* [PATCH 0/1] update bitbake to version 1.44.0
@ 2019-10-21 14:19 Cedric Hombourger
2019-10-21 14:19 ` [PATCH 1/1] bitbake: update " Cedric Hombourger
2019-11-08 8:05 ` [PATCH 0/1] update bitbake " Jan Kiszka
0 siblings, 2 replies; 11+ messages in thread
From: Cedric Hombourger @ 2019-10-21 14:19 UTC (permalink / raw)
To: isar-users; +Cc: Cedric Hombourger
It appears that the last update of Isar we did was last year. I would like to propose that
we re-align ourselves with upstream (this would fix some of the gitsm fetcher issues I have
been facing). No changes to bitbake were required, they however changed their API and
bitbake multiconfig:foo:bar no longer works (bitbake mc:foo:bar should be used instead).
Documentation and code using the old syntax was changed. The RECIPE-API-CHANGELOG document
was also changed. All changes were tested with our ci_build.sh script (no issues or failures
were observed).
Cedric Hombourger (1):
bitbake: update to version 1.44.0
RECIPE-API-CHANGELOG.md | 16 +
bitbake/.gitattributes | 2 +
bitbake/HEADER | 19 -
bitbake/LICENSE | 12 +-
bitbake/{COPYING => LICENSE.GPL-2.0-only} | 61 +-
bitbake/LICENSE.MIT | 25 +
bitbake/MANIFEST.in | 5 +-
bitbake/bin/bitbake | 16 +-
bitbake/bin/bitbake-diffsigs | 155 +-
bitbake/bin/bitbake-dumpsig | 95 +-
bitbake/bin/bitbake-hashclient | 170 ++
bitbake/bin/bitbake-hashserv | 62 +
bitbake/bin/bitbake-layers | 12 +-
bitbake/bin/bitbake-prserv | 4 +
bitbake/bin/bitbake-selftest | 16 +-
bitbake/bin/bitbake-worker | 19 +-
bitbake/bin/bitdoc | 14 +-
bitbake/bin/git-make-shallow | 4 +
bitbake/bin/toaster | 12 +-
bitbake/bin/toaster-eventreplay | 17 +-
bitbake/classes/base.bbclass | 2 +-
bitbake/contrib/dump_cache.py | 2 -
.../bitbake-user-manual-execution.xml | 78 +-
.../bitbake-user-manual-fetching.xml | 69 +-
.../bitbake-user-manual-hello.xml | 28 +-
.../bitbake-user-manual-intro.xml | 18 +-
.../bitbake-user-manual-metadata.xml | 179 +-
.../bitbake-user-manual-ref-variables.xml | 422 ++---
bitbake/doc/poky.ent | 8 -
bitbake/lib/bb/COW.py | 15 -
bitbake/lib/bb/__init__.py | 16 +-
bitbake/lib/bb/build.py | 151 +-
bitbake/lib/bb/cache.py | 109 +-
bitbake/lib/bb/cache_extra.py | 14 +-
bitbake/lib/bb/checksum.py | 12 +-
bitbake/lib/bb/codeparser.py | 13 +-
bitbake/lib/bb/command.py | 12 +-
bitbake/lib/bb/compat.py | 4 +
bitbake/lib/bb/cooker.py | 218 +--
bitbake/lib/bb/cookerdata.py | 44 +-
bitbake/lib/bb/daemonize.py | 4 +
bitbake/lib/bb/data.py | 23 +-
bitbake/lib/bb/data_smart.py | 25 +-
bitbake/lib/bb/event.py | 33 +-
bitbake/lib/bb/exceptions.py | 3 +
bitbake/lib/bb/fetch2/__init__.py | 55 +-
bitbake/lib/bb/fetch2/bzr.py | 12 +-
bitbake/lib/bb/fetch2/clearcase.py | 19 +-
bitbake/lib/bb/fetch2/cvs.py | 17 +-
bitbake/lib/bb/fetch2/git.py | 68 +-
bitbake/lib/bb/fetch2/gitannex.py | 14 +-
bitbake/lib/bb/fetch2/gitsm.py | 281 ++-
bitbake/lib/bb/fetch2/hg.py | 18 +-
bitbake/lib/bb/fetch2/local.py | 16 +-
bitbake/lib/bb/fetch2/npm.py | 69 +-
bitbake/lib/bb/fetch2/osc.py | 5 +-
bitbake/lib/bb/fetch2/perforce.py | 15 +-
bitbake/lib/bb/fetch2/repo.py | 16 +-
bitbake/lib/bb/fetch2/s3.py | 15 +-
bitbake/lib/bb/fetch2/sftp.py | 15 +-
bitbake/lib/bb/fetch2/ssh.py | 14 +-
bitbake/lib/bb/fetch2/svn.py | 98 +-
bitbake/lib/bb/fetch2/wget.py | 104 +-
bitbake/lib/bb/main.py | 41 +-
bitbake/lib/bb/methodpool.py | 15 +-
bitbake/lib/bb/monitordisk.py | 27 +-
bitbake/lib/bb/msg.py | 14 +-
bitbake/lib/bb/namedtuple_with_abc.py | 4 +-
bitbake/lib/bb/parse/__init__.py | 14 +-
bitbake/lib/bb/parse/ast.py | 15 +-
bitbake/lib/bb/parse/parse_py/BBHandler.py | 59 +-
bitbake/lib/bb/parse/parse_py/ConfHandler.py | 17 +-
bitbake/lib/bb/parse/parse_py/__init__.py | 17 +-
bitbake/lib/bb/persist_data.py | 234 ++-
bitbake/lib/bb/process.py | 4 +
bitbake/lib/bb/progress.py | 28 +-
bitbake/lib/bb/providers.py | 18 +-
bitbake/lib/bb/pysh/builtin.py | 710 --------
bitbake/lib/bb/pysh/interp.py | 1367 ---------------
bitbake/lib/bb/pysh/lsprof.py | 116 --
bitbake/lib/bb/pysh/pysh.py | 167 --
bitbake/lib/bb/pysh/pyshlex.py | 5 -
bitbake/lib/bb/pysh/pyshyacc.py | 17 +-
bitbake/lib/bb/pysh/sherrors.py | 26 -
bitbake/lib/bb/pysh/subprocess_fix.py | 77 -
bitbake/lib/bb/remotedata.py | 12 +-
bitbake/lib/bb/runqueue.py | 1523 ++++++++++-------
bitbake/lib/bb/server/__init__.py | 14 +-
bitbake/lib/bb/server/process.py | 123 +-
bitbake/lib/bb/server/xmlrpcclient.py | 12 +-
bitbake/lib/bb/server/xmlrpcserver.py | 12 +-
bitbake/lib/bb/siggen.py | 350 +++-
bitbake/lib/bb/taskdata.py | 17 +-
bitbake/lib/bb/tests/codeparser.py | 22 +-
bitbake/lib/bb/tests/cooker.py | 15 +-
bitbake/lib/bb/tests/cow.py | 17 +-
bitbake/lib/bb/tests/data.py | 39 +-
bitbake/lib/bb/tests/event.py | 23 +-
bitbake/lib/bb/tests/fetch.py | 308 +++-
bitbake/lib/bb/tests/parse.py | 33 +-
bitbake/lib/bb/tests/persist_data.py | 129 ++
.../tests/runqueue-tests/classes/base.bbclass | 262 +++
.../runqueue-tests/classes/image.bbclass | 5 +
.../runqueue-tests/classes/native.bbclass | 2 +
.../bb/tests/runqueue-tests/conf/bitbake.conf | 16 +
.../runqueue-tests/conf/multiconfig/mc1.conf | 1 +
.../runqueue-tests/conf/multiconfig/mc2.conf | 1 +
.../lib/bb/tests/runqueue-tests/recipes/a1.bb | 0
.../lib/bb/tests/runqueue-tests/recipes/b1.bb | 1 +
.../lib/bb/tests/runqueue-tests/recipes/c1.bb | 0
.../lib/bb/tests/runqueue-tests/recipes/d1.bb | 3 +
.../lib/bb/tests/runqueue-tests/recipes/e1.bb | 1 +
bitbake/lib/bb/tests/runqueue.py | 323 ++++
bitbake/lib/bb/tests/utils.py | 23 +-
bitbake/lib/bb/tinfoil.py | 12 +-
bitbake/lib/bb/ui/__init__.py | 12 +-
bitbake/lib/bb/ui/buildinfohelper.py | 22 +-
bitbake/lib/bb/ui/knotty.py | 60 +-
bitbake/lib/bb/ui/ncurses.py | 12 +-
bitbake/lib/bb/ui/taskexp.py | 12 +-
bitbake/lib/bb/ui/toasterui.py | 12 +-
bitbake/lib/bb/ui/uievent.py | 15 +-
bitbake/lib/bb/ui/uihelper.py | 16 +-
bitbake/lib/bb/utils.py | 49 +-
bitbake/lib/bblayers/__init__.py | 4 +
bitbake/lib/bblayers/action.py | 4 +
bitbake/lib/bblayers/common.py | 4 +
bitbake/lib/bblayers/layerindex.py | 6 +-
bitbake/lib/bblayers/query.py | 43 +-
bitbake/lib/bs4/dammit.py | 12 +-
bitbake/lib/bs4/element.py | 22 +-
bitbake/lib/hashserv/__init__.py | 93 +
bitbake/lib/hashserv/client.py | 157 ++
bitbake/lib/hashserv/server.py | 414 +++++
bitbake/lib/hashserv/tests.py | 142 ++
bitbake/lib/layerindexlib/__init__.py | 33 +-
bitbake/lib/layerindexlib/cooker.py | 12 +-
bitbake/lib/layerindexlib/plugin.py | 13 +-
bitbake/lib/layerindexlib/restapi.py | 12 +-
bitbake/lib/layerindexlib/tests/common.py | 12 +-
bitbake/lib/layerindexlib/tests/cooker.py | 12 +-
.../lib/layerindexlib/tests/layerindexobj.py | 12 +-
bitbake/lib/layerindexlib/tests/restapi.py | 12 +-
bitbake/lib/progressbar/__init__.py | 2 +
bitbake/lib/progressbar/compat.py | 2 +
bitbake/lib/progressbar/progressbar.py | 2 +
bitbake/lib/progressbar/widgets.py | 2 +
bitbake/lib/prserv/__init__.py | 4 +
bitbake/lib/prserv/db.py | 6 +-
bitbake/lib/prserv/serv.py | 4 +
bitbake/lib/pyinotify.py | 20 +-
bitbake/lib/toaster/bldcollector/admin.py | 4 +
bitbake/lib/toaster/bldcollector/urls.py | 13 +-
bitbake/lib/toaster/bldcollector/views.py | 12 +-
bitbake/lib/toaster/bldcontrol/admin.py | 4 +
.../lib/toaster/bldcontrol/bbcontroller.py | 16 +-
.../bldcontrol/localhostbecontroller.py | 16 +-
.../management/commands/checksettings.py | 4 +
.../management/commands/runbuilds.py | 4 +
bitbake/lib/toaster/bldcontrol/models.py | 4 +
bitbake/lib/toaster/bldcontrol/views.py | 4 +
bitbake/lib/toaster/manage.py | 4 +
bitbake/lib/toaster/orm/fixtures/oe-core.xml | 26 +-
bitbake/lib/toaster/orm/fixtures/poky.xml | 38 +-
.../orm/management/commands/lsupdates.py | 15 +-
bitbake/lib/toaster/orm/models.py | 25 +-
.../toaster/tests/browser/selenium_helpers.py | 17 +-
.../tests/browser/selenium_helpers_base.py | 17 +-
.../tests/browser/test_all_builds_page.py | 16 +-
.../tests/browser/test_all_projects_page.py | 16 +-
.../tests/browser/test_builddashboard_page.py | 16 +-
.../test_builddashboard_page_artifacts.py | 16 +-
.../test_builddashboard_page_recipes.py | 16 +-
.../browser/test_builddashboard_page_tasks.py | 16 +-
.../tests/browser/test_js_unit_tests.py | 16 +-
.../tests/browser/test_landing_page.py | 18 +-
.../tests/browser/test_layerdetails_page.py | 18 +-
.../browser/test_most_recent_builds_states.py | 18 +-
.../browser/test_new_custom_image_page.py | 16 +-
.../tests/browser/test_new_project_page.py | 16 +-
.../tests/browser/test_project_builds_page.py | 16 +-
.../tests/browser/test_project_config_page.py | 16 +-
.../tests/browser/test_project_page.py | 16 +-
.../lib/toaster/tests/browser/test_sample.py | 16 +-
.../toaster/tests/browser/test_task_page.py | 16 +-
.../tests/browser/test_toastertable_ui.py | 16 +-
bitbake/lib/toaster/tests/builds/buildtest.py | 16 +-
.../tests/builds/test_core_image_min.py | 17 +-
.../toaster/tests/commands/test_loaddata.py | 16 +-
.../toaster/tests/commands/test_lsupdates.py | 16 +-
.../toaster/tests/commands/test_runbuilds.py | 16 +-
bitbake/lib/toaster/tests/db/test_db.py | 2 +
.../lib/toaster/tests/eventreplay/__init__.py | 16 +-
.../tests/functional/functional_helpers.py | 16 +-
.../tests/functional/test_functional_basic.py | 16 +-
bitbake/lib/toaster/tests/views/test_views.py | 16 +-
bitbake/lib/toaster/toastergui/api.py | 13 +-
bitbake/lib/toaster/toastergui/buildtables.py | 15 +-
.../toastergui/static/js/importlayer.js | 12 +-
bitbake/lib/toaster/toastergui/tablefilter.py | 15 +-
bitbake/lib/toaster/toastergui/tables.py | 15 +-
.../templatetags/field_values_filter.py | 4 +
.../objects_to_dictionaries_filter.py | 4 +
.../templatetags/project_url_tag.py | 4 +
.../toastergui/templatetags/projecttags.py | 15 +-
bitbake/lib/toaster/toastergui/typeaheads.py | 12 +-
bitbake/lib/toaster/toastergui/urls.py | 12 +-
bitbake/lib/toaster/toastergui/views.py | 16 +-
bitbake/lib/toaster/toastergui/widgets.py | 15 +-
.../management/commands/builddelete.py | 4 +
.../management/commands/buildimport.py | 15 +-
.../management/commands/buildslist.py | 4 +
.../management/commands/checksocket.py | 16 +-
.../toastermain/management/commands/perf.py | 4 +
bitbake/lib/toaster/toastermain/settings.py | 15 +-
.../settings_production_example.py | 15 +-
.../lib/toaster/toastermain/settings_test.py | 15 +-
bitbake/lib/toaster/toastermain/urls.py | 15 +-
bitbake/lib/toaster/toastermain/wsgi.py | 7 +-
doc/user_manual.md | 38 +-
meta-isar/conf/conf-notes.txt | 6 +-
scripts/ci_build.sh | 62 +-
scripts/start_vm | 8 +-
testsuite/build_test/build_test.py | 2 +-
testsuite/start_vm.py | 2 +-
225 files changed, 5375 insertions(+), 6320 deletions(-)
create mode 100644 bitbake/.gitattributes
delete mode 100644 bitbake/HEADER
rename bitbake/{COPYING => LICENSE.GPL-2.0-only} (84%)
create mode 100644 bitbake/LICENSE.MIT
mode change 100755 => 120000 bitbake/bin/bitbake-dumpsig
create mode 100755 bitbake/bin/bitbake-hashclient
create mode 100755 bitbake/bin/bitbake-hashserv
delete mode 100644 bitbake/lib/bb/pysh/builtin.py
delete mode 100644 bitbake/lib/bb/pysh/interp.py
delete mode 100644 bitbake/lib/bb/pysh/lsprof.py
delete mode 100644 bitbake/lib/bb/pysh/pysh.py
delete mode 100644 bitbake/lib/bb/pysh/subprocess_fix.py
create mode 100644 bitbake/lib/bb/tests/persist_data.py
create mode 100644 bitbake/lib/bb/tests/runqueue-tests/classes/base.bbclass
create mode 100644 bitbake/lib/bb/tests/runqueue-tests/classes/image.bbclass
create mode 100644 bitbake/lib/bb/tests/runqueue-tests/classes/native.bbclass
create mode 100644 bitbake/lib/bb/tests/runqueue-tests/conf/bitbake.conf
create mode 100644 bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc1.conf
create mode 100644 bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc2.conf
create mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/a1.bb
create mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/b1.bb
create mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/c1.bb
create mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/d1.bb
create mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/e1.bb
create mode 100644 bitbake/lib/bb/tests/runqueue.py
create mode 100644 bitbake/lib/hashserv/__init__.py
create mode 100644 bitbake/lib/hashserv/client.py
create mode 100644 bitbake/lib/hashserv/server.py
create mode 100644 bitbake/lib/hashserv/tests.py
--
2.20.1
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH 1/1] bitbake: update to version 1.44.0
2019-10-21 14:19 [PATCH 0/1] update bitbake to version 1.44.0 Cedric Hombourger
@ 2019-10-21 14:19 ` Cedric Hombourger
2019-10-22 7:36 ` Henning Schild
2019-11-08 8:05 ` [PATCH 0/1] update bitbake " Jan Kiszka
1 sibling, 1 reply; 11+ messages in thread
From: Cedric Hombourger @ 2019-10-21 14:19 UTC (permalink / raw)
To: isar-users; +Cc: Cedric Hombourger
Found that the gitsm fetcher we ship is rather out-dated compared to
upstream and was failing to pull git sub-modules from repositories
such as azure-umqtt-c.
It shall be noted that bitbake has renamed its multiconfig targets:
the "multiconfig:" prefix was changed to "mc:". Updated our scripts
as well as documentation.
Origin: https://github.com/openembedded/bitbake.git
Commit 5d83d828cacb58ccb7c464e799c85fd2d2a50ccc (tag: 1.44.0)
Author: Richard Purdie <richard.purdie@linuxfoundation.org>
Date: Wed Oct 9 14:10:21 2019 +0100
bitbake: Update to version 1.44.0
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Cedric Hombourger <Cedric_Hombourger@mentor.com>
---
RECIPE-API-CHANGELOG.md | 16 +
bitbake/.gitattributes | 2 +
| 19 -
bitbake/LICENSE | 12 +-
bitbake/{COPYING => LICENSE.GPL-2.0-only} | 61 +-
bitbake/LICENSE.MIT | 25 +
bitbake/MANIFEST.in | 5 +-
bitbake/bin/bitbake | 16 +-
bitbake/bin/bitbake-diffsigs | 155 +-
bitbake/bin/bitbake-dumpsig | 95 +-
bitbake/bin/bitbake-hashclient | 170 ++
bitbake/bin/bitbake-hashserv | 62 +
bitbake/bin/bitbake-layers | 12 +-
bitbake/bin/bitbake-prserv | 4 +
bitbake/bin/bitbake-selftest | 16 +-
bitbake/bin/bitbake-worker | 19 +-
bitbake/bin/bitdoc | 14 +-
bitbake/bin/git-make-shallow | 4 +
bitbake/bin/toaster | 12 +-
bitbake/bin/toaster-eventreplay | 17 +-
bitbake/classes/base.bbclass | 2 +-
bitbake/contrib/dump_cache.py | 2 -
.../bitbake-user-manual-execution.xml | 78 +-
.../bitbake-user-manual-fetching.xml | 69 +-
.../bitbake-user-manual-hello.xml | 28 +-
.../bitbake-user-manual-intro.xml | 18 +-
.../bitbake-user-manual-metadata.xml | 179 +-
.../bitbake-user-manual-ref-variables.xml | 422 ++---
bitbake/doc/poky.ent | 8 -
bitbake/lib/bb/COW.py | 15 -
bitbake/lib/bb/__init__.py | 16 +-
bitbake/lib/bb/build.py | 151 +-
bitbake/lib/bb/cache.py | 109 +-
| 14 +-
bitbake/lib/bb/checksum.py | 12 +-
bitbake/lib/bb/codeparser.py | 13 +-
bitbake/lib/bb/command.py | 12 +-
bitbake/lib/bb/compat.py | 4 +
bitbake/lib/bb/cooker.py | 218 +--
bitbake/lib/bb/cookerdata.py | 44 +-
bitbake/lib/bb/daemonize.py | 4 +
bitbake/lib/bb/data.py | 23 +-
bitbake/lib/bb/data_smart.py | 25 +-
bitbake/lib/bb/event.py | 33 +-
bitbake/lib/bb/exceptions.py | 3 +
bitbake/lib/bb/fetch2/__init__.py | 55 +-
bitbake/lib/bb/fetch2/bzr.py | 12 +-
bitbake/lib/bb/fetch2/clearcase.py | 19 +-
bitbake/lib/bb/fetch2/cvs.py | 17 +-
bitbake/lib/bb/fetch2/git.py | 68 +-
bitbake/lib/bb/fetch2/gitannex.py | 14 +-
bitbake/lib/bb/fetch2/gitsm.py | 281 ++-
bitbake/lib/bb/fetch2/hg.py | 18 +-
bitbake/lib/bb/fetch2/local.py | 16 +-
bitbake/lib/bb/fetch2/npm.py | 69 +-
bitbake/lib/bb/fetch2/osc.py | 5 +-
bitbake/lib/bb/fetch2/perforce.py | 15 +-
bitbake/lib/bb/fetch2/repo.py | 16 +-
bitbake/lib/bb/fetch2/s3.py | 15 +-
bitbake/lib/bb/fetch2/sftp.py | 15 +-
bitbake/lib/bb/fetch2/ssh.py | 14 +-
bitbake/lib/bb/fetch2/svn.py | 98 +-
bitbake/lib/bb/fetch2/wget.py | 104 +-
bitbake/lib/bb/main.py | 41 +-
bitbake/lib/bb/methodpool.py | 15 +-
bitbake/lib/bb/monitordisk.py | 27 +-
bitbake/lib/bb/msg.py | 14 +-
bitbake/lib/bb/namedtuple_with_abc.py | 4 +-
bitbake/lib/bb/parse/__init__.py | 14 +-
bitbake/lib/bb/parse/ast.py | 15 +-
bitbake/lib/bb/parse/parse_py/BBHandler.py | 59 +-
bitbake/lib/bb/parse/parse_py/ConfHandler.py | 17 +-
bitbake/lib/bb/parse/parse_py/__init__.py | 17 +-
bitbake/lib/bb/persist_data.py | 234 ++-
bitbake/lib/bb/process.py | 4 +
bitbake/lib/bb/progress.py | 28 +-
bitbake/lib/bb/providers.py | 18 +-
bitbake/lib/bb/pysh/builtin.py | 710 --------
bitbake/lib/bb/pysh/interp.py | 1367 ---------------
bitbake/lib/bb/pysh/lsprof.py | 116 --
bitbake/lib/bb/pysh/pysh.py | 167 --
bitbake/lib/bb/pysh/pyshlex.py | 5 -
bitbake/lib/bb/pysh/pyshyacc.py | 17 +-
bitbake/lib/bb/pysh/sherrors.py | 26 -
bitbake/lib/bb/pysh/subprocess_fix.py | 77 -
bitbake/lib/bb/remotedata.py | 12 +-
bitbake/lib/bb/runqueue.py | 1523 ++++++++++-------
bitbake/lib/bb/server/__init__.py | 14 +-
bitbake/lib/bb/server/process.py | 123 +-
bitbake/lib/bb/server/xmlrpcclient.py | 12 +-
bitbake/lib/bb/server/xmlrpcserver.py | 12 +-
bitbake/lib/bb/siggen.py | 350 +++-
bitbake/lib/bb/taskdata.py | 17 +-
bitbake/lib/bb/tests/codeparser.py | 22 +-
bitbake/lib/bb/tests/cooker.py | 15 +-
bitbake/lib/bb/tests/cow.py | 17 +-
bitbake/lib/bb/tests/data.py | 39 +-
bitbake/lib/bb/tests/event.py | 23 +-
bitbake/lib/bb/tests/fetch.py | 308 +++-
bitbake/lib/bb/tests/parse.py | 33 +-
bitbake/lib/bb/tests/persist_data.py | 129 ++
.../tests/runqueue-tests/classes/base.bbclass | 262 +++
.../runqueue-tests/classes/image.bbclass | 5 +
.../runqueue-tests/classes/native.bbclass | 2 +
.../bb/tests/runqueue-tests/conf/bitbake.conf | 16 +
.../runqueue-tests/conf/multiconfig/mc1.conf | 1 +
.../runqueue-tests/conf/multiconfig/mc2.conf | 1 +
.../lib/bb/tests/runqueue-tests/recipes/a1.bb | 0
.../lib/bb/tests/runqueue-tests/recipes/b1.bb | 1 +
.../lib/bb/tests/runqueue-tests/recipes/c1.bb | 0
.../lib/bb/tests/runqueue-tests/recipes/d1.bb | 3 +
.../lib/bb/tests/runqueue-tests/recipes/e1.bb | 1 +
bitbake/lib/bb/tests/runqueue.py | 323 ++++
bitbake/lib/bb/tests/utils.py | 23 +-
bitbake/lib/bb/tinfoil.py | 12 +-
bitbake/lib/bb/ui/__init__.py | 12 +-
bitbake/lib/bb/ui/buildinfohelper.py | 22 +-
bitbake/lib/bb/ui/knotty.py | 60 +-
bitbake/lib/bb/ui/ncurses.py | 12 +-
bitbake/lib/bb/ui/taskexp.py | 12 +-
bitbake/lib/bb/ui/toasterui.py | 12 +-
bitbake/lib/bb/ui/uievent.py | 15 +-
bitbake/lib/bb/ui/uihelper.py | 16 +-
bitbake/lib/bb/utils.py | 49 +-
bitbake/lib/bblayers/__init__.py | 4 +
bitbake/lib/bblayers/action.py | 4 +
bitbake/lib/bblayers/common.py | 4 +
bitbake/lib/bblayers/layerindex.py | 6 +-
bitbake/lib/bblayers/query.py | 43 +-
bitbake/lib/bs4/dammit.py | 12 +-
bitbake/lib/bs4/element.py | 22 +-
bitbake/lib/hashserv/__init__.py | 93 +
bitbake/lib/hashserv/client.py | 157 ++
bitbake/lib/hashserv/server.py | 414 +++++
bitbake/lib/hashserv/tests.py | 142 ++
bitbake/lib/layerindexlib/__init__.py | 33 +-
bitbake/lib/layerindexlib/cooker.py | 12 +-
bitbake/lib/layerindexlib/plugin.py | 13 +-
bitbake/lib/layerindexlib/restapi.py | 12 +-
bitbake/lib/layerindexlib/tests/common.py | 12 +-
bitbake/lib/layerindexlib/tests/cooker.py | 12 +-
.../lib/layerindexlib/tests/layerindexobj.py | 12 +-
bitbake/lib/layerindexlib/tests/restapi.py | 12 +-
bitbake/lib/progressbar/__init__.py | 2 +
bitbake/lib/progressbar/compat.py | 2 +
bitbake/lib/progressbar/progressbar.py | 2 +
bitbake/lib/progressbar/widgets.py | 2 +
bitbake/lib/prserv/__init__.py | 4 +
bitbake/lib/prserv/db.py | 6 +-
bitbake/lib/prserv/serv.py | 4 +
bitbake/lib/pyinotify.py | 20 +-
bitbake/lib/toaster/bldcollector/admin.py | 4 +
bitbake/lib/toaster/bldcollector/urls.py | 13 +-
bitbake/lib/toaster/bldcollector/views.py | 12 +-
bitbake/lib/toaster/bldcontrol/admin.py | 4 +
.../lib/toaster/bldcontrol/bbcontroller.py | 16 +-
.../bldcontrol/localhostbecontroller.py | 16 +-
.../management/commands/checksettings.py | 4 +
.../management/commands/runbuilds.py | 4 +
bitbake/lib/toaster/bldcontrol/models.py | 4 +
bitbake/lib/toaster/bldcontrol/views.py | 4 +
bitbake/lib/toaster/manage.py | 4 +
bitbake/lib/toaster/orm/fixtures/oe-core.xml | 26 +-
bitbake/lib/toaster/orm/fixtures/poky.xml | 38 +-
.../orm/management/commands/lsupdates.py | 15 +-
bitbake/lib/toaster/orm/models.py | 25 +-
.../toaster/tests/browser/selenium_helpers.py | 17 +-
.../tests/browser/selenium_helpers_base.py | 17 +-
.../tests/browser/test_all_builds_page.py | 16 +-
.../tests/browser/test_all_projects_page.py | 16 +-
.../tests/browser/test_builddashboard_page.py | 16 +-
.../test_builddashboard_page_artifacts.py | 16 +-
.../test_builddashboard_page_recipes.py | 16 +-
.../browser/test_builddashboard_page_tasks.py | 16 +-
.../tests/browser/test_js_unit_tests.py | 16 +-
.../tests/browser/test_landing_page.py | 18 +-
.../tests/browser/test_layerdetails_page.py | 18 +-
.../browser/test_most_recent_builds_states.py | 18 +-
.../browser/test_new_custom_image_page.py | 16 +-
.../tests/browser/test_new_project_page.py | 16 +-
.../tests/browser/test_project_builds_page.py | 16 +-
.../tests/browser/test_project_config_page.py | 16 +-
.../tests/browser/test_project_page.py | 16 +-
.../lib/toaster/tests/browser/test_sample.py | 16 +-
.../toaster/tests/browser/test_task_page.py | 16 +-
.../tests/browser/test_toastertable_ui.py | 16 +-
bitbake/lib/toaster/tests/builds/buildtest.py | 16 +-
.../tests/builds/test_core_image_min.py | 17 +-
.../toaster/tests/commands/test_loaddata.py | 16 +-
.../toaster/tests/commands/test_lsupdates.py | 16 +-
.../toaster/tests/commands/test_runbuilds.py | 16 +-
bitbake/lib/toaster/tests/db/test_db.py | 2 +
.../lib/toaster/tests/eventreplay/__init__.py | 16 +-
.../tests/functional/functional_helpers.py | 16 +-
.../tests/functional/test_functional_basic.py | 16 +-
bitbake/lib/toaster/tests/views/test_views.py | 16 +-
bitbake/lib/toaster/toastergui/api.py | 13 +-
bitbake/lib/toaster/toastergui/buildtables.py | 15 +-
.../toastergui/static/js/importlayer.js | 12 +-
bitbake/lib/toaster/toastergui/tablefilter.py | 15 +-
bitbake/lib/toaster/toastergui/tables.py | 15 +-
.../templatetags/field_values_filter.py | 4 +
.../objects_to_dictionaries_filter.py | 4 +
.../templatetags/project_url_tag.py | 4 +
.../toastergui/templatetags/projecttags.py | 15 +-
bitbake/lib/toaster/toastergui/typeaheads.py | 12 +-
bitbake/lib/toaster/toastergui/urls.py | 12 +-
bitbake/lib/toaster/toastergui/views.py | 16 +-
bitbake/lib/toaster/toastergui/widgets.py | 15 +-
.../management/commands/builddelete.py | 4 +
.../management/commands/buildimport.py | 15 +-
.../management/commands/buildslist.py | 4 +
.../management/commands/checksocket.py | 16 +-
.../toastermain/management/commands/perf.py | 4 +
bitbake/lib/toaster/toastermain/settings.py | 15 +-
.../settings_production_example.py | 15 +-
.../lib/toaster/toastermain/settings_test.py | 15 +-
bitbake/lib/toaster/toastermain/urls.py | 15 +-
bitbake/lib/toaster/toastermain/wsgi.py | 7 +-
doc/user_manual.md | 38 +-
meta-isar/conf/conf-notes.txt | 6 +-
scripts/ci_build.sh | 62 +-
scripts/start_vm | 8 +-
testsuite/build_test/build_test.py | 2 +-
testsuite/start_vm.py | 2 +-
225 files changed, 5375 insertions(+), 6320 deletions(-)
create mode 100644 bitbake/.gitattributes
delete mode 100644 bitbake/HEADER
rename bitbake/{COPYING => LICENSE.GPL-2.0-only} (84%)
create mode 100644 bitbake/LICENSE.MIT
mode change 100755 => 120000 bitbake/bin/bitbake-dumpsig
create mode 100755 bitbake/bin/bitbake-hashclient
create mode 100755 bitbake/bin/bitbake-hashserv
delete mode 100644 bitbake/lib/bb/pysh/builtin.py
delete mode 100644 bitbake/lib/bb/pysh/interp.py
delete mode 100644 bitbake/lib/bb/pysh/lsprof.py
delete mode 100644 bitbake/lib/bb/pysh/pysh.py
delete mode 100644 bitbake/lib/bb/pysh/subprocess_fix.py
create mode 100644 bitbake/lib/bb/tests/persist_data.py
create mode 100644 bitbake/lib/bb/tests/runqueue-tests/classes/base.bbclass
create mode 100644 bitbake/lib/bb/tests/runqueue-tests/classes/image.bbclass
create mode 100644 bitbake/lib/bb/tests/runqueue-tests/classes/native.bbclass
create mode 100644 bitbake/lib/bb/tests/runqueue-tests/conf/bitbake.conf
create mode 100644 bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc1.conf
create mode 100644 bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc2.conf
create mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/a1.bb
create mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/b1.bb
create mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/c1.bb
create mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/d1.bb
create mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/e1.bb
create mode 100644 bitbake/lib/bb/tests/runqueue.py
create mode 100644 bitbake/lib/hashserv/__init__.py
create mode 100644 bitbake/lib/hashserv/client.py
create mode 100644 bitbake/lib/hashserv/server.py
create mode 100644 bitbake/lib/hashserv/tests.py
diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index bbef1a3..81ec430 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -176,3 +176,19 @@ Otherwise set a encrypted root password like this:
USERS += "root"
USER_root[password] = "$6$rounds=10000$RXeWrnFmkY$DtuS/OmsAS2cCEDo0BF5qQsizIrq6jPgXnwv3PHqREJeKd1sXdHX/ayQtuQWVDHe0KIO0/sVH8dvQm1KthF0d/"
```
+
+### multiconfig build targets were renamed
+
+bitbake was upgraded to version 1.44.0 where "multiconfig" build targets were
+renamed "mc". As an example, builds for the qemuarm-stretch machine should now
+be done as follows:
+
+```
+bitbake mc:qemuarm-stretch:isar-image-base
+```
+
+The old syntax is no longer supported and will produce an error:
+
+```
+bitbake multiconfig:qemuarm-stretch:isar-image-base
+```
diff --git a/bitbake/.gitattributes b/bitbake/.gitattributes
new file mode 100644
index 0000000..e4f8f62
--- /dev/null
+++ b/bitbake/.gitattributes
@@ -0,0 +1,2 @@
+*min.js binary
+*min.css binary
diff --git a/bitbake/HEADER b/bitbake/HEADER
deleted file mode 100644
index 9859255..0000000
--- a/bitbake/HEADER
+++ /dev/null
@@ -1,19 +0,0 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
-#
-# <one line to give the program's name and a brief idea of what it does.>
-# Copyright (C) <year> <name of author>
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
diff --git a/bitbake/LICENSE b/bitbake/LICENSE
index 7d4e5f4..8458042 100644
--- a/bitbake/LICENSE
+++ b/bitbake/LICENSE
@@ -1,4 +1,13 @@
-BitBake is licensed under the GNU General Public License version 2.0. See COPYING for further details.
+BitBake is licensed under the GNU General Public License version 2.0. See
+LICENSE.GPL-2.0-only for further details.
+
+Individual files contain the following style tags instead of the full license text:
+
+ SPDX-License-Identifier: GPL-2.0-only
+
+This enables machine processing of license information based on the SPDX
+License Identifiers that are here available: http://spdx.org/licenses/
+
The following external components are distributed with this software:
@@ -17,3 +26,4 @@ Foundation and individual contributors.
* Font Awesome fonts redistributed under the SIL Open Font License 1.1
* simplediff is distributed under the zlib license.
+
diff --git a/bitbake/COPYING b/bitbake/LICENSE.GPL-2.0-only
similarity index 84%
rename from bitbake/COPYING
rename to bitbake/LICENSE.GPL-2.0-only
index d511905..5db3c0a 100644
--- a/bitbake/COPYING
+++ b/bitbake/LICENSE.GPL-2.0-only
@@ -279,61 +279,10 @@ POSSIBILITY OF SUCH DAMAGES.
END OF TERMS AND CONDITIONS
- How to Apply These Terms to Your New Programs
+Note:
+Individual files contain the following tag instead of the full license text.
- If you develop a new program, and you want it to be of the greatest
-possible use to the public, the best way to achieve this is to make it
-free software which everyone can redistribute and change under these terms.
+ SPDX-License-Identifier: GPL-2.0-only
- To do so, attach the following notices to the program. It is safest
-to attach them to the start of each source file to most effectively
-convey the exclusion of warranty; and each file should have at least
-the "copyright" line and a pointer to where the full notice is found.
-
- <one line to give the program's name and a brief idea of what it does.>
- Copyright (C) <year> <name of author>
-
- This program is free software; you can redistribute it and/or modify
- it under the terms of the GNU General Public License as published by
- the Free Software Foundation; either version 2 of the License, or
- (at your option) any later version.
-
- This program is distributed in the hope that it will be useful,
- but WITHOUT ANY WARRANTY; without even the implied warranty of
- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- GNU General Public License for more details.
-
- You should have received a copy of the GNU General Public License along
- with this program; if not, write to the Free Software Foundation, Inc.,
- 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-Also add information on how to contact you by electronic and paper mail.
-
-If the program is interactive, make it output a short notice like this
-when it starts in an interactive mode:
-
- Gnomovision version 69, Copyright (C) year name of author
- Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
- This is free software, and you are welcome to redistribute it
- under certain conditions; type `show c' for details.
-
-The hypothetical commands `show w' and `show c' should show the appropriate
-parts of the General Public License. Of course, the commands you use may
-be called something other than `show w' and `show c'; they could even be
-mouse-clicks or menu items--whatever suits your program.
-
-You should also get your employer (if you work as a programmer) or your
-school, if any, to sign a "copyright disclaimer" for the program, if
-necessary. Here is a sample; alter the names:
-
- Yoyodyne, Inc., hereby disclaims all copyright interest in the program
- `Gnomovision' (which makes passes at compilers) written by James Hacker.
-
- <signature of Ty Coon>, 1 April 1989
- Ty Coon, President of Vice
-
-This General Public License does not permit incorporating your program into
-proprietary programs. If your program is a subroutine library, you may
-consider it more useful to permit linking proprietary applications with the
-library. If this is what you want to do, use the GNU Lesser General
-Public License instead of this License.
+This enables machine processing of license information based on the SPDX
+License Identifiers that are here available: http://spdx.org/licenses/
diff --git a/bitbake/LICENSE.MIT b/bitbake/LICENSE.MIT
new file mode 100644
index 0000000..a6919eb
--- /dev/null
+++ b/bitbake/LICENSE.MIT
@@ -0,0 +1,25 @@
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+Note:
+Individual files contain the following tag instead of the full license text.
+
+ SPDX-License-Identifier: MIT
+
+This enables machine processing of license information based on the SPDX
+License Identifiers that are here available: http://spdx.org/licenses/
diff --git a/bitbake/MANIFEST.in b/bitbake/MANIFEST.in
index b197378..f24969a 100644
--- a/bitbake/MANIFEST.in
+++ b/bitbake/MANIFEST.in
@@ -1,6 +1,8 @@
-include COPYING
include ChangeLog
include AUTHORS
+include LICENSE
+include LICENSE.GPL-2.0-only
+include LICENSE.MIT
include contrib/*
include contrib/vim/*/*
include conf/*
@@ -8,4 +10,3 @@ include classes/*
include doc/*
include doc/manual/*
include ez_setup.py
-include HEADER
diff --git a/bitbake/bin/bitbake b/bitbake/bin/bitbake
index 57dec2a..66d08f8 100755
--- a/bitbake/bin/bitbake
+++ b/bitbake/bin/bitbake
@@ -1,6 +1,4 @@
#!/usr/bin/env python3
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# Copyright (C) 2003, 2004 Chris Larson
# Copyright (C) 2003, 2004 Phil Blundell
@@ -9,18 +7,8 @@
# Copyright (C) 2005 ROAD GmbH
# Copyright (C) 2006 Richard Purdie
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import os
import sys
@@ -38,7 +26,7 @@ from bb.main import bitbake_main, BitBakeConfigParameters, BBMainException
if sys.getfilesystemencoding() != "utf-8":
sys.exit("Please use a locale setting which supports UTF-8 (such as LANG=en_US.UTF-8).\nPython can't change the filesystem locale after loading so we need a UTF-8 when Python starts or things won't work.")
-__version__ = "1.40.0"
+__version__ = "1.44.0"
if __name__ == "__main__":
if __version__ != bb.__version__:
diff --git a/bitbake/bin/bitbake-diffsigs b/bitbake/bin/bitbake-diffsigs
index 4e6bbdd..19420a2 100755
--- a/bitbake/bin/bitbake-diffsigs
+++ b/bitbake/bin/bitbake-diffsigs
@@ -1,27 +1,16 @@
#!/usr/bin/env python3
-# bitbake-diffsigs
-# BitBake task signature data comparison utility
+# bitbake-diffsigs / bitbake-dumpsig
+# BitBake task signature data dump and comparison utility
#
# Copyright (C) 2012-2013, 2017 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import os
import sys
import warnings
-import fnmatch
import argparse
import logging
import pickle
@@ -32,7 +21,10 @@ import bb.tinfoil
import bb.siggen
import bb.msg
-logger = bb.msg.logger_create('bitbake-diffsigs')
+myname = os.path.basename(sys.argv[0])
+logger = bb.msg.logger_create(myname)
+
+is_dump = myname == 'bitbake-dumpsig'
def find_siginfo(tinfoil, pn, taskname, sigs=None):
result = None
@@ -59,8 +51,8 @@ def find_siginfo(tinfoil, pn, taskname, sigs=None):
sys.exit(2)
return result
-def find_compare_task(bbhandler, pn, taskname, sig1=None, sig2=None, color=False):
- """ Find the most recent signature files for the specified PN/task and compare them """
+def find_siginfo_task(bbhandler, pn, taskname, sig1=None, sig2=None):
+ """ Find the most recent signature files for the specified PN/task """
if not taskname.startswith('do_'):
taskname = 'do_%s' % taskname
@@ -79,73 +71,81 @@ def find_compare_task(bbhandler, pn, taskname, sig1=None, sig2=None, color=False
latestfiles = [sigfiles[sig1], sigfiles[sig2]]
else:
filedates = find_siginfo(bbhandler, pn, taskname)
- latestfiles = sorted(filedates.keys(), key=lambda f: filedates[f])[-3:]
+ latestfiles = sorted(filedates.keys(), key=lambda f: filedates[f])[-2:]
if not latestfiles:
logger.error('No sigdata files found matching %s %s' % (pn, taskname))
sys.exit(1)
- elif len(latestfiles) < 2:
- logger.error('Only one matching sigdata file found for the specified task (%s %s)' % (pn, taskname))
- sys.exit(1)
- # Define recursion callback
- def recursecb(key, hash1, hash2):
- hashes = [hash1, hash2]
- hashfiles = find_siginfo(bbhandler, key, None, hashes)
-
- recout = []
- if len(hashfiles) == 0:
- recout.append("Unable to find matching sigdata for %s with hashes %s or %s" % (key, hash1, hash2))
- elif not hash1 in hashfiles:
- recout.append("Unable to find matching sigdata for %s with hash %s" % (key, hash1))
- elif not hash2 in hashfiles:
- recout.append("Unable to find matching sigdata for %s with hash %s" % (key, hash2))
- else:
- out2 = bb.siggen.compare_sigfiles(hashfiles[hash1], hashfiles[hash2], recursecb, color=color)
- for change in out2:
- for line in change.splitlines():
- recout.append(' ' + line)
+ return latestfiles
+
- return recout
+# Define recursion callback
+def recursecb(key, hash1, hash2):
+ hashes = [hash1, hash2]
+ hashfiles = find_siginfo(tinfoil, key, None, hashes)
- # Recurse into signature comparison
- logger.debug("Signature file (previous): %s" % latestfiles[-2])
- logger.debug("Signature file (latest): %s" % latestfiles[-1])
- output = bb.siggen.compare_sigfiles(latestfiles[-2], latestfiles[-1], recursecb, color=color)
- if output:
- print('\n'.join(output))
- sys.exit(0)
+ recout = []
+ if len(hashfiles) == 0:
+ recout.append("Unable to find matching sigdata for %s with hashes %s or %s" % (key, hash1, hash2))
+ elif not hash1 in hashfiles:
+ recout.append("Unable to find matching sigdata for %s with hash %s" % (key, hash1))
+ elif not hash2 in hashfiles:
+ recout.append("Unable to find matching sigdata for %s with hash %s" % (key, hash2))
+ else:
+ out2 = bb.siggen.compare_sigfiles(hashfiles[hash1], hashfiles[hash2], recursecb, color=color)
+ for change in out2:
+ for line in change.splitlines():
+ recout.append(' ' + line)
+ return recout
parser = argparse.ArgumentParser(
- description="Compares siginfo/sigdata files written out by BitBake")
+ description=("Dumps" if is_dump else "Compares") + " siginfo/sigdata files written out by BitBake")
-parser.add_argument('-d', '--debug',
+parser.add_argument('-D', '--debug',
help='Enable debug output',
action='store_true')
-parser.add_argument('--color',
- help='Colorize output (where %(metavar)s is %(choices)s)',
- choices=['auto', 'always', 'never'], default='auto', metavar='color')
+if is_dump:
+ parser.add_argument("-t", "--task",
+ help="find the signature data file for the last run of the specified task",
+ action="store", dest="taskargs", nargs=2, metavar=('recipename', 'taskname'))
+
+ parser.add_argument("sigdatafile1",
+ help="Signature file to dump. Not used when using -t/--task.",
+ action="store", nargs='?', metavar="sigdatafile")
+else:
+ parser.add_argument('-c', '--color',
+ help='Colorize the output (where %(metavar)s is %(choices)s)',
+ choices=['auto', 'always', 'never'], default='auto', metavar='color')
-parser.add_argument("-t", "--task",
- help="find the signature data files for last two runs of the specified task and compare them",
- action="store", dest="taskargs", nargs=2, metavar=('recipename', 'taskname'))
+ parser.add_argument('-d', '--dump',
+ help='Dump the last signature data instead of comparing (equivalent to using bitbake-dumpsig)',
+ action='store_true')
-parser.add_argument("-s", "--signature",
- help="With -t/--task, specify the signatures to look for instead of taking the last two",
- action="store", dest="sigargs", nargs=2, metavar=('fromsig', 'tosig'))
+ parser.add_argument("-t", "--task",
+ help="find the signature data files for the last two runs of the specified task and compare them",
+ action="store", dest="taskargs", nargs=2, metavar=('recipename', 'taskname'))
-parser.add_argument("sigdatafile1",
- help="First signature file to compare (or signature file to dump, if second not specified). Not used when using -t/--task.",
- action="store", nargs='?')
+ parser.add_argument("-s", "--signature",
+ help="With -t/--task, specify the signatures to look for instead of taking the last two",
+ action="store", dest="sigargs", nargs=2, metavar=('fromsig', 'tosig'))
-parser.add_argument("sigdatafile2",
- help="Second signature file to compare",
- action="store", nargs='?')
+ parser.add_argument("sigdatafile1",
+ help="First signature file to compare (or signature file to dump, if second not specified). Not used when using -t/--task.",
+ action="store", nargs='?')
+ parser.add_argument("sigdatafile2",
+ help="Second signature file to compare",
+ action="store", nargs='?')
options = parser.parse_args()
+if is_dump:
+ options.color = 'never'
+ options.dump = True
+ options.sigdatafile2 = None
+ options.sigargs = None
if options.debug:
logger.setLevel(logging.DEBUG)
@@ -155,17 +155,32 @@ color = (options.color == 'always' or (options.color == 'auto' and sys.stdout.is
if options.taskargs:
with bb.tinfoil.Tinfoil() as tinfoil:
tinfoil.prepare(config_only=True)
- if options.sigargs:
- find_compare_task(tinfoil, options.taskargs[0], options.taskargs[1], options.sigargs[0], options.sigargs[1], color=color)
+ if not options.dump and options.sigargs:
+ files = find_siginfo_task(tinfoil, options.taskargs[0], options.taskargs[1], options.sigargs[0], options.sigargs[1])
+ else:
+ files = find_siginfo_task(tinfoil, options.taskargs[0], options.taskargs[1])
+
+ if options.dump:
+ logger.debug("Signature file: %s" % files[-1])
+ output = bb.siggen.dump_sigfile(files[-1])
else:
- find_compare_task(tinfoil, options.taskargs[0], options.taskargs[1], color=color)
+ if len(files) < 2:
+ logger.error('Only one matching sigdata file found for the specified task (%s %s)' % (options.taskargs[0], options.taskargs[1]))
+ sys.exit(1)
+
+ # Recurse into signature comparison
+ logger.debug("Signature file (previous): %s" % files[-2])
+ logger.debug("Signature file (latest): %s" % files[-1])
+ output = bb.siggen.compare_sigfiles(files[-2], files[-1], recursecb, color=color)
else:
if options.sigargs:
logger.error('-s/--signature can only be used together with -t/--task')
sys.exit(1)
try:
- if options.sigdatafile1 and options.sigdatafile2:
- output = bb.siggen.compare_sigfiles(options.sigdatafile1, options.sigdatafile2, color=color)
+ if not options.dump and options.sigdatafile1 and options.sigdatafile2:
+ with bb.tinfoil.Tinfoil() as tinfoil:
+ tinfoil.prepare(config_only=True)
+ output = bb.siggen.compare_sigfiles(options.sigdatafile1, options.sigdatafile2, recursecb, color=color)
elif options.sigdatafile1:
output = bb.siggen.dump_sigfile(options.sigdatafile1)
else:
@@ -179,5 +194,5 @@ else:
logger.error('Invalid signature data - ensure you are specifying sigdata/siginfo files')
sys.exit(1)
- if output:
- print('\n'.join(output))
+if output:
+ print('\n'.join(output))
diff --git a/bitbake/bin/bitbake-dumpsig b/bitbake/bin/bitbake-dumpsig
deleted file mode 100755
index 95ebd93..0000000
--- a/bitbake/bin/bitbake-dumpsig
+++ /dev/null
@@ -1,94 +0,0 @@
-#!/usr/bin/env python3
-
-# bitbake-dumpsig
-# BitBake task signature dump utility
-#
-# Copyright (C) 2013 Intel Corporation
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-import os
-import sys
-import warnings
-import optparse
-import logging
-import pickle
-
-sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(sys.argv[0])), 'lib'))
-
-import bb.tinfoil
-import bb.siggen
-import bb.msg
-
-logger = bb.msg.logger_create('bitbake-dumpsig')
-
-def find_siginfo_task(bbhandler, pn, taskname):
- """ Find the most recent signature file for the specified PN/task """
-
- if not hasattr(bb.siggen, 'find_siginfo'):
- logger.error('Metadata does not support finding signature data files')
- sys.exit(1)
-
- if not taskname.startswith('do_'):
- taskname = 'do_%s' % taskname
-
- filedates = bb.siggen.find_siginfo(pn, taskname, None, bbhandler.config_data)
- latestfiles = sorted(filedates.keys(), key=lambda f: filedates[f])[-1:]
- if not latestfiles:
- logger.error('No sigdata files found matching %s %s' % (pn, taskname))
- sys.exit(1)
-
- return latestfiles[0]
-
-parser = optparse.OptionParser(
- description = "Dumps siginfo/sigdata files written out by BitBake",
- usage = """
- %prog -t recipename taskname
- %prog sigdatafile""")
-
-parser.add_option("-D", "--debug",
- help = "enable debug",
- action = "store_true", dest="debug", default = False)
-
-parser.add_option("-t", "--task",
- help = "find the signature data file for the specified task",
- action="store", dest="taskargs", nargs=2, metavar='recipename taskname')
-
-options, args = parser.parse_args(sys.argv)
-
-if options.debug:
- logger.setLevel(logging.DEBUG)
-
-if options.taskargs:
- tinfoil = bb.tinfoil.Tinfoil()
- tinfoil.prepare(config_only = True)
- file = find_siginfo_task(tinfoil, options.taskargs[0], options.taskargs[1])
- logger.debug("Signature file: %s" % file)
-elif len(args) == 1:
- parser.print_help()
- sys.exit(0)
-else:
- file = args[1]
-
-try:
- output = bb.siggen.dump_sigfile(file)
-except IOError as e:
- logger.error(str(e))
- sys.exit(1)
-except (pickle.UnpicklingError, EOFError):
- logger.error('Invalid signature data - ensure you are specifying a sigdata/siginfo file')
- sys.exit(1)
-
-if output:
- print('\n'.join(output))
diff --git a/bitbake/bin/bitbake-dumpsig b/bitbake/bin/bitbake-dumpsig
new file mode 120000
index 0000000..b1e8489
--- /dev/null
+++ b/bitbake/bin/bitbake-dumpsig
@@ -0,0 +1 @@
+bitbake-diffsigs
\ No newline at end of file
diff --git a/bitbake/bin/bitbake-hashclient b/bitbake/bin/bitbake-hashclient
new file mode 100755
index 0000000..29ab65f
--- /dev/null
+++ b/bitbake/bin/bitbake-hashclient
@@ -0,0 +1,170 @@
+#! /usr/bin/env python3
+#
+# Copyright (C) 2019 Garmin Ltd.
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+import argparse
+import hashlib
+import logging
+import os
+import pprint
+import sys
+import threading
+import time
+
+try:
+ import tqdm
+ ProgressBar = tqdm.tqdm
+except ImportError:
+ class ProgressBar(object):
+ def __init__(self, *args, **kwargs):
+ pass
+
+ def __enter__(self):
+ return self
+
+ def __exit__(self, *args, **kwargs):
+ pass
+
+ def update(self):
+ pass
+
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib'))
+
+import hashserv
+
+DEFAULT_ADDRESS = 'unix://./hashserve.sock'
+METHOD = 'stress.test.method'
+
+
+def main():
+ def handle_stats(args, client):
+ if args.reset:
+ s = client.reset_stats()
+ else:
+ s = client.get_stats()
+ pprint.pprint(s)
+ return 0
+
+ def handle_stress(args, client):
+ def thread_main(pbar, lock):
+ nonlocal found_hashes
+ nonlocal missed_hashes
+ nonlocal max_time
+
+ client = hashserv.create_client(args.address)
+
+ for i in range(args.requests):
+ taskhash = hashlib.sha256()
+ taskhash.update(args.taskhash_seed.encode('utf-8'))
+ taskhash.update(str(i).encode('utf-8'))
+
+ start_time = time.perf_counter()
+ l = client.get_unihash(METHOD, taskhash.hexdigest())
+ elapsed = time.perf_counter() - start_time
+
+ with lock:
+ if l:
+ found_hashes += 1
+ else:
+ missed_hashes += 1
+
+ max_time = max(elapsed, max_time)
+ pbar.update()
+
+ max_time = 0
+ found_hashes = 0
+ missed_hashes = 0
+ lock = threading.Lock()
+ total_requests = args.clients * args.requests
+ start_time = time.perf_counter()
+ with ProgressBar(total=total_requests) as pbar:
+ threads = [threading.Thread(target=thread_main, args=(pbar, lock), daemon=False) for _ in range(args.clients)]
+ for t in threads:
+ t.start()
+
+ for t in threads:
+ t.join()
+
+ elapsed = time.perf_counter() - start_time
+ with lock:
+ print("%d requests in %.1fs. %.1f requests per second" % (total_requests, elapsed, total_requests / elapsed))
+ print("Average request time %.8fs" % (elapsed / total_requests))
+ print("Max request time was %.8fs" % max_time)
+ print("Found %d hashes, missed %d" % (found_hashes, missed_hashes))
+
+ if args.report:
+ with ProgressBar(total=args.requests) as pbar:
+ for i in range(args.requests):
+ taskhash = hashlib.sha256()
+ taskhash.update(args.taskhash_seed.encode('utf-8'))
+ taskhash.update(str(i).encode('utf-8'))
+
+ outhash = hashlib.sha256()
+ outhash.update(args.outhash_seed.encode('utf-8'))
+ outhash.update(str(i).encode('utf-8'))
+
+ client.report_unihash(taskhash.hexdigest(), METHOD, outhash.hexdigest(), taskhash.hexdigest())
+
+ with lock:
+ pbar.update()
+
+ parser = argparse.ArgumentParser(description='Hash Equivalence Client')
+ parser.add_argument('--address', default=DEFAULT_ADDRESS, help='Server address (default "%(default)s")')
+ parser.add_argument('--log', default='WARNING', help='Set logging level')
+
+ subparsers = parser.add_subparsers()
+
+ stats_parser = subparsers.add_parser('stats', help='Show server stats')
+ stats_parser.add_argument('--reset', action='store_true',
+ help='Reset server stats')
+ stats_parser.set_defaults(func=handle_stats)
+
+ stress_parser = subparsers.add_parser('stress', help='Run stress test')
+ stress_parser.add_argument('--clients', type=int, default=10,
+ help='Number of simultaneous clients')
+ stress_parser.add_argument('--requests', type=int, default=1000,
+ help='Number of requests each client will perform')
+ stress_parser.add_argument('--report', action='store_true',
+ help='Report new hashes')
+ stress_parser.add_argument('--taskhash-seed', default='',
+ help='Include string in taskhash')
+ stress_parser.add_argument('--outhash-seed', default='',
+ help='Include string in outhash')
+ stress_parser.set_defaults(func=handle_stress)
+
+ args = parser.parse_args()
+
+ logger = logging.getLogger('hashserv')
+
+ level = getattr(logging, args.log.upper(), None)
+ if not isinstance(level, int):
+ raise ValueError('Invalid log level: %s' % args.log)
+
+ logger.setLevel(level)
+ console = logging.StreamHandler()
+ console.setLevel(level)
+ logger.addHandler(console)
+
+ func = getattr(args, 'func', None)
+ if func:
+ client = hashserv.create_client(args.address)
+ # Try to establish a connection to the server now to detect failures
+ # early
+ client.connect()
+
+ return func(args, client)
+
+ return 0
+
+
+if __name__ == '__main__':
+ try:
+ ret = main()
+ except Exception:
+ ret = 1
+ import traceback
+ traceback.print_exc()
+ sys.exit(ret)
diff --git a/bitbake/bin/bitbake-hashserv b/bitbake/bin/bitbake-hashserv
new file mode 100755
index 0000000..1bc1f91
--- /dev/null
+++ b/bitbake/bin/bitbake-hashserv
@@ -0,0 +1,62 @@
+#! /usr/bin/env python3
+#
+# Copyright (C) 2018 Garmin Ltd.
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+import os
+import sys
+import logging
+import argparse
+import sqlite3
+
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib'))
+
+import hashserv
+
+VERSION = "1.0.0"
+
+DEFAULT_BIND = 'unix://./hashserve.sock'
+
+
+def main():
+ parser = argparse.ArgumentParser(description='Hash Equivalence Reference Server. Version=%s' % VERSION,
+ epilog='''The bind address is the path to a unix domain socket if it is
+ prefixed with "unix://". Otherwise, it is an IP address
+ and port in form ADDRESS:PORT. To bind to all addresses, leave
+ the ADDRESS empty, e.g. "--bind :8686". To bind to a specific
+ IPv6 address, enclose the address in "[]", e.g.
+ "--bind [::1]:8686"'''
+ )
+
+ parser.add_argument('--bind', default=DEFAULT_BIND, help='Bind address (default "%(default)s")')
+ parser.add_argument('--database', default='./hashserv.db', help='Database file (default "%(default)s")')
+ parser.add_argument('--log', default='WARNING', help='Set logging level')
+
+ args = parser.parse_args()
+
+ logger = logging.getLogger('hashserv')
+
+ level = getattr(logging, args.log.upper(), None)
+ if not isinstance(level, int):
+ raise ValueError('Invalid log level: %s' % args.log)
+
+ logger.setLevel(level)
+ console = logging.StreamHandler()
+ console.setLevel(level)
+ logger.addHandler(console)
+
+ server = hashserv.create_server(args.bind, args.database)
+ server.serve_forever()
+ return 0
+
+
+if __name__ == '__main__':
+ try:
+ ret = main()
+ except Exception:
+ ret = 1
+ import traceback
+ traceback.print_exc()
+ sys.exit(ret)
diff --git a/bitbake/bin/bitbake-layers b/bitbake/bin/bitbake-layers
index d184011..a884dc1 100755
--- a/bitbake/bin/bitbake-layers
+++ b/bitbake/bin/bitbake-layers
@@ -7,18 +7,8 @@
# Copyright (C) 2011 Mentor Graphics Corporation
# Copyright (C) 2011-2015 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import logging
import os
diff --git a/bitbake/bin/bitbake-prserv b/bitbake/bin/bitbake-prserv
index f38d2dd..1e9b6cb 100755
--- a/bitbake/bin/bitbake-prserv
+++ b/bitbake/bin/bitbake-prserv
@@ -1,4 +1,8 @@
#!/usr/bin/env python3
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
import os
import sys,logging
import optparse
diff --git a/bitbake/bin/bitbake-selftest b/bitbake/bin/bitbake-selftest
index cfa7ac5..041a271 100755
--- a/bitbake/bin/bitbake-selftest
+++ b/bitbake/bin/bitbake-selftest
@@ -2,18 +2,8 @@
#
# Copyright (C) 2012 Richard Purdie
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import os
import sys, logging
@@ -22,6 +12,7 @@ sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib
import unittest
try:
import bb
+ import hashserv
import layerindexlib
except RuntimeError as exc:
sys.exit(str(exc))
@@ -33,7 +24,10 @@ tests = ["bb.tests.codeparser",
"bb.tests.event",
"bb.tests.fetch",
"bb.tests.parse",
+ "bb.tests.persist_data",
+ "bb.tests.runqueue",
"bb.tests.utils",
+ "hashserv.tests",
"layerindexlib.tests.layerindexobj",
"layerindexlib.tests.restapi",
"layerindexlib.tests.cooker"]
diff --git a/bitbake/bin/bitbake-worker b/bitbake/bin/bitbake-worker
index e925054..6776cad 100755
--- a/bitbake/bin/bitbake-worker
+++ b/bitbake/bin/bitbake-worker
@@ -1,4 +1,7 @@
#!/usr/bin/env python3
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
import os
import sys
@@ -136,7 +139,7 @@ def sigterm_handler(signum, frame):
os.killpg(0, signal.SIGTERM)
sys.exit()
-def fork_off_task(cfg, data, databuilder, workerdata, fn, task, taskname, appends, taskdepdata, extraconfigdata, quieterrors=False, dry_run_exec=False):
+def fork_off_task(cfg, data, databuilder, workerdata, fn, task, taskname, taskhash, unihash, appends, taskdepdata, extraconfigdata, quieterrors=False, dry_run_exec=False):
# We need to setup the environment BEFORE the fork, since
# a fork() or exec*() activates PSEUDO...
@@ -231,10 +234,13 @@ def fork_off_task(cfg, data, databuilder, workerdata, fn, task, taskname, append
the_data.setVar(varname, value)
bb.parse.siggen.set_taskdata(workerdata["sigdata"])
+ if "newhashes" in workerdata:
+ bb.parse.siggen.set_taskhashes(workerdata["newhashes"])
ret = 0
the_data = bb_cache.loadDataFull(fn, appends)
- the_data.setVar('BB_TASKHASH', workerdata["runq_hash"][task])
+ the_data.setVar('BB_TASKHASH', taskhash)
+ the_data.setVar('BB_UNIHASH', unihash)
bb.utils.set_process_name("%s:%s" % (the_data.getVar("PN"), taskname.replace("do_", "")))
@@ -373,6 +379,7 @@ class BitbakeWorker(object):
self.handle_item(b"cookerconfig", self.handle_cookercfg)
self.handle_item(b"extraconfigdata", self.handle_extraconfigdata)
self.handle_item(b"workerdata", self.handle_workerdata)
+ self.handle_item(b"newtaskhashes", self.handle_newtaskhashes)
self.handle_item(b"runtask", self.handle_runtask)
self.handle_item(b"finishnow", self.handle_finishnow)
self.handle_item(b"ping", self.handle_ping)
@@ -411,6 +418,10 @@ class BitbakeWorker(object):
bb.msg.loggerDefaultDomains = self.workerdata["logdefaultdomain"]
for mc in self.databuilder.mcdata:
self.databuilder.mcdata[mc].setVar("PRSERV_HOST", self.workerdata["prhost"])
+ self.databuilder.mcdata[mc].setVar("BB_HASHSERVE", self.workerdata["hashservaddr"])
+
+ def handle_newtaskhashes(self, data):
+ self.workerdata["newhashes"] = pickle.loads(data)
def handle_ping(self, _):
workerlog_write("Handling ping\n")
@@ -425,10 +436,10 @@ class BitbakeWorker(object):
sys.exit(0)
def handle_runtask(self, data):
- fn, task, taskname, quieterrors, appends, taskdepdata, dry_run_exec = pickle.loads(data)
+ fn, task, taskname, taskhash, unihash, quieterrors, appends, taskdepdata, dry_run_exec = pickle.loads(data)
workerlog_write("Handling runtask %s %s %s\n" % (task, fn, taskname))
- pid, pipein, pipeout = fork_off_task(self.cookercfg, self.data, self.databuilder, self.workerdata, fn, task, taskname, appends, taskdepdata, self.extraconfigdata, quieterrors, dry_run_exec)
+ pid, pipein, pipeout = fork_off_task(self.cookercfg, self.data, self.databuilder, self.workerdata, fn, task, taskname, taskhash, unihash, appends, taskdepdata, self.extraconfigdata, quieterrors, dry_run_exec)
self.build_pids[pid] = task
self.build_pipes[pid] = runQueueWorkerPipe(pipein, pipeout)
diff --git a/bitbake/bin/bitdoc b/bitbake/bin/bitdoc
index 2744678..9bd02be 100755
--- a/bitbake/bin/bitdoc
+++ b/bitbake/bin/bitdoc
@@ -1,21 +1,9 @@
#!/usr/bin/env python3
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# Copyright (C) 2005 Holger Hans Peter Freyther
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import optparse, os, sys
diff --git a/bitbake/bin/git-make-shallow b/bitbake/bin/git-make-shallow
index 296d3a3..57069f7 100755
--- a/bitbake/bin/git-make-shallow
+++ b/bitbake/bin/git-make-shallow
@@ -1,4 +1,8 @@
#!/usr/bin/env python3
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
"""git-make-shallow: make the current git repository shallow
Remove the history of the specified revisions, then optionally filter the
diff --git a/bitbake/bin/toaster b/bitbake/bin/toaster
index ecf66fa..c3472df 100755
--- a/bitbake/bin/toaster
+++ b/bitbake/bin/toaster
@@ -3,19 +3,9 @@
# toaster - shell script to start Toaster
# Copyright (C) 2013-2015 Intel Corp.
-
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; either version 2 of the License, or
-# (at your option) any later version.
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
+# SPDX-License-Identifier: GPL-2.0-or-later
#
-# You should have received a copy of the GNU General Public License
-# along with this program. If not, see http://www.gnu.org/licenses/.
HELP="
Usage: source toaster start|stop [webport=<address:port>] [noweb] [nobuild] [toasterdir]
diff --git a/bitbake/bin/toaster-eventreplay b/bitbake/bin/toaster-eventreplay
index 80967a0..8fa4ab7 100755
--- a/bitbake/bin/toaster-eventreplay
+++ b/bitbake/bin/toaster-eventreplay
@@ -1,25 +1,12 @@
#!/usr/bin/env python3
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# Copyright (C) 2014 Alex Damian
#
+# SPDX-License-Identifier: GPL-2.0-only
+#
# This file re-uses code spread throughout other Bitbake source files.
# As such, all other copyrights belong to their own right holders.
#
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
"""
This command takes a filename as a single parameter. The filename is read
diff --git a/bitbake/classes/base.bbclass b/bitbake/classes/base.bbclass
index 71d9ec1..08441fe 100644
--- a/bitbake/classes/base.bbclass
+++ b/bitbake/classes/base.bbclass
@@ -44,7 +44,7 @@ python do_showdata() {
# emit the metadata which isnt valid shell
for e in bb.data.keys(d):
if d.getVarFlag(e, 'python', False):
- bb.plain("\npython %s () {\n%s}" % (e, d.getVar(e, True)))
+ bb.plain("\npython %s () {\n%s}" % (e, d.getVar(e)))
}
addtask listtasks
diff --git a/bitbake/contrib/dump_cache.py b/bitbake/contrib/dump_cache.py
index 8963ca4..c6723cb 100755
--- a/bitbake/contrib/dump_cache.py
+++ b/bitbake/contrib/dump_cache.py
@@ -1,6 +1,4 @@
#!/usr/bin/env python3
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# Copyright (C) 2012, 2018 Wind River Systems, Inc.
#
diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
index f1caaec..46dafee 100644
--- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
+++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
@@ -31,7 +31,7 @@
<para>
Prior to executing BitBake, you should take advantage of available
parallel thread execution on your build host by setting the
- <link linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
+ <link linkend='var-bb-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
variable in your project's <filename>local.conf</filename>
configuration file.
</para>
@@ -87,9 +87,9 @@
<para>
The <filename>layer.conf</filename> files are used to
construct key variables such as
- <link linkend='var-BBPATH'><filename>BBPATH</filename></link>
+ <link linkend='var-bb-BBPATH'><filename>BBPATH</filename></link>
and
- <link linkend='var-BBFILES'><filename>BBFILES</filename></link>.
+ <link linkend='var-bb-BBFILES'><filename>BBFILES</filename></link>.
<filename>BBPATH</filename> is used to search for
configuration and class files under the
<filename>conf</filename> and <filename>classes</filename>
@@ -117,19 +117,19 @@
at certain variables, including:
<itemizedlist>
<listitem><para>
- <link linkend='var-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>
+ <link linkend='var-bb-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>
</para></listitem>
<listitem><para>
- <link linkend='var-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link>
+ <link linkend='var-bb-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link>
</para></listitem>
<listitem><para>
- <link linkend='var-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link>
+ <link linkend='var-bb-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link>
</para></listitem>
<listitem><para>
- <link linkend='var-BB_ORIGENV'><filename>BB_ORIGENV</filename></link>
+ <link linkend='var-bb-BB_ORIGENV'><filename>BB_ORIGENV</filename></link>
</para></listitem>
<listitem><para>
- <link linkend='var-BITBAKE_UI'><filename>BITBAKE_UI</filename></link>
+ <link linkend='var-bb-BITBAKE_UI'><filename>BITBAKE_UI</filename></link>
</para></listitem>
</itemizedlist>
The first four variables in this list relate to how BitBake treats shell
@@ -156,7 +156,7 @@
BitBake first searches the current working directory for an
optional <filename>conf/bblayers.conf</filename> configuration file.
This file is expected to contain a
- <link linkend='var-BBLAYERS'><filename>BBLAYERS</filename></link>
+ <link linkend='var-bb-BBLAYERS'><filename>BBLAYERS</filename></link>
variable that is a space-delimited list of 'layer' directories.
Recall that if BitBake cannot find a <filename>bblayers.conf</filename>
file, then it is assumed the user has set the <filename>BBPATH</filename>
@@ -166,10 +166,10 @@
<para>
For each directory (layer) in this list, a <filename>conf/layer.conf</filename>
file is located and parsed with the
- <link linkend='var-LAYERDIR'><filename>LAYERDIR</filename></link>
+ <link linkend='var-bb-LAYERDIR'><filename>LAYERDIR</filename></link>
variable being set to the directory where the layer was found.
The idea is these files automatically set up
- <link linkend='var-BBPATH'><filename>BBPATH</filename></link>
+ <link linkend='var-bb-BBPATH'><filename>BBPATH</filename></link>
and other variables correctly for a given build directory.
</para>
@@ -189,7 +189,7 @@
depending on the environment variables previously
mentioned or set in the configuration files.
The
- "<link linkend='ref-variables-glos'>Variables Glossary</link>"
+ "<link linkend='ref-bb-variables-glos'>Variables Glossary</link>"
chapter presents a full list of variables.
</para>
@@ -204,7 +204,7 @@
<para>
The <filename>base.bbclass</filename> file is always included.
Other classes that are specified in the configuration using the
- <link linkend='var-INHERIT'><filename>INHERIT</filename></link>
+ <link linkend='var-bb-INHERIT'><filename>INHERIT</filename></link>
variable are also included.
BitBake searches for class files in a
<filename>classes</filename> subdirectory under
@@ -270,7 +270,7 @@
<para>
During the configuration phase, BitBake will have set
- <link linkend='var-BBFILES'><filename>BBFILES</filename></link>.
+ <link linkend='var-bb-BBFILES'><filename>BBFILES</filename></link>.
BitBake now uses it to construct a list of recipes to parse,
along with any append files (<filename>.bbappend</filename>)
to apply.
@@ -292,7 +292,7 @@
Any inherit statements cause BitBake to find and
then parse class files (<filename>.bbclass</filename>)
using
- <link linkend='var-BBPATH'><filename>BBPATH</filename></link>
+ <link linkend='var-bb-BBPATH'><filename>BBPATH</filename></link>
as the search path.
Finally, BitBake parses in order any append files found in
<filename>BBFILES</filename>.
@@ -303,8 +303,8 @@
pieces of metadata.
For example, in <filename>bitbake.conf</filename> the recipe
name and version are used to set the variables
- <link linkend='var-PN'><filename>PN</filename></link> and
- <link linkend='var-PV'><filename>PV</filename></link>:
+ <link linkend='var-bb-PN'><filename>PN</filename></link> and
+ <link linkend='var-bb-PV'><filename>PV</filename></link>:
<literallayout class='monospaced'>
PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}"
PV = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[1] or '1.0'}"
@@ -336,7 +336,7 @@
recipe information.
The validity of this cache is determined by first computing a
checksum of the base configuration data (see
- <link linkend='var-BB_HASHCONFIG_WHITELIST'><filename>BB_HASHCONFIG_WHITELIST</filename></link>)
+ <link linkend='var-bb-BB_HASHCONFIG_WHITELIST'><filename>BB_HASHCONFIG_WHITELIST</filename></link>)
and then checking if the checksum matches.
If that checksum matches what is in the cache and the recipe
and class files have not changed, Bitbake is able to use
@@ -384,9 +384,9 @@
the recipe can be known.
Each recipe's <filename>PROVIDES</filename> list is created
implicitly through the recipe's
- <link linkend='var-PN'><filename>PN</filename></link> variable
+ <link linkend='var-bb-PN'><filename>PN</filename></link> variable
and explicitly through the recipe's
- <link linkend='var-PROVIDES'><filename>PROVIDES</filename></link>
+ <link linkend='var-bb-PROVIDES'><filename>PROVIDES</filename></link>
variable, which is optional.
</para>
@@ -427,7 +427,7 @@
PREFERRED_PROVIDER_virtual/kernel = "linux-yocto"
</literallayout>
The default
- <link linkend='var-PREFERRED_PROVIDER'><filename>PREFERRED_PROVIDER</filename></link>
+ <link linkend='var-bb-PREFERRED_PROVIDER'><filename>PREFERRED_PROVIDER</filename></link>
is the provider with the same name as the target.
Bitbake iterates through each target it needs to build and
resolves them and their dependencies using this process.
@@ -439,10 +439,10 @@
BitBake defaults to the highest version of a provider.
Version comparisons are made using the same method as Debian.
You can use the
- <link linkend='var-PREFERRED_VERSION'><filename>PREFERRED_VERSION</filename></link>
+ <link linkend='var-bb-PREFERRED_VERSION'><filename>PREFERRED_VERSION</filename></link>
variable to specify a particular version.
You can influence the order by using the
- <link linkend='var-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link>
+ <link linkend='var-bb-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link>
variable.
</para>
@@ -464,7 +464,7 @@
BitBake defaults to selecting the most recent
version, unless otherwise specified.
If the recipe in question has a
- <link linkend='var-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link>
+ <link linkend='var-bb-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link>
set lower than the other recipes (default is 0), then
it will not be selected.
This allows the person or persons maintaining
@@ -475,9 +475,9 @@
<para>
If the first recipe is named <filename>a_1.1.bb</filename>, then the
- <link linkend='var-PN'><filename>PN</filename></link> variable
+ <link linkend='var-bb-PN'><filename>PN</filename></link> variable
will be set to “a”, and the
- <link linkend='var-PV'><filename>PV</filename></link>
+ <link linkend='var-bb-PV'><filename>PV</filename></link>
variable will be set to 1.1.
</para>
@@ -532,11 +532,11 @@
<para>
Dependencies are defined through several variables.
You can find information about variables BitBake uses in
- the <link linkend='ref-variables-glos'>Variables Glossary</link>
+ the <link linkend='ref-bb-variables-glos'>Variables Glossary</link>
near the end of this manual.
At a basic level, it is sufficient to know that BitBake uses the
- <link linkend='var-DEPENDS'><filename>DEPENDS</filename></link> and
- <link linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link> variables when
+ <link linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link> and
+ <link linkend='var-bb-RDEPENDS'><filename>RDEPENDS</filename></link> variables when
calculating dependencies.
</para>
@@ -560,7 +560,7 @@
<para>
The build now starts with BitBake forking off threads up to the limit set in the
- <link linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
+ <link linkend='var-bb-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
variable.
BitBake continues to fork threads as long as there are tasks ready to run,
those tasks have all their dependencies met, and the thread threshold has not been
@@ -574,7 +574,7 @@
<para>
As each task completes, a timestamp is written to the directory specified by the
- <link linkend='var-STAMP'><filename>STAMP</filename></link> variable.
+ <link linkend='var-bb-STAMP'><filename>STAMP</filename></link> variable.
On subsequent runs, BitBake looks in the build directory within
<filename>tmp/stamps</filename> and does not rerun
tasks that are already completed unless a timestamp is found to be invalid.
@@ -618,7 +618,7 @@
<para>
Tasks can be either a shell task or a Python task.
For shell tasks, BitBake writes a shell script to
- <filename>${</filename><link linkend='var-T'><filename>T</filename></link><filename>}/run.do_taskname.pid</filename>
+ <filename>${</filename><link linkend='var-bb-T'><filename>T</filename></link><filename>}/run.do_taskname.pid</filename>
and then executes the script.
The generated shell script contains all the exported variables,
and the shell functions with all variables expanded.
@@ -645,10 +645,10 @@
behavior:
<itemizedlist>
<listitem><para>
- <link linkend='var-BB_SCHEDULER'><filename>BB_SCHEDULER</filename></link>
+ <link linkend='var-bb-BB_SCHEDULER'><filename>BB_SCHEDULER</filename></link>
</para></listitem>
<listitem><para>
- <link linkend='var-BB_SCHEDULERS'><filename>BB_SCHEDULERS</filename></link>
+ <link linkend='var-bb-BB_SCHEDULERS'><filename>BB_SCHEDULERS</filename></link>
</para></listitem>
</itemizedlist>
It is possible to have functions run before and after a task's main
@@ -684,7 +684,7 @@
The simplistic approach for excluding the working directory is to set
it to some fixed value and create the checksum for the "run" script.
BitBake goes one step better and uses the
- <link linkend='var-BB_HASHBASE_WHITELIST'><filename>BB_HASHBASE_WHITELIST</filename></link>
+ <link linkend='var-bb-BB_HASHBASE_WHITELIST'><filename>BB_HASHBASE_WHITELIST</filename></link>
variable to define a list of variables that should never be included
when generating the signatures.
</para>
@@ -795,7 +795,7 @@
This results in any metadata change that changes the task hash, automatically
causing the task to be run again.
This removes the need to bump
- <link linkend='var-PR'><filename>PR</filename></link>
+ <link linkend='var-bb-PR'><filename>PR</filename></link>
values, and changes to metadata automatically ripple across the build.
</para>
@@ -884,7 +884,7 @@
<para>
BitBake first calls the function defined by the
- <link linkend='var-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></link>
+ <link linkend='var-bb-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></link>
variable with a list of tasks and corresponding
hashes it wants to build.
This function is designed to be fast and returns a list
@@ -908,7 +908,7 @@
For example, it is pointless to obtain a compiler if you
already have the compiled binary.
To handle this, BitBake calls the
- <link linkend='var-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></link>
+ <link linkend='var-bb-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></link>
function for each successful setscene task to know whether or not it needs
to obtain the dependencies of that task.
</para>
@@ -916,7 +916,7 @@
<para>
Finally, after all the setscene tasks have executed, BitBake calls the
function listed in
- <link linkend='var-BB_SETSCENE_VERIFY_FUNCTION2'><filename>BB_SETSCENE_VERIFY_FUNCTION2</filename></link>
+ <link linkend='var-bb-BB_SETSCENE_VERIFY_FUNCTION2'><filename>BB_SETSCENE_VERIFY_FUNCTION2</filename></link>
with the list of tasks BitBake thinks has been "covered".
The metadata can then ensure that this list is correct and can
inform BitBake that it wants specific tasks to be run regardless
diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml
index 29ae486..6840408 100644
--- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml
+++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml
@@ -44,7 +44,7 @@
</literallayout>
This code sets up an instance of the fetch class.
The instance uses a space-separated list of URLs from the
- <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
+ <link linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link>
variable and then calls the <filename>download</filename>
method to download the files.
</para>
@@ -78,7 +78,7 @@
<listitem><para><emphasis>Pre-mirror Sites:</emphasis>
BitBake first uses pre-mirrors to try and find source files.
These locations are defined using the
- <link linkend='var-PREMIRRORS'><filename>PREMIRRORS</filename></link>
+ <link linkend='var-bb-PREMIRRORS'><filename>PREMIRRORS</filename></link>
variable.
</para></listitem>
<listitem><para><emphasis>Source URI:</emphasis>
@@ -88,7 +88,7 @@
<listitem><para><emphasis>Mirror Sites:</emphasis>
If fetch failures occur, BitBake next uses mirror locations as
defined by the
- <link linkend='var-MIRRORS'><filename>MIRRORS</filename></link>
+ <link linkend='var-bb-MIRRORS'><filename>MIRRORS</filename></link>
variable.
</para></listitem>
</itemizedlist>
@@ -144,7 +144,7 @@
Any source files that are not local (i.e.
downloaded from the Internet) are placed into the download
directory, which is specified by the
- <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
+ <link linkend='var-bb-DL_DIR'><filename>DL_DIR</filename></link>
variable.
</para>
@@ -184,11 +184,11 @@
<para>
If
- <link linkend='var-BB_STRICT_CHECKSUM'><filename>BB_STRICT_CHECKSUM</filename></link>
+ <link linkend='var-bb-BB_STRICT_CHECKSUM'><filename>BB_STRICT_CHECKSUM</filename></link>
is set, any download without a checksum triggers an
error message.
The
- <link linkend='var-BB_NO_NETWORK'><filename>BB_NO_NETWORK</filename></link>
+ <link linkend='var-bb-BB_NO_NETWORK'><filename>BB_NO_NETWORK</filename></link>
variable can be used to make any attempted network access a fatal
error, which is useful for checking that mirrors are complete
as well as other things.
@@ -265,11 +265,11 @@
The filename you specify within the URL can be
either an absolute or relative path to a file.
If the filename is relative, the contents of the
- <link linkend='var-FILESPATH'><filename>FILESPATH</filename></link>
+ <link linkend='var-bb-FILESPATH'><filename>FILESPATH</filename></link>
variable is used in the same way
<filename>PATH</filename> is used to find executables.
If the file cannot be found, it is assumed that it is available in
- <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
+ <link linkend='var-bb-DL_DIR'><filename>DL_DIR</filename></link>
by the time the <filename>download()</filename> method is called.
</para>
@@ -304,7 +304,7 @@
allows the name of the downloaded file to be specified.
Specifying the name of the downloaded file is useful
for avoiding collisions in
- <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
+ <link linkend='var-bb-DL_DIR'><filename>DL_DIR</filename></link>
when dealing with multiple files that have the same name.
</para>
@@ -355,7 +355,7 @@
A special value of "now" causes the checkout to
be updated on every build.
</para></listitem>
- <listitem><para><emphasis><link linkend='var-CVSDIR'><filename>CVSDIR</filename></link>:</emphasis>
+ <listitem><para><emphasis><link linkend='var-bb-CVSDIR'><filename>CVSDIR</filename></link>:</emphasis>
Specifies where a temporary checkout is saved.
The location is often <filename>DL_DIR/cvs</filename>.
</para></listitem>
@@ -395,7 +395,7 @@
<listitem><para><emphasis>"date":</emphasis>
Specifies a date.
If no "date" is specified, the
- <link linkend='var-SRCDATE'><filename>SRCDATE</filename></link>
+ <link linkend='var-bb-SRCDATE'><filename>SRCDATE</filename></link>
of the configuration is used to checkout a specific date.
The special value of "now" causes the checkout to be
updated on every build.
@@ -406,7 +406,7 @@
to which the module is unpacked.
You are forcing the module into a special
directory relative to
- <link linkend='var-CVSDIR'><filename>CVSDIR</filename></link>.
+ <link linkend='var-bb-CVSDIR'><filename>CVSDIR</filename></link>.
</para></listitem>
<listitem><para><emphasis>"rsh"</emphasis>
Used in conjunction with the "method" parameter.
@@ -448,7 +448,7 @@
<filename>FETCHCMD_svn</filename>, which defaults
to "svn".
The fetcher's temporary working directory is set by
- <link linkend='var-SVNDIR'><filename>SVNDIR</filename></link>,
+ <link linkend='var-bb-SVNDIR'><filename>SVNDIR</filename></link>,
which is usually <filename>DL_DIR/svn</filename>.
</para>
@@ -509,7 +509,7 @@
source control system.
The fetcher works by creating a bare clone of the
remote into
- <link linkend='var-GITDIR'><filename>GITDIR</filename></link>,
+ <link linkend='var-bb-GITDIR'><filename>GITDIR</filename></link>,
which is usually <filename>DL_DIR/git2</filename>.
This bare clone is then cloned into the work directory during the
unpack stage when a specific tree is checked out.
@@ -588,6 +588,14 @@
The name of the path in which to place the checkout.
By default, the path is <filename>git/</filename>.
</para></listitem>
+ <listitem><para><emphasis>"usehead":</emphasis>
+ Enables local <filename>git://</filename> URLs to use the
+ current branch HEAD as the revision for use with
+ <filename>AUTOREV</filename>.
+ The "usehead" parameter implies no branch and only works
+ when the transfer protocol is
+ <filename>file://</filename>.
+ </para></listitem>
</itemizedlist>
Here are some example URLs:
<literallayout class='monospaced'>
@@ -604,7 +612,7 @@
This fetcher submodule inherits from the
<link linkend='git-fetcher'>Git fetcher</link> and extends
that fetcher's behavior by fetching a repository's submodules.
- <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
+ <link linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link>
is passed to the Git fetcher as described in the
"<link linkend='git-fetcher'>Git Fetcher (<filename>git://</filename>)</link>"
section.
@@ -639,9 +647,9 @@
<para>
To use this fetcher, make sure your recipe has proper
- <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>,
- <link linkend='var-SRCREV'><filename>SRCREV</filename></link>, and
- <link linkend='var-PV'><filename>PV</filename></link> settings.
+ <link linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link>,
+ <link linkend='var-bb-SRCREV'><filename>SRCREV</filename></link>, and
+ <link linkend='var-bb-PV'><filename>PV</filename></link> settings.
Here is an example:
<literallayout class='monospaced'>
SRC_URI = "ccrc://cc.example.org/ccrc;vob=/example_vob;module=/example_module"
@@ -726,15 +734,15 @@
<filename>FETCHCMD_p4</filename>, which defaults
to "p4".
The fetcher's temporary working directory is set by
- <link linkend='var-P4DIR'><filename>P4DIR</filename></link>,
+ <link linkend='var-bb-P4DIR'><filename>P4DIR</filename></link>,
which defaults to "DL_DIR/p4".
</para>
<para>
To use this fetcher, make sure your recipe has proper
- <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>,
- <link linkend='var-SRCREV'><filename>SRCREV</filename></link>, and
- <link linkend='var-PV'><filename>PV</filename></link> values.
+ <link linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link>,
+ <link linkend='var-bb-SRCREV'><filename>SRCREV</filename></link>, and
+ <link linkend='var-bb-PV'><filename>PV</filename></link> values.
The p4 executable is able to use the config file defined by your
system's <filename>P4CONFIG</filename> environment variable in
order to define the Perforce server URL and port, username, and
@@ -785,9 +793,9 @@
<filename>google-repo</filename> source control system.
The fetcher works by initiating and syncing sources of the
repository into
- <link linkend='var-REPODIR'><filename>REPODIR</filename></link>,
+ <link linkend='var-bb-REPODIR'><filename>REPODIR</filename></link>,
which is usually
- <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link><filename>/repo</filename>.
+ <link linkend='var-bb-DL_DIR'><filename>DL_DIR</filename></link><filename>/repo</filename>.
</para>
<para>
@@ -824,19 +832,22 @@
Bazaar (<filename>bzr://</filename>)
</para></listitem>
<listitem><para>
- Trees using Git Annex (<filename>gitannex://</filename>)
+ Mercurial (<filename>hg://</filename>)
</para></listitem>
<listitem><para>
- Secure FTP (<filename>sftp://</filename>)
+ npm (<filename>npm://</filename>)
</para></listitem>
<listitem><para>
- Secure Shell (<filename>ssh://</filename>)
+ OSC (<filename>osc://</filename>)
</para></listitem>
<listitem><para>
- OSC (<filename>osc://</filename>)
+ Secure FTP (<filename>sftp://</filename>)
</para></listitem>
<listitem><para>
- Mercurial (<filename>hg://</filename>)
+ Secure Shell (<filename>ssh://</filename>)
+ </para></listitem>
+ <listitem><para>
+ Trees using Git Annex (<filename>gitannex://</filename>)
</para></listitem>
</itemizedlist>
No documentation currently exists for these lesser used
diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.xml b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.xml
index 9076f0f..39066e4 100644
--- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.xml
+++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.xml
@@ -194,7 +194,7 @@
<para>
When you run BitBake, it begins looking for metadata files.
The
- <link linkend='var-BBPATH'><filename>BBPATH</filename></link>
+ <link linkend='var-bb-BBPATH'><filename>BBPATH</filename></link>
variable is what tells BitBake where to look for those files.
<filename>BBPATH</filename> is not set and you need to set it.
Without <filename>BBPATH</filename>, Bitbake cannot
@@ -273,14 +273,14 @@
some editor to create the <filename>bitbake.conf</filename>
so that it contains the following:
<literallayout class='monospaced'>
- <link linkend='var-PN'>PN</link> = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}"
+ <link linkend='var-bb-PN'>PN</link> = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}"
</literallayout>
<literallayout class='monospaced'>
- TMPDIR = "${<link linkend='var-TOPDIR'>TOPDIR</link>}/tmp"
- <link linkend='var-CACHE'>CACHE</link> = "${TMPDIR}/cache"
- <link linkend='var-STAMP'>STAMP</link> = "${TMPDIR}/${PN}/stamps"
- <link linkend='var-T'>T</link> = "${TMPDIR}/${PN}/work"
- <link linkend='var-B'>B</link> = "${TMPDIR}/${PN}"
+ TMPDIR = "${<link linkend='var-bb-TOPDIR'>TOPDIR</link>}/tmp"
+ <link linkend='var-bb-CACHE'>CACHE</link> = "${TMPDIR}/cache"
+ <link linkend='var-bb-STAMP'>STAMP</link> = "${TMPDIR}/${PN}/stamps"
+ <link linkend='var-bb-T'>T</link> = "${TMPDIR}/${PN}/work"
+ <link linkend='var-bb-B'>B</link> = "${TMPDIR}/${PN}"
</literallayout>
<note>
Without a value for <filename>PN</filename>, the
@@ -402,12 +402,12 @@
Move to the <filename>conf</filename> directory and create a
<filename>layer.conf</filename> file that has the following:
<literallayout class='monospaced'>
- BBPATH .= ":${<link linkend='var-LAYERDIR'>LAYERDIR</link>}"
+ BBPATH .= ":${<link linkend='var-bb-LAYERDIR'>LAYERDIR</link>}"
- <link linkend='var-BBFILES'>BBFILES</link> += "${LAYERDIR}/*.bb"
+ <link linkend='var-bb-BBFILES'>BBFILES</link> += "${LAYERDIR}/*.bb"
- <link linkend='var-BBFILE_COLLECTIONS'>BBFILE_COLLECTIONS</link> += "mylayer"
- <link linkend='var-BBFILE_PATTERN'>BBFILE_PATTERN_mylayer</link> := "^${LAYERDIR_RE}/"
+ <link linkend='var-bb-BBFILE_COLLECTIONS'>BBFILE_COLLECTIONS</link> += "mylayer"
+ <link linkend='var-bb-BBFILE_PATTERN'>BBFILE_PATTERN_mylayer</link> := "^${LAYERDIR_RE}/"
</literallayout>
For information on these variables, click the links
to go to the definitions in the glossary.</para>
@@ -416,9 +416,9 @@
a recipe file named <filename>printhello.bb</filename> that
has the following:
<literallayout class='monospaced'>
- <link linkend='var-DESCRIPTION'>DESCRIPTION</link> = "Prints Hello World"
- <link linkend='var-PN'>PN</link> = 'printhello'
- <link linkend='var-PV'>PV</link> = '1'
+ <link linkend='var-bb-DESCRIPTION'>DESCRIPTION</link> = "Prints Hello World"
+ <link linkend='var-bb-PN'>PN</link> = 'printhello'
+ <link linkend='var-bb-PV'>PV</link> = '1'
python do_build() {
bb.plain("********************");
diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml
index f7d312a..8f2a960 100644
--- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml
+++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml
@@ -781,7 +781,7 @@
target, you must also enable BitBake to perform multiple
configuration builds.
Enabling is accomplished by setting the
- <link linkend='var-BBMULTICONFIG'><filename>BBMULTICONFIG</filename></link>
+ <link linkend='var-bb-BBMULTICONFIG'><filename>BBMULTICONFIG</filename></link>
variable in the <filename>local.conf</filename>
configuration file.
As an example, suppose you had configuration files
@@ -791,7 +791,7 @@
The following statement in the
<filename>local.conf</filename> file both enables
BitBake to perform multiple configuration builds and
- specifies the two multiconfigs:
+ specifies the two extra multiconfigs:
<literallayout class='monospaced'>
BBMULTICONFIG = "target1 target2"
</literallayout>
@@ -803,13 +803,13 @@
builds, use the following command form to start the
builds:
<literallayout class='monospaced'>
- $ bitbake [multiconfig:<replaceable>multiconfigname</replaceable>:]<replaceable>target</replaceable> [[[multiconfig:<replaceable>multiconfigname</replaceable>:]<replaceable>target</replaceable>] ... ]
+ $ bitbake [mc:<replaceable>multiconfigname</replaceable>:]<replaceable>target</replaceable> [[[mc:<replaceable>multiconfigname</replaceable>:]<replaceable>target</replaceable>] ... ]
</literallayout>
- Here is an example for two multiconfigs:
+ Here is an example for two extra multiconfigs:
<filename>target1</filename> and
<filename>target2</filename>:
<literallayout class='monospaced'>
- $ bitbake multiconfig:target1:<replaceable>target</replaceable> multiconfig:target2:<replaceable>target</replaceable>
+ $ bitbake mc::<replaceable>target</replaceable> mc:target1:<replaceable>target</replaceable> mc:target2:<replaceable>target</replaceable>
</literallayout>
</para>
</section>
@@ -837,13 +837,13 @@
build, you must declare the dependencies in the recipe
using the following statement form:
<literallayout class='monospaced'>
- <replaceable>task_or_package</replaceable>[mcdepends] = "multiconfig:<replaceable>from_multiconfig</replaceable>:<replaceable>to_multiconfig</replaceable>:<replaceable>recipe_name</replaceable>:<replaceable>task_on_which_to_depend</replaceable>"
+ <replaceable>task_or_package</replaceable>[mcdepends] = "mc:<replaceable>from_multiconfig</replaceable>:<replaceable>to_multiconfig</replaceable>:<replaceable>recipe_name</replaceable>:<replaceable>task_on_which_to_depend</replaceable>"
</literallayout>
To better show how to use this statement, consider an
example with two multiconfigs: <filename>target1</filename>
and <filename>target2</filename>:
<literallayout class='monospaced'>
- <replaceable>image_task</replaceable>[mcdepends] = "multiconfig:target1:target2:<replaceable>image2</replaceable>:<replaceable>rootfs_task</replaceable>"
+ <replaceable>image_task</replaceable>[mcdepends] = "mc:target1:target2:<replaceable>image2</replaceable>:<replaceable>rootfs_task</replaceable>"
</literallayout>
In this example, the
<replaceable>from_multiconfig</replaceable> is "target1" and
@@ -859,7 +859,7 @@
Once you set up this dependency, you can build the
"target1" multiconfig using a BitBake command as follows:
<literallayout class='monospaced'>
- $ bitbake multiconfig:target1:<replaceable>image1</replaceable>
+ $ bitbake mc:target1:<replaceable>image1</replaceable>
</literallayout>
This command executes all the tasks needed to create
<replaceable>image1</replaceable> for the "target1"
@@ -875,7 +875,7 @@
Consider this change to the statement in the
<replaceable>image1</replaceable> recipe:
<literallayout class='monospaced'>
- <replaceable>image_task</replaceable>[mcdepends] = "multiconfig:target1:target2:<replaceable>image2</replaceable>:<replaceable>image_task</replaceable>"
+ <replaceable>image_task</replaceable>[mcdepends] = "mc:target1:target2:<replaceable>image2</replaceable>:<replaceable>image_task</replaceable>"
</literallayout>
In this case, BitBake must create
<replaceable>image2</replaceable> for the "target2"
diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml
index 2490f6e..421364c 100644
--- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml
+++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml
@@ -61,6 +61,78 @@
</para>
</section>
+ <section id='modifying-existing-variables'>
+ <title>Modifying Existing Variables</title>
+
+ <para>
+ Sometimes you need to modify existing variables.
+ Following are some cases where you might find you want to
+ modify an existing variable:
+ <itemizedlist>
+ <listitem><para>
+ Customize a recipe that uses the variable.
+ </para></listitem>
+ <listitem><para>
+ Change a variable's default value used in a
+ <filename>*.bbclass</filename> file.
+ </para></listitem>
+ <listitem><para>
+ Change the variable in a <filename>*.bbappend</filename>
+ file to override the variable in the original recipe.
+ </para></listitem>
+ <listitem><para>
+ Change the variable in a configuration file so that the
+ value overrides an existing configuration.
+ </para></listitem>
+ </itemizedlist>
+ </para>
+
+ <para>
+ Changing a variable value can sometimes depend on how the
+ value was originally assigned and also on the desired
+ intent of the change.
+ In particular, when you append a value to a variable that
+ has a default value, the resulting value might not be what
+ you expect.
+ In this case, the value you provide might replace the value
+ rather than append to the default value.
+ </para>
+
+ <para>
+ If after you have changed a variable's value and something
+ unexplained occurs, you can use BitBake to check the actual
+ value of the suspect variable.
+ You can make these checks for both configuration and recipe
+ level changes:
+ <itemizedlist>
+ <listitem><para>
+ For configuration changes, use the following:
+ <literallayout class='monospaced'>
+ $ bitbake -e
+ </literallayout>
+ This command displays variable values after the
+ configuration files (i.e. <filename>local.conf</filename>,
+ <filename>bblayers.conf</filename>,
+ <filename>bitbake.conf</filename> and so forth) have
+ been parsed.
+ <note>
+ Variables that are exported to the environment are
+ preceded by the string "export" in the command's
+ output.
+ </note>
+ </para></listitem>
+ <listitem><para>
+ For recipe changes, use the following:
+ <literallayout class='monospaced'>
+ $ bitbake <replaceable>recipe</replaceable> -e | grep VARIABLE="
+ </literallayout>
+ This command checks to see if the variable actually
+ makes it into a specific recipe.
+ </para></listitem>
+ </itemizedlist>
+ </para>
+ </section>
+
<section id='line-joining'>
<title>Line Joining</title>
@@ -297,9 +369,8 @@
<para>
These operators differ from the ":=", ".=", "=.", "+=", and "=+"
- operators in that their effects are deferred
- until after parsing completes rather than being immediately
- applied.
+ operators in that their effects are applied at variable
+ expansion time rather than being immediately applied.
Here are some examples:
<literallayout class='monospaced'>
B = "bval"
@@ -348,18 +419,22 @@
FOO = "123 456 789 123456 123 456 123 456"
FOO_remove = "123"
FOO_remove = "456"
- FOO2 = "abc def ghi abcdef abc def abc def"
- FOO2_remove = "abc def"
+ FOO2 = " abc def ghi abcdef abc def abc def def"
+ FOO2_remove = " \
+ def \
+ abc \
+ ghi \
+ "
</literallayout>
The variable <filename>FOO</filename> becomes
- " 789 123456 "
+ " 789 123456 "
and <filename>FOO2</filename> becomes
- " ghi abcdef ".
+ " jkl abcdef ".
</para>
<para>
Like "_append" and "_prepend", "_remove"
- is deferred until after parsing completes.
+ is applied at variable expansion time.
</para>
</section>
@@ -503,7 +578,7 @@
</section>
<section id='unsetting-variables'>
- <title>Unseting variables</title>
+ <title>Unsetting variables</title>
<para>
It is possible to completely remove a variable or a variable flag
@@ -595,7 +670,7 @@
<para>
BitBake uses
- <link linkend='var-OVERRIDES'><filename>OVERRIDES</filename></link>
+ <link linkend='var-bb-OVERRIDES'><filename>OVERRIDES</filename></link>
to control what variables are overridden after BitBake
parses recipes and configuration files.
This section describes how you can use
@@ -705,7 +780,7 @@
<para>Internally, this is implemented by prepending
the task (e.g. "task-compile:") to the value of
- <link linkend='var-OVERRIDES'><filename>OVERRIDES</filename></link>
+ <link linkend='var-bb-OVERRIDES'><filename>OVERRIDES</filename></link>
for the local datastore of the <filename>do_compile</filename>
task.</para>
@@ -724,17 +799,15 @@
<title>Key Expansion</title>
<para>
- Key expansion happens when the BitBake datastore is finalized
- just before BitBake expands overrides.
+ Key expansion happens when the BitBake datastore is finalized.
To better understand this, consider the following example:
<literallayout class='monospaced'>
A${B} = "X"
B = "2"
A2 = "Y"
</literallayout>
- In this case, after all the parsing is complete, and
- before any overrides are handled, BitBake expands
- <filename>${B}</filename> into "2".
+ In this case, after all the parsing is complete,
+ BitBake expands <filename>${B}</filename> into "2".
This expansion causes <filename>A2</filename>, which was
set to "Y" before the expansion, to become "X".
</para>
@@ -868,7 +941,7 @@
<para>
BitBake uses the
- <link linkend='var-BBPATH'><filename>BBPATH</filename></link>
+ <link linkend='var-bb-BBPATH'><filename>BBPATH</filename></link>
variable to locate needed include and class files.
Additionally, BitBake searches the current directory for
<filename>include</filename> and <filename>require</filename>
@@ -1086,7 +1159,7 @@
<para>
When creating a configuration file (<filename>.conf</filename>),
you can use the
- <link linkend='var-INHERIT'><filename>INHERIT</filename></link>
+ <link linkend='var-bb-INHERIT'><filename>INHERIT</filename></link>
configuration directive to inherit a class.
BitBake only supports this directive when used within
a configuration file.
@@ -1370,7 +1443,7 @@
</para></listitem>
<listitem><para>
BitBake-style Python functions generate a separate
- <filename>${</filename><link linkend='var-T'><filename>T</filename></link><filename>}/run.</filename><replaceable>function-name</replaceable><filename>.</filename><replaceable>pid</replaceable>
+ <filename>${</filename><link linkend='var-bb-T'><filename>T</filename></link><filename>}/run.</filename><replaceable>function-name</replaceable><filename>.</filename><replaceable>pid</replaceable>
script that is executed to run the function, and also
generate a log file in
<filename>${T}/log.</filename><replaceable>function-name</replaceable><filename>.</filename><replaceable>pid</replaceable>
@@ -1773,7 +1846,7 @@
things exported or listed in its whitelist to ensure that the build
environment is reproducible and consistent.
You can prevent this "cleaning" by setting the
- <link linkend='var-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link>
+ <link linkend='var-bb-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link>
variable.
</note>
Consequently, if you do want something to get passed into the
@@ -1783,9 +1856,9 @@
Tell BitBake to load what you want from the environment
into the datastore.
You can do so through the
- <link linkend='var-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>
+ <link linkend='var-bb-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>
and
- <link linkend='var-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link>
+ <link linkend='var-bb-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link>
variables.
For example, assume you want to prevent the build system from
accessing your <filename>$HOME/.ccache</filename>
@@ -1824,7 +1897,7 @@
from the original execution environment.
Bitbake saves a copy of the original environment into
a special variable named
- <link linkend='var-BB_ORIGENV'><filename>BB_ORIGENV</filename></link>.
+ <link linkend='var-bb-BB_ORIGENV'><filename>BB_ORIGENV</filename></link>.
</para>
<para>
@@ -1883,7 +1956,7 @@
<listitem><para><emphasis><filename>[depends]</filename>:</emphasis>
Controls inter-task dependencies.
See the
- <link linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
+ <link linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link>
variable and the
"<link linkend='inter-task-dependencies'>Inter-Task Dependencies</link>"
section for more information.
@@ -1891,7 +1964,7 @@
<listitem><para><emphasis><filename>[deptask]</filename>:</emphasis>
Controls task build-time dependencies.
See the
- <link linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
+ <link linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link>
variable and the
"<link linkend='build-dependencies'>Build Dependencies</link>"
section for more information.
@@ -1937,7 +2010,7 @@
of cores but certain tasks need to be rate-limited due to various
kinds of resource constraints (e.g. to avoid network throttling).
<filename>number_threads</filename> works similarly to the
- <link linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
+ <link linkend='var-bb-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
variable but is task-specific.</para>
<para>Set the value globally.
@@ -1971,9 +2044,9 @@
<listitem><para><emphasis><filename>[rdepends]</filename>:</emphasis>
Controls inter-task runtime dependencies.
See the
- <link linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link>
+ <link linkend='var-bb-RDEPENDS'><filename>RDEPENDS</filename></link>
variable, the
- <link linkend='var-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
+ <link linkend='var-bb-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
variable, and the
"<link linkend='inter-task-dependencies'>Inter-Task Dependencies</link>"
section for more information.
@@ -1981,9 +2054,9 @@
<listitem><para><emphasis><filename>[rdeptask]</filename>:</emphasis>
Controls task runtime dependencies.
See the
- <link linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link>
+ <link linkend='var-bb-RDEPENDS'><filename>RDEPENDS</filename></link>
variable, the
- <link linkend='var-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
+ <link linkend='var-bb-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
variable, and the
"<link linkend='runtime-dependencies'>Runtime Dependencies</link>"
section for more information.
@@ -1996,9 +2069,9 @@
<listitem><para><emphasis><filename>[recrdeptask]</filename>:</emphasis>
Controls task recursive runtime dependencies.
See the
- <link linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link>
+ <link linkend='var-bb-RDEPENDS'><filename>RDEPENDS</filename></link>
variable, the
- <link linkend='var-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
+ <link linkend='var-bb-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
variable, and the
"<link linkend='recursive-dependencies'>Recursive Dependencies</link>"
section for more information.
@@ -2127,7 +2200,7 @@
Any given datastore only has one such event executed
against it, however.
If
- <link linkende='var-BB_INVALIDCONF'><filename>BB_INVALIDCONF</filename></link>
+ <link linkende='var-bb-BB_INVALIDCONF'><filename>BB_INVALIDCONF</filename></link>
is set in the datastore by the event handler, the
configuration is reparsed and a new event triggered,
allowing the metadata to update configuration.
@@ -2256,17 +2329,17 @@
from a single recipe file multiple incarnations of that
recipe file where all incarnations are buildable.
These features are enabled through the
- <link linkend='var-BBCLASSEXTEND'><filename>BBCLASSEXTEND</filename></link>
+ <link linkend='var-bb-BBCLASSEXTEND'><filename>BBCLASSEXTEND</filename></link>
and
- <link linkend='var-BBVERSIONS'><filename>BBVERSIONS</filename></link>
+ <link linkend='var-bb-BBVERSIONS'><filename>BBVERSIONS</filename></link>
variables.
<note>
The mechanism for this class extension is extremely
specific to the implementation.
Usually, the recipe's
- <link linkend='var-PROVIDES'><filename>PROVIDES</filename></link>,
- <link linkend='var-PN'><filename>PN</filename></link>, and
- <link linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
+ <link linkend='var-bb-PROVIDES'><filename>PROVIDES</filename></link>,
+ <link linkend='var-bb-PN'><filename>PN</filename></link>, and
+ <link linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link>
variables would need to be modified by the extension class.
For specific examples, see the OE-Core
<filename>native</filename>, <filename>nativesdk</filename>,
@@ -2287,7 +2360,7 @@
project from a single recipe file.
You can also specify conditional metadata
(using the
- <link linkend='var-OVERRIDES'><filename>OVERRIDES</filename></link>
+ <link linkend='var-bb-OVERRIDES'><filename>OVERRIDES</filename></link>
mechanism) for a single version, or an optionally named range of versions.
Here is an example:
<literallayout class='monospaced'>
@@ -2306,7 +2379,7 @@
into overrides, but it is also made available for the metadata to use
in the variable that defines the base recipe versions for use in
<filename>file://</filename> search paths
- (<link linkend='var-FILESPATH'><filename>FILESPATH</filename></link>).
+ (<link linkend='var-bb-FILESPATH'><filename>FILESPATH</filename></link>).
</para></listitem>
</itemizedlist>
</para>
@@ -2408,7 +2481,7 @@
<para>
BitBake uses the
- <link linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
+ <link linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link>
variable to manage build time dependencies.
The <filename>[deptask]</filename> varflag for tasks
signifies the task of each
@@ -2429,9 +2502,9 @@
<para>
BitBake uses the
- <link linkend='var-PACKAGES'><filename>PACKAGES</filename></link>,
- <link linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link>, and
- <link linkend='var-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
+ <link linkend='var-bb-PACKAGES'><filename>PACKAGES</filename></link>,
+ <link linkend='var-bb-RDEPENDS'><filename>RDEPENDS</filename></link>, and
+ <link linkend='var-bb-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
variables to manage runtime dependencies.
</para>
@@ -2686,7 +2759,7 @@
<para>
These checksums are stored in
- <link linkend='var-STAMP'><filename>STAMP</filename></link>.
+ <link linkend='var-bb-STAMP'><filename>STAMP</filename></link>.
You can examine the checksums using the following BitBake command:
<literallayout class='monospaced'>
$ bitbake-dumpsigs
@@ -2708,44 +2781,44 @@
The following list describes related variables:
<itemizedlist>
<listitem><para>
- <link linkend='var-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></link>:
+ <link linkend='var-bb-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></link>:
Specifies the name of the function to call during
the "setscene" part of the task's execution in order
to validate the list of task hashes.
</para></listitem>
<listitem><para>
- <link linkend='var-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></link>:
+ <link linkend='var-bb-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></link>:
Specifies a function BitBake calls that determines
whether BitBake requires a setscene dependency to
be met.
</para></listitem>
<listitem><para>
- <link linkend='var-BB_SETSCENE_VERIFY_FUNCTION2'><filename>BB_SETSCENE_VERIFY_FUNCTION2</filename></link>:
+ <link linkend='var-bb-BB_SETSCENE_VERIFY_FUNCTION2'><filename>BB_SETSCENE_VERIFY_FUNCTION2</filename></link>:
Specifies a function to call that verifies the list of
planned task execution before the main task execution
happens.
</para></listitem>
<listitem><para>
- <link linkend='var-BB_STAMP_POLICY'><filename>BB_STAMP_POLICY</filename></link>:
+ <link linkend='var-bb-BB_STAMP_POLICY'><filename>BB_STAMP_POLICY</filename></link>:
Defines the mode for comparing timestamps of stamp files.
</para></listitem>
<listitem><para>
- <link linkend='var-BB_STAMP_WHITELIST'><filename>BB_STAMP_WHITELIST</filename></link>:
+ <link linkend='var-bb-BB_STAMP_WHITELIST'><filename>BB_STAMP_WHITELIST</filename></link>:
Lists stamp files that are looked at when the stamp policy
is "whitelist".
</para></listitem>
<listitem><para>
- <link linkend='var-BB_TASKHASH'><filename>BB_TASKHASH</filename></link>:
+ <link linkend='var-bb-BB_TASKHASH'><filename>BB_TASKHASH</filename></link>:
Within an executing task, this variable holds the hash
of the task as returned by the currently enabled
signature generator.
</para></listitem>
<listitem><para>
- <link linkend='var-STAMP'><filename>STAMP</filename></link>:
+ <link linkend='var-bb-STAMP'><filename>STAMP</filename></link>:
The base path to create stamp files.
</para></listitem>
<listitem><para>
- <link linkend='var-STAMPCLEAN'><filename>STAMPCLEAN</filename></link>:
+ <link linkend='var-bb-STAMPCLEAN'><filename>STAMPCLEAN</filename></link>:
Again, the base path to create stamp files but can use wildcards
for matching a range of files for clean operations.
</para></listitem>
diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml
index a84b2bc..aca6741 100644
--- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml
+++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml
@@ -3,7 +3,7 @@
[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
<!-- Dummy chapter -->
-<chapter id='ref-variables-glos'>
+<chapter id='ref-bb-variables-glos'>
<title>Variables Glossary</title>
@@ -34,29 +34,29 @@
</itemizedlist>
</note>
-<glossary id='ref-variables-glossary'>
+<glossary id='ref-bb-variables-glossary'>
<para>
- <link linkend='var-ASSUME_PROVIDED'>A</link>
- <link linkend='var-B'>B</link>
- <link linkend='var-CACHE'>C</link>
- <link linkend='var-DEFAULT_PREFERENCE'>D</link>
- <link linkend='var-EXCLUDE_FROM_WORLD'>E</link>
- <link linkend='var-FAKEROOT'>F</link>
- <link linkend='var-GITDIR'>G</link>
- <link linkend='var-HGDIR'>H</link>
-<!-- <link linkend='var-ICECC_DISABLED'>I</link> -->
+ <link linkend='var-bb-ASSUME_PROVIDED'>A</link>
+ <link linkend='var-bb-B'>B</link>
+ <link linkend='var-bb-CACHE'>C</link>
+ <link linkend='var-bb-DEFAULT_PREFERENCE'>D</link>
+ <link linkend='var-bb-EXCLUDE_FROM_WORLD'>E</link>
+ <link linkend='var-bb-FAKEROOT'>F</link>
+ <link linkend='var-bb-GITDIR'>G</link>
+ <link linkend='var-bb-HGDIR'>H</link>
+ <link linkend='var-bb-INHERIT'>I</link>
<!-- <link linkend='var-glossary-j'>J</link> -->
<!-- <link linkend='var-KARCH'>K</link> -->
- <link linkend='var-LAYERDEPENDS'>L</link>
- <link linkend='var-MIRRORS'>M</link>
+ <link linkend='var-bb-LAYERDEPENDS'>L</link>
+ <link linkend='var-bb-MIRRORS'>M</link>
<!-- <link linkend='var-glossary-n'>N</link> -->
- <link linkend='var-OVERRIDES'>O</link>
- <link linkend='var-P4DIR'>P</link>
+ <link linkend='var-bb-OVERRIDES'>O</link>
+ <link linkend='var-bb-P4DIR'>P</link>
<!-- <link linkend='var-QMAKE_PROFILES'>Q</link> -->
- <link linkend='var-RDEPENDS'>R</link>
- <link linkend='var-SECTION'>S</link>
- <link linkend='var-T'>T</link>
+ <link linkend='var-bb-RDEPENDS'>R</link>
+ <link linkend='var-bb-SECTION'>S</link>
+ <link linkend='var-bb-T'>T</link>
<!-- <link linkend='var-UBOOT_CONFIG'>U</link> -->
<!-- <link linkend='var-glossary-v'>V</link> -->
<!-- <link linkend='var-WARN_QA'>W</link> -->
@@ -65,13 +65,13 @@
<!-- <link linkend='var-glossary-z'>Z</link>-->
</para>
- <glossdiv id='var-glossary-a'><title>A</title>
+ <glossdiv id='var-bb-glossary-a'><title>A</title>
- <glossentry id='var-ASSUME_PROVIDED'><glossterm>ASSUME_PROVIDED</glossterm>
+ <glossentry id='var-bb-ASSUME_PROVIDED'><glossterm>ASSUME_PROVIDED</glossterm>
<glossdef>
<para>
Lists recipe names
- (<link linkend='var-PN'><filename>PN</filename></link>
+ (<link linkend='var-bb-PN'><filename>PN</filename></link>
values) BitBake does not attempt to build.
Instead, BitBake assumes these recipes have already been
built.
@@ -91,9 +91,9 @@
</glossdiv>
- <glossdiv id='var-glossary-b'><title>B</title>
+ <glossdiv id='var-bb-glossary-b'><title>B</title>
- <glossentry id='var-B'><glossterm>B</glossterm>
+ <glossentry id='var-bb-B'><glossterm>B</glossterm>
<glossdef>
<para>
The directory in which BitBake executes functions
@@ -102,7 +102,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BB_ALLOWED_NETWORKS'><glossterm>BB_ALLOWED_NETWORKS</glossterm>
+ <glossentry id='var-bb-BB_ALLOWED_NETWORKS'><glossterm>BB_ALLOWED_NETWORKS</glossterm>
<glossdef>
<para>
Specifies a space-delimited list of hosts that the fetcher
@@ -111,7 +111,7 @@
<itemizedlist>
<listitem><para>
This host list is only used if
- <link linkend='var-BB_NO_NETWORK'><filename>BB_NO_NETWORK</filename></link>
+ <link linkend='var-bb-BB_NO_NETWORK'><filename>BB_NO_NETWORK</filename></link>
is either not set or set to "0".
</para></listitem>
<listitem><para>
@@ -151,13 +151,13 @@
</itemizedlist>
Using <filename>BB_ALLOWED_NETWORKS</filename> in
conjunction with
- <link linkend='var-PREMIRRORS'><filename>PREMIRRORS</filename></link>
+ <link linkend='var-bb-PREMIRRORS'><filename>PREMIRRORS</filename></link>
is very useful.
Adding the host you want to use to
<filename>PREMIRRORS</filename> results in the source code
being fetched from an allowed location and avoids raising
an error when a host that is not allowed is in a
- <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
+ <link linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link>
statement.
This is because the fetcher does not attempt to use the
host listed in <filename>SRC_URI</filename> after a
@@ -167,7 +167,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BB_CONSOLELOG'><glossterm>BB_CONSOLELOG</glossterm>
+ <glossentry id='var-bb-BB_CONSOLELOG'><glossterm>BB_CONSOLELOG</glossterm>
<glossdef>
<para>
Specifies the path to a log file into which BitBake's user
@@ -176,7 +176,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BB_CURRENTTASK'><glossterm>BB_CURRENTTASK</glossterm>
+ <glossentry id='var-bb-BB_CURRENTTASK'><glossterm>BB_CURRENTTASK</glossterm>
<glossdef>
<para>
Contains the name of the currently running task.
@@ -186,7 +186,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BB_DANGLINGAPPENDS_WARNONLY'><glossterm>BB_DANGLINGAPPENDS_WARNONLY</glossterm>
+ <glossentry id='var-bb-BB_DANGLINGAPPENDS_WARNONLY'><glossterm>BB_DANGLINGAPPENDS_WARNONLY</glossterm>
<glossdef>
<para>
Defines how BitBake handles situations where an append
@@ -208,7 +208,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BB_DEFAULT_TASK'><glossterm>BB_DEFAULT_TASK</glossterm>
+ <glossentry id='var-bb-BB_DEFAULT_TASK'><glossterm>BB_DEFAULT_TASK</glossterm>
<glossdef>
<para>
The default task to use when none is specified (e.g.
@@ -219,7 +219,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BB_DISKMON_DIRS'><glossterm>BB_DISKMON_DIRS</glossterm>
+ <glossentry id='var-bb-BB_DISKMON_DIRS'><glossterm>BB_DISKMON_DIRS</glossterm>
<glossdef>
<para>
Monitors disk space and available inodes during the build
@@ -245,7 +245,7 @@
build when a threshold is broken.
Subsequent warnings are issued as
defined by the
- <link linkend='var-BB_DISKMON_WARNINTERVAL'>BB_DISKMON_WARNINTERVAL</link> variable,
+ <link linkend='var-bb-BB_DISKMON_WARNINTERVAL'>BB_DISKMON_WARNINTERVAL</link> variable,
which must be defined.
<dir> is:
@@ -275,7 +275,7 @@
BB_DISKMON_DIRS = "ABORT,${TMPDIR},,100K"
</literallayout>
The first example works only if you also set
- the <link linkend='var-BB_DISKMON_WARNINTERVAL'><filename>BB_DISKMON_WARNINTERVAL</filename></link> variable.
+ the <link linkend='var-bb-BB_DISKMON_WARNINTERVAL'><filename>BB_DISKMON_WARNINTERVAL</filename></link> variable.
This example causes the build system to immediately
abort when either the disk space in <filename>${TMPDIR}</filename> drops
below 1 Gbyte or the available free inodes drops below
@@ -309,7 +309,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BB_DISKMON_WARNINTERVAL'><glossterm>BB_DISKMON_WARNINTERVAL</glossterm>
+ <glossentry id='var-bb-BB_DISKMON_WARNINTERVAL'><glossterm>BB_DISKMON_WARNINTERVAL</glossterm>
<glossdef>
<para>
Defines the disk space and free inode warning intervals.
@@ -319,7 +319,7 @@
If you are going to use the
<filename>BB_DISKMON_WARNINTERVAL</filename> variable, you must
also use the
- <link linkend='var-BB_DISKMON_DIRS'><filename>BB_DISKMON_DIRS</filename></link> variable
+ <link linkend='var-bb-BB_DISKMON_DIRS'><filename>BB_DISKMON_DIRS</filename></link> variable
and define its action as "WARN".
During the build, subsequent warnings are issued each time
disk space or number of free inodes further reduces by
@@ -374,7 +374,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BB_ENV_WHITELIST'><glossterm>BB_ENV_WHITELIST</glossterm>
+ <glossentry id='var-bb-BB_ENV_WHITELIST'><glossterm>BB_ENV_WHITELIST</glossterm>
<glossdef>
<para>
Specifies the internal whitelist of variables to allow
@@ -382,11 +382,11 @@
datastore.
If the value of this variable is not specified
(which is the default), the following list is used:
- <link linkend='var-BBPATH'><filename>BBPATH</filename></link>,
- <link linkend='var-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link>,
- <link linkend='var-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>,
+ <link linkend='var-bb-BBPATH'><filename>BBPATH</filename></link>,
+ <link linkend='var-bb-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link>,
+ <link linkend='var-bb-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>,
and
- <link linkend='var-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link>.
+ <link linkend='var-bb-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link>.
<note>
You must set this variable in the external environment
in order for it to work.
@@ -395,7 +395,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BB_ENV_EXTRAWHITE'><glossterm>BB_ENV_EXTRAWHITE</glossterm>
+ <glossentry id='var-bb-BB_ENV_EXTRAWHITE'><glossterm>BB_ENV_EXTRAWHITE</glossterm>
<glossdef>
<para>
Specifies an additional set of variables to allow through
@@ -403,7 +403,7 @@
datastore.
This list of variables are on top of the internal list
set in
- <link linkend='var-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>.
+ <link linkend='var-bb-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>.
<note>
You must set this variable in the external
environment in order for it to work.
@@ -412,22 +412,22 @@
</glossdef>
</glossentry>
- <glossentry id='var-BB_FETCH_PREMIRRORONLY'><glossterm>BB_FETCH_PREMIRRORONLY</glossterm>
+ <glossentry id='var-bb-BB_FETCH_PREMIRRORONLY'><glossterm>BB_FETCH_PREMIRRORONLY</glossterm>
<glossdef>
<para>
When set to "1", causes BitBake's fetcher module to only
search
- <link linkend='var-PREMIRRORS'><filename>PREMIRRORS</filename></link>
+ <link linkend='var-bb-PREMIRRORS'><filename>PREMIRRORS</filename></link>
for files.
BitBake will not search the main
- <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
+ <link linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link>
or
- <link linkend='var-MIRRORS'><filename>MIRRORS</filename></link>.
+ <link linkend='var-bb-MIRRORS'><filename>MIRRORS</filename></link>.
</para>
</glossdef>
</glossentry>
- <glossentry id='var-BB_FILENAME'><glossterm>BB_FILENAME</glossterm>
+ <glossentry id='var-bb-BB_FILENAME'><glossterm>BB_FILENAME</glossterm>
<glossdef>
<para>
Contains the filename of the recipe that owns the currently
@@ -440,12 +440,12 @@
</glossdef>
</glossentry>
- <glossentry id='var-BB_GENERATE_MIRROR_TARBALLS'><glossterm>BB_GENERATE_MIRROR_TARBALLS</glossterm>
+ <glossentry id='var-bb-BB_GENERATE_MIRROR_TARBALLS'><glossterm>BB_GENERATE_MIRROR_TARBALLS</glossterm>
<glossdef>
<para>
Causes tarballs of the Git repositories, including the
Git metadata, to be placed in the
- <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
+ <link linkend='var-bb-DL_DIR'><filename>DL_DIR</filename></link>
directory.
Anyone wishing to create a source mirror would want to
enable this variable.
@@ -461,7 +461,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BB_HASHCONFIG_WHITELIST'><glossterm>BB_HASHCONFIG_WHITELIST</glossterm>
+ <glossentry id='var-bb-BB_HASHCONFIG_WHITELIST'><glossterm>BB_HASHCONFIG_WHITELIST</glossterm>
<glossdef>
<para>
Lists variables that are excluded from base configuration
@@ -485,7 +485,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BB_HASHBASE_WHITELIST'><glossterm>BB_HASHBASE_WHITELIST</glossterm>
+ <glossentry id='var-bb-BB_HASHBASE_WHITELIST'><glossterm>BB_HASHBASE_WHITELIST</glossterm>
<glossdef>
<para>
Lists variables that are excluded from checksum and
@@ -500,7 +500,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BB_HASHCHECK_FUNCTION'><glossterm>BB_HASHCHECK_FUNCTION</glossterm>
+ <glossentry id='var-bb-BB_HASHCHECK_FUNCTION'><glossterm>BB_HASHCHECK_FUNCTION</glossterm>
<glossdef>
<para>
Specifies the name of the function to call during the
@@ -524,7 +524,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BB_INVALIDCONF'><glossterm>BB_INVALIDCONF</glossterm>
+ <glossentry id='var-bb-BB_INVALIDCONF'><glossterm>BB_INVALIDCONF</glossterm>
<glossdef>
<para>
Used in combination with the
@@ -539,11 +539,11 @@
</glossdef>
</glossentry>
- <glossentry id='var-BB_LOGFMT'><glossterm>BB_LOGFMT</glossterm>
+ <glossentry id='var-bb-BB_LOGFMT'><glossterm>BB_LOGFMT</glossterm>
<glossdef>
<para>
Specifies the name of the log files saved into
- <filename>${</filename><link linkend='var-T'><filename>T</filename></link><filename>}</filename>.
+ <filename>${</filename><link linkend='var-bb-T'><filename>T</filename></link><filename>}</filename>.
By default, the <filename>BB_LOGFMT</filename> variable
is undefined and the log file names get created using the
following form:
@@ -556,7 +556,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BB_NICE_LEVEL'><glossterm>BB_NICE_LEVEL</glossterm>
+ <glossentry id='var-bb-BB_NICE_LEVEL'><glossterm>BB_NICE_LEVEL</glossterm>
<glossdef>
<para>
Allows BitBake to run at a specific priority
@@ -564,13 +564,13 @@
System permissions usually mean that BitBake can reduce its
priority but not raise it again.
See
- <link linkend='var-BB_TASK_NICE_LEVEL'><filename>BB_TASK_NICE_LEVEL</filename></link>
+ <link linkend='var-bb-BB_TASK_NICE_LEVEL'><filename>BB_TASK_NICE_LEVEL</filename></link>
for additional information.
</para>
</glossdef>
</glossentry>
- <glossentry id='var-BB_NO_NETWORK'><glossterm>BB_NO_NETWORK</glossterm>
+ <glossentry id='var-bb-BB_NO_NETWORK'><glossterm>BB_NO_NETWORK</glossterm>
<glossdef>
<para>
Disables network access in the BitBake fetcher modules.
@@ -587,7 +587,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BB_NUMBER_THREADS'><glossterm>BB_NUMBER_THREADS</glossterm>
+ <glossentry id='var-bb-BB_NUMBER_THREADS'><glossterm>BB_NUMBER_THREADS</glossterm>
<glossdef>
<para>
The maximum number of tasks BitBake should run in parallel
@@ -599,7 +599,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BB_NUMBER_PARSE_THREADS'><glossterm>BB_NUMBER_PARSE_THREADS</glossterm>
+ <glossentry id='var-bb-BB_NUMBER_PARSE_THREADS'><glossterm>BB_NUMBER_PARSE_THREADS</glossterm>
<glossdef>
<para>
Sets the number of threads BitBake uses when parsing.
@@ -609,7 +609,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BB_ORIGENV'><glossterm>BB_ORIGENV</glossterm>
+ <glossentry id='var-bb-BB_ORIGENV'><glossterm>BB_ORIGENV</glossterm>
<glossdef>
<para>
Contains a copy of the original external environment in
@@ -625,7 +625,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BB_PRESERVE_ENV'><glossterm>BB_PRESERVE_ENV</glossterm>
+ <glossentry id='var-bb-BB_PRESERVE_ENV'><glossterm>BB_PRESERVE_ENV</glossterm>
<glossdef>
<para>
Disables whitelisting and instead allows all variables
@@ -639,12 +639,12 @@
</glossdef>
</glossentry>
- <glossentry id='var-BB_RUNFMT'><glossterm>BB_RUNFMT</glossterm>
+ <glossentry id='var-bb-BB_RUNFMT'><glossterm>BB_RUNFMT</glossterm>
<glossdef>
<para>
Specifies the name of the executable script files
(i.e. run files) saved into
- <filename>${</filename><link linkend='var-T'><filename>T</filename></link><filename>}</filename>.
+ <filename>${</filename><link linkend='var-bb-T'><filename>T</filename></link><filename>}</filename>.
By default, the <filename>BB_RUNFMT</filename> variable
is undefined and the run file names get created using the
following form:
@@ -657,7 +657,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BB_RUNTASK'><glossterm>BB_RUNTASK</glossterm>
+ <glossentry id='var-bb-BB_RUNTASK'><glossterm>BB_RUNTASK</glossterm>
<glossdef>
<para>
Contains the name of the currently executing task.
@@ -669,7 +669,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BB_SCHEDULER'><glossterm>BB_SCHEDULER</glossterm>
+ <glossentry id='var-bb-BB_SCHEDULER'><glossterm>BB_SCHEDULER</glossterm>
<glossdef>
<para>
Selects the name of the scheduler to use for the
@@ -695,7 +695,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BB_SCHEDULERS'><glossterm>BB_SCHEDULERS</glossterm>
+ <glossentry id='var-bb-BB_SCHEDULERS'><glossterm>BB_SCHEDULERS</glossterm>
<glossdef>
<para>
Defines custom schedulers to import.
@@ -705,13 +705,13 @@
<para>
For information how to select a scheduler, see the
- <link linkend='var-BB_SCHEDULER'><filename>BB_SCHEDULER</filename></link>
+ <link linkend='var-bb-BB_SCHEDULER'><filename>BB_SCHEDULER</filename></link>
variable.
</para>
</glossdef>
</glossentry>
- <glossentry id='var-BB_SETSCENE_DEPVALID'><glossterm>BB_SETSCENE_DEPVALID</glossterm>
+ <glossentry id='var-bb-BB_SETSCENE_DEPVALID'><glossterm>BB_SETSCENE_DEPVALID</glossterm>
<glossdef>
<para>
Specifies a function BitBake calls that determines
@@ -731,7 +731,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BB_SETSCENE_VERIFY_FUNCTION2'><glossterm>BB_SETSCENE_VERIFY_FUNCTION2</glossterm>
+ <glossentry id='var-bb-BB_SETSCENE_VERIFY_FUNCTION2'><glossterm>BB_SETSCENE_VERIFY_FUNCTION2</glossterm>
<glossdef>
<para>
Specifies a function to call that verifies the list of
@@ -752,7 +752,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BB_SIGNATURE_EXCLUDE_FLAGS'><glossterm>BB_SIGNATURE_EXCLUDE_FLAGS</glossterm>
+ <glossentry id='var-bb-BB_SIGNATURE_EXCLUDE_FLAGS'><glossterm>BB_SIGNATURE_EXCLUDE_FLAGS</glossterm>
<glossdef>
<para>
Lists variable flags (varflags)
@@ -771,7 +771,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BB_SIGNATURE_HANDLER'><glossterm>BB_SIGNATURE_HANDLER</glossterm>
+ <glossentry id='var-bb-BB_SIGNATURE_HANDLER'><glossterm>BB_SIGNATURE_HANDLER</glossterm>
<glossdef>
<para>
Defines the name of the signature handler BitBake uses.
@@ -790,7 +790,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BB_SRCREV_POLICY'><glossterm>BB_SRCREV_POLICY</glossterm>
+ <glossentry id='var-bb-BB_SRCREV_POLICY'><glossterm>BB_SRCREV_POLICY</glossterm>
<glossdef>
<para>
Defines the behavior of the fetcher when it interacts with
@@ -817,7 +817,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BB_STAMP_POLICY'><glossterm>BB_STAMP_POLICY</glossterm>
+ <glossentry id='var-bb-BB_STAMP_POLICY'><glossterm>BB_STAMP_POLICY</glossterm>
<glossdef>
<para>
Defines the mode used for how timestamps of stamp files
@@ -836,7 +836,7 @@
<listitem><para><emphasis>whitelist</emphasis> -
Identical to "full" mode except timestamp
comparisons are made for recipes listed in the
- <link linkend='var-BB_STAMP_WHITELIST'><filename>BB_STAMP_WHITELIST</filename></link>
+ <link linkend='var-bb-BB_STAMP_WHITELIST'><filename>BB_STAMP_WHITELIST</filename></link>
variable.
</para></listitem>
</itemizedlist>
@@ -848,19 +848,19 @@
</glossdef>
</glossentry>
- <glossentry id='var-BB_STAMP_WHITELIST'><glossterm>BB_STAMP_WHITELIST</glossterm>
+ <glossentry id='var-bb-BB_STAMP_WHITELIST'><glossterm>BB_STAMP_WHITELIST</glossterm>
<glossdef>
<para>
Lists files whose stamp file timestamps are compared when
the stamp policy mode is set to "whitelist".
For information on stamp policies, see the
- <link linkend='var-BB_STAMP_POLICY'><filename>BB_STAMP_POLICY</filename></link>
+ <link linkend='var-bb-BB_STAMP_POLICY'><filename>BB_STAMP_POLICY</filename></link>
variable.
</para>
</glossdef>
</glossentry>
- <glossentry id='var-BB_STRICT_CHECKSUM'><glossterm>BB_STRICT_CHECKSUM</glossterm>
+ <glossentry id='var-bb-BB_STRICT_CHECKSUM'><glossterm>BB_STRICT_CHECKSUM</glossterm>
<glossdef>
<para>
Sets a more strict checksum mechanism for non-local URLs.
@@ -871,7 +871,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BB_TASK_IONICE_LEVEL'><glossterm>BB_TASK_IONICE_LEVEL</glossterm>
+ <glossentry id='var-bb-BB_TASK_IONICE_LEVEL'><glossterm>BB_TASK_IONICE_LEVEL</glossterm>
<glossdef>
<para>
Allows adjustment of a task's Input/Output priority.
@@ -882,7 +882,7 @@
variable to adjust the I/O priority of these tasks.
<note>
This variable works similarly to the
- <link linkend='var-BB_TASK_NICE_LEVEL'><filename>BB_TASK_NICE_LEVEL</filename></link>
+ <link linkend='var-bb-BB_TASK_NICE_LEVEL'><filename>BB_TASK_NICE_LEVEL</filename></link>
variable except with a task's I/O priorities.
</note>
</para>
@@ -921,7 +921,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BB_TASK_NICE_LEVEL'><glossterm>BB_TASK_NICE_LEVEL</glossterm>
+ <glossentry id='var-bb-BB_TASK_NICE_LEVEL'><glossterm>BB_TASK_NICE_LEVEL</glossterm>
<glossdef>
<para>
Allows specific tasks to change their priority
@@ -940,7 +940,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BB_TASKHASH'><glossterm>BB_TASKHASH</glossterm>
+ <glossentry id='var-bb-BB_TASKHASH'><glossterm>BB_TASKHASH</glossterm>
<glossdef>
<para>
Within an executing task, this variable holds the hash
@@ -950,7 +950,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BB_VERBOSE_LOGS'><glossterm>BB_VERBOSE_LOGS</glossterm>
+ <glossentry id='var-bb-BB_VERBOSE_LOGS'><glossterm>BB_VERBOSE_LOGS</glossterm>
<glossdef>
<para>
Controls how verbose BitBake is during builds.
@@ -960,7 +960,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BB_WORKERCONTEXT'><glossterm>BB_WORKERCONTEXT</glossterm>
+ <glossentry id='var-bb-BB_WORKERCONTEXT'><glossterm>BB_WORKERCONTEXT</glossterm>
<glossdef>
<para>
Specifies if the current context is executing a task.
@@ -973,7 +973,7 @@
</glossentry>
- <glossentry id='var-BBCLASSEXTEND'><glossterm>BBCLASSEXTEND</glossterm>
+ <glossentry id='var-bb-BBCLASSEXTEND'><glossterm>BBCLASSEXTEND</glossterm>
<glossdef>
<para>
Allows you to extend a recipe so that it builds variants
@@ -1009,7 +1009,7 @@
<filename>_class-native</filename>.
For example, to generate a native version of a recipe,
a
- <link linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
+ <link linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link>
on "foo" is rewritten to a <filename>DEPENDS</filename>
on "foo-native".
</para>
@@ -1028,7 +1028,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BBDEBUG'><glossterm>BBDEBUG</glossterm>
+ <glossentry id='var-bb-BBDEBUG'><glossterm>BBDEBUG</glossterm>
<glossdef>
<para>
Sets the BitBake debug output level to a specific value
@@ -1042,7 +1042,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BBFILE_COLLECTIONS'><glossterm>BBFILE_COLLECTIONS</glossterm>
+ <glossentry id='var-bb-BBFILE_COLLECTIONS'><glossterm>BBFILE_COLLECTIONS</glossterm>
<glossdef>
<para>Lists the names of configured layers.
These names are used to find the other <filename>BBFILE_*</filename>
@@ -1053,10 +1053,10 @@
</glossdef>
</glossentry>
- <glossentry id='var-BBFILE_PATTERN'><glossterm>BBFILE_PATTERN</glossterm>
+ <glossentry id='var-bb-BBFILE_PATTERN'><glossterm>BBFILE_PATTERN</glossterm>
<glossdef>
<para>Variable that expands to match files from
- <link linkend='var-BBFILES'><filename>BBFILES</filename></link>
+ <link linkend='var-bb-BBFILES'><filename>BBFILES</filename></link>
in a particular layer.
This variable is used in the <filename>conf/layer.conf</filename> file and must
be suffixed with the name of the specific layer (e.g.
@@ -1064,7 +1064,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BBFILE_PRIORITY'><glossterm>BBFILE_PRIORITY</glossterm>
+ <glossentry id='var-bb-BBFILE_PRIORITY'><glossterm>BBFILE_PRIORITY</glossterm>
<glossdef>
<para>Assigns the priority for recipe files in each layer.</para>
<para>This variable is useful in situations where the same recipe appears in
@@ -1074,7 +1074,7 @@
letting you control the precedence for the multiple layers.
The precedence established through this variable stands regardless of a
recipe's version
- (<link linkend='var-PV'><filename>PV</filename></link> variable).
+ (<link linkend='var-bb-PV'><filename>PV</filename></link> variable).
For example, a layer that has a recipe with a higher <filename>PV</filename> value but for
which the <filename>BBFILE_PRIORITY</filename> is set to have a lower precedence still has a
lower precedence.</para>
@@ -1083,7 +1083,7 @@
For example, the value 6 has a higher precedence than the value 5.
If not specified, the <filename>BBFILE_PRIORITY</filename> variable is set based on layer
dependencies (see the
- <filename><link linkend='var-LAYERDEPENDS'>LAYERDEPENDS</link></filename> variable for
+ <filename><link linkend='var-bb-LAYERDEPENDS'>LAYERDEPENDS</link></filename> variable for
more information.
The default priority, if unspecified
for a layer with no dependencies, is the lowest defined priority + 1
@@ -1095,7 +1095,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BBFILES'><glossterm>BBFILES</glossterm>
+ <glossentry id='var-bb-BBFILES'><glossterm>BBFILES</glossterm>
<glossdef>
<para>
A space-separated list of recipe files BitBake uses to
@@ -1113,7 +1113,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BBINCLUDED'><glossterm>BBINCLUDED</glossterm>
+ <glossentry id='var-bb-BBINCLUDED'><glossterm>BBINCLUDED</glossterm>
<glossdef>
<para>
Contains a space-separated list of all of all files that
@@ -1123,7 +1123,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BBINCLUDELOGS'><glossterm>BBINCLUDELOGS</glossterm>
+ <glossentry id='var-bb-BBINCLUDELOGS'><glossterm>BBINCLUDELOGS</glossterm>
<glossdef>
<para>
If set to a value, enables printing the task log when
@@ -1132,11 +1132,11 @@
</glossdef>
</glossentry>
- <glossentry id='var-BBINCLUDELOGS_LINES'><glossterm>BBINCLUDELOGS_LINES</glossterm>
+ <glossentry id='var-bb-BBINCLUDELOGS_LINES'><glossterm>BBINCLUDELOGS_LINES</glossterm>
<glossdef>
<para>
If
- <link linkend='var-BBINCLUDELOGS'><filename>BBINCLUDELOGS</filename></link>
+ <link linkend='var-bb-BBINCLUDELOGS'><filename>BBINCLUDELOGS</filename></link>
is set, specifies the maximum number of lines from the
task log file to print when reporting a failed task.
If you do not set <filename>BBINCLUDELOGS_LINES</filename>,
@@ -1145,7 +1145,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BBLAYERS'><glossterm>BBLAYERS</glossterm>
+ <glossentry id='var-bb-BBLAYERS'><glossterm>BBLAYERS</glossterm>
<glossdef>
<para>Lists the layers to enable during the build.
This variable is defined in the <filename>bblayers.conf</filename> configuration
@@ -1166,7 +1166,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BBLAYERS_FETCH_DIR'><glossterm>BBLAYERS_FETCH_DIR</glossterm>
+ <glossentry id='var-bb-BBLAYERS_FETCH_DIR'><glossterm>BBLAYERS_FETCH_DIR</glossterm>
<glossdef>
<para>
Sets the base location where layers are stored.
@@ -1178,7 +1178,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BBMASK'><glossterm>BBMASK</glossterm>
+ <glossentry id='var-bb-BBMASK'><glossterm>BBMASK</glossterm>
<glossdef>
<para>
Prevents BitBake from processing recipes and recipe
@@ -1236,7 +1236,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BBMULTICONFIG'><glossterm>BBMULTICONFIG</glossterm>
+ <glossentry id='var-bb-BBMULTICONFIG'><glossterm>BBMULTICONFIG</glossterm>
<info>
BBMULTICONFIG[doc] = "Enables BitBake to perform multiple configuration builds and lists each separate configuration (multiconfig)."
</info>
@@ -1275,7 +1275,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BBPATH'><glossterm>BBPATH</glossterm>
+ <glossentry id='var-bb-BBPATH'><glossterm>BBPATH</glossterm>
<glossdef>
<para>
Used by BitBake to locate class
@@ -1302,7 +1302,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BBSERVER'><glossterm>BBSERVER</glossterm>
+ <glossentry id='var-bb-BBSERVER'><glossterm>BBSERVER</glossterm>
<glossdef>
<para>
Points to the server that runs memory-resident BitBake.
@@ -1312,7 +1312,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BBTARGETS'><glossterm>BBTARGETS</glossterm>
+ <glossentry id='var-bb-BBTARGETS'><glossterm>BBTARGETS</glossterm>
<glossdef>
<para>
Allows you to use a configuration file to add to the list
@@ -1321,14 +1321,14 @@
</glossdef>
</glossentry>
- <glossentry id='var-BBVERSIONS'><glossterm>BBVERSIONS</glossterm>
+ <glossentry id='var-bb-BBVERSIONS'><glossterm>BBVERSIONS</glossterm>
<glossdef>
<para>
Allows a single recipe to build multiple versions of a
project from a single recipe file.
You also able to specify conditional metadata
using the
- <link linkend='var-OVERRIDES'><filename>OVERRIDES</filename></link>
+ <link linkend='var-bb-OVERRIDES'><filename>OVERRIDES</filename></link>
mechanism for a single version or for an optionally named
range of versions.
</para>
@@ -1342,7 +1342,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BITBAKE_UI'><glossterm>BITBAKE_UI</glossterm>
+ <glossentry id='var-bb-BITBAKE_UI'><glossterm>BITBAKE_UI</glossterm>
<glossdef>
<para>
Used to specify the UI module to use when running BitBake.
@@ -1356,7 +1356,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BUILDNAME'><glossterm>BUILDNAME</glossterm>
+ <glossentry id='var-bb-BUILDNAME'><glossterm>BUILDNAME</glossterm>
<glossdef>
<para>
A name assigned to the build.
@@ -1366,7 +1366,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-BZRDIR'><glossterm>BZRDIR</glossterm>
+ <glossentry id='var-bb-BZRDIR'><glossterm>BZRDIR</glossterm>
<glossdef>
<para>
The directory in which files checked out of a Bazaar
@@ -1377,9 +1377,9 @@
</glossdiv>
- <glossdiv id='var-glossary-c'><title>C</title>
+ <glossdiv id='var-bb-glossary-c'><title>C</title>
- <glossentry id='var-CACHE'><glossterm>CACHE</glossterm>
+ <glossentry id='var-bb-CACHE'><glossterm>CACHE</glossterm>
<glossdef>
<para>
Specifies the directory BitBake uses to store a cache
@@ -1389,7 +1389,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-CVSDIR'><glossterm>CVSDIR</glossterm>
+ <glossentry id='var-bb-CVSDIR'><glossterm>CVSDIR</glossterm>
<glossdef>
<para>
The directory in which files checked out under the
@@ -1400,9 +1400,9 @@
</glossdiv>
- <glossdiv id='var-glossary-d'><title>D</title>
+ <glossdiv id='var-bb-glossary-d'><title>D</title>
- <glossentry id='var-DEFAULT_PREFERENCE'><glossterm>DEFAULT_PREFERENCE</glossterm>
+ <glossentry id='var-bb-DEFAULT_PREFERENCE'><glossterm>DEFAULT_PREFERENCE</glossterm>
<glossdef>
<para>
Specifies a weak bias for recipe selection priority.
@@ -1413,20 +1413,20 @@
piece of software.
Using the variable in this way causes the stable version
of the recipe to build by default in the absence of
- <filename><link linkend='var-PREFERRED_VERSION'>PREFERRED_VERSION</link></filename>
+ <filename><link linkend='var-bb-PREFERRED_VERSION'>PREFERRED_VERSION</link></filename>
being used to build the development version.
</para>
<note>
The bias provided by <filename>DEFAULT_PREFERENCE</filename>
is weak and is overridden by
- <filename><link linkend='var-BBFILE_PRIORITY'>BBFILE_PRIORITY</link></filename>
+ <filename><link linkend='var-bb-BBFILE_PRIORITY'>BBFILE_PRIORITY</link></filename>
if that variable is different between two layers
that contain different versions of the same recipe.
</note>
</glossdef>
</glossentry>
- <glossentry id='var-DEPENDS'><glossterm>DEPENDS</glossterm>
+ <glossentry id='var-bb-DEPENDS'><glossterm>DEPENDS</glossterm>
<glossdef>
<para>
Lists a recipe's build-time dependencies
@@ -1451,13 +1451,13 @@
<para>
For information on runtime dependencies, see the
- <link linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link>
+ <link linkend='var-bb-RDEPENDS'><filename>RDEPENDS</filename></link>
variable.
</para>
</glossdef>
</glossentry>
- <glossentry id='var-DESCRIPTION'><glossterm>DESCRIPTION</glossterm>
+ <glossentry id='var-bb-DESCRIPTION'><glossterm>DESCRIPTION</glossterm>
<glossdef>
<para>
A long description for the recipe.
@@ -1465,7 +1465,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-DL_DIR'><glossterm>DL_DIR</glossterm>
+ <glossentry id='var-bb-DL_DIR'><glossterm>DL_DIR</glossterm>
<glossdef>
<para>
The central download directory used by the build process to
@@ -1474,7 +1474,7 @@
suitable for mirroring for everything except Git
repositories.
If you want tarballs of Git repositories, use the
- <link linkend='var-BB_GENERATE_MIRROR_TARBALLS'><filename>BB_GENERATE_MIRROR_TARBALLS</filename></link>
+ <link linkend='var-bb-BB_GENERATE_MIRROR_TARBALLS'><filename>BB_GENERATE_MIRROR_TARBALLS</filename></link>
variable.
</para>
</glossdef>
@@ -1482,9 +1482,9 @@
</glossentry>
</glossdiv>
- <glossdiv id='var-glossary-e'><title>E</title>
+ <glossdiv id='var-bb-glossary-e'><title>E</title>
- <glossentry id='var-EXCLUDE_FROM_WORLD'><glossterm>EXCLUDE_FROM_WORLD</glossterm>
+ <glossentry id='var-bb-EXCLUDE_FROM_WORLD'><glossterm>EXCLUDE_FROM_WORLD</glossterm>
<glossdef>
<para>
Directs BitBake to exclude a recipe from world builds (i.e.
@@ -1512,9 +1512,9 @@
</glossdiv>
- <glossdiv id='var-glossary-f'><title>F</title>
+ <glossdiv id='var-bb-glossary-f'><title>F</title>
- <glossentry id='var-FAKEROOT'><glossterm>FAKEROOT</glossterm>
+ <glossentry id='var-bb-FAKEROOT'><glossterm>FAKEROOT</glossterm>
<glossdef>
<para>
Contains the command to use when running a shell script
@@ -1527,19 +1527,19 @@
</glossdef>
</glossentry>
- <glossentry id='var-FAKEROOTBASEENV'><glossterm>FAKEROOTBASEENV</glossterm>
+ <glossentry id='var-bb-FAKEROOTBASEENV'><glossterm>FAKEROOTBASEENV</glossterm>
<glossdef>
<para>
Lists environment variables to set when executing
the command defined by
- <link linkend='var-FAKEROOTCMD'><filename>FAKEROOTCMD</filename></link>
+ <link linkend='var-bb-FAKEROOTCMD'><filename>FAKEROOTCMD</filename></link>
that starts the bitbake-worker process
in the fakeroot environment.
</para>
</glossdef>
</glossentry>
- <glossentry id='var-FAKEROOTCMD'><glossterm>FAKEROOTCMD</glossterm>
+ <glossentry id='var-bb-FAKEROOTCMD'><glossterm>FAKEROOTCMD</glossterm>
<glossdef>
<para>
Contains the command that starts the bitbake-worker
@@ -1548,7 +1548,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-FAKEROOTDIRS'><glossterm>FAKEROOTDIRS</glossterm>
+ <glossentry id='var-bb-FAKEROOTDIRS'><glossterm>FAKEROOTDIRS</glossterm>
<glossdef>
<para>
Lists directories to create before running a task in
@@ -1557,33 +1557,33 @@
</glossdef>
</glossentry>
- <glossentry id='var-FAKEROOTENV'><glossterm>FAKEROOTENV</glossterm>
+ <glossentry id='var-bb-FAKEROOTENV'><glossterm>FAKEROOTENV</glossterm>
<glossdef>
<para>
Lists environment variables to set when running a task
in the fakeroot environment.
For additional information on environment variables and
the fakeroot environment, see the
- <link linkend='var-FAKEROOTBASEENV'><filename>FAKEROOTBASEENV</filename></link>
+ <link linkend='var-bb-FAKEROOTBASEENV'><filename>FAKEROOTBASEENV</filename></link>
variable.
</para>
</glossdef>
</glossentry>
- <glossentry id='var-FAKEROOTNOENV'><glossterm>FAKEROOTNOENV</glossterm>
+ <glossentry id='var-bb-FAKEROOTNOENV'><glossterm>FAKEROOTNOENV</glossterm>
<glossdef>
<para>
Lists environment variables to set when running a task
that is not in the fakeroot environment.
For additional information on environment variables and
the fakeroot environment, see the
- <link linkend='var-FAKEROOTENV'><filename>FAKEROOTENV</filename></link>
+ <link linkend='var-bb-FAKEROOTENV'><filename>FAKEROOTENV</filename></link>
variable.
</para>
</glossdef>
</glossentry>
- <glossentry id='var-FETCHCMD'><glossterm>FETCHCMD</glossterm>
+ <glossentry id='var-bb-FETCHCMD'><glossterm>FETCHCMD</glossterm>
<glossdef>
<para>
Defines the command the BitBake fetcher module
@@ -1595,7 +1595,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-FILE'><glossterm>FILE</glossterm>
+ <glossentry id='var-bb-FILE'><glossterm>FILE</glossterm>
<glossdef>
<para>
Points at the current file.
@@ -1607,7 +1607,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-FILESPATH'><glossterm>FILESPATH</glossterm>
+ <glossentry id='var-bb-FILESPATH'><glossterm>FILESPATH</glossterm>
<glossdef>
<para>
Specifies directories BitBake uses when searching for
@@ -1625,9 +1625,9 @@
</glossdiv>
- <glossdiv id='var-glossary-g'><title>G</title>
+ <glossdiv id='var-bb-glossary-g'><title>G</title>
- <glossentry id='var-GITDIR'><glossterm>GITDIR</glossterm>
+ <glossentry id='var-bb-GITDIR'><glossterm>GITDIR</glossterm>
<glossdef>
<para>
The directory in which a local copy of a Git repository
@@ -1639,9 +1639,9 @@
</glossdiv>
- <glossdiv id='var-glossary-h'><title>H</title>
+ <glossdiv id='var-bb-glossary-h'><title>H</title>
- <glossentry id='var-HGDIR'><glossterm>HGDIR</glossterm>
+ <glossentry id='var-bb-HGDIR'><glossterm>HGDIR</glossterm>
<glossdef>
<para>
The directory in which files checked out of a Mercurial
@@ -1650,7 +1650,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-HOMEPAGE'><glossterm>HOMEPAGE</glossterm>
+ <glossentry id='var-bb-HOMEPAGE'><glossterm>HOMEPAGE</glossterm>
<glossdef>
<para>Website where more information about the software the recipe is building
can be found.</para>
@@ -1659,9 +1659,9 @@
</glossdiv>
- <glossdiv id='var-glossary-i'><title>I</title>
+ <glossdiv id='var-bb-glossary-i'><title>I</title>
- <glossentry id='var-INHERIT'><glossterm>INHERIT</glossterm>
+ <glossentry id='var-bb-INHERIT'><glossterm>INHERIT</glossterm>
<glossdef>
<para>
Causes the named class or classes to be inherited globally.
@@ -1691,15 +1691,15 @@
</glossdiv>
-->
- <glossdiv id='var-glossary-l'><title>L</title>
+ <glossdiv id='var-bb-glossary-l'><title>L</title>
- <glossentry id='var-LAYERDEPENDS'><glossterm>LAYERDEPENDS</glossterm>
+ <glossentry id='var-bb-LAYERDEPENDS'><glossterm>LAYERDEPENDS</glossterm>
<glossdef>
<para>Lists the layers, separated by spaces, upon which this recipe depends.
Optionally, you can specify a specific layer version for a dependency
by adding it to the end of the layer name with a colon, (e.g. "anotherlayer:3"
to be compared against
- <link linkend='var-LAYERVERSION'><filename>LAYERVERSION</filename></link><filename>_anotherlayer</filename>
+ <link linkend='var-bb-LAYERVERSION'><filename>LAYERVERSION</filename></link><filename>_anotherlayer</filename>
in this case).
BitBake produces an error if any dependency is missing or
the version numbers do not match exactly (if specified).</para>
@@ -1710,7 +1710,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-LAYERDIR'><glossterm>LAYERDIR</glossterm>
+ <glossentry id='var-bb-LAYERDIR'><glossterm>LAYERDIR</glossterm>
<glossdef>
<para>When used inside the <filename>layer.conf</filename> configuration
file, this variable provides the path of the current layer.
@@ -1719,22 +1719,22 @@
</glossdef>
</glossentry>
- <glossentry id='var-LAYERDIR_RE'><glossterm>LAYERDIR_RE</glossterm>
+ <glossentry id='var-bb-LAYERDIR_RE'><glossterm>LAYERDIR_RE</glossterm>
<glossdef>
<para>When used inside the <filename>layer.conf</filename> configuration
file, this variable provides the path of the current layer,
escaped for use in a regular expression
- (<link linkend='var-BBFILE_PATTERN'><filename>BBFILE_PATTERN</filename></link>).
+ (<link linkend='var-bb-BBFILE_PATTERN'><filename>BBFILE_PATTERN</filename></link>).
This variable is not available outside of <filename>layer.conf</filename>
and references are expanded immediately when parsing of the file completes.</para>
</glossdef>
</glossentry>
- <glossentry id='var-LAYERVERSION'><glossterm>LAYERVERSION</glossterm>
+ <glossentry id='var-bb-LAYERVERSION'><glossterm>LAYERVERSION</glossterm>
<glossdef>
<para>Optionally specifies the version of a layer as a single number.
You can use this variable within
- <link linkend='var-LAYERDEPENDS'><filename>LAYERDEPENDS</filename></link>
+ <link linkend='var-bb-LAYERDEPENDS'><filename>LAYERDEPENDS</filename></link>
for another layer in order to depend on a specific version
of the layer.</para>
<para>
@@ -1744,7 +1744,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-LICENSE'><glossterm>LICENSE</glossterm>
+ <glossentry id='var-bb-LICENSE'><glossterm>LICENSE</glossterm>
<glossdef>
<para>
The list of source licenses for the recipe.
@@ -1754,9 +1754,9 @@
</glossdiv>
- <glossdiv id='var-glossary-m'><title>M</title>
+ <glossdiv id='var-bb-glossary-m'><title>M</title>
- <glossentry id='var-MIRRORS'><glossterm>MIRRORS</glossterm>
+ <glossentry id='var-bb-MIRRORS'><glossterm>MIRRORS</glossterm>
<glossdef>
<para>
Specifies additional paths from which BitBake gets source code.
@@ -1764,14 +1764,14 @@
tries the local download directory.
If that location fails, the build system tries locations
defined by
- <link linkend='var-PREMIRRORS'><filename>PREMIRRORS</filename></link>,
+ <link linkend='var-bb-PREMIRRORS'><filename>PREMIRRORS</filename></link>,
the upstream source, and then locations specified by
<filename>MIRRORS</filename> in that order.
</para>
</glossdef>
</glossentry>
- <glossentry id='var-MULTI_PROVIDER_WHITELIST'><glossterm>MULTI_PROVIDER_WHITELIST</glossterm>
+ <glossentry id='var-bb-MULTI_PROVIDER_WHITELIST'><glossterm>MULTI_PROVIDER_WHITELIST</glossterm>
<glossdef>
<para>
Allows you to suppress BitBake warnings caused when
@@ -1804,9 +1804,9 @@
</glossdiv>
-->
- <glossdiv id='var-glossary-o'><title>O</title>
+ <glossdiv id='var-bb-glossary-o'><title>O</title>
- <glossentry id='var-OVERRIDES'><glossterm>OVERRIDES</glossterm>
+ <glossentry id='var-bb-OVERRIDES'><glossterm>OVERRIDES</glossterm>
<glossdef>
<para>
BitBake uses <filename>OVERRIDES</filename> to control
@@ -1829,9 +1829,9 @@
</glossentry>
</glossdiv>
- <glossdiv id='var-glossary-p'><title>P</title>
+ <glossdiv id='var-bb-glossary-p'><title>P</title>
- <glossentry id='var-P4DIR'><glossterm>P4DIR</glossterm>
+ <glossentry id='var-bb-P4DIR'><glossterm>P4DIR</glossterm>
<glossdef>
<para>
The directory in which a local copy of a Perforce depot
@@ -1840,14 +1840,14 @@
</glossdef>
</glossentry>
- <glossentry id='var-PACKAGES'><glossterm>PACKAGES</glossterm>
+ <glossentry id='var-bb-PACKAGES'><glossterm>PACKAGES</glossterm>
<glossdef>
<para>The list of packages the recipe creates.
</para>
</glossdef>
</glossentry>
- <glossentry id='var-PACKAGES_DYNAMIC'><glossterm>PACKAGES_DYNAMIC</glossterm>
+ <glossentry id='var-bb-PACKAGES_DYNAMIC'><glossterm>PACKAGES_DYNAMIC</glossterm>
<glossdef>
<para>
A promise that your recipe satisfies runtime dependencies
@@ -1856,7 +1856,7 @@
does not actually satisfy the dependencies, it only states that
they should be satisfied.
For example, if a hard, runtime dependency
- (<link linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link>)
+ (<link linkend='var-bb-RDEPENDS'><filename>RDEPENDS</filename></link>)
of another package is satisfied during the build
through the <filename>PACKAGES_DYNAMIC</filename>
variable, but a package with the module name is never actually
@@ -1865,7 +1865,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-PE'><glossterm>PE</glossterm>
+ <glossentry id='var-bb-PE'><glossterm>PE</glossterm>
<glossdef>
<para>
The epoch of the recipe.
@@ -1877,7 +1877,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-PERSISTENT_DIR'><glossterm>PERSISTENT_DIR</glossterm>
+ <glossentry id='var-bb-PERSISTENT_DIR'><glossterm>PERSISTENT_DIR</glossterm>
<glossdef>
<para>
Specifies the directory BitBake uses to store data that
@@ -1889,7 +1889,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-PF'><glossterm>PF</glossterm>
+ <glossentry id='var-bb-PF'><glossterm>PF</glossterm>
<glossdef>
<para>
Specifies the recipe or package name and includes all version and revision
@@ -1899,27 +1899,27 @@
</glossdef>
</glossentry>
- <glossentry id='var-PN'><glossterm>PN</glossterm>
+ <glossentry id='var-bb-PN'><glossterm>PN</glossterm>
<glossdef>
<para>The recipe name.</para>
</glossdef>
</glossentry>
- <glossentry id='var-PR'><glossterm>PR</glossterm>
+ <glossentry id='var-bb-PR'><glossterm>PR</glossterm>
<glossdef>
<para>The revision of the recipe.
</para>
</glossdef>
</glossentry>
- <glossentry id='var-PREFERRED_PROVIDER'><glossterm>PREFERRED_PROVIDER</glossterm>
+ <glossentry id='var-bb-PREFERRED_PROVIDER'><glossterm>PREFERRED_PROVIDER</glossterm>
<glossdef>
<para>
Determines which recipe should be given preference when
multiple recipes provide the same item.
You should always suffix the variable with the name of the
provided item, and you should set it to the
- <link linkend='var-PN'><filename>PN</filename></link>
+ <link linkend='var-bb-PN'><filename>PN</filename></link>
of the recipe to which you want to give precedence.
Some examples:
<literallayout class='monospaced'>
@@ -1931,14 +1931,14 @@
</glossdef>
</glossentry>
- <glossentry id='var-PREFERRED_PROVIDERS'><glossterm>PREFERRED_PROVIDERS</glossterm>
+ <glossentry id='var-bb-PREFERRED_PROVIDERS'><glossterm>PREFERRED_PROVIDERS</glossterm>
<glossdef>
<para>
Determines which recipe should be given preference for
cases where multiple recipes provide the same item.
Functionally,
<filename>PREFERRED_PROVIDERS</filename> is identical to
- <link linkend='var-PREFERRED_PROVIDER'><filename>PREFERRED_PROVIDER</filename></link>.
+ <link linkend='var-bb-PREFERRED_PROVIDER'><filename>PREFERRED_PROVIDER</filename></link>.
However, the <filename>PREFERRED_PROVIDERS</filename>
variable lets you define preferences for multiple
situations using the following form:
@@ -1954,15 +1954,15 @@
</glossdef>
</glossentry>
- <glossentry id='var-PREFERRED_VERSION'><glossterm>PREFERRED_VERSION</glossterm>
+ <glossentry id='var-bb-PREFERRED_VERSION'><glossterm>PREFERRED_VERSION</glossterm>
<glossdef>
<para>
If there are multiple versions of recipes available, this
variable determines which recipe should be given preference.
You must always suffix the variable with the
- <link linkend='var-PN'><filename>PN</filename></link>
+ <link linkend='var-bb-PN'><filename>PN</filename></link>
you want to select, and you should set
- <link linkend='var-PV'><filename>PV</filename></link>
+ <link linkend='var-bb-PV'><filename>PV</filename></link>
accordingly for precedence.
</para>
@@ -1989,7 +1989,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-PREMIRRORS'><glossterm>PREMIRRORS</glossterm>
+ <glossentry id='var-bb-PREMIRRORS'><glossterm>PREMIRRORS</glossterm>
<glossdef>
<para>
Specifies additional paths from which BitBake gets source code.
@@ -1998,7 +1998,7 @@
If that location fails, the build system tries locations
defined by <filename>PREMIRRORS</filename>, the upstream
source, and then locations specified by
- <link linkend='var-MIRRORS'><filename>MIRRORS</filename></link>
+ <link linkend='var-bb-MIRRORS'><filename>MIRRORS</filename></link>
in that order.
</para>
@@ -2022,20 +2022,20 @@
</glossdef>
</glossentry>
- <glossentry id='var-PROVIDES'><glossterm>PROVIDES</glossterm>
+ <glossentry id='var-bb-PROVIDES'><glossterm>PROVIDES</glossterm>
<glossdef>
<para>
A list of aliases by which a particular recipe can be
known.
By default, a recipe's own
- <filename><link linkend='var-PN'>PN</link></filename>
+ <filename><link linkend='var-bb-PN'>PN</link></filename>
is implicitly already in its <filename>PROVIDES</filename>
list.
If a recipe uses <filename>PROVIDES</filename>, the
additional aliases are synonyms for the recipe and can
be useful satisfying dependencies of other recipes during
the build as specified by
- <filename><link linkend='var-DEPENDS'>DEPENDS</link></filename>.
+ <filename><link linkend='var-bb-DEPENDS'>DEPENDS</link></filename>.
</para>
<para>
@@ -2059,7 +2059,7 @@
virtual target in <filename>PROVIDES</filename>.
Recipes that depend on the functionality in question can
include the virtual target in
- <link linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
+ <link linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link>
to leave the choice of provider open.
</para>
@@ -2072,11 +2072,11 @@
</glossdef>
</glossentry>
- <glossentry id='var-PRSERV_HOST'><glossterm>PRSERV_HOST</glossterm>
+ <glossentry id='var-bb-PRSERV_HOST'><glossterm>PRSERV_HOST</glossterm>
<glossdef>
<para>
The network based
- <link linkend='var-PR'><filename>PR</filename></link>
+ <link linkend='var-bb-PR'><filename>PR</filename></link>
service host and port.
</para>
@@ -2094,7 +2094,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-PV'><glossterm>PV</glossterm>
+ <glossentry id='var-bb-PV'><glossterm>PV</glossterm>
<glossdef>
<para>The version of the recipe.
</para>
@@ -2108,9 +2108,9 @@
</glossdiv>
-->
- <glossdiv id='var-glossary-r'><title>R</title>
+ <glossdiv id='var-bb-glossary-r'><title>R</title>
- <glossentry id='var-RDEPENDS'><glossterm>RDEPENDS</glossterm>
+ <glossentry id='var-bb-RDEPENDS'><glossterm>RDEPENDS</glossterm>
<glossdef>
<para>
Lists a package's runtime dependencies (i.e. other packages)
@@ -2165,13 +2165,13 @@
<para>
For information on build-time dependencies, see the
- <link linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
+ <link linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link>
variable.
</para>
</glossdef>
</glossentry>
- <glossentry id='var-REPODIR'><glossterm>REPODIR</glossterm>
+ <glossentry id='var-bb-REPODIR'><glossterm>REPODIR</glossterm>
<glossdef>
<para>
The directory in which a local copy of a
@@ -2181,14 +2181,14 @@
</glossdef>
</glossentry>
- <glossentry id='var-RPROVIDES'><glossterm>RPROVIDES</glossterm>
+ <glossentry id='var-bb-RPROVIDES'><glossterm>RPROVIDES</glossterm>
<glossdef>
<para>
A list of package name aliases that a package also provides.
These aliases are useful for satisfying runtime dependencies
of other packages both during the build and on the target
(as specified by
- <filename><link linkend='var-RDEPENDS'>RDEPENDS</link></filename>).
+ <filename><link linkend='var-bb-RDEPENDS'>RDEPENDS</link></filename>).
</para>
<para>
As with all package-controlling variables, you must always
@@ -2201,7 +2201,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-RRECOMMENDS'><glossterm>RRECOMMENDS</glossterm>
+ <glossentry id='var-bb-RRECOMMENDS'><glossterm>RRECOMMENDS</glossterm>
<glossdef>
<para>
A list of packages that extends the usability of a package
@@ -2210,7 +2210,7 @@
packages in order to successfully build, but needs them for
the extended usability.
To specify runtime dependencies for packages, see the
- <filename><link linkend='var-RDEPENDS'>RDEPENDS</link></filename>
+ <filename><link linkend='var-bb-RDEPENDS'>RDEPENDS</link></filename>
variable.
</para>
@@ -2243,15 +2243,15 @@
</glossdiv>
- <glossdiv id='var-glossary-s'><title>S</title>
+ <glossdiv id='var-bb-glossary-s'><title>S</title>
- <glossentry id='var-SECTION'><glossterm>SECTION</glossterm>
+ <glossentry id='var-bb-SECTION'><glossterm>SECTION</glossterm>
<glossdef>
<para>The section in which packages should be categorized.</para>
</glossdef>
</glossentry>
- <glossentry id='var-SRC_URI'><glossterm>SRC_URI</glossterm>
+ <glossentry id='var-bb-SRC_URI'><glossterm>SRC_URI</glossterm>
<glossdef>
<para>
The list of source files - local or remote.
@@ -2272,7 +2272,7 @@
the metadata,
from the local machine.
The path is relative to the
- <link linkend='var-FILESPATH'><filename>FILESPATH</filename></link>
+ <link linkend='var-bb-FILESPATH'><filename>FILESPATH</filename></link>
variable.</para></listitem>
<listitem><para><emphasis><filename>bzr://</filename> -</emphasis> Fetches files from a
Bazaar revision control repository.</para></listitem>
@@ -2322,7 +2322,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-SRCDATE'><glossterm>SRCDATE</glossterm>
+ <glossentry id='var-bb-SRCDATE'><glossterm>SRCDATE</glossterm>
<glossdef>
<para>
The date of the source code used to build the package.
@@ -2331,7 +2331,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-SRCREV'><glossterm>SRCREV</glossterm>
+ <glossentry id='var-bb-SRCREV'><glossterm>SRCREV</glossterm>
<glossdef>
<para>
The revision of the source code used to build the package.
@@ -2344,13 +2344,13 @@
</glossdef>
</glossentry>
- <glossentry id='var-SRCREV_FORMAT'><glossterm>SRCREV_FORMAT</glossterm>
+ <glossentry id='var-bb-SRCREV_FORMAT'><glossterm>SRCREV_FORMAT</glossterm>
<glossdef>
<para>
Helps construct valid
- <link linkend='var-SRCREV'><filename>SRCREV</filename></link>
+ <link linkend='var-bb-SRCREV'><filename>SRCREV</filename></link>
values when multiple source controlled URLs are used in
- <link linkend='var-SRC_URI'><filename>SRC_URI</filename></link>.
+ <link linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link>.
</para>
<para>
@@ -2371,7 +2371,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-STAMP'><glossterm>STAMP</glossterm>
+ <glossentry id='var-bb-STAMP'><glossterm>STAMP</glossterm>
<glossdef>
<para>
Specifies the base path used to create recipe stamp files.
@@ -2381,12 +2381,12 @@
</glossdef>
</glossentry>
- <glossentry id='var-STAMPCLEAN'><glossterm>STAMPCLEAN</glossterm>
+ <glossentry id='var-bb-STAMPCLEAN'><glossterm>STAMPCLEAN</glossterm>
<glossdef>
<para>
Specifies the base path used to create recipe stamp files.
Unlike the
- <link linkend='var-STAMP'><filename>STAMP</filename></link>
+ <link linkend='var-bb-STAMP'><filename>STAMP</filename></link>
variable, <filename>STAMPCLEAN</filename> can contain
wildcards to match the range of files a clean operation
should remove.
@@ -2396,7 +2396,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-SUMMARY'><glossterm>SUMMARY</glossterm>
+ <glossentry id='var-bb-SUMMARY'><glossterm>SUMMARY</glossterm>
<glossdef>
<para>
A short summary for the recipe, which is 72 characters or less.
@@ -2404,7 +2404,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-SVNDIR'><glossterm>SVNDIR</glossterm>
+ <glossentry id='var-bb-SVNDIR'><glossterm>SVNDIR</glossterm>
<glossdef>
<para>
The directory in which files checked out of a Subversion
@@ -2415,9 +2415,9 @@
</glossdiv>
- <glossdiv id='var-glossary-t'><title>T</title>
+ <glossdiv id='var-bb-glossary-t'><title>T</title>
- <glossentry id='var-T'><glossterm>T</glossterm>
+ <glossentry id='var-bb-T'><glossterm>T</glossterm>
<glossdef>
<para>Points to a directory were BitBake places
temporary files, which consist mostly of task logs and
@@ -2426,7 +2426,7 @@
</glossdef>
</glossentry>
- <glossentry id='var-TOPDIR'><glossterm>TOPDIR</glossterm>
+ <glossentry id='var-bb-TOPDIR'><glossterm>TOPDIR</glossterm>
<glossdef>
<para>
Points to the build directory.
diff --git a/bitbake/doc/poky.ent b/bitbake/doc/poky.ent
index c032e14..85d9c83 100644
--- a/bitbake/doc/poky.ent
+++ b/bitbake/doc/poky.ent
@@ -17,13 +17,6 @@
<!ENTITY OE_DOCS_URL "http://docs.openembedded.org">
<!ENTITY OH_HOME_URL "http://o-hand.com">
<!ENTITY BITBAKE_HOME_URL "http://developer.berlios.de/projects/bitbake/">
-<!ENTITY ECLIPSE_MAIN_URL "http://www.eclipse.org/downloads">
-<!ENTITY ECLIPSE_DL_URL "http://download.eclipse.org">
-<!ENTITY ECLIPSE_DL_PLUGIN_URL "&YOCTO_DL_URL;/releases/eclipse-plugin/&DISTRO;">
-<!ENTITY ECLIPSE_UPDATES_URL "&ECLIPSE_DL_URL;/tm/updates/3.3">
-<!ENTITY ECLIPSE_INDIGO_URL "&ECLIPSE_DL_URL;/releases/indigo">
-<!ENTITY ECLIPSE_JUNO_URL "&ECLIPSE_DL_URL;/releases/juno">
-<!ENTITY ECLIPSE_INDIGO_CDT_URL "&ECLIPSE_DL_URL;tools/cdt/releases/indigo">
<!ENTITY YOCTO_DOCS_URL "&YOCTO_HOME_URL;/docs">
<!ENTITY YOCTO_SOURCES_URL "&YOCTO_HOME_URL;/sources/">
<!ENTITY YOCTO_AB_PORT_URL "&YOCTO_AB_URL;:8010">
@@ -31,7 +24,6 @@
<!ENTITY YOCTO_POKY_URL "&YOCTO_DL_URL;/releases/poky/">
<!ENTITY YOCTO_RELEASE_DL_URL "&YOCTO_DL_URL;/releases/yocto/yocto-&DISTRO;">
<!ENTITY YOCTO_TOOLCHAIN_DL_URL "&YOCTO_RELEASE_DL_URL;/toolchain/">
-<!ENTITY YOCTO_ECLIPSE_DL_URL "&YOCTO_RELEASE_DL_URL;/eclipse-plugin/indigo;">
<!ENTITY YOCTO_ADTINSTALLER_DL_URL "&YOCTO_RELEASE_DL_URL;/adt_installer">
<!ENTITY YOCTO_POKY_DL_URL "&YOCTO_RELEASE_DL_URL;/&YOCTO_POKY;.tar.bz2">
<!ENTITY YOCTO_MACHINES_DL_URL "&YOCTO_RELEASE_DL_URL;/machines">
diff --git a/bitbake/lib/bb/COW.py b/bitbake/lib/bb/COW.py
index 7817473..d26e981 100644
--- a/bitbake/lib/bb/COW.py
+++ b/bitbake/lib/bb/COW.py
@@ -1,23 +1,8 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# This is a copy on write dictionary and set which abuses classes to try and be nice and fast.
#
# Copyright (C) 2006 Tim Ansell
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-#
#Please Note:
# Be careful when using mutable types (ie Dict and Lists) - operations involving these are SLOW.
# Assign a file to __warn__ to get warnings about slow operations.
diff --git a/bitbake/lib/bb/__init__.py b/bitbake/lib/bb/__init__.py
index 4bc47c8..c144311 100644
--- a/bitbake/lib/bb/__init__.py
+++ b/bitbake/lib/bb/__init__.py
@@ -1,5 +1,3 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# BitBake Build System Python Library
#
@@ -8,20 +6,10 @@
#
# Based on Gentoo's portage.py.
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-__version__ = "1.40.0"
+__version__ = "1.44.0"
import sys
if sys.version_info < (3, 4, 0):
diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py
index 3e2a94e..30a2ba2 100644
--- a/bitbake/lib/bb/build.py
+++ b/bitbake/lib/bb/build.py
@@ -1,5 +1,3 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# BitBake 'Build' implementation
#
@@ -10,18 +8,7 @@
#
# Based on Gentoo's portage.py.
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+# SPDX-License-Identifier: GPL-2.0-only
#
# Based on functions from the base bb module, Copyright 2003 Holger Schurig
@@ -67,23 +54,6 @@ else:
builtins['bb'] = bb
builtins['os'] = os
-class FuncFailed(Exception):
- def __init__(self, name = None, logfile = None):
- self.logfile = logfile
- self.name = name
- if name:
- self.msg = 'Function failed: %s' % name
- else:
- self.msg = "Function failed"
-
- def __str__(self):
- if self.logfile and os.path.exists(self.logfile):
- msg = ("%s (log file is located at %s)" %
- (self.msg, self.logfile))
- else:
- msg = self.msg
- return msg
-
class TaskBase(event.Event):
"""Base class for task events"""
@@ -176,15 +146,33 @@ class LogTee(object):
def __repr__(self):
return '<LogTee {0}>'.format(self.name)
+
def flush(self):
self.outfile.flush()
-#
-# pythonexception allows the python exceptions generated to be raised
-# as the real exceptions (not FuncFailed) and without a backtrace at the
-# origin of the failure.
-#
-def exec_func(func, d, dirs = None, pythonexception=False):
+
+class StdoutNoopContextManager:
+ """
+ This class acts like sys.stdout, but adds noop __enter__ and __exit__ methods.
+ """
+ def __enter__(self):
+ return sys.stdout
+
+ def __exit__(self, *exc_info):
+ pass
+
+ def write(self, string):
+ return sys.stdout.write(string)
+
+ def flush(self):
+ sys.stdout.flush()
+
+ @property
+ def name(self):
+ return sys.stdout.name
+
+
+def exec_func(func, d, dirs = None):
"""Execute a BB 'function'"""
try:
@@ -256,7 +244,7 @@ def exec_func(func, d, dirs = None, pythonexception=False):
with bb.utils.fileslocked(lockfiles):
if ispython:
- exec_func_python(func, d, runfile, cwd=adir, pythonexception=pythonexception)
+ exec_func_python(func, d, runfile, cwd=adir)
else:
exec_func_shell(func, d, runfile, cwd=adir)
@@ -276,7 +264,7 @@ _functionfmt = """
{function}(d)
"""
logformatter = bb.msg.BBLogFormatter("%(levelname)s: %(message)s")
-def exec_func_python(func, d, runfile, cwd=None, pythonexception=False):
+def exec_func_python(func, d, runfile, cwd=None):
"""Execute a python BB 'function'"""
code = _functionfmt.format(function=func)
@@ -301,13 +289,7 @@ def exec_func_python(func, d, runfile, cwd=None, pythonexception=False):
bb.methodpool.insert_method(func, text, fn, lineno - 1)
comp = utils.better_compile(code, func, "exec_python_func() autogenerated")
- utils.better_exec(comp, {"d": d}, code, "exec_python_func() autogenerated", pythonexception=pythonexception)
- except (bb.parse.SkipRecipe, bb.build.FuncFailed):
- raise
- except:
- if pythonexception:
- raise
- raise FuncFailed(func, None)
+ utils.better_exec(comp, {"d": d}, code, "exec_python_func() autogenerated")
finally:
bb.debug(2, "Python function %s finished" % func)
@@ -335,6 +317,42 @@ trap 'bb_exit_handler' 0
set -e
'''
+def create_progress_handler(func, progress, logfile, d):
+ if progress == 'percent':
+ # Use default regex
+ return bb.progress.BasicProgressHandler(d, outfile=logfile)
+ elif progress.startswith('percent:'):
+ # Use specified regex
+ return bb.progress.BasicProgressHandler(d, regex=progress.split(':', 1)[1], outfile=logfile)
+ elif progress.startswith('outof:'):
+ # Use specified regex
+ return bb.progress.OutOfProgressHandler(d, regex=progress.split(':', 1)[1], outfile=logfile)
+ elif progress.startswith("custom:"):
+ # Use a custom progress handler that was injected via OE_EXTRA_IMPORTS or __builtins__
+ import functools
+ from types import ModuleType
+
+ parts = progress.split(":", 2)
+ _, cls, otherargs = parts[0], parts[1], (parts[2] or None) if parts[2:] else None
+ if cls:
+ def resolve(x, y):
+ if not x:
+ return None
+ if isinstance(x, ModuleType):
+ return getattr(x, y, None)
+ return x.get(y)
+ cls_obj = functools.reduce(resolve, cls.split("."), bb.utils._context)
+ if not cls_obj:
+ # Fall-back on __builtins__
+ cls_obj = functools.reduce(lambda x, y: x.get(y), cls.split("."), __builtins__)
+ if cls_obj:
+ return cls_obj(d, outfile=logfile, otherargs=otherargs)
+ bb.warn('%s: unknown custom progress handler in task progress varflag value "%s", ignoring' % (func, cls))
+ else:
+ bb.warn('%s: invalid task progress varflag value "%s", ignoring' % (func, progress))
+
+ return logfile
+
def exec_func_shell(func, d, runfile, cwd=None):
"""Execute a shell function from the metadata
@@ -372,23 +390,13 @@ exit $ret
cmd = [fakerootcmd, runfile]
if bb.msg.loggerDefaultVerbose:
- logfile = LogTee(logger, sys.stdout)
+ logfile = LogTee(logger, StdoutNoopContextManager())
else:
- logfile = sys.stdout
+ logfile = StdoutNoopContextManager()
progress = d.getVarFlag(func, 'progress')
if progress:
- if progress == 'percent':
- # Use default regex
- logfile = bb.progress.BasicProgressHandler(d, outfile=logfile)
- elif progress.startswith('percent:'):
- # Use specified regex
- logfile = bb.progress.BasicProgressHandler(d, regex=progress.split(':', 1)[1], outfile=logfile)
- elif progress.startswith('outof:'):
- # Use specified regex
- logfile = bb.progress.OutOfProgressHandler(d, regex=progress.split(':', 1)[1], outfile=logfile)
- else:
- bb.warn('%s: invalid task progress varflag value "%s", ignoring' % (func, progress))
+ logfile = create_progress_handler(func, progress, logfile, d)
fifobuffer = bytearray()
def readfifo(data):
@@ -407,6 +415,8 @@ exit $ret
bb.plain(value)
elif cmd == 'bbnote':
bb.note(value)
+ elif cmd == 'bbverbnote':
+ bb.verbnote(value)
elif cmd == 'bbwarn':
bb.warn(value)
elif cmd == 'bberror':
@@ -436,13 +446,8 @@ exit $ret
with open(fifopath, 'r+b', buffering=0) as fifo:
try:
bb.debug(2, "Executing shell function %s" % func)
-
- try:
- with open(os.devnull, 'r+') as stdin:
- bb.process.run(cmd, shell=False, stdin=stdin, log=logfile, extrafiles=[(fifo,readfifo)])
- except bb.process.CmdError:
- logfn = d.getVar('BB_LOGFILE')
- raise FuncFailed(func, logfn)
+ with open(os.devnull, 'r+') as stdin, logfile:
+ bb.process.run(cmd, shell=False, stdin=stdin, log=logfile, extrafiles=[(fifo,readfifo)])
finally:
os.unlink(fifopath)
@@ -570,9 +575,6 @@ def _exec_task(fn, task, d, quieterr):
event.fire(TaskStarted(task, logfn, flags, localdata), localdata)
except (bb.BBHandledException, SystemExit):
return 1
- except FuncFailed as exc:
- logger.error(str(exc))
- return 1
try:
for func in (prefuncs or '').split():
@@ -580,7 +582,10 @@ def _exec_task(fn, task, d, quieterr):
exec_func(task, localdata)
for func in (postfuncs or '').split():
exec_func(func, localdata)
- except FuncFailed as exc:
+ except bb.BBHandledException:
+ event.fire(TaskFailed(task, logfn, localdata, True), localdata)
+ return 1
+ except Exception as exc:
if quieterr:
event.fire(TaskFailedSilent(task, logfn, localdata), localdata)
else:
@@ -588,9 +593,6 @@ def _exec_task(fn, task, d, quieterr):
logger.error(str(exc))
event.fire(TaskFailed(task, logfn, localdata, errprinted), localdata)
return 1
- except bb.BBHandledException:
- event.fire(TaskFailed(task, logfn, localdata, True), localdata)
- return 1
finally:
sys.stdout.flush()
sys.stderr.flush()
@@ -814,6 +816,9 @@ def add_tasks(tasklist, d):
task_deps['parents'][task] = []
if 'deps' in flags:
for dep in flags['deps']:
+ # Check and warn for "addtask task after foo" while foo does not exist
+ #if not dep in tasklist:
+ # bb.warn('%s: dependent task %s for %s does not exist' % (d.getVar('PN'), dep, task))
dep = d.expand(dep)
task_deps['parents'][task].append(dep)
diff --git a/bitbake/lib/bb/cache.py b/bitbake/lib/bb/cache.py
index 258d679..b6f7da5 100644
--- a/bitbake/lib/bb/cache.py
+++ b/bitbake/lib/bb/cache.py
@@ -1,5 +1,3 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# BitBake Cache implementation
#
@@ -15,18 +13,8 @@
# Copyright (C) 2005 Holger Hans Peter Freyther
# Copyright (C) 2005 ROAD GmbH
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import os
import sys
@@ -95,21 +83,21 @@ class CoreRecipeInfo(RecipeInfoCommon):
self.appends = self.listvar('__BBAPPEND', metadata)
self.nocache = self.getvar('BB_DONT_CACHE', metadata)
+ self.provides = self.depvar('PROVIDES', metadata)
+ self.rprovides = self.depvar('RPROVIDES', metadata)
+ self.pn = self.getvar('PN', metadata) or bb.parse.vars_from_file(filename,metadata)[0]
+ self.packages = self.listvar('PACKAGES', metadata)
+ if not self.packages:
+ self.packages.append(self.pn)
+ self.packages_dynamic = self.listvar('PACKAGES_DYNAMIC', metadata)
+
self.skipreason = self.getvar('__SKIPPED', metadata)
if self.skipreason:
- self.pn = self.getvar('PN', metadata) or bb.parse.BBHandler.vars_from_file(filename,metadata)[0]
self.skipped = True
- self.provides = self.depvar('PROVIDES', metadata)
- self.rprovides = self.depvar('RPROVIDES', metadata)
return
self.tasks = metadata.getVar('__BBTASKS', False)
- self.pn = self.getvar('PN', metadata)
- self.packages = self.listvar('PACKAGES', metadata)
- if not self.packages:
- self.packages.append(self.pn)
-
self.basetaskhashes = self.taskvar('BB_BASEHASH', self.tasks, metadata)
self.hashfilename = self.getvar('BB_HASHFILENAME', metadata)
@@ -125,11 +113,8 @@ class CoreRecipeInfo(RecipeInfoCommon):
self.stampclean = self.getvar('STAMPCLEAN', metadata)
self.stamp_extrainfo = self.flaglist('stamp-extra-info', self.tasks, metadata)
self.file_checksums = self.flaglist('file-checksums', self.tasks, metadata, True)
- self.packages_dynamic = self.listvar('PACKAGES_DYNAMIC', metadata)
self.depends = self.depvar('DEPENDS', metadata)
- self.provides = self.depvar('PROVIDES', metadata)
self.rdepends = self.depvar('RDEPENDS', metadata)
- self.rprovides = self.depvar('RPROVIDES', metadata)
self.rrecommends = self.depvar('RRECOMMENDS', metadata)
self.rprovides_pkg = self.pkgvar('RPROVIDES', self.packages, metadata)
self.rdepends_pkg = self.pkgvar('RDEPENDS', self.packages, metadata)
@@ -235,7 +220,7 @@ class CoreRecipeInfo(RecipeInfoCommon):
cachedata.hashfn[fn] = self.hashfilename
for task, taskhash in self.basetaskhashes.items():
- identifier = '%s.%s' % (fn, task)
+ identifier = '%s:%s' % (fn, task)
cachedata.basetaskhash[identifier] = taskhash
cachedata.inherits[fn] = self.inherits
@@ -249,7 +234,7 @@ def virtualfn2realfn(virtualfn):
Convert a virtual file name to a real one + the associated subclass keyword
"""
mc = ""
- if virtualfn.startswith('multiconfig:'):
+ if virtualfn.startswith('mc:'):
elems = virtualfn.split(':')
mc = elems[1]
virtualfn = ":".join(elems[2:])
@@ -270,7 +255,7 @@ def realfn2virtual(realfn, cls, mc):
if cls:
realfn = "virtual:" + cls + ":" + realfn
if mc:
- realfn = "multiconfig:" + mc + ":" + realfn
+ realfn = "mc:" + mc + ":" + realfn
return realfn
def variant2virtual(realfn, variant):
@@ -279,11 +264,11 @@ def variant2virtual(realfn, variant):
"""
if variant == "":
return realfn
- if variant.startswith("multiconfig:"):
+ if variant.startswith("mc:"):
elems = variant.split(":")
if elems[2]:
- return "multiconfig:" + elems[1] + ":virtual:" + ":".join(elems[2:]) + ":" + realfn
- return "multiconfig:" + elems[1] + ":" + realfn
+ return "mc:" + elems[1] + ":virtual:" + ":".join(elems[2:]) + ":" + realfn
+ return "mc:" + elems[1] + ":" + realfn
return "virtual:" + variant + ":" + realfn
def parse_recipe(bb_data, bbfile, appends, mc=''):
@@ -361,7 +346,7 @@ class NoCache(object):
bb_data = self.databuilder.mcdata[mc].createCopy()
newstores = parse_recipe(bb_data, bbfile, appends, mc)
for ns in newstores:
- datastores["multiconfig:%s:%s" % (mc, ns)] = newstores[ns]
+ datastores["mc:%s:%s" % (mc, ns)] = newstores[ns]
return datastores
@@ -411,6 +396,15 @@ class Cache(NoCache):
else:
logger.debug(1, "Cache file %s not found, building..." % self.cachefile)
+ # We don't use the symlink, its just for debugging convinience
+ symlink = os.path.join(self.cachedir, "bb_cache.dat")
+ if os.path.exists(symlink):
+ bb.utils.remove(symlink)
+ try:
+ os.symlink(os.path.basename(self.cachefile), symlink)
+ except OSError:
+ pass
+
def load_cachefile(self):
cachesize = 0
previous_progress = 0
@@ -889,3 +883,56 @@ class MultiProcessCache(object):
p.dump([data, self.__class__.CACHE_VERSION])
bb.utils.unlockfile(glf)
+
+
+class SimpleCache(object):
+ """
+ BitBake multi-process cache implementation
+
+ Used by the codeparser & file checksum caches
+ """
+
+ def __init__(self, version):
+ self.cachefile = None
+ self.cachedata = None
+ self.cacheversion = version
+
+ def init_cache(self, d, cache_file_name=None, defaultdata=None):
+ cachedir = (d.getVar("PERSISTENT_DIR") or
+ d.getVar("CACHE"))
+ if not cachedir:
+ return defaultdata
+
+ bb.utils.mkdirhier(cachedir)
+ self.cachefile = os.path.join(cachedir,
+ cache_file_name or self.__class__.cache_file_name)
+ logger.debug(1, "Using cache in '%s'", self.cachefile)
+
+ glf = bb.utils.lockfile(self.cachefile + ".lock")
+
+ try:
+ with open(self.cachefile, "rb") as f:
+ p = pickle.Unpickler(f)
+ data, version = p.load()
+ except:
+ bb.utils.unlockfile(glf)
+ return defaultdata
+
+ bb.utils.unlockfile(glf)
+
+ if version != self.cacheversion:
+ return defaultdata
+
+ return data
+
+ def save(self, data):
+ if not self.cachefile:
+ return
+
+ glf = bb.utils.lockfile(self.cachefile + ".lock")
+
+ with open(self.cachefile, "wb") as f:
+ p = pickle.Pickler(f, -1)
+ p.dump([data, self.cacheversion])
+
+ bb.utils.unlockfile(glf)
--git a/bitbake/lib/bb/cache_extra.py b/bitbake/lib/bb/cache_extra.py
index 83f4959..bf4226d 100644
--- a/bitbake/lib/bb/cache_extra.py
+++ b/bitbake/lib/bb/cache_extra.py
@@ -1,5 +1,3 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# Extra RecipeInfo will be all defined in this file. Currently,
# Only Hob (Image Creator) Requests some extra fields. So
@@ -12,18 +10,8 @@
# Copyright (C) 2011, Intel Corporation. All rights reserved.
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from bb.cache import RecipeInfoCommon
diff --git a/bitbake/lib/bb/checksum.py b/bitbake/lib/bb/checksum.py
index 4e1598f..5bc8a8f 100644
--- a/bitbake/lib/bb/checksum.py
+++ b/bitbake/lib/bb/checksum.py
@@ -2,18 +2,8 @@
#
# Copyright (C) 2012 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import glob
import operator
diff --git a/bitbake/lib/bb/codeparser.py b/bitbake/lib/bb/codeparser.py
index ddd1b97..fd2c473 100644
--- a/bitbake/lib/bb/codeparser.py
+++ b/bitbake/lib/bb/codeparser.py
@@ -1,3 +1,7 @@
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
"""
BitBake code parser
@@ -33,7 +37,7 @@ from bb.cache import MultiProcessCache
logger = logging.getLogger('BitBake.CodeParser')
def bbhash(s):
- return hashlib.md5(s.encode("utf-8")).hexdigest()
+ return hashlib.sha256(s.encode("utf-8")).hexdigest()
def check_indent(codestr):
"""If the code is indented, add a top level piece of code to 'remove' the indentation"""
@@ -140,7 +144,7 @@ class CodeParserCache(MultiProcessCache):
# so that an existing cache gets invalidated. Additionally you'll need
# to increment __cache_version__ in cache.py in order to ensure that old
# recipe caches don't trigger "Taskhash mismatch" errors.
- CACHE_VERSION = 10
+ CACHE_VERSION = 11
def __init__(self):
MultiProcessCache.__init__(self)
@@ -368,8 +372,9 @@ class ShellParser():
def _parse_shell(self, value):
try:
tokens, _ = pyshyacc.parse(value, eof=True, debug=False)
- except pyshlex.NeedMore:
- raise sherrors.ShellSyntaxError("Unexpected EOF")
+ except Exception:
+ bb.error('Error during parse shell code, the last 5 lines are:\n%s' % '\n'.join(value.split('\n')[-5:]))
+ raise
self.process_tokens(tokens)
diff --git a/bitbake/lib/bb/command.py b/bitbake/lib/bb/command.py
index 6c966e3..378f389 100644
--- a/bitbake/lib/bb/command.py
+++ b/bitbake/lib/bb/command.py
@@ -6,18 +6,8 @@ Provide an interface to interact with the bitbake server through 'commands'
# Copyright (C) 2006-2007 Richard Purdie
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
"""
The bitbake server takes 'commands' from its UI/commandline.
diff --git a/bitbake/lib/bb/compat.py b/bitbake/lib/bb/compat.py
index de1923d..4935668 100644
--- a/bitbake/lib/bb/compat.py
+++ b/bitbake/lib/bb/compat.py
@@ -1,3 +1,7 @@
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
"""Code pulled from future python versions, here for compatibility"""
from collections import MutableMapping, KeysView, ValuesView, ItemsView, OrderedDict
diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index 16681ba..20ef04d 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -1,6 +1,3 @@
-#!/usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# Copyright (C) 2003, 2004 Chris Larson
# Copyright (C) 2003, 2004 Phil Blundell
@@ -9,19 +6,8 @@
# Copyright (C) 2005 ROAD GmbH
# Copyright (C) 2006 - 2007 Richard Purdie
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
import sys, os, glob, os.path, re, time
import atexit
@@ -45,6 +31,7 @@ import pyinotify
import json
import pickle
import codecs
+import hashserv
logger = logging.getLogger("BitBake")
collectlog = logging.getLogger("BitBake.Collection")
@@ -175,27 +162,45 @@ class BBCooker:
self.configuration = configuration
+ bb.debug(1, "BBCooker starting %s" % time.time())
+ sys.stdout.flush()
+
self.configwatcher = pyinotify.WatchManager()
+ bb.debug(1, "BBCooker pyinotify1 %s" % time.time())
+ sys.stdout.flush()
+
self.configwatcher.bbseen = []
self.configwatcher.bbwatchedfiles = []
self.confignotifier = pyinotify.Notifier(self.configwatcher, self.config_notifications)
+ bb.debug(1, "BBCooker pyinotify2 %s" % time.time())
+ sys.stdout.flush()
self.watchmask = pyinotify.IN_CLOSE_WRITE | pyinotify.IN_CREATE | pyinotify.IN_DELETE | \
pyinotify.IN_DELETE_SELF | pyinotify.IN_MODIFY | pyinotify.IN_MOVE_SELF | \
pyinotify.IN_MOVED_FROM | pyinotify.IN_MOVED_TO
self.watcher = pyinotify.WatchManager()
+ bb.debug(1, "BBCooker pyinotify3 %s" % time.time())
+ sys.stdout.flush()
self.watcher.bbseen = []
self.watcher.bbwatchedfiles = []
self.notifier = pyinotify.Notifier(self.watcher, self.notifications)
+ bb.debug(1, "BBCooker pyinotify complete %s" % time.time())
+ sys.stdout.flush()
+
# If being called by something like tinfoil, we need to clean cached data
# which may now be invalid
bb.parse.clear_cache()
bb.parse.BBHandler.cached_statements = {}
self.ui_cmdline = None
+ self.hashserv = None
+ self.hashservaddr = None
self.initConfigurationData()
+ bb.debug(1, "BBCooker parsed base configuration %s" % time.time())
+ sys.stdout.flush()
+
# we log all events to a file if so directed
if self.configuration.writeeventlog:
# register the log file writer as UI Handler
@@ -233,6 +238,9 @@ class BBCooker:
# Let SIGHUP exit as SIGTERM
signal.signal(signal.SIGHUP, self.sigterm_exception)
+ bb.debug(1, "BBCooker startup complete %s" % time.time())
+ sys.stdout.flush()
+
def process_inotify_updates(self):
for n in [self.confignotifier, self.notifier]:
if n.check_events(timeout=0):
@@ -367,13 +375,12 @@ class BBCooker:
# Copy of the data store which has been expanded.
# Used for firing events and accessing variables where expansion needs to be accounted for
#
- bb.parse.init_parser(self.data)
-
if CookerFeatures.BASEDATASTORE_TRACKING in self.featureset:
self.disableDataTracking()
- self.data.renameVar("__depends", "__base_depends")
- self.add_filewatch(self.data.getVar("__base_depends", False), self.configwatcher)
+ for mc in self.databuilder.mcdata.values():
+ mc.renameVar("__depends", "__base_depends")
+ self.add_filewatch(mc.getVar("__base_depends", False), self.configwatcher)
self.baseconfig_valid = True
self.parsecache_valid = False
@@ -385,6 +392,22 @@ class BBCooker:
except prserv.serv.PRServiceConfigError as e:
bb.fatal("Unable to start PR Server, exitting")
+ if self.data.getVar("BB_HASHSERVE") == "auto":
+ # Create a new hash server bound to a unix domain socket
+ if not self.hashserv:
+ dbfile = (self.data.getVar("PERSISTENT_DIR") or self.data.getVar("CACHE")) + "/hashserv.db"
+ self.hashservaddr = "unix://%s/hashserve.sock" % self.data.getVar("TOPDIR")
+ self.hashserv = hashserv.create_server(self.hashservaddr, dbfile, sync=False)
+ self.hashserv.process = multiprocessing.Process(target=self.hashserv.serve_forever)
+ self.hashserv.process.start()
+ self.data.setVar("BB_HASHSERVE", self.hashservaddr)
+ self.databuilder.origdata.setVar("BB_HASHSERVE", self.hashservaddr)
+ self.databuilder.data.setVar("BB_HASHSERVE", self.hashservaddr)
+ for mc in self.databuilder.mcdata:
+ self.databuilder.mcdata[mc].setVar("BB_HASHSERVE", self.hashservaddr)
+
+ bb.parse.init_parser(self.data)
+
def enableDataTracking(self):
self.configuration.tracking = True
if hasattr(self, "data"):
@@ -488,6 +511,7 @@ class BBCooker:
"""
fn = None
envdata = None
+ mc = ''
if not pkgs_to_build:
pkgs_to_build = []
@@ -496,6 +520,12 @@ class BBCooker:
self.enableDataTracking()
self.reset()
+ def mc_base(p):
+ if p.startswith('mc:'):
+ s = p.split(':')
+ if len(s) == 2:
+ return s[1]
+ return None
if buildfile:
# Parse the configuration here. We need to do it explicitly here since
@@ -506,18 +536,16 @@ class BBCooker:
fn = self.matchFile(fn)
fn = bb.cache.realfn2virtual(fn, cls, mc)
elif len(pkgs_to_build) == 1:
- ignore = self.data.getVar("ASSUME_PROVIDED") or ""
- if pkgs_to_build[0] in set(ignore.split()):
- bb.fatal("%s is in ASSUME_PROVIDED" % pkgs_to_build[0])
+ mc = mc_base(pkgs_to_build[0])
+ if not mc:
+ ignore = self.data.getVar("ASSUME_PROVIDED") or ""
+ if pkgs_to_build[0] in set(ignore.split()):
+ bb.fatal("%s is in ASSUME_PROVIDED" % pkgs_to_build[0])
- taskdata, runlist = self.buildTaskData(pkgs_to_build, None, self.configuration.abort, allowincomplete=True)
+ taskdata, runlist = self.buildTaskData(pkgs_to_build, None, self.configuration.abort, allowincomplete=True)
- mc = runlist[0][0]
- fn = runlist[0][3]
- else:
- envdata = self.data
- data.expandKeys(envdata)
- parse.ast.runAnonFuncs(envdata)
+ mc = runlist[0][0]
+ fn = runlist[0][3]
if fn:
try:
@@ -526,6 +554,12 @@ class BBCooker:
except Exception as e:
parselog.exception("Unable to read %s", fn)
raise
+ else:
+ if not mc in self.databuilder.mcdata:
+ bb.fatal('Not multiconfig named "%s" found' % mc)
+ envdata = self.databuilder.mcdata[mc]
+ data.expandKeys(envdata)
+ parse.ast.runAnonFuncs(envdata)
# Display history
with closing(StringIO()) as env:
@@ -565,10 +599,10 @@ class BBCooker:
wildcard = False
# Wild card expansion:
- # Replace string such as "multiconfig:*:bash"
- # into "multiconfig:A:bash multiconfig:B:bash bash"
+ # Replace string such as "mc:*:bash"
+ # into "mc:A:bash mc:B:bash bash"
for k in targetlist:
- if k.startswith("multiconfig:"):
+ if k.startswith("mc:"):
if wildcard:
bb.fatal('multiconfig conflict')
if k.split(":")[1] == "*":
@@ -601,7 +635,7 @@ class BBCooker:
runlist = []
for k in fulltargetlist:
mc = ""
- if k.startswith("multiconfig:"):
+ if k.startswith("mc:"):
mc = k.split(":")[1]
k = ":".join(k.split(":")[2:])
ktask = task
@@ -620,13 +654,22 @@ class BBCooker:
runlist.append([mc, k, ktask, fn])
bb.event.fire(bb.event.TreeDataPreparationProgress(current, len(fulltargetlist)), self.data)
- mcdeps = taskdata[mc].get_mcdepends()
+ havemc = False
+ for mc in self.multiconfigs:
+ if taskdata[mc].get_mcdepends():
+ havemc = True
+
# No need to do check providers if there are no mcdeps or not an mc build
- if mcdeps and mc:
- # Make sure we can provide the multiconfig dependency
+ if havemc or len(self.multiconfigs) > 1:
seen = set()
new = True
+ # Make sure we can provide the multiconfig dependency
while new:
+ mcdeps = set()
+ # Add unresolved first, so we can get multiconfig indirect dependencies on time
+ for mc in self.multiconfigs:
+ taskdata[mc].add_unresolved(localdata[mc], self.recipecaches[mc])
+ mcdeps |= set(taskdata[mc].get_mcdepends())
new = False
for mc in self.multiconfigs:
for k in mcdeps:
@@ -641,6 +684,7 @@ class BBCooker:
taskdata[depmc].add_provider(localdata[depmc], self.recipecaches[depmc], l[3])
seen.add(k)
new = True
+
for mc in self.multiconfigs:
taskdata[mc].add_unresolved(localdata[mc], self.recipecaches[mc])
@@ -676,7 +720,7 @@ class BBCooker:
@staticmethod
def add_mc_prefix(mc, pn):
if mc:
- return "multiconfig:%s:%s" % (mc, pn)
+ return "mc:%s:%s" % (mc, pn)
return pn
def buildDependTree(self, rq, taskdata):
@@ -875,6 +919,10 @@ class BBCooker:
os.unlink('package-depends.dot')
except FileNotFoundError:
pass
+ try:
+ os.unlink('recipe-depends.dot')
+ except FileNotFoundError:
+ pass
with open('task-depends.dot', 'w') as f:
f.write("digraph depends {\n")
@@ -888,27 +936,6 @@ class BBCooker:
f.write("}\n")
logger.info("Task dependencies saved to 'task-depends.dot'")
- with open('recipe-depends.dot', 'w') as f:
- f.write("digraph depends {\n")
- pndeps = {}
- for task in sorted(depgraph["tdepends"]):
- (pn, taskname) = task.rsplit(".", 1)
- if pn not in pndeps:
- pndeps[pn] = set()
- for dep in sorted(depgraph["tdepends"][task]):
- (deppn, deptaskname) = dep.rsplit(".", 1)
- pndeps[pn].add(deppn)
- for pn in sorted(pndeps):
- fn = depgraph["pn"][pn]["filename"]
- version = depgraph["pn"][pn]["version"]
- f.write('"%s" [label="%s\\n%s\\n%s"]\n' % (pn, pn, version, fn))
- for dep in sorted(pndeps[pn]):
- if dep == pn:
- continue
- f.write('"%s" -> "%s"\n' % (pn, dep))
- f.write("}\n")
- logger.info("Flattened recipe dependencies saved to 'recipe-depends.dot'")
-
def show_appends_with_no_recipes(self):
# Determine which bbappends haven't been applied
@@ -1191,8 +1218,8 @@ class BBCooker:
continue
elif regex == "":
parselog.debug(1, "BBFILE_PATTERN_%s is empty" % c)
+ cre = re.compile('^NULL$')
errors = False
- continue
else:
try:
cre = re.compile(regex)
@@ -1453,7 +1480,7 @@ class BBCooker:
ntargets = []
for target in runlist:
if target[0]:
- ntargets.append("multiconfig:%s:%s:%s" % (target[0], target[1], target[2]))
+ ntargets.append("mc:%s:%s:%s" % (target[0], target[1], target[2]))
ntargets.append("%s:%s" % (target[1], target[2]))
for mc in self.multiconfigs:
@@ -1576,6 +1603,9 @@ class BBCooker:
for pkg in pkgs_to_build:
if pkg in ignore:
parselog.warning("Explicit target \"%s\" is in ASSUME_PROVIDED, ignoring" % pkg)
+ if pkg.startswith("multiconfig:"):
+ pkgs_to_build.remove(pkg)
+ pkgs_to_build.append(pkg.replace("multiconfig:", "mc:"))
if 'world' in pkgs_to_build:
pkgs_to_build.remove('world')
@@ -1583,7 +1613,7 @@ class BBCooker:
bb.providers.buildWorldTargetList(self.recipecaches[mc], task)
for t in self.recipecaches[mc].world_target:
if mc:
- t = "multiconfig:" + mc + ":" + t
+ t = "mc:" + mc + ":" + t
pkgs_to_build.append(t)
if 'universe' in pkgs_to_build:
@@ -1602,7 +1632,7 @@ class BBCooker:
bb.debug(1, "Skipping %s for universe tasks as task %s doesn't exist" % (t, task))
continue
if mc:
- t = "multiconfig:" + mc + ":" + t
+ t = "mc:" + mc + ":" + t
pkgs_to_build.append(t)
return pkgs_to_build
@@ -1615,9 +1645,11 @@ class BBCooker:
def post_serve(self):
prserv.serv.auto_shutdown()
+ if self.hashserv:
+ self.hashserv.process.terminate()
+ self.hashserv.process.join()
bb.event.fire(CookerExit(), self.data)
-
def shutdown(self, force = False):
if force:
self.state = state.forceshutdown
@@ -1632,6 +1664,7 @@ class BBCooker:
def reset(self):
self.initConfigurationData()
+ self.handlePRServ()
def clientComplete(self):
"""Called when the client is done using the server"""
@@ -1865,35 +1898,6 @@ class ParsingFailure(Exception):
self.recipe = recipe
Exception.__init__(self, realexception, recipe)
-class Feeder(multiprocessing.Process):
- def __init__(self, jobs, to_parsers, quit):
- self.quit = quit
- self.jobs = jobs
- self.to_parsers = to_parsers
- multiprocessing.Process.__init__(self)
-
- def run(self):
- while True:
- try:
- quit = self.quit.get_nowait()
- except queue.Empty:
- pass
- else:
- if quit == 'cancel':
- self.to_parsers.cancel_join_thread()
- break
-
- try:
- job = self.jobs.pop()
- except IndexError:
- break
-
- try:
- self.to_parsers.put(job, timeout=0.5)
- except queue.Full:
- self.jobs.insert(0, job)
- continue
-
class Parser(multiprocessing.Process):
def __init__(self, jobs, results, quit, init, profile):
self.jobs = jobs
@@ -1940,11 +1944,8 @@ class Parser(multiprocessing.Process):
result = pending.pop()
else:
try:
- job = self.jobs.get(timeout=0.25)
- except queue.Empty:
- continue
-
- if job is None:
+ job = self.jobs.pop()
+ except IndexError:
break
result = self.parse(*job)
@@ -2028,14 +2029,15 @@ class CookerParser(object):
multiprocessing.util.Finalize(None, bb.codeparser.parser_cache_save, exitpriority=1)
multiprocessing.util.Finalize(None, bb.fetch.fetcher_parse_save, exitpriority=1)
- self.feeder_quit = multiprocessing.Queue(maxsize=1)
self.parser_quit = multiprocessing.Queue(maxsize=self.num_processes)
- self.jobs = multiprocessing.Queue(maxsize=self.num_processes)
self.result_queue = multiprocessing.Queue()
- self.feeder = Feeder(self.willparse, self.jobs, self.feeder_quit)
- self.feeder.start()
+
+ def chunkify(lst,n):
+ return [lst[i::n] for i in range(n)]
+ self.jobs = chunkify(self.willparse, self.num_processes)
+
for i in range(0, self.num_processes):
- parser = Parser(self.jobs, self.result_queue, self.parser_quit, init, self.cooker.configuration.profile)
+ parser = Parser(self.jobs[i], self.result_queue, self.parser_quit, init, self.cooker.configuration.profile)
parser.start()
self.process_names.append(parser.name)
self.processes.append(parser)
@@ -2056,17 +2058,20 @@ class CookerParser(object):
self.total)
bb.event.fire(event, self.cfgdata)
- self.feeder_quit.put(None)
for process in self.processes:
self.parser_quit.put(None)
else:
- self.feeder_quit.put('cancel')
-
self.parser_quit.cancel_join_thread()
for process in self.processes:
self.parser_quit.put(None)
- self.jobs.cancel_join_thread()
+ # Cleanup the queue before call process.join(), otherwise there might be
+ # deadlocks.
+ while True:
+ try:
+ self.result_queue.get(timeout=0.25)
+ except queue.Empty:
+ break
for process in self.processes:
if force:
@@ -2074,7 +2079,6 @@ class CookerParser(object):
process.terminate()
else:
process.join()
- self.feeder.join()
sync = threading.Thread(target=self.bb_cache.sync)
sync.start()
diff --git a/bitbake/lib/bb/cookerdata.py b/bitbake/lib/bb/cookerdata.py
index 5df66e6..472423f 100644
--- a/bitbake/lib/bb/cookerdata.py
+++ b/bitbake/lib/bb/cookerdata.py
@@ -1,6 +1,3 @@
-#!/usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# Copyright (C) 2003, 2004 Chris Larson
# Copyright (C) 2003, 2004 Phil Blundell
@@ -9,23 +6,14 @@
# Copyright (C) 2005 ROAD GmbH
# Copyright (C) 2006 Richard Purdie
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import logging
import os
import re
import sys
+import hashlib
from functools import wraps
import bb
from bb import data
@@ -134,6 +122,7 @@ class CookerConfiguration(object):
self.profile = False
self.nosetscene = False
self.setsceneonly = False
+ self.skipsetscene = False
self.invalidate_stamp = False
self.dump_signatures = []
self.dry_run = False
@@ -279,12 +268,13 @@ class CookerDataBuilder(object):
self.mcdata = {}
def parseBaseConfiguration(self):
+ data_hash = hashlib.sha256()
try:
- bb.parse.init_parser(self.basedata)
self.data = self.parseConfigurationFiles(self.prefiles, self.postfiles)
if self.data.getVar("BB_WORKERCONTEXT", False) is None:
bb.fetch.fetcher_init(self.data)
+ bb.parse.init_parser(self.data)
bb.codeparser.parser_cache_init(self.data)
bb.event.fire(bb.event.ConfigParsed(), self.data)
@@ -302,7 +292,7 @@ class CookerDataBuilder(object):
bb.event.fire(bb.event.ConfigParsed(), self.data)
bb.parse.init_parser(self.data)
- self.data_hash = self.data.get_hash()
+ data_hash.update(self.data.get_hash().encode('utf-8'))
self.mcdata[''] = self.data
multiconfig = (self.data.getVar("BBMULTICONFIG") or "").split()
@@ -310,9 +300,11 @@ class CookerDataBuilder(object):
mcdata = self.parseConfigurationFiles(self.prefiles, self.postfiles, config)
bb.event.fire(bb.event.ConfigParsed(), mcdata)
self.mcdata[config] = mcdata
+ data_hash.update(mcdata.get_hash().encode('utf-8'))
if multiconfig:
bb.event.fire(bb.event.MultiConfigParsed(self.mcdata), self.data)
+ self.data_hash = data_hash.hexdigest()
except (SyntaxError, bb.BBHandledException):
raise bb.BBHandledException
except bb.data_smart.ExpansionError as e:
@@ -354,14 +346,24 @@ class CookerDataBuilder(object):
data = parse_config_file(layerconf, data)
layers = (data.getVar('BBLAYERS') or "").split()
+ broken_layers = []
data = bb.data.createCopy(data)
approved = bb.utils.approved_variables()
+
+ # Check whether present layer directories exist
for layer in layers:
if not os.path.isdir(layer):
- parselog.critical("Layer directory '%s' does not exist! "
- "Please check BBLAYERS in %s" % (layer, layerconf))
- sys.exit(1)
+ broken_layers.append(layer)
+
+ if broken_layers:
+ parselog.critical("The following layer directories do not exist:")
+ for layer in broken_layers:
+ parselog.critical(" %s", layer)
+ parselog.critical("Please check BBLAYERS in %s" % (layerconf))
+ sys.exit(1)
+
+ for layer in layers:
parselog.debug(2, "Adding layer %s", layer)
if 'HOME' in approved and '~' in layer:
layer = os.path.expanduser(layer)
@@ -391,7 +393,11 @@ class CookerDataBuilder(object):
bb.fatal("BBFILES_DYNAMIC entries must be of the form <collection name>:<filename pattern>, not:\n %s" % "\n ".join(invalid))
layerseries = set((data.getVar("LAYERSERIES_CORENAMES") or "").split())
+ collections_tmp = collections[:]
for c in collections:
+ collections_tmp.remove(c)
+ if c in collections_tmp:
+ bb.fatal("Found duplicated BBFILE_COLLECTIONS '%s', check bblayers.conf or layer.conf to fix it." % c)
compat = set((data.getVar("LAYERSERIES_COMPAT_%s" % c) or "").split())
if compat and not (compat & layerseries):
bb.fatal("Layer %s is not compatible with the core layer which only supports these series: %s (layer is compatible with %s)"
diff --git a/bitbake/lib/bb/daemonize.py b/bitbake/lib/bb/daemonize.py
index c937675..f01e6ec 100644
--- a/bitbake/lib/bb/daemonize.py
+++ b/bitbake/lib/bb/daemonize.py
@@ -1,3 +1,7 @@
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
"""
Python Daemonizing helper
diff --git a/bitbake/lib/bb/data.py b/bitbake/lib/bb/data.py
index d66d98c..0d75d0c 100644
--- a/bitbake/lib/bb/data.py
+++ b/bitbake/lib/bb/data.py
@@ -1,5 +1,3 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
"""
BitBake 'Data' implementations
@@ -22,18 +20,7 @@ the speed is more critical here.
# Copyright (C) 2003, 2004 Chris Larson
# Copyright (C) 2005 Holger Hans Peter Freyther
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+# SPDX-License-Identifier: GPL-2.0-only
#
# Based on functions from the base bb module, Copyright 2003 Holger Schurig
@@ -143,7 +130,7 @@ def emit_var(var, o=sys.__stdout__, d = init(), all=False):
if all:
oval = d.getVar(var, False)
val = d.getVar(var)
- except (KeyboardInterrupt, bb.build.FuncFailed):
+ except (KeyboardInterrupt):
raise
except Exception as exc:
o.write('# expansion of %s threw %s: %s\n' % (var, exc.__class__.__name__, str(exc)))
@@ -322,8 +309,6 @@ def build_dependencies(key, keys, shelldeps, varflagsexcl, d):
if varflags.get("python"):
value = d.getVarFlag(key, "_content", False)
parser = bb.codeparser.PythonParser(key, logger)
- if value and "\t" in value:
- logger.warning("Variable %s contains tabs, please remove these (%s)" % (key, d.getVar("FILE")))
parser.parse_python(value, filename=varflags.get("filename"), lineno=varflags.get("lineno"))
deps = deps | parser.references
deps = deps | (keys & parser.execs)
@@ -437,8 +422,8 @@ def generate_dependency_hash(tasklist, gendeps, lookupcache, whitelist, fn):
var = lookupcache[dep]
if var is not None:
data = data + str(var)
- k = fn + "." + task
- basehash[k] = hashlib.md5(data.encode("utf-8")).hexdigest()
+ k = fn + ":" + task
+ basehash[k] = hashlib.sha256(data.encode("utf-8")).hexdigest()
taskdeps[task] = alldeps
return taskdeps, basehash
diff --git a/bitbake/lib/bb/data_smart.py b/bitbake/lib/bb/data_smart.py
index 67af380..dd5c618 100644
--- a/bitbake/lib/bb/data_smart.py
+++ b/bitbake/lib/bb/data_smart.py
@@ -1,5 +1,3 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
"""
BitBake Smart Dictionary Implementation
@@ -14,18 +12,8 @@ BitBake build tools.
# Copyright (C) 2005 Uli Luckas
# Copyright (C) 2005 ROAD GmbH
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
# Based on functions from the base bb module, Copyright 2003 Holger Schurig
import copy, re, sys, traceback
@@ -39,10 +27,11 @@ from bb.COW import COWDictBase
logger = logging.getLogger("BitBake.Data")
__setvar_keyword__ = ["_append", "_prepend", "_remove"]
-__setvar_regexp__ = re.compile('(?P<base>.*?)(?P<keyword>_append|_prepend|_remove)(_(?P<add>[^A-Z]*))?$')
-__expand_var_regexp__ = re.compile(r"\${[^{}@\n\t :]+}")
+__setvar_regexp__ = re.compile(r'(?P<base>.*?)(?P<keyword>_append|_prepend|_remove)(_(?P<add>[^A-Z]*))?$')
+__expand_var_regexp__ = re.compile(r"\${[a-zA-Z0-9\-_+./~]+?}")
__expand_python_regexp__ = re.compile(r"\${@.+?}")
-__whitespace_split__ = re.compile('(\s)')
+__whitespace_split__ = re.compile(r'(\s)')
+__override_regexp__ = re.compile(r'[a-z0-9]+')
def infer_caller_details(loginfo, parent = False, varval = True):
"""Save the caller the trouble of specifying everything."""
@@ -597,7 +586,7 @@ class DataSmart(MutableMapping):
# aka pay the cookie monster
override = var[var.rfind('_')+1:]
shortvar = var[:var.rfind('_')]
- while override and override.islower():
+ while override and __override_regexp__.match(override):
if shortvar not in self.overridedata:
self.overridedata[shortvar] = []
if [var, override] not in self.overridedata[shortvar]:
@@ -1073,4 +1062,4 @@ class DataSmart(MutableMapping):
data.update({i:value})
data_str = str([(k, data[k]) for k in sorted(data.keys())])
- return hashlib.md5(data_str.encode("utf-8")).hexdigest()
+ return hashlib.sha256(data_str.encode("utf-8")).hexdigest()
diff --git a/bitbake/lib/bb/event.py b/bitbake/lib/bb/event.py
index 5b1b094..d44621e 100644
--- a/bitbake/lib/bb/event.py
+++ b/bitbake/lib/bb/event.py
@@ -1,5 +1,3 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
"""
BitBake 'Event' implementation
@@ -9,18 +7,8 @@ BitBake build tools.
# Copyright (C) 2003, 2004 Chris Larson
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import os, sys
import warnings
@@ -136,6 +124,7 @@ def fire_class_handlers(event, d):
ui_queue = []
@atexit.register
def print_ui_queue():
+ global ui_queue
"""If we're exiting before a UI has been spawned, display any queued
LogRecords to the console."""
logger = logging.getLogger("BitBake")
@@ -180,6 +169,7 @@ def print_ui_queue():
logger.removeHandler(stderr)
else:
logger.removeHandler(stdout)
+ ui_queue = []
def fire_ui_handlers(event, d):
global _thread_lock
@@ -414,23 +404,6 @@ class RecipeTaskPreProcess(RecipeEvent):
class RecipeParsed(RecipeEvent):
""" Recipe Parsing Complete """
-class StampUpdate(Event):
- """Trigger for any adjustment of the stamp files to happen"""
-
- def __init__(self, targets, stampfns):
- self._targets = targets
- self._stampfns = stampfns
- Event.__init__(self)
-
- def getStampPrefix(self):
- return self._stampfns
-
- def getTargets(self):
- return self._targets
-
- stampPrefix = property(getStampPrefix)
- targets = property(getTargets)
-
class BuildBase(Event):
"""Base class for bitbake build events"""
diff --git a/bitbake/lib/bb/exceptions.py b/bitbake/lib/bb/exceptions.py
index cd71343..ecbad59 100644
--- a/bitbake/lib/bb/exceptions.py
+++ b/bitbake/lib/bb/exceptions.py
@@ -1,3 +1,6 @@
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
import inspect
import traceback
diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
index 572b71a..1f5f8f1 100644
--- a/bitbake/lib/bb/fetch2/__init__.py
+++ b/bitbake/lib/bb/fetch2/__init__.py
@@ -1,5 +1,3 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
"""
BitBake 'Fetch' implementations
@@ -10,18 +8,7 @@ BitBake build tools.
# Copyright (C) 2003, 2004 Chris Larson
# Copyright (C) 2012 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+# SPDX-License-Identifier: GPL-2.0-only
#
# Based on functions from the base bb module, Copyright 2003 Holger Schurig
@@ -256,7 +243,7 @@ class URI(object):
# Identify if the URI is relative or not
if urlp.scheme in self._relative_schemes and \
- re.compile("^\w+:(?!//)").match(uri):
+ re.compile(r"^\w+:(?!//)").match(uri):
self.relative = True
if not self.relative:
@@ -524,7 +511,7 @@ def fetcher_parse_save():
def fetcher_parse_done():
_checksum_cache.save_merge()
-def fetcher_compare_revisions():
+def fetcher_compare_revisions(d):
"""
Compare the revisions in the persistant cache with current values and
return true/false on whether they've changed.
@@ -777,7 +764,8 @@ def get_srcrev(d, method_name='sortable_revision'):
#
format = d.getVar('SRCREV_FORMAT')
if not format:
- raise FetchError("The SRCREV_FORMAT variable must be set when multiple SCMs are used.")
+ raise FetchError("The SRCREV_FORMAT variable must be set when multiple SCMs are used.\n"\
+ "The SCMs are:\n%s" % '\n'.join(scms))
name_to_rev = {}
seenautoinc = False
@@ -855,10 +843,18 @@ def runfetchcmd(cmd, d, quiet=False, cleanup=None, log=None, workdir=None):
if val:
cmd = 'export ' + var + '=\"%s\"; %s' % (val, cmd)
+ # Ensure that a _PYTHON_SYSCONFIGDATA_NAME value set by a recipe
+ # (for example via python3native.bbclass since warrior) is not set for
+ # host Python (otherwise tools like git-make-shallow will fail)
+ cmd = 'unset _PYTHON_SYSCONFIGDATA_NAME; ' + cmd
+
# Disable pseudo as it may affect ssh, potentially causing it to hang.
cmd = 'export PSEUDO_DISABLED=1; ' + cmd
- logger.debug(1, "Running %s", cmd)
+ if workdir:
+ logger.debug(1, "Running '%s' in %s" % (cmd, workdir))
+ else:
+ logger.debug(1, "Running %s", cmd)
success = False
error_message = ""
@@ -894,7 +890,7 @@ def check_network_access(d, info, url):
log remote network access, and error if BB_NO_NETWORK is set or the given
URI is untrusted
"""
- if d.getVar("BB_NO_NETWORK") == "1":
+ if bb.utils.to_boolean(d.getVar("BB_NO_NETWORK")):
raise NetworkAccess(url, info)
elif not trusted_network(d, url):
raise UntrustedUrl(url, info)
@@ -966,7 +962,8 @@ def rename_bad_checksum(ud, suffix):
new_localpath = "%s_bad-checksum_%s" % (ud.localpath, suffix)
bb.warn("Renaming %s to %s" % (ud.localpath, new_localpath))
- bb.utils.movefile(ud.localpath, new_localpath)
+ if not bb.utils.movefile(ud.localpath, new_localpath):
+ bb.warn("Renaming %s to %s failed, grep movefile in log.do_fetch to see why" % (ud.localpath, new_localpath))
def try_mirror_url(fetch, origud, ud, ld, check = False):
@@ -1027,7 +1024,7 @@ def try_mirror_url(fetch, origud, ud, ld, check = False):
raise
except IOError as e:
- if e.errno in [os.errno.ESTALE]:
+ if e.errno in [errno.ESTALE]:
logger.warning("Stale Error Observed %s." % ud.url)
return False
raise
@@ -1094,7 +1091,7 @@ def trusted_network(d, url):
BB_ALLOWED_NETWORKS is set globally or for a specific recipe.
Note: modifies SRC_URI & mirrors.
"""
- if d.getVar('BB_NO_NETWORK') == "1":
+ if bb.utils.to_boolean(d.getVar("BB_NO_NETWORK")):
return True
pkgname = d.expand(d.getVar('PN', False))
@@ -1403,7 +1400,7 @@ class FetchMethod(object):
Fetch urls
Assumes localpath was called first
"""
- raise NoMethodError(url)
+ raise NoMethodError(urldata.url)
def unpack(self, urldata, rootdir, data):
iterate = False
@@ -1469,7 +1466,7 @@ class FetchMethod(object):
else:
cmd = 'rpm2cpio.sh %s | cpio -id' % (file)
elif file.endswith('.deb') or file.endswith('.ipk'):
- output = subprocess.check_output('ar -t %s' % file, preexec_fn=subprocess_setup, shell=True)
+ output = subprocess.check_output(['ar', '-t', file], preexec_fn=subprocess_setup)
datafile = None
if output:
for line in output.decode().splitlines():
@@ -1547,7 +1544,7 @@ class FetchMethod(object):
Check the status of a URL
Assumes localpath was called first
"""
- logger.info("URL %s could not be checked for status since no method exists.", url)
+ logger.info("URL %s could not be checked for status since no method exists.", urldata.url)
return True
def latest_revision(self, ud, d, name):
@@ -1555,7 +1552,7 @@ class FetchMethod(object):
Look in the cache for the latest revision, if not present ask the SCM.
"""
if not hasattr(self, "_latest_revision"):
- raise ParameterError("The fetcher for this URL does not support _latest_revision", url)
+ raise ParameterError("The fetcher for this URL does not support _latest_revision", ud.url)
revs = bb.persist_data.persist('BB_URI_HEADREVS', d)
key = self.generate_revision_key(ud, d, name)
@@ -1638,7 +1635,7 @@ class Fetch(object):
urls = self.urls
network = self.d.getVar("BB_NO_NETWORK")
- premirroronly = (self.d.getVar("BB_FETCH_PREMIRRORONLY") == "1")
+ premirroronly = bb.utils.to_boolean(self.d.getVar("BB_FETCH_PREMIRRORONLY"))
for u in urls:
ud = self.ud[u]
@@ -1716,7 +1713,7 @@ class Fetch(object):
update_stamp(ud, self.d)
except IOError as e:
- if e.errno in [os.errno.ESTALE]:
+ if e.errno in [errno.ESTALE]:
logger.error("Stale Error Observed %s." % u)
raise ChecksumError("Stale Error Detected")
@@ -1786,7 +1783,7 @@ class Fetch(object):
for url in urls:
if url not in self.ud:
- self.ud[url] = FetchData(url, d)
+ self.ud[url] = FetchData(url, self.d)
ud = self.ud[url]
ud.setup_localpath(self.d)
diff --git a/bitbake/lib/bb/fetch2/bzr.py b/bitbake/lib/bb/fetch2/bzr.py
index 658502f..c56d875 100644
--- a/bitbake/lib/bb/fetch2/bzr.py
+++ b/bitbake/lib/bb/fetch2/bzr.py
@@ -10,18 +10,8 @@ BitBake 'Fetch' implementation for bzr.
# BitBake build tools.
# Copyright (C) 2003, 2004 Chris Larson
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import os
import sys
diff --git a/bitbake/lib/bb/fetch2/clearcase.py b/bitbake/lib/bb/fetch2/clearcase.py
index 3a6573d..3dd93ad 100644
--- a/bitbake/lib/bb/fetch2/clearcase.py
+++ b/bitbake/lib/bb/fetch2/clearcase.py
@@ -1,5 +1,3 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
"""
BitBake 'Fetch' clearcase implementation
@@ -47,18 +45,7 @@ User credentials:
"""
# Copyright (C) 2014 Siemens AG
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+# SPDX-License-Identifier: GPL-2.0-only
#
import os
@@ -67,6 +54,8 @@ import shutil
import bb
from bb.fetch2 import FetchMethod
from bb.fetch2 import FetchError
+from bb.fetch2 import MissingParameterError
+from bb.fetch2 import ParameterError
from bb.fetch2 import runfetchcmd
from bb.fetch2 import logger
@@ -92,7 +81,7 @@ class ClearCase(FetchMethod):
if 'protocol' in ud.parm:
ud.proto = ud.parm['protocol']
if not ud.proto in ('http', 'https'):
- raise fetch2.ParameterError("Invalid protocol type", ud.url)
+ raise ParameterError("Invalid protocol type", ud.url)
ud.vob = ''
if 'vob' in ud.parm:
diff --git a/bitbake/lib/bb/fetch2/cvs.py b/bitbake/lib/bb/fetch2/cvs.py
index 0e0a319..1b35ba4 100644
--- a/bitbake/lib/bb/fetch2/cvs.py
+++ b/bitbake/lib/bb/fetch2/cvs.py
@@ -1,5 +1,3 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
"""
BitBake 'Fetch' implementations
@@ -10,20 +8,9 @@ BitBake build tools.
# Copyright (C) 2003, 2004 Chris Larson
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-#
-#Based on functions from the base bb module, Copyright 2003 Holger Schurig
+# Based on functions from the base bb module, Copyright 2003 Holger Schurig
#
import os
diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index 59a2ee8..2d1d2ca 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -1,5 +1,3 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
"""
BitBake 'Fetch' git implementation
@@ -55,20 +53,10 @@ Supported SRC_URI options are:
"""
-#Copyright (C) 2005 Richard Purdie
+# Copyright (C) 2005 Richard Purdie
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import collections
import errno
@@ -199,7 +187,7 @@ class Git(FetchMethod):
depth_default = 1
ud.shallow_depths = collections.defaultdict(lambda: depth_default)
- revs_default = d.getVar("BB_GIT_SHALLOW_REVS", True)
+ revs_default = d.getVar("BB_GIT_SHALLOW_REVS")
ud.shallow_revs = []
ud.branches = {}
for pos, name in enumerate(ud.names):
@@ -318,7 +306,7 @@ class Git(FetchMethod):
def try_premirror(self, ud, d):
# If we don't do this, updating an existing checkout with only premirrors
# is not possible
- if d.getVar("BB_FETCH_PREMIRRORONLY") is not None:
+ if bb.utils.to_boolean(d.getVar("BB_FETCH_PREMIRRORONLY")):
return True
if os.path.exists(ud.clonedir):
return False
@@ -476,6 +464,8 @@ class Git(FetchMethod):
if os.path.exists(destdir):
bb.utils.prunedir(destdir)
+ need_lfs = ud.parm.get("lfs", "1") == "1"
+
source_found = False
source_error = []
@@ -503,6 +493,13 @@ class Git(FetchMethod):
repourl = self._get_repo_url(ud)
runfetchcmd("%s remote set-url origin %s" % (ud.basecmd, repourl), d, workdir=destdir)
+
+ if self._contains_lfs(ud, d, destdir):
+ if need_lfs and not self._find_git_lfs(d):
+ raise bb.fetch2.FetchError("Repository %s has LFS content, install git-lfs on host to download (or set lfs=0 to ignore it)" % (repourl))
+ else:
+ bb.note("Repository %s has LFS content but it is not being fetched" % (repourl))
+
if not ud.nocheckout:
if subdir != "":
runfetchcmd("%s read-tree %s%s" % (ud.basecmd, ud.revisions[ud.names[0]], readpathspec), d,
@@ -522,9 +519,17 @@ class Git(FetchMethod):
def clean(self, ud, d):
""" clean the git directory """
- bb.utils.remove(ud.localpath, True)
- bb.utils.remove(ud.fullmirror)
- bb.utils.remove(ud.fullmirror + ".done")
+ to_remove = [ud.localpath, ud.fullmirror, ud.fullmirror + ".done"]
+ # The localpath is a symlink to clonedir when it is cloned from a
+ # mirror, so remove both of them.
+ if os.path.islink(ud.localpath):
+ clonedir = os.path.realpath(ud.localpath)
+ to_remove.append(clonedir)
+
+ for r in to_remove:
+ if os.path.exists(r):
+ bb.note('Removing %s' % r)
+ bb.utils.remove(r, True)
def supports_srcrev(self):
return True
@@ -545,6 +550,27 @@ class Git(FetchMethod):
raise bb.fetch2.FetchError("The command '%s' gave output with more then 1 line unexpectedly, output: '%s'" % (cmd, output))
return output.split()[0] != "0"
+ def _contains_lfs(self, ud, d, wd):
+ """
+ Check if the repository has 'lfs' (large file) content
+ """
+ cmd = "%s grep lfs HEAD:.gitattributes | wc -l" % (
+ ud.basecmd)
+ try:
+ output = runfetchcmd(cmd, d, quiet=True, workdir=wd)
+ if int(output) > 0:
+ return True
+ except (bb.fetch2.FetchError,ValueError):
+ pass
+ return False
+
+ def _find_git_lfs(self, d):
+ """
+ Return True if git-lfs can be found, False otherwise.
+ """
+ import shutil
+ return shutil.which("git-lfs", path=d.getVar('PATH')) is not None
+
def _get_repo_url(self, ud):
"""
Return the repository URL
@@ -615,7 +641,7 @@ class Git(FetchMethod):
"""
pupver = ('', '')
- tagregex = re.compile(d.getVar('UPSTREAM_CHECK_GITTAGREGEX') or "(?P<pver>([0-9][\.|_]?)+)")
+ tagregex = re.compile(d.getVar('UPSTREAM_CHECK_GITTAGREGEX') or r"(?P<pver>([0-9][\.|_]?)+)")
try:
output = self._lsremote(ud, d, "refs/tags/*")
except (bb.fetch2.FetchError, bb.fetch2.NetworkAccess) as e:
@@ -630,7 +656,7 @@ class Git(FetchMethod):
tag_head = line.split("/")[-1]
# Ignore non-released branches
- m = re.search("(alpha|beta|rc|final)+", tag_head)
+ m = re.search(r"(alpha|beta|rc|final)+", tag_head)
if m:
continue
diff --git a/bitbake/lib/bb/fetch2/gitannex.py b/bitbake/lib/bb/fetch2/gitannex.py
index a9b69ca..1d497dc 100644
--- a/bitbake/lib/bb/fetch2/gitannex.py
+++ b/bitbake/lib/bb/fetch2/gitannex.py
@@ -1,5 +1,3 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
"""
BitBake 'Fetch' git annex implementation
"""
@@ -7,18 +5,8 @@ BitBake 'Fetch' git annex implementation
# Copyright (C) 2014 Otavio Salvador
# Copyright (C) 2014 O.S. Systems Software LTDA.
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import os
import bb
diff --git a/bitbake/lib/bb/fetch2/gitsm.py b/bitbake/lib/bb/fetch2/gitsm.py
index 35729db..c622771 100644
--- a/bitbake/lib/bb/fetch2/gitsm.py
+++ b/bitbake/lib/bb/fetch2/gitsm.py
@@ -1,5 +1,3 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
"""
BitBake 'Fetch' git submodules implementation
@@ -16,18 +14,8 @@ NOTE: Switching a SRC_URI from "git://" to "gitsm://" requires a clean of your r
# Copyright (C) 2013 Richard Purdie
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import os
import bb
@@ -45,60 +33,97 @@ class GitSM(Git):
"""
return ud.type in ['gitsm']
- @staticmethod
- def parse_gitmodules(gitmodules):
- modules = {}
- module = ""
- for line in gitmodules.splitlines():
- if line.startswith('[submodule'):
- module = line.split('"')[1]
- modules[module] = {}
- elif module and line.strip().startswith('path'):
- path = line.split('=')[1].strip()
- modules[module]['path'] = path
- elif module and line.strip().startswith('url'):
- url = line.split('=')[1].strip()
- modules[module]['url'] = url
- return modules
-
- def update_submodules(self, ud, d):
+ def process_submodules(self, ud, workdir, function, d):
+ """
+ Iterate over all of the submodules in this repository and execute
+ the 'function' for each of them.
+ """
+
submodules = []
paths = {}
+ revision = {}
uris = {}
- local_paths = {}
-
+ subrevision = {}
+
+ def parse_gitmodules(gitmodules):
+ modules = {}
+ module = ""
+ for line in gitmodules.splitlines():
+ if line.startswith('[submodule'):
+ module = line.split('"')[1]
+ modules[module] = {}
+ elif module and line.strip().startswith('path'):
+ path = line.split('=')[1].strip()
+ modules[module]['path'] = path
+ elif module and line.strip().startswith('url'):
+ url = line.split('=')[1].strip()
+ modules[module]['url'] = url
+ return modules
+
+ # Collect the defined submodules, and their attributes
for name in ud.names:
try:
- gitmodules = runfetchcmd("%s show %s:.gitmodules" % (ud.basecmd, ud.revisions[name]), d, quiet=True, workdir=ud.clonedir)
+ gitmodules = runfetchcmd("%s show %s:.gitmodules" % (ud.basecmd, ud.revisions[name]), d, quiet=True, workdir=workdir)
except:
# No submodules to update
continue
- for m, md in self.parse_gitmodules(gitmodules).items():
+ for m, md in parse_gitmodules(gitmodules).items():
+ try:
+ module_hash = runfetchcmd("%s ls-tree -z -d %s %s" % (ud.basecmd, ud.revisions[name], md['path']), d, quiet=True, workdir=workdir)
+ except:
+ # If the command fails, we don't have a valid file to check. If it doesn't
+ # fail -- it still might be a failure, see next check...
+ module_hash = ""
+
+ if not module_hash:
+ logger.debug(1, "submodule %s is defined, but is not initialized in the repository. Skipping", m)
+ continue
+
submodules.append(m)
paths[m] = md['path']
+ revision[m] = ud.revisions[name]
uris[m] = md['url']
+ subrevision[m] = module_hash.split()[2]
+
+ # Convert relative to absolute uri based on parent uri
if uris[m].startswith('..'):
newud = copy.copy(ud)
- newud.path = os.path.realpath(os.path.join(newud.path, md['url']))
+ newud.path = os.path.realpath(os.path.join(newud.path, uris[m]))
uris[m] = Git._get_repo_url(self, newud)
for module in submodules:
- module_hash = runfetchcmd("%s ls-tree -z -d %s %s" % (ud.basecmd, ud.revisions[name], paths[module]), d, quiet=True, workdir=ud.clonedir)
- module_hash = module_hash.split()[2]
+ # Translate the module url into a SRC_URI
+
+ if "://" in uris[module]:
+ # Properly formated URL already
+ proto = uris[module].split(':', 1)[0]
+ url = uris[module].replace('%s:' % proto, 'gitsm:', 1)
+ else:
+ if ":" in uris[module]:
+ # Most likely an SSH style reference
+ proto = "ssh"
+ if ":/" in uris[module]:
+ # Absolute reference, easy to convert..
+ url = "gitsm://" + uris[module].replace(':/', '/', 1)
+ else:
+ # Relative reference, no way to know if this is right!
+ logger.warning("Submodule included by %s refers to relative ssh reference %s. References may fail if not absolute." % (ud.url, uris[module]))
+ url = "gitsm://" + uris[module].replace(':', '/', 1)
+ else:
+ # This has to be a file reference
+ proto = "file"
+ url = "gitsm://" + uris[module]
- # Build new SRC_URI
- proto = uris[module].split(':', 1)[0]
- url = uris[module].replace('%s:' % proto, 'gitsm:', 1)
url += ';protocol=%s' % proto
url += ";name=%s" % module
- url += ";bareclone=1;nocheckout=1;nobranch=1"
+ url += ";subpath=%s" % module
ld = d.createCopy()
# Not necessary to set SRC_URI, since we're passing the URI to
# Fetch.
#ld.setVar('SRC_URI', url)
- ld.setVar('SRCREV_%s' % module, module_hash)
+ ld.setVar('SRCREV_%s' % module, subrevision[module])
# Workaround for issues with SRCPV/SRCREV_FORMAT errors
# error refer to 'multiple' repositories. Only the repository
@@ -106,145 +131,85 @@ class GitSM(Git):
ld.setVar('SRCPV', d.getVar('SRCPV'))
ld.setVar('SRCREV_FORMAT', module)
- newfetch = Fetch([url], ld, cache=False)
- newfetch.download()
- local_paths[module] = newfetch.localpath(url)
+ function(ud, url, module, paths[module], ld)
- # Correct the submodule references to the local download version...
- runfetchcmd("%(basecmd)s config submodule.%(module)s.url %(url)s" % {'basecmd': ud.basecmd, 'module': module, 'url' : local_paths[module]}, d, workdir=ud.clonedir)
-
- symlink_path = os.path.join(ud.clonedir, 'modules', paths[module])
- if not os.path.exists(symlink_path):
- try:
- os.makedirs(os.path.dirname(symlink_path), exist_ok=True)
- except OSError:
- pass
- os.symlink(local_paths[module], symlink_path)
-
- return True
+ return submodules != []
def need_update(self, ud, d):
- main_repo_needs_update = Git.need_update(self, ud, d)
-
- # First check that the main repository has enough history fetched. If it doesn't, then we don't
- # even have the .gitmodules and gitlinks for the submodules to attempt asking whether the
- # submodules' histories are recent enough.
- if main_repo_needs_update:
+ if Git.need_update(self, ud, d):
return True
- # Now check that the submodule histories are new enough. The git-submodule command doesn't have
- # any clean interface for doing this aside from just attempting the checkout (with network
- # fetched disabled).
- return not self.update_submodules(ud, d)
+ try:
+ # Check for the nugget dropped by the download operation
+ known_srcrevs = runfetchcmd("%s config --get-all bitbake.srcrev" % \
+ (ud.basecmd), d, workdir=ud.clonedir)
- def download(self, ud, d):
- Git.download(self, ud, d)
+ if ud.revisions[ud.names[0]] not in known_srcrevs.split():
+ return True
+ except bb.fetch2.FetchError:
+ # No srcrev nuggets, so this is new and needs to be updated
+ return True
- if not ud.shallow or ud.localpath != ud.fullshallow:
- self.update_submodules(ud, d)
+ return False
- def copy_submodules(self, submodules, ud, destdir, d):
- if ud.bareclone:
- repo_conf = destdir
- else:
- repo_conf = os.path.join(destdir, '.git')
+ def download(self, ud, d):
+ def download_submodule(ud, url, module, modpath, d):
+ url += ";bareclone=1;nobranch=1"
- if submodules and not os.path.exists(os.path.join(repo_conf, 'modules')):
- os.mkdir(os.path.join(repo_conf, 'modules'))
+ # Is the following still needed?
+ #url += ";nocheckout=1"
- for module, md in submodules.items():
- srcpath = os.path.join(ud.clonedir, 'modules', md['path'])
- modpath = os.path.join(repo_conf, 'modules', md['path'])
+ try:
+ newfetch = Fetch([url], d, cache=False)
+ newfetch.download()
+ # Drop a nugget to add each of the srcrevs we've fetched (used by need_update)
+ runfetchcmd("%s config --add bitbake.srcrev %s" % \
+ (ud.basecmd, ud.revisions[ud.names[0]]), d, workdir=ud.clonedir)
+ except Exception as e:
+ logger.error('gitsm: submodule download failed: %s %s' % (type(e).__name__, str(e)))
+ raise
- if os.path.exists(srcpath):
- if os.path.exists(os.path.join(srcpath, '.git')):
- srcpath = os.path.join(srcpath, '.git')
+ Git.download(self, ud, d)
+ self.process_submodules(ud, ud.clonedir, download_submodule, d)
- target = modpath
- if os.path.exists(modpath):
- target = os.path.dirname(modpath)
+ def unpack(self, ud, destdir, d):
+ def unpack_submodules(ud, url, module, modpath, d):
+ url += ";bareclone=1;nobranch=1"
- os.makedirs(os.path.dirname(target), exist_ok=True)
- runfetchcmd("cp -fpLR %s %s" % (srcpath, target), d)
- elif os.path.exists(modpath):
- # Module already exists, likely unpacked from a shallow mirror clone
- pass
+ # Figure out where we clone over the bare submodules...
+ if ud.bareclone:
+ repo_conf = ud.destdir
else:
- # This is fatal, as we do NOT want git-submodule to hit the network
- raise bb.fetch2.FetchError('Submodule %s does not exist in %s or %s.' % (module, srcpath, modpath))
-
- def clone_shallow_local(self, ud, dest, d):
- super(GitSM, self).clone_shallow_local(ud, dest, d)
+ repo_conf = os.path.join(ud.destdir, '.git')
- # Copy over the submodules' fetched histories too.
- repo_conf = os.path.join(dest, '.git')
-
- submodules = []
- for name in ud.names:
try:
- gitmodules = runfetchcmd("%s show %s:.gitmodules" % (ud.basecmd, ud.revision), d, quiet=True, workdir=dest)
- except:
- # No submodules to update
- continue
+ newfetch = Fetch([url], d, cache=False)
+ newfetch.unpack(root=os.path.dirname(os.path.join(repo_conf, 'modules', module)))
+ except Exception as e:
+ logger.error('gitsm: submodule unpack failed: %s %s' % (type(e).__name__, str(e)))
+ raise
- submodules = self.parse_gitmodules(gitmodules)
- self.copy_submodules(submodules, ud, dest, d)
+ local_path = newfetch.localpath(url)
- def unpack(self, ud, destdir, d):
- Git.unpack(self, ud, destdir, d)
+ # Correct the submodule references to the local download version...
+ runfetchcmd("%(basecmd)s config submodule.%(module)s.url %(url)s" % {'basecmd': ud.basecmd, 'module': module, 'url' : local_path}, d, workdir=ud.destdir)
- # Copy over the submodules' fetched histories too.
- if ud.bareclone:
- repo_conf = ud.destdir
- else:
- repo_conf = os.path.join(ud.destdir, '.git')
+ if ud.shallow:
+ runfetchcmd("%(basecmd)s config submodule.%(module)s.shallow true" % {'basecmd': ud.basecmd, 'module': module}, d, workdir=ud.destdir)
- update_submodules = False
- paths = {}
- uris = {}
- local_paths = {}
- for name in ud.names:
+ # Ensure the submodule repository is NOT set to bare, since we're checking it out...
try:
- gitmodules = runfetchcmd("%s show HEAD:.gitmodules" % (ud.basecmd), d, quiet=True, workdir=ud.destdir)
+ runfetchcmd("%s config core.bare false" % (ud.basecmd), d, quiet=True, workdir=os.path.join(repo_conf, 'modules', module))
except:
- # No submodules to update
- continue
-
- submodules = self.parse_gitmodules(gitmodules)
- self.copy_submodules(submodules, ud, ud.destdir, d)
-
- submodules_queue = [(module, os.path.join(repo_conf, 'modules', md['path'])) for module, md in submodules.items()]
- while len(submodules_queue) != 0:
- module, modpath = submodules_queue.pop()
-
- # add submodule children recursively
- try:
- gitmodules = runfetchcmd("%s show HEAD:.gitmodules" % (ud.basecmd), d, quiet=True, workdir=modpath)
- for m, md in self.parse_gitmodules(gitmodules).items():
- submodules_queue.append([m, os.path.join(modpath, 'modules', md['path'])])
- except:
- # no children
- pass
-
+ logger.error("Unable to set git config core.bare to false for %s" % os.path.join(repo_conf, 'modules', module))
+ raise
- # There are submodules to update
- update_submodules = True
-
- # Determine (from the submodule) the correct url to reference
- try:
- output = runfetchcmd("%(basecmd)s config remote.origin.url" % {'basecmd': ud.basecmd}, d, workdir=modpath)
- except bb.fetch2.FetchError as e:
- # No remote url defined in this submodule
- continue
-
- local_paths[module] = output
-
- # Setup the local URL properly (like git submodule init or sync would do...)
- runfetchcmd("%(basecmd)s config submodule.%(module)s.url %(url)s" % {'basecmd': ud.basecmd, 'module': module, 'url' : local_paths[module]}, d, workdir=ud.destdir)
+ Git.unpack(self, ud, destdir, d)
- # Ensure the submodule repository is NOT set to bare, since we're checking it out...
- runfetchcmd("%s config core.bare false" % (ud.basecmd), d, quiet=True, workdir=modpath)
+ ret = self.process_submodules(ud, ud.destdir, unpack_submodules, d)
- if update_submodules:
- # Run submodule update, this sets up the directories -- without touching the config
+ if not ud.bareclone and ret:
+ # All submodules should already be downloaded and configured in the tree. This simply sets
+ # up the configuration and checks out the files. The main project config should remain
+ # unmodified, and no download from the internet should occur.
runfetchcmd("%s submodule update --recursive --no-fetch" % (ud.basecmd), d, quiet=True, workdir=ud.destdir)
diff --git a/bitbake/lib/bb/fetch2/hg.py b/bitbake/lib/bb/fetch2/hg.py
index 936d043..15d729e 100644
--- a/bitbake/lib/bb/fetch2/hg.py
+++ b/bitbake/lib/bb/fetch2/hg.py
@@ -1,5 +1,3 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
"""
BitBake 'Fetch' implementation for mercurial DRCS (hg).
@@ -9,20 +7,10 @@ BitBake 'Fetch' implementation for mercurial DRCS (hg).
# Copyright (C) 2004 Marcin Juszkiewicz
# Copyright (C) 2007 Robert Schuster
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+# SPDX-License-Identifier: GPL-2.0-only
#
# Based on functions from the base bb module, Copyright 2003 Holger Schurig
+#
import os
import sys
@@ -99,7 +87,7 @@ class Hg(FetchMethod):
def try_premirror(self, ud, d):
# If we don't do this, updating an existing checkout with only premirrors
# is not possible
- if d.getVar("BB_FETCH_PREMIRRORONLY") is not None:
+ if bb.utils.to_boolean(d.getVar("BB_FETCH_PREMIRRORONLY")):
return True
if os.path.exists(ud.moddir):
return False
diff --git a/bitbake/lib/bb/fetch2/local.py b/bitbake/lib/bb/fetch2/local.py
index a114ac1..01d9ff9 100644
--- a/bitbake/lib/bb/fetch2/local.py
+++ b/bitbake/lib/bb/fetch2/local.py
@@ -1,5 +1,3 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
"""
BitBake 'Fetch' implementations
@@ -10,20 +8,10 @@ BitBake build tools.
# Copyright (C) 2003, 2004 Chris Larson
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+# SPDX-License-Identifier: GPL-2.0-only
#
# Based on functions from the base bb module, Copyright 2003 Holger Schurig
+#
import os
import urllib.request, urllib.parse, urllib.error
diff --git a/bitbake/lib/bb/fetch2/npm.py b/bitbake/lib/bb/fetch2/npm.py
index 65bf5a3..9700e61 100644
--- a/bitbake/lib/bb/fetch2/npm.py
+++ b/bitbake/lib/bb/fetch2/npm.py
@@ -1,5 +1,6 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
"""
BitBake 'Fetch' NPM implementation
@@ -100,11 +101,19 @@ class Npm(FetchMethod):
return False
return True
- def _runwget(self, ud, d, command, quiet):
- logger.debug(2, "Fetching %s using command '%s'" % (ud.url, command))
- bb.fetch2.check_network_access(d, command, ud.url)
+ def _runpack(self, ud, d, pkgfullname: str, quiet=False) -> str:
+ """
+ Runs npm pack on a full package name.
+ Returns the filename of the downloaded package
+ """
+ bb.fetch2.check_network_access(d, pkgfullname, ud.registry)
dldir = d.getVar("DL_DIR")
- runfetchcmd(command, d, quiet, workdir=dldir)
+ dldir = os.path.join(dldir, ud.prefixdir)
+
+ command = "npm pack {} --registry {}".format(pkgfullname, ud.registry)
+ logger.debug(2, "Fetching {} using command '{}' in {}".format(pkgfullname, command, dldir))
+ filename = runfetchcmd(command, d, quiet, workdir=dldir)
+ return filename.rstrip()
def _unpackdep(self, ud, pkg, data, destdir, dldir, d):
file = data[pkg]['tgz']
@@ -150,20 +159,11 @@ class Npm(FetchMethod):
Parse the output of npm view --json; the last JSON result
is assumed to be the one that we're interested in.
'''
- pdata = None
- outdeps = {}
- datalines = []
- bracelevel = 0
- for line in output.splitlines():
- if bracelevel:
- datalines.append(line)
- elif '{' in line:
- datalines = []
- datalines.append(line)
- bracelevel = bracelevel + line.count('{') - line.count('}')
- if datalines:
- pdata = json.loads('\n'.join(datalines))
- return pdata
+ pdata = json.loads(output);
+ try:
+ return pdata[-1]
+ except:
+ return pdata
def _getdependencies(self, pkg, data, version, d, ud, optional=False, fetchedlist=None):
if fetchedlist is None:
@@ -171,6 +171,9 @@ class Npm(FetchMethod):
pkgfullname = pkg
if version != '*' and not '/' in version:
pkgfullname += "@'%s'" % version
+ if pkgfullname in fetchedlist:
+ return
+
logger.debug(2, "Calling getdeps on %s" % pkg)
fetchcmd = "npm view %s --json --registry %s" % (pkgfullname, ud.registry)
output = runfetchcmd(fetchcmd, d, True)
@@ -190,15 +193,10 @@ class Npm(FetchMethod):
if (not blacklist and 'linux' not in pkg_os) or '!linux' in pkg_os:
logger.debug(2, "Skipping %s since it's incompatible with Linux" % pkg)
return
- #logger.debug(2, "Output URL is %s - %s - %s" % (ud.basepath, ud.basename, ud.localfile))
- outputurl = pdata['dist']['tarball']
+ filename = self._runpack(ud, d, pkgfullname)
data[pkg] = {}
- data[pkg]['tgz'] = os.path.basename(outputurl)
- if outputurl in fetchedlist:
- return
-
- self._runwget(ud, d, "%s --directory-prefix=%s %s" % (self.basecmd, ud.prefixdir, outputurl), False)
- fetchedlist.append(outputurl)
+ data[pkg]['tgz'] = filename
+ fetchedlist.append(pkgfullname)
dependencies = pdata.get('dependencies', {})
optionalDependencies = pdata.get('optionalDependencies', {})
@@ -225,17 +223,12 @@ class Npm(FetchMethod):
if obj == pkg:
self._getshrinkeddependencies(obj, data['dependencies'][obj], data['dependencies'][obj]['version'], d, ud, lockdown, manifest, False)
return
- outputurl = "invalid"
- if ('resolved' not in data) or (not data['resolved'].startswith('http://') and not data['resolved'].startswith('https://')):
- # will be the case for ${PN}
- fetchcmd = "npm view %s@%s dist.tarball --registry %s" % (pkg, version, ud.registry)
- logger.debug(2, "Found this matching URL: %s" % str(fetchcmd))
- outputurl = runfetchcmd(fetchcmd, d, True)
- else:
- outputurl = data['resolved']
- self._runwget(ud, d, "%s --directory-prefix=%s %s" % (self.basecmd, ud.prefixdir, outputurl), False)
+
+ pkgnameWithVersion = "{}@{}".format(pkg, version)
+ logger.debug(2, "Get dependencies for {}".format(pkgnameWithVersion))
+ filename = self._runpack(ud, d, pkgnameWithVersion)
manifest[pkg] = {}
- manifest[pkg]['tgz'] = os.path.basename(outputurl).rstrip()
+ manifest[pkg]['tgz'] = filename
manifest[pkg]['deps'] = {}
if pkg in lockdown:
diff --git a/bitbake/lib/bb/fetch2/osc.py b/bitbake/lib/bb/fetch2/osc.py
index 6c60456..3e56715 100644
--- a/bitbake/lib/bb/fetch2/osc.py
+++ b/bitbake/lib/bb/fetch2/osc.py
@@ -1,5 +1,6 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
"""
Bitbake "Fetch" implementation for osc (Opensuse build service client).
Based on the svn "Fetch" implementation.
diff --git a/bitbake/lib/bb/fetch2/perforce.py b/bitbake/lib/bb/fetch2/perforce.py
index 903a8e6..54d001e 100644
--- a/bitbake/lib/bb/fetch2/perforce.py
+++ b/bitbake/lib/bb/fetch2/perforce.py
@@ -1,5 +1,3 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
"""
BitBake 'Fetch' implementation for perforce
@@ -8,18 +6,7 @@ BitBake 'Fetch' implementation for perforce
# Copyright (C) 2003, 2004 Chris Larson
# Copyright (C) 2016 Kodak Alaris, Inc.
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+# SPDX-License-Identifier: GPL-2.0-only
#
# Based on functions from the base bb module, Copyright 2003 Holger Schurig
diff --git a/bitbake/lib/bb/fetch2/repo.py b/bitbake/lib/bb/fetch2/repo.py
index 8c7e818..2bdbbd4 100644
--- a/bitbake/lib/bb/fetch2/repo.py
+++ b/bitbake/lib/bb/fetch2/repo.py
@@ -1,5 +1,3 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
"""
BitBake "Fetch" repo (git) implementation
@@ -8,20 +6,10 @@ BitBake "Fetch" repo (git) implementation
# Copyright (C) 2009 Tom Rini <trini@embeddedalley.com>
#
# Based on git.py which is:
-#Copyright (C) 2005 Richard Purdie
+# Copyright (C) 2005 Richard Purdie
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import os
import bb
diff --git a/bitbake/lib/bb/fetch2/s3.py b/bitbake/lib/bb/fetch2/s3.py
index 1629288..ffca73c 100644
--- a/bitbake/lib/bb/fetch2/s3.py
+++ b/bitbake/lib/bb/fetch2/s3.py
@@ -1,5 +1,3 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
"""
BitBake 'Fetch' implementation for Amazon AWS S3.
@@ -13,18 +11,7 @@ The aws tool must be correctly installed and configured prior to use.
# Based in part on bb.fetch2.wget:
# Copyright (C) 2003, 2004 Chris Larson
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+# SPDX-License-Identifier: GPL-2.0-only
#
# Based on functions from the base bb module, Copyright 2003 Holger Schurig
diff --git a/bitbake/lib/bb/fetch2/sftp.py b/bitbake/lib/bb/fetch2/sftp.py
index 81884a6..f87f292 100644
--- a/bitbake/lib/bb/fetch2/sftp.py
+++ b/bitbake/lib/bb/fetch2/sftp.py
@@ -1,5 +1,3 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
"""
BitBake SFTP Fetch implementation
@@ -44,18 +42,7 @@ SRC_URI = "sftp://user@host.example.com/dir/path.file.txt"
# Based in part on bb.fetch2.wget:
# Copyright (C) 2003, 2004 Chris Larson
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+# SPDX-License-Identifier: GPL-2.0-only
#
# Based on functions from the base bb module, Copyright 2003 Holger Schurig
diff --git a/bitbake/lib/bb/fetch2/ssh.py b/bitbake/lib/bb/fetch2/ssh.py
index 6047ee4..f5be060 100644
--- a/bitbake/lib/bb/fetch2/ssh.py
+++ b/bitbake/lib/bb/fetch2/ssh.py
@@ -1,5 +1,3 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
'''
BitBake 'Fetch' implementations
@@ -29,18 +27,8 @@ IETF secsh internet draft:
# Copyright 2003 Holger Schurig
#
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import re, os
from bb.fetch2 import FetchMethod
diff --git a/bitbake/lib/bb/fetch2/svn.py b/bitbake/lib/bb/fetch2/svn.py
index ed70bcf..96d666b 100644
--- a/bitbake/lib/bb/fetch2/svn.py
+++ b/bitbake/lib/bb/fetch2/svn.py
@@ -1,5 +1,3 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
"""
BitBake 'Fetch' implementation for svn.
@@ -8,18 +6,7 @@ BitBake 'Fetch' implementation for svn.
# Copyright (C) 2003, 2004 Chris Larson
# Copyright (C) 2004 Marcin Juszkiewicz
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+# SPDX-License-Identifier: GPL-2.0-only
#
# Based on functions from the base bb module, Copyright 2003 Holger Schurig
@@ -63,6 +50,9 @@ class Svn(FetchMethod):
relpath = self._strip_leading_slashes(ud.path)
ud.pkgdir = os.path.join(svndir, ud.host, relpath)
ud.moddir = os.path.join(ud.pkgdir, ud.module)
+ # Protects the repository from concurrent updates, e.g. from two
+ # recipes fetching different revisions at the same time
+ ud.svnlock = os.path.join(ud.pkgdir, "svn.lock")
ud.setup_revisions(d)
@@ -101,6 +91,13 @@ class Svn(FetchMethod):
svncmd = "%s log --limit 1 %s %s://%s/%s/" % (ud.basecmd, " ".join(options), proto, svnroot, ud.module)
else:
suffix = ""
+
+ # externals may be either 'allowed' or 'nowarn', but not both. Allowed
+ # will not issue a warning, but will log to the debug buffer what has likely
+ # been downloaded by SVN.
+ if not ("externals" in ud.parm and ud.parm["externals"] == "allowed"):
+ options.append("--ignore-externals")
+
if ud.revision:
options.append("-r %s" % ud.revision)
suffix = "@%s" % (ud.revision)
@@ -123,35 +120,52 @@ class Svn(FetchMethod):
logger.debug(2, "Fetch: checking for module directory '" + ud.moddir + "'")
- if os.access(os.path.join(ud.moddir, '.svn'), os.R_OK):
- svnupdatecmd = self._buildsvncommand(ud, d, "update")
- logger.info("Update " + ud.url)
- # We need to attempt to run svn upgrade first in case its an older working format
- try:
- runfetchcmd(ud.basecmd + " upgrade", d, workdir=ud.moddir)
- except FetchError:
- pass
- logger.debug(1, "Running %s", svnupdatecmd)
- bb.fetch2.check_network_access(d, svnupdatecmd, ud.url)
- runfetchcmd(svnupdatecmd, d, workdir=ud.moddir)
- else:
- svnfetchcmd = self._buildsvncommand(ud, d, "fetch")
- logger.info("Fetch " + ud.url)
- # check out sources there
- bb.utils.mkdirhier(ud.pkgdir)
- logger.debug(1, "Running %s", svnfetchcmd)
- bb.fetch2.check_network_access(d, svnfetchcmd, ud.url)
- runfetchcmd(svnfetchcmd, d, workdir=ud.pkgdir)
-
- scmdata = ud.parm.get("scmdata", "")
- if scmdata == "keep":
- tar_flags = ""
- else:
- tar_flags = "--exclude='.svn'"
+ lf = bb.utils.lockfile(ud.svnlock)
+
+ try:
+ if os.access(os.path.join(ud.moddir, '.svn'), os.R_OK):
+ svnupdatecmd = self._buildsvncommand(ud, d, "update")
+ logger.info("Update " + ud.url)
+ # We need to attempt to run svn upgrade first in case its an older working format
+ try:
+ runfetchcmd(ud.basecmd + " upgrade", d, workdir=ud.moddir)
+ except FetchError:
+ pass
+ logger.debug(1, "Running %s", svnupdatecmd)
+ bb.fetch2.check_network_access(d, svnupdatecmd, ud.url)
+ runfetchcmd(svnupdatecmd, d, workdir=ud.moddir)
+ else:
+ svnfetchcmd = self._buildsvncommand(ud, d, "fetch")
+ logger.info("Fetch " + ud.url)
+ # check out sources there
+ bb.utils.mkdirhier(ud.pkgdir)
+ logger.debug(1, "Running %s", svnfetchcmd)
+ bb.fetch2.check_network_access(d, svnfetchcmd, ud.url)
+ runfetchcmd(svnfetchcmd, d, workdir=ud.pkgdir)
+
+ if not ("externals" in ud.parm and ud.parm["externals"] == "nowarn"):
+ # Warn the user if this had externals (won't catch them all)
+ output = runfetchcmd("svn propget svn:externals || true", d, workdir=ud.moddir)
+ if output:
+ if "--ignore-externals" in svnfetchcmd.split():
+ bb.warn("%s contains svn:externals." % ud.url)
+ bb.warn("These should be added to the recipe SRC_URI as necessary.")
+ bb.warn("svn fetch has ignored externals:\n%s" % output)
+ bb.warn("To disable this warning add ';externals=nowarn' to the url.")
+ else:
+ bb.debug(1, "svn repository has externals:\n%s" % output)
+
+ scmdata = ud.parm.get("scmdata", "")
+ if scmdata == "keep":
+ tar_flags = ""
+ else:
+ tar_flags = "--exclude='.svn'"
- # tar them up to a defined filename
- runfetchcmd("tar %s -czf %s %s" % (tar_flags, ud.localpath, ud.path_spec), d,
- cleanup=[ud.localpath], workdir=ud.pkgdir)
+ # tar them up to a defined filename
+ runfetchcmd("tar %s -czf %s %s" % (tar_flags, ud.localpath, ud.path_spec), d,
+ cleanup=[ud.localpath], workdir=ud.pkgdir)
+ finally:
+ bb.utils.unlockfile(lf)
def clean(self, ud, d):
""" Clean SVN specific files and dirs """
diff --git a/bitbake/lib/bb/fetch2/wget.py b/bitbake/lib/bb/fetch2/wget.py
index 8f505b6..725586d 100644
--- a/bitbake/lib/bb/fetch2/wget.py
+++ b/bitbake/lib/bb/fetch2/wget.py
@@ -1,5 +1,3 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
"""
BitBake 'Fetch' implementations
@@ -10,18 +8,7 @@ BitBake build tools.
# Copyright (C) 2003, 2004 Chris Larson
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+# SPDX-License-Identifier: GPL-2.0-only
#
# Based on functions from the base bb module, Copyright 2003 Holger Schurig
@@ -33,11 +20,14 @@ import logging
import errno
import bb
import bb.progress
+import socket
+import http.client
import urllib.request, urllib.parse, urllib.error
from bb.fetch2 import FetchMethod
from bb.fetch2 import FetchError
from bb.fetch2 import logger
from bb.fetch2 import runfetchcmd
+from bb.fetch2 import FetchConnectionCache
from bb.utils import export_proxies
from bs4 import BeautifulSoup
from bs4 import SoupStrainer
@@ -132,10 +122,6 @@ class Wget(FetchMethod):
return True
def checkstatus(self, fetch, ud, d, try_again=True):
- import urllib.request, urllib.error, urllib.parse, socket, http.client
- from urllib.response import addinfourl
- from bb.fetch2 import FetchConnectionCache
-
class HTTPConnectionCache(http.client.HTTPConnection):
if fetch.connection_cache:
def connect(self):
@@ -168,7 +154,7 @@ class Wget(FetchMethod):
"""
host = req.host
if not host:
- raise urlllib2.URLError('no host given')
+ raise urllib.error.URLError('no host given')
h = http_class(host, timeout=req.timeout) # will parse host:port
h.set_debuglevel(self._debuglevel)
@@ -185,7 +171,7 @@ class Wget(FetchMethod):
# request.
# Don't close connection when connection_cache is enabled,
- if fetch.connection_cache is None:
+ if fetch.connection_cache is None:
headers["Connection"] = "close"
else:
headers["Connection"] = "Keep-Alive" # Works for HTTP/1.0
@@ -252,7 +238,7 @@ class Wget(FetchMethod):
pass
closed = False
- resp = addinfourl(fp_dummy(), r.msg, req.get_full_url())
+ resp = urllib.response.addinfourl(fp_dummy(), r.msg, req.get_full_url())
resp.code = r.status
resp.msg = r.reason
@@ -271,17 +257,18 @@ class Wget(FetchMethod):
fp.read()
fp.close()
- newheaders = dict((k,v) for k,v in list(req.headers.items())
- if k.lower() not in ("content-length", "content-type"))
- return self.parent.open(urllib.request.Request(req.get_full_url(),
- headers=newheaders,
- origin_req_host=req.origin_req_host,
- unverifiable=True))
+ if req.get_method() != 'GET':
+ newheaders = dict((k, v) for k, v in list(req.headers.items())
+ if k.lower() not in ("content-length", "content-type"))
+ return self.parent.open(urllib.request.Request(req.get_full_url(),
+ headers=newheaders,
+ origin_req_host=req.origin_req_host,
+ unverifiable=True))
- """
- Some servers (e.g. GitHub archives, hosted on Amazon S3) return 403
- Forbidden when they actually mean 405 Method Not Allowed.
- """
+ raise urllib.request.HTTPError(req, code, msg, headers, None)
+
+ # Some servers (e.g. GitHub archives, hosted on Amazon S3) return 403
+ # Forbidden when they actually mean 405 Method Not Allowed.
http_error_403 = http_error_405
@@ -292,15 +279,15 @@ class Wget(FetchMethod):
"""
def redirect_request(self, req, fp, code, msg, headers, newurl):
newreq = urllib.request.HTTPRedirectHandler.redirect_request(self, req, fp, code, msg, headers, newurl)
- newreq.get_method = lambda: req.get_method()
+ newreq.get_method = req.get_method
return newreq
exported_proxies = export_proxies(d)
handlers = [FixedHTTPRedirectHandler, HTTPMethodFallback]
- if export_proxies:
+ if exported_proxies:
handlers.append(urllib.request.ProxyHandler())
handlers.append(CacheHTTPHandler())
- # XXX: Since Python 2.7.9 ssl cert validation is enabled by default
+ # Since Python 2.7.9 ssl cert validation is enabled by default
# see PEP-0476, this causes verification errors on some https servers
# so disable by default.
import ssl
@@ -319,19 +306,19 @@ class Wget(FetchMethod):
'''Adds Basic auth to http request, pass in login:password as string'''
import base64
encodeuser = base64.b64encode(login_str.encode('utf-8')).decode("utf-8")
- authheader = "Basic %s" % encodeuser
+ authheader = "Basic %s" % encodeuser
r.add_header("Authorization", authheader)
- if ud.user:
- add_basic_auth(ud.user, r)
+ if ud.user and ud.pswd:
+ add_basic_auth(ud.user + ':' + ud.pswd, r)
try:
- import netrc, urllib.parse
+ import netrc
n = netrc.netrc()
login, unused, password = n.authenticators(urllib.parse.urlparse(uri).hostname)
add_basic_auth("%s:%s" % (login, password), r)
except (TypeError, ImportError, IOError, netrc.NetrcParseError):
- pass
+ pass
with opener.open(r) as response:
pass
@@ -396,18 +383,14 @@ class Wget(FetchMethod):
(oldpn, oldpv, oldsuffix) = old
(newpn, newpv, newsuffix) = new
- """
- Check for a new suffix type that we have never heard of before
- """
- if (newsuffix):
+ # Check for a new suffix type that we have never heard of before
+ if newsuffix:
m = self.suffix_regex_comp.search(newsuffix)
if not m:
bb.warn("%s has a possible unknown suffix: %s" % (newpn, newsuffix))
return False
- """
- Not our package so ignore it
- """
+ # Not our package so ignore it
if oldpn != newpn:
return False
@@ -473,15 +456,14 @@ class Wget(FetchMethod):
return ""
- def _check_latest_version_by_dir(self, dirver, package, package_regex,
- current_version, ud, d):
+ def _check_latest_version_by_dir(self, dirver, package, package_regex, current_version, ud, d):
"""
- Scan every directory in order to get upstream version.
+ Scan every directory in order to get upstream version.
"""
version_dir = ['', '', '']
version = ['', '', '']
- dirver_regex = re.compile("(?P<pfx>\D*)(?P<ver>(\d+[\.\-_])+(\d+))")
+ dirver_regex = re.compile(r"(?P<pfx>\D*)(?P<ver>(\d+[\.\-_])+(\d+))")
s = dirver_regex.search(dirver)
if s:
version_dir[1] = s.group('ver')
@@ -541,26 +523,26 @@ class Wget(FetchMethod):
gst-fluendo-mp3
"""
# match most patterns which uses "-" as separator to version digits
- pn_prefix1 = "[a-zA-Z][a-zA-Z0-9]*([-_][a-zA-Z]\w+)*\+?[-_]"
+ pn_prefix1 = r"[a-zA-Z][a-zA-Z0-9]*([-_][a-zA-Z]\w+)*\+?[-_]"
# a loose pattern such as for unzip552.tar.gz
- pn_prefix2 = "[a-zA-Z]+"
+ pn_prefix2 = r"[a-zA-Z]+"
# a loose pattern such as for 80325-quicky-0.4.tar.gz
- pn_prefix3 = "[0-9]+[-]?[a-zA-Z]+"
+ pn_prefix3 = r"[0-9]+[-]?[a-zA-Z]+"
# Save the Package Name (pn) Regex for use later
- pn_regex = "(%s|%s|%s)" % (pn_prefix1, pn_prefix2, pn_prefix3)
+ pn_regex = r"(%s|%s|%s)" % (pn_prefix1, pn_prefix2, pn_prefix3)
# match version
- pver_regex = "(([A-Z]*\d+[a-zA-Z]*[\.\-_]*)+)"
+ pver_regex = r"(([A-Z]*\d+[a-zA-Z]*[\.\-_]*)+)"
# match arch
parch_regex = "-source|_all_"
# src.rpm extension was added only for rpm package. Can be removed if the rpm
# packaged will always be considered as having to be manually upgraded
- psuffix_regex = "(tar\.gz|tgz|tar\.bz2|zip|xz|tar\.lz|rpm|bz2|orig\.tar\.gz|tar\.xz|src\.tar\.gz|src\.tgz|svnr\d+\.tar\.bz2|stable\.tar\.gz|src\.rpm)"
+ psuffix_regex = r"(tar\.gz|tgz|tar\.bz2|zip|xz|tar\.lz|rpm|bz2|orig\.tar\.gz|tar\.xz|src\.tar\.gz|src\.tgz|svnr\d+\.tar\.bz2|stable\.tar\.gz|src\.rpm)"
# match name, version and archive type of a package
- package_regex_comp = re.compile("(?P<name>%s?\.?v?)(?P<pver>%s)(?P<arch>%s)?[\.-](?P<type>%s$)"
+ package_regex_comp = re.compile(r"(?P<name>%s?\.?v?)(?P<pver>%s)(?P<arch>%s)?[\.-](?P<type>%s$)"
% (pn_regex, pver_regex, parch_regex, psuffix_regex))
self.suffix_regex_comp = re.compile(psuffix_regex)
@@ -572,7 +554,7 @@ class Wget(FetchMethod):
version = self._parse_path(package_regex_comp, package)
if version:
package_custom_regex_comp = re.compile(
- "(?P<name>%s)(?P<pver>%s)(?P<arch>%s)?[\.-](?P<type>%s)" %
+ r"(?P<name>%s)(?P<pver>%s)(?P<arch>%s)?[\.-](?P<type>%s)" %
(re.escape(version[0]), pver_regex, parch_regex, psuffix_regex))
else:
package_custom_regex_comp = None
@@ -589,7 +571,7 @@ class Wget(FetchMethod):
current_version = ['', d.getVar('PV'), '']
"""possible to have no version in pkg name, such as spectrum-fw"""
- if not re.search("\d+", package):
+ if not re.search(r"\d+", package):
current_version[1] = re.sub('_', '.', current_version[1])
current_version[1] = re.sub('-', '.', current_version[1])
return (current_version[1], '')
@@ -607,13 +589,13 @@ class Wget(FetchMethod):
# search for version matches on folders inside the path, like:
# "5.7" in http://download.gnome.org/sources/${PN}/5.7/${PN}-${PV}.tar.gz
- dirver_regex = re.compile("(?P<dirver>[^/]*(\d+\.)*\d+([-_]r\d+)*)/")
+ dirver_regex = re.compile(r"(?P<dirver>[^/]*(\d+\.)*\d+([-_]r\d+)*)/")
m = dirver_regex.search(path)
if m:
pn = d.getVar('PN')
dirver = m.group('dirver')
- dirver_pn_regex = re.compile("%s\d?" % (re.escape(pn)))
+ dirver_pn_regex = re.compile(r"%s\d?" % (re.escape(pn)))
if not dirver_pn_regex.search(dirver):
return (self._check_latest_version_by_dir(dirver,
package, package_regex, current_version, ud, d), '')
diff --git a/bitbake/lib/bb/main.py b/bitbake/lib/bb/main.py
index 732a315..af2880f 100755
--- a/bitbake/lib/bb/main.py
+++ b/bitbake/lib/bb/main.py
@@ -1,6 +1,3 @@
-#!/usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# Copyright (C) 2003, 2004 Chris Larson
# Copyright (C) 2003, 2004 Phil Blundell
@@ -9,18 +6,8 @@
# Copyright (C) 2005 ROAD GmbH
# Copyright (C) 2006 Richard Purdie
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import os
import sys
@@ -267,6 +254,11 @@ class BitBakeConfigParameters(cookerdata.ConfigParameters):
help="Do not run any setscene tasks. sstate will be ignored and "
"everything needed, built.")
+ parser.add_option("", "--skip-setscene", action="store_true",
+ dest="skipsetscene", default=False,
+ help="Skip setscene tasks if they would be executed. Tasks previously "
+ "restored from sstate will be kept, unlike --no-setscene")
+
parser.add_option("", "--setscene-only", action="store_true",
dest="setsceneonly", default=False,
help="Only run setscene tasks, don't run any real tasks.")
@@ -448,7 +440,7 @@ def setup_bitbake(configParams, configuration, extrafeatures=None):
else:
logger.info("Reconnecting to bitbake server...")
if not os.path.exists(sockname):
- print("Previous bitbake instance shutting down?, waiting to retry...")
+ logger.info("Previous bitbake instance shutting down?, waiting to retry...")
i = 0
lock = None
# Wait for 5s or until we can get the lock
@@ -460,12 +452,7 @@ def setup_bitbake(configParams, configuration, extrafeatures=None):
bb.utils.unlockfile(lock)
raise bb.server.process.ProcessTimeout("Bitbake still shutting down as socket exists but no lock?")
if not configParams.server_only:
- try:
- server_connection = bb.server.process.connectProcessServer(sockname, featureset)
- except EOFError:
- # The server may have been shutting down but not closed the socket yet. If that happened,
- # ignore it.
- pass
+ server_connection = bb.server.process.connectProcessServer(sockname, featureset)
if server_connection or configParams.server_only:
break
@@ -475,12 +462,14 @@ def setup_bitbake(configParams, configuration, extrafeatures=None):
if not retries:
raise
retries -= 1
- if isinstance(e, (bb.server.process.ProcessTimeout, BrokenPipeError)):
- logger.info("Retrying server connection...")
+ tryno = 8 - retries
+ if isinstance(e, (bb.server.process.ProcessTimeout, BrokenPipeError, EOFError)):
+ logger.info("Retrying server connection (#%d)..." % tryno)
else:
- logger.info("Retrying server connection... (%s)" % traceback.format_exc())
+ logger.info("Retrying server connection (#%d)... (%s)" % (tryno, traceback.format_exc()))
if not retries:
- bb.fatal("Unable to connect to bitbake server, or start one")
+ bb.fatal("Unable to connect to bitbake server, or start one (server startup failures would be in bitbake-cookerdaemon.log).")
+ bb.event.print_ui_queue()
if retries < 5:
time.sleep(5)
@@ -502,7 +491,7 @@ def setup_bitbake(configParams, configuration, extrafeatures=None):
def lockBitbake():
topdir = bb.cookerdata.findTopdir()
if not topdir:
- bb.error("Unable to find conf/bblayers.conf or conf/bitbake.conf. BBAPTH is unset and/or not in a build directory?")
+ bb.error("Unable to find conf/bblayers.conf or conf/bitbake.conf. BBPATH is unset and/or not in a build directory?")
raise BBMainFatal
lockfile = topdir + "/bitbake.lock"
return topdir, bb.utils.lockfile(lockfile, False, False)
diff --git a/bitbake/lib/bb/methodpool.py b/bitbake/lib/bb/methodpool.py
index 49aed33..51783ac 100644
--- a/bitbake/lib/bb/methodpool.py
+++ b/bitbake/lib/bb/methodpool.py
@@ -1,21 +1,8 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
-#
#
# Copyright (C) 2006 Holger Hans Peter Freyther
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from bb.utils import better_compile, better_exec
diff --git a/bitbake/lib/bb/monitordisk.py b/bitbake/lib/bb/monitordisk.py
index 833cd3d..1a25b00 100644
--- a/bitbake/lib/bb/monitordisk.py
+++ b/bitbake/lib/bb/monitordisk.py
@@ -1,21 +1,8 @@
-#!/usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# Copyright (C) 2012 Robert Yang
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import os, logging, re, sys
import bb
@@ -28,16 +15,16 @@ def convertGMK(unit):
""" Convert the space unit G, M, K, the unit is case-insensitive """
- unitG = re.match('([1-9][0-9]*)[gG]\s?$', unit)
+ unitG = re.match(r'([1-9][0-9]*)[gG]\s?$', unit)
if unitG:
return int(unitG.group(1)) * (1024 ** 3)
- unitM = re.match('([1-9][0-9]*)[mM]\s?$', unit)
+ unitM = re.match(r'([1-9][0-9]*)[mM]\s?$', unit)
if unitM:
return int(unitM.group(1)) * (1024 ** 2)
- unitK = re.match('([1-9][0-9]*)[kK]\s?$', unit)
+ unitK = re.match(r'([1-9][0-9]*)[kK]\s?$', unit)
if unitK:
return int(unitK.group(1)) * 1024
- unitN = re.match('([1-9][0-9]*)\s?$', unit)
+ unitN = re.match(r'([1-9][0-9]*)\s?$', unit)
if unitN:
return int(unitN.group(1))
else:
@@ -83,7 +70,7 @@ def getDiskData(BBDirs, configuration):
for pathSpaceInode in BBDirs.split():
# The input format is: "dir,space,inode", dir is a must, space
# and inode are optional
- pathSpaceInodeRe = re.match('([^,]*),([^,]*),([^,]*),?(.*)', pathSpaceInode)
+ pathSpaceInodeRe = re.match(r'([^,]*),([^,]*),([^,]*),?(.*)', pathSpaceInode)
if not pathSpaceInodeRe:
printErr("Invalid value in BB_DISKMON_DIRS: %s" % pathSpaceInode)
return None
@@ -147,7 +134,7 @@ def getInterval(configuration):
else:
# The disk space or inode interval is optional, but it should
# have a correct value once it is specified
- intervalRe = re.match('([^,]*),?\s*(.*)', interval)
+ intervalRe = re.match(r'([^,]*),?\s*(.*)', interval)
if intervalRe:
intervalSpace = intervalRe.group(1)
if intervalSpace:
diff --git a/bitbake/lib/bb/msg.py b/bitbake/lib/bb/msg.py
index 96f077e..6216eb3 100644
--- a/bitbake/lib/bb/msg.py
+++ b/bitbake/lib/bb/msg.py
@@ -1,5 +1,3 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
"""
BitBake 'msg' implementation
@@ -9,18 +7,8 @@ Message handling infrastructure for bitbake
# Copyright (C) 2006 Richard Purdie
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import sys
import copy
diff --git a/bitbake/lib/bb/namedtuple_with_abc.py b/bitbake/lib/bb/namedtuple_with_abc.py
index 32f2fc6..646aed6 100644
--- a/bitbake/lib/bb/namedtuple_with_abc.py
+++ b/bitbake/lib/bb/namedtuple_with_abc.py
@@ -1,6 +1,8 @@
# http://code.activestate.com/recipes/577629-namedtupleabc-abstract-base-class-mix-in-for-named/
-#!/usr/bin/env python
# Copyright (c) 2011 Jan Kaliszewski (zuo). Available under the MIT License.
+#
+# SPDX-License-Identifier: MIT
+#
"""
namedtuple_with_abc.py:
diff --git a/bitbake/lib/bb/parse/__init__.py b/bitbake/lib/bb/parse/__init__.py
index 5397d57..76e180b 100644
--- a/bitbake/lib/bb/parse/__init__.py
+++ b/bitbake/lib/bb/parse/__init__.py
@@ -9,20 +9,10 @@ File parsers for the BitBake build tools.
# Copyright (C) 2003, 2004 Chris Larson
# Copyright (C) 2003, 2004 Phil Blundell
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+# SPDX-License-Identifier: GPL-2.0-only
#
# Based on functions from the base bb module, Copyright 2003 Holger Schurig
+#
handlers = []
diff --git a/bitbake/lib/bb/parse/ast.py b/bitbake/lib/bb/parse/ast.py
index 6d7c80b..f0911e6 100644
--- a/bitbake/lib/bb/parse/ast.py
+++ b/bitbake/lib/bb/parse/ast.py
@@ -1,5 +1,3 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
"""
AbstractSyntaxTree classes for the Bitbake language
"""
@@ -8,19 +6,8 @@
# Copyright (C) 2003, 2004 Phil Blundell
# Copyright (C) 2009 Holger Hans Peter Freyther
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
import re
import string
diff --git a/bitbake/lib/bb/parse/parse_py/BBHandler.py b/bitbake/lib/bb/parse/parse_py/BBHandler.py
index 01fc47e..6f7cf82 100644
--- a/bitbake/lib/bb/parse/parse_py/BBHandler.py
+++ b/bitbake/lib/bb/parse/parse_py/BBHandler.py
@@ -1,6 +1,3 @@
-#!/usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
"""
class for handling .bb files
@@ -12,19 +9,8 @@
# Copyright (C) 2003, 2004 Chris Larson
# Copyright (C) 2003, 2004 Phil Blundell
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
import re, bb, os
import logging
@@ -38,14 +24,15 @@ from .ConfHandler import include, init
# For compatibility
bb.deprecate_import(__name__, "bb.parse", ["vars_from_file"])
-__func_start_regexp__ = re.compile( r"(((?P<py>python)|(?P<fr>fakeroot))\s*)*(?P<func>[\w\.\-\+\{\}\$]+)?\s*\(\s*\)\s*{$" )
-__inherit_regexp__ = re.compile( r"inherit\s+(.+)" )
-__export_func_regexp__ = re.compile( r"EXPORT_FUNCTIONS\s+(.+)" )
-__addtask_regexp__ = re.compile("addtask\s+(?P<func>\w+)\s*((before\s*(?P<before>((.*(?=after))|(.*))))|(after\s*(?P<after>((.*(?=before))|(.*)))))*")
-__deltask_regexp__ = re.compile("deltask\s+(?P<func>\w+)")
-__addhandler_regexp__ = re.compile( r"addhandler\s+(.+)" )
-__def_regexp__ = re.compile( r"def\s+(\w+).*:" )
-__python_func_regexp__ = re.compile( r"(\s+.*)|(^$)|(^#)" )
+__func_start_regexp__ = re.compile(r"(((?P<py>python)|(?P<fr>fakeroot))\s*)*(?P<func>[\w\.\-\+\{\}\$]+)?\s*\(\s*\)\s*{$" )
+__inherit_regexp__ = re.compile(r"inherit\s+(.+)" )
+__export_func_regexp__ = re.compile(r"EXPORT_FUNCTIONS\s+(.+)" )
+__addtask_regexp__ = re.compile(r"addtask\s+(?P<func>\w+)\s*((before\s*(?P<before>((.*(?=after))|(.*))))|(after\s*(?P<after>((.*(?=before))|(.*)))))*")
+__deltask_regexp__ = re.compile(r"deltask\s+(?P<func>\w+)(?P<ignores>.*)")
+__addhandler_regexp__ = re.compile(r"addhandler\s+(.+)" )
+__def_regexp__ = re.compile(r"def\s+(\w+).*:" )
+__python_func_regexp__ = re.compile(r"(\s+.*)|(^$)|(^#)" )
+__python_tab_regexp__ = re.compile(r" *\t")
__infunc__ = []
__inpython__ = False
@@ -160,6 +147,16 @@ def handle(fn, d, include):
def feeder(lineno, s, fn, root, statements, eof=False):
global __func_start_regexp__, __inherit_regexp__, __export_func_regexp__, __addtask_regexp__, __addhandler_regexp__, __def_regexp__, __python_func_regexp__, __inpython__, __infunc__, __body__, bb, __residue__, __classname__
+
+ # Check tabs in python functions:
+ # - def py_funcname(): covered by __inpython__
+ # - python(): covered by '__anonymous' == __infunc__[0]
+ # - python funcname(): covered by __infunc__[3]
+ if __inpython__ or (__infunc__ and ('__anonymous' == __infunc__[0] or __infunc__[3])):
+ tab = __python_tab_regexp__.match(s)
+ if tab:
+ bb.warn('python should use 4 spaces indentation, but found tabs in %s, line %s' % (root, lineno))
+
if __infunc__:
if s == '}':
__body__.append('')
@@ -225,11 +222,27 @@ def feeder(lineno, s, fn, root, statements, eof=False):
m = __addtask_regexp__.match(s)
if m:
+ if len(m.group().split()) == 2:
+ # Check and warn for "addtask task1 task2"
+ m2 = re.match(r"addtask\s+(?P<func>\w+)(?P<ignores>.*)", s)
+ if m2 and m2.group('ignores'):
+ logger.warning('addtask ignored: "%s"' % m2.group('ignores'))
+
+ # Check and warn for "addtask task1 before task2 before task3", the
+ # similar to "after"
+ taskexpression = s.split()
+ for word in ('before', 'after'):
+ if taskexpression.count(word) > 1:
+ logger.warning("addtask contained multiple '%s' keywords, only one is supported" % word)
+
ast.handleAddTask(statements, fn, lineno, m)
return
m = __deltask_regexp__.match(s)
if m:
+ # Check and warn "for deltask task1 task2"
+ if m.group('ignores'):
+ logger.warning('deltask ignored: "%s"' % m.group('ignores'))
ast.handleDelTask(statements, fn, lineno, m)
return
diff --git a/bitbake/lib/bb/parse/parse_py/ConfHandler.py b/bitbake/lib/bb/parse/parse_py/ConfHandler.py
index 9d3ebe1..2e84b91 100644
--- a/bitbake/lib/bb/parse/parse_py/ConfHandler.py
+++ b/bitbake/lib/bb/parse/parse_py/ConfHandler.py
@@ -1,6 +1,3 @@
-#!/usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
"""
class for handling configuration data files
@@ -11,18 +8,8 @@
# Copyright (C) 2003, 2004 Chris Larson
# Copyright (C) 2003, 2004 Phil Blundell
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import errno
import re
@@ -147,7 +134,7 @@ def handle(fn, data, include):
continue
s = s.rstrip()
while s[-1] == '\\':
- s2 = f.readline().strip()
+ s2 = f.readline().rstrip()
lineno = lineno + 1
if (not s2 or s2 and s2[0] != "#") and s[0] == "#" :
bb.fatal("There is a confusing multiline, partially commented expression on line %s of file %s (%s).\nPlease clarify whether this is all a comment or should be parsed." % (lineno, fn, s))
diff --git a/bitbake/lib/bb/parse/parse_py/__init__.py b/bitbake/lib/bb/parse/parse_py/__init__.py
index 3e658d0..f508afa 100644
--- a/bitbake/lib/bb/parse/parse_py/__init__.py
+++ b/bitbake/lib/bb/parse/parse_py/__init__.py
@@ -1,6 +1,3 @@
-#!/usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
"""
BitBake Parsers
@@ -11,20 +8,10 @@ File parsers for the BitBake build tools.
# Copyright (C) 2003, 2004 Chris Larson
# Copyright (C) 2003, 2004 Phil Blundell
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+# SPDX-License-Identifier: GPL-2.0-only
#
# Based on functions from the base bb module, Copyright 2003 Holger Schurig
+#
from __future__ import absolute_import
from . import ConfHandler
diff --git a/bitbake/lib/bb/persist_data.py b/bitbake/lib/bb/persist_data.py
index bef7018..de8f87a 100644
--- a/bitbake/lib/bb/persist_data.py
+++ b/bitbake/lib/bb/persist_data.py
@@ -8,18 +8,8 @@ currently, providing a key/value store accessed by 'domain'.
# Copyright (C) 2007 Richard Purdie
# Copyright (C) 2010 Chris Larson <chris_larson@mentor.com>
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import collections
import logging
@@ -29,6 +19,7 @@ import warnings
from bb.compat import total_ordering
from collections import Mapping
import sqlite3
+import contextlib
sqlversion = sqlite3.sqlite_version_info
if sqlversion[0] < 3 or (sqlversion[0] == 3 and sqlversion[1] < 3):
@@ -36,84 +27,181 @@ if sqlversion[0] < 3 or (sqlversion[0] == 3 and sqlversion[1] < 3):
logger = logging.getLogger("BitBake.PersistData")
-if hasattr(sqlite3, 'enable_shared_cache'):
- try:
- sqlite3.enable_shared_cache(True)
- except sqlite3.OperationalError:
- pass
-
@total_ordering
class SQLTable(collections.MutableMapping):
+ class _Decorators(object):
+ @staticmethod
+ def retry(*, reconnect=True):
+ """
+ Decorator that restarts a function if a database locked sqlite
+ exception occurs. If reconnect is True, the database connection
+ will be closed and reopened each time a failure occurs
+ """
+ def retry_wrapper(f):
+ def wrap_func(self, *args, **kwargs):
+ # Reconnect if necessary
+ if self.connection is None and reconnect:
+ self.reconnect()
+
+ count = 0
+ while True:
+ try:
+ return f(self, *args, **kwargs)
+ except sqlite3.OperationalError as exc:
+ if count < 500 and ('is locked' in str(exc) or 'locking protocol' in str(exc)):
+ count = count + 1
+ if reconnect:
+ self.reconnect()
+ continue
+ raise
+ return wrap_func
+ return retry_wrapper
+
+ @staticmethod
+ def transaction(f):
+ """
+ Decorator that starts a database transaction and creates a database
+ cursor for performing queries. If no exception is thrown, the
+ database results are commited. If an exception occurs, the database
+ is rolled back. In all cases, the cursor is closed after the
+ function ends.
+
+ Note that the cursor is passed as an extra argument to the function
+ after `self` and before any of the normal arguments
+ """
+ def wrap_func(self, *args, **kwargs):
+ # Context manager will COMMIT the database on success,
+ # or ROLLBACK on an exception
+ with self.connection:
+ # Automatically close the cursor when done
+ with contextlib.closing(self.connection.cursor()) as cursor:
+ return f(self, cursor, *args, **kwargs)
+ return wrap_func
+
"""Object representing a table/domain in the database"""
def __init__(self, cachefile, table):
self.cachefile = cachefile
self.table = table
- self.cursor = connect(self.cachefile)
-
- self._execute("CREATE TABLE IF NOT EXISTS %s(key TEXT, value TEXT);"
- % table)
-
- def _execute(self, *query):
- """Execute a query, waiting to acquire a lock if necessary"""
- count = 0
- while True:
- try:
- return self.cursor.execute(*query)
- except sqlite3.OperationalError as exc:
- if 'database is locked' in str(exc) and count < 500:
- count = count + 1
+
+ self.connection = None
+ self._execute_single("CREATE TABLE IF NOT EXISTS %s(key TEXT PRIMARY KEY NOT NULL, value TEXT);" % table)
+
+ @_Decorators.retry(reconnect=False)
+ @_Decorators.transaction
+ def _setup_database(self, cursor):
+ cursor.execute("pragma synchronous = off;")
+ # Enable WAL and keep the autocheckpoint length small (the default is
+ # usually 1000). Persistent caches are usually read-mostly, so keeping
+ # this short will keep readers running quickly
+ cursor.execute("pragma journal_mode = WAL;")
+ cursor.execute("pragma wal_autocheckpoint = 100;")
+
+ def reconnect(self):
+ if self.connection is not None:
+ self.connection.close()
+ self.connection = sqlite3.connect(self.cachefile, timeout=5)
+ self.connection.text_factory = str
+ self._setup_database()
+
+ @_Decorators.retry()
+ @_Decorators.transaction
+ def _execute_single(self, cursor, *query):
+ """
+ Executes a single query and discards the results. This correctly closes
+ the database cursor when finished
+ """
+ cursor.execute(*query)
+
+ @_Decorators.retry()
+ def _row_iter(self, f, *query):
+ """
+ Helper function that returns a row iterator. Each time __next__ is
+ called on the iterator, the provided function is evaluated to determine
+ the return value
+ """
+ class CursorIter(object):
+ def __init__(self, cursor):
+ self.cursor = cursor
+
+ def __iter__(self):
+ return self
+
+ def __next__(self):
+ row = self.cursor.fetchone()
+ if row is None:
self.cursor.close()
- self.cursor = connect(self.cachefile)
- continue
- raise
+ raise StopIteration
+ return f(row)
+
+ def __enter__(self):
+ return self
+
+ def __exit__(self, typ, value, traceback):
+ self.cursor.close()
+ return False
+
+ cursor = self.connection.cursor()
+ try:
+ cursor.execute(*query)
+ return CursorIter(cursor)
+ except:
+ cursor.close()
def __enter__(self):
- self.cursor.__enter__()
+ self.connection.__enter__()
return self
def __exit__(self, *excinfo):
- self.cursor.__exit__(*excinfo)
-
- def __getitem__(self, key):
- data = self._execute("SELECT * from %s where key=?;" %
- self.table, [key])
- for row in data:
+ self.connection.__exit__(*excinfo)
+
+ @_Decorators.retry()
+ @_Decorators.transaction
+ def __getitem__(self, cursor, key):
+ cursor.execute("SELECT * from %s where key=?;" % self.table, [key])
+ row = cursor.fetchone()
+ if row is not None:
return row[1]
raise KeyError(key)
- def __delitem__(self, key):
+ @_Decorators.retry()
+ @_Decorators.transaction
+ def __delitem__(self, cursor, key):
if key not in self:
raise KeyError(key)
- self._execute("DELETE from %s where key=?;" % self.table, [key])
+ cursor.execute("DELETE from %s where key=?;" % self.table, [key])
- def __setitem__(self, key, value):
+ @_Decorators.retry()
+ @_Decorators.transaction
+ def __setitem__(self, cursor, key, value):
if not isinstance(key, str):
raise TypeError('Only string keys are supported')
elif not isinstance(value, str):
raise TypeError('Only string values are supported')
- data = self._execute("SELECT * from %s where key=?;" %
- self.table, [key])
- exists = len(list(data))
- if exists:
- self._execute("UPDATE %s SET value=? WHERE key=?;" % self.table,
- [value, key])
+ cursor.execute("SELECT * from %s where key=?;" % self.table, [key])
+ row = cursor.fetchone()
+ if row is not None:
+ cursor.execute("UPDATE %s SET value=? WHERE key=?;" % self.table, [value, key])
else:
- self._execute("INSERT into %s(key, value) values (?, ?);" %
- self.table, [key, value])
-
- def __contains__(self, key):
- return key in set(self)
-
- def __len__(self):
- data = self._execute("SELECT COUNT(key) FROM %s;" % self.table)
- for row in data:
+ cursor.execute("INSERT into %s(key, value) values (?, ?);" % self.table, [key, value])
+
+ @_Decorators.retry()
+ @_Decorators.transaction
+ def __contains__(self, cursor, key):
+ cursor.execute('SELECT * from %s where key=?;' % self.table, [key])
+ return cursor.fetchone() is not None
+
+ @_Decorators.retry()
+ @_Decorators.transaction
+ def __len__(self, cursor):
+ cursor.execute("SELECT COUNT(key) FROM %s;" % self.table)
+ row = cursor.fetchone()
+ if row is not None:
return row[0]
def __iter__(self):
- data = self._execute("SELECT key FROM %s;" % self.table)
- return (row[0] for row in data)
+ return self._row_iter(lambda row: row[0], "SELECT key from %s;" % self.table)
def __lt__(self, other):
if not isinstance(other, Mapping):
@@ -122,25 +210,27 @@ class SQLTable(collections.MutableMapping):
return len(self) < len(other)
def get_by_pattern(self, pattern):
- data = self._execute("SELECT * FROM %s WHERE key LIKE ?;" %
- self.table, [pattern])
- return [row[1] for row in data]
+ return self._row_iter(lambda row: row[1], "SELECT * FROM %s WHERE key LIKE ?;" %
+ self.table, [pattern])
def values(self):
return list(self.itervalues())
def itervalues(self):
- data = self._execute("SELECT value FROM %s;" % self.table)
- return (row[0] for row in data)
+ return self._row_iter(lambda row: row[0], "SELECT value FROM %s;" %
+ self.table)
def items(self):
return list(self.iteritems())
def iteritems(self):
- return self._execute("SELECT * FROM %s;" % self.table)
+ return self._row_iter(lambda row: (row[0], row[1]), "SELECT * FROM %s;" %
+ self.table)
- def clear(self):
- self._execute("DELETE FROM %s;" % self.table)
+ @_Decorators.retry()
+ @_Decorators.transaction
+ def clear(self, cursor):
+ cursor.execute("DELETE FROM %s;" % self.table)
def has_key(self, key):
return key in self
@@ -194,12 +284,6 @@ class PersistData(object):
"""
del self.data[domain][key]
-def connect(database):
- connection = sqlite3.connect(database, timeout=5, isolation_level=None)
- connection.execute("pragma synchronous = off;")
- connection.text_factory = str
- return connection
-
def persist(domain, d):
"""Convenience factory for SQLTable objects based upon metadata"""
import bb.utils
diff --git a/bitbake/lib/bb/process.py b/bitbake/lib/bb/process.py
index e69697c..2dc472a 100644
--- a/bitbake/lib/bb/process.py
+++ b/bitbake/lib/bb/process.py
@@ -1,3 +1,7 @@
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
import logging
import signal
import subprocess
diff --git a/bitbake/lib/bb/progress.py b/bitbake/lib/bb/progress.py
index f54d1c7..4022caa 100644
--- a/bitbake/lib/bb/progress.py
+++ b/bitbake/lib/bb/progress.py
@@ -4,18 +4,8 @@ BitBake progress handling code
# Copyright (C) 2016 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import sys
import re
@@ -23,6 +13,7 @@ import time
import inspect
import bb.event
import bb.build
+from bb.build import StdoutNoopContextManager
class ProgressHandler(object):
"""
@@ -37,7 +28,14 @@ class ProgressHandler(object):
if outfile:
self._outfile = outfile
else:
- self._outfile = sys.stdout
+ self._outfile = StdoutNoopContextManager()
+
+ def __enter__(self):
+ self._outfile.__enter__()
+ return self
+
+ def __exit__(self, *excinfo):
+ self._outfile.__exit__(*excinfo)
def _fire_progress(self, taskprogress, rate=None):
"""Internal function to fire the progress event"""
@@ -157,6 +155,12 @@ class MultiStageProgressReporter(object):
self._stage_total = None
self._callers = []
+ def __enter__(self):
+ return self
+
+ def __exit__(self, *excinfo):
+ pass
+
def _fire_progress(self, taskprogress):
bb.event.fire(bb.build.TaskProgress(taskprogress), self._data)
diff --git a/bitbake/lib/bb/providers.py b/bitbake/lib/bb/providers.py
index c2aa98c..f80963c 100644
--- a/bitbake/lib/bb/providers.py
+++ b/bitbake/lib/bb/providers.py
@@ -1,5 +1,3 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# Copyright (C) 2003, 2004 Chris Larson
# Copyright (C) 2003, 2004 Phil Blundell
@@ -8,18 +6,8 @@
# Copyright (C) 2005 ROAD GmbH
# Copyright (C) 2006 Richard Purdie
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import re
import logging
@@ -129,7 +117,7 @@ def findPreferredProvider(pn, cfgData, dataCache, pkg_pn = None, item = None):
preferred_v = cfgData.getVar("PREFERRED_VERSION")
if preferred_v:
- m = re.match('(\d+:)*(.*)(_.*)*', preferred_v)
+ m = re.match(r'(\d+:)*(.*)(_.*)*', preferred_v)
if m:
if m.group(1):
preferred_e = m.group(1)[:-1]
@@ -384,7 +372,7 @@ def getRuntimeProviders(dataCache, rdepend):
# Only search dynamic packages if we can't find anything in other variables
for pattern in dataCache.packages_dynamic:
- pattern = pattern.replace('+', "\+")
+ pattern = pattern.replace(r'+', r"\+")
if pattern in regexp_cache:
regexp = regexp_cache[pattern]
else:
diff --git a/bitbake/lib/bb/pysh/builtin.py b/bitbake/lib/bb/pysh/builtin.py
deleted file mode 100644
index a8814dc..0000000
--- a/bitbake/lib/bb/pysh/builtin.py
+++ /dev/null
@@ -1,710 +0,0 @@
-# builtin.py - builtins and utilities definitions for pysh.
-#
-# Copyright 2007 Patrick Mezard
-#
-# This software may be used and distributed according to the terms
-# of the GNU General Public License, incorporated herein by reference.
-
-"""Builtin and internal utilities implementations.
-
-- Beware not to use python interpreter environment as if it were the shell
-environment. For instance, commands working directory must be explicitely handled
-through env['PWD'] instead of relying on python working directory.
-"""
-import errno
-import optparse
-import os
-import re
-import subprocess
-import sys
-import time
-
-def has_subprocess_bug():
- return getattr(subprocess, 'list2cmdline') and \
- ( subprocess.list2cmdline(['']) == '' or \
- subprocess.list2cmdline(['foo|bar']) == 'foo|bar')
-
-# Detect python bug 1634343: "subprocess swallows empty arguments under win32"
-# <http://sourceforge.net/tracker/index.php?func=detail&aid=1634343&group_id=5470&atid=105470>
-# Also detect: "[ 1710802 ] subprocess must escape redirection characters under win32"
-# <http://sourceforge.net/tracker/index.php?func=detail&aid=1710802&group_id=5470&atid=105470>
-if has_subprocess_bug():
- import subprocess_fix
- subprocess.list2cmdline = subprocess_fix.list2cmdline
-
-from sherrors import *
-
-class NonExitingParser(optparse.OptionParser):
- """OptionParser default behaviour upon error is to print the error message and
- exit. Raise a utility error instead.
- """
- def error(self, msg):
- raise UtilityError(msg)
-
-#-------------------------------------------------------------------------------
-# set special builtin
-#-------------------------------------------------------------------------------
-OPT_SET = NonExitingParser(usage="set - set or unset options and positional parameters")
-OPT_SET.add_option( '-f', action='store_true', dest='has_f', default=False,
- help='The shell shall disable pathname expansion.')
-OPT_SET.add_option('-e', action='store_true', dest='has_e', default=False,
- help="""When this option is on, if a simple command fails for any of the \
- reasons listed in Consequences of Shell Errors or returns an exit status \
- value >0, and is not part of the compound list following a while, until, \
- or if keyword, and is not a part of an AND or OR list, and is not a \
- pipeline preceded by the ! reserved word, then the shell shall immediately \
- exit.""")
-OPT_SET.add_option('-x', action='store_true', dest='has_x', default=False,
- help="""The shell shall write to standard error a trace for each command \
- after it expands the command and before it executes it. It is unspecified \
- whether the command that turns tracing off is traced.""")
-
-def builtin_set(name, args, interp, env, stdin, stdout, stderr, debugflags):
- if 'debug-utility' in debugflags:
- print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n')
-
- option, args = OPT_SET.parse_args(args)
- env = interp.get_env()
-
- if option.has_f:
- env.set_opt('-f')
- if option.has_e:
- env.set_opt('-e')
- if option.has_x:
- env.set_opt('-x')
- return 0
-
-#-------------------------------------------------------------------------------
-# shift special builtin
-#-------------------------------------------------------------------------------
-def builtin_shift(name, args, interp, env, stdin, stdout, stderr, debugflags):
- if 'debug-utility' in debugflags:
- print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n')
-
- params = interp.get_env().get_positional_args()
- if args:
- try:
- n = int(args[0])
- if n > len(params):
- raise ValueError()
- except ValueError:
- return 1
- else:
- n = 1
-
- params[:n] = []
- interp.get_env().set_positional_args(params)
- return 0
-
-#-------------------------------------------------------------------------------
-# export special builtin
-#-------------------------------------------------------------------------------
-OPT_EXPORT = NonExitingParser(usage="set - set or unset options and positional parameters")
-OPT_EXPORT.add_option('-p', action='store_true', dest='has_p', default=False)
-
-def builtin_export(name, args, interp, env, stdin, stdout, stderr, debugflags):
- if 'debug-utility' in debugflags:
- print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n')
-
- option, args = OPT_EXPORT.parse_args(args)
- if option.has_p:
- raise NotImplementedError()
-
- for arg in args:
- try:
- name, value = arg.split('=', 1)
- except ValueError:
- name, value = arg, None
- env = interp.get_env().export(name, value)
-
- return 0
-
-#-------------------------------------------------------------------------------
-# return special builtin
-#-------------------------------------------------------------------------------
-def builtin_return(name, args, interp, env, stdin, stdout, stderr, debugflags):
- if 'debug-utility' in debugflags:
- print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n')
- res = 0
- if args:
- try:
- res = int(args[0])
- except ValueError:
- res = 0
- if not 0<=res<=255:
- res = 0
-
- # BUG: should be last executed command exit code
- raise ReturnSignal(res)
-
-#-------------------------------------------------------------------------------
-# trap special builtin
-#-------------------------------------------------------------------------------
-def builtin_trap(name, args, interp, env, stdin, stdout, stderr, debugflags):
- if 'debug-utility' in debugflags:
- print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n')
- if len(args) < 2:
- stderr.write('trap: usage: trap [[arg] signal_spec ...]\n')
- return 2
-
- action = args[0]
- for sig in args[1:]:
- try:
- env.traps[sig] = action
- except Exception as e:
- stderr.write('trap: %s\n' % str(e))
- return 0
-
-#-------------------------------------------------------------------------------
-# unset special builtin
-#-------------------------------------------------------------------------------
-OPT_UNSET = NonExitingParser("unset - unset values and attributes of variables and functions")
-OPT_UNSET.add_option( '-f', action='store_true', dest='has_f', default=False)
-OPT_UNSET.add_option( '-v', action='store_true', dest='has_v', default=False)
-
-def builtin_unset(name, args, interp, env, stdin, stdout, stderr, debugflags):
- if 'debug-utility' in debugflags:
- print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n')
-
- option, args = OPT_UNSET.parse_args(args)
-
- status = 0
- env = interp.get_env()
- for arg in args:
- try:
- if option.has_f:
- env.remove_function(arg)
- else:
- del env[arg]
- except KeyError:
- pass
- except VarAssignmentError:
- status = 1
-
- return status
-
-#-------------------------------------------------------------------------------
-# wait special builtin
-#-------------------------------------------------------------------------------
-def builtin_wait(name, args, interp, env, stdin, stdout, stderr, debugflags):
- if 'debug-utility' in debugflags:
- print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n')
-
- return interp.wait([int(arg) for arg in args])
-
-#-------------------------------------------------------------------------------
-# cat utility
-#-------------------------------------------------------------------------------
-def utility_cat(name, args, interp, env, stdin, stdout, stderr, debugflags):
- if 'debug-utility' in debugflags:
- print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n')
-
- if not args:
- args = ['-']
-
- status = 0
- for arg in args:
- if arg == '-':
- data = stdin.read()
- else:
- path = os.path.join(env['PWD'], arg)
- try:
- f = file(path, 'rb')
- try:
- data = f.read()
- finally:
- f.close()
- except IOError as e:
- if e.errno != errno.ENOENT:
- raise
- status = 1
- continue
- stdout.write(data)
- stdout.flush()
- return status
-
-#-------------------------------------------------------------------------------
-# cd utility
-#-------------------------------------------------------------------------------
-OPT_CD = NonExitingParser("cd - change the working directory")
-
-def utility_cd(name, args, interp, env, stdin, stdout, stderr, debugflags):
- if 'debug-utility' in debugflags:
- print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n')
-
- option, args = OPT_CD.parse_args(args)
- env = interp.get_env()
-
- directory = None
- printdir = False
- if not args:
- home = env.get('HOME')
- if home:
- # Unspecified, do nothing
- return 0
- else:
- directory = home
- elif len(args)==1:
- directory = args[0]
- if directory=='-':
- if 'OLDPWD' not in env:
- raise UtilityError("OLDPWD not set")
- printdir = True
- directory = env['OLDPWD']
- else:
- raise UtilityError("too many arguments")
-
- curpath = None
- # Absolute directories will be handled correctly by the os.path.join call.
- if not directory.startswith('.') and not directory.startswith('..'):
- cdpaths = env.get('CDPATH', '.').split(';')
- for cdpath in cdpaths:
- p = os.path.join(cdpath, directory)
- if os.path.isdir(p):
- curpath = p
- break
-
- if curpath is None:
- curpath = directory
- curpath = os.path.join(env['PWD'], directory)
-
- env['OLDPWD'] = env['PWD']
- env['PWD'] = curpath
- if printdir:
- stdout.write('%s\n' % curpath)
- return 0
-
-#-------------------------------------------------------------------------------
-# colon utility
-#-------------------------------------------------------------------------------
-def utility_colon(name, args, interp, env, stdin, stdout, stderr, debugflags):
- if 'debug-utility' in debugflags:
- print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n')
- return 0
-
-#-------------------------------------------------------------------------------
-# echo utility
-#-------------------------------------------------------------------------------
-def utility_echo(name, args, interp, env, stdin, stdout, stderr, debugflags):
- if 'debug-utility' in debugflags:
- print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n')
-
- # Echo only takes arguments, no options. Use printf if you need fancy stuff.
- output = ' '.join(args) + '\n'
- stdout.write(output)
- stdout.flush()
- return 0
-
-#-------------------------------------------------------------------------------
-# egrep utility
-#-------------------------------------------------------------------------------
-# egrep is usually a shell script.
-# Unfortunately, pysh does not support shell scripts *with arguments* right now,
-# so the redirection is implemented here, assuming grep is available.
-def utility_egrep(name, args, interp, env, stdin, stdout, stderr, debugflags):
- if 'debug-utility' in debugflags:
- print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n')
-
- return run_command('grep', ['-E'] + args, interp, env, stdin, stdout,
- stderr, debugflags)
-
-#-------------------------------------------------------------------------------
-# env utility
-#-------------------------------------------------------------------------------
-def utility_env(name, args, interp, env, stdin, stdout, stderr, debugflags):
- if 'debug-utility' in debugflags:
- print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n')
-
- if args and args[0]=='-i':
- raise NotImplementedError('env: -i option is not implemented')
-
- i = 0
- for arg in args:
- if '=' not in arg:
- break
- # Update the current environment
- name, value = arg.split('=', 1)
- env[name] = value
- i += 1
-
- if args[i:]:
- # Find then execute the specified interpreter
- utility = env.find_in_path(args[i])
- if not utility:
- return 127
- args[i:i+1] = utility
- name = args[i]
- args = args[i+1:]
- try:
- return run_command(name, args, interp, env, stdin, stdout, stderr,
- debugflags)
- except UtilityError:
- stderr.write('env: failed to execute %s' % ' '.join([name]+args))
- return 126
- else:
- for pair in env.get_variables().iteritems():
- stdout.write('%s=%s\n' % pair)
- return 0
-
-#-------------------------------------------------------------------------------
-# exit utility
-#-------------------------------------------------------------------------------
-def utility_exit(name, args, interp, env, stdin, stdout, stderr, debugflags):
- if 'debug-utility' in debugflags:
- print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n')
-
- res = None
- if args:
- try:
- res = int(args[0])
- except ValueError:
- res = None
- if not 0<=res<=255:
- res = None
-
- if res is None:
- # BUG: should be last executed command exit code
- res = 0
-
- raise ExitSignal(res)
-
-#-------------------------------------------------------------------------------
-# fgrep utility
-#-------------------------------------------------------------------------------
-# see egrep
-def utility_fgrep(name, args, interp, env, stdin, stdout, stderr, debugflags):
- if 'debug-utility' in debugflags:
- print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n')
-
- return run_command('grep', ['-F'] + args, interp, env, stdin, stdout,
- stderr, debugflags)
-
-#-------------------------------------------------------------------------------
-# gunzip utility
-#-------------------------------------------------------------------------------
-# see egrep
-def utility_gunzip(name, args, interp, env, stdin, stdout, stderr, debugflags):
- if 'debug-utility' in debugflags:
- print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n')
-
- return run_command('gzip', ['-d'] + args, interp, env, stdin, stdout,
- stderr, debugflags)
-
-#-------------------------------------------------------------------------------
-# kill utility
-#-------------------------------------------------------------------------------
-def utility_kill(name, args, interp, env, stdin, stdout, stderr, debugflags):
- if 'debug-utility' in debugflags:
- print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n')
-
- for arg in args:
- pid = int(arg)
- status = subprocess.call(['pskill', '/T', str(pid)],
- shell=True,
- stdout=subprocess.PIPE,
- stderr=subprocess.PIPE)
- # pskill is asynchronous, hence the stupid polling loop
- while 1:
- p = subprocess.Popen(['pslist', str(pid)],
- shell=True,
- stdout=subprocess.PIPE,
- stderr=subprocess.STDOUT)
- output = p.communicate()[0]
- if ('process %d was not' % pid) in output:
- break
- time.sleep(1)
- return status
-
-#-------------------------------------------------------------------------------
-# mkdir utility
-#-------------------------------------------------------------------------------
-OPT_MKDIR = NonExitingParser("mkdir - make directories.")
-OPT_MKDIR.add_option('-p', action='store_true', dest='has_p', default=False)
-
-def utility_mkdir(name, args, interp, env, stdin, stdout, stderr, debugflags):
- if 'debug-utility' in debugflags:
- print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n')
-
- # TODO: implement umask
- # TODO: implement proper utility error report
- option, args = OPT_MKDIR.parse_args(args)
- for arg in args:
- path = os.path.join(env['PWD'], arg)
- if option.has_p:
- try:
- os.makedirs(path)
- except IOError as e:
- if e.errno != errno.EEXIST:
- raise
- else:
- os.mkdir(path)
- return 0
-
-#-------------------------------------------------------------------------------
-# netstat utility
-#-------------------------------------------------------------------------------
-def utility_netstat(name, args, interp, env, stdin, stdout, stderr, debugflags):
- # Do you really expect me to implement netstat ?
- # This empty form is enough for Mercurial tests since it's
- # supposed to generate nothing upon success. Faking this test
- # is not a big deal either.
- if 'debug-utility' in debugflags:
- print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n')
- return 0
-
-#-------------------------------------------------------------------------------
-# pwd utility
-#-------------------------------------------------------------------------------
-OPT_PWD = NonExitingParser("pwd - return working directory name")
-OPT_PWD.add_option('-L', action='store_true', dest='has_L', default=True,
- help="""If the PWD environment variable contains an absolute pathname of \
- the current directory that does not contain the filenames dot or dot-dot, \
- pwd shall write this pathname to standard output. Otherwise, the -L option \
- shall behave as the -P option.""")
-OPT_PWD.add_option('-P', action='store_true', dest='has_L', default=False,
- help="""The absolute pathname written shall not contain filenames that, in \
- the context of the pathname, refer to files of type symbolic link.""")
-
-def utility_pwd(name, args, interp, env, stdin, stdout, stderr, debugflags):
- if 'debug-utility' in debugflags:
- print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n')
-
- option, args = OPT_PWD.parse_args(args)
- stdout.write('%s\n' % env['PWD'])
- return 0
-
-#-------------------------------------------------------------------------------
-# printf utility
-#-------------------------------------------------------------------------------
-RE_UNESCAPE = re.compile(r'(\\x[a-zA-Z0-9]{2}|\\[0-7]{1,3}|\\.)')
-
-def utility_printf(name, args, interp, env, stdin, stdout, stderr, debugflags):
- if 'debug-utility' in debugflags:
- print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n')
-
- def replace(m):
- assert m.group()
- g = m.group()[1:]
- if g.startswith('x'):
- return chr(int(g[1:], 16))
- if len(g) <= 3 and len([c for c in g if c in '01234567']) == len(g):
- # Yay, an octal number
- return chr(int(g, 8))
- return {
- 'a': '\a',
- 'b': '\b',
- 'f': '\f',
- 'n': '\n',
- 'r': '\r',
- 't': '\t',
- 'v': '\v',
- '\\': '\\',
- }.get(g)
-
- # Convert escape sequences
- format = re.sub(RE_UNESCAPE, replace, args[0])
- stdout.write(format % tuple(args[1:]))
- return 0
-
-#-------------------------------------------------------------------------------
-# true utility
-#-------------------------------------------------------------------------------
-def utility_true(name, args, interp, env, stdin, stdout, stderr, debugflags):
- if 'debug-utility' in debugflags:
- print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n')
- return 0
-
-#-------------------------------------------------------------------------------
-# sed utility
-#-------------------------------------------------------------------------------
-RE_SED = re.compile(r'^s(.).*\1[a-zA-Z]*$')
-
-# cygwin sed fails with some expressions when they do not end with a single space.
-# see unit tests for details. Interestingly, the same expressions works perfectly
-# in cygwin shell.
-def utility_sed(name, args, interp, env, stdin, stdout, stderr, debugflags):
- if 'debug-utility' in debugflags:
- print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n')
-
- # Scan pattern arguments and append a space if necessary
- for i in range(len(args)):
- if not RE_SED.search(args[i]):
- continue
- args[i] = args[i] + ' '
-
- return run_command(name, args, interp, env, stdin, stdout,
- stderr, debugflags)
-
-#-------------------------------------------------------------------------------
-# sleep utility
-#-------------------------------------------------------------------------------
-def utility_sleep(name, args, interp, env, stdin, stdout, stderr, debugflags):
- if 'debug-utility' in debugflags:
- print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n')
- time.sleep(int(args[0]))
- return 0
-
-#-------------------------------------------------------------------------------
-# sort utility
-#-------------------------------------------------------------------------------
-OPT_SORT = NonExitingParser("sort - sort, merge, or sequence check text files")
-
-def utility_sort(name, args, interp, env, stdin, stdout, stderr, debugflags):
-
- def sort(path):
- if path == '-':
- lines = stdin.readlines()
- else:
- try:
- f = file(path)
- try:
- lines = f.readlines()
- finally:
- f.close()
- except IOError as e:
- stderr.write(str(e) + '\n')
- return 1
-
- if lines and lines[-1][-1]!='\n':
- lines[-1] = lines[-1] + '\n'
- return lines
-
- if 'debug-utility' in debugflags:
- print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n')
-
- option, args = OPT_SORT.parse_args(args)
- alllines = []
-
- if len(args)<=0:
- args += ['-']
-
- # Load all files lines
- curdir = os.getcwd()
- try:
- os.chdir(env['PWD'])
- for path in args:
- alllines += sort(path)
- finally:
- os.chdir(curdir)
-
- alllines.sort()
- for line in alllines:
- stdout.write(line)
- return 0
-
-#-------------------------------------------------------------------------------
-# hg utility
-#-------------------------------------------------------------------------------
-
-hgcommands = [
- 'add',
- 'addremove',
- 'commit', 'ci',
- 'debugrename',
- 'debugwalk',
- 'falabala', # Dummy command used in a mercurial test
- 'incoming',
- 'locate',
- 'pull',
- 'push',
- 'qinit',
- 'remove', 'rm',
- 'rename', 'mv',
- 'revert',
- 'showconfig',
- 'status', 'st',
- 'strip',
- ]
-
-def rewriteslashes(name, args):
- # Several hg commands output file paths, rewrite the separators
- if len(args) > 1 and name.lower().endswith('python') \
- and args[0].endswith('hg'):
- for cmd in hgcommands:
- if cmd in args[1:]:
- return True
-
- # svn output contains many paths with OS specific separators.
- # Normalize these to unix paths.
- base = os.path.basename(name)
- if base.startswith('svn'):
- return True
-
- return False
-
-def rewritehg(output):
- if not output:
- return output
- # Rewrite os specific messages
- output = output.replace(': The system cannot find the file specified',
- ': No such file or directory')
- output = re.sub(': Access is denied.*$', ': Permission denied', output)
- output = output.replace(': No connection could be made because the target machine actively refused it',
- ': Connection refused')
- return output
-
-
-def run_command(name, args, interp, env, stdin, stdout,
- stderr, debugflags):
- # Execute the command
- if 'debug-utility' in debugflags:
- print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n')
-
- hgbin = interp.options().hgbinary
- ishg = hgbin and ('hg' in name or args and 'hg' in args[0])
- unixoutput = 'cygwin' in name or ishg
-
- exec_env = env.get_variables()
- try:
- # BUG: comparing file descriptor is clearly not a reliable way to tell
- # whether they point on the same underlying object. But in pysh limited
- # scope this is usually right, we do not expect complicated redirections
- # besides usual 2>&1.
- # Still there is one case we have but cannot deal with is when stdout
- # and stderr are redirected *by pysh caller*. This the reason for the
- # --redirect pysh() option.
- # Now, we want to know they are the same because we sometimes need to
- # transform the command output, mostly remove CR-LF to ensure that
- # command output is unix-like. Cygwin utilies are a special case because
- # they explicitely set their output streams to binary mode, so we have
- # nothing to do. For all others commands, we have to guess whether they
- # are sending text data, in which case the transformation must be done.
- # Again, the NUL character test is unreliable but should be enough for
- # hg tests.
- redirected = stdout.fileno()==stderr.fileno()
- if not redirected:
- p = subprocess.Popen([name] + args, cwd=env['PWD'], env=exec_env,
- stdin=stdin, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
- else:
- p = subprocess.Popen([name] + args, cwd=env['PWD'], env=exec_env,
- stdin=stdin, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
- out, err = p.communicate()
- except WindowsError as e:
- raise UtilityError(str(e))
-
- if not unixoutput:
- def encode(s):
- if '\0' in s:
- return s
- return s.replace('\r\n', '\n')
- else:
- encode = lambda s: s
-
- if rewriteslashes(name, args):
- encode1_ = encode
- def encode(s):
- s = encode1_(s)
- s = s.replace('\\\\', '\\')
- s = s.replace('\\', '/')
- return s
-
- if ishg:
- encode2_ = encode
- def encode(s):
- return rewritehg(encode2_(s))
-
- stdout.write(encode(out))
- if not redirected:
- stderr.write(encode(err))
- return p.returncode
-
diff --git a/bitbake/lib/bb/pysh/interp.py b/bitbake/lib/bb/pysh/interp.py
deleted file mode 100644
index d14ecf3..0000000
--- a/bitbake/lib/bb/pysh/interp.py
+++ /dev/null
@@ -1,1367 +0,0 @@
-# interp.py - shell interpreter for pysh.
-#
-# Copyright 2007 Patrick Mezard
-#
-# This software may be used and distributed according to the terms
-# of the GNU General Public License, incorporated herein by reference.
-
-"""Implement the shell interpreter.
-
-Most references are made to "The Open Group Base Specifications Issue 6".
-<http://www.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html>
-"""
-# TODO: document the fact input streams must implement fileno() so Popen will work correctly.
-# it requires non-stdin stream to be implemented as files. Still to be tested...
-# DOC: pathsep is used in PATH instead of ':'. Clearly, there are path syntax issues here.
-# TODO: stop command execution upon error.
-# TODO: sort out the filename/io_number mess. It should be possible to use filenames only.
-# TODO: review subshell implementation
-# TODO: test environment cloning for non-special builtins
-# TODO: set -x should not rebuild commands from tokens, assignments/redirections are lost
-# TODO: unit test for variable assignment
-# TODO: test error management wrt error type/utility type
-# TODO: test for binary output everywhere
-# BUG: debug-parsing does not pass log file to PLY. Maybe a PLY upgrade is necessary.
-import base64
-import cPickle as pickle
-import errno
-import glob
-import os
-import re
-import subprocess
-import sys
-import tempfile
-
-try:
- s = set()
- del s
-except NameError:
- from Set import Set as set
-
-import builtin
-from sherrors import *
-import pyshlex
-import pyshyacc
-
-def mappend(func, *args, **kargs):
- """Like map but assume func returns a list. Returned lists are merged into
- a single one.
- """
- return reduce(lambda a,b: a+b, map(func, *args, **kargs), [])
-
-class FileWrapper:
- """File object wrapper to ease debugging.
-
- Allow mode checking and implement file duplication through a simple
- reference counting scheme. Not sure the latter is really useful since
- only real file descriptors can be used.
- """
- def __init__(self, mode, file, close=True):
- if mode not in ('r', 'w', 'a'):
- raise IOError('invalid mode: %s' % mode)
- self._mode = mode
- self._close = close
- if isinstance(file, FileWrapper):
- if file._refcount[0] <= 0:
- raise IOError(0, 'Error')
- self._refcount = file._refcount
- self._refcount[0] += 1
- self._file = file._file
- else:
- self._refcount = [1]
- self._file = file
-
- def dup(self):
- return FileWrapper(self._mode, self, self._close)
-
- def fileno(self):
- """fileno() should be only necessary for input streams."""
- return self._file.fileno()
-
- def read(self, size=-1):
- if self._mode!='r':
- raise IOError(0, 'Error')
- return self._file.read(size)
-
- def readlines(self, *args, **kwargs):
- return self._file.readlines(*args, **kwargs)
-
- def write(self, s):
- if self._mode not in ('w', 'a'):
- raise IOError(0, 'Error')
- return self._file.write(s)
-
- def flush(self):
- self._file.flush()
-
- def close(self):
- if not self._refcount:
- return
- assert self._refcount[0] > 0
-
- self._refcount[0] -= 1
- if self._refcount[0] == 0:
- self._mode = 'c'
- if self._close:
- self._file.close()
- self._refcount = None
-
- def mode(self):
- return self._mode
-
- def __getattr__(self, name):
- if name == 'name':
- self.name = getattr(self._file, name)
- return self.name
- else:
- raise AttributeError(name)
-
- def __del__(self):
- self.close()
-
-
-def win32_open_devnull(mode):
- return open('NUL', mode)
-
-
-class Redirections:
- """Stores open files and their mapping to pseudo-sh file descriptor.
- """
- # BUG: redirections are not handled correctly: 1>&3 2>&3 3>&4 does
- # not make 1 to redirect to 4
- def __init__(self, stdin=None, stdout=None, stderr=None):
- self._descriptors = {}
- if stdin is not None:
- self._add_descriptor(0, stdin)
- if stdout is not None:
- self._add_descriptor(1, stdout)
- if stderr is not None:
- self._add_descriptor(2, stderr)
-
- def add_here_document(self, interp, name, content, io_number=None):
- if io_number is None:
- io_number = 0
-
- if name==pyshlex.unquote_wordtree(name):
- content = interp.expand_here_document(('TOKEN', content))
-
- # Write document content in a temporary file
- tmp = tempfile.TemporaryFile()
- try:
- tmp.write(content)
- tmp.flush()
- tmp.seek(0)
- self._add_descriptor(io_number, FileWrapper('r', tmp))
- except:
- tmp.close()
- raise
-
- def add(self, interp, op, filename, io_number=None):
- if op not in ('<', '>', '>|', '>>', '>&'):
- # TODO: add descriptor duplication and here_documents
- raise RedirectionError('Unsupported redirection operator "%s"' % op)
-
- if io_number is not None:
- io_number = int(io_number)
-
- if (op == '>&' and filename.isdigit()) or filename=='-':
- # No expansion for file descriptors, quote them if you want a filename
- fullname = filename
- else:
- if filename.startswith('/'):
- # TODO: win32 kludge
- if filename=='/dev/null':
- fullname = 'NUL'
- else:
- # TODO: handle absolute pathnames, they are unlikely to exist on the
- # current platform (win32 for instance).
- raise NotImplementedError()
- else:
- fullname = interp.expand_redirection(('TOKEN', filename))
- if not fullname:
- raise RedirectionError('%s: ambiguous redirect' % filename)
- # Build absolute path based on PWD
- fullname = os.path.join(interp.get_env()['PWD'], fullname)
-
- if op=='<':
- return self._add_input_redirection(interp, fullname, io_number)
- elif op in ('>', '>|'):
- clobber = ('>|'==op)
- return self._add_output_redirection(interp, fullname, io_number, clobber)
- elif op=='>>':
- return self._add_output_appending(interp, fullname, io_number)
- elif op=='>&':
- return self._dup_output_descriptor(fullname, io_number)
-
- def close(self):
- if self._descriptors is not None:
- for desc in self._descriptors.itervalues():
- desc.flush()
- desc.close()
- self._descriptors = None
-
- def stdin(self):
- return self._descriptors[0]
-
- def stdout(self):
- return self._descriptors[1]
-
- def stderr(self):
- return self._descriptors[2]
-
- def clone(self):
- clone = Redirections()
- for desc, fileobj in self._descriptors.iteritems():
- clone._descriptors[desc] = fileobj.dup()
- return clone
-
- def _add_output_redirection(self, interp, filename, io_number, clobber):
- if io_number is None:
- # io_number default to standard output
- io_number = 1
-
- if not clobber and interp.get_env().has_opt('-C') and os.path.isfile(filename):
- # File already exist in no-clobber mode, bail out
- raise RedirectionError('File "%s" already exists' % filename)
-
- # Open and register
- self._add_file_descriptor(io_number, filename, 'w')
-
- def _add_output_appending(self, interp, filename, io_number):
- if io_number is None:
- io_number = 1
- self._add_file_descriptor(io_number, filename, 'a')
-
- def _add_input_redirection(self, interp, filename, io_number):
- if io_number is None:
- io_number = 0
- self._add_file_descriptor(io_number, filename, 'r')
-
- def _add_file_descriptor(self, io_number, filename, mode):
- try:
- if filename.startswith('/'):
- if filename=='/dev/null':
- f = win32_open_devnull(mode+'b')
- else:
- # TODO: handle absolute pathnames, they are unlikely to exist on the
- # current platform (win32 for instance).
- raise NotImplementedError('cannot open absolute path %s' % repr(filename))
- else:
- f = file(filename, mode+'b')
- except IOError as e:
- raise RedirectionError(str(e))
-
- wrapper = None
- try:
- wrapper = FileWrapper(mode, f)
- f = None
- self._add_descriptor(io_number, wrapper)
- except:
- if f: f.close()
- if wrapper: wrapper.close()
- raise
-
- def _dup_output_descriptor(self, source_fd, dest_fd):
- if source_fd is None:
- source_fd = 1
- self._dup_file_descriptor(source_fd, dest_fd, 'w')
-
- def _dup_file_descriptor(self, source_fd, dest_fd, mode):
- source_fd = int(source_fd)
- if source_fd not in self._descriptors:
- raise RedirectionError('"%s" is not a valid file descriptor' % str(source_fd))
- source = self._descriptors[source_fd]
-
- if source.mode()!=mode:
- raise RedirectionError('Descriptor %s cannot be duplicated in mode "%s"' % (str(source), mode))
-
- if dest_fd=='-':
- # Close the source descriptor
- del self._descriptors[source_fd]
- source.close()
- else:
- dest_fd = int(dest_fd)
- if dest_fd not in self._descriptors:
- raise RedirectionError('Cannot replace file descriptor %s' % str(dest_fd))
-
- dest = self._descriptors[dest_fd]
- if dest.mode()!=mode:
- raise RedirectionError('Descriptor %s cannot be cannot be redirected in mode "%s"' % (str(dest), mode))
-
- self._descriptors[dest_fd] = source.dup()
- dest.close()
-
- def _add_descriptor(self, io_number, file):
- io_number = int(io_number)
-
- if io_number in self._descriptors:
- # Close the current descriptor
- d = self._descriptors[io_number]
- del self._descriptors[io_number]
- d.close()
-
- self._descriptors[io_number] = file
-
- def __str__(self):
- names = [('%d=%r' % (k, getattr(v, 'name', None))) for k,v
- in self._descriptors.iteritems()]
- names = ','.join(names)
- return 'Redirections(%s)' % names
-
- def __del__(self):
- self.close()
-
-def cygwin_to_windows_path(path):
- """Turn /cygdrive/c/foo into c:/foo, or return path if it
- is not a cygwin path.
- """
- if not path.startswith('/cygdrive/'):
- return path
- path = path[len('/cygdrive/'):]
- path = path[:1] + ':' + path[1:]
- return path
-
-def win32_to_unix_path(path):
- if path is not None:
- path = path.replace('\\', '/')
- return path
-
-_RE_SHEBANG = re.compile(r'^\#!\s?([^\s]+)(?:\s([^\s]+))?')
-_SHEBANG_CMDS = {
- '/usr/bin/env': 'env',
- '/bin/sh': 'pysh',
- 'python': 'python',
-}
-
-def resolve_shebang(path, ignoreshell=False):
- """Return a list of arguments as shebang interpreter call or an empty list
- if path does not refer to an executable script.
- See <http://www.opengroup.org/austin/docs/austin_51r2.txt>.
-
- ignoreshell - set to True to ignore sh shebangs. Return an empty list instead.
- """
- try:
- f = file(path)
- try:
- # At most 80 characters in the first line
- header = f.read(80).splitlines()[0]
- finally:
- f.close()
-
- m = _RE_SHEBANG.search(header)
- if not m:
- return []
- cmd, arg = m.group(1,2)
- if os.path.isfile(cmd):
- # Keep this one, the hg script for instance contains a weird windows
- # shebang referencing the current python install.
- cmdfile = os.path.basename(cmd).lower()
- if cmdfile == 'python.exe':
- cmd = 'python'
- pass
- elif cmd not in _SHEBANG_CMDS:
- raise CommandNotFound('Unknown interpreter "%s" referenced in '\
- 'shebang' % header)
- cmd = _SHEBANG_CMDS.get(cmd)
- if cmd is None or (ignoreshell and cmd == 'pysh'):
- return []
- if arg is None:
- return [cmd, win32_to_unix_path(path)]
- return [cmd, arg, win32_to_unix_path(path)]
- except IOError as e:
- if e.errno!=errno.ENOENT and \
- (e.errno!=errno.EPERM and not os.path.isdir(path)): # Opening a directory raises EPERM
- raise
- return []
-
-def win32_find_in_path(name, path):
- if isinstance(path, str):
- path = path.split(os.pathsep)
-
- exts = os.environ.get('PATHEXT', '').lower().split(os.pathsep)
- for p in path:
- p_name = os.path.join(p, name)
-
- prefix = resolve_shebang(p_name)
- if prefix:
- return prefix
-
- for ext in exts:
- p_name_ext = p_name + ext
- if os.path.exists(p_name_ext):
- return [win32_to_unix_path(p_name_ext)]
- return []
-
-class Traps(dict):
- def __setitem__(self, key, value):
- if key not in ('EXIT',):
- raise NotImplementedError()
- super(Traps, self).__setitem__(key, value)
-
-# IFS white spaces character class
-_IFS_WHITESPACES = (' ', '\t', '\n')
-
-class Environment:
- """Environment holds environment variables, export table, function
- definitions and whatever is defined in 2.12 "Shell Execution Environment",
- redirection excepted.
- """
- def __init__(self, pwd):
- self._opt = set() #Shell options
-
- self._functions = {}
- self._env = {'?': '0', '#': '0'}
- self._exported = set([
- 'HOME', 'IFS', 'PATH'
- ])
-
- # Set environment vars with side-effects
- self._ifs_ws = None # Set of IFS whitespace characters
- self._ifs_re = None # Regular expression used to split between words using IFS classes
- self['IFS'] = ''.join(_IFS_WHITESPACES) #Default environment values
- self['PWD'] = pwd
- self.traps = Traps()
-
- def clone(self, subshell=False):
- env = Environment(self['PWD'])
- env._opt = set(self._opt)
- for k,v in self.get_variables().iteritems():
- if k in self._exported:
- env.export(k,v)
- elif subshell:
- env[k] = v
-
- if subshell:
- env._functions = dict(self._functions)
-
- return env
-
- def __getitem__(self, key):
- if key in ('@', '*', '-', '$'):
- raise NotImplementedError('%s is not implemented' % repr(key))
- return self._env[key]
-
- def get(self, key, defval=None):
- try:
- return self[key]
- except KeyError:
- return defval
-
- def __setitem__(self, key, value):
- if key=='IFS':
- # Update the whitespace/non-whitespace classes
- self._update_ifs(value)
- elif key=='PWD':
- pwd = os.path.abspath(value)
- if not os.path.isdir(pwd):
- raise VarAssignmentError('Invalid directory %s' % value)
- value = pwd
- elif key in ('?', '!'):
- value = str(int(value))
- self._env[key] = value
-
- def __delitem__(self, key):
- if key in ('IFS', 'PWD', '?'):
- raise VarAssignmentError('%s cannot be unset' % key)
- del self._env[key]
-
- def __contains__(self, item):
- return item in self._env
-
- def set_positional_args(self, args):
- """Set the content of 'args' as positional argument from 1 to len(args).
- Return previous argument as a list of strings.
- """
- # Save and remove previous arguments
- prevargs = []
- for i in range(int(self._env['#'])):
- i = str(i+1)
- prevargs.append(self._env[i])
- del self._env[i]
- self._env['#'] = '0'
-
- #Set new ones
- for i,arg in enumerate(args):
- self._env[str(i+1)] = str(arg)
- self._env['#'] = str(len(args))
-
- return prevargs
-
- def get_positional_args(self):
- return [self._env[str(i+1)] for i in range(int(self._env['#']))]
-
- def get_variables(self):
- return dict(self._env)
-
- def export(self, key, value=None):
- if value is not None:
- self[key] = value
- self._exported.add(key)
-
- def get_exported(self):
- return [(k,self._env.get(k)) for k in self._exported]
-
- def split_fields(self, word):
- if not self._ifs_ws or not word:
- return [word]
- return re.split(self._ifs_re, word)
-
- def _update_ifs(self, value):
- """Update the split_fields related variables when IFS character set is
- changed.
- """
- # TODO: handle NULL IFS
-
- # Separate characters in whitespace and non-whitespace
- chars = set(value)
- ws = [c for c in chars if c in _IFS_WHITESPACES]
- nws = [c for c in chars if c not in _IFS_WHITESPACES]
-
- # Keep whitespaces in a string for left and right stripping
- self._ifs_ws = ''.join(ws)
-
- # Build a regexp to split fields
- trailing = '[' + ''.join([re.escape(c) for c in ws]) + ']'
- if nws:
- # First, the single non-whitespace occurence.
- nws = '[' + ''.join([re.escape(c) for c in nws]) + ']'
- nws = '(?:' + trailing + '*' + nws + trailing + '*' + '|' + trailing + '+)'
- else:
- # Then mix all parts with quantifiers
- nws = trailing + '+'
- self._ifs_re = re.compile(nws)
-
- def has_opt(self, opt, val=None):
- return (opt, val) in self._opt
-
- def set_opt(self, opt, val=None):
- self._opt.add((opt, val))
-
- def find_in_path(self, name, pwd=False):
- path = self._env.get('PATH', '').split(os.pathsep)
- if pwd:
- path[:0] = [self['PWD']]
- if os.name == 'nt':
- return win32_find_in_path(name, self._env.get('PATH', ''))
- else:
- raise NotImplementedError()
-
- def define_function(self, name, body):
- if not is_name(name):
- raise ShellSyntaxError('%s is not a valid function name' % repr(name))
- self._functions[name] = body
-
- def remove_function(self, name):
- del self._functions[name]
-
- def is_function(self, name):
- return name in self._functions
-
- def get_function(self, name):
- return self._functions.get(name)
-
-
-name_charset = 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_'
-name_charset = dict(zip(name_charset,name_charset))
-
-def match_name(s):
- """Return the length in characters of the longest prefix made of name
- allowed characters in s.
- """
- for i,c in enumerate(s):
- if c not in name_charset:
- return s[:i]
- return s
-
-def is_name(s):
- return len([c for c in s if c not in name_charset])<=0
-
-def is_special_param(c):
- return len(c)==1 and c in ('@','*','#','?','-','$','!','0')
-
-def utility_not_implemented(name, *args, **kwargs):
- raise NotImplementedError('%s utility is not implemented' % name)
-
-
-class Utility:
- """Define utilities properties:
- func -- utility callable. See builtin module for utility samples.
- is_special -- see XCU 2.8.
- """
- def __init__(self, func, is_special=0):
- self.func = func
- self.is_special = bool(is_special)
-
-
-def encodeargs(args):
- def encodearg(s):
- lines = base64.encodestring(s)
- lines = [l.splitlines()[0] for l in lines]
- return ''.join(lines)
-
- s = pickle.dumps(args)
- return encodearg(s)
-
-def decodeargs(s):
- s = base64.decodestring(s)
- return pickle.loads(s)
-
-
-class GlobError(Exception):
- pass
-
-class Options:
- def __init__(self):
- # True if Mercurial operates with binary streams
- self.hgbinary = True
-
-class Interpreter:
- # Implementation is very basic: the execute() method just makes a DFS on the
- # AST and execute nodes one by one. Nodes are tuple (name,obj) where name
- # is a string identifier and obj the AST element returned by the parser.
- #
- # Handler are named after the node identifiers.
- # TODO: check node names and remove the switch in execute with some
- # dynamic getattr() call to find node handlers.
- """Shell interpreter.
-
- The following debugging flags can be passed:
- debug-parsing - enable PLY debugging.
- debug-tree - print the generated AST.
- debug-cmd - trace command execution before word expansion, plus exit status.
- debug-utility - trace utility execution.
- """
-
- # List supported commands.
- COMMANDS = {
- 'cat': Utility(builtin.utility_cat,),
- 'cd': Utility(builtin.utility_cd,),
- ':': Utility(builtin.utility_colon,),
- 'echo': Utility(builtin.utility_echo),
- 'env': Utility(builtin.utility_env),
- 'exit': Utility(builtin.utility_exit),
- 'export': Utility(builtin.builtin_export, is_special=1),
- 'egrep': Utility(builtin.utility_egrep),
- 'fgrep': Utility(builtin.utility_fgrep),
- 'gunzip': Utility(builtin.utility_gunzip),
- 'kill': Utility(builtin.utility_kill),
- 'mkdir': Utility(builtin.utility_mkdir),
- 'netstat': Utility(builtin.utility_netstat),
- 'printf': Utility(builtin.utility_printf),
- 'pwd': Utility(builtin.utility_pwd),
- 'return': Utility(builtin.builtin_return, is_special=1),
- 'sed': Utility(builtin.utility_sed,),
- 'set': Utility(builtin.builtin_set,),
- 'shift': Utility(builtin.builtin_shift,),
- 'sleep': Utility(builtin.utility_sleep,),
- 'sort': Utility(builtin.utility_sort,),
- 'trap': Utility(builtin.builtin_trap, is_special=1),
- 'true': Utility(builtin.utility_true),
- 'unset': Utility(builtin.builtin_unset, is_special=1),
- 'wait': Utility(builtin.builtin_wait, is_special=1),
- }
-
- def __init__(self, pwd, debugflags = [], env=None, redirs=None, stdin=None,
- stdout=None, stderr=None, opts=Options()):
- self._env = env
- if self._env is None:
- self._env = Environment(pwd)
- self._children = {}
-
- self._redirs = redirs
- self._close_redirs = False
-
- if self._redirs is None:
- if stdin is None:
- stdin = sys.stdin
- if stdout is None:
- stdout = sys.stdout
- if stderr is None:
- stderr = sys.stderr
- stdin = FileWrapper('r', stdin, False)
- stdout = FileWrapper('w', stdout, False)
- stderr = FileWrapper('w', stderr, False)
- self._redirs = Redirections(stdin, stdout, stderr)
- self._close_redirs = True
-
- self._debugflags = list(debugflags)
- self._logfile = sys.stderr
- self._options = opts
-
- def close(self):
- """Must be called when the interpreter is no longer used."""
- script = self._env.traps.get('EXIT')
- if script:
- try:
- self.execute_script(script=script)
- except:
- pass
-
- if self._redirs is not None and self._close_redirs:
- self._redirs.close()
- self._redirs = None
-
- def log(self, s):
- self._logfile.write(s)
- self._logfile.flush()
-
- def __getitem__(self, key):
- return self._env[key]
-
- def __setitem__(self, key, value):
- self._env[key] = value
-
- def options(self):
- return self._options
-
- def redirect(self, redirs, ios):
- def add_redir(io):
- if isinstance(io, pyshyacc.IORedirect):
- redirs.add(self, io.op, io.filename, io.io_number)
- else:
- redirs.add_here_document(self, io.name, io.content, io.io_number)
-
- map(add_redir, ios)
- return redirs
-
- def execute_script(self, script=None, ast=None, sourced=False,
- scriptpath=None):
- """If script is not None, parse the input. Otherwise takes the supplied
- AST. Then execute the AST.
- Return the script exit status.
- """
- try:
- if scriptpath is not None:
- self._env['0'] = os.path.abspath(scriptpath)
-
- if script is not None:
- debug_parsing = ('debug-parsing' in self._debugflags)
- cmds, script = pyshyacc.parse(script, True, debug_parsing)
- if 'debug-tree' in self._debugflags:
- pyshyacc.print_commands(cmds, self._logfile)
- self._logfile.flush()
- else:
- cmds, script = ast, ''
-
- status = 0
- for cmd in cmds:
- try:
- status = self.execute(cmd)
- except ExitSignal as e:
- if sourced:
- raise
- status = int(e.args[0])
- return status
- except ShellError:
- self._env['?'] = 1
- raise
- if 'debug-utility' in self._debugflags or 'debug-cmd' in self._debugflags:
- self.log('returncode ' + str(status)+ '\n')
- return status
- except CommandNotFound as e:
- print >>self._redirs.stderr, str(e)
- self._redirs.stderr.flush()
- # Command not found by non-interactive shell
- # return 127
- raise
- except RedirectionError as e:
- # TODO: should be handled depending on the utility status
- print >>self._redirs.stderr, str(e)
- self._redirs.stderr.flush()
- # Command not found by non-interactive shell
- # return 127
- raise
-
- def dotcommand(self, env, args):
- if len(args) < 1:
- raise ShellError('. expects at least one argument')
- path = args[0]
- if '/' not in path:
- found = env.find_in_path(args[0], True)
- if found:
- path = found[0]
- script = file(path).read()
- return self.execute_script(script=script, sourced=True)
-
- def execute(self, token, redirs=None):
- """Execute and AST subtree with supplied redirections overriding default
- interpreter ones.
- Return the exit status.
- """
- if not token:
- return 0
-
- if redirs is None:
- redirs = self._redirs
-
- if isinstance(token, list):
- # Commands sequence
- res = 0
- for t in token:
- res = self.execute(t, redirs)
- return res
-
- type, value = token
- status = 0
- if type=='simple_command':
- redirs_copy = redirs.clone()
- try:
- # TODO: define and handle command return values
- # TODO: implement set -e
- status = self._execute_simple_command(value, redirs_copy)
- finally:
- redirs_copy.close()
- elif type=='pipeline':
- status = self._execute_pipeline(value, redirs)
- elif type=='and_or':
- status = self._execute_and_or(value, redirs)
- elif type=='for_clause':
- status = self._execute_for_clause(value, redirs)
- elif type=='while_clause':
- status = self._execute_while_clause(value, redirs)
- elif type=='function_definition':
- status = self._execute_function_definition(value, redirs)
- elif type=='brace_group':
- status = self._execute_brace_group(value, redirs)
- elif type=='if_clause':
- status = self._execute_if_clause(value, redirs)
- elif type=='subshell':
- status = self.subshell(ast=value.cmds, redirs=redirs)
- elif type=='async':
- status = self._asynclist(value)
- elif type=='redirect_list':
- redirs_copy = self.redirect(redirs.clone(), value.redirs)
- try:
- status = self.execute(value.cmd, redirs_copy)
- finally:
- redirs_copy.close()
- else:
- raise NotImplementedError('Unsupported token type ' + type)
-
- if status < 0:
- status = 255
- return status
-
- def _execute_if_clause(self, if_clause, redirs):
- cond_status = self.execute(if_clause.cond, redirs)
- if cond_status==0:
- return self.execute(if_clause.if_cmds, redirs)
- else:
- return self.execute(if_clause.else_cmds, redirs)
-
- def _execute_brace_group(self, group, redirs):
- status = 0
- for cmd in group.cmds:
- status = self.execute(cmd, redirs)
- return status
-
- def _execute_function_definition(self, fundef, redirs):
- self._env.define_function(fundef.name, fundef.body)
- return 0
-
- def _execute_while_clause(self, while_clause, redirs):
- status = 0
- while 1:
- cond_status = 0
- for cond in while_clause.condition:
- cond_status = self.execute(cond, redirs)
-
- if cond_status:
- break
-
- for cmd in while_clause.cmds:
- status = self.execute(cmd, redirs)
-
- return status
-
- def _execute_for_clause(self, for_clause, redirs):
- if not is_name(for_clause.name):
- raise ShellSyntaxError('%s is not a valid name' % repr(for_clause.name))
- items = mappend(self.expand_token, for_clause.items)
-
- status = 0
- for item in items:
- self._env[for_clause.name] = item
- for cmd in for_clause.cmds:
- status = self.execute(cmd, redirs)
- return status
-
- def _execute_and_or(self, or_and, redirs):
- res = self.execute(or_and.left, redirs)
- if (or_and.op=='&&' and res==0) or (or_and.op!='&&' and res!=0):
- res = self.execute(or_and.right, redirs)
- return res
-
- def _execute_pipeline(self, pipeline, redirs):
- if len(pipeline.commands)==1:
- status = self.execute(pipeline.commands[0], redirs)
- else:
- # Execute all commands one after the other
- status = 0
- inpath, outpath = None, None
- try:
- # Commands inputs and outputs cannot really be plugged as done
- # by a real shell. Run commands sequentially and chain their
- # input/output throught temporary files.
- tmpfd, inpath = tempfile.mkstemp()
- os.close(tmpfd)
- tmpfd, outpath = tempfile.mkstemp()
- os.close(tmpfd)
-
- inpath = win32_to_unix_path(inpath)
- outpath = win32_to_unix_path(outpath)
-
- for i, cmd in enumerate(pipeline.commands):
- call_redirs = redirs.clone()
- try:
- if i!=0:
- call_redirs.add(self, '<', inpath)
- if i!=len(pipeline.commands)-1:
- call_redirs.add(self, '>', outpath)
-
- status = self.execute(cmd, call_redirs)
-
- # Chain inputs/outputs
- inpath, outpath = outpath, inpath
- finally:
- call_redirs.close()
- finally:
- if inpath: os.remove(inpath)
- if outpath: os.remove(outpath)
-
- if pipeline.reverse_status:
- status = int(not status)
- self._env['?'] = status
- return status
-
- def _execute_function(self, name, args, interp, env, stdin, stdout, stderr, *others):
- assert interp is self
-
- func = env.get_function(name)
- #Set positional parameters
- prevargs = None
- try:
- prevargs = env.set_positional_args(args)
- try:
- redirs = Redirections(stdin.dup(), stdout.dup(), stderr.dup())
- try:
- status = self.execute(func, redirs)
- finally:
- redirs.close()
- except ReturnSignal as e:
- status = int(e.args[0])
- env['?'] = status
- return status
- finally:
- #Reset positional parameters
- if prevargs is not None:
- env.set_positional_args(prevargs)
-
- def _execute_simple_command(self, token, redirs):
- """Can raise ReturnSignal when return builtin is called, ExitSignal when
- exit is called, and other shell exceptions upon builtin failures.
- """
- debug_command = 'debug-cmd' in self._debugflags
- if debug_command:
- self.log('word' + repr(token.words) + '\n')
- self.log('assigns' + repr(token.assigns) + '\n')
- self.log('redirs' + repr(token.redirs) + '\n')
-
- is_special = None
- env = self._env
-
- try:
- # Word expansion
- args = []
- for word in token.words:
- args += self.expand_token(word)
- if is_special is None and args:
- is_special = env.is_function(args[0]) or \
- (args[0] in self.COMMANDS and self.COMMANDS[args[0]].is_special)
-
- if debug_command:
- self.log('_execute_simple_command' + str(args) + '\n')
-
- if not args:
- # Redirections happen is a subshell
- redirs = redirs.clone()
- elif not is_special:
- env = self._env.clone()
-
- # Redirections
- self.redirect(redirs, token.redirs)
-
- # Variables assignments
- res = 0
- for type,(k,v) in token.assigns:
- status, expanded = self.expand_variable((k,v))
- if status is not None:
- res = status
- if args:
- env.export(k, expanded)
- else:
- env[k] = expanded
-
- if args and args[0] in ('.', 'source'):
- res = self.dotcommand(env, args[1:])
- elif args:
- if args[0] in self.COMMANDS:
- command = self.COMMANDS[args[0]]
- elif env.is_function(args[0]):
- command = Utility(self._execute_function, is_special=True)
- else:
- if not '/' in args[0].replace('\\', '/'):
- cmd = env.find_in_path(args[0])
- if not cmd:
- # TODO: test error code on unknown command => 127
- raise CommandNotFound('Unknown command: "%s"' % args[0])
- else:
- # Handle commands like '/cygdrive/c/foo.bat'
- cmd = cygwin_to_windows_path(args[0])
- if not os.path.exists(cmd):
- raise CommandNotFound('%s: No such file or directory' % args[0])
- shebang = resolve_shebang(cmd)
- if shebang:
- cmd = shebang
- else:
- cmd = [cmd]
- args[0:1] = cmd
- command = Utility(builtin.run_command)
-
- # Command execution
- if 'debug-cmd' in self._debugflags:
- self.log('redirections ' + str(redirs) + '\n')
-
- res = command.func(args[0], args[1:], self, env,
- redirs.stdin(), redirs.stdout(),
- redirs.stderr(), self._debugflags)
-
- if self._env.has_opt('-x'):
- # Trace command execution in shell environment
- # BUG: would be hard to reproduce a real shell behaviour since
- # the AST is not annotated with source lines/tokens.
- self._redirs.stdout().write(' '.join(args))
-
- except ReturnSignal:
- raise
- except ShellError as e:
- if is_special or isinstance(e, (ExitSignal,
- ShellSyntaxError, ExpansionError)):
- raise e
- self._redirs.stderr().write(str(e)+'\n')
- return 1
-
- return res
-
- def expand_token(self, word):
- """Expand a word as specified in [2.6 Word Expansions]. Return the list
- of expanded words.
- """
- status, wtrees = self._expand_word(word)
- return map(pyshlex.wordtree_as_string, wtrees)
-
- def expand_variable(self, word):
- """Return a status code (or None if no command expansion occurred)
- and a single word.
- """
- status, wtrees = self._expand_word(word, pathname=False, split=False)
- words = map(pyshlex.wordtree_as_string, wtrees)
- assert len(words)==1
- return status, words[0]
-
- def expand_here_document(self, word):
- """Return the expanded document as a single word. The here document is
- assumed to be unquoted.
- """
- status, wtrees = self._expand_word(word, pathname=False,
- split=False, here_document=True)
- words = map(pyshlex.wordtree_as_string, wtrees)
- assert len(words)==1
- return words[0]
-
- def expand_redirection(self, word):
- """Return a single word."""
- return self.expand_variable(word)[1]
-
- def get_env(self):
- return self._env
-
- def _expand_word(self, token, pathname=True, split=True, here_document=False):
- wtree = pyshlex.make_wordtree(token[1], here_document=here_document)
-
- # TODO: implement tilde expansion
- def expand(wtree):
- """Return a pseudo wordtree: the tree or its subelements can be empty
- lists when no value result from the expansion.
- """
- status = None
- for part in wtree:
- if not isinstance(part, list):
- continue
- if part[0]in ("'", '\\'):
- continue
- elif part[0] in ('`', '$('):
- status, result = self._expand_command(part)
- part[:] = result
- elif part[0] in ('$', '${'):
- part[:] = self._expand_parameter(part, wtree[0]=='"', split)
- elif part[0] in ('', '"'):
- status, result = expand(part)
- part[:] = result
- else:
- raise NotImplementedError('%s expansion is not implemented'
- % part[0])
- # [] is returned when an expansion result in no-field,
- # like an empty $@
- wtree = [p for p in wtree if p != []]
- if len(wtree) < 3:
- return status, []
- return status, wtree
-
- status, wtree = expand(wtree)
- if len(wtree) == 0:
- return status, wtree
- wtree = pyshlex.normalize_wordtree(wtree)
-
- if split:
- wtrees = self._split_fields(wtree)
- else:
- wtrees = [wtree]
-
- if pathname:
- wtrees = mappend(self._expand_pathname, wtrees)
-
- wtrees = map(self._remove_quotes, wtrees)
- return status, wtrees
-
- def _expand_command(self, wtree):
- # BUG: there is something to do with backslashes and quoted
- # characters here
- command = pyshlex.wordtree_as_string(wtree[1:-1])
- status, output = self.subshell_output(command)
- return status, ['', output, '']
-
- def _expand_parameter(self, wtree, quoted=False, split=False):
- """Return a valid wtree or an empty list when no parameter results."""
- # Get the parameter name
- # TODO: implement weird expansion rules with ':'
- name = pyshlex.wordtree_as_string(wtree[1:-1])
- if not is_name(name) and not is_special_param(name):
- raise ExpansionError('Bad substitution "%s"' % name)
- # TODO: implement special parameters
- if name in ('@', '*'):
- args = self._env.get_positional_args()
- if len(args) == 0:
- return []
- if len(args)<2:
- return ['', ''.join(args), '']
-
- sep = self._env.get('IFS', '')[:1]
- if split and quoted and name=='@':
- # Introduce a new token to tell the caller that these parameters
- # cause a split as specified in 2.5.2
- return ['@'] + args + ['']
- else:
- return ['', sep.join(args), '']
-
- return ['', self._env.get(name, ''), '']
-
- def _split_fields(self, wtree):
- def is_empty(split):
- return split==['', '', '']
-
- def split_positional(quoted):
- # Return a list of wtree split according positional parameters rules.
- # All remaining '@' groups are removed.
- assert quoted[0]=='"'
-
- splits = [[]]
- for part in quoted:
- if not isinstance(part, list) or part[0]!='@':
- splits[-1].append(part)
- else:
- # Empty or single argument list were dealt with already
- assert len(part)>3
- # First argument must join with the beginning part of the original word
- splits[-1].append(part[1])
- # Create double-quotes expressions for every argument after the first
- for arg in part[2:-1]:
- splits[-1].append('"')
- splits.append(['"', arg])
- return splits
-
- # At this point, all expansions but pathnames have occured. Only quoted
- # and positional sequences remain. Thus, all candidates for field splitting
- # are in the tree root, or are positional splits ('@') and lie in root
- # children.
- if not wtree or wtree[0] not in ('', '"'):
- # The whole token is quoted or empty, nothing to split
- return [wtree]
-
- if wtree[0]=='"':
- wtree = ['', wtree, '']
-
- result = [['', '']]
- for part in wtree[1:-1]:
- if isinstance(part, list):
- if part[0]=='"':
- splits = split_positional(part)
- if len(splits)<=1:
- result[-1] += [part, '']
- else:
- # Terminate the current split
- result[-1] += [splits[0], '']
- result += splits[1:-1]
- # Create a new split
- result += [['', splits[-1], '']]
- else:
- result[-1] += [part, '']
- else:
- splits = self._env.split_fields(part)
- if len(splits)<=1:
- # No split
- result[-1][-1] += part
- else:
- # Terminate the current resulting part and create a new one
- result[-1][-1] += splits[0]
- result[-1].append('')
- result += [['', r, ''] for r in splits[1:-1]]
- result += [['', splits[-1]]]
- result[-1].append('')
-
- # Leading and trailing empty groups come from leading/trailing blanks
- if result and is_empty(result[-1]):
- result[-1:] = []
- if result and is_empty(result[0]):
- result[:1] = []
- return result
-
- def _expand_pathname(self, wtree):
- """See [2.6.6 Pathname Expansion]."""
- if self._env.has_opt('-f'):
- return [wtree]
-
- # All expansions have been performed, only quoted sequences should remain
- # in the tree. Generate the pattern by folding the tree, escaping special
- # characters when appear quoted
- special_chars = '*?[]'
-
- def make_pattern(wtree):
- subpattern = []
- for part in wtree[1:-1]:
- if isinstance(part, list):
- part = make_pattern(part)
- elif wtree[0]!='':
- for c in part:
- # Meta-characters cannot be quoted
- if c in special_chars:
- raise GlobError()
- subpattern.append(part)
- return ''.join(subpattern)
-
- def pwd_glob(pattern):
- cwd = os.getcwd()
- os.chdir(self._env['PWD'])
- try:
- return glob.glob(pattern)
- finally:
- os.chdir(cwd)
-
- #TODO: check working directory issues here wrt relative patterns
- try:
- pattern = make_pattern(wtree)
- paths = pwd_glob(pattern)
- except GlobError:
- # BUG: Meta-characters were found in quoted sequences. The should
- # have been used literally but this is unsupported in current glob module.
- # Instead we consider the whole tree must be used literally and
- # therefore there is no point in globbing. This is wrong when meta
- # characters are mixed with quoted meta in the same pattern like:
- # < foo*"py*" >
- paths = []
-
- if not paths:
- return [wtree]
- return [['', path, ''] for path in paths]
-
- def _remove_quotes(self, wtree):
- """See [2.6.7 Quote Removal]."""
-
- def unquote(wtree):
- unquoted = []
- for part in wtree[1:-1]:
- if isinstance(part, list):
- part = unquote(part)
- unquoted.append(part)
- return ''.join(unquoted)
-
- return ['', unquote(wtree), '']
-
- def subshell(self, script=None, ast=None, redirs=None):
- """Execute the script or AST in a subshell, with inherited redirections
- if redirs is not None.
- """
- if redirs:
- sub_redirs = redirs
- else:
- sub_redirs = redirs.clone()
-
- subshell = None
- try:
- subshell = Interpreter(None, self._debugflags, self._env.clone(True),
- sub_redirs, opts=self._options)
- return subshell.execute_script(script, ast)
- finally:
- if not redirs: sub_redirs.close()
- if subshell: subshell.close()
-
- def subshell_output(self, script):
- """Execute the script in a subshell and return the captured output."""
- # Create temporary file to capture subshell output
- tmpfd, tmppath = tempfile.mkstemp()
- try:
- tmpfile = os.fdopen(tmpfd, 'wb')
- stdout = FileWrapper('w', tmpfile)
-
- redirs = Redirections(self._redirs.stdin().dup(),
- stdout,
- self._redirs.stderr().dup())
- try:
- status = self.subshell(script=script, redirs=redirs)
- finally:
- redirs.close()
- redirs = None
-
- # Extract subshell standard output
- tmpfile = open(tmppath, 'rb')
- try:
- output = tmpfile.read()
- return status, output.rstrip('\n')
- finally:
- tmpfile.close()
- finally:
- os.remove(tmppath)
-
- def _asynclist(self, cmd):
- args = (self._env.get_variables(), cmd)
- arg = encodeargs(args)
- assert len(args) < 30*1024
- cmd = ['pysh.bat', '--ast', '-c', arg]
- p = subprocess.Popen(cmd, cwd=self._env['PWD'])
- self._children[p.pid] = p
- self._env['!'] = p.pid
- return 0
-
- def wait(self, pids=None):
- if not pids:
- pids = self._children.keys()
-
- status = 127
- for pid in pids:
- if pid not in self._children:
- continue
- p = self._children.pop(pid)
- status = p.wait()
-
- return status
-
diff --git a/bitbake/lib/bb/pysh/lsprof.py b/bitbake/lib/bb/pysh/lsprof.py
deleted file mode 100644
index b1831c2..0000000
--- a/bitbake/lib/bb/pysh/lsprof.py
+++ /dev/null
@@ -1,116 +0,0 @@
-#! /usr/bin/env python
-
-import sys
-from _lsprof import Profiler, profiler_entry
-
-__all__ = ['profile', 'Stats']
-
-def profile(f, *args, **kwds):
- """XXX docstring"""
- p = Profiler()
- p.enable(subcalls=True, builtins=True)
- try:
- f(*args, **kwds)
- finally:
- p.disable()
- return Stats(p.getstats())
-
-
-class Stats(object):
- """XXX docstring"""
-
- def __init__(self, data):
- self.data = data
-
- def sort(self, crit="inlinetime"):
- """XXX docstring"""
- if crit not in profiler_entry.__dict__:
- raise ValueError("Can't sort by %s" % crit)
- self.data.sort(lambda b, a: cmp(getattr(a, crit),
- getattr(b, crit)))
- for e in self.data:
- if e.calls:
- e.calls.sort(lambda b, a: cmp(getattr(a, crit),
- getattr(b, crit)))
-
- def pprint(self, top=None, file=None, limit=None, climit=None):
- """XXX docstring"""
- if file is None:
- file = sys.stdout
- d = self.data
- if top is not None:
- d = d[:top]
- cols = "% 12s %12s %11.4f %11.4f %s\n"
- hcols = "% 12s %12s %12s %12s %s\n"
- cols2 = "+%12s %12s %11.4f %11.4f + %s\n"
- file.write(hcols % ("CallCount", "Recursive", "Total(ms)",
- "Inline(ms)", "module:lineno(function)"))
- count = 0
- for e in d:
- file.write(cols % (e.callcount, e.reccallcount, e.totaltime,
- e.inlinetime, label(e.code)))
- count += 1
- if limit is not None and count == limit:
- return
- ccount = 0
- if e.calls:
- for se in e.calls:
- file.write(cols % ("+%s" % se.callcount, se.reccallcount,
- se.totaltime, se.inlinetime,
- "+%s" % label(se.code)))
- count += 1
- ccount += 1
- if limit is not None and count == limit:
- return
- if climit is not None and ccount == climit:
- break
-
- def freeze(self):
- """Replace all references to code objects with string
- descriptions; this makes it possible to pickle the instance."""
-
- # this code is probably rather ickier than it needs to be!
- for i in range(len(self.data)):
- e = self.data[i]
- if not isinstance(e.code, str):
- self.data[i] = type(e)((label(e.code),) + e[1:])
- if e.calls:
- for j in range(len(e.calls)):
- se = e.calls[j]
- if not isinstance(se.code, str):
- e.calls[j] = type(se)((label(se.code),) + se[1:])
-
-_fn2mod = {}
-
-def label(code):
- if isinstance(code, str):
- return code
- try:
- mname = _fn2mod[code.co_filename]
- except KeyError:
- for k, v in sys.modules.items():
- if v is None:
- continue
- if not hasattr(v, '__file__'):
- continue
- if not isinstance(v.__file__, str):
- continue
- if v.__file__.startswith(code.co_filename):
- mname = _fn2mod[code.co_filename] = k
- break
- else:
- mname = _fn2mod[code.co_filename] = '<%s>'%code.co_filename
-
- return '%s:%d(%s)' % (mname, code.co_firstlineno, code.co_name)
-
-
-if __name__ == '__main__':
- import os
- sys.argv = sys.argv[1:]
- if not sys.argv:
- print >> sys.stderr, "usage: lsprof.py <script> <arguments...>"
- sys.exit(2)
- sys.path.insert(0, os.path.abspath(os.path.dirname(sys.argv[0])))
- stats = profile(execfile, sys.argv[0], globals(), locals())
- stats.sort()
- stats.pprint()
diff --git a/bitbake/lib/bb/pysh/pysh.py b/bitbake/lib/bb/pysh/pysh.py
deleted file mode 100644
index b4e6145..0000000
--- a/bitbake/lib/bb/pysh/pysh.py
+++ /dev/null
@@ -1,167 +0,0 @@
-# pysh.py - command processing for pysh.
-#
-# Copyright 2007 Patrick Mezard
-#
-# This software may be used and distributed according to the terms
-# of the GNU General Public License, incorporated herein by reference.
-
-import optparse
-import os
-import sys
-
-import interp
-
-SH_OPT = optparse.OptionParser(prog='pysh', usage="%prog [OPTIONS]", version='0.1')
-SH_OPT.add_option('-c', action='store_true', dest='command_string', default=None,
- help='A string that shall be interpreted by the shell as one or more commands')
-SH_OPT.add_option('--redirect-to', dest='redirect_to', default=None,
- help='Redirect script commands stdout and stderr to the specified file')
-# See utility_command in builtin.py about the reason for this flag.
-SH_OPT.add_option('--redirected', dest='redirected', action='store_true', default=False,
- help='Tell the interpreter that stdout and stderr are actually the same objects, which is really stdout')
-SH_OPT.add_option('--debug-parsing', action='store_true', dest='debug_parsing', default=False,
- help='Trace PLY execution')
-SH_OPT.add_option('--debug-tree', action='store_true', dest='debug_tree', default=False,
- help='Display the generated syntax tree.')
-SH_OPT.add_option('--debug-cmd', action='store_true', dest='debug_cmd', default=False,
- help='Trace command execution before parameters expansion and exit status.')
-SH_OPT.add_option('--debug-utility', action='store_true', dest='debug_utility', default=False,
- help='Trace utility calls, after parameters expansions')
-SH_OPT.add_option('--ast', action='store_true', dest='ast', default=False,
- help='Encoded commands to execute in a subprocess')
-SH_OPT.add_option('--profile', action='store_true', default=False,
- help='Profile pysh run')
-
-
-def split_args(args):
- # Separate shell arguments from command ones
- # Just stop at the first argument not starting with a dash. I know, this is completely broken,
- # it ignores files starting with a dash or may take option values for command file. This is not
- # supposed to happen for now
- command_index = len(args)
- for i,arg in enumerate(args):
- if not arg.startswith('-'):
- command_index = i
- break
-
- return args[:command_index], args[command_index:]
-
-
-def fixenv(env):
- path = env.get('PATH')
- if path is not None:
- parts = path.split(os.pathsep)
- # Remove Windows utilities from PATH, they are useless at best and
- # some of them (find) may be confused with other utilities.
- parts = [p for p in parts if 'system32' not in p.lower()]
- env['PATH'] = os.pathsep.join(parts)
- if env.get('HOME') is None:
- # Several utilities, including cvsps, cannot work without
- # a defined HOME directory.
- env['HOME'] = os.path.expanduser('~')
- return env
-
-def _sh(cwd, shargs, cmdargs, options, debugflags=None, env=None):
- if os.environ.get('PYSH_TEXT') != '1':
- import msvcrt
- for fp in (sys.stdin, sys.stdout, sys.stderr):
- msvcrt.setmode(fp.fileno(), os.O_BINARY)
-
- hgbin = os.environ.get('PYSH_HGTEXT') != '1'
-
- if debugflags is None:
- debugflags = []
- if options.debug_parsing: debugflags.append('debug-parsing')
- if options.debug_utility: debugflags.append('debug-utility')
- if options.debug_cmd: debugflags.append('debug-cmd')
- if options.debug_tree: debugflags.append('debug-tree')
-
- if env is None:
- env = fixenv(dict(os.environ))
- if cwd is None:
- cwd = os.getcwd()
-
- if not cmdargs:
- # Nothing to do
- return 0
-
- ast = None
- command_file = None
- if options.command_string:
- input = cmdargs[0]
- if not options.ast:
- input += '\n'
- else:
- args, input = interp.decodeargs(input), None
- env, ast = args
- cwd = env.get('PWD', cwd)
- else:
- command_file = cmdargs[0]
- arguments = cmdargs[1:]
-
- prefix = interp.resolve_shebang(command_file, ignoreshell=True)
- if prefix:
- input = ' '.join(prefix + [command_file] + arguments)
- else:
- # Read commands from file
- f = file(command_file)
- try:
- # Trailing newline to help the parser
- input = f.read() + '\n'
- finally:
- f.close()
-
- redirect = None
- try:
- if options.redirected:
- stdout = sys.stdout
- stderr = stdout
- elif options.redirect_to:
- redirect = open(options.redirect_to, 'wb')
- stdout = redirect
- stderr = redirect
- else:
- stdout = sys.stdout
- stderr = sys.stderr
-
- # TODO: set arguments to environment variables
- opts = interp.Options()
- opts.hgbinary = hgbin
- ip = interp.Interpreter(cwd, debugflags, stdout=stdout, stderr=stderr,
- opts=opts)
- try:
- # Export given environment in shell object
- for k,v in env.iteritems():
- ip.get_env().export(k,v)
- return ip.execute_script(input, ast, scriptpath=command_file)
- finally:
- ip.close()
- finally:
- if redirect is not None:
- redirect.close()
-
-def sh(cwd=None, args=None, debugflags=None, env=None):
- if args is None:
- args = sys.argv[1:]
- shargs, cmdargs = split_args(args)
- options, shargs = SH_OPT.parse_args(shargs)
-
- if options.profile:
- import lsprof
- p = lsprof.Profiler()
- p.enable(subcalls=True)
- try:
- return _sh(cwd, shargs, cmdargs, options, debugflags, env)
- finally:
- p.disable()
- stats = lsprof.Stats(p.getstats())
- stats.sort()
- stats.pprint(top=10, file=sys.stderr, climit=5)
- else:
- return _sh(cwd, shargs, cmdargs, options, debugflags, env)
-
-def main():
- sys.exit(sh())
-
-if __name__=='__main__':
- main()
diff --git a/bitbake/lib/bb/pysh/pyshlex.py b/bitbake/lib/bb/pysh/pyshlex.py
index fbf094b..a42c294 100644
--- a/bitbake/lib/bb/pysh/pyshlex.py
+++ b/bitbake/lib/bb/pysh/pyshlex.py
@@ -13,11 +13,6 @@
# PLY in pull mode. It was designed to work incrementally and it would not be
# that hard to enable pull mode.
import re
-try:
- s = set()
- del s
-except NameError:
- from Set import Set as set
from ply import lex
from bb.pysh.sherrors import *
diff --git a/bitbake/lib/bb/pysh/pyshyacc.py b/bitbake/lib/bb/pysh/pyshyacc.py
index ba4cefd..de565dc 100644
--- a/bitbake/lib/bb/pysh/pyshyacc.py
+++ b/bitbake/lib/bb/pysh/pyshyacc.py
@@ -636,13 +636,16 @@ def p_empty(p):
def p_error(p):
msg = []
w = msg.append
- w('%r\n' % p)
- w('followed by:\n')
- for i in range(5):
- n = yacc.token()
- if not n:
- break
- w(' %r\n' % n)
+ if p:
+ w('%r\n' % p)
+ w('followed by:\n')
+ for i in range(5):
+ n = yacc.token()
+ if not n:
+ break
+ w(' %r\n' % n)
+ else:
+ w('Unexpected EOF')
raise sherrors.ShellSyntaxError(''.join(msg))
# Build the parser
diff --git a/bitbake/lib/bb/pysh/sherrors.py b/bitbake/lib/bb/pysh/sherrors.py
index 49d0533..3fe8e47 100644
--- a/bitbake/lib/bb/pysh/sherrors.py
+++ b/bitbake/lib/bb/pysh/sherrors.py
@@ -13,29 +13,3 @@ class ShellError(Exception):
class ShellSyntaxError(ShellError):
pass
-
-class UtilityError(ShellError):
- """Raised upon utility syntax error (option or operand error)."""
- pass
-
-class ExpansionError(ShellError):
- pass
-
-class CommandNotFound(ShellError):
- """Specified command was not found."""
- pass
-
-class RedirectionError(ShellError):
- pass
-
-class VarAssignmentError(ShellError):
- """Variable assignment error."""
- pass
-
-class ExitSignal(ShellError):
- """Exit signal."""
- pass
-
-class ReturnSignal(ShellError):
- """Exit signal."""
- pass
diff --git a/bitbake/lib/bb/pysh/subprocess_fix.py b/bitbake/lib/bb/pysh/subprocess_fix.py
deleted file mode 100644
index 46eca22..0000000
--- a/bitbake/lib/bb/pysh/subprocess_fix.py
+++ /dev/null
@@ -1,77 +0,0 @@
-# subprocess - Subprocesses with accessible I/O streams
-#
-# For more information about this module, see PEP 324.
-#
-# This module should remain compatible with Python 2.2, see PEP 291.
-#
-# Copyright (c) 2003-2005 by Peter Astrand <astrand@lysator.liu.se>
-#
-# Licensed to PSF under a Contributor Agreement.
-# See http://www.python.org/2.4/license for licensing details.
-
-def list2cmdline(seq):
- """
- Translate a sequence of arguments into a command line
- string, using the same rules as the MS C runtime:
-
- 1) Arguments are delimited by white space, which is either a
- space or a tab.
-
- 2) A string surrounded by double quotation marks is
- interpreted as a single argument, regardless of white space
- contained within. A quoted string can be embedded in an
- argument.
-
- 3) A double quotation mark preceded by a backslash is
- interpreted as a literal double quotation mark.
-
- 4) Backslashes are interpreted literally, unless they
- immediately precede a double quotation mark.
-
- 5) If backslashes immediately precede a double quotation mark,
- every pair of backslashes is interpreted as a literal
- backslash. If the number of backslashes is odd, the last
- backslash escapes the next double quotation mark as
- described in rule 3.
- """
-
- # See
- # http://msdn.microsoft.com/library/en-us/vccelng/htm/progs_12.asp
- result = []
- needquote = False
- for arg in seq:
- bs_buf = []
-
- # Add a space to separate this argument from the others
- if result:
- result.append(' ')
-
- needquote = (" " in arg) or ("\t" in arg) or ("|" in arg) or arg == ""
- if needquote:
- result.append('"')
-
- for c in arg:
- if c == '\\':
- # Don't know if we need to double yet.
- bs_buf.append(c)
- elif c == '"':
- # Double backspaces.
- result.append('\\' * len(bs_buf)*2)
- bs_buf = []
- result.append('\\"')
- else:
- # Normal char
- if bs_buf:
- result.extend(bs_buf)
- bs_buf = []
- result.append(c)
-
- # Add remaining backspaces, if any.
- if bs_buf:
- result.extend(bs_buf)
-
- if needquote:
- result.extend(bs_buf)
- result.append('"')
-
- return ''.join(result)
diff --git a/bitbake/lib/bb/remotedata.py b/bitbake/lib/bb/remotedata.py
index 68ecffc..7391e1b 100644
--- a/bitbake/lib/bb/remotedata.py
+++ b/bitbake/lib/bb/remotedata.py
@@ -6,18 +6,8 @@ Provides support for using a datastore from the bitbake client
# Copyright (C) 2016 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import bb.data
diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
index 9ce06c4..1804943 100644
--- a/bitbake/lib/bb/runqueue.py
+++ b/bitbake/lib/bb/runqueue.py
@@ -1,6 +1,3 @@
-#!/usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
"""
BitBake 'RunQueue' implementation
@@ -9,18 +6,8 @@ Handles preparation and execution of a queue of tasks
# Copyright (C) 2006-2007 Richard Purdie
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import copy
import os
@@ -37,11 +24,13 @@ from bb import monitordisk
import subprocess
import pickle
from multiprocessing import Process
+import shlex
+import pprint
bblogger = logging.getLogger("BitBake")
logger = logging.getLogger("BitBake.RunQueue")
-__find_md5__ = re.compile( r'(?i)(?<![a-z0-9])[a-f0-9]{32}(?![a-z0-9])' )
+__find_sha256__ = re.compile( r'(?i)(?<![a-z0-9])[a-f0-9]{64}(?![a-z0-9])' )
def fn_from_tid(tid):
return tid.rsplit(":", 1)[0]
@@ -49,17 +38,22 @@ def fn_from_tid(tid):
def taskname_from_tid(tid):
return tid.rsplit(":", 1)[1]
+def mc_from_tid(tid):
+ if tid.startswith('mc:'):
+ return tid.split(':')[1]
+ return ""
+
def split_tid(tid):
(mc, fn, taskname, _) = split_tid_mcfn(tid)
return (mc, fn, taskname)
def split_tid_mcfn(tid):
- if tid.startswith('multiconfig:'):
+ if tid.startswith('mc:'):
elems = tid.split(':')
mc = elems[1]
fn = ":".join(elems[2:-1])
taskname = elems[-1]
- mcfn = "multiconfig:" + mc + ":" + fn
+ mcfn = "mc:" + mc + ":" + fn
else:
tid = tid.rsplit(":", 1)
mc = ""
@@ -71,9 +65,17 @@ def split_tid_mcfn(tid):
def build_tid(mc, fn, taskname):
if mc:
- return "multiconfig:" + mc + ":" + fn + ":" + taskname
+ return "mc:" + mc + ":" + fn + ":" + taskname
return fn + ":" + taskname
+# Index used to pair up potentially matching multiconfig tasks
+# We match on PN, taskname and hash being equal
+def pending_hash_index(tid, rqdata):
+ (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
+ pn = rqdata.dataCaches[mc].pkg_fn[taskfn]
+ h = rqdata.runtaskentries[tid].unihash
+ return pn + ":" + "taskname" + h
+
class RunQueueStats:
"""
Holds statistics on the tasks handled by the associated runQueue
@@ -109,8 +111,6 @@ class RunQueueStats:
# runQueue state machine
runQueuePrepare = 2
runQueueSceneInit = 3
-runQueueSceneRun = 4
-runQueueRunInit = 5
runQueueRunning = 6
runQueueFailed = 7
runQueueCleanUp = 8
@@ -133,7 +133,7 @@ class RunQueueScheduler(object):
self.prio_map = [self.rqdata.runtaskentries.keys()]
- self.buildable = []
+ self.buildable = set()
self.skip_maxthread = {}
self.stamps = {}
for tid in self.rqdata.runtaskentries:
@@ -148,8 +148,11 @@ class RunQueueScheduler(object):
"""
Return the id of the first task we find that is buildable
"""
- self.buildable = [x for x in self.buildable if x not in self.rq.runq_running]
- if not self.buildable:
+ buildable = set(self.buildable)
+ buildable.difference_update(self.rq.runq_running)
+ buildable.difference_update(self.rq.holdoff_tasks)
+ buildable.intersection_update(self.rq.tasks_covered | self.rq.tasks_notcovered)
+ if not buildable:
return None
# Filter out tasks that have a max number of threads that have been exceeded
@@ -165,8 +168,8 @@ class RunQueueScheduler(object):
else:
skip_buildable[rtaskname] = 1
- if len(self.buildable) == 1:
- tid = self.buildable[0]
+ if len(buildable) == 1:
+ tid = buildable.pop()
taskname = taskname_from_tid(tid)
if taskname in skip_buildable and skip_buildable[taskname] >= int(self.skip_maxthread[taskname]):
return None
@@ -181,7 +184,7 @@ class RunQueueScheduler(object):
best = None
bestprio = None
- for tid in self.buildable:
+ for tid in buildable:
taskname = taskname_from_tid(tid)
if taskname in skip_buildable and skip_buildable[taskname] >= int(self.skip_maxthread[taskname]):
continue
@@ -203,7 +206,12 @@ class RunQueueScheduler(object):
return self.next_buildable_task()
def newbuildable(self, task):
- self.buildable.append(task)
+ self.buildable.add(task)
+ # Once tasks are running we don't need to worry about them again
+ self.buildable.difference_update(self.rq.runq_running)
+
+ def removebuildable(self, task):
+ self.buildable.remove(task)
def describe_task(self, taskid):
result = 'ID %s' % taskid
@@ -346,6 +354,7 @@ class RunTaskEntry(object):
self.depends = set()
self.revdeps = set()
self.hash = None
+ self.unihash = None
self.task = None
self.weight = 1
@@ -385,6 +394,9 @@ class RunQueueData:
def get_task_hash(self, tid):
return self.runtaskentries[tid].hash
+ def get_task_unihash(self, tid):
+ return self.runtaskentries[tid].unihash
+
def get_user_idstring(self, tid, task_name_suffix = ""):
return tid + task_name_suffix
@@ -405,6 +417,9 @@ class RunQueueData:
explored_deps = {}
msgs = []
+ class TooManyLoops(Exception):
+ pass
+
def chain_reorder(chain):
"""
Reorder a dependency chain so the lowest task id is first
@@ -457,7 +472,7 @@ class RunQueueData:
msgs.append("\n")
if len(valid_chains) > 10:
msgs.append("Aborted dependency loops search after 10 matches.\n")
- return msgs
+ raise TooManyLoops
continue
scan = False
if revdep not in explored_deps:
@@ -476,8 +491,11 @@ class RunQueueData:
explored_deps[tid] = total_deps
- for task in tasks:
- find_chains(task, [])
+ try:
+ for task in tasks:
+ find_chains(task, [])
+ except TooManyLoops:
+ pass
return msgs
@@ -833,6 +851,20 @@ class RunQueueData:
for depend in depends:
mark_active(depend, depth+1)
+ def invalidate_task(tid, error_nostamp):
+ (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
+ taskdep = self.dataCaches[mc].task_deps[taskfn]
+ if fn + ":" + taskname not in taskData[mc].taskentries:
+ logger.warning("Task %s does not exist, invalidating this task will have no effect" % taskname)
+ if 'nostamp' in taskdep and taskname in taskdep['nostamp']:
+ if error_nostamp:
+ bb.fatal("Task %s is marked nostamp, cannot invalidate this task" % taskname)
+ else:
+ bb.debug(1, "Task %s is marked nostamp, cannot invalidate this task" % taskname)
+ else:
+ logger.verbose("Invalidate task %s, %s", taskname, fn)
+ bb.parse.siggen.invalidate_task(taskname, self.dataCaches[mc], taskfn)
+
self.target_tids = []
for (mc, target, task, fn) in self.targets:
@@ -901,6 +933,8 @@ class RunQueueData:
for tid in list(runall_tids):
mark_active(tid,1)
+ if self.cooker.configuration.force:
+ invalidate_task(tid, False)
for tid in list(self.runtaskentries.keys()):
if tid not in runq_build:
@@ -922,6 +956,8 @@ class RunQueueData:
for tid in list(runonly_tids):
mark_active(tid,1)
+ if self.cooker.configuration.force:
+ invalidate_task(tid, False)
for tid in list(self.runtaskentries.keys()):
if tid not in runq_build:
@@ -1098,20 +1134,6 @@ class RunQueueData:
continue
self.runq_setscene_tids.append(tid)
- def invalidate_task(tid, error_nostamp):
- (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
- taskdep = self.dataCaches[mc].task_deps[taskfn]
- if fn + ":" + taskname not in taskData[mc].taskentries:
- logger.warning("Task %s does not exist, invalidating this task will have no effect" % taskname)
- if 'nostamp' in taskdep and taskname in taskdep['nostamp']:
- if error_nostamp:
- bb.fatal("Task %s is marked nostamp, cannot invalidate this task" % taskname)
- else:
- bb.debug(1, "Task %s is marked nostamp, cannot invalidate this task" % taskname)
- else:
- logger.verbose("Invalidate task %s, %s", taskname, fn)
- bb.parse.siggen.invalidate_task(taskname, self.dataCaches[mc], taskfn)
-
self.init_progress_reporter.next_stage()
# Invalidate task if force mode active
@@ -1142,6 +1164,8 @@ class RunQueueData:
self.init_progress_reporter.next_stage()
+ bb.parse.siggen.set_setscene_tasks(self.runq_setscene_tids)
+
# Iterate over the task list and call into the siggen code
dealtwith = set()
todeal = set(self.runtaskentries)
@@ -1150,18 +1174,20 @@ class RunQueueData:
if len(self.runtaskentries[tid].depends - dealtwith) == 0:
dealtwith.add(tid)
todeal.remove(tid)
- procdep = []
- for dep in self.runtaskentries[tid].depends:
- procdep.append(fn_from_tid(dep) + "." + taskname_from_tid(dep))
- (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
- self.runtaskentries[tid].hash = bb.parse.siggen.get_taskhash(taskfn, taskname, procdep, self.dataCaches[mc])
- task = self.runtaskentries[tid].task
+ self.prepare_task_hash(tid)
bb.parse.siggen.writeout_file_checksum_cache()
#self.dump_data()
return len(self.runtaskentries)
+ def prepare_task_hash(self, tid):
+ procdep = []
+ for dep in self.runtaskentries[tid].depends:
+ procdep.append(dep)
+ self.runtaskentries[tid].hash = bb.parse.siggen.get_taskhash(tid, procdep, self.dataCaches[mc_from_tid(tid)])
+ self.runtaskentries[tid].unihash = bb.parse.siggen.get_unihash(tid)
+
def dump_data(self):
"""
Dump some debug information on the internal data structures
@@ -1187,7 +1213,6 @@ class RunQueue:
self.stamppolicy = cfgData.getVar("BB_STAMP_POLICY") or "perfile"
self.hashvalidate = cfgData.getVar("BB_HASHCHECK_FUNCTION") or None
- self.setsceneverify = cfgData.getVar("BB_SETSCENE_VERIFY_FUNCTION2") or None
self.depvalidate = cfgData.getVar("BB_SETSCENE_DEPVALID") or None
self.state = runQueuePrepare
@@ -1196,7 +1221,7 @@ class RunQueue:
# Invoked at regular time intervals via the bitbake heartbeat event
# while the build is running. We generate a unique name for the handler
# here, just in case that there ever is more than one RunQueue instance,
- # start the handler when reaching runQueueSceneRun, and stop it when
+ # start the handler when reaching runQueueSceneInit, and stop it when
# done with the build.
self.dm = monitordisk.diskMonitor(cfgData)
self.dm_event_handler_name = '_bb_diskmonitor_' + str(id(self))
@@ -1213,28 +1238,23 @@ class RunQueue:
if fakeroot:
magic = magic + "beef"
mcdata = self.cooker.databuilder.mcdata[mc]
- fakerootcmd = mcdata.getVar("FAKEROOTCMD")
+ fakerootcmd = shlex.split(mcdata.getVar("FAKEROOTCMD"))
fakerootenv = (mcdata.getVar("FAKEROOTBASEENV") or "").split()
env = os.environ.copy()
for key, value in (var.split('=') for var in fakerootenv):
env[key] = value
- worker = subprocess.Popen([fakerootcmd, "bitbake-worker", magic], stdout=subprocess.PIPE, stdin=subprocess.PIPE, env=env)
+ worker = subprocess.Popen(fakerootcmd + ["bitbake-worker", magic], stdout=subprocess.PIPE, stdin=subprocess.PIPE, env=env)
else:
worker = subprocess.Popen(["bitbake-worker", magic], stdout=subprocess.PIPE, stdin=subprocess.PIPE)
bb.utils.nonblockingfd(worker.stdout)
workerpipe = runQueuePipe(worker.stdout, None, self.cfgData, self, rqexec)
- runqhash = {}
- for tid in self.rqdata.runtaskentries:
- runqhash[tid] = self.rqdata.runtaskentries[tid].hash
-
workerdata = {
"taskdeps" : self.rqdata.dataCaches[mc].task_deps,
"fakerootenv" : self.rqdata.dataCaches[mc].fakerootenv,
"fakerootdirs" : self.rqdata.dataCaches[mc].fakerootdirs,
"fakerootnoenv" : self.rqdata.dataCaches[mc].fakerootnoenv,
"sigdata" : bb.parse.siggen.get_taskdata(),
- "runq_hash" : runqhash,
"logdefaultdebug" : bb.msg.loggerDefaultDebugLevel,
"logdefaultverbose" : bb.msg.loggerDefaultVerbose,
"logdefaultverboselogs" : bb.msg.loggerVerboseLogs,
@@ -1243,6 +1263,7 @@ class RunQueue:
"buildname" : self.cfgData.getVar("BUILDNAME"),
"date" : self.cfgData.getVar("DATE"),
"time" : self.cfgData.getVar("TIME"),
+ "hashservaddr" : self.cooker.hashservaddr,
}
worker.stdin.write(b"<cookerconfig>" + pickle.dumps(self.cooker.configuration) + b"</cookerconfig>")
@@ -1376,6 +1397,31 @@ class RunQueue:
cache[tid] = iscurrent
return iscurrent
+ def validate_hashes(self, tocheck, data, currentcount=0, siginfo=False):
+ valid = set()
+ if self.hashvalidate:
+ sq_data = {}
+ sq_data['hash'] = {}
+ sq_data['hashfn'] = {}
+ sq_data['unihash'] = {}
+ for tid in tocheck:
+ (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
+ sq_data['hash'][tid] = self.rqdata.runtaskentries[tid].hash
+ sq_data['hashfn'][tid] = self.rqdata.dataCaches[mc].hashfn[taskfn]
+ sq_data['unihash'][tid] = self.rqdata.runtaskentries[tid].unihash
+
+ valid = self.validate_hash(sq_data, data, siginfo, currentcount)
+
+ return valid
+
+ def validate_hash(self, sq_data, d, siginfo, currentcount):
+ locs = {"sq_data" : sq_data, "d" : d, "siginfo" : siginfo, "currentcount" : currentcount}
+
+ # Metadata has **kwargs so args can be added, sq_data can also gain new fields
+ call = self.hashvalidate + "(sq_data, d, siginfo=siginfo, currentcount=currentcount)"
+
+ return bb.utils.better_eval(call, locs)
+
def _execute_runqueue(self):
"""
Run the tasks in a queue prepared by rqdata.prepare()
@@ -1386,7 +1432,6 @@ class RunQueue:
retval = True
if self.state is runQueuePrepare:
- self.rqexe = RunQueueExecuteDummy(self)
# NOTE: if you add, remove or significantly refactor the stages of this
# process then you should recalculate the weightings here. This is quite
# easy to do - just change the next line temporarily to pass debug=True as
@@ -1400,15 +1445,23 @@ class RunQueue:
self.state = runQueueComplete
else:
self.state = runQueueSceneInit
- self.rqdata.init_progress_reporter.next_stage()
-
- # we are ready to run, emit dependency info to any UI or class which
- # needs it
- depgraph = self.cooker.buildDependTree(self, self.rqdata.taskData)
- self.rqdata.init_progress_reporter.next_stage()
- bb.event.fire(bb.event.DepTreeGenerated(depgraph), self.cooker.data)
+ bb.parse.siggen.save_unitaskhashes()
if self.state is runQueueSceneInit:
+ self.rqdata.init_progress_reporter.next_stage()
+
+ # we are ready to run, emit dependency info to any UI or class which
+ # needs it
+ depgraph = self.cooker.buildDependTree(self, self.rqdata.taskData)
+ self.rqdata.init_progress_reporter.next_stage()
+ bb.event.fire(bb.event.DepTreeGenerated(depgraph), self.cooker.data)
+
+ if not self.dm_event_handler_registered:
+ res = bb.event.register(self.dm_event_handler_name,
+ lambda x: self.dm.check(self) if self.state in [runQueueRunning, runQueueCleanUp] else False,
+ ('bb.event.HeartbeatEvent',))
+ self.dm_event_handler_registered = True
+
dump = self.cooker.configuration.dump_signatures
if dump:
self.rqdata.init_progress_reporter.finish()
@@ -1418,29 +1471,23 @@ class RunQueue:
if 'printdiff' in dump:
self.write_diffscenetasks(invalidtasks)
self.state = runQueueComplete
- else:
- self.rqdata.init_progress_reporter.next_stage()
- self.start_worker()
- self.rqdata.init_progress_reporter.next_stage()
- self.rqexe = RunQueueExecuteScenequeue(self)
-
- if self.state is runQueueSceneRun:
- if not self.dm_event_handler_registered:
- res = bb.event.register(self.dm_event_handler_name,
- lambda x: self.dm.check(self) if self.state in [runQueueSceneRun, runQueueRunning, runQueueCleanUp] else False,
- ('bb.event.HeartbeatEvent',))
- self.dm_event_handler_registered = True
- retval = self.rqexe.execute()
- if self.state is runQueueRunInit:
- if self.cooker.configuration.setsceneonly:
- self.state = runQueueComplete
- else:
- # Just in case we didn't setscene
- self.rqdata.init_progress_reporter.finish()
- logger.info("Executing RunQueue Tasks")
- self.rqexe = RunQueueExecuteTasks(self)
- self.state = runQueueRunning
+ if self.state is runQueueSceneInit:
+ self.rqdata.init_progress_reporter.next_stage()
+ self.start_worker()
+ self.rqdata.init_progress_reporter.next_stage()
+ self.rqexe = RunQueueExecute(self)
+
+ # If we don't have any setscene functions, skip execution
+ if len(self.rqdata.runq_setscene_tids) == 0:
+ logger.info('No setscene tasks')
+ for tid in self.rqdata.runtaskentries:
+ if len(self.rqdata.runtaskentries[tid].depends) == 0:
+ self.rqexe.setbuildable(tid)
+ self.rqexe.tasks_notcovered.add(tid)
+ self.rqexe.sqdone = True
+ logger.info('Executing Tasks')
+ self.state = runQueueRunning
if self.state is runQueueRunning:
retval = self.rqexe.execute()
@@ -1455,12 +1502,14 @@ class RunQueue:
self.dm_event_handler_registered = False
if build_done and self.rqexe:
+ bb.parse.siggen.save_unitaskhashes()
self.teardown_workers()
- if self.rqexe.stats.failed:
- logger.info("Tasks Summary: Attempted %d tasks of which %d didn't need to be rerun and %d failed.", self.rqexe.stats.completed + self.rqexe.stats.failed, self.rqexe.stats.skipped, self.rqexe.stats.failed)
- else:
- # Let's avoid the word "failed" if nothing actually did
- logger.info("Tasks Summary: Attempted %d tasks of which %d didn't need to be rerun and all succeeded.", self.rqexe.stats.completed, self.rqexe.stats.skipped)
+ if self.rqexe:
+ if self.rqexe.stats.failed:
+ logger.info("Tasks Summary: Attempted %d tasks of which %d didn't need to be rerun and %d failed.", self.rqexe.stats.completed + self.rqexe.stats.failed, self.rqexe.stats.skipped, self.rqexe.stats.failed)
+ else:
+ # Let's avoid the word "failed" if nothing actually did
+ logger.info("Tasks Summary: Attempted %d tasks of which %d didn't need to be rerun and all succeeded.", self.rqexe.stats.completed, self.rqexe.stats.skipped)
if self.state is runQueueFailed:
raise bb.runqueue.TaskFailure(self.rqexe.failed_tids)
@@ -1543,15 +1592,8 @@ class RunQueue:
def print_diffscenetasks(self):
- valid = []
- sq_hash = []
- sq_hashfn = []
- sq_fn = []
- sq_taskname = []
- sq_task = []
noexec = []
- stamppresent = []
- valid_new = set()
+ tocheck = set()
for tid in self.rqdata.runtaskentries:
(mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
@@ -1561,21 +1603,9 @@ class RunQueue:
noexec.append(tid)
continue
- sq_fn.append(fn)
- sq_hashfn.append(self.rqdata.dataCaches[mc].hashfn[taskfn])
- sq_hash.append(self.rqdata.runtaskentries[tid].hash)
- sq_taskname.append(taskname)
- sq_task.append(tid)
- locs = { "sq_fn" : sq_fn, "sq_task" : sq_taskname, "sq_hash" : sq_hash, "sq_hashfn" : sq_hashfn, "d" : self.cooker.data }
- try:
- call = self.hashvalidate + "(sq_fn, sq_task, sq_hash, sq_hashfn, d, siginfo=True)"
- valid = bb.utils.better_eval(call, locs)
- # Handle version with no siginfo parameter
- except TypeError:
- call = self.hashvalidate + "(sq_fn, sq_task, sq_hash, sq_hashfn, d)"
- valid = bb.utils.better_eval(call, locs)
- for v in valid:
- valid_new.add(sq_task[v])
+ tocheck.add(tid)
+
+ valid_new = self.validate_hashes(tocheck, self.cooker.data, 0, True)
# Tasks which are both setscene and noexec never care about dependencies
# We therefore find tasks which are setscene and noexec and mark their
@@ -1634,7 +1664,7 @@ class RunQueue:
recout = []
if len(hashfiles) == 2:
out2 = bb.siggen.compare_sigfiles(hashfiles[hash1], hashfiles[hash2], recursecb)
- recout.extend(list(' ' + l for l in out2))
+ recout.extend(list(' ' + l for l in out2))
else:
recout.append("Unable to find matching sigdata for %s with hashes %s or %s" % (key, hash1, hash2))
@@ -1655,10 +1685,11 @@ class RunQueue:
matches = {k : v for k, v in iter(matches.items()) if h not in k}
if matches:
latestmatch = sorted(matches.keys(), key=lambda f: matches[f])[-1]
- prevh = __find_md5__.search(latestmatch).group(0)
+ prevh = __find_sha256__.search(latestmatch).group(0)
output = bb.siggen.compare_sigfiles(latestmatch, match, recursecb)
bb.plain("\nTask %s:%s couldn't be used from the cache because:\n We need hash %s, closest matching task was %s\n " % (pn, taskname, h, prevh) + '\n '.join(output))
+
class RunQueueExecute:
def __init__(self, rq):
@@ -1670,6 +1701,13 @@ class RunQueueExecute:
self.number_tasks = int(self.cfgData.getVar("BB_NUMBER_THREADS") or 1)
self.scheduler = self.cfgData.getVar("BB_SCHEDULER") or "speed"
+ self.sq_buildable = set()
+ self.sq_running = set()
+ self.sq_live = set()
+
+ self.updated_taskhash_queue = []
+ self.pending_migrations = set()
+
self.runq_buildable = set()
self.runq_running = set()
self.runq_complete = set()
@@ -1677,9 +1715,17 @@ class RunQueueExecute:
self.build_stamps = {}
self.build_stamps2 = []
self.failed_tids = []
+ self.sq_deferred = {}
self.stampcache = {}
+ self.holdoff_tasks = set()
+ self.holdoff_need_update = True
+ self.sqdone = False
+
+ self.stats = RunQueueStats(len(self.rqdata.runtaskentries))
+ self.sq_stats = RunQueueStats(len(self.rqdata.runq_setscene_tids))
+
for mc in rq.worker:
rq.worker[mc].pipe.setrunqueueexec(self)
for mc in rq.fakeworker:
@@ -1688,6 +1734,34 @@ class RunQueueExecute:
if self.number_tasks <= 0:
bb.fatal("Invalid BB_NUMBER_THREADS %s" % self.number_tasks)
+ # List of setscene tasks which we've covered
+ self.scenequeue_covered = set()
+ # List of tasks which are covered (including setscene ones)
+ self.tasks_covered = set()
+ self.tasks_scenequeue_done = set()
+ self.scenequeue_notcovered = set()
+ self.tasks_notcovered = set()
+ self.scenequeue_notneeded = set()
+
+ # We can't skip specified target tasks which aren't setscene tasks
+ self.cantskip = set(self.rqdata.target_tids)
+ self.cantskip.difference_update(self.rqdata.runq_setscene_tids)
+ self.cantskip.intersection_update(self.rqdata.runtaskentries)
+
+ schedulers = self.get_schedulers()
+ for scheduler in schedulers:
+ if self.scheduler == scheduler.name:
+ self.sched = scheduler(self, self.rqdata)
+ logger.debug(1, "Using runqueue scheduler '%s'", scheduler.name)
+ break
+ else:
+ bb.fatal("Invalid scheduler '%s'. Available schedulers: %s" %
+ (self.scheduler, ", ".join(obj.name for obj in schedulers)))
+
+ #if len(self.rqdata.runq_setscene_tids) > 0:
+ self.sqdata = SQData()
+ build_scenequeue_data(self.sqdata, self.rqdata, self.rq, self.cooker, self.stampcache, self)
+
def runqueue_process_waitpid(self, task, status):
# self.build_stamps[pid] may not exist when use shared work directory.
@@ -1695,10 +1769,17 @@ class RunQueueExecute:
self.build_stamps2.remove(self.build_stamps[task])
del self.build_stamps[task]
- if status != 0:
- self.task_fail(task, status)
+ if task in self.sq_live:
+ if status != 0:
+ self.sq_task_fail(task, status)
+ else:
+ self.sq_task_complete(task)
+ self.sq_live.remove(task)
else:
- self.task_complete(task)
+ if status != 0:
+ self.task_fail(task, status)
+ else:
+ self.task_complete(task)
return True
def finish_now(self):
@@ -1727,8 +1808,9 @@ class RunQueueExecute:
def finish(self):
self.rq.state = runQueueCleanUp
- if self.stats.active > 0:
- bb.event.fire(runQueueExitWait(self.stats.active), self.cfgData)
+ active = self.stats.active + self.sq_stats.active
+ if active > 0:
+ bb.event.fire(runQueueExitWait(active), self.cfgData)
self.rq.read_workers()
return self.rq.active_fds()
@@ -1739,10 +1821,14 @@ class RunQueueExecute:
self.rq.state = runQueueComplete
return True
- def check_dependencies(self, task, taskdeps, setscene = False):
+ # Used by setscene only
+ def check_dependencies(self, task, taskdeps):
if not self.rq.depvalidate:
return False
+ # Must not edit parent data
+ taskdeps = set(taskdeps)
+
taskdata = {}
taskdeps.add(task)
for dep in taskdeps:
@@ -1755,121 +1841,10 @@ class RunQueueExecute:
return valid
def can_start_task(self):
- can_start = self.stats.active < self.number_tasks
+ active = self.stats.active + self.sq_stats.active
+ can_start = active < self.number_tasks
return can_start
-class RunQueueExecuteDummy(RunQueueExecute):
- def __init__(self, rq):
- self.rq = rq
- self.stats = RunQueueStats(0)
-
- def finish(self):
- self.rq.state = runQueueComplete
- return
-
-class RunQueueExecuteTasks(RunQueueExecute):
- def __init__(self, rq):
- RunQueueExecute.__init__(self, rq)
-
- self.stats = RunQueueStats(len(self.rqdata.runtaskentries))
-
- self.stampcache = {}
-
- initial_covered = self.rq.scenequeue_covered.copy()
-
- # Mark initial buildable tasks
- for tid in self.rqdata.runtaskentries:
- if len(self.rqdata.runtaskentries[tid].depends) == 0:
- self.runq_buildable.add(tid)
- if len(self.rqdata.runtaskentries[tid].revdeps) > 0 and self.rqdata.runtaskentries[tid].revdeps.issubset(self.rq.scenequeue_covered):
- self.rq.scenequeue_covered.add(tid)
-
- found = True
- while found:
- found = False
- for tid in self.rqdata.runtaskentries:
- if tid in self.rq.scenequeue_covered:
- continue
- logger.debug(1, 'Considering %s: %s' % (tid, str(self.rqdata.runtaskentries[tid].revdeps)))
-
- if len(self.rqdata.runtaskentries[tid].revdeps) > 0 and self.rqdata.runtaskentries[tid].revdeps.issubset(self.rq.scenequeue_covered):
- if tid in self.rq.scenequeue_notcovered:
- continue
- found = True
- self.rq.scenequeue_covered.add(tid)
-
- logger.debug(1, 'Skip list (pre setsceneverify) %s', sorted(self.rq.scenequeue_covered))
-
- # Allow the metadata to elect for setscene tasks to run anyway
- covered_remove = set()
- if self.rq.setsceneverify:
- invalidtasks = []
- tasknames = {}
- fns = {}
- for tid in self.rqdata.runtaskentries:
- (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
- taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn]
- fns[tid] = taskfn
- tasknames[tid] = taskname
- if 'noexec' in taskdep and taskname in taskdep['noexec']:
- continue
- if self.rq.check_stamp_task(tid, taskname + "_setscene", cache=self.stampcache):
- logger.debug(2, 'Setscene stamp current for task %s', tid)
- continue
- if self.rq.check_stamp_task(tid, taskname, recurse = True, cache=self.stampcache):
- logger.debug(2, 'Normal stamp current for task %s', tid)
- continue
- invalidtasks.append(tid)
-
- call = self.rq.setsceneverify + "(covered, tasknames, fns, d, invalidtasks=invalidtasks)"
- locs = { "covered" : self.rq.scenequeue_covered, "tasknames" : tasknames, "fns" : fns, "d" : self.cooker.data, "invalidtasks" : invalidtasks }
- covered_remove = bb.utils.better_eval(call, locs)
-
- def removecoveredtask(tid):
- (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
- taskname = taskname + '_setscene'
- bb.build.del_stamp(taskname, self.rqdata.dataCaches[mc], taskfn)
- self.rq.scenequeue_covered.remove(tid)
-
- toremove = covered_remove | self.rq.scenequeue_notcovered
- for task in toremove:
- logger.debug(1, 'Not skipping task %s due to setsceneverify', task)
- while toremove:
- covered_remove = []
- for task in toremove:
- if task in self.rq.scenequeue_covered:
- removecoveredtask(task)
- for deptask in self.rqdata.runtaskentries[task].depends:
- if deptask not in self.rq.scenequeue_covered:
- continue
- if deptask in toremove or deptask in covered_remove or deptask in initial_covered:
- continue
- logger.debug(1, 'Task %s depends on task %s so not skipping' % (task, deptask))
- covered_remove.append(deptask)
- toremove = covered_remove
-
- logger.debug(1, 'Full skip list %s', self.rq.scenequeue_covered)
-
-
- for mc in self.rqdata.dataCaches:
- target_pairs = []
- for tid in self.rqdata.target_tids:
- (tidmc, fn, taskname, _) = split_tid_mcfn(tid)
- if tidmc == mc:
- target_pairs.append((fn, taskname))
-
- event.fire(bb.event.StampUpdate(target_pairs, self.rqdata.dataCaches[mc].stamp), self.cfgData)
-
- schedulers = self.get_schedulers()
- for scheduler in schedulers:
- if self.scheduler == scheduler.name:
- self.sched = scheduler(self, self.rqdata)
- logger.debug(1, "Using runqueue scheduler '%s'", scheduler.name)
- break
- else:
- bb.fatal("Invalid scheduler '%s'. Available schedulers: %s" %
- (self.scheduler, ", ".join(obj.name for obj in schedulers)))
-
def get_schedulers(self):
schedulers = set(obj for obj in globals().values()
if type(obj) is type and
@@ -1941,67 +1916,172 @@ class RunQueueExecuteTasks(RunQueueExecute):
self.stats.taskSkipped()
self.stats.taskCompleted()
+ def summarise_scenequeue_errors(self):
+ err = False
+ if not self.sqdone:
+ logger.debug(1, 'We could skip tasks %s', "\n".join(sorted(self.scenequeue_covered)))
+ completeevent = sceneQueueComplete(self.sq_stats, self.rq)
+ bb.event.fire(completeevent, self.cfgData)
+ if self.sq_deferred:
+ logger.error("Scenequeue had deferred entries: %s" % pprint.pformat(self.sq_deferred))
+ err = True
+ if self.updated_taskhash_queue:
+ logger.error("Scenequeue had unprocessed changed taskhash entries: %s" % pprint.pformat(self.updated_taskhash_queue))
+ err = True
+ if self.holdoff_tasks:
+ logger.error("Scenequeue had holdoff tasks: %s" % pprint.pformat(self.holdoff_tasks))
+ err = True
+
+ for tid in self.rqdata.runq_setscene_tids:
+ if tid not in self.scenequeue_covered and tid not in self.scenequeue_notcovered:
+ err = True
+ logger.error("Setscene Task %s was never marked as covered or not covered" % tid)
+ if tid not in self.sq_buildable:
+ err = True
+ logger.error("Setscene Task %s was never marked as buildable" % tid)
+ if tid not in self.sq_running:
+ err = True
+ logger.error("Setscene Task %s was never marked as running" % tid)
+
+ for x in self.rqdata.runtaskentries:
+ if x not in self.tasks_covered and x not in self.tasks_notcovered:
+ logger.error("Task %s was never moved from the setscene queue" % x)
+ err = True
+ if x not in self.tasks_scenequeue_done:
+ logger.error("Task %s was never processed by the setscene code" % x)
+ err = True
+ if len(self.rqdata.runtaskentries[x].depends) == 0 and x not in self.runq_buildable:
+ logger.error("Task %s was never marked as buildable by the setscene code" % x)
+ err = True
+ return err
+
+
def execute(self):
"""
- Run the tasks in a queue prepared by rqdata.prepare()
+ Run the tasks in a queue prepared by prepare_runqueue
"""
- if self.rqdata.setscenewhitelist is not None and not self.rqdata.setscenewhitelist_checked:
- self.rqdata.setscenewhitelist_checked = True
-
- # Check tasks that are going to run against the whitelist
- def check_norun_task(tid, showerror=False):
- (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
- # Ignore covered tasks
- if tid in self.rq.scenequeue_covered:
- return False
- # Ignore stamped tasks
- if self.rq.check_stamp_task(tid, taskname, cache=self.stampcache):
- return False
- # Ignore noexec tasks
- taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn]
- if 'noexec' in taskdep and taskname in taskdep['noexec']:
- return False
+ self.rq.read_workers()
+ self.process_possible_migrations()
- pn = self.rqdata.dataCaches[mc].pkg_fn[taskfn]
- if not check_setscene_enforce_whitelist(pn, taskname, self.rqdata.setscenewhitelist):
- if showerror:
- if tid in self.rqdata.runq_setscene_tids:
- logger.error('Task %s.%s attempted to execute unexpectedly and should have been setscened' % (pn, taskname))
+ task = None
+ if not self.sqdone and self.can_start_task():
+ # Find the next setscene to run
+ for nexttask in sorted(self.rqdata.runq_setscene_tids):
+ if nexttask in self.sq_buildable and nexttask not in self.sq_running and self.sqdata.stamps[nexttask] not in self.build_stamps.values():
+ if nexttask not in self.sqdata.unskippable and len(self.sqdata.sq_revdeps[nexttask]) > 0 and self.sqdata.sq_revdeps[nexttask].issubset(self.scenequeue_covered) and self.check_dependencies(nexttask, self.sqdata.sq_revdeps[nexttask]):
+ if nexttask not in self.rqdata.target_tids:
+ logger.debug(2, "Skipping setscene for task %s" % nexttask)
+ self.sq_task_skip(nexttask)
+ self.scenequeue_notneeded.add(nexttask)
+ if nexttask in self.sq_deferred:
+ del self.sq_deferred[nexttask]
+ return True
+ # If covered tasks are running, need to wait for them to complete
+ for t in self.sqdata.sq_covered_tasks[nexttask]:
+ if t in self.runq_running and t not in self.runq_complete:
+ continue
+ if nexttask in self.sq_deferred:
+ if self.sq_deferred[nexttask] not in self.runq_complete:
+ continue
+ logger.debug(1, "Task %s no longer deferred" % nexttask)
+ del self.sq_deferred[nexttask]
+ valid = self.rq.validate_hashes(set([nexttask]), self.cooker.data, 0, False)
+ if not valid:
+ logger.debug(1, "%s didn't become valid, skipping setscene" % nexttask)
+ self.sq_task_failoutright(nexttask)
+ return True
else:
- logger.error('Task %s.%s attempted to execute unexpectedly' % (pn, taskname))
- return True
- return False
- # Look to see if any tasks that we think shouldn't run are going to
- unexpected = False
- for tid in self.rqdata.runtaskentries:
- if check_norun_task(tid):
- unexpected = True
+ self.sqdata.outrightfail.remove(nexttask)
+ if nexttask in self.sqdata.outrightfail:
+ logger.debug(2, 'No package found, so skipping setscene task %s', nexttask)
+ self.sq_task_failoutright(nexttask)
+ return True
+ if nexttask in self.sqdata.unskippable:
+ logger.debug(2, "Setscene task %s is unskippable" % nexttask)
+ task = nexttask
break
- if unexpected:
- # Run through the tasks in the rough order they'd have executed and print errors
- # (since the order can be useful - usually missing sstate for the last few tasks
- # is the cause of the problem)
- task = self.sched.next()
- while task is not None:
- check_norun_task(task, showerror=True)
- self.task_skip(task, 'Setscene enforcement check')
- task = self.sched.next()
+ if task is not None:
+ (mc, fn, taskname, taskfn) = split_tid_mcfn(task)
+ taskname = taskname + "_setscene"
+ if self.rq.check_stamp_task(task, taskname_from_tid(task), recurse = True, cache=self.stampcache):
+ logger.debug(2, 'Stamp for underlying task %s is current, so skipping setscene variant', task)
+ self.sq_task_failoutright(task)
+ return True
- self.rq.state = runQueueCleanUp
+ if self.cooker.configuration.force:
+ if task in self.rqdata.target_tids:
+ self.sq_task_failoutright(task)
+ return True
+
+ if self.rq.check_stamp_task(task, taskname, cache=self.stampcache):
+ logger.debug(2, 'Setscene stamp current task %s, so skip it and its dependencies', task)
+ self.sq_task_skip(task)
return True
- self.rq.read_workers()
+ if self.cooker.configuration.skipsetscene:
+ logger.debug(2, 'No setscene tasks should be executed. Skipping %s', task)
+ self.sq_task_failoutright(task)
+ return True
- if self.stats.total == 0:
- # nothing to do
- self.rq.state = runQueueCleanUp
+ startevent = sceneQueueTaskStarted(task, self.sq_stats, self.rq)
+ bb.event.fire(startevent, self.cfgData)
+
+ taskdepdata = self.sq_build_taskdepdata(task)
+
+ taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn]
+ taskhash = self.rqdata.get_task_hash(task)
+ unihash = self.rqdata.get_task_unihash(task)
+ if 'fakeroot' in taskdep and taskname in taskdep['fakeroot'] and not self.cooker.configuration.dry_run:
+ if not mc in self.rq.fakeworker:
+ self.rq.start_fakeworker(self, mc)
+ self.rq.fakeworker[mc].process.stdin.write(b"<runtask>" + pickle.dumps((taskfn, task, taskname, taskhash, unihash, True, self.cooker.collection.get_file_appends(taskfn), taskdepdata, False)) + b"</runtask>")
+ self.rq.fakeworker[mc].process.stdin.flush()
+ else:
+ self.rq.worker[mc].process.stdin.write(b"<runtask>" + pickle.dumps((taskfn, task, taskname, taskhash, unihash, True, self.cooker.collection.get_file_appends(taskfn), taskdepdata, False)) + b"</runtask>")
+ self.rq.worker[mc].process.stdin.flush()
- task = self.sched.next()
+ self.build_stamps[task] = bb.build.stampfile(taskname, self.rqdata.dataCaches[mc], taskfn, noextra=True)
+ self.build_stamps2.append(self.build_stamps[task])
+ self.sq_running.add(task)
+ self.sq_live.add(task)
+ self.sq_stats.taskActive()
+ if self.can_start_task():
+ return True
+
+ self.update_holdofftasks()
+
+ if not self.sq_live and not self.sqdone and not self.sq_deferred and not self.updated_taskhash_queue and not self.holdoff_tasks:
+ logger.info("Setscene tasks completed")
+
+ err = self.summarise_scenequeue_errors()
+ if err:
+ self.rq.state = runQueueFailed
+ return True
+
+ if self.cooker.configuration.setsceneonly:
+ self.rq.state = runQueueComplete
+ return True
+ self.sqdone = True
+
+ if self.stats.total == 0:
+ # nothing to do
+ self.rq.state = runQueueComplete
+ return True
+
+ if self.cooker.configuration.setsceneonly:
+ task = None
+ else:
+ task = self.sched.next()
if task is not None:
(mc, fn, taskname, taskfn) = split_tid_mcfn(task)
- if task in self.rq.scenequeue_covered:
+ if self.rqdata.setscenewhitelist is not None:
+ if self.check_setscenewhitelist(task):
+ self.task_fail(task, "setscene whitelist")
+ return True
+
+ if task in self.tasks_covered:
logger.debug(2, "Setscene covered task %s", task)
self.task_skip(task, "covered")
return True
@@ -2030,6 +2110,8 @@ class RunQueueExecuteTasks(RunQueueExecute):
taskdepdata = self.build_taskdepdata(task)
taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn]
+ taskhash = self.rqdata.get_task_hash(task)
+ unihash = self.rqdata.get_task_unihash(task)
if 'fakeroot' in taskdep and taskname in taskdep['fakeroot'] and not (self.cooker.configuration.dry_run or self.rqdata.setscene_enforce):
if not mc in self.rq.fakeworker:
try:
@@ -2039,10 +2121,10 @@ class RunQueueExecuteTasks(RunQueueExecute):
self.rq.state = runQueueFailed
self.stats.taskFailed()
return True
- self.rq.fakeworker[mc].process.stdin.write(b"<runtask>" + pickle.dumps((taskfn, task, taskname, False, self.cooker.collection.get_file_appends(taskfn), taskdepdata, self.rqdata.setscene_enforce)) + b"</runtask>")
+ self.rq.fakeworker[mc].process.stdin.write(b"<runtask>" + pickle.dumps((taskfn, task, taskname, taskhash, unihash, False, self.cooker.collection.get_file_appends(taskfn), taskdepdata, self.rqdata.setscene_enforce)) + b"</runtask>")
self.rq.fakeworker[mc].process.stdin.flush()
else:
- self.rq.worker[mc].process.stdin.write(b"<runtask>" + pickle.dumps((taskfn, task, taskname, False, self.cooker.collection.get_file_appends(taskfn), taskdepdata, self.rqdata.setscene_enforce)) + b"</runtask>")
+ self.rq.worker[mc].process.stdin.write(b"<runtask>" + pickle.dumps((taskfn, task, taskname, taskhash, unihash, False, self.cooker.collection.get_file_appends(taskfn), taskdepdata, self.rqdata.setscene_enforce)) + b"</runtask>")
self.rq.worker[mc].process.stdin.flush()
self.build_stamps[task] = bb.build.stampfile(taskname, self.rqdata.dataCaches[mc], taskfn, noextra=True)
@@ -2052,30 +2134,58 @@ class RunQueueExecuteTasks(RunQueueExecute):
if self.can_start_task():
return True
- if self.stats.active > 0:
+ if self.stats.active > 0 or self.sq_stats.active > 0:
self.rq.read_workers()
return self.rq.active_fds()
+ # No more tasks can be run. If we have deferred setscene tasks we should run them.
+ if self.sq_deferred:
+ tid = self.sq_deferred.pop(list(self.sq_deferred.keys())[0])
+ logger.warning("Runqeueue deadlocked on deferred tasks, forcing task %s" % tid)
+ self.sq_task_failoutright(tid)
+ return True
+
if len(self.failed_tids) != 0:
self.rq.state = runQueueFailed
return True
# Sanity Checks
+ err = self.summarise_scenequeue_errors()
for task in self.rqdata.runtaskentries:
if task not in self.runq_buildable:
logger.error("Task %s never buildable!", task)
- if task not in self.runq_running:
+ err = True
+ elif task not in self.runq_running:
logger.error("Task %s never ran!", task)
- if task not in self.runq_complete:
+ err = True
+ elif task not in self.runq_complete:
logger.error("Task %s never completed!", task)
- self.rq.state = runQueueComplete
+ err = True
+
+ if err:
+ self.rq.state = runQueueFailed
+ else:
+ self.rq.state = runQueueComplete
return True
+ def filtermcdeps(self, task, mc, deps):
+ ret = set()
+ for dep in deps:
+ thismc = mc_from_tid(dep)
+ if thismc != mc:
+ continue
+ ret.add(dep)
+ return ret
+
+ # We filter out multiconfig dependencies from taskdepdata we pass to the tasks
+ # as most code can't handle them
def build_taskdepdata(self, task):
taskdepdata = {}
- next = self.rqdata.runtaskentries[task].depends
+ mc = mc_from_tid(task)
+ next = self.rqdata.runtaskentries[task].depends.copy()
next.add(task)
+ next = self.filtermcdeps(task, mc, next)
while next:
additional = []
for revdep in next:
@@ -2084,7 +2194,9 @@ class RunQueueExecuteTasks(RunQueueExecute):
deps = self.rqdata.runtaskentries[revdep].depends
provides = self.rqdata.dataCaches[mc].fn_provides[taskfn]
taskhash = self.rqdata.runtaskentries[revdep].hash
- taskdepdata[revdep] = [pn, taskname, fn, deps, provides, taskhash]
+ unihash = self.rqdata.runtaskentries[revdep].unihash
+ deps = self.filtermcdeps(task, mc, deps)
+ taskdepdata[revdep] = [pn, taskname, fn, deps, provides, taskhash, unihash]
for revdep2 in deps:
if revdep2 not in taskdepdata:
additional.append(revdep2)
@@ -2093,269 +2205,197 @@ class RunQueueExecuteTasks(RunQueueExecute):
#bb.note("Task %s: " % task + str(taskdepdata).replace("], ", "],\n"))
return taskdepdata
-class RunQueueExecuteScenequeue(RunQueueExecute):
- def __init__(self, rq):
- RunQueueExecute.__init__(self, rq)
-
- self.scenequeue_covered = set()
- self.scenequeue_notcovered = set()
- self.scenequeue_notneeded = set()
+ def update_holdofftasks(self):
- # If we don't have any setscene functions, skip this step
- if len(self.rqdata.runq_setscene_tids) == 0:
- rq.scenequeue_covered = set()
- rq.scenequeue_notcovered = set()
- rq.state = runQueueRunInit
+ if not self.holdoff_need_update:
return
- self.stats = RunQueueStats(len(self.rqdata.runq_setscene_tids))
+ notcovered = set(self.scenequeue_notcovered)
+ notcovered |= self.cantskip
+ for tid in self.scenequeue_notcovered:
+ notcovered |= self.sqdata.sq_covered_tasks[tid]
+ notcovered |= self.sqdata.unskippable.difference(self.rqdata.runq_setscene_tids)
+ notcovered.intersection_update(self.tasks_scenequeue_done)
- sq_revdeps = {}
- sq_revdeps_new = {}
- sq_revdeps_squash = {}
- self.sq_harddeps = {}
- self.stamps = {}
-
- # We need to construct a dependency graph for the setscene functions. Intermediate
- # dependencies between the setscene tasks only complicate the code. This code
- # therefore aims to collapse the huge runqueue dependency tree into a smaller one
- # only containing the setscene functions.
-
- self.rqdata.init_progress_reporter.next_stage()
-
- # First process the chains up to the first setscene task.
- endpoints = {}
- for tid in self.rqdata.runtaskentries:
- sq_revdeps[tid] = copy.copy(self.rqdata.runtaskentries[tid].revdeps)
- sq_revdeps_new[tid] = set()
- if (len(sq_revdeps[tid]) == 0) and tid not in self.rqdata.runq_setscene_tids:
- #bb.warn("Added endpoint %s" % (tid))
- endpoints[tid] = set()
+ covered = set(self.scenequeue_covered)
+ for tid in self.scenequeue_covered:
+ covered |= self.sqdata.sq_covered_tasks[tid]
+ covered.difference_update(notcovered)
+ covered.intersection_update(self.tasks_scenequeue_done)
- self.rqdata.init_progress_reporter.next_stage()
+ for tid in notcovered | covered:
+ if len(self.rqdata.runtaskentries[tid].depends) == 0:
+ self.setbuildable(tid)
+ elif self.rqdata.runtaskentries[tid].depends.issubset(self.runq_complete):
+ self.setbuildable(tid)
- # Secondly process the chains between setscene tasks.
- for tid in self.rqdata.runq_setscene_tids:
- #bb.warn("Added endpoint 2 %s" % (tid))
- for dep in self.rqdata.runtaskentries[tid].depends:
- if tid in sq_revdeps[dep]:
- sq_revdeps[dep].remove(tid)
- if dep not in endpoints:
- endpoints[dep] = set()
- #bb.warn(" Added endpoint 3 %s" % (dep))
- endpoints[dep].add(tid)
-
- self.rqdata.init_progress_reporter.next_stage()
-
- def process_endpoints(endpoints):
- newendpoints = {}
- for point, task in endpoints.items():
- tasks = set()
- if task:
- tasks |= task
- if sq_revdeps_new[point]:
- tasks |= sq_revdeps_new[point]
- sq_revdeps_new[point] = set()
- if point in self.rqdata.runq_setscene_tids:
- sq_revdeps_new[point] = tasks
- tasks = set()
- continue
- for dep in self.rqdata.runtaskentries[point].depends:
- if point in sq_revdeps[dep]:
- sq_revdeps[dep].remove(point)
- if tasks:
- sq_revdeps_new[dep] |= tasks
- if len(sq_revdeps[dep]) == 0 and dep not in self.rqdata.runq_setscene_tids:
- newendpoints[dep] = task
- if len(newendpoints) != 0:
- process_endpoints(newendpoints)
-
- process_endpoints(endpoints)
-
- self.rqdata.init_progress_reporter.next_stage()
-
- # Build a list of setscene tasks which are "unskippable"
- # These are direct endpoints referenced by the build
- endpoints2 = {}
- sq_revdeps2 = {}
- sq_revdeps_new2 = {}
- def process_endpoints2(endpoints):
- newendpoints = {}
- for point, task in endpoints.items():
- tasks = set([point])
- if task:
- tasks |= task
- if sq_revdeps_new2[point]:
- tasks |= sq_revdeps_new2[point]
- sq_revdeps_new2[point] = set()
- if point in self.rqdata.runq_setscene_tids:
- sq_revdeps_new2[point] = tasks
- for dep in self.rqdata.runtaskentries[point].depends:
- if point in sq_revdeps2[dep]:
- sq_revdeps2[dep].remove(point)
- if tasks:
- sq_revdeps_new2[dep] |= tasks
- if (len(sq_revdeps2[dep]) == 0 or len(sq_revdeps_new2[dep]) != 0) and dep not in self.rqdata.runq_setscene_tids:
- newendpoints[dep] = tasks
- if len(newendpoints) != 0:
- process_endpoints2(newendpoints)
- for tid in self.rqdata.runtaskentries:
- sq_revdeps2[tid] = copy.copy(self.rqdata.runtaskentries[tid].revdeps)
- sq_revdeps_new2[tid] = set()
- if (len(sq_revdeps2[tid]) == 0) and tid not in self.rqdata.runq_setscene_tids:
- endpoints2[tid] = set()
- process_endpoints2(endpoints2)
- self.unskippable = []
- for tid in self.rqdata.runq_setscene_tids:
- if sq_revdeps_new2[tid]:
- self.unskippable.append(tid)
+ self.tasks_covered = covered
+ self.tasks_notcovered = notcovered
- self.rqdata.init_progress_reporter.next_stage(len(self.rqdata.runtaskentries))
+ self.holdoff_tasks = set()
- for taskcounter, tid in enumerate(self.rqdata.runtaskentries):
- if tid in self.rqdata.runq_setscene_tids:
- deps = set()
- for dep in sq_revdeps_new[tid]:
- deps.add(dep)
- sq_revdeps_squash[tid] = deps
- elif len(sq_revdeps_new[tid]) != 0:
- bb.msg.fatal("RunQueue", "Something went badly wrong during scenequeue generation, aborting. Please report this problem.")
- self.rqdata.init_progress_reporter.update(taskcounter)
-
- self.rqdata.init_progress_reporter.next_stage()
-
- # Resolve setscene inter-task dependencies
- # e.g. do_sometask_setscene[depends] = "targetname:do_someothertask_setscene"
- # Note that anything explicitly depended upon will have its reverse dependencies removed to avoid circular dependencies
for tid in self.rqdata.runq_setscene_tids:
- (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
- realtid = tid + "_setscene"
- idepends = self.rqdata.taskData[mc].taskentries[realtid].idepends
- self.stamps[tid] = bb.build.stampfile(taskname + "_setscene", self.rqdata.dataCaches[mc], taskfn, noextra=True)
- for (depname, idependtask) in idepends:
-
- if depname not in self.rqdata.taskData[mc].build_targets:
- continue
+ if tid not in self.scenequeue_covered and tid not in self.scenequeue_notcovered:
+ self.holdoff_tasks.add(tid)
- depfn = self.rqdata.taskData[mc].build_targets[depname][0]
- if depfn is None:
- continue
- deptid = depfn + ":" + idependtask.replace("_setscene", "")
- if deptid not in self.rqdata.runtaskentries:
- bb.msg.fatal("RunQueue", "Task %s depends upon non-existent task %s:%s" % (realtid, depfn, idependtask))
-
- if not deptid in self.sq_harddeps:
- self.sq_harddeps[deptid] = set()
- self.sq_harddeps[deptid].add(tid)
-
- sq_revdeps_squash[tid].add(deptid)
- # Have to zero this to avoid circular dependencies
- sq_revdeps_squash[deptid] = set()
-
- self.rqdata.init_progress_reporter.next_stage()
-
- for task in self.sq_harddeps:
- for dep in self.sq_harddeps[task]:
- sq_revdeps_squash[dep].add(task)
-
- self.rqdata.init_progress_reporter.next_stage()
-
- #for tid in sq_revdeps_squash:
- # for dep in sq_revdeps_squash[tid]:
- # data = data + "\n %s" % dep
- # bb.warn("Task %s_setscene: is %s " % (tid, data
-
- self.sq_deps = {}
- self.sq_revdeps = sq_revdeps_squash
- self.sq_revdeps2 = copy.deepcopy(self.sq_revdeps)
-
- for tid in self.sq_revdeps:
- self.sq_deps[tid] = set()
- for tid in self.sq_revdeps:
- for dep in self.sq_revdeps[tid]:
- self.sq_deps[dep].add(tid)
-
- self.rqdata.init_progress_reporter.next_stage()
-
- for tid in self.sq_revdeps:
- if len(self.sq_revdeps[tid]) == 0:
- self.runq_buildable.add(tid)
-
- self.rqdata.init_progress_reporter.finish()
-
- self.outrightfail = []
- if self.rq.hashvalidate:
- sq_hash = []
- sq_hashfn = []
- sq_fn = []
- sq_taskname = []
- sq_task = []
- noexec = []
- stamppresent = []
- for tid in self.sq_revdeps:
- (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
+ for tid in self.holdoff_tasks.copy():
+ for dep in self.sqdata.sq_covered_tasks[tid]:
+ if dep not in self.runq_complete:
+ self.holdoff_tasks.add(dep)
- taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn]
+ self.holdoff_need_update = False
- if 'noexec' in taskdep and taskname in taskdep['noexec']:
- noexec.append(tid)
- self.task_skip(tid)
- bb.build.make_stamp(taskname + "_setscene", self.rqdata.dataCaches[mc], taskfn)
- continue
+ def process_possible_migrations(self):
- if self.rq.check_stamp_task(tid, taskname + "_setscene", cache=self.stampcache):
- logger.debug(2, 'Setscene stamp current for task %s', tid)
- stamppresent.append(tid)
- self.task_skip(tid)
- continue
+ changed = set()
+ for tid, unihash in self.updated_taskhash_queue.copy():
+ if tid in self.runq_running and tid not in self.runq_complete:
+ continue
- if self.rq.check_stamp_task(tid, taskname, recurse = True, cache=self.stampcache):
- logger.debug(2, 'Normal stamp current for task %s', tid)
- stamppresent.append(tid)
- self.task_skip(tid)
- continue
+ self.updated_taskhash_queue.remove((tid, unihash))
+
+ if unihash != self.rqdata.runtaskentries[tid].unihash:
+ logger.info("Task %s unihash changed to %s" % (tid, unihash))
+ self.rqdata.runtaskentries[tid].unihash = unihash
+ bb.parse.siggen.set_unihash(tid, unihash)
+
+ # Work out all tasks which depend on this one
+ total = set()
+ next = set(self.rqdata.runtaskentries[tid].revdeps)
+ while next:
+ current = next.copy()
+ total = total |next
+ next = set()
+ for ntid in current:
+ next |= self.rqdata.runtaskentries[ntid].revdeps
+ next.difference_update(total)
+
+ # Now iterate those tasks in dependency order to regenerate their taskhash/unihash
+ done = set()
+ next = set(self.rqdata.runtaskentries[tid].revdeps)
+ while next:
+ current = next.copy()
+ next = set()
+ for tid in current:
+ if not self.rqdata.runtaskentries[tid].depends.isdisjoint(total):
+ continue
+ procdep = []
+ for dep in self.rqdata.runtaskentries[tid].depends:
+ procdep.append(dep)
+ orighash = self.rqdata.runtaskentries[tid].hash
+ self.rqdata.runtaskentries[tid].hash = bb.parse.siggen.get_taskhash(tid, procdep, self.rqdata.dataCaches[mc_from_tid(tid)])
+ origuni = self.rqdata.runtaskentries[tid].unihash
+ self.rqdata.runtaskentries[tid].unihash = bb.parse.siggen.get_unihash(tid)
+ logger.debug(1, "Task %s hash changes: %s->%s %s->%s" % (tid, orighash, self.rqdata.runtaskentries[tid].hash, origuni, self.rqdata.runtaskentries[tid].unihash))
+ next |= self.rqdata.runtaskentries[tid].revdeps
+ changed.add(tid)
+ total.remove(tid)
+ next.intersection_update(total)
+
+ if changed:
+ for mc in self.rq.worker:
+ self.rq.worker[mc].process.stdin.write(b"<newtaskhashes>" + pickle.dumps(bb.parse.siggen.get_taskhashes()) + b"</newtaskhashes>")
+ for mc in self.rq.fakeworker:
+ self.rq.fakeworker[mc].process.stdin.write(b"<newtaskhashes>" + pickle.dumps(bb.parse.siggen.get_taskhashes()) + b"</newtaskhashes>")
+
+ logger.debug(1, pprint.pformat("Tasks changed:\n%s" % (changed)))
+
+ for tid in changed:
+ if tid not in self.rqdata.runq_setscene_tids:
+ continue
+ if tid in self.runq_running:
+ continue
+ if tid in self.scenequeue_covered:
+ # Potentially risky, should we report this hash as a match?
+ logger.info("Already covered setscene for %s so ignoring rehash" % (tid))
+ continue
+ if tid not in self.pending_migrations:
+ self.pending_migrations.add(tid)
+
+ for tid in self.pending_migrations.copy():
+ valid = True
+ # Check no tasks this covers are running
+ for dep in self.sqdata.sq_covered_tasks[tid]:
+ if dep in self.runq_running and dep not in self.runq_complete:
+ logger.debug(2, "Task %s is running which blocks setscene for %s from running" % (dep, tid))
+ valid = False
+ break
+ if not valid:
+ continue
- sq_fn.append(fn)
- sq_hashfn.append(self.rqdata.dataCaches[mc].hashfn[taskfn])
- sq_hash.append(self.rqdata.runtaskentries[tid].hash)
- sq_taskname.append(taskname)
- sq_task.append(tid)
+ self.pending_migrations.remove(tid)
+ changed = True
- self.cooker.data.setVar("BB_SETSCENE_STAMPCURRENT_COUNT", len(stamppresent))
+ if tid in self.tasks_scenequeue_done:
+ self.tasks_scenequeue_done.remove(tid)
+ for dep in self.sqdata.sq_covered_tasks[tid]:
+ if dep not in self.runq_complete:
+ if dep in self.tasks_scenequeue_done and dep not in self.sqdata.unskippable:
+ self.tasks_scenequeue_done.remove(dep)
+
+ if tid in self.sq_buildable:
+ self.sq_buildable.remove(tid)
+ if tid in self.sq_running:
+ self.sq_running.remove(tid)
+ if self.sqdata.sq_revdeps[tid].issubset(self.scenequeue_covered | self.scenequeue_notcovered):
+ if tid not in self.sq_buildable:
+ self.sq_buildable.add(tid)
+ if len(self.sqdata.sq_revdeps[tid]) == 0:
+ self.sq_buildable.add(tid)
+
+ if tid in self.sqdata.outrightfail:
+ self.sqdata.outrightfail.remove(tid)
+ if tid in self.scenequeue_notcovered:
+ self.scenequeue_notcovered.remove(tid)
+ if tid in self.scenequeue_covered:
+ self.scenequeue_covered.remove(tid)
+ if tid in self.scenequeue_notneeded:
+ self.scenequeue_notneeded.remove(tid)
- call = self.rq.hashvalidate + "(sq_fn, sq_task, sq_hash, sq_hashfn, d)"
- locs = { "sq_fn" : sq_fn, "sq_task" : sq_taskname, "sq_hash" : sq_hash, "sq_hashfn" : sq_hashfn, "d" : self.cooker.data }
- valid = bb.utils.better_eval(call, locs)
+ (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
+ self.sqdata.stamps[tid] = bb.build.stampfile(taskname + "_setscene", self.rqdata.dataCaches[mc], taskfn, noextra=True)
- self.cooker.data.delVar("BB_SETSCENE_STAMPCURRENT_COUNT")
+ if tid in self.stampcache:
+ del self.stampcache[tid]
- valid_new = stamppresent
- for v in valid:
- valid_new.append(sq_task[v])
+ if tid in self.build_stamps:
+ del self.build_stamps[tid]
- for tid in self.sq_revdeps:
- if tid not in valid_new and tid not in noexec:
- logger.debug(2, 'No package found, so skipping setscene task %s', tid)
- self.outrightfail.append(tid)
+ logger.info("Setscene task %s now valid and being rerun" % tid)
+ self.sqdone = False
+ update_scenequeue_data([tid], self.sqdata, self.rqdata, self.rq, self.cooker, self.stampcache, self)
- logger.info('Executing SetScene Tasks')
+ if changed:
+ self.holdoff_need_update = True
- self.rq.state = runQueueSceneRun
+ def scenequeue_updatecounters(self, task, fail=False):
- def scenequeue_updatecounters(self, task, fail = False):
- for dep in self.sq_deps[task]:
- if fail and task in self.sq_harddeps and dep in self.sq_harddeps[task]:
+ for dep in sorted(self.sqdata.sq_deps[task]):
+ if fail and task in self.sqdata.sq_harddeps and dep in self.sqdata.sq_harddeps[task]:
logger.debug(2, "%s was unavailable and is a hard dependency of %s so skipping" % (task, dep))
- self.scenequeue_updatecounters(dep, fail)
+ self.sq_task_failoutright(dep)
continue
- if task not in self.sq_revdeps2[dep]:
- # May already have been removed by the fail case above
- continue
- self.sq_revdeps2[dep].remove(task)
- if len(self.sq_revdeps2[dep]) == 0:
- self.runq_buildable.add(dep)
+ if self.sqdata.sq_revdeps[dep].issubset(self.scenequeue_covered | self.scenequeue_notcovered):
+ if dep not in self.sq_buildable:
+ self.sq_buildable.add(dep)
- def task_completeoutright(self, task):
+ next = set([task])
+ while next:
+ new = set()
+ for t in sorted(next):
+ self.tasks_scenequeue_done.add(t)
+ # Look down the dependency chain for non-setscene things which this task depends on
+ # and mark as 'done'
+ for dep in self.rqdata.runtaskentries[t].depends:
+ if dep in self.rqdata.runq_setscene_tids or dep in self.tasks_scenequeue_done:
+ continue
+ if self.rqdata.runtaskentries[dep].revdeps.issubset(self.tasks_scenequeue_done):
+ new.add(dep)
+ next = new
+
+ self.holdoff_need_update = True
+
+ def sq_task_completeoutright(self, task):
"""
Mark a task as completed
Look at the reverse dependencies and mark any task with
@@ -2366,7 +2406,7 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
self.scenequeue_covered.add(task)
self.scenequeue_updatecounters(task)
- def check_taskfail(self, task):
+ def sq_check_taskfail(self, task):
if self.rqdata.setscenewhitelist is not None:
realtask = task.split('_setscene')[0]
(mc, fn, taskname, taskfn) = split_tid_mcfn(realtask)
@@ -2375,130 +2415,34 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
logger.error('Task %s.%s failed' % (pn, taskname + "_setscene"))
self.rq.state = runQueueCleanUp
- def task_complete(self, task):
- self.stats.taskCompleted()
- bb.event.fire(sceneQueueTaskCompleted(task, self.stats, self.rq), self.cfgData)
- self.task_completeoutright(task)
+ def sq_task_complete(self, task):
+ self.sq_stats.taskCompleted()
+ bb.event.fire(sceneQueueTaskCompleted(task, self.sq_stats, self.rq), self.cfgData)
+ self.sq_task_completeoutright(task)
- def task_fail(self, task, result):
- self.stats.taskFailed()
- bb.event.fire(sceneQueueTaskFailed(task, self.stats, result, self), self.cfgData)
+ def sq_task_fail(self, task, result):
+ self.sq_stats.taskFailed()
+ bb.event.fire(sceneQueueTaskFailed(task, self.sq_stats, result, self), self.cfgData)
self.scenequeue_notcovered.add(task)
self.scenequeue_updatecounters(task, True)
- self.check_taskfail(task)
+ self.sq_check_taskfail(task)
- def task_failoutright(self, task):
- self.runq_running.add(task)
- self.runq_buildable.add(task)
- self.stats.taskSkipped()
- self.stats.taskCompleted()
+ def sq_task_failoutright(self, task):
+ self.sq_running.add(task)
+ self.sq_buildable.add(task)
+ self.sq_stats.taskSkipped()
+ self.sq_stats.taskCompleted()
self.scenequeue_notcovered.add(task)
self.scenequeue_updatecounters(task, True)
- def task_skip(self, task):
- self.runq_running.add(task)
- self.runq_buildable.add(task)
- self.task_completeoutright(task)
- self.stats.taskSkipped()
- self.stats.taskCompleted()
-
- def execute(self):
- """
- Run the tasks in a queue prepared by prepare_runqueue
- """
-
- self.rq.read_workers()
-
- task = None
- if self.can_start_task():
- # Find the next setscene to run
- for nexttask in self.rqdata.runq_setscene_tids:
- if nexttask in self.runq_buildable and nexttask not in self.runq_running and self.stamps[nexttask] not in self.build_stamps.values():
- if nexttask in self.unskippable:
- logger.debug(2, "Setscene task %s is unskippable" % nexttask)
- if nexttask not in self.unskippable and len(self.sq_revdeps[nexttask]) > 0 and self.sq_revdeps[nexttask].issubset(self.scenequeue_covered) and self.check_dependencies(nexttask, self.sq_revdeps[nexttask], True):
- fn = fn_from_tid(nexttask)
- foundtarget = False
-
- if nexttask in self.rqdata.target_tids:
- foundtarget = True
- if not foundtarget:
- logger.debug(2, "Skipping setscene for task %s" % nexttask)
- self.task_skip(nexttask)
- self.scenequeue_notneeded.add(nexttask)
- return True
- if nexttask in self.outrightfail:
- self.task_failoutright(nexttask)
- return True
- task = nexttask
- break
- if task is not None:
- (mc, fn, taskname, taskfn) = split_tid_mcfn(task)
- taskname = taskname + "_setscene"
- if self.rq.check_stamp_task(task, taskname_from_tid(task), recurse = True, cache=self.stampcache):
- logger.debug(2, 'Stamp for underlying task %s is current, so skipping setscene variant', task)
- self.task_failoutright(task)
- return True
-
- if self.cooker.configuration.force:
- if task in self.rqdata.target_tids:
- self.task_failoutright(task)
- return True
-
- if self.rq.check_stamp_task(task, taskname, cache=self.stampcache):
- logger.debug(2, 'Setscene stamp current task %s, so skip it and its dependencies', task)
- self.task_skip(task)
- return True
-
- startevent = sceneQueueTaskStarted(task, self.stats, self.rq)
- bb.event.fire(startevent, self.cfgData)
+ def sq_task_skip(self, task):
+ self.sq_running.add(task)
+ self.sq_buildable.add(task)
+ self.sq_task_completeoutright(task)
+ self.sq_stats.taskSkipped()
+ self.sq_stats.taskCompleted()
- taskdepdata = self.build_taskdepdata(task)
-
- taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn]
- if 'fakeroot' in taskdep and taskname in taskdep['fakeroot'] and not self.cooker.configuration.dry_run:
- if not mc in self.rq.fakeworker:
- self.rq.start_fakeworker(self, mc)
- self.rq.fakeworker[mc].process.stdin.write(b"<runtask>" + pickle.dumps((taskfn, task, taskname, True, self.cooker.collection.get_file_appends(taskfn), taskdepdata, False)) + b"</runtask>")
- self.rq.fakeworker[mc].process.stdin.flush()
- else:
- self.rq.worker[mc].process.stdin.write(b"<runtask>" + pickle.dumps((taskfn, task, taskname, True, self.cooker.collection.get_file_appends(taskfn), taskdepdata, False)) + b"</runtask>")
- self.rq.worker[mc].process.stdin.flush()
-
- self.build_stamps[task] = bb.build.stampfile(taskname, self.rqdata.dataCaches[mc], taskfn, noextra=True)
- self.build_stamps2.append(self.build_stamps[task])
- self.runq_running.add(task)
- self.stats.taskActive()
- if self.can_start_task():
- return True
-
- if self.stats.active > 0:
- self.rq.read_workers()
- return self.rq.active_fds()
-
- #for tid in self.sq_revdeps:
- # if tid not in self.runq_running:
- # buildable = tid in self.runq_buildable
- # revdeps = self.sq_revdeps[tid]
- # bb.warn("Found we didn't run %s %s %s" % (tid, buildable, str(revdeps)))
-
- self.rq.scenequeue_covered = self.scenequeue_covered
- self.rq.scenequeue_notcovered = self.scenequeue_notcovered
-
- logger.debug(1, 'We can skip tasks %s', "\n".join(sorted(self.rq.scenequeue_covered)))
-
- self.rq.state = runQueueRunInit
-
- completeevent = sceneQueueComplete(self.stats, self.rq)
- bb.event.fire(completeevent, self.cfgData)
-
- return True
-
- def runqueue_process_waitpid(self, task, status):
- RunQueueExecute.runqueue_process_waitpid(self, task, status)
-
-
- def build_taskdepdata(self, task):
+ def sq_build_taskdepdata(self, task):
def getsetscenedeps(tid):
deps = set()
(mc, fn, taskname, _) = split_tid_mcfn(tid)
@@ -2526,7 +2470,8 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
deps = getsetscenedeps(revdep)
provides = self.rqdata.dataCaches[mc].fn_provides[taskfn]
taskhash = self.rqdata.runtaskentries[revdep].hash
- taskdepdata[revdep] = [pn, taskname, fn, deps, provides, taskhash]
+ unihash = self.rqdata.runtaskentries[revdep].unihash
+ taskdepdata[revdep] = [pn, taskname, fn, deps, provides, taskhash, unihash]
for revdep2 in deps:
if revdep2 not in taskdepdata:
additional.append(revdep2)
@@ -2535,6 +2480,279 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
#bb.note("Task %s: " % task + str(taskdepdata).replace("], ", "],\n"))
return taskdepdata
+ def check_setscenewhitelist(self, tid):
+ # Check task that is going to run against the whitelist
+ (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
+ # Ignore covered tasks
+ if tid in self.tasks_covered:
+ return False
+ # Ignore stamped tasks
+ if self.rq.check_stamp_task(tid, taskname, cache=self.stampcache):
+ return False
+ # Ignore noexec tasks
+ taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn]
+ if 'noexec' in taskdep and taskname in taskdep['noexec']:
+ return False
+
+ pn = self.rqdata.dataCaches[mc].pkg_fn[taskfn]
+ if not check_setscene_enforce_whitelist(pn, taskname, self.rqdata.setscenewhitelist):
+ if tid in self.rqdata.runq_setscene_tids:
+ msg = 'Task %s.%s attempted to execute unexpectedly and should have been setscened' % (pn, taskname)
+ else:
+ msg = 'Task %s.%s attempted to execute unexpectedly' % (pn, taskname)
+ logger.error(msg + '\nThis is usually due to missing setscene tasks. Those missing in this build were: %s' % pprint.pformat(self.scenequeue_notcovered))
+ return True
+ return False
+
+class SQData(object):
+ def __init__(self):
+ # SceneQueue dependencies
+ self.sq_deps = {}
+ # SceneQueue reverse dependencies
+ self.sq_revdeps = {}
+ # Injected inter-setscene task dependencies
+ self.sq_harddeps = {}
+ # Cache of stamp files so duplicates can't run in parallel
+ self.stamps = {}
+ # Setscene tasks directly depended upon by the build
+ self.unskippable = set()
+ # List of setscene tasks which aren't present
+ self.outrightfail = set()
+ # A list of normal tasks a setscene task covers
+ self.sq_covered_tasks = {}
+
+def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache, sqrq):
+
+ sq_revdeps = {}
+ sq_revdeps_squash = {}
+ sq_collated_deps = {}
+
+ # We need to construct a dependency graph for the setscene functions. Intermediate
+ # dependencies between the setscene tasks only complicate the code. This code
+ # therefore aims to collapse the huge runqueue dependency tree into a smaller one
+ # only containing the setscene functions.
+
+ rqdata.init_progress_reporter.next_stage()
+
+ # First process the chains up to the first setscene task.
+ endpoints = {}
+ for tid in rqdata.runtaskentries:
+ sq_revdeps[tid] = copy.copy(rqdata.runtaskentries[tid].revdeps)
+ sq_revdeps_squash[tid] = set()
+ if (len(sq_revdeps[tid]) == 0) and tid not in rqdata.runq_setscene_tids:
+ #bb.warn("Added endpoint %s" % (tid))
+ endpoints[tid] = set()
+
+ rqdata.init_progress_reporter.next_stage()
+
+ # Secondly process the chains between setscene tasks.
+ for tid in rqdata.runq_setscene_tids:
+ sq_collated_deps[tid] = set()
+ #bb.warn("Added endpoint 2 %s" % (tid))
+ for dep in rqdata.runtaskentries[tid].depends:
+ if tid in sq_revdeps[dep]:
+ sq_revdeps[dep].remove(tid)
+ if dep not in endpoints:
+ endpoints[dep] = set()
+ #bb.warn(" Added endpoint 3 %s" % (dep))
+ endpoints[dep].add(tid)
+
+ rqdata.init_progress_reporter.next_stage()
+
+ def process_endpoints(endpoints):
+ newendpoints = {}
+ for point, task in endpoints.items():
+ tasks = set()
+ if task:
+ tasks |= task
+ if sq_revdeps_squash[point]:
+ tasks |= sq_revdeps_squash[point]
+ if point not in rqdata.runq_setscene_tids:
+ for t in tasks:
+ sq_collated_deps[t].add(point)
+ sq_revdeps_squash[point] = set()
+ if point in rqdata.runq_setscene_tids:
+ sq_revdeps_squash[point] = tasks
+ tasks = set()
+ continue
+ for dep in rqdata.runtaskentries[point].depends:
+ if point in sq_revdeps[dep]:
+ sq_revdeps[dep].remove(point)
+ if tasks:
+ sq_revdeps_squash[dep] |= tasks
+ if len(sq_revdeps[dep]) == 0 and dep not in rqdata.runq_setscene_tids:
+ newendpoints[dep] = task
+ if len(newendpoints) != 0:
+ process_endpoints(newendpoints)
+
+ process_endpoints(endpoints)
+
+ rqdata.init_progress_reporter.next_stage()
+
+ # Build a list of tasks which are "unskippable"
+ # These are direct endpoints referenced by the build upto and including setscene tasks
+ # Take the build endpoints (no revdeps) and find the sstate tasks they depend upon
+ new = True
+ for tid in rqdata.runtaskentries:
+ if len(rqdata.runtaskentries[tid].revdeps) == 0:
+ sqdata.unskippable.add(tid)
+ sqdata.unskippable |= sqrq.cantskip
+ while new:
+ new = False
+ orig = sqdata.unskippable.copy()
+ for tid in sorted(orig, reverse=True):
+ if tid in rqdata.runq_setscene_tids:
+ continue
+ if len(rqdata.runtaskentries[tid].depends) == 0:
+ # These are tasks which have no setscene tasks in their chain, need to mark as directly buildable
+ sqrq.setbuildable(tid)
+ sqdata.unskippable |= rqdata.runtaskentries[tid].depends
+ if sqdata.unskippable != orig:
+ new = True
+
+ sqrq.tasks_scenequeue_done |= sqdata.unskippable.difference(rqdata.runq_setscene_tids)
+
+ rqdata.init_progress_reporter.next_stage(len(rqdata.runtaskentries))
+
+ # Sanity check all dependencies could be changed to setscene task references
+ for taskcounter, tid in enumerate(rqdata.runtaskentries):
+ if tid in rqdata.runq_setscene_tids:
+ pass
+ elif len(sq_revdeps_squash[tid]) != 0:
+ bb.msg.fatal("RunQueue", "Something went badly wrong during scenequeue generation, aborting. Please report this problem.")
+ else:
+ del sq_revdeps_squash[tid]
+ rqdata.init_progress_reporter.update(taskcounter)
+
+ rqdata.init_progress_reporter.next_stage()
+
+ # Resolve setscene inter-task dependencies
+ # e.g. do_sometask_setscene[depends] = "targetname:do_someothertask_setscene"
+ # Note that anything explicitly depended upon will have its reverse dependencies removed to avoid circular dependencies
+ for tid in rqdata.runq_setscene_tids:
+ (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
+ realtid = tid + "_setscene"
+ idepends = rqdata.taskData[mc].taskentries[realtid].idepends
+ sqdata.stamps[tid] = bb.build.stampfile(taskname + "_setscene", rqdata.dataCaches[mc], taskfn, noextra=True)
+ for (depname, idependtask) in idepends:
+
+ if depname not in rqdata.taskData[mc].build_targets:
+ continue
+
+ depfn = rqdata.taskData[mc].build_targets[depname][0]
+ if depfn is None:
+ continue
+ deptid = depfn + ":" + idependtask.replace("_setscene", "")
+ if deptid not in rqdata.runtaskentries:
+ bb.msg.fatal("RunQueue", "Task %s depends upon non-existent task %s:%s" % (realtid, depfn, idependtask))
+
+ if not deptid in sqdata.sq_harddeps:
+ sqdata.sq_harddeps[deptid] = set()
+ sqdata.sq_harddeps[deptid].add(tid)
+
+ sq_revdeps_squash[tid].add(deptid)
+ # Have to zero this to avoid circular dependencies
+ sq_revdeps_squash[deptid] = set()
+
+ rqdata.init_progress_reporter.next_stage()
+
+ for task in sqdata.sq_harddeps:
+ for dep in sqdata.sq_harddeps[task]:
+ sq_revdeps_squash[dep].add(task)
+
+ rqdata.init_progress_reporter.next_stage()
+
+ #for tid in sq_revdeps_squash:
+ # data = ""
+ # for dep in sq_revdeps_squash[tid]:
+ # data = data + "\n %s" % dep
+ # bb.warn("Task %s_setscene: is %s " % (tid, data))
+
+ sqdata.sq_revdeps = sq_revdeps_squash
+ sqdata.sq_covered_tasks = sq_collated_deps
+
+ # Build reverse version of revdeps to populate deps structure
+ for tid in sqdata.sq_revdeps:
+ sqdata.sq_deps[tid] = set()
+ for tid in sqdata.sq_revdeps:
+ for dep in sqdata.sq_revdeps[tid]:
+ sqdata.sq_deps[dep].add(tid)
+
+ rqdata.init_progress_reporter.next_stage()
+
+ sqdata.multiconfigs = set()
+ for tid in sqdata.sq_revdeps:
+ sqdata.multiconfigs.add(mc_from_tid(tid))
+ if len(sqdata.sq_revdeps[tid]) == 0:
+ sqrq.sq_buildable.add(tid)
+
+ rqdata.init_progress_reporter.finish()
+
+ sqdata.noexec = set()
+ sqdata.stamppresent = set()
+ sqdata.valid = set()
+
+ update_scenequeue_data(sqdata.sq_revdeps, sqdata, rqdata, rq, cooker, stampcache, sqrq)
+
+def update_scenequeue_data(tids, sqdata, rqdata, rq, cooker, stampcache, sqrq):
+
+ tocheck = set()
+
+ for tid in sorted(tids):
+ if tid in sqdata.stamppresent:
+ sqdata.stamppresent.remove(tid)
+ if tid in sqdata.valid:
+ sqdata.valid.remove(tid)
+
+ (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
+
+ taskdep = rqdata.dataCaches[mc].task_deps[taskfn]
+
+ if 'noexec' in taskdep and taskname in taskdep['noexec']:
+ sqdata.noexec.add(tid)
+ sqrq.sq_task_skip(tid)
+ bb.build.make_stamp(taskname + "_setscene", rqdata.dataCaches[mc], taskfn)
+ continue
+
+ if rq.check_stamp_task(tid, taskname + "_setscene", cache=stampcache):
+ logger.debug(2, 'Setscene stamp current for task %s', tid)
+ sqdata.stamppresent.add(tid)
+ sqrq.sq_task_skip(tid)
+ continue
+
+ if rq.check_stamp_task(tid, taskname, recurse = True, cache=stampcache):
+ logger.debug(2, 'Normal stamp current for task %s', tid)
+ sqdata.stamppresent.add(tid)
+ sqrq.sq_task_skip(tid)
+ continue
+
+ tocheck.add(tid)
+
+ sqdata.valid |= rq.validate_hashes(tocheck, cooker.data, len(sqdata.stamppresent), False)
+
+ sqdata.hashes = {}
+ for mc in sorted(sqdata.multiconfigs):
+ for tid in sorted(sqdata.sq_revdeps):
+ if mc_from_tid(tid) != mc:
+ continue
+ if tid in sqdata.stamppresent:
+ continue
+ if tid in sqdata.valid:
+ continue
+ if tid in sqdata.noexec:
+ continue
+ if tid in sqrq.scenequeue_notcovered:
+ continue
+ sqdata.outrightfail.add(tid)
+
+ h = pending_hash_index(tid, rqdata)
+ if h not in sqdata.hashes:
+ sqdata.hashes[h] = tid
+ else:
+ sqrq.sq_deferred[tid] = sqdata.hashes[h]
+ bb.warn("Deferring %s after %s" % (tid, sqdata.hashes[h]))
+
+
class TaskFailure(Exception):
"""
Exception raised when a task in a runqueue fails
@@ -2641,6 +2859,15 @@ class runQueueTaskSkipped(runQueueEvent):
runQueueEvent.__init__(self, task, stats, rq)
self.reason = reason
+class taskUniHashUpdate(bb.event.Event):
+ """
+ Base runQueue event class
+ """
+ def __init__(self, task, unihash):
+ self.taskid = task
+ self.unihash = unihash
+ bb.event.Event.__init__(self)
+
class runQueuePipe():
"""
Abstraction for a pipe between a worker thread and the server
@@ -2683,6 +2910,8 @@ class runQueuePipe():
except ValueError as e:
bb.msg.fatal("RunQueue", "failed load pickle '%s': '%s'" % (e, self.queue[7:index]))
bb.event.fire_from_worker(event, self.d)
+ if isinstance(event, taskUniHashUpdate):
+ self.rqexec.updated_taskhash_queue.append((event.taskid, event.unihash))
found = True
self.queue = self.queue[index+8:]
index = self.queue.find(b"</event>")
diff --git a/bitbake/lib/bb/server/__init__.py b/bitbake/lib/bb/server/__init__.py
index 5a3fba9..b6f7513 100644
--- a/bitbake/lib/bb/server/__init__.py
+++ b/bitbake/lib/bb/server/__init__.py
@@ -5,17 +5,5 @@
# Copyright (C) 2006 - 2008 Richard Purdie
# Copyright (C) 2013 Alexandru Damian
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
-
diff --git a/bitbake/lib/bb/server/process.py b/bitbake/lib/bb/server/process.py
index 4e0d9c2..69aae62 100644
--- a/bitbake/lib/bb/server/process.py
+++ b/bitbake/lib/bb/server/process.py
@@ -3,18 +3,8 @@
#
# Copyright (C) 2010 Bob Foerster <robert@erafx.com>
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
"""
This module implements a multiprocessing.Process based server for bitbake.
@@ -130,6 +120,7 @@ class ProcessServer(multiprocessing.Process):
bb.utils.set_process_name("Cooker")
ready = []
+ newconnections = []
self.controllersock = False
fds = [self.sock]
@@ -138,37 +129,48 @@ class ProcessServer(multiprocessing.Process):
print("Entering server connection loop")
def disconnect_client(self, fds):
- if not self.haveui:
- return
print("Disconnecting Client")
- fds.remove(self.controllersock)
- fds.remove(self.command_channel)
- bb.event.unregister_UIHhandler(self.event_handle, True)
- self.command_channel_reply.writer.close()
- self.event_writer.writer.close()
- del self.event_writer
- self.controllersock.close()
- self.controllersock = False
- self.haveui = False
- self.lastui = time.time()
- self.cooker.clientComplete()
- if self.timeout is None:
+ if self.controllersock:
+ fds.remove(self.controllersock)
+ self.controllersock.close()
+ self.controllersock = False
+ if self.haveui:
+ fds.remove(self.command_channel)
+ bb.event.unregister_UIHhandler(self.event_handle, True)
+ self.command_channel_reply.writer.close()
+ self.event_writer.writer.close()
+ self.command_channel.close()
+ self.command_channel = False
+ del self.event_writer
+ self.lastui = time.time()
+ self.cooker.clientComplete()
+ self.haveui = False
+ ready = select.select(fds,[],[],0)[0]
+ if newconnections:
+ print("Starting new client")
+ conn = newconnections.pop(-1)
+ fds.append(conn)
+ self.controllersock = conn
+ elif self.timeout is None and not ready:
print("No timeout, exiting.")
self.quit = True
while not self.quit:
if self.sock in ready:
- self.controllersock, address = self.sock.accept()
- if self.haveui:
- print("Dropping connection attempt as we have a UI %s" % (str(ready)))
- self.controllersock.close()
- else:
- print("Accepting %s" % (str(ready)))
- fds.append(self.controllersock)
+ while select.select([self.sock],[],[],0)[0]:
+ controllersock, address = self.sock.accept()
+ if self.controllersock:
+ print("Queuing %s (%s)" % (str(ready), str(newconnections)))
+ newconnections.append(controllersock)
+ else:
+ print("Accepting %s (%s)" % (str(ready), str(newconnections)))
+ self.controllersock = controllersock
+ fds.append(controllersock)
if self.controllersock in ready:
try:
- print("Connecting Client")
+ print("Processing Client")
ui_fds = recvfds(self.controllersock, 3)
+ print("Connecting Client")
# Where to write events to
writer = ConnectionWriter(ui_fds[0])
@@ -239,6 +241,12 @@ class ProcessServer(multiprocessing.Process):
while not lock:
with bb.utils.timeout(3):
lock = bb.utils.lockfile(lockfile, shared=False, retry=False, block=True)
+ if lock:
+ # We hold the lock so we can remove the file (hide stale pid data)
+ bb.utils.remove(lockfile)
+ bb.utils.unlockfile(lock)
+ return
+
if not lock:
# Some systems may not have lsof available
procs = None
@@ -259,10 +267,6 @@ class ProcessServer(multiprocessing.Process):
if procs:
msg += ":\n%s" % str(procs)
print(msg)
- return
- # We hold the lock so we can remove the file (hide stale pid data)
- bb.utils.remove(lockfile)
- bb.utils.unlockfile(lock)
def idle_commands(self, delay, fds=None):
nextsleep = delay
@@ -398,36 +402,45 @@ class BitBakeServer(object):
os.close(self.readypipein)
ready = ConnectionReader(self.readypipe)
- r = ready.poll(30)
+ r = ready.poll(5)
+ if not r:
+ bb.note("Bitbake server didn't start within 5 seconds, waiting for 90")
+ r = ready.poll(90)
if r:
try:
r = ready.get()
except EOFError:
# Trap the child exitting/closing the pipe and error out
r = None
- if not r or r != "ready":
+ if not r or r[0] != "r":
ready.close()
- bb.error("Unable to start bitbake server")
+ bb.error("Unable to start bitbake server (%s)" % str(r))
if os.path.exists(logfile):
logstart_re = re.compile(self.start_log_format % ('([0-9]+)', '([0-9-]+ [0-9:.]+)'))
started = False
lines = []
+ lastlines = []
with open(logfile, "r") as f:
for line in f:
if started:
lines.append(line)
else:
+ lastlines.append(line)
res = logstart_re.match(line.rstrip())
if res:
ldatetime = datetime.datetime.strptime(res.group(2), self.start_log_datetime_format)
if ldatetime >= startdatetime:
started = True
lines.append(line)
+ if len(lastlines) > 60:
+ lastlines = lastlines[-60:]
if lines:
- if len(lines) > 10:
- bb.error("Last 10 lines of server log for this session (%s):\n%s" % (logfile, "".join(lines[-10:])))
+ if len(lines) > 60:
+ bb.error("Last 60 lines of server log for this session (%s):\n%s" % (logfile, "".join(lines[-60:])))
else:
bb.error("Server log for this session (%s):\n%s" % (logfile, "".join(lines)))
+ elif lastlines:
+ bb.error("Server didn't start, last 60 loglines (%s):\n%s" % (logfile, "".join(lastlines)))
else:
bb.error("%s doesn't exist" % logfile)
@@ -437,17 +450,24 @@ class BitBakeServer(object):
def _startServer(self):
print(self.start_log_format % (os.getpid(), datetime.datetime.now().strftime(self.start_log_datetime_format)))
+ sys.stdout.flush()
+
server = ProcessServer(self.bitbake_lock, self.sock, self.sockname)
self.configuration.setServerRegIdleCallback(server.register_idle_function)
os.close(self.readypipe)
writer = ConnectionWriter(self.readypipein)
- self.cooker = bb.cooker.BBCooker(self.configuration, self.featureset)
- writer.send("ready")
+ try:
+ self.cooker = bb.cooker.BBCooker(self.configuration, self.featureset)
+ except bb.BBHandledException:
+ return None
+ writer.send("r")
writer.close()
server.cooker = self.cooker
server.server_timeout = self.configuration.server_timeout
server.xmlrpcinterface = self.configuration.xmlrpcinterface
print("Started bitbake server pid %d" % os.getpid())
+ sys.stdout.flush()
+
server.start()
def connectProcessServer(sockname, featureset):
@@ -459,10 +479,20 @@ def connectProcessServer(sockname, featureset):
readfd = writefd = readfd1 = writefd1 = readfd2 = writefd2 = None
eq = command_chan_recv = command_chan = None
+ sock.settimeout(10)
+
try:
try:
os.chdir(os.path.dirname(sockname))
- sock.connect(os.path.basename(sockname))
+ finished = False
+ while not finished:
+ try:
+ sock.connect(os.path.basename(sockname))
+ finished = True
+ except IOError as e:
+ if e.errno == errno.EWOULDBLOCK:
+ pass
+ raise
finally:
os.chdir(cwd)
@@ -493,7 +523,8 @@ def connectProcessServer(sockname, featureset):
command_chan.close()
for i in [writefd, readfd1, writefd2]:
try:
- os.close(i)
+ if i:
+ os.close(i)
except OSError:
pass
sock.close()
diff --git a/bitbake/lib/bb/server/xmlrpcclient.py b/bitbake/lib/bb/server/xmlrpcclient.py
index 4661a9e..c054c3c 100644
--- a/bitbake/lib/bb/server/xmlrpcclient.py
+++ b/bitbake/lib/bb/server/xmlrpcclient.py
@@ -4,18 +4,8 @@
# Copyright (C) 2006 - 2007 Michael 'Mickey' Lauer
# Copyright (C) 2006 - 2008 Richard Purdie
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import os
import sys
diff --git a/bitbake/lib/bb/server/xmlrpcserver.py b/bitbake/lib/bb/server/xmlrpcserver.py
index 875b128..54fa32f 100644
--- a/bitbake/lib/bb/server/xmlrpcserver.py
+++ b/bitbake/lib/bb/server/xmlrpcserver.py
@@ -4,18 +4,8 @@
# Copyright (C) 2006 - 2007 Michael 'Mickey' Lauer
# Copyright (C) 2006 - 2008 Richard Purdie
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import os
import sys
diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py
index fdbb2a3..a4bb1ff 100644
--- a/bitbake/lib/bb/siggen.py
+++ b/bitbake/lib/bb/siggen.py
@@ -1,3 +1,7 @@
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
import hashlib
import logging
import os
@@ -8,6 +12,8 @@ import bb.data
import difflib
import simplediff
from bb.checksum import FileChecksumCache
+from bb import runqueue
+import hashserv
logger = logging.getLogger('BitBake.SigGen')
@@ -37,12 +43,18 @@ class SignatureGenerator(object):
self.runtaskdeps = {}
self.file_checksum_values = {}
self.taints = {}
+ self.unitaskhashes = {}
+ self.setscenetasks = {}
def finalise(self, fn, d, varient):
return
- def get_taskhash(self, fn, task, deps, dataCache):
- return "0"
+ def get_unihash(self, tid):
+ return self.taskhash[tid]
+
+ def get_taskhash(self, tid, deps, dataCache):
+ self.taskhash[tid] = hashlib.sha256(tid.encode("utf-8")).hexdigest()
+ return self.taskhash[tid]
def writeout_file_checksum_cache(self):
"""Write/update the file checksum cache onto disk"""
@@ -64,14 +76,25 @@ class SignatureGenerator(object):
return
def get_taskdata(self):
- return (self.runtaskdeps, self.taskhash, self.file_checksum_values, self.taints, self.basehash)
+ return (self.runtaskdeps, self.taskhash, self.file_checksum_values, self.taints, self.basehash, self.unitaskhashes, self.setscenetasks)
def set_taskdata(self, data):
- self.runtaskdeps, self.taskhash, self.file_checksum_values, self.taints, self.basehash = data
+ self.runtaskdeps, self.taskhash, self.file_checksum_values, self.taints, self.basehash, self.unitaskhashes, self.setscenetasks = data
def reset(self, data):
self.__init__(data)
+ def get_taskhashes(self):
+ return self.taskhash, self.unitaskhashes
+
+ def set_taskhashes(self, hashes):
+ self.taskhash, self.unitaskhashes = hashes
+
+ def save_unitaskhashes(self):
+ return
+
+ def set_setscene_tasks(self, setscene_tasks):
+ return
class SignatureGeneratorBasic(SignatureGenerator):
"""
@@ -87,7 +110,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
self.taints = {}
self.gendeps = {}
self.lookupcache = {}
- self.pkgnameextract = re.compile("(?P<fn>.*)\..*")
+ self.setscenetasks = {}
self.basewhitelist = set((data.getVar("BB_HASHBASE_WHITELIST") or "").split())
self.taskwhitelist = None
self.init_rundepcheck(data)
@@ -98,6 +121,9 @@ class SignatureGeneratorBasic(SignatureGenerator):
else:
self.checksum_cache = None
+ self.unihash_cache = bb.cache.SimpleCache("1")
+ self.unitaskhashes = self.unihash_cache.init_cache(data, "bb_unihashes.dat", {})
+
def init_rundepcheck(self, data):
self.taskwhitelist = data.getVar("BB_HASHTASK_WHITELIST") or None
if self.taskwhitelist:
@@ -113,10 +139,16 @@ class SignatureGeneratorBasic(SignatureGenerator):
taskdeps, basehash = bb.data.generate_dependency_hash(tasklist, gendeps, lookupcache, self.basewhitelist, fn)
for task in tasklist:
- k = fn + "." + task
- if not ignore_mismatch and k in self.basehash and self.basehash[k] != basehash[k]:
- bb.error("When reparsing %s, the basehash value changed from %s to %s. The metadata is not deterministic and this needs to be fixed." % (k, self.basehash[k], basehash[k]))
- self.basehash[k] = basehash[k]
+ tid = fn + ":" + task
+ if not ignore_mismatch and tid in self.basehash and self.basehash[tid] != basehash[tid]:
+ bb.error("When reparsing %s, the basehash value changed from %s to %s. The metadata is not deterministic and this needs to be fixed." % (tid, self.basehash[tid], basehash[tid]))
+ bb.error("The following commands may help:")
+ cmd = "$ bitbake %s -c%s" % (d.getVar('PN'), task)
+ # Make sure sigdata is dumped before run printdiff
+ bb.error("%s -Snone" % cmd)
+ bb.error("Then:")
+ bb.error("%s -Sprintdiff\n" % cmd)
+ self.basehash[tid] = basehash[tid]
self.taskdeps[fn] = taskdeps
self.gendeps[fn] = gendeps
@@ -124,6 +156,9 @@ class SignatureGeneratorBasic(SignatureGenerator):
return taskdeps
+ def set_setscene_tasks(self, setscene_tasks):
+ self.setscenetasks = setscene_tasks
+
def finalise(self, fn, d, variant):
mc = d.getVar("__BBMULTICONFIG", False) or ""
@@ -143,7 +178,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
# self.dump_sigtask(fn, task, d.getVar("STAMP"), False)
for task in taskdeps:
- d.setVar("BB_BASEHASH_task-%s" % task, self.basehash[fn + "." + task])
+ d.setVar("BB_BASEHASH_task-%s" % task, self.basehash[fn + ":" + task])
def rundep_check(self, fn, recipename, task, dep, depname, dataCache):
# Return True if we should keep the dependency, False to drop it
@@ -163,31 +198,26 @@ class SignatureGeneratorBasic(SignatureGenerator):
pass
return taint
- def get_taskhash(self, fn, task, deps, dataCache):
+ def get_taskhash(self, tid, deps, dataCache):
- mc = ''
- if fn.startswith('multiconfig:'):
- mc = fn.split(':')[1]
- k = fn + "." + task
+ (mc, _, task, fn) = bb.runqueue.split_tid_mcfn(tid)
- data = dataCache.basetaskhash[k]
- self.basehash[k] = data
- self.runtaskdeps[k] = []
- self.file_checksum_values[k] = []
+ data = dataCache.basetaskhash[tid]
+ self.basehash[tid] = data
+ self.runtaskdeps[tid] = []
+ self.file_checksum_values[tid] = []
recipename = dataCache.pkg_fn[fn]
for dep in sorted(deps, key=clean_basepath):
- pkgname = self.pkgnameextract.search(dep).group('fn')
- if mc:
- depmc = pkgname.split(':')[1]
- if mc != depmc:
- continue
- depname = dataCache.pkg_fn[pkgname]
+ (depmc, _, deptaskname, depfn) = bb.runqueue.split_tid_mcfn(dep)
+ if mc != depmc:
+ continue
+ depname = dataCache.pkg_fn[depfn]
if not self.rundep_check(fn, recipename, task, dep, depname, dataCache):
continue
if dep not in self.taskhash:
bb.fatal("%s is not in taskhash, caller isn't calling in dependency order?" % dep)
- data = data + self.taskhash[dep]
- self.runtaskdeps[k].append(dep)
+ data = data + self.get_unihash(dep)
+ self.runtaskdeps[tid].append(dep)
if task in dataCache.file_checksums[fn]:
if self.checksum_cache:
@@ -195,7 +225,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
else:
checksums = bb.fetch2.get_file_checksums(dataCache.file_checksums[fn][task], recipename)
for (f,cs) in checksums:
- self.file_checksum_values[k].append((f,cs))
+ self.file_checksum_values[tid].append((f,cs))
if cs:
data = data + cs
@@ -205,16 +235,16 @@ class SignatureGeneratorBasic(SignatureGenerator):
import uuid
taint = str(uuid.uuid4())
data = data + taint
- self.taints[k] = "nostamp:" + taint
+ self.taints[tid] = "nostamp:" + taint
taint = self.read_taint(fn, task, dataCache.stamp[fn])
if taint:
data = data + taint
- self.taints[k] = taint
- logger.warning("%s is tainted from a forced run" % k)
+ self.taints[tid] = taint
+ logger.warning("%s is tainted from a forced run" % tid)
- h = hashlib.md5(data.encode("utf-8")).hexdigest()
- self.taskhash[k] = h
+ h = hashlib.sha256(data.encode("utf-8")).hexdigest()
+ self.taskhash[tid] = h
#d.setVar("BB_TASKHASH_task-%s" % task, taskhash[task])
return h
@@ -227,17 +257,20 @@ class SignatureGeneratorBasic(SignatureGenerator):
bb.fetch2.fetcher_parse_save()
bb.fetch2.fetcher_parse_done()
+ def save_unitaskhashes(self):
+ self.unihash_cache.save(self.unitaskhashes)
+
def dump_sigtask(self, fn, task, stampbase, runtime):
- k = fn + "." + task
+ tid = fn + ":" + task
referencestamp = stampbase
if isinstance(runtime, str) and runtime.startswith("customfile"):
sigfile = stampbase
referencestamp = runtime[11:]
- elif runtime and k in self.taskhash:
- sigfile = stampbase + "." + task + ".sigdata" + "." + self.taskhash[k]
+ elif runtime and tid in self.taskhash:
+ sigfile = stampbase + "." + task + ".sigdata" + "." + self.get_unihash(tid)
else:
- sigfile = stampbase + "." + task + ".sigbasedata" + "." + self.basehash[k]
+ sigfile = stampbase + "." + task + ".sigbasedata" + "." + self.basehash[tid]
bb.utils.mkdirhier(os.path.dirname(sigfile))
@@ -246,7 +279,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
data['basewhitelist'] = self.basewhitelist
data['taskwhitelist'] = self.taskwhitelist
data['taskdeps'] = self.taskdeps[fn][task]
- data['basehash'] = self.basehash[k]
+ data['basehash'] = self.basehash[tid]
data['gendeps'] = {}
data['varvals'] = {}
data['varvals'][task] = self.lookupcache[fn][task]
@@ -256,30 +289,31 @@ class SignatureGeneratorBasic(SignatureGenerator):
data['gendeps'][dep] = self.gendeps[fn][dep]
data['varvals'][dep] = self.lookupcache[fn][dep]
- if runtime and k in self.taskhash:
- data['runtaskdeps'] = self.runtaskdeps[k]
- data['file_checksum_values'] = [(os.path.basename(f), cs) for f,cs in self.file_checksum_values[k]]
+ if runtime and tid in self.taskhash:
+ data['runtaskdeps'] = self.runtaskdeps[tid]
+ data['file_checksum_values'] = [(os.path.basename(f), cs) for f,cs in self.file_checksum_values[tid]]
data['runtaskhashes'] = {}
for dep in data['runtaskdeps']:
- data['runtaskhashes'][dep] = self.taskhash[dep]
- data['taskhash'] = self.taskhash[k]
+ data['runtaskhashes'][dep] = self.get_unihash(dep)
+ data['taskhash'] = self.taskhash[tid]
+ data['unihash'] = self.get_unihash(tid)
taint = self.read_taint(fn, task, referencestamp)
if taint:
data['taint'] = taint
- if runtime and k in self.taints:
- if 'nostamp:' in self.taints[k]:
- data['taint'] = self.taints[k]
+ if runtime and tid in self.taints:
+ if 'nostamp:' in self.taints[tid]:
+ data['taint'] = self.taints[tid]
computed_basehash = calc_basehash(data)
- if computed_basehash != self.basehash[k]:
- bb.error("Basehash mismatch %s versus %s for %s" % (computed_basehash, self.basehash[k], k))
- if runtime and k in self.taskhash:
+ if computed_basehash != self.basehash[tid]:
+ bb.error("Basehash mismatch %s versus %s for %s" % (computed_basehash, self.basehash[tid], tid))
+ if runtime and tid in self.taskhash:
computed_taskhash = calc_taskhash(data)
- if computed_taskhash != self.taskhash[k]:
- bb.error("Taskhash mismatch %s versus %s for %s" % (computed_taskhash, self.taskhash[k], k))
- sigfile = sigfile.replace(self.taskhash[k], computed_taskhash)
+ if computed_taskhash != self.taskhash[tid]:
+ bb.error("Taskhash mismatch %s versus %s for %s" % (computed_taskhash, self.taskhash[tid], tid))
+ sigfile = sigfile.replace(self.taskhash[tid], computed_taskhash)
fd, tmpfile = tempfile.mkstemp(dir=os.path.dirname(sigfile), prefix="sigtask.")
try:
@@ -299,30 +333,34 @@ class SignatureGeneratorBasic(SignatureGenerator):
if fn in self.taskdeps:
for task in self.taskdeps[fn]:
tid = fn + ":" + task
- (mc, _, _) = bb.runqueue.split_tid(tid)
- k = fn + "." + task
- if k not in self.taskhash:
+ mc = bb.runqueue.mc_from_tid(tid)
+ if tid not in self.taskhash:
continue
- if dataCaches[mc].basetaskhash[k] != self.basehash[k]:
- bb.error("Bitbake's cached basehash does not match the one we just generated (%s)!" % k)
- bb.error("The mismatched hashes were %s and %s" % (dataCaches[mc].basetaskhash[k], self.basehash[k]))
+ if dataCaches[mc].basetaskhash[tid] != self.basehash[tid]:
+ bb.error("Bitbake's cached basehash does not match the one we just generated (%s)!" % tid)
+ bb.error("The mismatched hashes were %s and %s" % (dataCaches[mc].basetaskhash[tid], self.basehash[tid]))
self.dump_sigtask(fn, task, dataCaches[mc].stamp[fn], True)
class SignatureGeneratorBasicHash(SignatureGeneratorBasic):
name = "basichash"
+ def get_stampfile_hash(self, tid):
+ if tid in self.taskhash:
+ return self.taskhash[tid]
+
+ # If task is not in basehash, then error
+ return self.basehash[tid]
+
def stampfile(self, stampbase, fn, taskname, extrainfo, clean=False):
if taskname != "do_setscene" and taskname.endswith("_setscene"):
- k = fn + "." + taskname[:-9]
+ tid = fn + ":" + taskname[:-9]
else:
- k = fn + "." + taskname
+ tid = fn + ":" + taskname
if clean:
h = "*"
- elif k in self.taskhash:
- h = self.taskhash[k]
else:
- # If k is not in basehash, then error
- h = self.basehash[k]
+ h = self.get_stampfile_hash(tid)
+
return ("%s.%s.%s.%s" % (stampbase, taskname, h, extrainfo)).rstrip('.')
def stampcleanmask(self, stampbase, fn, taskname, extrainfo):
@@ -332,6 +370,172 @@ class SignatureGeneratorBasicHash(SignatureGeneratorBasic):
bb.note("Tainting hash to force rebuild of task %s, %s" % (fn, task))
bb.build.write_taint(task, d, fn)
+class SignatureGeneratorUniHashMixIn(object):
+ def get_taskdata(self):
+ return (self.server, self.method) + super().get_taskdata()
+
+ def set_taskdata(self, data):
+ self.server, self.method = data[:2]
+ super().set_taskdata(data[2:])
+
+ def client(self):
+ if getattr(self, '_client', None) is None:
+ self._client = hashserv.create_client(self.server)
+ return self._client
+
+ def __get_task_unihash_key(self, tid):
+ # TODO: The key only *needs* to be the taskhash, the tid is just
+ # convenient
+ return '%s:%s' % (tid.rsplit("/", 1)[1], self.taskhash[tid])
+
+ def get_stampfile_hash(self, tid):
+ if tid in self.taskhash:
+ # If a unique hash is reported, use it as the stampfile hash. This
+ # ensures that if a task won't be re-run if the taskhash changes,
+ # but it would result in the same output hash
+ unihash = self.unitaskhashes.get(self.__get_task_unihash_key(tid), None)
+ if unihash is not None:
+ return unihash
+
+ return super().get_stampfile_hash(tid)
+
+ def set_unihash(self, tid, unihash):
+ self.unitaskhashes[self.__get_task_unihash_key(tid)] = unihash
+
+ def get_unihash(self, tid):
+ taskhash = self.taskhash[tid]
+
+ # If its not a setscene task we can return
+ if self.setscenetasks and tid not in self.setscenetasks:
+ return taskhash
+
+ key = self.__get_task_unihash_key(tid)
+
+ # TODO: This cache can grow unbounded. It probably only needs to keep
+ # for each task
+ unihash = self.unitaskhashes.get(key, None)
+ if unihash is not None:
+ return unihash
+
+ # In the absence of being able to discover a unique hash from the
+ # server, make it be equivalent to the taskhash. The unique "hash" only
+ # really needs to be a unique string (not even necessarily a hash), but
+ # making it match the taskhash has a few advantages:
+ #
+ # 1) All of the sstate code that assumes hashes can be the same
+ # 2) It provides maximal compatibility with builders that don't use
+ # an equivalency server
+ # 3) The value is easy for multiple independent builders to derive the
+ # same unique hash from the same input. This means that if the
+ # independent builders find the same taskhash, but it isn't reported
+ # to the server, there is a better chance that they will agree on
+ # the unique hash.
+ unihash = taskhash
+
+ try:
+ data = self.client().get_unihash(self.method, self.taskhash[tid])
+ if data:
+ unihash = data
+ # A unique hash equal to the taskhash is not very interesting,
+ # so it is reported it at debug level 2. If they differ, that
+ # is much more interesting, so it is reported at debug level 1
+ bb.debug((1, 2)[unihash == taskhash], 'Found unihash %s in place of %s for %s from %s' % (unihash, taskhash, tid, self.server))
+ else:
+ bb.debug(2, 'No reported unihash for %s:%s from %s' % (tid, taskhash, self.server))
+ except hashserv.client.HashConnectionError as e:
+ bb.warn('Error contacting Hash Equivalence Server %s: %s' % (self.server, str(e)))
+
+ self.unitaskhashes[key] = unihash
+ return unihash
+
+ def report_unihash(self, path, task, d):
+ import importlib
+
+ taskhash = d.getVar('BB_TASKHASH')
+ unihash = d.getVar('BB_UNIHASH')
+ report_taskdata = d.getVar('SSTATE_HASHEQUIV_REPORT_TASKDATA') == '1'
+ tempdir = d.getVar('T')
+ fn = d.getVar('BB_FILENAME')
+ tid = fn + ':do_' + task
+ key = tid.rsplit("/", 1)[1] + ':' + taskhash
+
+ if self.setscenetasks and tid not in self.setscenetasks:
+ return
+
+ # Sanity checks
+ cache_unihash = self.unitaskhashes.get(key, None)
+ if cache_unihash is None:
+ bb.fatal('%s not in unihash cache. Please report this error' % key)
+
+ if cache_unihash != unihash:
+ bb.fatal("Cache unihash %s doesn't match BB_UNIHASH %s" % (cache_unihash, unihash))
+
+ sigfile = None
+ sigfile_name = "depsig.do_%s.%d" % (task, os.getpid())
+ sigfile_link = "depsig.do_%s" % task
+
+ try:
+ sigfile = open(os.path.join(tempdir, sigfile_name), 'w+b')
+
+ locs = {'path': path, 'sigfile': sigfile, 'task': task, 'd': d}
+
+ if "." in self.method:
+ (module, method) = self.method.rsplit('.', 1)
+ locs['method'] = getattr(importlib.import_module(module), method)
+ outhash = bb.utils.better_eval('method(path, sigfile, task, d)', locs)
+ else:
+ outhash = bb.utils.better_eval(self.method + '(path, sigfile, task, d)', locs)
+
+ try:
+ extra_data = {}
+
+ owner = d.getVar('SSTATE_HASHEQUIV_OWNER')
+ if owner:
+ extra_data['owner'] = owner
+
+ if report_taskdata:
+ sigfile.seek(0)
+
+ extra_data['PN'] = d.getVar('PN')
+ extra_data['PV'] = d.getVar('PV')
+ extra_data['PR'] = d.getVar('PR')
+ extra_data['task'] = task
+ extra_data['outhash_siginfo'] = sigfile.read().decode('utf-8')
+
+ data = self.client().report_unihash(taskhash, self.method, outhash, unihash, extra_data)
+ new_unihash = data['unihash']
+
+ if new_unihash != unihash:
+ bb.debug(1, 'Task %s unihash changed %s -> %s by server %s' % (taskhash, unihash, new_unihash, self.server))
+ bb.event.fire(bb.runqueue.taskUniHashUpdate(fn + ':do_' + task, new_unihash), d)
+ else:
+ bb.debug(1, 'Reported task %s as unihash %s to %s' % (taskhash, unihash, self.server))
+ except hashserv.client.HashConnectionError as e:
+ bb.warn('Error contacting Hash Equivalence Server %s: %s' % (self.server, str(e)))
+ finally:
+ if sigfile:
+ sigfile.close()
+
+ sigfile_link_path = os.path.join(tempdir, sigfile_link)
+ bb.utils.remove(sigfile_link_path)
+
+ try:
+ os.symlink(sigfile_name, sigfile_link_path)
+ except OSError:
+ pass
+
+
+#
+# Dummy class used for bitbake-selftest
+#
+class SignatureGeneratorTestEquivHash(SignatureGeneratorUniHashMixIn, SignatureGeneratorBasicHash):
+ name = "TestEquivHash"
+ def init_rundepcheck(self, data):
+ super().init_rundepcheck(data)
+ self.server = data.getVar('BB_HASHSERVE')
+ self.method = "sstate_output_hash"
+
+
def dump_this_task(outfile, d):
import bb.parse
fn = d.getVar("BB_FILENAME")
@@ -392,13 +596,13 @@ def list_inline_diff(oldlist, newlist, colors=None):
def clean_basepath(a):
mc = None
- if a.startswith("multiconfig:"):
+ if a.startswith("mc:"):
_, mc, a = a.split(":", 2)
b = a.rsplit("/", 2)[1] + '/' + a.rsplit("/", 2)[2]
if a.startswith("virtual:"):
b = b + ":" + a.rsplit(":", 1)[0]
if mc:
- b = b + ":multiconfig:" + mc
+ b = b + ":mc:" + mc
return b
def clean_basepaths(a):
@@ -623,6 +827,10 @@ def compare_sigfiles(a, b, recursecb=None, color=False, collapsed=False):
a_taint = a_data.get('taint', None)
b_taint = b_data.get('taint', None)
if a_taint != b_taint:
+ if a_taint and a_taint.startswith('nostamp:'):
+ a_taint = a_taint.replace('nostamp:', 'nostamp(uuid4):')
+ if b_taint and b_taint.startswith('nostamp:'):
+ b_taint = b_taint.replace('nostamp:', 'nostamp(uuid4):')
output.append(color_format("{color_title}Taint (by forced/invalidated task) changed{color_default} from %s to %s") % (a_taint, b_taint))
return output
@@ -642,7 +850,7 @@ def calc_basehash(sigdata):
if val is not None:
basedata = basedata + str(val)
- return hashlib.md5(basedata.encode("utf-8")).hexdigest()
+ return hashlib.sha256(basedata.encode("utf-8")).hexdigest()
def calc_taskhash(sigdata):
data = sigdata['basehash']
@@ -660,7 +868,7 @@ def calc_taskhash(sigdata):
else:
data = data + sigdata['taint']
- return hashlib.md5(data.encode("utf-8")).hexdigest()
+ return hashlib.sha256(data.encode("utf-8")).hexdigest()
def dump_sigfile(a):
@@ -695,7 +903,11 @@ def dump_sigfile(a):
output.append("Hash for dependent task %s is %s" % (dep, a_data['runtaskhashes'][dep]))
if 'taint' in a_data:
- output.append("Tainted (by forced/invalidated task): %s" % a_data['taint'])
+ if a_data['taint'].startswith('nostamp:'):
+ msg = a_data['taint'].replace('nostamp:', 'nostamp(uuid4):')
+ else:
+ msg = a_data['taint']
+ output.append("Tainted (by forced/invalidated task): %s" % msg)
if 'task' in a_data:
computed_basehash = calc_basehash(a_data)
diff --git a/bitbake/lib/bb/taskdata.py b/bitbake/lib/bb/taskdata.py
index 94e822c..8c25e09 100644
--- a/bitbake/lib/bb/taskdata.py
+++ b/bitbake/lib/bb/taskdata.py
@@ -1,6 +1,3 @@
-#!/usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
"""
BitBake 'TaskData' implementation
@@ -10,18 +7,8 @@ Task data collection and handling
# Copyright (C) 2006 Richard Purdie
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import logging
import re
@@ -93,7 +80,7 @@ class TaskData:
def add_mcdepends(task):
for dep in task_deps['mcdepends'][task].split():
if len(dep.split(':')) != 5:
- bb.msg.fatal("TaskData", "Error for %s:%s[%s], multiconfig dependency %s does not contain exactly four ':' characters.\n Task '%s' should be specified in the form 'multiconfig:fromMC:toMC:packagename:task'" % (fn, task, 'mcdepends', dep, 'mcdepends'))
+ bb.msg.fatal("TaskData", "Error for %s:%s[%s], multiconfig dependency %s does not contain exactly four ':' characters.\n Task '%s' should be specified in the form 'mc:fromMC:toMC:packagename:task'" % (fn, task, 'mcdepends', dep, 'mcdepends'))
if dep not in self.mcdepends:
self.mcdepends.append(dep)
diff --git a/bitbake/lib/bb/tests/codeparser.py b/bitbake/lib/bb/tests/codeparser.py
index e30e78c..826a2d2 100644
--- a/bitbake/lib/bb/tests/codeparser.py
+++ b/bitbake/lib/bb/tests/codeparser.py
@@ -1,23 +1,10 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# BitBake Test for codeparser.py
#
# Copyright (C) 2010 Chris Larson
# Copyright (C) 2012 Richard Purdie
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+# SPDX-License-Identifier: GPL-2.0-only
#
import unittest
@@ -123,6 +110,13 @@ ${D}${libdir}/pkgconfig/*.pc
self.parseExpression("sed -i -e 's:IP{:I${:g' $pc")
self.assertExecs(set(["sed"]))
+ def test_parameter_expansion_modifiers(self):
+ # - and + are also valid modifiers for parameter expansion, but are
+ # valid characters in bitbake variable names, so are not included here
+ for i in ('=', ':-', ':=', '?', ':?', ':+', '#', '%', '##', '%%'):
+ name = "foo%sbar" % i
+ self.parseExpression("${%s}" % name)
+ self.assertNotIn(name, self.references)
def test_until(self):
self.parseExpression("until false; do echo true; done")
diff --git a/bitbake/lib/bb/tests/cooker.py b/bitbake/lib/bb/tests/cooker.py
index 2b44236..090916e 100644
--- a/bitbake/lib/bb/tests/cooker.py
+++ b/bitbake/lib/bb/tests/cooker.py
@@ -1,20 +1,7 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# BitBake Tests for cooker.py
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+# SPDX-License-Identifier: GPL-2.0-only
#
import unittest
diff --git a/bitbake/lib/bb/tests/cow.py b/bitbake/lib/bb/tests/cow.py
index d149d84..b4af4bb 100644
--- a/bitbake/lib/bb/tests/cow.py
+++ b/bitbake/lib/bb/tests/cow.py
@@ -1,22 +1,9 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# BitBake Tests for Copy-on-Write (cow.py)
#
-# Copyright 2006 Holger Freyther <freyther@handhelds.org>
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+# Copyright 2006 Holger Freyther <freyther@handhelds.org>
#
import unittest
diff --git a/bitbake/lib/bb/tests/data.py b/bitbake/lib/bb/tests/data.py
index db3e201..3e49984 100644
--- a/bitbake/lib/bb/tests/data.py
+++ b/bitbake/lib/bb/tests/data.py
@@ -1,23 +1,10 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# BitBake Tests for the Data Store (data.py/data_smart.py)
#
# Copyright (C) 2010 Chris Larson
# Copyright (C) 2012 Richard Purdie
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+# SPDX-License-Identifier: GPL-2.0-only
#
import unittest
@@ -394,6 +381,28 @@ class TestOverrides(unittest.TestCase):
self.d.setVar("OVERRIDES", "foo:bar:some_val")
self.assertEqual(self.d.getVar("TEST"), " testvalue5")
+ def test_append_and_override_1(self):
+ self.d.setVar("TEST_append", "testvalue2")
+ self.d.setVar("TEST_bar", "testvalue3")
+ self.assertEqual(self.d.getVar("TEST"), "testvalue3testvalue2")
+
+ def test_append_and_override_2(self):
+ self.d.setVar("TEST_append_bar", "testvalue2")
+ self.assertEqual(self.d.getVar("TEST"), "testvaluetestvalue2")
+
+ def test_append_and_override_3(self):
+ self.d.setVar("TEST_bar_append", "testvalue2")
+ self.assertEqual(self.d.getVar("TEST"), "testvalue2")
+
+ # Test an override with _<numeric> in it based on a real world OE issue
+ def test_underscore_override(self):
+ self.d.setVar("TARGET_ARCH", "x86_64")
+ self.d.setVar("PN", "test-${TARGET_ARCH}")
+ self.d.setVar("VERSION", "1")
+ self.d.setVar("VERSION_pn-test-${TARGET_ARCH}", "2")
+ self.d.setVar("OVERRIDES", "pn-${PN}")
+ bb.data.expandKeys(self.d)
+ self.assertEqual(self.d.getVar("VERSION"), "2")
class TestKeyExpansion(unittest.TestCase):
def setUp(self):
@@ -470,7 +479,7 @@ class TaskHash(unittest.TestCase):
tasklist, gendeps, lookupcache = bb.data.generate_dependencies(d)
taskdeps, basehash = bb.data.generate_dependency_hash(tasklist, gendeps, lookupcache, set(), "somefile")
bb.warn(str(lookupcache))
- return basehash["somefile." + taskname]
+ return basehash["somefile:" + taskname]
d = bb.data.init()
d.setVar("__BBTASKS", ["mytask"])
diff --git a/bitbake/lib/bb/tests/event.py b/bitbake/lib/bb/tests/event.py
index d3a5f62..9229b63 100644
--- a/bitbake/lib/bb/tests/event.py
+++ b/bitbake/lib/bb/tests/event.py
@@ -1,22 +1,9 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# BitBake Tests for the Event implementation (event.py)
#
# Copyright (C) 2017 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+# SPDX-License-Identifier: GPL-2.0-only
#
import unittest
@@ -574,14 +561,6 @@ class EventClassesTest(unittest.TestCase):
self.assertEqual(event.fn(1), callback(1))
self.assertEqual(event.pid, EventClassesTest._worker_pid)
- def test_StampUpdate(self):
- targets = ["foo", "bar"]
- stampfns = [lambda:"foobar"]
- event = bb.event.StampUpdate(targets, stampfns)
- self.assertEqual(event.targets, targets)
- self.assertEqual(event.stampPrefix, stampfns)
- self.assertEqual(event.pid, EventClassesTest._worker_pid)
-
def test_BuildBase(self):
""" Test base class for bitbake build events """
name = "foo"
diff --git a/bitbake/lib/bb/tests/fetch.py b/bitbake/lib/bb/tests/fetch.py
index 6848095..a0b656b 100644
--- a/bitbake/lib/bb/tests/fetch.py
+++ b/bitbake/lib/bb/tests/fetch.py
@@ -1,22 +1,9 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# BitBake Tests for the Fetcher (fetch2/)
#
# Copyright (C) 2012 Richard Purdie
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+# SPDX-License-Identifier: GPL-2.0-only
#
import unittest
@@ -893,12 +880,201 @@ class FetcherNetworkTest(FetcherTest):
@skipIfNoNetwork()
def test_git_submodule(self):
- fetcher = bb.fetch.Fetch(["gitsm://git.yoctoproject.org/git-submodule-test;rev=f12e57f2edf0aa534cf1616fa983d165a92b0842"], self.d)
+ # URL with ssh submodules
+ url = "gitsm://git.yoctoproject.org/git-submodule-test;branch=ssh-gitsm-tests;rev=049da4a6cb198d7c0302e9e8b243a1443cb809a7"
+ # Original URL (comment this if you have ssh access to git.yoctoproject.org)
+ url = "gitsm://git.yoctoproject.org/git-submodule-test;branch=master;rev=a2885dd7d25380d23627e7544b7bbb55014b16ee"
+ fetcher = bb.fetch.Fetch([url], self.d)
+ fetcher.download()
+ # Previous cwd has been deleted
+ os.chdir(os.path.dirname(self.unpackdir))
+ fetcher.unpack(self.unpackdir)
+
+ repo_path = os.path.join(self.tempdir, 'unpacked', 'git')
+ self.assertTrue(os.path.exists(repo_path), msg='Unpacked repository missing')
+ self.assertTrue(os.path.exists(os.path.join(repo_path, 'bitbake')), msg='bitbake submodule missing')
+ self.assertFalse(os.path.exists(os.path.join(repo_path, 'na')), msg='uninitialized submodule present')
+
+ # Only when we're running the extended test with a submodule's submodule, can we check this.
+ if os.path.exists(os.path.join(repo_path, 'bitbake-gitsm-test1')):
+ self.assertTrue(os.path.exists(os.path.join(repo_path, 'bitbake-gitsm-test1', 'bitbake')), msg='submodule of submodule missing')
+
+ @skipIfNoNetwork()
+ def test_git_submodule_dbus_broker(self):
+ # The following external repositories have show failures in fetch and unpack operations
+ # We want to avoid regressions!
+ url = "gitsm://github.com/bus1/dbus-broker;protocol=git;rev=fc874afa0992d0c75ec25acb43d344679f0ee7d2"
+ fetcher = bb.fetch.Fetch([url], self.d)
+ fetcher.download()
+ # Previous cwd has been deleted
+ os.chdir(os.path.dirname(self.unpackdir))
+ fetcher.unpack(self.unpackdir)
+
+ repo_path = os.path.join(self.tempdir, 'unpacked', 'git')
+ self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/subprojects/c-dvar/config')), msg='Missing submodule config "subprojects/c-dvar"')
+ self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/subprojects/c-list/config')), msg='Missing submodule config "subprojects/c-list"')
+ self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/subprojects/c-rbtree/config')), msg='Missing submodule config "subprojects/c-rbtree"')
+ self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/subprojects/c-sundry/config')), msg='Missing submodule config "subprojects/c-sundry"')
+ self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/subprojects/c-utf8/config')), msg='Missing submodule config "subprojects/c-utf8"')
+
+ @skipIfNoNetwork()
+ def test_git_submodule_CLI11(self):
+ url = "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=bd4dc911847d0cde7a6b41dfa626a85aab213baf"
+ fetcher = bb.fetch.Fetch([url], self.d)
+ fetcher.download()
+ # Previous cwd has been deleted
+ os.chdir(os.path.dirname(self.unpackdir))
+ fetcher.unpack(self.unpackdir)
+
+ repo_path = os.path.join(self.tempdir, 'unpacked', 'git')
+ self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/googletest/config')), msg='Missing submodule config "extern/googletest"')
+ self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/json/config')), msg='Missing submodule config "extern/json"')
+ self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/sanitizers/config')), msg='Missing submodule config "extern/sanitizers"')
+
+ @skipIfNoNetwork()
+ def test_git_submodule_update_CLI11(self):
+ """ Prevent regression on update detection not finding missing submodule, or modules without needed commits """
+ url = "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=cf6a99fa69aaefe477cc52e3ef4a7d2d7fa40714"
+ fetcher = bb.fetch.Fetch([url], self.d)
+ fetcher.download()
+
+ # CLI11 that pulls in a newer nlohmann-json
+ url = "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=49ac989a9527ee9bb496de9ded7b4872c2e0e5ca"
+ fetcher = bb.fetch.Fetch([url], self.d)
+ fetcher.download()
+ # Previous cwd has been deleted
+ os.chdir(os.path.dirname(self.unpackdir))
+ fetcher.unpack(self.unpackdir)
+
+ repo_path = os.path.join(self.tempdir, 'unpacked', 'git')
+ self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/googletest/config')), msg='Missing submodule config "extern/googletest"')
+ self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/json/config')), msg='Missing submodule config "extern/json"')
+ self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/extern/sanitizers/config')), msg='Missing submodule config "extern/sanitizers"')
+
+ @skipIfNoNetwork()
+ def test_git_submodule_aktualizr(self):
+ url = "gitsm://github.com/advancedtelematic/aktualizr;branch=master;protocol=git;rev=d00d1a04cc2366d1a5f143b84b9f507f8bd32c44"
+ fetcher = bb.fetch.Fetch([url], self.d)
fetcher.download()
# Previous cwd has been deleted
os.chdir(os.path.dirname(self.unpackdir))
fetcher.unpack(self.unpackdir)
+ repo_path = os.path.join(self.tempdir, 'unpacked', 'git')
+ self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/partial/extern/isotp-c/config')), msg='Missing submodule config "partial/extern/isotp-c/config"')
+ self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/partial/extern/isotp-c/modules/deps/bitfield-c/config')), msg='Missing submodule config "partial/extern/isotp-c/modules/deps/bitfield-c/config"')
+ self.assertTrue(os.path.exists(os.path.join(repo_path, 'partial/extern/isotp-c/deps/bitfield-c/.git')), msg="Submodule of submodule isotp-c did not unpack properly")
+ self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/tests/tuf-test-vectors/config')), msg='Missing submodule config "tests/tuf-test-vectors/config"')
+ self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/third_party/googletest/config')), msg='Missing submodule config "third_party/googletest/config"')
+ self.assertTrue(os.path.exists(os.path.join(repo_path, '.git/modules/third_party/HdrHistogram_c/config')), msg='Missing submodule config "third_party/HdrHistogram_c/config"')
+
+ @skipIfNoNetwork()
+ def test_git_submodule_iotedge(self):
+ """ Prevent regression on deeply nested submodules not being checked out properly, even though they were fetched. """
+
+ # This repository also has submodules where the module (name), path and url do not align
+ url = "gitsm://github.com/azure/iotedge.git;protocol=git;rev=d76e0316c6f324345d77c48a83ce836d09392699"
+ fetcher = bb.fetch.Fetch([url], self.d)
+ fetcher.download()
+ # Previous cwd has been deleted
+ os.chdir(os.path.dirname(self.unpackdir))
+ fetcher.unpack(self.unpackdir)
+
+ repo_path = os.path.join(self.tempdir, 'unpacked', 'git')
+
+ self.assertTrue(os.path.exists(os.path.join(repo_path, 'edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared/README.md')), msg='Missing submodule checkout')
+ self.assertTrue(os.path.exists(os.path.join(repo_path, 'edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared/testtools/ctest/README.md')), msg='Missing submodule checkout')
+ self.assertTrue(os.path.exists(os.path.join(repo_path, 'edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared/testtools/testrunner/readme.md')), msg='Missing submodule checkout')
+ self.assertTrue(os.path.exists(os.path.join(repo_path, 'edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared/testtools/umock-c/readme.md')), msg='Missing submodule checkout')
+ self.assertTrue(os.path.exists(os.path.join(repo_path, 'edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared/testtools/umock-c/deps/ctest/README.md')), msg='Missing submodule checkout')
+ self.assertTrue(os.path.exists(os.path.join(repo_path, 'edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared/testtools/umock-c/deps/testrunner/readme.md')), msg='Missing submodule checkout')
+ self.assertTrue(os.path.exists(os.path.join(repo_path, 'edgelet/hsm-sys/azure-iot-hsm-c/deps/utpm/README.md')), msg='Missing submodule checkout')
+ self.assertTrue(os.path.exists(os.path.join(repo_path, 'edgelet/hsm-sys/azure-iot-hsm-c/deps/utpm/deps/c-utility/README.md')), msg='Missing submodule checkout')
+ self.assertTrue(os.path.exists(os.path.join(repo_path, 'edgelet/hsm-sys/azure-iot-hsm-c/deps/utpm/deps/c-utility/testtools/ctest/README.md')), msg='Missing submodule checkout')
+ self.assertTrue(os.path.exists(os.path.join(repo_path, 'edgelet/hsm-sys/azure-iot-hsm-c/deps/utpm/deps/c-utility/testtools/testrunner/readme.md')), msg='Missing submodule checkout')
+ self.assertTrue(os.path.exists(os.path.join(repo_path, 'edgelet/hsm-sys/azure-iot-hsm-c/deps/utpm/deps/c-utility/testtools/umock-c/readme.md')), msg='Missing submodule checkout')
+ self.assertTrue(os.path.exists(os.path.join(repo_path, 'edgelet/hsm-sys/azure-iot-hsm-c/deps/utpm/deps/c-utility/testtools/umock-c/deps/ctest/README.md')), msg='Missing submodule checkout')
+ self.assertTrue(os.path.exists(os.path.join(repo_path, 'edgelet/hsm-sys/azure-iot-hsm-c/deps/utpm/deps/c-utility/testtools/umock-c/deps/testrunner/readme.md')), msg='Missing submodule checkout')
+
+class SVNTest(FetcherTest):
+ def skipIfNoSvn():
+ import shutil
+ if not shutil.which("svn"):
+ return unittest.skip("svn not installed, tests being skipped")
+
+ if not shutil.which("svnadmin"):
+ return unittest.skip("svnadmin not installed, tests being skipped")
+
+ return lambda f: f
+
+ @skipIfNoSvn()
+ def setUp(self):
+ """ Create a local repository """
+
+ super(SVNTest, self).setUp()
+
+ # Create something we can fetch
+ src_dir = tempfile.mkdtemp(dir=self.tempdir,
+ prefix='svnfetch_srcdir_')
+ src_dir = os.path.abspath(src_dir)
+ bb.process.run("echo readme > README.md", cwd=src_dir)
+
+ # Store it in a local SVN repository
+ repo_dir = tempfile.mkdtemp(dir=self.tempdir,
+ prefix='svnfetch_localrepo_')
+ repo_dir = os.path.abspath(repo_dir)
+ bb.process.run("svnadmin create project", cwd=repo_dir)
+
+ self.repo_url = "file://%s/project" % repo_dir
+ bb.process.run("svn import --non-interactive -m 'Initial import' %s %s/trunk" % (src_dir, self.repo_url),
+ cwd=repo_dir)
+
+ bb.process.run("svn co %s svnfetch_co" % self.repo_url, cwd=self.tempdir)
+ # Github will emulate SVN. Use this to check if we're downloding...
+ bb.process.run("svn propset svn:externals 'bitbake http://github.com/openembedded/bitbake' .",
+ cwd=os.path.join(self.tempdir, 'svnfetch_co', 'trunk'))
+ bb.process.run("svn commit --non-interactive -m 'Add external'",
+ cwd=os.path.join(self.tempdir, 'svnfetch_co', 'trunk'))
+
+ self.src_dir = src_dir
+ self.repo_dir = repo_dir
+
+ @skipIfNoSvn()
+ def tearDown(self):
+ os.chdir(self.origdir)
+ if os.environ.get("BB_TMPDIR_NOCLEAN") == "yes":
+ print("Not cleaning up %s. Please remove manually." % self.tempdir)
+ else:
+ bb.utils.prunedir(self.tempdir)
+
+ @skipIfNoSvn()
+ @skipIfNoNetwork()
+ def test_noexternal_svn(self):
+ # Always match the rev count from setUp (currently rev 2)
+ url = "svn://%s;module=trunk;protocol=file;rev=2" % self.repo_url.replace('file://', '')
+ fetcher = bb.fetch.Fetch([url], self.d)
+ fetcher.download()
+ os.chdir(os.path.dirname(self.unpackdir))
+ fetcher.unpack(self.unpackdir)
+
+ self.assertTrue(os.path.exists(os.path.join(self.unpackdir, 'trunk')), msg="Missing trunk")
+ self.assertTrue(os.path.exists(os.path.join(self.unpackdir, 'trunk', 'README.md')), msg="Missing contents")
+ self.assertFalse(os.path.exists(os.path.join(self.unpackdir, 'trunk/bitbake/trunk')), msg="External dir should NOT exist")
+ self.assertFalse(os.path.exists(os.path.join(self.unpackdir, 'trunk/bitbake/trunk', 'README')), msg="External README should NOT exit")
+
+ @skipIfNoSvn()
+ def test_external_svn(self):
+ # Always match the rev count from setUp (currently rev 2)
+ url = "svn://%s;module=trunk;protocol=file;externals=allowed;rev=2" % self.repo_url.replace('file://', '')
+ fetcher = bb.fetch.Fetch([url], self.d)
+ fetcher.download()
+ os.chdir(os.path.dirname(self.unpackdir))
+ fetcher.unpack(self.unpackdir)
+
+ self.assertTrue(os.path.exists(os.path.join(self.unpackdir, 'trunk')), msg="Missing trunk")
+ self.assertTrue(os.path.exists(os.path.join(self.unpackdir, 'trunk', 'README.md')), msg="Missing contents")
+ self.assertTrue(os.path.exists(os.path.join(self.unpackdir, 'trunk/bitbake/trunk')), msg="External dir should exist")
+ self.assertTrue(os.path.exists(os.path.join(self.unpackdir, 'trunk/bitbake/trunk', 'README')), msg="External README should exit")
class TrustedNetworksTest(FetcherTest):
def test_trusted_network(self):
@@ -1024,8 +1200,8 @@ class FetchLatestVersionTest(FetcherTest):
# packages with valid UPSTREAM_CHECK_URI and UPSTREAM_CHECK_REGEX
("cups", "http://www.cups.org/software/1.7.2/cups-1.7.2-source.tar.bz2", "https://github.com/apple/cups/releases", "(?P<name>cups\-)(?P<pver>((\d+[\.\-_]*)+))\-source\.tar\.gz")
: "2.0.0",
- ("db", "http://download.oracle.com/berkeley-db/db-5.3.21.tar.gz", "http://www.oracle.com/technetwork/products/berkeleydb/downloads/index-082944.html", "http://download.oracle.com/otn/berkeley-db/(?P<name>db-)(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz")
- : "6.1.19",
+ ("db", "http://download.oracle.com/berkeley-db/db-5.3.21.tar.gz", "http://ftp.debian.org/debian/pool/main/d/db5.3/", "(?P<name>db5\.3_)(?P<pver>\d+(\.\d+)+).+\.orig\.tar\.xz")
+ : "5.3.10",
}
@skipIfNoNetwork()
@@ -1280,7 +1456,7 @@ class GitShallowTest(FetcherTest):
def fetch(self, uri=None):
if uri is None:
- uris = self.d.getVar('SRC_URI', True).split()
+ uris = self.d.getVar('SRC_URI').split()
uri = uris[0]
d = self.d
else:
@@ -1312,6 +1488,7 @@ class GitShallowTest(FetcherTest):
# fetch and unpack, from the shallow tarball
bb.utils.remove(self.gitdir, recurse=True)
bb.utils.remove(ud.clonedir, recurse=True)
+ bb.utils.remove(ud.clonedir.replace('gitsource', 'gitsubmodule'), recurse=True)
# confirm that the unpacked repo is used when no git clone or git
# mirror tarball is available
@@ -1338,7 +1515,7 @@ class GitShallowTest(FetcherTest):
srcrev = self.git('rev-parse HEAD', cwd=self.srcdir).strip()
self.d.setVar('SRCREV', srcrev)
- uri = self.d.getVar('SRC_URI', True).split()[0]
+ uri = self.d.getVar('SRC_URI').split()[0]
uri = '%s;nobranch=1;bare=1' % uri
self.fetch_shallow(uri)
@@ -1466,6 +1643,7 @@ class GitShallowTest(FetcherTest):
self.git('config --add remote.origin.url "%s"' % smdir, cwd=smdir)
self.git('config --add remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*"', cwd=smdir)
self.add_empty_file('asub', cwd=smdir)
+ self.add_empty_file('bsub', cwd=smdir)
self.git('submodule init', cwd=self.srcdir)
self.git('submodule add file://%s' % smdir, cwd=self.srcdir)
@@ -1475,10 +1653,16 @@ class GitShallowTest(FetcherTest):
uri = 'gitsm://%s;protocol=file;subdir=${S}' % self.srcdir
fetcher, ud = self.fetch_shallow(uri)
+ # Verify the main repository is shallow
self.assertRevCount(1)
- assert './.git/modules/' in bb.process.run('tar -tzf %s' % os.path.join(self.dldir, ud.mirrortarballs[0]))[0]
+
+ # Verify the gitsubmodule directory is present
assert os.listdir(os.path.join(self.gitdir, 'gitsubmodule'))
+ # Verify the submodule is also shallow
+ self.assertRevCount(1, cwd=os.path.join(self.gitdir, 'gitsubmodule'))
+
+
if any(os.path.exists(os.path.join(p, 'git-annex')) for p in os.environ.get('PATH').split(':')):
def test_shallow_annex(self):
self.add_empty_file('a')
@@ -1510,7 +1694,7 @@ class GitShallowTest(FetcherTest):
self.add_empty_file('f')
self.assertRevCount(7, cwd=self.srcdir)
- uri = self.d.getVar('SRC_URI', True).split()[0]
+ uri = self.d.getVar('SRC_URI').split()[0]
uri = '%s;branch=master,a_branch;name=master,a_branch' % uri
self.d.setVar('BB_GIT_SHALLOW_DEPTH', '0')
@@ -1536,7 +1720,7 @@ class GitShallowTest(FetcherTest):
self.add_empty_file('f')
self.assertRevCount(7, cwd=self.srcdir)
- uri = self.d.getVar('SRC_URI', True).split()[0]
+ uri = self.d.getVar('SRC_URI').split()[0]
uri = '%s;branch=master,a_branch;name=master,a_branch' % uri
self.d.setVar('BB_GIT_SHALLOW_DEPTH', '0')
@@ -1724,3 +1908,83 @@ class GitShallowTest(FetcherTest):
dir = os.listdir(self.unpackdir + "/git/")
self.assertIn("fstests.doap", dir)
+
+class GitLfsTest(FetcherTest):
+ def setUp(self):
+ FetcherTest.setUp(self)
+
+ self.gitdir = os.path.join(self.tempdir, 'git')
+ self.srcdir = os.path.join(self.tempdir, 'gitsource')
+
+ self.d.setVar('WORKDIR', self.tempdir)
+ self.d.setVar('S', self.gitdir)
+ self.d.delVar('PREMIRRORS')
+ self.d.delVar('MIRRORS')
+
+ self.d.setVar('SRCREV', '${AUTOREV}')
+ self.d.setVar('AUTOREV', '${@bb.fetch2.get_autorev(d)}')
+
+ bb.utils.mkdirhier(self.srcdir)
+ self.git('init', cwd=self.srcdir)
+ with open(os.path.join(self.srcdir, '.gitattributes'), 'wt') as attrs:
+ attrs.write('*.mp3 filter=lfs -text')
+ self.git(['add', '.gitattributes'], cwd=self.srcdir)
+ self.git(['commit', '-m', "attributes", '.gitattributes'], cwd=self.srcdir)
+
+ def git(self, cmd, cwd=None):
+ if isinstance(cmd, str):
+ cmd = 'git ' + cmd
+ else:
+ cmd = ['git'] + cmd
+ if cwd is None:
+ cwd = self.gitdir
+ return bb.process.run(cmd, cwd=cwd)[0]
+
+ def fetch(self, uri=None):
+ uris = self.d.getVar('SRC_URI').split()
+ uri = uris[0]
+ d = self.d
+
+ fetcher = bb.fetch2.Fetch(uris, d)
+ fetcher.download()
+ ud = fetcher.ud[uri]
+ return fetcher, ud
+
+ def test_lfs_enabled(self):
+ import shutil
+
+ uri = 'git://%s;protocol=file;subdir=${S};lfs=1' % self.srcdir
+ self.d.setVar('SRC_URI', uri)
+
+ fetcher, ud = self.fetch()
+ self.assertIsNotNone(ud.method._find_git_lfs)
+
+ # If git-lfs can be found, the unpack should be successful
+ ud.method._find_git_lfs = lambda d: True
+ shutil.rmtree(self.gitdir, ignore_errors=True)
+ fetcher.unpack(self.d.getVar('WORKDIR'))
+
+ # If git-lfs cannot be found, the unpack should throw an error
+ with self.assertRaises(bb.fetch2.FetchError):
+ ud.method._find_git_lfs = lambda d: False
+ shutil.rmtree(self.gitdir, ignore_errors=True)
+ fetcher.unpack(self.d.getVar('WORKDIR'))
+
+ def test_lfs_disabled(self):
+ import shutil
+
+ uri = 'git://%s;protocol=file;subdir=${S};lfs=0' % self.srcdir
+ self.d.setVar('SRC_URI', uri)
+
+ fetcher, ud = self.fetch()
+ self.assertIsNotNone(ud.method._find_git_lfs)
+
+ # If git-lfs can be found, the unpack should be successful
+ ud.method._find_git_lfs = lambda d: True
+ shutil.rmtree(self.gitdir, ignore_errors=True)
+ fetcher.unpack(self.d.getVar('WORKDIR'))
+
+ # If git-lfs cannot be found, the unpack should be successful
+ ud.method._find_git_lfs = lambda d: False
+ shutil.rmtree(self.gitdir, ignore_errors=True)
+ fetcher.unpack(self.d.getVar('WORKDIR'))
diff --git a/bitbake/lib/bb/tests/parse.py b/bitbake/lib/bb/tests/parse.py
index 1bc4740..9afd1b2 100644
--- a/bitbake/lib/bb/tests/parse.py
+++ b/bitbake/lib/bb/tests/parse.py
@@ -1,22 +1,9 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# BitBake Test for lib/bb/parse/
#
# Copyright (C) 2015 Richard Purdie
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+# SPDX-License-Identifier: GPL-2.0-only
#
import unittest
@@ -187,3 +174,21 @@ python () {
self.assertEqual(d1.getVar("VAR_var"), "B")
self.assertEqual(d2.getVar("VAR_var"), None)
+ addtask_deltask = """
+addtask do_patch after do_foo after do_unpack before do_configure before do_compile
+addtask do_fetch do_patch
+
+deltask do_fetch do_patch
+"""
+ def test_parse_addtask_deltask(self):
+ import sys
+ f = self.parsehelper(self.addtask_deltask)
+ d = bb.parse.handle(f.name, self.d)['']
+
+ stdout = sys.stdout.getvalue()
+ self.assertTrue("addtask contained multiple 'before' keywords" in stdout)
+ self.assertTrue("addtask contained multiple 'after' keywords" in stdout)
+ self.assertTrue('addtask ignored: " do_patch"' in stdout)
+ self.assertTrue('deltask ignored: " do_patch"' in stdout)
+ #self.assertTrue('dependent task do_foo for do_patch does not exist' in stdout)
+
diff --git a/bitbake/lib/bb/tests/persist_data.py b/bitbake/lib/bb/tests/persist_data.py
new file mode 100644
index 0000000..f641b5a
--- /dev/null
+++ b/bitbake/lib/bb/tests/persist_data.py
@@ -0,0 +1,129 @@
+#
+# BitBake Test for lib/bb/persist_data/
+#
+# Copyright (C) 2018 Garmin Ltd.
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+import unittest
+import bb.data
+import bb.persist_data
+import tempfile
+import threading
+
+class PersistDataTest(unittest.TestCase):
+ def _create_data(self):
+ return bb.persist_data.persist('TEST_PERSIST_DATA', self.d)
+
+ def setUp(self):
+ self.d = bb.data.init()
+ self.tempdir = tempfile.TemporaryDirectory()
+ self.d['PERSISTENT_DIR'] = self.tempdir.name
+ self.data = self._create_data()
+ self.items = {
+ 'A1': '1',
+ 'B1': '2',
+ 'C2': '3'
+ }
+ self.stress_count = 10000
+ self.thread_count = 5
+
+ for k,v in self.items.items():
+ self.data[k] = v
+
+ def tearDown(self):
+ self.tempdir.cleanup()
+
+ def _iter_helper(self, seen, iterator):
+ with iter(iterator):
+ for v in iterator:
+ self.assertTrue(v in seen)
+ seen.remove(v)
+ self.assertEqual(len(seen), 0, '%s not seen' % seen)
+
+ def test_get(self):
+ for k, v in self.items.items():
+ self.assertEqual(self.data[k], v)
+
+ self.assertIsNone(self.data.get('D'))
+ with self.assertRaises(KeyError):
+ self.data['D']
+
+ def test_set(self):
+ for k, v in self.items.items():
+ self.data[k] += '-foo'
+
+ for k, v in self.items.items():
+ self.assertEqual(self.data[k], v + '-foo')
+
+ def test_delete(self):
+ self.data['D'] = '4'
+ self.assertEqual(self.data['D'], '4')
+ del self.data['D']
+ self.assertIsNone(self.data.get('D'))
+ with self.assertRaises(KeyError):
+ self.data['D']
+
+ def test_contains(self):
+ for k in self.items:
+ self.assertTrue(k in self.data)
+ self.assertTrue(self.data.has_key(k))
+ self.assertFalse('NotFound' in self.data)
+ self.assertFalse(self.data.has_key('NotFound'))
+
+ def test_len(self):
+ self.assertEqual(len(self.data), len(self.items))
+
+ def test_iter(self):
+ self._iter_helper(set(self.items.keys()), self.data)
+
+ def test_itervalues(self):
+ self._iter_helper(set(self.items.values()), self.data.itervalues())
+
+ def test_iteritems(self):
+ self._iter_helper(set(self.items.items()), self.data.iteritems())
+
+ def test_get_by_pattern(self):
+ self._iter_helper({'1', '2'}, self.data.get_by_pattern('_1'))
+
+ def _stress_read(self, data):
+ for i in range(self.stress_count):
+ for k in self.items:
+ data[k]
+
+ def _stress_write(self, data):
+ for i in range(self.stress_count):
+ for k, v in self.items.items():
+ data[k] = v + str(i)
+
+ def _validate_stress(self):
+ for k, v in self.items.items():
+ self.assertEqual(self.data[k], v + str(self.stress_count - 1))
+
+ def test_stress(self):
+ self._stress_read(self.data)
+ self._stress_write(self.data)
+ self._validate_stress()
+
+ def test_stress_threads(self):
+ def read_thread():
+ data = self._create_data()
+ self._stress_read(data)
+
+ def write_thread():
+ data = self._create_data()
+ self._stress_write(data)
+
+ threads = []
+ for i in range(self.thread_count):
+ threads.append(threading.Thread(target=read_thread))
+ threads.append(threading.Thread(target=write_thread))
+
+ for t in threads:
+ t.start()
+ self._stress_read(self.data)
+ for t in threads:
+ t.join()
+ self._validate_stress()
+
diff --git a/bitbake/lib/bb/tests/runqueue-tests/classes/base.bbclass b/bitbake/lib/bb/tests/runqueue-tests/classes/base.bbclass
new file mode 100644
index 0000000..b57650d
--- /dev/null
+++ b/bitbake/lib/bb/tests/runqueue-tests/classes/base.bbclass
@@ -0,0 +1,262 @@
+SLOWTASKS ??= ""
+SSTATEVALID ??= ""
+
+def stamptask(d):
+ import time
+
+ thistask = d.expand("${PN}:${BB_CURRENTTASK}")
+ stampname = d.expand("${TOPDIR}/%s.run" % thistask)
+ with open(stampname, "a+") as f:
+ f.write(d.getVar("BB_UNIHASH") + "\n")
+
+ if d.getVar("BB_CURRENT_MC") != "default":
+ thistask = d.expand("${BB_CURRENT_MC}:${PN}:${BB_CURRENTTASK}")
+ if thistask in d.getVar("SLOWTASKS").split():
+ bb.note("Slowing task %s" % thistask)
+ time.sleep(0.5)
+ if d.getVar("BB_HASHSERVE"):
+ task = d.getVar("BB_CURRENTTASK")
+ if task in ['package', 'package_qa', 'packagedata', 'package_write_ipk', 'package_write_rpm', 'populate_lic', 'populate_sysroot']:
+ bb.parse.siggen.report_unihash(os.getcwd(), d.getVar("BB_CURRENTTASK"), d)
+
+ with open(d.expand("${TOPDIR}/task.log"), "a+") as f:
+ f.write(thistask + "\n")
+
+
+def sstate_output_hash(path, sigfile, task, d):
+ import hashlib
+ h = hashlib.sha256()
+ h.update(d.expand("${PN}:${BB_CURRENTTASK}").encode('utf-8'))
+ return h.hexdigest()
+
+python do_fetch() {
+ # fetch
+ stamptask(d)
+}
+python do_unpack() {
+ # unpack
+ stamptask(d)
+}
+python do_patch() {
+ # patch
+ stamptask(d)
+}
+python do_populate_lic() {
+ # populate_lic
+ stamptask(d)
+}
+python do_prepare_recipe_sysroot() {
+ # prepare_recipe_sysroot
+ stamptask(d)
+}
+python do_configure() {
+ # configure
+ stamptask(d)
+}
+python do_compile() {
+ # compile
+ stamptask(d)
+}
+python do_install() {
+ # install
+ stamptask(d)
+}
+python do_populate_sysroot() {
+ # populate_sysroot
+ stamptask(d)
+}
+python do_package() {
+ # package
+ stamptask(d)
+}
+python do_package_write_ipk() {
+ # package_write_ipk
+ stamptask(d)
+}
+python do_package_write_rpm() {
+ # package_write_rpm
+ stamptask(d)
+}
+python do_packagedata() {
+ # packagedata
+ stamptask(d)
+}
+python do_package_qa() {
+ # package_qa
+ stamptask(d)
+}
+python do_build() {
+ # build
+ stamptask(d)
+}
+do_prepare_recipe_sysroot[deptask] = "do_populate_sysroot"
+do_package[deptask] += "do_packagedata"
+do_build[recrdeptask] += "do_deploy"
+do_build[recrdeptask] += "do_package_write_ipk"
+do_build[recrdeptask] += "do_package_write_rpm"
+do_package_qa[rdeptask] = "do_packagedata"
+do_populate_lic_deploy[recrdeptask] += "do_populate_lic do_deploy"
+
+DEBIANRDEP = "do_packagedata"
+oo_package_write_ipk[rdeptask] = "${DEBIANRDEP}"
+do_package_write_rpm[rdeptask] = "${DEBIANRDEP}"
+
+addtask fetch
+addtask unpack after do_fetch
+addtask patch after do_unpack
+addtask prepare_recipe_sysroot after do_patch
+addtask configure after do_prepare_recipe_sysroot
+addtask compile after do_configure
+addtask install after do_compile
+addtask populate_sysroot after do_install
+addtask package after do_install
+addtask package_write_ipk after do_packagedata do_package
+addtask package_write_rpm after do_packagedata do_package
+addtask packagedata after do_package
+addtask package_qa after do_package
+addtask build after do_package_qa do_package_write_rpm do_package_write_ipk do_populate_sysroot
+
+python do_package_setscene() {
+ stamptask(d)
+}
+python do_package_qa_setscene() {
+ stamptask(d)
+}
+python do_package_write_ipk_setscene() {
+ stamptask(d)
+}
+python do_package_write_rpm_setscene() {
+ stamptask(d)
+}
+python do_packagedata_setscene() {
+ stamptask(d)
+}
+python do_populate_lic_setscene() {
+ stamptask(d)
+}
+python do_populate_sysroot_setscene() {
+ stamptask(d)
+}
+
+addtask package_setscene
+addtask package_qa_setscene
+addtask package_write_ipk_setscene
+addtask package_write_rpm_setscene
+addtask packagedata_setscene
+addtask populate_lic_setscene
+addtask populate_sysroot_setscene
+
+BB_SETSCENE_DEPVALID = "setscene_depvalid"
+
+def setscene_depvalid(task, taskdependees, notneeded, d, log=None):
+ # taskdependees is a dict of tasks which depend on task, each being a 3 item list of [PN, TASKNAME, FILENAME]
+ # task is included in taskdependees too
+ # Return - False - We need this dependency
+ # - True - We can skip this dependency
+ import re
+
+ def logit(msg, log):
+ if log is not None:
+ log.append(msg)
+ else:
+ bb.debug(2, msg)
+
+ logit("Considering setscene task: %s" % (str(taskdependees[task])), log)
+
+ def isNativeCross(x):
+ return x.endswith("-native") or "-cross-" in x or "-crosssdk" in x or x.endswith("-cross")
+
+ # We only need to trigger populate_lic through direct dependencies
+ if taskdependees[task][1] == "do_populate_lic":
+ return True
+
+ # We only need to trigger packagedata through direct dependencies
+ # but need to preserve packagedata on packagedata links
+ if taskdependees[task][1] == "do_packagedata":
+ for dep in taskdependees:
+ if taskdependees[dep][1] == "do_packagedata":
+ return False
+ return True
+
+ for dep in taskdependees:
+ logit(" considering dependency: %s" % (str(taskdependees[dep])), log)
+ if task == dep:
+ continue
+ if dep in notneeded:
+ continue
+ # do_package_write_* and do_package doesn't need do_package
+ if taskdependees[task][1] == "do_package" and taskdependees[dep][1] in ['do_package', 'do_package_write_ipk', 'do_package_write_rpm', 'do_packagedata', 'do_package_qa']:
+ continue
+ # do_package_write_* need do_populate_sysroot as they're mainly postinstall dependencies
+ if taskdependees[task][1] == "do_populate_sysroot" and taskdependees[dep][1] in ['do_package_write_ipk', 'do_package_write_rpm']:
+ return False
+ # do_package/packagedata/package_qa don't need do_populate_sysroot
+ if taskdependees[task][1] == "do_populate_sysroot" and taskdependees[dep][1] in ['do_package', 'do_packagedata', 'do_package_qa']:
+ continue
+ # Native/Cross packages don't exist and are noexec anyway
+ if isNativeCross(taskdependees[dep][0]) and taskdependees[dep][1] in ['do_package_write_ipk', 'do_package_write_rpm', 'do_packagedata', 'do_package', 'do_package_qa']:
+ continue
+
+ # This is due to the [depends] in useradd.bbclass complicating matters
+ # The logic *is* reversed here due to the way hard setscene dependencies are injected
+ if (taskdependees[task][1] == 'do_package' or taskdependees[task][1] == 'do_populate_sysroot') and taskdependees[dep][0].endswith(('shadow-native', 'shadow-sysroot', 'base-passwd', 'pseudo-native')) and taskdependees[dep][1] == 'do_populate_sysroot':
+ continue
+
+ # Consider sysroot depending on sysroot tasks
+ if taskdependees[task][1] == 'do_populate_sysroot' and taskdependees[dep][1] == 'do_populate_sysroot':
+ # Native/Cross populate_sysroot need their dependencies
+ if isNativeCross(taskdependees[task][0]) and isNativeCross(taskdependees[dep][0]):
+ return False
+ # Target populate_sysroot depended on by cross tools need to be installed
+ if isNativeCross(taskdependees[dep][0]):
+ return False
+ # Native/cross tools depended upon by target sysroot are not needed
+ # Add an exception for shadow-native as required by useradd.bbclass
+ if isNativeCross(taskdependees[task][0]) and taskdependees[task][0] != 'shadow-native':
+ continue
+ # Target populate_sysroot need their dependencies
+ return False
+
+
+ if taskdependees[dep][1] == "do_populate_lic":
+ continue
+
+ # Safe fallthrough default
+ logit(" Default setscene dependency fall through due to dependency: %s" % (str(taskdependees[dep])), log)
+ return False
+ return True
+
+BB_HASHCHECK_FUNCTION = "sstate_checkhashes"
+
+def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, **kwargs):
+
+ found = set()
+ missed = set()
+
+ valid = d.getVar("SSTATEVALID").split()
+
+ for tid in sorted(sq_data['hash']):
+ n = os.path.basename(bb.runqueue.fn_from_tid(tid)).split(".")[0] + ":do_" + bb.runqueue.taskname_from_tid(tid)[3:]
+ print(n)
+ stampfile = d.expand("${TOPDIR}/%s.run" % n.replace("do_", ""))
+ if n in valid:
+ bb.note("SState: Found valid sstate for %s" % n)
+ found.add(tid)
+ elif n + ":" + sq_data['hash'][tid] in valid:
+ bb.note("SState: Found valid sstate for %s" % n)
+ found.add(tid)
+ elif os.path.exists(stampfile):
+ with open(stampfile, "r") as f:
+ hash = f.readline().strip()
+ if hash == sq_data['hash'][tid]:
+ bb.note("SState: Found valid sstate for %s (already run)" % n)
+ found.add(tid)
+ else:
+ bb.note("SState: sstate hash didn't match previous run for %s (%s vs %s)" % (n, sq_data['hash'][tid], hash))
+ missed.add(tid)
+ else:
+ missed.add(tid)
+ bb.note("SState: Found no valid sstate for %s (%s)" % (n, sq_data['hash'][tid]))
+
+ return found
+
diff --git a/bitbake/lib/bb/tests/runqueue-tests/classes/image.bbclass b/bitbake/lib/bb/tests/runqueue-tests/classes/image.bbclass
new file mode 100644
index 0000000..da9ff11
--- /dev/null
+++ b/bitbake/lib/bb/tests/runqueue-tests/classes/image.bbclass
@@ -0,0 +1,5 @@
+do_rootfs[recrdeptask] += "do_package_write_deb do_package_qa"
+do_rootfs[recrdeptask] += "do_package_write_ipk do_package_qa"
+do_rootfs[recrdeptask] += "do_package_write_rpm do_package_qa
+do_rootfs[recrdeptask] += "do_packagedata"
+do_rootfs[recrdeptask] += "do_populate_lic"
diff --git a/bitbake/lib/bb/tests/runqueue-tests/classes/native.bbclass b/bitbake/lib/bb/tests/runqueue-tests/classes/native.bbclass
new file mode 100644
index 0000000..7eaaee5
--- /dev/null
+++ b/bitbake/lib/bb/tests/runqueue-tests/classes/native.bbclass
@@ -0,0 +1,2 @@
+RECIPERDEPTASK = "do_populate_sysroot"
+do_populate_sysroot[rdeptask] = "${RECIPERDEPTASK}"
diff --git a/bitbake/lib/bb/tests/runqueue-tests/conf/bitbake.conf b/bitbake/lib/bb/tests/runqueue-tests/conf/bitbake.conf
new file mode 100644
index 0000000..5e451fc
--- /dev/null
+++ b/bitbake/lib/bb/tests/runqueue-tests/conf/bitbake.conf
@@ -0,0 +1,16 @@
+CACHE = "${TOPDIR}/cache"
+THISDIR = "${@os.path.dirname(d.getVar('FILE'))}"
+COREBASE := "${@os.path.normpath(os.path.dirname(d.getVar('FILE')+'/../../'))}"
+BBFILES = "${COREBASE}/recipes/*.bb"
+PROVIDES = "${PN}"
+PN = "${@bb.parse.vars_from_file(d.getVar('FILE', False),d)[0]}"
+PF = "${BB_CURRENT_MC}:${PN}"
+export PATH
+TMPDIR ??= "${TOPDIR}"
+STAMP = "${TMPDIR}/stamps/${PN}"
+T = "${TMPDIR}/workdir/${PN}/temp"
+BB_NUMBER_THREADS = "4"
+
+BB_HASHBASE_WHITELIST = "BB_CURRENT_MC BB_HASHSERVE TMPDIR TOPDIR SLOWTASKS SSTATEVALID FILE"
+
+include conf/multiconfig/${BB_CURRENT_MC}.conf
diff --git a/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc1.conf b/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc1.conf
new file mode 100644
index 0000000..ecf23e1
--- /dev/null
+++ b/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc1.conf
@@ -0,0 +1 @@
+TMPDIR = "${TOPDIR}/mc1/"
diff --git a/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc2.conf b/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc2.conf
new file mode 100644
index 0000000..eef338e
--- /dev/null
+++ b/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc2.conf
@@ -0,0 +1 @@
+TMPDIR = "${TOPDIR}/mc2/"
diff --git a/bitbake/lib/bb/tests/runqueue-tests/recipes/a1.bb b/bitbake/lib/bb/tests/runqueue-tests/recipes/a1.bb
new file mode 100644
index 0000000..e69de29
diff --git a/bitbake/lib/bb/tests/runqueue-tests/recipes/b1.bb b/bitbake/lib/bb/tests/runqueue-tests/recipes/b1.bb
new file mode 100644
index 0000000..c0b288e
--- /dev/null
+++ b/bitbake/lib/bb/tests/runqueue-tests/recipes/b1.bb
@@ -0,0 +1 @@
+DEPENDS = "a1"
\ No newline at end of file
diff --git a/bitbake/lib/bb/tests/runqueue-tests/recipes/c1.bb b/bitbake/lib/bb/tests/runqueue-tests/recipes/c1.bb
new file mode 100644
index 0000000..e69de29
diff --git a/bitbake/lib/bb/tests/runqueue-tests/recipes/d1.bb b/bitbake/lib/bb/tests/runqueue-tests/recipes/d1.bb
new file mode 100644
index 0000000..5ba1975
--- /dev/null
+++ b/bitbake/lib/bb/tests/runqueue-tests/recipes/d1.bb
@@ -0,0 +1,3 @@
+DEPENDS = "a1"
+
+do_package_setscene[depends] = "a1:do_populate_sysroot_setscene"
diff --git a/bitbake/lib/bb/tests/runqueue-tests/recipes/e1.bb b/bitbake/lib/bb/tests/runqueue-tests/recipes/e1.bb
new file mode 100644
index 0000000..1588bc8
--- /dev/null
+++ b/bitbake/lib/bb/tests/runqueue-tests/recipes/e1.bb
@@ -0,0 +1 @@
+DEPENDS = "b1"
\ No newline at end of file
diff --git a/bitbake/lib/bb/tests/runqueue.py b/bitbake/lib/bb/tests/runqueue.py
new file mode 100644
index 0000000..5e64391
--- /dev/null
+++ b/bitbake/lib/bb/tests/runqueue.py
@@ -0,0 +1,323 @@
+#
+# BitBake Tests for runqueue task processing
+#
+# Copyright (C) 2019 Richard Purdie
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+import unittest
+import bb
+import os
+import tempfile
+import subprocess
+import sys
+import time
+
+#
+# TODO:
+# Add tests on task ordering (X happens before Y after Z)
+#
+
+class RunQueueTests(unittest.TestCase):
+
+ alltasks = ['package', 'fetch', 'unpack', 'patch', 'prepare_recipe_sysroot', 'configure',
+ 'compile', 'install', 'packagedata', 'package_qa', 'package_write_rpm', 'package_write_ipk',
+ 'populate_sysroot', 'build']
+ a1_sstatevalid = "a1:do_package a1:do_package_qa a1:do_packagedata a1:do_package_write_ipk a1:do_package_write_rpm a1:do_populate_lic a1:do_populate_sysroot"
+ b1_sstatevalid = "b1:do_package b1:do_package_qa b1:do_packagedata b1:do_package_write_ipk b1:do_package_write_rpm b1:do_populate_lic b1:do_populate_sysroot"
+
+ def run_bitbakecmd(self, cmd, builddir, sstatevalid="", slowtasks="", extraenv=None, cleanup=False):
+ env = os.environ.copy()
+ env["BBPATH"] = os.path.realpath(os.path.join(os.path.dirname(__file__), "runqueue-tests"))
+ env["BB_ENV_EXTRAWHITE"] = "SSTATEVALID SLOWTASKS"
+ env["SSTATEVALID"] = sstatevalid
+ env["SLOWTASKS"] = slowtasks
+ if extraenv:
+ for k in extraenv:
+ env[k] = extraenv[k]
+ env["BB_ENV_EXTRAWHITE"] = env["BB_ENV_EXTRAWHITE"] + " " + k
+ try:
+ output = subprocess.check_output(cmd, env=env, stderr=subprocess.STDOUT,universal_newlines=True, cwd=builddir)
+ print(output)
+ except subprocess.CalledProcessError as e:
+ self.fail("Command %s failed with %s" % (cmd, e.output))
+ tasks = []
+ tasklog = builddir + "/task.log"
+ if os.path.exists(tasklog):
+ with open(tasklog, "r") as f:
+ tasks = [line.rstrip() for line in f]
+ if cleanup:
+ os.remove(tasklog)
+ return tasks
+
+ def test_no_setscenevalid(self):
+ with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
+ cmd = ["bitbake", "a1"]
+ sstatevalid = ""
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
+ expected = ['a1:' + x for x in self.alltasks]
+ self.assertEqual(set(tasks), set(expected))
+
+ def test_single_setscenevalid(self):
+ with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
+ cmd = ["bitbake", "a1"]
+ sstatevalid = "a1:do_package"
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
+ expected = ['a1:package_setscene', 'a1:fetch', 'a1:unpack', 'a1:patch', 'a1:prepare_recipe_sysroot', 'a1:configure',
+ 'a1:compile', 'a1:install', 'a1:packagedata', 'a1:package_qa', 'a1:package_write_rpm', 'a1:package_write_ipk',
+ 'a1:populate_sysroot', 'a1:build']
+ self.assertEqual(set(tasks), set(expected))
+
+ def test_intermediate_setscenevalid(self):
+ with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
+ cmd = ["bitbake", "a1"]
+ sstatevalid = "a1:do_package a1:do_populate_sysroot"
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
+ expected = ['a1:package_setscene', 'a1:packagedata', 'a1:package_qa', 'a1:package_write_rpm', 'a1:package_write_ipk',
+ 'a1:populate_sysroot_setscene', 'a1:build']
+ self.assertEqual(set(tasks), set(expected))
+
+ def test_intermediate_notcovered(self):
+ with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
+ cmd = ["bitbake", "a1"]
+ sstatevalid = "a1:do_package_qa a1:do_packagedata a1:do_package_write_ipk a1:do_package_write_rpm a1:do_populate_lic a1:do_populate_sysroot"
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
+ expected = ['a1:package_write_ipk_setscene', 'a1:package_write_rpm_setscene', 'a1:packagedata_setscene',
+ 'a1:package_qa_setscene', 'a1:build', 'a1:populate_sysroot_setscene']
+ self.assertEqual(set(tasks), set(expected))
+
+ def test_all_setscenevalid(self):
+ with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
+ cmd = ["bitbake", "a1"]
+ sstatevalid = self.a1_sstatevalid
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
+ expected = ['a1:package_write_ipk_setscene', 'a1:package_write_rpm_setscene', 'a1:packagedata_setscene',
+ 'a1:package_qa_setscene', 'a1:build', 'a1:populate_sysroot_setscene']
+ self.assertEqual(set(tasks), set(expected))
+
+ def test_no_settasks(self):
+ with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
+ cmd = ["bitbake", "a1", "-c", "patch"]
+ sstatevalid = self.a1_sstatevalid
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
+ expected = ['a1:fetch', 'a1:unpack', 'a1:patch']
+ self.assertEqual(set(tasks), set(expected))
+
+ def test_mix_covered_notcovered(self):
+ with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
+ cmd = ["bitbake", "a1:do_patch", "a1:do_populate_sysroot"]
+ sstatevalid = self.a1_sstatevalid
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
+ expected = ['a1:fetch', 'a1:unpack', 'a1:patch', 'a1:populate_sysroot_setscene']
+ self.assertEqual(set(tasks), set(expected))
+
+
+ # Test targets with intermediate setscene tasks alongside a target with no intermediate setscene tasks
+ def test_mixed_direct_tasks_setscene_tasks(self):
+ with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
+ cmd = ["bitbake", "c1:do_patch", "a1"]
+ sstatevalid = self.a1_sstatevalid
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
+ expected = ['c1:fetch', 'c1:unpack', 'c1:patch', 'a1:package_write_ipk_setscene', 'a1:package_write_rpm_setscene', 'a1:packagedata_setscene',
+ 'a1:package_qa_setscene', 'a1:build', 'a1:populate_sysroot_setscene']
+ self.assertEqual(set(tasks), set(expected))
+
+ # This test slows down the execution of do_package_setscene until after other real tasks have
+ # started running which tests for a bug where tasks were being lost from the buildable list of real
+ # tasks if they weren't in tasks_covered or tasks_notcovered
+ def test_slow_setscene(self):
+ with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
+ cmd = ["bitbake", "a1"]
+ sstatevalid = "a1:do_package"
+ slowtasks = "a1:package_setscene"
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid, slowtasks)
+ expected = ['a1:package_setscene', 'a1:fetch', 'a1:unpack', 'a1:patch', 'a1:prepare_recipe_sysroot', 'a1:configure',
+ 'a1:compile', 'a1:install', 'a1:packagedata', 'a1:package_qa', 'a1:package_write_rpm', 'a1:package_write_ipk',
+ 'a1:populate_sysroot', 'a1:build']
+ self.assertEqual(set(tasks), set(expected))
+
+ def test_setscenewhitelist(self):
+ with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
+ cmd = ["bitbake", "a1"]
+ extraenv = {
+ "BB_SETSCENE_ENFORCE" : "1",
+ "BB_SETSCENE_ENFORCE_WHITELIST" : "a1:do_package_write_rpm a1:do_build"
+ }
+ sstatevalid = "a1:do_package a1:do_package_qa a1:do_packagedata a1:do_package_write_ipk a1:do_populate_lic a1:do_populate_sysroot"
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid, extraenv=extraenv)
+ expected = ['a1:packagedata_setscene', 'a1:package_qa_setscene', 'a1:package_write_ipk_setscene',
+ 'a1:populate_sysroot_setscene', 'a1:package_setscene']
+ self.assertEqual(set(tasks), set(expected))
+
+ # Tests for problems with dependencies between setscene tasks
+ def test_no_setscenevalid_harddeps(self):
+ with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
+ cmd = ["bitbake", "d1"]
+ sstatevalid = ""
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
+ expected = ['a1:package', 'a1:fetch', 'a1:unpack', 'a1:patch', 'a1:prepare_recipe_sysroot', 'a1:configure',
+ 'a1:compile', 'a1:install', 'a1:packagedata', 'a1:package_write_rpm', 'a1:package_write_ipk',
+ 'a1:populate_sysroot', 'd1:package', 'd1:fetch', 'd1:unpack', 'd1:patch', 'd1:prepare_recipe_sysroot', 'd1:configure',
+ 'd1:compile', 'd1:install', 'd1:packagedata', 'd1:package_qa', 'd1:package_write_rpm', 'd1:package_write_ipk',
+ 'd1:populate_sysroot', 'd1:build']
+ self.assertEqual(set(tasks), set(expected))
+
+ def test_no_setscenevalid_withdeps(self):
+ with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
+ cmd = ["bitbake", "b1"]
+ sstatevalid = ""
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
+ expected = ['a1:' + x for x in self.alltasks] + ['b1:' + x for x in self.alltasks]
+ expected.remove('a1:build')
+ expected.remove('a1:package_qa')
+ self.assertEqual(set(tasks), set(expected))
+
+ def test_single_a1_setscenevalid_withdeps(self):
+ with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
+ cmd = ["bitbake", "b1"]
+ sstatevalid = "a1:do_package"
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
+ expected = ['a1:package_setscene', 'a1:fetch', 'a1:unpack', 'a1:patch', 'a1:prepare_recipe_sysroot', 'a1:configure',
+ 'a1:compile', 'a1:install', 'a1:packagedata', 'a1:package_write_rpm', 'a1:package_write_ipk',
+ 'a1:populate_sysroot'] + ['b1:' + x for x in self.alltasks]
+ self.assertEqual(set(tasks), set(expected))
+
+ def test_single_b1_setscenevalid_withdeps(self):
+ with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
+ cmd = ["bitbake", "b1"]
+ sstatevalid = "b1:do_package"
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
+ expected = ['a1:package', 'a1:fetch', 'a1:unpack', 'a1:patch', 'a1:prepare_recipe_sysroot', 'a1:configure',
+ 'a1:compile', 'a1:install', 'a1:packagedata', 'a1:package_write_rpm', 'a1:package_write_ipk',
+ 'a1:populate_sysroot', 'b1:package_setscene'] + ['b1:' + x for x in self.alltasks]
+ expected.remove('b1:package')
+ self.assertEqual(set(tasks), set(expected))
+
+ def test_intermediate_setscenevalid_withdeps(self):
+ with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
+ cmd = ["bitbake", "b1"]
+ sstatevalid = "a1:do_package a1:do_populate_sysroot b1:do_package"
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
+ expected = ['a1:package_setscene', 'a1:packagedata', 'a1:package_write_rpm', 'a1:package_write_ipk',
+ 'a1:populate_sysroot_setscene', 'b1:package_setscene'] + ['b1:' + x for x in self.alltasks]
+ expected.remove('b1:package')
+ self.assertEqual(set(tasks), set(expected))
+
+ def test_all_setscenevalid_withdeps(self):
+ with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
+ cmd = ["bitbake", "b1"]
+ sstatevalid = self.a1_sstatevalid + " " + self.b1_sstatevalid
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
+ expected = ['a1:package_write_ipk_setscene', 'a1:package_write_rpm_setscene', 'a1:packagedata_setscene',
+ 'b1:build', 'a1:populate_sysroot_setscene', 'b1:package_write_ipk_setscene', 'b1:package_write_rpm_setscene',
+ 'b1:packagedata_setscene', 'b1:package_qa_setscene', 'b1:populate_sysroot_setscene']
+ self.assertEqual(set(tasks), set(expected))
+
+ def test_multiconfig_setscene_optimise(self):
+ with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
+ extraenv = {
+ "BBMULTICONFIG" : "mc1 mc2",
+ "BB_SIGNATURE_HANDLER" : "basic"
+ }
+ cmd = ["bitbake", "b1", "mc:mc1:b1", "mc:mc2:b1"]
+ setscenetasks = ['package_write_ipk_setscene', 'package_write_rpm_setscene', 'packagedata_setscene',
+ 'populate_sysroot_setscene', 'package_qa_setscene']
+ sstatevalid = ""
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid, extraenv=extraenv)
+ expected = ['a1:' + x for x in self.alltasks] + ['b1:' + x for x in self.alltasks] + \
+ ['mc1:b1:' + x for x in setscenetasks] + ['mc1:a1:' + x for x in setscenetasks] + \
+ ['mc2:b1:' + x for x in setscenetasks] + ['mc2:a1:' + x for x in setscenetasks] + \
+ ['mc1:b1:build', 'mc2:b1:build']
+ for x in ['mc1:a1:package_qa_setscene', 'mc2:a1:package_qa_setscene', 'a1:build', 'a1:package_qa']:
+ expected.remove(x)
+ self.assertEqual(set(tasks), set(expected))
+
+
+ @unittest.skipIf(sys.version_info < (3, 5, 0), 'Python 3.5 or later required')
+ def test_hashserv_single(self):
+ with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
+ extraenv = {
+ "BB_HASHSERVE" : "auto",
+ "BB_SIGNATURE_HANDLER" : "TestEquivHash"
+ }
+ cmd = ["bitbake", "a1", "b1"]
+ setscenetasks = ['package_write_ipk_setscene', 'package_write_rpm_setscene', 'packagedata_setscene',
+ 'populate_sysroot_setscene', 'package_qa_setscene']
+ sstatevalid = ""
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid, extraenv=extraenv, cleanup=True)
+ expected = ['a1:' + x for x in self.alltasks] + ['b1:' + x for x in self.alltasks]
+ self.assertEqual(set(tasks), set(expected))
+ cmd = ["bitbake", "a1", "-c", "install", "-f"]
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid, extraenv=extraenv, cleanup=True)
+ expected = ['a1:install']
+ self.assertEqual(set(tasks), set(expected))
+ cmd = ["bitbake", "a1", "b1"]
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid, extraenv=extraenv, cleanup=True)
+ expected = ['a1:populate_sysroot', 'a1:package', 'a1:package_write_rpm_setscene', 'a1:packagedata_setscene',
+ 'a1:package_write_ipk_setscene', 'a1:package_qa_setscene']
+ self.assertEqual(set(tasks), set(expected))
+
+ self.shutdown(tempdir)
+
+ @unittest.skipIf(sys.version_info < (3, 5, 0), 'Python 3.5 or later required')
+ def test_hashserv_double(self):
+ with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
+ extraenv = {
+ "BB_HASHSERVE" : "auto",
+ "BB_SIGNATURE_HANDLER" : "TestEquivHash"
+ }
+ cmd = ["bitbake", "a1", "b1", "e1"]
+ setscenetasks = ['package_write_ipk_setscene', 'package_write_rpm_setscene', 'packagedata_setscene',
+ 'populate_sysroot_setscene', 'package_qa_setscene']
+ sstatevalid = ""
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid, extraenv=extraenv, cleanup=True)
+ expected = ['a1:' + x for x in self.alltasks] + ['b1:' + x for x in self.alltasks] + ['e1:' + x for x in self.alltasks]
+ self.assertEqual(set(tasks), set(expected))
+ cmd = ["bitbake", "a1", "b1", "-c", "install", "-fn"]
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid, extraenv=extraenv, cleanup=True)
+ cmd = ["bitbake", "e1"]
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid, extraenv=extraenv, cleanup=True)
+ expected = ['a1:package', 'a1:install', 'b1:package', 'b1:install', 'a1:populate_sysroot', 'b1:populate_sysroot',
+ 'a1:package_write_ipk_setscene', 'b1:packagedata_setscene', 'b1:package_write_rpm_setscene',
+ 'a1:package_write_rpm_setscene', 'b1:package_write_ipk_setscene', 'a1:packagedata_setscene']
+ self.assertEqual(set(tasks), set(expected))
+
+ self.shutdown(tempdir)
+
+ @unittest.skipIf(sys.version_info < (3, 5, 0), 'Python 3.5 or later required')
+ def test_hashserv_multiple_setscene(self):
+ # Runs e1:do_package_setscene twice
+ with tempfile.TemporaryDirectory(prefix="runqueuetest") as tempdir:
+ extraenv = {
+ "BB_HASHSERVE" : "auto",
+ "BB_SIGNATURE_HANDLER" : "TestEquivHash"
+ }
+ cmd = ["bitbake", "a1", "b1", "e1"]
+ setscenetasks = ['package_write_ipk_setscene', 'package_write_rpm_setscene', 'packagedata_setscene',
+ 'populate_sysroot_setscene', 'package_qa_setscene']
+ sstatevalid = ""
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid, extraenv=extraenv, cleanup=True)
+ expected = ['a1:' + x for x in self.alltasks] + ['b1:' + x for x in self.alltasks] + ['e1:' + x for x in self.alltasks]
+ self.assertEqual(set(tasks), set(expected))
+ cmd = ["bitbake", "a1", "b1", "-c", "install", "-fn"]
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid, extraenv=extraenv, cleanup=True)
+ cmd = ["bitbake", "e1"]
+ sstatevalid = "e1:do_package"
+ tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid, extraenv=extraenv, cleanup=True, slowtasks="a1:populate_sysroot b1:populate_sysroot")
+ expected = ['a1:package', 'a1:install', 'b1:package', 'b1:install', 'a1:populate_sysroot', 'b1:populate_sysroot',
+ 'a1:package_write_ipk_setscene', 'b1:packagedata_setscene', 'b1:package_write_rpm_setscene',
+ 'a1:package_write_rpm_setscene', 'b1:package_write_ipk_setscene', 'a1:packagedata_setscene',
+ 'e1:package_setscene']
+ self.assertEqual(set(tasks), set(expected))
+ for i in expected:
+ self.assertEqual(tasks.count(i), 1, "%s not in task list once" % i)
+
+ self.shutdown(tempdir)
+
+ def shutdown(self, tempdir):
+ # Wait for the hashserve socket to disappear else we'll see races with the tempdir cleanup
+ while os.path.exists(tempdir + "/hashserve.sock"):
+ time.sleep(0.5)
+
+
diff --git a/bitbake/lib/bb/tests/utils.py b/bitbake/lib/bb/tests/utils.py
index 2f4ccf3..f4adf1d 100644
--- a/bitbake/lib/bb/tests/utils.py
+++ b/bitbake/lib/bb/tests/utils.py
@@ -1,22 +1,9 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# BitBake Tests for utils.py
#
# Copyright (C) 2012 Richard Purdie
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+# SPDX-License-Identifier: GPL-2.0-only
#
import unittest
@@ -42,6 +29,14 @@ class VerCmpString(unittest.TestCase):
self.assertTrue(result < 0)
result = bb.utils.vercmp_string('1.1', '1.0+1.1-beta1')
self.assertTrue(result > 0)
+ result = bb.utils.vercmp_string('1a', '1a1')
+ self.assertTrue(result < 0)
+ result = bb.utils.vercmp_string('1a1', '1a')
+ self.assertTrue(result > 0)
+ result = bb.utils.vercmp_string('1.', '1.1')
+ self.assertTrue(result < 0)
+ result = bb.utils.vercmp_string('1.1', '1.')
+ self.assertTrue(result > 0)
def test_explode_dep_versions(self):
correctresult = {"foo" : ["= 1.10"]}
diff --git a/bitbake/lib/bb/tinfoil.py b/bitbake/lib/bb/tinfoil.py
index 368264f..0a1b913 100644
--- a/bitbake/lib/bb/tinfoil.py
+++ b/bitbake/lib/bb/tinfoil.py
@@ -4,18 +4,8 @@
# Copyright (C) 2011 Mentor Graphics Corporation
# Copyright (C) 2006-2012 Richard Purdie
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import logging
import os
diff --git a/bitbake/lib/bb/ui/__init__.py b/bitbake/lib/bb/ui/__init__.py
index a4805ed..4b7ac36 100644
--- a/bitbake/lib/bb/ui/__init__.py
+++ b/bitbake/lib/bb/ui/__init__.py
@@ -3,15 +3,5 @@
#
# Copyright (C) 2006-2007 Richard Purdie
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
diff --git a/bitbake/lib/bb/ui/buildinfohelper.py b/bitbake/lib/bb/ui/buildinfohelper.py
index 31323d2..5cbca97 100644
--- a/bitbake/lib/bb/ui/buildinfohelper.py
+++ b/bitbake/lib/bb/ui/buildinfohelper.py
@@ -3,18 +3,8 @@
#
# Copyright (C) 2013 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import sys
import bb
@@ -656,6 +646,9 @@ class ORMWrapper(object):
Target_Installed_Package.objects.create(target = target_obj, package = packagedict[p]['object'])
packagedeps_objs = []
+ pattern_so = re.compile(r'.*\.so(\.\d*)?$')
+ pattern_lib = re.compile(r'.*\-suffix(\d*)?$')
+ pattern_ko = re.compile(r'^kernel-module-.*')
for p in packagedict:
for (px,deptype) in packagedict[p]['depends']:
if deptype == 'depends':
@@ -664,6 +657,13 @@ class ORMWrapper(object):
tdeptype = Package_Dependency.TYPE_TRECOMMENDS
try:
+ # Skip known non-package objects like libraries and kernel modules
+ if pattern_so.match(px) or pattern_lib.match(px):
+ logger.info("Toaster does not add library file dependencies to packages (%s,%s)", p, px)
+ continue
+ if pattern_ko.match(px):
+ logger.info("Toaster does not add kernel module dependencies to packages (%s,%s)", p, px)
+ continue
packagedeps_objs.append(Package_Dependency(
package = packagedict[p]['object'],
depends_on = packagedict[px]['object'],
diff --git a/bitbake/lib/bb/ui/knotty.py b/bitbake/lib/bb/ui/knotty.py
index fa88e6c..35736ad 100644
--- a/bitbake/lib/bb/ui/knotty.py
+++ b/bitbake/lib/bb/ui/knotty.py
@@ -5,18 +5,8 @@
#
# Copyright (C) 2006-2012 Richard Purdie
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from __future__ import division
@@ -222,6 +212,23 @@ class TerminalFilter(object):
sys.stdout.flush()
self.footer_present = False
+ def elapsed(self, sec):
+ hrs = int(sec / 3600.0)
+ sec -= hrs * 3600
+ min = int(sec / 60.0)
+ sec -= min * 60
+ if hrs > 0:
+ return "%dh%dm%ds" % (hrs, min, sec)
+ elif min > 0:
+ return "%dm%ds" % (min, sec)
+ else:
+ return "%ds" % (sec)
+
+ def keepAlive(self, t):
+ if not self.cuu:
+ print("Bitbake still alive (%ds)" % t)
+ sys.stdout.flush()
+
def updateFooter(self):
if not self.cuu:
return
@@ -258,7 +265,7 @@ class TerminalFilter(object):
else:
start_time = activetasks[t].get("starttime", None)
if start_time:
- tasks.append("%s - %ds (pid %s)" % (activetasks[t]["title"], currenttime - start_time, t))
+ tasks.append("%s - %s (pid %s)" % (activetasks[t]["title"], self.elapsed(currenttime - start_time), t))
else:
tasks.append("%s (pid %s)" % (activetasks[t]["title"], t))
@@ -293,8 +300,8 @@ class TerminalFilter(object):
if start_time:
pbar.start_time = start_time
pbar.setmessage('%s:%s' % (tasknum, pbar.msg.split(':', 1)[1]))
+ pbar.setextra(rate)
if progress > -1:
- pbar.setextra(rate)
content = pbar.update(progress)
else:
content = pbar.update(1)
@@ -455,11 +462,17 @@ def main(server, eventHandler, params, tf = TerminalFilter):
warnings = 0
taskfailures = []
+ printinterval = 5000
+ lastprint = time.time()
+
termfilter = tf(main, helper, console, errconsole, format, params.options.quiet)
atexit.register(termfilter.finish)
while True:
try:
+ if (lastprint + printinterval) <= time.time():
+ termfilter.keepAlive(printinterval)
+ printinterval += 5000
event = eventHandler.waitEvent(0)
if event is None:
if main.shutdown > 1:
@@ -488,6 +501,8 @@ def main(server, eventHandler, params, tf = TerminalFilter):
continue
if isinstance(event, logging.LogRecord):
+ lastprint = time.time()
+ printinterval = 5000
if event.levelno >= format.ERROR:
errors = errors + 1
return_value = 1
@@ -645,7 +660,6 @@ def main(server, eventHandler, params, tf = TerminalFilter):
# ignore
if isinstance(event, (bb.event.BuildBase,
bb.event.MetadataEvent,
- bb.event.StampUpdate,
bb.event.ConfigParsed,
bb.event.MultiConfigParsed,
bb.event.RecipeParsed,
@@ -675,17 +689,27 @@ def main(server, eventHandler, params, tf = TerminalFilter):
if params.observe_only:
print("\nKeyboard Interrupt, exiting observer...")
main.shutdown = 2
- if not params.observe_only and main.shutdown == 1:
+
+ def state_force_shutdown():
print("\nSecond Keyboard Interrupt, stopping...\n")
_, error = server.runCommand(["stateForceShutdown"])
if error:
logger.error("Unable to cleanly stop: %s" % error)
+
+ if not params.observe_only and main.shutdown == 1:
+ state_force_shutdown()
+
if not params.observe_only and main.shutdown == 0:
print("\nKeyboard Interrupt, closing down...\n")
interrupted = True
- _, error = server.runCommand(["stateShutdown"])
- if error:
- logger.error("Unable to cleanly shutdown: %s" % error)
+ # Capture the second KeyboardInterrupt during stateShutdown is running
+ try:
+ _, error = server.runCommand(["stateShutdown"])
+ if error:
+ logger.error("Unable to cleanly shutdown: %s" % error)
+ except KeyboardInterrupt:
+ state_force_shutdown()
+
main.shutdown = main.shutdown + 1
pass
except Exception as e:
diff --git a/bitbake/lib/bb/ui/ncurses.py b/bitbake/lib/bb/ui/ncurses.py
index 8690c52..c422732 100644
--- a/bitbake/lib/bb/ui/ncurses.py
+++ b/bitbake/lib/bb/ui/ncurses.py
@@ -6,18 +6,8 @@
# Copyright (C) 2006 Michael 'Mickey' Lauer
# Copyright (C) 2006-2007 Richard Purdie
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
"""
We have the following windows:
diff --git a/bitbake/lib/bb/ui/taskexp.py b/bitbake/lib/bb/ui/taskexp.py
index 8305d70..50a943c 100644
--- a/bitbake/lib/bb/ui/taskexp.py
+++ b/bitbake/lib/bb/ui/taskexp.py
@@ -4,18 +4,8 @@
# Copyright (C) 2007 Ross Burton
# Copyright (C) 2007 - 2008 Richard Purdie
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import sys
import gi
diff --git a/bitbake/lib/bb/ui/toasterui.py b/bitbake/lib/bb/ui/toasterui.py
index 88cec37..51892c9 100644
--- a/bitbake/lib/bb/ui/toasterui.py
+++ b/bitbake/lib/bb/ui/toasterui.py
@@ -7,18 +7,8 @@
# Copyright (C) 2006-2012 Richard Purdie
# Copyright (C) 2013 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from __future__ import division
import time
diff --git a/bitbake/lib/bb/ui/uievent.py b/bitbake/lib/bb/ui/uievent.py
index 9542b91..fedb050 100644
--- a/bitbake/lib/bb/ui/uievent.py
+++ b/bitbake/lib/bb/ui/uievent.py
@@ -1,22 +1,9 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# Copyright (C) 2006 - 2007 Michael 'Mickey' Lauer
# Copyright (C) 2006 - 2007 Richard Purdie
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
"""
Use this class to fork off a thread to recieve event callbacks from the bitbake
diff --git a/bitbake/lib/bb/ui/uihelper.py b/bitbake/lib/bb/ui/uihelper.py
index 963c1ea..c8dd7df 100644
--- a/bitbake/lib/bb/ui/uihelper.py
+++ b/bitbake/lib/bb/ui/uihelper.py
@@ -1,21 +1,9 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
# Copyright (C) 2006 - 2007 Michael 'Mickey' Lauer
# Copyright (C) 2006 - 2007 Richard Purdie
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import bb.build
import time
@@ -52,7 +40,7 @@ class BBUIHelper:
self.running_pids.remove(event.pid)
self.failed_tasks.append( { 'title' : "%s %s" % (event._package, event._task)})
self.needUpdate = True
- elif isinstance(event, bb.runqueue.runQueueTaskStarted) or isinstance(event, bb.runqueue.sceneQueueTaskStarted):
+ elif isinstance(event, bb.runqueue.runQueueTaskStarted):
self.tasknumber_current = event.stats.completed + event.stats.active + event.stats.failed + 1
self.tasknumber_total = event.stats.total
self.needUpdate = True
diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
index 13bb5f2..d035949 100644
--- a/bitbake/lib/bb/utils.py
+++ b/bitbake/lib/bb/utils.py
@@ -1,23 +1,11 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
"""
BitBake Utility Functions
"""
# Copyright (C) 2004 Michael Lauer
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import re, fcntl, os, string, stat, shutil, time
import sys
@@ -121,6 +109,10 @@ def vercmp_part(a, b):
return -1
elif oa > ob:
return 1
+ elif ca is None:
+ return -1
+ elif cb is None:
+ return 1
elif ca < cb:
return -1
elif ca > cb:
@@ -402,7 +394,7 @@ def better_exec(code, context, text = None, realfile = "<code>", pythonexception
code = better_compile(code, realfile, realfile)
try:
exec(code, get_context(), context)
- except (bb.BBHandledException, bb.parse.SkipRecipe, bb.build.FuncFailed, bb.data_smart.ExpansionError):
+ except (bb.BBHandledException, bb.parse.SkipRecipe, bb.data_smart.ExpansionError):
# Error already shown so passthrough, no need for traceback
raise
except Exception as e:
@@ -685,7 +677,7 @@ def _check_unsafe_delete_path(path):
return True
return False
-def remove(path, recurse=False):
+def remove(path, recurse=False, ionice=False):
"""Equivalent to rm -f or rm -rf"""
if not path:
return
@@ -694,7 +686,10 @@ def remove(path, recurse=False):
if _check_unsafe_delete_path(path):
raise Exception('bb.utils.remove: called with dangerous path "%s" and recurse=True, refusing to delete!' % path)
# shutil.rmtree(name) would be ideal but its too slow
- subprocess.check_call(['rm', '-rf'] + glob.glob(path))
+ cmd = []
+ if ionice:
+ cmd = ['ionice', '-c', '3']
+ subprocess.check_call(cmd + ['rm', '-rf'] + glob.glob(path))
return
for name in glob.glob(path):
try:
@@ -703,20 +698,12 @@ def remove(path, recurse=False):
if exc.errno != errno.ENOENT:
raise
-def prunedir(topdir):
+def prunedir(topdir, ionice=False):
# Delete everything reachable from the directory named in 'topdir'.
# CAUTION: This is dangerous!
if _check_unsafe_delete_path(topdir):
raise Exception('bb.utils.prunedir: called with dangerous path "%s", refusing to delete!' % topdir)
- for root, dirs, files in os.walk(topdir, topdown = False):
- for name in files:
- os.remove(os.path.join(root, name))
- for name in dirs:
- if os.path.islink(os.path.join(root, name)):
- os.remove(os.path.join(root, name))
- else:
- os.rmdir(os.path.join(root, name))
- os.rmdir(topdir)
+ remove(topdir, recurse=True, ionice=ionice)
#
# Could also use return re.compile("(%s)" % "|".join(map(re.escape, suffixes))).sub(lambda mo: "", var)
@@ -726,8 +713,8 @@ def prune_suffix(var, suffixes, d):
# See if var ends with any of the suffixes listed and
# remove it if found
for suffix in suffixes:
- if var.endswith(suffix):
- return var.replace(suffix, "")
+ if suffix and var.endswith(suffix):
+ return var[:-len(suffix)]
return var
def mkdirhier(directory):
@@ -738,7 +725,7 @@ def mkdirhier(directory):
try:
os.makedirs(directory)
except OSError as e:
- if e.errno != errno.EEXIST:
+ if e.errno != errno.EEXIST or not os.path.isdir(directory):
raise e
def movefile(src, dest, newmtime = None, sstat = None):
@@ -796,7 +783,7 @@ def movefile(src, dest, newmtime = None, sstat = None):
os.rename(src, destpath)
renamefailed = 0
except Exception as e:
- if e[0] != errno.EXDEV:
+ if e.errno != errno.EXDEV:
# Some random error.
print("movefile: Failed to move", src, "to", dest, e)
return None
@@ -1505,6 +1492,8 @@ def ioprio_set(who, cls, value):
NR_ioprio_set = 251
elif _unamearch[0] == "i" and _unamearch[2:3] == "86":
NR_ioprio_set = 289
+ elif _unamearch == "aarch64":
+ NR_ioprio_set = 30
if NR_ioprio_set:
ioprio = value | (cls << IOPRIO_CLASS_SHIFT)
diff --git a/bitbake/lib/bblayers/__init__.py b/bitbake/lib/bblayers/__init__.py
index 3ad9513..4e7c09d 100644
--- a/bitbake/lib/bblayers/__init__.py
+++ b/bitbake/lib/bblayers/__init__.py
@@ -1,2 +1,6 @@
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)
diff --git a/bitbake/lib/bblayers/action.py b/bitbake/lib/bblayers/action.py
index a3f658f..d6459d6 100644
--- a/bitbake/lib/bblayers/action.py
+++ b/bitbake/lib/bblayers/action.py
@@ -1,3 +1,7 @@
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
import fnmatch
import logging
import os
diff --git a/bitbake/lib/bblayers/common.py b/bitbake/lib/bblayers/common.py
index 98515ce..c5657d3 100644
--- a/bitbake/lib/bblayers/common.py
+++ b/bitbake/lib/bblayers/common.py
@@ -1,3 +1,7 @@
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
import argparse
import logging
import os
diff --git a/bitbake/lib/bblayers/layerindex.py b/bitbake/lib/bblayers/layerindex.py
index 9f02a9d..57cd902 100644
--- a/bitbake/lib/bblayers/layerindex.py
+++ b/bitbake/lib/bblayers/layerindex.py
@@ -1,3 +1,7 @@
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
import layerindexlib
import argparse
@@ -28,7 +32,7 @@ class LayerIndexPlugin(ActionPlugin):
layerdir = os.path.join(repodir, subdir)
if not os.path.exists(repodir):
if fetch_layer:
- result = subprocess.call('git clone %s %s' % (url, repodir), shell = True)
+ result = subprocess.call(['git', 'clone', url, repodir])
if result:
logger.error("Failed to download %s" % url)
return None, None, None
diff --git a/bitbake/lib/bblayers/query.py b/bitbake/lib/bblayers/query.py
index 9294dfa..7db49c8 100644
--- a/bitbake/lib/bblayers/query.py
+++ b/bitbake/lib/bblayers/query.py
@@ -1,3 +1,7 @@
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
import collections
import fnmatch
import logging
@@ -42,7 +46,7 @@ layer, with the preferred version first. Note that skipped recipes that
are overlayed will also be listed, with a " (skipped)" suffix.
"""
- items_listed = self.list_recipes('Overlayed recipes', None, True, args.same_version, args.filenames, True, None)
+ items_listed = self.list_recipes('Overlayed recipes', None, True, args.same_version, args.filenames, False, True, None, False, None)
# Check for overlayed .bbclass files
classes = collections.defaultdict(list)
@@ -108,9 +112,9 @@ skipped recipes will also be listed, with a " (skipped)" suffix.
title = 'Matching recipes:'
else:
title = 'Available recipes:'
- self.list_recipes(title, args.pnspec, False, False, args.filenames, args.multiple, inheritlist)
+ self.list_recipes(title, args.pnspec, False, False, args.filenames, args.recipes_only, args.multiple, args.layer, args.bare, inheritlist)
- def list_recipes(self, title, pnspec, show_overlayed_only, show_same_ver_only, show_filenames, show_multi_provider_only, inherits):
+ def list_recipes(self, title, pnspec, show_overlayed_only, show_same_ver_only, show_filenames, show_recipes_only, show_multi_provider_only, selected_layer, bare, inherits):
if inherits:
bbpath = str(self.tinfoil.config_data.getVar('BBPATH'))
for classname in inherits:
@@ -140,24 +144,30 @@ skipped recipes will also be listed, with a " (skipped)" suffix.
preferred_versions[p] = (ver, fn)
def print_item(f, pn, ver, layer, ispref):
- if f in skiplist:
- skipped = ' (skipped)'
- else:
- skipped = ''
- if show_filenames:
- if ispref:
- logger.plain("%s%s", f, skipped)
+ if not selected_layer or layer == selected_layer:
+ if not bare and f in skiplist:
+ skipped = ' (skipped)'
else:
- logger.plain(" %s%s", f, skipped)
- else:
- if ispref:
- logger.plain("%s:", pn)
- logger.plain(" %s %s%s", layer.ljust(20), ver, skipped)
+ skipped = ''
+ if show_filenames:
+ if ispref:
+ logger.plain("%s%s", f, skipped)
+ else:
+ logger.plain(" %s%s", f, skipped)
+ elif show_recipes_only:
+ if pn not in show_unique_pn:
+ show_unique_pn.append(pn)
+ logger.plain("%s%s", pn, skipped)
+ else:
+ if ispref:
+ logger.plain("%s:", pn)
+ logger.plain(" %s %s%s", layer.ljust(20), ver, skipped)
global_inherit = (self.tinfoil.config_data.getVar('INHERIT') or "").split()
cls_re = re.compile('classes/')
preffiles = []
+ show_unique_pn = []
items_listed = False
for p in sorted(pkg_pn):
if pnspec:
@@ -489,8 +499,11 @@ NOTE: .bbappend files can impact the dependencies.
parser_show_recipes = self.add_command(sp, 'show-recipes', self.do_show_recipes)
parser_show_recipes.add_argument('-f', '--filenames', help='instead of the default formatting, list filenames of higher priority recipes with the ones they overlay indented underneath', action='store_true')
+ parser_show_recipes.add_argument('-r', '--recipes-only', help='instead of the default formatting, list recipes only', action='store_true')
parser_show_recipes.add_argument('-m', '--multiple', help='only list where multiple recipes (in the same layer or different layers) exist for the same recipe name', action='store_true')
parser_show_recipes.add_argument('-i', '--inherits', help='only list recipes that inherit the named class(es) - separate multiple classes using , (without spaces)', metavar='CLASS', default='')
+ parser_show_recipes.add_argument('-l', '--layer', help='only list recipes from the selected layer', default='')
+ parser_show_recipes.add_argument('-b', '--bare', help='output just names without the "(skipped)" marker', action='store_true')
parser_show_recipes.add_argument('pnspec', nargs='*', help='optional recipe name specification (wildcards allowed, enclose in quotes to avoid shell expansion)')
parser_show_appends = self.add_command(sp, 'show-appends', self.do_show_appends)
diff --git a/bitbake/lib/bs4/dammit.py b/bitbake/lib/bs4/dammit.py
index 68d419f..805aa90 100644
--- a/bitbake/lib/bs4/dammit.py
+++ b/bitbake/lib/bs4/dammit.py
@@ -45,9 +45,9 @@ except ImportError:
pass
xml_encoding_re = re.compile(
- '^<\?.*encoding=[\'"](.*?)[\'"].*\?>'.encode(), re.I)
+ r'^<\?.*encoding=[\'"](.*?)[\'"].*\?>'.encode(), re.I)
html_meta_re = re.compile(
- '<\s*meta[^>]+charset\s*=\s*["\']?([^>]*?)[ /;\'">]'.encode(), re.I)
+ r'<\s*meta[^>]+charset\s*=\s*["\']?([^>]*?)[ /;\'">]'.encode(), re.I)
class EntitySubstitution(object):
@@ -80,11 +80,11 @@ class EntitySubstitution(object):
">": "gt",
}
- BARE_AMPERSAND_OR_BRACKET = re.compile("([<>]|"
- "&(?!#\d+;|#x[0-9a-fA-F]+;|\w+;)"
- ")")
+ BARE_AMPERSAND_OR_BRACKET = re.compile(r"([<>]|"
+ r"&(?!#\d+;|#x[0-9a-fA-F]+;|\w+;)"
+ r")")
- AMPERSAND_OR_BRACKET = re.compile("([<>&])")
+ AMPERSAND_OR_BRACKET = re.compile(r"([<>&])")
@classmethod
def _substitute_html_entity(cls, matchobj):
diff --git a/bitbake/lib/bs4/element.py b/bitbake/lib/bs4/element.py
index 0e62c2e..3775a60 100644
--- a/bitbake/lib/bs4/element.py
+++ b/bitbake/lib/bs4/element.py
@@ -1,7 +1,7 @@
__license__ = "MIT"
from pdb import set_trace
-import collections
+import collections.abc
import re
import sys
import warnings
@@ -10,7 +10,7 @@ from bs4.dammit import EntitySubstitution
DEFAULT_OUTPUT_ENCODING = "utf-8"
PY3K = (sys.version_info[0] > 2)
-whitespace_re = re.compile("\s+")
+whitespace_re = re.compile(r"\s+")
def _alias(attr):
"""Alias one attribute name to another for backward compatibility"""
@@ -67,7 +67,7 @@ class ContentMetaAttributeValue(AttributeValueWithCharsetSubstitution):
The value of the 'content' attribute will be one of these objects.
"""
- CHARSET_RE = re.compile("((^|;)\s*charset=)([^;]*)", re.M)
+ CHARSET_RE = re.compile(r"((^|;)\s*charset=)([^;]*)", re.M)
def __new__(cls, original_value):
match = cls.CHARSET_RE.search(original_value)
@@ -155,7 +155,7 @@ class PageElement(object):
def format_string(self, s, formatter='minimal'):
"""Format the given string using the given formatter."""
- if not isinstance(formatter, collections.Callable):
+ if not isinstance(formatter, collections.abc.Callable):
formatter = self._formatter_for_name(formatter)
if formatter is None:
output = s
@@ -580,7 +580,7 @@ class PageElement(object):
# Methods for supporting CSS selectors.
- tag_name_re = re.compile('^[a-zA-Z0-9][-.a-zA-Z0-9:_]*$')
+ tag_name_re = re.compile(r'^[a-zA-Z0-9][-.a-zA-Z0-9:_]*$')
# /^([a-zA-Z0-9][-.a-zA-Z0-9:_]*)\[(\w+)([=~\|\^\$\*]?)=?"?([^\]"]*)"?\]$/
# \---------------------------/ \---/\-------------/ \-------/
@@ -1077,7 +1077,7 @@ class Tag(PageElement):
# First off, turn a string formatter into a function. This
# will stop the lookup from happening over and over again.
- if not isinstance(formatter, collections.Callable):
+ if not isinstance(formatter, collections.abc.Callable):
formatter = self._formatter_for_name(formatter)
attrs = []
@@ -1181,7 +1181,7 @@ class Tag(PageElement):
"""
# First off, turn a string formatter into a function. This
# will stop the lookup from happening over and over again.
- if not isinstance(formatter, collections.Callable):
+ if not isinstance(formatter, collections.abc.Callable):
formatter = self._formatter_for_name(formatter)
pretty_print = (indent_level is not None)
@@ -1364,7 +1364,7 @@ class Tag(PageElement):
if tag_name == '':
raise ValueError(
"A pseudo-class must be prefixed with a tag name.")
- pseudo_attributes = re.match('([a-zA-Z\d-]+)\(([a-zA-Z\d]+)\)', pseudo)
+ pseudo_attributes = re.match(r'([a-zA-Z\d-]+)\(([a-zA-Z\d]+)\)', pseudo)
found = []
if pseudo_attributes is None:
pseudo_type = pseudo
@@ -1562,7 +1562,7 @@ class SoupStrainer(object):
def _normalize_search_value(self, value):
# Leave it alone if it's a Unicode string, a callable, a
# regular expression, a boolean, or None.
- if (isinstance(value, str) or isinstance(value, collections.Callable) or hasattr(value, 'match')
+ if (isinstance(value, str) or isinstance(value, collections.abc.Callable) or hasattr(value, 'match')
or isinstance(value, bool) or value is None):
return value
@@ -1602,7 +1602,7 @@ class SoupStrainer(object):
markup = markup_name
markup_attrs = markup
call_function_with_tag_data = (
- isinstance(self.name, collections.Callable)
+ isinstance(self.name, collections.abc.Callable)
and not isinstance(markup_name, Tag))
if ((not self.name)
@@ -1688,7 +1688,7 @@ class SoupStrainer(object):
# True matches any non-None value.
return markup is not None
- if isinstance(match_against, collections.Callable):
+ if isinstance(match_against, collections.abc.Callable):
return match_against(markup)
# Custom callables take the tag as an argument, but all
diff --git a/bitbake/lib/hashserv/__init__.py b/bitbake/lib/hashserv/__init__.py
new file mode 100644
index 0000000..c331862
--- /dev/null
+++ b/bitbake/lib/hashserv/__init__.py
@@ -0,0 +1,93 @@
+# Copyright (C) 2018-2019 Garmin Ltd.
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+from contextlib import closing
+import re
+import sqlite3
+
+UNIX_PREFIX = "unix://"
+
+ADDR_TYPE_UNIX = 0
+ADDR_TYPE_TCP = 1
+
+
+def setup_database(database, sync=True):
+ db = sqlite3.connect(database)
+ db.row_factory = sqlite3.Row
+
+ with closing(db.cursor()) as cursor:
+ cursor.execute('''
+ CREATE TABLE IF NOT EXISTS tasks_v2 (
+ id INTEGER PRIMARY KEY AUTOINCREMENT,
+ method TEXT NOT NULL,
+ outhash TEXT NOT NULL,
+ taskhash TEXT NOT NULL,
+ unihash TEXT NOT NULL,
+ created DATETIME,
+
+ -- Optional fields
+ owner TEXT,
+ PN TEXT,
+ PV TEXT,
+ PR TEXT,
+ task TEXT,
+ outhash_siginfo TEXT,
+
+ UNIQUE(method, outhash, taskhash)
+ )
+ ''')
+ cursor.execute('PRAGMA journal_mode = WAL')
+ cursor.execute('PRAGMA synchronous = %s' % ('NORMAL' if sync else 'OFF'))
+
+ # Drop old indexes
+ cursor.execute('DROP INDEX IF EXISTS taskhash_lookup')
+ cursor.execute('DROP INDEX IF EXISTS outhash_lookup')
+
+ # Create new indexes
+ cursor.execute('CREATE INDEX IF NOT EXISTS taskhash_lookup_v2 ON tasks_v2 (method, taskhash, created)')
+ cursor.execute('CREATE INDEX IF NOT EXISTS outhash_lookup_v2 ON tasks_v2 (method, outhash)')
+
+ return db
+
+
+def parse_address(addr):
+ if addr.startswith(UNIX_PREFIX):
+ return (ADDR_TYPE_UNIX, (addr[len(UNIX_PREFIX):],))
+ else:
+ m = re.match(r'\[(?P<host>[^\]]*)\]:(?P<port>\d+)$', addr)
+ if m is not None:
+ host = m.group('host')
+ port = m.group('port')
+ else:
+ host, port = addr.split(':')
+
+ return (ADDR_TYPE_TCP, (host, int(port)))
+
+
+def create_server(addr, dbname, *, sync=True):
+ from . import server
+ db = setup_database(dbname, sync=sync)
+ s = server.Server(db)
+
+ (typ, a) = parse_address(addr)
+ if typ == ADDR_TYPE_UNIX:
+ s.start_unix_server(*a)
+ else:
+ s.start_tcp_server(*a)
+
+ return s
+
+
+def create_client(addr):
+ from . import client
+ c = client.Client()
+
+ (typ, a) = parse_address(addr)
+ if typ == ADDR_TYPE_UNIX:
+ c.connect_unix(*a)
+ else:
+ c.connect_tcp(*a)
+
+ return c
diff --git a/bitbake/lib/hashserv/client.py b/bitbake/lib/hashserv/client.py
new file mode 100644
index 0000000..f659566
--- /dev/null
+++ b/bitbake/lib/hashserv/client.py
@@ -0,0 +1,157 @@
+# Copyright (C) 2019 Garmin Ltd.
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+from contextlib import closing
+import json
+import logging
+import socket
+import os
+
+
+logger = logging.getLogger('hashserv.client')
+
+
+class HashConnectionError(Exception):
+ pass
+
+
+class Client(object):
+ MODE_NORMAL = 0
+ MODE_GET_STREAM = 1
+
+ def __init__(self):
+ self._socket = None
+ self.reader = None
+ self.writer = None
+ self.mode = self.MODE_NORMAL
+
+ def connect_tcp(self, address, port):
+ def connect_sock():
+ s = socket.create_connection((address, port))
+
+ s.setsockopt(socket.SOL_TCP, socket.TCP_NODELAY, 1)
+ s.setsockopt(socket.SOL_TCP, socket.TCP_QUICKACK, 1)
+ s.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
+ return s
+
+ self._connect_sock = connect_sock
+
+ def connect_unix(self, path):
+ def connect_sock():
+ s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+ # AF_UNIX has path length issues so chdir here to workaround
+ cwd = os.getcwd()
+ try:
+ os.chdir(os.path.dirname(path))
+ s.connect(os.path.basename(path))
+ finally:
+ os.chdir(cwd)
+ return s
+
+ self._connect_sock = connect_sock
+
+ def connect(self):
+ if self._socket is None:
+ self._socket = self._connect_sock()
+
+ self.reader = self._socket.makefile('r', encoding='utf-8')
+ self.writer = self._socket.makefile('w', encoding='utf-8')
+
+ self.writer.write('OEHASHEQUIV 1.0\n\n')
+ self.writer.flush()
+
+ # Restore mode if the socket is being re-created
+ cur_mode = self.mode
+ self.mode = self.MODE_NORMAL
+ self._set_mode(cur_mode)
+
+ return self._socket
+
+ def close(self):
+ if self._socket is not None:
+ self._socket.close()
+ self._socket = None
+ self.reader = None
+ self.writer = None
+
+ def _send_wrapper(self, proc):
+ count = 0
+ while True:
+ try:
+ self.connect()
+ return proc()
+ except (OSError, HashConnectionError, json.JSONDecodeError, UnicodeDecodeError) as e:
+ logger.warning('Error talking to server: %s' % e)
+ if count >= 3:
+ if not isinstance(e, HashConnectionError):
+ raise HashConnectionError(str(e))
+ raise e
+ self.close()
+ count += 1
+
+ def send_message(self, msg):
+ def proc():
+ self.writer.write('%s\n' % json.dumps(msg))
+ self.writer.flush()
+
+ l = self.reader.readline()
+ if not l:
+ raise HashConnectionError('Connection closed')
+
+ if not l.endswith('\n'):
+ raise HashConnectionError('Bad message %r' % message)
+
+ return json.loads(l)
+
+ return self._send_wrapper(proc)
+
+ def send_stream(self, msg):
+ def proc():
+ self.writer.write("%s\n" % msg)
+ self.writer.flush()
+ l = self.reader.readline()
+ if not l:
+ raise HashConnectionError('Connection closed')
+ return l.rstrip()
+
+ return self._send_wrapper(proc)
+
+ def _set_mode(self, new_mode):
+ if new_mode == self.MODE_NORMAL and self.mode == self.MODE_GET_STREAM:
+ r = self.send_stream('END')
+ if r != 'ok':
+ raise HashConnectionError('Bad response from server %r' % r)
+ elif new_mode == self.MODE_GET_STREAM and self.mode == self.MODE_NORMAL:
+ r = self.send_message({'get-stream': None})
+ if r != 'ok':
+ raise HashConnectionError('Bad response from server %r' % r)
+ elif new_mode != self.mode:
+ raise Exception('Undefined mode transition %r -> %r' % (self.mode, new_mode))
+
+ self.mode = new_mode
+
+ def get_unihash(self, method, taskhash):
+ self._set_mode(self.MODE_GET_STREAM)
+ r = self.send_stream('%s %s' % (method, taskhash))
+ if not r:
+ return None
+ return r
+
+ def report_unihash(self, taskhash, method, outhash, unihash, extra={}):
+ self._set_mode(self.MODE_NORMAL)
+ m = extra.copy()
+ m['taskhash'] = taskhash
+ m['method'] = method
+ m['outhash'] = outhash
+ m['unihash'] = unihash
+ return self.send_message({'report': m})
+
+ def get_stats(self):
+ self._set_mode(self.MODE_NORMAL)
+ return self.send_message({'get-stats': None})
+
+ def reset_stats(self):
+ self._set_mode(self.MODE_NORMAL)
+ return self.send_message({'reset-stats': None})
diff --git a/bitbake/lib/hashserv/server.py b/bitbake/lib/hashserv/server.py
new file mode 100644
index 0000000..0aff776
--- /dev/null
+++ b/bitbake/lib/hashserv/server.py
@@ -0,0 +1,414 @@
+# Copyright (C) 2019 Garmin Ltd.
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+from contextlib import closing
+from datetime import datetime
+import asyncio
+import json
+import logging
+import math
+import os
+import signal
+import socket
+import time
+
+logger = logging.getLogger('hashserv.server')
+
+
+class Measurement(object):
+ def __init__(self, sample):
+ self.sample = sample
+
+ def start(self):
+ self.start_time = time.perf_counter()
+
+ def end(self):
+ self.sample.add(time.perf_counter() - self.start_time)
+
+ def __enter__(self):
+ self.start()
+ return self
+
+ def __exit__(self, *args, **kwargs):
+ self.end()
+
+
+class Sample(object):
+ def __init__(self, stats):
+ self.stats = stats
+ self.num_samples = 0
+ self.elapsed = 0
+
+ def measure(self):
+ return Measurement(self)
+
+ def __enter__(self):
+ return self
+
+ def __exit__(self, *args, **kwargs):
+ self.end()
+
+ def add(self, elapsed):
+ self.num_samples += 1
+ self.elapsed += elapsed
+
+ def end(self):
+ if self.num_samples:
+ self.stats.add(self.elapsed)
+ self.num_samples = 0
+ self.elapsed = 0
+
+
+class Stats(object):
+ def __init__(self):
+ self.reset()
+
+ def reset(self):
+ self.num = 0
+ self.total_time = 0
+ self.max_time = 0
+ self.m = 0
+ self.s = 0
+ self.current_elapsed = None
+
+ def add(self, elapsed):
+ self.num += 1
+ if self.num == 1:
+ self.m = elapsed
+ self.s = 0
+ else:
+ last_m = self.m
+ self.m = last_m + (elapsed - last_m) / self.num
+ self.s = self.s + (elapsed - last_m) * (elapsed - self.m)
+
+ self.total_time += elapsed
+
+ if self.max_time < elapsed:
+ self.max_time = elapsed
+
+ def start_sample(self):
+ return Sample(self)
+
+ @property
+ def average(self):
+ if self.num == 0:
+ return 0
+ return self.total_time / self.num
+
+ @property
+ def stdev(self):
+ if self.num <= 1:
+ return 0
+ return math.sqrt(self.s / (self.num - 1))
+
+ def todict(self):
+ return {k: getattr(self, k) for k in ('num', 'total_time', 'max_time', 'average', 'stdev')}
+
+
+class ServerClient(object):
+ def __init__(self, reader, writer, db, request_stats):
+ self.reader = reader
+ self.writer = writer
+ self.db = db
+ self.request_stats = request_stats
+
+ async def process_requests(self):
+ try:
+ self.addr = self.writer.get_extra_info('peername')
+ logger.debug('Client %r connected' % (self.addr,))
+
+ # Read protocol and version
+ protocol = await self.reader.readline()
+ if protocol is None:
+ return
+
+ (proto_name, proto_version) = protocol.decode('utf-8').rstrip().split()
+ if proto_name != 'OEHASHEQUIV' or proto_version != '1.0':
+ return
+
+ # Read headers. Currently, no headers are implemented, so look for
+ # an empty line to signal the end of the headers
+ while True:
+ line = await self.reader.readline()
+ if line is None:
+ return
+
+ line = line.decode('utf-8').rstrip()
+ if not line:
+ break
+
+ # Handle messages
+ handlers = {
+ 'get': self.handle_get,
+ 'report': self.handle_report,
+ 'get-stream': self.handle_get_stream,
+ 'get-stats': self.handle_get_stats,
+ 'reset-stats': self.handle_reset_stats,
+ }
+
+ while True:
+ d = await self.read_message()
+ if d is None:
+ break
+
+ for k in handlers.keys():
+ if k in d:
+ logger.debug('Handling %s' % k)
+ if 'stream' in k:
+ await handlers[k](d[k])
+ else:
+ with self.request_stats.start_sample() as self.request_sample, \
+ self.request_sample.measure():
+ await handlers[k](d[k])
+ break
+ else:
+ logger.warning("Unrecognized command %r" % d)
+ break
+
+ await self.writer.drain()
+ finally:
+ self.writer.close()
+
+ def write_message(self, msg):
+ self.writer.write(('%s\n' % json.dumps(msg)).encode('utf-8'))
+
+ async def read_message(self):
+ l = await self.reader.readline()
+ if not l:
+ return None
+
+ try:
+ message = l.decode('utf-8')
+
+ if not message.endswith('\n'):
+ return None
+
+ return json.loads(message)
+ except (json.JSONDecodeError, UnicodeDecodeError) as e:
+ logger.error('Bad message from client: %r' % message)
+ raise e
+
+ async def handle_get(self, request):
+ method = request['method']
+ taskhash = request['taskhash']
+
+ row = self.query_equivalent(method, taskhash)
+ if row is not None:
+ logger.debug('Found equivalent task %s -> %s', (row['taskhash'], row['unihash']))
+ d = {k: row[k] for k in ('taskhash', 'method', 'unihash')}
+
+ self.write_message(d)
+ else:
+ self.write_message(None)
+
+ async def handle_get_stream(self, request):
+ self.write_message('ok')
+
+ while True:
+ l = await self.reader.readline()
+ if not l:
+ return
+
+ try:
+ # This inner loop is very sensitive and must be as fast as
+ # possible (which is why the request sample is handled manually
+ # instead of using 'with', and also why logging statements are
+ # commented out.
+ self.request_sample = self.request_stats.start_sample()
+ request_measure = self.request_sample.measure()
+ request_measure.start()
+
+ l = l.decode('utf-8').rstrip()
+ if l == 'END':
+ self.writer.write('ok\n'.encode('utf-8'))
+ return
+
+ (method, taskhash) = l.split()
+ #logger.debug('Looking up %s %s' % (method, taskhash))
+ row = self.query_equivalent(method, taskhash)
+ if row is not None:
+ msg = ('%s\n' % row['unihash']).encode('utf-8')
+ #logger.debug('Found equivalent task %s -> %s', (row['taskhash'], row['unihash']))
+ else:
+ msg = '\n'.encode('utf-8')
+
+ self.writer.write(msg)
+ finally:
+ request_measure.end()
+ self.request_sample.end()
+
+ await self.writer.drain()
+
+ async def handle_report(self, data):
+ with closing(self.db.cursor()) as cursor:
+ cursor.execute('''
+ -- Find tasks with a matching outhash (that is, tasks that
+ -- are equivalent)
+ SELECT taskhash, method, unihash FROM tasks_v2 WHERE method=:method AND outhash=:outhash
+
+ -- If there is an exact match on the taskhash, return it.
+ -- Otherwise return the oldest matching outhash of any
+ -- taskhash
+ ORDER BY CASE WHEN taskhash=:taskhash THEN 1 ELSE 2 END,
+ created ASC
+
+ -- Only return one row
+ LIMIT 1
+ ''', {k: data[k] for k in ('method', 'outhash', 'taskhash')})
+
+ row = cursor.fetchone()
+
+ # If no matching outhash was found, or one *was* found but it
+ # wasn't an exact match on the taskhash, a new entry for this
+ # taskhash should be added
+ if row is None or row['taskhash'] != data['taskhash']:
+ # If a row matching the outhash was found, the unihash for
+ # the new taskhash should be the same as that one.
+ # Otherwise the caller provided unihash is used.
+ unihash = data['unihash']
+ if row is not None:
+ unihash = row['unihash']
+
+ insert_data = {
+ 'method': data['method'],
+ 'outhash': data['outhash'],
+ 'taskhash': data['taskhash'],
+ 'unihash': unihash,
+ 'created': datetime.now()
+ }
+
+ for k in ('owner', 'PN', 'PV', 'PR', 'task', 'outhash_siginfo'):
+ if k in data:
+ insert_data[k] = data[k]
+
+ cursor.execute('''INSERT INTO tasks_v2 (%s) VALUES (%s)''' % (
+ ', '.join(sorted(insert_data.keys())),
+ ', '.join(':' + k for k in sorted(insert_data.keys()))),
+ insert_data)
+
+ self.db.commit()
+
+ logger.info('Adding taskhash %s with unihash %s',
+ data['taskhash'], unihash)
+
+ d = {
+ 'taskhash': data['taskhash'],
+ 'method': data['method'],
+ 'unihash': unihash
+ }
+ else:
+ d = {k: row[k] for k in ('taskhash', 'method', 'unihash')}
+
+ self.write_message(d)
+
+ async def handle_get_stats(self, request):
+ d = {
+ 'requests': self.request_stats.todict(),
+ }
+
+ self.write_message(d)
+
+ async def handle_reset_stats(self, request):
+ d = {
+ 'requests': self.request_stats.todict(),
+ }
+
+ self.request_stats.reset()
+ self.write_message(d)
+
+ def query_equivalent(self, method, taskhash):
+ # This is part of the inner loop and must be as fast as possible
+ try:
+ cursor = self.db.cursor()
+ cursor.execute('SELECT taskhash, method, unihash FROM tasks_v2 WHERE method=:method AND taskhash=:taskhash ORDER BY created ASC LIMIT 1',
+ {'method': method, 'taskhash': taskhash})
+ return cursor.fetchone()
+ except:
+ cursor.close()
+
+
+class Server(object):
+ def __init__(self, db, loop=None):
+ self.request_stats = Stats()
+ self.db = db
+
+ if loop is None:
+ self.loop = asyncio.new_event_loop()
+ self.close_loop = True
+ else:
+ self.loop = loop
+ self.close_loop = False
+
+ self._cleanup_socket = None
+
+ def start_tcp_server(self, host, port):
+ self.server = self.loop.run_until_complete(
+ asyncio.start_server(self.handle_client, host, port, loop=self.loop)
+ )
+
+ for s in self.server.sockets:
+ logger.info('Listening on %r' % (s.getsockname(),))
+ # Newer python does this automatically. Do it manually here for
+ # maximum compatibility
+ s.setsockopt(socket.SOL_TCP, socket.TCP_NODELAY, 1)
+ s.setsockopt(socket.SOL_TCP, socket.TCP_QUICKACK, 1)
+
+ name = self.server.sockets[0].getsockname()
+ if self.server.sockets[0].family == socket.AF_INET6:
+ self.address = "[%s]:%d" % (name[0], name[1])
+ else:
+ self.address = "%s:%d" % (name[0], name[1])
+
+ def start_unix_server(self, path):
+ def cleanup():
+ os.unlink(path)
+
+ cwd = os.getcwd()
+ try:
+ # Work around path length limits in AF_UNIX
+ os.chdir(os.path.dirname(path))
+ self.server = self.loop.run_until_complete(
+ asyncio.start_unix_server(self.handle_client, os.path.basename(path), loop=self.loop)
+ )
+ finally:
+ os.chdir(cwd)
+
+ logger.info('Listening on %r' % path)
+
+ self._cleanup_socket = cleanup
+ self.address = "unix://%s" % os.path.abspath(path)
+
+ async def handle_client(self, reader, writer):
+ # writer.transport.set_write_buffer_limits(0)
+ try:
+ client = ServerClient(reader, writer, self.db, self.request_stats)
+ await client.process_requests()
+ except Exception as e:
+ import traceback
+ logger.error('Error from client: %s' % str(e), exc_info=True)
+ traceback.print_exc()
+ writer.close()
+ logger.info('Client disconnected')
+
+ def serve_forever(self):
+ def signal_handler():
+ self.loop.stop()
+
+ self.loop.add_signal_handler(signal.SIGTERM, signal_handler)
+
+ try:
+ self.loop.run_forever()
+ except KeyboardInterrupt:
+ pass
+
+ self.server.close()
+ self.loop.run_until_complete(self.server.wait_closed())
+ logger.info('Server shutting down')
+
+ if self.close_loop:
+ self.loop.close()
+
+ if self._cleanup_socket is not None:
+ self._cleanup_socket()
diff --git a/bitbake/lib/hashserv/tests.py b/bitbake/lib/hashserv/tests.py
new file mode 100644
index 0000000..a5472a9
--- /dev/null
+++ b/bitbake/lib/hashserv/tests.py
@@ -0,0 +1,142 @@
+#! /usr/bin/env python3
+#
+# Copyright (C) 2018-2019 Garmin Ltd.
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+from . import create_server, create_client
+import hashlib
+import logging
+import multiprocessing
+import sys
+import tempfile
+import threading
+import unittest
+
+
+class TestHashEquivalenceServer(object):
+ METHOD = 'TestMethod'
+
+ def _run_server(self):
+ # logging.basicConfig(level=logging.DEBUG, filename='bbhashserv.log', filemode='w',
+ # format='%(levelname)s %(filename)s:%(lineno)d %(message)s')
+ self.server.serve_forever()
+
+ def setUp(self):
+ if sys.version_info < (3, 5, 0):
+ self.skipTest('Python 3.5 or later required')
+
+ self.temp_dir = tempfile.TemporaryDirectory(prefix='bb-hashserv')
+ self.dbfile = os.path.join(self.temp_dir.name, 'db.sqlite')
+
+ self.server = create_server(self.get_server_addr(), self.dbfile)
+ self.server_thread = multiprocessing.Process(target=self._run_server)
+ self.server_thread.start()
+ self.client = create_client(self.server.address)
+
+ def tearDown(self):
+ # Shutdown server
+ s = getattr(self, 'server', None)
+ if s is not None:
+ self.server_thread.terminate()
+ self.server_thread.join()
+ self.client.close()
+ self.temp_dir.cleanup()
+
+ def test_create_hash(self):
+ # Simple test that hashes can be created
+ taskhash = '35788efcb8dfb0a02659d81cf2bfd695fb30faf9'
+ outhash = '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5970796f'
+ unihash = 'f46d3fbb439bd9b921095da657a4de906510d2cd'
+
+ result = self.client.get_unihash(self.METHOD, taskhash)
+ self.assertIsNone(result, msg='Found unexpected task, %r' % result)
+
+ result = self.client.report_unihash(taskhash, self.METHOD, outhash, unihash)
+ self.assertEqual(result['unihash'], unihash, 'Server returned bad unihash')
+
+ def test_create_equivalent(self):
+ # Tests that a second reported task with the same outhash will be
+ # assigned the same unihash
+ taskhash = '53b8dce672cb6d0c73170be43f540460bfc347b4'
+ outhash = '5a9cb1649625f0bf41fc7791b635cd9c2d7118c7f021ba87dcd03f72b67ce7a8'
+ unihash = 'f37918cc02eb5a520b1aff86faacbc0a38124646'
+
+ result = self.client.report_unihash(taskhash, self.METHOD, outhash, unihash)
+ self.assertEqual(result['unihash'], unihash, 'Server returned bad unihash')
+
+ # Report a different task with the same outhash. The returned unihash
+ # should match the first task
+ taskhash2 = '3bf6f1e89d26205aec90da04854fbdbf73afe6b4'
+ unihash2 = 'af36b199320e611fbb16f1f277d3ee1d619ca58b'
+ result = self.client.report_unihash(taskhash2, self.METHOD, outhash, unihash2)
+ self.assertEqual(result['unihash'], unihash, 'Server returned bad unihash')
+
+ def test_duplicate_taskhash(self):
+ # Tests that duplicate reports of the same taskhash with different
+ # outhash & unihash always return the unihash from the first reported
+ # taskhash
+ taskhash = '8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a'
+ outhash = 'afe240a439959ce86f5e322f8c208e1fedefea9e813f2140c81af866cc9edf7e'
+ unihash = '218e57509998197d570e2c98512d0105985dffc9'
+ self.client.report_unihash(taskhash, self.METHOD, outhash, unihash)
+
+ result = self.client.get_unihash(self.METHOD, taskhash)
+ self.assertEqual(result, unihash)
+
+ outhash2 = '0904a7fe3dc712d9fd8a74a616ddca2a825a8ee97adf0bd3fc86082c7639914d'
+ unihash2 = 'ae9a7d252735f0dafcdb10e2e02561ca3a47314c'
+ self.client.report_unihash(taskhash, self.METHOD, outhash2, unihash2)
+
+ result = self.client.get_unihash(self.METHOD, taskhash)
+ self.assertEqual(result, unihash)
+
+ outhash3 = '77623a549b5b1a31e3732dfa8fe61d7ce5d44b3370f253c5360e136b852967b4'
+ unihash3 = '9217a7d6398518e5dc002ed58f2cbbbc78696603'
+ self.client.report_unihash(taskhash, self.METHOD, outhash3, unihash3)
+
+ result = self.client.get_unihash(self.METHOD, taskhash)
+ self.assertEqual(result, unihash)
+
+ def test_stress(self):
+ def query_server(failures):
+ client = Client(self.server.address)
+ try:
+ for i in range(1000):
+ taskhash = hashlib.sha256()
+ taskhash.update(str(i).encode('utf-8'))
+ taskhash = taskhash.hexdigest()
+ result = client.get_unihash(self.METHOD, taskhash)
+ if result != taskhash:
+ failures.append("taskhash mismatch: %s != %s" % (result, taskhash))
+ finally:
+ client.close()
+
+ # Report hashes
+ for i in range(1000):
+ taskhash = hashlib.sha256()
+ taskhash.update(str(i).encode('utf-8'))
+ taskhash = taskhash.hexdigest()
+ self.client.report_unihash(taskhash, self.METHOD, taskhash, taskhash)
+
+ failures = []
+ threads = [threading.Thread(target=query_server, args=(failures,)) for t in range(100)]
+
+ for t in threads:
+ t.start()
+
+ for t in threads:
+ t.join()
+
+ self.assertFalse(failures)
+
+
+class TestHashEquivalenceUnixServer(TestHashEquivalenceServer, unittest.TestCase):
+ def get_server_addr(self):
+ return "unix://" + os.path.join(self.temp_dir.name, 'sock')
+
+
+class TestHashEquivalenceTCPServer(TestHashEquivalenceServer, unittest.TestCase):
+ def get_server_addr(self):
+ return "localhost:0"
diff --git a/bitbake/lib/layerindexlib/__init__.py b/bitbake/lib/layerindexlib/__init__.py
index cb79cb3..77196b4 100644
--- a/bitbake/lib/layerindexlib/__init__.py
+++ b/bitbake/lib/layerindexlib/__init__.py
@@ -1,17 +1,7 @@
# Copyright (C) 2016-2018 Wind River Systems, Inc.
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
-# See the GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program; if not, write to the Free Software
-# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
import datetime
@@ -386,7 +376,7 @@ layerBranches set. If not, they are effectively blank.'''
invalid.append(name)
- def _resolve_dependencies(layerbranches, ignores, dependencies, invalid):
+ def _resolve_dependencies(layerbranches, ignores, dependencies, invalid, processed=None):
for layerbranch in layerbranches:
if ignores and layerbranch.layer.name in ignores:
continue
@@ -398,6 +388,13 @@ layerBranches set. If not, they are effectively blank.'''
if ignores and deplayerbranch.layer.name in ignores:
continue
+ # Since this is depth first, we need to know what we're currently processing
+ # in order to avoid infinite recursion on a loop.
+ if processed and deplayerbranch.layer.name in processed:
+ # We have found a recursion...
+ logger.warning('Circular layer dependency found: %s -> %s' % (processed, deplayerbranch.layer.name))
+ continue
+
# This little block is why we can't re-use the LayerIndexObj version,
# we must be able to satisfy each dependencies across layer indexes and
# use the layer index order for priority. (r stands for replacement below)
@@ -421,7 +418,17 @@ layerBranches set. If not, they are effectively blank.'''
# New dependency, we need to resolve it now... depth-first
if deplayerbranch.layer.name not in dependencies:
- (dependencies, invalid) = _resolve_dependencies([deplayerbranch], ignores, dependencies, invalid)
+ # Avoid recursion on this branch.
+ # We copy so we don't end up polluting the depth-first branch with other
+ # branches. Duplication between individual branches IS expected and
+ # handled by 'dependencies' processing.
+ if not processed:
+ local_processed = []
+ else:
+ local_processed = processed.copy()
+ local_processed.append(deplayerbranch.layer.name)
+
+ (dependencies, invalid) = _resolve_dependencies([deplayerbranch], ignores, dependencies, invalid, local_processed)
if deplayerbranch.layer.name not in dependencies:
dependencies[deplayerbranch.layer.name] = [deplayerbranch, layerdependency]
diff --git a/bitbake/lib/layerindexlib/cooker.py b/bitbake/lib/layerindexlib/cooker.py
index 848f0e2..604a961 100644
--- a/bitbake/lib/layerindexlib/cooker.py
+++ b/bitbake/lib/layerindexlib/cooker.py
@@ -1,17 +1,7 @@
# Copyright (C) 2016-2018 Wind River Systems, Inc.
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
-# See the GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program; if not, write to the Free Software
-# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
import logging
import json
diff --git a/bitbake/lib/layerindexlib/plugin.py b/bitbake/lib/layerindexlib/plugin.py
index 92a2e97..7015a1a 100644
--- a/bitbake/lib/layerindexlib/plugin.py
+++ b/bitbake/lib/layerindexlib/plugin.py
@@ -1,18 +1,7 @@
# Copyright (C) 2016-2018 Wind River Systems, Inc.
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
-# See the GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program; if not, write to the Free Software
-# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-
# The file contains:
# LayerIndex exceptions
# Plugin base class
diff --git a/bitbake/lib/layerindexlib/restapi.py b/bitbake/lib/layerindexlib/restapi.py
index d08eb20..21fd144 100644
--- a/bitbake/lib/layerindexlib/restapi.py
+++ b/bitbake/lib/layerindexlib/restapi.py
@@ -1,17 +1,7 @@
# Copyright (C) 2016-2018 Wind River Systems, Inc.
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
-# See the GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program; if not, write to the Free Software
-# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
import logging
import json
diff --git a/bitbake/lib/layerindexlib/tests/common.py b/bitbake/lib/layerindexlib/tests/common.py
index 22a5458..077382f 100644
--- a/bitbake/lib/layerindexlib/tests/common.py
+++ b/bitbake/lib/layerindexlib/tests/common.py
@@ -1,17 +1,7 @@
# Copyright (C) 2017-2018 Wind River Systems, Inc.
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
-# See the GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program; if not, write to the Free Software
-# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
import unittest
import tempfile
diff --git a/bitbake/lib/layerindexlib/tests/cooker.py b/bitbake/lib/layerindexlib/tests/cooker.py
index fdbf091..1fa102e 100644
--- a/bitbake/lib/layerindexlib/tests/cooker.py
+++ b/bitbake/lib/layerindexlib/tests/cooker.py
@@ -1,17 +1,7 @@
# Copyright (C) 2018 Wind River Systems, Inc.
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
-# See the GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program; if not, write to the Free Software
-# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
import unittest
import tempfile
diff --git a/bitbake/lib/layerindexlib/tests/layerindexobj.py b/bitbake/lib/layerindexlib/tests/layerindexobj.py
index e2fbb95..0c5ec88 100644
--- a/bitbake/lib/layerindexlib/tests/layerindexobj.py
+++ b/bitbake/lib/layerindexlib/tests/layerindexobj.py
@@ -1,17 +1,7 @@
# Copyright (C) 2017-2018 Wind River Systems, Inc.
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
-# See the GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program; if not, write to the Free Software
-# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
import unittest
import tempfile
diff --git a/bitbake/lib/layerindexlib/tests/restapi.py b/bitbake/lib/layerindexlib/tests/restapi.py
index 5876695..6d8dc00 100644
--- a/bitbake/lib/layerindexlib/tests/restapi.py
+++ b/bitbake/lib/layerindexlib/tests/restapi.py
@@ -1,17 +1,7 @@
# Copyright (C) 2017-2018 Wind River Systems, Inc.
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
-# See the GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program; if not, write to the Free Software
-# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
import unittest
import tempfile
diff --git a/bitbake/lib/progressbar/__init__.py b/bitbake/lib/progressbar/__init__.py
index fbab744..c545a62 100644
--- a/bitbake/lib/progressbar/__init__.py
+++ b/bitbake/lib/progressbar/__init__.py
@@ -4,6 +4,8 @@
# progressbar - Text progress bar library for Python.
# Copyright (c) 2005 Nilton Volpato
#
+# SPDX-License-Identifier: LGPL-2.1-or-later OR BSD-3-Clause-Clear
+#
# This library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
# License as published by the Free Software Foundation; either
diff --git a/bitbake/lib/progressbar/compat.py b/bitbake/lib/progressbar/compat.py
index a39f4a1..9804e0b 100644
--- a/bitbake/lib/progressbar/compat.py
+++ b/bitbake/lib/progressbar/compat.py
@@ -3,6 +3,8 @@
# progressbar - Text progress bar library for Python.
# Copyright (c) 2005 Nilton Volpato
#
+# SPDX-License-Identifier: LGPL-2.1-or-later OR BSD-3-Clause-Clear
+#
# This library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
# License as published by the Free Software Foundation; either
diff --git a/bitbake/lib/progressbar/progressbar.py b/bitbake/lib/progressbar/progressbar.py
index 2873ad6..e2b6ba1 100644
--- a/bitbake/lib/progressbar/progressbar.py
+++ b/bitbake/lib/progressbar/progressbar.py
@@ -5,6 +5,8 @@
#
# (With some small changes after importing into BitBake)
#
+# SPDX-License-Identifier: LGPL-2.1-or-later OR BSD-3-Clause-Clear
+#
# This library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
# License as published by the Free Software Foundation; either
diff --git a/bitbake/lib/progressbar/widgets.py b/bitbake/lib/progressbar/widgets.py
index 77285ca..0772aa5 100644
--- a/bitbake/lib/progressbar/widgets.py
+++ b/bitbake/lib/progressbar/widgets.py
@@ -3,6 +3,8 @@
# progressbar - Text progress bar library for Python.
# Copyright (c) 2005 Nilton Volpato
#
+# SPDX-License-Identifier: LGPL-2.1-or-later OR BSD-3-Clause-Clear
+#
# This library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
# License as published by the Free Software Foundation; either
diff --git a/bitbake/lib/prserv/__init__.py b/bitbake/lib/prserv/__init__.py
index c3cb73a..9961040 100644
--- a/bitbake/lib/prserv/__init__.py
+++ b/bitbake/lib/prserv/__init__.py
@@ -1,3 +1,7 @@
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
__version__ = "1.0.0"
import os, time
diff --git a/bitbake/lib/prserv/db.py b/bitbake/lib/prserv/db.py
index 495d09f..117d8c0 100644
--- a/bitbake/lib/prserv/db.py
+++ b/bitbake/lib/prserv/db.py
@@ -1,3 +1,7 @@
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
import logging
import os.path
import errno
@@ -253,7 +257,7 @@ class PRData(object):
self.connection=sqlite3.connect(self.filename, isolation_level="EXCLUSIVE", check_same_thread = False)
self.connection.row_factory=sqlite3.Row
self.connection.execute("pragma synchronous = off;")
- self.connection.execute("PRAGMA journal_mode = WAL;")
+ self.connection.execute("PRAGMA journal_mode = MEMORY;")
self._tables={}
def disconnect(self):
diff --git a/bitbake/lib/prserv/serv.py b/bitbake/lib/prserv/serv.py
index 6a99728..be3acec 100644
--- a/bitbake/lib/prserv/serv.py
+++ b/bitbake/lib/prserv/serv.py
@@ -1,3 +1,7 @@
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
import os,sys,logging
import signal, time
from xmlrpc.server import SimpleXMLRPCServer, SimpleXMLRPCRequestHandler
diff --git a/bitbake/lib/pyinotify.py b/bitbake/lib/pyinotify.py
index 4eb03b0..1528a22 100644
--- a/bitbake/lib/pyinotify.py
+++ b/bitbake/lib/pyinotify.py
@@ -1,25 +1,9 @@
-#!/usr/bin/env python
-
+#
# pyinotify.py - python interface to inotify
# Copyright (c) 2005-2015 Sebastien Martini <seb@dbzteam.org>
#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
+# SPDX-License-Identifier: MIT
#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-# THE SOFTWARE.
"""
pyinotify
diff --git a/bitbake/lib/toaster/bldcollector/admin.py b/bitbake/lib/toaster/bldcollector/admin.py
index 1f2e07f..feaa888 100644
--- a/bitbake/lib/toaster/bldcollector/admin.py
+++ b/bitbake/lib/toaster/bldcollector/admin.py
@@ -1,3 +1,7 @@
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
from django.contrib import admin
from orm.models import BitbakeVersion, Release, ToasterSetting, Layer_Version
from django import forms
diff --git a/bitbake/lib/toaster/bldcollector/urls.py b/bitbake/lib/toaster/bldcollector/urls.py
index 888175d..8eb1e34 100644
--- a/bitbake/lib/toaster/bldcollector/urls.py
+++ b/bitbake/lib/toaster/bldcollector/urls.py
@@ -3,19 +3,8 @@
#
# Copyright (C) 2014-2017 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
from django.conf.urls import include, url
diff --git a/bitbake/lib/toaster/bldcollector/views.py b/bitbake/lib/toaster/bldcollector/views.py
index f32fa4d..c708b41 100644
--- a/bitbake/lib/toaster/bldcollector/views.py
+++ b/bitbake/lib/toaster/bldcollector/views.py
@@ -3,18 +3,8 @@
#
# Copyright (C) 2014 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from django.views.decorators.cache import cache_control
from django.core.urlresolvers import reverse
diff --git a/bitbake/lib/toaster/bldcontrol/admin.py b/bitbake/lib/toaster/bldcontrol/admin.py
index fcbe5f5..e85c30e 100644
--- a/bitbake/lib/toaster/bldcontrol/admin.py
+++ b/bitbake/lib/toaster/bldcontrol/admin.py
@@ -1,3 +1,7 @@
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
from django.contrib import admin
from django.contrib.admin.filters import RelatedFieldListFilter
from .models import BuildEnvironment
diff --git a/bitbake/lib/toaster/bldcontrol/bbcontroller.py b/bitbake/lib/toaster/bldcontrol/bbcontroller.py
index 5195600..301df18 100644
--- a/bitbake/lib/toaster/bldcontrol/bbcontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/bbcontroller.py
@@ -1,24 +1,10 @@
#
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
-#
# BitBake Toaster Implementation
#
# Copyright (C) 2014 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
import os
import sys
diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
index 9490635..39ea736 100644
--- a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -1,24 +1,10 @@
#
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
-#
# BitBake Toaster Implementation
#
# Copyright (C) 2014 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
import os
import sys
diff --git a/bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py b/bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py
index 14298d9..fe2c4dc 100644
--- a/bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py
+++ b/bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py
@@ -1,3 +1,7 @@
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
from django.core.management.base import BaseCommand, CommandError
from django.db import transaction
diff --git a/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py b/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py
index 6a55dd4..50ec409 100644
--- a/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py
+++ b/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py
@@ -1,3 +1,7 @@
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
from django.core.management.base import BaseCommand
from django.db import transaction
from django.db.models import Q
diff --git a/bitbake/lib/toaster/bldcontrol/models.py b/bitbake/lib/toaster/bldcontrol/models.py
index 409614b..bcffcf5 100644
--- a/bitbake/lib/toaster/bldcontrol/models.py
+++ b/bitbake/lib/toaster/bldcontrol/models.py
@@ -1,3 +1,7 @@
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
from __future__ import unicode_literals
from django.db import models
from django.core.validators import MaxValueValidator, MinValueValidator
diff --git a/bitbake/lib/toaster/bldcontrol/views.py b/bitbake/lib/toaster/bldcontrol/views.py
index 60f00ef..286d88b 100644
--- a/bitbake/lib/toaster/bldcontrol/views.py
+++ b/bitbake/lib/toaster/bldcontrol/views.py
@@ -1 +1,5 @@
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
# Create your views here.
diff --git a/bitbake/lib/toaster/manage.py b/bitbake/lib/toaster/manage.py
index 0c7ea50..ae32619 100755
--- a/bitbake/lib/toaster/manage.py
+++ b/bitbake/lib/toaster/manage.py
@@ -1,4 +1,8 @@
#!/usr/bin/env python3
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
import os
import sys
diff --git a/bitbake/lib/toaster/orm/fixtures/oe-core.xml b/bitbake/lib/toaster/orm/fixtures/oe-core.xml
index fec93ab..a723f5a 100644
--- a/bitbake/lib/toaster/orm/fixtures/oe-core.xml
+++ b/bitbake/lib/toaster/orm/fixtures/oe-core.xml
@@ -8,9 +8,9 @@
<!-- Bitbake versions which correspond to the metadata release -->
<object model="orm.bitbakeversion" pk="1">
- <field type="CharField" name="name">sumo</field>
+ <field type="CharField" name="name">warrior</field>
<field type="CharField" name="giturl">git://git.openembedded.org/bitbake</field>
- <field type="CharField" name="branch">1.38</field>
+ <field type="CharField" name="branch">1.42</field>
</object>
<object model="orm.bitbakeversion" pk="2">
<field type="CharField" name="name">HEAD</field>
@@ -23,18 +23,18 @@
<field type="CharField" name="branch">master</field>
</object>
<object model="orm.bitbakeversion" pk="4">
- <field type="CharField" name="name">thud</field>
+ <field type="CharField" name="name">zeus</field>
<field type="CharField" name="giturl">git://git.openembedded.org/bitbake</field>
- <field type="CharField" name="branch">1.40</field>
+ <field type="CharField" name="branch">1.44</field>
</object>
<!-- Releases available -->
<object model="orm.release" pk="1">
- <field type="CharField" name="name">sumo</field>
- <field type="CharField" name="description">Openembedded Sumo</field>
+ <field type="CharField" name="name">warrior</field>
+ <field type="CharField" name="description">Openembedded Warrior</field>
<field rel="ManyToOneRel" to="orm.bitbakeversion" name="bitbake_version">1</field>
- <field type="CharField" name="branch_name">sumo</field>
- <field type="TextField" name="helptext">Toaster will run your builds using the tip of the <a href=\"http://cgit.openembedded.org/openembedded-core/log/?h=sumo\">OpenEmbedded Sumo</a> branch.</field>
+ <field type="CharField" name="branch_name">warrior</field>
+ <field type="TextField" name="helptext">Toaster will run your builds using the tip of the <a href=\"http://cgit.openembedded.org/openembedded-core/log/?h=warrior\">OpenEmbedded Warrior</a> branch.</field>
</object>
<object model="orm.release" pk="2">
<field type="CharField" name="name">local</field>
@@ -51,11 +51,11 @@
<field type="TextField" name="helptext">Toaster will run your builds using the tip of the <a href=\"http://cgit.openembedded.org/openembedded-core/log/\">OpenEmbedded master</a> branch.</field>
</object>
<object model="orm.release" pk="4">
- <field type="CharField" name="name">thud</field>
- <field type="CharField" name="description">Openembedded Rocko</field>
- <field rel="ManyToOneRel" to="orm.bitbakeversion" name="bitbake_version">1</field>
- <field type="CharField" name="branch_name">thud</field>
- <field type="TextField" name="helptext">Toaster will run your builds using the tip of the <a href=\"http://cgit.openembedded.org/openembedded-core/log/?h=thud\">OpenEmbedded Thud</a> branch.</field>
+ <field type="CharField" name="name">zeus</field>
+ <field type="CharField" name="description">Openembedded Zeus</field>
+ <field rel="ManyToOneRel" to="orm.bitbakeversion" name="bitbake_version">4</field>
+ <field type="CharField" name="branch_name">zeus</field>
+ <field type="TextField" name="helptext">Toaster will run your builds using the tip of the <a href=\"http://cgit.openembedded.org/openembedded-core/log/?h=zeus\">OpenEmbedded Zeus</a> branch.</field>
</object>
<!-- Default layers for each release -->
diff --git a/bitbake/lib/toaster/orm/fixtures/poky.xml b/bitbake/lib/toaster/orm/fixtures/poky.xml
index fb9a771..7992383 100644
--- a/bitbake/lib/toaster/orm/fixtures/poky.xml
+++ b/bitbake/lib/toaster/orm/fixtures/poky.xml
@@ -8,9 +8,9 @@
<!-- Bitbake versions which correspond to the metadata release -->
<object model="orm.bitbakeversion" pk="1">
- <field type="CharField" name="name">sumo</field>
+ <field type="CharField" name="name">warrior</field>
<field type="CharField" name="giturl">git://git.yoctoproject.org/poky</field>
- <field type="CharField" name="branch">sumo</field>
+ <field type="CharField" name="branch">warrior</field>
<field type="CharField" name="dirpath">bitbake</field>
</object>
<object model="orm.bitbakeversion" pk="2">
@@ -26,20 +26,20 @@
<field type="CharField" name="dirpath">bitbake</field>
</object>
<object model="orm.bitbakeversion" pk="4">
- <field type="CharField" name="name">thud</field>
+ <field type="CharField" name="name">zeus</field>
<field type="CharField" name="giturl">git://git.yoctoproject.org/poky</field>
- <field type="CharField" name="branch">thud</field>
+ <field type="CharField" name="branch">zeus</field>
<field type="CharField" name="dirpath">bitbake</field>
</object>
<!-- Releases available -->
<object model="orm.release" pk="1">
- <field type="CharField" name="name">sumo</field>
- <field type="CharField" name="description">Yocto Project 2.5 "Sumo"</field>
+ <field type="CharField" name="name">warrior</field>
+ <field type="CharField" name="description">Yocto Project 2.7 "Warrior"</field>
<field rel="ManyToOneRel" to="orm.bitbakeversion" name="bitbake_version">1</field>
- <field type="CharField" name="branch_name">sumo</field>
- <field type="TextField" name="helptext">Toaster will run your builds using the tip of the <a href="http://git.yoctoproject.org/cgit/cgit.cgi/poky/log/?h=sumo">Yocto Project Sumo branch</a>.</field>
+ <field type="CharField" name="branch_name">warrior</field>
+ <field type="TextField" name="helptext">Toaster will run your builds using the tip of the <a href="http://git.yoctoproject.org/cgit/cgit.cgi/poky/log/?h=warrior">Yocto Project Warrior branch</a>.</field>
</object>
<object model="orm.release" pk="2">
<field type="CharField" name="name">local</field>
@@ -56,11 +56,11 @@
<field type="TextField" name="helptext">Toaster will run your builds using the tip of the <a href="http://git.yoctoproject.org/cgit/cgit.cgi/poky/log/">Yocto Project Master branch</a>.</field>
</object>
<object model="orm.release" pk="4">
- <field type="CharField" name="name">rocko</field>
- <field type="CharField" name="description">Yocto Project 2.6 "Thud"</field>
- <field rel="ManyToOneRel" to="orm.bitbakeversion" name="bitbake_version">1</field>
- <field type="CharField" name="branch_name">thud</field>
- <field type="TextField" name="helptext">Toaster will run your builds using the tip of the <a href="http://git.yoctoproject.org/cgit/cgit.cgi/poky/log/?h=thud">Yocto Project Thud branch</a>.</field>
+ <field type="CharField" name="name">zeus</field>
+ <field type="CharField" name="description">Yocto Project 3.0 "Zeus"</field>
+ <field rel="ManyToOneRel" to="orm.bitbakeversion" name="bitbake_version">4</field>
+ <field type="CharField" name="branch_name">zeus</field>
+ <field type="TextField" name="helptext">Toaster will run your builds using the tip of the <a href="http://git.yoctoproject.org/cgit/cgit.cgi/poky/log/?h=zeus">Yocto Project Zeus branch</a>.</field>
</object>
<!-- Default project layers for each release -->
@@ -130,7 +130,7 @@
<field rel="ManyToOneRel" to="orm.layer" name="layer">1</field>
<field type="IntegerField" name="layer_source">0</field>
<field rel="ManyToOneRel" to="orm.release" name="release">1</field>
- <field type="CharField" name="branch">sumo</field>
+ <field type="CharField" name="branch">warrior</field>
<field type="CharField" name="dirpath">meta</field>
</object>
<object model="orm.layer_version" pk="2">
@@ -152,7 +152,7 @@
<field rel="ManyToOneRel" to="orm.layer" name="layer">1</field>
<field type="IntegerField" name="layer_source">0</field>
<field rel="ManyToOneRel" to="orm.release" name="release">4</field>
- <field type="CharField" name="branch">rocko</field>
+ <field type="CharField" name="branch">zeus</field>
<field type="CharField" name="dirpath">meta</field>
</object>
@@ -168,7 +168,7 @@
<field rel="ManyToOneRel" to="orm.layer" name="layer">2</field>
<field type="IntegerField" name="layer_source">0</field>
<field rel="ManyToOneRel" to="orm.release" name="release">1</field>
- <field type="CharField" name="branch">sumo</field>
+ <field type="CharField" name="branch">warrior</field>
<field type="CharField" name="dirpath">meta-poky</field>
</object>
<object model="orm.layer_version" pk="6">
@@ -190,7 +190,7 @@
<field rel="ManyToOneRel" to="orm.layer" name="layer">2</field>
<field type="IntegerField" name="layer_source">0</field>
<field rel="ManyToOneRel" to="orm.release" name="release">4</field>
- <field type="CharField" name="branch">rocko</field>
+ <field type="CharField" name="branch">zeus</field>
<field type="CharField" name="dirpath">meta-poky</field>
</object>
@@ -206,7 +206,7 @@
<field rel="ManyToOneRel" to="orm.layer" name="layer">3</field>
<field type="IntegerField" name="layer_source">0</field>
<field rel="ManyToOneRel" to="orm.release" name="release">1</field>
- <field type="CharField" name="branch">sumo</field>
+ <field type="CharField" name="branch">warrior</field>
<field type="CharField" name="dirpath">meta-yocto-bsp</field>
</object>
<object model="orm.layer_version" pk="10">
@@ -228,7 +228,7 @@
<field rel="ManyToOneRel" to="orm.layer" name="layer">3</field>
<field type="IntegerField" name="layer_source">0</field>
<field rel="ManyToOneRel" to="orm.release" name="release">4</field>
- <field type="CharField" name="branch">rocko</field>
+ <field type="CharField" name="branch">zeus</field>
<field type="CharField" name="dirpath">meta-yocto-bsp</field>
</object>
</django-objects>
diff --git a/bitbake/lib/toaster/orm/management/commands/lsupdates.py b/bitbake/lib/toaster/orm/management/commands/lsupdates.py
index 66114ff..5b5abbb 100644
--- a/bitbake/lib/toaster/orm/management/commands/lsupdates.py
+++ b/bitbake/lib/toaster/orm/management/commands/lsupdates.py
@@ -1,23 +1,10 @@
#
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
-#
# BitBake Toaster Implementation
#
# Copyright (C) 2016-2017 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from django.core.management.base import BaseCommand
diff --git a/bitbake/lib/toaster/orm/models.py b/bitbake/lib/toaster/orm/models.py
index 7720290..bb6b5de 100644
--- a/bitbake/lib/toaster/orm/models.py
+++ b/bitbake/lib/toaster/orm/models.py
@@ -1,23 +1,10 @@
#
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
-#
# BitBake Toaster Implementation
#
# Copyright (C) 2013 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from __future__ import unicode_literals
@@ -978,12 +965,12 @@ class TargetSDKFile(models.Model):
class Target_Image_File(models.Model):
# valid suffixes for image files produced by a build
SUFFIXES = {
- 'btrfs', 'cpio', 'cpio.gz', 'cpio.lz4', 'cpio.lzma', 'cpio.xz',
- 'cramfs', 'elf', 'ext2', 'ext2.bz2', 'ext2.gz', 'ext2.lzma', 'ext4',
- 'ext4.gz', 'ext3', 'ext3.gz', 'hdddirect', 'hddimg', 'iso', 'jffs2',
- 'jffs2.sum', 'multiubi', 'qcow2', 'squashfs', 'squashfs-lzo',
+ 'btrfs', 'container', 'cpio', 'cpio.gz', 'cpio.lz4', 'cpio.lzma',
+ 'cpio.xz', 'cramfs', 'ext2', 'ext2.bz2', 'ext2.gz', 'ext2.lzma',
+ 'ext3', 'ext3.gz', 'ext4', 'ext4.gz', 'f2fs', 'hddimg', 'iso', 'jffs2',
+ 'jffs2.sum', 'multiubi', 'squashfs', 'squashfs-lz4', 'squashfs-lzo',
'squashfs-xz', 'tar', 'tar.bz2', 'tar.gz', 'tar.lz4', 'tar.xz', 'ubi',
- 'ubifs', 'vdi', 'vmdk', 'wic', 'wic.bmap', 'wic.bz2', 'wic.gz', 'wic.lzma'
+ 'ubifs', 'wic', 'wic.bz2', 'wic.gz', 'wic.lzma'
}
target = models.ForeignKey(Target)
diff --git a/bitbake/lib/toaster/tests/browser/selenium_helpers.py b/bitbake/lib/toaster/tests/browser/selenium_helpers.py
index 08711e4..02d4f4b 100644
--- a/bitbake/lib/toaster/tests/browser/selenium_helpers.py
+++ b/bitbake/lib/toaster/tests/browser/selenium_helpers.py
@@ -1,23 +1,10 @@
-#! /usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#! /usr/bin/env python3
#
# BitBake Toaster Implementation
#
# Copyright (C) 2013-2016 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+# SPDX-License-Identifier: GPL-2.0-only
#
# The Wait class and some of SeleniumDriverHelper and SeleniumTestCase are
# modified from Patchwork, released under the same licence terms as Toaster:
diff --git a/bitbake/lib/toaster/tests/browser/selenium_helpers_base.py b/bitbake/lib/toaster/tests/browser/selenium_helpers_base.py
index 156d639..6c94684 100644
--- a/bitbake/lib/toaster/tests/browser/selenium_helpers_base.py
+++ b/bitbake/lib/toaster/tests/browser/selenium_helpers_base.py
@@ -1,23 +1,10 @@
-#! /usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#! /usr/bin/env python3
#
# BitBake Toaster Implementation
#
# Copyright (C) 2013-2016 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+# SPDX-License-Identifier: GPL-2.0-only
#
# The Wait class and some of SeleniumDriverHelper and SeleniumTestCase are
# modified from Patchwork, released under the same licence terms as Toaster:
diff --git a/bitbake/lib/toaster/tests/browser/test_all_builds_page.py b/bitbake/lib/toaster/tests/browser/test_all_builds_page.py
index b86f29b..fba627b 100644
--- a/bitbake/lib/toaster/tests/browser/test_all_builds_page.py
+++ b/bitbake/lib/toaster/tests/browser/test_all_builds_page.py
@@ -1,23 +1,11 @@
-#! /usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#! /usr/bin/env python3
#
# BitBake Toaster Implementation
#
# Copyright (C) 2013-2016 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import re
diff --git a/bitbake/lib/toaster/tests/browser/test_all_projects_page.py b/bitbake/lib/toaster/tests/browser/test_all_projects_page.py
index 44da640..afd2d35 100644
--- a/bitbake/lib/toaster/tests/browser/test_all_projects_page.py
+++ b/bitbake/lib/toaster/tests/browser/test_all_projects_page.py
@@ -1,23 +1,11 @@
-#! /usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#! /usr/bin/env python3
#
# BitBake Toaster Implementation
#
# Copyright (C) 2013-2016 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import re
diff --git a/bitbake/lib/toaster/tests/browser/test_builddashboard_page.py b/bitbake/lib/toaster/tests/browser/test_builddashboard_page.py
index f8ccb54..d972aff 100644
--- a/bitbake/lib/toaster/tests/browser/test_builddashboard_page.py
+++ b/bitbake/lib/toaster/tests/browser/test_builddashboard_page.py
@@ -1,23 +1,11 @@
-#! /usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#! /usr/bin/env python3
#
# BitBake Toaster Implementation
#
# Copyright (C) 2013-2016 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from django.core.urlresolvers import reverse
from django.utils import timezone
diff --git a/bitbake/lib/toaster/tests/browser/test_builddashboard_page_artifacts.py b/bitbake/lib/toaster/tests/browser/test_builddashboard_page_artifacts.py
index 1c627ad..e2623e8 100644
--- a/bitbake/lib/toaster/tests/browser/test_builddashboard_page_artifacts.py
+++ b/bitbake/lib/toaster/tests/browser/test_builddashboard_page_artifacts.py
@@ -1,23 +1,11 @@
-#! /usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#! /usr/bin/env python3
#
# BitBake Toaster Implementation
#
# Copyright (C) 2013-2016 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from django.core.urlresolvers import reverse
from django.utils import timezone
diff --git a/bitbake/lib/toaster/tests/browser/test_builddashboard_page_recipes.py b/bitbake/lib/toaster/tests/browser/test_builddashboard_page_recipes.py
index ed18324..c542d45 100644
--- a/bitbake/lib/toaster/tests/browser/test_builddashboard_page_recipes.py
+++ b/bitbake/lib/toaster/tests/browser/test_builddashboard_page_recipes.py
@@ -1,23 +1,11 @@
-#! /usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#! /usr/bin/env python3
#
# BitBake Toaster Implementation
#
# Copyright (C) 2013-2016 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from django.core.urlresolvers import reverse
from django.utils import timezone
diff --git a/bitbake/lib/toaster/tests/browser/test_builddashboard_page_tasks.py b/bitbake/lib/toaster/tests/browser/test_builddashboard_page_tasks.py
index da50f16..22acb47 100644
--- a/bitbake/lib/toaster/tests/browser/test_builddashboard_page_tasks.py
+++ b/bitbake/lib/toaster/tests/browser/test_builddashboard_page_tasks.py
@@ -1,23 +1,11 @@
-#! /usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#! /usr/bin/env python3
#
# BitBake Toaster Implementation
#
# Copyright (C) 2013-2016 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from django.core.urlresolvers import reverse
from django.utils import timezone
diff --git a/bitbake/lib/toaster/tests/browser/test_js_unit_tests.py b/bitbake/lib/toaster/tests/browser/test_js_unit_tests.py
index 3c0b962..e8b4295 100644
--- a/bitbake/lib/toaster/tests/browser/test_js_unit_tests.py
+++ b/bitbake/lib/toaster/tests/browser/test_js_unit_tests.py
@@ -1,23 +1,11 @@
-#! /usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#! /usr/bin/env python3
#
# BitBake Toaster Implementation
#
# Copyright (C) 2013-2016 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
"""
Run the js unit tests
diff --git a/bitbake/lib/toaster/tests/browser/test_landing_page.py b/bitbake/lib/toaster/tests/browser/test_landing_page.py
index 4d4cd66..0790198 100644
--- a/bitbake/lib/toaster/tests/browser/test_landing_page.py
+++ b/bitbake/lib/toaster/tests/browser/test_landing_page.py
@@ -1,23 +1,11 @@
-#! /usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#! /usr/bin/env python3
#
# BitBake Toaster Implementation
#
-# Copyright (C) 2013-2016 Intel Corporation
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
+# Copyright (C) 2013-2016 Intel Corporation
#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from django.core.urlresolvers import reverse
from django.utils import timezone
diff --git a/bitbake/lib/toaster/tests/browser/test_layerdetails_page.py b/bitbake/lib/toaster/tests/browser/test_layerdetails_page.py
index f24fb09..f81e696 100644
--- a/bitbake/lib/toaster/tests/browser/test_layerdetails_page.py
+++ b/bitbake/lib/toaster/tests/browser/test_layerdetails_page.py
@@ -1,23 +1,11 @@
-#! /usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#! /usr/bin/env python3
#
# BitBake Toaster Implementation
#
-# Copyright (C) 2013-2016 Intel Corporation
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
+# Copyright (C) 2013-2016 Intel Corporation
#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from django.core.urlresolvers import reverse
from tests.browser.selenium_helpers import SeleniumTestCase
diff --git a/bitbake/lib/toaster/tests/browser/test_most_recent_builds_states.py b/bitbake/lib/toaster/tests/browser/test_most_recent_builds_states.py
index abc0b0b..15d25dc 100644
--- a/bitbake/lib/toaster/tests/browser/test_most_recent_builds_states.py
+++ b/bitbake/lib/toaster/tests/browser/test_most_recent_builds_states.py
@@ -1,23 +1,11 @@
-#! /usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#! /usr/bin/env python3
#
# BitBake Toaster Implementation
#
-# Copyright (C) 2013-2016 Intel Corporation
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
+# Copyright (C) 2013-2016 Intel Corporation
#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from django.core.urlresolvers import reverse
from django.utils import timezone
diff --git a/bitbake/lib/toaster/tests/browser/test_new_custom_image_page.py b/bitbake/lib/toaster/tests/browser/test_new_custom_image_page.py
index ab5a8e6..0aa3b7a 100644
--- a/bitbake/lib/toaster/tests/browser/test_new_custom_image_page.py
+++ b/bitbake/lib/toaster/tests/browser/test_new_custom_image_page.py
@@ -1,23 +1,11 @@
-#! /usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#! /usr/bin/env python3
#
# BitBake Toaster Implementation
#
# Copyright (C) 2013-2016 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from django.core.urlresolvers import reverse
from tests.browser.selenium_helpers import SeleniumTestCase
diff --git a/bitbake/lib/toaster/tests/browser/test_new_project_page.py b/bitbake/lib/toaster/tests/browser/test_new_project_page.py
index 77e5f15..8e56bb0 100644
--- a/bitbake/lib/toaster/tests/browser/test_new_project_page.py
+++ b/bitbake/lib/toaster/tests/browser/test_new_project_page.py
@@ -1,23 +1,11 @@
-#! /usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#! /usr/bin/env python3
#
# BitBake Toaster Implementation
#
# Copyright (C) 2013-2016 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from django.core.urlresolvers import reverse
from tests.browser.selenium_helpers import SeleniumTestCase
diff --git a/bitbake/lib/toaster/tests/browser/test_project_builds_page.py b/bitbake/lib/toaster/tests/browser/test_project_builds_page.py
index 9fe91ab..47fb10b 100644
--- a/bitbake/lib/toaster/tests/browser/test_project_builds_page.py
+++ b/bitbake/lib/toaster/tests/browser/test_project_builds_page.py
@@ -1,23 +1,11 @@
-#! /usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#! /usr/bin/env python3
#
# BitBake Toaster Implementation
#
# Copyright (C) 2013-2016 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import re
diff --git a/bitbake/lib/toaster/tests/browser/test_project_config_page.py b/bitbake/lib/toaster/tests/browser/test_project_config_page.py
index 0710084..2816eb9 100644
--- a/bitbake/lib/toaster/tests/browser/test_project_config_page.py
+++ b/bitbake/lib/toaster/tests/browser/test_project_config_page.py
@@ -1,23 +1,11 @@
-#! /usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#! /usr/bin/env python3
#
# BitBake Toaster Implementation
#
# Copyright (C) 2013-2016 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import re
diff --git a/bitbake/lib/toaster/tests/browser/test_project_page.py b/bitbake/lib/toaster/tests/browser/test_project_page.py
index 0186463..8b5e1b6 100644
--- a/bitbake/lib/toaster/tests/browser/test_project_page.py
+++ b/bitbake/lib/toaster/tests/browser/test_project_page.py
@@ -1,23 +1,11 @@
-#! /usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#! /usr/bin/env python3
#
# BitBake Toaster Implementation
#
# Copyright (C) 2013-2016 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from django.core.urlresolvers import reverse
from django.utils import timezone
diff --git a/bitbake/lib/toaster/tests/browser/test_sample.py b/bitbake/lib/toaster/tests/browser/test_sample.py
index 20ec53c..f4ad670 100644
--- a/bitbake/lib/toaster/tests/browser/test_sample.py
+++ b/bitbake/lib/toaster/tests/browser/test_sample.py
@@ -1,23 +1,11 @@
-#! /usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#! /usr/bin/env python3
#
# BitBake Toaster Implementation
#
# Copyright (C) 2013-2016 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
"""
A small example test demonstrating the basics of writing a test with
diff --git a/bitbake/lib/toaster/tests/browser/test_task_page.py b/bitbake/lib/toaster/tests/browser/test_task_page.py
index 690d116..26f3dca 100644
--- a/bitbake/lib/toaster/tests/browser/test_task_page.py
+++ b/bitbake/lib/toaster/tests/browser/test_task_page.py
@@ -1,23 +1,11 @@
-#! /usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#! /usr/bin/env python3
#
# BitBake Toaster Implementation
#
# Copyright (C) 2013-2016 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from django.core.urlresolvers import reverse
from django.utils import timezone
diff --git a/bitbake/lib/toaster/tests/browser/test_toastertable_ui.py b/bitbake/lib/toaster/tests/browser/test_toastertable_ui.py
index 53ddf30..ef78cbb 100644
--- a/bitbake/lib/toaster/tests/browser/test_toastertable_ui.py
+++ b/bitbake/lib/toaster/tests/browser/test_toastertable_ui.py
@@ -1,23 +1,11 @@
-#! /usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#! /usr/bin/env python3
#
# BitBake Toaster Implementation
#
# Copyright (C) 2013-2016 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from datetime import datetime
diff --git a/bitbake/lib/toaster/tests/builds/buildtest.py b/bitbake/lib/toaster/tests/builds/buildtest.py
index 5a56a11..872bbd3 100644
--- a/bitbake/lib/toaster/tests/builds/buildtest.py
+++ b/bitbake/lib/toaster/tests/builds/buildtest.py
@@ -1,23 +1,11 @@
-#! /usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#! /usr/bin/env python3
#
# BitBake Toaster Implementation
#
# Copyright (C) 2016 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import os
import sys
diff --git a/bitbake/lib/toaster/tests/builds/test_core_image_min.py b/bitbake/lib/toaster/tests/builds/test_core_image_min.py
index 586f4a8..44b6cbe 100644
--- a/bitbake/lib/toaster/tests/builds/test_core_image_min.py
+++ b/bitbake/lib/toaster/tests/builds/test_core_image_min.py
@@ -1,24 +1,11 @@
-#! /usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#! /usr/bin/env python3
#
# BitBake Toaster Implementation
#
# Copyright (C) 2016 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
# Tests were part of openembedded-core oe selftest Authored by: Lucian Musat
# Ionut Chisanovici, Paul Eggleton and Cristian Iorga
diff --git a/bitbake/lib/toaster/tests/commands/test_loaddata.py b/bitbake/lib/toaster/tests/commands/test_loaddata.py
index 951f6ff..9e8d555 100644
--- a/bitbake/lib/toaster/tests/commands/test_loaddata.py
+++ b/bitbake/lib/toaster/tests/commands/test_loaddata.py
@@ -1,23 +1,11 @@
-#! /usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#! /usr/bin/env python3
#
# BitBake Toaster Implementation
#
# Copyright (C) 2016 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from django.test import TestCase
from django.core import management
diff --git a/bitbake/lib/toaster/tests/commands/test_lsupdates.py b/bitbake/lib/toaster/tests/commands/test_lsupdates.py
index 49897a4..3c4fbe0 100644
--- a/bitbake/lib/toaster/tests/commands/test_lsupdates.py
+++ b/bitbake/lib/toaster/tests/commands/test_lsupdates.py
@@ -1,23 +1,11 @@
-#! /usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#! /usr/bin/env python3
#
# BitBake Toaster Implementation
#
# Copyright (C) 2016 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from django.test import TestCase
from django.core import management
diff --git a/bitbake/lib/toaster/tests/commands/test_runbuilds.py b/bitbake/lib/toaster/tests/commands/test_runbuilds.py
index 3e63483..e223b95 100644
--- a/bitbake/lib/toaster/tests/commands/test_runbuilds.py
+++ b/bitbake/lib/toaster/tests/commands/test_runbuilds.py
@@ -1,23 +1,11 @@
-#! /usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#! /usr/bin/env python3
#
# BitBake Toaster Implementation
#
# Copyright (C) 2016 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import os
diff --git a/bitbake/lib/toaster/tests/db/test_db.py b/bitbake/lib/toaster/tests/db/test_db.py
index a0f5f6e..0410422 100644
--- a/bitbake/lib/toaster/tests/db/test_db.py
+++ b/bitbake/lib/toaster/tests/db/test_db.py
@@ -2,6 +2,8 @@
#
# Copyright (c) 2016 Damien Lespiau
#
+# SPDX-License-Identifier: MIT
+#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
diff --git a/bitbake/lib/toaster/tests/eventreplay/__init__.py b/bitbake/lib/toaster/tests/eventreplay/__init__.py
index 6956619..8ed6792 100644
--- a/bitbake/lib/toaster/tests/eventreplay/__init__.py
+++ b/bitbake/lib/toaster/tests/eventreplay/__init__.py
@@ -1,23 +1,11 @@
-#! /usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#! /usr/bin/env python3
#
# BitBake Toaster Implementation
#
# Copyright (C) 2016 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
# Tests were part of openembedded-core oe selftest Authored by: Lucian Musat
# Ionut Chisanovici, Paul Eggleton and Cristian Iorga
diff --git a/bitbake/lib/toaster/tests/functional/functional_helpers.py b/bitbake/lib/toaster/tests/functional/functional_helpers.py
index 486078a..455c408 100644
--- a/bitbake/lib/toaster/tests/functional/functional_helpers.py
+++ b/bitbake/lib/toaster/tests/functional/functional_helpers.py
@@ -1,23 +1,11 @@
-#! /usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#! /usr/bin/env python3
#
# BitBake Toaster functional tests implementation
#
# Copyright (C) 2017 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import os
import logging
diff --git a/bitbake/lib/toaster/tests/functional/test_functional_basic.py b/bitbake/lib/toaster/tests/functional/test_functional_basic.py
index cfa2b0f..56c84fb 100644
--- a/bitbake/lib/toaster/tests/functional/test_functional_basic.py
+++ b/bitbake/lib/toaster/tests/functional/test_functional_basic.py
@@ -1,23 +1,11 @@
-#! /usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#! /usr/bin/env python3
#
# BitBake Toaster functional tests implementation
#
# Copyright (C) 2017 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import time
import re
diff --git a/bitbake/lib/toaster/tests/views/test_views.py b/bitbake/lib/toaster/tests/views/test_views.py
index 1463077..68d9e9d 100644
--- a/bitbake/lib/toaster/tests/views/test_views.py
+++ b/bitbake/lib/toaster/tests/views/test_views.py
@@ -1,23 +1,11 @@
-#! /usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#! /usr/bin/env python3
#
# BitBake Toaster Implementation
#
# Copyright (C) 2013-2015 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
"""Test cases for Toaster GUI and ReST."""
diff --git a/bitbake/lib/toaster/toastergui/api.py b/bitbake/lib/toaster/toastergui/api.py
index 564d595..8b49b3e 100644
--- a/bitbake/lib/toaster/toastergui/api.py
+++ b/bitbake/lib/toaster/toastergui/api.py
@@ -3,19 +3,8 @@
#
# Copyright (C) 2016 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
# Please run flake8 on this file before sending patches
import os
diff --git a/bitbake/lib/toaster/toastergui/buildtables.py b/bitbake/lib/toaster/toastergui/buildtables.py
index 755a7c2..327059d 100644
--- a/bitbake/lib/toaster/toastergui/buildtables.py
+++ b/bitbake/lib/toaster/toastergui/buildtables.py
@@ -1,23 +1,10 @@
#
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
-#
# BitBake Toaster Implementation
#
# Copyright (C) 2016 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from orm.models import Build, Task, Target, Package
from django.db.models import Q, Sum
diff --git a/bitbake/lib/toaster/toastergui/static/js/importlayer.js b/bitbake/lib/toaster/toastergui/static/js/importlayer.js
index 2964839..8e2032d 100644
--- a/bitbake/lib/toaster/toastergui/static/js/importlayer.js
+++ b/bitbake/lib/toaster/toastergui/static/js/importlayer.js
@@ -17,11 +17,15 @@ function importLayerPageInit (ctx) {
var currentLayerDepSelection;
var validLayerName = /^(\w|-)+$/;
+ /* Catch 'disable' race condition between type-ahead started and "input change" */
+ var typeAheadStarted = 0;
+
libtoaster.makeTypeahead(layerDepInput,
libtoaster.ctx.layersTypeAheadUrl,
{ include_added: "true" }, function(item){
currentLayerDepSelection = item;
layerDepBtn.removeAttr("disabled");
+ typeAheadStarted = 1;
});
layerDepInput.on("typeahead:select", function(event, data){
@@ -34,7 +38,10 @@ function importLayerPageInit (ctx) {
// disable the "Add layer" button when the layer input typeahead is empty
// or not in the typeahead choices
layerDepInput.on("input change", function(){
- layerDepBtn.attr("disabled","disabled");
+ if (0 == typeAheadStarted) {
+ layerDepBtn.attr("disabled","disabled");
+ }
+ typeAheadStarted = 0;
});
/* We automatically add "openembedded-core" layer for convenience as a
@@ -50,6 +57,7 @@ function importLayerPageInit (ctx) {
});
layerDepBtn.click(function(){
+ typeAheadStarted = 0;
if (currentLayerDepSelection == undefined)
return;
@@ -77,7 +85,7 @@ function importLayerPageInit (ctx) {
$("#layer-deps-list").append(newLayerDep);
- libtoaster.getLayerDepsForProject(currentLayerDepSelection.layerdetailurl,
+ libtoaster.getLayerDepsForProject(currentLayerDepSelection.xhrLayerUrl,
function (data){
/* These are the dependencies of the layer added as a dependency */
if (data.list.length > 0) {
diff --git a/bitbake/lib/toaster/toastergui/tablefilter.py b/bitbake/lib/toaster/toastergui/tablefilter.py
index 65454e1..ffef795 100644
--- a/bitbake/lib/toaster/toastergui/tablefilter.py
+++ b/bitbake/lib/toaster/toastergui/tablefilter.py
@@ -1,23 +1,10 @@
#
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
-#
# BitBake Toaster Implementation
#
# Copyright (C) 2015 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from django.db.models import Q, Max, Min
from django.utils import dateparse, timezone
diff --git a/bitbake/lib/toaster/toastergui/tables.py b/bitbake/lib/toaster/toastergui/tables.py
index 9ff756b..b3ea222 100644
--- a/bitbake/lib/toaster/toastergui/tables.py
+++ b/bitbake/lib/toaster/toastergui/tables.py
@@ -1,23 +1,10 @@
#
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
-#
# BitBake Toaster Implementation
#
# Copyright (C) 2015 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from toastergui.widgets import ToasterTable
from orm.models import Recipe, ProjectLayer, Layer_Version, Machine, Project
diff --git a/bitbake/lib/toaster/toastergui/templatetags/field_values_filter.py b/bitbake/lib/toaster/toastergui/templatetags/field_values_filter.py
index 5a73af7..eb48339 100644
--- a/bitbake/lib/toaster/toastergui/templatetags/field_values_filter.py
+++ b/bitbake/lib/toaster/toastergui/templatetags/field_values_filter.py
@@ -1,3 +1,7 @@
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
from django import template
register = template.Library()
diff --git a/bitbake/lib/toaster/toastergui/templatetags/objects_to_dictionaries_filter.py b/bitbake/lib/toaster/toastergui/templatetags/objects_to_dictionaries_filter.py
index 0dcc7d2..048d533 100644
--- a/bitbake/lib/toaster/toastergui/templatetags/objects_to_dictionaries_filter.py
+++ b/bitbake/lib/toaster/toastergui/templatetags/objects_to_dictionaries_filter.py
@@ -1,3 +1,7 @@
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
from django import template
import json
diff --git a/bitbake/lib/toaster/toastergui/templatetags/project_url_tag.py b/bitbake/lib/toaster/toastergui/templatetags/project_url_tag.py
index 04770ac..71e0925 100644
--- a/bitbake/lib/toaster/toastergui/templatetags/project_url_tag.py
+++ b/bitbake/lib/toaster/toastergui/templatetags/project_url_tag.py
@@ -1,3 +1,7 @@
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
from django import template
from django.core.urlresolvers import reverse
diff --git a/bitbake/lib/toaster/toastergui/templatetags/projecttags.py b/bitbake/lib/toaster/toastergui/templatetags/projecttags.py
index b170a16..1dbab3b 100644
--- a/bitbake/lib/toaster/toastergui/templatetags/projecttags.py
+++ b/bitbake/lib/toaster/toastergui/templatetags/projecttags.py
@@ -1,23 +1,10 @@
#
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
-#
# BitBake Toaster Implementation
#
# Copyright (C) 2013 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from datetime import datetime, timedelta
from os.path import relpath
diff --git a/bitbake/lib/toaster/toastergui/typeaheads.py b/bitbake/lib/toaster/toastergui/typeaheads.py
index 5aa0f8d..fd750ff 100644
--- a/bitbake/lib/toaster/toastergui/typeaheads.py
+++ b/bitbake/lib/toaster/toastergui/typeaheads.py
@@ -3,18 +3,8 @@
#
# Copyright (C) 2015 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
import subprocess
diff --git a/bitbake/lib/toaster/toastergui/urls.py b/bitbake/lib/toaster/toastergui/urls.py
index dc03e30..673d9ae 100644
--- a/bitbake/lib/toaster/toastergui/urls.py
+++ b/bitbake/lib/toaster/toastergui/urls.py
@@ -3,18 +3,8 @@
#
# Copyright (C) 2013-2017 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from django.conf.urls import include, url
from django.views.generic import RedirectView, TemplateView
diff --git a/bitbake/lib/toaster/toastergui/views.py b/bitbake/lib/toaster/toastergui/views.py
index c712b06..d7acaff 100644
--- a/bitbake/lib/toaster/toastergui/views.py
+++ b/bitbake/lib/toaster/toastergui/views.py
@@ -1,24 +1,10 @@
#
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
-#
# BitBake Toaster Implementation
#
# Copyright (C) 2013 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-
import re
diff --git a/bitbake/lib/toaster/toastergui/widgets.py b/bitbake/lib/toaster/toastergui/widgets.py
index db5c3aa..645f458 100644
--- a/bitbake/lib/toaster/toastergui/widgets.py
+++ b/bitbake/lib/toaster/toastergui/widgets.py
@@ -1,23 +1,10 @@
#
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
-#
# BitBake Toaster Implementation
#
# Copyright (C) 2015 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from django.views.generic import View, TemplateView
from django.views.decorators.cache import cache_control
diff --git a/bitbake/lib/toaster/toastermain/management/commands/builddelete.py b/bitbake/lib/toaster/toastermain/management/commands/builddelete.py
index bf69a8f..c2d773a 100644
--- a/bitbake/lib/toaster/toastermain/management/commands/builddelete.py
+++ b/bitbake/lib/toaster/toastermain/management/commands/builddelete.py
@@ -1,3 +1,7 @@
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
from django.core.management.base import BaseCommand, CommandError
from django.core.exceptions import ObjectDoesNotExist
from orm.models import Build
diff --git a/bitbake/lib/toaster/toastermain/management/commands/buildimport.py b/bitbake/lib/toaster/toastermain/management/commands/buildimport.py
index 2d57ab5..408ad44 100644
--- a/bitbake/lib/toaster/toastermain/management/commands/buildimport.py
+++ b/bitbake/lib/toaster/toastermain/management/commands/buildimport.py
@@ -1,23 +1,10 @@
#
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
-#
# BitBake Toaster Implementation
#
# Copyright (C) 2018 Wind River Systems
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
# buildimport: import a project for project specific configuration
#
diff --git a/bitbake/lib/toaster/toastermain/management/commands/buildslist.py b/bitbake/lib/toaster/toastermain/management/commands/buildslist.py
index 70b5812..1ed2022 100644
--- a/bitbake/lib/toaster/toastermain/management/commands/buildslist.py
+++ b/bitbake/lib/toaster/toastermain/management/commands/buildslist.py
@@ -1,3 +1,7 @@
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
from django.core.management.base import BaseCommand, CommandError
from orm.models import Build
import os
diff --git a/bitbake/lib/toaster/toastermain/management/commands/checksocket.py b/bitbake/lib/toaster/toastermain/management/commands/checksocket.py
index 0399b86..811fd5d 100644
--- a/bitbake/lib/toaster/toastermain/management/commands/checksocket.py
+++ b/bitbake/lib/toaster/toastermain/management/commands/checksocket.py
@@ -1,23 +1,11 @@
-#!/usr/bin/env python
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#!/usr/bin/env python3
#
# BitBake Toaster Implementation
#
# Copyright (C) 2015 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
"""Custom management command checksocket."""
diff --git a/bitbake/lib/toaster/toastermain/management/commands/perf.py b/bitbake/lib/toaster/toastermain/management/commands/perf.py
index 6b450bb..7d629fb 100644
--- a/bitbake/lib/toaster/toastermain/management/commands/perf.py
+++ b/bitbake/lib/toaster/toastermain/management/commands/perf.py
@@ -1,3 +1,7 @@
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
from django.core.management.base import BaseCommand
from django.test.client import Client
import os, sys, re
diff --git a/bitbake/lib/toaster/toastermain/settings.py b/bitbake/lib/toaster/toastermain/settings.py
index 13541d3..74501fa 100644
--- a/bitbake/lib/toaster/toastermain/settings.py
+++ b/bitbake/lib/toaster/toastermain/settings.py
@@ -1,23 +1,10 @@
#
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
-#
# BitBake Toaster Implementation
#
# Copyright (C) 2013 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
# Django settings for Toaster project.
diff --git a/bitbake/lib/toaster/toastermain/settings_production_example.py b/bitbake/lib/toaster/toastermain/settings_production_example.py
index 61a2888..6cd0f52 100644
--- a/bitbake/lib/toaster/toastermain/settings_production_example.py
+++ b/bitbake/lib/toaster/toastermain/settings_production_example.py
@@ -1,23 +1,10 @@
#
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
-#
# BitBake Toaster Implementation
#
# Copyright (C) 2016 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
# See Django documentation for more information about deployment
# https://docs.djangoproject.com/en/1.8/howto/deployment/checklist/
diff --git a/bitbake/lib/toaster/toastermain/settings_test.py b/bitbake/lib/toaster/toastermain/settings_test.py
index a322711..6538d9e 100644
--- a/bitbake/lib/toaster/toastermain/settings_test.py
+++ b/bitbake/lib/toaster/toastermain/settings_test.py
@@ -1,23 +1,10 @@
#
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
-#
# BitBake Toaster Implementation
#
# Copyright (C) 2016 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
# Django settings for Toaster project.
diff --git a/bitbake/lib/toaster/toastermain/urls.py b/bitbake/lib/toaster/toastermain/urls.py
index e2fb0ae..ac77bc3 100644
--- a/bitbake/lib/toaster/toastermain/urls.py
+++ b/bitbake/lib/toaster/toastermain/urls.py
@@ -1,23 +1,10 @@
#
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
-#
# BitBake Toaster Implementation
#
# Copyright (C) 2013 Intel Corporation
#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
+# SPDX-License-Identifier: GPL-2.0-only
#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
from django.conf.urls import include, url
from django.views.generic import RedirectView, TemplateView
diff --git a/bitbake/lib/toaster/toastermain/wsgi.py b/bitbake/lib/toaster/toastermain/wsgi.py
index 031b314..4c31283 100644
--- a/bitbake/lib/toaster/toastermain/wsgi.py
+++ b/bitbake/lib/toaster/toastermain/wsgi.py
@@ -1,7 +1,8 @@
-"""
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+"""
WSGI config for Toaster project.
This module contains the WSGI application used by Django's development server
diff --git a/doc/user_manual.md b/doc/user_manual.md
index c2657da..96aa2e3 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -135,8 +135,8 @@ DISTRO_ARCH ??= "armhf"
Then, call `bitbake` with image names, e.g.:
```
-bitbake multiconfig:qemuarm-buster:isar-image-base \
- multiconfig:qemuarm-buster:isar-image-debug
+bitbake mc:qemuarm-buster:isar-image-base \
+ mc:qemuarm-buster:isar-image-debug
```
The following images are created:
@@ -169,14 +169,14 @@ The following command will produce `isar-image-base` images for all targets:
```
$ bitbake \
- multiconfig:qemuarm-stretch:isar-image-base \
- multiconfig:qemuarm-buster:isar-image-base \
- multiconfig:qemuarm64-stretch:isar-image-base \
- multiconfig:qemui386-stretch:isar-image-base \
- multiconfig:qemui386-buster:isar-image-base \
- multiconfig:qemuamd64-stretch:isar-image-base \
- multiconfig:qemuamd64-buster:isar-image-base \
- multiconfig:rpi-stretch:isar-image-base
+ mc:qemuarm-stretch:isar-image-base \
+ mc:qemuarm-buster:isar-image-base \
+ mc:qemuarm64-stretch:isar-image-base \
+ mc:qemui386-stretch:isar-image-base \
+ mc:qemui386-buster:isar-image-base \
+ mc:qemuamd64-stretch:isar-image-base \
+ mc:qemuamd64-buster:isar-image-base \
+ mc:rpi-stretch:isar-image-base
```
Created images are:
@@ -197,9 +197,9 @@ tmp/deploy/images/rpi/isar-image-base.rpi-sdimg
A bootable disk image is generated if you set IMAGE_TYPE to 'wic-img'. Behind the scenes a tool called `wic` is used to assemble the images. It is controlled by a `.wks` file which you can choose with changing WKS_FILE. Some examples in the tree use that feature already.
```
# Generate an image for the `i386` target architecture
- $ bitbake multiconfig:qemui386-buster:isar-image-base
+ $ bitbake mc:qemui386-buster:isar-image-base
# Similarly, for the `amd64` target architecture, in this case EFI
- $ bitbake multiconfig:qemuamd64-buster:isar-image-base
+ $ bitbake mc:qemuamd64-buster:isar-image-base
```
Variables may be used in `.wks.in` files; Isar will expand them and generate a regular `.wks` file before generating the disk image using `wic`.
@@ -303,7 +303,7 @@ following variables define the default configuration to build for:
- `DISTRO_ARCH` - The Debian architecture to build for (e.g., `armhf`).
If BitBake is called with multiconfig targets (e.g.,
-`multiconfig:qemuarm-buster:isar-image-base`), the following variable defines
+`mc:qemuarm-buster:isar-image-base`), the following variable defines
all supported configurations:
- `BBMULTICONFIG` - The list of the complete configuration definition files.
@@ -687,7 +687,7 @@ Debian cross-compilation works out of the box starting from Debian stretch distr
Just like OpenEmbedded, Isar supports a devshell target for all dpkg package
recipes. This target opens a terminal inside the buildchroot that runs the
package build. To invoke it, just call
-`bitbake multiconfig:${MACHINE}-${DISTRO}:<package_name> -c devshell`.
+`bitbake mc:${MACHINE}-${DISTRO}:<package_name> -c devshell`.
## Create an ISAR SDK root filesystem
@@ -705,7 +705,7 @@ target binary artifacts. Developer chroots to sdk rootfs and develops applicatio
### Solution
User manually triggers creation of SDK root filesystem for his target platform by launching the task `do_populate_sdk` for target image, f.e.
-`bitbake -c do_populate_sdk multiconfig:${MACHINE}-${DISTRO}:isar-image-base`.
+`bitbake -c do_populate_sdk mc:${MACHINE}-${DISTRO}:isar-image-base`.
The resulting SDK rootfs is archived into `tmp/deploy/images/${MACHINE}/sdk-${DISTRO}-${DISTRO_ARCH}.tar.xz`.
It is additionally available for direct use under `tmp/deploy/images/${MACHINE}/sdk-${DISTRO}-${DISTRO_ARCH}/`.
@@ -717,7 +717,7 @@ One may chroot into the SDK and install required target packages with the help o
- Trigger creation of SDK root filesystem
```
-bitbake -c do_populate_sdk multiconfig:qemuarm-buster:isar-image-base
+bitbake -c do_populate_sdk mc:qemuarm-buster:isar-image-base
```
- Mount the following directories in chroot by passing resulting rootfs as an argument to the script `mount_chroot.sh`:
@@ -801,7 +801,7 @@ BASE_REPO_KEY = "file://<absolute_path_to_your_pub_key_file>"'
- Trigger creation of local apt caching Debian packages during image generation.
```
-bitbake -c cache_base_repo multiconfig:qemuarm-buster:isar-image-base
+bitbake -c cache_base_repo mc:qemuarm-buster:isar-image-base
```
- Set `ISAR_USE_CACHED_BASE_REPO` in `conf/local.conf`:
@@ -820,7 +820,7 @@ sudo rm -rf tmp
- Trigger again generation of image (now using local caching repo):
```
-bitbake multiconfig:qemuarm-buster:isar-image-base
+bitbake mc:qemuarm-buster:isar-image-base
```
### Limitation
@@ -860,5 +860,5 @@ DISTRO_APT_SOURCES_append = " conf/distro/docker-buster.list"
And build the corresponding image target:
```
-bitbake multiconfig:qemuarm64-buster:isar-image-base
+bitbake mc:qemuarm64-buster:isar-image-base
```
diff --git a/meta-isar/conf/conf-notes.txt b/meta-isar/conf/conf-notes.txt
index 74df97c..c557cd3 100644
--- a/meta-isar/conf/conf-notes.txt
+++ b/meta-isar/conf/conf-notes.txt
@@ -1,4 +1,4 @@
Common targets are:
- multiconfig:qemuarm-buster:isar-image-base
- multiconfig:qemuamd64-buster:isar-image-base
- multiconfig:rpi-stretch:isar-image-base
+ mc:qemuarm-buster:isar-image-base
+ mc:qemuamd64-buster:isar-image-base
+ mc:rpi-stretch:isar-image-base
diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index 713d1c2..a8556a6 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -21,40 +21,40 @@ BUILD_DIR=./build
BB_ARGS="-v"
TARGETS_SET="\
- multiconfig:qemuarm-stretch:isar-image-base \
- multiconfig:qemuarm-buster:isar-image-base \
- multiconfig:qemuarm-bullseye:isar-image-base \
- multiconfig:qemuarm64-stretch:isar-image-base \
- multiconfig:qemui386-stretch:isar-image-base \
- multiconfig:qemui386-buster:isar-image-base \
- multiconfig:qemui386-bullseye:isar-image-base \
- multiconfig:qemuamd64-stretch:isar-image-base \
- multiconfig:qemuamd64-buster:isar-image-base \
- multiconfig:qemuamd64-buster-tgz:isar-image-base \
- multiconfig:qemuamd64-bullseye:isar-image-base \
- multiconfig:qemumipsel-stretch:isar-image-base \
- multiconfig:qemumipsel-buster:isar-image-base \
- multiconfig:qemumipsel-bullseye:isar-image-base \
- multiconfig:nand-ubi-demo-buster:isar-image-ubi \
- multiconfig:rpi-stretch:isar-image-base"
+ mc:qemuarm-stretch:isar-image-base \
+ mc:qemuarm-buster:isar-image-base \
+ mc:qemuarm-bullseye:isar-image-base \
+ mc:qemuarm64-stretch:isar-image-base \
+ mc:qemui386-stretch:isar-image-base \
+ mc:qemui386-buster:isar-image-base \
+ mc:qemui386-bullseye:isar-image-base \
+ mc:qemuamd64-stretch:isar-image-base \
+ mc:qemuamd64-buster:isar-image-base \
+ mc:qemuamd64-buster-tgz:isar-image-base \
+ mc:qemuamd64-bullseye:isar-image-base \
+ mc:qemumipsel-stretch:isar-image-base \
+ mc:qemumipsel-buster:isar-image-base \
+ mc:qemumipsel-bullseye:isar-image-base \
+ mc:nand-ubi-demo-buster:isar-image-ubi \
+ mc:rpi-stretch:isar-image-base"
# qemu-user-static of <= buster too old to build that
- # multiconfig:qemuarm64-buster:isar-image-base
- # multiconfig:qemuarm64-bullseye:isar-image-base
+ # mc:qemuarm64-buster:isar-image-base
+ # mc:qemuarm64-bullseye:isar-image-base
CROSS_TARGETS_SET="\
- multiconfig:qemuarm-stretch:isar-image-base \
- multiconfig:qemuarm-buster:isar-image-base \
- multiconfig:qemuarm-bullseye:isar-image-base \
- multiconfig:qemuarm64-stretch:isar-image-base \
- multiconfig:qemuamd64-stretch:isar-image-base \
- multiconfig:de0-nano-soc-stretch:isar-image-base \
- multiconfig:rpi-stretch:isar-image-base"
+ mc:qemuarm-stretch:isar-image-base \
+ mc:qemuarm-buster:isar-image-base \
+ mc:qemuarm-bullseye:isar-image-base \
+ mc:qemuarm64-stretch:isar-image-base \
+ mc:qemuamd64-stretch:isar-image-base \
+ mc:de0-nano-soc-stretch:isar-image-base \
+ mc:rpi-stretch:isar-image-base"
REPRO_TARGETS_SET="\
- multiconfig:qemuarm-stretch:isar-image-base \
- multiconfig:qemuarm64-stretch:isar-image-base \
- multiconfig:qemuamd64-stretch:isar-image-base \
- multiconfig:qemuarm-buster:isar-image-base"
+ mc:qemuarm-stretch:isar-image-base \
+ mc:qemuarm64-stretch:isar-image-base \
+ mc:qemuamd64-stretch:isar-image-base \
+ mc:qemuarm-buster:isar-image-base"
show_help() {
@@ -162,7 +162,7 @@ sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?= "1"/g' conf/local.c
bitbake $BB_ARGS $CROSS_TARGETS_SET
while [ -e bitbake.sock ]; do sleep 1; done
# In addition test SDK creation
-bitbake $BB_ARGS -c do_populate_sdk multiconfig:qemuarm-stretch:isar-image-base
+bitbake $BB_ARGS -c do_populate_sdk mc:qemuarm-stretch:isar-image-base
while [ -e bitbake.sock ]; do sleep 1; done
if [ -z "$FAST_BUILD" ]; then
@@ -175,6 +175,6 @@ fi
cp -a "${ISARROOT}/meta/classes/dpkg-base.bbclass" "${ISARROOT}/meta/classes/dpkg-base.bbclass.ci-backup"
echo -e "do_fetch_append() {\n\n}" >> "${ISARROOT}/meta/classes/dpkg-base.bbclass"
-bitbake $BB_ARGS multiconfig:qemuamd64-stretch:isar-image-base
+bitbake $BB_ARGS mc:qemuamd64-stretch:isar-image-base
mv "${ISARROOT}/meta/classes/dpkg-base.bbclass.ci-backup" "${ISARROOT}/meta/classes/dpkg-base.bbclass"
diff --git a/scripts/start_vm b/scripts/start_vm
index 9f07a03..71b55f2 100755
--- a/scripts/start_vm
+++ b/scripts/start_vm
@@ -101,7 +101,7 @@ do
shift
done
-eval $(bitbake -e multiconfig:qemu$ARCH-$DISTRO:isar-image-base | grep "^DEPLOY_DIR_IMAGE=")
+eval $(bitbake -e mc:qemu$ARCH-$DISTRO:isar-image-base | grep "^DEPLOY_DIR_IMAGE=")
readonly IMAGE_DIR=$DEPLOY_DIR_IMAGE
readonly ISARROOT="$(dirname "$0")"/..
@@ -112,13 +112,13 @@ eval "$(egrep 'MACHINE_SERIAL' $MACHINE_CONF |bb2sh)"
readonly CONFIG_CONF=$ISARROOT/meta-isar/conf/multiconfig/qemu$ARCH-$DISTRO.conf
eval "$(egrep 'QEMU_' $CONFIG_CONF |bb2sh)"
-eval $(bitbake -e multiconfig:qemu$ARCH-$DISTRO:isar-image-base | grep "^IMAGE_TYPE=")
+eval $(bitbake -e mc:qemu$ARCH-$DISTRO:isar-image-base | grep "^IMAGE_TYPE=")
case "$IMAGE_TYPE" in
ext4-img)
readonly ROOTFS_IMAGE=isar-image-base-debian-$DISTRO-qemu$ARCH.ext4.img
- eval $(bitbake -e multiconfig:qemu$ARCH-$DISTRO:isar-image-base | grep "^KERNEL_IMAGE=")
- eval $(bitbake -e multiconfig:qemu$ARCH-$DISTRO:isar-image-base | grep "^INITRD_IMAGE=")
+ eval $(bitbake -e mc:qemu$ARCH-$DISTRO:isar-image-base | grep "^KERNEL_IMAGE=")
+ eval $(bitbake -e mc:qemu$ARCH-$DISTRO:isar-image-base | grep "^INITRD_IMAGE=")
QKERNEL=$IMAGE_DIR/${KERNEL_IMAGE}
QINITRD=/dev/null
[ -n "$INITRD_IMAGE" ] && QINITRD=$IMAGE_DIR/${INITRD_IMAGE}
diff --git a/testsuite/build_test/build_test.py b/testsuite/build_test/build_test.py
index 4220d3a..7a55c2f 100644
--- a/testsuite/build_test/build_test.py
+++ b/testsuite/build_test/build_test.py
@@ -22,7 +22,7 @@ class BuildTest(Test):
#isar_root = dirname(__file__) + '/..'
os.chdir(build_dir)
- cmdline = ['bitbake', 'multiconfig:qemu' + arch + '-' + distro + ':isar-image-base']
+ cmdline = ['bitbake', 'mc:qemu' + arch + '-' + distro + ':isar-image-base']
p1 = subprocess32.run(cmdline)
if p1.returncode:
diff --git a/testsuite/start_vm.py b/testsuite/start_vm.py
index 02a4b51..a3e32ac 100755
--- a/testsuite/start_vm.py
+++ b/testsuite/start_vm.py
@@ -10,7 +10,7 @@ import sys
import time
def get_bitbake_env(arch, distro):
- multiconfig = 'multiconfig:qemu' + arch + '-' + distro + ':isar-image-base'
+ multiconfig = 'mc:qemu' + arch + '-' + distro + ':isar-image-base'
output = subprocess.check_output(['bitbake', '-e', str(multiconfig)])
return output
--
2.20.1
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 1/1] bitbake: update to version 1.44.0
2019-10-21 14:19 ` [PATCH 1/1] bitbake: update " Cedric Hombourger
@ 2019-10-22 7:36 ` Henning Schild
2019-10-22 7:41 ` Cedric Hombourger
0 siblings, 1 reply; 11+ messages in thread
From: Henning Schild @ 2019-10-22 7:36 UTC (permalink / raw)
To: Cedric Hombourger; +Cc: isar-users
Hi Cedric,
i would suggest a split between the actual bump and the following
Isar-changes. But i do not care too much and both ways have their pros
and cons.
Henning
Am Mon, 21 Oct 2019 16:19:58 +0200
schrieb Cedric Hombourger <Cedric_Hombourger@mentor.com>:
> Found that the gitsm fetcher we ship is rather out-dated compared to
> upstream and was failing to pull git sub-modules from repositories
> such as azure-umqtt-c.
>
> It shall be noted that bitbake has renamed its multiconfig targets:
> the "multiconfig:" prefix was changed to "mc:". Updated our scripts
> as well as documentation.
>
> Origin: https://github.com/openembedded/bitbake.git
> Commit 5d83d828cacb58ccb7c464e799c85fd2d2a50ccc (tag: 1.44.0)
> Author: Richard Purdie <richard.purdie@linuxfoundation.org>
> Date: Wed Oct 9 14:10:21 2019 +0100
>
> bitbake: Update to version 1.44.0
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>
> Signed-off-by: Cedric Hombourger <Cedric_Hombourger@mentor.com>
> ---
> RECIPE-API-CHANGELOG.md | 16 +
> bitbake/.gitattributes | 2 +
> bitbake/HEADER | 19 -
> bitbake/LICENSE | 12 +-
> bitbake/{COPYING => LICENSE.GPL-2.0-only} | 61 +-
> bitbake/LICENSE.MIT | 25 +
> bitbake/MANIFEST.in | 5 +-
> bitbake/bin/bitbake | 16 +-
> bitbake/bin/bitbake-diffsigs | 155 +-
> bitbake/bin/bitbake-dumpsig | 95 +-
> bitbake/bin/bitbake-hashclient | 170 ++
> bitbake/bin/bitbake-hashserv | 62 +
> bitbake/bin/bitbake-layers | 12 +-
> bitbake/bin/bitbake-prserv | 4 +
> bitbake/bin/bitbake-selftest | 16 +-
> bitbake/bin/bitbake-worker | 19 +-
> bitbake/bin/bitdoc | 14 +-
> bitbake/bin/git-make-shallow | 4 +
> bitbake/bin/toaster | 12 +-
> bitbake/bin/toaster-eventreplay | 17 +-
> bitbake/classes/base.bbclass | 2 +-
> bitbake/contrib/dump_cache.py | 2 -
> .../bitbake-user-manual-execution.xml | 78 +-
> .../bitbake-user-manual-fetching.xml | 69 +-
> .../bitbake-user-manual-hello.xml | 28 +-
> .../bitbake-user-manual-intro.xml | 18 +-
> .../bitbake-user-manual-metadata.xml | 179 +-
> .../bitbake-user-manual-ref-variables.xml | 422 ++---
> bitbake/doc/poky.ent | 8 -
> bitbake/lib/bb/COW.py | 15 -
> bitbake/lib/bb/__init__.py | 16 +-
> bitbake/lib/bb/build.py | 151 +-
> bitbake/lib/bb/cache.py | 109 +-
> bitbake/lib/bb/cache_extra.py | 14 +-
> bitbake/lib/bb/checksum.py | 12 +-
> bitbake/lib/bb/codeparser.py | 13 +-
> bitbake/lib/bb/command.py | 12 +-
> bitbake/lib/bb/compat.py | 4 +
> bitbake/lib/bb/cooker.py | 218 +--
> bitbake/lib/bb/cookerdata.py | 44 +-
> bitbake/lib/bb/daemonize.py | 4 +
> bitbake/lib/bb/data.py | 23 +-
> bitbake/lib/bb/data_smart.py | 25 +-
> bitbake/lib/bb/event.py | 33 +-
> bitbake/lib/bb/exceptions.py | 3 +
> bitbake/lib/bb/fetch2/__init__.py | 55 +-
> bitbake/lib/bb/fetch2/bzr.py | 12 +-
> bitbake/lib/bb/fetch2/clearcase.py | 19 +-
> bitbake/lib/bb/fetch2/cvs.py | 17 +-
> bitbake/lib/bb/fetch2/git.py | 68 +-
> bitbake/lib/bb/fetch2/gitannex.py | 14 +-
> bitbake/lib/bb/fetch2/gitsm.py | 281 ++-
> bitbake/lib/bb/fetch2/hg.py | 18 +-
> bitbake/lib/bb/fetch2/local.py | 16 +-
> bitbake/lib/bb/fetch2/npm.py | 69 +-
> bitbake/lib/bb/fetch2/osc.py | 5 +-
> bitbake/lib/bb/fetch2/perforce.py | 15 +-
> bitbake/lib/bb/fetch2/repo.py | 16 +-
> bitbake/lib/bb/fetch2/s3.py | 15 +-
> bitbake/lib/bb/fetch2/sftp.py | 15 +-
> bitbake/lib/bb/fetch2/ssh.py | 14 +-
> bitbake/lib/bb/fetch2/svn.py | 98 +-
> bitbake/lib/bb/fetch2/wget.py | 104 +-
> bitbake/lib/bb/main.py | 41 +-
> bitbake/lib/bb/methodpool.py | 15 +-
> bitbake/lib/bb/monitordisk.py | 27 +-
> bitbake/lib/bb/msg.py | 14 +-
> bitbake/lib/bb/namedtuple_with_abc.py | 4 +-
> bitbake/lib/bb/parse/__init__.py | 14 +-
> bitbake/lib/bb/parse/ast.py | 15 +-
> bitbake/lib/bb/parse/parse_py/BBHandler.py | 59 +-
> bitbake/lib/bb/parse/parse_py/ConfHandler.py | 17 +-
> bitbake/lib/bb/parse/parse_py/__init__.py | 17 +-
> bitbake/lib/bb/persist_data.py | 234 ++-
> bitbake/lib/bb/process.py | 4 +
> bitbake/lib/bb/progress.py | 28 +-
> bitbake/lib/bb/providers.py | 18 +-
> bitbake/lib/bb/pysh/builtin.py | 710 --------
> bitbake/lib/bb/pysh/interp.py | 1367 ---------------
> bitbake/lib/bb/pysh/lsprof.py | 116 --
> bitbake/lib/bb/pysh/pysh.py | 167 --
> bitbake/lib/bb/pysh/pyshlex.py | 5 -
> bitbake/lib/bb/pysh/pyshyacc.py | 17 +-
> bitbake/lib/bb/pysh/sherrors.py | 26 -
> bitbake/lib/bb/pysh/subprocess_fix.py | 77 -
> bitbake/lib/bb/remotedata.py | 12 +-
> bitbake/lib/bb/runqueue.py | 1523
> ++++++++++------- bitbake/lib/bb/server/__init__.py |
> 14 +- bitbake/lib/bb/server/process.py | 123 +-
> bitbake/lib/bb/server/xmlrpcclient.py | 12 +-
> bitbake/lib/bb/server/xmlrpcserver.py | 12 +-
> bitbake/lib/bb/siggen.py | 350 +++-
> bitbake/lib/bb/taskdata.py | 17 +-
> bitbake/lib/bb/tests/codeparser.py | 22 +-
> bitbake/lib/bb/tests/cooker.py | 15 +-
> bitbake/lib/bb/tests/cow.py | 17 +-
> bitbake/lib/bb/tests/data.py | 39 +-
> bitbake/lib/bb/tests/event.py | 23 +-
> bitbake/lib/bb/tests/fetch.py | 308 +++-
> bitbake/lib/bb/tests/parse.py | 33 +-
> bitbake/lib/bb/tests/persist_data.py | 129 ++
> .../tests/runqueue-tests/classes/base.bbclass | 262 +++
> .../runqueue-tests/classes/image.bbclass | 5 +
> .../runqueue-tests/classes/native.bbclass | 2 +
> .../bb/tests/runqueue-tests/conf/bitbake.conf | 16 +
> .../runqueue-tests/conf/multiconfig/mc1.conf | 1 +
> .../runqueue-tests/conf/multiconfig/mc2.conf | 1 +
> .../lib/bb/tests/runqueue-tests/recipes/a1.bb | 0
> .../lib/bb/tests/runqueue-tests/recipes/b1.bb | 1 +
> .../lib/bb/tests/runqueue-tests/recipes/c1.bb | 0
> .../lib/bb/tests/runqueue-tests/recipes/d1.bb | 3 +
> .../lib/bb/tests/runqueue-tests/recipes/e1.bb | 1 +
> bitbake/lib/bb/tests/runqueue.py | 323 ++++
> bitbake/lib/bb/tests/utils.py | 23 +-
> bitbake/lib/bb/tinfoil.py | 12 +-
> bitbake/lib/bb/ui/__init__.py | 12 +-
> bitbake/lib/bb/ui/buildinfohelper.py | 22 +-
> bitbake/lib/bb/ui/knotty.py | 60 +-
> bitbake/lib/bb/ui/ncurses.py | 12 +-
> bitbake/lib/bb/ui/taskexp.py | 12 +-
> bitbake/lib/bb/ui/toasterui.py | 12 +-
> bitbake/lib/bb/ui/uievent.py | 15 +-
> bitbake/lib/bb/ui/uihelper.py | 16 +-
> bitbake/lib/bb/utils.py | 49 +-
> bitbake/lib/bblayers/__init__.py | 4 +
> bitbake/lib/bblayers/action.py | 4 +
> bitbake/lib/bblayers/common.py | 4 +
> bitbake/lib/bblayers/layerindex.py | 6 +-
> bitbake/lib/bblayers/query.py | 43 +-
> bitbake/lib/bs4/dammit.py | 12 +-
> bitbake/lib/bs4/element.py | 22 +-
> bitbake/lib/hashserv/__init__.py | 93 +
> bitbake/lib/hashserv/client.py | 157 ++
> bitbake/lib/hashserv/server.py | 414 +++++
> bitbake/lib/hashserv/tests.py | 142 ++
> bitbake/lib/layerindexlib/__init__.py | 33 +-
> bitbake/lib/layerindexlib/cooker.py | 12 +-
> bitbake/lib/layerindexlib/plugin.py | 13 +-
> bitbake/lib/layerindexlib/restapi.py | 12 +-
> bitbake/lib/layerindexlib/tests/common.py | 12 +-
> bitbake/lib/layerindexlib/tests/cooker.py | 12 +-
> .../lib/layerindexlib/tests/layerindexobj.py | 12 +-
> bitbake/lib/layerindexlib/tests/restapi.py | 12 +-
> bitbake/lib/progressbar/__init__.py | 2 +
> bitbake/lib/progressbar/compat.py | 2 +
> bitbake/lib/progressbar/progressbar.py | 2 +
> bitbake/lib/progressbar/widgets.py | 2 +
> bitbake/lib/prserv/__init__.py | 4 +
> bitbake/lib/prserv/db.py | 6 +-
> bitbake/lib/prserv/serv.py | 4 +
> bitbake/lib/pyinotify.py | 20 +-
> bitbake/lib/toaster/bldcollector/admin.py | 4 +
> bitbake/lib/toaster/bldcollector/urls.py | 13 +-
> bitbake/lib/toaster/bldcollector/views.py | 12 +-
> bitbake/lib/toaster/bldcontrol/admin.py | 4 +
> .../lib/toaster/bldcontrol/bbcontroller.py | 16 +-
> .../bldcontrol/localhostbecontroller.py | 16 +-
> .../management/commands/checksettings.py | 4 +
> .../management/commands/runbuilds.py | 4 +
> bitbake/lib/toaster/bldcontrol/models.py | 4 +
> bitbake/lib/toaster/bldcontrol/views.py | 4 +
> bitbake/lib/toaster/manage.py | 4 +
> bitbake/lib/toaster/orm/fixtures/oe-core.xml | 26 +-
> bitbake/lib/toaster/orm/fixtures/poky.xml | 38 +-
> .../orm/management/commands/lsupdates.py | 15 +-
> bitbake/lib/toaster/orm/models.py | 25 +-
> .../toaster/tests/browser/selenium_helpers.py | 17 +-
> .../tests/browser/selenium_helpers_base.py | 17 +-
> .../tests/browser/test_all_builds_page.py | 16 +-
> .../tests/browser/test_all_projects_page.py | 16 +-
> .../tests/browser/test_builddashboard_page.py | 16 +-
> .../test_builddashboard_page_artifacts.py | 16 +-
> .../test_builddashboard_page_recipes.py | 16 +-
> .../browser/test_builddashboard_page_tasks.py | 16 +-
> .../tests/browser/test_js_unit_tests.py | 16 +-
> .../tests/browser/test_landing_page.py | 18 +-
> .../tests/browser/test_layerdetails_page.py | 18 +-
> .../browser/test_most_recent_builds_states.py | 18 +-
> .../browser/test_new_custom_image_page.py | 16 +-
> .../tests/browser/test_new_project_page.py | 16 +-
> .../tests/browser/test_project_builds_page.py | 16 +-
> .../tests/browser/test_project_config_page.py | 16 +-
> .../tests/browser/test_project_page.py | 16 +-
> .../lib/toaster/tests/browser/test_sample.py | 16 +-
> .../toaster/tests/browser/test_task_page.py | 16 +-
> .../tests/browser/test_toastertable_ui.py | 16 +-
> bitbake/lib/toaster/tests/builds/buildtest.py | 16 +-
> .../tests/builds/test_core_image_min.py | 17 +-
> .../toaster/tests/commands/test_loaddata.py | 16 +-
> .../toaster/tests/commands/test_lsupdates.py | 16 +-
> .../toaster/tests/commands/test_runbuilds.py | 16 +-
> bitbake/lib/toaster/tests/db/test_db.py | 2 +
> .../lib/toaster/tests/eventreplay/__init__.py | 16 +-
> .../tests/functional/functional_helpers.py | 16 +-
> .../tests/functional/test_functional_basic.py | 16 +-
> bitbake/lib/toaster/tests/views/test_views.py | 16 +-
> bitbake/lib/toaster/toastergui/api.py | 13 +-
> bitbake/lib/toaster/toastergui/buildtables.py | 15 +-
> .../toastergui/static/js/importlayer.js | 12 +-
> bitbake/lib/toaster/toastergui/tablefilter.py | 15 +-
> bitbake/lib/toaster/toastergui/tables.py | 15 +-
> .../templatetags/field_values_filter.py | 4 +
> .../objects_to_dictionaries_filter.py | 4 +
> .../templatetags/project_url_tag.py | 4 +
> .../toastergui/templatetags/projecttags.py | 15 +-
> bitbake/lib/toaster/toastergui/typeaheads.py | 12 +-
> bitbake/lib/toaster/toastergui/urls.py | 12 +-
> bitbake/lib/toaster/toastergui/views.py | 16 +-
> bitbake/lib/toaster/toastergui/widgets.py | 15 +-
> .../management/commands/builddelete.py | 4 +
> .../management/commands/buildimport.py | 15 +-
> .../management/commands/buildslist.py | 4 +
> .../management/commands/checksocket.py | 16 +-
> .../toastermain/management/commands/perf.py | 4 +
> bitbake/lib/toaster/toastermain/settings.py | 15 +-
> .../settings_production_example.py | 15 +-
> .../lib/toaster/toastermain/settings_test.py | 15 +-
> bitbake/lib/toaster/toastermain/urls.py | 15 +-
> bitbake/lib/toaster/toastermain/wsgi.py | 7 +-
> doc/user_manual.md | 38 +-
> meta-isar/conf/conf-notes.txt | 6 +-
> scripts/ci_build.sh | 62 +-
> scripts/start_vm | 8 +-
> testsuite/build_test/build_test.py | 2 +-
> testsuite/start_vm.py | 2 +-
> 225 files changed, 5375 insertions(+), 6320 deletions(-)
> create mode 100644 bitbake/.gitattributes
> delete mode 100644 bitbake/HEADER
> rename bitbake/{COPYING => LICENSE.GPL-2.0-only} (84%)
> create mode 100644 bitbake/LICENSE.MIT
> mode change 100755 => 120000 bitbake/bin/bitbake-dumpsig
> create mode 100755 bitbake/bin/bitbake-hashclient
> create mode 100755 bitbake/bin/bitbake-hashserv
> delete mode 100644 bitbake/lib/bb/pysh/builtin.py
> delete mode 100644 bitbake/lib/bb/pysh/interp.py
> delete mode 100644 bitbake/lib/bb/pysh/lsprof.py
> delete mode 100644 bitbake/lib/bb/pysh/pysh.py
> delete mode 100644 bitbake/lib/bb/pysh/subprocess_fix.py
> create mode 100644 bitbake/lib/bb/tests/persist_data.py
> create mode 100644
> bitbake/lib/bb/tests/runqueue-tests/classes/base.bbclass create mode
> 100644 bitbake/lib/bb/tests/runqueue-tests/classes/image.bbclass
> create mode 100644
> bitbake/lib/bb/tests/runqueue-tests/classes/native.bbclass create
> mode 100644 bitbake/lib/bb/tests/runqueue-tests/conf/bitbake.conf
> create mode 100644
> bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc1.conf create
> mode 100644
> bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc2.conf create
> mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/a1.bb create
> mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/b1.bb create
> mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/c1.bb create
> mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/d1.bb create
> mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/e1.bb create
> mode 100644 bitbake/lib/bb/tests/runqueue.py create mode 100644
> bitbake/lib/hashserv/__init__.py create mode 100644
> bitbake/lib/hashserv/client.py create mode 100644
> bitbake/lib/hashserv/server.py create mode 100644
> bitbake/lib/hashserv/tests.py
>
> diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
> index bbef1a3..81ec430 100644
> --- a/RECIPE-API-CHANGELOG.md
> +++ b/RECIPE-API-CHANGELOG.md
> @@ -176,3 +176,19 @@ Otherwise set a encrypted root password like
> this: USERS += "root"
> USER_root[password] =
> "$6$rounds=10000$RXeWrnFmkY$DtuS/OmsAS2cCEDo0BF5qQsizIrq6jPgXnwv3PHqREJeKd1sXdHX/ayQtuQWVDHe0KIO0/sVH8dvQm1KthF0d/"
> ``` +
> +### multiconfig build targets were renamed
> +
> +bitbake was upgraded to version 1.44.0 where "multiconfig" build
> targets were +renamed "mc". As an example, builds for the
> qemuarm-stretch machine should now +be done as follows:
> +
> +```
> +bitbake mc:qemuarm-stretch:isar-image-base
> +```
> +
> +The old syntax is no longer supported and will produce an error:
> +
> +```
> +bitbake multiconfig:qemuarm-stretch:isar-image-base
> +```
> diff --git a/bitbake/.gitattributes b/bitbake/.gitattributes
> new file mode 100644
> index 0000000..e4f8f62
> --- /dev/null
> +++ b/bitbake/.gitattributes
> @@ -0,0 +1,2 @@
> +*min.js binary
> +*min.css binary
> diff --git a/bitbake/HEADER b/bitbake/HEADER
> deleted file mode 100644
> index 9859255..0000000
> --- a/bitbake/HEADER
> +++ /dev/null
> @@ -1,19 +0,0 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> -#
> -# <one line to give the program's name and a brief idea of what it
> does.> -# Copyright (C) <year> <name of author>
> -#
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. -
> diff --git a/bitbake/LICENSE b/bitbake/LICENSE
> index 7d4e5f4..8458042 100644
> --- a/bitbake/LICENSE
> +++ b/bitbake/LICENSE
> @@ -1,4 +1,13 @@
> -BitBake is licensed under the GNU General Public License version
> 2.0. See COPYING for further details. +BitBake is licensed under the
> GNU General Public License version 2.0. See +LICENSE.GPL-2.0-only for
> further details. +
> +Individual files contain the following style tags instead of the
> full license text: +
> + SPDX-License-Identifier: GPL-2.0-only
> +
> +This enables machine processing of license information based on the
> SPDX +License Identifiers that are here available:
> http://spdx.org/licenses/ +
>
> The following external components are distributed with this software:
>
> @@ -17,3 +26,4 @@ Foundation and individual contributors.
> * Font Awesome fonts redistributed under the SIL Open Font License
> 1.1
> * simplediff is distributed under the zlib license.
> +
> diff --git a/bitbake/COPYING b/bitbake/LICENSE.GPL-2.0-only
> similarity index 84%
> rename from bitbake/COPYING
> rename to bitbake/LICENSE.GPL-2.0-only
> index d511905..5db3c0a 100644
> --- a/bitbake/COPYING
> +++ b/bitbake/LICENSE.GPL-2.0-only
> @@ -279,61 +279,10 @@ POSSIBILITY OF SUCH DAMAGES.
>
> END OF TERMS AND CONDITIONS
>
> - How to Apply These Terms to Your New Programs
> +Note:
> +Individual files contain the following tag instead of the full
> license text.
> - If you develop a new program, and you want it to be of the greatest
> -possible use to the public, the best way to achieve this is to make
> it -free software which everyone can redistribute and change under
> these terms.
> + SPDX-License-Identifier: GPL-2.0-only
>
> - To do so, attach the following notices to the program. It is
> safest -to attach them to the start of each source file to most
> effectively -convey the exclusion of warranty; and each file should
> have at least -the "copyright" line and a pointer to where the full
> notice is found. -
> - <one line to give the program's name and a brief idea of what it
> does.>
> - Copyright (C) <year> <name of author>
> -
> - This program is free software; you can redistribute it and/or
> modify
> - it under the terms of the GNU General Public License as
> published by
> - the Free Software Foundation; either version 2 of the License, or
> - (at your option) any later version.
> -
> - This program is distributed in the hope that it will be useful,
> - but WITHOUT ANY WARRANTY; without even the implied warranty of
> - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> - GNU General Public License for more details.
> -
> - You should have received a copy of the GNU General Public
> License along
> - with this program; if not, write to the Free Software
> Foundation, Inc.,
> - 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> -
> -Also add information on how to contact you by electronic and paper
> mail. -
> -If the program is interactive, make it output a short notice like
> this -when it starts in an interactive mode:
> -
> - Gnomovision version 69, Copyright (C) year name of author
> - Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type
> `show w'.
> - This is free software, and you are welcome to redistribute it
> - under certain conditions; type `show c' for details.
> -
> -The hypothetical commands `show w' and `show c' should show the
> appropriate -parts of the General Public License. Of course, the
> commands you use may -be called something other than `show w' and
> `show c'; they could even be -mouse-clicks or menu items--whatever
> suits your program. -
> -You should also get your employer (if you work as a programmer) or
> your -school, if any, to sign a "copyright disclaimer" for the
> program, if -necessary. Here is a sample; alter the names:
> -
> - Yoyodyne, Inc., hereby disclaims all copyright interest in the
> program
> - `Gnomovision' (which makes passes at compilers) written by James
> Hacker. -
> - <signature of Ty Coon>, 1 April 1989
> - Ty Coon, President of Vice
> -
> -This General Public License does not permit incorporating your
> program into -proprietary programs. If your program is a subroutine
> library, you may -consider it more useful to permit linking
> proprietary applications with the -library. If this is what you want
> to do, use the GNU Lesser General -Public License instead of this
> License. +This enables machine processing of license information
> based on the SPDX +License Identifiers that are here available:
> http://spdx.org/licenses/ diff --git a/bitbake/LICENSE.MIT
> b/bitbake/LICENSE.MIT new file mode 100644
> index 0000000..a6919eb
> --- /dev/null
> +++ b/bitbake/LICENSE.MIT
> @@ -0,0 +1,25 @@
> +Permission is hereby granted, free of charge, to any person
> obtaining a copy +of this software and associated documentation files
> (the "Software"), to deal +in the Software without restriction,
> including without limitation the rights +to use, copy, modify, merge,
> publish, distribute, sublicense, and/or sell +copies of the Software,
> and to permit persons to whom the Software is +furnished to do so,
> subject to the following conditions: +
> +The above copyright notice and this permission notice shall be
> included in +all copies or substantial portions of the Software.
> +
> +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
> EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
> MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND
> NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS
> BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN
> ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN
> CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
> +SOFTWARE. +
> +Note:
> +Individual files contain the following tag instead of the full
> license text. +
> + SPDX-License-Identifier: MIT
> +
> +This enables machine processing of license information based on the
> SPDX +License Identifiers that are here available:
> http://spdx.org/licenses/ diff --git a/bitbake/MANIFEST.in
> b/bitbake/MANIFEST.in index b197378..f24969a 100644
> --- a/bitbake/MANIFEST.in
> +++ b/bitbake/MANIFEST.in
> @@ -1,6 +1,8 @@
> -include COPYING
> include ChangeLog
> include AUTHORS
> +include LICENSE
> +include LICENSE.GPL-2.0-only
> +include LICENSE.MIT
> include contrib/*
> include contrib/vim/*/*
> include conf/*
> @@ -8,4 +10,3 @@ include classes/*
> include doc/*
> include doc/manual/*
> include ez_setup.py
> -include HEADER
> diff --git a/bitbake/bin/bitbake b/bitbake/bin/bitbake
> index 57dec2a..66d08f8 100755
> --- a/bitbake/bin/bitbake
> +++ b/bitbake/bin/bitbake
> @@ -1,6 +1,4 @@
> #!/usr/bin/env python3
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # Copyright (C) 2003, 2004 Chris Larson
> # Copyright (C) 2003, 2004 Phil Blundell
> @@ -9,18 +7,8 @@
> # Copyright (C) 2005 ROAD GmbH
> # Copyright (C) 2006 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import os
> import sys
> @@ -38,7 +26,7 @@ from bb.main import bitbake_main,
> BitBakeConfigParameters, BBMainException if
> sys.getfilesystemencoding() != "utf-8": sys.exit("Please use a locale
> setting which supports UTF-8 (such as LANG=en_US.UTF-8).\nPython
> can't change the filesystem locale after loading so we need a UTF-8
> when Python starts or things won't work.") -__version__ = "1.40.0"
> +__version__ = "1.44.0"
> if __name__ == "__main__":
> if __version__ != bb.__version__:
> diff --git a/bitbake/bin/bitbake-diffsigs
> b/bitbake/bin/bitbake-diffsigs index 4e6bbdd..19420a2 100755
> --- a/bitbake/bin/bitbake-diffsigs
> +++ b/bitbake/bin/bitbake-diffsigs
> @@ -1,27 +1,16 @@
> #!/usr/bin/env python3
>
> -# bitbake-diffsigs
> -# BitBake task signature data comparison utility
> +# bitbake-diffsigs / bitbake-dumpsig
> +# BitBake task signature data dump and comparison utility
> #
> # Copyright (C) 2012-2013, 2017 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import os
> import sys
> import warnings
> -import fnmatch
> import argparse
> import logging
> import pickle
> @@ -32,7 +21,10 @@ import bb.tinfoil
> import bb.siggen
> import bb.msg
>
> -logger = bb.msg.logger_create('bitbake-diffsigs')
> +myname = os.path.basename(sys.argv[0])
> +logger = bb.msg.logger_create(myname)
> +
> +is_dump = myname == 'bitbake-dumpsig'
>
> def find_siginfo(tinfoil, pn, taskname, sigs=None):
> result = None
> @@ -59,8 +51,8 @@ def find_siginfo(tinfoil, pn, taskname, sigs=None):
> sys.exit(2)
> return result
>
> -def find_compare_task(bbhandler, pn, taskname, sig1=None, sig2=None,
> color=False):
> - """ Find the most recent signature files for the specified
> PN/task and compare them """ +def find_siginfo_task(bbhandler, pn,
> taskname, sig1=None, sig2=None):
> + """ Find the most recent signature files for the specified
> PN/task """
> if not taskname.startswith('do_'):
> taskname = 'do_%s' % taskname
> @@ -79,73 +71,81 @@ def find_compare_task(bbhandler, pn, taskname,
> sig1=None, sig2=None, color=False latestfiles = [sigfiles[sig1],
> sigfiles[sig2]] else:
> filedates = find_siginfo(bbhandler, pn, taskname)
> - latestfiles = sorted(filedates.keys(), key=lambda f:
> filedates[f])[-3:]
> + latestfiles = sorted(filedates.keys(), key=lambda f:
> filedates[f])[-2:] if not latestfiles:
> logger.error('No sigdata files found matching %s %s' %
> (pn, taskname)) sys.exit(1)
> - elif len(latestfiles) < 2:
> - logger.error('Only one matching sigdata file found for
> the specified task (%s %s)' % (pn, taskname))
> - sys.exit(1)
>
> - # Define recursion callback
> - def recursecb(key, hash1, hash2):
> - hashes = [hash1, hash2]
> - hashfiles = find_siginfo(bbhandler, key, None, hashes)
> -
> - recout = []
> - if len(hashfiles) == 0:
> - recout.append("Unable to find matching sigdata for %s
> with hashes %s or %s" % (key, hash1, hash2))
> - elif not hash1 in hashfiles:
> - recout.append("Unable to find matching sigdata for %s
> with hash %s" % (key, hash1))
> - elif not hash2 in hashfiles:
> - recout.append("Unable to find matching sigdata for %s
> with hash %s" % (key, hash2))
> - else:
> - out2 = bb.siggen.compare_sigfiles(hashfiles[hash1],
> hashfiles[hash2], recursecb, color=color)
> - for change in out2:
> - for line in change.splitlines():
> - recout.append(' ' + line)
> + return latestfiles
> +
>
> - return recout
> +# Define recursion callback
> +def recursecb(key, hash1, hash2):
> + hashes = [hash1, hash2]
> + hashfiles = find_siginfo(tinfoil, key, None, hashes)
>
> - # Recurse into signature comparison
> - logger.debug("Signature file (previous): %s" % latestfiles[-2])
> - logger.debug("Signature file (latest): %s" % latestfiles[-1])
> - output = bb.siggen.compare_sigfiles(latestfiles[-2],
> latestfiles[-1], recursecb, color=color)
> - if output:
> - print('\n'.join(output))
> - sys.exit(0)
> + recout = []
> + if len(hashfiles) == 0:
> + recout.append("Unable to find matching sigdata for %s with
> hashes %s or %s" % (key, hash1, hash2))
> + elif not hash1 in hashfiles:
> + recout.append("Unable to find matching sigdata for %s with
> hash %s" % (key, hash1))
> + elif not hash2 in hashfiles:
> + recout.append("Unable to find matching sigdata for %s with
> hash %s" % (key, hash2))
> + else:
> + out2 = bb.siggen.compare_sigfiles(hashfiles[hash1],
> hashfiles[hash2], recursecb, color=color)
> + for change in out2:
> + for line in change.splitlines():
> + recout.append(' ' + line)
>
> + return recout
>
>
> parser = argparse.ArgumentParser(
> - description="Compares siginfo/sigdata files written out by
> BitBake")
> + description=("Dumps" if is_dump else "Compares") + "
> siginfo/sigdata files written out by BitBake")
> -parser.add_argument('-d', '--debug',
> +parser.add_argument('-D', '--debug',
> help='Enable debug output',
> action='store_true')
>
> -parser.add_argument('--color',
> - help='Colorize output (where %(metavar)s is %(choices)s)',
> - choices=['auto', 'always', 'never'], default='auto',
> metavar='color') +if is_dump:
> + parser.add_argument("-t", "--task",
> + help="find the signature data file for the last run of
> the specified task",
> + action="store", dest="taskargs", nargs=2,
> metavar=('recipename', 'taskname')) +
> + parser.add_argument("sigdatafile1",
> + help="Signature file to dump. Not used when using
> -t/--task.",
> + action="store", nargs='?', metavar="sigdatafile")
> +else:
> + parser.add_argument('-c', '--color',
> + help='Colorize the output (where %(metavar)s is
> %(choices)s)',
> + choices=['auto', 'always', 'never'], default='auto',
> metavar='color')
> -parser.add_argument("-t", "--task",
> - help="find the signature data files for last two runs of the
> specified task and compare them",
> - action="store", dest="taskargs", nargs=2,
> metavar=('recipename', 'taskname'))
> + parser.add_argument('-d', '--dump',
> + help='Dump the last signature data instead of comparing
> (equivalent to using bitbake-dumpsig)',
> + action='store_true')
>
> -parser.add_argument("-s", "--signature",
> - help="With -t/--task, specify the signatures to look for
> instead of taking the last two",
> - action="store", dest="sigargs", nargs=2, metavar=('fromsig',
> 'tosig'))
> + parser.add_argument("-t", "--task",
> + help="find the signature data files for the last two
> runs of the specified task and compare them",
> + action="store", dest="taskargs", nargs=2,
> metavar=('recipename', 'taskname'))
> -parser.add_argument("sigdatafile1",
> - help="First signature file to compare (or signature file to
> dump, if second not specified). Not used when using -t/--task.",
> - action="store", nargs='?')
> + parser.add_argument("-s", "--signature",
> + help="With -t/--task, specify the signatures to look for
> instead of taking the last two",
> + action="store", dest="sigargs", nargs=2,
> metavar=('fromsig', 'tosig'))
> -parser.add_argument("sigdatafile2",
> - help="Second signature file to compare",
> - action="store", nargs='?')
> + parser.add_argument("sigdatafile1",
> + help="First signature file to compare (or signature file
> to dump, if second not specified). Not used when using -t/--task.",
> + action="store", nargs='?')
>
> + parser.add_argument("sigdatafile2",
> + help="Second signature file to compare",
> + action="store", nargs='?')
>
> options = parser.parse_args()
> +if is_dump:
> + options.color = 'never'
> + options.dump = True
> + options.sigdatafile2 = None
> + options.sigargs = None
>
> if options.debug:
> logger.setLevel(logging.DEBUG)
> @@ -155,17 +155,32 @@ color = (options.color == 'always' or
> (options.color == 'auto' and sys.stdout.is if options.taskargs:
> with bb.tinfoil.Tinfoil() as tinfoil:
> tinfoil.prepare(config_only=True)
> - if options.sigargs:
> - find_compare_task(tinfoil, options.taskargs[0],
> options.taskargs[1], options.sigargs[0], options.sigargs[1],
> color=color)
> + if not options.dump and options.sigargs:
> + files = find_siginfo_task(tinfoil, options.taskargs[0],
> options.taskargs[1], options.sigargs[0], options.sigargs[1])
> + else:
> + files = find_siginfo_task(tinfoil, options.taskargs[0],
> options.taskargs[1]) +
> + if options.dump:
> + logger.debug("Signature file: %s" % files[-1])
> + output = bb.siggen.dump_sigfile(files[-1])
> else:
> - find_compare_task(tinfoil, options.taskargs[0],
> options.taskargs[1], color=color)
> + if len(files) < 2:
> + logger.error('Only one matching sigdata file found
> for the specified task (%s %s)' % (options.taskargs[0],
> options.taskargs[1]))
> + sys.exit(1)
> +
> + # Recurse into signature comparison
> + logger.debug("Signature file (previous): %s" % files[-2])
> + logger.debug("Signature file (latest): %s" % files[-1])
> + output = bb.siggen.compare_sigfiles(files[-2],
> files[-1], recursecb, color=color) else:
> if options.sigargs:
> logger.error('-s/--signature can only be used together with
> -t/--task') sys.exit(1)
> try:
> - if options.sigdatafile1 and options.sigdatafile2:
> - output =
> bb.siggen.compare_sigfiles(options.sigdatafile1,
> options.sigdatafile2, color=color)
> + if not options.dump and options.sigdatafile1 and
> options.sigdatafile2:
> + with bb.tinfoil.Tinfoil() as tinfoil:
> + tinfoil.prepare(config_only=True)
> + output =
> bb.siggen.compare_sigfiles(options.sigdatafile1,
> options.sigdatafile2, recursecb, color=color) elif
> options.sigdatafile1: output =
> bb.siggen.dump_sigfile(options.sigdatafile1) else: @@ -179,5 +194,5
> @@ else: logger.error('Invalid signature data - ensure you are
> specifying sigdata/siginfo files') sys.exit(1)
>
> - if output:
> - print('\n'.join(output))
> +if output:
> + print('\n'.join(output))
> diff --git a/bitbake/bin/bitbake-dumpsig b/bitbake/bin/bitbake-dumpsig
> deleted file mode 100755
> index 95ebd93..0000000
> --- a/bitbake/bin/bitbake-dumpsig
> +++ /dev/null
> @@ -1,94 +0,0 @@
> -#!/usr/bin/env python3
> -
> -# bitbake-dumpsig
> -# BitBake task signature dump utility
> -#
> -# Copyright (C) 2013 Intel Corporation
> -#
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. -
> -import os
> -import sys
> -import warnings
> -import optparse
> -import logging
> -import pickle
> -
> -sys.path.insert(0,
> os.path.join(os.path.dirname(os.path.dirname(sys.argv[0])), 'lib')) -
> -import bb.tinfoil
> -import bb.siggen
> -import bb.msg
> -
> -logger = bb.msg.logger_create('bitbake-dumpsig')
> -
> -def find_siginfo_task(bbhandler, pn, taskname):
> - """ Find the most recent signature file for the specified
> PN/task """ -
> - if not hasattr(bb.siggen, 'find_siginfo'):
> - logger.error('Metadata does not support finding signature
> data files')
> - sys.exit(1)
> -
> - if not taskname.startswith('do_'):
> - taskname = 'do_%s' % taskname
> -
> - filedates = bb.siggen.find_siginfo(pn, taskname, None,
> bbhandler.config_data)
> - latestfiles = sorted(filedates.keys(), key=lambda f:
> filedates[f])[-1:]
> - if not latestfiles:
> - logger.error('No sigdata files found matching %s %s' % (pn,
> taskname))
> - sys.exit(1)
> -
> - return latestfiles[0]
> -
> -parser = optparse.OptionParser(
> - description = "Dumps siginfo/sigdata files written out by
> BitBake",
> - usage = """
> - %prog -t recipename taskname
> - %prog sigdatafile""")
> -
> -parser.add_option("-D", "--debug",
> - help = "enable debug",
> - action = "store_true", dest="debug", default = False)
> -
> -parser.add_option("-t", "--task",
> - help = "find the signature data file for the specified task",
> - action="store", dest="taskargs", nargs=2,
> metavar='recipename taskname') -
> -options, args = parser.parse_args(sys.argv)
> -
> -if options.debug:
> - logger.setLevel(logging.DEBUG)
> -
> -if options.taskargs:
> - tinfoil = bb.tinfoil.Tinfoil()
> - tinfoil.prepare(config_only = True)
> - file = find_siginfo_task(tinfoil, options.taskargs[0],
> options.taskargs[1])
> - logger.debug("Signature file: %s" % file)
> -elif len(args) == 1:
> - parser.print_help()
> - sys.exit(0)
> -else:
> - file = args[1]
> -
> -try:
> - output = bb.siggen.dump_sigfile(file)
> -except IOError as e:
> - logger.error(str(e))
> - sys.exit(1)
> -except (pickle.UnpicklingError, EOFError):
> - logger.error('Invalid signature data - ensure you are specifying
> a sigdata/siginfo file')
> - sys.exit(1)
> -
> -if output:
> - print('\n'.join(output))
> diff --git a/bitbake/bin/bitbake-dumpsig b/bitbake/bin/bitbake-dumpsig
> new file mode 120000
> index 0000000..b1e8489
> --- /dev/null
> +++ b/bitbake/bin/bitbake-dumpsig
> @@ -0,0 +1 @@
> +bitbake-diffsigs
> \ No newline at end of file
> diff --git a/bitbake/bin/bitbake-hashclient
> b/bitbake/bin/bitbake-hashclient new file mode 100755
> index 0000000..29ab65f
> --- /dev/null
> +++ b/bitbake/bin/bitbake-hashclient
> @@ -0,0 +1,170 @@
> +#! /usr/bin/env python3
> +#
> +# Copyright (C) 2019 Garmin Ltd.
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> +import argparse
> +import hashlib
> +import logging
> +import os
> +import pprint
> +import sys
> +import threading
> +import time
> +
> +try:
> + import tqdm
> + ProgressBar = tqdm.tqdm
> +except ImportError:
> + class ProgressBar(object):
> + def __init__(self, *args, **kwargs):
> + pass
> +
> + def __enter__(self):
> + return self
> +
> + def __exit__(self, *args, **kwargs):
> + pass
> +
> + def update(self):
> + pass
> +
> +sys.path.insert(0,
> os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib')) +
> +import hashserv
> +
> +DEFAULT_ADDRESS = 'unix://./hashserve.sock'
> +METHOD = 'stress.test.method'
> +
> +
> +def main():
> + def handle_stats(args, client):
> + if args.reset:
> + s = client.reset_stats()
> + else:
> + s = client.get_stats()
> + pprint.pprint(s)
> + return 0
> +
> + def handle_stress(args, client):
> + def thread_main(pbar, lock):
> + nonlocal found_hashes
> + nonlocal missed_hashes
> + nonlocal max_time
> +
> + client = hashserv.create_client(args.address)
> +
> + for i in range(args.requests):
> + taskhash = hashlib.sha256()
> + taskhash.update(args.taskhash_seed.encode('utf-8'))
> + taskhash.update(str(i).encode('utf-8'))
> +
> + start_time = time.perf_counter()
> + l = client.get_unihash(METHOD, taskhash.hexdigest())
> + elapsed = time.perf_counter() - start_time
> +
> + with lock:
> + if l:
> + found_hashes += 1
> + else:
> + missed_hashes += 1
> +
> + max_time = max(elapsed, max_time)
> + pbar.update()
> +
> + max_time = 0
> + found_hashes = 0
> + missed_hashes = 0
> + lock = threading.Lock()
> + total_requests = args.clients * args.requests
> + start_time = time.perf_counter()
> + with ProgressBar(total=total_requests) as pbar:
> + threads = [threading.Thread(target=thread_main,
> args=(pbar, lock), daemon=False) for _ in range(args.clients)]
> + for t in threads:
> + t.start()
> +
> + for t in threads:
> + t.join()
> +
> + elapsed = time.perf_counter() - start_time
> + with lock:
> + print("%d requests in %.1fs. %.1f requests per second" %
> (total_requests, elapsed, total_requests / elapsed))
> + print("Average request time %.8fs" % (elapsed /
> total_requests))
> + print("Max request time was %.8fs" % max_time)
> + print("Found %d hashes, missed %d" % (found_hashes,
> missed_hashes)) +
> + if args.report:
> + with ProgressBar(total=args.requests) as pbar:
> + for i in range(args.requests):
> + taskhash = hashlib.sha256()
> +
> taskhash.update(args.taskhash_seed.encode('utf-8'))
> + taskhash.update(str(i).encode('utf-8'))
> +
> + outhash = hashlib.sha256()
> + outhash.update(args.outhash_seed.encode('utf-8'))
> + outhash.update(str(i).encode('utf-8'))
> +
> + client.report_unihash(taskhash.hexdigest(),
> METHOD, outhash.hexdigest(), taskhash.hexdigest()) +
> + with lock:
> + pbar.update()
> +
> + parser = argparse.ArgumentParser(description='Hash Equivalence
> Client')
> + parser.add_argument('--address', default=DEFAULT_ADDRESS,
> help='Server address (default "%(default)s")')
> + parser.add_argument('--log', default='WARNING', help='Set
> logging level') +
> + subparsers = parser.add_subparsers()
> +
> + stats_parser = subparsers.add_parser('stats', help='Show server
> stats')
> + stats_parser.add_argument('--reset', action='store_true',
> + help='Reset server stats')
> + stats_parser.set_defaults(func=handle_stats)
> +
> + stress_parser = subparsers.add_parser('stress', help='Run stress
> test')
> + stress_parser.add_argument('--clients', type=int, default=10,
> + help='Number of simultaneous clients')
> + stress_parser.add_argument('--requests', type=int, default=1000,
> + help='Number of requests each client
> will perform')
> + stress_parser.add_argument('--report', action='store_true',
> + help='Report new hashes')
> + stress_parser.add_argument('--taskhash-seed', default='',
> + help='Include string in taskhash')
> + stress_parser.add_argument('--outhash-seed', default='',
> + help='Include string in outhash')
> + stress_parser.set_defaults(func=handle_stress)
> +
> + args = parser.parse_args()
> +
> + logger = logging.getLogger('hashserv')
> +
> + level = getattr(logging, args.log.upper(), None)
> + if not isinstance(level, int):
> + raise ValueError('Invalid log level: %s' % args.log)
> +
> + logger.setLevel(level)
> + console = logging.StreamHandler()
> + console.setLevel(level)
> + logger.addHandler(console)
> +
> + func = getattr(args, 'func', None)
> + if func:
> + client = hashserv.create_client(args.address)
> + # Try to establish a connection to the server now to detect
> failures
> + # early
> + client.connect()
> +
> + return func(args, client)
> +
> + return 0
> +
> +
> +if __name__ == '__main__':
> + try:
> + ret = main()
> + except Exception:
> + ret = 1
> + import traceback
> + traceback.print_exc()
> + sys.exit(ret)
> diff --git a/bitbake/bin/bitbake-hashserv
> b/bitbake/bin/bitbake-hashserv new file mode 100755
> index 0000000..1bc1f91
> --- /dev/null
> +++ b/bitbake/bin/bitbake-hashserv
> @@ -0,0 +1,62 @@
> +#! /usr/bin/env python3
> +#
> +# Copyright (C) 2018 Garmin Ltd.
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> +import os
> +import sys
> +import logging
> +import argparse
> +import sqlite3
> +
> +sys.path.insert(0,
> os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib')) +
> +import hashserv
> +
> +VERSION = "1.0.0"
> +
> +DEFAULT_BIND = 'unix://./hashserve.sock'
> +
> +
> +def main():
> + parser = argparse.ArgumentParser(description='Hash Equivalence
> Reference Server. Version=%s' % VERSION,
> + epilog='''The bind address is
> the path to a unix domain socket if it is
> + prefixed with
> "unix://". Otherwise, it is an IP address
> + and port in form
> ADDRESS:PORT. To bind to all addresses, leave
> + the ADDRESS empty,
> e.g. "--bind :8686". To bind to a specific
> + IPv6 address, enclose
> the address in "[]", e.g.
> + "--bind [::1]:8686"'''
> + )
> +
> + parser.add_argument('--bind', default=DEFAULT_BIND, help='Bind
> address (default "%(default)s")')
> + parser.add_argument('--database', default='./hashserv.db',
> help='Database file (default "%(default)s")')
> + parser.add_argument('--log', default='WARNING', help='Set
> logging level') +
> + args = parser.parse_args()
> +
> + logger = logging.getLogger('hashserv')
> +
> + level = getattr(logging, args.log.upper(), None)
> + if not isinstance(level, int):
> + raise ValueError('Invalid log level: %s' % args.log)
> +
> + logger.setLevel(level)
> + console = logging.StreamHandler()
> + console.setLevel(level)
> + logger.addHandler(console)
> +
> + server = hashserv.create_server(args.bind, args.database)
> + server.serve_forever()
> + return 0
> +
> +
> +if __name__ == '__main__':
> + try:
> + ret = main()
> + except Exception:
> + ret = 1
> + import traceback
> + traceback.print_exc()
> + sys.exit(ret)
> diff --git a/bitbake/bin/bitbake-layers b/bitbake/bin/bitbake-layers
> index d184011..a884dc1 100755
> --- a/bitbake/bin/bitbake-layers
> +++ b/bitbake/bin/bitbake-layers
> @@ -7,18 +7,8 @@
> # Copyright (C) 2011 Mentor Graphics Corporation
> # Copyright (C) 2011-2015 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import logging
> import os
> diff --git a/bitbake/bin/bitbake-prserv b/bitbake/bin/bitbake-prserv
> index f38d2dd..1e9b6cb 100755
> --- a/bitbake/bin/bitbake-prserv
> +++ b/bitbake/bin/bitbake-prserv
> @@ -1,4 +1,8 @@
> #!/usr/bin/env python3
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> import os
> import sys,logging
> import optparse
> diff --git a/bitbake/bin/bitbake-selftest
> b/bitbake/bin/bitbake-selftest index cfa7ac5..041a271 100755
> --- a/bitbake/bin/bitbake-selftest
> +++ b/bitbake/bin/bitbake-selftest
> @@ -2,18 +2,8 @@
> #
> # Copyright (C) 2012 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import os
> import sys, logging
> @@ -22,6 +12,7 @@ sys.path.insert(0,
> os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib import
> unittest try:
> import bb
> + import hashserv
> import layerindexlib
> except RuntimeError as exc:
> sys.exit(str(exc))
> @@ -33,7 +24,10 @@ tests = ["bb.tests.codeparser",
> "bb.tests.event",
> "bb.tests.fetch",
> "bb.tests.parse",
> + "bb.tests.persist_data",
> + "bb.tests.runqueue",
> "bb.tests.utils",
> + "hashserv.tests",
> "layerindexlib.tests.layerindexobj",
> "layerindexlib.tests.restapi",
> "layerindexlib.tests.cooker"]
> diff --git a/bitbake/bin/bitbake-worker b/bitbake/bin/bitbake-worker
> index e925054..6776cad 100755
> --- a/bitbake/bin/bitbake-worker
> +++ b/bitbake/bin/bitbake-worker
> @@ -1,4 +1,7 @@
> #!/usr/bin/env python3
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
>
> import os
> import sys
> @@ -136,7 +139,7 @@ def sigterm_handler(signum, frame):
> os.killpg(0, signal.SIGTERM)
> sys.exit()
>
> -def fork_off_task(cfg, data, databuilder, workerdata, fn, task,
> taskname, appends, taskdepdata, extraconfigdata, quieterrors=False,
> dry_run_exec=False): +def fork_off_task(cfg, data, databuilder,
> workerdata, fn, task, taskname, taskhash, unihash, appends,
> taskdepdata, extraconfigdata, quieterrors=False, dry_run_exec=False):
> # We need to setup the environment BEFORE the fork, since # a fork()
> or exec*() activates PSEUDO... @@ -231,10 +234,13 @@ def
> fork_off_task(cfg, data, databuilder, workerdata, fn, task, taskname,
> append the_data.setVar(varname, value)
> bb.parse.siggen.set_taskdata(workerdata["sigdata"])
> + if "newhashes" in workerdata:
> +
> bb.parse.siggen.set_taskhashes(workerdata["newhashes"]) ret = 0
>
> the_data = bb_cache.loadDataFull(fn, appends)
> - the_data.setVar('BB_TASKHASH',
> workerdata["runq_hash"][task])
> + the_data.setVar('BB_TASKHASH', taskhash)
> + the_data.setVar('BB_UNIHASH', unihash)
>
> bb.utils.set_process_name("%s:%s" %
> (the_data.getVar("PN"), taskname.replace("do_", "")))
> @@ -373,6 +379,7 @@ class BitbakeWorker(object):
> self.handle_item(b"cookerconfig",
> self.handle_cookercfg) self.handle_item(b"extraconfigdata",
> self.handle_extraconfigdata) self.handle_item(b"workerdata",
> self.handle_workerdata)
> + self.handle_item(b"newtaskhashes",
> self.handle_newtaskhashes) self.handle_item(b"runtask",
> self.handle_runtask) self.handle_item(b"finishnow",
> self.handle_finishnow) self.handle_item(b"ping", self.handle_ping)
> @@ -411,6 +418,10 @@ class BitbakeWorker(object):
> bb.msg.loggerDefaultDomains =
> self.workerdata["logdefaultdomain"] for mc in self.databuilder.mcdata:
> self.databuilder.mcdata[mc].setVar("PRSERV_HOST",
> self.workerdata["prhost"])
> + self.databuilder.mcdata[mc].setVar("BB_HASHSERVE",
> self.workerdata["hashservaddr"]) +
> + def handle_newtaskhashes(self, data):
> + self.workerdata["newhashes"] = pickle.loads(data)
>
> def handle_ping(self, _):
> workerlog_write("Handling ping\n")
> @@ -425,10 +436,10 @@ class BitbakeWorker(object):
> sys.exit(0)
>
> def handle_runtask(self, data):
> - fn, task, taskname, quieterrors, appends, taskdepdata,
> dry_run_exec = pickle.loads(data)
> + fn, task, taskname, taskhash, unihash, quieterrors, appends,
> taskdepdata, dry_run_exec = pickle.loads(data)
> workerlog_write("Handling runtask %s %s %s\n" % (task, fn, taskname))
> - pid, pipein, pipeout = fork_off_task(self.cookercfg,
> self.data, self.databuilder, self.workerdata, fn, task, taskname,
> appends, taskdepdata, self.extraconfigdata, quieterrors, dry_run_exec)
> + pid, pipein, pipeout = fork_off_task(self.cookercfg,
> self.data, self.databuilder, self.workerdata, fn, task, taskname,
> taskhash, unihash, appends, taskdepdata, self.extraconfigdata,
> quieterrors, dry_run_exec) self.build_pids[pid] = task
> self.build_pipes[pid] = runQueueWorkerPipe(pipein, pipeout) diff
> --git a/bitbake/bin/bitdoc b/bitbake/bin/bitdoc index
> 2744678..9bd02be 100755 --- a/bitbake/bin/bitdoc
> +++ b/bitbake/bin/bitdoc
> @@ -1,21 +1,9 @@
> #!/usr/bin/env python3
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # Copyright (C) 2005 Holger Hans Peter Freyther
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import optparse, os, sys
>
> diff --git a/bitbake/bin/git-make-shallow
> b/bitbake/bin/git-make-shallow index 296d3a3..57069f7 100755
> --- a/bitbake/bin/git-make-shallow
> +++ b/bitbake/bin/git-make-shallow
> @@ -1,4 +1,8 @@
> #!/usr/bin/env python3
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> """git-make-shallow: make the current git repository shallow
>
> Remove the history of the specified revisions, then optionally
> filter the diff --git a/bitbake/bin/toaster b/bitbake/bin/toaster
> index ecf66fa..c3472df 100755
> --- a/bitbake/bin/toaster
> +++ b/bitbake/bin/toaster
> @@ -3,19 +3,9 @@
> # toaster - shell script to start Toaster
>
> # Copyright (C) 2013-2015 Intel Corp.
> -
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License as
> published by -# the Free Software Foundation; either version 2 of the
> License, or -# (at your option) any later version.
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# SPDX-License-Identifier: GPL-2.0-or-later
> #
> -# You should have received a copy of the GNU General Public License
> -# along with this program. If not, see http://www.gnu.org/licenses/.
>
> HELP="
> Usage: source toaster start|stop [webport=<address:port>] [noweb]
> [nobuild] [toasterdir] diff --git a/bitbake/bin/toaster-eventreplay
> b/bitbake/bin/toaster-eventreplay index 80967a0..8fa4ab7 100755
> --- a/bitbake/bin/toaster-eventreplay
> +++ b/bitbake/bin/toaster-eventreplay
> @@ -1,25 +1,12 @@
> #!/usr/bin/env python3
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # Copyright (C) 2014 Alex Damian
> #
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> # This file re-uses code spread throughout other Bitbake source
> files. # As such, all other copyrights belong to their own right
> holders. #
> -#
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> """
> This command takes a filename as a single parameter. The filename is
> read diff --git a/bitbake/classes/base.bbclass
> b/bitbake/classes/base.bbclass index 71d9ec1..08441fe 100644
> --- a/bitbake/classes/base.bbclass
> +++ b/bitbake/classes/base.bbclass
> @@ -44,7 +44,7 @@ python do_showdata() {
> # emit the metadata which isnt valid shell
> for e in bb.data.keys(d):
> if d.getVarFlag(e, 'python', False):
> - bb.plain("\npython %s () {\n%s}" % (e,
> d.getVar(e, True)))
> + bb.plain("\npython %s () {\n%s}" % (e,
> d.getVar(e))) }
>
> addtask listtasks
> diff --git a/bitbake/contrib/dump_cache.py
> b/bitbake/contrib/dump_cache.py index 8963ca4..c6723cb 100755
> --- a/bitbake/contrib/dump_cache.py
> +++ b/bitbake/contrib/dump_cache.py
> @@ -1,6 +1,4 @@
> #!/usr/bin/env python3
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # Copyright (C) 2012, 2018 Wind River Systems, Inc.
> #
> diff --git
> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
> index f1caaec..46dafee 100644 ---
> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
> +++
> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
> @@ -31,7 +31,7 @@ <para> Prior to executing BitBake, you should take
> advantage of available parallel thread execution on your build host
> by setting the
> - <link
> linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
> + <link
> linkend='var-bb-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
> variable in your project's <filename>local.conf</filename>
> configuration file. </para>
> @@ -87,9 +87,9 @@
> <para>
> The <filename>layer.conf</filename> files are used to
> construct key variables such as
> - <link
> linkend='var-BBPATH'><filename>BBPATH</filename></link>
> + <link
> linkend='var-bb-BBPATH'><filename>BBPATH</filename></link> and
> - <link
> linkend='var-BBFILES'><filename>BBFILES</filename></link>.
> + <link
> linkend='var-bb-BBFILES'><filename>BBFILES</filename></link>.
> <filename>BBPATH</filename> is used to search for configuration and
> class files under the <filename>conf</filename> and
> <filename>classes</filename> @@ -117,19 +117,19 @@
> at certain variables, including:
> <itemizedlist>
> <listitem><para>
> - <link
> linkend='var-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>
> + <link
> linkend='var-bb-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>
> </para></listitem> <listitem><para>
> - <link
> linkend='var-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link>
> + <link
> linkend='var-bb-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link>
> </para></listitem> <listitem><para>
> - <link
> linkend='var-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link>
> + <link
> linkend='var-bb-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link>
> </para></listitem> <listitem><para>
> - <link
> linkend='var-BB_ORIGENV'><filename>BB_ORIGENV</filename></link>
> + <link
> linkend='var-bb-BB_ORIGENV'><filename>BB_ORIGENV</filename></link>
> </para></listitem> <listitem><para>
> - <link
> linkend='var-BITBAKE_UI'><filename>BITBAKE_UI</filename></link>
> + <link
> linkend='var-bb-BITBAKE_UI'><filename>BITBAKE_UI</filename></link>
> </para></listitem> </itemizedlist>
> The first four variables in this list relate to how
> BitBake treats shell @@ -156,7 +156,7 @@
> BitBake first searches the current working directory for
> an optional <filename>conf/bblayers.conf</filename> configuration
> file. This file is expected to contain a
> - <link
> linkend='var-BBLAYERS'><filename>BBLAYERS</filename></link>
> + <link
> linkend='var-bb-BBLAYERS'><filename>BBLAYERS</filename></link>
> variable that is a space-delimited list of 'layer' directories.
> Recall that if BitBake cannot find a
> <filename>bblayers.conf</filename> file, then it is assumed the user
> has set the <filename>BBPATH</filename> @@ -166,10 +166,10 @@ <para>
> For each directory (layer) in this list, a
> <filename>conf/layer.conf</filename> file is located and parsed with
> the
> - <link
> linkend='var-LAYERDIR'><filename>LAYERDIR</filename></link>
> + <link
> linkend='var-bb-LAYERDIR'><filename>LAYERDIR</filename></link>
> variable being set to the directory where the layer was found. The
> idea is these files automatically set up
> - <link
> linkend='var-BBPATH'><filename>BBPATH</filename></link>
> + <link
> linkend='var-bb-BBPATH'><filename>BBPATH</filename></link> and other
> variables correctly for a given build directory. </para>
>
> @@ -189,7 +189,7 @@
> depending on the environment variables previously
> mentioned or set in the configuration files.
> The
> - "<link linkend='ref-variables-glos'>Variables
> Glossary</link>"
> + "<link linkend='ref-bb-variables-glos'>Variables
> Glossary</link>" chapter presents a full list of variables.
> </para>
>
> @@ -204,7 +204,7 @@
> <para>
> The <filename>base.bbclass</filename> file is always
> included. Other classes that are specified in the configuration using
> the
> - <link
> linkend='var-INHERIT'><filename>INHERIT</filename></link>
> + <link
> linkend='var-bb-INHERIT'><filename>INHERIT</filename></link> variable
> are also included. BitBake searches for class files in a
> <filename>classes</filename> subdirectory under
> @@ -270,7 +270,7 @@
>
> <para>
> During the configuration phase, BitBake will have set
> - <link
> linkend='var-BBFILES'><filename>BBFILES</filename></link>.
> + <link
> linkend='var-bb-BBFILES'><filename>BBFILES</filename></link>. BitBake
> now uses it to construct a list of recipes to parse, along with any
> append files (<filename>.bbappend</filename>) to apply.
> @@ -292,7 +292,7 @@
> Any inherit statements cause BitBake to find and
> then parse class files (<filename>.bbclass</filename>)
> using
> - <link
> linkend='var-BBPATH'><filename>BBPATH</filename></link>
> + <link
> linkend='var-bb-BBPATH'><filename>BBPATH</filename></link> as the
> search path. Finally, BitBake parses in order any append files found
> in <filename>BBFILES</filename>.
> @@ -303,8 +303,8 @@
> pieces of metadata.
> For example, in <filename>bitbake.conf</filename> the
> recipe name and version are used to set the variables
> - <link linkend='var-PN'><filename>PN</filename></link> and
> - <link linkend='var-PV'><filename>PV</filename></link>:
> + <link linkend='var-bb-PN'><filename>PN</filename></link>
> and
> + <link linkend='var-bb-PV'><filename>PV</filename></link>:
> <literallayout class='monospaced'>
> PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE',
> False),d)[0] or 'defaultpkgname'}" PV =
> "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[1]
> or '1.0'}" @@ -336,7 +336,7 @@ recipe information.
> The validity of this cache is determined by first
> computing a checksum of the base configuration data (see
> - <link
> linkend='var-BB_HASHCONFIG_WHITELIST'><filename>BB_HASHCONFIG_WHITELIST</filename></link>)
> + <link
> linkend='var-bb-BB_HASHCONFIG_WHITELIST'><filename>BB_HASHCONFIG_WHITELIST</filename></link>)
> and then checking if the checksum matches. If that checksum matches
> what is in the cache and the recipe and class files have not changed,
> Bitbake is able to use @@ -384,9 +384,9 @@
> the recipe can be known.
> Each recipe's <filename>PROVIDES</filename> list is
> created implicitly through the recipe's
> - <link linkend='var-PN'><filename>PN</filename></link>
> variable
> + <link linkend='var-bb-PN'><filename>PN</filename></link>
> variable and explicitly through the recipe's
> - <link
> linkend='var-PROVIDES'><filename>PROVIDES</filename></link>
> + <link
> linkend='var-bb-PROVIDES'><filename>PROVIDES</filename></link>
> variable, which is optional. </para>
>
> @@ -427,7 +427,7 @@
> PREFERRED_PROVIDER_virtual/kernel = "linux-yocto"
> </literallayout>
> The default
> - <link
> linkend='var-PREFERRED_PROVIDER'><filename>PREFERRED_PROVIDER</filename></link>
> + <link
> linkend='var-bb-PREFERRED_PROVIDER'><filename>PREFERRED_PROVIDER</filename></link>
> is the provider with the same name as the target. Bitbake iterates
> through each target it needs to build and resolves them and their
> dependencies using this process. @@ -439,10 +439,10 @@
> BitBake defaults to the highest version of a provider.
> Version comparisons are made using the same method as
> Debian. You can use the
> - <link
> linkend='var-PREFERRED_VERSION'><filename>PREFERRED_VERSION</filename></link>
> + <link
> linkend='var-bb-PREFERRED_VERSION'><filename>PREFERRED_VERSION</filename></link>
> variable to specify a particular version. You can influence the order
> by using the
> - <link
> linkend='var-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link>
> + <link
> linkend='var-bb-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link>
> variable. </para>
>
> @@ -464,7 +464,7 @@
> BitBake defaults to selecting the most recent
> version, unless otherwise specified.
> If the recipe in question has a
> - <link
> linkend='var-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link>
> + <link
> linkend='var-bb-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link>
> set lower than the other recipes (default is 0), then it will not be
> selected. This allows the person or persons maintaining
> @@ -475,9 +475,9 @@
>
> <para>
> If the first recipe is named
> <filename>a_1.1.bb</filename>, then the
> - <link linkend='var-PN'><filename>PN</filename></link>
> variable
> + <link linkend='var-bb-PN'><filename>PN</filename></link>
> variable will be set to “a”, and the
> - <link linkend='var-PV'><filename>PV</filename></link>
> + <link linkend='var-bb-PV'><filename>PV</filename></link>
> variable will be set to 1.1.
> </para>
>
> @@ -532,11 +532,11 @@
> <para>
> Dependencies are defined through several variables.
> You can find information about variables BitBake uses in
> - the <link linkend='ref-variables-glos'>Variables
> Glossary</link>
> + the <link linkend='ref-bb-variables-glos'>Variables
> Glossary</link> near the end of this manual.
> At a basic level, it is sufficient to know that BitBake
> uses the
> - <link
> linkend='var-DEPENDS'><filename>DEPENDS</filename></link> and
> - <link
> linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link> variables
> when
> + <link
> linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link> and
> + <link
> linkend='var-bb-RDEPENDS'><filename>RDEPENDS</filename></link>
> variables when calculating dependencies. </para>
>
> @@ -560,7 +560,7 @@
>
> <para>
> The build now starts with BitBake forking off threads up
> to the limit set in the
> - <link
> linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
> + <link
> linkend='var-bb-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
> variable. BitBake continues to fork threads as long as there are
> tasks ready to run, those tasks have all their dependencies met, and
> the thread threshold has not been @@ -574,7 +574,7 @@
>
> <para>
> As each task completes, a timestamp is written to the
> directory specified by the
> - <link
> linkend='var-STAMP'><filename>STAMP</filename></link> variable.
> + <link
> linkend='var-bb-STAMP'><filename>STAMP</filename></link> variable. On
> subsequent runs, BitBake looks in the build directory within
> <filename>tmp/stamps</filename> and does not rerun tasks that are
> already completed unless a timestamp is found to be invalid. @@
> -618,7 +618,7 @@ <para>
> Tasks can be either a shell task or a Python task.
> For shell tasks, BitBake writes a shell script to
> - <filename>${</filename><link
> linkend='var-T'><filename>T</filename></link><filename>}/run.do_taskname.pid</filename>
> + <filename>${</filename><link
> linkend='var-bb-T'><filename>T</filename></link><filename>}/run.do_taskname.pid</filename>
> and then executes the script. The generated shell script contains all
> the exported variables, and the shell functions with all variables
> expanded. @@ -645,10 +645,10 @@
> behavior:
> <itemizedlist>
> <listitem><para>
> - <link
> linkend='var-BB_SCHEDULER'><filename>BB_SCHEDULER</filename></link>
> + <link
> linkend='var-bb-BB_SCHEDULER'><filename>BB_SCHEDULER</filename></link>
> </para></listitem> <listitem><para>
> - <link
> linkend='var-BB_SCHEDULERS'><filename>BB_SCHEDULERS</filename></link>
> + <link
> linkend='var-bb-BB_SCHEDULERS'><filename>BB_SCHEDULERS</filename></link>
> </para></listitem> </itemizedlist>
> It is possible to have functions run before and after a
> task's main @@ -684,7 +684,7 @@
> The simplistic approach for excluding the working
> directory is to set it to some fixed value and create the checksum
> for the "run" script. BitBake goes one step better and uses the
> - <link
> linkend='var-BB_HASHBASE_WHITELIST'><filename>BB_HASHBASE_WHITELIST</filename></link>
> + <link
> linkend='var-bb-BB_HASHBASE_WHITELIST'><filename>BB_HASHBASE_WHITELIST</filename></link>
> variable to define a list of variables that should never be included
> when generating the signatures. </para>
> @@ -795,7 +795,7 @@
> This results in any metadata change that changes the
> task hash, automatically causing the task to be run again.
> This removes the need to bump
> - <link linkend='var-PR'><filename>PR</filename></link>
> + <link linkend='var-bb-PR'><filename>PR</filename></link>
> values, and changes to metadata automatically ripple
> across the build. </para>
>
> @@ -884,7 +884,7 @@
>
> <para>
> BitBake first calls the function defined by the
> - <link
> linkend='var-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></link>
> + <link
> linkend='var-bb-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></link>
> variable with a list of tasks and corresponding hashes it wants to
> build. This function is designed to be fast and returns a list
> @@ -908,7 +908,7 @@
> For example, it is pointless to obtain a compiler if you
> already have the compiled binary.
> To handle this, BitBake calls the
> - <link
> linkend='var-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></link>
> + <link
> linkend='var-bb-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></link>
> function for each successful setscene task to know whether or not it
> needs to obtain the dependencies of that task. </para>
> @@ -916,7 +916,7 @@
> <para>
> Finally, after all the setscene tasks have executed,
> BitBake calls the function listed in
> - <link
> linkend='var-BB_SETSCENE_VERIFY_FUNCTION2'><filename>BB_SETSCENE_VERIFY_FUNCTION2</filename></link>
> + <link
> linkend='var-bb-BB_SETSCENE_VERIFY_FUNCTION2'><filename>BB_SETSCENE_VERIFY_FUNCTION2</filename></link>
> with the list of tasks BitBake thinks has been "covered". The
> metadata can then ensure that this list is correct and can inform
> BitBake that it wants specific tasks to be run regardless diff --git
> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml
> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml
> index 29ae486..6840408 100644 ---
> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml
> +++
> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml @@
> -44,7 +44,7 @@ </literallayout> This code sets up an instance of the
> fetch class. The instance uses a space-separated list of URLs from the
> - <link
> linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
> + <link
> linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link> variable
> and then calls the <filename>download</filename> method to download
> the files. </para>
> @@ -78,7 +78,7 @@
> <listitem><para><emphasis>Pre-mirror
> Sites:</emphasis> BitBake first uses pre-mirrors to try and find
> source files. These locations are defined using the
> - <link
> linkend='var-PREMIRRORS'><filename>PREMIRRORS</filename></link>
> + <link
> linkend='var-bb-PREMIRRORS'><filename>PREMIRRORS</filename></link>
> variable. </para></listitem>
> <listitem><para><emphasis>Source URI:</emphasis>
> @@ -88,7 +88,7 @@
> <listitem><para><emphasis>Mirror Sites:</emphasis>
> If fetch failures occur, BitBake next uses
> mirror locations as defined by the
> - <link
> linkend='var-MIRRORS'><filename>MIRRORS</filename></link>
> + <link
> linkend='var-bb-MIRRORS'><filename>MIRRORS</filename></link> variable.
> </para></listitem>
> </itemizedlist>
> @@ -144,7 +144,7 @@
> Any source files that are not local (i.e.
> downloaded from the Internet) are placed into the
> download directory, which is specified by the
> - <link
> linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
> + <link
> linkend='var-bb-DL_DIR'><filename>DL_DIR</filename></link> variable.
> </para>
>
> @@ -184,11 +184,11 @@
>
> <para>
> If
> - <link
> linkend='var-BB_STRICT_CHECKSUM'><filename>BB_STRICT_CHECKSUM</filename></link>
> + <link
> linkend='var-bb-BB_STRICT_CHECKSUM'><filename>BB_STRICT_CHECKSUM</filename></link>
> is set, any download without a checksum triggers an error message.
> The
> - <link
> linkend='var-BB_NO_NETWORK'><filename>BB_NO_NETWORK</filename></link>
> + <link
> linkend='var-bb-BB_NO_NETWORK'><filename>BB_NO_NETWORK</filename></link>
> variable can be used to make any attempted network access a fatal
> error, which is useful for checking that mirrors are complete as well
> as other things. @@ -265,11 +265,11 @@
> The filename you specify within the URL can be
> either an absolute or relative path to a file.
> If the filename is relative, the contents of the
> - <link
> linkend='var-FILESPATH'><filename>FILESPATH</filename></link>
> + <link
> linkend='var-bb-FILESPATH'><filename>FILESPATH</filename></link>
> variable is used in the same way <filename>PATH</filename> is used to
> find executables. If the file cannot be found, it is assumed that it
> is available in
> - <link
> linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
> + <link
> linkend='var-bb-DL_DIR'><filename>DL_DIR</filename></link> by the
> time the <filename>download()</filename> method is called. </para>
>
> @@ -304,7 +304,7 @@
> allows the name of the downloaded file to be
> specified. Specifying the name of the downloaded file is useful
> for avoiding collisions in
> - <link
> linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
> + <link
> linkend='var-bb-DL_DIR'><filename>DL_DIR</filename></link> when
> dealing with multiple files that have the same name. </para>
>
> @@ -355,7 +355,7 @@
> A special value of "now" causes the checkout
> to be updated on every build.
> </para></listitem>
> - <listitem><para><emphasis><link
> linkend='var-CVSDIR'><filename>CVSDIR</filename></link>:</emphasis>
> + <listitem><para><emphasis><link
> linkend='var-bb-CVSDIR'><filename>CVSDIR</filename></link>:</emphasis>
> Specifies where a temporary checkout is saved. The location is often
> <filename>DL_DIR/cvs</filename>. </para></listitem>
> @@ -395,7 +395,7 @@
> <listitem><para><emphasis>"date":</emphasis>
> Specifies a date.
> If no "date" is specified, the
> - <link
> linkend='var-SRCDATE'><filename>SRCDATE</filename></link>
> + <link
> linkend='var-bb-SRCDATE'><filename>SRCDATE</filename></link> of the
> configuration is used to checkout a specific date. The special value
> of "now" causes the checkout to be updated on every build.
> @@ -406,7 +406,7 @@
> to which the module is unpacked.
> You are forcing the module into a special
> directory relative to
> - <link
> linkend='var-CVSDIR'><filename>CVSDIR</filename></link>.
> + <link
> linkend='var-bb-CVSDIR'><filename>CVSDIR</filename></link>.
> </para></listitem> <listitem><para><emphasis>"rsh"</emphasis>
> Used in conjunction with the "method"
> parameter. @@ -448,7 +448,7 @@
> <filename>FETCHCMD_svn</filename>, which defaults
> to "svn".
> The fetcher's temporary working directory is set by
> - <link
> linkend='var-SVNDIR'><filename>SVNDIR</filename></link>,
> + <link
> linkend='var-bb-SVNDIR'><filename>SVNDIR</filename></link>, which is
> usually <filename>DL_DIR/svn</filename>. </para>
>
> @@ -509,7 +509,7 @@
> source control system.
> The fetcher works by creating a bare clone of the
> remote into
> - <link
> linkend='var-GITDIR'><filename>GITDIR</filename></link>,
> + <link
> linkend='var-bb-GITDIR'><filename>GITDIR</filename></link>, which is
> usually <filename>DL_DIR/git2</filename>. This bare clone is then
> cloned into the work directory during the unpack stage when a
> specific tree is checked out. @@ -588,6 +588,14 @@
> The name of the path in which to place the
> checkout. By default, the path is <filename>git/</filename>.
> </para></listitem>
> + <listitem><para><emphasis>"usehead":</emphasis>
> + Enables local <filename>git://</filename>
> URLs to use the
> + current branch HEAD as the revision for use
> with
> + <filename>AUTOREV</filename>.
> + The "usehead" parameter implies no branch
> and only works
> + when the transfer protocol is
> + <filename>file://</filename>.
> + </para></listitem>
> </itemizedlist>
> Here are some example URLs:
> <literallayout class='monospaced'>
> @@ -604,7 +612,7 @@
> This fetcher submodule inherits from the
> <link linkend='git-fetcher'>Git fetcher</link> and
> extends that fetcher's behavior by fetching a repository's submodules.
> - <link
> linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
> + <link
> linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link> is
> passed to the Git fetcher as described in the "<link
> linkend='git-fetcher'>Git Fetcher
> (<filename>git://</filename>)</link>" section. @@ -639,9 +647,9 @@
>
> <para>
> To use this fetcher, make sure your recipe has proper
> - <link
> linkend='var-SRC_URI'><filename>SRC_URI</filename></link>,
> - <link
> linkend='var-SRCREV'><filename>SRCREV</filename></link>, and
> - <link
> linkend='var-PV'><filename>PV</filename></link> settings.
> + <link
> linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link>,
> + <link
> linkend='var-bb-SRCREV'><filename>SRCREV</filename></link>, and
> + <link
> linkend='var-bb-PV'><filename>PV</filename></link> settings. Here is
> an example: <literallayout class='monospaced'>
> SRC_URI =
> "ccrc://cc.example.org/ccrc;vob=/example_vob;module=/example_module"
> @@ -726,15 +734,15 @@ <filename>FETCHCMD_p4</filename>, which defaults
> to "p4".
> The fetcher's temporary working directory is set by
> - <link
> linkend='var-P4DIR'><filename>P4DIR</filename></link>,
> + <link
> linkend='var-bb-P4DIR'><filename>P4DIR</filename></link>, which
> defaults to "DL_DIR/p4". </para>
>
> <para>
> To use this fetcher, make sure your recipe has proper
> - <link
> linkend='var-SRC_URI'><filename>SRC_URI</filename></link>,
> - <link
> linkend='var-SRCREV'><filename>SRCREV</filename></link>, and
> - <link
> linkend='var-PV'><filename>PV</filename></link> values.
> + <link
> linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link>,
> + <link
> linkend='var-bb-SRCREV'><filename>SRCREV</filename></link>, and
> + <link
> linkend='var-bb-PV'><filename>PV</filename></link> values. The p4
> executable is able to use the config file defined by your system's
> <filename>P4CONFIG</filename> environment variable in order to define
> the Perforce server URL and port, username, and @@ -785,9 +793,9 @@
> <filename>google-repo</filename> source control
> system. The fetcher works by initiating and syncing sources of the
> repository into
> - <link
> linkend='var-REPODIR'><filename>REPODIR</filename></link>,
> + <link
> linkend='var-bb-REPODIR'><filename>REPODIR</filename></link>, which
> is usually
> - <link
> linkend='var-DL_DIR'><filename>DL_DIR</filename></link><filename>/repo</filename>.
> + <link
> linkend='var-bb-DL_DIR'><filename>DL_DIR</filename></link><filename>/repo</filename>.
> </para>
> <para>
> @@ -824,19 +832,22 @@
> Bazaar (<filename>bzr://</filename>)
> </para></listitem>
> <listitem><para>
> - Trees using Git Annex
> (<filename>gitannex://</filename>)
> + Mercurial (<filename>hg://</filename>)
> </para></listitem>
> <listitem><para>
> - Secure FTP (<filename>sftp://</filename>)
> + npm (<filename>npm://</filename>)
> </para></listitem>
> <listitem><para>
> - Secure Shell (<filename>ssh://</filename>)
> + OSC (<filename>osc://</filename>)
> </para></listitem>
> <listitem><para>
> - OSC (<filename>osc://</filename>)
> + Secure FTP (<filename>sftp://</filename>)
> </para></listitem>
> <listitem><para>
> - Mercurial (<filename>hg://</filename>)
> + Secure Shell (<filename>ssh://</filename>)
> + </para></listitem>
> + <listitem><para>
> + Trees using Git Annex
> (<filename>gitannex://</filename>) </para></listitem>
> </itemizedlist>
> No documentation currently exists for these lesser
> used diff --git
> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.xml
> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.xml index
> 9076f0f..39066e4 100644 ---
> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.xml +++
> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.xml @@
> -194,7 +194,7 @@ <para> When you run BitBake, it begins looking for
> metadata files. The
> - <link
> linkend='var-BBPATH'><filename>BBPATH</filename></link>
> + <link
> linkend='var-bb-BBPATH'><filename>BBPATH</filename></link> variable
> is what tells BitBake where to look for those files.
> <filename>BBPATH</filename> is not set and you need to set it.
> Without <filename>BBPATH</filename>, Bitbake cannot @@ -273,14
> +273,14 @@ some editor to create the <filename>bitbake.conf</filename>
> so that it contains the following:
> <literallayout class='monospaced'>
> - <link linkend='var-PN'>PN</link> =
> "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0]
> or 'defaultpkgname'}"
> + <link linkend='var-bb-PN'>PN</link> =
> "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0]
> or 'defaultpkgname'}" </literallayout> <literallayout
> class='monospaced'>
> - TMPDIR = "${<link linkend='var-TOPDIR'>TOPDIR</link>}/tmp"
> - <link linkend='var-CACHE'>CACHE</link> = "${TMPDIR}/cache"
> - <link linkend='var-STAMP'>STAMP</link> =
> "${TMPDIR}/${PN}/stamps"
> - <link linkend='var-T'>T</link> = "${TMPDIR}/${PN}/work"
> - <link linkend='var-B'>B</link> = "${TMPDIR}/${PN}"
> + TMPDIR = "${<link linkend='var-bb-TOPDIR'>TOPDIR</link>}/tmp"
> + <link linkend='var-bb-CACHE'>CACHE</link> = "${TMPDIR}/cache"
> + <link linkend='var-bb-STAMP'>STAMP</link> =
> "${TMPDIR}/${PN}/stamps"
> + <link linkend='var-bb-T'>T</link> = "${TMPDIR}/${PN}/work"
> + <link linkend='var-bb-B'>B</link> = "${TMPDIR}/${PN}"
> </literallayout>
> <note>
> Without a value for <filename>PN</filename>, the
> @@ -402,12 +402,12 @@
> Move to the <filename>conf</filename> directory and
> create a <filename>layer.conf</filename> file that has the following:
> <literallayout class='monospaced'>
> - BBPATH .= ":${<link linkend='var-LAYERDIR'>LAYERDIR</link>}"
> + BBPATH .= ":${<link linkend='var-bb-LAYERDIR'>LAYERDIR</link>}"
>
> - <link linkend='var-BBFILES'>BBFILES</link> += "${LAYERDIR}/*.bb"
> + <link linkend='var-bb-BBFILES'>BBFILES</link> +=
> "${LAYERDIR}/*.bb"
> - <link
> linkend='var-BBFILE_COLLECTIONS'>BBFILE_COLLECTIONS</link> +=
> "mylayer"
> - <link
> linkend='var-BBFILE_PATTERN'>BBFILE_PATTERN_mylayer</link> :=
> "^${LAYERDIR_RE}/"
> + <link
> linkend='var-bb-BBFILE_COLLECTIONS'>BBFILE_COLLECTIONS</link> +=
> "mylayer"
> + <link
> linkend='var-bb-BBFILE_PATTERN'>BBFILE_PATTERN_mylayer</link> :=
> "^${LAYERDIR_RE}/" </literallayout> For information on these
> variables, click the links to go to the definitions in the
> glossary.</para> @@ -416,9 +416,9 @@
> a recipe file named
> <filename>printhello.bb</filename> that has the following:
> <literallayout class='monospaced'>
> - <link linkend='var-DESCRIPTION'>DESCRIPTION</link> = "Prints
> Hello World"
> - <link linkend='var-PN'>PN</link> = 'printhello'
> - <link linkend='var-PV'>PV</link> = '1'
> + <link linkend='var-bb-DESCRIPTION'>DESCRIPTION</link> = "Prints
> Hello World"
> + <link linkend='var-bb-PN'>PN</link> = 'printhello'
> + <link linkend='var-bb-PV'>PV</link> = '1'
>
> python do_build() {
> bb.plain("********************");
> diff --git
> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml
> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml index
> f7d312a..8f2a960 100644 ---
> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml +++
> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml @@
> -781,7 +781,7 @@ target, you must also enable BitBake to perform
> multiple configuration builds. Enabling is accomplished by setting the
> - <link
> linkend='var-BBMULTICONFIG'><filename>BBMULTICONFIG</filename></link>
> + <link
> linkend='var-bb-BBMULTICONFIG'><filename>BBMULTICONFIG</filename></link>
> variable in the <filename>local.conf</filename> configuration file.
> As an example, suppose you had configuration
> files @@ -791,7 +791,7 @@
> The following statement in the
> <filename>local.conf</filename> file both enables
> BitBake to perform multiple configuration builds
> and
> - specifies the two multiconfigs:
> + specifies the two extra multiconfigs:
> <literallayout class='monospaced'>
> BBMULTICONFIG = "target1 target2"
> </literallayout>
> @@ -803,13 +803,13 @@
> builds, use the following command form to start
> the builds:
> <literallayout class='monospaced'>
> - $ bitbake
> [multiconfig:<replaceable>multiconfigname</replaceable>:]<replaceable>target</replaceable>
> [[[multiconfig:<replaceable>multiconfigname</replaceable>:]<replaceable>target</replaceable>] ... ]
> + $ bitbake
> [mc:<replaceable>multiconfigname</replaceable>:]<replaceable>target</replaceable>
> [[[mc:<replaceable>multiconfigname</replaceable>:]<replaceable>target</replaceable>] ... ]
> </literallayout>
> - Here is an example for two multiconfigs:
> + Here is an example for two extra multiconfigs:
> <filename>target1</filename> and
> <filename>target2</filename>:
> <literallayout class='monospaced'>
> - $ bitbake multiconfig:target1:<replaceable>target</replaceable>
> multiconfig:target2:<replaceable>target</replaceable>
> + $ bitbake mc::<replaceable>target</replaceable>
> mc:target1:<replaceable>target</replaceable>
> mc:target2:<replaceable>target</replaceable> </literallayout> </para>
> </section>
> @@ -837,13 +837,13 @@
> build, you must declare the dependencies in the
> recipe using the following statement form:
> <literallayout class='monospaced'>
> - <replaceable>task_or_package</replaceable>[mcdepends] =
> "multiconfig:<replaceable>from_multiconfig</replaceable>:<replaceable>to_multiconfig</replaceable>:<replaceable>recipe_name</replaceable>:<replaceable>task_on_which_to_depend</replaceable>"
> + <replaceable>task_or_package</replaceable>[mcdepends] =
> "mc:<replaceable>from_multiconfig</replaceable>:<replaceable>to_multiconfig</replaceable>:<replaceable>recipe_name</replaceable>:<replaceable>task_on_which_to_depend</replaceable>"
> </literallayout> To better show how to use this statement, consider an
> example with two multiconfigs:
> <filename>target1</filename> and <filename>target2</filename>:
> <literallayout class='monospaced'>
> - <replaceable>image_task</replaceable>[mcdepends] =
> "multiconfig:target1:target2:<replaceable>image2</replaceable>:<replaceable>rootfs_task</replaceable>"
> + <replaceable>image_task</replaceable>[mcdepends] =
> "mc:target1:target2:<replaceable>image2</replaceable>:<replaceable>rootfs_task</replaceable>"
> </literallayout> In this example, the
> <replaceable>from_multiconfig</replaceable> is
> "target1" and @@ -859,7 +859,7 @@
> Once you set up this dependency, you can build the
> "target1" multiconfig using a BitBake command as
> follows: <literallayout class='monospaced'>
> - $ bitbake multiconfig:target1:<replaceable>image1</replaceable>
> + $ bitbake mc:target1:<replaceable>image1</replaceable>
> </literallayout>
> This command executes all the tasks needed to
> create <replaceable>image1</replaceable> for the "target1"
> @@ -875,7 +875,7 @@
> Consider this change to the statement in the
> <replaceable>image1</replaceable> recipe:
> <literallayout class='monospaced'>
> - <replaceable>image_task</replaceable>[mcdepends] =
> "multiconfig:target1:target2:<replaceable>image2</replaceable>:<replaceable>image_task</replaceable>"
> + <replaceable>image_task</replaceable>[mcdepends] =
> "mc:target1:target2:<replaceable>image2</replaceable>:<replaceable>image_task</replaceable>"
> </literallayout> In this case, BitBake must create
> <replaceable>image2</replaceable> for the
> "target2" diff --git
> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml
> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml
> index 2490f6e..421364c 100644 ---
> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml
> +++
> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml @@
> -61,6 +61,78 @@ </para> </section>
> + <section id='modifying-existing-variables'>
> + <title>Modifying Existing Variables</title>
> +
> + <para>
> + Sometimes you need to modify existing variables.
> + Following are some cases where you might find you
> want to
> + modify an existing variable:
> + <itemizedlist>
> + <listitem><para>
> + Customize a recipe that uses the variable.
> + </para></listitem>
> + <listitem><para>
> + Change a variable's default value used in a
> + <filename>*.bbclass</filename> file.
> + </para></listitem>
> + <listitem><para>
> + Change the variable in a
> <filename>*.bbappend</filename>
> + file to override the variable in the
> original recipe.
> + </para></listitem>
> + <listitem><para>
> + Change the variable in a configuration file
> so that the
> + value overrides an existing configuration.
> + </para></listitem>
> + </itemizedlist>
> + </para>
> +
> + <para>
> + Changing a variable value can sometimes depend on
> how the
> + value was originally assigned and also on the desired
> + intent of the change.
> + In particular, when you append a value to a variable
> that
> + has a default value, the resulting value might not
> be what
> + you expect.
> + In this case, the value you provide might replace
> the value
> + rather than append to the default value.
> + </para>
> +
> + <para>
> + If after you have changed a variable's value and
> something
> + unexplained occurs, you can use BitBake to check the
> actual
> + value of the suspect variable.
> + You can make these checks for both configuration and
> recipe
> + level changes:
> + <itemizedlist>
> + <listitem><para>
> + For configuration changes, use the following:
> + <literallayout class='monospaced'>
> + $ bitbake -e
> + </literallayout>
> + This command displays variable values after
> the
> + configuration files (i.e.
> <filename>local.conf</filename>,
> + <filename>bblayers.conf</filename>,
> + <filename>bitbake.conf</filename> and so
> forth) have
> + been parsed.
> + <note>
> + Variables that are exported to the
> environment are
> + preceded by the string "export" in the
> command's
> + output.
> + </note>
> + </para></listitem>
> + <listitem><para>
> + For recipe changes, use the following:
> + <literallayout class='monospaced'>
> + $ bitbake <replaceable>recipe</replaceable> -e | grep VARIABLE="
> + </literallayout>
> + This command checks to see if the variable
> actually
> + makes it into a specific recipe.
> + </para></listitem>
> + </itemizedlist>
> + </para>
> + </section>
> +
> <section id='line-joining'>
> <title>Line Joining</title>
>
> @@ -297,9 +369,8 @@
>
> <para>
> These operators differ from the ":=", ".=", "=.",
> "+=", and "=+"
> - operators in that their effects are deferred
> - until after parsing completes rather than being
> immediately
> - applied.
> + operators in that their effects are applied at
> variable
> + expansion time rather than being immediately applied.
> Here are some examples:
> <literallayout class='monospaced'>
> B = "bval"
> @@ -348,18 +419,22 @@
> FOO = "123 456 789 123456 123 456 123 456"
> FOO_remove = "123"
> FOO_remove = "456"
> - FOO2 = "abc def ghi abcdef abc def abc def"
> - FOO2_remove = "abc def"
> + FOO2 = " abc def ghi abcdef abc def abc def def"
> + FOO2_remove = " \
> + def \
> + abc \
> + ghi \
> + "
> </literallayout>
> The variable <filename>FOO</filename> becomes
> - " 789 123456 "
> + " 789 123456 "
> and <filename>FOO2</filename> becomes
> - " ghi abcdef ".
> +
> " jkl abcdef ".
> </para>
> <para>
> Like "_append" and "_prepend", "_remove"
> - is deferred until after parsing completes.
> + is applied at variable expansion time.
> </para>
> </section>
>
> @@ -503,7 +578,7 @@
> </section>
>
> <section id='unsetting-variables'>
> - <title>Unseting variables</title>
> + <title>Unsetting variables</title>
>
> <para>
> It is possible to completely remove a variable or a
> variable flag @@ -595,7 +670,7 @@
>
> <para>
> BitBake uses
> - <link
> linkend='var-OVERRIDES'><filename>OVERRIDES</filename></link>
> + <link
> linkend='var-bb-OVERRIDES'><filename>OVERRIDES</filename></link> to
> control what variables are overridden after BitBake parses recipes
> and configuration files. This section describes how you can use
> @@ -705,7 +780,7 @@
>
> <para>Internally, this is implemented by
> prepending the task (e.g. "task-compile:") to the value of
> - <link
> linkend='var-OVERRIDES'><filename>OVERRIDES</filename></link>
> + <link
> linkend='var-bb-OVERRIDES'><filename>OVERRIDES</filename></link> for
> the local datastore of the <filename>do_compile</filename>
> task.</para>
> @@ -724,17 +799,15 @@
> <title>Key Expansion</title>
>
> <para>
> - Key expansion happens when the BitBake datastore is
> finalized
> - just before BitBake expands overrides.
> + Key expansion happens when the BitBake datastore is
> finalized. To better understand this, consider the following example:
> <literallayout class='monospaced'>
> A${B} = "X"
> B = "2"
> A2 = "Y"
> </literallayout>
> - In this case, after all the parsing is complete, and
> - before any overrides are handled, BitBake expands
> - <filename>${B}</filename> into "2".
> + In this case, after all the parsing is complete,
> + BitBake expands <filename>${B}</filename> into "2".
> This expansion causes <filename>A2</filename>, which
> was set to "Y" before the expansion, to become "X".
> </para>
> @@ -868,7 +941,7 @@
>
> <para>
> BitBake uses the
> - <link
> linkend='var-BBPATH'><filename>BBPATH</filename></link>
> + <link
> linkend='var-bb-BBPATH'><filename>BBPATH</filename></link> variable
> to locate needed include and class files. Additionally, BitBake
> searches the current directory for <filename>include</filename> and
> <filename>require</filename> @@ -1086,7 +1159,7 @@
> <para>
> When creating a configuration file
> (<filename>.conf</filename>), you can use the
> - <link
> linkend='var-INHERIT'><filename>INHERIT</filename></link>
> + <link
> linkend='var-bb-INHERIT'><filename>INHERIT</filename></link>
> configuration directive to inherit a class. BitBake only supports
> this directive when used within a configuration file.
> @@ -1370,7 +1443,7 @@
> </para></listitem>
> <listitem><para>
> BitBake-style Python functions generate a
> separate
> - <filename>${</filename><link
> linkend='var-T'><filename>T</filename></link><filename>}/run.</filename><replaceable>function-name</replaceable><filename>.</filename><replaceable>pid</replaceable>
> + <filename>${</filename><link
> linkend='var-bb-T'><filename>T</filename></link><filename>}/run.</filename><replaceable>function-name</replaceable><filename>.</filename><replaceable>pid</replaceable>
> script that is executed to run the function, and also generate a log
> file in
> <filename>${T}/log.</filename><replaceable>function-name</replaceable><filename>.</filename><replaceable>pid</replaceable>
> @@ -1773,7 +1846,7 @@ things exported or listed in its whitelist to
> ensure that the build environment is reproducible and consistent.
> You can prevent this "cleaning" by setting the
> - <link
> linkend='var-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link>
> + <link
> linkend='var-bb-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link>
> variable. </note>
> Consequently, if you do want something to get passed
> into the @@ -1783,9 +1856,9 @@
> Tell BitBake to load what you want from the
> environment into the datastore.
> You can do so through the
> - <link
> linkend='var-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>
> + <link
> linkend='var-bb-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>
> and
> - <link
> linkend='var-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link>
> + <link
> linkend='var-bb-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link>
> variables. For example, assume you want to prevent the build system
> from accessing your <filename>$HOME/.ccache</filename>
> @@ -1824,7 +1897,7 @@
> from the original execution environment.
> Bitbake saves a copy of the original environment into
> a special variable named
> - <link
> linkend='var-BB_ORIGENV'><filename>BB_ORIGENV</filename></link>.
> + <link
> linkend='var-bb-BB_ORIGENV'><filename>BB_ORIGENV</filename></link>.
> </para>
> <para>
> @@ -1883,7 +1956,7 @@
> <listitem><para><emphasis><filename>[depends]</filename>:</emphasis>
> Controls inter-task dependencies.
> See the
> - <link
> linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
> + <link
> linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link> variable
> and the "<link linkend='inter-task-dependencies'>Inter-Task
> Dependencies</link>" section for more information.
> @@ -1891,7 +1964,7 @@
> <listitem><para><emphasis><filename>[deptask]</filename>:</emphasis>
> Controls task build-time dependencies.
> See the
> - <link
> linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
> + <link
> linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link> variable
> and the "<link linkend='build-dependencies'>Build Dependencies</link>"
> section for more information.
> @@ -1937,7 +2010,7 @@
> of cores but certain tasks need to be
> rate-limited due to various kinds of resource constraints (e.g. to
> avoid network throttling). <filename>number_threads</filename> works
> similarly to the
> - <link
> linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
> + <link
> linkend='var-bb-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
> variable but is task-specific.</para>
> <para>Set the value globally.
> @@ -1971,9 +2044,9 @@
> <listitem><para><emphasis><filename>[rdepends]</filename>:</emphasis>
> Controls inter-task runtime dependencies.
> See the
> - <link
> linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link>
> + <link
> linkend='var-bb-RDEPENDS'><filename>RDEPENDS</filename></link>
> variable, the
> - <link
> linkend='var-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
> + <link
> linkend='var-bb-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
> variable, and the "<link linkend='inter-task-dependencies'>Inter-Task
> Dependencies</link>" section for more information.
> @@ -1981,9 +2054,9 @@
> <listitem><para><emphasis><filename>[rdeptask]</filename>:</emphasis>
> Controls task runtime dependencies.
> See the
> - <link
> linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link>
> + <link
> linkend='var-bb-RDEPENDS'><filename>RDEPENDS</filename></link>
> variable, the
> - <link
> linkend='var-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
> + <link
> linkend='var-bb-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
> variable, and the "<link linkend='runtime-dependencies'>Runtime
> Dependencies</link>" section for more information.
> @@ -1996,9 +2069,9 @@
> <listitem><para><emphasis><filename>[recrdeptask]</filename>:</emphasis>
> Controls task recursive runtime dependencies.
> See the
> - <link
> linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link>
> + <link
> linkend='var-bb-RDEPENDS'><filename>RDEPENDS</filename></link>
> variable, the
> - <link
> linkend='var-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
> + <link
> linkend='var-bb-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
> variable, and the "<link linkend='recursive-dependencies'>Recursive
> Dependencies</link>" section for more information.
> @@ -2127,7 +2200,7 @@
> Any given datastore only has one such event
> executed against it, however.
> If
> - <link
> linkende='var-BB_INVALIDCONF'><filename>BB_INVALIDCONF</filename></link>
> + <link
> linkende='var-bb-BB_INVALIDCONF'><filename>BB_INVALIDCONF</filename></link>
> is set in the datastore by the event handler, the configuration is
> reparsed and a new event triggered, allowing the metadata to update
> configuration. @@ -2256,17 +2329,17 @@
> from a single recipe file multiple incarnations of that
> recipe file where all incarnations are buildable.
> These features are enabled through the
> - <link
> linkend='var-BBCLASSEXTEND'><filename>BBCLASSEXTEND</filename></link>
> + <link
> linkend='var-bb-BBCLASSEXTEND'><filename>BBCLASSEXTEND</filename></link>
> and
> - <link
> linkend='var-BBVERSIONS'><filename>BBVERSIONS</filename></link>
> + <link
> linkend='var-bb-BBVERSIONS'><filename>BBVERSIONS</filename></link>
> variables. <note>
> The mechanism for this class extension is extremely
> specific to the implementation.
> Usually, the recipe's
> - <link
> linkend='var-PROVIDES'><filename>PROVIDES</filename></link>,
> - <link
> linkend='var-PN'><filename>PN</filename></link>, and
> - <link
> linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
> + <link
> linkend='var-bb-PROVIDES'><filename>PROVIDES</filename></link>,
> + <link
> linkend='var-bb-PN'><filename>PN</filename></link>, and
> + <link
> linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link>
> variables would need to be modified by the extension class. For
> specific examples, see the OE-Core <filename>native</filename>,
> <filename>nativesdk</filename>, @@ -2287,7 +2360,7 @@
> project from a single recipe file.
> You can also specify conditional metadata
> (using the
> - <link
> linkend='var-OVERRIDES'><filename>OVERRIDES</filename></link>
> + <link
> linkend='var-bb-OVERRIDES'><filename>OVERRIDES</filename></link>
> mechanism) for a single version, or an optionally named range of
> versions. Here is an example: <literallayout class='monospaced'>
> @@ -2306,7 +2379,7 @@
> into overrides, but it is also made available
> for the metadata to use in the variable that defines the base recipe
> versions for use in <filename>file://</filename> search paths
> - (<link
> linkend='var-FILESPATH'><filename>FILESPATH</filename></link>).
> + (<link
> linkend='var-bb-FILESPATH'><filename>FILESPATH</filename></link>).
> </para></listitem> </itemizedlist>
> </para>
> @@ -2408,7 +2481,7 @@
>
> <para>
> BitBake uses the
> - <link
> linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
> + <link
> linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link> variable
> to manage build time dependencies. The <filename>[deptask]</filename>
> varflag for tasks signifies the task of each
> @@ -2429,9 +2502,9 @@
>
> <para>
> BitBake uses the
> - <link
> linkend='var-PACKAGES'><filename>PACKAGES</filename></link>,
> - <link
> linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link>, and
> - <link
> linkend='var-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
> + <link
> linkend='var-bb-PACKAGES'><filename>PACKAGES</filename></link>,
> + <link
> linkend='var-bb-RDEPENDS'><filename>RDEPENDS</filename></link>, and
> + <link
> linkend='var-bb-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
> variables to manage runtime dependencies. </para>
>
> @@ -2686,7 +2759,7 @@
>
> <para>
> These checksums are stored in
> - <link
> linkend='var-STAMP'><filename>STAMP</filename></link>.
> + <link
> linkend='var-bb-STAMP'><filename>STAMP</filename></link>. You can
> examine the checksums using the following BitBake command:
> <literallayout class='monospaced'> $ bitbake-dumpsigs
> @@ -2708,44 +2781,44 @@
> The following list describes related variables:
> <itemizedlist>
> <listitem><para>
> - <link
> linkend='var-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></link>:
> + <link
> linkend='var-bb-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></link>:
> Specifies the name of the function to call during the "setscene" part
> of the task's execution in order to validate the list of task hashes.
> </para></listitem>
> <listitem><para>
> - <link
> linkend='var-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></link>:
> + <link
> linkend='var-bb-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></link>:
> Specifies a function BitBake calls that determines whether BitBake
> requires a setscene dependency to be met.
> </para></listitem>
> <listitem><para>
> - <link
> linkend='var-BB_SETSCENE_VERIFY_FUNCTION2'><filename>BB_SETSCENE_VERIFY_FUNCTION2</filename></link>:
> + <link
> linkend='var-bb-BB_SETSCENE_VERIFY_FUNCTION2'><filename>BB_SETSCENE_VERIFY_FUNCTION2</filename></link>:
> Specifies a function to call that verifies the list of planned task
> execution before the main task execution happens.
> </para></listitem>
> <listitem><para>
> - <link
> linkend='var-BB_STAMP_POLICY'><filename>BB_STAMP_POLICY</filename></link>:
> + <link
> linkend='var-bb-BB_STAMP_POLICY'><filename>BB_STAMP_POLICY</filename></link>:
> Defines the mode for comparing timestamps of stamp files.
> </para></listitem> <listitem><para>
> - <link
> linkend='var-BB_STAMP_WHITELIST'><filename>BB_STAMP_WHITELIST</filename></link>:
> + <link
> linkend='var-bb-BB_STAMP_WHITELIST'><filename>BB_STAMP_WHITELIST</filename></link>:
> Lists stamp files that are looked at when the stamp policy is
> "whitelist". </para></listitem>
> <listitem><para>
> - <link
> linkend='var-BB_TASKHASH'><filename>BB_TASKHASH</filename></link>:
> + <link
> linkend='var-bb-BB_TASKHASH'><filename>BB_TASKHASH</filename></link>:
> Within an executing task, this variable holds the hash of the task as
> returned by the currently enabled signature generator.
> </para></listitem>
> <listitem><para>
> - <link
> linkend='var-STAMP'><filename>STAMP</filename></link>:
> + <link
> linkend='var-bb-STAMP'><filename>STAMP</filename></link>: The base
> path to create stamp files. </para></listitem>
> <listitem><para>
> - <link
> linkend='var-STAMPCLEAN'><filename>STAMPCLEAN</filename></link>:
> + <link
> linkend='var-bb-STAMPCLEAN'><filename>STAMPCLEAN</filename></link>:
> Again, the base path to create stamp files but can use wildcards for
> matching a range of files for clean operations. </para></listitem>
> diff --git
> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml
> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml
> index a84b2bc..aca6741 100644 ---
> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml
> +++
> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml
> @@ -3,7 +3,7 @@ [<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
> <!-- Dummy chapter --> -<chapter id='ref-variables-glos'>
> +<chapter id='ref-bb-variables-glos'>
>
> <title>Variables Glossary</title>
>
> @@ -34,29 +34,29 @@
> </itemizedlist>
> </note>
>
> -<glossary id='ref-variables-glossary'>
> +<glossary id='ref-bb-variables-glossary'>
>
> <para>
> - <link linkend='var-ASSUME_PROVIDED'>A</link>
> - <link linkend='var-B'>B</link>
> - <link linkend='var-CACHE'>C</link>
> - <link linkend='var-DEFAULT_PREFERENCE'>D</link>
> - <link linkend='var-EXCLUDE_FROM_WORLD'>E</link>
> - <link linkend='var-FAKEROOT'>F</link>
> - <link linkend='var-GITDIR'>G</link>
> - <link linkend='var-HGDIR'>H</link>
> -<!-- <link linkend='var-ICECC_DISABLED'>I</link> -->
> + <link linkend='var-bb-ASSUME_PROVIDED'>A</link>
> + <link linkend='var-bb-B'>B</link>
> + <link linkend='var-bb-CACHE'>C</link>
> + <link linkend='var-bb-DEFAULT_PREFERENCE'>D</link>
> + <link linkend='var-bb-EXCLUDE_FROM_WORLD'>E</link>
> + <link linkend='var-bb-FAKEROOT'>F</link>
> + <link linkend='var-bb-GITDIR'>G</link>
> + <link linkend='var-bb-HGDIR'>H</link>
> + <link linkend='var-bb-INHERIT'>I</link>
> <!-- <link linkend='var-glossary-j'>J</link> -->
> <!-- <link linkend='var-KARCH'>K</link> -->
> - <link linkend='var-LAYERDEPENDS'>L</link>
> - <link linkend='var-MIRRORS'>M</link>
> + <link linkend='var-bb-LAYERDEPENDS'>L</link>
> + <link linkend='var-bb-MIRRORS'>M</link>
> <!-- <link linkend='var-glossary-n'>N</link> -->
> - <link linkend='var-OVERRIDES'>O</link>
> - <link linkend='var-P4DIR'>P</link>
> + <link linkend='var-bb-OVERRIDES'>O</link>
> + <link linkend='var-bb-P4DIR'>P</link>
> <!-- <link linkend='var-QMAKE_PROFILES'>Q</link> -->
> - <link linkend='var-RDEPENDS'>R</link>
> - <link linkend='var-SECTION'>S</link>
> - <link linkend='var-T'>T</link>
> + <link linkend='var-bb-RDEPENDS'>R</link>
> + <link linkend='var-bb-SECTION'>S</link>
> + <link linkend='var-bb-T'>T</link>
> <!-- <link linkend='var-UBOOT_CONFIG'>U</link> -->
> <!-- <link linkend='var-glossary-v'>V</link> -->
> <!-- <link linkend='var-WARN_QA'>W</link> -->
> @@ -65,13 +65,13 @@
> <!-- <link linkend='var-glossary-z'>Z</link>-->
> </para>
>
> - <glossdiv id='var-glossary-a'><title>A</title>
> + <glossdiv id='var-bb-glossary-a'><title>A</title>
>
> - <glossentry
> id='var-ASSUME_PROVIDED'><glossterm>ASSUME_PROVIDED</glossterm>
> + <glossentry
> id='var-bb-ASSUME_PROVIDED'><glossterm>ASSUME_PROVIDED</glossterm>
> <glossdef> <para>
> Lists recipe names
> - (<link
> linkend='var-PN'><filename>PN</filename></link>
> + (<link
> linkend='var-bb-PN'><filename>PN</filename></link> values) BitBake
> does not attempt to build. Instead, BitBake assumes these recipes
> have already been built.
> @@ -91,9 +91,9 @@
> </glossdiv>
>
>
> - <glossdiv id='var-glossary-b'><title>B</title>
> + <glossdiv id='var-bb-glossary-b'><title>B</title>
>
> - <glossentry id='var-B'><glossterm>B</glossterm>
> + <glossentry id='var-bb-B'><glossterm>B</glossterm>
> <glossdef>
> <para>
> The directory in which BitBake executes functions
> @@ -102,7 +102,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_ALLOWED_NETWORKS'><glossterm>BB_ALLOWED_NETWORKS</glossterm>
> + <glossentry
> id='var-bb-BB_ALLOWED_NETWORKS'><glossterm>BB_ALLOWED_NETWORKS</glossterm>
> <glossdef> <para>
> Specifies a space-delimited list of hosts that
> the fetcher @@ -111,7 +111,7 @@
> <itemizedlist>
> <listitem><para>
> This host list is only used if
> - <link
> linkend='var-BB_NO_NETWORK'><filename>BB_NO_NETWORK</filename></link>
> + <link
> linkend='var-bb-BB_NO_NETWORK'><filename>BB_NO_NETWORK</filename></link>
> is either not set or set to "0". </para></listitem>
> <listitem><para>
> @@ -151,13 +151,13 @@
> </itemizedlist>
> Using <filename>BB_ALLOWED_NETWORKS</filename> in
> conjunction with
> - <link
> linkend='var-PREMIRRORS'><filename>PREMIRRORS</filename></link>
> + <link
> linkend='var-bb-PREMIRRORS'><filename>PREMIRRORS</filename></link> is
> very useful. Adding the host you want to use to
> <filename>PREMIRRORS</filename> results in the
> source code being fetched from an allowed location and avoids raising
> an error when a host that is not allowed is in a
> - <link
> linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
> + <link
> linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link>
> statement. This is because the fetcher does not attempt to use the
> host listed in <filename>SRC_URI</filename>
> after a @@ -167,7 +167,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_CONSOLELOG'><glossterm>BB_CONSOLELOG</glossterm>
> + <glossentry
> id='var-bb-BB_CONSOLELOG'><glossterm>BB_CONSOLELOG</glossterm>
> <glossdef> <para>
> Specifies the path to a log file into which
> BitBake's user @@ -176,7 +176,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_CURRENTTASK'><glossterm>BB_CURRENTTASK</glossterm>
> + <glossentry
> id='var-bb-BB_CURRENTTASK'><glossterm>BB_CURRENTTASK</glossterm>
> <glossdef> <para>
> Contains the name of the currently running task.
> @@ -186,7 +186,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_DANGLINGAPPENDS_WARNONLY'><glossterm>BB_DANGLINGAPPENDS_WARNONLY</glossterm>
> + <glossentry
> id='var-bb-BB_DANGLINGAPPENDS_WARNONLY'><glossterm>BB_DANGLINGAPPENDS_WARNONLY</glossterm>
> <glossdef> <para>
> Defines how BitBake handles situations where an
> append @@ -208,7 +208,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_DEFAULT_TASK'><glossterm>BB_DEFAULT_TASK</glossterm>
> + <glossentry
> id='var-bb-BB_DEFAULT_TASK'><glossterm>BB_DEFAULT_TASK</glossterm>
> <glossdef> <para>
> The default task to use when none is specified
> (e.g. @@ -219,7 +219,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_DISKMON_DIRS'><glossterm>BB_DISKMON_DIRS</glossterm>
> + <glossentry
> id='var-bb-BB_DISKMON_DIRS'><glossterm>BB_DISKMON_DIRS</glossterm>
> <glossdef> <para>
> Monitors disk space and available inodes during
> the build @@ -245,7 +245,7 @@
> build when a threshold is broken.
> Subsequent warnings are issued as
> defined by the
> - <link
> linkend='var-BB_DISKMON_WARNINTERVAL'>BB_DISKMON_WARNINTERVAL</link>
> variable,
> + <link
> linkend='var-bb-BB_DISKMON_WARNINTERVAL'>BB_DISKMON_WARNINTERVAL</link>
> variable, which must be defined.
> <dir> is:
> @@ -275,7 +275,7 @@
> BB_DISKMON_DIRS = "ABORT,${TMPDIR},,100K"
> </literallayout>
> The first example works only if you also set
> - the <link
> linkend='var-BB_DISKMON_WARNINTERVAL'><filename>BB_DISKMON_WARNINTERVAL</filename></link>
> variable.
> + the <link
> linkend='var-bb-BB_DISKMON_WARNINTERVAL'><filename>BB_DISKMON_WARNINTERVAL</filename></link>
> variable. This example causes the build system to immediately abort
> when either the disk space in <filename>${TMPDIR}</filename> drops
> below 1 Gbyte or the available free inodes drops below @@ -309,7
> +309,7 @@ </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_DISKMON_WARNINTERVAL'><glossterm>BB_DISKMON_WARNINTERVAL</glossterm>
> + <glossentry
> id='var-bb-BB_DISKMON_WARNINTERVAL'><glossterm>BB_DISKMON_WARNINTERVAL</glossterm>
> <glossdef> <para>
> Defines the disk space and free inode warning
> intervals. @@ -319,7 +319,7 @@
> If you are going to use the
> <filename>BB_DISKMON_WARNINTERVAL</filename>
> variable, you must also use the
> - <link
> linkend='var-BB_DISKMON_DIRS'><filename>BB_DISKMON_DIRS</filename></link>
> variable
> + <link
> linkend='var-bb-BB_DISKMON_DIRS'><filename>BB_DISKMON_DIRS</filename></link>
> variable and define its action as "WARN". During the build,
> subsequent warnings are issued each time disk space or number of free
> inodes further reduces by @@ -374,7 +374,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_ENV_WHITELIST'><glossterm>BB_ENV_WHITELIST</glossterm>
> + <glossentry
> id='var-bb-BB_ENV_WHITELIST'><glossterm>BB_ENV_WHITELIST</glossterm>
> <glossdef> <para>
> Specifies the internal whitelist of variables to
> allow @@ -382,11 +382,11 @@
> datastore.
> If the value of this variable is not specified
> (which is the default), the following list is
> used:
> - <link
> linkend='var-BBPATH'><filename>BBPATH</filename></link>,
> - <link
> linkend='var-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link>,
> - <link
> linkend='var-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>,
> + <link
> linkend='var-bb-BBPATH'><filename>BBPATH</filename></link>,
> + <link
> linkend='var-bb-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link>,
> + <link
> linkend='var-bb-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>,
> and
> - <link
> linkend='var-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link>.
> + <link
> linkend='var-bb-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link>.
> <note> You must set this variable in the external environment
> in order for it to work.
> @@ -395,7 +395,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_ENV_EXTRAWHITE'><glossterm>BB_ENV_EXTRAWHITE</glossterm>
> + <glossentry
> id='var-bb-BB_ENV_EXTRAWHITE'><glossterm>BB_ENV_EXTRAWHITE</glossterm>
> <glossdef> <para>
> Specifies an additional set of variables to
> allow through @@ -403,7 +403,7 @@
> datastore.
> This list of variables are on top of the
> internal list set in
> - <link
> linkend='var-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>.
> + <link
> linkend='var-bb-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>.
> <note> You must set this variable in the external
> environment in order for it to work.
> @@ -412,22 +412,22 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_FETCH_PREMIRRORONLY'><glossterm>BB_FETCH_PREMIRRORONLY</glossterm>
> + <glossentry
> id='var-bb-BB_FETCH_PREMIRRORONLY'><glossterm>BB_FETCH_PREMIRRORONLY</glossterm>
> <glossdef> <para>
> When set to "1", causes BitBake's fetcher module
> to only search
> - <link
> linkend='var-PREMIRRORS'><filename>PREMIRRORS</filename></link>
> + <link
> linkend='var-bb-PREMIRRORS'><filename>PREMIRRORS</filename></link>
> for files. BitBake will not search the main
> - <link
> linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
> + <link
> linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link> or
> - <link
> linkend='var-MIRRORS'><filename>MIRRORS</filename></link>.
> + <link
> linkend='var-bb-MIRRORS'><filename>MIRRORS</filename></link>. </para>
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_FILENAME'><glossterm>BB_FILENAME</glossterm>
> + <glossentry
> id='var-bb-BB_FILENAME'><glossterm>BB_FILENAME</glossterm> <glossdef>
> <para>
> Contains the filename of the recipe that owns
> the currently @@ -440,12 +440,12 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_GENERATE_MIRROR_TARBALLS'><glossterm>BB_GENERATE_MIRROR_TARBALLS</glossterm>
> + <glossentry
> id='var-bb-BB_GENERATE_MIRROR_TARBALLS'><glossterm>BB_GENERATE_MIRROR_TARBALLS</glossterm>
> <glossdef> <para>
> Causes tarballs of the Git repositories,
> including the Git metadata, to be placed in the
> - <link
> linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
> + <link
> linkend='var-bb-DL_DIR'><filename>DL_DIR</filename></link> directory.
> Anyone wishing to create a source mirror would
> want to enable this variable.
> @@ -461,7 +461,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_HASHCONFIG_WHITELIST'><glossterm>BB_HASHCONFIG_WHITELIST</glossterm>
> + <glossentry
> id='var-bb-BB_HASHCONFIG_WHITELIST'><glossterm>BB_HASHCONFIG_WHITELIST</glossterm>
> <glossdef> <para>
> Lists variables that are excluded from base
> configuration @@ -485,7 +485,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_HASHBASE_WHITELIST'><glossterm>BB_HASHBASE_WHITELIST</glossterm>
> + <glossentry
> id='var-bb-BB_HASHBASE_WHITELIST'><glossterm>BB_HASHBASE_WHITELIST</glossterm>
> <glossdef> <para>
> Lists variables that are excluded from checksum
> and @@ -500,7 +500,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_HASHCHECK_FUNCTION'><glossterm>BB_HASHCHECK_FUNCTION</glossterm>
> + <glossentry
> id='var-bb-BB_HASHCHECK_FUNCTION'><glossterm>BB_HASHCHECK_FUNCTION</glossterm>
> <glossdef> <para>
> Specifies the name of the function to call
> during the @@ -524,7 +524,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_INVALIDCONF'><glossterm>BB_INVALIDCONF</glossterm>
> + <glossentry
> id='var-bb-BB_INVALIDCONF'><glossterm>BB_INVALIDCONF</glossterm>
> <glossdef> <para>
> Used in combination with the
> @@ -539,11 +539,11 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_LOGFMT'><glossterm>BB_LOGFMT</glossterm>
> + <glossentry
> id='var-bb-BB_LOGFMT'><glossterm>BB_LOGFMT</glossterm> <glossdef>
> <para>
> Specifies the name of the log files saved into
> - <filename>${</filename><link
> linkend='var-T'><filename>T</filename></link><filename>}</filename>.
> + <filename>${</filename><link
> linkend='var-bb-T'><filename>T</filename></link><filename>}</filename>.
> By default, the <filename>BB_LOGFMT</filename> variable is undefined
> and the log file names get created using the following form:
> @@ -556,7 +556,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_NICE_LEVEL'><glossterm>BB_NICE_LEVEL</glossterm>
> + <glossentry
> id='var-bb-BB_NICE_LEVEL'><glossterm>BB_NICE_LEVEL</glossterm>
> <glossdef> <para>
> Allows BitBake to run at a specific priority
> @@ -564,13 +564,13 @@
> System permissions usually mean that BitBake can
> reduce its priority but not raise it again.
> See
> - <link
> linkend='var-BB_TASK_NICE_LEVEL'><filename>BB_TASK_NICE_LEVEL</filename></link>
> + <link
> linkend='var-bb-BB_TASK_NICE_LEVEL'><filename>BB_TASK_NICE_LEVEL</filename></link>
> for additional information. </para>
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_NO_NETWORK'><glossterm>BB_NO_NETWORK</glossterm>
> + <glossentry
> id='var-bb-BB_NO_NETWORK'><glossterm>BB_NO_NETWORK</glossterm>
> <glossdef> <para>
> Disables network access in the BitBake fetcher
> modules. @@ -587,7 +587,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_NUMBER_THREADS'><glossterm>BB_NUMBER_THREADS</glossterm>
> + <glossentry
> id='var-bb-BB_NUMBER_THREADS'><glossterm>BB_NUMBER_THREADS</glossterm>
> <glossdef> <para>
> The maximum number of tasks BitBake should run
> in parallel @@ -599,7 +599,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_NUMBER_PARSE_THREADS'><glossterm>BB_NUMBER_PARSE_THREADS</glossterm>
> + <glossentry
> id='var-bb-BB_NUMBER_PARSE_THREADS'><glossterm>BB_NUMBER_PARSE_THREADS</glossterm>
> <glossdef> <para>
> Sets the number of threads BitBake uses when
> parsing. @@ -609,7 +609,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_ORIGENV'><glossterm>BB_ORIGENV</glossterm>
> + <glossentry
> id='var-bb-BB_ORIGENV'><glossterm>BB_ORIGENV</glossterm> <glossdef>
> <para>
> Contains a copy of the original external
> environment in @@ -625,7 +625,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_PRESERVE_ENV'><glossterm>BB_PRESERVE_ENV</glossterm>
> + <glossentry
> id='var-bb-BB_PRESERVE_ENV'><glossterm>BB_PRESERVE_ENV</glossterm>
> <glossdef> <para>
> Disables whitelisting and instead allows all
> variables @@ -639,12 +639,12 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_RUNFMT'><glossterm>BB_RUNFMT</glossterm>
> + <glossentry
> id='var-bb-BB_RUNFMT'><glossterm>BB_RUNFMT</glossterm> <glossdef>
> <para>
> Specifies the name of the executable script files
> (i.e. run files) saved into
> - <filename>${</filename><link
> linkend='var-T'><filename>T</filename></link><filename>}</filename>.
> + <filename>${</filename><link
> linkend='var-bb-T'><filename>T</filename></link><filename>}</filename>.
> By default, the <filename>BB_RUNFMT</filename> variable is undefined
> and the run file names get created using the following form:
> @@ -657,7 +657,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_RUNTASK'><glossterm>BB_RUNTASK</glossterm>
> + <glossentry
> id='var-bb-BB_RUNTASK'><glossterm>BB_RUNTASK</glossterm> <glossdef>
> <para>
> Contains the name of the currently executing
> task. @@ -669,7 +669,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_SCHEDULER'><glossterm>BB_SCHEDULER</glossterm>
> + <glossentry
> id='var-bb-BB_SCHEDULER'><glossterm>BB_SCHEDULER</glossterm>
> <glossdef> <para>
> Selects the name of the scheduler to use for the
> @@ -695,7 +695,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_SCHEDULERS'><glossterm>BB_SCHEDULERS</glossterm>
> + <glossentry
> id='var-bb-BB_SCHEDULERS'><glossterm>BB_SCHEDULERS</glossterm>
> <glossdef> <para>
> Defines custom schedulers to import.
> @@ -705,13 +705,13 @@
>
> <para>
> For information how to select a scheduler, see
> the
> - <link
> linkend='var-BB_SCHEDULER'><filename>BB_SCHEDULER</filename></link>
> + <link
> linkend='var-bb-BB_SCHEDULER'><filename>BB_SCHEDULER</filename></link>
> variable. </para>
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_SETSCENE_DEPVALID'><glossterm>BB_SETSCENE_DEPVALID</glossterm>
> + <glossentry
> id='var-bb-BB_SETSCENE_DEPVALID'><glossterm>BB_SETSCENE_DEPVALID</glossterm>
> <glossdef> <para>
> Specifies a function BitBake calls that
> determines @@ -731,7 +731,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_SETSCENE_VERIFY_FUNCTION2'><glossterm>BB_SETSCENE_VERIFY_FUNCTION2</glossterm>
> + <glossentry
> id='var-bb-BB_SETSCENE_VERIFY_FUNCTION2'><glossterm>BB_SETSCENE_VERIFY_FUNCTION2</glossterm>
> <glossdef> <para>
> Specifies a function to call that verifies the
> list of @@ -752,7 +752,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_SIGNATURE_EXCLUDE_FLAGS'><glossterm>BB_SIGNATURE_EXCLUDE_FLAGS</glossterm>
> + <glossentry
> id='var-bb-BB_SIGNATURE_EXCLUDE_FLAGS'><glossterm>BB_SIGNATURE_EXCLUDE_FLAGS</glossterm>
> <glossdef> <para>
> Lists variable flags (varflags)
> @@ -771,7 +771,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_SIGNATURE_HANDLER'><glossterm>BB_SIGNATURE_HANDLER</glossterm>
> + <glossentry
> id='var-bb-BB_SIGNATURE_HANDLER'><glossterm>BB_SIGNATURE_HANDLER</glossterm>
> <glossdef> <para>
> Defines the name of the signature handler
> BitBake uses. @@ -790,7 +790,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_SRCREV_POLICY'><glossterm>BB_SRCREV_POLICY</glossterm>
> + <glossentry
> id='var-bb-BB_SRCREV_POLICY'><glossterm>BB_SRCREV_POLICY</glossterm>
> <glossdef> <para>
> Defines the behavior of the fetcher when it
> interacts with @@ -817,7 +817,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_STAMP_POLICY'><glossterm>BB_STAMP_POLICY</glossterm>
> + <glossentry
> id='var-bb-BB_STAMP_POLICY'><glossterm>BB_STAMP_POLICY</glossterm>
> <glossdef> <para>
> Defines the mode used for how timestamps of
> stamp files @@ -836,7 +836,7 @@
> <listitem><para><emphasis>whitelist</emphasis>
> - Identical to "full" mode except timestamp
> comparisons are made for recipes listed
> in the
> - <link
> linkend='var-BB_STAMP_WHITELIST'><filename>BB_STAMP_WHITELIST</filename></link>
> + <link
> linkend='var-bb-BB_STAMP_WHITELIST'><filename>BB_STAMP_WHITELIST</filename></link>
> variable. </para></listitem>
> </itemizedlist>
> @@ -848,19 +848,19 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_STAMP_WHITELIST'><glossterm>BB_STAMP_WHITELIST</glossterm>
> + <glossentry
> id='var-bb-BB_STAMP_WHITELIST'><glossterm>BB_STAMP_WHITELIST</glossterm>
> <glossdef> <para>
> Lists files whose stamp file timestamps are
> compared when the stamp policy mode is set to "whitelist".
> For information on stamp policies, see the
> - <link
> linkend='var-BB_STAMP_POLICY'><filename>BB_STAMP_POLICY</filename></link>
> + <link
> linkend='var-bb-BB_STAMP_POLICY'><filename>BB_STAMP_POLICY</filename></link>
> variable. </para>
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_STRICT_CHECKSUM'><glossterm>BB_STRICT_CHECKSUM</glossterm>
> + <glossentry
> id='var-bb-BB_STRICT_CHECKSUM'><glossterm>BB_STRICT_CHECKSUM</glossterm>
> <glossdef> <para>
> Sets a more strict checksum mechanism for
> non-local URLs. @@ -871,7 +871,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_TASK_IONICE_LEVEL'><glossterm>BB_TASK_IONICE_LEVEL</glossterm>
> + <glossentry
> id='var-bb-BB_TASK_IONICE_LEVEL'><glossterm>BB_TASK_IONICE_LEVEL</glossterm>
> <glossdef> <para>
> Allows adjustment of a task's Input/Output
> priority. @@ -882,7 +882,7 @@
> variable to adjust the I/O priority of these
> tasks. <note>
> This variable works similarly to the
> - <link
> linkend='var-BB_TASK_NICE_LEVEL'><filename>BB_TASK_NICE_LEVEL</filename></link>
> + <link
> linkend='var-bb-BB_TASK_NICE_LEVEL'><filename>BB_TASK_NICE_LEVEL</filename></link>
> variable except with a task's I/O priorities. </note>
> </para>
> @@ -921,7 +921,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_TASK_NICE_LEVEL'><glossterm>BB_TASK_NICE_LEVEL</glossterm>
> + <glossentry
> id='var-bb-BB_TASK_NICE_LEVEL'><glossterm>BB_TASK_NICE_LEVEL</glossterm>
> <glossdef> <para>
> Allows specific tasks to change their priority
> @@ -940,7 +940,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_TASKHASH'><glossterm>BB_TASKHASH</glossterm>
> + <glossentry
> id='var-bb-BB_TASKHASH'><glossterm>BB_TASKHASH</glossterm> <glossdef>
> <para>
> Within an executing task, this variable holds
> the hash @@ -950,7 +950,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_VERBOSE_LOGS'><glossterm>BB_VERBOSE_LOGS</glossterm>
> + <glossentry
> id='var-bb-BB_VERBOSE_LOGS'><glossterm>BB_VERBOSE_LOGS</glossterm>
> <glossdef> <para>
> Controls how verbose BitBake is during builds.
> @@ -960,7 +960,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_WORKERCONTEXT'><glossterm>BB_WORKERCONTEXT</glossterm>
> + <glossentry
> id='var-bb-BB_WORKERCONTEXT'><glossterm>BB_WORKERCONTEXT</glossterm>
> <glossdef> <para>
> Specifies if the current context is executing a
> task. @@ -973,7 +973,7 @@
> </glossentry>
>
>
> - <glossentry
> id='var-BBCLASSEXTEND'><glossterm>BBCLASSEXTEND</glossterm>
> + <glossentry
> id='var-bb-BBCLASSEXTEND'><glossterm>BBCLASSEXTEND</glossterm>
> <glossdef> <para>
> Allows you to extend a recipe so that it builds
> variants @@ -1009,7 +1009,7 @@
> <filename>_class-native</filename>.
> For example, to generate a native version of
> a recipe, a
> - <link
> linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
> + <link
> linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link> on "foo"
> is rewritten to a <filename>DEPENDS</filename> on "foo-native".
> </para>
> @@ -1028,7 +1028,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-BBDEBUG'><glossterm>BBDEBUG</glossterm>
> + <glossentry
> id='var-bb-BBDEBUG'><glossterm>BBDEBUG</glossterm> <glossdef>
> <para>
> Sets the BitBake debug output level to a
> specific value @@ -1042,7 +1042,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BBFILE_COLLECTIONS'><glossterm>BBFILE_COLLECTIONS</glossterm>
> + <glossentry
> id='var-bb-BBFILE_COLLECTIONS'><glossterm>BBFILE_COLLECTIONS</glossterm>
> <glossdef> <para>Lists the names of configured layers.
> These names are used to find the other
> <filename>BBFILE_*</filename> @@ -1053,10 +1053,10 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BBFILE_PATTERN'><glossterm>BBFILE_PATTERN</glossterm>
> + <glossentry
> id='var-bb-BBFILE_PATTERN'><glossterm>BBFILE_PATTERN</glossterm>
> <glossdef> <para>Variable that expands to match files from
> - <link
> linkend='var-BBFILES'><filename>BBFILES</filename></link>
> + <link
> linkend='var-bb-BBFILES'><filename>BBFILES</filename></link> in a
> particular layer. This variable is used in the
> <filename>conf/layer.conf</filename> file and must be suffixed with
> the name of the specific layer (e.g. @@ -1064,7 +1064,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BBFILE_PRIORITY'><glossterm>BBFILE_PRIORITY</glossterm>
> + <glossentry
> id='var-bb-BBFILE_PRIORITY'><glossterm>BBFILE_PRIORITY</glossterm>
> <glossdef> <para>Assigns the priority for recipe files in each
> layer.</para> <para>This variable is useful in situations where the
> same recipe appears in @@ -1074,7 +1074,7 @@
> letting you control the precedence for the
> multiple layers. The precedence established through this variable
> stands regardless of a recipe's version
> - (<link
> linkend='var-PV'><filename>PV</filename></link> variable).
> + (<link
> linkend='var-bb-PV'><filename>PV</filename></link> variable). For
> example, a layer that has a recipe with a higher
> <filename>PV</filename> value but for which the
> <filename>BBFILE_PRIORITY</filename> is set to have a lower
> precedence still has a lower precedence.</para> @@ -1083,7 +1083,7 @@
> For example, the value 6 has a higher precedence than the value 5. If
> not specified, the <filename>BBFILE_PRIORITY</filename> variable is
> set based on layer dependencies (see the
> - <filename><link
> linkend='var-LAYERDEPENDS'>LAYERDEPENDS</link></filename> variable for
> + <filename><link
> linkend='var-bb-LAYERDEPENDS'>LAYERDEPENDS</link></filename> variable
> for more information. The default priority, if unspecified
> for a layer with no dependencies, is the lowest
> defined priority + 1 @@ -1095,7 +1095,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-BBFILES'><glossterm>BBFILES</glossterm>
> + <glossentry
> id='var-bb-BBFILES'><glossterm>BBFILES</glossterm> <glossdef>
> <para>
> A space-separated list of recipe files BitBake
> uses to @@ -1113,7 +1113,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BBINCLUDED'><glossterm>BBINCLUDED</glossterm>
> + <glossentry
> id='var-bb-BBINCLUDED'><glossterm>BBINCLUDED</glossterm> <glossdef>
> <para>
> Contains a space-separated list of all of all
> files that @@ -1123,7 +1123,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BBINCLUDELOGS'><glossterm>BBINCLUDELOGS</glossterm>
> + <glossentry
> id='var-bb-BBINCLUDELOGS'><glossterm>BBINCLUDELOGS</glossterm>
> <glossdef> <para>
> If set to a value, enables printing the task log
> when @@ -1132,11 +1132,11 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BBINCLUDELOGS_LINES'><glossterm>BBINCLUDELOGS_LINES</glossterm>
> + <glossentry
> id='var-bb-BBINCLUDELOGS_LINES'><glossterm>BBINCLUDELOGS_LINES</glossterm>
> <glossdef> <para>
> If
> - <link
> linkend='var-BBINCLUDELOGS'><filename>BBINCLUDELOGS</filename></link>
> + <link
> linkend='var-bb-BBINCLUDELOGS'><filename>BBINCLUDELOGS</filename></link>
> is set, specifies the maximum number of lines from the task log file
> to print when reporting a failed task. If you do not set
> <filename>BBINCLUDELOGS_LINES</filename>, @@ -1145,7 +1145,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-BBLAYERS'><glossterm>BBLAYERS</glossterm>
> + <glossentry
> id='var-bb-BBLAYERS'><glossterm>BBLAYERS</glossterm> <glossdef>
> <para>Lists the layers to enable during the build.
> This variable is defined in the
> <filename>bblayers.conf</filename> configuration @@ -1166,7 +1166,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BBLAYERS_FETCH_DIR'><glossterm>BBLAYERS_FETCH_DIR</glossterm>
> + <glossentry
> id='var-bb-BBLAYERS_FETCH_DIR'><glossterm>BBLAYERS_FETCH_DIR</glossterm>
> <glossdef> <para>
> Sets the base location where layers are stored.
> @@ -1178,7 +1178,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-BBMASK'><glossterm>BBMASK</glossterm>
> + <glossentry id='var-bb-BBMASK'><glossterm>BBMASK</glossterm>
> <glossdef>
> <para>
> Prevents BitBake from processing recipes and
> recipe @@ -1236,7 +1236,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BBMULTICONFIG'><glossterm>BBMULTICONFIG</glossterm>
> + <glossentry
> id='var-bb-BBMULTICONFIG'><glossterm>BBMULTICONFIG</glossterm> <info>
> BBMULTICONFIG[doc] = "Enables BitBake to perform
> multiple configuration builds and lists each separate configuration
> (multiconfig)." </info> @@ -1275,7 +1275,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-BBPATH'><glossterm>BBPATH</glossterm>
> + <glossentry id='var-bb-BBPATH'><glossterm>BBPATH</glossterm>
> <glossdef>
> <para>
> Used by BitBake to locate class
> @@ -1302,7 +1302,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-BBSERVER'><glossterm>BBSERVER</glossterm>
> + <glossentry
> id='var-bb-BBSERVER'><glossterm>BBSERVER</glossterm> <glossdef>
> <para>
> Points to the server that runs memory-resident
> BitBake. @@ -1312,7 +1312,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BBTARGETS'><glossterm>BBTARGETS</glossterm>
> + <glossentry
> id='var-bb-BBTARGETS'><glossterm>BBTARGETS</glossterm> <glossdef>
> <para>
> Allows you to use a configuration file to add to
> the list @@ -1321,14 +1321,14 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BBVERSIONS'><glossterm>BBVERSIONS</glossterm>
> + <glossentry
> id='var-bb-BBVERSIONS'><glossterm>BBVERSIONS</glossterm> <glossdef>
> <para>
> Allows a single recipe to build multiple
> versions of a project from a single recipe file.
> You also able to specify conditional metadata
> using the
> - <link
> linkend='var-OVERRIDES'><filename>OVERRIDES</filename></link>
> + <link
> linkend='var-bb-OVERRIDES'><filename>OVERRIDES</filename></link>
> mechanism for a single version or for an optionally named range of
> versions. </para>
> @@ -1342,7 +1342,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BITBAKE_UI'><glossterm>BITBAKE_UI</glossterm>
> + <glossentry
> id='var-bb-BITBAKE_UI'><glossterm>BITBAKE_UI</glossterm> <glossdef>
> <para>
> Used to specify the UI module to use when
> running BitBake. @@ -1356,7 +1356,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BUILDNAME'><glossterm>BUILDNAME</glossterm>
> + <glossentry
> id='var-bb-BUILDNAME'><glossterm>BUILDNAME</glossterm> <glossdef>
> <para>
> A name assigned to the build.
> @@ -1366,7 +1366,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-BZRDIR'><glossterm>BZRDIR</glossterm>
> + <glossentry id='var-bb-BZRDIR'><glossterm>BZRDIR</glossterm>
> <glossdef>
> <para>
> The directory in which files checked out of a
> Bazaar @@ -1377,9 +1377,9 @@
>
> </glossdiv>
>
> - <glossdiv id='var-glossary-c'><title>C</title>
> + <glossdiv id='var-bb-glossary-c'><title>C</title>
>
> - <glossentry id='var-CACHE'><glossterm>CACHE</glossterm>
> + <glossentry id='var-bb-CACHE'><glossterm>CACHE</glossterm>
> <glossdef>
> <para>
> Specifies the directory BitBake uses to store a
> cache @@ -1389,7 +1389,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-CVSDIR'><glossterm>CVSDIR</glossterm>
> + <glossentry id='var-bb-CVSDIR'><glossterm>CVSDIR</glossterm>
> <glossdef>
> <para>
> The directory in which files checked out under
> the @@ -1400,9 +1400,9 @@
>
> </glossdiv>
>
> - <glossdiv id='var-glossary-d'><title>D</title>
> + <glossdiv id='var-bb-glossary-d'><title>D</title>
>
> - <glossentry
> id='var-DEFAULT_PREFERENCE'><glossterm>DEFAULT_PREFERENCE</glossterm>
> + <glossentry
> id='var-bb-DEFAULT_PREFERENCE'><glossterm>DEFAULT_PREFERENCE</glossterm>
> <glossdef> <para>
> Specifies a weak bias for recipe selection
> priority. @@ -1413,20 +1413,20 @@
> piece of software.
> Using the variable in this way causes the stable
> version of the recipe to build by default in the absence of
> - <filename><link
> linkend='var-PREFERRED_VERSION'>PREFERRED_VERSION</link></filename>
> + <filename><link
> linkend='var-bb-PREFERRED_VERSION'>PREFERRED_VERSION</link></filename>
> being used to build the development version. </para>
> <note>
> The bias provided by
> <filename>DEFAULT_PREFERENCE</filename> is weak and is overridden by
> - <filename><link
> linkend='var-BBFILE_PRIORITY'>BBFILE_PRIORITY</link></filename>
> + <filename><link
> linkend='var-bb-BBFILE_PRIORITY'>BBFILE_PRIORITY</link></filename> if
> that variable is different between two layers that contain different
> versions of the same recipe. </note>
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-DEPENDS'><glossterm>DEPENDS</glossterm>
> + <glossentry
> id='var-bb-DEPENDS'><glossterm>DEPENDS</glossterm> <glossdef>
> <para>
> Lists a recipe's build-time dependencies
> @@ -1451,13 +1451,13 @@
>
> <para>
> For information on runtime dependencies, see the
> - <link
> linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link>
> + <link
> linkend='var-bb-RDEPENDS'><filename>RDEPENDS</filename></link>
> variable. </para>
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-DESCRIPTION'><glossterm>DESCRIPTION</glossterm>
> + <glossentry
> id='var-bb-DESCRIPTION'><glossterm>DESCRIPTION</glossterm> <glossdef>
> <para>
> A long description for the recipe.
> @@ -1465,7 +1465,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-DL_DIR'><glossterm>DL_DIR</glossterm>
> + <glossentry id='var-bb-DL_DIR'><glossterm>DL_DIR</glossterm>
> <glossdef>
> <para>
> The central download directory used by the build
> process to @@ -1474,7 +1474,7 @@
> suitable for mirroring for everything except Git
> repositories.
> If you want tarballs of Git repositories, use the
> - <link
> linkend='var-BB_GENERATE_MIRROR_TARBALLS'><filename>BB_GENERATE_MIRROR_TARBALLS</filename></link>
> + <link
> linkend='var-bb-BB_GENERATE_MIRROR_TARBALLS'><filename>BB_GENERATE_MIRROR_TARBALLS</filename></link>
> variable. </para>
> </glossdef>
> @@ -1482,9 +1482,9 @@
> </glossentry>
> </glossdiv>
>
> - <glossdiv id='var-glossary-e'><title>E</title>
> + <glossdiv id='var-bb-glossary-e'><title>E</title>
>
> - <glossentry
> id='var-EXCLUDE_FROM_WORLD'><glossterm>EXCLUDE_FROM_WORLD</glossterm>
> + <glossentry
> id='var-bb-EXCLUDE_FROM_WORLD'><glossterm>EXCLUDE_FROM_WORLD</glossterm>
> <glossdef> <para>
> Directs BitBake to exclude a recipe from world
> builds (i.e. @@ -1512,9 +1512,9 @@
>
> </glossdiv>
>
> - <glossdiv id='var-glossary-f'><title>F</title>
> + <glossdiv id='var-bb-glossary-f'><title>F</title>
>
> - <glossentry id='var-FAKEROOT'><glossterm>FAKEROOT</glossterm>
> + <glossentry
> id='var-bb-FAKEROOT'><glossterm>FAKEROOT</glossterm> <glossdef>
> <para>
> Contains the command to use when running a
> shell script @@ -1527,19 +1527,19 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-FAKEROOTBASEENV'><glossterm>FAKEROOTBASEENV</glossterm>
> + <glossentry
> id='var-bb-FAKEROOTBASEENV'><glossterm>FAKEROOTBASEENV</glossterm>
> <glossdef> <para>
> Lists environment variables to set when
> executing the command defined by
> - <link
> linkend='var-FAKEROOTCMD'><filename>FAKEROOTCMD</filename></link>
> + <link
> linkend='var-bb-FAKEROOTCMD'><filename>FAKEROOTCMD</filename></link>
> that starts the bitbake-worker process in the fakeroot environment.
> </para>
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-FAKEROOTCMD'><glossterm>FAKEROOTCMD</glossterm>
> + <glossentry
> id='var-bb-FAKEROOTCMD'><glossterm>FAKEROOTCMD</glossterm> <glossdef>
> <para>
> Contains the command that starts the
> bitbake-worker @@ -1548,7 +1548,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-FAKEROOTDIRS'><glossterm>FAKEROOTDIRS</glossterm>
> + <glossentry
> id='var-bb-FAKEROOTDIRS'><glossterm>FAKEROOTDIRS</glossterm>
> <glossdef> <para>
> Lists directories to create before running a
> task in @@ -1557,33 +1557,33 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-FAKEROOTENV'><glossterm>FAKEROOTENV</glossterm>
> + <glossentry
> id='var-bb-FAKEROOTENV'><glossterm>FAKEROOTENV</glossterm> <glossdef>
> <para>
> Lists environment variables to set when running
> a task in the fakeroot environment.
> For additional information on environment
> variables and the fakeroot environment, see the
> - <link
> linkend='var-FAKEROOTBASEENV'><filename>FAKEROOTBASEENV</filename></link>
> + <link
> linkend='var-bb-FAKEROOTBASEENV'><filename>FAKEROOTBASEENV</filename></link>
> variable. </para>
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-FAKEROOTNOENV'><glossterm>FAKEROOTNOENV</glossterm>
> + <glossentry
> id='var-bb-FAKEROOTNOENV'><glossterm>FAKEROOTNOENV</glossterm>
> <glossdef> <para>
> Lists environment variables to set when running
> a task that is not in the fakeroot environment.
> For additional information on environment
> variables and the fakeroot environment, see the
> - <link
> linkend='var-FAKEROOTENV'><filename>FAKEROOTENV</filename></link>
> + <link
> linkend='var-bb-FAKEROOTENV'><filename>FAKEROOTENV</filename></link>
> variable. </para>
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-FETCHCMD'><glossterm>FETCHCMD</glossterm>
> + <glossentry
> id='var-bb-FETCHCMD'><glossterm>FETCHCMD</glossterm> <glossdef>
> <para>
> Defines the command the BitBake fetcher module
> @@ -1595,7 +1595,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-FILE'><glossterm>FILE</glossterm>
> + <glossentry id='var-bb-FILE'><glossterm>FILE</glossterm>
> <glossdef>
> <para>
> Points at the current file.
> @@ -1607,7 +1607,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-FILESPATH'><glossterm>FILESPATH</glossterm>
> + <glossentry
> id='var-bb-FILESPATH'><glossterm>FILESPATH</glossterm> <glossdef>
> <para>
> Specifies directories BitBake uses when
> searching for @@ -1625,9 +1625,9 @@
> </glossdiv>
>
>
> - <glossdiv id='var-glossary-g'><title>G</title>
> + <glossdiv id='var-bb-glossary-g'><title>G</title>
>
> - <glossentry id='var-GITDIR'><glossterm>GITDIR</glossterm>
> + <glossentry id='var-bb-GITDIR'><glossterm>GITDIR</glossterm>
> <glossdef>
> <para>
> The directory in which a local copy of a Git
> repository @@ -1639,9 +1639,9 @@
> </glossdiv>
>
>
> - <glossdiv id='var-glossary-h'><title>H</title>
> + <glossdiv id='var-bb-glossary-h'><title>H</title>
>
> - <glossentry id='var-HGDIR'><glossterm>HGDIR</glossterm>
> + <glossentry id='var-bb-HGDIR'><glossterm>HGDIR</glossterm>
> <glossdef>
> <para>
> The directory in which files checked out of a
> Mercurial @@ -1650,7 +1650,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-HOMEPAGE'><glossterm>HOMEPAGE</glossterm>
> + <glossentry
> id='var-bb-HOMEPAGE'><glossterm>HOMEPAGE</glossterm> <glossdef>
> <para>Website where more information about the
> software the recipe is building can be found.</para>
> @@ -1659,9 +1659,9 @@
>
> </glossdiv>
>
> - <glossdiv id='var-glossary-i'><title>I</title>
> + <glossdiv id='var-bb-glossary-i'><title>I</title>
>
> - <glossentry id='var-INHERIT'><glossterm>INHERIT</glossterm>
> + <glossentry
> id='var-bb-INHERIT'><glossterm>INHERIT</glossterm> <glossdef>
> <para>
> Causes the named class or classes to be
> inherited globally. @@ -1691,15 +1691,15 @@
> </glossdiv>
> -->
>
> - <glossdiv id='var-glossary-l'><title>L</title>
> + <glossdiv id='var-bb-glossary-l'><title>L</title>
>
> - <glossentry
> id='var-LAYERDEPENDS'><glossterm>LAYERDEPENDS</glossterm>
> + <glossentry
> id='var-bb-LAYERDEPENDS'><glossterm>LAYERDEPENDS</glossterm>
> <glossdef> <para>Lists the layers, separated by spaces, upon which
> this recipe depends. Optionally, you can specify a specific layer
> version for a dependency by adding it to the end of the layer name
> with a colon, (e.g. "anotherlayer:3" to be compared against
> - <link
> linkend='var-LAYERVERSION'><filename>LAYERVERSION</filename></link><filename>_anotherlayer</filename>
> + <link
> linkend='var-bb-LAYERVERSION'><filename>LAYERVERSION</filename></link><filename>_anotherlayer</filename>
> in this case). BitBake produces an error if any dependency is missing
> or the version numbers do not match exactly (if specified).</para>
> @@ -1710,7 +1710,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-LAYERDIR'><glossterm>LAYERDIR</glossterm>
> + <glossentry
> id='var-bb-LAYERDIR'><glossterm>LAYERDIR</glossterm> <glossdef>
> <para>When used inside the
> <filename>layer.conf</filename> configuration file, this variable
> provides the path of the current layer. @@ -1719,22 +1719,22 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-LAYERDIR_RE'><glossterm>LAYERDIR_RE</glossterm>
> + <glossentry
> id='var-bb-LAYERDIR_RE'><glossterm>LAYERDIR_RE</glossterm> <glossdef>
> <para>When used inside the
> <filename>layer.conf</filename> configuration file, this variable
> provides the path of the current layer, escaped for use in a regular
> expression
> - (<link
> linkend='var-BBFILE_PATTERN'><filename>BBFILE_PATTERN</filename></link>).
> + (<link
> linkend='var-bb-BBFILE_PATTERN'><filename>BBFILE_PATTERN</filename></link>).
> This variable is not available outside of
> <filename>layer.conf</filename> and references are expanded
> immediately when parsing of the file completes.</para> </glossdef>
> </glossentry>
> - <glossentry
> id='var-LAYERVERSION'><glossterm>LAYERVERSION</glossterm>
> + <glossentry
> id='var-bb-LAYERVERSION'><glossterm>LAYERVERSION</glossterm>
> <glossdef> <para>Optionally specifies the version of a layer as a
> single number. You can use this variable within
> - <link
> linkend='var-LAYERDEPENDS'><filename>LAYERDEPENDS</filename></link>
> + <link
> linkend='var-bb-LAYERDEPENDS'><filename>LAYERDEPENDS</filename></link>
> for another layer in order to depend on a specific version of the
> layer.</para> <para>
> @@ -1744,7 +1744,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-LICENSE'><glossterm>LICENSE</glossterm>
> + <glossentry
> id='var-bb-LICENSE'><glossterm>LICENSE</glossterm> <glossdef>
> <para>
> The list of source licenses for the recipe.
> @@ -1754,9 +1754,9 @@
>
> </glossdiv>
>
> - <glossdiv id='var-glossary-m'><title>M</title>
> + <glossdiv id='var-bb-glossary-m'><title>M</title>
>
> - <glossentry id='var-MIRRORS'><glossterm>MIRRORS</glossterm>
> + <glossentry
> id='var-bb-MIRRORS'><glossterm>MIRRORS</glossterm> <glossdef>
> <para>
> Specifies additional paths from which BitBake
> gets source code. @@ -1764,14 +1764,14 @@
> tries the local download directory.
> If that location fails, the build system tries
> locations defined by
> - <link
> linkend='var-PREMIRRORS'><filename>PREMIRRORS</filename></link>,
> + <link
> linkend='var-bb-PREMIRRORS'><filename>PREMIRRORS</filename></link>,
> the upstream source, and then locations specified by
> <filename>MIRRORS</filename> in that order. </para>
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-MULTI_PROVIDER_WHITELIST'><glossterm>MULTI_PROVIDER_WHITELIST</glossterm>
> + <glossentry
> id='var-bb-MULTI_PROVIDER_WHITELIST'><glossterm>MULTI_PROVIDER_WHITELIST</glossterm>
> <glossdef> <para>
> Allows you to suppress BitBake warnings caused
> when @@ -1804,9 +1804,9 @@
> </glossdiv>
> -->
>
> - <glossdiv id='var-glossary-o'><title>O</title>
> + <glossdiv id='var-bb-glossary-o'><title>O</title>
>
> - <glossentry
> id='var-OVERRIDES'><glossterm>OVERRIDES</glossterm>
> + <glossentry
> id='var-bb-OVERRIDES'><glossterm>OVERRIDES</glossterm> <glossdef>
> <para>
> BitBake uses <filename>OVERRIDES</filename> to
> control @@ -1829,9 +1829,9 @@
> </glossentry>
> </glossdiv>
>
> - <glossdiv id='var-glossary-p'><title>P</title>
> + <glossdiv id='var-bb-glossary-p'><title>P</title>
>
> - <glossentry id='var-P4DIR'><glossterm>P4DIR</glossterm>
> + <glossentry id='var-bb-P4DIR'><glossterm>P4DIR</glossterm>
> <glossdef>
> <para>
> The directory in which a local copy of a
> Perforce depot @@ -1840,14 +1840,14 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-PACKAGES'><glossterm>PACKAGES</glossterm>
> + <glossentry
> id='var-bb-PACKAGES'><glossterm>PACKAGES</glossterm> <glossdef>
> <para>The list of packages the recipe creates.
> </para>
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-PACKAGES_DYNAMIC'><glossterm>PACKAGES_DYNAMIC</glossterm>
> + <glossentry
> id='var-bb-PACKAGES_DYNAMIC'><glossterm>PACKAGES_DYNAMIC</glossterm>
> <glossdef> <para>
> A promise that your recipe satisfies runtime
> dependencies @@ -1856,7 +1856,7 @@
> does not actually satisfy the dependencies, it
> only states that they should be satisfied.
> For example, if a hard, runtime dependency
> - (<link
> linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link>)
> + (<link
> linkend='var-bb-RDEPENDS'><filename>RDEPENDS</filename></link>) of
> another package is satisfied during the build through the
> <filename>PACKAGES_DYNAMIC</filename> variable, but a package with
> the module name is never actually @@ -1865,7 +1865,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-PE'><glossterm>PE</glossterm>
> + <glossentry id='var-bb-PE'><glossterm>PE</glossterm>
> <glossdef>
> <para>
> The epoch of the recipe.
> @@ -1877,7 +1877,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-PERSISTENT_DIR'><glossterm>PERSISTENT_DIR</glossterm>
> + <glossentry
> id='var-bb-PERSISTENT_DIR'><glossterm>PERSISTENT_DIR</glossterm>
> <glossdef> <para>
> Specifies the directory BitBake uses to store
> data that @@ -1889,7 +1889,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-PF'><glossterm>PF</glossterm>
> + <glossentry id='var-bb-PF'><glossterm>PF</glossterm>
> <glossdef>
> <para>
> Specifies the recipe or package name and
> includes all version and revision @@ -1899,27 +1899,27 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-PN'><glossterm>PN</glossterm>
> + <glossentry id='var-bb-PN'><glossterm>PN</glossterm>
> <glossdef>
> <para>The recipe name.</para>
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-PR'><glossterm>PR</glossterm>
> + <glossentry id='var-bb-PR'><glossterm>PR</glossterm>
> <glossdef>
> <para>The revision of the recipe.
> </para>
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-PREFERRED_PROVIDER'><glossterm>PREFERRED_PROVIDER</glossterm>
> + <glossentry
> id='var-bb-PREFERRED_PROVIDER'><glossterm>PREFERRED_PROVIDER</glossterm>
> <glossdef> <para>
> Determines which recipe should be given
> preference when multiple recipes provide the same item.
> You should always suffix the variable with the
> name of the provided item, and you should set it to the
> - <link
> linkend='var-PN'><filename>PN</filename></link>
> + <link
> linkend='var-bb-PN'><filename>PN</filename></link> of the recipe to
> which you want to give precedence. Some examples:
> <literallayout class='monospaced'>
> @@ -1931,14 +1931,14 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-PREFERRED_PROVIDERS'><glossterm>PREFERRED_PROVIDERS</glossterm>
> + <glossentry
> id='var-bb-PREFERRED_PROVIDERS'><glossterm>PREFERRED_PROVIDERS</glossterm>
> <glossdef> <para>
> Determines which recipe should be given
> preference for cases where multiple recipes provide the same item.
> Functionally,
> <filename>PREFERRED_PROVIDERS</filename> is
> identical to
> - <link
> linkend='var-PREFERRED_PROVIDER'><filename>PREFERRED_PROVIDER</filename></link>.
> + <link
> linkend='var-bb-PREFERRED_PROVIDER'><filename>PREFERRED_PROVIDER</filename></link>.
> However, the <filename>PREFERRED_PROVIDERS</filename> variable lets
> you define preferences for multiple situations using the following
> form: @@ -1954,15 +1954,15 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-PREFERRED_VERSION'><glossterm>PREFERRED_VERSION</glossterm>
> + <glossentry
> id='var-bb-PREFERRED_VERSION'><glossterm>PREFERRED_VERSION</glossterm>
> <glossdef> <para>
> If there are multiple versions of recipes
> available, this variable determines which recipe should be given
> preference. You must always suffix the variable with the
> - <link
> linkend='var-PN'><filename>PN</filename></link>
> + <link
> linkend='var-bb-PN'><filename>PN</filename></link> you want to
> select, and you should set
> - <link
> linkend='var-PV'><filename>PV</filename></link>
> + <link
> linkend='var-bb-PV'><filename>PV</filename></link> accordingly for
> precedence. </para>
>
> @@ -1989,7 +1989,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-PREMIRRORS'><glossterm>PREMIRRORS</glossterm>
> + <glossentry
> id='var-bb-PREMIRRORS'><glossterm>PREMIRRORS</glossterm> <glossdef>
> <para>
> Specifies additional paths from which BitBake
> gets source code. @@ -1998,7 +1998,7 @@
> If that location fails, the build system tries
> locations defined by <filename>PREMIRRORS</filename>, the upstream
> source, and then locations specified by
> - <link
> linkend='var-MIRRORS'><filename>MIRRORS</filename></link>
> + <link
> linkend='var-bb-MIRRORS'><filename>MIRRORS</filename></link> in that
> order. </para>
>
> @@ -2022,20 +2022,20 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-PROVIDES'><glossterm>PROVIDES</glossterm>
> + <glossentry
> id='var-bb-PROVIDES'><glossterm>PROVIDES</glossterm> <glossdef>
> <para>
> A list of aliases by which a particular recipe
> can be known.
> By default, a recipe's own
> - <filename><link
> linkend='var-PN'>PN</link></filename>
> + <filename><link
> linkend='var-bb-PN'>PN</link></filename> is implicitly already in its
> <filename>PROVIDES</filename> list.
> If a recipe uses <filename>PROVIDES</filename>,
> the additional aliases are synonyms for the recipe and can
> be useful satisfying dependencies of other
> recipes during the build as specified by
> - <filename><link
> linkend='var-DEPENDS'>DEPENDS</link></filename>.
> + <filename><link
> linkend='var-bb-DEPENDS'>DEPENDS</link></filename>. </para>
>
> <para>
> @@ -2059,7 +2059,7 @@
> virtual target in <filename>PROVIDES</filename>.
> Recipes that depend on the functionality in
> question can include the virtual target in
> - <link
> linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
> + <link
> linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link> to leave
> the choice of provider open. </para>
>
> @@ -2072,11 +2072,11 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-PRSERV_HOST'><glossterm>PRSERV_HOST</glossterm>
> + <glossentry
> id='var-bb-PRSERV_HOST'><glossterm>PRSERV_HOST</glossterm> <glossdef>
> <para>
> The network based
> - <link
> linkend='var-PR'><filename>PR</filename></link>
> + <link
> linkend='var-bb-PR'><filename>PR</filename></link> service host and
> port. </para>
>
> @@ -2094,7 +2094,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-PV'><glossterm>PV</glossterm>
> + <glossentry id='var-bb-PV'><glossterm>PV</glossterm>
> <glossdef>
> <para>The version of the recipe.
> </para>
> @@ -2108,9 +2108,9 @@
> </glossdiv>
> -->
>
> - <glossdiv id='var-glossary-r'><title>R</title>
> + <glossdiv id='var-bb-glossary-r'><title>R</title>
>
> - <glossentry id='var-RDEPENDS'><glossterm>RDEPENDS</glossterm>
> + <glossentry
> id='var-bb-RDEPENDS'><glossterm>RDEPENDS</glossterm> <glossdef>
> <para>
> Lists a package's runtime dependencies (i.e.
> other packages) @@ -2165,13 +2165,13 @@
>
> <para>
> For information on build-time dependencies, see
> the
> - <link
> linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
> + <link
> linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link> variable.
> </para>
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-REPODIR'><glossterm>REPODIR</glossterm>
> + <glossentry
> id='var-bb-REPODIR'><glossterm>REPODIR</glossterm> <glossdef>
> <para>
> The directory in which a local copy of a
> @@ -2181,14 +2181,14 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-RPROVIDES'><glossterm>RPROVIDES</glossterm>
> + <glossentry
> id='var-bb-RPROVIDES'><glossterm>RPROVIDES</glossterm> <glossdef>
> <para>
> A list of package name aliases that a package
> also provides. These aliases are useful for satisfying runtime
> dependencies of other packages both during the build and on the target
> (as specified by
> - <filename><link
> linkend='var-RDEPENDS'>RDEPENDS</link></filename>).
> + <filename><link
> linkend='var-bb-RDEPENDS'>RDEPENDS</link></filename>). </para>
> <para>
> As with all package-controlling variables, you
> must always @@ -2201,7 +2201,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-RRECOMMENDS'><glossterm>RRECOMMENDS</glossterm>
> + <glossentry
> id='var-bb-RRECOMMENDS'><glossterm>RRECOMMENDS</glossterm> <glossdef>
> <para>
> A list of packages that extends the usability of
> a package @@ -2210,7 +2210,7 @@
> packages in order to successfully build, but
> needs them for the extended usability.
> To specify runtime dependencies for packages,
> see the
> - <filename><link
> linkend='var-RDEPENDS'>RDEPENDS</link></filename>
> + <filename><link
> linkend='var-bb-RDEPENDS'>RDEPENDS</link></filename> variable.
> </para>
>
> @@ -2243,15 +2243,15 @@
>
> </glossdiv>
>
> - <glossdiv id='var-glossary-s'><title>S</title>
> + <glossdiv id='var-bb-glossary-s'><title>S</title>
>
> - <glossentry id='var-SECTION'><glossterm>SECTION</glossterm>
> + <glossentry
> id='var-bb-SECTION'><glossterm>SECTION</glossterm> <glossdef>
> <para>The section in which packages should be
> categorized.</para> </glossdef>
> </glossentry>
>
> - <glossentry id='var-SRC_URI'><glossterm>SRC_URI</glossterm>
> + <glossentry
> id='var-bb-SRC_URI'><glossterm>SRC_URI</glossterm> <glossdef>
> <para>
> The list of source files - local or remote.
> @@ -2272,7 +2272,7 @@
> the metadata,
> from the local machine.
> The path is relative to the
> - <link
> linkend='var-FILESPATH'><filename>FILESPATH</filename></link>
> + <link
> linkend='var-bb-FILESPATH'><filename>FILESPATH</filename></link>
> variable.</para></listitem>
> <listitem><para><emphasis><filename>bzr://</filename> -</emphasis>
> Fetches files from a Bazaar revision control
> repository.</para></listitem> @@ -2322,7 +2322,7 @@ </glossdef>
> </glossentry>
>
> - <glossentry id='var-SRCDATE'><glossterm>SRCDATE</glossterm>
> + <glossentry
> id='var-bb-SRCDATE'><glossterm>SRCDATE</glossterm> <glossdef>
> <para>
> The date of the source code used to build the
> package. @@ -2331,7 +2331,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-SRCREV'><glossterm>SRCREV</glossterm>
> + <glossentry id='var-bb-SRCREV'><glossterm>SRCREV</glossterm>
> <glossdef>
> <para>
> The revision of the source code used to build
> the package. @@ -2344,13 +2344,13 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-SRCREV_FORMAT'><glossterm>SRCREV_FORMAT</glossterm>
> + <glossentry
> id='var-bb-SRCREV_FORMAT'><glossterm>SRCREV_FORMAT</glossterm>
> <glossdef> <para>
> Helps construct valid
> - <link
> linkend='var-SRCREV'><filename>SRCREV</filename></link>
> + <link
> linkend='var-bb-SRCREV'><filename>SRCREV</filename></link> values
> when multiple source controlled URLs are used in
> - <link
> linkend='var-SRC_URI'><filename>SRC_URI</filename></link>.
> + <link
> linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link>. </para>
>
> <para>
> @@ -2371,7 +2371,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-STAMP'><glossterm>STAMP</glossterm>
> + <glossentry id='var-bb-STAMP'><glossterm>STAMP</glossterm>
> <glossdef>
> <para>
> Specifies the base path used to create recipe
> stamp files. @@ -2381,12 +2381,12 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-STAMPCLEAN'><glossterm>STAMPCLEAN</glossterm>
> + <glossentry
> id='var-bb-STAMPCLEAN'><glossterm>STAMPCLEAN</glossterm> <glossdef>
> <para>
> Specifies the base path used to create recipe
> stamp files. Unlike the
> - <link
> linkend='var-STAMP'><filename>STAMP</filename></link>
> + <link
> linkend='var-bb-STAMP'><filename>STAMP</filename></link> variable,
> <filename>STAMPCLEAN</filename> can contain wildcards to match the
> range of files a clean operation should remove.
> @@ -2396,7 +2396,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-SUMMARY'><glossterm>SUMMARY</glossterm>
> + <glossentry
> id='var-bb-SUMMARY'><glossterm>SUMMARY</glossterm> <glossdef>
> <para>
> A short summary for the recipe, which is 72
> characters or less. @@ -2404,7 +2404,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-SVNDIR'><glossterm>SVNDIR</glossterm>
> + <glossentry id='var-bb-SVNDIR'><glossterm>SVNDIR</glossterm>
> <glossdef>
> <para>
> The directory in which files checked out of a
> Subversion @@ -2415,9 +2415,9 @@
>
> </glossdiv>
>
> - <glossdiv id='var-glossary-t'><title>T</title>
> + <glossdiv id='var-bb-glossary-t'><title>T</title>
>
> - <glossentry id='var-T'><glossterm>T</glossterm>
> + <glossentry id='var-bb-T'><glossterm>T</glossterm>
> <glossdef>
> <para>Points to a directory were BitBake places
> temporary files, which consist mostly of task
> logs and @@ -2426,7 +2426,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-TOPDIR'><glossterm>TOPDIR</glossterm>
> + <glossentry id='var-bb-TOPDIR'><glossterm>TOPDIR</glossterm>
> <glossdef>
> <para>
> Points to the build directory.
> diff --git a/bitbake/doc/poky.ent b/bitbake/doc/poky.ent
> index c032e14..85d9c83 100644
> --- a/bitbake/doc/poky.ent
> +++ b/bitbake/doc/poky.ent
> @@ -17,13 +17,6 @@
> <!ENTITY OE_DOCS_URL "http://docs.openembedded.org">
> <!ENTITY OH_HOME_URL "http://o-hand.com">
> <!ENTITY BITBAKE_HOME_URL
> "http://developer.berlios.de/projects/bitbake/"> -<!ENTITY
> ECLIPSE_MAIN_URL "http://www.eclipse.org/downloads"> -<!ENTITY
> ECLIPSE_DL_URL "http://download.eclipse.org"> -<!ENTITY
> ECLIPSE_DL_PLUGIN_URL
> "&YOCTO_DL_URL;/releases/eclipse-plugin/&DISTRO;"> -<!ENTITY
> ECLIPSE_UPDATES_URL "&ECLIPSE_DL_URL;/tm/updates/3.3"> -<!ENTITY
> ECLIPSE_INDIGO_URL "&ECLIPSE_DL_URL;/releases/indigo"> -<!ENTITY
> ECLIPSE_JUNO_URL "&ECLIPSE_DL_URL;/releases/juno"> -<!ENTITY
> ECLIPSE_INDIGO_CDT_URL "&ECLIPSE_DL_URL;tools/cdt/releases/indigo">
> <!ENTITY YOCTO_DOCS_URL "&YOCTO_HOME_URL;/docs"> <!ENTITY
> YOCTO_SOURCES_URL "&YOCTO_HOME_URL;/sources/"> <!ENTITY
> YOCTO_AB_PORT_URL "&YOCTO_AB_URL;:8010"> @@ -31,7 +24,6 @@ <!ENTITY
> YOCTO_POKY_URL "&YOCTO_DL_URL;/releases/poky/"> <!ENTITY
> YOCTO_RELEASE_DL_URL "&YOCTO_DL_URL;/releases/yocto/yocto-&DISTRO;">
> <!ENTITY YOCTO_TOOLCHAIN_DL_URL "&YOCTO_RELEASE_DL_URL;/toolchain/">
> -<!ENTITY YOCTO_ECLIPSE_DL_URL
> "&YOCTO_RELEASE_DL_URL;/eclipse-plugin/indigo;"> <!ENTITY
> YOCTO_ADTINSTALLER_DL_URL "&YOCTO_RELEASE_DL_URL;/adt_installer">
> <!ENTITY YOCTO_POKY_DL_URL
> "&YOCTO_RELEASE_DL_URL;/&YOCTO_POKY;.tar.bz2"> <!ENTITY
> YOCTO_MACHINES_DL_URL "&YOCTO_RELEASE_DL_URL;/machines"> diff --git
> a/bitbake/lib/bb/COW.py b/bitbake/lib/bb/COW.py index
> 7817473..d26e981 100644 --- a/bitbake/lib/bb/COW.py +++
> b/bitbake/lib/bb/COW.py @@ -1,23 +1,8 @@ -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # This is a copy on write dictionary and set which abuses classes to
> try and be nice and fast. #
> # Copyright (C) 2006 Tim Ansell
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. -#
> #Please Note:
> # Be careful when using mutable types (ie Dict and Lists) -
> operations involving these are SLOW. # Assign a file to __warn__ to
> get warnings about slow operations. diff --git
> a/bitbake/lib/bb/__init__.py b/bitbake/lib/bb/__init__.py index
> 4bc47c8..c144311 100644 --- a/bitbake/lib/bb/__init__.py
> +++ b/bitbake/lib/bb/__init__.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # BitBake Build System Python Library
> #
> @@ -8,20 +6,10 @@
> #
> # Based on Gentoo's portage.py.
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> -__version__ = "1.40.0"
> +__version__ = "1.44.0"
>
> import sys
> if sys.version_info < (3, 4, 0):
> diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py
> index 3e2a94e..30a2ba2 100644
> --- a/bitbake/lib/bb/build.py
> +++ b/bitbake/lib/bb/build.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # BitBake 'Build' implementation
> #
> @@ -10,18 +8,7 @@
> #
> # Based on Gentoo's portage.py.
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
> # Based on functions from the base bb module, Copyright 2003 Holger
> Schurig
> @@ -67,23 +54,6 @@ else:
> builtins['bb'] = bb
> builtins['os'] = os
>
> -class FuncFailed(Exception):
> - def __init__(self, name = None, logfile = None):
> - self.logfile = logfile
> - self.name = name
> - if name:
> - self.msg = 'Function failed: %s' % name
> - else:
> - self.msg = "Function failed"
> -
> - def __str__(self):
> - if self.logfile and os.path.exists(self.logfile):
> - msg = ("%s (log file is located at %s)" %
> - (self.msg, self.logfile))
> - else:
> - msg = self.msg
> - return msg
> -
> class TaskBase(event.Event):
> """Base class for task events"""
>
> @@ -176,15 +146,33 @@ class LogTee(object):
>
> def __repr__(self):
> return '<LogTee {0}>'.format(self.name)
> +
> def flush(self):
> self.outfile.flush()
>
> -#
> -# pythonexception allows the python exceptions generated to be raised
> -# as the real exceptions (not FuncFailed) and without a backtrace at
> the -# origin of the failure.
> -#
> -def exec_func(func, d, dirs = None, pythonexception=False):
> +
> +class StdoutNoopContextManager:
> + """
> + This class acts like sys.stdout, but adds noop __enter__ and
> __exit__ methods.
> + """
> + def __enter__(self):
> + return sys.stdout
> +
> + def __exit__(self, *exc_info):
> + pass
> +
> + def write(self, string):
> + return sys.stdout.write(string)
> +
> + def flush(self):
> + sys.stdout.flush()
> +
> + @property
> + def name(self):
> + return sys.stdout.name
> +
> +
> +def exec_func(func, d, dirs = None):
> """Execute a BB 'function'"""
>
> try:
> @@ -256,7 +244,7 @@ def exec_func(func, d, dirs = None,
> pythonexception=False):
> with bb.utils.fileslocked(lockfiles):
> if ispython:
> - exec_func_python(func, d, runfile, cwd=adir,
> pythonexception=pythonexception)
> + exec_func_python(func, d, runfile, cwd=adir)
> else:
> exec_func_shell(func, d, runfile, cwd=adir)
>
> @@ -276,7 +264,7 @@ _functionfmt = """
> {function}(d)
> """
> logformatter = bb.msg.BBLogFormatter("%(levelname)s: %(message)s")
> -def exec_func_python(func, d, runfile, cwd=None,
> pythonexception=False): +def exec_func_python(func, d, runfile,
> cwd=None): """Execute a python BB 'function'"""
>
> code = _functionfmt.format(function=func)
> @@ -301,13 +289,7 @@ def exec_func_python(func, d, runfile, cwd=None,
> pythonexception=False): bb.methodpool.insert_method(func, text, fn,
> lineno - 1)
> comp = utils.better_compile(code, func, "exec_python_func()
> autogenerated")
> - utils.better_exec(comp, {"d": d}, code, "exec_python_func()
> autogenerated", pythonexception=pythonexception)
> - except (bb.parse.SkipRecipe, bb.build.FuncFailed):
> - raise
> - except:
> - if pythonexception:
> - raise
> - raise FuncFailed(func, None)
> + utils.better_exec(comp, {"d": d}, code, "exec_python_func()
> autogenerated") finally:
> bb.debug(2, "Python function %s finished" % func)
>
> @@ -335,6 +317,42 @@ trap 'bb_exit_handler' 0
> set -e
> '''
>
> +def create_progress_handler(func, progress, logfile, d):
> + if progress == 'percent':
> + # Use default regex
> + return bb.progress.BasicProgressHandler(d, outfile=logfile)
> + elif progress.startswith('percent:'):
> + # Use specified regex
> + return bb.progress.BasicProgressHandler(d,
> regex=progress.split(':', 1)[1], outfile=logfile)
> + elif progress.startswith('outof:'):
> + # Use specified regex
> + return bb.progress.OutOfProgressHandler(d,
> regex=progress.split(':', 1)[1], outfile=logfile)
> + elif progress.startswith("custom:"):
> + # Use a custom progress handler that was injected via
> OE_EXTRA_IMPORTS or __builtins__
> + import functools
> + from types import ModuleType
> +
> + parts = progress.split(":", 2)
> + _, cls, otherargs = parts[0], parts[1], (parts[2] or None)
> if parts[2:] else None
> + if cls:
> + def resolve(x, y):
> + if not x:
> + return None
> + if isinstance(x, ModuleType):
> + return getattr(x, y, None)
> + return x.get(y)
> + cls_obj = functools.reduce(resolve, cls.split("."),
> bb.utils._context)
> + if not cls_obj:
> + # Fall-back on __builtins__
> + cls_obj = functools.reduce(lambda x, y: x.get(y),
> cls.split("."), __builtins__)
> + if cls_obj:
> + return cls_obj(d, outfile=logfile,
> otherargs=otherargs)
> + bb.warn('%s: unknown custom progress handler in task
> progress varflag value "%s", ignoring' % (func, cls))
> + else:
> + bb.warn('%s: invalid task progress varflag value "%s",
> ignoring' % (func, progress)) +
> + return logfile
> +
> def exec_func_shell(func, d, runfile, cwd=None):
> """Execute a shell function from the metadata
>
> @@ -372,23 +390,13 @@ exit $ret
> cmd = [fakerootcmd, runfile]
>
> if bb.msg.loggerDefaultVerbose:
> - logfile = LogTee(logger, sys.stdout)
> + logfile = LogTee(logger, StdoutNoopContextManager())
> else:
> - logfile = sys.stdout
> + logfile = StdoutNoopContextManager()
>
> progress = d.getVarFlag(func, 'progress')
> if progress:
> - if progress == 'percent':
> - # Use default regex
> - logfile = bb.progress.BasicProgressHandler(d,
> outfile=logfile)
> - elif progress.startswith('percent:'):
> - # Use specified regex
> - logfile = bb.progress.BasicProgressHandler(d,
> regex=progress.split(':', 1)[1], outfile=logfile)
> - elif progress.startswith('outof:'):
> - # Use specified regex
> - logfile = bb.progress.OutOfProgressHandler(d,
> regex=progress.split(':', 1)[1], outfile=logfile)
> - else:
> - bb.warn('%s: invalid task progress varflag value "%s",
> ignoring' % (func, progress))
> + logfile = create_progress_handler(func, progress, logfile, d)
>
> fifobuffer = bytearray()
> def readfifo(data):
> @@ -407,6 +415,8 @@ exit $ret
> bb.plain(value)
> elif cmd == 'bbnote':
> bb.note(value)
> + elif cmd == 'bbverbnote':
> + bb.verbnote(value)
> elif cmd == 'bbwarn':
> bb.warn(value)
> elif cmd == 'bberror':
> @@ -436,13 +446,8 @@ exit $ret
> with open(fifopath, 'r+b', buffering=0) as fifo:
> try:
> bb.debug(2, "Executing shell function %s" % func)
> -
> - try:
> - with open(os.devnull, 'r+') as stdin:
> - bb.process.run(cmd, shell=False, stdin=stdin,
> log=logfile, extrafiles=[(fifo,readfifo)])
> - except bb.process.CmdError:
> - logfn = d.getVar('BB_LOGFILE')
> - raise FuncFailed(func, logfn)
> + with open(os.devnull, 'r+') as stdin, logfile:
> + bb.process.run(cmd, shell=False, stdin=stdin,
> log=logfile, extrafiles=[(fifo,readfifo)]) finally:
> os.unlink(fifopath)
>
> @@ -570,9 +575,6 @@ def _exec_task(fn, task, d, quieterr):
> event.fire(TaskStarted(task, logfn, flags, localdata),
> localdata) except (bb.BBHandledException, SystemExit):
> return 1
> - except FuncFailed as exc:
> - logger.error(str(exc))
> - return 1
>
> try:
> for func in (prefuncs or '').split():
> @@ -580,7 +582,10 @@ def _exec_task(fn, task, d, quieterr):
> exec_func(task, localdata)
> for func in (postfuncs or '').split():
> exec_func(func, localdata)
> - except FuncFailed as exc:
> + except bb.BBHandledException:
> + event.fire(TaskFailed(task, logfn, localdata, True),
> localdata)
> + return 1
> + except Exception as exc:
> if quieterr:
> event.fire(TaskFailedSilent(task, logfn, localdata),
> localdata) else:
> @@ -588,9 +593,6 @@ def _exec_task(fn, task, d, quieterr):
> logger.error(str(exc))
> event.fire(TaskFailed(task, logfn, localdata,
> errprinted), localdata) return 1
> - except bb.BBHandledException:
> - event.fire(TaskFailed(task, logfn, localdata, True),
> localdata)
> - return 1
> finally:
> sys.stdout.flush()
> sys.stderr.flush()
> @@ -814,6 +816,9 @@ def add_tasks(tasklist, d):
> task_deps['parents'][task] = []
> if 'deps' in flags:
> for dep in flags['deps']:
> + # Check and warn for "addtask task after foo" while
> foo does not exist
> + #if not dep in tasklist:
> + # bb.warn('%s: dependent task %s for %s does not
> exist' % (d.getVar('PN'), dep, task)) dep = d.expand(dep)
> task_deps['parents'][task].append(dep)
>
> diff --git a/bitbake/lib/bb/cache.py b/bitbake/lib/bb/cache.py
> index 258d679..b6f7da5 100644
> --- a/bitbake/lib/bb/cache.py
> +++ b/bitbake/lib/bb/cache.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # BitBake Cache implementation
> #
> @@ -15,18 +13,8 @@
> # Copyright (C) 2005 Holger Hans Peter Freyther
> # Copyright (C) 2005 ROAD GmbH
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import os
> import sys
> @@ -95,21 +83,21 @@ class CoreRecipeInfo(RecipeInfoCommon):
> self.appends = self.listvar('__BBAPPEND', metadata)
> self.nocache = self.getvar('BB_DONT_CACHE', metadata)
>
> + self.provides = self.depvar('PROVIDES', metadata)
> + self.rprovides = self.depvar('RPROVIDES', metadata)
> + self.pn = self.getvar('PN', metadata) or
> bb.parse.vars_from_file(filename,metadata)[0]
> + self.packages = self.listvar('PACKAGES', metadata)
> + if not self.packages:
> + self.packages.append(self.pn)
> + self.packages_dynamic = self.listvar('PACKAGES_DYNAMIC',
> metadata) +
> self.skipreason = self.getvar('__SKIPPED', metadata)
> if self.skipreason:
> - self.pn = self.getvar('PN', metadata) or
> bb.parse.BBHandler.vars_from_file(filename,metadata)[0] self.skipped
> = True
> - self.provides = self.depvar('PROVIDES', metadata)
> - self.rprovides = self.depvar('RPROVIDES', metadata)
> return
>
> self.tasks = metadata.getVar('__BBTASKS', False)
>
> - self.pn = self.getvar('PN', metadata)
> - self.packages = self.listvar('PACKAGES', metadata)
> - if not self.packages:
> - self.packages.append(self.pn)
> -
> self.basetaskhashes = self.taskvar('BB_BASEHASH',
> self.tasks, metadata) self.hashfilename =
> self.getvar('BB_HASHFILENAME', metadata)
> @@ -125,11 +113,8 @@ class CoreRecipeInfo(RecipeInfoCommon):
> self.stampclean = self.getvar('STAMPCLEAN', metadata)
> self.stamp_extrainfo = self.flaglist('stamp-extra-info',
> self.tasks, metadata) self.file_checksums =
> self.flaglist('file-checksums', self.tasks, metadata, True)
> - self.packages_dynamic = self.listvar('PACKAGES_DYNAMIC',
> metadata) self.depends = self.depvar('DEPENDS', metadata)
> - self.provides = self.depvar('PROVIDES', metadata)
> self.rdepends = self.depvar('RDEPENDS', metadata)
> - self.rprovides = self.depvar('RPROVIDES', metadata)
> self.rrecommends = self.depvar('RRECOMMENDS', metadata)
> self.rprovides_pkg = self.pkgvar('RPROVIDES',
> self.packages, metadata) self.rdepends_pkg =
> self.pkgvar('RDEPENDS', self.packages, metadata) @@ -235,7 +220,7 @@
> class CoreRecipeInfo(RecipeInfoCommon):
> cachedata.hashfn[fn] = self.hashfilename
> for task, taskhash in self.basetaskhashes.items():
> - identifier = '%s.%s' % (fn, task)
> + identifier = '%s:%s' % (fn, task)
> cachedata.basetaskhash[identifier] = taskhash
>
> cachedata.inherits[fn] = self.inherits
> @@ -249,7 +234,7 @@ def virtualfn2realfn(virtualfn):
> Convert a virtual file name to a real one + the associated
> subclass keyword """
> mc = ""
> - if virtualfn.startswith('multiconfig:'):
> + if virtualfn.startswith('mc:'):
> elems = virtualfn.split(':')
> mc = elems[1]
> virtualfn = ":".join(elems[2:])
> @@ -270,7 +255,7 @@ def realfn2virtual(realfn, cls, mc):
> if cls:
> realfn = "virtual:" + cls + ":" + realfn
> if mc:
> - realfn = "multiconfig:" + mc + ":" + realfn
> + realfn = "mc:" + mc + ":" + realfn
> return realfn
>
> def variant2virtual(realfn, variant):
> @@ -279,11 +264,11 @@ def variant2virtual(realfn, variant):
> """
> if variant == "":
> return realfn
> - if variant.startswith("multiconfig:"):
> + if variant.startswith("mc:"):
> elems = variant.split(":")
> if elems[2]:
> - return "multiconfig:" + elems[1] + ":virtual:" +
> ":".join(elems[2:]) + ":" + realfn
> - return "multiconfig:" + elems[1] + ":" + realfn
> + return "mc:" + elems[1] + ":virtual:" +
> ":".join(elems[2:]) + ":" + realfn
> + return "mc:" + elems[1] + ":" + realfn
> return "virtual:" + variant + ":" + realfn
>
> def parse_recipe(bb_data, bbfile, appends, mc=''):
> @@ -361,7 +346,7 @@ class NoCache(object):
> bb_data = self.databuilder.mcdata[mc].createCopy()
> newstores = parse_recipe(bb_data, bbfile, appends, mc)
> for ns in newstores:
> - datastores["multiconfig:%s:%s" % (mc, ns)] =
> newstores[ns]
> + datastores["mc:%s:%s" % (mc, ns)] = newstores[ns]
>
> return datastores
>
> @@ -411,6 +396,15 @@ class Cache(NoCache):
> else:
> logger.debug(1, "Cache file %s not found, building..." %
> self.cachefile)
> + # We don't use the symlink, its just for debugging
> convinience
> + symlink = os.path.join(self.cachedir, "bb_cache.dat")
> + if os.path.exists(symlink):
> + bb.utils.remove(symlink)
> + try:
> + os.symlink(os.path.basename(self.cachefile), symlink)
> + except OSError:
> + pass
> +
> def load_cachefile(self):
> cachesize = 0
> previous_progress = 0
> @@ -889,3 +883,56 @@ class MultiProcessCache(object):
> p.dump([data, self.__class__.CACHE_VERSION])
>
> bb.utils.unlockfile(glf)
> +
> +
> +class SimpleCache(object):
> + """
> + BitBake multi-process cache implementation
> +
> + Used by the codeparser & file checksum caches
> + """
> +
> + def __init__(self, version):
> + self.cachefile = None
> + self.cachedata = None
> + self.cacheversion = version
> +
> + def init_cache(self, d, cache_file_name=None, defaultdata=None):
> + cachedir = (d.getVar("PERSISTENT_DIR") or
> + d.getVar("CACHE"))
> + if not cachedir:
> + return defaultdata
> +
> + bb.utils.mkdirhier(cachedir)
> + self.cachefile = os.path.join(cachedir,
> + cache_file_name or
> self.__class__.cache_file_name)
> + logger.debug(1, "Using cache in '%s'", self.cachefile)
> +
> + glf = bb.utils.lockfile(self.cachefile + ".lock")
> +
> + try:
> + with open(self.cachefile, "rb") as f:
> + p = pickle.Unpickler(f)
> + data, version = p.load()
> + except:
> + bb.utils.unlockfile(glf)
> + return defaultdata
> +
> + bb.utils.unlockfile(glf)
> +
> + if version != self.cacheversion:
> + return defaultdata
> +
> + return data
> +
> + def save(self, data):
> + if not self.cachefile:
> + return
> +
> + glf = bb.utils.lockfile(self.cachefile + ".lock")
> +
> + with open(self.cachefile, "wb") as f:
> + p = pickle.Pickler(f, -1)
> + p.dump([data, self.cacheversion])
> +
> + bb.utils.unlockfile(glf)
> diff --git a/bitbake/lib/bb/cache_extra.py
> b/bitbake/lib/bb/cache_extra.py index 83f4959..bf4226d 100644
> --- a/bitbake/lib/bb/cache_extra.py
> +++ b/bitbake/lib/bb/cache_extra.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # Extra RecipeInfo will be all defined in this file. Currently,
> # Only Hob (Image Creator) Requests some extra fields. So
> @@ -12,18 +10,8 @@
>
> # Copyright (C) 2011, Intel Corporation. All rights reserved.
>
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from bb.cache import RecipeInfoCommon
>
> diff --git a/bitbake/lib/bb/checksum.py b/bitbake/lib/bb/checksum.py
> index 4e1598f..5bc8a8f 100644
> --- a/bitbake/lib/bb/checksum.py
> +++ b/bitbake/lib/bb/checksum.py
> @@ -2,18 +2,8 @@
> #
> # Copyright (C) 2012 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import glob
> import operator
> diff --git a/bitbake/lib/bb/codeparser.py
> b/bitbake/lib/bb/codeparser.py index ddd1b97..fd2c473 100644
> --- a/bitbake/lib/bb/codeparser.py
> +++ b/bitbake/lib/bb/codeparser.py
> @@ -1,3 +1,7 @@
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> """
> BitBake code parser
>
> @@ -33,7 +37,7 @@ from bb.cache import MultiProcessCache
> logger = logging.getLogger('BitBake.CodeParser')
>
> def bbhash(s):
> - return hashlib.md5(s.encode("utf-8")).hexdigest()
> + return hashlib.sha256(s.encode("utf-8")).hexdigest()
>
> def check_indent(codestr):
> """If the code is indented, add a top level piece of code to
> 'remove' the indentation""" @@ -140,7 +144,7 @@ class
> CodeParserCache(MultiProcessCache): # so that an existing cache gets
> invalidated. Additionally you'll need # to increment
> __cache_version__ in cache.py in order to ensure that old # recipe
> caches don't trigger "Taskhash mismatch" errors.
> - CACHE_VERSION = 10
> + CACHE_VERSION = 11
>
> def __init__(self):
> MultiProcessCache.__init__(self)
> @@ -368,8 +372,9 @@ class ShellParser():
> def _parse_shell(self, value):
> try:
> tokens, _ = pyshyacc.parse(value, eof=True, debug=False)
> - except pyshlex.NeedMore:
> - raise sherrors.ShellSyntaxError("Unexpected EOF")
> + except Exception:
> + bb.error('Error during parse shell code, the last 5
> lines are:\n%s' % '\n'.join(value.split('\n')[-5:]))
> + raise
>
> self.process_tokens(tokens)
>
> diff --git a/bitbake/lib/bb/command.py b/bitbake/lib/bb/command.py
> index 6c966e3..378f389 100644
> --- a/bitbake/lib/bb/command.py
> +++ b/bitbake/lib/bb/command.py
> @@ -6,18 +6,8 @@ Provide an interface to interact with the bitbake
> server through 'commands'
> # Copyright (C) 2006-2007 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> """
> The bitbake server takes 'commands' from its UI/commandline.
> diff --git a/bitbake/lib/bb/compat.py b/bitbake/lib/bb/compat.py
> index de1923d..4935668 100644
> --- a/bitbake/lib/bb/compat.py
> +++ b/bitbake/lib/bb/compat.py
> @@ -1,3 +1,7 @@
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> """Code pulled from future python versions, here for compatibility"""
>
> from collections import MutableMapping, KeysView, ValuesView,
> ItemsView, OrderedDict diff --git a/bitbake/lib/bb/cooker.py
> b/bitbake/lib/bb/cooker.py index 16681ba..20ef04d 100644
> --- a/bitbake/lib/bb/cooker.py
> +++ b/bitbake/lib/bb/cooker.py
> @@ -1,6 +1,3 @@
> -#!/usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # Copyright (C) 2003, 2004 Chris Larson
> # Copyright (C) 2003, 2004 Phil Blundell
> @@ -9,19 +6,8 @@
> # Copyright (C) 2005 ROAD GmbH
> # Copyright (C) 2006 - 2007 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. -
>
> import sys, os, glob, os.path, re, time
> import atexit
> @@ -45,6 +31,7 @@ import pyinotify
> import json
> import pickle
> import codecs
> +import hashserv
>
> logger = logging.getLogger("BitBake")
> collectlog = logging.getLogger("BitBake.Collection")
> @@ -175,27 +162,45 @@ class BBCooker:
>
> self.configuration = configuration
>
> + bb.debug(1, "BBCooker starting %s" % time.time())
> + sys.stdout.flush()
> +
> self.configwatcher = pyinotify.WatchManager()
> + bb.debug(1, "BBCooker pyinotify1 %s" % time.time())
> + sys.stdout.flush()
> +
> self.configwatcher.bbseen = []
> self.configwatcher.bbwatchedfiles = []
> self.confignotifier = pyinotify.Notifier(self.configwatcher,
> self.config_notifications)
> + bb.debug(1, "BBCooker pyinotify2 %s" % time.time())
> + sys.stdout.flush()
> self.watchmask = pyinotify.IN_CLOSE_WRITE |
> pyinotify.IN_CREATE | pyinotify.IN_DELETE | \
> pyinotify.IN_DELETE_SELF | pyinotify.IN_MODIFY |
> pyinotify.IN_MOVE_SELF | \ pyinotify.IN_MOVED_FROM |
> pyinotify.IN_MOVED_TO self.watcher = pyinotify.WatchManager()
> + bb.debug(1, "BBCooker pyinotify3 %s" % time.time())
> + sys.stdout.flush()
> self.watcher.bbseen = []
> self.watcher.bbwatchedfiles = []
> self.notifier = pyinotify.Notifier(self.watcher,
> self.notifications)
> + bb.debug(1, "BBCooker pyinotify complete %s" % time.time())
> + sys.stdout.flush()
> +
> # If being called by something like tinfoil, we need to
> clean cached data # which may now be invalid
> bb.parse.clear_cache()
> bb.parse.BBHandler.cached_statements = {}
>
> self.ui_cmdline = None
> + self.hashserv = None
> + self.hashservaddr = None
>
> self.initConfigurationData()
>
> + bb.debug(1, "BBCooker parsed base configuration %s" %
> time.time())
> + sys.stdout.flush()
> +
> # we log all events to a file if so directed
> if self.configuration.writeeventlog:
> # register the log file writer as UI Handler
> @@ -233,6 +238,9 @@ class BBCooker:
> # Let SIGHUP exit as SIGTERM
> signal.signal(signal.SIGHUP, self.sigterm_exception)
>
> + bb.debug(1, "BBCooker startup complete %s" % time.time())
> + sys.stdout.flush()
> +
> def process_inotify_updates(self):
> for n in [self.confignotifier, self.notifier]:
> if n.check_events(timeout=0):
> @@ -367,13 +375,12 @@ class BBCooker:
> # Copy of the data store which has been expanded.
> # Used for firing events and accessing variables where
> expansion needs to be accounted for #
> - bb.parse.init_parser(self.data)
> -
> if CookerFeatures.BASEDATASTORE_TRACKING in self.featureset:
> self.disableDataTracking()
>
> - self.data.renameVar("__depends", "__base_depends")
> - self.add_filewatch(self.data.getVar("__base_depends",
> False), self.configwatcher)
> + for mc in self.databuilder.mcdata.values():
> + mc.renameVar("__depends", "__base_depends")
> + self.add_filewatch(mc.getVar("__base_depends", False),
> self.configwatcher)
> self.baseconfig_valid = True
> self.parsecache_valid = False
> @@ -385,6 +392,22 @@ class BBCooker:
> except prserv.serv.PRServiceConfigError as e:
> bb.fatal("Unable to start PR Server, exitting")
>
> + if self.data.getVar("BB_HASHSERVE") == "auto":
> + # Create a new hash server bound to a unix domain socket
> + if not self.hashserv:
> + dbfile = (self.data.getVar("PERSISTENT_DIR") or
> self.data.getVar("CACHE")) + "/hashserv.db"
> + self.hashservaddr = "unix://%s/hashserve.sock" %
> self.data.getVar("TOPDIR")
> + self.hashserv =
> hashserv.create_server(self.hashservaddr, dbfile, sync=False)
> + self.hashserv.process =
> multiprocessing.Process(target=self.hashserv.serve_forever)
> + self.hashserv.process.start()
> + self.data.setVar("BB_HASHSERVE", self.hashservaddr)
> + self.databuilder.origdata.setVar("BB_HASHSERVE",
> self.hashservaddr)
> + self.databuilder.data.setVar("BB_HASHSERVE",
> self.hashservaddr)
> + for mc in self.databuilder.mcdata:
> + self.databuilder.mcdata[mc].setVar("BB_HASHSERVE",
> self.hashservaddr) +
> + bb.parse.init_parser(self.data)
> +
> def enableDataTracking(self):
> self.configuration.tracking = True
> if hasattr(self, "data"):
> @@ -488,6 +511,7 @@ class BBCooker:
> """
> fn = None
> envdata = None
> + mc = ''
> if not pkgs_to_build:
> pkgs_to_build = []
>
> @@ -496,6 +520,12 @@ class BBCooker:
> self.enableDataTracking()
> self.reset()
>
> + def mc_base(p):
> + if p.startswith('mc:'):
> + s = p.split(':')
> + if len(s) == 2:
> + return s[1]
> + return None
>
> if buildfile:
> # Parse the configuration here. We need to do it
> explicitly here since @@ -506,18 +536,16 @@ class BBCooker:
> fn = self.matchFile(fn)
> fn = bb.cache.realfn2virtual(fn, cls, mc)
> elif len(pkgs_to_build) == 1:
> - ignore = self.data.getVar("ASSUME_PROVIDED") or ""
> - if pkgs_to_build[0] in set(ignore.split()):
> - bb.fatal("%s is in ASSUME_PROVIDED" %
> pkgs_to_build[0])
> + mc = mc_base(pkgs_to_build[0])
> + if not mc:
> + ignore = self.data.getVar("ASSUME_PROVIDED") or ""
> + if pkgs_to_build[0] in set(ignore.split()):
> + bb.fatal("%s is in ASSUME_PROVIDED" %
> pkgs_to_build[0])
> - taskdata, runlist = self.buildTaskData(pkgs_to_build,
> None, self.configuration.abort, allowincomplete=True)
> + taskdata, runlist =
> self.buildTaskData(pkgs_to_build, None, self.configuration.abort,
> allowincomplete=True)
> - mc = runlist[0][0]
> - fn = runlist[0][3]
> - else:
> - envdata = self.data
> - data.expandKeys(envdata)
> - parse.ast.runAnonFuncs(envdata)
> + mc = runlist[0][0]
> + fn = runlist[0][3]
>
> if fn:
> try:
> @@ -526,6 +554,12 @@ class BBCooker:
> except Exception as e:
> parselog.exception("Unable to read %s", fn)
> raise
> + else:
> + if not mc in self.databuilder.mcdata:
> + bb.fatal('Not multiconfig named "%s" found' % mc)
> + envdata = self.databuilder.mcdata[mc]
> + data.expandKeys(envdata)
> + parse.ast.runAnonFuncs(envdata)
>
> # Display history
> with closing(StringIO()) as env:
> @@ -565,10 +599,10 @@ class BBCooker:
> wildcard = False
>
> # Wild card expansion:
> - # Replace string such as "multiconfig:*:bash"
> - # into "multiconfig:A:bash multiconfig:B:bash bash"
> + # Replace string such as "mc:*:bash"
> + # into "mc:A:bash mc:B:bash bash"
> for k in targetlist:
> - if k.startswith("multiconfig:"):
> + if k.startswith("mc:"):
> if wildcard:
> bb.fatal('multiconfig conflict')
> if k.split(":")[1] == "*":
> @@ -601,7 +635,7 @@ class BBCooker:
> runlist = []
> for k in fulltargetlist:
> mc = ""
> - if k.startswith("multiconfig:"):
> + if k.startswith("mc:"):
> mc = k.split(":")[1]
> k = ":".join(k.split(":")[2:])
> ktask = task
> @@ -620,13 +654,22 @@ class BBCooker:
> runlist.append([mc, k, ktask, fn])
> bb.event.fire(bb.event.TreeDataPreparationProgress(current,
> len(fulltargetlist)), self.data)
> - mcdeps = taskdata[mc].get_mcdepends()
> + havemc = False
> + for mc in self.multiconfigs:
> + if taskdata[mc].get_mcdepends():
> + havemc = True
> +
> # No need to do check providers if there are no mcdeps or
> not an mc build
> - if mcdeps and mc:
> - # Make sure we can provide the multiconfig dependency
> + if havemc or len(self.multiconfigs) > 1:
> seen = set()
> new = True
> + # Make sure we can provide the multiconfig dependency
> while new:
> + mcdeps = set()
> + # Add unresolved first, so we can get multiconfig
> indirect dependencies on time
> + for mc in self.multiconfigs:
> + taskdata[mc].add_unresolved(localdata[mc],
> self.recipecaches[mc])
> + mcdeps |= set(taskdata[mc].get_mcdepends())
> new = False
> for mc in self.multiconfigs:
> for k in mcdeps:
> @@ -641,6 +684,7 @@ class BBCooker:
> taskdata[depmc].add_provider(localdata[depmc],
> self.recipecaches[depmc], l[3]) seen.add(k)
> new = True
> +
> for mc in self.multiconfigs:
> taskdata[mc].add_unresolved(localdata[mc],
> self.recipecaches[mc])
> @@ -676,7 +720,7 @@ class BBCooker:
> @staticmethod
> def add_mc_prefix(mc, pn):
> if mc:
> - return "multiconfig:%s:%s" % (mc, pn)
> + return "mc:%s:%s" % (mc, pn)
> return pn
>
> def buildDependTree(self, rq, taskdata):
> @@ -875,6 +919,10 @@ class BBCooker:
> os.unlink('package-depends.dot')
> except FileNotFoundError:
> pass
> + try:
> + os.unlink('recipe-depends.dot')
> + except FileNotFoundError:
> + pass
>
> with open('task-depends.dot', 'w') as f:
> f.write("digraph depends {\n")
> @@ -888,27 +936,6 @@ class BBCooker:
> f.write("}\n")
> logger.info("Task dependencies saved to 'task-depends.dot'")
>
> - with open('recipe-depends.dot', 'w') as f:
> - f.write("digraph depends {\n")
> - pndeps = {}
> - for task in sorted(depgraph["tdepends"]):
> - (pn, taskname) = task.rsplit(".", 1)
> - if pn not in pndeps:
> - pndeps[pn] = set()
> - for dep in sorted(depgraph["tdepends"][task]):
> - (deppn, deptaskname) = dep.rsplit(".", 1)
> - pndeps[pn].add(deppn)
> - for pn in sorted(pndeps):
> - fn = depgraph["pn"][pn]["filename"]
> - version = depgraph["pn"][pn]["version"]
> - f.write('"%s" [label="%s\\n%s\\n%s"]\n' % (pn, pn,
> version, fn))
> - for dep in sorted(pndeps[pn]):
> - if dep == pn:
> - continue
> - f.write('"%s" -> "%s"\n' % (pn, dep))
> - f.write("}\n")
> - logger.info("Flattened recipe dependencies saved to
> 'recipe-depends.dot'") -
> def show_appends_with_no_recipes(self):
> # Determine which bbappends haven't been applied
>
> @@ -1191,8 +1218,8 @@ class BBCooker:
> continue
> elif regex == "":
> parselog.debug(1, "BBFILE_PATTERN_%s is empty" %
> c)
> + cre = re.compile('^NULL$')
> errors = False
> - continue
> else:
> try:
> cre = re.compile(regex)
> @@ -1453,7 +1480,7 @@ class BBCooker:
> ntargets = []
> for target in runlist:
> if target[0]:
> - ntargets.append("multiconfig:%s:%s:%s" % (target[0],
> target[1], target[2]))
> + ntargets.append("mc:%s:%s:%s" % (target[0],
> target[1], target[2])) ntargets.append("%s:%s" % (target[1],
> target[2]))
> for mc in self.multiconfigs:
> @@ -1576,6 +1603,9 @@ class BBCooker:
> for pkg in pkgs_to_build:
> if pkg in ignore:
> parselog.warning("Explicit target \"%s\" is in
> ASSUME_PROVIDED, ignoring" % pkg)
> + if pkg.startswith("multiconfig:"):
> + pkgs_to_build.remove(pkg)
> + pkgs_to_build.append(pkg.replace("multiconfig:",
> "mc:"))
> if 'world' in pkgs_to_build:
> pkgs_to_build.remove('world')
> @@ -1583,7 +1613,7 @@ class BBCooker:
> bb.providers.buildWorldTargetList(self.recipecaches[mc],
> task) for t in self.recipecaches[mc].world_target:
> if mc:
> - t = "multiconfig:" + mc + ":" + t
> + t = "mc:" + mc + ":" + t
> pkgs_to_build.append(t)
>
> if 'universe' in pkgs_to_build:
> @@ -1602,7 +1632,7 @@ class BBCooker:
> bb.debug(1, "Skipping %s for universe
> tasks as task %s doesn't exist" % (t, task)) continue
> if mc:
> - t = "multiconfig:" + mc + ":" + t
> + t = "mc:" + mc + ":" + t
> pkgs_to_build.append(t)
>
> return pkgs_to_build
> @@ -1615,9 +1645,11 @@ class BBCooker:
>
> def post_serve(self):
> prserv.serv.auto_shutdown()
> + if self.hashserv:
> + self.hashserv.process.terminate()
> + self.hashserv.process.join()
> bb.event.fire(CookerExit(), self.data)
>
> -
> def shutdown(self, force = False):
> if force:
> self.state = state.forceshutdown
> @@ -1632,6 +1664,7 @@ class BBCooker:
>
> def reset(self):
> self.initConfigurationData()
> + self.handlePRServ()
>
> def clientComplete(self):
> """Called when the client is done using the server"""
> @@ -1865,35 +1898,6 @@ class ParsingFailure(Exception):
> self.recipe = recipe
> Exception.__init__(self, realexception, recipe)
>
> -class Feeder(multiprocessing.Process):
> - def __init__(self, jobs, to_parsers, quit):
> - self.quit = quit
> - self.jobs = jobs
> - self.to_parsers = to_parsers
> - multiprocessing.Process.__init__(self)
> -
> - def run(self):
> - while True:
> - try:
> - quit = self.quit.get_nowait()
> - except queue.Empty:
> - pass
> - else:
> - if quit == 'cancel':
> - self.to_parsers.cancel_join_thread()
> - break
> -
> - try:
> - job = self.jobs.pop()
> - except IndexError:
> - break
> -
> - try:
> - self.to_parsers.put(job, timeout=0.5)
> - except queue.Full:
> - self.jobs.insert(0, job)
> - continue
> -
> class Parser(multiprocessing.Process):
> def __init__(self, jobs, results, quit, init, profile):
> self.jobs = jobs
> @@ -1940,11 +1944,8 @@ class Parser(multiprocessing.Process):
> result = pending.pop()
> else:
> try:
> - job = self.jobs.get(timeout=0.25)
> - except queue.Empty:
> - continue
> -
> - if job is None:
> + job = self.jobs.pop()
> + except IndexError:
> break
> result = self.parse(*job)
>
> @@ -2028,14 +2029,15 @@ class CookerParser(object):
> multiprocessing.util.Finalize(None,
> bb.codeparser.parser_cache_save, exitpriority=1)
> multiprocessing.util.Finalize(None, bb.fetch.fetcher_parse_save,
> exitpriority=1)
> - self.feeder_quit = multiprocessing.Queue(maxsize=1)
> self.parser_quit =
> multiprocessing.Queue(maxsize=self.num_processes)
> - self.jobs =
> multiprocessing.Queue(maxsize=self.num_processes) self.result_queue =
> multiprocessing.Queue()
> - self.feeder = Feeder(self.willparse, self.jobs,
> self.feeder_quit)
> - self.feeder.start()
> +
> + def chunkify(lst,n):
> + return [lst[i::n] for i in range(n)]
> + self.jobs = chunkify(self.willparse, self.num_processes)
> +
> for i in range(0, self.num_processes):
> - parser = Parser(self.jobs, self.result_queue,
> self.parser_quit, init, self.cooker.configuration.profile)
> + parser = Parser(self.jobs[i], self.result_queue,
> self.parser_quit, init, self.cooker.configuration.profile)
> parser.start() self.process_names.append(parser.name)
> self.processes.append(parser)
> @@ -2056,17 +2058,20 @@ class CookerParser(object):
> self.total)
>
> bb.event.fire(event, self.cfgdata)
> - self.feeder_quit.put(None)
> for process in self.processes:
> self.parser_quit.put(None)
> else:
> - self.feeder_quit.put('cancel')
> -
> self.parser_quit.cancel_join_thread()
> for process in self.processes:
> self.parser_quit.put(None)
>
> - self.jobs.cancel_join_thread()
> + # Cleanup the queue before call process.join(), otherwise
> there might be
> + # deadlocks.
> + while True:
> + try:
> + self.result_queue.get(timeout=0.25)
> + except queue.Empty:
> + break
>
> for process in self.processes:
> if force:
> @@ -2074,7 +2079,6 @@ class CookerParser(object):
> process.terminate()
> else:
> process.join()
> - self.feeder.join()
>
> sync = threading.Thread(target=self.bb_cache.sync)
> sync.start()
> diff --git a/bitbake/lib/bb/cookerdata.py
> b/bitbake/lib/bb/cookerdata.py index 5df66e6..472423f 100644
> --- a/bitbake/lib/bb/cookerdata.py
> +++ b/bitbake/lib/bb/cookerdata.py
> @@ -1,6 +1,3 @@
> -#!/usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # Copyright (C) 2003, 2004 Chris Larson
> # Copyright (C) 2003, 2004 Phil Blundell
> @@ -9,23 +6,14 @@
> # Copyright (C) 2005 ROAD GmbH
> # Copyright (C) 2006 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import logging
> import os
> import re
> import sys
> +import hashlib
> from functools import wraps
> import bb
> from bb import data
> @@ -134,6 +122,7 @@ class CookerConfiguration(object):
> self.profile = False
> self.nosetscene = False
> self.setsceneonly = False
> + self.skipsetscene = False
> self.invalidate_stamp = False
> self.dump_signatures = []
> self.dry_run = False
> @@ -279,12 +268,13 @@ class CookerDataBuilder(object):
> self.mcdata = {}
>
> def parseBaseConfiguration(self):
> + data_hash = hashlib.sha256()
> try:
> - bb.parse.init_parser(self.basedata)
> self.data = self.parseConfigurationFiles(self.prefiles,
> self.postfiles)
> if self.data.getVar("BB_WORKERCONTEXT", False) is None:
> bb.fetch.fetcher_init(self.data)
> + bb.parse.init_parser(self.data)
> bb.codeparser.parser_cache_init(self.data)
>
> bb.event.fire(bb.event.ConfigParsed(), self.data)
> @@ -302,7 +292,7 @@ class CookerDataBuilder(object):
> bb.event.fire(bb.event.ConfigParsed(), self.data)
>
> bb.parse.init_parser(self.data)
> - self.data_hash = self.data.get_hash()
> + data_hash.update(self.data.get_hash().encode('utf-8'))
> self.mcdata[''] = self.data
>
> multiconfig = (self.data.getVar("BBMULTICONFIG") or
> "").split() @@ -310,9 +300,11 @@ class CookerDataBuilder(object):
> mcdata = self.parseConfigurationFiles(self.prefiles,
> self.postfiles, config) bb.event.fire(bb.event.ConfigParsed(), mcdata)
> self.mcdata[config] = mcdata
> + data_hash.update(mcdata.get_hash().encode('utf-8'))
> if multiconfig:
> bb.event.fire(bb.event.MultiConfigParsed(self.mcdata),
> self.data)
> + self.data_hash = data_hash.hexdigest()
> except (SyntaxError, bb.BBHandledException):
> raise bb.BBHandledException
> except bb.data_smart.ExpansionError as e:
> @@ -354,14 +346,24 @@ class CookerDataBuilder(object):
> data = parse_config_file(layerconf, data)
>
> layers = (data.getVar('BBLAYERS') or "").split()
> + broken_layers = []
>
> data = bb.data.createCopy(data)
> approved = bb.utils.approved_variables()
> +
> + # Check whether present layer directories exist
> for layer in layers:
> if not os.path.isdir(layer):
> - parselog.critical("Layer directory '%s' does not
> exist! "
> - "Please check BBLAYERS in %s"
> % (layer, layerconf))
> - sys.exit(1)
> + broken_layers.append(layer)
> +
> + if broken_layers:
> + parselog.critical("The following layer directories
> do not exist:")
> + for layer in broken_layers:
> + parselog.critical(" %s", layer)
> + parselog.critical("Please check BBLAYERS in %s" %
> (layerconf))
> + sys.exit(1)
> +
> + for layer in layers:
> parselog.debug(2, "Adding layer %s", layer)
> if 'HOME' in approved and '~' in layer:
> layer = os.path.expanduser(layer)
> @@ -391,7 +393,11 @@ class CookerDataBuilder(object):
> bb.fatal("BBFILES_DYNAMIC entries must be of the
> form <collection name>:<filename pattern>, not:\n %s" % "\n
> ".join(invalid)) layerseries =
> set((data.getVar("LAYERSERIES_CORENAMES") or "").split())
> + collections_tmp = collections[:]
> for c in collections:
> + collections_tmp.remove(c)
> + if c in collections_tmp:
> + bb.fatal("Found duplicated BBFILE_COLLECTIONS
> '%s', check bblayers.conf or layer.conf to fix it." % c) compat =
> set((data.getVar("LAYERSERIES_COMPAT_%s" % c) or "").split()) if
> compat and not (compat & layerseries): bb.fatal("Layer %s is not
> compatible with the core layer which only supports these series: %s
> (layer is compatible with %s)" diff --git
> a/bitbake/lib/bb/daemonize.py b/bitbake/lib/bb/daemonize.py index
> c937675..f01e6ec 100644 --- a/bitbake/lib/bb/daemonize.py +++
> b/bitbake/lib/bb/daemonize.py @@ -1,3 +1,7 @@
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> """
> Python Daemonizing helper
>
> diff --git a/bitbake/lib/bb/data.py b/bitbake/lib/bb/data.py
> index d66d98c..0d75d0c 100644
> --- a/bitbake/lib/bb/data.py
> +++ b/bitbake/lib/bb/data.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake 'Data' implementations
>
> @@ -22,18 +20,7 @@ the speed is more critical here.
> # Copyright (C) 2003, 2004 Chris Larson
> # Copyright (C) 2005 Holger Hans Peter Freyther
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
> # Based on functions from the base bb module, Copyright 2003 Holger
> Schurig
> @@ -143,7 +130,7 @@ def emit_var(var, o=sys.__stdout__, d = init(),
> all=False): if all:
> oval = d.getVar(var, False)
> val = d.getVar(var)
> - except (KeyboardInterrupt, bb.build.FuncFailed):
> + except (KeyboardInterrupt):
> raise
> except Exception as exc:
> o.write('# expansion of %s threw %s: %s\n' % (var,
> exc.__class__.__name__, str(exc))) @@ -322,8 +309,6 @@ def
> build_dependencies(key, keys, shelldeps, varflagsexcl, d): if
> varflags.get("python"): value = d.getVarFlag(key, "_content", False)
> parser = bb.codeparser.PythonParser(key, logger)
> - if value and "\t" in value:
> - logger.warning("Variable %s contains tabs,
> please remove these (%s)" % (key, d.getVar("FILE")))
> parser.parse_python(value, filename=varflags.get("filename"),
> lineno=varflags.get("lineno")) deps = deps | parser.references deps =
> deps | (keys & parser.execs) @@ -437,8 +422,8 @@ def
> generate_dependency_hash(tasklist, gendeps, lookupcache, whitelist,
> fn): var = lookupcache[dep] if var is not None:
> data = data + str(var)
> - k = fn + "." + task
> - basehash[k] = hashlib.md5(data.encode("utf-8")).hexdigest()
> + k = fn + ":" + task
> + basehash[k] =
> hashlib.sha256(data.encode("utf-8")).hexdigest() taskdeps[task] =
> alldeps
> return taskdeps, basehash
> diff --git a/bitbake/lib/bb/data_smart.py
> b/bitbake/lib/bb/data_smart.py index 67af380..dd5c618 100644
> --- a/bitbake/lib/bb/data_smart.py
> +++ b/bitbake/lib/bb/data_smart.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake Smart Dictionary Implementation
>
> @@ -14,18 +12,8 @@ BitBake build tools.
> # Copyright (C) 2005 Uli Luckas
> # Copyright (C) 2005 ROAD GmbH
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. # Based on functions from the base bb module,
> Copyright 2003 Holger Schurig
> import copy, re, sys, traceback
> @@ -39,10 +27,11 @@ from bb.COW import COWDictBase
> logger = logging.getLogger("BitBake.Data")
>
> __setvar_keyword__ = ["_append", "_prepend", "_remove"]
> -__setvar_regexp__ =
> re.compile('(?P<base>.*?)(?P<keyword>_append|_prepend|_remove)(_(?P<add>[^A-Z]*))?$')
> -__expand_var_regexp__ = re.compile(r"\${[^{}@\n\t :]+}")
> +__setvar_regexp__ =
> re.compile(r'(?P<base>.*?)(?P<keyword>_append|_prepend|_remove)(_(?P<add>[^A-Z]*))?$')
> +__expand_var_regexp__ = re.compile(r"\${[a-zA-Z0-9\-_+./~]+?}")
> __expand_python_regexp__ = re.compile(r"\${@.+?}")
> -__whitespace_split__ = re.compile('(\s)') +__whitespace_split__ =
> re.compile(r'(\s)') +__override_regexp__ = re.compile(r'[a-z0-9]+')
> def infer_caller_details(loginfo, parent = False, varval = True):
> """Save the caller the trouble of specifying everything."""
> @@ -597,7 +586,7 @@ class DataSmart(MutableMapping):
> # aka pay the cookie monster
> override = var[var.rfind('_')+1:]
> shortvar = var[:var.rfind('_')]
> - while override and override.islower():
> + while override and __override_regexp__.match(override):
> if shortvar not in self.overridedata:
> self.overridedata[shortvar] = []
> if [var, override] not in self.overridedata[shortvar]:
> @@ -1073,4 +1062,4 @@ class DataSmart(MutableMapping):
> data.update({i:value})
>
> data_str = str([(k, data[k]) for k in sorted(data.keys())])
> - return hashlib.md5(data_str.encode("utf-8")).hexdigest()
> + return hashlib.sha256(data_str.encode("utf-8")).hexdigest()
> diff --git a/bitbake/lib/bb/event.py b/bitbake/lib/bb/event.py
> index 5b1b094..d44621e 100644
> --- a/bitbake/lib/bb/event.py
> +++ b/bitbake/lib/bb/event.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake 'Event' implementation
>
> @@ -9,18 +7,8 @@ BitBake build tools.
>
> # Copyright (C) 2003, 2004 Chris Larson
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import os, sys
> import warnings
> @@ -136,6 +124,7 @@ def fire_class_handlers(event, d):
> ui_queue = []
> @atexit.register
> def print_ui_queue():
> + global ui_queue
> """If we're exiting before a UI has been spawned, display any
> queued LogRecords to the console."""
> logger = logging.getLogger("BitBake")
> @@ -180,6 +169,7 @@ def print_ui_queue():
> logger.removeHandler(stderr)
> else:
> logger.removeHandler(stdout)
> + ui_queue = []
>
> def fire_ui_handlers(event, d):
> global _thread_lock
> @@ -414,23 +404,6 @@ class RecipeTaskPreProcess(RecipeEvent):
> class RecipeParsed(RecipeEvent):
> """ Recipe Parsing Complete """
>
> -class StampUpdate(Event):
> - """Trigger for any adjustment of the stamp files to happen"""
> -
> - def __init__(self, targets, stampfns):
> - self._targets = targets
> - self._stampfns = stampfns
> - Event.__init__(self)
> -
> - def getStampPrefix(self):
> - return self._stampfns
> -
> - def getTargets(self):
> - return self._targets
> -
> - stampPrefix = property(getStampPrefix)
> - targets = property(getTargets)
> -
> class BuildBase(Event):
> """Base class for bitbake build events"""
>
> diff --git a/bitbake/lib/bb/exceptions.py
> b/bitbake/lib/bb/exceptions.py index cd71343..ecbad59 100644
> --- a/bitbake/lib/bb/exceptions.py
> +++ b/bitbake/lib/bb/exceptions.py
> @@ -1,3 +1,6 @@
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
>
> import inspect
> import traceback
> diff --git a/bitbake/lib/bb/fetch2/__init__.py
> b/bitbake/lib/bb/fetch2/__init__.py index 572b71a..1f5f8f1 100644
> --- a/bitbake/lib/bb/fetch2/__init__.py
> +++ b/bitbake/lib/bb/fetch2/__init__.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake 'Fetch' implementations
>
> @@ -10,18 +8,7 @@ BitBake build tools.
> # Copyright (C) 2003, 2004 Chris Larson
> # Copyright (C) 2012 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
> # Based on functions from the base bb module, Copyright 2003 Holger
> Schurig
> @@ -256,7 +243,7 @@ class URI(object):
>
> # Identify if the URI is relative or not
> if urlp.scheme in self._relative_schemes and \
> - re.compile("^\w+:(?!//)").match(uri):
> + re.compile(r"^\w+:(?!//)").match(uri):
> self.relative = True
>
> if not self.relative:
> @@ -524,7 +511,7 @@ def fetcher_parse_save():
> def fetcher_parse_done():
> _checksum_cache.save_merge()
>
> -def fetcher_compare_revisions():
> +def fetcher_compare_revisions(d):
> """
> Compare the revisions in the persistant cache with current
> values and return true/false on whether they've changed.
> @@ -777,7 +764,8 @@ def get_srcrev(d,
> method_name='sortable_revision'): #
> format = d.getVar('SRCREV_FORMAT')
> if not format:
> - raise FetchError("The SRCREV_FORMAT variable must be set
> when multiple SCMs are used.")
> + raise FetchError("The SRCREV_FORMAT variable must be set
> when multiple SCMs are used.\n"\
> + "The SCMs are:\n%s" % '\n'.join(scms))
>
> name_to_rev = {}
> seenautoinc = False
> @@ -855,10 +843,18 @@ def runfetchcmd(cmd, d, quiet=False,
> cleanup=None, log=None, workdir=None): if val:
> cmd = 'export ' + var + '=\"%s\"; %s' % (val, cmd)
>
> + # Ensure that a _PYTHON_SYSCONFIGDATA_NAME value set by a recipe
> + # (for example via python3native.bbclass since warrior) is not
> set for
> + # host Python (otherwise tools like git-make-shallow will fail)
> + cmd = 'unset _PYTHON_SYSCONFIGDATA_NAME; ' + cmd
> +
> # Disable pseudo as it may affect ssh, potentially causing it to
> hang. cmd = 'export PSEUDO_DISABLED=1; ' + cmd
>
> - logger.debug(1, "Running %s", cmd)
> + if workdir:
> + logger.debug(1, "Running '%s' in %s" % (cmd, workdir))
> + else:
> + logger.debug(1, "Running %s", cmd)
>
> success = False
> error_message = ""
> @@ -894,7 +890,7 @@ def check_network_access(d, info, url):
> log remote network access, and error if BB_NO_NETWORK is set or
> the given URI is untrusted
> """
> - if d.getVar("BB_NO_NETWORK") == "1":
> + if bb.utils.to_boolean(d.getVar("BB_NO_NETWORK")):
> raise NetworkAccess(url, info)
> elif not trusted_network(d, url):
> raise UntrustedUrl(url, info)
> @@ -966,7 +962,8 @@ def rename_bad_checksum(ud, suffix):
>
> new_localpath = "%s_bad-checksum_%s" % (ud.localpath, suffix)
> bb.warn("Renaming %s to %s" % (ud.localpath, new_localpath))
> - bb.utils.movefile(ud.localpath, new_localpath)
> + if not bb.utils.movefile(ud.localpath, new_localpath):
> + bb.warn("Renaming %s to %s failed, grep movefile in
> log.do_fetch to see why" % (ud.localpath, new_localpath))
>
> def try_mirror_url(fetch, origud, ud, ld, check = False):
> @@ -1027,7 +1024,7 @@ def try_mirror_url(fetch, origud, ud, ld, check
> = False): raise
>
> except IOError as e:
> - if e.errno in [os.errno.ESTALE]:
> + if e.errno in [errno.ESTALE]:
> logger.warning("Stale Error Observed %s." % ud.url)
> return False
> raise
> @@ -1094,7 +1091,7 @@ def trusted_network(d, url):
> BB_ALLOWED_NETWORKS is set globally or for a specific recipe.
> Note: modifies SRC_URI & mirrors.
> """
> - if d.getVar('BB_NO_NETWORK') == "1":
> + if bb.utils.to_boolean(d.getVar("BB_NO_NETWORK")):
> return True
>
> pkgname = d.expand(d.getVar('PN', False))
> @@ -1403,7 +1400,7 @@ class FetchMethod(object):
> Fetch urls
> Assumes localpath was called first
> """
> - raise NoMethodError(url)
> + raise NoMethodError(urldata.url)
>
> def unpack(self, urldata, rootdir, data):
> iterate = False
> @@ -1469,7 +1466,7 @@ class FetchMethod(object):
> else:
> cmd = 'rpm2cpio.sh %s | cpio -id' % (file)
> elif file.endswith('.deb') or file.endswith('.ipk'):
> - output = subprocess.check_output('ar -t %s' % file,
> preexec_fn=subprocess_setup, shell=True)
> + output = subprocess.check_output(['ar', '-t', file],
> preexec_fn=subprocess_setup) datafile = None
> if output:
> for line in output.decode().splitlines():
> @@ -1547,7 +1544,7 @@ class FetchMethod(object):
> Check the status of a URL
> Assumes localpath was called first
> """
> - logger.info("URL %s could not be checked for status since no
> method exists.", url)
> + logger.info("URL %s could not be checked for status since no
> method exists.", urldata.url) return True
>
> def latest_revision(self, ud, d, name):
> @@ -1555,7 +1552,7 @@ class FetchMethod(object):
> Look in the cache for the latest revision, if not present
> ask the SCM. """
> if not hasattr(self, "_latest_revision"):
> - raise ParameterError("The fetcher for this URL does not
> support _latest_revision", url)
> + raise ParameterError("The fetcher for this URL does not
> support _latest_revision", ud.url)
> revs = bb.persist_data.persist('BB_URI_HEADREVS', d)
> key = self.generate_revision_key(ud, d, name)
> @@ -1638,7 +1635,7 @@ class Fetch(object):
> urls = self.urls
>
> network = self.d.getVar("BB_NO_NETWORK")
> - premirroronly = (self.d.getVar("BB_FETCH_PREMIRRORONLY") ==
> "1")
> + premirroronly =
> bb.utils.to_boolean(self.d.getVar("BB_FETCH_PREMIRRORONLY"))
> for u in urls:
> ud = self.ud[u]
> @@ -1716,7 +1713,7 @@ class Fetch(object):
> update_stamp(ud, self.d)
>
> except IOError as e:
> - if e.errno in [os.errno.ESTALE]:
> + if e.errno in [errno.ESTALE]:
> logger.error("Stale Error Observed %s." % u)
> raise ChecksumError("Stale Error Detected")
>
> @@ -1786,7 +1783,7 @@ class Fetch(object):
>
> for url in urls:
> if url not in self.ud:
> - self.ud[url] = FetchData(url, d)
> + self.ud[url] = FetchData(url, self.d)
> ud = self.ud[url]
> ud.setup_localpath(self.d)
>
> diff --git a/bitbake/lib/bb/fetch2/bzr.py
> b/bitbake/lib/bb/fetch2/bzr.py index 658502f..c56d875 100644
> --- a/bitbake/lib/bb/fetch2/bzr.py
> +++ b/bitbake/lib/bb/fetch2/bzr.py
> @@ -10,18 +10,8 @@ BitBake 'Fetch' implementation for bzr.
> # BitBake build tools.
> # Copyright (C) 2003, 2004 Chris Larson
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import os
> import sys
> diff --git a/bitbake/lib/bb/fetch2/clearcase.py
> b/bitbake/lib/bb/fetch2/clearcase.py index 3a6573d..3dd93ad 100644
> --- a/bitbake/lib/bb/fetch2/clearcase.py
> +++ b/bitbake/lib/bb/fetch2/clearcase.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake 'Fetch' clearcase implementation
>
> @@ -47,18 +45,7 @@ User credentials:
> """
> # Copyright (C) 2014 Siemens AG
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
>
> import os
> @@ -67,6 +54,8 @@ import shutil
> import bb
> from bb.fetch2 import FetchMethod
> from bb.fetch2 import FetchError
> +from bb.fetch2 import MissingParameterError
> +from bb.fetch2 import ParameterError
> from bb.fetch2 import runfetchcmd
> from bb.fetch2 import logger
>
> @@ -92,7 +81,7 @@ class ClearCase(FetchMethod):
> if 'protocol' in ud.parm:
> ud.proto = ud.parm['protocol']
> if not ud.proto in ('http', 'https'):
> - raise fetch2.ParameterError("Invalid protocol type",
> ud.url)
> + raise ParameterError("Invalid protocol type", ud.url)
>
> ud.vob = ''
> if 'vob' in ud.parm:
> diff --git a/bitbake/lib/bb/fetch2/cvs.py
> b/bitbake/lib/bb/fetch2/cvs.py index 0e0a319..1b35ba4 100644
> --- a/bitbake/lib/bb/fetch2/cvs.py
> +++ b/bitbake/lib/bb/fetch2/cvs.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake 'Fetch' implementations
>
> @@ -10,20 +8,9 @@ BitBake build tools.
>
> # Copyright (C) 2003, 2004 Chris Larson
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. -#
> -#Based on functions from the base bb module, Copyright 2003 Holger
> Schurig +# Based on functions from the base bb module, Copyright 2003
> Holger Schurig #
>
> import os
> diff --git a/bitbake/lib/bb/fetch2/git.py
> b/bitbake/lib/bb/fetch2/git.py index 59a2ee8..2d1d2ca 100644
> --- a/bitbake/lib/bb/fetch2/git.py
> +++ b/bitbake/lib/bb/fetch2/git.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake 'Fetch' git implementation
>
> @@ -55,20 +53,10 @@ Supported SRC_URI options are:
>
> """
>
> -#Copyright (C) 2005 Richard Purdie
> +# Copyright (C) 2005 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import collections
> import errno
> @@ -199,7 +187,7 @@ class Git(FetchMethod):
> depth_default = 1
> ud.shallow_depths = collections.defaultdict(lambda:
> depth_default)
> - revs_default = d.getVar("BB_GIT_SHALLOW_REVS", True)
> + revs_default = d.getVar("BB_GIT_SHALLOW_REVS")
> ud.shallow_revs = []
> ud.branches = {}
> for pos, name in enumerate(ud.names):
> @@ -318,7 +306,7 @@ class Git(FetchMethod):
> def try_premirror(self, ud, d):
> # If we don't do this, updating an existing checkout with
> only premirrors # is not possible
> - if d.getVar("BB_FETCH_PREMIRRORONLY") is not None:
> + if bb.utils.to_boolean(d.getVar("BB_FETCH_PREMIRRORONLY")):
> return True
> if os.path.exists(ud.clonedir):
> return False
> @@ -476,6 +464,8 @@ class Git(FetchMethod):
> if os.path.exists(destdir):
> bb.utils.prunedir(destdir)
>
> + need_lfs = ud.parm.get("lfs", "1") == "1"
> +
> source_found = False
> source_error = []
>
> @@ -503,6 +493,13 @@ class Git(FetchMethod):
>
> repourl = self._get_repo_url(ud)
> runfetchcmd("%s remote set-url origin %s" % (ud.basecmd,
> repourl), d, workdir=destdir) +
> + if self._contains_lfs(ud, d, destdir):
> + if need_lfs and not self._find_git_lfs(d):
> + raise bb.fetch2.FetchError("Repository %s has LFS
> content, install git-lfs on host to download (or set lfs=0 to ignore
> it)" % (repourl))
> + else:
> + bb.note("Repository %s has LFS content but it is not
> being fetched" % (repourl)) +
> if not ud.nocheckout:
> if subdir != "":
> runfetchcmd("%s read-tree %s%s" % (ud.basecmd,
> ud.revisions[ud.names[0]], readpathspec), d, @@ -522,9 +519,17 @@
> class Git(FetchMethod): def clean(self, ud, d):
> """ clean the git directory """
>
> - bb.utils.remove(ud.localpath, True)
> - bb.utils.remove(ud.fullmirror)
> - bb.utils.remove(ud.fullmirror + ".done")
> + to_remove = [ud.localpath, ud.fullmirror, ud.fullmirror +
> ".done"]
> + # The localpath is a symlink to clonedir when it is cloned
> from a
> + # mirror, so remove both of them.
> + if os.path.islink(ud.localpath):
> + clonedir = os.path.realpath(ud.localpath)
> + to_remove.append(clonedir)
> +
> + for r in to_remove:
> + if os.path.exists(r):
> + bb.note('Removing %s' % r)
> + bb.utils.remove(r, True)
>
> def supports_srcrev(self):
> return True
> @@ -545,6 +550,27 @@ class Git(FetchMethod):
> raise bb.fetch2.FetchError("The command '%s' gave output
> with more then 1 line unexpectedly, output: '%s'" % (cmd, output))
> return output.split()[0] != "0"
> + def _contains_lfs(self, ud, d, wd):
> + """
> + Check if the repository has 'lfs' (large file) content
> + """
> + cmd = "%s grep lfs HEAD:.gitattributes | wc -l" % (
> + ud.basecmd)
> + try:
> + output = runfetchcmd(cmd, d, quiet=True, workdir=wd)
> + if int(output) > 0:
> + return True
> + except (bb.fetch2.FetchError,ValueError):
> + pass
> + return False
> +
> + def _find_git_lfs(self, d):
> + """
> + Return True if git-lfs can be found, False otherwise.
> + """
> + import shutil
> + return shutil.which("git-lfs", path=d.getVar('PATH')) is not
> None +
> def _get_repo_url(self, ud):
> """
> Return the repository URL
> @@ -615,7 +641,7 @@ class Git(FetchMethod):
> """
> pupver = ('', '')
>
> - tagregex = re.compile(d.getVar('UPSTREAM_CHECK_GITTAGREGEX')
> or "(?P<pver>([0-9][\.|_]?)+)")
> + tagregex = re.compile(d.getVar('UPSTREAM_CHECK_GITTAGREGEX')
> or r"(?P<pver>([0-9][\.|_]?)+)") try:
> output = self._lsremote(ud, d, "refs/tags/*")
> except (bb.fetch2.FetchError, bb.fetch2.NetworkAccess) as e:
> @@ -630,7 +656,7 @@ class Git(FetchMethod):
>
> tag_head = line.split("/")[-1]
> # Ignore non-released branches
> - m = re.search("(alpha|beta|rc|final)+", tag_head)
> + m = re.search(r"(alpha|beta|rc|final)+", tag_head)
> if m:
> continue
>
> diff --git a/bitbake/lib/bb/fetch2/gitannex.py
> b/bitbake/lib/bb/fetch2/gitannex.py index a9b69ca..1d497dc 100644
> --- a/bitbake/lib/bb/fetch2/gitannex.py
> +++ b/bitbake/lib/bb/fetch2/gitannex.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake 'Fetch' git annex implementation
> """
> @@ -7,18 +5,8 @@ BitBake 'Fetch' git annex implementation
> # Copyright (C) 2014 Otavio Salvador
> # Copyright (C) 2014 O.S. Systems Software LTDA.
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import os
> import bb
> diff --git a/bitbake/lib/bb/fetch2/gitsm.py
> b/bitbake/lib/bb/fetch2/gitsm.py index 35729db..c622771 100644
> --- a/bitbake/lib/bb/fetch2/gitsm.py
> +++ b/bitbake/lib/bb/fetch2/gitsm.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake 'Fetch' git submodules implementation
>
> @@ -16,18 +14,8 @@ NOTE: Switching a SRC_URI from "git://" to
> "gitsm://" requires a clean of your r
> # Copyright (C) 2013 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import os
> import bb
> @@ -45,60 +33,97 @@ class GitSM(Git):
> """
> return ud.type in ['gitsm']
>
> - @staticmethod
> - def parse_gitmodules(gitmodules):
> - modules = {}
> - module = ""
> - for line in gitmodules.splitlines():
> - if line.startswith('[submodule'):
> - module = line.split('"')[1]
> - modules[module] = {}
> - elif module and line.strip().startswith('path'):
> - path = line.split('=')[1].strip()
> - modules[module]['path'] = path
> - elif module and line.strip().startswith('url'):
> - url = line.split('=')[1].strip()
> - modules[module]['url'] = url
> - return modules
> -
> - def update_submodules(self, ud, d):
> + def process_submodules(self, ud, workdir, function, d):
> + """
> + Iterate over all of the submodules in this repository and
> execute
> + the 'function' for each of them.
> + """
> +
> submodules = []
> paths = {}
> + revision = {}
> uris = {}
> - local_paths = {}
> -
> + subrevision = {}
> +
> + def parse_gitmodules(gitmodules):
> + modules = {}
> + module = ""
> + for line in gitmodules.splitlines():
> + if line.startswith('[submodule'):
> + module = line.split('"')[1]
> + modules[module] = {}
> + elif module and line.strip().startswith('path'):
> + path = line.split('=')[1].strip()
> + modules[module]['path'] = path
> + elif module and line.strip().startswith('url'):
> + url = line.split('=')[1].strip()
> + modules[module]['url'] = url
> + return modules
> +
> + # Collect the defined submodules, and their attributes
> for name in ud.names:
> try:
> - gitmodules = runfetchcmd("%s show %s:.gitmodules" %
> (ud.basecmd, ud.revisions[name]), d, quiet=True, workdir=ud.clonedir)
> + gitmodules = runfetchcmd("%s show %s:.gitmodules" %
> (ud.basecmd, ud.revisions[name]), d, quiet=True, workdir=workdir)
> except: # No submodules to update
> continue
>
> - for m, md in self.parse_gitmodules(gitmodules).items():
> + for m, md in parse_gitmodules(gitmodules).items():
> + try:
> + module_hash = runfetchcmd("%s ls-tree -z -d %s
> %s" % (ud.basecmd, ud.revisions[name], md['path']), d, quiet=True,
> workdir=workdir)
> + except:
> + # If the command fails, we don't have a valid
> file to check. If it doesn't
> + # fail -- it still might be a failure, see next
> check...
> + module_hash = ""
> +
> + if not module_hash:
> + logger.debug(1, "submodule %s is defined, but is
> not initialized in the repository. Skipping", m)
> + continue
> +
> submodules.append(m)
> paths[m] = md['path']
> + revision[m] = ud.revisions[name]
> uris[m] = md['url']
> + subrevision[m] = module_hash.split()[2]
> +
> + # Convert relative to absolute uri based on parent
> uri if uris[m].startswith('..'):
> newud = copy.copy(ud)
> - newud.path =
> os.path.realpath(os.path.join(newud.path, md['url']))
> + newud.path =
> os.path.realpath(os.path.join(newud.path, uris[m])) uris[m] =
> Git._get_repo_url(self, newud)
> for module in submodules:
> - module_hash = runfetchcmd("%s ls-tree -z -d %s %s" %
> (ud.basecmd, ud.revisions[name], paths[module]), d, quiet=True,
> workdir=ud.clonedir)
> - module_hash = module_hash.split()[2]
> + # Translate the module url into a SRC_URI
> +
> + if "://" in uris[module]:
> + # Properly formated URL already
> + proto = uris[module].split(':', 1)[0]
> + url = uris[module].replace('%s:' % proto, 'gitsm:',
> 1)
> + else:
> + if ":" in uris[module]:
> + # Most likely an SSH style reference
> + proto = "ssh"
> + if ":/" in uris[module]:
> + # Absolute reference, easy to convert..
> + url = "gitsm://" +
> uris[module].replace(':/', '/', 1)
> + else:
> + # Relative reference, no way to know if this
> is right!
> + logger.warning("Submodule included by %s
> refers to relative ssh reference %s. References may fail if not
> absolute." % (ud.url, uris[module]))
> + url = "gitsm://" + uris[module].replace(':',
> '/', 1)
> + else:
> + # This has to be a file reference
> + proto = "file"
> + url = "gitsm://" + uris[module]
>
> - # Build new SRC_URI
> - proto = uris[module].split(':', 1)[0]
> - url = uris[module].replace('%s:' % proto, 'gitsm:', 1)
> url += ';protocol=%s' % proto
> url += ";name=%s" % module
> - url += ";bareclone=1;nocheckout=1;nobranch=1"
> + url += ";subpath=%s" % module
>
> ld = d.createCopy()
> # Not necessary to set SRC_URI, since we're passing the
> URI to # Fetch.
> #ld.setVar('SRC_URI', url)
> - ld.setVar('SRCREV_%s' % module, module_hash)
> + ld.setVar('SRCREV_%s' % module, subrevision[module])
>
> # Workaround for issues with SRCPV/SRCREV_FORMAT errors
> # error refer to 'multiple' repositories. Only the
> repository @@ -106,145 +131,85 @@ class GitSM(Git):
> ld.setVar('SRCPV', d.getVar('SRCPV'))
> ld.setVar('SRCREV_FORMAT', module)
>
> - newfetch = Fetch([url], ld, cache=False)
> - newfetch.download()
> - local_paths[module] = newfetch.localpath(url)
> + function(ud, url, module, paths[module], ld)
>
> - # Correct the submodule references to the local download
> version...
> - runfetchcmd("%(basecmd)s config submodule.%(module)s.url
> %(url)s" % {'basecmd': ud.basecmd, 'module': module, 'url' :
> local_paths[module]}, d, workdir=ud.clonedir) -
> - symlink_path = os.path.join(ud.clonedir, 'modules',
> paths[module])
> - if not os.path.exists(symlink_path):
> - try:
> - os.makedirs(os.path.dirname(symlink_path),
> exist_ok=True)
> - except OSError:
> - pass
> - os.symlink(local_paths[module], symlink_path)
> -
> - return True
> + return submodules != []
>
> def need_update(self, ud, d):
> - main_repo_needs_update = Git.need_update(self, ud, d)
> -
> - # First check that the main repository has enough history
> fetched. If it doesn't, then we don't
> - # even have the .gitmodules and gitlinks for the submodules
> to attempt asking whether the
> - # submodules' histories are recent enough.
> - if main_repo_needs_update:
> + if Git.need_update(self, ud, d):
> return True
>
> - # Now check that the submodule histories are new enough. The
> git-submodule command doesn't have
> - # any clean interface for doing this aside from just
> attempting the checkout (with network
> - # fetched disabled).
> - return not self.update_submodules(ud, d)
> + try:
> + # Check for the nugget dropped by the download operation
> + known_srcrevs = runfetchcmd("%s config --get-all
> bitbake.srcrev" % \
> + (ud.basecmd), d, workdir=ud.clonedir)
>
> - def download(self, ud, d):
> - Git.download(self, ud, d)
> + if ud.revisions[ud.names[0]] not in
> known_srcrevs.split():
> + return True
> + except bb.fetch2.FetchError:
> + # No srcrev nuggets, so this is new and needs to be
> updated
> + return True
>
> - if not ud.shallow or ud.localpath != ud.fullshallow:
> - self.update_submodules(ud, d)
> + return False
>
> - def copy_submodules(self, submodules, ud, destdir, d):
> - if ud.bareclone:
> - repo_conf = destdir
> - else:
> - repo_conf = os.path.join(destdir, '.git')
> + def download(self, ud, d):
> + def download_submodule(ud, url, module, modpath, d):
> + url += ";bareclone=1;nobranch=1"
>
> - if submodules and not os.path.exists(os.path.join(repo_conf,
> 'modules')):
> - os.mkdir(os.path.join(repo_conf, 'modules'))
> + # Is the following still needed?
> + #url += ";nocheckout=1"
>
> - for module, md in submodules.items():
> - srcpath = os.path.join(ud.clonedir, 'modules',
> md['path'])
> - modpath = os.path.join(repo_conf, 'modules', md['path'])
> + try:
> + newfetch = Fetch([url], d, cache=False)
> + newfetch.download()
> + # Drop a nugget to add each of the srcrevs we've
> fetched (used by need_update)
> + runfetchcmd("%s config --add bitbake.srcrev %s" % \
> + (ud.basecmd, ud.revisions[ud.names[0]]),
> d, workdir=ud.clonedir)
> + except Exception as e:
> + logger.error('gitsm: submodule download failed: %s
> %s' % (type(e).__name__, str(e)))
> + raise
>
> - if os.path.exists(srcpath):
> - if os.path.exists(os.path.join(srcpath, '.git')):
> - srcpath = os.path.join(srcpath, '.git')
> + Git.download(self, ud, d)
> + self.process_submodules(ud, ud.clonedir, download_submodule,
> d)
> - target = modpath
> - if os.path.exists(modpath):
> - target = os.path.dirname(modpath)
> + def unpack(self, ud, destdir, d):
> + def unpack_submodules(ud, url, module, modpath, d):
> + url += ";bareclone=1;nobranch=1"
>
> - os.makedirs(os.path.dirname(target), exist_ok=True)
> - runfetchcmd("cp -fpLR %s %s" % (srcpath, target), d)
> - elif os.path.exists(modpath):
> - # Module already exists, likely unpacked from a
> shallow mirror clone
> - pass
> + # Figure out where we clone over the bare submodules...
> + if ud.bareclone:
> + repo_conf = ud.destdir
> else:
> - # This is fatal, as we do NOT want git-submodule to
> hit the network
> - raise bb.fetch2.FetchError('Submodule %s does not
> exist in %s or %s.' % (module, srcpath, modpath)) -
> - def clone_shallow_local(self, ud, dest, d):
> - super(GitSM, self).clone_shallow_local(ud, dest, d)
> + repo_conf = os.path.join(ud.destdir, '.git')
>
> - # Copy over the submodules' fetched histories too.
> - repo_conf = os.path.join(dest, '.git')
> -
> - submodules = []
> - for name in ud.names:
> try:
> - gitmodules = runfetchcmd("%s show %s:.gitmodules" %
> (ud.basecmd, ud.revision), d, quiet=True, workdir=dest)
> - except:
> - # No submodules to update
> - continue
> + newfetch = Fetch([url], d, cache=False)
> +
> newfetch.unpack(root=os.path.dirname(os.path.join(repo_conf,
> 'modules', module)))
> + except Exception as e:
> + logger.error('gitsm: submodule unpack failed: %s %s'
> % (type(e).__name__, str(e)))
> + raise
>
> - submodules = self.parse_gitmodules(gitmodules)
> - self.copy_submodules(submodules, ud, dest, d)
> + local_path = newfetch.localpath(url)
>
> - def unpack(self, ud, destdir, d):
> - Git.unpack(self, ud, destdir, d)
> + # Correct the submodule references to the local download
> version...
> + runfetchcmd("%(basecmd)s config submodule.%(module)s.url
> %(url)s" % {'basecmd': ud.basecmd, 'module': module, 'url' :
> local_path}, d, workdir=ud.destdir)
> - # Copy over the submodules' fetched histories too.
> - if ud.bareclone:
> - repo_conf = ud.destdir
> - else:
> - repo_conf = os.path.join(ud.destdir, '.git')
> + if ud.shallow:
> + runfetchcmd("%(basecmd)s config
> submodule.%(module)s.shallow true" % {'basecmd': ud.basecmd,
> 'module': module}, d, workdir=ud.destdir)
> - update_submodules = False
> - paths = {}
> - uris = {}
> - local_paths = {}
> - for name in ud.names:
> + # Ensure the submodule repository is NOT set to bare,
> since we're checking it out... try:
> - gitmodules = runfetchcmd("%s show HEAD:.gitmodules"
> % (ud.basecmd), d, quiet=True, workdir=ud.destdir)
> + runfetchcmd("%s config core.bare false" %
> (ud.basecmd), d, quiet=True, workdir=os.path.join(repo_conf,
> 'modules', module)) except:
> - # No submodules to update
> - continue
> -
> - submodules = self.parse_gitmodules(gitmodules)
> - self.copy_submodules(submodules, ud, ud.destdir, d)
> -
> - submodules_queue = [(module, os.path.join(repo_conf,
> 'modules', md['path'])) for module, md in submodules.items()]
> - while len(submodules_queue) != 0:
> - module, modpath = submodules_queue.pop()
> -
> - # add submodule children recursively
> - try:
> - gitmodules = runfetchcmd("%s show
> HEAD:.gitmodules" % (ud.basecmd), d, quiet=True, workdir=modpath)
> - for m, md in
> self.parse_gitmodules(gitmodules).items():
> - submodules_queue.append([m,
> os.path.join(modpath, 'modules', md['path'])])
> - except:
> - # no children
> - pass
> -
> + logger.error("Unable to set git config core.bare to
> false for %s" % os.path.join(repo_conf, 'modules', module))
> + raise
>
> - # There are submodules to update
> - update_submodules = True
> -
> - # Determine (from the submodule) the correct url to
> reference
> - try:
> - output = runfetchcmd("%(basecmd)s config
> remote.origin.url" % {'basecmd': ud.basecmd}, d, workdir=modpath)
> - except bb.fetch2.FetchError as e:
> - # No remote url defined in this submodule
> - continue
> -
> - local_paths[module] = output
> -
> - # Setup the local URL properly (like git submodule
> init or sync would do...)
> - runfetchcmd("%(basecmd)s config
> submodule.%(module)s.url %(url)s" % {'basecmd': ud.basecmd, 'module':
> module, 'url' : local_paths[module]}, d, workdir=ud.destdir)
> + Git.unpack(self, ud, destdir, d)
>
> - # Ensure the submodule repository is NOT set to
> bare, since we're checking it out...
> - runfetchcmd("%s config core.bare false" %
> (ud.basecmd), d, quiet=True, workdir=modpath)
> + ret = self.process_submodules(ud, ud.destdir,
> unpack_submodules, d)
> - if update_submodules:
> - # Run submodule update, this sets up the directories --
> without touching the config
> + if not ud.bareclone and ret:
> + # All submodules should already be downloaded and
> configured in the tree. This simply sets
> + # up the configuration and checks out the files. The
> main project config should remain
> + # unmodified, and no download from the internet should
> occur. runfetchcmd("%s submodule update --recursive --no-fetch" %
> (ud.basecmd), d, quiet=True, workdir=ud.destdir) diff --git
> a/bitbake/lib/bb/fetch2/hg.py b/bitbake/lib/bb/fetch2/hg.py index
> 936d043..15d729e 100644 --- a/bitbake/lib/bb/fetch2/hg.py
> +++ b/bitbake/lib/bb/fetch2/hg.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake 'Fetch' implementation for mercurial DRCS (hg).
>
> @@ -9,20 +7,10 @@ BitBake 'Fetch' implementation for mercurial DRCS
> (hg). # Copyright (C) 2004 Marcin Juszkiewicz
> # Copyright (C) 2007 Robert Schuster
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
> # Based on functions from the base bb module, Copyright 2003 Holger
> Schurig +#
>
> import os
> import sys
> @@ -99,7 +87,7 @@ class Hg(FetchMethod):
> def try_premirror(self, ud, d):
> # If we don't do this, updating an existing checkout with
> only premirrors # is not possible
> - if d.getVar("BB_FETCH_PREMIRRORONLY") is not None:
> + if bb.utils.to_boolean(d.getVar("BB_FETCH_PREMIRRORONLY")):
> return True
> if os.path.exists(ud.moddir):
> return False
> diff --git a/bitbake/lib/bb/fetch2/local.py
> b/bitbake/lib/bb/fetch2/local.py index a114ac1..01d9ff9 100644
> --- a/bitbake/lib/bb/fetch2/local.py
> +++ b/bitbake/lib/bb/fetch2/local.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake 'Fetch' implementations
>
> @@ -10,20 +8,10 @@ BitBake build tools.
>
> # Copyright (C) 2003, 2004 Chris Larson
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
> # Based on functions from the base bb module, Copyright 2003 Holger
> Schurig +#
>
> import os
> import urllib.request, urllib.parse, urllib.error
> diff --git a/bitbake/lib/bb/fetch2/npm.py
> b/bitbake/lib/bb/fetch2/npm.py index 65bf5a3..9700e61 100644
> --- a/bitbake/lib/bb/fetch2/npm.py
> +++ b/bitbake/lib/bb/fetch2/npm.py
> @@ -1,5 +1,6 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> """
> BitBake 'Fetch' NPM implementation
>
> @@ -100,11 +101,19 @@ class Npm(FetchMethod):
> return False
> return True
>
> - def _runwget(self, ud, d, command, quiet):
> - logger.debug(2, "Fetching %s using command '%s'" % (ud.url,
> command))
> - bb.fetch2.check_network_access(d, command, ud.url)
> + def _runpack(self, ud, d, pkgfullname: str, quiet=False) -> str:
> + """
> + Runs npm pack on a full package name.
> + Returns the filename of the downloaded package
> + """
> + bb.fetch2.check_network_access(d, pkgfullname, ud.registry)
> dldir = d.getVar("DL_DIR")
> - runfetchcmd(command, d, quiet, workdir=dldir)
> + dldir = os.path.join(dldir, ud.prefixdir)
> +
> + command = "npm pack {} --registry {}".format(pkgfullname,
> ud.registry)
> + logger.debug(2, "Fetching {} using command '{}' in
> {}".format(pkgfullname, command, dldir))
> + filename = runfetchcmd(command, d, quiet, workdir=dldir)
> + return filename.rstrip()
>
> def _unpackdep(self, ud, pkg, data, destdir, dldir, d):
> file = data[pkg]['tgz']
> @@ -150,20 +159,11 @@ class Npm(FetchMethod):
> Parse the output of npm view --json; the last JSON result
> is assumed to be the one that we're interested in.
> '''
> - pdata = None
> - outdeps = {}
> - datalines = []
> - bracelevel = 0
> - for line in output.splitlines():
> - if bracelevel:
> - datalines.append(line)
> - elif '{' in line:
> - datalines = []
> - datalines.append(line)
> - bracelevel = bracelevel + line.count('{') -
> line.count('}')
> - if datalines:
> - pdata = json.loads('\n'.join(datalines))
> - return pdata
> + pdata = json.loads(output);
> + try:
> + return pdata[-1]
> + except:
> + return pdata
>
> def _getdependencies(self, pkg, data, version, d, ud,
> optional=False, fetchedlist=None): if fetchedlist is None:
> @@ -171,6 +171,9 @@ class Npm(FetchMethod):
> pkgfullname = pkg
> if version != '*' and not '/' in version:
> pkgfullname += "@'%s'" % version
> + if pkgfullname in fetchedlist:
> + return
> +
> logger.debug(2, "Calling getdeps on %s" % pkg)
> fetchcmd = "npm view %s --json --registry %s" %
> (pkgfullname, ud.registry) output = runfetchcmd(fetchcmd, d, True)
> @@ -190,15 +193,10 @@ class Npm(FetchMethod):
> if (not blacklist and 'linux' not in pkg_os) or
> '!linux' in pkg_os: logger.debug(2, "Skipping %s since it's
> incompatible with Linux" % pkg) return
> - #logger.debug(2, "Output URL is %s - %s - %s" %
> (ud.basepath, ud.basename, ud.localfile))
> - outputurl = pdata['dist']['tarball']
> + filename = self._runpack(ud, d, pkgfullname)
> data[pkg] = {}
> - data[pkg]['tgz'] = os.path.basename(outputurl)
> - if outputurl in fetchedlist:
> - return
> -
> - self._runwget(ud, d, "%s --directory-prefix=%s %s" %
> (self.basecmd, ud.prefixdir, outputurl), False)
> - fetchedlist.append(outputurl)
> + data[pkg]['tgz'] = filename
> + fetchedlist.append(pkgfullname)
>
> dependencies = pdata.get('dependencies', {})
> optionalDependencies = pdata.get('optionalDependencies', {})
> @@ -225,17 +223,12 @@ class Npm(FetchMethod):
> if obj == pkg:
> self._getshrinkeddependencies(obj,
> data['dependencies'][obj], data['dependencies'][obj]['version'], d,
> ud, lockdown, manifest, False) return
> - outputurl = "invalid"
> - if ('resolved' not in data) or (not
> data['resolved'].startswith('http://') and not
> data['resolved'].startswith('https://')):
> - # will be the case for ${PN}
> - fetchcmd = "npm view %s@%s dist.tarball --registry %s" %
> (pkg, version, ud.registry)
> - logger.debug(2, "Found this matching URL: %s" %
> str(fetchcmd))
> - outputurl = runfetchcmd(fetchcmd, d, True)
> - else:
> - outputurl = data['resolved']
> - self._runwget(ud, d, "%s --directory-prefix=%s %s" %
> (self.basecmd, ud.prefixdir, outputurl), False) +
> + pkgnameWithVersion = "{}@{}".format(pkg, version)
> + logger.debug(2, "Get dependencies for
> {}".format(pkgnameWithVersion))
> + filename = self._runpack(ud, d, pkgnameWithVersion)
> manifest[pkg] = {}
> - manifest[pkg]['tgz'] = os.path.basename(outputurl).rstrip()
> + manifest[pkg]['tgz'] = filename
> manifest[pkg]['deps'] = {}
>
> if pkg in lockdown:
> diff --git a/bitbake/lib/bb/fetch2/osc.py
> b/bitbake/lib/bb/fetch2/osc.py index 6c60456..3e56715 100644
> --- a/bitbake/lib/bb/fetch2/osc.py
> +++ b/bitbake/lib/bb/fetch2/osc.py
> @@ -1,5 +1,6 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> """
> Bitbake "Fetch" implementation for osc (Opensuse build service
> client). Based on the svn "Fetch" implementation.
> diff --git a/bitbake/lib/bb/fetch2/perforce.py
> b/bitbake/lib/bb/fetch2/perforce.py index 903a8e6..54d001e 100644
> --- a/bitbake/lib/bb/fetch2/perforce.py
> +++ b/bitbake/lib/bb/fetch2/perforce.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake 'Fetch' implementation for perforce
>
> @@ -8,18 +6,7 @@ BitBake 'Fetch' implementation for perforce
> # Copyright (C) 2003, 2004 Chris Larson
> # Copyright (C) 2016 Kodak Alaris, Inc.
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
> # Based on functions from the base bb module, Copyright 2003 Holger
> Schurig
> diff --git a/bitbake/lib/bb/fetch2/repo.py
> b/bitbake/lib/bb/fetch2/repo.py index 8c7e818..2bdbbd4 100644
> --- a/bitbake/lib/bb/fetch2/repo.py
> +++ b/bitbake/lib/bb/fetch2/repo.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake "Fetch" repo (git) implementation
>
> @@ -8,20 +6,10 @@ BitBake "Fetch" repo (git) implementation
> # Copyright (C) 2009 Tom Rini <trini@embeddedalley.com>
> #
> # Based on git.py which is:
> -#Copyright (C) 2005 Richard Purdie
> +# Copyright (C) 2005 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import os
> import bb
> diff --git a/bitbake/lib/bb/fetch2/s3.py b/bitbake/lib/bb/fetch2/s3.py
> index 1629288..ffca73c 100644
> --- a/bitbake/lib/bb/fetch2/s3.py
> +++ b/bitbake/lib/bb/fetch2/s3.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake 'Fetch' implementation for Amazon AWS S3.
>
> @@ -13,18 +11,7 @@ The aws tool must be correctly installed and
> configured prior to use. # Based in part on bb.fetch2.wget:
> # Copyright (C) 2003, 2004 Chris Larson
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
> # Based on functions from the base bb module, Copyright 2003 Holger
> Schurig
> diff --git a/bitbake/lib/bb/fetch2/sftp.py
> b/bitbake/lib/bb/fetch2/sftp.py index 81884a6..f87f292 100644
> --- a/bitbake/lib/bb/fetch2/sftp.py
> +++ b/bitbake/lib/bb/fetch2/sftp.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake SFTP Fetch implementation
>
> @@ -44,18 +42,7 @@ SRC_URI =
> "sftp://user@host.example.com/dir/path.file.txt" # Based in part on
> bb.fetch2.wget: # Copyright (C) 2003, 2004 Chris Larson
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
> # Based on functions from the base bb module, Copyright 2003 Holger
> Schurig
> diff --git a/bitbake/lib/bb/fetch2/ssh.py
> b/bitbake/lib/bb/fetch2/ssh.py index 6047ee4..f5be060 100644
> --- a/bitbake/lib/bb/fetch2/ssh.py
> +++ b/bitbake/lib/bb/fetch2/ssh.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> '''
> BitBake 'Fetch' implementations
>
> @@ -29,18 +27,8 @@ IETF secsh internet draft:
> # Copyright 2003 Holger Schurig
> #
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import re, os
> from bb.fetch2 import FetchMethod
> diff --git a/bitbake/lib/bb/fetch2/svn.py
> b/bitbake/lib/bb/fetch2/svn.py index ed70bcf..96d666b 100644
> --- a/bitbake/lib/bb/fetch2/svn.py
> +++ b/bitbake/lib/bb/fetch2/svn.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake 'Fetch' implementation for svn.
>
> @@ -8,18 +6,7 @@ BitBake 'Fetch' implementation for svn.
> # Copyright (C) 2003, 2004 Chris Larson
> # Copyright (C) 2004 Marcin Juszkiewicz
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
> # Based on functions from the base bb module, Copyright 2003 Holger
> Schurig
> @@ -63,6 +50,9 @@ class Svn(FetchMethod):
> relpath = self._strip_leading_slashes(ud.path)
> ud.pkgdir = os.path.join(svndir, ud.host, relpath)
> ud.moddir = os.path.join(ud.pkgdir, ud.module)
> + # Protects the repository from concurrent updates, e.g. from
> two
> + # recipes fetching different revisions at the same time
> + ud.svnlock = os.path.join(ud.pkgdir, "svn.lock")
>
> ud.setup_revisions(d)
>
> @@ -101,6 +91,13 @@ class Svn(FetchMethod):
> svncmd = "%s log --limit 1 %s %s://%s/%s/" %
> (ud.basecmd, " ".join(options), proto, svnroot, ud.module) else:
> suffix = ""
> +
> + # externals may be either 'allowed' or 'nowarn', but not
> both. Allowed
> + # will not issue a warning, but will log to the debug
> buffer what has likely
> + # been downloaded by SVN.
> + if not ("externals" in ud.parm and ud.parm["externals"]
> == "allowed"):
> + options.append("--ignore-externals")
> +
> if ud.revision:
> options.append("-r %s" % ud.revision)
> suffix = "@%s" % (ud.revision)
> @@ -123,35 +120,52 @@ class Svn(FetchMethod):
>
> logger.debug(2, "Fetch: checking for module directory '" +
> ud.moddir + "'")
> - if os.access(os.path.join(ud.moddir, '.svn'), os.R_OK):
> - svnupdatecmd = self._buildsvncommand(ud, d, "update")
> - logger.info("Update " + ud.url)
> - # We need to attempt to run svn upgrade first in case
> its an older working format
> - try:
> - runfetchcmd(ud.basecmd + " upgrade", d,
> workdir=ud.moddir)
> - except FetchError:
> - pass
> - logger.debug(1, "Running %s", svnupdatecmd)
> - bb.fetch2.check_network_access(d, svnupdatecmd, ud.url)
> - runfetchcmd(svnupdatecmd, d, workdir=ud.moddir)
> - else:
> - svnfetchcmd = self._buildsvncommand(ud, d, "fetch")
> - logger.info("Fetch " + ud.url)
> - # check out sources there
> - bb.utils.mkdirhier(ud.pkgdir)
> - logger.debug(1, "Running %s", svnfetchcmd)
> - bb.fetch2.check_network_access(d, svnfetchcmd, ud.url)
> - runfetchcmd(svnfetchcmd, d, workdir=ud.pkgdir)
> -
> - scmdata = ud.parm.get("scmdata", "")
> - if scmdata == "keep":
> - tar_flags = ""
> - else:
> - tar_flags = "--exclude='.svn'"
> + lf = bb.utils.lockfile(ud.svnlock)
> +
> + try:
> + if os.access(os.path.join(ud.moddir, '.svn'), os.R_OK):
> + svnupdatecmd = self._buildsvncommand(ud, d, "update")
> + logger.info("Update " + ud.url)
> + # We need to attempt to run svn upgrade first in
> case its an older working format
> + try:
> + runfetchcmd(ud.basecmd + " upgrade", d,
> workdir=ud.moddir)
> + except FetchError:
> + pass
> + logger.debug(1, "Running %s", svnupdatecmd)
> + bb.fetch2.check_network_access(d, svnupdatecmd,
> ud.url)
> + runfetchcmd(svnupdatecmd, d, workdir=ud.moddir)
> + else:
> + svnfetchcmd = self._buildsvncommand(ud, d, "fetch")
> + logger.info("Fetch " + ud.url)
> + # check out sources there
> + bb.utils.mkdirhier(ud.pkgdir)
> + logger.debug(1, "Running %s", svnfetchcmd)
> + bb.fetch2.check_network_access(d, svnfetchcmd,
> ud.url)
> + runfetchcmd(svnfetchcmd, d, workdir=ud.pkgdir)
> +
> + if not ("externals" in ud.parm and ud.parm["externals"]
> == "nowarn"):
> + # Warn the user if this had externals (won't catch
> them all)
> + output = runfetchcmd("svn propget svn:externals ||
> true", d, workdir=ud.moddir)
> + if output:
> + if "--ignore-externals" in svnfetchcmd.split():
> + bb.warn("%s contains svn:externals." %
> ud.url)
> + bb.warn("These should be added to the recipe
> SRC_URI as necessary.")
> + bb.warn("svn fetch has ignored
> externals:\n%s" % output)
> + bb.warn("To disable this warning add
> ';externals=nowarn' to the url.")
> + else:
> + bb.debug(1, "svn repository has
> externals:\n%s" % output) +
> + scmdata = ud.parm.get("scmdata", "")
> + if scmdata == "keep":
> + tar_flags = ""
> + else:
> + tar_flags = "--exclude='.svn'"
>
> - # tar them up to a defined filename
> - runfetchcmd("tar %s -czf %s %s" % (tar_flags, ud.localpath,
> ud.path_spec), d,
> - cleanup=[ud.localpath], workdir=ud.pkgdir)
> + # tar them up to a defined filename
> + runfetchcmd("tar %s -czf %s %s" % (tar_flags,
> ud.localpath, ud.path_spec), d,
> + cleanup=[ud.localpath], workdir=ud.pkgdir)
> + finally:
> + bb.utils.unlockfile(lf)
>
> def clean(self, ud, d):
> """ Clean SVN specific files and dirs """
> diff --git a/bitbake/lib/bb/fetch2/wget.py
> b/bitbake/lib/bb/fetch2/wget.py index 8f505b6..725586d 100644
> --- a/bitbake/lib/bb/fetch2/wget.py
> +++ b/bitbake/lib/bb/fetch2/wget.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake 'Fetch' implementations
>
> @@ -10,18 +8,7 @@ BitBake build tools.
>
> # Copyright (C) 2003, 2004 Chris Larson
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
> # Based on functions from the base bb module, Copyright 2003 Holger
> Schurig
> @@ -33,11 +20,14 @@ import logging
> import errno
> import bb
> import bb.progress
> +import socket
> +import http.client
> import urllib.request, urllib.parse, urllib.error
> from bb.fetch2 import FetchMethod
> from bb.fetch2 import FetchError
> from bb.fetch2 import logger
> from bb.fetch2 import runfetchcmd
> +from bb.fetch2 import FetchConnectionCache
> from bb.utils import export_proxies
> from bs4 import BeautifulSoup
> from bs4 import SoupStrainer
> @@ -132,10 +122,6 @@ class Wget(FetchMethod):
> return True
>
> def checkstatus(self, fetch, ud, d, try_again=True):
> - import urllib.request, urllib.error, urllib.parse, socket,
> http.client
> - from urllib.response import addinfourl
> - from bb.fetch2 import FetchConnectionCache
> -
> class HTTPConnectionCache(http.client.HTTPConnection):
> if fetch.connection_cache:
> def connect(self):
> @@ -168,7 +154,7 @@ class Wget(FetchMethod):
> """
> host = req.host
> if not host:
> - raise urlllib2.URLError('no host given')
> + raise urllib.error.URLError('no host given')
>
> h = http_class(host, timeout=req.timeout) # will
> parse host:port h.set_debuglevel(self._debuglevel)
> @@ -185,7 +171,7 @@ class Wget(FetchMethod):
> # request.
>
> # Don't close connection when connection_cache is
> enabled,
> - if fetch.connection_cache is None:
> + if fetch.connection_cache is None:
> headers["Connection"] = "close"
> else:
> headers["Connection"] = "Keep-Alive" # Works for
> HTTP/1.0 @@ -252,7 +238,7 @@ class Wget(FetchMethod):
> pass
> closed = False
>
> - resp = addinfourl(fp_dummy(), r.msg,
> req.get_full_url())
> + resp = urllib.response.addinfourl(fp_dummy(), r.msg,
> req.get_full_url()) resp.code = r.status
> resp.msg = r.reason
>
> @@ -271,17 +257,18 @@ class Wget(FetchMethod):
> fp.read()
> fp.close()
>
> - newheaders = dict((k,v) for k,v in
> list(req.headers.items())
> - if k.lower() not in
> ("content-length", "content-type"))
> - return
> self.parent.open(urllib.request.Request(req.get_full_url(),
> -
> headers=newheaders,
> -
> origin_req_host=req.origin_req_host,
> -
> unverifiable=True))
> + if req.get_method() != 'GET':
> + newheaders = dict((k, v) for k, v in
> list(req.headers.items())
> + if k.lower() not in
> ("content-length", "content-type"))
> + return
> self.parent.open(urllib.request.Request(req.get_full_url(),
> +
> headers=newheaders,
> +
> origin_req_host=req.origin_req_host,
> +
> unverifiable=True))
> - """
> - Some servers (e.g. GitHub archives, hosted on Amazon S3)
> return 403
> - Forbidden when they actually mean 405 Method Not Allowed.
> - """
> + raise urllib.request.HTTPError(req, code, msg,
> headers, None) +
> + # Some servers (e.g. GitHub archives, hosted on Amazon
> S3) return 403
> + # Forbidden when they actually mean 405 Method Not
> Allowed. http_error_403 = http_error_405
>
>
> @@ -292,15 +279,15 @@ class Wget(FetchMethod):
> """
> def redirect_request(self, req, fp, code, msg, headers,
> newurl): newreq =
> urllib.request.HTTPRedirectHandler.redirect_request(self, req, fp,
> code, msg, headers, newurl)
> - newreq.get_method = lambda: req.get_method()
> + newreq.get_method = req.get_method
> return newreq
> exported_proxies = export_proxies(d)
>
> handlers = [FixedHTTPRedirectHandler, HTTPMethodFallback]
> - if export_proxies:
> + if exported_proxies:
> handlers.append(urllib.request.ProxyHandler())
> handlers.append(CacheHTTPHandler())
> - # XXX: Since Python 2.7.9 ssl cert validation is enabled by
> default
> + # Since Python 2.7.9 ssl cert validation is enabled by
> default # see PEP-0476, this causes verification errors on some https
> servers # so disable by default.
> import ssl
> @@ -319,19 +306,19 @@ class Wget(FetchMethod):
> '''Adds Basic auth to http request, pass in
> login:password as string''' import base64
> encodeuser =
> base64.b64encode(login_str.encode('utf-8')).decode("utf-8")
> - authheader = "Basic %s" % encodeuser
> + authheader = "Basic %s" % encodeuser
> r.add_header("Authorization", authheader)
>
> - if ud.user:
> - add_basic_auth(ud.user, r)
> + if ud.user and ud.pswd:
> + add_basic_auth(ud.user + ':' + ud.pswd, r)
>
> try:
> - import netrc, urllib.parse
> + import netrc
> n = netrc.netrc()
> login, unused, password =
> n.authenticators(urllib.parse.urlparse(uri).hostname)
> add_basic_auth("%s:%s" % (login, password), r) except (TypeError,
> ImportError, IOError, netrc.NetrcParseError):
> - pass
> + pass
>
> with opener.open(r) as response:
> pass
> @@ -396,18 +383,14 @@ class Wget(FetchMethod):
> (oldpn, oldpv, oldsuffix) = old
> (newpn, newpv, newsuffix) = new
>
> - """
> - Check for a new suffix type that we have never heard of
> before
> - """
> - if (newsuffix):
> + # Check for a new suffix type that we have never heard of
> before
> + if newsuffix:
> m = self.suffix_regex_comp.search(newsuffix)
> if not m:
> bb.warn("%s has a possible unknown suffix: %s" %
> (newpn, newsuffix)) return False
>
> - """
> - Not our package so ignore it
> - """
> + # Not our package so ignore it
> if oldpn != newpn:
> return False
>
> @@ -473,15 +456,14 @@ class Wget(FetchMethod):
>
> return ""
>
> - def _check_latest_version_by_dir(self, dirver, package,
> package_regex,
> - current_version, ud, d):
> + def _check_latest_version_by_dir(self, dirver, package,
> package_regex, current_version, ud, d): """
> - Scan every directory in order to get upstream version.
> + Scan every directory in order to get upstream version.
> """
> version_dir = ['', '', '']
> version = ['', '', '']
>
> - dirver_regex =
> re.compile("(?P<pfx>\D*)(?P<ver>(\d+[\.\-_])+(\d+))")
> + dirver_regex =
> re.compile(r"(?P<pfx>\D*)(?P<ver>(\d+[\.\-_])+(\d+))") s =
> dirver_regex.search(dirver) if s:
> version_dir[1] = s.group('ver')
> @@ -541,26 +523,26 @@ class Wget(FetchMethod):
> gst-fluendo-mp3
> """
> # match most patterns which uses "-" as separator to version
> digits
> - pn_prefix1 = "[a-zA-Z][a-zA-Z0-9]*([-_][a-zA-Z]\w+)*\+?[-_]"
> + pn_prefix1 = r"[a-zA-Z][a-zA-Z0-9]*([-_][a-zA-Z]\w+)*\+?[-_]"
> # a loose pattern such as for unzip552.tar.gz
> - pn_prefix2 = "[a-zA-Z]+"
> + pn_prefix2 = r"[a-zA-Z]+"
> # a loose pattern such as for 80325-quicky-0.4.tar.gz
> - pn_prefix3 = "[0-9]+[-]?[a-zA-Z]+"
> + pn_prefix3 = r"[0-9]+[-]?[a-zA-Z]+"
> # Save the Package Name (pn) Regex for use later
> - pn_regex = "(%s|%s|%s)" % (pn_prefix1, pn_prefix2,
> pn_prefix3)
> + pn_regex = r"(%s|%s|%s)" % (pn_prefix1, pn_prefix2,
> pn_prefix3)
> # match version
> - pver_regex = "(([A-Z]*\d+[a-zA-Z]*[\.\-_]*)+)"
> + pver_regex = r"(([A-Z]*\d+[a-zA-Z]*[\.\-_]*)+)"
>
> # match arch
> parch_regex = "-source|_all_"
>
> # src.rpm extension was added only for rpm package. Can be
> removed if the rpm # packaged will always be considered as having to
> be manually upgraded
> - psuffix_regex =
> "(tar\.gz|tgz|tar\.bz2|zip|xz|tar\.lz|rpm|bz2|orig\.tar\.gz|tar\.xz|src\.tar\.gz|src\.tgz|svnr\d+\.tar\.bz2|stable\.tar\.gz|src\.rpm)"
> + psuffix_regex =
> r"(tar\.gz|tgz|tar\.bz2|zip|xz|tar\.lz|rpm|bz2|orig\.tar\.gz|tar\.xz|src\.tar\.gz|src\.tgz|svnr\d+\.tar\.bz2|stable\.tar\.gz|src\.rpm)"
> # match name, version and archive type of a package
> - package_regex_comp =
> re.compile("(?P<name>%s?\.?v?)(?P<pver>%s)(?P<arch>%s)?[\.-](?P<type>%s$)"
> + package_regex_comp =
> re.compile(r"(?P<name>%s?\.?v?)(?P<pver>%s)(?P<arch>%s)?[\.-](?P<type>%s$)"
> % (pn_regex, pver_regex, parch_regex, psuffix_regex))
> self.suffix_regex_comp = re.compile(psuffix_regex)
> @@ -572,7 +554,7 @@ class Wget(FetchMethod):
> version = self._parse_path(package_regex_comp, package)
> if version:
> package_custom_regex_comp = re.compile(
> -
> "(?P<name>%s)(?P<pver>%s)(?P<arch>%s)?[\.-](?P<type>%s)" %
> +
> r"(?P<name>%s)(?P<pver>%s)(?P<arch>%s)?[\.-](?P<type>%s)" %
> (re.escape(version[0]), pver_regex, parch_regex, psuffix_regex)) else:
> package_custom_regex_comp = None
> @@ -589,7 +571,7 @@ class Wget(FetchMethod):
> current_version = ['', d.getVar('PV'), '']
>
> """possible to have no version in pkg name, such as
> spectrum-fw"""
> - if not re.search("\d+", package):
> + if not re.search(r"\d+", package):
> current_version[1] = re.sub('_', '.', current_version[1])
> current_version[1] = re.sub('-', '.', current_version[1])
> return (current_version[1], '')
> @@ -607,13 +589,13 @@ class Wget(FetchMethod):
>
> # search for version matches on folders inside the path,
> like: # "5.7" in
> http://download.gnome.org/sources/${PN}/5.7/${PN}-${PV}.tar.gz
> - dirver_regex =
> re.compile("(?P<dirver>[^/]*(\d+\.)*\d+([-_]r\d+)*)/")
> + dirver_regex =
> re.compile(r"(?P<dirver>[^/]*(\d+\.)*\d+([-_]r\d+)*)/") m =
> dirver_regex.search(path) if m:
> pn = d.getVar('PN')
> dirver = m.group('dirver')
>
> - dirver_pn_regex = re.compile("%s\d?" %
> (re.escape(pn)))
> + dirver_pn_regex = re.compile(r"%s\d?" %
> (re.escape(pn))) if not dirver_pn_regex.search(dirver):
> return (self._check_latest_version_by_dir(dirver,
> package, package_regex, current_version, ud,
> d), '') diff --git a/bitbake/lib/bb/main.py b/bitbake/lib/bb/main.py
> index 732a315..af2880f 100755
> --- a/bitbake/lib/bb/main.py
> +++ b/bitbake/lib/bb/main.py
> @@ -1,6 +1,3 @@
> -#!/usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # Copyright (C) 2003, 2004 Chris Larson
> # Copyright (C) 2003, 2004 Phil Blundell
> @@ -9,18 +6,8 @@
> # Copyright (C) 2005 ROAD GmbH
> # Copyright (C) 2006 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import os
> import sys
> @@ -267,6 +254,11 @@ class
> BitBakeConfigParameters(cookerdata.ConfigParameters): help="Do not
> run any setscene tasks. sstate will be ignored and " "everything
> needed, built.")
> + parser.add_option("", "--skip-setscene", action="store_true",
> + dest="skipsetscene", default=False,
> + help="Skip setscene tasks if they would be
> executed. Tasks previously "
> + "restored from sstate will be kept,
> unlike --no-setscene") +
> parser.add_option("", "--setscene-only", action="store_true",
> dest="setsceneonly", default=False,
> help="Only run setscene tasks, don't run
> any real tasks.") @@ -448,7 +440,7 @@ def setup_bitbake(configParams,
> configuration, extrafeatures=None): else:
> logger.info("Reconnecting to bitbake server...")
> if not os.path.exists(sockname):
> - print("Previous bitbake instance shutting
> down?, waiting to retry...")
> + logger.info("Previous bitbake instance
> shutting down?, waiting to retry...") i = 0
> lock = None
> # Wait for 5s or until we can get the lock
> @@ -460,12 +452,7 @@ def setup_bitbake(configParams, configuration,
> extrafeatures=None): bb.utils.unlockfile(lock)
> raise
> bb.server.process.ProcessTimeout("Bitbake still shutting down as
> socket exists but no lock?") if not configParams.server_only:
> - try:
> - server_connection =
> bb.server.process.connectProcessServer(sockname, featureset)
> - except EOFError:
> - # The server may have been shutting down but
> not closed the socket yet. If that happened,
> - # ignore it.
> - pass
> + server_connection =
> bb.server.process.connectProcessServer(sockname, featureset)
> if server_connection or configParams.server_only:
> break
> @@ -475,12 +462,14 @@ def setup_bitbake(configParams, configuration,
> extrafeatures=None): if not retries:
> raise
> retries -= 1
> - if isinstance(e, (bb.server.process.ProcessTimeout,
> BrokenPipeError)):
> - logger.info("Retrying server connection...")
> + tryno = 8 - retries
> + if isinstance(e, (bb.server.process.ProcessTimeout,
> BrokenPipeError, EOFError)):
> + logger.info("Retrying server connection
> (#%d)..." % tryno) else:
> - logger.info("Retrying server connection... (%s)"
> % traceback.format_exc())
> + logger.info("Retrying server connection (#%d)...
> (%s)" % (tryno, traceback.format_exc())) if not retries:
> - bb.fatal("Unable to connect to bitbake server, or
> start one")
> + bb.fatal("Unable to connect to bitbake server, or
> start one (server startup failures would be in
> bitbake-cookerdaemon.log).")
> + bb.event.print_ui_queue()
> if retries < 5:
> time.sleep(5)
>
> @@ -502,7 +491,7 @@ def setup_bitbake(configParams, configuration,
> extrafeatures=None): def lockBitbake():
> topdir = bb.cookerdata.findTopdir()
> if not topdir:
> - bb.error("Unable to find conf/bblayers.conf or
> conf/bitbake.conf. BBAPTH is unset and/or not in a build directory?")
> + bb.error("Unable to find conf/bblayers.conf or
> conf/bitbake.conf. BBPATH is unset and/or not in a build directory?")
> raise BBMainFatal lockfile = topdir + "/bitbake.lock"
> return topdir, bb.utils.lockfile(lockfile, False, False)
> diff --git a/bitbake/lib/bb/methodpool.py
> b/bitbake/lib/bb/methodpool.py index 49aed33..51783ac 100644
> --- a/bitbake/lib/bb/methodpool.py
> +++ b/bitbake/lib/bb/methodpool.py
> @@ -1,21 +1,8 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> -#
> #
> # Copyright (C) 2006 Holger Hans Peter Freyther
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from bb.utils import better_compile, better_exec
>
> diff --git a/bitbake/lib/bb/monitordisk.py
> b/bitbake/lib/bb/monitordisk.py index 833cd3d..1a25b00 100644
> --- a/bitbake/lib/bb/monitordisk.py
> +++ b/bitbake/lib/bb/monitordisk.py
> @@ -1,21 +1,8 @@
> -#!/usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # Copyright (C) 2012 Robert Yang
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import os, logging, re, sys
> import bb
> @@ -28,16 +15,16 @@ def convertGMK(unit):
>
> """ Convert the space unit G, M, K, the unit is case-insensitive
> """
> - unitG = re.match('([1-9][0-9]*)[gG]\s?$', unit)
> + unitG = re.match(r'([1-9][0-9]*)[gG]\s?$', unit)
> if unitG:
> return int(unitG.group(1)) * (1024 ** 3)
> - unitM = re.match('([1-9][0-9]*)[mM]\s?$', unit)
> + unitM = re.match(r'([1-9][0-9]*)[mM]\s?$', unit)
> if unitM:
> return int(unitM.group(1)) * (1024 ** 2)
> - unitK = re.match('([1-9][0-9]*)[kK]\s?$', unit)
> + unitK = re.match(r'([1-9][0-9]*)[kK]\s?$', unit)
> if unitK:
> return int(unitK.group(1)) * 1024
> - unitN = re.match('([1-9][0-9]*)\s?$', unit)
> + unitN = re.match(r'([1-9][0-9]*)\s?$', unit)
> if unitN:
> return int(unitN.group(1))
> else:
> @@ -83,7 +70,7 @@ def getDiskData(BBDirs, configuration):
> for pathSpaceInode in BBDirs.split():
> # The input format is: "dir,space,inode", dir is a must,
> space # and inode are optional
> - pathSpaceInodeRe = re.match('([^,]*),([^,]*),([^,]*),?(.*)',
> pathSpaceInode)
> + pathSpaceInodeRe =
> re.match(r'([^,]*),([^,]*),([^,]*),?(.*)', pathSpaceInode) if not
> pathSpaceInodeRe: printErr("Invalid value in BB_DISKMON_DIRS: %s" %
> pathSpaceInode) return None
> @@ -147,7 +134,7 @@ def getInterval(configuration):
> else:
> # The disk space or inode interval is optional, but it should
> # have a correct value once it is specified
> - intervalRe = re.match('([^,]*),?\s*(.*)', interval)
> + intervalRe = re.match(r'([^,]*),?\s*(.*)', interval)
> if intervalRe:
> intervalSpace = intervalRe.group(1)
> if intervalSpace:
> diff --git a/bitbake/lib/bb/msg.py b/bitbake/lib/bb/msg.py
> index 96f077e..6216eb3 100644
> --- a/bitbake/lib/bb/msg.py
> +++ b/bitbake/lib/bb/msg.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake 'msg' implementation
>
> @@ -9,18 +7,8 @@ Message handling infrastructure for bitbake
>
> # Copyright (C) 2006 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import sys
> import copy
> diff --git a/bitbake/lib/bb/namedtuple_with_abc.py
> b/bitbake/lib/bb/namedtuple_with_abc.py index 32f2fc6..646aed6 100644
> --- a/bitbake/lib/bb/namedtuple_with_abc.py
> +++ b/bitbake/lib/bb/namedtuple_with_abc.py
> @@ -1,6 +1,8 @@
> #
> http://code.activestate.com/recipes/577629-namedtupleabc-abstract-base-class-mix-in-for-named/
> -#!/usr/bin/env python # Copyright (c) 2011 Jan Kaliszewski (zuo).
> Available under the MIT License. +#
> +# SPDX-License-Identifier: MIT
> +#
>
> """
> namedtuple_with_abc.py:
> diff --git a/bitbake/lib/bb/parse/__init__.py
> b/bitbake/lib/bb/parse/__init__.py index 5397d57..76e180b 100644
> --- a/bitbake/lib/bb/parse/__init__.py
> +++ b/bitbake/lib/bb/parse/__init__.py
> @@ -9,20 +9,10 @@ File parsers for the BitBake build tools.
> # Copyright (C) 2003, 2004 Chris Larson
> # Copyright (C) 2003, 2004 Phil Blundell
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
> # Based on functions from the base bb module, Copyright 2003 Holger
> Schurig +#
>
> handlers = []
>
> diff --git a/bitbake/lib/bb/parse/ast.py b/bitbake/lib/bb/parse/ast.py
> index 6d7c80b..f0911e6 100644
> --- a/bitbake/lib/bb/parse/ast.py
> +++ b/bitbake/lib/bb/parse/ast.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> AbstractSyntaxTree classes for the Bitbake language
> """
> @@ -8,19 +6,8 @@
> # Copyright (C) 2003, 2004 Phil Blundell
> # Copyright (C) 2009 Holger Hans Peter Freyther
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. -
>
> import re
> import string
> diff --git a/bitbake/lib/bb/parse/parse_py/BBHandler.py
> b/bitbake/lib/bb/parse/parse_py/BBHandler.py index 01fc47e..6f7cf82
> 100644 --- a/bitbake/lib/bb/parse/parse_py/BBHandler.py
> +++ b/bitbake/lib/bb/parse/parse_py/BBHandler.py
> @@ -1,6 +1,3 @@
> -#!/usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> class for handling .bb files
>
> @@ -12,19 +9,8 @@
> # Copyright (C) 2003, 2004 Chris Larson
> # Copyright (C) 2003, 2004 Phil Blundell
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. -
>
> import re, bb, os
> import logging
> @@ -38,14 +24,15 @@ from .ConfHandler import include, init
> # For compatibility
> bb.deprecate_import(__name__, "bb.parse", ["vars_from_file"])
>
> -__func_start_regexp__ =
> re.compile( r"(((?P<py>python)|(?P<fr>fakeroot))\s*)*(?P<func>[\w\.\-\+\{\}\$]+)?\s*\(\s*\)\s*{$" )
> -__inherit_regexp__ = re.compile( r"inherit\s+(.+)" )
> -__export_func_regexp__ = re.compile( r"EXPORT_FUNCTIONS\s+(.+)" )
> -__addtask_regexp__ =
> re.compile("addtask\s+(?P<func>\w+)\s*((before\s*(?P<before>((.*(?=after))|(.*))))|(after\s*(?P<after>((.*(?=before))|(.*)))))*")
> -__deltask_regexp__ = re.compile("deltask\s+(?P<func>\w+)")
> -__addhandler_regexp__ = re.compile( r"addhandler\s+(.+)" )
> -__def_regexp__ = re.compile( r"def\s+(\w+).*:" )
> -__python_func_regexp__ = re.compile( r"(\s+.*)|(^$)|(^#)" )
> +__func_start_regexp__ =
> re.compile(r"(((?P<py>python)|(?P<fr>fakeroot))\s*)*(?P<func>[\w\.\-\+\{\}\$]+)?\s*\(\s*\)\s*{$" )
> +__inherit_regexp__ = re.compile(r"inherit\s+(.+)" )
> +__export_func_regexp__ = re.compile(r"EXPORT_FUNCTIONS\s+(.+)" )
> +__addtask_regexp__ =
> re.compile(r"addtask\s+(?P<func>\w+)\s*((before\s*(?P<before>((.*(?=after))|(.*))))|(after\s*(?P<after>((.*(?=before))|(.*)))))*")
> +__deltask_regexp__ =
> re.compile(r"deltask\s+(?P<func>\w+)(?P<ignores>.*)")
> +__addhandler_regexp__ = re.compile(r"addhandler\s+(.+)" )
> +__def_regexp__ = re.compile(r"def\s+(\w+).*:" )
> +__python_func_regexp__ = re.compile(r"(\s+.*)|(^$)|(^#)" )
> +__python_tab_regexp__ = re.compile(r" *\t") __infunc__ = []
> __inpython__ = False @@ -160,6 +147,16 @@ def handle(fn, d, include):
> def feeder(lineno, s, fn, root, statements, eof=False): global
> __func_start_regexp__, __inherit_regexp__, __export_func_regexp__,
> __addtask_regexp__, __addhandler_regexp__, __def_regexp__,
> __python_func_regexp__, __inpython__, __infunc__, __body__, bb,
> __residue__, __classname__ +
> + # Check tabs in python functions:
> + # - def py_funcname(): covered by __inpython__
> + # - python(): covered by '__anonymous' == __infunc__[0]
> + # - python funcname(): covered by __infunc__[3]
> + if __inpython__ or (__infunc__ and ('__anonymous' ==
> __infunc__[0] or __infunc__[3])):
> + tab = __python_tab_regexp__.match(s)
> + if tab:
> + bb.warn('python should use 4 spaces indentation, but
> found tabs in %s, line %s' % (root, lineno)) +
> if __infunc__:
> if s == '}':
> __body__.append('')
> @@ -225,11 +222,27 @@ def feeder(lineno, s, fn, root, statements,
> eof=False):
> m = __addtask_regexp__.match(s)
> if m:
> + if len(m.group().split()) == 2:
> + # Check and warn for "addtask task1 task2"
> + m2 = re.match(r"addtask\s+(?P<func>\w+)(?P<ignores>.*)",
> s)
> + if m2 and m2.group('ignores'):
> + logger.warning('addtask ignored: "%s"' %
> m2.group('ignores')) +
> + # Check and warn for "addtask task1 before task2 before
> task3", the
> + # similar to "after"
> + taskexpression = s.split()
> + for word in ('before', 'after'):
> + if taskexpression.count(word) > 1:
> + logger.warning("addtask contained multiple '%s'
> keywords, only one is supported" % word) +
> ast.handleAddTask(statements, fn, lineno, m)
> return
>
> m = __deltask_regexp__.match(s)
> if m:
> + # Check and warn "for deltask task1 task2"
> + if m.group('ignores'):
> + logger.warning('deltask ignored: "%s"' %
> m.group('ignores')) ast.handleDelTask(statements, fn, lineno, m)
> return
>
> diff --git a/bitbake/lib/bb/parse/parse_py/ConfHandler.py
> b/bitbake/lib/bb/parse/parse_py/ConfHandler.py index 9d3ebe1..2e84b91
> 100644 --- a/bitbake/lib/bb/parse/parse_py/ConfHandler.py
> +++ b/bitbake/lib/bb/parse/parse_py/ConfHandler.py
> @@ -1,6 +1,3 @@
> -#!/usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> class for handling configuration data files
>
> @@ -11,18 +8,8 @@
> # Copyright (C) 2003, 2004 Chris Larson
> # Copyright (C) 2003, 2004 Phil Blundell
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import errno
> import re
> @@ -147,7 +134,7 @@ def handle(fn, data, include):
> continue
> s = s.rstrip()
> while s[-1] == '\\':
> - s2 = f.readline().strip()
> + s2 = f.readline().rstrip()
> lineno = lineno + 1
> if (not s2 or s2 and s2[0] != "#") and s[0] == "#" :
> bb.fatal("There is a confusing multiline, partially
> commented expression on line %s of file %s (%s).\nPlease clarify
> whether this is all a comment or should be parsed." % (lineno, fn,
> s)) diff --git a/bitbake/lib/bb/parse/parse_py/__init__.py
> b/bitbake/lib/bb/parse/parse_py/__init__.py index 3e658d0..f508afa
> 100644 --- a/bitbake/lib/bb/parse/parse_py/__init__.py +++
> b/bitbake/lib/bb/parse/parse_py/__init__.py @@ -1,6 +1,3 @@
> -#!/usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake Parsers
>
> @@ -11,20 +8,10 @@ File parsers for the BitBake build tools.
> # Copyright (C) 2003, 2004 Chris Larson
> # Copyright (C) 2003, 2004 Phil Blundell
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
> # Based on functions from the base bb module, Copyright 2003 Holger
> Schurig +#
>
> from __future__ import absolute_import
> from . import ConfHandler
> diff --git a/bitbake/lib/bb/persist_data.py
> b/bitbake/lib/bb/persist_data.py index bef7018..de8f87a 100644
> --- a/bitbake/lib/bb/persist_data.py
> +++ b/bitbake/lib/bb/persist_data.py
> @@ -8,18 +8,8 @@ currently, providing a key/value store accessed by
> 'domain'. # Copyright (C) 2007 Richard Purdie
> # Copyright (C) 2010 Chris Larson <chris_larson@mentor.com>
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import collections
> import logging
> @@ -29,6 +19,7 @@ import warnings
> from bb.compat import total_ordering
> from collections import Mapping
> import sqlite3
> +import contextlib
>
> sqlversion = sqlite3.sqlite_version_info
> if sqlversion[0] < 3 or (sqlversion[0] == 3 and sqlversion[1] < 3):
> @@ -36,84 +27,181 @@ if sqlversion[0] < 3 or (sqlversion[0] == 3 and
> sqlversion[1] < 3):
>
> logger = logging.getLogger("BitBake.PersistData")
> -if hasattr(sqlite3, 'enable_shared_cache'):
> - try:
> - sqlite3.enable_shared_cache(True)
> - except sqlite3.OperationalError:
> - pass
> -
>
> @total_ordering
> class SQLTable(collections.MutableMapping):
> + class _Decorators(object):
> + @staticmethod
> + def retry(*, reconnect=True):
> + """
> + Decorator that restarts a function if a database locked
> sqlite
> + exception occurs. If reconnect is True, the database
> connection
> + will be closed and reopened each time a failure occurs
> + """
> + def retry_wrapper(f):
> + def wrap_func(self, *args, **kwargs):
> + # Reconnect if necessary
> + if self.connection is None and reconnect:
> + self.reconnect()
> +
> + count = 0
> + while True:
> + try:
> + return f(self, *args, **kwargs)
> + except sqlite3.OperationalError as exc:
> + if count < 500 and ('is locked' in
> str(exc) or 'locking protocol' in str(exc)):
> + count = count + 1
> + if reconnect:
> + self.reconnect()
> + continue
> + raise
> + return wrap_func
> + return retry_wrapper
> +
> + @staticmethod
> + def transaction(f):
> + """
> + Decorator that starts a database transaction and creates
> a database
> + cursor for performing queries. If no exception is
> thrown, the
> + database results are commited. If an exception occurs,
> the database
> + is rolled back. In all cases, the cursor is closed after
> the
> + function ends.
> +
> + Note that the cursor is passed as an extra argument to
> the function
> + after `self` and before any of the normal arguments
> + """
> + def wrap_func(self, *args, **kwargs):
> + # Context manager will COMMIT the database on
> success,
> + # or ROLLBACK on an exception
> + with self.connection:
> + # Automatically close the cursor when done
> + with
> contextlib.closing(self.connection.cursor()) as cursor:
> + return f(self, cursor, *args, **kwargs)
> + return wrap_func
> +
> """Object representing a table/domain in the database"""
> def __init__(self, cachefile, table):
> self.cachefile = cachefile
> self.table = table
> - self.cursor = connect(self.cachefile)
> -
> - self._execute("CREATE TABLE IF NOT EXISTS %s(key TEXT, value
> TEXT);"
> - % table)
> -
> - def _execute(self, *query):
> - """Execute a query, waiting to acquire a lock if necessary"""
> - count = 0
> - while True:
> - try:
> - return self.cursor.execute(*query)
> - except sqlite3.OperationalError as exc:
> - if 'database is locked' in str(exc) and count < 500:
> - count = count + 1
> +
> + self.connection = None
> + self._execute_single("CREATE TABLE IF NOT EXISTS %s(key TEXT
> PRIMARY KEY NOT NULL, value TEXT);" % table) +
> + @_Decorators.retry(reconnect=False)
> + @_Decorators.transaction
> + def _setup_database(self, cursor):
> + cursor.execute("pragma synchronous = off;")
> + # Enable WAL and keep the autocheckpoint length small (the
> default is
> + # usually 1000). Persistent caches are usually read-mostly,
> so keeping
> + # this short will keep readers running quickly
> + cursor.execute("pragma journal_mode = WAL;")
> + cursor.execute("pragma wal_autocheckpoint = 100;")
> +
> + def reconnect(self):
> + if self.connection is not None:
> + self.connection.close()
> + self.connection = sqlite3.connect(self.cachefile, timeout=5)
> + self.connection.text_factory = str
> + self._setup_database()
> +
> + @_Decorators.retry()
> + @_Decorators.transaction
> + def _execute_single(self, cursor, *query):
> + """
> + Executes a single query and discards the results. This
> correctly closes
> + the database cursor when finished
> + """
> + cursor.execute(*query)
> +
> + @_Decorators.retry()
> + def _row_iter(self, f, *query):
> + """
> + Helper function that returns a row iterator. Each time
> __next__ is
> + called on the iterator, the provided function is evaluated
> to determine
> + the return value
> + """
> + class CursorIter(object):
> + def __init__(self, cursor):
> + self.cursor = cursor
> +
> + def __iter__(self):
> + return self
> +
> + def __next__(self):
> + row = self.cursor.fetchone()
> + if row is None:
> self.cursor.close()
> - self.cursor = connect(self.cachefile)
> - continue
> - raise
> + raise StopIteration
> + return f(row)
> +
> + def __enter__(self):
> + return self
> +
> + def __exit__(self, typ, value, traceback):
> + self.cursor.close()
> + return False
> +
> + cursor = self.connection.cursor()
> + try:
> + cursor.execute(*query)
> + return CursorIter(cursor)
> + except:
> + cursor.close()
>
> def __enter__(self):
> - self.cursor.__enter__()
> + self.connection.__enter__()
> return self
>
> def __exit__(self, *excinfo):
> - self.cursor.__exit__(*excinfo)
> -
> - def __getitem__(self, key):
> - data = self._execute("SELECT * from %s where key=?;" %
> - self.table, [key])
> - for row in data:
> + self.connection.__exit__(*excinfo)
> +
> + @_Decorators.retry()
> + @_Decorators.transaction
> + def __getitem__(self, cursor, key):
> + cursor.execute("SELECT * from %s where key=?;" % self.table,
> [key])
> + row = cursor.fetchone()
> + if row is not None:
> return row[1]
> raise KeyError(key)
>
> - def __delitem__(self, key):
> + @_Decorators.retry()
> + @_Decorators.transaction
> + def __delitem__(self, cursor, key):
> if key not in self:
> raise KeyError(key)
> - self._execute("DELETE from %s where key=?;" % self.table,
> [key])
> + cursor.execute("DELETE from %s where key=?;" % self.table,
> [key])
> - def __setitem__(self, key, value):
> + @_Decorators.retry()
> + @_Decorators.transaction
> + def __setitem__(self, cursor, key, value):
> if not isinstance(key, str):
> raise TypeError('Only string keys are supported')
> elif not isinstance(value, str):
> raise TypeError('Only string values are supported')
>
> - data = self._execute("SELECT * from %s where key=?;" %
> - self.table, [key])
> - exists = len(list(data))
> - if exists:
> - self._execute("UPDATE %s SET value=? WHERE key=?;" %
> self.table,
> - [value, key])
> + cursor.execute("SELECT * from %s where key=?;" % self.table,
> [key])
> + row = cursor.fetchone()
> + if row is not None:
> + cursor.execute("UPDATE %s SET value=? WHERE key=?;" %
> self.table, [value, key]) else:
> - self._execute("INSERT into %s(key, value) values
> (?, ?);" %
> - self.table, [key, value])
> -
> - def __contains__(self, key):
> - return key in set(self)
> -
> - def __len__(self):
> - data = self._execute("SELECT COUNT(key) FROM %s;" %
> self.table)
> - for row in data:
> + cursor.execute("INSERT into %s(key, value) values
> (?, ?);" % self.table, [key, value]) +
> + @_Decorators.retry()
> + @_Decorators.transaction
> + def __contains__(self, cursor, key):
> + cursor.execute('SELECT * from %s where key=?;' % self.table,
> [key])
> + return cursor.fetchone() is not None
> +
> + @_Decorators.retry()
> + @_Decorators.transaction
> + def __len__(self, cursor):
> + cursor.execute("SELECT COUNT(key) FROM %s;" % self.table)
> + row = cursor.fetchone()
> + if row is not None:
> return row[0]
>
> def __iter__(self):
> - data = self._execute("SELECT key FROM %s;" % self.table)
> - return (row[0] for row in data)
> + return self._row_iter(lambda row: row[0], "SELECT key from
> %s;" % self.table)
> def __lt__(self, other):
> if not isinstance(other, Mapping):
> @@ -122,25 +210,27 @@ class SQLTable(collections.MutableMapping):
> return len(self) < len(other)
>
> def get_by_pattern(self, pattern):
> - data = self._execute("SELECT * FROM %s WHERE key LIKE ?;" %
> - self.table, [pattern])
> - return [row[1] for row in data]
> + return self._row_iter(lambda row: row[1], "SELECT * FROM %s
> WHERE key LIKE ?;" %
> + self.table, [pattern])
>
> def values(self):
> return list(self.itervalues())
>
> def itervalues(self):
> - data = self._execute("SELECT value FROM %s;" % self.table)
> - return (row[0] for row in data)
> + return self._row_iter(lambda row: row[0], "SELECT value FROM
> %s;" %
> + self.table)
>
> def items(self):
> return list(self.iteritems())
>
> def iteritems(self):
> - return self._execute("SELECT * FROM %s;" % self.table)
> + return self._row_iter(lambda row: (row[0], row[1]), "SELECT
> * FROM %s;" %
> + self.table)
>
> - def clear(self):
> - self._execute("DELETE FROM %s;" % self.table)
> + @_Decorators.retry()
> + @_Decorators.transaction
> + def clear(self, cursor):
> + cursor.execute("DELETE FROM %s;" % self.table)
>
> def has_key(self, key):
> return key in self
> @@ -194,12 +284,6 @@ class PersistData(object):
> """
> del self.data[domain][key]
>
> -def connect(database):
> - connection = sqlite3.connect(database, timeout=5,
> isolation_level=None)
> - connection.execute("pragma synchronous = off;")
> - connection.text_factory = str
> - return connection
> -
> def persist(domain, d):
> """Convenience factory for SQLTable objects based upon
> metadata""" import bb.utils
> diff --git a/bitbake/lib/bb/process.py b/bitbake/lib/bb/process.py
> index e69697c..2dc472a 100644
> --- a/bitbake/lib/bb/process.py
> +++ b/bitbake/lib/bb/process.py
> @@ -1,3 +1,7 @@
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> import logging
> import signal
> import subprocess
> diff --git a/bitbake/lib/bb/progress.py b/bitbake/lib/bb/progress.py
> index f54d1c7..4022caa 100644
> --- a/bitbake/lib/bb/progress.py
> +++ b/bitbake/lib/bb/progress.py
> @@ -4,18 +4,8 @@ BitBake progress handling code
>
> # Copyright (C) 2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import sys
> import re
> @@ -23,6 +13,7 @@ import time
> import inspect
> import bb.event
> import bb.build
> +from bb.build import StdoutNoopContextManager
>
> class ProgressHandler(object):
> """
> @@ -37,7 +28,14 @@ class ProgressHandler(object):
> if outfile:
> self._outfile = outfile
> else:
> - self._outfile = sys.stdout
> + self._outfile = StdoutNoopContextManager()
> +
> + def __enter__(self):
> + self._outfile.__enter__()
> + return self
> +
> + def __exit__(self, *excinfo):
> + self._outfile.__exit__(*excinfo)
>
> def _fire_progress(self, taskprogress, rate=None):
> """Internal function to fire the progress event"""
> @@ -157,6 +155,12 @@ class MultiStageProgressReporter(object):
> self._stage_total = None
> self._callers = []
>
> + def __enter__(self):
> + return self
> +
> + def __exit__(self, *excinfo):
> + pass
> +
> def _fire_progress(self, taskprogress):
> bb.event.fire(bb.build.TaskProgress(taskprogress),
> self._data)
> diff --git a/bitbake/lib/bb/providers.py b/bitbake/lib/bb/providers.py
> index c2aa98c..f80963c 100644
> --- a/bitbake/lib/bb/providers.py
> +++ b/bitbake/lib/bb/providers.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # Copyright (C) 2003, 2004 Chris Larson
> # Copyright (C) 2003, 2004 Phil Blundell
> @@ -8,18 +6,8 @@
> # Copyright (C) 2005 ROAD GmbH
> # Copyright (C) 2006 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import re
> import logging
> @@ -129,7 +117,7 @@ def findPreferredProvider(pn, cfgData, dataCache,
> pkg_pn = None, item = None): preferred_v =
> cfgData.getVar("PREFERRED_VERSION")
> if preferred_v:
> - m = re.match('(\d+:)*(.*)(_.*)*', preferred_v)
> + m = re.match(r'(\d+:)*(.*)(_.*)*', preferred_v)
> if m:
> if m.group(1):
> preferred_e = m.group(1)[:-1]
> @@ -384,7 +372,7 @@ def getRuntimeProviders(dataCache, rdepend):
>
> # Only search dynamic packages if we can't find anything in
> other variables for pattern in dataCache.packages_dynamic:
> - pattern = pattern.replace('+', "\+")
> + pattern = pattern.replace(r'+', r"\+")
> if pattern in regexp_cache:
> regexp = regexp_cache[pattern]
> else:
> diff --git a/bitbake/lib/bb/pysh/builtin.py
> b/bitbake/lib/bb/pysh/builtin.py deleted file mode 100644
> index a8814dc..0000000
> --- a/bitbake/lib/bb/pysh/builtin.py
> +++ /dev/null
> @@ -1,710 +0,0 @@
> -# builtin.py - builtins and utilities definitions for pysh.
> -#
> -# Copyright 2007 Patrick Mezard
> -#
> -# This software may be used and distributed according to the terms
> -# of the GNU General Public License, incorporated herein by
> reference. -
> -"""Builtin and internal utilities implementations.
> -
> -- Beware not to use python interpreter environment as if it were the
> shell -environment. For instance, commands working directory must be
> explicitely handled -through env['PWD'] instead of relying on python
> working directory. -"""
> -import errno
> -import optparse
> -import os
> -import re
> -import subprocess
> -import sys
> -import time
> -
> -def has_subprocess_bug():
> - return getattr(subprocess, 'list2cmdline') and \
> - ( subprocess.list2cmdline(['']) == '' or \
> - subprocess.list2cmdline(['foo|bar']) == 'foo|bar')
> -
> -# Detect python bug 1634343: "subprocess swallows empty arguments
> under win32" -#
> <http://sourceforge.net/tracker/index.php?func=detail&aid=1634343&group_id=5470&atid=105470>
> -# Also detect: "[ 1710802 ] subprocess must escape redirection
> characters under win32" -#
> <http://sourceforge.net/tracker/index.php?func=detail&aid=1710802&group_id=5470&atid=105470>
> -if has_subprocess_bug():
> - import subprocess_fix
> - subprocess.list2cmdline = subprocess_fix.list2cmdline
> -
> -from sherrors import *
> -
> -class NonExitingParser(optparse.OptionParser):
> - """OptionParser default behaviour upon error is to print the
> error message and
> - exit. Raise a utility error instead.
> - """
> - def error(self, msg):
> - raise UtilityError(msg)
> -
> -#-------------------------------------------------------------------------------
> -# set special builtin
> -#-------------------------------------------------------------------------------
> -OPT_SET = NonExitingParser(usage="set - set or unset options and
> positional parameters") -OPT_SET.add_option( '-f',
> action='store_true', dest='has_f', default=False,
> - help='The shell shall disable pathname expansion.')
> -OPT_SET.add_option('-e', action='store_true', dest='has_e',
> default=False,
> - help="""When this option is on, if a simple command fails for
> any of the \
> - reasons listed in Consequences of Shell Errors or returns an
> exit status \
> - value >0, and is not part of the compound list following a
> while, until, \
> - or if keyword, and is not a part of an AND or OR list, and is
> not a \
> - pipeline preceded by the ! reserved word, then the shell shall
> immediately \
> - exit.""")
> -OPT_SET.add_option('-x', action='store_true', dest='has_x',
> default=False,
> - help="""The shell shall write to standard error a trace for each
> command \
> - after it expands the command and before it executes it. It is
> unspecified \
> - whether the command that turns tracing off is traced.""")
> -
> -def builtin_set(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n') -
> - option, args = OPT_SET.parse_args(args)
> - env = interp.get_env()
> -
> - if option.has_f:
> - env.set_opt('-f')
> - if option.has_e:
> - env.set_opt('-e')
> - if option.has_x:
> - env.set_opt('-x')
> - return 0
> -
> -#-------------------------------------------------------------------------------
> -# shift special builtin
> -#-------------------------------------------------------------------------------
> -def builtin_shift(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> -
> - params = interp.get_env().get_positional_args()
> - if args:
> - try:
> - n = int(args[0])
> - if n > len(params):
> - raise ValueError()
> - except ValueError:
> - return 1
> - else:
> - n = 1
> -
> - params[:n] = []
> - interp.get_env().set_positional_args(params)
> - return 0
> -
> -#-------------------------------------------------------------------------------
> -# export special builtin
> -#-------------------------------------------------------------------------------
> -OPT_EXPORT = NonExitingParser(usage="set - set or unset options and
> positional parameters") -OPT_EXPORT.add_option('-p',
> action='store_true', dest='has_p', default=False) -
> -def builtin_export(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> -
> - option, args = OPT_EXPORT.parse_args(args)
> - if option.has_p:
> - raise NotImplementedError()
> -
> - for arg in args:
> - try:
> - name, value = arg.split('=', 1)
> - except ValueError:
> - name, value = arg, None
> - env = interp.get_env().export(name, value)
> -
> - return 0
> -
> -#-------------------------------------------------------------------------------
> -# return special builtin
> -#-------------------------------------------------------------------------------
> -def builtin_return(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> - res = 0
> - if args:
> - try:
> - res = int(args[0])
> - except ValueError:
> - res = 0
> - if not 0<=res<=255:
> - res = 0
> -
> - # BUG: should be last executed command exit code
> - raise ReturnSignal(res)
> -
> -#-------------------------------------------------------------------------------
> -# trap special builtin
> -#-------------------------------------------------------------------------------
> -def builtin_trap(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> - if len(args) < 2:
> - stderr.write('trap: usage: trap [[arg] signal_spec ...]\n')
> - return 2
> -
> - action = args[0]
> - for sig in args[1:]:
> - try:
> - env.traps[sig] = action
> - except Exception as e:
> - stderr.write('trap: %s\n' % str(e))
> - return 0
> -
> -#-------------------------------------------------------------------------------
> -# unset special builtin
> -#-------------------------------------------------------------------------------
> -OPT_UNSET = NonExitingParser("unset - unset values and attributes of
> variables and functions") -OPT_UNSET.add_option( '-f',
> action='store_true', dest='has_f', default=False)
> -OPT_UNSET.add_option( '-v', action='store_true', dest='has_v',
> default=False) - -def builtin_unset(name, args, interp, env, stdin,
> stdout, stderr, debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> -
> - option, args = OPT_UNSET.parse_args(args)
> -
> - status = 0
> - env = interp.get_env()
> - for arg in args:
> - try:
> - if option.has_f:
> - env.remove_function(arg)
> - else:
> - del env[arg]
> - except KeyError:
> - pass
> - except VarAssignmentError:
> - status = 1
> -
> - return status
> -
> -#-------------------------------------------------------------------------------
> -# wait special builtin
> -#-------------------------------------------------------------------------------
> -def builtin_wait(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n') -
> - return interp.wait([int(arg) for arg in args])
> -
> -#-------------------------------------------------------------------------------
> -# cat utility
> -#-------------------------------------------------------------------------------
> -def utility_cat(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n') -
> - if not args:
> - args = ['-']
> -
> - status = 0
> - for arg in args:
> - if arg == '-':
> - data = stdin.read()
> - else:
> - path = os.path.join(env['PWD'], arg)
> - try:
> - f = file(path, 'rb')
> - try:
> - data = f.read()
> - finally:
> - f.close()
> - except IOError as e:
> - if e.errno != errno.ENOENT:
> - raise
> - status = 1
> - continue
> - stdout.write(data)
> - stdout.flush()
> - return status
> -
> -#-------------------------------------------------------------------------------
> -# cd utility
> -#-------------------------------------------------------------------------------
> -OPT_CD = NonExitingParser("cd - change the working directory")
> -
> -def utility_cd(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n') -
> - option, args = OPT_CD.parse_args(args)
> - env = interp.get_env()
> -
> - directory = None
> - printdir = False
> - if not args:
> - home = env.get('HOME')
> - if home:
> - # Unspecified, do nothing
> - return 0
> - else:
> - directory = home
> - elif len(args)==1:
> - directory = args[0]
> - if directory=='-':
> - if 'OLDPWD' not in env:
> - raise UtilityError("OLDPWD not set")
> - printdir = True
> - directory = env['OLDPWD']
> - else:
> - raise UtilityError("too many arguments")
> -
> - curpath = None
> - # Absolute directories will be handled correctly by the
> os.path.join call.
> - if not directory.startswith('.') and not
> directory.startswith('..'):
> - cdpaths = env.get('CDPATH', '.').split(';')
> - for cdpath in cdpaths:
> - p = os.path.join(cdpath, directory)
> - if os.path.isdir(p):
> - curpath = p
> - break
> -
> - if curpath is None:
> - curpath = directory
> - curpath = os.path.join(env['PWD'], directory)
> -
> - env['OLDPWD'] = env['PWD']
> - env['PWD'] = curpath
> - if printdir:
> - stdout.write('%s\n' % curpath)
> - return 0
> -
> -#-------------------------------------------------------------------------------
> -# colon utility
> -#-------------------------------------------------------------------------------
> -def utility_colon(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> - return 0
> -
> -#-------------------------------------------------------------------------------
> -# echo utility
> -#-------------------------------------------------------------------------------
> -def utility_echo(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> -
> - # Echo only takes arguments, no options. Use printf if you need
> fancy stuff.
> - output = ' '.join(args) + '\n'
> - stdout.write(output)
> - stdout.flush()
> - return 0
> -
> -#-------------------------------------------------------------------------------
> -# egrep utility
> -#-------------------------------------------------------------------------------
> -# egrep is usually a shell script.
> -# Unfortunately, pysh does not support shell scripts *with
> arguments* right now, -# so the redirection is implemented here,
> assuming grep is available. -def utility_egrep(name, args, interp,
> env, stdin, stdout, stderr, debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> -
> - return run_command('grep', ['-E'] + args, interp, env, stdin,
> stdout,
> - stderr, debugflags)
> -
> -#-------------------------------------------------------------------------------
> -# env utility
> -#-------------------------------------------------------------------------------
> -def utility_env(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> -
> - if args and args[0]=='-i':
> - raise NotImplementedError('env: -i option is not
> implemented')
> -
> - i = 0
> - for arg in args:
> - if '=' not in arg:
> - break
> - # Update the current environment
> - name, value = arg.split('=', 1)
> - env[name] = value
> - i += 1
> -
> - if args[i:]:
> - # Find then execute the specified interpreter
> - utility = env.find_in_path(args[i])
> - if not utility:
> - return 127
> - args[i:i+1] = utility
> - name = args[i]
> - args = args[i+1:]
> - try:
> - return run_command(name, args, interp, env, stdin,
> stdout, stderr,
> - debugflags)
> - except UtilityError:
> - stderr.write('env: failed to execute %s' % '
> '.join([name]+args))
> - return 126
> - else:
> - for pair in env.get_variables().iteritems():
> - stdout.write('%s=%s\n' % pair)
> - return 0
> -
> -#-------------------------------------------------------------------------------
> -# exit utility
> -#-------------------------------------------------------------------------------
> -def utility_exit(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> -
> - res = None
> - if args:
> - try:
> - res = int(args[0])
> - except ValueError:
> - res = None
> - if not 0<=res<=255:
> - res = None
> -
> - if res is None:
> - # BUG: should be last executed command exit code
> - res = 0
> -
> - raise ExitSignal(res)
> -
> -#-------------------------------------------------------------------------------
> -# fgrep utility
> -#-------------------------------------------------------------------------------
> -# see egrep
> -def utility_fgrep(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> -
> - return run_command('grep', ['-F'] + args, interp, env, stdin,
> stdout,
> - stderr, debugflags)
> -
> -#-------------------------------------------------------------------------------
> -# gunzip utility
> -#-------------------------------------------------------------------------------
> -# see egrep
> -def utility_gunzip(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> -
> - return run_command('gzip', ['-d'] + args, interp, env, stdin,
> stdout,
> - stderr, debugflags)
> -
> -#-------------------------------------------------------------------------------
> -# kill utility
> -#-------------------------------------------------------------------------------
> -def utility_kill(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> -
> - for arg in args:
> - pid = int(arg)
> - status = subprocess.call(['pskill', '/T', str(pid)],
> - shell=True,
> - stdout=subprocess.PIPE,
> - stderr=subprocess.PIPE)
> - # pskill is asynchronous, hence the stupid polling loop
> - while 1:
> - p = subprocess.Popen(['pslist', str(pid)],
> - shell=True,
> - stdout=subprocess.PIPE,
> - stderr=subprocess.STDOUT)
> - output = p.communicate()[0]
> - if ('process %d was not' % pid) in output:
> - break
> - time.sleep(1)
> - return status
> -
> -#-------------------------------------------------------------------------------
> -# mkdir utility
> -#-------------------------------------------------------------------------------
> -OPT_MKDIR = NonExitingParser("mkdir - make directories.")
> -OPT_MKDIR.add_option('-p', action='store_true', dest='has_p',
> default=False) -
> -def utility_mkdir(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> -
> - # TODO: implement umask
> - # TODO: implement proper utility error report
> - option, args = OPT_MKDIR.parse_args(args)
> - for arg in args:
> - path = os.path.join(env['PWD'], arg)
> - if option.has_p:
> - try:
> - os.makedirs(path)
> - except IOError as e:
> - if e.errno != errno.EEXIST:
> - raise
> - else:
> - os.mkdir(path)
> - return 0
> -
> -#-------------------------------------------------------------------------------
> -# netstat utility
> -#-------------------------------------------------------------------------------
> -def utility_netstat(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - # Do you really expect me to implement netstat ?
> - # This empty form is enough for Mercurial tests since it's
> - # supposed to generate nothing upon success. Faking this test
> - # is not a big deal either.
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> - return 0
> -
> -#-------------------------------------------------------------------------------
> -# pwd utility
> -#-------------------------------------------------------------------------------
> -OPT_PWD = NonExitingParser("pwd - return working directory name")
> -OPT_PWD.add_option('-L', action='store_true', dest='has_L',
> default=True,
> - help="""If the PWD environment variable contains an absolute
> pathname of \
> - the current directory that does not contain the filenames dot or
> dot-dot, \
> - pwd shall write this pathname to standard output. Otherwise, the
> -L option \
> - shall behave as the -P option.""")
> -OPT_PWD.add_option('-P', action='store_true', dest='has_L',
> default=False,
> - help="""The absolute pathname written shall not contain
> filenames that, in \
> - the context of the pathname, refer to files of type symbolic
> link.""") -
> -def utility_pwd(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n') -
> - option, args = OPT_PWD.parse_args(args)
> - stdout.write('%s\n' % env['PWD'])
> - return 0
> -
> -#-------------------------------------------------------------------------------
> -# printf utility
> -#-------------------------------------------------------------------------------
> -RE_UNESCAPE = re.compile(r'(\\x[a-zA-Z0-9]{2}|\\[0-7]{1,3}|\\.)')
> -
> -def utility_printf(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> -
> - def replace(m):
> - assert m.group()
> - g = m.group()[1:]
> - if g.startswith('x'):
> - return chr(int(g[1:], 16))
> - if len(g) <= 3 and len([c for c in g if c in '01234567']) ==
> len(g):
> - # Yay, an octal number
> - return chr(int(g, 8))
> - return {
> - 'a': '\a',
> - 'b': '\b',
> - 'f': '\f',
> - 'n': '\n',
> - 'r': '\r',
> - 't': '\t',
> - 'v': '\v',
> - '\\': '\\',
> - }.get(g)
> -
> - # Convert escape sequences
> - format = re.sub(RE_UNESCAPE, replace, args[0])
> - stdout.write(format % tuple(args[1:]))
> - return 0
> -
> -#-------------------------------------------------------------------------------
> -# true utility
> -#-------------------------------------------------------------------------------
> -def utility_true(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> - return 0
> -
> -#-------------------------------------------------------------------------------
> -# sed utility
> -#-------------------------------------------------------------------------------
> -RE_SED = re.compile(r'^s(.).*\1[a-zA-Z]*$')
> -
> -# cygwin sed fails with some expressions when they do not end with a
> single space. -# see unit tests for details. Interestingly, the same
> expressions works perfectly -# in cygwin shell.
> -def utility_sed(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> -
> - # Scan pattern arguments and append a space if necessary
> - for i in range(len(args)):
> - if not RE_SED.search(args[i]):
> - continue
> - args[i] = args[i] + ' '
> -
> - return run_command(name, args, interp, env, stdin, stdout,
> - stderr, debugflags)
> -
> -#-------------------------------------------------------------------------------
> -# sleep utility
> -#-------------------------------------------------------------------------------
> -def utility_sleep(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> - time.sleep(int(args[0]))
> - return 0
> -
> -#-------------------------------------------------------------------------------
> -# sort utility
> -#-------------------------------------------------------------------------------
> -OPT_SORT = NonExitingParser("sort - sort, merge, or sequence check
> text files") -
> -def utility_sort(name, args, interp, env, stdin, stdout, stderr,
> debugflags): -
> - def sort(path):
> - if path == '-':
> - lines = stdin.readlines()
> - else:
> - try:
> - f = file(path)
> - try:
> - lines = f.readlines()
> - finally:
> - f.close()
> - except IOError as e:
> - stderr.write(str(e) + '\n')
> - return 1
> -
> - if lines and lines[-1][-1]!='\n':
> - lines[-1] = lines[-1] + '\n'
> - return lines
> -
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> -
> - option, args = OPT_SORT.parse_args(args)
> - alllines = []
> -
> - if len(args)<=0:
> - args += ['-']
> -
> - # Load all files lines
> - curdir = os.getcwd()
> - try:
> - os.chdir(env['PWD'])
> - for path in args:
> - alllines += sort(path)
> - finally:
> - os.chdir(curdir)
> -
> - alllines.sort()
> - for line in alllines:
> - stdout.write(line)
> - return 0
> -
> -#-------------------------------------------------------------------------------
> -# hg utility
> -#-------------------------------------------------------------------------------
> -
> -hgcommands = [
> - 'add',
> - 'addremove',
> - 'commit', 'ci',
> - 'debugrename',
> - 'debugwalk',
> - 'falabala', # Dummy command used in a mercurial test
> - 'incoming',
> - 'locate',
> - 'pull',
> - 'push',
> - 'qinit',
> - 'remove', 'rm',
> - 'rename', 'mv',
> - 'revert',
> - 'showconfig',
> - 'status', 'st',
> - 'strip',
> - ]
> -
> -def rewriteslashes(name, args):
> - # Several hg commands output file paths, rewrite the separators
> - if len(args) > 1 and name.lower().endswith('python') \
> - and args[0].endswith('hg'):
> - for cmd in hgcommands:
> - if cmd in args[1:]:
> - return True
> -
> - # svn output contains many paths with OS specific separators.
> - # Normalize these to unix paths.
> - base = os.path.basename(name)
> - if base.startswith('svn'):
> - return True
> -
> - return False
> -
> -def rewritehg(output):
> - if not output:
> - return output
> - # Rewrite os specific messages
> - output = output.replace(': The system cannot find the file
> specified',
> - ': No such file or directory')
> - output = re.sub(': Access is denied.*$', ': Permission denied',
> output)
> - output = output.replace(': No connection could be made because
> the target machine actively refused it',
> - ': Connection refused')
> - return output
> -
> -
> -def run_command(name, args, interp, env, stdin, stdout,
> - stderr, debugflags):
> - # Execute the command
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n') -
> - hgbin = interp.options().hgbinary
> - ishg = hgbin and ('hg' in name or args and 'hg' in args[0])
> - unixoutput = 'cygwin' in name or ishg
> -
> - exec_env = env.get_variables()
> - try:
> - # BUG: comparing file descriptor is clearly not a reliable
> way to tell
> - # whether they point on the same underlying object. But in
> pysh limited
> - # scope this is usually right, we do not expect complicated
> redirections
> - # besides usual 2>&1.
> - # Still there is one case we have but cannot deal with is
> when stdout
> - # and stderr are redirected *by pysh caller*. This the
> reason for the
> - # --redirect pysh() option.
> - # Now, we want to know they are the same because we
> sometimes need to
> - # transform the command output, mostly remove CR-LF to
> ensure that
> - # command output is unix-like. Cygwin utilies are a special
> case because
> - # they explicitely set their output streams to binary mode,
> so we have
> - # nothing to do. For all others commands, we have to guess
> whether they
> - # are sending text data, in which case the transformation
> must be done.
> - # Again, the NUL character test is unreliable but should be
> enough for
> - # hg tests.
> - redirected = stdout.fileno()==stderr.fileno()
> - if not redirected:
> - p = subprocess.Popen([name] + args, cwd=env['PWD'],
> env=exec_env,
> - stdin=stdin, stdout=subprocess.PIPE,
> stderr=subprocess.PIPE)
> - else:
> - p = subprocess.Popen([name] + args, cwd=env['PWD'],
> env=exec_env,
> - stdin=stdin, stdout=subprocess.PIPE,
> stderr=subprocess.STDOUT)
> - out, err = p.communicate()
> - except WindowsError as e:
> - raise UtilityError(str(e))
> -
> - if not unixoutput:
> - def encode(s):
> - if '\0' in s:
> - return s
> - return s.replace('\r\n', '\n')
> - else:
> - encode = lambda s: s
> -
> - if rewriteslashes(name, args):
> - encode1_ = encode
> - def encode(s):
> - s = encode1_(s)
> - s = s.replace('\\\\', '\\')
> - s = s.replace('\\', '/')
> - return s
> -
> - if ishg:
> - encode2_ = encode
> - def encode(s):
> - return rewritehg(encode2_(s))
> -
> - stdout.write(encode(out))
> - if not redirected:
> - stderr.write(encode(err))
> - return p.returncode
> -
> diff --git a/bitbake/lib/bb/pysh/interp.py
> b/bitbake/lib/bb/pysh/interp.py deleted file mode 100644
> index d14ecf3..0000000
> --- a/bitbake/lib/bb/pysh/interp.py
> +++ /dev/null
> @@ -1,1367 +0,0 @@
> -# interp.py - shell interpreter for pysh.
> -#
> -# Copyright 2007 Patrick Mezard
> -#
> -# This software may be used and distributed according to the terms
> -# of the GNU General Public License, incorporated herein by
> reference. -
> -"""Implement the shell interpreter.
> -
> -Most references are made to "The Open Group Base Specifications
> Issue 6".
> -<http://www.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html>
> -""" -# TODO: document the fact input streams must implement fileno()
> so Popen will work correctly. -# it requires non-stdin stream to be
> implemented as files. Still to be tested... -# DOC: pathsep is used
> in PATH instead of ':'. Clearly, there are path syntax issues here.
> -# TODO: stop command execution upon error. -# TODO: sort out the
> filename/io_number mess. It should be possible to use filenames only.
> -# TODO: review subshell implementation -# TODO: test environment
> cloning for non-special builtins -# TODO: set -x should not rebuild
> commands from tokens, assignments/redirections are lost -# TODO: unit
> test for variable assignment -# TODO: test error management wrt error
> type/utility type -# TODO: test for binary output everywhere
> -# BUG: debug-parsing does not pass log file to PLY. Maybe a PLY
> upgrade is necessary. -import base64
> -import cPickle as pickle
> -import errno
> -import glob
> -import os
> -import re
> -import subprocess
> -import sys
> -import tempfile
> -
> -try:
> - s = set()
> - del s
> -except NameError:
> - from Set import Set as set
> -
> -import builtin
> -from sherrors import *
> -import pyshlex
> -import pyshyacc
> -
> -def mappend(func, *args, **kargs):
> - """Like map but assume func returns a list. Returned lists are
> merged into
> - a single one.
> - """
> - return reduce(lambda a,b: a+b, map(func, *args, **kargs), [])
> -
> -class FileWrapper:
> - """File object wrapper to ease debugging.
> -
> - Allow mode checking and implement file duplication through a
> simple
> - reference counting scheme. Not sure the latter is really useful
> since
> - only real file descriptors can be used.
> - """
> - def __init__(self, mode, file, close=True):
> - if mode not in ('r', 'w', 'a'):
> - raise IOError('invalid mode: %s' % mode)
> - self._mode = mode
> - self._close = close
> - if isinstance(file, FileWrapper):
> - if file._refcount[0] <= 0:
> - raise IOError(0, 'Error')
> - self._refcount = file._refcount
> - self._refcount[0] += 1
> - self._file = file._file
> - else:
> - self._refcount = [1]
> - self._file = file
> -
> - def dup(self):
> - return FileWrapper(self._mode, self, self._close)
> -
> - def fileno(self):
> - """fileno() should be only necessary for input streams."""
> - return self._file.fileno()
> -
> - def read(self, size=-1):
> - if self._mode!='r':
> - raise IOError(0, 'Error')
> - return self._file.read(size)
> -
> - def readlines(self, *args, **kwargs):
> - return self._file.readlines(*args, **kwargs)
> -
> - def write(self, s):
> - if self._mode not in ('w', 'a'):
> - raise IOError(0, 'Error')
> - return self._file.write(s)
> -
> - def flush(self):
> - self._file.flush()
> -
> - def close(self):
> - if not self._refcount:
> - return
> - assert self._refcount[0] > 0
> -
> - self._refcount[0] -= 1
> - if self._refcount[0] == 0:
> - self._mode = 'c'
> - if self._close:
> - self._file.close()
> - self._refcount = None
> -
> - def mode(self):
> - return self._mode
> -
> - def __getattr__(self, name):
> - if name == 'name':
> - self.name = getattr(self._file, name)
> - return self.name
> - else:
> - raise AttributeError(name)
> -
> - def __del__(self):
> - self.close()
> -
> -
> -def win32_open_devnull(mode):
> - return open('NUL', mode)
> -
> -
> -class Redirections:
> - """Stores open files and their mapping to pseudo-sh file
> descriptor.
> - """
> - # BUG: redirections are not handled correctly: 1>&3 2>&3 3>&4
> does
> - # not make 1 to redirect to 4
> - def __init__(self, stdin=None, stdout=None, stderr=None):
> - self._descriptors = {}
> - if stdin is not None:
> - self._add_descriptor(0, stdin)
> - if stdout is not None:
> - self._add_descriptor(1, stdout)
> - if stderr is not None:
> - self._add_descriptor(2, stderr)
> -
> - def add_here_document(self, interp, name, content,
> io_number=None):
> - if io_number is None:
> - io_number = 0
> -
> - if name==pyshlex.unquote_wordtree(name):
> - content = interp.expand_here_document(('TOKEN', content))
> -
> - # Write document content in a temporary file
> - tmp = tempfile.TemporaryFile()
> - try:
> - tmp.write(content)
> - tmp.flush()
> - tmp.seek(0)
> - self._add_descriptor(io_number, FileWrapper('r', tmp))
> - except:
> - tmp.close()
> - raise
> -
> - def add(self, interp, op, filename, io_number=None):
> - if op not in ('<', '>', '>|', '>>', '>&'):
> - # TODO: add descriptor duplication and
> here_documents
> - raise RedirectionError('Unsupported redirection operator
> "%s"' % op)
> -
> - if io_number is not None:
> - io_number = int(io_number)
> -
> - if (op == '>&' and filename.isdigit()) or filename=='-':
> - # No expansion for file descriptors, quote them if you
> want a filename
> - fullname = filename
> - else:
> - if filename.startswith('/'):
> - # TODO: win32 kludge
> - if filename=='/dev/null':
> - fullname = 'NUL'
> - else:
> - # TODO: handle absolute pathnames, they are
> unlikely to exist on the
> - # current platform (win32 for instance).
> - raise NotImplementedError()
> - else:
> - fullname = interp.expand_redirection(('TOKEN',
> filename))
> - if not fullname:
> - raise RedirectionError('%s: ambiguous redirect'
> % filename)
> - # Build absolute path based on PWD
> - fullname = os.path.join(interp.get_env()['PWD'],
> fullname)
> -
> - if op=='<':
> - return self._add_input_redirection(interp, fullname,
> io_number)
> - elif op in ('>', '>|'):
> - clobber = ('>|'==op)
> - return self._add_output_redirection(interp, fullname,
> io_number, clobber)
> - elif op=='>>':
> - return self._add_output_appending(interp, fullname,
> io_number)
> - elif op=='>&':
> - return self._dup_output_descriptor(fullname, io_number)
> -
> - def close(self):
> - if self._descriptors is not None:
> - for desc in self._descriptors.itervalues():
> - desc.flush()
> - desc.close()
> - self._descriptors = None
> -
> - def stdin(self):
> - return self._descriptors[0]
> -
> - def stdout(self):
> - return self._descriptors[1]
> -
> - def stderr(self):
> - return self._descriptors[2]
> -
> - def clone(self):
> - clone = Redirections()
> - for desc, fileobj in self._descriptors.iteritems():
> - clone._descriptors[desc] = fileobj.dup()
> - return clone
> -
> - def _add_output_redirection(self, interp, filename, io_number,
> clobber):
> - if io_number is None:
> - # io_number default to standard output
> - io_number = 1
> -
> - if not clobber and interp.get_env().has_opt('-C') and
> os.path.isfile(filename):
> - # File already exist in no-clobber mode, bail out
> - raise RedirectionError('File "%s" already exists' %
> filename)
> -
> - # Open and register
> - self._add_file_descriptor(io_number, filename, 'w')
> -
> - def _add_output_appending(self, interp, filename, io_number):
> - if io_number is None:
> - io_number = 1
> - self._add_file_descriptor(io_number, filename, 'a')
> -
> - def _add_input_redirection(self, interp, filename, io_number):
> - if io_number is None:
> - io_number = 0
> - self._add_file_descriptor(io_number, filename, 'r')
> -
> - def _add_file_descriptor(self, io_number, filename, mode):
> - try:
> - if filename.startswith('/'):
> - if filename=='/dev/null':
> - f = win32_open_devnull(mode+'b')
> - else:
> - # TODO: handle absolute pathnames, they are
> unlikely to exist on the
> - # current platform (win32 for instance).
> - raise NotImplementedError('cannot open absolute
> path %s' % repr(filename))
> - else:
> - f = file(filename, mode+'b')
> - except IOError as e:
> - raise RedirectionError(str(e))
> -
> - wrapper = None
> - try:
> - wrapper = FileWrapper(mode, f)
> - f = None
> - self._add_descriptor(io_number, wrapper)
> - except:
> - if f: f.close()
> - if wrapper: wrapper.close()
> - raise
> -
> - def _dup_output_descriptor(self, source_fd, dest_fd):
> - if source_fd is None:
> - source_fd = 1
> - self._dup_file_descriptor(source_fd, dest_fd, 'w')
> -
> - def _dup_file_descriptor(self, source_fd, dest_fd, mode):
> - source_fd = int(source_fd)
> - if source_fd not in self._descriptors:
> - raise RedirectionError('"%s" is not a valid file
> descriptor' % str(source_fd))
> - source = self._descriptors[source_fd]
> -
> - if source.mode()!=mode:
> - raise RedirectionError('Descriptor %s cannot be
> duplicated in mode "%s"' % (str(source), mode))
> -
> - if dest_fd=='-':
> - # Close the source descriptor
> - del self._descriptors[source_fd]
> - source.close()
> - else:
> - dest_fd = int(dest_fd)
> - if dest_fd not in self._descriptors:
> - raise RedirectionError('Cannot replace file
> descriptor %s' % str(dest_fd))
> -
> - dest = self._descriptors[dest_fd]
> - if dest.mode()!=mode:
> - raise RedirectionError('Descriptor %s cannot be
> cannot be redirected in mode "%s"' % (str(dest), mode))
> -
> - self._descriptors[dest_fd] = source.dup()
> - dest.close()
> -
> - def _add_descriptor(self, io_number, file):
> - io_number = int(io_number)
> -
> - if io_number in self._descriptors:
> - # Close the current descriptor
> - d = self._descriptors[io_number]
> - del self._descriptors[io_number]
> - d.close()
> -
> - self._descriptors[io_number] = file
> -
> - def __str__(self):
> - names = [('%d=%r' % (k, getattr(v, 'name', None))) for k,v
> - in self._descriptors.iteritems()]
> - names = ','.join(names)
> - return 'Redirections(%s)' % names
> -
> - def __del__(self):
> - self.close()
> -
> -def cygwin_to_windows_path(path):
> - """Turn /cygdrive/c/foo into c:/foo, or return path if it
> - is not a cygwin path.
> - """
> - if not path.startswith('/cygdrive/'):
> - return path
> - path = path[len('/cygdrive/'):]
> - path = path[:1] + ':' + path[1:]
> - return path
> -
> -def win32_to_unix_path(path):
> - if path is not None:
> - path = path.replace('\\', '/')
> - return path
> -
> -_RE_SHEBANG = re.compile(r'^\#!\s?([^\s]+)(?:\s([^\s]+))?')
> -_SHEBANG_CMDS = {
> - '/usr/bin/env': 'env',
> - '/bin/sh': 'pysh',
> - 'python': 'python',
> -}
> -
> -def resolve_shebang(path, ignoreshell=False):
> - """Return a list of arguments as shebang interpreter call or an
> empty list
> - if path does not refer to an executable script.
> - See <http://www.opengroup.org/austin/docs/austin_51r2.txt>.
> -
> - ignoreshell - set to True to ignore sh shebangs. Return an empty
> list instead.
> - """
> - try:
> - f = file(path)
> - try:
> - # At most 80 characters in the first line
> - header = f.read(80).splitlines()[0]
> - finally:
> - f.close()
> -
> - m = _RE_SHEBANG.search(header)
> - if not m:
> - return []
> - cmd, arg = m.group(1,2)
> - if os.path.isfile(cmd):
> - # Keep this one, the hg script for instance contains a
> weird windows
> - # shebang referencing the current python install.
> - cmdfile = os.path.basename(cmd).lower()
> - if cmdfile == 'python.exe':
> - cmd = 'python'
> - pass
> - elif cmd not in _SHEBANG_CMDS:
> - raise CommandNotFound('Unknown interpreter "%s"
> referenced in '\
> - 'shebang' % header)
> - cmd = _SHEBANG_CMDS.get(cmd)
> - if cmd is None or (ignoreshell and cmd == 'pysh'):
> - return []
> - if arg is None:
> - return [cmd, win32_to_unix_path(path)]
> - return [cmd, arg, win32_to_unix_path(path)]
> - except IOError as e:
> - if e.errno!=errno.ENOENT and \
> - (e.errno!=errno.EPERM and not os.path.isdir(path)): #
> Opening a directory raises EPERM
> - raise
> - return []
> -
> -def win32_find_in_path(name, path):
> - if isinstance(path, str):
> - path = path.split(os.pathsep)
> -
> - exts = os.environ.get('PATHEXT', '').lower().split(os.pathsep)
> - for p in path:
> - p_name = os.path.join(p, name)
> -
> - prefix = resolve_shebang(p_name)
> - if prefix:
> - return prefix
> -
> - for ext in exts:
> - p_name_ext = p_name + ext
> - if os.path.exists(p_name_ext):
> - return [win32_to_unix_path(p_name_ext)]
> - return []
> -
> -class Traps(dict):
> - def __setitem__(self, key, value):
> - if key not in ('EXIT',):
> - raise NotImplementedError()
> - super(Traps, self).__setitem__(key, value)
> -
> -# IFS white spaces character class
> -_IFS_WHITESPACES = (' ', '\t', '\n')
> -
> -class Environment:
> - """Environment holds environment variables, export table,
> function
> - definitions and whatever is defined in 2.12 "Shell Execution
> Environment",
> - redirection excepted.
> - """
> - def __init__(self, pwd):
> - self._opt = set() #Shell options
> -
> - self._functions = {}
> - self._env = {'?': '0', '#': '0'}
> - self._exported = set([
> - 'HOME', 'IFS', 'PATH'
> - ])
> -
> - # Set environment vars with side-effects
> - self._ifs_ws = None # Set of IFS whitespace characters
> - self._ifs_re = None # Regular expression used to split
> between words using IFS classes
> - self['IFS'] = ''.join(_IFS_WHITESPACES) #Default environment
> values
> - self['PWD'] = pwd
> - self.traps = Traps()
> -
> - def clone(self, subshell=False):
> - env = Environment(self['PWD'])
> - env._opt = set(self._opt)
> - for k,v in self.get_variables().iteritems():
> - if k in self._exported:
> - env.export(k,v)
> - elif subshell:
> - env[k] = v
> -
> - if subshell:
> - env._functions = dict(self._functions)
> -
> - return env
> -
> - def __getitem__(self, key):
> - if key in ('@', '*', '-', '$'):
> - raise NotImplementedError('%s is not implemented' %
> repr(key))
> - return self._env[key]
> -
> - def get(self, key, defval=None):
> - try:
> - return self[key]
> - except KeyError:
> - return defval
> -
> - def __setitem__(self, key, value):
> - if key=='IFS':
> - # Update the whitespace/non-whitespace classes
> - self._update_ifs(value)
> - elif key=='PWD':
> - pwd = os.path.abspath(value)
> - if not os.path.isdir(pwd):
> - raise VarAssignmentError('Invalid directory %s' %
> value)
> - value = pwd
> - elif key in ('?', '!'):
> - value = str(int(value))
> - self._env[key] = value
> -
> - def __delitem__(self, key):
> - if key in ('IFS', 'PWD', '?'):
> - raise VarAssignmentError('%s cannot be unset' % key)
> - del self._env[key]
> -
> - def __contains__(self, item):
> - return item in self._env
> -
> - def set_positional_args(self, args):
> - """Set the content of 'args' as positional argument from 1
> to len(args).
> - Return previous argument as a list of strings.
> - """
> - # Save and remove previous arguments
> - prevargs = []
> - for i in range(int(self._env['#'])):
> - i = str(i+1)
> - prevargs.append(self._env[i])
> - del self._env[i]
> - self._env['#'] = '0'
> -
> - #Set new ones
> - for i,arg in enumerate(args):
> - self._env[str(i+1)] = str(arg)
> - self._env['#'] = str(len(args))
> -
> - return prevargs
> -
> - def get_positional_args(self):
> - return [self._env[str(i+1)] for i in
> range(int(self._env['#']))]
> -
> - def get_variables(self):
> - return dict(self._env)
> -
> - def export(self, key, value=None):
> - if value is not None:
> - self[key] = value
> - self._exported.add(key)
> -
> - def get_exported(self):
> - return [(k,self._env.get(k)) for k in self._exported]
> -
> - def split_fields(self, word):
> - if not self._ifs_ws or not word:
> - return [word]
> - return re.split(self._ifs_re, word)
> -
> - def _update_ifs(self, value):
> - """Update the split_fields related variables when IFS
> character set is
> - changed.
> - """
> - # TODO: handle NULL IFS
> -
> - # Separate characters in whitespace and non-whitespace
> - chars = set(value)
> - ws = [c for c in chars if c in _IFS_WHITESPACES]
> - nws = [c for c in chars if c not in _IFS_WHITESPACES]
> -
> - # Keep whitespaces in a string for left and right stripping
> - self._ifs_ws = ''.join(ws)
> -
> - # Build a regexp to split fields
> - trailing = '[' + ''.join([re.escape(c) for c in ws]) + ']'
> - if nws:
> - # First, the single non-whitespace occurence.
> - nws = '[' + ''.join([re.escape(c) for c in nws]) + ']'
> - nws = '(?:' + trailing + '*' + nws + trailing + '*' +
> '|' + trailing + '+)'
> - else:
> - # Then mix all parts with quantifiers
> - nws = trailing + '+'
> - self._ifs_re = re.compile(nws)
> -
> - def has_opt(self, opt, val=None):
> - return (opt, val) in self._opt
> -
> - def set_opt(self, opt, val=None):
> - self._opt.add((opt, val))
> -
> - def find_in_path(self, name, pwd=False):
> - path = self._env.get('PATH', '').split(os.pathsep)
> - if pwd:
> - path[:0] = [self['PWD']]
> - if os.name == 'nt':
> - return win32_find_in_path(name, self._env.get('PATH',
> ''))
> - else:
> - raise NotImplementedError()
> -
> - def define_function(self, name, body):
> - if not is_name(name):
> - raise ShellSyntaxError('%s is not a valid function name'
> % repr(name))
> - self._functions[name] = body
> -
> - def remove_function(self, name):
> - del self._functions[name]
> -
> - def is_function(self, name):
> - return name in self._functions
> -
> - def get_function(self, name):
> - return self._functions.get(name)
> -
> -
> -name_charset =
> 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_'
> -name_charset = dict(zip(name_charset,name_charset))
> -
> -def match_name(s):
> - """Return the length in characters of the longest prefix made of
> name
> - allowed characters in s.
> - """
> - for i,c in enumerate(s):
> - if c not in name_charset:
> - return s[:i]
> - return s
> -
> -def is_name(s):
> - return len([c for c in s if c not in name_charset])<=0
> -
> -def is_special_param(c):
> - return len(c)==1 and c in ('@','*','#','?','-','$','!','0')
> -
> -def utility_not_implemented(name, *args, **kwargs):
> - raise NotImplementedError('%s utility is not implemented' % name)
> -
> -
> -class Utility:
> - """Define utilities properties:
> - func -- utility callable. See builtin module for utility samples.
> - is_special -- see XCU 2.8.
> - """
> - def __init__(self, func, is_special=0):
> - self.func = func
> - self.is_special = bool(is_special)
> -
> -
> -def encodeargs(args):
> - def encodearg(s):
> - lines = base64.encodestring(s)
> - lines = [l.splitlines()[0] for l in lines]
> - return ''.join(lines)
> -
> - s = pickle.dumps(args)
> - return encodearg(s)
> -
> -def decodeargs(s):
> - s = base64.decodestring(s)
> - return pickle.loads(s)
> -
> -
> -class GlobError(Exception):
> - pass
> -
> -class Options:
> - def __init__(self):
> - # True if Mercurial operates with binary streams
> - self.hgbinary = True
> -
> -class Interpreter:
> - # Implementation is very basic: the execute() method just makes
> a DFS on the
> - # AST and execute nodes one by one. Nodes are tuple (name,obj)
> where name
> - # is a string identifier and obj the AST element returned by the
> parser.
> - #
> - # Handler are named after the node identifiers.
> - # TODO: check node names and remove the switch in execute with
> some
> - # dynamic getattr() call to find node handlers.
> - """Shell interpreter.
> -
> - The following debugging flags can be passed:
> - debug-parsing - enable PLY debugging.
> - debug-tree - print the generated AST.
> - debug-cmd - trace command execution before word expansion, plus
> exit status.
> - debug-utility - trace utility execution.
> - """
> -
> - # List supported commands.
> - COMMANDS = {
> - 'cat': Utility(builtin.utility_cat,),
> - 'cd': Utility(builtin.utility_cd,),
> - ':': Utility(builtin.utility_colon,),
> - 'echo': Utility(builtin.utility_echo),
> - 'env': Utility(builtin.utility_env),
> - 'exit': Utility(builtin.utility_exit),
> - 'export': Utility(builtin.builtin_export,
> is_special=1),
> - 'egrep': Utility(builtin.utility_egrep),
> - 'fgrep': Utility(builtin.utility_fgrep),
> - 'gunzip': Utility(builtin.utility_gunzip),
> - 'kill': Utility(builtin.utility_kill),
> - 'mkdir': Utility(builtin.utility_mkdir),
> - 'netstat': Utility(builtin.utility_netstat),
> - 'printf': Utility(builtin.utility_printf),
> - 'pwd': Utility(builtin.utility_pwd),
> - 'return': Utility(builtin.builtin_return,
> is_special=1),
> - 'sed': Utility(builtin.utility_sed,),
> - 'set': Utility(builtin.builtin_set,),
> - 'shift': Utility(builtin.builtin_shift,),
> - 'sleep': Utility(builtin.utility_sleep,),
> - 'sort': Utility(builtin.utility_sort,),
> - 'trap': Utility(builtin.builtin_trap,
> is_special=1),
> - 'true': Utility(builtin.utility_true),
> - 'unset': Utility(builtin.builtin_unset,
> is_special=1),
> - 'wait': Utility(builtin.builtin_wait,
> is_special=1),
> - }
> -
> - def __init__(self, pwd, debugflags = [], env=None, redirs=None,
> stdin=None,
> - stdout=None, stderr=None, opts=Options()):
> - self._env = env
> - if self._env is None:
> - self._env = Environment(pwd)
> - self._children = {}
> -
> - self._redirs = redirs
> - self._close_redirs = False
> -
> - if self._redirs is None:
> - if stdin is None:
> - stdin = sys.stdin
> - if stdout is None:
> - stdout = sys.stdout
> - if stderr is None:
> - stderr = sys.stderr
> - stdin = FileWrapper('r', stdin, False)
> - stdout = FileWrapper('w', stdout, False)
> - stderr = FileWrapper('w', stderr, False)
> - self._redirs = Redirections(stdin, stdout, stderr)
> - self._close_redirs = True
> -
> - self._debugflags = list(debugflags)
> - self._logfile = sys.stderr
> - self._options = opts
> -
> - def close(self):
> - """Must be called when the interpreter is no longer used."""
> - script = self._env.traps.get('EXIT')
> - if script:
> - try:
> - self.execute_script(script=script)
> - except:
> - pass
> -
> - if self._redirs is not None and self._close_redirs:
> - self._redirs.close()
> - self._redirs = None
> -
> - def log(self, s):
> - self._logfile.write(s)
> - self._logfile.flush()
> -
> - def __getitem__(self, key):
> - return self._env[key]
> -
> - def __setitem__(self, key, value):
> - self._env[key] = value
> -
> - def options(self):
> - return self._options
> -
> - def redirect(self, redirs, ios):
> - def add_redir(io):
> - if isinstance(io, pyshyacc.IORedirect):
> - redirs.add(self, io.op, io.filename, io.io_number)
> - else:
> - redirs.add_here_document(self, io.name, io.content,
> io.io_number)
> -
> - map(add_redir, ios)
> - return redirs
> -
> - def execute_script(self, script=None, ast=None, sourced=False,
> - scriptpath=None):
> - """If script is not None, parse the input. Otherwise takes
> the supplied
> - AST. Then execute the AST.
> - Return the script exit status.
> - """
> - try:
> - if scriptpath is not None:
> - self._env['0'] = os.path.abspath(scriptpath)
> -
> - if script is not None:
> - debug_parsing = ('debug-parsing' in
> self._debugflags)
> - cmds, script = pyshyacc.parse(script, True,
> debug_parsing)
> - if 'debug-tree' in self._debugflags:
> - pyshyacc.print_commands(cmds, self._logfile)
> - self._logfile.flush()
> - else:
> - cmds, script = ast, ''
> -
> - status = 0
> - for cmd in cmds:
> - try:
> - status = self.execute(cmd)
> - except ExitSignal as e:
> - if sourced:
> - raise
> - status = int(e.args[0])
> - return status
> - except ShellError:
> - self._env['?'] = 1
> - raise
> - if 'debug-utility' in self._debugflags or
> 'debug-cmd' in self._debugflags:
> - self.log('returncode ' + str(status)+ '\n')
> - return status
> - except CommandNotFound as e:
> - print >>self._redirs.stderr, str(e)
> - self._redirs.stderr.flush()
> - # Command not found by non-interactive shell
> - # return 127
> - raise
> - except RedirectionError as e:
> - # TODO: should be handled depending on the utility status
> - print >>self._redirs.stderr, str(e)
> - self._redirs.stderr.flush()
> - # Command not found by non-interactive shell
> - # return 127
> - raise
> -
> - def dotcommand(self, env, args):
> - if len(args) < 1:
> - raise ShellError('. expects at least one argument')
> - path = args[0]
> - if '/' not in path:
> - found = env.find_in_path(args[0], True)
> - if found:
> - path = found[0]
> - script = file(path).read()
> - return self.execute_script(script=script, sourced=True)
> -
> - def execute(self, token, redirs=None):
> - """Execute and AST subtree with supplied redirections
> overriding default
> - interpreter ones.
> - Return the exit status.
> - """
> - if not token:
> - return 0
> -
> - if redirs is None:
> - redirs = self._redirs
> -
> - if isinstance(token, list):
> - # Commands sequence
> - res = 0
> - for t in token:
> - res = self.execute(t, redirs)
> - return res
> -
> - type, value = token
> - status = 0
> - if type=='simple_command':
> - redirs_copy = redirs.clone()
> - try:
> - # TODO: define and handle command return values
> - # TODO: implement set -e
> - status = self._execute_simple_command(value,
> redirs_copy)
> - finally:
> - redirs_copy.close()
> - elif type=='pipeline':
> - status = self._execute_pipeline(value, redirs)
> - elif type=='and_or':
> - status = self._execute_and_or(value, redirs)
> - elif type=='for_clause':
> - status = self._execute_for_clause(value, redirs)
> - elif type=='while_clause':
> - status = self._execute_while_clause(value, redirs)
> - elif type=='function_definition':
> - status = self._execute_function_definition(value, redirs)
> - elif type=='brace_group':
> - status = self._execute_brace_group(value, redirs)
> - elif type=='if_clause':
> - status = self._execute_if_clause(value, redirs)
> - elif type=='subshell':
> - status = self.subshell(ast=value.cmds, redirs=redirs)
> - elif type=='async':
> - status = self._asynclist(value)
> - elif type=='redirect_list':
> - redirs_copy = self.redirect(redirs.clone(), value.redirs)
> - try:
> - status = self.execute(value.cmd, redirs_copy)
> - finally:
> - redirs_copy.close()
> - else:
> - raise NotImplementedError('Unsupported token type ' +
> type) -
> - if status < 0:
> - status = 255
> - return status
> -
> - def _execute_if_clause(self, if_clause, redirs):
> - cond_status = self.execute(if_clause.cond, redirs)
> - if cond_status==0:
> - return self.execute(if_clause.if_cmds, redirs)
> - else:
> - return self.execute(if_clause.else_cmds, redirs)
> -
> - def _execute_brace_group(self, group, redirs):
> - status = 0
> - for cmd in group.cmds:
> - status = self.execute(cmd, redirs)
> - return status
> -
> - def _execute_function_definition(self, fundef, redirs):
> - self._env.define_function(fundef.name, fundef.body)
> - return 0
> -
> - def _execute_while_clause(self, while_clause, redirs):
> - status = 0
> - while 1:
> - cond_status = 0
> - for cond in while_clause.condition:
> - cond_status = self.execute(cond, redirs)
> -
> - if cond_status:
> - break
> -
> - for cmd in while_clause.cmds:
> - status = self.execute(cmd, redirs)
> -
> - return status
> -
> - def _execute_for_clause(self, for_clause, redirs):
> - if not is_name(for_clause.name):
> - raise ShellSyntaxError('%s is not a valid name' %
> repr(for_clause.name))
> - items = mappend(self.expand_token, for_clause.items)
> -
> - status = 0
> - for item in items:
> - self._env[for_clause.name] = item
> - for cmd in for_clause.cmds:
> - status = self.execute(cmd, redirs)
> - return status
> -
> - def _execute_and_or(self, or_and, redirs):
> - res = self.execute(or_and.left, redirs)
> - if (or_and.op=='&&' and res==0) or (or_and.op!='&&' and
> res!=0):
> - res = self.execute(or_and.right, redirs)
> - return res
> -
> - def _execute_pipeline(self, pipeline, redirs):
> - if len(pipeline.commands)==1:
> - status = self.execute(pipeline.commands[0], redirs)
> - else:
> - # Execute all commands one after the other
> - status = 0
> - inpath, outpath = None, None
> - try:
> - # Commands inputs and outputs cannot really be
> plugged as done
> - # by a real shell. Run commands sequentially and
> chain their
> - # input/output throught temporary files.
> - tmpfd, inpath = tempfile.mkstemp()
> - os.close(tmpfd)
> - tmpfd, outpath = tempfile.mkstemp()
> - os.close(tmpfd)
> -
> - inpath = win32_to_unix_path(inpath)
> - outpath = win32_to_unix_path(outpath)
> -
> - for i, cmd in enumerate(pipeline.commands):
> - call_redirs = redirs.clone()
> - try:
> - if i!=0:
> - call_redirs.add(self, '<', inpath)
> - if i!=len(pipeline.commands)-1:
> - call_redirs.add(self, '>', outpath)
> -
> - status = self.execute(cmd, call_redirs)
> -
> - # Chain inputs/outputs
> - inpath, outpath = outpath, inpath
> - finally:
> - call_redirs.close()
> - finally:
> - if inpath: os.remove(inpath)
> - if outpath: os.remove(outpath)
> -
> - if pipeline.reverse_status:
> - status = int(not status)
> - self._env['?'] = status
> - return status
> -
> - def _execute_function(self, name, args, interp, env, stdin,
> stdout, stderr, *others):
> - assert interp is self
> -
> - func = env.get_function(name)
> - #Set positional parameters
> - prevargs = None
> - try:
> - prevargs = env.set_positional_args(args)
> - try:
> - redirs = Redirections(stdin.dup(), stdout.dup(),
> stderr.dup())
> - try:
> - status = self.execute(func, redirs)
> - finally:
> - redirs.close()
> - except ReturnSignal as e:
> - status = int(e.args[0])
> - env['?'] = status
> - return status
> - finally:
> - #Reset positional parameters
> - if prevargs is not None:
> - env.set_positional_args(prevargs)
> -
> - def _execute_simple_command(self, token, redirs):
> - """Can raise ReturnSignal when return builtin is called,
> ExitSignal when
> - exit is called, and other shell exceptions upon builtin
> failures.
> - """
> - debug_command = 'debug-cmd' in self._debugflags
> - if debug_command:
> - self.log('word' + repr(token.words) + '\n')
> - self.log('assigns' + repr(token.assigns) + '\n')
> - self.log('redirs' + repr(token.redirs) + '\n')
> -
> - is_special = None
> - env = self._env
> -
> - try:
> - # Word expansion
> - args = []
> - for word in token.words:
> - args += self.expand_token(word)
> - if is_special is None and args:
> - is_special = env.is_function(args[0]) or \
> - (args[0] in self.COMMANDS and
> self.COMMANDS[args[0]].is_special)
> -
> - if debug_command:
> - self.log('_execute_simple_command' + str(args) +
> '\n')
> -
> - if not args:
> - # Redirections happen is a subshell
> - redirs = redirs.clone()
> - elif not is_special:
> - env = self._env.clone()
> -
> - # Redirections
> - self.redirect(redirs, token.redirs)
> -
> - # Variables assignments
> - res = 0
> - for type,(k,v) in token.assigns:
> - status, expanded = self.expand_variable((k,v))
> - if status is not None:
> - res = status
> - if args:
> - env.export(k, expanded)
> - else:
> - env[k] = expanded
> -
> - if args and args[0] in ('.', 'source'):
> - res = self.dotcommand(env, args[1:])
> - elif args:
> - if args[0] in self.COMMANDS:
> - command = self.COMMANDS[args[0]]
> - elif env.is_function(args[0]):
> - command = Utility(self._execute_function,
> is_special=True)
> - else:
> - if not '/' in args[0].replace('\\', '/'):
> - cmd = env.find_in_path(args[0])
> - if not cmd:
> - # TODO: test error code on unknown
> command => 127
> - raise CommandNotFound('Unknown command:
> "%s"' % args[0])
> - else:
> - # Handle commands like '/cygdrive/c/foo.bat'
> - cmd = cygwin_to_windows_path(args[0])
> - if not os.path.exists(cmd):
> - raise CommandNotFound('%s: No such file
> or directory' % args[0])
> - shebang = resolve_shebang(cmd)
> - if shebang:
> - cmd = shebang
> - else:
> - cmd = [cmd]
> - args[0:1] = cmd
> - command = Utility(builtin.run_command)
> -
> - # Command execution
> - if 'debug-cmd' in self._debugflags:
> - self.log('redirections ' + str(redirs) + '\n')
> -
> - res = command.func(args[0], args[1:], self, env,
> - redirs.stdin(), redirs.stdout(),
> - redirs.stderr(), self._debugflags)
> -
> - if self._env.has_opt('-x'):
> - # Trace command execution in shell environment
> - # BUG: would be hard to reproduce a real shell
> behaviour since
> - # the AST is not annotated with source lines/tokens.
> - self._redirs.stdout().write(' '.join(args))
> -
> - except ReturnSignal:
> - raise
> - except ShellError as e:
> - if is_special or isinstance(e, (ExitSignal,
> - ShellSyntaxError,
> ExpansionError)):
> - raise e
> - self._redirs.stderr().write(str(e)+'\n')
> - return 1
> -
> - return res
> -
> - def expand_token(self, word):
> - """Expand a word as specified in [2.6 Word Expansions].
> Return the list
> - of expanded words.
> - """
> - status, wtrees = self._expand_word(word)
> - return map(pyshlex.wordtree_as_string, wtrees)
> -
> - def expand_variable(self, word):
> - """Return a status code (or None if no command expansion
> occurred)
> - and a single word.
> - """
> - status, wtrees = self._expand_word(word, pathname=False,
> split=False)
> - words = map(pyshlex.wordtree_as_string, wtrees)
> - assert len(words)==1
> - return status, words[0]
> -
> - def expand_here_document(self, word):
> - """Return the expanded document as a single word. The here
> document is
> - assumed to be unquoted.
> - """
> - status, wtrees = self._expand_word(word, pathname=False,
> - split=False,
> here_document=True)
> - words = map(pyshlex.wordtree_as_string, wtrees)
> - assert len(words)==1
> - return words[0]
> -
> - def expand_redirection(self, word):
> - """Return a single word."""
> - return self.expand_variable(word)[1]
> -
> - def get_env(self):
> - return self._env
> -
> - def _expand_word(self, token, pathname=True, split=True,
> here_document=False):
> - wtree = pyshlex.make_wordtree(token[1],
> here_document=here_document)
> -
> - # TODO: implement tilde expansion
> - def expand(wtree):
> - """Return a pseudo wordtree: the tree or its subelements
> can be empty
> - lists when no value result from the expansion.
> - """
> - status = None
> - for part in wtree:
> - if not isinstance(part, list):
> - continue
> - if part[0]in ("'", '\\'):
> - continue
> - elif part[0] in ('`', '$('):
> - status, result = self._expand_command(part)
> - part[:] = result
> - elif part[0] in ('$', '${'):
> - part[:] = self._expand_parameter(part,
> wtree[0]=='"', split)
> - elif part[0] in ('', '"'):
> - status, result = expand(part)
> - part[:] = result
> - else:
> - raise NotImplementedError('%s expansion is not
> implemented'
> - % part[0])
> - # [] is returned when an expansion result in no-field,
> - # like an empty $@
> - wtree = [p for p in wtree if p != []]
> - if len(wtree) < 3:
> - return status, []
> - return status, wtree
> -
> - status, wtree = expand(wtree)
> - if len(wtree) == 0:
> - return status, wtree
> - wtree = pyshlex.normalize_wordtree(wtree)
> -
> - if split:
> - wtrees = self._split_fields(wtree)
> - else:
> - wtrees = [wtree]
> -
> - if pathname:
> - wtrees = mappend(self._expand_pathname, wtrees)
> -
> - wtrees = map(self._remove_quotes, wtrees)
> - return status, wtrees
> -
> - def _expand_command(self, wtree):
> - # BUG: there is something to do with backslashes and quoted
> - # characters here
> - command = pyshlex.wordtree_as_string(wtree[1:-1])
> - status, output = self.subshell_output(command)
> - return status, ['', output, '']
> -
> - def _expand_parameter(self, wtree, quoted=False, split=False):
> - """Return a valid wtree or an empty list when no parameter
> results."""
> - # Get the parameter name
> - # TODO: implement weird expansion rules with ':'
> - name = pyshlex.wordtree_as_string(wtree[1:-1])
> - if not is_name(name) and not is_special_param(name):
> - raise ExpansionError('Bad substitution "%s"' % name)
> - # TODO: implement special parameters
> - if name in ('@', '*'):
> - args = self._env.get_positional_args()
> - if len(args) == 0:
> - return []
> - if len(args)<2:
> - return ['', ''.join(args), '']
> -
> - sep = self._env.get('IFS', '')[:1]
> - if split and quoted and name=='@':
> - # Introduce a new token to tell the caller that
> these parameters
> - # cause a split as specified in 2.5.2
> - return ['@'] + args + ['']
> - else:
> - return ['', sep.join(args), '']
> -
> - return ['', self._env.get(name, ''), '']
> -
> - def _split_fields(self, wtree):
> - def is_empty(split):
> - return split==['', '', '']
> -
> - def split_positional(quoted):
> - # Return a list of wtree split according positional
> parameters rules.
> - # All remaining '@' groups are removed.
> - assert quoted[0]=='"'
> -
> - splits = [[]]
> - for part in quoted:
> - if not isinstance(part, list) or part[0]!='@':
> - splits[-1].append(part)
> - else:
> - # Empty or single argument list were dealt with
> already
> - assert len(part)>3
> - # First argument must join with the beginning
> part of the original word
> - splits[-1].append(part[1])
> - # Create double-quotes expressions for every
> argument after the first
> - for arg in part[2:-1]:
> - splits[-1].append('"')
> - splits.append(['"', arg])
> - return splits
> -
> - # At this point, all expansions but pathnames have occured.
> Only quoted
> - # and positional sequences remain. Thus, all candidates for
> field splitting
> - # are in the tree root, or are positional splits ('@') and
> lie in root
> - # children.
> - if not wtree or wtree[0] not in ('', '"'):
> - # The whole token is quoted or empty, nothing to split
> - return [wtree]
> -
> - if wtree[0]=='"':
> - wtree = ['', wtree, '']
> -
> - result = [['', '']]
> - for part in wtree[1:-1]:
> - if isinstance(part, list):
> - if part[0]=='"':
> - splits = split_positional(part)
> - if len(splits)<=1:
> - result[-1] += [part, '']
> - else:
> - # Terminate the current split
> - result[-1] += [splits[0], '']
> - result += splits[1:-1]
> - # Create a new split
> - result += [['', splits[-1], '']]
> - else:
> - result[-1] += [part, '']
> - else:
> - splits = self._env.split_fields(part)
> - if len(splits)<=1:
> - # No split
> - result[-1][-1] += part
> - else:
> - # Terminate the current resulting part and
> create a new one
> - result[-1][-1] += splits[0]
> - result[-1].append('')
> - result += [['', r, ''] for r in splits[1:-1]]
> - result += [['', splits[-1]]]
> - result[-1].append('')
> -
> - # Leading and trailing empty groups come from
> leading/trailing blanks
> - if result and is_empty(result[-1]):
> - result[-1:] = []
> - if result and is_empty(result[0]):
> - result[:1] = []
> - return result
> -
> - def _expand_pathname(self, wtree):
> - """See [2.6.6 Pathname Expansion]."""
> - if self._env.has_opt('-f'):
> - return [wtree]
> -
> - # All expansions have been performed, only quoted sequences
> should remain
> - # in the tree. Generate the pattern by folding the tree,
> escaping special
> - # characters when appear quoted
> - special_chars = '*?[]'
> -
> - def make_pattern(wtree):
> - subpattern = []
> - for part in wtree[1:-1]:
> - if isinstance(part, list):
> - part = make_pattern(part)
> - elif wtree[0]!='':
> - for c in part:
> - # Meta-characters cannot be quoted
> - if c in special_chars:
> - raise GlobError()
> - subpattern.append(part)
> - return ''.join(subpattern)
> -
> - def pwd_glob(pattern):
> - cwd = os.getcwd()
> - os.chdir(self._env['PWD'])
> - try:
> - return glob.glob(pattern)
> - finally:
> - os.chdir(cwd)
> -
> - #TODO: check working directory issues here wrt relative
> patterns
> - try:
> - pattern = make_pattern(wtree)
> - paths = pwd_glob(pattern)
> - except GlobError:
> - # BUG: Meta-characters were found in quoted sequences.
> The should
> - # have been used literally but this is unsupported in
> current glob module.
> - # Instead we consider the whole tree must be used
> literally and
> - # therefore there is no point in globbing. This is wrong
> when meta
> - # characters are mixed with quoted meta in the same
> pattern like:
> - # < foo*"py*" >
> - paths = []
> -
> - if not paths:
> - return [wtree]
> - return [['', path, ''] for path in paths]
> -
> - def _remove_quotes(self, wtree):
> - """See [2.6.7 Quote Removal]."""
> -
> - def unquote(wtree):
> - unquoted = []
> - for part in wtree[1:-1]:
> - if isinstance(part, list):
> - part = unquote(part)
> - unquoted.append(part)
> - return ''.join(unquoted)
> -
> - return ['', unquote(wtree), '']
> -
> - def subshell(self, script=None, ast=None, redirs=None):
> - """Execute the script or AST in a subshell, with inherited
> redirections
> - if redirs is not None.
> - """
> - if redirs:
> - sub_redirs = redirs
> - else:
> - sub_redirs = redirs.clone()
> -
> - subshell = None
> - try:
> - subshell = Interpreter(None, self._debugflags,
> self._env.clone(True),
> - sub_redirs, opts=self._options)
> - return subshell.execute_script(script, ast)
> - finally:
> - if not redirs: sub_redirs.close()
> - if subshell: subshell.close()
> -
> - def subshell_output(self, script):
> - """Execute the script in a subshell and return the captured
> output."""
> - # Create temporary file to capture subshell output
> - tmpfd, tmppath = tempfile.mkstemp()
> - try:
> - tmpfile = os.fdopen(tmpfd, 'wb')
> - stdout = FileWrapper('w', tmpfile)
> -
> - redirs = Redirections(self._redirs.stdin().dup(),
> - stdout,
> -
> self._redirs.stderr().dup())
> - try:
> - status = self.subshell(script=script, redirs=redirs)
> - finally:
> - redirs.close()
> - redirs = None
> -
> - # Extract subshell standard output
> - tmpfile = open(tmppath, 'rb')
> - try:
> - output = tmpfile.read()
> - return status, output.rstrip('\n')
> - finally:
> - tmpfile.close()
> - finally:
> - os.remove(tmppath)
> -
> - def _asynclist(self, cmd):
> - args = (self._env.get_variables(), cmd)
> - arg = encodeargs(args)
> - assert len(args) < 30*1024
> - cmd = ['pysh.bat', '--ast', '-c', arg]
> - p = subprocess.Popen(cmd, cwd=self._env['PWD'])
> - self._children[p.pid] = p
> - self._env['!'] = p.pid
> - return 0
> -
> - def wait(self, pids=None):
> - if not pids:
> - pids = self._children.keys()
> -
> - status = 127
> - for pid in pids:
> - if pid not in self._children:
> - continue
> - p = self._children.pop(pid)
> - status = p.wait()
> -
> - return status
> -
> diff --git a/bitbake/lib/bb/pysh/lsprof.py
> b/bitbake/lib/bb/pysh/lsprof.py deleted file mode 100644
> index b1831c2..0000000
> --- a/bitbake/lib/bb/pysh/lsprof.py
> +++ /dev/null
> @@ -1,116 +0,0 @@
> -#! /usr/bin/env python
> -
> -import sys
> -from _lsprof import Profiler, profiler_entry
> -
> -__all__ = ['profile', 'Stats']
> -
> -def profile(f, *args, **kwds):
> - """XXX docstring"""
> - p = Profiler()
> - p.enable(subcalls=True, builtins=True)
> - try:
> - f(*args, **kwds)
> - finally:
> - p.disable()
> - return Stats(p.getstats())
> -
> -
> -class Stats(object):
> - """XXX docstring"""
> -
> - def __init__(self, data):
> - self.data = data
> -
> - def sort(self, crit="inlinetime"):
> - """XXX docstring"""
> - if crit not in profiler_entry.__dict__:
> - raise ValueError("Can't sort by %s" % crit)
> - self.data.sort(lambda b, a: cmp(getattr(a, crit),
> - getattr(b, crit)))
> - for e in self.data:
> - if e.calls:
> - e.calls.sort(lambda b, a: cmp(getattr(a, crit),
> - getattr(b, crit)))
> -
> - def pprint(self, top=None, file=None, limit=None, climit=None):
> - """XXX docstring"""
> - if file is None:
> - file = sys.stdout
> - d = self.data
> - if top is not None:
> - d = d[:top]
> - cols = "% 12s %12s %11.4f %11.4f %s\n"
> - hcols = "% 12s %12s %12s %12s %s\n"
> - cols2 = "+%12s %12s %11.4f %11.4f + %s\n"
> - file.write(hcols % ("CallCount", "Recursive", "Total(ms)",
> - "Inline(ms)", "module:lineno(function)"))
> - count = 0
> - for e in d:
> - file.write(cols % (e.callcount, e.reccallcount,
> e.totaltime,
> - e.inlinetime, label(e.code)))
> - count += 1
> - if limit is not None and count == limit:
> - return
> - ccount = 0
> - if e.calls:
> - for se in e.calls:
> - file.write(cols % ("+%s" % se.callcount,
> se.reccallcount,
> - se.totaltime, se.inlinetime,
> - "+%s" % label(se.code)))
> - count += 1
> - ccount += 1
> - if limit is not None and count == limit:
> - return
> - if climit is not None and ccount == climit:
> - break
> -
> - def freeze(self):
> - """Replace all references to code objects with string
> - descriptions; this makes it possible to pickle the
> instance.""" -
> - # this code is probably rather ickier than it needs to be!
> - for i in range(len(self.data)):
> - e = self.data[i]
> - if not isinstance(e.code, str):
> - self.data[i] = type(e)((label(e.code),) + e[1:])
> - if e.calls:
> - for j in range(len(e.calls)):
> - se = e.calls[j]
> - if not isinstance(se.code, str):
> - e.calls[j] = type(se)((label(se.code),) +
> se[1:]) -
> -_fn2mod = {}
> -
> -def label(code):
> - if isinstance(code, str):
> - return code
> - try:
> - mname = _fn2mod[code.co_filename]
> - except KeyError:
> - for k, v in sys.modules.items():
> - if v is None:
> - continue
> - if not hasattr(v, '__file__'):
> - continue
> - if not isinstance(v.__file__, str):
> - continue
> - if v.__file__.startswith(code.co_filename):
> - mname = _fn2mod[code.co_filename] = k
> - break
> - else:
> - mname = _fn2mod[code.co_filename] =
> '<%s>'%code.co_filename -
> - return '%s:%d(%s)' % (mname, code.co_firstlineno, code.co_name)
> -
> -
> -if __name__ == '__main__':
> - import os
> - sys.argv = sys.argv[1:]
> - if not sys.argv:
> - print >> sys.stderr, "usage: lsprof.py <script>
> <arguments...>"
> - sys.exit(2)
> - sys.path.insert(0, os.path.abspath(os.path.dirname(sys.argv[0])))
> - stats = profile(execfile, sys.argv[0], globals(), locals())
> - stats.sort()
> - stats.pprint()
> diff --git a/bitbake/lib/bb/pysh/pysh.py b/bitbake/lib/bb/pysh/pysh.py
> deleted file mode 100644
> index b4e6145..0000000
> --- a/bitbake/lib/bb/pysh/pysh.py
> +++ /dev/null
> @@ -1,167 +0,0 @@
> -# pysh.py - command processing for pysh.
> -#
> -# Copyright 2007 Patrick Mezard
> -#
> -# This software may be used and distributed according to the terms
> -# of the GNU General Public License, incorporated herein by
> reference. -
> -import optparse
> -import os
> -import sys
> -
> -import interp
> -
> -SH_OPT = optparse.OptionParser(prog='pysh', usage="%prog [OPTIONS]",
> version='0.1') -SH_OPT.add_option('-c', action='store_true',
> dest='command_string', default=None,
> - help='A string that shall be interpreted by the shell as one or
> more commands') -SH_OPT.add_option('--redirect-to',
> dest='redirect_to', default=None,
> - help='Redirect script commands stdout and stderr to the
> specified file') -# See utility_command in builtin.py about the
> reason for this flag. -SH_OPT.add_option('--redirected',
> dest='redirected', action='store_true', default=False,
> - help='Tell the interpreter that stdout and stderr are actually
> the same objects, which is really stdout')
> -SH_OPT.add_option('--debug-parsing', action='store_true',
> dest='debug_parsing', default=False,
> - help='Trace PLY execution')
> -SH_OPT.add_option('--debug-tree', action='store_true',
> dest='debug_tree', default=False,
> - help='Display the generated syntax tree.')
> -SH_OPT.add_option('--debug-cmd', action='store_true',
> dest='debug_cmd', default=False,
> - help='Trace command execution before parameters expansion and
> exit status.') -SH_OPT.add_option('--debug-utility',
> action='store_true', dest='debug_utility', default=False,
> - help='Trace utility calls, after parameters expansions')
> -SH_OPT.add_option('--ast', action='store_true', dest='ast',
> default=False,
> - help='Encoded commands to execute in a subprocess')
> -SH_OPT.add_option('--profile', action='store_true', default=False,
> - help='Profile pysh run')
> -
> -
> -def split_args(args):
> - # Separate shell arguments from command ones
> - # Just stop at the first argument not starting with a dash. I
> know, this is completely broken,
> - # it ignores files starting with a dash or may take option
> values for command file. This is not
> - # supposed to happen for now
> - command_index = len(args)
> - for i,arg in enumerate(args):
> - if not arg.startswith('-'):
> - command_index = i
> - break
> -
> - return args[:command_index], args[command_index:]
> -
> -
> -def fixenv(env):
> - path = env.get('PATH')
> - if path is not None:
> - parts = path.split(os.pathsep)
> - # Remove Windows utilities from PATH, they are useless at
> best and
> - # some of them (find) may be confused with other utilities.
> - parts = [p for p in parts if 'system32' not in p.lower()]
> - env['PATH'] = os.pathsep.join(parts)
> - if env.get('HOME') is None:
> - # Several utilities, including cvsps, cannot work without
> - # a defined HOME directory.
> - env['HOME'] = os.path.expanduser('~')
> - return env
> -
> -def _sh(cwd, shargs, cmdargs, options, debugflags=None, env=None):
> - if os.environ.get('PYSH_TEXT') != '1':
> - import msvcrt
> - for fp in (sys.stdin, sys.stdout, sys.stderr):
> - msvcrt.setmode(fp.fileno(), os.O_BINARY)
> -
> - hgbin = os.environ.get('PYSH_HGTEXT') != '1'
> -
> - if debugflags is None:
> - debugflags = []
> - if options.debug_parsing:
> debugflags.append('debug-parsing')
> - if options.debug_utility:
> debugflags.append('debug-utility')
> - if options.debug_cmd: debugflags.append('debug-cmd')
> - if options.debug_tree: debugflags.append('debug-tree')
> -
> - if env is None:
> - env = fixenv(dict(os.environ))
> - if cwd is None:
> - cwd = os.getcwd()
> -
> - if not cmdargs:
> - # Nothing to do
> - return 0
> -
> - ast = None
> - command_file = None
> - if options.command_string:
> - input = cmdargs[0]
> - if not options.ast:
> - input += '\n'
> - else:
> - args, input = interp.decodeargs(input), None
> - env, ast = args
> - cwd = env.get('PWD', cwd)
> - else:
> - command_file = cmdargs[0]
> - arguments = cmdargs[1:]
> -
> - prefix = interp.resolve_shebang(command_file,
> ignoreshell=True)
> - if prefix:
> - input = ' '.join(prefix + [command_file] + arguments)
> - else:
> - # Read commands from file
> - f = file(command_file)
> - try:
> - # Trailing newline to help the parser
> - input = f.read() + '\n'
> - finally:
> - f.close()
> -
> - redirect = None
> - try:
> - if options.redirected:
> - stdout = sys.stdout
> - stderr = stdout
> - elif options.redirect_to:
> - redirect = open(options.redirect_to, 'wb')
> - stdout = redirect
> - stderr = redirect
> - else:
> - stdout = sys.stdout
> - stderr = sys.stderr
> -
> - # TODO: set arguments to environment variables
> - opts = interp.Options()
> - opts.hgbinary = hgbin
> - ip = interp.Interpreter(cwd, debugflags, stdout=stdout,
> stderr=stderr,
> - opts=opts)
> - try:
> - # Export given environment in shell object
> - for k,v in env.iteritems():
> - ip.get_env().export(k,v)
> - return ip.execute_script(input, ast,
> scriptpath=command_file)
> - finally:
> - ip.close()
> - finally:
> - if redirect is not None:
> - redirect.close()
> -
> -def sh(cwd=None, args=None, debugflags=None, env=None):
> - if args is None:
> - args = sys.argv[1:]
> - shargs, cmdargs = split_args(args)
> - options, shargs = SH_OPT.parse_args(shargs)
> -
> - if options.profile:
> - import lsprof
> - p = lsprof.Profiler()
> - p.enable(subcalls=True)
> - try:
> - return _sh(cwd, shargs, cmdargs, options, debugflags,
> env)
> - finally:
> - p.disable()
> - stats = lsprof.Stats(p.getstats())
> - stats.sort()
> - stats.pprint(top=10, file=sys.stderr, climit=5)
> - else:
> - return _sh(cwd, shargs, cmdargs, options, debugflags, env)
> -
> -def main():
> - sys.exit(sh())
> -
> -if __name__=='__main__':
> - main()
> diff --git a/bitbake/lib/bb/pysh/pyshlex.py
> b/bitbake/lib/bb/pysh/pyshlex.py index fbf094b..a42c294 100644
> --- a/bitbake/lib/bb/pysh/pyshlex.py
> +++ b/bitbake/lib/bb/pysh/pyshlex.py
> @@ -13,11 +13,6 @@
> # PLY in pull mode. It was designed to work incrementally and it
> would not be # that hard to enable pull mode.
> import re
> -try:
> - s = set()
> - del s
> -except NameError:
> - from Set import Set as set
>
> from ply import lex
> from bb.pysh.sherrors import *
> diff --git a/bitbake/lib/bb/pysh/pyshyacc.py
> b/bitbake/lib/bb/pysh/pyshyacc.py index ba4cefd..de565dc 100644
> --- a/bitbake/lib/bb/pysh/pyshyacc.py
> +++ b/bitbake/lib/bb/pysh/pyshyacc.py
> @@ -636,13 +636,16 @@ def p_empty(p):
> def p_error(p):
> msg = []
> w = msg.append
> - w('%r\n' % p)
> - w('followed by:\n')
> - for i in range(5):
> - n = yacc.token()
> - if not n:
> - break
> - w(' %r\n' % n)
> + if p:
> + w('%r\n' % p)
> + w('followed by:\n')
> + for i in range(5):
> + n = yacc.token()
> + if not n:
> + break
> + w(' %r\n' % n)
> + else:
> + w('Unexpected EOF')
> raise sherrors.ShellSyntaxError(''.join(msg))
>
> # Build the parser
> diff --git a/bitbake/lib/bb/pysh/sherrors.py
> b/bitbake/lib/bb/pysh/sherrors.py index 49d0533..3fe8e47 100644
> --- a/bitbake/lib/bb/pysh/sherrors.py
> +++ b/bitbake/lib/bb/pysh/sherrors.py
> @@ -13,29 +13,3 @@ class ShellError(Exception):
>
> class ShellSyntaxError(ShellError):
> pass
> -
> -class UtilityError(ShellError):
> - """Raised upon utility syntax error (option or operand error)."""
> - pass
> -
> -class ExpansionError(ShellError):
> - pass
> -
> -class CommandNotFound(ShellError):
> - """Specified command was not found."""
> - pass
> -
> -class RedirectionError(ShellError):
> - pass
> -
> -class VarAssignmentError(ShellError):
> - """Variable assignment error."""
> - pass
> -
> -class ExitSignal(ShellError):
> - """Exit signal."""
> - pass
> -
> -class ReturnSignal(ShellError):
> - """Exit signal."""
> - pass
> diff --git a/bitbake/lib/bb/pysh/subprocess_fix.py
> b/bitbake/lib/bb/pysh/subprocess_fix.py deleted file mode 100644
> index 46eca22..0000000
> --- a/bitbake/lib/bb/pysh/subprocess_fix.py
> +++ /dev/null
> @@ -1,77 +0,0 @@
> -# subprocess - Subprocesses with accessible I/O streams
> -#
> -# For more information about this module, see PEP 324.
> -#
> -# This module should remain compatible with Python 2.2, see PEP 291.
> -#
> -# Copyright (c) 2003-2005 by Peter Astrand <astrand@lysator.liu.se>
> -#
> -# Licensed to PSF under a Contributor Agreement.
> -# See http://www.python.org/2.4/license for licensing details.
> -
> -def list2cmdline(seq):
> - """
> - Translate a sequence of arguments into a command line
> - string, using the same rules as the MS C runtime:
> -
> - 1) Arguments are delimited by white space, which is either a
> - space or a tab.
> -
> - 2) A string surrounded by double quotation marks is
> - interpreted as a single argument, regardless of white space
> - contained within. A quoted string can be embedded in an
> - argument.
> -
> - 3) A double quotation mark preceded by a backslash is
> - interpreted as a literal double quotation mark.
> -
> - 4) Backslashes are interpreted literally, unless they
> - immediately precede a double quotation mark.
> -
> - 5) If backslashes immediately precede a double quotation mark,
> - every pair of backslashes is interpreted as a literal
> - backslash. If the number of backslashes is odd, the last
> - backslash escapes the next double quotation mark as
> - described in rule 3.
> - """
> -
> - # See
> - #
> http://msdn.microsoft.com/library/en-us/vccelng/htm/progs_12.asp
> - result = []
> - needquote = False
> - for arg in seq:
> - bs_buf = []
> -
> - # Add a space to separate this argument from the others
> - if result:
> - result.append(' ')
> -
> - needquote = (" " in arg) or ("\t" in arg) or ("|" in arg) or
> arg == ""
> - if needquote:
> - result.append('"')
> -
> - for c in arg:
> - if c == '\\':
> - # Don't know if we need to double yet.
> - bs_buf.append(c)
> - elif c == '"':
> - # Double backspaces.
> - result.append('\\' * len(bs_buf)*2)
> - bs_buf = []
> - result.append('\\"')
> - else:
> - # Normal char
> - if bs_buf:
> - result.extend(bs_buf)
> - bs_buf = []
> - result.append(c)
> -
> - # Add remaining backspaces, if any.
> - if bs_buf:
> - result.extend(bs_buf)
> -
> - if needquote:
> - result.extend(bs_buf)
> - result.append('"')
> -
> - return ''.join(result)
> diff --git a/bitbake/lib/bb/remotedata.py
> b/bitbake/lib/bb/remotedata.py index 68ecffc..7391e1b 100644
> --- a/bitbake/lib/bb/remotedata.py
> +++ b/bitbake/lib/bb/remotedata.py
> @@ -6,18 +6,8 @@ Provides support for using a datastore from the
> bitbake client
> # Copyright (C) 2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import bb.data
>
> diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
> index 9ce06c4..1804943 100644
> --- a/bitbake/lib/bb/runqueue.py
> +++ b/bitbake/lib/bb/runqueue.py
> @@ -1,6 +1,3 @@
> -#!/usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake 'RunQueue' implementation
>
> @@ -9,18 +6,8 @@ Handles preparation and execution of a queue of tasks
>
> # Copyright (C) 2006-2007 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import copy
> import os
> @@ -37,11 +24,13 @@ from bb import monitordisk
> import subprocess
> import pickle
> from multiprocessing import Process
> +import shlex
> +import pprint
>
> bblogger = logging.getLogger("BitBake")
> logger = logging.getLogger("BitBake.RunQueue")
>
> -__find_md5__ =
> re.compile( r'(?i)(?<![a-z0-9])[a-f0-9]{32}(?![a-z0-9])' )
> +__find_sha256__ =
> re.compile( r'(?i)(?<![a-z0-9])[a-f0-9]{64}(?![a-z0-9])' ) def
> fn_from_tid(tid): return tid.rsplit(":", 1)[0]
> @@ -49,17 +38,22 @@ def fn_from_tid(tid):
> def taskname_from_tid(tid):
> return tid.rsplit(":", 1)[1]
>
> +def mc_from_tid(tid):
> + if tid.startswith('mc:'):
> + return tid.split(':')[1]
> + return ""
> +
> def split_tid(tid):
> (mc, fn, taskname, _) = split_tid_mcfn(tid)
> return (mc, fn, taskname)
>
> def split_tid_mcfn(tid):
> - if tid.startswith('multiconfig:'):
> + if tid.startswith('mc:'):
> elems = tid.split(':')
> mc = elems[1]
> fn = ":".join(elems[2:-1])
> taskname = elems[-1]
> - mcfn = "multiconfig:" + mc + ":" + fn
> + mcfn = "mc:" + mc + ":" + fn
> else:
> tid = tid.rsplit(":", 1)
> mc = ""
> @@ -71,9 +65,17 @@ def split_tid_mcfn(tid):
>
> def build_tid(mc, fn, taskname):
> if mc:
> - return "multiconfig:" + mc + ":" + fn + ":" + taskname
> + return "mc:" + mc + ":" + fn + ":" + taskname
> return fn + ":" + taskname
>
> +# Index used to pair up potentially matching multiconfig tasks
> +# We match on PN, taskname and hash being equal
> +def pending_hash_index(tid, rqdata):
> + (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
> + pn = rqdata.dataCaches[mc].pkg_fn[taskfn]
> + h = rqdata.runtaskentries[tid].unihash
> + return pn + ":" + "taskname" + h
> +
> class RunQueueStats:
> """
> Holds statistics on the tasks handled by the associated runQueue
> @@ -109,8 +111,6 @@ class RunQueueStats:
> # runQueue state machine
> runQueuePrepare = 2
> runQueueSceneInit = 3
> -runQueueSceneRun = 4
> -runQueueRunInit = 5
> runQueueRunning = 6
> runQueueFailed = 7
> runQueueCleanUp = 8
> @@ -133,7 +133,7 @@ class RunQueueScheduler(object):
>
> self.prio_map = [self.rqdata.runtaskentries.keys()]
>
> - self.buildable = []
> + self.buildable = set()
> self.skip_maxthread = {}
> self.stamps = {}
> for tid in self.rqdata.runtaskentries:
> @@ -148,8 +148,11 @@ class RunQueueScheduler(object):
> """
> Return the id of the first task we find that is buildable
> """
> - self.buildable = [x for x in self.buildable if x not in
> self.rq.runq_running]
> - if not self.buildable:
> + buildable = set(self.buildable)
> + buildable.difference_update(self.rq.runq_running)
> + buildable.difference_update(self.rq.holdoff_tasks)
> + buildable.intersection_update(self.rq.tasks_covered |
> self.rq.tasks_notcovered)
> + if not buildable:
> return None
>
> # Filter out tasks that have a max number of threads that
> have been exceeded @@ -165,8 +168,8 @@ class
> RunQueueScheduler(object): else:
> skip_buildable[rtaskname] = 1
>
> - if len(self.buildable) == 1:
> - tid = self.buildable[0]
> + if len(buildable) == 1:
> + tid = buildable.pop()
> taskname = taskname_from_tid(tid)
> if taskname in skip_buildable and
> skip_buildable[taskname] >= int(self.skip_maxthread[taskname]):
> return None @@ -181,7 +184,7 @@ class RunQueueScheduler(object):
>
> best = None
> bestprio = None
> - for tid in self.buildable:
> + for tid in buildable:
> taskname = taskname_from_tid(tid)
> if taskname in skip_buildable and
> skip_buildable[taskname] >= int(self.skip_maxthread[taskname]):
> continue @@ -203,7 +206,12 @@ class RunQueueScheduler(object):
> return self.next_buildable_task()
>
> def newbuildable(self, task):
> - self.buildable.append(task)
> + self.buildable.add(task)
> + # Once tasks are running we don't need to worry about them
> again
> + self.buildable.difference_update(self.rq.runq_running)
> +
> + def removebuildable(self, task):
> + self.buildable.remove(task)
>
> def describe_task(self, taskid):
> result = 'ID %s' % taskid
> @@ -346,6 +354,7 @@ class RunTaskEntry(object):
> self.depends = set()
> self.revdeps = set()
> self.hash = None
> + self.unihash = None
> self.task = None
> self.weight = 1
>
> @@ -385,6 +394,9 @@ class RunQueueData:
> def get_task_hash(self, tid):
> return self.runtaskentries[tid].hash
>
> + def get_task_unihash(self, tid):
> + return self.runtaskentries[tid].unihash
> +
> def get_user_idstring(self, tid, task_name_suffix = ""):
> return tid + task_name_suffix
>
> @@ -405,6 +417,9 @@ class RunQueueData:
> explored_deps = {}
> msgs = []
>
> + class TooManyLoops(Exception):
> + pass
> +
> def chain_reorder(chain):
> """
> Reorder a dependency chain so the lowest task id is first
> @@ -457,7 +472,7 @@ class RunQueueData:
> msgs.append("\n")
> if len(valid_chains) > 10:
> msgs.append("Aborted dependency loops search
> after 10 matches.\n")
> - return msgs
> + raise TooManyLoops
> continue
> scan = False
> if revdep not in explored_deps:
> @@ -476,8 +491,11 @@ class RunQueueData:
>
> explored_deps[tid] = total_deps
>
> - for task in tasks:
> - find_chains(task, [])
> + try:
> + for task in tasks:
> + find_chains(task, [])
> + except TooManyLoops:
> + pass
>
> return msgs
>
> @@ -833,6 +851,20 @@ class RunQueueData:
> for depend in depends:
> mark_active(depend, depth+1)
>
> + def invalidate_task(tid, error_nostamp):
> + (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
> + taskdep = self.dataCaches[mc].task_deps[taskfn]
> + if fn + ":" + taskname not in taskData[mc].taskentries:
> + logger.warning("Task %s does not exist, invalidating
> this task will have no effect" % taskname)
> + if 'nostamp' in taskdep and taskname in
> taskdep['nostamp']:
> + if error_nostamp:
> + bb.fatal("Task %s is marked nostamp, cannot
> invalidate this task" % taskname)
> + else:
> + bb.debug(1, "Task %s is marked nostamp, cannot
> invalidate this task" % taskname)
> + else:
> + logger.verbose("Invalidate task %s, %s", taskname,
> fn)
> + bb.parse.siggen.invalidate_task(taskname,
> self.dataCaches[mc], taskfn) +
> self.target_tids = []
> for (mc, target, task, fn) in self.targets:
>
> @@ -901,6 +933,8 @@ class RunQueueData:
>
> for tid in list(runall_tids):
> mark_active(tid,1)
> + if self.cooker.configuration.force:
> + invalidate_task(tid, False)
>
> for tid in list(self.runtaskentries.keys()):
> if tid not in runq_build:
> @@ -922,6 +956,8 @@ class RunQueueData:
>
> for tid in list(runonly_tids):
> mark_active(tid,1)
> + if self.cooker.configuration.force:
> + invalidate_task(tid, False)
>
> for tid in list(self.runtaskentries.keys()):
> if tid not in runq_build:
> @@ -1098,20 +1134,6 @@ class RunQueueData:
> continue
> self.runq_setscene_tids.append(tid)
>
> - def invalidate_task(tid, error_nostamp):
> - (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
> - taskdep = self.dataCaches[mc].task_deps[taskfn]
> - if fn + ":" + taskname not in taskData[mc].taskentries:
> - logger.warning("Task %s does not exist, invalidating
> this task will have no effect" % taskname)
> - if 'nostamp' in taskdep and taskname in
> taskdep['nostamp']:
> - if error_nostamp:
> - bb.fatal("Task %s is marked nostamp, cannot
> invalidate this task" % taskname)
> - else:
> - bb.debug(1, "Task %s is marked nostamp, cannot
> invalidate this task" % taskname)
> - else:
> - logger.verbose("Invalidate task %s, %s", taskname,
> fn)
> - bb.parse.siggen.invalidate_task(taskname,
> self.dataCaches[mc], taskfn) -
> self.init_progress_reporter.next_stage()
>
> # Invalidate task if force mode active
> @@ -1142,6 +1164,8 @@ class RunQueueData:
>
> self.init_progress_reporter.next_stage()
>
> + bb.parse.siggen.set_setscene_tasks(self.runq_setscene_tids)
> +
> # Iterate over the task list and call into the siggen code
> dealtwith = set()
> todeal = set(self.runtaskentries)
> @@ -1150,18 +1174,20 @@ class RunQueueData:
> if len(self.runtaskentries[tid].depends - dealtwith)
> == 0: dealtwith.add(tid)
> todeal.remove(tid)
> - procdep = []
> - for dep in self.runtaskentries[tid].depends:
> - procdep.append(fn_from_tid(dep) + "." +
> taskname_from_tid(dep))
> - (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
> - self.runtaskentries[tid].hash =
> bb.parse.siggen.get_taskhash(taskfn, taskname, procdep,
> self.dataCaches[mc])
> - task = self.runtaskentries[tid].task
> + self.prepare_task_hash(tid)
>
> bb.parse.siggen.writeout_file_checksum_cache()
>
> #self.dump_data()
> return len(self.runtaskentries)
>
> + def prepare_task_hash(self, tid):
> + procdep = []
> + for dep in self.runtaskentries[tid].depends:
> + procdep.append(dep)
> + self.runtaskentries[tid].hash =
> bb.parse.siggen.get_taskhash(tid, procdep,
> self.dataCaches[mc_from_tid(tid)])
> + self.runtaskentries[tid].unihash =
> bb.parse.siggen.get_unihash(tid) +
> def dump_data(self):
> """
> Dump some debug information on the internal data structures
> @@ -1187,7 +1213,6 @@ class RunQueue:
>
> self.stamppolicy = cfgData.getVar("BB_STAMP_POLICY") or
> "perfile" self.hashvalidate = cfgData.getVar("BB_HASHCHECK_FUNCTION")
> or None
> - self.setsceneverify =
> cfgData.getVar("BB_SETSCENE_VERIFY_FUNCTION2") or None
> self.depvalidate = cfgData.getVar("BB_SETSCENE_DEPVALID") or None
> self.state = runQueuePrepare
> @@ -1196,7 +1221,7 @@ class RunQueue:
> # Invoked at regular time intervals via the bitbake
> heartbeat event # while the build is running. We generate a unique
> name for the handler # here, just in case that there ever is more
> than one RunQueue instance,
> - # start the handler when reaching runQueueSceneRun, and stop
> it when
> + # start the handler when reaching runQueueSceneInit, and
> stop it when # done with the build.
> self.dm = monitordisk.diskMonitor(cfgData)
> self.dm_event_handler_name = '_bb_diskmonitor_' +
> str(id(self)) @@ -1213,28 +1238,23 @@ class RunQueue:
> if fakeroot:
> magic = magic + "beef"
> mcdata = self.cooker.databuilder.mcdata[mc]
> - fakerootcmd = mcdata.getVar("FAKEROOTCMD")
> + fakerootcmd = shlex.split(mcdata.getVar("FAKEROOTCMD"))
> fakerootenv = (mcdata.getVar("FAKEROOTBASEENV") or
> "").split() env = os.environ.copy()
> for key, value in (var.split('=') for var in
> fakerootenv): env[key] = value
> - worker = subprocess.Popen([fakerootcmd,
> "bitbake-worker", magic], stdout=subprocess.PIPE,
> stdin=subprocess.PIPE, env=env)
> + worker = subprocess.Popen(fakerootcmd +
> ["bitbake-worker", magic], stdout=subprocess.PIPE,
> stdin=subprocess.PIPE, env=env) else: worker =
> subprocess.Popen(["bitbake-worker", magic], stdout=subprocess.PIPE,
> stdin=subprocess.PIPE) bb.utils.nonblockingfd(worker.stdout)
> workerpipe = runQueuePipe(worker.stdout, None, self.cfgData, self,
> rqexec)
> - runqhash = {}
> - for tid in self.rqdata.runtaskentries:
> - runqhash[tid] = self.rqdata.runtaskentries[tid].hash
> -
> workerdata = {
> "taskdeps" : self.rqdata.dataCaches[mc].task_deps,
> "fakerootenv" : self.rqdata.dataCaches[mc].fakerootenv,
> "fakerootdirs" : self.rqdata.dataCaches[mc].fakerootdirs,
> "fakerootnoenv" :
> self.rqdata.dataCaches[mc].fakerootnoenv, "sigdata" :
> bb.parse.siggen.get_taskdata(),
> - "runq_hash" : runqhash,
> "logdefaultdebug" : bb.msg.loggerDefaultDebugLevel,
> "logdefaultverbose" : bb.msg.loggerDefaultVerbose,
> "logdefaultverboselogs" : bb.msg.loggerVerboseLogs,
> @@ -1243,6 +1263,7 @@ class RunQueue:
> "buildname" : self.cfgData.getVar("BUILDNAME"),
> "date" : self.cfgData.getVar("DATE"),
> "time" : self.cfgData.getVar("TIME"),
> + "hashservaddr" : self.cooker.hashservaddr,
> }
>
> worker.stdin.write(b"<cookerconfig>" +
> pickle.dumps(self.cooker.configuration) + b"</cookerconfig>") @@
> -1376,6 +1397,31 @@ class RunQueue: cache[tid] = iscurrent
> return iscurrent
>
> + def validate_hashes(self, tocheck, data, currentcount=0,
> siginfo=False):
> + valid = set()
> + if self.hashvalidate:
> + sq_data = {}
> + sq_data['hash'] = {}
> + sq_data['hashfn'] = {}
> + sq_data['unihash'] = {}
> + for tid in tocheck:
> + (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
> + sq_data['hash'][tid] =
> self.rqdata.runtaskentries[tid].hash
> + sq_data['hashfn'][tid] =
> self.rqdata.dataCaches[mc].hashfn[taskfn]
> + sq_data['unihash'][tid] =
> self.rqdata.runtaskentries[tid].unihash +
> + valid = self.validate_hash(sq_data, data, siginfo,
> currentcount) +
> + return valid
> +
> + def validate_hash(self, sq_data, d, siginfo, currentcount):
> + locs = {"sq_data" : sq_data, "d" : d, "siginfo" : siginfo,
> "currentcount" : currentcount} +
> + # Metadata has **kwargs so args can be added, sq_data can
> also gain new fields
> + call = self.hashvalidate + "(sq_data, d, siginfo=siginfo,
> currentcount=currentcount)" +
> + return bb.utils.better_eval(call, locs)
> +
> def _execute_runqueue(self):
> """
> Run the tasks in a queue prepared by rqdata.prepare()
> @@ -1386,7 +1432,6 @@ class RunQueue:
> retval = True
>
> if self.state is runQueuePrepare:
> - self.rqexe = RunQueueExecuteDummy(self)
> # NOTE: if you add, remove or significantly refactor the
> stages of this # process then you should recalculate the weightings
> here. This is quite # easy to do - just change the next line
> temporarily to pass debug=True as @@ -1400,15 +1445,23 @@ class
> RunQueue: self.state = runQueueComplete
> else:
> self.state = runQueueSceneInit
> - self.rqdata.init_progress_reporter.next_stage()
> -
> - # we are ready to run, emit dependency info to any
> UI or class which
> - # needs it
> - depgraph = self.cooker.buildDependTree(self,
> self.rqdata.taskData)
> - self.rqdata.init_progress_reporter.next_stage()
> - bb.event.fire(bb.event.DepTreeGenerated(depgraph),
> self.cooker.data)
> + bb.parse.siggen.save_unitaskhashes()
>
> if self.state is runQueueSceneInit:
> + self.rqdata.init_progress_reporter.next_stage()
> +
> + # we are ready to run, emit dependency info to any UI
> or class which
> + # needs it
> + depgraph = self.cooker.buildDependTree(self,
> self.rqdata.taskData)
> + self.rqdata.init_progress_reporter.next_stage()
> + bb.event.fire(bb.event.DepTreeGenerated(depgraph),
> self.cooker.data) +
> + if not self.dm_event_handler_registered:
> + res = bb.event.register(self.dm_event_handler_name,
> + lambda x:
> self.dm.check(self) if self.state in [runQueueRunning,
> runQueueCleanUp] else False,
> +
> ('bb.event.HeartbeatEvent',))
> + self.dm_event_handler_registered = True
> +
> dump = self.cooker.configuration.dump_signatures
> if dump:
> self.rqdata.init_progress_reporter.finish()
> @@ -1418,29 +1471,23 @@ class RunQueue:
> if 'printdiff' in dump:
> self.write_diffscenetasks(invalidtasks)
> self.state = runQueueComplete
> - else:
> - self.rqdata.init_progress_reporter.next_stage()
> - self.start_worker()
> - self.rqdata.init_progress_reporter.next_stage()
> - self.rqexe = RunQueueExecuteScenequeue(self)
> -
> - if self.state is runQueueSceneRun:
> - if not self.dm_event_handler_registered:
> - res = bb.event.register(self.dm_event_handler_name,
> - lambda x:
> self.dm.check(self) if self.state in [runQueueSceneRun,
> runQueueRunning, runQueueCleanUp] else False,
> -
> ('bb.event.HeartbeatEvent',))
> - self.dm_event_handler_registered = True
> - retval = self.rqexe.execute()
>
> - if self.state is runQueueRunInit:
> - if self.cooker.configuration.setsceneonly:
> - self.state = runQueueComplete
> - else:
> - # Just in case we didn't setscene
> - self.rqdata.init_progress_reporter.finish()
> - logger.info("Executing RunQueue Tasks")
> - self.rqexe = RunQueueExecuteTasks(self)
> - self.state = runQueueRunning
> + if self.state is runQueueSceneInit:
> + self.rqdata.init_progress_reporter.next_stage()
> + self.start_worker()
> + self.rqdata.init_progress_reporter.next_stage()
> + self.rqexe = RunQueueExecute(self)
> +
> + # If we don't have any setscene functions, skip execution
> + if len(self.rqdata.runq_setscene_tids) == 0:
> + logger.info('No setscene tasks')
> + for tid in self.rqdata.runtaskentries:
> + if len(self.rqdata.runtaskentries[tid].depends)
> == 0:
> + self.rqexe.setbuildable(tid)
> + self.rqexe.tasks_notcovered.add(tid)
> + self.rqexe.sqdone = True
> + logger.info('Executing Tasks')
> + self.state = runQueueRunning
>
> if self.state is runQueueRunning:
> retval = self.rqexe.execute()
> @@ -1455,12 +1502,14 @@ class RunQueue:
> self.dm_event_handler_registered = False
>
> if build_done and self.rqexe:
> + bb.parse.siggen.save_unitaskhashes()
> self.teardown_workers()
> - if self.rqexe.stats.failed:
> - logger.info("Tasks Summary: Attempted %d tasks of
> which %d didn't need to be rerun and %d failed.",
> self.rqexe.stats.completed + self.rqexe.stats.failed,
> self.rqexe.stats.skipped, self.rqexe.stats.failed)
> - else:
> - # Let's avoid the word "failed" if nothing actually
> did
> - logger.info("Tasks Summary: Attempted %d tasks of
> which %d didn't need to be rerun and all succeeded.",
> self.rqexe.stats.completed, self.rqexe.stats.skipped)
> + if self.rqexe:
> + if self.rqexe.stats.failed:
> + logger.info("Tasks Summary: Attempted %d tasks
> of which %d didn't need to be rerun and %d failed.",
> self.rqexe.stats.completed + self.rqexe.stats.failed,
> self.rqexe.stats.skipped, self.rqexe.stats.failed)
> + else:
> + # Let's avoid the word "failed" if nothing
> actually did
> + logger.info("Tasks Summary: Attempted %d tasks
> of which %d didn't need to be rerun and all succeeded.",
> self.rqexe.stats.completed, self.rqexe.stats.skipped) if self.state
> is runQueueFailed: raise
> bb.runqueue.TaskFailure(self.rqexe.failed_tids) @@ -1543,15 +1592,8
> @@ class RunQueue:
> def print_diffscenetasks(self):
>
> - valid = []
> - sq_hash = []
> - sq_hashfn = []
> - sq_fn = []
> - sq_taskname = []
> - sq_task = []
> noexec = []
> - stamppresent = []
> - valid_new = set()
> + tocheck = set()
>
> for tid in self.rqdata.runtaskentries:
> (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
> @@ -1561,21 +1603,9 @@ class RunQueue:
> noexec.append(tid)
> continue
>
> - sq_fn.append(fn)
> -
> sq_hashfn.append(self.rqdata.dataCaches[mc].hashfn[taskfn])
> - sq_hash.append(self.rqdata.runtaskentries[tid].hash)
> - sq_taskname.append(taskname)
> - sq_task.append(tid)
> - locs = { "sq_fn" : sq_fn, "sq_task" : sq_taskname,
> "sq_hash" : sq_hash, "sq_hashfn" : sq_hashfn, "d" : self.cooker.data }
> - try:
> - call = self.hashvalidate + "(sq_fn, sq_task, sq_hash,
> sq_hashfn, d, siginfo=True)"
> - valid = bb.utils.better_eval(call, locs)
> - # Handle version with no siginfo parameter
> - except TypeError:
> - call = self.hashvalidate + "(sq_fn, sq_task, sq_hash,
> sq_hashfn, d)"
> - valid = bb.utils.better_eval(call, locs)
> - for v in valid:
> - valid_new.add(sq_task[v])
> + tocheck.add(tid)
> +
> + valid_new = self.validate_hashes(tocheck, self.cooker.data,
> 0, True)
> # Tasks which are both setscene and noexec never care about
> dependencies # We therefore find tasks which are setscene and noexec
> and mark their @@ -1634,7 +1664,7 @@ class RunQueue:
> recout = []
> if len(hashfiles) == 2:
> out2 = bb.siggen.compare_sigfiles(hashfiles[hash1],
> hashfiles[hash2], recursecb)
> - recout.extend(list(' ' + l for l in out2))
> + recout.extend(list(' ' + l for l in out2))
> else:
> recout.append("Unable to find matching sigdata for
> %s with hashes %s or %s" % (key, hash1, hash2))
> @@ -1655,10 +1685,11 @@ class RunQueue:
> matches = {k : v for k, v in iter(matches.items()) if h
> not in k} if matches:
> latestmatch = sorted(matches.keys(), key=lambda f:
> matches[f])[-1]
> - prevh = __find_md5__.search(latestmatch).group(0)
> + prevh = __find_sha256__.search(latestmatch).group(0)
> output = bb.siggen.compare_sigfiles(latestmatch,
> match, recursecb) bb.plain("\nTask %s:%s couldn't be used from the
> cache because:\n We need hash %s, closest matching task was %s\n "
> % (pn, taskname, h, prevh) + '\n '.join(output)) +
> class RunQueueExecute:
>
> def __init__(self, rq):
> @@ -1670,6 +1701,13 @@ class RunQueueExecute:
> self.number_tasks =
> int(self.cfgData.getVar("BB_NUMBER_THREADS") or 1) self.scheduler =
> self.cfgData.getVar("BB_SCHEDULER") or "speed"
> + self.sq_buildable = set()
> + self.sq_running = set()
> + self.sq_live = set()
> +
> + self.updated_taskhash_queue = []
> + self.pending_migrations = set()
> +
> self.runq_buildable = set()
> self.runq_running = set()
> self.runq_complete = set()
> @@ -1677,9 +1715,17 @@ class RunQueueExecute:
> self.build_stamps = {}
> self.build_stamps2 = []
> self.failed_tids = []
> + self.sq_deferred = {}
>
> self.stampcache = {}
>
> + self.holdoff_tasks = set()
> + self.holdoff_need_update = True
> + self.sqdone = False
> +
> + self.stats = RunQueueStats(len(self.rqdata.runtaskentries))
> + self.sq_stats =
> RunQueueStats(len(self.rqdata.runq_setscene_tids)) +
> for mc in rq.worker:
> rq.worker[mc].pipe.setrunqueueexec(self)
> for mc in rq.fakeworker:
> @@ -1688,6 +1734,34 @@ class RunQueueExecute:
> if self.number_tasks <= 0:
> bb.fatal("Invalid BB_NUMBER_THREADS %s" %
> self.number_tasks)
> + # List of setscene tasks which we've covered
> + self.scenequeue_covered = set()
> + # List of tasks which are covered (including setscene ones)
> + self.tasks_covered = set()
> + self.tasks_scenequeue_done = set()
> + self.scenequeue_notcovered = set()
> + self.tasks_notcovered = set()
> + self.scenequeue_notneeded = set()
> +
> + # We can't skip specified target tasks which aren't setscene
> tasks
> + self.cantskip = set(self.rqdata.target_tids)
> +
> self.cantskip.difference_update(self.rqdata.runq_setscene_tids)
> + self.cantskip.intersection_update(self.rqdata.runtaskentries)
> +
> + schedulers = self.get_schedulers()
> + for scheduler in schedulers:
> + if self.scheduler == scheduler.name:
> + self.sched = scheduler(self, self.rqdata)
> + logger.debug(1, "Using runqueue scheduler '%s'",
> scheduler.name)
> + break
> + else:
> + bb.fatal("Invalid scheduler '%s'. Available schedulers:
> %s" %
> + (self.scheduler, ", ".join(obj.name for obj in
> schedulers))) +
> + #if len(self.rqdata.runq_setscene_tids) > 0:
> + self.sqdata = SQData()
> + build_scenequeue_data(self.sqdata, self.rqdata, self.rq,
> self.cooker, self.stampcache, self) +
> def runqueue_process_waitpid(self, task, status):
>
> # self.build_stamps[pid] may not exist when use shared work
> directory. @@ -1695,10 +1769,17 @@ class RunQueueExecute:
> self.build_stamps2.remove(self.build_stamps[task])
> del self.build_stamps[task]
>
> - if status != 0:
> - self.task_fail(task, status)
> + if task in self.sq_live:
> + if status != 0:
> + self.sq_task_fail(task, status)
> + else:
> + self.sq_task_complete(task)
> + self.sq_live.remove(task)
> else:
> - self.task_complete(task)
> + if status != 0:
> + self.task_fail(task, status)
> + else:
> + self.task_complete(task)
> return True
>
> def finish_now(self):
> @@ -1727,8 +1808,9 @@ class RunQueueExecute:
> def finish(self):
> self.rq.state = runQueueCleanUp
>
> - if self.stats.active > 0:
> - bb.event.fire(runQueueExitWait(self.stats.active),
> self.cfgData)
> + active = self.stats.active + self.sq_stats.active
> + if active > 0:
> + bb.event.fire(runQueueExitWait(active), self.cfgData)
> self.rq.read_workers()
> return self.rq.active_fds()
>
> @@ -1739,10 +1821,14 @@ class RunQueueExecute:
> self.rq.state = runQueueComplete
> return True
>
> - def check_dependencies(self, task, taskdeps, setscene = False):
> + # Used by setscene only
> + def check_dependencies(self, task, taskdeps):
> if not self.rq.depvalidate:
> return False
>
> + # Must not edit parent data
> + taskdeps = set(taskdeps)
> +
> taskdata = {}
> taskdeps.add(task)
> for dep in taskdeps:
> @@ -1755,121 +1841,10 @@ class RunQueueExecute:
> return valid
>
> def can_start_task(self):
> - can_start = self.stats.active < self.number_tasks
> + active = self.stats.active + self.sq_stats.active
> + can_start = active < self.number_tasks
> return can_start
>
> -class RunQueueExecuteDummy(RunQueueExecute):
> - def __init__(self, rq):
> - self.rq = rq
> - self.stats = RunQueueStats(0)
> -
> - def finish(self):
> - self.rq.state = runQueueComplete
> - return
> -
> -class RunQueueExecuteTasks(RunQueueExecute):
> - def __init__(self, rq):
> - RunQueueExecute.__init__(self, rq)
> -
> - self.stats = RunQueueStats(len(self.rqdata.runtaskentries))
> -
> - self.stampcache = {}
> -
> - initial_covered = self.rq.scenequeue_covered.copy()
> -
> - # Mark initial buildable tasks
> - for tid in self.rqdata.runtaskentries:
> - if len(self.rqdata.runtaskentries[tid].depends) == 0:
> - self.runq_buildable.add(tid)
> - if len(self.rqdata.runtaskentries[tid].revdeps) > 0 and
> self.rqdata.runtaskentries[tid].revdeps.issubset(self.rq.scenequeue_covered):
> - self.rq.scenequeue_covered.add(tid)
> -
> - found = True
> - while found:
> - found = False
> - for tid in self.rqdata.runtaskentries:
> - if tid in self.rq.scenequeue_covered:
> - continue
> - logger.debug(1, 'Considering %s: %s' % (tid,
> str(self.rqdata.runtaskentries[tid].revdeps))) -
> - if len(self.rqdata.runtaskentries[tid].revdeps) > 0
> and
> self.rqdata.runtaskentries[tid].revdeps.issubset(self.rq.scenequeue_covered):
> - if tid in self.rq.scenequeue_notcovered:
> - continue
> - found = True
> - self.rq.scenequeue_covered.add(tid)
> -
> - logger.debug(1, 'Skip list (pre setsceneverify) %s',
> sorted(self.rq.scenequeue_covered)) -
> - # Allow the metadata to elect for setscene tasks to run
> anyway
> - covered_remove = set()
> - if self.rq.setsceneverify:
> - invalidtasks = []
> - tasknames = {}
> - fns = {}
> - for tid in self.rqdata.runtaskentries:
> - (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
> - taskdep =
> self.rqdata.dataCaches[mc].task_deps[taskfn]
> - fns[tid] = taskfn
> - tasknames[tid] = taskname
> - if 'noexec' in taskdep and taskname in
> taskdep['noexec']:
> - continue
> - if self.rq.check_stamp_task(tid, taskname +
> "_setscene", cache=self.stampcache):
> - logger.debug(2, 'Setscene stamp current for task
> %s', tid)
> - continue
> - if self.rq.check_stamp_task(tid, taskname, recurse =
> True, cache=self.stampcache):
> - logger.debug(2, 'Normal stamp current for task
> %s', tid)
> - continue
> - invalidtasks.append(tid)
> -
> - call = self.rq.setsceneverify + "(covered, tasknames,
> fns, d, invalidtasks=invalidtasks)"
> - locs = { "covered" : self.rq.scenequeue_covered,
> "tasknames" : tasknames, "fns" : fns, "d" : self.cooker.data,
> "invalidtasks" : invalidtasks }
> - covered_remove = bb.utils.better_eval(call, locs)
> -
> - def removecoveredtask(tid):
> - (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
> - taskname = taskname + '_setscene'
> - bb.build.del_stamp(taskname, self.rqdata.dataCaches[mc],
> taskfn)
> - self.rq.scenequeue_covered.remove(tid)
> -
> - toremove = covered_remove | self.rq.scenequeue_notcovered
> - for task in toremove:
> - logger.debug(1, 'Not skipping task %s due to
> setsceneverify', task)
> - while toremove:
> - covered_remove = []
> - for task in toremove:
> - if task in self.rq.scenequeue_covered:
> - removecoveredtask(task)
> - for deptask in
> self.rqdata.runtaskentries[task].depends:
> - if deptask not in self.rq.scenequeue_covered:
> - continue
> - if deptask in toremove or deptask in
> covered_remove or deptask in initial_covered:
> - continue
> - logger.debug(1, 'Task %s depends on task %s so
> not skipping' % (task, deptask))
> - covered_remove.append(deptask)
> - toremove = covered_remove
> -
> - logger.debug(1, 'Full skip list %s',
> self.rq.scenequeue_covered) -
> -
> - for mc in self.rqdata.dataCaches:
> - target_pairs = []
> - for tid in self.rqdata.target_tids:
> - (tidmc, fn, taskname, _) = split_tid_mcfn(tid)
> - if tidmc == mc:
> - target_pairs.append((fn, taskname))
> -
> - event.fire(bb.event.StampUpdate(target_pairs,
> self.rqdata.dataCaches[mc].stamp), self.cfgData) -
> - schedulers = self.get_schedulers()
> - for scheduler in schedulers:
> - if self.scheduler == scheduler.name:
> - self.sched = scheduler(self, self.rqdata)
> - logger.debug(1, "Using runqueue scheduler '%s'",
> scheduler.name)
> - break
> - else:
> - bb.fatal("Invalid scheduler '%s'. Available schedulers:
> %s" %
> - (self.scheduler, ", ".join(obj.name for obj in
> schedulers))) -
> def get_schedulers(self):
> schedulers = set(obj for obj in globals().values()
> if type(obj) is type and
> @@ -1941,67 +1916,172 @@ class RunQueueExecuteTasks(RunQueueExecute):
> self.stats.taskSkipped()
> self.stats.taskCompleted()
>
> + def summarise_scenequeue_errors(self):
> + err = False
> + if not self.sqdone:
> + logger.debug(1, 'We could skip tasks %s',
> "\n".join(sorted(self.scenequeue_covered)))
> + completeevent = sceneQueueComplete(self.sq_stats,
> self.rq)
> + bb.event.fire(completeevent, self.cfgData)
> + if self.sq_deferred:
> + logger.error("Scenequeue had deferred entries: %s" %
> pprint.pformat(self.sq_deferred))
> + err = True
> + if self.updated_taskhash_queue:
> + logger.error("Scenequeue had unprocessed changed
> taskhash entries: %s" % pprint.pformat(self.updated_taskhash_queue))
> + err = True
> + if self.holdoff_tasks:
> + logger.error("Scenequeue had holdoff tasks: %s" %
> pprint.pformat(self.holdoff_tasks))
> + err = True
> +
> + for tid in self.rqdata.runq_setscene_tids:
> + if tid not in self.scenequeue_covered and tid not in
> self.scenequeue_notcovered:
> + err = True
> + logger.error("Setscene Task %s was never marked as
> covered or not covered" % tid)
> + if tid not in self.sq_buildable:
> + err = True
> + logger.error("Setscene Task %s was never marked as
> buildable" % tid)
> + if tid not in self.sq_running:
> + err = True
> + logger.error("Setscene Task %s was never marked as
> running" % tid) +
> + for x in self.rqdata.runtaskentries:
> + if x not in self.tasks_covered and x not in
> self.tasks_notcovered:
> + logger.error("Task %s was never moved from the
> setscene queue" % x)
> + err = True
> + if x not in self.tasks_scenequeue_done:
> + logger.error("Task %s was never processed by the
> setscene code" % x)
> + err = True
> + if len(self.rqdata.runtaskentries[x].depends) == 0 and x
> not in self.runq_buildable:
> + logger.error("Task %s was never marked as buildable
> by the setscene code" % x)
> + err = True
> + return err
> +
> +
> def execute(self):
> """
> - Run the tasks in a queue prepared by rqdata.prepare()
> + Run the tasks in a queue prepared by prepare_runqueue
> """
>
> - if self.rqdata.setscenewhitelist is not None and not
> self.rqdata.setscenewhitelist_checked:
> - self.rqdata.setscenewhitelist_checked = True
> -
> - # Check tasks that are going to run against the whitelist
> - def check_norun_task(tid, showerror=False):
> - (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
> - # Ignore covered tasks
> - if tid in self.rq.scenequeue_covered:
> - return False
> - # Ignore stamped tasks
> - if self.rq.check_stamp_task(tid, taskname,
> cache=self.stampcache):
> - return False
> - # Ignore noexec tasks
> - taskdep =
> self.rqdata.dataCaches[mc].task_deps[taskfn]
> - if 'noexec' in taskdep and taskname in
> taskdep['noexec']:
> - return False
> + self.rq.read_workers()
> + self.process_possible_migrations()
>
> - pn = self.rqdata.dataCaches[mc].pkg_fn[taskfn]
> - if not check_setscene_enforce_whitelist(pn,
> taskname, self.rqdata.setscenewhitelist):
> - if showerror:
> - if tid in self.rqdata.runq_setscene_tids:
> - logger.error('Task %s.%s attempted to
> execute unexpectedly and should have been setscened' % (pn, taskname))
> + task = None
> + if not self.sqdone and self.can_start_task():
> + # Find the next setscene to run
> + for nexttask in sorted(self.rqdata.runq_setscene_tids):
> + if nexttask in self.sq_buildable and nexttask not in
> self.sq_running and self.sqdata.stamps[nexttask] not in
> self.build_stamps.values():
> + if nexttask not in self.sqdata.unskippable and
> len(self.sqdata.sq_revdeps[nexttask]) > 0 and
> self.sqdata.sq_revdeps[nexttask].issubset(self.scenequeue_covered)
> and self.check_dependencies(nexttask,
> self.sqdata.sq_revdeps[nexttask]):
> + if nexttask not in self.rqdata.target_tids:
> + logger.debug(2, "Skipping setscene for
> task %s" % nexttask)
> + self.sq_task_skip(nexttask)
> + self.scenequeue_notneeded.add(nexttask)
> + if nexttask in self.sq_deferred:
> + del self.sq_deferred[nexttask]
> + return True
> + # If covered tasks are running, need to wait for
> them to complete
> + for t in self.sqdata.sq_covered_tasks[nexttask]:
> + if t in self.runq_running and t not in
> self.runq_complete:
> + continue
> + if nexttask in self.sq_deferred:
> + if self.sq_deferred[nexttask] not in
> self.runq_complete:
> + continue
> + logger.debug(1, "Task %s no longer deferred"
> % nexttask)
> + del self.sq_deferred[nexttask]
> + valid =
> self.rq.validate_hashes(set([nexttask]), self.cooker.data, 0, False)
> + if not valid:
> + logger.debug(1, "%s didn't become valid,
> skipping setscene" % nexttask)
> + self.sq_task_failoutright(nexttask)
> + return True
> else:
> - logger.error('Task %s.%s attempted to
> execute unexpectedly' % (pn, taskname))
> - return True
> - return False
> - # Look to see if any tasks that we think shouldn't run
> are going to
> - unexpected = False
> - for tid in self.rqdata.runtaskentries:
> - if check_norun_task(tid):
> - unexpected = True
> + self.sqdata.outrightfail.remove(nexttask)
> + if nexttask in self.sqdata.outrightfail:
> + logger.debug(2, 'No package found, so
> skipping setscene task %s', nexttask)
> + self.sq_task_failoutright(nexttask)
> + return True
> + if nexttask in self.sqdata.unskippable:
> + logger.debug(2, "Setscene task %s is
> unskippable" % nexttask)
> + task = nexttask
> break
> - if unexpected:
> - # Run through the tasks in the rough order they'd
> have executed and print errors
> - # (since the order can be useful - usually missing
> sstate for the last few tasks
> - # is the cause of the problem)
> - task = self.sched.next()
> - while task is not None:
> - check_norun_task(task, showerror=True)
> - self.task_skip(task, 'Setscene enforcement
> check')
> - task = self.sched.next()
> + if task is not None:
> + (mc, fn, taskname, taskfn) = split_tid_mcfn(task)
> + taskname = taskname + "_setscene"
> + if self.rq.check_stamp_task(task,
> taskname_from_tid(task), recurse = True, cache=self.stampcache):
> + logger.debug(2, 'Stamp for underlying task %s is
> current, so skipping setscene variant', task)
> + self.sq_task_failoutright(task)
> + return True
>
> - self.rq.state = runQueueCleanUp
> + if self.cooker.configuration.force:
> + if task in self.rqdata.target_tids:
> + self.sq_task_failoutright(task)
> + return True
> +
> + if self.rq.check_stamp_task(task, taskname,
> cache=self.stampcache):
> + logger.debug(2, 'Setscene stamp current task %s, so
> skip it and its dependencies', task)
> + self.sq_task_skip(task)
> return True
>
> - self.rq.read_workers()
> + if self.cooker.configuration.skipsetscene:
> + logger.debug(2, 'No setscene tasks should be
> executed. Skipping %s', task)
> + self.sq_task_failoutright(task)
> + return True
>
> - if self.stats.total == 0:
> - # nothing to do
> - self.rq.state = runQueueCleanUp
> + startevent = sceneQueueTaskStarted(task, self.sq_stats,
> self.rq)
> + bb.event.fire(startevent, self.cfgData)
> +
> + taskdepdata = self.sq_build_taskdepdata(task)
> +
> + taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn]
> + taskhash = self.rqdata.get_task_hash(task)
> + unihash = self.rqdata.get_task_unihash(task)
> + if 'fakeroot' in taskdep and taskname in
> taskdep['fakeroot'] and not self.cooker.configuration.dry_run:
> + if not mc in self.rq.fakeworker:
> + self.rq.start_fakeworker(self, mc)
> +
> self.rq.fakeworker[mc].process.stdin.write(b"<runtask>" +
> pickle.dumps((taskfn, task, taskname, taskhash, unihash, True,
> self.cooker.collection.get_file_appends(taskfn), taskdepdata, False))
> + b"</runtask>")
> + self.rq.fakeworker[mc].process.stdin.flush()
> + else:
> + self.rq.worker[mc].process.stdin.write(b"<runtask>"
> + pickle.dumps((taskfn, task, taskname, taskhash, unihash, True,
> self.cooker.collection.get_file_appends(taskfn), taskdepdata, False))
> + b"</runtask>")
> + self.rq.worker[mc].process.stdin.flush()
>
> - task = self.sched.next()
> + self.build_stamps[task] = bb.build.stampfile(taskname,
> self.rqdata.dataCaches[mc], taskfn, noextra=True)
> + self.build_stamps2.append(self.build_stamps[task])
> + self.sq_running.add(task)
> + self.sq_live.add(task)
> + self.sq_stats.taskActive()
> + if self.can_start_task():
> + return True
> +
> + self.update_holdofftasks()
> +
> + if not self.sq_live and not self.sqdone and not
> self.sq_deferred and not self.updated_taskhash_queue and not
> self.holdoff_tasks:
> + logger.info("Setscene tasks completed")
> +
> + err = self.summarise_scenequeue_errors()
> + if err:
> + self.rq.state = runQueueFailed
> + return True
> +
> + if self.cooker.configuration.setsceneonly:
> + self.rq.state = runQueueComplete
> + return True
> + self.sqdone = True
> +
> + if self.stats.total == 0:
> + # nothing to do
> + self.rq.state = runQueueComplete
> + return True
> +
> + if self.cooker.configuration.setsceneonly:
> + task = None
> + else:
> + task = self.sched.next()
> if task is not None:
> (mc, fn, taskname, taskfn) = split_tid_mcfn(task)
>
> - if task in self.rq.scenequeue_covered:
> + if self.rqdata.setscenewhitelist is not None:
> + if self.check_setscenewhitelist(task):
> + self.task_fail(task, "setscene whitelist")
> + return True
> +
> + if task in self.tasks_covered:
> logger.debug(2, "Setscene covered task %s", task)
> self.task_skip(task, "covered")
> return True
> @@ -2030,6 +2110,8 @@ class RunQueueExecuteTasks(RunQueueExecute):
> taskdepdata = self.build_taskdepdata(task)
>
> taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn]
> + taskhash = self.rqdata.get_task_hash(task)
> + unihash = self.rqdata.get_task_unihash(task)
> if 'fakeroot' in taskdep and taskname in
> taskdep['fakeroot'] and not (self.cooker.configuration.dry_run or
> self.rqdata.setscene_enforce): if not mc in self.rq.fakeworker: try:
> @@ -2039,10 +2121,10 @@ class RunQueueExecuteTasks(RunQueueExecute):
> self.rq.state = runQueueFailed
> self.stats.taskFailed()
> return True
> -
> self.rq.fakeworker[mc].process.stdin.write(b"<runtask>" +
> pickle.dumps((taskfn, task, taskname, False,
> self.cooker.collection.get_file_appends(taskfn), taskdepdata,
> self.rqdata.setscene_enforce)) + b"</runtask>")
> +
> self.rq.fakeworker[mc].process.stdin.write(b"<runtask>" +
> pickle.dumps((taskfn, task, taskname, taskhash, unihash, False,
> self.cooker.collection.get_file_appends(taskfn), taskdepdata,
> self.rqdata.setscene_enforce)) + b"</runtask>")
> self.rq.fakeworker[mc].process.stdin.flush() else:
> - self.rq.worker[mc].process.stdin.write(b"<runtask>"
> + pickle.dumps((taskfn, task, taskname, False,
> self.cooker.collection.get_file_appends(taskfn), taskdepdata,
> self.rqdata.setscene_enforce)) + b"</runtask>")
> + self.rq.worker[mc].process.stdin.write(b"<runtask>"
> + pickle.dumps((taskfn, task, taskname, taskhash, unihash, False,
> self.cooker.collection.get_file_appends(taskfn), taskdepdata,
> self.rqdata.setscene_enforce)) + b"</runtask>")
> self.rq.worker[mc].process.stdin.flush() self.build_stamps[task] =
> bb.build.stampfile(taskname, self.rqdata.dataCaches[mc], taskfn,
> noextra=True) @@ -2052,30 +2134,58 @@ class
> RunQueueExecuteTasks(RunQueueExecute): if self.can_start_task():
> return True
> - if self.stats.active > 0:
> + if self.stats.active > 0 or self.sq_stats.active > 0:
> self.rq.read_workers()
> return self.rq.active_fds()
>
> + # No more tasks can be run. If we have deferred setscene
> tasks we should run them.
> + if self.sq_deferred:
> + tid =
> self.sq_deferred.pop(list(self.sq_deferred.keys())[0])
> + logger.warning("Runqeueue deadlocked on deferred tasks,
> forcing task %s" % tid)
> + self.sq_task_failoutright(tid)
> + return True
> +
> if len(self.failed_tids) != 0:
> self.rq.state = runQueueFailed
> return True
>
> # Sanity Checks
> + err = self.summarise_scenequeue_errors()
> for task in self.rqdata.runtaskentries:
> if task not in self.runq_buildable:
> logger.error("Task %s never buildable!", task)
> - if task not in self.runq_running:
> + err = True
> + elif task not in self.runq_running:
> logger.error("Task %s never ran!", task)
> - if task not in self.runq_complete:
> + err = True
> + elif task not in self.runq_complete:
> logger.error("Task %s never completed!", task)
> - self.rq.state = runQueueComplete
> + err = True
> +
> + if err:
> + self.rq.state = runQueueFailed
> + else:
> + self.rq.state = runQueueComplete
>
> return True
>
> + def filtermcdeps(self, task, mc, deps):
> + ret = set()
> + for dep in deps:
> + thismc = mc_from_tid(dep)
> + if thismc != mc:
> + continue
> + ret.add(dep)
> + return ret
> +
> + # We filter out multiconfig dependencies from taskdepdata we
> pass to the tasks
> + # as most code can't handle them
> def build_taskdepdata(self, task):
> taskdepdata = {}
> - next = self.rqdata.runtaskentries[task].depends
> + mc = mc_from_tid(task)
> + next = self.rqdata.runtaskentries[task].depends.copy()
> next.add(task)
> + next = self.filtermcdeps(task, mc, next)
> while next:
> additional = []
> for revdep in next:
> @@ -2084,7 +2194,9 @@ class RunQueueExecuteTasks(RunQueueExecute):
> deps = self.rqdata.runtaskentries[revdep].depends
> provides =
> self.rqdata.dataCaches[mc].fn_provides[taskfn] taskhash =
> self.rqdata.runtaskentries[revdep].hash
> - taskdepdata[revdep] = [pn, taskname, fn, deps,
> provides, taskhash]
> + unihash = self.rqdata.runtaskentries[revdep].unihash
> + deps = self.filtermcdeps(task, mc, deps)
> + taskdepdata[revdep] = [pn, taskname, fn, deps,
> provides, taskhash, unihash] for revdep2 in deps:
> if revdep2 not in taskdepdata:
> additional.append(revdep2)
> @@ -2093,269 +2205,197 @@ class RunQueueExecuteTasks(RunQueueExecute):
> #bb.note("Task %s: " % task + str(taskdepdata).replace("],
> ", "],\n")) return taskdepdata
>
> -class RunQueueExecuteScenequeue(RunQueueExecute):
> - def __init__(self, rq):
> - RunQueueExecute.__init__(self, rq)
> -
> - self.scenequeue_covered = set()
> - self.scenequeue_notcovered = set()
> - self.scenequeue_notneeded = set()
> + def update_holdofftasks(self):
>
> - # If we don't have any setscene functions, skip this step
> - if len(self.rqdata.runq_setscene_tids) == 0:
> - rq.scenequeue_covered = set()
> - rq.scenequeue_notcovered = set()
> - rq.state = runQueueRunInit
> + if not self.holdoff_need_update:
> return
>
> - self.stats =
> RunQueueStats(len(self.rqdata.runq_setscene_tids))
> + notcovered = set(self.scenequeue_notcovered)
> + notcovered |= self.cantskip
> + for tid in self.scenequeue_notcovered:
> + notcovered |= self.sqdata.sq_covered_tasks[tid]
> + notcovered |=
> self.sqdata.unskippable.difference(self.rqdata.runq_setscene_tids)
> + notcovered.intersection_update(self.tasks_scenequeue_done)
>
> - sq_revdeps = {}
> - sq_revdeps_new = {}
> - sq_revdeps_squash = {}
> - self.sq_harddeps = {}
> - self.stamps = {}
> -
> - # We need to construct a dependency graph for the setscene
> functions. Intermediate
> - # dependencies between the setscene tasks only complicate
> the code. This code
> - # therefore aims to collapse the huge runqueue dependency
> tree into a smaller one
> - # only containing the setscene functions.
> -
> - self.rqdata.init_progress_reporter.next_stage()
> -
> - # First process the chains up to the first setscene task.
> - endpoints = {}
> - for tid in self.rqdata.runtaskentries:
> - sq_revdeps[tid] =
> copy.copy(self.rqdata.runtaskentries[tid].revdeps)
> - sq_revdeps_new[tid] = set()
> - if (len(sq_revdeps[tid]) == 0) and tid not in
> self.rqdata.runq_setscene_tids:
> - #bb.warn("Added endpoint %s" % (tid))
> - endpoints[tid] = set()
> + covered = set(self.scenequeue_covered)
> + for tid in self.scenequeue_covered:
> + covered |= self.sqdata.sq_covered_tasks[tid]
> + covered.difference_update(notcovered)
> + covered.intersection_update(self.tasks_scenequeue_done)
>
> - self.rqdata.init_progress_reporter.next_stage()
> + for tid in notcovered | covered:
> + if len(self.rqdata.runtaskentries[tid].depends) == 0:
> + self.setbuildable(tid)
> + elif
> self.rqdata.runtaskentries[tid].depends.issubset(self.runq_complete):
> + self.setbuildable(tid)
>
> - # Secondly process the chains between setscene tasks.
> - for tid in self.rqdata.runq_setscene_tids:
> - #bb.warn("Added endpoint 2 %s" % (tid))
> - for dep in self.rqdata.runtaskentries[tid].depends:
> - if tid in sq_revdeps[dep]:
> - sq_revdeps[dep].remove(tid)
> - if dep not in endpoints:
> - endpoints[dep] = set()
> - #bb.warn(" Added endpoint 3 %s" % (dep))
> - endpoints[dep].add(tid)
> -
> - self.rqdata.init_progress_reporter.next_stage()
> -
> - def process_endpoints(endpoints):
> - newendpoints = {}
> - for point, task in endpoints.items():
> - tasks = set()
> - if task:
> - tasks |= task
> - if sq_revdeps_new[point]:
> - tasks |= sq_revdeps_new[point]
> - sq_revdeps_new[point] = set()
> - if point in self.rqdata.runq_setscene_tids:
> - sq_revdeps_new[point] = tasks
> - tasks = set()
> - continue
> - for dep in self.rqdata.runtaskentries[point].depends:
> - if point in sq_revdeps[dep]:
> - sq_revdeps[dep].remove(point)
> - if tasks:
> - sq_revdeps_new[dep] |= tasks
> - if len(sq_revdeps[dep]) == 0 and dep not in
> self.rqdata.runq_setscene_tids:
> - newendpoints[dep] = task
> - if len(newendpoints) != 0:
> - process_endpoints(newendpoints)
> -
> - process_endpoints(endpoints)
> -
> - self.rqdata.init_progress_reporter.next_stage()
> -
> - # Build a list of setscene tasks which are "unskippable"
> - # These are direct endpoints referenced by the build
> - endpoints2 = {}
> - sq_revdeps2 = {}
> - sq_revdeps_new2 = {}
> - def process_endpoints2(endpoints):
> - newendpoints = {}
> - for point, task in endpoints.items():
> - tasks = set([point])
> - if task:
> - tasks |= task
> - if sq_revdeps_new2[point]:
> - tasks |= sq_revdeps_new2[point]
> - sq_revdeps_new2[point] = set()
> - if point in self.rqdata.runq_setscene_tids:
> - sq_revdeps_new2[point] = tasks
> - for dep in self.rqdata.runtaskentries[point].depends:
> - if point in sq_revdeps2[dep]:
> - sq_revdeps2[dep].remove(point)
> - if tasks:
> - sq_revdeps_new2[dep] |= tasks
> - if (len(sq_revdeps2[dep]) == 0 or
> len(sq_revdeps_new2[dep]) != 0) and dep not in
> self.rqdata.runq_setscene_tids:
> - newendpoints[dep] = tasks
> - if len(newendpoints) != 0:
> - process_endpoints2(newendpoints)
> - for tid in self.rqdata.runtaskentries:
> - sq_revdeps2[tid] =
> copy.copy(self.rqdata.runtaskentries[tid].revdeps)
> - sq_revdeps_new2[tid] = set()
> - if (len(sq_revdeps2[tid]) == 0) and tid not in
> self.rqdata.runq_setscene_tids:
> - endpoints2[tid] = set()
> - process_endpoints2(endpoints2)
> - self.unskippable = []
> - for tid in self.rqdata.runq_setscene_tids:
> - if sq_revdeps_new2[tid]:
> - self.unskippable.append(tid)
> + self.tasks_covered = covered
> + self.tasks_notcovered = notcovered
>
> -
> self.rqdata.init_progress_reporter.next_stage(len(self.rqdata.runtaskentries))
> + self.holdoff_tasks = set()
>
> - for taskcounter, tid in
> enumerate(self.rqdata.runtaskentries):
> - if tid in self.rqdata.runq_setscene_tids:
> - deps = set()
> - for dep in sq_revdeps_new[tid]:
> - deps.add(dep)
> - sq_revdeps_squash[tid] = deps
> - elif len(sq_revdeps_new[tid]) != 0:
> - bb.msg.fatal("RunQueue", "Something went badly wrong
> during scenequeue generation, aborting. Please report this problem.")
> - self.rqdata.init_progress_reporter.update(taskcounter)
> -
> - self.rqdata.init_progress_reporter.next_stage()
> -
> - # Resolve setscene inter-task dependencies
> - # e.g. do_sometask_setscene[depends] =
> "targetname:do_someothertask_setscene"
> - # Note that anything explicitly depended upon will have its
> reverse dependencies removed to avoid circular dependencies for tid
> in self.rqdata.runq_setscene_tids:
> - (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
> - realtid = tid + "_setscene"
> - idepends =
> self.rqdata.taskData[mc].taskentries[realtid].idepends
> - self.stamps[tid] = bb.build.stampfile(taskname +
> "_setscene", self.rqdata.dataCaches[mc], taskfn, noextra=True)
> - for (depname, idependtask) in idepends:
> -
> - if depname not in
> self.rqdata.taskData[mc].build_targets:
> - continue
> + if tid not in self.scenequeue_covered and tid not in
> self.scenequeue_notcovered:
> + self.holdoff_tasks.add(tid)
>
> - depfn =
> self.rqdata.taskData[mc].build_targets[depname][0]
> - if depfn is None:
> - continue
> - deptid = depfn + ":" +
> idependtask.replace("_setscene", "")
> - if deptid not in self.rqdata.runtaskentries:
> - bb.msg.fatal("RunQueue", "Task %s depends
> upon non-existent task %s:%s" % (realtid, depfn, idependtask)) -
> - if not deptid in self.sq_harddeps:
> - self.sq_harddeps[deptid] = set()
> - self.sq_harddeps[deptid].add(tid)
> -
> - sq_revdeps_squash[tid].add(deptid)
> - # Have to zero this to avoid circular
> dependencies
> - sq_revdeps_squash[deptid] = set()
> -
> - self.rqdata.init_progress_reporter.next_stage()
> -
> - for task in self.sq_harddeps:
> - for dep in self.sq_harddeps[task]:
> - sq_revdeps_squash[dep].add(task)
> -
> - self.rqdata.init_progress_reporter.next_stage()
> -
> - #for tid in sq_revdeps_squash:
> - # for dep in sq_revdeps_squash[tid]:
> - # data = data + "\n %s" % dep
> - # bb.warn("Task %s_setscene: is %s " % (tid, data
> -
> - self.sq_deps = {}
> - self.sq_revdeps = sq_revdeps_squash
> - self.sq_revdeps2 = copy.deepcopy(self.sq_revdeps)
> -
> - for tid in self.sq_revdeps:
> - self.sq_deps[tid] = set()
> - for tid in self.sq_revdeps:
> - for dep in self.sq_revdeps[tid]:
> - self.sq_deps[dep].add(tid)
> -
> - self.rqdata.init_progress_reporter.next_stage()
> -
> - for tid in self.sq_revdeps:
> - if len(self.sq_revdeps[tid]) == 0:
> - self.runq_buildable.add(tid)
> -
> - self.rqdata.init_progress_reporter.finish()
> -
> - self.outrightfail = []
> - if self.rq.hashvalidate:
> - sq_hash = []
> - sq_hashfn = []
> - sq_fn = []
> - sq_taskname = []
> - sq_task = []
> - noexec = []
> - stamppresent = []
> - for tid in self.sq_revdeps:
> - (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
> + for tid in self.holdoff_tasks.copy():
> + for dep in self.sqdata.sq_covered_tasks[tid]:
> + if dep not in self.runq_complete:
> + self.holdoff_tasks.add(dep)
>
> - taskdep =
> self.rqdata.dataCaches[mc].task_deps[taskfn]
> + self.holdoff_need_update = False
>
> - if 'noexec' in taskdep and taskname in
> taskdep['noexec']:
> - noexec.append(tid)
> - self.task_skip(tid)
> - bb.build.make_stamp(taskname + "_setscene",
> self.rqdata.dataCaches[mc], taskfn)
> - continue
> + def process_possible_migrations(self):
>
> - if self.rq.check_stamp_task(tid, taskname +
> "_setscene", cache=self.stampcache):
> - logger.debug(2, 'Setscene stamp current for task
> %s', tid)
> - stamppresent.append(tid)
> - self.task_skip(tid)
> - continue
> + changed = set()
> + for tid, unihash in self.updated_taskhash_queue.copy():
> + if tid in self.runq_running and tid not in
> self.runq_complete:
> + continue
>
> - if self.rq.check_stamp_task(tid, taskname, recurse =
> True, cache=self.stampcache):
> - logger.debug(2, 'Normal stamp current for task
> %s', tid)
> - stamppresent.append(tid)
> - self.task_skip(tid)
> - continue
> + self.updated_taskhash_queue.remove((tid, unihash))
> +
> + if unihash != self.rqdata.runtaskentries[tid].unihash:
> + logger.info("Task %s unihash changed to %s" % (tid,
> unihash))
> + self.rqdata.runtaskentries[tid].unihash = unihash
> + bb.parse.siggen.set_unihash(tid, unihash)
> +
> + # Work out all tasks which depend on this one
> + total = set()
> + next = set(self.rqdata.runtaskentries[tid].revdeps)
> + while next:
> + current = next.copy()
> + total = total |next
> + next = set()
> + for ntid in current:
> + next |=
> self.rqdata.runtaskentries[ntid].revdeps
> + next.difference_update(total)
> +
> + # Now iterate those tasks in dependency order to
> regenerate their taskhash/unihash
> + done = set()
> + next = set(self.rqdata.runtaskentries[tid].revdeps)
> + while next:
> + current = next.copy()
> + next = set()
> + for tid in current:
> + if not
> self.rqdata.runtaskentries[tid].depends.isdisjoint(total):
> + continue
> + procdep = []
> + for dep in
> self.rqdata.runtaskentries[tid].depends:
> + procdep.append(dep)
> + orighash =
> self.rqdata.runtaskentries[tid].hash
> + self.rqdata.runtaskentries[tid].hash =
> bb.parse.siggen.get_taskhash(tid, procdep,
> self.rqdata.dataCaches[mc_from_tid(tid)])
> + origuni =
> self.rqdata.runtaskentries[tid].unihash
> + self.rqdata.runtaskentries[tid].unihash =
> bb.parse.siggen.get_unihash(tid)
> + logger.debug(1, "Task %s hash changes:
> %s->%s %s->%s" % (tid, orighash,
> self.rqdata.runtaskentries[tid].hash, origuni,
> self.rqdata.runtaskentries[tid].unihash))
> + next |=
> self.rqdata.runtaskentries[tid].revdeps
> + changed.add(tid)
> + total.remove(tid)
> + next.intersection_update(total)
> +
> + if changed:
> + for mc in self.rq.worker:
> +
> self.rq.worker[mc].process.stdin.write(b"<newtaskhashes>" +
> pickle.dumps(bb.parse.siggen.get_taskhashes()) + b"</newtaskhashes>")
> + for mc in self.rq.fakeworker:
> +
> self.rq.fakeworker[mc].process.stdin.write(b"<newtaskhashes>" +
> pickle.dumps(bb.parse.siggen.get_taskhashes()) + b"</newtaskhashes>")
> +
> + logger.debug(1, pprint.pformat("Tasks changed:\n%s" %
> (changed))) +
> + for tid in changed:
> + if tid not in self.rqdata.runq_setscene_tids:
> + continue
> + if tid in self.runq_running:
> + continue
> + if tid in self.scenequeue_covered:
> + # Potentially risky, should we report this hash as a
> match?
> + logger.info("Already covered setscene for %s so
> ignoring rehash" % (tid))
> + continue
> + if tid not in self.pending_migrations:
> + self.pending_migrations.add(tid)
> +
> + for tid in self.pending_migrations.copy():
> + valid = True
> + # Check no tasks this covers are running
> + for dep in self.sqdata.sq_covered_tasks[tid]:
> + if dep in self.runq_running and dep not in
> self.runq_complete:
> + logger.debug(2, "Task %s is running which blocks
> setscene for %s from running" % (dep, tid))
> + valid = False
> + break
> + if not valid:
> + continue
>
> - sq_fn.append(fn)
> -
> sq_hashfn.append(self.rqdata.dataCaches[mc].hashfn[taskfn])
> - sq_hash.append(self.rqdata.runtaskentries[tid].hash)
> - sq_taskname.append(taskname)
> - sq_task.append(tid)
> + self.pending_migrations.remove(tid)
> + changed = True
>
> -
> self.cooker.data.setVar("BB_SETSCENE_STAMPCURRENT_COUNT",
> len(stamppresent))
> + if tid in self.tasks_scenequeue_done:
> + self.tasks_scenequeue_done.remove(tid)
> + for dep in self.sqdata.sq_covered_tasks[tid]:
> + if dep not in self.runq_complete:
> + if dep in self.tasks_scenequeue_done and dep not
> in self.sqdata.unskippable:
> + self.tasks_scenequeue_done.remove(dep)
> +
> + if tid in self.sq_buildable:
> + self.sq_buildable.remove(tid)
> + if tid in self.sq_running:
> + self.sq_running.remove(tid)
> + if
> self.sqdata.sq_revdeps[tid].issubset(self.scenequeue_covered |
> self.scenequeue_notcovered):
> + if tid not in self.sq_buildable:
> + self.sq_buildable.add(tid)
> + if len(self.sqdata.sq_revdeps[tid]) == 0:
> + self.sq_buildable.add(tid)
> +
> + if tid in self.sqdata.outrightfail:
> + self.sqdata.outrightfail.remove(tid)
> + if tid in self.scenequeue_notcovered:
> + self.scenequeue_notcovered.remove(tid)
> + if tid in self.scenequeue_covered:
> + self.scenequeue_covered.remove(tid)
> + if tid in self.scenequeue_notneeded:
> + self.scenequeue_notneeded.remove(tid)
>
> - call = self.rq.hashvalidate + "(sq_fn, sq_task, sq_hash,
> sq_hashfn, d)"
> - locs = { "sq_fn" : sq_fn, "sq_task" : sq_taskname,
> "sq_hash" : sq_hash, "sq_hashfn" : sq_hashfn, "d" : self.cooker.data }
> - valid = bb.utils.better_eval(call, locs)
> + (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
> + self.sqdata.stamps[tid] = bb.build.stampfile(taskname +
> "_setscene", self.rqdata.dataCaches[mc], taskfn, noextra=True)
> - self.cooker.data.delVar("BB_SETSCENE_STAMPCURRENT_COUNT")
> + if tid in self.stampcache:
> + del self.stampcache[tid]
>
> - valid_new = stamppresent
> - for v in valid:
> - valid_new.append(sq_task[v])
> + if tid in self.build_stamps:
> + del self.build_stamps[tid]
>
> - for tid in self.sq_revdeps:
> - if tid not in valid_new and tid not in noexec:
> - logger.debug(2, 'No package found, so skipping
> setscene task %s', tid)
> - self.outrightfail.append(tid)
> + logger.info("Setscene task %s now valid and being rerun"
> % tid)
> + self.sqdone = False
> + update_scenequeue_data([tid], self.sqdata, self.rqdata,
> self.rq, self.cooker, self.stampcache, self)
> - logger.info('Executing SetScene Tasks')
> + if changed:
> + self.holdoff_need_update = True
>
> - self.rq.state = runQueueSceneRun
> + def scenequeue_updatecounters(self, task, fail=False):
>
> - def scenequeue_updatecounters(self, task, fail = False):
> - for dep in self.sq_deps[task]:
> - if fail and task in self.sq_harddeps and dep in
> self.sq_harddeps[task]:
> + for dep in sorted(self.sqdata.sq_deps[task]):
> + if fail and task in self.sqdata.sq_harddeps and dep in
> self.sqdata.sq_harddeps[task]: logger.debug(2, "%s was unavailable
> and is a hard dependency of %s so skipping" % (task, dep))
> - self.scenequeue_updatecounters(dep, fail)
> + self.sq_task_failoutright(dep)
> continue
> - if task not in self.sq_revdeps2[dep]:
> - # May already have been removed by the fail case
> above
> - continue
> - self.sq_revdeps2[dep].remove(task)
> - if len(self.sq_revdeps2[dep]) == 0:
> - self.runq_buildable.add(dep)
> + if
> self.sqdata.sq_revdeps[dep].issubset(self.scenequeue_covered |
> self.scenequeue_notcovered):
> + if dep not in self.sq_buildable:
> + self.sq_buildable.add(dep)
>
> - def task_completeoutright(self, task):
> + next = set([task])
> + while next:
> + new = set()
> + for t in sorted(next):
> + self.tasks_scenequeue_done.add(t)
> + # Look down the dependency chain for non-setscene
> things which this task depends on
> + # and mark as 'done'
> + for dep in self.rqdata.runtaskentries[t].depends:
> + if dep in self.rqdata.runq_setscene_tids or dep
> in self.tasks_scenequeue_done:
> + continue
> + if
> self.rqdata.runtaskentries[dep].revdeps.issubset(self.tasks_scenequeue_done):
> + new.add(dep)
> + next = new
> +
> + self.holdoff_need_update = True
> +
> + def sq_task_completeoutright(self, task):
> """
> Mark a task as completed
> Look at the reverse dependencies and mark any task with
> @@ -2366,7 +2406,7 @@ class
> RunQueueExecuteScenequeue(RunQueueExecute):
> self.scenequeue_covered.add(task) self.scenequeue_updatecounters(task)
>
> - def check_taskfail(self, task):
> + def sq_check_taskfail(self, task):
> if self.rqdata.setscenewhitelist is not None:
> realtask = task.split('_setscene')[0]
> (mc, fn, taskname, taskfn) = split_tid_mcfn(realtask)
> @@ -2375,130 +2415,34 @@ class
> RunQueueExecuteScenequeue(RunQueueExecute): logger.error('Task %s.%s
> failed' % (pn, taskname + "_setscene")) self.rq.state =
> runQueueCleanUp
> - def task_complete(self, task):
> - self.stats.taskCompleted()
> - bb.event.fire(sceneQueueTaskCompleted(task, self.stats,
> self.rq), self.cfgData)
> - self.task_completeoutright(task)
> + def sq_task_complete(self, task):
> + self.sq_stats.taskCompleted()
> + bb.event.fire(sceneQueueTaskCompleted(task, self.sq_stats,
> self.rq), self.cfgData)
> + self.sq_task_completeoutright(task)
>
> - def task_fail(self, task, result):
> - self.stats.taskFailed()
> - bb.event.fire(sceneQueueTaskFailed(task, self.stats, result,
> self), self.cfgData)
> + def sq_task_fail(self, task, result):
> + self.sq_stats.taskFailed()
> + bb.event.fire(sceneQueueTaskFailed(task, self.sq_stats,
> result, self), self.cfgData) self.scenequeue_notcovered.add(task)
> self.scenequeue_updatecounters(task, True)
> - self.check_taskfail(task)
> + self.sq_check_taskfail(task)
>
> - def task_failoutright(self, task):
> - self.runq_running.add(task)
> - self.runq_buildable.add(task)
> - self.stats.taskSkipped()
> - self.stats.taskCompleted()
> + def sq_task_failoutright(self, task):
> + self.sq_running.add(task)
> + self.sq_buildable.add(task)
> + self.sq_stats.taskSkipped()
> + self.sq_stats.taskCompleted()
> self.scenequeue_notcovered.add(task)
> self.scenequeue_updatecounters(task, True)
>
> - def task_skip(self, task):
> - self.runq_running.add(task)
> - self.runq_buildable.add(task)
> - self.task_completeoutright(task)
> - self.stats.taskSkipped()
> - self.stats.taskCompleted()
> -
> - def execute(self):
> - """
> - Run the tasks in a queue prepared by prepare_runqueue
> - """
> -
> - self.rq.read_workers()
> -
> - task = None
> - if self.can_start_task():
> - # Find the next setscene to run
> - for nexttask in self.rqdata.runq_setscene_tids:
> - if nexttask in self.runq_buildable and nexttask not
> in self.runq_running and self.stamps[nexttask] not in
> self.build_stamps.values():
> - if nexttask in self.unskippable:
> - logger.debug(2, "Setscene task %s is
> unskippable" % nexttask)
> - if nexttask not in self.unskippable and
> len(self.sq_revdeps[nexttask]) > 0 and
> self.sq_revdeps[nexttask].issubset(self.scenequeue_covered) and
> self.check_dependencies(nexttask, self.sq_revdeps[nexttask], True):
> - fn = fn_from_tid(nexttask)
> - foundtarget = False
> -
> - if nexttask in self.rqdata.target_tids:
> - foundtarget = True
> - if not foundtarget:
> - logger.debug(2, "Skipping setscene for
> task %s" % nexttask)
> - self.task_skip(nexttask)
> - self.scenequeue_notneeded.add(nexttask)
> - return True
> - if nexttask in self.outrightfail:
> - self.task_failoutright(nexttask)
> - return True
> - task = nexttask
> - break
> - if task is not None:
> - (mc, fn, taskname, taskfn) = split_tid_mcfn(task)
> - taskname = taskname + "_setscene"
> - if self.rq.check_stamp_task(task,
> taskname_from_tid(task), recurse = True, cache=self.stampcache):
> - logger.debug(2, 'Stamp for underlying task %s is
> current, so skipping setscene variant', task)
> - self.task_failoutright(task)
> - return True
> -
> - if self.cooker.configuration.force:
> - if task in self.rqdata.target_tids:
> - self.task_failoutright(task)
> - return True
> -
> - if self.rq.check_stamp_task(task, taskname,
> cache=self.stampcache):
> - logger.debug(2, 'Setscene stamp current task %s, so
> skip it and its dependencies', task)
> - self.task_skip(task)
> - return True
> -
> - startevent = sceneQueueTaskStarted(task, self.stats,
> self.rq)
> - bb.event.fire(startevent, self.cfgData)
> + def sq_task_skip(self, task):
> + self.sq_running.add(task)
> + self.sq_buildable.add(task)
> + self.sq_task_completeoutright(task)
> + self.sq_stats.taskSkipped()
> + self.sq_stats.taskCompleted()
>
> - taskdepdata = self.build_taskdepdata(task)
> -
> - taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn]
> - if 'fakeroot' in taskdep and taskname in
> taskdep['fakeroot'] and not self.cooker.configuration.dry_run:
> - if not mc in self.rq.fakeworker:
> - self.rq.start_fakeworker(self, mc)
> -
> self.rq.fakeworker[mc].process.stdin.write(b"<runtask>" +
> pickle.dumps((taskfn, task, taskname, True,
> self.cooker.collection.get_file_appends(taskfn), taskdepdata, False))
> + b"</runtask>")
> - self.rq.fakeworker[mc].process.stdin.flush()
> - else:
> - self.rq.worker[mc].process.stdin.write(b"<runtask>"
> + pickle.dumps((taskfn, task, taskname, True,
> self.cooker.collection.get_file_appends(taskfn), taskdepdata, False))
> + b"</runtask>")
> - self.rq.worker[mc].process.stdin.flush()
> -
> - self.build_stamps[task] = bb.build.stampfile(taskname,
> self.rqdata.dataCaches[mc], taskfn, noextra=True)
> - self.build_stamps2.append(self.build_stamps[task])
> - self.runq_running.add(task)
> - self.stats.taskActive()
> - if self.can_start_task():
> - return True
> -
> - if self.stats.active > 0:
> - self.rq.read_workers()
> - return self.rq.active_fds()
> -
> - #for tid in self.sq_revdeps:
> - # if tid not in self.runq_running:
> - # buildable = tid in self.runq_buildable
> - # revdeps = self.sq_revdeps[tid]
> - # bb.warn("Found we didn't run %s %s %s" % (tid,
> buildable, str(revdeps))) -
> - self.rq.scenequeue_covered = self.scenequeue_covered
> - self.rq.scenequeue_notcovered = self.scenequeue_notcovered
> -
> - logger.debug(1, 'We can skip tasks %s',
> "\n".join(sorted(self.rq.scenequeue_covered))) -
> - self.rq.state = runQueueRunInit
> -
> - completeevent = sceneQueueComplete(self.stats, self.rq)
> - bb.event.fire(completeevent, self.cfgData)
> -
> - return True
> -
> - def runqueue_process_waitpid(self, task, status):
> - RunQueueExecute.runqueue_process_waitpid(self, task, status)
> -
> -
> - def build_taskdepdata(self, task):
> + def sq_build_taskdepdata(self, task):
> def getsetscenedeps(tid):
> deps = set()
> (mc, fn, taskname, _) = split_tid_mcfn(tid)
> @@ -2526,7 +2470,8 @@ class
> RunQueueExecuteScenequeue(RunQueueExecute): deps =
> getsetscenedeps(revdep) provides =
> self.rqdata.dataCaches[mc].fn_provides[taskfn] taskhash =
> self.rqdata.runtaskentries[revdep].hash
> - taskdepdata[revdep] = [pn, taskname, fn, deps,
> provides, taskhash]
> + unihash = self.rqdata.runtaskentries[revdep].unihash
> + taskdepdata[revdep] = [pn, taskname, fn, deps,
> provides, taskhash, unihash] for revdep2 in deps:
> if revdep2 not in taskdepdata:
> additional.append(revdep2)
> @@ -2535,6 +2480,279 @@ class
> RunQueueExecuteScenequeue(RunQueueExecute): #bb.note("Task %s: " %
> task + str(taskdepdata).replace("], ", "],\n")) return taskdepdata
>
> + def check_setscenewhitelist(self, tid):
> + # Check task that is going to run against the whitelist
> + (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
> + # Ignore covered tasks
> + if tid in self.tasks_covered:
> + return False
> + # Ignore stamped tasks
> + if self.rq.check_stamp_task(tid, taskname,
> cache=self.stampcache):
> + return False
> + # Ignore noexec tasks
> + taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn]
> + if 'noexec' in taskdep and taskname in taskdep['noexec']:
> + return False
> +
> + pn = self.rqdata.dataCaches[mc].pkg_fn[taskfn]
> + if not check_setscene_enforce_whitelist(pn, taskname,
> self.rqdata.setscenewhitelist):
> + if tid in self.rqdata.runq_setscene_tids:
> + msg = 'Task %s.%s attempted to execute unexpectedly
> and should have been setscened' % (pn, taskname)
> + else:
> + msg = 'Task %s.%s attempted to execute unexpectedly'
> % (pn, taskname)
> + logger.error(msg + '\nThis is usually due to missing
> setscene tasks. Those missing in this build were: %s' %
> pprint.pformat(self.scenequeue_notcovered))
> + return True
> + return False
> +
> +class SQData(object):
> + def __init__(self):
> + # SceneQueue dependencies
> + self.sq_deps = {}
> + # SceneQueue reverse dependencies
> + self.sq_revdeps = {}
> + # Injected inter-setscene task dependencies
> + self.sq_harddeps = {}
> + # Cache of stamp files so duplicates can't run in parallel
> + self.stamps = {}
> + # Setscene tasks directly depended upon by the build
> + self.unskippable = set()
> + # List of setscene tasks which aren't present
> + self.outrightfail = set()
> + # A list of normal tasks a setscene task covers
> + self.sq_covered_tasks = {}
> +
> +def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache,
> sqrq): +
> + sq_revdeps = {}
> + sq_revdeps_squash = {}
> + sq_collated_deps = {}
> +
> + # We need to construct a dependency graph for the setscene
> functions. Intermediate
> + # dependencies between the setscene tasks only complicate the
> code. This code
> + # therefore aims to collapse the huge runqueue dependency tree
> into a smaller one
> + # only containing the setscene functions.
> +
> + rqdata.init_progress_reporter.next_stage()
> +
> + # First process the chains up to the first setscene task.
> + endpoints = {}
> + for tid in rqdata.runtaskentries:
> + sq_revdeps[tid] =
> copy.copy(rqdata.runtaskentries[tid].revdeps)
> + sq_revdeps_squash[tid] = set()
> + if (len(sq_revdeps[tid]) == 0) and tid not in
> rqdata.runq_setscene_tids:
> + #bb.warn("Added endpoint %s" % (tid))
> + endpoints[tid] = set()
> +
> + rqdata.init_progress_reporter.next_stage()
> +
> + # Secondly process the chains between setscene tasks.
> + for tid in rqdata.runq_setscene_tids:
> + sq_collated_deps[tid] = set()
> + #bb.warn("Added endpoint 2 %s" % (tid))
> + for dep in rqdata.runtaskentries[tid].depends:
> + if tid in sq_revdeps[dep]:
> + sq_revdeps[dep].remove(tid)
> + if dep not in endpoints:
> + endpoints[dep] = set()
> + #bb.warn(" Added endpoint 3 %s" % (dep))
> + endpoints[dep].add(tid)
> +
> + rqdata.init_progress_reporter.next_stage()
> +
> + def process_endpoints(endpoints):
> + newendpoints = {}
> + for point, task in endpoints.items():
> + tasks = set()
> + if task:
> + tasks |= task
> + if sq_revdeps_squash[point]:
> + tasks |= sq_revdeps_squash[point]
> + if point not in rqdata.runq_setscene_tids:
> + for t in tasks:
> + sq_collated_deps[t].add(point)
> + sq_revdeps_squash[point] = set()
> + if point in rqdata.runq_setscene_tids:
> + sq_revdeps_squash[point] = tasks
> + tasks = set()
> + continue
> + for dep in rqdata.runtaskentries[point].depends:
> + if point in sq_revdeps[dep]:
> + sq_revdeps[dep].remove(point)
> + if tasks:
> + sq_revdeps_squash[dep] |= tasks
> + if len(sq_revdeps[dep]) == 0 and dep not in
> rqdata.runq_setscene_tids:
> + newendpoints[dep] = task
> + if len(newendpoints) != 0:
> + process_endpoints(newendpoints)
> +
> + process_endpoints(endpoints)
> +
> + rqdata.init_progress_reporter.next_stage()
> +
> + # Build a list of tasks which are "unskippable"
> + # These are direct endpoints referenced by the build upto and
> including setscene tasks
> + # Take the build endpoints (no revdeps) and find the sstate
> tasks they depend upon
> + new = True
> + for tid in rqdata.runtaskentries:
> + if len(rqdata.runtaskentries[tid].revdeps) == 0:
> + sqdata.unskippable.add(tid)
> + sqdata.unskippable |= sqrq.cantskip
> + while new:
> + new = False
> + orig = sqdata.unskippable.copy()
> + for tid in sorted(orig, reverse=True):
> + if tid in rqdata.runq_setscene_tids:
> + continue
> + if len(rqdata.runtaskentries[tid].depends) == 0:
> + # These are tasks which have no setscene tasks in
> their chain, need to mark as directly buildable
> + sqrq.setbuildable(tid)
> + sqdata.unskippable |= rqdata.runtaskentries[tid].depends
> + if sqdata.unskippable != orig:
> + new = True
> +
> + sqrq.tasks_scenequeue_done |=
> sqdata.unskippable.difference(rqdata.runq_setscene_tids) +
> +
> rqdata.init_progress_reporter.next_stage(len(rqdata.runtaskentries)) +
> + # Sanity check all dependencies could be changed to setscene
> task references
> + for taskcounter, tid in enumerate(rqdata.runtaskentries):
> + if tid in rqdata.runq_setscene_tids:
> + pass
> + elif len(sq_revdeps_squash[tid]) != 0:
> + bb.msg.fatal("RunQueue", "Something went badly wrong
> during scenequeue generation, aborting. Please report this problem.")
> + else:
> + del sq_revdeps_squash[tid]
> + rqdata.init_progress_reporter.update(taskcounter)
> +
> + rqdata.init_progress_reporter.next_stage()
> +
> + # Resolve setscene inter-task dependencies
> + # e.g. do_sometask_setscene[depends] =
> "targetname:do_someothertask_setscene"
> + # Note that anything explicitly depended upon will have its
> reverse dependencies removed to avoid circular dependencies
> + for tid in rqdata.runq_setscene_tids:
> + (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
> + realtid = tid + "_setscene"
> + idepends = rqdata.taskData[mc].taskentries[realtid].idepends
> + sqdata.stamps[tid] = bb.build.stampfile(taskname +
> "_setscene", rqdata.dataCaches[mc], taskfn, noextra=True)
> + for (depname, idependtask) in idepends:
> +
> + if depname not in rqdata.taskData[mc].build_targets:
> + continue
> +
> + depfn = rqdata.taskData[mc].build_targets[depname][0]
> + if depfn is None:
> + continue
> + deptid = depfn + ":" + idependtask.replace("_setscene",
> "")
> + if deptid not in rqdata.runtaskentries:
> + bb.msg.fatal("RunQueue", "Task %s depends upon
> non-existent task %s:%s" % (realtid, depfn, idependtask)) +
> + if not deptid in sqdata.sq_harddeps:
> + sqdata.sq_harddeps[deptid] = set()
> + sqdata.sq_harddeps[deptid].add(tid)
> +
> + sq_revdeps_squash[tid].add(deptid)
> + # Have to zero this to avoid circular dependencies
> + sq_revdeps_squash[deptid] = set()
> +
> + rqdata.init_progress_reporter.next_stage()
> +
> + for task in sqdata.sq_harddeps:
> + for dep in sqdata.sq_harddeps[task]:
> + sq_revdeps_squash[dep].add(task)
> +
> + rqdata.init_progress_reporter.next_stage()
> +
> + #for tid in sq_revdeps_squash:
> + # data = ""
> + # for dep in sq_revdeps_squash[tid]:
> + # data = data + "\n %s" % dep
> + # bb.warn("Task %s_setscene: is %s " % (tid, data))
> +
> + sqdata.sq_revdeps = sq_revdeps_squash
> + sqdata.sq_covered_tasks = sq_collated_deps
> +
> + # Build reverse version of revdeps to populate deps structure
> + for tid in sqdata.sq_revdeps:
> + sqdata.sq_deps[tid] = set()
> + for tid in sqdata.sq_revdeps:
> + for dep in sqdata.sq_revdeps[tid]:
> + sqdata.sq_deps[dep].add(tid)
> +
> + rqdata.init_progress_reporter.next_stage()
> +
> + sqdata.multiconfigs = set()
> + for tid in sqdata.sq_revdeps:
> + sqdata.multiconfigs.add(mc_from_tid(tid))
> + if len(sqdata.sq_revdeps[tid]) == 0:
> + sqrq.sq_buildable.add(tid)
> +
> + rqdata.init_progress_reporter.finish()
> +
> + sqdata.noexec = set()
> + sqdata.stamppresent = set()
> + sqdata.valid = set()
> +
> + update_scenequeue_data(sqdata.sq_revdeps, sqdata, rqdata, rq,
> cooker, stampcache, sqrq) +
> +def update_scenequeue_data(tids, sqdata, rqdata, rq, cooker,
> stampcache, sqrq): +
> + tocheck = set()
> +
> + for tid in sorted(tids):
> + if tid in sqdata.stamppresent:
> + sqdata.stamppresent.remove(tid)
> + if tid in sqdata.valid:
> + sqdata.valid.remove(tid)
> +
> + (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
> +
> + taskdep = rqdata.dataCaches[mc].task_deps[taskfn]
> +
> + if 'noexec' in taskdep and taskname in taskdep['noexec']:
> + sqdata.noexec.add(tid)
> + sqrq.sq_task_skip(tid)
> + bb.build.make_stamp(taskname + "_setscene",
> rqdata.dataCaches[mc], taskfn)
> + continue
> +
> + if rq.check_stamp_task(tid, taskname + "_setscene",
> cache=stampcache):
> + logger.debug(2, 'Setscene stamp current for task %s',
> tid)
> + sqdata.stamppresent.add(tid)
> + sqrq.sq_task_skip(tid)
> + continue
> +
> + if rq.check_stamp_task(tid, taskname, recurse = True,
> cache=stampcache):
> + logger.debug(2, 'Normal stamp current for task %s', tid)
> + sqdata.stamppresent.add(tid)
> + sqrq.sq_task_skip(tid)
> + continue
> +
> + tocheck.add(tid)
> +
> + sqdata.valid |= rq.validate_hashes(tocheck, cooker.data,
> len(sqdata.stamppresent), False) +
> + sqdata.hashes = {}
> + for mc in sorted(sqdata.multiconfigs):
> + for tid in sorted(sqdata.sq_revdeps):
> + if mc_from_tid(tid) != mc:
> + continue
> + if tid in sqdata.stamppresent:
> + continue
> + if tid in sqdata.valid:
> + continue
> + if tid in sqdata.noexec:
> + continue
> + if tid in sqrq.scenequeue_notcovered:
> + continue
> + sqdata.outrightfail.add(tid)
> +
> + h = pending_hash_index(tid, rqdata)
> + if h not in sqdata.hashes:
> + sqdata.hashes[h] = tid
> + else:
> + sqrq.sq_deferred[tid] = sqdata.hashes[h]
> + bb.warn("Deferring %s after %s" % (tid,
> sqdata.hashes[h])) +
> +
> class TaskFailure(Exception):
> """
> Exception raised when a task in a runqueue fails
> @@ -2641,6 +2859,15 @@ class runQueueTaskSkipped(runQueueEvent):
> runQueueEvent.__init__(self, task, stats, rq)
> self.reason = reason
>
> +class taskUniHashUpdate(bb.event.Event):
> + """
> + Base runQueue event class
> + """
> + def __init__(self, task, unihash):
> + self.taskid = task
> + self.unihash = unihash
> + bb.event.Event.__init__(self)
> +
> class runQueuePipe():
> """
> Abstraction for a pipe between a worker thread and the server
> @@ -2683,6 +2910,8 @@ class runQueuePipe():
> except ValueError as e:
> bb.msg.fatal("RunQueue", "failed load pickle
> '%s': '%s'" % (e, self.queue[7:index]))
> bb.event.fire_from_worker(event, self.d)
> + if isinstance(event, taskUniHashUpdate):
> +
> self.rqexec.updated_taskhash_queue.append((event.taskid,
> event.unihash)) found = True self.queue = self.queue[index+8:]
> index = self.queue.find(b"</event>")
> diff --git a/bitbake/lib/bb/server/__init__.py
> b/bitbake/lib/bb/server/__init__.py index 5a3fba9..b6f7513 100644
> --- a/bitbake/lib/bb/server/__init__.py
> +++ b/bitbake/lib/bb/server/__init__.py
> @@ -5,17 +5,5 @@
> # Copyright (C) 2006 - 2008 Richard Purdie
> # Copyright (C) 2013 Alexandru Damian
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. -
> -
> diff --git a/bitbake/lib/bb/server/process.py
> b/bitbake/lib/bb/server/process.py index 4e0d9c2..69aae62 100644
> --- a/bitbake/lib/bb/server/process.py
> +++ b/bitbake/lib/bb/server/process.py
> @@ -3,18 +3,8 @@
> #
> # Copyright (C) 2010 Bob Foerster <robert@erafx.com>
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> """
> This module implements a multiprocessing.Process based server
> for bitbake. @@ -130,6 +120,7 @@ class
> ProcessServer(multiprocessing.Process):
> bb.utils.set_process_name("Cooker")
> ready = []
> + newconnections = []
>
> self.controllersock = False
> fds = [self.sock]
> @@ -138,37 +129,48 @@ class ProcessServer(multiprocessing.Process):
> print("Entering server connection loop")
>
> def disconnect_client(self, fds):
> - if not self.haveui:
> - return
> print("Disconnecting Client")
> - fds.remove(self.controllersock)
> - fds.remove(self.command_channel)
> - bb.event.unregister_UIHhandler(self.event_handle, True)
> - self.command_channel_reply.writer.close()
> - self.event_writer.writer.close()
> - del self.event_writer
> - self.controllersock.close()
> - self.controllersock = False
> - self.haveui = False
> - self.lastui = time.time()
> - self.cooker.clientComplete()
> - if self.timeout is None:
> + if self.controllersock:
> + fds.remove(self.controllersock)
> + self.controllersock.close()
> + self.controllersock = False
> + if self.haveui:
> + fds.remove(self.command_channel)
> + bb.event.unregister_UIHhandler(self.event_handle,
> True)
> + self.command_channel_reply.writer.close()
> + self.event_writer.writer.close()
> + self.command_channel.close()
> + self.command_channel = False
> + del self.event_writer
> + self.lastui = time.time()
> + self.cooker.clientComplete()
> + self.haveui = False
> + ready = select.select(fds,[],[],0)[0]
> + if newconnections:
> + print("Starting new client")
> + conn = newconnections.pop(-1)
> + fds.append(conn)
> + self.controllersock = conn
> + elif self.timeout is None and not ready:
> print("No timeout, exiting.")
> self.quit = True
>
> while not self.quit:
> if self.sock in ready:
> - self.controllersock, address = self.sock.accept()
> - if self.haveui:
> - print("Dropping connection attempt as we have a
> UI %s" % (str(ready)))
> - self.controllersock.close()
> - else:
> - print("Accepting %s" % (str(ready)))
> - fds.append(self.controllersock)
> + while select.select([self.sock],[],[],0)[0]:
> + controllersock, address = self.sock.accept()
> + if self.controllersock:
> + print("Queuing %s (%s)" % (str(ready),
> str(newconnections)))
> + newconnections.append(controllersock)
> + else:
> + print("Accepting %s (%s)" % (str(ready),
> str(newconnections)))
> + self.controllersock = controllersock
> + fds.append(controllersock)
> if self.controllersock in ready:
> try:
> - print("Connecting Client")
> + print("Processing Client")
> ui_fds = recvfds(self.controllersock, 3)
> + print("Connecting Client")
>
> # Where to write events to
> writer = ConnectionWriter(ui_fds[0])
> @@ -239,6 +241,12 @@ class ProcessServer(multiprocessing.Process):
> while not lock:
> with bb.utils.timeout(3):
> lock = bb.utils.lockfile(lockfile, shared=False,
> retry=False, block=True)
> + if lock:
> + # We hold the lock so we can remove the file
> (hide stale pid data)
> + bb.utils.remove(lockfile)
> + bb.utils.unlockfile(lock)
> + return
> +
> if not lock:
> # Some systems may not have lsof available
> procs = None
> @@ -259,10 +267,6 @@ class ProcessServer(multiprocessing.Process):
> if procs:
> msg += ":\n%s" % str(procs)
> print(msg)
> - return
> - # We hold the lock so we can remove the file (hide stale pid
> data)
> - bb.utils.remove(lockfile)
> - bb.utils.unlockfile(lock)
>
> def idle_commands(self, delay, fds=None):
> nextsleep = delay
> @@ -398,36 +402,45 @@ class BitBakeServer(object):
> os.close(self.readypipein)
>
> ready = ConnectionReader(self.readypipe)
> - r = ready.poll(30)
> + r = ready.poll(5)
> + if not r:
> + bb.note("Bitbake server didn't start within 5 seconds,
> waiting for 90")
> + r = ready.poll(90)
> if r:
> try:
> r = ready.get()
> except EOFError:
> # Trap the child exitting/closing the pipe and error
> out r = None
> - if not r or r != "ready":
> + if not r or r[0] != "r":
> ready.close()
> - bb.error("Unable to start bitbake server")
> + bb.error("Unable to start bitbake server (%s)" % str(r))
> if os.path.exists(logfile):
> logstart_re = re.compile(self.start_log_format %
> ('([0-9]+)', '([0-9-]+ [0-9:.]+)')) started = False
> lines = []
> + lastlines = []
> with open(logfile, "r") as f:
> for line in f:
> if started:
> lines.append(line)
> else:
> + lastlines.append(line)
> res = logstart_re.match(line.rstrip())
> if res:
> ldatetime =
> datetime.datetime.strptime(res.group(2),
> self.start_log_datetime_format) if ldatetime >= startdatetime:
> started = True lines.append(line)
> + if len(lastlines) > 60:
> + lastlines = lastlines[-60:]
> if lines:
> - if len(lines) > 10:
> - bb.error("Last 10 lines of server log for
> this session (%s):\n%s" % (logfile, "".join(lines[-10:])))
> + if len(lines) > 60:
> + bb.error("Last 60 lines of server log for
> this session (%s):\n%s" % (logfile, "".join(lines[-60:]))) else:
> bb.error("Server log for this session
> (%s):\n%s" % (logfile, "".join(lines)))
> + elif lastlines:
> + bb.error("Server didn't start, last 60
> loglines (%s):\n%s" % (logfile, "".join(lastlines))) else:
> bb.error("%s doesn't exist" % logfile)
>
> @@ -437,17 +450,24 @@ class BitBakeServer(object):
>
> def _startServer(self):
> print(self.start_log_format % (os.getpid(),
> datetime.datetime.now().strftime(self.start_log_datetime_format)))
> + sys.stdout.flush()
> +
> server = ProcessServer(self.bitbake_lock, self.sock,
> self.sockname)
> self.configuration.setServerRegIdleCallback(server.register_idle_function)
> os.close(self.readypipe) writer = ConnectionWriter(self.readypipein)
> - self.cooker = bb.cooker.BBCooker(self.configuration,
> self.featureset)
> - writer.send("ready")
> + try:
> + self.cooker = bb.cooker.BBCooker(self.configuration,
> self.featureset)
> + except bb.BBHandledException:
> + return None
> + writer.send("r")
> writer.close()
> server.cooker = self.cooker
> server.server_timeout = self.configuration.server_timeout
> server.xmlrpcinterface = self.configuration.xmlrpcinterface
> print("Started bitbake server pid %d" % os.getpid())
> + sys.stdout.flush()
> +
> server.start()
>
> def connectProcessServer(sockname, featureset):
> @@ -459,10 +479,20 @@ def connectProcessServer(sockname, featureset):
> readfd = writefd = readfd1 = writefd1 = readfd2 = writefd2 = None
> eq = command_chan_recv = command_chan = None
>
> + sock.settimeout(10)
> +
> try:
> try:
> os.chdir(os.path.dirname(sockname))
> - sock.connect(os.path.basename(sockname))
> + finished = False
> + while not finished:
> + try:
> + sock.connect(os.path.basename(sockname))
> + finished = True
> + except IOError as e:
> + if e.errno == errno.EWOULDBLOCK:
> + pass
> + raise
> finally:
> os.chdir(cwd)
>
> @@ -493,7 +523,8 @@ def connectProcessServer(sockname, featureset):
> command_chan.close()
> for i in [writefd, readfd1, writefd2]:
> try:
> - os.close(i)
> + if i:
> + os.close(i)
> except OSError:
> pass
> sock.close()
> diff --git a/bitbake/lib/bb/server/xmlrpcclient.py
> b/bitbake/lib/bb/server/xmlrpcclient.py index 4661a9e..c054c3c 100644
> --- a/bitbake/lib/bb/server/xmlrpcclient.py
> +++ b/bitbake/lib/bb/server/xmlrpcclient.py
> @@ -4,18 +4,8 @@
> # Copyright (C) 2006 - 2007 Michael 'Mickey' Lauer
> # Copyright (C) 2006 - 2008 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import os
> import sys
> diff --git a/bitbake/lib/bb/server/xmlrpcserver.py
> b/bitbake/lib/bb/server/xmlrpcserver.py index 875b128..54fa32f 100644
> --- a/bitbake/lib/bb/server/xmlrpcserver.py
> +++ b/bitbake/lib/bb/server/xmlrpcserver.py
> @@ -4,18 +4,8 @@
> # Copyright (C) 2006 - 2007 Michael 'Mickey' Lauer
> # Copyright (C) 2006 - 2008 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import os
> import sys
> diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py
> index fdbb2a3..a4bb1ff 100644
> --- a/bitbake/lib/bb/siggen.py
> +++ b/bitbake/lib/bb/siggen.py
> @@ -1,3 +1,7 @@
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> import hashlib
> import logging
> import os
> @@ -8,6 +12,8 @@ import bb.data
> import difflib
> import simplediff
> from bb.checksum import FileChecksumCache
> +from bb import runqueue
> +import hashserv
>
> logger = logging.getLogger('BitBake.SigGen')
>
> @@ -37,12 +43,18 @@ class SignatureGenerator(object):
> self.runtaskdeps = {}
> self.file_checksum_values = {}
> self.taints = {}
> + self.unitaskhashes = {}
> + self.setscenetasks = {}
>
> def finalise(self, fn, d, varient):
> return
>
> - def get_taskhash(self, fn, task, deps, dataCache):
> - return "0"
> + def get_unihash(self, tid):
> + return self.taskhash[tid]
> +
> + def get_taskhash(self, tid, deps, dataCache):
> + self.taskhash[tid] =
> hashlib.sha256(tid.encode("utf-8")).hexdigest()
> + return self.taskhash[tid]
>
> def writeout_file_checksum_cache(self):
> """Write/update the file checksum cache onto disk"""
> @@ -64,14 +76,25 @@ class SignatureGenerator(object):
> return
>
> def get_taskdata(self):
> - return (self.runtaskdeps, self.taskhash,
> self.file_checksum_values, self.taints, self.basehash)
> + return (self.runtaskdeps, self.taskhash,
> self.file_checksum_values, self.taints, self.basehash,
> self.unitaskhashes, self.setscenetasks) def set_taskdata(self, data):
> - self.runtaskdeps, self.taskhash, self.file_checksum_values,
> self.taints, self.basehash = data
> + self.runtaskdeps, self.taskhash, self.file_checksum_values,
> self.taints, self.basehash, self.unitaskhashes, self.setscenetasks =
> data def reset(self, data):
> self.__init__(data)
>
> + def get_taskhashes(self):
> + return self.taskhash, self.unitaskhashes
> +
> + def set_taskhashes(self, hashes):
> + self.taskhash, self.unitaskhashes = hashes
> +
> + def save_unitaskhashes(self):
> + return
> +
> + def set_setscene_tasks(self, setscene_tasks):
> + return
>
> class SignatureGeneratorBasic(SignatureGenerator):
> """
> @@ -87,7 +110,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
> self.taints = {}
> self.gendeps = {}
> self.lookupcache = {}
> - self.pkgnameextract = re.compile("(?P<fn>.*)\..*")
> + self.setscenetasks = {}
> self.basewhitelist =
> set((data.getVar("BB_HASHBASE_WHITELIST") or "").split())
> self.taskwhitelist = None self.init_rundepcheck(data)
> @@ -98,6 +121,9 @@ class SignatureGeneratorBasic(SignatureGenerator):
> else:
> self.checksum_cache = None
>
> + self.unihash_cache = bb.cache.SimpleCache("1")
> + self.unitaskhashes = self.unihash_cache.init_cache(data,
> "bb_unihashes.dat", {}) +
> def init_rundepcheck(self, data):
> self.taskwhitelist = data.getVar("BB_HASHTASK_WHITELIST") or
> None if self.taskwhitelist:
> @@ -113,10 +139,16 @@ class
> SignatureGeneratorBasic(SignatureGenerator): taskdeps, basehash =
> bb.data.generate_dependency_hash(tasklist, gendeps, lookupcache,
> self.basewhitelist, fn) for task in tasklist:
> - k = fn + "." + task
> - if not ignore_mismatch and k in self.basehash and
> self.basehash[k] != basehash[k]:
> - bb.error("When reparsing %s, the basehash value
> changed from %s to %s. The metadata is not deterministic and this
> needs to be fixed." % (k, self.basehash[k], basehash[k]))
> - self.basehash[k] = basehash[k]
> + tid = fn + ":" + task
> + if not ignore_mismatch and tid in self.basehash and
> self.basehash[tid] != basehash[tid]:
> + bb.error("When reparsing %s, the basehash value
> changed from %s to %s. The metadata is not deterministic and this
> needs to be fixed." % (tid, self.basehash[tid], basehash[tid]))
> + bb.error("The following commands may help:")
> + cmd = "$ bitbake %s -c%s" % (d.getVar('PN'), task)
> + # Make sure sigdata is dumped before run printdiff
> + bb.error("%s -Snone" % cmd)
> + bb.error("Then:")
> + bb.error("%s -Sprintdiff\n" % cmd)
> + self.basehash[tid] = basehash[tid]
>
> self.taskdeps[fn] = taskdeps
> self.gendeps[fn] = gendeps
> @@ -124,6 +156,9 @@ class SignatureGeneratorBasic(SignatureGenerator):
>
> return taskdeps
>
> + def set_setscene_tasks(self, setscene_tasks):
> + self.setscenetasks = setscene_tasks
> +
> def finalise(self, fn, d, variant):
>
> mc = d.getVar("__BBMULTICONFIG", False) or ""
> @@ -143,7 +178,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
> # self.dump_sigtask(fn, task, d.getVar("STAMP"), False)
>
> for task in taskdeps:
> - d.setVar("BB_BASEHASH_task-%s" % task, self.basehash[fn
> + "." + task])
> + d.setVar("BB_BASEHASH_task-%s" % task, self.basehash[fn
> + ":" + task])
> def rundep_check(self, fn, recipename, task, dep, depname,
> dataCache): # Return True if we should keep the dependency, False to
> drop it @@ -163,31 +198,26 @@ class
> SignatureGeneratorBasic(SignatureGenerator): pass
> return taint
>
> - def get_taskhash(self, fn, task, deps, dataCache):
> + def get_taskhash(self, tid, deps, dataCache):
>
> - mc = ''
> - if fn.startswith('multiconfig:'):
> - mc = fn.split(':')[1]
> - k = fn + "." + task
> + (mc, _, task, fn) = bb.runqueue.split_tid_mcfn(tid)
>
> - data = dataCache.basetaskhash[k]
> - self.basehash[k] = data
> - self.runtaskdeps[k] = []
> - self.file_checksum_values[k] = []
> + data = dataCache.basetaskhash[tid]
> + self.basehash[tid] = data
> + self.runtaskdeps[tid] = []
> + self.file_checksum_values[tid] = []
> recipename = dataCache.pkg_fn[fn]
> for dep in sorted(deps, key=clean_basepath):
> - pkgname = self.pkgnameextract.search(dep).group('fn')
> - if mc:
> - depmc = pkgname.split(':')[1]
> - if mc != depmc:
> - continue
> - depname = dataCache.pkg_fn[pkgname]
> + (depmc, _, deptaskname, depfn) =
> bb.runqueue.split_tid_mcfn(dep)
> + if mc != depmc:
> + continue
> + depname = dataCache.pkg_fn[depfn]
> if not self.rundep_check(fn, recipename, task, dep,
> depname, dataCache): continue
> if dep not in self.taskhash:
> bb.fatal("%s is not in taskhash, caller isn't
> calling in dependency order?" % dep)
> - data = data + self.taskhash[dep]
> - self.runtaskdeps[k].append(dep)
> + data = data + self.get_unihash(dep)
> + self.runtaskdeps[tid].append(dep)
>
> if task in dataCache.file_checksums[fn]:
> if self.checksum_cache:
> @@ -195,7 +225,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
> else:
> checksums =
> bb.fetch2.get_file_checksums(dataCache.file_checksums[fn][task],
> recipename) for (f,cs) in checksums:
> - self.file_checksum_values[k].append((f,cs))
> + self.file_checksum_values[tid].append((f,cs))
> if cs:
> data = data + cs
>
> @@ -205,16 +235,16 @@ class
> SignatureGeneratorBasic(SignatureGenerator): import uuid
> taint = str(uuid.uuid4())
> data = data + taint
> - self.taints[k] = "nostamp:" + taint
> + self.taints[tid] = "nostamp:" + taint
>
> taint = self.read_taint(fn, task, dataCache.stamp[fn])
> if taint:
> data = data + taint
> - self.taints[k] = taint
> - logger.warning("%s is tainted from a forced run" % k)
> + self.taints[tid] = taint
> + logger.warning("%s is tainted from a forced run" % tid)
>
> - h = hashlib.md5(data.encode("utf-8")).hexdigest()
> - self.taskhash[k] = h
> + h = hashlib.sha256(data.encode("utf-8")).hexdigest()
> + self.taskhash[tid] = h
> #d.setVar("BB_TASKHASH_task-%s" % task, taskhash[task])
> return h
>
> @@ -227,17 +257,20 @@ class
> SignatureGeneratorBasic(SignatureGenerator):
> bb.fetch2.fetcher_parse_save() bb.fetch2.fetcher_parse_done()
>
> + def save_unitaskhashes(self):
> + self.unihash_cache.save(self.unitaskhashes)
> +
> def dump_sigtask(self, fn, task, stampbase, runtime):
>
> - k = fn + "." + task
> + tid = fn + ":" + task
> referencestamp = stampbase
> if isinstance(runtime, str) and
> runtime.startswith("customfile"): sigfile = stampbase
> referencestamp = runtime[11:]
> - elif runtime and k in self.taskhash:
> - sigfile = stampbase + "." + task + ".sigdata" + "." +
> self.taskhash[k]
> + elif runtime and tid in self.taskhash:
> + sigfile = stampbase + "." + task + ".sigdata" + "." +
> self.get_unihash(tid) else:
> - sigfile = stampbase + "." + task + ".sigbasedata" + "."
> + self.basehash[k]
> + sigfile = stampbase + "." + task + ".sigbasedata" + "."
> + self.basehash[tid]
> bb.utils.mkdirhier(os.path.dirname(sigfile))
>
> @@ -246,7 +279,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
> data['basewhitelist'] = self.basewhitelist
> data['taskwhitelist'] = self.taskwhitelist
> data['taskdeps'] = self.taskdeps[fn][task]
> - data['basehash'] = self.basehash[k]
> + data['basehash'] = self.basehash[tid]
> data['gendeps'] = {}
> data['varvals'] = {}
> data['varvals'][task] = self.lookupcache[fn][task]
> @@ -256,30 +289,31 @@ class
> SignatureGeneratorBasic(SignatureGenerator): data['gendeps'][dep] =
> self.gendeps[fn][dep] data['varvals'][dep] = self.lookupcache[fn][dep]
>
> - if runtime and k in self.taskhash:
> - data['runtaskdeps'] = self.runtaskdeps[k]
> - data['file_checksum_values'] = [(os.path.basename(f),
> cs) for f,cs in self.file_checksum_values[k]]
> + if runtime and tid in self.taskhash:
> + data['runtaskdeps'] = self.runtaskdeps[tid]
> + data['file_checksum_values'] = [(os.path.basename(f),
> cs) for f,cs in self.file_checksum_values[tid]] data['runtaskhashes']
> = {} for dep in data['runtaskdeps']:
> - data['runtaskhashes'][dep] = self.taskhash[dep]
> - data['taskhash'] = self.taskhash[k]
> + data['runtaskhashes'][dep] = self.get_unihash(dep)
> + data['taskhash'] = self.taskhash[tid]
> + data['unihash'] = self.get_unihash(tid)
>
> taint = self.read_taint(fn, task, referencestamp)
> if taint:
> data['taint'] = taint
>
> - if runtime and k in self.taints:
> - if 'nostamp:' in self.taints[k]:
> - data['taint'] = self.taints[k]
> + if runtime and tid in self.taints:
> + if 'nostamp:' in self.taints[tid]:
> + data['taint'] = self.taints[tid]
>
> computed_basehash = calc_basehash(data)
> - if computed_basehash != self.basehash[k]:
> - bb.error("Basehash mismatch %s versus %s for %s" %
> (computed_basehash, self.basehash[k], k))
> - if runtime and k in self.taskhash:
> + if computed_basehash != self.basehash[tid]:
> + bb.error("Basehash mismatch %s versus %s for %s" %
> (computed_basehash, self.basehash[tid], tid))
> + if runtime and tid in self.taskhash:
> computed_taskhash = calc_taskhash(data)
> - if computed_taskhash != self.taskhash[k]:
> - bb.error("Taskhash mismatch %s versus %s for %s" %
> (computed_taskhash, self.taskhash[k], k))
> - sigfile = sigfile.replace(self.taskhash[k],
> computed_taskhash)
> + if computed_taskhash != self.taskhash[tid]:
> + bb.error("Taskhash mismatch %s versus %s for %s" %
> (computed_taskhash, self.taskhash[tid], tid))
> + sigfile = sigfile.replace(self.taskhash[tid],
> computed_taskhash)
> fd, tmpfile = tempfile.mkstemp(dir=os.path.dirname(sigfile),
> prefix="sigtask.") try:
> @@ -299,30 +333,34 @@ class
> SignatureGeneratorBasic(SignatureGenerator): if fn in self.taskdeps:
> for task in self.taskdeps[fn]:
> tid = fn + ":" + task
> - (mc, _, _) = bb.runqueue.split_tid(tid)
> - k = fn + "." + task
> - if k not in self.taskhash:
> + mc = bb.runqueue.mc_from_tid(tid)
> + if tid not in self.taskhash:
> continue
> - if dataCaches[mc].basetaskhash[k] !=
> self.basehash[k]:
> - bb.error("Bitbake's cached basehash does not
> match the one we just generated (%s)!" % k)
> - bb.error("The mismatched hashes were %s and %s"
> % (dataCaches[mc].basetaskhash[k], self.basehash[k]))
> + if dataCaches[mc].basetaskhash[tid] !=
> self.basehash[tid]:
> + bb.error("Bitbake's cached basehash does not
> match the one we just generated (%s)!" % tid)
> + bb.error("The mismatched hashes were %s and %s"
> % (dataCaches[mc].basetaskhash[tid], self.basehash[tid]))
> self.dump_sigtask(fn, task, dataCaches[mc].stamp[fn], True)
> class SignatureGeneratorBasicHash(SignatureGeneratorBasic):
> name = "basichash"
>
> + def get_stampfile_hash(self, tid):
> + if tid in self.taskhash:
> + return self.taskhash[tid]
> +
> + # If task is not in basehash, then error
> + return self.basehash[tid]
> +
> def stampfile(self, stampbase, fn, taskname, extrainfo,
> clean=False): if taskname != "do_setscene" and
> taskname.endswith("_setscene"):
> - k = fn + "." + taskname[:-9]
> + tid = fn + ":" + taskname[:-9]
> else:
> - k = fn + "." + taskname
> + tid = fn + ":" + taskname
> if clean:
> h = "*"
> - elif k in self.taskhash:
> - h = self.taskhash[k]
> else:
> - # If k is not in basehash, then error
> - h = self.basehash[k]
> + h = self.get_stampfile_hash(tid)
> +
> return ("%s.%s.%s.%s" % (stampbase, taskname, h,
> extrainfo)).rstrip('.')
> def stampcleanmask(self, stampbase, fn, taskname, extrainfo):
> @@ -332,6 +370,172 @@ class
> SignatureGeneratorBasicHash(SignatureGeneratorBasic):
> bb.note("Tainting hash to force rebuild of task %s, %s" % (fn, task))
> bb.build.write_taint(task, d, fn)
> +class SignatureGeneratorUniHashMixIn(object):
> + def get_taskdata(self):
> + return (self.server, self.method) + super().get_taskdata()
> +
> + def set_taskdata(self, data):
> + self.server, self.method = data[:2]
> + super().set_taskdata(data[2:])
> +
> + def client(self):
> + if getattr(self, '_client', None) is None:
> + self._client = hashserv.create_client(self.server)
> + return self._client
> +
> + def __get_task_unihash_key(self, tid):
> + # TODO: The key only *needs* to be the taskhash, the tid is
> just
> + # convenient
> + return '%s:%s' % (tid.rsplit("/", 1)[1], self.taskhash[tid])
> +
> + def get_stampfile_hash(self, tid):
> + if tid in self.taskhash:
> + # If a unique hash is reported, use it as the stampfile
> hash. This
> + # ensures that if a task won't be re-run if the taskhash
> changes,
> + # but it would result in the same output hash
> + unihash =
> self.unitaskhashes.get(self.__get_task_unihash_key(tid), None)
> + if unihash is not None:
> + return unihash
> +
> + return super().get_stampfile_hash(tid)
> +
> + def set_unihash(self, tid, unihash):
> + self.unitaskhashes[self.__get_task_unihash_key(tid)] =
> unihash +
> + def get_unihash(self, tid):
> + taskhash = self.taskhash[tid]
> +
> + # If its not a setscene task we can return
> + if self.setscenetasks and tid not in self.setscenetasks:
> + return taskhash
> +
> + key = self.__get_task_unihash_key(tid)
> +
> + # TODO: This cache can grow unbounded. It probably only
> needs to keep
> + # for each task
> + unihash = self.unitaskhashes.get(key, None)
> + if unihash is not None:
> + return unihash
> +
> + # In the absence of being able to discover a unique hash
> from the
> + # server, make it be equivalent to the taskhash. The unique
> "hash" only
> + # really needs to be a unique string (not even necessarily a
> hash), but
> + # making it match the taskhash has a few advantages:
> + #
> + # 1) All of the sstate code that assumes hashes can be the
> same
> + # 2) It provides maximal compatibility with builders that
> don't use
> + # an equivalency server
> + # 3) The value is easy for multiple independent builders to
> derive the
> + # same unique hash from the same input. This means that
> if the
> + # independent builders find the same taskhash, but it
> isn't reported
> + # to the server, there is a better chance that they will
> agree on
> + # the unique hash.
> + unihash = taskhash
> +
> + try:
> + data = self.client().get_unihash(self.method,
> self.taskhash[tid])
> + if data:
> + unihash = data
> + # A unique hash equal to the taskhash is not very
> interesting,
> + # so it is reported it at debug level 2. If they
> differ, that
> + # is much more interesting, so it is reported at
> debug level 1
> + bb.debug((1, 2)[unihash == taskhash], 'Found unihash
> %s in place of %s for %s from %s' % (unihash, taskhash, tid,
> self.server))
> + else:
> + bb.debug(2, 'No reported unihash for %s:%s from %s'
> % (tid, taskhash, self.server))
> + except hashserv.client.HashConnectionError as e:
> + bb.warn('Error contacting Hash Equivalence Server %s:
> %s' % (self.server, str(e))) +
> + self.unitaskhashes[key] = unihash
> + return unihash
> +
> + def report_unihash(self, path, task, d):
> + import importlib
> +
> + taskhash = d.getVar('BB_TASKHASH')
> + unihash = d.getVar('BB_UNIHASH')
> + report_taskdata =
> d.getVar('SSTATE_HASHEQUIV_REPORT_TASKDATA') == '1'
> + tempdir = d.getVar('T')
> + fn = d.getVar('BB_FILENAME')
> + tid = fn + ':do_' + task
> + key = tid.rsplit("/", 1)[1] + ':' + taskhash
> +
> + if self.setscenetasks and tid not in self.setscenetasks:
> + return
> +
> + # Sanity checks
> + cache_unihash = self.unitaskhashes.get(key, None)
> + if cache_unihash is None:
> + bb.fatal('%s not in unihash cache. Please report this
> error' % key) +
> + if cache_unihash != unihash:
> + bb.fatal("Cache unihash %s doesn't match BB_UNIHASH %s"
> % (cache_unihash, unihash)) +
> + sigfile = None
> + sigfile_name = "depsig.do_%s.%d" % (task, os.getpid())
> + sigfile_link = "depsig.do_%s" % task
> +
> + try:
> + sigfile = open(os.path.join(tempdir, sigfile_name),
> 'w+b') +
> + locs = {'path': path, 'sigfile': sigfile, 'task': task,
> 'd': d} +
> + if "." in self.method:
> + (module, method) = self.method.rsplit('.', 1)
> + locs['method'] =
> getattr(importlib.import_module(module), method)
> + outhash = bb.utils.better_eval('method(path,
> sigfile, task, d)', locs)
> + else:
> + outhash = bb.utils.better_eval(self.method + '(path,
> sigfile, task, d)', locs) +
> + try:
> + extra_data = {}
> +
> + owner = d.getVar('SSTATE_HASHEQUIV_OWNER')
> + if owner:
> + extra_data['owner'] = owner
> +
> + if report_taskdata:
> + sigfile.seek(0)
> +
> + extra_data['PN'] = d.getVar('PN')
> + extra_data['PV'] = d.getVar('PV')
> + extra_data['PR'] = d.getVar('PR')
> + extra_data['task'] = task
> + extra_data['outhash_siginfo'] =
> sigfile.read().decode('utf-8') +
> + data = self.client().report_unihash(taskhash,
> self.method, outhash, unihash, extra_data)
> + new_unihash = data['unihash']
> +
> + if new_unihash != unihash:
> + bb.debug(1, 'Task %s unihash changed %s -> %s by
> server %s' % (taskhash, unihash, new_unihash, self.server))
> + bb.event.fire(bb.runqueue.taskUniHashUpdate(fn +
> ':do_' + task, new_unihash), d)
> + else:
> + bb.debug(1, 'Reported task %s as unihash %s to
> %s' % (taskhash, unihash, self.server))
> + except hashserv.client.HashConnectionError as e:
> + bb.warn('Error contacting Hash Equivalence Server
> %s: %s' % (self.server, str(e)))
> + finally:
> + if sigfile:
> + sigfile.close()
> +
> + sigfile_link_path = os.path.join(tempdir,
> sigfile_link)
> + bb.utils.remove(sigfile_link_path)
> +
> + try:
> + os.symlink(sigfile_name, sigfile_link_path)
> + except OSError:
> + pass
> +
> +
> +#
> +# Dummy class used for bitbake-selftest
> +#
> +class
> SignatureGeneratorTestEquivHash(SignatureGeneratorUniHashMixIn,
> SignatureGeneratorBasicHash):
> + name = "TestEquivHash"
> + def init_rundepcheck(self, data):
> + super().init_rundepcheck(data)
> + self.server = data.getVar('BB_HASHSERVE')
> + self.method = "sstate_output_hash"
> +
> +
> def dump_this_task(outfile, d):
> import bb.parse
> fn = d.getVar("BB_FILENAME")
> @@ -392,13 +596,13 @@ def list_inline_diff(oldlist, newlist,
> colors=None):
> def clean_basepath(a):
> mc = None
> - if a.startswith("multiconfig:"):
> + if a.startswith("mc:"):
> _, mc, a = a.split(":", 2)
> b = a.rsplit("/", 2)[1] + '/' + a.rsplit("/", 2)[2]
> if a.startswith("virtual:"):
> b = b + ":" + a.rsplit(":", 1)[0]
> if mc:
> - b = b + ":multiconfig:" + mc
> + b = b + ":mc:" + mc
> return b
>
> def clean_basepaths(a):
> @@ -623,6 +827,10 @@ def compare_sigfiles(a, b, recursecb=None,
> color=False, collapsed=False): a_taint = a_data.get('taint', None)
> b_taint = b_data.get('taint', None)
> if a_taint != b_taint:
> + if a_taint and a_taint.startswith('nostamp:'):
> + a_taint = a_taint.replace('nostamp:', 'nostamp(uuid4):')
> + if b_taint and b_taint.startswith('nostamp:'):
> + b_taint = b_taint.replace('nostamp:', 'nostamp(uuid4):')
> output.append(color_format("{color_title}Taint (by
> forced/invalidated task) changed{color_default} from %s to %s") %
> (a_taint, b_taint)) return output
> @@ -642,7 +850,7 @@ def calc_basehash(sigdata):
> if val is not None:
> basedata = basedata + str(val)
>
> - return hashlib.md5(basedata.encode("utf-8")).hexdigest()
> + return hashlib.sha256(basedata.encode("utf-8")).hexdigest()
>
> def calc_taskhash(sigdata):
> data = sigdata['basehash']
> @@ -660,7 +868,7 @@ def calc_taskhash(sigdata):
> else:
> data = data + sigdata['taint']
>
> - return hashlib.md5(data.encode("utf-8")).hexdigest()
> + return hashlib.sha256(data.encode("utf-8")).hexdigest()
>
>
> def dump_sigfile(a):
> @@ -695,7 +903,11 @@ def dump_sigfile(a):
> output.append("Hash for dependent task %s is %s" % (dep,
> a_data['runtaskhashes'][dep]))
> if 'taint' in a_data:
> - output.append("Tainted (by forced/invalidated task): %s" %
> a_data['taint'])
> + if a_data['taint'].startswith('nostamp:'):
> + msg = a_data['taint'].replace('nostamp:',
> 'nostamp(uuid4):')
> + else:
> + msg = a_data['taint']
> + output.append("Tainted (by forced/invalidated task): %s" %
> msg)
> if 'task' in a_data:
> computed_basehash = calc_basehash(a_data)
> diff --git a/bitbake/lib/bb/taskdata.py b/bitbake/lib/bb/taskdata.py
> index 94e822c..8c25e09 100644
> --- a/bitbake/lib/bb/taskdata.py
> +++ b/bitbake/lib/bb/taskdata.py
> @@ -1,6 +1,3 @@
> -#!/usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake 'TaskData' implementation
>
> @@ -10,18 +7,8 @@ Task data collection and handling
>
> # Copyright (C) 2006 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import logging
> import re
> @@ -93,7 +80,7 @@ class TaskData:
> def add_mcdepends(task):
> for dep in task_deps['mcdepends'][task].split():
> if len(dep.split(':')) != 5:
> - bb.msg.fatal("TaskData", "Error for %s:%s[%s],
> multiconfig dependency %s does not contain exactly four ':'
> characters.\n Task '%s' should be specified in the form
> 'multiconfig:fromMC:toMC:packagename:task'" % (fn, task, 'mcdepends',
> dep, 'mcdepends'))
> + bb.msg.fatal("TaskData", "Error for %s:%s[%s],
> multiconfig dependency %s does not contain exactly four ':'
> characters.\n Task '%s' should be specified in the form
> 'mc:fromMC:toMC:packagename:task'" % (fn, task, 'mcdepends', dep,
> 'mcdepends')) if dep not in self.mcdepends:
> self.mcdepends.append(dep) diff --git
> a/bitbake/lib/bb/tests/codeparser.py
> b/bitbake/lib/bb/tests/codeparser.py index e30e78c..826a2d2 100644
> --- a/bitbake/lib/bb/tests/codeparser.py +++
> b/bitbake/lib/bb/tests/codeparser.py @@ -1,23 +1,10 @@ -#
> ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4;
> indent-tabs-mode: nil -*- #
> # BitBake Test for codeparser.py
> #
> # Copyright (C) 2010 Chris Larson
> # Copyright (C) 2012 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
>
> import unittest
> @@ -123,6 +110,13 @@ ${D}${libdir}/pkgconfig/*.pc
> self.parseExpression("sed -i -e 's:IP{:I${:g' $pc")
> self.assertExecs(set(["sed"]))
>
> + def test_parameter_expansion_modifiers(self):
> + # - and + are also valid modifiers for parameter expansion,
> but are
> + # valid characters in bitbake variable names, so are not
> included here
> + for i in ('=', ':-', ':=', '?', ':?', ':+', '#', '%', '##',
> '%%'):
> + name = "foo%sbar" % i
> + self.parseExpression("${%s}" % name)
> + self.assertNotIn(name, self.references)
>
> def test_until(self):
> self.parseExpression("until false; do echo true; done")
> diff --git a/bitbake/lib/bb/tests/cooker.py
> b/bitbake/lib/bb/tests/cooker.py index 2b44236..090916e 100644
> --- a/bitbake/lib/bb/tests/cooker.py
> +++ b/bitbake/lib/bb/tests/cooker.py
> @@ -1,20 +1,7 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # BitBake Tests for cooker.py
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
>
> import unittest
> diff --git a/bitbake/lib/bb/tests/cow.py b/bitbake/lib/bb/tests/cow.py
> index d149d84..b4af4bb 100644
> --- a/bitbake/lib/bb/tests/cow.py
> +++ b/bitbake/lib/bb/tests/cow.py
> @@ -1,22 +1,9 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # BitBake Tests for Copy-on-Write (cow.py)
> #
> -# Copyright 2006 Holger Freyther <freyther@handhelds.org>
> -#
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# Copyright 2006 Holger Freyther
> <freyther@handhelds.org> #
>
> import unittest
> diff --git a/bitbake/lib/bb/tests/data.py
> b/bitbake/lib/bb/tests/data.py index db3e201..3e49984 100644
> --- a/bitbake/lib/bb/tests/data.py
> +++ b/bitbake/lib/bb/tests/data.py
> @@ -1,23 +1,10 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # BitBake Tests for the Data Store (data.py/data_smart.py)
> #
> # Copyright (C) 2010 Chris Larson
> # Copyright (C) 2012 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
>
> import unittest
> @@ -394,6 +381,28 @@ class TestOverrides(unittest.TestCase):
> self.d.setVar("OVERRIDES", "foo:bar:some_val")
> self.assertEqual(self.d.getVar("TEST"), " testvalue5")
>
> + def test_append_and_override_1(self):
> + self.d.setVar("TEST_append", "testvalue2")
> + self.d.setVar("TEST_bar", "testvalue3")
> + self.assertEqual(self.d.getVar("TEST"),
> "testvalue3testvalue2") +
> + def test_append_and_override_2(self):
> + self.d.setVar("TEST_append_bar", "testvalue2")
> + self.assertEqual(self.d.getVar("TEST"),
> "testvaluetestvalue2") +
> + def test_append_and_override_3(self):
> + self.d.setVar("TEST_bar_append", "testvalue2")
> + self.assertEqual(self.d.getVar("TEST"), "testvalue2")
> +
> + # Test an override with _<numeric> in it based on a real world
> OE issue
> + def test_underscore_override(self):
> + self.d.setVar("TARGET_ARCH", "x86_64")
> + self.d.setVar("PN", "test-${TARGET_ARCH}")
> + self.d.setVar("VERSION", "1")
> + self.d.setVar("VERSION_pn-test-${TARGET_ARCH}", "2")
> + self.d.setVar("OVERRIDES", "pn-${PN}")
> + bb.data.expandKeys(self.d)
> + self.assertEqual(self.d.getVar("VERSION"), "2")
>
> class TestKeyExpansion(unittest.TestCase):
> def setUp(self):
> @@ -470,7 +479,7 @@ class TaskHash(unittest.TestCase):
> tasklist, gendeps, lookupcache =
> bb.data.generate_dependencies(d) taskdeps, basehash =
> bb.data.generate_dependency_hash(tasklist, gendeps, lookupcache,
> set(), "somefile") bb.warn(str(lookupcache))
> - return basehash["somefile." + taskname]
> + return basehash["somefile:" + taskname]
>
> d = bb.data.init()
> d.setVar("__BBTASKS", ["mytask"])
> diff --git a/bitbake/lib/bb/tests/event.py
> b/bitbake/lib/bb/tests/event.py index d3a5f62..9229b63 100644
> --- a/bitbake/lib/bb/tests/event.py
> +++ b/bitbake/lib/bb/tests/event.py
> @@ -1,22 +1,9 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # BitBake Tests for the Event implementation (event.py)
> #
> # Copyright (C) 2017 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
>
> import unittest
> @@ -574,14 +561,6 @@ class EventClassesTest(unittest.TestCase):
> self.assertEqual(event.fn(1), callback(1))
> self.assertEqual(event.pid, EventClassesTest._worker_pid)
>
> - def test_StampUpdate(self):
> - targets = ["foo", "bar"]
> - stampfns = [lambda:"foobar"]
> - event = bb.event.StampUpdate(targets, stampfns)
> - self.assertEqual(event.targets, targets)
> - self.assertEqual(event.stampPrefix, stampfns)
> - self.assertEqual(event.pid, EventClassesTest._worker_pid)
> -
> def test_BuildBase(self):
> """ Test base class for bitbake build events """
> name = "foo"
> diff --git a/bitbake/lib/bb/tests/fetch.py
> b/bitbake/lib/bb/tests/fetch.py index 6848095..a0b656b 100644
> --- a/bitbake/lib/bb/tests/fetch.py
> +++ b/bitbake/lib/bb/tests/fetch.py
> @@ -1,22 +1,9 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # BitBake Tests for the Fetcher (fetch2/)
> #
> # Copyright (C) 2012 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
>
> import unittest
> @@ -893,12 +880,201 @@ class FetcherNetworkTest(FetcherTest):
>
> @skipIfNoNetwork()
> def test_git_submodule(self):
> - fetcher =
> bb.fetch.Fetch(["gitsm://git.yoctoproject.org/git-submodule-test;rev=f12e57f2edf0aa534cf1616fa983d165a92b0842"],
> self.d)
> + # URL with ssh submodules
> + url =
> "gitsm://git.yoctoproject.org/git-submodule-test;branch=ssh-gitsm-tests;rev=049da4a6cb198d7c0302e9e8b243a1443cb809a7"
> + # Original URL (comment this if you have ssh access to
> git.yoctoproject.org)
> + url =
> "gitsm://git.yoctoproject.org/git-submodule-test;branch=master;rev=a2885dd7d25380d23627e7544b7bbb55014b16ee"
> + fetcher = bb.fetch.Fetch([url], self.d)
> + fetcher.download()
> + # Previous cwd has been deleted
> + os.chdir(os.path.dirname(self.unpackdir))
> + fetcher.unpack(self.unpackdir)
> +
> + repo_path = os.path.join(self.tempdir, 'unpacked', 'git')
> + self.assertTrue(os.path.exists(repo_path), msg='Unpacked
> repository missing')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> 'bitbake')), msg='bitbake submodule missing')
> + self.assertFalse(os.path.exists(os.path.join(repo_path,
> 'na')), msg='uninitialized submodule present') +
> + # Only when we're running the extended test with a
> submodule's submodule, can we check this.
> + if os.path.exists(os.path.join(repo_path,
> 'bitbake-gitsm-test1')):
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> 'bitbake-gitsm-test1', 'bitbake')), msg='submodule of submodule
> missing') +
> + @skipIfNoNetwork()
> + def test_git_submodule_dbus_broker(self):
> + # The following external repositories have show failures in
> fetch and unpack operations
> + # We want to avoid regressions!
> + url =
> "gitsm://github.com/bus1/dbus-broker;protocol=git;rev=fc874afa0992d0c75ec25acb43d344679f0ee7d2"
> + fetcher = bb.fetch.Fetch([url], self.d)
> + fetcher.download()
> + # Previous cwd has been deleted
> + os.chdir(os.path.dirname(self.unpackdir))
> + fetcher.unpack(self.unpackdir)
> +
> + repo_path = os.path.join(self.tempdir, 'unpacked', 'git')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> '.git/modules/subprojects/c-dvar/config')), msg='Missing submodule
> config "subprojects/c-dvar"')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> '.git/modules/subprojects/c-list/config')), msg='Missing submodule
> config "subprojects/c-list"')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> '.git/modules/subprojects/c-rbtree/config')), msg='Missing submodule
> config "subprojects/c-rbtree"')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> '.git/modules/subprojects/c-sundry/config')), msg='Missing submodule
> config "subprojects/c-sundry"')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> '.git/modules/subprojects/c-utf8/config')), msg='Missing submodule
> config "subprojects/c-utf8"') +
> + @skipIfNoNetwork()
> + def test_git_submodule_CLI11(self):
> + url =
> "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=bd4dc911847d0cde7a6b41dfa626a85aab213baf"
> + fetcher = bb.fetch.Fetch([url], self.d)
> + fetcher.download()
> + # Previous cwd has been deleted
> + os.chdir(os.path.dirname(self.unpackdir))
> + fetcher.unpack(self.unpackdir)
> +
> + repo_path = os.path.join(self.tempdir, 'unpacked', 'git')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> '.git/modules/extern/googletest/config')), msg='Missing submodule
> config "extern/googletest"')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> '.git/modules/extern/json/config')), msg='Missing submodule config
> "extern/json"')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> '.git/modules/extern/sanitizers/config')), msg='Missing submodule
> config "extern/sanitizers"') +
> + @skipIfNoNetwork()
> + def test_git_submodule_update_CLI11(self):
> + """ Prevent regression on update detection not finding
> missing submodule, or modules without needed commits """
> + url =
> "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=cf6a99fa69aaefe477cc52e3ef4a7d2d7fa40714"
> + fetcher = bb.fetch.Fetch([url], self.d)
> + fetcher.download()
> +
> + # CLI11 that pulls in a newer nlohmann-json
> + url =
> "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=49ac989a9527ee9bb496de9ded7b4872c2e0e5ca"
> + fetcher = bb.fetch.Fetch([url], self.d)
> + fetcher.download()
> + # Previous cwd has been deleted
> + os.chdir(os.path.dirname(self.unpackdir))
> + fetcher.unpack(self.unpackdir)
> +
> + repo_path = os.path.join(self.tempdir, 'unpacked', 'git')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> '.git/modules/extern/googletest/config')), msg='Missing submodule
> config "extern/googletest"')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> '.git/modules/extern/json/config')), msg='Missing submodule config
> "extern/json"')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> '.git/modules/extern/sanitizers/config')), msg='Missing submodule
> config "extern/sanitizers"') +
> + @skipIfNoNetwork()
> + def test_git_submodule_aktualizr(self):
> + url =
> "gitsm://github.com/advancedtelematic/aktualizr;branch=master;protocol=git;rev=d00d1a04cc2366d1a5f143b84b9f507f8bd32c44"
> + fetcher = bb.fetch.Fetch([url], self.d)
> fetcher.download()
> # Previous cwd has been deleted
> os.chdir(os.path.dirname(self.unpackdir))
> fetcher.unpack(self.unpackdir)
>
> + repo_path = os.path.join(self.tempdir, 'unpacked', 'git')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> '.git/modules/partial/extern/isotp-c/config')), msg='Missing
> submodule config "partial/extern/isotp-c/config"')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> '.git/modules/partial/extern/isotp-c/modules/deps/bitfield-c/config')),
> msg='Missing submodule config
> "partial/extern/isotp-c/modules/deps/bitfield-c/config"')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> 'partial/extern/isotp-c/deps/bitfield-c/.git')), msg="Submodule of
> submodule isotp-c did not unpack properly")
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> '.git/modules/tests/tuf-test-vectors/config')), msg='Missing
> submodule config "tests/tuf-test-vectors/config"')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> '.git/modules/third_party/googletest/config')), msg='Missing
> submodule config "third_party/googletest/config"')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> '.git/modules/third_party/HdrHistogram_c/config')), msg='Missing
> submodule config "third_party/HdrHistogram_c/config"') +
> + @skipIfNoNetwork()
> + def test_git_submodule_iotedge(self):
> + """ Prevent regression on deeply nested submodules not being
> checked out properly, even though they were fetched. """ +
> + # This repository also has submodules where the module
> (name), path and url do not align
> + url =
> "gitsm://github.com/azure/iotedge.git;protocol=git;rev=d76e0316c6f324345d77c48a83ce836d09392699"
> + fetcher = bb.fetch.Fetch([url], self.d)
> + fetcher.download()
> + # Previous cwd has been deleted
> + os.chdir(os.path.dirname(self.unpackdir))
> + fetcher.unpack(self.unpackdir)
> +
> + repo_path = os.path.join(self.tempdir, 'unpacked', 'git')
> +
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared/README.md')),
> msg='Missing submodule checkout')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared/testtools/ctest/README.md')),
> msg='Missing submodule checkout')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared/testtools/testrunner/readme.md')),
> msg='Missing submodule checkout')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared/testtools/umock-c/readme.md')),
> msg='Missing submodule checkout')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared/testtools/umock-c/deps/ctest/README.md')),
> msg='Missing submodule checkout')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared/testtools/umock-c/deps/testrunner/readme.md')),
> msg='Missing submodule checkout')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/utpm/README.md')), msg='Missing
> submodule checkout')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/utpm/deps/c-utility/README.md')),
> msg='Missing submodule checkout')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/utpm/deps/c-utility/testtools/ctest/README.md')),
> msg='Missing submodule checkout')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/utpm/deps/c-utility/testtools/testrunner/readme.md')),
> msg='Missing submodule checkout')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/utpm/deps/c-utility/testtools/umock-c/readme.md')),
> msg='Missing submodule checkout')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/utpm/deps/c-utility/testtools/umock-c/deps/ctest/README.md')),
> msg='Missing submodule checkout')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/utpm/deps/c-utility/testtools/umock-c/deps/testrunner/readme.md')),
> msg='Missing submodule checkout') + +class SVNTest(FetcherTest):
> + def skipIfNoSvn():
> + import shutil
> + if not shutil.which("svn"):
> + return unittest.skip("svn not installed, tests being
> skipped") +
> + if not shutil.which("svnadmin"):
> + return unittest.skip("svnadmin not installed, tests
> being skipped") +
> + return lambda f: f
> +
> + @skipIfNoSvn()
> + def setUp(self):
> + """ Create a local repository """
> +
> + super(SVNTest, self).setUp()
> +
> + # Create something we can fetch
> + src_dir = tempfile.mkdtemp(dir=self.tempdir,
> + prefix='svnfetch_srcdir_')
> + src_dir = os.path.abspath(src_dir)
> + bb.process.run("echo readme > README.md", cwd=src_dir)
> +
> + # Store it in a local SVN repository
> + repo_dir = tempfile.mkdtemp(dir=self.tempdir,
> + prefix='svnfetch_localrepo_')
> + repo_dir = os.path.abspath(repo_dir)
> + bb.process.run("svnadmin create project", cwd=repo_dir)
> +
> + self.repo_url = "file://%s/project" % repo_dir
> + bb.process.run("svn import --non-interactive -m 'Initial
> import' %s %s/trunk" % (src_dir, self.repo_url),
> + cwd=repo_dir)
> +
> + bb.process.run("svn co %s svnfetch_co" % self.repo_url,
> cwd=self.tempdir)
> + # Github will emulate SVN. Use this to check if we're
> downloding...
> + bb.process.run("svn propset svn:externals 'bitbake
> http://github.com/openembedded/bitbake' .",
> + cwd=os.path.join(self.tempdir, 'svnfetch_co',
> 'trunk'))
> + bb.process.run("svn commit --non-interactive -m 'Add
> external'",
> + cwd=os.path.join(self.tempdir, 'svnfetch_co',
> 'trunk')) +
> + self.src_dir = src_dir
> + self.repo_dir = repo_dir
> +
> + @skipIfNoSvn()
> + def tearDown(self):
> + os.chdir(self.origdir)
> + if os.environ.get("BB_TMPDIR_NOCLEAN") == "yes":
> + print("Not cleaning up %s. Please remove manually." %
> self.tempdir)
> + else:
> + bb.utils.prunedir(self.tempdir)
> +
> + @skipIfNoSvn()
> + @skipIfNoNetwork()
> + def test_noexternal_svn(self):
> + # Always match the rev count from setUp (currently rev 2)
> + url = "svn://%s;module=trunk;protocol=file;rev=2" %
> self.repo_url.replace('file://', '')
> + fetcher = bb.fetch.Fetch([url], self.d)
> + fetcher.download()
> + os.chdir(os.path.dirname(self.unpackdir))
> + fetcher.unpack(self.unpackdir)
> +
> + self.assertTrue(os.path.exists(os.path.join(self.unpackdir,
> 'trunk')), msg="Missing trunk")
> + self.assertTrue(os.path.exists(os.path.join(self.unpackdir,
> 'trunk', 'README.md')), msg="Missing contents")
> + self.assertFalse(os.path.exists(os.path.join(self.unpackdir,
> 'trunk/bitbake/trunk')), msg="External dir should NOT exist")
> + self.assertFalse(os.path.exists(os.path.join(self.unpackdir,
> 'trunk/bitbake/trunk', 'README')), msg="External README should NOT
> exit") +
> + @skipIfNoSvn()
> + def test_external_svn(self):
> + # Always match the rev count from setUp (currently rev 2)
> + url =
> "svn://%s;module=trunk;protocol=file;externals=allowed;rev=2" %
> self.repo_url.replace('file://', '')
> + fetcher = bb.fetch.Fetch([url], self.d)
> + fetcher.download()
> + os.chdir(os.path.dirname(self.unpackdir))
> + fetcher.unpack(self.unpackdir)
> +
> + self.assertTrue(os.path.exists(os.path.join(self.unpackdir,
> 'trunk')), msg="Missing trunk")
> + self.assertTrue(os.path.exists(os.path.join(self.unpackdir,
> 'trunk', 'README.md')), msg="Missing contents")
> + self.assertTrue(os.path.exists(os.path.join(self.unpackdir,
> 'trunk/bitbake/trunk')), msg="External dir should exist")
> + self.assertTrue(os.path.exists(os.path.join(self.unpackdir,
> 'trunk/bitbake/trunk', 'README')), msg="External README should exit")
> class TrustedNetworksTest(FetcherTest):
> def test_trusted_network(self):
> @@ -1024,8 +1200,8 @@ class FetchLatestVersionTest(FetcherTest):
> # packages with valid UPSTREAM_CHECK_URI and
> UPSTREAM_CHECK_REGEX ("cups",
> "http://www.cups.org/software/1.7.2/cups-1.7.2-source.tar.bz2",
> "https://github.com/apple/cups/releases",
> "(?P<name>cups\-)(?P<pver>((\d+[\.\-_]*)+))\-source\.tar\.gz") :
> "2.0.0",
> - ("db",
> "http://download.oracle.com/berkeley-db/db-5.3.21.tar.gz",
> "http://www.oracle.com/technetwork/products/berkeleydb/downloads/index-082944.html",
> "http://download.oracle.com/otn/berkeley-db/(?P<name>db-)(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz")
> - : "6.1.19",
> + ("db",
> "http://download.oracle.com/berkeley-db/db-5.3.21.tar.gz",
> "http://ftp.debian.org/debian/pool/main/d/db5.3/",
> "(?P<name>db5\.3_)(?P<pver>\d+(\.\d+)+).+\.orig\.tar\.xz")
> + : "5.3.10",
> }
>
> @skipIfNoNetwork()
> @@ -1280,7 +1456,7 @@ class GitShallowTest(FetcherTest):
>
> def fetch(self, uri=None):
> if uri is None:
> - uris = self.d.getVar('SRC_URI', True).split()
> + uris = self.d.getVar('SRC_URI').split()
> uri = uris[0]
> d = self.d
> else:
> @@ -1312,6 +1488,7 @@ class GitShallowTest(FetcherTest):
> # fetch and unpack, from the shallow tarball
> bb.utils.remove(self.gitdir, recurse=True)
> bb.utils.remove(ud.clonedir, recurse=True)
> + bb.utils.remove(ud.clonedir.replace('gitsource',
> 'gitsubmodule'), recurse=True)
> # confirm that the unpacked repo is used when no git clone
> or git # mirror tarball is available
> @@ -1338,7 +1515,7 @@ class GitShallowTest(FetcherTest):
>
> srcrev = self.git('rev-parse HEAD', cwd=self.srcdir).strip()
> self.d.setVar('SRCREV', srcrev)
> - uri = self.d.getVar('SRC_URI', True).split()[0]
> + uri = self.d.getVar('SRC_URI').split()[0]
> uri = '%s;nobranch=1;bare=1' % uri
>
> self.fetch_shallow(uri)
> @@ -1466,6 +1643,7 @@ class GitShallowTest(FetcherTest):
> self.git('config --add remote.origin.url "%s"' % smdir,
> cwd=smdir) self.git('config --add remote.origin.fetch
> "+refs/heads/*:refs/remotes/origin/*"', cwd=smdir)
> self.add_empty_file('asub', cwd=smdir)
> + self.add_empty_file('bsub', cwd=smdir)
>
> self.git('submodule init', cwd=self.srcdir)
> self.git('submodule add file://%s' % smdir, cwd=self.srcdir)
> @@ -1475,10 +1653,16 @@ class GitShallowTest(FetcherTest):
> uri = 'gitsm://%s;protocol=file;subdir=${S}' % self.srcdir
> fetcher, ud = self.fetch_shallow(uri)
>
> + # Verify the main repository is shallow
> self.assertRevCount(1)
> - assert './.git/modules/' in bb.process.run('tar -tzf %s' %
> os.path.join(self.dldir, ud.mirrortarballs[0]))[0] +
> + # Verify the gitsubmodule directory is present
> assert os.listdir(os.path.join(self.gitdir, 'gitsubmodule'))
>
> + # Verify the submodule is also shallow
> + self.assertRevCount(1, cwd=os.path.join(self.gitdir,
> 'gitsubmodule')) +
> +
> if any(os.path.exists(os.path.join(p, 'git-annex')) for p in
> os.environ.get('PATH').split(':')): def test_shallow_annex(self):
> self.add_empty_file('a')
> @@ -1510,7 +1694,7 @@ class GitShallowTest(FetcherTest):
> self.add_empty_file('f')
> self.assertRevCount(7, cwd=self.srcdir)
>
> - uri = self.d.getVar('SRC_URI', True).split()[0]
> + uri = self.d.getVar('SRC_URI').split()[0]
> uri = '%s;branch=master,a_branch;name=master,a_branch' % uri
>
> self.d.setVar('BB_GIT_SHALLOW_DEPTH', '0')
> @@ -1536,7 +1720,7 @@ class GitShallowTest(FetcherTest):
> self.add_empty_file('f')
> self.assertRevCount(7, cwd=self.srcdir)
>
> - uri = self.d.getVar('SRC_URI', True).split()[0]
> + uri = self.d.getVar('SRC_URI').split()[0]
> uri = '%s;branch=master,a_branch;name=master,a_branch' % uri
>
> self.d.setVar('BB_GIT_SHALLOW_DEPTH', '0')
> @@ -1724,3 +1908,83 @@ class GitShallowTest(FetcherTest):
>
> dir = os.listdir(self.unpackdir + "/git/")
> self.assertIn("fstests.doap", dir)
> +
> +class GitLfsTest(FetcherTest):
> + def setUp(self):
> + FetcherTest.setUp(self)
> +
> + self.gitdir = os.path.join(self.tempdir, 'git')
> + self.srcdir = os.path.join(self.tempdir, 'gitsource')
> +
> + self.d.setVar('WORKDIR', self.tempdir)
> + self.d.setVar('S', self.gitdir)
> + self.d.delVar('PREMIRRORS')
> + self.d.delVar('MIRRORS')
> +
> + self.d.setVar('SRCREV', '${AUTOREV}')
> + self.d.setVar('AUTOREV', '${@bb.fetch2.get_autorev(d)}')
> +
> + bb.utils.mkdirhier(self.srcdir)
> + self.git('init', cwd=self.srcdir)
> + with open(os.path.join(self.srcdir, '.gitattributes'), 'wt')
> as attrs:
> + attrs.write('*.mp3 filter=lfs -text')
> + self.git(['add', '.gitattributes'], cwd=self.srcdir)
> + self.git(['commit', '-m', "attributes", '.gitattributes'],
> cwd=self.srcdir) +
> + def git(self, cmd, cwd=None):
> + if isinstance(cmd, str):
> + cmd = 'git ' + cmd
> + else:
> + cmd = ['git'] + cmd
> + if cwd is None:
> + cwd = self.gitdir
> + return bb.process.run(cmd, cwd=cwd)[0]
> +
> + def fetch(self, uri=None):
> + uris = self.d.getVar('SRC_URI').split()
> + uri = uris[0]
> + d = self.d
> +
> + fetcher = bb.fetch2.Fetch(uris, d)
> + fetcher.download()
> + ud = fetcher.ud[uri]
> + return fetcher, ud
> +
> + def test_lfs_enabled(self):
> + import shutil
> +
> + uri = 'git://%s;protocol=file;subdir=${S};lfs=1' %
> self.srcdir
> + self.d.setVar('SRC_URI', uri)
> +
> + fetcher, ud = self.fetch()
> + self.assertIsNotNone(ud.method._find_git_lfs)
> +
> + # If git-lfs can be found, the unpack should be successful
> + ud.method._find_git_lfs = lambda d: True
> + shutil.rmtree(self.gitdir, ignore_errors=True)
> + fetcher.unpack(self.d.getVar('WORKDIR'))
> +
> + # If git-lfs cannot be found, the unpack should throw an
> error
> + with self.assertRaises(bb.fetch2.FetchError):
> + ud.method._find_git_lfs = lambda d: False
> + shutil.rmtree(self.gitdir, ignore_errors=True)
> + fetcher.unpack(self.d.getVar('WORKDIR'))
> +
> + def test_lfs_disabled(self):
> + import shutil
> +
> + uri = 'git://%s;protocol=file;subdir=${S};lfs=0' %
> self.srcdir
> + self.d.setVar('SRC_URI', uri)
> +
> + fetcher, ud = self.fetch()
> + self.assertIsNotNone(ud.method._find_git_lfs)
> +
> + # If git-lfs can be found, the unpack should be successful
> + ud.method._find_git_lfs = lambda d: True
> + shutil.rmtree(self.gitdir, ignore_errors=True)
> + fetcher.unpack(self.d.getVar('WORKDIR'))
> +
> + # If git-lfs cannot be found, the unpack should be successful
> + ud.method._find_git_lfs = lambda d: False
> + shutil.rmtree(self.gitdir, ignore_errors=True)
> + fetcher.unpack(self.d.getVar('WORKDIR'))
> diff --git a/bitbake/lib/bb/tests/parse.py
> b/bitbake/lib/bb/tests/parse.py index 1bc4740..9afd1b2 100644
> --- a/bitbake/lib/bb/tests/parse.py
> +++ b/bitbake/lib/bb/tests/parse.py
> @@ -1,22 +1,9 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # BitBake Test for lib/bb/parse/
> #
> # Copyright (C) 2015 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
>
> import unittest
> @@ -187,3 +174,21 @@ python () {
> self.assertEqual(d1.getVar("VAR_var"), "B")
> self.assertEqual(d2.getVar("VAR_var"), None)
>
> + addtask_deltask = """
> +addtask do_patch after do_foo after do_unpack before do_configure
> before do_compile +addtask do_fetch do_patch
> +
> +deltask do_fetch do_patch
> +"""
> + def test_parse_addtask_deltask(self):
> + import sys
> + f = self.parsehelper(self.addtask_deltask)
> + d = bb.parse.handle(f.name, self.d)['']
> +
> + stdout = sys.stdout.getvalue()
> + self.assertTrue("addtask contained multiple 'before'
> keywords" in stdout)
> + self.assertTrue("addtask contained multiple 'after'
> keywords" in stdout)
> + self.assertTrue('addtask ignored: " do_patch"' in stdout)
> + self.assertTrue('deltask ignored: " do_patch"' in stdout)
> + #self.assertTrue('dependent task do_foo for do_patch does
> not exist' in stdout) +
> diff --git a/bitbake/lib/bb/tests/persist_data.py
> b/bitbake/lib/bb/tests/persist_data.py new file mode 100644
> index 0000000..f641b5a
> --- /dev/null
> +++ b/bitbake/lib/bb/tests/persist_data.py
> @@ -0,0 +1,129 @@
> +#
> +# BitBake Test for lib/bb/persist_data/
> +#
> +# Copyright (C) 2018 Garmin Ltd.
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> +import unittest
> +import bb.data
> +import bb.persist_data
> +import tempfile
> +import threading
> +
> +class PersistDataTest(unittest.TestCase):
> + def _create_data(self):
> + return bb.persist_data.persist('TEST_PERSIST_DATA', self.d)
> +
> + def setUp(self):
> + self.d = bb.data.init()
> + self.tempdir = tempfile.TemporaryDirectory()
> + self.d['PERSISTENT_DIR'] = self.tempdir.name
> + self.data = self._create_data()
> + self.items = {
> + 'A1': '1',
> + 'B1': '2',
> + 'C2': '3'
> + }
> + self.stress_count = 10000
> + self.thread_count = 5
> +
> + for k,v in self.items.items():
> + self.data[k] = v
> +
> + def tearDown(self):
> + self.tempdir.cleanup()
> +
> + def _iter_helper(self, seen, iterator):
> + with iter(iterator):
> + for v in iterator:
> + self.assertTrue(v in seen)
> + seen.remove(v)
> + self.assertEqual(len(seen), 0, '%s not seen' % seen)
> +
> + def test_get(self):
> + for k, v in self.items.items():
> + self.assertEqual(self.data[k], v)
> +
> + self.assertIsNone(self.data.get('D'))
> + with self.assertRaises(KeyError):
> + self.data['D']
> +
> + def test_set(self):
> + for k, v in self.items.items():
> + self.data[k] += '-foo'
> +
> + for k, v in self.items.items():
> + self.assertEqual(self.data[k], v + '-foo')
> +
> + def test_delete(self):
> + self.data['D'] = '4'
> + self.assertEqual(self.data['D'], '4')
> + del self.data['D']
> + self.assertIsNone(self.data.get('D'))
> + with self.assertRaises(KeyError):
> + self.data['D']
> +
> + def test_contains(self):
> + for k in self.items:
> + self.assertTrue(k in self.data)
> + self.assertTrue(self.data.has_key(k))
> + self.assertFalse('NotFound' in self.data)
> + self.assertFalse(self.data.has_key('NotFound'))
> +
> + def test_len(self):
> + self.assertEqual(len(self.data), len(self.items))
> +
> + def test_iter(self):
> + self._iter_helper(set(self.items.keys()), self.data)
> +
> + def test_itervalues(self):
> + self._iter_helper(set(self.items.values()),
> self.data.itervalues()) +
> + def test_iteritems(self):
> + self._iter_helper(set(self.items.items()),
> self.data.iteritems()) +
> + def test_get_by_pattern(self):
> + self._iter_helper({'1', '2'}, self.data.get_by_pattern('_1'))
> +
> + def _stress_read(self, data):
> + for i in range(self.stress_count):
> + for k in self.items:
> + data[k]
> +
> + def _stress_write(self, data):
> + for i in range(self.stress_count):
> + for k, v in self.items.items():
> + data[k] = v + str(i)
> +
> + def _validate_stress(self):
> + for k, v in self.items.items():
> + self.assertEqual(self.data[k], v + str(self.stress_count
> - 1)) +
> + def test_stress(self):
> + self._stress_read(self.data)
> + self._stress_write(self.data)
> + self._validate_stress()
> +
> + def test_stress_threads(self):
> + def read_thread():
> + data = self._create_data()
> + self._stress_read(data)
> +
> + def write_thread():
> + data = self._create_data()
> + self._stress_write(data)
> +
> + threads = []
> + for i in range(self.thread_count):
> + threads.append(threading.Thread(target=read_thread))
> + threads.append(threading.Thread(target=write_thread))
> +
> + for t in threads:
> + t.start()
> + self._stress_read(self.data)
> + for t in threads:
> + t.join()
> + self._validate_stress()
> +
> diff --git a/bitbake/lib/bb/tests/runqueue-tests/classes/base.bbclass
> b/bitbake/lib/bb/tests/runqueue-tests/classes/base.bbclass new file
> mode 100644 index 0000000..b57650d
> --- /dev/null
> +++ b/bitbake/lib/bb/tests/runqueue-tests/classes/base.bbclass
> @@ -0,0 +1,262 @@
> +SLOWTASKS ??= ""
> +SSTATEVALID ??= ""
> +
> +def stamptask(d):
> + import time
> +
> + thistask = d.expand("${PN}:${BB_CURRENTTASK}")
> + stampname = d.expand("${TOPDIR}/%s.run" % thistask)
> + with open(stampname, "a+") as f:
> + f.write(d.getVar("BB_UNIHASH") + "\n")
> +
> + if d.getVar("BB_CURRENT_MC") != "default":
> + thistask =
> d.expand("${BB_CURRENT_MC}:${PN}:${BB_CURRENTTASK}")
> + if thistask in d.getVar("SLOWTASKS").split():
> + bb.note("Slowing task %s" % thistask)
> + time.sleep(0.5)
> + if d.getVar("BB_HASHSERVE"):
> + task = d.getVar("BB_CURRENTTASK")
> + if task in ['package', 'package_qa', 'packagedata',
> 'package_write_ipk', 'package_write_rpm', 'populate_lic',
> 'populate_sysroot']:
> + bb.parse.siggen.report_unihash(os.getcwd(),
> d.getVar("BB_CURRENTTASK"), d) +
> + with open(d.expand("${TOPDIR}/task.log"), "a+") as f:
> + f.write(thistask + "\n")
> +
> +
> +def sstate_output_hash(path, sigfile, task, d):
> + import hashlib
> + h = hashlib.sha256()
> + h.update(d.expand("${PN}:${BB_CURRENTTASK}").encode('utf-8'))
> + return h.hexdigest()
> +
> +python do_fetch() {
> + # fetch
> + stamptask(d)
> +}
> +python do_unpack() {
> + # unpack
> + stamptask(d)
> +}
> +python do_patch() {
> + # patch
> + stamptask(d)
> +}
> +python do_populate_lic() {
> + # populate_lic
> + stamptask(d)
> +}
> +python do_prepare_recipe_sysroot() {
> + # prepare_recipe_sysroot
> + stamptask(d)
> +}
> +python do_configure() {
> + # configure
> + stamptask(d)
> +}
> +python do_compile() {
> + # compile
> + stamptask(d)
> +}
> +python do_install() {
> + # install
> + stamptask(d)
> +}
> +python do_populate_sysroot() {
> + # populate_sysroot
> + stamptask(d)
> +}
> +python do_package() {
> + # package
> + stamptask(d)
> +}
> +python do_package_write_ipk() {
> + # package_write_ipk
> + stamptask(d)
> +}
> +python do_package_write_rpm() {
> + # package_write_rpm
> + stamptask(d)
> +}
> +python do_packagedata() {
> + # packagedata
> + stamptask(d)
> +}
> +python do_package_qa() {
> + # package_qa
> + stamptask(d)
> +}
> +python do_build() {
> + # build
> + stamptask(d)
> +}
> +do_prepare_recipe_sysroot[deptask] = "do_populate_sysroot"
> +do_package[deptask] += "do_packagedata"
> +do_build[recrdeptask] += "do_deploy"
> +do_build[recrdeptask] += "do_package_write_ipk"
> +do_build[recrdeptask] += "do_package_write_rpm"
> +do_package_qa[rdeptask] = "do_packagedata"
> +do_populate_lic_deploy[recrdeptask] += "do_populate_lic do_deploy"
> +
> +DEBIANRDEP = "do_packagedata"
> +oo_package_write_ipk[rdeptask] = "${DEBIANRDEP}"
> +do_package_write_rpm[rdeptask] = "${DEBIANRDEP}"
> +
> +addtask fetch
> +addtask unpack after do_fetch
> +addtask patch after do_unpack
> +addtask prepare_recipe_sysroot after do_patch
> +addtask configure after do_prepare_recipe_sysroot
> +addtask compile after do_configure
> +addtask install after do_compile
> +addtask populate_sysroot after do_install
> +addtask package after do_install
> +addtask package_write_ipk after do_packagedata do_package
> +addtask package_write_rpm after do_packagedata do_package
> +addtask packagedata after do_package
> +addtask package_qa after do_package
> +addtask build after do_package_qa do_package_write_rpm
> do_package_write_ipk do_populate_sysroot +
> +python do_package_setscene() {
> + stamptask(d)
> +}
> +python do_package_qa_setscene() {
> + stamptask(d)
> +}
> +python do_package_write_ipk_setscene() {
> + stamptask(d)
> +}
> +python do_package_write_rpm_setscene() {
> + stamptask(d)
> +}
> +python do_packagedata_setscene() {
> + stamptask(d)
> +}
> +python do_populate_lic_setscene() {
> + stamptask(d)
> +}
> +python do_populate_sysroot_setscene() {
> + stamptask(d)
> +}
> +
> +addtask package_setscene
> +addtask package_qa_setscene
> +addtask package_write_ipk_setscene
> +addtask package_write_rpm_setscene
> +addtask packagedata_setscene
> +addtask populate_lic_setscene
> +addtask populate_sysroot_setscene
> +
> +BB_SETSCENE_DEPVALID = "setscene_depvalid"
> +
> +def setscene_depvalid(task, taskdependees, notneeded, d, log=None):
> + # taskdependees is a dict of tasks which depend on task, each
> being a 3 item list of [PN, TASKNAME, FILENAME]
> + # task is included in taskdependees too
> + # Return - False - We need this dependency
> + # - True - We can skip this dependency
> + import re
> +
> + def logit(msg, log):
> + if log is not None:
> + log.append(msg)
> + else:
> + bb.debug(2, msg)
> +
> + logit("Considering setscene task: %s" %
> (str(taskdependees[task])), log) +
> + def isNativeCross(x):
> + return x.endswith("-native") or "-cross-" in x or
> "-crosssdk" in x or x.endswith("-cross") +
> + # We only need to trigger populate_lic through direct
> dependencies
> + if taskdependees[task][1] == "do_populate_lic":
> + return True
> +
> + # We only need to trigger packagedata through direct dependencies
> + # but need to preserve packagedata on packagedata links
> + if taskdependees[task][1] == "do_packagedata":
> + for dep in taskdependees:
> + if taskdependees[dep][1] == "do_packagedata":
> + return False
> + return True
> +
> + for dep in taskdependees:
> + logit(" considering dependency: %s" %
> (str(taskdependees[dep])), log)
> + if task == dep:
> + continue
> + if dep in notneeded:
> + continue
> + # do_package_write_* and do_package doesn't need do_package
> + if taskdependees[task][1] == "do_package" and
> taskdependees[dep][1] in ['do_package', 'do_package_write_ipk',
> 'do_package_write_rpm', 'do_packagedata', 'do_package_qa']:
> + continue
> + # do_package_write_* need do_populate_sysroot as they're
> mainly postinstall dependencies
> + if taskdependees[task][1] == "do_populate_sysroot" and
> taskdependees[dep][1] in ['do_package_write_ipk',
> 'do_package_write_rpm']:
> + return False
> + # do_package/packagedata/package_qa don't need
> do_populate_sysroot
> + if taskdependees[task][1] == "do_populate_sysroot" and
> taskdependees[dep][1] in ['do_package', 'do_packagedata',
> 'do_package_qa']:
> + continue
> + # Native/Cross packages don't exist and are noexec anyway
> + if isNativeCross(taskdependees[dep][0]) and
> taskdependees[dep][1] in ['do_package_write_ipk',
> 'do_package_write_rpm', 'do_packagedata', 'do_package',
> 'do_package_qa']:
> + continue
> +
> + # This is due to the [depends] in useradd.bbclass
> complicating matters
> + # The logic *is* reversed here due to the way hard setscene
> dependencies are injected
> + if (taskdependees[task][1] == 'do_package' or
> taskdependees[task][1] == 'do_populate_sysroot') and
> taskdependees[dep][0].endswith(('shadow-native', 'shadow-sysroot',
> 'base-passwd', 'pseudo-native')) and taskdependees[dep][1] ==
> 'do_populate_sysroot':
> + continue
> +
> + # Consider sysroot depending on sysroot tasks
> + if taskdependees[task][1] == 'do_populate_sysroot' and
> taskdependees[dep][1] == 'do_populate_sysroot':
> + # Native/Cross populate_sysroot need their dependencies
> + if isNativeCross(taskdependees[task][0]) and
> isNativeCross(taskdependees[dep][0]):
> + return False
> + # Target populate_sysroot depended on by cross tools
> need to be installed
> + if isNativeCross(taskdependees[dep][0]):
> + return False
> + # Native/cross tools depended upon by target sysroot are
> not needed
> + # Add an exception for shadow-native as required by
> useradd.bbclass
> + if isNativeCross(taskdependees[task][0]) and
> taskdependees[task][0] != 'shadow-native':
> + continue
> + # Target populate_sysroot need their dependencies
> + return False
> +
> +
> + if taskdependees[dep][1] == "do_populate_lic":
> + continue
> +
> + # Safe fallthrough default
> + logit(" Default setscene dependency fall through due to
> dependency: %s" % (str(taskdependees[dep])), log)
> + return False
> + return True
> +
> +BB_HASHCHECK_FUNCTION = "sstate_checkhashes"
> +
> +def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0,
> **kwargs): +
> + found = set()
> + missed = set()
> +
> + valid = d.getVar("SSTATEVALID").split()
> +
> + for tid in sorted(sq_data['hash']):
> + n =
> os.path.basename(bb.runqueue.fn_from_tid(tid)).split(".")[0] + ":do_"
> + bb.runqueue.taskname_from_tid(tid)[3:]
> + print(n)
> + stampfile = d.expand("${TOPDIR}/%s.run" % n.replace("do_",
> ""))
> + if n in valid:
> + bb.note("SState: Found valid sstate for %s" % n)
> + found.add(tid)
> + elif n + ":" + sq_data['hash'][tid] in valid:
> + bb.note("SState: Found valid sstate for %s" % n)
> + found.add(tid)
> + elif os.path.exists(stampfile):
> + with open(stampfile, "r") as f:
> + hash = f.readline().strip()
> + if hash == sq_data['hash'][tid]:
> + bb.note("SState: Found valid sstate for %s (already
> run)" % n)
> + found.add(tid)
> + else:
> + bb.note("SState: sstate hash didn't match previous
> run for %s (%s vs %s)" % (n, sq_data['hash'][tid], hash))
> + missed.add(tid)
> + else:
> + missed.add(tid)
> + bb.note("SState: Found no valid sstate for %s (%s)" %
> (n, sq_data['hash'][tid])) +
> + return found
> +
> diff --git
> a/bitbake/lib/bb/tests/runqueue-tests/classes/image.bbclass
> b/bitbake/lib/bb/tests/runqueue-tests/classes/image.bbclass new file
> mode 100644 index 0000000..da9ff11 --- /dev/null
> +++ b/bitbake/lib/bb/tests/runqueue-tests/classes/image.bbclass
> @@ -0,0 +1,5 @@
> +do_rootfs[recrdeptask] += "do_package_write_deb do_package_qa"
> +do_rootfs[recrdeptask] += "do_package_write_ipk do_package_qa"
> +do_rootfs[recrdeptask] += "do_package_write_rpm do_package_qa
> +do_rootfs[recrdeptask] += "do_packagedata"
> +do_rootfs[recrdeptask] += "do_populate_lic"
> diff --git
> a/bitbake/lib/bb/tests/runqueue-tests/classes/native.bbclass
> b/bitbake/lib/bb/tests/runqueue-tests/classes/native.bbclass new file
> mode 100644 index 0000000..7eaaee5 --- /dev/null
> +++ b/bitbake/lib/bb/tests/runqueue-tests/classes/native.bbclass
> @@ -0,0 +1,2 @@
> +RECIPERDEPTASK = "do_populate_sysroot"
> +do_populate_sysroot[rdeptask] = "${RECIPERDEPTASK}"
> diff --git a/bitbake/lib/bb/tests/runqueue-tests/conf/bitbake.conf
> b/bitbake/lib/bb/tests/runqueue-tests/conf/bitbake.conf new file mode
> 100644 index 0000000..5e451fc
> --- /dev/null
> +++ b/bitbake/lib/bb/tests/runqueue-tests/conf/bitbake.conf
> @@ -0,0 +1,16 @@
> +CACHE = "${TOPDIR}/cache"
> +THISDIR = "${@os.path.dirname(d.getVar('FILE'))}"
> +COREBASE :=
> "${@os.path.normpath(os.path.dirname(d.getVar('FILE')+'/../../'))}"
> +BBFILES = "${COREBASE}/recipes/*.bb" +PROVIDES = "${PN}"
> +PN = "${@bb.parse.vars_from_file(d.getVar('FILE', False),d)[0]}"
> +PF = "${BB_CURRENT_MC}:${PN}"
> +export PATH
> +TMPDIR ??= "${TOPDIR}"
> +STAMP = "${TMPDIR}/stamps/${PN}"
> +T = "${TMPDIR}/workdir/${PN}/temp"
> +BB_NUMBER_THREADS = "4"
> +
> +BB_HASHBASE_WHITELIST = "BB_CURRENT_MC BB_HASHSERVE TMPDIR TOPDIR
> SLOWTASKS SSTATEVALID FILE" +
> +include conf/multiconfig/${BB_CURRENT_MC}.conf
> diff --git
> a/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc1.conf
> b/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc1.conf new
> file mode 100644 index 0000000..ecf23e1 --- /dev/null
> +++ b/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc1.conf
> @@ -0,0 +1 @@
> +TMPDIR = "${TOPDIR}/mc1/"
> diff --git
> a/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc2.conf
> b/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc2.conf new
> file mode 100644 index 0000000..eef338e --- /dev/null
> +++ b/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc2.conf
> @@ -0,0 +1 @@
> +TMPDIR = "${TOPDIR}/mc2/"
> diff --git a/bitbake/lib/bb/tests/runqueue-tests/recipes/a1.bb
> b/bitbake/lib/bb/tests/runqueue-tests/recipes/a1.bb new file mode
> 100644 index 0000000..e69de29
> diff --git a/bitbake/lib/bb/tests/runqueue-tests/recipes/b1.bb
> b/bitbake/lib/bb/tests/runqueue-tests/recipes/b1.bb new file mode
> 100644 index 0000000..c0b288e
> --- /dev/null
> +++ b/bitbake/lib/bb/tests/runqueue-tests/recipes/b1.bb
> @@ -0,0 +1 @@
> +DEPENDS = "a1"
> \ No newline at end of file
> diff --git a/bitbake/lib/bb/tests/runqueue-tests/recipes/c1.bb
> b/bitbake/lib/bb/tests/runqueue-tests/recipes/c1.bb new file mode
> 100644 index 0000000..e69de29
> diff --git a/bitbake/lib/bb/tests/runqueue-tests/recipes/d1.bb
> b/bitbake/lib/bb/tests/runqueue-tests/recipes/d1.bb new file mode
> 100644 index 0000000..5ba1975
> --- /dev/null
> +++ b/bitbake/lib/bb/tests/runqueue-tests/recipes/d1.bb
> @@ -0,0 +1,3 @@
> +DEPENDS = "a1"
> +
> +do_package_setscene[depends] = "a1:do_populate_sysroot_setscene"
> diff --git a/bitbake/lib/bb/tests/runqueue-tests/recipes/e1.bb
> b/bitbake/lib/bb/tests/runqueue-tests/recipes/e1.bb new file mode
> 100644 index 0000000..1588bc8
> --- /dev/null
> +++ b/bitbake/lib/bb/tests/runqueue-tests/recipes/e1.bb
> @@ -0,0 +1 @@
> +DEPENDS = "b1"
> \ No newline at end of file
> diff --git a/bitbake/lib/bb/tests/runqueue.py
> b/bitbake/lib/bb/tests/runqueue.py new file mode 100644
> index 0000000..5e64391
> --- /dev/null
> +++ b/bitbake/lib/bb/tests/runqueue.py
> @@ -0,0 +1,323 @@
> +#
> +# BitBake Tests for runqueue task processing
> +#
> +# Copyright (C) 2019 Richard Purdie
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> +import unittest
> +import bb
> +import os
> +import tempfile
> +import subprocess
> +import sys
> +import time
> +
> +#
> +# TODO:
> +# Add tests on task ordering (X happens before Y after Z)
> +#
> +
> +class RunQueueTests(unittest.TestCase):
> +
> + alltasks = ['package', 'fetch', 'unpack', 'patch',
> 'prepare_recipe_sysroot', 'configure',
> + 'compile', 'install', 'packagedata', 'package_qa',
> 'package_write_rpm', 'package_write_ipk',
> + 'populate_sysroot', 'build']
> + a1_sstatevalid = "a1:do_package a1:do_package_qa
> a1:do_packagedata a1:do_package_write_ipk a1:do_package_write_rpm
> a1:do_populate_lic a1:do_populate_sysroot"
> + b1_sstatevalid = "b1:do_package b1:do_package_qa
> b1:do_packagedata b1:do_package_write_ipk b1:do_package_write_rpm
> b1:do_populate_lic b1:do_populate_sysroot" +
> + def run_bitbakecmd(self, cmd, builddir, sstatevalid="",
> slowtasks="", extraenv=None, cleanup=False):
> + env = os.environ.copy()
> + env["BBPATH"] =
> os.path.realpath(os.path.join(os.path.dirname(__file__),
> "runqueue-tests"))
> + env["BB_ENV_EXTRAWHITE"] = "SSTATEVALID SLOWTASKS"
> + env["SSTATEVALID"] = sstatevalid
> + env["SLOWTASKS"] = slowtasks
> + if extraenv:
> + for k in extraenv:
> + env[k] = extraenv[k]
> + env["BB_ENV_EXTRAWHITE"] = env["BB_ENV_EXTRAWHITE"]
> + " " + k
> + try:
> + output = subprocess.check_output(cmd, env=env,
> stderr=subprocess.STDOUT,universal_newlines=True, cwd=builddir)
> + print(output)
> + except subprocess.CalledProcessError as e:
> + self.fail("Command %s failed with %s" % (cmd, e.output))
> + tasks = []
> + tasklog = builddir + "/task.log"
> + if os.path.exists(tasklog):
> + with open(tasklog, "r") as f:
> + tasks = [line.rstrip() for line in f]
> + if cleanup:
> + os.remove(tasklog)
> + return tasks
> +
> + def test_no_setscenevalid(self):
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + cmd = ["bitbake", "a1"]
> + sstatevalid = ""
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
> + expected = ['a1:' + x for x in self.alltasks]
> + self.assertEqual(set(tasks), set(expected))
> +
> + def test_single_setscenevalid(self):
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + cmd = ["bitbake", "a1"]
> + sstatevalid = "a1:do_package"
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
> + expected = ['a1:package_setscene', 'a1:fetch',
> 'a1:unpack', 'a1:patch', 'a1:prepare_recipe_sysroot', 'a1:configure',
> + 'a1:compile', 'a1:install',
> 'a1:packagedata', 'a1:package_qa', 'a1:package_write_rpm',
> 'a1:package_write_ipk',
> + 'a1:populate_sysroot', 'a1:build']
> + self.assertEqual(set(tasks), set(expected))
> +
> + def test_intermediate_setscenevalid(self):
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + cmd = ["bitbake", "a1"]
> + sstatevalid = "a1:do_package a1:do_populate_sysroot"
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
> + expected = ['a1:package_setscene', 'a1:packagedata',
> 'a1:package_qa', 'a1:package_write_rpm', 'a1:package_write_ipk',
> + 'a1:populate_sysroot_setscene', 'a1:build']
> + self.assertEqual(set(tasks), set(expected))
> +
> + def test_intermediate_notcovered(self):
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + cmd = ["bitbake", "a1"]
> + sstatevalid = "a1:do_package_qa a1:do_packagedata
> a1:do_package_write_ipk a1:do_package_write_rpm a1:do_populate_lic
> a1:do_populate_sysroot"
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
> + expected = ['a1:package_write_ipk_setscene',
> 'a1:package_write_rpm_setscene', 'a1:packagedata_setscene',
> + 'a1:package_qa_setscene', 'a1:build',
> 'a1:populate_sysroot_setscene']
> + self.assertEqual(set(tasks), set(expected))
> +
> + def test_all_setscenevalid(self):
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + cmd = ["bitbake", "a1"]
> + sstatevalid = self.a1_sstatevalid
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
> + expected = ['a1:package_write_ipk_setscene',
> 'a1:package_write_rpm_setscene', 'a1:packagedata_setscene',
> + 'a1:package_qa_setscene', 'a1:build',
> 'a1:populate_sysroot_setscene']
> + self.assertEqual(set(tasks), set(expected))
> +
> + def test_no_settasks(self):
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + cmd = ["bitbake", "a1", "-c", "patch"]
> + sstatevalid = self.a1_sstatevalid
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
> + expected = ['a1:fetch', 'a1:unpack', 'a1:patch']
> + self.assertEqual(set(tasks), set(expected))
> +
> + def test_mix_covered_notcovered(self):
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + cmd = ["bitbake", "a1:do_patch",
> "a1:do_populate_sysroot"]
> + sstatevalid = self.a1_sstatevalid
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
> + expected = ['a1:fetch', 'a1:unpack', 'a1:patch',
> 'a1:populate_sysroot_setscene']
> + self.assertEqual(set(tasks), set(expected))
> +
> +
> + # Test targets with intermediate setscene tasks alongside a
> target with no intermediate setscene tasks
> + def test_mixed_direct_tasks_setscene_tasks(self):
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + cmd = ["bitbake", "c1:do_patch", "a1"]
> + sstatevalid = self.a1_sstatevalid
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
> + expected = ['c1:fetch', 'c1:unpack', 'c1:patch',
> 'a1:package_write_ipk_setscene', 'a1:package_write_rpm_setscene',
> 'a1:packagedata_setscene',
> + 'a1:package_qa_setscene', 'a1:build',
> 'a1:populate_sysroot_setscene']
> + self.assertEqual(set(tasks), set(expected))
> +
> + # This test slows down the execution of do_package_setscene
> until after other real tasks have
> + # started running which tests for a bug where tasks were being
> lost from the buildable list of real
> + # tasks if they weren't in tasks_covered or tasks_notcovered
> + def test_slow_setscene(self):
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + cmd = ["bitbake", "a1"]
> + sstatevalid = "a1:do_package"
> + slowtasks = "a1:package_setscene"
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
> slowtasks)
> + expected = ['a1:package_setscene', 'a1:fetch',
> 'a1:unpack', 'a1:patch', 'a1:prepare_recipe_sysroot', 'a1:configure',
> + 'a1:compile', 'a1:install',
> 'a1:packagedata', 'a1:package_qa', 'a1:package_write_rpm',
> 'a1:package_write_ipk',
> + 'a1:populate_sysroot', 'a1:build']
> + self.assertEqual(set(tasks), set(expected))
> +
> + def test_setscenewhitelist(self):
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + cmd = ["bitbake", "a1"]
> + extraenv = {
> + "BB_SETSCENE_ENFORCE" : "1",
> + "BB_SETSCENE_ENFORCE_WHITELIST" :
> "a1:do_package_write_rpm a1:do_build"
> + }
> + sstatevalid = "a1:do_package a1:do_package_qa
> a1:do_packagedata a1:do_package_write_ipk a1:do_populate_lic
> a1:do_populate_sysroot"
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
> extraenv=extraenv)
> + expected = ['a1:packagedata_setscene',
> 'a1:package_qa_setscene', 'a1:package_write_ipk_setscene',
> + 'a1:populate_sysroot_setscene',
> 'a1:package_setscene']
> + self.assertEqual(set(tasks), set(expected))
> +
> + # Tests for problems with dependencies between setscene tasks
> + def test_no_setscenevalid_harddeps(self):
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + cmd = ["bitbake", "d1"]
> + sstatevalid = ""
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
> + expected = ['a1:package', 'a1:fetch', 'a1:unpack',
> 'a1:patch', 'a1:prepare_recipe_sysroot', 'a1:configure',
> + 'a1:compile', 'a1:install',
> 'a1:packagedata', 'a1:package_write_rpm', 'a1:package_write_ipk',
> + 'a1:populate_sysroot', 'd1:package',
> 'd1:fetch', 'd1:unpack', 'd1:patch', 'd1:prepare_recipe_sysroot',
> 'd1:configure',
> + 'd1:compile', 'd1:install',
> 'd1:packagedata', 'd1:package_qa', 'd1:package_write_rpm',
> 'd1:package_write_ipk',
> + 'd1:populate_sysroot', 'd1:build']
> + self.assertEqual(set(tasks), set(expected))
> +
> + def test_no_setscenevalid_withdeps(self):
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + cmd = ["bitbake", "b1"]
> + sstatevalid = ""
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
> + expected = ['a1:' + x for x in self.alltasks] + ['b1:' +
> x for x in self.alltasks]
> + expected.remove('a1:build')
> + expected.remove('a1:package_qa')
> + self.assertEqual(set(tasks), set(expected))
> +
> + def test_single_a1_setscenevalid_withdeps(self):
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + cmd = ["bitbake", "b1"]
> + sstatevalid = "a1:do_package"
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
> + expected = ['a1:package_setscene', 'a1:fetch',
> 'a1:unpack', 'a1:patch', 'a1:prepare_recipe_sysroot', 'a1:configure',
> + 'a1:compile', 'a1:install',
> 'a1:packagedata', 'a1:package_write_rpm', 'a1:package_write_ipk',
> + 'a1:populate_sysroot'] + ['b1:' + x for x in
> self.alltasks]
> + self.assertEqual(set(tasks), set(expected))
> +
> + def test_single_b1_setscenevalid_withdeps(self):
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + cmd = ["bitbake", "b1"]
> + sstatevalid = "b1:do_package"
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
> + expected = ['a1:package', 'a1:fetch', 'a1:unpack',
> 'a1:patch', 'a1:prepare_recipe_sysroot', 'a1:configure',
> + 'a1:compile', 'a1:install',
> 'a1:packagedata', 'a1:package_write_rpm', 'a1:package_write_ipk',
> + 'a1:populate_sysroot',
> 'b1:package_setscene'] + ['b1:' + x for x in self.alltasks]
> + expected.remove('b1:package')
> + self.assertEqual(set(tasks), set(expected))
> +
> + def test_intermediate_setscenevalid_withdeps(self):
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + cmd = ["bitbake", "b1"]
> + sstatevalid = "a1:do_package a1:do_populate_sysroot
> b1:do_package"
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
> + expected = ['a1:package_setscene', 'a1:packagedata',
> 'a1:package_write_rpm', 'a1:package_write_ipk',
> + 'a1:populate_sysroot_setscene',
> 'b1:package_setscene'] + ['b1:' + x for x in self.alltasks]
> + expected.remove('b1:package')
> + self.assertEqual(set(tasks), set(expected))
> +
> + def test_all_setscenevalid_withdeps(self):
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + cmd = ["bitbake", "b1"]
> + sstatevalid = self.a1_sstatevalid + " " +
> self.b1_sstatevalid
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
> + expected = ['a1:package_write_ipk_setscene',
> 'a1:package_write_rpm_setscene', 'a1:packagedata_setscene',
> + 'b1:build', 'a1:populate_sysroot_setscene',
> 'b1:package_write_ipk_setscene', 'b1:package_write_rpm_setscene',
> + 'b1:packagedata_setscene',
> 'b1:package_qa_setscene', 'b1:populate_sysroot_setscene']
> + self.assertEqual(set(tasks), set(expected))
> +
> + def test_multiconfig_setscene_optimise(self):
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + extraenv = {
> + "BBMULTICONFIG" : "mc1 mc2",
> + "BB_SIGNATURE_HANDLER" : "basic"
> + }
> + cmd = ["bitbake", "b1", "mc:mc1:b1", "mc:mc2:b1"]
> + setscenetasks = ['package_write_ipk_setscene',
> 'package_write_rpm_setscene', 'packagedata_setscene',
> + 'populate_sysroot_setscene',
> 'package_qa_setscene']
> + sstatevalid = ""
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
> extraenv=extraenv)
> + expected = ['a1:' + x for x in self.alltasks] + ['b1:' +
> x for x in self.alltasks] + \
> + ['mc1:b1:' + x for x in setscenetasks] +
> ['mc1:a1:' + x for x in setscenetasks] + \
> + ['mc2:b1:' + x for x in setscenetasks] +
> ['mc2:a1:' + x for x in setscenetasks] + \
> + ['mc1:b1:build', 'mc2:b1:build']
> + for x in ['mc1:a1:package_qa_setscene',
> 'mc2:a1:package_qa_setscene', 'a1:build', 'a1:package_qa']:
> + expected.remove(x)
> + self.assertEqual(set(tasks), set(expected))
> +
> +
> + @unittest.skipIf(sys.version_info < (3, 5, 0), 'Python 3.5 or
> later required')
> + def test_hashserv_single(self):
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + extraenv = {
> + "BB_HASHSERVE" : "auto",
> + "BB_SIGNATURE_HANDLER" : "TestEquivHash"
> + }
> + cmd = ["bitbake", "a1", "b1"]
> + setscenetasks = ['package_write_ipk_setscene',
> 'package_write_rpm_setscene', 'packagedata_setscene',
> + 'populate_sysroot_setscene',
> 'package_qa_setscene']
> + sstatevalid = ""
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
> extraenv=extraenv, cleanup=True)
> + expected = ['a1:' + x for x in self.alltasks] + ['b1:' +
> x for x in self.alltasks]
> + self.assertEqual(set(tasks), set(expected))
> + cmd = ["bitbake", "a1", "-c", "install", "-f"]
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
> extraenv=extraenv, cleanup=True)
> + expected = ['a1:install']
> + self.assertEqual(set(tasks), set(expected))
> + cmd = ["bitbake", "a1", "b1"]
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
> extraenv=extraenv, cleanup=True)
> + expected = ['a1:populate_sysroot', 'a1:package',
> 'a1:package_write_rpm_setscene', 'a1:packagedata_setscene',
> + 'a1:package_write_ipk_setscene',
> 'a1:package_qa_setscene']
> + self.assertEqual(set(tasks), set(expected))
> +
> + self.shutdown(tempdir)
> +
> + @unittest.skipIf(sys.version_info < (3, 5, 0), 'Python 3.5 or
> later required')
> + def test_hashserv_double(self):
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + extraenv = {
> + "BB_HASHSERVE" : "auto",
> + "BB_SIGNATURE_HANDLER" : "TestEquivHash"
> + }
> + cmd = ["bitbake", "a1", "b1", "e1"]
> + setscenetasks = ['package_write_ipk_setscene',
> 'package_write_rpm_setscene', 'packagedata_setscene',
> + 'populate_sysroot_setscene',
> 'package_qa_setscene']
> + sstatevalid = ""
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
> extraenv=extraenv, cleanup=True)
> + expected = ['a1:' + x for x in self.alltasks] + ['b1:' +
> x for x in self.alltasks] + ['e1:' + x for x in self.alltasks]
> + self.assertEqual(set(tasks), set(expected))
> + cmd = ["bitbake", "a1", "b1", "-c", "install", "-fn"]
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
> extraenv=extraenv, cleanup=True)
> + cmd = ["bitbake", "e1"]
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
> extraenv=extraenv, cleanup=True)
> + expected = ['a1:package', 'a1:install', 'b1:package',
> 'b1:install', 'a1:populate_sysroot', 'b1:populate_sysroot',
> + 'a1:package_write_ipk_setscene',
> 'b1:packagedata_setscene', 'b1:package_write_rpm_setscene',
> + 'a1:package_write_rpm_setscene',
> 'b1:package_write_ipk_setscene', 'a1:packagedata_setscene']
> + self.assertEqual(set(tasks), set(expected))
> +
> + self.shutdown(tempdir)
> +
> + @unittest.skipIf(sys.version_info < (3, 5, 0), 'Python 3.5 or
> later required')
> + def test_hashserv_multiple_setscene(self):
> + # Runs e1:do_package_setscene twice
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + extraenv = {
> + "BB_HASHSERVE" : "auto",
> + "BB_SIGNATURE_HANDLER" : "TestEquivHash"
> + }
> + cmd = ["bitbake", "a1", "b1", "e1"]
> + setscenetasks = ['package_write_ipk_setscene',
> 'package_write_rpm_setscene', 'packagedata_setscene',
> + 'populate_sysroot_setscene',
> 'package_qa_setscene']
> + sstatevalid = ""
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
> extraenv=extraenv, cleanup=True)
> + expected = ['a1:' + x for x in self.alltasks] + ['b1:' +
> x for x in self.alltasks] + ['e1:' + x for x in self.alltasks]
> + self.assertEqual(set(tasks), set(expected))
> + cmd = ["bitbake", "a1", "b1", "-c", "install", "-fn"]
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
> extraenv=extraenv, cleanup=True)
> + cmd = ["bitbake", "e1"]
> + sstatevalid = "e1:do_package"
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
> extraenv=extraenv, cleanup=True, slowtasks="a1:populate_sysroot
> b1:populate_sysroot")
> + expected = ['a1:package', 'a1:install', 'b1:package',
> 'b1:install', 'a1:populate_sysroot', 'b1:populate_sysroot',
> + 'a1:package_write_ipk_setscene',
> 'b1:packagedata_setscene', 'b1:package_write_rpm_setscene',
> + 'a1:package_write_rpm_setscene',
> 'b1:package_write_ipk_setscene', 'a1:packagedata_setscene',
> + 'e1:package_setscene']
> + self.assertEqual(set(tasks), set(expected))
> + for i in expected:
> + self.assertEqual(tasks.count(i), 1, "%s not in task
> list once" % i) +
> + self.shutdown(tempdir)
> +
> + def shutdown(self, tempdir):
> + # Wait for the hashserve socket to disappear else we'll see
> races with the tempdir cleanup
> + while os.path.exists(tempdir + "/hashserve.sock"):
> + time.sleep(0.5)
> +
> +
> diff --git a/bitbake/lib/bb/tests/utils.py
> b/bitbake/lib/bb/tests/utils.py index 2f4ccf3..f4adf1d 100644
> --- a/bitbake/lib/bb/tests/utils.py
> +++ b/bitbake/lib/bb/tests/utils.py
> @@ -1,22 +1,9 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # BitBake Tests for utils.py
> #
> # Copyright (C) 2012 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
>
> import unittest
> @@ -42,6 +29,14 @@ class VerCmpString(unittest.TestCase):
> self.assertTrue(result < 0)
> result = bb.utils.vercmp_string('1.1', '1.0+1.1-beta1')
> self.assertTrue(result > 0)
> + result = bb.utils.vercmp_string('1a', '1a1')
> + self.assertTrue(result < 0)
> + result = bb.utils.vercmp_string('1a1', '1a')
> + self.assertTrue(result > 0)
> + result = bb.utils.vercmp_string('1.', '1.1')
> + self.assertTrue(result < 0)
> + result = bb.utils.vercmp_string('1.1', '1.')
> + self.assertTrue(result > 0)
>
> def test_explode_dep_versions(self):
> correctresult = {"foo" : ["= 1.10"]}
> diff --git a/bitbake/lib/bb/tinfoil.py b/bitbake/lib/bb/tinfoil.py
> index 368264f..0a1b913 100644
> --- a/bitbake/lib/bb/tinfoil.py
> +++ b/bitbake/lib/bb/tinfoil.py
> @@ -4,18 +4,8 @@
> # Copyright (C) 2011 Mentor Graphics Corporation
> # Copyright (C) 2006-2012 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import logging
> import os
> diff --git a/bitbake/lib/bb/ui/__init__.py
> b/bitbake/lib/bb/ui/__init__.py index a4805ed..4b7ac36 100644
> --- a/bitbake/lib/bb/ui/__init__.py
> +++ b/bitbake/lib/bb/ui/__init__.py
> @@ -3,15 +3,5 @@
> #
> # Copyright (C) 2006-2007 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. diff --git a/bitbake/lib/bb/ui/buildinfohelper.py
> b/bitbake/lib/bb/ui/buildinfohelper.py index 31323d2..5cbca97 100644
> --- a/bitbake/lib/bb/ui/buildinfohelper.py
> +++ b/bitbake/lib/bb/ui/buildinfohelper.py
> @@ -3,18 +3,8 @@
> #
> # Copyright (C) 2013 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import sys
> import bb
> @@ -656,6 +646,9 @@ class ORMWrapper(object):
> Target_Installed_Package.objects.create(target =
> target_obj, package = packagedict[p]['object'])
> packagedeps_objs = []
> + pattern_so = re.compile(r'.*\.so(\.\d*)?$')
> + pattern_lib = re.compile(r'.*\-suffix(\d*)?$')
> + pattern_ko = re.compile(r'^kernel-module-.*')
> for p in packagedict:
> for (px,deptype) in packagedict[p]['depends']:
> if deptype == 'depends':
> @@ -664,6 +657,13 @@ class ORMWrapper(object):
> tdeptype = Package_Dependency.TYPE_TRECOMMENDS
>
> try:
> + # Skip known non-package objects like libraries
> and kernel modules
> + if pattern_so.match(px) or pattern_lib.match(px):
> + logger.info("Toaster does not add library
> file dependencies to packages (%s,%s)", p, px)
> + continue
> + if pattern_ko.match(px):
> + logger.info("Toaster does not add kernel
> module dependencies to packages (%s,%s)", p, px)
> + continue
> packagedeps_objs.append(Package_Dependency(
> package = packagedict[p]['object'],
> depends_on = packagedict[px]['object'],
> diff --git a/bitbake/lib/bb/ui/knotty.py b/bitbake/lib/bb/ui/knotty.py
> index fa88e6c..35736ad 100644
> --- a/bitbake/lib/bb/ui/knotty.py
> +++ b/bitbake/lib/bb/ui/knotty.py
> @@ -5,18 +5,8 @@
> #
> # Copyright (C) 2006-2012 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from __future__ import division
>
> @@ -222,6 +212,23 @@ class TerminalFilter(object):
> sys.stdout.flush()
> self.footer_present = False
>
> + def elapsed(self, sec):
> + hrs = int(sec / 3600.0)
> + sec -= hrs * 3600
> + min = int(sec / 60.0)
> + sec -= min * 60
> + if hrs > 0:
> + return "%dh%dm%ds" % (hrs, min, sec)
> + elif min > 0:
> + return "%dm%ds" % (min, sec)
> + else:
> + return "%ds" % (sec)
> +
> + def keepAlive(self, t):
> + if not self.cuu:
> + print("Bitbake still alive (%ds)" % t)
> + sys.stdout.flush()
> +
> def updateFooter(self):
> if not self.cuu:
> return
> @@ -258,7 +265,7 @@ class TerminalFilter(object):
> else:
> start_time = activetasks[t].get("starttime", None)
> if start_time:
> - tasks.append("%s - %ds (pid %s)" %
> (activetasks[t]["title"], currenttime - start_time, t))
> + tasks.append("%s - %s (pid %s)" %
> (activetasks[t]["title"], self.elapsed(currenttime - start_time), t))
> else: tasks.append("%s (pid %s)" % (activetasks[t]["title"], t))
>
> @@ -293,8 +300,8 @@ class TerminalFilter(object):
> if start_time:
> pbar.start_time = start_time
> pbar.setmessage('%s:%s' % (tasknum,
> pbar.msg.split(':', 1)[1]))
> + pbar.setextra(rate)
> if progress > -1:
> - pbar.setextra(rate)
> content = pbar.update(progress)
> else:
> content = pbar.update(1)
> @@ -455,11 +462,17 @@ def main(server, eventHandler, params, tf =
> TerminalFilter): warnings = 0
> taskfailures = []
>
> + printinterval = 5000
> + lastprint = time.time()
> +
> termfilter = tf(main, helper, console, errconsole, format,
> params.options.quiet) atexit.register(termfilter.finish)
>
> while True:
> try:
> + if (lastprint + printinterval) <= time.time():
> + termfilter.keepAlive(printinterval)
> + printinterval += 5000
> event = eventHandler.waitEvent(0)
> if event is None:
> if main.shutdown > 1:
> @@ -488,6 +501,8 @@ def main(server, eventHandler, params, tf =
> TerminalFilter): continue
>
> if isinstance(event, logging.LogRecord):
> + lastprint = time.time()
> + printinterval = 5000
> if event.levelno >= format.ERROR:
> errors = errors + 1
> return_value = 1
> @@ -645,7 +660,6 @@ def main(server, eventHandler, params, tf =
> TerminalFilter): # ignore
> if isinstance(event, (bb.event.BuildBase,
> bb.event.MetadataEvent,
> - bb.event.StampUpdate,
> bb.event.ConfigParsed,
> bb.event.MultiConfigParsed,
> bb.event.RecipeParsed,
> @@ -675,17 +689,27 @@ def main(server, eventHandler, params, tf =
> TerminalFilter): if params.observe_only:
> print("\nKeyboard Interrupt, exiting observer...")
> main.shutdown = 2
> - if not params.observe_only and main.shutdown == 1:
> +
> + def state_force_shutdown():
> print("\nSecond Keyboard Interrupt, stopping...\n")
> _, error = server.runCommand(["stateForceShutdown"])
> if error:
> logger.error("Unable to cleanly stop: %s" %
> error) +
> + if not params.observe_only and main.shutdown == 1:
> + state_force_shutdown()
> +
> if not params.observe_only and main.shutdown == 0:
> print("\nKeyboard Interrupt, closing down...\n")
> interrupted = True
> - _, error = server.runCommand(["stateShutdown"])
> - if error:
> - logger.error("Unable to cleanly shutdown: %s" %
> error)
> + # Capture the second KeyboardInterrupt during
> stateShutdown is running
> + try:
> + _, error = server.runCommand(["stateShutdown"])
> + if error:
> + logger.error("Unable to cleanly shutdown:
> %s" % error)
> + except KeyboardInterrupt:
> + state_force_shutdown()
> +
> main.shutdown = main.shutdown + 1
> pass
> except Exception as e:
> diff --git a/bitbake/lib/bb/ui/ncurses.py
> b/bitbake/lib/bb/ui/ncurses.py index 8690c52..c422732 100644
> --- a/bitbake/lib/bb/ui/ncurses.py
> +++ b/bitbake/lib/bb/ui/ncurses.py
> @@ -6,18 +6,8 @@
> # Copyright (C) 2006 Michael 'Mickey' Lauer
> # Copyright (C) 2006-2007 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> """
> We have the following windows:
> diff --git a/bitbake/lib/bb/ui/taskexp.py
> b/bitbake/lib/bb/ui/taskexp.py index 8305d70..50a943c 100644
> --- a/bitbake/lib/bb/ui/taskexp.py
> +++ b/bitbake/lib/bb/ui/taskexp.py
> @@ -4,18 +4,8 @@
> # Copyright (C) 2007 Ross Burton
> # Copyright (C) 2007 - 2008 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import sys
> import gi
> diff --git a/bitbake/lib/bb/ui/toasterui.py
> b/bitbake/lib/bb/ui/toasterui.py index 88cec37..51892c9 100644
> --- a/bitbake/lib/bb/ui/toasterui.py
> +++ b/bitbake/lib/bb/ui/toasterui.py
> @@ -7,18 +7,8 @@
> # Copyright (C) 2006-2012 Richard Purdie
> # Copyright (C) 2013 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from __future__ import division
> import time
> diff --git a/bitbake/lib/bb/ui/uievent.py
> b/bitbake/lib/bb/ui/uievent.py index 9542b91..fedb050 100644
> --- a/bitbake/lib/bb/ui/uievent.py
> +++ b/bitbake/lib/bb/ui/uievent.py
> @@ -1,22 +1,9 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # Copyright (C) 2006 - 2007 Michael 'Mickey' Lauer
> # Copyright (C) 2006 - 2007 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. -
>
> """
> Use this class to fork off a thread to recieve event callbacks from
> the bitbake diff --git a/bitbake/lib/bb/ui/uihelper.py
> b/bitbake/lib/bb/ui/uihelper.py index 963c1ea..c8dd7df 100644
> --- a/bitbake/lib/bb/ui/uihelper.py
> +++ b/bitbake/lib/bb/ui/uihelper.py
> @@ -1,21 +1,9 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # Copyright (C) 2006 - 2007 Michael 'Mickey' Lauer
> # Copyright (C) 2006 - 2007 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import bb.build
> import time
> @@ -52,7 +40,7 @@ class BBUIHelper:
> self.running_pids.remove(event.pid)
> self.failed_tasks.append( { 'title' : "%s %s" %
> (event._package, event._task)}) self.needUpdate = True
> - elif isinstance(event, bb.runqueue.runQueueTaskStarted) or
> isinstance(event, bb.runqueue.sceneQueueTaskStarted):
> + elif isinstance(event, bb.runqueue.runQueueTaskStarted):
> self.tasknumber_current = event.stats.completed +
> event.stats.active + event.stats.failed + 1 self.tasknumber_total =
> event.stats.total self.needUpdate = True
> diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
> index 13bb5f2..d035949 100644
> --- a/bitbake/lib/bb/utils.py
> +++ b/bitbake/lib/bb/utils.py
> @@ -1,23 +1,11 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake Utility Functions
> """
>
> # Copyright (C) 2004 Michael Lauer
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import re, fcntl, os, string, stat, shutil, time
> import sys
> @@ -121,6 +109,10 @@ def vercmp_part(a, b):
> return -1
> elif oa > ob:
> return 1
> + elif ca is None:
> + return -1
> + elif cb is None:
> + return 1
> elif ca < cb:
> return -1
> elif ca > cb:
> @@ -402,7 +394,7 @@ def better_exec(code, context, text = None,
> realfile = "<code>", pythonexception code = better_compile(code,
> realfile, realfile) try:
> exec(code, get_context(), context)
> - except (bb.BBHandledException, bb.parse.SkipRecipe,
> bb.build.FuncFailed, bb.data_smart.ExpansionError):
> + except (bb.BBHandledException, bb.parse.SkipRecipe,
> bb.data_smart.ExpansionError): # Error already shown so passthrough,
> no need for traceback raise
> except Exception as e:
> @@ -685,7 +677,7 @@ def _check_unsafe_delete_path(path):
> return True
> return False
>
> -def remove(path, recurse=False):
> +def remove(path, recurse=False, ionice=False):
> """Equivalent to rm -f or rm -rf"""
> if not path:
> return
> @@ -694,7 +686,10 @@ def remove(path, recurse=False):
> if _check_unsafe_delete_path(path):
> raise Exception('bb.utils.remove: called with
> dangerous path "%s" and recurse=True, refusing to delete!' % path) #
> shutil.rmtree(name) would be ideal but its too slow
> - subprocess.check_call(['rm', '-rf'] + glob.glob(path))
> + cmd = []
> + if ionice:
> + cmd = ['ionice', '-c', '3']
> + subprocess.check_call(cmd + ['rm', '-rf'] + glob.glob(path))
> return
> for name in glob.glob(path):
> try:
> @@ -703,20 +698,12 @@ def remove(path, recurse=False):
> if exc.errno != errno.ENOENT:
> raise
>
> -def prunedir(topdir):
> +def prunedir(topdir, ionice=False):
> # Delete everything reachable from the directory named in
> 'topdir'. # CAUTION: This is dangerous!
> if _check_unsafe_delete_path(topdir):
> raise Exception('bb.utils.prunedir: called with dangerous
> path "%s", refusing to delete!' % topdir)
> - for root, dirs, files in os.walk(topdir, topdown = False):
> - for name in files:
> - os.remove(os.path.join(root, name))
> - for name in dirs:
> - if os.path.islink(os.path.join(root, name)):
> - os.remove(os.path.join(root, name))
> - else:
> - os.rmdir(os.path.join(root, name))
> - os.rmdir(topdir)
> + remove(topdir, recurse=True, ionice=ionice)
>
> #
> # Could also use return re.compile("(%s)" % "|".join(map(re.escape,
> suffixes))).sub(lambda mo: "", var) @@ -726,8 +713,8 @@ def
> prune_suffix(var, suffixes, d): # See if var ends with any of the
> suffixes listed and # remove it if found
> for suffix in suffixes:
> - if var.endswith(suffix):
> - return var.replace(suffix, "")
> + if suffix and var.endswith(suffix):
> + return var[:-len(suffix)]
> return var
>
> def mkdirhier(directory):
> @@ -738,7 +725,7 @@ def mkdirhier(directory):
> try:
> os.makedirs(directory)
> except OSError as e:
> - if e.errno != errno.EEXIST:
> + if e.errno != errno.EEXIST or not os.path.isdir(directory):
> raise e
>
> def movefile(src, dest, newmtime = None, sstat = None):
> @@ -796,7 +783,7 @@ def movefile(src, dest, newmtime = None, sstat =
> None): os.rename(src, destpath)
> renamefailed = 0
> except Exception as e:
> - if e[0] != errno.EXDEV:
> + if e.errno != errno.EXDEV:
> # Some random error.
> print("movefile: Failed to move", src, "to", dest, e)
> return None
> @@ -1505,6 +1492,8 @@ def ioprio_set(who, cls, value):
> NR_ioprio_set = 251
> elif _unamearch[0] == "i" and _unamearch[2:3] == "86":
> NR_ioprio_set = 289
> + elif _unamearch == "aarch64":
> + NR_ioprio_set = 30
>
> if NR_ioprio_set:
> ioprio = value | (cls << IOPRIO_CLASS_SHIFT)
> diff --git a/bitbake/lib/bblayers/__init__.py
> b/bitbake/lib/bblayers/__init__.py index 3ad9513..4e7c09d 100644
> --- a/bitbake/lib/bblayers/__init__.py
> +++ b/bitbake/lib/bblayers/__init__.py
> @@ -1,2 +1,6 @@
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> from pkgutil import extend_path
> __path__ = extend_path(__path__, __name__)
> diff --git a/bitbake/lib/bblayers/action.py
> b/bitbake/lib/bblayers/action.py index a3f658f..d6459d6 100644
> --- a/bitbake/lib/bblayers/action.py
> +++ b/bitbake/lib/bblayers/action.py
> @@ -1,3 +1,7 @@
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> import fnmatch
> import logging
> import os
> diff --git a/bitbake/lib/bblayers/common.py
> b/bitbake/lib/bblayers/common.py index 98515ce..c5657d3 100644
> --- a/bitbake/lib/bblayers/common.py
> +++ b/bitbake/lib/bblayers/common.py
> @@ -1,3 +1,7 @@
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> import argparse
> import logging
> import os
> diff --git a/bitbake/lib/bblayers/layerindex.py
> b/bitbake/lib/bblayers/layerindex.py index 9f02a9d..57cd902 100644
> --- a/bitbake/lib/bblayers/layerindex.py
> +++ b/bitbake/lib/bblayers/layerindex.py
> @@ -1,3 +1,7 @@
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> import layerindexlib
>
> import argparse
> @@ -28,7 +32,7 @@ class LayerIndexPlugin(ActionPlugin):
> layerdir = os.path.join(repodir, subdir)
> if not os.path.exists(repodir):
> if fetch_layer:
> - result = subprocess.call('git clone %s %s' % (url,
> repodir), shell = True)
> + result = subprocess.call(['git', 'clone', url,
> repodir]) if result:
> logger.error("Failed to download %s" % url)
> return None, None, None
> diff --git a/bitbake/lib/bblayers/query.py
> b/bitbake/lib/bblayers/query.py index 9294dfa..7db49c8 100644
> --- a/bitbake/lib/bblayers/query.py
> +++ b/bitbake/lib/bblayers/query.py
> @@ -1,3 +1,7 @@
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> import collections
> import fnmatch
> import logging
> @@ -42,7 +46,7 @@ layer, with the preferred version first. Note that
> skipped recipes that are overlayed will also be listed, with a
> " (skipped)" suffix. """
>
> - items_listed = self.list_recipes('Overlayed recipes', None,
> True, args.same_version, args.filenames, True, None)
> + items_listed = self.list_recipes('Overlayed recipes', None,
> True, args.same_version, args.filenames, False, True, None, False,
> None) # Check for overlayed .bbclass files
> classes = collections.defaultdict(list)
> @@ -108,9 +112,9 @@ skipped recipes will also be listed, with a
> " (skipped)" suffix. title = 'Matching recipes:'
> else:
> title = 'Available recipes:'
> - self.list_recipes(title, args.pnspec, False, False,
> args.filenames, args.multiple, inheritlist)
> + self.list_recipes(title, args.pnspec, False, False,
> args.filenames, args.recipes_only, args.multiple, args.layer,
> args.bare, inheritlist)
> - def list_recipes(self, title, pnspec, show_overlayed_only,
> show_same_ver_only, show_filenames, show_multi_provider_only,
> inherits):
> + def list_recipes(self, title, pnspec, show_overlayed_only,
> show_same_ver_only, show_filenames, show_recipes_only,
> show_multi_provider_only, selected_layer, bare, inherits): if
> inherits: bbpath = str(self.tinfoil.config_data.getVar('BBPATH')) for
> classname in inherits: @@ -140,24 +144,30 @@ skipped recipes will
> also be listed, with a " (skipped)" suffix. preferred_versions[p] =
> (ver, fn)
> def print_item(f, pn, ver, layer, ispref):
> - if f in skiplist:
> - skipped = ' (skipped)'
> - else:
> - skipped = ''
> - if show_filenames:
> - if ispref:
> - logger.plain("%s%s", f, skipped)
> + if not selected_layer or layer == selected_layer:
> + if not bare and f in skiplist:
> + skipped = ' (skipped)'
> else:
> - logger.plain(" %s%s", f, skipped)
> - else:
> - if ispref:
> - logger.plain("%s:", pn)
> - logger.plain(" %s %s%s", layer.ljust(20), ver,
> skipped)
> + skipped = ''
> + if show_filenames:
> + if ispref:
> + logger.plain("%s%s", f, skipped)
> + else:
> + logger.plain(" %s%s", f, skipped)
> + elif show_recipes_only:
> + if pn not in show_unique_pn:
> + show_unique_pn.append(pn)
> + logger.plain("%s%s", pn, skipped)
> + else:
> + if ispref:
> + logger.plain("%s:", pn)
> + logger.plain(" %s %s%s", layer.ljust(20), ver,
> skipped)
> global_inherit = (self.tinfoil.config_data.getVar('INHERIT')
> or "").split() cls_re = re.compile('classes/')
>
> preffiles = []
> + show_unique_pn = []
> items_listed = False
> for p in sorted(pkg_pn):
> if pnspec:
> @@ -489,8 +499,11 @@ NOTE: .bbappend files can impact the
> dependencies.
> parser_show_recipes = self.add_command(sp, 'show-recipes',
> self.do_show_recipes) parser_show_recipes.add_argument('-f',
> '--filenames', help='instead of the default formatting, list
> filenames of higher priority recipes with the ones they overlay
> indented underneath', action='store_true')
> + parser_show_recipes.add_argument('-r', '--recipes-only',
> help='instead of the default formatting, list recipes only',
> action='store_true') parser_show_recipes.add_argument('-m',
> '--multiple', help='only list where multiple recipes (in the same
> layer or different layers) exist for the same recipe name',
> action='store_true') parser_show_recipes.add_argument('-i',
> '--inherits', help='only list recipes that inherit the named
> class(es) - separate multiple classes using , (without spaces)',
> metavar='CLASS', default='')
> + parser_show_recipes.add_argument('-l', '--layer', help='only
> list recipes from the selected layer', default='')
> + parser_show_recipes.add_argument('-b', '--bare',
> help='output just names without the "(skipped)" marker',
> action='store_true') parser_show_recipes.add_argument('pnspec',
> nargs='*', help='optional recipe name specification (wildcards
> allowed, enclose in quotes to avoid shell expansion)')
> parser_show_appends = self.add_command(sp, 'show-appends',
> self.do_show_appends) diff --git a/bitbake/lib/bs4/dammit.py
> b/bitbake/lib/bs4/dammit.py index 68d419f..805aa90 100644 ---
> a/bitbake/lib/bs4/dammit.py +++ b/bitbake/lib/bs4/dammit.py @@ -45,9
> +45,9 @@ except ImportError: pass
>
> xml_encoding_re = re.compile(
> - '^<\?.*encoding=[\'"](.*?)[\'"].*\?>'.encode(), re.I)
> + r'^<\?.*encoding=[\'"](.*?)[\'"].*\?>'.encode(), re.I)
> html_meta_re = re.compile(
> - '<\s*meta[^>]+charset\s*=\s*["\']?([^>]*?)[ /;\'">]'.encode(),
> re.I)
> + r'<\s*meta[^>]+charset\s*=\s*["\']?([^>]*?)[ /;\'">]'.encode(),
> re.I)
> class EntitySubstitution(object):
>
> @@ -80,11 +80,11 @@ class EntitySubstitution(object):
> ">": "gt",
> }
>
> - BARE_AMPERSAND_OR_BRACKET = re.compile("([<>]|"
> -
> "&(?!#\d+;|#x[0-9a-fA-F]+;|\w+;)"
> - ")")
> + BARE_AMPERSAND_OR_BRACKET = re.compile(r"([<>]|"
> +
> r"&(?!#\d+;|#x[0-9a-fA-F]+;|\w+;)"
> + r")")
>
> - AMPERSAND_OR_BRACKET = re.compile("([<>&])")
> + AMPERSAND_OR_BRACKET = re.compile(r"([<>&])")
>
> @classmethod
> def _substitute_html_entity(cls, matchobj):
> diff --git a/bitbake/lib/bs4/element.py b/bitbake/lib/bs4/element.py
> index 0e62c2e..3775a60 100644
> --- a/bitbake/lib/bs4/element.py
> +++ b/bitbake/lib/bs4/element.py
> @@ -1,7 +1,7 @@
> __license__ = "MIT"
>
> from pdb import set_trace
> -import collections
> +import collections.abc
> import re
> import sys
> import warnings
> @@ -10,7 +10,7 @@ from bs4.dammit import EntitySubstitution
> DEFAULT_OUTPUT_ENCODING = "utf-8"
> PY3K = (sys.version_info[0] > 2)
>
> -whitespace_re = re.compile("\s+")
> +whitespace_re = re.compile(r"\s+")
>
> def _alias(attr):
> """Alias one attribute name to another for backward
> compatibility""" @@ -67,7 +67,7 @@ class
> ContentMetaAttributeValue(AttributeValueWithCharsetSubstitution): The
> value of the 'content' attribute will be one of these objects. """
>
> - CHARSET_RE = re.compile("((^|;)\s*charset=)([^;]*)", re.M)
> + CHARSET_RE = re.compile(r"((^|;)\s*charset=)([^;]*)", re.M)
>
> def __new__(cls, original_value):
> match = cls.CHARSET_RE.search(original_value)
> @@ -155,7 +155,7 @@ class PageElement(object):
>
> def format_string(self, s, formatter='minimal'):
> """Format the given string using the given formatter."""
> - if not isinstance(formatter, collections.Callable):
> + if not isinstance(formatter, collections.abc.Callable):
> formatter = self._formatter_for_name(formatter)
> if formatter is None:
> output = s
> @@ -580,7 +580,7 @@ class PageElement(object):
>
> # Methods for supporting CSS selectors.
>
> - tag_name_re = re.compile('^[a-zA-Z0-9][-.a-zA-Z0-9:_]*$')
> + tag_name_re = re.compile(r'^[a-zA-Z0-9][-.a-zA-Z0-9:_]*$')
>
> # /^([a-zA-Z0-9][-.a-zA-Z0-9:_]*)\[(\w+)([=~\|\^\$\*]?)=?"?([^\]"]*)"?\]$/
> # \---------------------------/ \---/\-------------/
> \-------/ @@ -1077,7 +1077,7 @@ class Tag(PageElement):
>
> # First off, turn a string formatter into a function. This
> # will stop the lookup from happening over and over again.
> - if not isinstance(formatter, collections.Callable):
> + if not isinstance(formatter, collections.abc.Callable):
> formatter = self._formatter_for_name(formatter)
>
> attrs = []
> @@ -1181,7 +1181,7 @@ class Tag(PageElement):
> """
> # First off, turn a string formatter into a function. This
> # will stop the lookup from happening over and over again.
> - if not isinstance(formatter, collections.Callable):
> + if not isinstance(formatter, collections.abc.Callable):
> formatter = self._formatter_for_name(formatter)
>
> pretty_print = (indent_level is not None)
> @@ -1364,7 +1364,7 @@ class Tag(PageElement):
> if tag_name == '':
> raise ValueError(
> "A pseudo-class must be prefixed with a tag
> name.")
> - pseudo_attributes =
> re.match('([a-zA-Z\d-]+)\(([a-zA-Z\d]+)\)', pseudo)
> + pseudo_attributes =
> re.match(r'([a-zA-Z\d-]+)\(([a-zA-Z\d]+)\)', pseudo) found = []
> if pseudo_attributes is None:
> pseudo_type = pseudo
> @@ -1562,7 +1562,7 @@ class SoupStrainer(object):
> def _normalize_search_value(self, value):
> # Leave it alone if it's a Unicode string, a callable, a
> # regular expression, a boolean, or None.
> - if (isinstance(value, str) or isinstance(value,
> collections.Callable) or hasattr(value, 'match')
> + if (isinstance(value, str) or isinstance(value,
> collections.abc.Callable) or hasattr(value, 'match') or
> isinstance(value, bool) or value is None): return value
>
> @@ -1602,7 +1602,7 @@ class SoupStrainer(object):
> markup = markup_name
> markup_attrs = markup
> call_function_with_tag_data = (
> - isinstance(self.name, collections.Callable)
> + isinstance(self.name, collections.abc.Callable)
> and not isinstance(markup_name, Tag))
>
> if ((not self.name)
> @@ -1688,7 +1688,7 @@ class SoupStrainer(object):
> # True matches any non-None value.
> return markup is not None
>
> - if isinstance(match_against, collections.Callable):
> + if isinstance(match_against, collections.abc.Callable):
> return match_against(markup)
>
> # Custom callables take the tag as an argument, but all
> diff --git a/bitbake/lib/hashserv/__init__.py
> b/bitbake/lib/hashserv/__init__.py new file mode 100644
> index 0000000..c331862
> --- /dev/null
> +++ b/bitbake/lib/hashserv/__init__.py
> @@ -0,0 +1,93 @@
> +# Copyright (C) 2018-2019 Garmin Ltd.
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> +from contextlib import closing
> +import re
> +import sqlite3
> +
> +UNIX_PREFIX = "unix://"
> +
> +ADDR_TYPE_UNIX = 0
> +ADDR_TYPE_TCP = 1
> +
> +
> +def setup_database(database, sync=True):
> + db = sqlite3.connect(database)
> + db.row_factory = sqlite3.Row
> +
> + with closing(db.cursor()) as cursor:
> + cursor.execute('''
> + CREATE TABLE IF NOT EXISTS tasks_v2 (
> + id INTEGER PRIMARY KEY AUTOINCREMENT,
> + method TEXT NOT NULL,
> + outhash TEXT NOT NULL,
> + taskhash TEXT NOT NULL,
> + unihash TEXT NOT NULL,
> + created DATETIME,
> +
> + -- Optional fields
> + owner TEXT,
> + PN TEXT,
> + PV TEXT,
> + PR TEXT,
> + task TEXT,
> + outhash_siginfo TEXT,
> +
> + UNIQUE(method, outhash, taskhash)
> + )
> + ''')
> + cursor.execute('PRAGMA journal_mode = WAL')
> + cursor.execute('PRAGMA synchronous = %s' % ('NORMAL' if sync
> else 'OFF')) +
> + # Drop old indexes
> + cursor.execute('DROP INDEX IF EXISTS taskhash_lookup')
> + cursor.execute('DROP INDEX IF EXISTS outhash_lookup')
> +
> + # Create new indexes
> + cursor.execute('CREATE INDEX IF NOT EXISTS
> taskhash_lookup_v2 ON tasks_v2 (method, taskhash, created)')
> + cursor.execute('CREATE INDEX IF NOT EXISTS outhash_lookup_v2
> ON tasks_v2 (method, outhash)') +
> + return db
> +
> +
> +def parse_address(addr):
> + if addr.startswith(UNIX_PREFIX):
> + return (ADDR_TYPE_UNIX, (addr[len(UNIX_PREFIX):],))
> + else:
> + m = re.match(r'\[(?P<host>[^\]]*)\]:(?P<port>\d+)$', addr)
> + if m is not None:
> + host = m.group('host')
> + port = m.group('port')
> + else:
> + host, port = addr.split(':')
> +
> + return (ADDR_TYPE_TCP, (host, int(port)))
> +
> +
> +def create_server(addr, dbname, *, sync=True):
> + from . import server
> + db = setup_database(dbname, sync=sync)
> + s = server.Server(db)
> +
> + (typ, a) = parse_address(addr)
> + if typ == ADDR_TYPE_UNIX:
> + s.start_unix_server(*a)
> + else:
> + s.start_tcp_server(*a)
> +
> + return s
> +
> +
> +def create_client(addr):
> + from . import client
> + c = client.Client()
> +
> + (typ, a) = parse_address(addr)
> + if typ == ADDR_TYPE_UNIX:
> + c.connect_unix(*a)
> + else:
> + c.connect_tcp(*a)
> +
> + return c
> diff --git a/bitbake/lib/hashserv/client.py
> b/bitbake/lib/hashserv/client.py new file mode 100644
> index 0000000..f659566
> --- /dev/null
> +++ b/bitbake/lib/hashserv/client.py
> @@ -0,0 +1,157 @@
> +# Copyright (C) 2019 Garmin Ltd.
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> +from contextlib import closing
> +import json
> +import logging
> +import socket
> +import os
> +
> +
> +logger = logging.getLogger('hashserv.client')
> +
> +
> +class HashConnectionError(Exception):
> + pass
> +
> +
> +class Client(object):
> + MODE_NORMAL = 0
> + MODE_GET_STREAM = 1
> +
> + def __init__(self):
> + self._socket = None
> + self.reader = None
> + self.writer = None
> + self.mode = self.MODE_NORMAL
> +
> + def connect_tcp(self, address, port):
> + def connect_sock():
> + s = socket.create_connection((address, port))
> +
> + s.setsockopt(socket.SOL_TCP, socket.TCP_NODELAY, 1)
> + s.setsockopt(socket.SOL_TCP, socket.TCP_QUICKACK, 1)
> + s.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
> + return s
> +
> + self._connect_sock = connect_sock
> +
> + def connect_unix(self, path):
> + def connect_sock():
> + s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
> + # AF_UNIX has path length issues so chdir here to
> workaround
> + cwd = os.getcwd()
> + try:
> + os.chdir(os.path.dirname(path))
> + s.connect(os.path.basename(path))
> + finally:
> + os.chdir(cwd)
> + return s
> +
> + self._connect_sock = connect_sock
> +
> + def connect(self):
> + if self._socket is None:
> + self._socket = self._connect_sock()
> +
> + self.reader = self._socket.makefile('r',
> encoding='utf-8')
> + self.writer = self._socket.makefile('w',
> encoding='utf-8') +
> + self.writer.write('OEHASHEQUIV 1.0\n\n')
> + self.writer.flush()
> +
> + # Restore mode if the socket is being re-created
> + cur_mode = self.mode
> + self.mode = self.MODE_NORMAL
> + self._set_mode(cur_mode)
> +
> + return self._socket
> +
> + def close(self):
> + if self._socket is not None:
> + self._socket.close()
> + self._socket = None
> + self.reader = None
> + self.writer = None
> +
> + def _send_wrapper(self, proc):
> + count = 0
> + while True:
> + try:
> + self.connect()
> + return proc()
> + except (OSError, HashConnectionError,
> json.JSONDecodeError, UnicodeDecodeError) as e:
> + logger.warning('Error talking to server: %s' % e)
> + if count >= 3:
> + if not isinstance(e, HashConnectionError):
> + raise HashConnectionError(str(e))
> + raise e
> + self.close()
> + count += 1
> +
> + def send_message(self, msg):
> + def proc():
> + self.writer.write('%s\n' % json.dumps(msg))
> + self.writer.flush()
> +
> + l = self.reader.readline()
> + if not l:
> + raise HashConnectionError('Connection closed')
> +
> + if not l.endswith('\n'):
> + raise HashConnectionError('Bad message %r' % message)
> +
> + return json.loads(l)
> +
> + return self._send_wrapper(proc)
> +
> + def send_stream(self, msg):
> + def proc():
> + self.writer.write("%s\n" % msg)
> + self.writer.flush()
> + l = self.reader.readline()
> + if not l:
> + raise HashConnectionError('Connection closed')
> + return l.rstrip()
> +
> + return self._send_wrapper(proc)
> +
> + def _set_mode(self, new_mode):
> + if new_mode == self.MODE_NORMAL and self.mode ==
> self.MODE_GET_STREAM:
> + r = self.send_stream('END')
> + if r != 'ok':
> + raise HashConnectionError('Bad response from server
> %r' % r)
> + elif new_mode == self.MODE_GET_STREAM and self.mode ==
> self.MODE_NORMAL:
> + r = self.send_message({'get-stream': None})
> + if r != 'ok':
> + raise HashConnectionError('Bad response from server
> %r' % r)
> + elif new_mode != self.mode:
> + raise Exception('Undefined mode transition %r -> %r' %
> (self.mode, new_mode)) +
> + self.mode = new_mode
> +
> + def get_unihash(self, method, taskhash):
> + self._set_mode(self.MODE_GET_STREAM)
> + r = self.send_stream('%s %s' % (method, taskhash))
> + if not r:
> + return None
> + return r
> +
> + def report_unihash(self, taskhash, method, outhash, unihash,
> extra={}):
> + self._set_mode(self.MODE_NORMAL)
> + m = extra.copy()
> + m['taskhash'] = taskhash
> + m['method'] = method
> + m['outhash'] = outhash
> + m['unihash'] = unihash
> + return self.send_message({'report': m})
> +
> + def get_stats(self):
> + self._set_mode(self.MODE_NORMAL)
> + return self.send_message({'get-stats': None})
> +
> + def reset_stats(self):
> + self._set_mode(self.MODE_NORMAL)
> + return self.send_message({'reset-stats': None})
> diff --git a/bitbake/lib/hashserv/server.py
> b/bitbake/lib/hashserv/server.py new file mode 100644
> index 0000000..0aff776
> --- /dev/null
> +++ b/bitbake/lib/hashserv/server.py
> @@ -0,0 +1,414 @@
> +# Copyright (C) 2019 Garmin Ltd.
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> +from contextlib import closing
> +from datetime import datetime
> +import asyncio
> +import json
> +import logging
> +import math
> +import os
> +import signal
> +import socket
> +import time
> +
> +logger = logging.getLogger('hashserv.server')
> +
> +
> +class Measurement(object):
> + def __init__(self, sample):
> + self.sample = sample
> +
> + def start(self):
> + self.start_time = time.perf_counter()
> +
> + def end(self):
> + self.sample.add(time.perf_counter() - self.start_time)
> +
> + def __enter__(self):
> + self.start()
> + return self
> +
> + def __exit__(self, *args, **kwargs):
> + self.end()
> +
> +
> +class Sample(object):
> + def __init__(self, stats):
> + self.stats = stats
> + self.num_samples = 0
> + self.elapsed = 0
> +
> + def measure(self):
> + return Measurement(self)
> +
> + def __enter__(self):
> + return self
> +
> + def __exit__(self, *args, **kwargs):
> + self.end()
> +
> + def add(self, elapsed):
> + self.num_samples += 1
> + self.elapsed += elapsed
> +
> + def end(self):
> + if self.num_samples:
> + self.stats.add(self.elapsed)
> + self.num_samples = 0
> + self.elapsed = 0
> +
> +
> +class Stats(object):
> + def __init__(self):
> + self.reset()
> +
> + def reset(self):
> + self.num = 0
> + self.total_time = 0
> + self.max_time = 0
> + self.m = 0
> + self.s = 0
> + self.current_elapsed = None
> +
> + def add(self, elapsed):
> + self.num += 1
> + if self.num == 1:
> + self.m = elapsed
> + self.s = 0
> + else:
> + last_m = self.m
> + self.m = last_m + (elapsed - last_m) / self.num
> + self.s = self.s + (elapsed - last_m) * (elapsed - self.m)
> +
> + self.total_time += elapsed
> +
> + if self.max_time < elapsed:
> + self.max_time = elapsed
> +
> + def start_sample(self):
> + return Sample(self)
> +
> + @property
> + def average(self):
> + if self.num == 0:
> + return 0
> + return self.total_time / self.num
> +
> + @property
> + def stdev(self):
> + if self.num <= 1:
> + return 0
> + return math.sqrt(self.s / (self.num - 1))
> +
> + def todict(self):
> + return {k: getattr(self, k) for k in ('num', 'total_time',
> 'max_time', 'average', 'stdev')} +
> +
> +class ServerClient(object):
> + def __init__(self, reader, writer, db, request_stats):
> + self.reader = reader
> + self.writer = writer
> + self.db = db
> + self.request_stats = request_stats
> +
> + async def process_requests(self):
> + try:
> + self.addr = self.writer.get_extra_info('peername')
> + logger.debug('Client %r connected' % (self.addr,))
> +
> + # Read protocol and version
> + protocol = await self.reader.readline()
> + if protocol is None:
> + return
> +
> + (proto_name, proto_version) =
> protocol.decode('utf-8').rstrip().split()
> + if proto_name != 'OEHASHEQUIV' or proto_version != '1.0':
> + return
> +
> + # Read headers. Currently, no headers are implemented,
> so look for
> + # an empty line to signal the end of the headers
> + while True:
> + line = await self.reader.readline()
> + if line is None:
> + return
> +
> + line = line.decode('utf-8').rstrip()
> + if not line:
> + break
> +
> + # Handle messages
> + handlers = {
> + 'get': self.handle_get,
> + 'report': self.handle_report,
> + 'get-stream': self.handle_get_stream,
> + 'get-stats': self.handle_get_stats,
> + 'reset-stats': self.handle_reset_stats,
> + }
> +
> + while True:
> + d = await self.read_message()
> + if d is None:
> + break
> +
> + for k in handlers.keys():
> + if k in d:
> + logger.debug('Handling %s' % k)
> + if 'stream' in k:
> + await handlers[k](d[k])
> + else:
> + with self.request_stats.start_sample()
> as self.request_sample, \
> + self.request_sample.measure():
> + await handlers[k](d[k])
> + break
> + else:
> + logger.warning("Unrecognized command %r" % d)
> + break
> +
> + await self.writer.drain()
> + finally:
> + self.writer.close()
> +
> + def write_message(self, msg):
> + self.writer.write(('%s\n' % json.dumps(msg)).encode('utf-8'))
> +
> + async def read_message(self):
> + l = await self.reader.readline()
> + if not l:
> + return None
> +
> + try:
> + message = l.decode('utf-8')
> +
> + if not message.endswith('\n'):
> + return None
> +
> + return json.loads(message)
> + except (json.JSONDecodeError, UnicodeDecodeError) as e:
> + logger.error('Bad message from client: %r' % message)
> + raise e
> +
> + async def handle_get(self, request):
> + method = request['method']
> + taskhash = request['taskhash']
> +
> + row = self.query_equivalent(method, taskhash)
> + if row is not None:
> + logger.debug('Found equivalent task %s -> %s',
> (row['taskhash'], row['unihash']))
> + d = {k: row[k] for k in ('taskhash', 'method',
> 'unihash')} +
> + self.write_message(d)
> + else:
> + self.write_message(None)
> +
> + async def handle_get_stream(self, request):
> + self.write_message('ok')
> +
> + while True:
> + l = await self.reader.readline()
> + if not l:
> + return
> +
> + try:
> + # This inner loop is very sensitive and must be as
> fast as
> + # possible (which is why the request sample is
> handled manually
> + # instead of using 'with', and also why logging
> statements are
> + # commented out.
> + self.request_sample =
> self.request_stats.start_sample()
> + request_measure = self.request_sample.measure()
> + request_measure.start()
> +
> + l = l.decode('utf-8').rstrip()
> + if l == 'END':
> + self.writer.write('ok\n'.encode('utf-8'))
> + return
> +
> + (method, taskhash) = l.split()
> + #logger.debug('Looking up %s %s' % (method,
> taskhash))
> + row = self.query_equivalent(method, taskhash)
> + if row is not None:
> + msg = ('%s\n' % row['unihash']).encode('utf-8')
> + #logger.debug('Found equivalent task %s -> %s',
> (row['taskhash'], row['unihash']))
> + else:
> + msg = '\n'.encode('utf-8')
> +
> + self.writer.write(msg)
> + finally:
> + request_measure.end()
> + self.request_sample.end()
> +
> + await self.writer.drain()
> +
> + async def handle_report(self, data):
> + with closing(self.db.cursor()) as cursor:
> + cursor.execute('''
> + -- Find tasks with a matching outhash (that is,
> tasks that
> + -- are equivalent)
> + SELECT taskhash, method, unihash FROM tasks_v2 WHERE
> method=:method AND outhash=:outhash +
> + -- If there is an exact match on the taskhash,
> return it.
> + -- Otherwise return the oldest matching outhash of
> any
> + -- taskhash
> + ORDER BY CASE WHEN taskhash=:taskhash THEN 1 ELSE 2
> END,
> + created ASC
> +
> + -- Only return one row
> + LIMIT 1
> + ''', {k: data[k] for k in ('method', 'outhash',
> 'taskhash')}) +
> + row = cursor.fetchone()
> +
> + # If no matching outhash was found, or one *was* found
> but it
> + # wasn't an exact match on the taskhash, a new entry for
> this
> + # taskhash should be added
> + if row is None or row['taskhash'] != data['taskhash']:
> + # If a row matching the outhash was found, the
> unihash for
> + # the new taskhash should be the same as that one.
> + # Otherwise the caller provided unihash is used.
> + unihash = data['unihash']
> + if row is not None:
> + unihash = row['unihash']
> +
> + insert_data = {
> + 'method': data['method'],
> + 'outhash': data['outhash'],
> + 'taskhash': data['taskhash'],
> + 'unihash': unihash,
> + 'created': datetime.now()
> + }
> +
> + for k in ('owner', 'PN', 'PV', 'PR', 'task',
> 'outhash_siginfo'):
> + if k in data:
> + insert_data[k] = data[k]
> +
> + cursor.execute('''INSERT INTO tasks_v2 (%s) VALUES
> (%s)''' % (
> + ', '.join(sorted(insert_data.keys())),
> + ', '.join(':' + k for k in
> sorted(insert_data.keys()))),
> + insert_data)
> +
> + self.db.commit()
> +
> + logger.info('Adding taskhash %s with unihash %s',
> + data['taskhash'], unihash)
> +
> + d = {
> + 'taskhash': data['taskhash'],
> + 'method': data['method'],
> + 'unihash': unihash
> + }
> + else:
> + d = {k: row[k] for k in ('taskhash', 'method',
> 'unihash')} +
> + self.write_message(d)
> +
> + async def handle_get_stats(self, request):
> + d = {
> + 'requests': self.request_stats.todict(),
> + }
> +
> + self.write_message(d)
> +
> + async def handle_reset_stats(self, request):
> + d = {
> + 'requests': self.request_stats.todict(),
> + }
> +
> + self.request_stats.reset()
> + self.write_message(d)
> +
> + def query_equivalent(self, method, taskhash):
> + # This is part of the inner loop and must be as fast as
> possible
> + try:
> + cursor = self.db.cursor()
> + cursor.execute('SELECT taskhash, method, unihash FROM
> tasks_v2 WHERE method=:method AND taskhash=:taskhash ORDER BY created
> ASC LIMIT 1',
> + {'method': method, 'taskhash': taskhash})
> + return cursor.fetchone()
> + except:
> + cursor.close()
> +
> +
> +class Server(object):
> + def __init__(self, db, loop=None):
> + self.request_stats = Stats()
> + self.db = db
> +
> + if loop is None:
> + self.loop = asyncio.new_event_loop()
> + self.close_loop = True
> + else:
> + self.loop = loop
> + self.close_loop = False
> +
> + self._cleanup_socket = None
> +
> + def start_tcp_server(self, host, port):
> + self.server = self.loop.run_until_complete(
> + asyncio.start_server(self.handle_client, host, port,
> loop=self.loop)
> + )
> +
> + for s in self.server.sockets:
> + logger.info('Listening on %r' % (s.getsockname(),))
> + # Newer python does this automatically. Do it manually
> here for
> + # maximum compatibility
> + s.setsockopt(socket.SOL_TCP, socket.TCP_NODELAY, 1)
> + s.setsockopt(socket.SOL_TCP, socket.TCP_QUICKACK, 1)
> +
> + name = self.server.sockets[0].getsockname()
> + if self.server.sockets[0].family == socket.AF_INET6:
> + self.address = "[%s]:%d" % (name[0], name[1])
> + else:
> + self.address = "%s:%d" % (name[0], name[1])
> +
> + def start_unix_server(self, path):
> + def cleanup():
> + os.unlink(path)
> +
> + cwd = os.getcwd()
> + try:
> + # Work around path length limits in AF_UNIX
> + os.chdir(os.path.dirname(path))
> + self.server = self.loop.run_until_complete(
> + asyncio.start_unix_server(self.handle_client,
> os.path.basename(path), loop=self.loop)
> + )
> + finally:
> + os.chdir(cwd)
> +
> + logger.info('Listening on %r' % path)
> +
> + self._cleanup_socket = cleanup
> + self.address = "unix://%s" % os.path.abspath(path)
> +
> + async def handle_client(self, reader, writer):
> + # writer.transport.set_write_buffer_limits(0)
> + try:
> + client = ServerClient(reader, writer, self.db,
> self.request_stats)
> + await client.process_requests()
> + except Exception as e:
> + import traceback
> + logger.error('Error from client: %s' % str(e),
> exc_info=True)
> + traceback.print_exc()
> + writer.close()
> + logger.info('Client disconnected')
> +
> + def serve_forever(self):
> + def signal_handler():
> + self.loop.stop()
> +
> + self.loop.add_signal_handler(signal.SIGTERM, signal_handler)
> +
> + try:
> + self.loop.run_forever()
> + except KeyboardInterrupt:
> + pass
> +
> + self.server.close()
> + self.loop.run_until_complete(self.server.wait_closed())
> + logger.info('Server shutting down')
> +
> + if self.close_loop:
> + self.loop.close()
> +
> + if self._cleanup_socket is not None:
> + self._cleanup_socket()
> diff --git a/bitbake/lib/hashserv/tests.py
> b/bitbake/lib/hashserv/tests.py new file mode 100644
> index 0000000..a5472a9
> --- /dev/null
> +++ b/bitbake/lib/hashserv/tests.py
> @@ -0,0 +1,142 @@
> +#! /usr/bin/env python3
> +#
> +# Copyright (C) 2018-2019 Garmin Ltd.
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> +from . import create_server, create_client
> +import hashlib
> +import logging
> +import multiprocessing
> +import sys
> +import tempfile
> +import threading
> +import unittest
> +
> +
> +class TestHashEquivalenceServer(object):
> + METHOD = 'TestMethod'
> +
> + def _run_server(self):
> + # logging.basicConfig(level=logging.DEBUG,
> filename='bbhashserv.log', filemode='w',
> + # format='%(levelname)s
> %(filename)s:%(lineno)d %(message)s')
> + self.server.serve_forever()
> +
> + def setUp(self):
> + if sys.version_info < (3, 5, 0):
> + self.skipTest('Python 3.5 or later required')
> +
> + self.temp_dir =
> tempfile.TemporaryDirectory(prefix='bb-hashserv')
> + self.dbfile = os.path.join(self.temp_dir.name, 'db.sqlite')
> +
> + self.server = create_server(self.get_server_addr(),
> self.dbfile)
> + self.server_thread =
> multiprocessing.Process(target=self._run_server)
> + self.server_thread.start()
> + self.client = create_client(self.server.address)
> +
> + def tearDown(self):
> + # Shutdown server
> + s = getattr(self, 'server', None)
> + if s is not None:
> + self.server_thread.terminate()
> + self.server_thread.join()
> + self.client.close()
> + self.temp_dir.cleanup()
> +
> + def test_create_hash(self):
> + # Simple test that hashes can be created
> + taskhash = '35788efcb8dfb0a02659d81cf2bfd695fb30faf9'
> + outhash =
> '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5970796f'
> + unihash = 'f46d3fbb439bd9b921095da657a4de906510d2cd'
> +
> + result = self.client.get_unihash(self.METHOD, taskhash)
> + self.assertIsNone(result, msg='Found unexpected task, %r' %
> result) +
> + result = self.client.report_unihash(taskhash, self.METHOD,
> outhash, unihash)
> + self.assertEqual(result['unihash'], unihash, 'Server
> returned bad unihash') +
> + def test_create_equivalent(self):
> + # Tests that a second reported task with the same outhash
> will be
> + # assigned the same unihash
> + taskhash = '53b8dce672cb6d0c73170be43f540460bfc347b4'
> + outhash =
> '5a9cb1649625f0bf41fc7791b635cd9c2d7118c7f021ba87dcd03f72b67ce7a8'
> + unihash = 'f37918cc02eb5a520b1aff86faacbc0a38124646'
> +
> + result = self.client.report_unihash(taskhash, self.METHOD,
> outhash, unihash)
> + self.assertEqual(result['unihash'], unihash, 'Server
> returned bad unihash') +
> + # Report a different task with the same outhash. The
> returned unihash
> + # should match the first task
> + taskhash2 = '3bf6f1e89d26205aec90da04854fbdbf73afe6b4'
> + unihash2 = 'af36b199320e611fbb16f1f277d3ee1d619ca58b'
> + result = self.client.report_unihash(taskhash2, self.METHOD,
> outhash, unihash2)
> + self.assertEqual(result['unihash'], unihash, 'Server
> returned bad unihash') +
> + def test_duplicate_taskhash(self):
> + # Tests that duplicate reports of the same taskhash with
> different
> + # outhash & unihash always return the unihash from the first
> reported
> + # taskhash
> + taskhash = '8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a'
> + outhash =
> 'afe240a439959ce86f5e322f8c208e1fedefea9e813f2140c81af866cc9edf7e'
> + unihash = '218e57509998197d570e2c98512d0105985dffc9'
> + self.client.report_unihash(taskhash, self.METHOD, outhash,
> unihash) +
> + result = self.client.get_unihash(self.METHOD, taskhash)
> + self.assertEqual(result, unihash)
> +
> + outhash2 =
> '0904a7fe3dc712d9fd8a74a616ddca2a825a8ee97adf0bd3fc86082c7639914d'
> + unihash2 = 'ae9a7d252735f0dafcdb10e2e02561ca3a47314c'
> + self.client.report_unihash(taskhash, self.METHOD, outhash2,
> unihash2) +
> + result = self.client.get_unihash(self.METHOD, taskhash)
> + self.assertEqual(result, unihash)
> +
> + outhash3 =
> '77623a549b5b1a31e3732dfa8fe61d7ce5d44b3370f253c5360e136b852967b4'
> + unihash3 = '9217a7d6398518e5dc002ed58f2cbbbc78696603'
> + self.client.report_unihash(taskhash, self.METHOD, outhash3,
> unihash3) +
> + result = self.client.get_unihash(self.METHOD, taskhash)
> + self.assertEqual(result, unihash)
> +
> + def test_stress(self):
> + def query_server(failures):
> + client = Client(self.server.address)
> + try:
> + for i in range(1000):
> + taskhash = hashlib.sha256()
> + taskhash.update(str(i).encode('utf-8'))
> + taskhash = taskhash.hexdigest()
> + result = client.get_unihash(self.METHOD,
> taskhash)
> + if result != taskhash:
> + failures.append("taskhash mismatch: %s !=
> %s" % (result, taskhash))
> + finally:
> + client.close()
> +
> + # Report hashes
> + for i in range(1000):
> + taskhash = hashlib.sha256()
> + taskhash.update(str(i).encode('utf-8'))
> + taskhash = taskhash.hexdigest()
> + self.client.report_unihash(taskhash, self.METHOD,
> taskhash, taskhash) +
> + failures = []
> + threads = [threading.Thread(target=query_server,
> args=(failures,)) for t in range(100)] +
> + for t in threads:
> + t.start()
> +
> + for t in threads:
> + t.join()
> +
> + self.assertFalse(failures)
> +
> +
> +class TestHashEquivalenceUnixServer(TestHashEquivalenceServer,
> unittest.TestCase):
> + def get_server_addr(self):
> + return "unix://" + os.path.join(self.temp_dir.name, 'sock')
> +
> +
> +class TestHashEquivalenceTCPServer(TestHashEquivalenceServer,
> unittest.TestCase):
> + def get_server_addr(self):
> + return "localhost:0"
> diff --git a/bitbake/lib/layerindexlib/__init__.py
> b/bitbake/lib/layerindexlib/__init__.py index cb79cb3..77196b4 100644
> --- a/bitbake/lib/layerindexlib/__init__.py
> +++ b/bitbake/lib/layerindexlib/__init__.py
> @@ -1,17 +1,7 @@
> # Copyright (C) 2016-2018 Wind River Systems, Inc.
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
> -# See the GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> -# along with this program; if not, write to the Free Software
> -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
> 02111-1307 USA
> import datetime
>
> @@ -386,7 +376,7 @@ layerBranches set. If not, they are effectively
> blank.''' invalid.append(name)
>
>
> - def _resolve_dependencies(layerbranches, ignores,
> dependencies, invalid):
> + def _resolve_dependencies(layerbranches, ignores,
> dependencies, invalid, processed=None): for layerbranch in
> layerbranches: if ignores and layerbranch.layer.name in ignores:
> continue
> @@ -398,6 +388,13 @@ layerBranches set. If not, they are effectively
> blank.''' if ignores and deplayerbranch.layer.name in ignores:
> continue
>
> + # Since this is depth first, we need to know
> what we're currently processing
> + # in order to avoid infinite recursion on a loop.
> + if processed and deplayerbranch.layer.name in
> processed:
> + # We have found a recursion...
> + logger.warning('Circular layer dependency
> found: %s -> %s' % (processed, deplayerbranch.layer.name))
> + continue
> +
> # This little block is why we can't re-use the
> LayerIndexObj version, # we must be able to satisfy each dependencies
> across layer indexes and # use the layer index order for priority.
> (r stands for replacement below) @@ -421,7 +418,17 @@ layerBranches
> set. If not, they are effectively blank.'''
> # New dependency, we need to resolve it now...
> depth-first if deplayerbranch.layer.name not in dependencies:
> - (dependencies, invalid) =
> _resolve_dependencies([deplayerbranch], ignores, dependencies,
> invalid)
> + # Avoid recursion on this branch.
> + # We copy so we don't end up polluting the
> depth-first branch with other
> + # branches. Duplication between individual
> branches IS expected and
> + # handled by 'dependencies' processing.
> + if not processed:
> + local_processed = []
> + else:
> + local_processed = processed.copy()
> +
> local_processed.append(deplayerbranch.layer.name) +
> + (dependencies, invalid) =
> _resolve_dependencies([deplayerbranch], ignores, dependencies,
> invalid, local_processed) if deplayerbranch.layer.name not in
> dependencies: dependencies[deplayerbranch.layer.name] =
> [deplayerbranch, layerdependency] diff --git
> a/bitbake/lib/layerindexlib/cooker.py
> b/bitbake/lib/layerindexlib/cooker.py index 848f0e2..604a961 100644
> --- a/bitbake/lib/layerindexlib/cooker.py +++
> b/bitbake/lib/layerindexlib/cooker.py @@ -1,17 +1,7 @@
> # Copyright (C) 2016-2018 Wind River Systems, Inc.
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
> -# See the GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> -# along with this program; if not, write to the Free Software
> -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
> 02111-1307 USA
> import logging
> import json
> diff --git a/bitbake/lib/layerindexlib/plugin.py
> b/bitbake/lib/layerindexlib/plugin.py index 92a2e97..7015a1a 100644
> --- a/bitbake/lib/layerindexlib/plugin.py
> +++ b/bitbake/lib/layerindexlib/plugin.py
> @@ -1,18 +1,7 @@
> # Copyright (C) 2016-2018 Wind River Systems, Inc.
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
> -# See the GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> -# along with this program; if not, write to the Free Software
> -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
> 02111-1307 USA -
> # The file contains:
> # LayerIndex exceptions
> # Plugin base class
> diff --git a/bitbake/lib/layerindexlib/restapi.py
> b/bitbake/lib/layerindexlib/restapi.py index d08eb20..21fd144 100644
> --- a/bitbake/lib/layerindexlib/restapi.py
> +++ b/bitbake/lib/layerindexlib/restapi.py
> @@ -1,17 +1,7 @@
> # Copyright (C) 2016-2018 Wind River Systems, Inc.
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
> -# See the GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> -# along with this program; if not, write to the Free Software
> -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
> 02111-1307 USA
> import logging
> import json
> diff --git a/bitbake/lib/layerindexlib/tests/common.py
> b/bitbake/lib/layerindexlib/tests/common.py index 22a5458..077382f
> 100644 --- a/bitbake/lib/layerindexlib/tests/common.py
> +++ b/bitbake/lib/layerindexlib/tests/common.py
> @@ -1,17 +1,7 @@
> # Copyright (C) 2017-2018 Wind River Systems, Inc.
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
> -# See the GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> -# along with this program; if not, write to the Free Software
> -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
> 02111-1307 USA
> import unittest
> import tempfile
> diff --git a/bitbake/lib/layerindexlib/tests/cooker.py
> b/bitbake/lib/layerindexlib/tests/cooker.py index fdbf091..1fa102e
> 100644 --- a/bitbake/lib/layerindexlib/tests/cooker.py
> +++ b/bitbake/lib/layerindexlib/tests/cooker.py
> @@ -1,17 +1,7 @@
> # Copyright (C) 2018 Wind River Systems, Inc.
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
> -# See the GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> -# along with this program; if not, write to the Free Software
> -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
> 02111-1307 USA
> import unittest
> import tempfile
> diff --git a/bitbake/lib/layerindexlib/tests/layerindexobj.py
> b/bitbake/lib/layerindexlib/tests/layerindexobj.py index
> e2fbb95..0c5ec88 100644 ---
> a/bitbake/lib/layerindexlib/tests/layerindexobj.py +++
> b/bitbake/lib/layerindexlib/tests/layerindexobj.py @@ -1,17 +1,7 @@
> # Copyright (C) 2017-2018 Wind River Systems, Inc.
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
> -# See the GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> -# along with this program; if not, write to the Free Software
> -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
> 02111-1307 USA
> import unittest
> import tempfile
> diff --git a/bitbake/lib/layerindexlib/tests/restapi.py
> b/bitbake/lib/layerindexlib/tests/restapi.py index 5876695..6d8dc00
> 100644 --- a/bitbake/lib/layerindexlib/tests/restapi.py
> +++ b/bitbake/lib/layerindexlib/tests/restapi.py
> @@ -1,17 +1,7 @@
> # Copyright (C) 2017-2018 Wind River Systems, Inc.
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
> -# See the GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> -# along with this program; if not, write to the Free Software
> -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
> 02111-1307 USA
> import unittest
> import tempfile
> diff --git a/bitbake/lib/progressbar/__init__.py
> b/bitbake/lib/progressbar/__init__.py index fbab744..c545a62 100644
> --- a/bitbake/lib/progressbar/__init__.py
> +++ b/bitbake/lib/progressbar/__init__.py
> @@ -4,6 +4,8 @@
> # progressbar - Text progress bar library for Python.
> # Copyright (c) 2005 Nilton Volpato
> #
> +# SPDX-License-Identifier: LGPL-2.1-or-later OR BSD-3-Clause-Clear
> +#
> # This library is free software; you can redistribute it and/or
> # modify it under the terms of the GNU Lesser General Public
> # License as published by the Free Software Foundation; either
> diff --git a/bitbake/lib/progressbar/compat.py
> b/bitbake/lib/progressbar/compat.py index a39f4a1..9804e0b 100644
> --- a/bitbake/lib/progressbar/compat.py
> +++ b/bitbake/lib/progressbar/compat.py
> @@ -3,6 +3,8 @@
> # progressbar - Text progress bar library for Python.
> # Copyright (c) 2005 Nilton Volpato
> #
> +# SPDX-License-Identifier: LGPL-2.1-or-later OR BSD-3-Clause-Clear
> +#
> # This library is free software; you can redistribute it and/or
> # modify it under the terms of the GNU Lesser General Public
> # License as published by the Free Software Foundation; either
> diff --git a/bitbake/lib/progressbar/progressbar.py
> b/bitbake/lib/progressbar/progressbar.py index 2873ad6..e2b6ba1 100644
> --- a/bitbake/lib/progressbar/progressbar.py
> +++ b/bitbake/lib/progressbar/progressbar.py
> @@ -5,6 +5,8 @@
> #
> # (With some small changes after importing into BitBake)
> #
> +# SPDX-License-Identifier: LGPL-2.1-or-later OR BSD-3-Clause-Clear
> +#
> # This library is free software; you can redistribute it and/or
> # modify it under the terms of the GNU Lesser General Public
> # License as published by the Free Software Foundation; either
> diff --git a/bitbake/lib/progressbar/widgets.py
> b/bitbake/lib/progressbar/widgets.py index 77285ca..0772aa5 100644
> --- a/bitbake/lib/progressbar/widgets.py
> +++ b/bitbake/lib/progressbar/widgets.py
> @@ -3,6 +3,8 @@
> # progressbar - Text progress bar library for Python.
> # Copyright (c) 2005 Nilton Volpato
> #
> +# SPDX-License-Identifier: LGPL-2.1-or-later OR BSD-3-Clause-Clear
> +#
> # This library is free software; you can redistribute it and/or
> # modify it under the terms of the GNU Lesser General Public
> # License as published by the Free Software Foundation; either
> diff --git a/bitbake/lib/prserv/__init__.py
> b/bitbake/lib/prserv/__init__.py index c3cb73a..9961040 100644
> --- a/bitbake/lib/prserv/__init__.py
> +++ b/bitbake/lib/prserv/__init__.py
> @@ -1,3 +1,7 @@
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> __version__ = "1.0.0"
>
> import os, time
> diff --git a/bitbake/lib/prserv/db.py b/bitbake/lib/prserv/db.py
> index 495d09f..117d8c0 100644
> --- a/bitbake/lib/prserv/db.py
> +++ b/bitbake/lib/prserv/db.py
> @@ -1,3 +1,7 @@
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> import logging
> import os.path
> import errno
> @@ -253,7 +257,7 @@ class PRData(object):
> self.connection=sqlite3.connect(self.filename,
> isolation_level="EXCLUSIVE", check_same_thread = False)
> self.connection.row_factory=sqlite3.Row
> self.connection.execute("pragma synchronous = off;")
> - self.connection.execute("PRAGMA journal_mode = WAL;")
> + self.connection.execute("PRAGMA journal_mode = MEMORY;")
> self._tables={}
>
> def disconnect(self):
> diff --git a/bitbake/lib/prserv/serv.py b/bitbake/lib/prserv/serv.py
> index 6a99728..be3acec 100644
> --- a/bitbake/lib/prserv/serv.py
> +++ b/bitbake/lib/prserv/serv.py
> @@ -1,3 +1,7 @@
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> import os,sys,logging
> import signal, time
> from xmlrpc.server import SimpleXMLRPCServer,
> SimpleXMLRPCRequestHandler diff --git a/bitbake/lib/pyinotify.py
> b/bitbake/lib/pyinotify.py index 4eb03b0..1528a22 100644
> --- a/bitbake/lib/pyinotify.py
> +++ b/bitbake/lib/pyinotify.py
> @@ -1,25 +1,9 @@
> -#!/usr/bin/env python
> -
> +#
> # pyinotify.py - python interface to inotify
> # Copyright (c) 2005-2015 Sebastien Martini <seb@dbzteam.org>
> #
> -# Permission is hereby granted, free of charge, to any person
> obtaining a copy -# of this software and associated documentation
> files (the "Software"), to deal -# in the Software without
> restriction, including without limitation the rights -# to use, copy,
> modify, merge, publish, distribute, sublicense, and/or sell -# copies
> of the Software, and to permit persons to whom the Software is -#
> furnished to do so, subject to the following conditions: -#
> -# The above copyright notice and this permission notice shall be
> included in -# all copies or substantial portions of the Software.
> +# SPDX-License-Identifier: MIT
> #
> -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
> EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
> MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND
> NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT
> HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY,
> WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -#
> OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
> DEALINGS IN -# THE SOFTWARE. """
> pyinotify
>
> diff --git a/bitbake/lib/toaster/bldcollector/admin.py
> b/bitbake/lib/toaster/bldcollector/admin.py index 1f2e07f..feaa888
> 100644 --- a/bitbake/lib/toaster/bldcollector/admin.py
> +++ b/bitbake/lib/toaster/bldcollector/admin.py
> @@ -1,3 +1,7 @@
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> from django.contrib import admin
> from orm.models import BitbakeVersion, Release, ToasterSetting,
> Layer_Version from django import forms
> diff --git a/bitbake/lib/toaster/bldcollector/urls.py
> b/bitbake/lib/toaster/bldcollector/urls.py index 888175d..8eb1e34
> 100644 --- a/bitbake/lib/toaster/bldcollector/urls.py
> +++ b/bitbake/lib/toaster/bldcollector/urls.py
> @@ -3,19 +3,8 @@
> #
> # Copyright (C) 2014-2017 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. -
>
> from django.conf.urls import include, url
>
> diff --git a/bitbake/lib/toaster/bldcollector/views.py
> b/bitbake/lib/toaster/bldcollector/views.py index f32fa4d..c708b41
> 100644 --- a/bitbake/lib/toaster/bldcollector/views.py
> +++ b/bitbake/lib/toaster/bldcollector/views.py
> @@ -3,18 +3,8 @@
> #
> # Copyright (C) 2014 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from django.views.decorators.cache import cache_control
> from django.core.urlresolvers import reverse
> diff --git a/bitbake/lib/toaster/bldcontrol/admin.py
> b/bitbake/lib/toaster/bldcontrol/admin.py index fcbe5f5..e85c30e
> 100644 --- a/bitbake/lib/toaster/bldcontrol/admin.py
> +++ b/bitbake/lib/toaster/bldcontrol/admin.py
> @@ -1,3 +1,7 @@
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> from django.contrib import admin
> from django.contrib.admin.filters import RelatedFieldListFilter
> from .models import BuildEnvironment
> diff --git a/bitbake/lib/toaster/bldcontrol/bbcontroller.py
> b/bitbake/lib/toaster/bldcontrol/bbcontroller.py index
> 5195600..301df18 100644 ---
> a/bitbake/lib/toaster/bldcontrol/bbcontroller.py +++
> b/bitbake/lib/toaster/bldcontrol/bbcontroller.py @@ -1,24 +1,10 @@
> #
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> -#
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2014 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. -
>
> import os
> import sys
> diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
> b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py index
> 9490635..39ea736 100644 ---
> a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py +++
> b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py @@ -1,24
> +1,10 @@ #
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> -#
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2014 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. -
>
> import os
> import sys
> diff --git
> a/bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py
> b/bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py
> index 14298d9..fe2c4dc 100644 ---
> a/bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py
> +++
> b/bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py
> @@ -1,3 +1,7 @@ +# +# SPDX-License-Identifier: GPL-2.0-only +#
> +
> from django.core.management.base import BaseCommand, CommandError
> from django.db import transaction
>
> diff --git
> a/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py
> b/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py
> index 6a55dd4..50ec409 100644 ---
> a/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py +++
> b/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py @@
> -1,3 +1,7 @@ +# +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> from django.core.management.base import BaseCommand
> from django.db import transaction
> from django.db.models import Q
> diff --git a/bitbake/lib/toaster/bldcontrol/models.py
> b/bitbake/lib/toaster/bldcontrol/models.py index 409614b..bcffcf5
> 100644 --- a/bitbake/lib/toaster/bldcontrol/models.py
> +++ b/bitbake/lib/toaster/bldcontrol/models.py
> @@ -1,3 +1,7 @@
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> from __future__ import unicode_literals
> from django.db import models
> from django.core.validators import MaxValueValidator,
> MinValueValidator diff --git
> a/bitbake/lib/toaster/bldcontrol/views.py
> b/bitbake/lib/toaster/bldcontrol/views.py index 60f00ef..286d88b
> 100644 --- a/bitbake/lib/toaster/bldcontrol/views.py +++
> b/bitbake/lib/toaster/bldcontrol/views.py @@ -1 +1,5 @@
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> # Create your views here.
> diff --git a/bitbake/lib/toaster/manage.py
> b/bitbake/lib/toaster/manage.py index 0c7ea50..ae32619 100755
> --- a/bitbake/lib/toaster/manage.py
> +++ b/bitbake/lib/toaster/manage.py
> @@ -1,4 +1,8 @@
> #!/usr/bin/env python3
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> import os
> import sys
>
> diff --git a/bitbake/lib/toaster/orm/fixtures/oe-core.xml
> b/bitbake/lib/toaster/orm/fixtures/oe-core.xml index fec93ab..a723f5a
> 100644 --- a/bitbake/lib/toaster/orm/fixtures/oe-core.xml
> +++ b/bitbake/lib/toaster/orm/fixtures/oe-core.xml
> @@ -8,9 +8,9 @@
>
> <!-- Bitbake versions which correspond to the metadata release -->
> <object model="orm.bitbakeversion" pk="1">
> - <field type="CharField" name="name">sumo</field>
> + <field type="CharField" name="name">warrior</field>
> <field type="CharField"
> name="giturl">git://git.openembedded.org/bitbake</field>
> - <field type="CharField" name="branch">1.38</field>
> + <field type="CharField" name="branch">1.42</field>
> </object>
> <object model="orm.bitbakeversion" pk="2">
> <field type="CharField" name="name">HEAD</field>
> @@ -23,18 +23,18 @@
> <field type="CharField" name="branch">master</field>
> </object>
> <object model="orm.bitbakeversion" pk="4">
> - <field type="CharField" name="name">thud</field>
> + <field type="CharField" name="name">zeus</field>
> <field type="CharField"
> name="giturl">git://git.openembedded.org/bitbake</field>
> - <field type="CharField" name="branch">1.40</field>
> + <field type="CharField" name="branch">1.44</field>
> </object>
>
> <!-- Releases available -->
> <object model="orm.release" pk="1">
> - <field type="CharField" name="name">sumo</field>
> - <field type="CharField" name="description">Openembedded
> Sumo</field>
> + <field type="CharField" name="name">warrior</field>
> + <field type="CharField" name="description">Openembedded
> Warrior</field> <field rel="ManyToOneRel" to="orm.bitbakeversion"
> name="bitbake_version">1</field>
> - <field type="CharField" name="branch_name">sumo</field>
> - <field type="TextField" name="helptext">Toaster will run your
> builds using the tip of the <a
> href=\"http://cgit.openembedded.org/openembedded-core/log/?h=sumo\">OpenEmbedded
> Sumo</a> branch.</field>
> + <field type="CharField" name="branch_name">warrior</field>
> + <field type="TextField" name="helptext">Toaster will run your
> builds using the tip of the <a
> href=\"http://cgit.openembedded.org/openembedded-core/log/?h=warrior\">OpenEmbedded
> Warrior</a> branch.</field> </object> <object
> model="orm.release" pk="2"> <field type="CharField"
> name="name">local</field> @@ -51,11 +51,11 @@ <field type="TextField"
> name="helptext">Toaster will run your builds using the tip of the
> <a
> href=\"http://cgit.openembedded.org/openembedded-core/log/\">OpenEmbedded
> master</a> branch.</field> </object> <object
> model="orm.release" pk="4">
> - <field type="CharField" name="name">thud</field>
> - <field type="CharField" name="description">Openembedded
> Rocko</field>
> - <field rel="ManyToOneRel" to="orm.bitbakeversion"
> name="bitbake_version">1</field>
> - <field type="CharField" name="branch_name">thud</field>
> - <field type="TextField" name="helptext">Toaster will run your
> builds using the tip of the <a
> href=\"http://cgit.openembedded.org/openembedded-core/log/?h=thud\">OpenEmbedded
> Thud</a> branch.</field>
> + <field type="CharField" name="name">zeus</field>
> + <field type="CharField" name="description">Openembedded
> Zeus</field>
> + <field rel="ManyToOneRel" to="orm.bitbakeversion"
> name="bitbake_version">4</field>
> + <field type="CharField" name="branch_name">zeus</field>
> + <field type="TextField" name="helptext">Toaster will run your
> builds using the tip of the <a
> href=\"http://cgit.openembedded.org/openembedded-core/log/?h=zeus\">OpenEmbedded
> Zeus</a> branch.</field> </object> <!-- Default layers for each
> release --> diff --git a/bitbake/lib/toaster/orm/fixtures/poky.xml
> b/bitbake/lib/toaster/orm/fixtures/poky.xml index fb9a771..7992383
> 100644 --- a/bitbake/lib/toaster/orm/fixtures/poky.xml
> +++ b/bitbake/lib/toaster/orm/fixtures/poky.xml
> @@ -8,9 +8,9 @@
>
> <!-- Bitbake versions which correspond to the metadata release -->
> <object model="orm.bitbakeversion" pk="1">
> - <field type="CharField" name="name">sumo</field>
> + <field type="CharField" name="name">warrior</field>
> <field type="CharField"
> name="giturl">git://git.yoctoproject.org/poky</field>
> - <field type="CharField" name="branch">sumo</field>
> + <field type="CharField" name="branch">warrior</field>
> <field type="CharField" name="dirpath">bitbake</field>
> </object>
> <object model="orm.bitbakeversion" pk="2">
> @@ -26,20 +26,20 @@
> <field type="CharField" name="dirpath">bitbake</field>
> </object>
> <object model="orm.bitbakeversion" pk="4">
> - <field type="CharField" name="name">thud</field>
> + <field type="CharField" name="name">zeus</field>
> <field type="CharField"
> name="giturl">git://git.yoctoproject.org/poky</field>
> - <field type="CharField" name="branch">thud</field>
> + <field type="CharField" name="branch">zeus</field>
> <field type="CharField" name="dirpath">bitbake</field>
> </object>
>
>
> <!-- Releases available -->
> <object model="orm.release" pk="1">
> - <field type="CharField" name="name">sumo</field>
> - <field type="CharField" name="description">Yocto Project 2.5
> "Sumo"</field>
> + <field type="CharField" name="name">warrior</field>
> + <field type="CharField" name="description">Yocto Project 2.7
> "Warrior"</field> <field rel="ManyToOneRel" to="orm.bitbakeversion"
> name="bitbake_version">1</field>
> - <field type="CharField" name="branch_name">sumo</field>
> - <field type="TextField" name="helptext">Toaster will run your
> builds using the tip of the <a
> href="http://git.yoctoproject.org/cgit/cgit.cgi/poky/log/?h=sumo">Yocto
> Project Sumo branch</a>.</field>
> + <field type="CharField" name="branch_name">warrior</field>
> + <field type="TextField" name="helptext">Toaster will run your
> builds using the tip of the <a
> href="http://git.yoctoproject.org/cgit/cgit.cgi/poky/log/?h=warrior">Yocto
> Project Warrior branch</a>.</field> </object> <object
> model="orm.release" pk="2"> <field type="CharField"
> name="name">local</field> @@ -56,11 +56,11 @@ <field type="TextField"
> name="helptext">Toaster will run your builds using the tip of the
> <a
> href="http://git.yoctoproject.org/cgit/cgit.cgi/poky/log/">Yocto
> Project Master branch</a>.</field> </object> <object
> model="orm.release" pk="4">
> - <field type="CharField" name="name">rocko</field>
> - <field type="CharField" name="description">Yocto Project 2.6
> "Thud"</field>
> - <field rel="ManyToOneRel" to="orm.bitbakeversion"
> name="bitbake_version">1</field>
> - <field type="CharField" name="branch_name">thud</field>
> - <field type="TextField" name="helptext">Toaster will run your
> builds using the tip of the <a
> href="http://git.yoctoproject.org/cgit/cgit.cgi/poky/log/?h=thud">Yocto
> Project Thud branch</a>.</field>
> + <field type="CharField" name="name">zeus</field>
> + <field type="CharField" name="description">Yocto Project 3.0
> "Zeus"</field>
> + <field rel="ManyToOneRel" to="orm.bitbakeversion"
> name="bitbake_version">4</field>
> + <field type="CharField" name="branch_name">zeus</field>
> + <field type="TextField" name="helptext">Toaster will run your
> builds using the tip of the <a
> href="http://git.yoctoproject.org/cgit/cgit.cgi/poky/log/?h=zeus">Yocto
> Project Zeus branch</a>.</field> </object> <!-- Default project
> layers for each release --> @@ -130,7 +130,7 @@
> <field rel="ManyToOneRel" to="orm.layer" name="layer">1</field>
> <field type="IntegerField" name="layer_source">0</field>
> <field rel="ManyToOneRel" to="orm.release"
> name="release">1</field>
> - <field type="CharField" name="branch">sumo</field>
> + <field type="CharField" name="branch">warrior</field>
> <field type="CharField" name="dirpath">meta</field>
> </object>
> <object model="orm.layer_version" pk="2">
> @@ -152,7 +152,7 @@
> <field rel="ManyToOneRel" to="orm.layer" name="layer">1</field>
> <field type="IntegerField" name="layer_source">0</field>
> <field rel="ManyToOneRel" to="orm.release"
> name="release">4</field>
> - <field type="CharField" name="branch">rocko</field>
> + <field type="CharField" name="branch">zeus</field>
> <field type="CharField" name="dirpath">meta</field>
> </object>
>
> @@ -168,7 +168,7 @@
> <field rel="ManyToOneRel" to="orm.layer" name="layer">2</field>
> <field type="IntegerField" name="layer_source">0</field>
> <field rel="ManyToOneRel" to="orm.release"
> name="release">1</field>
> - <field type="CharField" name="branch">sumo</field>
> + <field type="CharField" name="branch">warrior</field>
> <field type="CharField" name="dirpath">meta-poky</field>
> </object>
> <object model="orm.layer_version" pk="6">
> @@ -190,7 +190,7 @@
> <field rel="ManyToOneRel" to="orm.layer" name="layer">2</field>
> <field type="IntegerField" name="layer_source">0</field>
> <field rel="ManyToOneRel" to="orm.release"
> name="release">4</field>
> - <field type="CharField" name="branch">rocko</field>
> + <field type="CharField" name="branch">zeus</field>
> <field type="CharField" name="dirpath">meta-poky</field>
> </object>
>
> @@ -206,7 +206,7 @@
> <field rel="ManyToOneRel" to="orm.layer" name="layer">3</field>
> <field type="IntegerField" name="layer_source">0</field>
> <field rel="ManyToOneRel" to="orm.release"
> name="release">1</field>
> - <field type="CharField" name="branch">sumo</field>
> + <field type="CharField" name="branch">warrior</field>
> <field type="CharField" name="dirpath">meta-yocto-bsp</field>
> </object>
> <object model="orm.layer_version" pk="10">
> @@ -228,7 +228,7 @@
> <field rel="ManyToOneRel" to="orm.layer" name="layer">3</field>
> <field type="IntegerField" name="layer_source">0</field>
> <field rel="ManyToOneRel" to="orm.release"
> name="release">4</field>
> - <field type="CharField" name="branch">rocko</field>
> + <field type="CharField" name="branch">zeus</field>
> <field type="CharField" name="dirpath">meta-yocto-bsp</field>
> </object>
> </django-objects>
> diff --git a/bitbake/lib/toaster/orm/management/commands/lsupdates.py
> b/bitbake/lib/toaster/orm/management/commands/lsupdates.py index
> 66114ff..5b5abbb 100644 ---
> a/bitbake/lib/toaster/orm/management/commands/lsupdates.py +++
> b/bitbake/lib/toaster/orm/management/commands/lsupdates.py @@ -1,23
> +1,10 @@ #
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> -#
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2016-2017 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from django.core.management.base import BaseCommand
>
> diff --git a/bitbake/lib/toaster/orm/models.py
> b/bitbake/lib/toaster/orm/models.py index 7720290..bb6b5de 100644
> --- a/bitbake/lib/toaster/orm/models.py
> +++ b/bitbake/lib/toaster/orm/models.py
> @@ -1,23 +1,10 @@
> #
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> -#
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from __future__ import unicode_literals
>
> @@ -978,12 +965,12 @@ class TargetSDKFile(models.Model):
> class Target_Image_File(models.Model):
> # valid suffixes for image files produced by a build
> SUFFIXES = {
> - 'btrfs', 'cpio', 'cpio.gz', 'cpio.lz4', 'cpio.lzma',
> 'cpio.xz',
> - 'cramfs', 'elf', 'ext2', 'ext2.bz2', 'ext2.gz', 'ext2.lzma',
> 'ext4',
> - 'ext4.gz', 'ext3', 'ext3.gz', 'hdddirect', 'hddimg', 'iso',
> 'jffs2',
> - 'jffs2.sum', 'multiubi', 'qcow2', 'squashfs', 'squashfs-lzo',
> + 'btrfs', 'container', 'cpio', 'cpio.gz', 'cpio.lz4',
> 'cpio.lzma',
> + 'cpio.xz', 'cramfs', 'ext2', 'ext2.bz2', 'ext2.gz',
> 'ext2.lzma',
> + 'ext3', 'ext3.gz', 'ext4', 'ext4.gz', 'f2fs', 'hddimg',
> 'iso', 'jffs2',
> + 'jffs2.sum', 'multiubi', 'squashfs', 'squashfs-lz4',
> 'squashfs-lzo', 'squashfs-xz', 'tar', 'tar.bz2', 'tar.gz', 'tar.lz4',
> 'tar.xz', 'ubi',
> - 'ubifs', 'vdi', 'vmdk', 'wic', 'wic.bmap', 'wic.bz2',
> 'wic.gz', 'wic.lzma'
> + 'ubifs', 'wic', 'wic.bz2', 'wic.gz', 'wic.lzma'
> }
>
> target = models.ForeignKey(Target)
> diff --git a/bitbake/lib/toaster/tests/browser/selenium_helpers.py
> b/bitbake/lib/toaster/tests/browser/selenium_helpers.py index
> 08711e4..02d4f4b 100644 ---
> a/bitbake/lib/toaster/tests/browser/selenium_helpers.py +++
> b/bitbake/lib/toaster/tests/browser/selenium_helpers.py @@ -1,23
> +1,10 @@ -#! /usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013-2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
> # The Wait class and some of SeleniumDriverHelper and
> SeleniumTestCase are # modified from Patchwork, released under the
> same licence terms as Toaster: diff --git
> a/bitbake/lib/toaster/tests/browser/selenium_helpers_base.py
> b/bitbake/lib/toaster/tests/browser/selenium_helpers_base.py index
> 156d639..6c94684 100644 ---
> a/bitbake/lib/toaster/tests/browser/selenium_helpers_base.py +++
> b/bitbake/lib/toaster/tests/browser/selenium_helpers_base.py @@ -1,23
> +1,10 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013-2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
> # The Wait class and some of SeleniumDriverHelper and
> SeleniumTestCase are # modified from Patchwork, released under the
> same licence terms as Toaster: diff --git
> a/bitbake/lib/toaster/tests/browser/test_all_builds_page.py
> b/bitbake/lib/toaster/tests/browser/test_all_builds_page.py index
> b86f29b..fba627b 100644 ---
> a/bitbake/lib/toaster/tests/browser/test_all_builds_page.py +++
> b/bitbake/lib/toaster/tests/browser/test_all_builds_page.py @@ -1,23
> +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013-2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import re
>
> diff --git
> a/bitbake/lib/toaster/tests/browser/test_all_projects_page.py
> b/bitbake/lib/toaster/tests/browser/test_all_projects_page.py index
> 44da640..afd2d35 100644 ---
> a/bitbake/lib/toaster/tests/browser/test_all_projects_page.py +++
> b/bitbake/lib/toaster/tests/browser/test_all_projects_page.py @@
> -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013-2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import re
>
> diff --git
> a/bitbake/lib/toaster/tests/browser/test_builddashboard_page.py
> b/bitbake/lib/toaster/tests/browser/test_builddashboard_page.py index
> f8ccb54..d972aff 100644 ---
> a/bitbake/lib/toaster/tests/browser/test_builddashboard_page.py +++
> b/bitbake/lib/toaster/tests/browser/test_builddashboard_page.py @@
> -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013-2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from django.core.urlresolvers import reverse
> from django.utils import timezone
> diff --git
> a/bitbake/lib/toaster/tests/browser/test_builddashboard_page_artifacts.py
> b/bitbake/lib/toaster/tests/browser/test_builddashboard_page_artifacts.py
> index 1c627ad..e2623e8 100644 ---
> a/bitbake/lib/toaster/tests/browser/test_builddashboard_page_artifacts.py
> +++
> b/bitbake/lib/toaster/tests/browser/test_builddashboard_page_artifacts.py
> @@ -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et -#
> -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3 #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013-2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from django.core.urlresolvers import reverse
> from django.utils import timezone
> diff --git
> a/bitbake/lib/toaster/tests/browser/test_builddashboard_page_recipes.py
> b/bitbake/lib/toaster/tests/browser/test_builddashboard_page_recipes.py
> index ed18324..c542d45 100644 ---
> a/bitbake/lib/toaster/tests/browser/test_builddashboard_page_recipes.py
> +++
> b/bitbake/lib/toaster/tests/browser/test_builddashboard_page_recipes.py
> @@ -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et -#
> -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3 #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013-2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from django.core.urlresolvers import reverse
> from django.utils import timezone
> diff --git
> a/bitbake/lib/toaster/tests/browser/test_builddashboard_page_tasks.py
> b/bitbake/lib/toaster/tests/browser/test_builddashboard_page_tasks.py
> index da50f16..22acb47 100644 ---
> a/bitbake/lib/toaster/tests/browser/test_builddashboard_page_tasks.py
> +++
> b/bitbake/lib/toaster/tests/browser/test_builddashboard_page_tasks.py
> @@ -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et -#
> -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3 #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013-2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from django.core.urlresolvers import reverse
> from django.utils import timezone
> diff --git a/bitbake/lib/toaster/tests/browser/test_js_unit_tests.py
> b/bitbake/lib/toaster/tests/browser/test_js_unit_tests.py index
> 3c0b962..e8b4295 100644 ---
> a/bitbake/lib/toaster/tests/browser/test_js_unit_tests.py +++
> b/bitbake/lib/toaster/tests/browser/test_js_unit_tests.py @@ -1,23
> +1,11 @@ -#! /usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013-2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> """
> Run the js unit tests
> diff --git a/bitbake/lib/toaster/tests/browser/test_landing_page.py
> b/bitbake/lib/toaster/tests/browser/test_landing_page.py index
> 4d4cd66..0790198 100644 ---
> a/bitbake/lib/toaster/tests/browser/test_landing_page.py +++
> b/bitbake/lib/toaster/tests/browser/test_landing_page.py @@ -1,23
> +1,11 @@ -#! /usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> -# Copyright (C) 2013-2016 Intel Corporation
> -#
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# Copyright (C) 2013-2016 Intel Corporation
> #
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from django.core.urlresolvers import reverse
> from django.utils import timezone
> diff --git
> a/bitbake/lib/toaster/tests/browser/test_layerdetails_page.py
> b/bitbake/lib/toaster/tests/browser/test_layerdetails_page.py index
> f24fb09..f81e696 100644 ---
> a/bitbake/lib/toaster/tests/browser/test_layerdetails_page.py +++
> b/bitbake/lib/toaster/tests/browser/test_layerdetails_page.py @@
> -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> -# Copyright (C) 2013-2016 Intel Corporation
> -#
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# Copyright (C) 2013-2016 Intel Corporation
> #
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from django.core.urlresolvers import reverse
> from tests.browser.selenium_helpers import SeleniumTestCase
> diff --git
> a/bitbake/lib/toaster/tests/browser/test_most_recent_builds_states.py
> b/bitbake/lib/toaster/tests/browser/test_most_recent_builds_states.py
> index abc0b0b..15d25dc 100644 ---
> a/bitbake/lib/toaster/tests/browser/test_most_recent_builds_states.py
> +++
> b/bitbake/lib/toaster/tests/browser/test_most_recent_builds_states.py
> @@ -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et -#
> -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3 #
> # BitBake Toaster Implementation
> #
> -# Copyright (C) 2013-2016 Intel Corporation
> -#
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# Copyright (C) 2013-2016 Intel Corporation
> #
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from django.core.urlresolvers import reverse
> from django.utils import timezone
> diff --git
> a/bitbake/lib/toaster/tests/browser/test_new_custom_image_page.py
> b/bitbake/lib/toaster/tests/browser/test_new_custom_image_page.py
> index ab5a8e6..0aa3b7a 100644 ---
> a/bitbake/lib/toaster/tests/browser/test_new_custom_image_page.py +++
> b/bitbake/lib/toaster/tests/browser/test_new_custom_image_page.py @@
> -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013-2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from django.core.urlresolvers import reverse
> from tests.browser.selenium_helpers import SeleniumTestCase
> diff --git
> a/bitbake/lib/toaster/tests/browser/test_new_project_page.py
> b/bitbake/lib/toaster/tests/browser/test_new_project_page.py index
> 77e5f15..8e56bb0 100644 ---
> a/bitbake/lib/toaster/tests/browser/test_new_project_page.py +++
> b/bitbake/lib/toaster/tests/browser/test_new_project_page.py @@ -1,23
> +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013-2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from django.core.urlresolvers import reverse
> from tests.browser.selenium_helpers import SeleniumTestCase
> diff --git
> a/bitbake/lib/toaster/tests/browser/test_project_builds_page.py
> b/bitbake/lib/toaster/tests/browser/test_project_builds_page.py index
> 9fe91ab..47fb10b 100644 ---
> a/bitbake/lib/toaster/tests/browser/test_project_builds_page.py +++
> b/bitbake/lib/toaster/tests/browser/test_project_builds_page.py @@
> -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013-2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import re
>
> diff --git
> a/bitbake/lib/toaster/tests/browser/test_project_config_page.py
> b/bitbake/lib/toaster/tests/browser/test_project_config_page.py index
> 0710084..2816eb9 100644 ---
> a/bitbake/lib/toaster/tests/browser/test_project_config_page.py +++
> b/bitbake/lib/toaster/tests/browser/test_project_config_page.py @@
> -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013-2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import re
>
> diff --git a/bitbake/lib/toaster/tests/browser/test_project_page.py
> b/bitbake/lib/toaster/tests/browser/test_project_page.py index
> 0186463..8b5e1b6 100644 ---
> a/bitbake/lib/toaster/tests/browser/test_project_page.py +++
> b/bitbake/lib/toaster/tests/browser/test_project_page.py @@ -1,23
> +1,11 @@ -#! /usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013-2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from django.core.urlresolvers import reverse
> from django.utils import timezone
> diff --git a/bitbake/lib/toaster/tests/browser/test_sample.py
> b/bitbake/lib/toaster/tests/browser/test_sample.py index
> 20ec53c..f4ad670 100644 ---
> a/bitbake/lib/toaster/tests/browser/test_sample.py +++
> b/bitbake/lib/toaster/tests/browser/test_sample.py @@ -1,23 +1,11 @@
> -#! /usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013-2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> """
> A small example test demonstrating the basics of writing a test with
> diff --git a/bitbake/lib/toaster/tests/browser/test_task_page.py
> b/bitbake/lib/toaster/tests/browser/test_task_page.py index
> 690d116..26f3dca 100644 ---
> a/bitbake/lib/toaster/tests/browser/test_task_page.py +++
> b/bitbake/lib/toaster/tests/browser/test_task_page.py @@ -1,23 +1,11
> @@ -#! /usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013-2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from django.core.urlresolvers import reverse
> from django.utils import timezone
> diff --git
> a/bitbake/lib/toaster/tests/browser/test_toastertable_ui.py
> b/bitbake/lib/toaster/tests/browser/test_toastertable_ui.py index
> 53ddf30..ef78cbb 100644 ---
> a/bitbake/lib/toaster/tests/browser/test_toastertable_ui.py +++
> b/bitbake/lib/toaster/tests/browser/test_toastertable_ui.py @@ -1,23
> +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013-2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from datetime import datetime
>
> diff --git a/bitbake/lib/toaster/tests/builds/buildtest.py
> b/bitbake/lib/toaster/tests/builds/buildtest.py index
> 5a56a11..872bbd3 100644 ---
> a/bitbake/lib/toaster/tests/builds/buildtest.py +++
> b/bitbake/lib/toaster/tests/builds/buildtest.py @@ -1,23 +1,11 @@
> -#! /usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import os
> import sys
> diff --git a/bitbake/lib/toaster/tests/builds/test_core_image_min.py
> b/bitbake/lib/toaster/tests/builds/test_core_image_min.py index
> 586f4a8..44b6cbe 100644 ---
> a/bitbake/lib/toaster/tests/builds/test_core_image_min.py +++
> b/bitbake/lib/toaster/tests/builds/test_core_image_min.py @@ -1,24
> +1,11 @@ -#! /usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. -
> # Tests were part of openembedded-core oe selftest Authored by:
> Lucian Musat # Ionut Chisanovici, Paul Eggleton and Cristian Iorga
>
> diff --git a/bitbake/lib/toaster/tests/commands/test_loaddata.py
> b/bitbake/lib/toaster/tests/commands/test_loaddata.py index
> 951f6ff..9e8d555 100644 ---
> a/bitbake/lib/toaster/tests/commands/test_loaddata.py +++
> b/bitbake/lib/toaster/tests/commands/test_loaddata.py @@ -1,23 +1,11
> @@ -#! /usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from django.test import TestCase
> from django.core import management
> diff --git a/bitbake/lib/toaster/tests/commands/test_lsupdates.py
> b/bitbake/lib/toaster/tests/commands/test_lsupdates.py index
> 49897a4..3c4fbe0 100644 ---
> a/bitbake/lib/toaster/tests/commands/test_lsupdates.py +++
> b/bitbake/lib/toaster/tests/commands/test_lsupdates.py @@ -1,23 +1,11
> @@ -#! /usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from django.test import TestCase
> from django.core import management
> diff --git a/bitbake/lib/toaster/tests/commands/test_runbuilds.py
> b/bitbake/lib/toaster/tests/commands/test_runbuilds.py index
> 3e63483..e223b95 100644 ---
> a/bitbake/lib/toaster/tests/commands/test_runbuilds.py +++
> b/bitbake/lib/toaster/tests/commands/test_runbuilds.py @@ -1,23 +1,11
> @@ -#! /usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import os
>
> diff --git a/bitbake/lib/toaster/tests/db/test_db.py
> b/bitbake/lib/toaster/tests/db/test_db.py index a0f5f6e..0410422
> 100644 --- a/bitbake/lib/toaster/tests/db/test_db.py
> +++ b/bitbake/lib/toaster/tests/db/test_db.py
> @@ -2,6 +2,8 @@
> #
> # Copyright (c) 2016 Damien Lespiau
> #
> +# SPDX-License-Identifier: MIT
> +#
> # Permission is hereby granted, free of charge, to any person
> obtaining a copy # of this software and associated documentation
> files (the "Software"), to deal # in the Software without
> restriction, including without limitation the rights diff --git
> a/bitbake/lib/toaster/tests/eventreplay/__init__.py
> b/bitbake/lib/toaster/tests/eventreplay/__init__.py index
> 6956619..8ed6792 100644 ---
> a/bitbake/lib/toaster/tests/eventreplay/__init__.py +++
> b/bitbake/lib/toaster/tests/eventreplay/__init__.py @@ -1,23 +1,11 @@
> -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> # Tests were part of openembedded-core oe selftest Authored by:
> Lucian Musat # Ionut Chisanovici, Paul Eggleton and Cristian Iorga
> diff --git
> a/bitbake/lib/toaster/tests/functional/functional_helpers.py
> b/bitbake/lib/toaster/tests/functional/functional_helpers.py index
> 486078a..455c408 100644 ---
> a/bitbake/lib/toaster/tests/functional/functional_helpers.py +++
> b/bitbake/lib/toaster/tests/functional/functional_helpers.py @@ -1,23
> +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster functional tests implementation
> #
> # Copyright (C) 2017 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import os
> import logging
> diff --git
> a/bitbake/lib/toaster/tests/functional/test_functional_basic.py
> b/bitbake/lib/toaster/tests/functional/test_functional_basic.py index
> cfa2b0f..56c84fb 100644 ---
> a/bitbake/lib/toaster/tests/functional/test_functional_basic.py +++
> b/bitbake/lib/toaster/tests/functional/test_functional_basic.py @@
> -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster functional tests implementation
> #
> # Copyright (C) 2017 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import time
> import re
> diff --git a/bitbake/lib/toaster/tests/views/test_views.py
> b/bitbake/lib/toaster/tests/views/test_views.py index
> 1463077..68d9e9d 100644 ---
> a/bitbake/lib/toaster/tests/views/test_views.py +++
> b/bitbake/lib/toaster/tests/views/test_views.py @@ -1,23 +1,11 @@
> -#! /usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013-2015 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> """Test cases for Toaster GUI and ReST."""
>
> diff --git a/bitbake/lib/toaster/toastergui/api.py
> b/bitbake/lib/toaster/toastergui/api.py index 564d595..8b49b3e 100644
> --- a/bitbake/lib/toaster/toastergui/api.py
> +++ b/bitbake/lib/toaster/toastergui/api.py
> @@ -3,19 +3,8 @@
> #
> # Copyright (C) 2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. -
> # Please run flake8 on this file before sending patches
>
> import os
> diff --git a/bitbake/lib/toaster/toastergui/buildtables.py
> b/bitbake/lib/toaster/toastergui/buildtables.py index
> 755a7c2..327059d 100644 ---
> a/bitbake/lib/toaster/toastergui/buildtables.py +++
> b/bitbake/lib/toaster/toastergui/buildtables.py @@ -1,23 +1,10 @@
> #
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> -#
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from orm.models import Build, Task, Target, Package
> from django.db.models import Q, Sum
> diff --git a/bitbake/lib/toaster/toastergui/static/js/importlayer.js
> b/bitbake/lib/toaster/toastergui/static/js/importlayer.js index
> 2964839..8e2032d 100644 ---
> a/bitbake/lib/toaster/toastergui/static/js/importlayer.js +++
> b/bitbake/lib/toaster/toastergui/static/js/importlayer.js @@ -17,11
> +17,15 @@ function importLayerPageInit (ctx) { var
> currentLayerDepSelection; var validLayerName = /^(\w|-)+$/;
>
> + /* Catch 'disable' race condition between type-ahead started and
> "input change" */
> + var typeAheadStarted = 0;
> +
> libtoaster.makeTypeahead(layerDepInput,
> libtoaster.ctx.layersTypeAheadUrl,
> { include_added: "true" }, function(item){
> currentLayerDepSelection = item;
> layerDepBtn.removeAttr("disabled");
> + typeAheadStarted = 1;
> });
>
> layerDepInput.on("typeahead:select", function(event, data){
> @@ -34,7 +38,10 @@ function importLayerPageInit (ctx) {
> // disable the "Add layer" button when the layer input typeahead
> is empty // or not in the typeahead choices
> layerDepInput.on("input change", function(){
> - layerDepBtn.attr("disabled","disabled");
> + if (0 == typeAheadStarted) {
> + layerDepBtn.attr("disabled","disabled");
> + }
> + typeAheadStarted = 0;
> });
>
> /* We automatically add "openembedded-core" layer for convenience
> as a @@ -50,6 +57,7 @@ function importLayerPageInit (ctx) {
> });
>
> layerDepBtn.click(function(){
> + typeAheadStarted = 0;
> if (currentLayerDepSelection == undefined)
> return;
>
> @@ -77,7 +85,7 @@ function importLayerPageInit (ctx) {
>
> $("#layer-deps-list").append(newLayerDep);
>
> -
> libtoaster.getLayerDepsForProject(currentLayerDepSelection.layerdetailurl,
> +
> libtoaster.getLayerDepsForProject(currentLayerDepSelection.xhrLayerUrl,
> function (data){ /* These are the dependencies of the layer added as
> a dependency */ if (data.list.length > 0) {
> diff --git a/bitbake/lib/toaster/toastergui/tablefilter.py
> b/bitbake/lib/toaster/toastergui/tablefilter.py index
> 65454e1..ffef795 100644 ---
> a/bitbake/lib/toaster/toastergui/tablefilter.py +++
> b/bitbake/lib/toaster/toastergui/tablefilter.py @@ -1,23 +1,10 @@
> #
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> -#
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2015 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from django.db.models import Q, Max, Min
> from django.utils import dateparse, timezone
> diff --git a/bitbake/lib/toaster/toastergui/tables.py
> b/bitbake/lib/toaster/toastergui/tables.py index 9ff756b..b3ea222
> 100644 --- a/bitbake/lib/toaster/toastergui/tables.py
> +++ b/bitbake/lib/toaster/toastergui/tables.py
> @@ -1,23 +1,10 @@
> #
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> -#
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2015 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from toastergui.widgets import ToasterTable
> from orm.models import Recipe, ProjectLayer, Layer_Version, Machine,
> Project diff --git
> a/bitbake/lib/toaster/toastergui/templatetags/field_values_filter.py
> b/bitbake/lib/toaster/toastergui/templatetags/field_values_filter.py
> index 5a73af7..eb48339 100644 ---
> a/bitbake/lib/toaster/toastergui/templatetags/field_values_filter.py
> +++
> b/bitbake/lib/toaster/toastergui/templatetags/field_values_filter.py
> @@ -1,3 +1,7 @@ +# +# SPDX-License-Identifier: GPL-2.0-only +#
> +
> from django import template
>
> register = template.Library()
> diff --git
> a/bitbake/lib/toaster/toastergui/templatetags/objects_to_dictionaries_filter.py
> b/bitbake/lib/toaster/toastergui/templatetags/objects_to_dictionaries_filter.py
> index 0dcc7d2..048d533 100644 ---
> a/bitbake/lib/toaster/toastergui/templatetags/objects_to_dictionaries_filter.py
> +++
> b/bitbake/lib/toaster/toastergui/templatetags/objects_to_dictionaries_filter.py
> @@ -1,3 +1,7 @@ +# +# SPDX-License-Identifier: GPL-2.0-only +#
> +
> from django import template
> import json
>
> diff --git
> a/bitbake/lib/toaster/toastergui/templatetags/project_url_tag.py
> b/bitbake/lib/toaster/toastergui/templatetags/project_url_tag.py
> index 04770ac..71e0925 100644 ---
> a/bitbake/lib/toaster/toastergui/templatetags/project_url_tag.py +++
> b/bitbake/lib/toaster/toastergui/templatetags/project_url_tag.py @@
> -1,3 +1,7 @@ +# +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> from django import template
> from django.core.urlresolvers import reverse
>
> diff --git
> a/bitbake/lib/toaster/toastergui/templatetags/projecttags.py
> b/bitbake/lib/toaster/toastergui/templatetags/projecttags.py index
> b170a16..1dbab3b 100644 ---
> a/bitbake/lib/toaster/toastergui/templatetags/projecttags.py +++
> b/bitbake/lib/toaster/toastergui/templatetags/projecttags.py @@ -1,23
> +1,10 @@ # -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> -#
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from datetime import datetime, timedelta
> from os.path import relpath
> diff --git a/bitbake/lib/toaster/toastergui/typeaheads.py
> b/bitbake/lib/toaster/toastergui/typeaheads.py index 5aa0f8d..fd750ff
> 100644 --- a/bitbake/lib/toaster/toastergui/typeaheads.py
> +++ b/bitbake/lib/toaster/toastergui/typeaheads.py
> @@ -3,18 +3,8 @@
> #
> # Copyright (C) 2015 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import subprocess
>
> diff --git a/bitbake/lib/toaster/toastergui/urls.py
> b/bitbake/lib/toaster/toastergui/urls.py index dc03e30..673d9ae 100644
> --- a/bitbake/lib/toaster/toastergui/urls.py
> +++ b/bitbake/lib/toaster/toastergui/urls.py
> @@ -3,18 +3,8 @@
> #
> # Copyright (C) 2013-2017 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from django.conf.urls import include, url
> from django.views.generic import RedirectView, TemplateView
> diff --git a/bitbake/lib/toaster/toastergui/views.py
> b/bitbake/lib/toaster/toastergui/views.py index c712b06..d7acaff
> 100644 --- a/bitbake/lib/toaster/toastergui/views.py
> +++ b/bitbake/lib/toaster/toastergui/views.py
> @@ -1,24 +1,10 @@
> #
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> -#
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. -
>
> import re
>
> diff --git a/bitbake/lib/toaster/toastergui/widgets.py
> b/bitbake/lib/toaster/toastergui/widgets.py index db5c3aa..645f458
> 100644 --- a/bitbake/lib/toaster/toastergui/widgets.py
> +++ b/bitbake/lib/toaster/toastergui/widgets.py
> @@ -1,23 +1,10 @@
> #
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> -#
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2015 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from django.views.generic import View, TemplateView
> from django.views.decorators.cache import cache_control
> diff --git
> a/bitbake/lib/toaster/toastermain/management/commands/builddelete.py
> b/bitbake/lib/toaster/toastermain/management/commands/builddelete.py
> index bf69a8f..c2d773a 100644 ---
> a/bitbake/lib/toaster/toastermain/management/commands/builddelete.py
> +++
> b/bitbake/lib/toaster/toastermain/management/commands/builddelete.py
> @@ -1,3 +1,7 @@ +# +# SPDX-License-Identifier: GPL-2.0-only +#
> +
> from django.core.management.base import BaseCommand, CommandError
> from django.core.exceptions import ObjectDoesNotExist
> from orm.models import Build
> diff --git
> a/bitbake/lib/toaster/toastermain/management/commands/buildimport.py
> b/bitbake/lib/toaster/toastermain/management/commands/buildimport.py
> index 2d57ab5..408ad44 100644 ---
> a/bitbake/lib/toaster/toastermain/management/commands/buildimport.py
> +++
> b/bitbake/lib/toaster/toastermain/management/commands/buildimport.py
> @@ -1,23 +1,10 @@ # -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4;
> c-basic-offset: 4; indent-tabs-mode: nil -*- -#
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2018 Wind River Systems
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> # buildimport: import a project for project specific configuration
> #
> diff --git
> a/bitbake/lib/toaster/toastermain/management/commands/buildslist.py
> b/bitbake/lib/toaster/toastermain/management/commands/buildslist.py
> index 70b5812..1ed2022 100644 ---
> a/bitbake/lib/toaster/toastermain/management/commands/buildslist.py
> +++
> b/bitbake/lib/toaster/toastermain/management/commands/buildslist.py
> @@ -1,3 +1,7 @@ +# +# SPDX-License-Identifier: GPL-2.0-only +#
> +
> from django.core.management.base import BaseCommand, CommandError
> from orm.models import Build
> import os
> diff --git
> a/bitbake/lib/toaster/toastermain/management/commands/checksocket.py
> b/bitbake/lib/toaster/toastermain/management/commands/checksocket.py
> index 0399b86..811fd5d 100644 ---
> a/bitbake/lib/toaster/toastermain/management/commands/checksocket.py
> +++
> b/bitbake/lib/toaster/toastermain/management/commands/checksocket.py
> @@ -1,23 +1,11 @@ -#!/usr/bin/env python -# ex:ts=4:sw=4:sts=4:et -#
> -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#!/usr/bin/env python3 #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2015 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> """Custom management command checksocket."""
>
> diff --git
> a/bitbake/lib/toaster/toastermain/management/commands/perf.py
> b/bitbake/lib/toaster/toastermain/management/commands/perf.py index
> 6b450bb..7d629fb 100644 ---
> a/bitbake/lib/toaster/toastermain/management/commands/perf.py +++
> b/bitbake/lib/toaster/toastermain/management/commands/perf.py @@ -1,3
> +1,7 @@ +# +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> from django.core.management.base import BaseCommand
> from django.test.client import Client
> import os, sys, re
> diff --git a/bitbake/lib/toaster/toastermain/settings.py
> b/bitbake/lib/toaster/toastermain/settings.py index 13541d3..74501fa
> 100644 --- a/bitbake/lib/toaster/toastermain/settings.py
> +++ b/bitbake/lib/toaster/toastermain/settings.py
> @@ -1,23 +1,10 @@
> #
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> -#
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> # Django settings for Toaster project.
>
> diff --git
> a/bitbake/lib/toaster/toastermain/settings_production_example.py
> b/bitbake/lib/toaster/toastermain/settings_production_example.py
> index 61a2888..6cd0f52 100644 ---
> a/bitbake/lib/toaster/toastermain/settings_production_example.py +++
> b/bitbake/lib/toaster/toastermain/settings_production_example.py @@
> -1,23 +1,10 @@ # -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> -#
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> # See Django documentation for more information about deployment
> # https://docs.djangoproject.com/en/1.8/howto/deployment/checklist/
> diff --git a/bitbake/lib/toaster/toastermain/settings_test.py
> b/bitbake/lib/toaster/toastermain/settings_test.py index
> a322711..6538d9e 100644 ---
> a/bitbake/lib/toaster/toastermain/settings_test.py +++
> b/bitbake/lib/toaster/toastermain/settings_test.py @@ -1,23 +1,10 @@
> #
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> -#
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> # Django settings for Toaster project.
>
> diff --git a/bitbake/lib/toaster/toastermain/urls.py
> b/bitbake/lib/toaster/toastermain/urls.py index e2fb0ae..ac77bc3
> 100644 --- a/bitbake/lib/toaster/toastermain/urls.py
> +++ b/bitbake/lib/toaster/toastermain/urls.py
> @@ -1,23 +1,10 @@
> #
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> -#
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from django.conf.urls import include, url
> from django.views.generic import RedirectView, TemplateView
> diff --git a/bitbake/lib/toaster/toastermain/wsgi.py
> b/bitbake/lib/toaster/toastermain/wsgi.py index 031b314..4c31283
> 100644 --- a/bitbake/lib/toaster/toastermain/wsgi.py
> +++ b/bitbake/lib/toaster/toastermain/wsgi.py
> @@ -1,7 +1,8 @@
> -"""
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> +"""
> WSGI config for Toaster project.
>
> This module contains the WSGI application used by Django's
> development server diff --git a/doc/user_manual.md
> b/doc/user_manual.md index c2657da..96aa2e3 100644
> --- a/doc/user_manual.md
> +++ b/doc/user_manual.md
> @@ -135,8 +135,8 @@ DISTRO_ARCH ??= "armhf"
> Then, call `bitbake` with image names, e.g.:
>
> ```
> -bitbake multiconfig:qemuarm-buster:isar-image-base \
> - multiconfig:qemuarm-buster:isar-image-debug
> +bitbake mc:qemuarm-buster:isar-image-base \
> + mc:qemuarm-buster:isar-image-debug
> ```
>
> The following images are created:
> @@ -169,14 +169,14 @@ The following command will produce
> `isar-image-base` images for all targets:
> ```
> $ bitbake \
> - multiconfig:qemuarm-stretch:isar-image-base \
> - multiconfig:qemuarm-buster:isar-image-base \
> - multiconfig:qemuarm64-stretch:isar-image-base \
> - multiconfig:qemui386-stretch:isar-image-base \
> - multiconfig:qemui386-buster:isar-image-base \
> - multiconfig:qemuamd64-stretch:isar-image-base \
> - multiconfig:qemuamd64-buster:isar-image-base \
> - multiconfig:rpi-stretch:isar-image-base
> + mc:qemuarm-stretch:isar-image-base \
> + mc:qemuarm-buster:isar-image-base \
> + mc:qemuarm64-stretch:isar-image-base \
> + mc:qemui386-stretch:isar-image-base \
> + mc:qemui386-buster:isar-image-base \
> + mc:qemuamd64-stretch:isar-image-base \
> + mc:qemuamd64-buster:isar-image-base \
> + mc:rpi-stretch:isar-image-base
> ```
>
> Created images are:
> @@ -197,9 +197,9 @@ tmp/deploy/images/rpi/isar-image-base.rpi-sdimg
> A bootable disk image is generated if you set IMAGE_TYPE to
> 'wic-img'. Behind the scenes a tool called `wic` is used to assemble
> the images. It is controlled by a `.wks` file which you can choose
> with changing WKS_FILE. Some examples in the tree use that feature
> already. ``` # Generate an image for the `i386` target architecture
> - $ bitbake multiconfig:qemui386-buster:isar-image-base
> + $ bitbake mc:qemui386-buster:isar-image-base
> # Similarly, for the `amd64` target architecture, in this case EFI
> - $ bitbake multiconfig:qemuamd64-buster:isar-image-base
> + $ bitbake mc:qemuamd64-buster:isar-image-base
> ```
>
> Variables may be used in `.wks.in` files; Isar will expand them and
> generate a regular `.wks` file before generating the disk image using
> `wic`. @@ -303,7 +303,7 @@ following variables define the default
> configuration to build for:
> - `DISTRO_ARCH` - The Debian architecture to build for (e.g.,
> `armhf`).
> If BitBake is called with multiconfig targets (e.g.,
> -`multiconfig:qemuarm-buster:isar-image-base`), the following
> variable defines +`mc:qemuarm-buster:isar-image-base`), the following
> variable defines all supported configurations:
>
> - `BBMULTICONFIG` - The list of the complete configuration
> definition files. @@ -687,7 +687,7 @@ Debian cross-compilation works
> out of the box starting from Debian stretch distr Just like
> OpenEmbedded, Isar supports a devshell target for all dpkg package
> recipes. This target opens a terminal inside the buildchroot that
> runs the package build. To invoke it, just call -`bitbake
> multiconfig:${MACHINE}-${DISTRO}:<package_name> -c devshell`.
> +`bitbake mc:${MACHINE}-${DISTRO}:<package_name> -c devshell`.
>
> ## Create an ISAR SDK root filesystem
> @@ -705,7 +705,7 @@ target binary artifacts. Developer chroots to sdk
> rootfs and develops applicatio ### Solution
>
> User manually triggers creation of SDK root filesystem for his
> target platform by launching the task `do_populate_sdk` for target
> image, f.e. -`bitbake -c do_populate_sdk
> multiconfig:${MACHINE}-${DISTRO}:isar-image-base`. +`bitbake -c
> do_populate_sdk mc:${MACHINE}-${DISTRO}:isar-image-base`. The
> resulting SDK rootfs is archived into
> `tmp/deploy/images/${MACHINE}/sdk-${DISTRO}-${DISTRO_ARCH}.tar.xz`.
> It is additionally available for direct use under
> `tmp/deploy/images/${MACHINE}/sdk-${DISTRO}-${DISTRO_ARCH}/`. @@
> -717,7 +717,7 @@ One may chroot into the SDK and install required
> target packages with the help o
> - Trigger creation of SDK root filesystem
>
> ```
> -bitbake -c do_populate_sdk multiconfig:qemuarm-buster:isar-image-base
> +bitbake -c do_populate_sdk mc:qemuarm-buster:isar-image-base
> ```
>
> - Mount the following directories in chroot by passing resulting
> rootfs as an argument to the script `mount_chroot.sh`: @@ -801,7
> +801,7 @@ BASE_REPO_KEY =
> "file://<absolute_path_to_your_pub_key_file>"'
> - Trigger creation of local apt caching Debian packages during
> image generation.
> ```
> -bitbake -c cache_base_repo multiconfig:qemuarm-buster:isar-image-base
> +bitbake -c cache_base_repo mc:qemuarm-buster:isar-image-base
> ```
>
> - Set `ISAR_USE_CACHED_BASE_REPO` in `conf/local.conf`:
> @@ -820,7 +820,7 @@ sudo rm -rf tmp
> - Trigger again generation of image (now using local caching repo):
>
> ```
> -bitbake multiconfig:qemuarm-buster:isar-image-base
> +bitbake mc:qemuarm-buster:isar-image-base
> ```
>
> ### Limitation
> @@ -860,5 +860,5 @@ DISTRO_APT_SOURCES_append = "
> conf/distro/docker-buster.list" And build the corresponding image
> target:
> ```
> -bitbake multiconfig:qemuarm64-buster:isar-image-base
> +bitbake mc:qemuarm64-buster:isar-image-base
> ```
> diff --git a/meta-isar/conf/conf-notes.txt
> b/meta-isar/conf/conf-notes.txt index 74df97c..c557cd3 100644
> --- a/meta-isar/conf/conf-notes.txt
> +++ b/meta-isar/conf/conf-notes.txt
> @@ -1,4 +1,4 @@
> Common targets are:
> - multiconfig:qemuarm-buster:isar-image-base
> - multiconfig:qemuamd64-buster:isar-image-base
> - multiconfig:rpi-stretch:isar-image-base
> + mc:qemuarm-buster:isar-image-base
> + mc:qemuamd64-buster:isar-image-base
> + mc:rpi-stretch:isar-image-base
> diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
> index 713d1c2..a8556a6 100755
> --- a/scripts/ci_build.sh
> +++ b/scripts/ci_build.sh
> @@ -21,40 +21,40 @@ BUILD_DIR=./build
> BB_ARGS="-v"
>
> TARGETS_SET="\
> - multiconfig:qemuarm-stretch:isar-image-base \
> - multiconfig:qemuarm-buster:isar-image-base \
> - multiconfig:qemuarm-bullseye:isar-image-base \
> - multiconfig:qemuarm64-stretch:isar-image-base \
> - multiconfig:qemui386-stretch:isar-image-base \
> - multiconfig:qemui386-buster:isar-image-base \
> - multiconfig:qemui386-bullseye:isar-image-base \
> - multiconfig:qemuamd64-stretch:isar-image-base \
> - multiconfig:qemuamd64-buster:isar-image-base \
> - multiconfig:qemuamd64-buster-tgz:isar-image-base \
> - multiconfig:qemuamd64-bullseye:isar-image-base \
> - multiconfig:qemumipsel-stretch:isar-image-base \
> - multiconfig:qemumipsel-buster:isar-image-base \
> - multiconfig:qemumipsel-bullseye:isar-image-base \
> - multiconfig:nand-ubi-demo-buster:isar-image-ubi \
> - multiconfig:rpi-stretch:isar-image-base"
> + mc:qemuarm-stretch:isar-image-base \
> + mc:qemuarm-buster:isar-image-base \
> + mc:qemuarm-bullseye:isar-image-base \
> + mc:qemuarm64-stretch:isar-image-base \
> + mc:qemui386-stretch:isar-image-base \
> + mc:qemui386-buster:isar-image-base \
> + mc:qemui386-bullseye:isar-image-base \
> + mc:qemuamd64-stretch:isar-image-base \
> + mc:qemuamd64-buster:isar-image-base \
> + mc:qemuamd64-buster-tgz:isar-image-base \
> + mc:qemuamd64-bullseye:isar-image-base \
> + mc:qemumipsel-stretch:isar-image-base \
> + mc:qemumipsel-buster:isar-image-base \
> + mc:qemumipsel-bullseye:isar-image-base \
> + mc:nand-ubi-demo-buster:isar-image-ubi \
> + mc:rpi-stretch:isar-image-base"
> # qemu-user-static of <= buster too old to build that
> - # multiconfig:qemuarm64-buster:isar-image-base
> - # multiconfig:qemuarm64-bullseye:isar-image-base
> + # mc:qemuarm64-buster:isar-image-base
> + # mc:qemuarm64-bullseye:isar-image-base
>
> CROSS_TARGETS_SET="\
> - multiconfig:qemuarm-stretch:isar-image-base \
> - multiconfig:qemuarm-buster:isar-image-base \
> - multiconfig:qemuarm-bullseye:isar-image-base \
> - multiconfig:qemuarm64-stretch:isar-image-base \
> - multiconfig:qemuamd64-stretch:isar-image-base \
> - multiconfig:de0-nano-soc-stretch:isar-image-base \
> - multiconfig:rpi-stretch:isar-image-base"
> + mc:qemuarm-stretch:isar-image-base \
> + mc:qemuarm-buster:isar-image-base \
> + mc:qemuarm-bullseye:isar-image-base \
> + mc:qemuarm64-stretch:isar-image-base \
> + mc:qemuamd64-stretch:isar-image-base \
> + mc:de0-nano-soc-stretch:isar-image-base \
> + mc:rpi-stretch:isar-image-base"
>
> REPRO_TARGETS_SET="\
> - multiconfig:qemuarm-stretch:isar-image-base \
> - multiconfig:qemuarm64-stretch:isar-image-base \
> - multiconfig:qemuamd64-stretch:isar-image-base \
> - multiconfig:qemuarm-buster:isar-image-base"
> + mc:qemuarm-stretch:isar-image-base \
> + mc:qemuarm64-stretch:isar-image-base \
> + mc:qemuamd64-stretch:isar-image-base \
> + mc:qemuarm-buster:isar-image-base"
>
>
> show_help() {
> @@ -162,7 +162,7 @@ sed -i -e 's/ISAR_CROSS_COMPILE ?=
> "0"/ISAR_CROSS_COMPILE ?= "1"/g' conf/local.c bitbake $BB_ARGS
> $CROSS_TARGETS_SET while [ -e bitbake.sock ]; do sleep 1; done
> # In addition test SDK creation
> -bitbake $BB_ARGS -c do_populate_sdk
> multiconfig:qemuarm-stretch:isar-image-base +bitbake $BB_ARGS -c
> do_populate_sdk mc:qemuarm-stretch:isar-image-base while [ -e
> bitbake.sock ]; do sleep 1; done
> if [ -z "$FAST_BUILD" ]; then
> @@ -175,6 +175,6 @@ fi
> cp -a "${ISARROOT}/meta/classes/dpkg-base.bbclass"
> "${ISARROOT}/meta/classes/dpkg-base.bbclass.ci-backup" echo -e
> "do_fetch_append() {\n\n}" >>
> "${ISARROOT}/meta/classes/dpkg-base.bbclass" -bitbake $BB_ARGS
> multiconfig:qemuamd64-stretch:isar-image-base +bitbake $BB_ARGS
> mc:qemuamd64-stretch:isar-image-base
> mv "${ISARROOT}/meta/classes/dpkg-base.bbclass.ci-backup"
> "${ISARROOT}/meta/classes/dpkg-base.bbclass" diff --git
> a/scripts/start_vm b/scripts/start_vm index 9f07a03..71b55f2 100755
> --- a/scripts/start_vm
> +++ b/scripts/start_vm
> @@ -101,7 +101,7 @@ do
> shift
> done
>
> -eval $(bitbake -e multiconfig:qemu$ARCH-$DISTRO:isar-image-base |
> grep "^DEPLOY_DIR_IMAGE=") +eval $(bitbake -e
> mc:qemu$ARCH-$DISTRO:isar-image-base | grep "^DEPLOY_DIR_IMAGE=")
> readonly IMAGE_DIR=$DEPLOY_DIR_IMAGE
> readonly ISARROOT="$(dirname "$0")"/..
> @@ -112,13 +112,13 @@ eval "$(egrep 'MACHINE_SERIAL' $MACHINE_CONF
> |bb2sh)" readonly
> CONFIG_CONF=$ISARROOT/meta-isar/conf/multiconfig/qemu$ARCH-$DISTRO.conf
> eval "$(egrep 'QEMU_' $CONFIG_CONF |bb2sh)"
> -eval $(bitbake -e multiconfig:qemu$ARCH-$DISTRO:isar-image-base |
> grep "^IMAGE_TYPE=") +eval $(bitbake -e
> mc:qemu$ARCH-$DISTRO:isar-image-base | grep "^IMAGE_TYPE=") case
> "$IMAGE_TYPE" in ext4-img)
> readonly
> ROOTFS_IMAGE=isar-image-base-debian-$DISTRO-qemu$ARCH.ext4.img
> - eval $(bitbake -e multiconfig:qemu$ARCH-$DISTRO:isar-image-base
> | grep "^KERNEL_IMAGE=")
> - eval $(bitbake -e multiconfig:qemu$ARCH-$DISTRO:isar-image-base
> | grep "^INITRD_IMAGE=")
> + eval $(bitbake -e mc:qemu$ARCH-$DISTRO:isar-image-base | grep
> "^KERNEL_IMAGE=")
> + eval $(bitbake -e mc:qemu$ARCH-$DISTRO:isar-image-base | grep
> "^INITRD_IMAGE=") QKERNEL=$IMAGE_DIR/${KERNEL_IMAGE}
> QINITRD=/dev/null
> [ -n "$INITRD_IMAGE" ] && QINITRD=$IMAGE_DIR/${INITRD_IMAGE}
> diff --git a/testsuite/build_test/build_test.py
> b/testsuite/build_test/build_test.py index 4220d3a..7a55c2f 100644
> --- a/testsuite/build_test/build_test.py
> +++ b/testsuite/build_test/build_test.py
> @@ -22,7 +22,7 @@ class BuildTest(Test):
>
> #isar_root = dirname(__file__) + '/..'
> os.chdir(build_dir)
> - cmdline = ['bitbake', 'multiconfig:qemu' + arch + '-' +
> distro + ':isar-image-base']
> + cmdline = ['bitbake', 'mc:qemu' + arch + '-' + distro +
> ':isar-image-base'] p1 = subprocess32.run(cmdline)
>
> if p1.returncode:
> diff --git a/testsuite/start_vm.py b/testsuite/start_vm.py
> index 02a4b51..a3e32ac 100755
> --- a/testsuite/start_vm.py
> +++ b/testsuite/start_vm.py
> @@ -10,7 +10,7 @@ import sys
> import time
>
> def get_bitbake_env(arch, distro):
> - multiconfig = 'multiconfig:qemu' + arch + '-' + distro +
> ':isar-image-base'
> + multiconfig = 'mc:qemu' + arch + '-' + distro +
> ':isar-image-base' output = subprocess.check_output(['bitbake', '-e',
> str(multiconfig)]) return output
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 1/1] bitbake: update to version 1.44.0
2019-10-22 7:36 ` Henning Schild
@ 2019-10-22 7:41 ` Cedric Hombourger
2019-10-22 15:56 ` Baurzhan Ismagulov
0 siblings, 1 reply; 11+ messages in thread
From: Cedric Hombourger @ 2019-10-22 7:41 UTC (permalink / raw)
To: Henning Schild; +Cc: isar-users
Hi Henning
On 10/22/2019 9:36 AM, Henning Schild wrote:
> Hi Cedric,
>
> i would suggest a split between the actual bump and the following
> Isar-changes. But i do not care too much and both ways have their pros
> and cons.
I have debated (with myself :)) and I agree that there are pros and cons
with each approach.
I ended up submitting a single commit at the end of the day to support
people needing to "git bisect".
I have otherwise no issues resubmitting the changes if we prefer to have
the bitbake upgrade clearly separated from the Isar changes (either way
is fine with me - @maintainers just let me know what your preference is!)
> Henning
>
> Am Mon, 21 Oct 2019 16:19:58 +0200
> schrieb Cedric Hombourger <Cedric_Hombourger@mentor.com>:
>
>> Found that the gitsm fetcher we ship is rather out-dated compared to
>> upstream and was failing to pull git sub-modules from repositories
>> such as azure-umqtt-c.
>>
>> It shall be noted that bitbake has renamed its multiconfig targets:
>> the "multiconfig:" prefix was changed to "mc:". Updated our scripts
>> as well as documentation.
>>
>> Origin: https://github.com/openembedded/bitbake.git
>> Commit 5d83d828cacb58ccb7c464e799c85fd2d2a50ccc (tag: 1.44.0)
>> Author: Richard Purdie <richard.purdie@linuxfoundation.org>
>> Date: Wed Oct 9 14:10:21 2019 +0100
>>
>> bitbake: Update to version 1.44.0
>>
>> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>>
>> Signed-off-by: Cedric Hombourger <Cedric_Hombourger@mentor.com>
>> ---
>> RECIPE-API-CHANGELOG.md | 16 +
>> bitbake/.gitattributes | 2 +
>> bitbake/HEADER | 19 -
>> bitbake/LICENSE | 12 +-
>> bitbake/{COPYING => LICENSE.GPL-2.0-only} | 61 +-
>> bitbake/LICENSE.MIT | 25 +
>> bitbake/MANIFEST.in | 5 +-
>> bitbake/bin/bitbake | 16 +-
>> bitbake/bin/bitbake-diffsigs | 155 +-
>> bitbake/bin/bitbake-dumpsig | 95 +-
>> bitbake/bin/bitbake-hashclient | 170 ++
>> bitbake/bin/bitbake-hashserv | 62 +
>> bitbake/bin/bitbake-layers | 12 +-
>> bitbake/bin/bitbake-prserv | 4 +
>> bitbake/bin/bitbake-selftest | 16 +-
>> bitbake/bin/bitbake-worker | 19 +-
>> bitbake/bin/bitdoc | 14 +-
>> bitbake/bin/git-make-shallow | 4 +
>> bitbake/bin/toaster | 12 +-
>> bitbake/bin/toaster-eventreplay | 17 +-
>> bitbake/classes/base.bbclass | 2 +-
>> bitbake/contrib/dump_cache.py | 2 -
>> .../bitbake-user-manual-execution.xml | 78 +-
>> .../bitbake-user-manual-fetching.xml | 69 +-
>> .../bitbake-user-manual-hello.xml | 28 +-
>> .../bitbake-user-manual-intro.xml | 18 +-
>> .../bitbake-user-manual-metadata.xml | 179 +-
>> .../bitbake-user-manual-ref-variables.xml | 422 ++---
>> bitbake/doc/poky.ent | 8 -
>> bitbake/lib/bb/COW.py | 15 -
>> bitbake/lib/bb/__init__.py | 16 +-
>> bitbake/lib/bb/build.py | 151 +-
>> bitbake/lib/bb/cache.py | 109 +-
>> bitbake/lib/bb/cache_extra.py | 14 +-
>> bitbake/lib/bb/checksum.py | 12 +-
>> bitbake/lib/bb/codeparser.py | 13 +-
>> bitbake/lib/bb/command.py | 12 +-
>> bitbake/lib/bb/compat.py | 4 +
>> bitbake/lib/bb/cooker.py | 218 +--
>> bitbake/lib/bb/cookerdata.py | 44 +-
>> bitbake/lib/bb/daemonize.py | 4 +
>> bitbake/lib/bb/data.py | 23 +-
>> bitbake/lib/bb/data_smart.py | 25 +-
>> bitbake/lib/bb/event.py | 33 +-
>> bitbake/lib/bb/exceptions.py | 3 +
>> bitbake/lib/bb/fetch2/__init__.py | 55 +-
>> bitbake/lib/bb/fetch2/bzr.py | 12 +-
>> bitbake/lib/bb/fetch2/clearcase.py | 19 +-
>> bitbake/lib/bb/fetch2/cvs.py | 17 +-
>> bitbake/lib/bb/fetch2/git.py | 68 +-
>> bitbake/lib/bb/fetch2/gitannex.py | 14 +-
>> bitbake/lib/bb/fetch2/gitsm.py | 281 ++-
>> bitbake/lib/bb/fetch2/hg.py | 18 +-
>> bitbake/lib/bb/fetch2/local.py | 16 +-
>> bitbake/lib/bb/fetch2/npm.py | 69 +-
>> bitbake/lib/bb/fetch2/osc.py | 5 +-
>> bitbake/lib/bb/fetch2/perforce.py | 15 +-
>> bitbake/lib/bb/fetch2/repo.py | 16 +-
>> bitbake/lib/bb/fetch2/s3.py | 15 +-
>> bitbake/lib/bb/fetch2/sftp.py | 15 +-
>> bitbake/lib/bb/fetch2/ssh.py | 14 +-
>> bitbake/lib/bb/fetch2/svn.py | 98 +-
>> bitbake/lib/bb/fetch2/wget.py | 104 +-
>> bitbake/lib/bb/main.py | 41 +-
>> bitbake/lib/bb/methodpool.py | 15 +-
>> bitbake/lib/bb/monitordisk.py | 27 +-
>> bitbake/lib/bb/msg.py | 14 +-
>> bitbake/lib/bb/namedtuple_with_abc.py | 4 +-
>> bitbake/lib/bb/parse/__init__.py | 14 +-
>> bitbake/lib/bb/parse/ast.py | 15 +-
>> bitbake/lib/bb/parse/parse_py/BBHandler.py | 59 +-
>> bitbake/lib/bb/parse/parse_py/ConfHandler.py | 17 +-
>> bitbake/lib/bb/parse/parse_py/__init__.py | 17 +-
>> bitbake/lib/bb/persist_data.py | 234 ++-
>> bitbake/lib/bb/process.py | 4 +
>> bitbake/lib/bb/progress.py | 28 +-
>> bitbake/lib/bb/providers.py | 18 +-
>> bitbake/lib/bb/pysh/builtin.py | 710 --------
>> bitbake/lib/bb/pysh/interp.py | 1367 ---------------
>> bitbake/lib/bb/pysh/lsprof.py | 116 --
>> bitbake/lib/bb/pysh/pysh.py | 167 --
>> bitbake/lib/bb/pysh/pyshlex.py | 5 -
>> bitbake/lib/bb/pysh/pyshyacc.py | 17 +-
>> bitbake/lib/bb/pysh/sherrors.py | 26 -
>> bitbake/lib/bb/pysh/subprocess_fix.py | 77 -
>> bitbake/lib/bb/remotedata.py | 12 +-
>> bitbake/lib/bb/runqueue.py | 1523
>> ++++++++++------- bitbake/lib/bb/server/__init__.py |
>> 14 +- bitbake/lib/bb/server/process.py | 123 +-
>> bitbake/lib/bb/server/xmlrpcclient.py | 12 +-
>> bitbake/lib/bb/server/xmlrpcserver.py | 12 +-
>> bitbake/lib/bb/siggen.py | 350 +++-
>> bitbake/lib/bb/taskdata.py | 17 +-
>> bitbake/lib/bb/tests/codeparser.py | 22 +-
>> bitbake/lib/bb/tests/cooker.py | 15 +-
>> bitbake/lib/bb/tests/cow.py | 17 +-
>> bitbake/lib/bb/tests/data.py | 39 +-
>> bitbake/lib/bb/tests/event.py | 23 +-
>> bitbake/lib/bb/tests/fetch.py | 308 +++-
>> bitbake/lib/bb/tests/parse.py | 33 +-
>> bitbake/lib/bb/tests/persist_data.py | 129 ++
>> .../tests/runqueue-tests/classes/base.bbclass | 262 +++
>> .../runqueue-tests/classes/image.bbclass | 5 +
>> .../runqueue-tests/classes/native.bbclass | 2 +
>> .../bb/tests/runqueue-tests/conf/bitbake.conf | 16 +
>> .../runqueue-tests/conf/multiconfig/mc1.conf | 1 +
>> .../runqueue-tests/conf/multiconfig/mc2.conf | 1 +
>> .../lib/bb/tests/runqueue-tests/recipes/a1.bb | 0
>> .../lib/bb/tests/runqueue-tests/recipes/b1.bb | 1 +
>> .../lib/bb/tests/runqueue-tests/recipes/c1.bb | 0
>> .../lib/bb/tests/runqueue-tests/recipes/d1.bb | 3 +
>> .../lib/bb/tests/runqueue-tests/recipes/e1.bb | 1 +
>> bitbake/lib/bb/tests/runqueue.py | 323 ++++
>> bitbake/lib/bb/tests/utils.py | 23 +-
>> bitbake/lib/bb/tinfoil.py | 12 +-
>> bitbake/lib/bb/ui/__init__.py | 12 +-
>> bitbake/lib/bb/ui/buildinfohelper.py | 22 +-
>> bitbake/lib/bb/ui/knotty.py | 60 +-
>> bitbake/lib/bb/ui/ncurses.py | 12 +-
>> bitbake/lib/bb/ui/taskexp.py | 12 +-
>> bitbake/lib/bb/ui/toasterui.py | 12 +-
>> bitbake/lib/bb/ui/uievent.py | 15 +-
>> bitbake/lib/bb/ui/uihelper.py | 16 +-
>> bitbake/lib/bb/utils.py | 49 +-
>> bitbake/lib/bblayers/__init__.py | 4 +
>> bitbake/lib/bblayers/action.py | 4 +
>> bitbake/lib/bblayers/common.py | 4 +
>> bitbake/lib/bblayers/layerindex.py | 6 +-
>> bitbake/lib/bblayers/query.py | 43 +-
>> bitbake/lib/bs4/dammit.py | 12 +-
>> bitbake/lib/bs4/element.py | 22 +-
>> bitbake/lib/hashserv/__init__.py | 93 +
>> bitbake/lib/hashserv/client.py | 157 ++
>> bitbake/lib/hashserv/server.py | 414 +++++
>> bitbake/lib/hashserv/tests.py | 142 ++
>> bitbake/lib/layerindexlib/__init__.py | 33 +-
>> bitbake/lib/layerindexlib/cooker.py | 12 +-
>> bitbake/lib/layerindexlib/plugin.py | 13 +-
>> bitbake/lib/layerindexlib/restapi.py | 12 +-
>> bitbake/lib/layerindexlib/tests/common.py | 12 +-
>> bitbake/lib/layerindexlib/tests/cooker.py | 12 +-
>> .../lib/layerindexlib/tests/layerindexobj.py | 12 +-
>> bitbake/lib/layerindexlib/tests/restapi.py | 12 +-
>> bitbake/lib/progressbar/__init__.py | 2 +
>> bitbake/lib/progressbar/compat.py | 2 +
>> bitbake/lib/progressbar/progressbar.py | 2 +
>> bitbake/lib/progressbar/widgets.py | 2 +
>> bitbake/lib/prserv/__init__.py | 4 +
>> bitbake/lib/prserv/db.py | 6 +-
>> bitbake/lib/prserv/serv.py | 4 +
>> bitbake/lib/pyinotify.py | 20 +-
>> bitbake/lib/toaster/bldcollector/admin.py | 4 +
>> bitbake/lib/toaster/bldcollector/urls.py | 13 +-
>> bitbake/lib/toaster/bldcollector/views.py | 12 +-
>> bitbake/lib/toaster/bldcontrol/admin.py | 4 +
>> .../lib/toaster/bldcontrol/bbcontroller.py | 16 +-
>> .../bldcontrol/localhostbecontroller.py | 16 +-
>> .../management/commands/checksettings.py | 4 +
>> .../management/commands/runbuilds.py | 4 +
>> bitbake/lib/toaster/bldcontrol/models.py | 4 +
>> bitbake/lib/toaster/bldcontrol/views.py | 4 +
>> bitbake/lib/toaster/manage.py | 4 +
>> bitbake/lib/toaster/orm/fixtures/oe-core.xml | 26 +-
>> bitbake/lib/toaster/orm/fixtures/poky.xml | 38 +-
>> .../orm/management/commands/lsupdates.py | 15 +-
>> bitbake/lib/toaster/orm/models.py | 25 +-
>> .../toaster/tests/browser/selenium_helpers.py | 17 +-
>> .../tests/browser/selenium_helpers_base.py | 17 +-
>> .../tests/browser/test_all_builds_page.py | 16 +-
>> .../tests/browser/test_all_projects_page.py | 16 +-
>> .../tests/browser/test_builddashboard_page.py | 16 +-
>> .../test_builddashboard_page_artifacts.py | 16 +-
>> .../test_builddashboard_page_recipes.py | 16 +-
>> .../browser/test_builddashboard_page_tasks.py | 16 +-
>> .../tests/browser/test_js_unit_tests.py | 16 +-
>> .../tests/browser/test_landing_page.py | 18 +-
>> .../tests/browser/test_layerdetails_page.py | 18 +-
>> .../browser/test_most_recent_builds_states.py | 18 +-
>> .../browser/test_new_custom_image_page.py | 16 +-
>> .../tests/browser/test_new_project_page.py | 16 +-
>> .../tests/browser/test_project_builds_page.py | 16 +-
>> .../tests/browser/test_project_config_page.py | 16 +-
>> .../tests/browser/test_project_page.py | 16 +-
>> .../lib/toaster/tests/browser/test_sample.py | 16 +-
>> .../toaster/tests/browser/test_task_page.py | 16 +-
>> .../tests/browser/test_toastertable_ui.py | 16 +-
>> bitbake/lib/toaster/tests/builds/buildtest.py | 16 +-
>> .../tests/builds/test_core_image_min.py | 17 +-
>> .../toaster/tests/commands/test_loaddata.py | 16 +-
>> .../toaster/tests/commands/test_lsupdates.py | 16 +-
>> .../toaster/tests/commands/test_runbuilds.py | 16 +-
>> bitbake/lib/toaster/tests/db/test_db.py | 2 +
>> .../lib/toaster/tests/eventreplay/__init__.py | 16 +-
>> .../tests/functional/functional_helpers.py | 16 +-
>> .../tests/functional/test_functional_basic.py | 16 +-
>> bitbake/lib/toaster/tests/views/test_views.py | 16 +-
>> bitbake/lib/toaster/toastergui/api.py | 13 +-
>> bitbake/lib/toaster/toastergui/buildtables.py | 15 +-
>> .../toastergui/static/js/importlayer.js | 12 +-
>> bitbake/lib/toaster/toastergui/tablefilter.py | 15 +-
>> bitbake/lib/toaster/toastergui/tables.py | 15 +-
>> .../templatetags/field_values_filter.py | 4 +
>> .../objects_to_dictionaries_filter.py | 4 +
>> .../templatetags/project_url_tag.py | 4 +
>> .../toastergui/templatetags/projecttags.py | 15 +-
>> bitbake/lib/toaster/toastergui/typeaheads.py | 12 +-
>> bitbake/lib/toaster/toastergui/urls.py | 12 +-
>> bitbake/lib/toaster/toastergui/views.py | 16 +-
>> bitbake/lib/toaster/toastergui/widgets.py | 15 +-
>> .../management/commands/builddelete.py | 4 +
>> .../management/commands/buildimport.py | 15 +-
>> .../management/commands/buildslist.py | 4 +
>> .../management/commands/checksocket.py | 16 +-
>> .../toastermain/management/commands/perf.py | 4 +
>> bitbake/lib/toaster/toastermain/settings.py | 15 +-
>> .../settings_production_example.py | 15 +-
>> .../lib/toaster/toastermain/settings_test.py | 15 +-
>> bitbake/lib/toaster/toastermain/urls.py | 15 +-
>> bitbake/lib/toaster/toastermain/wsgi.py | 7 +-
>> doc/user_manual.md | 38 +-
>> meta-isar/conf/conf-notes.txt | 6 +-
>> scripts/ci_build.sh | 62 +-
>> scripts/start_vm | 8 +-
>> testsuite/build_test/build_test.py | 2 +-
>> testsuite/start_vm.py | 2 +-
>> 225 files changed, 5375 insertions(+), 6320 deletions(-)
>> create mode 100644 bitbake/.gitattributes
>> delete mode 100644 bitbake/HEADER
>> rename bitbake/{COPYING => LICENSE.GPL-2.0-only} (84%)
>> create mode 100644 bitbake/LICENSE.MIT
>> mode change 100755 => 120000 bitbake/bin/bitbake-dumpsig
>> create mode 100755 bitbake/bin/bitbake-hashclient
>> create mode 100755 bitbake/bin/bitbake-hashserv
>> delete mode 100644 bitbake/lib/bb/pysh/builtin.py
>> delete mode 100644 bitbake/lib/bb/pysh/interp.py
>> delete mode 100644 bitbake/lib/bb/pysh/lsprof.py
>> delete mode 100644 bitbake/lib/bb/pysh/pysh.py
>> delete mode 100644 bitbake/lib/bb/pysh/subprocess_fix.py
>> create mode 100644 bitbake/lib/bb/tests/persist_data.py
>> create mode 100644
>> bitbake/lib/bb/tests/runqueue-tests/classes/base.bbclass create mode
>> 100644 bitbake/lib/bb/tests/runqueue-tests/classes/image.bbclass
>> create mode 100644
>> bitbake/lib/bb/tests/runqueue-tests/classes/native.bbclass create
>> mode 100644 bitbake/lib/bb/tests/runqueue-tests/conf/bitbake.conf
>> create mode 100644
>> bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc1.conf create
>> mode 100644
>> bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc2.conf create
>> mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/a1.bb create
>> mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/b1.bb create
>> mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/c1.bb create
>> mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/d1.bb create
>> mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/e1.bb create
>> mode 100644 bitbake/lib/bb/tests/runqueue.py create mode 100644
>> bitbake/lib/hashserv/__init__.py create mode 100644
>> bitbake/lib/hashserv/client.py create mode 100644
>> bitbake/lib/hashserv/server.py create mode 100644
>> bitbake/lib/hashserv/tests.py
>>
>> diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
>> index bbef1a3..81ec430 100644
>> --- a/RECIPE-API-CHANGELOG.md
>> +++ b/RECIPE-API-CHANGELOG.md
>> @@ -176,3 +176,19 @@ Otherwise set a encrypted root password like
>> this: USERS += "root"
>> USER_root[password] =
>> "$6$rounds=10000$RXeWrnFmkY$DtuS/OmsAS2cCEDo0BF5qQsizIrq6jPgXnwv3PHqREJeKd1sXdHX/ayQtuQWVDHe0KIO0/sVH8dvQm1KthF0d/"
>> ``` +
>> +### multiconfig build targets were renamed
>> +
>> +bitbake was upgraded to version 1.44.0 where "multiconfig" build
>> targets were +renamed "mc". As an example, builds for the
>> qemuarm-stretch machine should now +be done as follows:
>> +
>> +```
>> +bitbake mc:qemuarm-stretch:isar-image-base
>> +```
>> +
>> +The old syntax is no longer supported and will produce an error:
>> +
>> +```
>> +bitbake multiconfig:qemuarm-stretch:isar-image-base
>> +```
>> diff --git a/bitbake/.gitattributes b/bitbake/.gitattributes
>> new file mode 100644
>> index 0000000..e4f8f62
>> --- /dev/null
>> +++ b/bitbake/.gitattributes
>> @@ -0,0 +1,2 @@
>> +*min.js binary
>> +*min.css binary
>> diff --git a/bitbake/HEADER b/bitbake/HEADER
>> deleted file mode 100644
>> index 9859255..0000000
>> --- a/bitbake/HEADER
>> +++ /dev/null
>> @@ -1,19 +0,0 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> -#
>> -# <one line to give the program's name and a brief idea of what it
>> does.> -# Copyright (C) <year> <name of author>
>> -#
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. -
>> diff --git a/bitbake/LICENSE b/bitbake/LICENSE
>> index 7d4e5f4..8458042 100644
>> --- a/bitbake/LICENSE
>> +++ b/bitbake/LICENSE
>> @@ -1,4 +1,13 @@
>> -BitBake is licensed under the GNU General Public License version
>> 2.0. See COPYING for further details. +BitBake is licensed under the
>> GNU General Public License version 2.0. See +LICENSE.GPL-2.0-only for
>> further details. +
>> +Individual files contain the following style tags instead of the
>> full license text: +
>> + SPDX-License-Identifier: GPL-2.0-only
>> +
>> +This enables machine processing of license information based on the
>> SPDX +License Identifiers that are here available:
>> http://spdx.org/licenses/ +
>>
>> The following external components are distributed with this software:
>>
>> @@ -17,3 +26,4 @@ Foundation and individual contributors.
>> * Font Awesome fonts redistributed under the SIL Open Font License
>> 1.1
>> * simplediff is distributed under the zlib license.
>> +
>> diff --git a/bitbake/COPYING b/bitbake/LICENSE.GPL-2.0-only
>> similarity index 84%
>> rename from bitbake/COPYING
>> rename to bitbake/LICENSE.GPL-2.0-only
>> index d511905..5db3c0a 100644
>> --- a/bitbake/COPYING
>> +++ b/bitbake/LICENSE.GPL-2.0-only
>> @@ -279,61 +279,10 @@ POSSIBILITY OF SUCH DAMAGES.
>>
>> END OF TERMS AND CONDITIONS
>>
>> - How to Apply These Terms to Your New Programs
>> +Note:
>> +Individual files contain the following tag instead of the full
>> license text.
>> - If you develop a new program, and you want it to be of the greatest
>> -possible use to the public, the best way to achieve this is to make
>> it -free software which everyone can redistribute and change under
>> these terms.
>> + SPDX-License-Identifier: GPL-2.0-only
>>
>> - To do so, attach the following notices to the program. It is
>> safest -to attach them to the start of each source file to most
>> effectively -convey the exclusion of warranty; and each file should
>> have at least -the "copyright" line and a pointer to where the full
>> notice is found. -
>> - <one line to give the program's name and a brief idea of what it
>> does.>
>> - Copyright (C) <year> <name of author>
>> -
>> - This program is free software; you can redistribute it and/or
>> modify
>> - it under the terms of the GNU General Public License as
>> published by
>> - the Free Software Foundation; either version 2 of the License, or
>> - (at your option) any later version.
>> -
>> - This program is distributed in the hope that it will be useful,
>> - but WITHOUT ANY WARRANTY; without even the implied warranty of
>> - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> - GNU General Public License for more details.
>> -
>> - You should have received a copy of the GNU General Public
>> License along
>> - with this program; if not, write to the Free Software
>> Foundation, Inc.,
>> - 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
>> -
>> -Also add information on how to contact you by electronic and paper
>> mail. -
>> -If the program is interactive, make it output a short notice like
>> this -when it starts in an interactive mode:
>> -
>> - Gnomovision version 69, Copyright (C) year name of author
>> - Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type
>> `show w'.
>> - This is free software, and you are welcome to redistribute it
>> - under certain conditions; type `show c' for details.
>> -
>> -The hypothetical commands `show w' and `show c' should show the
>> appropriate -parts of the General Public License. Of course, the
>> commands you use may -be called something other than `show w' and
>> `show c'; they could even be -mouse-clicks or menu items--whatever
>> suits your program. -
>> -You should also get your employer (if you work as a programmer) or
>> your -school, if any, to sign a "copyright disclaimer" for the
>> program, if -necessary. Here is a sample; alter the names:
>> -
>> - Yoyodyne, Inc., hereby disclaims all copyright interest in the
>> program
>> - `Gnomovision' (which makes passes at compilers) written by James
>> Hacker. -
>> - <signature of Ty Coon>, 1 April 1989
>> - Ty Coon, President of Vice
>> -
>> -This General Public License does not permit incorporating your
>> program into -proprietary programs. If your program is a subroutine
>> library, you may -consider it more useful to permit linking
>> proprietary applications with the -library. If this is what you want
>> to do, use the GNU Lesser General -Public License instead of this
>> License. +This enables machine processing of license information
>> based on the SPDX +License Identifiers that are here available:
>> http://spdx.org/licenses/ diff --git a/bitbake/LICENSE.MIT
>> b/bitbake/LICENSE.MIT new file mode 100644
>> index 0000000..a6919eb
>> --- /dev/null
>> +++ b/bitbake/LICENSE.MIT
>> @@ -0,0 +1,25 @@
>> +Permission is hereby granted, free of charge, to any person
>> obtaining a copy +of this software and associated documentation files
>> (the "Software"), to deal +in the Software without restriction,
>> including without limitation the rights +to use, copy, modify, merge,
>> publish, distribute, sublicense, and/or sell +copies of the Software,
>> and to permit persons to whom the Software is +furnished to do so,
>> subject to the following conditions: +
>> +The above copyright notice and this permission notice shall be
>> included in +all copies or substantial portions of the Software.
>> +
>> +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
>> EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
>> MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND
>> NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS
>> BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN
>> ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN
>> CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
>> +SOFTWARE. +
>> +Note:
>> +Individual files contain the following tag instead of the full
>> license text. +
>> + SPDX-License-Identifier: MIT
>> +
>> +This enables machine processing of license information based on the
>> SPDX +License Identifiers that are here available:
>> http://spdx.org/licenses/ diff --git a/bitbake/MANIFEST.in
>> b/bitbake/MANIFEST.in index b197378..f24969a 100644
>> --- a/bitbake/MANIFEST.in
>> +++ b/bitbake/MANIFEST.in
>> @@ -1,6 +1,8 @@
>> -include COPYING
>> include ChangeLog
>> include AUTHORS
>> +include LICENSE
>> +include LICENSE.GPL-2.0-only
>> +include LICENSE.MIT
>> include contrib/*
>> include contrib/vim/*/*
>> include conf/*
>> @@ -8,4 +10,3 @@ include classes/*
>> include doc/*
>> include doc/manual/*
>> include ez_setup.py
>> -include HEADER
>> diff --git a/bitbake/bin/bitbake b/bitbake/bin/bitbake
>> index 57dec2a..66d08f8 100755
>> --- a/bitbake/bin/bitbake
>> +++ b/bitbake/bin/bitbake
>> @@ -1,6 +1,4 @@
>> #!/usr/bin/env python3
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # Copyright (C) 2003, 2004 Chris Larson
>> # Copyright (C) 2003, 2004 Phil Blundell
>> @@ -9,18 +7,8 @@
>> # Copyright (C) 2005 ROAD GmbH
>> # Copyright (C) 2006 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import os
>> import sys
>> @@ -38,7 +26,7 @@ from bb.main import bitbake_main,
>> BitBakeConfigParameters, BBMainException if
>> sys.getfilesystemencoding() != "utf-8": sys.exit("Please use a locale
>> setting which supports UTF-8 (such as LANG=en_US.UTF-8).\nPython
>> can't change the filesystem locale after loading so we need a UTF-8
>> when Python starts or things won't work.") -__version__ = "1.40.0"
>> +__version__ = "1.44.0"
>> if __name__ == "__main__":
>> if __version__ != bb.__version__:
>> diff --git a/bitbake/bin/bitbake-diffsigs
>> b/bitbake/bin/bitbake-diffsigs index 4e6bbdd..19420a2 100755
>> --- a/bitbake/bin/bitbake-diffsigs
>> +++ b/bitbake/bin/bitbake-diffsigs
>> @@ -1,27 +1,16 @@
>> #!/usr/bin/env python3
>>
>> -# bitbake-diffsigs
>> -# BitBake task signature data comparison utility
>> +# bitbake-diffsigs / bitbake-dumpsig
>> +# BitBake task signature data dump and comparison utility
>> #
>> # Copyright (C) 2012-2013, 2017 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import os
>> import sys
>> import warnings
>> -import fnmatch
>> import argparse
>> import logging
>> import pickle
>> @@ -32,7 +21,10 @@ import bb.tinfoil
>> import bb.siggen
>> import bb.msg
>>
>> -logger = bb.msg.logger_create('bitbake-diffsigs')
>> +myname = os.path.basename(sys.argv[0])
>> +logger = bb.msg.logger_create(myname)
>> +
>> +is_dump = myname == 'bitbake-dumpsig'
>>
>> def find_siginfo(tinfoil, pn, taskname, sigs=None):
>> result = None
>> @@ -59,8 +51,8 @@ def find_siginfo(tinfoil, pn, taskname, sigs=None):
>> sys.exit(2)
>> return result
>>
>> -def find_compare_task(bbhandler, pn, taskname, sig1=None, sig2=None,
>> color=False):
>> - """ Find the most recent signature files for the specified
>> PN/task and compare them """ +def find_siginfo_task(bbhandler, pn,
>> taskname, sig1=None, sig2=None):
>> + """ Find the most recent signature files for the specified
>> PN/task """
>> if not taskname.startswith('do_'):
>> taskname = 'do_%s' % taskname
>> @@ -79,73 +71,81 @@ def find_compare_task(bbhandler, pn, taskname,
>> sig1=None, sig2=None, color=False latestfiles = [sigfiles[sig1],
>> sigfiles[sig2]] else:
>> filedates = find_siginfo(bbhandler, pn, taskname)
>> - latestfiles = sorted(filedates.keys(), key=lambda f:
>> filedates[f])[-3:]
>> + latestfiles = sorted(filedates.keys(), key=lambda f:
>> filedates[f])[-2:] if not latestfiles:
>> logger.error('No sigdata files found matching %s %s' %
>> (pn, taskname)) sys.exit(1)
>> - elif len(latestfiles) < 2:
>> - logger.error('Only one matching sigdata file found for
>> the specified task (%s %s)' % (pn, taskname))
>> - sys.exit(1)
>>
>> - # Define recursion callback
>> - def recursecb(key, hash1, hash2):
>> - hashes = [hash1, hash2]
>> - hashfiles = find_siginfo(bbhandler, key, None, hashes)
>> -
>> - recout = []
>> - if len(hashfiles) == 0:
>> - recout.append("Unable to find matching sigdata for %s
>> with hashes %s or %s" % (key, hash1, hash2))
>> - elif not hash1 in hashfiles:
>> - recout.append("Unable to find matching sigdata for %s
>> with hash %s" % (key, hash1))
>> - elif not hash2 in hashfiles:
>> - recout.append("Unable to find matching sigdata for %s
>> with hash %s" % (key, hash2))
>> - else:
>> - out2 = bb.siggen.compare_sigfiles(hashfiles[hash1],
>> hashfiles[hash2], recursecb, color=color)
>> - for change in out2:
>> - for line in change.splitlines():
>> - recout.append(' ' + line)
>> + return latestfiles
>> +
>>
>> - return recout
>> +# Define recursion callback
>> +def recursecb(key, hash1, hash2):
>> + hashes = [hash1, hash2]
>> + hashfiles = find_siginfo(tinfoil, key, None, hashes)
>>
>> - # Recurse into signature comparison
>> - logger.debug("Signature file (previous): %s" % latestfiles[-2])
>> - logger.debug("Signature file (latest): %s" % latestfiles[-1])
>> - output = bb.siggen.compare_sigfiles(latestfiles[-2],
>> latestfiles[-1], recursecb, color=color)
>> - if output:
>> - print('\n'.join(output))
>> - sys.exit(0)
>> + recout = []
>> + if len(hashfiles) == 0:
>> + recout.append("Unable to find matching sigdata for %s with
>> hashes %s or %s" % (key, hash1, hash2))
>> + elif not hash1 in hashfiles:
>> + recout.append("Unable to find matching sigdata for %s with
>> hash %s" % (key, hash1))
>> + elif not hash2 in hashfiles:
>> + recout.append("Unable to find matching sigdata for %s with
>> hash %s" % (key, hash2))
>> + else:
>> + out2 = bb.siggen.compare_sigfiles(hashfiles[hash1],
>> hashfiles[hash2], recursecb, color=color)
>> + for change in out2:
>> + for line in change.splitlines():
>> + recout.append(' ' + line)
>>
>> + return recout
>>
>>
>> parser = argparse.ArgumentParser(
>> - description="Compares siginfo/sigdata files written out by
>> BitBake")
>> + description=("Dumps" if is_dump else "Compares") + "
>> siginfo/sigdata files written out by BitBake")
>> -parser.add_argument('-d', '--debug',
>> +parser.add_argument('-D', '--debug',
>> help='Enable debug output',
>> action='store_true')
>>
>> -parser.add_argument('--color',
>> - help='Colorize output (where %(metavar)s is %(choices)s)',
>> - choices=['auto', 'always', 'never'], default='auto',
>> metavar='color') +if is_dump:
>> + parser.add_argument("-t", "--task",
>> + help="find the signature data file for the last run of
>> the specified task",
>> + action="store", dest="taskargs", nargs=2,
>> metavar=('recipename', 'taskname')) +
>> + parser.add_argument("sigdatafile1",
>> + help="Signature file to dump. Not used when using
>> -t/--task.",
>> + action="store", nargs='?', metavar="sigdatafile")
>> +else:
>> + parser.add_argument('-c', '--color',
>> + help='Colorize the output (where %(metavar)s is
>> %(choices)s)',
>> + choices=['auto', 'always', 'never'], default='auto',
>> metavar='color')
>> -parser.add_argument("-t", "--task",
>> - help="find the signature data files for last two runs of the
>> specified task and compare them",
>> - action="store", dest="taskargs", nargs=2,
>> metavar=('recipename', 'taskname'))
>> + parser.add_argument('-d', '--dump',
>> + help='Dump the last signature data instead of comparing
>> (equivalent to using bitbake-dumpsig)',
>> + action='store_true')
>>
>> -parser.add_argument("-s", "--signature",
>> - help="With -t/--task, specify the signatures to look for
>> instead of taking the last two",
>> - action="store", dest="sigargs", nargs=2, metavar=('fromsig',
>> 'tosig'))
>> + parser.add_argument("-t", "--task",
>> + help="find the signature data files for the last two
>> runs of the specified task and compare them",
>> + action="store", dest="taskargs", nargs=2,
>> metavar=('recipename', 'taskname'))
>> -parser.add_argument("sigdatafile1",
>> - help="First signature file to compare (or signature file to
>> dump, if second not specified). Not used when using -t/--task.",
>> - action="store", nargs='?')
>> + parser.add_argument("-s", "--signature",
>> + help="With -t/--task, specify the signatures to look for
>> instead of taking the last two",
>> + action="store", dest="sigargs", nargs=2,
>> metavar=('fromsig', 'tosig'))
>> -parser.add_argument("sigdatafile2",
>> - help="Second signature file to compare",
>> - action="store", nargs='?')
>> + parser.add_argument("sigdatafile1",
>> + help="First signature file to compare (or signature file
>> to dump, if second not specified). Not used when using -t/--task.",
>> + action="store", nargs='?')
>>
>> + parser.add_argument("sigdatafile2",
>> + help="Second signature file to compare",
>> + action="store", nargs='?')
>>
>> options = parser.parse_args()
>> +if is_dump:
>> + options.color = 'never'
>> + options.dump = True
>> + options.sigdatafile2 = None
>> + options.sigargs = None
>>
>> if options.debug:
>> logger.setLevel(logging.DEBUG)
>> @@ -155,17 +155,32 @@ color = (options.color == 'always' or
>> (options.color == 'auto' and sys.stdout.is if options.taskargs:
>> with bb.tinfoil.Tinfoil() as tinfoil:
>> tinfoil.prepare(config_only=True)
>> - if options.sigargs:
>> - find_compare_task(tinfoil, options.taskargs[0],
>> options.taskargs[1], options.sigargs[0], options.sigargs[1],
>> color=color)
>> + if not options.dump and options.sigargs:
>> + files = find_siginfo_task(tinfoil, options.taskargs[0],
>> options.taskargs[1], options.sigargs[0], options.sigargs[1])
>> + else:
>> + files = find_siginfo_task(tinfoil, options.taskargs[0],
>> options.taskargs[1]) +
>> + if options.dump:
>> + logger.debug("Signature file: %s" % files[-1])
>> + output = bb.siggen.dump_sigfile(files[-1])
>> else:
>> - find_compare_task(tinfoil, options.taskargs[0],
>> options.taskargs[1], color=color)
>> + if len(files) < 2:
>> + logger.error('Only one matching sigdata file found
>> for the specified task (%s %s)' % (options.taskargs[0],
>> options.taskargs[1]))
>> + sys.exit(1)
>> +
>> + # Recurse into signature comparison
>> + logger.debug("Signature file (previous): %s" % files[-2])
>> + logger.debug("Signature file (latest): %s" % files[-1])
>> + output = bb.siggen.compare_sigfiles(files[-2],
>> files[-1], recursecb, color=color) else:
>> if options.sigargs:
>> logger.error('-s/--signature can only be used together with
>> -t/--task') sys.exit(1)
>> try:
>> - if options.sigdatafile1 and options.sigdatafile2:
>> - output =
>> bb.siggen.compare_sigfiles(options.sigdatafile1,
>> options.sigdatafile2, color=color)
>> + if not options.dump and options.sigdatafile1 and
>> options.sigdatafile2:
>> + with bb.tinfoil.Tinfoil() as tinfoil:
>> + tinfoil.prepare(config_only=True)
>> + output =
>> bb.siggen.compare_sigfiles(options.sigdatafile1,
>> options.sigdatafile2, recursecb, color=color) elif
>> options.sigdatafile1: output =
>> bb.siggen.dump_sigfile(options.sigdatafile1) else: @@ -179,5 +194,5
>> @@ else: logger.error('Invalid signature data - ensure you are
>> specifying sigdata/siginfo files') sys.exit(1)
>>
>> - if output:
>> - print('\n'.join(output))
>> +if output:
>> + print('\n'.join(output))
>> diff --git a/bitbake/bin/bitbake-dumpsig b/bitbake/bin/bitbake-dumpsig
>> deleted file mode 100755
>> index 95ebd93..0000000
>> --- a/bitbake/bin/bitbake-dumpsig
>> +++ /dev/null
>> @@ -1,94 +0,0 @@
>> -#!/usr/bin/env python3
>> -
>> -# bitbake-dumpsig
>> -# BitBake task signature dump utility
>> -#
>> -# Copyright (C) 2013 Intel Corporation
>> -#
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. -
>> -import os
>> -import sys
>> -import warnings
>> -import optparse
>> -import logging
>> -import pickle
>> -
>> -sys.path.insert(0,
>> os.path.join(os.path.dirname(os.path.dirname(sys.argv[0])), 'lib')) -
>> -import bb.tinfoil
>> -import bb.siggen
>> -import bb.msg
>> -
>> -logger = bb.msg.logger_create('bitbake-dumpsig')
>> -
>> -def find_siginfo_task(bbhandler, pn, taskname):
>> - """ Find the most recent signature file for the specified
>> PN/task """ -
>> - if not hasattr(bb.siggen, 'find_siginfo'):
>> - logger.error('Metadata does not support finding signature
>> data files')
>> - sys.exit(1)
>> -
>> - if not taskname.startswith('do_'):
>> - taskname = 'do_%s' % taskname
>> -
>> - filedates = bb.siggen.find_siginfo(pn, taskname, None,
>> bbhandler.config_data)
>> - latestfiles = sorted(filedates.keys(), key=lambda f:
>> filedates[f])[-1:]
>> - if not latestfiles:
>> - logger.error('No sigdata files found matching %s %s' % (pn,
>> taskname))
>> - sys.exit(1)
>> -
>> - return latestfiles[0]
>> -
>> -parser = optparse.OptionParser(
>> - description = "Dumps siginfo/sigdata files written out by
>> BitBake",
>> - usage = """
>> - %prog -t recipename taskname
>> - %prog sigdatafile""")
>> -
>> -parser.add_option("-D", "--debug",
>> - help = "enable debug",
>> - action = "store_true", dest="debug", default = False)
>> -
>> -parser.add_option("-t", "--task",
>> - help = "find the signature data file for the specified task",
>> - action="store", dest="taskargs", nargs=2,
>> metavar='recipename taskname') -
>> -options, args = parser.parse_args(sys.argv)
>> -
>> -if options.debug:
>> - logger.setLevel(logging.DEBUG)
>> -
>> -if options.taskargs:
>> - tinfoil = bb.tinfoil.Tinfoil()
>> - tinfoil.prepare(config_only = True)
>> - file = find_siginfo_task(tinfoil, options.taskargs[0],
>> options.taskargs[1])
>> - logger.debug("Signature file: %s" % file)
>> -elif len(args) == 1:
>> - parser.print_help()
>> - sys.exit(0)
>> -else:
>> - file = args[1]
>> -
>> -try:
>> - output = bb.siggen.dump_sigfile(file)
>> -except IOError as e:
>> - logger.error(str(e))
>> - sys.exit(1)
>> -except (pickle.UnpicklingError, EOFError):
>> - logger.error('Invalid signature data - ensure you are specifying
>> a sigdata/siginfo file')
>> - sys.exit(1)
>> -
>> -if output:
>> - print('\n'.join(output))
>> diff --git a/bitbake/bin/bitbake-dumpsig b/bitbake/bin/bitbake-dumpsig
>> new file mode 120000
>> index 0000000..b1e8489
>> --- /dev/null
>> +++ b/bitbake/bin/bitbake-dumpsig
>> @@ -0,0 +1 @@
>> +bitbake-diffsigs
>> \ No newline at end of file
>> diff --git a/bitbake/bin/bitbake-hashclient
>> b/bitbake/bin/bitbake-hashclient new file mode 100755
>> index 0000000..29ab65f
>> --- /dev/null
>> +++ b/bitbake/bin/bitbake-hashclient
>> @@ -0,0 +1,170 @@
>> +#! /usr/bin/env python3
>> +#
>> +# Copyright (C) 2019 Garmin Ltd.
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> +import argparse
>> +import hashlib
>> +import logging
>> +import os
>> +import pprint
>> +import sys
>> +import threading
>> +import time
>> +
>> +try:
>> + import tqdm
>> + ProgressBar = tqdm.tqdm
>> +except ImportError:
>> + class ProgressBar(object):
>> + def __init__(self, *args, **kwargs):
>> + pass
>> +
>> + def __enter__(self):
>> + return self
>> +
>> + def __exit__(self, *args, **kwargs):
>> + pass
>> +
>> + def update(self):
>> + pass
>> +
>> +sys.path.insert(0,
>> os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib')) +
>> +import hashserv
>> +
>> +DEFAULT_ADDRESS = 'unix://./hashserve.sock'
>> +METHOD = 'stress.test.method'
>> +
>> +
>> +def main():
>> + def handle_stats(args, client):
>> + if args.reset:
>> + s = client.reset_stats()
>> + else:
>> + s = client.get_stats()
>> + pprint.pprint(s)
>> + return 0
>> +
>> + def handle_stress(args, client):
>> + def thread_main(pbar, lock):
>> + nonlocal found_hashes
>> + nonlocal missed_hashes
>> + nonlocal max_time
>> +
>> + client = hashserv.create_client(args.address)
>> +
>> + for i in range(args.requests):
>> + taskhash = hashlib.sha256()
>> + taskhash.update(args.taskhash_seed.encode('utf-8'))
>> + taskhash.update(str(i).encode('utf-8'))
>> +
>> + start_time = time.perf_counter()
>> + l = client.get_unihash(METHOD, taskhash.hexdigest())
>> + elapsed = time.perf_counter() - start_time
>> +
>> + with lock:
>> + if l:
>> + found_hashes += 1
>> + else:
>> + missed_hashes += 1
>> +
>> + max_time = max(elapsed, max_time)
>> + pbar.update()
>> +
>> + max_time = 0
>> + found_hashes = 0
>> + missed_hashes = 0
>> + lock = threading.Lock()
>> + total_requests = args.clients * args.requests
>> + start_time = time.perf_counter()
>> + with ProgressBar(total=total_requests) as pbar:
>> + threads = [threading.Thread(target=thread_main,
>> args=(pbar, lock), daemon=False) for _ in range(args.clients)]
>> + for t in threads:
>> + t.start()
>> +
>> + for t in threads:
>> + t.join()
>> +
>> + elapsed = time.perf_counter() - start_time
>> + with lock:
>> + print("%d requests in %.1fs. %.1f requests per second" %
>> (total_requests, elapsed, total_requests / elapsed))
>> + print("Average request time %.8fs" % (elapsed /
>> total_requests))
>> + print("Max request time was %.8fs" % max_time)
>> + print("Found %d hashes, missed %d" % (found_hashes,
>> missed_hashes)) +
>> + if args.report:
>> + with ProgressBar(total=args.requests) as pbar:
>> + for i in range(args.requests):
>> + taskhash = hashlib.sha256()
>> +
>> taskhash.update(args.taskhash_seed.encode('utf-8'))
>> + taskhash.update(str(i).encode('utf-8'))
>> +
>> + outhash = hashlib.sha256()
>> + outhash.update(args.outhash_seed.encode('utf-8'))
>> + outhash.update(str(i).encode('utf-8'))
>> +
>> + client.report_unihash(taskhash.hexdigest(),
>> METHOD, outhash.hexdigest(), taskhash.hexdigest()) +
>> + with lock:
>> + pbar.update()
>> +
>> + parser = argparse.ArgumentParser(description='Hash Equivalence
>> Client')
>> + parser.add_argument('--address', default=DEFAULT_ADDRESS,
>> help='Server address (default "%(default)s")')
>> + parser.add_argument('--log', default='WARNING', help='Set
>> logging level') +
>> + subparsers = parser.add_subparsers()
>> +
>> + stats_parser = subparsers.add_parser('stats', help='Show server
>> stats')
>> + stats_parser.add_argument('--reset', action='store_true',
>> + help='Reset server stats')
>> + stats_parser.set_defaults(func=handle_stats)
>> +
>> + stress_parser = subparsers.add_parser('stress', help='Run stress
>> test')
>> + stress_parser.add_argument('--clients', type=int, default=10,
>> + help='Number of simultaneous clients')
>> + stress_parser.add_argument('--requests', type=int, default=1000,
>> + help='Number of requests each client
>> will perform')
>> + stress_parser.add_argument('--report', action='store_true',
>> + help='Report new hashes')
>> + stress_parser.add_argument('--taskhash-seed', default='',
>> + help='Include string in taskhash')
>> + stress_parser.add_argument('--outhash-seed', default='',
>> + help='Include string in outhash')
>> + stress_parser.set_defaults(func=handle_stress)
>> +
>> + args = parser.parse_args()
>> +
>> + logger = logging.getLogger('hashserv')
>> +
>> + level = getattr(logging, args.log.upper(), None)
>> + if not isinstance(level, int):
>> + raise ValueError('Invalid log level: %s' % args.log)
>> +
>> + logger.setLevel(level)
>> + console = logging.StreamHandler()
>> + console.setLevel(level)
>> + logger.addHandler(console)
>> +
>> + func = getattr(args, 'func', None)
>> + if func:
>> + client = hashserv.create_client(args.address)
>> + # Try to establish a connection to the server now to detect
>> failures
>> + # early
>> + client.connect()
>> +
>> + return func(args, client)
>> +
>> + return 0
>> +
>> +
>> +if __name__ == '__main__':
>> + try:
>> + ret = main()
>> + except Exception:
>> + ret = 1
>> + import traceback
>> + traceback.print_exc()
>> + sys.exit(ret)
>> diff --git a/bitbake/bin/bitbake-hashserv
>> b/bitbake/bin/bitbake-hashserv new file mode 100755
>> index 0000000..1bc1f91
>> --- /dev/null
>> +++ b/bitbake/bin/bitbake-hashserv
>> @@ -0,0 +1,62 @@
>> +#! /usr/bin/env python3
>> +#
>> +# Copyright (C) 2018 Garmin Ltd.
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> +import os
>> +import sys
>> +import logging
>> +import argparse
>> +import sqlite3
>> +
>> +sys.path.insert(0,
>> os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib')) +
>> +import hashserv
>> +
>> +VERSION = "1.0.0"
>> +
>> +DEFAULT_BIND = 'unix://./hashserve.sock'
>> +
>> +
>> +def main():
>> + parser = argparse.ArgumentParser(description='Hash Equivalence
>> Reference Server. Version=%s' % VERSION,
>> + epilog='''The bind address is
>> the path to a unix domain socket if it is
>> + prefixed with
>> "unix://". Otherwise, it is an IP address
>> + and port in form
>> ADDRESS:PORT. To bind to all addresses, leave
>> + the ADDRESS empty,
>> e.g. "--bind :8686". To bind to a specific
>> + IPv6 address, enclose
>> the address in "[]", e.g.
>> + "--bind [::1]:8686"'''
>> + )
>> +
>> + parser.add_argument('--bind', default=DEFAULT_BIND, help='Bind
>> address (default "%(default)s")')
>> + parser.add_argument('--database', default='./hashserv.db',
>> help='Database file (default "%(default)s")')
>> + parser.add_argument('--log', default='WARNING', help='Set
>> logging level') +
>> + args = parser.parse_args()
>> +
>> + logger = logging.getLogger('hashserv')
>> +
>> + level = getattr(logging, args.log.upper(), None)
>> + if not isinstance(level, int):
>> + raise ValueError('Invalid log level: %s' % args.log)
>> +
>> + logger.setLevel(level)
>> + console = logging.StreamHandler()
>> + console.setLevel(level)
>> + logger.addHandler(console)
>> +
>> + server = hashserv.create_server(args.bind, args.database)
>> + server.serve_forever()
>> + return 0
>> +
>> +
>> +if __name__ == '__main__':
>> + try:
>> + ret = main()
>> + except Exception:
>> + ret = 1
>> + import traceback
>> + traceback.print_exc()
>> + sys.exit(ret)
>> diff --git a/bitbake/bin/bitbake-layers b/bitbake/bin/bitbake-layers
>> index d184011..a884dc1 100755
>> --- a/bitbake/bin/bitbake-layers
>> +++ b/bitbake/bin/bitbake-layers
>> @@ -7,18 +7,8 @@
>> # Copyright (C) 2011 Mentor Graphics Corporation
>> # Copyright (C) 2011-2015 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import logging
>> import os
>> diff --git a/bitbake/bin/bitbake-prserv b/bitbake/bin/bitbake-prserv
>> index f38d2dd..1e9b6cb 100755
>> --- a/bitbake/bin/bitbake-prserv
>> +++ b/bitbake/bin/bitbake-prserv
>> @@ -1,4 +1,8 @@
>> #!/usr/bin/env python3
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> import os
>> import sys,logging
>> import optparse
>> diff --git a/bitbake/bin/bitbake-selftest
>> b/bitbake/bin/bitbake-selftest index cfa7ac5..041a271 100755
>> --- a/bitbake/bin/bitbake-selftest
>> +++ b/bitbake/bin/bitbake-selftest
>> @@ -2,18 +2,8 @@
>> #
>> # Copyright (C) 2012 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import os
>> import sys, logging
>> @@ -22,6 +12,7 @@ sys.path.insert(0,
>> os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib import
>> unittest try:
>> import bb
>> + import hashserv
>> import layerindexlib
>> except RuntimeError as exc:
>> sys.exit(str(exc))
>> @@ -33,7 +24,10 @@ tests = ["bb.tests.codeparser",
>> "bb.tests.event",
>> "bb.tests.fetch",
>> "bb.tests.parse",
>> + "bb.tests.persist_data",
>> + "bb.tests.runqueue",
>> "bb.tests.utils",
>> + "hashserv.tests",
>> "layerindexlib.tests.layerindexobj",
>> "layerindexlib.tests.restapi",
>> "layerindexlib.tests.cooker"]
>> diff --git a/bitbake/bin/bitbake-worker b/bitbake/bin/bitbake-worker
>> index e925054..6776cad 100755
>> --- a/bitbake/bin/bitbake-worker
>> +++ b/bitbake/bin/bitbake-worker
>> @@ -1,4 +1,7 @@
>> #!/usr/bin/env python3
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>>
>> import os
>> import sys
>> @@ -136,7 +139,7 @@ def sigterm_handler(signum, frame):
>> os.killpg(0, signal.SIGTERM)
>> sys.exit()
>>
>> -def fork_off_task(cfg, data, databuilder, workerdata, fn, task,
>> taskname, appends, taskdepdata, extraconfigdata, quieterrors=False,
>> dry_run_exec=False): +def fork_off_task(cfg, data, databuilder,
>> workerdata, fn, task, taskname, taskhash, unihash, appends,
>> taskdepdata, extraconfigdata, quieterrors=False, dry_run_exec=False):
>> # We need to setup the environment BEFORE the fork, since # a fork()
>> or exec*() activates PSEUDO... @@ -231,10 +234,13 @@ def
>> fork_off_task(cfg, data, databuilder, workerdata, fn, task, taskname,
>> append the_data.setVar(varname, value)
>> bb.parse.siggen.set_taskdata(workerdata["sigdata"])
>> + if "newhashes" in workerdata:
>> +
>> bb.parse.siggen.set_taskhashes(workerdata["newhashes"]) ret = 0
>>
>> the_data = bb_cache.loadDataFull(fn, appends)
>> - the_data.setVar('BB_TASKHASH',
>> workerdata["runq_hash"][task])
>> + the_data.setVar('BB_TASKHASH', taskhash)
>> + the_data.setVar('BB_UNIHASH', unihash)
>>
>> bb.utils.set_process_name("%s:%s" %
>> (the_data.getVar("PN"), taskname.replace("do_", "")))
>> @@ -373,6 +379,7 @@ class BitbakeWorker(object):
>> self.handle_item(b"cookerconfig",
>> self.handle_cookercfg) self.handle_item(b"extraconfigdata",
>> self.handle_extraconfigdata) self.handle_item(b"workerdata",
>> self.handle_workerdata)
>> + self.handle_item(b"newtaskhashes",
>> self.handle_newtaskhashes) self.handle_item(b"runtask",
>> self.handle_runtask) self.handle_item(b"finishnow",
>> self.handle_finishnow) self.handle_item(b"ping", self.handle_ping)
>> @@ -411,6 +418,10 @@ class BitbakeWorker(object):
>> bb.msg.loggerDefaultDomains =
>> self.workerdata["logdefaultdomain"] for mc in self.databuilder.mcdata:
>> self.databuilder.mcdata[mc].setVar("PRSERV_HOST",
>> self.workerdata["prhost"])
>> + self.databuilder.mcdata[mc].setVar("BB_HASHSERVE",
>> self.workerdata["hashservaddr"]) +
>> + def handle_newtaskhashes(self, data):
>> + self.workerdata["newhashes"] = pickle.loads(data)
>>
>> def handle_ping(self, _):
>> workerlog_write("Handling ping\n")
>> @@ -425,10 +436,10 @@ class BitbakeWorker(object):
>> sys.exit(0)
>>
>> def handle_runtask(self, data):
>> - fn, task, taskname, quieterrors, appends, taskdepdata,
>> dry_run_exec = pickle.loads(data)
>> + fn, task, taskname, taskhash, unihash, quieterrors, appends,
>> taskdepdata, dry_run_exec = pickle.loads(data)
>> workerlog_write("Handling runtask %s %s %s\n" % (task, fn, taskname))
>> - pid, pipein, pipeout = fork_off_task(self.cookercfg,
>> self.data, self.databuilder, self.workerdata, fn, task, taskname,
>> appends, taskdepdata, self.extraconfigdata, quieterrors, dry_run_exec)
>> + pid, pipein, pipeout = fork_off_task(self.cookercfg,
>> self.data, self.databuilder, self.workerdata, fn, task, taskname,
>> taskhash, unihash, appends, taskdepdata, self.extraconfigdata,
>> quieterrors, dry_run_exec) self.build_pids[pid] = task
>> self.build_pipes[pid] = runQueueWorkerPipe(pipein, pipeout) diff
>> --git a/bitbake/bin/bitdoc b/bitbake/bin/bitdoc index
>> 2744678..9bd02be 100755 --- a/bitbake/bin/bitdoc
>> +++ b/bitbake/bin/bitdoc
>> @@ -1,21 +1,9 @@
>> #!/usr/bin/env python3
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # Copyright (C) 2005 Holger Hans Peter Freyther
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import optparse, os, sys
>>
>> diff --git a/bitbake/bin/git-make-shallow
>> b/bitbake/bin/git-make-shallow index 296d3a3..57069f7 100755
>> --- a/bitbake/bin/git-make-shallow
>> +++ b/bitbake/bin/git-make-shallow
>> @@ -1,4 +1,8 @@
>> #!/usr/bin/env python3
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> """git-make-shallow: make the current git repository shallow
>>
>> Remove the history of the specified revisions, then optionally
>> filter the diff --git a/bitbake/bin/toaster b/bitbake/bin/toaster
>> index ecf66fa..c3472df 100755
>> --- a/bitbake/bin/toaster
>> +++ b/bitbake/bin/toaster
>> @@ -3,19 +3,9 @@
>> # toaster - shell script to start Toaster
>>
>> # Copyright (C) 2013-2015 Intel Corp.
>> -
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License as
>> published by -# the Free Software Foundation; either version 2 of the
>> License, or -# (at your option) any later version.
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# SPDX-License-Identifier: GPL-2.0-or-later
>> #
>> -# You should have received a copy of the GNU General Public License
>> -# along with this program. If not, see http://www.gnu.org/licenses/.
>>
>> HELP="
>> Usage: source toaster start|stop [webport=<address:port>] [noweb]
>> [nobuild] [toasterdir] diff --git a/bitbake/bin/toaster-eventreplay
>> b/bitbake/bin/toaster-eventreplay index 80967a0..8fa4ab7 100755
>> --- a/bitbake/bin/toaster-eventreplay
>> +++ b/bitbake/bin/toaster-eventreplay
>> @@ -1,25 +1,12 @@
>> #!/usr/bin/env python3
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # Copyright (C) 2014 Alex Damian
>> #
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> # This file re-uses code spread throughout other Bitbake source
>> files. # As such, all other copyrights belong to their own right
>> holders. #
>> -#
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> """
>> This command takes a filename as a single parameter. The filename is
>> read diff --git a/bitbake/classes/base.bbclass
>> b/bitbake/classes/base.bbclass index 71d9ec1..08441fe 100644
>> --- a/bitbake/classes/base.bbclass
>> +++ b/bitbake/classes/base.bbclass
>> @@ -44,7 +44,7 @@ python do_showdata() {
>> # emit the metadata which isnt valid shell
>> for e in bb.data.keys(d):
>> if d.getVarFlag(e, 'python', False):
>> - bb.plain("\npython %s () {\n%s}" % (e,
>> d.getVar(e, True)))
>> + bb.plain("\npython %s () {\n%s}" % (e,
>> d.getVar(e))) }
>>
>> addtask listtasks
>> diff --git a/bitbake/contrib/dump_cache.py
>> b/bitbake/contrib/dump_cache.py index 8963ca4..c6723cb 100755
>> --- a/bitbake/contrib/dump_cache.py
>> +++ b/bitbake/contrib/dump_cache.py
>> @@ -1,6 +1,4 @@
>> #!/usr/bin/env python3
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # Copyright (C) 2012, 2018 Wind River Systems, Inc.
>> #
>> diff --git
>> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
>> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
>> index f1caaec..46dafee 100644 ---
>> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
>> +++
>> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
>> @@ -31,7 +31,7 @@ <para> Prior to executing BitBake, you should take
>> advantage of available parallel thread execution on your build host
>> by setting the
>> - <link
>> linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
>> + <link
>> linkend='var-bb-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
>> variable in your project's <filename>local.conf</filename>
>> configuration file. </para>
>> @@ -87,9 +87,9 @@
>> <para>
>> The <filename>layer.conf</filename> files are used to
>> construct key variables such as
>> - <link
>> linkend='var-BBPATH'><filename>BBPATH</filename></link>
>> + <link
>> linkend='var-bb-BBPATH'><filename>BBPATH</filename></link> and
>> - <link
>> linkend='var-BBFILES'><filename>BBFILES</filename></link>.
>> + <link
>> linkend='var-bb-BBFILES'><filename>BBFILES</filename></link>.
>> <filename>BBPATH</filename> is used to search for configuration and
>> class files under the <filename>conf</filename> and
>> <filename>classes</filename> @@ -117,19 +117,19 @@
>> at certain variables, including:
>> <itemizedlist>
>> <listitem><para>
>> - <link
>> linkend='var-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>
>> + <link
>> linkend='var-bb-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>
>> </para></listitem> <listitem><para>
>> - <link
>> linkend='var-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link>
>> + <link
>> linkend='var-bb-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link>
>> </para></listitem> <listitem><para>
>> - <link
>> linkend='var-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link>
>> + <link
>> linkend='var-bb-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link>
>> </para></listitem> <listitem><para>
>> - <link
>> linkend='var-BB_ORIGENV'><filename>BB_ORIGENV</filename></link>
>> + <link
>> linkend='var-bb-BB_ORIGENV'><filename>BB_ORIGENV</filename></link>
>> </para></listitem> <listitem><para>
>> - <link
>> linkend='var-BITBAKE_UI'><filename>BITBAKE_UI</filename></link>
>> + <link
>> linkend='var-bb-BITBAKE_UI'><filename>BITBAKE_UI</filename></link>
>> </para></listitem> </itemizedlist>
>> The first four variables in this list relate to how
>> BitBake treats shell @@ -156,7 +156,7 @@
>> BitBake first searches the current working directory for
>> an optional <filename>conf/bblayers.conf</filename> configuration
>> file. This file is expected to contain a
>> - <link
>> linkend='var-BBLAYERS'><filename>BBLAYERS</filename></link>
>> + <link
>> linkend='var-bb-BBLAYERS'><filename>BBLAYERS</filename></link>
>> variable that is a space-delimited list of 'layer' directories.
>> Recall that if BitBake cannot find a
>> <filename>bblayers.conf</filename> file, then it is assumed the user
>> has set the <filename>BBPATH</filename> @@ -166,10 +166,10 @@ <para>
>> For each directory (layer) in this list, a
>> <filename>conf/layer.conf</filename> file is located and parsed with
>> the
>> - <link
>> linkend='var-LAYERDIR'><filename>LAYERDIR</filename></link>
>> + <link
>> linkend='var-bb-LAYERDIR'><filename>LAYERDIR</filename></link>
>> variable being set to the directory where the layer was found. The
>> idea is these files automatically set up
>> - <link
>> linkend='var-BBPATH'><filename>BBPATH</filename></link>
>> + <link
>> linkend='var-bb-BBPATH'><filename>BBPATH</filename></link> and other
>> variables correctly for a given build directory. </para>
>>
>> @@ -189,7 +189,7 @@
>> depending on the environment variables previously
>> mentioned or set in the configuration files.
>> The
>> - "<link linkend='ref-variables-glos'>Variables
>> Glossary</link>"
>> + "<link linkend='ref-bb-variables-glos'>Variables
>> Glossary</link>" chapter presents a full list of variables.
>> </para>
>>
>> @@ -204,7 +204,7 @@
>> <para>
>> The <filename>base.bbclass</filename> file is always
>> included. Other classes that are specified in the configuration using
>> the
>> - <link
>> linkend='var-INHERIT'><filename>INHERIT</filename></link>
>> + <link
>> linkend='var-bb-INHERIT'><filename>INHERIT</filename></link> variable
>> are also included. BitBake searches for class files in a
>> <filename>classes</filename> subdirectory under
>> @@ -270,7 +270,7 @@
>>
>> <para>
>> During the configuration phase, BitBake will have set
>> - <link
>> linkend='var-BBFILES'><filename>BBFILES</filename></link>.
>> + <link
>> linkend='var-bb-BBFILES'><filename>BBFILES</filename></link>. BitBake
>> now uses it to construct a list of recipes to parse, along with any
>> append files (<filename>.bbappend</filename>) to apply.
>> @@ -292,7 +292,7 @@
>> Any inherit statements cause BitBake to find and
>> then parse class files (<filename>.bbclass</filename>)
>> using
>> - <link
>> linkend='var-BBPATH'><filename>BBPATH</filename></link>
>> + <link
>> linkend='var-bb-BBPATH'><filename>BBPATH</filename></link> as the
>> search path. Finally, BitBake parses in order any append files found
>> in <filename>BBFILES</filename>.
>> @@ -303,8 +303,8 @@
>> pieces of metadata.
>> For example, in <filename>bitbake.conf</filename> the
>> recipe name and version are used to set the variables
>> - <link linkend='var-PN'><filename>PN</filename></link> and
>> - <link linkend='var-PV'><filename>PV</filename></link>:
>> + <link linkend='var-bb-PN'><filename>PN</filename></link>
>> and
>> + <link linkend='var-bb-PV'><filename>PV</filename></link>:
>> <literallayout class='monospaced'>
>> PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE',
>> False),d)[0] or 'defaultpkgname'}" PV =
>> "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[1]
>> or '1.0'}" @@ -336,7 +336,7 @@ recipe information.
>> The validity of this cache is determined by first
>> computing a checksum of the base configuration data (see
>> - <link
>> linkend='var-BB_HASHCONFIG_WHITELIST'><filename>BB_HASHCONFIG_WHITELIST</filename></link>)
>> + <link
>> linkend='var-bb-BB_HASHCONFIG_WHITELIST'><filename>BB_HASHCONFIG_WHITELIST</filename></link>)
>> and then checking if the checksum matches. If that checksum matches
>> what is in the cache and the recipe and class files have not changed,
>> Bitbake is able to use @@ -384,9 +384,9 @@
>> the recipe can be known.
>> Each recipe's <filename>PROVIDES</filename> list is
>> created implicitly through the recipe's
>> - <link linkend='var-PN'><filename>PN</filename></link>
>> variable
>> + <link linkend='var-bb-PN'><filename>PN</filename></link>
>> variable and explicitly through the recipe's
>> - <link
>> linkend='var-PROVIDES'><filename>PROVIDES</filename></link>
>> + <link
>> linkend='var-bb-PROVIDES'><filename>PROVIDES</filename></link>
>> variable, which is optional. </para>
>>
>> @@ -427,7 +427,7 @@
>> PREFERRED_PROVIDER_virtual/kernel = "linux-yocto"
>> </literallayout>
>> The default
>> - <link
>> linkend='var-PREFERRED_PROVIDER'><filename>PREFERRED_PROVIDER</filename></link>
>> + <link
>> linkend='var-bb-PREFERRED_PROVIDER'><filename>PREFERRED_PROVIDER</filename></link>
>> is the provider with the same name as the target. Bitbake iterates
>> through each target it needs to build and resolves them and their
>> dependencies using this process. @@ -439,10 +439,10 @@
>> BitBake defaults to the highest version of a provider.
>> Version comparisons are made using the same method as
>> Debian. You can use the
>> - <link
>> linkend='var-PREFERRED_VERSION'><filename>PREFERRED_VERSION</filename></link>
>> + <link
>> linkend='var-bb-PREFERRED_VERSION'><filename>PREFERRED_VERSION</filename></link>
>> variable to specify a particular version. You can influence the order
>> by using the
>> - <link
>> linkend='var-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link>
>> + <link
>> linkend='var-bb-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link>
>> variable. </para>
>>
>> @@ -464,7 +464,7 @@
>> BitBake defaults to selecting the most recent
>> version, unless otherwise specified.
>> If the recipe in question has a
>> - <link
>> linkend='var-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link>
>> + <link
>> linkend='var-bb-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link>
>> set lower than the other recipes (default is 0), then it will not be
>> selected. This allows the person or persons maintaining
>> @@ -475,9 +475,9 @@
>>
>> <para>
>> If the first recipe is named
>> <filename>a_1.1.bb</filename>, then the
>> - <link linkend='var-PN'><filename>PN</filename></link>
>> variable
>> + <link linkend='var-bb-PN'><filename>PN</filename></link>
>> variable will be set to “a”, and the
>> - <link linkend='var-PV'><filename>PV</filename></link>
>> + <link linkend='var-bb-PV'><filename>PV</filename></link>
>> variable will be set to 1.1.
>> </para>
>>
>> @@ -532,11 +532,11 @@
>> <para>
>> Dependencies are defined through several variables.
>> You can find information about variables BitBake uses in
>> - the <link linkend='ref-variables-glos'>Variables
>> Glossary</link>
>> + the <link linkend='ref-bb-variables-glos'>Variables
>> Glossary</link> near the end of this manual.
>> At a basic level, it is sufficient to know that BitBake
>> uses the
>> - <link
>> linkend='var-DEPENDS'><filename>DEPENDS</filename></link> and
>> - <link
>> linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link> variables
>> when
>> + <link
>> linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link> and
>> + <link
>> linkend='var-bb-RDEPENDS'><filename>RDEPENDS</filename></link>
>> variables when calculating dependencies. </para>
>>
>> @@ -560,7 +560,7 @@
>>
>> <para>
>> The build now starts with BitBake forking off threads up
>> to the limit set in the
>> - <link
>> linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
>> + <link
>> linkend='var-bb-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
>> variable. BitBake continues to fork threads as long as there are
>> tasks ready to run, those tasks have all their dependencies met, and
>> the thread threshold has not been @@ -574,7 +574,7 @@
>>
>> <para>
>> As each task completes, a timestamp is written to the
>> directory specified by the
>> - <link
>> linkend='var-STAMP'><filename>STAMP</filename></link> variable.
>> + <link
>> linkend='var-bb-STAMP'><filename>STAMP</filename></link> variable. On
>> subsequent runs, BitBake looks in the build directory within
>> <filename>tmp/stamps</filename> and does not rerun tasks that are
>> already completed unless a timestamp is found to be invalid. @@
>> -618,7 +618,7 @@ <para>
>> Tasks can be either a shell task or a Python task.
>> For shell tasks, BitBake writes a shell script to
>> - <filename>${</filename><link
>> linkend='var-T'><filename>T</filename></link><filename>}/run.do_taskname.pid</filename>
>> + <filename>${</filename><link
>> linkend='var-bb-T'><filename>T</filename></link><filename>}/run.do_taskname.pid</filename>
>> and then executes the script. The generated shell script contains all
>> the exported variables, and the shell functions with all variables
>> expanded. @@ -645,10 +645,10 @@
>> behavior:
>> <itemizedlist>
>> <listitem><para>
>> - <link
>> linkend='var-BB_SCHEDULER'><filename>BB_SCHEDULER</filename></link>
>> + <link
>> linkend='var-bb-BB_SCHEDULER'><filename>BB_SCHEDULER</filename></link>
>> </para></listitem> <listitem><para>
>> - <link
>> linkend='var-BB_SCHEDULERS'><filename>BB_SCHEDULERS</filename></link>
>> + <link
>> linkend='var-bb-BB_SCHEDULERS'><filename>BB_SCHEDULERS</filename></link>
>> </para></listitem> </itemizedlist>
>> It is possible to have functions run before and after a
>> task's main @@ -684,7 +684,7 @@
>> The simplistic approach for excluding the working
>> directory is to set it to some fixed value and create the checksum
>> for the "run" script. BitBake goes one step better and uses the
>> - <link
>> linkend='var-BB_HASHBASE_WHITELIST'><filename>BB_HASHBASE_WHITELIST</filename></link>
>> + <link
>> linkend='var-bb-BB_HASHBASE_WHITELIST'><filename>BB_HASHBASE_WHITELIST</filename></link>
>> variable to define a list of variables that should never be included
>> when generating the signatures. </para>
>> @@ -795,7 +795,7 @@
>> This results in any metadata change that changes the
>> task hash, automatically causing the task to be run again.
>> This removes the need to bump
>> - <link linkend='var-PR'><filename>PR</filename></link>
>> + <link linkend='var-bb-PR'><filename>PR</filename></link>
>> values, and changes to metadata automatically ripple
>> across the build. </para>
>>
>> @@ -884,7 +884,7 @@
>>
>> <para>
>> BitBake first calls the function defined by the
>> - <link
>> linkend='var-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></link>
>> + <link
>> linkend='var-bb-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></link>
>> variable with a list of tasks and corresponding hashes it wants to
>> build. This function is designed to be fast and returns a list
>> @@ -908,7 +908,7 @@
>> For example, it is pointless to obtain a compiler if you
>> already have the compiled binary.
>> To handle this, BitBake calls the
>> - <link
>> linkend='var-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></link>
>> + <link
>> linkend='var-bb-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></link>
>> function for each successful setscene task to know whether or not it
>> needs to obtain the dependencies of that task. </para>
>> @@ -916,7 +916,7 @@
>> <para>
>> Finally, after all the setscene tasks have executed,
>> BitBake calls the function listed in
>> - <link
>> linkend='var-BB_SETSCENE_VERIFY_FUNCTION2'><filename>BB_SETSCENE_VERIFY_FUNCTION2</filename></link>
>> + <link
>> linkend='var-bb-BB_SETSCENE_VERIFY_FUNCTION2'><filename>BB_SETSCENE_VERIFY_FUNCTION2</filename></link>
>> with the list of tasks BitBake thinks has been "covered". The
>> metadata can then ensure that this list is correct and can inform
>> BitBake that it wants specific tasks to be run regardless diff --git
>> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml
>> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml
>> index 29ae486..6840408 100644 ---
>> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml
>> +++
>> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml @@
>> -44,7 +44,7 @@ </literallayout> This code sets up an instance of the
>> fetch class. The instance uses a space-separated list of URLs from the
>> - <link
>> linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
>> + <link
>> linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link> variable
>> and then calls the <filename>download</filename> method to download
>> the files. </para>
>> @@ -78,7 +78,7 @@
>> <listitem><para><emphasis>Pre-mirror
>> Sites:</emphasis> BitBake first uses pre-mirrors to try and find
>> source files. These locations are defined using the
>> - <link
>> linkend='var-PREMIRRORS'><filename>PREMIRRORS</filename></link>
>> + <link
>> linkend='var-bb-PREMIRRORS'><filename>PREMIRRORS</filename></link>
>> variable. </para></listitem>
>> <listitem><para><emphasis>Source URI:</emphasis>
>> @@ -88,7 +88,7 @@
>> <listitem><para><emphasis>Mirror Sites:</emphasis>
>> If fetch failures occur, BitBake next uses
>> mirror locations as defined by the
>> - <link
>> linkend='var-MIRRORS'><filename>MIRRORS</filename></link>
>> + <link
>> linkend='var-bb-MIRRORS'><filename>MIRRORS</filename></link> variable.
>> </para></listitem>
>> </itemizedlist>
>> @@ -144,7 +144,7 @@
>> Any source files that are not local (i.e.
>> downloaded from the Internet) are placed into the
>> download directory, which is specified by the
>> - <link
>> linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
>> + <link
>> linkend='var-bb-DL_DIR'><filename>DL_DIR</filename></link> variable.
>> </para>
>>
>> @@ -184,11 +184,11 @@
>>
>> <para>
>> If
>> - <link
>> linkend='var-BB_STRICT_CHECKSUM'><filename>BB_STRICT_CHECKSUM</filename></link>
>> + <link
>> linkend='var-bb-BB_STRICT_CHECKSUM'><filename>BB_STRICT_CHECKSUM</filename></link>
>> is set, any download without a checksum triggers an error message.
>> The
>> - <link
>> linkend='var-BB_NO_NETWORK'><filename>BB_NO_NETWORK</filename></link>
>> + <link
>> linkend='var-bb-BB_NO_NETWORK'><filename>BB_NO_NETWORK</filename></link>
>> variable can be used to make any attempted network access a fatal
>> error, which is useful for checking that mirrors are complete as well
>> as other things. @@ -265,11 +265,11 @@
>> The filename you specify within the URL can be
>> either an absolute or relative path to a file.
>> If the filename is relative, the contents of the
>> - <link
>> linkend='var-FILESPATH'><filename>FILESPATH</filename></link>
>> + <link
>> linkend='var-bb-FILESPATH'><filename>FILESPATH</filename></link>
>> variable is used in the same way <filename>PATH</filename> is used to
>> find executables. If the file cannot be found, it is assumed that it
>> is available in
>> - <link
>> linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
>> + <link
>> linkend='var-bb-DL_DIR'><filename>DL_DIR</filename></link> by the
>> time the <filename>download()</filename> method is called. </para>
>>
>> @@ -304,7 +304,7 @@
>> allows the name of the downloaded file to be
>> specified. Specifying the name of the downloaded file is useful
>> for avoiding collisions in
>> - <link
>> linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
>> + <link
>> linkend='var-bb-DL_DIR'><filename>DL_DIR</filename></link> when
>> dealing with multiple files that have the same name. </para>
>>
>> @@ -355,7 +355,7 @@
>> A special value of "now" causes the checkout
>> to be updated on every build.
>> </para></listitem>
>> - <listitem><para><emphasis><link
>> linkend='var-CVSDIR'><filename>CVSDIR</filename></link>:</emphasis>
>> + <listitem><para><emphasis><link
>> linkend='var-bb-CVSDIR'><filename>CVSDIR</filename></link>:</emphasis>
>> Specifies where a temporary checkout is saved. The location is often
>> <filename>DL_DIR/cvs</filename>. </para></listitem>
>> @@ -395,7 +395,7 @@
>> <listitem><para><emphasis>"date":</emphasis>
>> Specifies a date.
>> If no "date" is specified, the
>> - <link
>> linkend='var-SRCDATE'><filename>SRCDATE</filename></link>
>> + <link
>> linkend='var-bb-SRCDATE'><filename>SRCDATE</filename></link> of the
>> configuration is used to checkout a specific date. The special value
>> of "now" causes the checkout to be updated on every build.
>> @@ -406,7 +406,7 @@
>> to which the module is unpacked.
>> You are forcing the module into a special
>> directory relative to
>> - <link
>> linkend='var-CVSDIR'><filename>CVSDIR</filename></link>.
>> + <link
>> linkend='var-bb-CVSDIR'><filename>CVSDIR</filename></link>.
>> </para></listitem> <listitem><para><emphasis>"rsh"</emphasis>
>> Used in conjunction with the "method"
>> parameter. @@ -448,7 +448,7 @@
>> <filename>FETCHCMD_svn</filename>, which defaults
>> to "svn".
>> The fetcher's temporary working directory is set by
>> - <link
>> linkend='var-SVNDIR'><filename>SVNDIR</filename></link>,
>> + <link
>> linkend='var-bb-SVNDIR'><filename>SVNDIR</filename></link>, which is
>> usually <filename>DL_DIR/svn</filename>. </para>
>>
>> @@ -509,7 +509,7 @@
>> source control system.
>> The fetcher works by creating a bare clone of the
>> remote into
>> - <link
>> linkend='var-GITDIR'><filename>GITDIR</filename></link>,
>> + <link
>> linkend='var-bb-GITDIR'><filename>GITDIR</filename></link>, which is
>> usually <filename>DL_DIR/git2</filename>. This bare clone is then
>> cloned into the work directory during the unpack stage when a
>> specific tree is checked out. @@ -588,6 +588,14 @@
>> The name of the path in which to place the
>> checkout. By default, the path is <filename>git/</filename>.
>> </para></listitem>
>> + <listitem><para><emphasis>"usehead":</emphasis>
>> + Enables local <filename>git://</filename>
>> URLs to use the
>> + current branch HEAD as the revision for use
>> with
>> + <filename>AUTOREV</filename>.
>> + The "usehead" parameter implies no branch
>> and only works
>> + when the transfer protocol is
>> + <filename>file://</filename>.
>> + </para></listitem>
>> </itemizedlist>
>> Here are some example URLs:
>> <literallayout class='monospaced'>
>> @@ -604,7 +612,7 @@
>> This fetcher submodule inherits from the
>> <link linkend='git-fetcher'>Git fetcher</link> and
>> extends that fetcher's behavior by fetching a repository's submodules.
>> - <link
>> linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
>> + <link
>> linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link> is
>> passed to the Git fetcher as described in the "<link
>> linkend='git-fetcher'>Git Fetcher
>> (<filename>git://</filename>)</link>" section. @@ -639,9 +647,9 @@
>>
>> <para>
>> To use this fetcher, make sure your recipe has proper
>> - <link
>> linkend='var-SRC_URI'><filename>SRC_URI</filename></link>,
>> - <link
>> linkend='var-SRCREV'><filename>SRCREV</filename></link>, and
>> - <link
>> linkend='var-PV'><filename>PV</filename></link> settings.
>> + <link
>> linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link>,
>> + <link
>> linkend='var-bb-SRCREV'><filename>SRCREV</filename></link>, and
>> + <link
>> linkend='var-bb-PV'><filename>PV</filename></link> settings. Here is
>> an example: <literallayout class='monospaced'>
>> SRC_URI =
>> "ccrc://cc.example.org/ccrc;vob=/example_vob;module=/example_module"
>> @@ -726,15 +734,15 @@ <filename>FETCHCMD_p4</filename>, which defaults
>> to "p4".
>> The fetcher's temporary working directory is set by
>> - <link
>> linkend='var-P4DIR'><filename>P4DIR</filename></link>,
>> + <link
>> linkend='var-bb-P4DIR'><filename>P4DIR</filename></link>, which
>> defaults to "DL_DIR/p4". </para>
>>
>> <para>
>> To use this fetcher, make sure your recipe has proper
>> - <link
>> linkend='var-SRC_URI'><filename>SRC_URI</filename></link>,
>> - <link
>> linkend='var-SRCREV'><filename>SRCREV</filename></link>, and
>> - <link
>> linkend='var-PV'><filename>PV</filename></link> values.
>> + <link
>> linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link>,
>> + <link
>> linkend='var-bb-SRCREV'><filename>SRCREV</filename></link>, and
>> + <link
>> linkend='var-bb-PV'><filename>PV</filename></link> values. The p4
>> executable is able to use the config file defined by your system's
>> <filename>P4CONFIG</filename> environment variable in order to define
>> the Perforce server URL and port, username, and @@ -785,9 +793,9 @@
>> <filename>google-repo</filename> source control
>> system. The fetcher works by initiating and syncing sources of the
>> repository into
>> - <link
>> linkend='var-REPODIR'><filename>REPODIR</filename></link>,
>> + <link
>> linkend='var-bb-REPODIR'><filename>REPODIR</filename></link>, which
>> is usually
>> - <link
>> linkend='var-DL_DIR'><filename>DL_DIR</filename></link><filename>/repo</filename>.
>> + <link
>> linkend='var-bb-DL_DIR'><filename>DL_DIR</filename></link><filename>/repo</filename>.
>> </para>
>> <para>
>> @@ -824,19 +832,22 @@
>> Bazaar (<filename>bzr://</filename>)
>> </para></listitem>
>> <listitem><para>
>> - Trees using Git Annex
>> (<filename>gitannex://</filename>)
>> + Mercurial (<filename>hg://</filename>)
>> </para></listitem>
>> <listitem><para>
>> - Secure FTP (<filename>sftp://</filename>)
>> + npm (<filename>npm://</filename>)
>> </para></listitem>
>> <listitem><para>
>> - Secure Shell (<filename>ssh://</filename>)
>> + OSC (<filename>osc://</filename>)
>> </para></listitem>
>> <listitem><para>
>> - OSC (<filename>osc://</filename>)
>> + Secure FTP (<filename>sftp://</filename>)
>> </para></listitem>
>> <listitem><para>
>> - Mercurial (<filename>hg://</filename>)
>> + Secure Shell (<filename>ssh://</filename>)
>> + </para></listitem>
>> + <listitem><para>
>> + Trees using Git Annex
>> (<filename>gitannex://</filename>) </para></listitem>
>> </itemizedlist>
>> No documentation currently exists for these lesser
>> used diff --git
>> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.xml
>> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.xml index
>> 9076f0f..39066e4 100644 ---
>> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.xml +++
>> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.xml @@
>> -194,7 +194,7 @@ <para> When you run BitBake, it begins looking for
>> metadata files. The
>> - <link
>> linkend='var-BBPATH'><filename>BBPATH</filename></link>
>> + <link
>> linkend='var-bb-BBPATH'><filename>BBPATH</filename></link> variable
>> is what tells BitBake where to look for those files.
>> <filename>BBPATH</filename> is not set and you need to set it.
>> Without <filename>BBPATH</filename>, Bitbake cannot @@ -273,14
>> +273,14 @@ some editor to create the <filename>bitbake.conf</filename>
>> so that it contains the following:
>> <literallayout class='monospaced'>
>> - <link linkend='var-PN'>PN</link> =
>> "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0]
>> or 'defaultpkgname'}"
>> + <link linkend='var-bb-PN'>PN</link> =
>> "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0]
>> or 'defaultpkgname'}" </literallayout> <literallayout
>> class='monospaced'>
>> - TMPDIR = "${<link linkend='var-TOPDIR'>TOPDIR</link>}/tmp"
>> - <link linkend='var-CACHE'>CACHE</link> = "${TMPDIR}/cache"
>> - <link linkend='var-STAMP'>STAMP</link> =
>> "${TMPDIR}/${PN}/stamps"
>> - <link linkend='var-T'>T</link> = "${TMPDIR}/${PN}/work"
>> - <link linkend='var-B'>B</link> = "${TMPDIR}/${PN}"
>> + TMPDIR = "${<link linkend='var-bb-TOPDIR'>TOPDIR</link>}/tmp"
>> + <link linkend='var-bb-CACHE'>CACHE</link> = "${TMPDIR}/cache"
>> + <link linkend='var-bb-STAMP'>STAMP</link> =
>> "${TMPDIR}/${PN}/stamps"
>> + <link linkend='var-bb-T'>T</link> = "${TMPDIR}/${PN}/work"
>> + <link linkend='var-bb-B'>B</link> = "${TMPDIR}/${PN}"
>> </literallayout>
>> <note>
>> Without a value for <filename>PN</filename>, the
>> @@ -402,12 +402,12 @@
>> Move to the <filename>conf</filename> directory and
>> create a <filename>layer.conf</filename> file that has the following:
>> <literallayout class='monospaced'>
>> - BBPATH .= ":${<link linkend='var-LAYERDIR'>LAYERDIR</link>}"
>> + BBPATH .= ":${<link linkend='var-bb-LAYERDIR'>LAYERDIR</link>}"
>>
>> - <link linkend='var-BBFILES'>BBFILES</link> += "${LAYERDIR}/*.bb"
>> + <link linkend='var-bb-BBFILES'>BBFILES</link> +=
>> "${LAYERDIR}/*.bb"
>> - <link
>> linkend='var-BBFILE_COLLECTIONS'>BBFILE_COLLECTIONS</link> +=
>> "mylayer"
>> - <link
>> linkend='var-BBFILE_PATTERN'>BBFILE_PATTERN_mylayer</link> :=
>> "^${LAYERDIR_RE}/"
>> + <link
>> linkend='var-bb-BBFILE_COLLECTIONS'>BBFILE_COLLECTIONS</link> +=
>> "mylayer"
>> + <link
>> linkend='var-bb-BBFILE_PATTERN'>BBFILE_PATTERN_mylayer</link> :=
>> "^${LAYERDIR_RE}/" </literallayout> For information on these
>> variables, click the links to go to the definitions in the
>> glossary.</para> @@ -416,9 +416,9 @@
>> a recipe file named
>> <filename>printhello.bb</filename> that has the following:
>> <literallayout class='monospaced'>
>> - <link linkend='var-DESCRIPTION'>DESCRIPTION</link> = "Prints
>> Hello World"
>> - <link linkend='var-PN'>PN</link> = 'printhello'
>> - <link linkend='var-PV'>PV</link> = '1'
>> + <link linkend='var-bb-DESCRIPTION'>DESCRIPTION</link> = "Prints
>> Hello World"
>> + <link linkend='var-bb-PN'>PN</link> = 'printhello'
>> + <link linkend='var-bb-PV'>PV</link> = '1'
>>
>> python do_build() {
>> bb.plain("********************");
>> diff --git
>> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml
>> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml index
>> f7d312a..8f2a960 100644 ---
>> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml +++
>> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml @@
>> -781,7 +781,7 @@ target, you must also enable BitBake to perform
>> multiple configuration builds. Enabling is accomplished by setting the
>> - <link
>> linkend='var-BBMULTICONFIG'><filename>BBMULTICONFIG</filename></link>
>> + <link
>> linkend='var-bb-BBMULTICONFIG'><filename>BBMULTICONFIG</filename></link>
>> variable in the <filename>local.conf</filename> configuration file.
>> As an example, suppose you had configuration
>> files @@ -791,7 +791,7 @@
>> The following statement in the
>> <filename>local.conf</filename> file both enables
>> BitBake to perform multiple configuration builds
>> and
>> - specifies the two multiconfigs:
>> + specifies the two extra multiconfigs:
>> <literallayout class='monospaced'>
>> BBMULTICONFIG = "target1 target2"
>> </literallayout>
>> @@ -803,13 +803,13 @@
>> builds, use the following command form to start
>> the builds:
>> <literallayout class='monospaced'>
>> - $ bitbake
>> [multiconfig:<replaceable>multiconfigname</replaceable>:]<replaceable>target</replaceable>
>> [[[multiconfig:<replaceable>multiconfigname</replaceable>:]<replaceable>target</replaceable>] ... ]
>> + $ bitbake
>> [mc:<replaceable>multiconfigname</replaceable>:]<replaceable>target</replaceable>
>> [[[mc:<replaceable>multiconfigname</replaceable>:]<replaceable>target</replaceable>] ... ]
>> </literallayout>
>> - Here is an example for two multiconfigs:
>> + Here is an example for two extra multiconfigs:
>> <filename>target1</filename> and
>> <filename>target2</filename>:
>> <literallayout class='monospaced'>
>> - $ bitbake multiconfig:target1:<replaceable>target</replaceable>
>> multiconfig:target2:<replaceable>target</replaceable>
>> + $ bitbake mc::<replaceable>target</replaceable>
>> mc:target1:<replaceable>target</replaceable>
>> mc:target2:<replaceable>target</replaceable> </literallayout> </para>
>> </section>
>> @@ -837,13 +837,13 @@
>> build, you must declare the dependencies in the
>> recipe using the following statement form:
>> <literallayout class='monospaced'>
>> - <replaceable>task_or_package</replaceable>[mcdepends] =
>> "multiconfig:<replaceable>from_multiconfig</replaceable>:<replaceable>to_multiconfig</replaceable>:<replaceable>recipe_name</replaceable>:<replaceable>task_on_which_to_depend</replaceable>"
>> + <replaceable>task_or_package</replaceable>[mcdepends] =
>> "mc:<replaceable>from_multiconfig</replaceable>:<replaceable>to_multiconfig</replaceable>:<replaceable>recipe_name</replaceable>:<replaceable>task_on_which_to_depend</replaceable>"
>> </literallayout> To better show how to use this statement, consider an
>> example with two multiconfigs:
>> <filename>target1</filename> and <filename>target2</filename>:
>> <literallayout class='monospaced'>
>> - <replaceable>image_task</replaceable>[mcdepends] =
>> "multiconfig:target1:target2:<replaceable>image2</replaceable>:<replaceable>rootfs_task</replaceable>"
>> + <replaceable>image_task</replaceable>[mcdepends] =
>> "mc:target1:target2:<replaceable>image2</replaceable>:<replaceable>rootfs_task</replaceable>"
>> </literallayout> In this example, the
>> <replaceable>from_multiconfig</replaceable> is
>> "target1" and @@ -859,7 +859,7 @@
>> Once you set up this dependency, you can build the
>> "target1" multiconfig using a BitBake command as
>> follows: <literallayout class='monospaced'>
>> - $ bitbake multiconfig:target1:<replaceable>image1</replaceable>
>> + $ bitbake mc:target1:<replaceable>image1</replaceable>
>> </literallayout>
>> This command executes all the tasks needed to
>> create <replaceable>image1</replaceable> for the "target1"
>> @@ -875,7 +875,7 @@
>> Consider this change to the statement in the
>> <replaceable>image1</replaceable> recipe:
>> <literallayout class='monospaced'>
>> - <replaceable>image_task</replaceable>[mcdepends] =
>> "multiconfig:target1:target2:<replaceable>image2</replaceable>:<replaceable>image_task</replaceable>"
>> + <replaceable>image_task</replaceable>[mcdepends] =
>> "mc:target1:target2:<replaceable>image2</replaceable>:<replaceable>image_task</replaceable>"
>> </literallayout> In this case, BitBake must create
>> <replaceable>image2</replaceable> for the
>> "target2" diff --git
>> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml
>> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml
>> index 2490f6e..421364c 100644 ---
>> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml
>> +++
>> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml @@
>> -61,6 +61,78 @@ </para> </section>
>> + <section id='modifying-existing-variables'>
>> + <title>Modifying Existing Variables</title>
>> +
>> + <para>
>> + Sometimes you need to modify existing variables.
>> + Following are some cases where you might find you
>> want to
>> + modify an existing variable:
>> + <itemizedlist>
>> + <listitem><para>
>> + Customize a recipe that uses the variable.
>> + </para></listitem>
>> + <listitem><para>
>> + Change a variable's default value used in a
>> + <filename>*.bbclass</filename> file.
>> + </para></listitem>
>> + <listitem><para>
>> + Change the variable in a
>> <filename>*.bbappend</filename>
>> + file to override the variable in the
>> original recipe.
>> + </para></listitem>
>> + <listitem><para>
>> + Change the variable in a configuration file
>> so that the
>> + value overrides an existing configuration.
>> + </para></listitem>
>> + </itemizedlist>
>> + </para>
>> +
>> + <para>
>> + Changing a variable value can sometimes depend on
>> how the
>> + value was originally assigned and also on the desired
>> + intent of the change.
>> + In particular, when you append a value to a variable
>> that
>> + has a default value, the resulting value might not
>> be what
>> + you expect.
>> + In this case, the value you provide might replace
>> the value
>> + rather than append to the default value.
>> + </para>
>> +
>> + <para>
>> + If after you have changed a variable's value and
>> something
>> + unexplained occurs, you can use BitBake to check the
>> actual
>> + value of the suspect variable.
>> + You can make these checks for both configuration and
>> recipe
>> + level changes:
>> + <itemizedlist>
>> + <listitem><para>
>> + For configuration changes, use the following:
>> + <literallayout class='monospaced'>
>> + $ bitbake -e
>> + </literallayout>
>> + This command displays variable values after
>> the
>> + configuration files (i.e.
>> <filename>local.conf</filename>,
>> + <filename>bblayers.conf</filename>,
>> + <filename>bitbake.conf</filename> and so
>> forth) have
>> + been parsed.
>> + <note>
>> + Variables that are exported to the
>> environment are
>> + preceded by the string "export" in the
>> command's
>> + output.
>> + </note>
>> + </para></listitem>
>> + <listitem><para>
>> + For recipe changes, use the following:
>> + <literallayout class='monospaced'>
>> + $ bitbake <replaceable>recipe</replaceable> -e | grep VARIABLE="
>> + </literallayout>
>> + This command checks to see if the variable
>> actually
>> + makes it into a specific recipe.
>> + </para></listitem>
>> + </itemizedlist>
>> + </para>
>> + </section>
>> +
>> <section id='line-joining'>
>> <title>Line Joining</title>
>>
>> @@ -297,9 +369,8 @@
>>
>> <para>
>> These operators differ from the ":=", ".=", "=.",
>> "+=", and "=+"
>> - operators in that their effects are deferred
>> - until after parsing completes rather than being
>> immediately
>> - applied.
>> + operators in that their effects are applied at
>> variable
>> + expansion time rather than being immediately applied.
>> Here are some examples:
>> <literallayout class='monospaced'>
>> B = "bval"
>> @@ -348,18 +419,22 @@
>> FOO = "123 456 789 123456 123 456 123 456"
>> FOO_remove = "123"
>> FOO_remove = "456"
>> - FOO2 = "abc def ghi abcdef abc def abc def"
>> - FOO2_remove = "abc def"
>> + FOO2 = " abc def ghi abcdef abc def abc def def"
>> + FOO2_remove = " \
>> + def \
>> + abc \
>> + ghi \
>> + "
>> </literallayout>
>> The variable <filename>FOO</filename> becomes
>> - " 789 123456 "
>> + " 789 123456 "
>> and <filename>FOO2</filename> becomes
>> - " ghi abcdef ".
>> +
>> " jkl abcdef ".
>> </para>
>> <para>
>> Like "_append" and "_prepend", "_remove"
>> - is deferred until after parsing completes.
>> + is applied at variable expansion time.
>> </para>
>> </section>
>>
>> @@ -503,7 +578,7 @@
>> </section>
>>
>> <section id='unsetting-variables'>
>> - <title>Unseting variables</title>
>> + <title>Unsetting variables</title>
>>
>> <para>
>> It is possible to completely remove a variable or a
>> variable flag @@ -595,7 +670,7 @@
>>
>> <para>
>> BitBake uses
>> - <link
>> linkend='var-OVERRIDES'><filename>OVERRIDES</filename></link>
>> + <link
>> linkend='var-bb-OVERRIDES'><filename>OVERRIDES</filename></link> to
>> control what variables are overridden after BitBake parses recipes
>> and configuration files. This section describes how you can use
>> @@ -705,7 +780,7 @@
>>
>> <para>Internally, this is implemented by
>> prepending the task (e.g. "task-compile:") to the value of
>> - <link
>> linkend='var-OVERRIDES'><filename>OVERRIDES</filename></link>
>> + <link
>> linkend='var-bb-OVERRIDES'><filename>OVERRIDES</filename></link> for
>> the local datastore of the <filename>do_compile</filename>
>> task.</para>
>> @@ -724,17 +799,15 @@
>> <title>Key Expansion</title>
>>
>> <para>
>> - Key expansion happens when the BitBake datastore is
>> finalized
>> - just before BitBake expands overrides.
>> + Key expansion happens when the BitBake datastore is
>> finalized. To better understand this, consider the following example:
>> <literallayout class='monospaced'>
>> A${B} = "X"
>> B = "2"
>> A2 = "Y"
>> </literallayout>
>> - In this case, after all the parsing is complete, and
>> - before any overrides are handled, BitBake expands
>> - <filename>${B}</filename> into "2".
>> + In this case, after all the parsing is complete,
>> + BitBake expands <filename>${B}</filename> into "2".
>> This expansion causes <filename>A2</filename>, which
>> was set to "Y" before the expansion, to become "X".
>> </para>
>> @@ -868,7 +941,7 @@
>>
>> <para>
>> BitBake uses the
>> - <link
>> linkend='var-BBPATH'><filename>BBPATH</filename></link>
>> + <link
>> linkend='var-bb-BBPATH'><filename>BBPATH</filename></link> variable
>> to locate needed include and class files. Additionally, BitBake
>> searches the current directory for <filename>include</filename> and
>> <filename>require</filename> @@ -1086,7 +1159,7 @@
>> <para>
>> When creating a configuration file
>> (<filename>.conf</filename>), you can use the
>> - <link
>> linkend='var-INHERIT'><filename>INHERIT</filename></link>
>> + <link
>> linkend='var-bb-INHERIT'><filename>INHERIT</filename></link>
>> configuration directive to inherit a class. BitBake only supports
>> this directive when used within a configuration file.
>> @@ -1370,7 +1443,7 @@
>> </para></listitem>
>> <listitem><para>
>> BitBake-style Python functions generate a
>> separate
>> - <filename>${</filename><link
>> linkend='var-T'><filename>T</filename></link><filename>}/run.</filename><replaceable>function-name</replaceable><filename>.</filename><replaceable>pid</replaceable>
>> + <filename>${</filename><link
>> linkend='var-bb-T'><filename>T</filename></link><filename>}/run.</filename><replaceable>function-name</replaceable><filename>.</filename><replaceable>pid</replaceable>
>> script that is executed to run the function, and also generate a log
>> file in
>> <filename>${T}/log.</filename><replaceable>function-name</replaceable><filename>.</filename><replaceable>pid</replaceable>
>> @@ -1773,7 +1846,7 @@ things exported or listed in its whitelist to
>> ensure that the build environment is reproducible and consistent.
>> You can prevent this "cleaning" by setting the
>> - <link
>> linkend='var-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link>
>> + <link
>> linkend='var-bb-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link>
>> variable. </note>
>> Consequently, if you do want something to get passed
>> into the @@ -1783,9 +1856,9 @@
>> Tell BitBake to load what you want from the
>> environment into the datastore.
>> You can do so through the
>> - <link
>> linkend='var-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>
>> + <link
>> linkend='var-bb-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>
>> and
>> - <link
>> linkend='var-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link>
>> + <link
>> linkend='var-bb-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link>
>> variables. For example, assume you want to prevent the build system
>> from accessing your <filename>$HOME/.ccache</filename>
>> @@ -1824,7 +1897,7 @@
>> from the original execution environment.
>> Bitbake saves a copy of the original environment into
>> a special variable named
>> - <link
>> linkend='var-BB_ORIGENV'><filename>BB_ORIGENV</filename></link>.
>> + <link
>> linkend='var-bb-BB_ORIGENV'><filename>BB_ORIGENV</filename></link>.
>> </para>
>> <para>
>> @@ -1883,7 +1956,7 @@
>> <listitem><para><emphasis><filename>[depends]</filename>:</emphasis>
>> Controls inter-task dependencies.
>> See the
>> - <link
>> linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
>> + <link
>> linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link> variable
>> and the "<link linkend='inter-task-dependencies'>Inter-Task
>> Dependencies</link>" section for more information.
>> @@ -1891,7 +1964,7 @@
>> <listitem><para><emphasis><filename>[deptask]</filename>:</emphasis>
>> Controls task build-time dependencies.
>> See the
>> - <link
>> linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
>> + <link
>> linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link> variable
>> and the "<link linkend='build-dependencies'>Build Dependencies</link>"
>> section for more information.
>> @@ -1937,7 +2010,7 @@
>> of cores but certain tasks need to be
>> rate-limited due to various kinds of resource constraints (e.g. to
>> avoid network throttling). <filename>number_threads</filename> works
>> similarly to the
>> - <link
>> linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
>> + <link
>> linkend='var-bb-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
>> variable but is task-specific.</para>
>> <para>Set the value globally.
>> @@ -1971,9 +2044,9 @@
>> <listitem><para><emphasis><filename>[rdepends]</filename>:</emphasis>
>> Controls inter-task runtime dependencies.
>> See the
>> - <link
>> linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link>
>> + <link
>> linkend='var-bb-RDEPENDS'><filename>RDEPENDS</filename></link>
>> variable, the
>> - <link
>> linkend='var-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
>> + <link
>> linkend='var-bb-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
>> variable, and the "<link linkend='inter-task-dependencies'>Inter-Task
>> Dependencies</link>" section for more information.
>> @@ -1981,9 +2054,9 @@
>> <listitem><para><emphasis><filename>[rdeptask]</filename>:</emphasis>
>> Controls task runtime dependencies.
>> See the
>> - <link
>> linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link>
>> + <link
>> linkend='var-bb-RDEPENDS'><filename>RDEPENDS</filename></link>
>> variable, the
>> - <link
>> linkend='var-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
>> + <link
>> linkend='var-bb-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
>> variable, and the "<link linkend='runtime-dependencies'>Runtime
>> Dependencies</link>" section for more information.
>> @@ -1996,9 +2069,9 @@
>> <listitem><para><emphasis><filename>[recrdeptask]</filename>:</emphasis>
>> Controls task recursive runtime dependencies.
>> See the
>> - <link
>> linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link>
>> + <link
>> linkend='var-bb-RDEPENDS'><filename>RDEPENDS</filename></link>
>> variable, the
>> - <link
>> linkend='var-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
>> + <link
>> linkend='var-bb-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
>> variable, and the "<link linkend='recursive-dependencies'>Recursive
>> Dependencies</link>" section for more information.
>> @@ -2127,7 +2200,7 @@
>> Any given datastore only has one such event
>> executed against it, however.
>> If
>> - <link
>> linkende='var-BB_INVALIDCONF'><filename>BB_INVALIDCONF</filename></link>
>> + <link
>> linkende='var-bb-BB_INVALIDCONF'><filename>BB_INVALIDCONF</filename></link>
>> is set in the datastore by the event handler, the configuration is
>> reparsed and a new event triggered, allowing the metadata to update
>> configuration. @@ -2256,17 +2329,17 @@
>> from a single recipe file multiple incarnations of that
>> recipe file where all incarnations are buildable.
>> These features are enabled through the
>> - <link
>> linkend='var-BBCLASSEXTEND'><filename>BBCLASSEXTEND</filename></link>
>> + <link
>> linkend='var-bb-BBCLASSEXTEND'><filename>BBCLASSEXTEND</filename></link>
>> and
>> - <link
>> linkend='var-BBVERSIONS'><filename>BBVERSIONS</filename></link>
>> + <link
>> linkend='var-bb-BBVERSIONS'><filename>BBVERSIONS</filename></link>
>> variables. <note>
>> The mechanism for this class extension is extremely
>> specific to the implementation.
>> Usually, the recipe's
>> - <link
>> linkend='var-PROVIDES'><filename>PROVIDES</filename></link>,
>> - <link
>> linkend='var-PN'><filename>PN</filename></link>, and
>> - <link
>> linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
>> + <link
>> linkend='var-bb-PROVIDES'><filename>PROVIDES</filename></link>,
>> + <link
>> linkend='var-bb-PN'><filename>PN</filename></link>, and
>> + <link
>> linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link>
>> variables would need to be modified by the extension class. For
>> specific examples, see the OE-Core <filename>native</filename>,
>> <filename>nativesdk</filename>, @@ -2287,7 +2360,7 @@
>> project from a single recipe file.
>> You can also specify conditional metadata
>> (using the
>> - <link
>> linkend='var-OVERRIDES'><filename>OVERRIDES</filename></link>
>> + <link
>> linkend='var-bb-OVERRIDES'><filename>OVERRIDES</filename></link>
>> mechanism) for a single version, or an optionally named range of
>> versions. Here is an example: <literallayout class='monospaced'>
>> @@ -2306,7 +2379,7 @@
>> into overrides, but it is also made available
>> for the metadata to use in the variable that defines the base recipe
>> versions for use in <filename>file://</filename> search paths
>> - (<link
>> linkend='var-FILESPATH'><filename>FILESPATH</filename></link>).
>> + (<link
>> linkend='var-bb-FILESPATH'><filename>FILESPATH</filename></link>).
>> </para></listitem> </itemizedlist>
>> </para>
>> @@ -2408,7 +2481,7 @@
>>
>> <para>
>> BitBake uses the
>> - <link
>> linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
>> + <link
>> linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link> variable
>> to manage build time dependencies. The <filename>[deptask]</filename>
>> varflag for tasks signifies the task of each
>> @@ -2429,9 +2502,9 @@
>>
>> <para>
>> BitBake uses the
>> - <link
>> linkend='var-PACKAGES'><filename>PACKAGES</filename></link>,
>> - <link
>> linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link>, and
>> - <link
>> linkend='var-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
>> + <link
>> linkend='var-bb-PACKAGES'><filename>PACKAGES</filename></link>,
>> + <link
>> linkend='var-bb-RDEPENDS'><filename>RDEPENDS</filename></link>, and
>> + <link
>> linkend='var-bb-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
>> variables to manage runtime dependencies. </para>
>>
>> @@ -2686,7 +2759,7 @@
>>
>> <para>
>> These checksums are stored in
>> - <link
>> linkend='var-STAMP'><filename>STAMP</filename></link>.
>> + <link
>> linkend='var-bb-STAMP'><filename>STAMP</filename></link>. You can
>> examine the checksums using the following BitBake command:
>> <literallayout class='monospaced'> $ bitbake-dumpsigs
>> @@ -2708,44 +2781,44 @@
>> The following list describes related variables:
>> <itemizedlist>
>> <listitem><para>
>> - <link
>> linkend='var-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></link>:
>> + <link
>> linkend='var-bb-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></link>:
>> Specifies the name of the function to call during the "setscene" part
>> of the task's execution in order to validate the list of task hashes.
>> </para></listitem>
>> <listitem><para>
>> - <link
>> linkend='var-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></link>:
>> + <link
>> linkend='var-bb-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></link>:
>> Specifies a function BitBake calls that determines whether BitBake
>> requires a setscene dependency to be met.
>> </para></listitem>
>> <listitem><para>
>> - <link
>> linkend='var-BB_SETSCENE_VERIFY_FUNCTION2'><filename>BB_SETSCENE_VERIFY_FUNCTION2</filename></link>:
>> + <link
>> linkend='var-bb-BB_SETSCENE_VERIFY_FUNCTION2'><filename>BB_SETSCENE_VERIFY_FUNCTION2</filename></link>:
>> Specifies a function to call that verifies the list of planned task
>> execution before the main task execution happens.
>> </para></listitem>
>> <listitem><para>
>> - <link
>> linkend='var-BB_STAMP_POLICY'><filename>BB_STAMP_POLICY</filename></link>:
>> + <link
>> linkend='var-bb-BB_STAMP_POLICY'><filename>BB_STAMP_POLICY</filename></link>:
>> Defines the mode for comparing timestamps of stamp files.
>> </para></listitem> <listitem><para>
>> - <link
>> linkend='var-BB_STAMP_WHITELIST'><filename>BB_STAMP_WHITELIST</filename></link>:
>> + <link
>> linkend='var-bb-BB_STAMP_WHITELIST'><filename>BB_STAMP_WHITELIST</filename></link>:
>> Lists stamp files that are looked at when the stamp policy is
>> "whitelist". </para></listitem>
>> <listitem><para>
>> - <link
>> linkend='var-BB_TASKHASH'><filename>BB_TASKHASH</filename></link>:
>> + <link
>> linkend='var-bb-BB_TASKHASH'><filename>BB_TASKHASH</filename></link>:
>> Within an executing task, this variable holds the hash of the task as
>> returned by the currently enabled signature generator.
>> </para></listitem>
>> <listitem><para>
>> - <link
>> linkend='var-STAMP'><filename>STAMP</filename></link>:
>> + <link
>> linkend='var-bb-STAMP'><filename>STAMP</filename></link>: The base
>> path to create stamp files. </para></listitem>
>> <listitem><para>
>> - <link
>> linkend='var-STAMPCLEAN'><filename>STAMPCLEAN</filename></link>:
>> + <link
>> linkend='var-bb-STAMPCLEAN'><filename>STAMPCLEAN</filename></link>:
>> Again, the base path to create stamp files but can use wildcards for
>> matching a range of files for clean operations. </para></listitem>
>> diff --git
>> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml
>> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml
>> index a84b2bc..aca6741 100644 ---
>> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml
>> +++
>> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml
>> @@ -3,7 +3,7 @@ [<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
>> <!-- Dummy chapter --> -<chapter id='ref-variables-glos'>
>> +<chapter id='ref-bb-variables-glos'>
>>
>> <title>Variables Glossary</title>
>>
>> @@ -34,29 +34,29 @@
>> </itemizedlist>
>> </note>
>>
>> -<glossary id='ref-variables-glossary'>
>> +<glossary id='ref-bb-variables-glossary'>
>>
>> <para>
>> - <link linkend='var-ASSUME_PROVIDED'>A</link>
>> - <link linkend='var-B'>B</link>
>> - <link linkend='var-CACHE'>C</link>
>> - <link linkend='var-DEFAULT_PREFERENCE'>D</link>
>> - <link linkend='var-EXCLUDE_FROM_WORLD'>E</link>
>> - <link linkend='var-FAKEROOT'>F</link>
>> - <link linkend='var-GITDIR'>G</link>
>> - <link linkend='var-HGDIR'>H</link>
>> -<!-- <link linkend='var-ICECC_DISABLED'>I</link> -->
>> + <link linkend='var-bb-ASSUME_PROVIDED'>A</link>
>> + <link linkend='var-bb-B'>B</link>
>> + <link linkend='var-bb-CACHE'>C</link>
>> + <link linkend='var-bb-DEFAULT_PREFERENCE'>D</link>
>> + <link linkend='var-bb-EXCLUDE_FROM_WORLD'>E</link>
>> + <link linkend='var-bb-FAKEROOT'>F</link>
>> + <link linkend='var-bb-GITDIR'>G</link>
>> + <link linkend='var-bb-HGDIR'>H</link>
>> + <link linkend='var-bb-INHERIT'>I</link>
>> <!-- <link linkend='var-glossary-j'>J</link> -->
>> <!-- <link linkend='var-KARCH'>K</link> -->
>> - <link linkend='var-LAYERDEPENDS'>L</link>
>> - <link linkend='var-MIRRORS'>M</link>
>> + <link linkend='var-bb-LAYERDEPENDS'>L</link>
>> + <link linkend='var-bb-MIRRORS'>M</link>
>> <!-- <link linkend='var-glossary-n'>N</link> -->
>> - <link linkend='var-OVERRIDES'>O</link>
>> - <link linkend='var-P4DIR'>P</link>
>> + <link linkend='var-bb-OVERRIDES'>O</link>
>> + <link linkend='var-bb-P4DIR'>P</link>
>> <!-- <link linkend='var-QMAKE_PROFILES'>Q</link> -->
>> - <link linkend='var-RDEPENDS'>R</link>
>> - <link linkend='var-SECTION'>S</link>
>> - <link linkend='var-T'>T</link>
>> + <link linkend='var-bb-RDEPENDS'>R</link>
>> + <link linkend='var-bb-SECTION'>S</link>
>> + <link linkend='var-bb-T'>T</link>
>> <!-- <link linkend='var-UBOOT_CONFIG'>U</link> -->
>> <!-- <link linkend='var-glossary-v'>V</link> -->
>> <!-- <link linkend='var-WARN_QA'>W</link> -->
>> @@ -65,13 +65,13 @@
>> <!-- <link linkend='var-glossary-z'>Z</link>-->
>> </para>
>>
>> - <glossdiv id='var-glossary-a'><title>A</title>
>> + <glossdiv id='var-bb-glossary-a'><title>A</title>
>>
>> - <glossentry
>> id='var-ASSUME_PROVIDED'><glossterm>ASSUME_PROVIDED</glossterm>
>> + <glossentry
>> id='var-bb-ASSUME_PROVIDED'><glossterm>ASSUME_PROVIDED</glossterm>
>> <glossdef> <para>
>> Lists recipe names
>> - (<link
>> linkend='var-PN'><filename>PN</filename></link>
>> + (<link
>> linkend='var-bb-PN'><filename>PN</filename></link> values) BitBake
>> does not attempt to build. Instead, BitBake assumes these recipes
>> have already been built.
>> @@ -91,9 +91,9 @@
>> </glossdiv>
>>
>>
>> - <glossdiv id='var-glossary-b'><title>B</title>
>> + <glossdiv id='var-bb-glossary-b'><title>B</title>
>>
>> - <glossentry id='var-B'><glossterm>B</glossterm>
>> + <glossentry id='var-bb-B'><glossterm>B</glossterm>
>> <glossdef>
>> <para>
>> The directory in which BitBake executes functions
>> @@ -102,7 +102,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_ALLOWED_NETWORKS'><glossterm>BB_ALLOWED_NETWORKS</glossterm>
>> + <glossentry
>> id='var-bb-BB_ALLOWED_NETWORKS'><glossterm>BB_ALLOWED_NETWORKS</glossterm>
>> <glossdef> <para>
>> Specifies a space-delimited list of hosts that
>> the fetcher @@ -111,7 +111,7 @@
>> <itemizedlist>
>> <listitem><para>
>> This host list is only used if
>> - <link
>> linkend='var-BB_NO_NETWORK'><filename>BB_NO_NETWORK</filename></link>
>> + <link
>> linkend='var-bb-BB_NO_NETWORK'><filename>BB_NO_NETWORK</filename></link>
>> is either not set or set to "0". </para></listitem>
>> <listitem><para>
>> @@ -151,13 +151,13 @@
>> </itemizedlist>
>> Using <filename>BB_ALLOWED_NETWORKS</filename> in
>> conjunction with
>> - <link
>> linkend='var-PREMIRRORS'><filename>PREMIRRORS</filename></link>
>> + <link
>> linkend='var-bb-PREMIRRORS'><filename>PREMIRRORS</filename></link> is
>> very useful. Adding the host you want to use to
>> <filename>PREMIRRORS</filename> results in the
>> source code being fetched from an allowed location and avoids raising
>> an error when a host that is not allowed is in a
>> - <link
>> linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
>> + <link
>> linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link>
>> statement. This is because the fetcher does not attempt to use the
>> host listed in <filename>SRC_URI</filename>
>> after a @@ -167,7 +167,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_CONSOLELOG'><glossterm>BB_CONSOLELOG</glossterm>
>> + <glossentry
>> id='var-bb-BB_CONSOLELOG'><glossterm>BB_CONSOLELOG</glossterm>
>> <glossdef> <para>
>> Specifies the path to a log file into which
>> BitBake's user @@ -176,7 +176,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_CURRENTTASK'><glossterm>BB_CURRENTTASK</glossterm>
>> + <glossentry
>> id='var-bb-BB_CURRENTTASK'><glossterm>BB_CURRENTTASK</glossterm>
>> <glossdef> <para>
>> Contains the name of the currently running task.
>> @@ -186,7 +186,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_DANGLINGAPPENDS_WARNONLY'><glossterm>BB_DANGLINGAPPENDS_WARNONLY</glossterm>
>> + <glossentry
>> id='var-bb-BB_DANGLINGAPPENDS_WARNONLY'><glossterm>BB_DANGLINGAPPENDS_WARNONLY</glossterm>
>> <glossdef> <para>
>> Defines how BitBake handles situations where an
>> append @@ -208,7 +208,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_DEFAULT_TASK'><glossterm>BB_DEFAULT_TASK</glossterm>
>> + <glossentry
>> id='var-bb-BB_DEFAULT_TASK'><glossterm>BB_DEFAULT_TASK</glossterm>
>> <glossdef> <para>
>> The default task to use when none is specified
>> (e.g. @@ -219,7 +219,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_DISKMON_DIRS'><glossterm>BB_DISKMON_DIRS</glossterm>
>> + <glossentry
>> id='var-bb-BB_DISKMON_DIRS'><glossterm>BB_DISKMON_DIRS</glossterm>
>> <glossdef> <para>
>> Monitors disk space and available inodes during
>> the build @@ -245,7 +245,7 @@
>> build when a threshold is broken.
>> Subsequent warnings are issued as
>> defined by the
>> - <link
>> linkend='var-BB_DISKMON_WARNINTERVAL'>BB_DISKMON_WARNINTERVAL</link>
>> variable,
>> + <link
>> linkend='var-bb-BB_DISKMON_WARNINTERVAL'>BB_DISKMON_WARNINTERVAL</link>
>> variable, which must be defined.
>> <dir> is:
>> @@ -275,7 +275,7 @@
>> BB_DISKMON_DIRS = "ABORT,${TMPDIR},,100K"
>> </literallayout>
>> The first example works only if you also set
>> - the <link
>> linkend='var-BB_DISKMON_WARNINTERVAL'><filename>BB_DISKMON_WARNINTERVAL</filename></link>
>> variable.
>> + the <link
>> linkend='var-bb-BB_DISKMON_WARNINTERVAL'><filename>BB_DISKMON_WARNINTERVAL</filename></link>
>> variable. This example causes the build system to immediately abort
>> when either the disk space in <filename>${TMPDIR}</filename> drops
>> below 1 Gbyte or the available free inodes drops below @@ -309,7
>> +309,7 @@ </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_DISKMON_WARNINTERVAL'><glossterm>BB_DISKMON_WARNINTERVAL</glossterm>
>> + <glossentry
>> id='var-bb-BB_DISKMON_WARNINTERVAL'><glossterm>BB_DISKMON_WARNINTERVAL</glossterm>
>> <glossdef> <para>
>> Defines the disk space and free inode warning
>> intervals. @@ -319,7 +319,7 @@
>> If you are going to use the
>> <filename>BB_DISKMON_WARNINTERVAL</filename>
>> variable, you must also use the
>> - <link
>> linkend='var-BB_DISKMON_DIRS'><filename>BB_DISKMON_DIRS</filename></link>
>> variable
>> + <link
>> linkend='var-bb-BB_DISKMON_DIRS'><filename>BB_DISKMON_DIRS</filename></link>
>> variable and define its action as "WARN". During the build,
>> subsequent warnings are issued each time disk space or number of free
>> inodes further reduces by @@ -374,7 +374,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_ENV_WHITELIST'><glossterm>BB_ENV_WHITELIST</glossterm>
>> + <glossentry
>> id='var-bb-BB_ENV_WHITELIST'><glossterm>BB_ENV_WHITELIST</glossterm>
>> <glossdef> <para>
>> Specifies the internal whitelist of variables to
>> allow @@ -382,11 +382,11 @@
>> datastore.
>> If the value of this variable is not specified
>> (which is the default), the following list is
>> used:
>> - <link
>> linkend='var-BBPATH'><filename>BBPATH</filename></link>,
>> - <link
>> linkend='var-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link>,
>> - <link
>> linkend='var-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>,
>> + <link
>> linkend='var-bb-BBPATH'><filename>BBPATH</filename></link>,
>> + <link
>> linkend='var-bb-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link>,
>> + <link
>> linkend='var-bb-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>,
>> and
>> - <link
>> linkend='var-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link>.
>> + <link
>> linkend='var-bb-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link>.
>> <note> You must set this variable in the external environment
>> in order for it to work.
>> @@ -395,7 +395,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_ENV_EXTRAWHITE'><glossterm>BB_ENV_EXTRAWHITE</glossterm>
>> + <glossentry
>> id='var-bb-BB_ENV_EXTRAWHITE'><glossterm>BB_ENV_EXTRAWHITE</glossterm>
>> <glossdef> <para>
>> Specifies an additional set of variables to
>> allow through @@ -403,7 +403,7 @@
>> datastore.
>> This list of variables are on top of the
>> internal list set in
>> - <link
>> linkend='var-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>.
>> + <link
>> linkend='var-bb-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>.
>> <note> You must set this variable in the external
>> environment in order for it to work.
>> @@ -412,22 +412,22 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_FETCH_PREMIRRORONLY'><glossterm>BB_FETCH_PREMIRRORONLY</glossterm>
>> + <glossentry
>> id='var-bb-BB_FETCH_PREMIRRORONLY'><glossterm>BB_FETCH_PREMIRRORONLY</glossterm>
>> <glossdef> <para>
>> When set to "1", causes BitBake's fetcher module
>> to only search
>> - <link
>> linkend='var-PREMIRRORS'><filename>PREMIRRORS</filename></link>
>> + <link
>> linkend='var-bb-PREMIRRORS'><filename>PREMIRRORS</filename></link>
>> for files. BitBake will not search the main
>> - <link
>> linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
>> + <link
>> linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link> or
>> - <link
>> linkend='var-MIRRORS'><filename>MIRRORS</filename></link>.
>> + <link
>> linkend='var-bb-MIRRORS'><filename>MIRRORS</filename></link>. </para>
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_FILENAME'><glossterm>BB_FILENAME</glossterm>
>> + <glossentry
>> id='var-bb-BB_FILENAME'><glossterm>BB_FILENAME</glossterm> <glossdef>
>> <para>
>> Contains the filename of the recipe that owns
>> the currently @@ -440,12 +440,12 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_GENERATE_MIRROR_TARBALLS'><glossterm>BB_GENERATE_MIRROR_TARBALLS</glossterm>
>> + <glossentry
>> id='var-bb-BB_GENERATE_MIRROR_TARBALLS'><glossterm>BB_GENERATE_MIRROR_TARBALLS</glossterm>
>> <glossdef> <para>
>> Causes tarballs of the Git repositories,
>> including the Git metadata, to be placed in the
>> - <link
>> linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
>> + <link
>> linkend='var-bb-DL_DIR'><filename>DL_DIR</filename></link> directory.
>> Anyone wishing to create a source mirror would
>> want to enable this variable.
>> @@ -461,7 +461,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_HASHCONFIG_WHITELIST'><glossterm>BB_HASHCONFIG_WHITELIST</glossterm>
>> + <glossentry
>> id='var-bb-BB_HASHCONFIG_WHITELIST'><glossterm>BB_HASHCONFIG_WHITELIST</glossterm>
>> <glossdef> <para>
>> Lists variables that are excluded from base
>> configuration @@ -485,7 +485,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_HASHBASE_WHITELIST'><glossterm>BB_HASHBASE_WHITELIST</glossterm>
>> + <glossentry
>> id='var-bb-BB_HASHBASE_WHITELIST'><glossterm>BB_HASHBASE_WHITELIST</glossterm>
>> <glossdef> <para>
>> Lists variables that are excluded from checksum
>> and @@ -500,7 +500,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_HASHCHECK_FUNCTION'><glossterm>BB_HASHCHECK_FUNCTION</glossterm>
>> + <glossentry
>> id='var-bb-BB_HASHCHECK_FUNCTION'><glossterm>BB_HASHCHECK_FUNCTION</glossterm>
>> <glossdef> <para>
>> Specifies the name of the function to call
>> during the @@ -524,7 +524,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_INVALIDCONF'><glossterm>BB_INVALIDCONF</glossterm>
>> + <glossentry
>> id='var-bb-BB_INVALIDCONF'><glossterm>BB_INVALIDCONF</glossterm>
>> <glossdef> <para>
>> Used in combination with the
>> @@ -539,11 +539,11 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_LOGFMT'><glossterm>BB_LOGFMT</glossterm>
>> + <glossentry
>> id='var-bb-BB_LOGFMT'><glossterm>BB_LOGFMT</glossterm> <glossdef>
>> <para>
>> Specifies the name of the log files saved into
>> - <filename>${</filename><link
>> linkend='var-T'><filename>T</filename></link><filename>}</filename>.
>> + <filename>${</filename><link
>> linkend='var-bb-T'><filename>T</filename></link><filename>}</filename>.
>> By default, the <filename>BB_LOGFMT</filename> variable is undefined
>> and the log file names get created using the following form:
>> @@ -556,7 +556,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_NICE_LEVEL'><glossterm>BB_NICE_LEVEL</glossterm>
>> + <glossentry
>> id='var-bb-BB_NICE_LEVEL'><glossterm>BB_NICE_LEVEL</glossterm>
>> <glossdef> <para>
>> Allows BitBake to run at a specific priority
>> @@ -564,13 +564,13 @@
>> System permissions usually mean that BitBake can
>> reduce its priority but not raise it again.
>> See
>> - <link
>> linkend='var-BB_TASK_NICE_LEVEL'><filename>BB_TASK_NICE_LEVEL</filename></link>
>> + <link
>> linkend='var-bb-BB_TASK_NICE_LEVEL'><filename>BB_TASK_NICE_LEVEL</filename></link>
>> for additional information. </para>
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_NO_NETWORK'><glossterm>BB_NO_NETWORK</glossterm>
>> + <glossentry
>> id='var-bb-BB_NO_NETWORK'><glossterm>BB_NO_NETWORK</glossterm>
>> <glossdef> <para>
>> Disables network access in the BitBake fetcher
>> modules. @@ -587,7 +587,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_NUMBER_THREADS'><glossterm>BB_NUMBER_THREADS</glossterm>
>> + <glossentry
>> id='var-bb-BB_NUMBER_THREADS'><glossterm>BB_NUMBER_THREADS</glossterm>
>> <glossdef> <para>
>> The maximum number of tasks BitBake should run
>> in parallel @@ -599,7 +599,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_NUMBER_PARSE_THREADS'><glossterm>BB_NUMBER_PARSE_THREADS</glossterm>
>> + <glossentry
>> id='var-bb-BB_NUMBER_PARSE_THREADS'><glossterm>BB_NUMBER_PARSE_THREADS</glossterm>
>> <glossdef> <para>
>> Sets the number of threads BitBake uses when
>> parsing. @@ -609,7 +609,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_ORIGENV'><glossterm>BB_ORIGENV</glossterm>
>> + <glossentry
>> id='var-bb-BB_ORIGENV'><glossterm>BB_ORIGENV</glossterm> <glossdef>
>> <para>
>> Contains a copy of the original external
>> environment in @@ -625,7 +625,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_PRESERVE_ENV'><glossterm>BB_PRESERVE_ENV</glossterm>
>> + <glossentry
>> id='var-bb-BB_PRESERVE_ENV'><glossterm>BB_PRESERVE_ENV</glossterm>
>> <glossdef> <para>
>> Disables whitelisting and instead allows all
>> variables @@ -639,12 +639,12 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_RUNFMT'><glossterm>BB_RUNFMT</glossterm>
>> + <glossentry
>> id='var-bb-BB_RUNFMT'><glossterm>BB_RUNFMT</glossterm> <glossdef>
>> <para>
>> Specifies the name of the executable script files
>> (i.e. run files) saved into
>> - <filename>${</filename><link
>> linkend='var-T'><filename>T</filename></link><filename>}</filename>.
>> + <filename>${</filename><link
>> linkend='var-bb-T'><filename>T</filename></link><filename>}</filename>.
>> By default, the <filename>BB_RUNFMT</filename> variable is undefined
>> and the run file names get created using the following form:
>> @@ -657,7 +657,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_RUNTASK'><glossterm>BB_RUNTASK</glossterm>
>> + <glossentry
>> id='var-bb-BB_RUNTASK'><glossterm>BB_RUNTASK</glossterm> <glossdef>
>> <para>
>> Contains the name of the currently executing
>> task. @@ -669,7 +669,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_SCHEDULER'><glossterm>BB_SCHEDULER</glossterm>
>> + <glossentry
>> id='var-bb-BB_SCHEDULER'><glossterm>BB_SCHEDULER</glossterm>
>> <glossdef> <para>
>> Selects the name of the scheduler to use for the
>> @@ -695,7 +695,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_SCHEDULERS'><glossterm>BB_SCHEDULERS</glossterm>
>> + <glossentry
>> id='var-bb-BB_SCHEDULERS'><glossterm>BB_SCHEDULERS</glossterm>
>> <glossdef> <para>
>> Defines custom schedulers to import.
>> @@ -705,13 +705,13 @@
>>
>> <para>
>> For information how to select a scheduler, see
>> the
>> - <link
>> linkend='var-BB_SCHEDULER'><filename>BB_SCHEDULER</filename></link>
>> + <link
>> linkend='var-bb-BB_SCHEDULER'><filename>BB_SCHEDULER</filename></link>
>> variable. </para>
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_SETSCENE_DEPVALID'><glossterm>BB_SETSCENE_DEPVALID</glossterm>
>> + <glossentry
>> id='var-bb-BB_SETSCENE_DEPVALID'><glossterm>BB_SETSCENE_DEPVALID</glossterm>
>> <glossdef> <para>
>> Specifies a function BitBake calls that
>> determines @@ -731,7 +731,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_SETSCENE_VERIFY_FUNCTION2'><glossterm>BB_SETSCENE_VERIFY_FUNCTION2</glossterm>
>> + <glossentry
>> id='var-bb-BB_SETSCENE_VERIFY_FUNCTION2'><glossterm>BB_SETSCENE_VERIFY_FUNCTION2</glossterm>
>> <glossdef> <para>
>> Specifies a function to call that verifies the
>> list of @@ -752,7 +752,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_SIGNATURE_EXCLUDE_FLAGS'><glossterm>BB_SIGNATURE_EXCLUDE_FLAGS</glossterm>
>> + <glossentry
>> id='var-bb-BB_SIGNATURE_EXCLUDE_FLAGS'><glossterm>BB_SIGNATURE_EXCLUDE_FLAGS</glossterm>
>> <glossdef> <para>
>> Lists variable flags (varflags)
>> @@ -771,7 +771,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_SIGNATURE_HANDLER'><glossterm>BB_SIGNATURE_HANDLER</glossterm>
>> + <glossentry
>> id='var-bb-BB_SIGNATURE_HANDLER'><glossterm>BB_SIGNATURE_HANDLER</glossterm>
>> <glossdef> <para>
>> Defines the name of the signature handler
>> BitBake uses. @@ -790,7 +790,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_SRCREV_POLICY'><glossterm>BB_SRCREV_POLICY</glossterm>
>> + <glossentry
>> id='var-bb-BB_SRCREV_POLICY'><glossterm>BB_SRCREV_POLICY</glossterm>
>> <glossdef> <para>
>> Defines the behavior of the fetcher when it
>> interacts with @@ -817,7 +817,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_STAMP_POLICY'><glossterm>BB_STAMP_POLICY</glossterm>
>> + <glossentry
>> id='var-bb-BB_STAMP_POLICY'><glossterm>BB_STAMP_POLICY</glossterm>
>> <glossdef> <para>
>> Defines the mode used for how timestamps of
>> stamp files @@ -836,7 +836,7 @@
>> <listitem><para><emphasis>whitelist</emphasis>
>> - Identical to "full" mode except timestamp
>> comparisons are made for recipes listed
>> in the
>> - <link
>> linkend='var-BB_STAMP_WHITELIST'><filename>BB_STAMP_WHITELIST</filename></link>
>> + <link
>> linkend='var-bb-BB_STAMP_WHITELIST'><filename>BB_STAMP_WHITELIST</filename></link>
>> variable. </para></listitem>
>> </itemizedlist>
>> @@ -848,19 +848,19 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_STAMP_WHITELIST'><glossterm>BB_STAMP_WHITELIST</glossterm>
>> + <glossentry
>> id='var-bb-BB_STAMP_WHITELIST'><glossterm>BB_STAMP_WHITELIST</glossterm>
>> <glossdef> <para>
>> Lists files whose stamp file timestamps are
>> compared when the stamp policy mode is set to "whitelist".
>> For information on stamp policies, see the
>> - <link
>> linkend='var-BB_STAMP_POLICY'><filename>BB_STAMP_POLICY</filename></link>
>> + <link
>> linkend='var-bb-BB_STAMP_POLICY'><filename>BB_STAMP_POLICY</filename></link>
>> variable. </para>
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_STRICT_CHECKSUM'><glossterm>BB_STRICT_CHECKSUM</glossterm>
>> + <glossentry
>> id='var-bb-BB_STRICT_CHECKSUM'><glossterm>BB_STRICT_CHECKSUM</glossterm>
>> <glossdef> <para>
>> Sets a more strict checksum mechanism for
>> non-local URLs. @@ -871,7 +871,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_TASK_IONICE_LEVEL'><glossterm>BB_TASK_IONICE_LEVEL</glossterm>
>> + <glossentry
>> id='var-bb-BB_TASK_IONICE_LEVEL'><glossterm>BB_TASK_IONICE_LEVEL</glossterm>
>> <glossdef> <para>
>> Allows adjustment of a task's Input/Output
>> priority. @@ -882,7 +882,7 @@
>> variable to adjust the I/O priority of these
>> tasks. <note>
>> This variable works similarly to the
>> - <link
>> linkend='var-BB_TASK_NICE_LEVEL'><filename>BB_TASK_NICE_LEVEL</filename></link>
>> + <link
>> linkend='var-bb-BB_TASK_NICE_LEVEL'><filename>BB_TASK_NICE_LEVEL</filename></link>
>> variable except with a task's I/O priorities. </note>
>> </para>
>> @@ -921,7 +921,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_TASK_NICE_LEVEL'><glossterm>BB_TASK_NICE_LEVEL</glossterm>
>> + <glossentry
>> id='var-bb-BB_TASK_NICE_LEVEL'><glossterm>BB_TASK_NICE_LEVEL</glossterm>
>> <glossdef> <para>
>> Allows specific tasks to change their priority
>> @@ -940,7 +940,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_TASKHASH'><glossterm>BB_TASKHASH</glossterm>
>> + <glossentry
>> id='var-bb-BB_TASKHASH'><glossterm>BB_TASKHASH</glossterm> <glossdef>
>> <para>
>> Within an executing task, this variable holds
>> the hash @@ -950,7 +950,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_VERBOSE_LOGS'><glossterm>BB_VERBOSE_LOGS</glossterm>
>> + <glossentry
>> id='var-bb-BB_VERBOSE_LOGS'><glossterm>BB_VERBOSE_LOGS</glossterm>
>> <glossdef> <para>
>> Controls how verbose BitBake is during builds.
>> @@ -960,7 +960,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_WORKERCONTEXT'><glossterm>BB_WORKERCONTEXT</glossterm>
>> + <glossentry
>> id='var-bb-BB_WORKERCONTEXT'><glossterm>BB_WORKERCONTEXT</glossterm>
>> <glossdef> <para>
>> Specifies if the current context is executing a
>> task. @@ -973,7 +973,7 @@
>> </glossentry>
>>
>>
>> - <glossentry
>> id='var-BBCLASSEXTEND'><glossterm>BBCLASSEXTEND</glossterm>
>> + <glossentry
>> id='var-bb-BBCLASSEXTEND'><glossterm>BBCLASSEXTEND</glossterm>
>> <glossdef> <para>
>> Allows you to extend a recipe so that it builds
>> variants @@ -1009,7 +1009,7 @@
>> <filename>_class-native</filename>.
>> For example, to generate a native version of
>> a recipe, a
>> - <link
>> linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
>> + <link
>> linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link> on "foo"
>> is rewritten to a <filename>DEPENDS</filename> on "foo-native".
>> </para>
>> @@ -1028,7 +1028,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-BBDEBUG'><glossterm>BBDEBUG</glossterm>
>> + <glossentry
>> id='var-bb-BBDEBUG'><glossterm>BBDEBUG</glossterm> <glossdef>
>> <para>
>> Sets the BitBake debug output level to a
>> specific value @@ -1042,7 +1042,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BBFILE_COLLECTIONS'><glossterm>BBFILE_COLLECTIONS</glossterm>
>> + <glossentry
>> id='var-bb-BBFILE_COLLECTIONS'><glossterm>BBFILE_COLLECTIONS</glossterm>
>> <glossdef> <para>Lists the names of configured layers.
>> These names are used to find the other
>> <filename>BBFILE_*</filename> @@ -1053,10 +1053,10 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BBFILE_PATTERN'><glossterm>BBFILE_PATTERN</glossterm>
>> + <glossentry
>> id='var-bb-BBFILE_PATTERN'><glossterm>BBFILE_PATTERN</glossterm>
>> <glossdef> <para>Variable that expands to match files from
>> - <link
>> linkend='var-BBFILES'><filename>BBFILES</filename></link>
>> + <link
>> linkend='var-bb-BBFILES'><filename>BBFILES</filename></link> in a
>> particular layer. This variable is used in the
>> <filename>conf/layer.conf</filename> file and must be suffixed with
>> the name of the specific layer (e.g. @@ -1064,7 +1064,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BBFILE_PRIORITY'><glossterm>BBFILE_PRIORITY</glossterm>
>> + <glossentry
>> id='var-bb-BBFILE_PRIORITY'><glossterm>BBFILE_PRIORITY</glossterm>
>> <glossdef> <para>Assigns the priority for recipe files in each
>> layer.</para> <para>This variable is useful in situations where the
>> same recipe appears in @@ -1074,7 +1074,7 @@
>> letting you control the precedence for the
>> multiple layers. The precedence established through this variable
>> stands regardless of a recipe's version
>> - (<link
>> linkend='var-PV'><filename>PV</filename></link> variable).
>> + (<link
>> linkend='var-bb-PV'><filename>PV</filename></link> variable). For
>> example, a layer that has a recipe with a higher
>> <filename>PV</filename> value but for which the
>> <filename>BBFILE_PRIORITY</filename> is set to have a lower
>> precedence still has a lower precedence.</para> @@ -1083,7 +1083,7 @@
>> For example, the value 6 has a higher precedence than the value 5. If
>> not specified, the <filename>BBFILE_PRIORITY</filename> variable is
>> set based on layer dependencies (see the
>> - <filename><link
>> linkend='var-LAYERDEPENDS'>LAYERDEPENDS</link></filename> variable for
>> + <filename><link
>> linkend='var-bb-LAYERDEPENDS'>LAYERDEPENDS</link></filename> variable
>> for more information. The default priority, if unspecified
>> for a layer with no dependencies, is the lowest
>> defined priority + 1 @@ -1095,7 +1095,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-BBFILES'><glossterm>BBFILES</glossterm>
>> + <glossentry
>> id='var-bb-BBFILES'><glossterm>BBFILES</glossterm> <glossdef>
>> <para>
>> A space-separated list of recipe files BitBake
>> uses to @@ -1113,7 +1113,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BBINCLUDED'><glossterm>BBINCLUDED</glossterm>
>> + <glossentry
>> id='var-bb-BBINCLUDED'><glossterm>BBINCLUDED</glossterm> <glossdef>
>> <para>
>> Contains a space-separated list of all of all
>> files that @@ -1123,7 +1123,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BBINCLUDELOGS'><glossterm>BBINCLUDELOGS</glossterm>
>> + <glossentry
>> id='var-bb-BBINCLUDELOGS'><glossterm>BBINCLUDELOGS</glossterm>
>> <glossdef> <para>
>> If set to a value, enables printing the task log
>> when @@ -1132,11 +1132,11 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BBINCLUDELOGS_LINES'><glossterm>BBINCLUDELOGS_LINES</glossterm>
>> + <glossentry
>> id='var-bb-BBINCLUDELOGS_LINES'><glossterm>BBINCLUDELOGS_LINES</glossterm>
>> <glossdef> <para>
>> If
>> - <link
>> linkend='var-BBINCLUDELOGS'><filename>BBINCLUDELOGS</filename></link>
>> + <link
>> linkend='var-bb-BBINCLUDELOGS'><filename>BBINCLUDELOGS</filename></link>
>> is set, specifies the maximum number of lines from the task log file
>> to print when reporting a failed task. If you do not set
>> <filename>BBINCLUDELOGS_LINES</filename>, @@ -1145,7 +1145,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-BBLAYERS'><glossterm>BBLAYERS</glossterm>
>> + <glossentry
>> id='var-bb-BBLAYERS'><glossterm>BBLAYERS</glossterm> <glossdef>
>> <para>Lists the layers to enable during the build.
>> This variable is defined in the
>> <filename>bblayers.conf</filename> configuration @@ -1166,7 +1166,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BBLAYERS_FETCH_DIR'><glossterm>BBLAYERS_FETCH_DIR</glossterm>
>> + <glossentry
>> id='var-bb-BBLAYERS_FETCH_DIR'><glossterm>BBLAYERS_FETCH_DIR</glossterm>
>> <glossdef> <para>
>> Sets the base location where layers are stored.
>> @@ -1178,7 +1178,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-BBMASK'><glossterm>BBMASK</glossterm>
>> + <glossentry id='var-bb-BBMASK'><glossterm>BBMASK</glossterm>
>> <glossdef>
>> <para>
>> Prevents BitBake from processing recipes and
>> recipe @@ -1236,7 +1236,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BBMULTICONFIG'><glossterm>BBMULTICONFIG</glossterm>
>> + <glossentry
>> id='var-bb-BBMULTICONFIG'><glossterm>BBMULTICONFIG</glossterm> <info>
>> BBMULTICONFIG[doc] = "Enables BitBake to perform
>> multiple configuration builds and lists each separate configuration
>> (multiconfig)." </info> @@ -1275,7 +1275,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-BBPATH'><glossterm>BBPATH</glossterm>
>> + <glossentry id='var-bb-BBPATH'><glossterm>BBPATH</glossterm>
>> <glossdef>
>> <para>
>> Used by BitBake to locate class
>> @@ -1302,7 +1302,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-BBSERVER'><glossterm>BBSERVER</glossterm>
>> + <glossentry
>> id='var-bb-BBSERVER'><glossterm>BBSERVER</glossterm> <glossdef>
>> <para>
>> Points to the server that runs memory-resident
>> BitBake. @@ -1312,7 +1312,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BBTARGETS'><glossterm>BBTARGETS</glossterm>
>> + <glossentry
>> id='var-bb-BBTARGETS'><glossterm>BBTARGETS</glossterm> <glossdef>
>> <para>
>> Allows you to use a configuration file to add to
>> the list @@ -1321,14 +1321,14 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BBVERSIONS'><glossterm>BBVERSIONS</glossterm>
>> + <glossentry
>> id='var-bb-BBVERSIONS'><glossterm>BBVERSIONS</glossterm> <glossdef>
>> <para>
>> Allows a single recipe to build multiple
>> versions of a project from a single recipe file.
>> You also able to specify conditional metadata
>> using the
>> - <link
>> linkend='var-OVERRIDES'><filename>OVERRIDES</filename></link>
>> + <link
>> linkend='var-bb-OVERRIDES'><filename>OVERRIDES</filename></link>
>> mechanism for a single version or for an optionally named range of
>> versions. </para>
>> @@ -1342,7 +1342,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BITBAKE_UI'><glossterm>BITBAKE_UI</glossterm>
>> + <glossentry
>> id='var-bb-BITBAKE_UI'><glossterm>BITBAKE_UI</glossterm> <glossdef>
>> <para>
>> Used to specify the UI module to use when
>> running BitBake. @@ -1356,7 +1356,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BUILDNAME'><glossterm>BUILDNAME</glossterm>
>> + <glossentry
>> id='var-bb-BUILDNAME'><glossterm>BUILDNAME</glossterm> <glossdef>
>> <para>
>> A name assigned to the build.
>> @@ -1366,7 +1366,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-BZRDIR'><glossterm>BZRDIR</glossterm>
>> + <glossentry id='var-bb-BZRDIR'><glossterm>BZRDIR</glossterm>
>> <glossdef>
>> <para>
>> The directory in which files checked out of a
>> Bazaar @@ -1377,9 +1377,9 @@
>>
>> </glossdiv>
>>
>> - <glossdiv id='var-glossary-c'><title>C</title>
>> + <glossdiv id='var-bb-glossary-c'><title>C</title>
>>
>> - <glossentry id='var-CACHE'><glossterm>CACHE</glossterm>
>> + <glossentry id='var-bb-CACHE'><glossterm>CACHE</glossterm>
>> <glossdef>
>> <para>
>> Specifies the directory BitBake uses to store a
>> cache @@ -1389,7 +1389,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-CVSDIR'><glossterm>CVSDIR</glossterm>
>> + <glossentry id='var-bb-CVSDIR'><glossterm>CVSDIR</glossterm>
>> <glossdef>
>> <para>
>> The directory in which files checked out under
>> the @@ -1400,9 +1400,9 @@
>>
>> </glossdiv>
>>
>> - <glossdiv id='var-glossary-d'><title>D</title>
>> + <glossdiv id='var-bb-glossary-d'><title>D</title>
>>
>> - <glossentry
>> id='var-DEFAULT_PREFERENCE'><glossterm>DEFAULT_PREFERENCE</glossterm>
>> + <glossentry
>> id='var-bb-DEFAULT_PREFERENCE'><glossterm>DEFAULT_PREFERENCE</glossterm>
>> <glossdef> <para>
>> Specifies a weak bias for recipe selection
>> priority. @@ -1413,20 +1413,20 @@
>> piece of software.
>> Using the variable in this way causes the stable
>> version of the recipe to build by default in the absence of
>> - <filename><link
>> linkend='var-PREFERRED_VERSION'>PREFERRED_VERSION</link></filename>
>> + <filename><link
>> linkend='var-bb-PREFERRED_VERSION'>PREFERRED_VERSION</link></filename>
>> being used to build the development version. </para>
>> <note>
>> The bias provided by
>> <filename>DEFAULT_PREFERENCE</filename> is weak and is overridden by
>> - <filename><link
>> linkend='var-BBFILE_PRIORITY'>BBFILE_PRIORITY</link></filename>
>> + <filename><link
>> linkend='var-bb-BBFILE_PRIORITY'>BBFILE_PRIORITY</link></filename> if
>> that variable is different between two layers that contain different
>> versions of the same recipe. </note>
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-DEPENDS'><glossterm>DEPENDS</glossterm>
>> + <glossentry
>> id='var-bb-DEPENDS'><glossterm>DEPENDS</glossterm> <glossdef>
>> <para>
>> Lists a recipe's build-time dependencies
>> @@ -1451,13 +1451,13 @@
>>
>> <para>
>> For information on runtime dependencies, see the
>> - <link
>> linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link>
>> + <link
>> linkend='var-bb-RDEPENDS'><filename>RDEPENDS</filename></link>
>> variable. </para>
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-DESCRIPTION'><glossterm>DESCRIPTION</glossterm>
>> + <glossentry
>> id='var-bb-DESCRIPTION'><glossterm>DESCRIPTION</glossterm> <glossdef>
>> <para>
>> A long description for the recipe.
>> @@ -1465,7 +1465,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-DL_DIR'><glossterm>DL_DIR</glossterm>
>> + <glossentry id='var-bb-DL_DIR'><glossterm>DL_DIR</glossterm>
>> <glossdef>
>> <para>
>> The central download directory used by the build
>> process to @@ -1474,7 +1474,7 @@
>> suitable for mirroring for everything except Git
>> repositories.
>> If you want tarballs of Git repositories, use the
>> - <link
>> linkend='var-BB_GENERATE_MIRROR_TARBALLS'><filename>BB_GENERATE_MIRROR_TARBALLS</filename></link>
>> + <link
>> linkend='var-bb-BB_GENERATE_MIRROR_TARBALLS'><filename>BB_GENERATE_MIRROR_TARBALLS</filename></link>
>> variable. </para>
>> </glossdef>
>> @@ -1482,9 +1482,9 @@
>> </glossentry>
>> </glossdiv>
>>
>> - <glossdiv id='var-glossary-e'><title>E</title>
>> + <glossdiv id='var-bb-glossary-e'><title>E</title>
>>
>> - <glossentry
>> id='var-EXCLUDE_FROM_WORLD'><glossterm>EXCLUDE_FROM_WORLD</glossterm>
>> + <glossentry
>> id='var-bb-EXCLUDE_FROM_WORLD'><glossterm>EXCLUDE_FROM_WORLD</glossterm>
>> <glossdef> <para>
>> Directs BitBake to exclude a recipe from world
>> builds (i.e. @@ -1512,9 +1512,9 @@
>>
>> </glossdiv>
>>
>> - <glossdiv id='var-glossary-f'><title>F</title>
>> + <glossdiv id='var-bb-glossary-f'><title>F</title>
>>
>> - <glossentry id='var-FAKEROOT'><glossterm>FAKEROOT</glossterm>
>> + <glossentry
>> id='var-bb-FAKEROOT'><glossterm>FAKEROOT</glossterm> <glossdef>
>> <para>
>> Contains the command to use when running a
>> shell script @@ -1527,19 +1527,19 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-FAKEROOTBASEENV'><glossterm>FAKEROOTBASEENV</glossterm>
>> + <glossentry
>> id='var-bb-FAKEROOTBASEENV'><glossterm>FAKEROOTBASEENV</glossterm>
>> <glossdef> <para>
>> Lists environment variables to set when
>> executing the command defined by
>> - <link
>> linkend='var-FAKEROOTCMD'><filename>FAKEROOTCMD</filename></link>
>> + <link
>> linkend='var-bb-FAKEROOTCMD'><filename>FAKEROOTCMD</filename></link>
>> that starts the bitbake-worker process in the fakeroot environment.
>> </para>
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-FAKEROOTCMD'><glossterm>FAKEROOTCMD</glossterm>
>> + <glossentry
>> id='var-bb-FAKEROOTCMD'><glossterm>FAKEROOTCMD</glossterm> <glossdef>
>> <para>
>> Contains the command that starts the
>> bitbake-worker @@ -1548,7 +1548,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-FAKEROOTDIRS'><glossterm>FAKEROOTDIRS</glossterm>
>> + <glossentry
>> id='var-bb-FAKEROOTDIRS'><glossterm>FAKEROOTDIRS</glossterm>
>> <glossdef> <para>
>> Lists directories to create before running a
>> task in @@ -1557,33 +1557,33 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-FAKEROOTENV'><glossterm>FAKEROOTENV</glossterm>
>> + <glossentry
>> id='var-bb-FAKEROOTENV'><glossterm>FAKEROOTENV</glossterm> <glossdef>
>> <para>
>> Lists environment variables to set when running
>> a task in the fakeroot environment.
>> For additional information on environment
>> variables and the fakeroot environment, see the
>> - <link
>> linkend='var-FAKEROOTBASEENV'><filename>FAKEROOTBASEENV</filename></link>
>> + <link
>> linkend='var-bb-FAKEROOTBASEENV'><filename>FAKEROOTBASEENV</filename></link>
>> variable. </para>
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-FAKEROOTNOENV'><glossterm>FAKEROOTNOENV</glossterm>
>> + <glossentry
>> id='var-bb-FAKEROOTNOENV'><glossterm>FAKEROOTNOENV</glossterm>
>> <glossdef> <para>
>> Lists environment variables to set when running
>> a task that is not in the fakeroot environment.
>> For additional information on environment
>> variables and the fakeroot environment, see the
>> - <link
>> linkend='var-FAKEROOTENV'><filename>FAKEROOTENV</filename></link>
>> + <link
>> linkend='var-bb-FAKEROOTENV'><filename>FAKEROOTENV</filename></link>
>> variable. </para>
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-FETCHCMD'><glossterm>FETCHCMD</glossterm>
>> + <glossentry
>> id='var-bb-FETCHCMD'><glossterm>FETCHCMD</glossterm> <glossdef>
>> <para>
>> Defines the command the BitBake fetcher module
>> @@ -1595,7 +1595,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-FILE'><glossterm>FILE</glossterm>
>> + <glossentry id='var-bb-FILE'><glossterm>FILE</glossterm>
>> <glossdef>
>> <para>
>> Points at the current file.
>> @@ -1607,7 +1607,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-FILESPATH'><glossterm>FILESPATH</glossterm>
>> + <glossentry
>> id='var-bb-FILESPATH'><glossterm>FILESPATH</glossterm> <glossdef>
>> <para>
>> Specifies directories BitBake uses when
>> searching for @@ -1625,9 +1625,9 @@
>> </glossdiv>
>>
>>
>> - <glossdiv id='var-glossary-g'><title>G</title>
>> + <glossdiv id='var-bb-glossary-g'><title>G</title>
>>
>> - <glossentry id='var-GITDIR'><glossterm>GITDIR</glossterm>
>> + <glossentry id='var-bb-GITDIR'><glossterm>GITDIR</glossterm>
>> <glossdef>
>> <para>
>> The directory in which a local copy of a Git
>> repository @@ -1639,9 +1639,9 @@
>> </glossdiv>
>>
>>
>> - <glossdiv id='var-glossary-h'><title>H</title>
>> + <glossdiv id='var-bb-glossary-h'><title>H</title>
>>
>> - <glossentry id='var-HGDIR'><glossterm>HGDIR</glossterm>
>> + <glossentry id='var-bb-HGDIR'><glossterm>HGDIR</glossterm>
>> <glossdef>
>> <para>
>> The directory in which files checked out of a
>> Mercurial @@ -1650,7 +1650,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-HOMEPAGE'><glossterm>HOMEPAGE</glossterm>
>> + <glossentry
>> id='var-bb-HOMEPAGE'><glossterm>HOMEPAGE</glossterm> <glossdef>
>> <para>Website where more information about the
>> software the recipe is building can be found.</para>
>> @@ -1659,9 +1659,9 @@
>>
>> </glossdiv>
>>
>> - <glossdiv id='var-glossary-i'><title>I</title>
>> + <glossdiv id='var-bb-glossary-i'><title>I</title>
>>
>> - <glossentry id='var-INHERIT'><glossterm>INHERIT</glossterm>
>> + <glossentry
>> id='var-bb-INHERIT'><glossterm>INHERIT</glossterm> <glossdef>
>> <para>
>> Causes the named class or classes to be
>> inherited globally. @@ -1691,15 +1691,15 @@
>> </glossdiv>
>> -->
>>
>> - <glossdiv id='var-glossary-l'><title>L</title>
>> + <glossdiv id='var-bb-glossary-l'><title>L</title>
>>
>> - <glossentry
>> id='var-LAYERDEPENDS'><glossterm>LAYERDEPENDS</glossterm>
>> + <glossentry
>> id='var-bb-LAYERDEPENDS'><glossterm>LAYERDEPENDS</glossterm>
>> <glossdef> <para>Lists the layers, separated by spaces, upon which
>> this recipe depends. Optionally, you can specify a specific layer
>> version for a dependency by adding it to the end of the layer name
>> with a colon, (e.g. "anotherlayer:3" to be compared against
>> - <link
>> linkend='var-LAYERVERSION'><filename>LAYERVERSION</filename></link><filename>_anotherlayer</filename>
>> + <link
>> linkend='var-bb-LAYERVERSION'><filename>LAYERVERSION</filename></link><filename>_anotherlayer</filename>
>> in this case). BitBake produces an error if any dependency is missing
>> or the version numbers do not match exactly (if specified).</para>
>> @@ -1710,7 +1710,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-LAYERDIR'><glossterm>LAYERDIR</glossterm>
>> + <glossentry
>> id='var-bb-LAYERDIR'><glossterm>LAYERDIR</glossterm> <glossdef>
>> <para>When used inside the
>> <filename>layer.conf</filename> configuration file, this variable
>> provides the path of the current layer. @@ -1719,22 +1719,22 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-LAYERDIR_RE'><glossterm>LAYERDIR_RE</glossterm>
>> + <glossentry
>> id='var-bb-LAYERDIR_RE'><glossterm>LAYERDIR_RE</glossterm> <glossdef>
>> <para>When used inside the
>> <filename>layer.conf</filename> configuration file, this variable
>> provides the path of the current layer, escaped for use in a regular
>> expression
>> - (<link
>> linkend='var-BBFILE_PATTERN'><filename>BBFILE_PATTERN</filename></link>).
>> + (<link
>> linkend='var-bb-BBFILE_PATTERN'><filename>BBFILE_PATTERN</filename></link>).
>> This variable is not available outside of
>> <filename>layer.conf</filename> and references are expanded
>> immediately when parsing of the file completes.</para> </glossdef>
>> </glossentry>
>> - <glossentry
>> id='var-LAYERVERSION'><glossterm>LAYERVERSION</glossterm>
>> + <glossentry
>> id='var-bb-LAYERVERSION'><glossterm>LAYERVERSION</glossterm>
>> <glossdef> <para>Optionally specifies the version of a layer as a
>> single number. You can use this variable within
>> - <link
>> linkend='var-LAYERDEPENDS'><filename>LAYERDEPENDS</filename></link>
>> + <link
>> linkend='var-bb-LAYERDEPENDS'><filename>LAYERDEPENDS</filename></link>
>> for another layer in order to depend on a specific version of the
>> layer.</para> <para>
>> @@ -1744,7 +1744,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-LICENSE'><glossterm>LICENSE</glossterm>
>> + <glossentry
>> id='var-bb-LICENSE'><glossterm>LICENSE</glossterm> <glossdef>
>> <para>
>> The list of source licenses for the recipe.
>> @@ -1754,9 +1754,9 @@
>>
>> </glossdiv>
>>
>> - <glossdiv id='var-glossary-m'><title>M</title>
>> + <glossdiv id='var-bb-glossary-m'><title>M</title>
>>
>> - <glossentry id='var-MIRRORS'><glossterm>MIRRORS</glossterm>
>> + <glossentry
>> id='var-bb-MIRRORS'><glossterm>MIRRORS</glossterm> <glossdef>
>> <para>
>> Specifies additional paths from which BitBake
>> gets source code. @@ -1764,14 +1764,14 @@
>> tries the local download directory.
>> If that location fails, the build system tries
>> locations defined by
>> - <link
>> linkend='var-PREMIRRORS'><filename>PREMIRRORS</filename></link>,
>> + <link
>> linkend='var-bb-PREMIRRORS'><filename>PREMIRRORS</filename></link>,
>> the upstream source, and then locations specified by
>> <filename>MIRRORS</filename> in that order. </para>
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-MULTI_PROVIDER_WHITELIST'><glossterm>MULTI_PROVIDER_WHITELIST</glossterm>
>> + <glossentry
>> id='var-bb-MULTI_PROVIDER_WHITELIST'><glossterm>MULTI_PROVIDER_WHITELIST</glossterm>
>> <glossdef> <para>
>> Allows you to suppress BitBake warnings caused
>> when @@ -1804,9 +1804,9 @@
>> </glossdiv>
>> -->
>>
>> - <glossdiv id='var-glossary-o'><title>O</title>
>> + <glossdiv id='var-bb-glossary-o'><title>O</title>
>>
>> - <glossentry
>> id='var-OVERRIDES'><glossterm>OVERRIDES</glossterm>
>> + <glossentry
>> id='var-bb-OVERRIDES'><glossterm>OVERRIDES</glossterm> <glossdef>
>> <para>
>> BitBake uses <filename>OVERRIDES</filename> to
>> control @@ -1829,9 +1829,9 @@
>> </glossentry>
>> </glossdiv>
>>
>> - <glossdiv id='var-glossary-p'><title>P</title>
>> + <glossdiv id='var-bb-glossary-p'><title>P</title>
>>
>> - <glossentry id='var-P4DIR'><glossterm>P4DIR</glossterm>
>> + <glossentry id='var-bb-P4DIR'><glossterm>P4DIR</glossterm>
>> <glossdef>
>> <para>
>> The directory in which a local copy of a
>> Perforce depot @@ -1840,14 +1840,14 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-PACKAGES'><glossterm>PACKAGES</glossterm>
>> + <glossentry
>> id='var-bb-PACKAGES'><glossterm>PACKAGES</glossterm> <glossdef>
>> <para>The list of packages the recipe creates.
>> </para>
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-PACKAGES_DYNAMIC'><glossterm>PACKAGES_DYNAMIC</glossterm>
>> + <glossentry
>> id='var-bb-PACKAGES_DYNAMIC'><glossterm>PACKAGES_DYNAMIC</glossterm>
>> <glossdef> <para>
>> A promise that your recipe satisfies runtime
>> dependencies @@ -1856,7 +1856,7 @@
>> does not actually satisfy the dependencies, it
>> only states that they should be satisfied.
>> For example, if a hard, runtime dependency
>> - (<link
>> linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link>)
>> + (<link
>> linkend='var-bb-RDEPENDS'><filename>RDEPENDS</filename></link>) of
>> another package is satisfied during the build through the
>> <filename>PACKAGES_DYNAMIC</filename> variable, but a package with
>> the module name is never actually @@ -1865,7 +1865,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-PE'><glossterm>PE</glossterm>
>> + <glossentry id='var-bb-PE'><glossterm>PE</glossterm>
>> <glossdef>
>> <para>
>> The epoch of the recipe.
>> @@ -1877,7 +1877,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-PERSISTENT_DIR'><glossterm>PERSISTENT_DIR</glossterm>
>> + <glossentry
>> id='var-bb-PERSISTENT_DIR'><glossterm>PERSISTENT_DIR</glossterm>
>> <glossdef> <para>
>> Specifies the directory BitBake uses to store
>> data that @@ -1889,7 +1889,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-PF'><glossterm>PF</glossterm>
>> + <glossentry id='var-bb-PF'><glossterm>PF</glossterm>
>> <glossdef>
>> <para>
>> Specifies the recipe or package name and
>> includes all version and revision @@ -1899,27 +1899,27 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-PN'><glossterm>PN</glossterm>
>> + <glossentry id='var-bb-PN'><glossterm>PN</glossterm>
>> <glossdef>
>> <para>The recipe name.</para>
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-PR'><glossterm>PR</glossterm>
>> + <glossentry id='var-bb-PR'><glossterm>PR</glossterm>
>> <glossdef>
>> <para>The revision of the recipe.
>> </para>
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-PREFERRED_PROVIDER'><glossterm>PREFERRED_PROVIDER</glossterm>
>> + <glossentry
>> id='var-bb-PREFERRED_PROVIDER'><glossterm>PREFERRED_PROVIDER</glossterm>
>> <glossdef> <para>
>> Determines which recipe should be given
>> preference when multiple recipes provide the same item.
>> You should always suffix the variable with the
>> name of the provided item, and you should set it to the
>> - <link
>> linkend='var-PN'><filename>PN</filename></link>
>> + <link
>> linkend='var-bb-PN'><filename>PN</filename></link> of the recipe to
>> which you want to give precedence. Some examples:
>> <literallayout class='monospaced'>
>> @@ -1931,14 +1931,14 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-PREFERRED_PROVIDERS'><glossterm>PREFERRED_PROVIDERS</glossterm>
>> + <glossentry
>> id='var-bb-PREFERRED_PROVIDERS'><glossterm>PREFERRED_PROVIDERS</glossterm>
>> <glossdef> <para>
>> Determines which recipe should be given
>> preference for cases where multiple recipes provide the same item.
>> Functionally,
>> <filename>PREFERRED_PROVIDERS</filename> is
>> identical to
>> - <link
>> linkend='var-PREFERRED_PROVIDER'><filename>PREFERRED_PROVIDER</filename></link>.
>> + <link
>> linkend='var-bb-PREFERRED_PROVIDER'><filename>PREFERRED_PROVIDER</filename></link>.
>> However, the <filename>PREFERRED_PROVIDERS</filename> variable lets
>> you define preferences for multiple situations using the following
>> form: @@ -1954,15 +1954,15 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-PREFERRED_VERSION'><glossterm>PREFERRED_VERSION</glossterm>
>> + <glossentry
>> id='var-bb-PREFERRED_VERSION'><glossterm>PREFERRED_VERSION</glossterm>
>> <glossdef> <para>
>> If there are multiple versions of recipes
>> available, this variable determines which recipe should be given
>> preference. You must always suffix the variable with the
>> - <link
>> linkend='var-PN'><filename>PN</filename></link>
>> + <link
>> linkend='var-bb-PN'><filename>PN</filename></link> you want to
>> select, and you should set
>> - <link
>> linkend='var-PV'><filename>PV</filename></link>
>> + <link
>> linkend='var-bb-PV'><filename>PV</filename></link> accordingly for
>> precedence. </para>
>>
>> @@ -1989,7 +1989,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-PREMIRRORS'><glossterm>PREMIRRORS</glossterm>
>> + <glossentry
>> id='var-bb-PREMIRRORS'><glossterm>PREMIRRORS</glossterm> <glossdef>
>> <para>
>> Specifies additional paths from which BitBake
>> gets source code. @@ -1998,7 +1998,7 @@
>> If that location fails, the build system tries
>> locations defined by <filename>PREMIRRORS</filename>, the upstream
>> source, and then locations specified by
>> - <link
>> linkend='var-MIRRORS'><filename>MIRRORS</filename></link>
>> + <link
>> linkend='var-bb-MIRRORS'><filename>MIRRORS</filename></link> in that
>> order. </para>
>>
>> @@ -2022,20 +2022,20 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-PROVIDES'><glossterm>PROVIDES</glossterm>
>> + <glossentry
>> id='var-bb-PROVIDES'><glossterm>PROVIDES</glossterm> <glossdef>
>> <para>
>> A list of aliases by which a particular recipe
>> can be known.
>> By default, a recipe's own
>> - <filename><link
>> linkend='var-PN'>PN</link></filename>
>> + <filename><link
>> linkend='var-bb-PN'>PN</link></filename> is implicitly already in its
>> <filename>PROVIDES</filename> list.
>> If a recipe uses <filename>PROVIDES</filename>,
>> the additional aliases are synonyms for the recipe and can
>> be useful satisfying dependencies of other
>> recipes during the build as specified by
>> - <filename><link
>> linkend='var-DEPENDS'>DEPENDS</link></filename>.
>> + <filename><link
>> linkend='var-bb-DEPENDS'>DEPENDS</link></filename>. </para>
>>
>> <para>
>> @@ -2059,7 +2059,7 @@
>> virtual target in <filename>PROVIDES</filename>.
>> Recipes that depend on the functionality in
>> question can include the virtual target in
>> - <link
>> linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
>> + <link
>> linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link> to leave
>> the choice of provider open. </para>
>>
>> @@ -2072,11 +2072,11 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-PRSERV_HOST'><glossterm>PRSERV_HOST</glossterm>
>> + <glossentry
>> id='var-bb-PRSERV_HOST'><glossterm>PRSERV_HOST</glossterm> <glossdef>
>> <para>
>> The network based
>> - <link
>> linkend='var-PR'><filename>PR</filename></link>
>> + <link
>> linkend='var-bb-PR'><filename>PR</filename></link> service host and
>> port. </para>
>>
>> @@ -2094,7 +2094,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-PV'><glossterm>PV</glossterm>
>> + <glossentry id='var-bb-PV'><glossterm>PV</glossterm>
>> <glossdef>
>> <para>The version of the recipe.
>> </para>
>> @@ -2108,9 +2108,9 @@
>> </glossdiv>
>> -->
>>
>> - <glossdiv id='var-glossary-r'><title>R</title>
>> + <glossdiv id='var-bb-glossary-r'><title>R</title>
>>
>> - <glossentry id='var-RDEPENDS'><glossterm>RDEPENDS</glossterm>
>> + <glossentry
>> id='var-bb-RDEPENDS'><glossterm>RDEPENDS</glossterm> <glossdef>
>> <para>
>> Lists a package's runtime dependencies (i.e.
>> other packages) @@ -2165,13 +2165,13 @@
>>
>> <para>
>> For information on build-time dependencies, see
>> the
>> - <link
>> linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
>> + <link
>> linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link> variable.
>> </para>
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-REPODIR'><glossterm>REPODIR</glossterm>
>> + <glossentry
>> id='var-bb-REPODIR'><glossterm>REPODIR</glossterm> <glossdef>
>> <para>
>> The directory in which a local copy of a
>> @@ -2181,14 +2181,14 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-RPROVIDES'><glossterm>RPROVIDES</glossterm>
>> + <glossentry
>> id='var-bb-RPROVIDES'><glossterm>RPROVIDES</glossterm> <glossdef>
>> <para>
>> A list of package name aliases that a package
>> also provides. These aliases are useful for satisfying runtime
>> dependencies of other packages both during the build and on the target
>> (as specified by
>> - <filename><link
>> linkend='var-RDEPENDS'>RDEPENDS</link></filename>).
>> + <filename><link
>> linkend='var-bb-RDEPENDS'>RDEPENDS</link></filename>). </para>
>> <para>
>> As with all package-controlling variables, you
>> must always @@ -2201,7 +2201,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-RRECOMMENDS'><glossterm>RRECOMMENDS</glossterm>
>> + <glossentry
>> id='var-bb-RRECOMMENDS'><glossterm>RRECOMMENDS</glossterm> <glossdef>
>> <para>
>> A list of packages that extends the usability of
>> a package @@ -2210,7 +2210,7 @@
>> packages in order to successfully build, but
>> needs them for the extended usability.
>> To specify runtime dependencies for packages,
>> see the
>> - <filename><link
>> linkend='var-RDEPENDS'>RDEPENDS</link></filename>
>> + <filename><link
>> linkend='var-bb-RDEPENDS'>RDEPENDS</link></filename> variable.
>> </para>
>>
>> @@ -2243,15 +2243,15 @@
>>
>> </glossdiv>
>>
>> - <glossdiv id='var-glossary-s'><title>S</title>
>> + <glossdiv id='var-bb-glossary-s'><title>S</title>
>>
>> - <glossentry id='var-SECTION'><glossterm>SECTION</glossterm>
>> + <glossentry
>> id='var-bb-SECTION'><glossterm>SECTION</glossterm> <glossdef>
>> <para>The section in which packages should be
>> categorized.</para> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-SRC_URI'><glossterm>SRC_URI</glossterm>
>> + <glossentry
>> id='var-bb-SRC_URI'><glossterm>SRC_URI</glossterm> <glossdef>
>> <para>
>> The list of source files - local or remote.
>> @@ -2272,7 +2272,7 @@
>> the metadata,
>> from the local machine.
>> The path is relative to the
>> - <link
>> linkend='var-FILESPATH'><filename>FILESPATH</filename></link>
>> + <link
>> linkend='var-bb-FILESPATH'><filename>FILESPATH</filename></link>
>> variable.</para></listitem>
>> <listitem><para><emphasis><filename>bzr://</filename> -</emphasis>
>> Fetches files from a Bazaar revision control
>> repository.</para></listitem> @@ -2322,7 +2322,7 @@ </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-SRCDATE'><glossterm>SRCDATE</glossterm>
>> + <glossentry
>> id='var-bb-SRCDATE'><glossterm>SRCDATE</glossterm> <glossdef>
>> <para>
>> The date of the source code used to build the
>> package. @@ -2331,7 +2331,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-SRCREV'><glossterm>SRCREV</glossterm>
>> + <glossentry id='var-bb-SRCREV'><glossterm>SRCREV</glossterm>
>> <glossdef>
>> <para>
>> The revision of the source code used to build
>> the package. @@ -2344,13 +2344,13 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-SRCREV_FORMAT'><glossterm>SRCREV_FORMAT</glossterm>
>> + <glossentry
>> id='var-bb-SRCREV_FORMAT'><glossterm>SRCREV_FORMAT</glossterm>
>> <glossdef> <para>
>> Helps construct valid
>> - <link
>> linkend='var-SRCREV'><filename>SRCREV</filename></link>
>> + <link
>> linkend='var-bb-SRCREV'><filename>SRCREV</filename></link> values
>> when multiple source controlled URLs are used in
>> - <link
>> linkend='var-SRC_URI'><filename>SRC_URI</filename></link>.
>> + <link
>> linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link>. </para>
>>
>> <para>
>> @@ -2371,7 +2371,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-STAMP'><glossterm>STAMP</glossterm>
>> + <glossentry id='var-bb-STAMP'><glossterm>STAMP</glossterm>
>> <glossdef>
>> <para>
>> Specifies the base path used to create recipe
>> stamp files. @@ -2381,12 +2381,12 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-STAMPCLEAN'><glossterm>STAMPCLEAN</glossterm>
>> + <glossentry
>> id='var-bb-STAMPCLEAN'><glossterm>STAMPCLEAN</glossterm> <glossdef>
>> <para>
>> Specifies the base path used to create recipe
>> stamp files. Unlike the
>> - <link
>> linkend='var-STAMP'><filename>STAMP</filename></link>
>> + <link
>> linkend='var-bb-STAMP'><filename>STAMP</filename></link> variable,
>> <filename>STAMPCLEAN</filename> can contain wildcards to match the
>> range of files a clean operation should remove.
>> @@ -2396,7 +2396,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-SUMMARY'><glossterm>SUMMARY</glossterm>
>> + <glossentry
>> id='var-bb-SUMMARY'><glossterm>SUMMARY</glossterm> <glossdef>
>> <para>
>> A short summary for the recipe, which is 72
>> characters or less. @@ -2404,7 +2404,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-SVNDIR'><glossterm>SVNDIR</glossterm>
>> + <glossentry id='var-bb-SVNDIR'><glossterm>SVNDIR</glossterm>
>> <glossdef>
>> <para>
>> The directory in which files checked out of a
>> Subversion @@ -2415,9 +2415,9 @@
>>
>> </glossdiv>
>>
>> - <glossdiv id='var-glossary-t'><title>T</title>
>> + <glossdiv id='var-bb-glossary-t'><title>T</title>
>>
>> - <glossentry id='var-T'><glossterm>T</glossterm>
>> + <glossentry id='var-bb-T'><glossterm>T</glossterm>
>> <glossdef>
>> <para>Points to a directory were BitBake places
>> temporary files, which consist mostly of task
>> logs and @@ -2426,7 +2426,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-TOPDIR'><glossterm>TOPDIR</glossterm>
>> + <glossentry id='var-bb-TOPDIR'><glossterm>TOPDIR</glossterm>
>> <glossdef>
>> <para>
>> Points to the build directory.
>> diff --git a/bitbake/doc/poky.ent b/bitbake/doc/poky.ent
>> index c032e14..85d9c83 100644
>> --- a/bitbake/doc/poky.ent
>> +++ b/bitbake/doc/poky.ent
>> @@ -17,13 +17,6 @@
>> <!ENTITY OE_DOCS_URL "http://docs.openembedded.org">
>> <!ENTITY OH_HOME_URL "http://o-hand.com">
>> <!ENTITY BITBAKE_HOME_URL
>> "http://developer.berlios.de/projects/bitbake/"> -<!ENTITY
>> ECLIPSE_MAIN_URL "http://www.eclipse.org/downloads"> -<!ENTITY
>> ECLIPSE_DL_URL "http://download.eclipse.org"> -<!ENTITY
>> ECLIPSE_DL_PLUGIN_URL
>> "&YOCTO_DL_URL;/releases/eclipse-plugin/&DISTRO;"> -<!ENTITY
>> ECLIPSE_UPDATES_URL "&ECLIPSE_DL_URL;/tm/updates/3.3"> -<!ENTITY
>> ECLIPSE_INDIGO_URL "&ECLIPSE_DL_URL;/releases/indigo"> -<!ENTITY
>> ECLIPSE_JUNO_URL "&ECLIPSE_DL_URL;/releases/juno"> -<!ENTITY
>> ECLIPSE_INDIGO_CDT_URL "&ECLIPSE_DL_URL;tools/cdt/releases/indigo">
>> <!ENTITY YOCTO_DOCS_URL "&YOCTO_HOME_URL;/docs"> <!ENTITY
>> YOCTO_SOURCES_URL "&YOCTO_HOME_URL;/sources/"> <!ENTITY
>> YOCTO_AB_PORT_URL "&YOCTO_AB_URL;:8010"> @@ -31,7 +24,6 @@ <!ENTITY
>> YOCTO_POKY_URL "&YOCTO_DL_URL;/releases/poky/"> <!ENTITY
>> YOCTO_RELEASE_DL_URL "&YOCTO_DL_URL;/releases/yocto/yocto-&DISTRO;">
>> <!ENTITY YOCTO_TOOLCHAIN_DL_URL "&YOCTO_RELEASE_DL_URL;/toolchain/">
>> -<!ENTITY YOCTO_ECLIPSE_DL_URL
>> "&YOCTO_RELEASE_DL_URL;/eclipse-plugin/indigo;"> <!ENTITY
>> YOCTO_ADTINSTALLER_DL_URL "&YOCTO_RELEASE_DL_URL;/adt_installer">
>> <!ENTITY YOCTO_POKY_DL_URL
>> "&YOCTO_RELEASE_DL_URL;/&YOCTO_POKY;.tar.bz2"> <!ENTITY
>> YOCTO_MACHINES_DL_URL "&YOCTO_RELEASE_DL_URL;/machines"> diff --git
>> a/bitbake/lib/bb/COW.py b/bitbake/lib/bb/COW.py index
>> 7817473..d26e981 100644 --- a/bitbake/lib/bb/COW.py +++
>> b/bitbake/lib/bb/COW.py @@ -1,23 +1,8 @@ -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # This is a copy on write dictionary and set which abuses classes to
>> try and be nice and fast. #
>> # Copyright (C) 2006 Tim Ansell
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. -#
>> #Please Note:
>> # Be careful when using mutable types (ie Dict and Lists) -
>> operations involving these are SLOW. # Assign a file to __warn__ to
>> get warnings about slow operations. diff --git
>> a/bitbake/lib/bb/__init__.py b/bitbake/lib/bb/__init__.py index
>> 4bc47c8..c144311 100644 --- a/bitbake/lib/bb/__init__.py
>> +++ b/bitbake/lib/bb/__init__.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # BitBake Build System Python Library
>> #
>> @@ -8,20 +6,10 @@
>> #
>> # Based on Gentoo's portage.py.
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> -__version__ = "1.40.0"
>> +__version__ = "1.44.0"
>>
>> import sys
>> if sys.version_info < (3, 4, 0):
>> diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py
>> index 3e2a94e..30a2ba2 100644
>> --- a/bitbake/lib/bb/build.py
>> +++ b/bitbake/lib/bb/build.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # BitBake 'Build' implementation
>> #
>> @@ -10,18 +8,7 @@
>> #
>> # Based on Gentoo's portage.py.
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> # Based on functions from the base bb module, Copyright 2003 Holger
>> Schurig
>> @@ -67,23 +54,6 @@ else:
>> builtins['bb'] = bb
>> builtins['os'] = os
>>
>> -class FuncFailed(Exception):
>> - def __init__(self, name = None, logfile = None):
>> - self.logfile = logfile
>> - self.name = name
>> - if name:
>> - self.msg = 'Function failed: %s' % name
>> - else:
>> - self.msg = "Function failed"
>> -
>> - def __str__(self):
>> - if self.logfile and os.path.exists(self.logfile):
>> - msg = ("%s (log file is located at %s)" %
>> - (self.msg, self.logfile))
>> - else:
>> - msg = self.msg
>> - return msg
>> -
>> class TaskBase(event.Event):
>> """Base class for task events"""
>>
>> @@ -176,15 +146,33 @@ class LogTee(object):
>>
>> def __repr__(self):
>> return '<LogTee {0}>'.format(self.name)
>> +
>> def flush(self):
>> self.outfile.flush()
>>
>> -#
>> -# pythonexception allows the python exceptions generated to be raised
>> -# as the real exceptions (not FuncFailed) and without a backtrace at
>> the -# origin of the failure.
>> -#
>> -def exec_func(func, d, dirs = None, pythonexception=False):
>> +
>> +class StdoutNoopContextManager:
>> + """
>> + This class acts like sys.stdout, but adds noop __enter__ and
>> __exit__ methods.
>> + """
>> + def __enter__(self):
>> + return sys.stdout
>> +
>> + def __exit__(self, *exc_info):
>> + pass
>> +
>> + def write(self, string):
>> + return sys.stdout.write(string)
>> +
>> + def flush(self):
>> + sys.stdout.flush()
>> +
>> + @property
>> + def name(self):
>> + return sys.stdout.name
>> +
>> +
>> +def exec_func(func, d, dirs = None):
>> """Execute a BB 'function'"""
>>
>> try:
>> @@ -256,7 +244,7 @@ def exec_func(func, d, dirs = None,
>> pythonexception=False):
>> with bb.utils.fileslocked(lockfiles):
>> if ispython:
>> - exec_func_python(func, d, runfile, cwd=adir,
>> pythonexception=pythonexception)
>> + exec_func_python(func, d, runfile, cwd=adir)
>> else:
>> exec_func_shell(func, d, runfile, cwd=adir)
>>
>> @@ -276,7 +264,7 @@ _functionfmt = """
>> {function}(d)
>> """
>> logformatter = bb.msg.BBLogFormatter("%(levelname)s: %(message)s")
>> -def exec_func_python(func, d, runfile, cwd=None,
>> pythonexception=False): +def exec_func_python(func, d, runfile,
>> cwd=None): """Execute a python BB 'function'"""
>>
>> code = _functionfmt.format(function=func)
>> @@ -301,13 +289,7 @@ def exec_func_python(func, d, runfile, cwd=None,
>> pythonexception=False): bb.methodpool.insert_method(func, text, fn,
>> lineno - 1)
>> comp = utils.better_compile(code, func, "exec_python_func()
>> autogenerated")
>> - utils.better_exec(comp, {"d": d}, code, "exec_python_func()
>> autogenerated", pythonexception=pythonexception)
>> - except (bb.parse.SkipRecipe, bb.build.FuncFailed):
>> - raise
>> - except:
>> - if pythonexception:
>> - raise
>> - raise FuncFailed(func, None)
>> + utils.better_exec(comp, {"d": d}, code, "exec_python_func()
>> autogenerated") finally:
>> bb.debug(2, "Python function %s finished" % func)
>>
>> @@ -335,6 +317,42 @@ trap 'bb_exit_handler' 0
>> set -e
>> '''
>>
>> +def create_progress_handler(func, progress, logfile, d):
>> + if progress == 'percent':
>> + # Use default regex
>> + return bb.progress.BasicProgressHandler(d, outfile=logfile)
>> + elif progress.startswith('percent:'):
>> + # Use specified regex
>> + return bb.progress.BasicProgressHandler(d,
>> regex=progress.split(':', 1)[1], outfile=logfile)
>> + elif progress.startswith('outof:'):
>> + # Use specified regex
>> + return bb.progress.OutOfProgressHandler(d,
>> regex=progress.split(':', 1)[1], outfile=logfile)
>> + elif progress.startswith("custom:"):
>> + # Use a custom progress handler that was injected via
>> OE_EXTRA_IMPORTS or __builtins__
>> + import functools
>> + from types import ModuleType
>> +
>> + parts = progress.split(":", 2)
>> + _, cls, otherargs = parts[0], parts[1], (parts[2] or None)
>> if parts[2:] else None
>> + if cls:
>> + def resolve(x, y):
>> + if not x:
>> + return None
>> + if isinstance(x, ModuleType):
>> + return getattr(x, y, None)
>> + return x.get(y)
>> + cls_obj = functools.reduce(resolve, cls.split("."),
>> bb.utils._context)
>> + if not cls_obj:
>> + # Fall-back on __builtins__
>> + cls_obj = functools.reduce(lambda x, y: x.get(y),
>> cls.split("."), __builtins__)
>> + if cls_obj:
>> + return cls_obj(d, outfile=logfile,
>> otherargs=otherargs)
>> + bb.warn('%s: unknown custom progress handler in task
>> progress varflag value "%s", ignoring' % (func, cls))
>> + else:
>> + bb.warn('%s: invalid task progress varflag value "%s",
>> ignoring' % (func, progress)) +
>> + return logfile
>> +
>> def exec_func_shell(func, d, runfile, cwd=None):
>> """Execute a shell function from the metadata
>>
>> @@ -372,23 +390,13 @@ exit $ret
>> cmd = [fakerootcmd, runfile]
>>
>> if bb.msg.loggerDefaultVerbose:
>> - logfile = LogTee(logger, sys.stdout)
>> + logfile = LogTee(logger, StdoutNoopContextManager())
>> else:
>> - logfile = sys.stdout
>> + logfile = StdoutNoopContextManager()
>>
>> progress = d.getVarFlag(func, 'progress')
>> if progress:
>> - if progress == 'percent':
>> - # Use default regex
>> - logfile = bb.progress.BasicProgressHandler(d,
>> outfile=logfile)
>> - elif progress.startswith('percent:'):
>> - # Use specified regex
>> - logfile = bb.progress.BasicProgressHandler(d,
>> regex=progress.split(':', 1)[1], outfile=logfile)
>> - elif progress.startswith('outof:'):
>> - # Use specified regex
>> - logfile = bb.progress.OutOfProgressHandler(d,
>> regex=progress.split(':', 1)[1], outfile=logfile)
>> - else:
>> - bb.warn('%s: invalid task progress varflag value "%s",
>> ignoring' % (func, progress))
>> + logfile = create_progress_handler(func, progress, logfile, d)
>>
>> fifobuffer = bytearray()
>> def readfifo(data):
>> @@ -407,6 +415,8 @@ exit $ret
>> bb.plain(value)
>> elif cmd == 'bbnote':
>> bb.note(value)
>> + elif cmd == 'bbverbnote':
>> + bb.verbnote(value)
>> elif cmd == 'bbwarn':
>> bb.warn(value)
>> elif cmd == 'bberror':
>> @@ -436,13 +446,8 @@ exit $ret
>> with open(fifopath, 'r+b', buffering=0) as fifo:
>> try:
>> bb.debug(2, "Executing shell function %s" % func)
>> -
>> - try:
>> - with open(os.devnull, 'r+') as stdin:
>> - bb.process.run(cmd, shell=False, stdin=stdin,
>> log=logfile, extrafiles=[(fifo,readfifo)])
>> - except bb.process.CmdError:
>> - logfn = d.getVar('BB_LOGFILE')
>> - raise FuncFailed(func, logfn)
>> + with open(os.devnull, 'r+') as stdin, logfile:
>> + bb.process.run(cmd, shell=False, stdin=stdin,
>> log=logfile, extrafiles=[(fifo,readfifo)]) finally:
>> os.unlink(fifopath)
>>
>> @@ -570,9 +575,6 @@ def _exec_task(fn, task, d, quieterr):
>> event.fire(TaskStarted(task, logfn, flags, localdata),
>> localdata) except (bb.BBHandledException, SystemExit):
>> return 1
>> - except FuncFailed as exc:
>> - logger.error(str(exc))
>> - return 1
>>
>> try:
>> for func in (prefuncs or '').split():
>> @@ -580,7 +582,10 @@ def _exec_task(fn, task, d, quieterr):
>> exec_func(task, localdata)
>> for func in (postfuncs or '').split():
>> exec_func(func, localdata)
>> - except FuncFailed as exc:
>> + except bb.BBHandledException:
>> + event.fire(TaskFailed(task, logfn, localdata, True),
>> localdata)
>> + return 1
>> + except Exception as exc:
>> if quieterr:
>> event.fire(TaskFailedSilent(task, logfn, localdata),
>> localdata) else:
>> @@ -588,9 +593,6 @@ def _exec_task(fn, task, d, quieterr):
>> logger.error(str(exc))
>> event.fire(TaskFailed(task, logfn, localdata,
>> errprinted), localdata) return 1
>> - except bb.BBHandledException:
>> - event.fire(TaskFailed(task, logfn, localdata, True),
>> localdata)
>> - return 1
>> finally:
>> sys.stdout.flush()
>> sys.stderr.flush()
>> @@ -814,6 +816,9 @@ def add_tasks(tasklist, d):
>> task_deps['parents'][task] = []
>> if 'deps' in flags:
>> for dep in flags['deps']:
>> + # Check and warn for "addtask task after foo" while
>> foo does not exist
>> + #if not dep in tasklist:
>> + # bb.warn('%s: dependent task %s for %s does not
>> exist' % (d.getVar('PN'), dep, task)) dep = d.expand(dep)
>> task_deps['parents'][task].append(dep)
>>
>> diff --git a/bitbake/lib/bb/cache.py b/bitbake/lib/bb/cache.py
>> index 258d679..b6f7da5 100644
>> --- a/bitbake/lib/bb/cache.py
>> +++ b/bitbake/lib/bb/cache.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # BitBake Cache implementation
>> #
>> @@ -15,18 +13,8 @@
>> # Copyright (C) 2005 Holger Hans Peter Freyther
>> # Copyright (C) 2005 ROAD GmbH
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import os
>> import sys
>> @@ -95,21 +83,21 @@ class CoreRecipeInfo(RecipeInfoCommon):
>> self.appends = self.listvar('__BBAPPEND', metadata)
>> self.nocache = self.getvar('BB_DONT_CACHE', metadata)
>>
>> + self.provides = self.depvar('PROVIDES', metadata)
>> + self.rprovides = self.depvar('RPROVIDES', metadata)
>> + self.pn = self.getvar('PN', metadata) or
>> bb.parse.vars_from_file(filename,metadata)[0]
>> + self.packages = self.listvar('PACKAGES', metadata)
>> + if not self.packages:
>> + self.packages.append(self.pn)
>> + self.packages_dynamic = self.listvar('PACKAGES_DYNAMIC',
>> metadata) +
>> self.skipreason = self.getvar('__SKIPPED', metadata)
>> if self.skipreason:
>> - self.pn = self.getvar('PN', metadata) or
>> bb.parse.BBHandler.vars_from_file(filename,metadata)[0] self.skipped
>> = True
>> - self.provides = self.depvar('PROVIDES', metadata)
>> - self.rprovides = self.depvar('RPROVIDES', metadata)
>> return
>>
>> self.tasks = metadata.getVar('__BBTASKS', False)
>>
>> - self.pn = self.getvar('PN', metadata)
>> - self.packages = self.listvar('PACKAGES', metadata)
>> - if not self.packages:
>> - self.packages.append(self.pn)
>> -
>> self.basetaskhashes = self.taskvar('BB_BASEHASH',
>> self.tasks, metadata) self.hashfilename =
>> self.getvar('BB_HASHFILENAME', metadata)
>> @@ -125,11 +113,8 @@ class CoreRecipeInfo(RecipeInfoCommon):
>> self.stampclean = self.getvar('STAMPCLEAN', metadata)
>> self.stamp_extrainfo = self.flaglist('stamp-extra-info',
>> self.tasks, metadata) self.file_checksums =
>> self.flaglist('file-checksums', self.tasks, metadata, True)
>> - self.packages_dynamic = self.listvar('PACKAGES_DYNAMIC',
>> metadata) self.depends = self.depvar('DEPENDS', metadata)
>> - self.provides = self.depvar('PROVIDES', metadata)
>> self.rdepends = self.depvar('RDEPENDS', metadata)
>> - self.rprovides = self.depvar('RPROVIDES', metadata)
>> self.rrecommends = self.depvar('RRECOMMENDS', metadata)
>> self.rprovides_pkg = self.pkgvar('RPROVIDES',
>> self.packages, metadata) self.rdepends_pkg =
>> self.pkgvar('RDEPENDS', self.packages, metadata) @@ -235,7 +220,7 @@
>> class CoreRecipeInfo(RecipeInfoCommon):
>> cachedata.hashfn[fn] = self.hashfilename
>> for task, taskhash in self.basetaskhashes.items():
>> - identifier = '%s.%s' % (fn, task)
>> + identifier = '%s:%s' % (fn, task)
>> cachedata.basetaskhash[identifier] = taskhash
>>
>> cachedata.inherits[fn] = self.inherits
>> @@ -249,7 +234,7 @@ def virtualfn2realfn(virtualfn):
>> Convert a virtual file name to a real one + the associated
>> subclass keyword """
>> mc = ""
>> - if virtualfn.startswith('multiconfig:'):
>> + if virtualfn.startswith('mc:'):
>> elems = virtualfn.split(':')
>> mc = elems[1]
>> virtualfn = ":".join(elems[2:])
>> @@ -270,7 +255,7 @@ def realfn2virtual(realfn, cls, mc):
>> if cls:
>> realfn = "virtual:" + cls + ":" + realfn
>> if mc:
>> - realfn = "multiconfig:" + mc + ":" + realfn
>> + realfn = "mc:" + mc + ":" + realfn
>> return realfn
>>
>> def variant2virtual(realfn, variant):
>> @@ -279,11 +264,11 @@ def variant2virtual(realfn, variant):
>> """
>> if variant == "":
>> return realfn
>> - if variant.startswith("multiconfig:"):
>> + if variant.startswith("mc:"):
>> elems = variant.split(":")
>> if elems[2]:
>> - return "multiconfig:" + elems[1] + ":virtual:" +
>> ":".join(elems[2:]) + ":" + realfn
>> - return "multiconfig:" + elems[1] + ":" + realfn
>> + return "mc:" + elems[1] + ":virtual:" +
>> ":".join(elems[2:]) + ":" + realfn
>> + return "mc:" + elems[1] + ":" + realfn
>> return "virtual:" + variant + ":" + realfn
>>
>> def parse_recipe(bb_data, bbfile, appends, mc=''):
>> @@ -361,7 +346,7 @@ class NoCache(object):
>> bb_data = self.databuilder.mcdata[mc].createCopy()
>> newstores = parse_recipe(bb_data, bbfile, appends, mc)
>> for ns in newstores:
>> - datastores["multiconfig:%s:%s" % (mc, ns)] =
>> newstores[ns]
>> + datastores["mc:%s:%s" % (mc, ns)] = newstores[ns]
>>
>> return datastores
>>
>> @@ -411,6 +396,15 @@ class Cache(NoCache):
>> else:
>> logger.debug(1, "Cache file %s not found, building..." %
>> self.cachefile)
>> + # We don't use the symlink, its just for debugging
>> convinience
>> + symlink = os.path.join(self.cachedir, "bb_cache.dat")
>> + if os.path.exists(symlink):
>> + bb.utils.remove(symlink)
>> + try:
>> + os.symlink(os.path.basename(self.cachefile), symlink)
>> + except OSError:
>> + pass
>> +
>> def load_cachefile(self):
>> cachesize = 0
>> previous_progress = 0
>> @@ -889,3 +883,56 @@ class MultiProcessCache(object):
>> p.dump([data, self.__class__.CACHE_VERSION])
>>
>> bb.utils.unlockfile(glf)
>> +
>> +
>> +class SimpleCache(object):
>> + """
>> + BitBake multi-process cache implementation
>> +
>> + Used by the codeparser & file checksum caches
>> + """
>> +
>> + def __init__(self, version):
>> + self.cachefile = None
>> + self.cachedata = None
>> + self.cacheversion = version
>> +
>> + def init_cache(self, d, cache_file_name=None, defaultdata=None):
>> + cachedir = (d.getVar("PERSISTENT_DIR") or
>> + d.getVar("CACHE"))
>> + if not cachedir:
>> + return defaultdata
>> +
>> + bb.utils.mkdirhier(cachedir)
>> + self.cachefile = os.path.join(cachedir,
>> + cache_file_name or
>> self.__class__.cache_file_name)
>> + logger.debug(1, "Using cache in '%s'", self.cachefile)
>> +
>> + glf = bb.utils.lockfile(self.cachefile + ".lock")
>> +
>> + try:
>> + with open(self.cachefile, "rb") as f:
>> + p = pickle.Unpickler(f)
>> + data, version = p.load()
>> + except:
>> + bb.utils.unlockfile(glf)
>> + return defaultdata
>> +
>> + bb.utils.unlockfile(glf)
>> +
>> + if version != self.cacheversion:
>> + return defaultdata
>> +
>> + return data
>> +
>> + def save(self, data):
>> + if not self.cachefile:
>> + return
>> +
>> + glf = bb.utils.lockfile(self.cachefile + ".lock")
>> +
>> + with open(self.cachefile, "wb") as f:
>> + p = pickle.Pickler(f, -1)
>> + p.dump([data, self.cacheversion])
>> +
>> + bb.utils.unlockfile(glf)
>> diff --git a/bitbake/lib/bb/cache_extra.py
>> b/bitbake/lib/bb/cache_extra.py index 83f4959..bf4226d 100644
>> --- a/bitbake/lib/bb/cache_extra.py
>> +++ b/bitbake/lib/bb/cache_extra.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # Extra RecipeInfo will be all defined in this file. Currently,
>> # Only Hob (Image Creator) Requests some extra fields. So
>> @@ -12,18 +10,8 @@
>>
>> # Copyright (C) 2011, Intel Corporation. All rights reserved.
>>
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from bb.cache import RecipeInfoCommon
>>
>> diff --git a/bitbake/lib/bb/checksum.py b/bitbake/lib/bb/checksum.py
>> index 4e1598f..5bc8a8f 100644
>> --- a/bitbake/lib/bb/checksum.py
>> +++ b/bitbake/lib/bb/checksum.py
>> @@ -2,18 +2,8 @@
>> #
>> # Copyright (C) 2012 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import glob
>> import operator
>> diff --git a/bitbake/lib/bb/codeparser.py
>> b/bitbake/lib/bb/codeparser.py index ddd1b97..fd2c473 100644
>> --- a/bitbake/lib/bb/codeparser.py
>> +++ b/bitbake/lib/bb/codeparser.py
>> @@ -1,3 +1,7 @@
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> """
>> BitBake code parser
>>
>> @@ -33,7 +37,7 @@ from bb.cache import MultiProcessCache
>> logger = logging.getLogger('BitBake.CodeParser')
>>
>> def bbhash(s):
>> - return hashlib.md5(s.encode("utf-8")).hexdigest()
>> + return hashlib.sha256(s.encode("utf-8")).hexdigest()
>>
>> def check_indent(codestr):
>> """If the code is indented, add a top level piece of code to
>> 'remove' the indentation""" @@ -140,7 +144,7 @@ class
>> CodeParserCache(MultiProcessCache): # so that an existing cache gets
>> invalidated. Additionally you'll need # to increment
>> __cache_version__ in cache.py in order to ensure that old # recipe
>> caches don't trigger "Taskhash mismatch" errors.
>> - CACHE_VERSION = 10
>> + CACHE_VERSION = 11
>>
>> def __init__(self):
>> MultiProcessCache.__init__(self)
>> @@ -368,8 +372,9 @@ class ShellParser():
>> def _parse_shell(self, value):
>> try:
>> tokens, _ = pyshyacc.parse(value, eof=True, debug=False)
>> - except pyshlex.NeedMore:
>> - raise sherrors.ShellSyntaxError("Unexpected EOF")
>> + except Exception:
>> + bb.error('Error during parse shell code, the last 5
>> lines are:\n%s' % '\n'.join(value.split('\n')[-5:]))
>> + raise
>>
>> self.process_tokens(tokens)
>>
>> diff --git a/bitbake/lib/bb/command.py b/bitbake/lib/bb/command.py
>> index 6c966e3..378f389 100644
>> --- a/bitbake/lib/bb/command.py
>> +++ b/bitbake/lib/bb/command.py
>> @@ -6,18 +6,8 @@ Provide an interface to interact with the bitbake
>> server through 'commands'
>> # Copyright (C) 2006-2007 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> """
>> The bitbake server takes 'commands' from its UI/commandline.
>> diff --git a/bitbake/lib/bb/compat.py b/bitbake/lib/bb/compat.py
>> index de1923d..4935668 100644
>> --- a/bitbake/lib/bb/compat.py
>> +++ b/bitbake/lib/bb/compat.py
>> @@ -1,3 +1,7 @@
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> """Code pulled from future python versions, here for compatibility"""
>>
>> from collections import MutableMapping, KeysView, ValuesView,
>> ItemsView, OrderedDict diff --git a/bitbake/lib/bb/cooker.py
>> b/bitbake/lib/bb/cooker.py index 16681ba..20ef04d 100644
>> --- a/bitbake/lib/bb/cooker.py
>> +++ b/bitbake/lib/bb/cooker.py
>> @@ -1,6 +1,3 @@
>> -#!/usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # Copyright (C) 2003, 2004 Chris Larson
>> # Copyright (C) 2003, 2004 Phil Blundell
>> @@ -9,19 +6,8 @@
>> # Copyright (C) 2005 ROAD GmbH
>> # Copyright (C) 2006 - 2007 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. -
>>
>> import sys, os, glob, os.path, re, time
>> import atexit
>> @@ -45,6 +31,7 @@ import pyinotify
>> import json
>> import pickle
>> import codecs
>> +import hashserv
>>
>> logger = logging.getLogger("BitBake")
>> collectlog = logging.getLogger("BitBake.Collection")
>> @@ -175,27 +162,45 @@ class BBCooker:
>>
>> self.configuration = configuration
>>
>> + bb.debug(1, "BBCooker starting %s" % time.time())
>> + sys.stdout.flush()
>> +
>> self.configwatcher = pyinotify.WatchManager()
>> + bb.debug(1, "BBCooker pyinotify1 %s" % time.time())
>> + sys.stdout.flush()
>> +
>> self.configwatcher.bbseen = []
>> self.configwatcher.bbwatchedfiles = []
>> self.confignotifier = pyinotify.Notifier(self.configwatcher,
>> self.config_notifications)
>> + bb.debug(1, "BBCooker pyinotify2 %s" % time.time())
>> + sys.stdout.flush()
>> self.watchmask = pyinotify.IN_CLOSE_WRITE |
>> pyinotify.IN_CREATE | pyinotify.IN_DELETE | \
>> pyinotify.IN_DELETE_SELF | pyinotify.IN_MODIFY |
>> pyinotify.IN_MOVE_SELF | \ pyinotify.IN_MOVED_FROM |
>> pyinotify.IN_MOVED_TO self.watcher = pyinotify.WatchManager()
>> + bb.debug(1, "BBCooker pyinotify3 %s" % time.time())
>> + sys.stdout.flush()
>> self.watcher.bbseen = []
>> self.watcher.bbwatchedfiles = []
>> self.notifier = pyinotify.Notifier(self.watcher,
>> self.notifications)
>> + bb.debug(1, "BBCooker pyinotify complete %s" % time.time())
>> + sys.stdout.flush()
>> +
>> # If being called by something like tinfoil, we need to
>> clean cached data # which may now be invalid
>> bb.parse.clear_cache()
>> bb.parse.BBHandler.cached_statements = {}
>>
>> self.ui_cmdline = None
>> + self.hashserv = None
>> + self.hashservaddr = None
>>
>> self.initConfigurationData()
>>
>> + bb.debug(1, "BBCooker parsed base configuration %s" %
>> time.time())
>> + sys.stdout.flush()
>> +
>> # we log all events to a file if so directed
>> if self.configuration.writeeventlog:
>> # register the log file writer as UI Handler
>> @@ -233,6 +238,9 @@ class BBCooker:
>> # Let SIGHUP exit as SIGTERM
>> signal.signal(signal.SIGHUP, self.sigterm_exception)
>>
>> + bb.debug(1, "BBCooker startup complete %s" % time.time())
>> + sys.stdout.flush()
>> +
>> def process_inotify_updates(self):
>> for n in [self.confignotifier, self.notifier]:
>> if n.check_events(timeout=0):
>> @@ -367,13 +375,12 @@ class BBCooker:
>> # Copy of the data store which has been expanded.
>> # Used for firing events and accessing variables where
>> expansion needs to be accounted for #
>> - bb.parse.init_parser(self.data)
>> -
>> if CookerFeatures.BASEDATASTORE_TRACKING in self.featureset:
>> self.disableDataTracking()
>>
>> - self.data.renameVar("__depends", "__base_depends")
>> - self.add_filewatch(self.data.getVar("__base_depends",
>> False), self.configwatcher)
>> + for mc in self.databuilder.mcdata.values():
>> + mc.renameVar("__depends", "__base_depends")
>> + self.add_filewatch(mc.getVar("__base_depends", False),
>> self.configwatcher)
>> self.baseconfig_valid = True
>> self.parsecache_valid = False
>> @@ -385,6 +392,22 @@ class BBCooker:
>> except prserv.serv.PRServiceConfigError as e:
>> bb.fatal("Unable to start PR Server, exitting")
>>
>> + if self.data.getVar("BB_HASHSERVE") == "auto":
>> + # Create a new hash server bound to a unix domain socket
>> + if not self.hashserv:
>> + dbfile = (self.data.getVar("PERSISTENT_DIR") or
>> self.data.getVar("CACHE")) + "/hashserv.db"
>> + self.hashservaddr = "unix://%s/hashserve.sock" %
>> self.data.getVar("TOPDIR")
>> + self.hashserv =
>> hashserv.create_server(self.hashservaddr, dbfile, sync=False)
>> + self.hashserv.process =
>> multiprocessing.Process(target=self.hashserv.serve_forever)
>> + self.hashserv.process.start()
>> + self.data.setVar("BB_HASHSERVE", self.hashservaddr)
>> + self.databuilder.origdata.setVar("BB_HASHSERVE",
>> self.hashservaddr)
>> + self.databuilder.data.setVar("BB_HASHSERVE",
>> self.hashservaddr)
>> + for mc in self.databuilder.mcdata:
>> + self.databuilder.mcdata[mc].setVar("BB_HASHSERVE",
>> self.hashservaddr) +
>> + bb.parse.init_parser(self.data)
>> +
>> def enableDataTracking(self):
>> self.configuration.tracking = True
>> if hasattr(self, "data"):
>> @@ -488,6 +511,7 @@ class BBCooker:
>> """
>> fn = None
>> envdata = None
>> + mc = ''
>> if not pkgs_to_build:
>> pkgs_to_build = []
>>
>> @@ -496,6 +520,12 @@ class BBCooker:
>> self.enableDataTracking()
>> self.reset()
>>
>> + def mc_base(p):
>> + if p.startswith('mc:'):
>> + s = p.split(':')
>> + if len(s) == 2:
>> + return s[1]
>> + return None
>>
>> if buildfile:
>> # Parse the configuration here. We need to do it
>> explicitly here since @@ -506,18 +536,16 @@ class BBCooker:
>> fn = self.matchFile(fn)
>> fn = bb.cache.realfn2virtual(fn, cls, mc)
>> elif len(pkgs_to_build) == 1:
>> - ignore = self.data.getVar("ASSUME_PROVIDED") or ""
>> - if pkgs_to_build[0] in set(ignore.split()):
>> - bb.fatal("%s is in ASSUME_PROVIDED" %
>> pkgs_to_build[0])
>> + mc = mc_base(pkgs_to_build[0])
>> + if not mc:
>> + ignore = self.data.getVar("ASSUME_PROVIDED") or ""
>> + if pkgs_to_build[0] in set(ignore.split()):
>> + bb.fatal("%s is in ASSUME_PROVIDED" %
>> pkgs_to_build[0])
>> - taskdata, runlist = self.buildTaskData(pkgs_to_build,
>> None, self.configuration.abort, allowincomplete=True)
>> + taskdata, runlist =
>> self.buildTaskData(pkgs_to_build, None, self.configuration.abort,
>> allowincomplete=True)
>> - mc = runlist[0][0]
>> - fn = runlist[0][3]
>> - else:
>> - envdata = self.data
>> - data.expandKeys(envdata)
>> - parse.ast.runAnonFuncs(envdata)
>> + mc = runlist[0][0]
>> + fn = runlist[0][3]
>>
>> if fn:
>> try:
>> @@ -526,6 +554,12 @@ class BBCooker:
>> except Exception as e:
>> parselog.exception("Unable to read %s", fn)
>> raise
>> + else:
>> + if not mc in self.databuilder.mcdata:
>> + bb.fatal('Not multiconfig named "%s" found' % mc)
>> + envdata = self.databuilder.mcdata[mc]
>> + data.expandKeys(envdata)
>> + parse.ast.runAnonFuncs(envdata)
>>
>> # Display history
>> with closing(StringIO()) as env:
>> @@ -565,10 +599,10 @@ class BBCooker:
>> wildcard = False
>>
>> # Wild card expansion:
>> - # Replace string such as "multiconfig:*:bash"
>> - # into "multiconfig:A:bash multiconfig:B:bash bash"
>> + # Replace string such as "mc:*:bash"
>> + # into "mc:A:bash mc:B:bash bash"
>> for k in targetlist:
>> - if k.startswith("multiconfig:"):
>> + if k.startswith("mc:"):
>> if wildcard:
>> bb.fatal('multiconfig conflict')
>> if k.split(":")[1] == "*":
>> @@ -601,7 +635,7 @@ class BBCooker:
>> runlist = []
>> for k in fulltargetlist:
>> mc = ""
>> - if k.startswith("multiconfig:"):
>> + if k.startswith("mc:"):
>> mc = k.split(":")[1]
>> k = ":".join(k.split(":")[2:])
>> ktask = task
>> @@ -620,13 +654,22 @@ class BBCooker:
>> runlist.append([mc, k, ktask, fn])
>> bb.event.fire(bb.event.TreeDataPreparationProgress(current,
>> len(fulltargetlist)), self.data)
>> - mcdeps = taskdata[mc].get_mcdepends()
>> + havemc = False
>> + for mc in self.multiconfigs:
>> + if taskdata[mc].get_mcdepends():
>> + havemc = True
>> +
>> # No need to do check providers if there are no mcdeps or
>> not an mc build
>> - if mcdeps and mc:
>> - # Make sure we can provide the multiconfig dependency
>> + if havemc or len(self.multiconfigs) > 1:
>> seen = set()
>> new = True
>> + # Make sure we can provide the multiconfig dependency
>> while new:
>> + mcdeps = set()
>> + # Add unresolved first, so we can get multiconfig
>> indirect dependencies on time
>> + for mc in self.multiconfigs:
>> + taskdata[mc].add_unresolved(localdata[mc],
>> self.recipecaches[mc])
>> + mcdeps |= set(taskdata[mc].get_mcdepends())
>> new = False
>> for mc in self.multiconfigs:
>> for k in mcdeps:
>> @@ -641,6 +684,7 @@ class BBCooker:
>> taskdata[depmc].add_provider(localdata[depmc],
>> self.recipecaches[depmc], l[3]) seen.add(k)
>> new = True
>> +
>> for mc in self.multiconfigs:
>> taskdata[mc].add_unresolved(localdata[mc],
>> self.recipecaches[mc])
>> @@ -676,7 +720,7 @@ class BBCooker:
>> @staticmethod
>> def add_mc_prefix(mc, pn):
>> if mc:
>> - return "multiconfig:%s:%s" % (mc, pn)
>> + return "mc:%s:%s" % (mc, pn)
>> return pn
>>
>> def buildDependTree(self, rq, taskdata):
>> @@ -875,6 +919,10 @@ class BBCooker:
>> os.unlink('package-depends.dot')
>> except FileNotFoundError:
>> pass
>> + try:
>> + os.unlink('recipe-depends.dot')
>> + except FileNotFoundError:
>> + pass
>>
>> with open('task-depends.dot', 'w') as f:
>> f.write("digraph depends {\n")
>> @@ -888,27 +936,6 @@ class BBCooker:
>> f.write("}\n")
>> logger.info("Task dependencies saved to 'task-depends.dot'")
>>
>> - with open('recipe-depends.dot', 'w') as f:
>> - f.write("digraph depends {\n")
>> - pndeps = {}
>> - for task in sorted(depgraph["tdepends"]):
>> - (pn, taskname) = task.rsplit(".", 1)
>> - if pn not in pndeps:
>> - pndeps[pn] = set()
>> - for dep in sorted(depgraph["tdepends"][task]):
>> - (deppn, deptaskname) = dep.rsplit(".", 1)
>> - pndeps[pn].add(deppn)
>> - for pn in sorted(pndeps):
>> - fn = depgraph["pn"][pn]["filename"]
>> - version = depgraph["pn"][pn]["version"]
>> - f.write('"%s" [label="%s\\n%s\\n%s"]\n' % (pn, pn,
>> version, fn))
>> - for dep in sorted(pndeps[pn]):
>> - if dep == pn:
>> - continue
>> - f.write('"%s" -> "%s"\n' % (pn, dep))
>> - f.write("}\n")
>> - logger.info("Flattened recipe dependencies saved to
>> 'recipe-depends.dot'") -
>> def show_appends_with_no_recipes(self):
>> # Determine which bbappends haven't been applied
>>
>> @@ -1191,8 +1218,8 @@ class BBCooker:
>> continue
>> elif regex == "":
>> parselog.debug(1, "BBFILE_PATTERN_%s is empty" %
>> c)
>> + cre = re.compile('^NULL$')
>> errors = False
>> - continue
>> else:
>> try:
>> cre = re.compile(regex)
>> @@ -1453,7 +1480,7 @@ class BBCooker:
>> ntargets = []
>> for target in runlist:
>> if target[0]:
>> - ntargets.append("multiconfig:%s:%s:%s" % (target[0],
>> target[1], target[2]))
>> + ntargets.append("mc:%s:%s:%s" % (target[0],
>> target[1], target[2])) ntargets.append("%s:%s" % (target[1],
>> target[2]))
>> for mc in self.multiconfigs:
>> @@ -1576,6 +1603,9 @@ class BBCooker:
>> for pkg in pkgs_to_build:
>> if pkg in ignore:
>> parselog.warning("Explicit target \"%s\" is in
>> ASSUME_PROVIDED, ignoring" % pkg)
>> + if pkg.startswith("multiconfig:"):
>> + pkgs_to_build.remove(pkg)
>> + pkgs_to_build.append(pkg.replace("multiconfig:",
>> "mc:"))
>> if 'world' in pkgs_to_build:
>> pkgs_to_build.remove('world')
>> @@ -1583,7 +1613,7 @@ class BBCooker:
>> bb.providers.buildWorldTargetList(self.recipecaches[mc],
>> task) for t in self.recipecaches[mc].world_target:
>> if mc:
>> - t = "multiconfig:" + mc + ":" + t
>> + t = "mc:" + mc + ":" + t
>> pkgs_to_build.append(t)
>>
>> if 'universe' in pkgs_to_build:
>> @@ -1602,7 +1632,7 @@ class BBCooker:
>> bb.debug(1, "Skipping %s for universe
>> tasks as task %s doesn't exist" % (t, task)) continue
>> if mc:
>> - t = "multiconfig:" + mc + ":" + t
>> + t = "mc:" + mc + ":" + t
>> pkgs_to_build.append(t)
>>
>> return pkgs_to_build
>> @@ -1615,9 +1645,11 @@ class BBCooker:
>>
>> def post_serve(self):
>> prserv.serv.auto_shutdown()
>> + if self.hashserv:
>> + self.hashserv.process.terminate()
>> + self.hashserv.process.join()
>> bb.event.fire(CookerExit(), self.data)
>>
>> -
>> def shutdown(self, force = False):
>> if force:
>> self.state = state.forceshutdown
>> @@ -1632,6 +1664,7 @@ class BBCooker:
>>
>> def reset(self):
>> self.initConfigurationData()
>> + self.handlePRServ()
>>
>> def clientComplete(self):
>> """Called when the client is done using the server"""
>> @@ -1865,35 +1898,6 @@ class ParsingFailure(Exception):
>> self.recipe = recipe
>> Exception.__init__(self, realexception, recipe)
>>
>> -class Feeder(multiprocessing.Process):
>> - def __init__(self, jobs, to_parsers, quit):
>> - self.quit = quit
>> - self.jobs = jobs
>> - self.to_parsers = to_parsers
>> - multiprocessing.Process.__init__(self)
>> -
>> - def run(self):
>> - while True:
>> - try:
>> - quit = self.quit.get_nowait()
>> - except queue.Empty:
>> - pass
>> - else:
>> - if quit == 'cancel':
>> - self.to_parsers.cancel_join_thread()
>> - break
>> -
>> - try:
>> - job = self.jobs.pop()
>> - except IndexError:
>> - break
>> -
>> - try:
>> - self.to_parsers.put(job, timeout=0.5)
>> - except queue.Full:
>> - self.jobs.insert(0, job)
>> - continue
>> -
>> class Parser(multiprocessing.Process):
>> def __init__(self, jobs, results, quit, init, profile):
>> self.jobs = jobs
>> @@ -1940,11 +1944,8 @@ class Parser(multiprocessing.Process):
>> result = pending.pop()
>> else:
>> try:
>> - job = self.jobs.get(timeout=0.25)
>> - except queue.Empty:
>> - continue
>> -
>> - if job is None:
>> + job = self.jobs.pop()
>> + except IndexError:
>> break
>> result = self.parse(*job)
>>
>> @@ -2028,14 +2029,15 @@ class CookerParser(object):
>> multiprocessing.util.Finalize(None,
>> bb.codeparser.parser_cache_save, exitpriority=1)
>> multiprocessing.util.Finalize(None, bb.fetch.fetcher_parse_save,
>> exitpriority=1)
>> - self.feeder_quit = multiprocessing.Queue(maxsize=1)
>> self.parser_quit =
>> multiprocessing.Queue(maxsize=self.num_processes)
>> - self.jobs =
>> multiprocessing.Queue(maxsize=self.num_processes) self.result_queue =
>> multiprocessing.Queue()
>> - self.feeder = Feeder(self.willparse, self.jobs,
>> self.feeder_quit)
>> - self.feeder.start()
>> +
>> + def chunkify(lst,n):
>> + return [lst[i::n] for i in range(n)]
>> + self.jobs = chunkify(self.willparse, self.num_processes)
>> +
>> for i in range(0, self.num_processes):
>> - parser = Parser(self.jobs, self.result_queue,
>> self.parser_quit, init, self.cooker.configuration.profile)
>> + parser = Parser(self.jobs[i], self.result_queue,
>> self.parser_quit, init, self.cooker.configuration.profile)
>> parser.start() self.process_names.append(parser.name)
>> self.processes.append(parser)
>> @@ -2056,17 +2058,20 @@ class CookerParser(object):
>> self.total)
>>
>> bb.event.fire(event, self.cfgdata)
>> - self.feeder_quit.put(None)
>> for process in self.processes:
>> self.parser_quit.put(None)
>> else:
>> - self.feeder_quit.put('cancel')
>> -
>> self.parser_quit.cancel_join_thread()
>> for process in self.processes:
>> self.parser_quit.put(None)
>>
>> - self.jobs.cancel_join_thread()
>> + # Cleanup the queue before call process.join(), otherwise
>> there might be
>> + # deadlocks.
>> + while True:
>> + try:
>> + self.result_queue.get(timeout=0.25)
>> + except queue.Empty:
>> + break
>>
>> for process in self.processes:
>> if force:
>> @@ -2074,7 +2079,6 @@ class CookerParser(object):
>> process.terminate()
>> else:
>> process.join()
>> - self.feeder.join()
>>
>> sync = threading.Thread(target=self.bb_cache.sync)
>> sync.start()
>> diff --git a/bitbake/lib/bb/cookerdata.py
>> b/bitbake/lib/bb/cookerdata.py index 5df66e6..472423f 100644
>> --- a/bitbake/lib/bb/cookerdata.py
>> +++ b/bitbake/lib/bb/cookerdata.py
>> @@ -1,6 +1,3 @@
>> -#!/usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # Copyright (C) 2003, 2004 Chris Larson
>> # Copyright (C) 2003, 2004 Phil Blundell
>> @@ -9,23 +6,14 @@
>> # Copyright (C) 2005 ROAD GmbH
>> # Copyright (C) 2006 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import logging
>> import os
>> import re
>> import sys
>> +import hashlib
>> from functools import wraps
>> import bb
>> from bb import data
>> @@ -134,6 +122,7 @@ class CookerConfiguration(object):
>> self.profile = False
>> self.nosetscene = False
>> self.setsceneonly = False
>> + self.skipsetscene = False
>> self.invalidate_stamp = False
>> self.dump_signatures = []
>> self.dry_run = False
>> @@ -279,12 +268,13 @@ class CookerDataBuilder(object):
>> self.mcdata = {}
>>
>> def parseBaseConfiguration(self):
>> + data_hash = hashlib.sha256()
>> try:
>> - bb.parse.init_parser(self.basedata)
>> self.data = self.parseConfigurationFiles(self.prefiles,
>> self.postfiles)
>> if self.data.getVar("BB_WORKERCONTEXT", False) is None:
>> bb.fetch.fetcher_init(self.data)
>> + bb.parse.init_parser(self.data)
>> bb.codeparser.parser_cache_init(self.data)
>>
>> bb.event.fire(bb.event.ConfigParsed(), self.data)
>> @@ -302,7 +292,7 @@ class CookerDataBuilder(object):
>> bb.event.fire(bb.event.ConfigParsed(), self.data)
>>
>> bb.parse.init_parser(self.data)
>> - self.data_hash = self.data.get_hash()
>> + data_hash.update(self.data.get_hash().encode('utf-8'))
>> self.mcdata[''] = self.data
>>
>> multiconfig = (self.data.getVar("BBMULTICONFIG") or
>> "").split() @@ -310,9 +300,11 @@ class CookerDataBuilder(object):
>> mcdata = self.parseConfigurationFiles(self.prefiles,
>> self.postfiles, config) bb.event.fire(bb.event.ConfigParsed(), mcdata)
>> self.mcdata[config] = mcdata
>> + data_hash.update(mcdata.get_hash().encode('utf-8'))
>> if multiconfig:
>> bb.event.fire(bb.event.MultiConfigParsed(self.mcdata),
>> self.data)
>> + self.data_hash = data_hash.hexdigest()
>> except (SyntaxError, bb.BBHandledException):
>> raise bb.BBHandledException
>> except bb.data_smart.ExpansionError as e:
>> @@ -354,14 +346,24 @@ class CookerDataBuilder(object):
>> data = parse_config_file(layerconf, data)
>>
>> layers = (data.getVar('BBLAYERS') or "").split()
>> + broken_layers = []
>>
>> data = bb.data.createCopy(data)
>> approved = bb.utils.approved_variables()
>> +
>> + # Check whether present layer directories exist
>> for layer in layers:
>> if not os.path.isdir(layer):
>> - parselog.critical("Layer directory '%s' does not
>> exist! "
>> - "Please check BBLAYERS in %s"
>> % (layer, layerconf))
>> - sys.exit(1)
>> + broken_layers.append(layer)
>> +
>> + if broken_layers:
>> + parselog.critical("The following layer directories
>> do not exist:")
>> + for layer in broken_layers:
>> + parselog.critical(" %s", layer)
>> + parselog.critical("Please check BBLAYERS in %s" %
>> (layerconf))
>> + sys.exit(1)
>> +
>> + for layer in layers:
>> parselog.debug(2, "Adding layer %s", layer)
>> if 'HOME' in approved and '~' in layer:
>> layer = os.path.expanduser(layer)
>> @@ -391,7 +393,11 @@ class CookerDataBuilder(object):
>> bb.fatal("BBFILES_DYNAMIC entries must be of the
>> form <collection name>:<filename pattern>, not:\n %s" % "\n
>> ".join(invalid)) layerseries =
>> set((data.getVar("LAYERSERIES_CORENAMES") or "").split())
>> + collections_tmp = collections[:]
>> for c in collections:
>> + collections_tmp.remove(c)
>> + if c in collections_tmp:
>> + bb.fatal("Found duplicated BBFILE_COLLECTIONS
>> '%s', check bblayers.conf or layer.conf to fix it." % c) compat =
>> set((data.getVar("LAYERSERIES_COMPAT_%s" % c) or "").split()) if
>> compat and not (compat & layerseries): bb.fatal("Layer %s is not
>> compatible with the core layer which only supports these series: %s
>> (layer is compatible with %s)" diff --git
>> a/bitbake/lib/bb/daemonize.py b/bitbake/lib/bb/daemonize.py index
>> c937675..f01e6ec 100644 --- a/bitbake/lib/bb/daemonize.py +++
>> b/bitbake/lib/bb/daemonize.py @@ -1,3 +1,7 @@
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> """
>> Python Daemonizing helper
>>
>> diff --git a/bitbake/lib/bb/data.py b/bitbake/lib/bb/data.py
>> index d66d98c..0d75d0c 100644
>> --- a/bitbake/lib/bb/data.py
>> +++ b/bitbake/lib/bb/data.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake 'Data' implementations
>>
>> @@ -22,18 +20,7 @@ the speed is more critical here.
>> # Copyright (C) 2003, 2004 Chris Larson
>> # Copyright (C) 2005 Holger Hans Peter Freyther
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> # Based on functions from the base bb module, Copyright 2003 Holger
>> Schurig
>> @@ -143,7 +130,7 @@ def emit_var(var, o=sys.__stdout__, d = init(),
>> all=False): if all:
>> oval = d.getVar(var, False)
>> val = d.getVar(var)
>> - except (KeyboardInterrupt, bb.build.FuncFailed):
>> + except (KeyboardInterrupt):
>> raise
>> except Exception as exc:
>> o.write('# expansion of %s threw %s: %s\n' % (var,
>> exc.__class__.__name__, str(exc))) @@ -322,8 +309,6 @@ def
>> build_dependencies(key, keys, shelldeps, varflagsexcl, d): if
>> varflags.get("python"): value = d.getVarFlag(key, "_content", False)
>> parser = bb.codeparser.PythonParser(key, logger)
>> - if value and "\t" in value:
>> - logger.warning("Variable %s contains tabs,
>> please remove these (%s)" % (key, d.getVar("FILE")))
>> parser.parse_python(value, filename=varflags.get("filename"),
>> lineno=varflags.get("lineno")) deps = deps | parser.references deps =
>> deps | (keys & parser.execs) @@ -437,8 +422,8 @@ def
>> generate_dependency_hash(tasklist, gendeps, lookupcache, whitelist,
>> fn): var = lookupcache[dep] if var is not None:
>> data = data + str(var)
>> - k = fn + "." + task
>> - basehash[k] = hashlib.md5(data.encode("utf-8")).hexdigest()
>> + k = fn + ":" + task
>> + basehash[k] =
>> hashlib.sha256(data.encode("utf-8")).hexdigest() taskdeps[task] =
>> alldeps
>> return taskdeps, basehash
>> diff --git a/bitbake/lib/bb/data_smart.py
>> b/bitbake/lib/bb/data_smart.py index 67af380..dd5c618 100644
>> --- a/bitbake/lib/bb/data_smart.py
>> +++ b/bitbake/lib/bb/data_smart.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake Smart Dictionary Implementation
>>
>> @@ -14,18 +12,8 @@ BitBake build tools.
>> # Copyright (C) 2005 Uli Luckas
>> # Copyright (C) 2005 ROAD GmbH
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. # Based on functions from the base bb module,
>> Copyright 2003 Holger Schurig
>> import copy, re, sys, traceback
>> @@ -39,10 +27,11 @@ from bb.COW import COWDictBase
>> logger = logging.getLogger("BitBake.Data")
>>
>> __setvar_keyword__ = ["_append", "_prepend", "_remove"]
>> -__setvar_regexp__ =
>> re.compile('(?P<base>.*?)(?P<keyword>_append|_prepend|_remove)(_(?P<add>[^A-Z]*))?$')
>> -__expand_var_regexp__ = re.compile(r"\${[^{}@\n\t :]+}")
>> +__setvar_regexp__ =
>> re.compile(r'(?P<base>.*?)(?P<keyword>_append|_prepend|_remove)(_(?P<add>[^A-Z]*))?$')
>> +__expand_var_regexp__ = re.compile(r"\${[a-zA-Z0-9\-_+./~]+?}")
>> __expand_python_regexp__ = re.compile(r"\${@.+?}")
>> -__whitespace_split__ = re.compile('(\s)') +__whitespace_split__ =
>> re.compile(r'(\s)') +__override_regexp__ = re.compile(r'[a-z0-9]+')
>> def infer_caller_details(loginfo, parent = False, varval = True):
>> """Save the caller the trouble of specifying everything."""
>> @@ -597,7 +586,7 @@ class DataSmart(MutableMapping):
>> # aka pay the cookie monster
>> override = var[var.rfind('_')+1:]
>> shortvar = var[:var.rfind('_')]
>> - while override and override.islower():
>> + while override and __override_regexp__.match(override):
>> if shortvar not in self.overridedata:
>> self.overridedata[shortvar] = []
>> if [var, override] not in self.overridedata[shortvar]:
>> @@ -1073,4 +1062,4 @@ class DataSmart(MutableMapping):
>> data.update({i:value})
>>
>> data_str = str([(k, data[k]) for k in sorted(data.keys())])
>> - return hashlib.md5(data_str.encode("utf-8")).hexdigest()
>> + return hashlib.sha256(data_str.encode("utf-8")).hexdigest()
>> diff --git a/bitbake/lib/bb/event.py b/bitbake/lib/bb/event.py
>> index 5b1b094..d44621e 100644
>> --- a/bitbake/lib/bb/event.py
>> +++ b/bitbake/lib/bb/event.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake 'Event' implementation
>>
>> @@ -9,18 +7,8 @@ BitBake build tools.
>>
>> # Copyright (C) 2003, 2004 Chris Larson
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import os, sys
>> import warnings
>> @@ -136,6 +124,7 @@ def fire_class_handlers(event, d):
>> ui_queue = []
>> @atexit.register
>> def print_ui_queue():
>> + global ui_queue
>> """If we're exiting before a UI has been spawned, display any
>> queued LogRecords to the console."""
>> logger = logging.getLogger("BitBake")
>> @@ -180,6 +169,7 @@ def print_ui_queue():
>> logger.removeHandler(stderr)
>> else:
>> logger.removeHandler(stdout)
>> + ui_queue = []
>>
>> def fire_ui_handlers(event, d):
>> global _thread_lock
>> @@ -414,23 +404,6 @@ class RecipeTaskPreProcess(RecipeEvent):
>> class RecipeParsed(RecipeEvent):
>> """ Recipe Parsing Complete """
>>
>> -class StampUpdate(Event):
>> - """Trigger for any adjustment of the stamp files to happen"""
>> -
>> - def __init__(self, targets, stampfns):
>> - self._targets = targets
>> - self._stampfns = stampfns
>> - Event.__init__(self)
>> -
>> - def getStampPrefix(self):
>> - return self._stampfns
>> -
>> - def getTargets(self):
>> - return self._targets
>> -
>> - stampPrefix = property(getStampPrefix)
>> - targets = property(getTargets)
>> -
>> class BuildBase(Event):
>> """Base class for bitbake build events"""
>>
>> diff --git a/bitbake/lib/bb/exceptions.py
>> b/bitbake/lib/bb/exceptions.py index cd71343..ecbad59 100644
>> --- a/bitbake/lib/bb/exceptions.py
>> +++ b/bitbake/lib/bb/exceptions.py
>> @@ -1,3 +1,6 @@
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>>
>> import inspect
>> import traceback
>> diff --git a/bitbake/lib/bb/fetch2/__init__.py
>> b/bitbake/lib/bb/fetch2/__init__.py index 572b71a..1f5f8f1 100644
>> --- a/bitbake/lib/bb/fetch2/__init__.py
>> +++ b/bitbake/lib/bb/fetch2/__init__.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake 'Fetch' implementations
>>
>> @@ -10,18 +8,7 @@ BitBake build tools.
>> # Copyright (C) 2003, 2004 Chris Larson
>> # Copyright (C) 2012 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> # Based on functions from the base bb module, Copyright 2003 Holger
>> Schurig
>> @@ -256,7 +243,7 @@ class URI(object):
>>
>> # Identify if the URI is relative or not
>> if urlp.scheme in self._relative_schemes and \
>> - re.compile("^\w+:(?!//)").match(uri):
>> + re.compile(r"^\w+:(?!//)").match(uri):
>> self.relative = True
>>
>> if not self.relative:
>> @@ -524,7 +511,7 @@ def fetcher_parse_save():
>> def fetcher_parse_done():
>> _checksum_cache.save_merge()
>>
>> -def fetcher_compare_revisions():
>> +def fetcher_compare_revisions(d):
>> """
>> Compare the revisions in the persistant cache with current
>> values and return true/false on whether they've changed.
>> @@ -777,7 +764,8 @@ def get_srcrev(d,
>> method_name='sortable_revision'): #
>> format = d.getVar('SRCREV_FORMAT')
>> if not format:
>> - raise FetchError("The SRCREV_FORMAT variable must be set
>> when multiple SCMs are used.")
>> + raise FetchError("The SRCREV_FORMAT variable must be set
>> when multiple SCMs are used.\n"\
>> + "The SCMs are:\n%s" % '\n'.join(scms))
>>
>> name_to_rev = {}
>> seenautoinc = False
>> @@ -855,10 +843,18 @@ def runfetchcmd(cmd, d, quiet=False,
>> cleanup=None, log=None, workdir=None): if val:
>> cmd = 'export ' + var + '=\"%s\"; %s' % (val, cmd)
>>
>> + # Ensure that a _PYTHON_SYSCONFIGDATA_NAME value set by a recipe
>> + # (for example via python3native.bbclass since warrior) is not
>> set for
>> + # host Python (otherwise tools like git-make-shallow will fail)
>> + cmd = 'unset _PYTHON_SYSCONFIGDATA_NAME; ' + cmd
>> +
>> # Disable pseudo as it may affect ssh, potentially causing it to
>> hang. cmd = 'export PSEUDO_DISABLED=1; ' + cmd
>>
>> - logger.debug(1, "Running %s", cmd)
>> + if workdir:
>> + logger.debug(1, "Running '%s' in %s" % (cmd, workdir))
>> + else:
>> + logger.debug(1, "Running %s", cmd)
>>
>> success = False
>> error_message = ""
>> @@ -894,7 +890,7 @@ def check_network_access(d, info, url):
>> log remote network access, and error if BB_NO_NETWORK is set or
>> the given URI is untrusted
>> """
>> - if d.getVar("BB_NO_NETWORK") == "1":
>> + if bb.utils.to_boolean(d.getVar("BB_NO_NETWORK")):
>> raise NetworkAccess(url, info)
>> elif not trusted_network(d, url):
>> raise UntrustedUrl(url, info)
>> @@ -966,7 +962,8 @@ def rename_bad_checksum(ud, suffix):
>>
>> new_localpath = "%s_bad-checksum_%s" % (ud.localpath, suffix)
>> bb.warn("Renaming %s to %s" % (ud.localpath, new_localpath))
>> - bb.utils.movefile(ud.localpath, new_localpath)
>> + if not bb.utils.movefile(ud.localpath, new_localpath):
>> + bb.warn("Renaming %s to %s failed, grep movefile in
>> log.do_fetch to see why" % (ud.localpath, new_localpath))
>>
>> def try_mirror_url(fetch, origud, ud, ld, check = False):
>> @@ -1027,7 +1024,7 @@ def try_mirror_url(fetch, origud, ud, ld, check
>> = False): raise
>>
>> except IOError as e:
>> - if e.errno in [os.errno.ESTALE]:
>> + if e.errno in [errno.ESTALE]:
>> logger.warning("Stale Error Observed %s." % ud.url)
>> return False
>> raise
>> @@ -1094,7 +1091,7 @@ def trusted_network(d, url):
>> BB_ALLOWED_NETWORKS is set globally or for a specific recipe.
>> Note: modifies SRC_URI & mirrors.
>> """
>> - if d.getVar('BB_NO_NETWORK') == "1":
>> + if bb.utils.to_boolean(d.getVar("BB_NO_NETWORK")):
>> return True
>>
>> pkgname = d.expand(d.getVar('PN', False))
>> @@ -1403,7 +1400,7 @@ class FetchMethod(object):
>> Fetch urls
>> Assumes localpath was called first
>> """
>> - raise NoMethodError(url)
>> + raise NoMethodError(urldata.url)
>>
>> def unpack(self, urldata, rootdir, data):
>> iterate = False
>> @@ -1469,7 +1466,7 @@ class FetchMethod(object):
>> else:
>> cmd = 'rpm2cpio.sh %s | cpio -id' % (file)
>> elif file.endswith('.deb') or file.endswith('.ipk'):
>> - output = subprocess.check_output('ar -t %s' % file,
>> preexec_fn=subprocess_setup, shell=True)
>> + output = subprocess.check_output(['ar', '-t', file],
>> preexec_fn=subprocess_setup) datafile = None
>> if output:
>> for line in output.decode().splitlines():
>> @@ -1547,7 +1544,7 @@ class FetchMethod(object):
>> Check the status of a URL
>> Assumes localpath was called first
>> """
>> - logger.info("URL %s could not be checked for status since no
>> method exists.", url)
>> + logger.info("URL %s could not be checked for status since no
>> method exists.", urldata.url) return True
>>
>> def latest_revision(self, ud, d, name):
>> @@ -1555,7 +1552,7 @@ class FetchMethod(object):
>> Look in the cache for the latest revision, if not present
>> ask the SCM. """
>> if not hasattr(self, "_latest_revision"):
>> - raise ParameterError("The fetcher for this URL does not
>> support _latest_revision", url)
>> + raise ParameterError("The fetcher for this URL does not
>> support _latest_revision", ud.url)
>> revs = bb.persist_data.persist('BB_URI_HEADREVS', d)
>> key = self.generate_revision_key(ud, d, name)
>> @@ -1638,7 +1635,7 @@ class Fetch(object):
>> urls = self.urls
>>
>> network = self.d.getVar("BB_NO_NETWORK")
>> - premirroronly = (self.d.getVar("BB_FETCH_PREMIRRORONLY") ==
>> "1")
>> + premirroronly =
>> bb.utils.to_boolean(self.d.getVar("BB_FETCH_PREMIRRORONLY"))
>> for u in urls:
>> ud = self.ud[u]
>> @@ -1716,7 +1713,7 @@ class Fetch(object):
>> update_stamp(ud, self.d)
>>
>> except IOError as e:
>> - if e.errno in [os.errno.ESTALE]:
>> + if e.errno in [errno.ESTALE]:
>> logger.error("Stale Error Observed %s." % u)
>> raise ChecksumError("Stale Error Detected")
>>
>> @@ -1786,7 +1783,7 @@ class Fetch(object):
>>
>> for url in urls:
>> if url not in self.ud:
>> - self.ud[url] = FetchData(url, d)
>> + self.ud[url] = FetchData(url, self.d)
>> ud = self.ud[url]
>> ud.setup_localpath(self.d)
>>
>> diff --git a/bitbake/lib/bb/fetch2/bzr.py
>> b/bitbake/lib/bb/fetch2/bzr.py index 658502f..c56d875 100644
>> --- a/bitbake/lib/bb/fetch2/bzr.py
>> +++ b/bitbake/lib/bb/fetch2/bzr.py
>> @@ -10,18 +10,8 @@ BitBake 'Fetch' implementation for bzr.
>> # BitBake build tools.
>> # Copyright (C) 2003, 2004 Chris Larson
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import os
>> import sys
>> diff --git a/bitbake/lib/bb/fetch2/clearcase.py
>> b/bitbake/lib/bb/fetch2/clearcase.py index 3a6573d..3dd93ad 100644
>> --- a/bitbake/lib/bb/fetch2/clearcase.py
>> +++ b/bitbake/lib/bb/fetch2/clearcase.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake 'Fetch' clearcase implementation
>>
>> @@ -47,18 +45,7 @@ User credentials:
>> """
>> # Copyright (C) 2014 Siemens AG
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>>
>> import os
>> @@ -67,6 +54,8 @@ import shutil
>> import bb
>> from bb.fetch2 import FetchMethod
>> from bb.fetch2 import FetchError
>> +from bb.fetch2 import MissingParameterError
>> +from bb.fetch2 import ParameterError
>> from bb.fetch2 import runfetchcmd
>> from bb.fetch2 import logger
>>
>> @@ -92,7 +81,7 @@ class ClearCase(FetchMethod):
>> if 'protocol' in ud.parm:
>> ud.proto = ud.parm['protocol']
>> if not ud.proto in ('http', 'https'):
>> - raise fetch2.ParameterError("Invalid protocol type",
>> ud.url)
>> + raise ParameterError("Invalid protocol type", ud.url)
>>
>> ud.vob = ''
>> if 'vob' in ud.parm:
>> diff --git a/bitbake/lib/bb/fetch2/cvs.py
>> b/bitbake/lib/bb/fetch2/cvs.py index 0e0a319..1b35ba4 100644
>> --- a/bitbake/lib/bb/fetch2/cvs.py
>> +++ b/bitbake/lib/bb/fetch2/cvs.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake 'Fetch' implementations
>>
>> @@ -10,20 +8,9 @@ BitBake build tools.
>>
>> # Copyright (C) 2003, 2004 Chris Larson
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. -#
>> -#Based on functions from the base bb module, Copyright 2003 Holger
>> Schurig +# Based on functions from the base bb module, Copyright 2003
>> Holger Schurig #
>>
>> import os
>> diff --git a/bitbake/lib/bb/fetch2/git.py
>> b/bitbake/lib/bb/fetch2/git.py index 59a2ee8..2d1d2ca 100644
>> --- a/bitbake/lib/bb/fetch2/git.py
>> +++ b/bitbake/lib/bb/fetch2/git.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake 'Fetch' git implementation
>>
>> @@ -55,20 +53,10 @@ Supported SRC_URI options are:
>>
>> """
>>
>> -#Copyright (C) 2005 Richard Purdie
>> +# Copyright (C) 2005 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import collections
>> import errno
>> @@ -199,7 +187,7 @@ class Git(FetchMethod):
>> depth_default = 1
>> ud.shallow_depths = collections.defaultdict(lambda:
>> depth_default)
>> - revs_default = d.getVar("BB_GIT_SHALLOW_REVS", True)
>> + revs_default = d.getVar("BB_GIT_SHALLOW_REVS")
>> ud.shallow_revs = []
>> ud.branches = {}
>> for pos, name in enumerate(ud.names):
>> @@ -318,7 +306,7 @@ class Git(FetchMethod):
>> def try_premirror(self, ud, d):
>> # If we don't do this, updating an existing checkout with
>> only premirrors # is not possible
>> - if d.getVar("BB_FETCH_PREMIRRORONLY") is not None:
>> + if bb.utils.to_boolean(d.getVar("BB_FETCH_PREMIRRORONLY")):
>> return True
>> if os.path.exists(ud.clonedir):
>> return False
>> @@ -476,6 +464,8 @@ class Git(FetchMethod):
>> if os.path.exists(destdir):
>> bb.utils.prunedir(destdir)
>>
>> + need_lfs = ud.parm.get("lfs", "1") == "1"
>> +
>> source_found = False
>> source_error = []
>>
>> @@ -503,6 +493,13 @@ class Git(FetchMethod):
>>
>> repourl = self._get_repo_url(ud)
>> runfetchcmd("%s remote set-url origin %s" % (ud.basecmd,
>> repourl), d, workdir=destdir) +
>> + if self._contains_lfs(ud, d, destdir):
>> + if need_lfs and not self._find_git_lfs(d):
>> + raise bb.fetch2.FetchError("Repository %s has LFS
>> content, install git-lfs on host to download (or set lfs=0 to ignore
>> it)" % (repourl))
>> + else:
>> + bb.note("Repository %s has LFS content but it is not
>> being fetched" % (repourl)) +
>> if not ud.nocheckout:
>> if subdir != "":
>> runfetchcmd("%s read-tree %s%s" % (ud.basecmd,
>> ud.revisions[ud.names[0]], readpathspec), d, @@ -522,9 +519,17 @@
>> class Git(FetchMethod): def clean(self, ud, d):
>> """ clean the git directory """
>>
>> - bb.utils.remove(ud.localpath, True)
>> - bb.utils.remove(ud.fullmirror)
>> - bb.utils.remove(ud.fullmirror + ".done")
>> + to_remove = [ud.localpath, ud.fullmirror, ud.fullmirror +
>> ".done"]
>> + # The localpath is a symlink to clonedir when it is cloned
>> from a
>> + # mirror, so remove both of them.
>> + if os.path.islink(ud.localpath):
>> + clonedir = os.path.realpath(ud.localpath)
>> + to_remove.append(clonedir)
>> +
>> + for r in to_remove:
>> + if os.path.exists(r):
>> + bb.note('Removing %s' % r)
>> + bb.utils.remove(r, True)
>>
>> def supports_srcrev(self):
>> return True
>> @@ -545,6 +550,27 @@ class Git(FetchMethod):
>> raise bb.fetch2.FetchError("The command '%s' gave output
>> with more then 1 line unexpectedly, output: '%s'" % (cmd, output))
>> return output.split()[0] != "0"
>> + def _contains_lfs(self, ud, d, wd):
>> + """
>> + Check if the repository has 'lfs' (large file) content
>> + """
>> + cmd = "%s grep lfs HEAD:.gitattributes | wc -l" % (
>> + ud.basecmd)
>> + try:
>> + output = runfetchcmd(cmd, d, quiet=True, workdir=wd)
>> + if int(output) > 0:
>> + return True
>> + except (bb.fetch2.FetchError,ValueError):
>> + pass
>> + return False
>> +
>> + def _find_git_lfs(self, d):
>> + """
>> + Return True if git-lfs can be found, False otherwise.
>> + """
>> + import shutil
>> + return shutil.which("git-lfs", path=d.getVar('PATH')) is not
>> None +
>> def _get_repo_url(self, ud):
>> """
>> Return the repository URL
>> @@ -615,7 +641,7 @@ class Git(FetchMethod):
>> """
>> pupver = ('', '')
>>
>> - tagregex = re.compile(d.getVar('UPSTREAM_CHECK_GITTAGREGEX')
>> or "(?P<pver>([0-9][\.|_]?)+)")
>> + tagregex = re.compile(d.getVar('UPSTREAM_CHECK_GITTAGREGEX')
>> or r"(?P<pver>([0-9][\.|_]?)+)") try:
>> output = self._lsremote(ud, d, "refs/tags/*")
>> except (bb.fetch2.FetchError, bb.fetch2.NetworkAccess) as e:
>> @@ -630,7 +656,7 @@ class Git(FetchMethod):
>>
>> tag_head = line.split("/")[-1]
>> # Ignore non-released branches
>> - m = re.search("(alpha|beta|rc|final)+", tag_head)
>> + m = re.search(r"(alpha|beta|rc|final)+", tag_head)
>> if m:
>> continue
>>
>> diff --git a/bitbake/lib/bb/fetch2/gitannex.py
>> b/bitbake/lib/bb/fetch2/gitannex.py index a9b69ca..1d497dc 100644
>> --- a/bitbake/lib/bb/fetch2/gitannex.py
>> +++ b/bitbake/lib/bb/fetch2/gitannex.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake 'Fetch' git annex implementation
>> """
>> @@ -7,18 +5,8 @@ BitBake 'Fetch' git annex implementation
>> # Copyright (C) 2014 Otavio Salvador
>> # Copyright (C) 2014 O.S. Systems Software LTDA.
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import os
>> import bb
>> diff --git a/bitbake/lib/bb/fetch2/gitsm.py
>> b/bitbake/lib/bb/fetch2/gitsm.py index 35729db..c622771 100644
>> --- a/bitbake/lib/bb/fetch2/gitsm.py
>> +++ b/bitbake/lib/bb/fetch2/gitsm.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake 'Fetch' git submodules implementation
>>
>> @@ -16,18 +14,8 @@ NOTE: Switching a SRC_URI from "git://" to
>> "gitsm://" requires a clean of your r
>> # Copyright (C) 2013 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import os
>> import bb
>> @@ -45,60 +33,97 @@ class GitSM(Git):
>> """
>> return ud.type in ['gitsm']
>>
>> - @staticmethod
>> - def parse_gitmodules(gitmodules):
>> - modules = {}
>> - module = ""
>> - for line in gitmodules.splitlines():
>> - if line.startswith('[submodule'):
>> - module = line.split('"')[1]
>> - modules[module] = {}
>> - elif module and line.strip().startswith('path'):
>> - path = line.split('=')[1].strip()
>> - modules[module]['path'] = path
>> - elif module and line.strip().startswith('url'):
>> - url = line.split('=')[1].strip()
>> - modules[module]['url'] = url
>> - return modules
>> -
>> - def update_submodules(self, ud, d):
>> + def process_submodules(self, ud, workdir, function, d):
>> + """
>> + Iterate over all of the submodules in this repository and
>> execute
>> + the 'function' for each of them.
>> + """
>> +
>> submodules = []
>> paths = {}
>> + revision = {}
>> uris = {}
>> - local_paths = {}
>> -
>> + subrevision = {}
>> +
>> + def parse_gitmodules(gitmodules):
>> + modules = {}
>> + module = ""
>> + for line in gitmodules.splitlines():
>> + if line.startswith('[submodule'):
>> + module = line.split('"')[1]
>> + modules[module] = {}
>> + elif module and line.strip().startswith('path'):
>> + path = line.split('=')[1].strip()
>> + modules[module]['path'] = path
>> + elif module and line.strip().startswith('url'):
>> + url = line.split('=')[1].strip()
>> + modules[module]['url'] = url
>> + return modules
>> +
>> + # Collect the defined submodules, and their attributes
>> for name in ud.names:
>> try:
>> - gitmodules = runfetchcmd("%s show %s:.gitmodules" %
>> (ud.basecmd, ud.revisions[name]), d, quiet=True, workdir=ud.clonedir)
>> + gitmodules = runfetchcmd("%s show %s:.gitmodules" %
>> (ud.basecmd, ud.revisions[name]), d, quiet=True, workdir=workdir)
>> except: # No submodules to update
>> continue
>>
>> - for m, md in self.parse_gitmodules(gitmodules).items():
>> + for m, md in parse_gitmodules(gitmodules).items():
>> + try:
>> + module_hash = runfetchcmd("%s ls-tree -z -d %s
>> %s" % (ud.basecmd, ud.revisions[name], md['path']), d, quiet=True,
>> workdir=workdir)
>> + except:
>> + # If the command fails, we don't have a valid
>> file to check. If it doesn't
>> + # fail -- it still might be a failure, see next
>> check...
>> + module_hash = ""
>> +
>> + if not module_hash:
>> + logger.debug(1, "submodule %s is defined, but is
>> not initialized in the repository. Skipping", m)
>> + continue
>> +
>> submodules.append(m)
>> paths[m] = md['path']
>> + revision[m] = ud.revisions[name]
>> uris[m] = md['url']
>> + subrevision[m] = module_hash.split()[2]
>> +
>> + # Convert relative to absolute uri based on parent
>> uri if uris[m].startswith('..'):
>> newud = copy.copy(ud)
>> - newud.path =
>> os.path.realpath(os.path.join(newud.path, md['url']))
>> + newud.path =
>> os.path.realpath(os.path.join(newud.path, uris[m])) uris[m] =
>> Git._get_repo_url(self, newud)
>> for module in submodules:
>> - module_hash = runfetchcmd("%s ls-tree -z -d %s %s" %
>> (ud.basecmd, ud.revisions[name], paths[module]), d, quiet=True,
>> workdir=ud.clonedir)
>> - module_hash = module_hash.split()[2]
>> + # Translate the module url into a SRC_URI
>> +
>> + if "://" in uris[module]:
>> + # Properly formated URL already
>> + proto = uris[module].split(':', 1)[0]
>> + url = uris[module].replace('%s:' % proto, 'gitsm:',
>> 1)
>> + else:
>> + if ":" in uris[module]:
>> + # Most likely an SSH style reference
>> + proto = "ssh"
>> + if ":/" in uris[module]:
>> + # Absolute reference, easy to convert..
>> + url = "gitsm://" +
>> uris[module].replace(':/', '/', 1)
>> + else:
>> + # Relative reference, no way to know if this
>> is right!
>> + logger.warning("Submodule included by %s
>> refers to relative ssh reference %s. References may fail if not
>> absolute." % (ud.url, uris[module]))
>> + url = "gitsm://" + uris[module].replace(':',
>> '/', 1)
>> + else:
>> + # This has to be a file reference
>> + proto = "file"
>> + url = "gitsm://" + uris[module]
>>
>> - # Build new SRC_URI
>> - proto = uris[module].split(':', 1)[0]
>> - url = uris[module].replace('%s:' % proto, 'gitsm:', 1)
>> url += ';protocol=%s' % proto
>> url += ";name=%s" % module
>> - url += ";bareclone=1;nocheckout=1;nobranch=1"
>> + url += ";subpath=%s" % module
>>
>> ld = d.createCopy()
>> # Not necessary to set SRC_URI, since we're passing the
>> URI to # Fetch.
>> #ld.setVar('SRC_URI', url)
>> - ld.setVar('SRCREV_%s' % module, module_hash)
>> + ld.setVar('SRCREV_%s' % module, subrevision[module])
>>
>> # Workaround for issues with SRCPV/SRCREV_FORMAT errors
>> # error refer to 'multiple' repositories. Only the
>> repository @@ -106,145 +131,85 @@ class GitSM(Git):
>> ld.setVar('SRCPV', d.getVar('SRCPV'))
>> ld.setVar('SRCREV_FORMAT', module)
>>
>> - newfetch = Fetch([url], ld, cache=False)
>> - newfetch.download()
>> - local_paths[module] = newfetch.localpath(url)
>> + function(ud, url, module, paths[module], ld)
>>
>> - # Correct the submodule references to the local download
>> version...
>> - runfetchcmd("%(basecmd)s config submodule.%(module)s.url
>> %(url)s" % {'basecmd': ud.basecmd, 'module': module, 'url' :
>> local_paths[module]}, d, workdir=ud.clonedir) -
>> - symlink_path = os.path.join(ud.clonedir, 'modules',
>> paths[module])
>> - if not os.path.exists(symlink_path):
>> - try:
>> - os.makedirs(os.path.dirname(symlink_path),
>> exist_ok=True)
>> - except OSError:
>> - pass
>> - os.symlink(local_paths[module], symlink_path)
>> -
>> - return True
>> + return submodules != []
>>
>> def need_update(self, ud, d):
>> - main_repo_needs_update = Git.need_update(self, ud, d)
>> -
>> - # First check that the main repository has enough history
>> fetched. If it doesn't, then we don't
>> - # even have the .gitmodules and gitlinks for the submodules
>> to attempt asking whether the
>> - # submodules' histories are recent enough.
>> - if main_repo_needs_update:
>> + if Git.need_update(self, ud, d):
>> return True
>>
>> - # Now check that the submodule histories are new enough. The
>> git-submodule command doesn't have
>> - # any clean interface for doing this aside from just
>> attempting the checkout (with network
>> - # fetched disabled).
>> - return not self.update_submodules(ud, d)
>> + try:
>> + # Check for the nugget dropped by the download operation
>> + known_srcrevs = runfetchcmd("%s config --get-all
>> bitbake.srcrev" % \
>> + (ud.basecmd), d, workdir=ud.clonedir)
>>
>> - def download(self, ud, d):
>> - Git.download(self, ud, d)
>> + if ud.revisions[ud.names[0]] not in
>> known_srcrevs.split():
>> + return True
>> + except bb.fetch2.FetchError:
>> + # No srcrev nuggets, so this is new and needs to be
>> updated
>> + return True
>>
>> - if not ud.shallow or ud.localpath != ud.fullshallow:
>> - self.update_submodules(ud, d)
>> + return False
>>
>> - def copy_submodules(self, submodules, ud, destdir, d):
>> - if ud.bareclone:
>> - repo_conf = destdir
>> - else:
>> - repo_conf = os.path.join(destdir, '.git')
>> + def download(self, ud, d):
>> + def download_submodule(ud, url, module, modpath, d):
>> + url += ";bareclone=1;nobranch=1"
>>
>> - if submodules and not os.path.exists(os.path.join(repo_conf,
>> 'modules')):
>> - os.mkdir(os.path.join(repo_conf, 'modules'))
>> + # Is the following still needed?
>> + #url += ";nocheckout=1"
>>
>> - for module, md in submodules.items():
>> - srcpath = os.path.join(ud.clonedir, 'modules',
>> md['path'])
>> - modpath = os.path.join(repo_conf, 'modules', md['path'])
>> + try:
>> + newfetch = Fetch([url], d, cache=False)
>> + newfetch.download()
>> + # Drop a nugget to add each of the srcrevs we've
>> fetched (used by need_update)
>> + runfetchcmd("%s config --add bitbake.srcrev %s" % \
>> + (ud.basecmd, ud.revisions[ud.names[0]]),
>> d, workdir=ud.clonedir)
>> + except Exception as e:
>> + logger.error('gitsm: submodule download failed: %s
>> %s' % (type(e).__name__, str(e)))
>> + raise
>>
>> - if os.path.exists(srcpath):
>> - if os.path.exists(os.path.join(srcpath, '.git')):
>> - srcpath = os.path.join(srcpath, '.git')
>> + Git.download(self, ud, d)
>> + self.process_submodules(ud, ud.clonedir, download_submodule,
>> d)
>> - target = modpath
>> - if os.path.exists(modpath):
>> - target = os.path.dirname(modpath)
>> + def unpack(self, ud, destdir, d):
>> + def unpack_submodules(ud, url, module, modpath, d):
>> + url += ";bareclone=1;nobranch=1"
>>
>> - os.makedirs(os.path.dirname(target), exist_ok=True)
>> - runfetchcmd("cp -fpLR %s %s" % (srcpath, target), d)
>> - elif os.path.exists(modpath):
>> - # Module already exists, likely unpacked from a
>> shallow mirror clone
>> - pass
>> + # Figure out where we clone over the bare submodules...
>> + if ud.bareclone:
>> + repo_conf = ud.destdir
>> else:
>> - # This is fatal, as we do NOT want git-submodule to
>> hit the network
>> - raise bb.fetch2.FetchError('Submodule %s does not
>> exist in %s or %s.' % (module, srcpath, modpath)) -
>> - def clone_shallow_local(self, ud, dest, d):
>> - super(GitSM, self).clone_shallow_local(ud, dest, d)
>> + repo_conf = os.path.join(ud.destdir, '.git')
>>
>> - # Copy over the submodules' fetched histories too.
>> - repo_conf = os.path.join(dest, '.git')
>> -
>> - submodules = []
>> - for name in ud.names:
>> try:
>> - gitmodules = runfetchcmd("%s show %s:.gitmodules" %
>> (ud.basecmd, ud.revision), d, quiet=True, workdir=dest)
>> - except:
>> - # No submodules to update
>> - continue
>> + newfetch = Fetch([url], d, cache=False)
>> +
>> newfetch.unpack(root=os.path.dirname(os.path.join(repo_conf,
>> 'modules', module)))
>> + except Exception as e:
>> + logger.error('gitsm: submodule unpack failed: %s %s'
>> % (type(e).__name__, str(e)))
>> + raise
>>
>> - submodules = self.parse_gitmodules(gitmodules)
>> - self.copy_submodules(submodules, ud, dest, d)
>> + local_path = newfetch.localpath(url)
>>
>> - def unpack(self, ud, destdir, d):
>> - Git.unpack(self, ud, destdir, d)
>> + # Correct the submodule references to the local download
>> version...
>> + runfetchcmd("%(basecmd)s config submodule.%(module)s.url
>> %(url)s" % {'basecmd': ud.basecmd, 'module': module, 'url' :
>> local_path}, d, workdir=ud.destdir)
>> - # Copy over the submodules' fetched histories too.
>> - if ud.bareclone:
>> - repo_conf = ud.destdir
>> - else:
>> - repo_conf = os.path.join(ud.destdir, '.git')
>> + if ud.shallow:
>> + runfetchcmd("%(basecmd)s config
>> submodule.%(module)s.shallow true" % {'basecmd': ud.basecmd,
>> 'module': module}, d, workdir=ud.destdir)
>> - update_submodules = False
>> - paths = {}
>> - uris = {}
>> - local_paths = {}
>> - for name in ud.names:
>> + # Ensure the submodule repository is NOT set to bare,
>> since we're checking it out... try:
>> - gitmodules = runfetchcmd("%s show HEAD:.gitmodules"
>> % (ud.basecmd), d, quiet=True, workdir=ud.destdir)
>> + runfetchcmd("%s config core.bare false" %
>> (ud.basecmd), d, quiet=True, workdir=os.path.join(repo_conf,
>> 'modules', module)) except:
>> - # No submodules to update
>> - continue
>> -
>> - submodules = self.parse_gitmodules(gitmodules)
>> - self.copy_submodules(submodules, ud, ud.destdir, d)
>> -
>> - submodules_queue = [(module, os.path.join(repo_conf,
>> 'modules', md['path'])) for module, md in submodules.items()]
>> - while len(submodules_queue) != 0:
>> - module, modpath = submodules_queue.pop()
>> -
>> - # add submodule children recursively
>> - try:
>> - gitmodules = runfetchcmd("%s show
>> HEAD:.gitmodules" % (ud.basecmd), d, quiet=True, workdir=modpath)
>> - for m, md in
>> self.parse_gitmodules(gitmodules).items():
>> - submodules_queue.append([m,
>> os.path.join(modpath, 'modules', md['path'])])
>> - except:
>> - # no children
>> - pass
>> -
>> + logger.error("Unable to set git config core.bare to
>> false for %s" % os.path.join(repo_conf, 'modules', module))
>> + raise
>>
>> - # There are submodules to update
>> - update_submodules = True
>> -
>> - # Determine (from the submodule) the correct url to
>> reference
>> - try:
>> - output = runfetchcmd("%(basecmd)s config
>> remote.origin.url" % {'basecmd': ud.basecmd}, d, workdir=modpath)
>> - except bb.fetch2.FetchError as e:
>> - # No remote url defined in this submodule
>> - continue
>> -
>> - local_paths[module] = output
>> -
>> - # Setup the local URL properly (like git submodule
>> init or sync would do...)
>> - runfetchcmd("%(basecmd)s config
>> submodule.%(module)s.url %(url)s" % {'basecmd': ud.basecmd, 'module':
>> module, 'url' : local_paths[module]}, d, workdir=ud.destdir)
>> + Git.unpack(self, ud, destdir, d)
>>
>> - # Ensure the submodule repository is NOT set to
>> bare, since we're checking it out...
>> - runfetchcmd("%s config core.bare false" %
>> (ud.basecmd), d, quiet=True, workdir=modpath)
>> + ret = self.process_submodules(ud, ud.destdir,
>> unpack_submodules, d)
>> - if update_submodules:
>> - # Run submodule update, this sets up the directories --
>> without touching the config
>> + if not ud.bareclone and ret:
>> + # All submodules should already be downloaded and
>> configured in the tree. This simply sets
>> + # up the configuration and checks out the files. The
>> main project config should remain
>> + # unmodified, and no download from the internet should
>> occur. runfetchcmd("%s submodule update --recursive --no-fetch" %
>> (ud.basecmd), d, quiet=True, workdir=ud.destdir) diff --git
>> a/bitbake/lib/bb/fetch2/hg.py b/bitbake/lib/bb/fetch2/hg.py index
>> 936d043..15d729e 100644 --- a/bitbake/lib/bb/fetch2/hg.py
>> +++ b/bitbake/lib/bb/fetch2/hg.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake 'Fetch' implementation for mercurial DRCS (hg).
>>
>> @@ -9,20 +7,10 @@ BitBake 'Fetch' implementation for mercurial DRCS
>> (hg). # Copyright (C) 2004 Marcin Juszkiewicz
>> # Copyright (C) 2007 Robert Schuster
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> # Based on functions from the base bb module, Copyright 2003 Holger
>> Schurig +#
>>
>> import os
>> import sys
>> @@ -99,7 +87,7 @@ class Hg(FetchMethod):
>> def try_premirror(self, ud, d):
>> # If we don't do this, updating an existing checkout with
>> only premirrors # is not possible
>> - if d.getVar("BB_FETCH_PREMIRRORONLY") is not None:
>> + if bb.utils.to_boolean(d.getVar("BB_FETCH_PREMIRRORONLY")):
>> return True
>> if os.path.exists(ud.moddir):
>> return False
>> diff --git a/bitbake/lib/bb/fetch2/local.py
>> b/bitbake/lib/bb/fetch2/local.py index a114ac1..01d9ff9 100644
>> --- a/bitbake/lib/bb/fetch2/local.py
>> +++ b/bitbake/lib/bb/fetch2/local.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake 'Fetch' implementations
>>
>> @@ -10,20 +8,10 @@ BitBake build tools.
>>
>> # Copyright (C) 2003, 2004 Chris Larson
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> # Based on functions from the base bb module, Copyright 2003 Holger
>> Schurig +#
>>
>> import os
>> import urllib.request, urllib.parse, urllib.error
>> diff --git a/bitbake/lib/bb/fetch2/npm.py
>> b/bitbake/lib/bb/fetch2/npm.py index 65bf5a3..9700e61 100644
>> --- a/bitbake/lib/bb/fetch2/npm.py
>> +++ b/bitbake/lib/bb/fetch2/npm.py
>> @@ -1,5 +1,6 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> """
>> BitBake 'Fetch' NPM implementation
>>
>> @@ -100,11 +101,19 @@ class Npm(FetchMethod):
>> return False
>> return True
>>
>> - def _runwget(self, ud, d, command, quiet):
>> - logger.debug(2, "Fetching %s using command '%s'" % (ud.url,
>> command))
>> - bb.fetch2.check_network_access(d, command, ud.url)
>> + def _runpack(self, ud, d, pkgfullname: str, quiet=False) -> str:
>> + """
>> + Runs npm pack on a full package name.
>> + Returns the filename of the downloaded package
>> + """
>> + bb.fetch2.check_network_access(d, pkgfullname, ud.registry)
>> dldir = d.getVar("DL_DIR")
>> - runfetchcmd(command, d, quiet, workdir=dldir)
>> + dldir = os.path.join(dldir, ud.prefixdir)
>> +
>> + command = "npm pack {} --registry {}".format(pkgfullname,
>> ud.registry)
>> + logger.debug(2, "Fetching {} using command '{}' in
>> {}".format(pkgfullname, command, dldir))
>> + filename = runfetchcmd(command, d, quiet, workdir=dldir)
>> + return filename.rstrip()
>>
>> def _unpackdep(self, ud, pkg, data, destdir, dldir, d):
>> file = data[pkg]['tgz']
>> @@ -150,20 +159,11 @@ class Npm(FetchMethod):
>> Parse the output of npm view --json; the last JSON result
>> is assumed to be the one that we're interested in.
>> '''
>> - pdata = None
>> - outdeps = {}
>> - datalines = []
>> - bracelevel = 0
>> - for line in output.splitlines():
>> - if bracelevel:
>> - datalines.append(line)
>> - elif '{' in line:
>> - datalines = []
>> - datalines.append(line)
>> - bracelevel = bracelevel + line.count('{') -
>> line.count('}')
>> - if datalines:
>> - pdata = json.loads('\n'.join(datalines))
>> - return pdata
>> + pdata = json.loads(output);
>> + try:
>> + return pdata[-1]
>> + except:
>> + return pdata
>>
>> def _getdependencies(self, pkg, data, version, d, ud,
>> optional=False, fetchedlist=None): if fetchedlist is None:
>> @@ -171,6 +171,9 @@ class Npm(FetchMethod):
>> pkgfullname = pkg
>> if version != '*' and not '/' in version:
>> pkgfullname += "@'%s'" % version
>> + if pkgfullname in fetchedlist:
>> + return
>> +
>> logger.debug(2, "Calling getdeps on %s" % pkg)
>> fetchcmd = "npm view %s --json --registry %s" %
>> (pkgfullname, ud.registry) output = runfetchcmd(fetchcmd, d, True)
>> @@ -190,15 +193,10 @@ class Npm(FetchMethod):
>> if (not blacklist and 'linux' not in pkg_os) or
>> '!linux' in pkg_os: logger.debug(2, "Skipping %s since it's
>> incompatible with Linux" % pkg) return
>> - #logger.debug(2, "Output URL is %s - %s - %s" %
>> (ud.basepath, ud.basename, ud.localfile))
>> - outputurl = pdata['dist']['tarball']
>> + filename = self._runpack(ud, d, pkgfullname)
>> data[pkg] = {}
>> - data[pkg]['tgz'] = os.path.basename(outputurl)
>> - if outputurl in fetchedlist:
>> - return
>> -
>> - self._runwget(ud, d, "%s --directory-prefix=%s %s" %
>> (self.basecmd, ud.prefixdir, outputurl), False)
>> - fetchedlist.append(outputurl)
>> + data[pkg]['tgz'] = filename
>> + fetchedlist.append(pkgfullname)
>>
>> dependencies = pdata.get('dependencies', {})
>> optionalDependencies = pdata.get('optionalDependencies', {})
>> @@ -225,17 +223,12 @@ class Npm(FetchMethod):
>> if obj == pkg:
>> self._getshrinkeddependencies(obj,
>> data['dependencies'][obj], data['dependencies'][obj]['version'], d,
>> ud, lockdown, manifest, False) return
>> - outputurl = "invalid"
>> - if ('resolved' not in data) or (not
>> data['resolved'].startswith('http://') and not
>> data['resolved'].startswith('https://')):
>> - # will be the case for ${PN}
>> - fetchcmd = "npm view %s@%s dist.tarball --registry %s" %
>> (pkg, version, ud.registry)
>> - logger.debug(2, "Found this matching URL: %s" %
>> str(fetchcmd))
>> - outputurl = runfetchcmd(fetchcmd, d, True)
>> - else:
>> - outputurl = data['resolved']
>> - self._runwget(ud, d, "%s --directory-prefix=%s %s" %
>> (self.basecmd, ud.prefixdir, outputurl), False) +
>> + pkgnameWithVersion = "{}@{}".format(pkg, version)
>> + logger.debug(2, "Get dependencies for
>> {}".format(pkgnameWithVersion))
>> + filename = self._runpack(ud, d, pkgnameWithVersion)
>> manifest[pkg] = {}
>> - manifest[pkg]['tgz'] = os.path.basename(outputurl).rstrip()
>> + manifest[pkg]['tgz'] = filename
>> manifest[pkg]['deps'] = {}
>>
>> if pkg in lockdown:
>> diff --git a/bitbake/lib/bb/fetch2/osc.py
>> b/bitbake/lib/bb/fetch2/osc.py index 6c60456..3e56715 100644
>> --- a/bitbake/lib/bb/fetch2/osc.py
>> +++ b/bitbake/lib/bb/fetch2/osc.py
>> @@ -1,5 +1,6 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> """
>> Bitbake "Fetch" implementation for osc (Opensuse build service
>> client). Based on the svn "Fetch" implementation.
>> diff --git a/bitbake/lib/bb/fetch2/perforce.py
>> b/bitbake/lib/bb/fetch2/perforce.py index 903a8e6..54d001e 100644
>> --- a/bitbake/lib/bb/fetch2/perforce.py
>> +++ b/bitbake/lib/bb/fetch2/perforce.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake 'Fetch' implementation for perforce
>>
>> @@ -8,18 +6,7 @@ BitBake 'Fetch' implementation for perforce
>> # Copyright (C) 2003, 2004 Chris Larson
>> # Copyright (C) 2016 Kodak Alaris, Inc.
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> # Based on functions from the base bb module, Copyright 2003 Holger
>> Schurig
>> diff --git a/bitbake/lib/bb/fetch2/repo.py
>> b/bitbake/lib/bb/fetch2/repo.py index 8c7e818..2bdbbd4 100644
>> --- a/bitbake/lib/bb/fetch2/repo.py
>> +++ b/bitbake/lib/bb/fetch2/repo.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake "Fetch" repo (git) implementation
>>
>> @@ -8,20 +6,10 @@ BitBake "Fetch" repo (git) implementation
>> # Copyright (C) 2009 Tom Rini <trini@embeddedalley.com>
>> #
>> # Based on git.py which is:
>> -#Copyright (C) 2005 Richard Purdie
>> +# Copyright (C) 2005 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import os
>> import bb
>> diff --git a/bitbake/lib/bb/fetch2/s3.py b/bitbake/lib/bb/fetch2/s3.py
>> index 1629288..ffca73c 100644
>> --- a/bitbake/lib/bb/fetch2/s3.py
>> +++ b/bitbake/lib/bb/fetch2/s3.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake 'Fetch' implementation for Amazon AWS S3.
>>
>> @@ -13,18 +11,7 @@ The aws tool must be correctly installed and
>> configured prior to use. # Based in part on bb.fetch2.wget:
>> # Copyright (C) 2003, 2004 Chris Larson
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> # Based on functions from the base bb module, Copyright 2003 Holger
>> Schurig
>> diff --git a/bitbake/lib/bb/fetch2/sftp.py
>> b/bitbake/lib/bb/fetch2/sftp.py index 81884a6..f87f292 100644
>> --- a/bitbake/lib/bb/fetch2/sftp.py
>> +++ b/bitbake/lib/bb/fetch2/sftp.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake SFTP Fetch implementation
>>
>> @@ -44,18 +42,7 @@ SRC_URI =
>> "sftp://user@host.example.com/dir/path.file.txt" # Based in part on
>> bb.fetch2.wget: # Copyright (C) 2003, 2004 Chris Larson
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> # Based on functions from the base bb module, Copyright 2003 Holger
>> Schurig
>> diff --git a/bitbake/lib/bb/fetch2/ssh.py
>> b/bitbake/lib/bb/fetch2/ssh.py index 6047ee4..f5be060 100644
>> --- a/bitbake/lib/bb/fetch2/ssh.py
>> +++ b/bitbake/lib/bb/fetch2/ssh.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> '''
>> BitBake 'Fetch' implementations
>>
>> @@ -29,18 +27,8 @@ IETF secsh internet draft:
>> # Copyright 2003 Holger Schurig
>> #
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import re, os
>> from bb.fetch2 import FetchMethod
>> diff --git a/bitbake/lib/bb/fetch2/svn.py
>> b/bitbake/lib/bb/fetch2/svn.py index ed70bcf..96d666b 100644
>> --- a/bitbake/lib/bb/fetch2/svn.py
>> +++ b/bitbake/lib/bb/fetch2/svn.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake 'Fetch' implementation for svn.
>>
>> @@ -8,18 +6,7 @@ BitBake 'Fetch' implementation for svn.
>> # Copyright (C) 2003, 2004 Chris Larson
>> # Copyright (C) 2004 Marcin Juszkiewicz
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> # Based on functions from the base bb module, Copyright 2003 Holger
>> Schurig
>> @@ -63,6 +50,9 @@ class Svn(FetchMethod):
>> relpath = self._strip_leading_slashes(ud.path)
>> ud.pkgdir = os.path.join(svndir, ud.host, relpath)
>> ud.moddir = os.path.join(ud.pkgdir, ud.module)
>> + # Protects the repository from concurrent updates, e.g. from
>> two
>> + # recipes fetching different revisions at the same time
>> + ud.svnlock = os.path.join(ud.pkgdir, "svn.lock")
>>
>> ud.setup_revisions(d)
>>
>> @@ -101,6 +91,13 @@ class Svn(FetchMethod):
>> svncmd = "%s log --limit 1 %s %s://%s/%s/" %
>> (ud.basecmd, " ".join(options), proto, svnroot, ud.module) else:
>> suffix = ""
>> +
>> + # externals may be either 'allowed' or 'nowarn', but not
>> both. Allowed
>> + # will not issue a warning, but will log to the debug
>> buffer what has likely
>> + # been downloaded by SVN.
>> + if not ("externals" in ud.parm and ud.parm["externals"]
>> == "allowed"):
>> + options.append("--ignore-externals")
>> +
>> if ud.revision:
>> options.append("-r %s" % ud.revision)
>> suffix = "@%s" % (ud.revision)
>> @@ -123,35 +120,52 @@ class Svn(FetchMethod):
>>
>> logger.debug(2, "Fetch: checking for module directory '" +
>> ud.moddir + "'")
>> - if os.access(os.path.join(ud.moddir, '.svn'), os.R_OK):
>> - svnupdatecmd = self._buildsvncommand(ud, d, "update")
>> - logger.info("Update " + ud.url)
>> - # We need to attempt to run svn upgrade first in case
>> its an older working format
>> - try:
>> - runfetchcmd(ud.basecmd + " upgrade", d,
>> workdir=ud.moddir)
>> - except FetchError:
>> - pass
>> - logger.debug(1, "Running %s", svnupdatecmd)
>> - bb.fetch2.check_network_access(d, svnupdatecmd, ud.url)
>> - runfetchcmd(svnupdatecmd, d, workdir=ud.moddir)
>> - else:
>> - svnfetchcmd = self._buildsvncommand(ud, d, "fetch")
>> - logger.info("Fetch " + ud.url)
>> - # check out sources there
>> - bb.utils.mkdirhier(ud.pkgdir)
>> - logger.debug(1, "Running %s", svnfetchcmd)
>> - bb.fetch2.check_network_access(d, svnfetchcmd, ud.url)
>> - runfetchcmd(svnfetchcmd, d, workdir=ud.pkgdir)
>> -
>> - scmdata = ud.parm.get("scmdata", "")
>> - if scmdata == "keep":
>> - tar_flags = ""
>> - else:
>> - tar_flags = "--exclude='.svn'"
>> + lf = bb.utils.lockfile(ud.svnlock)
>> +
>> + try:
>> + if os.access(os.path.join(ud.moddir, '.svn'), os.R_OK):
>> + svnupdatecmd = self._buildsvncommand(ud, d, "update")
>> + logger.info("Update " + ud.url)
>> + # We need to attempt to run svn upgrade first in
>> case its an older working format
>> + try:
>> + runfetchcmd(ud.basecmd + " upgrade", d,
>> workdir=ud.moddir)
>> + except FetchError:
>> + pass
>> + logger.debug(1, "Running %s", svnupdatecmd)
>> + bb.fetch2.check_network_access(d, svnupdatecmd,
>> ud.url)
>> + runfetchcmd(svnupdatecmd, d, workdir=ud.moddir)
>> + else:
>> + svnfetchcmd = self._buildsvncommand(ud, d, "fetch")
>> + logger.info("Fetch " + ud.url)
>> + # check out sources there
>> + bb.utils.mkdirhier(ud.pkgdir)
>> + logger.debug(1, "Running %s", svnfetchcmd)
>> + bb.fetch2.check_network_access(d, svnfetchcmd,
>> ud.url)
>> + runfetchcmd(svnfetchcmd, d, workdir=ud.pkgdir)
>> +
>> + if not ("externals" in ud.parm and ud.parm["externals"]
>> == "nowarn"):
>> + # Warn the user if this had externals (won't catch
>> them all)
>> + output = runfetchcmd("svn propget svn:externals ||
>> true", d, workdir=ud.moddir)
>> + if output:
>> + if "--ignore-externals" in svnfetchcmd.split():
>> + bb.warn("%s contains svn:externals." %
>> ud.url)
>> + bb.warn("These should be added to the recipe
>> SRC_URI as necessary.")
>> + bb.warn("svn fetch has ignored
>> externals:\n%s" % output)
>> + bb.warn("To disable this warning add
>> ';externals=nowarn' to the url.")
>> + else:
>> + bb.debug(1, "svn repository has
>> externals:\n%s" % output) +
>> + scmdata = ud.parm.get("scmdata", "")
>> + if scmdata == "keep":
>> + tar_flags = ""
>> + else:
>> + tar_flags = "--exclude='.svn'"
>>
>> - # tar them up to a defined filename
>> - runfetchcmd("tar %s -czf %s %s" % (tar_flags, ud.localpath,
>> ud.path_spec), d,
>> - cleanup=[ud.localpath], workdir=ud.pkgdir)
>> + # tar them up to a defined filename
>> + runfetchcmd("tar %s -czf %s %s" % (tar_flags,
>> ud.localpath, ud.path_spec), d,
>> + cleanup=[ud.localpath], workdir=ud.pkgdir)
>> + finally:
>> + bb.utils.unlockfile(lf)
>>
>> def clean(self, ud, d):
>> """ Clean SVN specific files and dirs """
>> diff --git a/bitbake/lib/bb/fetch2/wget.py
>> b/bitbake/lib/bb/fetch2/wget.py index 8f505b6..725586d 100644
>> --- a/bitbake/lib/bb/fetch2/wget.py
>> +++ b/bitbake/lib/bb/fetch2/wget.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake 'Fetch' implementations
>>
>> @@ -10,18 +8,7 @@ BitBake build tools.
>>
>> # Copyright (C) 2003, 2004 Chris Larson
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> # Based on functions from the base bb module, Copyright 2003 Holger
>> Schurig
>> @@ -33,11 +20,14 @@ import logging
>> import errno
>> import bb
>> import bb.progress
>> +import socket
>> +import http.client
>> import urllib.request, urllib.parse, urllib.error
>> from bb.fetch2 import FetchMethod
>> from bb.fetch2 import FetchError
>> from bb.fetch2 import logger
>> from bb.fetch2 import runfetchcmd
>> +from bb.fetch2 import FetchConnectionCache
>> from bb.utils import export_proxies
>> from bs4 import BeautifulSoup
>> from bs4 import SoupStrainer
>> @@ -132,10 +122,6 @@ class Wget(FetchMethod):
>> return True
>>
>> def checkstatus(self, fetch, ud, d, try_again=True):
>> - import urllib.request, urllib.error, urllib.parse, socket,
>> http.client
>> - from urllib.response import addinfourl
>> - from bb.fetch2 import FetchConnectionCache
>> -
>> class HTTPConnectionCache(http.client.HTTPConnection):
>> if fetch.connection_cache:
>> def connect(self):
>> @@ -168,7 +154,7 @@ class Wget(FetchMethod):
>> """
>> host = req.host
>> if not host:
>> - raise urlllib2.URLError('no host given')
>> + raise urllib.error.URLError('no host given')
>>
>> h = http_class(host, timeout=req.timeout) # will
>> parse host:port h.set_debuglevel(self._debuglevel)
>> @@ -185,7 +171,7 @@ class Wget(FetchMethod):
>> # request.
>>
>> # Don't close connection when connection_cache is
>> enabled,
>> - if fetch.connection_cache is None:
>> + if fetch.connection_cache is None:
>> headers["Connection"] = "close"
>> else:
>> headers["Connection"] = "Keep-Alive" # Works for
>> HTTP/1.0 @@ -252,7 +238,7 @@ class Wget(FetchMethod):
>> pass
>> closed = False
>>
>> - resp = addinfourl(fp_dummy(), r.msg,
>> req.get_full_url())
>> + resp = urllib.response.addinfourl(fp_dummy(), r.msg,
>> req.get_full_url()) resp.code = r.status
>> resp.msg = r.reason
>>
>> @@ -271,17 +257,18 @@ class Wget(FetchMethod):
>> fp.read()
>> fp.close()
>>
>> - newheaders = dict((k,v) for k,v in
>> list(req.headers.items())
>> - if k.lower() not in
>> ("content-length", "content-type"))
>> - return
>> self.parent.open(urllib.request.Request(req.get_full_url(),
>> -
>> headers=newheaders,
>> -
>> origin_req_host=req.origin_req_host,
>> -
>> unverifiable=True))
>> + if req.get_method() != 'GET':
>> + newheaders = dict((k, v) for k, v in
>> list(req.headers.items())
>> + if k.lower() not in
>> ("content-length", "content-type"))
>> + return
>> self.parent.open(urllib.request.Request(req.get_full_url(),
>> +
>> headers=newheaders,
>> +
>> origin_req_host=req.origin_req_host,
>> +
>> unverifiable=True))
>> - """
>> - Some servers (e.g. GitHub archives, hosted on Amazon S3)
>> return 403
>> - Forbidden when they actually mean 405 Method Not Allowed.
>> - """
>> + raise urllib.request.HTTPError(req, code, msg,
>> headers, None) +
>> + # Some servers (e.g. GitHub archives, hosted on Amazon
>> S3) return 403
>> + # Forbidden when they actually mean 405 Method Not
>> Allowed. http_error_403 = http_error_405
>>
>>
>> @@ -292,15 +279,15 @@ class Wget(FetchMethod):
>> """
>> def redirect_request(self, req, fp, code, msg, headers,
>> newurl): newreq =
>> urllib.request.HTTPRedirectHandler.redirect_request(self, req, fp,
>> code, msg, headers, newurl)
>> - newreq.get_method = lambda: req.get_method()
>> + newreq.get_method = req.get_method
>> return newreq
>> exported_proxies = export_proxies(d)
>>
>> handlers = [FixedHTTPRedirectHandler, HTTPMethodFallback]
>> - if export_proxies:
>> + if exported_proxies:
>> handlers.append(urllib.request.ProxyHandler())
>> handlers.append(CacheHTTPHandler())
>> - # XXX: Since Python 2.7.9 ssl cert validation is enabled by
>> default
>> + # Since Python 2.7.9 ssl cert validation is enabled by
>> default # see PEP-0476, this causes verification errors on some https
>> servers # so disable by default.
>> import ssl
>> @@ -319,19 +306,19 @@ class Wget(FetchMethod):
>> '''Adds Basic auth to http request, pass in
>> login:password as string''' import base64
>> encodeuser =
>> base64.b64encode(login_str.encode('utf-8')).decode("utf-8")
>> - authheader = "Basic %s" % encodeuser
>> + authheader = "Basic %s" % encodeuser
>> r.add_header("Authorization", authheader)
>>
>> - if ud.user:
>> - add_basic_auth(ud.user, r)
>> + if ud.user and ud.pswd:
>> + add_basic_auth(ud.user + ':' + ud.pswd, r)
>>
>> try:
>> - import netrc, urllib.parse
>> + import netrc
>> n = netrc.netrc()
>> login, unused, password =
>> n.authenticators(urllib.parse.urlparse(uri).hostname)
>> add_basic_auth("%s:%s" % (login, password), r) except (TypeError,
>> ImportError, IOError, netrc.NetrcParseError):
>> - pass
>> + pass
>>
>> with opener.open(r) as response:
>> pass
>> @@ -396,18 +383,14 @@ class Wget(FetchMethod):
>> (oldpn, oldpv, oldsuffix) = old
>> (newpn, newpv, newsuffix) = new
>>
>> - """
>> - Check for a new suffix type that we have never heard of
>> before
>> - """
>> - if (newsuffix):
>> + # Check for a new suffix type that we have never heard of
>> before
>> + if newsuffix:
>> m = self.suffix_regex_comp.search(newsuffix)
>> if not m:
>> bb.warn("%s has a possible unknown suffix: %s" %
>> (newpn, newsuffix)) return False
>>
>> - """
>> - Not our package so ignore it
>> - """
>> + # Not our package so ignore it
>> if oldpn != newpn:
>> return False
>>
>> @@ -473,15 +456,14 @@ class Wget(FetchMethod):
>>
>> return ""
>>
>> - def _check_latest_version_by_dir(self, dirver, package,
>> package_regex,
>> - current_version, ud, d):
>> + def _check_latest_version_by_dir(self, dirver, package,
>> package_regex, current_version, ud, d): """
>> - Scan every directory in order to get upstream version.
>> + Scan every directory in order to get upstream version.
>> """
>> version_dir = ['', '', '']
>> version = ['', '', '']
>>
>> - dirver_regex =
>> re.compile("(?P<pfx>\D*)(?P<ver>(\d+[\.\-_])+(\d+))")
>> + dirver_regex =
>> re.compile(r"(?P<pfx>\D*)(?P<ver>(\d+[\.\-_])+(\d+))") s =
>> dirver_regex.search(dirver) if s:
>> version_dir[1] = s.group('ver')
>> @@ -541,26 +523,26 @@ class Wget(FetchMethod):
>> gst-fluendo-mp3
>> """
>> # match most patterns which uses "-" as separator to version
>> digits
>> - pn_prefix1 = "[a-zA-Z][a-zA-Z0-9]*([-_][a-zA-Z]\w+)*\+?[-_]"
>> + pn_prefix1 = r"[a-zA-Z][a-zA-Z0-9]*([-_][a-zA-Z]\w+)*\+?[-_]"
>> # a loose pattern such as for unzip552.tar.gz
>> - pn_prefix2 = "[a-zA-Z]+"
>> + pn_prefix2 = r"[a-zA-Z]+"
>> # a loose pattern such as for 80325-quicky-0.4.tar.gz
>> - pn_prefix3 = "[0-9]+[-]?[a-zA-Z]+"
>> + pn_prefix3 = r"[0-9]+[-]?[a-zA-Z]+"
>> # Save the Package Name (pn) Regex for use later
>> - pn_regex = "(%s|%s|%s)" % (pn_prefix1, pn_prefix2,
>> pn_prefix3)
>> + pn_regex = r"(%s|%s|%s)" % (pn_prefix1, pn_prefix2,
>> pn_prefix3)
>> # match version
>> - pver_regex = "(([A-Z]*\d+[a-zA-Z]*[\.\-_]*)+)"
>> + pver_regex = r"(([A-Z]*\d+[a-zA-Z]*[\.\-_]*)+)"
>>
>> # match arch
>> parch_regex = "-source|_all_"
>>
>> # src.rpm extension was added only for rpm package. Can be
>> removed if the rpm # packaged will always be considered as having to
>> be manually upgraded
>> - psuffix_regex =
>> "(tar\.gz|tgz|tar\.bz2|zip|xz|tar\.lz|rpm|bz2|orig\.tar\.gz|tar\.xz|src\.tar\.gz|src\.tgz|svnr\d+\.tar\.bz2|stable\.tar\.gz|src\.rpm)"
>> + psuffix_regex =
>> r"(tar\.gz|tgz|tar\.bz2|zip|xz|tar\.lz|rpm|bz2|orig\.tar\.gz|tar\.xz|src\.tar\.gz|src\.tgz|svnr\d+\.tar\.bz2|stable\.tar\.gz|src\.rpm)"
>> # match name, version and archive type of a package
>> - package_regex_comp =
>> re.compile("(?P<name>%s?\.?v?)(?P<pver>%s)(?P<arch>%s)?[\.-](?P<type>%s$)"
>> + package_regex_comp =
>> re.compile(r"(?P<name>%s?\.?v?)(?P<pver>%s)(?P<arch>%s)?[\.-](?P<type>%s$)"
>> % (pn_regex, pver_regex, parch_regex, psuffix_regex))
>> self.suffix_regex_comp = re.compile(psuffix_regex)
>> @@ -572,7 +554,7 @@ class Wget(FetchMethod):
>> version = self._parse_path(package_regex_comp, package)
>> if version:
>> package_custom_regex_comp = re.compile(
>> -
>> "(?P<name>%s)(?P<pver>%s)(?P<arch>%s)?[\.-](?P<type>%s)" %
>> +
>> r"(?P<name>%s)(?P<pver>%s)(?P<arch>%s)?[\.-](?P<type>%s)" %
>> (re.escape(version[0]), pver_regex, parch_regex, psuffix_regex)) else:
>> package_custom_regex_comp = None
>> @@ -589,7 +571,7 @@ class Wget(FetchMethod):
>> current_version = ['', d.getVar('PV'), '']
>>
>> """possible to have no version in pkg name, such as
>> spectrum-fw"""
>> - if not re.search("\d+", package):
>> + if not re.search(r"\d+", package):
>> current_version[1] = re.sub('_', '.', current_version[1])
>> current_version[1] = re.sub('-', '.', current_version[1])
>> return (current_version[1], '')
>> @@ -607,13 +589,13 @@ class Wget(FetchMethod):
>>
>> # search for version matches on folders inside the path,
>> like: # "5.7" in
>> http://download.gnome.org/sources/${PN}/5.7/${PN}-${PV}.tar.gz
>> - dirver_regex =
>> re.compile("(?P<dirver>[^/]*(\d+\.)*\d+([-_]r\d+)*)/")
>> + dirver_regex =
>> re.compile(r"(?P<dirver>[^/]*(\d+\.)*\d+([-_]r\d+)*)/") m =
>> dirver_regex.search(path) if m:
>> pn = d.getVar('PN')
>> dirver = m.group('dirver')
>>
>> - dirver_pn_regex = re.compile("%s\d?" %
>> (re.escape(pn)))
>> + dirver_pn_regex = re.compile(r"%s\d?" %
>> (re.escape(pn))) if not dirver_pn_regex.search(dirver):
>> return (self._check_latest_version_by_dir(dirver,
>> package, package_regex, current_version, ud,
>> d), '') diff --git a/bitbake/lib/bb/main.py b/bitbake/lib/bb/main.py
>> index 732a315..af2880f 100755
>> --- a/bitbake/lib/bb/main.py
>> +++ b/bitbake/lib/bb/main.py
>> @@ -1,6 +1,3 @@
>> -#!/usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # Copyright (C) 2003, 2004 Chris Larson
>> # Copyright (C) 2003, 2004 Phil Blundell
>> @@ -9,18 +6,8 @@
>> # Copyright (C) 2005 ROAD GmbH
>> # Copyright (C) 2006 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import os
>> import sys
>> @@ -267,6 +254,11 @@ class
>> BitBakeConfigParameters(cookerdata.ConfigParameters): help="Do not
>> run any setscene tasks. sstate will be ignored and " "everything
>> needed, built.")
>> + parser.add_option("", "--skip-setscene", action="store_true",
>> + dest="skipsetscene", default=False,
>> + help="Skip setscene tasks if they would be
>> executed. Tasks previously "
>> + "restored from sstate will be kept,
>> unlike --no-setscene") +
>> parser.add_option("", "--setscene-only", action="store_true",
>> dest="setsceneonly", default=False,
>> help="Only run setscene tasks, don't run
>> any real tasks.") @@ -448,7 +440,7 @@ def setup_bitbake(configParams,
>> configuration, extrafeatures=None): else:
>> logger.info("Reconnecting to bitbake server...")
>> if not os.path.exists(sockname):
>> - print("Previous bitbake instance shutting
>> down?, waiting to retry...")
>> + logger.info("Previous bitbake instance
>> shutting down?, waiting to retry...") i = 0
>> lock = None
>> # Wait for 5s or until we can get the lock
>> @@ -460,12 +452,7 @@ def setup_bitbake(configParams, configuration,
>> extrafeatures=None): bb.utils.unlockfile(lock)
>> raise
>> bb.server.process.ProcessTimeout("Bitbake still shutting down as
>> socket exists but no lock?") if not configParams.server_only:
>> - try:
>> - server_connection =
>> bb.server.process.connectProcessServer(sockname, featureset)
>> - except EOFError:
>> - # The server may have been shutting down but
>> not closed the socket yet. If that happened,
>> - # ignore it.
>> - pass
>> + server_connection =
>> bb.server.process.connectProcessServer(sockname, featureset)
>> if server_connection or configParams.server_only:
>> break
>> @@ -475,12 +462,14 @@ def setup_bitbake(configParams, configuration,
>> extrafeatures=None): if not retries:
>> raise
>> retries -= 1
>> - if isinstance(e, (bb.server.process.ProcessTimeout,
>> BrokenPipeError)):
>> - logger.info("Retrying server connection...")
>> + tryno = 8 - retries
>> + if isinstance(e, (bb.server.process.ProcessTimeout,
>> BrokenPipeError, EOFError)):
>> + logger.info("Retrying server connection
>> (#%d)..." % tryno) else:
>> - logger.info("Retrying server connection... (%s)"
>> % traceback.format_exc())
>> + logger.info("Retrying server connection (#%d)...
>> (%s)" % (tryno, traceback.format_exc())) if not retries:
>> - bb.fatal("Unable to connect to bitbake server, or
>> start one")
>> + bb.fatal("Unable to connect to bitbake server, or
>> start one (server startup failures would be in
>> bitbake-cookerdaemon.log).")
>> + bb.event.print_ui_queue()
>> if retries < 5:
>> time.sleep(5)
>>
>> @@ -502,7 +491,7 @@ def setup_bitbake(configParams, configuration,
>> extrafeatures=None): def lockBitbake():
>> topdir = bb.cookerdata.findTopdir()
>> if not topdir:
>> - bb.error("Unable to find conf/bblayers.conf or
>> conf/bitbake.conf. BBAPTH is unset and/or not in a build directory?")
>> + bb.error("Unable to find conf/bblayers.conf or
>> conf/bitbake.conf. BBPATH is unset and/or not in a build directory?")
>> raise BBMainFatal lockfile = topdir + "/bitbake.lock"
>> return topdir, bb.utils.lockfile(lockfile, False, False)
>> diff --git a/bitbake/lib/bb/methodpool.py
>> b/bitbake/lib/bb/methodpool.py index 49aed33..51783ac 100644
>> --- a/bitbake/lib/bb/methodpool.py
>> +++ b/bitbake/lib/bb/methodpool.py
>> @@ -1,21 +1,8 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> -#
>> #
>> # Copyright (C) 2006 Holger Hans Peter Freyther
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from bb.utils import better_compile, better_exec
>>
>> diff --git a/bitbake/lib/bb/monitordisk.py
>> b/bitbake/lib/bb/monitordisk.py index 833cd3d..1a25b00 100644
>> --- a/bitbake/lib/bb/monitordisk.py
>> +++ b/bitbake/lib/bb/monitordisk.py
>> @@ -1,21 +1,8 @@
>> -#!/usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # Copyright (C) 2012 Robert Yang
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import os, logging, re, sys
>> import bb
>> @@ -28,16 +15,16 @@ def convertGMK(unit):
>>
>> """ Convert the space unit G, M, K, the unit is case-insensitive
>> """
>> - unitG = re.match('([1-9][0-9]*)[gG]\s?$', unit)
>> + unitG = re.match(r'([1-9][0-9]*)[gG]\s?$', unit)
>> if unitG:
>> return int(unitG.group(1)) * (1024 ** 3)
>> - unitM = re.match('([1-9][0-9]*)[mM]\s?$', unit)
>> + unitM = re.match(r'([1-9][0-9]*)[mM]\s?$', unit)
>> if unitM:
>> return int(unitM.group(1)) * (1024 ** 2)
>> - unitK = re.match('([1-9][0-9]*)[kK]\s?$', unit)
>> + unitK = re.match(r'([1-9][0-9]*)[kK]\s?$', unit)
>> if unitK:
>> return int(unitK.group(1)) * 1024
>> - unitN = re.match('([1-9][0-9]*)\s?$', unit)
>> + unitN = re.match(r'([1-9][0-9]*)\s?$', unit)
>> if unitN:
>> return int(unitN.group(1))
>> else:
>> @@ -83,7 +70,7 @@ def getDiskData(BBDirs, configuration):
>> for pathSpaceInode in BBDirs.split():
>> # The input format is: "dir,space,inode", dir is a must,
>> space # and inode are optional
>> - pathSpaceInodeRe = re.match('([^,]*),([^,]*),([^,]*),?(.*)',
>> pathSpaceInode)
>> + pathSpaceInodeRe =
>> re.match(r'([^,]*),([^,]*),([^,]*),?(.*)', pathSpaceInode) if not
>> pathSpaceInodeRe: printErr("Invalid value in BB_DISKMON_DIRS: %s" %
>> pathSpaceInode) return None
>> @@ -147,7 +134,7 @@ def getInterval(configuration):
>> else:
>> # The disk space or inode interval is optional, but it should
>> # have a correct value once it is specified
>> - intervalRe = re.match('([^,]*),?\s*(.*)', interval)
>> + intervalRe = re.match(r'([^,]*),?\s*(.*)', interval)
>> if intervalRe:
>> intervalSpace = intervalRe.group(1)
>> if intervalSpace:
>> diff --git a/bitbake/lib/bb/msg.py b/bitbake/lib/bb/msg.py
>> index 96f077e..6216eb3 100644
>> --- a/bitbake/lib/bb/msg.py
>> +++ b/bitbake/lib/bb/msg.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake 'msg' implementation
>>
>> @@ -9,18 +7,8 @@ Message handling infrastructure for bitbake
>>
>> # Copyright (C) 2006 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import sys
>> import copy
>> diff --git a/bitbake/lib/bb/namedtuple_with_abc.py
>> b/bitbake/lib/bb/namedtuple_with_abc.py index 32f2fc6..646aed6 100644
>> --- a/bitbake/lib/bb/namedtuple_with_abc.py
>> +++ b/bitbake/lib/bb/namedtuple_with_abc.py
>> @@ -1,6 +1,8 @@
>> #
>> http://code.activestate.com/recipes/577629-namedtupleabc-abstract-base-class-mix-in-for-named/
>> -#!/usr/bin/env python # Copyright (c) 2011 Jan Kaliszewski (zuo).
>> Available under the MIT License. +#
>> +# SPDX-License-Identifier: MIT
>> +#
>>
>> """
>> namedtuple_with_abc.py:
>> diff --git a/bitbake/lib/bb/parse/__init__.py
>> b/bitbake/lib/bb/parse/__init__.py index 5397d57..76e180b 100644
>> --- a/bitbake/lib/bb/parse/__init__.py
>> +++ b/bitbake/lib/bb/parse/__init__.py
>> @@ -9,20 +9,10 @@ File parsers for the BitBake build tools.
>> # Copyright (C) 2003, 2004 Chris Larson
>> # Copyright (C) 2003, 2004 Phil Blundell
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> # Based on functions from the base bb module, Copyright 2003 Holger
>> Schurig +#
>>
>> handlers = []
>>
>> diff --git a/bitbake/lib/bb/parse/ast.py b/bitbake/lib/bb/parse/ast.py
>> index 6d7c80b..f0911e6 100644
>> --- a/bitbake/lib/bb/parse/ast.py
>> +++ b/bitbake/lib/bb/parse/ast.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> AbstractSyntaxTree classes for the Bitbake language
>> """
>> @@ -8,19 +6,8 @@
>> # Copyright (C) 2003, 2004 Phil Blundell
>> # Copyright (C) 2009 Holger Hans Peter Freyther
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. -
>>
>> import re
>> import string
>> diff --git a/bitbake/lib/bb/parse/parse_py/BBHandler.py
>> b/bitbake/lib/bb/parse/parse_py/BBHandler.py index 01fc47e..6f7cf82
>> 100644 --- a/bitbake/lib/bb/parse/parse_py/BBHandler.py
>> +++ b/bitbake/lib/bb/parse/parse_py/BBHandler.py
>> @@ -1,6 +1,3 @@
>> -#!/usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> class for handling .bb files
>>
>> @@ -12,19 +9,8 @@
>> # Copyright (C) 2003, 2004 Chris Larson
>> # Copyright (C) 2003, 2004 Phil Blundell
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. -
>>
>> import re, bb, os
>> import logging
>> @@ -38,14 +24,15 @@ from .ConfHandler import include, init
>> # For compatibility
>> bb.deprecate_import(__name__, "bb.parse", ["vars_from_file"])
>>
>> -__func_start_regexp__ =
>> re.compile( r"(((?P<py>python)|(?P<fr>fakeroot))\s*)*(?P<func>[\w\.\-\+\{\}\$]+)?\s*\(\s*\)\s*{$" )
>> -__inherit_regexp__ = re.compile( r"inherit\s+(.+)" )
>> -__export_func_regexp__ = re.compile( r"EXPORT_FUNCTIONS\s+(.+)" )
>> -__addtask_regexp__ =
>> re.compile("addtask\s+(?P<func>\w+)\s*((before\s*(?P<before>((.*(?=after))|(.*))))|(after\s*(?P<after>((.*(?=before))|(.*)))))*")
>> -__deltask_regexp__ = re.compile("deltask\s+(?P<func>\w+)")
>> -__addhandler_regexp__ = re.compile( r"addhandler\s+(.+)" )
>> -__def_regexp__ = re.compile( r"def\s+(\w+).*:" )
>> -__python_func_regexp__ = re.compile( r"(\s+.*)|(^$)|(^#)" )
>> +__func_start_regexp__ =
>> re.compile(r"(((?P<py>python)|(?P<fr>fakeroot))\s*)*(?P<func>[\w\.\-\+\{\}\$]+)?\s*\(\s*\)\s*{$" )
>> +__inherit_regexp__ = re.compile(r"inherit\s+(.+)" )
>> +__export_func_regexp__ = re.compile(r"EXPORT_FUNCTIONS\s+(.+)" )
>> +__addtask_regexp__ =
>> re.compile(r"addtask\s+(?P<func>\w+)\s*((before\s*(?P<before>((.*(?=after))|(.*))))|(after\s*(?P<after>((.*(?=before))|(.*)))))*")
>> +__deltask_regexp__ =
>> re.compile(r"deltask\s+(?P<func>\w+)(?P<ignores>.*)")
>> +__addhandler_regexp__ = re.compile(r"addhandler\s+(.+)" )
>> +__def_regexp__ = re.compile(r"def\s+(\w+).*:" )
>> +__python_func_regexp__ = re.compile(r"(\s+.*)|(^$)|(^#)" )
>> +__python_tab_regexp__ = re.compile(r" *\t") __infunc__ = []
>> __inpython__ = False @@ -160,6 +147,16 @@ def handle(fn, d, include):
>> def feeder(lineno, s, fn, root, statements, eof=False): global
>> __func_start_regexp__, __inherit_regexp__, __export_func_regexp__,
>> __addtask_regexp__, __addhandler_regexp__, __def_regexp__,
>> __python_func_regexp__, __inpython__, __infunc__, __body__, bb,
>> __residue__, __classname__ +
>> + # Check tabs in python functions:
>> + # - def py_funcname(): covered by __inpython__
>> + # - python(): covered by '__anonymous' == __infunc__[0]
>> + # - python funcname(): covered by __infunc__[3]
>> + if __inpython__ or (__infunc__ and ('__anonymous' ==
>> __infunc__[0] or __infunc__[3])):
>> + tab = __python_tab_regexp__.match(s)
>> + if tab:
>> + bb.warn('python should use 4 spaces indentation, but
>> found tabs in %s, line %s' % (root, lineno)) +
>> if __infunc__:
>> if s == '}':
>> __body__.append('')
>> @@ -225,11 +222,27 @@ def feeder(lineno, s, fn, root, statements,
>> eof=False):
>> m = __addtask_regexp__.match(s)
>> if m:
>> + if len(m.group().split()) == 2:
>> + # Check and warn for "addtask task1 task2"
>> + m2 = re.match(r"addtask\s+(?P<func>\w+)(?P<ignores>.*)",
>> s)
>> + if m2 and m2.group('ignores'):
>> + logger.warning('addtask ignored: "%s"' %
>> m2.group('ignores')) +
>> + # Check and warn for "addtask task1 before task2 before
>> task3", the
>> + # similar to "after"
>> + taskexpression = s.split()
>> + for word in ('before', 'after'):
>> + if taskexpression.count(word) > 1:
>> + logger.warning("addtask contained multiple '%s'
>> keywords, only one is supported" % word) +
>> ast.handleAddTask(statements, fn, lineno, m)
>> return
>>
>> m = __deltask_regexp__.match(s)
>> if m:
>> + # Check and warn "for deltask task1 task2"
>> + if m.group('ignores'):
>> + logger.warning('deltask ignored: "%s"' %
>> m.group('ignores')) ast.handleDelTask(statements, fn, lineno, m)
>> return
>>
>> diff --git a/bitbake/lib/bb/parse/parse_py/ConfHandler.py
>> b/bitbake/lib/bb/parse/parse_py/ConfHandler.py index 9d3ebe1..2e84b91
>> 100644 --- a/bitbake/lib/bb/parse/parse_py/ConfHandler.py
>> +++ b/bitbake/lib/bb/parse/parse_py/ConfHandler.py
>> @@ -1,6 +1,3 @@
>> -#!/usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> class for handling configuration data files
>>
>> @@ -11,18 +8,8 @@
>> # Copyright (C) 2003, 2004 Chris Larson
>> # Copyright (C) 2003, 2004 Phil Blundell
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import errno
>> import re
>> @@ -147,7 +134,7 @@ def handle(fn, data, include):
>> continue
>> s = s.rstrip()
>> while s[-1] == '\\':
>> - s2 = f.readline().strip()
>> + s2 = f.readline().rstrip()
>> lineno = lineno + 1
>> if (not s2 or s2 and s2[0] != "#") and s[0] == "#" :
>> bb.fatal("There is a confusing multiline, partially
>> commented expression on line %s of file %s (%s).\nPlease clarify
>> whether this is all a comment or should be parsed." % (lineno, fn,
>> s)) diff --git a/bitbake/lib/bb/parse/parse_py/__init__.py
>> b/bitbake/lib/bb/parse/parse_py/__init__.py index 3e658d0..f508afa
>> 100644 --- a/bitbake/lib/bb/parse/parse_py/__init__.py +++
>> b/bitbake/lib/bb/parse/parse_py/__init__.py @@ -1,6 +1,3 @@
>> -#!/usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake Parsers
>>
>> @@ -11,20 +8,10 @@ File parsers for the BitBake build tools.
>> # Copyright (C) 2003, 2004 Chris Larson
>> # Copyright (C) 2003, 2004 Phil Blundell
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> # Based on functions from the base bb module, Copyright 2003 Holger
>> Schurig +#
>>
>> from __future__ import absolute_import
>> from . import ConfHandler
>> diff --git a/bitbake/lib/bb/persist_data.py
>> b/bitbake/lib/bb/persist_data.py index bef7018..de8f87a 100644
>> --- a/bitbake/lib/bb/persist_data.py
>> +++ b/bitbake/lib/bb/persist_data.py
>> @@ -8,18 +8,8 @@ currently, providing a key/value store accessed by
>> 'domain'. # Copyright (C) 2007 Richard Purdie
>> # Copyright (C) 2010 Chris Larson <chris_larson@mentor.com>
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import collections
>> import logging
>> @@ -29,6 +19,7 @@ import warnings
>> from bb.compat import total_ordering
>> from collections import Mapping
>> import sqlite3
>> +import contextlib
>>
>> sqlversion = sqlite3.sqlite_version_info
>> if sqlversion[0] < 3 or (sqlversion[0] == 3 and sqlversion[1] < 3):
>> @@ -36,84 +27,181 @@ if sqlversion[0] < 3 or (sqlversion[0] == 3 and
>> sqlversion[1] < 3):
>>
>> logger = logging.getLogger("BitBake.PersistData")
>> -if hasattr(sqlite3, 'enable_shared_cache'):
>> - try:
>> - sqlite3.enable_shared_cache(True)
>> - except sqlite3.OperationalError:
>> - pass
>> -
>>
>> @total_ordering
>> class SQLTable(collections.MutableMapping):
>> + class _Decorators(object):
>> + @staticmethod
>> + def retry(*, reconnect=True):
>> + """
>> + Decorator that restarts a function if a database locked
>> sqlite
>> + exception occurs. If reconnect is True, the database
>> connection
>> + will be closed and reopened each time a failure occurs
>> + """
>> + def retry_wrapper(f):
>> + def wrap_func(self, *args, **kwargs):
>> + # Reconnect if necessary
>> + if self.connection is None and reconnect:
>> + self.reconnect()
>> +
>> + count = 0
>> + while True:
>> + try:
>> + return f(self, *args, **kwargs)
>> + except sqlite3.OperationalError as exc:
>> + if count < 500 and ('is locked' in
>> str(exc) or 'locking protocol' in str(exc)):
>> + count = count + 1
>> + if reconnect:
>> + self.reconnect()
>> + continue
>> + raise
>> + return wrap_func
>> + return retry_wrapper
>> +
>> + @staticmethod
>> + def transaction(f):
>> + """
>> + Decorator that starts a database transaction and creates
>> a database
>> + cursor for performing queries. If no exception is
>> thrown, the
>> + database results are commited. If an exception occurs,
>> the database
>> + is rolled back. In all cases, the cursor is closed after
>> the
>> + function ends.
>> +
>> + Note that the cursor is passed as an extra argument to
>> the function
>> + after `self` and before any of the normal arguments
>> + """
>> + def wrap_func(self, *args, **kwargs):
>> + # Context manager will COMMIT the database on
>> success,
>> + # or ROLLBACK on an exception
>> + with self.connection:
>> + # Automatically close the cursor when done
>> + with
>> contextlib.closing(self.connection.cursor()) as cursor:
>> + return f(self, cursor, *args, **kwargs)
>> + return wrap_func
>> +
>> """Object representing a table/domain in the database"""
>> def __init__(self, cachefile, table):
>> self.cachefile = cachefile
>> self.table = table
>> - self.cursor = connect(self.cachefile)
>> -
>> - self._execute("CREATE TABLE IF NOT EXISTS %s(key TEXT, value
>> TEXT);"
>> - % table)
>> -
>> - def _execute(self, *query):
>> - """Execute a query, waiting to acquire a lock if necessary"""
>> - count = 0
>> - while True:
>> - try:
>> - return self.cursor.execute(*query)
>> - except sqlite3.OperationalError as exc:
>> - if 'database is locked' in str(exc) and count < 500:
>> - count = count + 1
>> +
>> + self.connection = None
>> + self._execute_single("CREATE TABLE IF NOT EXISTS %s(key TEXT
>> PRIMARY KEY NOT NULL, value TEXT);" % table) +
>> + @_Decorators.retry(reconnect=False)
>> + @_Decorators.transaction
>> + def _setup_database(self, cursor):
>> + cursor.execute("pragma synchronous = off;")
>> + # Enable WAL and keep the autocheckpoint length small (the
>> default is
>> + # usually 1000). Persistent caches are usually read-mostly,
>> so keeping
>> + # this short will keep readers running quickly
>> + cursor.execute("pragma journal_mode = WAL;")
>> + cursor.execute("pragma wal_autocheckpoint = 100;")
>> +
>> + def reconnect(self):
>> + if self.connection is not None:
>> + self.connection.close()
>> + self.connection = sqlite3.connect(self.cachefile, timeout=5)
>> + self.connection.text_factory = str
>> + self._setup_database()
>> +
>> + @_Decorators.retry()
>> + @_Decorators.transaction
>> + def _execute_single(self, cursor, *query):
>> + """
>> + Executes a single query and discards the results. This
>> correctly closes
>> + the database cursor when finished
>> + """
>> + cursor.execute(*query)
>> +
>> + @_Decorators.retry()
>> + def _row_iter(self, f, *query):
>> + """
>> + Helper function that returns a row iterator. Each time
>> __next__ is
>> + called on the iterator, the provided function is evaluated
>> to determine
>> + the return value
>> + """
>> + class CursorIter(object):
>> + def __init__(self, cursor):
>> + self.cursor = cursor
>> +
>> + def __iter__(self):
>> + return self
>> +
>> + def __next__(self):
>> + row = self.cursor.fetchone()
>> + if row is None:
>> self.cursor.close()
>> - self.cursor = connect(self.cachefile)
>> - continue
>> - raise
>> + raise StopIteration
>> + return f(row)
>> +
>> + def __enter__(self):
>> + return self
>> +
>> + def __exit__(self, typ, value, traceback):
>> + self.cursor.close()
>> + return False
>> +
>> + cursor = self.connection.cursor()
>> + try:
>> + cursor.execute(*query)
>> + return CursorIter(cursor)
>> + except:
>> + cursor.close()
>>
>> def __enter__(self):
>> - self.cursor.__enter__()
>> + self.connection.__enter__()
>> return self
>>
>> def __exit__(self, *excinfo):
>> - self.cursor.__exit__(*excinfo)
>> -
>> - def __getitem__(self, key):
>> - data = self._execute("SELECT * from %s where key=?;" %
>> - self.table, [key])
>> - for row in data:
>> + self.connection.__exit__(*excinfo)
>> +
>> + @_Decorators.retry()
>> + @_Decorators.transaction
>> + def __getitem__(self, cursor, key):
>> + cursor.execute("SELECT * from %s where key=?;" % self.table,
>> [key])
>> + row = cursor.fetchone()
>> + if row is not None:
>> return row[1]
>> raise KeyError(key)
>>
>> - def __delitem__(self, key):
>> + @_Decorators.retry()
>> + @_Decorators.transaction
>> + def __delitem__(self, cursor, key):
>> if key not in self:
>> raise KeyError(key)
>> - self._execute("DELETE from %s where key=?;" % self.table,
>> [key])
>> + cursor.execute("DELETE from %s where key=?;" % self.table,
>> [key])
>> - def __setitem__(self, key, value):
>> + @_Decorators.retry()
>> + @_Decorators.transaction
>> + def __setitem__(self, cursor, key, value):
>> if not isinstance(key, str):
>> raise TypeError('Only string keys are supported')
>> elif not isinstance(value, str):
>> raise TypeError('Only string values are supported')
>>
>> - data = self._execute("SELECT * from %s where key=?;" %
>> - self.table, [key])
>> - exists = len(list(data))
>> - if exists:
>> - self._execute("UPDATE %s SET value=? WHERE key=?;" %
>> self.table,
>> - [value, key])
>> + cursor.execute("SELECT * from %s where key=?;" % self.table,
>> [key])
>> + row = cursor.fetchone()
>> + if row is not None:
>> + cursor.execute("UPDATE %s SET value=? WHERE key=?;" %
>> self.table, [value, key]) else:
>> - self._execute("INSERT into %s(key, value) values
>> (?, ?);" %
>> - self.table, [key, value])
>> -
>> - def __contains__(self, key):
>> - return key in set(self)
>> -
>> - def __len__(self):
>> - data = self._execute("SELECT COUNT(key) FROM %s;" %
>> self.table)
>> - for row in data:
>> + cursor.execute("INSERT into %s(key, value) values
>> (?, ?);" % self.table, [key, value]) +
>> + @_Decorators.retry()
>> + @_Decorators.transaction
>> + def __contains__(self, cursor, key):
>> + cursor.execute('SELECT * from %s where key=?;' % self.table,
>> [key])
>> + return cursor.fetchone() is not None
>> +
>> + @_Decorators.retry()
>> + @_Decorators.transaction
>> + def __len__(self, cursor):
>> + cursor.execute("SELECT COUNT(key) FROM %s;" % self.table)
>> + row = cursor.fetchone()
>> + if row is not None:
>> return row[0]
>>
>> def __iter__(self):
>> - data = self._execute("SELECT key FROM %s;" % self.table)
>> - return (row[0] for row in data)
>> + return self._row_iter(lambda row: row[0], "SELECT key from
>> %s;" % self.table)
>> def __lt__(self, other):
>> if not isinstance(other, Mapping):
>> @@ -122,25 +210,27 @@ class SQLTable(collections.MutableMapping):
>> return len(self) < len(other)
>>
>> def get_by_pattern(self, pattern):
>> - data = self._execute("SELECT * FROM %s WHERE key LIKE ?;" %
>> - self.table, [pattern])
>> - return [row[1] for row in data]
>> + return self._row_iter(lambda row: row[1], "SELECT * FROM %s
>> WHERE key LIKE ?;" %
>> + self.table, [pattern])
>>
>> def values(self):
>> return list(self.itervalues())
>>
>> def itervalues(self):
>> - data = self._execute("SELECT value FROM %s;" % self.table)
>> - return (row[0] for row in data)
>> + return self._row_iter(lambda row: row[0], "SELECT value FROM
>> %s;" %
>> + self.table)
>>
>> def items(self):
>> return list(self.iteritems())
>>
>> def iteritems(self):
>> - return self._execute("SELECT * FROM %s;" % self.table)
>> + return self._row_iter(lambda row: (row[0], row[1]), "SELECT
>> * FROM %s;" %
>> + self.table)
>>
>> - def clear(self):
>> - self._execute("DELETE FROM %s;" % self.table)
>> + @_Decorators.retry()
>> + @_Decorators.transaction
>> + def clear(self, cursor):
>> + cursor.execute("DELETE FROM %s;" % self.table)
>>
>> def has_key(self, key):
>> return key in self
>> @@ -194,12 +284,6 @@ class PersistData(object):
>> """
>> del self.data[domain][key]
>>
>> -def connect(database):
>> - connection = sqlite3.connect(database, timeout=5,
>> isolation_level=None)
>> - connection.execute("pragma synchronous = off;")
>> - connection.text_factory = str
>> - return connection
>> -
>> def persist(domain, d):
>> """Convenience factory for SQLTable objects based upon
>> metadata""" import bb.utils
>> diff --git a/bitbake/lib/bb/process.py b/bitbake/lib/bb/process.py
>> index e69697c..2dc472a 100644
>> --- a/bitbake/lib/bb/process.py
>> +++ b/bitbake/lib/bb/process.py
>> @@ -1,3 +1,7 @@
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> import logging
>> import signal
>> import subprocess
>> diff --git a/bitbake/lib/bb/progress.py b/bitbake/lib/bb/progress.py
>> index f54d1c7..4022caa 100644
>> --- a/bitbake/lib/bb/progress.py
>> +++ b/bitbake/lib/bb/progress.py
>> @@ -4,18 +4,8 @@ BitBake progress handling code
>>
>> # Copyright (C) 2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import sys
>> import re
>> @@ -23,6 +13,7 @@ import time
>> import inspect
>> import bb.event
>> import bb.build
>> +from bb.build import StdoutNoopContextManager
>>
>> class ProgressHandler(object):
>> """
>> @@ -37,7 +28,14 @@ class ProgressHandler(object):
>> if outfile:
>> self._outfile = outfile
>> else:
>> - self._outfile = sys.stdout
>> + self._outfile = StdoutNoopContextManager()
>> +
>> + def __enter__(self):
>> + self._outfile.__enter__()
>> + return self
>> +
>> + def __exit__(self, *excinfo):
>> + self._outfile.__exit__(*excinfo)
>>
>> def _fire_progress(self, taskprogress, rate=None):
>> """Internal function to fire the progress event"""
>> @@ -157,6 +155,12 @@ class MultiStageProgressReporter(object):
>> self._stage_total = None
>> self._callers = []
>>
>> + def __enter__(self):
>> + return self
>> +
>> + def __exit__(self, *excinfo):
>> + pass
>> +
>> def _fire_progress(self, taskprogress):
>> bb.event.fire(bb.build.TaskProgress(taskprogress),
>> self._data)
>> diff --git a/bitbake/lib/bb/providers.py b/bitbake/lib/bb/providers.py
>> index c2aa98c..f80963c 100644
>> --- a/bitbake/lib/bb/providers.py
>> +++ b/bitbake/lib/bb/providers.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # Copyright (C) 2003, 2004 Chris Larson
>> # Copyright (C) 2003, 2004 Phil Blundell
>> @@ -8,18 +6,8 @@
>> # Copyright (C) 2005 ROAD GmbH
>> # Copyright (C) 2006 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import re
>> import logging
>> @@ -129,7 +117,7 @@ def findPreferredProvider(pn, cfgData, dataCache,
>> pkg_pn = None, item = None): preferred_v =
>> cfgData.getVar("PREFERRED_VERSION")
>> if preferred_v:
>> - m = re.match('(\d+:)*(.*)(_.*)*', preferred_v)
>> + m = re.match(r'(\d+:)*(.*)(_.*)*', preferred_v)
>> if m:
>> if m.group(1):
>> preferred_e = m.group(1)[:-1]
>> @@ -384,7 +372,7 @@ def getRuntimeProviders(dataCache, rdepend):
>>
>> # Only search dynamic packages if we can't find anything in
>> other variables for pattern in dataCache.packages_dynamic:
>> - pattern = pattern.replace('+', "\+")
>> + pattern = pattern.replace(r'+', r"\+")
>> if pattern in regexp_cache:
>> regexp = regexp_cache[pattern]
>> else:
>> diff --git a/bitbake/lib/bb/pysh/builtin.py
>> b/bitbake/lib/bb/pysh/builtin.py deleted file mode 100644
>> index a8814dc..0000000
>> --- a/bitbake/lib/bb/pysh/builtin.py
>> +++ /dev/null
>> @@ -1,710 +0,0 @@
>> -# builtin.py - builtins and utilities definitions for pysh.
>> -#
>> -# Copyright 2007 Patrick Mezard
>> -#
>> -# This software may be used and distributed according to the terms
>> -# of the GNU General Public License, incorporated herein by
>> reference. -
>> -"""Builtin and internal utilities implementations.
>> -
>> -- Beware not to use python interpreter environment as if it were the
>> shell -environment. For instance, commands working directory must be
>> explicitely handled -through env['PWD'] instead of relying on python
>> working directory. -"""
>> -import errno
>> -import optparse
>> -import os
>> -import re
>> -import subprocess
>> -import sys
>> -import time
>> -
>> -def has_subprocess_bug():
>> - return getattr(subprocess, 'list2cmdline') and \
>> - ( subprocess.list2cmdline(['']) == '' or \
>> - subprocess.list2cmdline(['foo|bar']) == 'foo|bar')
>> -
>> -# Detect python bug 1634343: "subprocess swallows empty arguments
>> under win32" -#
>> <http://sourceforge.net/tracker/index.php?func=detail&aid=1634343&group_id=5470&atid=105470>
>> -# Also detect: "[ 1710802 ] subprocess must escape redirection
>> characters under win32" -#
>> <http://sourceforge.net/tracker/index.php?func=detail&aid=1710802&group_id=5470&atid=105470>
>> -if has_subprocess_bug():
>> - import subprocess_fix
>> - subprocess.list2cmdline = subprocess_fix.list2cmdline
>> -
>> -from sherrors import *
>> -
>> -class NonExitingParser(optparse.OptionParser):
>> - """OptionParser default behaviour upon error is to print the
>> error message and
>> - exit. Raise a utility error instead.
>> - """
>> - def error(self, msg):
>> - raise UtilityError(msg)
>> -
>> -#-------------------------------------------------------------------------------
>> -# set special builtin
>> -#-------------------------------------------------------------------------------
>> -OPT_SET = NonExitingParser(usage="set - set or unset options and
>> positional parameters") -OPT_SET.add_option( '-f',
>> action='store_true', dest='has_f', default=False,
>> - help='The shell shall disable pathname expansion.')
>> -OPT_SET.add_option('-e', action='store_true', dest='has_e',
>> default=False,
>> - help="""When this option is on, if a simple command fails for
>> any of the \
>> - reasons listed in Consequences of Shell Errors or returns an
>> exit status \
>> - value >0, and is not part of the compound list following a
>> while, until, \
>> - or if keyword, and is not a part of an AND or OR list, and is
>> not a \
>> - pipeline preceded by the ! reserved word, then the shell shall
>> immediately \
>> - exit.""")
>> -OPT_SET.add_option('-x', action='store_true', dest='has_x',
>> default=False,
>> - help="""The shell shall write to standard error a trace for each
>> command \
>> - after it expands the command and before it executes it. It is
>> unspecified \
>> - whether the command that turns tracing off is traced.""")
>> -
>> -def builtin_set(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n') -
>> - option, args = OPT_SET.parse_args(args)
>> - env = interp.get_env()
>> -
>> - if option.has_f:
>> - env.set_opt('-f')
>> - if option.has_e:
>> - env.set_opt('-e')
>> - if option.has_x:
>> - env.set_opt('-x')
>> - return 0
>> -
>> -#-------------------------------------------------------------------------------
>> -# shift special builtin
>> -#-------------------------------------------------------------------------------
>> -def builtin_shift(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> -
>> - params = interp.get_env().get_positional_args()
>> - if args:
>> - try:
>> - n = int(args[0])
>> - if n > len(params):
>> - raise ValueError()
>> - except ValueError:
>> - return 1
>> - else:
>> - n = 1
>> -
>> - params[:n] = []
>> - interp.get_env().set_positional_args(params)
>> - return 0
>> -
>> -#-------------------------------------------------------------------------------
>> -# export special builtin
>> -#-------------------------------------------------------------------------------
>> -OPT_EXPORT = NonExitingParser(usage="set - set or unset options and
>> positional parameters") -OPT_EXPORT.add_option('-p',
>> action='store_true', dest='has_p', default=False) -
>> -def builtin_export(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> -
>> - option, args = OPT_EXPORT.parse_args(args)
>> - if option.has_p:
>> - raise NotImplementedError()
>> -
>> - for arg in args:
>> - try:
>> - name, value = arg.split('=', 1)
>> - except ValueError:
>> - name, value = arg, None
>> - env = interp.get_env().export(name, value)
>> -
>> - return 0
>> -
>> -#-------------------------------------------------------------------------------
>> -# return special builtin
>> -#-------------------------------------------------------------------------------
>> -def builtin_return(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> - res = 0
>> - if args:
>> - try:
>> - res = int(args[0])
>> - except ValueError:
>> - res = 0
>> - if not 0<=res<=255:
>> - res = 0
>> -
>> - # BUG: should be last executed command exit code
>> - raise ReturnSignal(res)
>> -
>> -#-------------------------------------------------------------------------------
>> -# trap special builtin
>> -#-------------------------------------------------------------------------------
>> -def builtin_trap(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> - if len(args) < 2:
>> - stderr.write('trap: usage: trap [[arg] signal_spec ...]\n')
>> - return 2
>> -
>> - action = args[0]
>> - for sig in args[1:]:
>> - try:
>> - env.traps[sig] = action
>> - except Exception as e:
>> - stderr.write('trap: %s\n' % str(e))
>> - return 0
>> -
>> -#-------------------------------------------------------------------------------
>> -# unset special builtin
>> -#-------------------------------------------------------------------------------
>> -OPT_UNSET = NonExitingParser("unset - unset values and attributes of
>> variables and functions") -OPT_UNSET.add_option( '-f',
>> action='store_true', dest='has_f', default=False)
>> -OPT_UNSET.add_option( '-v', action='store_true', dest='has_v',
>> default=False) - -def builtin_unset(name, args, interp, env, stdin,
>> stdout, stderr, debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> -
>> - option, args = OPT_UNSET.parse_args(args)
>> -
>> - status = 0
>> - env = interp.get_env()
>> - for arg in args:
>> - try:
>> - if option.has_f:
>> - env.remove_function(arg)
>> - else:
>> - del env[arg]
>> - except KeyError:
>> - pass
>> - except VarAssignmentError:
>> - status = 1
>> -
>> - return status
>> -
>> -#-------------------------------------------------------------------------------
>> -# wait special builtin
>> -#-------------------------------------------------------------------------------
>> -def builtin_wait(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n') -
>> - return interp.wait([int(arg) for arg in args])
>> -
>> -#-------------------------------------------------------------------------------
>> -# cat utility
>> -#-------------------------------------------------------------------------------
>> -def utility_cat(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n') -
>> - if not args:
>> - args = ['-']
>> -
>> - status = 0
>> - for arg in args:
>> - if arg == '-':
>> - data = stdin.read()
>> - else:
>> - path = os.path.join(env['PWD'], arg)
>> - try:
>> - f = file(path, 'rb')
>> - try:
>> - data = f.read()
>> - finally:
>> - f.close()
>> - except IOError as e:
>> - if e.errno != errno.ENOENT:
>> - raise
>> - status = 1
>> - continue
>> - stdout.write(data)
>> - stdout.flush()
>> - return status
>> -
>> -#-------------------------------------------------------------------------------
>> -# cd utility
>> -#-------------------------------------------------------------------------------
>> -OPT_CD = NonExitingParser("cd - change the working directory")
>> -
>> -def utility_cd(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n') -
>> - option, args = OPT_CD.parse_args(args)
>> - env = interp.get_env()
>> -
>> - directory = None
>> - printdir = False
>> - if not args:
>> - home = env.get('HOME')
>> - if home:
>> - # Unspecified, do nothing
>> - return 0
>> - else:
>> - directory = home
>> - elif len(args)==1:
>> - directory = args[0]
>> - if directory=='-':
>> - if 'OLDPWD' not in env:
>> - raise UtilityError("OLDPWD not set")
>> - printdir = True
>> - directory = env['OLDPWD']
>> - else:
>> - raise UtilityError("too many arguments")
>> -
>> - curpath = None
>> - # Absolute directories will be handled correctly by the
>> os.path.join call.
>> - if not directory.startswith('.') and not
>> directory.startswith('..'):
>> - cdpaths = env.get('CDPATH', '.').split(';')
>> - for cdpath in cdpaths:
>> - p = os.path.join(cdpath, directory)
>> - if os.path.isdir(p):
>> - curpath = p
>> - break
>> -
>> - if curpath is None:
>> - curpath = directory
>> - curpath = os.path.join(env['PWD'], directory)
>> -
>> - env['OLDPWD'] = env['PWD']
>> - env['PWD'] = curpath
>> - if printdir:
>> - stdout.write('%s\n' % curpath)
>> - return 0
>> -
>> -#-------------------------------------------------------------------------------
>> -# colon utility
>> -#-------------------------------------------------------------------------------
>> -def utility_colon(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> - return 0
>> -
>> -#-------------------------------------------------------------------------------
>> -# echo utility
>> -#-------------------------------------------------------------------------------
>> -def utility_echo(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> -
>> - # Echo only takes arguments, no options. Use printf if you need
>> fancy stuff.
>> - output = ' '.join(args) + '\n'
>> - stdout.write(output)
>> - stdout.flush()
>> - return 0
>> -
>> -#-------------------------------------------------------------------------------
>> -# egrep utility
>> -#-------------------------------------------------------------------------------
>> -# egrep is usually a shell script.
>> -# Unfortunately, pysh does not support shell scripts *with
>> arguments* right now, -# so the redirection is implemented here,
>> assuming grep is available. -def utility_egrep(name, args, interp,
>> env, stdin, stdout, stderr, debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> -
>> - return run_command('grep', ['-E'] + args, interp, env, stdin,
>> stdout,
>> - stderr, debugflags)
>> -
>> -#-------------------------------------------------------------------------------
>> -# env utility
>> -#-------------------------------------------------------------------------------
>> -def utility_env(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> -
>> - if args and args[0]=='-i':
>> - raise NotImplementedError('env: -i option is not
>> implemented')
>> -
>> - i = 0
>> - for arg in args:
>> - if '=' not in arg:
>> - break
>> - # Update the current environment
>> - name, value = arg.split('=', 1)
>> - env[name] = value
>> - i += 1
>> -
>> - if args[i:]:
>> - # Find then execute the specified interpreter
>> - utility = env.find_in_path(args[i])
>> - if not utility:
>> - return 127
>> - args[i:i+1] = utility
>> - name = args[i]
>> - args = args[i+1:]
>> - try:
>> - return run_command(name, args, interp, env, stdin,
>> stdout, stderr,
>> - debugflags)
>> - except UtilityError:
>> - stderr.write('env: failed to execute %s' % '
>> '.join([name]+args))
>> - return 126
>> - else:
>> - for pair in env.get_variables().iteritems():
>> - stdout.write('%s=%s\n' % pair)
>> - return 0
>> -
>> -#-------------------------------------------------------------------------------
>> -# exit utility
>> -#-------------------------------------------------------------------------------
>> -def utility_exit(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> -
>> - res = None
>> - if args:
>> - try:
>> - res = int(args[0])
>> - except ValueError:
>> - res = None
>> - if not 0<=res<=255:
>> - res = None
>> -
>> - if res is None:
>> - # BUG: should be last executed command exit code
>> - res = 0
>> -
>> - raise ExitSignal(res)
>> -
>> -#-------------------------------------------------------------------------------
>> -# fgrep utility
>> -#-------------------------------------------------------------------------------
>> -# see egrep
>> -def utility_fgrep(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> -
>> - return run_command('grep', ['-F'] + args, interp, env, stdin,
>> stdout,
>> - stderr, debugflags)
>> -
>> -#-------------------------------------------------------------------------------
>> -# gunzip utility
>> -#-------------------------------------------------------------------------------
>> -# see egrep
>> -def utility_gunzip(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> -
>> - return run_command('gzip', ['-d'] + args, interp, env, stdin,
>> stdout,
>> - stderr, debugflags)
>> -
>> -#-------------------------------------------------------------------------------
>> -# kill utility
>> -#-------------------------------------------------------------------------------
>> -def utility_kill(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> -
>> - for arg in args:
>> - pid = int(arg)
>> - status = subprocess.call(['pskill', '/T', str(pid)],
>> - shell=True,
>> - stdout=subprocess.PIPE,
>> - stderr=subprocess.PIPE)
>> - # pskill is asynchronous, hence the stupid polling loop
>> - while 1:
>> - p = subprocess.Popen(['pslist', str(pid)],
>> - shell=True,
>> - stdout=subprocess.PIPE,
>> - stderr=subprocess.STDOUT)
>> - output = p.communicate()[0]
>> - if ('process %d was not' % pid) in output:
>> - break
>> - time.sleep(1)
>> - return status
>> -
>> -#-------------------------------------------------------------------------------
>> -# mkdir utility
>> -#-------------------------------------------------------------------------------
>> -OPT_MKDIR = NonExitingParser("mkdir - make directories.")
>> -OPT_MKDIR.add_option('-p', action='store_true', dest='has_p',
>> default=False) -
>> -def utility_mkdir(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> -
>> - # TODO: implement umask
>> - # TODO: implement proper utility error report
>> - option, args = OPT_MKDIR.parse_args(args)
>> - for arg in args:
>> - path = os.path.join(env['PWD'], arg)
>> - if option.has_p:
>> - try:
>> - os.makedirs(path)
>> - except IOError as e:
>> - if e.errno != errno.EEXIST:
>> - raise
>> - else:
>> - os.mkdir(path)
>> - return 0
>> -
>> -#-------------------------------------------------------------------------------
>> -# netstat utility
>> -#-------------------------------------------------------------------------------
>> -def utility_netstat(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - # Do you really expect me to implement netstat ?
>> - # This empty form is enough for Mercurial tests since it's
>> - # supposed to generate nothing upon success. Faking this test
>> - # is not a big deal either.
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> - return 0
>> -
>> -#-------------------------------------------------------------------------------
>> -# pwd utility
>> -#-------------------------------------------------------------------------------
>> -OPT_PWD = NonExitingParser("pwd - return working directory name")
>> -OPT_PWD.add_option('-L', action='store_true', dest='has_L',
>> default=True,
>> - help="""If the PWD environment variable contains an absolute
>> pathname of \
>> - the current directory that does not contain the filenames dot or
>> dot-dot, \
>> - pwd shall write this pathname to standard output. Otherwise, the
>> -L option \
>> - shall behave as the -P option.""")
>> -OPT_PWD.add_option('-P', action='store_true', dest='has_L',
>> default=False,
>> - help="""The absolute pathname written shall not contain
>> filenames that, in \
>> - the context of the pathname, refer to files of type symbolic
>> link.""") -
>> -def utility_pwd(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n') -
>> - option, args = OPT_PWD.parse_args(args)
>> - stdout.write('%s\n' % env['PWD'])
>> - return 0
>> -
>> -#-------------------------------------------------------------------------------
>> -# printf utility
>> -#-------------------------------------------------------------------------------
>> -RE_UNESCAPE = re.compile(r'(\\x[a-zA-Z0-9]{2}|\\[0-7]{1,3}|\\.)')
>> -
>> -def utility_printf(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> -
>> - def replace(m):
>> - assert m.group()
>> - g = m.group()[1:]
>> - if g.startswith('x'):
>> - return chr(int(g[1:], 16))
>> - if len(g) <= 3 and len([c for c in g if c in '01234567']) ==
>> len(g):
>> - # Yay, an octal number
>> - return chr(int(g, 8))
>> - return {
>> - 'a': '\a',
>> - 'b': '\b',
>> - 'f': '\f',
>> - 'n': '\n',
>> - 'r': '\r',
>> - 't': '\t',
>> - 'v': '\v',
>> - '\\': '\\',
>> - }.get(g)
>> -
>> - # Convert escape sequences
>> - format = re.sub(RE_UNESCAPE, replace, args[0])
>> - stdout.write(format % tuple(args[1:]))
>> - return 0
>> -
>> -#-------------------------------------------------------------------------------
>> -# true utility
>> -#-------------------------------------------------------------------------------
>> -def utility_true(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> - return 0
>> -
>> -#-------------------------------------------------------------------------------
>> -# sed utility
>> -#-------------------------------------------------------------------------------
>> -RE_SED = re.compile(r'^s(.).*\1[a-zA-Z]*$')
>> -
>> -# cygwin sed fails with some expressions when they do not end with a
>> single space. -# see unit tests for details. Interestingly, the same
>> expressions works perfectly -# in cygwin shell.
>> -def utility_sed(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> -
>> - # Scan pattern arguments and append a space if necessary
>> - for i in range(len(args)):
>> - if not RE_SED.search(args[i]):
>> - continue
>> - args[i] = args[i] + ' '
>> -
>> - return run_command(name, args, interp, env, stdin, stdout,
>> - stderr, debugflags)
>> -
>> -#-------------------------------------------------------------------------------
>> -# sleep utility
>> -#-------------------------------------------------------------------------------
>> -def utility_sleep(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> - time.sleep(int(args[0]))
>> - return 0
>> -
>> -#-------------------------------------------------------------------------------
>> -# sort utility
>> -#-------------------------------------------------------------------------------
>> -OPT_SORT = NonExitingParser("sort - sort, merge, or sequence check
>> text files") -
>> -def utility_sort(name, args, interp, env, stdin, stdout, stderr,
>> debugflags): -
>> - def sort(path):
>> - if path == '-':
>> - lines = stdin.readlines()
>> - else:
>> - try:
>> - f = file(path)
>> - try:
>> - lines = f.readlines()
>> - finally:
>> - f.close()
>> - except IOError as e:
>> - stderr.write(str(e) + '\n')
>> - return 1
>> -
>> - if lines and lines[-1][-1]!='\n':
>> - lines[-1] = lines[-1] + '\n'
>> - return lines
>> -
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> -
>> - option, args = OPT_SORT.parse_args(args)
>> - alllines = []
>> -
>> - if len(args)<=0:
>> - args += ['-']
>> -
>> - # Load all files lines
>> - curdir = os.getcwd()
>> - try:
>> - os.chdir(env['PWD'])
>> - for path in args:
>> - alllines += sort(path)
>> - finally:
>> - os.chdir(curdir)
>> -
>> - alllines.sort()
>> - for line in alllines:
>> - stdout.write(line)
>> - return 0
>> -
>> -#-------------------------------------------------------------------------------
>> -# hg utility
>> -#-------------------------------------------------------------------------------
>> -
>> -hgcommands = [
>> - 'add',
>> - 'addremove',
>> - 'commit', 'ci',
>> - 'debugrename',
>> - 'debugwalk',
>> - 'falabala', # Dummy command used in a mercurial test
>> - 'incoming',
>> - 'locate',
>> - 'pull',
>> - 'push',
>> - 'qinit',
>> - 'remove', 'rm',
>> - 'rename', 'mv',
>> - 'revert',
>> - 'showconfig',
>> - 'status', 'st',
>> - 'strip',
>> - ]
>> -
>> -def rewriteslashes(name, args):
>> - # Several hg commands output file paths, rewrite the separators
>> - if len(args) > 1 and name.lower().endswith('python') \
>> - and args[0].endswith('hg'):
>> - for cmd in hgcommands:
>> - if cmd in args[1:]:
>> - return True
>> -
>> - # svn output contains many paths with OS specific separators.
>> - # Normalize these to unix paths.
>> - base = os.path.basename(name)
>> - if base.startswith('svn'):
>> - return True
>> -
>> - return False
>> -
>> -def rewritehg(output):
>> - if not output:
>> - return output
>> - # Rewrite os specific messages
>> - output = output.replace(': The system cannot find the file
>> specified',
>> - ': No such file or directory')
>> - output = re.sub(': Access is denied.*$', ': Permission denied',
>> output)
>> - output = output.replace(': No connection could be made because
>> the target machine actively refused it',
>> - ': Connection refused')
>> - return output
>> -
>> -
>> -def run_command(name, args, interp, env, stdin, stdout,
>> - stderr, debugflags):
>> - # Execute the command
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n') -
>> - hgbin = interp.options().hgbinary
>> - ishg = hgbin and ('hg' in name or args and 'hg' in args[0])
>> - unixoutput = 'cygwin' in name or ishg
>> -
>> - exec_env = env.get_variables()
>> - try:
>> - # BUG: comparing file descriptor is clearly not a reliable
>> way to tell
>> - # whether they point on the same underlying object. But in
>> pysh limited
>> - # scope this is usually right, we do not expect complicated
>> redirections
>> - # besides usual 2>&1.
>> - # Still there is one case we have but cannot deal with is
>> when stdout
>> - # and stderr are redirected *by pysh caller*. This the
>> reason for the
>> - # --redirect pysh() option.
>> - # Now, we want to know they are the same because we
>> sometimes need to
>> - # transform the command output, mostly remove CR-LF to
>> ensure that
>> - # command output is unix-like. Cygwin utilies are a special
>> case because
>> - # they explicitely set their output streams to binary mode,
>> so we have
>> - # nothing to do. For all others commands, we have to guess
>> whether they
>> - # are sending text data, in which case the transformation
>> must be done.
>> - # Again, the NUL character test is unreliable but should be
>> enough for
>> - # hg tests.
>> - redirected = stdout.fileno()==stderr.fileno()
>> - if not redirected:
>> - p = subprocess.Popen([name] + args, cwd=env['PWD'],
>> env=exec_env,
>> - stdin=stdin, stdout=subprocess.PIPE,
>> stderr=subprocess.PIPE)
>> - else:
>> - p = subprocess.Popen([name] + args, cwd=env['PWD'],
>> env=exec_env,
>> - stdin=stdin, stdout=subprocess.PIPE,
>> stderr=subprocess.STDOUT)
>> - out, err = p.communicate()
>> - except WindowsError as e:
>> - raise UtilityError(str(e))
>> -
>> - if not unixoutput:
>> - def encode(s):
>> - if '\0' in s:
>> - return s
>> - return s.replace('\r\n', '\n')
>> - else:
>> - encode = lambda s: s
>> -
>> - if rewriteslashes(name, args):
>> - encode1_ = encode
>> - def encode(s):
>> - s = encode1_(s)
>> - s = s.replace('\\\\', '\\')
>> - s = s.replace('\\', '/')
>> - return s
>> -
>> - if ishg:
>> - encode2_ = encode
>> - def encode(s):
>> - return rewritehg(encode2_(s))
>> -
>> - stdout.write(encode(out))
>> - if not redirected:
>> - stderr.write(encode(err))
>> - return p.returncode
>> -
>> diff --git a/bitbake/lib/bb/pysh/interp.py
>> b/bitbake/lib/bb/pysh/interp.py deleted file mode 100644
>> index d14ecf3..0000000
>> --- a/bitbake/lib/bb/pysh/interp.py
>> +++ /dev/null
>> @@ -1,1367 +0,0 @@
>> -# interp.py - shell interpreter for pysh.
>> -#
>> -# Copyright 2007 Patrick Mezard
>> -#
>> -# This software may be used and distributed according to the terms
>> -# of the GNU General Public License, incorporated herein by
>> reference. -
>> -"""Implement the shell interpreter.
>> -
>> -Most references are made to "The Open Group Base Specifications
>> Issue 6".
>> -<http://www.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html>
>> -""" -# TODO: document the fact input streams must implement fileno()
>> so Popen will work correctly. -# it requires non-stdin stream to be
>> implemented as files. Still to be tested... -# DOC: pathsep is used
>> in PATH instead of ':'. Clearly, there are path syntax issues here.
>> -# TODO: stop command execution upon error. -# TODO: sort out the
>> filename/io_number mess. It should be possible to use filenames only.
>> -# TODO: review subshell implementation -# TODO: test environment
>> cloning for non-special builtins -# TODO: set -x should not rebuild
>> commands from tokens, assignments/redirections are lost -# TODO: unit
>> test for variable assignment -# TODO: test error management wrt error
>> type/utility type -# TODO: test for binary output everywhere
>> -# BUG: debug-parsing does not pass log file to PLY. Maybe a PLY
>> upgrade is necessary. -import base64
>> -import cPickle as pickle
>> -import errno
>> -import glob
>> -import os
>> -import re
>> -import subprocess
>> -import sys
>> -import tempfile
>> -
>> -try:
>> - s = set()
>> - del s
>> -except NameError:
>> - from Set import Set as set
>> -
>> -import builtin
>> -from sherrors import *
>> -import pyshlex
>> -import pyshyacc
>> -
>> -def mappend(func, *args, **kargs):
>> - """Like map but assume func returns a list. Returned lists are
>> merged into
>> - a single one.
>> - """
>> - return reduce(lambda a,b: a+b, map(func, *args, **kargs), [])
>> -
>> -class FileWrapper:
>> - """File object wrapper to ease debugging.
>> -
>> - Allow mode checking and implement file duplication through a
>> simple
>> - reference counting scheme. Not sure the latter is really useful
>> since
>> - only real file descriptors can be used.
>> - """
>> - def __init__(self, mode, file, close=True):
>> - if mode not in ('r', 'w', 'a'):
>> - raise IOError('invalid mode: %s' % mode)
>> - self._mode = mode
>> - self._close = close
>> - if isinstance(file, FileWrapper):
>> - if file._refcount[0] <= 0:
>> - raise IOError(0, 'Error')
>> - self._refcount = file._refcount
>> - self._refcount[0] += 1
>> - self._file = file._file
>> - else:
>> - self._refcount = [1]
>> - self._file = file
>> -
>> - def dup(self):
>> - return FileWrapper(self._mode, self, self._close)
>> -
>> - def fileno(self):
>> - """fileno() should be only necessary for input streams."""
>> - return self._file.fileno()
>> -
>> - def read(self, size=-1):
>> - if self._mode!='r':
>> - raise IOError(0, 'Error')
>> - return self._file.read(size)
>> -
>> - def readlines(self, *args, **kwargs):
>> - return self._file.readlines(*args, **kwargs)
>> -
>> - def write(self, s):
>> - if self._mode not in ('w', 'a'):
>> - raise IOError(0, 'Error')
>> - return self._file.write(s)
>> -
>> - def flush(self):
>> - self._file.flush()
>> -
>> - def close(self):
>> - if not self._refcount:
>> - return
>> - assert self._refcount[0] > 0
>> -
>> - self._refcount[0] -= 1
>> - if self._refcount[0] == 0:
>> - self._mode = 'c'
>> - if self._close:
>> - self._file.close()
>> - self._refcount = None
>> -
>> - def mode(self):
>> - return self._mode
>> -
>> - def __getattr__(self, name):
>> - if name == 'name':
>> - self.name = getattr(self._file, name)
>> - return self.name
>> - else:
>> - raise AttributeError(name)
>> -
>> - def __del__(self):
>> - self.close()
>> -
>> -
>> -def win32_open_devnull(mode):
>> - return open('NUL', mode)
>> -
>> -
>> -class Redirections:
>> - """Stores open files and their mapping to pseudo-sh file
>> descriptor.
>> - """
>> - # BUG: redirections are not handled correctly: 1>&3 2>&3 3>&4
>> does
>> - # not make 1 to redirect to 4
>> - def __init__(self, stdin=None, stdout=None, stderr=None):
>> - self._descriptors = {}
>> - if stdin is not None:
>> - self._add_descriptor(0, stdin)
>> - if stdout is not None:
>> - self._add_descriptor(1, stdout)
>> - if stderr is not None:
>> - self._add_descriptor(2, stderr)
>> -
>> - def add_here_document(self, interp, name, content,
>> io_number=None):
>> - if io_number is None:
>> - io_number = 0
>> -
>> - if name==pyshlex.unquote_wordtree(name):
>> - content = interp.expand_here_document(('TOKEN', content))
>> -
>> - # Write document content in a temporary file
>> - tmp = tempfile.TemporaryFile()
>> - try:
>> - tmp.write(content)
>> - tmp.flush()
>> - tmp.seek(0)
>> - self._add_descriptor(io_number, FileWrapper('r', tmp))
>> - except:
>> - tmp.close()
>> - raise
>> -
>> - def add(self, interp, op, filename, io_number=None):
>> - if op not in ('<', '>', '>|', '>>', '>&'):
>> - # TODO: add descriptor duplication and
>> here_documents
>> - raise RedirectionError('Unsupported redirection operator
>> "%s"' % op)
>> -
>> - if io_number is not None:
>> - io_number = int(io_number)
>> -
>> - if (op == '>&' and filename.isdigit()) or filename=='-':
>> - # No expansion for file descriptors, quote them if you
>> want a filename
>> - fullname = filename
>> - else:
>> - if filename.startswith('/'):
>> - # TODO: win32 kludge
>> - if filename=='/dev/null':
>> - fullname = 'NUL'
>> - else:
>> - # TODO: handle absolute pathnames, they are
>> unlikely to exist on the
>> - # current platform (win32 for instance).
>> - raise NotImplementedError()
>> - else:
>> - fullname = interp.expand_redirection(('TOKEN',
>> filename))
>> - if not fullname:
>> - raise RedirectionError('%s: ambiguous redirect'
>> % filename)
>> - # Build absolute path based on PWD
>> - fullname = os.path.join(interp.get_env()['PWD'],
>> fullname)
>> -
>> - if op=='<':
>> - return self._add_input_redirection(interp, fullname,
>> io_number)
>> - elif op in ('>', '>|'):
>> - clobber = ('>|'==op)
>> - return self._add_output_redirection(interp, fullname,
>> io_number, clobber)
>> - elif op=='>>':
>> - return self._add_output_appending(interp, fullname,
>> io_number)
>> - elif op=='>&':
>> - return self._dup_output_descriptor(fullname, io_number)
>> -
>> - def close(self):
>> - if self._descriptors is not None:
>> - for desc in self._descriptors.itervalues():
>> - desc.flush()
>> - desc.close()
>> - self._descriptors = None
>> -
>> - def stdin(self):
>> - return self._descriptors[0]
>> -
>> - def stdout(self):
>> - return self._descriptors[1]
>> -
>> - def stderr(self):
>> - return self._descriptors[2]
>> -
>> - def clone(self):
>> - clone = Redirections()
>> - for desc, fileobj in self._descriptors.iteritems():
>> - clone._descriptors[desc] = fileobj.dup()
>> - return clone
>> -
>> - def _add_output_redirection(self, interp, filename, io_number,
>> clobber):
>> - if io_number is None:
>> - # io_number default to standard output
>> - io_number = 1
>> -
>> - if not clobber and interp.get_env().has_opt('-C') and
>> os.path.isfile(filename):
>> - # File already exist in no-clobber mode, bail out
>> - raise RedirectionError('File "%s" already exists' %
>> filename)
>> -
>> - # Open and register
>> - self._add_file_descriptor(io_number, filename, 'w')
>> -
>> - def _add_output_appending(self, interp, filename, io_number):
>> - if io_number is None:
>> - io_number = 1
>> - self._add_file_descriptor(io_number, filename, 'a')
>> -
>> - def _add_input_redirection(self, interp, filename, io_number):
>> - if io_number is None:
>> - io_number = 0
>> - self._add_file_descriptor(io_number, filename, 'r')
>> -
>> - def _add_file_descriptor(self, io_number, filename, mode):
>> - try:
>> - if filename.startswith('/'):
>> - if filename=='/dev/null':
>> - f = win32_open_devnull(mode+'b')
>> - else:
>> - # TODO: handle absolute pathnames, they are
>> unlikely to exist on the
>> - # current platform (win32 for instance).
>> - raise NotImplementedError('cannot open absolute
>> path %s' % repr(filename))
>> - else:
>> - f = file(filename, mode+'b')
>> - except IOError as e:
>> - raise RedirectionError(str(e))
>> -
>> - wrapper = None
>> - try:
>> - wrapper = FileWrapper(mode, f)
>> - f = None
>> - self._add_descriptor(io_number, wrapper)
>> - except:
>> - if f: f.close()
>> - if wrapper: wrapper.close()
>> - raise
>> -
>> - def _dup_output_descriptor(self, source_fd, dest_fd):
>> - if source_fd is None:
>> - source_fd = 1
>> - self._dup_file_descriptor(source_fd, dest_fd, 'w')
>> -
>> - def _dup_file_descriptor(self, source_fd, dest_fd, mode):
>> - source_fd = int(source_fd)
>> - if source_fd not in self._descriptors:
>> - raise RedirectionError('"%s" is not a valid file
>> descriptor' % str(source_fd))
>> - source = self._descriptors[source_fd]
>> -
>> - if source.mode()!=mode:
>> - raise RedirectionError('Descriptor %s cannot be
>> duplicated in mode "%s"' % (str(source), mode))
>> -
>> - if dest_fd=='-':
>> - # Close the source descriptor
>> - del self._descriptors[source_fd]
>> - source.close()
>> - else:
>> - dest_fd = int(dest_fd)
>> - if dest_fd not in self._descriptors:
>> - raise RedirectionError('Cannot replace file
>> descriptor %s' % str(dest_fd))
>> -
>> - dest = self._descriptors[dest_fd]
>> - if dest.mode()!=mode:
>> - raise RedirectionError('Descriptor %s cannot be
>> cannot be redirected in mode "%s"' % (str(dest), mode))
>> -
>> - self._descriptors[dest_fd] = source.dup()
>> - dest.close()
>> -
>> - def _add_descriptor(self, io_number, file):
>> - io_number = int(io_number)
>> -
>> - if io_number in self._descriptors:
>> - # Close the current descriptor
>> - d = self._descriptors[io_number]
>> - del self._descriptors[io_number]
>> - d.close()
>> -
>> - self._descriptors[io_number] = file
>> -
>> - def __str__(self):
>> - names = [('%d=%r' % (k, getattr(v, 'name', None))) for k,v
>> - in self._descriptors.iteritems()]
>> - names = ','.join(names)
>> - return 'Redirections(%s)' % names
>> -
>> - def __del__(self):
>> - self.close()
>> -
>> -def cygwin_to_windows_path(path):
>> - """Turn /cygdrive/c/foo into c:/foo, or return path if it
>> - is not a cygwin path.
>> - """
>> - if not path.startswith('/cygdrive/'):
>> - return path
>> - path = path[len('/cygdrive/'):]
>> - path = path[:1] + ':' + path[1:]
>> - return path
>> -
>> -def win32_to_unix_path(path):
>> - if path is not None:
>> - path = path.replace('\\', '/')
>> - return path
>> -
>> -_RE_SHEBANG = re.compile(r'^\#!\s?([^\s]+)(?:\s([^\s]+))?')
>> -_SHEBANG_CMDS = {
>> - '/usr/bin/env': 'env',
>> - '/bin/sh': 'pysh',
>> - 'python': 'python',
>> -}
>> -
>> -def resolve_shebang(path, ignoreshell=False):
>> - """Return a list of arguments as shebang interpreter call or an
>> empty list
>> - if path does not refer to an executable script.
>> - See <http://www.opengroup.org/austin/docs/austin_51r2.txt>.
>> -
>> - ignoreshell - set to True to ignore sh shebangs. Return an empty
>> list instead.
>> - """
>> - try:
>> - f = file(path)
>> - try:
>> - # At most 80 characters in the first line
>> - header = f.read(80).splitlines()[0]
>> - finally:
>> - f.close()
>> -
>> - m = _RE_SHEBANG.search(header)
>> - if not m:
>> - return []
>> - cmd, arg = m.group(1,2)
>> - if os.path.isfile(cmd):
>> - # Keep this one, the hg script for instance contains a
>> weird windows
>> - # shebang referencing the current python install.
>> - cmdfile = os.path.basename(cmd).lower()
>> - if cmdfile == 'python.exe':
>> - cmd = 'python'
>> - pass
>> - elif cmd not in _SHEBANG_CMDS:
>> - raise CommandNotFound('Unknown interpreter "%s"
>> referenced in '\
>> - 'shebang' % header)
>> - cmd = _SHEBANG_CMDS.get(cmd)
>> - if cmd is None or (ignoreshell and cmd == 'pysh'):
>> - return []
>> - if arg is None:
>> - return [cmd, win32_to_unix_path(path)]
>> - return [cmd, arg, win32_to_unix_path(path)]
>> - except IOError as e:
>> - if e.errno!=errno.ENOENT and \
>> - (e.errno!=errno.EPERM and not os.path.isdir(path)): #
>> Opening a directory raises EPERM
>> - raise
>> - return []
>> -
>> -def win32_find_in_path(name, path):
>> - if isinstance(path, str):
>> - path = path.split(os.pathsep)
>> -
>> - exts = os.environ.get('PATHEXT', '').lower().split(os.pathsep)
>> - for p in path:
>> - p_name = os.path.join(p, name)
>> -
>> - prefix = resolve_shebang(p_name)
>> - if prefix:
>> - return prefix
>> -
>> - for ext in exts:
>> - p_name_ext = p_name + ext
>> - if os.path.exists(p_name_ext):
>> - return [win32_to_unix_path(p_name_ext)]
>> - return []
>> -
>> -class Traps(dict):
>> - def __setitem__(self, key, value):
>> - if key not in ('EXIT',):
>> - raise NotImplementedError()
>> - super(Traps, self).__setitem__(key, value)
>> -
>> -# IFS white spaces character class
>> -_IFS_WHITESPACES = (' ', '\t', '\n')
>> -
>> -class Environment:
>> - """Environment holds environment variables, export table,
>> function
>> - definitions and whatever is defined in 2.12 "Shell Execution
>> Environment",
>> - redirection excepted.
>> - """
>> - def __init__(self, pwd):
>> - self._opt = set() #Shell options
>> -
>> - self._functions = {}
>> - self._env = {'?': '0', '#': '0'}
>> - self._exported = set([
>> - 'HOME', 'IFS', 'PATH'
>> - ])
>> -
>> - # Set environment vars with side-effects
>> - self._ifs_ws = None # Set of IFS whitespace characters
>> - self._ifs_re = None # Regular expression used to split
>> between words using IFS classes
>> - self['IFS'] = ''.join(_IFS_WHITESPACES) #Default environment
>> values
>> - self['PWD'] = pwd
>> - self.traps = Traps()
>> -
>> - def clone(self, subshell=False):
>> - env = Environment(self['PWD'])
>> - env._opt = set(self._opt)
>> - for k,v in self.get_variables().iteritems():
>> - if k in self._exported:
>> - env.export(k,v)
>> - elif subshell:
>> - env[k] = v
>> -
>> - if subshell:
>> - env._functions = dict(self._functions)
>> -
>> - return env
>> -
>> - def __getitem__(self, key):
>> - if key in ('@', '*', '-', '$'):
>> - raise NotImplementedError('%s is not implemented' %
>> repr(key))
>> - return self._env[key]
>> -
>> - def get(self, key, defval=None):
>> - try:
>> - return self[key]
>> - except KeyError:
>> - return defval
>> -
>> - def __setitem__(self, key, value):
>> - if key=='IFS':
>> - # Update the whitespace/non-whitespace classes
>> - self._update_ifs(value)
>> - elif key=='PWD':
>> - pwd = os.path.abspath(value)
>> - if not os.path.isdir(pwd):
>> - raise VarAssignmentError('Invalid directory %s' %
>> value)
>> - value = pwd
>> - elif key in ('?', '!'):
>> - value = str(int(value))
>> - self._env[key] = value
>> -
>> - def __delitem__(self, key):
>> - if key in ('IFS', 'PWD', '?'):
>> - raise VarAssignmentError('%s cannot be unset' % key)
>> - del self._env[key]
>> -
>> - def __contains__(self, item):
>> - return item in self._env
>> -
>> - def set_positional_args(self, args):
>> - """Set the content of 'args' as positional argument from 1
>> to len(args).
>> - Return previous argument as a list of strings.
>> - """
>> - # Save and remove previous arguments
>> - prevargs = []
>> - for i in range(int(self._env['#'])):
>> - i = str(i+1)
>> - prevargs.append(self._env[i])
>> - del self._env[i]
>> - self._env['#'] = '0'
>> -
>> - #Set new ones
>> - for i,arg in enumerate(args):
>> - self._env[str(i+1)] = str(arg)
>> - self._env['#'] = str(len(args))
>> -
>> - return prevargs
>> -
>> - def get_positional_args(self):
>> - return [self._env[str(i+1)] for i in
>> range(int(self._env['#']))]
>> -
>> - def get_variables(self):
>> - return dict(self._env)
>> -
>> - def export(self, key, value=None):
>> - if value is not None:
>> - self[key] = value
>> - self._exported.add(key)
>> -
>> - def get_exported(self):
>> - return [(k,self._env.get(k)) for k in self._exported]
>> -
>> - def split_fields(self, word):
>> - if not self._ifs_ws or not word:
>> - return [word]
>> - return re.split(self._ifs_re, word)
>> -
>> - def _update_ifs(self, value):
>> - """Update the split_fields related variables when IFS
>> character set is
>> - changed.
>> - """
>> - # TODO: handle NULL IFS
>> -
>> - # Separate characters in whitespace and non-whitespace
>> - chars = set(value)
>> - ws = [c for c in chars if c in _IFS_WHITESPACES]
>> - nws = [c for c in chars if c not in _IFS_WHITESPACES]
>> -
>> - # Keep whitespaces in a string for left and right stripping
>> - self._ifs_ws = ''.join(ws)
>> -
>> - # Build a regexp to split fields
>> - trailing = '[' + ''.join([re.escape(c) for c in ws]) + ']'
>> - if nws:
>> - # First, the single non-whitespace occurence.
>> - nws = '[' + ''.join([re.escape(c) for c in nws]) + ']'
>> - nws = '(?:' + trailing + '*' + nws + trailing + '*' +
>> '|' + trailing + '+)'
>> - else:
>> - # Then mix all parts with quantifiers
>> - nws = trailing + '+'
>> - self._ifs_re = re.compile(nws)
>> -
>> - def has_opt(self, opt, val=None):
>> - return (opt, val) in self._opt
>> -
>> - def set_opt(self, opt, val=None):
>> - self._opt.add((opt, val))
>> -
>> - def find_in_path(self, name, pwd=False):
>> - path = self._env.get('PATH', '').split(os.pathsep)
>> - if pwd:
>> - path[:0] = [self['PWD']]
>> - if os.name == 'nt':
>> - return win32_find_in_path(name, self._env.get('PATH',
>> ''))
>> - else:
>> - raise NotImplementedError()
>> -
>> - def define_function(self, name, body):
>> - if not is_name(name):
>> - raise ShellSyntaxError('%s is not a valid function name'
>> % repr(name))
>> - self._functions[name] = body
>> -
>> - def remove_function(self, name):
>> - del self._functions[name]
>> -
>> - def is_function(self, name):
>> - return name in self._functions
>> -
>> - def get_function(self, name):
>> - return self._functions.get(name)
>> -
>> -
>> -name_charset =
>> 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_'
>> -name_charset = dict(zip(name_charset,name_charset))
>> -
>> -def match_name(s):
>> - """Return the length in characters of the longest prefix made of
>> name
>> - allowed characters in s.
>> - """
>> - for i,c in enumerate(s):
>> - if c not in name_charset:
>> - return s[:i]
>> - return s
>> -
>> -def is_name(s):
>> - return len([c for c in s if c not in name_charset])<=0
>> -
>> -def is_special_param(c):
>> - return len(c)==1 and c in ('@','*','#','?','-','$','!','0')
>> -
>> -def utility_not_implemented(name, *args, **kwargs):
>> - raise NotImplementedError('%s utility is not implemented' % name)
>> -
>> -
>> -class Utility:
>> - """Define utilities properties:
>> - func -- utility callable. See builtin module for utility samples.
>> - is_special -- see XCU 2.8.
>> - """
>> - def __init__(self, func, is_special=0):
>> - self.func = func
>> - self.is_special = bool(is_special)
>> -
>> -
>> -def encodeargs(args):
>> - def encodearg(s):
>> - lines = base64.encodestring(s)
>> - lines = [l.splitlines()[0] for l in lines]
>> - return ''.join(lines)
>> -
>> - s = pickle.dumps(args)
>> - return encodearg(s)
>> -
>> -def decodeargs(s):
>> - s = base64.decodestring(s)
>> - return pickle.loads(s)
>> -
>> -
>> -class GlobError(Exception):
>> - pass
>> -
>> -class Options:
>> - def __init__(self):
>> - # True if Mercurial operates with binary streams
>> - self.hgbinary = True
>> -
>> -class Interpreter:
>> - # Implementation is very basic: the execute() method just makes
>> a DFS on the
>> - # AST and execute nodes one by one. Nodes are tuple (name,obj)
>> where name
>> - # is a string identifier and obj the AST element returned by the
>> parser.
>> - #
>> - # Handler are named after the node identifiers.
>> - # TODO: check node names and remove the switch in execute with
>> some
>> - # dynamic getattr() call to find node handlers.
>> - """Shell interpreter.
>> -
>> - The following debugging flags can be passed:
>> - debug-parsing - enable PLY debugging.
>> - debug-tree - print the generated AST.
>> - debug-cmd - trace command execution before word expansion, plus
>> exit status.
>> - debug-utility - trace utility execution.
>> - """
>> -
>> - # List supported commands.
>> - COMMANDS = {
>> - 'cat': Utility(builtin.utility_cat,),
>> - 'cd': Utility(builtin.utility_cd,),
>> - ':': Utility(builtin.utility_colon,),
>> - 'echo': Utility(builtin.utility_echo),
>> - 'env': Utility(builtin.utility_env),
>> - 'exit': Utility(builtin.utility_exit),
>> - 'export': Utility(builtin.builtin_export,
>> is_special=1),
>> - 'egrep': Utility(builtin.utility_egrep),
>> - 'fgrep': Utility(builtin.utility_fgrep),
>> - 'gunzip': Utility(builtin.utility_gunzip),
>> - 'kill': Utility(builtin.utility_kill),
>> - 'mkdir': Utility(builtin.utility_mkdir),
>> - 'netstat': Utility(builtin.utility_netstat),
>> - 'printf': Utility(builtin.utility_printf),
>> - 'pwd': Utility(builtin.utility_pwd),
>> - 'return': Utility(builtin.builtin_return,
>> is_special=1),
>> - 'sed': Utility(builtin.utility_sed,),
>> - 'set': Utility(builtin.builtin_set,),
>> - 'shift': Utility(builtin.builtin_shift,),
>> - 'sleep': Utility(builtin.utility_sleep,),
>> - 'sort': Utility(builtin.utility_sort,),
>> - 'trap': Utility(builtin.builtin_trap,
>> is_special=1),
>> - 'true': Utility(builtin.utility_true),
>> - 'unset': Utility(builtin.builtin_unset,
>> is_special=1),
>> - 'wait': Utility(builtin.builtin_wait,
>> is_special=1),
>> - }
>> -
>> - def __init__(self, pwd, debugflags = [], env=None, redirs=None,
>> stdin=None,
>> - stdout=None, stderr=None, opts=Options()):
>> - self._env = env
>> - if self._env is None:
>> - self._env = Environment(pwd)
>> - self._children = {}
>> -
>> - self._redirs = redirs
>> - self._close_redirs = False
>> -
>> - if self._redirs is None:
>> - if stdin is None:
>> - stdin = sys.stdin
>> - if stdout is None:
>> - stdout = sys.stdout
>> - if stderr is None:
>> - stderr = sys.stderr
>> - stdin = FileWrapper('r', stdin, False)
>> - stdout = FileWrapper('w', stdout, False)
>> - stderr = FileWrapper('w', stderr, False)
>> - self._redirs = Redirections(stdin, stdout, stderr)
>> - self._close_redirs = True
>> -
>> - self._debugflags = list(debugflags)
>> - self._logfile = sys.stderr
>> - self._options = opts
>> -
>> - def close(self):
>> - """Must be called when the interpreter is no longer used."""
>> - script = self._env.traps.get('EXIT')
>> - if script:
>> - try:
>> - self.execute_script(script=script)
>> - except:
>> - pass
>> -
>> - if self._redirs is not None and self._close_redirs:
>> - self._redirs.close()
>> - self._redirs = None
>> -
>> - def log(self, s):
>> - self._logfile.write(s)
>> - self._logfile.flush()
>> -
>> - def __getitem__(self, key):
>> - return self._env[key]
>> -
>> - def __setitem__(self, key, value):
>> - self._env[key] = value
>> -
>> - def options(self):
>> - return self._options
>> -
>> - def redirect(self, redirs, ios):
>> - def add_redir(io):
>> - if isinstance(io, pyshyacc.IORedirect):
>> - redirs.add(self, io.op, io.filename, io.io_number)
>> - else:
>> - redirs.add_here_document(self, io.name, io.content,
>> io.io_number)
>> -
>> - map(add_redir, ios)
>> - return redirs
>> -
>> - def execute_script(self, script=None, ast=None, sourced=False,
>> - scriptpath=None):
>> - """If script is not None, parse the input. Otherwise takes
>> the supplied
>> - AST. Then execute the AST.
>> - Return the script exit status.
>> - """
>> - try:
>> - if scriptpath is not None:
>> - self._env['0'] = os.path.abspath(scriptpath)
>> -
>> - if script is not None:
>> - debug_parsing = ('debug-parsing' in
>> self._debugflags)
>> - cmds, script = pyshyacc.parse(script, True,
>> debug_parsing)
>> - if 'debug-tree' in self._debugflags:
>> - pyshyacc.print_commands(cmds, self._logfile)
>> - self._logfile.flush()
>> - else:
>> - cmds, script = ast, ''
>> -
>> - status = 0
>> - for cmd in cmds:
>> - try:
>> - status = self.execute(cmd)
>> - except ExitSignal as e:
>> - if sourced:
>> - raise
>> - status = int(e.args[0])
>> - return status
>> - except ShellError:
>> - self._env['?'] = 1
>> - raise
>> - if 'debug-utility' in self._debugflags or
>> 'debug-cmd' in self._debugflags:
>> - self.log('returncode ' + str(status)+ '\n')
>> - return status
>> - except CommandNotFound as e:
>> - print >>self._redirs.stderr, str(e)
>> - self._redirs.stderr.flush()
>> - # Command not found by non-interactive shell
>> - # return 127
>> - raise
>> - except RedirectionError as e:
>> - # TODO: should be handled depending on the utility status
>> - print >>self._redirs.stderr, str(e)
>> - self._redirs.stderr.flush()
>> - # Command not found by non-interactive shell
>> - # return 127
>> - raise
>> -
>> - def dotcommand(self, env, args):
>> - if len(args) < 1:
>> - raise ShellError('. expects at least one argument')
>> - path = args[0]
>> - if '/' not in path:
>> - found = env.find_in_path(args[0], True)
>> - if found:
>> - path = found[0]
>> - script = file(path).read()
>> - return self.execute_script(script=script, sourced=True)
>> -
>> - def execute(self, token, redirs=None):
>> - """Execute and AST subtree with supplied redirections
>> overriding default
>> - interpreter ones.
>> - Return the exit status.
>> - """
>> - if not token:
>> - return 0
>> -
>> - if redirs is None:
>> - redirs = self._redirs
>> -
>> - if isinstance(token, list):
>> - # Commands sequence
>> - res = 0
>> - for t in token:
>> - res = self.execute(t, redirs)
>> - return res
>> -
>> - type, value = token
>> - status = 0
>> - if type=='simple_command':
>> - redirs_copy = redirs.clone()
>> - try:
>> - # TODO: define and handle command return values
>> - # TODO: implement set -e
>> - status = self._execute_simple_command(value,
>> redirs_copy)
>> - finally:
>> - redirs_copy.close()
>> - elif type=='pipeline':
>> - status = self._execute_pipeline(value, redirs)
>> - elif type=='and_or':
>> - status = self._execute_and_or(value, redirs)
>> - elif type=='for_clause':
>> - status = self._execute_for_clause(value, redirs)
>> - elif type=='while_clause':
>> - status = self._execute_while_clause(value, redirs)
>> - elif type=='function_definition':
>> - status = self._execute_function_definition(value, redirs)
>> - elif type=='brace_group':
>> - status = self._execute_brace_group(value, redirs)
>> - elif type=='if_clause':
>> - status = self._execute_if_clause(value, redirs)
>> - elif type=='subshell':
>> - status = self.subshell(ast=value.cmds, redirs=redirs)
>> - elif type=='async':
>> - status = self._asynclist(value)
>> - elif type=='redirect_list':
>> - redirs_copy = self.redirect(redirs.clone(), value.redirs)
>> - try:
>> - status = self.execute(value.cmd, redirs_copy)
>> - finally:
>> - redirs_copy.close()
>> - else:
>> - raise NotImplementedError('Unsupported token type ' +
>> type) -
>> - if status < 0:
>> - status = 255
>> - return status
>> -
>> - def _execute_if_clause(self, if_clause, redirs):
>> - cond_status = self.execute(if_clause.cond, redirs)
>> - if cond_status==0:
>> - return self.execute(if_clause.if_cmds, redirs)
>> - else:
>> - return self.execute(if_clause.else_cmds, redirs)
>> -
>> - def _execute_brace_group(self, group, redirs):
>> - status = 0
>> - for cmd in group.cmds:
>> - status = self.execute(cmd, redirs)
>> - return status
>> -
>> - def _execute_function_definition(self, fundef, redirs):
>> - self._env.define_function(fundef.name, fundef.body)
>> - return 0
>> -
>> - def _execute_while_clause(self, while_clause, redirs):
>> - status = 0
>> - while 1:
>> - cond_status = 0
>> - for cond in while_clause.condition:
>> - cond_status = self.execute(cond, redirs)
>> -
>> - if cond_status:
>> - break
>> -
>> - for cmd in while_clause.cmds:
>> - status = self.execute(cmd, redirs)
>> -
>> - return status
>> -
>> - def _execute_for_clause(self, for_clause, redirs):
>> - if not is_name(for_clause.name):
>> - raise ShellSyntaxError('%s is not a valid name' %
>> repr(for_clause.name))
>> - items = mappend(self.expand_token, for_clause.items)
>> -
>> - status = 0
>> - for item in items:
>> - self._env[for_clause.name] = item
>> - for cmd in for_clause.cmds:
>> - status = self.execute(cmd, redirs)
>> - return status
>> -
>> - def _execute_and_or(self, or_and, redirs):
>> - res = self.execute(or_and.left, redirs)
>> - if (or_and.op=='&&' and res==0) or (or_and.op!='&&' and
>> res!=0):
>> - res = self.execute(or_and.right, redirs)
>> - return res
>> -
>> - def _execute_pipeline(self, pipeline, redirs):
>> - if len(pipeline.commands)==1:
>> - status = self.execute(pipeline.commands[0], redirs)
>> - else:
>> - # Execute all commands one after the other
>> - status = 0
>> - inpath, outpath = None, None
>> - try:
>> - # Commands inputs and outputs cannot really be
>> plugged as done
>> - # by a real shell. Run commands sequentially and
>> chain their
>> - # input/output throught temporary files.
>> - tmpfd, inpath = tempfile.mkstemp()
>> - os.close(tmpfd)
>> - tmpfd, outpath = tempfile.mkstemp()
>> - os.close(tmpfd)
>> -
>> - inpath = win32_to_unix_path(inpath)
>> - outpath = win32_to_unix_path(outpath)
>> -
>> - for i, cmd in enumerate(pipeline.commands):
>> - call_redirs = redirs.clone()
>> - try:
>> - if i!=0:
>> - call_redirs.add(self, '<', inpath)
>> - if i!=len(pipeline.commands)-1:
>> - call_redirs.add(self, '>', outpath)
>> -
>> - status = self.execute(cmd, call_redirs)
>> -
>> - # Chain inputs/outputs
>> - inpath, outpath = outpath, inpath
>> - finally:
>> - call_redirs.close()
>> - finally:
>> - if inpath: os.remove(inpath)
>> - if outpath: os.remove(outpath)
>> -
>> - if pipeline.reverse_status:
>> - status = int(not status)
>> - self._env['?'] = status
>> - return status
>> -
>> - def _execute_function(self, name, args, interp, env, stdin,
>> stdout, stderr, *others):
>> - assert interp is self
>> -
>> - func = env.get_function(name)
>> - #Set positional parameters
>> - prevargs = None
>> - try:
>> - prevargs = env.set_positional_args(args)
>> - try:
>> - redirs = Redirections(stdin.dup(), stdout.dup(),
>> stderr.dup())
>> - try:
>> - status = self.execute(func, redirs)
>> - finally:
>> - redirs.close()
>> - except ReturnSignal as e:
>> - status = int(e.args[0])
>> - env['?'] = status
>> - return status
>> - finally:
>> - #Reset positional parameters
>> - if prevargs is not None:
>> - env.set_positional_args(prevargs)
>> -
>> - def _execute_simple_command(self, token, redirs):
>> - """Can raise ReturnSignal when return builtin is called,
>> ExitSignal when
>> - exit is called, and other shell exceptions upon builtin
>> failures.
>> - """
>> - debug_command = 'debug-cmd' in self._debugflags
>> - if debug_command:
>> - self.log('word' + repr(token.words) + '\n')
>> - self.log('assigns' + repr(token.assigns) + '\n')
>> - self.log('redirs' + repr(token.redirs) + '\n')
>> -
>> - is_special = None
>> - env = self._env
>> -
>> - try:
>> - # Word expansion
>> - args = []
>> - for word in token.words:
>> - args += self.expand_token(word)
>> - if is_special is None and args:
>> - is_special = env.is_function(args[0]) or \
>> - (args[0] in self.COMMANDS and
>> self.COMMANDS[args[0]].is_special)
>> -
>> - if debug_command:
>> - self.log('_execute_simple_command' + str(args) +
>> '\n')
>> -
>> - if not args:
>> - # Redirections happen is a subshell
>> - redirs = redirs.clone()
>> - elif not is_special:
>> - env = self._env.clone()
>> -
>> - # Redirections
>> - self.redirect(redirs, token.redirs)
>> -
>> - # Variables assignments
>> - res = 0
>> - for type,(k,v) in token.assigns:
>> - status, expanded = self.expand_variable((k,v))
>> - if status is not None:
>> - res = status
>> - if args:
>> - env.export(k, expanded)
>> - else:
>> - env[k] = expanded
>> -
>> - if args and args[0] in ('.', 'source'):
>> - res = self.dotcommand(env, args[1:])
>> - elif args:
>> - if args[0] in self.COMMANDS:
>> - command = self.COMMANDS[args[0]]
>> - elif env.is_function(args[0]):
>> - command = Utility(self._execute_function,
>> is_special=True)
>> - else:
>> - if not '/' in args[0].replace('\\', '/'):
>> - cmd = env.find_in_path(args[0])
>> - if not cmd:
>> - # TODO: test error code on unknown
>> command => 127
>> - raise CommandNotFound('Unknown command:
>> "%s"' % args[0])
>> - else:
>> - # Handle commands like '/cygdrive/c/foo.bat'
>> - cmd = cygwin_to_windows_path(args[0])
>> - if not os.path.exists(cmd):
>> - raise CommandNotFound('%s: No such file
>> or directory' % args[0])
>> - shebang = resolve_shebang(cmd)
>> - if shebang:
>> - cmd = shebang
>> - else:
>> - cmd = [cmd]
>> - args[0:1] = cmd
>> - command = Utility(builtin.run_command)
>> -
>> - # Command execution
>> - if 'debug-cmd' in self._debugflags:
>> - self.log('redirections ' + str(redirs) + '\n')
>> -
>> - res = command.func(args[0], args[1:], self, env,
>> - redirs.stdin(), redirs.stdout(),
>> - redirs.stderr(), self._debugflags)
>> -
>> - if self._env.has_opt('-x'):
>> - # Trace command execution in shell environment
>> - # BUG: would be hard to reproduce a real shell
>> behaviour since
>> - # the AST is not annotated with source lines/tokens.
>> - self._redirs.stdout().write(' '.join(args))
>> -
>> - except ReturnSignal:
>> - raise
>> - except ShellError as e:
>> - if is_special or isinstance(e, (ExitSignal,
>> - ShellSyntaxError,
>> ExpansionError)):
>> - raise e
>> - self._redirs.stderr().write(str(e)+'\n')
>> - return 1
>> -
>> - return res
>> -
>> - def expand_token(self, word):
>> - """Expand a word as specified in [2.6 Word Expansions].
>> Return the list
>> - of expanded words.
>> - """
>> - status, wtrees = self._expand_word(word)
>> - return map(pyshlex.wordtree_as_string, wtrees)
>> -
>> - def expand_variable(self, word):
>> - """Return a status code (or None if no command expansion
>> occurred)
>> - and a single word.
>> - """
>> - status, wtrees = self._expand_word(word, pathname=False,
>> split=False)
>> - words = map(pyshlex.wordtree_as_string, wtrees)
>> - assert len(words)==1
>> - return status, words[0]
>> -
>> - def expand_here_document(self, word):
>> - """Return the expanded document as a single word. The here
>> document is
>> - assumed to be unquoted.
>> - """
>> - status, wtrees = self._expand_word(word, pathname=False,
>> - split=False,
>> here_document=True)
>> - words = map(pyshlex.wordtree_as_string, wtrees)
>> - assert len(words)==1
>> - return words[0]
>> -
>> - def expand_redirection(self, word):
>> - """Return a single word."""
>> - return self.expand_variable(word)[1]
>> -
>> - def get_env(self):
>> - return self._env
>> -
>> - def _expand_word(self, token, pathname=True, split=True,
>> here_document=False):
>> - wtree = pyshlex.make_wordtree(token[1],
>> here_document=here_document)
>> -
>> - # TODO: implement tilde expansion
>> - def expand(wtree):
>> - """Return a pseudo wordtree: the tree or its subelements
>> can be empty
>> - lists when no value result from the expansion.
>> - """
>> - status = None
>> - for part in wtree:
>> - if not isinstance(part, list):
>> - continue
>> - if part[0]in ("'", '\\'):
>> - continue
>> - elif part[0] in ('`', '$('):
>> - status, result = self._expand_command(part)
>> - part[:] = result
>> - elif part[0] in ('$', '${'):
>> - part[:] = self._expand_parameter(part,
>> wtree[0]=='"', split)
>> - elif part[0] in ('', '"'):
>> - status, result = expand(part)
>> - part[:] = result
>> - else:
>> - raise NotImplementedError('%s expansion is not
>> implemented'
>> - % part[0])
>> - # [] is returned when an expansion result in no-field,
>> - # like an empty $@
>> - wtree = [p for p in wtree if p != []]
>> - if len(wtree) < 3:
>> - return status, []
>> - return status, wtree
>> -
>> - status, wtree = expand(wtree)
>> - if len(wtree) == 0:
>> - return status, wtree
>> - wtree = pyshlex.normalize_wordtree(wtree)
>> -
>> - if split:
>> - wtrees = self._split_fields(wtree)
>> - else:
>> - wtrees = [wtree]
>> -
>> - if pathname:
>> - wtrees = mappend(self._expand_pathname, wtrees)
>> -
>> - wtrees = map(self._remove_quotes, wtrees)
>> - return status, wtrees
>> -
>> - def _expand_command(self, wtree):
>> - # BUG: there is something to do with backslashes and quoted
>> - # characters here
>> - command = pyshlex.wordtree_as_string(wtree[1:-1])
>> - status, output = self.subshell_output(command)
>> - return status, ['', output, '']
>> -
>> - def _expand_parameter(self, wtree, quoted=False, split=False):
>> - """Return a valid wtree or an empty list when no parameter
>> results."""
>> - # Get the parameter name
>> - # TODO: implement weird expansion rules with ':'
>> - name = pyshlex.wordtree_as_string(wtree[1:-1])
>> - if not is_name(name) and not is_special_param(name):
>> - raise ExpansionError('Bad substitution "%s"' % name)
>> - # TODO: implement special parameters
>> - if name in ('@', '*'):
>> - args = self._env.get_positional_args()
>> - if len(args) == 0:
>> - return []
>> - if len(args)<2:
>> - return ['', ''.join(args), '']
>> -
>> - sep = self._env.get('IFS', '')[:1]
>> - if split and quoted and name=='@':
>> - # Introduce a new token to tell the caller that
>> these parameters
>> - # cause a split as specified in 2.5.2
>> - return ['@'] + args + ['']
>> - else:
>> - return ['', sep.join(args), '']
>> -
>> - return ['', self._env.get(name, ''), '']
>> -
>> - def _split_fields(self, wtree):
>> - def is_empty(split):
>> - return split==['', '', '']
>> -
>> - def split_positional(quoted):
>> - # Return a list of wtree split according positional
>> parameters rules.
>> - # All remaining '@' groups are removed.
>> - assert quoted[0]=='"'
>> -
>> - splits = [[]]
>> - for part in quoted:
>> - if not isinstance(part, list) or part[0]!='@':
>> - splits[-1].append(part)
>> - else:
>> - # Empty or single argument list were dealt with
>> already
>> - assert len(part)>3
>> - # First argument must join with the beginning
>> part of the original word
>> - splits[-1].append(part[1])
>> - # Create double-quotes expressions for every
>> argument after the first
>> - for arg in part[2:-1]:
>> - splits[-1].append('"')
>> - splits.append(['"', arg])
>> - return splits
>> -
>> - # At this point, all expansions but pathnames have occured.
>> Only quoted
>> - # and positional sequences remain. Thus, all candidates for
>> field splitting
>> - # are in the tree root, or are positional splits ('@') and
>> lie in root
>> - # children.
>> - if not wtree or wtree[0] not in ('', '"'):
>> - # The whole token is quoted or empty, nothing to split
>> - return [wtree]
>> -
>> - if wtree[0]=='"':
>> - wtree = ['', wtree, '']
>> -
>> - result = [['', '']]
>> - for part in wtree[1:-1]:
>> - if isinstance(part, list):
>> - if part[0]=='"':
>> - splits = split_positional(part)
>> - if len(splits)<=1:
>> - result[-1] += [part, '']
>> - else:
>> - # Terminate the current split
>> - result[-1] += [splits[0], '']
>> - result += splits[1:-1]
>> - # Create a new split
>> - result += [['', splits[-1], '']]
>> - else:
>> - result[-1] += [part, '']
>> - else:
>> - splits = self._env.split_fields(part)
>> - if len(splits)<=1:
>> - # No split
>> - result[-1][-1] += part
>> - else:
>> - # Terminate the current resulting part and
>> create a new one
>> - result[-1][-1] += splits[0]
>> - result[-1].append('')
>> - result += [['', r, ''] for r in splits[1:-1]]
>> - result += [['', splits[-1]]]
>> - result[-1].append('')
>> -
>> - # Leading and trailing empty groups come from
>> leading/trailing blanks
>> - if result and is_empty(result[-1]):
>> - result[-1:] = []
>> - if result and is_empty(result[0]):
>> - result[:1] = []
>> - return result
>> -
>> - def _expand_pathname(self, wtree):
>> - """See [2.6.6 Pathname Expansion]."""
>> - if self._env.has_opt('-f'):
>> - return [wtree]
>> -
>> - # All expansions have been performed, only quoted sequences
>> should remain
>> - # in the tree. Generate the pattern by folding the tree,
>> escaping special
>> - # characters when appear quoted
>> - special_chars = '*?[]'
>> -
>> - def make_pattern(wtree):
>> - subpattern = []
>> - for part in wtree[1:-1]:
>> - if isinstance(part, list):
>> - part = make_pattern(part)
>> - elif wtree[0]!='':
>> - for c in part:
>> - # Meta-characters cannot be quoted
>> - if c in special_chars:
>> - raise GlobError()
>> - subpattern.append(part)
>> - return ''.join(subpattern)
>> -
>> - def pwd_glob(pattern):
>> - cwd = os.getcwd()
>> - os.chdir(self._env['PWD'])
>> - try:
>> - return glob.glob(pattern)
>> - finally:
>> - os.chdir(cwd)
>> -
>> - #TODO: check working directory issues here wrt relative
>> patterns
>> - try:
>> - pattern = make_pattern(wtree)
>> - paths = pwd_glob(pattern)
>> - except GlobError:
>> - # BUG: Meta-characters were found in quoted sequences.
>> The should
>> - # have been used literally but this is unsupported in
>> current glob module.
>> - # Instead we consider the whole tree must be used
>> literally and
>> - # therefore there is no point in globbing. This is wrong
>> when meta
>> - # characters are mixed with quoted meta in the same
>> pattern like:
>> - # < foo*"py*" >
>> - paths = []
>> -
>> - if not paths:
>> - return [wtree]
>> - return [['', path, ''] for path in paths]
>> -
>> - def _remove_quotes(self, wtree):
>> - """See [2.6.7 Quote Removal]."""
>> -
>> - def unquote(wtree):
>> - unquoted = []
>> - for part in wtree[1:-1]:
>> - if isinstance(part, list):
>> - part = unquote(part)
>> - unquoted.append(part)
>> - return ''.join(unquoted)
>> -
>> - return ['', unquote(wtree), '']
>> -
>> - def subshell(self, script=None, ast=None, redirs=None):
>> - """Execute the script or AST in a subshell, with inherited
>> redirections
>> - if redirs is not None.
>> - """
>> - if redirs:
>> - sub_redirs = redirs
>> - else:
>> - sub_redirs = redirs.clone()
>> -
>> - subshell = None
>> - try:
>> - subshell = Interpreter(None, self._debugflags,
>> self._env.clone(True),
>> - sub_redirs, opts=self._options)
>> - return subshell.execute_script(script, ast)
>> - finally:
>> - if not redirs: sub_redirs.close()
>> - if subshell: subshell.close()
>> -
>> - def subshell_output(self, script):
>> - """Execute the script in a subshell and return the captured
>> output."""
>> - # Create temporary file to capture subshell output
>> - tmpfd, tmppath = tempfile.mkstemp()
>> - try:
>> - tmpfile = os.fdopen(tmpfd, 'wb')
>> - stdout = FileWrapper('w', tmpfile)
>> -
>> - redirs = Redirections(self._redirs.stdin().dup(),
>> - stdout,
>> -
>> self._redirs.stderr().dup())
>> - try:
>> - status = self.subshell(script=script, redirs=redirs)
>> - finally:
>> - redirs.close()
>> - redirs = None
>> -
>> - # Extract subshell standard output
>> - tmpfile = open(tmppath, 'rb')
>> - try:
>> - output = tmpfile.read()
>> - return status, output.rstrip('\n')
>> - finally:
>> - tmpfile.close()
>> - finally:
>> - os.remove(tmppath)
>> -
>> - def _asynclist(self, cmd):
>> - args = (self._env.get_variables(), cmd)
>> - arg = encodeargs(args)
>> - assert len(args) < 30*1024
>> - cmd = ['pysh.bat', '--ast', '-c', arg]
>> - p = subprocess.Popen(cmd, cwd=self._env['PWD'])
>> - self._children[p.pid] = p
>> - self._env['!'] = p.pid
>> - return 0
>> -
>> - def wait(self, pids=None):
>> - if not pids:
>> - pids = self._children.keys()
>> -
>> - status = 127
>> - for pid in pids:
>> - if pid not in self._children:
>> - continue
>> - p = self._children.pop(pid)
>> - status = p.wait()
>> -
>> - return status
>> -
>> diff --git a/bitbake/lib/bb/pysh/lsprof.py
>> b/bitbake/lib/bb/pysh/lsprof.py deleted file mode 100644
>> index b1831c2..0000000
>> --- a/bitbake/lib/bb/pysh/lsprof.py
>> +++ /dev/null
>> @@ -1,116 +0,0 @@
>> -#! /usr/bin/env python
>> -
>> -import sys
>> -from _lsprof import Profiler, profiler_entry
>> -
>> -__all__ = ['profile', 'Stats']
>> -
>> -def profile(f, *args, **kwds):
>> - """XXX docstring"""
>> - p = Profiler()
>> - p.enable(subcalls=True, builtins=True)
>> - try:
>> - f(*args, **kwds)
>> - finally:
>> - p.disable()
>> - return Stats(p.getstats())
>> -
>> -
>> -class Stats(object):
>> - """XXX docstring"""
>> -
>> - def __init__(self, data):
>> - self.data = data
>> -
>> - def sort(self, crit="inlinetime"):
>> - """XXX docstring"""
>> - if crit not in profiler_entry.__dict__:
>> - raise ValueError("Can't sort by %s" % crit)
>> - self.data.sort(lambda b, a: cmp(getattr(a, crit),
>> - getattr(b, crit)))
>> - for e in self.data:
>> - if e.calls:
>> - e.calls.sort(lambda b, a: cmp(getattr(a, crit),
>> - getattr(b, crit)))
>> -
>> - def pprint(self, top=None, file=None, limit=None, climit=None):
>> - """XXX docstring"""
>> - if file is None:
>> - file = sys.stdout
>> - d = self.data
>> - if top is not None:
>> - d = d[:top]
>> - cols = "% 12s %12s %11.4f %11.4f %s\n"
>> - hcols = "% 12s %12s %12s %12s %s\n"
>> - cols2 = "+%12s %12s %11.4f %11.4f + %s\n"
>> - file.write(hcols % ("CallCount", "Recursive", "Total(ms)",
>> - "Inline(ms)", "module:lineno(function)"))
>> - count = 0
>> - for e in d:
>> - file.write(cols % (e.callcount, e.reccallcount,
>> e.totaltime,
>> - e.inlinetime, label(e.code)))
>> - count += 1
>> - if limit is not None and count == limit:
>> - return
>> - ccount = 0
>> - if e.calls:
>> - for se in e.calls:
>> - file.write(cols % ("+%s" % se.callcount,
>> se.reccallcount,
>> - se.totaltime, se.inlinetime,
>> - "+%s" % label(se.code)))
>> - count += 1
>> - ccount += 1
>> - if limit is not None and count == limit:
>> - return
>> - if climit is not None and ccount == climit:
>> - break
>> -
>> - def freeze(self):
>> - """Replace all references to code objects with string
>> - descriptions; this makes it possible to pickle the
>> instance.""" -
>> - # this code is probably rather ickier than it needs to be!
>> - for i in range(len(self.data)):
>> - e = self.data[i]
>> - if not isinstance(e.code, str):
>> - self.data[i] = type(e)((label(e.code),) + e[1:])
>> - if e.calls:
>> - for j in range(len(e.calls)):
>> - se = e.calls[j]
>> - if not isinstance(se.code, str):
>> - e.calls[j] = type(se)((label(se.code),) +
>> se[1:]) -
>> -_fn2mod = {}
>> -
>> -def label(code):
>> - if isinstance(code, str):
>> - return code
>> - try:
>> - mname = _fn2mod[code.co_filename]
>> - except KeyError:
>> - for k, v in sys.modules.items():
>> - if v is None:
>> - continue
>> - if not hasattr(v, '__file__'):
>> - continue
>> - if not isinstance(v.__file__, str):
>> - continue
>> - if v.__file__.startswith(code.co_filename):
>> - mname = _fn2mod[code.co_filename] = k
>> - break
>> - else:
>> - mname = _fn2mod[code.co_filename] =
>> '<%s>'%code.co_filename -
>> - return '%s:%d(%s)' % (mname, code.co_firstlineno, code.co_name)
>> -
>> -
>> -if __name__ == '__main__':
>> - import os
>> - sys.argv = sys.argv[1:]
>> - if not sys.argv:
>> - print >> sys.stderr, "usage: lsprof.py <script>
>> <arguments...>"
>> - sys.exit(2)
>> - sys.path.insert(0, os.path.abspath(os.path.dirname(sys.argv[0])))
>> - stats = profile(execfile, sys.argv[0], globals(), locals())
>> - stats.sort()
>> - stats.pprint()
>> diff --git a/bitbake/lib/bb/pysh/pysh.py b/bitbake/lib/bb/pysh/pysh.py
>> deleted file mode 100644
>> index b4e6145..0000000
>> --- a/bitbake/lib/bb/pysh/pysh.py
>> +++ /dev/null
>> @@ -1,167 +0,0 @@
>> -# pysh.py - command processing for pysh.
>> -#
>> -# Copyright 2007 Patrick Mezard
>> -#
>> -# This software may be used and distributed according to the terms
>> -# of the GNU General Public License, incorporated herein by
>> reference. -
>> -import optparse
>> -import os
>> -import sys
>> -
>> -import interp
>> -
>> -SH_OPT = optparse.OptionParser(prog='pysh', usage="%prog [OPTIONS]",
>> version='0.1') -SH_OPT.add_option('-c', action='store_true',
>> dest='command_string', default=None,
>> - help='A string that shall be interpreted by the shell as one or
>> more commands') -SH_OPT.add_option('--redirect-to',
>> dest='redirect_to', default=None,
>> - help='Redirect script commands stdout and stderr to the
>> specified file') -# See utility_command in builtin.py about the
>> reason for this flag. -SH_OPT.add_option('--redirected',
>> dest='redirected', action='store_true', default=False,
>> - help='Tell the interpreter that stdout and stderr are actually
>> the same objects, which is really stdout')
>> -SH_OPT.add_option('--debug-parsing', action='store_true',
>> dest='debug_parsing', default=False,
>> - help='Trace PLY execution')
>> -SH_OPT.add_option('--debug-tree', action='store_true',
>> dest='debug_tree', default=False,
>> - help='Display the generated syntax tree.')
>> -SH_OPT.add_option('--debug-cmd', action='store_true',
>> dest='debug_cmd', default=False,
>> - help='Trace command execution before parameters expansion and
>> exit status.') -SH_OPT.add_option('--debug-utility',
>> action='store_true', dest='debug_utility', default=False,
>> - help='Trace utility calls, after parameters expansions')
>> -SH_OPT.add_option('--ast', action='store_true', dest='ast',
>> default=False,
>> - help='Encoded commands to execute in a subprocess')
>> -SH_OPT.add_option('--profile', action='store_true', default=False,
>> - help='Profile pysh run')
>> -
>> -
>> -def split_args(args):
>> - # Separate shell arguments from command ones
>> - # Just stop at the first argument not starting with a dash. I
>> know, this is completely broken,
>> - # it ignores files starting with a dash or may take option
>> values for command file. This is not
>> - # supposed to happen for now
>> - command_index = len(args)
>> - for i,arg in enumerate(args):
>> - if not arg.startswith('-'):
>> - command_index = i
>> - break
>> -
>> - return args[:command_index], args[command_index:]
>> -
>> -
>> -def fixenv(env):
>> - path = env.get('PATH')
>> - if path is not None:
>> - parts = path.split(os.pathsep)
>> - # Remove Windows utilities from PATH, they are useless at
>> best and
>> - # some of them (find) may be confused with other utilities.
>> - parts = [p for p in parts if 'system32' not in p.lower()]
>> - env['PATH'] = os.pathsep.join(parts)
>> - if env.get('HOME') is None:
>> - # Several utilities, including cvsps, cannot work without
>> - # a defined HOME directory.
>> - env['HOME'] = os.path.expanduser('~')
>> - return env
>> -
>> -def _sh(cwd, shargs, cmdargs, options, debugflags=None, env=None):
>> - if os.environ.get('PYSH_TEXT') != '1':
>> - import msvcrt
>> - for fp in (sys.stdin, sys.stdout, sys.stderr):
>> - msvcrt.setmode(fp.fileno(), os.O_BINARY)
>> -
>> - hgbin = os.environ.get('PYSH_HGTEXT') != '1'
>> -
>> - if debugflags is None:
>> - debugflags = []
>> - if options.debug_parsing:
>> debugflags.append('debug-parsing')
>> - if options.debug_utility:
>> debugflags.append('debug-utility')
>> - if options.debug_cmd: debugflags.append('debug-cmd')
>> - if options.debug_tree: debugflags.append('debug-tree')
>> -
>> - if env is None:
>> - env = fixenv(dict(os.environ))
>> - if cwd is None:
>> - cwd = os.getcwd()
>> -
>> - if not cmdargs:
>> - # Nothing to do
>> - return 0
>> -
>> - ast = None
>> - command_file = None
>> - if options.command_string:
>> - input = cmdargs[0]
>> - if not options.ast:
>> - input += '\n'
>> - else:
>> - args, input = interp.decodeargs(input), None
>> - env, ast = args
>> - cwd = env.get('PWD', cwd)
>> - else:
>> - command_file = cmdargs[0]
>> - arguments = cmdargs[1:]
>> -
>> - prefix = interp.resolve_shebang(command_file,
>> ignoreshell=True)
>> - if prefix:
>> - input = ' '.join(prefix + [command_file] + arguments)
>> - else:
>> - # Read commands from file
>> - f = file(command_file)
>> - try:
>> - # Trailing newline to help the parser
>> - input = f.read() + '\n'
>> - finally:
>> - f.close()
>> -
>> - redirect = None
>> - try:
>> - if options.redirected:
>> - stdout = sys.stdout
>> - stderr = stdout
>> - elif options.redirect_to:
>> - redirect = open(options.redirect_to, 'wb')
>> - stdout = redirect
>> - stderr = redirect
>> - else:
>> - stdout = sys.stdout
>> - stderr = sys.stderr
>> -
>> - # TODO: set arguments to environment variables
>> - opts = interp.Options()
>> - opts.hgbinary = hgbin
>> - ip = interp.Interpreter(cwd, debugflags, stdout=stdout,
>> stderr=stderr,
>> - opts=opts)
>> - try:
>> - # Export given environment in shell object
>> - for k,v in env.iteritems():
>> - ip.get_env().export(k,v)
>> - return ip.execute_script(input, ast,
>> scriptpath=command_file)
>> - finally:
>> - ip.close()
>> - finally:
>> - if redirect is not None:
>> - redirect.close()
>> -
>> -def sh(cwd=None, args=None, debugflags=None, env=None):
>> - if args is None:
>> - args = sys.argv[1:]
>> - shargs, cmdargs = split_args(args)
>> - options, shargs = SH_OPT.parse_args(shargs)
>> -
>> - if options.profile:
>> - import lsprof
>> - p = lsprof.Profiler()
>> - p.enable(subcalls=True)
>> - try:
>> - return _sh(cwd, shargs, cmdargs, options, debugflags,
>> env)
>> - finally:
>> - p.disable()
>> - stats = lsprof.Stats(p.getstats())
>> - stats.sort()
>> - stats.pprint(top=10, file=sys.stderr, climit=5)
>> - else:
>> - return _sh(cwd, shargs, cmdargs, options, debugflags, env)
>> -
>> -def main():
>> - sys.exit(sh())
>> -
>> -if __name__=='__main__':
>> - main()
>> diff --git a/bitbake/lib/bb/pysh/pyshlex.py
>> b/bitbake/lib/bb/pysh/pyshlex.py index fbf094b..a42c294 100644
>> --- a/bitbake/lib/bb/pysh/pyshlex.py
>> +++ b/bitbake/lib/bb/pysh/pyshlex.py
>> @@ -13,11 +13,6 @@
>> # PLY in pull mode. It was designed to work incrementally and it
>> would not be # that hard to enable pull mode.
>> import re
>> -try:
>> - s = set()
>> - del s
>> -except NameError:
>> - from Set import Set as set
>>
>> from ply import lex
>> from bb.pysh.sherrors import *
>> diff --git a/bitbake/lib/bb/pysh/pyshyacc.py
>> b/bitbake/lib/bb/pysh/pyshyacc.py index ba4cefd..de565dc 100644
>> --- a/bitbake/lib/bb/pysh/pyshyacc.py
>> +++ b/bitbake/lib/bb/pysh/pyshyacc.py
>> @@ -636,13 +636,16 @@ def p_empty(p):
>> def p_error(p):
>> msg = []
>> w = msg.append
>> - w('%r\n' % p)
>> - w('followed by:\n')
>> - for i in range(5):
>> - n = yacc.token()
>> - if not n:
>> - break
>> - w(' %r\n' % n)
>> + if p:
>> + w('%r\n' % p)
>> + w('followed by:\n')
>> + for i in range(5):
>> + n = yacc.token()
>> + if not n:
>> + break
>> + w(' %r\n' % n)
>> + else:
>> + w('Unexpected EOF')
>> raise sherrors.ShellSyntaxError(''.join(msg))
>>
>> # Build the parser
>> diff --git a/bitbake/lib/bb/pysh/sherrors.py
>> b/bitbake/lib/bb/pysh/sherrors.py index 49d0533..3fe8e47 100644
>> --- a/bitbake/lib/bb/pysh/sherrors.py
>> +++ b/bitbake/lib/bb/pysh/sherrors.py
>> @@ -13,29 +13,3 @@ class ShellError(Exception):
>>
>> class ShellSyntaxError(ShellError):
>> pass
>> -
>> -class UtilityError(ShellError):
>> - """Raised upon utility syntax error (option or operand error)."""
>> - pass
>> -
>> -class ExpansionError(ShellError):
>> - pass
>> -
>> -class CommandNotFound(ShellError):
>> - """Specified command was not found."""
>> - pass
>> -
>> -class RedirectionError(ShellError):
>> - pass
>> -
>> -class VarAssignmentError(ShellError):
>> - """Variable assignment error."""
>> - pass
>> -
>> -class ExitSignal(ShellError):
>> - """Exit signal."""
>> - pass
>> -
>> -class ReturnSignal(ShellError):
>> - """Exit signal."""
>> - pass
>> diff --git a/bitbake/lib/bb/pysh/subprocess_fix.py
>> b/bitbake/lib/bb/pysh/subprocess_fix.py deleted file mode 100644
>> index 46eca22..0000000
>> --- a/bitbake/lib/bb/pysh/subprocess_fix.py
>> +++ /dev/null
>> @@ -1,77 +0,0 @@
>> -# subprocess - Subprocesses with accessible I/O streams
>> -#
>> -# For more information about this module, see PEP 324.
>> -#
>> -# This module should remain compatible with Python 2.2, see PEP 291.
>> -#
>> -# Copyright (c) 2003-2005 by Peter Astrand <astrand@lysator.liu.se>
>> -#
>> -# Licensed to PSF under a Contributor Agreement.
>> -# See http://www.python.org/2.4/license for licensing details.
>> -
>> -def list2cmdline(seq):
>> - """
>> - Translate a sequence of arguments into a command line
>> - string, using the same rules as the MS C runtime:
>> -
>> - 1) Arguments are delimited by white space, which is either a
>> - space or a tab.
>> -
>> - 2) A string surrounded by double quotation marks is
>> - interpreted as a single argument, regardless of white space
>> - contained within. A quoted string can be embedded in an
>> - argument.
>> -
>> - 3) A double quotation mark preceded by a backslash is
>> - interpreted as a literal double quotation mark.
>> -
>> - 4) Backslashes are interpreted literally, unless they
>> - immediately precede a double quotation mark.
>> -
>> - 5) If backslashes immediately precede a double quotation mark,
>> - every pair of backslashes is interpreted as a literal
>> - backslash. If the number of backslashes is odd, the last
>> - backslash escapes the next double quotation mark as
>> - described in rule 3.
>> - """
>> -
>> - # See
>> - #
>> http://msdn.microsoft.com/library/en-us/vccelng/htm/progs_12.asp
>> - result = []
>> - needquote = False
>> - for arg in seq:
>> - bs_buf = []
>> -
>> - # Add a space to separate this argument from the others
>> - if result:
>> - result.append(' ')
>> -
>> - needquote = (" " in arg) or ("\t" in arg) or ("|" in arg) or
>> arg == ""
>> - if needquote:
>> - result.append('"')
>> -
>> - for c in arg:
>> - if c == '\\':
>> - # Don't know if we need to double yet.
>> - bs_buf.append(c)
>> - elif c == '"':
>> - # Double backspaces.
>> - result.append('\\' * len(bs_buf)*2)
>> - bs_buf = []
>> - result.append('\\"')
>> - else:
>> - # Normal char
>> - if bs_buf:
>> - result.extend(bs_buf)
>> - bs_buf = []
>> - result.append(c)
>> -
>> - # Add remaining backspaces, if any.
>> - if bs_buf:
>> - result.extend(bs_buf)
>> -
>> - if needquote:
>> - result.extend(bs_buf)
>> - result.append('"')
>> -
>> - return ''.join(result)
>> diff --git a/bitbake/lib/bb/remotedata.py
>> b/bitbake/lib/bb/remotedata.py index 68ecffc..7391e1b 100644
>> --- a/bitbake/lib/bb/remotedata.py
>> +++ b/bitbake/lib/bb/remotedata.py
>> @@ -6,18 +6,8 @@ Provides support for using a datastore from the
>> bitbake client
>> # Copyright (C) 2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import bb.data
>>
>> diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
>> index 9ce06c4..1804943 100644
>> --- a/bitbake/lib/bb/runqueue.py
>> +++ b/bitbake/lib/bb/runqueue.py
>> @@ -1,6 +1,3 @@
>> -#!/usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake 'RunQueue' implementation
>>
>> @@ -9,18 +6,8 @@ Handles preparation and execution of a queue of tasks
>>
>> # Copyright (C) 2006-2007 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import copy
>> import os
>> @@ -37,11 +24,13 @@ from bb import monitordisk
>> import subprocess
>> import pickle
>> from multiprocessing import Process
>> +import shlex
>> +import pprint
>>
>> bblogger = logging.getLogger("BitBake")
>> logger = logging.getLogger("BitBake.RunQueue")
>>
>> -__find_md5__ =
>> re.compile( r'(?i)(?<![a-z0-9])[a-f0-9]{32}(?![a-z0-9])' )
>> +__find_sha256__ =
>> re.compile( r'(?i)(?<![a-z0-9])[a-f0-9]{64}(?![a-z0-9])' ) def
>> fn_from_tid(tid): return tid.rsplit(":", 1)[0]
>> @@ -49,17 +38,22 @@ def fn_from_tid(tid):
>> def taskname_from_tid(tid):
>> return tid.rsplit(":", 1)[1]
>>
>> +def mc_from_tid(tid):
>> + if tid.startswith('mc:'):
>> + return tid.split(':')[1]
>> + return ""
>> +
>> def split_tid(tid):
>> (mc, fn, taskname, _) = split_tid_mcfn(tid)
>> return (mc, fn, taskname)
>>
>> def split_tid_mcfn(tid):
>> - if tid.startswith('multiconfig:'):
>> + if tid.startswith('mc:'):
>> elems = tid.split(':')
>> mc = elems[1]
>> fn = ":".join(elems[2:-1])
>> taskname = elems[-1]
>> - mcfn = "multiconfig:" + mc + ":" + fn
>> + mcfn = "mc:" + mc + ":" + fn
>> else:
>> tid = tid.rsplit(":", 1)
>> mc = ""
>> @@ -71,9 +65,17 @@ def split_tid_mcfn(tid):
>>
>> def build_tid(mc, fn, taskname):
>> if mc:
>> - return "multiconfig:" + mc + ":" + fn + ":" + taskname
>> + return "mc:" + mc + ":" + fn + ":" + taskname
>> return fn + ":" + taskname
>>
>> +# Index used to pair up potentially matching multiconfig tasks
>> +# We match on PN, taskname and hash being equal
>> +def pending_hash_index(tid, rqdata):
>> + (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
>> + pn = rqdata.dataCaches[mc].pkg_fn[taskfn]
>> + h = rqdata.runtaskentries[tid].unihash
>> + return pn + ":" + "taskname" + h
>> +
>> class RunQueueStats:
>> """
>> Holds statistics on the tasks handled by the associated runQueue
>> @@ -109,8 +111,6 @@ class RunQueueStats:
>> # runQueue state machine
>> runQueuePrepare = 2
>> runQueueSceneInit = 3
>> -runQueueSceneRun = 4
>> -runQueueRunInit = 5
>> runQueueRunning = 6
>> runQueueFailed = 7
>> runQueueCleanUp = 8
>> @@ -133,7 +133,7 @@ class RunQueueScheduler(object):
>>
>> self.prio_map = [self.rqdata.runtaskentries.keys()]
>>
>> - self.buildable = []
>> + self.buildable = set()
>> self.skip_maxthread = {}
>> self.stamps = {}
>> for tid in self.rqdata.runtaskentries:
>> @@ -148,8 +148,11 @@ class RunQueueScheduler(object):
>> """
>> Return the id of the first task we find that is buildable
>> """
>> - self.buildable = [x for x in self.buildable if x not in
>> self.rq.runq_running]
>> - if not self.buildable:
>> + buildable = set(self.buildable)
>> + buildable.difference_update(self.rq.runq_running)
>> + buildable.difference_update(self.rq.holdoff_tasks)
>> + buildable.intersection_update(self.rq.tasks_covered |
>> self.rq.tasks_notcovered)
>> + if not buildable:
>> return None
>>
>> # Filter out tasks that have a max number of threads that
>> have been exceeded @@ -165,8 +168,8 @@ class
>> RunQueueScheduler(object): else:
>> skip_buildable[rtaskname] = 1
>>
>> - if len(self.buildable) == 1:
>> - tid = self.buildable[0]
>> + if len(buildable) == 1:
>> + tid = buildable.pop()
>> taskname = taskname_from_tid(tid)
>> if taskname in skip_buildable and
>> skip_buildable[taskname] >= int(self.skip_maxthread[taskname]):
>> return None @@ -181,7 +184,7 @@ class RunQueueScheduler(object):
>>
>> best = None
>> bestprio = None
>> - for tid in self.buildable:
>> + for tid in buildable:
>> taskname = taskname_from_tid(tid)
>> if taskname in skip_buildable and
>> skip_buildable[taskname] >= int(self.skip_maxthread[taskname]):
>> continue @@ -203,7 +206,12 @@ class RunQueueScheduler(object):
>> return self.next_buildable_task()
>>
>> def newbuildable(self, task):
>> - self.buildable.append(task)
>> + self.buildable.add(task)
>> + # Once tasks are running we don't need to worry about them
>> again
>> + self.buildable.difference_update(self.rq.runq_running)
>> +
>> + def removebuildable(self, task):
>> + self.buildable.remove(task)
>>
>> def describe_task(self, taskid):
>> result = 'ID %s' % taskid
>> @@ -346,6 +354,7 @@ class RunTaskEntry(object):
>> self.depends = set()
>> self.revdeps = set()
>> self.hash = None
>> + self.unihash = None
>> self.task = None
>> self.weight = 1
>>
>> @@ -385,6 +394,9 @@ class RunQueueData:
>> def get_task_hash(self, tid):
>> return self.runtaskentries[tid].hash
>>
>> + def get_task_unihash(self, tid):
>> + return self.runtaskentries[tid].unihash
>> +
>> def get_user_idstring(self, tid, task_name_suffix = ""):
>> return tid + task_name_suffix
>>
>> @@ -405,6 +417,9 @@ class RunQueueData:
>> explored_deps = {}
>> msgs = []
>>
>> + class TooManyLoops(Exception):
>> + pass
>> +
>> def chain_reorder(chain):
>> """
>> Reorder a dependency chain so the lowest task id is first
>> @@ -457,7 +472,7 @@ class RunQueueData:
>> msgs.append("\n")
>> if len(valid_chains) > 10:
>> msgs.append("Aborted dependency loops search
>> after 10 matches.\n")
>> - return msgs
>> + raise TooManyLoops
>> continue
>> scan = False
>> if revdep not in explored_deps:
>> @@ -476,8 +491,11 @@ class RunQueueData:
>>
>> explored_deps[tid] = total_deps
>>
>> - for task in tasks:
>> - find_chains(task, [])
>> + try:
>> + for task in tasks:
>> + find_chains(task, [])
>> + except TooManyLoops:
>> + pass
>>
>> return msgs
>>
>> @@ -833,6 +851,20 @@ class RunQueueData:
>> for depend in depends:
>> mark_active(depend, depth+1)
>>
>> + def invalidate_task(tid, error_nostamp):
>> + (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
>> + taskdep = self.dataCaches[mc].task_deps[taskfn]
>> + if fn + ":" + taskname not in taskData[mc].taskentries:
>> + logger.warning("Task %s does not exist, invalidating
>> this task will have no effect" % taskname)
>> + if 'nostamp' in taskdep and taskname in
>> taskdep['nostamp']:
>> + if error_nostamp:
>> + bb.fatal("Task %s is marked nostamp, cannot
>> invalidate this task" % taskname)
>> + else:
>> + bb.debug(1, "Task %s is marked nostamp, cannot
>> invalidate this task" % taskname)
>> + else:
>> + logger.verbose("Invalidate task %s, %s", taskname,
>> fn)
>> + bb.parse.siggen.invalidate_task(taskname,
>> self.dataCaches[mc], taskfn) +
>> self.target_tids = []
>> for (mc, target, task, fn) in self.targets:
>>
>> @@ -901,6 +933,8 @@ class RunQueueData:
>>
>> for tid in list(runall_tids):
>> mark_active(tid,1)
>> + if self.cooker.configuration.force:
>> + invalidate_task(tid, False)
>>
>> for tid in list(self.runtaskentries.keys()):
>> if tid not in runq_build:
>> @@ -922,6 +956,8 @@ class RunQueueData:
>>
>> for tid in list(runonly_tids):
>> mark_active(tid,1)
>> + if self.cooker.configuration.force:
>> + invalidate_task(tid, False)
>>
>> for tid in list(self.runtaskentries.keys()):
>> if tid not in runq_build:
>> @@ -1098,20 +1134,6 @@ class RunQueueData:
>> continue
>> self.runq_setscene_tids.append(tid)
>>
>> - def invalidate_task(tid, error_nostamp):
>> - (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
>> - taskdep = self.dataCaches[mc].task_deps[taskfn]
>> - if fn + ":" + taskname not in taskData[mc].taskentries:
>> - logger.warning("Task %s does not exist, invalidating
>> this task will have no effect" % taskname)
>> - if 'nostamp' in taskdep and taskname in
>> taskdep['nostamp']:
>> - if error_nostamp:
>> - bb.fatal("Task %s is marked nostamp, cannot
>> invalidate this task" % taskname)
>> - else:
>> - bb.debug(1, "Task %s is marked nostamp, cannot
>> invalidate this task" % taskname)
>> - else:
>> - logger.verbose("Invalidate task %s, %s", taskname,
>> fn)
>> - bb.parse.siggen.invalidate_task(taskname,
>> self.dataCaches[mc], taskfn) -
>> self.init_progress_reporter.next_stage()
>>
>> # Invalidate task if force mode active
>> @@ -1142,6 +1164,8 @@ class RunQueueData:
>>
>> self.init_progress_reporter.next_stage()
>>
>> + bb.parse.siggen.set_setscene_tasks(self.runq_setscene_tids)
>> +
>> # Iterate over the task list and call into the siggen code
>> dealtwith = set()
>> todeal = set(self.runtaskentries)
>> @@ -1150,18 +1174,20 @@ class RunQueueData:
>> if len(self.runtaskentries[tid].depends - dealtwith)
>> == 0: dealtwith.add(tid)
>> todeal.remove(tid)
>> - procdep = []
>> - for dep in self.runtaskentries[tid].depends:
>> - procdep.append(fn_from_tid(dep) + "." +
>> taskname_from_tid(dep))
>> - (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
>> - self.runtaskentries[tid].hash =
>> bb.parse.siggen.get_taskhash(taskfn, taskname, procdep,
>> self.dataCaches[mc])
>> - task = self.runtaskentries[tid].task
>> + self.prepare_task_hash(tid)
>>
>> bb.parse.siggen.writeout_file_checksum_cache()
>>
>> #self.dump_data()
>> return len(self.runtaskentries)
>>
>> + def prepare_task_hash(self, tid):
>> + procdep = []
>> + for dep in self.runtaskentries[tid].depends:
>> + procdep.append(dep)
>> + self.runtaskentries[tid].hash =
>> bb.parse.siggen.get_taskhash(tid, procdep,
>> self.dataCaches[mc_from_tid(tid)])
>> + self.runtaskentries[tid].unihash =
>> bb.parse.siggen.get_unihash(tid) +
>> def dump_data(self):
>> """
>> Dump some debug information on the internal data structures
>> @@ -1187,7 +1213,6 @@ class RunQueue:
>>
>> self.stamppolicy = cfgData.getVar("BB_STAMP_POLICY") or
>> "perfile" self.hashvalidate = cfgData.getVar("BB_HASHCHECK_FUNCTION")
>> or None
>> - self.setsceneverify =
>> cfgData.getVar("BB_SETSCENE_VERIFY_FUNCTION2") or None
>> self.depvalidate = cfgData.getVar("BB_SETSCENE_DEPVALID") or None
>> self.state = runQueuePrepare
>> @@ -1196,7 +1221,7 @@ class RunQueue:
>> # Invoked at regular time intervals via the bitbake
>> heartbeat event # while the build is running. We generate a unique
>> name for the handler # here, just in case that there ever is more
>> than one RunQueue instance,
>> - # start the handler when reaching runQueueSceneRun, and stop
>> it when
>> + # start the handler when reaching runQueueSceneInit, and
>> stop it when # done with the build.
>> self.dm = monitordisk.diskMonitor(cfgData)
>> self.dm_event_handler_name = '_bb_diskmonitor_' +
>> str(id(self)) @@ -1213,28 +1238,23 @@ class RunQueue:
>> if fakeroot:
>> magic = magic + "beef"
>> mcdata = self.cooker.databuilder.mcdata[mc]
>> - fakerootcmd = mcdata.getVar("FAKEROOTCMD")
>> + fakerootcmd = shlex.split(mcdata.getVar("FAKEROOTCMD"))
>> fakerootenv = (mcdata.getVar("FAKEROOTBASEENV") or
>> "").split() env = os.environ.copy()
>> for key, value in (var.split('=') for var in
>> fakerootenv): env[key] = value
>> - worker = subprocess.Popen([fakerootcmd,
>> "bitbake-worker", magic], stdout=subprocess.PIPE,
>> stdin=subprocess.PIPE, env=env)
>> + worker = subprocess.Popen(fakerootcmd +
>> ["bitbake-worker", magic], stdout=subprocess.PIPE,
>> stdin=subprocess.PIPE, env=env) else: worker =
>> subprocess.Popen(["bitbake-worker", magic], stdout=subprocess.PIPE,
>> stdin=subprocess.PIPE) bb.utils.nonblockingfd(worker.stdout)
>> workerpipe = runQueuePipe(worker.stdout, None, self.cfgData, self,
>> rqexec)
>> - runqhash = {}
>> - for tid in self.rqdata.runtaskentries:
>> - runqhash[tid] = self.rqdata.runtaskentries[tid].hash
>> -
>> workerdata = {
>> "taskdeps" : self.rqdata.dataCaches[mc].task_deps,
>> "fakerootenv" : self.rqdata.dataCaches[mc].fakerootenv,
>> "fakerootdirs" : self.rqdata.dataCaches[mc].fakerootdirs,
>> "fakerootnoenv" :
>> self.rqdata.dataCaches[mc].fakerootnoenv, "sigdata" :
>> bb.parse.siggen.get_taskdata(),
>> - "runq_hash" : runqhash,
>> "logdefaultdebug" : bb.msg.loggerDefaultDebugLevel,
>> "logdefaultverbose" : bb.msg.loggerDefaultVerbose,
>> "logdefaultverboselogs" : bb.msg.loggerVerboseLogs,
>> @@ -1243,6 +1263,7 @@ class RunQueue:
>> "buildname" : self.cfgData.getVar("BUILDNAME"),
>> "date" : self.cfgData.getVar("DATE"),
>> "time" : self.cfgData.getVar("TIME"),
>> + "hashservaddr" : self.cooker.hashservaddr,
>> }
>>
>> worker.stdin.write(b"<cookerconfig>" +
>> pickle.dumps(self.cooker.configuration) + b"</cookerconfig>") @@
>> -1376,6 +1397,31 @@ class RunQueue: cache[tid] = iscurrent
>> return iscurrent
>>
>> + def validate_hashes(self, tocheck, data, currentcount=0,
>> siginfo=False):
>> + valid = set()
>> + if self.hashvalidate:
>> + sq_data = {}
>> + sq_data['hash'] = {}
>> + sq_data['hashfn'] = {}
>> + sq_data['unihash'] = {}
>> + for tid in tocheck:
>> + (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
>> + sq_data['hash'][tid] =
>> self.rqdata.runtaskentries[tid].hash
>> + sq_data['hashfn'][tid] =
>> self.rqdata.dataCaches[mc].hashfn[taskfn]
>> + sq_data['unihash'][tid] =
>> self.rqdata.runtaskentries[tid].unihash +
>> + valid = self.validate_hash(sq_data, data, siginfo,
>> currentcount) +
>> + return valid
>> +
>> + def validate_hash(self, sq_data, d, siginfo, currentcount):
>> + locs = {"sq_data" : sq_data, "d" : d, "siginfo" : siginfo,
>> "currentcount" : currentcount} +
>> + # Metadata has **kwargs so args can be added, sq_data can
>> also gain new fields
>> + call = self.hashvalidate + "(sq_data, d, siginfo=siginfo,
>> currentcount=currentcount)" +
>> + return bb.utils.better_eval(call, locs)
>> +
>> def _execute_runqueue(self):
>> """
>> Run the tasks in a queue prepared by rqdata.prepare()
>> @@ -1386,7 +1432,6 @@ class RunQueue:
>> retval = True
>>
>> if self.state is runQueuePrepare:
>> - self.rqexe = RunQueueExecuteDummy(self)
>> # NOTE: if you add, remove or significantly refactor the
>> stages of this # process then you should recalculate the weightings
>> here. This is quite # easy to do - just change the next line
>> temporarily to pass debug=True as @@ -1400,15 +1445,23 @@ class
>> RunQueue: self.state = runQueueComplete
>> else:
>> self.state = runQueueSceneInit
>> - self.rqdata.init_progress_reporter.next_stage()
>> -
>> - # we are ready to run, emit dependency info to any
>> UI or class which
>> - # needs it
>> - depgraph = self.cooker.buildDependTree(self,
>> self.rqdata.taskData)
>> - self.rqdata.init_progress_reporter.next_stage()
>> - bb.event.fire(bb.event.DepTreeGenerated(depgraph),
>> self.cooker.data)
>> + bb.parse.siggen.save_unitaskhashes()
>>
>> if self.state is runQueueSceneInit:
>> + self.rqdata.init_progress_reporter.next_stage()
>> +
>> + # we are ready to run, emit dependency info to any UI
>> or class which
>> + # needs it
>> + depgraph = self.cooker.buildDependTree(self,
>> self.rqdata.taskData)
>> + self.rqdata.init_progress_reporter.next_stage()
>> + bb.event.fire(bb.event.DepTreeGenerated(depgraph),
>> self.cooker.data) +
>> + if not self.dm_event_handler_registered:
>> + res = bb.event.register(self.dm_event_handler_name,
>> + lambda x:
>> self.dm.check(self) if self.state in [runQueueRunning,
>> runQueueCleanUp] else False,
>> +
>> ('bb.event.HeartbeatEvent',))
>> + self.dm_event_handler_registered = True
>> +
>> dump = self.cooker.configuration.dump_signatures
>> if dump:
>> self.rqdata.init_progress_reporter.finish()
>> @@ -1418,29 +1471,23 @@ class RunQueue:
>> if 'printdiff' in dump:
>> self.write_diffscenetasks(invalidtasks)
>> self.state = runQueueComplete
>> - else:
>> - self.rqdata.init_progress_reporter.next_stage()
>> - self.start_worker()
>> - self.rqdata.init_progress_reporter.next_stage()
>> - self.rqexe = RunQueueExecuteScenequeue(self)
>> -
>> - if self.state is runQueueSceneRun:
>> - if not self.dm_event_handler_registered:
>> - res = bb.event.register(self.dm_event_handler_name,
>> - lambda x:
>> self.dm.check(self) if self.state in [runQueueSceneRun,
>> runQueueRunning, runQueueCleanUp] else False,
>> -
>> ('bb.event.HeartbeatEvent',))
>> - self.dm_event_handler_registered = True
>> - retval = self.rqexe.execute()
>>
>> - if self.state is runQueueRunInit:
>> - if self.cooker.configuration.setsceneonly:
>> - self.state = runQueueComplete
>> - else:
>> - # Just in case we didn't setscene
>> - self.rqdata.init_progress_reporter.finish()
>> - logger.info("Executing RunQueue Tasks")
>> - self.rqexe = RunQueueExecuteTasks(self)
>> - self.state = runQueueRunning
>> + if self.state is runQueueSceneInit:
>> + self.rqdata.init_progress_reporter.next_stage()
>> + self.start_worker()
>> + self.rqdata.init_progress_reporter.next_stage()
>> + self.rqexe = RunQueueExecute(self)
>> +
>> + # If we don't have any setscene functions, skip execution
>> + if len(self.rqdata.runq_setscene_tids) == 0:
>> + logger.info('No setscene tasks')
>> + for tid in self.rqdata.runtaskentries:
>> + if len(self.rqdata.runtaskentries[tid].depends)
>> == 0:
>> + self.rqexe.setbuildable(tid)
>> + self.rqexe.tasks_notcovered.add(tid)
>> + self.rqexe.sqdone = True
>> + logger.info('Executing Tasks')
>> + self.state = runQueueRunning
>>
>> if self.state is runQueueRunning:
>> retval = self.rqexe.execute()
>> @@ -1455,12 +1502,14 @@ class RunQueue:
>> self.dm_event_handler_registered = False
>>
>> if build_done and self.rqexe:
>> + bb.parse.siggen.save_unitaskhashes()
>> self.teardown_workers()
>> - if self.rqexe.stats.failed:
>> - logger.info("Tasks Summary: Attempted %d tasks of
>> which %d didn't need to be rerun and %d failed.",
>> self.rqexe.stats.completed + self.rqexe.stats.failed,
>> self.rqexe.stats.skipped, self.rqexe.stats.failed)
>> - else:
>> - # Let's avoid the word "failed" if nothing actually
>> did
>> - logger.info("Tasks Summary: Attempted %d tasks of
>> which %d didn't need to be rerun and all succeeded.",
>> self.rqexe.stats.completed, self.rqexe.stats.skipped)
>> + if self.rqexe:
>> + if self.rqexe.stats.failed:
>> + logger.info("Tasks Summary: Attempted %d tasks
>> of which %d didn't need to be rerun and %d failed.",
>> self.rqexe.stats.completed + self.rqexe.stats.failed,
>> self.rqexe.stats.skipped, self.rqexe.stats.failed)
>> + else:
>> + # Let's avoid the word "failed" if nothing
>> actually did
>> + logger.info("Tasks Summary: Attempted %d tasks
>> of which %d didn't need to be rerun and all succeeded.",
>> self.rqexe.stats.completed, self.rqexe.stats.skipped) if self.state
>> is runQueueFailed: raise
>> bb.runqueue.TaskFailure(self.rqexe.failed_tids) @@ -1543,15 +1592,8
>> @@ class RunQueue:
>> def print_diffscenetasks(self):
>>
>> - valid = []
>> - sq_hash = []
>> - sq_hashfn = []
>> - sq_fn = []
>> - sq_taskname = []
>> - sq_task = []
>> noexec = []
>> - stamppresent = []
>> - valid_new = set()
>> + tocheck = set()
>>
>> for tid in self.rqdata.runtaskentries:
>> (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
>> @@ -1561,21 +1603,9 @@ class RunQueue:
>> noexec.append(tid)
>> continue
>>
>> - sq_fn.append(fn)
>> -
>> sq_hashfn.append(self.rqdata.dataCaches[mc].hashfn[taskfn])
>> - sq_hash.append(self.rqdata.runtaskentries[tid].hash)
>> - sq_taskname.append(taskname)
>> - sq_task.append(tid)
>> - locs = { "sq_fn" : sq_fn, "sq_task" : sq_taskname,
>> "sq_hash" : sq_hash, "sq_hashfn" : sq_hashfn, "d" : self.cooker.data }
>> - try:
>> - call = self.hashvalidate + "(sq_fn, sq_task, sq_hash,
>> sq_hashfn, d, siginfo=True)"
>> - valid = bb.utils.better_eval(call, locs)
>> - # Handle version with no siginfo parameter
>> - except TypeError:
>> - call = self.hashvalidate + "(sq_fn, sq_task, sq_hash,
>> sq_hashfn, d)"
>> - valid = bb.utils.better_eval(call, locs)
>> - for v in valid:
>> - valid_new.add(sq_task[v])
>> + tocheck.add(tid)
>> +
>> + valid_new = self.validate_hashes(tocheck, self.cooker.data,
>> 0, True)
>> # Tasks which are both setscene and noexec never care about
>> dependencies # We therefore find tasks which are setscene and noexec
>> and mark their @@ -1634,7 +1664,7 @@ class RunQueue:
>> recout = []
>> if len(hashfiles) == 2:
>> out2 = bb.siggen.compare_sigfiles(hashfiles[hash1],
>> hashfiles[hash2], recursecb)
>> - recout.extend(list(' ' + l for l in out2))
>> + recout.extend(list(' ' + l for l in out2))
>> else:
>> recout.append("Unable to find matching sigdata for
>> %s with hashes %s or %s" % (key, hash1, hash2))
>> @@ -1655,10 +1685,11 @@ class RunQueue:
>> matches = {k : v for k, v in iter(matches.items()) if h
>> not in k} if matches:
>> latestmatch = sorted(matches.keys(), key=lambda f:
>> matches[f])[-1]
>> - prevh = __find_md5__.search(latestmatch).group(0)
>> + prevh = __find_sha256__.search(latestmatch).group(0)
>> output = bb.siggen.compare_sigfiles(latestmatch,
>> match, recursecb) bb.plain("\nTask %s:%s couldn't be used from the
>> cache because:\n We need hash %s, closest matching task was %s\n "
>> % (pn, taskname, h, prevh) + '\n '.join(output)) +
>> class RunQueueExecute:
>>
>> def __init__(self, rq):
>> @@ -1670,6 +1701,13 @@ class RunQueueExecute:
>> self.number_tasks =
>> int(self.cfgData.getVar("BB_NUMBER_THREADS") or 1) self.scheduler =
>> self.cfgData.getVar("BB_SCHEDULER") or "speed"
>> + self.sq_buildable = set()
>> + self.sq_running = set()
>> + self.sq_live = set()
>> +
>> + self.updated_taskhash_queue = []
>> + self.pending_migrations = set()
>> +
>> self.runq_buildable = set()
>> self.runq_running = set()
>> self.runq_complete = set()
>> @@ -1677,9 +1715,17 @@ class RunQueueExecute:
>> self.build_stamps = {}
>> self.build_stamps2 = []
>> self.failed_tids = []
>> + self.sq_deferred = {}
>>
>> self.stampcache = {}
>>
>> + self.holdoff_tasks = set()
>> + self.holdoff_need_update = True
>> + self.sqdone = False
>> +
>> + self.stats = RunQueueStats(len(self.rqdata.runtaskentries))
>> + self.sq_stats =
>> RunQueueStats(len(self.rqdata.runq_setscene_tids)) +
>> for mc in rq.worker:
>> rq.worker[mc].pipe.setrunqueueexec(self)
>> for mc in rq.fakeworker:
>> @@ -1688,6 +1734,34 @@ class RunQueueExecute:
>> if self.number_tasks <= 0:
>> bb.fatal("Invalid BB_NUMBER_THREADS %s" %
>> self.number_tasks)
>> + # List of setscene tasks which we've covered
>> + self.scenequeue_covered = set()
>> + # List of tasks which are covered (including setscene ones)
>> + self.tasks_covered = set()
>> + self.tasks_scenequeue_done = set()
>> + self.scenequeue_notcovered = set()
>> + self.tasks_notcovered = set()
>> + self.scenequeue_notneeded = set()
>> +
>> + # We can't skip specified target tasks which aren't setscene
>> tasks
>> + self.cantskip = set(self.rqdata.target_tids)
>> +
>> self.cantskip.difference_update(self.rqdata.runq_setscene_tids)
>> + self.cantskip.intersection_update(self.rqdata.runtaskentries)
>> +
>> + schedulers = self.get_schedulers()
>> + for scheduler in schedulers:
>> + if self.scheduler == scheduler.name:
>> + self.sched = scheduler(self, self.rqdata)
>> + logger.debug(1, "Using runqueue scheduler '%s'",
>> scheduler.name)
>> + break
>> + else:
>> + bb.fatal("Invalid scheduler '%s'. Available schedulers:
>> %s" %
>> + (self.scheduler, ", ".join(obj.name for obj in
>> schedulers))) +
>> + #if len(self.rqdata.runq_setscene_tids) > 0:
>> + self.sqdata = SQData()
>> + build_scenequeue_data(self.sqdata, self.rqdata, self.rq,
>> self.cooker, self.stampcache, self) +
>> def runqueue_process_waitpid(self, task, status):
>>
>> # self.build_stamps[pid] may not exist when use shared work
>> directory. @@ -1695,10 +1769,17 @@ class RunQueueExecute:
>> self.build_stamps2.remove(self.build_stamps[task])
>> del self.build_stamps[task]
>>
>> - if status != 0:
>> - self.task_fail(task, status)
>> + if task in self.sq_live:
>> + if status != 0:
>> + self.sq_task_fail(task, status)
>> + else:
>> + self.sq_task_complete(task)
>> + self.sq_live.remove(task)
>> else:
>> - self.task_complete(task)
>> + if status != 0:
>> + self.task_fail(task, status)
>> + else:
>> + self.task_complete(task)
>> return True
>>
>> def finish_now(self):
>> @@ -1727,8 +1808,9 @@ class RunQueueExecute:
>> def finish(self):
>> self.rq.state = runQueueCleanUp
>>
>> - if self.stats.active > 0:
>> - bb.event.fire(runQueueExitWait(self.stats.active),
>> self.cfgData)
>> + active = self.stats.active + self.sq_stats.active
>> + if active > 0:
>> + bb.event.fire(runQueueExitWait(active), self.cfgData)
>> self.rq.read_workers()
>> return self.rq.active_fds()
>>
>> @@ -1739,10 +1821,14 @@ class RunQueueExecute:
>> self.rq.state = runQueueComplete
>> return True
>>
>> - def check_dependencies(self, task, taskdeps, setscene = False):
>> + # Used by setscene only
>> + def check_dependencies(self, task, taskdeps):
>> if not self.rq.depvalidate:
>> return False
>>
>> + # Must not edit parent data
>> + taskdeps = set(taskdeps)
>> +
>> taskdata = {}
>> taskdeps.add(task)
>> for dep in taskdeps:
>> @@ -1755,121 +1841,10 @@ class RunQueueExecute:
>> return valid
>>
>> def can_start_task(self):
>> - can_start = self.stats.active < self.number_tasks
>> + active = self.stats.active + self.sq_stats.active
>> + can_start = active < self.number_tasks
>> return can_start
>>
>> -class RunQueueExecuteDummy(RunQueueExecute):
>> - def __init__(self, rq):
>> - self.rq = rq
>> - self.stats = RunQueueStats(0)
>> -
>> - def finish(self):
>> - self.rq.state = runQueueComplete
>> - return
>> -
>> -class RunQueueExecuteTasks(RunQueueExecute):
>> - def __init__(self, rq):
>> - RunQueueExecute.__init__(self, rq)
>> -
>> - self.stats = RunQueueStats(len(self.rqdata.runtaskentries))
>> -
>> - self.stampcache = {}
>> -
>> - initial_covered = self.rq.scenequeue_covered.copy()
>> -
>> - # Mark initial buildable tasks
>> - for tid in self.rqdata.runtaskentries:
>> - if len(self.rqdata.runtaskentries[tid].depends) == 0:
>> - self.runq_buildable.add(tid)
>> - if len(self.rqdata.runtaskentries[tid].revdeps) > 0 and
>> self.rqdata.runtaskentries[tid].revdeps.issubset(self.rq.scenequeue_covered):
>> - self.rq.scenequeue_covered.add(tid)
>> -
>> - found = True
>> - while found:
>> - found = False
>> - for tid in self.rqdata.runtaskentries:
>> - if tid in self.rq.scenequeue_covered:
>> - continue
>> - logger.debug(1, 'Considering %s: %s' % (tid,
>> str(self.rqdata.runtaskentries[tid].revdeps))) -
>> - if len(self.rqdata.runtaskentries[tid].revdeps) > 0
>> and
>> self.rqdata.runtaskentries[tid].revdeps.issubset(self.rq.scenequeue_covered):
>> - if tid in self.rq.scenequeue_notcovered:
>> - continue
>> - found = True
>> - self.rq.scenequeue_covered.add(tid)
>> -
>> - logger.debug(1, 'Skip list (pre setsceneverify) %s',
>> sorted(self.rq.scenequeue_covered)) -
>> - # Allow the metadata to elect for setscene tasks to run
>> anyway
>> - covered_remove = set()
>> - if self.rq.setsceneverify:
>> - invalidtasks = []
>> - tasknames = {}
>> - fns = {}
>> - for tid in self.rqdata.runtaskentries:
>> - (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
>> - taskdep =
>> self.rqdata.dataCaches[mc].task_deps[taskfn]
>> - fns[tid] = taskfn
>> - tasknames[tid] = taskname
>> - if 'noexec' in taskdep and taskname in
>> taskdep['noexec']:
>> - continue
>> - if self.rq.check_stamp_task(tid, taskname +
>> "_setscene", cache=self.stampcache):
>> - logger.debug(2, 'Setscene stamp current for task
>> %s', tid)
>> - continue
>> - if self.rq.check_stamp_task(tid, taskname, recurse =
>> True, cache=self.stampcache):
>> - logger.debug(2, 'Normal stamp current for task
>> %s', tid)
>> - continue
>> - invalidtasks.append(tid)
>> -
>> - call = self.rq.setsceneverify + "(covered, tasknames,
>> fns, d, invalidtasks=invalidtasks)"
>> - locs = { "covered" : self.rq.scenequeue_covered,
>> "tasknames" : tasknames, "fns" : fns, "d" : self.cooker.data,
>> "invalidtasks" : invalidtasks }
>> - covered_remove = bb.utils.better_eval(call, locs)
>> -
>> - def removecoveredtask(tid):
>> - (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
>> - taskname = taskname + '_setscene'
>> - bb.build.del_stamp(taskname, self.rqdata.dataCaches[mc],
>> taskfn)
>> - self.rq.scenequeue_covered.remove(tid)
>> -
>> - toremove = covered_remove | self.rq.scenequeue_notcovered
>> - for task in toremove:
>> - logger.debug(1, 'Not skipping task %s due to
>> setsceneverify', task)
>> - while toremove:
>> - covered_remove = []
>> - for task in toremove:
>> - if task in self.rq.scenequeue_covered:
>> - removecoveredtask(task)
>> - for deptask in
>> self.rqdata.runtaskentries[task].depends:
>> - if deptask not in self.rq.scenequeue_covered:
>> - continue
>> - if deptask in toremove or deptask in
>> covered_remove or deptask in initial_covered:
>> - continue
>> - logger.debug(1, 'Task %s depends on task %s so
>> not skipping' % (task, deptask))
>> - covered_remove.append(deptask)
>> - toremove = covered_remove
>> -
>> - logger.debug(1, 'Full skip list %s',
>> self.rq.scenequeue_covered) -
>> -
>> - for mc in self.rqdata.dataCaches:
>> - target_pairs = []
>> - for tid in self.rqdata.target_tids:
>> - (tidmc, fn, taskname, _) = split_tid_mcfn(tid)
>> - if tidmc == mc:
>> - target_pairs.append((fn, taskname))
>> -
>> - event.fire(bb.event.StampUpdate(target_pairs,
>> self.rqdata.dataCaches[mc].stamp), self.cfgData) -
>> - schedulers = self.get_schedulers()
>> - for scheduler in schedulers:
>> - if self.scheduler == scheduler.name:
>> - self.sched = scheduler(self, self.rqdata)
>> - logger.debug(1, "Using runqueue scheduler '%s'",
>> scheduler.name)
>> - break
>> - else:
>> - bb.fatal("Invalid scheduler '%s'. Available schedulers:
>> %s" %
>> - (self.scheduler, ", ".join(obj.name for obj in
>> schedulers))) -
>> def get_schedulers(self):
>> schedulers = set(obj for obj in globals().values()
>> if type(obj) is type and
>> @@ -1941,67 +1916,172 @@ class RunQueueExecuteTasks(RunQueueExecute):
>> self.stats.taskSkipped()
>> self.stats.taskCompleted()
>>
>> + def summarise_scenequeue_errors(self):
>> + err = False
>> + if not self.sqdone:
>> + logger.debug(1, 'We could skip tasks %s',
>> "\n".join(sorted(self.scenequeue_covered)))
>> + completeevent = sceneQueueComplete(self.sq_stats,
>> self.rq)
>> + bb.event.fire(completeevent, self.cfgData)
>> + if self.sq_deferred:
>> + logger.error("Scenequeue had deferred entries: %s" %
>> pprint.pformat(self.sq_deferred))
>> + err = True
>> + if self.updated_taskhash_queue:
>> + logger.error("Scenequeue had unprocessed changed
>> taskhash entries: %s" % pprint.pformat(self.updated_taskhash_queue))
>> + err = True
>> + if self.holdoff_tasks:
>> + logger.error("Scenequeue had holdoff tasks: %s" %
>> pprint.pformat(self.holdoff_tasks))
>> + err = True
>> +
>> + for tid in self.rqdata.runq_setscene_tids:
>> + if tid not in self.scenequeue_covered and tid not in
>> self.scenequeue_notcovered:
>> + err = True
>> + logger.error("Setscene Task %s was never marked as
>> covered or not covered" % tid)
>> + if tid not in self.sq_buildable:
>> + err = True
>> + logger.error("Setscene Task %s was never marked as
>> buildable" % tid)
>> + if tid not in self.sq_running:
>> + err = True
>> + logger.error("Setscene Task %s was never marked as
>> running" % tid) +
>> + for x in self.rqdata.runtaskentries:
>> + if x not in self.tasks_covered and x not in
>> self.tasks_notcovered:
>> + logger.error("Task %s was never moved from the
>> setscene queue" % x)
>> + err = True
>> + if x not in self.tasks_scenequeue_done:
>> + logger.error("Task %s was never processed by the
>> setscene code" % x)
>> + err = True
>> + if len(self.rqdata.runtaskentries[x].depends) == 0 and x
>> not in self.runq_buildable:
>> + logger.error("Task %s was never marked as buildable
>> by the setscene code" % x)
>> + err = True
>> + return err
>> +
>> +
>> def execute(self):
>> """
>> - Run the tasks in a queue prepared by rqdata.prepare()
>> + Run the tasks in a queue prepared by prepare_runqueue
>> """
>>
>> - if self.rqdata.setscenewhitelist is not None and not
>> self.rqdata.setscenewhitelist_checked:
>> - self.rqdata.setscenewhitelist_checked = True
>> -
>> - # Check tasks that are going to run against the whitelist
>> - def check_norun_task(tid, showerror=False):
>> - (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
>> - # Ignore covered tasks
>> - if tid in self.rq.scenequeue_covered:
>> - return False
>> - # Ignore stamped tasks
>> - if self.rq.check_stamp_task(tid, taskname,
>> cache=self.stampcache):
>> - return False
>> - # Ignore noexec tasks
>> - taskdep =
>> self.rqdata.dataCaches[mc].task_deps[taskfn]
>> - if 'noexec' in taskdep and taskname in
>> taskdep['noexec']:
>> - return False
>> + self.rq.read_workers()
>> + self.process_possible_migrations()
>>
>> - pn = self.rqdata.dataCaches[mc].pkg_fn[taskfn]
>> - if not check_setscene_enforce_whitelist(pn,
>> taskname, self.rqdata.setscenewhitelist):
>> - if showerror:
>> - if tid in self.rqdata.runq_setscene_tids:
>> - logger.error('Task %s.%s attempted to
>> execute unexpectedly and should have been setscened' % (pn, taskname))
>> + task = None
>> + if not self.sqdone and self.can_start_task():
>> + # Find the next setscene to run
>> + for nexttask in sorted(self.rqdata.runq_setscene_tids):
>> + if nexttask in self.sq_buildable and nexttask not in
>> self.sq_running and self.sqdata.stamps[nexttask] not in
>> self.build_stamps.values():
>> + if nexttask not in self.sqdata.unskippable and
>> len(self.sqdata.sq_revdeps[nexttask]) > 0 and
>> self.sqdata.sq_revdeps[nexttask].issubset(self.scenequeue_covered)
>> and self.check_dependencies(nexttask,
>> self.sqdata.sq_revdeps[nexttask]):
>> + if nexttask not in self.rqdata.target_tids:
>> + logger.debug(2, "Skipping setscene for
>> task %s" % nexttask)
>> + self.sq_task_skip(nexttask)
>> + self.scenequeue_notneeded.add(nexttask)
>> + if nexttask in self.sq_deferred:
>> + del self.sq_deferred[nexttask]
>> + return True
>> + # If covered tasks are running, need to wait for
>> them to complete
>> + for t in self.sqdata.sq_covered_tasks[nexttask]:
>> + if t in self.runq_running and t not in
>> self.runq_complete:
>> + continue
>> + if nexttask in self.sq_deferred:
>> + if self.sq_deferred[nexttask] not in
>> self.runq_complete:
>> + continue
>> + logger.debug(1, "Task %s no longer deferred"
>> % nexttask)
>> + del self.sq_deferred[nexttask]
>> + valid =
>> self.rq.validate_hashes(set([nexttask]), self.cooker.data, 0, False)
>> + if not valid:
>> + logger.debug(1, "%s didn't become valid,
>> skipping setscene" % nexttask)
>> + self.sq_task_failoutright(nexttask)
>> + return True
>> else:
>> - logger.error('Task %s.%s attempted to
>> execute unexpectedly' % (pn, taskname))
>> - return True
>> - return False
>> - # Look to see if any tasks that we think shouldn't run
>> are going to
>> - unexpected = False
>> - for tid in self.rqdata.runtaskentries:
>> - if check_norun_task(tid):
>> - unexpected = True
>> + self.sqdata.outrightfail.remove(nexttask)
>> + if nexttask in self.sqdata.outrightfail:
>> + logger.debug(2, 'No package found, so
>> skipping setscene task %s', nexttask)
>> + self.sq_task_failoutright(nexttask)
>> + return True
>> + if nexttask in self.sqdata.unskippable:
>> + logger.debug(2, "Setscene task %s is
>> unskippable" % nexttask)
>> + task = nexttask
>> break
>> - if unexpected:
>> - # Run through the tasks in the rough order they'd
>> have executed and print errors
>> - # (since the order can be useful - usually missing
>> sstate for the last few tasks
>> - # is the cause of the problem)
>> - task = self.sched.next()
>> - while task is not None:
>> - check_norun_task(task, showerror=True)
>> - self.task_skip(task, 'Setscene enforcement
>> check')
>> - task = self.sched.next()
>> + if task is not None:
>> + (mc, fn, taskname, taskfn) = split_tid_mcfn(task)
>> + taskname = taskname + "_setscene"
>> + if self.rq.check_stamp_task(task,
>> taskname_from_tid(task), recurse = True, cache=self.stampcache):
>> + logger.debug(2, 'Stamp for underlying task %s is
>> current, so skipping setscene variant', task)
>> + self.sq_task_failoutright(task)
>> + return True
>>
>> - self.rq.state = runQueueCleanUp
>> + if self.cooker.configuration.force:
>> + if task in self.rqdata.target_tids:
>> + self.sq_task_failoutright(task)
>> + return True
>> +
>> + if self.rq.check_stamp_task(task, taskname,
>> cache=self.stampcache):
>> + logger.debug(2, 'Setscene stamp current task %s, so
>> skip it and its dependencies', task)
>> + self.sq_task_skip(task)
>> return True
>>
>> - self.rq.read_workers()
>> + if self.cooker.configuration.skipsetscene:
>> + logger.debug(2, 'No setscene tasks should be
>> executed. Skipping %s', task)
>> + self.sq_task_failoutright(task)
>> + return True
>>
>> - if self.stats.total == 0:
>> - # nothing to do
>> - self.rq.state = runQueueCleanUp
>> + startevent = sceneQueueTaskStarted(task, self.sq_stats,
>> self.rq)
>> + bb.event.fire(startevent, self.cfgData)
>> +
>> + taskdepdata = self.sq_build_taskdepdata(task)
>> +
>> + taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn]
>> + taskhash = self.rqdata.get_task_hash(task)
>> + unihash = self.rqdata.get_task_unihash(task)
>> + if 'fakeroot' in taskdep and taskname in
>> taskdep['fakeroot'] and not self.cooker.configuration.dry_run:
>> + if not mc in self.rq.fakeworker:
>> + self.rq.start_fakeworker(self, mc)
>> +
>> self.rq.fakeworker[mc].process.stdin.write(b"<runtask>" +
>> pickle.dumps((taskfn, task, taskname, taskhash, unihash, True,
>> self.cooker.collection.get_file_appends(taskfn), taskdepdata, False))
>> + b"</runtask>")
>> + self.rq.fakeworker[mc].process.stdin.flush()
>> + else:
>> + self.rq.worker[mc].process.stdin.write(b"<runtask>"
>> + pickle.dumps((taskfn, task, taskname, taskhash, unihash, True,
>> self.cooker.collection.get_file_appends(taskfn), taskdepdata, False))
>> + b"</runtask>")
>> + self.rq.worker[mc].process.stdin.flush()
>>
>> - task = self.sched.next()
>> + self.build_stamps[task] = bb.build.stampfile(taskname,
>> self.rqdata.dataCaches[mc], taskfn, noextra=True)
>> + self.build_stamps2.append(self.build_stamps[task])
>> + self.sq_running.add(task)
>> + self.sq_live.add(task)
>> + self.sq_stats.taskActive()
>> + if self.can_start_task():
>> + return True
>> +
>> + self.update_holdofftasks()
>> +
>> + if not self.sq_live and not self.sqdone and not
>> self.sq_deferred and not self.updated_taskhash_queue and not
>> self.holdoff_tasks:
>> + logger.info("Setscene tasks completed")
>> +
>> + err = self.summarise_scenequeue_errors()
>> + if err:
>> + self.rq.state = runQueueFailed
>> + return True
>> +
>> + if self.cooker.configuration.setsceneonly:
>> + self.rq.state = runQueueComplete
>> + return True
>> + self.sqdone = True
>> +
>> + if self.stats.total == 0:
>> + # nothing to do
>> + self.rq.state = runQueueComplete
>> + return True
>> +
>> + if self.cooker.configuration.setsceneonly:
>> + task = None
>> + else:
>> + task = self.sched.next()
>> if task is not None:
>> (mc, fn, taskname, taskfn) = split_tid_mcfn(task)
>>
>> - if task in self.rq.scenequeue_covered:
>> + if self.rqdata.setscenewhitelist is not None:
>> + if self.check_setscenewhitelist(task):
>> + self.task_fail(task, "setscene whitelist")
>> + return True
>> +
>> + if task in self.tasks_covered:
>> logger.debug(2, "Setscene covered task %s", task)
>> self.task_skip(task, "covered")
>> return True
>> @@ -2030,6 +2110,8 @@ class RunQueueExecuteTasks(RunQueueExecute):
>> taskdepdata = self.build_taskdepdata(task)
>>
>> taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn]
>> + taskhash = self.rqdata.get_task_hash(task)
>> + unihash = self.rqdata.get_task_unihash(task)
>> if 'fakeroot' in taskdep and taskname in
>> taskdep['fakeroot'] and not (self.cooker.configuration.dry_run or
>> self.rqdata.setscene_enforce): if not mc in self.rq.fakeworker: try:
>> @@ -2039,10 +2121,10 @@ class RunQueueExecuteTasks(RunQueueExecute):
>> self.rq.state = runQueueFailed
>> self.stats.taskFailed()
>> return True
>> -
>> self.rq.fakeworker[mc].process.stdin.write(b"<runtask>" +
>> pickle.dumps((taskfn, task, taskname, False,
>> self.cooker.collection.get_file_appends(taskfn), taskdepdata,
>> self.rqdata.setscene_enforce)) + b"</runtask>")
>> +
>> self.rq.fakeworker[mc].process.stdin.write(b"<runtask>" +
>> pickle.dumps((taskfn, task, taskname, taskhash, unihash, False,
>> self.cooker.collection.get_file_appends(taskfn), taskdepdata,
>> self.rqdata.setscene_enforce)) + b"</runtask>")
>> self.rq.fakeworker[mc].process.stdin.flush() else:
>> - self.rq.worker[mc].process.stdin.write(b"<runtask>"
>> + pickle.dumps((taskfn, task, taskname, False,
>> self.cooker.collection.get_file_appends(taskfn), taskdepdata,
>> self.rqdata.setscene_enforce)) + b"</runtask>")
>> + self.rq.worker[mc].process.stdin.write(b"<runtask>"
>> + pickle.dumps((taskfn, task, taskname, taskhash, unihash, False,
>> self.cooker.collection.get_file_appends(taskfn), taskdepdata,
>> self.rqdata.setscene_enforce)) + b"</runtask>")
>> self.rq.worker[mc].process.stdin.flush() self.build_stamps[task] =
>> bb.build.stampfile(taskname, self.rqdata.dataCaches[mc], taskfn,
>> noextra=True) @@ -2052,30 +2134,58 @@ class
>> RunQueueExecuteTasks(RunQueueExecute): if self.can_start_task():
>> return True
>> - if self.stats.active > 0:
>> + if self.stats.active > 0 or self.sq_stats.active > 0:
>> self.rq.read_workers()
>> return self.rq.active_fds()
>>
>> + # No more tasks can be run. If we have deferred setscene
>> tasks we should run them.
>> + if self.sq_deferred:
>> + tid =
>> self.sq_deferred.pop(list(self.sq_deferred.keys())[0])
>> + logger.warning("Runqeueue deadlocked on deferred tasks,
>> forcing task %s" % tid)
>> + self.sq_task_failoutright(tid)
>> + return True
>> +
>> if len(self.failed_tids) != 0:
>> self.rq.state = runQueueFailed
>> return True
>>
>> # Sanity Checks
>> + err = self.summarise_scenequeue_errors()
>> for task in self.rqdata.runtaskentries:
>> if task not in self.runq_buildable:
>> logger.error("Task %s never buildable!", task)
>> - if task not in self.runq_running:
>> + err = True
>> + elif task not in self.runq_running:
>> logger.error("Task %s never ran!", task)
>> - if task not in self.runq_complete:
>> + err = True
>> + elif task not in self.runq_complete:
>> logger.error("Task %s never completed!", task)
>> - self.rq.state = runQueueComplete
>> + err = True
>> +
>> + if err:
>> + self.rq.state = runQueueFailed
>> + else:
>> + self.rq.state = runQueueComplete
>>
>> return True
>>
>> + def filtermcdeps(self, task, mc, deps):
>> + ret = set()
>> + for dep in deps:
>> + thismc = mc_from_tid(dep)
>> + if thismc != mc:
>> + continue
>> + ret.add(dep)
>> + return ret
>> +
>> + # We filter out multiconfig dependencies from taskdepdata we
>> pass to the tasks
>> + # as most code can't handle them
>> def build_taskdepdata(self, task):
>> taskdepdata = {}
>> - next = self.rqdata.runtaskentries[task].depends
>> + mc = mc_from_tid(task)
>> + next = self.rqdata.runtaskentries[task].depends.copy()
>> next.add(task)
>> + next = self.filtermcdeps(task, mc, next)
>> while next:
>> additional = []
>> for revdep in next:
>> @@ -2084,7 +2194,9 @@ class RunQueueExecuteTasks(RunQueueExecute):
>> deps = self.rqdata.runtaskentries[revdep].depends
>> provides =
>> self.rqdata.dataCaches[mc].fn_provides[taskfn] taskhash =
>> self.rqdata.runtaskentries[revdep].hash
>> - taskdepdata[revdep] = [pn, taskname, fn, deps,
>> provides, taskhash]
>> + unihash = self.rqdata.runtaskentries[revdep].unihash
>> + deps = self.filtermcdeps(task, mc, deps)
>> + taskdepdata[revdep] = [pn, taskname, fn, deps,
>> provides, taskhash, unihash] for revdep2 in deps:
>> if revdep2 not in taskdepdata:
>> additional.append(revdep2)
>> @@ -2093,269 +2205,197 @@ class RunQueueExecuteTasks(RunQueueExecute):
>> #bb.note("Task %s: " % task + str(taskdepdata).replace("],
>> ", "],\n")) return taskdepdata
>>
>> -class RunQueueExecuteScenequeue(RunQueueExecute):
>> - def __init__(self, rq):
>> - RunQueueExecute.__init__(self, rq)
>> -
>> - self.scenequeue_covered = set()
>> - self.scenequeue_notcovered = set()
>> - self.scenequeue_notneeded = set()
>> + def update_holdofftasks(self):
>>
>> - # If we don't have any setscene functions, skip this step
>> - if len(self.rqdata.runq_setscene_tids) == 0:
>> - rq.scenequeue_covered = set()
>> - rq.scenequeue_notcovered = set()
>> - rq.state = runQueueRunInit
>> + if not self.holdoff_need_update:
>> return
>>
>> - self.stats =
>> RunQueueStats(len(self.rqdata.runq_setscene_tids))
>> + notcovered = set(self.scenequeue_notcovered)
>> + notcovered |= self.cantskip
>> + for tid in self.scenequeue_notcovered:
>> + notcovered |= self.sqdata.sq_covered_tasks[tid]
>> + notcovered |=
>> self.sqdata.unskippable.difference(self.rqdata.runq_setscene_tids)
>> + notcovered.intersection_update(self.tasks_scenequeue_done)
>>
>> - sq_revdeps = {}
>> - sq_revdeps_new = {}
>> - sq_revdeps_squash = {}
>> - self.sq_harddeps = {}
>> - self.stamps = {}
>> -
>> - # We need to construct a dependency graph for the setscene
>> functions. Intermediate
>> - # dependencies between the setscene tasks only complicate
>> the code. This code
>> - # therefore aims to collapse the huge runqueue dependency
>> tree into a smaller one
>> - # only containing the setscene functions.
>> -
>> - self.rqdata.init_progress_reporter.next_stage()
>> -
>> - # First process the chains up to the first setscene task.
>> - endpoints = {}
>> - for tid in self.rqdata.runtaskentries:
>> - sq_revdeps[tid] =
>> copy.copy(self.rqdata.runtaskentries[tid].revdeps)
>> - sq_revdeps_new[tid] = set()
>> - if (len(sq_revdeps[tid]) == 0) and tid not in
>> self.rqdata.runq_setscene_tids:
>> - #bb.warn("Added endpoint %s" % (tid))
>> - endpoints[tid] = set()
>> + covered = set(self.scenequeue_covered)
>> + for tid in self.scenequeue_covered:
>> + covered |= self.sqdata.sq_covered_tasks[tid]
>> + covered.difference_update(notcovered)
>> + covered.intersection_update(self.tasks_scenequeue_done)
>>
>> - self.rqdata.init_progress_reporter.next_stage()
>> + for tid in notcovered | covered:
>> + if len(self.rqdata.runtaskentries[tid].depends) == 0:
>> + self.setbuildable(tid)
>> + elif
>> self.rqdata.runtaskentries[tid].depends.issubset(self.runq_complete):
>> + self.setbuildable(tid)
>>
>> - # Secondly process the chains between setscene tasks.
>> - for tid in self.rqdata.runq_setscene_tids:
>> - #bb.warn("Added endpoint 2 %s" % (tid))
>> - for dep in self.rqdata.runtaskentries[tid].depends:
>> - if tid in sq_revdeps[dep]:
>> - sq_revdeps[dep].remove(tid)
>> - if dep not in endpoints:
>> - endpoints[dep] = set()
>> - #bb.warn(" Added endpoint 3 %s" % (dep))
>> - endpoints[dep].add(tid)
>> -
>> - self.rqdata.init_progress_reporter.next_stage()
>> -
>> - def process_endpoints(endpoints):
>> - newendpoints = {}
>> - for point, task in endpoints.items():
>> - tasks = set()
>> - if task:
>> - tasks |= task
>> - if sq_revdeps_new[point]:
>> - tasks |= sq_revdeps_new[point]
>> - sq_revdeps_new[point] = set()
>> - if point in self.rqdata.runq_setscene_tids:
>> - sq_revdeps_new[point] = tasks
>> - tasks = set()
>> - continue
>> - for dep in self.rqdata.runtaskentries[point].depends:
>> - if point in sq_revdeps[dep]:
>> - sq_revdeps[dep].remove(point)
>> - if tasks:
>> - sq_revdeps_new[dep] |= tasks
>> - if len(sq_revdeps[dep]) == 0 and dep not in
>> self.rqdata.runq_setscene_tids:
>> - newendpoints[dep] = task
>> - if len(newendpoints) != 0:
>> - process_endpoints(newendpoints)
>> -
>> - process_endpoints(endpoints)
>> -
>> - self.rqdata.init_progress_reporter.next_stage()
>> -
>> - # Build a list of setscene tasks which are "unskippable"
>> - # These are direct endpoints referenced by the build
>> - endpoints2 = {}
>> - sq_revdeps2 = {}
>> - sq_revdeps_new2 = {}
>> - def process_endpoints2(endpoints):
>> - newendpoints = {}
>> - for point, task in endpoints.items():
>> - tasks = set([point])
>> - if task:
>> - tasks |= task
>> - if sq_revdeps_new2[point]:
>> - tasks |= sq_revdeps_new2[point]
>> - sq_revdeps_new2[point] = set()
>> - if point in self.rqdata.runq_setscene_tids:
>> - sq_revdeps_new2[point] = tasks
>> - for dep in self.rqdata.runtaskentries[point].depends:
>> - if point in sq_revdeps2[dep]:
>> - sq_revdeps2[dep].remove(point)
>> - if tasks:
>> - sq_revdeps_new2[dep] |= tasks
>> - if (len(sq_revdeps2[dep]) == 0 or
>> len(sq_revdeps_new2[dep]) != 0) and dep not in
>> self.rqdata.runq_setscene_tids:
>> - newendpoints[dep] = tasks
>> - if len(newendpoints) != 0:
>> - process_endpoints2(newendpoints)
>> - for tid in self.rqdata.runtaskentries:
>> - sq_revdeps2[tid] =
>> copy.copy(self.rqdata.runtaskentries[tid].revdeps)
>> - sq_revdeps_new2[tid] = set()
>> - if (len(sq_revdeps2[tid]) == 0) and tid not in
>> self.rqdata.runq_setscene_tids:
>> - endpoints2[tid] = set()
>> - process_endpoints2(endpoints2)
>> - self.unskippable = []
>> - for tid in self.rqdata.runq_setscene_tids:
>> - if sq_revdeps_new2[tid]:
>> - self.unskippable.append(tid)
>> + self.tasks_covered = covered
>> + self.tasks_notcovered = notcovered
>>
>> -
>> self.rqdata.init_progress_reporter.next_stage(len(self.rqdata.runtaskentries))
>> + self.holdoff_tasks = set()
>>
>> - for taskcounter, tid in
>> enumerate(self.rqdata.runtaskentries):
>> - if tid in self.rqdata.runq_setscene_tids:
>> - deps = set()
>> - for dep in sq_revdeps_new[tid]:
>> - deps.add(dep)
>> - sq_revdeps_squash[tid] = deps
>> - elif len(sq_revdeps_new[tid]) != 0:
>> - bb.msg.fatal("RunQueue", "Something went badly wrong
>> during scenequeue generation, aborting. Please report this problem.")
>> - self.rqdata.init_progress_reporter.update(taskcounter)
>> -
>> - self.rqdata.init_progress_reporter.next_stage()
>> -
>> - # Resolve setscene inter-task dependencies
>> - # e.g. do_sometask_setscene[depends] =
>> "targetname:do_someothertask_setscene"
>> - # Note that anything explicitly depended upon will have its
>> reverse dependencies removed to avoid circular dependencies for tid
>> in self.rqdata.runq_setscene_tids:
>> - (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
>> - realtid = tid + "_setscene"
>> - idepends =
>> self.rqdata.taskData[mc].taskentries[realtid].idepends
>> - self.stamps[tid] = bb.build.stampfile(taskname +
>> "_setscene", self.rqdata.dataCaches[mc], taskfn, noextra=True)
>> - for (depname, idependtask) in idepends:
>> -
>> - if depname not in
>> self.rqdata.taskData[mc].build_targets:
>> - continue
>> + if tid not in self.scenequeue_covered and tid not in
>> self.scenequeue_notcovered:
>> + self.holdoff_tasks.add(tid)
>>
>> - depfn =
>> self.rqdata.taskData[mc].build_targets[depname][0]
>> - if depfn is None:
>> - continue
>> - deptid = depfn + ":" +
>> idependtask.replace("_setscene", "")
>> - if deptid not in self.rqdata.runtaskentries:
>> - bb.msg.fatal("RunQueue", "Task %s depends
>> upon non-existent task %s:%s" % (realtid, depfn, idependtask)) -
>> - if not deptid in self.sq_harddeps:
>> - self.sq_harddeps[deptid] = set()
>> - self.sq_harddeps[deptid].add(tid)
>> -
>> - sq_revdeps_squash[tid].add(deptid)
>> - # Have to zero this to avoid circular
>> dependencies
>> - sq_revdeps_squash[deptid] = set()
>> -
>> - self.rqdata.init_progress_reporter.next_stage()
>> -
>> - for task in self.sq_harddeps:
>> - for dep in self.sq_harddeps[task]:
>> - sq_revdeps_squash[dep].add(task)
>> -
>> - self.rqdata.init_progress_reporter.next_stage()
>> -
>> - #for tid in sq_revdeps_squash:
>> - # for dep in sq_revdeps_squash[tid]:
>> - # data = data + "\n %s" % dep
>> - # bb.warn("Task %s_setscene: is %s " % (tid, data
>> -
>> - self.sq_deps = {}
>> - self.sq_revdeps = sq_revdeps_squash
>> - self.sq_revdeps2 = copy.deepcopy(self.sq_revdeps)
>> -
>> - for tid in self.sq_revdeps:
>> - self.sq_deps[tid] = set()
>> - for tid in self.sq_revdeps:
>> - for dep in self.sq_revdeps[tid]:
>> - self.sq_deps[dep].add(tid)
>> -
>> - self.rqdata.init_progress_reporter.next_stage()
>> -
>> - for tid in self.sq_revdeps:
>> - if len(self.sq_revdeps[tid]) == 0:
>> - self.runq_buildable.add(tid)
>> -
>> - self.rqdata.init_progress_reporter.finish()
>> -
>> - self.outrightfail = []
>> - if self.rq.hashvalidate:
>> - sq_hash = []
>> - sq_hashfn = []
>> - sq_fn = []
>> - sq_taskname = []
>> - sq_task = []
>> - noexec = []
>> - stamppresent = []
>> - for tid in self.sq_revdeps:
>> - (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
>> + for tid in self.holdoff_tasks.copy():
>> + for dep in self.sqdata.sq_covered_tasks[tid]:
>> + if dep not in self.runq_complete:
>> + self.holdoff_tasks.add(dep)
>>
>> - taskdep =
>> self.rqdata.dataCaches[mc].task_deps[taskfn]
>> + self.holdoff_need_update = False
>>
>> - if 'noexec' in taskdep and taskname in
>> taskdep['noexec']:
>> - noexec.append(tid)
>> - self.task_skip(tid)
>> - bb.build.make_stamp(taskname + "_setscene",
>> self.rqdata.dataCaches[mc], taskfn)
>> - continue
>> + def process_possible_migrations(self):
>>
>> - if self.rq.check_stamp_task(tid, taskname +
>> "_setscene", cache=self.stampcache):
>> - logger.debug(2, 'Setscene stamp current for task
>> %s', tid)
>> - stamppresent.append(tid)
>> - self.task_skip(tid)
>> - continue
>> + changed = set()
>> + for tid, unihash in self.updated_taskhash_queue.copy():
>> + if tid in self.runq_running and tid not in
>> self.runq_complete:
>> + continue
>>
>> - if self.rq.check_stamp_task(tid, taskname, recurse =
>> True, cache=self.stampcache):
>> - logger.debug(2, 'Normal stamp current for task
>> %s', tid)
>> - stamppresent.append(tid)
>> - self.task_skip(tid)
>> - continue
>> + self.updated_taskhash_queue.remove((tid, unihash))
>> +
>> + if unihash != self.rqdata.runtaskentries[tid].unihash:
>> + logger.info("Task %s unihash changed to %s" % (tid,
>> unihash))
>> + self.rqdata.runtaskentries[tid].unihash = unihash
>> + bb.parse.siggen.set_unihash(tid, unihash)
>> +
>> + # Work out all tasks which depend on this one
>> + total = set()
>> + next = set(self.rqdata.runtaskentries[tid].revdeps)
>> + while next:
>> + current = next.copy()
>> + total = total |next
>> + next = set()
>> + for ntid in current:
>> + next |=
>> self.rqdata.runtaskentries[ntid].revdeps
>> + next.difference_update(total)
>> +
>> + # Now iterate those tasks in dependency order to
>> regenerate their taskhash/unihash
>> + done = set()
>> + next = set(self.rqdata.runtaskentries[tid].revdeps)
>> + while next:
>> + current = next.copy()
>> + next = set()
>> + for tid in current:
>> + if not
>> self.rqdata.runtaskentries[tid].depends.isdisjoint(total):
>> + continue
>> + procdep = []
>> + for dep in
>> self.rqdata.runtaskentries[tid].depends:
>> + procdep.append(dep)
>> + orighash =
>> self.rqdata.runtaskentries[tid].hash
>> + self.rqdata.runtaskentries[tid].hash =
>> bb.parse.siggen.get_taskhash(tid, procdep,
>> self.rqdata.dataCaches[mc_from_tid(tid)])
>> + origuni =
>> self.rqdata.runtaskentries[tid].unihash
>> + self.rqdata.runtaskentries[tid].unihash =
>> bb.parse.siggen.get_unihash(tid)
>> + logger.debug(1, "Task %s hash changes:
>> %s->%s %s->%s" % (tid, orighash,
>> self.rqdata.runtaskentries[tid].hash, origuni,
>> self.rqdata.runtaskentries[tid].unihash))
>> + next |=
>> self.rqdata.runtaskentries[tid].revdeps
>> + changed.add(tid)
>> + total.remove(tid)
>> + next.intersection_update(total)
>> +
>> + if changed:
>> + for mc in self.rq.worker:
>> +
>> self.rq.worker[mc].process.stdin.write(b"<newtaskhashes>" +
>> pickle.dumps(bb.parse.siggen.get_taskhashes()) + b"</newtaskhashes>")
>> + for mc in self.rq.fakeworker:
>> +
>> self.rq.fakeworker[mc].process.stdin.write(b"<newtaskhashes>" +
>> pickle.dumps(bb.parse.siggen.get_taskhashes()) + b"</newtaskhashes>")
>> +
>> + logger.debug(1, pprint.pformat("Tasks changed:\n%s" %
>> (changed))) +
>> + for tid in changed:
>> + if tid not in self.rqdata.runq_setscene_tids:
>> + continue
>> + if tid in self.runq_running:
>> + continue
>> + if tid in self.scenequeue_covered:
>> + # Potentially risky, should we report this hash as a
>> match?
>> + logger.info("Already covered setscene for %s so
>> ignoring rehash" % (tid))
>> + continue
>> + if tid not in self.pending_migrations:
>> + self.pending_migrations.add(tid)
>> +
>> + for tid in self.pending_migrations.copy():
>> + valid = True
>> + # Check no tasks this covers are running
>> + for dep in self.sqdata.sq_covered_tasks[tid]:
>> + if dep in self.runq_running and dep not in
>> self.runq_complete:
>> + logger.debug(2, "Task %s is running which blocks
>> setscene for %s from running" % (dep, tid))
>> + valid = False
>> + break
>> + if not valid:
>> + continue
>>
>> - sq_fn.append(fn)
>> -
>> sq_hashfn.append(self.rqdata.dataCaches[mc].hashfn[taskfn])
>> - sq_hash.append(self.rqdata.runtaskentries[tid].hash)
>> - sq_taskname.append(taskname)
>> - sq_task.append(tid)
>> + self.pending_migrations.remove(tid)
>> + changed = True
>>
>> -
>> self.cooker.data.setVar("BB_SETSCENE_STAMPCURRENT_COUNT",
>> len(stamppresent))
>> + if tid in self.tasks_scenequeue_done:
>> + self.tasks_scenequeue_done.remove(tid)
>> + for dep in self.sqdata.sq_covered_tasks[tid]:
>> + if dep not in self.runq_complete:
>> + if dep in self.tasks_scenequeue_done and dep not
>> in self.sqdata.unskippable:
>> + self.tasks_scenequeue_done.remove(dep)
>> +
>> + if tid in self.sq_buildable:
>> + self.sq_buildable.remove(tid)
>> + if tid in self.sq_running:
>> + self.sq_running.remove(tid)
>> + if
>> self.sqdata.sq_revdeps[tid].issubset(self.scenequeue_covered |
>> self.scenequeue_notcovered):
>> + if tid not in self.sq_buildable:
>> + self.sq_buildable.add(tid)
>> + if len(self.sqdata.sq_revdeps[tid]) == 0:
>> + self.sq_buildable.add(tid)
>> +
>> + if tid in self.sqdata.outrightfail:
>> + self.sqdata.outrightfail.remove(tid)
>> + if tid in self.scenequeue_notcovered:
>> + self.scenequeue_notcovered.remove(tid)
>> + if tid in self.scenequeue_covered:
>> + self.scenequeue_covered.remove(tid)
>> + if tid in self.scenequeue_notneeded:
>> + self.scenequeue_notneeded.remove(tid)
>>
>> - call = self.rq.hashvalidate + "(sq_fn, sq_task, sq_hash,
>> sq_hashfn, d)"
>> - locs = { "sq_fn" : sq_fn, "sq_task" : sq_taskname,
>> "sq_hash" : sq_hash, "sq_hashfn" : sq_hashfn, "d" : self.cooker.data }
>> - valid = bb.utils.better_eval(call, locs)
>> + (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
>> + self.sqdata.stamps[tid] = bb.build.stampfile(taskname +
>> "_setscene", self.rqdata.dataCaches[mc], taskfn, noextra=True)
>> - self.cooker.data.delVar("BB_SETSCENE_STAMPCURRENT_COUNT")
>> + if tid in self.stampcache:
>> + del self.stampcache[tid]
>>
>> - valid_new = stamppresent
>> - for v in valid:
>> - valid_new.append(sq_task[v])
>> + if tid in self.build_stamps:
>> + del self.build_stamps[tid]
>>
>> - for tid in self.sq_revdeps:
>> - if tid not in valid_new and tid not in noexec:
>> - logger.debug(2, 'No package found, so skipping
>> setscene task %s', tid)
>> - self.outrightfail.append(tid)
>> + logger.info("Setscene task %s now valid and being rerun"
>> % tid)
>> + self.sqdone = False
>> + update_scenequeue_data([tid], self.sqdata, self.rqdata,
>> self.rq, self.cooker, self.stampcache, self)
>> - logger.info('Executing SetScene Tasks')
>> + if changed:
>> + self.holdoff_need_update = True
>>
>> - self.rq.state = runQueueSceneRun
>> + def scenequeue_updatecounters(self, task, fail=False):
>>
>> - def scenequeue_updatecounters(self, task, fail = False):
>> - for dep in self.sq_deps[task]:
>> - if fail and task in self.sq_harddeps and dep in
>> self.sq_harddeps[task]:
>> + for dep in sorted(self.sqdata.sq_deps[task]):
>> + if fail and task in self.sqdata.sq_harddeps and dep in
>> self.sqdata.sq_harddeps[task]: logger.debug(2, "%s was unavailable
>> and is a hard dependency of %s so skipping" % (task, dep))
>> - self.scenequeue_updatecounters(dep, fail)
>> + self.sq_task_failoutright(dep)
>> continue
>> - if task not in self.sq_revdeps2[dep]:
>> - # May already have been removed by the fail case
>> above
>> - continue
>> - self.sq_revdeps2[dep].remove(task)
>> - if len(self.sq_revdeps2[dep]) == 0:
>> - self.runq_buildable.add(dep)
>> + if
>> self.sqdata.sq_revdeps[dep].issubset(self.scenequeue_covered |
>> self.scenequeue_notcovered):
>> + if dep not in self.sq_buildable:
>> + self.sq_buildable.add(dep)
>>
>> - def task_completeoutright(self, task):
>> + next = set([task])
>> + while next:
>> + new = set()
>> + for t in sorted(next):
>> + self.tasks_scenequeue_done.add(t)
>> + # Look down the dependency chain for non-setscene
>> things which this task depends on
>> + # and mark as 'done'
>> + for dep in self.rqdata.runtaskentries[t].depends:
>> + if dep in self.rqdata.runq_setscene_tids or dep
>> in self.tasks_scenequeue_done:
>> + continue
>> + if
>> self.rqdata.runtaskentries[dep].revdeps.issubset(self.tasks_scenequeue_done):
>> + new.add(dep)
>> + next = new
>> +
>> + self.holdoff_need_update = True
>> +
>> + def sq_task_completeoutright(self, task):
>> """
>> Mark a task as completed
>> Look at the reverse dependencies and mark any task with
>> @@ -2366,7 +2406,7 @@ class
>> RunQueueExecuteScenequeue(RunQueueExecute):
>> self.scenequeue_covered.add(task) self.scenequeue_updatecounters(task)
>>
>> - def check_taskfail(self, task):
>> + def sq_check_taskfail(self, task):
>> if self.rqdata.setscenewhitelist is not None:
>> realtask = task.split('_setscene')[0]
>> (mc, fn, taskname, taskfn) = split_tid_mcfn(realtask)
>> @@ -2375,130 +2415,34 @@ class
>> RunQueueExecuteScenequeue(RunQueueExecute): logger.error('Task %s.%s
>> failed' % (pn, taskname + "_setscene")) self.rq.state =
>> runQueueCleanUp
>> - def task_complete(self, task):
>> - self.stats.taskCompleted()
>> - bb.event.fire(sceneQueueTaskCompleted(task, self.stats,
>> self.rq), self.cfgData)
>> - self.task_completeoutright(task)
>> + def sq_task_complete(self, task):
>> + self.sq_stats.taskCompleted()
>> + bb.event.fire(sceneQueueTaskCompleted(task, self.sq_stats,
>> self.rq), self.cfgData)
>> + self.sq_task_completeoutright(task)
>>
>> - def task_fail(self, task, result):
>> - self.stats.taskFailed()
>> - bb.event.fire(sceneQueueTaskFailed(task, self.stats, result,
>> self), self.cfgData)
>> + def sq_task_fail(self, task, result):
>> + self.sq_stats.taskFailed()
>> + bb.event.fire(sceneQueueTaskFailed(task, self.sq_stats,
>> result, self), self.cfgData) self.scenequeue_notcovered.add(task)
>> self.scenequeue_updatecounters(task, True)
>> - self.check_taskfail(task)
>> + self.sq_check_taskfail(task)
>>
>> - def task_failoutright(self, task):
>> - self.runq_running.add(task)
>> - self.runq_buildable.add(task)
>> - self.stats.taskSkipped()
>> - self.stats.taskCompleted()
>> + def sq_task_failoutright(self, task):
>> + self.sq_running.add(task)
>> + self.sq_buildable.add(task)
>> + self.sq_stats.taskSkipped()
>> + self.sq_stats.taskCompleted()
>> self.scenequeue_notcovered.add(task)
>> self.scenequeue_updatecounters(task, True)
>>
>> - def task_skip(self, task):
>> - self.runq_running.add(task)
>> - self.runq_buildable.add(task)
>> - self.task_completeoutright(task)
>> - self.stats.taskSkipped()
>> - self.stats.taskCompleted()
>> -
>> - def execute(self):
>> - """
>> - Run the tasks in a queue prepared by prepare_runqueue
>> - """
>> -
>> - self.rq.read_workers()
>> -
>> - task = None
>> - if self.can_start_task():
>> - # Find the next setscene to run
>> - for nexttask in self.rqdata.runq_setscene_tids:
>> - if nexttask in self.runq_buildable and nexttask not
>> in self.runq_running and self.stamps[nexttask] not in
>> self.build_stamps.values():
>> - if nexttask in self.unskippable:
>> - logger.debug(2, "Setscene task %s is
>> unskippable" % nexttask)
>> - if nexttask not in self.unskippable and
>> len(self.sq_revdeps[nexttask]) > 0 and
>> self.sq_revdeps[nexttask].issubset(self.scenequeue_covered) and
>> self.check_dependencies(nexttask, self.sq_revdeps[nexttask], True):
>> - fn = fn_from_tid(nexttask)
>> - foundtarget = False
>> -
>> - if nexttask in self.rqdata.target_tids:
>> - foundtarget = True
>> - if not foundtarget:
>> - logger.debug(2, "Skipping setscene for
>> task %s" % nexttask)
>> - self.task_skip(nexttask)
>> - self.scenequeue_notneeded.add(nexttask)
>> - return True
>> - if nexttask in self.outrightfail:
>> - self.task_failoutright(nexttask)
>> - return True
>> - task = nexttask
>> - break
>> - if task is not None:
>> - (mc, fn, taskname, taskfn) = split_tid_mcfn(task)
>> - taskname = taskname + "_setscene"
>> - if self.rq.check_stamp_task(task,
>> taskname_from_tid(task), recurse = True, cache=self.stampcache):
>> - logger.debug(2, 'Stamp for underlying task %s is
>> current, so skipping setscene variant', task)
>> - self.task_failoutright(task)
>> - return True
>> -
>> - if self.cooker.configuration.force:
>> - if task in self.rqdata.target_tids:
>> - self.task_failoutright(task)
>> - return True
>> -
>> - if self.rq.check_stamp_task(task, taskname,
>> cache=self.stampcache):
>> - logger.debug(2, 'Setscene stamp current task %s, so
>> skip it and its dependencies', task)
>> - self.task_skip(task)
>> - return True
>> -
>> - startevent = sceneQueueTaskStarted(task, self.stats,
>> self.rq)
>> - bb.event.fire(startevent, self.cfgData)
>> + def sq_task_skip(self, task):
>> + self.sq_running.add(task)
>> + self.sq_buildable.add(task)
>> + self.sq_task_completeoutright(task)
>> + self.sq_stats.taskSkipped()
>> + self.sq_stats.taskCompleted()
>>
>> - taskdepdata = self.build_taskdepdata(task)
>> -
>> - taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn]
>> - if 'fakeroot' in taskdep and taskname in
>> taskdep['fakeroot'] and not self.cooker.configuration.dry_run:
>> - if not mc in self.rq.fakeworker:
>> - self.rq.start_fakeworker(self, mc)
>> -
>> self.rq.fakeworker[mc].process.stdin.write(b"<runtask>" +
>> pickle.dumps((taskfn, task, taskname, True,
>> self.cooker.collection.get_file_appends(taskfn), taskdepdata, False))
>> + b"</runtask>")
>> - self.rq.fakeworker[mc].process.stdin.flush()
>> - else:
>> - self.rq.worker[mc].process.stdin.write(b"<runtask>"
>> + pickle.dumps((taskfn, task, taskname, True,
>> self.cooker.collection.get_file_appends(taskfn), taskdepdata, False))
>> + b"</runtask>")
>> - self.rq.worker[mc].process.stdin.flush()
>> -
>> - self.build_stamps[task] = bb.build.stampfile(taskname,
>> self.rqdata.dataCaches[mc], taskfn, noextra=True)
>> - self.build_stamps2.append(self.build_stamps[task])
>> - self.runq_running.add(task)
>> - self.stats.taskActive()
>> - if self.can_start_task():
>> - return True
>> -
>> - if self.stats.active > 0:
>> - self.rq.read_workers()
>> - return self.rq.active_fds()
>> -
>> - #for tid in self.sq_revdeps:
>> - # if tid not in self.runq_running:
>> - # buildable = tid in self.runq_buildable
>> - # revdeps = self.sq_revdeps[tid]
>> - # bb.warn("Found we didn't run %s %s %s" % (tid,
>> buildable, str(revdeps))) -
>> - self.rq.scenequeue_covered = self.scenequeue_covered
>> - self.rq.scenequeue_notcovered = self.scenequeue_notcovered
>> -
>> - logger.debug(1, 'We can skip tasks %s',
>> "\n".join(sorted(self.rq.scenequeue_covered))) -
>> - self.rq.state = runQueueRunInit
>> -
>> - completeevent = sceneQueueComplete(self.stats, self.rq)
>> - bb.event.fire(completeevent, self.cfgData)
>> -
>> - return True
>> -
>> - def runqueue_process_waitpid(self, task, status):
>> - RunQueueExecute.runqueue_process_waitpid(self, task, status)
>> -
>> -
>> - def build_taskdepdata(self, task):
>> + def sq_build_taskdepdata(self, task):
>> def getsetscenedeps(tid):
>> deps = set()
>> (mc, fn, taskname, _) = split_tid_mcfn(tid)
>> @@ -2526,7 +2470,8 @@ class
>> RunQueueExecuteScenequeue(RunQueueExecute): deps =
>> getsetscenedeps(revdep) provides =
>> self.rqdata.dataCaches[mc].fn_provides[taskfn] taskhash =
>> self.rqdata.runtaskentries[revdep].hash
>> - taskdepdata[revdep] = [pn, taskname, fn, deps,
>> provides, taskhash]
>> + unihash = self.rqdata.runtaskentries[revdep].unihash
>> + taskdepdata[revdep] = [pn, taskname, fn, deps,
>> provides, taskhash, unihash] for revdep2 in deps:
>> if revdep2 not in taskdepdata:
>> additional.append(revdep2)
>> @@ -2535,6 +2480,279 @@ class
>> RunQueueExecuteScenequeue(RunQueueExecute): #bb.note("Task %s: " %
>> task + str(taskdepdata).replace("], ", "],\n")) return taskdepdata
>>
>> + def check_setscenewhitelist(self, tid):
>> + # Check task that is going to run against the whitelist
>> + (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
>> + # Ignore covered tasks
>> + if tid in self.tasks_covered:
>> + return False
>> + # Ignore stamped tasks
>> + if self.rq.check_stamp_task(tid, taskname,
>> cache=self.stampcache):
>> + return False
>> + # Ignore noexec tasks
>> + taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn]
>> + if 'noexec' in taskdep and taskname in taskdep['noexec']:
>> + return False
>> +
>> + pn = self.rqdata.dataCaches[mc].pkg_fn[taskfn]
>> + if not check_setscene_enforce_whitelist(pn, taskname,
>> self.rqdata.setscenewhitelist):
>> + if tid in self.rqdata.runq_setscene_tids:
>> + msg = 'Task %s.%s attempted to execute unexpectedly
>> and should have been setscened' % (pn, taskname)
>> + else:
>> + msg = 'Task %s.%s attempted to execute unexpectedly'
>> % (pn, taskname)
>> + logger.error(msg + '\nThis is usually due to missing
>> setscene tasks. Those missing in this build were: %s' %
>> pprint.pformat(self.scenequeue_notcovered))
>> + return True
>> + return False
>> +
>> +class SQData(object):
>> + def __init__(self):
>> + # SceneQueue dependencies
>> + self.sq_deps = {}
>> + # SceneQueue reverse dependencies
>> + self.sq_revdeps = {}
>> + # Injected inter-setscene task dependencies
>> + self.sq_harddeps = {}
>> + # Cache of stamp files so duplicates can't run in parallel
>> + self.stamps = {}
>> + # Setscene tasks directly depended upon by the build
>> + self.unskippable = set()
>> + # List of setscene tasks which aren't present
>> + self.outrightfail = set()
>> + # A list of normal tasks a setscene task covers
>> + self.sq_covered_tasks = {}
>> +
>> +def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache,
>> sqrq): +
>> + sq_revdeps = {}
>> + sq_revdeps_squash = {}
>> + sq_collated_deps = {}
>> +
>> + # We need to construct a dependency graph for the setscene
>> functions. Intermediate
>> + # dependencies between the setscene tasks only complicate the
>> code. This code
>> + # therefore aims to collapse the huge runqueue dependency tree
>> into a smaller one
>> + # only containing the setscene functions.
>> +
>> + rqdata.init_progress_reporter.next_stage()
>> +
>> + # First process the chains up to the first setscene task.
>> + endpoints = {}
>> + for tid in rqdata.runtaskentries:
>> + sq_revdeps[tid] =
>> copy.copy(rqdata.runtaskentries[tid].revdeps)
>> + sq_revdeps_squash[tid] = set()
>> + if (len(sq_revdeps[tid]) == 0) and tid not in
>> rqdata.runq_setscene_tids:
>> + #bb.warn("Added endpoint %s" % (tid))
>> + endpoints[tid] = set()
>> +
>> + rqdata.init_progress_reporter.next_stage()
>> +
>> + # Secondly process the chains between setscene tasks.
>> + for tid in rqdata.runq_setscene_tids:
>> + sq_collated_deps[tid] = set()
>> + #bb.warn("Added endpoint 2 %s" % (tid))
>> + for dep in rqdata.runtaskentries[tid].depends:
>> + if tid in sq_revdeps[dep]:
>> + sq_revdeps[dep].remove(tid)
>> + if dep not in endpoints:
>> + endpoints[dep] = set()
>> + #bb.warn(" Added endpoint 3 %s" % (dep))
>> + endpoints[dep].add(tid)
>> +
>> + rqdata.init_progress_reporter.next_stage()
>> +
>> + def process_endpoints(endpoints):
>> + newendpoints = {}
>> + for point, task in endpoints.items():
>> + tasks = set()
>> + if task:
>> + tasks |= task
>> + if sq_revdeps_squash[point]:
>> + tasks |= sq_revdeps_squash[point]
>> + if point not in rqdata.runq_setscene_tids:
>> + for t in tasks:
>> + sq_collated_deps[t].add(point)
>> + sq_revdeps_squash[point] = set()
>> + if point in rqdata.runq_setscene_tids:
>> + sq_revdeps_squash[point] = tasks
>> + tasks = set()
>> + continue
>> + for dep in rqdata.runtaskentries[point].depends:
>> + if point in sq_revdeps[dep]:
>> + sq_revdeps[dep].remove(point)
>> + if tasks:
>> + sq_revdeps_squash[dep] |= tasks
>> + if len(sq_revdeps[dep]) == 0 and dep not in
>> rqdata.runq_setscene_tids:
>> + newendpoints[dep] = task
>> + if len(newendpoints) != 0:
>> + process_endpoints(newendpoints)
>> +
>> + process_endpoints(endpoints)
>> +
>> + rqdata.init_progress_reporter.next_stage()
>> +
>> + # Build a list of tasks which are "unskippable"
>> + # These are direct endpoints referenced by the build upto and
>> including setscene tasks
>> + # Take the build endpoints (no revdeps) and find the sstate
>> tasks they depend upon
>> + new = True
>> + for tid in rqdata.runtaskentries:
>> + if len(rqdata.runtaskentries[tid].revdeps) == 0:
>> + sqdata.unskippable.add(tid)
>> + sqdata.unskippable |= sqrq.cantskip
>> + while new:
>> + new = False
>> + orig = sqdata.unskippable.copy()
>> + for tid in sorted(orig, reverse=True):
>> + if tid in rqdata.runq_setscene_tids:
>> + continue
>> + if len(rqdata.runtaskentries[tid].depends) == 0:
>> + # These are tasks which have no setscene tasks in
>> their chain, need to mark as directly buildable
>> + sqrq.setbuildable(tid)
>> + sqdata.unskippable |= rqdata.runtaskentries[tid].depends
>> + if sqdata.unskippable != orig:
>> + new = True
>> +
>> + sqrq.tasks_scenequeue_done |=
>> sqdata.unskippable.difference(rqdata.runq_setscene_tids) +
>> +
>> rqdata.init_progress_reporter.next_stage(len(rqdata.runtaskentries)) +
>> + # Sanity check all dependencies could be changed to setscene
>> task references
>> + for taskcounter, tid in enumerate(rqdata.runtaskentries):
>> + if tid in rqdata.runq_setscene_tids:
>> + pass
>> + elif len(sq_revdeps_squash[tid]) != 0:
>> + bb.msg.fatal("RunQueue", "Something went badly wrong
>> during scenequeue generation, aborting. Please report this problem.")
>> + else:
>> + del sq_revdeps_squash[tid]
>> + rqdata.init_progress_reporter.update(taskcounter)
>> +
>> + rqdata.init_progress_reporter.next_stage()
>> +
>> + # Resolve setscene inter-task dependencies
>> + # e.g. do_sometask_setscene[depends] =
>> "targetname:do_someothertask_setscene"
>> + # Note that anything explicitly depended upon will have its
>> reverse dependencies removed to avoid circular dependencies
>> + for tid in rqdata.runq_setscene_tids:
>> + (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
>> + realtid = tid + "_setscene"
>> + idepends = rqdata.taskData[mc].taskentries[realtid].idepends
>> + sqdata.stamps[tid] = bb.build.stampfile(taskname +
>> "_setscene", rqdata.dataCaches[mc], taskfn, noextra=True)
>> + for (depname, idependtask) in idepends:
>> +
>> + if depname not in rqdata.taskData[mc].build_targets:
>> + continue
>> +
>> + depfn = rqdata.taskData[mc].build_targets[depname][0]
>> + if depfn is None:
>> + continue
>> + deptid = depfn + ":" + idependtask.replace("_setscene",
>> "")
>> + if deptid not in rqdata.runtaskentries:
>> + bb.msg.fatal("RunQueue", "Task %s depends upon
>> non-existent task %s:%s" % (realtid, depfn, idependtask)) +
>> + if not deptid in sqdata.sq_harddeps:
>> + sqdata.sq_harddeps[deptid] = set()
>> + sqdata.sq_harddeps[deptid].add(tid)
>> +
>> + sq_revdeps_squash[tid].add(deptid)
>> + # Have to zero this to avoid circular dependencies
>> + sq_revdeps_squash[deptid] = set()
>> +
>> + rqdata.init_progress_reporter.next_stage()
>> +
>> + for task in sqdata.sq_harddeps:
>> + for dep in sqdata.sq_harddeps[task]:
>> + sq_revdeps_squash[dep].add(task)
>> +
>> + rqdata.init_progress_reporter.next_stage()
>> +
>> + #for tid in sq_revdeps_squash:
>> + # data = ""
>> + # for dep in sq_revdeps_squash[tid]:
>> + # data = data + "\n %s" % dep
>> + # bb.warn("Task %s_setscene: is %s " % (tid, data))
>> +
>> + sqdata.sq_revdeps = sq_revdeps_squash
>> + sqdata.sq_covered_tasks = sq_collated_deps
>> +
>> + # Build reverse version of revdeps to populate deps structure
>> + for tid in sqdata.sq_revdeps:
>> + sqdata.sq_deps[tid] = set()
>> + for tid in sqdata.sq_revdeps:
>> + for dep in sqdata.sq_revdeps[tid]:
>> + sqdata.sq_deps[dep].add(tid)
>> +
>> + rqdata.init_progress_reporter.next_stage()
>> +
>> + sqdata.multiconfigs = set()
>> + for tid in sqdata.sq_revdeps:
>> + sqdata.multiconfigs.add(mc_from_tid(tid))
>> + if len(sqdata.sq_revdeps[tid]) == 0:
>> + sqrq.sq_buildable.add(tid)
>> +
>> + rqdata.init_progress_reporter.finish()
>> +
>> + sqdata.noexec = set()
>> + sqdata.stamppresent = set()
>> + sqdata.valid = set()
>> +
>> + update_scenequeue_data(sqdata.sq_revdeps, sqdata, rqdata, rq,
>> cooker, stampcache, sqrq) +
>> +def update_scenequeue_data(tids, sqdata, rqdata, rq, cooker,
>> stampcache, sqrq): +
>> + tocheck = set()
>> +
>> + for tid in sorted(tids):
>> + if tid in sqdata.stamppresent:
>> + sqdata.stamppresent.remove(tid)
>> + if tid in sqdata.valid:
>> + sqdata.valid.remove(tid)
>> +
>> + (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
>> +
>> + taskdep = rqdata.dataCaches[mc].task_deps[taskfn]
>> +
>> + if 'noexec' in taskdep and taskname in taskdep['noexec']:
>> + sqdata.noexec.add(tid)
>> + sqrq.sq_task_skip(tid)
>> + bb.build.make_stamp(taskname + "_setscene",
>> rqdata.dataCaches[mc], taskfn)
>> + continue
>> +
>> + if rq.check_stamp_task(tid, taskname + "_setscene",
>> cache=stampcache):
>> + logger.debug(2, 'Setscene stamp current for task %s',
>> tid)
>> + sqdata.stamppresent.add(tid)
>> + sqrq.sq_task_skip(tid)
>> + continue
>> +
>> + if rq.check_stamp_task(tid, taskname, recurse = True,
>> cache=stampcache):
>> + logger.debug(2, 'Normal stamp current for task %s', tid)
>> + sqdata.stamppresent.add(tid)
>> + sqrq.sq_task_skip(tid)
>> + continue
>> +
>> + tocheck.add(tid)
>> +
>> + sqdata.valid |= rq.validate_hashes(tocheck, cooker.data,
>> len(sqdata.stamppresent), False) +
>> + sqdata.hashes = {}
>> + for mc in sorted(sqdata.multiconfigs):
>> + for tid in sorted(sqdata.sq_revdeps):
>> + if mc_from_tid(tid) != mc:
>> + continue
>> + if tid in sqdata.stamppresent:
>> + continue
>> + if tid in sqdata.valid:
>> + continue
>> + if tid in sqdata.noexec:
>> + continue
>> + if tid in sqrq.scenequeue_notcovered:
>> + continue
>> + sqdata.outrightfail.add(tid)
>> +
>> + h = pending_hash_index(tid, rqdata)
>> + if h not in sqdata.hashes:
>> + sqdata.hashes[h] = tid
>> + else:
>> + sqrq.sq_deferred[tid] = sqdata.hashes[h]
>> + bb.warn("Deferring %s after %s" % (tid,
>> sqdata.hashes[h])) +
>> +
>> class TaskFailure(Exception):
>> """
>> Exception raised when a task in a runqueue fails
>> @@ -2641,6 +2859,15 @@ class runQueueTaskSkipped(runQueueEvent):
>> runQueueEvent.__init__(self, task, stats, rq)
>> self.reason = reason
>>
>> +class taskUniHashUpdate(bb.event.Event):
>> + """
>> + Base runQueue event class
>> + """
>> + def __init__(self, task, unihash):
>> + self.taskid = task
>> + self.unihash = unihash
>> + bb.event.Event.__init__(self)
>> +
>> class runQueuePipe():
>> """
>> Abstraction for a pipe between a worker thread and the server
>> @@ -2683,6 +2910,8 @@ class runQueuePipe():
>> except ValueError as e:
>> bb.msg.fatal("RunQueue", "failed load pickle
>> '%s': '%s'" % (e, self.queue[7:index]))
>> bb.event.fire_from_worker(event, self.d)
>> + if isinstance(event, taskUniHashUpdate):
>> +
>> self.rqexec.updated_taskhash_queue.append((event.taskid,
>> event.unihash)) found = True self.queue = self.queue[index+8:]
>> index = self.queue.find(b"</event>")
>> diff --git a/bitbake/lib/bb/server/__init__.py
>> b/bitbake/lib/bb/server/__init__.py index 5a3fba9..b6f7513 100644
>> --- a/bitbake/lib/bb/server/__init__.py
>> +++ b/bitbake/lib/bb/server/__init__.py
>> @@ -5,17 +5,5 @@
>> # Copyright (C) 2006 - 2008 Richard Purdie
>> # Copyright (C) 2013 Alexandru Damian
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. -
>> -
>> diff --git a/bitbake/lib/bb/server/process.py
>> b/bitbake/lib/bb/server/process.py index 4e0d9c2..69aae62 100644
>> --- a/bitbake/lib/bb/server/process.py
>> +++ b/bitbake/lib/bb/server/process.py
>> @@ -3,18 +3,8 @@
>> #
>> # Copyright (C) 2010 Bob Foerster <robert@erafx.com>
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> """
>> This module implements a multiprocessing.Process based server
>> for bitbake. @@ -130,6 +120,7 @@ class
>> ProcessServer(multiprocessing.Process):
>> bb.utils.set_process_name("Cooker")
>> ready = []
>> + newconnections = []
>>
>> self.controllersock = False
>> fds = [self.sock]
>> @@ -138,37 +129,48 @@ class ProcessServer(multiprocessing.Process):
>> print("Entering server connection loop")
>>
>> def disconnect_client(self, fds):
>> - if not self.haveui:
>> - return
>> print("Disconnecting Client")
>> - fds.remove(self.controllersock)
>> - fds.remove(self.command_channel)
>> - bb.event.unregister_UIHhandler(self.event_handle, True)
>> - self.command_channel_reply.writer.close()
>> - self.event_writer.writer.close()
>> - del self.event_writer
>> - self.controllersock.close()
>> - self.controllersock = False
>> - self.haveui = False
>> - self.lastui = time.time()
>> - self.cooker.clientComplete()
>> - if self.timeout is None:
>> + if self.controllersock:
>> + fds.remove(self.controllersock)
>> + self.controllersock.close()
>> + self.controllersock = False
>> + if self.haveui:
>> + fds.remove(self.command_channel)
>> + bb.event.unregister_UIHhandler(self.event_handle,
>> True)
>> + self.command_channel_reply.writer.close()
>> + self.event_writer.writer.close()
>> + self.command_channel.close()
>> + self.command_channel = False
>> + del self.event_writer
>> + self.lastui = time.time()
>> + self.cooker.clientComplete()
>> + self.haveui = False
>> + ready = select.select(fds,[],[],0)[0]
>> + if newconnections:
>> + print("Starting new client")
>> + conn = newconnections.pop(-1)
>> + fds.append(conn)
>> + self.controllersock = conn
>> + elif self.timeout is None and not ready:
>> print("No timeout, exiting.")
>> self.quit = True
>>
>> while not self.quit:
>> if self.sock in ready:
>> - self.controllersock, address = self.sock.accept()
>> - if self.haveui:
>> - print("Dropping connection attempt as we have a
>> UI %s" % (str(ready)))
>> - self.controllersock.close()
>> - else:
>> - print("Accepting %s" % (str(ready)))
>> - fds.append(self.controllersock)
>> + while select.select([self.sock],[],[],0)[0]:
>> + controllersock, address = self.sock.accept()
>> + if self.controllersock:
>> + print("Queuing %s (%s)" % (str(ready),
>> str(newconnections)))
>> + newconnections.append(controllersock)
>> + else:
>> + print("Accepting %s (%s)" % (str(ready),
>> str(newconnections)))
>> + self.controllersock = controllersock
>> + fds.append(controllersock)
>> if self.controllersock in ready:
>> try:
>> - print("Connecting Client")
>> + print("Processing Client")
>> ui_fds = recvfds(self.controllersock, 3)
>> + print("Connecting Client")
>>
>> # Where to write events to
>> writer = ConnectionWriter(ui_fds[0])
>> @@ -239,6 +241,12 @@ class ProcessServer(multiprocessing.Process):
>> while not lock:
>> with bb.utils.timeout(3):
>> lock = bb.utils.lockfile(lockfile, shared=False,
>> retry=False, block=True)
>> + if lock:
>> + # We hold the lock so we can remove the file
>> (hide stale pid data)
>> + bb.utils.remove(lockfile)
>> + bb.utils.unlockfile(lock)
>> + return
>> +
>> if not lock:
>> # Some systems may not have lsof available
>> procs = None
>> @@ -259,10 +267,6 @@ class ProcessServer(multiprocessing.Process):
>> if procs:
>> msg += ":\n%s" % str(procs)
>> print(msg)
>> - return
>> - # We hold the lock so we can remove the file (hide stale pid
>> data)
>> - bb.utils.remove(lockfile)
>> - bb.utils.unlockfile(lock)
>>
>> def idle_commands(self, delay, fds=None):
>> nextsleep = delay
>> @@ -398,36 +402,45 @@ class BitBakeServer(object):
>> os.close(self.readypipein)
>>
>> ready = ConnectionReader(self.readypipe)
>> - r = ready.poll(30)
>> + r = ready.poll(5)
>> + if not r:
>> + bb.note("Bitbake server didn't start within 5 seconds,
>> waiting for 90")
>> + r = ready.poll(90)
>> if r:
>> try:
>> r = ready.get()
>> except EOFError:
>> # Trap the child exitting/closing the pipe and error
>> out r = None
>> - if not r or r != "ready":
>> + if not r or r[0] != "r":
>> ready.close()
>> - bb.error("Unable to start bitbake server")
>> + bb.error("Unable to start bitbake server (%s)" % str(r))
>> if os.path.exists(logfile):
>> logstart_re = re.compile(self.start_log_format %
>> ('([0-9]+)', '([0-9-]+ [0-9:.]+)')) started = False
>> lines = []
>> + lastlines = []
>> with open(logfile, "r") as f:
>> for line in f:
>> if started:
>> lines.append(line)
>> else:
>> + lastlines.append(line)
>> res = logstart_re.match(line.rstrip())
>> if res:
>> ldatetime =
>> datetime.datetime.strptime(res.group(2),
>> self.start_log_datetime_format) if ldatetime >= startdatetime:
>> started = True lines.append(line)
>> + if len(lastlines) > 60:
>> + lastlines = lastlines[-60:]
>> if lines:
>> - if len(lines) > 10:
>> - bb.error("Last 10 lines of server log for
>> this session (%s):\n%s" % (logfile, "".join(lines[-10:])))
>> + if len(lines) > 60:
>> + bb.error("Last 60 lines of server log for
>> this session (%s):\n%s" % (logfile, "".join(lines[-60:]))) else:
>> bb.error("Server log for this session
>> (%s):\n%s" % (logfile, "".join(lines)))
>> + elif lastlines:
>> + bb.error("Server didn't start, last 60
>> loglines (%s):\n%s" % (logfile, "".join(lastlines))) else:
>> bb.error("%s doesn't exist" % logfile)
>>
>> @@ -437,17 +450,24 @@ class BitBakeServer(object):
>>
>> def _startServer(self):
>> print(self.start_log_format % (os.getpid(),
>> datetime.datetime.now().strftime(self.start_log_datetime_format)))
>> + sys.stdout.flush()
>> +
>> server = ProcessServer(self.bitbake_lock, self.sock,
>> self.sockname)
>> self.configuration.setServerRegIdleCallback(server.register_idle_function)
>> os.close(self.readypipe) writer = ConnectionWriter(self.readypipein)
>> - self.cooker = bb.cooker.BBCooker(self.configuration,
>> self.featureset)
>> - writer.send("ready")
>> + try:
>> + self.cooker = bb.cooker.BBCooker(self.configuration,
>> self.featureset)
>> + except bb.BBHandledException:
>> + return None
>> + writer.send("r")
>> writer.close()
>> server.cooker = self.cooker
>> server.server_timeout = self.configuration.server_timeout
>> server.xmlrpcinterface = self.configuration.xmlrpcinterface
>> print("Started bitbake server pid %d" % os.getpid())
>> + sys.stdout.flush()
>> +
>> server.start()
>>
>> def connectProcessServer(sockname, featureset):
>> @@ -459,10 +479,20 @@ def connectProcessServer(sockname, featureset):
>> readfd = writefd = readfd1 = writefd1 = readfd2 = writefd2 = None
>> eq = command_chan_recv = command_chan = None
>>
>> + sock.settimeout(10)
>> +
>> try:
>> try:
>> os.chdir(os.path.dirname(sockname))
>> - sock.connect(os.path.basename(sockname))
>> + finished = False
>> + while not finished:
>> + try:
>> + sock.connect(os.path.basename(sockname))
>> + finished = True
>> + except IOError as e:
>> + if e.errno == errno.EWOULDBLOCK:
>> + pass
>> + raise
>> finally:
>> os.chdir(cwd)
>>
>> @@ -493,7 +523,8 @@ def connectProcessServer(sockname, featureset):
>> command_chan.close()
>> for i in [writefd, readfd1, writefd2]:
>> try:
>> - os.close(i)
>> + if i:
>> + os.close(i)
>> except OSError:
>> pass
>> sock.close()
>> diff --git a/bitbake/lib/bb/server/xmlrpcclient.py
>> b/bitbake/lib/bb/server/xmlrpcclient.py index 4661a9e..c054c3c 100644
>> --- a/bitbake/lib/bb/server/xmlrpcclient.py
>> +++ b/bitbake/lib/bb/server/xmlrpcclient.py
>> @@ -4,18 +4,8 @@
>> # Copyright (C) 2006 - 2007 Michael 'Mickey' Lauer
>> # Copyright (C) 2006 - 2008 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import os
>> import sys
>> diff --git a/bitbake/lib/bb/server/xmlrpcserver.py
>> b/bitbake/lib/bb/server/xmlrpcserver.py index 875b128..54fa32f 100644
>> --- a/bitbake/lib/bb/server/xmlrpcserver.py
>> +++ b/bitbake/lib/bb/server/xmlrpcserver.py
>> @@ -4,18 +4,8 @@
>> # Copyright (C) 2006 - 2007 Michael 'Mickey' Lauer
>> # Copyright (C) 2006 - 2008 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import os
>> import sys
>> diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py
>> index fdbb2a3..a4bb1ff 100644
>> --- a/bitbake/lib/bb/siggen.py
>> +++ b/bitbake/lib/bb/siggen.py
>> @@ -1,3 +1,7 @@
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> import hashlib
>> import logging
>> import os
>> @@ -8,6 +12,8 @@ import bb.data
>> import difflib
>> import simplediff
>> from bb.checksum import FileChecksumCache
>> +from bb import runqueue
>> +import hashserv
>>
>> logger = logging.getLogger('BitBake.SigGen')
>>
>> @@ -37,12 +43,18 @@ class SignatureGenerator(object):
>> self.runtaskdeps = {}
>> self.file_checksum_values = {}
>> self.taints = {}
>> + self.unitaskhashes = {}
>> + self.setscenetasks = {}
>>
>> def finalise(self, fn, d, varient):
>> return
>>
>> - def get_taskhash(self, fn, task, deps, dataCache):
>> - return "0"
>> + def get_unihash(self, tid):
>> + return self.taskhash[tid]
>> +
>> + def get_taskhash(self, tid, deps, dataCache):
>> + self.taskhash[tid] =
>> hashlib.sha256(tid.encode("utf-8")).hexdigest()
>> + return self.taskhash[tid]
>>
>> def writeout_file_checksum_cache(self):
>> """Write/update the file checksum cache onto disk"""
>> @@ -64,14 +76,25 @@ class SignatureGenerator(object):
>> return
>>
>> def get_taskdata(self):
>> - return (self.runtaskdeps, self.taskhash,
>> self.file_checksum_values, self.taints, self.basehash)
>> + return (self.runtaskdeps, self.taskhash,
>> self.file_checksum_values, self.taints, self.basehash,
>> self.unitaskhashes, self.setscenetasks) def set_taskdata(self, data):
>> - self.runtaskdeps, self.taskhash, self.file_checksum_values,
>> self.taints, self.basehash = data
>> + self.runtaskdeps, self.taskhash, self.file_checksum_values,
>> self.taints, self.basehash, self.unitaskhashes, self.setscenetasks =
>> data def reset(self, data):
>> self.__init__(data)
>>
>> + def get_taskhashes(self):
>> + return self.taskhash, self.unitaskhashes
>> +
>> + def set_taskhashes(self, hashes):
>> + self.taskhash, self.unitaskhashes = hashes
>> +
>> + def save_unitaskhashes(self):
>> + return
>> +
>> + def set_setscene_tasks(self, setscene_tasks):
>> + return
>>
>> class SignatureGeneratorBasic(SignatureGenerator):
>> """
>> @@ -87,7 +110,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
>> self.taints = {}
>> self.gendeps = {}
>> self.lookupcache = {}
>> - self.pkgnameextract = re.compile("(?P<fn>.*)\..*")
>> + self.setscenetasks = {}
>> self.basewhitelist =
>> set((data.getVar("BB_HASHBASE_WHITELIST") or "").split())
>> self.taskwhitelist = None self.init_rundepcheck(data)
>> @@ -98,6 +121,9 @@ class SignatureGeneratorBasic(SignatureGenerator):
>> else:
>> self.checksum_cache = None
>>
>> + self.unihash_cache = bb.cache.SimpleCache("1")
>> + self.unitaskhashes = self.unihash_cache.init_cache(data,
>> "bb_unihashes.dat", {}) +
>> def init_rundepcheck(self, data):
>> self.taskwhitelist = data.getVar("BB_HASHTASK_WHITELIST") or
>> None if self.taskwhitelist:
>> @@ -113,10 +139,16 @@ class
>> SignatureGeneratorBasic(SignatureGenerator): taskdeps, basehash =
>> bb.data.generate_dependency_hash(tasklist, gendeps, lookupcache,
>> self.basewhitelist, fn) for task in tasklist:
>> - k = fn + "." + task
>> - if not ignore_mismatch and k in self.basehash and
>> self.basehash[k] != basehash[k]:
>> - bb.error("When reparsing %s, the basehash value
>> changed from %s to %s. The metadata is not deterministic and this
>> needs to be fixed." % (k, self.basehash[k], basehash[k]))
>> - self.basehash[k] = basehash[k]
>> + tid = fn + ":" + task
>> + if not ignore_mismatch and tid in self.basehash and
>> self.basehash[tid] != basehash[tid]:
>> + bb.error("When reparsing %s, the basehash value
>> changed from %s to %s. The metadata is not deterministic and this
>> needs to be fixed." % (tid, self.basehash[tid], basehash[tid]))
>> + bb.error("The following commands may help:")
>> + cmd = "$ bitbake %s -c%s" % (d.getVar('PN'), task)
>> + # Make sure sigdata is dumped before run printdiff
>> + bb.error("%s -Snone" % cmd)
>> + bb.error("Then:")
>> + bb.error("%s -Sprintdiff\n" % cmd)
>> + self.basehash[tid] = basehash[tid]
>>
>> self.taskdeps[fn] = taskdeps
>> self.gendeps[fn] = gendeps
>> @@ -124,6 +156,9 @@ class SignatureGeneratorBasic(SignatureGenerator):
>>
>> return taskdeps
>>
>> + def set_setscene_tasks(self, setscene_tasks):
>> + self.setscenetasks = setscene_tasks
>> +
>> def finalise(self, fn, d, variant):
>>
>> mc = d.getVar("__BBMULTICONFIG", False) or ""
>> @@ -143,7 +178,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
>> # self.dump_sigtask(fn, task, d.getVar("STAMP"), False)
>>
>> for task in taskdeps:
>> - d.setVar("BB_BASEHASH_task-%s" % task, self.basehash[fn
>> + "." + task])
>> + d.setVar("BB_BASEHASH_task-%s" % task, self.basehash[fn
>> + ":" + task])
>> def rundep_check(self, fn, recipename, task, dep, depname,
>> dataCache): # Return True if we should keep the dependency, False to
>> drop it @@ -163,31 +198,26 @@ class
>> SignatureGeneratorBasic(SignatureGenerator): pass
>> return taint
>>
>> - def get_taskhash(self, fn, task, deps, dataCache):
>> + def get_taskhash(self, tid, deps, dataCache):
>>
>> - mc = ''
>> - if fn.startswith('multiconfig:'):
>> - mc = fn.split(':')[1]
>> - k = fn + "." + task
>> + (mc, _, task, fn) = bb.runqueue.split_tid_mcfn(tid)
>>
>> - data = dataCache.basetaskhash[k]
>> - self.basehash[k] = data
>> - self.runtaskdeps[k] = []
>> - self.file_checksum_values[k] = []
>> + data = dataCache.basetaskhash[tid]
>> + self.basehash[tid] = data
>> + self.runtaskdeps[tid] = []
>> + self.file_checksum_values[tid] = []
>> recipename = dataCache.pkg_fn[fn]
>> for dep in sorted(deps, key=clean_basepath):
>> - pkgname = self.pkgnameextract.search(dep).group('fn')
>> - if mc:
>> - depmc = pkgname.split(':')[1]
>> - if mc != depmc:
>> - continue
>> - depname = dataCache.pkg_fn[pkgname]
>> + (depmc, _, deptaskname, depfn) =
>> bb.runqueue.split_tid_mcfn(dep)
>> + if mc != depmc:
>> + continue
>> + depname = dataCache.pkg_fn[depfn]
>> if not self.rundep_check(fn, recipename, task, dep,
>> depname, dataCache): continue
>> if dep not in self.taskhash:
>> bb.fatal("%s is not in taskhash, caller isn't
>> calling in dependency order?" % dep)
>> - data = data + self.taskhash[dep]
>> - self.runtaskdeps[k].append(dep)
>> + data = data + self.get_unihash(dep)
>> + self.runtaskdeps[tid].append(dep)
>>
>> if task in dataCache.file_checksums[fn]:
>> if self.checksum_cache:
>> @@ -195,7 +225,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
>> else:
>> checksums =
>> bb.fetch2.get_file_checksums(dataCache.file_checksums[fn][task],
>> recipename) for (f,cs) in checksums:
>> - self.file_checksum_values[k].append((f,cs))
>> + self.file_checksum_values[tid].append((f,cs))
>> if cs:
>> data = data + cs
>>
>> @@ -205,16 +235,16 @@ class
>> SignatureGeneratorBasic(SignatureGenerator): import uuid
>> taint = str(uuid.uuid4())
>> data = data + taint
>> - self.taints[k] = "nostamp:" + taint
>> + self.taints[tid] = "nostamp:" + taint
>>
>> taint = self.read_taint(fn, task, dataCache.stamp[fn])
>> if taint:
>> data = data + taint
>> - self.taints[k] = taint
>> - logger.warning("%s is tainted from a forced run" % k)
>> + self.taints[tid] = taint
>> + logger.warning("%s is tainted from a forced run" % tid)
>>
>> - h = hashlib.md5(data.encode("utf-8")).hexdigest()
>> - self.taskhash[k] = h
>> + h = hashlib.sha256(data.encode("utf-8")).hexdigest()
>> + self.taskhash[tid] = h
>> #d.setVar("BB_TASKHASH_task-%s" % task, taskhash[task])
>> return h
>>
>> @@ -227,17 +257,20 @@ class
>> SignatureGeneratorBasic(SignatureGenerator):
>> bb.fetch2.fetcher_parse_save() bb.fetch2.fetcher_parse_done()
>>
>> + def save_unitaskhashes(self):
>> + self.unihash_cache.save(self.unitaskhashes)
>> +
>> def dump_sigtask(self, fn, task, stampbase, runtime):
>>
>> - k = fn + "." + task
>> + tid = fn + ":" + task
>> referencestamp = stampbase
>> if isinstance(runtime, str) and
>> runtime.startswith("customfile"): sigfile = stampbase
>> referencestamp = runtime[11:]
>> - elif runtime and k in self.taskhash:
>> - sigfile = stampbase + "." + task + ".sigdata" + "." +
>> self.taskhash[k]
>> + elif runtime and tid in self.taskhash:
>> + sigfile = stampbase + "." + task + ".sigdata" + "." +
>> self.get_unihash(tid) else:
>> - sigfile = stampbase + "." + task + ".sigbasedata" + "."
>> + self.basehash[k]
>> + sigfile = stampbase + "." + task + ".sigbasedata" + "."
>> + self.basehash[tid]
>> bb.utils.mkdirhier(os.path.dirname(sigfile))
>>
>> @@ -246,7 +279,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
>> data['basewhitelist'] = self.basewhitelist
>> data['taskwhitelist'] = self.taskwhitelist
>> data['taskdeps'] = self.taskdeps[fn][task]
>> - data['basehash'] = self.basehash[k]
>> + data['basehash'] = self.basehash[tid]
>> data['gendeps'] = {}
>> data['varvals'] = {}
>> data['varvals'][task] = self.lookupcache[fn][task]
>> @@ -256,30 +289,31 @@ class
>> SignatureGeneratorBasic(SignatureGenerator): data['gendeps'][dep] =
>> self.gendeps[fn][dep] data['varvals'][dep] = self.lookupcache[fn][dep]
>>
>> - if runtime and k in self.taskhash:
>> - data['runtaskdeps'] = self.runtaskdeps[k]
>> - data['file_checksum_values'] = [(os.path.basename(f),
>> cs) for f,cs in self.file_checksum_values[k]]
>> + if runtime and tid in self.taskhash:
>> + data['runtaskdeps'] = self.runtaskdeps[tid]
>> + data['file_checksum_values'] = [(os.path.basename(f),
>> cs) for f,cs in self.file_checksum_values[tid]] data['runtaskhashes']
>> = {} for dep in data['runtaskdeps']:
>> - data['runtaskhashes'][dep] = self.taskhash[dep]
>> - data['taskhash'] = self.taskhash[k]
>> + data['runtaskhashes'][dep] = self.get_unihash(dep)
>> + data['taskhash'] = self.taskhash[tid]
>> + data['unihash'] = self.get_unihash(tid)
>>
>> taint = self.read_taint(fn, task, referencestamp)
>> if taint:
>> data['taint'] = taint
>>
>> - if runtime and k in self.taints:
>> - if 'nostamp:' in self.taints[k]:
>> - data['taint'] = self.taints[k]
>> + if runtime and tid in self.taints:
>> + if 'nostamp:' in self.taints[tid]:
>> + data['taint'] = self.taints[tid]
>>
>> computed_basehash = calc_basehash(data)
>> - if computed_basehash != self.basehash[k]:
>> - bb.error("Basehash mismatch %s versus %s for %s" %
>> (computed_basehash, self.basehash[k], k))
>> - if runtime and k in self.taskhash:
>> + if computed_basehash != self.basehash[tid]:
>> + bb.error("Basehash mismatch %s versus %s for %s" %
>> (computed_basehash, self.basehash[tid], tid))
>> + if runtime and tid in self.taskhash:
>> computed_taskhash = calc_taskhash(data)
>> - if computed_taskhash != self.taskhash[k]:
>> - bb.error("Taskhash mismatch %s versus %s for %s" %
>> (computed_taskhash, self.taskhash[k], k))
>> - sigfile = sigfile.replace(self.taskhash[k],
>> computed_taskhash)
>> + if computed_taskhash != self.taskhash[tid]:
>> + bb.error("Taskhash mismatch %s versus %s for %s" %
>> (computed_taskhash, self.taskhash[tid], tid))
>> + sigfile = sigfile.replace(self.taskhash[tid],
>> computed_taskhash)
>> fd, tmpfile = tempfile.mkstemp(dir=os.path.dirname(sigfile),
>> prefix="sigtask.") try:
>> @@ -299,30 +333,34 @@ class
>> SignatureGeneratorBasic(SignatureGenerator): if fn in self.taskdeps:
>> for task in self.taskdeps[fn]:
>> tid = fn + ":" + task
>> - (mc, _, _) = bb.runqueue.split_tid(tid)
>> - k = fn + "." + task
>> - if k not in self.taskhash:
>> + mc = bb.runqueue.mc_from_tid(tid)
>> + if tid not in self.taskhash:
>> continue
>> - if dataCaches[mc].basetaskhash[k] !=
>> self.basehash[k]:
>> - bb.error("Bitbake's cached basehash does not
>> match the one we just generated (%s)!" % k)
>> - bb.error("The mismatched hashes were %s and %s"
>> % (dataCaches[mc].basetaskhash[k], self.basehash[k]))
>> + if dataCaches[mc].basetaskhash[tid] !=
>> self.basehash[tid]:
>> + bb.error("Bitbake's cached basehash does not
>> match the one we just generated (%s)!" % tid)
>> + bb.error("The mismatched hashes were %s and %s"
>> % (dataCaches[mc].basetaskhash[tid], self.basehash[tid]))
>> self.dump_sigtask(fn, task, dataCaches[mc].stamp[fn], True)
>> class SignatureGeneratorBasicHash(SignatureGeneratorBasic):
>> name = "basichash"
>>
>> + def get_stampfile_hash(self, tid):
>> + if tid in self.taskhash:
>> + return self.taskhash[tid]
>> +
>> + # If task is not in basehash, then error
>> + return self.basehash[tid]
>> +
>> def stampfile(self, stampbase, fn, taskname, extrainfo,
>> clean=False): if taskname != "do_setscene" and
>> taskname.endswith("_setscene"):
>> - k = fn + "." + taskname[:-9]
>> + tid = fn + ":" + taskname[:-9]
>> else:
>> - k = fn + "." + taskname
>> + tid = fn + ":" + taskname
>> if clean:
>> h = "*"
>> - elif k in self.taskhash:
>> - h = self.taskhash[k]
>> else:
>> - # If k is not in basehash, then error
>> - h = self.basehash[k]
>> + h = self.get_stampfile_hash(tid)
>> +
>> return ("%s.%s.%s.%s" % (stampbase, taskname, h,
>> extrainfo)).rstrip('.')
>> def stampcleanmask(self, stampbase, fn, taskname, extrainfo):
>> @@ -332,6 +370,172 @@ class
>> SignatureGeneratorBasicHash(SignatureGeneratorBasic):
>> bb.note("Tainting hash to force rebuild of task %s, %s" % (fn, task))
>> bb.build.write_taint(task, d, fn)
>> +class SignatureGeneratorUniHashMixIn(object):
>> + def get_taskdata(self):
>> + return (self.server, self.method) + super().get_taskdata()
>> +
>> + def set_taskdata(self, data):
>> + self.server, self.method = data[:2]
>> + super().set_taskdata(data[2:])
>> +
>> + def client(self):
>> + if getattr(self, '_client', None) is None:
>> + self._client = hashserv.create_client(self.server)
>> + return self._client
>> +
>> + def __get_task_unihash_key(self, tid):
>> + # TODO: The key only *needs* to be the taskhash, the tid is
>> just
>> + # convenient
>> + return '%s:%s' % (tid.rsplit("/", 1)[1], self.taskhash[tid])
>> +
>> + def get_stampfile_hash(self, tid):
>> + if tid in self.taskhash:
>> + # If a unique hash is reported, use it as the stampfile
>> hash. This
>> + # ensures that if a task won't be re-run if the taskhash
>> changes,
>> + # but it would result in the same output hash
>> + unihash =
>> self.unitaskhashes.get(self.__get_task_unihash_key(tid), None)
>> + if unihash is not None:
>> + return unihash
>> +
>> + return super().get_stampfile_hash(tid)
>> +
>> + def set_unihash(self, tid, unihash):
>> + self.unitaskhashes[self.__get_task_unihash_key(tid)] =
>> unihash +
>> + def get_unihash(self, tid):
>> + taskhash = self.taskhash[tid]
>> +
>> + # If its not a setscene task we can return
>> + if self.setscenetasks and tid not in self.setscenetasks:
>> + return taskhash
>> +
>> + key = self.__get_task_unihash_key(tid)
>> +
>> + # TODO: This cache can grow unbounded. It probably only
>> needs to keep
>> + # for each task
>> + unihash = self.unitaskhashes.get(key, None)
>> + if unihash is not None:
>> + return unihash
>> +
>> + # In the absence of being able to discover a unique hash
>> from the
>> + # server, make it be equivalent to the taskhash. The unique
>> "hash" only
>> + # really needs to be a unique string (not even necessarily a
>> hash), but
>> + # making it match the taskhash has a few advantages:
>> + #
>> + # 1) All of the sstate code that assumes hashes can be the
>> same
>> + # 2) It provides maximal compatibility with builders that
>> don't use
>> + # an equivalency server
>> + # 3) The value is easy for multiple independent builders to
>> derive the
>> + # same unique hash from the same input. This means that
>> if the
>> + # independent builders find the same taskhash, but it
>> isn't reported
>> + # to the server, there is a better chance that they will
>> agree on
>> + # the unique hash.
>> + unihash = taskhash
>> +
>> + try:
>> + data = self.client().get_unihash(self.method,
>> self.taskhash[tid])
>> + if data:
>> + unihash = data
>> + # A unique hash equal to the taskhash is not very
>> interesting,
>> + # so it is reported it at debug level 2. If they
>> differ, that
>> + # is much more interesting, so it is reported at
>> debug level 1
>> + bb.debug((1, 2)[unihash == taskhash], 'Found unihash
>> %s in place of %s for %s from %s' % (unihash, taskhash, tid,
>> self.server))
>> + else:
>> + bb.debug(2, 'No reported unihash for %s:%s from %s'
>> % (tid, taskhash, self.server))
>> + except hashserv.client.HashConnectionError as e:
>> + bb.warn('Error contacting Hash Equivalence Server %s:
>> %s' % (self.server, str(e))) +
>> + self.unitaskhashes[key] = unihash
>> + return unihash
>> +
>> + def report_unihash(self, path, task, d):
>> + import importlib
>> +
>> + taskhash = d.getVar('BB_TASKHASH')
>> + unihash = d.getVar('BB_UNIHASH')
>> + report_taskdata =
>> d.getVar('SSTATE_HASHEQUIV_REPORT_TASKDATA') == '1'
>> + tempdir = d.getVar('T')
>> + fn = d.getVar('BB_FILENAME')
>> + tid = fn + ':do_' + task
>> + key = tid.rsplit("/", 1)[1] + ':' + taskhash
>> +
>> + if self.setscenetasks and tid not in self.setscenetasks:
>> + return
>> +
>> + # Sanity checks
>> + cache_unihash = self.unitaskhashes.get(key, None)
>> + if cache_unihash is None:
>> + bb.fatal('%s not in unihash cache. Please report this
>> error' % key) +
>> + if cache_unihash != unihash:
>> + bb.fatal("Cache unihash %s doesn't match BB_UNIHASH %s"
>> % (cache_unihash, unihash)) +
>> + sigfile = None
>> + sigfile_name = "depsig.do_%s.%d" % (task, os.getpid())
>> + sigfile_link = "depsig.do_%s" % task
>> +
>> + try:
>> + sigfile = open(os.path.join(tempdir, sigfile_name),
>> 'w+b') +
>> + locs = {'path': path, 'sigfile': sigfile, 'task': task,
>> 'd': d} +
>> + if "." in self.method:
>> + (module, method) = self.method.rsplit('.', 1)
>> + locs['method'] =
>> getattr(importlib.import_module(module), method)
>> + outhash = bb.utils.better_eval('method(path,
>> sigfile, task, d)', locs)
>> + else:
>> + outhash = bb.utils.better_eval(self.method + '(path,
>> sigfile, task, d)', locs) +
>> + try:
>> + extra_data = {}
>> +
>> + owner = d.getVar('SSTATE_HASHEQUIV_OWNER')
>> + if owner:
>> + extra_data['owner'] = owner
>> +
>> + if report_taskdata:
>> + sigfile.seek(0)
>> +
>> + extra_data['PN'] = d.getVar('PN')
>> + extra_data['PV'] = d.getVar('PV')
>> + extra_data['PR'] = d.getVar('PR')
>> + extra_data['task'] = task
>> + extra_data['outhash_siginfo'] =
>> sigfile.read().decode('utf-8') +
>> + data = self.client().report_unihash(taskhash,
>> self.method, outhash, unihash, extra_data)
>> + new_unihash = data['unihash']
>> +
>> + if new_unihash != unihash:
>> + bb.debug(1, 'Task %s unihash changed %s -> %s by
>> server %s' % (taskhash, unihash, new_unihash, self.server))
>> + bb.event.fire(bb.runqueue.taskUniHashUpdate(fn +
>> ':do_' + task, new_unihash), d)
>> + else:
>> + bb.debug(1, 'Reported task %s as unihash %s to
>> %s' % (taskhash, unihash, self.server))
>> + except hashserv.client.HashConnectionError as e:
>> + bb.warn('Error contacting Hash Equivalence Server
>> %s: %s' % (self.server, str(e)))
>> + finally:
>> + if sigfile:
>> + sigfile.close()
>> +
>> + sigfile_link_path = os.path.join(tempdir,
>> sigfile_link)
>> + bb.utils.remove(sigfile_link_path)
>> +
>> + try:
>> + os.symlink(sigfile_name, sigfile_link_path)
>> + except OSError:
>> + pass
>> +
>> +
>> +#
>> +# Dummy class used for bitbake-selftest
>> +#
>> +class
>> SignatureGeneratorTestEquivHash(SignatureGeneratorUniHashMixIn,
>> SignatureGeneratorBasicHash):
>> + name = "TestEquivHash"
>> + def init_rundepcheck(self, data):
>> + super().init_rundepcheck(data)
>> + self.server = data.getVar('BB_HASHSERVE')
>> + self.method = "sstate_output_hash"
>> +
>> +
>> def dump_this_task(outfile, d):
>> import bb.parse
>> fn = d.getVar("BB_FILENAME")
>> @@ -392,13 +596,13 @@ def list_inline_diff(oldlist, newlist,
>> colors=None):
>> def clean_basepath(a):
>> mc = None
>> - if a.startswith("multiconfig:"):
>> + if a.startswith("mc:"):
>> _, mc, a = a.split(":", 2)
>> b = a.rsplit("/", 2)[1] + '/' + a.rsplit("/", 2)[2]
>> if a.startswith("virtual:"):
>> b = b + ":" + a.rsplit(":", 1)[0]
>> if mc:
>> - b = b + ":multiconfig:" + mc
>> + b = b + ":mc:" + mc
>> return b
>>
>> def clean_basepaths(a):
>> @@ -623,6 +827,10 @@ def compare_sigfiles(a, b, recursecb=None,
>> color=False, collapsed=False): a_taint = a_data.get('taint', None)
>> b_taint = b_data.get('taint', None)
>> if a_taint != b_taint:
>> + if a_taint and a_taint.startswith('nostamp:'):
>> + a_taint = a_taint.replace('nostamp:', 'nostamp(uuid4):')
>> + if b_taint and b_taint.startswith('nostamp:'):
>> + b_taint = b_taint.replace('nostamp:', 'nostamp(uuid4):')
>> output.append(color_format("{color_title}Taint (by
>> forced/invalidated task) changed{color_default} from %s to %s") %
>> (a_taint, b_taint)) return output
>> @@ -642,7 +850,7 @@ def calc_basehash(sigdata):
>> if val is not None:
>> basedata = basedata + str(val)
>>
>> - return hashlib.md5(basedata.encode("utf-8")).hexdigest()
>> + return hashlib.sha256(basedata.encode("utf-8")).hexdigest()
>>
>> def calc_taskhash(sigdata):
>> data = sigdata['basehash']
>> @@ -660,7 +868,7 @@ def calc_taskhash(sigdata):
>> else:
>> data = data + sigdata['taint']
>>
>> - return hashlib.md5(data.encode("utf-8")).hexdigest()
>> + return hashlib.sha256(data.encode("utf-8")).hexdigest()
>>
>>
>> def dump_sigfile(a):
>> @@ -695,7 +903,11 @@ def dump_sigfile(a):
>> output.append("Hash for dependent task %s is %s" % (dep,
>> a_data['runtaskhashes'][dep]))
>> if 'taint' in a_data:
>> - output.append("Tainted (by forced/invalidated task): %s" %
>> a_data['taint'])
>> + if a_data['taint'].startswith('nostamp:'):
>> + msg = a_data['taint'].replace('nostamp:',
>> 'nostamp(uuid4):')
>> + else:
>> + msg = a_data['taint']
>> + output.append("Tainted (by forced/invalidated task): %s" %
>> msg)
>> if 'task' in a_data:
>> computed_basehash = calc_basehash(a_data)
>> diff --git a/bitbake/lib/bb/taskdata.py b/bitbake/lib/bb/taskdata.py
>> index 94e822c..8c25e09 100644
>> --- a/bitbake/lib/bb/taskdata.py
>> +++ b/bitbake/lib/bb/taskdata.py
>> @@ -1,6 +1,3 @@
>> -#!/usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake 'TaskData' implementation
>>
>> @@ -10,18 +7,8 @@ Task data collection and handling
>>
>> # Copyright (C) 2006 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import logging
>> import re
>> @@ -93,7 +80,7 @@ class TaskData:
>> def add_mcdepends(task):
>> for dep in task_deps['mcdepends'][task].split():
>> if len(dep.split(':')) != 5:
>> - bb.msg.fatal("TaskData", "Error for %s:%s[%s],
>> multiconfig dependency %s does not contain exactly four ':'
>> characters.\n Task '%s' should be specified in the form
>> 'multiconfig:fromMC:toMC:packagename:task'" % (fn, task, 'mcdepends',
>> dep, 'mcdepends'))
>> + bb.msg.fatal("TaskData", "Error for %s:%s[%s],
>> multiconfig dependency %s does not contain exactly four ':'
>> characters.\n Task '%s' should be specified in the form
>> 'mc:fromMC:toMC:packagename:task'" % (fn, task, 'mcdepends', dep,
>> 'mcdepends')) if dep not in self.mcdepends:
>> self.mcdepends.append(dep) diff --git
>> a/bitbake/lib/bb/tests/codeparser.py
>> b/bitbake/lib/bb/tests/codeparser.py index e30e78c..826a2d2 100644
>> --- a/bitbake/lib/bb/tests/codeparser.py +++
>> b/bitbake/lib/bb/tests/codeparser.py @@ -1,23 +1,10 @@ -#
>> ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4;
>> indent-tabs-mode: nil -*- #
>> # BitBake Test for codeparser.py
>> #
>> # Copyright (C) 2010 Chris Larson
>> # Copyright (C) 2012 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>>
>> import unittest
>> @@ -123,6 +110,13 @@ ${D}${libdir}/pkgconfig/*.pc
>> self.parseExpression("sed -i -e 's:IP{:I${:g' $pc")
>> self.assertExecs(set(["sed"]))
>>
>> + def test_parameter_expansion_modifiers(self):
>> + # - and + are also valid modifiers for parameter expansion,
>> but are
>> + # valid characters in bitbake variable names, so are not
>> included here
>> + for i in ('=', ':-', ':=', '?', ':?', ':+', '#', '%', '##',
>> '%%'):
>> + name = "foo%sbar" % i
>> + self.parseExpression("${%s}" % name)
>> + self.assertNotIn(name, self.references)
>>
>> def test_until(self):
>> self.parseExpression("until false; do echo true; done")
>> diff --git a/bitbake/lib/bb/tests/cooker.py
>> b/bitbake/lib/bb/tests/cooker.py index 2b44236..090916e 100644
>> --- a/bitbake/lib/bb/tests/cooker.py
>> +++ b/bitbake/lib/bb/tests/cooker.py
>> @@ -1,20 +1,7 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # BitBake Tests for cooker.py
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>>
>> import unittest
>> diff --git a/bitbake/lib/bb/tests/cow.py b/bitbake/lib/bb/tests/cow.py
>> index d149d84..b4af4bb 100644
>> --- a/bitbake/lib/bb/tests/cow.py
>> +++ b/bitbake/lib/bb/tests/cow.py
>> @@ -1,22 +1,9 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # BitBake Tests for Copy-on-Write (cow.py)
>> #
>> -# Copyright 2006 Holger Freyther <freyther@handhelds.org>
>> -#
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# Copyright 2006 Holger Freyther
>> <freyther@handhelds.org> #
>>
>> import unittest
>> diff --git a/bitbake/lib/bb/tests/data.py
>> b/bitbake/lib/bb/tests/data.py index db3e201..3e49984 100644
>> --- a/bitbake/lib/bb/tests/data.py
>> +++ b/bitbake/lib/bb/tests/data.py
>> @@ -1,23 +1,10 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # BitBake Tests for the Data Store (data.py/data_smart.py)
>> #
>> # Copyright (C) 2010 Chris Larson
>> # Copyright (C) 2012 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>>
>> import unittest
>> @@ -394,6 +381,28 @@ class TestOverrides(unittest.TestCase):
>> self.d.setVar("OVERRIDES", "foo:bar:some_val")
>> self.assertEqual(self.d.getVar("TEST"), " testvalue5")
>>
>> + def test_append_and_override_1(self):
>> + self.d.setVar("TEST_append", "testvalue2")
>> + self.d.setVar("TEST_bar", "testvalue3")
>> + self.assertEqual(self.d.getVar("TEST"),
>> "testvalue3testvalue2") +
>> + def test_append_and_override_2(self):
>> + self.d.setVar("TEST_append_bar", "testvalue2")
>> + self.assertEqual(self.d.getVar("TEST"),
>> "testvaluetestvalue2") +
>> + def test_append_and_override_3(self):
>> + self.d.setVar("TEST_bar_append", "testvalue2")
>> + self.assertEqual(self.d.getVar("TEST"), "testvalue2")
>> +
>> + # Test an override with _<numeric> in it based on a real world
>> OE issue
>> + def test_underscore_override(self):
>> + self.d.setVar("TARGET_ARCH", "x86_64")
>> + self.d.setVar("PN", "test-${TARGET_ARCH}")
>> + self.d.setVar("VERSION", "1")
>> + self.d.setVar("VERSION_pn-test-${TARGET_ARCH}", "2")
>> + self.d.setVar("OVERRIDES", "pn-${PN}")
>> + bb.data.expandKeys(self.d)
>> + self.assertEqual(self.d.getVar("VERSION"), "2")
>>
>> class TestKeyExpansion(unittest.TestCase):
>> def setUp(self):
>> @@ -470,7 +479,7 @@ class TaskHash(unittest.TestCase):
>> tasklist, gendeps, lookupcache =
>> bb.data.generate_dependencies(d) taskdeps, basehash =
>> bb.data.generate_dependency_hash(tasklist, gendeps, lookupcache,
>> set(), "somefile") bb.warn(str(lookupcache))
>> - return basehash["somefile." + taskname]
>> + return basehash["somefile:" + taskname]
>>
>> d = bb.data.init()
>> d.setVar("__BBTASKS", ["mytask"])
>> diff --git a/bitbake/lib/bb/tests/event.py
>> b/bitbake/lib/bb/tests/event.py index d3a5f62..9229b63 100644
>> --- a/bitbake/lib/bb/tests/event.py
>> +++ b/bitbake/lib/bb/tests/event.py
>> @@ -1,22 +1,9 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # BitBake Tests for the Event implementation (event.py)
>> #
>> # Copyright (C) 2017 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>>
>> import unittest
>> @@ -574,14 +561,6 @@ class EventClassesTest(unittest.TestCase):
>> self.assertEqual(event.fn(1), callback(1))
>> self.assertEqual(event.pid, EventClassesTest._worker_pid)
>>
>> - def test_StampUpdate(self):
>> - targets = ["foo", "bar"]
>> - stampfns = [lambda:"foobar"]
>> - event = bb.event.StampUpdate(targets, stampfns)
>> - self.assertEqual(event.targets, targets)
>> - self.assertEqual(event.stampPrefix, stampfns)
>> - self.assertEqual(event.pid, EventClassesTest._worker_pid)
>> -
>> def test_BuildBase(self):
>> """ Test base class for bitbake build events """
>> name = "foo"
>> diff --git a/bitbake/lib/bb/tests/fetch.py
>> b/bitbake/lib/bb/tests/fetch.py index 6848095..a0b656b 100644
>> --- a/bitbake/lib/bb/tests/fetch.py
>> +++ b/bitbake/lib/bb/tests/fetch.py
>> @@ -1,22 +1,9 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # BitBake Tests for the Fetcher (fetch2/)
>> #
>> # Copyright (C) 2012 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>>
>> import unittest
>> @@ -893,12 +880,201 @@ class FetcherNetworkTest(FetcherTest):
>>
>> @skipIfNoNetwork()
>> def test_git_submodule(self):
>> - fetcher =
>> bb.fetch.Fetch(["gitsm://git.yoctoproject.org/git-submodule-test;rev=f12e57f2edf0aa534cf1616fa983d165a92b0842"],
>> self.d)
>> + # URL with ssh submodules
>> + url =
>> "gitsm://git.yoctoproject.org/git-submodule-test;branch=ssh-gitsm-tests;rev=049da4a6cb198d7c0302e9e8b243a1443cb809a7"
>> + # Original URL (comment this if you have ssh access to
>> git.yoctoproject.org)
>> + url =
>> "gitsm://git.yoctoproject.org/git-submodule-test;branch=master;rev=a2885dd7d25380d23627e7544b7bbb55014b16ee"
>> + fetcher = bb.fetch.Fetch([url], self.d)
>> + fetcher.download()
>> + # Previous cwd has been deleted
>> + os.chdir(os.path.dirname(self.unpackdir))
>> + fetcher.unpack(self.unpackdir)
>> +
>> + repo_path = os.path.join(self.tempdir, 'unpacked', 'git')
>> + self.assertTrue(os.path.exists(repo_path), msg='Unpacked
>> repository missing')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> 'bitbake')), msg='bitbake submodule missing')
>> + self.assertFalse(os.path.exists(os.path.join(repo_path,
>> 'na')), msg='uninitialized submodule present') +
>> + # Only when we're running the extended test with a
>> submodule's submodule, can we check this.
>> + if os.path.exists(os.path.join(repo_path,
>> 'bitbake-gitsm-test1')):
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> 'bitbake-gitsm-test1', 'bitbake')), msg='submodule of submodule
>> missing') +
>> + @skipIfNoNetwork()
>> + def test_git_submodule_dbus_broker(self):
>> + # The following external repositories have show failures in
>> fetch and unpack operations
>> + # We want to avoid regressions!
>> + url =
>> "gitsm://github.com/bus1/dbus-broker;protocol=git;rev=fc874afa0992d0c75ec25acb43d344679f0ee7d2"
>> + fetcher = bb.fetch.Fetch([url], self.d)
>> + fetcher.download()
>> + # Previous cwd has been deleted
>> + os.chdir(os.path.dirname(self.unpackdir))
>> + fetcher.unpack(self.unpackdir)
>> +
>> + repo_path = os.path.join(self.tempdir, 'unpacked', 'git')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> '.git/modules/subprojects/c-dvar/config')), msg='Missing submodule
>> config "subprojects/c-dvar"')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> '.git/modules/subprojects/c-list/config')), msg='Missing submodule
>> config "subprojects/c-list"')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> '.git/modules/subprojects/c-rbtree/config')), msg='Missing submodule
>> config "subprojects/c-rbtree"')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> '.git/modules/subprojects/c-sundry/config')), msg='Missing submodule
>> config "subprojects/c-sundry"')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> '.git/modules/subprojects/c-utf8/config')), msg='Missing submodule
>> config "subprojects/c-utf8"') +
>> + @skipIfNoNetwork()
>> + def test_git_submodule_CLI11(self):
>> + url =
>> "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=bd4dc911847d0cde7a6b41dfa626a85aab213baf"
>> + fetcher = bb.fetch.Fetch([url], self.d)
>> + fetcher.download()
>> + # Previous cwd has been deleted
>> + os.chdir(os.path.dirname(self.unpackdir))
>> + fetcher.unpack(self.unpackdir)
>> +
>> + repo_path = os.path.join(self.tempdir, 'unpacked', 'git')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> '.git/modules/extern/googletest/config')), msg='Missing submodule
>> config "extern/googletest"')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> '.git/modules/extern/json/config')), msg='Missing submodule config
>> "extern/json"')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> '.git/modules/extern/sanitizers/config')), msg='Missing submodule
>> config "extern/sanitizers"') +
>> + @skipIfNoNetwork()
>> + def test_git_submodule_update_CLI11(self):
>> + """ Prevent regression on update detection not finding
>> missing submodule, or modules without needed commits """
>> + url =
>> "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=cf6a99fa69aaefe477cc52e3ef4a7d2d7fa40714"
>> + fetcher = bb.fetch.Fetch([url], self.d)
>> + fetcher.download()
>> +
>> + # CLI11 that pulls in a newer nlohmann-json
>> + url =
>> "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=49ac989a9527ee9bb496de9ded7b4872c2e0e5ca"
>> + fetcher = bb.fetch.Fetch([url], self.d)
>> + fetcher.download()
>> + # Previous cwd has been deleted
>> + os.chdir(os.path.dirname(self.unpackdir))
>> + fetcher.unpack(self.unpackdir)
>> +
>> + repo_path = os.path.join(self.tempdir, 'unpacked', 'git')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> '.git/modules/extern/googletest/config')), msg='Missing submodule
>> config "extern/googletest"')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> '.git/modules/extern/json/config')), msg='Missing submodule config
>> "extern/json"')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> '.git/modules/extern/sanitizers/config')), msg='Missing submodule
>> config "extern/sanitizers"') +
>> + @skipIfNoNetwork()
>> + def test_git_submodule_aktualizr(self):
>> + url =
>> "gitsm://github.com/advancedtelematic/aktualizr;branch=master;protocol=git;rev=d00d1a04cc2366d1a5f143b84b9f507f8bd32c44"
>> + fetcher = bb.fetch.Fetch([url], self.d)
>> fetcher.download()
>> # Previous cwd has been deleted
>> os.chdir(os.path.dirname(self.unpackdir))
>> fetcher.unpack(self.unpackdir)
>>
>> + repo_path = os.path.join(self.tempdir, 'unpacked', 'git')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> '.git/modules/partial/extern/isotp-c/config')), msg='Missing
>> submodule config "partial/extern/isotp-c/config"')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> '.git/modules/partial/extern/isotp-c/modules/deps/bitfield-c/config')),
>> msg='Missing submodule config
>> "partial/extern/isotp-c/modules/deps/bitfield-c/config"')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> 'partial/extern/isotp-c/deps/bitfield-c/.git')), msg="Submodule of
>> submodule isotp-c did not unpack properly")
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> '.git/modules/tests/tuf-test-vectors/config')), msg='Missing
>> submodule config "tests/tuf-test-vectors/config"')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> '.git/modules/third_party/googletest/config')), msg='Missing
>> submodule config "third_party/googletest/config"')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> '.git/modules/third_party/HdrHistogram_c/config')), msg='Missing
>> submodule config "third_party/HdrHistogram_c/config"') +
>> + @skipIfNoNetwork()
>> + def test_git_submodule_iotedge(self):
>> + """ Prevent regression on deeply nested submodules not being
>> checked out properly, even though they were fetched. """ +
>> + # This repository also has submodules where the module
>> (name), path and url do not align
>> + url =
>> "gitsm://github.com/azure/iotedge.git;protocol=git;rev=d76e0316c6f324345d77c48a83ce836d09392699"
>> + fetcher = bb.fetch.Fetch([url], self.d)
>> + fetcher.download()
>> + # Previous cwd has been deleted
>> + os.chdir(os.path.dirname(self.unpackdir))
>> + fetcher.unpack(self.unpackdir)
>> +
>> + repo_path = os.path.join(self.tempdir, 'unpacked', 'git')
>> +
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared/README.md')),
>> msg='Missing submodule checkout')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared/testtools/ctest/README.md')),
>> msg='Missing submodule checkout')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared/testtools/testrunner/readme.md')),
>> msg='Missing submodule checkout')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared/testtools/umock-c/readme.md')),
>> msg='Missing submodule checkout')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared/testtools/umock-c/deps/ctest/README.md')),
>> msg='Missing submodule checkout')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared/testtools/umock-c/deps/testrunner/readme.md')),
>> msg='Missing submodule checkout')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/utpm/README.md')), msg='Missing
>> submodule checkout')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/utpm/deps/c-utility/README.md')),
>> msg='Missing submodule checkout')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/utpm/deps/c-utility/testtools/ctest/README.md')),
>> msg='Missing submodule checkout')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/utpm/deps/c-utility/testtools/testrunner/readme.md')),
>> msg='Missing submodule checkout')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/utpm/deps/c-utility/testtools/umock-c/readme.md')),
>> msg='Missing submodule checkout')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/utpm/deps/c-utility/testtools/umock-c/deps/ctest/README.md')),
>> msg='Missing submodule checkout')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/utpm/deps/c-utility/testtools/umock-c/deps/testrunner/readme.md')),
>> msg='Missing submodule checkout') + +class SVNTest(FetcherTest):
>> + def skipIfNoSvn():
>> + import shutil
>> + if not shutil.which("svn"):
>> + return unittest.skip("svn not installed, tests being
>> skipped") +
>> + if not shutil.which("svnadmin"):
>> + return unittest.skip("svnadmin not installed, tests
>> being skipped") +
>> + return lambda f: f
>> +
>> + @skipIfNoSvn()
>> + def setUp(self):
>> + """ Create a local repository """
>> +
>> + super(SVNTest, self).setUp()
>> +
>> + # Create something we can fetch
>> + src_dir = tempfile.mkdtemp(dir=self.tempdir,
>> + prefix='svnfetch_srcdir_')
>> + src_dir = os.path.abspath(src_dir)
>> + bb.process.run("echo readme > README.md", cwd=src_dir)
>> +
>> + # Store it in a local SVN repository
>> + repo_dir = tempfile.mkdtemp(dir=self.tempdir,
>> + prefix='svnfetch_localrepo_')
>> + repo_dir = os.path.abspath(repo_dir)
>> + bb.process.run("svnadmin create project", cwd=repo_dir)
>> +
>> + self.repo_url = "file://%s/project" % repo_dir
>> + bb.process.run("svn import --non-interactive -m 'Initial
>> import' %s %s/trunk" % (src_dir, self.repo_url),
>> + cwd=repo_dir)
>> +
>> + bb.process.run("svn co %s svnfetch_co" % self.repo_url,
>> cwd=self.tempdir)
>> + # Github will emulate SVN. Use this to check if we're
>> downloding...
>> + bb.process.run("svn propset svn:externals 'bitbake
>> http://github.com/openembedded/bitbake' .",
>> + cwd=os.path.join(self.tempdir, 'svnfetch_co',
>> 'trunk'))
>> + bb.process.run("svn commit --non-interactive -m 'Add
>> external'",
>> + cwd=os.path.join(self.tempdir, 'svnfetch_co',
>> 'trunk')) +
>> + self.src_dir = src_dir
>> + self.repo_dir = repo_dir
>> +
>> + @skipIfNoSvn()
>> + def tearDown(self):
>> + os.chdir(self.origdir)
>> + if os.environ.get("BB_TMPDIR_NOCLEAN") == "yes":
>> + print("Not cleaning up %s. Please remove manually." %
>> self.tempdir)
>> + else:
>> + bb.utils.prunedir(self.tempdir)
>> +
>> + @skipIfNoSvn()
>> + @skipIfNoNetwork()
>> + def test_noexternal_svn(self):
>> + # Always match the rev count from setUp (currently rev 2)
>> + url = "svn://%s;module=trunk;protocol=file;rev=2" %
>> self.repo_url.replace('file://', '')
>> + fetcher = bb.fetch.Fetch([url], self.d)
>> + fetcher.download()
>> + os.chdir(os.path.dirname(self.unpackdir))
>> + fetcher.unpack(self.unpackdir)
>> +
>> + self.assertTrue(os.path.exists(os.path.join(self.unpackdir,
>> 'trunk')), msg="Missing trunk")
>> + self.assertTrue(os.path.exists(os.path.join(self.unpackdir,
>> 'trunk', 'README.md')), msg="Missing contents")
>> + self.assertFalse(os.path.exists(os.path.join(self.unpackdir,
>> 'trunk/bitbake/trunk')), msg="External dir should NOT exist")
>> + self.assertFalse(os.path.exists(os.path.join(self.unpackdir,
>> 'trunk/bitbake/trunk', 'README')), msg="External README should NOT
>> exit") +
>> + @skipIfNoSvn()
>> + def test_external_svn(self):
>> + # Always match the rev count from setUp (currently rev 2)
>> + url =
>> "svn://%s;module=trunk;protocol=file;externals=allowed;rev=2" %
>> self.repo_url.replace('file://', '')
>> + fetcher = bb.fetch.Fetch([url], self.d)
>> + fetcher.download()
>> + os.chdir(os.path.dirname(self.unpackdir))
>> + fetcher.unpack(self.unpackdir)
>> +
>> + self.assertTrue(os.path.exists(os.path.join(self.unpackdir,
>> 'trunk')), msg="Missing trunk")
>> + self.assertTrue(os.path.exists(os.path.join(self.unpackdir,
>> 'trunk', 'README.md')), msg="Missing contents")
>> + self.assertTrue(os.path.exists(os.path.join(self.unpackdir,
>> 'trunk/bitbake/trunk')), msg="External dir should exist")
>> + self.assertTrue(os.path.exists(os.path.join(self.unpackdir,
>> 'trunk/bitbake/trunk', 'README')), msg="External README should exit")
>> class TrustedNetworksTest(FetcherTest):
>> def test_trusted_network(self):
>> @@ -1024,8 +1200,8 @@ class FetchLatestVersionTest(FetcherTest):
>> # packages with valid UPSTREAM_CHECK_URI and
>> UPSTREAM_CHECK_REGEX ("cups",
>> "http://www.cups.org/software/1.7.2/cups-1.7.2-source.tar.bz2",
>> "https://github.com/apple/cups/releases",
>> "(?P<name>cups\-)(?P<pver>((\d+[\.\-_]*)+))\-source\.tar\.gz") :
>> "2.0.0",
>> - ("db",
>> "http://download.oracle.com/berkeley-db/db-5.3.21.tar.gz",
>> "http://www.oracle.com/technetwork/products/berkeleydb/downloads/index-082944.html",
>> "http://download.oracle.com/otn/berkeley-db/(?P<name>db-)(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz")
>> - : "6.1.19",
>> + ("db",
>> "http://download.oracle.com/berkeley-db/db-5.3.21.tar.gz",
>> "http://ftp.debian.org/debian/pool/main/d/db5.3/",
>> "(?P<name>db5\.3_)(?P<pver>\d+(\.\d+)+).+\.orig\.tar\.xz")
>> + : "5.3.10",
>> }
>>
>> @skipIfNoNetwork()
>> @@ -1280,7 +1456,7 @@ class GitShallowTest(FetcherTest):
>>
>> def fetch(self, uri=None):
>> if uri is None:
>> - uris = self.d.getVar('SRC_URI', True).split()
>> + uris = self.d.getVar('SRC_URI').split()
>> uri = uris[0]
>> d = self.d
>> else:
>> @@ -1312,6 +1488,7 @@ class GitShallowTest(FetcherTest):
>> # fetch and unpack, from the shallow tarball
>> bb.utils.remove(self.gitdir, recurse=True)
>> bb.utils.remove(ud.clonedir, recurse=True)
>> + bb.utils.remove(ud.clonedir.replace('gitsource',
>> 'gitsubmodule'), recurse=True)
>> # confirm that the unpacked repo is used when no git clone
>> or git # mirror tarball is available
>> @@ -1338,7 +1515,7 @@ class GitShallowTest(FetcherTest):
>>
>> srcrev = self.git('rev-parse HEAD', cwd=self.srcdir).strip()
>> self.d.setVar('SRCREV', srcrev)
>> - uri = self.d.getVar('SRC_URI', True).split()[0]
>> + uri = self.d.getVar('SRC_URI').split()[0]
>> uri = '%s;nobranch=1;bare=1' % uri
>>
>> self.fetch_shallow(uri)
>> @@ -1466,6 +1643,7 @@ class GitShallowTest(FetcherTest):
>> self.git('config --add remote.origin.url "%s"' % smdir,
>> cwd=smdir) self.git('config --add remote.origin.fetch
>> "+refs/heads/*:refs/remotes/origin/*"', cwd=smdir)
>> self.add_empty_file('asub', cwd=smdir)
>> + self.add_empty_file('bsub', cwd=smdir)
>>
>> self.git('submodule init', cwd=self.srcdir)
>> self.git('submodule add file://%s' % smdir, cwd=self.srcdir)
>> @@ -1475,10 +1653,16 @@ class GitShallowTest(FetcherTest):
>> uri = 'gitsm://%s;protocol=file;subdir=${S}' % self.srcdir
>> fetcher, ud = self.fetch_shallow(uri)
>>
>> + # Verify the main repository is shallow
>> self.assertRevCount(1)
>> - assert './.git/modules/' in bb.process.run('tar -tzf %s' %
>> os.path.join(self.dldir, ud.mirrortarballs[0]))[0] +
>> + # Verify the gitsubmodule directory is present
>> assert os.listdir(os.path.join(self.gitdir, 'gitsubmodule'))
>>
>> + # Verify the submodule is also shallow
>> + self.assertRevCount(1, cwd=os.path.join(self.gitdir,
>> 'gitsubmodule')) +
>> +
>> if any(os.path.exists(os.path.join(p, 'git-annex')) for p in
>> os.environ.get('PATH').split(':')): def test_shallow_annex(self):
>> self.add_empty_file('a')
>> @@ -1510,7 +1694,7 @@ class GitShallowTest(FetcherTest):
>> self.add_empty_file('f')
>> self.assertRevCount(7, cwd=self.srcdir)
>>
>> - uri = self.d.getVar('SRC_URI', True).split()[0]
>> + uri = self.d.getVar('SRC_URI').split()[0]
>> uri = '%s;branch=master,a_branch;name=master,a_branch' % uri
>>
>> self.d.setVar('BB_GIT_SHALLOW_DEPTH', '0')
>> @@ -1536,7 +1720,7 @@ class GitShallowTest(FetcherTest):
>> self.add_empty_file('f')
>> self.assertRevCount(7, cwd=self.srcdir)
>>
>> - uri = self.d.getVar('SRC_URI', True).split()[0]
>> + uri = self.d.getVar('SRC_URI').split()[0]
>> uri = '%s;branch=master,a_branch;name=master,a_branch' % uri
>>
>> self.d.setVar('BB_GIT_SHALLOW_DEPTH', '0')
>> @@ -1724,3 +1908,83 @@ class GitShallowTest(FetcherTest):
>>
>> dir = os.listdir(self.unpackdir + "/git/")
>> self.assertIn("fstests.doap", dir)
>> +
>> +class GitLfsTest(FetcherTest):
>> + def setUp(self):
>> + FetcherTest.setUp(self)
>> +
>> + self.gitdir = os.path.join(self.tempdir, 'git')
>> + self.srcdir = os.path.join(self.tempdir, 'gitsource')
>> +
>> + self.d.setVar('WORKDIR', self.tempdir)
>> + self.d.setVar('S', self.gitdir)
>> + self.d.delVar('PREMIRRORS')
>> + self.d.delVar('MIRRORS')
>> +
>> + self.d.setVar('SRCREV', '${AUTOREV}')
>> + self.d.setVar('AUTOREV', '${@bb.fetch2.get_autorev(d)}')
>> +
>> + bb.utils.mkdirhier(self.srcdir)
>> + self.git('init', cwd=self.srcdir)
>> + with open(os.path.join(self.srcdir, '.gitattributes'), 'wt')
>> as attrs:
>> + attrs.write('*.mp3 filter=lfs -text')
>> + self.git(['add', '.gitattributes'], cwd=self.srcdir)
>> + self.git(['commit', '-m', "attributes", '.gitattributes'],
>> cwd=self.srcdir) +
>> + def git(self, cmd, cwd=None):
>> + if isinstance(cmd, str):
>> + cmd = 'git ' + cmd
>> + else:
>> + cmd = ['git'] + cmd
>> + if cwd is None:
>> + cwd = self.gitdir
>> + return bb.process.run(cmd, cwd=cwd)[0]
>> +
>> + def fetch(self, uri=None):
>> + uris = self.d.getVar('SRC_URI').split()
>> + uri = uris[0]
>> + d = self.d
>> +
>> + fetcher = bb.fetch2.Fetch(uris, d)
>> + fetcher.download()
>> + ud = fetcher.ud[uri]
>> + return fetcher, ud
>> +
>> + def test_lfs_enabled(self):
>> + import shutil
>> +
>> + uri = 'git://%s;protocol=file;subdir=${S};lfs=1' %
>> self.srcdir
>> + self.d.setVar('SRC_URI', uri)
>> +
>> + fetcher, ud = self.fetch()
>> + self.assertIsNotNone(ud.method._find_git_lfs)
>> +
>> + # If git-lfs can be found, the unpack should be successful
>> + ud.method._find_git_lfs = lambda d: True
>> + shutil.rmtree(self.gitdir, ignore_errors=True)
>> + fetcher.unpack(self.d.getVar('WORKDIR'))
>> +
>> + # If git-lfs cannot be found, the unpack should throw an
>> error
>> + with self.assertRaises(bb.fetch2.FetchError):
>> + ud.method._find_git_lfs = lambda d: False
>> + shutil.rmtree(self.gitdir, ignore_errors=True)
>> + fetcher.unpack(self.d.getVar('WORKDIR'))
>> +
>> + def test_lfs_disabled(self):
>> + import shutil
>> +
>> + uri = 'git://%s;protocol=file;subdir=${S};lfs=0' %
>> self.srcdir
>> + self.d.setVar('SRC_URI', uri)
>> +
>> + fetcher, ud = self.fetch()
>> + self.assertIsNotNone(ud.method._find_git_lfs)
>> +
>> + # If git-lfs can be found, the unpack should be successful
>> + ud.method._find_git_lfs = lambda d: True
>> + shutil.rmtree(self.gitdir, ignore_errors=True)
>> + fetcher.unpack(self.d.getVar('WORKDIR'))
>> +
>> + # If git-lfs cannot be found, the unpack should be successful
>> + ud.method._find_git_lfs = lambda d: False
>> + shutil.rmtree(self.gitdir, ignore_errors=True)
>> + fetcher.unpack(self.d.getVar('WORKDIR'))
>> diff --git a/bitbake/lib/bb/tests/parse.py
>> b/bitbake/lib/bb/tests/parse.py index 1bc4740..9afd1b2 100644
>> --- a/bitbake/lib/bb/tests/parse.py
>> +++ b/bitbake/lib/bb/tests/parse.py
>> @@ -1,22 +1,9 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # BitBake Test for lib/bb/parse/
>> #
>> # Copyright (C) 2015 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>>
>> import unittest
>> @@ -187,3 +174,21 @@ python () {
>> self.assertEqual(d1.getVar("VAR_var"), "B")
>> self.assertEqual(d2.getVar("VAR_var"), None)
>>
>> + addtask_deltask = """
>> +addtask do_patch after do_foo after do_unpack before do_configure
>> before do_compile +addtask do_fetch do_patch
>> +
>> +deltask do_fetch do_patch
>> +"""
>> + def test_parse_addtask_deltask(self):
>> + import sys
>> + f = self.parsehelper(self.addtask_deltask)
>> + d = bb.parse.handle(f.name, self.d)['']
>> +
>> + stdout = sys.stdout.getvalue()
>> + self.assertTrue("addtask contained multiple 'before'
>> keywords" in stdout)
>> + self.assertTrue("addtask contained multiple 'after'
>> keywords" in stdout)
>> + self.assertTrue('addtask ignored: " do_patch"' in stdout)
>> + self.assertTrue('deltask ignored: " do_patch"' in stdout)
>> + #self.assertTrue('dependent task do_foo for do_patch does
>> not exist' in stdout) +
>> diff --git a/bitbake/lib/bb/tests/persist_data.py
>> b/bitbake/lib/bb/tests/persist_data.py new file mode 100644
>> index 0000000..f641b5a
>> --- /dev/null
>> +++ b/bitbake/lib/bb/tests/persist_data.py
>> @@ -0,0 +1,129 @@
>> +#
>> +# BitBake Test for lib/bb/persist_data/
>> +#
>> +# Copyright (C) 2018 Garmin Ltd.
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> +import unittest
>> +import bb.data
>> +import bb.persist_data
>> +import tempfile
>> +import threading
>> +
>> +class PersistDataTest(unittest.TestCase):
>> + def _create_data(self):
>> + return bb.persist_data.persist('TEST_PERSIST_DATA', self.d)
>> +
>> + def setUp(self):
>> + self.d = bb.data.init()
>> + self.tempdir = tempfile.TemporaryDirectory()
>> + self.d['PERSISTENT_DIR'] = self.tempdir.name
>> + self.data = self._create_data()
>> + self.items = {
>> + 'A1': '1',
>> + 'B1': '2',
>> + 'C2': '3'
>> + }
>> + self.stress_count = 10000
>> + self.thread_count = 5
>> +
>> + for k,v in self.items.items():
>> + self.data[k] = v
>> +
>> + def tearDown(self):
>> + self.tempdir.cleanup()
>> +
>> + def _iter_helper(self, seen, iterator):
>> + with iter(iterator):
>> + for v in iterator:
>> + self.assertTrue(v in seen)
>> + seen.remove(v)
>> + self.assertEqual(len(seen), 0, '%s not seen' % seen)
>> +
>> + def test_get(self):
>> + for k, v in self.items.items():
>> + self.assertEqual(self.data[k], v)
>> +
>> + self.assertIsNone(self.data.get('D'))
>> + with self.assertRaises(KeyError):
>> + self.data['D']
>> +
>> + def test_set(self):
>> + for k, v in self.items.items():
>> + self.data[k] += '-foo'
>> +
>> + for k, v in self.items.items():
>> + self.assertEqual(self.data[k], v + '-foo')
>> +
>> + def test_delete(self):
>> + self.data['D'] = '4'
>> + self.assertEqual(self.data['D'], '4')
>> + del self.data['D']
>> + self.assertIsNone(self.data.get('D'))
>> + with self.assertRaises(KeyError):
>> + self.data['D']
>> +
>> + def test_contains(self):
>> + for k in self.items:
>> + self.assertTrue(k in self.data)
>> + self.assertTrue(self.data.has_key(k))
>> + self.assertFalse('NotFound' in self.data)
>> + self.assertFalse(self.data.has_key('NotFound'))
>> +
>> + def test_len(self):
>> + self.assertEqual(len(self.data), len(self.items))
>> +
>> + def test_iter(self):
>> + self._iter_helper(set(self.items.keys()), self.data)
>> +
>> + def test_itervalues(self):
>> + self._iter_helper(set(self.items.values()),
>> self.data.itervalues()) +
>> + def test_iteritems(self):
>> + self._iter_helper(set(self.items.items()),
>> self.data.iteritems()) +
>> + def test_get_by_pattern(self):
>> + self._iter_helper({'1', '2'}, self.data.get_by_pattern('_1'))
>> +
>> + def _stress_read(self, data):
>> + for i in range(self.stress_count):
>> + for k in self.items:
>> + data[k]
>> +
>> + def _stress_write(self, data):
>> + for i in range(self.stress_count):
>> + for k, v in self.items.items():
>> + data[k] = v + str(i)
>> +
>> + def _validate_stress(self):
>> + for k, v in self.items.items():
>> + self.assertEqual(self.data[k], v + str(self.stress_count
>> - 1)) +
>> + def test_stress(self):
>> + self._stress_read(self.data)
>> + self._stress_write(self.data)
>> + self._validate_stress()
>> +
>> + def test_stress_threads(self):
>> + def read_thread():
>> + data = self._create_data()
>> + self._stress_read(data)
>> +
>> + def write_thread():
>> + data = self._create_data()
>> + self._stress_write(data)
>> +
>> + threads = []
>> + for i in range(self.thread_count):
>> + threads.append(threading.Thread(target=read_thread))
>> + threads.append(threading.Thread(target=write_thread))
>> +
>> + for t in threads:
>> + t.start()
>> + self._stress_read(self.data)
>> + for t in threads:
>> + t.join()
>> + self._validate_stress()
>> +
>> diff --git a/bitbake/lib/bb/tests/runqueue-tests/classes/base.bbclass
>> b/bitbake/lib/bb/tests/runqueue-tests/classes/base.bbclass new file
>> mode 100644 index 0000000..b57650d
>> --- /dev/null
>> +++ b/bitbake/lib/bb/tests/runqueue-tests/classes/base.bbclass
>> @@ -0,0 +1,262 @@
>> +SLOWTASKS ??= ""
>> +SSTATEVALID ??= ""
>> +
>> +def stamptask(d):
>> + import time
>> +
>> + thistask = d.expand("${PN}:${BB_CURRENTTASK}")
>> + stampname = d.expand("${TOPDIR}/%s.run" % thistask)
>> + with open(stampname, "a+") as f:
>> + f.write(d.getVar("BB_UNIHASH") + "\n")
>> +
>> + if d.getVar("BB_CURRENT_MC") != "default":
>> + thistask =
>> d.expand("${BB_CURRENT_MC}:${PN}:${BB_CURRENTTASK}")
>> + if thistask in d.getVar("SLOWTASKS").split():
>> + bb.note("Slowing task %s" % thistask)
>> + time.sleep(0.5)
>> + if d.getVar("BB_HASHSERVE"):
>> + task = d.getVar("BB_CURRENTTASK")
>> + if task in ['package', 'package_qa', 'packagedata',
>> 'package_write_ipk', 'package_write_rpm', 'populate_lic',
>> 'populate_sysroot']:
>> + bb.parse.siggen.report_unihash(os.getcwd(),
>> d.getVar("BB_CURRENTTASK"), d) +
>> + with open(d.expand("${TOPDIR}/task.log"), "a+") as f:
>> + f.write(thistask + "\n")
>> +
>> +
>> +def sstate_output_hash(path, sigfile, task, d):
>> + import hashlib
>> + h = hashlib.sha256()
>> + h.update(d.expand("${PN}:${BB_CURRENTTASK}").encode('utf-8'))
>> + return h.hexdigest()
>> +
>> +python do_fetch() {
>> + # fetch
>> + stamptask(d)
>> +}
>> +python do_unpack() {
>> + # unpack
>> + stamptask(d)
>> +}
>> +python do_patch() {
>> + # patch
>> + stamptask(d)
>> +}
>> +python do_populate_lic() {
>> + # populate_lic
>> + stamptask(d)
>> +}
>> +python do_prepare_recipe_sysroot() {
>> + # prepare_recipe_sysroot
>> + stamptask(d)
>> +}
>> +python do_configure() {
>> + # configure
>> + stamptask(d)
>> +}
>> +python do_compile() {
>> + # compile
>> + stamptask(d)
>> +}
>> +python do_install() {
>> + # install
>> + stamptask(d)
>> +}
>> +python do_populate_sysroot() {
>> + # populate_sysroot
>> + stamptask(d)
>> +}
>> +python do_package() {
>> + # package
>> + stamptask(d)
>> +}
>> +python do_package_write_ipk() {
>> + # package_write_ipk
>> + stamptask(d)
>> +}
>> +python do_package_write_rpm() {
>> + # package_write_rpm
>> + stamptask(d)
>> +}
>> +python do_packagedata() {
>> + # packagedata
>> + stamptask(d)
>> +}
>> +python do_package_qa() {
>> + # package_qa
>> + stamptask(d)
>> +}
>> +python do_build() {
>> + # build
>> + stamptask(d)
>> +}
>> +do_prepare_recipe_sysroot[deptask] = "do_populate_sysroot"
>> +do_package[deptask] += "do_packagedata"
>> +do_build[recrdeptask] += "do_deploy"
>> +do_build[recrdeptask] += "do_package_write_ipk"
>> +do_build[recrdeptask] += "do_package_write_rpm"
>> +do_package_qa[rdeptask] = "do_packagedata"
>> +do_populate_lic_deploy[recrdeptask] += "do_populate_lic do_deploy"
>> +
>> +DEBIANRDEP = "do_packagedata"
>> +oo_package_write_ipk[rdeptask] = "${DEBIANRDEP}"
>> +do_package_write_rpm[rdeptask] = "${DEBIANRDEP}"
>> +
>> +addtask fetch
>> +addtask unpack after do_fetch
>> +addtask patch after do_unpack
>> +addtask prepare_recipe_sysroot after do_patch
>> +addtask configure after do_prepare_recipe_sysroot
>> +addtask compile after do_configure
>> +addtask install after do_compile
>> +addtask populate_sysroot after do_install
>> +addtask package after do_install
>> +addtask package_write_ipk after do_packagedata do_package
>> +addtask package_write_rpm after do_packagedata do_package
>> +addtask packagedata after do_package
>> +addtask package_qa after do_package
>> +addtask build after do_package_qa do_package_write_rpm
>> do_package_write_ipk do_populate_sysroot +
>> +python do_package_setscene() {
>> + stamptask(d)
>> +}
>> +python do_package_qa_setscene() {
>> + stamptask(d)
>> +}
>> +python do_package_write_ipk_setscene() {
>> + stamptask(d)
>> +}
>> +python do_package_write_rpm_setscene() {
>> + stamptask(d)
>> +}
>> +python do_packagedata_setscene() {
>> + stamptask(d)
>> +}
>> +python do_populate_lic_setscene() {
>> + stamptask(d)
>> +}
>> +python do_populate_sysroot_setscene() {
>> + stamptask(d)
>> +}
>> +
>> +addtask package_setscene
>> +addtask package_qa_setscene
>> +addtask package_write_ipk_setscene
>> +addtask package_write_rpm_setscene
>> +addtask packagedata_setscene
>> +addtask populate_lic_setscene
>> +addtask populate_sysroot_setscene
>> +
>> +BB_SETSCENE_DEPVALID = "setscene_depvalid"
>> +
>> +def setscene_depvalid(task, taskdependees, notneeded, d, log=None):
>> + # taskdependees is a dict of tasks which depend on task, each
>> being a 3 item list of [PN, TASKNAME, FILENAME]
>> + # task is included in taskdependees too
>> + # Return - False - We need this dependency
>> + # - True - We can skip this dependency
>> + import re
>> +
>> + def logit(msg, log):
>> + if log is not None:
>> + log.append(msg)
>> + else:
>> + bb.debug(2, msg)
>> +
>> + logit("Considering setscene task: %s" %
>> (str(taskdependees[task])), log) +
>> + def isNativeCross(x):
>> + return x.endswith("-native") or "-cross-" in x or
>> "-crosssdk" in x or x.endswith("-cross") +
>> + # We only need to trigger populate_lic through direct
>> dependencies
>> + if taskdependees[task][1] == "do_populate_lic":
>> + return True
>> +
>> + # We only need to trigger packagedata through direct dependencies
>> + # but need to preserve packagedata on packagedata links
>> + if taskdependees[task][1] == "do_packagedata":
>> + for dep in taskdependees:
>> + if taskdependees[dep][1] == "do_packagedata":
>> + return False
>> + return True
>> +
>> + for dep in taskdependees:
>> + logit(" considering dependency: %s" %
>> (str(taskdependees[dep])), log)
>> + if task == dep:
>> + continue
>> + if dep in notneeded:
>> + continue
>> + # do_package_write_* and do_package doesn't need do_package
>> + if taskdependees[task][1] == "do_package" and
>> taskdependees[dep][1] in ['do_package', 'do_package_write_ipk',
>> 'do_package_write_rpm', 'do_packagedata', 'do_package_qa']:
>> + continue
>> + # do_package_write_* need do_populate_sysroot as they're
>> mainly postinstall dependencies
>> + if taskdependees[task][1] == "do_populate_sysroot" and
>> taskdependees[dep][1] in ['do_package_write_ipk',
>> 'do_package_write_rpm']:
>> + return False
>> + # do_package/packagedata/package_qa don't need
>> do_populate_sysroot
>> + if taskdependees[task][1] == "do_populate_sysroot" and
>> taskdependees[dep][1] in ['do_package', 'do_packagedata',
>> 'do_package_qa']:
>> + continue
>> + # Native/Cross packages don't exist and are noexec anyway
>> + if isNativeCross(taskdependees[dep][0]) and
>> taskdependees[dep][1] in ['do_package_write_ipk',
>> 'do_package_write_rpm', 'do_packagedata', 'do_package',
>> 'do_package_qa']:
>> + continue
>> +
>> + # This is due to the [depends] in useradd.bbclass
>> complicating matters
>> + # The logic *is* reversed here due to the way hard setscene
>> dependencies are injected
>> + if (taskdependees[task][1] == 'do_package' or
>> taskdependees[task][1] == 'do_populate_sysroot') and
>> taskdependees[dep][0].endswith(('shadow-native', 'shadow-sysroot',
>> 'base-passwd', 'pseudo-native')) and taskdependees[dep][1] ==
>> 'do_populate_sysroot':
>> + continue
>> +
>> + # Consider sysroot depending on sysroot tasks
>> + if taskdependees[task][1] == 'do_populate_sysroot' and
>> taskdependees[dep][1] == 'do_populate_sysroot':
>> + # Native/Cross populate_sysroot need their dependencies
>> + if isNativeCross(taskdependees[task][0]) and
>> isNativeCross(taskdependees[dep][0]):
>> + return False
>> + # Target populate_sysroot depended on by cross tools
>> need to be installed
>> + if isNativeCross(taskdependees[dep][0]):
>> + return False
>> + # Native/cross tools depended upon by target sysroot are
>> not needed
>> + # Add an exception for shadow-native as required by
>> useradd.bbclass
>> + if isNativeCross(taskdependees[task][0]) and
>> taskdependees[task][0] != 'shadow-native':
>> + continue
>> + # Target populate_sysroot need their dependencies
>> + return False
>> +
>> +
>> + if taskdependees[dep][1] == "do_populate_lic":
>> + continue
>> +
>> + # Safe fallthrough default
>> + logit(" Default setscene dependency fall through due to
>> dependency: %s" % (str(taskdependees[dep])), log)
>> + return False
>> + return True
>> +
>> +BB_HASHCHECK_FUNCTION = "sstate_checkhashes"
>> +
>> +def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0,
>> **kwargs): +
>> + found = set()
>> + missed = set()
>> +
>> + valid = d.getVar("SSTATEVALID").split()
>> +
>> + for tid in sorted(sq_data['hash']):
>> + n =
>> os.path.basename(bb.runqueue.fn_from_tid(tid)).split(".")[0] + ":do_"
>> + bb.runqueue.taskname_from_tid(tid)[3:]
>> + print(n)
>> + stampfile = d.expand("${TOPDIR}/%s.run" % n.replace("do_",
>> ""))
>> + if n in valid:
>> + bb.note("SState: Found valid sstate for %s" % n)
>> + found.add(tid)
>> + elif n + ":" + sq_data['hash'][tid] in valid:
>> + bb.note("SState: Found valid sstate for %s" % n)
>> + found.add(tid)
>> + elif os.path.exists(stampfile):
>> + with open(stampfile, "r") as f:
>> + hash = f.readline().strip()
>> + if hash == sq_data['hash'][tid]:
>> + bb.note("SState: Found valid sstate for %s (already
>> run)" % n)
>> + found.add(tid)
>> + else:
>> + bb.note("SState: sstate hash didn't match previous
>> run for %s (%s vs %s)" % (n, sq_data['hash'][tid], hash))
>> + missed.add(tid)
>> + else:
>> + missed.add(tid)
>> + bb.note("SState: Found no valid sstate for %s (%s)" %
>> (n, sq_data['hash'][tid])) +
>> + return found
>> +
>> diff --git
>> a/bitbake/lib/bb/tests/runqueue-tests/classes/image.bbclass
>> b/bitbake/lib/bb/tests/runqueue-tests/classes/image.bbclass new file
>> mode 100644 index 0000000..da9ff11 --- /dev/null
>> +++ b/bitbake/lib/bb/tests/runqueue-tests/classes/image.bbclass
>> @@ -0,0 +1,5 @@
>> +do_rootfs[recrdeptask] += "do_package_write_deb do_package_qa"
>> +do_rootfs[recrdeptask] += "do_package_write_ipk do_package_qa"
>> +do_rootfs[recrdeptask] += "do_package_write_rpm do_package_qa
>> +do_rootfs[recrdeptask] += "do_packagedata"
>> +do_rootfs[recrdeptask] += "do_populate_lic"
>> diff --git
>> a/bitbake/lib/bb/tests/runqueue-tests/classes/native.bbclass
>> b/bitbake/lib/bb/tests/runqueue-tests/classes/native.bbclass new file
>> mode 100644 index 0000000..7eaaee5 --- /dev/null
>> +++ b/bitbake/lib/bb/tests/runqueue-tests/classes/native.bbclass
>> @@ -0,0 +1,2 @@
>> +RECIPERDEPTASK = "do_populate_sysroot"
>> +do_populate_sysroot[rdeptask] = "${RECIPERDEPTASK}"
>> diff --git a/bitbake/lib/bb/tests/runqueue-tests/conf/bitbake.conf
>> b/bitbake/lib/bb/tests/runqueue-tests/conf/bitbake.conf new file mode
>> 100644 index 0000000..5e451fc
>> --- /dev/null
>> +++ b/bitbake/lib/bb/tests/runqueue-tests/conf/bitbake.conf
>> @@ -0,0 +1,16 @@
>> +CACHE = "${TOPDIR}/cache"
>> +THISDIR = "${@os.path.dirname(d.getVar('FILE'))}"
>> +COREBASE :=
>> "${@os.path.normpath(os.path.dirname(d.getVar('FILE')+'/../../'))}"
>> +BBFILES = "${COREBASE}/recipes/*.bb" +PROVIDES = "${PN}"
>> +PN = "${@bb.parse.vars_from_file(d.getVar('FILE', False),d)[0]}"
>> +PF = "${BB_CURRENT_MC}:${PN}"
>> +export PATH
>> +TMPDIR ??= "${TOPDIR}"
>> +STAMP = "${TMPDIR}/stamps/${PN}"
>> +T = "${TMPDIR}/workdir/${PN}/temp"
>> +BB_NUMBER_THREADS = "4"
>> +
>> +BB_HASHBASE_WHITELIST = "BB_CURRENT_MC BB_HASHSERVE TMPDIR TOPDIR
>> SLOWTASKS SSTATEVALID FILE" +
>> +include conf/multiconfig/${BB_CURRENT_MC}.conf
>> diff --git
>> a/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc1.conf
>> b/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc1.conf new
>> file mode 100644 index 0000000..ecf23e1 --- /dev/null
>> +++ b/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc1.conf
>> @@ -0,0 +1 @@
>> +TMPDIR = "${TOPDIR}/mc1/"
>> diff --git
>> a/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc2.conf
>> b/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc2.conf new
>> file mode 100644 index 0000000..eef338e --- /dev/null
>> +++ b/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc2.conf
>> @@ -0,0 +1 @@
>> +TMPDIR = "${TOPDIR}/mc2/"
>> diff --git a/bitbake/lib/bb/tests/runqueue-tests/recipes/a1.bb
>> b/bitbake/lib/bb/tests/runqueue-tests/recipes/a1.bb new file mode
>> 100644 index 0000000..e69de29
>> diff --git a/bitbake/lib/bb/tests/runqueue-tests/recipes/b1.bb
>> b/bitbake/lib/bb/tests/runqueue-tests/recipes/b1.bb new file mode
>> 100644 index 0000000..c0b288e
>> --- /dev/null
>> +++ b/bitbake/lib/bb/tests/runqueue-tests/recipes/b1.bb
>> @@ -0,0 +1 @@
>> +DEPENDS = "a1"
>> \ No newline at end of file
>> diff --git a/bitbake/lib/bb/tests/runqueue-tests/recipes/c1.bb
>> b/bitbake/lib/bb/tests/runqueue-tests/recipes/c1.bb new file mode
>> 100644 index 0000000..e69de29
>> diff --git a/bitbake/lib/bb/tests/runqueue-tests/recipes/d1.bb
>> b/bitbake/lib/bb/tests/runqueue-tests/recipes/d1.bb new file mode
>> 100644 index 0000000..5ba1975
>> --- /dev/null
>> +++ b/bitbake/lib/bb/tests/runqueue-tests/recipes/d1.bb
>> @@ -0,0 +1,3 @@
>> +DEPENDS = "a1"
>> +
>> +do_package_setscene[depends] = "a1:do_populate_sysroot_setscene"
>> diff --git a/bitbake/lib/bb/tests/runqueue-tests/recipes/e1.bb
>> b/bitbake/lib/bb/tests/runqueue-tests/recipes/e1.bb new file mode
>> 100644 index 0000000..1588bc8
>> --- /dev/null
>> +++ b/bitbake/lib/bb/tests/runqueue-tests/recipes/e1.bb
>> @@ -0,0 +1 @@
>> +DEPENDS = "b1"
>> \ No newline at end of file
>> diff --git a/bitbake/lib/bb/tests/runqueue.py
>> b/bitbake/lib/bb/tests/runqueue.py new file mode 100644
>> index 0000000..5e64391
>> --- /dev/null
>> +++ b/bitbake/lib/bb/tests/runqueue.py
>> @@ -0,0 +1,323 @@
>> +#
>> +# BitBake Tests for runqueue task processing
>> +#
>> +# Copyright (C) 2019 Richard Purdie
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> +import unittest
>> +import bb
>> +import os
>> +import tempfile
>> +import subprocess
>> +import sys
>> +import time
>> +
>> +#
>> +# TODO:
>> +# Add tests on task ordering (X happens before Y after Z)
>> +#
>> +
>> +class RunQueueTests(unittest.TestCase):
>> +
>> + alltasks = ['package', 'fetch', 'unpack', 'patch',
>> 'prepare_recipe_sysroot', 'configure',
>> + 'compile', 'install', 'packagedata', 'package_qa',
>> 'package_write_rpm', 'package_write_ipk',
>> + 'populate_sysroot', 'build']
>> + a1_sstatevalid = "a1:do_package a1:do_package_qa
>> a1:do_packagedata a1:do_package_write_ipk a1:do_package_write_rpm
>> a1:do_populate_lic a1:do_populate_sysroot"
>> + b1_sstatevalid = "b1:do_package b1:do_package_qa
>> b1:do_packagedata b1:do_package_write_ipk b1:do_package_write_rpm
>> b1:do_populate_lic b1:do_populate_sysroot" +
>> + def run_bitbakecmd(self, cmd, builddir, sstatevalid="",
>> slowtasks="", extraenv=None, cleanup=False):
>> + env = os.environ.copy()
>> + env["BBPATH"] =
>> os.path.realpath(os.path.join(os.path.dirname(__file__),
>> "runqueue-tests"))
>> + env["BB_ENV_EXTRAWHITE"] = "SSTATEVALID SLOWTASKS"
>> + env["SSTATEVALID"] = sstatevalid
>> + env["SLOWTASKS"] = slowtasks
>> + if extraenv:
>> + for k in extraenv:
>> + env[k] = extraenv[k]
>> + env["BB_ENV_EXTRAWHITE"] = env["BB_ENV_EXTRAWHITE"]
>> + " " + k
>> + try:
>> + output = subprocess.check_output(cmd, env=env,
>> stderr=subprocess.STDOUT,universal_newlines=True, cwd=builddir)
>> + print(output)
>> + except subprocess.CalledProcessError as e:
>> + self.fail("Command %s failed with %s" % (cmd, e.output))
>> + tasks = []
>> + tasklog = builddir + "/task.log"
>> + if os.path.exists(tasklog):
>> + with open(tasklog, "r") as f:
>> + tasks = [line.rstrip() for line in f]
>> + if cleanup:
>> + os.remove(tasklog)
>> + return tasks
>> +
>> + def test_no_setscenevalid(self):
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + cmd = ["bitbake", "a1"]
>> + sstatevalid = ""
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
>> + expected = ['a1:' + x for x in self.alltasks]
>> + self.assertEqual(set(tasks), set(expected))
>> +
>> + def test_single_setscenevalid(self):
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + cmd = ["bitbake", "a1"]
>> + sstatevalid = "a1:do_package"
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
>> + expected = ['a1:package_setscene', 'a1:fetch',
>> 'a1:unpack', 'a1:patch', 'a1:prepare_recipe_sysroot', 'a1:configure',
>> + 'a1:compile', 'a1:install',
>> 'a1:packagedata', 'a1:package_qa', 'a1:package_write_rpm',
>> 'a1:package_write_ipk',
>> + 'a1:populate_sysroot', 'a1:build']
>> + self.assertEqual(set(tasks), set(expected))
>> +
>> + def test_intermediate_setscenevalid(self):
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + cmd = ["bitbake", "a1"]
>> + sstatevalid = "a1:do_package a1:do_populate_sysroot"
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
>> + expected = ['a1:package_setscene', 'a1:packagedata',
>> 'a1:package_qa', 'a1:package_write_rpm', 'a1:package_write_ipk',
>> + 'a1:populate_sysroot_setscene', 'a1:build']
>> + self.assertEqual(set(tasks), set(expected))
>> +
>> + def test_intermediate_notcovered(self):
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + cmd = ["bitbake", "a1"]
>> + sstatevalid = "a1:do_package_qa a1:do_packagedata
>> a1:do_package_write_ipk a1:do_package_write_rpm a1:do_populate_lic
>> a1:do_populate_sysroot"
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
>> + expected = ['a1:package_write_ipk_setscene',
>> 'a1:package_write_rpm_setscene', 'a1:packagedata_setscene',
>> + 'a1:package_qa_setscene', 'a1:build',
>> 'a1:populate_sysroot_setscene']
>> + self.assertEqual(set(tasks), set(expected))
>> +
>> + def test_all_setscenevalid(self):
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + cmd = ["bitbake", "a1"]
>> + sstatevalid = self.a1_sstatevalid
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
>> + expected = ['a1:package_write_ipk_setscene',
>> 'a1:package_write_rpm_setscene', 'a1:packagedata_setscene',
>> + 'a1:package_qa_setscene', 'a1:build',
>> 'a1:populate_sysroot_setscene']
>> + self.assertEqual(set(tasks), set(expected))
>> +
>> + def test_no_settasks(self):
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + cmd = ["bitbake", "a1", "-c", "patch"]
>> + sstatevalid = self.a1_sstatevalid
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
>> + expected = ['a1:fetch', 'a1:unpack', 'a1:patch']
>> + self.assertEqual(set(tasks), set(expected))
>> +
>> + def test_mix_covered_notcovered(self):
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + cmd = ["bitbake", "a1:do_patch",
>> "a1:do_populate_sysroot"]
>> + sstatevalid = self.a1_sstatevalid
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
>> + expected = ['a1:fetch', 'a1:unpack', 'a1:patch',
>> 'a1:populate_sysroot_setscene']
>> + self.assertEqual(set(tasks), set(expected))
>> +
>> +
>> + # Test targets with intermediate setscene tasks alongside a
>> target with no intermediate setscene tasks
>> + def test_mixed_direct_tasks_setscene_tasks(self):
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + cmd = ["bitbake", "c1:do_patch", "a1"]
>> + sstatevalid = self.a1_sstatevalid
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
>> + expected = ['c1:fetch', 'c1:unpack', 'c1:patch',
>> 'a1:package_write_ipk_setscene', 'a1:package_write_rpm_setscene',
>> 'a1:packagedata_setscene',
>> + 'a1:package_qa_setscene', 'a1:build',
>> 'a1:populate_sysroot_setscene']
>> + self.assertEqual(set(tasks), set(expected))
>> +
>> + # This test slows down the execution of do_package_setscene
>> until after other real tasks have
>> + # started running which tests for a bug where tasks were being
>> lost from the buildable list of real
>> + # tasks if they weren't in tasks_covered or tasks_notcovered
>> + def test_slow_setscene(self):
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + cmd = ["bitbake", "a1"]
>> + sstatevalid = "a1:do_package"
>> + slowtasks = "a1:package_setscene"
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
>> slowtasks)
>> + expected = ['a1:package_setscene', 'a1:fetch',
>> 'a1:unpack', 'a1:patch', 'a1:prepare_recipe_sysroot', 'a1:configure',
>> + 'a1:compile', 'a1:install',
>> 'a1:packagedata', 'a1:package_qa', 'a1:package_write_rpm',
>> 'a1:package_write_ipk',
>> + 'a1:populate_sysroot', 'a1:build']
>> + self.assertEqual(set(tasks), set(expected))
>> +
>> + def test_setscenewhitelist(self):
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + cmd = ["bitbake", "a1"]
>> + extraenv = {
>> + "BB_SETSCENE_ENFORCE" : "1",
>> + "BB_SETSCENE_ENFORCE_WHITELIST" :
>> "a1:do_package_write_rpm a1:do_build"
>> + }
>> + sstatevalid = "a1:do_package a1:do_package_qa
>> a1:do_packagedata a1:do_package_write_ipk a1:do_populate_lic
>> a1:do_populate_sysroot"
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
>> extraenv=extraenv)
>> + expected = ['a1:packagedata_setscene',
>> 'a1:package_qa_setscene', 'a1:package_write_ipk_setscene',
>> + 'a1:populate_sysroot_setscene',
>> 'a1:package_setscene']
>> + self.assertEqual(set(tasks), set(expected))
>> +
>> + # Tests for problems with dependencies between setscene tasks
>> + def test_no_setscenevalid_harddeps(self):
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + cmd = ["bitbake", "d1"]
>> + sstatevalid = ""
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
>> + expected = ['a1:package', 'a1:fetch', 'a1:unpack',
>> 'a1:patch', 'a1:prepare_recipe_sysroot', 'a1:configure',
>> + 'a1:compile', 'a1:install',
>> 'a1:packagedata', 'a1:package_write_rpm', 'a1:package_write_ipk',
>> + 'a1:populate_sysroot', 'd1:package',
>> 'd1:fetch', 'd1:unpack', 'd1:patch', 'd1:prepare_recipe_sysroot',
>> 'd1:configure',
>> + 'd1:compile', 'd1:install',
>> 'd1:packagedata', 'd1:package_qa', 'd1:package_write_rpm',
>> 'd1:package_write_ipk',
>> + 'd1:populate_sysroot', 'd1:build']
>> + self.assertEqual(set(tasks), set(expected))
>> +
>> + def test_no_setscenevalid_withdeps(self):
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + cmd = ["bitbake", "b1"]
>> + sstatevalid = ""
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
>> + expected = ['a1:' + x for x in self.alltasks] + ['b1:' +
>> x for x in self.alltasks]
>> + expected.remove('a1:build')
>> + expected.remove('a1:package_qa')
>> + self.assertEqual(set(tasks), set(expected))
>> +
>> + def test_single_a1_setscenevalid_withdeps(self):
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + cmd = ["bitbake", "b1"]
>> + sstatevalid = "a1:do_package"
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
>> + expected = ['a1:package_setscene', 'a1:fetch',
>> 'a1:unpack', 'a1:patch', 'a1:prepare_recipe_sysroot', 'a1:configure',
>> + 'a1:compile', 'a1:install',
>> 'a1:packagedata', 'a1:package_write_rpm', 'a1:package_write_ipk',
>> + 'a1:populate_sysroot'] + ['b1:' + x for x in
>> self.alltasks]
>> + self.assertEqual(set(tasks), set(expected))
>> +
>> + def test_single_b1_setscenevalid_withdeps(self):
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + cmd = ["bitbake", "b1"]
>> + sstatevalid = "b1:do_package"
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
>> + expected = ['a1:package', 'a1:fetch', 'a1:unpack',
>> 'a1:patch', 'a1:prepare_recipe_sysroot', 'a1:configure',
>> + 'a1:compile', 'a1:install',
>> 'a1:packagedata', 'a1:package_write_rpm', 'a1:package_write_ipk',
>> + 'a1:populate_sysroot',
>> 'b1:package_setscene'] + ['b1:' + x for x in self.alltasks]
>> + expected.remove('b1:package')
>> + self.assertEqual(set(tasks), set(expected))
>> +
>> + def test_intermediate_setscenevalid_withdeps(self):
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + cmd = ["bitbake", "b1"]
>> + sstatevalid = "a1:do_package a1:do_populate_sysroot
>> b1:do_package"
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
>> + expected = ['a1:package_setscene', 'a1:packagedata',
>> 'a1:package_write_rpm', 'a1:package_write_ipk',
>> + 'a1:populate_sysroot_setscene',
>> 'b1:package_setscene'] + ['b1:' + x for x in self.alltasks]
>> + expected.remove('b1:package')
>> + self.assertEqual(set(tasks), set(expected))
>> +
>> + def test_all_setscenevalid_withdeps(self):
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + cmd = ["bitbake", "b1"]
>> + sstatevalid = self.a1_sstatevalid + " " +
>> self.b1_sstatevalid
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
>> + expected = ['a1:package_write_ipk_setscene',
>> 'a1:package_write_rpm_setscene', 'a1:packagedata_setscene',
>> + 'b1:build', 'a1:populate_sysroot_setscene',
>> 'b1:package_write_ipk_setscene', 'b1:package_write_rpm_setscene',
>> + 'b1:packagedata_setscene',
>> 'b1:package_qa_setscene', 'b1:populate_sysroot_setscene']
>> + self.assertEqual(set(tasks), set(expected))
>> +
>> + def test_multiconfig_setscene_optimise(self):
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + extraenv = {
>> + "BBMULTICONFIG" : "mc1 mc2",
>> + "BB_SIGNATURE_HANDLER" : "basic"
>> + }
>> + cmd = ["bitbake", "b1", "mc:mc1:b1", "mc:mc2:b1"]
>> + setscenetasks = ['package_write_ipk_setscene',
>> 'package_write_rpm_setscene', 'packagedata_setscene',
>> + 'populate_sysroot_setscene',
>> 'package_qa_setscene']
>> + sstatevalid = ""
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
>> extraenv=extraenv)
>> + expected = ['a1:' + x for x in self.alltasks] + ['b1:' +
>> x for x in self.alltasks] + \
>> + ['mc1:b1:' + x for x in setscenetasks] +
>> ['mc1:a1:' + x for x in setscenetasks] + \
>> + ['mc2:b1:' + x for x in setscenetasks] +
>> ['mc2:a1:' + x for x in setscenetasks] + \
>> + ['mc1:b1:build', 'mc2:b1:build']
>> + for x in ['mc1:a1:package_qa_setscene',
>> 'mc2:a1:package_qa_setscene', 'a1:build', 'a1:package_qa']:
>> + expected.remove(x)
>> + self.assertEqual(set(tasks), set(expected))
>> +
>> +
>> + @unittest.skipIf(sys.version_info < (3, 5, 0), 'Python 3.5 or
>> later required')
>> + def test_hashserv_single(self):
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + extraenv = {
>> + "BB_HASHSERVE" : "auto",
>> + "BB_SIGNATURE_HANDLER" : "TestEquivHash"
>> + }
>> + cmd = ["bitbake", "a1", "b1"]
>> + setscenetasks = ['package_write_ipk_setscene',
>> 'package_write_rpm_setscene', 'packagedata_setscene',
>> + 'populate_sysroot_setscene',
>> 'package_qa_setscene']
>> + sstatevalid = ""
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
>> extraenv=extraenv, cleanup=True)
>> + expected = ['a1:' + x for x in self.alltasks] + ['b1:' +
>> x for x in self.alltasks]
>> + self.assertEqual(set(tasks), set(expected))
>> + cmd = ["bitbake", "a1", "-c", "install", "-f"]
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
>> extraenv=extraenv, cleanup=True)
>> + expected = ['a1:install']
>> + self.assertEqual(set(tasks), set(expected))
>> + cmd = ["bitbake", "a1", "b1"]
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
>> extraenv=extraenv, cleanup=True)
>> + expected = ['a1:populate_sysroot', 'a1:package',
>> 'a1:package_write_rpm_setscene', 'a1:packagedata_setscene',
>> + 'a1:package_write_ipk_setscene',
>> 'a1:package_qa_setscene']
>> + self.assertEqual(set(tasks), set(expected))
>> +
>> + self.shutdown(tempdir)
>> +
>> + @unittest.skipIf(sys.version_info < (3, 5, 0), 'Python 3.5 or
>> later required')
>> + def test_hashserv_double(self):
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + extraenv = {
>> + "BB_HASHSERVE" : "auto",
>> + "BB_SIGNATURE_HANDLER" : "TestEquivHash"
>> + }
>> + cmd = ["bitbake", "a1", "b1", "e1"]
>> + setscenetasks = ['package_write_ipk_setscene',
>> 'package_write_rpm_setscene', 'packagedata_setscene',
>> + 'populate_sysroot_setscene',
>> 'package_qa_setscene']
>> + sstatevalid = ""
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
>> extraenv=extraenv, cleanup=True)
>> + expected = ['a1:' + x for x in self.alltasks] + ['b1:' +
>> x for x in self.alltasks] + ['e1:' + x for x in self.alltasks]
>> + self.assertEqual(set(tasks), set(expected))
>> + cmd = ["bitbake", "a1", "b1", "-c", "install", "-fn"]
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
>> extraenv=extraenv, cleanup=True)
>> + cmd = ["bitbake", "e1"]
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
>> extraenv=extraenv, cleanup=True)
>> + expected = ['a1:package', 'a1:install', 'b1:package',
>> 'b1:install', 'a1:populate_sysroot', 'b1:populate_sysroot',
>> + 'a1:package_write_ipk_setscene',
>> 'b1:packagedata_setscene', 'b1:package_write_rpm_setscene',
>> + 'a1:package_write_rpm_setscene',
>> 'b1:package_write_ipk_setscene', 'a1:packagedata_setscene']
>> + self.assertEqual(set(tasks), set(expected))
>> +
>> + self.shutdown(tempdir)
>> +
>> + @unittest.skipIf(sys.version_info < (3, 5, 0), 'Python 3.5 or
>> later required')
>> + def test_hashserv_multiple_setscene(self):
>> + # Runs e1:do_package_setscene twice
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + extraenv = {
>> + "BB_HASHSERVE" : "auto",
>> + "BB_SIGNATURE_HANDLER" : "TestEquivHash"
>> + }
>> + cmd = ["bitbake", "a1", "b1", "e1"]
>> + setscenetasks = ['package_write_ipk_setscene',
>> 'package_write_rpm_setscene', 'packagedata_setscene',
>> + 'populate_sysroot_setscene',
>> 'package_qa_setscene']
>> + sstatevalid = ""
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
>> extraenv=extraenv, cleanup=True)
>> + expected = ['a1:' + x for x in self.alltasks] + ['b1:' +
>> x for x in self.alltasks] + ['e1:' + x for x in self.alltasks]
>> + self.assertEqual(set(tasks), set(expected))
>> + cmd = ["bitbake", "a1", "b1", "-c", "install", "-fn"]
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
>> extraenv=extraenv, cleanup=True)
>> + cmd = ["bitbake", "e1"]
>> + sstatevalid = "e1:do_package"
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
>> extraenv=extraenv, cleanup=True, slowtasks="a1:populate_sysroot
>> b1:populate_sysroot")
>> + expected = ['a1:package', 'a1:install', 'b1:package',
>> 'b1:install', 'a1:populate_sysroot', 'b1:populate_sysroot',
>> + 'a1:package_write_ipk_setscene',
>> 'b1:packagedata_setscene', 'b1:package_write_rpm_setscene',
>> + 'a1:package_write_rpm_setscene',
>> 'b1:package_write_ipk_setscene', 'a1:packagedata_setscene',
>> + 'e1:package_setscene']
>> + self.assertEqual(set(tasks), set(expected))
>> + for i in expected:
>> + self.assertEqual(tasks.count(i), 1, "%s not in task
>> list once" % i) +
>> + self.shutdown(tempdir)
>> +
>> + def shutdown(self, tempdir):
>> + # Wait for the hashserve socket to disappear else we'll see
>> races with the tempdir cleanup
>> + while os.path.exists(tempdir + "/hashserve.sock"):
>> + time.sleep(0.5)
>> +
>> +
>> diff --git a/bitbake/lib/bb/tests/utils.py
>> b/bitbake/lib/bb/tests/utils.py index 2f4ccf3..f4adf1d 100644
>> --- a/bitbake/lib/bb/tests/utils.py
>> +++ b/bitbake/lib/bb/tests/utils.py
>> @@ -1,22 +1,9 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # BitBake Tests for utils.py
>> #
>> # Copyright (C) 2012 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>>
>> import unittest
>> @@ -42,6 +29,14 @@ class VerCmpString(unittest.TestCase):
>> self.assertTrue(result < 0)
>> result = bb.utils.vercmp_string('1.1', '1.0+1.1-beta1')
>> self.assertTrue(result > 0)
>> + result = bb.utils.vercmp_string('1a', '1a1')
>> + self.assertTrue(result < 0)
>> + result = bb.utils.vercmp_string('1a1', '1a')
>> + self.assertTrue(result > 0)
>> + result = bb.utils.vercmp_string('1.', '1.1')
>> + self.assertTrue(result < 0)
>> + result = bb.utils.vercmp_string('1.1', '1.')
>> + self.assertTrue(result > 0)
>>
>> def test_explode_dep_versions(self):
>> correctresult = {"foo" : ["= 1.10"]}
>> diff --git a/bitbake/lib/bb/tinfoil.py b/bitbake/lib/bb/tinfoil.py
>> index 368264f..0a1b913 100644
>> --- a/bitbake/lib/bb/tinfoil.py
>> +++ b/bitbake/lib/bb/tinfoil.py
>> @@ -4,18 +4,8 @@
>> # Copyright (C) 2011 Mentor Graphics Corporation
>> # Copyright (C) 2006-2012 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import logging
>> import os
>> diff --git a/bitbake/lib/bb/ui/__init__.py
>> b/bitbake/lib/bb/ui/__init__.py index a4805ed..4b7ac36 100644
>> --- a/bitbake/lib/bb/ui/__init__.py
>> +++ b/bitbake/lib/bb/ui/__init__.py
>> @@ -3,15 +3,5 @@
>> #
>> # Copyright (C) 2006-2007 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. diff --git a/bitbake/lib/bb/ui/buildinfohelper.py
>> b/bitbake/lib/bb/ui/buildinfohelper.py index 31323d2..5cbca97 100644
>> --- a/bitbake/lib/bb/ui/buildinfohelper.py
>> +++ b/bitbake/lib/bb/ui/buildinfohelper.py
>> @@ -3,18 +3,8 @@
>> #
>> # Copyright (C) 2013 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import sys
>> import bb
>> @@ -656,6 +646,9 @@ class ORMWrapper(object):
>> Target_Installed_Package.objects.create(target =
>> target_obj, package = packagedict[p]['object'])
>> packagedeps_objs = []
>> + pattern_so = re.compile(r'.*\.so(\.\d*)?$')
>> + pattern_lib = re.compile(r'.*\-suffix(\d*)?$')
>> + pattern_ko = re.compile(r'^kernel-module-.*')
>> for p in packagedict:
>> for (px,deptype) in packagedict[p]['depends']:
>> if deptype == 'depends':
>> @@ -664,6 +657,13 @@ class ORMWrapper(object):
>> tdeptype = Package_Dependency.TYPE_TRECOMMENDS
>>
>> try:
>> + # Skip known non-package objects like libraries
>> and kernel modules
>> + if pattern_so.match(px) or pattern_lib.match(px):
>> + logger.info("Toaster does not add library
>> file dependencies to packages (%s,%s)", p, px)
>> + continue
>> + if pattern_ko.match(px):
>> + logger.info("Toaster does not add kernel
>> module dependencies to packages (%s,%s)", p, px)
>> + continue
>> packagedeps_objs.append(Package_Dependency(
>> package = packagedict[p]['object'],
>> depends_on = packagedict[px]['object'],
>> diff --git a/bitbake/lib/bb/ui/knotty.py b/bitbake/lib/bb/ui/knotty.py
>> index fa88e6c..35736ad 100644
>> --- a/bitbake/lib/bb/ui/knotty.py
>> +++ b/bitbake/lib/bb/ui/knotty.py
>> @@ -5,18 +5,8 @@
>> #
>> # Copyright (C) 2006-2012 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from __future__ import division
>>
>> @@ -222,6 +212,23 @@ class TerminalFilter(object):
>> sys.stdout.flush()
>> self.footer_present = False
>>
>> + def elapsed(self, sec):
>> + hrs = int(sec / 3600.0)
>> + sec -= hrs * 3600
>> + min = int(sec / 60.0)
>> + sec -= min * 60
>> + if hrs > 0:
>> + return "%dh%dm%ds" % (hrs, min, sec)
>> + elif min > 0:
>> + return "%dm%ds" % (min, sec)
>> + else:
>> + return "%ds" % (sec)
>> +
>> + def keepAlive(self, t):
>> + if not self.cuu:
>> + print("Bitbake still alive (%ds)" % t)
>> + sys.stdout.flush()
>> +
>> def updateFooter(self):
>> if not self.cuu:
>> return
>> @@ -258,7 +265,7 @@ class TerminalFilter(object):
>> else:
>> start_time = activetasks[t].get("starttime", None)
>> if start_time:
>> - tasks.append("%s - %ds (pid %s)" %
>> (activetasks[t]["title"], currenttime - start_time, t))
>> + tasks.append("%s - %s (pid %s)" %
>> (activetasks[t]["title"], self.elapsed(currenttime - start_time), t))
>> else: tasks.append("%s (pid %s)" % (activetasks[t]["title"], t))
>>
>> @@ -293,8 +300,8 @@ class TerminalFilter(object):
>> if start_time:
>> pbar.start_time = start_time
>> pbar.setmessage('%s:%s' % (tasknum,
>> pbar.msg.split(':', 1)[1]))
>> + pbar.setextra(rate)
>> if progress > -1:
>> - pbar.setextra(rate)
>> content = pbar.update(progress)
>> else:
>> content = pbar.update(1)
>> @@ -455,11 +462,17 @@ def main(server, eventHandler, params, tf =
>> TerminalFilter): warnings = 0
>> taskfailures = []
>>
>> + printinterval = 5000
>> + lastprint = time.time()
>> +
>> termfilter = tf(main, helper, console, errconsole, format,
>> params.options.quiet) atexit.register(termfilter.finish)
>>
>> while True:
>> try:
>> + if (lastprint + printinterval) <= time.time():
>> + termfilter.keepAlive(printinterval)
>> + printinterval += 5000
>> event = eventHandler.waitEvent(0)
>> if event is None:
>> if main.shutdown > 1:
>> @@ -488,6 +501,8 @@ def main(server, eventHandler, params, tf =
>> TerminalFilter): continue
>>
>> if isinstance(event, logging.LogRecord):
>> + lastprint = time.time()
>> + printinterval = 5000
>> if event.levelno >= format.ERROR:
>> errors = errors + 1
>> return_value = 1
>> @@ -645,7 +660,6 @@ def main(server, eventHandler, params, tf =
>> TerminalFilter): # ignore
>> if isinstance(event, (bb.event.BuildBase,
>> bb.event.MetadataEvent,
>> - bb.event.StampUpdate,
>> bb.event.ConfigParsed,
>> bb.event.MultiConfigParsed,
>> bb.event.RecipeParsed,
>> @@ -675,17 +689,27 @@ def main(server, eventHandler, params, tf =
>> TerminalFilter): if params.observe_only:
>> print("\nKeyboard Interrupt, exiting observer...")
>> main.shutdown = 2
>> - if not params.observe_only and main.shutdown == 1:
>> +
>> + def state_force_shutdown():
>> print("\nSecond Keyboard Interrupt, stopping...\n")
>> _, error = server.runCommand(["stateForceShutdown"])
>> if error:
>> logger.error("Unable to cleanly stop: %s" %
>> error) +
>> + if not params.observe_only and main.shutdown == 1:
>> + state_force_shutdown()
>> +
>> if not params.observe_only and main.shutdown == 0:
>> print("\nKeyboard Interrupt, closing down...\n")
>> interrupted = True
>> - _, error = server.runCommand(["stateShutdown"])
>> - if error:
>> - logger.error("Unable to cleanly shutdown: %s" %
>> error)
>> + # Capture the second KeyboardInterrupt during
>> stateShutdown is running
>> + try:
>> + _, error = server.runCommand(["stateShutdown"])
>> + if error:
>> + logger.error("Unable to cleanly shutdown:
>> %s" % error)
>> + except KeyboardInterrupt:
>> + state_force_shutdown()
>> +
>> main.shutdown = main.shutdown + 1
>> pass
>> except Exception as e:
>> diff --git a/bitbake/lib/bb/ui/ncurses.py
>> b/bitbake/lib/bb/ui/ncurses.py index 8690c52..c422732 100644
>> --- a/bitbake/lib/bb/ui/ncurses.py
>> +++ b/bitbake/lib/bb/ui/ncurses.py
>> @@ -6,18 +6,8 @@
>> # Copyright (C) 2006 Michael 'Mickey' Lauer
>> # Copyright (C) 2006-2007 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> """
>> We have the following windows:
>> diff --git a/bitbake/lib/bb/ui/taskexp.py
>> b/bitbake/lib/bb/ui/taskexp.py index 8305d70..50a943c 100644
>> --- a/bitbake/lib/bb/ui/taskexp.py
>> +++ b/bitbake/lib/bb/ui/taskexp.py
>> @@ -4,18 +4,8 @@
>> # Copyright (C) 2007 Ross Burton
>> # Copyright (C) 2007 - 2008 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import sys
>> import gi
>> diff --git a/bitbake/lib/bb/ui/toasterui.py
>> b/bitbake/lib/bb/ui/toasterui.py index 88cec37..51892c9 100644
>> --- a/bitbake/lib/bb/ui/toasterui.py
>> +++ b/bitbake/lib/bb/ui/toasterui.py
>> @@ -7,18 +7,8 @@
>> # Copyright (C) 2006-2012 Richard Purdie
>> # Copyright (C) 2013 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from __future__ import division
>> import time
>> diff --git a/bitbake/lib/bb/ui/uievent.py
>> b/bitbake/lib/bb/ui/uievent.py index 9542b91..fedb050 100644
>> --- a/bitbake/lib/bb/ui/uievent.py
>> +++ b/bitbake/lib/bb/ui/uievent.py
>> @@ -1,22 +1,9 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # Copyright (C) 2006 - 2007 Michael 'Mickey' Lauer
>> # Copyright (C) 2006 - 2007 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. -
>>
>> """
>> Use this class to fork off a thread to recieve event callbacks from
>> the bitbake diff --git a/bitbake/lib/bb/ui/uihelper.py
>> b/bitbake/lib/bb/ui/uihelper.py index 963c1ea..c8dd7df 100644
>> --- a/bitbake/lib/bb/ui/uihelper.py
>> +++ b/bitbake/lib/bb/ui/uihelper.py
>> @@ -1,21 +1,9 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # Copyright (C) 2006 - 2007 Michael 'Mickey' Lauer
>> # Copyright (C) 2006 - 2007 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import bb.build
>> import time
>> @@ -52,7 +40,7 @@ class BBUIHelper:
>> self.running_pids.remove(event.pid)
>> self.failed_tasks.append( { 'title' : "%s %s" %
>> (event._package, event._task)}) self.needUpdate = True
>> - elif isinstance(event, bb.runqueue.runQueueTaskStarted) or
>> isinstance(event, bb.runqueue.sceneQueueTaskStarted):
>> + elif isinstance(event, bb.runqueue.runQueueTaskStarted):
>> self.tasknumber_current = event.stats.completed +
>> event.stats.active + event.stats.failed + 1 self.tasknumber_total =
>> event.stats.total self.needUpdate = True
>> diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
>> index 13bb5f2..d035949 100644
>> --- a/bitbake/lib/bb/utils.py
>> +++ b/bitbake/lib/bb/utils.py
>> @@ -1,23 +1,11 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake Utility Functions
>> """
>>
>> # Copyright (C) 2004 Michael Lauer
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import re, fcntl, os, string, stat, shutil, time
>> import sys
>> @@ -121,6 +109,10 @@ def vercmp_part(a, b):
>> return -1
>> elif oa > ob:
>> return 1
>> + elif ca is None:
>> + return -1
>> + elif cb is None:
>> + return 1
>> elif ca < cb:
>> return -1
>> elif ca > cb:
>> @@ -402,7 +394,7 @@ def better_exec(code, context, text = None,
>> realfile = "<code>", pythonexception code = better_compile(code,
>> realfile, realfile) try:
>> exec(code, get_context(), context)
>> - except (bb.BBHandledException, bb.parse.SkipRecipe,
>> bb.build.FuncFailed, bb.data_smart.ExpansionError):
>> + except (bb.BBHandledException, bb.parse.SkipRecipe,
>> bb.data_smart.ExpansionError): # Error already shown so passthrough,
>> no need for traceback raise
>> except Exception as e:
>> @@ -685,7 +677,7 @@ def _check_unsafe_delete_path(path):
>> return True
>> return False
>>
>> -def remove(path, recurse=False):
>> +def remove(path, recurse=False, ionice=False):
>> """Equivalent to rm -f or rm -rf"""
>> if not path:
>> return
>> @@ -694,7 +686,10 @@ def remove(path, recurse=False):
>> if _check_unsafe_delete_path(path):
>> raise Exception('bb.utils.remove: called with
>> dangerous path "%s" and recurse=True, refusing to delete!' % path) #
>> shutil.rmtree(name) would be ideal but its too slow
>> - subprocess.check_call(['rm', '-rf'] + glob.glob(path))
>> + cmd = []
>> + if ionice:
>> + cmd = ['ionice', '-c', '3']
>> + subprocess.check_call(cmd + ['rm', '-rf'] + glob.glob(path))
>> return
>> for name in glob.glob(path):
>> try:
>> @@ -703,20 +698,12 @@ def remove(path, recurse=False):
>> if exc.errno != errno.ENOENT:
>> raise
>>
>> -def prunedir(topdir):
>> +def prunedir(topdir, ionice=False):
>> # Delete everything reachable from the directory named in
>> 'topdir'. # CAUTION: This is dangerous!
>> if _check_unsafe_delete_path(topdir):
>> raise Exception('bb.utils.prunedir: called with dangerous
>> path "%s", refusing to delete!' % topdir)
>> - for root, dirs, files in os.walk(topdir, topdown = False):
>> - for name in files:
>> - os.remove(os.path.join(root, name))
>> - for name in dirs:
>> - if os.path.islink(os.path.join(root, name)):
>> - os.remove(os.path.join(root, name))
>> - else:
>> - os.rmdir(os.path.join(root, name))
>> - os.rmdir(topdir)
>> + remove(topdir, recurse=True, ionice=ionice)
>>
>> #
>> # Could also use return re.compile("(%s)" % "|".join(map(re.escape,
>> suffixes))).sub(lambda mo: "", var) @@ -726,8 +713,8 @@ def
>> prune_suffix(var, suffixes, d): # See if var ends with any of the
>> suffixes listed and # remove it if found
>> for suffix in suffixes:
>> - if var.endswith(suffix):
>> - return var.replace(suffix, "")
>> + if suffix and var.endswith(suffix):
>> + return var[:-len(suffix)]
>> return var
>>
>> def mkdirhier(directory):
>> @@ -738,7 +725,7 @@ def mkdirhier(directory):
>> try:
>> os.makedirs(directory)
>> except OSError as e:
>> - if e.errno != errno.EEXIST:
>> + if e.errno != errno.EEXIST or not os.path.isdir(directory):
>> raise e
>>
>> def movefile(src, dest, newmtime = None, sstat = None):
>> @@ -796,7 +783,7 @@ def movefile(src, dest, newmtime = None, sstat =
>> None): os.rename(src, destpath)
>> renamefailed = 0
>> except Exception as e:
>> - if e[0] != errno.EXDEV:
>> + if e.errno != errno.EXDEV:
>> # Some random error.
>> print("movefile: Failed to move", src, "to", dest, e)
>> return None
>> @@ -1505,6 +1492,8 @@ def ioprio_set(who, cls, value):
>> NR_ioprio_set = 251
>> elif _unamearch[0] == "i" and _unamearch[2:3] == "86":
>> NR_ioprio_set = 289
>> + elif _unamearch == "aarch64":
>> + NR_ioprio_set = 30
>>
>> if NR_ioprio_set:
>> ioprio = value | (cls << IOPRIO_CLASS_SHIFT)
>> diff --git a/bitbake/lib/bblayers/__init__.py
>> b/bitbake/lib/bblayers/__init__.py index 3ad9513..4e7c09d 100644
>> --- a/bitbake/lib/bblayers/__init__.py
>> +++ b/bitbake/lib/bblayers/__init__.py
>> @@ -1,2 +1,6 @@
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> from pkgutil import extend_path
>> __path__ = extend_path(__path__, __name__)
>> diff --git a/bitbake/lib/bblayers/action.py
>> b/bitbake/lib/bblayers/action.py index a3f658f..d6459d6 100644
>> --- a/bitbake/lib/bblayers/action.py
>> +++ b/bitbake/lib/bblayers/action.py
>> @@ -1,3 +1,7 @@
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> import fnmatch
>> import logging
>> import os
>> diff --git a/bitbake/lib/bblayers/common.py
>> b/bitbake/lib/bblayers/common.py index 98515ce..c5657d3 100644
>> --- a/bitbake/lib/bblayers/common.py
>> +++ b/bitbake/lib/bblayers/common.py
>> @@ -1,3 +1,7 @@
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> import argparse
>> import logging
>> import os
>> diff --git a/bitbake/lib/bblayers/layerindex.py
>> b/bitbake/lib/bblayers/layerindex.py index 9f02a9d..57cd902 100644
>> --- a/bitbake/lib/bblayers/layerindex.py
>> +++ b/bitbake/lib/bblayers/layerindex.py
>> @@ -1,3 +1,7 @@
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> import layerindexlib
>>
>> import argparse
>> @@ -28,7 +32,7 @@ class LayerIndexPlugin(ActionPlugin):
>> layerdir = os.path.join(repodir, subdir)
>> if not os.path.exists(repodir):
>> if fetch_layer:
>> - result = subprocess.call('git clone %s %s' % (url,
>> repodir), shell = True)
>> + result = subprocess.call(['git', 'clone', url,
>> repodir]) if result:
>> logger.error("Failed to download %s" % url)
>> return None, None, None
>> diff --git a/bitbake/lib/bblayers/query.py
>> b/bitbake/lib/bblayers/query.py index 9294dfa..7db49c8 100644
>> --- a/bitbake/lib/bblayers/query.py
>> +++ b/bitbake/lib/bblayers/query.py
>> @@ -1,3 +1,7 @@
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> import collections
>> import fnmatch
>> import logging
>> @@ -42,7 +46,7 @@ layer, with the preferred version first. Note that
>> skipped recipes that are overlayed will also be listed, with a
>> " (skipped)" suffix. """
>>
>> - items_listed = self.list_recipes('Overlayed recipes', None,
>> True, args.same_version, args.filenames, True, None)
>> + items_listed = self.list_recipes('Overlayed recipes', None,
>> True, args.same_version, args.filenames, False, True, None, False,
>> None) # Check for overlayed .bbclass files
>> classes = collections.defaultdict(list)
>> @@ -108,9 +112,9 @@ skipped recipes will also be listed, with a
>> " (skipped)" suffix. title = 'Matching recipes:'
>> else:
>> title = 'Available recipes:'
>> - self.list_recipes(title, args.pnspec, False, False,
>> args.filenames, args.multiple, inheritlist)
>> + self.list_recipes(title, args.pnspec, False, False,
>> args.filenames, args.recipes_only, args.multiple, args.layer,
>> args.bare, inheritlist)
>> - def list_recipes(self, title, pnspec, show_overlayed_only,
>> show_same_ver_only, show_filenames, show_multi_provider_only,
>> inherits):
>> + def list_recipes(self, title, pnspec, show_overlayed_only,
>> show_same_ver_only, show_filenames, show_recipes_only,
>> show_multi_provider_only, selected_layer, bare, inherits): if
>> inherits: bbpath = str(self.tinfoil.config_data.getVar('BBPATH')) for
>> classname in inherits: @@ -140,24 +144,30 @@ skipped recipes will
>> also be listed, with a " (skipped)" suffix. preferred_versions[p] =
>> (ver, fn)
>> def print_item(f, pn, ver, layer, ispref):
>> - if f in skiplist:
>> - skipped = ' (skipped)'
>> - else:
>> - skipped = ''
>> - if show_filenames:
>> - if ispref:
>> - logger.plain("%s%s", f, skipped)
>> + if not selected_layer or layer == selected_layer:
>> + if not bare and f in skiplist:
>> + skipped = ' (skipped)'
>> else:
>> - logger.plain(" %s%s", f, skipped)
>> - else:
>> - if ispref:
>> - logger.plain("%s:", pn)
>> - logger.plain(" %s %s%s", layer.ljust(20), ver,
>> skipped)
>> + skipped = ''
>> + if show_filenames:
>> + if ispref:
>> + logger.plain("%s%s", f, skipped)
>> + else:
>> + logger.plain(" %s%s", f, skipped)
>> + elif show_recipes_only:
>> + if pn not in show_unique_pn:
>> + show_unique_pn.append(pn)
>> + logger.plain("%s%s", pn, skipped)
>> + else:
>> + if ispref:
>> + logger.plain("%s:", pn)
>> + logger.plain(" %s %s%s", layer.ljust(20), ver,
>> skipped)
>> global_inherit = (self.tinfoil.config_data.getVar('INHERIT')
>> or "").split() cls_re = re.compile('classes/')
>>
>> preffiles = []
>> + show_unique_pn = []
>> items_listed = False
>> for p in sorted(pkg_pn):
>> if pnspec:
>> @@ -489,8 +499,11 @@ NOTE: .bbappend files can impact the
>> dependencies.
>> parser_show_recipes = self.add_command(sp, 'show-recipes',
>> self.do_show_recipes) parser_show_recipes.add_argument('-f',
>> '--filenames', help='instead of the default formatting, list
>> filenames of higher priority recipes with the ones they overlay
>> indented underneath', action='store_true')
>> + parser_show_recipes.add_argument('-r', '--recipes-only',
>> help='instead of the default formatting, list recipes only',
>> action='store_true') parser_show_recipes.add_argument('-m',
>> '--multiple', help='only list where multiple recipes (in the same
>> layer or different layers) exist for the same recipe name',
>> action='store_true') parser_show_recipes.add_argument('-i',
>> '--inherits', help='only list recipes that inherit the named
>> class(es) - separate multiple classes using , (without spaces)',
>> metavar='CLASS', default='')
>> + parser_show_recipes.add_argument('-l', '--layer', help='only
>> list recipes from the selected layer', default='')
>> + parser_show_recipes.add_argument('-b', '--bare',
>> help='output just names without the "(skipped)" marker',
>> action='store_true') parser_show_recipes.add_argument('pnspec',
>> nargs='*', help='optional recipe name specification (wildcards
>> allowed, enclose in quotes to avoid shell expansion)')
>> parser_show_appends = self.add_command(sp, 'show-appends',
>> self.do_show_appends) diff --git a/bitbake/lib/bs4/dammit.py
>> b/bitbake/lib/bs4/dammit.py index 68d419f..805aa90 100644 ---
>> a/bitbake/lib/bs4/dammit.py +++ b/bitbake/lib/bs4/dammit.py @@ -45,9
>> +45,9 @@ except ImportError: pass
>>
>> xml_encoding_re = re.compile(
>> - '^<\?.*encoding=[\'"](.*?)[\'"].*\?>'.encode(), re.I)
>> + r'^<\?.*encoding=[\'"](.*?)[\'"].*\?>'.encode(), re.I)
>> html_meta_re = re.compile(
>> - '<\s*meta[^>]+charset\s*=\s*["\']?([^>]*?)[ /;\'">]'.encode(),
>> re.I)
>> + r'<\s*meta[^>]+charset\s*=\s*["\']?([^>]*?)[ /;\'">]'.encode(),
>> re.I)
>> class EntitySubstitution(object):
>>
>> @@ -80,11 +80,11 @@ class EntitySubstitution(object):
>> ">": "gt",
>> }
>>
>> - BARE_AMPERSAND_OR_BRACKET = re.compile("([<>]|"
>> -
>> "&(?!#\d+;|#x[0-9a-fA-F]+;|\w+;)"
>> - ")")
>> + BARE_AMPERSAND_OR_BRACKET = re.compile(r"([<>]|"
>> +
>> r"&(?!#\d+;|#x[0-9a-fA-F]+;|\w+;)"
>> + r")")
>>
>> - AMPERSAND_OR_BRACKET = re.compile("([<>&])")
>> + AMPERSAND_OR_BRACKET = re.compile(r"([<>&])")
>>
>> @classmethod
>> def _substitute_html_entity(cls, matchobj):
>> diff --git a/bitbake/lib/bs4/element.py b/bitbake/lib/bs4/element.py
>> index 0e62c2e..3775a60 100644
>> --- a/bitbake/lib/bs4/element.py
>> +++ b/bitbake/lib/bs4/element.py
>> @@ -1,7 +1,7 @@
>> __license__ = "MIT"
>>
>> from pdb import set_trace
>> -import collections
>> +import collections.abc
>> import re
>> import sys
>> import warnings
>> @@ -10,7 +10,7 @@ from bs4.dammit import EntitySubstitution
>> DEFAULT_OUTPUT_ENCODING = "utf-8"
>> PY3K = (sys.version_info[0] > 2)
>>
>> -whitespace_re = re.compile("\s+")
>> +whitespace_re = re.compile(r"\s+")
>>
>> def _alias(attr):
>> """Alias one attribute name to another for backward
>> compatibility""" @@ -67,7 +67,7 @@ class
>> ContentMetaAttributeValue(AttributeValueWithCharsetSubstitution): The
>> value of the 'content' attribute will be one of these objects. """
>>
>> - CHARSET_RE = re.compile("((^|;)\s*charset=)([^;]*)", re.M)
>> + CHARSET_RE = re.compile(r"((^|;)\s*charset=)([^;]*)", re.M)
>>
>> def __new__(cls, original_value):
>> match = cls.CHARSET_RE.search(original_value)
>> @@ -155,7 +155,7 @@ class PageElement(object):
>>
>> def format_string(self, s, formatter='minimal'):
>> """Format the given string using the given formatter."""
>> - if not isinstance(formatter, collections.Callable):
>> + if not isinstance(formatter, collections.abc.Callable):
>> formatter = self._formatter_for_name(formatter)
>> if formatter is None:
>> output = s
>> @@ -580,7 +580,7 @@ class PageElement(object):
>>
>> # Methods for supporting CSS selectors.
>>
>> - tag_name_re = re.compile('^[a-zA-Z0-9][-.a-zA-Z0-9:_]*$')
>> + tag_name_re = re.compile(r'^[a-zA-Z0-9][-.a-zA-Z0-9:_]*$')
>>
>> # /^([a-zA-Z0-9][-.a-zA-Z0-9:_]*)\[(\w+)([=~\|\^\$\*]?)=?"?([^\]"]*)"?\]$/
>> # \---------------------------/ \---/\-------------/
>> \-------/ @@ -1077,7 +1077,7 @@ class Tag(PageElement):
>>
>> # First off, turn a string formatter into a function. This
>> # will stop the lookup from happening over and over again.
>> - if not isinstance(formatter, collections.Callable):
>> + if not isinstance(formatter, collections.abc.Callable):
>> formatter = self._formatter_for_name(formatter)
>>
>> attrs = []
>> @@ -1181,7 +1181,7 @@ class Tag(PageElement):
>> """
>> # First off, turn a string formatter into a function. This
>> # will stop the lookup from happening over and over again.
>> - if not isinstance(formatter, collections.Callable):
>> + if not isinstance(formatter, collections.abc.Callable):
>> formatter = self._formatter_for_name(formatter)
>>
>> pretty_print = (indent_level is not None)
>> @@ -1364,7 +1364,7 @@ class Tag(PageElement):
>> if tag_name == '':
>> raise ValueError(
>> "A pseudo-class must be prefixed with a tag
>> name.")
>> - pseudo_attributes =
>> re.match('([a-zA-Z\d-]+)\(([a-zA-Z\d]+)\)', pseudo)
>> + pseudo_attributes =
>> re.match(r'([a-zA-Z\d-]+)\(([a-zA-Z\d]+)\)', pseudo) found = []
>> if pseudo_attributes is None:
>> pseudo_type = pseudo
>> @@ -1562,7 +1562,7 @@ class SoupStrainer(object):
>> def _normalize_search_value(self, value):
>> # Leave it alone if it's a Unicode string, a callable, a
>> # regular expression, a boolean, or None.
>> - if (isinstance(value, str) or isinstance(value,
>> collections.Callable) or hasattr(value, 'match')
>> + if (isinstance(value, str) or isinstance(value,
>> collections.abc.Callable) or hasattr(value, 'match') or
>> isinstance(value, bool) or value is None): return value
>>
>> @@ -1602,7 +1602,7 @@ class SoupStrainer(object):
>> markup = markup_name
>> markup_attrs = markup
>> call_function_with_tag_data = (
>> - isinstance(self.name, collections.Callable)
>> + isinstance(self.name, collections.abc.Callable)
>> and not isinstance(markup_name, Tag))
>>
>> if ((not self.name)
>> @@ -1688,7 +1688,7 @@ class SoupStrainer(object):
>> # True matches any non-None value.
>> return markup is not None
>>
>> - if isinstance(match_against, collections.Callable):
>> + if isinstance(match_against, collections.abc.Callable):
>> return match_against(markup)
>>
>> # Custom callables take the tag as an argument, but all
>> diff --git a/bitbake/lib/hashserv/__init__.py
>> b/bitbake/lib/hashserv/__init__.py new file mode 100644
>> index 0000000..c331862
>> --- /dev/null
>> +++ b/bitbake/lib/hashserv/__init__.py
>> @@ -0,0 +1,93 @@
>> +# Copyright (C) 2018-2019 Garmin Ltd.
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> +from contextlib import closing
>> +import re
>> +import sqlite3
>> +
>> +UNIX_PREFIX = "unix://"
>> +
>> +ADDR_TYPE_UNIX = 0
>> +ADDR_TYPE_TCP = 1
>> +
>> +
>> +def setup_database(database, sync=True):
>> + db = sqlite3.connect(database)
>> + db.row_factory = sqlite3.Row
>> +
>> + with closing(db.cursor()) as cursor:
>> + cursor.execute('''
>> + CREATE TABLE IF NOT EXISTS tasks_v2 (
>> + id INTEGER PRIMARY KEY AUTOINCREMENT,
>> + method TEXT NOT NULL,
>> + outhash TEXT NOT NULL,
>> + taskhash TEXT NOT NULL,
>> + unihash TEXT NOT NULL,
>> + created DATETIME,
>> +
>> + -- Optional fields
>> + owner TEXT,
>> + PN TEXT,
>> + PV TEXT,
>> + PR TEXT,
>> + task TEXT,
>> + outhash_siginfo TEXT,
>> +
>> + UNIQUE(method, outhash, taskhash)
>> + )
>> + ''')
>> + cursor.execute('PRAGMA journal_mode = WAL')
>> + cursor.execute('PRAGMA synchronous = %s' % ('NORMAL' if sync
>> else 'OFF')) +
>> + # Drop old indexes
>> + cursor.execute('DROP INDEX IF EXISTS taskhash_lookup')
>> + cursor.execute('DROP INDEX IF EXISTS outhash_lookup')
>> +
>> + # Create new indexes
>> + cursor.execute('CREATE INDEX IF NOT EXISTS
>> taskhash_lookup_v2 ON tasks_v2 (method, taskhash, created)')
>> + cursor.execute('CREATE INDEX IF NOT EXISTS outhash_lookup_v2
>> ON tasks_v2 (method, outhash)') +
>> + return db
>> +
>> +
>> +def parse_address(addr):
>> + if addr.startswith(UNIX_PREFIX):
>> + return (ADDR_TYPE_UNIX, (addr[len(UNIX_PREFIX):],))
>> + else:
>> + m = re.match(r'\[(?P<host>[^\]]*)\]:(?P<port>\d+)$', addr)
>> + if m is not None:
>> + host = m.group('host')
>> + port = m.group('port')
>> + else:
>> + host, port = addr.split(':')
>> +
>> + return (ADDR_TYPE_TCP, (host, int(port)))
>> +
>> +
>> +def create_server(addr, dbname, *, sync=True):
>> + from . import server
>> + db = setup_database(dbname, sync=sync)
>> + s = server.Server(db)
>> +
>> + (typ, a) = parse_address(addr)
>> + if typ == ADDR_TYPE_UNIX:
>> + s.start_unix_server(*a)
>> + else:
>> + s.start_tcp_server(*a)
>> +
>> + return s
>> +
>> +
>> +def create_client(addr):
>> + from . import client
>> + c = client.Client()
>> +
>> + (typ, a) = parse_address(addr)
>> + if typ == ADDR_TYPE_UNIX:
>> + c.connect_unix(*a)
>> + else:
>> + c.connect_tcp(*a)
>> +
>> + return c
>> diff --git a/bitbake/lib/hashserv/client.py
>> b/bitbake/lib/hashserv/client.py new file mode 100644
>> index 0000000..f659566
>> --- /dev/null
>> +++ b/bitbake/lib/hashserv/client.py
>> @@ -0,0 +1,157 @@
>> +# Copyright (C) 2019 Garmin Ltd.
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> +from contextlib import closing
>> +import json
>> +import logging
>> +import socket
>> +import os
>> +
>> +
>> +logger = logging.getLogger('hashserv.client')
>> +
>> +
>> +class HashConnectionError(Exception):
>> + pass
>> +
>> +
>> +class Client(object):
>> + MODE_NORMAL = 0
>> + MODE_GET_STREAM = 1
>> +
>> + def __init__(self):
>> + self._socket = None
>> + self.reader = None
>> + self.writer = None
>> + self.mode = self.MODE_NORMAL
>> +
>> + def connect_tcp(self, address, port):
>> + def connect_sock():
>> + s = socket.create_connection((address, port))
>> +
>> + s.setsockopt(socket.SOL_TCP, socket.TCP_NODELAY, 1)
>> + s.setsockopt(socket.SOL_TCP, socket.TCP_QUICKACK, 1)
>> + s.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
>> + return s
>> +
>> + self._connect_sock = connect_sock
>> +
>> + def connect_unix(self, path):
>> + def connect_sock():
>> + s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
>> + # AF_UNIX has path length issues so chdir here to
>> workaround
>> + cwd = os.getcwd()
>> + try:
>> + os.chdir(os.path.dirname(path))
>> + s.connect(os.path.basename(path))
>> + finally:
>> + os.chdir(cwd)
>> + return s
>> +
>> + self._connect_sock = connect_sock
>> +
>> + def connect(self):
>> + if self._socket is None:
>> + self._socket = self._connect_sock()
>> +
>> + self.reader = self._socket.makefile('r',
>> encoding='utf-8')
>> + self.writer = self._socket.makefile('w',
>> encoding='utf-8') +
>> + self.writer.write('OEHASHEQUIV 1.0\n\n')
>> + self.writer.flush()
>> +
>> + # Restore mode if the socket is being re-created
>> + cur_mode = self.mode
>> + self.mode = self.MODE_NORMAL
>> + self._set_mode(cur_mode)
>> +
>> + return self._socket
>> +
>> + def close(self):
>> + if self._socket is not None:
>> + self._socket.close()
>> + self._socket = None
>> + self.reader = None
>> + self.writer = None
>> +
>> + def _send_wrapper(self, proc):
>> + count = 0
>> + while True:
>> + try:
>> + self.connect()
>> + return proc()
>> + except (OSError, HashConnectionError,
>> json.JSONDecodeError, UnicodeDecodeError) as e:
>> + logger.warning('Error talking to server: %s' % e)
>> + if count >= 3:
>> + if not isinstance(e, HashConnectionError):
>> + raise HashConnectionError(str(e))
>> + raise e
>> + self.close()
>> + count += 1
>> +
>> + def send_message(self, msg):
>> + def proc():
>> + self.writer.write('%s\n' % json.dumps(msg))
>> + self.writer.flush()
>> +
>> + l = self.reader.readline()
>> + if not l:
>> + raise HashConnectionError('Connection closed')
>> +
>> + if not l.endswith('\n'):
>> + raise HashConnectionError('Bad message %r' % message)
>> +
>> + return json.loads(l)
>> +
>> + return self._send_wrapper(proc)
>> +
>> + def send_stream(self, msg):
>> + def proc():
>> + self.writer.write("%s\n" % msg)
>> + self.writer.flush()
>> + l = self.reader.readline()
>> + if not l:
>> + raise HashConnectionError('Connection closed')
>> + return l.rstrip()
>> +
>> + return self._send_wrapper(proc)
>> +
>> + def _set_mode(self, new_mode):
>> + if new_mode == self.MODE_NORMAL and self.mode ==
>> self.MODE_GET_STREAM:
>> + r = self.send_stream('END')
>> + if r != 'ok':
>> + raise HashConnectionError('Bad response from server
>> %r' % r)
>> + elif new_mode == self.MODE_GET_STREAM and self.mode ==
>> self.MODE_NORMAL:
>> + r = self.send_message({'get-stream': None})
>> + if r != 'ok':
>> + raise HashConnectionError('Bad response from server
>> %r' % r)
>> + elif new_mode != self.mode:
>> + raise Exception('Undefined mode transition %r -> %r' %
>> (self.mode, new_mode)) +
>> + self.mode = new_mode
>> +
>> + def get_unihash(self, method, taskhash):
>> + self._set_mode(self.MODE_GET_STREAM)
>> + r = self.send_stream('%s %s' % (method, taskhash))
>> + if not r:
>> + return None
>> + return r
>> +
>> + def report_unihash(self, taskhash, method, outhash, unihash,
>> extra={}):
>> + self._set_mode(self.MODE_NORMAL)
>> + m = extra.copy()
>> + m['taskhash'] = taskhash
>> + m['method'] = method
>> + m['outhash'] = outhash
>> + m['unihash'] = unihash
>> + return self.send_message({'report': m})
>> +
>> + def get_stats(self):
>> + self._set_mode(self.MODE_NORMAL)
>> + return self.send_message({'get-stats': None})
>> +
>> + def reset_stats(self):
>> + self._set_mode(self.MODE_NORMAL)
>> + return self.send_message({'reset-stats': None})
>> diff --git a/bitbake/lib/hashserv/server.py
>> b/bitbake/lib/hashserv/server.py new file mode 100644
>> index 0000000..0aff776
>> --- /dev/null
>> +++ b/bitbake/lib/hashserv/server.py
>> @@ -0,0 +1,414 @@
>> +# Copyright (C) 2019 Garmin Ltd.
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> +from contextlib import closing
>> +from datetime import datetime
>> +import asyncio
>> +import json
>> +import logging
>> +import math
>> +import os
>> +import signal
>> +import socket
>> +import time
>> +
>> +logger = logging.getLogger('hashserv.server')
>> +
>> +
>> +class Measurement(object):
>> + def __init__(self, sample):
>> + self.sample = sample
>> +
>> + def start(self):
>> + self.start_time = time.perf_counter()
>> +
>> + def end(self):
>> + self.sample.add(time.perf_counter() - self.start_time)
>> +
>> + def __enter__(self):
>> + self.start()
>> + return self
>> +
>> + def __exit__(self, *args, **kwargs):
>> + self.end()
>> +
>> +
>> +class Sample(object):
>> + def __init__(self, stats):
>> + self.stats = stats
>> + self.num_samples = 0
>> + self.elapsed = 0
>> +
>> + def measure(self):
>> + return Measurement(self)
>> +
>> + def __enter__(self):
>> + return self
>> +
>> + def __exit__(self, *args, **kwargs):
>> + self.end()
>> +
>> + def add(self, elapsed):
>> + self.num_samples += 1
>> + self.elapsed += elapsed
>> +
>> + def end(self):
>> + if self.num_samples:
>> + self.stats.add(self.elapsed)
>> + self.num_samples = 0
>> + self.elapsed = 0
>> +
>> +
>> +class Stats(object):
>> + def __init__(self):
>> + self.reset()
>> +
>> + def reset(self):
>> + self.num = 0
>> + self.total_time = 0
>> + self.max_time = 0
>> + self.m = 0
>> + self.s = 0
>> + self.current_elapsed = None
>> +
>> + def add(self, elapsed):
>> + self.num += 1
>> + if self.num == 1:
>> + self.m = elapsed
>> + self.s = 0
>> + else:
>> + last_m = self.m
>> + self.m = last_m + (elapsed - last_m) / self.num
>> + self.s = self.s + (elapsed - last_m) * (elapsed - self.m)
>> +
>> + self.total_time += elapsed
>> +
>> + if self.max_time < elapsed:
>> + self.max_time = elapsed
>> +
>> + def start_sample(self):
>> + return Sample(self)
>> +
>> + @property
>> + def average(self):
>> + if self.num == 0:
>> + return 0
>> + return self.total_time / self.num
>> +
>> + @property
>> + def stdev(self):
>> + if self.num <= 1:
>> + return 0
>> + return math.sqrt(self.s / (self.num - 1))
>> +
>> + def todict(self):
>> + return {k: getattr(self, k) for k in ('num', 'total_time',
>> 'max_time', 'average', 'stdev')} +
>> +
>> +class ServerClient(object):
>> + def __init__(self, reader, writer, db, request_stats):
>> + self.reader = reader
>> + self.writer = writer
>> + self.db = db
>> + self.request_stats = request_stats
>> +
>> + async def process_requests(self):
>> + try:
>> + self.addr = self.writer.get_extra_info('peername')
>> + logger.debug('Client %r connected' % (self.addr,))
>> +
>> + # Read protocol and version
>> + protocol = await self.reader.readline()
>> + if protocol is None:
>> + return
>> +
>> + (proto_name, proto_version) =
>> protocol.decode('utf-8').rstrip().split()
>> + if proto_name != 'OEHASHEQUIV' or proto_version != '1.0':
>> + return
>> +
>> + # Read headers. Currently, no headers are implemented,
>> so look for
>> + # an empty line to signal the end of the headers
>> + while True:
>> + line = await self.reader.readline()
>> + if line is None:
>> + return
>> +
>> + line = line.decode('utf-8').rstrip()
>> + if not line:
>> + break
>> +
>> + # Handle messages
>> + handlers = {
>> + 'get': self.handle_get,
>> + 'report': self.handle_report,
>> + 'get-stream': self.handle_get_stream,
>> + 'get-stats': self.handle_get_stats,
>> + 'reset-stats': self.handle_reset_stats,
>> + }
>> +
>> + while True:
>> + d = await self.read_message()
>> + if d is None:
>> + break
>> +
>> + for k in handlers.keys():
>> + if k in d:
>> + logger.debug('Handling %s' % k)
>> + if 'stream' in k:
>> + await handlers[k](d[k])
>> + else:
>> + with self.request_stats.start_sample()
>> as self.request_sample, \
>> + self.request_sample.measure():
>> + await handlers[k](d[k])
>> + break
>> + else:
>> + logger.warning("Unrecognized command %r" % d)
>> + break
>> +
>> + await self.writer.drain()
>> + finally:
>> + self.writer.close()
>> +
>> + def write_message(self, msg):
>> + self.writer.write(('%s\n' % json.dumps(msg)).encode('utf-8'))
>> +
>> + async def read_message(self):
>> + l = await self.reader.readline()
>> + if not l:
>> + return None
>> +
>> + try:
>> + message = l.decode('utf-8')
>> +
>> + if not message.endswith('\n'):
>> + return None
>> +
>> + return json.loads(message)
>> + except (json.JSONDecodeError, UnicodeDecodeError) as e:
>> + logger.error('Bad message from client: %r' % message)
>> + raise e
>> +
>> + async def handle_get(self, request):
>> + method = request['method']
>> + taskhash = request['taskhash']
>> +
>> + row = self.query_equivalent(method, taskhash)
>> + if row is not None:
>> + logger.debug('Found equivalent task %s -> %s',
>> (row['taskhash'], row['unihash']))
>> + d = {k: row[k] for k in ('taskhash', 'method',
>> 'unihash')} +
>> + self.write_message(d)
>> + else:
>> + self.write_message(None)
>> +
>> + async def handle_get_stream(self, request):
>> + self.write_message('ok')
>> +
>> + while True:
>> + l = await self.reader.readline()
>> + if not l:
>> + return
>> +
>> + try:
>> + # This inner loop is very sensitive and must be as
>> fast as
>> + # possible (which is why the request sample is
>> handled manually
>> + # instead of using 'with', and also why logging
>> statements are
>> + # commented out.
>> + self.request_sample =
>> self.request_stats.start_sample()
>> + request_measure = self.request_sample.measure()
>> + request_measure.start()
>> +
>> + l = l.decode('utf-8').rstrip()
>> + if l == 'END':
>> + self.writer.write('ok\n'.encode('utf-8'))
>> + return
>> +
>> + (method, taskhash) = l.split()
>> + #logger.debug('Looking up %s %s' % (method,
>> taskhash))
>> + row = self.query_equivalent(method, taskhash)
>> + if row is not None:
>> + msg = ('%s\n' % row['unihash']).encode('utf-8')
>> + #logger.debug('Found equivalent task %s -> %s',
>> (row['taskhash'], row['unihash']))
>> + else:
>> + msg = '\n'.encode('utf-8')
>> +
>> + self.writer.write(msg)
>> + finally:
>> + request_measure.end()
>> + self.request_sample.end()
>> +
>> + await self.writer.drain()
>> +
>> + async def handle_report(self, data):
>> + with closing(self.db.cursor()) as cursor:
>> + cursor.execute('''
>> + -- Find tasks with a matching outhash (that is,
>> tasks that
>> + -- are equivalent)
>> + SELECT taskhash, method, unihash FROM tasks_v2 WHERE
>> method=:method AND outhash=:outhash +
>> + -- If there is an exact match on the taskhash,
>> return it.
>> + -- Otherwise return the oldest matching outhash of
>> any
>> + -- taskhash
>> + ORDER BY CASE WHEN taskhash=:taskhash THEN 1 ELSE 2
>> END,
>> + created ASC
>> +
>> + -- Only return one row
>> + LIMIT 1
>> + ''', {k: data[k] for k in ('method', 'outhash',
>> 'taskhash')}) +
>> + row = cursor.fetchone()
>> +
>> + # If no matching outhash was found, or one *was* found
>> but it
>> + # wasn't an exact match on the taskhash, a new entry for
>> this
>> + # taskhash should be added
>> + if row is None or row['taskhash'] != data['taskhash']:
>> + # If a row matching the outhash was found, the
>> unihash for
>> + # the new taskhash should be the same as that one.
>> + # Otherwise the caller provided unihash is used.
>> + unihash = data['unihash']
>> + if row is not None:
>> + unihash = row['unihash']
>> +
>> + insert_data = {
>> + 'method': data['method'],
>> + 'outhash': data['outhash'],
>> + 'taskhash': data['taskhash'],
>> + 'unihash': unihash,
>> + 'created': datetime.now()
>> + }
>> +
>> + for k in ('owner', 'PN', 'PV', 'PR', 'task',
>> 'outhash_siginfo'):
>> + if k in data:
>> + insert_data[k] = data[k]
>> +
>> + cursor.execute('''INSERT INTO tasks_v2 (%s) VALUES
>> (%s)''' % (
>> + ', '.join(sorted(insert_data.keys())),
>> + ', '.join(':' + k for k in
>> sorted(insert_data.keys()))),
>> + insert_data)
>> +
>> + self.db.commit()
>> +
>> + logger.info('Adding taskhash %s with unihash %s',
>> + data['taskhash'], unihash)
>> +
>> + d = {
>> + 'taskhash': data['taskhash'],
>> + 'method': data['method'],
>> + 'unihash': unihash
>> + }
>> + else:
>> + d = {k: row[k] for k in ('taskhash', 'method',
>> 'unihash')} +
>> + self.write_message(d)
>> +
>> + async def handle_get_stats(self, request):
>> + d = {
>> + 'requests': self.request_stats.todict(),
>> + }
>> +
>> + self.write_message(d)
>> +
>> + async def handle_reset_stats(self, request):
>> + d = {
>> + 'requests': self.request_stats.todict(),
>> + }
>> +
>> + self.request_stats.reset()
>> + self.write_message(d)
>> +
>> + def query_equivalent(self, method, taskhash):
>> + # This is part of the inner loop and must be as fast as
>> possible
>> + try:
>> + cursor = self.db.cursor()
>> + cursor.execute('SELECT taskhash, method, unihash FROM
>> tasks_v2 WHERE method=:method AND taskhash=:taskhash ORDER BY created
>> ASC LIMIT 1',
>> + {'method': method, 'taskhash': taskhash})
>> + return cursor.fetchone()
>> + except:
>> + cursor.close()
>> +
>> +
>> +class Server(object):
>> + def __init__(self, db, loop=None):
>> + self.request_stats = Stats()
>> + self.db = db
>> +
>> + if loop is None:
>> + self.loop = asyncio.new_event_loop()
>> + self.close_loop = True
>> + else:
>> + self.loop = loop
>> + self.close_loop = False
>> +
>> + self._cleanup_socket = None
>> +
>> + def start_tcp_server(self, host, port):
>> + self.server = self.loop.run_until_complete(
>> + asyncio.start_server(self.handle_client, host, port,
>> loop=self.loop)
>> + )
>> +
>> + for s in self.server.sockets:
>> + logger.info('Listening on %r' % (s.getsockname(),))
>> + # Newer python does this automatically. Do it manually
>> here for
>> + # maximum compatibility
>> + s.setsockopt(socket.SOL_TCP, socket.TCP_NODELAY, 1)
>> + s.setsockopt(socket.SOL_TCP, socket.TCP_QUICKACK, 1)
>> +
>> + name = self.server.sockets[0].getsockname()
>> + if self.server.sockets[0].family == socket.AF_INET6:
>> + self.address = "[%s]:%d" % (name[0], name[1])
>> + else:
>> + self.address = "%s:%d" % (name[0], name[1])
>> +
>> + def start_unix_server(self, path):
>> + def cleanup():
>> + os.unlink(path)
>> +
>> + cwd = os.getcwd()
>> + try:
>> + # Work around path length limits in AF_UNIX
>> + os.chdir(os.path.dirname(path))
>> + self.server = self.loop.run_until_complete(
>> + asyncio.start_unix_server(self.handle_client,
>> os.path.basename(path), loop=self.loop)
>> + )
>> + finally:
>> + os.chdir(cwd)
>> +
>> + logger.info('Listening on %r' % path)
>> +
>> + self._cleanup_socket = cleanup
>> + self.address = "unix://%s" % os.path.abspath(path)
>> +
>> + async def handle_client(self, reader, writer):
>> + # writer.transport.set_write_buffer_limits(0)
>> + try:
>> + client = ServerClient(reader, writer, self.db,
>> self.request_stats)
>> + await client.process_requests()
>> + except Exception as e:
>> + import traceback
>> + logger.error('Error from client: %s' % str(e),
>> exc_info=True)
>> + traceback.print_exc()
>> + writer.close()
>> + logger.info('Client disconnected')
>> +
>> + def serve_forever(self):
>> + def signal_handler():
>> + self.loop.stop()
>> +
>> + self.loop.add_signal_handler(signal.SIGTERM, signal_handler)
>> +
>> + try:
>> + self.loop.run_forever()
>> + except KeyboardInterrupt:
>> + pass
>> +
>> + self.server.close()
>> + self.loop.run_until_complete(self.server.wait_closed())
>> + logger.info('Server shutting down')
>> +
>> + if self.close_loop:
>> + self.loop.close()
>> +
>> + if self._cleanup_socket is not None:
>> + self._cleanup_socket()
>> diff --git a/bitbake/lib/hashserv/tests.py
>> b/bitbake/lib/hashserv/tests.py new file mode 100644
>> index 0000000..a5472a9
>> --- /dev/null
>> +++ b/bitbake/lib/hashserv/tests.py
>> @@ -0,0 +1,142 @@
>> +#! /usr/bin/env python3
>> +#
>> +# Copyright (C) 2018-2019 Garmin Ltd.
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> +from . import create_server, create_client
>> +import hashlib
>> +import logging
>> +import multiprocessing
>> +import sys
>> +import tempfile
>> +import threading
>> +import unittest
>> +
>> +
>> +class TestHashEquivalenceServer(object):
>> + METHOD = 'TestMethod'
>> +
>> + def _run_server(self):
>> + # logging.basicConfig(level=logging.DEBUG,
>> filename='bbhashserv.log', filemode='w',
>> + # format='%(levelname)s
>> %(filename)s:%(lineno)d %(message)s')
>> + self.server.serve_forever()
>> +
>> + def setUp(self):
>> + if sys.version_info < (3, 5, 0):
>> + self.skipTest('Python 3.5 or later required')
>> +
>> + self.temp_dir =
>> tempfile.TemporaryDirectory(prefix='bb-hashserv')
>> + self.dbfile = os.path.join(self.temp_dir.name, 'db.sqlite')
>> +
>> + self.server = create_server(self.get_server_addr(),
>> self.dbfile)
>> + self.server_thread =
>> multiprocessing.Process(target=self._run_server)
>> + self.server_thread.start()
>> + self.client = create_client(self.server.address)
>> +
>> + def tearDown(self):
>> + # Shutdown server
>> + s = getattr(self, 'server', None)
>> + if s is not None:
>> + self.server_thread.terminate()
>> + self.server_thread.join()
>> + self.client.close()
>> + self.temp_dir.cleanup()
>> +
>> + def test_create_hash(self):
>> + # Simple test that hashes can be created
>> + taskhash = '35788efcb8dfb0a02659d81cf2bfd695fb30faf9'
>> + outhash =
>> '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5970796f'
>> + unihash = 'f46d3fbb439bd9b921095da657a4de906510d2cd'
>> +
>> + result = self.client.get_unihash(self.METHOD, taskhash)
>> + self.assertIsNone(result, msg='Found unexpected task, %r' %
>> result) +
>> + result = self.client.report_unihash(taskhash, self.METHOD,
>> outhash, unihash)
>> + self.assertEqual(result['unihash'], unihash, 'Server
>> returned bad unihash') +
>> + def test_create_equivalent(self):
>> + # Tests that a second reported task with the same outhash
>> will be
>> + # assigned the same unihash
>> + taskhash = '53b8dce672cb6d0c73170be43f540460bfc347b4'
>> + outhash =
>> '5a9cb1649625f0bf41fc7791b635cd9c2d7118c7f021ba87dcd03f72b67ce7a8'
>> + unihash = 'f37918cc02eb5a520b1aff86faacbc0a38124646'
>> +
>> + result = self.client.report_unihash(taskhash, self.METHOD,
>> outhash, unihash)
>> + self.assertEqual(result['unihash'], unihash, 'Server
>> returned bad unihash') +
>> + # Report a different task with the same outhash. The
>> returned unihash
>> + # should match the first task
>> + taskhash2 = '3bf6f1e89d26205aec90da04854fbdbf73afe6b4'
>> + unihash2 = 'af36b199320e611fbb16f1f277d3ee1d619ca58b'
>> + result = self.client.report_unihash(taskhash2, self.METHOD,
>> outhash, unihash2)
>> + self.assertEqual(result['unihash'], unihash, 'Server
>> returned bad unihash') +
>> + def test_duplicate_taskhash(self):
>> + # Tests that duplicate reports of the same taskhash with
>> different
>> + # outhash & unihash always return the unihash from the first
>> reported
>> + # taskhash
>> + taskhash = '8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a'
>> + outhash =
>> 'afe240a439959ce86f5e322f8c208e1fedefea9e813f2140c81af866cc9edf7e'
>> + unihash = '218e57509998197d570e2c98512d0105985dffc9'
>> + self.client.report_unihash(taskhash, self.METHOD, outhash,
>> unihash) +
>> + result = self.client.get_unihash(self.METHOD, taskhash)
>> + self.assertEqual(result, unihash)
>> +
>> + outhash2 =
>> '0904a7fe3dc712d9fd8a74a616ddca2a825a8ee97adf0bd3fc86082c7639914d'
>> + unihash2 = 'ae9a7d252735f0dafcdb10e2e02561ca3a47314c'
>> + self.client.report_unihash(taskhash, self.METHOD, outhash2,
>> unihash2) +
>> + result = self.client.get_unihash(self.METHOD, taskhash)
>> + self.assertEqual(result, unihash)
>> +
>> + outhash3 =
>> '77623a549b5b1a31e3732dfa8fe61d7ce5d44b3370f253c5360e136b852967b4'
>> + unihash3 = '9217a7d6398518e5dc002ed58f2cbbbc78696603'
>> + self.client.report_unihash(taskhash, self.METHOD, outhash3,
>> unihash3) +
>> + result = self.client.get_unihash(self.METHOD, taskhash)
>> + self.assertEqual(result, unihash)
>> +
>> + def test_stress(self):
>> + def query_server(failures):
>> + client = Client(self.server.address)
>> + try:
>> + for i in range(1000):
>> + taskhash = hashlib.sha256()
>> + taskhash.update(str(i).encode('utf-8'))
>> + taskhash = taskhash.hexdigest()
>> + result = client.get_unihash(self.METHOD,
>> taskhash)
>> + if result != taskhash:
>> + failures.append("taskhash mismatch: %s !=
>> %s" % (result, taskhash))
>> + finally:
>> + client.close()
>> +
>> + # Report hashes
>> + for i in range(1000):
>> + taskhash = hashlib.sha256()
>> + taskhash.update(str(i).encode('utf-8'))
>> + taskhash = taskhash.hexdigest()
>> + self.client.report_unihash(taskhash, self.METHOD,
>> taskhash, taskhash) +
>> + failures = []
>> + threads = [threading.Thread(target=query_server,
>> args=(failures,)) for t in range(100)] +
>> + for t in threads:
>> + t.start()
>> +
>> + for t in threads:
>> + t.join()
>> +
>> + self.assertFalse(failures)
>> +
>> +
>> +class TestHashEquivalenceUnixServer(TestHashEquivalenceServer,
>> unittest.TestCase):
>> + def get_server_addr(self):
>> + return "unix://" + os.path.join(self.temp_dir.name, 'sock')
>> +
>> +
>> +class TestHashEquivalenceTCPServer(TestHashEquivalenceServer,
>> unittest.TestCase):
>> + def get_server_addr(self):
>> + return "localhost:0"
>> diff --git a/bitbake/lib/layerindexlib/__init__.py
>> b/bitbake/lib/layerindexlib/__init__.py index cb79cb3..77196b4 100644
>> --- a/bitbake/lib/layerindexlib/__init__.py
>> +++ b/bitbake/lib/layerindexlib/__init__.py
>> @@ -1,17 +1,7 @@
>> # Copyright (C) 2016-2018 Wind River Systems, Inc.
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>> -# See the GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> -# along with this program; if not, write to the Free Software
>> -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
>> 02111-1307 USA
>> import datetime
>>
>> @@ -386,7 +376,7 @@ layerBranches set. If not, they are effectively
>> blank.''' invalid.append(name)
>>
>>
>> - def _resolve_dependencies(layerbranches, ignores,
>> dependencies, invalid):
>> + def _resolve_dependencies(layerbranches, ignores,
>> dependencies, invalid, processed=None): for layerbranch in
>> layerbranches: if ignores and layerbranch.layer.name in ignores:
>> continue
>> @@ -398,6 +388,13 @@ layerBranches set. If not, they are effectively
>> blank.''' if ignores and deplayerbranch.layer.name in ignores:
>> continue
>>
>> + # Since this is depth first, we need to know
>> what we're currently processing
>> + # in order to avoid infinite recursion on a loop.
>> + if processed and deplayerbranch.layer.name in
>> processed:
>> + # We have found a recursion...
>> + logger.warning('Circular layer dependency
>> found: %s -> %s' % (processed, deplayerbranch.layer.name))
>> + continue
>> +
>> # This little block is why we can't re-use the
>> LayerIndexObj version, # we must be able to satisfy each dependencies
>> across layer indexes and # use the layer index order for priority.
>> (r stands for replacement below) @@ -421,7 +418,17 @@ layerBranches
>> set. If not, they are effectively blank.'''
>> # New dependency, we need to resolve it now...
>> depth-first if deplayerbranch.layer.name not in dependencies:
>> - (dependencies, invalid) =
>> _resolve_dependencies([deplayerbranch], ignores, dependencies,
>> invalid)
>> + # Avoid recursion on this branch.
>> + # We copy so we don't end up polluting the
>> depth-first branch with other
>> + # branches. Duplication between individual
>> branches IS expected and
>> + # handled by 'dependencies' processing.
>> + if not processed:
>> + local_processed = []
>> + else:
>> + local_processed = processed.copy()
>> +
>> local_processed.append(deplayerbranch.layer.name) +
>> + (dependencies, invalid) =
>> _resolve_dependencies([deplayerbranch], ignores, dependencies,
>> invalid, local_processed) if deplayerbranch.layer.name not in
>> dependencies: dependencies[deplayerbranch.layer.name] =
>> [deplayerbranch, layerdependency] diff --git
>> a/bitbake/lib/layerindexlib/cooker.py
>> b/bitbake/lib/layerindexlib/cooker.py index 848f0e2..604a961 100644
>> --- a/bitbake/lib/layerindexlib/cooker.py +++
>> b/bitbake/lib/layerindexlib/cooker.py @@ -1,17 +1,7 @@
>> # Copyright (C) 2016-2018 Wind River Systems, Inc.
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>> -# See the GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> -# along with this program; if not, write to the Free Software
>> -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
>> 02111-1307 USA
>> import logging
>> import json
>> diff --git a/bitbake/lib/layerindexlib/plugin.py
>> b/bitbake/lib/layerindexlib/plugin.py index 92a2e97..7015a1a 100644
>> --- a/bitbake/lib/layerindexlib/plugin.py
>> +++ b/bitbake/lib/layerindexlib/plugin.py
>> @@ -1,18 +1,7 @@
>> # Copyright (C) 2016-2018 Wind River Systems, Inc.
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>> -# See the GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> -# along with this program; if not, write to the Free Software
>> -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
>> 02111-1307 USA -
>> # The file contains:
>> # LayerIndex exceptions
>> # Plugin base class
>> diff --git a/bitbake/lib/layerindexlib/restapi.py
>> b/bitbake/lib/layerindexlib/restapi.py index d08eb20..21fd144 100644
>> --- a/bitbake/lib/layerindexlib/restapi.py
>> +++ b/bitbake/lib/layerindexlib/restapi.py
>> @@ -1,17 +1,7 @@
>> # Copyright (C) 2016-2018 Wind River Systems, Inc.
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>> -# See the GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> -# along with this program; if not, write to the Free Software
>> -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
>> 02111-1307 USA
>> import logging
>> import json
>> diff --git a/bitbake/lib/layerindexlib/tests/common.py
>> b/bitbake/lib/layerindexlib/tests/common.py index 22a5458..077382f
>> 100644 --- a/bitbake/lib/layerindexlib/tests/common.py
>> +++ b/bitbake/lib/layerindexlib/tests/common.py
>> @@ -1,17 +1,7 @@
>> # Copyright (C) 2017-2018 Wind River Systems, Inc.
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>> -# See the GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> -# along with this program; if not, write to the Free Software
>> -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
>> 02111-1307 USA
>> import unittest
>> import tempfile
>> diff --git a/bitbake/lib/layerindexlib/tests/cooker.py
>> b/bitbake/lib/layerindexlib/tests/cooker.py index fdbf091..1fa102e
>> 100644 --- a/bitbake/lib/layerindexlib/tests/cooker.py
>> +++ b/bitbake/lib/layerindexlib/tests/cooker.py
>> @@ -1,17 +1,7 @@
>> # Copyright (C) 2018 Wind River Systems, Inc.
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>> -# See the GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> -# along with this program; if not, write to the Free Software
>> -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
>> 02111-1307 USA
>> import unittest
>> import tempfile
>> diff --git a/bitbake/lib/layerindexlib/tests/layerindexobj.py
>> b/bitbake/lib/layerindexlib/tests/layerindexobj.py index
>> e2fbb95..0c5ec88 100644 ---
>> a/bitbake/lib/layerindexlib/tests/layerindexobj.py +++
>> b/bitbake/lib/layerindexlib/tests/layerindexobj.py @@ -1,17 +1,7 @@
>> # Copyright (C) 2017-2018 Wind River Systems, Inc.
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>> -# See the GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> -# along with this program; if not, write to the Free Software
>> -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
>> 02111-1307 USA
>> import unittest
>> import tempfile
>> diff --git a/bitbake/lib/layerindexlib/tests/restapi.py
>> b/bitbake/lib/layerindexlib/tests/restapi.py index 5876695..6d8dc00
>> 100644 --- a/bitbake/lib/layerindexlib/tests/restapi.py
>> +++ b/bitbake/lib/layerindexlib/tests/restapi.py
>> @@ -1,17 +1,7 @@
>> # Copyright (C) 2017-2018 Wind River Systems, Inc.
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>> -# See the GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> -# along with this program; if not, write to the Free Software
>> -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
>> 02111-1307 USA
>> import unittest
>> import tempfile
>> diff --git a/bitbake/lib/progressbar/__init__.py
>> b/bitbake/lib/progressbar/__init__.py index fbab744..c545a62 100644
>> --- a/bitbake/lib/progressbar/__init__.py
>> +++ b/bitbake/lib/progressbar/__init__.py
>> @@ -4,6 +4,8 @@
>> # progressbar - Text progress bar library for Python.
>> # Copyright (c) 2005 Nilton Volpato
>> #
>> +# SPDX-License-Identifier: LGPL-2.1-or-later OR BSD-3-Clause-Clear
>> +#
>> # This library is free software; you can redistribute it and/or
>> # modify it under the terms of the GNU Lesser General Public
>> # License as published by the Free Software Foundation; either
>> diff --git a/bitbake/lib/progressbar/compat.py
>> b/bitbake/lib/progressbar/compat.py index a39f4a1..9804e0b 100644
>> --- a/bitbake/lib/progressbar/compat.py
>> +++ b/bitbake/lib/progressbar/compat.py
>> @@ -3,6 +3,8 @@
>> # progressbar - Text progress bar library for Python.
>> # Copyright (c) 2005 Nilton Volpato
>> #
>> +# SPDX-License-Identifier: LGPL-2.1-or-later OR BSD-3-Clause-Clear
>> +#
>> # This library is free software; you can redistribute it and/or
>> # modify it under the terms of the GNU Lesser General Public
>> # License as published by the Free Software Foundation; either
>> diff --git a/bitbake/lib/progressbar/progressbar.py
>> b/bitbake/lib/progressbar/progressbar.py index 2873ad6..e2b6ba1 100644
>> --- a/bitbake/lib/progressbar/progressbar.py
>> +++ b/bitbake/lib/progressbar/progressbar.py
>> @@ -5,6 +5,8 @@
>> #
>> # (With some small changes after importing into BitBake)
>> #
>> +# SPDX-License-Identifier: LGPL-2.1-or-later OR BSD-3-Clause-Clear
>> +#
>> # This library is free software; you can redistribute it and/or
>> # modify it under the terms of the GNU Lesser General Public
>> # License as published by the Free Software Foundation; either
>> diff --git a/bitbake/lib/progressbar/widgets.py
>> b/bitbake/lib/progressbar/widgets.py index 77285ca..0772aa5 100644
>> --- a/bitbake/lib/progressbar/widgets.py
>> +++ b/bitbake/lib/progressbar/widgets.py
>> @@ -3,6 +3,8 @@
>> # progressbar - Text progress bar library for Python.
>> # Copyright (c) 2005 Nilton Volpato
>> #
>> +# SPDX-License-Identifier: LGPL-2.1-or-later OR BSD-3-Clause-Clear
>> +#
>> # This library is free software; you can redistribute it and/or
>> # modify it under the terms of the GNU Lesser General Public
>> # License as published by the Free Software Foundation; either
>> diff --git a/bitbake/lib/prserv/__init__.py
>> b/bitbake/lib/prserv/__init__.py index c3cb73a..9961040 100644
>> --- a/bitbake/lib/prserv/__init__.py
>> +++ b/bitbake/lib/prserv/__init__.py
>> @@ -1,3 +1,7 @@
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> __version__ = "1.0.0"
>>
>> import os, time
>> diff --git a/bitbake/lib/prserv/db.py b/bitbake/lib/prserv/db.py
>> index 495d09f..117d8c0 100644
>> --- a/bitbake/lib/prserv/db.py
>> +++ b/bitbake/lib/prserv/db.py
>> @@ -1,3 +1,7 @@
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> import logging
>> import os.path
>> import errno
>> @@ -253,7 +257,7 @@ class PRData(object):
>> self.connection=sqlite3.connect(self.filename,
>> isolation_level="EXCLUSIVE", check_same_thread = False)
>> self.connection.row_factory=sqlite3.Row
>> self.connection.execute("pragma synchronous = off;")
>> - self.connection.execute("PRAGMA journal_mode = WAL;")
>> + self.connection.execute("PRAGMA journal_mode = MEMORY;")
>> self._tables={}
>>
>> def disconnect(self):
>> diff --git a/bitbake/lib/prserv/serv.py b/bitbake/lib/prserv/serv.py
>> index 6a99728..be3acec 100644
>> --- a/bitbake/lib/prserv/serv.py
>> +++ b/bitbake/lib/prserv/serv.py
>> @@ -1,3 +1,7 @@
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> import os,sys,logging
>> import signal, time
>> from xmlrpc.server import SimpleXMLRPCServer,
>> SimpleXMLRPCRequestHandler diff --git a/bitbake/lib/pyinotify.py
>> b/bitbake/lib/pyinotify.py index 4eb03b0..1528a22 100644
>> --- a/bitbake/lib/pyinotify.py
>> +++ b/bitbake/lib/pyinotify.py
>> @@ -1,25 +1,9 @@
>> -#!/usr/bin/env python
>> -
>> +#
>> # pyinotify.py - python interface to inotify
>> # Copyright (c) 2005-2015 Sebastien Martini <seb@dbzteam.org>
>> #
>> -# Permission is hereby granted, free of charge, to any person
>> obtaining a copy -# of this software and associated documentation
>> files (the "Software"), to deal -# in the Software without
>> restriction, including without limitation the rights -# to use, copy,
>> modify, merge, publish, distribute, sublicense, and/or sell -# copies
>> of the Software, and to permit persons to whom the Software is -#
>> furnished to do so, subject to the following conditions: -#
>> -# The above copyright notice and this permission notice shall be
>> included in -# all copies or substantial portions of the Software.
>> +# SPDX-License-Identifier: MIT
>> #
>> -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
>> EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
>> MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND
>> NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT
>> HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY,
>> WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -#
>> OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
>> DEALINGS IN -# THE SOFTWARE. """
>> pyinotify
>>
>> diff --git a/bitbake/lib/toaster/bldcollector/admin.py
>> b/bitbake/lib/toaster/bldcollector/admin.py index 1f2e07f..feaa888
>> 100644 --- a/bitbake/lib/toaster/bldcollector/admin.py
>> +++ b/bitbake/lib/toaster/bldcollector/admin.py
>> @@ -1,3 +1,7 @@
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> from django.contrib import admin
>> from orm.models import BitbakeVersion, Release, ToasterSetting,
>> Layer_Version from django import forms
>> diff --git a/bitbake/lib/toaster/bldcollector/urls.py
>> b/bitbake/lib/toaster/bldcollector/urls.py index 888175d..8eb1e34
>> 100644 --- a/bitbake/lib/toaster/bldcollector/urls.py
>> +++ b/bitbake/lib/toaster/bldcollector/urls.py
>> @@ -3,19 +3,8 @@
>> #
>> # Copyright (C) 2014-2017 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. -
>>
>> from django.conf.urls import include, url
>>
>> diff --git a/bitbake/lib/toaster/bldcollector/views.py
>> b/bitbake/lib/toaster/bldcollector/views.py index f32fa4d..c708b41
>> 100644 --- a/bitbake/lib/toaster/bldcollector/views.py
>> +++ b/bitbake/lib/toaster/bldcollector/views.py
>> @@ -3,18 +3,8 @@
>> #
>> # Copyright (C) 2014 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from django.views.decorators.cache import cache_control
>> from django.core.urlresolvers import reverse
>> diff --git a/bitbake/lib/toaster/bldcontrol/admin.py
>> b/bitbake/lib/toaster/bldcontrol/admin.py index fcbe5f5..e85c30e
>> 100644 --- a/bitbake/lib/toaster/bldcontrol/admin.py
>> +++ b/bitbake/lib/toaster/bldcontrol/admin.py
>> @@ -1,3 +1,7 @@
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> from django.contrib import admin
>> from django.contrib.admin.filters import RelatedFieldListFilter
>> from .models import BuildEnvironment
>> diff --git a/bitbake/lib/toaster/bldcontrol/bbcontroller.py
>> b/bitbake/lib/toaster/bldcontrol/bbcontroller.py index
>> 5195600..301df18 100644 ---
>> a/bitbake/lib/toaster/bldcontrol/bbcontroller.py +++
>> b/bitbake/lib/toaster/bldcontrol/bbcontroller.py @@ -1,24 +1,10 @@
>> #
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> -#
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2014 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. -
>>
>> import os
>> import sys
>> diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
>> b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py index
>> 9490635..39ea736 100644 ---
>> a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py +++
>> b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py @@ -1,24
>> +1,10 @@ #
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> -#
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2014 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. -
>>
>> import os
>> import sys
>> diff --git
>> a/bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py
>> b/bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py
>> index 14298d9..fe2c4dc 100644 ---
>> a/bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py
>> +++
>> b/bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py
>> @@ -1,3 +1,7 @@ +# +# SPDX-License-Identifier: GPL-2.0-only +#
>> +
>> from django.core.management.base import BaseCommand, CommandError
>> from django.db import transaction
>>
>> diff --git
>> a/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py
>> b/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py
>> index 6a55dd4..50ec409 100644 ---
>> a/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py +++
>> b/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py @@
>> -1,3 +1,7 @@ +# +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> from django.core.management.base import BaseCommand
>> from django.db import transaction
>> from django.db.models import Q
>> diff --git a/bitbake/lib/toaster/bldcontrol/models.py
>> b/bitbake/lib/toaster/bldcontrol/models.py index 409614b..bcffcf5
>> 100644 --- a/bitbake/lib/toaster/bldcontrol/models.py
>> +++ b/bitbake/lib/toaster/bldcontrol/models.py
>> @@ -1,3 +1,7 @@
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> from __future__ import unicode_literals
>> from django.db import models
>> from django.core.validators import MaxValueValidator,
>> MinValueValidator diff --git
>> a/bitbake/lib/toaster/bldcontrol/views.py
>> b/bitbake/lib/toaster/bldcontrol/views.py index 60f00ef..286d88b
>> 100644 --- a/bitbake/lib/toaster/bldcontrol/views.py +++
>> b/bitbake/lib/toaster/bldcontrol/views.py @@ -1 +1,5 @@
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> # Create your views here.
>> diff --git a/bitbake/lib/toaster/manage.py
>> b/bitbake/lib/toaster/manage.py index 0c7ea50..ae32619 100755
>> --- a/bitbake/lib/toaster/manage.py
>> +++ b/bitbake/lib/toaster/manage.py
>> @@ -1,4 +1,8 @@
>> #!/usr/bin/env python3
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> import os
>> import sys
>>
>> diff --git a/bitbake/lib/toaster/orm/fixtures/oe-core.xml
>> b/bitbake/lib/toaster/orm/fixtures/oe-core.xml index fec93ab..a723f5a
>> 100644 --- a/bitbake/lib/toaster/orm/fixtures/oe-core.xml
>> +++ b/bitbake/lib/toaster/orm/fixtures/oe-core.xml
>> @@ -8,9 +8,9 @@
>>
>> <!-- Bitbake versions which correspond to the metadata release -->
>> <object model="orm.bitbakeversion" pk="1">
>> - <field type="CharField" name="name">sumo</field>
>> + <field type="CharField" name="name">warrior</field>
>> <field type="CharField"
>> name="giturl">git://git.openembedded.org/bitbake</field>
>> - <field type="CharField" name="branch">1.38</field>
>> + <field type="CharField" name="branch">1.42</field>
>> </object>
>> <object model="orm.bitbakeversion" pk="2">
>> <field type="CharField" name="name">HEAD</field>
>> @@ -23,18 +23,18 @@
>> <field type="CharField" name="branch">master</field>
>> </object>
>> <object model="orm.bitbakeversion" pk="4">
>> - <field type="CharField" name="name">thud</field>
>> + <field type="CharField" name="name">zeus</field>
>> <field type="CharField"
>> name="giturl">git://git.openembedded.org/bitbake</field>
>> - <field type="CharField" name="branch">1.40</field>
>> + <field type="CharField" name="branch">1.44</field>
>> </object>
>>
>> <!-- Releases available -->
>> <object model="orm.release" pk="1">
>> - <field type="CharField" name="name">sumo</field>
>> - <field type="CharField" name="description">Openembedded
>> Sumo</field>
>> + <field type="CharField" name="name">warrior</field>
>> + <field type="CharField" name="description">Openembedded
>> Warrior</field> <field rel="ManyToOneRel" to="orm.bitbakeversion"
>> name="bitbake_version">1</field>
>> - <field type="CharField" name="branch_name">sumo</field>
>> - <field type="TextField" name="helptext">Toaster will run your
>> builds using the tip of the <a
>> href=\"http://cgit.openembedded.org/openembedded-core/log/?h=sumo\">OpenEmbedded
>> Sumo</a> branch.</field>
>> + <field type="CharField" name="branch_name">warrior</field>
>> + <field type="TextField" name="helptext">Toaster will run your
>> builds using the tip of the <a
>> href=\"http://cgit.openembedded.org/openembedded-core/log/?h=warrior\">OpenEmbedded
>> Warrior</a> branch.</field> </object> <object
>> model="orm.release" pk="2"> <field type="CharField"
>> name="name">local</field> @@ -51,11 +51,11 @@ <field type="TextField"
>> name="helptext">Toaster will run your builds using the tip of the
>> <a
>> href=\"http://cgit.openembedded.org/openembedded-core/log/\">OpenEmbedded
>> master</a> branch.</field> </object> <object
>> model="orm.release" pk="4">
>> - <field type="CharField" name="name">thud</field>
>> - <field type="CharField" name="description">Openembedded
>> Rocko</field>
>> - <field rel="ManyToOneRel" to="orm.bitbakeversion"
>> name="bitbake_version">1</field>
>> - <field type="CharField" name="branch_name">thud</field>
>> - <field type="TextField" name="helptext">Toaster will run your
>> builds using the tip of the <a
>> href=\"http://cgit.openembedded.org/openembedded-core/log/?h=thud\">OpenEmbedded
>> Thud</a> branch.</field>
>> + <field type="CharField" name="name">zeus</field>
>> + <field type="CharField" name="description">Openembedded
>> Zeus</field>
>> + <field rel="ManyToOneRel" to="orm.bitbakeversion"
>> name="bitbake_version">4</field>
>> + <field type="CharField" name="branch_name">zeus</field>
>> + <field type="TextField" name="helptext">Toaster will run your
>> builds using the tip of the <a
>> href=\"http://cgit.openembedded.org/openembedded-core/log/?h=zeus\">OpenEmbedded
>> Zeus</a> branch.</field> </object> <!-- Default layers for each
>> release --> diff --git a/bitbake/lib/toaster/orm/fixtures/poky.xml
>> b/bitbake/lib/toaster/orm/fixtures/poky.xml index fb9a771..7992383
>> 100644 --- a/bitbake/lib/toaster/orm/fixtures/poky.xml
>> +++ b/bitbake/lib/toaster/orm/fixtures/poky.xml
>> @@ -8,9 +8,9 @@
>>
>> <!-- Bitbake versions which correspond to the metadata release -->
>> <object model="orm.bitbakeversion" pk="1">
>> - <field type="CharField" name="name">sumo</field>
>> + <field type="CharField" name="name">warrior</field>
>> <field type="CharField"
>> name="giturl">git://git.yoctoproject.org/poky</field>
>> - <field type="CharField" name="branch">sumo</field>
>> + <field type="CharField" name="branch">warrior</field>
>> <field type="CharField" name="dirpath">bitbake</field>
>> </object>
>> <object model="orm.bitbakeversion" pk="2">
>> @@ -26,20 +26,20 @@
>> <field type="CharField" name="dirpath">bitbake</field>
>> </object>
>> <object model="orm.bitbakeversion" pk="4">
>> - <field type="CharField" name="name">thud</field>
>> + <field type="CharField" name="name">zeus</field>
>> <field type="CharField"
>> name="giturl">git://git.yoctoproject.org/poky</field>
>> - <field type="CharField" name="branch">thud</field>
>> + <field type="CharField" name="branch">zeus</field>
>> <field type="CharField" name="dirpath">bitbake</field>
>> </object>
>>
>>
>> <!-- Releases available -->
>> <object model="orm.release" pk="1">
>> - <field type="CharField" name="name">sumo</field>
>> - <field type="CharField" name="description">Yocto Project 2.5
>> "Sumo"</field>
>> + <field type="CharField" name="name">warrior</field>
>> + <field type="CharField" name="description">Yocto Project 2.7
>> "Warrior"</field> <field rel="ManyToOneRel" to="orm.bitbakeversion"
>> name="bitbake_version">1</field>
>> - <field type="CharField" name="branch_name">sumo</field>
>> - <field type="TextField" name="helptext">Toaster will run your
>> builds using the tip of the <a
>> href="http://git.yoctoproject.org/cgit/cgit.cgi/poky/log/?h=sumo">Yocto
>> Project Sumo branch</a>.</field>
>> + <field type="CharField" name="branch_name">warrior</field>
>> + <field type="TextField" name="helptext">Toaster will run your
>> builds using the tip of the <a
>> href="http://git.yoctoproject.org/cgit/cgit.cgi/poky/log/?h=warrior">Yocto
>> Project Warrior branch</a>.</field> </object> <object
>> model="orm.release" pk="2"> <field type="CharField"
>> name="name">local</field> @@ -56,11 +56,11 @@ <field type="TextField"
>> name="helptext">Toaster will run your builds using the tip of the
>> <a
>> href="http://git.yoctoproject.org/cgit/cgit.cgi/poky/log/">Yocto
>> Project Master branch</a>.</field> </object> <object
>> model="orm.release" pk="4">
>> - <field type="CharField" name="name">rocko</field>
>> - <field type="CharField" name="description">Yocto Project 2.6
>> "Thud"</field>
>> - <field rel="ManyToOneRel" to="orm.bitbakeversion"
>> name="bitbake_version">1</field>
>> - <field type="CharField" name="branch_name">thud</field>
>> - <field type="TextField" name="helptext">Toaster will run your
>> builds using the tip of the <a
>> href="http://git.yoctoproject.org/cgit/cgit.cgi/poky/log/?h=thud">Yocto
>> Project Thud branch</a>.</field>
>> + <field type="CharField" name="name">zeus</field>
>> + <field type="CharField" name="description">Yocto Project 3.0
>> "Zeus"</field>
>> + <field rel="ManyToOneRel" to="orm.bitbakeversion"
>> name="bitbake_version">4</field>
>> + <field type="CharField" name="branch_name">zeus</field>
>> + <field type="TextField" name="helptext">Toaster will run your
>> builds using the tip of the <a
>> href="http://git.yoctoproject.org/cgit/cgit.cgi/poky/log/?h=zeus">Yocto
>> Project Zeus branch</a>.</field> </object> <!-- Default project
>> layers for each release --> @@ -130,7 +130,7 @@
>> <field rel="ManyToOneRel" to="orm.layer" name="layer">1</field>
>> <field type="IntegerField" name="layer_source">0</field>
>> <field rel="ManyToOneRel" to="orm.release"
>> name="release">1</field>
>> - <field type="CharField" name="branch">sumo</field>
>> + <field type="CharField" name="branch">warrior</field>
>> <field type="CharField" name="dirpath">meta</field>
>> </object>
>> <object model="orm.layer_version" pk="2">
>> @@ -152,7 +152,7 @@
>> <field rel="ManyToOneRel" to="orm.layer" name="layer">1</field>
>> <field type="IntegerField" name="layer_source">0</field>
>> <field rel="ManyToOneRel" to="orm.release"
>> name="release">4</field>
>> - <field type="CharField" name="branch">rocko</field>
>> + <field type="CharField" name="branch">zeus</field>
>> <field type="CharField" name="dirpath">meta</field>
>> </object>
>>
>> @@ -168,7 +168,7 @@
>> <field rel="ManyToOneRel" to="orm.layer" name="layer">2</field>
>> <field type="IntegerField" name="layer_source">0</field>
>> <field rel="ManyToOneRel" to="orm.release"
>> name="release">1</field>
>> - <field type="CharField" name="branch">sumo</field>
>> + <field type="CharField" name="branch">warrior</field>
>> <field type="CharField" name="dirpath">meta-poky</field>
>> </object>
>> <object model="orm.layer_version" pk="6">
>> @@ -190,7 +190,7 @@
>> <field rel="ManyToOneRel" to="orm.layer" name="layer">2</field>
>> <field type="IntegerField" name="layer_source">0</field>
>> <field rel="ManyToOneRel" to="orm.release"
>> name="release">4</field>
>> - <field type="CharField" name="branch">rocko</field>
>> + <field type="CharField" name="branch">zeus</field>
>> <field type="CharField" name="dirpath">meta-poky</field>
>> </object>
>>
>> @@ -206,7 +206,7 @@
>> <field rel="ManyToOneRel" to="orm.layer" name="layer">3</field>
>> <field type="IntegerField" name="layer_source">0</field>
>> <field rel="ManyToOneRel" to="orm.release"
>> name="release">1</field>
>> - <field type="CharField" name="branch">sumo</field>
>> + <field type="CharField" name="branch">warrior</field>
>> <field type="CharField" name="dirpath">meta-yocto-bsp</field>
>> </object>
>> <object model="orm.layer_version" pk="10">
>> @@ -228,7 +228,7 @@
>> <field rel="ManyToOneRel" to="orm.layer" name="layer">3</field>
>> <field type="IntegerField" name="layer_source">0</field>
>> <field rel="ManyToOneRel" to="orm.release"
>> name="release">4</field>
>> - <field type="CharField" name="branch">rocko</field>
>> + <field type="CharField" name="branch">zeus</field>
>> <field type="CharField" name="dirpath">meta-yocto-bsp</field>
>> </object>
>> </django-objects>
>> diff --git a/bitbake/lib/toaster/orm/management/commands/lsupdates.py
>> b/bitbake/lib/toaster/orm/management/commands/lsupdates.py index
>> 66114ff..5b5abbb 100644 ---
>> a/bitbake/lib/toaster/orm/management/commands/lsupdates.py +++
>> b/bitbake/lib/toaster/orm/management/commands/lsupdates.py @@ -1,23
>> +1,10 @@ #
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> -#
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2016-2017 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from django.core.management.base import BaseCommand
>>
>> diff --git a/bitbake/lib/toaster/orm/models.py
>> b/bitbake/lib/toaster/orm/models.py index 7720290..bb6b5de 100644
>> --- a/bitbake/lib/toaster/orm/models.py
>> +++ b/bitbake/lib/toaster/orm/models.py
>> @@ -1,23 +1,10 @@
>> #
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> -#
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from __future__ import unicode_literals
>>
>> @@ -978,12 +965,12 @@ class TargetSDKFile(models.Model):
>> class Target_Image_File(models.Model):
>> # valid suffixes for image files produced by a build
>> SUFFIXES = {
>> - 'btrfs', 'cpio', 'cpio.gz', 'cpio.lz4', 'cpio.lzma',
>> 'cpio.xz',
>> - 'cramfs', 'elf', 'ext2', 'ext2.bz2', 'ext2.gz', 'ext2.lzma',
>> 'ext4',
>> - 'ext4.gz', 'ext3', 'ext3.gz', 'hdddirect', 'hddimg', 'iso',
>> 'jffs2',
>> - 'jffs2.sum', 'multiubi', 'qcow2', 'squashfs', 'squashfs-lzo',
>> + 'btrfs', 'container', 'cpio', 'cpio.gz', 'cpio.lz4',
>> 'cpio.lzma',
>> + 'cpio.xz', 'cramfs', 'ext2', 'ext2.bz2', 'ext2.gz',
>> 'ext2.lzma',
>> + 'ext3', 'ext3.gz', 'ext4', 'ext4.gz', 'f2fs', 'hddimg',
>> 'iso', 'jffs2',
>> + 'jffs2.sum', 'multiubi', 'squashfs', 'squashfs-lz4',
>> 'squashfs-lzo', 'squashfs-xz', 'tar', 'tar.bz2', 'tar.gz', 'tar.lz4',
>> 'tar.xz', 'ubi',
>> - 'ubifs', 'vdi', 'vmdk', 'wic', 'wic.bmap', 'wic.bz2',
>> 'wic.gz', 'wic.lzma'
>> + 'ubifs', 'wic', 'wic.bz2', 'wic.gz', 'wic.lzma'
>> }
>>
>> target = models.ForeignKey(Target)
>> diff --git a/bitbake/lib/toaster/tests/browser/selenium_helpers.py
>> b/bitbake/lib/toaster/tests/browser/selenium_helpers.py index
>> 08711e4..02d4f4b 100644 ---
>> a/bitbake/lib/toaster/tests/browser/selenium_helpers.py +++
>> b/bitbake/lib/toaster/tests/browser/selenium_helpers.py @@ -1,23
>> +1,10 @@ -#! /usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> # The Wait class and some of SeleniumDriverHelper and
>> SeleniumTestCase are # modified from Patchwork, released under the
>> same licence terms as Toaster: diff --git
>> a/bitbake/lib/toaster/tests/browser/selenium_helpers_base.py
>> b/bitbake/lib/toaster/tests/browser/selenium_helpers_base.py index
>> 156d639..6c94684 100644 ---
>> a/bitbake/lib/toaster/tests/browser/selenium_helpers_base.py +++
>> b/bitbake/lib/toaster/tests/browser/selenium_helpers_base.py @@ -1,23
>> +1,10 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> # The Wait class and some of SeleniumDriverHelper and
>> SeleniumTestCase are # modified from Patchwork, released under the
>> same licence terms as Toaster: diff --git
>> a/bitbake/lib/toaster/tests/browser/test_all_builds_page.py
>> b/bitbake/lib/toaster/tests/browser/test_all_builds_page.py index
>> b86f29b..fba627b 100644 ---
>> a/bitbake/lib/toaster/tests/browser/test_all_builds_page.py +++
>> b/bitbake/lib/toaster/tests/browser/test_all_builds_page.py @@ -1,23
>> +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import re
>>
>> diff --git
>> a/bitbake/lib/toaster/tests/browser/test_all_projects_page.py
>> b/bitbake/lib/toaster/tests/browser/test_all_projects_page.py index
>> 44da640..afd2d35 100644 ---
>> a/bitbake/lib/toaster/tests/browser/test_all_projects_page.py +++
>> b/bitbake/lib/toaster/tests/browser/test_all_projects_page.py @@
>> -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import re
>>
>> diff --git
>> a/bitbake/lib/toaster/tests/browser/test_builddashboard_page.py
>> b/bitbake/lib/toaster/tests/browser/test_builddashboard_page.py index
>> f8ccb54..d972aff 100644 ---
>> a/bitbake/lib/toaster/tests/browser/test_builddashboard_page.py +++
>> b/bitbake/lib/toaster/tests/browser/test_builddashboard_page.py @@
>> -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from django.core.urlresolvers import reverse
>> from django.utils import timezone
>> diff --git
>> a/bitbake/lib/toaster/tests/browser/test_builddashboard_page_artifacts.py
>> b/bitbake/lib/toaster/tests/browser/test_builddashboard_page_artifacts.py
>> index 1c627ad..e2623e8 100644 ---
>> a/bitbake/lib/toaster/tests/browser/test_builddashboard_page_artifacts.py
>> +++
>> b/bitbake/lib/toaster/tests/browser/test_builddashboard_page_artifacts.py
>> @@ -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et -#
>> -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3 #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from django.core.urlresolvers import reverse
>> from django.utils import timezone
>> diff --git
>> a/bitbake/lib/toaster/tests/browser/test_builddashboard_page_recipes.py
>> b/bitbake/lib/toaster/tests/browser/test_builddashboard_page_recipes.py
>> index ed18324..c542d45 100644 ---
>> a/bitbake/lib/toaster/tests/browser/test_builddashboard_page_recipes.py
>> +++
>> b/bitbake/lib/toaster/tests/browser/test_builddashboard_page_recipes.py
>> @@ -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et -#
>> -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3 #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from django.core.urlresolvers import reverse
>> from django.utils import timezone
>> diff --git
>> a/bitbake/lib/toaster/tests/browser/test_builddashboard_page_tasks.py
>> b/bitbake/lib/toaster/tests/browser/test_builddashboard_page_tasks.py
>> index da50f16..22acb47 100644 ---
>> a/bitbake/lib/toaster/tests/browser/test_builddashboard_page_tasks.py
>> +++
>> b/bitbake/lib/toaster/tests/browser/test_builddashboard_page_tasks.py
>> @@ -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et -#
>> -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3 #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from django.core.urlresolvers import reverse
>> from django.utils import timezone
>> diff --git a/bitbake/lib/toaster/tests/browser/test_js_unit_tests.py
>> b/bitbake/lib/toaster/tests/browser/test_js_unit_tests.py index
>> 3c0b962..e8b4295 100644 ---
>> a/bitbake/lib/toaster/tests/browser/test_js_unit_tests.py +++
>> b/bitbake/lib/toaster/tests/browser/test_js_unit_tests.py @@ -1,23
>> +1,11 @@ -#! /usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> """
>> Run the js unit tests
>> diff --git a/bitbake/lib/toaster/tests/browser/test_landing_page.py
>> b/bitbake/lib/toaster/tests/browser/test_landing_page.py index
>> 4d4cd66..0790198 100644 ---
>> a/bitbake/lib/toaster/tests/browser/test_landing_page.py +++
>> b/bitbake/lib/toaster/tests/browser/test_landing_page.py @@ -1,23
>> +1,11 @@ -#! /usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> -# Copyright (C) 2013-2016 Intel Corporation
>> -#
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from django.core.urlresolvers import reverse
>> from django.utils import timezone
>> diff --git
>> a/bitbake/lib/toaster/tests/browser/test_layerdetails_page.py
>> b/bitbake/lib/toaster/tests/browser/test_layerdetails_page.py index
>> f24fb09..f81e696 100644 ---
>> a/bitbake/lib/toaster/tests/browser/test_layerdetails_page.py +++
>> b/bitbake/lib/toaster/tests/browser/test_layerdetails_page.py @@
>> -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> -# Copyright (C) 2013-2016 Intel Corporation
>> -#
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from django.core.urlresolvers import reverse
>> from tests.browser.selenium_helpers import SeleniumTestCase
>> diff --git
>> a/bitbake/lib/toaster/tests/browser/test_most_recent_builds_states.py
>> b/bitbake/lib/toaster/tests/browser/test_most_recent_builds_states.py
>> index abc0b0b..15d25dc 100644 ---
>> a/bitbake/lib/toaster/tests/browser/test_most_recent_builds_states.py
>> +++
>> b/bitbake/lib/toaster/tests/browser/test_most_recent_builds_states.py
>> @@ -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et -#
>> -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3 #
>> # BitBake Toaster Implementation
>> #
>> -# Copyright (C) 2013-2016 Intel Corporation
>> -#
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from django.core.urlresolvers import reverse
>> from django.utils import timezone
>> diff --git
>> a/bitbake/lib/toaster/tests/browser/test_new_custom_image_page.py
>> b/bitbake/lib/toaster/tests/browser/test_new_custom_image_page.py
>> index ab5a8e6..0aa3b7a 100644 ---
>> a/bitbake/lib/toaster/tests/browser/test_new_custom_image_page.py +++
>> b/bitbake/lib/toaster/tests/browser/test_new_custom_image_page.py @@
>> -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from django.core.urlresolvers import reverse
>> from tests.browser.selenium_helpers import SeleniumTestCase
>> diff --git
>> a/bitbake/lib/toaster/tests/browser/test_new_project_page.py
>> b/bitbake/lib/toaster/tests/browser/test_new_project_page.py index
>> 77e5f15..8e56bb0 100644 ---
>> a/bitbake/lib/toaster/tests/browser/test_new_project_page.py +++
>> b/bitbake/lib/toaster/tests/browser/test_new_project_page.py @@ -1,23
>> +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from django.core.urlresolvers import reverse
>> from tests.browser.selenium_helpers import SeleniumTestCase
>> diff --git
>> a/bitbake/lib/toaster/tests/browser/test_project_builds_page.py
>> b/bitbake/lib/toaster/tests/browser/test_project_builds_page.py index
>> 9fe91ab..47fb10b 100644 ---
>> a/bitbake/lib/toaster/tests/browser/test_project_builds_page.py +++
>> b/bitbake/lib/toaster/tests/browser/test_project_builds_page.py @@
>> -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import re
>>
>> diff --git
>> a/bitbake/lib/toaster/tests/browser/test_project_config_page.py
>> b/bitbake/lib/toaster/tests/browser/test_project_config_page.py index
>> 0710084..2816eb9 100644 ---
>> a/bitbake/lib/toaster/tests/browser/test_project_config_page.py +++
>> b/bitbake/lib/toaster/tests/browser/test_project_config_page.py @@
>> -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import re
>>
>> diff --git a/bitbake/lib/toaster/tests/browser/test_project_page.py
>> b/bitbake/lib/toaster/tests/browser/test_project_page.py index
>> 0186463..8b5e1b6 100644 ---
>> a/bitbake/lib/toaster/tests/browser/test_project_page.py +++
>> b/bitbake/lib/toaster/tests/browser/test_project_page.py @@ -1,23
>> +1,11 @@ -#! /usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from django.core.urlresolvers import reverse
>> from django.utils import timezone
>> diff --git a/bitbake/lib/toaster/tests/browser/test_sample.py
>> b/bitbake/lib/toaster/tests/browser/test_sample.py index
>> 20ec53c..f4ad670 100644 ---
>> a/bitbake/lib/toaster/tests/browser/test_sample.py +++
>> b/bitbake/lib/toaster/tests/browser/test_sample.py @@ -1,23 +1,11 @@
>> -#! /usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> """
>> A small example test demonstrating the basics of writing a test with
>> diff --git a/bitbake/lib/toaster/tests/browser/test_task_page.py
>> b/bitbake/lib/toaster/tests/browser/test_task_page.py index
>> 690d116..26f3dca 100644 ---
>> a/bitbake/lib/toaster/tests/browser/test_task_page.py +++
>> b/bitbake/lib/toaster/tests/browser/test_task_page.py @@ -1,23 +1,11
>> @@ -#! /usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from django.core.urlresolvers import reverse
>> from django.utils import timezone
>> diff --git
>> a/bitbake/lib/toaster/tests/browser/test_toastertable_ui.py
>> b/bitbake/lib/toaster/tests/browser/test_toastertable_ui.py index
>> 53ddf30..ef78cbb 100644 ---
>> a/bitbake/lib/toaster/tests/browser/test_toastertable_ui.py +++
>> b/bitbake/lib/toaster/tests/browser/test_toastertable_ui.py @@ -1,23
>> +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from datetime import datetime
>>
>> diff --git a/bitbake/lib/toaster/tests/builds/buildtest.py
>> b/bitbake/lib/toaster/tests/builds/buildtest.py index
>> 5a56a11..872bbd3 100644 ---
>> a/bitbake/lib/toaster/tests/builds/buildtest.py +++
>> b/bitbake/lib/toaster/tests/builds/buildtest.py @@ -1,23 +1,11 @@
>> -#! /usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import os
>> import sys
>> diff --git a/bitbake/lib/toaster/tests/builds/test_core_image_min.py
>> b/bitbake/lib/toaster/tests/builds/test_core_image_min.py index
>> 586f4a8..44b6cbe 100644 ---
>> a/bitbake/lib/toaster/tests/builds/test_core_image_min.py +++
>> b/bitbake/lib/toaster/tests/builds/test_core_image_min.py @@ -1,24
>> +1,11 @@ -#! /usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. -
>> # Tests were part of openembedded-core oe selftest Authored by:
>> Lucian Musat # Ionut Chisanovici, Paul Eggleton and Cristian Iorga
>>
>> diff --git a/bitbake/lib/toaster/tests/commands/test_loaddata.py
>> b/bitbake/lib/toaster/tests/commands/test_loaddata.py index
>> 951f6ff..9e8d555 100644 ---
>> a/bitbake/lib/toaster/tests/commands/test_loaddata.py +++
>> b/bitbake/lib/toaster/tests/commands/test_loaddata.py @@ -1,23 +1,11
>> @@ -#! /usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from django.test import TestCase
>> from django.core import management
>> diff --git a/bitbake/lib/toaster/tests/commands/test_lsupdates.py
>> b/bitbake/lib/toaster/tests/commands/test_lsupdates.py index
>> 49897a4..3c4fbe0 100644 ---
>> a/bitbake/lib/toaster/tests/commands/test_lsupdates.py +++
>> b/bitbake/lib/toaster/tests/commands/test_lsupdates.py @@ -1,23 +1,11
>> @@ -#! /usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from django.test import TestCase
>> from django.core import management
>> diff --git a/bitbake/lib/toaster/tests/commands/test_runbuilds.py
>> b/bitbake/lib/toaster/tests/commands/test_runbuilds.py index
>> 3e63483..e223b95 100644 ---
>> a/bitbake/lib/toaster/tests/commands/test_runbuilds.py +++
>> b/bitbake/lib/toaster/tests/commands/test_runbuilds.py @@ -1,23 +1,11
>> @@ -#! /usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import os
>>
>> diff --git a/bitbake/lib/toaster/tests/db/test_db.py
>> b/bitbake/lib/toaster/tests/db/test_db.py index a0f5f6e..0410422
>> 100644 --- a/bitbake/lib/toaster/tests/db/test_db.py
>> +++ b/bitbake/lib/toaster/tests/db/test_db.py
>> @@ -2,6 +2,8 @@
>> #
>> # Copyright (c) 2016 Damien Lespiau
>> #
>> +# SPDX-License-Identifier: MIT
>> +#
>> # Permission is hereby granted, free of charge, to any person
>> obtaining a copy # of this software and associated documentation
>> files (the "Software"), to deal # in the Software without
>> restriction, including without limitation the rights diff --git
>> a/bitbake/lib/toaster/tests/eventreplay/__init__.py
>> b/bitbake/lib/toaster/tests/eventreplay/__init__.py index
>> 6956619..8ed6792 100644 ---
>> a/bitbake/lib/toaster/tests/eventreplay/__init__.py +++
>> b/bitbake/lib/toaster/tests/eventreplay/__init__.py @@ -1,23 +1,11 @@
>> -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> # Tests were part of openembedded-core oe selftest Authored by:
>> Lucian Musat # Ionut Chisanovici, Paul Eggleton and Cristian Iorga
>> diff --git
>> a/bitbake/lib/toaster/tests/functional/functional_helpers.py
>> b/bitbake/lib/toaster/tests/functional/functional_helpers.py index
>> 486078a..455c408 100644 ---
>> a/bitbake/lib/toaster/tests/functional/functional_helpers.py +++
>> b/bitbake/lib/toaster/tests/functional/functional_helpers.py @@ -1,23
>> +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster functional tests implementation
>> #
>> # Copyright (C) 2017 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import os
>> import logging
>> diff --git
>> a/bitbake/lib/toaster/tests/functional/test_functional_basic.py
>> b/bitbake/lib/toaster/tests/functional/test_functional_basic.py index
>> cfa2b0f..56c84fb 100644 ---
>> a/bitbake/lib/toaster/tests/functional/test_functional_basic.py +++
>> b/bitbake/lib/toaster/tests/functional/test_functional_basic.py @@
>> -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster functional tests implementation
>> #
>> # Copyright (C) 2017 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import time
>> import re
>> diff --git a/bitbake/lib/toaster/tests/views/test_views.py
>> b/bitbake/lib/toaster/tests/views/test_views.py index
>> 1463077..68d9e9d 100644 ---
>> a/bitbake/lib/toaster/tests/views/test_views.py +++
>> b/bitbake/lib/toaster/tests/views/test_views.py @@ -1,23 +1,11 @@
>> -#! /usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013-2015 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> """Test cases for Toaster GUI and ReST."""
>>
>> diff --git a/bitbake/lib/toaster/toastergui/api.py
>> b/bitbake/lib/toaster/toastergui/api.py index 564d595..8b49b3e 100644
>> --- a/bitbake/lib/toaster/toastergui/api.py
>> +++ b/bitbake/lib/toaster/toastergui/api.py
>> @@ -3,19 +3,8 @@
>> #
>> # Copyright (C) 2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. -
>> # Please run flake8 on this file before sending patches
>>
>> import os
>> diff --git a/bitbake/lib/toaster/toastergui/buildtables.py
>> b/bitbake/lib/toaster/toastergui/buildtables.py index
>> 755a7c2..327059d 100644 ---
>> a/bitbake/lib/toaster/toastergui/buildtables.py +++
>> b/bitbake/lib/toaster/toastergui/buildtables.py @@ -1,23 +1,10 @@
>> #
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> -#
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from orm.models import Build, Task, Target, Package
>> from django.db.models import Q, Sum
>> diff --git a/bitbake/lib/toaster/toastergui/static/js/importlayer.js
>> b/bitbake/lib/toaster/toastergui/static/js/importlayer.js index
>> 2964839..8e2032d 100644 ---
>> a/bitbake/lib/toaster/toastergui/static/js/importlayer.js +++
>> b/bitbake/lib/toaster/toastergui/static/js/importlayer.js @@ -17,11
>> +17,15 @@ function importLayerPageInit (ctx) { var
>> currentLayerDepSelection; var validLayerName = /^(\w|-)+$/;
>>
>> + /* Catch 'disable' race condition between type-ahead started and
>> "input change" */
>> + var typeAheadStarted = 0;
>> +
>> libtoaster.makeTypeahead(layerDepInput,
>> libtoaster.ctx.layersTypeAheadUrl,
>> { include_added: "true" }, function(item){
>> currentLayerDepSelection = item;
>> layerDepBtn.removeAttr("disabled");
>> + typeAheadStarted = 1;
>> });
>>
>> layerDepInput.on("typeahead:select", function(event, data){
>> @@ -34,7 +38,10 @@ function importLayerPageInit (ctx) {
>> // disable the "Add layer" button when the layer input typeahead
>> is empty // or not in the typeahead choices
>> layerDepInput.on("input change", function(){
>> - layerDepBtn.attr("disabled","disabled");
>> + if (0 == typeAheadStarted) {
>> + layerDepBtn.attr("disabled","disabled");
>> + }
>> + typeAheadStarted = 0;
>> });
>>
>> /* We automatically add "openembedded-core" layer for convenience
>> as a @@ -50,6 +57,7 @@ function importLayerPageInit (ctx) {
>> });
>>
>> layerDepBtn.click(function(){
>> + typeAheadStarted = 0;
>> if (currentLayerDepSelection == undefined)
>> return;
>>
>> @@ -77,7 +85,7 @@ function importLayerPageInit (ctx) {
>>
>> $("#layer-deps-list").append(newLayerDep);
>>
>> -
>> libtoaster.getLayerDepsForProject(currentLayerDepSelection.layerdetailurl,
>> +
>> libtoaster.getLayerDepsForProject(currentLayerDepSelection.xhrLayerUrl,
>> function (data){ /* These are the dependencies of the layer added as
>> a dependency */ if (data.list.length > 0) {
>> diff --git a/bitbake/lib/toaster/toastergui/tablefilter.py
>> b/bitbake/lib/toaster/toastergui/tablefilter.py index
>> 65454e1..ffef795 100644 ---
>> a/bitbake/lib/toaster/toastergui/tablefilter.py +++
>> b/bitbake/lib/toaster/toastergui/tablefilter.py @@ -1,23 +1,10 @@
>> #
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> -#
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2015 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from django.db.models import Q, Max, Min
>> from django.utils import dateparse, timezone
>> diff --git a/bitbake/lib/toaster/toastergui/tables.py
>> b/bitbake/lib/toaster/toastergui/tables.py index 9ff756b..b3ea222
>> 100644 --- a/bitbake/lib/toaster/toastergui/tables.py
>> +++ b/bitbake/lib/toaster/toastergui/tables.py
>> @@ -1,23 +1,10 @@
>> #
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> -#
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2015 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from toastergui.widgets import ToasterTable
>> from orm.models import Recipe, ProjectLayer, Layer_Version, Machine,
>> Project diff --git
>> a/bitbake/lib/toaster/toastergui/templatetags/field_values_filter.py
>> b/bitbake/lib/toaster/toastergui/templatetags/field_values_filter.py
>> index 5a73af7..eb48339 100644 ---
>> a/bitbake/lib/toaster/toastergui/templatetags/field_values_filter.py
>> +++
>> b/bitbake/lib/toaster/toastergui/templatetags/field_values_filter.py
>> @@ -1,3 +1,7 @@ +# +# SPDX-License-Identifier: GPL-2.0-only +#
>> +
>> from django import template
>>
>> register = template.Library()
>> diff --git
>> a/bitbake/lib/toaster/toastergui/templatetags/objects_to_dictionaries_filter.py
>> b/bitbake/lib/toaster/toastergui/templatetags/objects_to_dictionaries_filter.py
>> index 0dcc7d2..048d533 100644 ---
>> a/bitbake/lib/toaster/toastergui/templatetags/objects_to_dictionaries_filter.py
>> +++
>> b/bitbake/lib/toaster/toastergui/templatetags/objects_to_dictionaries_filter.py
>> @@ -1,3 +1,7 @@ +# +# SPDX-License-Identifier: GPL-2.0-only +#
>> +
>> from django import template
>> import json
>>
>> diff --git
>> a/bitbake/lib/toaster/toastergui/templatetags/project_url_tag.py
>> b/bitbake/lib/toaster/toastergui/templatetags/project_url_tag.py
>> index 04770ac..71e0925 100644 ---
>> a/bitbake/lib/toaster/toastergui/templatetags/project_url_tag.py +++
>> b/bitbake/lib/toaster/toastergui/templatetags/project_url_tag.py @@
>> -1,3 +1,7 @@ +# +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> from django import template
>> from django.core.urlresolvers import reverse
>>
>> diff --git
>> a/bitbake/lib/toaster/toastergui/templatetags/projecttags.py
>> b/bitbake/lib/toaster/toastergui/templatetags/projecttags.py index
>> b170a16..1dbab3b 100644 ---
>> a/bitbake/lib/toaster/toastergui/templatetags/projecttags.py +++
>> b/bitbake/lib/toaster/toastergui/templatetags/projecttags.py @@ -1,23
>> +1,10 @@ # -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> -#
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from datetime import datetime, timedelta
>> from os.path import relpath
>> diff --git a/bitbake/lib/toaster/toastergui/typeaheads.py
>> b/bitbake/lib/toaster/toastergui/typeaheads.py index 5aa0f8d..fd750ff
>> 100644 --- a/bitbake/lib/toaster/toastergui/typeaheads.py
>> +++ b/bitbake/lib/toaster/toastergui/typeaheads.py
>> @@ -3,18 +3,8 @@
>> #
>> # Copyright (C) 2015 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import subprocess
>>
>> diff --git a/bitbake/lib/toaster/toastergui/urls.py
>> b/bitbake/lib/toaster/toastergui/urls.py index dc03e30..673d9ae 100644
>> --- a/bitbake/lib/toaster/toastergui/urls.py
>> +++ b/bitbake/lib/toaster/toastergui/urls.py
>> @@ -3,18 +3,8 @@
>> #
>> # Copyright (C) 2013-2017 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from django.conf.urls import include, url
>> from django.views.generic import RedirectView, TemplateView
>> diff --git a/bitbake/lib/toaster/toastergui/views.py
>> b/bitbake/lib/toaster/toastergui/views.py index c712b06..d7acaff
>> 100644 --- a/bitbake/lib/toaster/toastergui/views.py
>> +++ b/bitbake/lib/toaster/toastergui/views.py
>> @@ -1,24 +1,10 @@
>> #
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> -#
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. -
>>
>> import re
>>
>> diff --git a/bitbake/lib/toaster/toastergui/widgets.py
>> b/bitbake/lib/toaster/toastergui/widgets.py index db5c3aa..645f458
>> 100644 --- a/bitbake/lib/toaster/toastergui/widgets.py
>> +++ b/bitbake/lib/toaster/toastergui/widgets.py
>> @@ -1,23 +1,10 @@
>> #
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> -#
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2015 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from django.views.generic import View, TemplateView
>> from django.views.decorators.cache import cache_control
>> diff --git
>> a/bitbake/lib/toaster/toastermain/management/commands/builddelete.py
>> b/bitbake/lib/toaster/toastermain/management/commands/builddelete.py
>> index bf69a8f..c2d773a 100644 ---
>> a/bitbake/lib/toaster/toastermain/management/commands/builddelete.py
>> +++
>> b/bitbake/lib/toaster/toastermain/management/commands/builddelete.py
>> @@ -1,3 +1,7 @@ +# +# SPDX-License-Identifier: GPL-2.0-only +#
>> +
>> from django.core.management.base import BaseCommand, CommandError
>> from django.core.exceptions import ObjectDoesNotExist
>> from orm.models import Build
>> diff --git
>> a/bitbake/lib/toaster/toastermain/management/commands/buildimport.py
>> b/bitbake/lib/toaster/toastermain/management/commands/buildimport.py
>> index 2d57ab5..408ad44 100644 ---
>> a/bitbake/lib/toaster/toastermain/management/commands/buildimport.py
>> +++
>> b/bitbake/lib/toaster/toastermain/management/commands/buildimport.py
>> @@ -1,23 +1,10 @@ # -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4;
>> c-basic-offset: 4; indent-tabs-mode: nil -*- -#
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2018 Wind River Systems
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> # buildimport: import a project for project specific configuration
>> #
>> diff --git
>> a/bitbake/lib/toaster/toastermain/management/commands/buildslist.py
>> b/bitbake/lib/toaster/toastermain/management/commands/buildslist.py
>> index 70b5812..1ed2022 100644 ---
>> a/bitbake/lib/toaster/toastermain/management/commands/buildslist.py
>> +++
>> b/bitbake/lib/toaster/toastermain/management/commands/buildslist.py
>> @@ -1,3 +1,7 @@ +# +# SPDX-License-Identifier: GPL-2.0-only +#
>> +
>> from django.core.management.base import BaseCommand, CommandError
>> from orm.models import Build
>> import os
>> diff --git
>> a/bitbake/lib/toaster/toastermain/management/commands/checksocket.py
>> b/bitbake/lib/toaster/toastermain/management/commands/checksocket.py
>> index 0399b86..811fd5d 100644 ---
>> a/bitbake/lib/toaster/toastermain/management/commands/checksocket.py
>> +++
>> b/bitbake/lib/toaster/toastermain/management/commands/checksocket.py
>> @@ -1,23 +1,11 @@ -#!/usr/bin/env python -# ex:ts=4:sw=4:sts=4:et -#
>> -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#!/usr/bin/env python3 #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2015 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> """Custom management command checksocket."""
>>
>> diff --git
>> a/bitbake/lib/toaster/toastermain/management/commands/perf.py
>> b/bitbake/lib/toaster/toastermain/management/commands/perf.py index
>> 6b450bb..7d629fb 100644 ---
>> a/bitbake/lib/toaster/toastermain/management/commands/perf.py +++
>> b/bitbake/lib/toaster/toastermain/management/commands/perf.py @@ -1,3
>> +1,7 @@ +# +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> from django.core.management.base import BaseCommand
>> from django.test.client import Client
>> import os, sys, re
>> diff --git a/bitbake/lib/toaster/toastermain/settings.py
>> b/bitbake/lib/toaster/toastermain/settings.py index 13541d3..74501fa
>> 100644 --- a/bitbake/lib/toaster/toastermain/settings.py
>> +++ b/bitbake/lib/toaster/toastermain/settings.py
>> @@ -1,23 +1,10 @@
>> #
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> -#
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> # Django settings for Toaster project.
>>
>> diff --git
>> a/bitbake/lib/toaster/toastermain/settings_production_example.py
>> b/bitbake/lib/toaster/toastermain/settings_production_example.py
>> index 61a2888..6cd0f52 100644 ---
>> a/bitbake/lib/toaster/toastermain/settings_production_example.py +++
>> b/bitbake/lib/toaster/toastermain/settings_production_example.py @@
>> -1,23 +1,10 @@ # -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> -#
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> # See Django documentation for more information about deployment
>> # https://docs.djangoproject.com/en/1.8/howto/deployment/checklist/
>> diff --git a/bitbake/lib/toaster/toastermain/settings_test.py
>> b/bitbake/lib/toaster/toastermain/settings_test.py index
>> a322711..6538d9e 100644 ---
>> a/bitbake/lib/toaster/toastermain/settings_test.py +++
>> b/bitbake/lib/toaster/toastermain/settings_test.py @@ -1,23 +1,10 @@
>> #
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> -#
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> # Django settings for Toaster project.
>>
>> diff --git a/bitbake/lib/toaster/toastermain/urls.py
>> b/bitbake/lib/toaster/toastermain/urls.py index e2fb0ae..ac77bc3
>> 100644 --- a/bitbake/lib/toaster/toastermain/urls.py
>> +++ b/bitbake/lib/toaster/toastermain/urls.py
>> @@ -1,23 +1,10 @@
>> #
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> -#
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from django.conf.urls import include, url
>> from django.views.generic import RedirectView, TemplateView
>> diff --git a/bitbake/lib/toaster/toastermain/wsgi.py
>> b/bitbake/lib/toaster/toastermain/wsgi.py index 031b314..4c31283
>> 100644 --- a/bitbake/lib/toaster/toastermain/wsgi.py
>> +++ b/bitbake/lib/toaster/toastermain/wsgi.py
>> @@ -1,7 +1,8 @@
>> -"""
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> +"""
>> WSGI config for Toaster project.
>>
>> This module contains the WSGI application used by Django's
>> development server diff --git a/doc/user_manual.md
>> b/doc/user_manual.md index c2657da..96aa2e3 100644
>> --- a/doc/user_manual.md
>> +++ b/doc/user_manual.md
>> @@ -135,8 +135,8 @@ DISTRO_ARCH ??= "armhf"
>> Then, call `bitbake` with image names, e.g.:
>>
>> ```
>> -bitbake multiconfig:qemuarm-buster:isar-image-base \
>> - multiconfig:qemuarm-buster:isar-image-debug
>> +bitbake mc:qemuarm-buster:isar-image-base \
>> + mc:qemuarm-buster:isar-image-debug
>> ```
>>
>> The following images are created:
>> @@ -169,14 +169,14 @@ The following command will produce
>> `isar-image-base` images for all targets:
>> ```
>> $ bitbake \
>> - multiconfig:qemuarm-stretch:isar-image-base \
>> - multiconfig:qemuarm-buster:isar-image-base \
>> - multiconfig:qemuarm64-stretch:isar-image-base \
>> - multiconfig:qemui386-stretch:isar-image-base \
>> - multiconfig:qemui386-buster:isar-image-base \
>> - multiconfig:qemuamd64-stretch:isar-image-base \
>> - multiconfig:qemuamd64-buster:isar-image-base \
>> - multiconfig:rpi-stretch:isar-image-base
>> + mc:qemuarm-stretch:isar-image-base \
>> + mc:qemuarm-buster:isar-image-base \
>> + mc:qemuarm64-stretch:isar-image-base \
>> + mc:qemui386-stretch:isar-image-base \
>> + mc:qemui386-buster:isar-image-base \
>> + mc:qemuamd64-stretch:isar-image-base \
>> + mc:qemuamd64-buster:isar-image-base \
>> + mc:rpi-stretch:isar-image-base
>> ```
>>
>> Created images are:
>> @@ -197,9 +197,9 @@ tmp/deploy/images/rpi/isar-image-base.rpi-sdimg
>> A bootable disk image is generated if you set IMAGE_TYPE to
>> 'wic-img'. Behind the scenes a tool called `wic` is used to assemble
>> the images. It is controlled by a `.wks` file which you can choose
>> with changing WKS_FILE. Some examples in the tree use that feature
>> already. ``` # Generate an image for the `i386` target architecture
>> - $ bitbake multiconfig:qemui386-buster:isar-image-base
>> + $ bitbake mc:qemui386-buster:isar-image-base
>> # Similarly, for the `amd64` target architecture, in this case EFI
>> - $ bitbake multiconfig:qemuamd64-buster:isar-image-base
>> + $ bitbake mc:qemuamd64-buster:isar-image-base
>> ```
>>
>> Variables may be used in `.wks.in` files; Isar will expand them and
>> generate a regular `.wks` file before generating the disk image using
>> `wic`. @@ -303,7 +303,7 @@ following variables define the default
>> configuration to build for:
>> - `DISTRO_ARCH` - The Debian architecture to build for (e.g.,
>> `armhf`).
>> If BitBake is called with multiconfig targets (e.g.,
>> -`multiconfig:qemuarm-buster:isar-image-base`), the following
>> variable defines +`mc:qemuarm-buster:isar-image-base`), the following
>> variable defines all supported configurations:
>>
>> - `BBMULTICONFIG` - The list of the complete configuration
>> definition files. @@ -687,7 +687,7 @@ Debian cross-compilation works
>> out of the box starting from Debian stretch distr Just like
>> OpenEmbedded, Isar supports a devshell target for all dpkg package
>> recipes. This target opens a terminal inside the buildchroot that
>> runs the package build. To invoke it, just call -`bitbake
>> multiconfig:${MACHINE}-${DISTRO}:<package_name> -c devshell`.
>> +`bitbake mc:${MACHINE}-${DISTRO}:<package_name> -c devshell`.
>>
>> ## Create an ISAR SDK root filesystem
>> @@ -705,7 +705,7 @@ target binary artifacts. Developer chroots to sdk
>> rootfs and develops applicatio ### Solution
>>
>> User manually triggers creation of SDK root filesystem for his
>> target platform by launching the task `do_populate_sdk` for target
>> image, f.e. -`bitbake -c do_populate_sdk
>> multiconfig:${MACHINE}-${DISTRO}:isar-image-base`. +`bitbake -c
>> do_populate_sdk mc:${MACHINE}-${DISTRO}:isar-image-base`. The
>> resulting SDK rootfs is archived into
>> `tmp/deploy/images/${MACHINE}/sdk-${DISTRO}-${DISTRO_ARCH}.tar.xz`.
>> It is additionally available for direct use under
>> `tmp/deploy/images/${MACHINE}/sdk-${DISTRO}-${DISTRO_ARCH}/`. @@
>> -717,7 +717,7 @@ One may chroot into the SDK and install required
>> target packages with the help o
>> - Trigger creation of SDK root filesystem
>>
>> ```
>> -bitbake -c do_populate_sdk multiconfig:qemuarm-buster:isar-image-base
>> +bitbake -c do_populate_sdk mc:qemuarm-buster:isar-image-base
>> ```
>>
>> - Mount the following directories in chroot by passing resulting
>> rootfs as an argument to the script `mount_chroot.sh`: @@ -801,7
>> +801,7 @@ BASE_REPO_KEY =
>> "file://<absolute_path_to_your_pub_key_file>"'
>> - Trigger creation of local apt caching Debian packages during
>> image generation.
>> ```
>> -bitbake -c cache_base_repo multiconfig:qemuarm-buster:isar-image-base
>> +bitbake -c cache_base_repo mc:qemuarm-buster:isar-image-base
>> ```
>>
>> - Set `ISAR_USE_CACHED_BASE_REPO` in `conf/local.conf`:
>> @@ -820,7 +820,7 @@ sudo rm -rf tmp
>> - Trigger again generation of image (now using local caching repo):
>>
>> ```
>> -bitbake multiconfig:qemuarm-buster:isar-image-base
>> +bitbake mc:qemuarm-buster:isar-image-base
>> ```
>>
>> ### Limitation
>> @@ -860,5 +860,5 @@ DISTRO_APT_SOURCES_append = "
>> conf/distro/docker-buster.list" And build the corresponding image
>> target:
>> ```
>> -bitbake multiconfig:qemuarm64-buster:isar-image-base
>> +bitbake mc:qemuarm64-buster:isar-image-base
>> ```
>> diff --git a/meta-isar/conf/conf-notes.txt
>> b/meta-isar/conf/conf-notes.txt index 74df97c..c557cd3 100644
>> --- a/meta-isar/conf/conf-notes.txt
>> +++ b/meta-isar/conf/conf-notes.txt
>> @@ -1,4 +1,4 @@
>> Common targets are:
>> - multiconfig:qemuarm-buster:isar-image-base
>> - multiconfig:qemuamd64-buster:isar-image-base
>> - multiconfig:rpi-stretch:isar-image-base
>> + mc:qemuarm-buster:isar-image-base
>> + mc:qemuamd64-buster:isar-image-base
>> + mc:rpi-stretch:isar-image-base
>> diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
>> index 713d1c2..a8556a6 100755
>> --- a/scripts/ci_build.sh
>> +++ b/scripts/ci_build.sh
>> @@ -21,40 +21,40 @@ BUILD_DIR=./build
>> BB_ARGS="-v"
>>
>> TARGETS_SET="\
>> - multiconfig:qemuarm-stretch:isar-image-base \
>> - multiconfig:qemuarm-buster:isar-image-base \
>> - multiconfig:qemuarm-bullseye:isar-image-base \
>> - multiconfig:qemuarm64-stretch:isar-image-base \
>> - multiconfig:qemui386-stretch:isar-image-base \
>> - multiconfig:qemui386-buster:isar-image-base \
>> - multiconfig:qemui386-bullseye:isar-image-base \
>> - multiconfig:qemuamd64-stretch:isar-image-base \
>> - multiconfig:qemuamd64-buster:isar-image-base \
>> - multiconfig:qemuamd64-buster-tgz:isar-image-base \
>> - multiconfig:qemuamd64-bullseye:isar-image-base \
>> - multiconfig:qemumipsel-stretch:isar-image-base \
>> - multiconfig:qemumipsel-buster:isar-image-base \
>> - multiconfig:qemumipsel-bullseye:isar-image-base \
>> - multiconfig:nand-ubi-demo-buster:isar-image-ubi \
>> - multiconfig:rpi-stretch:isar-image-base"
>> + mc:qemuarm-stretch:isar-image-base \
>> + mc:qemuarm-buster:isar-image-base \
>> + mc:qemuarm-bullseye:isar-image-base \
>> + mc:qemuarm64-stretch:isar-image-base \
>> + mc:qemui386-stretch:isar-image-base \
>> + mc:qemui386-buster:isar-image-base \
>> + mc:qemui386-bullseye:isar-image-base \
>> + mc:qemuamd64-stretch:isar-image-base \
>> + mc:qemuamd64-buster:isar-image-base \
>> + mc:qemuamd64-buster-tgz:isar-image-base \
>> + mc:qemuamd64-bullseye:isar-image-base \
>> + mc:qemumipsel-stretch:isar-image-base \
>> + mc:qemumipsel-buster:isar-image-base \
>> + mc:qemumipsel-bullseye:isar-image-base \
>> + mc:nand-ubi-demo-buster:isar-image-ubi \
>> + mc:rpi-stretch:isar-image-base"
>> # qemu-user-static of <= buster too old to build that
>> - # multiconfig:qemuarm64-buster:isar-image-base
>> - # multiconfig:qemuarm64-bullseye:isar-image-base
>> + # mc:qemuarm64-buster:isar-image-base
>> + # mc:qemuarm64-bullseye:isar-image-base
>>
>> CROSS_TARGETS_SET="\
>> - multiconfig:qemuarm-stretch:isar-image-base \
>> - multiconfig:qemuarm-buster:isar-image-base \
>> - multiconfig:qemuarm-bullseye:isar-image-base \
>> - multiconfig:qemuarm64-stretch:isar-image-base \
>> - multiconfig:qemuamd64-stretch:isar-image-base \
>> - multiconfig:de0-nano-soc-stretch:isar-image-base \
>> - multiconfig:rpi-stretch:isar-image-base"
>> + mc:qemuarm-stretch:isar-image-base \
>> + mc:qemuarm-buster:isar-image-base \
>> + mc:qemuarm-bullseye:isar-image-base \
>> + mc:qemuarm64-stretch:isar-image-base \
>> + mc:qemuamd64-stretch:isar-image-base \
>> + mc:de0-nano-soc-stretch:isar-image-base \
>> + mc:rpi-stretch:isar-image-base"
>>
>> REPRO_TARGETS_SET="\
>> - multiconfig:qemuarm-stretch:isar-image-base \
>> - multiconfig:qemuarm64-stretch:isar-image-base \
>> - multiconfig:qemuamd64-stretch:isar-image-base \
>> - multiconfig:qemuarm-buster:isar-image-base"
>> + mc:qemuarm-stretch:isar-image-base \
>> + mc:qemuarm64-stretch:isar-image-base \
>> + mc:qemuamd64-stretch:isar-image-base \
>> + mc:qemuarm-buster:isar-image-base"
>>
>>
>> show_help() {
>> @@ -162,7 +162,7 @@ sed -i -e 's/ISAR_CROSS_COMPILE ?=
>> "0"/ISAR_CROSS_COMPILE ?= "1"/g' conf/local.c bitbake $BB_ARGS
>> $CROSS_TARGETS_SET while [ -e bitbake.sock ]; do sleep 1; done
>> # In addition test SDK creation
>> -bitbake $BB_ARGS -c do_populate_sdk
>> multiconfig:qemuarm-stretch:isar-image-base +bitbake $BB_ARGS -c
>> do_populate_sdk mc:qemuarm-stretch:isar-image-base while [ -e
>> bitbake.sock ]; do sleep 1; done
>> if [ -z "$FAST_BUILD" ]; then
>> @@ -175,6 +175,6 @@ fi
>> cp -a "${ISARROOT}/meta/classes/dpkg-base.bbclass"
>> "${ISARROOT}/meta/classes/dpkg-base.bbclass.ci-backup" echo -e
>> "do_fetch_append() {\n\n}" >>
>> "${ISARROOT}/meta/classes/dpkg-base.bbclass" -bitbake $BB_ARGS
>> multiconfig:qemuamd64-stretch:isar-image-base +bitbake $BB_ARGS
>> mc:qemuamd64-stretch:isar-image-base
>> mv "${ISARROOT}/meta/classes/dpkg-base.bbclass.ci-backup"
>> "${ISARROOT}/meta/classes/dpkg-base.bbclass" diff --git
>> a/scripts/start_vm b/scripts/start_vm index 9f07a03..71b55f2 100755
>> --- a/scripts/start_vm
>> +++ b/scripts/start_vm
>> @@ -101,7 +101,7 @@ do
>> shift
>> done
>>
>> -eval $(bitbake -e multiconfig:qemu$ARCH-$DISTRO:isar-image-base |
>> grep "^DEPLOY_DIR_IMAGE=") +eval $(bitbake -e
>> mc:qemu$ARCH-$DISTRO:isar-image-base | grep "^DEPLOY_DIR_IMAGE=")
>> readonly IMAGE_DIR=$DEPLOY_DIR_IMAGE
>> readonly ISARROOT="$(dirname "$0")"/..
>> @@ -112,13 +112,13 @@ eval "$(egrep 'MACHINE_SERIAL' $MACHINE_CONF
>> |bb2sh)" readonly
>> CONFIG_CONF=$ISARROOT/meta-isar/conf/multiconfig/qemu$ARCH-$DISTRO.conf
>> eval "$(egrep 'QEMU_' $CONFIG_CONF |bb2sh)"
>> -eval $(bitbake -e multiconfig:qemu$ARCH-$DISTRO:isar-image-base |
>> grep "^IMAGE_TYPE=") +eval $(bitbake -e
>> mc:qemu$ARCH-$DISTRO:isar-image-base | grep "^IMAGE_TYPE=") case
>> "$IMAGE_TYPE" in ext4-img)
>> readonly
>> ROOTFS_IMAGE=isar-image-base-debian-$DISTRO-qemu$ARCH.ext4.img
>> - eval $(bitbake -e multiconfig:qemu$ARCH-$DISTRO:isar-image-base
>> | grep "^KERNEL_IMAGE=")
>> - eval $(bitbake -e multiconfig:qemu$ARCH-$DISTRO:isar-image-base
>> | grep "^INITRD_IMAGE=")
>> + eval $(bitbake -e mc:qemu$ARCH-$DISTRO:isar-image-base | grep
>> "^KERNEL_IMAGE=")
>> + eval $(bitbake -e mc:qemu$ARCH-$DISTRO:isar-image-base | grep
>> "^INITRD_IMAGE=") QKERNEL=$IMAGE_DIR/${KERNEL_IMAGE}
>> QINITRD=/dev/null
>> [ -n "$INITRD_IMAGE" ] && QINITRD=$IMAGE_DIR/${INITRD_IMAGE}
>> diff --git a/testsuite/build_test/build_test.py
>> b/testsuite/build_test/build_test.py index 4220d3a..7a55c2f 100644
>> --- a/testsuite/build_test/build_test.py
>> +++ b/testsuite/build_test/build_test.py
>> @@ -22,7 +22,7 @@ class BuildTest(Test):
>>
>> #isar_root = dirname(__file__) + '/..'
>> os.chdir(build_dir)
>> - cmdline = ['bitbake', 'multiconfig:qemu' + arch + '-' +
>> distro + ':isar-image-base']
>> + cmdline = ['bitbake', 'mc:qemu' + arch + '-' + distro +
>> ':isar-image-base'] p1 = subprocess32.run(cmdline)
>>
>> if p1.returncode:
>> diff --git a/testsuite/start_vm.py b/testsuite/start_vm.py
>> index 02a4b51..a3e32ac 100755
>> --- a/testsuite/start_vm.py
>> +++ b/testsuite/start_vm.py
>> @@ -10,7 +10,7 @@ import sys
>> import time
>>
>> def get_bitbake_env(arch, distro):
>> - multiconfig = 'multiconfig:qemu' + arch + '-' + distro +
>> ':isar-image-base'
>> + multiconfig = 'mc:qemu' + arch + '-' + distro +
>> ':isar-image-base' output = subprocess.check_output(['bitbake', '-e',
>> str(multiconfig)]) return output
>>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 1/1] bitbake: update to version 1.44.0
2019-10-22 7:41 ` Cedric Hombourger
@ 2019-10-22 15:56 ` Baurzhan Ismagulov
2019-10-24 6:58 ` chombourger
0 siblings, 1 reply; 11+ messages in thread
From: Baurzhan Ismagulov @ 2019-10-22 15:56 UTC (permalink / raw)
To: isar-users
On Tue, Oct 22, 2019 at 09:41:59AM +0200, Cedric Hombourger wrote:
> I have debated (with myself :)) and I agree that there are pros and cons
> with each approach.
> I ended up submitting a single commit at the end of the day to support
> people needing to "git bisect".
> I have otherwise no issues resubmitting the changes if we prefer to have the
> bitbake upgrade clearly separated from the Isar changes (either way is fine
> with me - @maintainers just let me know what your preference is!)
Since the bitbake commit breaks stuff, I have to prefer :) a single commit. I
agree it is ugly.
With kind regards,
Baurzhan.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 1/1] bitbake: update to version 1.44.0
2019-10-22 15:56 ` Baurzhan Ismagulov
@ 2019-10-24 6:58 ` chombourger
2019-10-24 19:44 ` Baurzhan Ismagulov
0 siblings, 1 reply; 11+ messages in thread
From: chombourger @ 2019-10-24 6:58 UTC (permalink / raw)
To: isar-users
[-- Attachment #1.1: Type: text/plain, Size: 808 bytes --]
Thanks Baurzhan
Is this patch OK to merge then?
Cedric
On Tuesday, October 22, 2019 at 5:56:46 PM UTC+2, Baurzhan Ismagulov wrote:
>
> On Tue, Oct 22, 2019 at 09:41:59AM +0200, Cedric Hombourger wrote:
> > I have debated (with myself :)) and I agree that there are pros and cons
> > with each approach.
> > I ended up submitting a single commit at the end of the day to support
> > people needing to "git bisect".
> > I have otherwise no issues resubmitting the changes if we prefer to have
> the
> > bitbake upgrade clearly separated from the Isar changes (either way is
> fine
> > with me - @maintainers just let me know what your preference is!)
>
> Since the bitbake commit breaks stuff, I have to prefer :) a single
> commit. I
> agree it is ugly.
>
> With kind regards,
> Baurzhan.
>
[-- Attachment #1.2: Type: text/html, Size: 1047 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 1/1] bitbake: update to version 1.44.0
2019-10-24 6:58 ` chombourger
@ 2019-10-24 19:44 ` Baurzhan Ismagulov
2019-11-05 12:36 ` chombourger
0 siblings, 1 reply; 11+ messages in thread
From: Baurzhan Ismagulov @ 2019-10-24 19:44 UTC (permalink / raw)
To: isar-users
On Wed, Oct 23, 2019 at 11:58:06PM -0700, chombourger@gmail.com wrote:
> Is this patch OK to merge then?
I'd like to run it through the CI and skim through it. I'll let you know.
With kind regards,
Baurzhan.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 1/1] bitbake: update to version 1.44.0
2019-10-24 19:44 ` Baurzhan Ismagulov
@ 2019-11-05 12:36 ` chombourger
2019-11-06 6:01 ` Baurzhan Ismagulov
0 siblings, 1 reply; 11+ messages in thread
From: chombourger @ 2019-11-05 12:36 UTC (permalink / raw)
To: isar-users
[-- Attachment #1.1: Type: text/plain, Size: 532 bytes --]
Hi Baurzhan/maintainers
Any luck?
I have rebased this patch locally, had some trivial conflicts with the
RECIPE-API-CHANGELOG.md file.
Let me know if I should send you the rebased patch?
Thank you
Cedric
On Thursday, October 24, 2019 at 9:44:34 PM UTC+2, Baurzhan Ismagulov wrote:
>
> On Wed, Oct 23, 2019 at 11:58:06PM -0700, chomb...@gmail.com <javascript:>
> wrote:
> > Is this patch OK to merge then?
>
> I'd like to run it through the CI and skim through it. I'll let you know.
>
> With kind regards,
> Baurzhan.
>
[-- Attachment #1.2: Type: text/html, Size: 988 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 1/1] bitbake: update to version 1.44.0
2019-11-05 12:36 ` chombourger
@ 2019-11-06 6:01 ` Baurzhan Ismagulov
2019-11-06 6:02 ` Cedric Hombourger
0 siblings, 1 reply; 11+ messages in thread
From: Baurzhan Ismagulov @ 2019-11-06 6:01 UTC (permalink / raw)
To: isar-users
On Tue, Nov 05, 2019 at 04:36:58AM -0800, chombourger@gmail.com wrote:
> I have rebased this patch locally, had some trivial conflicts with the
> RECIPE-API-CHANGELOG.md file.
> Let me know if I should send you the rebased patch?
Thanks, no need to resend. Applied to next.
With kind regards,
Baurzhan.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 1/1] bitbake: update to version 1.44.0
2019-11-06 6:01 ` Baurzhan Ismagulov
@ 2019-11-06 6:02 ` Cedric Hombourger
0 siblings, 0 replies; 11+ messages in thread
From: Cedric Hombourger @ 2019-11-06 6:02 UTC (permalink / raw)
To: Baurzhan Ismagulov; +Cc: isar-users
> On Nov 6, 2019, at 7:01 AM, Baurzhan Ismagulov <ibr@radix50.net> wrote:
>
> On Tue, Nov 05, 2019 at 04:36:58AM -0800, chombourger@gmail.com wrote:
>> I have rebased this patch locally, had some trivial conflicts with the
>> RECIPE-API-CHANGELOG.md file.
>> Let me know if I should send you the rebased patch?
>
> Thanks, no need to resend. Applied to next.
Awesome, thank you!
>
> With kind regards,
> Baurzhan.
>
> --
> You received this message because you are subscribed to a topic in the Google Groups "isar-users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/isar-users/xXs46cNcceM/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to isar-users+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20191106060112.iyq6f4o2lypbg5u5%40yssyq.m.ilbers.de.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 0/1] update bitbake to version 1.44.0
2019-10-21 14:19 [PATCH 0/1] update bitbake to version 1.44.0 Cedric Hombourger
2019-10-21 14:19 ` [PATCH 1/1] bitbake: update " Cedric Hombourger
@ 2019-11-08 8:05 ` Jan Kiszka
1 sibling, 0 replies; 11+ messages in thread
From: Jan Kiszka @ 2019-11-08 8:05 UTC (permalink / raw)
To: Cedric Hombourger, isar-users
On 21.10.19 16:19, Cedric Hombourger wrote:
> It appears that the last update of Isar we did was last year. I would like to propose that
> we re-align ourselves with upstream (this would fix some of the gitsm fetcher issues I have
> been facing). No changes to bitbake were required, they however changed their API and
> bitbake multiconfig:foo:bar no longer works (bitbake mc:foo:bar should be used instead).
> Documentation and code using the old syntax was changed. The RECIPE-API-CHANGELOG document
> was also changed. All changes were tested with our ci_build.sh script (no issues or failures
> were observed).
Just in case others run into this:
While the official syntax is now "mc:", bitbake still stupports
"multiconfig:" as compat solution. Well, except you actually provide
multiple configs: Only the first entry seems to be properly converted to
"mc:" internally, and all other targets will fail the build...
So, we need to switch completely in downstream multiconfig projects as
well (so far I only know of jailhouse-images). The required kas change
is submitted, will require a new release there, though.
Jan
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2019-11-08 8:05 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-10-21 14:19 [PATCH 0/1] update bitbake to version 1.44.0 Cedric Hombourger
2019-10-21 14:19 ` [PATCH 1/1] bitbake: update " Cedric Hombourger
2019-10-22 7:36 ` Henning Schild
2019-10-22 7:41 ` Cedric Hombourger
2019-10-22 15:56 ` Baurzhan Ismagulov
2019-10-24 6:58 ` chombourger
2019-10-24 19:44 ` Baurzhan Ismagulov
2019-11-05 12:36 ` chombourger
2019-11-06 6:01 ` Baurzhan Ismagulov
2019-11-06 6:02 ` Cedric Hombourger
2019-11-08 8:05 ` [PATCH 0/1] update bitbake " Jan Kiszka
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox