From: Henning Schild <henning.schild@siemens.com>
To: Cedric Hombourger <Cedric_Hombourger@mentor.com>
Cc: <isar-users@googlegroups.com>
Subject: Re: [PATCH 1/1] bitbake: update to version 1.44.0
Date: Tue, 22 Oct 2019 09:36:09 +0200 [thread overview]
Message-ID: <20191022093609.5ed343a5@md1za8fc.ad001.siemens.net> (raw)
In-Reply-To: <1571667598-657-2-git-send-email-Cedric_Hombourger@mentor.com>
Hi Cedric,
i would suggest a split between the actual bump and the following
Isar-changes. But i do not care too much and both ways have their pros
and cons.
Henning
Am Mon, 21 Oct 2019 16:19:58 +0200
schrieb Cedric Hombourger <Cedric_Hombourger@mentor.com>:
> Found that the gitsm fetcher we ship is rather out-dated compared to
> upstream and was failing to pull git sub-modules from repositories
> such as azure-umqtt-c.
>
> It shall be noted that bitbake has renamed its multiconfig targets:
> the "multiconfig:" prefix was changed to "mc:". Updated our scripts
> as well as documentation.
>
> Origin: https://github.com/openembedded/bitbake.git
> Commit 5d83d828cacb58ccb7c464e799c85fd2d2a50ccc (tag: 1.44.0)
> Author: Richard Purdie <richard.purdie@linuxfoundation.org>
> Date: Wed Oct 9 14:10:21 2019 +0100
>
> bitbake: Update to version 1.44.0
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>
> Signed-off-by: Cedric Hombourger <Cedric_Hombourger@mentor.com>
> ---
> RECIPE-API-CHANGELOG.md | 16 +
> bitbake/.gitattributes | 2 +
> bitbake/HEADER | 19 -
> bitbake/LICENSE | 12 +-
> bitbake/{COPYING => LICENSE.GPL-2.0-only} | 61 +-
> bitbake/LICENSE.MIT | 25 +
> bitbake/MANIFEST.in | 5 +-
> bitbake/bin/bitbake | 16 +-
> bitbake/bin/bitbake-diffsigs | 155 +-
> bitbake/bin/bitbake-dumpsig | 95 +-
> bitbake/bin/bitbake-hashclient | 170 ++
> bitbake/bin/bitbake-hashserv | 62 +
> bitbake/bin/bitbake-layers | 12 +-
> bitbake/bin/bitbake-prserv | 4 +
> bitbake/bin/bitbake-selftest | 16 +-
> bitbake/bin/bitbake-worker | 19 +-
> bitbake/bin/bitdoc | 14 +-
> bitbake/bin/git-make-shallow | 4 +
> bitbake/bin/toaster | 12 +-
> bitbake/bin/toaster-eventreplay | 17 +-
> bitbake/classes/base.bbclass | 2 +-
> bitbake/contrib/dump_cache.py | 2 -
> .../bitbake-user-manual-execution.xml | 78 +-
> .../bitbake-user-manual-fetching.xml | 69 +-
> .../bitbake-user-manual-hello.xml | 28 +-
> .../bitbake-user-manual-intro.xml | 18 +-
> .../bitbake-user-manual-metadata.xml | 179 +-
> .../bitbake-user-manual-ref-variables.xml | 422 ++---
> bitbake/doc/poky.ent | 8 -
> bitbake/lib/bb/COW.py | 15 -
> bitbake/lib/bb/__init__.py | 16 +-
> bitbake/lib/bb/build.py | 151 +-
> bitbake/lib/bb/cache.py | 109 +-
> bitbake/lib/bb/cache_extra.py | 14 +-
> bitbake/lib/bb/checksum.py | 12 +-
> bitbake/lib/bb/codeparser.py | 13 +-
> bitbake/lib/bb/command.py | 12 +-
> bitbake/lib/bb/compat.py | 4 +
> bitbake/lib/bb/cooker.py | 218 +--
> bitbake/lib/bb/cookerdata.py | 44 +-
> bitbake/lib/bb/daemonize.py | 4 +
> bitbake/lib/bb/data.py | 23 +-
> bitbake/lib/bb/data_smart.py | 25 +-
> bitbake/lib/bb/event.py | 33 +-
> bitbake/lib/bb/exceptions.py | 3 +
> bitbake/lib/bb/fetch2/__init__.py | 55 +-
> bitbake/lib/bb/fetch2/bzr.py | 12 +-
> bitbake/lib/bb/fetch2/clearcase.py | 19 +-
> bitbake/lib/bb/fetch2/cvs.py | 17 +-
> bitbake/lib/bb/fetch2/git.py | 68 +-
> bitbake/lib/bb/fetch2/gitannex.py | 14 +-
> bitbake/lib/bb/fetch2/gitsm.py | 281 ++-
> bitbake/lib/bb/fetch2/hg.py | 18 +-
> bitbake/lib/bb/fetch2/local.py | 16 +-
> bitbake/lib/bb/fetch2/npm.py | 69 +-
> bitbake/lib/bb/fetch2/osc.py | 5 +-
> bitbake/lib/bb/fetch2/perforce.py | 15 +-
> bitbake/lib/bb/fetch2/repo.py | 16 +-
> bitbake/lib/bb/fetch2/s3.py | 15 +-
> bitbake/lib/bb/fetch2/sftp.py | 15 +-
> bitbake/lib/bb/fetch2/ssh.py | 14 +-
> bitbake/lib/bb/fetch2/svn.py | 98 +-
> bitbake/lib/bb/fetch2/wget.py | 104 +-
> bitbake/lib/bb/main.py | 41 +-
> bitbake/lib/bb/methodpool.py | 15 +-
> bitbake/lib/bb/monitordisk.py | 27 +-
> bitbake/lib/bb/msg.py | 14 +-
> bitbake/lib/bb/namedtuple_with_abc.py | 4 +-
> bitbake/lib/bb/parse/__init__.py | 14 +-
> bitbake/lib/bb/parse/ast.py | 15 +-
> bitbake/lib/bb/parse/parse_py/BBHandler.py | 59 +-
> bitbake/lib/bb/parse/parse_py/ConfHandler.py | 17 +-
> bitbake/lib/bb/parse/parse_py/__init__.py | 17 +-
> bitbake/lib/bb/persist_data.py | 234 ++-
> bitbake/lib/bb/process.py | 4 +
> bitbake/lib/bb/progress.py | 28 +-
> bitbake/lib/bb/providers.py | 18 +-
> bitbake/lib/bb/pysh/builtin.py | 710 --------
> bitbake/lib/bb/pysh/interp.py | 1367 ---------------
> bitbake/lib/bb/pysh/lsprof.py | 116 --
> bitbake/lib/bb/pysh/pysh.py | 167 --
> bitbake/lib/bb/pysh/pyshlex.py | 5 -
> bitbake/lib/bb/pysh/pyshyacc.py | 17 +-
> bitbake/lib/bb/pysh/sherrors.py | 26 -
> bitbake/lib/bb/pysh/subprocess_fix.py | 77 -
> bitbake/lib/bb/remotedata.py | 12 +-
> bitbake/lib/bb/runqueue.py | 1523
> ++++++++++------- bitbake/lib/bb/server/__init__.py |
> 14 +- bitbake/lib/bb/server/process.py | 123 +-
> bitbake/lib/bb/server/xmlrpcclient.py | 12 +-
> bitbake/lib/bb/server/xmlrpcserver.py | 12 +-
> bitbake/lib/bb/siggen.py | 350 +++-
> bitbake/lib/bb/taskdata.py | 17 +-
> bitbake/lib/bb/tests/codeparser.py | 22 +-
> bitbake/lib/bb/tests/cooker.py | 15 +-
> bitbake/lib/bb/tests/cow.py | 17 +-
> bitbake/lib/bb/tests/data.py | 39 +-
> bitbake/lib/bb/tests/event.py | 23 +-
> bitbake/lib/bb/tests/fetch.py | 308 +++-
> bitbake/lib/bb/tests/parse.py | 33 +-
> bitbake/lib/bb/tests/persist_data.py | 129 ++
> .../tests/runqueue-tests/classes/base.bbclass | 262 +++
> .../runqueue-tests/classes/image.bbclass | 5 +
> .../runqueue-tests/classes/native.bbclass | 2 +
> .../bb/tests/runqueue-tests/conf/bitbake.conf | 16 +
> .../runqueue-tests/conf/multiconfig/mc1.conf | 1 +
> .../runqueue-tests/conf/multiconfig/mc2.conf | 1 +
> .../lib/bb/tests/runqueue-tests/recipes/a1.bb | 0
> .../lib/bb/tests/runqueue-tests/recipes/b1.bb | 1 +
> .../lib/bb/tests/runqueue-tests/recipes/c1.bb | 0
> .../lib/bb/tests/runqueue-tests/recipes/d1.bb | 3 +
> .../lib/bb/tests/runqueue-tests/recipes/e1.bb | 1 +
> bitbake/lib/bb/tests/runqueue.py | 323 ++++
> bitbake/lib/bb/tests/utils.py | 23 +-
> bitbake/lib/bb/tinfoil.py | 12 +-
> bitbake/lib/bb/ui/__init__.py | 12 +-
> bitbake/lib/bb/ui/buildinfohelper.py | 22 +-
> bitbake/lib/bb/ui/knotty.py | 60 +-
> bitbake/lib/bb/ui/ncurses.py | 12 +-
> bitbake/lib/bb/ui/taskexp.py | 12 +-
> bitbake/lib/bb/ui/toasterui.py | 12 +-
> bitbake/lib/bb/ui/uievent.py | 15 +-
> bitbake/lib/bb/ui/uihelper.py | 16 +-
> bitbake/lib/bb/utils.py | 49 +-
> bitbake/lib/bblayers/__init__.py | 4 +
> bitbake/lib/bblayers/action.py | 4 +
> bitbake/lib/bblayers/common.py | 4 +
> bitbake/lib/bblayers/layerindex.py | 6 +-
> bitbake/lib/bblayers/query.py | 43 +-
> bitbake/lib/bs4/dammit.py | 12 +-
> bitbake/lib/bs4/element.py | 22 +-
> bitbake/lib/hashserv/__init__.py | 93 +
> bitbake/lib/hashserv/client.py | 157 ++
> bitbake/lib/hashserv/server.py | 414 +++++
> bitbake/lib/hashserv/tests.py | 142 ++
> bitbake/lib/layerindexlib/__init__.py | 33 +-
> bitbake/lib/layerindexlib/cooker.py | 12 +-
> bitbake/lib/layerindexlib/plugin.py | 13 +-
> bitbake/lib/layerindexlib/restapi.py | 12 +-
> bitbake/lib/layerindexlib/tests/common.py | 12 +-
> bitbake/lib/layerindexlib/tests/cooker.py | 12 +-
> .../lib/layerindexlib/tests/layerindexobj.py | 12 +-
> bitbake/lib/layerindexlib/tests/restapi.py | 12 +-
> bitbake/lib/progressbar/__init__.py | 2 +
> bitbake/lib/progressbar/compat.py | 2 +
> bitbake/lib/progressbar/progressbar.py | 2 +
> bitbake/lib/progressbar/widgets.py | 2 +
> bitbake/lib/prserv/__init__.py | 4 +
> bitbake/lib/prserv/db.py | 6 +-
> bitbake/lib/prserv/serv.py | 4 +
> bitbake/lib/pyinotify.py | 20 +-
> bitbake/lib/toaster/bldcollector/admin.py | 4 +
> bitbake/lib/toaster/bldcollector/urls.py | 13 +-
> bitbake/lib/toaster/bldcollector/views.py | 12 +-
> bitbake/lib/toaster/bldcontrol/admin.py | 4 +
> .../lib/toaster/bldcontrol/bbcontroller.py | 16 +-
> .../bldcontrol/localhostbecontroller.py | 16 +-
> .../management/commands/checksettings.py | 4 +
> .../management/commands/runbuilds.py | 4 +
> bitbake/lib/toaster/bldcontrol/models.py | 4 +
> bitbake/lib/toaster/bldcontrol/views.py | 4 +
> bitbake/lib/toaster/manage.py | 4 +
> bitbake/lib/toaster/orm/fixtures/oe-core.xml | 26 +-
> bitbake/lib/toaster/orm/fixtures/poky.xml | 38 +-
> .../orm/management/commands/lsupdates.py | 15 +-
> bitbake/lib/toaster/orm/models.py | 25 +-
> .../toaster/tests/browser/selenium_helpers.py | 17 +-
> .../tests/browser/selenium_helpers_base.py | 17 +-
> .../tests/browser/test_all_builds_page.py | 16 +-
> .../tests/browser/test_all_projects_page.py | 16 +-
> .../tests/browser/test_builddashboard_page.py | 16 +-
> .../test_builddashboard_page_artifacts.py | 16 +-
> .../test_builddashboard_page_recipes.py | 16 +-
> .../browser/test_builddashboard_page_tasks.py | 16 +-
> .../tests/browser/test_js_unit_tests.py | 16 +-
> .../tests/browser/test_landing_page.py | 18 +-
> .../tests/browser/test_layerdetails_page.py | 18 +-
> .../browser/test_most_recent_builds_states.py | 18 +-
> .../browser/test_new_custom_image_page.py | 16 +-
> .../tests/browser/test_new_project_page.py | 16 +-
> .../tests/browser/test_project_builds_page.py | 16 +-
> .../tests/browser/test_project_config_page.py | 16 +-
> .../tests/browser/test_project_page.py | 16 +-
> .../lib/toaster/tests/browser/test_sample.py | 16 +-
> .../toaster/tests/browser/test_task_page.py | 16 +-
> .../tests/browser/test_toastertable_ui.py | 16 +-
> bitbake/lib/toaster/tests/builds/buildtest.py | 16 +-
> .../tests/builds/test_core_image_min.py | 17 +-
> .../toaster/tests/commands/test_loaddata.py | 16 +-
> .../toaster/tests/commands/test_lsupdates.py | 16 +-
> .../toaster/tests/commands/test_runbuilds.py | 16 +-
> bitbake/lib/toaster/tests/db/test_db.py | 2 +
> .../lib/toaster/tests/eventreplay/__init__.py | 16 +-
> .../tests/functional/functional_helpers.py | 16 +-
> .../tests/functional/test_functional_basic.py | 16 +-
> bitbake/lib/toaster/tests/views/test_views.py | 16 +-
> bitbake/lib/toaster/toastergui/api.py | 13 +-
> bitbake/lib/toaster/toastergui/buildtables.py | 15 +-
> .../toastergui/static/js/importlayer.js | 12 +-
> bitbake/lib/toaster/toastergui/tablefilter.py | 15 +-
> bitbake/lib/toaster/toastergui/tables.py | 15 +-
> .../templatetags/field_values_filter.py | 4 +
> .../objects_to_dictionaries_filter.py | 4 +
> .../templatetags/project_url_tag.py | 4 +
> .../toastergui/templatetags/projecttags.py | 15 +-
> bitbake/lib/toaster/toastergui/typeaheads.py | 12 +-
> bitbake/lib/toaster/toastergui/urls.py | 12 +-
> bitbake/lib/toaster/toastergui/views.py | 16 +-
> bitbake/lib/toaster/toastergui/widgets.py | 15 +-
> .../management/commands/builddelete.py | 4 +
> .../management/commands/buildimport.py | 15 +-
> .../management/commands/buildslist.py | 4 +
> .../management/commands/checksocket.py | 16 +-
> .../toastermain/management/commands/perf.py | 4 +
> bitbake/lib/toaster/toastermain/settings.py | 15 +-
> .../settings_production_example.py | 15 +-
> .../lib/toaster/toastermain/settings_test.py | 15 +-
> bitbake/lib/toaster/toastermain/urls.py | 15 +-
> bitbake/lib/toaster/toastermain/wsgi.py | 7 +-
> doc/user_manual.md | 38 +-
> meta-isar/conf/conf-notes.txt | 6 +-
> scripts/ci_build.sh | 62 +-
> scripts/start_vm | 8 +-
> testsuite/build_test/build_test.py | 2 +-
> testsuite/start_vm.py | 2 +-
> 225 files changed, 5375 insertions(+), 6320 deletions(-)
> create mode 100644 bitbake/.gitattributes
> delete mode 100644 bitbake/HEADER
> rename bitbake/{COPYING => LICENSE.GPL-2.0-only} (84%)
> create mode 100644 bitbake/LICENSE.MIT
> mode change 100755 => 120000 bitbake/bin/bitbake-dumpsig
> create mode 100755 bitbake/bin/bitbake-hashclient
> create mode 100755 bitbake/bin/bitbake-hashserv
> delete mode 100644 bitbake/lib/bb/pysh/builtin.py
> delete mode 100644 bitbake/lib/bb/pysh/interp.py
> delete mode 100644 bitbake/lib/bb/pysh/lsprof.py
> delete mode 100644 bitbake/lib/bb/pysh/pysh.py
> delete mode 100644 bitbake/lib/bb/pysh/subprocess_fix.py
> create mode 100644 bitbake/lib/bb/tests/persist_data.py
> create mode 100644
> bitbake/lib/bb/tests/runqueue-tests/classes/base.bbclass create mode
> 100644 bitbake/lib/bb/tests/runqueue-tests/classes/image.bbclass
> create mode 100644
> bitbake/lib/bb/tests/runqueue-tests/classes/native.bbclass create
> mode 100644 bitbake/lib/bb/tests/runqueue-tests/conf/bitbake.conf
> create mode 100644
> bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc1.conf create
> mode 100644
> bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc2.conf create
> mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/a1.bb create
> mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/b1.bb create
> mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/c1.bb create
> mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/d1.bb create
> mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/e1.bb create
> mode 100644 bitbake/lib/bb/tests/runqueue.py create mode 100644
> bitbake/lib/hashserv/__init__.py create mode 100644
> bitbake/lib/hashserv/client.py create mode 100644
> bitbake/lib/hashserv/server.py create mode 100644
> bitbake/lib/hashserv/tests.py
>
> diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
> index bbef1a3..81ec430 100644
> --- a/RECIPE-API-CHANGELOG.md
> +++ b/RECIPE-API-CHANGELOG.md
> @@ -176,3 +176,19 @@ Otherwise set a encrypted root password like
> this: USERS += "root"
> USER_root[password] =
> "$6$rounds=10000$RXeWrnFmkY$DtuS/OmsAS2cCEDo0BF5qQsizIrq6jPgXnwv3PHqREJeKd1sXdHX/ayQtuQWVDHe0KIO0/sVH8dvQm1KthF0d/"
> ``` +
> +### multiconfig build targets were renamed
> +
> +bitbake was upgraded to version 1.44.0 where "multiconfig" build
> targets were +renamed "mc". As an example, builds for the
> qemuarm-stretch machine should now +be done as follows:
> +
> +```
> +bitbake mc:qemuarm-stretch:isar-image-base
> +```
> +
> +The old syntax is no longer supported and will produce an error:
> +
> +```
> +bitbake multiconfig:qemuarm-stretch:isar-image-base
> +```
> diff --git a/bitbake/.gitattributes b/bitbake/.gitattributes
> new file mode 100644
> index 0000000..e4f8f62
> --- /dev/null
> +++ b/bitbake/.gitattributes
> @@ -0,0 +1,2 @@
> +*min.js binary
> +*min.css binary
> diff --git a/bitbake/HEADER b/bitbake/HEADER
> deleted file mode 100644
> index 9859255..0000000
> --- a/bitbake/HEADER
> +++ /dev/null
> @@ -1,19 +0,0 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> -#
> -# <one line to give the program's name and a brief idea of what it
> does.> -# Copyright (C) <year> <name of author>
> -#
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. -
> diff --git a/bitbake/LICENSE b/bitbake/LICENSE
> index 7d4e5f4..8458042 100644
> --- a/bitbake/LICENSE
> +++ b/bitbake/LICENSE
> @@ -1,4 +1,13 @@
> -BitBake is licensed under the GNU General Public License version
> 2.0. See COPYING for further details. +BitBake is licensed under the
> GNU General Public License version 2.0. See +LICENSE.GPL-2.0-only for
> further details. +
> +Individual files contain the following style tags instead of the
> full license text: +
> + SPDX-License-Identifier: GPL-2.0-only
> +
> +This enables machine processing of license information based on the
> SPDX +License Identifiers that are here available:
> http://spdx.org/licenses/ +
>
> The following external components are distributed with this software:
>
> @@ -17,3 +26,4 @@ Foundation and individual contributors.
> * Font Awesome fonts redistributed under the SIL Open Font License
> 1.1
> * simplediff is distributed under the zlib license.
> +
> diff --git a/bitbake/COPYING b/bitbake/LICENSE.GPL-2.0-only
> similarity index 84%
> rename from bitbake/COPYING
> rename to bitbake/LICENSE.GPL-2.0-only
> index d511905..5db3c0a 100644
> --- a/bitbake/COPYING
> +++ b/bitbake/LICENSE.GPL-2.0-only
> @@ -279,61 +279,10 @@ POSSIBILITY OF SUCH DAMAGES.
>
> END OF TERMS AND CONDITIONS
>
> - How to Apply These Terms to Your New Programs
> +Note:
> +Individual files contain the following tag instead of the full
> license text.
> - If you develop a new program, and you want it to be of the greatest
> -possible use to the public, the best way to achieve this is to make
> it -free software which everyone can redistribute and change under
> these terms.
> + SPDX-License-Identifier: GPL-2.0-only
>
> - To do so, attach the following notices to the program. It is
> safest -to attach them to the start of each source file to most
> effectively -convey the exclusion of warranty; and each file should
> have at least -the "copyright" line and a pointer to where the full
> notice is found. -
> - <one line to give the program's name and a brief idea of what it
> does.>
> - Copyright (C) <year> <name of author>
> -
> - This program is free software; you can redistribute it and/or
> modify
> - it under the terms of the GNU General Public License as
> published by
> - the Free Software Foundation; either version 2 of the License, or
> - (at your option) any later version.
> -
> - This program is distributed in the hope that it will be useful,
> - but WITHOUT ANY WARRANTY; without even the implied warranty of
> - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> - GNU General Public License for more details.
> -
> - You should have received a copy of the GNU General Public
> License along
> - with this program; if not, write to the Free Software
> Foundation, Inc.,
> - 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> -
> -Also add information on how to contact you by electronic and paper
> mail. -
> -If the program is interactive, make it output a short notice like
> this -when it starts in an interactive mode:
> -
> - Gnomovision version 69, Copyright (C) year name of author
> - Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type
> `show w'.
> - This is free software, and you are welcome to redistribute it
> - under certain conditions; type `show c' for details.
> -
> -The hypothetical commands `show w' and `show c' should show the
> appropriate -parts of the General Public License. Of course, the
> commands you use may -be called something other than `show w' and
> `show c'; they could even be -mouse-clicks or menu items--whatever
> suits your program. -
> -You should also get your employer (if you work as a programmer) or
> your -school, if any, to sign a "copyright disclaimer" for the
> program, if -necessary. Here is a sample; alter the names:
> -
> - Yoyodyne, Inc., hereby disclaims all copyright interest in the
> program
> - `Gnomovision' (which makes passes at compilers) written by James
> Hacker. -
> - <signature of Ty Coon>, 1 April 1989
> - Ty Coon, President of Vice
> -
> -This General Public License does not permit incorporating your
> program into -proprietary programs. If your program is a subroutine
> library, you may -consider it more useful to permit linking
> proprietary applications with the -library. If this is what you want
> to do, use the GNU Lesser General -Public License instead of this
> License. +This enables machine processing of license information
> based on the SPDX +License Identifiers that are here available:
> http://spdx.org/licenses/ diff --git a/bitbake/LICENSE.MIT
> b/bitbake/LICENSE.MIT new file mode 100644
> index 0000000..a6919eb
> --- /dev/null
> +++ b/bitbake/LICENSE.MIT
> @@ -0,0 +1,25 @@
> +Permission is hereby granted, free of charge, to any person
> obtaining a copy +of this software and associated documentation files
> (the "Software"), to deal +in the Software without restriction,
> including without limitation the rights +to use, copy, modify, merge,
> publish, distribute, sublicense, and/or sell +copies of the Software,
> and to permit persons to whom the Software is +furnished to do so,
> subject to the following conditions: +
> +The above copyright notice and this permission notice shall be
> included in +all copies or substantial portions of the Software.
> +
> +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
> EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
> MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND
> NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS
> BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN
> ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN
> CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
> +SOFTWARE. +
> +Note:
> +Individual files contain the following tag instead of the full
> license text. +
> + SPDX-License-Identifier: MIT
> +
> +This enables machine processing of license information based on the
> SPDX +License Identifiers that are here available:
> http://spdx.org/licenses/ diff --git a/bitbake/MANIFEST.in
> b/bitbake/MANIFEST.in index b197378..f24969a 100644
> --- a/bitbake/MANIFEST.in
> +++ b/bitbake/MANIFEST.in
> @@ -1,6 +1,8 @@
> -include COPYING
> include ChangeLog
> include AUTHORS
> +include LICENSE
> +include LICENSE.GPL-2.0-only
> +include LICENSE.MIT
> include contrib/*
> include contrib/vim/*/*
> include conf/*
> @@ -8,4 +10,3 @@ include classes/*
> include doc/*
> include doc/manual/*
> include ez_setup.py
> -include HEADER
> diff --git a/bitbake/bin/bitbake b/bitbake/bin/bitbake
> index 57dec2a..66d08f8 100755
> --- a/bitbake/bin/bitbake
> +++ b/bitbake/bin/bitbake
> @@ -1,6 +1,4 @@
> #!/usr/bin/env python3
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # Copyright (C) 2003, 2004 Chris Larson
> # Copyright (C) 2003, 2004 Phil Blundell
> @@ -9,18 +7,8 @@
> # Copyright (C) 2005 ROAD GmbH
> # Copyright (C) 2006 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import os
> import sys
> @@ -38,7 +26,7 @@ from bb.main import bitbake_main,
> BitBakeConfigParameters, BBMainException if
> sys.getfilesystemencoding() != "utf-8": sys.exit("Please use a locale
> setting which supports UTF-8 (such as LANG=en_US.UTF-8).\nPython
> can't change the filesystem locale after loading so we need a UTF-8
> when Python starts or things won't work.") -__version__ = "1.40.0"
> +__version__ = "1.44.0"
> if __name__ == "__main__":
> if __version__ != bb.__version__:
> diff --git a/bitbake/bin/bitbake-diffsigs
> b/bitbake/bin/bitbake-diffsigs index 4e6bbdd..19420a2 100755
> --- a/bitbake/bin/bitbake-diffsigs
> +++ b/bitbake/bin/bitbake-diffsigs
> @@ -1,27 +1,16 @@
> #!/usr/bin/env python3
>
> -# bitbake-diffsigs
> -# BitBake task signature data comparison utility
> +# bitbake-diffsigs / bitbake-dumpsig
> +# BitBake task signature data dump and comparison utility
> #
> # Copyright (C) 2012-2013, 2017 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import os
> import sys
> import warnings
> -import fnmatch
> import argparse
> import logging
> import pickle
> @@ -32,7 +21,10 @@ import bb.tinfoil
> import bb.siggen
> import bb.msg
>
> -logger = bb.msg.logger_create('bitbake-diffsigs')
> +myname = os.path.basename(sys.argv[0])
> +logger = bb.msg.logger_create(myname)
> +
> +is_dump = myname == 'bitbake-dumpsig'
>
> def find_siginfo(tinfoil, pn, taskname, sigs=None):
> result = None
> @@ -59,8 +51,8 @@ def find_siginfo(tinfoil, pn, taskname, sigs=None):
> sys.exit(2)
> return result
>
> -def find_compare_task(bbhandler, pn, taskname, sig1=None, sig2=None,
> color=False):
> - """ Find the most recent signature files for the specified
> PN/task and compare them """ +def find_siginfo_task(bbhandler, pn,
> taskname, sig1=None, sig2=None):
> + """ Find the most recent signature files for the specified
> PN/task """
> if not taskname.startswith('do_'):
> taskname = 'do_%s' % taskname
> @@ -79,73 +71,81 @@ def find_compare_task(bbhandler, pn, taskname,
> sig1=None, sig2=None, color=False latestfiles = [sigfiles[sig1],
> sigfiles[sig2]] else:
> filedates = find_siginfo(bbhandler, pn, taskname)
> - latestfiles = sorted(filedates.keys(), key=lambda f:
> filedates[f])[-3:]
> + latestfiles = sorted(filedates.keys(), key=lambda f:
> filedates[f])[-2:] if not latestfiles:
> logger.error('No sigdata files found matching %s %s' %
> (pn, taskname)) sys.exit(1)
> - elif len(latestfiles) < 2:
> - logger.error('Only one matching sigdata file found for
> the specified task (%s %s)' % (pn, taskname))
> - sys.exit(1)
>
> - # Define recursion callback
> - def recursecb(key, hash1, hash2):
> - hashes = [hash1, hash2]
> - hashfiles = find_siginfo(bbhandler, key, None, hashes)
> -
> - recout = []
> - if len(hashfiles) == 0:
> - recout.append("Unable to find matching sigdata for %s
> with hashes %s or %s" % (key, hash1, hash2))
> - elif not hash1 in hashfiles:
> - recout.append("Unable to find matching sigdata for %s
> with hash %s" % (key, hash1))
> - elif not hash2 in hashfiles:
> - recout.append("Unable to find matching sigdata for %s
> with hash %s" % (key, hash2))
> - else:
> - out2 = bb.siggen.compare_sigfiles(hashfiles[hash1],
> hashfiles[hash2], recursecb, color=color)
> - for change in out2:
> - for line in change.splitlines():
> - recout.append(' ' + line)
> + return latestfiles
> +
>
> - return recout
> +# Define recursion callback
> +def recursecb(key, hash1, hash2):
> + hashes = [hash1, hash2]
> + hashfiles = find_siginfo(tinfoil, key, None, hashes)
>
> - # Recurse into signature comparison
> - logger.debug("Signature file (previous): %s" % latestfiles[-2])
> - logger.debug("Signature file (latest): %s" % latestfiles[-1])
> - output = bb.siggen.compare_sigfiles(latestfiles[-2],
> latestfiles[-1], recursecb, color=color)
> - if output:
> - print('\n'.join(output))
> - sys.exit(0)
> + recout = []
> + if len(hashfiles) == 0:
> + recout.append("Unable to find matching sigdata for %s with
> hashes %s or %s" % (key, hash1, hash2))
> + elif not hash1 in hashfiles:
> + recout.append("Unable to find matching sigdata for %s with
> hash %s" % (key, hash1))
> + elif not hash2 in hashfiles:
> + recout.append("Unable to find matching sigdata for %s with
> hash %s" % (key, hash2))
> + else:
> + out2 = bb.siggen.compare_sigfiles(hashfiles[hash1],
> hashfiles[hash2], recursecb, color=color)
> + for change in out2:
> + for line in change.splitlines():
> + recout.append(' ' + line)
>
> + return recout
>
>
> parser = argparse.ArgumentParser(
> - description="Compares siginfo/sigdata files written out by
> BitBake")
> + description=("Dumps" if is_dump else "Compares") + "
> siginfo/sigdata files written out by BitBake")
> -parser.add_argument('-d', '--debug',
> +parser.add_argument('-D', '--debug',
> help='Enable debug output',
> action='store_true')
>
> -parser.add_argument('--color',
> - help='Colorize output (where %(metavar)s is %(choices)s)',
> - choices=['auto', 'always', 'never'], default='auto',
> metavar='color') +if is_dump:
> + parser.add_argument("-t", "--task",
> + help="find the signature data file for the last run of
> the specified task",
> + action="store", dest="taskargs", nargs=2,
> metavar=('recipename', 'taskname')) +
> + parser.add_argument("sigdatafile1",
> + help="Signature file to dump. Not used when using
> -t/--task.",
> + action="store", nargs='?', metavar="sigdatafile")
> +else:
> + parser.add_argument('-c', '--color',
> + help='Colorize the output (where %(metavar)s is
> %(choices)s)',
> + choices=['auto', 'always', 'never'], default='auto',
> metavar='color')
> -parser.add_argument("-t", "--task",
> - help="find the signature data files for last two runs of the
> specified task and compare them",
> - action="store", dest="taskargs", nargs=2,
> metavar=('recipename', 'taskname'))
> + parser.add_argument('-d', '--dump',
> + help='Dump the last signature data instead of comparing
> (equivalent to using bitbake-dumpsig)',
> + action='store_true')
>
> -parser.add_argument("-s", "--signature",
> - help="With -t/--task, specify the signatures to look for
> instead of taking the last two",
> - action="store", dest="sigargs", nargs=2, metavar=('fromsig',
> 'tosig'))
> + parser.add_argument("-t", "--task",
> + help="find the signature data files for the last two
> runs of the specified task and compare them",
> + action="store", dest="taskargs", nargs=2,
> metavar=('recipename', 'taskname'))
> -parser.add_argument("sigdatafile1",
> - help="First signature file to compare (or signature file to
> dump, if second not specified). Not used when using -t/--task.",
> - action="store", nargs='?')
> + parser.add_argument("-s", "--signature",
> + help="With -t/--task, specify the signatures to look for
> instead of taking the last two",
> + action="store", dest="sigargs", nargs=2,
> metavar=('fromsig', 'tosig'))
> -parser.add_argument("sigdatafile2",
> - help="Second signature file to compare",
> - action="store", nargs='?')
> + parser.add_argument("sigdatafile1",
> + help="First signature file to compare (or signature file
> to dump, if second not specified). Not used when using -t/--task.",
> + action="store", nargs='?')
>
> + parser.add_argument("sigdatafile2",
> + help="Second signature file to compare",
> + action="store", nargs='?')
>
> options = parser.parse_args()
> +if is_dump:
> + options.color = 'never'
> + options.dump = True
> + options.sigdatafile2 = None
> + options.sigargs = None
>
> if options.debug:
> logger.setLevel(logging.DEBUG)
> @@ -155,17 +155,32 @@ color = (options.color == 'always' or
> (options.color == 'auto' and sys.stdout.is if options.taskargs:
> with bb.tinfoil.Tinfoil() as tinfoil:
> tinfoil.prepare(config_only=True)
> - if options.sigargs:
> - find_compare_task(tinfoil, options.taskargs[0],
> options.taskargs[1], options.sigargs[0], options.sigargs[1],
> color=color)
> + if not options.dump and options.sigargs:
> + files = find_siginfo_task(tinfoil, options.taskargs[0],
> options.taskargs[1], options.sigargs[0], options.sigargs[1])
> + else:
> + files = find_siginfo_task(tinfoil, options.taskargs[0],
> options.taskargs[1]) +
> + if options.dump:
> + logger.debug("Signature file: %s" % files[-1])
> + output = bb.siggen.dump_sigfile(files[-1])
> else:
> - find_compare_task(tinfoil, options.taskargs[0],
> options.taskargs[1], color=color)
> + if len(files) < 2:
> + logger.error('Only one matching sigdata file found
> for the specified task (%s %s)' % (options.taskargs[0],
> options.taskargs[1]))
> + sys.exit(1)
> +
> + # Recurse into signature comparison
> + logger.debug("Signature file (previous): %s" % files[-2])
> + logger.debug("Signature file (latest): %s" % files[-1])
> + output = bb.siggen.compare_sigfiles(files[-2],
> files[-1], recursecb, color=color) else:
> if options.sigargs:
> logger.error('-s/--signature can only be used together with
> -t/--task') sys.exit(1)
> try:
> - if options.sigdatafile1 and options.sigdatafile2:
> - output =
> bb.siggen.compare_sigfiles(options.sigdatafile1,
> options.sigdatafile2, color=color)
> + if not options.dump and options.sigdatafile1 and
> options.sigdatafile2:
> + with bb.tinfoil.Tinfoil() as tinfoil:
> + tinfoil.prepare(config_only=True)
> + output =
> bb.siggen.compare_sigfiles(options.sigdatafile1,
> options.sigdatafile2, recursecb, color=color) elif
> options.sigdatafile1: output =
> bb.siggen.dump_sigfile(options.sigdatafile1) else: @@ -179,5 +194,5
> @@ else: logger.error('Invalid signature data - ensure you are
> specifying sigdata/siginfo files') sys.exit(1)
>
> - if output:
> - print('\n'.join(output))
> +if output:
> + print('\n'.join(output))
> diff --git a/bitbake/bin/bitbake-dumpsig b/bitbake/bin/bitbake-dumpsig
> deleted file mode 100755
> index 95ebd93..0000000
> --- a/bitbake/bin/bitbake-dumpsig
> +++ /dev/null
> @@ -1,94 +0,0 @@
> -#!/usr/bin/env python3
> -
> -# bitbake-dumpsig
> -# BitBake task signature dump utility
> -#
> -# Copyright (C) 2013 Intel Corporation
> -#
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. -
> -import os
> -import sys
> -import warnings
> -import optparse
> -import logging
> -import pickle
> -
> -sys.path.insert(0,
> os.path.join(os.path.dirname(os.path.dirname(sys.argv[0])), 'lib')) -
> -import bb.tinfoil
> -import bb.siggen
> -import bb.msg
> -
> -logger = bb.msg.logger_create('bitbake-dumpsig')
> -
> -def find_siginfo_task(bbhandler, pn, taskname):
> - """ Find the most recent signature file for the specified
> PN/task """ -
> - if not hasattr(bb.siggen, 'find_siginfo'):
> - logger.error('Metadata does not support finding signature
> data files')
> - sys.exit(1)
> -
> - if not taskname.startswith('do_'):
> - taskname = 'do_%s' % taskname
> -
> - filedates = bb.siggen.find_siginfo(pn, taskname, None,
> bbhandler.config_data)
> - latestfiles = sorted(filedates.keys(), key=lambda f:
> filedates[f])[-1:]
> - if not latestfiles:
> - logger.error('No sigdata files found matching %s %s' % (pn,
> taskname))
> - sys.exit(1)
> -
> - return latestfiles[0]
> -
> -parser = optparse.OptionParser(
> - description = "Dumps siginfo/sigdata files written out by
> BitBake",
> - usage = """
> - %prog -t recipename taskname
> - %prog sigdatafile""")
> -
> -parser.add_option("-D", "--debug",
> - help = "enable debug",
> - action = "store_true", dest="debug", default = False)
> -
> -parser.add_option("-t", "--task",
> - help = "find the signature data file for the specified task",
> - action="store", dest="taskargs", nargs=2,
> metavar='recipename taskname') -
> -options, args = parser.parse_args(sys.argv)
> -
> -if options.debug:
> - logger.setLevel(logging.DEBUG)
> -
> -if options.taskargs:
> - tinfoil = bb.tinfoil.Tinfoil()
> - tinfoil.prepare(config_only = True)
> - file = find_siginfo_task(tinfoil, options.taskargs[0],
> options.taskargs[1])
> - logger.debug("Signature file: %s" % file)
> -elif len(args) == 1:
> - parser.print_help()
> - sys.exit(0)
> -else:
> - file = args[1]
> -
> -try:
> - output = bb.siggen.dump_sigfile(file)
> -except IOError as e:
> - logger.error(str(e))
> - sys.exit(1)
> -except (pickle.UnpicklingError, EOFError):
> - logger.error('Invalid signature data - ensure you are specifying
> a sigdata/siginfo file')
> - sys.exit(1)
> -
> -if output:
> - print('\n'.join(output))
> diff --git a/bitbake/bin/bitbake-dumpsig b/bitbake/bin/bitbake-dumpsig
> new file mode 120000
> index 0000000..b1e8489
> --- /dev/null
> +++ b/bitbake/bin/bitbake-dumpsig
> @@ -0,0 +1 @@
> +bitbake-diffsigs
> \ No newline at end of file
> diff --git a/bitbake/bin/bitbake-hashclient
> b/bitbake/bin/bitbake-hashclient new file mode 100755
> index 0000000..29ab65f
> --- /dev/null
> +++ b/bitbake/bin/bitbake-hashclient
> @@ -0,0 +1,170 @@
> +#! /usr/bin/env python3
> +#
> +# Copyright (C) 2019 Garmin Ltd.
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> +import argparse
> +import hashlib
> +import logging
> +import os
> +import pprint
> +import sys
> +import threading
> +import time
> +
> +try:
> + import tqdm
> + ProgressBar = tqdm.tqdm
> +except ImportError:
> + class ProgressBar(object):
> + def __init__(self, *args, **kwargs):
> + pass
> +
> + def __enter__(self):
> + return self
> +
> + def __exit__(self, *args, **kwargs):
> + pass
> +
> + def update(self):
> + pass
> +
> +sys.path.insert(0,
> os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib')) +
> +import hashserv
> +
> +DEFAULT_ADDRESS = 'unix://./hashserve.sock'
> +METHOD = 'stress.test.method'
> +
> +
> +def main():
> + def handle_stats(args, client):
> + if args.reset:
> + s = client.reset_stats()
> + else:
> + s = client.get_stats()
> + pprint.pprint(s)
> + return 0
> +
> + def handle_stress(args, client):
> + def thread_main(pbar, lock):
> + nonlocal found_hashes
> + nonlocal missed_hashes
> + nonlocal max_time
> +
> + client = hashserv.create_client(args.address)
> +
> + for i in range(args.requests):
> + taskhash = hashlib.sha256()
> + taskhash.update(args.taskhash_seed.encode('utf-8'))
> + taskhash.update(str(i).encode('utf-8'))
> +
> + start_time = time.perf_counter()
> + l = client.get_unihash(METHOD, taskhash.hexdigest())
> + elapsed = time.perf_counter() - start_time
> +
> + with lock:
> + if l:
> + found_hashes += 1
> + else:
> + missed_hashes += 1
> +
> + max_time = max(elapsed, max_time)
> + pbar.update()
> +
> + max_time = 0
> + found_hashes = 0
> + missed_hashes = 0
> + lock = threading.Lock()
> + total_requests = args.clients * args.requests
> + start_time = time.perf_counter()
> + with ProgressBar(total=total_requests) as pbar:
> + threads = [threading.Thread(target=thread_main,
> args=(pbar, lock), daemon=False) for _ in range(args.clients)]
> + for t in threads:
> + t.start()
> +
> + for t in threads:
> + t.join()
> +
> + elapsed = time.perf_counter() - start_time
> + with lock:
> + print("%d requests in %.1fs. %.1f requests per second" %
> (total_requests, elapsed, total_requests / elapsed))
> + print("Average request time %.8fs" % (elapsed /
> total_requests))
> + print("Max request time was %.8fs" % max_time)
> + print("Found %d hashes, missed %d" % (found_hashes,
> missed_hashes)) +
> + if args.report:
> + with ProgressBar(total=args.requests) as pbar:
> + for i in range(args.requests):
> + taskhash = hashlib.sha256()
> +
> taskhash.update(args.taskhash_seed.encode('utf-8'))
> + taskhash.update(str(i).encode('utf-8'))
> +
> + outhash = hashlib.sha256()
> + outhash.update(args.outhash_seed.encode('utf-8'))
> + outhash.update(str(i).encode('utf-8'))
> +
> + client.report_unihash(taskhash.hexdigest(),
> METHOD, outhash.hexdigest(), taskhash.hexdigest()) +
> + with lock:
> + pbar.update()
> +
> + parser = argparse.ArgumentParser(description='Hash Equivalence
> Client')
> + parser.add_argument('--address', default=DEFAULT_ADDRESS,
> help='Server address (default "%(default)s")')
> + parser.add_argument('--log', default='WARNING', help='Set
> logging level') +
> + subparsers = parser.add_subparsers()
> +
> + stats_parser = subparsers.add_parser('stats', help='Show server
> stats')
> + stats_parser.add_argument('--reset', action='store_true',
> + help='Reset server stats')
> + stats_parser.set_defaults(func=handle_stats)
> +
> + stress_parser = subparsers.add_parser('stress', help='Run stress
> test')
> + stress_parser.add_argument('--clients', type=int, default=10,
> + help='Number of simultaneous clients')
> + stress_parser.add_argument('--requests', type=int, default=1000,
> + help='Number of requests each client
> will perform')
> + stress_parser.add_argument('--report', action='store_true',
> + help='Report new hashes')
> + stress_parser.add_argument('--taskhash-seed', default='',
> + help='Include string in taskhash')
> + stress_parser.add_argument('--outhash-seed', default='',
> + help='Include string in outhash')
> + stress_parser.set_defaults(func=handle_stress)
> +
> + args = parser.parse_args()
> +
> + logger = logging.getLogger('hashserv')
> +
> + level = getattr(logging, args.log.upper(), None)
> + if not isinstance(level, int):
> + raise ValueError('Invalid log level: %s' % args.log)
> +
> + logger.setLevel(level)
> + console = logging.StreamHandler()
> + console.setLevel(level)
> + logger.addHandler(console)
> +
> + func = getattr(args, 'func', None)
> + if func:
> + client = hashserv.create_client(args.address)
> + # Try to establish a connection to the server now to detect
> failures
> + # early
> + client.connect()
> +
> + return func(args, client)
> +
> + return 0
> +
> +
> +if __name__ == '__main__':
> + try:
> + ret = main()
> + except Exception:
> + ret = 1
> + import traceback
> + traceback.print_exc()
> + sys.exit(ret)
> diff --git a/bitbake/bin/bitbake-hashserv
> b/bitbake/bin/bitbake-hashserv new file mode 100755
> index 0000000..1bc1f91
> --- /dev/null
> +++ b/bitbake/bin/bitbake-hashserv
> @@ -0,0 +1,62 @@
> +#! /usr/bin/env python3
> +#
> +# Copyright (C) 2018 Garmin Ltd.
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> +import os
> +import sys
> +import logging
> +import argparse
> +import sqlite3
> +
> +sys.path.insert(0,
> os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib')) +
> +import hashserv
> +
> +VERSION = "1.0.0"
> +
> +DEFAULT_BIND = 'unix://./hashserve.sock'
> +
> +
> +def main():
> + parser = argparse.ArgumentParser(description='Hash Equivalence
> Reference Server. Version=%s' % VERSION,
> + epilog='''The bind address is
> the path to a unix domain socket if it is
> + prefixed with
> "unix://". Otherwise, it is an IP address
> + and port in form
> ADDRESS:PORT. To bind to all addresses, leave
> + the ADDRESS empty,
> e.g. "--bind :8686". To bind to a specific
> + IPv6 address, enclose
> the address in "[]", e.g.
> + "--bind [::1]:8686"'''
> + )
> +
> + parser.add_argument('--bind', default=DEFAULT_BIND, help='Bind
> address (default "%(default)s")')
> + parser.add_argument('--database', default='./hashserv.db',
> help='Database file (default "%(default)s")')
> + parser.add_argument('--log', default='WARNING', help='Set
> logging level') +
> + args = parser.parse_args()
> +
> + logger = logging.getLogger('hashserv')
> +
> + level = getattr(logging, args.log.upper(), None)
> + if not isinstance(level, int):
> + raise ValueError('Invalid log level: %s' % args.log)
> +
> + logger.setLevel(level)
> + console = logging.StreamHandler()
> + console.setLevel(level)
> + logger.addHandler(console)
> +
> + server = hashserv.create_server(args.bind, args.database)
> + server.serve_forever()
> + return 0
> +
> +
> +if __name__ == '__main__':
> + try:
> + ret = main()
> + except Exception:
> + ret = 1
> + import traceback
> + traceback.print_exc()
> + sys.exit(ret)
> diff --git a/bitbake/bin/bitbake-layers b/bitbake/bin/bitbake-layers
> index d184011..a884dc1 100755
> --- a/bitbake/bin/bitbake-layers
> +++ b/bitbake/bin/bitbake-layers
> @@ -7,18 +7,8 @@
> # Copyright (C) 2011 Mentor Graphics Corporation
> # Copyright (C) 2011-2015 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import logging
> import os
> diff --git a/bitbake/bin/bitbake-prserv b/bitbake/bin/bitbake-prserv
> index f38d2dd..1e9b6cb 100755
> --- a/bitbake/bin/bitbake-prserv
> +++ b/bitbake/bin/bitbake-prserv
> @@ -1,4 +1,8 @@
> #!/usr/bin/env python3
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> import os
> import sys,logging
> import optparse
> diff --git a/bitbake/bin/bitbake-selftest
> b/bitbake/bin/bitbake-selftest index cfa7ac5..041a271 100755
> --- a/bitbake/bin/bitbake-selftest
> +++ b/bitbake/bin/bitbake-selftest
> @@ -2,18 +2,8 @@
> #
> # Copyright (C) 2012 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import os
> import sys, logging
> @@ -22,6 +12,7 @@ sys.path.insert(0,
> os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib import
> unittest try:
> import bb
> + import hashserv
> import layerindexlib
> except RuntimeError as exc:
> sys.exit(str(exc))
> @@ -33,7 +24,10 @@ tests = ["bb.tests.codeparser",
> "bb.tests.event",
> "bb.tests.fetch",
> "bb.tests.parse",
> + "bb.tests.persist_data",
> + "bb.tests.runqueue",
> "bb.tests.utils",
> + "hashserv.tests",
> "layerindexlib.tests.layerindexobj",
> "layerindexlib.tests.restapi",
> "layerindexlib.tests.cooker"]
> diff --git a/bitbake/bin/bitbake-worker b/bitbake/bin/bitbake-worker
> index e925054..6776cad 100755
> --- a/bitbake/bin/bitbake-worker
> +++ b/bitbake/bin/bitbake-worker
> @@ -1,4 +1,7 @@
> #!/usr/bin/env python3
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
>
> import os
> import sys
> @@ -136,7 +139,7 @@ def sigterm_handler(signum, frame):
> os.killpg(0, signal.SIGTERM)
> sys.exit()
>
> -def fork_off_task(cfg, data, databuilder, workerdata, fn, task,
> taskname, appends, taskdepdata, extraconfigdata, quieterrors=False,
> dry_run_exec=False): +def fork_off_task(cfg, data, databuilder,
> workerdata, fn, task, taskname, taskhash, unihash, appends,
> taskdepdata, extraconfigdata, quieterrors=False, dry_run_exec=False):
> # We need to setup the environment BEFORE the fork, since # a fork()
> or exec*() activates PSEUDO... @@ -231,10 +234,13 @@ def
> fork_off_task(cfg, data, databuilder, workerdata, fn, task, taskname,
> append the_data.setVar(varname, value)
> bb.parse.siggen.set_taskdata(workerdata["sigdata"])
> + if "newhashes" in workerdata:
> +
> bb.parse.siggen.set_taskhashes(workerdata["newhashes"]) ret = 0
>
> the_data = bb_cache.loadDataFull(fn, appends)
> - the_data.setVar('BB_TASKHASH',
> workerdata["runq_hash"][task])
> + the_data.setVar('BB_TASKHASH', taskhash)
> + the_data.setVar('BB_UNIHASH', unihash)
>
> bb.utils.set_process_name("%s:%s" %
> (the_data.getVar("PN"), taskname.replace("do_", "")))
> @@ -373,6 +379,7 @@ class BitbakeWorker(object):
> self.handle_item(b"cookerconfig",
> self.handle_cookercfg) self.handle_item(b"extraconfigdata",
> self.handle_extraconfigdata) self.handle_item(b"workerdata",
> self.handle_workerdata)
> + self.handle_item(b"newtaskhashes",
> self.handle_newtaskhashes) self.handle_item(b"runtask",
> self.handle_runtask) self.handle_item(b"finishnow",
> self.handle_finishnow) self.handle_item(b"ping", self.handle_ping)
> @@ -411,6 +418,10 @@ class BitbakeWorker(object):
> bb.msg.loggerDefaultDomains =
> self.workerdata["logdefaultdomain"] for mc in self.databuilder.mcdata:
> self.databuilder.mcdata[mc].setVar("PRSERV_HOST",
> self.workerdata["prhost"])
> + self.databuilder.mcdata[mc].setVar("BB_HASHSERVE",
> self.workerdata["hashservaddr"]) +
> + def handle_newtaskhashes(self, data):
> + self.workerdata["newhashes"] = pickle.loads(data)
>
> def handle_ping(self, _):
> workerlog_write("Handling ping\n")
> @@ -425,10 +436,10 @@ class BitbakeWorker(object):
> sys.exit(0)
>
> def handle_runtask(self, data):
> - fn, task, taskname, quieterrors, appends, taskdepdata,
> dry_run_exec = pickle.loads(data)
> + fn, task, taskname, taskhash, unihash, quieterrors, appends,
> taskdepdata, dry_run_exec = pickle.loads(data)
> workerlog_write("Handling runtask %s %s %s\n" % (task, fn, taskname))
> - pid, pipein, pipeout = fork_off_task(self.cookercfg,
> self.data, self.databuilder, self.workerdata, fn, task, taskname,
> appends, taskdepdata, self.extraconfigdata, quieterrors, dry_run_exec)
> + pid, pipein, pipeout = fork_off_task(self.cookercfg,
> self.data, self.databuilder, self.workerdata, fn, task, taskname,
> taskhash, unihash, appends, taskdepdata, self.extraconfigdata,
> quieterrors, dry_run_exec) self.build_pids[pid] = task
> self.build_pipes[pid] = runQueueWorkerPipe(pipein, pipeout) diff
> --git a/bitbake/bin/bitdoc b/bitbake/bin/bitdoc index
> 2744678..9bd02be 100755 --- a/bitbake/bin/bitdoc
> +++ b/bitbake/bin/bitdoc
> @@ -1,21 +1,9 @@
> #!/usr/bin/env python3
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # Copyright (C) 2005 Holger Hans Peter Freyther
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import optparse, os, sys
>
> diff --git a/bitbake/bin/git-make-shallow
> b/bitbake/bin/git-make-shallow index 296d3a3..57069f7 100755
> --- a/bitbake/bin/git-make-shallow
> +++ b/bitbake/bin/git-make-shallow
> @@ -1,4 +1,8 @@
> #!/usr/bin/env python3
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> """git-make-shallow: make the current git repository shallow
>
> Remove the history of the specified revisions, then optionally
> filter the diff --git a/bitbake/bin/toaster b/bitbake/bin/toaster
> index ecf66fa..c3472df 100755
> --- a/bitbake/bin/toaster
> +++ b/bitbake/bin/toaster
> @@ -3,19 +3,9 @@
> # toaster - shell script to start Toaster
>
> # Copyright (C) 2013-2015 Intel Corp.
> -
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License as
> published by -# the Free Software Foundation; either version 2 of the
> License, or -# (at your option) any later version.
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# SPDX-License-Identifier: GPL-2.0-or-later
> #
> -# You should have received a copy of the GNU General Public License
> -# along with this program. If not, see http://www.gnu.org/licenses/.
>
> HELP="
> Usage: source toaster start|stop [webport=<address:port>] [noweb]
> [nobuild] [toasterdir] diff --git a/bitbake/bin/toaster-eventreplay
> b/bitbake/bin/toaster-eventreplay index 80967a0..8fa4ab7 100755
> --- a/bitbake/bin/toaster-eventreplay
> +++ b/bitbake/bin/toaster-eventreplay
> @@ -1,25 +1,12 @@
> #!/usr/bin/env python3
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # Copyright (C) 2014 Alex Damian
> #
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> # This file re-uses code spread throughout other Bitbake source
> files. # As such, all other copyrights belong to their own right
> holders. #
> -#
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> """
> This command takes a filename as a single parameter. The filename is
> read diff --git a/bitbake/classes/base.bbclass
> b/bitbake/classes/base.bbclass index 71d9ec1..08441fe 100644
> --- a/bitbake/classes/base.bbclass
> +++ b/bitbake/classes/base.bbclass
> @@ -44,7 +44,7 @@ python do_showdata() {
> # emit the metadata which isnt valid shell
> for e in bb.data.keys(d):
> if d.getVarFlag(e, 'python', False):
> - bb.plain("\npython %s () {\n%s}" % (e,
> d.getVar(e, True)))
> + bb.plain("\npython %s () {\n%s}" % (e,
> d.getVar(e))) }
>
> addtask listtasks
> diff --git a/bitbake/contrib/dump_cache.py
> b/bitbake/contrib/dump_cache.py index 8963ca4..c6723cb 100755
> --- a/bitbake/contrib/dump_cache.py
> +++ b/bitbake/contrib/dump_cache.py
> @@ -1,6 +1,4 @@
> #!/usr/bin/env python3
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # Copyright (C) 2012, 2018 Wind River Systems, Inc.
> #
> diff --git
> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
> index f1caaec..46dafee 100644 ---
> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
> +++
> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
> @@ -31,7 +31,7 @@ <para> Prior to executing BitBake, you should take
> advantage of available parallel thread execution on your build host
> by setting the
> - <link
> linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
> + <link
> linkend='var-bb-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
> variable in your project's <filename>local.conf</filename>
> configuration file. </para>
> @@ -87,9 +87,9 @@
> <para>
> The <filename>layer.conf</filename> files are used to
> construct key variables such as
> - <link
> linkend='var-BBPATH'><filename>BBPATH</filename></link>
> + <link
> linkend='var-bb-BBPATH'><filename>BBPATH</filename></link> and
> - <link
> linkend='var-BBFILES'><filename>BBFILES</filename></link>.
> + <link
> linkend='var-bb-BBFILES'><filename>BBFILES</filename></link>.
> <filename>BBPATH</filename> is used to search for configuration and
> class files under the <filename>conf</filename> and
> <filename>classes</filename> @@ -117,19 +117,19 @@
> at certain variables, including:
> <itemizedlist>
> <listitem><para>
> - <link
> linkend='var-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>
> + <link
> linkend='var-bb-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>
> </para></listitem> <listitem><para>
> - <link
> linkend='var-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link>
> + <link
> linkend='var-bb-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link>
> </para></listitem> <listitem><para>
> - <link
> linkend='var-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link>
> + <link
> linkend='var-bb-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link>
> </para></listitem> <listitem><para>
> - <link
> linkend='var-BB_ORIGENV'><filename>BB_ORIGENV</filename></link>
> + <link
> linkend='var-bb-BB_ORIGENV'><filename>BB_ORIGENV</filename></link>
> </para></listitem> <listitem><para>
> - <link
> linkend='var-BITBAKE_UI'><filename>BITBAKE_UI</filename></link>
> + <link
> linkend='var-bb-BITBAKE_UI'><filename>BITBAKE_UI</filename></link>
> </para></listitem> </itemizedlist>
> The first four variables in this list relate to how
> BitBake treats shell @@ -156,7 +156,7 @@
> BitBake first searches the current working directory for
> an optional <filename>conf/bblayers.conf</filename> configuration
> file. This file is expected to contain a
> - <link
> linkend='var-BBLAYERS'><filename>BBLAYERS</filename></link>
> + <link
> linkend='var-bb-BBLAYERS'><filename>BBLAYERS</filename></link>
> variable that is a space-delimited list of 'layer' directories.
> Recall that if BitBake cannot find a
> <filename>bblayers.conf</filename> file, then it is assumed the user
> has set the <filename>BBPATH</filename> @@ -166,10 +166,10 @@ <para>
> For each directory (layer) in this list, a
> <filename>conf/layer.conf</filename> file is located and parsed with
> the
> - <link
> linkend='var-LAYERDIR'><filename>LAYERDIR</filename></link>
> + <link
> linkend='var-bb-LAYERDIR'><filename>LAYERDIR</filename></link>
> variable being set to the directory where the layer was found. The
> idea is these files automatically set up
> - <link
> linkend='var-BBPATH'><filename>BBPATH</filename></link>
> + <link
> linkend='var-bb-BBPATH'><filename>BBPATH</filename></link> and other
> variables correctly for a given build directory. </para>
>
> @@ -189,7 +189,7 @@
> depending on the environment variables previously
> mentioned or set in the configuration files.
> The
> - "<link linkend='ref-variables-glos'>Variables
> Glossary</link>"
> + "<link linkend='ref-bb-variables-glos'>Variables
> Glossary</link>" chapter presents a full list of variables.
> </para>
>
> @@ -204,7 +204,7 @@
> <para>
> The <filename>base.bbclass</filename> file is always
> included. Other classes that are specified in the configuration using
> the
> - <link
> linkend='var-INHERIT'><filename>INHERIT</filename></link>
> + <link
> linkend='var-bb-INHERIT'><filename>INHERIT</filename></link> variable
> are also included. BitBake searches for class files in a
> <filename>classes</filename> subdirectory under
> @@ -270,7 +270,7 @@
>
> <para>
> During the configuration phase, BitBake will have set
> - <link
> linkend='var-BBFILES'><filename>BBFILES</filename></link>.
> + <link
> linkend='var-bb-BBFILES'><filename>BBFILES</filename></link>. BitBake
> now uses it to construct a list of recipes to parse, along with any
> append files (<filename>.bbappend</filename>) to apply.
> @@ -292,7 +292,7 @@
> Any inherit statements cause BitBake to find and
> then parse class files (<filename>.bbclass</filename>)
> using
> - <link
> linkend='var-BBPATH'><filename>BBPATH</filename></link>
> + <link
> linkend='var-bb-BBPATH'><filename>BBPATH</filename></link> as the
> search path. Finally, BitBake parses in order any append files found
> in <filename>BBFILES</filename>.
> @@ -303,8 +303,8 @@
> pieces of metadata.
> For example, in <filename>bitbake.conf</filename> the
> recipe name and version are used to set the variables
> - <link linkend='var-PN'><filename>PN</filename></link> and
> - <link linkend='var-PV'><filename>PV</filename></link>:
> + <link linkend='var-bb-PN'><filename>PN</filename></link>
> and
> + <link linkend='var-bb-PV'><filename>PV</filename></link>:
> <literallayout class='monospaced'>
> PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE',
> False),d)[0] or 'defaultpkgname'}" PV =
> "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[1]
> or '1.0'}" @@ -336,7 +336,7 @@ recipe information.
> The validity of this cache is determined by first
> computing a checksum of the base configuration data (see
> - <link
> linkend='var-BB_HASHCONFIG_WHITELIST'><filename>BB_HASHCONFIG_WHITELIST</filename></link>)
> + <link
> linkend='var-bb-BB_HASHCONFIG_WHITELIST'><filename>BB_HASHCONFIG_WHITELIST</filename></link>)
> and then checking if the checksum matches. If that checksum matches
> what is in the cache and the recipe and class files have not changed,
> Bitbake is able to use @@ -384,9 +384,9 @@
> the recipe can be known.
> Each recipe's <filename>PROVIDES</filename> list is
> created implicitly through the recipe's
> - <link linkend='var-PN'><filename>PN</filename></link>
> variable
> + <link linkend='var-bb-PN'><filename>PN</filename></link>
> variable and explicitly through the recipe's
> - <link
> linkend='var-PROVIDES'><filename>PROVIDES</filename></link>
> + <link
> linkend='var-bb-PROVIDES'><filename>PROVIDES</filename></link>
> variable, which is optional. </para>
>
> @@ -427,7 +427,7 @@
> PREFERRED_PROVIDER_virtual/kernel = "linux-yocto"
> </literallayout>
> The default
> - <link
> linkend='var-PREFERRED_PROVIDER'><filename>PREFERRED_PROVIDER</filename></link>
> + <link
> linkend='var-bb-PREFERRED_PROVIDER'><filename>PREFERRED_PROVIDER</filename></link>
> is the provider with the same name as the target. Bitbake iterates
> through each target it needs to build and resolves them and their
> dependencies using this process. @@ -439,10 +439,10 @@
> BitBake defaults to the highest version of a provider.
> Version comparisons are made using the same method as
> Debian. You can use the
> - <link
> linkend='var-PREFERRED_VERSION'><filename>PREFERRED_VERSION</filename></link>
> + <link
> linkend='var-bb-PREFERRED_VERSION'><filename>PREFERRED_VERSION</filename></link>
> variable to specify a particular version. You can influence the order
> by using the
> - <link
> linkend='var-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link>
> + <link
> linkend='var-bb-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link>
> variable. </para>
>
> @@ -464,7 +464,7 @@
> BitBake defaults to selecting the most recent
> version, unless otherwise specified.
> If the recipe in question has a
> - <link
> linkend='var-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link>
> + <link
> linkend='var-bb-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link>
> set lower than the other recipes (default is 0), then it will not be
> selected. This allows the person or persons maintaining
> @@ -475,9 +475,9 @@
>
> <para>
> If the first recipe is named
> <filename>a_1.1.bb</filename>, then the
> - <link linkend='var-PN'><filename>PN</filename></link>
> variable
> + <link linkend='var-bb-PN'><filename>PN</filename></link>
> variable will be set to “a”, and the
> - <link linkend='var-PV'><filename>PV</filename></link>
> + <link linkend='var-bb-PV'><filename>PV</filename></link>
> variable will be set to 1.1.
> </para>
>
> @@ -532,11 +532,11 @@
> <para>
> Dependencies are defined through several variables.
> You can find information about variables BitBake uses in
> - the <link linkend='ref-variables-glos'>Variables
> Glossary</link>
> + the <link linkend='ref-bb-variables-glos'>Variables
> Glossary</link> near the end of this manual.
> At a basic level, it is sufficient to know that BitBake
> uses the
> - <link
> linkend='var-DEPENDS'><filename>DEPENDS</filename></link> and
> - <link
> linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link> variables
> when
> + <link
> linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link> and
> + <link
> linkend='var-bb-RDEPENDS'><filename>RDEPENDS</filename></link>
> variables when calculating dependencies. </para>
>
> @@ -560,7 +560,7 @@
>
> <para>
> The build now starts with BitBake forking off threads up
> to the limit set in the
> - <link
> linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
> + <link
> linkend='var-bb-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
> variable. BitBake continues to fork threads as long as there are
> tasks ready to run, those tasks have all their dependencies met, and
> the thread threshold has not been @@ -574,7 +574,7 @@
>
> <para>
> As each task completes, a timestamp is written to the
> directory specified by the
> - <link
> linkend='var-STAMP'><filename>STAMP</filename></link> variable.
> + <link
> linkend='var-bb-STAMP'><filename>STAMP</filename></link> variable. On
> subsequent runs, BitBake looks in the build directory within
> <filename>tmp/stamps</filename> and does not rerun tasks that are
> already completed unless a timestamp is found to be invalid. @@
> -618,7 +618,7 @@ <para>
> Tasks can be either a shell task or a Python task.
> For shell tasks, BitBake writes a shell script to
> - <filename>${</filename><link
> linkend='var-T'><filename>T</filename></link><filename>}/run.do_taskname.pid</filename>
> + <filename>${</filename><link
> linkend='var-bb-T'><filename>T</filename></link><filename>}/run.do_taskname.pid</filename>
> and then executes the script. The generated shell script contains all
> the exported variables, and the shell functions with all variables
> expanded. @@ -645,10 +645,10 @@
> behavior:
> <itemizedlist>
> <listitem><para>
> - <link
> linkend='var-BB_SCHEDULER'><filename>BB_SCHEDULER</filename></link>
> + <link
> linkend='var-bb-BB_SCHEDULER'><filename>BB_SCHEDULER</filename></link>
> </para></listitem> <listitem><para>
> - <link
> linkend='var-BB_SCHEDULERS'><filename>BB_SCHEDULERS</filename></link>
> + <link
> linkend='var-bb-BB_SCHEDULERS'><filename>BB_SCHEDULERS</filename></link>
> </para></listitem> </itemizedlist>
> It is possible to have functions run before and after a
> task's main @@ -684,7 +684,7 @@
> The simplistic approach for excluding the working
> directory is to set it to some fixed value and create the checksum
> for the "run" script. BitBake goes one step better and uses the
> - <link
> linkend='var-BB_HASHBASE_WHITELIST'><filename>BB_HASHBASE_WHITELIST</filename></link>
> + <link
> linkend='var-bb-BB_HASHBASE_WHITELIST'><filename>BB_HASHBASE_WHITELIST</filename></link>
> variable to define a list of variables that should never be included
> when generating the signatures. </para>
> @@ -795,7 +795,7 @@
> This results in any metadata change that changes the
> task hash, automatically causing the task to be run again.
> This removes the need to bump
> - <link linkend='var-PR'><filename>PR</filename></link>
> + <link linkend='var-bb-PR'><filename>PR</filename></link>
> values, and changes to metadata automatically ripple
> across the build. </para>
>
> @@ -884,7 +884,7 @@
>
> <para>
> BitBake first calls the function defined by the
> - <link
> linkend='var-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></link>
> + <link
> linkend='var-bb-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></link>
> variable with a list of tasks and corresponding hashes it wants to
> build. This function is designed to be fast and returns a list
> @@ -908,7 +908,7 @@
> For example, it is pointless to obtain a compiler if you
> already have the compiled binary.
> To handle this, BitBake calls the
> - <link
> linkend='var-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></link>
> + <link
> linkend='var-bb-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></link>
> function for each successful setscene task to know whether or not it
> needs to obtain the dependencies of that task. </para>
> @@ -916,7 +916,7 @@
> <para>
> Finally, after all the setscene tasks have executed,
> BitBake calls the function listed in
> - <link
> linkend='var-BB_SETSCENE_VERIFY_FUNCTION2'><filename>BB_SETSCENE_VERIFY_FUNCTION2</filename></link>
> + <link
> linkend='var-bb-BB_SETSCENE_VERIFY_FUNCTION2'><filename>BB_SETSCENE_VERIFY_FUNCTION2</filename></link>
> with the list of tasks BitBake thinks has been "covered". The
> metadata can then ensure that this list is correct and can inform
> BitBake that it wants specific tasks to be run regardless diff --git
> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml
> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml
> index 29ae486..6840408 100644 ---
> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml
> +++
> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml @@
> -44,7 +44,7 @@ </literallayout> This code sets up an instance of the
> fetch class. The instance uses a space-separated list of URLs from the
> - <link
> linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
> + <link
> linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link> variable
> and then calls the <filename>download</filename> method to download
> the files. </para>
> @@ -78,7 +78,7 @@
> <listitem><para><emphasis>Pre-mirror
> Sites:</emphasis> BitBake first uses pre-mirrors to try and find
> source files. These locations are defined using the
> - <link
> linkend='var-PREMIRRORS'><filename>PREMIRRORS</filename></link>
> + <link
> linkend='var-bb-PREMIRRORS'><filename>PREMIRRORS</filename></link>
> variable. </para></listitem>
> <listitem><para><emphasis>Source URI:</emphasis>
> @@ -88,7 +88,7 @@
> <listitem><para><emphasis>Mirror Sites:</emphasis>
> If fetch failures occur, BitBake next uses
> mirror locations as defined by the
> - <link
> linkend='var-MIRRORS'><filename>MIRRORS</filename></link>
> + <link
> linkend='var-bb-MIRRORS'><filename>MIRRORS</filename></link> variable.
> </para></listitem>
> </itemizedlist>
> @@ -144,7 +144,7 @@
> Any source files that are not local (i.e.
> downloaded from the Internet) are placed into the
> download directory, which is specified by the
> - <link
> linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
> + <link
> linkend='var-bb-DL_DIR'><filename>DL_DIR</filename></link> variable.
> </para>
>
> @@ -184,11 +184,11 @@
>
> <para>
> If
> - <link
> linkend='var-BB_STRICT_CHECKSUM'><filename>BB_STRICT_CHECKSUM</filename></link>
> + <link
> linkend='var-bb-BB_STRICT_CHECKSUM'><filename>BB_STRICT_CHECKSUM</filename></link>
> is set, any download without a checksum triggers an error message.
> The
> - <link
> linkend='var-BB_NO_NETWORK'><filename>BB_NO_NETWORK</filename></link>
> + <link
> linkend='var-bb-BB_NO_NETWORK'><filename>BB_NO_NETWORK</filename></link>
> variable can be used to make any attempted network access a fatal
> error, which is useful for checking that mirrors are complete as well
> as other things. @@ -265,11 +265,11 @@
> The filename you specify within the URL can be
> either an absolute or relative path to a file.
> If the filename is relative, the contents of the
> - <link
> linkend='var-FILESPATH'><filename>FILESPATH</filename></link>
> + <link
> linkend='var-bb-FILESPATH'><filename>FILESPATH</filename></link>
> variable is used in the same way <filename>PATH</filename> is used to
> find executables. If the file cannot be found, it is assumed that it
> is available in
> - <link
> linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
> + <link
> linkend='var-bb-DL_DIR'><filename>DL_DIR</filename></link> by the
> time the <filename>download()</filename> method is called. </para>
>
> @@ -304,7 +304,7 @@
> allows the name of the downloaded file to be
> specified. Specifying the name of the downloaded file is useful
> for avoiding collisions in
> - <link
> linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
> + <link
> linkend='var-bb-DL_DIR'><filename>DL_DIR</filename></link> when
> dealing with multiple files that have the same name. </para>
>
> @@ -355,7 +355,7 @@
> A special value of "now" causes the checkout
> to be updated on every build.
> </para></listitem>
> - <listitem><para><emphasis><link
> linkend='var-CVSDIR'><filename>CVSDIR</filename></link>:</emphasis>
> + <listitem><para><emphasis><link
> linkend='var-bb-CVSDIR'><filename>CVSDIR</filename></link>:</emphasis>
> Specifies where a temporary checkout is saved. The location is often
> <filename>DL_DIR/cvs</filename>. </para></listitem>
> @@ -395,7 +395,7 @@
> <listitem><para><emphasis>"date":</emphasis>
> Specifies a date.
> If no "date" is specified, the
> - <link
> linkend='var-SRCDATE'><filename>SRCDATE</filename></link>
> + <link
> linkend='var-bb-SRCDATE'><filename>SRCDATE</filename></link> of the
> configuration is used to checkout a specific date. The special value
> of "now" causes the checkout to be updated on every build.
> @@ -406,7 +406,7 @@
> to which the module is unpacked.
> You are forcing the module into a special
> directory relative to
> - <link
> linkend='var-CVSDIR'><filename>CVSDIR</filename></link>.
> + <link
> linkend='var-bb-CVSDIR'><filename>CVSDIR</filename></link>.
> </para></listitem> <listitem><para><emphasis>"rsh"</emphasis>
> Used in conjunction with the "method"
> parameter. @@ -448,7 +448,7 @@
> <filename>FETCHCMD_svn</filename>, which defaults
> to "svn".
> The fetcher's temporary working directory is set by
> - <link
> linkend='var-SVNDIR'><filename>SVNDIR</filename></link>,
> + <link
> linkend='var-bb-SVNDIR'><filename>SVNDIR</filename></link>, which is
> usually <filename>DL_DIR/svn</filename>. </para>
>
> @@ -509,7 +509,7 @@
> source control system.
> The fetcher works by creating a bare clone of the
> remote into
> - <link
> linkend='var-GITDIR'><filename>GITDIR</filename></link>,
> + <link
> linkend='var-bb-GITDIR'><filename>GITDIR</filename></link>, which is
> usually <filename>DL_DIR/git2</filename>. This bare clone is then
> cloned into the work directory during the unpack stage when a
> specific tree is checked out. @@ -588,6 +588,14 @@
> The name of the path in which to place the
> checkout. By default, the path is <filename>git/</filename>.
> </para></listitem>
> + <listitem><para><emphasis>"usehead":</emphasis>
> + Enables local <filename>git://</filename>
> URLs to use the
> + current branch HEAD as the revision for use
> with
> + <filename>AUTOREV</filename>.
> + The "usehead" parameter implies no branch
> and only works
> + when the transfer protocol is
> + <filename>file://</filename>.
> + </para></listitem>
> </itemizedlist>
> Here are some example URLs:
> <literallayout class='monospaced'>
> @@ -604,7 +612,7 @@
> This fetcher submodule inherits from the
> <link linkend='git-fetcher'>Git fetcher</link> and
> extends that fetcher's behavior by fetching a repository's submodules.
> - <link
> linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
> + <link
> linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link> is
> passed to the Git fetcher as described in the "<link
> linkend='git-fetcher'>Git Fetcher
> (<filename>git://</filename>)</link>" section. @@ -639,9 +647,9 @@
>
> <para>
> To use this fetcher, make sure your recipe has proper
> - <link
> linkend='var-SRC_URI'><filename>SRC_URI</filename></link>,
> - <link
> linkend='var-SRCREV'><filename>SRCREV</filename></link>, and
> - <link
> linkend='var-PV'><filename>PV</filename></link> settings.
> + <link
> linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link>,
> + <link
> linkend='var-bb-SRCREV'><filename>SRCREV</filename></link>, and
> + <link
> linkend='var-bb-PV'><filename>PV</filename></link> settings. Here is
> an example: <literallayout class='monospaced'>
> SRC_URI =
> "ccrc://cc.example.org/ccrc;vob=/example_vob;module=/example_module"
> @@ -726,15 +734,15 @@ <filename>FETCHCMD_p4</filename>, which defaults
> to "p4".
> The fetcher's temporary working directory is set by
> - <link
> linkend='var-P4DIR'><filename>P4DIR</filename></link>,
> + <link
> linkend='var-bb-P4DIR'><filename>P4DIR</filename></link>, which
> defaults to "DL_DIR/p4". </para>
>
> <para>
> To use this fetcher, make sure your recipe has proper
> - <link
> linkend='var-SRC_URI'><filename>SRC_URI</filename></link>,
> - <link
> linkend='var-SRCREV'><filename>SRCREV</filename></link>, and
> - <link
> linkend='var-PV'><filename>PV</filename></link> values.
> + <link
> linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link>,
> + <link
> linkend='var-bb-SRCREV'><filename>SRCREV</filename></link>, and
> + <link
> linkend='var-bb-PV'><filename>PV</filename></link> values. The p4
> executable is able to use the config file defined by your system's
> <filename>P4CONFIG</filename> environment variable in order to define
> the Perforce server URL and port, username, and @@ -785,9 +793,9 @@
> <filename>google-repo</filename> source control
> system. The fetcher works by initiating and syncing sources of the
> repository into
> - <link
> linkend='var-REPODIR'><filename>REPODIR</filename></link>,
> + <link
> linkend='var-bb-REPODIR'><filename>REPODIR</filename></link>, which
> is usually
> - <link
> linkend='var-DL_DIR'><filename>DL_DIR</filename></link><filename>/repo</filename>.
> + <link
> linkend='var-bb-DL_DIR'><filename>DL_DIR</filename></link><filename>/repo</filename>.
> </para>
> <para>
> @@ -824,19 +832,22 @@
> Bazaar (<filename>bzr://</filename>)
> </para></listitem>
> <listitem><para>
> - Trees using Git Annex
> (<filename>gitannex://</filename>)
> + Mercurial (<filename>hg://</filename>)
> </para></listitem>
> <listitem><para>
> - Secure FTP (<filename>sftp://</filename>)
> + npm (<filename>npm://</filename>)
> </para></listitem>
> <listitem><para>
> - Secure Shell (<filename>ssh://</filename>)
> + OSC (<filename>osc://</filename>)
> </para></listitem>
> <listitem><para>
> - OSC (<filename>osc://</filename>)
> + Secure FTP (<filename>sftp://</filename>)
> </para></listitem>
> <listitem><para>
> - Mercurial (<filename>hg://</filename>)
> + Secure Shell (<filename>ssh://</filename>)
> + </para></listitem>
> + <listitem><para>
> + Trees using Git Annex
> (<filename>gitannex://</filename>) </para></listitem>
> </itemizedlist>
> No documentation currently exists for these lesser
> used diff --git
> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.xml
> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.xml index
> 9076f0f..39066e4 100644 ---
> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.xml +++
> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.xml @@
> -194,7 +194,7 @@ <para> When you run BitBake, it begins looking for
> metadata files. The
> - <link
> linkend='var-BBPATH'><filename>BBPATH</filename></link>
> + <link
> linkend='var-bb-BBPATH'><filename>BBPATH</filename></link> variable
> is what tells BitBake where to look for those files.
> <filename>BBPATH</filename> is not set and you need to set it.
> Without <filename>BBPATH</filename>, Bitbake cannot @@ -273,14
> +273,14 @@ some editor to create the <filename>bitbake.conf</filename>
> so that it contains the following:
> <literallayout class='monospaced'>
> - <link linkend='var-PN'>PN</link> =
> "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0]
> or 'defaultpkgname'}"
> + <link linkend='var-bb-PN'>PN</link> =
> "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0]
> or 'defaultpkgname'}" </literallayout> <literallayout
> class='monospaced'>
> - TMPDIR = "${<link linkend='var-TOPDIR'>TOPDIR</link>}/tmp"
> - <link linkend='var-CACHE'>CACHE</link> = "${TMPDIR}/cache"
> - <link linkend='var-STAMP'>STAMP</link> =
> "${TMPDIR}/${PN}/stamps"
> - <link linkend='var-T'>T</link> = "${TMPDIR}/${PN}/work"
> - <link linkend='var-B'>B</link> = "${TMPDIR}/${PN}"
> + TMPDIR = "${<link linkend='var-bb-TOPDIR'>TOPDIR</link>}/tmp"
> + <link linkend='var-bb-CACHE'>CACHE</link> = "${TMPDIR}/cache"
> + <link linkend='var-bb-STAMP'>STAMP</link> =
> "${TMPDIR}/${PN}/stamps"
> + <link linkend='var-bb-T'>T</link> = "${TMPDIR}/${PN}/work"
> + <link linkend='var-bb-B'>B</link> = "${TMPDIR}/${PN}"
> </literallayout>
> <note>
> Without a value for <filename>PN</filename>, the
> @@ -402,12 +402,12 @@
> Move to the <filename>conf</filename> directory and
> create a <filename>layer.conf</filename> file that has the following:
> <literallayout class='monospaced'>
> - BBPATH .= ":${<link linkend='var-LAYERDIR'>LAYERDIR</link>}"
> + BBPATH .= ":${<link linkend='var-bb-LAYERDIR'>LAYERDIR</link>}"
>
> - <link linkend='var-BBFILES'>BBFILES</link> += "${LAYERDIR}/*.bb"
> + <link linkend='var-bb-BBFILES'>BBFILES</link> +=
> "${LAYERDIR}/*.bb"
> - <link
> linkend='var-BBFILE_COLLECTIONS'>BBFILE_COLLECTIONS</link> +=
> "mylayer"
> - <link
> linkend='var-BBFILE_PATTERN'>BBFILE_PATTERN_mylayer</link> :=
> "^${LAYERDIR_RE}/"
> + <link
> linkend='var-bb-BBFILE_COLLECTIONS'>BBFILE_COLLECTIONS</link> +=
> "mylayer"
> + <link
> linkend='var-bb-BBFILE_PATTERN'>BBFILE_PATTERN_mylayer</link> :=
> "^${LAYERDIR_RE}/" </literallayout> For information on these
> variables, click the links to go to the definitions in the
> glossary.</para> @@ -416,9 +416,9 @@
> a recipe file named
> <filename>printhello.bb</filename> that has the following:
> <literallayout class='monospaced'>
> - <link linkend='var-DESCRIPTION'>DESCRIPTION</link> = "Prints
> Hello World"
> - <link linkend='var-PN'>PN</link> = 'printhello'
> - <link linkend='var-PV'>PV</link> = '1'
> + <link linkend='var-bb-DESCRIPTION'>DESCRIPTION</link> = "Prints
> Hello World"
> + <link linkend='var-bb-PN'>PN</link> = 'printhello'
> + <link linkend='var-bb-PV'>PV</link> = '1'
>
> python do_build() {
> bb.plain("********************");
> diff --git
> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml
> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml index
> f7d312a..8f2a960 100644 ---
> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml +++
> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml @@
> -781,7 +781,7 @@ target, you must also enable BitBake to perform
> multiple configuration builds. Enabling is accomplished by setting the
> - <link
> linkend='var-BBMULTICONFIG'><filename>BBMULTICONFIG</filename></link>
> + <link
> linkend='var-bb-BBMULTICONFIG'><filename>BBMULTICONFIG</filename></link>
> variable in the <filename>local.conf</filename> configuration file.
> As an example, suppose you had configuration
> files @@ -791,7 +791,7 @@
> The following statement in the
> <filename>local.conf</filename> file both enables
> BitBake to perform multiple configuration builds
> and
> - specifies the two multiconfigs:
> + specifies the two extra multiconfigs:
> <literallayout class='monospaced'>
> BBMULTICONFIG = "target1 target2"
> </literallayout>
> @@ -803,13 +803,13 @@
> builds, use the following command form to start
> the builds:
> <literallayout class='monospaced'>
> - $ bitbake
> [multiconfig:<replaceable>multiconfigname</replaceable>:]<replaceable>target</replaceable>
> [[[multiconfig:<replaceable>multiconfigname</replaceable>:]<replaceable>target</replaceable>] ... ]
> + $ bitbake
> [mc:<replaceable>multiconfigname</replaceable>:]<replaceable>target</replaceable>
> [[[mc:<replaceable>multiconfigname</replaceable>:]<replaceable>target</replaceable>] ... ]
> </literallayout>
> - Here is an example for two multiconfigs:
> + Here is an example for two extra multiconfigs:
> <filename>target1</filename> and
> <filename>target2</filename>:
> <literallayout class='monospaced'>
> - $ bitbake multiconfig:target1:<replaceable>target</replaceable>
> multiconfig:target2:<replaceable>target</replaceable>
> + $ bitbake mc::<replaceable>target</replaceable>
> mc:target1:<replaceable>target</replaceable>
> mc:target2:<replaceable>target</replaceable> </literallayout> </para>
> </section>
> @@ -837,13 +837,13 @@
> build, you must declare the dependencies in the
> recipe using the following statement form:
> <literallayout class='monospaced'>
> - <replaceable>task_or_package</replaceable>[mcdepends] =
> "multiconfig:<replaceable>from_multiconfig</replaceable>:<replaceable>to_multiconfig</replaceable>:<replaceable>recipe_name</replaceable>:<replaceable>task_on_which_to_depend</replaceable>"
> + <replaceable>task_or_package</replaceable>[mcdepends] =
> "mc:<replaceable>from_multiconfig</replaceable>:<replaceable>to_multiconfig</replaceable>:<replaceable>recipe_name</replaceable>:<replaceable>task_on_which_to_depend</replaceable>"
> </literallayout> To better show how to use this statement, consider an
> example with two multiconfigs:
> <filename>target1</filename> and <filename>target2</filename>:
> <literallayout class='monospaced'>
> - <replaceable>image_task</replaceable>[mcdepends] =
> "multiconfig:target1:target2:<replaceable>image2</replaceable>:<replaceable>rootfs_task</replaceable>"
> + <replaceable>image_task</replaceable>[mcdepends] =
> "mc:target1:target2:<replaceable>image2</replaceable>:<replaceable>rootfs_task</replaceable>"
> </literallayout> In this example, the
> <replaceable>from_multiconfig</replaceable> is
> "target1" and @@ -859,7 +859,7 @@
> Once you set up this dependency, you can build the
> "target1" multiconfig using a BitBake command as
> follows: <literallayout class='monospaced'>
> - $ bitbake multiconfig:target1:<replaceable>image1</replaceable>
> + $ bitbake mc:target1:<replaceable>image1</replaceable>
> </literallayout>
> This command executes all the tasks needed to
> create <replaceable>image1</replaceable> for the "target1"
> @@ -875,7 +875,7 @@
> Consider this change to the statement in the
> <replaceable>image1</replaceable> recipe:
> <literallayout class='monospaced'>
> - <replaceable>image_task</replaceable>[mcdepends] =
> "multiconfig:target1:target2:<replaceable>image2</replaceable>:<replaceable>image_task</replaceable>"
> + <replaceable>image_task</replaceable>[mcdepends] =
> "mc:target1:target2:<replaceable>image2</replaceable>:<replaceable>image_task</replaceable>"
> </literallayout> In this case, BitBake must create
> <replaceable>image2</replaceable> for the
> "target2" diff --git
> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml
> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml
> index 2490f6e..421364c 100644 ---
> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml
> +++
> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml @@
> -61,6 +61,78 @@ </para> </section>
> + <section id='modifying-existing-variables'>
> + <title>Modifying Existing Variables</title>
> +
> + <para>
> + Sometimes you need to modify existing variables.
> + Following are some cases where you might find you
> want to
> + modify an existing variable:
> + <itemizedlist>
> + <listitem><para>
> + Customize a recipe that uses the variable.
> + </para></listitem>
> + <listitem><para>
> + Change a variable's default value used in a
> + <filename>*.bbclass</filename> file.
> + </para></listitem>
> + <listitem><para>
> + Change the variable in a
> <filename>*.bbappend</filename>
> + file to override the variable in the
> original recipe.
> + </para></listitem>
> + <listitem><para>
> + Change the variable in a configuration file
> so that the
> + value overrides an existing configuration.
> + </para></listitem>
> + </itemizedlist>
> + </para>
> +
> + <para>
> + Changing a variable value can sometimes depend on
> how the
> + value was originally assigned and also on the desired
> + intent of the change.
> + In particular, when you append a value to a variable
> that
> + has a default value, the resulting value might not
> be what
> + you expect.
> + In this case, the value you provide might replace
> the value
> + rather than append to the default value.
> + </para>
> +
> + <para>
> + If after you have changed a variable's value and
> something
> + unexplained occurs, you can use BitBake to check the
> actual
> + value of the suspect variable.
> + You can make these checks for both configuration and
> recipe
> + level changes:
> + <itemizedlist>
> + <listitem><para>
> + For configuration changes, use the following:
> + <literallayout class='monospaced'>
> + $ bitbake -e
> + </literallayout>
> + This command displays variable values after
> the
> + configuration files (i.e.
> <filename>local.conf</filename>,
> + <filename>bblayers.conf</filename>,
> + <filename>bitbake.conf</filename> and so
> forth) have
> + been parsed.
> + <note>
> + Variables that are exported to the
> environment are
> + preceded by the string "export" in the
> command's
> + output.
> + </note>
> + </para></listitem>
> + <listitem><para>
> + For recipe changes, use the following:
> + <literallayout class='monospaced'>
> + $ bitbake <replaceable>recipe</replaceable> -e | grep VARIABLE="
> + </literallayout>
> + This command checks to see if the variable
> actually
> + makes it into a specific recipe.
> + </para></listitem>
> + </itemizedlist>
> + </para>
> + </section>
> +
> <section id='line-joining'>
> <title>Line Joining</title>
>
> @@ -297,9 +369,8 @@
>
> <para>
> These operators differ from the ":=", ".=", "=.",
> "+=", and "=+"
> - operators in that their effects are deferred
> - until after parsing completes rather than being
> immediately
> - applied.
> + operators in that their effects are applied at
> variable
> + expansion time rather than being immediately applied.
> Here are some examples:
> <literallayout class='monospaced'>
> B = "bval"
> @@ -348,18 +419,22 @@
> FOO = "123 456 789 123456 123 456 123 456"
> FOO_remove = "123"
> FOO_remove = "456"
> - FOO2 = "abc def ghi abcdef abc def abc def"
> - FOO2_remove = "abc def"
> + FOO2 = " abc def ghi abcdef abc def abc def def"
> + FOO2_remove = " \
> + def \
> + abc \
> + ghi \
> + "
> </literallayout>
> The variable <filename>FOO</filename> becomes
> - " 789 123456 "
> + " 789 123456 "
> and <filename>FOO2</filename> becomes
> - " ghi abcdef ".
> +
> " jkl abcdef ".
> </para>
> <para>
> Like "_append" and "_prepend", "_remove"
> - is deferred until after parsing completes.
> + is applied at variable expansion time.
> </para>
> </section>
>
> @@ -503,7 +578,7 @@
> </section>
>
> <section id='unsetting-variables'>
> - <title>Unseting variables</title>
> + <title>Unsetting variables</title>
>
> <para>
> It is possible to completely remove a variable or a
> variable flag @@ -595,7 +670,7 @@
>
> <para>
> BitBake uses
> - <link
> linkend='var-OVERRIDES'><filename>OVERRIDES</filename></link>
> + <link
> linkend='var-bb-OVERRIDES'><filename>OVERRIDES</filename></link> to
> control what variables are overridden after BitBake parses recipes
> and configuration files. This section describes how you can use
> @@ -705,7 +780,7 @@
>
> <para>Internally, this is implemented by
> prepending the task (e.g. "task-compile:") to the value of
> - <link
> linkend='var-OVERRIDES'><filename>OVERRIDES</filename></link>
> + <link
> linkend='var-bb-OVERRIDES'><filename>OVERRIDES</filename></link> for
> the local datastore of the <filename>do_compile</filename>
> task.</para>
> @@ -724,17 +799,15 @@
> <title>Key Expansion</title>
>
> <para>
> - Key expansion happens when the BitBake datastore is
> finalized
> - just before BitBake expands overrides.
> + Key expansion happens when the BitBake datastore is
> finalized. To better understand this, consider the following example:
> <literallayout class='monospaced'>
> A${B} = "X"
> B = "2"
> A2 = "Y"
> </literallayout>
> - In this case, after all the parsing is complete, and
> - before any overrides are handled, BitBake expands
> - <filename>${B}</filename> into "2".
> + In this case, after all the parsing is complete,
> + BitBake expands <filename>${B}</filename> into "2".
> This expansion causes <filename>A2</filename>, which
> was set to "Y" before the expansion, to become "X".
> </para>
> @@ -868,7 +941,7 @@
>
> <para>
> BitBake uses the
> - <link
> linkend='var-BBPATH'><filename>BBPATH</filename></link>
> + <link
> linkend='var-bb-BBPATH'><filename>BBPATH</filename></link> variable
> to locate needed include and class files. Additionally, BitBake
> searches the current directory for <filename>include</filename> and
> <filename>require</filename> @@ -1086,7 +1159,7 @@
> <para>
> When creating a configuration file
> (<filename>.conf</filename>), you can use the
> - <link
> linkend='var-INHERIT'><filename>INHERIT</filename></link>
> + <link
> linkend='var-bb-INHERIT'><filename>INHERIT</filename></link>
> configuration directive to inherit a class. BitBake only supports
> this directive when used within a configuration file.
> @@ -1370,7 +1443,7 @@
> </para></listitem>
> <listitem><para>
> BitBake-style Python functions generate a
> separate
> - <filename>${</filename><link
> linkend='var-T'><filename>T</filename></link><filename>}/run.</filename><replaceable>function-name</replaceable><filename>.</filename><replaceable>pid</replaceable>
> + <filename>${</filename><link
> linkend='var-bb-T'><filename>T</filename></link><filename>}/run.</filename><replaceable>function-name</replaceable><filename>.</filename><replaceable>pid</replaceable>
> script that is executed to run the function, and also generate a log
> file in
> <filename>${T}/log.</filename><replaceable>function-name</replaceable><filename>.</filename><replaceable>pid</replaceable>
> @@ -1773,7 +1846,7 @@ things exported or listed in its whitelist to
> ensure that the build environment is reproducible and consistent.
> You can prevent this "cleaning" by setting the
> - <link
> linkend='var-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link>
> + <link
> linkend='var-bb-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link>
> variable. </note>
> Consequently, if you do want something to get passed
> into the @@ -1783,9 +1856,9 @@
> Tell BitBake to load what you want from the
> environment into the datastore.
> You can do so through the
> - <link
> linkend='var-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>
> + <link
> linkend='var-bb-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>
> and
> - <link
> linkend='var-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link>
> + <link
> linkend='var-bb-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link>
> variables. For example, assume you want to prevent the build system
> from accessing your <filename>$HOME/.ccache</filename>
> @@ -1824,7 +1897,7 @@
> from the original execution environment.
> Bitbake saves a copy of the original environment into
> a special variable named
> - <link
> linkend='var-BB_ORIGENV'><filename>BB_ORIGENV</filename></link>.
> + <link
> linkend='var-bb-BB_ORIGENV'><filename>BB_ORIGENV</filename></link>.
> </para>
> <para>
> @@ -1883,7 +1956,7 @@
> <listitem><para><emphasis><filename>[depends]</filename>:</emphasis>
> Controls inter-task dependencies.
> See the
> - <link
> linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
> + <link
> linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link> variable
> and the "<link linkend='inter-task-dependencies'>Inter-Task
> Dependencies</link>" section for more information.
> @@ -1891,7 +1964,7 @@
> <listitem><para><emphasis><filename>[deptask]</filename>:</emphasis>
> Controls task build-time dependencies.
> See the
> - <link
> linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
> + <link
> linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link> variable
> and the "<link linkend='build-dependencies'>Build Dependencies</link>"
> section for more information.
> @@ -1937,7 +2010,7 @@
> of cores but certain tasks need to be
> rate-limited due to various kinds of resource constraints (e.g. to
> avoid network throttling). <filename>number_threads</filename> works
> similarly to the
> - <link
> linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
> + <link
> linkend='var-bb-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
> variable but is task-specific.</para>
> <para>Set the value globally.
> @@ -1971,9 +2044,9 @@
> <listitem><para><emphasis><filename>[rdepends]</filename>:</emphasis>
> Controls inter-task runtime dependencies.
> See the
> - <link
> linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link>
> + <link
> linkend='var-bb-RDEPENDS'><filename>RDEPENDS</filename></link>
> variable, the
> - <link
> linkend='var-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
> + <link
> linkend='var-bb-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
> variable, and the "<link linkend='inter-task-dependencies'>Inter-Task
> Dependencies</link>" section for more information.
> @@ -1981,9 +2054,9 @@
> <listitem><para><emphasis><filename>[rdeptask]</filename>:</emphasis>
> Controls task runtime dependencies.
> See the
> - <link
> linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link>
> + <link
> linkend='var-bb-RDEPENDS'><filename>RDEPENDS</filename></link>
> variable, the
> - <link
> linkend='var-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
> + <link
> linkend='var-bb-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
> variable, and the "<link linkend='runtime-dependencies'>Runtime
> Dependencies</link>" section for more information.
> @@ -1996,9 +2069,9 @@
> <listitem><para><emphasis><filename>[recrdeptask]</filename>:</emphasis>
> Controls task recursive runtime dependencies.
> See the
> - <link
> linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link>
> + <link
> linkend='var-bb-RDEPENDS'><filename>RDEPENDS</filename></link>
> variable, the
> - <link
> linkend='var-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
> + <link
> linkend='var-bb-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
> variable, and the "<link linkend='recursive-dependencies'>Recursive
> Dependencies</link>" section for more information.
> @@ -2127,7 +2200,7 @@
> Any given datastore only has one such event
> executed against it, however.
> If
> - <link
> linkende='var-BB_INVALIDCONF'><filename>BB_INVALIDCONF</filename></link>
> + <link
> linkende='var-bb-BB_INVALIDCONF'><filename>BB_INVALIDCONF</filename></link>
> is set in the datastore by the event handler, the configuration is
> reparsed and a new event triggered, allowing the metadata to update
> configuration. @@ -2256,17 +2329,17 @@
> from a single recipe file multiple incarnations of that
> recipe file where all incarnations are buildable.
> These features are enabled through the
> - <link
> linkend='var-BBCLASSEXTEND'><filename>BBCLASSEXTEND</filename></link>
> + <link
> linkend='var-bb-BBCLASSEXTEND'><filename>BBCLASSEXTEND</filename></link>
> and
> - <link
> linkend='var-BBVERSIONS'><filename>BBVERSIONS</filename></link>
> + <link
> linkend='var-bb-BBVERSIONS'><filename>BBVERSIONS</filename></link>
> variables. <note>
> The mechanism for this class extension is extremely
> specific to the implementation.
> Usually, the recipe's
> - <link
> linkend='var-PROVIDES'><filename>PROVIDES</filename></link>,
> - <link
> linkend='var-PN'><filename>PN</filename></link>, and
> - <link
> linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
> + <link
> linkend='var-bb-PROVIDES'><filename>PROVIDES</filename></link>,
> + <link
> linkend='var-bb-PN'><filename>PN</filename></link>, and
> + <link
> linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link>
> variables would need to be modified by the extension class. For
> specific examples, see the OE-Core <filename>native</filename>,
> <filename>nativesdk</filename>, @@ -2287,7 +2360,7 @@
> project from a single recipe file.
> You can also specify conditional metadata
> (using the
> - <link
> linkend='var-OVERRIDES'><filename>OVERRIDES</filename></link>
> + <link
> linkend='var-bb-OVERRIDES'><filename>OVERRIDES</filename></link>
> mechanism) for a single version, or an optionally named range of
> versions. Here is an example: <literallayout class='monospaced'>
> @@ -2306,7 +2379,7 @@
> into overrides, but it is also made available
> for the metadata to use in the variable that defines the base recipe
> versions for use in <filename>file://</filename> search paths
> - (<link
> linkend='var-FILESPATH'><filename>FILESPATH</filename></link>).
> + (<link
> linkend='var-bb-FILESPATH'><filename>FILESPATH</filename></link>).
> </para></listitem> </itemizedlist>
> </para>
> @@ -2408,7 +2481,7 @@
>
> <para>
> BitBake uses the
> - <link
> linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
> + <link
> linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link> variable
> to manage build time dependencies. The <filename>[deptask]</filename>
> varflag for tasks signifies the task of each
> @@ -2429,9 +2502,9 @@
>
> <para>
> BitBake uses the
> - <link
> linkend='var-PACKAGES'><filename>PACKAGES</filename></link>,
> - <link
> linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link>, and
> - <link
> linkend='var-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
> + <link
> linkend='var-bb-PACKAGES'><filename>PACKAGES</filename></link>,
> + <link
> linkend='var-bb-RDEPENDS'><filename>RDEPENDS</filename></link>, and
> + <link
> linkend='var-bb-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
> variables to manage runtime dependencies. </para>
>
> @@ -2686,7 +2759,7 @@
>
> <para>
> These checksums are stored in
> - <link
> linkend='var-STAMP'><filename>STAMP</filename></link>.
> + <link
> linkend='var-bb-STAMP'><filename>STAMP</filename></link>. You can
> examine the checksums using the following BitBake command:
> <literallayout class='monospaced'> $ bitbake-dumpsigs
> @@ -2708,44 +2781,44 @@
> The following list describes related variables:
> <itemizedlist>
> <listitem><para>
> - <link
> linkend='var-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></link>:
> + <link
> linkend='var-bb-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></link>:
> Specifies the name of the function to call during the "setscene" part
> of the task's execution in order to validate the list of task hashes.
> </para></listitem>
> <listitem><para>
> - <link
> linkend='var-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></link>:
> + <link
> linkend='var-bb-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></link>:
> Specifies a function BitBake calls that determines whether BitBake
> requires a setscene dependency to be met.
> </para></listitem>
> <listitem><para>
> - <link
> linkend='var-BB_SETSCENE_VERIFY_FUNCTION2'><filename>BB_SETSCENE_VERIFY_FUNCTION2</filename></link>:
> + <link
> linkend='var-bb-BB_SETSCENE_VERIFY_FUNCTION2'><filename>BB_SETSCENE_VERIFY_FUNCTION2</filename></link>:
> Specifies a function to call that verifies the list of planned task
> execution before the main task execution happens.
> </para></listitem>
> <listitem><para>
> - <link
> linkend='var-BB_STAMP_POLICY'><filename>BB_STAMP_POLICY</filename></link>:
> + <link
> linkend='var-bb-BB_STAMP_POLICY'><filename>BB_STAMP_POLICY</filename></link>:
> Defines the mode for comparing timestamps of stamp files.
> </para></listitem> <listitem><para>
> - <link
> linkend='var-BB_STAMP_WHITELIST'><filename>BB_STAMP_WHITELIST</filename></link>:
> + <link
> linkend='var-bb-BB_STAMP_WHITELIST'><filename>BB_STAMP_WHITELIST</filename></link>:
> Lists stamp files that are looked at when the stamp policy is
> "whitelist". </para></listitem>
> <listitem><para>
> - <link
> linkend='var-BB_TASKHASH'><filename>BB_TASKHASH</filename></link>:
> + <link
> linkend='var-bb-BB_TASKHASH'><filename>BB_TASKHASH</filename></link>:
> Within an executing task, this variable holds the hash of the task as
> returned by the currently enabled signature generator.
> </para></listitem>
> <listitem><para>
> - <link
> linkend='var-STAMP'><filename>STAMP</filename></link>:
> + <link
> linkend='var-bb-STAMP'><filename>STAMP</filename></link>: The base
> path to create stamp files. </para></listitem>
> <listitem><para>
> - <link
> linkend='var-STAMPCLEAN'><filename>STAMPCLEAN</filename></link>:
> + <link
> linkend='var-bb-STAMPCLEAN'><filename>STAMPCLEAN</filename></link>:
> Again, the base path to create stamp files but can use wildcards for
> matching a range of files for clean operations. </para></listitem>
> diff --git
> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml
> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml
> index a84b2bc..aca6741 100644 ---
> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml
> +++
> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml
> @@ -3,7 +3,7 @@ [<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
> <!-- Dummy chapter --> -<chapter id='ref-variables-glos'>
> +<chapter id='ref-bb-variables-glos'>
>
> <title>Variables Glossary</title>
>
> @@ -34,29 +34,29 @@
> </itemizedlist>
> </note>
>
> -<glossary id='ref-variables-glossary'>
> +<glossary id='ref-bb-variables-glossary'>
>
> <para>
> - <link linkend='var-ASSUME_PROVIDED'>A</link>
> - <link linkend='var-B'>B</link>
> - <link linkend='var-CACHE'>C</link>
> - <link linkend='var-DEFAULT_PREFERENCE'>D</link>
> - <link linkend='var-EXCLUDE_FROM_WORLD'>E</link>
> - <link linkend='var-FAKEROOT'>F</link>
> - <link linkend='var-GITDIR'>G</link>
> - <link linkend='var-HGDIR'>H</link>
> -<!-- <link linkend='var-ICECC_DISABLED'>I</link> -->
> + <link linkend='var-bb-ASSUME_PROVIDED'>A</link>
> + <link linkend='var-bb-B'>B</link>
> + <link linkend='var-bb-CACHE'>C</link>
> + <link linkend='var-bb-DEFAULT_PREFERENCE'>D</link>
> + <link linkend='var-bb-EXCLUDE_FROM_WORLD'>E</link>
> + <link linkend='var-bb-FAKEROOT'>F</link>
> + <link linkend='var-bb-GITDIR'>G</link>
> + <link linkend='var-bb-HGDIR'>H</link>
> + <link linkend='var-bb-INHERIT'>I</link>
> <!-- <link linkend='var-glossary-j'>J</link> -->
> <!-- <link linkend='var-KARCH'>K</link> -->
> - <link linkend='var-LAYERDEPENDS'>L</link>
> - <link linkend='var-MIRRORS'>M</link>
> + <link linkend='var-bb-LAYERDEPENDS'>L</link>
> + <link linkend='var-bb-MIRRORS'>M</link>
> <!-- <link linkend='var-glossary-n'>N</link> -->
> - <link linkend='var-OVERRIDES'>O</link>
> - <link linkend='var-P4DIR'>P</link>
> + <link linkend='var-bb-OVERRIDES'>O</link>
> + <link linkend='var-bb-P4DIR'>P</link>
> <!-- <link linkend='var-QMAKE_PROFILES'>Q</link> -->
> - <link linkend='var-RDEPENDS'>R</link>
> - <link linkend='var-SECTION'>S</link>
> - <link linkend='var-T'>T</link>
> + <link linkend='var-bb-RDEPENDS'>R</link>
> + <link linkend='var-bb-SECTION'>S</link>
> + <link linkend='var-bb-T'>T</link>
> <!-- <link linkend='var-UBOOT_CONFIG'>U</link> -->
> <!-- <link linkend='var-glossary-v'>V</link> -->
> <!-- <link linkend='var-WARN_QA'>W</link> -->
> @@ -65,13 +65,13 @@
> <!-- <link linkend='var-glossary-z'>Z</link>-->
> </para>
>
> - <glossdiv id='var-glossary-a'><title>A</title>
> + <glossdiv id='var-bb-glossary-a'><title>A</title>
>
> - <glossentry
> id='var-ASSUME_PROVIDED'><glossterm>ASSUME_PROVIDED</glossterm>
> + <glossentry
> id='var-bb-ASSUME_PROVIDED'><glossterm>ASSUME_PROVIDED</glossterm>
> <glossdef> <para>
> Lists recipe names
> - (<link
> linkend='var-PN'><filename>PN</filename></link>
> + (<link
> linkend='var-bb-PN'><filename>PN</filename></link> values) BitBake
> does not attempt to build. Instead, BitBake assumes these recipes
> have already been built.
> @@ -91,9 +91,9 @@
> </glossdiv>
>
>
> - <glossdiv id='var-glossary-b'><title>B</title>
> + <glossdiv id='var-bb-glossary-b'><title>B</title>
>
> - <glossentry id='var-B'><glossterm>B</glossterm>
> + <glossentry id='var-bb-B'><glossterm>B</glossterm>
> <glossdef>
> <para>
> The directory in which BitBake executes functions
> @@ -102,7 +102,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_ALLOWED_NETWORKS'><glossterm>BB_ALLOWED_NETWORKS</glossterm>
> + <glossentry
> id='var-bb-BB_ALLOWED_NETWORKS'><glossterm>BB_ALLOWED_NETWORKS</glossterm>
> <glossdef> <para>
> Specifies a space-delimited list of hosts that
> the fetcher @@ -111,7 +111,7 @@
> <itemizedlist>
> <listitem><para>
> This host list is only used if
> - <link
> linkend='var-BB_NO_NETWORK'><filename>BB_NO_NETWORK</filename></link>
> + <link
> linkend='var-bb-BB_NO_NETWORK'><filename>BB_NO_NETWORK</filename></link>
> is either not set or set to "0". </para></listitem>
> <listitem><para>
> @@ -151,13 +151,13 @@
> </itemizedlist>
> Using <filename>BB_ALLOWED_NETWORKS</filename> in
> conjunction with
> - <link
> linkend='var-PREMIRRORS'><filename>PREMIRRORS</filename></link>
> + <link
> linkend='var-bb-PREMIRRORS'><filename>PREMIRRORS</filename></link> is
> very useful. Adding the host you want to use to
> <filename>PREMIRRORS</filename> results in the
> source code being fetched from an allowed location and avoids raising
> an error when a host that is not allowed is in a
> - <link
> linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
> + <link
> linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link>
> statement. This is because the fetcher does not attempt to use the
> host listed in <filename>SRC_URI</filename>
> after a @@ -167,7 +167,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_CONSOLELOG'><glossterm>BB_CONSOLELOG</glossterm>
> + <glossentry
> id='var-bb-BB_CONSOLELOG'><glossterm>BB_CONSOLELOG</glossterm>
> <glossdef> <para>
> Specifies the path to a log file into which
> BitBake's user @@ -176,7 +176,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_CURRENTTASK'><glossterm>BB_CURRENTTASK</glossterm>
> + <glossentry
> id='var-bb-BB_CURRENTTASK'><glossterm>BB_CURRENTTASK</glossterm>
> <glossdef> <para>
> Contains the name of the currently running task.
> @@ -186,7 +186,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_DANGLINGAPPENDS_WARNONLY'><glossterm>BB_DANGLINGAPPENDS_WARNONLY</glossterm>
> + <glossentry
> id='var-bb-BB_DANGLINGAPPENDS_WARNONLY'><glossterm>BB_DANGLINGAPPENDS_WARNONLY</glossterm>
> <glossdef> <para>
> Defines how BitBake handles situations where an
> append @@ -208,7 +208,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_DEFAULT_TASK'><glossterm>BB_DEFAULT_TASK</glossterm>
> + <glossentry
> id='var-bb-BB_DEFAULT_TASK'><glossterm>BB_DEFAULT_TASK</glossterm>
> <glossdef> <para>
> The default task to use when none is specified
> (e.g. @@ -219,7 +219,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_DISKMON_DIRS'><glossterm>BB_DISKMON_DIRS</glossterm>
> + <glossentry
> id='var-bb-BB_DISKMON_DIRS'><glossterm>BB_DISKMON_DIRS</glossterm>
> <glossdef> <para>
> Monitors disk space and available inodes during
> the build @@ -245,7 +245,7 @@
> build when a threshold is broken.
> Subsequent warnings are issued as
> defined by the
> - <link
> linkend='var-BB_DISKMON_WARNINTERVAL'>BB_DISKMON_WARNINTERVAL</link>
> variable,
> + <link
> linkend='var-bb-BB_DISKMON_WARNINTERVAL'>BB_DISKMON_WARNINTERVAL</link>
> variable, which must be defined.
> <dir> is:
> @@ -275,7 +275,7 @@
> BB_DISKMON_DIRS = "ABORT,${TMPDIR},,100K"
> </literallayout>
> The first example works only if you also set
> - the <link
> linkend='var-BB_DISKMON_WARNINTERVAL'><filename>BB_DISKMON_WARNINTERVAL</filename></link>
> variable.
> + the <link
> linkend='var-bb-BB_DISKMON_WARNINTERVAL'><filename>BB_DISKMON_WARNINTERVAL</filename></link>
> variable. This example causes the build system to immediately abort
> when either the disk space in <filename>${TMPDIR}</filename> drops
> below 1 Gbyte or the available free inodes drops below @@ -309,7
> +309,7 @@ </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_DISKMON_WARNINTERVAL'><glossterm>BB_DISKMON_WARNINTERVAL</glossterm>
> + <glossentry
> id='var-bb-BB_DISKMON_WARNINTERVAL'><glossterm>BB_DISKMON_WARNINTERVAL</glossterm>
> <glossdef> <para>
> Defines the disk space and free inode warning
> intervals. @@ -319,7 +319,7 @@
> If you are going to use the
> <filename>BB_DISKMON_WARNINTERVAL</filename>
> variable, you must also use the
> - <link
> linkend='var-BB_DISKMON_DIRS'><filename>BB_DISKMON_DIRS</filename></link>
> variable
> + <link
> linkend='var-bb-BB_DISKMON_DIRS'><filename>BB_DISKMON_DIRS</filename></link>
> variable and define its action as "WARN". During the build,
> subsequent warnings are issued each time disk space or number of free
> inodes further reduces by @@ -374,7 +374,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_ENV_WHITELIST'><glossterm>BB_ENV_WHITELIST</glossterm>
> + <glossentry
> id='var-bb-BB_ENV_WHITELIST'><glossterm>BB_ENV_WHITELIST</glossterm>
> <glossdef> <para>
> Specifies the internal whitelist of variables to
> allow @@ -382,11 +382,11 @@
> datastore.
> If the value of this variable is not specified
> (which is the default), the following list is
> used:
> - <link
> linkend='var-BBPATH'><filename>BBPATH</filename></link>,
> - <link
> linkend='var-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link>,
> - <link
> linkend='var-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>,
> + <link
> linkend='var-bb-BBPATH'><filename>BBPATH</filename></link>,
> + <link
> linkend='var-bb-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link>,
> + <link
> linkend='var-bb-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>,
> and
> - <link
> linkend='var-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link>.
> + <link
> linkend='var-bb-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link>.
> <note> You must set this variable in the external environment
> in order for it to work.
> @@ -395,7 +395,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_ENV_EXTRAWHITE'><glossterm>BB_ENV_EXTRAWHITE</glossterm>
> + <glossentry
> id='var-bb-BB_ENV_EXTRAWHITE'><glossterm>BB_ENV_EXTRAWHITE</glossterm>
> <glossdef> <para>
> Specifies an additional set of variables to
> allow through @@ -403,7 +403,7 @@
> datastore.
> This list of variables are on top of the
> internal list set in
> - <link
> linkend='var-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>.
> + <link
> linkend='var-bb-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>.
> <note> You must set this variable in the external
> environment in order for it to work.
> @@ -412,22 +412,22 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_FETCH_PREMIRRORONLY'><glossterm>BB_FETCH_PREMIRRORONLY</glossterm>
> + <glossentry
> id='var-bb-BB_FETCH_PREMIRRORONLY'><glossterm>BB_FETCH_PREMIRRORONLY</glossterm>
> <glossdef> <para>
> When set to "1", causes BitBake's fetcher module
> to only search
> - <link
> linkend='var-PREMIRRORS'><filename>PREMIRRORS</filename></link>
> + <link
> linkend='var-bb-PREMIRRORS'><filename>PREMIRRORS</filename></link>
> for files. BitBake will not search the main
> - <link
> linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
> + <link
> linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link> or
> - <link
> linkend='var-MIRRORS'><filename>MIRRORS</filename></link>.
> + <link
> linkend='var-bb-MIRRORS'><filename>MIRRORS</filename></link>. </para>
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_FILENAME'><glossterm>BB_FILENAME</glossterm>
> + <glossentry
> id='var-bb-BB_FILENAME'><glossterm>BB_FILENAME</glossterm> <glossdef>
> <para>
> Contains the filename of the recipe that owns
> the currently @@ -440,12 +440,12 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_GENERATE_MIRROR_TARBALLS'><glossterm>BB_GENERATE_MIRROR_TARBALLS</glossterm>
> + <glossentry
> id='var-bb-BB_GENERATE_MIRROR_TARBALLS'><glossterm>BB_GENERATE_MIRROR_TARBALLS</glossterm>
> <glossdef> <para>
> Causes tarballs of the Git repositories,
> including the Git metadata, to be placed in the
> - <link
> linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
> + <link
> linkend='var-bb-DL_DIR'><filename>DL_DIR</filename></link> directory.
> Anyone wishing to create a source mirror would
> want to enable this variable.
> @@ -461,7 +461,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_HASHCONFIG_WHITELIST'><glossterm>BB_HASHCONFIG_WHITELIST</glossterm>
> + <glossentry
> id='var-bb-BB_HASHCONFIG_WHITELIST'><glossterm>BB_HASHCONFIG_WHITELIST</glossterm>
> <glossdef> <para>
> Lists variables that are excluded from base
> configuration @@ -485,7 +485,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_HASHBASE_WHITELIST'><glossterm>BB_HASHBASE_WHITELIST</glossterm>
> + <glossentry
> id='var-bb-BB_HASHBASE_WHITELIST'><glossterm>BB_HASHBASE_WHITELIST</glossterm>
> <glossdef> <para>
> Lists variables that are excluded from checksum
> and @@ -500,7 +500,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_HASHCHECK_FUNCTION'><glossterm>BB_HASHCHECK_FUNCTION</glossterm>
> + <glossentry
> id='var-bb-BB_HASHCHECK_FUNCTION'><glossterm>BB_HASHCHECK_FUNCTION</glossterm>
> <glossdef> <para>
> Specifies the name of the function to call
> during the @@ -524,7 +524,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_INVALIDCONF'><glossterm>BB_INVALIDCONF</glossterm>
> + <glossentry
> id='var-bb-BB_INVALIDCONF'><glossterm>BB_INVALIDCONF</glossterm>
> <glossdef> <para>
> Used in combination with the
> @@ -539,11 +539,11 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_LOGFMT'><glossterm>BB_LOGFMT</glossterm>
> + <glossentry
> id='var-bb-BB_LOGFMT'><glossterm>BB_LOGFMT</glossterm> <glossdef>
> <para>
> Specifies the name of the log files saved into
> - <filename>${</filename><link
> linkend='var-T'><filename>T</filename></link><filename>}</filename>.
> + <filename>${</filename><link
> linkend='var-bb-T'><filename>T</filename></link><filename>}</filename>.
> By default, the <filename>BB_LOGFMT</filename> variable is undefined
> and the log file names get created using the following form:
> @@ -556,7 +556,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_NICE_LEVEL'><glossterm>BB_NICE_LEVEL</glossterm>
> + <glossentry
> id='var-bb-BB_NICE_LEVEL'><glossterm>BB_NICE_LEVEL</glossterm>
> <glossdef> <para>
> Allows BitBake to run at a specific priority
> @@ -564,13 +564,13 @@
> System permissions usually mean that BitBake can
> reduce its priority but not raise it again.
> See
> - <link
> linkend='var-BB_TASK_NICE_LEVEL'><filename>BB_TASK_NICE_LEVEL</filename></link>
> + <link
> linkend='var-bb-BB_TASK_NICE_LEVEL'><filename>BB_TASK_NICE_LEVEL</filename></link>
> for additional information. </para>
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_NO_NETWORK'><glossterm>BB_NO_NETWORK</glossterm>
> + <glossentry
> id='var-bb-BB_NO_NETWORK'><glossterm>BB_NO_NETWORK</glossterm>
> <glossdef> <para>
> Disables network access in the BitBake fetcher
> modules. @@ -587,7 +587,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_NUMBER_THREADS'><glossterm>BB_NUMBER_THREADS</glossterm>
> + <glossentry
> id='var-bb-BB_NUMBER_THREADS'><glossterm>BB_NUMBER_THREADS</glossterm>
> <glossdef> <para>
> The maximum number of tasks BitBake should run
> in parallel @@ -599,7 +599,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_NUMBER_PARSE_THREADS'><glossterm>BB_NUMBER_PARSE_THREADS</glossterm>
> + <glossentry
> id='var-bb-BB_NUMBER_PARSE_THREADS'><glossterm>BB_NUMBER_PARSE_THREADS</glossterm>
> <glossdef> <para>
> Sets the number of threads BitBake uses when
> parsing. @@ -609,7 +609,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_ORIGENV'><glossterm>BB_ORIGENV</glossterm>
> + <glossentry
> id='var-bb-BB_ORIGENV'><glossterm>BB_ORIGENV</glossterm> <glossdef>
> <para>
> Contains a copy of the original external
> environment in @@ -625,7 +625,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_PRESERVE_ENV'><glossterm>BB_PRESERVE_ENV</glossterm>
> + <glossentry
> id='var-bb-BB_PRESERVE_ENV'><glossterm>BB_PRESERVE_ENV</glossterm>
> <glossdef> <para>
> Disables whitelisting and instead allows all
> variables @@ -639,12 +639,12 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_RUNFMT'><glossterm>BB_RUNFMT</glossterm>
> + <glossentry
> id='var-bb-BB_RUNFMT'><glossterm>BB_RUNFMT</glossterm> <glossdef>
> <para>
> Specifies the name of the executable script files
> (i.e. run files) saved into
> - <filename>${</filename><link
> linkend='var-T'><filename>T</filename></link><filename>}</filename>.
> + <filename>${</filename><link
> linkend='var-bb-T'><filename>T</filename></link><filename>}</filename>.
> By default, the <filename>BB_RUNFMT</filename> variable is undefined
> and the run file names get created using the following form:
> @@ -657,7 +657,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_RUNTASK'><glossterm>BB_RUNTASK</glossterm>
> + <glossentry
> id='var-bb-BB_RUNTASK'><glossterm>BB_RUNTASK</glossterm> <glossdef>
> <para>
> Contains the name of the currently executing
> task. @@ -669,7 +669,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_SCHEDULER'><glossterm>BB_SCHEDULER</glossterm>
> + <glossentry
> id='var-bb-BB_SCHEDULER'><glossterm>BB_SCHEDULER</glossterm>
> <glossdef> <para>
> Selects the name of the scheduler to use for the
> @@ -695,7 +695,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_SCHEDULERS'><glossterm>BB_SCHEDULERS</glossterm>
> + <glossentry
> id='var-bb-BB_SCHEDULERS'><glossterm>BB_SCHEDULERS</glossterm>
> <glossdef> <para>
> Defines custom schedulers to import.
> @@ -705,13 +705,13 @@
>
> <para>
> For information how to select a scheduler, see
> the
> - <link
> linkend='var-BB_SCHEDULER'><filename>BB_SCHEDULER</filename></link>
> + <link
> linkend='var-bb-BB_SCHEDULER'><filename>BB_SCHEDULER</filename></link>
> variable. </para>
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_SETSCENE_DEPVALID'><glossterm>BB_SETSCENE_DEPVALID</glossterm>
> + <glossentry
> id='var-bb-BB_SETSCENE_DEPVALID'><glossterm>BB_SETSCENE_DEPVALID</glossterm>
> <glossdef> <para>
> Specifies a function BitBake calls that
> determines @@ -731,7 +731,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_SETSCENE_VERIFY_FUNCTION2'><glossterm>BB_SETSCENE_VERIFY_FUNCTION2</glossterm>
> + <glossentry
> id='var-bb-BB_SETSCENE_VERIFY_FUNCTION2'><glossterm>BB_SETSCENE_VERIFY_FUNCTION2</glossterm>
> <glossdef> <para>
> Specifies a function to call that verifies the
> list of @@ -752,7 +752,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_SIGNATURE_EXCLUDE_FLAGS'><glossterm>BB_SIGNATURE_EXCLUDE_FLAGS</glossterm>
> + <glossentry
> id='var-bb-BB_SIGNATURE_EXCLUDE_FLAGS'><glossterm>BB_SIGNATURE_EXCLUDE_FLAGS</glossterm>
> <glossdef> <para>
> Lists variable flags (varflags)
> @@ -771,7 +771,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_SIGNATURE_HANDLER'><glossterm>BB_SIGNATURE_HANDLER</glossterm>
> + <glossentry
> id='var-bb-BB_SIGNATURE_HANDLER'><glossterm>BB_SIGNATURE_HANDLER</glossterm>
> <glossdef> <para>
> Defines the name of the signature handler
> BitBake uses. @@ -790,7 +790,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_SRCREV_POLICY'><glossterm>BB_SRCREV_POLICY</glossterm>
> + <glossentry
> id='var-bb-BB_SRCREV_POLICY'><glossterm>BB_SRCREV_POLICY</glossterm>
> <glossdef> <para>
> Defines the behavior of the fetcher when it
> interacts with @@ -817,7 +817,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_STAMP_POLICY'><glossterm>BB_STAMP_POLICY</glossterm>
> + <glossentry
> id='var-bb-BB_STAMP_POLICY'><glossterm>BB_STAMP_POLICY</glossterm>
> <glossdef> <para>
> Defines the mode used for how timestamps of
> stamp files @@ -836,7 +836,7 @@
> <listitem><para><emphasis>whitelist</emphasis>
> - Identical to "full" mode except timestamp
> comparisons are made for recipes listed
> in the
> - <link
> linkend='var-BB_STAMP_WHITELIST'><filename>BB_STAMP_WHITELIST</filename></link>
> + <link
> linkend='var-bb-BB_STAMP_WHITELIST'><filename>BB_STAMP_WHITELIST</filename></link>
> variable. </para></listitem>
> </itemizedlist>
> @@ -848,19 +848,19 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_STAMP_WHITELIST'><glossterm>BB_STAMP_WHITELIST</glossterm>
> + <glossentry
> id='var-bb-BB_STAMP_WHITELIST'><glossterm>BB_STAMP_WHITELIST</glossterm>
> <glossdef> <para>
> Lists files whose stamp file timestamps are
> compared when the stamp policy mode is set to "whitelist".
> For information on stamp policies, see the
> - <link
> linkend='var-BB_STAMP_POLICY'><filename>BB_STAMP_POLICY</filename></link>
> + <link
> linkend='var-bb-BB_STAMP_POLICY'><filename>BB_STAMP_POLICY</filename></link>
> variable. </para>
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_STRICT_CHECKSUM'><glossterm>BB_STRICT_CHECKSUM</glossterm>
> + <glossentry
> id='var-bb-BB_STRICT_CHECKSUM'><glossterm>BB_STRICT_CHECKSUM</glossterm>
> <glossdef> <para>
> Sets a more strict checksum mechanism for
> non-local URLs. @@ -871,7 +871,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_TASK_IONICE_LEVEL'><glossterm>BB_TASK_IONICE_LEVEL</glossterm>
> + <glossentry
> id='var-bb-BB_TASK_IONICE_LEVEL'><glossterm>BB_TASK_IONICE_LEVEL</glossterm>
> <glossdef> <para>
> Allows adjustment of a task's Input/Output
> priority. @@ -882,7 +882,7 @@
> variable to adjust the I/O priority of these
> tasks. <note>
> This variable works similarly to the
> - <link
> linkend='var-BB_TASK_NICE_LEVEL'><filename>BB_TASK_NICE_LEVEL</filename></link>
> + <link
> linkend='var-bb-BB_TASK_NICE_LEVEL'><filename>BB_TASK_NICE_LEVEL</filename></link>
> variable except with a task's I/O priorities. </note>
> </para>
> @@ -921,7 +921,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_TASK_NICE_LEVEL'><glossterm>BB_TASK_NICE_LEVEL</glossterm>
> + <glossentry
> id='var-bb-BB_TASK_NICE_LEVEL'><glossterm>BB_TASK_NICE_LEVEL</glossterm>
> <glossdef> <para>
> Allows specific tasks to change their priority
> @@ -940,7 +940,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_TASKHASH'><glossterm>BB_TASKHASH</glossterm>
> + <glossentry
> id='var-bb-BB_TASKHASH'><glossterm>BB_TASKHASH</glossterm> <glossdef>
> <para>
> Within an executing task, this variable holds
> the hash @@ -950,7 +950,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_VERBOSE_LOGS'><glossterm>BB_VERBOSE_LOGS</glossterm>
> + <glossentry
> id='var-bb-BB_VERBOSE_LOGS'><glossterm>BB_VERBOSE_LOGS</glossterm>
> <glossdef> <para>
> Controls how verbose BitBake is during builds.
> @@ -960,7 +960,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BB_WORKERCONTEXT'><glossterm>BB_WORKERCONTEXT</glossterm>
> + <glossentry
> id='var-bb-BB_WORKERCONTEXT'><glossterm>BB_WORKERCONTEXT</glossterm>
> <glossdef> <para>
> Specifies if the current context is executing a
> task. @@ -973,7 +973,7 @@
> </glossentry>
>
>
> - <glossentry
> id='var-BBCLASSEXTEND'><glossterm>BBCLASSEXTEND</glossterm>
> + <glossentry
> id='var-bb-BBCLASSEXTEND'><glossterm>BBCLASSEXTEND</glossterm>
> <glossdef> <para>
> Allows you to extend a recipe so that it builds
> variants @@ -1009,7 +1009,7 @@
> <filename>_class-native</filename>.
> For example, to generate a native version of
> a recipe, a
> - <link
> linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
> + <link
> linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link> on "foo"
> is rewritten to a <filename>DEPENDS</filename> on "foo-native".
> </para>
> @@ -1028,7 +1028,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-BBDEBUG'><glossterm>BBDEBUG</glossterm>
> + <glossentry
> id='var-bb-BBDEBUG'><glossterm>BBDEBUG</glossterm> <glossdef>
> <para>
> Sets the BitBake debug output level to a
> specific value @@ -1042,7 +1042,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BBFILE_COLLECTIONS'><glossterm>BBFILE_COLLECTIONS</glossterm>
> + <glossentry
> id='var-bb-BBFILE_COLLECTIONS'><glossterm>BBFILE_COLLECTIONS</glossterm>
> <glossdef> <para>Lists the names of configured layers.
> These names are used to find the other
> <filename>BBFILE_*</filename> @@ -1053,10 +1053,10 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BBFILE_PATTERN'><glossterm>BBFILE_PATTERN</glossterm>
> + <glossentry
> id='var-bb-BBFILE_PATTERN'><glossterm>BBFILE_PATTERN</glossterm>
> <glossdef> <para>Variable that expands to match files from
> - <link
> linkend='var-BBFILES'><filename>BBFILES</filename></link>
> + <link
> linkend='var-bb-BBFILES'><filename>BBFILES</filename></link> in a
> particular layer. This variable is used in the
> <filename>conf/layer.conf</filename> file and must be suffixed with
> the name of the specific layer (e.g. @@ -1064,7 +1064,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BBFILE_PRIORITY'><glossterm>BBFILE_PRIORITY</glossterm>
> + <glossentry
> id='var-bb-BBFILE_PRIORITY'><glossterm>BBFILE_PRIORITY</glossterm>
> <glossdef> <para>Assigns the priority for recipe files in each
> layer.</para> <para>This variable is useful in situations where the
> same recipe appears in @@ -1074,7 +1074,7 @@
> letting you control the precedence for the
> multiple layers. The precedence established through this variable
> stands regardless of a recipe's version
> - (<link
> linkend='var-PV'><filename>PV</filename></link> variable).
> + (<link
> linkend='var-bb-PV'><filename>PV</filename></link> variable). For
> example, a layer that has a recipe with a higher
> <filename>PV</filename> value but for which the
> <filename>BBFILE_PRIORITY</filename> is set to have a lower
> precedence still has a lower precedence.</para> @@ -1083,7 +1083,7 @@
> For example, the value 6 has a higher precedence than the value 5. If
> not specified, the <filename>BBFILE_PRIORITY</filename> variable is
> set based on layer dependencies (see the
> - <filename><link
> linkend='var-LAYERDEPENDS'>LAYERDEPENDS</link></filename> variable for
> + <filename><link
> linkend='var-bb-LAYERDEPENDS'>LAYERDEPENDS</link></filename> variable
> for more information. The default priority, if unspecified
> for a layer with no dependencies, is the lowest
> defined priority + 1 @@ -1095,7 +1095,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-BBFILES'><glossterm>BBFILES</glossterm>
> + <glossentry
> id='var-bb-BBFILES'><glossterm>BBFILES</glossterm> <glossdef>
> <para>
> A space-separated list of recipe files BitBake
> uses to @@ -1113,7 +1113,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BBINCLUDED'><glossterm>BBINCLUDED</glossterm>
> + <glossentry
> id='var-bb-BBINCLUDED'><glossterm>BBINCLUDED</glossterm> <glossdef>
> <para>
> Contains a space-separated list of all of all
> files that @@ -1123,7 +1123,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BBINCLUDELOGS'><glossterm>BBINCLUDELOGS</glossterm>
> + <glossentry
> id='var-bb-BBINCLUDELOGS'><glossterm>BBINCLUDELOGS</glossterm>
> <glossdef> <para>
> If set to a value, enables printing the task log
> when @@ -1132,11 +1132,11 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BBINCLUDELOGS_LINES'><glossterm>BBINCLUDELOGS_LINES</glossterm>
> + <glossentry
> id='var-bb-BBINCLUDELOGS_LINES'><glossterm>BBINCLUDELOGS_LINES</glossterm>
> <glossdef> <para>
> If
> - <link
> linkend='var-BBINCLUDELOGS'><filename>BBINCLUDELOGS</filename></link>
> + <link
> linkend='var-bb-BBINCLUDELOGS'><filename>BBINCLUDELOGS</filename></link>
> is set, specifies the maximum number of lines from the task log file
> to print when reporting a failed task. If you do not set
> <filename>BBINCLUDELOGS_LINES</filename>, @@ -1145,7 +1145,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-BBLAYERS'><glossterm>BBLAYERS</glossterm>
> + <glossentry
> id='var-bb-BBLAYERS'><glossterm>BBLAYERS</glossterm> <glossdef>
> <para>Lists the layers to enable during the build.
> This variable is defined in the
> <filename>bblayers.conf</filename> configuration @@ -1166,7 +1166,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BBLAYERS_FETCH_DIR'><glossterm>BBLAYERS_FETCH_DIR</glossterm>
> + <glossentry
> id='var-bb-BBLAYERS_FETCH_DIR'><glossterm>BBLAYERS_FETCH_DIR</glossterm>
> <glossdef> <para>
> Sets the base location where layers are stored.
> @@ -1178,7 +1178,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-BBMASK'><glossterm>BBMASK</glossterm>
> + <glossentry id='var-bb-BBMASK'><glossterm>BBMASK</glossterm>
> <glossdef>
> <para>
> Prevents BitBake from processing recipes and
> recipe @@ -1236,7 +1236,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BBMULTICONFIG'><glossterm>BBMULTICONFIG</glossterm>
> + <glossentry
> id='var-bb-BBMULTICONFIG'><glossterm>BBMULTICONFIG</glossterm> <info>
> BBMULTICONFIG[doc] = "Enables BitBake to perform
> multiple configuration builds and lists each separate configuration
> (multiconfig)." </info> @@ -1275,7 +1275,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-BBPATH'><glossterm>BBPATH</glossterm>
> + <glossentry id='var-bb-BBPATH'><glossterm>BBPATH</glossterm>
> <glossdef>
> <para>
> Used by BitBake to locate class
> @@ -1302,7 +1302,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-BBSERVER'><glossterm>BBSERVER</glossterm>
> + <glossentry
> id='var-bb-BBSERVER'><glossterm>BBSERVER</glossterm> <glossdef>
> <para>
> Points to the server that runs memory-resident
> BitBake. @@ -1312,7 +1312,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BBTARGETS'><glossterm>BBTARGETS</glossterm>
> + <glossentry
> id='var-bb-BBTARGETS'><glossterm>BBTARGETS</glossterm> <glossdef>
> <para>
> Allows you to use a configuration file to add to
> the list @@ -1321,14 +1321,14 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BBVERSIONS'><glossterm>BBVERSIONS</glossterm>
> + <glossentry
> id='var-bb-BBVERSIONS'><glossterm>BBVERSIONS</glossterm> <glossdef>
> <para>
> Allows a single recipe to build multiple
> versions of a project from a single recipe file.
> You also able to specify conditional metadata
> using the
> - <link
> linkend='var-OVERRIDES'><filename>OVERRIDES</filename></link>
> + <link
> linkend='var-bb-OVERRIDES'><filename>OVERRIDES</filename></link>
> mechanism for a single version or for an optionally named range of
> versions. </para>
> @@ -1342,7 +1342,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BITBAKE_UI'><glossterm>BITBAKE_UI</glossterm>
> + <glossentry
> id='var-bb-BITBAKE_UI'><glossterm>BITBAKE_UI</glossterm> <glossdef>
> <para>
> Used to specify the UI module to use when
> running BitBake. @@ -1356,7 +1356,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-BUILDNAME'><glossterm>BUILDNAME</glossterm>
> + <glossentry
> id='var-bb-BUILDNAME'><glossterm>BUILDNAME</glossterm> <glossdef>
> <para>
> A name assigned to the build.
> @@ -1366,7 +1366,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-BZRDIR'><glossterm>BZRDIR</glossterm>
> + <glossentry id='var-bb-BZRDIR'><glossterm>BZRDIR</glossterm>
> <glossdef>
> <para>
> The directory in which files checked out of a
> Bazaar @@ -1377,9 +1377,9 @@
>
> </glossdiv>
>
> - <glossdiv id='var-glossary-c'><title>C</title>
> + <glossdiv id='var-bb-glossary-c'><title>C</title>
>
> - <glossentry id='var-CACHE'><glossterm>CACHE</glossterm>
> + <glossentry id='var-bb-CACHE'><glossterm>CACHE</glossterm>
> <glossdef>
> <para>
> Specifies the directory BitBake uses to store a
> cache @@ -1389,7 +1389,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-CVSDIR'><glossterm>CVSDIR</glossterm>
> + <glossentry id='var-bb-CVSDIR'><glossterm>CVSDIR</glossterm>
> <glossdef>
> <para>
> The directory in which files checked out under
> the @@ -1400,9 +1400,9 @@
>
> </glossdiv>
>
> - <glossdiv id='var-glossary-d'><title>D</title>
> + <glossdiv id='var-bb-glossary-d'><title>D</title>
>
> - <glossentry
> id='var-DEFAULT_PREFERENCE'><glossterm>DEFAULT_PREFERENCE</glossterm>
> + <glossentry
> id='var-bb-DEFAULT_PREFERENCE'><glossterm>DEFAULT_PREFERENCE</glossterm>
> <glossdef> <para>
> Specifies a weak bias for recipe selection
> priority. @@ -1413,20 +1413,20 @@
> piece of software.
> Using the variable in this way causes the stable
> version of the recipe to build by default in the absence of
> - <filename><link
> linkend='var-PREFERRED_VERSION'>PREFERRED_VERSION</link></filename>
> + <filename><link
> linkend='var-bb-PREFERRED_VERSION'>PREFERRED_VERSION</link></filename>
> being used to build the development version. </para>
> <note>
> The bias provided by
> <filename>DEFAULT_PREFERENCE</filename> is weak and is overridden by
> - <filename><link
> linkend='var-BBFILE_PRIORITY'>BBFILE_PRIORITY</link></filename>
> + <filename><link
> linkend='var-bb-BBFILE_PRIORITY'>BBFILE_PRIORITY</link></filename> if
> that variable is different between two layers that contain different
> versions of the same recipe. </note>
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-DEPENDS'><glossterm>DEPENDS</glossterm>
> + <glossentry
> id='var-bb-DEPENDS'><glossterm>DEPENDS</glossterm> <glossdef>
> <para>
> Lists a recipe's build-time dependencies
> @@ -1451,13 +1451,13 @@
>
> <para>
> For information on runtime dependencies, see the
> - <link
> linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link>
> + <link
> linkend='var-bb-RDEPENDS'><filename>RDEPENDS</filename></link>
> variable. </para>
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-DESCRIPTION'><glossterm>DESCRIPTION</glossterm>
> + <glossentry
> id='var-bb-DESCRIPTION'><glossterm>DESCRIPTION</glossterm> <glossdef>
> <para>
> A long description for the recipe.
> @@ -1465,7 +1465,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-DL_DIR'><glossterm>DL_DIR</glossterm>
> + <glossentry id='var-bb-DL_DIR'><glossterm>DL_DIR</glossterm>
> <glossdef>
> <para>
> The central download directory used by the build
> process to @@ -1474,7 +1474,7 @@
> suitable for mirroring for everything except Git
> repositories.
> If you want tarballs of Git repositories, use the
> - <link
> linkend='var-BB_GENERATE_MIRROR_TARBALLS'><filename>BB_GENERATE_MIRROR_TARBALLS</filename></link>
> + <link
> linkend='var-bb-BB_GENERATE_MIRROR_TARBALLS'><filename>BB_GENERATE_MIRROR_TARBALLS</filename></link>
> variable. </para>
> </glossdef>
> @@ -1482,9 +1482,9 @@
> </glossentry>
> </glossdiv>
>
> - <glossdiv id='var-glossary-e'><title>E</title>
> + <glossdiv id='var-bb-glossary-e'><title>E</title>
>
> - <glossentry
> id='var-EXCLUDE_FROM_WORLD'><glossterm>EXCLUDE_FROM_WORLD</glossterm>
> + <glossentry
> id='var-bb-EXCLUDE_FROM_WORLD'><glossterm>EXCLUDE_FROM_WORLD</glossterm>
> <glossdef> <para>
> Directs BitBake to exclude a recipe from world
> builds (i.e. @@ -1512,9 +1512,9 @@
>
> </glossdiv>
>
> - <glossdiv id='var-glossary-f'><title>F</title>
> + <glossdiv id='var-bb-glossary-f'><title>F</title>
>
> - <glossentry id='var-FAKEROOT'><glossterm>FAKEROOT</glossterm>
> + <glossentry
> id='var-bb-FAKEROOT'><glossterm>FAKEROOT</glossterm> <glossdef>
> <para>
> Contains the command to use when running a
> shell script @@ -1527,19 +1527,19 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-FAKEROOTBASEENV'><glossterm>FAKEROOTBASEENV</glossterm>
> + <glossentry
> id='var-bb-FAKEROOTBASEENV'><glossterm>FAKEROOTBASEENV</glossterm>
> <glossdef> <para>
> Lists environment variables to set when
> executing the command defined by
> - <link
> linkend='var-FAKEROOTCMD'><filename>FAKEROOTCMD</filename></link>
> + <link
> linkend='var-bb-FAKEROOTCMD'><filename>FAKEROOTCMD</filename></link>
> that starts the bitbake-worker process in the fakeroot environment.
> </para>
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-FAKEROOTCMD'><glossterm>FAKEROOTCMD</glossterm>
> + <glossentry
> id='var-bb-FAKEROOTCMD'><glossterm>FAKEROOTCMD</glossterm> <glossdef>
> <para>
> Contains the command that starts the
> bitbake-worker @@ -1548,7 +1548,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-FAKEROOTDIRS'><glossterm>FAKEROOTDIRS</glossterm>
> + <glossentry
> id='var-bb-FAKEROOTDIRS'><glossterm>FAKEROOTDIRS</glossterm>
> <glossdef> <para>
> Lists directories to create before running a
> task in @@ -1557,33 +1557,33 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-FAKEROOTENV'><glossterm>FAKEROOTENV</glossterm>
> + <glossentry
> id='var-bb-FAKEROOTENV'><glossterm>FAKEROOTENV</glossterm> <glossdef>
> <para>
> Lists environment variables to set when running
> a task in the fakeroot environment.
> For additional information on environment
> variables and the fakeroot environment, see the
> - <link
> linkend='var-FAKEROOTBASEENV'><filename>FAKEROOTBASEENV</filename></link>
> + <link
> linkend='var-bb-FAKEROOTBASEENV'><filename>FAKEROOTBASEENV</filename></link>
> variable. </para>
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-FAKEROOTNOENV'><glossterm>FAKEROOTNOENV</glossterm>
> + <glossentry
> id='var-bb-FAKEROOTNOENV'><glossterm>FAKEROOTNOENV</glossterm>
> <glossdef> <para>
> Lists environment variables to set when running
> a task that is not in the fakeroot environment.
> For additional information on environment
> variables and the fakeroot environment, see the
> - <link
> linkend='var-FAKEROOTENV'><filename>FAKEROOTENV</filename></link>
> + <link
> linkend='var-bb-FAKEROOTENV'><filename>FAKEROOTENV</filename></link>
> variable. </para>
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-FETCHCMD'><glossterm>FETCHCMD</glossterm>
> + <glossentry
> id='var-bb-FETCHCMD'><glossterm>FETCHCMD</glossterm> <glossdef>
> <para>
> Defines the command the BitBake fetcher module
> @@ -1595,7 +1595,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-FILE'><glossterm>FILE</glossterm>
> + <glossentry id='var-bb-FILE'><glossterm>FILE</glossterm>
> <glossdef>
> <para>
> Points at the current file.
> @@ -1607,7 +1607,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-FILESPATH'><glossterm>FILESPATH</glossterm>
> + <glossentry
> id='var-bb-FILESPATH'><glossterm>FILESPATH</glossterm> <glossdef>
> <para>
> Specifies directories BitBake uses when
> searching for @@ -1625,9 +1625,9 @@
> </glossdiv>
>
>
> - <glossdiv id='var-glossary-g'><title>G</title>
> + <glossdiv id='var-bb-glossary-g'><title>G</title>
>
> - <glossentry id='var-GITDIR'><glossterm>GITDIR</glossterm>
> + <glossentry id='var-bb-GITDIR'><glossterm>GITDIR</glossterm>
> <glossdef>
> <para>
> The directory in which a local copy of a Git
> repository @@ -1639,9 +1639,9 @@
> </glossdiv>
>
>
> - <glossdiv id='var-glossary-h'><title>H</title>
> + <glossdiv id='var-bb-glossary-h'><title>H</title>
>
> - <glossentry id='var-HGDIR'><glossterm>HGDIR</glossterm>
> + <glossentry id='var-bb-HGDIR'><glossterm>HGDIR</glossterm>
> <glossdef>
> <para>
> The directory in which files checked out of a
> Mercurial @@ -1650,7 +1650,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-HOMEPAGE'><glossterm>HOMEPAGE</glossterm>
> + <glossentry
> id='var-bb-HOMEPAGE'><glossterm>HOMEPAGE</glossterm> <glossdef>
> <para>Website where more information about the
> software the recipe is building can be found.</para>
> @@ -1659,9 +1659,9 @@
>
> </glossdiv>
>
> - <glossdiv id='var-glossary-i'><title>I</title>
> + <glossdiv id='var-bb-glossary-i'><title>I</title>
>
> - <glossentry id='var-INHERIT'><glossterm>INHERIT</glossterm>
> + <glossentry
> id='var-bb-INHERIT'><glossterm>INHERIT</glossterm> <glossdef>
> <para>
> Causes the named class or classes to be
> inherited globally. @@ -1691,15 +1691,15 @@
> </glossdiv>
> -->
>
> - <glossdiv id='var-glossary-l'><title>L</title>
> + <glossdiv id='var-bb-glossary-l'><title>L</title>
>
> - <glossentry
> id='var-LAYERDEPENDS'><glossterm>LAYERDEPENDS</glossterm>
> + <glossentry
> id='var-bb-LAYERDEPENDS'><glossterm>LAYERDEPENDS</glossterm>
> <glossdef> <para>Lists the layers, separated by spaces, upon which
> this recipe depends. Optionally, you can specify a specific layer
> version for a dependency by adding it to the end of the layer name
> with a colon, (e.g. "anotherlayer:3" to be compared against
> - <link
> linkend='var-LAYERVERSION'><filename>LAYERVERSION</filename></link><filename>_anotherlayer</filename>
> + <link
> linkend='var-bb-LAYERVERSION'><filename>LAYERVERSION</filename></link><filename>_anotherlayer</filename>
> in this case). BitBake produces an error if any dependency is missing
> or the version numbers do not match exactly (if specified).</para>
> @@ -1710,7 +1710,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-LAYERDIR'><glossterm>LAYERDIR</glossterm>
> + <glossentry
> id='var-bb-LAYERDIR'><glossterm>LAYERDIR</glossterm> <glossdef>
> <para>When used inside the
> <filename>layer.conf</filename> configuration file, this variable
> provides the path of the current layer. @@ -1719,22 +1719,22 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-LAYERDIR_RE'><glossterm>LAYERDIR_RE</glossterm>
> + <glossentry
> id='var-bb-LAYERDIR_RE'><glossterm>LAYERDIR_RE</glossterm> <glossdef>
> <para>When used inside the
> <filename>layer.conf</filename> configuration file, this variable
> provides the path of the current layer, escaped for use in a regular
> expression
> - (<link
> linkend='var-BBFILE_PATTERN'><filename>BBFILE_PATTERN</filename></link>).
> + (<link
> linkend='var-bb-BBFILE_PATTERN'><filename>BBFILE_PATTERN</filename></link>).
> This variable is not available outside of
> <filename>layer.conf</filename> and references are expanded
> immediately when parsing of the file completes.</para> </glossdef>
> </glossentry>
> - <glossentry
> id='var-LAYERVERSION'><glossterm>LAYERVERSION</glossterm>
> + <glossentry
> id='var-bb-LAYERVERSION'><glossterm>LAYERVERSION</glossterm>
> <glossdef> <para>Optionally specifies the version of a layer as a
> single number. You can use this variable within
> - <link
> linkend='var-LAYERDEPENDS'><filename>LAYERDEPENDS</filename></link>
> + <link
> linkend='var-bb-LAYERDEPENDS'><filename>LAYERDEPENDS</filename></link>
> for another layer in order to depend on a specific version of the
> layer.</para> <para>
> @@ -1744,7 +1744,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-LICENSE'><glossterm>LICENSE</glossterm>
> + <glossentry
> id='var-bb-LICENSE'><glossterm>LICENSE</glossterm> <glossdef>
> <para>
> The list of source licenses for the recipe.
> @@ -1754,9 +1754,9 @@
>
> </glossdiv>
>
> - <glossdiv id='var-glossary-m'><title>M</title>
> + <glossdiv id='var-bb-glossary-m'><title>M</title>
>
> - <glossentry id='var-MIRRORS'><glossterm>MIRRORS</glossterm>
> + <glossentry
> id='var-bb-MIRRORS'><glossterm>MIRRORS</glossterm> <glossdef>
> <para>
> Specifies additional paths from which BitBake
> gets source code. @@ -1764,14 +1764,14 @@
> tries the local download directory.
> If that location fails, the build system tries
> locations defined by
> - <link
> linkend='var-PREMIRRORS'><filename>PREMIRRORS</filename></link>,
> + <link
> linkend='var-bb-PREMIRRORS'><filename>PREMIRRORS</filename></link>,
> the upstream source, and then locations specified by
> <filename>MIRRORS</filename> in that order. </para>
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-MULTI_PROVIDER_WHITELIST'><glossterm>MULTI_PROVIDER_WHITELIST</glossterm>
> + <glossentry
> id='var-bb-MULTI_PROVIDER_WHITELIST'><glossterm>MULTI_PROVIDER_WHITELIST</glossterm>
> <glossdef> <para>
> Allows you to suppress BitBake warnings caused
> when @@ -1804,9 +1804,9 @@
> </glossdiv>
> -->
>
> - <glossdiv id='var-glossary-o'><title>O</title>
> + <glossdiv id='var-bb-glossary-o'><title>O</title>
>
> - <glossentry
> id='var-OVERRIDES'><glossterm>OVERRIDES</glossterm>
> + <glossentry
> id='var-bb-OVERRIDES'><glossterm>OVERRIDES</glossterm> <glossdef>
> <para>
> BitBake uses <filename>OVERRIDES</filename> to
> control @@ -1829,9 +1829,9 @@
> </glossentry>
> </glossdiv>
>
> - <glossdiv id='var-glossary-p'><title>P</title>
> + <glossdiv id='var-bb-glossary-p'><title>P</title>
>
> - <glossentry id='var-P4DIR'><glossterm>P4DIR</glossterm>
> + <glossentry id='var-bb-P4DIR'><glossterm>P4DIR</glossterm>
> <glossdef>
> <para>
> The directory in which a local copy of a
> Perforce depot @@ -1840,14 +1840,14 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-PACKAGES'><glossterm>PACKAGES</glossterm>
> + <glossentry
> id='var-bb-PACKAGES'><glossterm>PACKAGES</glossterm> <glossdef>
> <para>The list of packages the recipe creates.
> </para>
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-PACKAGES_DYNAMIC'><glossterm>PACKAGES_DYNAMIC</glossterm>
> + <glossentry
> id='var-bb-PACKAGES_DYNAMIC'><glossterm>PACKAGES_DYNAMIC</glossterm>
> <glossdef> <para>
> A promise that your recipe satisfies runtime
> dependencies @@ -1856,7 +1856,7 @@
> does not actually satisfy the dependencies, it
> only states that they should be satisfied.
> For example, if a hard, runtime dependency
> - (<link
> linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link>)
> + (<link
> linkend='var-bb-RDEPENDS'><filename>RDEPENDS</filename></link>) of
> another package is satisfied during the build through the
> <filename>PACKAGES_DYNAMIC</filename> variable, but a package with
> the module name is never actually @@ -1865,7 +1865,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-PE'><glossterm>PE</glossterm>
> + <glossentry id='var-bb-PE'><glossterm>PE</glossterm>
> <glossdef>
> <para>
> The epoch of the recipe.
> @@ -1877,7 +1877,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-PERSISTENT_DIR'><glossterm>PERSISTENT_DIR</glossterm>
> + <glossentry
> id='var-bb-PERSISTENT_DIR'><glossterm>PERSISTENT_DIR</glossterm>
> <glossdef> <para>
> Specifies the directory BitBake uses to store
> data that @@ -1889,7 +1889,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-PF'><glossterm>PF</glossterm>
> + <glossentry id='var-bb-PF'><glossterm>PF</glossterm>
> <glossdef>
> <para>
> Specifies the recipe or package name and
> includes all version and revision @@ -1899,27 +1899,27 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-PN'><glossterm>PN</glossterm>
> + <glossentry id='var-bb-PN'><glossterm>PN</glossterm>
> <glossdef>
> <para>The recipe name.</para>
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-PR'><glossterm>PR</glossterm>
> + <glossentry id='var-bb-PR'><glossterm>PR</glossterm>
> <glossdef>
> <para>The revision of the recipe.
> </para>
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-PREFERRED_PROVIDER'><glossterm>PREFERRED_PROVIDER</glossterm>
> + <glossentry
> id='var-bb-PREFERRED_PROVIDER'><glossterm>PREFERRED_PROVIDER</glossterm>
> <glossdef> <para>
> Determines which recipe should be given
> preference when multiple recipes provide the same item.
> You should always suffix the variable with the
> name of the provided item, and you should set it to the
> - <link
> linkend='var-PN'><filename>PN</filename></link>
> + <link
> linkend='var-bb-PN'><filename>PN</filename></link> of the recipe to
> which you want to give precedence. Some examples:
> <literallayout class='monospaced'>
> @@ -1931,14 +1931,14 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-PREFERRED_PROVIDERS'><glossterm>PREFERRED_PROVIDERS</glossterm>
> + <glossentry
> id='var-bb-PREFERRED_PROVIDERS'><glossterm>PREFERRED_PROVIDERS</glossterm>
> <glossdef> <para>
> Determines which recipe should be given
> preference for cases where multiple recipes provide the same item.
> Functionally,
> <filename>PREFERRED_PROVIDERS</filename> is
> identical to
> - <link
> linkend='var-PREFERRED_PROVIDER'><filename>PREFERRED_PROVIDER</filename></link>.
> + <link
> linkend='var-bb-PREFERRED_PROVIDER'><filename>PREFERRED_PROVIDER</filename></link>.
> However, the <filename>PREFERRED_PROVIDERS</filename> variable lets
> you define preferences for multiple situations using the following
> form: @@ -1954,15 +1954,15 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-PREFERRED_VERSION'><glossterm>PREFERRED_VERSION</glossterm>
> + <glossentry
> id='var-bb-PREFERRED_VERSION'><glossterm>PREFERRED_VERSION</glossterm>
> <glossdef> <para>
> If there are multiple versions of recipes
> available, this variable determines which recipe should be given
> preference. You must always suffix the variable with the
> - <link
> linkend='var-PN'><filename>PN</filename></link>
> + <link
> linkend='var-bb-PN'><filename>PN</filename></link> you want to
> select, and you should set
> - <link
> linkend='var-PV'><filename>PV</filename></link>
> + <link
> linkend='var-bb-PV'><filename>PV</filename></link> accordingly for
> precedence. </para>
>
> @@ -1989,7 +1989,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-PREMIRRORS'><glossterm>PREMIRRORS</glossterm>
> + <glossentry
> id='var-bb-PREMIRRORS'><glossterm>PREMIRRORS</glossterm> <glossdef>
> <para>
> Specifies additional paths from which BitBake
> gets source code. @@ -1998,7 +1998,7 @@
> If that location fails, the build system tries
> locations defined by <filename>PREMIRRORS</filename>, the upstream
> source, and then locations specified by
> - <link
> linkend='var-MIRRORS'><filename>MIRRORS</filename></link>
> + <link
> linkend='var-bb-MIRRORS'><filename>MIRRORS</filename></link> in that
> order. </para>
>
> @@ -2022,20 +2022,20 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-PROVIDES'><glossterm>PROVIDES</glossterm>
> + <glossentry
> id='var-bb-PROVIDES'><glossterm>PROVIDES</glossterm> <glossdef>
> <para>
> A list of aliases by which a particular recipe
> can be known.
> By default, a recipe's own
> - <filename><link
> linkend='var-PN'>PN</link></filename>
> + <filename><link
> linkend='var-bb-PN'>PN</link></filename> is implicitly already in its
> <filename>PROVIDES</filename> list.
> If a recipe uses <filename>PROVIDES</filename>,
> the additional aliases are synonyms for the recipe and can
> be useful satisfying dependencies of other
> recipes during the build as specified by
> - <filename><link
> linkend='var-DEPENDS'>DEPENDS</link></filename>.
> + <filename><link
> linkend='var-bb-DEPENDS'>DEPENDS</link></filename>. </para>
>
> <para>
> @@ -2059,7 +2059,7 @@
> virtual target in <filename>PROVIDES</filename>.
> Recipes that depend on the functionality in
> question can include the virtual target in
> - <link
> linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
> + <link
> linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link> to leave
> the choice of provider open. </para>
>
> @@ -2072,11 +2072,11 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-PRSERV_HOST'><glossterm>PRSERV_HOST</glossterm>
> + <glossentry
> id='var-bb-PRSERV_HOST'><glossterm>PRSERV_HOST</glossterm> <glossdef>
> <para>
> The network based
> - <link
> linkend='var-PR'><filename>PR</filename></link>
> + <link
> linkend='var-bb-PR'><filename>PR</filename></link> service host and
> port. </para>
>
> @@ -2094,7 +2094,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-PV'><glossterm>PV</glossterm>
> + <glossentry id='var-bb-PV'><glossterm>PV</glossterm>
> <glossdef>
> <para>The version of the recipe.
> </para>
> @@ -2108,9 +2108,9 @@
> </glossdiv>
> -->
>
> - <glossdiv id='var-glossary-r'><title>R</title>
> + <glossdiv id='var-bb-glossary-r'><title>R</title>
>
> - <glossentry id='var-RDEPENDS'><glossterm>RDEPENDS</glossterm>
> + <glossentry
> id='var-bb-RDEPENDS'><glossterm>RDEPENDS</glossterm> <glossdef>
> <para>
> Lists a package's runtime dependencies (i.e.
> other packages) @@ -2165,13 +2165,13 @@
>
> <para>
> For information on build-time dependencies, see
> the
> - <link
> linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
> + <link
> linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link> variable.
> </para>
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-REPODIR'><glossterm>REPODIR</glossterm>
> + <glossentry
> id='var-bb-REPODIR'><glossterm>REPODIR</glossterm> <glossdef>
> <para>
> The directory in which a local copy of a
> @@ -2181,14 +2181,14 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-RPROVIDES'><glossterm>RPROVIDES</glossterm>
> + <glossentry
> id='var-bb-RPROVIDES'><glossterm>RPROVIDES</glossterm> <glossdef>
> <para>
> A list of package name aliases that a package
> also provides. These aliases are useful for satisfying runtime
> dependencies of other packages both during the build and on the target
> (as specified by
> - <filename><link
> linkend='var-RDEPENDS'>RDEPENDS</link></filename>).
> + <filename><link
> linkend='var-bb-RDEPENDS'>RDEPENDS</link></filename>). </para>
> <para>
> As with all package-controlling variables, you
> must always @@ -2201,7 +2201,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-RRECOMMENDS'><glossterm>RRECOMMENDS</glossterm>
> + <glossentry
> id='var-bb-RRECOMMENDS'><glossterm>RRECOMMENDS</glossterm> <glossdef>
> <para>
> A list of packages that extends the usability of
> a package @@ -2210,7 +2210,7 @@
> packages in order to successfully build, but
> needs them for the extended usability.
> To specify runtime dependencies for packages,
> see the
> - <filename><link
> linkend='var-RDEPENDS'>RDEPENDS</link></filename>
> + <filename><link
> linkend='var-bb-RDEPENDS'>RDEPENDS</link></filename> variable.
> </para>
>
> @@ -2243,15 +2243,15 @@
>
> </glossdiv>
>
> - <glossdiv id='var-glossary-s'><title>S</title>
> + <glossdiv id='var-bb-glossary-s'><title>S</title>
>
> - <glossentry id='var-SECTION'><glossterm>SECTION</glossterm>
> + <glossentry
> id='var-bb-SECTION'><glossterm>SECTION</glossterm> <glossdef>
> <para>The section in which packages should be
> categorized.</para> </glossdef>
> </glossentry>
>
> - <glossentry id='var-SRC_URI'><glossterm>SRC_URI</glossterm>
> + <glossentry
> id='var-bb-SRC_URI'><glossterm>SRC_URI</glossterm> <glossdef>
> <para>
> The list of source files - local or remote.
> @@ -2272,7 +2272,7 @@
> the metadata,
> from the local machine.
> The path is relative to the
> - <link
> linkend='var-FILESPATH'><filename>FILESPATH</filename></link>
> + <link
> linkend='var-bb-FILESPATH'><filename>FILESPATH</filename></link>
> variable.</para></listitem>
> <listitem><para><emphasis><filename>bzr://</filename> -</emphasis>
> Fetches files from a Bazaar revision control
> repository.</para></listitem> @@ -2322,7 +2322,7 @@ </glossdef>
> </glossentry>
>
> - <glossentry id='var-SRCDATE'><glossterm>SRCDATE</glossterm>
> + <glossentry
> id='var-bb-SRCDATE'><glossterm>SRCDATE</glossterm> <glossdef>
> <para>
> The date of the source code used to build the
> package. @@ -2331,7 +2331,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-SRCREV'><glossterm>SRCREV</glossterm>
> + <glossentry id='var-bb-SRCREV'><glossterm>SRCREV</glossterm>
> <glossdef>
> <para>
> The revision of the source code used to build
> the package. @@ -2344,13 +2344,13 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-SRCREV_FORMAT'><glossterm>SRCREV_FORMAT</glossterm>
> + <glossentry
> id='var-bb-SRCREV_FORMAT'><glossterm>SRCREV_FORMAT</glossterm>
> <glossdef> <para>
> Helps construct valid
> - <link
> linkend='var-SRCREV'><filename>SRCREV</filename></link>
> + <link
> linkend='var-bb-SRCREV'><filename>SRCREV</filename></link> values
> when multiple source controlled URLs are used in
> - <link
> linkend='var-SRC_URI'><filename>SRC_URI</filename></link>.
> + <link
> linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link>. </para>
>
> <para>
> @@ -2371,7 +2371,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-STAMP'><glossterm>STAMP</glossterm>
> + <glossentry id='var-bb-STAMP'><glossterm>STAMP</glossterm>
> <glossdef>
> <para>
> Specifies the base path used to create recipe
> stamp files. @@ -2381,12 +2381,12 @@
> </glossdef>
> </glossentry>
>
> - <glossentry
> id='var-STAMPCLEAN'><glossterm>STAMPCLEAN</glossterm>
> + <glossentry
> id='var-bb-STAMPCLEAN'><glossterm>STAMPCLEAN</glossterm> <glossdef>
> <para>
> Specifies the base path used to create recipe
> stamp files. Unlike the
> - <link
> linkend='var-STAMP'><filename>STAMP</filename></link>
> + <link
> linkend='var-bb-STAMP'><filename>STAMP</filename></link> variable,
> <filename>STAMPCLEAN</filename> can contain wildcards to match the
> range of files a clean operation should remove.
> @@ -2396,7 +2396,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-SUMMARY'><glossterm>SUMMARY</glossterm>
> + <glossentry
> id='var-bb-SUMMARY'><glossterm>SUMMARY</glossterm> <glossdef>
> <para>
> A short summary for the recipe, which is 72
> characters or less. @@ -2404,7 +2404,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-SVNDIR'><glossterm>SVNDIR</glossterm>
> + <glossentry id='var-bb-SVNDIR'><glossterm>SVNDIR</glossterm>
> <glossdef>
> <para>
> The directory in which files checked out of a
> Subversion @@ -2415,9 +2415,9 @@
>
> </glossdiv>
>
> - <glossdiv id='var-glossary-t'><title>T</title>
> + <glossdiv id='var-bb-glossary-t'><title>T</title>
>
> - <glossentry id='var-T'><glossterm>T</glossterm>
> + <glossentry id='var-bb-T'><glossterm>T</glossterm>
> <glossdef>
> <para>Points to a directory were BitBake places
> temporary files, which consist mostly of task
> logs and @@ -2426,7 +2426,7 @@
> </glossdef>
> </glossentry>
>
> - <glossentry id='var-TOPDIR'><glossterm>TOPDIR</glossterm>
> + <glossentry id='var-bb-TOPDIR'><glossterm>TOPDIR</glossterm>
> <glossdef>
> <para>
> Points to the build directory.
> diff --git a/bitbake/doc/poky.ent b/bitbake/doc/poky.ent
> index c032e14..85d9c83 100644
> --- a/bitbake/doc/poky.ent
> +++ b/bitbake/doc/poky.ent
> @@ -17,13 +17,6 @@
> <!ENTITY OE_DOCS_URL "http://docs.openembedded.org">
> <!ENTITY OH_HOME_URL "http://o-hand.com">
> <!ENTITY BITBAKE_HOME_URL
> "http://developer.berlios.de/projects/bitbake/"> -<!ENTITY
> ECLIPSE_MAIN_URL "http://www.eclipse.org/downloads"> -<!ENTITY
> ECLIPSE_DL_URL "http://download.eclipse.org"> -<!ENTITY
> ECLIPSE_DL_PLUGIN_URL
> "&YOCTO_DL_URL;/releases/eclipse-plugin/&DISTRO;"> -<!ENTITY
> ECLIPSE_UPDATES_URL "&ECLIPSE_DL_URL;/tm/updates/3.3"> -<!ENTITY
> ECLIPSE_INDIGO_URL "&ECLIPSE_DL_URL;/releases/indigo"> -<!ENTITY
> ECLIPSE_JUNO_URL "&ECLIPSE_DL_URL;/releases/juno"> -<!ENTITY
> ECLIPSE_INDIGO_CDT_URL "&ECLIPSE_DL_URL;tools/cdt/releases/indigo">
> <!ENTITY YOCTO_DOCS_URL "&YOCTO_HOME_URL;/docs"> <!ENTITY
> YOCTO_SOURCES_URL "&YOCTO_HOME_URL;/sources/"> <!ENTITY
> YOCTO_AB_PORT_URL "&YOCTO_AB_URL;:8010"> @@ -31,7 +24,6 @@ <!ENTITY
> YOCTO_POKY_URL "&YOCTO_DL_URL;/releases/poky/"> <!ENTITY
> YOCTO_RELEASE_DL_URL "&YOCTO_DL_URL;/releases/yocto/yocto-&DISTRO;">
> <!ENTITY YOCTO_TOOLCHAIN_DL_URL "&YOCTO_RELEASE_DL_URL;/toolchain/">
> -<!ENTITY YOCTO_ECLIPSE_DL_URL
> "&YOCTO_RELEASE_DL_URL;/eclipse-plugin/indigo;"> <!ENTITY
> YOCTO_ADTINSTALLER_DL_URL "&YOCTO_RELEASE_DL_URL;/adt_installer">
> <!ENTITY YOCTO_POKY_DL_URL
> "&YOCTO_RELEASE_DL_URL;/&YOCTO_POKY;.tar.bz2"> <!ENTITY
> YOCTO_MACHINES_DL_URL "&YOCTO_RELEASE_DL_URL;/machines"> diff --git
> a/bitbake/lib/bb/COW.py b/bitbake/lib/bb/COW.py index
> 7817473..d26e981 100644 --- a/bitbake/lib/bb/COW.py +++
> b/bitbake/lib/bb/COW.py @@ -1,23 +1,8 @@ -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # This is a copy on write dictionary and set which abuses classes to
> try and be nice and fast. #
> # Copyright (C) 2006 Tim Ansell
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. -#
> #Please Note:
> # Be careful when using mutable types (ie Dict and Lists) -
> operations involving these are SLOW. # Assign a file to __warn__ to
> get warnings about slow operations. diff --git
> a/bitbake/lib/bb/__init__.py b/bitbake/lib/bb/__init__.py index
> 4bc47c8..c144311 100644 --- a/bitbake/lib/bb/__init__.py
> +++ b/bitbake/lib/bb/__init__.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # BitBake Build System Python Library
> #
> @@ -8,20 +6,10 @@
> #
> # Based on Gentoo's portage.py.
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> -__version__ = "1.40.0"
> +__version__ = "1.44.0"
>
> import sys
> if sys.version_info < (3, 4, 0):
> diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py
> index 3e2a94e..30a2ba2 100644
> --- a/bitbake/lib/bb/build.py
> +++ b/bitbake/lib/bb/build.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # BitBake 'Build' implementation
> #
> @@ -10,18 +8,7 @@
> #
> # Based on Gentoo's portage.py.
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
> # Based on functions from the base bb module, Copyright 2003 Holger
> Schurig
> @@ -67,23 +54,6 @@ else:
> builtins['bb'] = bb
> builtins['os'] = os
>
> -class FuncFailed(Exception):
> - def __init__(self, name = None, logfile = None):
> - self.logfile = logfile
> - self.name = name
> - if name:
> - self.msg = 'Function failed: %s' % name
> - else:
> - self.msg = "Function failed"
> -
> - def __str__(self):
> - if self.logfile and os.path.exists(self.logfile):
> - msg = ("%s (log file is located at %s)" %
> - (self.msg, self.logfile))
> - else:
> - msg = self.msg
> - return msg
> -
> class TaskBase(event.Event):
> """Base class for task events"""
>
> @@ -176,15 +146,33 @@ class LogTee(object):
>
> def __repr__(self):
> return '<LogTee {0}>'.format(self.name)
> +
> def flush(self):
> self.outfile.flush()
>
> -#
> -# pythonexception allows the python exceptions generated to be raised
> -# as the real exceptions (not FuncFailed) and without a backtrace at
> the -# origin of the failure.
> -#
> -def exec_func(func, d, dirs = None, pythonexception=False):
> +
> +class StdoutNoopContextManager:
> + """
> + This class acts like sys.stdout, but adds noop __enter__ and
> __exit__ methods.
> + """
> + def __enter__(self):
> + return sys.stdout
> +
> + def __exit__(self, *exc_info):
> + pass
> +
> + def write(self, string):
> + return sys.stdout.write(string)
> +
> + def flush(self):
> + sys.stdout.flush()
> +
> + @property
> + def name(self):
> + return sys.stdout.name
> +
> +
> +def exec_func(func, d, dirs = None):
> """Execute a BB 'function'"""
>
> try:
> @@ -256,7 +244,7 @@ def exec_func(func, d, dirs = None,
> pythonexception=False):
> with bb.utils.fileslocked(lockfiles):
> if ispython:
> - exec_func_python(func, d, runfile, cwd=adir,
> pythonexception=pythonexception)
> + exec_func_python(func, d, runfile, cwd=adir)
> else:
> exec_func_shell(func, d, runfile, cwd=adir)
>
> @@ -276,7 +264,7 @@ _functionfmt = """
> {function}(d)
> """
> logformatter = bb.msg.BBLogFormatter("%(levelname)s: %(message)s")
> -def exec_func_python(func, d, runfile, cwd=None,
> pythonexception=False): +def exec_func_python(func, d, runfile,
> cwd=None): """Execute a python BB 'function'"""
>
> code = _functionfmt.format(function=func)
> @@ -301,13 +289,7 @@ def exec_func_python(func, d, runfile, cwd=None,
> pythonexception=False): bb.methodpool.insert_method(func, text, fn,
> lineno - 1)
> comp = utils.better_compile(code, func, "exec_python_func()
> autogenerated")
> - utils.better_exec(comp, {"d": d}, code, "exec_python_func()
> autogenerated", pythonexception=pythonexception)
> - except (bb.parse.SkipRecipe, bb.build.FuncFailed):
> - raise
> - except:
> - if pythonexception:
> - raise
> - raise FuncFailed(func, None)
> + utils.better_exec(comp, {"d": d}, code, "exec_python_func()
> autogenerated") finally:
> bb.debug(2, "Python function %s finished" % func)
>
> @@ -335,6 +317,42 @@ trap 'bb_exit_handler' 0
> set -e
> '''
>
> +def create_progress_handler(func, progress, logfile, d):
> + if progress == 'percent':
> + # Use default regex
> + return bb.progress.BasicProgressHandler(d, outfile=logfile)
> + elif progress.startswith('percent:'):
> + # Use specified regex
> + return bb.progress.BasicProgressHandler(d,
> regex=progress.split(':', 1)[1], outfile=logfile)
> + elif progress.startswith('outof:'):
> + # Use specified regex
> + return bb.progress.OutOfProgressHandler(d,
> regex=progress.split(':', 1)[1], outfile=logfile)
> + elif progress.startswith("custom:"):
> + # Use a custom progress handler that was injected via
> OE_EXTRA_IMPORTS or __builtins__
> + import functools
> + from types import ModuleType
> +
> + parts = progress.split(":", 2)
> + _, cls, otherargs = parts[0], parts[1], (parts[2] or None)
> if parts[2:] else None
> + if cls:
> + def resolve(x, y):
> + if not x:
> + return None
> + if isinstance(x, ModuleType):
> + return getattr(x, y, None)
> + return x.get(y)
> + cls_obj = functools.reduce(resolve, cls.split("."),
> bb.utils._context)
> + if not cls_obj:
> + # Fall-back on __builtins__
> + cls_obj = functools.reduce(lambda x, y: x.get(y),
> cls.split("."), __builtins__)
> + if cls_obj:
> + return cls_obj(d, outfile=logfile,
> otherargs=otherargs)
> + bb.warn('%s: unknown custom progress handler in task
> progress varflag value "%s", ignoring' % (func, cls))
> + else:
> + bb.warn('%s: invalid task progress varflag value "%s",
> ignoring' % (func, progress)) +
> + return logfile
> +
> def exec_func_shell(func, d, runfile, cwd=None):
> """Execute a shell function from the metadata
>
> @@ -372,23 +390,13 @@ exit $ret
> cmd = [fakerootcmd, runfile]
>
> if bb.msg.loggerDefaultVerbose:
> - logfile = LogTee(logger, sys.stdout)
> + logfile = LogTee(logger, StdoutNoopContextManager())
> else:
> - logfile = sys.stdout
> + logfile = StdoutNoopContextManager()
>
> progress = d.getVarFlag(func, 'progress')
> if progress:
> - if progress == 'percent':
> - # Use default regex
> - logfile = bb.progress.BasicProgressHandler(d,
> outfile=logfile)
> - elif progress.startswith('percent:'):
> - # Use specified regex
> - logfile = bb.progress.BasicProgressHandler(d,
> regex=progress.split(':', 1)[1], outfile=logfile)
> - elif progress.startswith('outof:'):
> - # Use specified regex
> - logfile = bb.progress.OutOfProgressHandler(d,
> regex=progress.split(':', 1)[1], outfile=logfile)
> - else:
> - bb.warn('%s: invalid task progress varflag value "%s",
> ignoring' % (func, progress))
> + logfile = create_progress_handler(func, progress, logfile, d)
>
> fifobuffer = bytearray()
> def readfifo(data):
> @@ -407,6 +415,8 @@ exit $ret
> bb.plain(value)
> elif cmd == 'bbnote':
> bb.note(value)
> + elif cmd == 'bbverbnote':
> + bb.verbnote(value)
> elif cmd == 'bbwarn':
> bb.warn(value)
> elif cmd == 'bberror':
> @@ -436,13 +446,8 @@ exit $ret
> with open(fifopath, 'r+b', buffering=0) as fifo:
> try:
> bb.debug(2, "Executing shell function %s" % func)
> -
> - try:
> - with open(os.devnull, 'r+') as stdin:
> - bb.process.run(cmd, shell=False, stdin=stdin,
> log=logfile, extrafiles=[(fifo,readfifo)])
> - except bb.process.CmdError:
> - logfn = d.getVar('BB_LOGFILE')
> - raise FuncFailed(func, logfn)
> + with open(os.devnull, 'r+') as stdin, logfile:
> + bb.process.run(cmd, shell=False, stdin=stdin,
> log=logfile, extrafiles=[(fifo,readfifo)]) finally:
> os.unlink(fifopath)
>
> @@ -570,9 +575,6 @@ def _exec_task(fn, task, d, quieterr):
> event.fire(TaskStarted(task, logfn, flags, localdata),
> localdata) except (bb.BBHandledException, SystemExit):
> return 1
> - except FuncFailed as exc:
> - logger.error(str(exc))
> - return 1
>
> try:
> for func in (prefuncs or '').split():
> @@ -580,7 +582,10 @@ def _exec_task(fn, task, d, quieterr):
> exec_func(task, localdata)
> for func in (postfuncs or '').split():
> exec_func(func, localdata)
> - except FuncFailed as exc:
> + except bb.BBHandledException:
> + event.fire(TaskFailed(task, logfn, localdata, True),
> localdata)
> + return 1
> + except Exception as exc:
> if quieterr:
> event.fire(TaskFailedSilent(task, logfn, localdata),
> localdata) else:
> @@ -588,9 +593,6 @@ def _exec_task(fn, task, d, quieterr):
> logger.error(str(exc))
> event.fire(TaskFailed(task, logfn, localdata,
> errprinted), localdata) return 1
> - except bb.BBHandledException:
> - event.fire(TaskFailed(task, logfn, localdata, True),
> localdata)
> - return 1
> finally:
> sys.stdout.flush()
> sys.stderr.flush()
> @@ -814,6 +816,9 @@ def add_tasks(tasklist, d):
> task_deps['parents'][task] = []
> if 'deps' in flags:
> for dep in flags['deps']:
> + # Check and warn for "addtask task after foo" while
> foo does not exist
> + #if not dep in tasklist:
> + # bb.warn('%s: dependent task %s for %s does not
> exist' % (d.getVar('PN'), dep, task)) dep = d.expand(dep)
> task_deps['parents'][task].append(dep)
>
> diff --git a/bitbake/lib/bb/cache.py b/bitbake/lib/bb/cache.py
> index 258d679..b6f7da5 100644
> --- a/bitbake/lib/bb/cache.py
> +++ b/bitbake/lib/bb/cache.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # BitBake Cache implementation
> #
> @@ -15,18 +13,8 @@
> # Copyright (C) 2005 Holger Hans Peter Freyther
> # Copyright (C) 2005 ROAD GmbH
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import os
> import sys
> @@ -95,21 +83,21 @@ class CoreRecipeInfo(RecipeInfoCommon):
> self.appends = self.listvar('__BBAPPEND', metadata)
> self.nocache = self.getvar('BB_DONT_CACHE', metadata)
>
> + self.provides = self.depvar('PROVIDES', metadata)
> + self.rprovides = self.depvar('RPROVIDES', metadata)
> + self.pn = self.getvar('PN', metadata) or
> bb.parse.vars_from_file(filename,metadata)[0]
> + self.packages = self.listvar('PACKAGES', metadata)
> + if not self.packages:
> + self.packages.append(self.pn)
> + self.packages_dynamic = self.listvar('PACKAGES_DYNAMIC',
> metadata) +
> self.skipreason = self.getvar('__SKIPPED', metadata)
> if self.skipreason:
> - self.pn = self.getvar('PN', metadata) or
> bb.parse.BBHandler.vars_from_file(filename,metadata)[0] self.skipped
> = True
> - self.provides = self.depvar('PROVIDES', metadata)
> - self.rprovides = self.depvar('RPROVIDES', metadata)
> return
>
> self.tasks = metadata.getVar('__BBTASKS', False)
>
> - self.pn = self.getvar('PN', metadata)
> - self.packages = self.listvar('PACKAGES', metadata)
> - if not self.packages:
> - self.packages.append(self.pn)
> -
> self.basetaskhashes = self.taskvar('BB_BASEHASH',
> self.tasks, metadata) self.hashfilename =
> self.getvar('BB_HASHFILENAME', metadata)
> @@ -125,11 +113,8 @@ class CoreRecipeInfo(RecipeInfoCommon):
> self.stampclean = self.getvar('STAMPCLEAN', metadata)
> self.stamp_extrainfo = self.flaglist('stamp-extra-info',
> self.tasks, metadata) self.file_checksums =
> self.flaglist('file-checksums', self.tasks, metadata, True)
> - self.packages_dynamic = self.listvar('PACKAGES_DYNAMIC',
> metadata) self.depends = self.depvar('DEPENDS', metadata)
> - self.provides = self.depvar('PROVIDES', metadata)
> self.rdepends = self.depvar('RDEPENDS', metadata)
> - self.rprovides = self.depvar('RPROVIDES', metadata)
> self.rrecommends = self.depvar('RRECOMMENDS', metadata)
> self.rprovides_pkg = self.pkgvar('RPROVIDES',
> self.packages, metadata) self.rdepends_pkg =
> self.pkgvar('RDEPENDS', self.packages, metadata) @@ -235,7 +220,7 @@
> class CoreRecipeInfo(RecipeInfoCommon):
> cachedata.hashfn[fn] = self.hashfilename
> for task, taskhash in self.basetaskhashes.items():
> - identifier = '%s.%s' % (fn, task)
> + identifier = '%s:%s' % (fn, task)
> cachedata.basetaskhash[identifier] = taskhash
>
> cachedata.inherits[fn] = self.inherits
> @@ -249,7 +234,7 @@ def virtualfn2realfn(virtualfn):
> Convert a virtual file name to a real one + the associated
> subclass keyword """
> mc = ""
> - if virtualfn.startswith('multiconfig:'):
> + if virtualfn.startswith('mc:'):
> elems = virtualfn.split(':')
> mc = elems[1]
> virtualfn = ":".join(elems[2:])
> @@ -270,7 +255,7 @@ def realfn2virtual(realfn, cls, mc):
> if cls:
> realfn = "virtual:" + cls + ":" + realfn
> if mc:
> - realfn = "multiconfig:" + mc + ":" + realfn
> + realfn = "mc:" + mc + ":" + realfn
> return realfn
>
> def variant2virtual(realfn, variant):
> @@ -279,11 +264,11 @@ def variant2virtual(realfn, variant):
> """
> if variant == "":
> return realfn
> - if variant.startswith("multiconfig:"):
> + if variant.startswith("mc:"):
> elems = variant.split(":")
> if elems[2]:
> - return "multiconfig:" + elems[1] + ":virtual:" +
> ":".join(elems[2:]) + ":" + realfn
> - return "multiconfig:" + elems[1] + ":" + realfn
> + return "mc:" + elems[1] + ":virtual:" +
> ":".join(elems[2:]) + ":" + realfn
> + return "mc:" + elems[1] + ":" + realfn
> return "virtual:" + variant + ":" + realfn
>
> def parse_recipe(bb_data, bbfile, appends, mc=''):
> @@ -361,7 +346,7 @@ class NoCache(object):
> bb_data = self.databuilder.mcdata[mc].createCopy()
> newstores = parse_recipe(bb_data, bbfile, appends, mc)
> for ns in newstores:
> - datastores["multiconfig:%s:%s" % (mc, ns)] =
> newstores[ns]
> + datastores["mc:%s:%s" % (mc, ns)] = newstores[ns]
>
> return datastores
>
> @@ -411,6 +396,15 @@ class Cache(NoCache):
> else:
> logger.debug(1, "Cache file %s not found, building..." %
> self.cachefile)
> + # We don't use the symlink, its just for debugging
> convinience
> + symlink = os.path.join(self.cachedir, "bb_cache.dat")
> + if os.path.exists(symlink):
> + bb.utils.remove(symlink)
> + try:
> + os.symlink(os.path.basename(self.cachefile), symlink)
> + except OSError:
> + pass
> +
> def load_cachefile(self):
> cachesize = 0
> previous_progress = 0
> @@ -889,3 +883,56 @@ class MultiProcessCache(object):
> p.dump([data, self.__class__.CACHE_VERSION])
>
> bb.utils.unlockfile(glf)
> +
> +
> +class SimpleCache(object):
> + """
> + BitBake multi-process cache implementation
> +
> + Used by the codeparser & file checksum caches
> + """
> +
> + def __init__(self, version):
> + self.cachefile = None
> + self.cachedata = None
> + self.cacheversion = version
> +
> + def init_cache(self, d, cache_file_name=None, defaultdata=None):
> + cachedir = (d.getVar("PERSISTENT_DIR") or
> + d.getVar("CACHE"))
> + if not cachedir:
> + return defaultdata
> +
> + bb.utils.mkdirhier(cachedir)
> + self.cachefile = os.path.join(cachedir,
> + cache_file_name or
> self.__class__.cache_file_name)
> + logger.debug(1, "Using cache in '%s'", self.cachefile)
> +
> + glf = bb.utils.lockfile(self.cachefile + ".lock")
> +
> + try:
> + with open(self.cachefile, "rb") as f:
> + p = pickle.Unpickler(f)
> + data, version = p.load()
> + except:
> + bb.utils.unlockfile(glf)
> + return defaultdata
> +
> + bb.utils.unlockfile(glf)
> +
> + if version != self.cacheversion:
> + return defaultdata
> +
> + return data
> +
> + def save(self, data):
> + if not self.cachefile:
> + return
> +
> + glf = bb.utils.lockfile(self.cachefile + ".lock")
> +
> + with open(self.cachefile, "wb") as f:
> + p = pickle.Pickler(f, -1)
> + p.dump([data, self.cacheversion])
> +
> + bb.utils.unlockfile(glf)
> diff --git a/bitbake/lib/bb/cache_extra.py
> b/bitbake/lib/bb/cache_extra.py index 83f4959..bf4226d 100644
> --- a/bitbake/lib/bb/cache_extra.py
> +++ b/bitbake/lib/bb/cache_extra.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # Extra RecipeInfo will be all defined in this file. Currently,
> # Only Hob (Image Creator) Requests some extra fields. So
> @@ -12,18 +10,8 @@
>
> # Copyright (C) 2011, Intel Corporation. All rights reserved.
>
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from bb.cache import RecipeInfoCommon
>
> diff --git a/bitbake/lib/bb/checksum.py b/bitbake/lib/bb/checksum.py
> index 4e1598f..5bc8a8f 100644
> --- a/bitbake/lib/bb/checksum.py
> +++ b/bitbake/lib/bb/checksum.py
> @@ -2,18 +2,8 @@
> #
> # Copyright (C) 2012 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import glob
> import operator
> diff --git a/bitbake/lib/bb/codeparser.py
> b/bitbake/lib/bb/codeparser.py index ddd1b97..fd2c473 100644
> --- a/bitbake/lib/bb/codeparser.py
> +++ b/bitbake/lib/bb/codeparser.py
> @@ -1,3 +1,7 @@
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> """
> BitBake code parser
>
> @@ -33,7 +37,7 @@ from bb.cache import MultiProcessCache
> logger = logging.getLogger('BitBake.CodeParser')
>
> def bbhash(s):
> - return hashlib.md5(s.encode("utf-8")).hexdigest()
> + return hashlib.sha256(s.encode("utf-8")).hexdigest()
>
> def check_indent(codestr):
> """If the code is indented, add a top level piece of code to
> 'remove' the indentation""" @@ -140,7 +144,7 @@ class
> CodeParserCache(MultiProcessCache): # so that an existing cache gets
> invalidated. Additionally you'll need # to increment
> __cache_version__ in cache.py in order to ensure that old # recipe
> caches don't trigger "Taskhash mismatch" errors.
> - CACHE_VERSION = 10
> + CACHE_VERSION = 11
>
> def __init__(self):
> MultiProcessCache.__init__(self)
> @@ -368,8 +372,9 @@ class ShellParser():
> def _parse_shell(self, value):
> try:
> tokens, _ = pyshyacc.parse(value, eof=True, debug=False)
> - except pyshlex.NeedMore:
> - raise sherrors.ShellSyntaxError("Unexpected EOF")
> + except Exception:
> + bb.error('Error during parse shell code, the last 5
> lines are:\n%s' % '\n'.join(value.split('\n')[-5:]))
> + raise
>
> self.process_tokens(tokens)
>
> diff --git a/bitbake/lib/bb/command.py b/bitbake/lib/bb/command.py
> index 6c966e3..378f389 100644
> --- a/bitbake/lib/bb/command.py
> +++ b/bitbake/lib/bb/command.py
> @@ -6,18 +6,8 @@ Provide an interface to interact with the bitbake
> server through 'commands'
> # Copyright (C) 2006-2007 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> """
> The bitbake server takes 'commands' from its UI/commandline.
> diff --git a/bitbake/lib/bb/compat.py b/bitbake/lib/bb/compat.py
> index de1923d..4935668 100644
> --- a/bitbake/lib/bb/compat.py
> +++ b/bitbake/lib/bb/compat.py
> @@ -1,3 +1,7 @@
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> """Code pulled from future python versions, here for compatibility"""
>
> from collections import MutableMapping, KeysView, ValuesView,
> ItemsView, OrderedDict diff --git a/bitbake/lib/bb/cooker.py
> b/bitbake/lib/bb/cooker.py index 16681ba..20ef04d 100644
> --- a/bitbake/lib/bb/cooker.py
> +++ b/bitbake/lib/bb/cooker.py
> @@ -1,6 +1,3 @@
> -#!/usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # Copyright (C) 2003, 2004 Chris Larson
> # Copyright (C) 2003, 2004 Phil Blundell
> @@ -9,19 +6,8 @@
> # Copyright (C) 2005 ROAD GmbH
> # Copyright (C) 2006 - 2007 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. -
>
> import sys, os, glob, os.path, re, time
> import atexit
> @@ -45,6 +31,7 @@ import pyinotify
> import json
> import pickle
> import codecs
> +import hashserv
>
> logger = logging.getLogger("BitBake")
> collectlog = logging.getLogger("BitBake.Collection")
> @@ -175,27 +162,45 @@ class BBCooker:
>
> self.configuration = configuration
>
> + bb.debug(1, "BBCooker starting %s" % time.time())
> + sys.stdout.flush()
> +
> self.configwatcher = pyinotify.WatchManager()
> + bb.debug(1, "BBCooker pyinotify1 %s" % time.time())
> + sys.stdout.flush()
> +
> self.configwatcher.bbseen = []
> self.configwatcher.bbwatchedfiles = []
> self.confignotifier = pyinotify.Notifier(self.configwatcher,
> self.config_notifications)
> + bb.debug(1, "BBCooker pyinotify2 %s" % time.time())
> + sys.stdout.flush()
> self.watchmask = pyinotify.IN_CLOSE_WRITE |
> pyinotify.IN_CREATE | pyinotify.IN_DELETE | \
> pyinotify.IN_DELETE_SELF | pyinotify.IN_MODIFY |
> pyinotify.IN_MOVE_SELF | \ pyinotify.IN_MOVED_FROM |
> pyinotify.IN_MOVED_TO self.watcher = pyinotify.WatchManager()
> + bb.debug(1, "BBCooker pyinotify3 %s" % time.time())
> + sys.stdout.flush()
> self.watcher.bbseen = []
> self.watcher.bbwatchedfiles = []
> self.notifier = pyinotify.Notifier(self.watcher,
> self.notifications)
> + bb.debug(1, "BBCooker pyinotify complete %s" % time.time())
> + sys.stdout.flush()
> +
> # If being called by something like tinfoil, we need to
> clean cached data # which may now be invalid
> bb.parse.clear_cache()
> bb.parse.BBHandler.cached_statements = {}
>
> self.ui_cmdline = None
> + self.hashserv = None
> + self.hashservaddr = None
>
> self.initConfigurationData()
>
> + bb.debug(1, "BBCooker parsed base configuration %s" %
> time.time())
> + sys.stdout.flush()
> +
> # we log all events to a file if so directed
> if self.configuration.writeeventlog:
> # register the log file writer as UI Handler
> @@ -233,6 +238,9 @@ class BBCooker:
> # Let SIGHUP exit as SIGTERM
> signal.signal(signal.SIGHUP, self.sigterm_exception)
>
> + bb.debug(1, "BBCooker startup complete %s" % time.time())
> + sys.stdout.flush()
> +
> def process_inotify_updates(self):
> for n in [self.confignotifier, self.notifier]:
> if n.check_events(timeout=0):
> @@ -367,13 +375,12 @@ class BBCooker:
> # Copy of the data store which has been expanded.
> # Used for firing events and accessing variables where
> expansion needs to be accounted for #
> - bb.parse.init_parser(self.data)
> -
> if CookerFeatures.BASEDATASTORE_TRACKING in self.featureset:
> self.disableDataTracking()
>
> - self.data.renameVar("__depends", "__base_depends")
> - self.add_filewatch(self.data.getVar("__base_depends",
> False), self.configwatcher)
> + for mc in self.databuilder.mcdata.values():
> + mc.renameVar("__depends", "__base_depends")
> + self.add_filewatch(mc.getVar("__base_depends", False),
> self.configwatcher)
> self.baseconfig_valid = True
> self.parsecache_valid = False
> @@ -385,6 +392,22 @@ class BBCooker:
> except prserv.serv.PRServiceConfigError as e:
> bb.fatal("Unable to start PR Server, exitting")
>
> + if self.data.getVar("BB_HASHSERVE") == "auto":
> + # Create a new hash server bound to a unix domain socket
> + if not self.hashserv:
> + dbfile = (self.data.getVar("PERSISTENT_DIR") or
> self.data.getVar("CACHE")) + "/hashserv.db"
> + self.hashservaddr = "unix://%s/hashserve.sock" %
> self.data.getVar("TOPDIR")
> + self.hashserv =
> hashserv.create_server(self.hashservaddr, dbfile, sync=False)
> + self.hashserv.process =
> multiprocessing.Process(target=self.hashserv.serve_forever)
> + self.hashserv.process.start()
> + self.data.setVar("BB_HASHSERVE", self.hashservaddr)
> + self.databuilder.origdata.setVar("BB_HASHSERVE",
> self.hashservaddr)
> + self.databuilder.data.setVar("BB_HASHSERVE",
> self.hashservaddr)
> + for mc in self.databuilder.mcdata:
> + self.databuilder.mcdata[mc].setVar("BB_HASHSERVE",
> self.hashservaddr) +
> + bb.parse.init_parser(self.data)
> +
> def enableDataTracking(self):
> self.configuration.tracking = True
> if hasattr(self, "data"):
> @@ -488,6 +511,7 @@ class BBCooker:
> """
> fn = None
> envdata = None
> + mc = ''
> if not pkgs_to_build:
> pkgs_to_build = []
>
> @@ -496,6 +520,12 @@ class BBCooker:
> self.enableDataTracking()
> self.reset()
>
> + def mc_base(p):
> + if p.startswith('mc:'):
> + s = p.split(':')
> + if len(s) == 2:
> + return s[1]
> + return None
>
> if buildfile:
> # Parse the configuration here. We need to do it
> explicitly here since @@ -506,18 +536,16 @@ class BBCooker:
> fn = self.matchFile(fn)
> fn = bb.cache.realfn2virtual(fn, cls, mc)
> elif len(pkgs_to_build) == 1:
> - ignore = self.data.getVar("ASSUME_PROVIDED") or ""
> - if pkgs_to_build[0] in set(ignore.split()):
> - bb.fatal("%s is in ASSUME_PROVIDED" %
> pkgs_to_build[0])
> + mc = mc_base(pkgs_to_build[0])
> + if not mc:
> + ignore = self.data.getVar("ASSUME_PROVIDED") or ""
> + if pkgs_to_build[0] in set(ignore.split()):
> + bb.fatal("%s is in ASSUME_PROVIDED" %
> pkgs_to_build[0])
> - taskdata, runlist = self.buildTaskData(pkgs_to_build,
> None, self.configuration.abort, allowincomplete=True)
> + taskdata, runlist =
> self.buildTaskData(pkgs_to_build, None, self.configuration.abort,
> allowincomplete=True)
> - mc = runlist[0][0]
> - fn = runlist[0][3]
> - else:
> - envdata = self.data
> - data.expandKeys(envdata)
> - parse.ast.runAnonFuncs(envdata)
> + mc = runlist[0][0]
> + fn = runlist[0][3]
>
> if fn:
> try:
> @@ -526,6 +554,12 @@ class BBCooker:
> except Exception as e:
> parselog.exception("Unable to read %s", fn)
> raise
> + else:
> + if not mc in self.databuilder.mcdata:
> + bb.fatal('Not multiconfig named "%s" found' % mc)
> + envdata = self.databuilder.mcdata[mc]
> + data.expandKeys(envdata)
> + parse.ast.runAnonFuncs(envdata)
>
> # Display history
> with closing(StringIO()) as env:
> @@ -565,10 +599,10 @@ class BBCooker:
> wildcard = False
>
> # Wild card expansion:
> - # Replace string such as "multiconfig:*:bash"
> - # into "multiconfig:A:bash multiconfig:B:bash bash"
> + # Replace string such as "mc:*:bash"
> + # into "mc:A:bash mc:B:bash bash"
> for k in targetlist:
> - if k.startswith("multiconfig:"):
> + if k.startswith("mc:"):
> if wildcard:
> bb.fatal('multiconfig conflict')
> if k.split(":")[1] == "*":
> @@ -601,7 +635,7 @@ class BBCooker:
> runlist = []
> for k in fulltargetlist:
> mc = ""
> - if k.startswith("multiconfig:"):
> + if k.startswith("mc:"):
> mc = k.split(":")[1]
> k = ":".join(k.split(":")[2:])
> ktask = task
> @@ -620,13 +654,22 @@ class BBCooker:
> runlist.append([mc, k, ktask, fn])
> bb.event.fire(bb.event.TreeDataPreparationProgress(current,
> len(fulltargetlist)), self.data)
> - mcdeps = taskdata[mc].get_mcdepends()
> + havemc = False
> + for mc in self.multiconfigs:
> + if taskdata[mc].get_mcdepends():
> + havemc = True
> +
> # No need to do check providers if there are no mcdeps or
> not an mc build
> - if mcdeps and mc:
> - # Make sure we can provide the multiconfig dependency
> + if havemc or len(self.multiconfigs) > 1:
> seen = set()
> new = True
> + # Make sure we can provide the multiconfig dependency
> while new:
> + mcdeps = set()
> + # Add unresolved first, so we can get multiconfig
> indirect dependencies on time
> + for mc in self.multiconfigs:
> + taskdata[mc].add_unresolved(localdata[mc],
> self.recipecaches[mc])
> + mcdeps |= set(taskdata[mc].get_mcdepends())
> new = False
> for mc in self.multiconfigs:
> for k in mcdeps:
> @@ -641,6 +684,7 @@ class BBCooker:
> taskdata[depmc].add_provider(localdata[depmc],
> self.recipecaches[depmc], l[3]) seen.add(k)
> new = True
> +
> for mc in self.multiconfigs:
> taskdata[mc].add_unresolved(localdata[mc],
> self.recipecaches[mc])
> @@ -676,7 +720,7 @@ class BBCooker:
> @staticmethod
> def add_mc_prefix(mc, pn):
> if mc:
> - return "multiconfig:%s:%s" % (mc, pn)
> + return "mc:%s:%s" % (mc, pn)
> return pn
>
> def buildDependTree(self, rq, taskdata):
> @@ -875,6 +919,10 @@ class BBCooker:
> os.unlink('package-depends.dot')
> except FileNotFoundError:
> pass
> + try:
> + os.unlink('recipe-depends.dot')
> + except FileNotFoundError:
> + pass
>
> with open('task-depends.dot', 'w') as f:
> f.write("digraph depends {\n")
> @@ -888,27 +936,6 @@ class BBCooker:
> f.write("}\n")
> logger.info("Task dependencies saved to 'task-depends.dot'")
>
> - with open('recipe-depends.dot', 'w') as f:
> - f.write("digraph depends {\n")
> - pndeps = {}
> - for task in sorted(depgraph["tdepends"]):
> - (pn, taskname) = task.rsplit(".", 1)
> - if pn not in pndeps:
> - pndeps[pn] = set()
> - for dep in sorted(depgraph["tdepends"][task]):
> - (deppn, deptaskname) = dep.rsplit(".", 1)
> - pndeps[pn].add(deppn)
> - for pn in sorted(pndeps):
> - fn = depgraph["pn"][pn]["filename"]
> - version = depgraph["pn"][pn]["version"]
> - f.write('"%s" [label="%s\\n%s\\n%s"]\n' % (pn, pn,
> version, fn))
> - for dep in sorted(pndeps[pn]):
> - if dep == pn:
> - continue
> - f.write('"%s" -> "%s"\n' % (pn, dep))
> - f.write("}\n")
> - logger.info("Flattened recipe dependencies saved to
> 'recipe-depends.dot'") -
> def show_appends_with_no_recipes(self):
> # Determine which bbappends haven't been applied
>
> @@ -1191,8 +1218,8 @@ class BBCooker:
> continue
> elif regex == "":
> parselog.debug(1, "BBFILE_PATTERN_%s is empty" %
> c)
> + cre = re.compile('^NULL$')
> errors = False
> - continue
> else:
> try:
> cre = re.compile(regex)
> @@ -1453,7 +1480,7 @@ class BBCooker:
> ntargets = []
> for target in runlist:
> if target[0]:
> - ntargets.append("multiconfig:%s:%s:%s" % (target[0],
> target[1], target[2]))
> + ntargets.append("mc:%s:%s:%s" % (target[0],
> target[1], target[2])) ntargets.append("%s:%s" % (target[1],
> target[2]))
> for mc in self.multiconfigs:
> @@ -1576,6 +1603,9 @@ class BBCooker:
> for pkg in pkgs_to_build:
> if pkg in ignore:
> parselog.warning("Explicit target \"%s\" is in
> ASSUME_PROVIDED, ignoring" % pkg)
> + if pkg.startswith("multiconfig:"):
> + pkgs_to_build.remove(pkg)
> + pkgs_to_build.append(pkg.replace("multiconfig:",
> "mc:"))
> if 'world' in pkgs_to_build:
> pkgs_to_build.remove('world')
> @@ -1583,7 +1613,7 @@ class BBCooker:
> bb.providers.buildWorldTargetList(self.recipecaches[mc],
> task) for t in self.recipecaches[mc].world_target:
> if mc:
> - t = "multiconfig:" + mc + ":" + t
> + t = "mc:" + mc + ":" + t
> pkgs_to_build.append(t)
>
> if 'universe' in pkgs_to_build:
> @@ -1602,7 +1632,7 @@ class BBCooker:
> bb.debug(1, "Skipping %s for universe
> tasks as task %s doesn't exist" % (t, task)) continue
> if mc:
> - t = "multiconfig:" + mc + ":" + t
> + t = "mc:" + mc + ":" + t
> pkgs_to_build.append(t)
>
> return pkgs_to_build
> @@ -1615,9 +1645,11 @@ class BBCooker:
>
> def post_serve(self):
> prserv.serv.auto_shutdown()
> + if self.hashserv:
> + self.hashserv.process.terminate()
> + self.hashserv.process.join()
> bb.event.fire(CookerExit(), self.data)
>
> -
> def shutdown(self, force = False):
> if force:
> self.state = state.forceshutdown
> @@ -1632,6 +1664,7 @@ class BBCooker:
>
> def reset(self):
> self.initConfigurationData()
> + self.handlePRServ()
>
> def clientComplete(self):
> """Called when the client is done using the server"""
> @@ -1865,35 +1898,6 @@ class ParsingFailure(Exception):
> self.recipe = recipe
> Exception.__init__(self, realexception, recipe)
>
> -class Feeder(multiprocessing.Process):
> - def __init__(self, jobs, to_parsers, quit):
> - self.quit = quit
> - self.jobs = jobs
> - self.to_parsers = to_parsers
> - multiprocessing.Process.__init__(self)
> -
> - def run(self):
> - while True:
> - try:
> - quit = self.quit.get_nowait()
> - except queue.Empty:
> - pass
> - else:
> - if quit == 'cancel':
> - self.to_parsers.cancel_join_thread()
> - break
> -
> - try:
> - job = self.jobs.pop()
> - except IndexError:
> - break
> -
> - try:
> - self.to_parsers.put(job, timeout=0.5)
> - except queue.Full:
> - self.jobs.insert(0, job)
> - continue
> -
> class Parser(multiprocessing.Process):
> def __init__(self, jobs, results, quit, init, profile):
> self.jobs = jobs
> @@ -1940,11 +1944,8 @@ class Parser(multiprocessing.Process):
> result = pending.pop()
> else:
> try:
> - job = self.jobs.get(timeout=0.25)
> - except queue.Empty:
> - continue
> -
> - if job is None:
> + job = self.jobs.pop()
> + except IndexError:
> break
> result = self.parse(*job)
>
> @@ -2028,14 +2029,15 @@ class CookerParser(object):
> multiprocessing.util.Finalize(None,
> bb.codeparser.parser_cache_save, exitpriority=1)
> multiprocessing.util.Finalize(None, bb.fetch.fetcher_parse_save,
> exitpriority=1)
> - self.feeder_quit = multiprocessing.Queue(maxsize=1)
> self.parser_quit =
> multiprocessing.Queue(maxsize=self.num_processes)
> - self.jobs =
> multiprocessing.Queue(maxsize=self.num_processes) self.result_queue =
> multiprocessing.Queue()
> - self.feeder = Feeder(self.willparse, self.jobs,
> self.feeder_quit)
> - self.feeder.start()
> +
> + def chunkify(lst,n):
> + return [lst[i::n] for i in range(n)]
> + self.jobs = chunkify(self.willparse, self.num_processes)
> +
> for i in range(0, self.num_processes):
> - parser = Parser(self.jobs, self.result_queue,
> self.parser_quit, init, self.cooker.configuration.profile)
> + parser = Parser(self.jobs[i], self.result_queue,
> self.parser_quit, init, self.cooker.configuration.profile)
> parser.start() self.process_names.append(parser.name)
> self.processes.append(parser)
> @@ -2056,17 +2058,20 @@ class CookerParser(object):
> self.total)
>
> bb.event.fire(event, self.cfgdata)
> - self.feeder_quit.put(None)
> for process in self.processes:
> self.parser_quit.put(None)
> else:
> - self.feeder_quit.put('cancel')
> -
> self.parser_quit.cancel_join_thread()
> for process in self.processes:
> self.parser_quit.put(None)
>
> - self.jobs.cancel_join_thread()
> + # Cleanup the queue before call process.join(), otherwise
> there might be
> + # deadlocks.
> + while True:
> + try:
> + self.result_queue.get(timeout=0.25)
> + except queue.Empty:
> + break
>
> for process in self.processes:
> if force:
> @@ -2074,7 +2079,6 @@ class CookerParser(object):
> process.terminate()
> else:
> process.join()
> - self.feeder.join()
>
> sync = threading.Thread(target=self.bb_cache.sync)
> sync.start()
> diff --git a/bitbake/lib/bb/cookerdata.py
> b/bitbake/lib/bb/cookerdata.py index 5df66e6..472423f 100644
> --- a/bitbake/lib/bb/cookerdata.py
> +++ b/bitbake/lib/bb/cookerdata.py
> @@ -1,6 +1,3 @@
> -#!/usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # Copyright (C) 2003, 2004 Chris Larson
> # Copyright (C) 2003, 2004 Phil Blundell
> @@ -9,23 +6,14 @@
> # Copyright (C) 2005 ROAD GmbH
> # Copyright (C) 2006 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import logging
> import os
> import re
> import sys
> +import hashlib
> from functools import wraps
> import bb
> from bb import data
> @@ -134,6 +122,7 @@ class CookerConfiguration(object):
> self.profile = False
> self.nosetscene = False
> self.setsceneonly = False
> + self.skipsetscene = False
> self.invalidate_stamp = False
> self.dump_signatures = []
> self.dry_run = False
> @@ -279,12 +268,13 @@ class CookerDataBuilder(object):
> self.mcdata = {}
>
> def parseBaseConfiguration(self):
> + data_hash = hashlib.sha256()
> try:
> - bb.parse.init_parser(self.basedata)
> self.data = self.parseConfigurationFiles(self.prefiles,
> self.postfiles)
> if self.data.getVar("BB_WORKERCONTEXT", False) is None:
> bb.fetch.fetcher_init(self.data)
> + bb.parse.init_parser(self.data)
> bb.codeparser.parser_cache_init(self.data)
>
> bb.event.fire(bb.event.ConfigParsed(), self.data)
> @@ -302,7 +292,7 @@ class CookerDataBuilder(object):
> bb.event.fire(bb.event.ConfigParsed(), self.data)
>
> bb.parse.init_parser(self.data)
> - self.data_hash = self.data.get_hash()
> + data_hash.update(self.data.get_hash().encode('utf-8'))
> self.mcdata[''] = self.data
>
> multiconfig = (self.data.getVar("BBMULTICONFIG") or
> "").split() @@ -310,9 +300,11 @@ class CookerDataBuilder(object):
> mcdata = self.parseConfigurationFiles(self.prefiles,
> self.postfiles, config) bb.event.fire(bb.event.ConfigParsed(), mcdata)
> self.mcdata[config] = mcdata
> + data_hash.update(mcdata.get_hash().encode('utf-8'))
> if multiconfig:
> bb.event.fire(bb.event.MultiConfigParsed(self.mcdata),
> self.data)
> + self.data_hash = data_hash.hexdigest()
> except (SyntaxError, bb.BBHandledException):
> raise bb.BBHandledException
> except bb.data_smart.ExpansionError as e:
> @@ -354,14 +346,24 @@ class CookerDataBuilder(object):
> data = parse_config_file(layerconf, data)
>
> layers = (data.getVar('BBLAYERS') or "").split()
> + broken_layers = []
>
> data = bb.data.createCopy(data)
> approved = bb.utils.approved_variables()
> +
> + # Check whether present layer directories exist
> for layer in layers:
> if not os.path.isdir(layer):
> - parselog.critical("Layer directory '%s' does not
> exist! "
> - "Please check BBLAYERS in %s"
> % (layer, layerconf))
> - sys.exit(1)
> + broken_layers.append(layer)
> +
> + if broken_layers:
> + parselog.critical("The following layer directories
> do not exist:")
> + for layer in broken_layers:
> + parselog.critical(" %s", layer)
> + parselog.critical("Please check BBLAYERS in %s" %
> (layerconf))
> + sys.exit(1)
> +
> + for layer in layers:
> parselog.debug(2, "Adding layer %s", layer)
> if 'HOME' in approved and '~' in layer:
> layer = os.path.expanduser(layer)
> @@ -391,7 +393,11 @@ class CookerDataBuilder(object):
> bb.fatal("BBFILES_DYNAMIC entries must be of the
> form <collection name>:<filename pattern>, not:\n %s" % "\n
> ".join(invalid)) layerseries =
> set((data.getVar("LAYERSERIES_CORENAMES") or "").split())
> + collections_tmp = collections[:]
> for c in collections:
> + collections_tmp.remove(c)
> + if c in collections_tmp:
> + bb.fatal("Found duplicated BBFILE_COLLECTIONS
> '%s', check bblayers.conf or layer.conf to fix it." % c) compat =
> set((data.getVar("LAYERSERIES_COMPAT_%s" % c) or "").split()) if
> compat and not (compat & layerseries): bb.fatal("Layer %s is not
> compatible with the core layer which only supports these series: %s
> (layer is compatible with %s)" diff --git
> a/bitbake/lib/bb/daemonize.py b/bitbake/lib/bb/daemonize.py index
> c937675..f01e6ec 100644 --- a/bitbake/lib/bb/daemonize.py +++
> b/bitbake/lib/bb/daemonize.py @@ -1,3 +1,7 @@
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> """
> Python Daemonizing helper
>
> diff --git a/bitbake/lib/bb/data.py b/bitbake/lib/bb/data.py
> index d66d98c..0d75d0c 100644
> --- a/bitbake/lib/bb/data.py
> +++ b/bitbake/lib/bb/data.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake 'Data' implementations
>
> @@ -22,18 +20,7 @@ the speed is more critical here.
> # Copyright (C) 2003, 2004 Chris Larson
> # Copyright (C) 2005 Holger Hans Peter Freyther
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
> # Based on functions from the base bb module, Copyright 2003 Holger
> Schurig
> @@ -143,7 +130,7 @@ def emit_var(var, o=sys.__stdout__, d = init(),
> all=False): if all:
> oval = d.getVar(var, False)
> val = d.getVar(var)
> - except (KeyboardInterrupt, bb.build.FuncFailed):
> + except (KeyboardInterrupt):
> raise
> except Exception as exc:
> o.write('# expansion of %s threw %s: %s\n' % (var,
> exc.__class__.__name__, str(exc))) @@ -322,8 +309,6 @@ def
> build_dependencies(key, keys, shelldeps, varflagsexcl, d): if
> varflags.get("python"): value = d.getVarFlag(key, "_content", False)
> parser = bb.codeparser.PythonParser(key, logger)
> - if value and "\t" in value:
> - logger.warning("Variable %s contains tabs,
> please remove these (%s)" % (key, d.getVar("FILE")))
> parser.parse_python(value, filename=varflags.get("filename"),
> lineno=varflags.get("lineno")) deps = deps | parser.references deps =
> deps | (keys & parser.execs) @@ -437,8 +422,8 @@ def
> generate_dependency_hash(tasklist, gendeps, lookupcache, whitelist,
> fn): var = lookupcache[dep] if var is not None:
> data = data + str(var)
> - k = fn + "." + task
> - basehash[k] = hashlib.md5(data.encode("utf-8")).hexdigest()
> + k = fn + ":" + task
> + basehash[k] =
> hashlib.sha256(data.encode("utf-8")).hexdigest() taskdeps[task] =
> alldeps
> return taskdeps, basehash
> diff --git a/bitbake/lib/bb/data_smart.py
> b/bitbake/lib/bb/data_smart.py index 67af380..dd5c618 100644
> --- a/bitbake/lib/bb/data_smart.py
> +++ b/bitbake/lib/bb/data_smart.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake Smart Dictionary Implementation
>
> @@ -14,18 +12,8 @@ BitBake build tools.
> # Copyright (C) 2005 Uli Luckas
> # Copyright (C) 2005 ROAD GmbH
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. # Based on functions from the base bb module,
> Copyright 2003 Holger Schurig
> import copy, re, sys, traceback
> @@ -39,10 +27,11 @@ from bb.COW import COWDictBase
> logger = logging.getLogger("BitBake.Data")
>
> __setvar_keyword__ = ["_append", "_prepend", "_remove"]
> -__setvar_regexp__ =
> re.compile('(?P<base>.*?)(?P<keyword>_append|_prepend|_remove)(_(?P<add>[^A-Z]*))?$')
> -__expand_var_regexp__ = re.compile(r"\${[^{}@\n\t :]+}")
> +__setvar_regexp__ =
> re.compile(r'(?P<base>.*?)(?P<keyword>_append|_prepend|_remove)(_(?P<add>[^A-Z]*))?$')
> +__expand_var_regexp__ = re.compile(r"\${[a-zA-Z0-9\-_+./~]+?}")
> __expand_python_regexp__ = re.compile(r"\${@.+?}")
> -__whitespace_split__ = re.compile('(\s)') +__whitespace_split__ =
> re.compile(r'(\s)') +__override_regexp__ = re.compile(r'[a-z0-9]+')
> def infer_caller_details(loginfo, parent = False, varval = True):
> """Save the caller the trouble of specifying everything."""
> @@ -597,7 +586,7 @@ class DataSmart(MutableMapping):
> # aka pay the cookie monster
> override = var[var.rfind('_')+1:]
> shortvar = var[:var.rfind('_')]
> - while override and override.islower():
> + while override and __override_regexp__.match(override):
> if shortvar not in self.overridedata:
> self.overridedata[shortvar] = []
> if [var, override] not in self.overridedata[shortvar]:
> @@ -1073,4 +1062,4 @@ class DataSmart(MutableMapping):
> data.update({i:value})
>
> data_str = str([(k, data[k]) for k in sorted(data.keys())])
> - return hashlib.md5(data_str.encode("utf-8")).hexdigest()
> + return hashlib.sha256(data_str.encode("utf-8")).hexdigest()
> diff --git a/bitbake/lib/bb/event.py b/bitbake/lib/bb/event.py
> index 5b1b094..d44621e 100644
> --- a/bitbake/lib/bb/event.py
> +++ b/bitbake/lib/bb/event.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake 'Event' implementation
>
> @@ -9,18 +7,8 @@ BitBake build tools.
>
> # Copyright (C) 2003, 2004 Chris Larson
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import os, sys
> import warnings
> @@ -136,6 +124,7 @@ def fire_class_handlers(event, d):
> ui_queue = []
> @atexit.register
> def print_ui_queue():
> + global ui_queue
> """If we're exiting before a UI has been spawned, display any
> queued LogRecords to the console."""
> logger = logging.getLogger("BitBake")
> @@ -180,6 +169,7 @@ def print_ui_queue():
> logger.removeHandler(stderr)
> else:
> logger.removeHandler(stdout)
> + ui_queue = []
>
> def fire_ui_handlers(event, d):
> global _thread_lock
> @@ -414,23 +404,6 @@ class RecipeTaskPreProcess(RecipeEvent):
> class RecipeParsed(RecipeEvent):
> """ Recipe Parsing Complete """
>
> -class StampUpdate(Event):
> - """Trigger for any adjustment of the stamp files to happen"""
> -
> - def __init__(self, targets, stampfns):
> - self._targets = targets
> - self._stampfns = stampfns
> - Event.__init__(self)
> -
> - def getStampPrefix(self):
> - return self._stampfns
> -
> - def getTargets(self):
> - return self._targets
> -
> - stampPrefix = property(getStampPrefix)
> - targets = property(getTargets)
> -
> class BuildBase(Event):
> """Base class for bitbake build events"""
>
> diff --git a/bitbake/lib/bb/exceptions.py
> b/bitbake/lib/bb/exceptions.py index cd71343..ecbad59 100644
> --- a/bitbake/lib/bb/exceptions.py
> +++ b/bitbake/lib/bb/exceptions.py
> @@ -1,3 +1,6 @@
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
>
> import inspect
> import traceback
> diff --git a/bitbake/lib/bb/fetch2/__init__.py
> b/bitbake/lib/bb/fetch2/__init__.py index 572b71a..1f5f8f1 100644
> --- a/bitbake/lib/bb/fetch2/__init__.py
> +++ b/bitbake/lib/bb/fetch2/__init__.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake 'Fetch' implementations
>
> @@ -10,18 +8,7 @@ BitBake build tools.
> # Copyright (C) 2003, 2004 Chris Larson
> # Copyright (C) 2012 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
> # Based on functions from the base bb module, Copyright 2003 Holger
> Schurig
> @@ -256,7 +243,7 @@ class URI(object):
>
> # Identify if the URI is relative or not
> if urlp.scheme in self._relative_schemes and \
> - re.compile("^\w+:(?!//)").match(uri):
> + re.compile(r"^\w+:(?!//)").match(uri):
> self.relative = True
>
> if not self.relative:
> @@ -524,7 +511,7 @@ def fetcher_parse_save():
> def fetcher_parse_done():
> _checksum_cache.save_merge()
>
> -def fetcher_compare_revisions():
> +def fetcher_compare_revisions(d):
> """
> Compare the revisions in the persistant cache with current
> values and return true/false on whether they've changed.
> @@ -777,7 +764,8 @@ def get_srcrev(d,
> method_name='sortable_revision'): #
> format = d.getVar('SRCREV_FORMAT')
> if not format:
> - raise FetchError("The SRCREV_FORMAT variable must be set
> when multiple SCMs are used.")
> + raise FetchError("The SRCREV_FORMAT variable must be set
> when multiple SCMs are used.\n"\
> + "The SCMs are:\n%s" % '\n'.join(scms))
>
> name_to_rev = {}
> seenautoinc = False
> @@ -855,10 +843,18 @@ def runfetchcmd(cmd, d, quiet=False,
> cleanup=None, log=None, workdir=None): if val:
> cmd = 'export ' + var + '=\"%s\"; %s' % (val, cmd)
>
> + # Ensure that a _PYTHON_SYSCONFIGDATA_NAME value set by a recipe
> + # (for example via python3native.bbclass since warrior) is not
> set for
> + # host Python (otherwise tools like git-make-shallow will fail)
> + cmd = 'unset _PYTHON_SYSCONFIGDATA_NAME; ' + cmd
> +
> # Disable pseudo as it may affect ssh, potentially causing it to
> hang. cmd = 'export PSEUDO_DISABLED=1; ' + cmd
>
> - logger.debug(1, "Running %s", cmd)
> + if workdir:
> + logger.debug(1, "Running '%s' in %s" % (cmd, workdir))
> + else:
> + logger.debug(1, "Running %s", cmd)
>
> success = False
> error_message = ""
> @@ -894,7 +890,7 @@ def check_network_access(d, info, url):
> log remote network access, and error if BB_NO_NETWORK is set or
> the given URI is untrusted
> """
> - if d.getVar("BB_NO_NETWORK") == "1":
> + if bb.utils.to_boolean(d.getVar("BB_NO_NETWORK")):
> raise NetworkAccess(url, info)
> elif not trusted_network(d, url):
> raise UntrustedUrl(url, info)
> @@ -966,7 +962,8 @@ def rename_bad_checksum(ud, suffix):
>
> new_localpath = "%s_bad-checksum_%s" % (ud.localpath, suffix)
> bb.warn("Renaming %s to %s" % (ud.localpath, new_localpath))
> - bb.utils.movefile(ud.localpath, new_localpath)
> + if not bb.utils.movefile(ud.localpath, new_localpath):
> + bb.warn("Renaming %s to %s failed, grep movefile in
> log.do_fetch to see why" % (ud.localpath, new_localpath))
>
> def try_mirror_url(fetch, origud, ud, ld, check = False):
> @@ -1027,7 +1024,7 @@ def try_mirror_url(fetch, origud, ud, ld, check
> = False): raise
>
> except IOError as e:
> - if e.errno in [os.errno.ESTALE]:
> + if e.errno in [errno.ESTALE]:
> logger.warning("Stale Error Observed %s." % ud.url)
> return False
> raise
> @@ -1094,7 +1091,7 @@ def trusted_network(d, url):
> BB_ALLOWED_NETWORKS is set globally or for a specific recipe.
> Note: modifies SRC_URI & mirrors.
> """
> - if d.getVar('BB_NO_NETWORK') == "1":
> + if bb.utils.to_boolean(d.getVar("BB_NO_NETWORK")):
> return True
>
> pkgname = d.expand(d.getVar('PN', False))
> @@ -1403,7 +1400,7 @@ class FetchMethod(object):
> Fetch urls
> Assumes localpath was called first
> """
> - raise NoMethodError(url)
> + raise NoMethodError(urldata.url)
>
> def unpack(self, urldata, rootdir, data):
> iterate = False
> @@ -1469,7 +1466,7 @@ class FetchMethod(object):
> else:
> cmd = 'rpm2cpio.sh %s | cpio -id' % (file)
> elif file.endswith('.deb') or file.endswith('.ipk'):
> - output = subprocess.check_output('ar -t %s' % file,
> preexec_fn=subprocess_setup, shell=True)
> + output = subprocess.check_output(['ar', '-t', file],
> preexec_fn=subprocess_setup) datafile = None
> if output:
> for line in output.decode().splitlines():
> @@ -1547,7 +1544,7 @@ class FetchMethod(object):
> Check the status of a URL
> Assumes localpath was called first
> """
> - logger.info("URL %s could not be checked for status since no
> method exists.", url)
> + logger.info("URL %s could not be checked for status since no
> method exists.", urldata.url) return True
>
> def latest_revision(self, ud, d, name):
> @@ -1555,7 +1552,7 @@ class FetchMethod(object):
> Look in the cache for the latest revision, if not present
> ask the SCM. """
> if not hasattr(self, "_latest_revision"):
> - raise ParameterError("The fetcher for this URL does not
> support _latest_revision", url)
> + raise ParameterError("The fetcher for this URL does not
> support _latest_revision", ud.url)
> revs = bb.persist_data.persist('BB_URI_HEADREVS', d)
> key = self.generate_revision_key(ud, d, name)
> @@ -1638,7 +1635,7 @@ class Fetch(object):
> urls = self.urls
>
> network = self.d.getVar("BB_NO_NETWORK")
> - premirroronly = (self.d.getVar("BB_FETCH_PREMIRRORONLY") ==
> "1")
> + premirroronly =
> bb.utils.to_boolean(self.d.getVar("BB_FETCH_PREMIRRORONLY"))
> for u in urls:
> ud = self.ud[u]
> @@ -1716,7 +1713,7 @@ class Fetch(object):
> update_stamp(ud, self.d)
>
> except IOError as e:
> - if e.errno in [os.errno.ESTALE]:
> + if e.errno in [errno.ESTALE]:
> logger.error("Stale Error Observed %s." % u)
> raise ChecksumError("Stale Error Detected")
>
> @@ -1786,7 +1783,7 @@ class Fetch(object):
>
> for url in urls:
> if url not in self.ud:
> - self.ud[url] = FetchData(url, d)
> + self.ud[url] = FetchData(url, self.d)
> ud = self.ud[url]
> ud.setup_localpath(self.d)
>
> diff --git a/bitbake/lib/bb/fetch2/bzr.py
> b/bitbake/lib/bb/fetch2/bzr.py index 658502f..c56d875 100644
> --- a/bitbake/lib/bb/fetch2/bzr.py
> +++ b/bitbake/lib/bb/fetch2/bzr.py
> @@ -10,18 +10,8 @@ BitBake 'Fetch' implementation for bzr.
> # BitBake build tools.
> # Copyright (C) 2003, 2004 Chris Larson
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import os
> import sys
> diff --git a/bitbake/lib/bb/fetch2/clearcase.py
> b/bitbake/lib/bb/fetch2/clearcase.py index 3a6573d..3dd93ad 100644
> --- a/bitbake/lib/bb/fetch2/clearcase.py
> +++ b/bitbake/lib/bb/fetch2/clearcase.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake 'Fetch' clearcase implementation
>
> @@ -47,18 +45,7 @@ User credentials:
> """
> # Copyright (C) 2014 Siemens AG
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
>
> import os
> @@ -67,6 +54,8 @@ import shutil
> import bb
> from bb.fetch2 import FetchMethod
> from bb.fetch2 import FetchError
> +from bb.fetch2 import MissingParameterError
> +from bb.fetch2 import ParameterError
> from bb.fetch2 import runfetchcmd
> from bb.fetch2 import logger
>
> @@ -92,7 +81,7 @@ class ClearCase(FetchMethod):
> if 'protocol' in ud.parm:
> ud.proto = ud.parm['protocol']
> if not ud.proto in ('http', 'https'):
> - raise fetch2.ParameterError("Invalid protocol type",
> ud.url)
> + raise ParameterError("Invalid protocol type", ud.url)
>
> ud.vob = ''
> if 'vob' in ud.parm:
> diff --git a/bitbake/lib/bb/fetch2/cvs.py
> b/bitbake/lib/bb/fetch2/cvs.py index 0e0a319..1b35ba4 100644
> --- a/bitbake/lib/bb/fetch2/cvs.py
> +++ b/bitbake/lib/bb/fetch2/cvs.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake 'Fetch' implementations
>
> @@ -10,20 +8,9 @@ BitBake build tools.
>
> # Copyright (C) 2003, 2004 Chris Larson
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. -#
> -#Based on functions from the base bb module, Copyright 2003 Holger
> Schurig +# Based on functions from the base bb module, Copyright 2003
> Holger Schurig #
>
> import os
> diff --git a/bitbake/lib/bb/fetch2/git.py
> b/bitbake/lib/bb/fetch2/git.py index 59a2ee8..2d1d2ca 100644
> --- a/bitbake/lib/bb/fetch2/git.py
> +++ b/bitbake/lib/bb/fetch2/git.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake 'Fetch' git implementation
>
> @@ -55,20 +53,10 @@ Supported SRC_URI options are:
>
> """
>
> -#Copyright (C) 2005 Richard Purdie
> +# Copyright (C) 2005 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import collections
> import errno
> @@ -199,7 +187,7 @@ class Git(FetchMethod):
> depth_default = 1
> ud.shallow_depths = collections.defaultdict(lambda:
> depth_default)
> - revs_default = d.getVar("BB_GIT_SHALLOW_REVS", True)
> + revs_default = d.getVar("BB_GIT_SHALLOW_REVS")
> ud.shallow_revs = []
> ud.branches = {}
> for pos, name in enumerate(ud.names):
> @@ -318,7 +306,7 @@ class Git(FetchMethod):
> def try_premirror(self, ud, d):
> # If we don't do this, updating an existing checkout with
> only premirrors # is not possible
> - if d.getVar("BB_FETCH_PREMIRRORONLY") is not None:
> + if bb.utils.to_boolean(d.getVar("BB_FETCH_PREMIRRORONLY")):
> return True
> if os.path.exists(ud.clonedir):
> return False
> @@ -476,6 +464,8 @@ class Git(FetchMethod):
> if os.path.exists(destdir):
> bb.utils.prunedir(destdir)
>
> + need_lfs = ud.parm.get("lfs", "1") == "1"
> +
> source_found = False
> source_error = []
>
> @@ -503,6 +493,13 @@ class Git(FetchMethod):
>
> repourl = self._get_repo_url(ud)
> runfetchcmd("%s remote set-url origin %s" % (ud.basecmd,
> repourl), d, workdir=destdir) +
> + if self._contains_lfs(ud, d, destdir):
> + if need_lfs and not self._find_git_lfs(d):
> + raise bb.fetch2.FetchError("Repository %s has LFS
> content, install git-lfs on host to download (or set lfs=0 to ignore
> it)" % (repourl))
> + else:
> + bb.note("Repository %s has LFS content but it is not
> being fetched" % (repourl)) +
> if not ud.nocheckout:
> if subdir != "":
> runfetchcmd("%s read-tree %s%s" % (ud.basecmd,
> ud.revisions[ud.names[0]], readpathspec), d, @@ -522,9 +519,17 @@
> class Git(FetchMethod): def clean(self, ud, d):
> """ clean the git directory """
>
> - bb.utils.remove(ud.localpath, True)
> - bb.utils.remove(ud.fullmirror)
> - bb.utils.remove(ud.fullmirror + ".done")
> + to_remove = [ud.localpath, ud.fullmirror, ud.fullmirror +
> ".done"]
> + # The localpath is a symlink to clonedir when it is cloned
> from a
> + # mirror, so remove both of them.
> + if os.path.islink(ud.localpath):
> + clonedir = os.path.realpath(ud.localpath)
> + to_remove.append(clonedir)
> +
> + for r in to_remove:
> + if os.path.exists(r):
> + bb.note('Removing %s' % r)
> + bb.utils.remove(r, True)
>
> def supports_srcrev(self):
> return True
> @@ -545,6 +550,27 @@ class Git(FetchMethod):
> raise bb.fetch2.FetchError("The command '%s' gave output
> with more then 1 line unexpectedly, output: '%s'" % (cmd, output))
> return output.split()[0] != "0"
> + def _contains_lfs(self, ud, d, wd):
> + """
> + Check if the repository has 'lfs' (large file) content
> + """
> + cmd = "%s grep lfs HEAD:.gitattributes | wc -l" % (
> + ud.basecmd)
> + try:
> + output = runfetchcmd(cmd, d, quiet=True, workdir=wd)
> + if int(output) > 0:
> + return True
> + except (bb.fetch2.FetchError,ValueError):
> + pass
> + return False
> +
> + def _find_git_lfs(self, d):
> + """
> + Return True if git-lfs can be found, False otherwise.
> + """
> + import shutil
> + return shutil.which("git-lfs", path=d.getVar('PATH')) is not
> None +
> def _get_repo_url(self, ud):
> """
> Return the repository URL
> @@ -615,7 +641,7 @@ class Git(FetchMethod):
> """
> pupver = ('', '')
>
> - tagregex = re.compile(d.getVar('UPSTREAM_CHECK_GITTAGREGEX')
> or "(?P<pver>([0-9][\.|_]?)+)")
> + tagregex = re.compile(d.getVar('UPSTREAM_CHECK_GITTAGREGEX')
> or r"(?P<pver>([0-9][\.|_]?)+)") try:
> output = self._lsremote(ud, d, "refs/tags/*")
> except (bb.fetch2.FetchError, bb.fetch2.NetworkAccess) as e:
> @@ -630,7 +656,7 @@ class Git(FetchMethod):
>
> tag_head = line.split("/")[-1]
> # Ignore non-released branches
> - m = re.search("(alpha|beta|rc|final)+", tag_head)
> + m = re.search(r"(alpha|beta|rc|final)+", tag_head)
> if m:
> continue
>
> diff --git a/bitbake/lib/bb/fetch2/gitannex.py
> b/bitbake/lib/bb/fetch2/gitannex.py index a9b69ca..1d497dc 100644
> --- a/bitbake/lib/bb/fetch2/gitannex.py
> +++ b/bitbake/lib/bb/fetch2/gitannex.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake 'Fetch' git annex implementation
> """
> @@ -7,18 +5,8 @@ BitBake 'Fetch' git annex implementation
> # Copyright (C) 2014 Otavio Salvador
> # Copyright (C) 2014 O.S. Systems Software LTDA.
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import os
> import bb
> diff --git a/bitbake/lib/bb/fetch2/gitsm.py
> b/bitbake/lib/bb/fetch2/gitsm.py index 35729db..c622771 100644
> --- a/bitbake/lib/bb/fetch2/gitsm.py
> +++ b/bitbake/lib/bb/fetch2/gitsm.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake 'Fetch' git submodules implementation
>
> @@ -16,18 +14,8 @@ NOTE: Switching a SRC_URI from "git://" to
> "gitsm://" requires a clean of your r
> # Copyright (C) 2013 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import os
> import bb
> @@ -45,60 +33,97 @@ class GitSM(Git):
> """
> return ud.type in ['gitsm']
>
> - @staticmethod
> - def parse_gitmodules(gitmodules):
> - modules = {}
> - module = ""
> - for line in gitmodules.splitlines():
> - if line.startswith('[submodule'):
> - module = line.split('"')[1]
> - modules[module] = {}
> - elif module and line.strip().startswith('path'):
> - path = line.split('=')[1].strip()
> - modules[module]['path'] = path
> - elif module and line.strip().startswith('url'):
> - url = line.split('=')[1].strip()
> - modules[module]['url'] = url
> - return modules
> -
> - def update_submodules(self, ud, d):
> + def process_submodules(self, ud, workdir, function, d):
> + """
> + Iterate over all of the submodules in this repository and
> execute
> + the 'function' for each of them.
> + """
> +
> submodules = []
> paths = {}
> + revision = {}
> uris = {}
> - local_paths = {}
> -
> + subrevision = {}
> +
> + def parse_gitmodules(gitmodules):
> + modules = {}
> + module = ""
> + for line in gitmodules.splitlines():
> + if line.startswith('[submodule'):
> + module = line.split('"')[1]
> + modules[module] = {}
> + elif module and line.strip().startswith('path'):
> + path = line.split('=')[1].strip()
> + modules[module]['path'] = path
> + elif module and line.strip().startswith('url'):
> + url = line.split('=')[1].strip()
> + modules[module]['url'] = url
> + return modules
> +
> + # Collect the defined submodules, and their attributes
> for name in ud.names:
> try:
> - gitmodules = runfetchcmd("%s show %s:.gitmodules" %
> (ud.basecmd, ud.revisions[name]), d, quiet=True, workdir=ud.clonedir)
> + gitmodules = runfetchcmd("%s show %s:.gitmodules" %
> (ud.basecmd, ud.revisions[name]), d, quiet=True, workdir=workdir)
> except: # No submodules to update
> continue
>
> - for m, md in self.parse_gitmodules(gitmodules).items():
> + for m, md in parse_gitmodules(gitmodules).items():
> + try:
> + module_hash = runfetchcmd("%s ls-tree -z -d %s
> %s" % (ud.basecmd, ud.revisions[name], md['path']), d, quiet=True,
> workdir=workdir)
> + except:
> + # If the command fails, we don't have a valid
> file to check. If it doesn't
> + # fail -- it still might be a failure, see next
> check...
> + module_hash = ""
> +
> + if not module_hash:
> + logger.debug(1, "submodule %s is defined, but is
> not initialized in the repository. Skipping", m)
> + continue
> +
> submodules.append(m)
> paths[m] = md['path']
> + revision[m] = ud.revisions[name]
> uris[m] = md['url']
> + subrevision[m] = module_hash.split()[2]
> +
> + # Convert relative to absolute uri based on parent
> uri if uris[m].startswith('..'):
> newud = copy.copy(ud)
> - newud.path =
> os.path.realpath(os.path.join(newud.path, md['url']))
> + newud.path =
> os.path.realpath(os.path.join(newud.path, uris[m])) uris[m] =
> Git._get_repo_url(self, newud)
> for module in submodules:
> - module_hash = runfetchcmd("%s ls-tree -z -d %s %s" %
> (ud.basecmd, ud.revisions[name], paths[module]), d, quiet=True,
> workdir=ud.clonedir)
> - module_hash = module_hash.split()[2]
> + # Translate the module url into a SRC_URI
> +
> + if "://" in uris[module]:
> + # Properly formated URL already
> + proto = uris[module].split(':', 1)[0]
> + url = uris[module].replace('%s:' % proto, 'gitsm:',
> 1)
> + else:
> + if ":" in uris[module]:
> + # Most likely an SSH style reference
> + proto = "ssh"
> + if ":/" in uris[module]:
> + # Absolute reference, easy to convert..
> + url = "gitsm://" +
> uris[module].replace(':/', '/', 1)
> + else:
> + # Relative reference, no way to know if this
> is right!
> + logger.warning("Submodule included by %s
> refers to relative ssh reference %s. References may fail if not
> absolute." % (ud.url, uris[module]))
> + url = "gitsm://" + uris[module].replace(':',
> '/', 1)
> + else:
> + # This has to be a file reference
> + proto = "file"
> + url = "gitsm://" + uris[module]
>
> - # Build new SRC_URI
> - proto = uris[module].split(':', 1)[0]
> - url = uris[module].replace('%s:' % proto, 'gitsm:', 1)
> url += ';protocol=%s' % proto
> url += ";name=%s" % module
> - url += ";bareclone=1;nocheckout=1;nobranch=1"
> + url += ";subpath=%s" % module
>
> ld = d.createCopy()
> # Not necessary to set SRC_URI, since we're passing the
> URI to # Fetch.
> #ld.setVar('SRC_URI', url)
> - ld.setVar('SRCREV_%s' % module, module_hash)
> + ld.setVar('SRCREV_%s' % module, subrevision[module])
>
> # Workaround for issues with SRCPV/SRCREV_FORMAT errors
> # error refer to 'multiple' repositories. Only the
> repository @@ -106,145 +131,85 @@ class GitSM(Git):
> ld.setVar('SRCPV', d.getVar('SRCPV'))
> ld.setVar('SRCREV_FORMAT', module)
>
> - newfetch = Fetch([url], ld, cache=False)
> - newfetch.download()
> - local_paths[module] = newfetch.localpath(url)
> + function(ud, url, module, paths[module], ld)
>
> - # Correct the submodule references to the local download
> version...
> - runfetchcmd("%(basecmd)s config submodule.%(module)s.url
> %(url)s" % {'basecmd': ud.basecmd, 'module': module, 'url' :
> local_paths[module]}, d, workdir=ud.clonedir) -
> - symlink_path = os.path.join(ud.clonedir, 'modules',
> paths[module])
> - if not os.path.exists(symlink_path):
> - try:
> - os.makedirs(os.path.dirname(symlink_path),
> exist_ok=True)
> - except OSError:
> - pass
> - os.symlink(local_paths[module], symlink_path)
> -
> - return True
> + return submodules != []
>
> def need_update(self, ud, d):
> - main_repo_needs_update = Git.need_update(self, ud, d)
> -
> - # First check that the main repository has enough history
> fetched. If it doesn't, then we don't
> - # even have the .gitmodules and gitlinks for the submodules
> to attempt asking whether the
> - # submodules' histories are recent enough.
> - if main_repo_needs_update:
> + if Git.need_update(self, ud, d):
> return True
>
> - # Now check that the submodule histories are new enough. The
> git-submodule command doesn't have
> - # any clean interface for doing this aside from just
> attempting the checkout (with network
> - # fetched disabled).
> - return not self.update_submodules(ud, d)
> + try:
> + # Check for the nugget dropped by the download operation
> + known_srcrevs = runfetchcmd("%s config --get-all
> bitbake.srcrev" % \
> + (ud.basecmd), d, workdir=ud.clonedir)
>
> - def download(self, ud, d):
> - Git.download(self, ud, d)
> + if ud.revisions[ud.names[0]] not in
> known_srcrevs.split():
> + return True
> + except bb.fetch2.FetchError:
> + # No srcrev nuggets, so this is new and needs to be
> updated
> + return True
>
> - if not ud.shallow or ud.localpath != ud.fullshallow:
> - self.update_submodules(ud, d)
> + return False
>
> - def copy_submodules(self, submodules, ud, destdir, d):
> - if ud.bareclone:
> - repo_conf = destdir
> - else:
> - repo_conf = os.path.join(destdir, '.git')
> + def download(self, ud, d):
> + def download_submodule(ud, url, module, modpath, d):
> + url += ";bareclone=1;nobranch=1"
>
> - if submodules and not os.path.exists(os.path.join(repo_conf,
> 'modules')):
> - os.mkdir(os.path.join(repo_conf, 'modules'))
> + # Is the following still needed?
> + #url += ";nocheckout=1"
>
> - for module, md in submodules.items():
> - srcpath = os.path.join(ud.clonedir, 'modules',
> md['path'])
> - modpath = os.path.join(repo_conf, 'modules', md['path'])
> + try:
> + newfetch = Fetch([url], d, cache=False)
> + newfetch.download()
> + # Drop a nugget to add each of the srcrevs we've
> fetched (used by need_update)
> + runfetchcmd("%s config --add bitbake.srcrev %s" % \
> + (ud.basecmd, ud.revisions[ud.names[0]]),
> d, workdir=ud.clonedir)
> + except Exception as e:
> + logger.error('gitsm: submodule download failed: %s
> %s' % (type(e).__name__, str(e)))
> + raise
>
> - if os.path.exists(srcpath):
> - if os.path.exists(os.path.join(srcpath, '.git')):
> - srcpath = os.path.join(srcpath, '.git')
> + Git.download(self, ud, d)
> + self.process_submodules(ud, ud.clonedir, download_submodule,
> d)
> - target = modpath
> - if os.path.exists(modpath):
> - target = os.path.dirname(modpath)
> + def unpack(self, ud, destdir, d):
> + def unpack_submodules(ud, url, module, modpath, d):
> + url += ";bareclone=1;nobranch=1"
>
> - os.makedirs(os.path.dirname(target), exist_ok=True)
> - runfetchcmd("cp -fpLR %s %s" % (srcpath, target), d)
> - elif os.path.exists(modpath):
> - # Module already exists, likely unpacked from a
> shallow mirror clone
> - pass
> + # Figure out where we clone over the bare submodules...
> + if ud.bareclone:
> + repo_conf = ud.destdir
> else:
> - # This is fatal, as we do NOT want git-submodule to
> hit the network
> - raise bb.fetch2.FetchError('Submodule %s does not
> exist in %s or %s.' % (module, srcpath, modpath)) -
> - def clone_shallow_local(self, ud, dest, d):
> - super(GitSM, self).clone_shallow_local(ud, dest, d)
> + repo_conf = os.path.join(ud.destdir, '.git')
>
> - # Copy over the submodules' fetched histories too.
> - repo_conf = os.path.join(dest, '.git')
> -
> - submodules = []
> - for name in ud.names:
> try:
> - gitmodules = runfetchcmd("%s show %s:.gitmodules" %
> (ud.basecmd, ud.revision), d, quiet=True, workdir=dest)
> - except:
> - # No submodules to update
> - continue
> + newfetch = Fetch([url], d, cache=False)
> +
> newfetch.unpack(root=os.path.dirname(os.path.join(repo_conf,
> 'modules', module)))
> + except Exception as e:
> + logger.error('gitsm: submodule unpack failed: %s %s'
> % (type(e).__name__, str(e)))
> + raise
>
> - submodules = self.parse_gitmodules(gitmodules)
> - self.copy_submodules(submodules, ud, dest, d)
> + local_path = newfetch.localpath(url)
>
> - def unpack(self, ud, destdir, d):
> - Git.unpack(self, ud, destdir, d)
> + # Correct the submodule references to the local download
> version...
> + runfetchcmd("%(basecmd)s config submodule.%(module)s.url
> %(url)s" % {'basecmd': ud.basecmd, 'module': module, 'url' :
> local_path}, d, workdir=ud.destdir)
> - # Copy over the submodules' fetched histories too.
> - if ud.bareclone:
> - repo_conf = ud.destdir
> - else:
> - repo_conf = os.path.join(ud.destdir, '.git')
> + if ud.shallow:
> + runfetchcmd("%(basecmd)s config
> submodule.%(module)s.shallow true" % {'basecmd': ud.basecmd,
> 'module': module}, d, workdir=ud.destdir)
> - update_submodules = False
> - paths = {}
> - uris = {}
> - local_paths = {}
> - for name in ud.names:
> + # Ensure the submodule repository is NOT set to bare,
> since we're checking it out... try:
> - gitmodules = runfetchcmd("%s show HEAD:.gitmodules"
> % (ud.basecmd), d, quiet=True, workdir=ud.destdir)
> + runfetchcmd("%s config core.bare false" %
> (ud.basecmd), d, quiet=True, workdir=os.path.join(repo_conf,
> 'modules', module)) except:
> - # No submodules to update
> - continue
> -
> - submodules = self.parse_gitmodules(gitmodules)
> - self.copy_submodules(submodules, ud, ud.destdir, d)
> -
> - submodules_queue = [(module, os.path.join(repo_conf,
> 'modules', md['path'])) for module, md in submodules.items()]
> - while len(submodules_queue) != 0:
> - module, modpath = submodules_queue.pop()
> -
> - # add submodule children recursively
> - try:
> - gitmodules = runfetchcmd("%s show
> HEAD:.gitmodules" % (ud.basecmd), d, quiet=True, workdir=modpath)
> - for m, md in
> self.parse_gitmodules(gitmodules).items():
> - submodules_queue.append([m,
> os.path.join(modpath, 'modules', md['path'])])
> - except:
> - # no children
> - pass
> -
> + logger.error("Unable to set git config core.bare to
> false for %s" % os.path.join(repo_conf, 'modules', module))
> + raise
>
> - # There are submodules to update
> - update_submodules = True
> -
> - # Determine (from the submodule) the correct url to
> reference
> - try:
> - output = runfetchcmd("%(basecmd)s config
> remote.origin.url" % {'basecmd': ud.basecmd}, d, workdir=modpath)
> - except bb.fetch2.FetchError as e:
> - # No remote url defined in this submodule
> - continue
> -
> - local_paths[module] = output
> -
> - # Setup the local URL properly (like git submodule
> init or sync would do...)
> - runfetchcmd("%(basecmd)s config
> submodule.%(module)s.url %(url)s" % {'basecmd': ud.basecmd, 'module':
> module, 'url' : local_paths[module]}, d, workdir=ud.destdir)
> + Git.unpack(self, ud, destdir, d)
>
> - # Ensure the submodule repository is NOT set to
> bare, since we're checking it out...
> - runfetchcmd("%s config core.bare false" %
> (ud.basecmd), d, quiet=True, workdir=modpath)
> + ret = self.process_submodules(ud, ud.destdir,
> unpack_submodules, d)
> - if update_submodules:
> - # Run submodule update, this sets up the directories --
> without touching the config
> + if not ud.bareclone and ret:
> + # All submodules should already be downloaded and
> configured in the tree. This simply sets
> + # up the configuration and checks out the files. The
> main project config should remain
> + # unmodified, and no download from the internet should
> occur. runfetchcmd("%s submodule update --recursive --no-fetch" %
> (ud.basecmd), d, quiet=True, workdir=ud.destdir) diff --git
> a/bitbake/lib/bb/fetch2/hg.py b/bitbake/lib/bb/fetch2/hg.py index
> 936d043..15d729e 100644 --- a/bitbake/lib/bb/fetch2/hg.py
> +++ b/bitbake/lib/bb/fetch2/hg.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake 'Fetch' implementation for mercurial DRCS (hg).
>
> @@ -9,20 +7,10 @@ BitBake 'Fetch' implementation for mercurial DRCS
> (hg). # Copyright (C) 2004 Marcin Juszkiewicz
> # Copyright (C) 2007 Robert Schuster
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
> # Based on functions from the base bb module, Copyright 2003 Holger
> Schurig +#
>
> import os
> import sys
> @@ -99,7 +87,7 @@ class Hg(FetchMethod):
> def try_premirror(self, ud, d):
> # If we don't do this, updating an existing checkout with
> only premirrors # is not possible
> - if d.getVar("BB_FETCH_PREMIRRORONLY") is not None:
> + if bb.utils.to_boolean(d.getVar("BB_FETCH_PREMIRRORONLY")):
> return True
> if os.path.exists(ud.moddir):
> return False
> diff --git a/bitbake/lib/bb/fetch2/local.py
> b/bitbake/lib/bb/fetch2/local.py index a114ac1..01d9ff9 100644
> --- a/bitbake/lib/bb/fetch2/local.py
> +++ b/bitbake/lib/bb/fetch2/local.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake 'Fetch' implementations
>
> @@ -10,20 +8,10 @@ BitBake build tools.
>
> # Copyright (C) 2003, 2004 Chris Larson
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
> # Based on functions from the base bb module, Copyright 2003 Holger
> Schurig +#
>
> import os
> import urllib.request, urllib.parse, urllib.error
> diff --git a/bitbake/lib/bb/fetch2/npm.py
> b/bitbake/lib/bb/fetch2/npm.py index 65bf5a3..9700e61 100644
> --- a/bitbake/lib/bb/fetch2/npm.py
> +++ b/bitbake/lib/bb/fetch2/npm.py
> @@ -1,5 +1,6 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> """
> BitBake 'Fetch' NPM implementation
>
> @@ -100,11 +101,19 @@ class Npm(FetchMethod):
> return False
> return True
>
> - def _runwget(self, ud, d, command, quiet):
> - logger.debug(2, "Fetching %s using command '%s'" % (ud.url,
> command))
> - bb.fetch2.check_network_access(d, command, ud.url)
> + def _runpack(self, ud, d, pkgfullname: str, quiet=False) -> str:
> + """
> + Runs npm pack on a full package name.
> + Returns the filename of the downloaded package
> + """
> + bb.fetch2.check_network_access(d, pkgfullname, ud.registry)
> dldir = d.getVar("DL_DIR")
> - runfetchcmd(command, d, quiet, workdir=dldir)
> + dldir = os.path.join(dldir, ud.prefixdir)
> +
> + command = "npm pack {} --registry {}".format(pkgfullname,
> ud.registry)
> + logger.debug(2, "Fetching {} using command '{}' in
> {}".format(pkgfullname, command, dldir))
> + filename = runfetchcmd(command, d, quiet, workdir=dldir)
> + return filename.rstrip()
>
> def _unpackdep(self, ud, pkg, data, destdir, dldir, d):
> file = data[pkg]['tgz']
> @@ -150,20 +159,11 @@ class Npm(FetchMethod):
> Parse the output of npm view --json; the last JSON result
> is assumed to be the one that we're interested in.
> '''
> - pdata = None
> - outdeps = {}
> - datalines = []
> - bracelevel = 0
> - for line in output.splitlines():
> - if bracelevel:
> - datalines.append(line)
> - elif '{' in line:
> - datalines = []
> - datalines.append(line)
> - bracelevel = bracelevel + line.count('{') -
> line.count('}')
> - if datalines:
> - pdata = json.loads('\n'.join(datalines))
> - return pdata
> + pdata = json.loads(output);
> + try:
> + return pdata[-1]
> + except:
> + return pdata
>
> def _getdependencies(self, pkg, data, version, d, ud,
> optional=False, fetchedlist=None): if fetchedlist is None:
> @@ -171,6 +171,9 @@ class Npm(FetchMethod):
> pkgfullname = pkg
> if version != '*' and not '/' in version:
> pkgfullname += "@'%s'" % version
> + if pkgfullname in fetchedlist:
> + return
> +
> logger.debug(2, "Calling getdeps on %s" % pkg)
> fetchcmd = "npm view %s --json --registry %s" %
> (pkgfullname, ud.registry) output = runfetchcmd(fetchcmd, d, True)
> @@ -190,15 +193,10 @@ class Npm(FetchMethod):
> if (not blacklist and 'linux' not in pkg_os) or
> '!linux' in pkg_os: logger.debug(2, "Skipping %s since it's
> incompatible with Linux" % pkg) return
> - #logger.debug(2, "Output URL is %s - %s - %s" %
> (ud.basepath, ud.basename, ud.localfile))
> - outputurl = pdata['dist']['tarball']
> + filename = self._runpack(ud, d, pkgfullname)
> data[pkg] = {}
> - data[pkg]['tgz'] = os.path.basename(outputurl)
> - if outputurl in fetchedlist:
> - return
> -
> - self._runwget(ud, d, "%s --directory-prefix=%s %s" %
> (self.basecmd, ud.prefixdir, outputurl), False)
> - fetchedlist.append(outputurl)
> + data[pkg]['tgz'] = filename
> + fetchedlist.append(pkgfullname)
>
> dependencies = pdata.get('dependencies', {})
> optionalDependencies = pdata.get('optionalDependencies', {})
> @@ -225,17 +223,12 @@ class Npm(FetchMethod):
> if obj == pkg:
> self._getshrinkeddependencies(obj,
> data['dependencies'][obj], data['dependencies'][obj]['version'], d,
> ud, lockdown, manifest, False) return
> - outputurl = "invalid"
> - if ('resolved' not in data) or (not
> data['resolved'].startswith('http://') and not
> data['resolved'].startswith('https://')):
> - # will be the case for ${PN}
> - fetchcmd = "npm view %s@%s dist.tarball --registry %s" %
> (pkg, version, ud.registry)
> - logger.debug(2, "Found this matching URL: %s" %
> str(fetchcmd))
> - outputurl = runfetchcmd(fetchcmd, d, True)
> - else:
> - outputurl = data['resolved']
> - self._runwget(ud, d, "%s --directory-prefix=%s %s" %
> (self.basecmd, ud.prefixdir, outputurl), False) +
> + pkgnameWithVersion = "{}@{}".format(pkg, version)
> + logger.debug(2, "Get dependencies for
> {}".format(pkgnameWithVersion))
> + filename = self._runpack(ud, d, pkgnameWithVersion)
> manifest[pkg] = {}
> - manifest[pkg]['tgz'] = os.path.basename(outputurl).rstrip()
> + manifest[pkg]['tgz'] = filename
> manifest[pkg]['deps'] = {}
>
> if pkg in lockdown:
> diff --git a/bitbake/lib/bb/fetch2/osc.py
> b/bitbake/lib/bb/fetch2/osc.py index 6c60456..3e56715 100644
> --- a/bitbake/lib/bb/fetch2/osc.py
> +++ b/bitbake/lib/bb/fetch2/osc.py
> @@ -1,5 +1,6 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> """
> Bitbake "Fetch" implementation for osc (Opensuse build service
> client). Based on the svn "Fetch" implementation.
> diff --git a/bitbake/lib/bb/fetch2/perforce.py
> b/bitbake/lib/bb/fetch2/perforce.py index 903a8e6..54d001e 100644
> --- a/bitbake/lib/bb/fetch2/perforce.py
> +++ b/bitbake/lib/bb/fetch2/perforce.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake 'Fetch' implementation for perforce
>
> @@ -8,18 +6,7 @@ BitBake 'Fetch' implementation for perforce
> # Copyright (C) 2003, 2004 Chris Larson
> # Copyright (C) 2016 Kodak Alaris, Inc.
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
> # Based on functions from the base bb module, Copyright 2003 Holger
> Schurig
> diff --git a/bitbake/lib/bb/fetch2/repo.py
> b/bitbake/lib/bb/fetch2/repo.py index 8c7e818..2bdbbd4 100644
> --- a/bitbake/lib/bb/fetch2/repo.py
> +++ b/bitbake/lib/bb/fetch2/repo.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake "Fetch" repo (git) implementation
>
> @@ -8,20 +6,10 @@ BitBake "Fetch" repo (git) implementation
> # Copyright (C) 2009 Tom Rini <trini@embeddedalley.com>
> #
> # Based on git.py which is:
> -#Copyright (C) 2005 Richard Purdie
> +# Copyright (C) 2005 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import os
> import bb
> diff --git a/bitbake/lib/bb/fetch2/s3.py b/bitbake/lib/bb/fetch2/s3.py
> index 1629288..ffca73c 100644
> --- a/bitbake/lib/bb/fetch2/s3.py
> +++ b/bitbake/lib/bb/fetch2/s3.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake 'Fetch' implementation for Amazon AWS S3.
>
> @@ -13,18 +11,7 @@ The aws tool must be correctly installed and
> configured prior to use. # Based in part on bb.fetch2.wget:
> # Copyright (C) 2003, 2004 Chris Larson
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
> # Based on functions from the base bb module, Copyright 2003 Holger
> Schurig
> diff --git a/bitbake/lib/bb/fetch2/sftp.py
> b/bitbake/lib/bb/fetch2/sftp.py index 81884a6..f87f292 100644
> --- a/bitbake/lib/bb/fetch2/sftp.py
> +++ b/bitbake/lib/bb/fetch2/sftp.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake SFTP Fetch implementation
>
> @@ -44,18 +42,7 @@ SRC_URI =
> "sftp://user@host.example.com/dir/path.file.txt" # Based in part on
> bb.fetch2.wget: # Copyright (C) 2003, 2004 Chris Larson
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
> # Based on functions from the base bb module, Copyright 2003 Holger
> Schurig
> diff --git a/bitbake/lib/bb/fetch2/ssh.py
> b/bitbake/lib/bb/fetch2/ssh.py index 6047ee4..f5be060 100644
> --- a/bitbake/lib/bb/fetch2/ssh.py
> +++ b/bitbake/lib/bb/fetch2/ssh.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> '''
> BitBake 'Fetch' implementations
>
> @@ -29,18 +27,8 @@ IETF secsh internet draft:
> # Copyright 2003 Holger Schurig
> #
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import re, os
> from bb.fetch2 import FetchMethod
> diff --git a/bitbake/lib/bb/fetch2/svn.py
> b/bitbake/lib/bb/fetch2/svn.py index ed70bcf..96d666b 100644
> --- a/bitbake/lib/bb/fetch2/svn.py
> +++ b/bitbake/lib/bb/fetch2/svn.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake 'Fetch' implementation for svn.
>
> @@ -8,18 +6,7 @@ BitBake 'Fetch' implementation for svn.
> # Copyright (C) 2003, 2004 Chris Larson
> # Copyright (C) 2004 Marcin Juszkiewicz
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
> # Based on functions from the base bb module, Copyright 2003 Holger
> Schurig
> @@ -63,6 +50,9 @@ class Svn(FetchMethod):
> relpath = self._strip_leading_slashes(ud.path)
> ud.pkgdir = os.path.join(svndir, ud.host, relpath)
> ud.moddir = os.path.join(ud.pkgdir, ud.module)
> + # Protects the repository from concurrent updates, e.g. from
> two
> + # recipes fetching different revisions at the same time
> + ud.svnlock = os.path.join(ud.pkgdir, "svn.lock")
>
> ud.setup_revisions(d)
>
> @@ -101,6 +91,13 @@ class Svn(FetchMethod):
> svncmd = "%s log --limit 1 %s %s://%s/%s/" %
> (ud.basecmd, " ".join(options), proto, svnroot, ud.module) else:
> suffix = ""
> +
> + # externals may be either 'allowed' or 'nowarn', but not
> both. Allowed
> + # will not issue a warning, but will log to the debug
> buffer what has likely
> + # been downloaded by SVN.
> + if not ("externals" in ud.parm and ud.parm["externals"]
> == "allowed"):
> + options.append("--ignore-externals")
> +
> if ud.revision:
> options.append("-r %s" % ud.revision)
> suffix = "@%s" % (ud.revision)
> @@ -123,35 +120,52 @@ class Svn(FetchMethod):
>
> logger.debug(2, "Fetch: checking for module directory '" +
> ud.moddir + "'")
> - if os.access(os.path.join(ud.moddir, '.svn'), os.R_OK):
> - svnupdatecmd = self._buildsvncommand(ud, d, "update")
> - logger.info("Update " + ud.url)
> - # We need to attempt to run svn upgrade first in case
> its an older working format
> - try:
> - runfetchcmd(ud.basecmd + " upgrade", d,
> workdir=ud.moddir)
> - except FetchError:
> - pass
> - logger.debug(1, "Running %s", svnupdatecmd)
> - bb.fetch2.check_network_access(d, svnupdatecmd, ud.url)
> - runfetchcmd(svnupdatecmd, d, workdir=ud.moddir)
> - else:
> - svnfetchcmd = self._buildsvncommand(ud, d, "fetch")
> - logger.info("Fetch " + ud.url)
> - # check out sources there
> - bb.utils.mkdirhier(ud.pkgdir)
> - logger.debug(1, "Running %s", svnfetchcmd)
> - bb.fetch2.check_network_access(d, svnfetchcmd, ud.url)
> - runfetchcmd(svnfetchcmd, d, workdir=ud.pkgdir)
> -
> - scmdata = ud.parm.get("scmdata", "")
> - if scmdata == "keep":
> - tar_flags = ""
> - else:
> - tar_flags = "--exclude='.svn'"
> + lf = bb.utils.lockfile(ud.svnlock)
> +
> + try:
> + if os.access(os.path.join(ud.moddir, '.svn'), os.R_OK):
> + svnupdatecmd = self._buildsvncommand(ud, d, "update")
> + logger.info("Update " + ud.url)
> + # We need to attempt to run svn upgrade first in
> case its an older working format
> + try:
> + runfetchcmd(ud.basecmd + " upgrade", d,
> workdir=ud.moddir)
> + except FetchError:
> + pass
> + logger.debug(1, "Running %s", svnupdatecmd)
> + bb.fetch2.check_network_access(d, svnupdatecmd,
> ud.url)
> + runfetchcmd(svnupdatecmd, d, workdir=ud.moddir)
> + else:
> + svnfetchcmd = self._buildsvncommand(ud, d, "fetch")
> + logger.info("Fetch " + ud.url)
> + # check out sources there
> + bb.utils.mkdirhier(ud.pkgdir)
> + logger.debug(1, "Running %s", svnfetchcmd)
> + bb.fetch2.check_network_access(d, svnfetchcmd,
> ud.url)
> + runfetchcmd(svnfetchcmd, d, workdir=ud.pkgdir)
> +
> + if not ("externals" in ud.parm and ud.parm["externals"]
> == "nowarn"):
> + # Warn the user if this had externals (won't catch
> them all)
> + output = runfetchcmd("svn propget svn:externals ||
> true", d, workdir=ud.moddir)
> + if output:
> + if "--ignore-externals" in svnfetchcmd.split():
> + bb.warn("%s contains svn:externals." %
> ud.url)
> + bb.warn("These should be added to the recipe
> SRC_URI as necessary.")
> + bb.warn("svn fetch has ignored
> externals:\n%s" % output)
> + bb.warn("To disable this warning add
> ';externals=nowarn' to the url.")
> + else:
> + bb.debug(1, "svn repository has
> externals:\n%s" % output) +
> + scmdata = ud.parm.get("scmdata", "")
> + if scmdata == "keep":
> + tar_flags = ""
> + else:
> + tar_flags = "--exclude='.svn'"
>
> - # tar them up to a defined filename
> - runfetchcmd("tar %s -czf %s %s" % (tar_flags, ud.localpath,
> ud.path_spec), d,
> - cleanup=[ud.localpath], workdir=ud.pkgdir)
> + # tar them up to a defined filename
> + runfetchcmd("tar %s -czf %s %s" % (tar_flags,
> ud.localpath, ud.path_spec), d,
> + cleanup=[ud.localpath], workdir=ud.pkgdir)
> + finally:
> + bb.utils.unlockfile(lf)
>
> def clean(self, ud, d):
> """ Clean SVN specific files and dirs """
> diff --git a/bitbake/lib/bb/fetch2/wget.py
> b/bitbake/lib/bb/fetch2/wget.py index 8f505b6..725586d 100644
> --- a/bitbake/lib/bb/fetch2/wget.py
> +++ b/bitbake/lib/bb/fetch2/wget.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake 'Fetch' implementations
>
> @@ -10,18 +8,7 @@ BitBake build tools.
>
> # Copyright (C) 2003, 2004 Chris Larson
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
> # Based on functions from the base bb module, Copyright 2003 Holger
> Schurig
> @@ -33,11 +20,14 @@ import logging
> import errno
> import bb
> import bb.progress
> +import socket
> +import http.client
> import urllib.request, urllib.parse, urllib.error
> from bb.fetch2 import FetchMethod
> from bb.fetch2 import FetchError
> from bb.fetch2 import logger
> from bb.fetch2 import runfetchcmd
> +from bb.fetch2 import FetchConnectionCache
> from bb.utils import export_proxies
> from bs4 import BeautifulSoup
> from bs4 import SoupStrainer
> @@ -132,10 +122,6 @@ class Wget(FetchMethod):
> return True
>
> def checkstatus(self, fetch, ud, d, try_again=True):
> - import urllib.request, urllib.error, urllib.parse, socket,
> http.client
> - from urllib.response import addinfourl
> - from bb.fetch2 import FetchConnectionCache
> -
> class HTTPConnectionCache(http.client.HTTPConnection):
> if fetch.connection_cache:
> def connect(self):
> @@ -168,7 +154,7 @@ class Wget(FetchMethod):
> """
> host = req.host
> if not host:
> - raise urlllib2.URLError('no host given')
> + raise urllib.error.URLError('no host given')
>
> h = http_class(host, timeout=req.timeout) # will
> parse host:port h.set_debuglevel(self._debuglevel)
> @@ -185,7 +171,7 @@ class Wget(FetchMethod):
> # request.
>
> # Don't close connection when connection_cache is
> enabled,
> - if fetch.connection_cache is None:
> + if fetch.connection_cache is None:
> headers["Connection"] = "close"
> else:
> headers["Connection"] = "Keep-Alive" # Works for
> HTTP/1.0 @@ -252,7 +238,7 @@ class Wget(FetchMethod):
> pass
> closed = False
>
> - resp = addinfourl(fp_dummy(), r.msg,
> req.get_full_url())
> + resp = urllib.response.addinfourl(fp_dummy(), r.msg,
> req.get_full_url()) resp.code = r.status
> resp.msg = r.reason
>
> @@ -271,17 +257,18 @@ class Wget(FetchMethod):
> fp.read()
> fp.close()
>
> - newheaders = dict((k,v) for k,v in
> list(req.headers.items())
> - if k.lower() not in
> ("content-length", "content-type"))
> - return
> self.parent.open(urllib.request.Request(req.get_full_url(),
> -
> headers=newheaders,
> -
> origin_req_host=req.origin_req_host,
> -
> unverifiable=True))
> + if req.get_method() != 'GET':
> + newheaders = dict((k, v) for k, v in
> list(req.headers.items())
> + if k.lower() not in
> ("content-length", "content-type"))
> + return
> self.parent.open(urllib.request.Request(req.get_full_url(),
> +
> headers=newheaders,
> +
> origin_req_host=req.origin_req_host,
> +
> unverifiable=True))
> - """
> - Some servers (e.g. GitHub archives, hosted on Amazon S3)
> return 403
> - Forbidden when they actually mean 405 Method Not Allowed.
> - """
> + raise urllib.request.HTTPError(req, code, msg,
> headers, None) +
> + # Some servers (e.g. GitHub archives, hosted on Amazon
> S3) return 403
> + # Forbidden when they actually mean 405 Method Not
> Allowed. http_error_403 = http_error_405
>
>
> @@ -292,15 +279,15 @@ class Wget(FetchMethod):
> """
> def redirect_request(self, req, fp, code, msg, headers,
> newurl): newreq =
> urllib.request.HTTPRedirectHandler.redirect_request(self, req, fp,
> code, msg, headers, newurl)
> - newreq.get_method = lambda: req.get_method()
> + newreq.get_method = req.get_method
> return newreq
> exported_proxies = export_proxies(d)
>
> handlers = [FixedHTTPRedirectHandler, HTTPMethodFallback]
> - if export_proxies:
> + if exported_proxies:
> handlers.append(urllib.request.ProxyHandler())
> handlers.append(CacheHTTPHandler())
> - # XXX: Since Python 2.7.9 ssl cert validation is enabled by
> default
> + # Since Python 2.7.9 ssl cert validation is enabled by
> default # see PEP-0476, this causes verification errors on some https
> servers # so disable by default.
> import ssl
> @@ -319,19 +306,19 @@ class Wget(FetchMethod):
> '''Adds Basic auth to http request, pass in
> login:password as string''' import base64
> encodeuser =
> base64.b64encode(login_str.encode('utf-8')).decode("utf-8")
> - authheader = "Basic %s" % encodeuser
> + authheader = "Basic %s" % encodeuser
> r.add_header("Authorization", authheader)
>
> - if ud.user:
> - add_basic_auth(ud.user, r)
> + if ud.user and ud.pswd:
> + add_basic_auth(ud.user + ':' + ud.pswd, r)
>
> try:
> - import netrc, urllib.parse
> + import netrc
> n = netrc.netrc()
> login, unused, password =
> n.authenticators(urllib.parse.urlparse(uri).hostname)
> add_basic_auth("%s:%s" % (login, password), r) except (TypeError,
> ImportError, IOError, netrc.NetrcParseError):
> - pass
> + pass
>
> with opener.open(r) as response:
> pass
> @@ -396,18 +383,14 @@ class Wget(FetchMethod):
> (oldpn, oldpv, oldsuffix) = old
> (newpn, newpv, newsuffix) = new
>
> - """
> - Check for a new suffix type that we have never heard of
> before
> - """
> - if (newsuffix):
> + # Check for a new suffix type that we have never heard of
> before
> + if newsuffix:
> m = self.suffix_regex_comp.search(newsuffix)
> if not m:
> bb.warn("%s has a possible unknown suffix: %s" %
> (newpn, newsuffix)) return False
>
> - """
> - Not our package so ignore it
> - """
> + # Not our package so ignore it
> if oldpn != newpn:
> return False
>
> @@ -473,15 +456,14 @@ class Wget(FetchMethod):
>
> return ""
>
> - def _check_latest_version_by_dir(self, dirver, package,
> package_regex,
> - current_version, ud, d):
> + def _check_latest_version_by_dir(self, dirver, package,
> package_regex, current_version, ud, d): """
> - Scan every directory in order to get upstream version.
> + Scan every directory in order to get upstream version.
> """
> version_dir = ['', '', '']
> version = ['', '', '']
>
> - dirver_regex =
> re.compile("(?P<pfx>\D*)(?P<ver>(\d+[\.\-_])+(\d+))")
> + dirver_regex =
> re.compile(r"(?P<pfx>\D*)(?P<ver>(\d+[\.\-_])+(\d+))") s =
> dirver_regex.search(dirver) if s:
> version_dir[1] = s.group('ver')
> @@ -541,26 +523,26 @@ class Wget(FetchMethod):
> gst-fluendo-mp3
> """
> # match most patterns which uses "-" as separator to version
> digits
> - pn_prefix1 = "[a-zA-Z][a-zA-Z0-9]*([-_][a-zA-Z]\w+)*\+?[-_]"
> + pn_prefix1 = r"[a-zA-Z][a-zA-Z0-9]*([-_][a-zA-Z]\w+)*\+?[-_]"
> # a loose pattern such as for unzip552.tar.gz
> - pn_prefix2 = "[a-zA-Z]+"
> + pn_prefix2 = r"[a-zA-Z]+"
> # a loose pattern such as for 80325-quicky-0.4.tar.gz
> - pn_prefix3 = "[0-9]+[-]?[a-zA-Z]+"
> + pn_prefix3 = r"[0-9]+[-]?[a-zA-Z]+"
> # Save the Package Name (pn) Regex for use later
> - pn_regex = "(%s|%s|%s)" % (pn_prefix1, pn_prefix2,
> pn_prefix3)
> + pn_regex = r"(%s|%s|%s)" % (pn_prefix1, pn_prefix2,
> pn_prefix3)
> # match version
> - pver_regex = "(([A-Z]*\d+[a-zA-Z]*[\.\-_]*)+)"
> + pver_regex = r"(([A-Z]*\d+[a-zA-Z]*[\.\-_]*)+)"
>
> # match arch
> parch_regex = "-source|_all_"
>
> # src.rpm extension was added only for rpm package. Can be
> removed if the rpm # packaged will always be considered as having to
> be manually upgraded
> - psuffix_regex =
> "(tar\.gz|tgz|tar\.bz2|zip|xz|tar\.lz|rpm|bz2|orig\.tar\.gz|tar\.xz|src\.tar\.gz|src\.tgz|svnr\d+\.tar\.bz2|stable\.tar\.gz|src\.rpm)"
> + psuffix_regex =
> r"(tar\.gz|tgz|tar\.bz2|zip|xz|tar\.lz|rpm|bz2|orig\.tar\.gz|tar\.xz|src\.tar\.gz|src\.tgz|svnr\d+\.tar\.bz2|stable\.tar\.gz|src\.rpm)"
> # match name, version and archive type of a package
> - package_regex_comp =
> re.compile("(?P<name>%s?\.?v?)(?P<pver>%s)(?P<arch>%s)?[\.-](?P<type>%s$)"
> + package_regex_comp =
> re.compile(r"(?P<name>%s?\.?v?)(?P<pver>%s)(?P<arch>%s)?[\.-](?P<type>%s$)"
> % (pn_regex, pver_regex, parch_regex, psuffix_regex))
> self.suffix_regex_comp = re.compile(psuffix_regex)
> @@ -572,7 +554,7 @@ class Wget(FetchMethod):
> version = self._parse_path(package_regex_comp, package)
> if version:
> package_custom_regex_comp = re.compile(
> -
> "(?P<name>%s)(?P<pver>%s)(?P<arch>%s)?[\.-](?P<type>%s)" %
> +
> r"(?P<name>%s)(?P<pver>%s)(?P<arch>%s)?[\.-](?P<type>%s)" %
> (re.escape(version[0]), pver_regex, parch_regex, psuffix_regex)) else:
> package_custom_regex_comp = None
> @@ -589,7 +571,7 @@ class Wget(FetchMethod):
> current_version = ['', d.getVar('PV'), '']
>
> """possible to have no version in pkg name, such as
> spectrum-fw"""
> - if not re.search("\d+", package):
> + if not re.search(r"\d+", package):
> current_version[1] = re.sub('_', '.', current_version[1])
> current_version[1] = re.sub('-', '.', current_version[1])
> return (current_version[1], '')
> @@ -607,13 +589,13 @@ class Wget(FetchMethod):
>
> # search for version matches on folders inside the path,
> like: # "5.7" in
> http://download.gnome.org/sources/${PN}/5.7/${PN}-${PV}.tar.gz
> - dirver_regex =
> re.compile("(?P<dirver>[^/]*(\d+\.)*\d+([-_]r\d+)*)/")
> + dirver_regex =
> re.compile(r"(?P<dirver>[^/]*(\d+\.)*\d+([-_]r\d+)*)/") m =
> dirver_regex.search(path) if m:
> pn = d.getVar('PN')
> dirver = m.group('dirver')
>
> - dirver_pn_regex = re.compile("%s\d?" %
> (re.escape(pn)))
> + dirver_pn_regex = re.compile(r"%s\d?" %
> (re.escape(pn))) if not dirver_pn_regex.search(dirver):
> return (self._check_latest_version_by_dir(dirver,
> package, package_regex, current_version, ud,
> d), '') diff --git a/bitbake/lib/bb/main.py b/bitbake/lib/bb/main.py
> index 732a315..af2880f 100755
> --- a/bitbake/lib/bb/main.py
> +++ b/bitbake/lib/bb/main.py
> @@ -1,6 +1,3 @@
> -#!/usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # Copyright (C) 2003, 2004 Chris Larson
> # Copyright (C) 2003, 2004 Phil Blundell
> @@ -9,18 +6,8 @@
> # Copyright (C) 2005 ROAD GmbH
> # Copyright (C) 2006 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import os
> import sys
> @@ -267,6 +254,11 @@ class
> BitBakeConfigParameters(cookerdata.ConfigParameters): help="Do not
> run any setscene tasks. sstate will be ignored and " "everything
> needed, built.")
> + parser.add_option("", "--skip-setscene", action="store_true",
> + dest="skipsetscene", default=False,
> + help="Skip setscene tasks if they would be
> executed. Tasks previously "
> + "restored from sstate will be kept,
> unlike --no-setscene") +
> parser.add_option("", "--setscene-only", action="store_true",
> dest="setsceneonly", default=False,
> help="Only run setscene tasks, don't run
> any real tasks.") @@ -448,7 +440,7 @@ def setup_bitbake(configParams,
> configuration, extrafeatures=None): else:
> logger.info("Reconnecting to bitbake server...")
> if not os.path.exists(sockname):
> - print("Previous bitbake instance shutting
> down?, waiting to retry...")
> + logger.info("Previous bitbake instance
> shutting down?, waiting to retry...") i = 0
> lock = None
> # Wait for 5s or until we can get the lock
> @@ -460,12 +452,7 @@ def setup_bitbake(configParams, configuration,
> extrafeatures=None): bb.utils.unlockfile(lock)
> raise
> bb.server.process.ProcessTimeout("Bitbake still shutting down as
> socket exists but no lock?") if not configParams.server_only:
> - try:
> - server_connection =
> bb.server.process.connectProcessServer(sockname, featureset)
> - except EOFError:
> - # The server may have been shutting down but
> not closed the socket yet. If that happened,
> - # ignore it.
> - pass
> + server_connection =
> bb.server.process.connectProcessServer(sockname, featureset)
> if server_connection or configParams.server_only:
> break
> @@ -475,12 +462,14 @@ def setup_bitbake(configParams, configuration,
> extrafeatures=None): if not retries:
> raise
> retries -= 1
> - if isinstance(e, (bb.server.process.ProcessTimeout,
> BrokenPipeError)):
> - logger.info("Retrying server connection...")
> + tryno = 8 - retries
> + if isinstance(e, (bb.server.process.ProcessTimeout,
> BrokenPipeError, EOFError)):
> + logger.info("Retrying server connection
> (#%d)..." % tryno) else:
> - logger.info("Retrying server connection... (%s)"
> % traceback.format_exc())
> + logger.info("Retrying server connection (#%d)...
> (%s)" % (tryno, traceback.format_exc())) if not retries:
> - bb.fatal("Unable to connect to bitbake server, or
> start one")
> + bb.fatal("Unable to connect to bitbake server, or
> start one (server startup failures would be in
> bitbake-cookerdaemon.log).")
> + bb.event.print_ui_queue()
> if retries < 5:
> time.sleep(5)
>
> @@ -502,7 +491,7 @@ def setup_bitbake(configParams, configuration,
> extrafeatures=None): def lockBitbake():
> topdir = bb.cookerdata.findTopdir()
> if not topdir:
> - bb.error("Unable to find conf/bblayers.conf or
> conf/bitbake.conf. BBAPTH is unset and/or not in a build directory?")
> + bb.error("Unable to find conf/bblayers.conf or
> conf/bitbake.conf. BBPATH is unset and/or not in a build directory?")
> raise BBMainFatal lockfile = topdir + "/bitbake.lock"
> return topdir, bb.utils.lockfile(lockfile, False, False)
> diff --git a/bitbake/lib/bb/methodpool.py
> b/bitbake/lib/bb/methodpool.py index 49aed33..51783ac 100644
> --- a/bitbake/lib/bb/methodpool.py
> +++ b/bitbake/lib/bb/methodpool.py
> @@ -1,21 +1,8 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> -#
> #
> # Copyright (C) 2006 Holger Hans Peter Freyther
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from bb.utils import better_compile, better_exec
>
> diff --git a/bitbake/lib/bb/monitordisk.py
> b/bitbake/lib/bb/monitordisk.py index 833cd3d..1a25b00 100644
> --- a/bitbake/lib/bb/monitordisk.py
> +++ b/bitbake/lib/bb/monitordisk.py
> @@ -1,21 +1,8 @@
> -#!/usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # Copyright (C) 2012 Robert Yang
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import os, logging, re, sys
> import bb
> @@ -28,16 +15,16 @@ def convertGMK(unit):
>
> """ Convert the space unit G, M, K, the unit is case-insensitive
> """
> - unitG = re.match('([1-9][0-9]*)[gG]\s?$', unit)
> + unitG = re.match(r'([1-9][0-9]*)[gG]\s?$', unit)
> if unitG:
> return int(unitG.group(1)) * (1024 ** 3)
> - unitM = re.match('([1-9][0-9]*)[mM]\s?$', unit)
> + unitM = re.match(r'([1-9][0-9]*)[mM]\s?$', unit)
> if unitM:
> return int(unitM.group(1)) * (1024 ** 2)
> - unitK = re.match('([1-9][0-9]*)[kK]\s?$', unit)
> + unitK = re.match(r'([1-9][0-9]*)[kK]\s?$', unit)
> if unitK:
> return int(unitK.group(1)) * 1024
> - unitN = re.match('([1-9][0-9]*)\s?$', unit)
> + unitN = re.match(r'([1-9][0-9]*)\s?$', unit)
> if unitN:
> return int(unitN.group(1))
> else:
> @@ -83,7 +70,7 @@ def getDiskData(BBDirs, configuration):
> for pathSpaceInode in BBDirs.split():
> # The input format is: "dir,space,inode", dir is a must,
> space # and inode are optional
> - pathSpaceInodeRe = re.match('([^,]*),([^,]*),([^,]*),?(.*)',
> pathSpaceInode)
> + pathSpaceInodeRe =
> re.match(r'([^,]*),([^,]*),([^,]*),?(.*)', pathSpaceInode) if not
> pathSpaceInodeRe: printErr("Invalid value in BB_DISKMON_DIRS: %s" %
> pathSpaceInode) return None
> @@ -147,7 +134,7 @@ def getInterval(configuration):
> else:
> # The disk space or inode interval is optional, but it should
> # have a correct value once it is specified
> - intervalRe = re.match('([^,]*),?\s*(.*)', interval)
> + intervalRe = re.match(r'([^,]*),?\s*(.*)', interval)
> if intervalRe:
> intervalSpace = intervalRe.group(1)
> if intervalSpace:
> diff --git a/bitbake/lib/bb/msg.py b/bitbake/lib/bb/msg.py
> index 96f077e..6216eb3 100644
> --- a/bitbake/lib/bb/msg.py
> +++ b/bitbake/lib/bb/msg.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake 'msg' implementation
>
> @@ -9,18 +7,8 @@ Message handling infrastructure for bitbake
>
> # Copyright (C) 2006 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import sys
> import copy
> diff --git a/bitbake/lib/bb/namedtuple_with_abc.py
> b/bitbake/lib/bb/namedtuple_with_abc.py index 32f2fc6..646aed6 100644
> --- a/bitbake/lib/bb/namedtuple_with_abc.py
> +++ b/bitbake/lib/bb/namedtuple_with_abc.py
> @@ -1,6 +1,8 @@
> #
> http://code.activestate.com/recipes/577629-namedtupleabc-abstract-base-class-mix-in-for-named/
> -#!/usr/bin/env python # Copyright (c) 2011 Jan Kaliszewski (zuo).
> Available under the MIT License. +#
> +# SPDX-License-Identifier: MIT
> +#
>
> """
> namedtuple_with_abc.py:
> diff --git a/bitbake/lib/bb/parse/__init__.py
> b/bitbake/lib/bb/parse/__init__.py index 5397d57..76e180b 100644
> --- a/bitbake/lib/bb/parse/__init__.py
> +++ b/bitbake/lib/bb/parse/__init__.py
> @@ -9,20 +9,10 @@ File parsers for the BitBake build tools.
> # Copyright (C) 2003, 2004 Chris Larson
> # Copyright (C) 2003, 2004 Phil Blundell
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
> # Based on functions from the base bb module, Copyright 2003 Holger
> Schurig +#
>
> handlers = []
>
> diff --git a/bitbake/lib/bb/parse/ast.py b/bitbake/lib/bb/parse/ast.py
> index 6d7c80b..f0911e6 100644
> --- a/bitbake/lib/bb/parse/ast.py
> +++ b/bitbake/lib/bb/parse/ast.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> AbstractSyntaxTree classes for the Bitbake language
> """
> @@ -8,19 +6,8 @@
> # Copyright (C) 2003, 2004 Phil Blundell
> # Copyright (C) 2009 Holger Hans Peter Freyther
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. -
>
> import re
> import string
> diff --git a/bitbake/lib/bb/parse/parse_py/BBHandler.py
> b/bitbake/lib/bb/parse/parse_py/BBHandler.py index 01fc47e..6f7cf82
> 100644 --- a/bitbake/lib/bb/parse/parse_py/BBHandler.py
> +++ b/bitbake/lib/bb/parse/parse_py/BBHandler.py
> @@ -1,6 +1,3 @@
> -#!/usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> class for handling .bb files
>
> @@ -12,19 +9,8 @@
> # Copyright (C) 2003, 2004 Chris Larson
> # Copyright (C) 2003, 2004 Phil Blundell
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. -
>
> import re, bb, os
> import logging
> @@ -38,14 +24,15 @@ from .ConfHandler import include, init
> # For compatibility
> bb.deprecate_import(__name__, "bb.parse", ["vars_from_file"])
>
> -__func_start_regexp__ =
> re.compile( r"(((?P<py>python)|(?P<fr>fakeroot))\s*)*(?P<func>[\w\.\-\+\{\}\$]+)?\s*\(\s*\)\s*{$" )
> -__inherit_regexp__ = re.compile( r"inherit\s+(.+)" )
> -__export_func_regexp__ = re.compile( r"EXPORT_FUNCTIONS\s+(.+)" )
> -__addtask_regexp__ =
> re.compile("addtask\s+(?P<func>\w+)\s*((before\s*(?P<before>((.*(?=after))|(.*))))|(after\s*(?P<after>((.*(?=before))|(.*)))))*")
> -__deltask_regexp__ = re.compile("deltask\s+(?P<func>\w+)")
> -__addhandler_regexp__ = re.compile( r"addhandler\s+(.+)" )
> -__def_regexp__ = re.compile( r"def\s+(\w+).*:" )
> -__python_func_regexp__ = re.compile( r"(\s+.*)|(^$)|(^#)" )
> +__func_start_regexp__ =
> re.compile(r"(((?P<py>python)|(?P<fr>fakeroot))\s*)*(?P<func>[\w\.\-\+\{\}\$]+)?\s*\(\s*\)\s*{$" )
> +__inherit_regexp__ = re.compile(r"inherit\s+(.+)" )
> +__export_func_regexp__ = re.compile(r"EXPORT_FUNCTIONS\s+(.+)" )
> +__addtask_regexp__ =
> re.compile(r"addtask\s+(?P<func>\w+)\s*((before\s*(?P<before>((.*(?=after))|(.*))))|(after\s*(?P<after>((.*(?=before))|(.*)))))*")
> +__deltask_regexp__ =
> re.compile(r"deltask\s+(?P<func>\w+)(?P<ignores>.*)")
> +__addhandler_regexp__ = re.compile(r"addhandler\s+(.+)" )
> +__def_regexp__ = re.compile(r"def\s+(\w+).*:" )
> +__python_func_regexp__ = re.compile(r"(\s+.*)|(^$)|(^#)" )
> +__python_tab_regexp__ = re.compile(r" *\t") __infunc__ = []
> __inpython__ = False @@ -160,6 +147,16 @@ def handle(fn, d, include):
> def feeder(lineno, s, fn, root, statements, eof=False): global
> __func_start_regexp__, __inherit_regexp__, __export_func_regexp__,
> __addtask_regexp__, __addhandler_regexp__, __def_regexp__,
> __python_func_regexp__, __inpython__, __infunc__, __body__, bb,
> __residue__, __classname__ +
> + # Check tabs in python functions:
> + # - def py_funcname(): covered by __inpython__
> + # - python(): covered by '__anonymous' == __infunc__[0]
> + # - python funcname(): covered by __infunc__[3]
> + if __inpython__ or (__infunc__ and ('__anonymous' ==
> __infunc__[0] or __infunc__[3])):
> + tab = __python_tab_regexp__.match(s)
> + if tab:
> + bb.warn('python should use 4 spaces indentation, but
> found tabs in %s, line %s' % (root, lineno)) +
> if __infunc__:
> if s == '}':
> __body__.append('')
> @@ -225,11 +222,27 @@ def feeder(lineno, s, fn, root, statements,
> eof=False):
> m = __addtask_regexp__.match(s)
> if m:
> + if len(m.group().split()) == 2:
> + # Check and warn for "addtask task1 task2"
> + m2 = re.match(r"addtask\s+(?P<func>\w+)(?P<ignores>.*)",
> s)
> + if m2 and m2.group('ignores'):
> + logger.warning('addtask ignored: "%s"' %
> m2.group('ignores')) +
> + # Check and warn for "addtask task1 before task2 before
> task3", the
> + # similar to "after"
> + taskexpression = s.split()
> + for word in ('before', 'after'):
> + if taskexpression.count(word) > 1:
> + logger.warning("addtask contained multiple '%s'
> keywords, only one is supported" % word) +
> ast.handleAddTask(statements, fn, lineno, m)
> return
>
> m = __deltask_regexp__.match(s)
> if m:
> + # Check and warn "for deltask task1 task2"
> + if m.group('ignores'):
> + logger.warning('deltask ignored: "%s"' %
> m.group('ignores')) ast.handleDelTask(statements, fn, lineno, m)
> return
>
> diff --git a/bitbake/lib/bb/parse/parse_py/ConfHandler.py
> b/bitbake/lib/bb/parse/parse_py/ConfHandler.py index 9d3ebe1..2e84b91
> 100644 --- a/bitbake/lib/bb/parse/parse_py/ConfHandler.py
> +++ b/bitbake/lib/bb/parse/parse_py/ConfHandler.py
> @@ -1,6 +1,3 @@
> -#!/usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> class for handling configuration data files
>
> @@ -11,18 +8,8 @@
> # Copyright (C) 2003, 2004 Chris Larson
> # Copyright (C) 2003, 2004 Phil Blundell
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import errno
> import re
> @@ -147,7 +134,7 @@ def handle(fn, data, include):
> continue
> s = s.rstrip()
> while s[-1] == '\\':
> - s2 = f.readline().strip()
> + s2 = f.readline().rstrip()
> lineno = lineno + 1
> if (not s2 or s2 and s2[0] != "#") and s[0] == "#" :
> bb.fatal("There is a confusing multiline, partially
> commented expression on line %s of file %s (%s).\nPlease clarify
> whether this is all a comment or should be parsed." % (lineno, fn,
> s)) diff --git a/bitbake/lib/bb/parse/parse_py/__init__.py
> b/bitbake/lib/bb/parse/parse_py/__init__.py index 3e658d0..f508afa
> 100644 --- a/bitbake/lib/bb/parse/parse_py/__init__.py +++
> b/bitbake/lib/bb/parse/parse_py/__init__.py @@ -1,6 +1,3 @@
> -#!/usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake Parsers
>
> @@ -11,20 +8,10 @@ File parsers for the BitBake build tools.
> # Copyright (C) 2003, 2004 Chris Larson
> # Copyright (C) 2003, 2004 Phil Blundell
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
> # Based on functions from the base bb module, Copyright 2003 Holger
> Schurig +#
>
> from __future__ import absolute_import
> from . import ConfHandler
> diff --git a/bitbake/lib/bb/persist_data.py
> b/bitbake/lib/bb/persist_data.py index bef7018..de8f87a 100644
> --- a/bitbake/lib/bb/persist_data.py
> +++ b/bitbake/lib/bb/persist_data.py
> @@ -8,18 +8,8 @@ currently, providing a key/value store accessed by
> 'domain'. # Copyright (C) 2007 Richard Purdie
> # Copyright (C) 2010 Chris Larson <chris_larson@mentor.com>
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import collections
> import logging
> @@ -29,6 +19,7 @@ import warnings
> from bb.compat import total_ordering
> from collections import Mapping
> import sqlite3
> +import contextlib
>
> sqlversion = sqlite3.sqlite_version_info
> if sqlversion[0] < 3 or (sqlversion[0] == 3 and sqlversion[1] < 3):
> @@ -36,84 +27,181 @@ if sqlversion[0] < 3 or (sqlversion[0] == 3 and
> sqlversion[1] < 3):
>
> logger = logging.getLogger("BitBake.PersistData")
> -if hasattr(sqlite3, 'enable_shared_cache'):
> - try:
> - sqlite3.enable_shared_cache(True)
> - except sqlite3.OperationalError:
> - pass
> -
>
> @total_ordering
> class SQLTable(collections.MutableMapping):
> + class _Decorators(object):
> + @staticmethod
> + def retry(*, reconnect=True):
> + """
> + Decorator that restarts a function if a database locked
> sqlite
> + exception occurs. If reconnect is True, the database
> connection
> + will be closed and reopened each time a failure occurs
> + """
> + def retry_wrapper(f):
> + def wrap_func(self, *args, **kwargs):
> + # Reconnect if necessary
> + if self.connection is None and reconnect:
> + self.reconnect()
> +
> + count = 0
> + while True:
> + try:
> + return f(self, *args, **kwargs)
> + except sqlite3.OperationalError as exc:
> + if count < 500 and ('is locked' in
> str(exc) or 'locking protocol' in str(exc)):
> + count = count + 1
> + if reconnect:
> + self.reconnect()
> + continue
> + raise
> + return wrap_func
> + return retry_wrapper
> +
> + @staticmethod
> + def transaction(f):
> + """
> + Decorator that starts a database transaction and creates
> a database
> + cursor for performing queries. If no exception is
> thrown, the
> + database results are commited. If an exception occurs,
> the database
> + is rolled back. In all cases, the cursor is closed after
> the
> + function ends.
> +
> + Note that the cursor is passed as an extra argument to
> the function
> + after `self` and before any of the normal arguments
> + """
> + def wrap_func(self, *args, **kwargs):
> + # Context manager will COMMIT the database on
> success,
> + # or ROLLBACK on an exception
> + with self.connection:
> + # Automatically close the cursor when done
> + with
> contextlib.closing(self.connection.cursor()) as cursor:
> + return f(self, cursor, *args, **kwargs)
> + return wrap_func
> +
> """Object representing a table/domain in the database"""
> def __init__(self, cachefile, table):
> self.cachefile = cachefile
> self.table = table
> - self.cursor = connect(self.cachefile)
> -
> - self._execute("CREATE TABLE IF NOT EXISTS %s(key TEXT, value
> TEXT);"
> - % table)
> -
> - def _execute(self, *query):
> - """Execute a query, waiting to acquire a lock if necessary"""
> - count = 0
> - while True:
> - try:
> - return self.cursor.execute(*query)
> - except sqlite3.OperationalError as exc:
> - if 'database is locked' in str(exc) and count < 500:
> - count = count + 1
> +
> + self.connection = None
> + self._execute_single("CREATE TABLE IF NOT EXISTS %s(key TEXT
> PRIMARY KEY NOT NULL, value TEXT);" % table) +
> + @_Decorators.retry(reconnect=False)
> + @_Decorators.transaction
> + def _setup_database(self, cursor):
> + cursor.execute("pragma synchronous = off;")
> + # Enable WAL and keep the autocheckpoint length small (the
> default is
> + # usually 1000). Persistent caches are usually read-mostly,
> so keeping
> + # this short will keep readers running quickly
> + cursor.execute("pragma journal_mode = WAL;")
> + cursor.execute("pragma wal_autocheckpoint = 100;")
> +
> + def reconnect(self):
> + if self.connection is not None:
> + self.connection.close()
> + self.connection = sqlite3.connect(self.cachefile, timeout=5)
> + self.connection.text_factory = str
> + self._setup_database()
> +
> + @_Decorators.retry()
> + @_Decorators.transaction
> + def _execute_single(self, cursor, *query):
> + """
> + Executes a single query and discards the results. This
> correctly closes
> + the database cursor when finished
> + """
> + cursor.execute(*query)
> +
> + @_Decorators.retry()
> + def _row_iter(self, f, *query):
> + """
> + Helper function that returns a row iterator. Each time
> __next__ is
> + called on the iterator, the provided function is evaluated
> to determine
> + the return value
> + """
> + class CursorIter(object):
> + def __init__(self, cursor):
> + self.cursor = cursor
> +
> + def __iter__(self):
> + return self
> +
> + def __next__(self):
> + row = self.cursor.fetchone()
> + if row is None:
> self.cursor.close()
> - self.cursor = connect(self.cachefile)
> - continue
> - raise
> + raise StopIteration
> + return f(row)
> +
> + def __enter__(self):
> + return self
> +
> + def __exit__(self, typ, value, traceback):
> + self.cursor.close()
> + return False
> +
> + cursor = self.connection.cursor()
> + try:
> + cursor.execute(*query)
> + return CursorIter(cursor)
> + except:
> + cursor.close()
>
> def __enter__(self):
> - self.cursor.__enter__()
> + self.connection.__enter__()
> return self
>
> def __exit__(self, *excinfo):
> - self.cursor.__exit__(*excinfo)
> -
> - def __getitem__(self, key):
> - data = self._execute("SELECT * from %s where key=?;" %
> - self.table, [key])
> - for row in data:
> + self.connection.__exit__(*excinfo)
> +
> + @_Decorators.retry()
> + @_Decorators.transaction
> + def __getitem__(self, cursor, key):
> + cursor.execute("SELECT * from %s where key=?;" % self.table,
> [key])
> + row = cursor.fetchone()
> + if row is not None:
> return row[1]
> raise KeyError(key)
>
> - def __delitem__(self, key):
> + @_Decorators.retry()
> + @_Decorators.transaction
> + def __delitem__(self, cursor, key):
> if key not in self:
> raise KeyError(key)
> - self._execute("DELETE from %s where key=?;" % self.table,
> [key])
> + cursor.execute("DELETE from %s where key=?;" % self.table,
> [key])
> - def __setitem__(self, key, value):
> + @_Decorators.retry()
> + @_Decorators.transaction
> + def __setitem__(self, cursor, key, value):
> if not isinstance(key, str):
> raise TypeError('Only string keys are supported')
> elif not isinstance(value, str):
> raise TypeError('Only string values are supported')
>
> - data = self._execute("SELECT * from %s where key=?;" %
> - self.table, [key])
> - exists = len(list(data))
> - if exists:
> - self._execute("UPDATE %s SET value=? WHERE key=?;" %
> self.table,
> - [value, key])
> + cursor.execute("SELECT * from %s where key=?;" % self.table,
> [key])
> + row = cursor.fetchone()
> + if row is not None:
> + cursor.execute("UPDATE %s SET value=? WHERE key=?;" %
> self.table, [value, key]) else:
> - self._execute("INSERT into %s(key, value) values
> (?, ?);" %
> - self.table, [key, value])
> -
> - def __contains__(self, key):
> - return key in set(self)
> -
> - def __len__(self):
> - data = self._execute("SELECT COUNT(key) FROM %s;" %
> self.table)
> - for row in data:
> + cursor.execute("INSERT into %s(key, value) values
> (?, ?);" % self.table, [key, value]) +
> + @_Decorators.retry()
> + @_Decorators.transaction
> + def __contains__(self, cursor, key):
> + cursor.execute('SELECT * from %s where key=?;' % self.table,
> [key])
> + return cursor.fetchone() is not None
> +
> + @_Decorators.retry()
> + @_Decorators.transaction
> + def __len__(self, cursor):
> + cursor.execute("SELECT COUNT(key) FROM %s;" % self.table)
> + row = cursor.fetchone()
> + if row is not None:
> return row[0]
>
> def __iter__(self):
> - data = self._execute("SELECT key FROM %s;" % self.table)
> - return (row[0] for row in data)
> + return self._row_iter(lambda row: row[0], "SELECT key from
> %s;" % self.table)
> def __lt__(self, other):
> if not isinstance(other, Mapping):
> @@ -122,25 +210,27 @@ class SQLTable(collections.MutableMapping):
> return len(self) < len(other)
>
> def get_by_pattern(self, pattern):
> - data = self._execute("SELECT * FROM %s WHERE key LIKE ?;" %
> - self.table, [pattern])
> - return [row[1] for row in data]
> + return self._row_iter(lambda row: row[1], "SELECT * FROM %s
> WHERE key LIKE ?;" %
> + self.table, [pattern])
>
> def values(self):
> return list(self.itervalues())
>
> def itervalues(self):
> - data = self._execute("SELECT value FROM %s;" % self.table)
> - return (row[0] for row in data)
> + return self._row_iter(lambda row: row[0], "SELECT value FROM
> %s;" %
> + self.table)
>
> def items(self):
> return list(self.iteritems())
>
> def iteritems(self):
> - return self._execute("SELECT * FROM %s;" % self.table)
> + return self._row_iter(lambda row: (row[0], row[1]), "SELECT
> * FROM %s;" %
> + self.table)
>
> - def clear(self):
> - self._execute("DELETE FROM %s;" % self.table)
> + @_Decorators.retry()
> + @_Decorators.transaction
> + def clear(self, cursor):
> + cursor.execute("DELETE FROM %s;" % self.table)
>
> def has_key(self, key):
> return key in self
> @@ -194,12 +284,6 @@ class PersistData(object):
> """
> del self.data[domain][key]
>
> -def connect(database):
> - connection = sqlite3.connect(database, timeout=5,
> isolation_level=None)
> - connection.execute("pragma synchronous = off;")
> - connection.text_factory = str
> - return connection
> -
> def persist(domain, d):
> """Convenience factory for SQLTable objects based upon
> metadata""" import bb.utils
> diff --git a/bitbake/lib/bb/process.py b/bitbake/lib/bb/process.py
> index e69697c..2dc472a 100644
> --- a/bitbake/lib/bb/process.py
> +++ b/bitbake/lib/bb/process.py
> @@ -1,3 +1,7 @@
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> import logging
> import signal
> import subprocess
> diff --git a/bitbake/lib/bb/progress.py b/bitbake/lib/bb/progress.py
> index f54d1c7..4022caa 100644
> --- a/bitbake/lib/bb/progress.py
> +++ b/bitbake/lib/bb/progress.py
> @@ -4,18 +4,8 @@ BitBake progress handling code
>
> # Copyright (C) 2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import sys
> import re
> @@ -23,6 +13,7 @@ import time
> import inspect
> import bb.event
> import bb.build
> +from bb.build import StdoutNoopContextManager
>
> class ProgressHandler(object):
> """
> @@ -37,7 +28,14 @@ class ProgressHandler(object):
> if outfile:
> self._outfile = outfile
> else:
> - self._outfile = sys.stdout
> + self._outfile = StdoutNoopContextManager()
> +
> + def __enter__(self):
> + self._outfile.__enter__()
> + return self
> +
> + def __exit__(self, *excinfo):
> + self._outfile.__exit__(*excinfo)
>
> def _fire_progress(self, taskprogress, rate=None):
> """Internal function to fire the progress event"""
> @@ -157,6 +155,12 @@ class MultiStageProgressReporter(object):
> self._stage_total = None
> self._callers = []
>
> + def __enter__(self):
> + return self
> +
> + def __exit__(self, *excinfo):
> + pass
> +
> def _fire_progress(self, taskprogress):
> bb.event.fire(bb.build.TaskProgress(taskprogress),
> self._data)
> diff --git a/bitbake/lib/bb/providers.py b/bitbake/lib/bb/providers.py
> index c2aa98c..f80963c 100644
> --- a/bitbake/lib/bb/providers.py
> +++ b/bitbake/lib/bb/providers.py
> @@ -1,5 +1,3 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # Copyright (C) 2003, 2004 Chris Larson
> # Copyright (C) 2003, 2004 Phil Blundell
> @@ -8,18 +6,8 @@
> # Copyright (C) 2005 ROAD GmbH
> # Copyright (C) 2006 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import re
> import logging
> @@ -129,7 +117,7 @@ def findPreferredProvider(pn, cfgData, dataCache,
> pkg_pn = None, item = None): preferred_v =
> cfgData.getVar("PREFERRED_VERSION")
> if preferred_v:
> - m = re.match('(\d+:)*(.*)(_.*)*', preferred_v)
> + m = re.match(r'(\d+:)*(.*)(_.*)*', preferred_v)
> if m:
> if m.group(1):
> preferred_e = m.group(1)[:-1]
> @@ -384,7 +372,7 @@ def getRuntimeProviders(dataCache, rdepend):
>
> # Only search dynamic packages if we can't find anything in
> other variables for pattern in dataCache.packages_dynamic:
> - pattern = pattern.replace('+', "\+")
> + pattern = pattern.replace(r'+', r"\+")
> if pattern in regexp_cache:
> regexp = regexp_cache[pattern]
> else:
> diff --git a/bitbake/lib/bb/pysh/builtin.py
> b/bitbake/lib/bb/pysh/builtin.py deleted file mode 100644
> index a8814dc..0000000
> --- a/bitbake/lib/bb/pysh/builtin.py
> +++ /dev/null
> @@ -1,710 +0,0 @@
> -# builtin.py - builtins and utilities definitions for pysh.
> -#
> -# Copyright 2007 Patrick Mezard
> -#
> -# This software may be used and distributed according to the terms
> -# of the GNU General Public License, incorporated herein by
> reference. -
> -"""Builtin and internal utilities implementations.
> -
> -- Beware not to use python interpreter environment as if it were the
> shell -environment. For instance, commands working directory must be
> explicitely handled -through env['PWD'] instead of relying on python
> working directory. -"""
> -import errno
> -import optparse
> -import os
> -import re
> -import subprocess
> -import sys
> -import time
> -
> -def has_subprocess_bug():
> - return getattr(subprocess, 'list2cmdline') and \
> - ( subprocess.list2cmdline(['']) == '' or \
> - subprocess.list2cmdline(['foo|bar']) == 'foo|bar')
> -
> -# Detect python bug 1634343: "subprocess swallows empty arguments
> under win32" -#
> <http://sourceforge.net/tracker/index.php?func=detail&aid=1634343&group_id=5470&atid=105470>
> -# Also detect: "[ 1710802 ] subprocess must escape redirection
> characters under win32" -#
> <http://sourceforge.net/tracker/index.php?func=detail&aid=1710802&group_id=5470&atid=105470>
> -if has_subprocess_bug():
> - import subprocess_fix
> - subprocess.list2cmdline = subprocess_fix.list2cmdline
> -
> -from sherrors import *
> -
> -class NonExitingParser(optparse.OptionParser):
> - """OptionParser default behaviour upon error is to print the
> error message and
> - exit. Raise a utility error instead.
> - """
> - def error(self, msg):
> - raise UtilityError(msg)
> -
> -#-------------------------------------------------------------------------------
> -# set special builtin
> -#-------------------------------------------------------------------------------
> -OPT_SET = NonExitingParser(usage="set - set or unset options and
> positional parameters") -OPT_SET.add_option( '-f',
> action='store_true', dest='has_f', default=False,
> - help='The shell shall disable pathname expansion.')
> -OPT_SET.add_option('-e', action='store_true', dest='has_e',
> default=False,
> - help="""When this option is on, if a simple command fails for
> any of the \
> - reasons listed in Consequences of Shell Errors or returns an
> exit status \
> - value >0, and is not part of the compound list following a
> while, until, \
> - or if keyword, and is not a part of an AND or OR list, and is
> not a \
> - pipeline preceded by the ! reserved word, then the shell shall
> immediately \
> - exit.""")
> -OPT_SET.add_option('-x', action='store_true', dest='has_x',
> default=False,
> - help="""The shell shall write to standard error a trace for each
> command \
> - after it expands the command and before it executes it. It is
> unspecified \
> - whether the command that turns tracing off is traced.""")
> -
> -def builtin_set(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n') -
> - option, args = OPT_SET.parse_args(args)
> - env = interp.get_env()
> -
> - if option.has_f:
> - env.set_opt('-f')
> - if option.has_e:
> - env.set_opt('-e')
> - if option.has_x:
> - env.set_opt('-x')
> - return 0
> -
> -#-------------------------------------------------------------------------------
> -# shift special builtin
> -#-------------------------------------------------------------------------------
> -def builtin_shift(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> -
> - params = interp.get_env().get_positional_args()
> - if args:
> - try:
> - n = int(args[0])
> - if n > len(params):
> - raise ValueError()
> - except ValueError:
> - return 1
> - else:
> - n = 1
> -
> - params[:n] = []
> - interp.get_env().set_positional_args(params)
> - return 0
> -
> -#-------------------------------------------------------------------------------
> -# export special builtin
> -#-------------------------------------------------------------------------------
> -OPT_EXPORT = NonExitingParser(usage="set - set or unset options and
> positional parameters") -OPT_EXPORT.add_option('-p',
> action='store_true', dest='has_p', default=False) -
> -def builtin_export(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> -
> - option, args = OPT_EXPORT.parse_args(args)
> - if option.has_p:
> - raise NotImplementedError()
> -
> - for arg in args:
> - try:
> - name, value = arg.split('=', 1)
> - except ValueError:
> - name, value = arg, None
> - env = interp.get_env().export(name, value)
> -
> - return 0
> -
> -#-------------------------------------------------------------------------------
> -# return special builtin
> -#-------------------------------------------------------------------------------
> -def builtin_return(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> - res = 0
> - if args:
> - try:
> - res = int(args[0])
> - except ValueError:
> - res = 0
> - if not 0<=res<=255:
> - res = 0
> -
> - # BUG: should be last executed command exit code
> - raise ReturnSignal(res)
> -
> -#-------------------------------------------------------------------------------
> -# trap special builtin
> -#-------------------------------------------------------------------------------
> -def builtin_trap(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> - if len(args) < 2:
> - stderr.write('trap: usage: trap [[arg] signal_spec ...]\n')
> - return 2
> -
> - action = args[0]
> - for sig in args[1:]:
> - try:
> - env.traps[sig] = action
> - except Exception as e:
> - stderr.write('trap: %s\n' % str(e))
> - return 0
> -
> -#-------------------------------------------------------------------------------
> -# unset special builtin
> -#-------------------------------------------------------------------------------
> -OPT_UNSET = NonExitingParser("unset - unset values and attributes of
> variables and functions") -OPT_UNSET.add_option( '-f',
> action='store_true', dest='has_f', default=False)
> -OPT_UNSET.add_option( '-v', action='store_true', dest='has_v',
> default=False) - -def builtin_unset(name, args, interp, env, stdin,
> stdout, stderr, debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> -
> - option, args = OPT_UNSET.parse_args(args)
> -
> - status = 0
> - env = interp.get_env()
> - for arg in args:
> - try:
> - if option.has_f:
> - env.remove_function(arg)
> - else:
> - del env[arg]
> - except KeyError:
> - pass
> - except VarAssignmentError:
> - status = 1
> -
> - return status
> -
> -#-------------------------------------------------------------------------------
> -# wait special builtin
> -#-------------------------------------------------------------------------------
> -def builtin_wait(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n') -
> - return interp.wait([int(arg) for arg in args])
> -
> -#-------------------------------------------------------------------------------
> -# cat utility
> -#-------------------------------------------------------------------------------
> -def utility_cat(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n') -
> - if not args:
> - args = ['-']
> -
> - status = 0
> - for arg in args:
> - if arg == '-':
> - data = stdin.read()
> - else:
> - path = os.path.join(env['PWD'], arg)
> - try:
> - f = file(path, 'rb')
> - try:
> - data = f.read()
> - finally:
> - f.close()
> - except IOError as e:
> - if e.errno != errno.ENOENT:
> - raise
> - status = 1
> - continue
> - stdout.write(data)
> - stdout.flush()
> - return status
> -
> -#-------------------------------------------------------------------------------
> -# cd utility
> -#-------------------------------------------------------------------------------
> -OPT_CD = NonExitingParser("cd - change the working directory")
> -
> -def utility_cd(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n') -
> - option, args = OPT_CD.parse_args(args)
> - env = interp.get_env()
> -
> - directory = None
> - printdir = False
> - if not args:
> - home = env.get('HOME')
> - if home:
> - # Unspecified, do nothing
> - return 0
> - else:
> - directory = home
> - elif len(args)==1:
> - directory = args[0]
> - if directory=='-':
> - if 'OLDPWD' not in env:
> - raise UtilityError("OLDPWD not set")
> - printdir = True
> - directory = env['OLDPWD']
> - else:
> - raise UtilityError("too many arguments")
> -
> - curpath = None
> - # Absolute directories will be handled correctly by the
> os.path.join call.
> - if not directory.startswith('.') and not
> directory.startswith('..'):
> - cdpaths = env.get('CDPATH', '.').split(';')
> - for cdpath in cdpaths:
> - p = os.path.join(cdpath, directory)
> - if os.path.isdir(p):
> - curpath = p
> - break
> -
> - if curpath is None:
> - curpath = directory
> - curpath = os.path.join(env['PWD'], directory)
> -
> - env['OLDPWD'] = env['PWD']
> - env['PWD'] = curpath
> - if printdir:
> - stdout.write('%s\n' % curpath)
> - return 0
> -
> -#-------------------------------------------------------------------------------
> -# colon utility
> -#-------------------------------------------------------------------------------
> -def utility_colon(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> - return 0
> -
> -#-------------------------------------------------------------------------------
> -# echo utility
> -#-------------------------------------------------------------------------------
> -def utility_echo(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> -
> - # Echo only takes arguments, no options. Use printf if you need
> fancy stuff.
> - output = ' '.join(args) + '\n'
> - stdout.write(output)
> - stdout.flush()
> - return 0
> -
> -#-------------------------------------------------------------------------------
> -# egrep utility
> -#-------------------------------------------------------------------------------
> -# egrep is usually a shell script.
> -# Unfortunately, pysh does not support shell scripts *with
> arguments* right now, -# so the redirection is implemented here,
> assuming grep is available. -def utility_egrep(name, args, interp,
> env, stdin, stdout, stderr, debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> -
> - return run_command('grep', ['-E'] + args, interp, env, stdin,
> stdout,
> - stderr, debugflags)
> -
> -#-------------------------------------------------------------------------------
> -# env utility
> -#-------------------------------------------------------------------------------
> -def utility_env(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> -
> - if args and args[0]=='-i':
> - raise NotImplementedError('env: -i option is not
> implemented')
> -
> - i = 0
> - for arg in args:
> - if '=' not in arg:
> - break
> - # Update the current environment
> - name, value = arg.split('=', 1)
> - env[name] = value
> - i += 1
> -
> - if args[i:]:
> - # Find then execute the specified interpreter
> - utility = env.find_in_path(args[i])
> - if not utility:
> - return 127
> - args[i:i+1] = utility
> - name = args[i]
> - args = args[i+1:]
> - try:
> - return run_command(name, args, interp, env, stdin,
> stdout, stderr,
> - debugflags)
> - except UtilityError:
> - stderr.write('env: failed to execute %s' % '
> '.join([name]+args))
> - return 126
> - else:
> - for pair in env.get_variables().iteritems():
> - stdout.write('%s=%s\n' % pair)
> - return 0
> -
> -#-------------------------------------------------------------------------------
> -# exit utility
> -#-------------------------------------------------------------------------------
> -def utility_exit(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> -
> - res = None
> - if args:
> - try:
> - res = int(args[0])
> - except ValueError:
> - res = None
> - if not 0<=res<=255:
> - res = None
> -
> - if res is None:
> - # BUG: should be last executed command exit code
> - res = 0
> -
> - raise ExitSignal(res)
> -
> -#-------------------------------------------------------------------------------
> -# fgrep utility
> -#-------------------------------------------------------------------------------
> -# see egrep
> -def utility_fgrep(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> -
> - return run_command('grep', ['-F'] + args, interp, env, stdin,
> stdout,
> - stderr, debugflags)
> -
> -#-------------------------------------------------------------------------------
> -# gunzip utility
> -#-------------------------------------------------------------------------------
> -# see egrep
> -def utility_gunzip(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> -
> - return run_command('gzip', ['-d'] + args, interp, env, stdin,
> stdout,
> - stderr, debugflags)
> -
> -#-------------------------------------------------------------------------------
> -# kill utility
> -#-------------------------------------------------------------------------------
> -def utility_kill(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> -
> - for arg in args:
> - pid = int(arg)
> - status = subprocess.call(['pskill', '/T', str(pid)],
> - shell=True,
> - stdout=subprocess.PIPE,
> - stderr=subprocess.PIPE)
> - # pskill is asynchronous, hence the stupid polling loop
> - while 1:
> - p = subprocess.Popen(['pslist', str(pid)],
> - shell=True,
> - stdout=subprocess.PIPE,
> - stderr=subprocess.STDOUT)
> - output = p.communicate()[0]
> - if ('process %d was not' % pid) in output:
> - break
> - time.sleep(1)
> - return status
> -
> -#-------------------------------------------------------------------------------
> -# mkdir utility
> -#-------------------------------------------------------------------------------
> -OPT_MKDIR = NonExitingParser("mkdir - make directories.")
> -OPT_MKDIR.add_option('-p', action='store_true', dest='has_p',
> default=False) -
> -def utility_mkdir(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> -
> - # TODO: implement umask
> - # TODO: implement proper utility error report
> - option, args = OPT_MKDIR.parse_args(args)
> - for arg in args:
> - path = os.path.join(env['PWD'], arg)
> - if option.has_p:
> - try:
> - os.makedirs(path)
> - except IOError as e:
> - if e.errno != errno.EEXIST:
> - raise
> - else:
> - os.mkdir(path)
> - return 0
> -
> -#-------------------------------------------------------------------------------
> -# netstat utility
> -#-------------------------------------------------------------------------------
> -def utility_netstat(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - # Do you really expect me to implement netstat ?
> - # This empty form is enough for Mercurial tests since it's
> - # supposed to generate nothing upon success. Faking this test
> - # is not a big deal either.
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> - return 0
> -
> -#-------------------------------------------------------------------------------
> -# pwd utility
> -#-------------------------------------------------------------------------------
> -OPT_PWD = NonExitingParser("pwd - return working directory name")
> -OPT_PWD.add_option('-L', action='store_true', dest='has_L',
> default=True,
> - help="""If the PWD environment variable contains an absolute
> pathname of \
> - the current directory that does not contain the filenames dot or
> dot-dot, \
> - pwd shall write this pathname to standard output. Otherwise, the
> -L option \
> - shall behave as the -P option.""")
> -OPT_PWD.add_option('-P', action='store_true', dest='has_L',
> default=False,
> - help="""The absolute pathname written shall not contain
> filenames that, in \
> - the context of the pathname, refer to files of type symbolic
> link.""") -
> -def utility_pwd(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n') -
> - option, args = OPT_PWD.parse_args(args)
> - stdout.write('%s\n' % env['PWD'])
> - return 0
> -
> -#-------------------------------------------------------------------------------
> -# printf utility
> -#-------------------------------------------------------------------------------
> -RE_UNESCAPE = re.compile(r'(\\x[a-zA-Z0-9]{2}|\\[0-7]{1,3}|\\.)')
> -
> -def utility_printf(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> -
> - def replace(m):
> - assert m.group()
> - g = m.group()[1:]
> - if g.startswith('x'):
> - return chr(int(g[1:], 16))
> - if len(g) <= 3 and len([c for c in g if c in '01234567']) ==
> len(g):
> - # Yay, an octal number
> - return chr(int(g, 8))
> - return {
> - 'a': '\a',
> - 'b': '\b',
> - 'f': '\f',
> - 'n': '\n',
> - 'r': '\r',
> - 't': '\t',
> - 'v': '\v',
> - '\\': '\\',
> - }.get(g)
> -
> - # Convert escape sequences
> - format = re.sub(RE_UNESCAPE, replace, args[0])
> - stdout.write(format % tuple(args[1:]))
> - return 0
> -
> -#-------------------------------------------------------------------------------
> -# true utility
> -#-------------------------------------------------------------------------------
> -def utility_true(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> - return 0
> -
> -#-------------------------------------------------------------------------------
> -# sed utility
> -#-------------------------------------------------------------------------------
> -RE_SED = re.compile(r'^s(.).*\1[a-zA-Z]*$')
> -
> -# cygwin sed fails with some expressions when they do not end with a
> single space. -# see unit tests for details. Interestingly, the same
> expressions works perfectly -# in cygwin shell.
> -def utility_sed(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> -
> - # Scan pattern arguments and append a space if necessary
> - for i in range(len(args)):
> - if not RE_SED.search(args[i]):
> - continue
> - args[i] = args[i] + ' '
> -
> - return run_command(name, args, interp, env, stdin, stdout,
> - stderr, debugflags)
> -
> -#-------------------------------------------------------------------------------
> -# sleep utility
> -#-------------------------------------------------------------------------------
> -def utility_sleep(name, args, interp, env, stdin, stdout, stderr,
> debugflags):
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> - time.sleep(int(args[0]))
> - return 0
> -
> -#-------------------------------------------------------------------------------
> -# sort utility
> -#-------------------------------------------------------------------------------
> -OPT_SORT = NonExitingParser("sort - sort, merge, or sequence check
> text files") -
> -def utility_sort(name, args, interp, env, stdin, stdout, stderr,
> debugflags): -
> - def sort(path):
> - if path == '-':
> - lines = stdin.readlines()
> - else:
> - try:
> - f = file(path)
> - try:
> - lines = f.readlines()
> - finally:
> - f.close()
> - except IOError as e:
> - stderr.write(str(e) + '\n')
> - return 1
> -
> - if lines and lines[-1][-1]!='\n':
> - lines[-1] = lines[-1] + '\n'
> - return lines
> -
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n')
> -
> - option, args = OPT_SORT.parse_args(args)
> - alllines = []
> -
> - if len(args)<=0:
> - args += ['-']
> -
> - # Load all files lines
> - curdir = os.getcwd()
> - try:
> - os.chdir(env['PWD'])
> - for path in args:
> - alllines += sort(path)
> - finally:
> - os.chdir(curdir)
> -
> - alllines.sort()
> - for line in alllines:
> - stdout.write(line)
> - return 0
> -
> -#-------------------------------------------------------------------------------
> -# hg utility
> -#-------------------------------------------------------------------------------
> -
> -hgcommands = [
> - 'add',
> - 'addremove',
> - 'commit', 'ci',
> - 'debugrename',
> - 'debugwalk',
> - 'falabala', # Dummy command used in a mercurial test
> - 'incoming',
> - 'locate',
> - 'pull',
> - 'push',
> - 'qinit',
> - 'remove', 'rm',
> - 'rename', 'mv',
> - 'revert',
> - 'showconfig',
> - 'status', 'st',
> - 'strip',
> - ]
> -
> -def rewriteslashes(name, args):
> - # Several hg commands output file paths, rewrite the separators
> - if len(args) > 1 and name.lower().endswith('python') \
> - and args[0].endswith('hg'):
> - for cmd in hgcommands:
> - if cmd in args[1:]:
> - return True
> -
> - # svn output contains many paths with OS specific separators.
> - # Normalize these to unix paths.
> - base = os.path.basename(name)
> - if base.startswith('svn'):
> - return True
> -
> - return False
> -
> -def rewritehg(output):
> - if not output:
> - return output
> - # Rewrite os specific messages
> - output = output.replace(': The system cannot find the file
> specified',
> - ': No such file or directory')
> - output = re.sub(': Access is denied.*$', ': Permission denied',
> output)
> - output = output.replace(': No connection could be made because
> the target machine actively refused it',
> - ': Connection refused')
> - return output
> -
> -
> -def run_command(name, args, interp, env, stdin, stdout,
> - stderr, debugflags):
> - # Execute the command
> - if 'debug-utility' in debugflags:
> - print interp.log(' '.join([name, str(args), interp['PWD']])
> + '\n') -
> - hgbin = interp.options().hgbinary
> - ishg = hgbin and ('hg' in name or args and 'hg' in args[0])
> - unixoutput = 'cygwin' in name or ishg
> -
> - exec_env = env.get_variables()
> - try:
> - # BUG: comparing file descriptor is clearly not a reliable
> way to tell
> - # whether they point on the same underlying object. But in
> pysh limited
> - # scope this is usually right, we do not expect complicated
> redirections
> - # besides usual 2>&1.
> - # Still there is one case we have but cannot deal with is
> when stdout
> - # and stderr are redirected *by pysh caller*. This the
> reason for the
> - # --redirect pysh() option.
> - # Now, we want to know they are the same because we
> sometimes need to
> - # transform the command output, mostly remove CR-LF to
> ensure that
> - # command output is unix-like. Cygwin utilies are a special
> case because
> - # they explicitely set their output streams to binary mode,
> so we have
> - # nothing to do. For all others commands, we have to guess
> whether they
> - # are sending text data, in which case the transformation
> must be done.
> - # Again, the NUL character test is unreliable but should be
> enough for
> - # hg tests.
> - redirected = stdout.fileno()==stderr.fileno()
> - if not redirected:
> - p = subprocess.Popen([name] + args, cwd=env['PWD'],
> env=exec_env,
> - stdin=stdin, stdout=subprocess.PIPE,
> stderr=subprocess.PIPE)
> - else:
> - p = subprocess.Popen([name] + args, cwd=env['PWD'],
> env=exec_env,
> - stdin=stdin, stdout=subprocess.PIPE,
> stderr=subprocess.STDOUT)
> - out, err = p.communicate()
> - except WindowsError as e:
> - raise UtilityError(str(e))
> -
> - if not unixoutput:
> - def encode(s):
> - if '\0' in s:
> - return s
> - return s.replace('\r\n', '\n')
> - else:
> - encode = lambda s: s
> -
> - if rewriteslashes(name, args):
> - encode1_ = encode
> - def encode(s):
> - s = encode1_(s)
> - s = s.replace('\\\\', '\\')
> - s = s.replace('\\', '/')
> - return s
> -
> - if ishg:
> - encode2_ = encode
> - def encode(s):
> - return rewritehg(encode2_(s))
> -
> - stdout.write(encode(out))
> - if not redirected:
> - stderr.write(encode(err))
> - return p.returncode
> -
> diff --git a/bitbake/lib/bb/pysh/interp.py
> b/bitbake/lib/bb/pysh/interp.py deleted file mode 100644
> index d14ecf3..0000000
> --- a/bitbake/lib/bb/pysh/interp.py
> +++ /dev/null
> @@ -1,1367 +0,0 @@
> -# interp.py - shell interpreter for pysh.
> -#
> -# Copyright 2007 Patrick Mezard
> -#
> -# This software may be used and distributed according to the terms
> -# of the GNU General Public License, incorporated herein by
> reference. -
> -"""Implement the shell interpreter.
> -
> -Most references are made to "The Open Group Base Specifications
> Issue 6".
> -<http://www.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html>
> -""" -# TODO: document the fact input streams must implement fileno()
> so Popen will work correctly. -# it requires non-stdin stream to be
> implemented as files. Still to be tested... -# DOC: pathsep is used
> in PATH instead of ':'. Clearly, there are path syntax issues here.
> -# TODO: stop command execution upon error. -# TODO: sort out the
> filename/io_number mess. It should be possible to use filenames only.
> -# TODO: review subshell implementation -# TODO: test environment
> cloning for non-special builtins -# TODO: set -x should not rebuild
> commands from tokens, assignments/redirections are lost -# TODO: unit
> test for variable assignment -# TODO: test error management wrt error
> type/utility type -# TODO: test for binary output everywhere
> -# BUG: debug-parsing does not pass log file to PLY. Maybe a PLY
> upgrade is necessary. -import base64
> -import cPickle as pickle
> -import errno
> -import glob
> -import os
> -import re
> -import subprocess
> -import sys
> -import tempfile
> -
> -try:
> - s = set()
> - del s
> -except NameError:
> - from Set import Set as set
> -
> -import builtin
> -from sherrors import *
> -import pyshlex
> -import pyshyacc
> -
> -def mappend(func, *args, **kargs):
> - """Like map but assume func returns a list. Returned lists are
> merged into
> - a single one.
> - """
> - return reduce(lambda a,b: a+b, map(func, *args, **kargs), [])
> -
> -class FileWrapper:
> - """File object wrapper to ease debugging.
> -
> - Allow mode checking and implement file duplication through a
> simple
> - reference counting scheme. Not sure the latter is really useful
> since
> - only real file descriptors can be used.
> - """
> - def __init__(self, mode, file, close=True):
> - if mode not in ('r', 'w', 'a'):
> - raise IOError('invalid mode: %s' % mode)
> - self._mode = mode
> - self._close = close
> - if isinstance(file, FileWrapper):
> - if file._refcount[0] <= 0:
> - raise IOError(0, 'Error')
> - self._refcount = file._refcount
> - self._refcount[0] += 1
> - self._file = file._file
> - else:
> - self._refcount = [1]
> - self._file = file
> -
> - def dup(self):
> - return FileWrapper(self._mode, self, self._close)
> -
> - def fileno(self):
> - """fileno() should be only necessary for input streams."""
> - return self._file.fileno()
> -
> - def read(self, size=-1):
> - if self._mode!='r':
> - raise IOError(0, 'Error')
> - return self._file.read(size)
> -
> - def readlines(self, *args, **kwargs):
> - return self._file.readlines(*args, **kwargs)
> -
> - def write(self, s):
> - if self._mode not in ('w', 'a'):
> - raise IOError(0, 'Error')
> - return self._file.write(s)
> -
> - def flush(self):
> - self._file.flush()
> -
> - def close(self):
> - if not self._refcount:
> - return
> - assert self._refcount[0] > 0
> -
> - self._refcount[0] -= 1
> - if self._refcount[0] == 0:
> - self._mode = 'c'
> - if self._close:
> - self._file.close()
> - self._refcount = None
> -
> - def mode(self):
> - return self._mode
> -
> - def __getattr__(self, name):
> - if name == 'name':
> - self.name = getattr(self._file, name)
> - return self.name
> - else:
> - raise AttributeError(name)
> -
> - def __del__(self):
> - self.close()
> -
> -
> -def win32_open_devnull(mode):
> - return open('NUL', mode)
> -
> -
> -class Redirections:
> - """Stores open files and their mapping to pseudo-sh file
> descriptor.
> - """
> - # BUG: redirections are not handled correctly: 1>&3 2>&3 3>&4
> does
> - # not make 1 to redirect to 4
> - def __init__(self, stdin=None, stdout=None, stderr=None):
> - self._descriptors = {}
> - if stdin is not None:
> - self._add_descriptor(0, stdin)
> - if stdout is not None:
> - self._add_descriptor(1, stdout)
> - if stderr is not None:
> - self._add_descriptor(2, stderr)
> -
> - def add_here_document(self, interp, name, content,
> io_number=None):
> - if io_number is None:
> - io_number = 0
> -
> - if name==pyshlex.unquote_wordtree(name):
> - content = interp.expand_here_document(('TOKEN', content))
> -
> - # Write document content in a temporary file
> - tmp = tempfile.TemporaryFile()
> - try:
> - tmp.write(content)
> - tmp.flush()
> - tmp.seek(0)
> - self._add_descriptor(io_number, FileWrapper('r', tmp))
> - except:
> - tmp.close()
> - raise
> -
> - def add(self, interp, op, filename, io_number=None):
> - if op not in ('<', '>', '>|', '>>', '>&'):
> - # TODO: add descriptor duplication and
> here_documents
> - raise RedirectionError('Unsupported redirection operator
> "%s"' % op)
> -
> - if io_number is not None:
> - io_number = int(io_number)
> -
> - if (op == '>&' and filename.isdigit()) or filename=='-':
> - # No expansion for file descriptors, quote them if you
> want a filename
> - fullname = filename
> - else:
> - if filename.startswith('/'):
> - # TODO: win32 kludge
> - if filename=='/dev/null':
> - fullname = 'NUL'
> - else:
> - # TODO: handle absolute pathnames, they are
> unlikely to exist on the
> - # current platform (win32 for instance).
> - raise NotImplementedError()
> - else:
> - fullname = interp.expand_redirection(('TOKEN',
> filename))
> - if not fullname:
> - raise RedirectionError('%s: ambiguous redirect'
> % filename)
> - # Build absolute path based on PWD
> - fullname = os.path.join(interp.get_env()['PWD'],
> fullname)
> -
> - if op=='<':
> - return self._add_input_redirection(interp, fullname,
> io_number)
> - elif op in ('>', '>|'):
> - clobber = ('>|'==op)
> - return self._add_output_redirection(interp, fullname,
> io_number, clobber)
> - elif op=='>>':
> - return self._add_output_appending(interp, fullname,
> io_number)
> - elif op=='>&':
> - return self._dup_output_descriptor(fullname, io_number)
> -
> - def close(self):
> - if self._descriptors is not None:
> - for desc in self._descriptors.itervalues():
> - desc.flush()
> - desc.close()
> - self._descriptors = None
> -
> - def stdin(self):
> - return self._descriptors[0]
> -
> - def stdout(self):
> - return self._descriptors[1]
> -
> - def stderr(self):
> - return self._descriptors[2]
> -
> - def clone(self):
> - clone = Redirections()
> - for desc, fileobj in self._descriptors.iteritems():
> - clone._descriptors[desc] = fileobj.dup()
> - return clone
> -
> - def _add_output_redirection(self, interp, filename, io_number,
> clobber):
> - if io_number is None:
> - # io_number default to standard output
> - io_number = 1
> -
> - if not clobber and interp.get_env().has_opt('-C') and
> os.path.isfile(filename):
> - # File already exist in no-clobber mode, bail out
> - raise RedirectionError('File "%s" already exists' %
> filename)
> -
> - # Open and register
> - self._add_file_descriptor(io_number, filename, 'w')
> -
> - def _add_output_appending(self, interp, filename, io_number):
> - if io_number is None:
> - io_number = 1
> - self._add_file_descriptor(io_number, filename, 'a')
> -
> - def _add_input_redirection(self, interp, filename, io_number):
> - if io_number is None:
> - io_number = 0
> - self._add_file_descriptor(io_number, filename, 'r')
> -
> - def _add_file_descriptor(self, io_number, filename, mode):
> - try:
> - if filename.startswith('/'):
> - if filename=='/dev/null':
> - f = win32_open_devnull(mode+'b')
> - else:
> - # TODO: handle absolute pathnames, they are
> unlikely to exist on the
> - # current platform (win32 for instance).
> - raise NotImplementedError('cannot open absolute
> path %s' % repr(filename))
> - else:
> - f = file(filename, mode+'b')
> - except IOError as e:
> - raise RedirectionError(str(e))
> -
> - wrapper = None
> - try:
> - wrapper = FileWrapper(mode, f)
> - f = None
> - self._add_descriptor(io_number, wrapper)
> - except:
> - if f: f.close()
> - if wrapper: wrapper.close()
> - raise
> -
> - def _dup_output_descriptor(self, source_fd, dest_fd):
> - if source_fd is None:
> - source_fd = 1
> - self._dup_file_descriptor(source_fd, dest_fd, 'w')
> -
> - def _dup_file_descriptor(self, source_fd, dest_fd, mode):
> - source_fd = int(source_fd)
> - if source_fd not in self._descriptors:
> - raise RedirectionError('"%s" is not a valid file
> descriptor' % str(source_fd))
> - source = self._descriptors[source_fd]
> -
> - if source.mode()!=mode:
> - raise RedirectionError('Descriptor %s cannot be
> duplicated in mode "%s"' % (str(source), mode))
> -
> - if dest_fd=='-':
> - # Close the source descriptor
> - del self._descriptors[source_fd]
> - source.close()
> - else:
> - dest_fd = int(dest_fd)
> - if dest_fd not in self._descriptors:
> - raise RedirectionError('Cannot replace file
> descriptor %s' % str(dest_fd))
> -
> - dest = self._descriptors[dest_fd]
> - if dest.mode()!=mode:
> - raise RedirectionError('Descriptor %s cannot be
> cannot be redirected in mode "%s"' % (str(dest), mode))
> -
> - self._descriptors[dest_fd] = source.dup()
> - dest.close()
> -
> - def _add_descriptor(self, io_number, file):
> - io_number = int(io_number)
> -
> - if io_number in self._descriptors:
> - # Close the current descriptor
> - d = self._descriptors[io_number]
> - del self._descriptors[io_number]
> - d.close()
> -
> - self._descriptors[io_number] = file
> -
> - def __str__(self):
> - names = [('%d=%r' % (k, getattr(v, 'name', None))) for k,v
> - in self._descriptors.iteritems()]
> - names = ','.join(names)
> - return 'Redirections(%s)' % names
> -
> - def __del__(self):
> - self.close()
> -
> -def cygwin_to_windows_path(path):
> - """Turn /cygdrive/c/foo into c:/foo, or return path if it
> - is not a cygwin path.
> - """
> - if not path.startswith('/cygdrive/'):
> - return path
> - path = path[len('/cygdrive/'):]
> - path = path[:1] + ':' + path[1:]
> - return path
> -
> -def win32_to_unix_path(path):
> - if path is not None:
> - path = path.replace('\\', '/')
> - return path
> -
> -_RE_SHEBANG = re.compile(r'^\#!\s?([^\s]+)(?:\s([^\s]+))?')
> -_SHEBANG_CMDS = {
> - '/usr/bin/env': 'env',
> - '/bin/sh': 'pysh',
> - 'python': 'python',
> -}
> -
> -def resolve_shebang(path, ignoreshell=False):
> - """Return a list of arguments as shebang interpreter call or an
> empty list
> - if path does not refer to an executable script.
> - See <http://www.opengroup.org/austin/docs/austin_51r2.txt>.
> -
> - ignoreshell - set to True to ignore sh shebangs. Return an empty
> list instead.
> - """
> - try:
> - f = file(path)
> - try:
> - # At most 80 characters in the first line
> - header = f.read(80).splitlines()[0]
> - finally:
> - f.close()
> -
> - m = _RE_SHEBANG.search(header)
> - if not m:
> - return []
> - cmd, arg = m.group(1,2)
> - if os.path.isfile(cmd):
> - # Keep this one, the hg script for instance contains a
> weird windows
> - # shebang referencing the current python install.
> - cmdfile = os.path.basename(cmd).lower()
> - if cmdfile == 'python.exe':
> - cmd = 'python'
> - pass
> - elif cmd not in _SHEBANG_CMDS:
> - raise CommandNotFound('Unknown interpreter "%s"
> referenced in '\
> - 'shebang' % header)
> - cmd = _SHEBANG_CMDS.get(cmd)
> - if cmd is None or (ignoreshell and cmd == 'pysh'):
> - return []
> - if arg is None:
> - return [cmd, win32_to_unix_path(path)]
> - return [cmd, arg, win32_to_unix_path(path)]
> - except IOError as e:
> - if e.errno!=errno.ENOENT and \
> - (e.errno!=errno.EPERM and not os.path.isdir(path)): #
> Opening a directory raises EPERM
> - raise
> - return []
> -
> -def win32_find_in_path(name, path):
> - if isinstance(path, str):
> - path = path.split(os.pathsep)
> -
> - exts = os.environ.get('PATHEXT', '').lower().split(os.pathsep)
> - for p in path:
> - p_name = os.path.join(p, name)
> -
> - prefix = resolve_shebang(p_name)
> - if prefix:
> - return prefix
> -
> - for ext in exts:
> - p_name_ext = p_name + ext
> - if os.path.exists(p_name_ext):
> - return [win32_to_unix_path(p_name_ext)]
> - return []
> -
> -class Traps(dict):
> - def __setitem__(self, key, value):
> - if key not in ('EXIT',):
> - raise NotImplementedError()
> - super(Traps, self).__setitem__(key, value)
> -
> -# IFS white spaces character class
> -_IFS_WHITESPACES = (' ', '\t', '\n')
> -
> -class Environment:
> - """Environment holds environment variables, export table,
> function
> - definitions and whatever is defined in 2.12 "Shell Execution
> Environment",
> - redirection excepted.
> - """
> - def __init__(self, pwd):
> - self._opt = set() #Shell options
> -
> - self._functions = {}
> - self._env = {'?': '0', '#': '0'}
> - self._exported = set([
> - 'HOME', 'IFS', 'PATH'
> - ])
> -
> - # Set environment vars with side-effects
> - self._ifs_ws = None # Set of IFS whitespace characters
> - self._ifs_re = None # Regular expression used to split
> between words using IFS classes
> - self['IFS'] = ''.join(_IFS_WHITESPACES) #Default environment
> values
> - self['PWD'] = pwd
> - self.traps = Traps()
> -
> - def clone(self, subshell=False):
> - env = Environment(self['PWD'])
> - env._opt = set(self._opt)
> - for k,v in self.get_variables().iteritems():
> - if k in self._exported:
> - env.export(k,v)
> - elif subshell:
> - env[k] = v
> -
> - if subshell:
> - env._functions = dict(self._functions)
> -
> - return env
> -
> - def __getitem__(self, key):
> - if key in ('@', '*', '-', '$'):
> - raise NotImplementedError('%s is not implemented' %
> repr(key))
> - return self._env[key]
> -
> - def get(self, key, defval=None):
> - try:
> - return self[key]
> - except KeyError:
> - return defval
> -
> - def __setitem__(self, key, value):
> - if key=='IFS':
> - # Update the whitespace/non-whitespace classes
> - self._update_ifs(value)
> - elif key=='PWD':
> - pwd = os.path.abspath(value)
> - if not os.path.isdir(pwd):
> - raise VarAssignmentError('Invalid directory %s' %
> value)
> - value = pwd
> - elif key in ('?', '!'):
> - value = str(int(value))
> - self._env[key] = value
> -
> - def __delitem__(self, key):
> - if key in ('IFS', 'PWD', '?'):
> - raise VarAssignmentError('%s cannot be unset' % key)
> - del self._env[key]
> -
> - def __contains__(self, item):
> - return item in self._env
> -
> - def set_positional_args(self, args):
> - """Set the content of 'args' as positional argument from 1
> to len(args).
> - Return previous argument as a list of strings.
> - """
> - # Save and remove previous arguments
> - prevargs = []
> - for i in range(int(self._env['#'])):
> - i = str(i+1)
> - prevargs.append(self._env[i])
> - del self._env[i]
> - self._env['#'] = '0'
> -
> - #Set new ones
> - for i,arg in enumerate(args):
> - self._env[str(i+1)] = str(arg)
> - self._env['#'] = str(len(args))
> -
> - return prevargs
> -
> - def get_positional_args(self):
> - return [self._env[str(i+1)] for i in
> range(int(self._env['#']))]
> -
> - def get_variables(self):
> - return dict(self._env)
> -
> - def export(self, key, value=None):
> - if value is not None:
> - self[key] = value
> - self._exported.add(key)
> -
> - def get_exported(self):
> - return [(k,self._env.get(k)) for k in self._exported]
> -
> - def split_fields(self, word):
> - if not self._ifs_ws or not word:
> - return [word]
> - return re.split(self._ifs_re, word)
> -
> - def _update_ifs(self, value):
> - """Update the split_fields related variables when IFS
> character set is
> - changed.
> - """
> - # TODO: handle NULL IFS
> -
> - # Separate characters in whitespace and non-whitespace
> - chars = set(value)
> - ws = [c for c in chars if c in _IFS_WHITESPACES]
> - nws = [c for c in chars if c not in _IFS_WHITESPACES]
> -
> - # Keep whitespaces in a string for left and right stripping
> - self._ifs_ws = ''.join(ws)
> -
> - # Build a regexp to split fields
> - trailing = '[' + ''.join([re.escape(c) for c in ws]) + ']'
> - if nws:
> - # First, the single non-whitespace occurence.
> - nws = '[' + ''.join([re.escape(c) for c in nws]) + ']'
> - nws = '(?:' + trailing + '*' + nws + trailing + '*' +
> '|' + trailing + '+)'
> - else:
> - # Then mix all parts with quantifiers
> - nws = trailing + '+'
> - self._ifs_re = re.compile(nws)
> -
> - def has_opt(self, opt, val=None):
> - return (opt, val) in self._opt
> -
> - def set_opt(self, opt, val=None):
> - self._opt.add((opt, val))
> -
> - def find_in_path(self, name, pwd=False):
> - path = self._env.get('PATH', '').split(os.pathsep)
> - if pwd:
> - path[:0] = [self['PWD']]
> - if os.name == 'nt':
> - return win32_find_in_path(name, self._env.get('PATH',
> ''))
> - else:
> - raise NotImplementedError()
> -
> - def define_function(self, name, body):
> - if not is_name(name):
> - raise ShellSyntaxError('%s is not a valid function name'
> % repr(name))
> - self._functions[name] = body
> -
> - def remove_function(self, name):
> - del self._functions[name]
> -
> - def is_function(self, name):
> - return name in self._functions
> -
> - def get_function(self, name):
> - return self._functions.get(name)
> -
> -
> -name_charset =
> 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_'
> -name_charset = dict(zip(name_charset,name_charset))
> -
> -def match_name(s):
> - """Return the length in characters of the longest prefix made of
> name
> - allowed characters in s.
> - """
> - for i,c in enumerate(s):
> - if c not in name_charset:
> - return s[:i]
> - return s
> -
> -def is_name(s):
> - return len([c for c in s if c not in name_charset])<=0
> -
> -def is_special_param(c):
> - return len(c)==1 and c in ('@','*','#','?','-','$','!','0')
> -
> -def utility_not_implemented(name, *args, **kwargs):
> - raise NotImplementedError('%s utility is not implemented' % name)
> -
> -
> -class Utility:
> - """Define utilities properties:
> - func -- utility callable. See builtin module for utility samples.
> - is_special -- see XCU 2.8.
> - """
> - def __init__(self, func, is_special=0):
> - self.func = func
> - self.is_special = bool(is_special)
> -
> -
> -def encodeargs(args):
> - def encodearg(s):
> - lines = base64.encodestring(s)
> - lines = [l.splitlines()[0] for l in lines]
> - return ''.join(lines)
> -
> - s = pickle.dumps(args)
> - return encodearg(s)
> -
> -def decodeargs(s):
> - s = base64.decodestring(s)
> - return pickle.loads(s)
> -
> -
> -class GlobError(Exception):
> - pass
> -
> -class Options:
> - def __init__(self):
> - # True if Mercurial operates with binary streams
> - self.hgbinary = True
> -
> -class Interpreter:
> - # Implementation is very basic: the execute() method just makes
> a DFS on the
> - # AST and execute nodes one by one. Nodes are tuple (name,obj)
> where name
> - # is a string identifier and obj the AST element returned by the
> parser.
> - #
> - # Handler are named after the node identifiers.
> - # TODO: check node names and remove the switch in execute with
> some
> - # dynamic getattr() call to find node handlers.
> - """Shell interpreter.
> -
> - The following debugging flags can be passed:
> - debug-parsing - enable PLY debugging.
> - debug-tree - print the generated AST.
> - debug-cmd - trace command execution before word expansion, plus
> exit status.
> - debug-utility - trace utility execution.
> - """
> -
> - # List supported commands.
> - COMMANDS = {
> - 'cat': Utility(builtin.utility_cat,),
> - 'cd': Utility(builtin.utility_cd,),
> - ':': Utility(builtin.utility_colon,),
> - 'echo': Utility(builtin.utility_echo),
> - 'env': Utility(builtin.utility_env),
> - 'exit': Utility(builtin.utility_exit),
> - 'export': Utility(builtin.builtin_export,
> is_special=1),
> - 'egrep': Utility(builtin.utility_egrep),
> - 'fgrep': Utility(builtin.utility_fgrep),
> - 'gunzip': Utility(builtin.utility_gunzip),
> - 'kill': Utility(builtin.utility_kill),
> - 'mkdir': Utility(builtin.utility_mkdir),
> - 'netstat': Utility(builtin.utility_netstat),
> - 'printf': Utility(builtin.utility_printf),
> - 'pwd': Utility(builtin.utility_pwd),
> - 'return': Utility(builtin.builtin_return,
> is_special=1),
> - 'sed': Utility(builtin.utility_sed,),
> - 'set': Utility(builtin.builtin_set,),
> - 'shift': Utility(builtin.builtin_shift,),
> - 'sleep': Utility(builtin.utility_sleep,),
> - 'sort': Utility(builtin.utility_sort,),
> - 'trap': Utility(builtin.builtin_trap,
> is_special=1),
> - 'true': Utility(builtin.utility_true),
> - 'unset': Utility(builtin.builtin_unset,
> is_special=1),
> - 'wait': Utility(builtin.builtin_wait,
> is_special=1),
> - }
> -
> - def __init__(self, pwd, debugflags = [], env=None, redirs=None,
> stdin=None,
> - stdout=None, stderr=None, opts=Options()):
> - self._env = env
> - if self._env is None:
> - self._env = Environment(pwd)
> - self._children = {}
> -
> - self._redirs = redirs
> - self._close_redirs = False
> -
> - if self._redirs is None:
> - if stdin is None:
> - stdin = sys.stdin
> - if stdout is None:
> - stdout = sys.stdout
> - if stderr is None:
> - stderr = sys.stderr
> - stdin = FileWrapper('r', stdin, False)
> - stdout = FileWrapper('w', stdout, False)
> - stderr = FileWrapper('w', stderr, False)
> - self._redirs = Redirections(stdin, stdout, stderr)
> - self._close_redirs = True
> -
> - self._debugflags = list(debugflags)
> - self._logfile = sys.stderr
> - self._options = opts
> -
> - def close(self):
> - """Must be called when the interpreter is no longer used."""
> - script = self._env.traps.get('EXIT')
> - if script:
> - try:
> - self.execute_script(script=script)
> - except:
> - pass
> -
> - if self._redirs is not None and self._close_redirs:
> - self._redirs.close()
> - self._redirs = None
> -
> - def log(self, s):
> - self._logfile.write(s)
> - self._logfile.flush()
> -
> - def __getitem__(self, key):
> - return self._env[key]
> -
> - def __setitem__(self, key, value):
> - self._env[key] = value
> -
> - def options(self):
> - return self._options
> -
> - def redirect(self, redirs, ios):
> - def add_redir(io):
> - if isinstance(io, pyshyacc.IORedirect):
> - redirs.add(self, io.op, io.filename, io.io_number)
> - else:
> - redirs.add_here_document(self, io.name, io.content,
> io.io_number)
> -
> - map(add_redir, ios)
> - return redirs
> -
> - def execute_script(self, script=None, ast=None, sourced=False,
> - scriptpath=None):
> - """If script is not None, parse the input. Otherwise takes
> the supplied
> - AST. Then execute the AST.
> - Return the script exit status.
> - """
> - try:
> - if scriptpath is not None:
> - self._env['0'] = os.path.abspath(scriptpath)
> -
> - if script is not None:
> - debug_parsing = ('debug-parsing' in
> self._debugflags)
> - cmds, script = pyshyacc.parse(script, True,
> debug_parsing)
> - if 'debug-tree' in self._debugflags:
> - pyshyacc.print_commands(cmds, self._logfile)
> - self._logfile.flush()
> - else:
> - cmds, script = ast, ''
> -
> - status = 0
> - for cmd in cmds:
> - try:
> - status = self.execute(cmd)
> - except ExitSignal as e:
> - if sourced:
> - raise
> - status = int(e.args[0])
> - return status
> - except ShellError:
> - self._env['?'] = 1
> - raise
> - if 'debug-utility' in self._debugflags or
> 'debug-cmd' in self._debugflags:
> - self.log('returncode ' + str(status)+ '\n')
> - return status
> - except CommandNotFound as e:
> - print >>self._redirs.stderr, str(e)
> - self._redirs.stderr.flush()
> - # Command not found by non-interactive shell
> - # return 127
> - raise
> - except RedirectionError as e:
> - # TODO: should be handled depending on the utility status
> - print >>self._redirs.stderr, str(e)
> - self._redirs.stderr.flush()
> - # Command not found by non-interactive shell
> - # return 127
> - raise
> -
> - def dotcommand(self, env, args):
> - if len(args) < 1:
> - raise ShellError('. expects at least one argument')
> - path = args[0]
> - if '/' not in path:
> - found = env.find_in_path(args[0], True)
> - if found:
> - path = found[0]
> - script = file(path).read()
> - return self.execute_script(script=script, sourced=True)
> -
> - def execute(self, token, redirs=None):
> - """Execute and AST subtree with supplied redirections
> overriding default
> - interpreter ones.
> - Return the exit status.
> - """
> - if not token:
> - return 0
> -
> - if redirs is None:
> - redirs = self._redirs
> -
> - if isinstance(token, list):
> - # Commands sequence
> - res = 0
> - for t in token:
> - res = self.execute(t, redirs)
> - return res
> -
> - type, value = token
> - status = 0
> - if type=='simple_command':
> - redirs_copy = redirs.clone()
> - try:
> - # TODO: define and handle command return values
> - # TODO: implement set -e
> - status = self._execute_simple_command(value,
> redirs_copy)
> - finally:
> - redirs_copy.close()
> - elif type=='pipeline':
> - status = self._execute_pipeline(value, redirs)
> - elif type=='and_or':
> - status = self._execute_and_or(value, redirs)
> - elif type=='for_clause':
> - status = self._execute_for_clause(value, redirs)
> - elif type=='while_clause':
> - status = self._execute_while_clause(value, redirs)
> - elif type=='function_definition':
> - status = self._execute_function_definition(value, redirs)
> - elif type=='brace_group':
> - status = self._execute_brace_group(value, redirs)
> - elif type=='if_clause':
> - status = self._execute_if_clause(value, redirs)
> - elif type=='subshell':
> - status = self.subshell(ast=value.cmds, redirs=redirs)
> - elif type=='async':
> - status = self._asynclist(value)
> - elif type=='redirect_list':
> - redirs_copy = self.redirect(redirs.clone(), value.redirs)
> - try:
> - status = self.execute(value.cmd, redirs_copy)
> - finally:
> - redirs_copy.close()
> - else:
> - raise NotImplementedError('Unsupported token type ' +
> type) -
> - if status < 0:
> - status = 255
> - return status
> -
> - def _execute_if_clause(self, if_clause, redirs):
> - cond_status = self.execute(if_clause.cond, redirs)
> - if cond_status==0:
> - return self.execute(if_clause.if_cmds, redirs)
> - else:
> - return self.execute(if_clause.else_cmds, redirs)
> -
> - def _execute_brace_group(self, group, redirs):
> - status = 0
> - for cmd in group.cmds:
> - status = self.execute(cmd, redirs)
> - return status
> -
> - def _execute_function_definition(self, fundef, redirs):
> - self._env.define_function(fundef.name, fundef.body)
> - return 0
> -
> - def _execute_while_clause(self, while_clause, redirs):
> - status = 0
> - while 1:
> - cond_status = 0
> - for cond in while_clause.condition:
> - cond_status = self.execute(cond, redirs)
> -
> - if cond_status:
> - break
> -
> - for cmd in while_clause.cmds:
> - status = self.execute(cmd, redirs)
> -
> - return status
> -
> - def _execute_for_clause(self, for_clause, redirs):
> - if not is_name(for_clause.name):
> - raise ShellSyntaxError('%s is not a valid name' %
> repr(for_clause.name))
> - items = mappend(self.expand_token, for_clause.items)
> -
> - status = 0
> - for item in items:
> - self._env[for_clause.name] = item
> - for cmd in for_clause.cmds:
> - status = self.execute(cmd, redirs)
> - return status
> -
> - def _execute_and_or(self, or_and, redirs):
> - res = self.execute(or_and.left, redirs)
> - if (or_and.op=='&&' and res==0) or (or_and.op!='&&' and
> res!=0):
> - res = self.execute(or_and.right, redirs)
> - return res
> -
> - def _execute_pipeline(self, pipeline, redirs):
> - if len(pipeline.commands)==1:
> - status = self.execute(pipeline.commands[0], redirs)
> - else:
> - # Execute all commands one after the other
> - status = 0
> - inpath, outpath = None, None
> - try:
> - # Commands inputs and outputs cannot really be
> plugged as done
> - # by a real shell. Run commands sequentially and
> chain their
> - # input/output throught temporary files.
> - tmpfd, inpath = tempfile.mkstemp()
> - os.close(tmpfd)
> - tmpfd, outpath = tempfile.mkstemp()
> - os.close(tmpfd)
> -
> - inpath = win32_to_unix_path(inpath)
> - outpath = win32_to_unix_path(outpath)
> -
> - for i, cmd in enumerate(pipeline.commands):
> - call_redirs = redirs.clone()
> - try:
> - if i!=0:
> - call_redirs.add(self, '<', inpath)
> - if i!=len(pipeline.commands)-1:
> - call_redirs.add(self, '>', outpath)
> -
> - status = self.execute(cmd, call_redirs)
> -
> - # Chain inputs/outputs
> - inpath, outpath = outpath, inpath
> - finally:
> - call_redirs.close()
> - finally:
> - if inpath: os.remove(inpath)
> - if outpath: os.remove(outpath)
> -
> - if pipeline.reverse_status:
> - status = int(not status)
> - self._env['?'] = status
> - return status
> -
> - def _execute_function(self, name, args, interp, env, stdin,
> stdout, stderr, *others):
> - assert interp is self
> -
> - func = env.get_function(name)
> - #Set positional parameters
> - prevargs = None
> - try:
> - prevargs = env.set_positional_args(args)
> - try:
> - redirs = Redirections(stdin.dup(), stdout.dup(),
> stderr.dup())
> - try:
> - status = self.execute(func, redirs)
> - finally:
> - redirs.close()
> - except ReturnSignal as e:
> - status = int(e.args[0])
> - env['?'] = status
> - return status
> - finally:
> - #Reset positional parameters
> - if prevargs is not None:
> - env.set_positional_args(prevargs)
> -
> - def _execute_simple_command(self, token, redirs):
> - """Can raise ReturnSignal when return builtin is called,
> ExitSignal when
> - exit is called, and other shell exceptions upon builtin
> failures.
> - """
> - debug_command = 'debug-cmd' in self._debugflags
> - if debug_command:
> - self.log('word' + repr(token.words) + '\n')
> - self.log('assigns' + repr(token.assigns) + '\n')
> - self.log('redirs' + repr(token.redirs) + '\n')
> -
> - is_special = None
> - env = self._env
> -
> - try:
> - # Word expansion
> - args = []
> - for word in token.words:
> - args += self.expand_token(word)
> - if is_special is None and args:
> - is_special = env.is_function(args[0]) or \
> - (args[0] in self.COMMANDS and
> self.COMMANDS[args[0]].is_special)
> -
> - if debug_command:
> - self.log('_execute_simple_command' + str(args) +
> '\n')
> -
> - if not args:
> - # Redirections happen is a subshell
> - redirs = redirs.clone()
> - elif not is_special:
> - env = self._env.clone()
> -
> - # Redirections
> - self.redirect(redirs, token.redirs)
> -
> - # Variables assignments
> - res = 0
> - for type,(k,v) in token.assigns:
> - status, expanded = self.expand_variable((k,v))
> - if status is not None:
> - res = status
> - if args:
> - env.export(k, expanded)
> - else:
> - env[k] = expanded
> -
> - if args and args[0] in ('.', 'source'):
> - res = self.dotcommand(env, args[1:])
> - elif args:
> - if args[0] in self.COMMANDS:
> - command = self.COMMANDS[args[0]]
> - elif env.is_function(args[0]):
> - command = Utility(self._execute_function,
> is_special=True)
> - else:
> - if not '/' in args[0].replace('\\', '/'):
> - cmd = env.find_in_path(args[0])
> - if not cmd:
> - # TODO: test error code on unknown
> command => 127
> - raise CommandNotFound('Unknown command:
> "%s"' % args[0])
> - else:
> - # Handle commands like '/cygdrive/c/foo.bat'
> - cmd = cygwin_to_windows_path(args[0])
> - if not os.path.exists(cmd):
> - raise CommandNotFound('%s: No such file
> or directory' % args[0])
> - shebang = resolve_shebang(cmd)
> - if shebang:
> - cmd = shebang
> - else:
> - cmd = [cmd]
> - args[0:1] = cmd
> - command = Utility(builtin.run_command)
> -
> - # Command execution
> - if 'debug-cmd' in self._debugflags:
> - self.log('redirections ' + str(redirs) + '\n')
> -
> - res = command.func(args[0], args[1:], self, env,
> - redirs.stdin(), redirs.stdout(),
> - redirs.stderr(), self._debugflags)
> -
> - if self._env.has_opt('-x'):
> - # Trace command execution in shell environment
> - # BUG: would be hard to reproduce a real shell
> behaviour since
> - # the AST is not annotated with source lines/tokens.
> - self._redirs.stdout().write(' '.join(args))
> -
> - except ReturnSignal:
> - raise
> - except ShellError as e:
> - if is_special or isinstance(e, (ExitSignal,
> - ShellSyntaxError,
> ExpansionError)):
> - raise e
> - self._redirs.stderr().write(str(e)+'\n')
> - return 1
> -
> - return res
> -
> - def expand_token(self, word):
> - """Expand a word as specified in [2.6 Word Expansions].
> Return the list
> - of expanded words.
> - """
> - status, wtrees = self._expand_word(word)
> - return map(pyshlex.wordtree_as_string, wtrees)
> -
> - def expand_variable(self, word):
> - """Return a status code (or None if no command expansion
> occurred)
> - and a single word.
> - """
> - status, wtrees = self._expand_word(word, pathname=False,
> split=False)
> - words = map(pyshlex.wordtree_as_string, wtrees)
> - assert len(words)==1
> - return status, words[0]
> -
> - def expand_here_document(self, word):
> - """Return the expanded document as a single word. The here
> document is
> - assumed to be unquoted.
> - """
> - status, wtrees = self._expand_word(word, pathname=False,
> - split=False,
> here_document=True)
> - words = map(pyshlex.wordtree_as_string, wtrees)
> - assert len(words)==1
> - return words[0]
> -
> - def expand_redirection(self, word):
> - """Return a single word."""
> - return self.expand_variable(word)[1]
> -
> - def get_env(self):
> - return self._env
> -
> - def _expand_word(self, token, pathname=True, split=True,
> here_document=False):
> - wtree = pyshlex.make_wordtree(token[1],
> here_document=here_document)
> -
> - # TODO: implement tilde expansion
> - def expand(wtree):
> - """Return a pseudo wordtree: the tree or its subelements
> can be empty
> - lists when no value result from the expansion.
> - """
> - status = None
> - for part in wtree:
> - if not isinstance(part, list):
> - continue
> - if part[0]in ("'", '\\'):
> - continue
> - elif part[0] in ('`', '$('):
> - status, result = self._expand_command(part)
> - part[:] = result
> - elif part[0] in ('$', '${'):
> - part[:] = self._expand_parameter(part,
> wtree[0]=='"', split)
> - elif part[0] in ('', '"'):
> - status, result = expand(part)
> - part[:] = result
> - else:
> - raise NotImplementedError('%s expansion is not
> implemented'
> - % part[0])
> - # [] is returned when an expansion result in no-field,
> - # like an empty $@
> - wtree = [p for p in wtree if p != []]
> - if len(wtree) < 3:
> - return status, []
> - return status, wtree
> -
> - status, wtree = expand(wtree)
> - if len(wtree) == 0:
> - return status, wtree
> - wtree = pyshlex.normalize_wordtree(wtree)
> -
> - if split:
> - wtrees = self._split_fields(wtree)
> - else:
> - wtrees = [wtree]
> -
> - if pathname:
> - wtrees = mappend(self._expand_pathname, wtrees)
> -
> - wtrees = map(self._remove_quotes, wtrees)
> - return status, wtrees
> -
> - def _expand_command(self, wtree):
> - # BUG: there is something to do with backslashes and quoted
> - # characters here
> - command = pyshlex.wordtree_as_string(wtree[1:-1])
> - status, output = self.subshell_output(command)
> - return status, ['', output, '']
> -
> - def _expand_parameter(self, wtree, quoted=False, split=False):
> - """Return a valid wtree or an empty list when no parameter
> results."""
> - # Get the parameter name
> - # TODO: implement weird expansion rules with ':'
> - name = pyshlex.wordtree_as_string(wtree[1:-1])
> - if not is_name(name) and not is_special_param(name):
> - raise ExpansionError('Bad substitution "%s"' % name)
> - # TODO: implement special parameters
> - if name in ('@', '*'):
> - args = self._env.get_positional_args()
> - if len(args) == 0:
> - return []
> - if len(args)<2:
> - return ['', ''.join(args), '']
> -
> - sep = self._env.get('IFS', '')[:1]
> - if split and quoted and name=='@':
> - # Introduce a new token to tell the caller that
> these parameters
> - # cause a split as specified in 2.5.2
> - return ['@'] + args + ['']
> - else:
> - return ['', sep.join(args), '']
> -
> - return ['', self._env.get(name, ''), '']
> -
> - def _split_fields(self, wtree):
> - def is_empty(split):
> - return split==['', '', '']
> -
> - def split_positional(quoted):
> - # Return a list of wtree split according positional
> parameters rules.
> - # All remaining '@' groups are removed.
> - assert quoted[0]=='"'
> -
> - splits = [[]]
> - for part in quoted:
> - if not isinstance(part, list) or part[0]!='@':
> - splits[-1].append(part)
> - else:
> - # Empty or single argument list were dealt with
> already
> - assert len(part)>3
> - # First argument must join with the beginning
> part of the original word
> - splits[-1].append(part[1])
> - # Create double-quotes expressions for every
> argument after the first
> - for arg in part[2:-1]:
> - splits[-1].append('"')
> - splits.append(['"', arg])
> - return splits
> -
> - # At this point, all expansions but pathnames have occured.
> Only quoted
> - # and positional sequences remain. Thus, all candidates for
> field splitting
> - # are in the tree root, or are positional splits ('@') and
> lie in root
> - # children.
> - if not wtree or wtree[0] not in ('', '"'):
> - # The whole token is quoted or empty, nothing to split
> - return [wtree]
> -
> - if wtree[0]=='"':
> - wtree = ['', wtree, '']
> -
> - result = [['', '']]
> - for part in wtree[1:-1]:
> - if isinstance(part, list):
> - if part[0]=='"':
> - splits = split_positional(part)
> - if len(splits)<=1:
> - result[-1] += [part, '']
> - else:
> - # Terminate the current split
> - result[-1] += [splits[0], '']
> - result += splits[1:-1]
> - # Create a new split
> - result += [['', splits[-1], '']]
> - else:
> - result[-1] += [part, '']
> - else:
> - splits = self._env.split_fields(part)
> - if len(splits)<=1:
> - # No split
> - result[-1][-1] += part
> - else:
> - # Terminate the current resulting part and
> create a new one
> - result[-1][-1] += splits[0]
> - result[-1].append('')
> - result += [['', r, ''] for r in splits[1:-1]]
> - result += [['', splits[-1]]]
> - result[-1].append('')
> -
> - # Leading and trailing empty groups come from
> leading/trailing blanks
> - if result and is_empty(result[-1]):
> - result[-1:] = []
> - if result and is_empty(result[0]):
> - result[:1] = []
> - return result
> -
> - def _expand_pathname(self, wtree):
> - """See [2.6.6 Pathname Expansion]."""
> - if self._env.has_opt('-f'):
> - return [wtree]
> -
> - # All expansions have been performed, only quoted sequences
> should remain
> - # in the tree. Generate the pattern by folding the tree,
> escaping special
> - # characters when appear quoted
> - special_chars = '*?[]'
> -
> - def make_pattern(wtree):
> - subpattern = []
> - for part in wtree[1:-1]:
> - if isinstance(part, list):
> - part = make_pattern(part)
> - elif wtree[0]!='':
> - for c in part:
> - # Meta-characters cannot be quoted
> - if c in special_chars:
> - raise GlobError()
> - subpattern.append(part)
> - return ''.join(subpattern)
> -
> - def pwd_glob(pattern):
> - cwd = os.getcwd()
> - os.chdir(self._env['PWD'])
> - try:
> - return glob.glob(pattern)
> - finally:
> - os.chdir(cwd)
> -
> - #TODO: check working directory issues here wrt relative
> patterns
> - try:
> - pattern = make_pattern(wtree)
> - paths = pwd_glob(pattern)
> - except GlobError:
> - # BUG: Meta-characters were found in quoted sequences.
> The should
> - # have been used literally but this is unsupported in
> current glob module.
> - # Instead we consider the whole tree must be used
> literally and
> - # therefore there is no point in globbing. This is wrong
> when meta
> - # characters are mixed with quoted meta in the same
> pattern like:
> - # < foo*"py*" >
> - paths = []
> -
> - if not paths:
> - return [wtree]
> - return [['', path, ''] for path in paths]
> -
> - def _remove_quotes(self, wtree):
> - """See [2.6.7 Quote Removal]."""
> -
> - def unquote(wtree):
> - unquoted = []
> - for part in wtree[1:-1]:
> - if isinstance(part, list):
> - part = unquote(part)
> - unquoted.append(part)
> - return ''.join(unquoted)
> -
> - return ['', unquote(wtree), '']
> -
> - def subshell(self, script=None, ast=None, redirs=None):
> - """Execute the script or AST in a subshell, with inherited
> redirections
> - if redirs is not None.
> - """
> - if redirs:
> - sub_redirs = redirs
> - else:
> - sub_redirs = redirs.clone()
> -
> - subshell = None
> - try:
> - subshell = Interpreter(None, self._debugflags,
> self._env.clone(True),
> - sub_redirs, opts=self._options)
> - return subshell.execute_script(script, ast)
> - finally:
> - if not redirs: sub_redirs.close()
> - if subshell: subshell.close()
> -
> - def subshell_output(self, script):
> - """Execute the script in a subshell and return the captured
> output."""
> - # Create temporary file to capture subshell output
> - tmpfd, tmppath = tempfile.mkstemp()
> - try:
> - tmpfile = os.fdopen(tmpfd, 'wb')
> - stdout = FileWrapper('w', tmpfile)
> -
> - redirs = Redirections(self._redirs.stdin().dup(),
> - stdout,
> -
> self._redirs.stderr().dup())
> - try:
> - status = self.subshell(script=script, redirs=redirs)
> - finally:
> - redirs.close()
> - redirs = None
> -
> - # Extract subshell standard output
> - tmpfile = open(tmppath, 'rb')
> - try:
> - output = tmpfile.read()
> - return status, output.rstrip('\n')
> - finally:
> - tmpfile.close()
> - finally:
> - os.remove(tmppath)
> -
> - def _asynclist(self, cmd):
> - args = (self._env.get_variables(), cmd)
> - arg = encodeargs(args)
> - assert len(args) < 30*1024
> - cmd = ['pysh.bat', '--ast', '-c', arg]
> - p = subprocess.Popen(cmd, cwd=self._env['PWD'])
> - self._children[p.pid] = p
> - self._env['!'] = p.pid
> - return 0
> -
> - def wait(self, pids=None):
> - if not pids:
> - pids = self._children.keys()
> -
> - status = 127
> - for pid in pids:
> - if pid not in self._children:
> - continue
> - p = self._children.pop(pid)
> - status = p.wait()
> -
> - return status
> -
> diff --git a/bitbake/lib/bb/pysh/lsprof.py
> b/bitbake/lib/bb/pysh/lsprof.py deleted file mode 100644
> index b1831c2..0000000
> --- a/bitbake/lib/bb/pysh/lsprof.py
> +++ /dev/null
> @@ -1,116 +0,0 @@
> -#! /usr/bin/env python
> -
> -import sys
> -from _lsprof import Profiler, profiler_entry
> -
> -__all__ = ['profile', 'Stats']
> -
> -def profile(f, *args, **kwds):
> - """XXX docstring"""
> - p = Profiler()
> - p.enable(subcalls=True, builtins=True)
> - try:
> - f(*args, **kwds)
> - finally:
> - p.disable()
> - return Stats(p.getstats())
> -
> -
> -class Stats(object):
> - """XXX docstring"""
> -
> - def __init__(self, data):
> - self.data = data
> -
> - def sort(self, crit="inlinetime"):
> - """XXX docstring"""
> - if crit not in profiler_entry.__dict__:
> - raise ValueError("Can't sort by %s" % crit)
> - self.data.sort(lambda b, a: cmp(getattr(a, crit),
> - getattr(b, crit)))
> - for e in self.data:
> - if e.calls:
> - e.calls.sort(lambda b, a: cmp(getattr(a, crit),
> - getattr(b, crit)))
> -
> - def pprint(self, top=None, file=None, limit=None, climit=None):
> - """XXX docstring"""
> - if file is None:
> - file = sys.stdout
> - d = self.data
> - if top is not None:
> - d = d[:top]
> - cols = "% 12s %12s %11.4f %11.4f %s\n"
> - hcols = "% 12s %12s %12s %12s %s\n"
> - cols2 = "+%12s %12s %11.4f %11.4f + %s\n"
> - file.write(hcols % ("CallCount", "Recursive", "Total(ms)",
> - "Inline(ms)", "module:lineno(function)"))
> - count = 0
> - for e in d:
> - file.write(cols % (e.callcount, e.reccallcount,
> e.totaltime,
> - e.inlinetime, label(e.code)))
> - count += 1
> - if limit is not None and count == limit:
> - return
> - ccount = 0
> - if e.calls:
> - for se in e.calls:
> - file.write(cols % ("+%s" % se.callcount,
> se.reccallcount,
> - se.totaltime, se.inlinetime,
> - "+%s" % label(se.code)))
> - count += 1
> - ccount += 1
> - if limit is not None and count == limit:
> - return
> - if climit is not None and ccount == climit:
> - break
> -
> - def freeze(self):
> - """Replace all references to code objects with string
> - descriptions; this makes it possible to pickle the
> instance.""" -
> - # this code is probably rather ickier than it needs to be!
> - for i in range(len(self.data)):
> - e = self.data[i]
> - if not isinstance(e.code, str):
> - self.data[i] = type(e)((label(e.code),) + e[1:])
> - if e.calls:
> - for j in range(len(e.calls)):
> - se = e.calls[j]
> - if not isinstance(se.code, str):
> - e.calls[j] = type(se)((label(se.code),) +
> se[1:]) -
> -_fn2mod = {}
> -
> -def label(code):
> - if isinstance(code, str):
> - return code
> - try:
> - mname = _fn2mod[code.co_filename]
> - except KeyError:
> - for k, v in sys.modules.items():
> - if v is None:
> - continue
> - if not hasattr(v, '__file__'):
> - continue
> - if not isinstance(v.__file__, str):
> - continue
> - if v.__file__.startswith(code.co_filename):
> - mname = _fn2mod[code.co_filename] = k
> - break
> - else:
> - mname = _fn2mod[code.co_filename] =
> '<%s>'%code.co_filename -
> - return '%s:%d(%s)' % (mname, code.co_firstlineno, code.co_name)
> -
> -
> -if __name__ == '__main__':
> - import os
> - sys.argv = sys.argv[1:]
> - if not sys.argv:
> - print >> sys.stderr, "usage: lsprof.py <script>
> <arguments...>"
> - sys.exit(2)
> - sys.path.insert(0, os.path.abspath(os.path.dirname(sys.argv[0])))
> - stats = profile(execfile, sys.argv[0], globals(), locals())
> - stats.sort()
> - stats.pprint()
> diff --git a/bitbake/lib/bb/pysh/pysh.py b/bitbake/lib/bb/pysh/pysh.py
> deleted file mode 100644
> index b4e6145..0000000
> --- a/bitbake/lib/bb/pysh/pysh.py
> +++ /dev/null
> @@ -1,167 +0,0 @@
> -# pysh.py - command processing for pysh.
> -#
> -# Copyright 2007 Patrick Mezard
> -#
> -# This software may be used and distributed according to the terms
> -# of the GNU General Public License, incorporated herein by
> reference. -
> -import optparse
> -import os
> -import sys
> -
> -import interp
> -
> -SH_OPT = optparse.OptionParser(prog='pysh', usage="%prog [OPTIONS]",
> version='0.1') -SH_OPT.add_option('-c', action='store_true',
> dest='command_string', default=None,
> - help='A string that shall be interpreted by the shell as one or
> more commands') -SH_OPT.add_option('--redirect-to',
> dest='redirect_to', default=None,
> - help='Redirect script commands stdout and stderr to the
> specified file') -# See utility_command in builtin.py about the
> reason for this flag. -SH_OPT.add_option('--redirected',
> dest='redirected', action='store_true', default=False,
> - help='Tell the interpreter that stdout and stderr are actually
> the same objects, which is really stdout')
> -SH_OPT.add_option('--debug-parsing', action='store_true',
> dest='debug_parsing', default=False,
> - help='Trace PLY execution')
> -SH_OPT.add_option('--debug-tree', action='store_true',
> dest='debug_tree', default=False,
> - help='Display the generated syntax tree.')
> -SH_OPT.add_option('--debug-cmd', action='store_true',
> dest='debug_cmd', default=False,
> - help='Trace command execution before parameters expansion and
> exit status.') -SH_OPT.add_option('--debug-utility',
> action='store_true', dest='debug_utility', default=False,
> - help='Trace utility calls, after parameters expansions')
> -SH_OPT.add_option('--ast', action='store_true', dest='ast',
> default=False,
> - help='Encoded commands to execute in a subprocess')
> -SH_OPT.add_option('--profile', action='store_true', default=False,
> - help='Profile pysh run')
> -
> -
> -def split_args(args):
> - # Separate shell arguments from command ones
> - # Just stop at the first argument not starting with a dash. I
> know, this is completely broken,
> - # it ignores files starting with a dash or may take option
> values for command file. This is not
> - # supposed to happen for now
> - command_index = len(args)
> - for i,arg in enumerate(args):
> - if not arg.startswith('-'):
> - command_index = i
> - break
> -
> - return args[:command_index], args[command_index:]
> -
> -
> -def fixenv(env):
> - path = env.get('PATH')
> - if path is not None:
> - parts = path.split(os.pathsep)
> - # Remove Windows utilities from PATH, they are useless at
> best and
> - # some of them (find) may be confused with other utilities.
> - parts = [p for p in parts if 'system32' not in p.lower()]
> - env['PATH'] = os.pathsep.join(parts)
> - if env.get('HOME') is None:
> - # Several utilities, including cvsps, cannot work without
> - # a defined HOME directory.
> - env['HOME'] = os.path.expanduser('~')
> - return env
> -
> -def _sh(cwd, shargs, cmdargs, options, debugflags=None, env=None):
> - if os.environ.get('PYSH_TEXT') != '1':
> - import msvcrt
> - for fp in (sys.stdin, sys.stdout, sys.stderr):
> - msvcrt.setmode(fp.fileno(), os.O_BINARY)
> -
> - hgbin = os.environ.get('PYSH_HGTEXT') != '1'
> -
> - if debugflags is None:
> - debugflags = []
> - if options.debug_parsing:
> debugflags.append('debug-parsing')
> - if options.debug_utility:
> debugflags.append('debug-utility')
> - if options.debug_cmd: debugflags.append('debug-cmd')
> - if options.debug_tree: debugflags.append('debug-tree')
> -
> - if env is None:
> - env = fixenv(dict(os.environ))
> - if cwd is None:
> - cwd = os.getcwd()
> -
> - if not cmdargs:
> - # Nothing to do
> - return 0
> -
> - ast = None
> - command_file = None
> - if options.command_string:
> - input = cmdargs[0]
> - if not options.ast:
> - input += '\n'
> - else:
> - args, input = interp.decodeargs(input), None
> - env, ast = args
> - cwd = env.get('PWD', cwd)
> - else:
> - command_file = cmdargs[0]
> - arguments = cmdargs[1:]
> -
> - prefix = interp.resolve_shebang(command_file,
> ignoreshell=True)
> - if prefix:
> - input = ' '.join(prefix + [command_file] + arguments)
> - else:
> - # Read commands from file
> - f = file(command_file)
> - try:
> - # Trailing newline to help the parser
> - input = f.read() + '\n'
> - finally:
> - f.close()
> -
> - redirect = None
> - try:
> - if options.redirected:
> - stdout = sys.stdout
> - stderr = stdout
> - elif options.redirect_to:
> - redirect = open(options.redirect_to, 'wb')
> - stdout = redirect
> - stderr = redirect
> - else:
> - stdout = sys.stdout
> - stderr = sys.stderr
> -
> - # TODO: set arguments to environment variables
> - opts = interp.Options()
> - opts.hgbinary = hgbin
> - ip = interp.Interpreter(cwd, debugflags, stdout=stdout,
> stderr=stderr,
> - opts=opts)
> - try:
> - # Export given environment in shell object
> - for k,v in env.iteritems():
> - ip.get_env().export(k,v)
> - return ip.execute_script(input, ast,
> scriptpath=command_file)
> - finally:
> - ip.close()
> - finally:
> - if redirect is not None:
> - redirect.close()
> -
> -def sh(cwd=None, args=None, debugflags=None, env=None):
> - if args is None:
> - args = sys.argv[1:]
> - shargs, cmdargs = split_args(args)
> - options, shargs = SH_OPT.parse_args(shargs)
> -
> - if options.profile:
> - import lsprof
> - p = lsprof.Profiler()
> - p.enable(subcalls=True)
> - try:
> - return _sh(cwd, shargs, cmdargs, options, debugflags,
> env)
> - finally:
> - p.disable()
> - stats = lsprof.Stats(p.getstats())
> - stats.sort()
> - stats.pprint(top=10, file=sys.stderr, climit=5)
> - else:
> - return _sh(cwd, shargs, cmdargs, options, debugflags, env)
> -
> -def main():
> - sys.exit(sh())
> -
> -if __name__=='__main__':
> - main()
> diff --git a/bitbake/lib/bb/pysh/pyshlex.py
> b/bitbake/lib/bb/pysh/pyshlex.py index fbf094b..a42c294 100644
> --- a/bitbake/lib/bb/pysh/pyshlex.py
> +++ b/bitbake/lib/bb/pysh/pyshlex.py
> @@ -13,11 +13,6 @@
> # PLY in pull mode. It was designed to work incrementally and it
> would not be # that hard to enable pull mode.
> import re
> -try:
> - s = set()
> - del s
> -except NameError:
> - from Set import Set as set
>
> from ply import lex
> from bb.pysh.sherrors import *
> diff --git a/bitbake/lib/bb/pysh/pyshyacc.py
> b/bitbake/lib/bb/pysh/pyshyacc.py index ba4cefd..de565dc 100644
> --- a/bitbake/lib/bb/pysh/pyshyacc.py
> +++ b/bitbake/lib/bb/pysh/pyshyacc.py
> @@ -636,13 +636,16 @@ def p_empty(p):
> def p_error(p):
> msg = []
> w = msg.append
> - w('%r\n' % p)
> - w('followed by:\n')
> - for i in range(5):
> - n = yacc.token()
> - if not n:
> - break
> - w(' %r\n' % n)
> + if p:
> + w('%r\n' % p)
> + w('followed by:\n')
> + for i in range(5):
> + n = yacc.token()
> + if not n:
> + break
> + w(' %r\n' % n)
> + else:
> + w('Unexpected EOF')
> raise sherrors.ShellSyntaxError(''.join(msg))
>
> # Build the parser
> diff --git a/bitbake/lib/bb/pysh/sherrors.py
> b/bitbake/lib/bb/pysh/sherrors.py index 49d0533..3fe8e47 100644
> --- a/bitbake/lib/bb/pysh/sherrors.py
> +++ b/bitbake/lib/bb/pysh/sherrors.py
> @@ -13,29 +13,3 @@ class ShellError(Exception):
>
> class ShellSyntaxError(ShellError):
> pass
> -
> -class UtilityError(ShellError):
> - """Raised upon utility syntax error (option or operand error)."""
> - pass
> -
> -class ExpansionError(ShellError):
> - pass
> -
> -class CommandNotFound(ShellError):
> - """Specified command was not found."""
> - pass
> -
> -class RedirectionError(ShellError):
> - pass
> -
> -class VarAssignmentError(ShellError):
> - """Variable assignment error."""
> - pass
> -
> -class ExitSignal(ShellError):
> - """Exit signal."""
> - pass
> -
> -class ReturnSignal(ShellError):
> - """Exit signal."""
> - pass
> diff --git a/bitbake/lib/bb/pysh/subprocess_fix.py
> b/bitbake/lib/bb/pysh/subprocess_fix.py deleted file mode 100644
> index 46eca22..0000000
> --- a/bitbake/lib/bb/pysh/subprocess_fix.py
> +++ /dev/null
> @@ -1,77 +0,0 @@
> -# subprocess - Subprocesses with accessible I/O streams
> -#
> -# For more information about this module, see PEP 324.
> -#
> -# This module should remain compatible with Python 2.2, see PEP 291.
> -#
> -# Copyright (c) 2003-2005 by Peter Astrand <astrand@lysator.liu.se>
> -#
> -# Licensed to PSF under a Contributor Agreement.
> -# See http://www.python.org/2.4/license for licensing details.
> -
> -def list2cmdline(seq):
> - """
> - Translate a sequence of arguments into a command line
> - string, using the same rules as the MS C runtime:
> -
> - 1) Arguments are delimited by white space, which is either a
> - space or a tab.
> -
> - 2) A string surrounded by double quotation marks is
> - interpreted as a single argument, regardless of white space
> - contained within. A quoted string can be embedded in an
> - argument.
> -
> - 3) A double quotation mark preceded by a backslash is
> - interpreted as a literal double quotation mark.
> -
> - 4) Backslashes are interpreted literally, unless they
> - immediately precede a double quotation mark.
> -
> - 5) If backslashes immediately precede a double quotation mark,
> - every pair of backslashes is interpreted as a literal
> - backslash. If the number of backslashes is odd, the last
> - backslash escapes the next double quotation mark as
> - described in rule 3.
> - """
> -
> - # See
> - #
> http://msdn.microsoft.com/library/en-us/vccelng/htm/progs_12.asp
> - result = []
> - needquote = False
> - for arg in seq:
> - bs_buf = []
> -
> - # Add a space to separate this argument from the others
> - if result:
> - result.append(' ')
> -
> - needquote = (" " in arg) or ("\t" in arg) or ("|" in arg) or
> arg == ""
> - if needquote:
> - result.append('"')
> -
> - for c in arg:
> - if c == '\\':
> - # Don't know if we need to double yet.
> - bs_buf.append(c)
> - elif c == '"':
> - # Double backspaces.
> - result.append('\\' * len(bs_buf)*2)
> - bs_buf = []
> - result.append('\\"')
> - else:
> - # Normal char
> - if bs_buf:
> - result.extend(bs_buf)
> - bs_buf = []
> - result.append(c)
> -
> - # Add remaining backspaces, if any.
> - if bs_buf:
> - result.extend(bs_buf)
> -
> - if needquote:
> - result.extend(bs_buf)
> - result.append('"')
> -
> - return ''.join(result)
> diff --git a/bitbake/lib/bb/remotedata.py
> b/bitbake/lib/bb/remotedata.py index 68ecffc..7391e1b 100644
> --- a/bitbake/lib/bb/remotedata.py
> +++ b/bitbake/lib/bb/remotedata.py
> @@ -6,18 +6,8 @@ Provides support for using a datastore from the
> bitbake client
> # Copyright (C) 2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import bb.data
>
> diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
> index 9ce06c4..1804943 100644
> --- a/bitbake/lib/bb/runqueue.py
> +++ b/bitbake/lib/bb/runqueue.py
> @@ -1,6 +1,3 @@
> -#!/usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake 'RunQueue' implementation
>
> @@ -9,18 +6,8 @@ Handles preparation and execution of a queue of tasks
>
> # Copyright (C) 2006-2007 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import copy
> import os
> @@ -37,11 +24,13 @@ from bb import monitordisk
> import subprocess
> import pickle
> from multiprocessing import Process
> +import shlex
> +import pprint
>
> bblogger = logging.getLogger("BitBake")
> logger = logging.getLogger("BitBake.RunQueue")
>
> -__find_md5__ =
> re.compile( r'(?i)(?<![a-z0-9])[a-f0-9]{32}(?![a-z0-9])' )
> +__find_sha256__ =
> re.compile( r'(?i)(?<![a-z0-9])[a-f0-9]{64}(?![a-z0-9])' ) def
> fn_from_tid(tid): return tid.rsplit(":", 1)[0]
> @@ -49,17 +38,22 @@ def fn_from_tid(tid):
> def taskname_from_tid(tid):
> return tid.rsplit(":", 1)[1]
>
> +def mc_from_tid(tid):
> + if tid.startswith('mc:'):
> + return tid.split(':')[1]
> + return ""
> +
> def split_tid(tid):
> (mc, fn, taskname, _) = split_tid_mcfn(tid)
> return (mc, fn, taskname)
>
> def split_tid_mcfn(tid):
> - if tid.startswith('multiconfig:'):
> + if tid.startswith('mc:'):
> elems = tid.split(':')
> mc = elems[1]
> fn = ":".join(elems[2:-1])
> taskname = elems[-1]
> - mcfn = "multiconfig:" + mc + ":" + fn
> + mcfn = "mc:" + mc + ":" + fn
> else:
> tid = tid.rsplit(":", 1)
> mc = ""
> @@ -71,9 +65,17 @@ def split_tid_mcfn(tid):
>
> def build_tid(mc, fn, taskname):
> if mc:
> - return "multiconfig:" + mc + ":" + fn + ":" + taskname
> + return "mc:" + mc + ":" + fn + ":" + taskname
> return fn + ":" + taskname
>
> +# Index used to pair up potentially matching multiconfig tasks
> +# We match on PN, taskname and hash being equal
> +def pending_hash_index(tid, rqdata):
> + (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
> + pn = rqdata.dataCaches[mc].pkg_fn[taskfn]
> + h = rqdata.runtaskentries[tid].unihash
> + return pn + ":" + "taskname" + h
> +
> class RunQueueStats:
> """
> Holds statistics on the tasks handled by the associated runQueue
> @@ -109,8 +111,6 @@ class RunQueueStats:
> # runQueue state machine
> runQueuePrepare = 2
> runQueueSceneInit = 3
> -runQueueSceneRun = 4
> -runQueueRunInit = 5
> runQueueRunning = 6
> runQueueFailed = 7
> runQueueCleanUp = 8
> @@ -133,7 +133,7 @@ class RunQueueScheduler(object):
>
> self.prio_map = [self.rqdata.runtaskentries.keys()]
>
> - self.buildable = []
> + self.buildable = set()
> self.skip_maxthread = {}
> self.stamps = {}
> for tid in self.rqdata.runtaskentries:
> @@ -148,8 +148,11 @@ class RunQueueScheduler(object):
> """
> Return the id of the first task we find that is buildable
> """
> - self.buildable = [x for x in self.buildable if x not in
> self.rq.runq_running]
> - if not self.buildable:
> + buildable = set(self.buildable)
> + buildable.difference_update(self.rq.runq_running)
> + buildable.difference_update(self.rq.holdoff_tasks)
> + buildable.intersection_update(self.rq.tasks_covered |
> self.rq.tasks_notcovered)
> + if not buildable:
> return None
>
> # Filter out tasks that have a max number of threads that
> have been exceeded @@ -165,8 +168,8 @@ class
> RunQueueScheduler(object): else:
> skip_buildable[rtaskname] = 1
>
> - if len(self.buildable) == 1:
> - tid = self.buildable[0]
> + if len(buildable) == 1:
> + tid = buildable.pop()
> taskname = taskname_from_tid(tid)
> if taskname in skip_buildable and
> skip_buildable[taskname] >= int(self.skip_maxthread[taskname]):
> return None @@ -181,7 +184,7 @@ class RunQueueScheduler(object):
>
> best = None
> bestprio = None
> - for tid in self.buildable:
> + for tid in buildable:
> taskname = taskname_from_tid(tid)
> if taskname in skip_buildable and
> skip_buildable[taskname] >= int(self.skip_maxthread[taskname]):
> continue @@ -203,7 +206,12 @@ class RunQueueScheduler(object):
> return self.next_buildable_task()
>
> def newbuildable(self, task):
> - self.buildable.append(task)
> + self.buildable.add(task)
> + # Once tasks are running we don't need to worry about them
> again
> + self.buildable.difference_update(self.rq.runq_running)
> +
> + def removebuildable(self, task):
> + self.buildable.remove(task)
>
> def describe_task(self, taskid):
> result = 'ID %s' % taskid
> @@ -346,6 +354,7 @@ class RunTaskEntry(object):
> self.depends = set()
> self.revdeps = set()
> self.hash = None
> + self.unihash = None
> self.task = None
> self.weight = 1
>
> @@ -385,6 +394,9 @@ class RunQueueData:
> def get_task_hash(self, tid):
> return self.runtaskentries[tid].hash
>
> + def get_task_unihash(self, tid):
> + return self.runtaskentries[tid].unihash
> +
> def get_user_idstring(self, tid, task_name_suffix = ""):
> return tid + task_name_suffix
>
> @@ -405,6 +417,9 @@ class RunQueueData:
> explored_deps = {}
> msgs = []
>
> + class TooManyLoops(Exception):
> + pass
> +
> def chain_reorder(chain):
> """
> Reorder a dependency chain so the lowest task id is first
> @@ -457,7 +472,7 @@ class RunQueueData:
> msgs.append("\n")
> if len(valid_chains) > 10:
> msgs.append("Aborted dependency loops search
> after 10 matches.\n")
> - return msgs
> + raise TooManyLoops
> continue
> scan = False
> if revdep not in explored_deps:
> @@ -476,8 +491,11 @@ class RunQueueData:
>
> explored_deps[tid] = total_deps
>
> - for task in tasks:
> - find_chains(task, [])
> + try:
> + for task in tasks:
> + find_chains(task, [])
> + except TooManyLoops:
> + pass
>
> return msgs
>
> @@ -833,6 +851,20 @@ class RunQueueData:
> for depend in depends:
> mark_active(depend, depth+1)
>
> + def invalidate_task(tid, error_nostamp):
> + (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
> + taskdep = self.dataCaches[mc].task_deps[taskfn]
> + if fn + ":" + taskname not in taskData[mc].taskentries:
> + logger.warning("Task %s does not exist, invalidating
> this task will have no effect" % taskname)
> + if 'nostamp' in taskdep and taskname in
> taskdep['nostamp']:
> + if error_nostamp:
> + bb.fatal("Task %s is marked nostamp, cannot
> invalidate this task" % taskname)
> + else:
> + bb.debug(1, "Task %s is marked nostamp, cannot
> invalidate this task" % taskname)
> + else:
> + logger.verbose("Invalidate task %s, %s", taskname,
> fn)
> + bb.parse.siggen.invalidate_task(taskname,
> self.dataCaches[mc], taskfn) +
> self.target_tids = []
> for (mc, target, task, fn) in self.targets:
>
> @@ -901,6 +933,8 @@ class RunQueueData:
>
> for tid in list(runall_tids):
> mark_active(tid,1)
> + if self.cooker.configuration.force:
> + invalidate_task(tid, False)
>
> for tid in list(self.runtaskentries.keys()):
> if tid not in runq_build:
> @@ -922,6 +956,8 @@ class RunQueueData:
>
> for tid in list(runonly_tids):
> mark_active(tid,1)
> + if self.cooker.configuration.force:
> + invalidate_task(tid, False)
>
> for tid in list(self.runtaskentries.keys()):
> if tid not in runq_build:
> @@ -1098,20 +1134,6 @@ class RunQueueData:
> continue
> self.runq_setscene_tids.append(tid)
>
> - def invalidate_task(tid, error_nostamp):
> - (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
> - taskdep = self.dataCaches[mc].task_deps[taskfn]
> - if fn + ":" + taskname not in taskData[mc].taskentries:
> - logger.warning("Task %s does not exist, invalidating
> this task will have no effect" % taskname)
> - if 'nostamp' in taskdep and taskname in
> taskdep['nostamp']:
> - if error_nostamp:
> - bb.fatal("Task %s is marked nostamp, cannot
> invalidate this task" % taskname)
> - else:
> - bb.debug(1, "Task %s is marked nostamp, cannot
> invalidate this task" % taskname)
> - else:
> - logger.verbose("Invalidate task %s, %s", taskname,
> fn)
> - bb.parse.siggen.invalidate_task(taskname,
> self.dataCaches[mc], taskfn) -
> self.init_progress_reporter.next_stage()
>
> # Invalidate task if force mode active
> @@ -1142,6 +1164,8 @@ class RunQueueData:
>
> self.init_progress_reporter.next_stage()
>
> + bb.parse.siggen.set_setscene_tasks(self.runq_setscene_tids)
> +
> # Iterate over the task list and call into the siggen code
> dealtwith = set()
> todeal = set(self.runtaskentries)
> @@ -1150,18 +1174,20 @@ class RunQueueData:
> if len(self.runtaskentries[tid].depends - dealtwith)
> == 0: dealtwith.add(tid)
> todeal.remove(tid)
> - procdep = []
> - for dep in self.runtaskentries[tid].depends:
> - procdep.append(fn_from_tid(dep) + "." +
> taskname_from_tid(dep))
> - (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
> - self.runtaskentries[tid].hash =
> bb.parse.siggen.get_taskhash(taskfn, taskname, procdep,
> self.dataCaches[mc])
> - task = self.runtaskentries[tid].task
> + self.prepare_task_hash(tid)
>
> bb.parse.siggen.writeout_file_checksum_cache()
>
> #self.dump_data()
> return len(self.runtaskentries)
>
> + def prepare_task_hash(self, tid):
> + procdep = []
> + for dep in self.runtaskentries[tid].depends:
> + procdep.append(dep)
> + self.runtaskentries[tid].hash =
> bb.parse.siggen.get_taskhash(tid, procdep,
> self.dataCaches[mc_from_tid(tid)])
> + self.runtaskentries[tid].unihash =
> bb.parse.siggen.get_unihash(tid) +
> def dump_data(self):
> """
> Dump some debug information on the internal data structures
> @@ -1187,7 +1213,6 @@ class RunQueue:
>
> self.stamppolicy = cfgData.getVar("BB_STAMP_POLICY") or
> "perfile" self.hashvalidate = cfgData.getVar("BB_HASHCHECK_FUNCTION")
> or None
> - self.setsceneverify =
> cfgData.getVar("BB_SETSCENE_VERIFY_FUNCTION2") or None
> self.depvalidate = cfgData.getVar("BB_SETSCENE_DEPVALID") or None
> self.state = runQueuePrepare
> @@ -1196,7 +1221,7 @@ class RunQueue:
> # Invoked at regular time intervals via the bitbake
> heartbeat event # while the build is running. We generate a unique
> name for the handler # here, just in case that there ever is more
> than one RunQueue instance,
> - # start the handler when reaching runQueueSceneRun, and stop
> it when
> + # start the handler when reaching runQueueSceneInit, and
> stop it when # done with the build.
> self.dm = monitordisk.diskMonitor(cfgData)
> self.dm_event_handler_name = '_bb_diskmonitor_' +
> str(id(self)) @@ -1213,28 +1238,23 @@ class RunQueue:
> if fakeroot:
> magic = magic + "beef"
> mcdata = self.cooker.databuilder.mcdata[mc]
> - fakerootcmd = mcdata.getVar("FAKEROOTCMD")
> + fakerootcmd = shlex.split(mcdata.getVar("FAKEROOTCMD"))
> fakerootenv = (mcdata.getVar("FAKEROOTBASEENV") or
> "").split() env = os.environ.copy()
> for key, value in (var.split('=') for var in
> fakerootenv): env[key] = value
> - worker = subprocess.Popen([fakerootcmd,
> "bitbake-worker", magic], stdout=subprocess.PIPE,
> stdin=subprocess.PIPE, env=env)
> + worker = subprocess.Popen(fakerootcmd +
> ["bitbake-worker", magic], stdout=subprocess.PIPE,
> stdin=subprocess.PIPE, env=env) else: worker =
> subprocess.Popen(["bitbake-worker", magic], stdout=subprocess.PIPE,
> stdin=subprocess.PIPE) bb.utils.nonblockingfd(worker.stdout)
> workerpipe = runQueuePipe(worker.stdout, None, self.cfgData, self,
> rqexec)
> - runqhash = {}
> - for tid in self.rqdata.runtaskentries:
> - runqhash[tid] = self.rqdata.runtaskentries[tid].hash
> -
> workerdata = {
> "taskdeps" : self.rqdata.dataCaches[mc].task_deps,
> "fakerootenv" : self.rqdata.dataCaches[mc].fakerootenv,
> "fakerootdirs" : self.rqdata.dataCaches[mc].fakerootdirs,
> "fakerootnoenv" :
> self.rqdata.dataCaches[mc].fakerootnoenv, "sigdata" :
> bb.parse.siggen.get_taskdata(),
> - "runq_hash" : runqhash,
> "logdefaultdebug" : bb.msg.loggerDefaultDebugLevel,
> "logdefaultverbose" : bb.msg.loggerDefaultVerbose,
> "logdefaultverboselogs" : bb.msg.loggerVerboseLogs,
> @@ -1243,6 +1263,7 @@ class RunQueue:
> "buildname" : self.cfgData.getVar("BUILDNAME"),
> "date" : self.cfgData.getVar("DATE"),
> "time" : self.cfgData.getVar("TIME"),
> + "hashservaddr" : self.cooker.hashservaddr,
> }
>
> worker.stdin.write(b"<cookerconfig>" +
> pickle.dumps(self.cooker.configuration) + b"</cookerconfig>") @@
> -1376,6 +1397,31 @@ class RunQueue: cache[tid] = iscurrent
> return iscurrent
>
> + def validate_hashes(self, tocheck, data, currentcount=0,
> siginfo=False):
> + valid = set()
> + if self.hashvalidate:
> + sq_data = {}
> + sq_data['hash'] = {}
> + sq_data['hashfn'] = {}
> + sq_data['unihash'] = {}
> + for tid in tocheck:
> + (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
> + sq_data['hash'][tid] =
> self.rqdata.runtaskentries[tid].hash
> + sq_data['hashfn'][tid] =
> self.rqdata.dataCaches[mc].hashfn[taskfn]
> + sq_data['unihash'][tid] =
> self.rqdata.runtaskentries[tid].unihash +
> + valid = self.validate_hash(sq_data, data, siginfo,
> currentcount) +
> + return valid
> +
> + def validate_hash(self, sq_data, d, siginfo, currentcount):
> + locs = {"sq_data" : sq_data, "d" : d, "siginfo" : siginfo,
> "currentcount" : currentcount} +
> + # Metadata has **kwargs so args can be added, sq_data can
> also gain new fields
> + call = self.hashvalidate + "(sq_data, d, siginfo=siginfo,
> currentcount=currentcount)" +
> + return bb.utils.better_eval(call, locs)
> +
> def _execute_runqueue(self):
> """
> Run the tasks in a queue prepared by rqdata.prepare()
> @@ -1386,7 +1432,6 @@ class RunQueue:
> retval = True
>
> if self.state is runQueuePrepare:
> - self.rqexe = RunQueueExecuteDummy(self)
> # NOTE: if you add, remove or significantly refactor the
> stages of this # process then you should recalculate the weightings
> here. This is quite # easy to do - just change the next line
> temporarily to pass debug=True as @@ -1400,15 +1445,23 @@ class
> RunQueue: self.state = runQueueComplete
> else:
> self.state = runQueueSceneInit
> - self.rqdata.init_progress_reporter.next_stage()
> -
> - # we are ready to run, emit dependency info to any
> UI or class which
> - # needs it
> - depgraph = self.cooker.buildDependTree(self,
> self.rqdata.taskData)
> - self.rqdata.init_progress_reporter.next_stage()
> - bb.event.fire(bb.event.DepTreeGenerated(depgraph),
> self.cooker.data)
> + bb.parse.siggen.save_unitaskhashes()
>
> if self.state is runQueueSceneInit:
> + self.rqdata.init_progress_reporter.next_stage()
> +
> + # we are ready to run, emit dependency info to any UI
> or class which
> + # needs it
> + depgraph = self.cooker.buildDependTree(self,
> self.rqdata.taskData)
> + self.rqdata.init_progress_reporter.next_stage()
> + bb.event.fire(bb.event.DepTreeGenerated(depgraph),
> self.cooker.data) +
> + if not self.dm_event_handler_registered:
> + res = bb.event.register(self.dm_event_handler_name,
> + lambda x:
> self.dm.check(self) if self.state in [runQueueRunning,
> runQueueCleanUp] else False,
> +
> ('bb.event.HeartbeatEvent',))
> + self.dm_event_handler_registered = True
> +
> dump = self.cooker.configuration.dump_signatures
> if dump:
> self.rqdata.init_progress_reporter.finish()
> @@ -1418,29 +1471,23 @@ class RunQueue:
> if 'printdiff' in dump:
> self.write_diffscenetasks(invalidtasks)
> self.state = runQueueComplete
> - else:
> - self.rqdata.init_progress_reporter.next_stage()
> - self.start_worker()
> - self.rqdata.init_progress_reporter.next_stage()
> - self.rqexe = RunQueueExecuteScenequeue(self)
> -
> - if self.state is runQueueSceneRun:
> - if not self.dm_event_handler_registered:
> - res = bb.event.register(self.dm_event_handler_name,
> - lambda x:
> self.dm.check(self) if self.state in [runQueueSceneRun,
> runQueueRunning, runQueueCleanUp] else False,
> -
> ('bb.event.HeartbeatEvent',))
> - self.dm_event_handler_registered = True
> - retval = self.rqexe.execute()
>
> - if self.state is runQueueRunInit:
> - if self.cooker.configuration.setsceneonly:
> - self.state = runQueueComplete
> - else:
> - # Just in case we didn't setscene
> - self.rqdata.init_progress_reporter.finish()
> - logger.info("Executing RunQueue Tasks")
> - self.rqexe = RunQueueExecuteTasks(self)
> - self.state = runQueueRunning
> + if self.state is runQueueSceneInit:
> + self.rqdata.init_progress_reporter.next_stage()
> + self.start_worker()
> + self.rqdata.init_progress_reporter.next_stage()
> + self.rqexe = RunQueueExecute(self)
> +
> + # If we don't have any setscene functions, skip execution
> + if len(self.rqdata.runq_setscene_tids) == 0:
> + logger.info('No setscene tasks')
> + for tid in self.rqdata.runtaskentries:
> + if len(self.rqdata.runtaskentries[tid].depends)
> == 0:
> + self.rqexe.setbuildable(tid)
> + self.rqexe.tasks_notcovered.add(tid)
> + self.rqexe.sqdone = True
> + logger.info('Executing Tasks')
> + self.state = runQueueRunning
>
> if self.state is runQueueRunning:
> retval = self.rqexe.execute()
> @@ -1455,12 +1502,14 @@ class RunQueue:
> self.dm_event_handler_registered = False
>
> if build_done and self.rqexe:
> + bb.parse.siggen.save_unitaskhashes()
> self.teardown_workers()
> - if self.rqexe.stats.failed:
> - logger.info("Tasks Summary: Attempted %d tasks of
> which %d didn't need to be rerun and %d failed.",
> self.rqexe.stats.completed + self.rqexe.stats.failed,
> self.rqexe.stats.skipped, self.rqexe.stats.failed)
> - else:
> - # Let's avoid the word "failed" if nothing actually
> did
> - logger.info("Tasks Summary: Attempted %d tasks of
> which %d didn't need to be rerun and all succeeded.",
> self.rqexe.stats.completed, self.rqexe.stats.skipped)
> + if self.rqexe:
> + if self.rqexe.stats.failed:
> + logger.info("Tasks Summary: Attempted %d tasks
> of which %d didn't need to be rerun and %d failed.",
> self.rqexe.stats.completed + self.rqexe.stats.failed,
> self.rqexe.stats.skipped, self.rqexe.stats.failed)
> + else:
> + # Let's avoid the word "failed" if nothing
> actually did
> + logger.info("Tasks Summary: Attempted %d tasks
> of which %d didn't need to be rerun and all succeeded.",
> self.rqexe.stats.completed, self.rqexe.stats.skipped) if self.state
> is runQueueFailed: raise
> bb.runqueue.TaskFailure(self.rqexe.failed_tids) @@ -1543,15 +1592,8
> @@ class RunQueue:
> def print_diffscenetasks(self):
>
> - valid = []
> - sq_hash = []
> - sq_hashfn = []
> - sq_fn = []
> - sq_taskname = []
> - sq_task = []
> noexec = []
> - stamppresent = []
> - valid_new = set()
> + tocheck = set()
>
> for tid in self.rqdata.runtaskentries:
> (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
> @@ -1561,21 +1603,9 @@ class RunQueue:
> noexec.append(tid)
> continue
>
> - sq_fn.append(fn)
> -
> sq_hashfn.append(self.rqdata.dataCaches[mc].hashfn[taskfn])
> - sq_hash.append(self.rqdata.runtaskentries[tid].hash)
> - sq_taskname.append(taskname)
> - sq_task.append(tid)
> - locs = { "sq_fn" : sq_fn, "sq_task" : sq_taskname,
> "sq_hash" : sq_hash, "sq_hashfn" : sq_hashfn, "d" : self.cooker.data }
> - try:
> - call = self.hashvalidate + "(sq_fn, sq_task, sq_hash,
> sq_hashfn, d, siginfo=True)"
> - valid = bb.utils.better_eval(call, locs)
> - # Handle version with no siginfo parameter
> - except TypeError:
> - call = self.hashvalidate + "(sq_fn, sq_task, sq_hash,
> sq_hashfn, d)"
> - valid = bb.utils.better_eval(call, locs)
> - for v in valid:
> - valid_new.add(sq_task[v])
> + tocheck.add(tid)
> +
> + valid_new = self.validate_hashes(tocheck, self.cooker.data,
> 0, True)
> # Tasks which are both setscene and noexec never care about
> dependencies # We therefore find tasks which are setscene and noexec
> and mark their @@ -1634,7 +1664,7 @@ class RunQueue:
> recout = []
> if len(hashfiles) == 2:
> out2 = bb.siggen.compare_sigfiles(hashfiles[hash1],
> hashfiles[hash2], recursecb)
> - recout.extend(list(' ' + l for l in out2))
> + recout.extend(list(' ' + l for l in out2))
> else:
> recout.append("Unable to find matching sigdata for
> %s with hashes %s or %s" % (key, hash1, hash2))
> @@ -1655,10 +1685,11 @@ class RunQueue:
> matches = {k : v for k, v in iter(matches.items()) if h
> not in k} if matches:
> latestmatch = sorted(matches.keys(), key=lambda f:
> matches[f])[-1]
> - prevh = __find_md5__.search(latestmatch).group(0)
> + prevh = __find_sha256__.search(latestmatch).group(0)
> output = bb.siggen.compare_sigfiles(latestmatch,
> match, recursecb) bb.plain("\nTask %s:%s couldn't be used from the
> cache because:\n We need hash %s, closest matching task was %s\n "
> % (pn, taskname, h, prevh) + '\n '.join(output)) +
> class RunQueueExecute:
>
> def __init__(self, rq):
> @@ -1670,6 +1701,13 @@ class RunQueueExecute:
> self.number_tasks =
> int(self.cfgData.getVar("BB_NUMBER_THREADS") or 1) self.scheduler =
> self.cfgData.getVar("BB_SCHEDULER") or "speed"
> + self.sq_buildable = set()
> + self.sq_running = set()
> + self.sq_live = set()
> +
> + self.updated_taskhash_queue = []
> + self.pending_migrations = set()
> +
> self.runq_buildable = set()
> self.runq_running = set()
> self.runq_complete = set()
> @@ -1677,9 +1715,17 @@ class RunQueueExecute:
> self.build_stamps = {}
> self.build_stamps2 = []
> self.failed_tids = []
> + self.sq_deferred = {}
>
> self.stampcache = {}
>
> + self.holdoff_tasks = set()
> + self.holdoff_need_update = True
> + self.sqdone = False
> +
> + self.stats = RunQueueStats(len(self.rqdata.runtaskentries))
> + self.sq_stats =
> RunQueueStats(len(self.rqdata.runq_setscene_tids)) +
> for mc in rq.worker:
> rq.worker[mc].pipe.setrunqueueexec(self)
> for mc in rq.fakeworker:
> @@ -1688,6 +1734,34 @@ class RunQueueExecute:
> if self.number_tasks <= 0:
> bb.fatal("Invalid BB_NUMBER_THREADS %s" %
> self.number_tasks)
> + # List of setscene tasks which we've covered
> + self.scenequeue_covered = set()
> + # List of tasks which are covered (including setscene ones)
> + self.tasks_covered = set()
> + self.tasks_scenequeue_done = set()
> + self.scenequeue_notcovered = set()
> + self.tasks_notcovered = set()
> + self.scenequeue_notneeded = set()
> +
> + # We can't skip specified target tasks which aren't setscene
> tasks
> + self.cantskip = set(self.rqdata.target_tids)
> +
> self.cantskip.difference_update(self.rqdata.runq_setscene_tids)
> + self.cantskip.intersection_update(self.rqdata.runtaskentries)
> +
> + schedulers = self.get_schedulers()
> + for scheduler in schedulers:
> + if self.scheduler == scheduler.name:
> + self.sched = scheduler(self, self.rqdata)
> + logger.debug(1, "Using runqueue scheduler '%s'",
> scheduler.name)
> + break
> + else:
> + bb.fatal("Invalid scheduler '%s'. Available schedulers:
> %s" %
> + (self.scheduler, ", ".join(obj.name for obj in
> schedulers))) +
> + #if len(self.rqdata.runq_setscene_tids) > 0:
> + self.sqdata = SQData()
> + build_scenequeue_data(self.sqdata, self.rqdata, self.rq,
> self.cooker, self.stampcache, self) +
> def runqueue_process_waitpid(self, task, status):
>
> # self.build_stamps[pid] may not exist when use shared work
> directory. @@ -1695,10 +1769,17 @@ class RunQueueExecute:
> self.build_stamps2.remove(self.build_stamps[task])
> del self.build_stamps[task]
>
> - if status != 0:
> - self.task_fail(task, status)
> + if task in self.sq_live:
> + if status != 0:
> + self.sq_task_fail(task, status)
> + else:
> + self.sq_task_complete(task)
> + self.sq_live.remove(task)
> else:
> - self.task_complete(task)
> + if status != 0:
> + self.task_fail(task, status)
> + else:
> + self.task_complete(task)
> return True
>
> def finish_now(self):
> @@ -1727,8 +1808,9 @@ class RunQueueExecute:
> def finish(self):
> self.rq.state = runQueueCleanUp
>
> - if self.stats.active > 0:
> - bb.event.fire(runQueueExitWait(self.stats.active),
> self.cfgData)
> + active = self.stats.active + self.sq_stats.active
> + if active > 0:
> + bb.event.fire(runQueueExitWait(active), self.cfgData)
> self.rq.read_workers()
> return self.rq.active_fds()
>
> @@ -1739,10 +1821,14 @@ class RunQueueExecute:
> self.rq.state = runQueueComplete
> return True
>
> - def check_dependencies(self, task, taskdeps, setscene = False):
> + # Used by setscene only
> + def check_dependencies(self, task, taskdeps):
> if not self.rq.depvalidate:
> return False
>
> + # Must not edit parent data
> + taskdeps = set(taskdeps)
> +
> taskdata = {}
> taskdeps.add(task)
> for dep in taskdeps:
> @@ -1755,121 +1841,10 @@ class RunQueueExecute:
> return valid
>
> def can_start_task(self):
> - can_start = self.stats.active < self.number_tasks
> + active = self.stats.active + self.sq_stats.active
> + can_start = active < self.number_tasks
> return can_start
>
> -class RunQueueExecuteDummy(RunQueueExecute):
> - def __init__(self, rq):
> - self.rq = rq
> - self.stats = RunQueueStats(0)
> -
> - def finish(self):
> - self.rq.state = runQueueComplete
> - return
> -
> -class RunQueueExecuteTasks(RunQueueExecute):
> - def __init__(self, rq):
> - RunQueueExecute.__init__(self, rq)
> -
> - self.stats = RunQueueStats(len(self.rqdata.runtaskentries))
> -
> - self.stampcache = {}
> -
> - initial_covered = self.rq.scenequeue_covered.copy()
> -
> - # Mark initial buildable tasks
> - for tid in self.rqdata.runtaskentries:
> - if len(self.rqdata.runtaskentries[tid].depends) == 0:
> - self.runq_buildable.add(tid)
> - if len(self.rqdata.runtaskentries[tid].revdeps) > 0 and
> self.rqdata.runtaskentries[tid].revdeps.issubset(self.rq.scenequeue_covered):
> - self.rq.scenequeue_covered.add(tid)
> -
> - found = True
> - while found:
> - found = False
> - for tid in self.rqdata.runtaskentries:
> - if tid in self.rq.scenequeue_covered:
> - continue
> - logger.debug(1, 'Considering %s: %s' % (tid,
> str(self.rqdata.runtaskentries[tid].revdeps))) -
> - if len(self.rqdata.runtaskentries[tid].revdeps) > 0
> and
> self.rqdata.runtaskentries[tid].revdeps.issubset(self.rq.scenequeue_covered):
> - if tid in self.rq.scenequeue_notcovered:
> - continue
> - found = True
> - self.rq.scenequeue_covered.add(tid)
> -
> - logger.debug(1, 'Skip list (pre setsceneverify) %s',
> sorted(self.rq.scenequeue_covered)) -
> - # Allow the metadata to elect for setscene tasks to run
> anyway
> - covered_remove = set()
> - if self.rq.setsceneverify:
> - invalidtasks = []
> - tasknames = {}
> - fns = {}
> - for tid in self.rqdata.runtaskentries:
> - (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
> - taskdep =
> self.rqdata.dataCaches[mc].task_deps[taskfn]
> - fns[tid] = taskfn
> - tasknames[tid] = taskname
> - if 'noexec' in taskdep and taskname in
> taskdep['noexec']:
> - continue
> - if self.rq.check_stamp_task(tid, taskname +
> "_setscene", cache=self.stampcache):
> - logger.debug(2, 'Setscene stamp current for task
> %s', tid)
> - continue
> - if self.rq.check_stamp_task(tid, taskname, recurse =
> True, cache=self.stampcache):
> - logger.debug(2, 'Normal stamp current for task
> %s', tid)
> - continue
> - invalidtasks.append(tid)
> -
> - call = self.rq.setsceneverify + "(covered, tasknames,
> fns, d, invalidtasks=invalidtasks)"
> - locs = { "covered" : self.rq.scenequeue_covered,
> "tasknames" : tasknames, "fns" : fns, "d" : self.cooker.data,
> "invalidtasks" : invalidtasks }
> - covered_remove = bb.utils.better_eval(call, locs)
> -
> - def removecoveredtask(tid):
> - (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
> - taskname = taskname + '_setscene'
> - bb.build.del_stamp(taskname, self.rqdata.dataCaches[mc],
> taskfn)
> - self.rq.scenequeue_covered.remove(tid)
> -
> - toremove = covered_remove | self.rq.scenequeue_notcovered
> - for task in toremove:
> - logger.debug(1, 'Not skipping task %s due to
> setsceneverify', task)
> - while toremove:
> - covered_remove = []
> - for task in toremove:
> - if task in self.rq.scenequeue_covered:
> - removecoveredtask(task)
> - for deptask in
> self.rqdata.runtaskentries[task].depends:
> - if deptask not in self.rq.scenequeue_covered:
> - continue
> - if deptask in toremove or deptask in
> covered_remove or deptask in initial_covered:
> - continue
> - logger.debug(1, 'Task %s depends on task %s so
> not skipping' % (task, deptask))
> - covered_remove.append(deptask)
> - toremove = covered_remove
> -
> - logger.debug(1, 'Full skip list %s',
> self.rq.scenequeue_covered) -
> -
> - for mc in self.rqdata.dataCaches:
> - target_pairs = []
> - for tid in self.rqdata.target_tids:
> - (tidmc, fn, taskname, _) = split_tid_mcfn(tid)
> - if tidmc == mc:
> - target_pairs.append((fn, taskname))
> -
> - event.fire(bb.event.StampUpdate(target_pairs,
> self.rqdata.dataCaches[mc].stamp), self.cfgData) -
> - schedulers = self.get_schedulers()
> - for scheduler in schedulers:
> - if self.scheduler == scheduler.name:
> - self.sched = scheduler(self, self.rqdata)
> - logger.debug(1, "Using runqueue scheduler '%s'",
> scheduler.name)
> - break
> - else:
> - bb.fatal("Invalid scheduler '%s'. Available schedulers:
> %s" %
> - (self.scheduler, ", ".join(obj.name for obj in
> schedulers))) -
> def get_schedulers(self):
> schedulers = set(obj for obj in globals().values()
> if type(obj) is type and
> @@ -1941,67 +1916,172 @@ class RunQueueExecuteTasks(RunQueueExecute):
> self.stats.taskSkipped()
> self.stats.taskCompleted()
>
> + def summarise_scenequeue_errors(self):
> + err = False
> + if not self.sqdone:
> + logger.debug(1, 'We could skip tasks %s',
> "\n".join(sorted(self.scenequeue_covered)))
> + completeevent = sceneQueueComplete(self.sq_stats,
> self.rq)
> + bb.event.fire(completeevent, self.cfgData)
> + if self.sq_deferred:
> + logger.error("Scenequeue had deferred entries: %s" %
> pprint.pformat(self.sq_deferred))
> + err = True
> + if self.updated_taskhash_queue:
> + logger.error("Scenequeue had unprocessed changed
> taskhash entries: %s" % pprint.pformat(self.updated_taskhash_queue))
> + err = True
> + if self.holdoff_tasks:
> + logger.error("Scenequeue had holdoff tasks: %s" %
> pprint.pformat(self.holdoff_tasks))
> + err = True
> +
> + for tid in self.rqdata.runq_setscene_tids:
> + if tid not in self.scenequeue_covered and tid not in
> self.scenequeue_notcovered:
> + err = True
> + logger.error("Setscene Task %s was never marked as
> covered or not covered" % tid)
> + if tid not in self.sq_buildable:
> + err = True
> + logger.error("Setscene Task %s was never marked as
> buildable" % tid)
> + if tid not in self.sq_running:
> + err = True
> + logger.error("Setscene Task %s was never marked as
> running" % tid) +
> + for x in self.rqdata.runtaskentries:
> + if x not in self.tasks_covered and x not in
> self.tasks_notcovered:
> + logger.error("Task %s was never moved from the
> setscene queue" % x)
> + err = True
> + if x not in self.tasks_scenequeue_done:
> + logger.error("Task %s was never processed by the
> setscene code" % x)
> + err = True
> + if len(self.rqdata.runtaskentries[x].depends) == 0 and x
> not in self.runq_buildable:
> + logger.error("Task %s was never marked as buildable
> by the setscene code" % x)
> + err = True
> + return err
> +
> +
> def execute(self):
> """
> - Run the tasks in a queue prepared by rqdata.prepare()
> + Run the tasks in a queue prepared by prepare_runqueue
> """
>
> - if self.rqdata.setscenewhitelist is not None and not
> self.rqdata.setscenewhitelist_checked:
> - self.rqdata.setscenewhitelist_checked = True
> -
> - # Check tasks that are going to run against the whitelist
> - def check_norun_task(tid, showerror=False):
> - (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
> - # Ignore covered tasks
> - if tid in self.rq.scenequeue_covered:
> - return False
> - # Ignore stamped tasks
> - if self.rq.check_stamp_task(tid, taskname,
> cache=self.stampcache):
> - return False
> - # Ignore noexec tasks
> - taskdep =
> self.rqdata.dataCaches[mc].task_deps[taskfn]
> - if 'noexec' in taskdep and taskname in
> taskdep['noexec']:
> - return False
> + self.rq.read_workers()
> + self.process_possible_migrations()
>
> - pn = self.rqdata.dataCaches[mc].pkg_fn[taskfn]
> - if not check_setscene_enforce_whitelist(pn,
> taskname, self.rqdata.setscenewhitelist):
> - if showerror:
> - if tid in self.rqdata.runq_setscene_tids:
> - logger.error('Task %s.%s attempted to
> execute unexpectedly and should have been setscened' % (pn, taskname))
> + task = None
> + if not self.sqdone and self.can_start_task():
> + # Find the next setscene to run
> + for nexttask in sorted(self.rqdata.runq_setscene_tids):
> + if nexttask in self.sq_buildable and nexttask not in
> self.sq_running and self.sqdata.stamps[nexttask] not in
> self.build_stamps.values():
> + if nexttask not in self.sqdata.unskippable and
> len(self.sqdata.sq_revdeps[nexttask]) > 0 and
> self.sqdata.sq_revdeps[nexttask].issubset(self.scenequeue_covered)
> and self.check_dependencies(nexttask,
> self.sqdata.sq_revdeps[nexttask]):
> + if nexttask not in self.rqdata.target_tids:
> + logger.debug(2, "Skipping setscene for
> task %s" % nexttask)
> + self.sq_task_skip(nexttask)
> + self.scenequeue_notneeded.add(nexttask)
> + if nexttask in self.sq_deferred:
> + del self.sq_deferred[nexttask]
> + return True
> + # If covered tasks are running, need to wait for
> them to complete
> + for t in self.sqdata.sq_covered_tasks[nexttask]:
> + if t in self.runq_running and t not in
> self.runq_complete:
> + continue
> + if nexttask in self.sq_deferred:
> + if self.sq_deferred[nexttask] not in
> self.runq_complete:
> + continue
> + logger.debug(1, "Task %s no longer deferred"
> % nexttask)
> + del self.sq_deferred[nexttask]
> + valid =
> self.rq.validate_hashes(set([nexttask]), self.cooker.data, 0, False)
> + if not valid:
> + logger.debug(1, "%s didn't become valid,
> skipping setscene" % nexttask)
> + self.sq_task_failoutright(nexttask)
> + return True
> else:
> - logger.error('Task %s.%s attempted to
> execute unexpectedly' % (pn, taskname))
> - return True
> - return False
> - # Look to see if any tasks that we think shouldn't run
> are going to
> - unexpected = False
> - for tid in self.rqdata.runtaskentries:
> - if check_norun_task(tid):
> - unexpected = True
> + self.sqdata.outrightfail.remove(nexttask)
> + if nexttask in self.sqdata.outrightfail:
> + logger.debug(2, 'No package found, so
> skipping setscene task %s', nexttask)
> + self.sq_task_failoutright(nexttask)
> + return True
> + if nexttask in self.sqdata.unskippable:
> + logger.debug(2, "Setscene task %s is
> unskippable" % nexttask)
> + task = nexttask
> break
> - if unexpected:
> - # Run through the tasks in the rough order they'd
> have executed and print errors
> - # (since the order can be useful - usually missing
> sstate for the last few tasks
> - # is the cause of the problem)
> - task = self.sched.next()
> - while task is not None:
> - check_norun_task(task, showerror=True)
> - self.task_skip(task, 'Setscene enforcement
> check')
> - task = self.sched.next()
> + if task is not None:
> + (mc, fn, taskname, taskfn) = split_tid_mcfn(task)
> + taskname = taskname + "_setscene"
> + if self.rq.check_stamp_task(task,
> taskname_from_tid(task), recurse = True, cache=self.stampcache):
> + logger.debug(2, 'Stamp for underlying task %s is
> current, so skipping setscene variant', task)
> + self.sq_task_failoutright(task)
> + return True
>
> - self.rq.state = runQueueCleanUp
> + if self.cooker.configuration.force:
> + if task in self.rqdata.target_tids:
> + self.sq_task_failoutright(task)
> + return True
> +
> + if self.rq.check_stamp_task(task, taskname,
> cache=self.stampcache):
> + logger.debug(2, 'Setscene stamp current task %s, so
> skip it and its dependencies', task)
> + self.sq_task_skip(task)
> return True
>
> - self.rq.read_workers()
> + if self.cooker.configuration.skipsetscene:
> + logger.debug(2, 'No setscene tasks should be
> executed. Skipping %s', task)
> + self.sq_task_failoutright(task)
> + return True
>
> - if self.stats.total == 0:
> - # nothing to do
> - self.rq.state = runQueueCleanUp
> + startevent = sceneQueueTaskStarted(task, self.sq_stats,
> self.rq)
> + bb.event.fire(startevent, self.cfgData)
> +
> + taskdepdata = self.sq_build_taskdepdata(task)
> +
> + taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn]
> + taskhash = self.rqdata.get_task_hash(task)
> + unihash = self.rqdata.get_task_unihash(task)
> + if 'fakeroot' in taskdep and taskname in
> taskdep['fakeroot'] and not self.cooker.configuration.dry_run:
> + if not mc in self.rq.fakeworker:
> + self.rq.start_fakeworker(self, mc)
> +
> self.rq.fakeworker[mc].process.stdin.write(b"<runtask>" +
> pickle.dumps((taskfn, task, taskname, taskhash, unihash, True,
> self.cooker.collection.get_file_appends(taskfn), taskdepdata, False))
> + b"</runtask>")
> + self.rq.fakeworker[mc].process.stdin.flush()
> + else:
> + self.rq.worker[mc].process.stdin.write(b"<runtask>"
> + pickle.dumps((taskfn, task, taskname, taskhash, unihash, True,
> self.cooker.collection.get_file_appends(taskfn), taskdepdata, False))
> + b"</runtask>")
> + self.rq.worker[mc].process.stdin.flush()
>
> - task = self.sched.next()
> + self.build_stamps[task] = bb.build.stampfile(taskname,
> self.rqdata.dataCaches[mc], taskfn, noextra=True)
> + self.build_stamps2.append(self.build_stamps[task])
> + self.sq_running.add(task)
> + self.sq_live.add(task)
> + self.sq_stats.taskActive()
> + if self.can_start_task():
> + return True
> +
> + self.update_holdofftasks()
> +
> + if not self.sq_live and not self.sqdone and not
> self.sq_deferred and not self.updated_taskhash_queue and not
> self.holdoff_tasks:
> + logger.info("Setscene tasks completed")
> +
> + err = self.summarise_scenequeue_errors()
> + if err:
> + self.rq.state = runQueueFailed
> + return True
> +
> + if self.cooker.configuration.setsceneonly:
> + self.rq.state = runQueueComplete
> + return True
> + self.sqdone = True
> +
> + if self.stats.total == 0:
> + # nothing to do
> + self.rq.state = runQueueComplete
> + return True
> +
> + if self.cooker.configuration.setsceneonly:
> + task = None
> + else:
> + task = self.sched.next()
> if task is not None:
> (mc, fn, taskname, taskfn) = split_tid_mcfn(task)
>
> - if task in self.rq.scenequeue_covered:
> + if self.rqdata.setscenewhitelist is not None:
> + if self.check_setscenewhitelist(task):
> + self.task_fail(task, "setscene whitelist")
> + return True
> +
> + if task in self.tasks_covered:
> logger.debug(2, "Setscene covered task %s", task)
> self.task_skip(task, "covered")
> return True
> @@ -2030,6 +2110,8 @@ class RunQueueExecuteTasks(RunQueueExecute):
> taskdepdata = self.build_taskdepdata(task)
>
> taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn]
> + taskhash = self.rqdata.get_task_hash(task)
> + unihash = self.rqdata.get_task_unihash(task)
> if 'fakeroot' in taskdep and taskname in
> taskdep['fakeroot'] and not (self.cooker.configuration.dry_run or
> self.rqdata.setscene_enforce): if not mc in self.rq.fakeworker: try:
> @@ -2039,10 +2121,10 @@ class RunQueueExecuteTasks(RunQueueExecute):
> self.rq.state = runQueueFailed
> self.stats.taskFailed()
> return True
> -
> self.rq.fakeworker[mc].process.stdin.write(b"<runtask>" +
> pickle.dumps((taskfn, task, taskname, False,
> self.cooker.collection.get_file_appends(taskfn), taskdepdata,
> self.rqdata.setscene_enforce)) + b"</runtask>")
> +
> self.rq.fakeworker[mc].process.stdin.write(b"<runtask>" +
> pickle.dumps((taskfn, task, taskname, taskhash, unihash, False,
> self.cooker.collection.get_file_appends(taskfn), taskdepdata,
> self.rqdata.setscene_enforce)) + b"</runtask>")
> self.rq.fakeworker[mc].process.stdin.flush() else:
> - self.rq.worker[mc].process.stdin.write(b"<runtask>"
> + pickle.dumps((taskfn, task, taskname, False,
> self.cooker.collection.get_file_appends(taskfn), taskdepdata,
> self.rqdata.setscene_enforce)) + b"</runtask>")
> + self.rq.worker[mc].process.stdin.write(b"<runtask>"
> + pickle.dumps((taskfn, task, taskname, taskhash, unihash, False,
> self.cooker.collection.get_file_appends(taskfn), taskdepdata,
> self.rqdata.setscene_enforce)) + b"</runtask>")
> self.rq.worker[mc].process.stdin.flush() self.build_stamps[task] =
> bb.build.stampfile(taskname, self.rqdata.dataCaches[mc], taskfn,
> noextra=True) @@ -2052,30 +2134,58 @@ class
> RunQueueExecuteTasks(RunQueueExecute): if self.can_start_task():
> return True
> - if self.stats.active > 0:
> + if self.stats.active > 0 or self.sq_stats.active > 0:
> self.rq.read_workers()
> return self.rq.active_fds()
>
> + # No more tasks can be run. If we have deferred setscene
> tasks we should run them.
> + if self.sq_deferred:
> + tid =
> self.sq_deferred.pop(list(self.sq_deferred.keys())[0])
> + logger.warning("Runqeueue deadlocked on deferred tasks,
> forcing task %s" % tid)
> + self.sq_task_failoutright(tid)
> + return True
> +
> if len(self.failed_tids) != 0:
> self.rq.state = runQueueFailed
> return True
>
> # Sanity Checks
> + err = self.summarise_scenequeue_errors()
> for task in self.rqdata.runtaskentries:
> if task not in self.runq_buildable:
> logger.error("Task %s never buildable!", task)
> - if task not in self.runq_running:
> + err = True
> + elif task not in self.runq_running:
> logger.error("Task %s never ran!", task)
> - if task not in self.runq_complete:
> + err = True
> + elif task not in self.runq_complete:
> logger.error("Task %s never completed!", task)
> - self.rq.state = runQueueComplete
> + err = True
> +
> + if err:
> + self.rq.state = runQueueFailed
> + else:
> + self.rq.state = runQueueComplete
>
> return True
>
> + def filtermcdeps(self, task, mc, deps):
> + ret = set()
> + for dep in deps:
> + thismc = mc_from_tid(dep)
> + if thismc != mc:
> + continue
> + ret.add(dep)
> + return ret
> +
> + # We filter out multiconfig dependencies from taskdepdata we
> pass to the tasks
> + # as most code can't handle them
> def build_taskdepdata(self, task):
> taskdepdata = {}
> - next = self.rqdata.runtaskentries[task].depends
> + mc = mc_from_tid(task)
> + next = self.rqdata.runtaskentries[task].depends.copy()
> next.add(task)
> + next = self.filtermcdeps(task, mc, next)
> while next:
> additional = []
> for revdep in next:
> @@ -2084,7 +2194,9 @@ class RunQueueExecuteTasks(RunQueueExecute):
> deps = self.rqdata.runtaskentries[revdep].depends
> provides =
> self.rqdata.dataCaches[mc].fn_provides[taskfn] taskhash =
> self.rqdata.runtaskentries[revdep].hash
> - taskdepdata[revdep] = [pn, taskname, fn, deps,
> provides, taskhash]
> + unihash = self.rqdata.runtaskentries[revdep].unihash
> + deps = self.filtermcdeps(task, mc, deps)
> + taskdepdata[revdep] = [pn, taskname, fn, deps,
> provides, taskhash, unihash] for revdep2 in deps:
> if revdep2 not in taskdepdata:
> additional.append(revdep2)
> @@ -2093,269 +2205,197 @@ class RunQueueExecuteTasks(RunQueueExecute):
> #bb.note("Task %s: " % task + str(taskdepdata).replace("],
> ", "],\n")) return taskdepdata
>
> -class RunQueueExecuteScenequeue(RunQueueExecute):
> - def __init__(self, rq):
> - RunQueueExecute.__init__(self, rq)
> -
> - self.scenequeue_covered = set()
> - self.scenequeue_notcovered = set()
> - self.scenequeue_notneeded = set()
> + def update_holdofftasks(self):
>
> - # If we don't have any setscene functions, skip this step
> - if len(self.rqdata.runq_setscene_tids) == 0:
> - rq.scenequeue_covered = set()
> - rq.scenequeue_notcovered = set()
> - rq.state = runQueueRunInit
> + if not self.holdoff_need_update:
> return
>
> - self.stats =
> RunQueueStats(len(self.rqdata.runq_setscene_tids))
> + notcovered = set(self.scenequeue_notcovered)
> + notcovered |= self.cantskip
> + for tid in self.scenequeue_notcovered:
> + notcovered |= self.sqdata.sq_covered_tasks[tid]
> + notcovered |=
> self.sqdata.unskippable.difference(self.rqdata.runq_setscene_tids)
> + notcovered.intersection_update(self.tasks_scenequeue_done)
>
> - sq_revdeps = {}
> - sq_revdeps_new = {}
> - sq_revdeps_squash = {}
> - self.sq_harddeps = {}
> - self.stamps = {}
> -
> - # We need to construct a dependency graph for the setscene
> functions. Intermediate
> - # dependencies between the setscene tasks only complicate
> the code. This code
> - # therefore aims to collapse the huge runqueue dependency
> tree into a smaller one
> - # only containing the setscene functions.
> -
> - self.rqdata.init_progress_reporter.next_stage()
> -
> - # First process the chains up to the first setscene task.
> - endpoints = {}
> - for tid in self.rqdata.runtaskentries:
> - sq_revdeps[tid] =
> copy.copy(self.rqdata.runtaskentries[tid].revdeps)
> - sq_revdeps_new[tid] = set()
> - if (len(sq_revdeps[tid]) == 0) and tid not in
> self.rqdata.runq_setscene_tids:
> - #bb.warn("Added endpoint %s" % (tid))
> - endpoints[tid] = set()
> + covered = set(self.scenequeue_covered)
> + for tid in self.scenequeue_covered:
> + covered |= self.sqdata.sq_covered_tasks[tid]
> + covered.difference_update(notcovered)
> + covered.intersection_update(self.tasks_scenequeue_done)
>
> - self.rqdata.init_progress_reporter.next_stage()
> + for tid in notcovered | covered:
> + if len(self.rqdata.runtaskentries[tid].depends) == 0:
> + self.setbuildable(tid)
> + elif
> self.rqdata.runtaskentries[tid].depends.issubset(self.runq_complete):
> + self.setbuildable(tid)
>
> - # Secondly process the chains between setscene tasks.
> - for tid in self.rqdata.runq_setscene_tids:
> - #bb.warn("Added endpoint 2 %s" % (tid))
> - for dep in self.rqdata.runtaskentries[tid].depends:
> - if tid in sq_revdeps[dep]:
> - sq_revdeps[dep].remove(tid)
> - if dep not in endpoints:
> - endpoints[dep] = set()
> - #bb.warn(" Added endpoint 3 %s" % (dep))
> - endpoints[dep].add(tid)
> -
> - self.rqdata.init_progress_reporter.next_stage()
> -
> - def process_endpoints(endpoints):
> - newendpoints = {}
> - for point, task in endpoints.items():
> - tasks = set()
> - if task:
> - tasks |= task
> - if sq_revdeps_new[point]:
> - tasks |= sq_revdeps_new[point]
> - sq_revdeps_new[point] = set()
> - if point in self.rqdata.runq_setscene_tids:
> - sq_revdeps_new[point] = tasks
> - tasks = set()
> - continue
> - for dep in self.rqdata.runtaskentries[point].depends:
> - if point in sq_revdeps[dep]:
> - sq_revdeps[dep].remove(point)
> - if tasks:
> - sq_revdeps_new[dep] |= tasks
> - if len(sq_revdeps[dep]) == 0 and dep not in
> self.rqdata.runq_setscene_tids:
> - newendpoints[dep] = task
> - if len(newendpoints) != 0:
> - process_endpoints(newendpoints)
> -
> - process_endpoints(endpoints)
> -
> - self.rqdata.init_progress_reporter.next_stage()
> -
> - # Build a list of setscene tasks which are "unskippable"
> - # These are direct endpoints referenced by the build
> - endpoints2 = {}
> - sq_revdeps2 = {}
> - sq_revdeps_new2 = {}
> - def process_endpoints2(endpoints):
> - newendpoints = {}
> - for point, task in endpoints.items():
> - tasks = set([point])
> - if task:
> - tasks |= task
> - if sq_revdeps_new2[point]:
> - tasks |= sq_revdeps_new2[point]
> - sq_revdeps_new2[point] = set()
> - if point in self.rqdata.runq_setscene_tids:
> - sq_revdeps_new2[point] = tasks
> - for dep in self.rqdata.runtaskentries[point].depends:
> - if point in sq_revdeps2[dep]:
> - sq_revdeps2[dep].remove(point)
> - if tasks:
> - sq_revdeps_new2[dep] |= tasks
> - if (len(sq_revdeps2[dep]) == 0 or
> len(sq_revdeps_new2[dep]) != 0) and dep not in
> self.rqdata.runq_setscene_tids:
> - newendpoints[dep] = tasks
> - if len(newendpoints) != 0:
> - process_endpoints2(newendpoints)
> - for tid in self.rqdata.runtaskentries:
> - sq_revdeps2[tid] =
> copy.copy(self.rqdata.runtaskentries[tid].revdeps)
> - sq_revdeps_new2[tid] = set()
> - if (len(sq_revdeps2[tid]) == 0) and tid not in
> self.rqdata.runq_setscene_tids:
> - endpoints2[tid] = set()
> - process_endpoints2(endpoints2)
> - self.unskippable = []
> - for tid in self.rqdata.runq_setscene_tids:
> - if sq_revdeps_new2[tid]:
> - self.unskippable.append(tid)
> + self.tasks_covered = covered
> + self.tasks_notcovered = notcovered
>
> -
> self.rqdata.init_progress_reporter.next_stage(len(self.rqdata.runtaskentries))
> + self.holdoff_tasks = set()
>
> - for taskcounter, tid in
> enumerate(self.rqdata.runtaskentries):
> - if tid in self.rqdata.runq_setscene_tids:
> - deps = set()
> - for dep in sq_revdeps_new[tid]:
> - deps.add(dep)
> - sq_revdeps_squash[tid] = deps
> - elif len(sq_revdeps_new[tid]) != 0:
> - bb.msg.fatal("RunQueue", "Something went badly wrong
> during scenequeue generation, aborting. Please report this problem.")
> - self.rqdata.init_progress_reporter.update(taskcounter)
> -
> - self.rqdata.init_progress_reporter.next_stage()
> -
> - # Resolve setscene inter-task dependencies
> - # e.g. do_sometask_setscene[depends] =
> "targetname:do_someothertask_setscene"
> - # Note that anything explicitly depended upon will have its
> reverse dependencies removed to avoid circular dependencies for tid
> in self.rqdata.runq_setscene_tids:
> - (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
> - realtid = tid + "_setscene"
> - idepends =
> self.rqdata.taskData[mc].taskentries[realtid].idepends
> - self.stamps[tid] = bb.build.stampfile(taskname +
> "_setscene", self.rqdata.dataCaches[mc], taskfn, noextra=True)
> - for (depname, idependtask) in idepends:
> -
> - if depname not in
> self.rqdata.taskData[mc].build_targets:
> - continue
> + if tid not in self.scenequeue_covered and tid not in
> self.scenequeue_notcovered:
> + self.holdoff_tasks.add(tid)
>
> - depfn =
> self.rqdata.taskData[mc].build_targets[depname][0]
> - if depfn is None:
> - continue
> - deptid = depfn + ":" +
> idependtask.replace("_setscene", "")
> - if deptid not in self.rqdata.runtaskentries:
> - bb.msg.fatal("RunQueue", "Task %s depends
> upon non-existent task %s:%s" % (realtid, depfn, idependtask)) -
> - if not deptid in self.sq_harddeps:
> - self.sq_harddeps[deptid] = set()
> - self.sq_harddeps[deptid].add(tid)
> -
> - sq_revdeps_squash[tid].add(deptid)
> - # Have to zero this to avoid circular
> dependencies
> - sq_revdeps_squash[deptid] = set()
> -
> - self.rqdata.init_progress_reporter.next_stage()
> -
> - for task in self.sq_harddeps:
> - for dep in self.sq_harddeps[task]:
> - sq_revdeps_squash[dep].add(task)
> -
> - self.rqdata.init_progress_reporter.next_stage()
> -
> - #for tid in sq_revdeps_squash:
> - # for dep in sq_revdeps_squash[tid]:
> - # data = data + "\n %s" % dep
> - # bb.warn("Task %s_setscene: is %s " % (tid, data
> -
> - self.sq_deps = {}
> - self.sq_revdeps = sq_revdeps_squash
> - self.sq_revdeps2 = copy.deepcopy(self.sq_revdeps)
> -
> - for tid in self.sq_revdeps:
> - self.sq_deps[tid] = set()
> - for tid in self.sq_revdeps:
> - for dep in self.sq_revdeps[tid]:
> - self.sq_deps[dep].add(tid)
> -
> - self.rqdata.init_progress_reporter.next_stage()
> -
> - for tid in self.sq_revdeps:
> - if len(self.sq_revdeps[tid]) == 0:
> - self.runq_buildable.add(tid)
> -
> - self.rqdata.init_progress_reporter.finish()
> -
> - self.outrightfail = []
> - if self.rq.hashvalidate:
> - sq_hash = []
> - sq_hashfn = []
> - sq_fn = []
> - sq_taskname = []
> - sq_task = []
> - noexec = []
> - stamppresent = []
> - for tid in self.sq_revdeps:
> - (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
> + for tid in self.holdoff_tasks.copy():
> + for dep in self.sqdata.sq_covered_tasks[tid]:
> + if dep not in self.runq_complete:
> + self.holdoff_tasks.add(dep)
>
> - taskdep =
> self.rqdata.dataCaches[mc].task_deps[taskfn]
> + self.holdoff_need_update = False
>
> - if 'noexec' in taskdep and taskname in
> taskdep['noexec']:
> - noexec.append(tid)
> - self.task_skip(tid)
> - bb.build.make_stamp(taskname + "_setscene",
> self.rqdata.dataCaches[mc], taskfn)
> - continue
> + def process_possible_migrations(self):
>
> - if self.rq.check_stamp_task(tid, taskname +
> "_setscene", cache=self.stampcache):
> - logger.debug(2, 'Setscene stamp current for task
> %s', tid)
> - stamppresent.append(tid)
> - self.task_skip(tid)
> - continue
> + changed = set()
> + for tid, unihash in self.updated_taskhash_queue.copy():
> + if tid in self.runq_running and tid not in
> self.runq_complete:
> + continue
>
> - if self.rq.check_stamp_task(tid, taskname, recurse =
> True, cache=self.stampcache):
> - logger.debug(2, 'Normal stamp current for task
> %s', tid)
> - stamppresent.append(tid)
> - self.task_skip(tid)
> - continue
> + self.updated_taskhash_queue.remove((tid, unihash))
> +
> + if unihash != self.rqdata.runtaskentries[tid].unihash:
> + logger.info("Task %s unihash changed to %s" % (tid,
> unihash))
> + self.rqdata.runtaskentries[tid].unihash = unihash
> + bb.parse.siggen.set_unihash(tid, unihash)
> +
> + # Work out all tasks which depend on this one
> + total = set()
> + next = set(self.rqdata.runtaskentries[tid].revdeps)
> + while next:
> + current = next.copy()
> + total = total |next
> + next = set()
> + for ntid in current:
> + next |=
> self.rqdata.runtaskentries[ntid].revdeps
> + next.difference_update(total)
> +
> + # Now iterate those tasks in dependency order to
> regenerate their taskhash/unihash
> + done = set()
> + next = set(self.rqdata.runtaskentries[tid].revdeps)
> + while next:
> + current = next.copy()
> + next = set()
> + for tid in current:
> + if not
> self.rqdata.runtaskentries[tid].depends.isdisjoint(total):
> + continue
> + procdep = []
> + for dep in
> self.rqdata.runtaskentries[tid].depends:
> + procdep.append(dep)
> + orighash =
> self.rqdata.runtaskentries[tid].hash
> + self.rqdata.runtaskentries[tid].hash =
> bb.parse.siggen.get_taskhash(tid, procdep,
> self.rqdata.dataCaches[mc_from_tid(tid)])
> + origuni =
> self.rqdata.runtaskentries[tid].unihash
> + self.rqdata.runtaskentries[tid].unihash =
> bb.parse.siggen.get_unihash(tid)
> + logger.debug(1, "Task %s hash changes:
> %s->%s %s->%s" % (tid, orighash,
> self.rqdata.runtaskentries[tid].hash, origuni,
> self.rqdata.runtaskentries[tid].unihash))
> + next |=
> self.rqdata.runtaskentries[tid].revdeps
> + changed.add(tid)
> + total.remove(tid)
> + next.intersection_update(total)
> +
> + if changed:
> + for mc in self.rq.worker:
> +
> self.rq.worker[mc].process.stdin.write(b"<newtaskhashes>" +
> pickle.dumps(bb.parse.siggen.get_taskhashes()) + b"</newtaskhashes>")
> + for mc in self.rq.fakeworker:
> +
> self.rq.fakeworker[mc].process.stdin.write(b"<newtaskhashes>" +
> pickle.dumps(bb.parse.siggen.get_taskhashes()) + b"</newtaskhashes>")
> +
> + logger.debug(1, pprint.pformat("Tasks changed:\n%s" %
> (changed))) +
> + for tid in changed:
> + if tid not in self.rqdata.runq_setscene_tids:
> + continue
> + if tid in self.runq_running:
> + continue
> + if tid in self.scenequeue_covered:
> + # Potentially risky, should we report this hash as a
> match?
> + logger.info("Already covered setscene for %s so
> ignoring rehash" % (tid))
> + continue
> + if tid not in self.pending_migrations:
> + self.pending_migrations.add(tid)
> +
> + for tid in self.pending_migrations.copy():
> + valid = True
> + # Check no tasks this covers are running
> + for dep in self.sqdata.sq_covered_tasks[tid]:
> + if dep in self.runq_running and dep not in
> self.runq_complete:
> + logger.debug(2, "Task %s is running which blocks
> setscene for %s from running" % (dep, tid))
> + valid = False
> + break
> + if not valid:
> + continue
>
> - sq_fn.append(fn)
> -
> sq_hashfn.append(self.rqdata.dataCaches[mc].hashfn[taskfn])
> - sq_hash.append(self.rqdata.runtaskentries[tid].hash)
> - sq_taskname.append(taskname)
> - sq_task.append(tid)
> + self.pending_migrations.remove(tid)
> + changed = True
>
> -
> self.cooker.data.setVar("BB_SETSCENE_STAMPCURRENT_COUNT",
> len(stamppresent))
> + if tid in self.tasks_scenequeue_done:
> + self.tasks_scenequeue_done.remove(tid)
> + for dep in self.sqdata.sq_covered_tasks[tid]:
> + if dep not in self.runq_complete:
> + if dep in self.tasks_scenequeue_done and dep not
> in self.sqdata.unskippable:
> + self.tasks_scenequeue_done.remove(dep)
> +
> + if tid in self.sq_buildable:
> + self.sq_buildable.remove(tid)
> + if tid in self.sq_running:
> + self.sq_running.remove(tid)
> + if
> self.sqdata.sq_revdeps[tid].issubset(self.scenequeue_covered |
> self.scenequeue_notcovered):
> + if tid not in self.sq_buildable:
> + self.sq_buildable.add(tid)
> + if len(self.sqdata.sq_revdeps[tid]) == 0:
> + self.sq_buildable.add(tid)
> +
> + if tid in self.sqdata.outrightfail:
> + self.sqdata.outrightfail.remove(tid)
> + if tid in self.scenequeue_notcovered:
> + self.scenequeue_notcovered.remove(tid)
> + if tid in self.scenequeue_covered:
> + self.scenequeue_covered.remove(tid)
> + if tid in self.scenequeue_notneeded:
> + self.scenequeue_notneeded.remove(tid)
>
> - call = self.rq.hashvalidate + "(sq_fn, sq_task, sq_hash,
> sq_hashfn, d)"
> - locs = { "sq_fn" : sq_fn, "sq_task" : sq_taskname,
> "sq_hash" : sq_hash, "sq_hashfn" : sq_hashfn, "d" : self.cooker.data }
> - valid = bb.utils.better_eval(call, locs)
> + (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
> + self.sqdata.stamps[tid] = bb.build.stampfile(taskname +
> "_setscene", self.rqdata.dataCaches[mc], taskfn, noextra=True)
> - self.cooker.data.delVar("BB_SETSCENE_STAMPCURRENT_COUNT")
> + if tid in self.stampcache:
> + del self.stampcache[tid]
>
> - valid_new = stamppresent
> - for v in valid:
> - valid_new.append(sq_task[v])
> + if tid in self.build_stamps:
> + del self.build_stamps[tid]
>
> - for tid in self.sq_revdeps:
> - if tid not in valid_new and tid not in noexec:
> - logger.debug(2, 'No package found, so skipping
> setscene task %s', tid)
> - self.outrightfail.append(tid)
> + logger.info("Setscene task %s now valid and being rerun"
> % tid)
> + self.sqdone = False
> + update_scenequeue_data([tid], self.sqdata, self.rqdata,
> self.rq, self.cooker, self.stampcache, self)
> - logger.info('Executing SetScene Tasks')
> + if changed:
> + self.holdoff_need_update = True
>
> - self.rq.state = runQueueSceneRun
> + def scenequeue_updatecounters(self, task, fail=False):
>
> - def scenequeue_updatecounters(self, task, fail = False):
> - for dep in self.sq_deps[task]:
> - if fail and task in self.sq_harddeps and dep in
> self.sq_harddeps[task]:
> + for dep in sorted(self.sqdata.sq_deps[task]):
> + if fail and task in self.sqdata.sq_harddeps and dep in
> self.sqdata.sq_harddeps[task]: logger.debug(2, "%s was unavailable
> and is a hard dependency of %s so skipping" % (task, dep))
> - self.scenequeue_updatecounters(dep, fail)
> + self.sq_task_failoutright(dep)
> continue
> - if task not in self.sq_revdeps2[dep]:
> - # May already have been removed by the fail case
> above
> - continue
> - self.sq_revdeps2[dep].remove(task)
> - if len(self.sq_revdeps2[dep]) == 0:
> - self.runq_buildable.add(dep)
> + if
> self.sqdata.sq_revdeps[dep].issubset(self.scenequeue_covered |
> self.scenequeue_notcovered):
> + if dep not in self.sq_buildable:
> + self.sq_buildable.add(dep)
>
> - def task_completeoutright(self, task):
> + next = set([task])
> + while next:
> + new = set()
> + for t in sorted(next):
> + self.tasks_scenequeue_done.add(t)
> + # Look down the dependency chain for non-setscene
> things which this task depends on
> + # and mark as 'done'
> + for dep in self.rqdata.runtaskentries[t].depends:
> + if dep in self.rqdata.runq_setscene_tids or dep
> in self.tasks_scenequeue_done:
> + continue
> + if
> self.rqdata.runtaskentries[dep].revdeps.issubset(self.tasks_scenequeue_done):
> + new.add(dep)
> + next = new
> +
> + self.holdoff_need_update = True
> +
> + def sq_task_completeoutright(self, task):
> """
> Mark a task as completed
> Look at the reverse dependencies and mark any task with
> @@ -2366,7 +2406,7 @@ class
> RunQueueExecuteScenequeue(RunQueueExecute):
> self.scenequeue_covered.add(task) self.scenequeue_updatecounters(task)
>
> - def check_taskfail(self, task):
> + def sq_check_taskfail(self, task):
> if self.rqdata.setscenewhitelist is not None:
> realtask = task.split('_setscene')[0]
> (mc, fn, taskname, taskfn) = split_tid_mcfn(realtask)
> @@ -2375,130 +2415,34 @@ class
> RunQueueExecuteScenequeue(RunQueueExecute): logger.error('Task %s.%s
> failed' % (pn, taskname + "_setscene")) self.rq.state =
> runQueueCleanUp
> - def task_complete(self, task):
> - self.stats.taskCompleted()
> - bb.event.fire(sceneQueueTaskCompleted(task, self.stats,
> self.rq), self.cfgData)
> - self.task_completeoutright(task)
> + def sq_task_complete(self, task):
> + self.sq_stats.taskCompleted()
> + bb.event.fire(sceneQueueTaskCompleted(task, self.sq_stats,
> self.rq), self.cfgData)
> + self.sq_task_completeoutright(task)
>
> - def task_fail(self, task, result):
> - self.stats.taskFailed()
> - bb.event.fire(sceneQueueTaskFailed(task, self.stats, result,
> self), self.cfgData)
> + def sq_task_fail(self, task, result):
> + self.sq_stats.taskFailed()
> + bb.event.fire(sceneQueueTaskFailed(task, self.sq_stats,
> result, self), self.cfgData) self.scenequeue_notcovered.add(task)
> self.scenequeue_updatecounters(task, True)
> - self.check_taskfail(task)
> + self.sq_check_taskfail(task)
>
> - def task_failoutright(self, task):
> - self.runq_running.add(task)
> - self.runq_buildable.add(task)
> - self.stats.taskSkipped()
> - self.stats.taskCompleted()
> + def sq_task_failoutright(self, task):
> + self.sq_running.add(task)
> + self.sq_buildable.add(task)
> + self.sq_stats.taskSkipped()
> + self.sq_stats.taskCompleted()
> self.scenequeue_notcovered.add(task)
> self.scenequeue_updatecounters(task, True)
>
> - def task_skip(self, task):
> - self.runq_running.add(task)
> - self.runq_buildable.add(task)
> - self.task_completeoutright(task)
> - self.stats.taskSkipped()
> - self.stats.taskCompleted()
> -
> - def execute(self):
> - """
> - Run the tasks in a queue prepared by prepare_runqueue
> - """
> -
> - self.rq.read_workers()
> -
> - task = None
> - if self.can_start_task():
> - # Find the next setscene to run
> - for nexttask in self.rqdata.runq_setscene_tids:
> - if nexttask in self.runq_buildable and nexttask not
> in self.runq_running and self.stamps[nexttask] not in
> self.build_stamps.values():
> - if nexttask in self.unskippable:
> - logger.debug(2, "Setscene task %s is
> unskippable" % nexttask)
> - if nexttask not in self.unskippable and
> len(self.sq_revdeps[nexttask]) > 0 and
> self.sq_revdeps[nexttask].issubset(self.scenequeue_covered) and
> self.check_dependencies(nexttask, self.sq_revdeps[nexttask], True):
> - fn = fn_from_tid(nexttask)
> - foundtarget = False
> -
> - if nexttask in self.rqdata.target_tids:
> - foundtarget = True
> - if not foundtarget:
> - logger.debug(2, "Skipping setscene for
> task %s" % nexttask)
> - self.task_skip(nexttask)
> - self.scenequeue_notneeded.add(nexttask)
> - return True
> - if nexttask in self.outrightfail:
> - self.task_failoutright(nexttask)
> - return True
> - task = nexttask
> - break
> - if task is not None:
> - (mc, fn, taskname, taskfn) = split_tid_mcfn(task)
> - taskname = taskname + "_setscene"
> - if self.rq.check_stamp_task(task,
> taskname_from_tid(task), recurse = True, cache=self.stampcache):
> - logger.debug(2, 'Stamp for underlying task %s is
> current, so skipping setscene variant', task)
> - self.task_failoutright(task)
> - return True
> -
> - if self.cooker.configuration.force:
> - if task in self.rqdata.target_tids:
> - self.task_failoutright(task)
> - return True
> -
> - if self.rq.check_stamp_task(task, taskname,
> cache=self.stampcache):
> - logger.debug(2, 'Setscene stamp current task %s, so
> skip it and its dependencies', task)
> - self.task_skip(task)
> - return True
> -
> - startevent = sceneQueueTaskStarted(task, self.stats,
> self.rq)
> - bb.event.fire(startevent, self.cfgData)
> + def sq_task_skip(self, task):
> + self.sq_running.add(task)
> + self.sq_buildable.add(task)
> + self.sq_task_completeoutright(task)
> + self.sq_stats.taskSkipped()
> + self.sq_stats.taskCompleted()
>
> - taskdepdata = self.build_taskdepdata(task)
> -
> - taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn]
> - if 'fakeroot' in taskdep and taskname in
> taskdep['fakeroot'] and not self.cooker.configuration.dry_run:
> - if not mc in self.rq.fakeworker:
> - self.rq.start_fakeworker(self, mc)
> -
> self.rq.fakeworker[mc].process.stdin.write(b"<runtask>" +
> pickle.dumps((taskfn, task, taskname, True,
> self.cooker.collection.get_file_appends(taskfn), taskdepdata, False))
> + b"</runtask>")
> - self.rq.fakeworker[mc].process.stdin.flush()
> - else:
> - self.rq.worker[mc].process.stdin.write(b"<runtask>"
> + pickle.dumps((taskfn, task, taskname, True,
> self.cooker.collection.get_file_appends(taskfn), taskdepdata, False))
> + b"</runtask>")
> - self.rq.worker[mc].process.stdin.flush()
> -
> - self.build_stamps[task] = bb.build.stampfile(taskname,
> self.rqdata.dataCaches[mc], taskfn, noextra=True)
> - self.build_stamps2.append(self.build_stamps[task])
> - self.runq_running.add(task)
> - self.stats.taskActive()
> - if self.can_start_task():
> - return True
> -
> - if self.stats.active > 0:
> - self.rq.read_workers()
> - return self.rq.active_fds()
> -
> - #for tid in self.sq_revdeps:
> - # if tid not in self.runq_running:
> - # buildable = tid in self.runq_buildable
> - # revdeps = self.sq_revdeps[tid]
> - # bb.warn("Found we didn't run %s %s %s" % (tid,
> buildable, str(revdeps))) -
> - self.rq.scenequeue_covered = self.scenequeue_covered
> - self.rq.scenequeue_notcovered = self.scenequeue_notcovered
> -
> - logger.debug(1, 'We can skip tasks %s',
> "\n".join(sorted(self.rq.scenequeue_covered))) -
> - self.rq.state = runQueueRunInit
> -
> - completeevent = sceneQueueComplete(self.stats, self.rq)
> - bb.event.fire(completeevent, self.cfgData)
> -
> - return True
> -
> - def runqueue_process_waitpid(self, task, status):
> - RunQueueExecute.runqueue_process_waitpid(self, task, status)
> -
> -
> - def build_taskdepdata(self, task):
> + def sq_build_taskdepdata(self, task):
> def getsetscenedeps(tid):
> deps = set()
> (mc, fn, taskname, _) = split_tid_mcfn(tid)
> @@ -2526,7 +2470,8 @@ class
> RunQueueExecuteScenequeue(RunQueueExecute): deps =
> getsetscenedeps(revdep) provides =
> self.rqdata.dataCaches[mc].fn_provides[taskfn] taskhash =
> self.rqdata.runtaskentries[revdep].hash
> - taskdepdata[revdep] = [pn, taskname, fn, deps,
> provides, taskhash]
> + unihash = self.rqdata.runtaskentries[revdep].unihash
> + taskdepdata[revdep] = [pn, taskname, fn, deps,
> provides, taskhash, unihash] for revdep2 in deps:
> if revdep2 not in taskdepdata:
> additional.append(revdep2)
> @@ -2535,6 +2480,279 @@ class
> RunQueueExecuteScenequeue(RunQueueExecute): #bb.note("Task %s: " %
> task + str(taskdepdata).replace("], ", "],\n")) return taskdepdata
>
> + def check_setscenewhitelist(self, tid):
> + # Check task that is going to run against the whitelist
> + (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
> + # Ignore covered tasks
> + if tid in self.tasks_covered:
> + return False
> + # Ignore stamped tasks
> + if self.rq.check_stamp_task(tid, taskname,
> cache=self.stampcache):
> + return False
> + # Ignore noexec tasks
> + taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn]
> + if 'noexec' in taskdep and taskname in taskdep['noexec']:
> + return False
> +
> + pn = self.rqdata.dataCaches[mc].pkg_fn[taskfn]
> + if not check_setscene_enforce_whitelist(pn, taskname,
> self.rqdata.setscenewhitelist):
> + if tid in self.rqdata.runq_setscene_tids:
> + msg = 'Task %s.%s attempted to execute unexpectedly
> and should have been setscened' % (pn, taskname)
> + else:
> + msg = 'Task %s.%s attempted to execute unexpectedly'
> % (pn, taskname)
> + logger.error(msg + '\nThis is usually due to missing
> setscene tasks. Those missing in this build were: %s' %
> pprint.pformat(self.scenequeue_notcovered))
> + return True
> + return False
> +
> +class SQData(object):
> + def __init__(self):
> + # SceneQueue dependencies
> + self.sq_deps = {}
> + # SceneQueue reverse dependencies
> + self.sq_revdeps = {}
> + # Injected inter-setscene task dependencies
> + self.sq_harddeps = {}
> + # Cache of stamp files so duplicates can't run in parallel
> + self.stamps = {}
> + # Setscene tasks directly depended upon by the build
> + self.unskippable = set()
> + # List of setscene tasks which aren't present
> + self.outrightfail = set()
> + # A list of normal tasks a setscene task covers
> + self.sq_covered_tasks = {}
> +
> +def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache,
> sqrq): +
> + sq_revdeps = {}
> + sq_revdeps_squash = {}
> + sq_collated_deps = {}
> +
> + # We need to construct a dependency graph for the setscene
> functions. Intermediate
> + # dependencies between the setscene tasks only complicate the
> code. This code
> + # therefore aims to collapse the huge runqueue dependency tree
> into a smaller one
> + # only containing the setscene functions.
> +
> + rqdata.init_progress_reporter.next_stage()
> +
> + # First process the chains up to the first setscene task.
> + endpoints = {}
> + for tid in rqdata.runtaskentries:
> + sq_revdeps[tid] =
> copy.copy(rqdata.runtaskentries[tid].revdeps)
> + sq_revdeps_squash[tid] = set()
> + if (len(sq_revdeps[tid]) == 0) and tid not in
> rqdata.runq_setscene_tids:
> + #bb.warn("Added endpoint %s" % (tid))
> + endpoints[tid] = set()
> +
> + rqdata.init_progress_reporter.next_stage()
> +
> + # Secondly process the chains between setscene tasks.
> + for tid in rqdata.runq_setscene_tids:
> + sq_collated_deps[tid] = set()
> + #bb.warn("Added endpoint 2 %s" % (tid))
> + for dep in rqdata.runtaskentries[tid].depends:
> + if tid in sq_revdeps[dep]:
> + sq_revdeps[dep].remove(tid)
> + if dep not in endpoints:
> + endpoints[dep] = set()
> + #bb.warn(" Added endpoint 3 %s" % (dep))
> + endpoints[dep].add(tid)
> +
> + rqdata.init_progress_reporter.next_stage()
> +
> + def process_endpoints(endpoints):
> + newendpoints = {}
> + for point, task in endpoints.items():
> + tasks = set()
> + if task:
> + tasks |= task
> + if sq_revdeps_squash[point]:
> + tasks |= sq_revdeps_squash[point]
> + if point not in rqdata.runq_setscene_tids:
> + for t in tasks:
> + sq_collated_deps[t].add(point)
> + sq_revdeps_squash[point] = set()
> + if point in rqdata.runq_setscene_tids:
> + sq_revdeps_squash[point] = tasks
> + tasks = set()
> + continue
> + for dep in rqdata.runtaskentries[point].depends:
> + if point in sq_revdeps[dep]:
> + sq_revdeps[dep].remove(point)
> + if tasks:
> + sq_revdeps_squash[dep] |= tasks
> + if len(sq_revdeps[dep]) == 0 and dep not in
> rqdata.runq_setscene_tids:
> + newendpoints[dep] = task
> + if len(newendpoints) != 0:
> + process_endpoints(newendpoints)
> +
> + process_endpoints(endpoints)
> +
> + rqdata.init_progress_reporter.next_stage()
> +
> + # Build a list of tasks which are "unskippable"
> + # These are direct endpoints referenced by the build upto and
> including setscene tasks
> + # Take the build endpoints (no revdeps) and find the sstate
> tasks they depend upon
> + new = True
> + for tid in rqdata.runtaskentries:
> + if len(rqdata.runtaskentries[tid].revdeps) == 0:
> + sqdata.unskippable.add(tid)
> + sqdata.unskippable |= sqrq.cantskip
> + while new:
> + new = False
> + orig = sqdata.unskippable.copy()
> + for tid in sorted(orig, reverse=True):
> + if tid in rqdata.runq_setscene_tids:
> + continue
> + if len(rqdata.runtaskentries[tid].depends) == 0:
> + # These are tasks which have no setscene tasks in
> their chain, need to mark as directly buildable
> + sqrq.setbuildable(tid)
> + sqdata.unskippable |= rqdata.runtaskentries[tid].depends
> + if sqdata.unskippable != orig:
> + new = True
> +
> + sqrq.tasks_scenequeue_done |=
> sqdata.unskippable.difference(rqdata.runq_setscene_tids) +
> +
> rqdata.init_progress_reporter.next_stage(len(rqdata.runtaskentries)) +
> + # Sanity check all dependencies could be changed to setscene
> task references
> + for taskcounter, tid in enumerate(rqdata.runtaskentries):
> + if tid in rqdata.runq_setscene_tids:
> + pass
> + elif len(sq_revdeps_squash[tid]) != 0:
> + bb.msg.fatal("RunQueue", "Something went badly wrong
> during scenequeue generation, aborting. Please report this problem.")
> + else:
> + del sq_revdeps_squash[tid]
> + rqdata.init_progress_reporter.update(taskcounter)
> +
> + rqdata.init_progress_reporter.next_stage()
> +
> + # Resolve setscene inter-task dependencies
> + # e.g. do_sometask_setscene[depends] =
> "targetname:do_someothertask_setscene"
> + # Note that anything explicitly depended upon will have its
> reverse dependencies removed to avoid circular dependencies
> + for tid in rqdata.runq_setscene_tids:
> + (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
> + realtid = tid + "_setscene"
> + idepends = rqdata.taskData[mc].taskentries[realtid].idepends
> + sqdata.stamps[tid] = bb.build.stampfile(taskname +
> "_setscene", rqdata.dataCaches[mc], taskfn, noextra=True)
> + for (depname, idependtask) in idepends:
> +
> + if depname not in rqdata.taskData[mc].build_targets:
> + continue
> +
> + depfn = rqdata.taskData[mc].build_targets[depname][0]
> + if depfn is None:
> + continue
> + deptid = depfn + ":" + idependtask.replace("_setscene",
> "")
> + if deptid not in rqdata.runtaskentries:
> + bb.msg.fatal("RunQueue", "Task %s depends upon
> non-existent task %s:%s" % (realtid, depfn, idependtask)) +
> + if not deptid in sqdata.sq_harddeps:
> + sqdata.sq_harddeps[deptid] = set()
> + sqdata.sq_harddeps[deptid].add(tid)
> +
> + sq_revdeps_squash[tid].add(deptid)
> + # Have to zero this to avoid circular dependencies
> + sq_revdeps_squash[deptid] = set()
> +
> + rqdata.init_progress_reporter.next_stage()
> +
> + for task in sqdata.sq_harddeps:
> + for dep in sqdata.sq_harddeps[task]:
> + sq_revdeps_squash[dep].add(task)
> +
> + rqdata.init_progress_reporter.next_stage()
> +
> + #for tid in sq_revdeps_squash:
> + # data = ""
> + # for dep in sq_revdeps_squash[tid]:
> + # data = data + "\n %s" % dep
> + # bb.warn("Task %s_setscene: is %s " % (tid, data))
> +
> + sqdata.sq_revdeps = sq_revdeps_squash
> + sqdata.sq_covered_tasks = sq_collated_deps
> +
> + # Build reverse version of revdeps to populate deps structure
> + for tid in sqdata.sq_revdeps:
> + sqdata.sq_deps[tid] = set()
> + for tid in sqdata.sq_revdeps:
> + for dep in sqdata.sq_revdeps[tid]:
> + sqdata.sq_deps[dep].add(tid)
> +
> + rqdata.init_progress_reporter.next_stage()
> +
> + sqdata.multiconfigs = set()
> + for tid in sqdata.sq_revdeps:
> + sqdata.multiconfigs.add(mc_from_tid(tid))
> + if len(sqdata.sq_revdeps[tid]) == 0:
> + sqrq.sq_buildable.add(tid)
> +
> + rqdata.init_progress_reporter.finish()
> +
> + sqdata.noexec = set()
> + sqdata.stamppresent = set()
> + sqdata.valid = set()
> +
> + update_scenequeue_data(sqdata.sq_revdeps, sqdata, rqdata, rq,
> cooker, stampcache, sqrq) +
> +def update_scenequeue_data(tids, sqdata, rqdata, rq, cooker,
> stampcache, sqrq): +
> + tocheck = set()
> +
> + for tid in sorted(tids):
> + if tid in sqdata.stamppresent:
> + sqdata.stamppresent.remove(tid)
> + if tid in sqdata.valid:
> + sqdata.valid.remove(tid)
> +
> + (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
> +
> + taskdep = rqdata.dataCaches[mc].task_deps[taskfn]
> +
> + if 'noexec' in taskdep and taskname in taskdep['noexec']:
> + sqdata.noexec.add(tid)
> + sqrq.sq_task_skip(tid)
> + bb.build.make_stamp(taskname + "_setscene",
> rqdata.dataCaches[mc], taskfn)
> + continue
> +
> + if rq.check_stamp_task(tid, taskname + "_setscene",
> cache=stampcache):
> + logger.debug(2, 'Setscene stamp current for task %s',
> tid)
> + sqdata.stamppresent.add(tid)
> + sqrq.sq_task_skip(tid)
> + continue
> +
> + if rq.check_stamp_task(tid, taskname, recurse = True,
> cache=stampcache):
> + logger.debug(2, 'Normal stamp current for task %s', tid)
> + sqdata.stamppresent.add(tid)
> + sqrq.sq_task_skip(tid)
> + continue
> +
> + tocheck.add(tid)
> +
> + sqdata.valid |= rq.validate_hashes(tocheck, cooker.data,
> len(sqdata.stamppresent), False) +
> + sqdata.hashes = {}
> + for mc in sorted(sqdata.multiconfigs):
> + for tid in sorted(sqdata.sq_revdeps):
> + if mc_from_tid(tid) != mc:
> + continue
> + if tid in sqdata.stamppresent:
> + continue
> + if tid in sqdata.valid:
> + continue
> + if tid in sqdata.noexec:
> + continue
> + if tid in sqrq.scenequeue_notcovered:
> + continue
> + sqdata.outrightfail.add(tid)
> +
> + h = pending_hash_index(tid, rqdata)
> + if h not in sqdata.hashes:
> + sqdata.hashes[h] = tid
> + else:
> + sqrq.sq_deferred[tid] = sqdata.hashes[h]
> + bb.warn("Deferring %s after %s" % (tid,
> sqdata.hashes[h])) +
> +
> class TaskFailure(Exception):
> """
> Exception raised when a task in a runqueue fails
> @@ -2641,6 +2859,15 @@ class runQueueTaskSkipped(runQueueEvent):
> runQueueEvent.__init__(self, task, stats, rq)
> self.reason = reason
>
> +class taskUniHashUpdate(bb.event.Event):
> + """
> + Base runQueue event class
> + """
> + def __init__(self, task, unihash):
> + self.taskid = task
> + self.unihash = unihash
> + bb.event.Event.__init__(self)
> +
> class runQueuePipe():
> """
> Abstraction for a pipe between a worker thread and the server
> @@ -2683,6 +2910,8 @@ class runQueuePipe():
> except ValueError as e:
> bb.msg.fatal("RunQueue", "failed load pickle
> '%s': '%s'" % (e, self.queue[7:index]))
> bb.event.fire_from_worker(event, self.d)
> + if isinstance(event, taskUniHashUpdate):
> +
> self.rqexec.updated_taskhash_queue.append((event.taskid,
> event.unihash)) found = True self.queue = self.queue[index+8:]
> index = self.queue.find(b"</event>")
> diff --git a/bitbake/lib/bb/server/__init__.py
> b/bitbake/lib/bb/server/__init__.py index 5a3fba9..b6f7513 100644
> --- a/bitbake/lib/bb/server/__init__.py
> +++ b/bitbake/lib/bb/server/__init__.py
> @@ -5,17 +5,5 @@
> # Copyright (C) 2006 - 2008 Richard Purdie
> # Copyright (C) 2013 Alexandru Damian
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. -
> -
> diff --git a/bitbake/lib/bb/server/process.py
> b/bitbake/lib/bb/server/process.py index 4e0d9c2..69aae62 100644
> --- a/bitbake/lib/bb/server/process.py
> +++ b/bitbake/lib/bb/server/process.py
> @@ -3,18 +3,8 @@
> #
> # Copyright (C) 2010 Bob Foerster <robert@erafx.com>
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> """
> This module implements a multiprocessing.Process based server
> for bitbake. @@ -130,6 +120,7 @@ class
> ProcessServer(multiprocessing.Process):
> bb.utils.set_process_name("Cooker")
> ready = []
> + newconnections = []
>
> self.controllersock = False
> fds = [self.sock]
> @@ -138,37 +129,48 @@ class ProcessServer(multiprocessing.Process):
> print("Entering server connection loop")
>
> def disconnect_client(self, fds):
> - if not self.haveui:
> - return
> print("Disconnecting Client")
> - fds.remove(self.controllersock)
> - fds.remove(self.command_channel)
> - bb.event.unregister_UIHhandler(self.event_handle, True)
> - self.command_channel_reply.writer.close()
> - self.event_writer.writer.close()
> - del self.event_writer
> - self.controllersock.close()
> - self.controllersock = False
> - self.haveui = False
> - self.lastui = time.time()
> - self.cooker.clientComplete()
> - if self.timeout is None:
> + if self.controllersock:
> + fds.remove(self.controllersock)
> + self.controllersock.close()
> + self.controllersock = False
> + if self.haveui:
> + fds.remove(self.command_channel)
> + bb.event.unregister_UIHhandler(self.event_handle,
> True)
> + self.command_channel_reply.writer.close()
> + self.event_writer.writer.close()
> + self.command_channel.close()
> + self.command_channel = False
> + del self.event_writer
> + self.lastui = time.time()
> + self.cooker.clientComplete()
> + self.haveui = False
> + ready = select.select(fds,[],[],0)[0]
> + if newconnections:
> + print("Starting new client")
> + conn = newconnections.pop(-1)
> + fds.append(conn)
> + self.controllersock = conn
> + elif self.timeout is None and not ready:
> print("No timeout, exiting.")
> self.quit = True
>
> while not self.quit:
> if self.sock in ready:
> - self.controllersock, address = self.sock.accept()
> - if self.haveui:
> - print("Dropping connection attempt as we have a
> UI %s" % (str(ready)))
> - self.controllersock.close()
> - else:
> - print("Accepting %s" % (str(ready)))
> - fds.append(self.controllersock)
> + while select.select([self.sock],[],[],0)[0]:
> + controllersock, address = self.sock.accept()
> + if self.controllersock:
> + print("Queuing %s (%s)" % (str(ready),
> str(newconnections)))
> + newconnections.append(controllersock)
> + else:
> + print("Accepting %s (%s)" % (str(ready),
> str(newconnections)))
> + self.controllersock = controllersock
> + fds.append(controllersock)
> if self.controllersock in ready:
> try:
> - print("Connecting Client")
> + print("Processing Client")
> ui_fds = recvfds(self.controllersock, 3)
> + print("Connecting Client")
>
> # Where to write events to
> writer = ConnectionWriter(ui_fds[0])
> @@ -239,6 +241,12 @@ class ProcessServer(multiprocessing.Process):
> while not lock:
> with bb.utils.timeout(3):
> lock = bb.utils.lockfile(lockfile, shared=False,
> retry=False, block=True)
> + if lock:
> + # We hold the lock so we can remove the file
> (hide stale pid data)
> + bb.utils.remove(lockfile)
> + bb.utils.unlockfile(lock)
> + return
> +
> if not lock:
> # Some systems may not have lsof available
> procs = None
> @@ -259,10 +267,6 @@ class ProcessServer(multiprocessing.Process):
> if procs:
> msg += ":\n%s" % str(procs)
> print(msg)
> - return
> - # We hold the lock so we can remove the file (hide stale pid
> data)
> - bb.utils.remove(lockfile)
> - bb.utils.unlockfile(lock)
>
> def idle_commands(self, delay, fds=None):
> nextsleep = delay
> @@ -398,36 +402,45 @@ class BitBakeServer(object):
> os.close(self.readypipein)
>
> ready = ConnectionReader(self.readypipe)
> - r = ready.poll(30)
> + r = ready.poll(5)
> + if not r:
> + bb.note("Bitbake server didn't start within 5 seconds,
> waiting for 90")
> + r = ready.poll(90)
> if r:
> try:
> r = ready.get()
> except EOFError:
> # Trap the child exitting/closing the pipe and error
> out r = None
> - if not r or r != "ready":
> + if not r or r[0] != "r":
> ready.close()
> - bb.error("Unable to start bitbake server")
> + bb.error("Unable to start bitbake server (%s)" % str(r))
> if os.path.exists(logfile):
> logstart_re = re.compile(self.start_log_format %
> ('([0-9]+)', '([0-9-]+ [0-9:.]+)')) started = False
> lines = []
> + lastlines = []
> with open(logfile, "r") as f:
> for line in f:
> if started:
> lines.append(line)
> else:
> + lastlines.append(line)
> res = logstart_re.match(line.rstrip())
> if res:
> ldatetime =
> datetime.datetime.strptime(res.group(2),
> self.start_log_datetime_format) if ldatetime >= startdatetime:
> started = True lines.append(line)
> + if len(lastlines) > 60:
> + lastlines = lastlines[-60:]
> if lines:
> - if len(lines) > 10:
> - bb.error("Last 10 lines of server log for
> this session (%s):\n%s" % (logfile, "".join(lines[-10:])))
> + if len(lines) > 60:
> + bb.error("Last 60 lines of server log for
> this session (%s):\n%s" % (logfile, "".join(lines[-60:]))) else:
> bb.error("Server log for this session
> (%s):\n%s" % (logfile, "".join(lines)))
> + elif lastlines:
> + bb.error("Server didn't start, last 60
> loglines (%s):\n%s" % (logfile, "".join(lastlines))) else:
> bb.error("%s doesn't exist" % logfile)
>
> @@ -437,17 +450,24 @@ class BitBakeServer(object):
>
> def _startServer(self):
> print(self.start_log_format % (os.getpid(),
> datetime.datetime.now().strftime(self.start_log_datetime_format)))
> + sys.stdout.flush()
> +
> server = ProcessServer(self.bitbake_lock, self.sock,
> self.sockname)
> self.configuration.setServerRegIdleCallback(server.register_idle_function)
> os.close(self.readypipe) writer = ConnectionWriter(self.readypipein)
> - self.cooker = bb.cooker.BBCooker(self.configuration,
> self.featureset)
> - writer.send("ready")
> + try:
> + self.cooker = bb.cooker.BBCooker(self.configuration,
> self.featureset)
> + except bb.BBHandledException:
> + return None
> + writer.send("r")
> writer.close()
> server.cooker = self.cooker
> server.server_timeout = self.configuration.server_timeout
> server.xmlrpcinterface = self.configuration.xmlrpcinterface
> print("Started bitbake server pid %d" % os.getpid())
> + sys.stdout.flush()
> +
> server.start()
>
> def connectProcessServer(sockname, featureset):
> @@ -459,10 +479,20 @@ def connectProcessServer(sockname, featureset):
> readfd = writefd = readfd1 = writefd1 = readfd2 = writefd2 = None
> eq = command_chan_recv = command_chan = None
>
> + sock.settimeout(10)
> +
> try:
> try:
> os.chdir(os.path.dirname(sockname))
> - sock.connect(os.path.basename(sockname))
> + finished = False
> + while not finished:
> + try:
> + sock.connect(os.path.basename(sockname))
> + finished = True
> + except IOError as e:
> + if e.errno == errno.EWOULDBLOCK:
> + pass
> + raise
> finally:
> os.chdir(cwd)
>
> @@ -493,7 +523,8 @@ def connectProcessServer(sockname, featureset):
> command_chan.close()
> for i in [writefd, readfd1, writefd2]:
> try:
> - os.close(i)
> + if i:
> + os.close(i)
> except OSError:
> pass
> sock.close()
> diff --git a/bitbake/lib/bb/server/xmlrpcclient.py
> b/bitbake/lib/bb/server/xmlrpcclient.py index 4661a9e..c054c3c 100644
> --- a/bitbake/lib/bb/server/xmlrpcclient.py
> +++ b/bitbake/lib/bb/server/xmlrpcclient.py
> @@ -4,18 +4,8 @@
> # Copyright (C) 2006 - 2007 Michael 'Mickey' Lauer
> # Copyright (C) 2006 - 2008 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import os
> import sys
> diff --git a/bitbake/lib/bb/server/xmlrpcserver.py
> b/bitbake/lib/bb/server/xmlrpcserver.py index 875b128..54fa32f 100644
> --- a/bitbake/lib/bb/server/xmlrpcserver.py
> +++ b/bitbake/lib/bb/server/xmlrpcserver.py
> @@ -4,18 +4,8 @@
> # Copyright (C) 2006 - 2007 Michael 'Mickey' Lauer
> # Copyright (C) 2006 - 2008 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import os
> import sys
> diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py
> index fdbb2a3..a4bb1ff 100644
> --- a/bitbake/lib/bb/siggen.py
> +++ b/bitbake/lib/bb/siggen.py
> @@ -1,3 +1,7 @@
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> import hashlib
> import logging
> import os
> @@ -8,6 +12,8 @@ import bb.data
> import difflib
> import simplediff
> from bb.checksum import FileChecksumCache
> +from bb import runqueue
> +import hashserv
>
> logger = logging.getLogger('BitBake.SigGen')
>
> @@ -37,12 +43,18 @@ class SignatureGenerator(object):
> self.runtaskdeps = {}
> self.file_checksum_values = {}
> self.taints = {}
> + self.unitaskhashes = {}
> + self.setscenetasks = {}
>
> def finalise(self, fn, d, varient):
> return
>
> - def get_taskhash(self, fn, task, deps, dataCache):
> - return "0"
> + def get_unihash(self, tid):
> + return self.taskhash[tid]
> +
> + def get_taskhash(self, tid, deps, dataCache):
> + self.taskhash[tid] =
> hashlib.sha256(tid.encode("utf-8")).hexdigest()
> + return self.taskhash[tid]
>
> def writeout_file_checksum_cache(self):
> """Write/update the file checksum cache onto disk"""
> @@ -64,14 +76,25 @@ class SignatureGenerator(object):
> return
>
> def get_taskdata(self):
> - return (self.runtaskdeps, self.taskhash,
> self.file_checksum_values, self.taints, self.basehash)
> + return (self.runtaskdeps, self.taskhash,
> self.file_checksum_values, self.taints, self.basehash,
> self.unitaskhashes, self.setscenetasks) def set_taskdata(self, data):
> - self.runtaskdeps, self.taskhash, self.file_checksum_values,
> self.taints, self.basehash = data
> + self.runtaskdeps, self.taskhash, self.file_checksum_values,
> self.taints, self.basehash, self.unitaskhashes, self.setscenetasks =
> data def reset(self, data):
> self.__init__(data)
>
> + def get_taskhashes(self):
> + return self.taskhash, self.unitaskhashes
> +
> + def set_taskhashes(self, hashes):
> + self.taskhash, self.unitaskhashes = hashes
> +
> + def save_unitaskhashes(self):
> + return
> +
> + def set_setscene_tasks(self, setscene_tasks):
> + return
>
> class SignatureGeneratorBasic(SignatureGenerator):
> """
> @@ -87,7 +110,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
> self.taints = {}
> self.gendeps = {}
> self.lookupcache = {}
> - self.pkgnameextract = re.compile("(?P<fn>.*)\..*")
> + self.setscenetasks = {}
> self.basewhitelist =
> set((data.getVar("BB_HASHBASE_WHITELIST") or "").split())
> self.taskwhitelist = None self.init_rundepcheck(data)
> @@ -98,6 +121,9 @@ class SignatureGeneratorBasic(SignatureGenerator):
> else:
> self.checksum_cache = None
>
> + self.unihash_cache = bb.cache.SimpleCache("1")
> + self.unitaskhashes = self.unihash_cache.init_cache(data,
> "bb_unihashes.dat", {}) +
> def init_rundepcheck(self, data):
> self.taskwhitelist = data.getVar("BB_HASHTASK_WHITELIST") or
> None if self.taskwhitelist:
> @@ -113,10 +139,16 @@ class
> SignatureGeneratorBasic(SignatureGenerator): taskdeps, basehash =
> bb.data.generate_dependency_hash(tasklist, gendeps, lookupcache,
> self.basewhitelist, fn) for task in tasklist:
> - k = fn + "." + task
> - if not ignore_mismatch and k in self.basehash and
> self.basehash[k] != basehash[k]:
> - bb.error("When reparsing %s, the basehash value
> changed from %s to %s. The metadata is not deterministic and this
> needs to be fixed." % (k, self.basehash[k], basehash[k]))
> - self.basehash[k] = basehash[k]
> + tid = fn + ":" + task
> + if not ignore_mismatch and tid in self.basehash and
> self.basehash[tid] != basehash[tid]:
> + bb.error("When reparsing %s, the basehash value
> changed from %s to %s. The metadata is not deterministic and this
> needs to be fixed." % (tid, self.basehash[tid], basehash[tid]))
> + bb.error("The following commands may help:")
> + cmd = "$ bitbake %s -c%s" % (d.getVar('PN'), task)
> + # Make sure sigdata is dumped before run printdiff
> + bb.error("%s -Snone" % cmd)
> + bb.error("Then:")
> + bb.error("%s -Sprintdiff\n" % cmd)
> + self.basehash[tid] = basehash[tid]
>
> self.taskdeps[fn] = taskdeps
> self.gendeps[fn] = gendeps
> @@ -124,6 +156,9 @@ class SignatureGeneratorBasic(SignatureGenerator):
>
> return taskdeps
>
> + def set_setscene_tasks(self, setscene_tasks):
> + self.setscenetasks = setscene_tasks
> +
> def finalise(self, fn, d, variant):
>
> mc = d.getVar("__BBMULTICONFIG", False) or ""
> @@ -143,7 +178,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
> # self.dump_sigtask(fn, task, d.getVar("STAMP"), False)
>
> for task in taskdeps:
> - d.setVar("BB_BASEHASH_task-%s" % task, self.basehash[fn
> + "." + task])
> + d.setVar("BB_BASEHASH_task-%s" % task, self.basehash[fn
> + ":" + task])
> def rundep_check(self, fn, recipename, task, dep, depname,
> dataCache): # Return True if we should keep the dependency, False to
> drop it @@ -163,31 +198,26 @@ class
> SignatureGeneratorBasic(SignatureGenerator): pass
> return taint
>
> - def get_taskhash(self, fn, task, deps, dataCache):
> + def get_taskhash(self, tid, deps, dataCache):
>
> - mc = ''
> - if fn.startswith('multiconfig:'):
> - mc = fn.split(':')[1]
> - k = fn + "." + task
> + (mc, _, task, fn) = bb.runqueue.split_tid_mcfn(tid)
>
> - data = dataCache.basetaskhash[k]
> - self.basehash[k] = data
> - self.runtaskdeps[k] = []
> - self.file_checksum_values[k] = []
> + data = dataCache.basetaskhash[tid]
> + self.basehash[tid] = data
> + self.runtaskdeps[tid] = []
> + self.file_checksum_values[tid] = []
> recipename = dataCache.pkg_fn[fn]
> for dep in sorted(deps, key=clean_basepath):
> - pkgname = self.pkgnameextract.search(dep).group('fn')
> - if mc:
> - depmc = pkgname.split(':')[1]
> - if mc != depmc:
> - continue
> - depname = dataCache.pkg_fn[pkgname]
> + (depmc, _, deptaskname, depfn) =
> bb.runqueue.split_tid_mcfn(dep)
> + if mc != depmc:
> + continue
> + depname = dataCache.pkg_fn[depfn]
> if not self.rundep_check(fn, recipename, task, dep,
> depname, dataCache): continue
> if dep not in self.taskhash:
> bb.fatal("%s is not in taskhash, caller isn't
> calling in dependency order?" % dep)
> - data = data + self.taskhash[dep]
> - self.runtaskdeps[k].append(dep)
> + data = data + self.get_unihash(dep)
> + self.runtaskdeps[tid].append(dep)
>
> if task in dataCache.file_checksums[fn]:
> if self.checksum_cache:
> @@ -195,7 +225,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
> else:
> checksums =
> bb.fetch2.get_file_checksums(dataCache.file_checksums[fn][task],
> recipename) for (f,cs) in checksums:
> - self.file_checksum_values[k].append((f,cs))
> + self.file_checksum_values[tid].append((f,cs))
> if cs:
> data = data + cs
>
> @@ -205,16 +235,16 @@ class
> SignatureGeneratorBasic(SignatureGenerator): import uuid
> taint = str(uuid.uuid4())
> data = data + taint
> - self.taints[k] = "nostamp:" + taint
> + self.taints[tid] = "nostamp:" + taint
>
> taint = self.read_taint(fn, task, dataCache.stamp[fn])
> if taint:
> data = data + taint
> - self.taints[k] = taint
> - logger.warning("%s is tainted from a forced run" % k)
> + self.taints[tid] = taint
> + logger.warning("%s is tainted from a forced run" % tid)
>
> - h = hashlib.md5(data.encode("utf-8")).hexdigest()
> - self.taskhash[k] = h
> + h = hashlib.sha256(data.encode("utf-8")).hexdigest()
> + self.taskhash[tid] = h
> #d.setVar("BB_TASKHASH_task-%s" % task, taskhash[task])
> return h
>
> @@ -227,17 +257,20 @@ class
> SignatureGeneratorBasic(SignatureGenerator):
> bb.fetch2.fetcher_parse_save() bb.fetch2.fetcher_parse_done()
>
> + def save_unitaskhashes(self):
> + self.unihash_cache.save(self.unitaskhashes)
> +
> def dump_sigtask(self, fn, task, stampbase, runtime):
>
> - k = fn + "." + task
> + tid = fn + ":" + task
> referencestamp = stampbase
> if isinstance(runtime, str) and
> runtime.startswith("customfile"): sigfile = stampbase
> referencestamp = runtime[11:]
> - elif runtime and k in self.taskhash:
> - sigfile = stampbase + "." + task + ".sigdata" + "." +
> self.taskhash[k]
> + elif runtime and tid in self.taskhash:
> + sigfile = stampbase + "." + task + ".sigdata" + "." +
> self.get_unihash(tid) else:
> - sigfile = stampbase + "." + task + ".sigbasedata" + "."
> + self.basehash[k]
> + sigfile = stampbase + "." + task + ".sigbasedata" + "."
> + self.basehash[tid]
> bb.utils.mkdirhier(os.path.dirname(sigfile))
>
> @@ -246,7 +279,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
> data['basewhitelist'] = self.basewhitelist
> data['taskwhitelist'] = self.taskwhitelist
> data['taskdeps'] = self.taskdeps[fn][task]
> - data['basehash'] = self.basehash[k]
> + data['basehash'] = self.basehash[tid]
> data['gendeps'] = {}
> data['varvals'] = {}
> data['varvals'][task] = self.lookupcache[fn][task]
> @@ -256,30 +289,31 @@ class
> SignatureGeneratorBasic(SignatureGenerator): data['gendeps'][dep] =
> self.gendeps[fn][dep] data['varvals'][dep] = self.lookupcache[fn][dep]
>
> - if runtime and k in self.taskhash:
> - data['runtaskdeps'] = self.runtaskdeps[k]
> - data['file_checksum_values'] = [(os.path.basename(f),
> cs) for f,cs in self.file_checksum_values[k]]
> + if runtime and tid in self.taskhash:
> + data['runtaskdeps'] = self.runtaskdeps[tid]
> + data['file_checksum_values'] = [(os.path.basename(f),
> cs) for f,cs in self.file_checksum_values[tid]] data['runtaskhashes']
> = {} for dep in data['runtaskdeps']:
> - data['runtaskhashes'][dep] = self.taskhash[dep]
> - data['taskhash'] = self.taskhash[k]
> + data['runtaskhashes'][dep] = self.get_unihash(dep)
> + data['taskhash'] = self.taskhash[tid]
> + data['unihash'] = self.get_unihash(tid)
>
> taint = self.read_taint(fn, task, referencestamp)
> if taint:
> data['taint'] = taint
>
> - if runtime and k in self.taints:
> - if 'nostamp:' in self.taints[k]:
> - data['taint'] = self.taints[k]
> + if runtime and tid in self.taints:
> + if 'nostamp:' in self.taints[tid]:
> + data['taint'] = self.taints[tid]
>
> computed_basehash = calc_basehash(data)
> - if computed_basehash != self.basehash[k]:
> - bb.error("Basehash mismatch %s versus %s for %s" %
> (computed_basehash, self.basehash[k], k))
> - if runtime and k in self.taskhash:
> + if computed_basehash != self.basehash[tid]:
> + bb.error("Basehash mismatch %s versus %s for %s" %
> (computed_basehash, self.basehash[tid], tid))
> + if runtime and tid in self.taskhash:
> computed_taskhash = calc_taskhash(data)
> - if computed_taskhash != self.taskhash[k]:
> - bb.error("Taskhash mismatch %s versus %s for %s" %
> (computed_taskhash, self.taskhash[k], k))
> - sigfile = sigfile.replace(self.taskhash[k],
> computed_taskhash)
> + if computed_taskhash != self.taskhash[tid]:
> + bb.error("Taskhash mismatch %s versus %s for %s" %
> (computed_taskhash, self.taskhash[tid], tid))
> + sigfile = sigfile.replace(self.taskhash[tid],
> computed_taskhash)
> fd, tmpfile = tempfile.mkstemp(dir=os.path.dirname(sigfile),
> prefix="sigtask.") try:
> @@ -299,30 +333,34 @@ class
> SignatureGeneratorBasic(SignatureGenerator): if fn in self.taskdeps:
> for task in self.taskdeps[fn]:
> tid = fn + ":" + task
> - (mc, _, _) = bb.runqueue.split_tid(tid)
> - k = fn + "." + task
> - if k not in self.taskhash:
> + mc = bb.runqueue.mc_from_tid(tid)
> + if tid not in self.taskhash:
> continue
> - if dataCaches[mc].basetaskhash[k] !=
> self.basehash[k]:
> - bb.error("Bitbake's cached basehash does not
> match the one we just generated (%s)!" % k)
> - bb.error("The mismatched hashes were %s and %s"
> % (dataCaches[mc].basetaskhash[k], self.basehash[k]))
> + if dataCaches[mc].basetaskhash[tid] !=
> self.basehash[tid]:
> + bb.error("Bitbake's cached basehash does not
> match the one we just generated (%s)!" % tid)
> + bb.error("The mismatched hashes were %s and %s"
> % (dataCaches[mc].basetaskhash[tid], self.basehash[tid]))
> self.dump_sigtask(fn, task, dataCaches[mc].stamp[fn], True)
> class SignatureGeneratorBasicHash(SignatureGeneratorBasic):
> name = "basichash"
>
> + def get_stampfile_hash(self, tid):
> + if tid in self.taskhash:
> + return self.taskhash[tid]
> +
> + # If task is not in basehash, then error
> + return self.basehash[tid]
> +
> def stampfile(self, stampbase, fn, taskname, extrainfo,
> clean=False): if taskname != "do_setscene" and
> taskname.endswith("_setscene"):
> - k = fn + "." + taskname[:-9]
> + tid = fn + ":" + taskname[:-9]
> else:
> - k = fn + "." + taskname
> + tid = fn + ":" + taskname
> if clean:
> h = "*"
> - elif k in self.taskhash:
> - h = self.taskhash[k]
> else:
> - # If k is not in basehash, then error
> - h = self.basehash[k]
> + h = self.get_stampfile_hash(tid)
> +
> return ("%s.%s.%s.%s" % (stampbase, taskname, h,
> extrainfo)).rstrip('.')
> def stampcleanmask(self, stampbase, fn, taskname, extrainfo):
> @@ -332,6 +370,172 @@ class
> SignatureGeneratorBasicHash(SignatureGeneratorBasic):
> bb.note("Tainting hash to force rebuild of task %s, %s" % (fn, task))
> bb.build.write_taint(task, d, fn)
> +class SignatureGeneratorUniHashMixIn(object):
> + def get_taskdata(self):
> + return (self.server, self.method) + super().get_taskdata()
> +
> + def set_taskdata(self, data):
> + self.server, self.method = data[:2]
> + super().set_taskdata(data[2:])
> +
> + def client(self):
> + if getattr(self, '_client', None) is None:
> + self._client = hashserv.create_client(self.server)
> + return self._client
> +
> + def __get_task_unihash_key(self, tid):
> + # TODO: The key only *needs* to be the taskhash, the tid is
> just
> + # convenient
> + return '%s:%s' % (tid.rsplit("/", 1)[1], self.taskhash[tid])
> +
> + def get_stampfile_hash(self, tid):
> + if tid in self.taskhash:
> + # If a unique hash is reported, use it as the stampfile
> hash. This
> + # ensures that if a task won't be re-run if the taskhash
> changes,
> + # but it would result in the same output hash
> + unihash =
> self.unitaskhashes.get(self.__get_task_unihash_key(tid), None)
> + if unihash is not None:
> + return unihash
> +
> + return super().get_stampfile_hash(tid)
> +
> + def set_unihash(self, tid, unihash):
> + self.unitaskhashes[self.__get_task_unihash_key(tid)] =
> unihash +
> + def get_unihash(self, tid):
> + taskhash = self.taskhash[tid]
> +
> + # If its not a setscene task we can return
> + if self.setscenetasks and tid not in self.setscenetasks:
> + return taskhash
> +
> + key = self.__get_task_unihash_key(tid)
> +
> + # TODO: This cache can grow unbounded. It probably only
> needs to keep
> + # for each task
> + unihash = self.unitaskhashes.get(key, None)
> + if unihash is not None:
> + return unihash
> +
> + # In the absence of being able to discover a unique hash
> from the
> + # server, make it be equivalent to the taskhash. The unique
> "hash" only
> + # really needs to be a unique string (not even necessarily a
> hash), but
> + # making it match the taskhash has a few advantages:
> + #
> + # 1) All of the sstate code that assumes hashes can be the
> same
> + # 2) It provides maximal compatibility with builders that
> don't use
> + # an equivalency server
> + # 3) The value is easy for multiple independent builders to
> derive the
> + # same unique hash from the same input. This means that
> if the
> + # independent builders find the same taskhash, but it
> isn't reported
> + # to the server, there is a better chance that they will
> agree on
> + # the unique hash.
> + unihash = taskhash
> +
> + try:
> + data = self.client().get_unihash(self.method,
> self.taskhash[tid])
> + if data:
> + unihash = data
> + # A unique hash equal to the taskhash is not very
> interesting,
> + # so it is reported it at debug level 2. If they
> differ, that
> + # is much more interesting, so it is reported at
> debug level 1
> + bb.debug((1, 2)[unihash == taskhash], 'Found unihash
> %s in place of %s for %s from %s' % (unihash, taskhash, tid,
> self.server))
> + else:
> + bb.debug(2, 'No reported unihash for %s:%s from %s'
> % (tid, taskhash, self.server))
> + except hashserv.client.HashConnectionError as e:
> + bb.warn('Error contacting Hash Equivalence Server %s:
> %s' % (self.server, str(e))) +
> + self.unitaskhashes[key] = unihash
> + return unihash
> +
> + def report_unihash(self, path, task, d):
> + import importlib
> +
> + taskhash = d.getVar('BB_TASKHASH')
> + unihash = d.getVar('BB_UNIHASH')
> + report_taskdata =
> d.getVar('SSTATE_HASHEQUIV_REPORT_TASKDATA') == '1'
> + tempdir = d.getVar('T')
> + fn = d.getVar('BB_FILENAME')
> + tid = fn + ':do_' + task
> + key = tid.rsplit("/", 1)[1] + ':' + taskhash
> +
> + if self.setscenetasks and tid not in self.setscenetasks:
> + return
> +
> + # Sanity checks
> + cache_unihash = self.unitaskhashes.get(key, None)
> + if cache_unihash is None:
> + bb.fatal('%s not in unihash cache. Please report this
> error' % key) +
> + if cache_unihash != unihash:
> + bb.fatal("Cache unihash %s doesn't match BB_UNIHASH %s"
> % (cache_unihash, unihash)) +
> + sigfile = None
> + sigfile_name = "depsig.do_%s.%d" % (task, os.getpid())
> + sigfile_link = "depsig.do_%s" % task
> +
> + try:
> + sigfile = open(os.path.join(tempdir, sigfile_name),
> 'w+b') +
> + locs = {'path': path, 'sigfile': sigfile, 'task': task,
> 'd': d} +
> + if "." in self.method:
> + (module, method) = self.method.rsplit('.', 1)
> + locs['method'] =
> getattr(importlib.import_module(module), method)
> + outhash = bb.utils.better_eval('method(path,
> sigfile, task, d)', locs)
> + else:
> + outhash = bb.utils.better_eval(self.method + '(path,
> sigfile, task, d)', locs) +
> + try:
> + extra_data = {}
> +
> + owner = d.getVar('SSTATE_HASHEQUIV_OWNER')
> + if owner:
> + extra_data['owner'] = owner
> +
> + if report_taskdata:
> + sigfile.seek(0)
> +
> + extra_data['PN'] = d.getVar('PN')
> + extra_data['PV'] = d.getVar('PV')
> + extra_data['PR'] = d.getVar('PR')
> + extra_data['task'] = task
> + extra_data['outhash_siginfo'] =
> sigfile.read().decode('utf-8') +
> + data = self.client().report_unihash(taskhash,
> self.method, outhash, unihash, extra_data)
> + new_unihash = data['unihash']
> +
> + if new_unihash != unihash:
> + bb.debug(1, 'Task %s unihash changed %s -> %s by
> server %s' % (taskhash, unihash, new_unihash, self.server))
> + bb.event.fire(bb.runqueue.taskUniHashUpdate(fn +
> ':do_' + task, new_unihash), d)
> + else:
> + bb.debug(1, 'Reported task %s as unihash %s to
> %s' % (taskhash, unihash, self.server))
> + except hashserv.client.HashConnectionError as e:
> + bb.warn('Error contacting Hash Equivalence Server
> %s: %s' % (self.server, str(e)))
> + finally:
> + if sigfile:
> + sigfile.close()
> +
> + sigfile_link_path = os.path.join(tempdir,
> sigfile_link)
> + bb.utils.remove(sigfile_link_path)
> +
> + try:
> + os.symlink(sigfile_name, sigfile_link_path)
> + except OSError:
> + pass
> +
> +
> +#
> +# Dummy class used for bitbake-selftest
> +#
> +class
> SignatureGeneratorTestEquivHash(SignatureGeneratorUniHashMixIn,
> SignatureGeneratorBasicHash):
> + name = "TestEquivHash"
> + def init_rundepcheck(self, data):
> + super().init_rundepcheck(data)
> + self.server = data.getVar('BB_HASHSERVE')
> + self.method = "sstate_output_hash"
> +
> +
> def dump_this_task(outfile, d):
> import bb.parse
> fn = d.getVar("BB_FILENAME")
> @@ -392,13 +596,13 @@ def list_inline_diff(oldlist, newlist,
> colors=None):
> def clean_basepath(a):
> mc = None
> - if a.startswith("multiconfig:"):
> + if a.startswith("mc:"):
> _, mc, a = a.split(":", 2)
> b = a.rsplit("/", 2)[1] + '/' + a.rsplit("/", 2)[2]
> if a.startswith("virtual:"):
> b = b + ":" + a.rsplit(":", 1)[0]
> if mc:
> - b = b + ":multiconfig:" + mc
> + b = b + ":mc:" + mc
> return b
>
> def clean_basepaths(a):
> @@ -623,6 +827,10 @@ def compare_sigfiles(a, b, recursecb=None,
> color=False, collapsed=False): a_taint = a_data.get('taint', None)
> b_taint = b_data.get('taint', None)
> if a_taint != b_taint:
> + if a_taint and a_taint.startswith('nostamp:'):
> + a_taint = a_taint.replace('nostamp:', 'nostamp(uuid4):')
> + if b_taint and b_taint.startswith('nostamp:'):
> + b_taint = b_taint.replace('nostamp:', 'nostamp(uuid4):')
> output.append(color_format("{color_title}Taint (by
> forced/invalidated task) changed{color_default} from %s to %s") %
> (a_taint, b_taint)) return output
> @@ -642,7 +850,7 @@ def calc_basehash(sigdata):
> if val is not None:
> basedata = basedata + str(val)
>
> - return hashlib.md5(basedata.encode("utf-8")).hexdigest()
> + return hashlib.sha256(basedata.encode("utf-8")).hexdigest()
>
> def calc_taskhash(sigdata):
> data = sigdata['basehash']
> @@ -660,7 +868,7 @@ def calc_taskhash(sigdata):
> else:
> data = data + sigdata['taint']
>
> - return hashlib.md5(data.encode("utf-8")).hexdigest()
> + return hashlib.sha256(data.encode("utf-8")).hexdigest()
>
>
> def dump_sigfile(a):
> @@ -695,7 +903,11 @@ def dump_sigfile(a):
> output.append("Hash for dependent task %s is %s" % (dep,
> a_data['runtaskhashes'][dep]))
> if 'taint' in a_data:
> - output.append("Tainted (by forced/invalidated task): %s" %
> a_data['taint'])
> + if a_data['taint'].startswith('nostamp:'):
> + msg = a_data['taint'].replace('nostamp:',
> 'nostamp(uuid4):')
> + else:
> + msg = a_data['taint']
> + output.append("Tainted (by forced/invalidated task): %s" %
> msg)
> if 'task' in a_data:
> computed_basehash = calc_basehash(a_data)
> diff --git a/bitbake/lib/bb/taskdata.py b/bitbake/lib/bb/taskdata.py
> index 94e822c..8c25e09 100644
> --- a/bitbake/lib/bb/taskdata.py
> +++ b/bitbake/lib/bb/taskdata.py
> @@ -1,6 +1,3 @@
> -#!/usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake 'TaskData' implementation
>
> @@ -10,18 +7,8 @@ Task data collection and handling
>
> # Copyright (C) 2006 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import logging
> import re
> @@ -93,7 +80,7 @@ class TaskData:
> def add_mcdepends(task):
> for dep in task_deps['mcdepends'][task].split():
> if len(dep.split(':')) != 5:
> - bb.msg.fatal("TaskData", "Error for %s:%s[%s],
> multiconfig dependency %s does not contain exactly four ':'
> characters.\n Task '%s' should be specified in the form
> 'multiconfig:fromMC:toMC:packagename:task'" % (fn, task, 'mcdepends',
> dep, 'mcdepends'))
> + bb.msg.fatal("TaskData", "Error for %s:%s[%s],
> multiconfig dependency %s does not contain exactly four ':'
> characters.\n Task '%s' should be specified in the form
> 'mc:fromMC:toMC:packagename:task'" % (fn, task, 'mcdepends', dep,
> 'mcdepends')) if dep not in self.mcdepends:
> self.mcdepends.append(dep) diff --git
> a/bitbake/lib/bb/tests/codeparser.py
> b/bitbake/lib/bb/tests/codeparser.py index e30e78c..826a2d2 100644
> --- a/bitbake/lib/bb/tests/codeparser.py +++
> b/bitbake/lib/bb/tests/codeparser.py @@ -1,23 +1,10 @@ -#
> ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4;
> indent-tabs-mode: nil -*- #
> # BitBake Test for codeparser.py
> #
> # Copyright (C) 2010 Chris Larson
> # Copyright (C) 2012 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
>
> import unittest
> @@ -123,6 +110,13 @@ ${D}${libdir}/pkgconfig/*.pc
> self.parseExpression("sed -i -e 's:IP{:I${:g' $pc")
> self.assertExecs(set(["sed"]))
>
> + def test_parameter_expansion_modifiers(self):
> + # - and + are also valid modifiers for parameter expansion,
> but are
> + # valid characters in bitbake variable names, so are not
> included here
> + for i in ('=', ':-', ':=', '?', ':?', ':+', '#', '%', '##',
> '%%'):
> + name = "foo%sbar" % i
> + self.parseExpression("${%s}" % name)
> + self.assertNotIn(name, self.references)
>
> def test_until(self):
> self.parseExpression("until false; do echo true; done")
> diff --git a/bitbake/lib/bb/tests/cooker.py
> b/bitbake/lib/bb/tests/cooker.py index 2b44236..090916e 100644
> --- a/bitbake/lib/bb/tests/cooker.py
> +++ b/bitbake/lib/bb/tests/cooker.py
> @@ -1,20 +1,7 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # BitBake Tests for cooker.py
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
>
> import unittest
> diff --git a/bitbake/lib/bb/tests/cow.py b/bitbake/lib/bb/tests/cow.py
> index d149d84..b4af4bb 100644
> --- a/bitbake/lib/bb/tests/cow.py
> +++ b/bitbake/lib/bb/tests/cow.py
> @@ -1,22 +1,9 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # BitBake Tests for Copy-on-Write (cow.py)
> #
> -# Copyright 2006 Holger Freyther <freyther@handhelds.org>
> -#
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# Copyright 2006 Holger Freyther
> <freyther@handhelds.org> #
>
> import unittest
> diff --git a/bitbake/lib/bb/tests/data.py
> b/bitbake/lib/bb/tests/data.py index db3e201..3e49984 100644
> --- a/bitbake/lib/bb/tests/data.py
> +++ b/bitbake/lib/bb/tests/data.py
> @@ -1,23 +1,10 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # BitBake Tests for the Data Store (data.py/data_smart.py)
> #
> # Copyright (C) 2010 Chris Larson
> # Copyright (C) 2012 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
>
> import unittest
> @@ -394,6 +381,28 @@ class TestOverrides(unittest.TestCase):
> self.d.setVar("OVERRIDES", "foo:bar:some_val")
> self.assertEqual(self.d.getVar("TEST"), " testvalue5")
>
> + def test_append_and_override_1(self):
> + self.d.setVar("TEST_append", "testvalue2")
> + self.d.setVar("TEST_bar", "testvalue3")
> + self.assertEqual(self.d.getVar("TEST"),
> "testvalue3testvalue2") +
> + def test_append_and_override_2(self):
> + self.d.setVar("TEST_append_bar", "testvalue2")
> + self.assertEqual(self.d.getVar("TEST"),
> "testvaluetestvalue2") +
> + def test_append_and_override_3(self):
> + self.d.setVar("TEST_bar_append", "testvalue2")
> + self.assertEqual(self.d.getVar("TEST"), "testvalue2")
> +
> + # Test an override with _<numeric> in it based on a real world
> OE issue
> + def test_underscore_override(self):
> + self.d.setVar("TARGET_ARCH", "x86_64")
> + self.d.setVar("PN", "test-${TARGET_ARCH}")
> + self.d.setVar("VERSION", "1")
> + self.d.setVar("VERSION_pn-test-${TARGET_ARCH}", "2")
> + self.d.setVar("OVERRIDES", "pn-${PN}")
> + bb.data.expandKeys(self.d)
> + self.assertEqual(self.d.getVar("VERSION"), "2")
>
> class TestKeyExpansion(unittest.TestCase):
> def setUp(self):
> @@ -470,7 +479,7 @@ class TaskHash(unittest.TestCase):
> tasklist, gendeps, lookupcache =
> bb.data.generate_dependencies(d) taskdeps, basehash =
> bb.data.generate_dependency_hash(tasklist, gendeps, lookupcache,
> set(), "somefile") bb.warn(str(lookupcache))
> - return basehash["somefile." + taskname]
> + return basehash["somefile:" + taskname]
>
> d = bb.data.init()
> d.setVar("__BBTASKS", ["mytask"])
> diff --git a/bitbake/lib/bb/tests/event.py
> b/bitbake/lib/bb/tests/event.py index d3a5f62..9229b63 100644
> --- a/bitbake/lib/bb/tests/event.py
> +++ b/bitbake/lib/bb/tests/event.py
> @@ -1,22 +1,9 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # BitBake Tests for the Event implementation (event.py)
> #
> # Copyright (C) 2017 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
>
> import unittest
> @@ -574,14 +561,6 @@ class EventClassesTest(unittest.TestCase):
> self.assertEqual(event.fn(1), callback(1))
> self.assertEqual(event.pid, EventClassesTest._worker_pid)
>
> - def test_StampUpdate(self):
> - targets = ["foo", "bar"]
> - stampfns = [lambda:"foobar"]
> - event = bb.event.StampUpdate(targets, stampfns)
> - self.assertEqual(event.targets, targets)
> - self.assertEqual(event.stampPrefix, stampfns)
> - self.assertEqual(event.pid, EventClassesTest._worker_pid)
> -
> def test_BuildBase(self):
> """ Test base class for bitbake build events """
> name = "foo"
> diff --git a/bitbake/lib/bb/tests/fetch.py
> b/bitbake/lib/bb/tests/fetch.py index 6848095..a0b656b 100644
> --- a/bitbake/lib/bb/tests/fetch.py
> +++ b/bitbake/lib/bb/tests/fetch.py
> @@ -1,22 +1,9 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # BitBake Tests for the Fetcher (fetch2/)
> #
> # Copyright (C) 2012 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
>
> import unittest
> @@ -893,12 +880,201 @@ class FetcherNetworkTest(FetcherTest):
>
> @skipIfNoNetwork()
> def test_git_submodule(self):
> - fetcher =
> bb.fetch.Fetch(["gitsm://git.yoctoproject.org/git-submodule-test;rev=f12e57f2edf0aa534cf1616fa983d165a92b0842"],
> self.d)
> + # URL with ssh submodules
> + url =
> "gitsm://git.yoctoproject.org/git-submodule-test;branch=ssh-gitsm-tests;rev=049da4a6cb198d7c0302e9e8b243a1443cb809a7"
> + # Original URL (comment this if you have ssh access to
> git.yoctoproject.org)
> + url =
> "gitsm://git.yoctoproject.org/git-submodule-test;branch=master;rev=a2885dd7d25380d23627e7544b7bbb55014b16ee"
> + fetcher = bb.fetch.Fetch([url], self.d)
> + fetcher.download()
> + # Previous cwd has been deleted
> + os.chdir(os.path.dirname(self.unpackdir))
> + fetcher.unpack(self.unpackdir)
> +
> + repo_path = os.path.join(self.tempdir, 'unpacked', 'git')
> + self.assertTrue(os.path.exists(repo_path), msg='Unpacked
> repository missing')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> 'bitbake')), msg='bitbake submodule missing')
> + self.assertFalse(os.path.exists(os.path.join(repo_path,
> 'na')), msg='uninitialized submodule present') +
> + # Only when we're running the extended test with a
> submodule's submodule, can we check this.
> + if os.path.exists(os.path.join(repo_path,
> 'bitbake-gitsm-test1')):
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> 'bitbake-gitsm-test1', 'bitbake')), msg='submodule of submodule
> missing') +
> + @skipIfNoNetwork()
> + def test_git_submodule_dbus_broker(self):
> + # The following external repositories have show failures in
> fetch and unpack operations
> + # We want to avoid regressions!
> + url =
> "gitsm://github.com/bus1/dbus-broker;protocol=git;rev=fc874afa0992d0c75ec25acb43d344679f0ee7d2"
> + fetcher = bb.fetch.Fetch([url], self.d)
> + fetcher.download()
> + # Previous cwd has been deleted
> + os.chdir(os.path.dirname(self.unpackdir))
> + fetcher.unpack(self.unpackdir)
> +
> + repo_path = os.path.join(self.tempdir, 'unpacked', 'git')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> '.git/modules/subprojects/c-dvar/config')), msg='Missing submodule
> config "subprojects/c-dvar"')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> '.git/modules/subprojects/c-list/config')), msg='Missing submodule
> config "subprojects/c-list"')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> '.git/modules/subprojects/c-rbtree/config')), msg='Missing submodule
> config "subprojects/c-rbtree"')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> '.git/modules/subprojects/c-sundry/config')), msg='Missing submodule
> config "subprojects/c-sundry"')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> '.git/modules/subprojects/c-utf8/config')), msg='Missing submodule
> config "subprojects/c-utf8"') +
> + @skipIfNoNetwork()
> + def test_git_submodule_CLI11(self):
> + url =
> "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=bd4dc911847d0cde7a6b41dfa626a85aab213baf"
> + fetcher = bb.fetch.Fetch([url], self.d)
> + fetcher.download()
> + # Previous cwd has been deleted
> + os.chdir(os.path.dirname(self.unpackdir))
> + fetcher.unpack(self.unpackdir)
> +
> + repo_path = os.path.join(self.tempdir, 'unpacked', 'git')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> '.git/modules/extern/googletest/config')), msg='Missing submodule
> config "extern/googletest"')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> '.git/modules/extern/json/config')), msg='Missing submodule config
> "extern/json"')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> '.git/modules/extern/sanitizers/config')), msg='Missing submodule
> config "extern/sanitizers"') +
> + @skipIfNoNetwork()
> + def test_git_submodule_update_CLI11(self):
> + """ Prevent regression on update detection not finding
> missing submodule, or modules without needed commits """
> + url =
> "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=cf6a99fa69aaefe477cc52e3ef4a7d2d7fa40714"
> + fetcher = bb.fetch.Fetch([url], self.d)
> + fetcher.download()
> +
> + # CLI11 that pulls in a newer nlohmann-json
> + url =
> "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=49ac989a9527ee9bb496de9ded7b4872c2e0e5ca"
> + fetcher = bb.fetch.Fetch([url], self.d)
> + fetcher.download()
> + # Previous cwd has been deleted
> + os.chdir(os.path.dirname(self.unpackdir))
> + fetcher.unpack(self.unpackdir)
> +
> + repo_path = os.path.join(self.tempdir, 'unpacked', 'git')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> '.git/modules/extern/googletest/config')), msg='Missing submodule
> config "extern/googletest"')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> '.git/modules/extern/json/config')), msg='Missing submodule config
> "extern/json"')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> '.git/modules/extern/sanitizers/config')), msg='Missing submodule
> config "extern/sanitizers"') +
> + @skipIfNoNetwork()
> + def test_git_submodule_aktualizr(self):
> + url =
> "gitsm://github.com/advancedtelematic/aktualizr;branch=master;protocol=git;rev=d00d1a04cc2366d1a5f143b84b9f507f8bd32c44"
> + fetcher = bb.fetch.Fetch([url], self.d)
> fetcher.download()
> # Previous cwd has been deleted
> os.chdir(os.path.dirname(self.unpackdir))
> fetcher.unpack(self.unpackdir)
>
> + repo_path = os.path.join(self.tempdir, 'unpacked', 'git')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> '.git/modules/partial/extern/isotp-c/config')), msg='Missing
> submodule config "partial/extern/isotp-c/config"')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> '.git/modules/partial/extern/isotp-c/modules/deps/bitfield-c/config')),
> msg='Missing submodule config
> "partial/extern/isotp-c/modules/deps/bitfield-c/config"')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> 'partial/extern/isotp-c/deps/bitfield-c/.git')), msg="Submodule of
> submodule isotp-c did not unpack properly")
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> '.git/modules/tests/tuf-test-vectors/config')), msg='Missing
> submodule config "tests/tuf-test-vectors/config"')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> '.git/modules/third_party/googletest/config')), msg='Missing
> submodule config "third_party/googletest/config"')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> '.git/modules/third_party/HdrHistogram_c/config')), msg='Missing
> submodule config "third_party/HdrHistogram_c/config"') +
> + @skipIfNoNetwork()
> + def test_git_submodule_iotedge(self):
> + """ Prevent regression on deeply nested submodules not being
> checked out properly, even though they were fetched. """ +
> + # This repository also has submodules where the module
> (name), path and url do not align
> + url =
> "gitsm://github.com/azure/iotedge.git;protocol=git;rev=d76e0316c6f324345d77c48a83ce836d09392699"
> + fetcher = bb.fetch.Fetch([url], self.d)
> + fetcher.download()
> + # Previous cwd has been deleted
> + os.chdir(os.path.dirname(self.unpackdir))
> + fetcher.unpack(self.unpackdir)
> +
> + repo_path = os.path.join(self.tempdir, 'unpacked', 'git')
> +
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared/README.md')),
> msg='Missing submodule checkout')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared/testtools/ctest/README.md')),
> msg='Missing submodule checkout')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared/testtools/testrunner/readme.md')),
> msg='Missing submodule checkout')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared/testtools/umock-c/readme.md')),
> msg='Missing submodule checkout')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared/testtools/umock-c/deps/ctest/README.md')),
> msg='Missing submodule checkout')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared/testtools/umock-c/deps/testrunner/readme.md')),
> msg='Missing submodule checkout')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/utpm/README.md')), msg='Missing
> submodule checkout')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/utpm/deps/c-utility/README.md')),
> msg='Missing submodule checkout')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/utpm/deps/c-utility/testtools/ctest/README.md')),
> msg='Missing submodule checkout')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/utpm/deps/c-utility/testtools/testrunner/readme.md')),
> msg='Missing submodule checkout')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/utpm/deps/c-utility/testtools/umock-c/readme.md')),
> msg='Missing submodule checkout')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/utpm/deps/c-utility/testtools/umock-c/deps/ctest/README.md')),
> msg='Missing submodule checkout')
> + self.assertTrue(os.path.exists(os.path.join(repo_path,
> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/utpm/deps/c-utility/testtools/umock-c/deps/testrunner/readme.md')),
> msg='Missing submodule checkout') + +class SVNTest(FetcherTest):
> + def skipIfNoSvn():
> + import shutil
> + if not shutil.which("svn"):
> + return unittest.skip("svn not installed, tests being
> skipped") +
> + if not shutil.which("svnadmin"):
> + return unittest.skip("svnadmin not installed, tests
> being skipped") +
> + return lambda f: f
> +
> + @skipIfNoSvn()
> + def setUp(self):
> + """ Create a local repository """
> +
> + super(SVNTest, self).setUp()
> +
> + # Create something we can fetch
> + src_dir = tempfile.mkdtemp(dir=self.tempdir,
> + prefix='svnfetch_srcdir_')
> + src_dir = os.path.abspath(src_dir)
> + bb.process.run("echo readme > README.md", cwd=src_dir)
> +
> + # Store it in a local SVN repository
> + repo_dir = tempfile.mkdtemp(dir=self.tempdir,
> + prefix='svnfetch_localrepo_')
> + repo_dir = os.path.abspath(repo_dir)
> + bb.process.run("svnadmin create project", cwd=repo_dir)
> +
> + self.repo_url = "file://%s/project" % repo_dir
> + bb.process.run("svn import --non-interactive -m 'Initial
> import' %s %s/trunk" % (src_dir, self.repo_url),
> + cwd=repo_dir)
> +
> + bb.process.run("svn co %s svnfetch_co" % self.repo_url,
> cwd=self.tempdir)
> + # Github will emulate SVN. Use this to check if we're
> downloding...
> + bb.process.run("svn propset svn:externals 'bitbake
> http://github.com/openembedded/bitbake' .",
> + cwd=os.path.join(self.tempdir, 'svnfetch_co',
> 'trunk'))
> + bb.process.run("svn commit --non-interactive -m 'Add
> external'",
> + cwd=os.path.join(self.tempdir, 'svnfetch_co',
> 'trunk')) +
> + self.src_dir = src_dir
> + self.repo_dir = repo_dir
> +
> + @skipIfNoSvn()
> + def tearDown(self):
> + os.chdir(self.origdir)
> + if os.environ.get("BB_TMPDIR_NOCLEAN") == "yes":
> + print("Not cleaning up %s. Please remove manually." %
> self.tempdir)
> + else:
> + bb.utils.prunedir(self.tempdir)
> +
> + @skipIfNoSvn()
> + @skipIfNoNetwork()
> + def test_noexternal_svn(self):
> + # Always match the rev count from setUp (currently rev 2)
> + url = "svn://%s;module=trunk;protocol=file;rev=2" %
> self.repo_url.replace('file://', '')
> + fetcher = bb.fetch.Fetch([url], self.d)
> + fetcher.download()
> + os.chdir(os.path.dirname(self.unpackdir))
> + fetcher.unpack(self.unpackdir)
> +
> + self.assertTrue(os.path.exists(os.path.join(self.unpackdir,
> 'trunk')), msg="Missing trunk")
> + self.assertTrue(os.path.exists(os.path.join(self.unpackdir,
> 'trunk', 'README.md')), msg="Missing contents")
> + self.assertFalse(os.path.exists(os.path.join(self.unpackdir,
> 'trunk/bitbake/trunk')), msg="External dir should NOT exist")
> + self.assertFalse(os.path.exists(os.path.join(self.unpackdir,
> 'trunk/bitbake/trunk', 'README')), msg="External README should NOT
> exit") +
> + @skipIfNoSvn()
> + def test_external_svn(self):
> + # Always match the rev count from setUp (currently rev 2)
> + url =
> "svn://%s;module=trunk;protocol=file;externals=allowed;rev=2" %
> self.repo_url.replace('file://', '')
> + fetcher = bb.fetch.Fetch([url], self.d)
> + fetcher.download()
> + os.chdir(os.path.dirname(self.unpackdir))
> + fetcher.unpack(self.unpackdir)
> +
> + self.assertTrue(os.path.exists(os.path.join(self.unpackdir,
> 'trunk')), msg="Missing trunk")
> + self.assertTrue(os.path.exists(os.path.join(self.unpackdir,
> 'trunk', 'README.md')), msg="Missing contents")
> + self.assertTrue(os.path.exists(os.path.join(self.unpackdir,
> 'trunk/bitbake/trunk')), msg="External dir should exist")
> + self.assertTrue(os.path.exists(os.path.join(self.unpackdir,
> 'trunk/bitbake/trunk', 'README')), msg="External README should exit")
> class TrustedNetworksTest(FetcherTest):
> def test_trusted_network(self):
> @@ -1024,8 +1200,8 @@ class FetchLatestVersionTest(FetcherTest):
> # packages with valid UPSTREAM_CHECK_URI and
> UPSTREAM_CHECK_REGEX ("cups",
> "http://www.cups.org/software/1.7.2/cups-1.7.2-source.tar.bz2",
> "https://github.com/apple/cups/releases",
> "(?P<name>cups\-)(?P<pver>((\d+[\.\-_]*)+))\-source\.tar\.gz") :
> "2.0.0",
> - ("db",
> "http://download.oracle.com/berkeley-db/db-5.3.21.tar.gz",
> "http://www.oracle.com/technetwork/products/berkeleydb/downloads/index-082944.html",
> "http://download.oracle.com/otn/berkeley-db/(?P<name>db-)(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz")
> - : "6.1.19",
> + ("db",
> "http://download.oracle.com/berkeley-db/db-5.3.21.tar.gz",
> "http://ftp.debian.org/debian/pool/main/d/db5.3/",
> "(?P<name>db5\.3_)(?P<pver>\d+(\.\d+)+).+\.orig\.tar\.xz")
> + : "5.3.10",
> }
>
> @skipIfNoNetwork()
> @@ -1280,7 +1456,7 @@ class GitShallowTest(FetcherTest):
>
> def fetch(self, uri=None):
> if uri is None:
> - uris = self.d.getVar('SRC_URI', True).split()
> + uris = self.d.getVar('SRC_URI').split()
> uri = uris[0]
> d = self.d
> else:
> @@ -1312,6 +1488,7 @@ class GitShallowTest(FetcherTest):
> # fetch and unpack, from the shallow tarball
> bb.utils.remove(self.gitdir, recurse=True)
> bb.utils.remove(ud.clonedir, recurse=True)
> + bb.utils.remove(ud.clonedir.replace('gitsource',
> 'gitsubmodule'), recurse=True)
> # confirm that the unpacked repo is used when no git clone
> or git # mirror tarball is available
> @@ -1338,7 +1515,7 @@ class GitShallowTest(FetcherTest):
>
> srcrev = self.git('rev-parse HEAD', cwd=self.srcdir).strip()
> self.d.setVar('SRCREV', srcrev)
> - uri = self.d.getVar('SRC_URI', True).split()[0]
> + uri = self.d.getVar('SRC_URI').split()[0]
> uri = '%s;nobranch=1;bare=1' % uri
>
> self.fetch_shallow(uri)
> @@ -1466,6 +1643,7 @@ class GitShallowTest(FetcherTest):
> self.git('config --add remote.origin.url "%s"' % smdir,
> cwd=smdir) self.git('config --add remote.origin.fetch
> "+refs/heads/*:refs/remotes/origin/*"', cwd=smdir)
> self.add_empty_file('asub', cwd=smdir)
> + self.add_empty_file('bsub', cwd=smdir)
>
> self.git('submodule init', cwd=self.srcdir)
> self.git('submodule add file://%s' % smdir, cwd=self.srcdir)
> @@ -1475,10 +1653,16 @@ class GitShallowTest(FetcherTest):
> uri = 'gitsm://%s;protocol=file;subdir=${S}' % self.srcdir
> fetcher, ud = self.fetch_shallow(uri)
>
> + # Verify the main repository is shallow
> self.assertRevCount(1)
> - assert './.git/modules/' in bb.process.run('tar -tzf %s' %
> os.path.join(self.dldir, ud.mirrortarballs[0]))[0] +
> + # Verify the gitsubmodule directory is present
> assert os.listdir(os.path.join(self.gitdir, 'gitsubmodule'))
>
> + # Verify the submodule is also shallow
> + self.assertRevCount(1, cwd=os.path.join(self.gitdir,
> 'gitsubmodule')) +
> +
> if any(os.path.exists(os.path.join(p, 'git-annex')) for p in
> os.environ.get('PATH').split(':')): def test_shallow_annex(self):
> self.add_empty_file('a')
> @@ -1510,7 +1694,7 @@ class GitShallowTest(FetcherTest):
> self.add_empty_file('f')
> self.assertRevCount(7, cwd=self.srcdir)
>
> - uri = self.d.getVar('SRC_URI', True).split()[0]
> + uri = self.d.getVar('SRC_URI').split()[0]
> uri = '%s;branch=master,a_branch;name=master,a_branch' % uri
>
> self.d.setVar('BB_GIT_SHALLOW_DEPTH', '0')
> @@ -1536,7 +1720,7 @@ class GitShallowTest(FetcherTest):
> self.add_empty_file('f')
> self.assertRevCount(7, cwd=self.srcdir)
>
> - uri = self.d.getVar('SRC_URI', True).split()[0]
> + uri = self.d.getVar('SRC_URI').split()[0]
> uri = '%s;branch=master,a_branch;name=master,a_branch' % uri
>
> self.d.setVar('BB_GIT_SHALLOW_DEPTH', '0')
> @@ -1724,3 +1908,83 @@ class GitShallowTest(FetcherTest):
>
> dir = os.listdir(self.unpackdir + "/git/")
> self.assertIn("fstests.doap", dir)
> +
> +class GitLfsTest(FetcherTest):
> + def setUp(self):
> + FetcherTest.setUp(self)
> +
> + self.gitdir = os.path.join(self.tempdir, 'git')
> + self.srcdir = os.path.join(self.tempdir, 'gitsource')
> +
> + self.d.setVar('WORKDIR', self.tempdir)
> + self.d.setVar('S', self.gitdir)
> + self.d.delVar('PREMIRRORS')
> + self.d.delVar('MIRRORS')
> +
> + self.d.setVar('SRCREV', '${AUTOREV}')
> + self.d.setVar('AUTOREV', '${@bb.fetch2.get_autorev(d)}')
> +
> + bb.utils.mkdirhier(self.srcdir)
> + self.git('init', cwd=self.srcdir)
> + with open(os.path.join(self.srcdir, '.gitattributes'), 'wt')
> as attrs:
> + attrs.write('*.mp3 filter=lfs -text')
> + self.git(['add', '.gitattributes'], cwd=self.srcdir)
> + self.git(['commit', '-m', "attributes", '.gitattributes'],
> cwd=self.srcdir) +
> + def git(self, cmd, cwd=None):
> + if isinstance(cmd, str):
> + cmd = 'git ' + cmd
> + else:
> + cmd = ['git'] + cmd
> + if cwd is None:
> + cwd = self.gitdir
> + return bb.process.run(cmd, cwd=cwd)[0]
> +
> + def fetch(self, uri=None):
> + uris = self.d.getVar('SRC_URI').split()
> + uri = uris[0]
> + d = self.d
> +
> + fetcher = bb.fetch2.Fetch(uris, d)
> + fetcher.download()
> + ud = fetcher.ud[uri]
> + return fetcher, ud
> +
> + def test_lfs_enabled(self):
> + import shutil
> +
> + uri = 'git://%s;protocol=file;subdir=${S};lfs=1' %
> self.srcdir
> + self.d.setVar('SRC_URI', uri)
> +
> + fetcher, ud = self.fetch()
> + self.assertIsNotNone(ud.method._find_git_lfs)
> +
> + # If git-lfs can be found, the unpack should be successful
> + ud.method._find_git_lfs = lambda d: True
> + shutil.rmtree(self.gitdir, ignore_errors=True)
> + fetcher.unpack(self.d.getVar('WORKDIR'))
> +
> + # If git-lfs cannot be found, the unpack should throw an
> error
> + with self.assertRaises(bb.fetch2.FetchError):
> + ud.method._find_git_lfs = lambda d: False
> + shutil.rmtree(self.gitdir, ignore_errors=True)
> + fetcher.unpack(self.d.getVar('WORKDIR'))
> +
> + def test_lfs_disabled(self):
> + import shutil
> +
> + uri = 'git://%s;protocol=file;subdir=${S};lfs=0' %
> self.srcdir
> + self.d.setVar('SRC_URI', uri)
> +
> + fetcher, ud = self.fetch()
> + self.assertIsNotNone(ud.method._find_git_lfs)
> +
> + # If git-lfs can be found, the unpack should be successful
> + ud.method._find_git_lfs = lambda d: True
> + shutil.rmtree(self.gitdir, ignore_errors=True)
> + fetcher.unpack(self.d.getVar('WORKDIR'))
> +
> + # If git-lfs cannot be found, the unpack should be successful
> + ud.method._find_git_lfs = lambda d: False
> + shutil.rmtree(self.gitdir, ignore_errors=True)
> + fetcher.unpack(self.d.getVar('WORKDIR'))
> diff --git a/bitbake/lib/bb/tests/parse.py
> b/bitbake/lib/bb/tests/parse.py index 1bc4740..9afd1b2 100644
> --- a/bitbake/lib/bb/tests/parse.py
> +++ b/bitbake/lib/bb/tests/parse.py
> @@ -1,22 +1,9 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # BitBake Test for lib/bb/parse/
> #
> # Copyright (C) 2015 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
>
> import unittest
> @@ -187,3 +174,21 @@ python () {
> self.assertEqual(d1.getVar("VAR_var"), "B")
> self.assertEqual(d2.getVar("VAR_var"), None)
>
> + addtask_deltask = """
> +addtask do_patch after do_foo after do_unpack before do_configure
> before do_compile +addtask do_fetch do_patch
> +
> +deltask do_fetch do_patch
> +"""
> + def test_parse_addtask_deltask(self):
> + import sys
> + f = self.parsehelper(self.addtask_deltask)
> + d = bb.parse.handle(f.name, self.d)['']
> +
> + stdout = sys.stdout.getvalue()
> + self.assertTrue("addtask contained multiple 'before'
> keywords" in stdout)
> + self.assertTrue("addtask contained multiple 'after'
> keywords" in stdout)
> + self.assertTrue('addtask ignored: " do_patch"' in stdout)
> + self.assertTrue('deltask ignored: " do_patch"' in stdout)
> + #self.assertTrue('dependent task do_foo for do_patch does
> not exist' in stdout) +
> diff --git a/bitbake/lib/bb/tests/persist_data.py
> b/bitbake/lib/bb/tests/persist_data.py new file mode 100644
> index 0000000..f641b5a
> --- /dev/null
> +++ b/bitbake/lib/bb/tests/persist_data.py
> @@ -0,0 +1,129 @@
> +#
> +# BitBake Test for lib/bb/persist_data/
> +#
> +# Copyright (C) 2018 Garmin Ltd.
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> +import unittest
> +import bb.data
> +import bb.persist_data
> +import tempfile
> +import threading
> +
> +class PersistDataTest(unittest.TestCase):
> + def _create_data(self):
> + return bb.persist_data.persist('TEST_PERSIST_DATA', self.d)
> +
> + def setUp(self):
> + self.d = bb.data.init()
> + self.tempdir = tempfile.TemporaryDirectory()
> + self.d['PERSISTENT_DIR'] = self.tempdir.name
> + self.data = self._create_data()
> + self.items = {
> + 'A1': '1',
> + 'B1': '2',
> + 'C2': '3'
> + }
> + self.stress_count = 10000
> + self.thread_count = 5
> +
> + for k,v in self.items.items():
> + self.data[k] = v
> +
> + def tearDown(self):
> + self.tempdir.cleanup()
> +
> + def _iter_helper(self, seen, iterator):
> + with iter(iterator):
> + for v in iterator:
> + self.assertTrue(v in seen)
> + seen.remove(v)
> + self.assertEqual(len(seen), 0, '%s not seen' % seen)
> +
> + def test_get(self):
> + for k, v in self.items.items():
> + self.assertEqual(self.data[k], v)
> +
> + self.assertIsNone(self.data.get('D'))
> + with self.assertRaises(KeyError):
> + self.data['D']
> +
> + def test_set(self):
> + for k, v in self.items.items():
> + self.data[k] += '-foo'
> +
> + for k, v in self.items.items():
> + self.assertEqual(self.data[k], v + '-foo')
> +
> + def test_delete(self):
> + self.data['D'] = '4'
> + self.assertEqual(self.data['D'], '4')
> + del self.data['D']
> + self.assertIsNone(self.data.get('D'))
> + with self.assertRaises(KeyError):
> + self.data['D']
> +
> + def test_contains(self):
> + for k in self.items:
> + self.assertTrue(k in self.data)
> + self.assertTrue(self.data.has_key(k))
> + self.assertFalse('NotFound' in self.data)
> + self.assertFalse(self.data.has_key('NotFound'))
> +
> + def test_len(self):
> + self.assertEqual(len(self.data), len(self.items))
> +
> + def test_iter(self):
> + self._iter_helper(set(self.items.keys()), self.data)
> +
> + def test_itervalues(self):
> + self._iter_helper(set(self.items.values()),
> self.data.itervalues()) +
> + def test_iteritems(self):
> + self._iter_helper(set(self.items.items()),
> self.data.iteritems()) +
> + def test_get_by_pattern(self):
> + self._iter_helper({'1', '2'}, self.data.get_by_pattern('_1'))
> +
> + def _stress_read(self, data):
> + for i in range(self.stress_count):
> + for k in self.items:
> + data[k]
> +
> + def _stress_write(self, data):
> + for i in range(self.stress_count):
> + for k, v in self.items.items():
> + data[k] = v + str(i)
> +
> + def _validate_stress(self):
> + for k, v in self.items.items():
> + self.assertEqual(self.data[k], v + str(self.stress_count
> - 1)) +
> + def test_stress(self):
> + self._stress_read(self.data)
> + self._stress_write(self.data)
> + self._validate_stress()
> +
> + def test_stress_threads(self):
> + def read_thread():
> + data = self._create_data()
> + self._stress_read(data)
> +
> + def write_thread():
> + data = self._create_data()
> + self._stress_write(data)
> +
> + threads = []
> + for i in range(self.thread_count):
> + threads.append(threading.Thread(target=read_thread))
> + threads.append(threading.Thread(target=write_thread))
> +
> + for t in threads:
> + t.start()
> + self._stress_read(self.data)
> + for t in threads:
> + t.join()
> + self._validate_stress()
> +
> diff --git a/bitbake/lib/bb/tests/runqueue-tests/classes/base.bbclass
> b/bitbake/lib/bb/tests/runqueue-tests/classes/base.bbclass new file
> mode 100644 index 0000000..b57650d
> --- /dev/null
> +++ b/bitbake/lib/bb/tests/runqueue-tests/classes/base.bbclass
> @@ -0,0 +1,262 @@
> +SLOWTASKS ??= ""
> +SSTATEVALID ??= ""
> +
> +def stamptask(d):
> + import time
> +
> + thistask = d.expand("${PN}:${BB_CURRENTTASK}")
> + stampname = d.expand("${TOPDIR}/%s.run" % thistask)
> + with open(stampname, "a+") as f:
> + f.write(d.getVar("BB_UNIHASH") + "\n")
> +
> + if d.getVar("BB_CURRENT_MC") != "default":
> + thistask =
> d.expand("${BB_CURRENT_MC}:${PN}:${BB_CURRENTTASK}")
> + if thistask in d.getVar("SLOWTASKS").split():
> + bb.note("Slowing task %s" % thistask)
> + time.sleep(0.5)
> + if d.getVar("BB_HASHSERVE"):
> + task = d.getVar("BB_CURRENTTASK")
> + if task in ['package', 'package_qa', 'packagedata',
> 'package_write_ipk', 'package_write_rpm', 'populate_lic',
> 'populate_sysroot']:
> + bb.parse.siggen.report_unihash(os.getcwd(),
> d.getVar("BB_CURRENTTASK"), d) +
> + with open(d.expand("${TOPDIR}/task.log"), "a+") as f:
> + f.write(thistask + "\n")
> +
> +
> +def sstate_output_hash(path, sigfile, task, d):
> + import hashlib
> + h = hashlib.sha256()
> + h.update(d.expand("${PN}:${BB_CURRENTTASK}").encode('utf-8'))
> + return h.hexdigest()
> +
> +python do_fetch() {
> + # fetch
> + stamptask(d)
> +}
> +python do_unpack() {
> + # unpack
> + stamptask(d)
> +}
> +python do_patch() {
> + # patch
> + stamptask(d)
> +}
> +python do_populate_lic() {
> + # populate_lic
> + stamptask(d)
> +}
> +python do_prepare_recipe_sysroot() {
> + # prepare_recipe_sysroot
> + stamptask(d)
> +}
> +python do_configure() {
> + # configure
> + stamptask(d)
> +}
> +python do_compile() {
> + # compile
> + stamptask(d)
> +}
> +python do_install() {
> + # install
> + stamptask(d)
> +}
> +python do_populate_sysroot() {
> + # populate_sysroot
> + stamptask(d)
> +}
> +python do_package() {
> + # package
> + stamptask(d)
> +}
> +python do_package_write_ipk() {
> + # package_write_ipk
> + stamptask(d)
> +}
> +python do_package_write_rpm() {
> + # package_write_rpm
> + stamptask(d)
> +}
> +python do_packagedata() {
> + # packagedata
> + stamptask(d)
> +}
> +python do_package_qa() {
> + # package_qa
> + stamptask(d)
> +}
> +python do_build() {
> + # build
> + stamptask(d)
> +}
> +do_prepare_recipe_sysroot[deptask] = "do_populate_sysroot"
> +do_package[deptask] += "do_packagedata"
> +do_build[recrdeptask] += "do_deploy"
> +do_build[recrdeptask] += "do_package_write_ipk"
> +do_build[recrdeptask] += "do_package_write_rpm"
> +do_package_qa[rdeptask] = "do_packagedata"
> +do_populate_lic_deploy[recrdeptask] += "do_populate_lic do_deploy"
> +
> +DEBIANRDEP = "do_packagedata"
> +oo_package_write_ipk[rdeptask] = "${DEBIANRDEP}"
> +do_package_write_rpm[rdeptask] = "${DEBIANRDEP}"
> +
> +addtask fetch
> +addtask unpack after do_fetch
> +addtask patch after do_unpack
> +addtask prepare_recipe_sysroot after do_patch
> +addtask configure after do_prepare_recipe_sysroot
> +addtask compile after do_configure
> +addtask install after do_compile
> +addtask populate_sysroot after do_install
> +addtask package after do_install
> +addtask package_write_ipk after do_packagedata do_package
> +addtask package_write_rpm after do_packagedata do_package
> +addtask packagedata after do_package
> +addtask package_qa after do_package
> +addtask build after do_package_qa do_package_write_rpm
> do_package_write_ipk do_populate_sysroot +
> +python do_package_setscene() {
> + stamptask(d)
> +}
> +python do_package_qa_setscene() {
> + stamptask(d)
> +}
> +python do_package_write_ipk_setscene() {
> + stamptask(d)
> +}
> +python do_package_write_rpm_setscene() {
> + stamptask(d)
> +}
> +python do_packagedata_setscene() {
> + stamptask(d)
> +}
> +python do_populate_lic_setscene() {
> + stamptask(d)
> +}
> +python do_populate_sysroot_setscene() {
> + stamptask(d)
> +}
> +
> +addtask package_setscene
> +addtask package_qa_setscene
> +addtask package_write_ipk_setscene
> +addtask package_write_rpm_setscene
> +addtask packagedata_setscene
> +addtask populate_lic_setscene
> +addtask populate_sysroot_setscene
> +
> +BB_SETSCENE_DEPVALID = "setscene_depvalid"
> +
> +def setscene_depvalid(task, taskdependees, notneeded, d, log=None):
> + # taskdependees is a dict of tasks which depend on task, each
> being a 3 item list of [PN, TASKNAME, FILENAME]
> + # task is included in taskdependees too
> + # Return - False - We need this dependency
> + # - True - We can skip this dependency
> + import re
> +
> + def logit(msg, log):
> + if log is not None:
> + log.append(msg)
> + else:
> + bb.debug(2, msg)
> +
> + logit("Considering setscene task: %s" %
> (str(taskdependees[task])), log) +
> + def isNativeCross(x):
> + return x.endswith("-native") or "-cross-" in x or
> "-crosssdk" in x or x.endswith("-cross") +
> + # We only need to trigger populate_lic through direct
> dependencies
> + if taskdependees[task][1] == "do_populate_lic":
> + return True
> +
> + # We only need to trigger packagedata through direct dependencies
> + # but need to preserve packagedata on packagedata links
> + if taskdependees[task][1] == "do_packagedata":
> + for dep in taskdependees:
> + if taskdependees[dep][1] == "do_packagedata":
> + return False
> + return True
> +
> + for dep in taskdependees:
> + logit(" considering dependency: %s" %
> (str(taskdependees[dep])), log)
> + if task == dep:
> + continue
> + if dep in notneeded:
> + continue
> + # do_package_write_* and do_package doesn't need do_package
> + if taskdependees[task][1] == "do_package" and
> taskdependees[dep][1] in ['do_package', 'do_package_write_ipk',
> 'do_package_write_rpm', 'do_packagedata', 'do_package_qa']:
> + continue
> + # do_package_write_* need do_populate_sysroot as they're
> mainly postinstall dependencies
> + if taskdependees[task][1] == "do_populate_sysroot" and
> taskdependees[dep][1] in ['do_package_write_ipk',
> 'do_package_write_rpm']:
> + return False
> + # do_package/packagedata/package_qa don't need
> do_populate_sysroot
> + if taskdependees[task][1] == "do_populate_sysroot" and
> taskdependees[dep][1] in ['do_package', 'do_packagedata',
> 'do_package_qa']:
> + continue
> + # Native/Cross packages don't exist and are noexec anyway
> + if isNativeCross(taskdependees[dep][0]) and
> taskdependees[dep][1] in ['do_package_write_ipk',
> 'do_package_write_rpm', 'do_packagedata', 'do_package',
> 'do_package_qa']:
> + continue
> +
> + # This is due to the [depends] in useradd.bbclass
> complicating matters
> + # The logic *is* reversed here due to the way hard setscene
> dependencies are injected
> + if (taskdependees[task][1] == 'do_package' or
> taskdependees[task][1] == 'do_populate_sysroot') and
> taskdependees[dep][0].endswith(('shadow-native', 'shadow-sysroot',
> 'base-passwd', 'pseudo-native')) and taskdependees[dep][1] ==
> 'do_populate_sysroot':
> + continue
> +
> + # Consider sysroot depending on sysroot tasks
> + if taskdependees[task][1] == 'do_populate_sysroot' and
> taskdependees[dep][1] == 'do_populate_sysroot':
> + # Native/Cross populate_sysroot need their dependencies
> + if isNativeCross(taskdependees[task][0]) and
> isNativeCross(taskdependees[dep][0]):
> + return False
> + # Target populate_sysroot depended on by cross tools
> need to be installed
> + if isNativeCross(taskdependees[dep][0]):
> + return False
> + # Native/cross tools depended upon by target sysroot are
> not needed
> + # Add an exception for shadow-native as required by
> useradd.bbclass
> + if isNativeCross(taskdependees[task][0]) and
> taskdependees[task][0] != 'shadow-native':
> + continue
> + # Target populate_sysroot need their dependencies
> + return False
> +
> +
> + if taskdependees[dep][1] == "do_populate_lic":
> + continue
> +
> + # Safe fallthrough default
> + logit(" Default setscene dependency fall through due to
> dependency: %s" % (str(taskdependees[dep])), log)
> + return False
> + return True
> +
> +BB_HASHCHECK_FUNCTION = "sstate_checkhashes"
> +
> +def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0,
> **kwargs): +
> + found = set()
> + missed = set()
> +
> + valid = d.getVar("SSTATEVALID").split()
> +
> + for tid in sorted(sq_data['hash']):
> + n =
> os.path.basename(bb.runqueue.fn_from_tid(tid)).split(".")[0] + ":do_"
> + bb.runqueue.taskname_from_tid(tid)[3:]
> + print(n)
> + stampfile = d.expand("${TOPDIR}/%s.run" % n.replace("do_",
> ""))
> + if n in valid:
> + bb.note("SState: Found valid sstate for %s" % n)
> + found.add(tid)
> + elif n + ":" + sq_data['hash'][tid] in valid:
> + bb.note("SState: Found valid sstate for %s" % n)
> + found.add(tid)
> + elif os.path.exists(stampfile):
> + with open(stampfile, "r") as f:
> + hash = f.readline().strip()
> + if hash == sq_data['hash'][tid]:
> + bb.note("SState: Found valid sstate for %s (already
> run)" % n)
> + found.add(tid)
> + else:
> + bb.note("SState: sstate hash didn't match previous
> run for %s (%s vs %s)" % (n, sq_data['hash'][tid], hash))
> + missed.add(tid)
> + else:
> + missed.add(tid)
> + bb.note("SState: Found no valid sstate for %s (%s)" %
> (n, sq_data['hash'][tid])) +
> + return found
> +
> diff --git
> a/bitbake/lib/bb/tests/runqueue-tests/classes/image.bbclass
> b/bitbake/lib/bb/tests/runqueue-tests/classes/image.bbclass new file
> mode 100644 index 0000000..da9ff11 --- /dev/null
> +++ b/bitbake/lib/bb/tests/runqueue-tests/classes/image.bbclass
> @@ -0,0 +1,5 @@
> +do_rootfs[recrdeptask] += "do_package_write_deb do_package_qa"
> +do_rootfs[recrdeptask] += "do_package_write_ipk do_package_qa"
> +do_rootfs[recrdeptask] += "do_package_write_rpm do_package_qa
> +do_rootfs[recrdeptask] += "do_packagedata"
> +do_rootfs[recrdeptask] += "do_populate_lic"
> diff --git
> a/bitbake/lib/bb/tests/runqueue-tests/classes/native.bbclass
> b/bitbake/lib/bb/tests/runqueue-tests/classes/native.bbclass new file
> mode 100644 index 0000000..7eaaee5 --- /dev/null
> +++ b/bitbake/lib/bb/tests/runqueue-tests/classes/native.bbclass
> @@ -0,0 +1,2 @@
> +RECIPERDEPTASK = "do_populate_sysroot"
> +do_populate_sysroot[rdeptask] = "${RECIPERDEPTASK}"
> diff --git a/bitbake/lib/bb/tests/runqueue-tests/conf/bitbake.conf
> b/bitbake/lib/bb/tests/runqueue-tests/conf/bitbake.conf new file mode
> 100644 index 0000000..5e451fc
> --- /dev/null
> +++ b/bitbake/lib/bb/tests/runqueue-tests/conf/bitbake.conf
> @@ -0,0 +1,16 @@
> +CACHE = "${TOPDIR}/cache"
> +THISDIR = "${@os.path.dirname(d.getVar('FILE'))}"
> +COREBASE :=
> "${@os.path.normpath(os.path.dirname(d.getVar('FILE')+'/../../'))}"
> +BBFILES = "${COREBASE}/recipes/*.bb" +PROVIDES = "${PN}"
> +PN = "${@bb.parse.vars_from_file(d.getVar('FILE', False),d)[0]}"
> +PF = "${BB_CURRENT_MC}:${PN}"
> +export PATH
> +TMPDIR ??= "${TOPDIR}"
> +STAMP = "${TMPDIR}/stamps/${PN}"
> +T = "${TMPDIR}/workdir/${PN}/temp"
> +BB_NUMBER_THREADS = "4"
> +
> +BB_HASHBASE_WHITELIST = "BB_CURRENT_MC BB_HASHSERVE TMPDIR TOPDIR
> SLOWTASKS SSTATEVALID FILE" +
> +include conf/multiconfig/${BB_CURRENT_MC}.conf
> diff --git
> a/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc1.conf
> b/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc1.conf new
> file mode 100644 index 0000000..ecf23e1 --- /dev/null
> +++ b/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc1.conf
> @@ -0,0 +1 @@
> +TMPDIR = "${TOPDIR}/mc1/"
> diff --git
> a/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc2.conf
> b/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc2.conf new
> file mode 100644 index 0000000..eef338e --- /dev/null
> +++ b/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc2.conf
> @@ -0,0 +1 @@
> +TMPDIR = "${TOPDIR}/mc2/"
> diff --git a/bitbake/lib/bb/tests/runqueue-tests/recipes/a1.bb
> b/bitbake/lib/bb/tests/runqueue-tests/recipes/a1.bb new file mode
> 100644 index 0000000..e69de29
> diff --git a/bitbake/lib/bb/tests/runqueue-tests/recipes/b1.bb
> b/bitbake/lib/bb/tests/runqueue-tests/recipes/b1.bb new file mode
> 100644 index 0000000..c0b288e
> --- /dev/null
> +++ b/bitbake/lib/bb/tests/runqueue-tests/recipes/b1.bb
> @@ -0,0 +1 @@
> +DEPENDS = "a1"
> \ No newline at end of file
> diff --git a/bitbake/lib/bb/tests/runqueue-tests/recipes/c1.bb
> b/bitbake/lib/bb/tests/runqueue-tests/recipes/c1.bb new file mode
> 100644 index 0000000..e69de29
> diff --git a/bitbake/lib/bb/tests/runqueue-tests/recipes/d1.bb
> b/bitbake/lib/bb/tests/runqueue-tests/recipes/d1.bb new file mode
> 100644 index 0000000..5ba1975
> --- /dev/null
> +++ b/bitbake/lib/bb/tests/runqueue-tests/recipes/d1.bb
> @@ -0,0 +1,3 @@
> +DEPENDS = "a1"
> +
> +do_package_setscene[depends] = "a1:do_populate_sysroot_setscene"
> diff --git a/bitbake/lib/bb/tests/runqueue-tests/recipes/e1.bb
> b/bitbake/lib/bb/tests/runqueue-tests/recipes/e1.bb new file mode
> 100644 index 0000000..1588bc8
> --- /dev/null
> +++ b/bitbake/lib/bb/tests/runqueue-tests/recipes/e1.bb
> @@ -0,0 +1 @@
> +DEPENDS = "b1"
> \ No newline at end of file
> diff --git a/bitbake/lib/bb/tests/runqueue.py
> b/bitbake/lib/bb/tests/runqueue.py new file mode 100644
> index 0000000..5e64391
> --- /dev/null
> +++ b/bitbake/lib/bb/tests/runqueue.py
> @@ -0,0 +1,323 @@
> +#
> +# BitBake Tests for runqueue task processing
> +#
> +# Copyright (C) 2019 Richard Purdie
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> +import unittest
> +import bb
> +import os
> +import tempfile
> +import subprocess
> +import sys
> +import time
> +
> +#
> +# TODO:
> +# Add tests on task ordering (X happens before Y after Z)
> +#
> +
> +class RunQueueTests(unittest.TestCase):
> +
> + alltasks = ['package', 'fetch', 'unpack', 'patch',
> 'prepare_recipe_sysroot', 'configure',
> + 'compile', 'install', 'packagedata', 'package_qa',
> 'package_write_rpm', 'package_write_ipk',
> + 'populate_sysroot', 'build']
> + a1_sstatevalid = "a1:do_package a1:do_package_qa
> a1:do_packagedata a1:do_package_write_ipk a1:do_package_write_rpm
> a1:do_populate_lic a1:do_populate_sysroot"
> + b1_sstatevalid = "b1:do_package b1:do_package_qa
> b1:do_packagedata b1:do_package_write_ipk b1:do_package_write_rpm
> b1:do_populate_lic b1:do_populate_sysroot" +
> + def run_bitbakecmd(self, cmd, builddir, sstatevalid="",
> slowtasks="", extraenv=None, cleanup=False):
> + env = os.environ.copy()
> + env["BBPATH"] =
> os.path.realpath(os.path.join(os.path.dirname(__file__),
> "runqueue-tests"))
> + env["BB_ENV_EXTRAWHITE"] = "SSTATEVALID SLOWTASKS"
> + env["SSTATEVALID"] = sstatevalid
> + env["SLOWTASKS"] = slowtasks
> + if extraenv:
> + for k in extraenv:
> + env[k] = extraenv[k]
> + env["BB_ENV_EXTRAWHITE"] = env["BB_ENV_EXTRAWHITE"]
> + " " + k
> + try:
> + output = subprocess.check_output(cmd, env=env,
> stderr=subprocess.STDOUT,universal_newlines=True, cwd=builddir)
> + print(output)
> + except subprocess.CalledProcessError as e:
> + self.fail("Command %s failed with %s" % (cmd, e.output))
> + tasks = []
> + tasklog = builddir + "/task.log"
> + if os.path.exists(tasklog):
> + with open(tasklog, "r") as f:
> + tasks = [line.rstrip() for line in f]
> + if cleanup:
> + os.remove(tasklog)
> + return tasks
> +
> + def test_no_setscenevalid(self):
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + cmd = ["bitbake", "a1"]
> + sstatevalid = ""
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
> + expected = ['a1:' + x for x in self.alltasks]
> + self.assertEqual(set(tasks), set(expected))
> +
> + def test_single_setscenevalid(self):
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + cmd = ["bitbake", "a1"]
> + sstatevalid = "a1:do_package"
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
> + expected = ['a1:package_setscene', 'a1:fetch',
> 'a1:unpack', 'a1:patch', 'a1:prepare_recipe_sysroot', 'a1:configure',
> + 'a1:compile', 'a1:install',
> 'a1:packagedata', 'a1:package_qa', 'a1:package_write_rpm',
> 'a1:package_write_ipk',
> + 'a1:populate_sysroot', 'a1:build']
> + self.assertEqual(set(tasks), set(expected))
> +
> + def test_intermediate_setscenevalid(self):
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + cmd = ["bitbake", "a1"]
> + sstatevalid = "a1:do_package a1:do_populate_sysroot"
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
> + expected = ['a1:package_setscene', 'a1:packagedata',
> 'a1:package_qa', 'a1:package_write_rpm', 'a1:package_write_ipk',
> + 'a1:populate_sysroot_setscene', 'a1:build']
> + self.assertEqual(set(tasks), set(expected))
> +
> + def test_intermediate_notcovered(self):
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + cmd = ["bitbake", "a1"]
> + sstatevalid = "a1:do_package_qa a1:do_packagedata
> a1:do_package_write_ipk a1:do_package_write_rpm a1:do_populate_lic
> a1:do_populate_sysroot"
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
> + expected = ['a1:package_write_ipk_setscene',
> 'a1:package_write_rpm_setscene', 'a1:packagedata_setscene',
> + 'a1:package_qa_setscene', 'a1:build',
> 'a1:populate_sysroot_setscene']
> + self.assertEqual(set(tasks), set(expected))
> +
> + def test_all_setscenevalid(self):
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + cmd = ["bitbake", "a1"]
> + sstatevalid = self.a1_sstatevalid
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
> + expected = ['a1:package_write_ipk_setscene',
> 'a1:package_write_rpm_setscene', 'a1:packagedata_setscene',
> + 'a1:package_qa_setscene', 'a1:build',
> 'a1:populate_sysroot_setscene']
> + self.assertEqual(set(tasks), set(expected))
> +
> + def test_no_settasks(self):
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + cmd = ["bitbake", "a1", "-c", "patch"]
> + sstatevalid = self.a1_sstatevalid
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
> + expected = ['a1:fetch', 'a1:unpack', 'a1:patch']
> + self.assertEqual(set(tasks), set(expected))
> +
> + def test_mix_covered_notcovered(self):
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + cmd = ["bitbake", "a1:do_patch",
> "a1:do_populate_sysroot"]
> + sstatevalid = self.a1_sstatevalid
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
> + expected = ['a1:fetch', 'a1:unpack', 'a1:patch',
> 'a1:populate_sysroot_setscene']
> + self.assertEqual(set(tasks), set(expected))
> +
> +
> + # Test targets with intermediate setscene tasks alongside a
> target with no intermediate setscene tasks
> + def test_mixed_direct_tasks_setscene_tasks(self):
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + cmd = ["bitbake", "c1:do_patch", "a1"]
> + sstatevalid = self.a1_sstatevalid
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
> + expected = ['c1:fetch', 'c1:unpack', 'c1:patch',
> 'a1:package_write_ipk_setscene', 'a1:package_write_rpm_setscene',
> 'a1:packagedata_setscene',
> + 'a1:package_qa_setscene', 'a1:build',
> 'a1:populate_sysroot_setscene']
> + self.assertEqual(set(tasks), set(expected))
> +
> + # This test slows down the execution of do_package_setscene
> until after other real tasks have
> + # started running which tests for a bug where tasks were being
> lost from the buildable list of real
> + # tasks if they weren't in tasks_covered or tasks_notcovered
> + def test_slow_setscene(self):
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + cmd = ["bitbake", "a1"]
> + sstatevalid = "a1:do_package"
> + slowtasks = "a1:package_setscene"
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
> slowtasks)
> + expected = ['a1:package_setscene', 'a1:fetch',
> 'a1:unpack', 'a1:patch', 'a1:prepare_recipe_sysroot', 'a1:configure',
> + 'a1:compile', 'a1:install',
> 'a1:packagedata', 'a1:package_qa', 'a1:package_write_rpm',
> 'a1:package_write_ipk',
> + 'a1:populate_sysroot', 'a1:build']
> + self.assertEqual(set(tasks), set(expected))
> +
> + def test_setscenewhitelist(self):
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + cmd = ["bitbake", "a1"]
> + extraenv = {
> + "BB_SETSCENE_ENFORCE" : "1",
> + "BB_SETSCENE_ENFORCE_WHITELIST" :
> "a1:do_package_write_rpm a1:do_build"
> + }
> + sstatevalid = "a1:do_package a1:do_package_qa
> a1:do_packagedata a1:do_package_write_ipk a1:do_populate_lic
> a1:do_populate_sysroot"
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
> extraenv=extraenv)
> + expected = ['a1:packagedata_setscene',
> 'a1:package_qa_setscene', 'a1:package_write_ipk_setscene',
> + 'a1:populate_sysroot_setscene',
> 'a1:package_setscene']
> + self.assertEqual(set(tasks), set(expected))
> +
> + # Tests for problems with dependencies between setscene tasks
> + def test_no_setscenevalid_harddeps(self):
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + cmd = ["bitbake", "d1"]
> + sstatevalid = ""
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
> + expected = ['a1:package', 'a1:fetch', 'a1:unpack',
> 'a1:patch', 'a1:prepare_recipe_sysroot', 'a1:configure',
> + 'a1:compile', 'a1:install',
> 'a1:packagedata', 'a1:package_write_rpm', 'a1:package_write_ipk',
> + 'a1:populate_sysroot', 'd1:package',
> 'd1:fetch', 'd1:unpack', 'd1:patch', 'd1:prepare_recipe_sysroot',
> 'd1:configure',
> + 'd1:compile', 'd1:install',
> 'd1:packagedata', 'd1:package_qa', 'd1:package_write_rpm',
> 'd1:package_write_ipk',
> + 'd1:populate_sysroot', 'd1:build']
> + self.assertEqual(set(tasks), set(expected))
> +
> + def test_no_setscenevalid_withdeps(self):
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + cmd = ["bitbake", "b1"]
> + sstatevalid = ""
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
> + expected = ['a1:' + x for x in self.alltasks] + ['b1:' +
> x for x in self.alltasks]
> + expected.remove('a1:build')
> + expected.remove('a1:package_qa')
> + self.assertEqual(set(tasks), set(expected))
> +
> + def test_single_a1_setscenevalid_withdeps(self):
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + cmd = ["bitbake", "b1"]
> + sstatevalid = "a1:do_package"
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
> + expected = ['a1:package_setscene', 'a1:fetch',
> 'a1:unpack', 'a1:patch', 'a1:prepare_recipe_sysroot', 'a1:configure',
> + 'a1:compile', 'a1:install',
> 'a1:packagedata', 'a1:package_write_rpm', 'a1:package_write_ipk',
> + 'a1:populate_sysroot'] + ['b1:' + x for x in
> self.alltasks]
> + self.assertEqual(set(tasks), set(expected))
> +
> + def test_single_b1_setscenevalid_withdeps(self):
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + cmd = ["bitbake", "b1"]
> + sstatevalid = "b1:do_package"
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
> + expected = ['a1:package', 'a1:fetch', 'a1:unpack',
> 'a1:patch', 'a1:prepare_recipe_sysroot', 'a1:configure',
> + 'a1:compile', 'a1:install',
> 'a1:packagedata', 'a1:package_write_rpm', 'a1:package_write_ipk',
> + 'a1:populate_sysroot',
> 'b1:package_setscene'] + ['b1:' + x for x in self.alltasks]
> + expected.remove('b1:package')
> + self.assertEqual(set(tasks), set(expected))
> +
> + def test_intermediate_setscenevalid_withdeps(self):
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + cmd = ["bitbake", "b1"]
> + sstatevalid = "a1:do_package a1:do_populate_sysroot
> b1:do_package"
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
> + expected = ['a1:package_setscene', 'a1:packagedata',
> 'a1:package_write_rpm', 'a1:package_write_ipk',
> + 'a1:populate_sysroot_setscene',
> 'b1:package_setscene'] + ['b1:' + x for x in self.alltasks]
> + expected.remove('b1:package')
> + self.assertEqual(set(tasks), set(expected))
> +
> + def test_all_setscenevalid_withdeps(self):
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + cmd = ["bitbake", "b1"]
> + sstatevalid = self.a1_sstatevalid + " " +
> self.b1_sstatevalid
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
> + expected = ['a1:package_write_ipk_setscene',
> 'a1:package_write_rpm_setscene', 'a1:packagedata_setscene',
> + 'b1:build', 'a1:populate_sysroot_setscene',
> 'b1:package_write_ipk_setscene', 'b1:package_write_rpm_setscene',
> + 'b1:packagedata_setscene',
> 'b1:package_qa_setscene', 'b1:populate_sysroot_setscene']
> + self.assertEqual(set(tasks), set(expected))
> +
> + def test_multiconfig_setscene_optimise(self):
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + extraenv = {
> + "BBMULTICONFIG" : "mc1 mc2",
> + "BB_SIGNATURE_HANDLER" : "basic"
> + }
> + cmd = ["bitbake", "b1", "mc:mc1:b1", "mc:mc2:b1"]
> + setscenetasks = ['package_write_ipk_setscene',
> 'package_write_rpm_setscene', 'packagedata_setscene',
> + 'populate_sysroot_setscene',
> 'package_qa_setscene']
> + sstatevalid = ""
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
> extraenv=extraenv)
> + expected = ['a1:' + x for x in self.alltasks] + ['b1:' +
> x for x in self.alltasks] + \
> + ['mc1:b1:' + x for x in setscenetasks] +
> ['mc1:a1:' + x for x in setscenetasks] + \
> + ['mc2:b1:' + x for x in setscenetasks] +
> ['mc2:a1:' + x for x in setscenetasks] + \
> + ['mc1:b1:build', 'mc2:b1:build']
> + for x in ['mc1:a1:package_qa_setscene',
> 'mc2:a1:package_qa_setscene', 'a1:build', 'a1:package_qa']:
> + expected.remove(x)
> + self.assertEqual(set(tasks), set(expected))
> +
> +
> + @unittest.skipIf(sys.version_info < (3, 5, 0), 'Python 3.5 or
> later required')
> + def test_hashserv_single(self):
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + extraenv = {
> + "BB_HASHSERVE" : "auto",
> + "BB_SIGNATURE_HANDLER" : "TestEquivHash"
> + }
> + cmd = ["bitbake", "a1", "b1"]
> + setscenetasks = ['package_write_ipk_setscene',
> 'package_write_rpm_setscene', 'packagedata_setscene',
> + 'populate_sysroot_setscene',
> 'package_qa_setscene']
> + sstatevalid = ""
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
> extraenv=extraenv, cleanup=True)
> + expected = ['a1:' + x for x in self.alltasks] + ['b1:' +
> x for x in self.alltasks]
> + self.assertEqual(set(tasks), set(expected))
> + cmd = ["bitbake", "a1", "-c", "install", "-f"]
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
> extraenv=extraenv, cleanup=True)
> + expected = ['a1:install']
> + self.assertEqual(set(tasks), set(expected))
> + cmd = ["bitbake", "a1", "b1"]
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
> extraenv=extraenv, cleanup=True)
> + expected = ['a1:populate_sysroot', 'a1:package',
> 'a1:package_write_rpm_setscene', 'a1:packagedata_setscene',
> + 'a1:package_write_ipk_setscene',
> 'a1:package_qa_setscene']
> + self.assertEqual(set(tasks), set(expected))
> +
> + self.shutdown(tempdir)
> +
> + @unittest.skipIf(sys.version_info < (3, 5, 0), 'Python 3.5 or
> later required')
> + def test_hashserv_double(self):
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + extraenv = {
> + "BB_HASHSERVE" : "auto",
> + "BB_SIGNATURE_HANDLER" : "TestEquivHash"
> + }
> + cmd = ["bitbake", "a1", "b1", "e1"]
> + setscenetasks = ['package_write_ipk_setscene',
> 'package_write_rpm_setscene', 'packagedata_setscene',
> + 'populate_sysroot_setscene',
> 'package_qa_setscene']
> + sstatevalid = ""
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
> extraenv=extraenv, cleanup=True)
> + expected = ['a1:' + x for x in self.alltasks] + ['b1:' +
> x for x in self.alltasks] + ['e1:' + x for x in self.alltasks]
> + self.assertEqual(set(tasks), set(expected))
> + cmd = ["bitbake", "a1", "b1", "-c", "install", "-fn"]
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
> extraenv=extraenv, cleanup=True)
> + cmd = ["bitbake", "e1"]
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
> extraenv=extraenv, cleanup=True)
> + expected = ['a1:package', 'a1:install', 'b1:package',
> 'b1:install', 'a1:populate_sysroot', 'b1:populate_sysroot',
> + 'a1:package_write_ipk_setscene',
> 'b1:packagedata_setscene', 'b1:package_write_rpm_setscene',
> + 'a1:package_write_rpm_setscene',
> 'b1:package_write_ipk_setscene', 'a1:packagedata_setscene']
> + self.assertEqual(set(tasks), set(expected))
> +
> + self.shutdown(tempdir)
> +
> + @unittest.skipIf(sys.version_info < (3, 5, 0), 'Python 3.5 or
> later required')
> + def test_hashserv_multiple_setscene(self):
> + # Runs e1:do_package_setscene twice
> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
> tempdir:
> + extraenv = {
> + "BB_HASHSERVE" : "auto",
> + "BB_SIGNATURE_HANDLER" : "TestEquivHash"
> + }
> + cmd = ["bitbake", "a1", "b1", "e1"]
> + setscenetasks = ['package_write_ipk_setscene',
> 'package_write_rpm_setscene', 'packagedata_setscene',
> + 'populate_sysroot_setscene',
> 'package_qa_setscene']
> + sstatevalid = ""
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
> extraenv=extraenv, cleanup=True)
> + expected = ['a1:' + x for x in self.alltasks] + ['b1:' +
> x for x in self.alltasks] + ['e1:' + x for x in self.alltasks]
> + self.assertEqual(set(tasks), set(expected))
> + cmd = ["bitbake", "a1", "b1", "-c", "install", "-fn"]
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
> extraenv=extraenv, cleanup=True)
> + cmd = ["bitbake", "e1"]
> + sstatevalid = "e1:do_package"
> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
> extraenv=extraenv, cleanup=True, slowtasks="a1:populate_sysroot
> b1:populate_sysroot")
> + expected = ['a1:package', 'a1:install', 'b1:package',
> 'b1:install', 'a1:populate_sysroot', 'b1:populate_sysroot',
> + 'a1:package_write_ipk_setscene',
> 'b1:packagedata_setscene', 'b1:package_write_rpm_setscene',
> + 'a1:package_write_rpm_setscene',
> 'b1:package_write_ipk_setscene', 'a1:packagedata_setscene',
> + 'e1:package_setscene']
> + self.assertEqual(set(tasks), set(expected))
> + for i in expected:
> + self.assertEqual(tasks.count(i), 1, "%s not in task
> list once" % i) +
> + self.shutdown(tempdir)
> +
> + def shutdown(self, tempdir):
> + # Wait for the hashserve socket to disappear else we'll see
> races with the tempdir cleanup
> + while os.path.exists(tempdir + "/hashserve.sock"):
> + time.sleep(0.5)
> +
> +
> diff --git a/bitbake/lib/bb/tests/utils.py
> b/bitbake/lib/bb/tests/utils.py index 2f4ccf3..f4adf1d 100644
> --- a/bitbake/lib/bb/tests/utils.py
> +++ b/bitbake/lib/bb/tests/utils.py
> @@ -1,22 +1,9 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # BitBake Tests for utils.py
> #
> # Copyright (C) 2012 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
>
> import unittest
> @@ -42,6 +29,14 @@ class VerCmpString(unittest.TestCase):
> self.assertTrue(result < 0)
> result = bb.utils.vercmp_string('1.1', '1.0+1.1-beta1')
> self.assertTrue(result > 0)
> + result = bb.utils.vercmp_string('1a', '1a1')
> + self.assertTrue(result < 0)
> + result = bb.utils.vercmp_string('1a1', '1a')
> + self.assertTrue(result > 0)
> + result = bb.utils.vercmp_string('1.', '1.1')
> + self.assertTrue(result < 0)
> + result = bb.utils.vercmp_string('1.1', '1.')
> + self.assertTrue(result > 0)
>
> def test_explode_dep_versions(self):
> correctresult = {"foo" : ["= 1.10"]}
> diff --git a/bitbake/lib/bb/tinfoil.py b/bitbake/lib/bb/tinfoil.py
> index 368264f..0a1b913 100644
> --- a/bitbake/lib/bb/tinfoil.py
> +++ b/bitbake/lib/bb/tinfoil.py
> @@ -4,18 +4,8 @@
> # Copyright (C) 2011 Mentor Graphics Corporation
> # Copyright (C) 2006-2012 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import logging
> import os
> diff --git a/bitbake/lib/bb/ui/__init__.py
> b/bitbake/lib/bb/ui/__init__.py index a4805ed..4b7ac36 100644
> --- a/bitbake/lib/bb/ui/__init__.py
> +++ b/bitbake/lib/bb/ui/__init__.py
> @@ -3,15 +3,5 @@
> #
> # Copyright (C) 2006-2007 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. diff --git a/bitbake/lib/bb/ui/buildinfohelper.py
> b/bitbake/lib/bb/ui/buildinfohelper.py index 31323d2..5cbca97 100644
> --- a/bitbake/lib/bb/ui/buildinfohelper.py
> +++ b/bitbake/lib/bb/ui/buildinfohelper.py
> @@ -3,18 +3,8 @@
> #
> # Copyright (C) 2013 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import sys
> import bb
> @@ -656,6 +646,9 @@ class ORMWrapper(object):
> Target_Installed_Package.objects.create(target =
> target_obj, package = packagedict[p]['object'])
> packagedeps_objs = []
> + pattern_so = re.compile(r'.*\.so(\.\d*)?$')
> + pattern_lib = re.compile(r'.*\-suffix(\d*)?$')
> + pattern_ko = re.compile(r'^kernel-module-.*')
> for p in packagedict:
> for (px,deptype) in packagedict[p]['depends']:
> if deptype == 'depends':
> @@ -664,6 +657,13 @@ class ORMWrapper(object):
> tdeptype = Package_Dependency.TYPE_TRECOMMENDS
>
> try:
> + # Skip known non-package objects like libraries
> and kernel modules
> + if pattern_so.match(px) or pattern_lib.match(px):
> + logger.info("Toaster does not add library
> file dependencies to packages (%s,%s)", p, px)
> + continue
> + if pattern_ko.match(px):
> + logger.info("Toaster does not add kernel
> module dependencies to packages (%s,%s)", p, px)
> + continue
> packagedeps_objs.append(Package_Dependency(
> package = packagedict[p]['object'],
> depends_on = packagedict[px]['object'],
> diff --git a/bitbake/lib/bb/ui/knotty.py b/bitbake/lib/bb/ui/knotty.py
> index fa88e6c..35736ad 100644
> --- a/bitbake/lib/bb/ui/knotty.py
> +++ b/bitbake/lib/bb/ui/knotty.py
> @@ -5,18 +5,8 @@
> #
> # Copyright (C) 2006-2012 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from __future__ import division
>
> @@ -222,6 +212,23 @@ class TerminalFilter(object):
> sys.stdout.flush()
> self.footer_present = False
>
> + def elapsed(self, sec):
> + hrs = int(sec / 3600.0)
> + sec -= hrs * 3600
> + min = int(sec / 60.0)
> + sec -= min * 60
> + if hrs > 0:
> + return "%dh%dm%ds" % (hrs, min, sec)
> + elif min > 0:
> + return "%dm%ds" % (min, sec)
> + else:
> + return "%ds" % (sec)
> +
> + def keepAlive(self, t):
> + if not self.cuu:
> + print("Bitbake still alive (%ds)" % t)
> + sys.stdout.flush()
> +
> def updateFooter(self):
> if not self.cuu:
> return
> @@ -258,7 +265,7 @@ class TerminalFilter(object):
> else:
> start_time = activetasks[t].get("starttime", None)
> if start_time:
> - tasks.append("%s - %ds (pid %s)" %
> (activetasks[t]["title"], currenttime - start_time, t))
> + tasks.append("%s - %s (pid %s)" %
> (activetasks[t]["title"], self.elapsed(currenttime - start_time), t))
> else: tasks.append("%s (pid %s)" % (activetasks[t]["title"], t))
>
> @@ -293,8 +300,8 @@ class TerminalFilter(object):
> if start_time:
> pbar.start_time = start_time
> pbar.setmessage('%s:%s' % (tasknum,
> pbar.msg.split(':', 1)[1]))
> + pbar.setextra(rate)
> if progress > -1:
> - pbar.setextra(rate)
> content = pbar.update(progress)
> else:
> content = pbar.update(1)
> @@ -455,11 +462,17 @@ def main(server, eventHandler, params, tf =
> TerminalFilter): warnings = 0
> taskfailures = []
>
> + printinterval = 5000
> + lastprint = time.time()
> +
> termfilter = tf(main, helper, console, errconsole, format,
> params.options.quiet) atexit.register(termfilter.finish)
>
> while True:
> try:
> + if (lastprint + printinterval) <= time.time():
> + termfilter.keepAlive(printinterval)
> + printinterval += 5000
> event = eventHandler.waitEvent(0)
> if event is None:
> if main.shutdown > 1:
> @@ -488,6 +501,8 @@ def main(server, eventHandler, params, tf =
> TerminalFilter): continue
>
> if isinstance(event, logging.LogRecord):
> + lastprint = time.time()
> + printinterval = 5000
> if event.levelno >= format.ERROR:
> errors = errors + 1
> return_value = 1
> @@ -645,7 +660,6 @@ def main(server, eventHandler, params, tf =
> TerminalFilter): # ignore
> if isinstance(event, (bb.event.BuildBase,
> bb.event.MetadataEvent,
> - bb.event.StampUpdate,
> bb.event.ConfigParsed,
> bb.event.MultiConfigParsed,
> bb.event.RecipeParsed,
> @@ -675,17 +689,27 @@ def main(server, eventHandler, params, tf =
> TerminalFilter): if params.observe_only:
> print("\nKeyboard Interrupt, exiting observer...")
> main.shutdown = 2
> - if not params.observe_only and main.shutdown == 1:
> +
> + def state_force_shutdown():
> print("\nSecond Keyboard Interrupt, stopping...\n")
> _, error = server.runCommand(["stateForceShutdown"])
> if error:
> logger.error("Unable to cleanly stop: %s" %
> error) +
> + if not params.observe_only and main.shutdown == 1:
> + state_force_shutdown()
> +
> if not params.observe_only and main.shutdown == 0:
> print("\nKeyboard Interrupt, closing down...\n")
> interrupted = True
> - _, error = server.runCommand(["stateShutdown"])
> - if error:
> - logger.error("Unable to cleanly shutdown: %s" %
> error)
> + # Capture the second KeyboardInterrupt during
> stateShutdown is running
> + try:
> + _, error = server.runCommand(["stateShutdown"])
> + if error:
> + logger.error("Unable to cleanly shutdown:
> %s" % error)
> + except KeyboardInterrupt:
> + state_force_shutdown()
> +
> main.shutdown = main.shutdown + 1
> pass
> except Exception as e:
> diff --git a/bitbake/lib/bb/ui/ncurses.py
> b/bitbake/lib/bb/ui/ncurses.py index 8690c52..c422732 100644
> --- a/bitbake/lib/bb/ui/ncurses.py
> +++ b/bitbake/lib/bb/ui/ncurses.py
> @@ -6,18 +6,8 @@
> # Copyright (C) 2006 Michael 'Mickey' Lauer
> # Copyright (C) 2006-2007 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> """
> We have the following windows:
> diff --git a/bitbake/lib/bb/ui/taskexp.py
> b/bitbake/lib/bb/ui/taskexp.py index 8305d70..50a943c 100644
> --- a/bitbake/lib/bb/ui/taskexp.py
> +++ b/bitbake/lib/bb/ui/taskexp.py
> @@ -4,18 +4,8 @@
> # Copyright (C) 2007 Ross Burton
> # Copyright (C) 2007 - 2008 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import sys
> import gi
> diff --git a/bitbake/lib/bb/ui/toasterui.py
> b/bitbake/lib/bb/ui/toasterui.py index 88cec37..51892c9 100644
> --- a/bitbake/lib/bb/ui/toasterui.py
> +++ b/bitbake/lib/bb/ui/toasterui.py
> @@ -7,18 +7,8 @@
> # Copyright (C) 2006-2012 Richard Purdie
> # Copyright (C) 2013 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from __future__ import division
> import time
> diff --git a/bitbake/lib/bb/ui/uievent.py
> b/bitbake/lib/bb/ui/uievent.py index 9542b91..fedb050 100644
> --- a/bitbake/lib/bb/ui/uievent.py
> +++ b/bitbake/lib/bb/ui/uievent.py
> @@ -1,22 +1,9 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # Copyright (C) 2006 - 2007 Michael 'Mickey' Lauer
> # Copyright (C) 2006 - 2007 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. -
>
> """
> Use this class to fork off a thread to recieve event callbacks from
> the bitbake diff --git a/bitbake/lib/bb/ui/uihelper.py
> b/bitbake/lib/bb/ui/uihelper.py index 963c1ea..c8dd7df 100644
> --- a/bitbake/lib/bb/ui/uihelper.py
> +++ b/bitbake/lib/bb/ui/uihelper.py
> @@ -1,21 +1,9 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> # Copyright (C) 2006 - 2007 Michael 'Mickey' Lauer
> # Copyright (C) 2006 - 2007 Richard Purdie
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import bb.build
> import time
> @@ -52,7 +40,7 @@ class BBUIHelper:
> self.running_pids.remove(event.pid)
> self.failed_tasks.append( { 'title' : "%s %s" %
> (event._package, event._task)}) self.needUpdate = True
> - elif isinstance(event, bb.runqueue.runQueueTaskStarted) or
> isinstance(event, bb.runqueue.sceneQueueTaskStarted):
> + elif isinstance(event, bb.runqueue.runQueueTaskStarted):
> self.tasknumber_current = event.stats.completed +
> event.stats.active + event.stats.failed + 1 self.tasknumber_total =
> event.stats.total self.needUpdate = True
> diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
> index 13bb5f2..d035949 100644
> --- a/bitbake/lib/bb/utils.py
> +++ b/bitbake/lib/bb/utils.py
> @@ -1,23 +1,11 @@
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> """
> BitBake Utility Functions
> """
>
> # Copyright (C) 2004 Michael Lauer
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import re, fcntl, os, string, stat, shutil, time
> import sys
> @@ -121,6 +109,10 @@ def vercmp_part(a, b):
> return -1
> elif oa > ob:
> return 1
> + elif ca is None:
> + return -1
> + elif cb is None:
> + return 1
> elif ca < cb:
> return -1
> elif ca > cb:
> @@ -402,7 +394,7 @@ def better_exec(code, context, text = None,
> realfile = "<code>", pythonexception code = better_compile(code,
> realfile, realfile) try:
> exec(code, get_context(), context)
> - except (bb.BBHandledException, bb.parse.SkipRecipe,
> bb.build.FuncFailed, bb.data_smart.ExpansionError):
> + except (bb.BBHandledException, bb.parse.SkipRecipe,
> bb.data_smart.ExpansionError): # Error already shown so passthrough,
> no need for traceback raise
> except Exception as e:
> @@ -685,7 +677,7 @@ def _check_unsafe_delete_path(path):
> return True
> return False
>
> -def remove(path, recurse=False):
> +def remove(path, recurse=False, ionice=False):
> """Equivalent to rm -f or rm -rf"""
> if not path:
> return
> @@ -694,7 +686,10 @@ def remove(path, recurse=False):
> if _check_unsafe_delete_path(path):
> raise Exception('bb.utils.remove: called with
> dangerous path "%s" and recurse=True, refusing to delete!' % path) #
> shutil.rmtree(name) would be ideal but its too slow
> - subprocess.check_call(['rm', '-rf'] + glob.glob(path))
> + cmd = []
> + if ionice:
> + cmd = ['ionice', '-c', '3']
> + subprocess.check_call(cmd + ['rm', '-rf'] + glob.glob(path))
> return
> for name in glob.glob(path):
> try:
> @@ -703,20 +698,12 @@ def remove(path, recurse=False):
> if exc.errno != errno.ENOENT:
> raise
>
> -def prunedir(topdir):
> +def prunedir(topdir, ionice=False):
> # Delete everything reachable from the directory named in
> 'topdir'. # CAUTION: This is dangerous!
> if _check_unsafe_delete_path(topdir):
> raise Exception('bb.utils.prunedir: called with dangerous
> path "%s", refusing to delete!' % topdir)
> - for root, dirs, files in os.walk(topdir, topdown = False):
> - for name in files:
> - os.remove(os.path.join(root, name))
> - for name in dirs:
> - if os.path.islink(os.path.join(root, name)):
> - os.remove(os.path.join(root, name))
> - else:
> - os.rmdir(os.path.join(root, name))
> - os.rmdir(topdir)
> + remove(topdir, recurse=True, ionice=ionice)
>
> #
> # Could also use return re.compile("(%s)" % "|".join(map(re.escape,
> suffixes))).sub(lambda mo: "", var) @@ -726,8 +713,8 @@ def
> prune_suffix(var, suffixes, d): # See if var ends with any of the
> suffixes listed and # remove it if found
> for suffix in suffixes:
> - if var.endswith(suffix):
> - return var.replace(suffix, "")
> + if suffix and var.endswith(suffix):
> + return var[:-len(suffix)]
> return var
>
> def mkdirhier(directory):
> @@ -738,7 +725,7 @@ def mkdirhier(directory):
> try:
> os.makedirs(directory)
> except OSError as e:
> - if e.errno != errno.EEXIST:
> + if e.errno != errno.EEXIST or not os.path.isdir(directory):
> raise e
>
> def movefile(src, dest, newmtime = None, sstat = None):
> @@ -796,7 +783,7 @@ def movefile(src, dest, newmtime = None, sstat =
> None): os.rename(src, destpath)
> renamefailed = 0
> except Exception as e:
> - if e[0] != errno.EXDEV:
> + if e.errno != errno.EXDEV:
> # Some random error.
> print("movefile: Failed to move", src, "to", dest, e)
> return None
> @@ -1505,6 +1492,8 @@ def ioprio_set(who, cls, value):
> NR_ioprio_set = 251
> elif _unamearch[0] == "i" and _unamearch[2:3] == "86":
> NR_ioprio_set = 289
> + elif _unamearch == "aarch64":
> + NR_ioprio_set = 30
>
> if NR_ioprio_set:
> ioprio = value | (cls << IOPRIO_CLASS_SHIFT)
> diff --git a/bitbake/lib/bblayers/__init__.py
> b/bitbake/lib/bblayers/__init__.py index 3ad9513..4e7c09d 100644
> --- a/bitbake/lib/bblayers/__init__.py
> +++ b/bitbake/lib/bblayers/__init__.py
> @@ -1,2 +1,6 @@
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> from pkgutil import extend_path
> __path__ = extend_path(__path__, __name__)
> diff --git a/bitbake/lib/bblayers/action.py
> b/bitbake/lib/bblayers/action.py index a3f658f..d6459d6 100644
> --- a/bitbake/lib/bblayers/action.py
> +++ b/bitbake/lib/bblayers/action.py
> @@ -1,3 +1,7 @@
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> import fnmatch
> import logging
> import os
> diff --git a/bitbake/lib/bblayers/common.py
> b/bitbake/lib/bblayers/common.py index 98515ce..c5657d3 100644
> --- a/bitbake/lib/bblayers/common.py
> +++ b/bitbake/lib/bblayers/common.py
> @@ -1,3 +1,7 @@
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> import argparse
> import logging
> import os
> diff --git a/bitbake/lib/bblayers/layerindex.py
> b/bitbake/lib/bblayers/layerindex.py index 9f02a9d..57cd902 100644
> --- a/bitbake/lib/bblayers/layerindex.py
> +++ b/bitbake/lib/bblayers/layerindex.py
> @@ -1,3 +1,7 @@
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> import layerindexlib
>
> import argparse
> @@ -28,7 +32,7 @@ class LayerIndexPlugin(ActionPlugin):
> layerdir = os.path.join(repodir, subdir)
> if not os.path.exists(repodir):
> if fetch_layer:
> - result = subprocess.call('git clone %s %s' % (url,
> repodir), shell = True)
> + result = subprocess.call(['git', 'clone', url,
> repodir]) if result:
> logger.error("Failed to download %s" % url)
> return None, None, None
> diff --git a/bitbake/lib/bblayers/query.py
> b/bitbake/lib/bblayers/query.py index 9294dfa..7db49c8 100644
> --- a/bitbake/lib/bblayers/query.py
> +++ b/bitbake/lib/bblayers/query.py
> @@ -1,3 +1,7 @@
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> import collections
> import fnmatch
> import logging
> @@ -42,7 +46,7 @@ layer, with the preferred version first. Note that
> skipped recipes that are overlayed will also be listed, with a
> " (skipped)" suffix. """
>
> - items_listed = self.list_recipes('Overlayed recipes', None,
> True, args.same_version, args.filenames, True, None)
> + items_listed = self.list_recipes('Overlayed recipes', None,
> True, args.same_version, args.filenames, False, True, None, False,
> None) # Check for overlayed .bbclass files
> classes = collections.defaultdict(list)
> @@ -108,9 +112,9 @@ skipped recipes will also be listed, with a
> " (skipped)" suffix. title = 'Matching recipes:'
> else:
> title = 'Available recipes:'
> - self.list_recipes(title, args.pnspec, False, False,
> args.filenames, args.multiple, inheritlist)
> + self.list_recipes(title, args.pnspec, False, False,
> args.filenames, args.recipes_only, args.multiple, args.layer,
> args.bare, inheritlist)
> - def list_recipes(self, title, pnspec, show_overlayed_only,
> show_same_ver_only, show_filenames, show_multi_provider_only,
> inherits):
> + def list_recipes(self, title, pnspec, show_overlayed_only,
> show_same_ver_only, show_filenames, show_recipes_only,
> show_multi_provider_only, selected_layer, bare, inherits): if
> inherits: bbpath = str(self.tinfoil.config_data.getVar('BBPATH')) for
> classname in inherits: @@ -140,24 +144,30 @@ skipped recipes will
> also be listed, with a " (skipped)" suffix. preferred_versions[p] =
> (ver, fn)
> def print_item(f, pn, ver, layer, ispref):
> - if f in skiplist:
> - skipped = ' (skipped)'
> - else:
> - skipped = ''
> - if show_filenames:
> - if ispref:
> - logger.plain("%s%s", f, skipped)
> + if not selected_layer or layer == selected_layer:
> + if not bare and f in skiplist:
> + skipped = ' (skipped)'
> else:
> - logger.plain(" %s%s", f, skipped)
> - else:
> - if ispref:
> - logger.plain("%s:", pn)
> - logger.plain(" %s %s%s", layer.ljust(20), ver,
> skipped)
> + skipped = ''
> + if show_filenames:
> + if ispref:
> + logger.plain("%s%s", f, skipped)
> + else:
> + logger.plain(" %s%s", f, skipped)
> + elif show_recipes_only:
> + if pn not in show_unique_pn:
> + show_unique_pn.append(pn)
> + logger.plain("%s%s", pn, skipped)
> + else:
> + if ispref:
> + logger.plain("%s:", pn)
> + logger.plain(" %s %s%s", layer.ljust(20), ver,
> skipped)
> global_inherit = (self.tinfoil.config_data.getVar('INHERIT')
> or "").split() cls_re = re.compile('classes/')
>
> preffiles = []
> + show_unique_pn = []
> items_listed = False
> for p in sorted(pkg_pn):
> if pnspec:
> @@ -489,8 +499,11 @@ NOTE: .bbappend files can impact the
> dependencies.
> parser_show_recipes = self.add_command(sp, 'show-recipes',
> self.do_show_recipes) parser_show_recipes.add_argument('-f',
> '--filenames', help='instead of the default formatting, list
> filenames of higher priority recipes with the ones they overlay
> indented underneath', action='store_true')
> + parser_show_recipes.add_argument('-r', '--recipes-only',
> help='instead of the default formatting, list recipes only',
> action='store_true') parser_show_recipes.add_argument('-m',
> '--multiple', help='only list where multiple recipes (in the same
> layer or different layers) exist for the same recipe name',
> action='store_true') parser_show_recipes.add_argument('-i',
> '--inherits', help='only list recipes that inherit the named
> class(es) - separate multiple classes using , (without spaces)',
> metavar='CLASS', default='')
> + parser_show_recipes.add_argument('-l', '--layer', help='only
> list recipes from the selected layer', default='')
> + parser_show_recipes.add_argument('-b', '--bare',
> help='output just names without the "(skipped)" marker',
> action='store_true') parser_show_recipes.add_argument('pnspec',
> nargs='*', help='optional recipe name specification (wildcards
> allowed, enclose in quotes to avoid shell expansion)')
> parser_show_appends = self.add_command(sp, 'show-appends',
> self.do_show_appends) diff --git a/bitbake/lib/bs4/dammit.py
> b/bitbake/lib/bs4/dammit.py index 68d419f..805aa90 100644 ---
> a/bitbake/lib/bs4/dammit.py +++ b/bitbake/lib/bs4/dammit.py @@ -45,9
> +45,9 @@ except ImportError: pass
>
> xml_encoding_re = re.compile(
> - '^<\?.*encoding=[\'"](.*?)[\'"].*\?>'.encode(), re.I)
> + r'^<\?.*encoding=[\'"](.*?)[\'"].*\?>'.encode(), re.I)
> html_meta_re = re.compile(
> - '<\s*meta[^>]+charset\s*=\s*["\']?([^>]*?)[ /;\'">]'.encode(),
> re.I)
> + r'<\s*meta[^>]+charset\s*=\s*["\']?([^>]*?)[ /;\'">]'.encode(),
> re.I)
> class EntitySubstitution(object):
>
> @@ -80,11 +80,11 @@ class EntitySubstitution(object):
> ">": "gt",
> }
>
> - BARE_AMPERSAND_OR_BRACKET = re.compile("([<>]|"
> -
> "&(?!#\d+;|#x[0-9a-fA-F]+;|\w+;)"
> - ")")
> + BARE_AMPERSAND_OR_BRACKET = re.compile(r"([<>]|"
> +
> r"&(?!#\d+;|#x[0-9a-fA-F]+;|\w+;)"
> + r")")
>
> - AMPERSAND_OR_BRACKET = re.compile("([<>&])")
> + AMPERSAND_OR_BRACKET = re.compile(r"([<>&])")
>
> @classmethod
> def _substitute_html_entity(cls, matchobj):
> diff --git a/bitbake/lib/bs4/element.py b/bitbake/lib/bs4/element.py
> index 0e62c2e..3775a60 100644
> --- a/bitbake/lib/bs4/element.py
> +++ b/bitbake/lib/bs4/element.py
> @@ -1,7 +1,7 @@
> __license__ = "MIT"
>
> from pdb import set_trace
> -import collections
> +import collections.abc
> import re
> import sys
> import warnings
> @@ -10,7 +10,7 @@ from bs4.dammit import EntitySubstitution
> DEFAULT_OUTPUT_ENCODING = "utf-8"
> PY3K = (sys.version_info[0] > 2)
>
> -whitespace_re = re.compile("\s+")
> +whitespace_re = re.compile(r"\s+")
>
> def _alias(attr):
> """Alias one attribute name to another for backward
> compatibility""" @@ -67,7 +67,7 @@ class
> ContentMetaAttributeValue(AttributeValueWithCharsetSubstitution): The
> value of the 'content' attribute will be one of these objects. """
>
> - CHARSET_RE = re.compile("((^|;)\s*charset=)([^;]*)", re.M)
> + CHARSET_RE = re.compile(r"((^|;)\s*charset=)([^;]*)", re.M)
>
> def __new__(cls, original_value):
> match = cls.CHARSET_RE.search(original_value)
> @@ -155,7 +155,7 @@ class PageElement(object):
>
> def format_string(self, s, formatter='minimal'):
> """Format the given string using the given formatter."""
> - if not isinstance(formatter, collections.Callable):
> + if not isinstance(formatter, collections.abc.Callable):
> formatter = self._formatter_for_name(formatter)
> if formatter is None:
> output = s
> @@ -580,7 +580,7 @@ class PageElement(object):
>
> # Methods for supporting CSS selectors.
>
> - tag_name_re = re.compile('^[a-zA-Z0-9][-.a-zA-Z0-9:_]*$')
> + tag_name_re = re.compile(r'^[a-zA-Z0-9][-.a-zA-Z0-9:_]*$')
>
> # /^([a-zA-Z0-9][-.a-zA-Z0-9:_]*)\[(\w+)([=~\|\^\$\*]?)=?"?([^\]"]*)"?\]$/
> # \---------------------------/ \---/\-------------/
> \-------/ @@ -1077,7 +1077,7 @@ class Tag(PageElement):
>
> # First off, turn a string formatter into a function. This
> # will stop the lookup from happening over and over again.
> - if not isinstance(formatter, collections.Callable):
> + if not isinstance(formatter, collections.abc.Callable):
> formatter = self._formatter_for_name(formatter)
>
> attrs = []
> @@ -1181,7 +1181,7 @@ class Tag(PageElement):
> """
> # First off, turn a string formatter into a function. This
> # will stop the lookup from happening over and over again.
> - if not isinstance(formatter, collections.Callable):
> + if not isinstance(formatter, collections.abc.Callable):
> formatter = self._formatter_for_name(formatter)
>
> pretty_print = (indent_level is not None)
> @@ -1364,7 +1364,7 @@ class Tag(PageElement):
> if tag_name == '':
> raise ValueError(
> "A pseudo-class must be prefixed with a tag
> name.")
> - pseudo_attributes =
> re.match('([a-zA-Z\d-]+)\(([a-zA-Z\d]+)\)', pseudo)
> + pseudo_attributes =
> re.match(r'([a-zA-Z\d-]+)\(([a-zA-Z\d]+)\)', pseudo) found = []
> if pseudo_attributes is None:
> pseudo_type = pseudo
> @@ -1562,7 +1562,7 @@ class SoupStrainer(object):
> def _normalize_search_value(self, value):
> # Leave it alone if it's a Unicode string, a callable, a
> # regular expression, a boolean, or None.
> - if (isinstance(value, str) or isinstance(value,
> collections.Callable) or hasattr(value, 'match')
> + if (isinstance(value, str) or isinstance(value,
> collections.abc.Callable) or hasattr(value, 'match') or
> isinstance(value, bool) or value is None): return value
>
> @@ -1602,7 +1602,7 @@ class SoupStrainer(object):
> markup = markup_name
> markup_attrs = markup
> call_function_with_tag_data = (
> - isinstance(self.name, collections.Callable)
> + isinstance(self.name, collections.abc.Callable)
> and not isinstance(markup_name, Tag))
>
> if ((not self.name)
> @@ -1688,7 +1688,7 @@ class SoupStrainer(object):
> # True matches any non-None value.
> return markup is not None
>
> - if isinstance(match_against, collections.Callable):
> + if isinstance(match_against, collections.abc.Callable):
> return match_against(markup)
>
> # Custom callables take the tag as an argument, but all
> diff --git a/bitbake/lib/hashserv/__init__.py
> b/bitbake/lib/hashserv/__init__.py new file mode 100644
> index 0000000..c331862
> --- /dev/null
> +++ b/bitbake/lib/hashserv/__init__.py
> @@ -0,0 +1,93 @@
> +# Copyright (C) 2018-2019 Garmin Ltd.
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> +from contextlib import closing
> +import re
> +import sqlite3
> +
> +UNIX_PREFIX = "unix://"
> +
> +ADDR_TYPE_UNIX = 0
> +ADDR_TYPE_TCP = 1
> +
> +
> +def setup_database(database, sync=True):
> + db = sqlite3.connect(database)
> + db.row_factory = sqlite3.Row
> +
> + with closing(db.cursor()) as cursor:
> + cursor.execute('''
> + CREATE TABLE IF NOT EXISTS tasks_v2 (
> + id INTEGER PRIMARY KEY AUTOINCREMENT,
> + method TEXT NOT NULL,
> + outhash TEXT NOT NULL,
> + taskhash TEXT NOT NULL,
> + unihash TEXT NOT NULL,
> + created DATETIME,
> +
> + -- Optional fields
> + owner TEXT,
> + PN TEXT,
> + PV TEXT,
> + PR TEXT,
> + task TEXT,
> + outhash_siginfo TEXT,
> +
> + UNIQUE(method, outhash, taskhash)
> + )
> + ''')
> + cursor.execute('PRAGMA journal_mode = WAL')
> + cursor.execute('PRAGMA synchronous = %s' % ('NORMAL' if sync
> else 'OFF')) +
> + # Drop old indexes
> + cursor.execute('DROP INDEX IF EXISTS taskhash_lookup')
> + cursor.execute('DROP INDEX IF EXISTS outhash_lookup')
> +
> + # Create new indexes
> + cursor.execute('CREATE INDEX IF NOT EXISTS
> taskhash_lookup_v2 ON tasks_v2 (method, taskhash, created)')
> + cursor.execute('CREATE INDEX IF NOT EXISTS outhash_lookup_v2
> ON tasks_v2 (method, outhash)') +
> + return db
> +
> +
> +def parse_address(addr):
> + if addr.startswith(UNIX_PREFIX):
> + return (ADDR_TYPE_UNIX, (addr[len(UNIX_PREFIX):],))
> + else:
> + m = re.match(r'\[(?P<host>[^\]]*)\]:(?P<port>\d+)$', addr)
> + if m is not None:
> + host = m.group('host')
> + port = m.group('port')
> + else:
> + host, port = addr.split(':')
> +
> + return (ADDR_TYPE_TCP, (host, int(port)))
> +
> +
> +def create_server(addr, dbname, *, sync=True):
> + from . import server
> + db = setup_database(dbname, sync=sync)
> + s = server.Server(db)
> +
> + (typ, a) = parse_address(addr)
> + if typ == ADDR_TYPE_UNIX:
> + s.start_unix_server(*a)
> + else:
> + s.start_tcp_server(*a)
> +
> + return s
> +
> +
> +def create_client(addr):
> + from . import client
> + c = client.Client()
> +
> + (typ, a) = parse_address(addr)
> + if typ == ADDR_TYPE_UNIX:
> + c.connect_unix(*a)
> + else:
> + c.connect_tcp(*a)
> +
> + return c
> diff --git a/bitbake/lib/hashserv/client.py
> b/bitbake/lib/hashserv/client.py new file mode 100644
> index 0000000..f659566
> --- /dev/null
> +++ b/bitbake/lib/hashserv/client.py
> @@ -0,0 +1,157 @@
> +# Copyright (C) 2019 Garmin Ltd.
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> +from contextlib import closing
> +import json
> +import logging
> +import socket
> +import os
> +
> +
> +logger = logging.getLogger('hashserv.client')
> +
> +
> +class HashConnectionError(Exception):
> + pass
> +
> +
> +class Client(object):
> + MODE_NORMAL = 0
> + MODE_GET_STREAM = 1
> +
> + def __init__(self):
> + self._socket = None
> + self.reader = None
> + self.writer = None
> + self.mode = self.MODE_NORMAL
> +
> + def connect_tcp(self, address, port):
> + def connect_sock():
> + s = socket.create_connection((address, port))
> +
> + s.setsockopt(socket.SOL_TCP, socket.TCP_NODELAY, 1)
> + s.setsockopt(socket.SOL_TCP, socket.TCP_QUICKACK, 1)
> + s.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
> + return s
> +
> + self._connect_sock = connect_sock
> +
> + def connect_unix(self, path):
> + def connect_sock():
> + s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
> + # AF_UNIX has path length issues so chdir here to
> workaround
> + cwd = os.getcwd()
> + try:
> + os.chdir(os.path.dirname(path))
> + s.connect(os.path.basename(path))
> + finally:
> + os.chdir(cwd)
> + return s
> +
> + self._connect_sock = connect_sock
> +
> + def connect(self):
> + if self._socket is None:
> + self._socket = self._connect_sock()
> +
> + self.reader = self._socket.makefile('r',
> encoding='utf-8')
> + self.writer = self._socket.makefile('w',
> encoding='utf-8') +
> + self.writer.write('OEHASHEQUIV 1.0\n\n')
> + self.writer.flush()
> +
> + # Restore mode if the socket is being re-created
> + cur_mode = self.mode
> + self.mode = self.MODE_NORMAL
> + self._set_mode(cur_mode)
> +
> + return self._socket
> +
> + def close(self):
> + if self._socket is not None:
> + self._socket.close()
> + self._socket = None
> + self.reader = None
> + self.writer = None
> +
> + def _send_wrapper(self, proc):
> + count = 0
> + while True:
> + try:
> + self.connect()
> + return proc()
> + except (OSError, HashConnectionError,
> json.JSONDecodeError, UnicodeDecodeError) as e:
> + logger.warning('Error talking to server: %s' % e)
> + if count >= 3:
> + if not isinstance(e, HashConnectionError):
> + raise HashConnectionError(str(e))
> + raise e
> + self.close()
> + count += 1
> +
> + def send_message(self, msg):
> + def proc():
> + self.writer.write('%s\n' % json.dumps(msg))
> + self.writer.flush()
> +
> + l = self.reader.readline()
> + if not l:
> + raise HashConnectionError('Connection closed')
> +
> + if not l.endswith('\n'):
> + raise HashConnectionError('Bad message %r' % message)
> +
> + return json.loads(l)
> +
> + return self._send_wrapper(proc)
> +
> + def send_stream(self, msg):
> + def proc():
> + self.writer.write("%s\n" % msg)
> + self.writer.flush()
> + l = self.reader.readline()
> + if not l:
> + raise HashConnectionError('Connection closed')
> + return l.rstrip()
> +
> + return self._send_wrapper(proc)
> +
> + def _set_mode(self, new_mode):
> + if new_mode == self.MODE_NORMAL and self.mode ==
> self.MODE_GET_STREAM:
> + r = self.send_stream('END')
> + if r != 'ok':
> + raise HashConnectionError('Bad response from server
> %r' % r)
> + elif new_mode == self.MODE_GET_STREAM and self.mode ==
> self.MODE_NORMAL:
> + r = self.send_message({'get-stream': None})
> + if r != 'ok':
> + raise HashConnectionError('Bad response from server
> %r' % r)
> + elif new_mode != self.mode:
> + raise Exception('Undefined mode transition %r -> %r' %
> (self.mode, new_mode)) +
> + self.mode = new_mode
> +
> + def get_unihash(self, method, taskhash):
> + self._set_mode(self.MODE_GET_STREAM)
> + r = self.send_stream('%s %s' % (method, taskhash))
> + if not r:
> + return None
> + return r
> +
> + def report_unihash(self, taskhash, method, outhash, unihash,
> extra={}):
> + self._set_mode(self.MODE_NORMAL)
> + m = extra.copy()
> + m['taskhash'] = taskhash
> + m['method'] = method
> + m['outhash'] = outhash
> + m['unihash'] = unihash
> + return self.send_message({'report': m})
> +
> + def get_stats(self):
> + self._set_mode(self.MODE_NORMAL)
> + return self.send_message({'get-stats': None})
> +
> + def reset_stats(self):
> + self._set_mode(self.MODE_NORMAL)
> + return self.send_message({'reset-stats': None})
> diff --git a/bitbake/lib/hashserv/server.py
> b/bitbake/lib/hashserv/server.py new file mode 100644
> index 0000000..0aff776
> --- /dev/null
> +++ b/bitbake/lib/hashserv/server.py
> @@ -0,0 +1,414 @@
> +# Copyright (C) 2019 Garmin Ltd.
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> +from contextlib import closing
> +from datetime import datetime
> +import asyncio
> +import json
> +import logging
> +import math
> +import os
> +import signal
> +import socket
> +import time
> +
> +logger = logging.getLogger('hashserv.server')
> +
> +
> +class Measurement(object):
> + def __init__(self, sample):
> + self.sample = sample
> +
> + def start(self):
> + self.start_time = time.perf_counter()
> +
> + def end(self):
> + self.sample.add(time.perf_counter() - self.start_time)
> +
> + def __enter__(self):
> + self.start()
> + return self
> +
> + def __exit__(self, *args, **kwargs):
> + self.end()
> +
> +
> +class Sample(object):
> + def __init__(self, stats):
> + self.stats = stats
> + self.num_samples = 0
> + self.elapsed = 0
> +
> + def measure(self):
> + return Measurement(self)
> +
> + def __enter__(self):
> + return self
> +
> + def __exit__(self, *args, **kwargs):
> + self.end()
> +
> + def add(self, elapsed):
> + self.num_samples += 1
> + self.elapsed += elapsed
> +
> + def end(self):
> + if self.num_samples:
> + self.stats.add(self.elapsed)
> + self.num_samples = 0
> + self.elapsed = 0
> +
> +
> +class Stats(object):
> + def __init__(self):
> + self.reset()
> +
> + def reset(self):
> + self.num = 0
> + self.total_time = 0
> + self.max_time = 0
> + self.m = 0
> + self.s = 0
> + self.current_elapsed = None
> +
> + def add(self, elapsed):
> + self.num += 1
> + if self.num == 1:
> + self.m = elapsed
> + self.s = 0
> + else:
> + last_m = self.m
> + self.m = last_m + (elapsed - last_m) / self.num
> + self.s = self.s + (elapsed - last_m) * (elapsed - self.m)
> +
> + self.total_time += elapsed
> +
> + if self.max_time < elapsed:
> + self.max_time = elapsed
> +
> + def start_sample(self):
> + return Sample(self)
> +
> + @property
> + def average(self):
> + if self.num == 0:
> + return 0
> + return self.total_time / self.num
> +
> + @property
> + def stdev(self):
> + if self.num <= 1:
> + return 0
> + return math.sqrt(self.s / (self.num - 1))
> +
> + def todict(self):
> + return {k: getattr(self, k) for k in ('num', 'total_time',
> 'max_time', 'average', 'stdev')} +
> +
> +class ServerClient(object):
> + def __init__(self, reader, writer, db, request_stats):
> + self.reader = reader
> + self.writer = writer
> + self.db = db
> + self.request_stats = request_stats
> +
> + async def process_requests(self):
> + try:
> + self.addr = self.writer.get_extra_info('peername')
> + logger.debug('Client %r connected' % (self.addr,))
> +
> + # Read protocol and version
> + protocol = await self.reader.readline()
> + if protocol is None:
> + return
> +
> + (proto_name, proto_version) =
> protocol.decode('utf-8').rstrip().split()
> + if proto_name != 'OEHASHEQUIV' or proto_version != '1.0':
> + return
> +
> + # Read headers. Currently, no headers are implemented,
> so look for
> + # an empty line to signal the end of the headers
> + while True:
> + line = await self.reader.readline()
> + if line is None:
> + return
> +
> + line = line.decode('utf-8').rstrip()
> + if not line:
> + break
> +
> + # Handle messages
> + handlers = {
> + 'get': self.handle_get,
> + 'report': self.handle_report,
> + 'get-stream': self.handle_get_stream,
> + 'get-stats': self.handle_get_stats,
> + 'reset-stats': self.handle_reset_stats,
> + }
> +
> + while True:
> + d = await self.read_message()
> + if d is None:
> + break
> +
> + for k in handlers.keys():
> + if k in d:
> + logger.debug('Handling %s' % k)
> + if 'stream' in k:
> + await handlers[k](d[k])
> + else:
> + with self.request_stats.start_sample()
> as self.request_sample, \
> + self.request_sample.measure():
> + await handlers[k](d[k])
> + break
> + else:
> + logger.warning("Unrecognized command %r" % d)
> + break
> +
> + await self.writer.drain()
> + finally:
> + self.writer.close()
> +
> + def write_message(self, msg):
> + self.writer.write(('%s\n' % json.dumps(msg)).encode('utf-8'))
> +
> + async def read_message(self):
> + l = await self.reader.readline()
> + if not l:
> + return None
> +
> + try:
> + message = l.decode('utf-8')
> +
> + if not message.endswith('\n'):
> + return None
> +
> + return json.loads(message)
> + except (json.JSONDecodeError, UnicodeDecodeError) as e:
> + logger.error('Bad message from client: %r' % message)
> + raise e
> +
> + async def handle_get(self, request):
> + method = request['method']
> + taskhash = request['taskhash']
> +
> + row = self.query_equivalent(method, taskhash)
> + if row is not None:
> + logger.debug('Found equivalent task %s -> %s',
> (row['taskhash'], row['unihash']))
> + d = {k: row[k] for k in ('taskhash', 'method',
> 'unihash')} +
> + self.write_message(d)
> + else:
> + self.write_message(None)
> +
> + async def handle_get_stream(self, request):
> + self.write_message('ok')
> +
> + while True:
> + l = await self.reader.readline()
> + if not l:
> + return
> +
> + try:
> + # This inner loop is very sensitive and must be as
> fast as
> + # possible (which is why the request sample is
> handled manually
> + # instead of using 'with', and also why logging
> statements are
> + # commented out.
> + self.request_sample =
> self.request_stats.start_sample()
> + request_measure = self.request_sample.measure()
> + request_measure.start()
> +
> + l = l.decode('utf-8').rstrip()
> + if l == 'END':
> + self.writer.write('ok\n'.encode('utf-8'))
> + return
> +
> + (method, taskhash) = l.split()
> + #logger.debug('Looking up %s %s' % (method,
> taskhash))
> + row = self.query_equivalent(method, taskhash)
> + if row is not None:
> + msg = ('%s\n' % row['unihash']).encode('utf-8')
> + #logger.debug('Found equivalent task %s -> %s',
> (row['taskhash'], row['unihash']))
> + else:
> + msg = '\n'.encode('utf-8')
> +
> + self.writer.write(msg)
> + finally:
> + request_measure.end()
> + self.request_sample.end()
> +
> + await self.writer.drain()
> +
> + async def handle_report(self, data):
> + with closing(self.db.cursor()) as cursor:
> + cursor.execute('''
> + -- Find tasks with a matching outhash (that is,
> tasks that
> + -- are equivalent)
> + SELECT taskhash, method, unihash FROM tasks_v2 WHERE
> method=:method AND outhash=:outhash +
> + -- If there is an exact match on the taskhash,
> return it.
> + -- Otherwise return the oldest matching outhash of
> any
> + -- taskhash
> + ORDER BY CASE WHEN taskhash=:taskhash THEN 1 ELSE 2
> END,
> + created ASC
> +
> + -- Only return one row
> + LIMIT 1
> + ''', {k: data[k] for k in ('method', 'outhash',
> 'taskhash')}) +
> + row = cursor.fetchone()
> +
> + # If no matching outhash was found, or one *was* found
> but it
> + # wasn't an exact match on the taskhash, a new entry for
> this
> + # taskhash should be added
> + if row is None or row['taskhash'] != data['taskhash']:
> + # If a row matching the outhash was found, the
> unihash for
> + # the new taskhash should be the same as that one.
> + # Otherwise the caller provided unihash is used.
> + unihash = data['unihash']
> + if row is not None:
> + unihash = row['unihash']
> +
> + insert_data = {
> + 'method': data['method'],
> + 'outhash': data['outhash'],
> + 'taskhash': data['taskhash'],
> + 'unihash': unihash,
> + 'created': datetime.now()
> + }
> +
> + for k in ('owner', 'PN', 'PV', 'PR', 'task',
> 'outhash_siginfo'):
> + if k in data:
> + insert_data[k] = data[k]
> +
> + cursor.execute('''INSERT INTO tasks_v2 (%s) VALUES
> (%s)''' % (
> + ', '.join(sorted(insert_data.keys())),
> + ', '.join(':' + k for k in
> sorted(insert_data.keys()))),
> + insert_data)
> +
> + self.db.commit()
> +
> + logger.info('Adding taskhash %s with unihash %s',
> + data['taskhash'], unihash)
> +
> + d = {
> + 'taskhash': data['taskhash'],
> + 'method': data['method'],
> + 'unihash': unihash
> + }
> + else:
> + d = {k: row[k] for k in ('taskhash', 'method',
> 'unihash')} +
> + self.write_message(d)
> +
> + async def handle_get_stats(self, request):
> + d = {
> + 'requests': self.request_stats.todict(),
> + }
> +
> + self.write_message(d)
> +
> + async def handle_reset_stats(self, request):
> + d = {
> + 'requests': self.request_stats.todict(),
> + }
> +
> + self.request_stats.reset()
> + self.write_message(d)
> +
> + def query_equivalent(self, method, taskhash):
> + # This is part of the inner loop and must be as fast as
> possible
> + try:
> + cursor = self.db.cursor()
> + cursor.execute('SELECT taskhash, method, unihash FROM
> tasks_v2 WHERE method=:method AND taskhash=:taskhash ORDER BY created
> ASC LIMIT 1',
> + {'method': method, 'taskhash': taskhash})
> + return cursor.fetchone()
> + except:
> + cursor.close()
> +
> +
> +class Server(object):
> + def __init__(self, db, loop=None):
> + self.request_stats = Stats()
> + self.db = db
> +
> + if loop is None:
> + self.loop = asyncio.new_event_loop()
> + self.close_loop = True
> + else:
> + self.loop = loop
> + self.close_loop = False
> +
> + self._cleanup_socket = None
> +
> + def start_tcp_server(self, host, port):
> + self.server = self.loop.run_until_complete(
> + asyncio.start_server(self.handle_client, host, port,
> loop=self.loop)
> + )
> +
> + for s in self.server.sockets:
> + logger.info('Listening on %r' % (s.getsockname(),))
> + # Newer python does this automatically. Do it manually
> here for
> + # maximum compatibility
> + s.setsockopt(socket.SOL_TCP, socket.TCP_NODELAY, 1)
> + s.setsockopt(socket.SOL_TCP, socket.TCP_QUICKACK, 1)
> +
> + name = self.server.sockets[0].getsockname()
> + if self.server.sockets[0].family == socket.AF_INET6:
> + self.address = "[%s]:%d" % (name[0], name[1])
> + else:
> + self.address = "%s:%d" % (name[0], name[1])
> +
> + def start_unix_server(self, path):
> + def cleanup():
> + os.unlink(path)
> +
> + cwd = os.getcwd()
> + try:
> + # Work around path length limits in AF_UNIX
> + os.chdir(os.path.dirname(path))
> + self.server = self.loop.run_until_complete(
> + asyncio.start_unix_server(self.handle_client,
> os.path.basename(path), loop=self.loop)
> + )
> + finally:
> + os.chdir(cwd)
> +
> + logger.info('Listening on %r' % path)
> +
> + self._cleanup_socket = cleanup
> + self.address = "unix://%s" % os.path.abspath(path)
> +
> + async def handle_client(self, reader, writer):
> + # writer.transport.set_write_buffer_limits(0)
> + try:
> + client = ServerClient(reader, writer, self.db,
> self.request_stats)
> + await client.process_requests()
> + except Exception as e:
> + import traceback
> + logger.error('Error from client: %s' % str(e),
> exc_info=True)
> + traceback.print_exc()
> + writer.close()
> + logger.info('Client disconnected')
> +
> + def serve_forever(self):
> + def signal_handler():
> + self.loop.stop()
> +
> + self.loop.add_signal_handler(signal.SIGTERM, signal_handler)
> +
> + try:
> + self.loop.run_forever()
> + except KeyboardInterrupt:
> + pass
> +
> + self.server.close()
> + self.loop.run_until_complete(self.server.wait_closed())
> + logger.info('Server shutting down')
> +
> + if self.close_loop:
> + self.loop.close()
> +
> + if self._cleanup_socket is not None:
> + self._cleanup_socket()
> diff --git a/bitbake/lib/hashserv/tests.py
> b/bitbake/lib/hashserv/tests.py new file mode 100644
> index 0000000..a5472a9
> --- /dev/null
> +++ b/bitbake/lib/hashserv/tests.py
> @@ -0,0 +1,142 @@
> +#! /usr/bin/env python3
> +#
> +# Copyright (C) 2018-2019 Garmin Ltd.
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> +from . import create_server, create_client
> +import hashlib
> +import logging
> +import multiprocessing
> +import sys
> +import tempfile
> +import threading
> +import unittest
> +
> +
> +class TestHashEquivalenceServer(object):
> + METHOD = 'TestMethod'
> +
> + def _run_server(self):
> + # logging.basicConfig(level=logging.DEBUG,
> filename='bbhashserv.log', filemode='w',
> + # format='%(levelname)s
> %(filename)s:%(lineno)d %(message)s')
> + self.server.serve_forever()
> +
> + def setUp(self):
> + if sys.version_info < (3, 5, 0):
> + self.skipTest('Python 3.5 or later required')
> +
> + self.temp_dir =
> tempfile.TemporaryDirectory(prefix='bb-hashserv')
> + self.dbfile = os.path.join(self.temp_dir.name, 'db.sqlite')
> +
> + self.server = create_server(self.get_server_addr(),
> self.dbfile)
> + self.server_thread =
> multiprocessing.Process(target=self._run_server)
> + self.server_thread.start()
> + self.client = create_client(self.server.address)
> +
> + def tearDown(self):
> + # Shutdown server
> + s = getattr(self, 'server', None)
> + if s is not None:
> + self.server_thread.terminate()
> + self.server_thread.join()
> + self.client.close()
> + self.temp_dir.cleanup()
> +
> + def test_create_hash(self):
> + # Simple test that hashes can be created
> + taskhash = '35788efcb8dfb0a02659d81cf2bfd695fb30faf9'
> + outhash =
> '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5970796f'
> + unihash = 'f46d3fbb439bd9b921095da657a4de906510d2cd'
> +
> + result = self.client.get_unihash(self.METHOD, taskhash)
> + self.assertIsNone(result, msg='Found unexpected task, %r' %
> result) +
> + result = self.client.report_unihash(taskhash, self.METHOD,
> outhash, unihash)
> + self.assertEqual(result['unihash'], unihash, 'Server
> returned bad unihash') +
> + def test_create_equivalent(self):
> + # Tests that a second reported task with the same outhash
> will be
> + # assigned the same unihash
> + taskhash = '53b8dce672cb6d0c73170be43f540460bfc347b4'
> + outhash =
> '5a9cb1649625f0bf41fc7791b635cd9c2d7118c7f021ba87dcd03f72b67ce7a8'
> + unihash = 'f37918cc02eb5a520b1aff86faacbc0a38124646'
> +
> + result = self.client.report_unihash(taskhash, self.METHOD,
> outhash, unihash)
> + self.assertEqual(result['unihash'], unihash, 'Server
> returned bad unihash') +
> + # Report a different task with the same outhash. The
> returned unihash
> + # should match the first task
> + taskhash2 = '3bf6f1e89d26205aec90da04854fbdbf73afe6b4'
> + unihash2 = 'af36b199320e611fbb16f1f277d3ee1d619ca58b'
> + result = self.client.report_unihash(taskhash2, self.METHOD,
> outhash, unihash2)
> + self.assertEqual(result['unihash'], unihash, 'Server
> returned bad unihash') +
> + def test_duplicate_taskhash(self):
> + # Tests that duplicate reports of the same taskhash with
> different
> + # outhash & unihash always return the unihash from the first
> reported
> + # taskhash
> + taskhash = '8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a'
> + outhash =
> 'afe240a439959ce86f5e322f8c208e1fedefea9e813f2140c81af866cc9edf7e'
> + unihash = '218e57509998197d570e2c98512d0105985dffc9'
> + self.client.report_unihash(taskhash, self.METHOD, outhash,
> unihash) +
> + result = self.client.get_unihash(self.METHOD, taskhash)
> + self.assertEqual(result, unihash)
> +
> + outhash2 =
> '0904a7fe3dc712d9fd8a74a616ddca2a825a8ee97adf0bd3fc86082c7639914d'
> + unihash2 = 'ae9a7d252735f0dafcdb10e2e02561ca3a47314c'
> + self.client.report_unihash(taskhash, self.METHOD, outhash2,
> unihash2) +
> + result = self.client.get_unihash(self.METHOD, taskhash)
> + self.assertEqual(result, unihash)
> +
> + outhash3 =
> '77623a549b5b1a31e3732dfa8fe61d7ce5d44b3370f253c5360e136b852967b4'
> + unihash3 = '9217a7d6398518e5dc002ed58f2cbbbc78696603'
> + self.client.report_unihash(taskhash, self.METHOD, outhash3,
> unihash3) +
> + result = self.client.get_unihash(self.METHOD, taskhash)
> + self.assertEqual(result, unihash)
> +
> + def test_stress(self):
> + def query_server(failures):
> + client = Client(self.server.address)
> + try:
> + for i in range(1000):
> + taskhash = hashlib.sha256()
> + taskhash.update(str(i).encode('utf-8'))
> + taskhash = taskhash.hexdigest()
> + result = client.get_unihash(self.METHOD,
> taskhash)
> + if result != taskhash:
> + failures.append("taskhash mismatch: %s !=
> %s" % (result, taskhash))
> + finally:
> + client.close()
> +
> + # Report hashes
> + for i in range(1000):
> + taskhash = hashlib.sha256()
> + taskhash.update(str(i).encode('utf-8'))
> + taskhash = taskhash.hexdigest()
> + self.client.report_unihash(taskhash, self.METHOD,
> taskhash, taskhash) +
> + failures = []
> + threads = [threading.Thread(target=query_server,
> args=(failures,)) for t in range(100)] +
> + for t in threads:
> + t.start()
> +
> + for t in threads:
> + t.join()
> +
> + self.assertFalse(failures)
> +
> +
> +class TestHashEquivalenceUnixServer(TestHashEquivalenceServer,
> unittest.TestCase):
> + def get_server_addr(self):
> + return "unix://" + os.path.join(self.temp_dir.name, 'sock')
> +
> +
> +class TestHashEquivalenceTCPServer(TestHashEquivalenceServer,
> unittest.TestCase):
> + def get_server_addr(self):
> + return "localhost:0"
> diff --git a/bitbake/lib/layerindexlib/__init__.py
> b/bitbake/lib/layerindexlib/__init__.py index cb79cb3..77196b4 100644
> --- a/bitbake/lib/layerindexlib/__init__.py
> +++ b/bitbake/lib/layerindexlib/__init__.py
> @@ -1,17 +1,7 @@
> # Copyright (C) 2016-2018 Wind River Systems, Inc.
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
> -# See the GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> -# along with this program; if not, write to the Free Software
> -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
> 02111-1307 USA
> import datetime
>
> @@ -386,7 +376,7 @@ layerBranches set. If not, they are effectively
> blank.''' invalid.append(name)
>
>
> - def _resolve_dependencies(layerbranches, ignores,
> dependencies, invalid):
> + def _resolve_dependencies(layerbranches, ignores,
> dependencies, invalid, processed=None): for layerbranch in
> layerbranches: if ignores and layerbranch.layer.name in ignores:
> continue
> @@ -398,6 +388,13 @@ layerBranches set. If not, they are effectively
> blank.''' if ignores and deplayerbranch.layer.name in ignores:
> continue
>
> + # Since this is depth first, we need to know
> what we're currently processing
> + # in order to avoid infinite recursion on a loop.
> + if processed and deplayerbranch.layer.name in
> processed:
> + # We have found a recursion...
> + logger.warning('Circular layer dependency
> found: %s -> %s' % (processed, deplayerbranch.layer.name))
> + continue
> +
> # This little block is why we can't re-use the
> LayerIndexObj version, # we must be able to satisfy each dependencies
> across layer indexes and # use the layer index order for priority.
> (r stands for replacement below) @@ -421,7 +418,17 @@ layerBranches
> set. If not, they are effectively blank.'''
> # New dependency, we need to resolve it now...
> depth-first if deplayerbranch.layer.name not in dependencies:
> - (dependencies, invalid) =
> _resolve_dependencies([deplayerbranch], ignores, dependencies,
> invalid)
> + # Avoid recursion on this branch.
> + # We copy so we don't end up polluting the
> depth-first branch with other
> + # branches. Duplication between individual
> branches IS expected and
> + # handled by 'dependencies' processing.
> + if not processed:
> + local_processed = []
> + else:
> + local_processed = processed.copy()
> +
> local_processed.append(deplayerbranch.layer.name) +
> + (dependencies, invalid) =
> _resolve_dependencies([deplayerbranch], ignores, dependencies,
> invalid, local_processed) if deplayerbranch.layer.name not in
> dependencies: dependencies[deplayerbranch.layer.name] =
> [deplayerbranch, layerdependency] diff --git
> a/bitbake/lib/layerindexlib/cooker.py
> b/bitbake/lib/layerindexlib/cooker.py index 848f0e2..604a961 100644
> --- a/bitbake/lib/layerindexlib/cooker.py +++
> b/bitbake/lib/layerindexlib/cooker.py @@ -1,17 +1,7 @@
> # Copyright (C) 2016-2018 Wind River Systems, Inc.
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
> -# See the GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> -# along with this program; if not, write to the Free Software
> -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
> 02111-1307 USA
> import logging
> import json
> diff --git a/bitbake/lib/layerindexlib/plugin.py
> b/bitbake/lib/layerindexlib/plugin.py index 92a2e97..7015a1a 100644
> --- a/bitbake/lib/layerindexlib/plugin.py
> +++ b/bitbake/lib/layerindexlib/plugin.py
> @@ -1,18 +1,7 @@
> # Copyright (C) 2016-2018 Wind River Systems, Inc.
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
> -# See the GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> -# along with this program; if not, write to the Free Software
> -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
> 02111-1307 USA -
> # The file contains:
> # LayerIndex exceptions
> # Plugin base class
> diff --git a/bitbake/lib/layerindexlib/restapi.py
> b/bitbake/lib/layerindexlib/restapi.py index d08eb20..21fd144 100644
> --- a/bitbake/lib/layerindexlib/restapi.py
> +++ b/bitbake/lib/layerindexlib/restapi.py
> @@ -1,17 +1,7 @@
> # Copyright (C) 2016-2018 Wind River Systems, Inc.
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
> -# See the GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> -# along with this program; if not, write to the Free Software
> -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
> 02111-1307 USA
> import logging
> import json
> diff --git a/bitbake/lib/layerindexlib/tests/common.py
> b/bitbake/lib/layerindexlib/tests/common.py index 22a5458..077382f
> 100644 --- a/bitbake/lib/layerindexlib/tests/common.py
> +++ b/bitbake/lib/layerindexlib/tests/common.py
> @@ -1,17 +1,7 @@
> # Copyright (C) 2017-2018 Wind River Systems, Inc.
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
> -# See the GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> -# along with this program; if not, write to the Free Software
> -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
> 02111-1307 USA
> import unittest
> import tempfile
> diff --git a/bitbake/lib/layerindexlib/tests/cooker.py
> b/bitbake/lib/layerindexlib/tests/cooker.py index fdbf091..1fa102e
> 100644 --- a/bitbake/lib/layerindexlib/tests/cooker.py
> +++ b/bitbake/lib/layerindexlib/tests/cooker.py
> @@ -1,17 +1,7 @@
> # Copyright (C) 2018 Wind River Systems, Inc.
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
> -# See the GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> -# along with this program; if not, write to the Free Software
> -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
> 02111-1307 USA
> import unittest
> import tempfile
> diff --git a/bitbake/lib/layerindexlib/tests/layerindexobj.py
> b/bitbake/lib/layerindexlib/tests/layerindexobj.py index
> e2fbb95..0c5ec88 100644 ---
> a/bitbake/lib/layerindexlib/tests/layerindexobj.py +++
> b/bitbake/lib/layerindexlib/tests/layerindexobj.py @@ -1,17 +1,7 @@
> # Copyright (C) 2017-2018 Wind River Systems, Inc.
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
> -# See the GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> -# along with this program; if not, write to the Free Software
> -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
> 02111-1307 USA
> import unittest
> import tempfile
> diff --git a/bitbake/lib/layerindexlib/tests/restapi.py
> b/bitbake/lib/layerindexlib/tests/restapi.py index 5876695..6d8dc00
> 100644 --- a/bitbake/lib/layerindexlib/tests/restapi.py
> +++ b/bitbake/lib/layerindexlib/tests/restapi.py
> @@ -1,17 +1,7 @@
> # Copyright (C) 2017-2018 Wind River Systems, Inc.
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
> -# See the GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> -# along with this program; if not, write to the Free Software
> -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
> 02111-1307 USA
> import unittest
> import tempfile
> diff --git a/bitbake/lib/progressbar/__init__.py
> b/bitbake/lib/progressbar/__init__.py index fbab744..c545a62 100644
> --- a/bitbake/lib/progressbar/__init__.py
> +++ b/bitbake/lib/progressbar/__init__.py
> @@ -4,6 +4,8 @@
> # progressbar - Text progress bar library for Python.
> # Copyright (c) 2005 Nilton Volpato
> #
> +# SPDX-License-Identifier: LGPL-2.1-or-later OR BSD-3-Clause-Clear
> +#
> # This library is free software; you can redistribute it and/or
> # modify it under the terms of the GNU Lesser General Public
> # License as published by the Free Software Foundation; either
> diff --git a/bitbake/lib/progressbar/compat.py
> b/bitbake/lib/progressbar/compat.py index a39f4a1..9804e0b 100644
> --- a/bitbake/lib/progressbar/compat.py
> +++ b/bitbake/lib/progressbar/compat.py
> @@ -3,6 +3,8 @@
> # progressbar - Text progress bar library for Python.
> # Copyright (c) 2005 Nilton Volpato
> #
> +# SPDX-License-Identifier: LGPL-2.1-or-later OR BSD-3-Clause-Clear
> +#
> # This library is free software; you can redistribute it and/or
> # modify it under the terms of the GNU Lesser General Public
> # License as published by the Free Software Foundation; either
> diff --git a/bitbake/lib/progressbar/progressbar.py
> b/bitbake/lib/progressbar/progressbar.py index 2873ad6..e2b6ba1 100644
> --- a/bitbake/lib/progressbar/progressbar.py
> +++ b/bitbake/lib/progressbar/progressbar.py
> @@ -5,6 +5,8 @@
> #
> # (With some small changes after importing into BitBake)
> #
> +# SPDX-License-Identifier: LGPL-2.1-or-later OR BSD-3-Clause-Clear
> +#
> # This library is free software; you can redistribute it and/or
> # modify it under the terms of the GNU Lesser General Public
> # License as published by the Free Software Foundation; either
> diff --git a/bitbake/lib/progressbar/widgets.py
> b/bitbake/lib/progressbar/widgets.py index 77285ca..0772aa5 100644
> --- a/bitbake/lib/progressbar/widgets.py
> +++ b/bitbake/lib/progressbar/widgets.py
> @@ -3,6 +3,8 @@
> # progressbar - Text progress bar library for Python.
> # Copyright (c) 2005 Nilton Volpato
> #
> +# SPDX-License-Identifier: LGPL-2.1-or-later OR BSD-3-Clause-Clear
> +#
> # This library is free software; you can redistribute it and/or
> # modify it under the terms of the GNU Lesser General Public
> # License as published by the Free Software Foundation; either
> diff --git a/bitbake/lib/prserv/__init__.py
> b/bitbake/lib/prserv/__init__.py index c3cb73a..9961040 100644
> --- a/bitbake/lib/prserv/__init__.py
> +++ b/bitbake/lib/prserv/__init__.py
> @@ -1,3 +1,7 @@
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> __version__ = "1.0.0"
>
> import os, time
> diff --git a/bitbake/lib/prserv/db.py b/bitbake/lib/prserv/db.py
> index 495d09f..117d8c0 100644
> --- a/bitbake/lib/prserv/db.py
> +++ b/bitbake/lib/prserv/db.py
> @@ -1,3 +1,7 @@
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> import logging
> import os.path
> import errno
> @@ -253,7 +257,7 @@ class PRData(object):
> self.connection=sqlite3.connect(self.filename,
> isolation_level="EXCLUSIVE", check_same_thread = False)
> self.connection.row_factory=sqlite3.Row
> self.connection.execute("pragma synchronous = off;")
> - self.connection.execute("PRAGMA journal_mode = WAL;")
> + self.connection.execute("PRAGMA journal_mode = MEMORY;")
> self._tables={}
>
> def disconnect(self):
> diff --git a/bitbake/lib/prserv/serv.py b/bitbake/lib/prserv/serv.py
> index 6a99728..be3acec 100644
> --- a/bitbake/lib/prserv/serv.py
> +++ b/bitbake/lib/prserv/serv.py
> @@ -1,3 +1,7 @@
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> import os,sys,logging
> import signal, time
> from xmlrpc.server import SimpleXMLRPCServer,
> SimpleXMLRPCRequestHandler diff --git a/bitbake/lib/pyinotify.py
> b/bitbake/lib/pyinotify.py index 4eb03b0..1528a22 100644
> --- a/bitbake/lib/pyinotify.py
> +++ b/bitbake/lib/pyinotify.py
> @@ -1,25 +1,9 @@
> -#!/usr/bin/env python
> -
> +#
> # pyinotify.py - python interface to inotify
> # Copyright (c) 2005-2015 Sebastien Martini <seb@dbzteam.org>
> #
> -# Permission is hereby granted, free of charge, to any person
> obtaining a copy -# of this software and associated documentation
> files (the "Software"), to deal -# in the Software without
> restriction, including without limitation the rights -# to use, copy,
> modify, merge, publish, distribute, sublicense, and/or sell -# copies
> of the Software, and to permit persons to whom the Software is -#
> furnished to do so, subject to the following conditions: -#
> -# The above copyright notice and this permission notice shall be
> included in -# all copies or substantial portions of the Software.
> +# SPDX-License-Identifier: MIT
> #
> -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
> EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
> MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND
> NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT
> HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY,
> WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -#
> OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
> DEALINGS IN -# THE SOFTWARE. """
> pyinotify
>
> diff --git a/bitbake/lib/toaster/bldcollector/admin.py
> b/bitbake/lib/toaster/bldcollector/admin.py index 1f2e07f..feaa888
> 100644 --- a/bitbake/lib/toaster/bldcollector/admin.py
> +++ b/bitbake/lib/toaster/bldcollector/admin.py
> @@ -1,3 +1,7 @@
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> from django.contrib import admin
> from orm.models import BitbakeVersion, Release, ToasterSetting,
> Layer_Version from django import forms
> diff --git a/bitbake/lib/toaster/bldcollector/urls.py
> b/bitbake/lib/toaster/bldcollector/urls.py index 888175d..8eb1e34
> 100644 --- a/bitbake/lib/toaster/bldcollector/urls.py
> +++ b/bitbake/lib/toaster/bldcollector/urls.py
> @@ -3,19 +3,8 @@
> #
> # Copyright (C) 2014-2017 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. -
>
> from django.conf.urls import include, url
>
> diff --git a/bitbake/lib/toaster/bldcollector/views.py
> b/bitbake/lib/toaster/bldcollector/views.py index f32fa4d..c708b41
> 100644 --- a/bitbake/lib/toaster/bldcollector/views.py
> +++ b/bitbake/lib/toaster/bldcollector/views.py
> @@ -3,18 +3,8 @@
> #
> # Copyright (C) 2014 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from django.views.decorators.cache import cache_control
> from django.core.urlresolvers import reverse
> diff --git a/bitbake/lib/toaster/bldcontrol/admin.py
> b/bitbake/lib/toaster/bldcontrol/admin.py index fcbe5f5..e85c30e
> 100644 --- a/bitbake/lib/toaster/bldcontrol/admin.py
> +++ b/bitbake/lib/toaster/bldcontrol/admin.py
> @@ -1,3 +1,7 @@
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> from django.contrib import admin
> from django.contrib.admin.filters import RelatedFieldListFilter
> from .models import BuildEnvironment
> diff --git a/bitbake/lib/toaster/bldcontrol/bbcontroller.py
> b/bitbake/lib/toaster/bldcontrol/bbcontroller.py index
> 5195600..301df18 100644 ---
> a/bitbake/lib/toaster/bldcontrol/bbcontroller.py +++
> b/bitbake/lib/toaster/bldcontrol/bbcontroller.py @@ -1,24 +1,10 @@
> #
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> -#
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2014 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. -
>
> import os
> import sys
> diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
> b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py index
> 9490635..39ea736 100644 ---
> a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py +++
> b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py @@ -1,24
> +1,10 @@ #
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> -#
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2014 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. -
>
> import os
> import sys
> diff --git
> a/bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py
> b/bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py
> index 14298d9..fe2c4dc 100644 ---
> a/bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py
> +++
> b/bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py
> @@ -1,3 +1,7 @@ +# +# SPDX-License-Identifier: GPL-2.0-only +#
> +
> from django.core.management.base import BaseCommand, CommandError
> from django.db import transaction
>
> diff --git
> a/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py
> b/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py
> index 6a55dd4..50ec409 100644 ---
> a/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py +++
> b/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py @@
> -1,3 +1,7 @@ +# +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> from django.core.management.base import BaseCommand
> from django.db import transaction
> from django.db.models import Q
> diff --git a/bitbake/lib/toaster/bldcontrol/models.py
> b/bitbake/lib/toaster/bldcontrol/models.py index 409614b..bcffcf5
> 100644 --- a/bitbake/lib/toaster/bldcontrol/models.py
> +++ b/bitbake/lib/toaster/bldcontrol/models.py
> @@ -1,3 +1,7 @@
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> from __future__ import unicode_literals
> from django.db import models
> from django.core.validators import MaxValueValidator,
> MinValueValidator diff --git
> a/bitbake/lib/toaster/bldcontrol/views.py
> b/bitbake/lib/toaster/bldcontrol/views.py index 60f00ef..286d88b
> 100644 --- a/bitbake/lib/toaster/bldcontrol/views.py +++
> b/bitbake/lib/toaster/bldcontrol/views.py @@ -1 +1,5 @@
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> # Create your views here.
> diff --git a/bitbake/lib/toaster/manage.py
> b/bitbake/lib/toaster/manage.py index 0c7ea50..ae32619 100755
> --- a/bitbake/lib/toaster/manage.py
> +++ b/bitbake/lib/toaster/manage.py
> @@ -1,4 +1,8 @@
> #!/usr/bin/env python3
> +#
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> import os
> import sys
>
> diff --git a/bitbake/lib/toaster/orm/fixtures/oe-core.xml
> b/bitbake/lib/toaster/orm/fixtures/oe-core.xml index fec93ab..a723f5a
> 100644 --- a/bitbake/lib/toaster/orm/fixtures/oe-core.xml
> +++ b/bitbake/lib/toaster/orm/fixtures/oe-core.xml
> @@ -8,9 +8,9 @@
>
> <!-- Bitbake versions which correspond to the metadata release -->
> <object model="orm.bitbakeversion" pk="1">
> - <field type="CharField" name="name">sumo</field>
> + <field type="CharField" name="name">warrior</field>
> <field type="CharField"
> name="giturl">git://git.openembedded.org/bitbake</field>
> - <field type="CharField" name="branch">1.38</field>
> + <field type="CharField" name="branch">1.42</field>
> </object>
> <object model="orm.bitbakeversion" pk="2">
> <field type="CharField" name="name">HEAD</field>
> @@ -23,18 +23,18 @@
> <field type="CharField" name="branch">master</field>
> </object>
> <object model="orm.bitbakeversion" pk="4">
> - <field type="CharField" name="name">thud</field>
> + <field type="CharField" name="name">zeus</field>
> <field type="CharField"
> name="giturl">git://git.openembedded.org/bitbake</field>
> - <field type="CharField" name="branch">1.40</field>
> + <field type="CharField" name="branch">1.44</field>
> </object>
>
> <!-- Releases available -->
> <object model="orm.release" pk="1">
> - <field type="CharField" name="name">sumo</field>
> - <field type="CharField" name="description">Openembedded
> Sumo</field>
> + <field type="CharField" name="name">warrior</field>
> + <field type="CharField" name="description">Openembedded
> Warrior</field> <field rel="ManyToOneRel" to="orm.bitbakeversion"
> name="bitbake_version">1</field>
> - <field type="CharField" name="branch_name">sumo</field>
> - <field type="TextField" name="helptext">Toaster will run your
> builds using the tip of the <a
> href=\"http://cgit.openembedded.org/openembedded-core/log/?h=sumo\">OpenEmbedded
> Sumo</a> branch.</field>
> + <field type="CharField" name="branch_name">warrior</field>
> + <field type="TextField" name="helptext">Toaster will run your
> builds using the tip of the <a
> href=\"http://cgit.openembedded.org/openembedded-core/log/?h=warrior\">OpenEmbedded
> Warrior</a> branch.</field> </object> <object
> model="orm.release" pk="2"> <field type="CharField"
> name="name">local</field> @@ -51,11 +51,11 @@ <field type="TextField"
> name="helptext">Toaster will run your builds using the tip of the
> <a
> href=\"http://cgit.openembedded.org/openembedded-core/log/\">OpenEmbedded
> master</a> branch.</field> </object> <object
> model="orm.release" pk="4">
> - <field type="CharField" name="name">thud</field>
> - <field type="CharField" name="description">Openembedded
> Rocko</field>
> - <field rel="ManyToOneRel" to="orm.bitbakeversion"
> name="bitbake_version">1</field>
> - <field type="CharField" name="branch_name">thud</field>
> - <field type="TextField" name="helptext">Toaster will run your
> builds using the tip of the <a
> href=\"http://cgit.openembedded.org/openembedded-core/log/?h=thud\">OpenEmbedded
> Thud</a> branch.</field>
> + <field type="CharField" name="name">zeus</field>
> + <field type="CharField" name="description">Openembedded
> Zeus</field>
> + <field rel="ManyToOneRel" to="orm.bitbakeversion"
> name="bitbake_version">4</field>
> + <field type="CharField" name="branch_name">zeus</field>
> + <field type="TextField" name="helptext">Toaster will run your
> builds using the tip of the <a
> href=\"http://cgit.openembedded.org/openembedded-core/log/?h=zeus\">OpenEmbedded
> Zeus</a> branch.</field> </object> <!-- Default layers for each
> release --> diff --git a/bitbake/lib/toaster/orm/fixtures/poky.xml
> b/bitbake/lib/toaster/orm/fixtures/poky.xml index fb9a771..7992383
> 100644 --- a/bitbake/lib/toaster/orm/fixtures/poky.xml
> +++ b/bitbake/lib/toaster/orm/fixtures/poky.xml
> @@ -8,9 +8,9 @@
>
> <!-- Bitbake versions which correspond to the metadata release -->
> <object model="orm.bitbakeversion" pk="1">
> - <field type="CharField" name="name">sumo</field>
> + <field type="CharField" name="name">warrior</field>
> <field type="CharField"
> name="giturl">git://git.yoctoproject.org/poky</field>
> - <field type="CharField" name="branch">sumo</field>
> + <field type="CharField" name="branch">warrior</field>
> <field type="CharField" name="dirpath">bitbake</field>
> </object>
> <object model="orm.bitbakeversion" pk="2">
> @@ -26,20 +26,20 @@
> <field type="CharField" name="dirpath">bitbake</field>
> </object>
> <object model="orm.bitbakeversion" pk="4">
> - <field type="CharField" name="name">thud</field>
> + <field type="CharField" name="name">zeus</field>
> <field type="CharField"
> name="giturl">git://git.yoctoproject.org/poky</field>
> - <field type="CharField" name="branch">thud</field>
> + <field type="CharField" name="branch">zeus</field>
> <field type="CharField" name="dirpath">bitbake</field>
> </object>
>
>
> <!-- Releases available -->
> <object model="orm.release" pk="1">
> - <field type="CharField" name="name">sumo</field>
> - <field type="CharField" name="description">Yocto Project 2.5
> "Sumo"</field>
> + <field type="CharField" name="name">warrior</field>
> + <field type="CharField" name="description">Yocto Project 2.7
> "Warrior"</field> <field rel="ManyToOneRel" to="orm.bitbakeversion"
> name="bitbake_version">1</field>
> - <field type="CharField" name="branch_name">sumo</field>
> - <field type="TextField" name="helptext">Toaster will run your
> builds using the tip of the <a
> href="http://git.yoctoproject.org/cgit/cgit.cgi/poky/log/?h=sumo">Yocto
> Project Sumo branch</a>.</field>
> + <field type="CharField" name="branch_name">warrior</field>
> + <field type="TextField" name="helptext">Toaster will run your
> builds using the tip of the <a
> href="http://git.yoctoproject.org/cgit/cgit.cgi/poky/log/?h=warrior">Yocto
> Project Warrior branch</a>.</field> </object> <object
> model="orm.release" pk="2"> <field type="CharField"
> name="name">local</field> @@ -56,11 +56,11 @@ <field type="TextField"
> name="helptext">Toaster will run your builds using the tip of the
> <a
> href="http://git.yoctoproject.org/cgit/cgit.cgi/poky/log/">Yocto
> Project Master branch</a>.</field> </object> <object
> model="orm.release" pk="4">
> - <field type="CharField" name="name">rocko</field>
> - <field type="CharField" name="description">Yocto Project 2.6
> "Thud"</field>
> - <field rel="ManyToOneRel" to="orm.bitbakeversion"
> name="bitbake_version">1</field>
> - <field type="CharField" name="branch_name">thud</field>
> - <field type="TextField" name="helptext">Toaster will run your
> builds using the tip of the <a
> href="http://git.yoctoproject.org/cgit/cgit.cgi/poky/log/?h=thud">Yocto
> Project Thud branch</a>.</field>
> + <field type="CharField" name="name">zeus</field>
> + <field type="CharField" name="description">Yocto Project 3.0
> "Zeus"</field>
> + <field rel="ManyToOneRel" to="orm.bitbakeversion"
> name="bitbake_version">4</field>
> + <field type="CharField" name="branch_name">zeus</field>
> + <field type="TextField" name="helptext">Toaster will run your
> builds using the tip of the <a
> href="http://git.yoctoproject.org/cgit/cgit.cgi/poky/log/?h=zeus">Yocto
> Project Zeus branch</a>.</field> </object> <!-- Default project
> layers for each release --> @@ -130,7 +130,7 @@
> <field rel="ManyToOneRel" to="orm.layer" name="layer">1</field>
> <field type="IntegerField" name="layer_source">0</field>
> <field rel="ManyToOneRel" to="orm.release"
> name="release">1</field>
> - <field type="CharField" name="branch">sumo</field>
> + <field type="CharField" name="branch">warrior</field>
> <field type="CharField" name="dirpath">meta</field>
> </object>
> <object model="orm.layer_version" pk="2">
> @@ -152,7 +152,7 @@
> <field rel="ManyToOneRel" to="orm.layer" name="layer">1</field>
> <field type="IntegerField" name="layer_source">0</field>
> <field rel="ManyToOneRel" to="orm.release"
> name="release">4</field>
> - <field type="CharField" name="branch">rocko</field>
> + <field type="CharField" name="branch">zeus</field>
> <field type="CharField" name="dirpath">meta</field>
> </object>
>
> @@ -168,7 +168,7 @@
> <field rel="ManyToOneRel" to="orm.layer" name="layer">2</field>
> <field type="IntegerField" name="layer_source">0</field>
> <field rel="ManyToOneRel" to="orm.release"
> name="release">1</field>
> - <field type="CharField" name="branch">sumo</field>
> + <field type="CharField" name="branch">warrior</field>
> <field type="CharField" name="dirpath">meta-poky</field>
> </object>
> <object model="orm.layer_version" pk="6">
> @@ -190,7 +190,7 @@
> <field rel="ManyToOneRel" to="orm.layer" name="layer">2</field>
> <field type="IntegerField" name="layer_source">0</field>
> <field rel="ManyToOneRel" to="orm.release"
> name="release">4</field>
> - <field type="CharField" name="branch">rocko</field>
> + <field type="CharField" name="branch">zeus</field>
> <field type="CharField" name="dirpath">meta-poky</field>
> </object>
>
> @@ -206,7 +206,7 @@
> <field rel="ManyToOneRel" to="orm.layer" name="layer">3</field>
> <field type="IntegerField" name="layer_source">0</field>
> <field rel="ManyToOneRel" to="orm.release"
> name="release">1</field>
> - <field type="CharField" name="branch">sumo</field>
> + <field type="CharField" name="branch">warrior</field>
> <field type="CharField" name="dirpath">meta-yocto-bsp</field>
> </object>
> <object model="orm.layer_version" pk="10">
> @@ -228,7 +228,7 @@
> <field rel="ManyToOneRel" to="orm.layer" name="layer">3</field>
> <field type="IntegerField" name="layer_source">0</field>
> <field rel="ManyToOneRel" to="orm.release"
> name="release">4</field>
> - <field type="CharField" name="branch">rocko</field>
> + <field type="CharField" name="branch">zeus</field>
> <field type="CharField" name="dirpath">meta-yocto-bsp</field>
> </object>
> </django-objects>
> diff --git a/bitbake/lib/toaster/orm/management/commands/lsupdates.py
> b/bitbake/lib/toaster/orm/management/commands/lsupdates.py index
> 66114ff..5b5abbb 100644 ---
> a/bitbake/lib/toaster/orm/management/commands/lsupdates.py +++
> b/bitbake/lib/toaster/orm/management/commands/lsupdates.py @@ -1,23
> +1,10 @@ #
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> -#
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2016-2017 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from django.core.management.base import BaseCommand
>
> diff --git a/bitbake/lib/toaster/orm/models.py
> b/bitbake/lib/toaster/orm/models.py index 7720290..bb6b5de 100644
> --- a/bitbake/lib/toaster/orm/models.py
> +++ b/bitbake/lib/toaster/orm/models.py
> @@ -1,23 +1,10 @@
> #
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> -#
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from __future__ import unicode_literals
>
> @@ -978,12 +965,12 @@ class TargetSDKFile(models.Model):
> class Target_Image_File(models.Model):
> # valid suffixes for image files produced by a build
> SUFFIXES = {
> - 'btrfs', 'cpio', 'cpio.gz', 'cpio.lz4', 'cpio.lzma',
> 'cpio.xz',
> - 'cramfs', 'elf', 'ext2', 'ext2.bz2', 'ext2.gz', 'ext2.lzma',
> 'ext4',
> - 'ext4.gz', 'ext3', 'ext3.gz', 'hdddirect', 'hddimg', 'iso',
> 'jffs2',
> - 'jffs2.sum', 'multiubi', 'qcow2', 'squashfs', 'squashfs-lzo',
> + 'btrfs', 'container', 'cpio', 'cpio.gz', 'cpio.lz4',
> 'cpio.lzma',
> + 'cpio.xz', 'cramfs', 'ext2', 'ext2.bz2', 'ext2.gz',
> 'ext2.lzma',
> + 'ext3', 'ext3.gz', 'ext4', 'ext4.gz', 'f2fs', 'hddimg',
> 'iso', 'jffs2',
> + 'jffs2.sum', 'multiubi', 'squashfs', 'squashfs-lz4',
> 'squashfs-lzo', 'squashfs-xz', 'tar', 'tar.bz2', 'tar.gz', 'tar.lz4',
> 'tar.xz', 'ubi',
> - 'ubifs', 'vdi', 'vmdk', 'wic', 'wic.bmap', 'wic.bz2',
> 'wic.gz', 'wic.lzma'
> + 'ubifs', 'wic', 'wic.bz2', 'wic.gz', 'wic.lzma'
> }
>
> target = models.ForeignKey(Target)
> diff --git a/bitbake/lib/toaster/tests/browser/selenium_helpers.py
> b/bitbake/lib/toaster/tests/browser/selenium_helpers.py index
> 08711e4..02d4f4b 100644 ---
> a/bitbake/lib/toaster/tests/browser/selenium_helpers.py +++
> b/bitbake/lib/toaster/tests/browser/selenium_helpers.py @@ -1,23
> +1,10 @@ -#! /usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013-2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
> # The Wait class and some of SeleniumDriverHelper and
> SeleniumTestCase are # modified from Patchwork, released under the
> same licence terms as Toaster: diff --git
> a/bitbake/lib/toaster/tests/browser/selenium_helpers_base.py
> b/bitbake/lib/toaster/tests/browser/selenium_helpers_base.py index
> 156d639..6c94684 100644 ---
> a/bitbake/lib/toaster/tests/browser/selenium_helpers_base.py +++
> b/bitbake/lib/toaster/tests/browser/selenium_helpers_base.py @@ -1,23
> +1,10 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013-2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
> #
> # The Wait class and some of SeleniumDriverHelper and
> SeleniumTestCase are # modified from Patchwork, released under the
> same licence terms as Toaster: diff --git
> a/bitbake/lib/toaster/tests/browser/test_all_builds_page.py
> b/bitbake/lib/toaster/tests/browser/test_all_builds_page.py index
> b86f29b..fba627b 100644 ---
> a/bitbake/lib/toaster/tests/browser/test_all_builds_page.py +++
> b/bitbake/lib/toaster/tests/browser/test_all_builds_page.py @@ -1,23
> +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013-2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import re
>
> diff --git
> a/bitbake/lib/toaster/tests/browser/test_all_projects_page.py
> b/bitbake/lib/toaster/tests/browser/test_all_projects_page.py index
> 44da640..afd2d35 100644 ---
> a/bitbake/lib/toaster/tests/browser/test_all_projects_page.py +++
> b/bitbake/lib/toaster/tests/browser/test_all_projects_page.py @@
> -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013-2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import re
>
> diff --git
> a/bitbake/lib/toaster/tests/browser/test_builddashboard_page.py
> b/bitbake/lib/toaster/tests/browser/test_builddashboard_page.py index
> f8ccb54..d972aff 100644 ---
> a/bitbake/lib/toaster/tests/browser/test_builddashboard_page.py +++
> b/bitbake/lib/toaster/tests/browser/test_builddashboard_page.py @@
> -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013-2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from django.core.urlresolvers import reverse
> from django.utils import timezone
> diff --git
> a/bitbake/lib/toaster/tests/browser/test_builddashboard_page_artifacts.py
> b/bitbake/lib/toaster/tests/browser/test_builddashboard_page_artifacts.py
> index 1c627ad..e2623e8 100644 ---
> a/bitbake/lib/toaster/tests/browser/test_builddashboard_page_artifacts.py
> +++
> b/bitbake/lib/toaster/tests/browser/test_builddashboard_page_artifacts.py
> @@ -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et -#
> -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3 #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013-2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from django.core.urlresolvers import reverse
> from django.utils import timezone
> diff --git
> a/bitbake/lib/toaster/tests/browser/test_builddashboard_page_recipes.py
> b/bitbake/lib/toaster/tests/browser/test_builddashboard_page_recipes.py
> index ed18324..c542d45 100644 ---
> a/bitbake/lib/toaster/tests/browser/test_builddashboard_page_recipes.py
> +++
> b/bitbake/lib/toaster/tests/browser/test_builddashboard_page_recipes.py
> @@ -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et -#
> -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3 #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013-2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from django.core.urlresolvers import reverse
> from django.utils import timezone
> diff --git
> a/bitbake/lib/toaster/tests/browser/test_builddashboard_page_tasks.py
> b/bitbake/lib/toaster/tests/browser/test_builddashboard_page_tasks.py
> index da50f16..22acb47 100644 ---
> a/bitbake/lib/toaster/tests/browser/test_builddashboard_page_tasks.py
> +++
> b/bitbake/lib/toaster/tests/browser/test_builddashboard_page_tasks.py
> @@ -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et -#
> -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3 #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013-2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from django.core.urlresolvers import reverse
> from django.utils import timezone
> diff --git a/bitbake/lib/toaster/tests/browser/test_js_unit_tests.py
> b/bitbake/lib/toaster/tests/browser/test_js_unit_tests.py index
> 3c0b962..e8b4295 100644 ---
> a/bitbake/lib/toaster/tests/browser/test_js_unit_tests.py +++
> b/bitbake/lib/toaster/tests/browser/test_js_unit_tests.py @@ -1,23
> +1,11 @@ -#! /usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013-2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> """
> Run the js unit tests
> diff --git a/bitbake/lib/toaster/tests/browser/test_landing_page.py
> b/bitbake/lib/toaster/tests/browser/test_landing_page.py index
> 4d4cd66..0790198 100644 ---
> a/bitbake/lib/toaster/tests/browser/test_landing_page.py +++
> b/bitbake/lib/toaster/tests/browser/test_landing_page.py @@ -1,23
> +1,11 @@ -#! /usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> -# Copyright (C) 2013-2016 Intel Corporation
> -#
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# Copyright (C) 2013-2016 Intel Corporation
> #
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from django.core.urlresolvers import reverse
> from django.utils import timezone
> diff --git
> a/bitbake/lib/toaster/tests/browser/test_layerdetails_page.py
> b/bitbake/lib/toaster/tests/browser/test_layerdetails_page.py index
> f24fb09..f81e696 100644 ---
> a/bitbake/lib/toaster/tests/browser/test_layerdetails_page.py +++
> b/bitbake/lib/toaster/tests/browser/test_layerdetails_page.py @@
> -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> -# Copyright (C) 2013-2016 Intel Corporation
> -#
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# Copyright (C) 2013-2016 Intel Corporation
> #
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from django.core.urlresolvers import reverse
> from tests.browser.selenium_helpers import SeleniumTestCase
> diff --git
> a/bitbake/lib/toaster/tests/browser/test_most_recent_builds_states.py
> b/bitbake/lib/toaster/tests/browser/test_most_recent_builds_states.py
> index abc0b0b..15d25dc 100644 ---
> a/bitbake/lib/toaster/tests/browser/test_most_recent_builds_states.py
> +++
> b/bitbake/lib/toaster/tests/browser/test_most_recent_builds_states.py
> @@ -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et -#
> -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3 #
> # BitBake Toaster Implementation
> #
> -# Copyright (C) 2013-2016 Intel Corporation
> -#
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# Copyright (C) 2013-2016 Intel Corporation
> #
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from django.core.urlresolvers import reverse
> from django.utils import timezone
> diff --git
> a/bitbake/lib/toaster/tests/browser/test_new_custom_image_page.py
> b/bitbake/lib/toaster/tests/browser/test_new_custom_image_page.py
> index ab5a8e6..0aa3b7a 100644 ---
> a/bitbake/lib/toaster/tests/browser/test_new_custom_image_page.py +++
> b/bitbake/lib/toaster/tests/browser/test_new_custom_image_page.py @@
> -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013-2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from django.core.urlresolvers import reverse
> from tests.browser.selenium_helpers import SeleniumTestCase
> diff --git
> a/bitbake/lib/toaster/tests/browser/test_new_project_page.py
> b/bitbake/lib/toaster/tests/browser/test_new_project_page.py index
> 77e5f15..8e56bb0 100644 ---
> a/bitbake/lib/toaster/tests/browser/test_new_project_page.py +++
> b/bitbake/lib/toaster/tests/browser/test_new_project_page.py @@ -1,23
> +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013-2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from django.core.urlresolvers import reverse
> from tests.browser.selenium_helpers import SeleniumTestCase
> diff --git
> a/bitbake/lib/toaster/tests/browser/test_project_builds_page.py
> b/bitbake/lib/toaster/tests/browser/test_project_builds_page.py index
> 9fe91ab..47fb10b 100644 ---
> a/bitbake/lib/toaster/tests/browser/test_project_builds_page.py +++
> b/bitbake/lib/toaster/tests/browser/test_project_builds_page.py @@
> -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013-2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import re
>
> diff --git
> a/bitbake/lib/toaster/tests/browser/test_project_config_page.py
> b/bitbake/lib/toaster/tests/browser/test_project_config_page.py index
> 0710084..2816eb9 100644 ---
> a/bitbake/lib/toaster/tests/browser/test_project_config_page.py +++
> b/bitbake/lib/toaster/tests/browser/test_project_config_page.py @@
> -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013-2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import re
>
> diff --git a/bitbake/lib/toaster/tests/browser/test_project_page.py
> b/bitbake/lib/toaster/tests/browser/test_project_page.py index
> 0186463..8b5e1b6 100644 ---
> a/bitbake/lib/toaster/tests/browser/test_project_page.py +++
> b/bitbake/lib/toaster/tests/browser/test_project_page.py @@ -1,23
> +1,11 @@ -#! /usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013-2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from django.core.urlresolvers import reverse
> from django.utils import timezone
> diff --git a/bitbake/lib/toaster/tests/browser/test_sample.py
> b/bitbake/lib/toaster/tests/browser/test_sample.py index
> 20ec53c..f4ad670 100644 ---
> a/bitbake/lib/toaster/tests/browser/test_sample.py +++
> b/bitbake/lib/toaster/tests/browser/test_sample.py @@ -1,23 +1,11 @@
> -#! /usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013-2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> """
> A small example test demonstrating the basics of writing a test with
> diff --git a/bitbake/lib/toaster/tests/browser/test_task_page.py
> b/bitbake/lib/toaster/tests/browser/test_task_page.py index
> 690d116..26f3dca 100644 ---
> a/bitbake/lib/toaster/tests/browser/test_task_page.py +++
> b/bitbake/lib/toaster/tests/browser/test_task_page.py @@ -1,23 +1,11
> @@ -#! /usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013-2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from django.core.urlresolvers import reverse
> from django.utils import timezone
> diff --git
> a/bitbake/lib/toaster/tests/browser/test_toastertable_ui.py
> b/bitbake/lib/toaster/tests/browser/test_toastertable_ui.py index
> 53ddf30..ef78cbb 100644 ---
> a/bitbake/lib/toaster/tests/browser/test_toastertable_ui.py +++
> b/bitbake/lib/toaster/tests/browser/test_toastertable_ui.py @@ -1,23
> +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013-2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from datetime import datetime
>
> diff --git a/bitbake/lib/toaster/tests/builds/buildtest.py
> b/bitbake/lib/toaster/tests/builds/buildtest.py index
> 5a56a11..872bbd3 100644 ---
> a/bitbake/lib/toaster/tests/builds/buildtest.py +++
> b/bitbake/lib/toaster/tests/builds/buildtest.py @@ -1,23 +1,11 @@
> -#! /usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import os
> import sys
> diff --git a/bitbake/lib/toaster/tests/builds/test_core_image_min.py
> b/bitbake/lib/toaster/tests/builds/test_core_image_min.py index
> 586f4a8..44b6cbe 100644 ---
> a/bitbake/lib/toaster/tests/builds/test_core_image_min.py +++
> b/bitbake/lib/toaster/tests/builds/test_core_image_min.py @@ -1,24
> +1,11 @@ -#! /usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. -
> # Tests were part of openembedded-core oe selftest Authored by:
> Lucian Musat # Ionut Chisanovici, Paul Eggleton and Cristian Iorga
>
> diff --git a/bitbake/lib/toaster/tests/commands/test_loaddata.py
> b/bitbake/lib/toaster/tests/commands/test_loaddata.py index
> 951f6ff..9e8d555 100644 ---
> a/bitbake/lib/toaster/tests/commands/test_loaddata.py +++
> b/bitbake/lib/toaster/tests/commands/test_loaddata.py @@ -1,23 +1,11
> @@ -#! /usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from django.test import TestCase
> from django.core import management
> diff --git a/bitbake/lib/toaster/tests/commands/test_lsupdates.py
> b/bitbake/lib/toaster/tests/commands/test_lsupdates.py index
> 49897a4..3c4fbe0 100644 ---
> a/bitbake/lib/toaster/tests/commands/test_lsupdates.py +++
> b/bitbake/lib/toaster/tests/commands/test_lsupdates.py @@ -1,23 +1,11
> @@ -#! /usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from django.test import TestCase
> from django.core import management
> diff --git a/bitbake/lib/toaster/tests/commands/test_runbuilds.py
> b/bitbake/lib/toaster/tests/commands/test_runbuilds.py index
> 3e63483..e223b95 100644 ---
> a/bitbake/lib/toaster/tests/commands/test_runbuilds.py +++
> b/bitbake/lib/toaster/tests/commands/test_runbuilds.py @@ -1,23 +1,11
> @@ -#! /usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import os
>
> diff --git a/bitbake/lib/toaster/tests/db/test_db.py
> b/bitbake/lib/toaster/tests/db/test_db.py index a0f5f6e..0410422
> 100644 --- a/bitbake/lib/toaster/tests/db/test_db.py
> +++ b/bitbake/lib/toaster/tests/db/test_db.py
> @@ -2,6 +2,8 @@
> #
> # Copyright (c) 2016 Damien Lespiau
> #
> +# SPDX-License-Identifier: MIT
> +#
> # Permission is hereby granted, free of charge, to any person
> obtaining a copy # of this software and associated documentation
> files (the "Software"), to deal # in the Software without
> restriction, including without limitation the rights diff --git
> a/bitbake/lib/toaster/tests/eventreplay/__init__.py
> b/bitbake/lib/toaster/tests/eventreplay/__init__.py index
> 6956619..8ed6792 100644 ---
> a/bitbake/lib/toaster/tests/eventreplay/__init__.py +++
> b/bitbake/lib/toaster/tests/eventreplay/__init__.py @@ -1,23 +1,11 @@
> -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> # Tests were part of openembedded-core oe selftest Authored by:
> Lucian Musat # Ionut Chisanovici, Paul Eggleton and Cristian Iorga
> diff --git
> a/bitbake/lib/toaster/tests/functional/functional_helpers.py
> b/bitbake/lib/toaster/tests/functional/functional_helpers.py index
> 486078a..455c408 100644 ---
> a/bitbake/lib/toaster/tests/functional/functional_helpers.py +++
> b/bitbake/lib/toaster/tests/functional/functional_helpers.py @@ -1,23
> +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster functional tests implementation
> #
> # Copyright (C) 2017 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import os
> import logging
> diff --git
> a/bitbake/lib/toaster/tests/functional/test_functional_basic.py
> b/bitbake/lib/toaster/tests/functional/test_functional_basic.py index
> cfa2b0f..56c84fb 100644 ---
> a/bitbake/lib/toaster/tests/functional/test_functional_basic.py +++
> b/bitbake/lib/toaster/tests/functional/test_functional_basic.py @@
> -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster functional tests implementation
> #
> # Copyright (C) 2017 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import time
> import re
> diff --git a/bitbake/lib/toaster/tests/views/test_views.py
> b/bitbake/lib/toaster/tests/views/test_views.py index
> 1463077..68d9e9d 100644 ---
> a/bitbake/lib/toaster/tests/views/test_views.py +++
> b/bitbake/lib/toaster/tests/views/test_views.py @@ -1,23 +1,11 @@
> -#! /usr/bin/env python
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#! /usr/bin/env python3
> #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013-2015 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> """Test cases for Toaster GUI and ReST."""
>
> diff --git a/bitbake/lib/toaster/toastergui/api.py
> b/bitbake/lib/toaster/toastergui/api.py index 564d595..8b49b3e 100644
> --- a/bitbake/lib/toaster/toastergui/api.py
> +++ b/bitbake/lib/toaster/toastergui/api.py
> @@ -3,19 +3,8 @@
> #
> # Copyright (C) 2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. -
> # Please run flake8 on this file before sending patches
>
> import os
> diff --git a/bitbake/lib/toaster/toastergui/buildtables.py
> b/bitbake/lib/toaster/toastergui/buildtables.py index
> 755a7c2..327059d 100644 ---
> a/bitbake/lib/toaster/toastergui/buildtables.py +++
> b/bitbake/lib/toaster/toastergui/buildtables.py @@ -1,23 +1,10 @@
> #
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> -#
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from orm.models import Build, Task, Target, Package
> from django.db.models import Q, Sum
> diff --git a/bitbake/lib/toaster/toastergui/static/js/importlayer.js
> b/bitbake/lib/toaster/toastergui/static/js/importlayer.js index
> 2964839..8e2032d 100644 ---
> a/bitbake/lib/toaster/toastergui/static/js/importlayer.js +++
> b/bitbake/lib/toaster/toastergui/static/js/importlayer.js @@ -17,11
> +17,15 @@ function importLayerPageInit (ctx) { var
> currentLayerDepSelection; var validLayerName = /^(\w|-)+$/;
>
> + /* Catch 'disable' race condition between type-ahead started and
> "input change" */
> + var typeAheadStarted = 0;
> +
> libtoaster.makeTypeahead(layerDepInput,
> libtoaster.ctx.layersTypeAheadUrl,
> { include_added: "true" }, function(item){
> currentLayerDepSelection = item;
> layerDepBtn.removeAttr("disabled");
> + typeAheadStarted = 1;
> });
>
> layerDepInput.on("typeahead:select", function(event, data){
> @@ -34,7 +38,10 @@ function importLayerPageInit (ctx) {
> // disable the "Add layer" button when the layer input typeahead
> is empty // or not in the typeahead choices
> layerDepInput.on("input change", function(){
> - layerDepBtn.attr("disabled","disabled");
> + if (0 == typeAheadStarted) {
> + layerDepBtn.attr("disabled","disabled");
> + }
> + typeAheadStarted = 0;
> });
>
> /* We automatically add "openembedded-core" layer for convenience
> as a @@ -50,6 +57,7 @@ function importLayerPageInit (ctx) {
> });
>
> layerDepBtn.click(function(){
> + typeAheadStarted = 0;
> if (currentLayerDepSelection == undefined)
> return;
>
> @@ -77,7 +85,7 @@ function importLayerPageInit (ctx) {
>
> $("#layer-deps-list").append(newLayerDep);
>
> -
> libtoaster.getLayerDepsForProject(currentLayerDepSelection.layerdetailurl,
> +
> libtoaster.getLayerDepsForProject(currentLayerDepSelection.xhrLayerUrl,
> function (data){ /* These are the dependencies of the layer added as
> a dependency */ if (data.list.length > 0) {
> diff --git a/bitbake/lib/toaster/toastergui/tablefilter.py
> b/bitbake/lib/toaster/toastergui/tablefilter.py index
> 65454e1..ffef795 100644 ---
> a/bitbake/lib/toaster/toastergui/tablefilter.py +++
> b/bitbake/lib/toaster/toastergui/tablefilter.py @@ -1,23 +1,10 @@
> #
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> -#
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2015 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from django.db.models import Q, Max, Min
> from django.utils import dateparse, timezone
> diff --git a/bitbake/lib/toaster/toastergui/tables.py
> b/bitbake/lib/toaster/toastergui/tables.py index 9ff756b..b3ea222
> 100644 --- a/bitbake/lib/toaster/toastergui/tables.py
> +++ b/bitbake/lib/toaster/toastergui/tables.py
> @@ -1,23 +1,10 @@
> #
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> -#
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2015 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from toastergui.widgets import ToasterTable
> from orm.models import Recipe, ProjectLayer, Layer_Version, Machine,
> Project diff --git
> a/bitbake/lib/toaster/toastergui/templatetags/field_values_filter.py
> b/bitbake/lib/toaster/toastergui/templatetags/field_values_filter.py
> index 5a73af7..eb48339 100644 ---
> a/bitbake/lib/toaster/toastergui/templatetags/field_values_filter.py
> +++
> b/bitbake/lib/toaster/toastergui/templatetags/field_values_filter.py
> @@ -1,3 +1,7 @@ +# +# SPDX-License-Identifier: GPL-2.0-only +#
> +
> from django import template
>
> register = template.Library()
> diff --git
> a/bitbake/lib/toaster/toastergui/templatetags/objects_to_dictionaries_filter.py
> b/bitbake/lib/toaster/toastergui/templatetags/objects_to_dictionaries_filter.py
> index 0dcc7d2..048d533 100644 ---
> a/bitbake/lib/toaster/toastergui/templatetags/objects_to_dictionaries_filter.py
> +++
> b/bitbake/lib/toaster/toastergui/templatetags/objects_to_dictionaries_filter.py
> @@ -1,3 +1,7 @@ +# +# SPDX-License-Identifier: GPL-2.0-only +#
> +
> from django import template
> import json
>
> diff --git
> a/bitbake/lib/toaster/toastergui/templatetags/project_url_tag.py
> b/bitbake/lib/toaster/toastergui/templatetags/project_url_tag.py
> index 04770ac..71e0925 100644 ---
> a/bitbake/lib/toaster/toastergui/templatetags/project_url_tag.py +++
> b/bitbake/lib/toaster/toastergui/templatetags/project_url_tag.py @@
> -1,3 +1,7 @@ +# +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> from django import template
> from django.core.urlresolvers import reverse
>
> diff --git
> a/bitbake/lib/toaster/toastergui/templatetags/projecttags.py
> b/bitbake/lib/toaster/toastergui/templatetags/projecttags.py index
> b170a16..1dbab3b 100644 ---
> a/bitbake/lib/toaster/toastergui/templatetags/projecttags.py +++
> b/bitbake/lib/toaster/toastergui/templatetags/projecttags.py @@ -1,23
> +1,10 @@ # -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> -#
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from datetime import datetime, timedelta
> from os.path import relpath
> diff --git a/bitbake/lib/toaster/toastergui/typeaheads.py
> b/bitbake/lib/toaster/toastergui/typeaheads.py index 5aa0f8d..fd750ff
> 100644 --- a/bitbake/lib/toaster/toastergui/typeaheads.py
> +++ b/bitbake/lib/toaster/toastergui/typeaheads.py
> @@ -3,18 +3,8 @@
> #
> # Copyright (C) 2015 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> import subprocess
>
> diff --git a/bitbake/lib/toaster/toastergui/urls.py
> b/bitbake/lib/toaster/toastergui/urls.py index dc03e30..673d9ae 100644
> --- a/bitbake/lib/toaster/toastergui/urls.py
> +++ b/bitbake/lib/toaster/toastergui/urls.py
> @@ -3,18 +3,8 @@
> #
> # Copyright (C) 2013-2017 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from django.conf.urls import include, url
> from django.views.generic import RedirectView, TemplateView
> diff --git a/bitbake/lib/toaster/toastergui/views.py
> b/bitbake/lib/toaster/toastergui/views.py index c712b06..d7acaff
> 100644 --- a/bitbake/lib/toaster/toastergui/views.py
> +++ b/bitbake/lib/toaster/toastergui/views.py
> @@ -1,24 +1,10 @@
> #
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> -#
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA. -
>
> import re
>
> diff --git a/bitbake/lib/toaster/toastergui/widgets.py
> b/bitbake/lib/toaster/toastergui/widgets.py index db5c3aa..645f458
> 100644 --- a/bitbake/lib/toaster/toastergui/widgets.py
> +++ b/bitbake/lib/toaster/toastergui/widgets.py
> @@ -1,23 +1,10 @@
> #
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> -#
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2015 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from django.views.generic import View, TemplateView
> from django.views.decorators.cache import cache_control
> diff --git
> a/bitbake/lib/toaster/toastermain/management/commands/builddelete.py
> b/bitbake/lib/toaster/toastermain/management/commands/builddelete.py
> index bf69a8f..c2d773a 100644 ---
> a/bitbake/lib/toaster/toastermain/management/commands/builddelete.py
> +++
> b/bitbake/lib/toaster/toastermain/management/commands/builddelete.py
> @@ -1,3 +1,7 @@ +# +# SPDX-License-Identifier: GPL-2.0-only +#
> +
> from django.core.management.base import BaseCommand, CommandError
> from django.core.exceptions import ObjectDoesNotExist
> from orm.models import Build
> diff --git
> a/bitbake/lib/toaster/toastermain/management/commands/buildimport.py
> b/bitbake/lib/toaster/toastermain/management/commands/buildimport.py
> index 2d57ab5..408ad44 100644 ---
> a/bitbake/lib/toaster/toastermain/management/commands/buildimport.py
> +++
> b/bitbake/lib/toaster/toastermain/management/commands/buildimport.py
> @@ -1,23 +1,10 @@ # -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4;
> c-basic-offset: 4; indent-tabs-mode: nil -*- -#
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2018 Wind River Systems
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> # buildimport: import a project for project specific configuration
> #
> diff --git
> a/bitbake/lib/toaster/toastermain/management/commands/buildslist.py
> b/bitbake/lib/toaster/toastermain/management/commands/buildslist.py
> index 70b5812..1ed2022 100644 ---
> a/bitbake/lib/toaster/toastermain/management/commands/buildslist.py
> +++
> b/bitbake/lib/toaster/toastermain/management/commands/buildslist.py
> @@ -1,3 +1,7 @@ +# +# SPDX-License-Identifier: GPL-2.0-only +#
> +
> from django.core.management.base import BaseCommand, CommandError
> from orm.models import Build
> import os
> diff --git
> a/bitbake/lib/toaster/toastermain/management/commands/checksocket.py
> b/bitbake/lib/toaster/toastermain/management/commands/checksocket.py
> index 0399b86..811fd5d 100644 ---
> a/bitbake/lib/toaster/toastermain/management/commands/checksocket.py
> +++
> b/bitbake/lib/toaster/toastermain/management/commands/checksocket.py
> @@ -1,23 +1,11 @@ -#!/usr/bin/env python -# ex:ts=4:sw=4:sts=4:et -#
> -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> +#!/usr/bin/env python3 #
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2015 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> """Custom management command checksocket."""
>
> diff --git
> a/bitbake/lib/toaster/toastermain/management/commands/perf.py
> b/bitbake/lib/toaster/toastermain/management/commands/perf.py index
> 6b450bb..7d629fb 100644 ---
> a/bitbake/lib/toaster/toastermain/management/commands/perf.py +++
> b/bitbake/lib/toaster/toastermain/management/commands/perf.py @@ -1,3
> +1,7 @@ +# +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> from django.core.management.base import BaseCommand
> from django.test.client import Client
> import os, sys, re
> diff --git a/bitbake/lib/toaster/toastermain/settings.py
> b/bitbake/lib/toaster/toastermain/settings.py index 13541d3..74501fa
> 100644 --- a/bitbake/lib/toaster/toastermain/settings.py
> +++ b/bitbake/lib/toaster/toastermain/settings.py
> @@ -1,23 +1,10 @@
> #
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> -#
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> # Django settings for Toaster project.
>
> diff --git
> a/bitbake/lib/toaster/toastermain/settings_production_example.py
> b/bitbake/lib/toaster/toastermain/settings_production_example.py
> index 61a2888..6cd0f52 100644 ---
> a/bitbake/lib/toaster/toastermain/settings_production_example.py +++
> b/bitbake/lib/toaster/toastermain/settings_production_example.py @@
> -1,23 +1,10 @@ # -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> -#
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> # See Django documentation for more information about deployment
> # https://docs.djangoproject.com/en/1.8/howto/deployment/checklist/
> diff --git a/bitbake/lib/toaster/toastermain/settings_test.py
> b/bitbake/lib/toaster/toastermain/settings_test.py index
> a322711..6538d9e 100644 ---
> a/bitbake/lib/toaster/toastermain/settings_test.py +++
> b/bitbake/lib/toaster/toastermain/settings_test.py @@ -1,23 +1,10 @@
> #
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> -#
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2016 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> # Django settings for Toaster project.
>
> diff --git a/bitbake/lib/toaster/toastermain/urls.py
> b/bitbake/lib/toaster/toastermain/urls.py index e2fb0ae..ac77bc3
> 100644 --- a/bitbake/lib/toaster/toastermain/urls.py
> +++ b/bitbake/lib/toaster/toastermain/urls.py
> @@ -1,23 +1,10 @@
> #
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> -#
> # BitBake Toaster Implementation
> #
> # Copyright (C) 2013 Intel Corporation
> #
> -# This program is free software; you can redistribute it and/or
> modify -# it under the terms of the GNU General Public License
> version 2 as -# published by the Free Software Foundation.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> +# SPDX-License-Identifier: GPL-2.0-only
> #
> -# You should have received a copy of the GNU General Public License
> along -# with this program; if not, write to the Free Software
> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA.
> from django.conf.urls import include, url
> from django.views.generic import RedirectView, TemplateView
> diff --git a/bitbake/lib/toaster/toastermain/wsgi.py
> b/bitbake/lib/toaster/toastermain/wsgi.py index 031b314..4c31283
> 100644 --- a/bitbake/lib/toaster/toastermain/wsgi.py
> +++ b/bitbake/lib/toaster/toastermain/wsgi.py
> @@ -1,7 +1,8 @@
> -"""
> -# ex:ts=4:sw=4:sts=4:et
> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
> #
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +
> +"""
> WSGI config for Toaster project.
>
> This module contains the WSGI application used by Django's
> development server diff --git a/doc/user_manual.md
> b/doc/user_manual.md index c2657da..96aa2e3 100644
> --- a/doc/user_manual.md
> +++ b/doc/user_manual.md
> @@ -135,8 +135,8 @@ DISTRO_ARCH ??= "armhf"
> Then, call `bitbake` with image names, e.g.:
>
> ```
> -bitbake multiconfig:qemuarm-buster:isar-image-base \
> - multiconfig:qemuarm-buster:isar-image-debug
> +bitbake mc:qemuarm-buster:isar-image-base \
> + mc:qemuarm-buster:isar-image-debug
> ```
>
> The following images are created:
> @@ -169,14 +169,14 @@ The following command will produce
> `isar-image-base` images for all targets:
> ```
> $ bitbake \
> - multiconfig:qemuarm-stretch:isar-image-base \
> - multiconfig:qemuarm-buster:isar-image-base \
> - multiconfig:qemuarm64-stretch:isar-image-base \
> - multiconfig:qemui386-stretch:isar-image-base \
> - multiconfig:qemui386-buster:isar-image-base \
> - multiconfig:qemuamd64-stretch:isar-image-base \
> - multiconfig:qemuamd64-buster:isar-image-base \
> - multiconfig:rpi-stretch:isar-image-base
> + mc:qemuarm-stretch:isar-image-base \
> + mc:qemuarm-buster:isar-image-base \
> + mc:qemuarm64-stretch:isar-image-base \
> + mc:qemui386-stretch:isar-image-base \
> + mc:qemui386-buster:isar-image-base \
> + mc:qemuamd64-stretch:isar-image-base \
> + mc:qemuamd64-buster:isar-image-base \
> + mc:rpi-stretch:isar-image-base
> ```
>
> Created images are:
> @@ -197,9 +197,9 @@ tmp/deploy/images/rpi/isar-image-base.rpi-sdimg
> A bootable disk image is generated if you set IMAGE_TYPE to
> 'wic-img'. Behind the scenes a tool called `wic` is used to assemble
> the images. It is controlled by a `.wks` file which you can choose
> with changing WKS_FILE. Some examples in the tree use that feature
> already. ``` # Generate an image for the `i386` target architecture
> - $ bitbake multiconfig:qemui386-buster:isar-image-base
> + $ bitbake mc:qemui386-buster:isar-image-base
> # Similarly, for the `amd64` target architecture, in this case EFI
> - $ bitbake multiconfig:qemuamd64-buster:isar-image-base
> + $ bitbake mc:qemuamd64-buster:isar-image-base
> ```
>
> Variables may be used in `.wks.in` files; Isar will expand them and
> generate a regular `.wks` file before generating the disk image using
> `wic`. @@ -303,7 +303,7 @@ following variables define the default
> configuration to build for:
> - `DISTRO_ARCH` - The Debian architecture to build for (e.g.,
> `armhf`).
> If BitBake is called with multiconfig targets (e.g.,
> -`multiconfig:qemuarm-buster:isar-image-base`), the following
> variable defines +`mc:qemuarm-buster:isar-image-base`), the following
> variable defines all supported configurations:
>
> - `BBMULTICONFIG` - The list of the complete configuration
> definition files. @@ -687,7 +687,7 @@ Debian cross-compilation works
> out of the box starting from Debian stretch distr Just like
> OpenEmbedded, Isar supports a devshell target for all dpkg package
> recipes. This target opens a terminal inside the buildchroot that
> runs the package build. To invoke it, just call -`bitbake
> multiconfig:${MACHINE}-${DISTRO}:<package_name> -c devshell`.
> +`bitbake mc:${MACHINE}-${DISTRO}:<package_name> -c devshell`.
>
> ## Create an ISAR SDK root filesystem
> @@ -705,7 +705,7 @@ target binary artifacts. Developer chroots to sdk
> rootfs and develops applicatio ### Solution
>
> User manually triggers creation of SDK root filesystem for his
> target platform by launching the task `do_populate_sdk` for target
> image, f.e. -`bitbake -c do_populate_sdk
> multiconfig:${MACHINE}-${DISTRO}:isar-image-base`. +`bitbake -c
> do_populate_sdk mc:${MACHINE}-${DISTRO}:isar-image-base`. The
> resulting SDK rootfs is archived into
> `tmp/deploy/images/${MACHINE}/sdk-${DISTRO}-${DISTRO_ARCH}.tar.xz`.
> It is additionally available for direct use under
> `tmp/deploy/images/${MACHINE}/sdk-${DISTRO}-${DISTRO_ARCH}/`. @@
> -717,7 +717,7 @@ One may chroot into the SDK and install required
> target packages with the help o
> - Trigger creation of SDK root filesystem
>
> ```
> -bitbake -c do_populate_sdk multiconfig:qemuarm-buster:isar-image-base
> +bitbake -c do_populate_sdk mc:qemuarm-buster:isar-image-base
> ```
>
> - Mount the following directories in chroot by passing resulting
> rootfs as an argument to the script `mount_chroot.sh`: @@ -801,7
> +801,7 @@ BASE_REPO_KEY =
> "file://<absolute_path_to_your_pub_key_file>"'
> - Trigger creation of local apt caching Debian packages during
> image generation.
> ```
> -bitbake -c cache_base_repo multiconfig:qemuarm-buster:isar-image-base
> +bitbake -c cache_base_repo mc:qemuarm-buster:isar-image-base
> ```
>
> - Set `ISAR_USE_CACHED_BASE_REPO` in `conf/local.conf`:
> @@ -820,7 +820,7 @@ sudo rm -rf tmp
> - Trigger again generation of image (now using local caching repo):
>
> ```
> -bitbake multiconfig:qemuarm-buster:isar-image-base
> +bitbake mc:qemuarm-buster:isar-image-base
> ```
>
> ### Limitation
> @@ -860,5 +860,5 @@ DISTRO_APT_SOURCES_append = "
> conf/distro/docker-buster.list" And build the corresponding image
> target:
> ```
> -bitbake multiconfig:qemuarm64-buster:isar-image-base
> +bitbake mc:qemuarm64-buster:isar-image-base
> ```
> diff --git a/meta-isar/conf/conf-notes.txt
> b/meta-isar/conf/conf-notes.txt index 74df97c..c557cd3 100644
> --- a/meta-isar/conf/conf-notes.txt
> +++ b/meta-isar/conf/conf-notes.txt
> @@ -1,4 +1,4 @@
> Common targets are:
> - multiconfig:qemuarm-buster:isar-image-base
> - multiconfig:qemuamd64-buster:isar-image-base
> - multiconfig:rpi-stretch:isar-image-base
> + mc:qemuarm-buster:isar-image-base
> + mc:qemuamd64-buster:isar-image-base
> + mc:rpi-stretch:isar-image-base
> diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
> index 713d1c2..a8556a6 100755
> --- a/scripts/ci_build.sh
> +++ b/scripts/ci_build.sh
> @@ -21,40 +21,40 @@ BUILD_DIR=./build
> BB_ARGS="-v"
>
> TARGETS_SET="\
> - multiconfig:qemuarm-stretch:isar-image-base \
> - multiconfig:qemuarm-buster:isar-image-base \
> - multiconfig:qemuarm-bullseye:isar-image-base \
> - multiconfig:qemuarm64-stretch:isar-image-base \
> - multiconfig:qemui386-stretch:isar-image-base \
> - multiconfig:qemui386-buster:isar-image-base \
> - multiconfig:qemui386-bullseye:isar-image-base \
> - multiconfig:qemuamd64-stretch:isar-image-base \
> - multiconfig:qemuamd64-buster:isar-image-base \
> - multiconfig:qemuamd64-buster-tgz:isar-image-base \
> - multiconfig:qemuamd64-bullseye:isar-image-base \
> - multiconfig:qemumipsel-stretch:isar-image-base \
> - multiconfig:qemumipsel-buster:isar-image-base \
> - multiconfig:qemumipsel-bullseye:isar-image-base \
> - multiconfig:nand-ubi-demo-buster:isar-image-ubi \
> - multiconfig:rpi-stretch:isar-image-base"
> + mc:qemuarm-stretch:isar-image-base \
> + mc:qemuarm-buster:isar-image-base \
> + mc:qemuarm-bullseye:isar-image-base \
> + mc:qemuarm64-stretch:isar-image-base \
> + mc:qemui386-stretch:isar-image-base \
> + mc:qemui386-buster:isar-image-base \
> + mc:qemui386-bullseye:isar-image-base \
> + mc:qemuamd64-stretch:isar-image-base \
> + mc:qemuamd64-buster:isar-image-base \
> + mc:qemuamd64-buster-tgz:isar-image-base \
> + mc:qemuamd64-bullseye:isar-image-base \
> + mc:qemumipsel-stretch:isar-image-base \
> + mc:qemumipsel-buster:isar-image-base \
> + mc:qemumipsel-bullseye:isar-image-base \
> + mc:nand-ubi-demo-buster:isar-image-ubi \
> + mc:rpi-stretch:isar-image-base"
> # qemu-user-static of <= buster too old to build that
> - # multiconfig:qemuarm64-buster:isar-image-base
> - # multiconfig:qemuarm64-bullseye:isar-image-base
> + # mc:qemuarm64-buster:isar-image-base
> + # mc:qemuarm64-bullseye:isar-image-base
>
> CROSS_TARGETS_SET="\
> - multiconfig:qemuarm-stretch:isar-image-base \
> - multiconfig:qemuarm-buster:isar-image-base \
> - multiconfig:qemuarm-bullseye:isar-image-base \
> - multiconfig:qemuarm64-stretch:isar-image-base \
> - multiconfig:qemuamd64-stretch:isar-image-base \
> - multiconfig:de0-nano-soc-stretch:isar-image-base \
> - multiconfig:rpi-stretch:isar-image-base"
> + mc:qemuarm-stretch:isar-image-base \
> + mc:qemuarm-buster:isar-image-base \
> + mc:qemuarm-bullseye:isar-image-base \
> + mc:qemuarm64-stretch:isar-image-base \
> + mc:qemuamd64-stretch:isar-image-base \
> + mc:de0-nano-soc-stretch:isar-image-base \
> + mc:rpi-stretch:isar-image-base"
>
> REPRO_TARGETS_SET="\
> - multiconfig:qemuarm-stretch:isar-image-base \
> - multiconfig:qemuarm64-stretch:isar-image-base \
> - multiconfig:qemuamd64-stretch:isar-image-base \
> - multiconfig:qemuarm-buster:isar-image-base"
> + mc:qemuarm-stretch:isar-image-base \
> + mc:qemuarm64-stretch:isar-image-base \
> + mc:qemuamd64-stretch:isar-image-base \
> + mc:qemuarm-buster:isar-image-base"
>
>
> show_help() {
> @@ -162,7 +162,7 @@ sed -i -e 's/ISAR_CROSS_COMPILE ?=
> "0"/ISAR_CROSS_COMPILE ?= "1"/g' conf/local.c bitbake $BB_ARGS
> $CROSS_TARGETS_SET while [ -e bitbake.sock ]; do sleep 1; done
> # In addition test SDK creation
> -bitbake $BB_ARGS -c do_populate_sdk
> multiconfig:qemuarm-stretch:isar-image-base +bitbake $BB_ARGS -c
> do_populate_sdk mc:qemuarm-stretch:isar-image-base while [ -e
> bitbake.sock ]; do sleep 1; done
> if [ -z "$FAST_BUILD" ]; then
> @@ -175,6 +175,6 @@ fi
> cp -a "${ISARROOT}/meta/classes/dpkg-base.bbclass"
> "${ISARROOT}/meta/classes/dpkg-base.bbclass.ci-backup" echo -e
> "do_fetch_append() {\n\n}" >>
> "${ISARROOT}/meta/classes/dpkg-base.bbclass" -bitbake $BB_ARGS
> multiconfig:qemuamd64-stretch:isar-image-base +bitbake $BB_ARGS
> mc:qemuamd64-stretch:isar-image-base
> mv "${ISARROOT}/meta/classes/dpkg-base.bbclass.ci-backup"
> "${ISARROOT}/meta/classes/dpkg-base.bbclass" diff --git
> a/scripts/start_vm b/scripts/start_vm index 9f07a03..71b55f2 100755
> --- a/scripts/start_vm
> +++ b/scripts/start_vm
> @@ -101,7 +101,7 @@ do
> shift
> done
>
> -eval $(bitbake -e multiconfig:qemu$ARCH-$DISTRO:isar-image-base |
> grep "^DEPLOY_DIR_IMAGE=") +eval $(bitbake -e
> mc:qemu$ARCH-$DISTRO:isar-image-base | grep "^DEPLOY_DIR_IMAGE=")
> readonly IMAGE_DIR=$DEPLOY_DIR_IMAGE
> readonly ISARROOT="$(dirname "$0")"/..
> @@ -112,13 +112,13 @@ eval "$(egrep 'MACHINE_SERIAL' $MACHINE_CONF
> |bb2sh)" readonly
> CONFIG_CONF=$ISARROOT/meta-isar/conf/multiconfig/qemu$ARCH-$DISTRO.conf
> eval "$(egrep 'QEMU_' $CONFIG_CONF |bb2sh)"
> -eval $(bitbake -e multiconfig:qemu$ARCH-$DISTRO:isar-image-base |
> grep "^IMAGE_TYPE=") +eval $(bitbake -e
> mc:qemu$ARCH-$DISTRO:isar-image-base | grep "^IMAGE_TYPE=") case
> "$IMAGE_TYPE" in ext4-img)
> readonly
> ROOTFS_IMAGE=isar-image-base-debian-$DISTRO-qemu$ARCH.ext4.img
> - eval $(bitbake -e multiconfig:qemu$ARCH-$DISTRO:isar-image-base
> | grep "^KERNEL_IMAGE=")
> - eval $(bitbake -e multiconfig:qemu$ARCH-$DISTRO:isar-image-base
> | grep "^INITRD_IMAGE=")
> + eval $(bitbake -e mc:qemu$ARCH-$DISTRO:isar-image-base | grep
> "^KERNEL_IMAGE=")
> + eval $(bitbake -e mc:qemu$ARCH-$DISTRO:isar-image-base | grep
> "^INITRD_IMAGE=") QKERNEL=$IMAGE_DIR/${KERNEL_IMAGE}
> QINITRD=/dev/null
> [ -n "$INITRD_IMAGE" ] && QINITRD=$IMAGE_DIR/${INITRD_IMAGE}
> diff --git a/testsuite/build_test/build_test.py
> b/testsuite/build_test/build_test.py index 4220d3a..7a55c2f 100644
> --- a/testsuite/build_test/build_test.py
> +++ b/testsuite/build_test/build_test.py
> @@ -22,7 +22,7 @@ class BuildTest(Test):
>
> #isar_root = dirname(__file__) + '/..'
> os.chdir(build_dir)
> - cmdline = ['bitbake', 'multiconfig:qemu' + arch + '-' +
> distro + ':isar-image-base']
> + cmdline = ['bitbake', 'mc:qemu' + arch + '-' + distro +
> ':isar-image-base'] p1 = subprocess32.run(cmdline)
>
> if p1.returncode:
> diff --git a/testsuite/start_vm.py b/testsuite/start_vm.py
> index 02a4b51..a3e32ac 100755
> --- a/testsuite/start_vm.py
> +++ b/testsuite/start_vm.py
> @@ -10,7 +10,7 @@ import sys
> import time
>
> def get_bitbake_env(arch, distro):
> - multiconfig = 'multiconfig:qemu' + arch + '-' + distro +
> ':isar-image-base'
> + multiconfig = 'mc:qemu' + arch + '-' + distro +
> ':isar-image-base' output = subprocess.check_output(['bitbake', '-e',
> str(multiconfig)]) return output
>
next prev parent reply other threads:[~2019-10-22 7:36 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-10-21 14:19 [PATCH 0/1] update bitbake " Cedric Hombourger
2019-10-21 14:19 ` [PATCH 1/1] bitbake: update " Cedric Hombourger
2019-10-22 7:36 ` Henning Schild [this message]
2019-10-22 7:41 ` Cedric Hombourger
2019-10-22 15:56 ` Baurzhan Ismagulov
2019-10-24 6:58 ` chombourger
2019-10-24 19:44 ` Baurzhan Ismagulov
2019-11-05 12:36 ` chombourger
2019-11-06 6:01 ` Baurzhan Ismagulov
2019-11-06 6:02 ` Cedric Hombourger
2019-11-08 8:05 ` [PATCH 0/1] update bitbake " Jan Kiszka
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20191022093609.5ed343a5@md1za8fc.ad001.siemens.net \
--to=henning.schild@siemens.com \
--cc=Cedric_Hombourger@mentor.com \
--cc=isar-users@googlegroups.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox