From: Cedric Hombourger <cedric_hombourger@mentor.com>
To: Henning Schild <henning.schild@siemens.com>
Cc: <isar-users@googlegroups.com>
Subject: Re: [PATCH 1/1] bitbake: update to version 1.44.0
Date: Tue, 22 Oct 2019 09:41:59 +0200 [thread overview]
Message-ID: <514fe94a-882a-9b77-b554-f79b255a89d3@mentor.com> (raw)
In-Reply-To: <20191022093609.5ed343a5@md1za8fc.ad001.siemens.net>
Hi Henning
On 10/22/2019 9:36 AM, Henning Schild wrote:
> Hi Cedric,
>
> i would suggest a split between the actual bump and the following
> Isar-changes. But i do not care too much and both ways have their pros
> and cons.
I have debated (with myself :)) and I agree that there are pros and cons
with each approach.
I ended up submitting a single commit at the end of the day to support
people needing to "git bisect".
I have otherwise no issues resubmitting the changes if we prefer to have
the bitbake upgrade clearly separated from the Isar changes (either way
is fine with me - @maintainers just let me know what your preference is!)
> Henning
>
> Am Mon, 21 Oct 2019 16:19:58 +0200
> schrieb Cedric Hombourger <Cedric_Hombourger@mentor.com>:
>
>> Found that the gitsm fetcher we ship is rather out-dated compared to
>> upstream and was failing to pull git sub-modules from repositories
>> such as azure-umqtt-c.
>>
>> It shall be noted that bitbake has renamed its multiconfig targets:
>> the "multiconfig:" prefix was changed to "mc:". Updated our scripts
>> as well as documentation.
>>
>> Origin: https://github.com/openembedded/bitbake.git
>> Commit 5d83d828cacb58ccb7c464e799c85fd2d2a50ccc (tag: 1.44.0)
>> Author: Richard Purdie <richard.purdie@linuxfoundation.org>
>> Date: Wed Oct 9 14:10:21 2019 +0100
>>
>> bitbake: Update to version 1.44.0
>>
>> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>>
>> Signed-off-by: Cedric Hombourger <Cedric_Hombourger@mentor.com>
>> ---
>> RECIPE-API-CHANGELOG.md | 16 +
>> bitbake/.gitattributes | 2 +
>> bitbake/HEADER | 19 -
>> bitbake/LICENSE | 12 +-
>> bitbake/{COPYING => LICENSE.GPL-2.0-only} | 61 +-
>> bitbake/LICENSE.MIT | 25 +
>> bitbake/MANIFEST.in | 5 +-
>> bitbake/bin/bitbake | 16 +-
>> bitbake/bin/bitbake-diffsigs | 155 +-
>> bitbake/bin/bitbake-dumpsig | 95 +-
>> bitbake/bin/bitbake-hashclient | 170 ++
>> bitbake/bin/bitbake-hashserv | 62 +
>> bitbake/bin/bitbake-layers | 12 +-
>> bitbake/bin/bitbake-prserv | 4 +
>> bitbake/bin/bitbake-selftest | 16 +-
>> bitbake/bin/bitbake-worker | 19 +-
>> bitbake/bin/bitdoc | 14 +-
>> bitbake/bin/git-make-shallow | 4 +
>> bitbake/bin/toaster | 12 +-
>> bitbake/bin/toaster-eventreplay | 17 +-
>> bitbake/classes/base.bbclass | 2 +-
>> bitbake/contrib/dump_cache.py | 2 -
>> .../bitbake-user-manual-execution.xml | 78 +-
>> .../bitbake-user-manual-fetching.xml | 69 +-
>> .../bitbake-user-manual-hello.xml | 28 +-
>> .../bitbake-user-manual-intro.xml | 18 +-
>> .../bitbake-user-manual-metadata.xml | 179 +-
>> .../bitbake-user-manual-ref-variables.xml | 422 ++---
>> bitbake/doc/poky.ent | 8 -
>> bitbake/lib/bb/COW.py | 15 -
>> bitbake/lib/bb/__init__.py | 16 +-
>> bitbake/lib/bb/build.py | 151 +-
>> bitbake/lib/bb/cache.py | 109 +-
>> bitbake/lib/bb/cache_extra.py | 14 +-
>> bitbake/lib/bb/checksum.py | 12 +-
>> bitbake/lib/bb/codeparser.py | 13 +-
>> bitbake/lib/bb/command.py | 12 +-
>> bitbake/lib/bb/compat.py | 4 +
>> bitbake/lib/bb/cooker.py | 218 +--
>> bitbake/lib/bb/cookerdata.py | 44 +-
>> bitbake/lib/bb/daemonize.py | 4 +
>> bitbake/lib/bb/data.py | 23 +-
>> bitbake/lib/bb/data_smart.py | 25 +-
>> bitbake/lib/bb/event.py | 33 +-
>> bitbake/lib/bb/exceptions.py | 3 +
>> bitbake/lib/bb/fetch2/__init__.py | 55 +-
>> bitbake/lib/bb/fetch2/bzr.py | 12 +-
>> bitbake/lib/bb/fetch2/clearcase.py | 19 +-
>> bitbake/lib/bb/fetch2/cvs.py | 17 +-
>> bitbake/lib/bb/fetch2/git.py | 68 +-
>> bitbake/lib/bb/fetch2/gitannex.py | 14 +-
>> bitbake/lib/bb/fetch2/gitsm.py | 281 ++-
>> bitbake/lib/bb/fetch2/hg.py | 18 +-
>> bitbake/lib/bb/fetch2/local.py | 16 +-
>> bitbake/lib/bb/fetch2/npm.py | 69 +-
>> bitbake/lib/bb/fetch2/osc.py | 5 +-
>> bitbake/lib/bb/fetch2/perforce.py | 15 +-
>> bitbake/lib/bb/fetch2/repo.py | 16 +-
>> bitbake/lib/bb/fetch2/s3.py | 15 +-
>> bitbake/lib/bb/fetch2/sftp.py | 15 +-
>> bitbake/lib/bb/fetch2/ssh.py | 14 +-
>> bitbake/lib/bb/fetch2/svn.py | 98 +-
>> bitbake/lib/bb/fetch2/wget.py | 104 +-
>> bitbake/lib/bb/main.py | 41 +-
>> bitbake/lib/bb/methodpool.py | 15 +-
>> bitbake/lib/bb/monitordisk.py | 27 +-
>> bitbake/lib/bb/msg.py | 14 +-
>> bitbake/lib/bb/namedtuple_with_abc.py | 4 +-
>> bitbake/lib/bb/parse/__init__.py | 14 +-
>> bitbake/lib/bb/parse/ast.py | 15 +-
>> bitbake/lib/bb/parse/parse_py/BBHandler.py | 59 +-
>> bitbake/lib/bb/parse/parse_py/ConfHandler.py | 17 +-
>> bitbake/lib/bb/parse/parse_py/__init__.py | 17 +-
>> bitbake/lib/bb/persist_data.py | 234 ++-
>> bitbake/lib/bb/process.py | 4 +
>> bitbake/lib/bb/progress.py | 28 +-
>> bitbake/lib/bb/providers.py | 18 +-
>> bitbake/lib/bb/pysh/builtin.py | 710 --------
>> bitbake/lib/bb/pysh/interp.py | 1367 ---------------
>> bitbake/lib/bb/pysh/lsprof.py | 116 --
>> bitbake/lib/bb/pysh/pysh.py | 167 --
>> bitbake/lib/bb/pysh/pyshlex.py | 5 -
>> bitbake/lib/bb/pysh/pyshyacc.py | 17 +-
>> bitbake/lib/bb/pysh/sherrors.py | 26 -
>> bitbake/lib/bb/pysh/subprocess_fix.py | 77 -
>> bitbake/lib/bb/remotedata.py | 12 +-
>> bitbake/lib/bb/runqueue.py | 1523
>> ++++++++++------- bitbake/lib/bb/server/__init__.py |
>> 14 +- bitbake/lib/bb/server/process.py | 123 +-
>> bitbake/lib/bb/server/xmlrpcclient.py | 12 +-
>> bitbake/lib/bb/server/xmlrpcserver.py | 12 +-
>> bitbake/lib/bb/siggen.py | 350 +++-
>> bitbake/lib/bb/taskdata.py | 17 +-
>> bitbake/lib/bb/tests/codeparser.py | 22 +-
>> bitbake/lib/bb/tests/cooker.py | 15 +-
>> bitbake/lib/bb/tests/cow.py | 17 +-
>> bitbake/lib/bb/tests/data.py | 39 +-
>> bitbake/lib/bb/tests/event.py | 23 +-
>> bitbake/lib/bb/tests/fetch.py | 308 +++-
>> bitbake/lib/bb/tests/parse.py | 33 +-
>> bitbake/lib/bb/tests/persist_data.py | 129 ++
>> .../tests/runqueue-tests/classes/base.bbclass | 262 +++
>> .../runqueue-tests/classes/image.bbclass | 5 +
>> .../runqueue-tests/classes/native.bbclass | 2 +
>> .../bb/tests/runqueue-tests/conf/bitbake.conf | 16 +
>> .../runqueue-tests/conf/multiconfig/mc1.conf | 1 +
>> .../runqueue-tests/conf/multiconfig/mc2.conf | 1 +
>> .../lib/bb/tests/runqueue-tests/recipes/a1.bb | 0
>> .../lib/bb/tests/runqueue-tests/recipes/b1.bb | 1 +
>> .../lib/bb/tests/runqueue-tests/recipes/c1.bb | 0
>> .../lib/bb/tests/runqueue-tests/recipes/d1.bb | 3 +
>> .../lib/bb/tests/runqueue-tests/recipes/e1.bb | 1 +
>> bitbake/lib/bb/tests/runqueue.py | 323 ++++
>> bitbake/lib/bb/tests/utils.py | 23 +-
>> bitbake/lib/bb/tinfoil.py | 12 +-
>> bitbake/lib/bb/ui/__init__.py | 12 +-
>> bitbake/lib/bb/ui/buildinfohelper.py | 22 +-
>> bitbake/lib/bb/ui/knotty.py | 60 +-
>> bitbake/lib/bb/ui/ncurses.py | 12 +-
>> bitbake/lib/bb/ui/taskexp.py | 12 +-
>> bitbake/lib/bb/ui/toasterui.py | 12 +-
>> bitbake/lib/bb/ui/uievent.py | 15 +-
>> bitbake/lib/bb/ui/uihelper.py | 16 +-
>> bitbake/lib/bb/utils.py | 49 +-
>> bitbake/lib/bblayers/__init__.py | 4 +
>> bitbake/lib/bblayers/action.py | 4 +
>> bitbake/lib/bblayers/common.py | 4 +
>> bitbake/lib/bblayers/layerindex.py | 6 +-
>> bitbake/lib/bblayers/query.py | 43 +-
>> bitbake/lib/bs4/dammit.py | 12 +-
>> bitbake/lib/bs4/element.py | 22 +-
>> bitbake/lib/hashserv/__init__.py | 93 +
>> bitbake/lib/hashserv/client.py | 157 ++
>> bitbake/lib/hashserv/server.py | 414 +++++
>> bitbake/lib/hashserv/tests.py | 142 ++
>> bitbake/lib/layerindexlib/__init__.py | 33 +-
>> bitbake/lib/layerindexlib/cooker.py | 12 +-
>> bitbake/lib/layerindexlib/plugin.py | 13 +-
>> bitbake/lib/layerindexlib/restapi.py | 12 +-
>> bitbake/lib/layerindexlib/tests/common.py | 12 +-
>> bitbake/lib/layerindexlib/tests/cooker.py | 12 +-
>> .../lib/layerindexlib/tests/layerindexobj.py | 12 +-
>> bitbake/lib/layerindexlib/tests/restapi.py | 12 +-
>> bitbake/lib/progressbar/__init__.py | 2 +
>> bitbake/lib/progressbar/compat.py | 2 +
>> bitbake/lib/progressbar/progressbar.py | 2 +
>> bitbake/lib/progressbar/widgets.py | 2 +
>> bitbake/lib/prserv/__init__.py | 4 +
>> bitbake/lib/prserv/db.py | 6 +-
>> bitbake/lib/prserv/serv.py | 4 +
>> bitbake/lib/pyinotify.py | 20 +-
>> bitbake/lib/toaster/bldcollector/admin.py | 4 +
>> bitbake/lib/toaster/bldcollector/urls.py | 13 +-
>> bitbake/lib/toaster/bldcollector/views.py | 12 +-
>> bitbake/lib/toaster/bldcontrol/admin.py | 4 +
>> .../lib/toaster/bldcontrol/bbcontroller.py | 16 +-
>> .../bldcontrol/localhostbecontroller.py | 16 +-
>> .../management/commands/checksettings.py | 4 +
>> .../management/commands/runbuilds.py | 4 +
>> bitbake/lib/toaster/bldcontrol/models.py | 4 +
>> bitbake/lib/toaster/bldcontrol/views.py | 4 +
>> bitbake/lib/toaster/manage.py | 4 +
>> bitbake/lib/toaster/orm/fixtures/oe-core.xml | 26 +-
>> bitbake/lib/toaster/orm/fixtures/poky.xml | 38 +-
>> .../orm/management/commands/lsupdates.py | 15 +-
>> bitbake/lib/toaster/orm/models.py | 25 +-
>> .../toaster/tests/browser/selenium_helpers.py | 17 +-
>> .../tests/browser/selenium_helpers_base.py | 17 +-
>> .../tests/browser/test_all_builds_page.py | 16 +-
>> .../tests/browser/test_all_projects_page.py | 16 +-
>> .../tests/browser/test_builddashboard_page.py | 16 +-
>> .../test_builddashboard_page_artifacts.py | 16 +-
>> .../test_builddashboard_page_recipes.py | 16 +-
>> .../browser/test_builddashboard_page_tasks.py | 16 +-
>> .../tests/browser/test_js_unit_tests.py | 16 +-
>> .../tests/browser/test_landing_page.py | 18 +-
>> .../tests/browser/test_layerdetails_page.py | 18 +-
>> .../browser/test_most_recent_builds_states.py | 18 +-
>> .../browser/test_new_custom_image_page.py | 16 +-
>> .../tests/browser/test_new_project_page.py | 16 +-
>> .../tests/browser/test_project_builds_page.py | 16 +-
>> .../tests/browser/test_project_config_page.py | 16 +-
>> .../tests/browser/test_project_page.py | 16 +-
>> .../lib/toaster/tests/browser/test_sample.py | 16 +-
>> .../toaster/tests/browser/test_task_page.py | 16 +-
>> .../tests/browser/test_toastertable_ui.py | 16 +-
>> bitbake/lib/toaster/tests/builds/buildtest.py | 16 +-
>> .../tests/builds/test_core_image_min.py | 17 +-
>> .../toaster/tests/commands/test_loaddata.py | 16 +-
>> .../toaster/tests/commands/test_lsupdates.py | 16 +-
>> .../toaster/tests/commands/test_runbuilds.py | 16 +-
>> bitbake/lib/toaster/tests/db/test_db.py | 2 +
>> .../lib/toaster/tests/eventreplay/__init__.py | 16 +-
>> .../tests/functional/functional_helpers.py | 16 +-
>> .../tests/functional/test_functional_basic.py | 16 +-
>> bitbake/lib/toaster/tests/views/test_views.py | 16 +-
>> bitbake/lib/toaster/toastergui/api.py | 13 +-
>> bitbake/lib/toaster/toastergui/buildtables.py | 15 +-
>> .../toastergui/static/js/importlayer.js | 12 +-
>> bitbake/lib/toaster/toastergui/tablefilter.py | 15 +-
>> bitbake/lib/toaster/toastergui/tables.py | 15 +-
>> .../templatetags/field_values_filter.py | 4 +
>> .../objects_to_dictionaries_filter.py | 4 +
>> .../templatetags/project_url_tag.py | 4 +
>> .../toastergui/templatetags/projecttags.py | 15 +-
>> bitbake/lib/toaster/toastergui/typeaheads.py | 12 +-
>> bitbake/lib/toaster/toastergui/urls.py | 12 +-
>> bitbake/lib/toaster/toastergui/views.py | 16 +-
>> bitbake/lib/toaster/toastergui/widgets.py | 15 +-
>> .../management/commands/builddelete.py | 4 +
>> .../management/commands/buildimport.py | 15 +-
>> .../management/commands/buildslist.py | 4 +
>> .../management/commands/checksocket.py | 16 +-
>> .../toastermain/management/commands/perf.py | 4 +
>> bitbake/lib/toaster/toastermain/settings.py | 15 +-
>> .../settings_production_example.py | 15 +-
>> .../lib/toaster/toastermain/settings_test.py | 15 +-
>> bitbake/lib/toaster/toastermain/urls.py | 15 +-
>> bitbake/lib/toaster/toastermain/wsgi.py | 7 +-
>> doc/user_manual.md | 38 +-
>> meta-isar/conf/conf-notes.txt | 6 +-
>> scripts/ci_build.sh | 62 +-
>> scripts/start_vm | 8 +-
>> testsuite/build_test/build_test.py | 2 +-
>> testsuite/start_vm.py | 2 +-
>> 225 files changed, 5375 insertions(+), 6320 deletions(-)
>> create mode 100644 bitbake/.gitattributes
>> delete mode 100644 bitbake/HEADER
>> rename bitbake/{COPYING => LICENSE.GPL-2.0-only} (84%)
>> create mode 100644 bitbake/LICENSE.MIT
>> mode change 100755 => 120000 bitbake/bin/bitbake-dumpsig
>> create mode 100755 bitbake/bin/bitbake-hashclient
>> create mode 100755 bitbake/bin/bitbake-hashserv
>> delete mode 100644 bitbake/lib/bb/pysh/builtin.py
>> delete mode 100644 bitbake/lib/bb/pysh/interp.py
>> delete mode 100644 bitbake/lib/bb/pysh/lsprof.py
>> delete mode 100644 bitbake/lib/bb/pysh/pysh.py
>> delete mode 100644 bitbake/lib/bb/pysh/subprocess_fix.py
>> create mode 100644 bitbake/lib/bb/tests/persist_data.py
>> create mode 100644
>> bitbake/lib/bb/tests/runqueue-tests/classes/base.bbclass create mode
>> 100644 bitbake/lib/bb/tests/runqueue-tests/classes/image.bbclass
>> create mode 100644
>> bitbake/lib/bb/tests/runqueue-tests/classes/native.bbclass create
>> mode 100644 bitbake/lib/bb/tests/runqueue-tests/conf/bitbake.conf
>> create mode 100644
>> bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc1.conf create
>> mode 100644
>> bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc2.conf create
>> mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/a1.bb create
>> mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/b1.bb create
>> mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/c1.bb create
>> mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/d1.bb create
>> mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/e1.bb create
>> mode 100644 bitbake/lib/bb/tests/runqueue.py create mode 100644
>> bitbake/lib/hashserv/__init__.py create mode 100644
>> bitbake/lib/hashserv/client.py create mode 100644
>> bitbake/lib/hashserv/server.py create mode 100644
>> bitbake/lib/hashserv/tests.py
>>
>> diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
>> index bbef1a3..81ec430 100644
>> --- a/RECIPE-API-CHANGELOG.md
>> +++ b/RECIPE-API-CHANGELOG.md
>> @@ -176,3 +176,19 @@ Otherwise set a encrypted root password like
>> this: USERS += "root"
>> USER_root[password] =
>> "$6$rounds=10000$RXeWrnFmkY$DtuS/OmsAS2cCEDo0BF5qQsizIrq6jPgXnwv3PHqREJeKd1sXdHX/ayQtuQWVDHe0KIO0/sVH8dvQm1KthF0d/"
>> ``` +
>> +### multiconfig build targets were renamed
>> +
>> +bitbake was upgraded to version 1.44.0 where "multiconfig" build
>> targets were +renamed "mc". As an example, builds for the
>> qemuarm-stretch machine should now +be done as follows:
>> +
>> +```
>> +bitbake mc:qemuarm-stretch:isar-image-base
>> +```
>> +
>> +The old syntax is no longer supported and will produce an error:
>> +
>> +```
>> +bitbake multiconfig:qemuarm-stretch:isar-image-base
>> +```
>> diff --git a/bitbake/.gitattributes b/bitbake/.gitattributes
>> new file mode 100644
>> index 0000000..e4f8f62
>> --- /dev/null
>> +++ b/bitbake/.gitattributes
>> @@ -0,0 +1,2 @@
>> +*min.js binary
>> +*min.css binary
>> diff --git a/bitbake/HEADER b/bitbake/HEADER
>> deleted file mode 100644
>> index 9859255..0000000
>> --- a/bitbake/HEADER
>> +++ /dev/null
>> @@ -1,19 +0,0 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> -#
>> -# <one line to give the program's name and a brief idea of what it
>> does.> -# Copyright (C) <year> <name of author>
>> -#
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. -
>> diff --git a/bitbake/LICENSE b/bitbake/LICENSE
>> index 7d4e5f4..8458042 100644
>> --- a/bitbake/LICENSE
>> +++ b/bitbake/LICENSE
>> @@ -1,4 +1,13 @@
>> -BitBake is licensed under the GNU General Public License version
>> 2.0. See COPYING for further details. +BitBake is licensed under the
>> GNU General Public License version 2.0. See +LICENSE.GPL-2.0-only for
>> further details. +
>> +Individual files contain the following style tags instead of the
>> full license text: +
>> + SPDX-License-Identifier: GPL-2.0-only
>> +
>> +This enables machine processing of license information based on the
>> SPDX +License Identifiers that are here available:
>> http://spdx.org/licenses/ +
>>
>> The following external components are distributed with this software:
>>
>> @@ -17,3 +26,4 @@ Foundation and individual contributors.
>> * Font Awesome fonts redistributed under the SIL Open Font License
>> 1.1
>> * simplediff is distributed under the zlib license.
>> +
>> diff --git a/bitbake/COPYING b/bitbake/LICENSE.GPL-2.0-only
>> similarity index 84%
>> rename from bitbake/COPYING
>> rename to bitbake/LICENSE.GPL-2.0-only
>> index d511905..5db3c0a 100644
>> --- a/bitbake/COPYING
>> +++ b/bitbake/LICENSE.GPL-2.0-only
>> @@ -279,61 +279,10 @@ POSSIBILITY OF SUCH DAMAGES.
>>
>> END OF TERMS AND CONDITIONS
>>
>> - How to Apply These Terms to Your New Programs
>> +Note:
>> +Individual files contain the following tag instead of the full
>> license text.
>> - If you develop a new program, and you want it to be of the greatest
>> -possible use to the public, the best way to achieve this is to make
>> it -free software which everyone can redistribute and change under
>> these terms.
>> + SPDX-License-Identifier: GPL-2.0-only
>>
>> - To do so, attach the following notices to the program. It is
>> safest -to attach them to the start of each source file to most
>> effectively -convey the exclusion of warranty; and each file should
>> have at least -the "copyright" line and a pointer to where the full
>> notice is found. -
>> - <one line to give the program's name and a brief idea of what it
>> does.>
>> - Copyright (C) <year> <name of author>
>> -
>> - This program is free software; you can redistribute it and/or
>> modify
>> - it under the terms of the GNU General Public License as
>> published by
>> - the Free Software Foundation; either version 2 of the License, or
>> - (at your option) any later version.
>> -
>> - This program is distributed in the hope that it will be useful,
>> - but WITHOUT ANY WARRANTY; without even the implied warranty of
>> - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> - GNU General Public License for more details.
>> -
>> - You should have received a copy of the GNU General Public
>> License along
>> - with this program; if not, write to the Free Software
>> Foundation, Inc.,
>> - 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
>> -
>> -Also add information on how to contact you by electronic and paper
>> mail. -
>> -If the program is interactive, make it output a short notice like
>> this -when it starts in an interactive mode:
>> -
>> - Gnomovision version 69, Copyright (C) year name of author
>> - Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type
>> `show w'.
>> - This is free software, and you are welcome to redistribute it
>> - under certain conditions; type `show c' for details.
>> -
>> -The hypothetical commands `show w' and `show c' should show the
>> appropriate -parts of the General Public License. Of course, the
>> commands you use may -be called something other than `show w' and
>> `show c'; they could even be -mouse-clicks or menu items--whatever
>> suits your program. -
>> -You should also get your employer (if you work as a programmer) or
>> your -school, if any, to sign a "copyright disclaimer" for the
>> program, if -necessary. Here is a sample; alter the names:
>> -
>> - Yoyodyne, Inc., hereby disclaims all copyright interest in the
>> program
>> - `Gnomovision' (which makes passes at compilers) written by James
>> Hacker. -
>> - <signature of Ty Coon>, 1 April 1989
>> - Ty Coon, President of Vice
>> -
>> -This General Public License does not permit incorporating your
>> program into -proprietary programs. If your program is a subroutine
>> library, you may -consider it more useful to permit linking
>> proprietary applications with the -library. If this is what you want
>> to do, use the GNU Lesser General -Public License instead of this
>> License. +This enables machine processing of license information
>> based on the SPDX +License Identifiers that are here available:
>> http://spdx.org/licenses/ diff --git a/bitbake/LICENSE.MIT
>> b/bitbake/LICENSE.MIT new file mode 100644
>> index 0000000..a6919eb
>> --- /dev/null
>> +++ b/bitbake/LICENSE.MIT
>> @@ -0,0 +1,25 @@
>> +Permission is hereby granted, free of charge, to any person
>> obtaining a copy +of this software and associated documentation files
>> (the "Software"), to deal +in the Software without restriction,
>> including without limitation the rights +to use, copy, modify, merge,
>> publish, distribute, sublicense, and/or sell +copies of the Software,
>> and to permit persons to whom the Software is +furnished to do so,
>> subject to the following conditions: +
>> +The above copyright notice and this permission notice shall be
>> included in +all copies or substantial portions of the Software.
>> +
>> +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
>> EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
>> MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND
>> NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS
>> BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN
>> ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN
>> CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
>> +SOFTWARE. +
>> +Note:
>> +Individual files contain the following tag instead of the full
>> license text. +
>> + SPDX-License-Identifier: MIT
>> +
>> +This enables machine processing of license information based on the
>> SPDX +License Identifiers that are here available:
>> http://spdx.org/licenses/ diff --git a/bitbake/MANIFEST.in
>> b/bitbake/MANIFEST.in index b197378..f24969a 100644
>> --- a/bitbake/MANIFEST.in
>> +++ b/bitbake/MANIFEST.in
>> @@ -1,6 +1,8 @@
>> -include COPYING
>> include ChangeLog
>> include AUTHORS
>> +include LICENSE
>> +include LICENSE.GPL-2.0-only
>> +include LICENSE.MIT
>> include contrib/*
>> include contrib/vim/*/*
>> include conf/*
>> @@ -8,4 +10,3 @@ include classes/*
>> include doc/*
>> include doc/manual/*
>> include ez_setup.py
>> -include HEADER
>> diff --git a/bitbake/bin/bitbake b/bitbake/bin/bitbake
>> index 57dec2a..66d08f8 100755
>> --- a/bitbake/bin/bitbake
>> +++ b/bitbake/bin/bitbake
>> @@ -1,6 +1,4 @@
>> #!/usr/bin/env python3
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # Copyright (C) 2003, 2004 Chris Larson
>> # Copyright (C) 2003, 2004 Phil Blundell
>> @@ -9,18 +7,8 @@
>> # Copyright (C) 2005 ROAD GmbH
>> # Copyright (C) 2006 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import os
>> import sys
>> @@ -38,7 +26,7 @@ from bb.main import bitbake_main,
>> BitBakeConfigParameters, BBMainException if
>> sys.getfilesystemencoding() != "utf-8": sys.exit("Please use a locale
>> setting which supports UTF-8 (such as LANG=en_US.UTF-8).\nPython
>> can't change the filesystem locale after loading so we need a UTF-8
>> when Python starts or things won't work.") -__version__ = "1.40.0"
>> +__version__ = "1.44.0"
>> if __name__ == "__main__":
>> if __version__ != bb.__version__:
>> diff --git a/bitbake/bin/bitbake-diffsigs
>> b/bitbake/bin/bitbake-diffsigs index 4e6bbdd..19420a2 100755
>> --- a/bitbake/bin/bitbake-diffsigs
>> +++ b/bitbake/bin/bitbake-diffsigs
>> @@ -1,27 +1,16 @@
>> #!/usr/bin/env python3
>>
>> -# bitbake-diffsigs
>> -# BitBake task signature data comparison utility
>> +# bitbake-diffsigs / bitbake-dumpsig
>> +# BitBake task signature data dump and comparison utility
>> #
>> # Copyright (C) 2012-2013, 2017 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import os
>> import sys
>> import warnings
>> -import fnmatch
>> import argparse
>> import logging
>> import pickle
>> @@ -32,7 +21,10 @@ import bb.tinfoil
>> import bb.siggen
>> import bb.msg
>>
>> -logger = bb.msg.logger_create('bitbake-diffsigs')
>> +myname = os.path.basename(sys.argv[0])
>> +logger = bb.msg.logger_create(myname)
>> +
>> +is_dump = myname == 'bitbake-dumpsig'
>>
>> def find_siginfo(tinfoil, pn, taskname, sigs=None):
>> result = None
>> @@ -59,8 +51,8 @@ def find_siginfo(tinfoil, pn, taskname, sigs=None):
>> sys.exit(2)
>> return result
>>
>> -def find_compare_task(bbhandler, pn, taskname, sig1=None, sig2=None,
>> color=False):
>> - """ Find the most recent signature files for the specified
>> PN/task and compare them """ +def find_siginfo_task(bbhandler, pn,
>> taskname, sig1=None, sig2=None):
>> + """ Find the most recent signature files for the specified
>> PN/task """
>> if not taskname.startswith('do_'):
>> taskname = 'do_%s' % taskname
>> @@ -79,73 +71,81 @@ def find_compare_task(bbhandler, pn, taskname,
>> sig1=None, sig2=None, color=False latestfiles = [sigfiles[sig1],
>> sigfiles[sig2]] else:
>> filedates = find_siginfo(bbhandler, pn, taskname)
>> - latestfiles = sorted(filedates.keys(), key=lambda f:
>> filedates[f])[-3:]
>> + latestfiles = sorted(filedates.keys(), key=lambda f:
>> filedates[f])[-2:] if not latestfiles:
>> logger.error('No sigdata files found matching %s %s' %
>> (pn, taskname)) sys.exit(1)
>> - elif len(latestfiles) < 2:
>> - logger.error('Only one matching sigdata file found for
>> the specified task (%s %s)' % (pn, taskname))
>> - sys.exit(1)
>>
>> - # Define recursion callback
>> - def recursecb(key, hash1, hash2):
>> - hashes = [hash1, hash2]
>> - hashfiles = find_siginfo(bbhandler, key, None, hashes)
>> -
>> - recout = []
>> - if len(hashfiles) == 0:
>> - recout.append("Unable to find matching sigdata for %s
>> with hashes %s or %s" % (key, hash1, hash2))
>> - elif not hash1 in hashfiles:
>> - recout.append("Unable to find matching sigdata for %s
>> with hash %s" % (key, hash1))
>> - elif not hash2 in hashfiles:
>> - recout.append("Unable to find matching sigdata for %s
>> with hash %s" % (key, hash2))
>> - else:
>> - out2 = bb.siggen.compare_sigfiles(hashfiles[hash1],
>> hashfiles[hash2], recursecb, color=color)
>> - for change in out2:
>> - for line in change.splitlines():
>> - recout.append(' ' + line)
>> + return latestfiles
>> +
>>
>> - return recout
>> +# Define recursion callback
>> +def recursecb(key, hash1, hash2):
>> + hashes = [hash1, hash2]
>> + hashfiles = find_siginfo(tinfoil, key, None, hashes)
>>
>> - # Recurse into signature comparison
>> - logger.debug("Signature file (previous): %s" % latestfiles[-2])
>> - logger.debug("Signature file (latest): %s" % latestfiles[-1])
>> - output = bb.siggen.compare_sigfiles(latestfiles[-2],
>> latestfiles[-1], recursecb, color=color)
>> - if output:
>> - print('\n'.join(output))
>> - sys.exit(0)
>> + recout = []
>> + if len(hashfiles) == 0:
>> + recout.append("Unable to find matching sigdata for %s with
>> hashes %s or %s" % (key, hash1, hash2))
>> + elif not hash1 in hashfiles:
>> + recout.append("Unable to find matching sigdata for %s with
>> hash %s" % (key, hash1))
>> + elif not hash2 in hashfiles:
>> + recout.append("Unable to find matching sigdata for %s with
>> hash %s" % (key, hash2))
>> + else:
>> + out2 = bb.siggen.compare_sigfiles(hashfiles[hash1],
>> hashfiles[hash2], recursecb, color=color)
>> + for change in out2:
>> + for line in change.splitlines():
>> + recout.append(' ' + line)
>>
>> + return recout
>>
>>
>> parser = argparse.ArgumentParser(
>> - description="Compares siginfo/sigdata files written out by
>> BitBake")
>> + description=("Dumps" if is_dump else "Compares") + "
>> siginfo/sigdata files written out by BitBake")
>> -parser.add_argument('-d', '--debug',
>> +parser.add_argument('-D', '--debug',
>> help='Enable debug output',
>> action='store_true')
>>
>> -parser.add_argument('--color',
>> - help='Colorize output (where %(metavar)s is %(choices)s)',
>> - choices=['auto', 'always', 'never'], default='auto',
>> metavar='color') +if is_dump:
>> + parser.add_argument("-t", "--task",
>> + help="find the signature data file for the last run of
>> the specified task",
>> + action="store", dest="taskargs", nargs=2,
>> metavar=('recipename', 'taskname')) +
>> + parser.add_argument("sigdatafile1",
>> + help="Signature file to dump. Not used when using
>> -t/--task.",
>> + action="store", nargs='?', metavar="sigdatafile")
>> +else:
>> + parser.add_argument('-c', '--color',
>> + help='Colorize the output (where %(metavar)s is
>> %(choices)s)',
>> + choices=['auto', 'always', 'never'], default='auto',
>> metavar='color')
>> -parser.add_argument("-t", "--task",
>> - help="find the signature data files for last two runs of the
>> specified task and compare them",
>> - action="store", dest="taskargs", nargs=2,
>> metavar=('recipename', 'taskname'))
>> + parser.add_argument('-d', '--dump',
>> + help='Dump the last signature data instead of comparing
>> (equivalent to using bitbake-dumpsig)',
>> + action='store_true')
>>
>> -parser.add_argument("-s", "--signature",
>> - help="With -t/--task, specify the signatures to look for
>> instead of taking the last two",
>> - action="store", dest="sigargs", nargs=2, metavar=('fromsig',
>> 'tosig'))
>> + parser.add_argument("-t", "--task",
>> + help="find the signature data files for the last two
>> runs of the specified task and compare them",
>> + action="store", dest="taskargs", nargs=2,
>> metavar=('recipename', 'taskname'))
>> -parser.add_argument("sigdatafile1",
>> - help="First signature file to compare (or signature file to
>> dump, if second not specified). Not used when using -t/--task.",
>> - action="store", nargs='?')
>> + parser.add_argument("-s", "--signature",
>> + help="With -t/--task, specify the signatures to look for
>> instead of taking the last two",
>> + action="store", dest="sigargs", nargs=2,
>> metavar=('fromsig', 'tosig'))
>> -parser.add_argument("sigdatafile2",
>> - help="Second signature file to compare",
>> - action="store", nargs='?')
>> + parser.add_argument("sigdatafile1",
>> + help="First signature file to compare (or signature file
>> to dump, if second not specified). Not used when using -t/--task.",
>> + action="store", nargs='?')
>>
>> + parser.add_argument("sigdatafile2",
>> + help="Second signature file to compare",
>> + action="store", nargs='?')
>>
>> options = parser.parse_args()
>> +if is_dump:
>> + options.color = 'never'
>> + options.dump = True
>> + options.sigdatafile2 = None
>> + options.sigargs = None
>>
>> if options.debug:
>> logger.setLevel(logging.DEBUG)
>> @@ -155,17 +155,32 @@ color = (options.color == 'always' or
>> (options.color == 'auto' and sys.stdout.is if options.taskargs:
>> with bb.tinfoil.Tinfoil() as tinfoil:
>> tinfoil.prepare(config_only=True)
>> - if options.sigargs:
>> - find_compare_task(tinfoil, options.taskargs[0],
>> options.taskargs[1], options.sigargs[0], options.sigargs[1],
>> color=color)
>> + if not options.dump and options.sigargs:
>> + files = find_siginfo_task(tinfoil, options.taskargs[0],
>> options.taskargs[1], options.sigargs[0], options.sigargs[1])
>> + else:
>> + files = find_siginfo_task(tinfoil, options.taskargs[0],
>> options.taskargs[1]) +
>> + if options.dump:
>> + logger.debug("Signature file: %s" % files[-1])
>> + output = bb.siggen.dump_sigfile(files[-1])
>> else:
>> - find_compare_task(tinfoil, options.taskargs[0],
>> options.taskargs[1], color=color)
>> + if len(files) < 2:
>> + logger.error('Only one matching sigdata file found
>> for the specified task (%s %s)' % (options.taskargs[0],
>> options.taskargs[1]))
>> + sys.exit(1)
>> +
>> + # Recurse into signature comparison
>> + logger.debug("Signature file (previous): %s" % files[-2])
>> + logger.debug("Signature file (latest): %s" % files[-1])
>> + output = bb.siggen.compare_sigfiles(files[-2],
>> files[-1], recursecb, color=color) else:
>> if options.sigargs:
>> logger.error('-s/--signature can only be used together with
>> -t/--task') sys.exit(1)
>> try:
>> - if options.sigdatafile1 and options.sigdatafile2:
>> - output =
>> bb.siggen.compare_sigfiles(options.sigdatafile1,
>> options.sigdatafile2, color=color)
>> + if not options.dump and options.sigdatafile1 and
>> options.sigdatafile2:
>> + with bb.tinfoil.Tinfoil() as tinfoil:
>> + tinfoil.prepare(config_only=True)
>> + output =
>> bb.siggen.compare_sigfiles(options.sigdatafile1,
>> options.sigdatafile2, recursecb, color=color) elif
>> options.sigdatafile1: output =
>> bb.siggen.dump_sigfile(options.sigdatafile1) else: @@ -179,5 +194,5
>> @@ else: logger.error('Invalid signature data - ensure you are
>> specifying sigdata/siginfo files') sys.exit(1)
>>
>> - if output:
>> - print('\n'.join(output))
>> +if output:
>> + print('\n'.join(output))
>> diff --git a/bitbake/bin/bitbake-dumpsig b/bitbake/bin/bitbake-dumpsig
>> deleted file mode 100755
>> index 95ebd93..0000000
>> --- a/bitbake/bin/bitbake-dumpsig
>> +++ /dev/null
>> @@ -1,94 +0,0 @@
>> -#!/usr/bin/env python3
>> -
>> -# bitbake-dumpsig
>> -# BitBake task signature dump utility
>> -#
>> -# Copyright (C) 2013 Intel Corporation
>> -#
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. -
>> -import os
>> -import sys
>> -import warnings
>> -import optparse
>> -import logging
>> -import pickle
>> -
>> -sys.path.insert(0,
>> os.path.join(os.path.dirname(os.path.dirname(sys.argv[0])), 'lib')) -
>> -import bb.tinfoil
>> -import bb.siggen
>> -import bb.msg
>> -
>> -logger = bb.msg.logger_create('bitbake-dumpsig')
>> -
>> -def find_siginfo_task(bbhandler, pn, taskname):
>> - """ Find the most recent signature file for the specified
>> PN/task """ -
>> - if not hasattr(bb.siggen, 'find_siginfo'):
>> - logger.error('Metadata does not support finding signature
>> data files')
>> - sys.exit(1)
>> -
>> - if not taskname.startswith('do_'):
>> - taskname = 'do_%s' % taskname
>> -
>> - filedates = bb.siggen.find_siginfo(pn, taskname, None,
>> bbhandler.config_data)
>> - latestfiles = sorted(filedates.keys(), key=lambda f:
>> filedates[f])[-1:]
>> - if not latestfiles:
>> - logger.error('No sigdata files found matching %s %s' % (pn,
>> taskname))
>> - sys.exit(1)
>> -
>> - return latestfiles[0]
>> -
>> -parser = optparse.OptionParser(
>> - description = "Dumps siginfo/sigdata files written out by
>> BitBake",
>> - usage = """
>> - %prog -t recipename taskname
>> - %prog sigdatafile""")
>> -
>> -parser.add_option("-D", "--debug",
>> - help = "enable debug",
>> - action = "store_true", dest="debug", default = False)
>> -
>> -parser.add_option("-t", "--task",
>> - help = "find the signature data file for the specified task",
>> - action="store", dest="taskargs", nargs=2,
>> metavar='recipename taskname') -
>> -options, args = parser.parse_args(sys.argv)
>> -
>> -if options.debug:
>> - logger.setLevel(logging.DEBUG)
>> -
>> -if options.taskargs:
>> - tinfoil = bb.tinfoil.Tinfoil()
>> - tinfoil.prepare(config_only = True)
>> - file = find_siginfo_task(tinfoil, options.taskargs[0],
>> options.taskargs[1])
>> - logger.debug("Signature file: %s" % file)
>> -elif len(args) == 1:
>> - parser.print_help()
>> - sys.exit(0)
>> -else:
>> - file = args[1]
>> -
>> -try:
>> - output = bb.siggen.dump_sigfile(file)
>> -except IOError as e:
>> - logger.error(str(e))
>> - sys.exit(1)
>> -except (pickle.UnpicklingError, EOFError):
>> - logger.error('Invalid signature data - ensure you are specifying
>> a sigdata/siginfo file')
>> - sys.exit(1)
>> -
>> -if output:
>> - print('\n'.join(output))
>> diff --git a/bitbake/bin/bitbake-dumpsig b/bitbake/bin/bitbake-dumpsig
>> new file mode 120000
>> index 0000000..b1e8489
>> --- /dev/null
>> +++ b/bitbake/bin/bitbake-dumpsig
>> @@ -0,0 +1 @@
>> +bitbake-diffsigs
>> \ No newline at end of file
>> diff --git a/bitbake/bin/bitbake-hashclient
>> b/bitbake/bin/bitbake-hashclient new file mode 100755
>> index 0000000..29ab65f
>> --- /dev/null
>> +++ b/bitbake/bin/bitbake-hashclient
>> @@ -0,0 +1,170 @@
>> +#! /usr/bin/env python3
>> +#
>> +# Copyright (C) 2019 Garmin Ltd.
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> +import argparse
>> +import hashlib
>> +import logging
>> +import os
>> +import pprint
>> +import sys
>> +import threading
>> +import time
>> +
>> +try:
>> + import tqdm
>> + ProgressBar = tqdm.tqdm
>> +except ImportError:
>> + class ProgressBar(object):
>> + def __init__(self, *args, **kwargs):
>> + pass
>> +
>> + def __enter__(self):
>> + return self
>> +
>> + def __exit__(self, *args, **kwargs):
>> + pass
>> +
>> + def update(self):
>> + pass
>> +
>> +sys.path.insert(0,
>> os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib')) +
>> +import hashserv
>> +
>> +DEFAULT_ADDRESS = 'unix://./hashserve.sock'
>> +METHOD = 'stress.test.method'
>> +
>> +
>> +def main():
>> + def handle_stats(args, client):
>> + if args.reset:
>> + s = client.reset_stats()
>> + else:
>> + s = client.get_stats()
>> + pprint.pprint(s)
>> + return 0
>> +
>> + def handle_stress(args, client):
>> + def thread_main(pbar, lock):
>> + nonlocal found_hashes
>> + nonlocal missed_hashes
>> + nonlocal max_time
>> +
>> + client = hashserv.create_client(args.address)
>> +
>> + for i in range(args.requests):
>> + taskhash = hashlib.sha256()
>> + taskhash.update(args.taskhash_seed.encode('utf-8'))
>> + taskhash.update(str(i).encode('utf-8'))
>> +
>> + start_time = time.perf_counter()
>> + l = client.get_unihash(METHOD, taskhash.hexdigest())
>> + elapsed = time.perf_counter() - start_time
>> +
>> + with lock:
>> + if l:
>> + found_hashes += 1
>> + else:
>> + missed_hashes += 1
>> +
>> + max_time = max(elapsed, max_time)
>> + pbar.update()
>> +
>> + max_time = 0
>> + found_hashes = 0
>> + missed_hashes = 0
>> + lock = threading.Lock()
>> + total_requests = args.clients * args.requests
>> + start_time = time.perf_counter()
>> + with ProgressBar(total=total_requests) as pbar:
>> + threads = [threading.Thread(target=thread_main,
>> args=(pbar, lock), daemon=False) for _ in range(args.clients)]
>> + for t in threads:
>> + t.start()
>> +
>> + for t in threads:
>> + t.join()
>> +
>> + elapsed = time.perf_counter() - start_time
>> + with lock:
>> + print("%d requests in %.1fs. %.1f requests per second" %
>> (total_requests, elapsed, total_requests / elapsed))
>> + print("Average request time %.8fs" % (elapsed /
>> total_requests))
>> + print("Max request time was %.8fs" % max_time)
>> + print("Found %d hashes, missed %d" % (found_hashes,
>> missed_hashes)) +
>> + if args.report:
>> + with ProgressBar(total=args.requests) as pbar:
>> + for i in range(args.requests):
>> + taskhash = hashlib.sha256()
>> +
>> taskhash.update(args.taskhash_seed.encode('utf-8'))
>> + taskhash.update(str(i).encode('utf-8'))
>> +
>> + outhash = hashlib.sha256()
>> + outhash.update(args.outhash_seed.encode('utf-8'))
>> + outhash.update(str(i).encode('utf-8'))
>> +
>> + client.report_unihash(taskhash.hexdigest(),
>> METHOD, outhash.hexdigest(), taskhash.hexdigest()) +
>> + with lock:
>> + pbar.update()
>> +
>> + parser = argparse.ArgumentParser(description='Hash Equivalence
>> Client')
>> + parser.add_argument('--address', default=DEFAULT_ADDRESS,
>> help='Server address (default "%(default)s")')
>> + parser.add_argument('--log', default='WARNING', help='Set
>> logging level') +
>> + subparsers = parser.add_subparsers()
>> +
>> + stats_parser = subparsers.add_parser('stats', help='Show server
>> stats')
>> + stats_parser.add_argument('--reset', action='store_true',
>> + help='Reset server stats')
>> + stats_parser.set_defaults(func=handle_stats)
>> +
>> + stress_parser = subparsers.add_parser('stress', help='Run stress
>> test')
>> + stress_parser.add_argument('--clients', type=int, default=10,
>> + help='Number of simultaneous clients')
>> + stress_parser.add_argument('--requests', type=int, default=1000,
>> + help='Number of requests each client
>> will perform')
>> + stress_parser.add_argument('--report', action='store_true',
>> + help='Report new hashes')
>> + stress_parser.add_argument('--taskhash-seed', default='',
>> + help='Include string in taskhash')
>> + stress_parser.add_argument('--outhash-seed', default='',
>> + help='Include string in outhash')
>> + stress_parser.set_defaults(func=handle_stress)
>> +
>> + args = parser.parse_args()
>> +
>> + logger = logging.getLogger('hashserv')
>> +
>> + level = getattr(logging, args.log.upper(), None)
>> + if not isinstance(level, int):
>> + raise ValueError('Invalid log level: %s' % args.log)
>> +
>> + logger.setLevel(level)
>> + console = logging.StreamHandler()
>> + console.setLevel(level)
>> + logger.addHandler(console)
>> +
>> + func = getattr(args, 'func', None)
>> + if func:
>> + client = hashserv.create_client(args.address)
>> + # Try to establish a connection to the server now to detect
>> failures
>> + # early
>> + client.connect()
>> +
>> + return func(args, client)
>> +
>> + return 0
>> +
>> +
>> +if __name__ == '__main__':
>> + try:
>> + ret = main()
>> + except Exception:
>> + ret = 1
>> + import traceback
>> + traceback.print_exc()
>> + sys.exit(ret)
>> diff --git a/bitbake/bin/bitbake-hashserv
>> b/bitbake/bin/bitbake-hashserv new file mode 100755
>> index 0000000..1bc1f91
>> --- /dev/null
>> +++ b/bitbake/bin/bitbake-hashserv
>> @@ -0,0 +1,62 @@
>> +#! /usr/bin/env python3
>> +#
>> +# Copyright (C) 2018 Garmin Ltd.
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> +import os
>> +import sys
>> +import logging
>> +import argparse
>> +import sqlite3
>> +
>> +sys.path.insert(0,
>> os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib')) +
>> +import hashserv
>> +
>> +VERSION = "1.0.0"
>> +
>> +DEFAULT_BIND = 'unix://./hashserve.sock'
>> +
>> +
>> +def main():
>> + parser = argparse.ArgumentParser(description='Hash Equivalence
>> Reference Server. Version=%s' % VERSION,
>> + epilog='''The bind address is
>> the path to a unix domain socket if it is
>> + prefixed with
>> "unix://". Otherwise, it is an IP address
>> + and port in form
>> ADDRESS:PORT. To bind to all addresses, leave
>> + the ADDRESS empty,
>> e.g. "--bind :8686". To bind to a specific
>> + IPv6 address, enclose
>> the address in "[]", e.g.
>> + "--bind [::1]:8686"'''
>> + )
>> +
>> + parser.add_argument('--bind', default=DEFAULT_BIND, help='Bind
>> address (default "%(default)s")')
>> + parser.add_argument('--database', default='./hashserv.db',
>> help='Database file (default "%(default)s")')
>> + parser.add_argument('--log', default='WARNING', help='Set
>> logging level') +
>> + args = parser.parse_args()
>> +
>> + logger = logging.getLogger('hashserv')
>> +
>> + level = getattr(logging, args.log.upper(), None)
>> + if not isinstance(level, int):
>> + raise ValueError('Invalid log level: %s' % args.log)
>> +
>> + logger.setLevel(level)
>> + console = logging.StreamHandler()
>> + console.setLevel(level)
>> + logger.addHandler(console)
>> +
>> + server = hashserv.create_server(args.bind, args.database)
>> + server.serve_forever()
>> + return 0
>> +
>> +
>> +if __name__ == '__main__':
>> + try:
>> + ret = main()
>> + except Exception:
>> + ret = 1
>> + import traceback
>> + traceback.print_exc()
>> + sys.exit(ret)
>> diff --git a/bitbake/bin/bitbake-layers b/bitbake/bin/bitbake-layers
>> index d184011..a884dc1 100755
>> --- a/bitbake/bin/bitbake-layers
>> +++ b/bitbake/bin/bitbake-layers
>> @@ -7,18 +7,8 @@
>> # Copyright (C) 2011 Mentor Graphics Corporation
>> # Copyright (C) 2011-2015 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import logging
>> import os
>> diff --git a/bitbake/bin/bitbake-prserv b/bitbake/bin/bitbake-prserv
>> index f38d2dd..1e9b6cb 100755
>> --- a/bitbake/bin/bitbake-prserv
>> +++ b/bitbake/bin/bitbake-prserv
>> @@ -1,4 +1,8 @@
>> #!/usr/bin/env python3
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> import os
>> import sys,logging
>> import optparse
>> diff --git a/bitbake/bin/bitbake-selftest
>> b/bitbake/bin/bitbake-selftest index cfa7ac5..041a271 100755
>> --- a/bitbake/bin/bitbake-selftest
>> +++ b/bitbake/bin/bitbake-selftest
>> @@ -2,18 +2,8 @@
>> #
>> # Copyright (C) 2012 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import os
>> import sys, logging
>> @@ -22,6 +12,7 @@ sys.path.insert(0,
>> os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib import
>> unittest try:
>> import bb
>> + import hashserv
>> import layerindexlib
>> except RuntimeError as exc:
>> sys.exit(str(exc))
>> @@ -33,7 +24,10 @@ tests = ["bb.tests.codeparser",
>> "bb.tests.event",
>> "bb.tests.fetch",
>> "bb.tests.parse",
>> + "bb.tests.persist_data",
>> + "bb.tests.runqueue",
>> "bb.tests.utils",
>> + "hashserv.tests",
>> "layerindexlib.tests.layerindexobj",
>> "layerindexlib.tests.restapi",
>> "layerindexlib.tests.cooker"]
>> diff --git a/bitbake/bin/bitbake-worker b/bitbake/bin/bitbake-worker
>> index e925054..6776cad 100755
>> --- a/bitbake/bin/bitbake-worker
>> +++ b/bitbake/bin/bitbake-worker
>> @@ -1,4 +1,7 @@
>> #!/usr/bin/env python3
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>>
>> import os
>> import sys
>> @@ -136,7 +139,7 @@ def sigterm_handler(signum, frame):
>> os.killpg(0, signal.SIGTERM)
>> sys.exit()
>>
>> -def fork_off_task(cfg, data, databuilder, workerdata, fn, task,
>> taskname, appends, taskdepdata, extraconfigdata, quieterrors=False,
>> dry_run_exec=False): +def fork_off_task(cfg, data, databuilder,
>> workerdata, fn, task, taskname, taskhash, unihash, appends,
>> taskdepdata, extraconfigdata, quieterrors=False, dry_run_exec=False):
>> # We need to setup the environment BEFORE the fork, since # a fork()
>> or exec*() activates PSEUDO... @@ -231,10 +234,13 @@ def
>> fork_off_task(cfg, data, databuilder, workerdata, fn, task, taskname,
>> append the_data.setVar(varname, value)
>> bb.parse.siggen.set_taskdata(workerdata["sigdata"])
>> + if "newhashes" in workerdata:
>> +
>> bb.parse.siggen.set_taskhashes(workerdata["newhashes"]) ret = 0
>>
>> the_data = bb_cache.loadDataFull(fn, appends)
>> - the_data.setVar('BB_TASKHASH',
>> workerdata["runq_hash"][task])
>> + the_data.setVar('BB_TASKHASH', taskhash)
>> + the_data.setVar('BB_UNIHASH', unihash)
>>
>> bb.utils.set_process_name("%s:%s" %
>> (the_data.getVar("PN"), taskname.replace("do_", "")))
>> @@ -373,6 +379,7 @@ class BitbakeWorker(object):
>> self.handle_item(b"cookerconfig",
>> self.handle_cookercfg) self.handle_item(b"extraconfigdata",
>> self.handle_extraconfigdata) self.handle_item(b"workerdata",
>> self.handle_workerdata)
>> + self.handle_item(b"newtaskhashes",
>> self.handle_newtaskhashes) self.handle_item(b"runtask",
>> self.handle_runtask) self.handle_item(b"finishnow",
>> self.handle_finishnow) self.handle_item(b"ping", self.handle_ping)
>> @@ -411,6 +418,10 @@ class BitbakeWorker(object):
>> bb.msg.loggerDefaultDomains =
>> self.workerdata["logdefaultdomain"] for mc in self.databuilder.mcdata:
>> self.databuilder.mcdata[mc].setVar("PRSERV_HOST",
>> self.workerdata["prhost"])
>> + self.databuilder.mcdata[mc].setVar("BB_HASHSERVE",
>> self.workerdata["hashservaddr"]) +
>> + def handle_newtaskhashes(self, data):
>> + self.workerdata["newhashes"] = pickle.loads(data)
>>
>> def handle_ping(self, _):
>> workerlog_write("Handling ping\n")
>> @@ -425,10 +436,10 @@ class BitbakeWorker(object):
>> sys.exit(0)
>>
>> def handle_runtask(self, data):
>> - fn, task, taskname, quieterrors, appends, taskdepdata,
>> dry_run_exec = pickle.loads(data)
>> + fn, task, taskname, taskhash, unihash, quieterrors, appends,
>> taskdepdata, dry_run_exec = pickle.loads(data)
>> workerlog_write("Handling runtask %s %s %s\n" % (task, fn, taskname))
>> - pid, pipein, pipeout = fork_off_task(self.cookercfg,
>> self.data, self.databuilder, self.workerdata, fn, task, taskname,
>> appends, taskdepdata, self.extraconfigdata, quieterrors, dry_run_exec)
>> + pid, pipein, pipeout = fork_off_task(self.cookercfg,
>> self.data, self.databuilder, self.workerdata, fn, task, taskname,
>> taskhash, unihash, appends, taskdepdata, self.extraconfigdata,
>> quieterrors, dry_run_exec) self.build_pids[pid] = task
>> self.build_pipes[pid] = runQueueWorkerPipe(pipein, pipeout) diff
>> --git a/bitbake/bin/bitdoc b/bitbake/bin/bitdoc index
>> 2744678..9bd02be 100755 --- a/bitbake/bin/bitdoc
>> +++ b/bitbake/bin/bitdoc
>> @@ -1,21 +1,9 @@
>> #!/usr/bin/env python3
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # Copyright (C) 2005 Holger Hans Peter Freyther
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import optparse, os, sys
>>
>> diff --git a/bitbake/bin/git-make-shallow
>> b/bitbake/bin/git-make-shallow index 296d3a3..57069f7 100755
>> --- a/bitbake/bin/git-make-shallow
>> +++ b/bitbake/bin/git-make-shallow
>> @@ -1,4 +1,8 @@
>> #!/usr/bin/env python3
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> """git-make-shallow: make the current git repository shallow
>>
>> Remove the history of the specified revisions, then optionally
>> filter the diff --git a/bitbake/bin/toaster b/bitbake/bin/toaster
>> index ecf66fa..c3472df 100755
>> --- a/bitbake/bin/toaster
>> +++ b/bitbake/bin/toaster
>> @@ -3,19 +3,9 @@
>> # toaster - shell script to start Toaster
>>
>> # Copyright (C) 2013-2015 Intel Corp.
>> -
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License as
>> published by -# the Free Software Foundation; either version 2 of the
>> License, or -# (at your option) any later version.
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# SPDX-License-Identifier: GPL-2.0-or-later
>> #
>> -# You should have received a copy of the GNU General Public License
>> -# along with this program. If not, see http://www.gnu.org/licenses/.
>>
>> HELP="
>> Usage: source toaster start|stop [webport=<address:port>] [noweb]
>> [nobuild] [toasterdir] diff --git a/bitbake/bin/toaster-eventreplay
>> b/bitbake/bin/toaster-eventreplay index 80967a0..8fa4ab7 100755
>> --- a/bitbake/bin/toaster-eventreplay
>> +++ b/bitbake/bin/toaster-eventreplay
>> @@ -1,25 +1,12 @@
>> #!/usr/bin/env python3
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # Copyright (C) 2014 Alex Damian
>> #
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> # This file re-uses code spread throughout other Bitbake source
>> files. # As such, all other copyrights belong to their own right
>> holders. #
>> -#
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> """
>> This command takes a filename as a single parameter. The filename is
>> read diff --git a/bitbake/classes/base.bbclass
>> b/bitbake/classes/base.bbclass index 71d9ec1..08441fe 100644
>> --- a/bitbake/classes/base.bbclass
>> +++ b/bitbake/classes/base.bbclass
>> @@ -44,7 +44,7 @@ python do_showdata() {
>> # emit the metadata which isnt valid shell
>> for e in bb.data.keys(d):
>> if d.getVarFlag(e, 'python', False):
>> - bb.plain("\npython %s () {\n%s}" % (e,
>> d.getVar(e, True)))
>> + bb.plain("\npython %s () {\n%s}" % (e,
>> d.getVar(e))) }
>>
>> addtask listtasks
>> diff --git a/bitbake/contrib/dump_cache.py
>> b/bitbake/contrib/dump_cache.py index 8963ca4..c6723cb 100755
>> --- a/bitbake/contrib/dump_cache.py
>> +++ b/bitbake/contrib/dump_cache.py
>> @@ -1,6 +1,4 @@
>> #!/usr/bin/env python3
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # Copyright (C) 2012, 2018 Wind River Systems, Inc.
>> #
>> diff --git
>> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
>> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
>> index f1caaec..46dafee 100644 ---
>> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
>> +++
>> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
>> @@ -31,7 +31,7 @@ <para> Prior to executing BitBake, you should take
>> advantage of available parallel thread execution on your build host
>> by setting the
>> - <link
>> linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
>> + <link
>> linkend='var-bb-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
>> variable in your project's <filename>local.conf</filename>
>> configuration file. </para>
>> @@ -87,9 +87,9 @@
>> <para>
>> The <filename>layer.conf</filename> files are used to
>> construct key variables such as
>> - <link
>> linkend='var-BBPATH'><filename>BBPATH</filename></link>
>> + <link
>> linkend='var-bb-BBPATH'><filename>BBPATH</filename></link> and
>> - <link
>> linkend='var-BBFILES'><filename>BBFILES</filename></link>.
>> + <link
>> linkend='var-bb-BBFILES'><filename>BBFILES</filename></link>.
>> <filename>BBPATH</filename> is used to search for configuration and
>> class files under the <filename>conf</filename> and
>> <filename>classes</filename> @@ -117,19 +117,19 @@
>> at certain variables, including:
>> <itemizedlist>
>> <listitem><para>
>> - <link
>> linkend='var-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>
>> + <link
>> linkend='var-bb-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>
>> </para></listitem> <listitem><para>
>> - <link
>> linkend='var-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link>
>> + <link
>> linkend='var-bb-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link>
>> </para></listitem> <listitem><para>
>> - <link
>> linkend='var-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link>
>> + <link
>> linkend='var-bb-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link>
>> </para></listitem> <listitem><para>
>> - <link
>> linkend='var-BB_ORIGENV'><filename>BB_ORIGENV</filename></link>
>> + <link
>> linkend='var-bb-BB_ORIGENV'><filename>BB_ORIGENV</filename></link>
>> </para></listitem> <listitem><para>
>> - <link
>> linkend='var-BITBAKE_UI'><filename>BITBAKE_UI</filename></link>
>> + <link
>> linkend='var-bb-BITBAKE_UI'><filename>BITBAKE_UI</filename></link>
>> </para></listitem> </itemizedlist>
>> The first four variables in this list relate to how
>> BitBake treats shell @@ -156,7 +156,7 @@
>> BitBake first searches the current working directory for
>> an optional <filename>conf/bblayers.conf</filename> configuration
>> file. This file is expected to contain a
>> - <link
>> linkend='var-BBLAYERS'><filename>BBLAYERS</filename></link>
>> + <link
>> linkend='var-bb-BBLAYERS'><filename>BBLAYERS</filename></link>
>> variable that is a space-delimited list of 'layer' directories.
>> Recall that if BitBake cannot find a
>> <filename>bblayers.conf</filename> file, then it is assumed the user
>> has set the <filename>BBPATH</filename> @@ -166,10 +166,10 @@ <para>
>> For each directory (layer) in this list, a
>> <filename>conf/layer.conf</filename> file is located and parsed with
>> the
>> - <link
>> linkend='var-LAYERDIR'><filename>LAYERDIR</filename></link>
>> + <link
>> linkend='var-bb-LAYERDIR'><filename>LAYERDIR</filename></link>
>> variable being set to the directory where the layer was found. The
>> idea is these files automatically set up
>> - <link
>> linkend='var-BBPATH'><filename>BBPATH</filename></link>
>> + <link
>> linkend='var-bb-BBPATH'><filename>BBPATH</filename></link> and other
>> variables correctly for a given build directory. </para>
>>
>> @@ -189,7 +189,7 @@
>> depending on the environment variables previously
>> mentioned or set in the configuration files.
>> The
>> - "<link linkend='ref-variables-glos'>Variables
>> Glossary</link>"
>> + "<link linkend='ref-bb-variables-glos'>Variables
>> Glossary</link>" chapter presents a full list of variables.
>> </para>
>>
>> @@ -204,7 +204,7 @@
>> <para>
>> The <filename>base.bbclass</filename> file is always
>> included. Other classes that are specified in the configuration using
>> the
>> - <link
>> linkend='var-INHERIT'><filename>INHERIT</filename></link>
>> + <link
>> linkend='var-bb-INHERIT'><filename>INHERIT</filename></link> variable
>> are also included. BitBake searches for class files in a
>> <filename>classes</filename> subdirectory under
>> @@ -270,7 +270,7 @@
>>
>> <para>
>> During the configuration phase, BitBake will have set
>> - <link
>> linkend='var-BBFILES'><filename>BBFILES</filename></link>.
>> + <link
>> linkend='var-bb-BBFILES'><filename>BBFILES</filename></link>. BitBake
>> now uses it to construct a list of recipes to parse, along with any
>> append files (<filename>.bbappend</filename>) to apply.
>> @@ -292,7 +292,7 @@
>> Any inherit statements cause BitBake to find and
>> then parse class files (<filename>.bbclass</filename>)
>> using
>> - <link
>> linkend='var-BBPATH'><filename>BBPATH</filename></link>
>> + <link
>> linkend='var-bb-BBPATH'><filename>BBPATH</filename></link> as the
>> search path. Finally, BitBake parses in order any append files found
>> in <filename>BBFILES</filename>.
>> @@ -303,8 +303,8 @@
>> pieces of metadata.
>> For example, in <filename>bitbake.conf</filename> the
>> recipe name and version are used to set the variables
>> - <link linkend='var-PN'><filename>PN</filename></link> and
>> - <link linkend='var-PV'><filename>PV</filename></link>:
>> + <link linkend='var-bb-PN'><filename>PN</filename></link>
>> and
>> + <link linkend='var-bb-PV'><filename>PV</filename></link>:
>> <literallayout class='monospaced'>
>> PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE',
>> False),d)[0] or 'defaultpkgname'}" PV =
>> "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[1]
>> or '1.0'}" @@ -336,7 +336,7 @@ recipe information.
>> The validity of this cache is determined by first
>> computing a checksum of the base configuration data (see
>> - <link
>> linkend='var-BB_HASHCONFIG_WHITELIST'><filename>BB_HASHCONFIG_WHITELIST</filename></link>)
>> + <link
>> linkend='var-bb-BB_HASHCONFIG_WHITELIST'><filename>BB_HASHCONFIG_WHITELIST</filename></link>)
>> and then checking if the checksum matches. If that checksum matches
>> what is in the cache and the recipe and class files have not changed,
>> Bitbake is able to use @@ -384,9 +384,9 @@
>> the recipe can be known.
>> Each recipe's <filename>PROVIDES</filename> list is
>> created implicitly through the recipe's
>> - <link linkend='var-PN'><filename>PN</filename></link>
>> variable
>> + <link linkend='var-bb-PN'><filename>PN</filename></link>
>> variable and explicitly through the recipe's
>> - <link
>> linkend='var-PROVIDES'><filename>PROVIDES</filename></link>
>> + <link
>> linkend='var-bb-PROVIDES'><filename>PROVIDES</filename></link>
>> variable, which is optional. </para>
>>
>> @@ -427,7 +427,7 @@
>> PREFERRED_PROVIDER_virtual/kernel = "linux-yocto"
>> </literallayout>
>> The default
>> - <link
>> linkend='var-PREFERRED_PROVIDER'><filename>PREFERRED_PROVIDER</filename></link>
>> + <link
>> linkend='var-bb-PREFERRED_PROVIDER'><filename>PREFERRED_PROVIDER</filename></link>
>> is the provider with the same name as the target. Bitbake iterates
>> through each target it needs to build and resolves them and their
>> dependencies using this process. @@ -439,10 +439,10 @@
>> BitBake defaults to the highest version of a provider.
>> Version comparisons are made using the same method as
>> Debian. You can use the
>> - <link
>> linkend='var-PREFERRED_VERSION'><filename>PREFERRED_VERSION</filename></link>
>> + <link
>> linkend='var-bb-PREFERRED_VERSION'><filename>PREFERRED_VERSION</filename></link>
>> variable to specify a particular version. You can influence the order
>> by using the
>> - <link
>> linkend='var-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link>
>> + <link
>> linkend='var-bb-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link>
>> variable. </para>
>>
>> @@ -464,7 +464,7 @@
>> BitBake defaults to selecting the most recent
>> version, unless otherwise specified.
>> If the recipe in question has a
>> - <link
>> linkend='var-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link>
>> + <link
>> linkend='var-bb-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link>
>> set lower than the other recipes (default is 0), then it will not be
>> selected. This allows the person or persons maintaining
>> @@ -475,9 +475,9 @@
>>
>> <para>
>> If the first recipe is named
>> <filename>a_1.1.bb</filename>, then the
>> - <link linkend='var-PN'><filename>PN</filename></link>
>> variable
>> + <link linkend='var-bb-PN'><filename>PN</filename></link>
>> variable will be set to “a”, and the
>> - <link linkend='var-PV'><filename>PV</filename></link>
>> + <link linkend='var-bb-PV'><filename>PV</filename></link>
>> variable will be set to 1.1.
>> </para>
>>
>> @@ -532,11 +532,11 @@
>> <para>
>> Dependencies are defined through several variables.
>> You can find information about variables BitBake uses in
>> - the <link linkend='ref-variables-glos'>Variables
>> Glossary</link>
>> + the <link linkend='ref-bb-variables-glos'>Variables
>> Glossary</link> near the end of this manual.
>> At a basic level, it is sufficient to know that BitBake
>> uses the
>> - <link
>> linkend='var-DEPENDS'><filename>DEPENDS</filename></link> and
>> - <link
>> linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link> variables
>> when
>> + <link
>> linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link> and
>> + <link
>> linkend='var-bb-RDEPENDS'><filename>RDEPENDS</filename></link>
>> variables when calculating dependencies. </para>
>>
>> @@ -560,7 +560,7 @@
>>
>> <para>
>> The build now starts with BitBake forking off threads up
>> to the limit set in the
>> - <link
>> linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
>> + <link
>> linkend='var-bb-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
>> variable. BitBake continues to fork threads as long as there are
>> tasks ready to run, those tasks have all their dependencies met, and
>> the thread threshold has not been @@ -574,7 +574,7 @@
>>
>> <para>
>> As each task completes, a timestamp is written to the
>> directory specified by the
>> - <link
>> linkend='var-STAMP'><filename>STAMP</filename></link> variable.
>> + <link
>> linkend='var-bb-STAMP'><filename>STAMP</filename></link> variable. On
>> subsequent runs, BitBake looks in the build directory within
>> <filename>tmp/stamps</filename> and does not rerun tasks that are
>> already completed unless a timestamp is found to be invalid. @@
>> -618,7 +618,7 @@ <para>
>> Tasks can be either a shell task or a Python task.
>> For shell tasks, BitBake writes a shell script to
>> - <filename>${</filename><link
>> linkend='var-T'><filename>T</filename></link><filename>}/run.do_taskname.pid</filename>
>> + <filename>${</filename><link
>> linkend='var-bb-T'><filename>T</filename></link><filename>}/run.do_taskname.pid</filename>
>> and then executes the script. The generated shell script contains all
>> the exported variables, and the shell functions with all variables
>> expanded. @@ -645,10 +645,10 @@
>> behavior:
>> <itemizedlist>
>> <listitem><para>
>> - <link
>> linkend='var-BB_SCHEDULER'><filename>BB_SCHEDULER</filename></link>
>> + <link
>> linkend='var-bb-BB_SCHEDULER'><filename>BB_SCHEDULER</filename></link>
>> </para></listitem> <listitem><para>
>> - <link
>> linkend='var-BB_SCHEDULERS'><filename>BB_SCHEDULERS</filename></link>
>> + <link
>> linkend='var-bb-BB_SCHEDULERS'><filename>BB_SCHEDULERS</filename></link>
>> </para></listitem> </itemizedlist>
>> It is possible to have functions run before and after a
>> task's main @@ -684,7 +684,7 @@
>> The simplistic approach for excluding the working
>> directory is to set it to some fixed value and create the checksum
>> for the "run" script. BitBake goes one step better and uses the
>> - <link
>> linkend='var-BB_HASHBASE_WHITELIST'><filename>BB_HASHBASE_WHITELIST</filename></link>
>> + <link
>> linkend='var-bb-BB_HASHBASE_WHITELIST'><filename>BB_HASHBASE_WHITELIST</filename></link>
>> variable to define a list of variables that should never be included
>> when generating the signatures. </para>
>> @@ -795,7 +795,7 @@
>> This results in any metadata change that changes the
>> task hash, automatically causing the task to be run again.
>> This removes the need to bump
>> - <link linkend='var-PR'><filename>PR</filename></link>
>> + <link linkend='var-bb-PR'><filename>PR</filename></link>
>> values, and changes to metadata automatically ripple
>> across the build. </para>
>>
>> @@ -884,7 +884,7 @@
>>
>> <para>
>> BitBake first calls the function defined by the
>> - <link
>> linkend='var-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></link>
>> + <link
>> linkend='var-bb-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></link>
>> variable with a list of tasks and corresponding hashes it wants to
>> build. This function is designed to be fast and returns a list
>> @@ -908,7 +908,7 @@
>> For example, it is pointless to obtain a compiler if you
>> already have the compiled binary.
>> To handle this, BitBake calls the
>> - <link
>> linkend='var-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></link>
>> + <link
>> linkend='var-bb-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></link>
>> function for each successful setscene task to know whether or not it
>> needs to obtain the dependencies of that task. </para>
>> @@ -916,7 +916,7 @@
>> <para>
>> Finally, after all the setscene tasks have executed,
>> BitBake calls the function listed in
>> - <link
>> linkend='var-BB_SETSCENE_VERIFY_FUNCTION2'><filename>BB_SETSCENE_VERIFY_FUNCTION2</filename></link>
>> + <link
>> linkend='var-bb-BB_SETSCENE_VERIFY_FUNCTION2'><filename>BB_SETSCENE_VERIFY_FUNCTION2</filename></link>
>> with the list of tasks BitBake thinks has been "covered". The
>> metadata can then ensure that this list is correct and can inform
>> BitBake that it wants specific tasks to be run regardless diff --git
>> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml
>> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml
>> index 29ae486..6840408 100644 ---
>> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml
>> +++
>> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml @@
>> -44,7 +44,7 @@ </literallayout> This code sets up an instance of the
>> fetch class. The instance uses a space-separated list of URLs from the
>> - <link
>> linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
>> + <link
>> linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link> variable
>> and then calls the <filename>download</filename> method to download
>> the files. </para>
>> @@ -78,7 +78,7 @@
>> <listitem><para><emphasis>Pre-mirror
>> Sites:</emphasis> BitBake first uses pre-mirrors to try and find
>> source files. These locations are defined using the
>> - <link
>> linkend='var-PREMIRRORS'><filename>PREMIRRORS</filename></link>
>> + <link
>> linkend='var-bb-PREMIRRORS'><filename>PREMIRRORS</filename></link>
>> variable. </para></listitem>
>> <listitem><para><emphasis>Source URI:</emphasis>
>> @@ -88,7 +88,7 @@
>> <listitem><para><emphasis>Mirror Sites:</emphasis>
>> If fetch failures occur, BitBake next uses
>> mirror locations as defined by the
>> - <link
>> linkend='var-MIRRORS'><filename>MIRRORS</filename></link>
>> + <link
>> linkend='var-bb-MIRRORS'><filename>MIRRORS</filename></link> variable.
>> </para></listitem>
>> </itemizedlist>
>> @@ -144,7 +144,7 @@
>> Any source files that are not local (i.e.
>> downloaded from the Internet) are placed into the
>> download directory, which is specified by the
>> - <link
>> linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
>> + <link
>> linkend='var-bb-DL_DIR'><filename>DL_DIR</filename></link> variable.
>> </para>
>>
>> @@ -184,11 +184,11 @@
>>
>> <para>
>> If
>> - <link
>> linkend='var-BB_STRICT_CHECKSUM'><filename>BB_STRICT_CHECKSUM</filename></link>
>> + <link
>> linkend='var-bb-BB_STRICT_CHECKSUM'><filename>BB_STRICT_CHECKSUM</filename></link>
>> is set, any download without a checksum triggers an error message.
>> The
>> - <link
>> linkend='var-BB_NO_NETWORK'><filename>BB_NO_NETWORK</filename></link>
>> + <link
>> linkend='var-bb-BB_NO_NETWORK'><filename>BB_NO_NETWORK</filename></link>
>> variable can be used to make any attempted network access a fatal
>> error, which is useful for checking that mirrors are complete as well
>> as other things. @@ -265,11 +265,11 @@
>> The filename you specify within the URL can be
>> either an absolute or relative path to a file.
>> If the filename is relative, the contents of the
>> - <link
>> linkend='var-FILESPATH'><filename>FILESPATH</filename></link>
>> + <link
>> linkend='var-bb-FILESPATH'><filename>FILESPATH</filename></link>
>> variable is used in the same way <filename>PATH</filename> is used to
>> find executables. If the file cannot be found, it is assumed that it
>> is available in
>> - <link
>> linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
>> + <link
>> linkend='var-bb-DL_DIR'><filename>DL_DIR</filename></link> by the
>> time the <filename>download()</filename> method is called. </para>
>>
>> @@ -304,7 +304,7 @@
>> allows the name of the downloaded file to be
>> specified. Specifying the name of the downloaded file is useful
>> for avoiding collisions in
>> - <link
>> linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
>> + <link
>> linkend='var-bb-DL_DIR'><filename>DL_DIR</filename></link> when
>> dealing with multiple files that have the same name. </para>
>>
>> @@ -355,7 +355,7 @@
>> A special value of "now" causes the checkout
>> to be updated on every build.
>> </para></listitem>
>> - <listitem><para><emphasis><link
>> linkend='var-CVSDIR'><filename>CVSDIR</filename></link>:</emphasis>
>> + <listitem><para><emphasis><link
>> linkend='var-bb-CVSDIR'><filename>CVSDIR</filename></link>:</emphasis>
>> Specifies where a temporary checkout is saved. The location is often
>> <filename>DL_DIR/cvs</filename>. </para></listitem>
>> @@ -395,7 +395,7 @@
>> <listitem><para><emphasis>"date":</emphasis>
>> Specifies a date.
>> If no "date" is specified, the
>> - <link
>> linkend='var-SRCDATE'><filename>SRCDATE</filename></link>
>> + <link
>> linkend='var-bb-SRCDATE'><filename>SRCDATE</filename></link> of the
>> configuration is used to checkout a specific date. The special value
>> of "now" causes the checkout to be updated on every build.
>> @@ -406,7 +406,7 @@
>> to which the module is unpacked.
>> You are forcing the module into a special
>> directory relative to
>> - <link
>> linkend='var-CVSDIR'><filename>CVSDIR</filename></link>.
>> + <link
>> linkend='var-bb-CVSDIR'><filename>CVSDIR</filename></link>.
>> </para></listitem> <listitem><para><emphasis>"rsh"</emphasis>
>> Used in conjunction with the "method"
>> parameter. @@ -448,7 +448,7 @@
>> <filename>FETCHCMD_svn</filename>, which defaults
>> to "svn".
>> The fetcher's temporary working directory is set by
>> - <link
>> linkend='var-SVNDIR'><filename>SVNDIR</filename></link>,
>> + <link
>> linkend='var-bb-SVNDIR'><filename>SVNDIR</filename></link>, which is
>> usually <filename>DL_DIR/svn</filename>. </para>
>>
>> @@ -509,7 +509,7 @@
>> source control system.
>> The fetcher works by creating a bare clone of the
>> remote into
>> - <link
>> linkend='var-GITDIR'><filename>GITDIR</filename></link>,
>> + <link
>> linkend='var-bb-GITDIR'><filename>GITDIR</filename></link>, which is
>> usually <filename>DL_DIR/git2</filename>. This bare clone is then
>> cloned into the work directory during the unpack stage when a
>> specific tree is checked out. @@ -588,6 +588,14 @@
>> The name of the path in which to place the
>> checkout. By default, the path is <filename>git/</filename>.
>> </para></listitem>
>> + <listitem><para><emphasis>"usehead":</emphasis>
>> + Enables local <filename>git://</filename>
>> URLs to use the
>> + current branch HEAD as the revision for use
>> with
>> + <filename>AUTOREV</filename>.
>> + The "usehead" parameter implies no branch
>> and only works
>> + when the transfer protocol is
>> + <filename>file://</filename>.
>> + </para></listitem>
>> </itemizedlist>
>> Here are some example URLs:
>> <literallayout class='monospaced'>
>> @@ -604,7 +612,7 @@
>> This fetcher submodule inherits from the
>> <link linkend='git-fetcher'>Git fetcher</link> and
>> extends that fetcher's behavior by fetching a repository's submodules.
>> - <link
>> linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
>> + <link
>> linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link> is
>> passed to the Git fetcher as described in the "<link
>> linkend='git-fetcher'>Git Fetcher
>> (<filename>git://</filename>)</link>" section. @@ -639,9 +647,9 @@
>>
>> <para>
>> To use this fetcher, make sure your recipe has proper
>> - <link
>> linkend='var-SRC_URI'><filename>SRC_URI</filename></link>,
>> - <link
>> linkend='var-SRCREV'><filename>SRCREV</filename></link>, and
>> - <link
>> linkend='var-PV'><filename>PV</filename></link> settings.
>> + <link
>> linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link>,
>> + <link
>> linkend='var-bb-SRCREV'><filename>SRCREV</filename></link>, and
>> + <link
>> linkend='var-bb-PV'><filename>PV</filename></link> settings. Here is
>> an example: <literallayout class='monospaced'>
>> SRC_URI =
>> "ccrc://cc.example.org/ccrc;vob=/example_vob;module=/example_module"
>> @@ -726,15 +734,15 @@ <filename>FETCHCMD_p4</filename>, which defaults
>> to "p4".
>> The fetcher's temporary working directory is set by
>> - <link
>> linkend='var-P4DIR'><filename>P4DIR</filename></link>,
>> + <link
>> linkend='var-bb-P4DIR'><filename>P4DIR</filename></link>, which
>> defaults to "DL_DIR/p4". </para>
>>
>> <para>
>> To use this fetcher, make sure your recipe has proper
>> - <link
>> linkend='var-SRC_URI'><filename>SRC_URI</filename></link>,
>> - <link
>> linkend='var-SRCREV'><filename>SRCREV</filename></link>, and
>> - <link
>> linkend='var-PV'><filename>PV</filename></link> values.
>> + <link
>> linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link>,
>> + <link
>> linkend='var-bb-SRCREV'><filename>SRCREV</filename></link>, and
>> + <link
>> linkend='var-bb-PV'><filename>PV</filename></link> values. The p4
>> executable is able to use the config file defined by your system's
>> <filename>P4CONFIG</filename> environment variable in order to define
>> the Perforce server URL and port, username, and @@ -785,9 +793,9 @@
>> <filename>google-repo</filename> source control
>> system. The fetcher works by initiating and syncing sources of the
>> repository into
>> - <link
>> linkend='var-REPODIR'><filename>REPODIR</filename></link>,
>> + <link
>> linkend='var-bb-REPODIR'><filename>REPODIR</filename></link>, which
>> is usually
>> - <link
>> linkend='var-DL_DIR'><filename>DL_DIR</filename></link><filename>/repo</filename>.
>> + <link
>> linkend='var-bb-DL_DIR'><filename>DL_DIR</filename></link><filename>/repo</filename>.
>> </para>
>> <para>
>> @@ -824,19 +832,22 @@
>> Bazaar (<filename>bzr://</filename>)
>> </para></listitem>
>> <listitem><para>
>> - Trees using Git Annex
>> (<filename>gitannex://</filename>)
>> + Mercurial (<filename>hg://</filename>)
>> </para></listitem>
>> <listitem><para>
>> - Secure FTP (<filename>sftp://</filename>)
>> + npm (<filename>npm://</filename>)
>> </para></listitem>
>> <listitem><para>
>> - Secure Shell (<filename>ssh://</filename>)
>> + OSC (<filename>osc://</filename>)
>> </para></listitem>
>> <listitem><para>
>> - OSC (<filename>osc://</filename>)
>> + Secure FTP (<filename>sftp://</filename>)
>> </para></listitem>
>> <listitem><para>
>> - Mercurial (<filename>hg://</filename>)
>> + Secure Shell (<filename>ssh://</filename>)
>> + </para></listitem>
>> + <listitem><para>
>> + Trees using Git Annex
>> (<filename>gitannex://</filename>) </para></listitem>
>> </itemizedlist>
>> No documentation currently exists for these lesser
>> used diff --git
>> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.xml
>> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.xml index
>> 9076f0f..39066e4 100644 ---
>> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.xml +++
>> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.xml @@
>> -194,7 +194,7 @@ <para> When you run BitBake, it begins looking for
>> metadata files. The
>> - <link
>> linkend='var-BBPATH'><filename>BBPATH</filename></link>
>> + <link
>> linkend='var-bb-BBPATH'><filename>BBPATH</filename></link> variable
>> is what tells BitBake where to look for those files.
>> <filename>BBPATH</filename> is not set and you need to set it.
>> Without <filename>BBPATH</filename>, Bitbake cannot @@ -273,14
>> +273,14 @@ some editor to create the <filename>bitbake.conf</filename>
>> so that it contains the following:
>> <literallayout class='monospaced'>
>> - <link linkend='var-PN'>PN</link> =
>> "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0]
>> or 'defaultpkgname'}"
>> + <link linkend='var-bb-PN'>PN</link> =
>> "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0]
>> or 'defaultpkgname'}" </literallayout> <literallayout
>> class='monospaced'>
>> - TMPDIR = "${<link linkend='var-TOPDIR'>TOPDIR</link>}/tmp"
>> - <link linkend='var-CACHE'>CACHE</link> = "${TMPDIR}/cache"
>> - <link linkend='var-STAMP'>STAMP</link> =
>> "${TMPDIR}/${PN}/stamps"
>> - <link linkend='var-T'>T</link> = "${TMPDIR}/${PN}/work"
>> - <link linkend='var-B'>B</link> = "${TMPDIR}/${PN}"
>> + TMPDIR = "${<link linkend='var-bb-TOPDIR'>TOPDIR</link>}/tmp"
>> + <link linkend='var-bb-CACHE'>CACHE</link> = "${TMPDIR}/cache"
>> + <link linkend='var-bb-STAMP'>STAMP</link> =
>> "${TMPDIR}/${PN}/stamps"
>> + <link linkend='var-bb-T'>T</link> = "${TMPDIR}/${PN}/work"
>> + <link linkend='var-bb-B'>B</link> = "${TMPDIR}/${PN}"
>> </literallayout>
>> <note>
>> Without a value for <filename>PN</filename>, the
>> @@ -402,12 +402,12 @@
>> Move to the <filename>conf</filename> directory and
>> create a <filename>layer.conf</filename> file that has the following:
>> <literallayout class='monospaced'>
>> - BBPATH .= ":${<link linkend='var-LAYERDIR'>LAYERDIR</link>}"
>> + BBPATH .= ":${<link linkend='var-bb-LAYERDIR'>LAYERDIR</link>}"
>>
>> - <link linkend='var-BBFILES'>BBFILES</link> += "${LAYERDIR}/*.bb"
>> + <link linkend='var-bb-BBFILES'>BBFILES</link> +=
>> "${LAYERDIR}/*.bb"
>> - <link
>> linkend='var-BBFILE_COLLECTIONS'>BBFILE_COLLECTIONS</link> +=
>> "mylayer"
>> - <link
>> linkend='var-BBFILE_PATTERN'>BBFILE_PATTERN_mylayer</link> :=
>> "^${LAYERDIR_RE}/"
>> + <link
>> linkend='var-bb-BBFILE_COLLECTIONS'>BBFILE_COLLECTIONS</link> +=
>> "mylayer"
>> + <link
>> linkend='var-bb-BBFILE_PATTERN'>BBFILE_PATTERN_mylayer</link> :=
>> "^${LAYERDIR_RE}/" </literallayout> For information on these
>> variables, click the links to go to the definitions in the
>> glossary.</para> @@ -416,9 +416,9 @@
>> a recipe file named
>> <filename>printhello.bb</filename> that has the following:
>> <literallayout class='monospaced'>
>> - <link linkend='var-DESCRIPTION'>DESCRIPTION</link> = "Prints
>> Hello World"
>> - <link linkend='var-PN'>PN</link> = 'printhello'
>> - <link linkend='var-PV'>PV</link> = '1'
>> + <link linkend='var-bb-DESCRIPTION'>DESCRIPTION</link> = "Prints
>> Hello World"
>> + <link linkend='var-bb-PN'>PN</link> = 'printhello'
>> + <link linkend='var-bb-PV'>PV</link> = '1'
>>
>> python do_build() {
>> bb.plain("********************");
>> diff --git
>> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml
>> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml index
>> f7d312a..8f2a960 100644 ---
>> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml +++
>> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml @@
>> -781,7 +781,7 @@ target, you must also enable BitBake to perform
>> multiple configuration builds. Enabling is accomplished by setting the
>> - <link
>> linkend='var-BBMULTICONFIG'><filename>BBMULTICONFIG</filename></link>
>> + <link
>> linkend='var-bb-BBMULTICONFIG'><filename>BBMULTICONFIG</filename></link>
>> variable in the <filename>local.conf</filename> configuration file.
>> As an example, suppose you had configuration
>> files @@ -791,7 +791,7 @@
>> The following statement in the
>> <filename>local.conf</filename> file both enables
>> BitBake to perform multiple configuration builds
>> and
>> - specifies the two multiconfigs:
>> + specifies the two extra multiconfigs:
>> <literallayout class='monospaced'>
>> BBMULTICONFIG = "target1 target2"
>> </literallayout>
>> @@ -803,13 +803,13 @@
>> builds, use the following command form to start
>> the builds:
>> <literallayout class='monospaced'>
>> - $ bitbake
>> [multiconfig:<replaceable>multiconfigname</replaceable>:]<replaceable>target</replaceable>
>> [[[multiconfig:<replaceable>multiconfigname</replaceable>:]<replaceable>target</replaceable>] ... ]
>> + $ bitbake
>> [mc:<replaceable>multiconfigname</replaceable>:]<replaceable>target</replaceable>
>> [[[mc:<replaceable>multiconfigname</replaceable>:]<replaceable>target</replaceable>] ... ]
>> </literallayout>
>> - Here is an example for two multiconfigs:
>> + Here is an example for two extra multiconfigs:
>> <filename>target1</filename> and
>> <filename>target2</filename>:
>> <literallayout class='monospaced'>
>> - $ bitbake multiconfig:target1:<replaceable>target</replaceable>
>> multiconfig:target2:<replaceable>target</replaceable>
>> + $ bitbake mc::<replaceable>target</replaceable>
>> mc:target1:<replaceable>target</replaceable>
>> mc:target2:<replaceable>target</replaceable> </literallayout> </para>
>> </section>
>> @@ -837,13 +837,13 @@
>> build, you must declare the dependencies in the
>> recipe using the following statement form:
>> <literallayout class='monospaced'>
>> - <replaceable>task_or_package</replaceable>[mcdepends] =
>> "multiconfig:<replaceable>from_multiconfig</replaceable>:<replaceable>to_multiconfig</replaceable>:<replaceable>recipe_name</replaceable>:<replaceable>task_on_which_to_depend</replaceable>"
>> + <replaceable>task_or_package</replaceable>[mcdepends] =
>> "mc:<replaceable>from_multiconfig</replaceable>:<replaceable>to_multiconfig</replaceable>:<replaceable>recipe_name</replaceable>:<replaceable>task_on_which_to_depend</replaceable>"
>> </literallayout> To better show how to use this statement, consider an
>> example with two multiconfigs:
>> <filename>target1</filename> and <filename>target2</filename>:
>> <literallayout class='monospaced'>
>> - <replaceable>image_task</replaceable>[mcdepends] =
>> "multiconfig:target1:target2:<replaceable>image2</replaceable>:<replaceable>rootfs_task</replaceable>"
>> + <replaceable>image_task</replaceable>[mcdepends] =
>> "mc:target1:target2:<replaceable>image2</replaceable>:<replaceable>rootfs_task</replaceable>"
>> </literallayout> In this example, the
>> <replaceable>from_multiconfig</replaceable> is
>> "target1" and @@ -859,7 +859,7 @@
>> Once you set up this dependency, you can build the
>> "target1" multiconfig using a BitBake command as
>> follows: <literallayout class='monospaced'>
>> - $ bitbake multiconfig:target1:<replaceable>image1</replaceable>
>> + $ bitbake mc:target1:<replaceable>image1</replaceable>
>> </literallayout>
>> This command executes all the tasks needed to
>> create <replaceable>image1</replaceable> for the "target1"
>> @@ -875,7 +875,7 @@
>> Consider this change to the statement in the
>> <replaceable>image1</replaceable> recipe:
>> <literallayout class='monospaced'>
>> - <replaceable>image_task</replaceable>[mcdepends] =
>> "multiconfig:target1:target2:<replaceable>image2</replaceable>:<replaceable>image_task</replaceable>"
>> + <replaceable>image_task</replaceable>[mcdepends] =
>> "mc:target1:target2:<replaceable>image2</replaceable>:<replaceable>image_task</replaceable>"
>> </literallayout> In this case, BitBake must create
>> <replaceable>image2</replaceable> for the
>> "target2" diff --git
>> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml
>> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml
>> index 2490f6e..421364c 100644 ---
>> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml
>> +++
>> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml @@
>> -61,6 +61,78 @@ </para> </section>
>> + <section id='modifying-existing-variables'>
>> + <title>Modifying Existing Variables</title>
>> +
>> + <para>
>> + Sometimes you need to modify existing variables.
>> + Following are some cases where you might find you
>> want to
>> + modify an existing variable:
>> + <itemizedlist>
>> + <listitem><para>
>> + Customize a recipe that uses the variable.
>> + </para></listitem>
>> + <listitem><para>
>> + Change a variable's default value used in a
>> + <filename>*.bbclass</filename> file.
>> + </para></listitem>
>> + <listitem><para>
>> + Change the variable in a
>> <filename>*.bbappend</filename>
>> + file to override the variable in the
>> original recipe.
>> + </para></listitem>
>> + <listitem><para>
>> + Change the variable in a configuration file
>> so that the
>> + value overrides an existing configuration.
>> + </para></listitem>
>> + </itemizedlist>
>> + </para>
>> +
>> + <para>
>> + Changing a variable value can sometimes depend on
>> how the
>> + value was originally assigned and also on the desired
>> + intent of the change.
>> + In particular, when you append a value to a variable
>> that
>> + has a default value, the resulting value might not
>> be what
>> + you expect.
>> + In this case, the value you provide might replace
>> the value
>> + rather than append to the default value.
>> + </para>
>> +
>> + <para>
>> + If after you have changed a variable's value and
>> something
>> + unexplained occurs, you can use BitBake to check the
>> actual
>> + value of the suspect variable.
>> + You can make these checks for both configuration and
>> recipe
>> + level changes:
>> + <itemizedlist>
>> + <listitem><para>
>> + For configuration changes, use the following:
>> + <literallayout class='monospaced'>
>> + $ bitbake -e
>> + </literallayout>
>> + This command displays variable values after
>> the
>> + configuration files (i.e.
>> <filename>local.conf</filename>,
>> + <filename>bblayers.conf</filename>,
>> + <filename>bitbake.conf</filename> and so
>> forth) have
>> + been parsed.
>> + <note>
>> + Variables that are exported to the
>> environment are
>> + preceded by the string "export" in the
>> command's
>> + output.
>> + </note>
>> + </para></listitem>
>> + <listitem><para>
>> + For recipe changes, use the following:
>> + <literallayout class='monospaced'>
>> + $ bitbake <replaceable>recipe</replaceable> -e | grep VARIABLE="
>> + </literallayout>
>> + This command checks to see if the variable
>> actually
>> + makes it into a specific recipe.
>> + </para></listitem>
>> + </itemizedlist>
>> + </para>
>> + </section>
>> +
>> <section id='line-joining'>
>> <title>Line Joining</title>
>>
>> @@ -297,9 +369,8 @@
>>
>> <para>
>> These operators differ from the ":=", ".=", "=.",
>> "+=", and "=+"
>> - operators in that their effects are deferred
>> - until after parsing completes rather than being
>> immediately
>> - applied.
>> + operators in that their effects are applied at
>> variable
>> + expansion time rather than being immediately applied.
>> Here are some examples:
>> <literallayout class='monospaced'>
>> B = "bval"
>> @@ -348,18 +419,22 @@
>> FOO = "123 456 789 123456 123 456 123 456"
>> FOO_remove = "123"
>> FOO_remove = "456"
>> - FOO2 = "abc def ghi abcdef abc def abc def"
>> - FOO2_remove = "abc def"
>> + FOO2 = " abc def ghi abcdef abc def abc def def"
>> + FOO2_remove = " \
>> + def \
>> + abc \
>> + ghi \
>> + "
>> </literallayout>
>> The variable <filename>FOO</filename> becomes
>> - " 789 123456 "
>> + " 789 123456 "
>> and <filename>FOO2</filename> becomes
>> - " ghi abcdef ".
>> +
>> " jkl abcdef ".
>> </para>
>> <para>
>> Like "_append" and "_prepend", "_remove"
>> - is deferred until after parsing completes.
>> + is applied at variable expansion time.
>> </para>
>> </section>
>>
>> @@ -503,7 +578,7 @@
>> </section>
>>
>> <section id='unsetting-variables'>
>> - <title>Unseting variables</title>
>> + <title>Unsetting variables</title>
>>
>> <para>
>> It is possible to completely remove a variable or a
>> variable flag @@ -595,7 +670,7 @@
>>
>> <para>
>> BitBake uses
>> - <link
>> linkend='var-OVERRIDES'><filename>OVERRIDES</filename></link>
>> + <link
>> linkend='var-bb-OVERRIDES'><filename>OVERRIDES</filename></link> to
>> control what variables are overridden after BitBake parses recipes
>> and configuration files. This section describes how you can use
>> @@ -705,7 +780,7 @@
>>
>> <para>Internally, this is implemented by
>> prepending the task (e.g. "task-compile:") to the value of
>> - <link
>> linkend='var-OVERRIDES'><filename>OVERRIDES</filename></link>
>> + <link
>> linkend='var-bb-OVERRIDES'><filename>OVERRIDES</filename></link> for
>> the local datastore of the <filename>do_compile</filename>
>> task.</para>
>> @@ -724,17 +799,15 @@
>> <title>Key Expansion</title>
>>
>> <para>
>> - Key expansion happens when the BitBake datastore is
>> finalized
>> - just before BitBake expands overrides.
>> + Key expansion happens when the BitBake datastore is
>> finalized. To better understand this, consider the following example:
>> <literallayout class='monospaced'>
>> A${B} = "X"
>> B = "2"
>> A2 = "Y"
>> </literallayout>
>> - In this case, after all the parsing is complete, and
>> - before any overrides are handled, BitBake expands
>> - <filename>${B}</filename> into "2".
>> + In this case, after all the parsing is complete,
>> + BitBake expands <filename>${B}</filename> into "2".
>> This expansion causes <filename>A2</filename>, which
>> was set to "Y" before the expansion, to become "X".
>> </para>
>> @@ -868,7 +941,7 @@
>>
>> <para>
>> BitBake uses the
>> - <link
>> linkend='var-BBPATH'><filename>BBPATH</filename></link>
>> + <link
>> linkend='var-bb-BBPATH'><filename>BBPATH</filename></link> variable
>> to locate needed include and class files. Additionally, BitBake
>> searches the current directory for <filename>include</filename> and
>> <filename>require</filename> @@ -1086,7 +1159,7 @@
>> <para>
>> When creating a configuration file
>> (<filename>.conf</filename>), you can use the
>> - <link
>> linkend='var-INHERIT'><filename>INHERIT</filename></link>
>> + <link
>> linkend='var-bb-INHERIT'><filename>INHERIT</filename></link>
>> configuration directive to inherit a class. BitBake only supports
>> this directive when used within a configuration file.
>> @@ -1370,7 +1443,7 @@
>> </para></listitem>
>> <listitem><para>
>> BitBake-style Python functions generate a
>> separate
>> - <filename>${</filename><link
>> linkend='var-T'><filename>T</filename></link><filename>}/run.</filename><replaceable>function-name</replaceable><filename>.</filename><replaceable>pid</replaceable>
>> + <filename>${</filename><link
>> linkend='var-bb-T'><filename>T</filename></link><filename>}/run.</filename><replaceable>function-name</replaceable><filename>.</filename><replaceable>pid</replaceable>
>> script that is executed to run the function, and also generate a log
>> file in
>> <filename>${T}/log.</filename><replaceable>function-name</replaceable><filename>.</filename><replaceable>pid</replaceable>
>> @@ -1773,7 +1846,7 @@ things exported or listed in its whitelist to
>> ensure that the build environment is reproducible and consistent.
>> You can prevent this "cleaning" by setting the
>> - <link
>> linkend='var-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link>
>> + <link
>> linkend='var-bb-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link>
>> variable. </note>
>> Consequently, if you do want something to get passed
>> into the @@ -1783,9 +1856,9 @@
>> Tell BitBake to load what you want from the
>> environment into the datastore.
>> You can do so through the
>> - <link
>> linkend='var-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>
>> + <link
>> linkend='var-bb-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>
>> and
>> - <link
>> linkend='var-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link>
>> + <link
>> linkend='var-bb-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link>
>> variables. For example, assume you want to prevent the build system
>> from accessing your <filename>$HOME/.ccache</filename>
>> @@ -1824,7 +1897,7 @@
>> from the original execution environment.
>> Bitbake saves a copy of the original environment into
>> a special variable named
>> - <link
>> linkend='var-BB_ORIGENV'><filename>BB_ORIGENV</filename></link>.
>> + <link
>> linkend='var-bb-BB_ORIGENV'><filename>BB_ORIGENV</filename></link>.
>> </para>
>> <para>
>> @@ -1883,7 +1956,7 @@
>> <listitem><para><emphasis><filename>[depends]</filename>:</emphasis>
>> Controls inter-task dependencies.
>> See the
>> - <link
>> linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
>> + <link
>> linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link> variable
>> and the "<link linkend='inter-task-dependencies'>Inter-Task
>> Dependencies</link>" section for more information.
>> @@ -1891,7 +1964,7 @@
>> <listitem><para><emphasis><filename>[deptask]</filename>:</emphasis>
>> Controls task build-time dependencies.
>> See the
>> - <link
>> linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
>> + <link
>> linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link> variable
>> and the "<link linkend='build-dependencies'>Build Dependencies</link>"
>> section for more information.
>> @@ -1937,7 +2010,7 @@
>> of cores but certain tasks need to be
>> rate-limited due to various kinds of resource constraints (e.g. to
>> avoid network throttling). <filename>number_threads</filename> works
>> similarly to the
>> - <link
>> linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
>> + <link
>> linkend='var-bb-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link>
>> variable but is task-specific.</para>
>> <para>Set the value globally.
>> @@ -1971,9 +2044,9 @@
>> <listitem><para><emphasis><filename>[rdepends]</filename>:</emphasis>
>> Controls inter-task runtime dependencies.
>> See the
>> - <link
>> linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link>
>> + <link
>> linkend='var-bb-RDEPENDS'><filename>RDEPENDS</filename></link>
>> variable, the
>> - <link
>> linkend='var-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
>> + <link
>> linkend='var-bb-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
>> variable, and the "<link linkend='inter-task-dependencies'>Inter-Task
>> Dependencies</link>" section for more information.
>> @@ -1981,9 +2054,9 @@
>> <listitem><para><emphasis><filename>[rdeptask]</filename>:</emphasis>
>> Controls task runtime dependencies.
>> See the
>> - <link
>> linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link>
>> + <link
>> linkend='var-bb-RDEPENDS'><filename>RDEPENDS</filename></link>
>> variable, the
>> - <link
>> linkend='var-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
>> + <link
>> linkend='var-bb-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
>> variable, and the "<link linkend='runtime-dependencies'>Runtime
>> Dependencies</link>" section for more information.
>> @@ -1996,9 +2069,9 @@
>> <listitem><para><emphasis><filename>[recrdeptask]</filename>:</emphasis>
>> Controls task recursive runtime dependencies.
>> See the
>> - <link
>> linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link>
>> + <link
>> linkend='var-bb-RDEPENDS'><filename>RDEPENDS</filename></link>
>> variable, the
>> - <link
>> linkend='var-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
>> + <link
>> linkend='var-bb-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
>> variable, and the "<link linkend='recursive-dependencies'>Recursive
>> Dependencies</link>" section for more information.
>> @@ -2127,7 +2200,7 @@
>> Any given datastore only has one such event
>> executed against it, however.
>> If
>> - <link
>> linkende='var-BB_INVALIDCONF'><filename>BB_INVALIDCONF</filename></link>
>> + <link
>> linkende='var-bb-BB_INVALIDCONF'><filename>BB_INVALIDCONF</filename></link>
>> is set in the datastore by the event handler, the configuration is
>> reparsed and a new event triggered, allowing the metadata to update
>> configuration. @@ -2256,17 +2329,17 @@
>> from a single recipe file multiple incarnations of that
>> recipe file where all incarnations are buildable.
>> These features are enabled through the
>> - <link
>> linkend='var-BBCLASSEXTEND'><filename>BBCLASSEXTEND</filename></link>
>> + <link
>> linkend='var-bb-BBCLASSEXTEND'><filename>BBCLASSEXTEND</filename></link>
>> and
>> - <link
>> linkend='var-BBVERSIONS'><filename>BBVERSIONS</filename></link>
>> + <link
>> linkend='var-bb-BBVERSIONS'><filename>BBVERSIONS</filename></link>
>> variables. <note>
>> The mechanism for this class extension is extremely
>> specific to the implementation.
>> Usually, the recipe's
>> - <link
>> linkend='var-PROVIDES'><filename>PROVIDES</filename></link>,
>> - <link
>> linkend='var-PN'><filename>PN</filename></link>, and
>> - <link
>> linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
>> + <link
>> linkend='var-bb-PROVIDES'><filename>PROVIDES</filename></link>,
>> + <link
>> linkend='var-bb-PN'><filename>PN</filename></link>, and
>> + <link
>> linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link>
>> variables would need to be modified by the extension class. For
>> specific examples, see the OE-Core <filename>native</filename>,
>> <filename>nativesdk</filename>, @@ -2287,7 +2360,7 @@
>> project from a single recipe file.
>> You can also specify conditional metadata
>> (using the
>> - <link
>> linkend='var-OVERRIDES'><filename>OVERRIDES</filename></link>
>> + <link
>> linkend='var-bb-OVERRIDES'><filename>OVERRIDES</filename></link>
>> mechanism) for a single version, or an optionally named range of
>> versions. Here is an example: <literallayout class='monospaced'>
>> @@ -2306,7 +2379,7 @@
>> into overrides, but it is also made available
>> for the metadata to use in the variable that defines the base recipe
>> versions for use in <filename>file://</filename> search paths
>> - (<link
>> linkend='var-FILESPATH'><filename>FILESPATH</filename></link>).
>> + (<link
>> linkend='var-bb-FILESPATH'><filename>FILESPATH</filename></link>).
>> </para></listitem> </itemizedlist>
>> </para>
>> @@ -2408,7 +2481,7 @@
>>
>> <para>
>> BitBake uses the
>> - <link
>> linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
>> + <link
>> linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link> variable
>> to manage build time dependencies. The <filename>[deptask]</filename>
>> varflag for tasks signifies the task of each
>> @@ -2429,9 +2502,9 @@
>>
>> <para>
>> BitBake uses the
>> - <link
>> linkend='var-PACKAGES'><filename>PACKAGES</filename></link>,
>> - <link
>> linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link>, and
>> - <link
>> linkend='var-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
>> + <link
>> linkend='var-bb-PACKAGES'><filename>PACKAGES</filename></link>,
>> + <link
>> linkend='var-bb-RDEPENDS'><filename>RDEPENDS</filename></link>, and
>> + <link
>> linkend='var-bb-RRECOMMENDS'><filename>RRECOMMENDS</filename></link>
>> variables to manage runtime dependencies. </para>
>>
>> @@ -2686,7 +2759,7 @@
>>
>> <para>
>> These checksums are stored in
>> - <link
>> linkend='var-STAMP'><filename>STAMP</filename></link>.
>> + <link
>> linkend='var-bb-STAMP'><filename>STAMP</filename></link>. You can
>> examine the checksums using the following BitBake command:
>> <literallayout class='monospaced'> $ bitbake-dumpsigs
>> @@ -2708,44 +2781,44 @@
>> The following list describes related variables:
>> <itemizedlist>
>> <listitem><para>
>> - <link
>> linkend='var-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></link>:
>> + <link
>> linkend='var-bb-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></link>:
>> Specifies the name of the function to call during the "setscene" part
>> of the task's execution in order to validate the list of task hashes.
>> </para></listitem>
>> <listitem><para>
>> - <link
>> linkend='var-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></link>:
>> + <link
>> linkend='var-bb-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></link>:
>> Specifies a function BitBake calls that determines whether BitBake
>> requires a setscene dependency to be met.
>> </para></listitem>
>> <listitem><para>
>> - <link
>> linkend='var-BB_SETSCENE_VERIFY_FUNCTION2'><filename>BB_SETSCENE_VERIFY_FUNCTION2</filename></link>:
>> + <link
>> linkend='var-bb-BB_SETSCENE_VERIFY_FUNCTION2'><filename>BB_SETSCENE_VERIFY_FUNCTION2</filename></link>:
>> Specifies a function to call that verifies the list of planned task
>> execution before the main task execution happens.
>> </para></listitem>
>> <listitem><para>
>> - <link
>> linkend='var-BB_STAMP_POLICY'><filename>BB_STAMP_POLICY</filename></link>:
>> + <link
>> linkend='var-bb-BB_STAMP_POLICY'><filename>BB_STAMP_POLICY</filename></link>:
>> Defines the mode for comparing timestamps of stamp files.
>> </para></listitem> <listitem><para>
>> - <link
>> linkend='var-BB_STAMP_WHITELIST'><filename>BB_STAMP_WHITELIST</filename></link>:
>> + <link
>> linkend='var-bb-BB_STAMP_WHITELIST'><filename>BB_STAMP_WHITELIST</filename></link>:
>> Lists stamp files that are looked at when the stamp policy is
>> "whitelist". </para></listitem>
>> <listitem><para>
>> - <link
>> linkend='var-BB_TASKHASH'><filename>BB_TASKHASH</filename></link>:
>> + <link
>> linkend='var-bb-BB_TASKHASH'><filename>BB_TASKHASH</filename></link>:
>> Within an executing task, this variable holds the hash of the task as
>> returned by the currently enabled signature generator.
>> </para></listitem>
>> <listitem><para>
>> - <link
>> linkend='var-STAMP'><filename>STAMP</filename></link>:
>> + <link
>> linkend='var-bb-STAMP'><filename>STAMP</filename></link>: The base
>> path to create stamp files. </para></listitem>
>> <listitem><para>
>> - <link
>> linkend='var-STAMPCLEAN'><filename>STAMPCLEAN</filename></link>:
>> + <link
>> linkend='var-bb-STAMPCLEAN'><filename>STAMPCLEAN</filename></link>:
>> Again, the base path to create stamp files but can use wildcards for
>> matching a range of files for clean operations. </para></listitem>
>> diff --git
>> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml
>> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml
>> index a84b2bc..aca6741 100644 ---
>> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml
>> +++
>> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml
>> @@ -3,7 +3,7 @@ [<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
>> <!-- Dummy chapter --> -<chapter id='ref-variables-glos'>
>> +<chapter id='ref-bb-variables-glos'>
>>
>> <title>Variables Glossary</title>
>>
>> @@ -34,29 +34,29 @@
>> </itemizedlist>
>> </note>
>>
>> -<glossary id='ref-variables-glossary'>
>> +<glossary id='ref-bb-variables-glossary'>
>>
>> <para>
>> - <link linkend='var-ASSUME_PROVIDED'>A</link>
>> - <link linkend='var-B'>B</link>
>> - <link linkend='var-CACHE'>C</link>
>> - <link linkend='var-DEFAULT_PREFERENCE'>D</link>
>> - <link linkend='var-EXCLUDE_FROM_WORLD'>E</link>
>> - <link linkend='var-FAKEROOT'>F</link>
>> - <link linkend='var-GITDIR'>G</link>
>> - <link linkend='var-HGDIR'>H</link>
>> -<!-- <link linkend='var-ICECC_DISABLED'>I</link> -->
>> + <link linkend='var-bb-ASSUME_PROVIDED'>A</link>
>> + <link linkend='var-bb-B'>B</link>
>> + <link linkend='var-bb-CACHE'>C</link>
>> + <link linkend='var-bb-DEFAULT_PREFERENCE'>D</link>
>> + <link linkend='var-bb-EXCLUDE_FROM_WORLD'>E</link>
>> + <link linkend='var-bb-FAKEROOT'>F</link>
>> + <link linkend='var-bb-GITDIR'>G</link>
>> + <link linkend='var-bb-HGDIR'>H</link>
>> + <link linkend='var-bb-INHERIT'>I</link>
>> <!-- <link linkend='var-glossary-j'>J</link> -->
>> <!-- <link linkend='var-KARCH'>K</link> -->
>> - <link linkend='var-LAYERDEPENDS'>L</link>
>> - <link linkend='var-MIRRORS'>M</link>
>> + <link linkend='var-bb-LAYERDEPENDS'>L</link>
>> + <link linkend='var-bb-MIRRORS'>M</link>
>> <!-- <link linkend='var-glossary-n'>N</link> -->
>> - <link linkend='var-OVERRIDES'>O</link>
>> - <link linkend='var-P4DIR'>P</link>
>> + <link linkend='var-bb-OVERRIDES'>O</link>
>> + <link linkend='var-bb-P4DIR'>P</link>
>> <!-- <link linkend='var-QMAKE_PROFILES'>Q</link> -->
>> - <link linkend='var-RDEPENDS'>R</link>
>> - <link linkend='var-SECTION'>S</link>
>> - <link linkend='var-T'>T</link>
>> + <link linkend='var-bb-RDEPENDS'>R</link>
>> + <link linkend='var-bb-SECTION'>S</link>
>> + <link linkend='var-bb-T'>T</link>
>> <!-- <link linkend='var-UBOOT_CONFIG'>U</link> -->
>> <!-- <link linkend='var-glossary-v'>V</link> -->
>> <!-- <link linkend='var-WARN_QA'>W</link> -->
>> @@ -65,13 +65,13 @@
>> <!-- <link linkend='var-glossary-z'>Z</link>-->
>> </para>
>>
>> - <glossdiv id='var-glossary-a'><title>A</title>
>> + <glossdiv id='var-bb-glossary-a'><title>A</title>
>>
>> - <glossentry
>> id='var-ASSUME_PROVIDED'><glossterm>ASSUME_PROVIDED</glossterm>
>> + <glossentry
>> id='var-bb-ASSUME_PROVIDED'><glossterm>ASSUME_PROVIDED</glossterm>
>> <glossdef> <para>
>> Lists recipe names
>> - (<link
>> linkend='var-PN'><filename>PN</filename></link>
>> + (<link
>> linkend='var-bb-PN'><filename>PN</filename></link> values) BitBake
>> does not attempt to build. Instead, BitBake assumes these recipes
>> have already been built.
>> @@ -91,9 +91,9 @@
>> </glossdiv>
>>
>>
>> - <glossdiv id='var-glossary-b'><title>B</title>
>> + <glossdiv id='var-bb-glossary-b'><title>B</title>
>>
>> - <glossentry id='var-B'><glossterm>B</glossterm>
>> + <glossentry id='var-bb-B'><glossterm>B</glossterm>
>> <glossdef>
>> <para>
>> The directory in which BitBake executes functions
>> @@ -102,7 +102,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_ALLOWED_NETWORKS'><glossterm>BB_ALLOWED_NETWORKS</glossterm>
>> + <glossentry
>> id='var-bb-BB_ALLOWED_NETWORKS'><glossterm>BB_ALLOWED_NETWORKS</glossterm>
>> <glossdef> <para>
>> Specifies a space-delimited list of hosts that
>> the fetcher @@ -111,7 +111,7 @@
>> <itemizedlist>
>> <listitem><para>
>> This host list is only used if
>> - <link
>> linkend='var-BB_NO_NETWORK'><filename>BB_NO_NETWORK</filename></link>
>> + <link
>> linkend='var-bb-BB_NO_NETWORK'><filename>BB_NO_NETWORK</filename></link>
>> is either not set or set to "0". </para></listitem>
>> <listitem><para>
>> @@ -151,13 +151,13 @@
>> </itemizedlist>
>> Using <filename>BB_ALLOWED_NETWORKS</filename> in
>> conjunction with
>> - <link
>> linkend='var-PREMIRRORS'><filename>PREMIRRORS</filename></link>
>> + <link
>> linkend='var-bb-PREMIRRORS'><filename>PREMIRRORS</filename></link> is
>> very useful. Adding the host you want to use to
>> <filename>PREMIRRORS</filename> results in the
>> source code being fetched from an allowed location and avoids raising
>> an error when a host that is not allowed is in a
>> - <link
>> linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
>> + <link
>> linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link>
>> statement. This is because the fetcher does not attempt to use the
>> host listed in <filename>SRC_URI</filename>
>> after a @@ -167,7 +167,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_CONSOLELOG'><glossterm>BB_CONSOLELOG</glossterm>
>> + <glossentry
>> id='var-bb-BB_CONSOLELOG'><glossterm>BB_CONSOLELOG</glossterm>
>> <glossdef> <para>
>> Specifies the path to a log file into which
>> BitBake's user @@ -176,7 +176,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_CURRENTTASK'><glossterm>BB_CURRENTTASK</glossterm>
>> + <glossentry
>> id='var-bb-BB_CURRENTTASK'><glossterm>BB_CURRENTTASK</glossterm>
>> <glossdef> <para>
>> Contains the name of the currently running task.
>> @@ -186,7 +186,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_DANGLINGAPPENDS_WARNONLY'><glossterm>BB_DANGLINGAPPENDS_WARNONLY</glossterm>
>> + <glossentry
>> id='var-bb-BB_DANGLINGAPPENDS_WARNONLY'><glossterm>BB_DANGLINGAPPENDS_WARNONLY</glossterm>
>> <glossdef> <para>
>> Defines how BitBake handles situations where an
>> append @@ -208,7 +208,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_DEFAULT_TASK'><glossterm>BB_DEFAULT_TASK</glossterm>
>> + <glossentry
>> id='var-bb-BB_DEFAULT_TASK'><glossterm>BB_DEFAULT_TASK</glossterm>
>> <glossdef> <para>
>> The default task to use when none is specified
>> (e.g. @@ -219,7 +219,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_DISKMON_DIRS'><glossterm>BB_DISKMON_DIRS</glossterm>
>> + <glossentry
>> id='var-bb-BB_DISKMON_DIRS'><glossterm>BB_DISKMON_DIRS</glossterm>
>> <glossdef> <para>
>> Monitors disk space and available inodes during
>> the build @@ -245,7 +245,7 @@
>> build when a threshold is broken.
>> Subsequent warnings are issued as
>> defined by the
>> - <link
>> linkend='var-BB_DISKMON_WARNINTERVAL'>BB_DISKMON_WARNINTERVAL</link>
>> variable,
>> + <link
>> linkend='var-bb-BB_DISKMON_WARNINTERVAL'>BB_DISKMON_WARNINTERVAL</link>
>> variable, which must be defined.
>> <dir> is:
>> @@ -275,7 +275,7 @@
>> BB_DISKMON_DIRS = "ABORT,${TMPDIR},,100K"
>> </literallayout>
>> The first example works only if you also set
>> - the <link
>> linkend='var-BB_DISKMON_WARNINTERVAL'><filename>BB_DISKMON_WARNINTERVAL</filename></link>
>> variable.
>> + the <link
>> linkend='var-bb-BB_DISKMON_WARNINTERVAL'><filename>BB_DISKMON_WARNINTERVAL</filename></link>
>> variable. This example causes the build system to immediately abort
>> when either the disk space in <filename>${TMPDIR}</filename> drops
>> below 1 Gbyte or the available free inodes drops below @@ -309,7
>> +309,7 @@ </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_DISKMON_WARNINTERVAL'><glossterm>BB_DISKMON_WARNINTERVAL</glossterm>
>> + <glossentry
>> id='var-bb-BB_DISKMON_WARNINTERVAL'><glossterm>BB_DISKMON_WARNINTERVAL</glossterm>
>> <glossdef> <para>
>> Defines the disk space and free inode warning
>> intervals. @@ -319,7 +319,7 @@
>> If you are going to use the
>> <filename>BB_DISKMON_WARNINTERVAL</filename>
>> variable, you must also use the
>> - <link
>> linkend='var-BB_DISKMON_DIRS'><filename>BB_DISKMON_DIRS</filename></link>
>> variable
>> + <link
>> linkend='var-bb-BB_DISKMON_DIRS'><filename>BB_DISKMON_DIRS</filename></link>
>> variable and define its action as "WARN". During the build,
>> subsequent warnings are issued each time disk space or number of free
>> inodes further reduces by @@ -374,7 +374,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_ENV_WHITELIST'><glossterm>BB_ENV_WHITELIST</glossterm>
>> + <glossentry
>> id='var-bb-BB_ENV_WHITELIST'><glossterm>BB_ENV_WHITELIST</glossterm>
>> <glossdef> <para>
>> Specifies the internal whitelist of variables to
>> allow @@ -382,11 +382,11 @@
>> datastore.
>> If the value of this variable is not specified
>> (which is the default), the following list is
>> used:
>> - <link
>> linkend='var-BBPATH'><filename>BBPATH</filename></link>,
>> - <link
>> linkend='var-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link>,
>> - <link
>> linkend='var-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>,
>> + <link
>> linkend='var-bb-BBPATH'><filename>BBPATH</filename></link>,
>> + <link
>> linkend='var-bb-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link>,
>> + <link
>> linkend='var-bb-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>,
>> and
>> - <link
>> linkend='var-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link>.
>> + <link
>> linkend='var-bb-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link>.
>> <note> You must set this variable in the external environment
>> in order for it to work.
>> @@ -395,7 +395,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_ENV_EXTRAWHITE'><glossterm>BB_ENV_EXTRAWHITE</glossterm>
>> + <glossentry
>> id='var-bb-BB_ENV_EXTRAWHITE'><glossterm>BB_ENV_EXTRAWHITE</glossterm>
>> <glossdef> <para>
>> Specifies an additional set of variables to
>> allow through @@ -403,7 +403,7 @@
>> datastore.
>> This list of variables are on top of the
>> internal list set in
>> - <link
>> linkend='var-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>.
>> + <link
>> linkend='var-bb-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link>.
>> <note> You must set this variable in the external
>> environment in order for it to work.
>> @@ -412,22 +412,22 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_FETCH_PREMIRRORONLY'><glossterm>BB_FETCH_PREMIRRORONLY</glossterm>
>> + <glossentry
>> id='var-bb-BB_FETCH_PREMIRRORONLY'><glossterm>BB_FETCH_PREMIRRORONLY</glossterm>
>> <glossdef> <para>
>> When set to "1", causes BitBake's fetcher module
>> to only search
>> - <link
>> linkend='var-PREMIRRORS'><filename>PREMIRRORS</filename></link>
>> + <link
>> linkend='var-bb-PREMIRRORS'><filename>PREMIRRORS</filename></link>
>> for files. BitBake will not search the main
>> - <link
>> linkend='var-SRC_URI'><filename>SRC_URI</filename></link>
>> + <link
>> linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link> or
>> - <link
>> linkend='var-MIRRORS'><filename>MIRRORS</filename></link>.
>> + <link
>> linkend='var-bb-MIRRORS'><filename>MIRRORS</filename></link>. </para>
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_FILENAME'><glossterm>BB_FILENAME</glossterm>
>> + <glossentry
>> id='var-bb-BB_FILENAME'><glossterm>BB_FILENAME</glossterm> <glossdef>
>> <para>
>> Contains the filename of the recipe that owns
>> the currently @@ -440,12 +440,12 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_GENERATE_MIRROR_TARBALLS'><glossterm>BB_GENERATE_MIRROR_TARBALLS</glossterm>
>> + <glossentry
>> id='var-bb-BB_GENERATE_MIRROR_TARBALLS'><glossterm>BB_GENERATE_MIRROR_TARBALLS</glossterm>
>> <glossdef> <para>
>> Causes tarballs of the Git repositories,
>> including the Git metadata, to be placed in the
>> - <link
>> linkend='var-DL_DIR'><filename>DL_DIR</filename></link>
>> + <link
>> linkend='var-bb-DL_DIR'><filename>DL_DIR</filename></link> directory.
>> Anyone wishing to create a source mirror would
>> want to enable this variable.
>> @@ -461,7 +461,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_HASHCONFIG_WHITELIST'><glossterm>BB_HASHCONFIG_WHITELIST</glossterm>
>> + <glossentry
>> id='var-bb-BB_HASHCONFIG_WHITELIST'><glossterm>BB_HASHCONFIG_WHITELIST</glossterm>
>> <glossdef> <para>
>> Lists variables that are excluded from base
>> configuration @@ -485,7 +485,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_HASHBASE_WHITELIST'><glossterm>BB_HASHBASE_WHITELIST</glossterm>
>> + <glossentry
>> id='var-bb-BB_HASHBASE_WHITELIST'><glossterm>BB_HASHBASE_WHITELIST</glossterm>
>> <glossdef> <para>
>> Lists variables that are excluded from checksum
>> and @@ -500,7 +500,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_HASHCHECK_FUNCTION'><glossterm>BB_HASHCHECK_FUNCTION</glossterm>
>> + <glossentry
>> id='var-bb-BB_HASHCHECK_FUNCTION'><glossterm>BB_HASHCHECK_FUNCTION</glossterm>
>> <glossdef> <para>
>> Specifies the name of the function to call
>> during the @@ -524,7 +524,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_INVALIDCONF'><glossterm>BB_INVALIDCONF</glossterm>
>> + <glossentry
>> id='var-bb-BB_INVALIDCONF'><glossterm>BB_INVALIDCONF</glossterm>
>> <glossdef> <para>
>> Used in combination with the
>> @@ -539,11 +539,11 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_LOGFMT'><glossterm>BB_LOGFMT</glossterm>
>> + <glossentry
>> id='var-bb-BB_LOGFMT'><glossterm>BB_LOGFMT</glossterm> <glossdef>
>> <para>
>> Specifies the name of the log files saved into
>> - <filename>${</filename><link
>> linkend='var-T'><filename>T</filename></link><filename>}</filename>.
>> + <filename>${</filename><link
>> linkend='var-bb-T'><filename>T</filename></link><filename>}</filename>.
>> By default, the <filename>BB_LOGFMT</filename> variable is undefined
>> and the log file names get created using the following form:
>> @@ -556,7 +556,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_NICE_LEVEL'><glossterm>BB_NICE_LEVEL</glossterm>
>> + <glossentry
>> id='var-bb-BB_NICE_LEVEL'><glossterm>BB_NICE_LEVEL</glossterm>
>> <glossdef> <para>
>> Allows BitBake to run at a specific priority
>> @@ -564,13 +564,13 @@
>> System permissions usually mean that BitBake can
>> reduce its priority but not raise it again.
>> See
>> - <link
>> linkend='var-BB_TASK_NICE_LEVEL'><filename>BB_TASK_NICE_LEVEL</filename></link>
>> + <link
>> linkend='var-bb-BB_TASK_NICE_LEVEL'><filename>BB_TASK_NICE_LEVEL</filename></link>
>> for additional information. </para>
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_NO_NETWORK'><glossterm>BB_NO_NETWORK</glossterm>
>> + <glossentry
>> id='var-bb-BB_NO_NETWORK'><glossterm>BB_NO_NETWORK</glossterm>
>> <glossdef> <para>
>> Disables network access in the BitBake fetcher
>> modules. @@ -587,7 +587,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_NUMBER_THREADS'><glossterm>BB_NUMBER_THREADS</glossterm>
>> + <glossentry
>> id='var-bb-BB_NUMBER_THREADS'><glossterm>BB_NUMBER_THREADS</glossterm>
>> <glossdef> <para>
>> The maximum number of tasks BitBake should run
>> in parallel @@ -599,7 +599,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_NUMBER_PARSE_THREADS'><glossterm>BB_NUMBER_PARSE_THREADS</glossterm>
>> + <glossentry
>> id='var-bb-BB_NUMBER_PARSE_THREADS'><glossterm>BB_NUMBER_PARSE_THREADS</glossterm>
>> <glossdef> <para>
>> Sets the number of threads BitBake uses when
>> parsing. @@ -609,7 +609,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_ORIGENV'><glossterm>BB_ORIGENV</glossterm>
>> + <glossentry
>> id='var-bb-BB_ORIGENV'><glossterm>BB_ORIGENV</glossterm> <glossdef>
>> <para>
>> Contains a copy of the original external
>> environment in @@ -625,7 +625,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_PRESERVE_ENV'><glossterm>BB_PRESERVE_ENV</glossterm>
>> + <glossentry
>> id='var-bb-BB_PRESERVE_ENV'><glossterm>BB_PRESERVE_ENV</glossterm>
>> <glossdef> <para>
>> Disables whitelisting and instead allows all
>> variables @@ -639,12 +639,12 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_RUNFMT'><glossterm>BB_RUNFMT</glossterm>
>> + <glossentry
>> id='var-bb-BB_RUNFMT'><glossterm>BB_RUNFMT</glossterm> <glossdef>
>> <para>
>> Specifies the name of the executable script files
>> (i.e. run files) saved into
>> - <filename>${</filename><link
>> linkend='var-T'><filename>T</filename></link><filename>}</filename>.
>> + <filename>${</filename><link
>> linkend='var-bb-T'><filename>T</filename></link><filename>}</filename>.
>> By default, the <filename>BB_RUNFMT</filename> variable is undefined
>> and the run file names get created using the following form:
>> @@ -657,7 +657,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_RUNTASK'><glossterm>BB_RUNTASK</glossterm>
>> + <glossentry
>> id='var-bb-BB_RUNTASK'><glossterm>BB_RUNTASK</glossterm> <glossdef>
>> <para>
>> Contains the name of the currently executing
>> task. @@ -669,7 +669,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_SCHEDULER'><glossterm>BB_SCHEDULER</glossterm>
>> + <glossentry
>> id='var-bb-BB_SCHEDULER'><glossterm>BB_SCHEDULER</glossterm>
>> <glossdef> <para>
>> Selects the name of the scheduler to use for the
>> @@ -695,7 +695,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_SCHEDULERS'><glossterm>BB_SCHEDULERS</glossterm>
>> + <glossentry
>> id='var-bb-BB_SCHEDULERS'><glossterm>BB_SCHEDULERS</glossterm>
>> <glossdef> <para>
>> Defines custom schedulers to import.
>> @@ -705,13 +705,13 @@
>>
>> <para>
>> For information how to select a scheduler, see
>> the
>> - <link
>> linkend='var-BB_SCHEDULER'><filename>BB_SCHEDULER</filename></link>
>> + <link
>> linkend='var-bb-BB_SCHEDULER'><filename>BB_SCHEDULER</filename></link>
>> variable. </para>
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_SETSCENE_DEPVALID'><glossterm>BB_SETSCENE_DEPVALID</glossterm>
>> + <glossentry
>> id='var-bb-BB_SETSCENE_DEPVALID'><glossterm>BB_SETSCENE_DEPVALID</glossterm>
>> <glossdef> <para>
>> Specifies a function BitBake calls that
>> determines @@ -731,7 +731,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_SETSCENE_VERIFY_FUNCTION2'><glossterm>BB_SETSCENE_VERIFY_FUNCTION2</glossterm>
>> + <glossentry
>> id='var-bb-BB_SETSCENE_VERIFY_FUNCTION2'><glossterm>BB_SETSCENE_VERIFY_FUNCTION2</glossterm>
>> <glossdef> <para>
>> Specifies a function to call that verifies the
>> list of @@ -752,7 +752,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_SIGNATURE_EXCLUDE_FLAGS'><glossterm>BB_SIGNATURE_EXCLUDE_FLAGS</glossterm>
>> + <glossentry
>> id='var-bb-BB_SIGNATURE_EXCLUDE_FLAGS'><glossterm>BB_SIGNATURE_EXCLUDE_FLAGS</glossterm>
>> <glossdef> <para>
>> Lists variable flags (varflags)
>> @@ -771,7 +771,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_SIGNATURE_HANDLER'><glossterm>BB_SIGNATURE_HANDLER</glossterm>
>> + <glossentry
>> id='var-bb-BB_SIGNATURE_HANDLER'><glossterm>BB_SIGNATURE_HANDLER</glossterm>
>> <glossdef> <para>
>> Defines the name of the signature handler
>> BitBake uses. @@ -790,7 +790,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_SRCREV_POLICY'><glossterm>BB_SRCREV_POLICY</glossterm>
>> + <glossentry
>> id='var-bb-BB_SRCREV_POLICY'><glossterm>BB_SRCREV_POLICY</glossterm>
>> <glossdef> <para>
>> Defines the behavior of the fetcher when it
>> interacts with @@ -817,7 +817,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_STAMP_POLICY'><glossterm>BB_STAMP_POLICY</glossterm>
>> + <glossentry
>> id='var-bb-BB_STAMP_POLICY'><glossterm>BB_STAMP_POLICY</glossterm>
>> <glossdef> <para>
>> Defines the mode used for how timestamps of
>> stamp files @@ -836,7 +836,7 @@
>> <listitem><para><emphasis>whitelist</emphasis>
>> - Identical to "full" mode except timestamp
>> comparisons are made for recipes listed
>> in the
>> - <link
>> linkend='var-BB_STAMP_WHITELIST'><filename>BB_STAMP_WHITELIST</filename></link>
>> + <link
>> linkend='var-bb-BB_STAMP_WHITELIST'><filename>BB_STAMP_WHITELIST</filename></link>
>> variable. </para></listitem>
>> </itemizedlist>
>> @@ -848,19 +848,19 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_STAMP_WHITELIST'><glossterm>BB_STAMP_WHITELIST</glossterm>
>> + <glossentry
>> id='var-bb-BB_STAMP_WHITELIST'><glossterm>BB_STAMP_WHITELIST</glossterm>
>> <glossdef> <para>
>> Lists files whose stamp file timestamps are
>> compared when the stamp policy mode is set to "whitelist".
>> For information on stamp policies, see the
>> - <link
>> linkend='var-BB_STAMP_POLICY'><filename>BB_STAMP_POLICY</filename></link>
>> + <link
>> linkend='var-bb-BB_STAMP_POLICY'><filename>BB_STAMP_POLICY</filename></link>
>> variable. </para>
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_STRICT_CHECKSUM'><glossterm>BB_STRICT_CHECKSUM</glossterm>
>> + <glossentry
>> id='var-bb-BB_STRICT_CHECKSUM'><glossterm>BB_STRICT_CHECKSUM</glossterm>
>> <glossdef> <para>
>> Sets a more strict checksum mechanism for
>> non-local URLs. @@ -871,7 +871,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_TASK_IONICE_LEVEL'><glossterm>BB_TASK_IONICE_LEVEL</glossterm>
>> + <glossentry
>> id='var-bb-BB_TASK_IONICE_LEVEL'><glossterm>BB_TASK_IONICE_LEVEL</glossterm>
>> <glossdef> <para>
>> Allows adjustment of a task's Input/Output
>> priority. @@ -882,7 +882,7 @@
>> variable to adjust the I/O priority of these
>> tasks. <note>
>> This variable works similarly to the
>> - <link
>> linkend='var-BB_TASK_NICE_LEVEL'><filename>BB_TASK_NICE_LEVEL</filename></link>
>> + <link
>> linkend='var-bb-BB_TASK_NICE_LEVEL'><filename>BB_TASK_NICE_LEVEL</filename></link>
>> variable except with a task's I/O priorities. </note>
>> </para>
>> @@ -921,7 +921,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_TASK_NICE_LEVEL'><glossterm>BB_TASK_NICE_LEVEL</glossterm>
>> + <glossentry
>> id='var-bb-BB_TASK_NICE_LEVEL'><glossterm>BB_TASK_NICE_LEVEL</glossterm>
>> <glossdef> <para>
>> Allows specific tasks to change their priority
>> @@ -940,7 +940,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_TASKHASH'><glossterm>BB_TASKHASH</glossterm>
>> + <glossentry
>> id='var-bb-BB_TASKHASH'><glossterm>BB_TASKHASH</glossterm> <glossdef>
>> <para>
>> Within an executing task, this variable holds
>> the hash @@ -950,7 +950,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_VERBOSE_LOGS'><glossterm>BB_VERBOSE_LOGS</glossterm>
>> + <glossentry
>> id='var-bb-BB_VERBOSE_LOGS'><glossterm>BB_VERBOSE_LOGS</glossterm>
>> <glossdef> <para>
>> Controls how verbose BitBake is during builds.
>> @@ -960,7 +960,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BB_WORKERCONTEXT'><glossterm>BB_WORKERCONTEXT</glossterm>
>> + <glossentry
>> id='var-bb-BB_WORKERCONTEXT'><glossterm>BB_WORKERCONTEXT</glossterm>
>> <glossdef> <para>
>> Specifies if the current context is executing a
>> task. @@ -973,7 +973,7 @@
>> </glossentry>
>>
>>
>> - <glossentry
>> id='var-BBCLASSEXTEND'><glossterm>BBCLASSEXTEND</glossterm>
>> + <glossentry
>> id='var-bb-BBCLASSEXTEND'><glossterm>BBCLASSEXTEND</glossterm>
>> <glossdef> <para>
>> Allows you to extend a recipe so that it builds
>> variants @@ -1009,7 +1009,7 @@
>> <filename>_class-native</filename>.
>> For example, to generate a native version of
>> a recipe, a
>> - <link
>> linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
>> + <link
>> linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link> on "foo"
>> is rewritten to a <filename>DEPENDS</filename> on "foo-native".
>> </para>
>> @@ -1028,7 +1028,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-BBDEBUG'><glossterm>BBDEBUG</glossterm>
>> + <glossentry
>> id='var-bb-BBDEBUG'><glossterm>BBDEBUG</glossterm> <glossdef>
>> <para>
>> Sets the BitBake debug output level to a
>> specific value @@ -1042,7 +1042,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BBFILE_COLLECTIONS'><glossterm>BBFILE_COLLECTIONS</glossterm>
>> + <glossentry
>> id='var-bb-BBFILE_COLLECTIONS'><glossterm>BBFILE_COLLECTIONS</glossterm>
>> <glossdef> <para>Lists the names of configured layers.
>> These names are used to find the other
>> <filename>BBFILE_*</filename> @@ -1053,10 +1053,10 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BBFILE_PATTERN'><glossterm>BBFILE_PATTERN</glossterm>
>> + <glossentry
>> id='var-bb-BBFILE_PATTERN'><glossterm>BBFILE_PATTERN</glossterm>
>> <glossdef> <para>Variable that expands to match files from
>> - <link
>> linkend='var-BBFILES'><filename>BBFILES</filename></link>
>> + <link
>> linkend='var-bb-BBFILES'><filename>BBFILES</filename></link> in a
>> particular layer. This variable is used in the
>> <filename>conf/layer.conf</filename> file and must be suffixed with
>> the name of the specific layer (e.g. @@ -1064,7 +1064,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BBFILE_PRIORITY'><glossterm>BBFILE_PRIORITY</glossterm>
>> + <glossentry
>> id='var-bb-BBFILE_PRIORITY'><glossterm>BBFILE_PRIORITY</glossterm>
>> <glossdef> <para>Assigns the priority for recipe files in each
>> layer.</para> <para>This variable is useful in situations where the
>> same recipe appears in @@ -1074,7 +1074,7 @@
>> letting you control the precedence for the
>> multiple layers. The precedence established through this variable
>> stands regardless of a recipe's version
>> - (<link
>> linkend='var-PV'><filename>PV</filename></link> variable).
>> + (<link
>> linkend='var-bb-PV'><filename>PV</filename></link> variable). For
>> example, a layer that has a recipe with a higher
>> <filename>PV</filename> value but for which the
>> <filename>BBFILE_PRIORITY</filename> is set to have a lower
>> precedence still has a lower precedence.</para> @@ -1083,7 +1083,7 @@
>> For example, the value 6 has a higher precedence than the value 5. If
>> not specified, the <filename>BBFILE_PRIORITY</filename> variable is
>> set based on layer dependencies (see the
>> - <filename><link
>> linkend='var-LAYERDEPENDS'>LAYERDEPENDS</link></filename> variable for
>> + <filename><link
>> linkend='var-bb-LAYERDEPENDS'>LAYERDEPENDS</link></filename> variable
>> for more information. The default priority, if unspecified
>> for a layer with no dependencies, is the lowest
>> defined priority + 1 @@ -1095,7 +1095,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-BBFILES'><glossterm>BBFILES</glossterm>
>> + <glossentry
>> id='var-bb-BBFILES'><glossterm>BBFILES</glossterm> <glossdef>
>> <para>
>> A space-separated list of recipe files BitBake
>> uses to @@ -1113,7 +1113,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BBINCLUDED'><glossterm>BBINCLUDED</glossterm>
>> + <glossentry
>> id='var-bb-BBINCLUDED'><glossterm>BBINCLUDED</glossterm> <glossdef>
>> <para>
>> Contains a space-separated list of all of all
>> files that @@ -1123,7 +1123,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BBINCLUDELOGS'><glossterm>BBINCLUDELOGS</glossterm>
>> + <glossentry
>> id='var-bb-BBINCLUDELOGS'><glossterm>BBINCLUDELOGS</glossterm>
>> <glossdef> <para>
>> If set to a value, enables printing the task log
>> when @@ -1132,11 +1132,11 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BBINCLUDELOGS_LINES'><glossterm>BBINCLUDELOGS_LINES</glossterm>
>> + <glossentry
>> id='var-bb-BBINCLUDELOGS_LINES'><glossterm>BBINCLUDELOGS_LINES</glossterm>
>> <glossdef> <para>
>> If
>> - <link
>> linkend='var-BBINCLUDELOGS'><filename>BBINCLUDELOGS</filename></link>
>> + <link
>> linkend='var-bb-BBINCLUDELOGS'><filename>BBINCLUDELOGS</filename></link>
>> is set, specifies the maximum number of lines from the task log file
>> to print when reporting a failed task. If you do not set
>> <filename>BBINCLUDELOGS_LINES</filename>, @@ -1145,7 +1145,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-BBLAYERS'><glossterm>BBLAYERS</glossterm>
>> + <glossentry
>> id='var-bb-BBLAYERS'><glossterm>BBLAYERS</glossterm> <glossdef>
>> <para>Lists the layers to enable during the build.
>> This variable is defined in the
>> <filename>bblayers.conf</filename> configuration @@ -1166,7 +1166,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BBLAYERS_FETCH_DIR'><glossterm>BBLAYERS_FETCH_DIR</glossterm>
>> + <glossentry
>> id='var-bb-BBLAYERS_FETCH_DIR'><glossterm>BBLAYERS_FETCH_DIR</glossterm>
>> <glossdef> <para>
>> Sets the base location where layers are stored.
>> @@ -1178,7 +1178,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-BBMASK'><glossterm>BBMASK</glossterm>
>> + <glossentry id='var-bb-BBMASK'><glossterm>BBMASK</glossterm>
>> <glossdef>
>> <para>
>> Prevents BitBake from processing recipes and
>> recipe @@ -1236,7 +1236,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BBMULTICONFIG'><glossterm>BBMULTICONFIG</glossterm>
>> + <glossentry
>> id='var-bb-BBMULTICONFIG'><glossterm>BBMULTICONFIG</glossterm> <info>
>> BBMULTICONFIG[doc] = "Enables BitBake to perform
>> multiple configuration builds and lists each separate configuration
>> (multiconfig)." </info> @@ -1275,7 +1275,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-BBPATH'><glossterm>BBPATH</glossterm>
>> + <glossentry id='var-bb-BBPATH'><glossterm>BBPATH</glossterm>
>> <glossdef>
>> <para>
>> Used by BitBake to locate class
>> @@ -1302,7 +1302,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-BBSERVER'><glossterm>BBSERVER</glossterm>
>> + <glossentry
>> id='var-bb-BBSERVER'><glossterm>BBSERVER</glossterm> <glossdef>
>> <para>
>> Points to the server that runs memory-resident
>> BitBake. @@ -1312,7 +1312,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BBTARGETS'><glossterm>BBTARGETS</glossterm>
>> + <glossentry
>> id='var-bb-BBTARGETS'><glossterm>BBTARGETS</glossterm> <glossdef>
>> <para>
>> Allows you to use a configuration file to add to
>> the list @@ -1321,14 +1321,14 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BBVERSIONS'><glossterm>BBVERSIONS</glossterm>
>> + <glossentry
>> id='var-bb-BBVERSIONS'><glossterm>BBVERSIONS</glossterm> <glossdef>
>> <para>
>> Allows a single recipe to build multiple
>> versions of a project from a single recipe file.
>> You also able to specify conditional metadata
>> using the
>> - <link
>> linkend='var-OVERRIDES'><filename>OVERRIDES</filename></link>
>> + <link
>> linkend='var-bb-OVERRIDES'><filename>OVERRIDES</filename></link>
>> mechanism for a single version or for an optionally named range of
>> versions. </para>
>> @@ -1342,7 +1342,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BITBAKE_UI'><glossterm>BITBAKE_UI</glossterm>
>> + <glossentry
>> id='var-bb-BITBAKE_UI'><glossterm>BITBAKE_UI</glossterm> <glossdef>
>> <para>
>> Used to specify the UI module to use when
>> running BitBake. @@ -1356,7 +1356,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-BUILDNAME'><glossterm>BUILDNAME</glossterm>
>> + <glossentry
>> id='var-bb-BUILDNAME'><glossterm>BUILDNAME</glossterm> <glossdef>
>> <para>
>> A name assigned to the build.
>> @@ -1366,7 +1366,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-BZRDIR'><glossterm>BZRDIR</glossterm>
>> + <glossentry id='var-bb-BZRDIR'><glossterm>BZRDIR</glossterm>
>> <glossdef>
>> <para>
>> The directory in which files checked out of a
>> Bazaar @@ -1377,9 +1377,9 @@
>>
>> </glossdiv>
>>
>> - <glossdiv id='var-glossary-c'><title>C</title>
>> + <glossdiv id='var-bb-glossary-c'><title>C</title>
>>
>> - <glossentry id='var-CACHE'><glossterm>CACHE</glossterm>
>> + <glossentry id='var-bb-CACHE'><glossterm>CACHE</glossterm>
>> <glossdef>
>> <para>
>> Specifies the directory BitBake uses to store a
>> cache @@ -1389,7 +1389,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-CVSDIR'><glossterm>CVSDIR</glossterm>
>> + <glossentry id='var-bb-CVSDIR'><glossterm>CVSDIR</glossterm>
>> <glossdef>
>> <para>
>> The directory in which files checked out under
>> the @@ -1400,9 +1400,9 @@
>>
>> </glossdiv>
>>
>> - <glossdiv id='var-glossary-d'><title>D</title>
>> + <glossdiv id='var-bb-glossary-d'><title>D</title>
>>
>> - <glossentry
>> id='var-DEFAULT_PREFERENCE'><glossterm>DEFAULT_PREFERENCE</glossterm>
>> + <glossentry
>> id='var-bb-DEFAULT_PREFERENCE'><glossterm>DEFAULT_PREFERENCE</glossterm>
>> <glossdef> <para>
>> Specifies a weak bias for recipe selection
>> priority. @@ -1413,20 +1413,20 @@
>> piece of software.
>> Using the variable in this way causes the stable
>> version of the recipe to build by default in the absence of
>> - <filename><link
>> linkend='var-PREFERRED_VERSION'>PREFERRED_VERSION</link></filename>
>> + <filename><link
>> linkend='var-bb-PREFERRED_VERSION'>PREFERRED_VERSION</link></filename>
>> being used to build the development version. </para>
>> <note>
>> The bias provided by
>> <filename>DEFAULT_PREFERENCE</filename> is weak and is overridden by
>> - <filename><link
>> linkend='var-BBFILE_PRIORITY'>BBFILE_PRIORITY</link></filename>
>> + <filename><link
>> linkend='var-bb-BBFILE_PRIORITY'>BBFILE_PRIORITY</link></filename> if
>> that variable is different between two layers that contain different
>> versions of the same recipe. </note>
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-DEPENDS'><glossterm>DEPENDS</glossterm>
>> + <glossentry
>> id='var-bb-DEPENDS'><glossterm>DEPENDS</glossterm> <glossdef>
>> <para>
>> Lists a recipe's build-time dependencies
>> @@ -1451,13 +1451,13 @@
>>
>> <para>
>> For information on runtime dependencies, see the
>> - <link
>> linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link>
>> + <link
>> linkend='var-bb-RDEPENDS'><filename>RDEPENDS</filename></link>
>> variable. </para>
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-DESCRIPTION'><glossterm>DESCRIPTION</glossterm>
>> + <glossentry
>> id='var-bb-DESCRIPTION'><glossterm>DESCRIPTION</glossterm> <glossdef>
>> <para>
>> A long description for the recipe.
>> @@ -1465,7 +1465,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-DL_DIR'><glossterm>DL_DIR</glossterm>
>> + <glossentry id='var-bb-DL_DIR'><glossterm>DL_DIR</glossterm>
>> <glossdef>
>> <para>
>> The central download directory used by the build
>> process to @@ -1474,7 +1474,7 @@
>> suitable for mirroring for everything except Git
>> repositories.
>> If you want tarballs of Git repositories, use the
>> - <link
>> linkend='var-BB_GENERATE_MIRROR_TARBALLS'><filename>BB_GENERATE_MIRROR_TARBALLS</filename></link>
>> + <link
>> linkend='var-bb-BB_GENERATE_MIRROR_TARBALLS'><filename>BB_GENERATE_MIRROR_TARBALLS</filename></link>
>> variable. </para>
>> </glossdef>
>> @@ -1482,9 +1482,9 @@
>> </glossentry>
>> </glossdiv>
>>
>> - <glossdiv id='var-glossary-e'><title>E</title>
>> + <glossdiv id='var-bb-glossary-e'><title>E</title>
>>
>> - <glossentry
>> id='var-EXCLUDE_FROM_WORLD'><glossterm>EXCLUDE_FROM_WORLD</glossterm>
>> + <glossentry
>> id='var-bb-EXCLUDE_FROM_WORLD'><glossterm>EXCLUDE_FROM_WORLD</glossterm>
>> <glossdef> <para>
>> Directs BitBake to exclude a recipe from world
>> builds (i.e. @@ -1512,9 +1512,9 @@
>>
>> </glossdiv>
>>
>> - <glossdiv id='var-glossary-f'><title>F</title>
>> + <glossdiv id='var-bb-glossary-f'><title>F</title>
>>
>> - <glossentry id='var-FAKEROOT'><glossterm>FAKEROOT</glossterm>
>> + <glossentry
>> id='var-bb-FAKEROOT'><glossterm>FAKEROOT</glossterm> <glossdef>
>> <para>
>> Contains the command to use when running a
>> shell script @@ -1527,19 +1527,19 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-FAKEROOTBASEENV'><glossterm>FAKEROOTBASEENV</glossterm>
>> + <glossentry
>> id='var-bb-FAKEROOTBASEENV'><glossterm>FAKEROOTBASEENV</glossterm>
>> <glossdef> <para>
>> Lists environment variables to set when
>> executing the command defined by
>> - <link
>> linkend='var-FAKEROOTCMD'><filename>FAKEROOTCMD</filename></link>
>> + <link
>> linkend='var-bb-FAKEROOTCMD'><filename>FAKEROOTCMD</filename></link>
>> that starts the bitbake-worker process in the fakeroot environment.
>> </para>
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-FAKEROOTCMD'><glossterm>FAKEROOTCMD</glossterm>
>> + <glossentry
>> id='var-bb-FAKEROOTCMD'><glossterm>FAKEROOTCMD</glossterm> <glossdef>
>> <para>
>> Contains the command that starts the
>> bitbake-worker @@ -1548,7 +1548,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-FAKEROOTDIRS'><glossterm>FAKEROOTDIRS</glossterm>
>> + <glossentry
>> id='var-bb-FAKEROOTDIRS'><glossterm>FAKEROOTDIRS</glossterm>
>> <glossdef> <para>
>> Lists directories to create before running a
>> task in @@ -1557,33 +1557,33 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-FAKEROOTENV'><glossterm>FAKEROOTENV</glossterm>
>> + <glossentry
>> id='var-bb-FAKEROOTENV'><glossterm>FAKEROOTENV</glossterm> <glossdef>
>> <para>
>> Lists environment variables to set when running
>> a task in the fakeroot environment.
>> For additional information on environment
>> variables and the fakeroot environment, see the
>> - <link
>> linkend='var-FAKEROOTBASEENV'><filename>FAKEROOTBASEENV</filename></link>
>> + <link
>> linkend='var-bb-FAKEROOTBASEENV'><filename>FAKEROOTBASEENV</filename></link>
>> variable. </para>
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-FAKEROOTNOENV'><glossterm>FAKEROOTNOENV</glossterm>
>> + <glossentry
>> id='var-bb-FAKEROOTNOENV'><glossterm>FAKEROOTNOENV</glossterm>
>> <glossdef> <para>
>> Lists environment variables to set when running
>> a task that is not in the fakeroot environment.
>> For additional information on environment
>> variables and the fakeroot environment, see the
>> - <link
>> linkend='var-FAKEROOTENV'><filename>FAKEROOTENV</filename></link>
>> + <link
>> linkend='var-bb-FAKEROOTENV'><filename>FAKEROOTENV</filename></link>
>> variable. </para>
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-FETCHCMD'><glossterm>FETCHCMD</glossterm>
>> + <glossentry
>> id='var-bb-FETCHCMD'><glossterm>FETCHCMD</glossterm> <glossdef>
>> <para>
>> Defines the command the BitBake fetcher module
>> @@ -1595,7 +1595,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-FILE'><glossterm>FILE</glossterm>
>> + <glossentry id='var-bb-FILE'><glossterm>FILE</glossterm>
>> <glossdef>
>> <para>
>> Points at the current file.
>> @@ -1607,7 +1607,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-FILESPATH'><glossterm>FILESPATH</glossterm>
>> + <glossentry
>> id='var-bb-FILESPATH'><glossterm>FILESPATH</glossterm> <glossdef>
>> <para>
>> Specifies directories BitBake uses when
>> searching for @@ -1625,9 +1625,9 @@
>> </glossdiv>
>>
>>
>> - <glossdiv id='var-glossary-g'><title>G</title>
>> + <glossdiv id='var-bb-glossary-g'><title>G</title>
>>
>> - <glossentry id='var-GITDIR'><glossterm>GITDIR</glossterm>
>> + <glossentry id='var-bb-GITDIR'><glossterm>GITDIR</glossterm>
>> <glossdef>
>> <para>
>> The directory in which a local copy of a Git
>> repository @@ -1639,9 +1639,9 @@
>> </glossdiv>
>>
>>
>> - <glossdiv id='var-glossary-h'><title>H</title>
>> + <glossdiv id='var-bb-glossary-h'><title>H</title>
>>
>> - <glossentry id='var-HGDIR'><glossterm>HGDIR</glossterm>
>> + <glossentry id='var-bb-HGDIR'><glossterm>HGDIR</glossterm>
>> <glossdef>
>> <para>
>> The directory in which files checked out of a
>> Mercurial @@ -1650,7 +1650,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-HOMEPAGE'><glossterm>HOMEPAGE</glossterm>
>> + <glossentry
>> id='var-bb-HOMEPAGE'><glossterm>HOMEPAGE</glossterm> <glossdef>
>> <para>Website where more information about the
>> software the recipe is building can be found.</para>
>> @@ -1659,9 +1659,9 @@
>>
>> </glossdiv>
>>
>> - <glossdiv id='var-glossary-i'><title>I</title>
>> + <glossdiv id='var-bb-glossary-i'><title>I</title>
>>
>> - <glossentry id='var-INHERIT'><glossterm>INHERIT</glossterm>
>> + <glossentry
>> id='var-bb-INHERIT'><glossterm>INHERIT</glossterm> <glossdef>
>> <para>
>> Causes the named class or classes to be
>> inherited globally. @@ -1691,15 +1691,15 @@
>> </glossdiv>
>> -->
>>
>> - <glossdiv id='var-glossary-l'><title>L</title>
>> + <glossdiv id='var-bb-glossary-l'><title>L</title>
>>
>> - <glossentry
>> id='var-LAYERDEPENDS'><glossterm>LAYERDEPENDS</glossterm>
>> + <glossentry
>> id='var-bb-LAYERDEPENDS'><glossterm>LAYERDEPENDS</glossterm>
>> <glossdef> <para>Lists the layers, separated by spaces, upon which
>> this recipe depends. Optionally, you can specify a specific layer
>> version for a dependency by adding it to the end of the layer name
>> with a colon, (e.g. "anotherlayer:3" to be compared against
>> - <link
>> linkend='var-LAYERVERSION'><filename>LAYERVERSION</filename></link><filename>_anotherlayer</filename>
>> + <link
>> linkend='var-bb-LAYERVERSION'><filename>LAYERVERSION</filename></link><filename>_anotherlayer</filename>
>> in this case). BitBake produces an error if any dependency is missing
>> or the version numbers do not match exactly (if specified).</para>
>> @@ -1710,7 +1710,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-LAYERDIR'><glossterm>LAYERDIR</glossterm>
>> + <glossentry
>> id='var-bb-LAYERDIR'><glossterm>LAYERDIR</glossterm> <glossdef>
>> <para>When used inside the
>> <filename>layer.conf</filename> configuration file, this variable
>> provides the path of the current layer. @@ -1719,22 +1719,22 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-LAYERDIR_RE'><glossterm>LAYERDIR_RE</glossterm>
>> + <glossentry
>> id='var-bb-LAYERDIR_RE'><glossterm>LAYERDIR_RE</glossterm> <glossdef>
>> <para>When used inside the
>> <filename>layer.conf</filename> configuration file, this variable
>> provides the path of the current layer, escaped for use in a regular
>> expression
>> - (<link
>> linkend='var-BBFILE_PATTERN'><filename>BBFILE_PATTERN</filename></link>).
>> + (<link
>> linkend='var-bb-BBFILE_PATTERN'><filename>BBFILE_PATTERN</filename></link>).
>> This variable is not available outside of
>> <filename>layer.conf</filename> and references are expanded
>> immediately when parsing of the file completes.</para> </glossdef>
>> </glossentry>
>> - <glossentry
>> id='var-LAYERVERSION'><glossterm>LAYERVERSION</glossterm>
>> + <glossentry
>> id='var-bb-LAYERVERSION'><glossterm>LAYERVERSION</glossterm>
>> <glossdef> <para>Optionally specifies the version of a layer as a
>> single number. You can use this variable within
>> - <link
>> linkend='var-LAYERDEPENDS'><filename>LAYERDEPENDS</filename></link>
>> + <link
>> linkend='var-bb-LAYERDEPENDS'><filename>LAYERDEPENDS</filename></link>
>> for another layer in order to depend on a specific version of the
>> layer.</para> <para>
>> @@ -1744,7 +1744,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-LICENSE'><glossterm>LICENSE</glossterm>
>> + <glossentry
>> id='var-bb-LICENSE'><glossterm>LICENSE</glossterm> <glossdef>
>> <para>
>> The list of source licenses for the recipe.
>> @@ -1754,9 +1754,9 @@
>>
>> </glossdiv>
>>
>> - <glossdiv id='var-glossary-m'><title>M</title>
>> + <glossdiv id='var-bb-glossary-m'><title>M</title>
>>
>> - <glossentry id='var-MIRRORS'><glossterm>MIRRORS</glossterm>
>> + <glossentry
>> id='var-bb-MIRRORS'><glossterm>MIRRORS</glossterm> <glossdef>
>> <para>
>> Specifies additional paths from which BitBake
>> gets source code. @@ -1764,14 +1764,14 @@
>> tries the local download directory.
>> If that location fails, the build system tries
>> locations defined by
>> - <link
>> linkend='var-PREMIRRORS'><filename>PREMIRRORS</filename></link>,
>> + <link
>> linkend='var-bb-PREMIRRORS'><filename>PREMIRRORS</filename></link>,
>> the upstream source, and then locations specified by
>> <filename>MIRRORS</filename> in that order. </para>
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-MULTI_PROVIDER_WHITELIST'><glossterm>MULTI_PROVIDER_WHITELIST</glossterm>
>> + <glossentry
>> id='var-bb-MULTI_PROVIDER_WHITELIST'><glossterm>MULTI_PROVIDER_WHITELIST</glossterm>
>> <glossdef> <para>
>> Allows you to suppress BitBake warnings caused
>> when @@ -1804,9 +1804,9 @@
>> </glossdiv>
>> -->
>>
>> - <glossdiv id='var-glossary-o'><title>O</title>
>> + <glossdiv id='var-bb-glossary-o'><title>O</title>
>>
>> - <glossentry
>> id='var-OVERRIDES'><glossterm>OVERRIDES</glossterm>
>> + <glossentry
>> id='var-bb-OVERRIDES'><glossterm>OVERRIDES</glossterm> <glossdef>
>> <para>
>> BitBake uses <filename>OVERRIDES</filename> to
>> control @@ -1829,9 +1829,9 @@
>> </glossentry>
>> </glossdiv>
>>
>> - <glossdiv id='var-glossary-p'><title>P</title>
>> + <glossdiv id='var-bb-glossary-p'><title>P</title>
>>
>> - <glossentry id='var-P4DIR'><glossterm>P4DIR</glossterm>
>> + <glossentry id='var-bb-P4DIR'><glossterm>P4DIR</glossterm>
>> <glossdef>
>> <para>
>> The directory in which a local copy of a
>> Perforce depot @@ -1840,14 +1840,14 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-PACKAGES'><glossterm>PACKAGES</glossterm>
>> + <glossentry
>> id='var-bb-PACKAGES'><glossterm>PACKAGES</glossterm> <glossdef>
>> <para>The list of packages the recipe creates.
>> </para>
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-PACKAGES_DYNAMIC'><glossterm>PACKAGES_DYNAMIC</glossterm>
>> + <glossentry
>> id='var-bb-PACKAGES_DYNAMIC'><glossterm>PACKAGES_DYNAMIC</glossterm>
>> <glossdef> <para>
>> A promise that your recipe satisfies runtime
>> dependencies @@ -1856,7 +1856,7 @@
>> does not actually satisfy the dependencies, it
>> only states that they should be satisfied.
>> For example, if a hard, runtime dependency
>> - (<link
>> linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link>)
>> + (<link
>> linkend='var-bb-RDEPENDS'><filename>RDEPENDS</filename></link>) of
>> another package is satisfied during the build through the
>> <filename>PACKAGES_DYNAMIC</filename> variable, but a package with
>> the module name is never actually @@ -1865,7 +1865,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-PE'><glossterm>PE</glossterm>
>> + <glossentry id='var-bb-PE'><glossterm>PE</glossterm>
>> <glossdef>
>> <para>
>> The epoch of the recipe.
>> @@ -1877,7 +1877,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-PERSISTENT_DIR'><glossterm>PERSISTENT_DIR</glossterm>
>> + <glossentry
>> id='var-bb-PERSISTENT_DIR'><glossterm>PERSISTENT_DIR</glossterm>
>> <glossdef> <para>
>> Specifies the directory BitBake uses to store
>> data that @@ -1889,7 +1889,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-PF'><glossterm>PF</glossterm>
>> + <glossentry id='var-bb-PF'><glossterm>PF</glossterm>
>> <glossdef>
>> <para>
>> Specifies the recipe or package name and
>> includes all version and revision @@ -1899,27 +1899,27 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-PN'><glossterm>PN</glossterm>
>> + <glossentry id='var-bb-PN'><glossterm>PN</glossterm>
>> <glossdef>
>> <para>The recipe name.</para>
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-PR'><glossterm>PR</glossterm>
>> + <glossentry id='var-bb-PR'><glossterm>PR</glossterm>
>> <glossdef>
>> <para>The revision of the recipe.
>> </para>
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-PREFERRED_PROVIDER'><glossterm>PREFERRED_PROVIDER</glossterm>
>> + <glossentry
>> id='var-bb-PREFERRED_PROVIDER'><glossterm>PREFERRED_PROVIDER</glossterm>
>> <glossdef> <para>
>> Determines which recipe should be given
>> preference when multiple recipes provide the same item.
>> You should always suffix the variable with the
>> name of the provided item, and you should set it to the
>> - <link
>> linkend='var-PN'><filename>PN</filename></link>
>> + <link
>> linkend='var-bb-PN'><filename>PN</filename></link> of the recipe to
>> which you want to give precedence. Some examples:
>> <literallayout class='monospaced'>
>> @@ -1931,14 +1931,14 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-PREFERRED_PROVIDERS'><glossterm>PREFERRED_PROVIDERS</glossterm>
>> + <glossentry
>> id='var-bb-PREFERRED_PROVIDERS'><glossterm>PREFERRED_PROVIDERS</glossterm>
>> <glossdef> <para>
>> Determines which recipe should be given
>> preference for cases where multiple recipes provide the same item.
>> Functionally,
>> <filename>PREFERRED_PROVIDERS</filename> is
>> identical to
>> - <link
>> linkend='var-PREFERRED_PROVIDER'><filename>PREFERRED_PROVIDER</filename></link>.
>> + <link
>> linkend='var-bb-PREFERRED_PROVIDER'><filename>PREFERRED_PROVIDER</filename></link>.
>> However, the <filename>PREFERRED_PROVIDERS</filename> variable lets
>> you define preferences for multiple situations using the following
>> form: @@ -1954,15 +1954,15 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-PREFERRED_VERSION'><glossterm>PREFERRED_VERSION</glossterm>
>> + <glossentry
>> id='var-bb-PREFERRED_VERSION'><glossterm>PREFERRED_VERSION</glossterm>
>> <glossdef> <para>
>> If there are multiple versions of recipes
>> available, this variable determines which recipe should be given
>> preference. You must always suffix the variable with the
>> - <link
>> linkend='var-PN'><filename>PN</filename></link>
>> + <link
>> linkend='var-bb-PN'><filename>PN</filename></link> you want to
>> select, and you should set
>> - <link
>> linkend='var-PV'><filename>PV</filename></link>
>> + <link
>> linkend='var-bb-PV'><filename>PV</filename></link> accordingly for
>> precedence. </para>
>>
>> @@ -1989,7 +1989,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-PREMIRRORS'><glossterm>PREMIRRORS</glossterm>
>> + <glossentry
>> id='var-bb-PREMIRRORS'><glossterm>PREMIRRORS</glossterm> <glossdef>
>> <para>
>> Specifies additional paths from which BitBake
>> gets source code. @@ -1998,7 +1998,7 @@
>> If that location fails, the build system tries
>> locations defined by <filename>PREMIRRORS</filename>, the upstream
>> source, and then locations specified by
>> - <link
>> linkend='var-MIRRORS'><filename>MIRRORS</filename></link>
>> + <link
>> linkend='var-bb-MIRRORS'><filename>MIRRORS</filename></link> in that
>> order. </para>
>>
>> @@ -2022,20 +2022,20 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-PROVIDES'><glossterm>PROVIDES</glossterm>
>> + <glossentry
>> id='var-bb-PROVIDES'><glossterm>PROVIDES</glossterm> <glossdef>
>> <para>
>> A list of aliases by which a particular recipe
>> can be known.
>> By default, a recipe's own
>> - <filename><link
>> linkend='var-PN'>PN</link></filename>
>> + <filename><link
>> linkend='var-bb-PN'>PN</link></filename> is implicitly already in its
>> <filename>PROVIDES</filename> list.
>> If a recipe uses <filename>PROVIDES</filename>,
>> the additional aliases are synonyms for the recipe and can
>> be useful satisfying dependencies of other
>> recipes during the build as specified by
>> - <filename><link
>> linkend='var-DEPENDS'>DEPENDS</link></filename>.
>> + <filename><link
>> linkend='var-bb-DEPENDS'>DEPENDS</link></filename>. </para>
>>
>> <para>
>> @@ -2059,7 +2059,7 @@
>> virtual target in <filename>PROVIDES</filename>.
>> Recipes that depend on the functionality in
>> question can include the virtual target in
>> - <link
>> linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
>> + <link
>> linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link> to leave
>> the choice of provider open. </para>
>>
>> @@ -2072,11 +2072,11 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-PRSERV_HOST'><glossterm>PRSERV_HOST</glossterm>
>> + <glossentry
>> id='var-bb-PRSERV_HOST'><glossterm>PRSERV_HOST</glossterm> <glossdef>
>> <para>
>> The network based
>> - <link
>> linkend='var-PR'><filename>PR</filename></link>
>> + <link
>> linkend='var-bb-PR'><filename>PR</filename></link> service host and
>> port. </para>
>>
>> @@ -2094,7 +2094,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-PV'><glossterm>PV</glossterm>
>> + <glossentry id='var-bb-PV'><glossterm>PV</glossterm>
>> <glossdef>
>> <para>The version of the recipe.
>> </para>
>> @@ -2108,9 +2108,9 @@
>> </glossdiv>
>> -->
>>
>> - <glossdiv id='var-glossary-r'><title>R</title>
>> + <glossdiv id='var-bb-glossary-r'><title>R</title>
>>
>> - <glossentry id='var-RDEPENDS'><glossterm>RDEPENDS</glossterm>
>> + <glossentry
>> id='var-bb-RDEPENDS'><glossterm>RDEPENDS</glossterm> <glossdef>
>> <para>
>> Lists a package's runtime dependencies (i.e.
>> other packages) @@ -2165,13 +2165,13 @@
>>
>> <para>
>> For information on build-time dependencies, see
>> the
>> - <link
>> linkend='var-DEPENDS'><filename>DEPENDS</filename></link>
>> + <link
>> linkend='var-bb-DEPENDS'><filename>DEPENDS</filename></link> variable.
>> </para>
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-REPODIR'><glossterm>REPODIR</glossterm>
>> + <glossentry
>> id='var-bb-REPODIR'><glossterm>REPODIR</glossterm> <glossdef>
>> <para>
>> The directory in which a local copy of a
>> @@ -2181,14 +2181,14 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-RPROVIDES'><glossterm>RPROVIDES</glossterm>
>> + <glossentry
>> id='var-bb-RPROVIDES'><glossterm>RPROVIDES</glossterm> <glossdef>
>> <para>
>> A list of package name aliases that a package
>> also provides. These aliases are useful for satisfying runtime
>> dependencies of other packages both during the build and on the target
>> (as specified by
>> - <filename><link
>> linkend='var-RDEPENDS'>RDEPENDS</link></filename>).
>> + <filename><link
>> linkend='var-bb-RDEPENDS'>RDEPENDS</link></filename>). </para>
>> <para>
>> As with all package-controlling variables, you
>> must always @@ -2201,7 +2201,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-RRECOMMENDS'><glossterm>RRECOMMENDS</glossterm>
>> + <glossentry
>> id='var-bb-RRECOMMENDS'><glossterm>RRECOMMENDS</glossterm> <glossdef>
>> <para>
>> A list of packages that extends the usability of
>> a package @@ -2210,7 +2210,7 @@
>> packages in order to successfully build, but
>> needs them for the extended usability.
>> To specify runtime dependencies for packages,
>> see the
>> - <filename><link
>> linkend='var-RDEPENDS'>RDEPENDS</link></filename>
>> + <filename><link
>> linkend='var-bb-RDEPENDS'>RDEPENDS</link></filename> variable.
>> </para>
>>
>> @@ -2243,15 +2243,15 @@
>>
>> </glossdiv>
>>
>> - <glossdiv id='var-glossary-s'><title>S</title>
>> + <glossdiv id='var-bb-glossary-s'><title>S</title>
>>
>> - <glossentry id='var-SECTION'><glossterm>SECTION</glossterm>
>> + <glossentry
>> id='var-bb-SECTION'><glossterm>SECTION</glossterm> <glossdef>
>> <para>The section in which packages should be
>> categorized.</para> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-SRC_URI'><glossterm>SRC_URI</glossterm>
>> + <glossentry
>> id='var-bb-SRC_URI'><glossterm>SRC_URI</glossterm> <glossdef>
>> <para>
>> The list of source files - local or remote.
>> @@ -2272,7 +2272,7 @@
>> the metadata,
>> from the local machine.
>> The path is relative to the
>> - <link
>> linkend='var-FILESPATH'><filename>FILESPATH</filename></link>
>> + <link
>> linkend='var-bb-FILESPATH'><filename>FILESPATH</filename></link>
>> variable.</para></listitem>
>> <listitem><para><emphasis><filename>bzr://</filename> -</emphasis>
>> Fetches files from a Bazaar revision control
>> repository.</para></listitem> @@ -2322,7 +2322,7 @@ </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-SRCDATE'><glossterm>SRCDATE</glossterm>
>> + <glossentry
>> id='var-bb-SRCDATE'><glossterm>SRCDATE</glossterm> <glossdef>
>> <para>
>> The date of the source code used to build the
>> package. @@ -2331,7 +2331,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-SRCREV'><glossterm>SRCREV</glossterm>
>> + <glossentry id='var-bb-SRCREV'><glossterm>SRCREV</glossterm>
>> <glossdef>
>> <para>
>> The revision of the source code used to build
>> the package. @@ -2344,13 +2344,13 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-SRCREV_FORMAT'><glossterm>SRCREV_FORMAT</glossterm>
>> + <glossentry
>> id='var-bb-SRCREV_FORMAT'><glossterm>SRCREV_FORMAT</glossterm>
>> <glossdef> <para>
>> Helps construct valid
>> - <link
>> linkend='var-SRCREV'><filename>SRCREV</filename></link>
>> + <link
>> linkend='var-bb-SRCREV'><filename>SRCREV</filename></link> values
>> when multiple source controlled URLs are used in
>> - <link
>> linkend='var-SRC_URI'><filename>SRC_URI</filename></link>.
>> + <link
>> linkend='var-bb-SRC_URI'><filename>SRC_URI</filename></link>. </para>
>>
>> <para>
>> @@ -2371,7 +2371,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-STAMP'><glossterm>STAMP</glossterm>
>> + <glossentry id='var-bb-STAMP'><glossterm>STAMP</glossterm>
>> <glossdef>
>> <para>
>> Specifies the base path used to create recipe
>> stamp files. @@ -2381,12 +2381,12 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry
>> id='var-STAMPCLEAN'><glossterm>STAMPCLEAN</glossterm>
>> + <glossentry
>> id='var-bb-STAMPCLEAN'><glossterm>STAMPCLEAN</glossterm> <glossdef>
>> <para>
>> Specifies the base path used to create recipe
>> stamp files. Unlike the
>> - <link
>> linkend='var-STAMP'><filename>STAMP</filename></link>
>> + <link
>> linkend='var-bb-STAMP'><filename>STAMP</filename></link> variable,
>> <filename>STAMPCLEAN</filename> can contain wildcards to match the
>> range of files a clean operation should remove.
>> @@ -2396,7 +2396,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-SUMMARY'><glossterm>SUMMARY</glossterm>
>> + <glossentry
>> id='var-bb-SUMMARY'><glossterm>SUMMARY</glossterm> <glossdef>
>> <para>
>> A short summary for the recipe, which is 72
>> characters or less. @@ -2404,7 +2404,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-SVNDIR'><glossterm>SVNDIR</glossterm>
>> + <glossentry id='var-bb-SVNDIR'><glossterm>SVNDIR</glossterm>
>> <glossdef>
>> <para>
>> The directory in which files checked out of a
>> Subversion @@ -2415,9 +2415,9 @@
>>
>> </glossdiv>
>>
>> - <glossdiv id='var-glossary-t'><title>T</title>
>> + <glossdiv id='var-bb-glossary-t'><title>T</title>
>>
>> - <glossentry id='var-T'><glossterm>T</glossterm>
>> + <glossentry id='var-bb-T'><glossterm>T</glossterm>
>> <glossdef>
>> <para>Points to a directory were BitBake places
>> temporary files, which consist mostly of task
>> logs and @@ -2426,7 +2426,7 @@
>> </glossdef>
>> </glossentry>
>>
>> - <glossentry id='var-TOPDIR'><glossterm>TOPDIR</glossterm>
>> + <glossentry id='var-bb-TOPDIR'><glossterm>TOPDIR</glossterm>
>> <glossdef>
>> <para>
>> Points to the build directory.
>> diff --git a/bitbake/doc/poky.ent b/bitbake/doc/poky.ent
>> index c032e14..85d9c83 100644
>> --- a/bitbake/doc/poky.ent
>> +++ b/bitbake/doc/poky.ent
>> @@ -17,13 +17,6 @@
>> <!ENTITY OE_DOCS_URL "http://docs.openembedded.org">
>> <!ENTITY OH_HOME_URL "http://o-hand.com">
>> <!ENTITY BITBAKE_HOME_URL
>> "http://developer.berlios.de/projects/bitbake/"> -<!ENTITY
>> ECLIPSE_MAIN_URL "http://www.eclipse.org/downloads"> -<!ENTITY
>> ECLIPSE_DL_URL "http://download.eclipse.org"> -<!ENTITY
>> ECLIPSE_DL_PLUGIN_URL
>> "&YOCTO_DL_URL;/releases/eclipse-plugin/&DISTRO;"> -<!ENTITY
>> ECLIPSE_UPDATES_URL "&ECLIPSE_DL_URL;/tm/updates/3.3"> -<!ENTITY
>> ECLIPSE_INDIGO_URL "&ECLIPSE_DL_URL;/releases/indigo"> -<!ENTITY
>> ECLIPSE_JUNO_URL "&ECLIPSE_DL_URL;/releases/juno"> -<!ENTITY
>> ECLIPSE_INDIGO_CDT_URL "&ECLIPSE_DL_URL;tools/cdt/releases/indigo">
>> <!ENTITY YOCTO_DOCS_URL "&YOCTO_HOME_URL;/docs"> <!ENTITY
>> YOCTO_SOURCES_URL "&YOCTO_HOME_URL;/sources/"> <!ENTITY
>> YOCTO_AB_PORT_URL "&YOCTO_AB_URL;:8010"> @@ -31,7 +24,6 @@ <!ENTITY
>> YOCTO_POKY_URL "&YOCTO_DL_URL;/releases/poky/"> <!ENTITY
>> YOCTO_RELEASE_DL_URL "&YOCTO_DL_URL;/releases/yocto/yocto-&DISTRO;">
>> <!ENTITY YOCTO_TOOLCHAIN_DL_URL "&YOCTO_RELEASE_DL_URL;/toolchain/">
>> -<!ENTITY YOCTO_ECLIPSE_DL_URL
>> "&YOCTO_RELEASE_DL_URL;/eclipse-plugin/indigo;"> <!ENTITY
>> YOCTO_ADTINSTALLER_DL_URL "&YOCTO_RELEASE_DL_URL;/adt_installer">
>> <!ENTITY YOCTO_POKY_DL_URL
>> "&YOCTO_RELEASE_DL_URL;/&YOCTO_POKY;.tar.bz2"> <!ENTITY
>> YOCTO_MACHINES_DL_URL "&YOCTO_RELEASE_DL_URL;/machines"> diff --git
>> a/bitbake/lib/bb/COW.py b/bitbake/lib/bb/COW.py index
>> 7817473..d26e981 100644 --- a/bitbake/lib/bb/COW.py +++
>> b/bitbake/lib/bb/COW.py @@ -1,23 +1,8 @@ -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # This is a copy on write dictionary and set which abuses classes to
>> try and be nice and fast. #
>> # Copyright (C) 2006 Tim Ansell
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. -#
>> #Please Note:
>> # Be careful when using mutable types (ie Dict and Lists) -
>> operations involving these are SLOW. # Assign a file to __warn__ to
>> get warnings about slow operations. diff --git
>> a/bitbake/lib/bb/__init__.py b/bitbake/lib/bb/__init__.py index
>> 4bc47c8..c144311 100644 --- a/bitbake/lib/bb/__init__.py
>> +++ b/bitbake/lib/bb/__init__.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # BitBake Build System Python Library
>> #
>> @@ -8,20 +6,10 @@
>> #
>> # Based on Gentoo's portage.py.
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> -__version__ = "1.40.0"
>> +__version__ = "1.44.0"
>>
>> import sys
>> if sys.version_info < (3, 4, 0):
>> diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py
>> index 3e2a94e..30a2ba2 100644
>> --- a/bitbake/lib/bb/build.py
>> +++ b/bitbake/lib/bb/build.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # BitBake 'Build' implementation
>> #
>> @@ -10,18 +8,7 @@
>> #
>> # Based on Gentoo's portage.py.
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> # Based on functions from the base bb module, Copyright 2003 Holger
>> Schurig
>> @@ -67,23 +54,6 @@ else:
>> builtins['bb'] = bb
>> builtins['os'] = os
>>
>> -class FuncFailed(Exception):
>> - def __init__(self, name = None, logfile = None):
>> - self.logfile = logfile
>> - self.name = name
>> - if name:
>> - self.msg = 'Function failed: %s' % name
>> - else:
>> - self.msg = "Function failed"
>> -
>> - def __str__(self):
>> - if self.logfile and os.path.exists(self.logfile):
>> - msg = ("%s (log file is located at %s)" %
>> - (self.msg, self.logfile))
>> - else:
>> - msg = self.msg
>> - return msg
>> -
>> class TaskBase(event.Event):
>> """Base class for task events"""
>>
>> @@ -176,15 +146,33 @@ class LogTee(object):
>>
>> def __repr__(self):
>> return '<LogTee {0}>'.format(self.name)
>> +
>> def flush(self):
>> self.outfile.flush()
>>
>> -#
>> -# pythonexception allows the python exceptions generated to be raised
>> -# as the real exceptions (not FuncFailed) and without a backtrace at
>> the -# origin of the failure.
>> -#
>> -def exec_func(func, d, dirs = None, pythonexception=False):
>> +
>> +class StdoutNoopContextManager:
>> + """
>> + This class acts like sys.stdout, but adds noop __enter__ and
>> __exit__ methods.
>> + """
>> + def __enter__(self):
>> + return sys.stdout
>> +
>> + def __exit__(self, *exc_info):
>> + pass
>> +
>> + def write(self, string):
>> + return sys.stdout.write(string)
>> +
>> + def flush(self):
>> + sys.stdout.flush()
>> +
>> + @property
>> + def name(self):
>> + return sys.stdout.name
>> +
>> +
>> +def exec_func(func, d, dirs = None):
>> """Execute a BB 'function'"""
>>
>> try:
>> @@ -256,7 +244,7 @@ def exec_func(func, d, dirs = None,
>> pythonexception=False):
>> with bb.utils.fileslocked(lockfiles):
>> if ispython:
>> - exec_func_python(func, d, runfile, cwd=adir,
>> pythonexception=pythonexception)
>> + exec_func_python(func, d, runfile, cwd=adir)
>> else:
>> exec_func_shell(func, d, runfile, cwd=adir)
>>
>> @@ -276,7 +264,7 @@ _functionfmt = """
>> {function}(d)
>> """
>> logformatter = bb.msg.BBLogFormatter("%(levelname)s: %(message)s")
>> -def exec_func_python(func, d, runfile, cwd=None,
>> pythonexception=False): +def exec_func_python(func, d, runfile,
>> cwd=None): """Execute a python BB 'function'"""
>>
>> code = _functionfmt.format(function=func)
>> @@ -301,13 +289,7 @@ def exec_func_python(func, d, runfile, cwd=None,
>> pythonexception=False): bb.methodpool.insert_method(func, text, fn,
>> lineno - 1)
>> comp = utils.better_compile(code, func, "exec_python_func()
>> autogenerated")
>> - utils.better_exec(comp, {"d": d}, code, "exec_python_func()
>> autogenerated", pythonexception=pythonexception)
>> - except (bb.parse.SkipRecipe, bb.build.FuncFailed):
>> - raise
>> - except:
>> - if pythonexception:
>> - raise
>> - raise FuncFailed(func, None)
>> + utils.better_exec(comp, {"d": d}, code, "exec_python_func()
>> autogenerated") finally:
>> bb.debug(2, "Python function %s finished" % func)
>>
>> @@ -335,6 +317,42 @@ trap 'bb_exit_handler' 0
>> set -e
>> '''
>>
>> +def create_progress_handler(func, progress, logfile, d):
>> + if progress == 'percent':
>> + # Use default regex
>> + return bb.progress.BasicProgressHandler(d, outfile=logfile)
>> + elif progress.startswith('percent:'):
>> + # Use specified regex
>> + return bb.progress.BasicProgressHandler(d,
>> regex=progress.split(':', 1)[1], outfile=logfile)
>> + elif progress.startswith('outof:'):
>> + # Use specified regex
>> + return bb.progress.OutOfProgressHandler(d,
>> regex=progress.split(':', 1)[1], outfile=logfile)
>> + elif progress.startswith("custom:"):
>> + # Use a custom progress handler that was injected via
>> OE_EXTRA_IMPORTS or __builtins__
>> + import functools
>> + from types import ModuleType
>> +
>> + parts = progress.split(":", 2)
>> + _, cls, otherargs = parts[0], parts[1], (parts[2] or None)
>> if parts[2:] else None
>> + if cls:
>> + def resolve(x, y):
>> + if not x:
>> + return None
>> + if isinstance(x, ModuleType):
>> + return getattr(x, y, None)
>> + return x.get(y)
>> + cls_obj = functools.reduce(resolve, cls.split("."),
>> bb.utils._context)
>> + if not cls_obj:
>> + # Fall-back on __builtins__
>> + cls_obj = functools.reduce(lambda x, y: x.get(y),
>> cls.split("."), __builtins__)
>> + if cls_obj:
>> + return cls_obj(d, outfile=logfile,
>> otherargs=otherargs)
>> + bb.warn('%s: unknown custom progress handler in task
>> progress varflag value "%s", ignoring' % (func, cls))
>> + else:
>> + bb.warn('%s: invalid task progress varflag value "%s",
>> ignoring' % (func, progress)) +
>> + return logfile
>> +
>> def exec_func_shell(func, d, runfile, cwd=None):
>> """Execute a shell function from the metadata
>>
>> @@ -372,23 +390,13 @@ exit $ret
>> cmd = [fakerootcmd, runfile]
>>
>> if bb.msg.loggerDefaultVerbose:
>> - logfile = LogTee(logger, sys.stdout)
>> + logfile = LogTee(logger, StdoutNoopContextManager())
>> else:
>> - logfile = sys.stdout
>> + logfile = StdoutNoopContextManager()
>>
>> progress = d.getVarFlag(func, 'progress')
>> if progress:
>> - if progress == 'percent':
>> - # Use default regex
>> - logfile = bb.progress.BasicProgressHandler(d,
>> outfile=logfile)
>> - elif progress.startswith('percent:'):
>> - # Use specified regex
>> - logfile = bb.progress.BasicProgressHandler(d,
>> regex=progress.split(':', 1)[1], outfile=logfile)
>> - elif progress.startswith('outof:'):
>> - # Use specified regex
>> - logfile = bb.progress.OutOfProgressHandler(d,
>> regex=progress.split(':', 1)[1], outfile=logfile)
>> - else:
>> - bb.warn('%s: invalid task progress varflag value "%s",
>> ignoring' % (func, progress))
>> + logfile = create_progress_handler(func, progress, logfile, d)
>>
>> fifobuffer = bytearray()
>> def readfifo(data):
>> @@ -407,6 +415,8 @@ exit $ret
>> bb.plain(value)
>> elif cmd == 'bbnote':
>> bb.note(value)
>> + elif cmd == 'bbverbnote':
>> + bb.verbnote(value)
>> elif cmd == 'bbwarn':
>> bb.warn(value)
>> elif cmd == 'bberror':
>> @@ -436,13 +446,8 @@ exit $ret
>> with open(fifopath, 'r+b', buffering=0) as fifo:
>> try:
>> bb.debug(2, "Executing shell function %s" % func)
>> -
>> - try:
>> - with open(os.devnull, 'r+') as stdin:
>> - bb.process.run(cmd, shell=False, stdin=stdin,
>> log=logfile, extrafiles=[(fifo,readfifo)])
>> - except bb.process.CmdError:
>> - logfn = d.getVar('BB_LOGFILE')
>> - raise FuncFailed(func, logfn)
>> + with open(os.devnull, 'r+') as stdin, logfile:
>> + bb.process.run(cmd, shell=False, stdin=stdin,
>> log=logfile, extrafiles=[(fifo,readfifo)]) finally:
>> os.unlink(fifopath)
>>
>> @@ -570,9 +575,6 @@ def _exec_task(fn, task, d, quieterr):
>> event.fire(TaskStarted(task, logfn, flags, localdata),
>> localdata) except (bb.BBHandledException, SystemExit):
>> return 1
>> - except FuncFailed as exc:
>> - logger.error(str(exc))
>> - return 1
>>
>> try:
>> for func in (prefuncs or '').split():
>> @@ -580,7 +582,10 @@ def _exec_task(fn, task, d, quieterr):
>> exec_func(task, localdata)
>> for func in (postfuncs or '').split():
>> exec_func(func, localdata)
>> - except FuncFailed as exc:
>> + except bb.BBHandledException:
>> + event.fire(TaskFailed(task, logfn, localdata, True),
>> localdata)
>> + return 1
>> + except Exception as exc:
>> if quieterr:
>> event.fire(TaskFailedSilent(task, logfn, localdata),
>> localdata) else:
>> @@ -588,9 +593,6 @@ def _exec_task(fn, task, d, quieterr):
>> logger.error(str(exc))
>> event.fire(TaskFailed(task, logfn, localdata,
>> errprinted), localdata) return 1
>> - except bb.BBHandledException:
>> - event.fire(TaskFailed(task, logfn, localdata, True),
>> localdata)
>> - return 1
>> finally:
>> sys.stdout.flush()
>> sys.stderr.flush()
>> @@ -814,6 +816,9 @@ def add_tasks(tasklist, d):
>> task_deps['parents'][task] = []
>> if 'deps' in flags:
>> for dep in flags['deps']:
>> + # Check and warn for "addtask task after foo" while
>> foo does not exist
>> + #if not dep in tasklist:
>> + # bb.warn('%s: dependent task %s for %s does not
>> exist' % (d.getVar('PN'), dep, task)) dep = d.expand(dep)
>> task_deps['parents'][task].append(dep)
>>
>> diff --git a/bitbake/lib/bb/cache.py b/bitbake/lib/bb/cache.py
>> index 258d679..b6f7da5 100644
>> --- a/bitbake/lib/bb/cache.py
>> +++ b/bitbake/lib/bb/cache.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # BitBake Cache implementation
>> #
>> @@ -15,18 +13,8 @@
>> # Copyright (C) 2005 Holger Hans Peter Freyther
>> # Copyright (C) 2005 ROAD GmbH
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import os
>> import sys
>> @@ -95,21 +83,21 @@ class CoreRecipeInfo(RecipeInfoCommon):
>> self.appends = self.listvar('__BBAPPEND', metadata)
>> self.nocache = self.getvar('BB_DONT_CACHE', metadata)
>>
>> + self.provides = self.depvar('PROVIDES', metadata)
>> + self.rprovides = self.depvar('RPROVIDES', metadata)
>> + self.pn = self.getvar('PN', metadata) or
>> bb.parse.vars_from_file(filename,metadata)[0]
>> + self.packages = self.listvar('PACKAGES', metadata)
>> + if not self.packages:
>> + self.packages.append(self.pn)
>> + self.packages_dynamic = self.listvar('PACKAGES_DYNAMIC',
>> metadata) +
>> self.skipreason = self.getvar('__SKIPPED', metadata)
>> if self.skipreason:
>> - self.pn = self.getvar('PN', metadata) or
>> bb.parse.BBHandler.vars_from_file(filename,metadata)[0] self.skipped
>> = True
>> - self.provides = self.depvar('PROVIDES', metadata)
>> - self.rprovides = self.depvar('RPROVIDES', metadata)
>> return
>>
>> self.tasks = metadata.getVar('__BBTASKS', False)
>>
>> - self.pn = self.getvar('PN', metadata)
>> - self.packages = self.listvar('PACKAGES', metadata)
>> - if not self.packages:
>> - self.packages.append(self.pn)
>> -
>> self.basetaskhashes = self.taskvar('BB_BASEHASH',
>> self.tasks, metadata) self.hashfilename =
>> self.getvar('BB_HASHFILENAME', metadata)
>> @@ -125,11 +113,8 @@ class CoreRecipeInfo(RecipeInfoCommon):
>> self.stampclean = self.getvar('STAMPCLEAN', metadata)
>> self.stamp_extrainfo = self.flaglist('stamp-extra-info',
>> self.tasks, metadata) self.file_checksums =
>> self.flaglist('file-checksums', self.tasks, metadata, True)
>> - self.packages_dynamic = self.listvar('PACKAGES_DYNAMIC',
>> metadata) self.depends = self.depvar('DEPENDS', metadata)
>> - self.provides = self.depvar('PROVIDES', metadata)
>> self.rdepends = self.depvar('RDEPENDS', metadata)
>> - self.rprovides = self.depvar('RPROVIDES', metadata)
>> self.rrecommends = self.depvar('RRECOMMENDS', metadata)
>> self.rprovides_pkg = self.pkgvar('RPROVIDES',
>> self.packages, metadata) self.rdepends_pkg =
>> self.pkgvar('RDEPENDS', self.packages, metadata) @@ -235,7 +220,7 @@
>> class CoreRecipeInfo(RecipeInfoCommon):
>> cachedata.hashfn[fn] = self.hashfilename
>> for task, taskhash in self.basetaskhashes.items():
>> - identifier = '%s.%s' % (fn, task)
>> + identifier = '%s:%s' % (fn, task)
>> cachedata.basetaskhash[identifier] = taskhash
>>
>> cachedata.inherits[fn] = self.inherits
>> @@ -249,7 +234,7 @@ def virtualfn2realfn(virtualfn):
>> Convert a virtual file name to a real one + the associated
>> subclass keyword """
>> mc = ""
>> - if virtualfn.startswith('multiconfig:'):
>> + if virtualfn.startswith('mc:'):
>> elems = virtualfn.split(':')
>> mc = elems[1]
>> virtualfn = ":".join(elems[2:])
>> @@ -270,7 +255,7 @@ def realfn2virtual(realfn, cls, mc):
>> if cls:
>> realfn = "virtual:" + cls + ":" + realfn
>> if mc:
>> - realfn = "multiconfig:" + mc + ":" + realfn
>> + realfn = "mc:" + mc + ":" + realfn
>> return realfn
>>
>> def variant2virtual(realfn, variant):
>> @@ -279,11 +264,11 @@ def variant2virtual(realfn, variant):
>> """
>> if variant == "":
>> return realfn
>> - if variant.startswith("multiconfig:"):
>> + if variant.startswith("mc:"):
>> elems = variant.split(":")
>> if elems[2]:
>> - return "multiconfig:" + elems[1] + ":virtual:" +
>> ":".join(elems[2:]) + ":" + realfn
>> - return "multiconfig:" + elems[1] + ":" + realfn
>> + return "mc:" + elems[1] + ":virtual:" +
>> ":".join(elems[2:]) + ":" + realfn
>> + return "mc:" + elems[1] + ":" + realfn
>> return "virtual:" + variant + ":" + realfn
>>
>> def parse_recipe(bb_data, bbfile, appends, mc=''):
>> @@ -361,7 +346,7 @@ class NoCache(object):
>> bb_data = self.databuilder.mcdata[mc].createCopy()
>> newstores = parse_recipe(bb_data, bbfile, appends, mc)
>> for ns in newstores:
>> - datastores["multiconfig:%s:%s" % (mc, ns)] =
>> newstores[ns]
>> + datastores["mc:%s:%s" % (mc, ns)] = newstores[ns]
>>
>> return datastores
>>
>> @@ -411,6 +396,15 @@ class Cache(NoCache):
>> else:
>> logger.debug(1, "Cache file %s not found, building..." %
>> self.cachefile)
>> + # We don't use the symlink, its just for debugging
>> convinience
>> + symlink = os.path.join(self.cachedir, "bb_cache.dat")
>> + if os.path.exists(symlink):
>> + bb.utils.remove(symlink)
>> + try:
>> + os.symlink(os.path.basename(self.cachefile), symlink)
>> + except OSError:
>> + pass
>> +
>> def load_cachefile(self):
>> cachesize = 0
>> previous_progress = 0
>> @@ -889,3 +883,56 @@ class MultiProcessCache(object):
>> p.dump([data, self.__class__.CACHE_VERSION])
>>
>> bb.utils.unlockfile(glf)
>> +
>> +
>> +class SimpleCache(object):
>> + """
>> + BitBake multi-process cache implementation
>> +
>> + Used by the codeparser & file checksum caches
>> + """
>> +
>> + def __init__(self, version):
>> + self.cachefile = None
>> + self.cachedata = None
>> + self.cacheversion = version
>> +
>> + def init_cache(self, d, cache_file_name=None, defaultdata=None):
>> + cachedir = (d.getVar("PERSISTENT_DIR") or
>> + d.getVar("CACHE"))
>> + if not cachedir:
>> + return defaultdata
>> +
>> + bb.utils.mkdirhier(cachedir)
>> + self.cachefile = os.path.join(cachedir,
>> + cache_file_name or
>> self.__class__.cache_file_name)
>> + logger.debug(1, "Using cache in '%s'", self.cachefile)
>> +
>> + glf = bb.utils.lockfile(self.cachefile + ".lock")
>> +
>> + try:
>> + with open(self.cachefile, "rb") as f:
>> + p = pickle.Unpickler(f)
>> + data, version = p.load()
>> + except:
>> + bb.utils.unlockfile(glf)
>> + return defaultdata
>> +
>> + bb.utils.unlockfile(glf)
>> +
>> + if version != self.cacheversion:
>> + return defaultdata
>> +
>> + return data
>> +
>> + def save(self, data):
>> + if not self.cachefile:
>> + return
>> +
>> + glf = bb.utils.lockfile(self.cachefile + ".lock")
>> +
>> + with open(self.cachefile, "wb") as f:
>> + p = pickle.Pickler(f, -1)
>> + p.dump([data, self.cacheversion])
>> +
>> + bb.utils.unlockfile(glf)
>> diff --git a/bitbake/lib/bb/cache_extra.py
>> b/bitbake/lib/bb/cache_extra.py index 83f4959..bf4226d 100644
>> --- a/bitbake/lib/bb/cache_extra.py
>> +++ b/bitbake/lib/bb/cache_extra.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # Extra RecipeInfo will be all defined in this file. Currently,
>> # Only Hob (Image Creator) Requests some extra fields. So
>> @@ -12,18 +10,8 @@
>>
>> # Copyright (C) 2011, Intel Corporation. All rights reserved.
>>
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from bb.cache import RecipeInfoCommon
>>
>> diff --git a/bitbake/lib/bb/checksum.py b/bitbake/lib/bb/checksum.py
>> index 4e1598f..5bc8a8f 100644
>> --- a/bitbake/lib/bb/checksum.py
>> +++ b/bitbake/lib/bb/checksum.py
>> @@ -2,18 +2,8 @@
>> #
>> # Copyright (C) 2012 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import glob
>> import operator
>> diff --git a/bitbake/lib/bb/codeparser.py
>> b/bitbake/lib/bb/codeparser.py index ddd1b97..fd2c473 100644
>> --- a/bitbake/lib/bb/codeparser.py
>> +++ b/bitbake/lib/bb/codeparser.py
>> @@ -1,3 +1,7 @@
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> """
>> BitBake code parser
>>
>> @@ -33,7 +37,7 @@ from bb.cache import MultiProcessCache
>> logger = logging.getLogger('BitBake.CodeParser')
>>
>> def bbhash(s):
>> - return hashlib.md5(s.encode("utf-8")).hexdigest()
>> + return hashlib.sha256(s.encode("utf-8")).hexdigest()
>>
>> def check_indent(codestr):
>> """If the code is indented, add a top level piece of code to
>> 'remove' the indentation""" @@ -140,7 +144,7 @@ class
>> CodeParserCache(MultiProcessCache): # so that an existing cache gets
>> invalidated. Additionally you'll need # to increment
>> __cache_version__ in cache.py in order to ensure that old # recipe
>> caches don't trigger "Taskhash mismatch" errors.
>> - CACHE_VERSION = 10
>> + CACHE_VERSION = 11
>>
>> def __init__(self):
>> MultiProcessCache.__init__(self)
>> @@ -368,8 +372,9 @@ class ShellParser():
>> def _parse_shell(self, value):
>> try:
>> tokens, _ = pyshyacc.parse(value, eof=True, debug=False)
>> - except pyshlex.NeedMore:
>> - raise sherrors.ShellSyntaxError("Unexpected EOF")
>> + except Exception:
>> + bb.error('Error during parse shell code, the last 5
>> lines are:\n%s' % '\n'.join(value.split('\n')[-5:]))
>> + raise
>>
>> self.process_tokens(tokens)
>>
>> diff --git a/bitbake/lib/bb/command.py b/bitbake/lib/bb/command.py
>> index 6c966e3..378f389 100644
>> --- a/bitbake/lib/bb/command.py
>> +++ b/bitbake/lib/bb/command.py
>> @@ -6,18 +6,8 @@ Provide an interface to interact with the bitbake
>> server through 'commands'
>> # Copyright (C) 2006-2007 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> """
>> The bitbake server takes 'commands' from its UI/commandline.
>> diff --git a/bitbake/lib/bb/compat.py b/bitbake/lib/bb/compat.py
>> index de1923d..4935668 100644
>> --- a/bitbake/lib/bb/compat.py
>> +++ b/bitbake/lib/bb/compat.py
>> @@ -1,3 +1,7 @@
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> """Code pulled from future python versions, here for compatibility"""
>>
>> from collections import MutableMapping, KeysView, ValuesView,
>> ItemsView, OrderedDict diff --git a/bitbake/lib/bb/cooker.py
>> b/bitbake/lib/bb/cooker.py index 16681ba..20ef04d 100644
>> --- a/bitbake/lib/bb/cooker.py
>> +++ b/bitbake/lib/bb/cooker.py
>> @@ -1,6 +1,3 @@
>> -#!/usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # Copyright (C) 2003, 2004 Chris Larson
>> # Copyright (C) 2003, 2004 Phil Blundell
>> @@ -9,19 +6,8 @@
>> # Copyright (C) 2005 ROAD GmbH
>> # Copyright (C) 2006 - 2007 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. -
>>
>> import sys, os, glob, os.path, re, time
>> import atexit
>> @@ -45,6 +31,7 @@ import pyinotify
>> import json
>> import pickle
>> import codecs
>> +import hashserv
>>
>> logger = logging.getLogger("BitBake")
>> collectlog = logging.getLogger("BitBake.Collection")
>> @@ -175,27 +162,45 @@ class BBCooker:
>>
>> self.configuration = configuration
>>
>> + bb.debug(1, "BBCooker starting %s" % time.time())
>> + sys.stdout.flush()
>> +
>> self.configwatcher = pyinotify.WatchManager()
>> + bb.debug(1, "BBCooker pyinotify1 %s" % time.time())
>> + sys.stdout.flush()
>> +
>> self.configwatcher.bbseen = []
>> self.configwatcher.bbwatchedfiles = []
>> self.confignotifier = pyinotify.Notifier(self.configwatcher,
>> self.config_notifications)
>> + bb.debug(1, "BBCooker pyinotify2 %s" % time.time())
>> + sys.stdout.flush()
>> self.watchmask = pyinotify.IN_CLOSE_WRITE |
>> pyinotify.IN_CREATE | pyinotify.IN_DELETE | \
>> pyinotify.IN_DELETE_SELF | pyinotify.IN_MODIFY |
>> pyinotify.IN_MOVE_SELF | \ pyinotify.IN_MOVED_FROM |
>> pyinotify.IN_MOVED_TO self.watcher = pyinotify.WatchManager()
>> + bb.debug(1, "BBCooker pyinotify3 %s" % time.time())
>> + sys.stdout.flush()
>> self.watcher.bbseen = []
>> self.watcher.bbwatchedfiles = []
>> self.notifier = pyinotify.Notifier(self.watcher,
>> self.notifications)
>> + bb.debug(1, "BBCooker pyinotify complete %s" % time.time())
>> + sys.stdout.flush()
>> +
>> # If being called by something like tinfoil, we need to
>> clean cached data # which may now be invalid
>> bb.parse.clear_cache()
>> bb.parse.BBHandler.cached_statements = {}
>>
>> self.ui_cmdline = None
>> + self.hashserv = None
>> + self.hashservaddr = None
>>
>> self.initConfigurationData()
>>
>> + bb.debug(1, "BBCooker parsed base configuration %s" %
>> time.time())
>> + sys.stdout.flush()
>> +
>> # we log all events to a file if so directed
>> if self.configuration.writeeventlog:
>> # register the log file writer as UI Handler
>> @@ -233,6 +238,9 @@ class BBCooker:
>> # Let SIGHUP exit as SIGTERM
>> signal.signal(signal.SIGHUP, self.sigterm_exception)
>>
>> + bb.debug(1, "BBCooker startup complete %s" % time.time())
>> + sys.stdout.flush()
>> +
>> def process_inotify_updates(self):
>> for n in [self.confignotifier, self.notifier]:
>> if n.check_events(timeout=0):
>> @@ -367,13 +375,12 @@ class BBCooker:
>> # Copy of the data store which has been expanded.
>> # Used for firing events and accessing variables where
>> expansion needs to be accounted for #
>> - bb.parse.init_parser(self.data)
>> -
>> if CookerFeatures.BASEDATASTORE_TRACKING in self.featureset:
>> self.disableDataTracking()
>>
>> - self.data.renameVar("__depends", "__base_depends")
>> - self.add_filewatch(self.data.getVar("__base_depends",
>> False), self.configwatcher)
>> + for mc in self.databuilder.mcdata.values():
>> + mc.renameVar("__depends", "__base_depends")
>> + self.add_filewatch(mc.getVar("__base_depends", False),
>> self.configwatcher)
>> self.baseconfig_valid = True
>> self.parsecache_valid = False
>> @@ -385,6 +392,22 @@ class BBCooker:
>> except prserv.serv.PRServiceConfigError as e:
>> bb.fatal("Unable to start PR Server, exitting")
>>
>> + if self.data.getVar("BB_HASHSERVE") == "auto":
>> + # Create a new hash server bound to a unix domain socket
>> + if not self.hashserv:
>> + dbfile = (self.data.getVar("PERSISTENT_DIR") or
>> self.data.getVar("CACHE")) + "/hashserv.db"
>> + self.hashservaddr = "unix://%s/hashserve.sock" %
>> self.data.getVar("TOPDIR")
>> + self.hashserv =
>> hashserv.create_server(self.hashservaddr, dbfile, sync=False)
>> + self.hashserv.process =
>> multiprocessing.Process(target=self.hashserv.serve_forever)
>> + self.hashserv.process.start()
>> + self.data.setVar("BB_HASHSERVE", self.hashservaddr)
>> + self.databuilder.origdata.setVar("BB_HASHSERVE",
>> self.hashservaddr)
>> + self.databuilder.data.setVar("BB_HASHSERVE",
>> self.hashservaddr)
>> + for mc in self.databuilder.mcdata:
>> + self.databuilder.mcdata[mc].setVar("BB_HASHSERVE",
>> self.hashservaddr) +
>> + bb.parse.init_parser(self.data)
>> +
>> def enableDataTracking(self):
>> self.configuration.tracking = True
>> if hasattr(self, "data"):
>> @@ -488,6 +511,7 @@ class BBCooker:
>> """
>> fn = None
>> envdata = None
>> + mc = ''
>> if not pkgs_to_build:
>> pkgs_to_build = []
>>
>> @@ -496,6 +520,12 @@ class BBCooker:
>> self.enableDataTracking()
>> self.reset()
>>
>> + def mc_base(p):
>> + if p.startswith('mc:'):
>> + s = p.split(':')
>> + if len(s) == 2:
>> + return s[1]
>> + return None
>>
>> if buildfile:
>> # Parse the configuration here. We need to do it
>> explicitly here since @@ -506,18 +536,16 @@ class BBCooker:
>> fn = self.matchFile(fn)
>> fn = bb.cache.realfn2virtual(fn, cls, mc)
>> elif len(pkgs_to_build) == 1:
>> - ignore = self.data.getVar("ASSUME_PROVIDED") or ""
>> - if pkgs_to_build[0] in set(ignore.split()):
>> - bb.fatal("%s is in ASSUME_PROVIDED" %
>> pkgs_to_build[0])
>> + mc = mc_base(pkgs_to_build[0])
>> + if not mc:
>> + ignore = self.data.getVar("ASSUME_PROVIDED") or ""
>> + if pkgs_to_build[0] in set(ignore.split()):
>> + bb.fatal("%s is in ASSUME_PROVIDED" %
>> pkgs_to_build[0])
>> - taskdata, runlist = self.buildTaskData(pkgs_to_build,
>> None, self.configuration.abort, allowincomplete=True)
>> + taskdata, runlist =
>> self.buildTaskData(pkgs_to_build, None, self.configuration.abort,
>> allowincomplete=True)
>> - mc = runlist[0][0]
>> - fn = runlist[0][3]
>> - else:
>> - envdata = self.data
>> - data.expandKeys(envdata)
>> - parse.ast.runAnonFuncs(envdata)
>> + mc = runlist[0][0]
>> + fn = runlist[0][3]
>>
>> if fn:
>> try:
>> @@ -526,6 +554,12 @@ class BBCooker:
>> except Exception as e:
>> parselog.exception("Unable to read %s", fn)
>> raise
>> + else:
>> + if not mc in self.databuilder.mcdata:
>> + bb.fatal('Not multiconfig named "%s" found' % mc)
>> + envdata = self.databuilder.mcdata[mc]
>> + data.expandKeys(envdata)
>> + parse.ast.runAnonFuncs(envdata)
>>
>> # Display history
>> with closing(StringIO()) as env:
>> @@ -565,10 +599,10 @@ class BBCooker:
>> wildcard = False
>>
>> # Wild card expansion:
>> - # Replace string such as "multiconfig:*:bash"
>> - # into "multiconfig:A:bash multiconfig:B:bash bash"
>> + # Replace string such as "mc:*:bash"
>> + # into "mc:A:bash mc:B:bash bash"
>> for k in targetlist:
>> - if k.startswith("multiconfig:"):
>> + if k.startswith("mc:"):
>> if wildcard:
>> bb.fatal('multiconfig conflict')
>> if k.split(":")[1] == "*":
>> @@ -601,7 +635,7 @@ class BBCooker:
>> runlist = []
>> for k in fulltargetlist:
>> mc = ""
>> - if k.startswith("multiconfig:"):
>> + if k.startswith("mc:"):
>> mc = k.split(":")[1]
>> k = ":".join(k.split(":")[2:])
>> ktask = task
>> @@ -620,13 +654,22 @@ class BBCooker:
>> runlist.append([mc, k, ktask, fn])
>> bb.event.fire(bb.event.TreeDataPreparationProgress(current,
>> len(fulltargetlist)), self.data)
>> - mcdeps = taskdata[mc].get_mcdepends()
>> + havemc = False
>> + for mc in self.multiconfigs:
>> + if taskdata[mc].get_mcdepends():
>> + havemc = True
>> +
>> # No need to do check providers if there are no mcdeps or
>> not an mc build
>> - if mcdeps and mc:
>> - # Make sure we can provide the multiconfig dependency
>> + if havemc or len(self.multiconfigs) > 1:
>> seen = set()
>> new = True
>> + # Make sure we can provide the multiconfig dependency
>> while new:
>> + mcdeps = set()
>> + # Add unresolved first, so we can get multiconfig
>> indirect dependencies on time
>> + for mc in self.multiconfigs:
>> + taskdata[mc].add_unresolved(localdata[mc],
>> self.recipecaches[mc])
>> + mcdeps |= set(taskdata[mc].get_mcdepends())
>> new = False
>> for mc in self.multiconfigs:
>> for k in mcdeps:
>> @@ -641,6 +684,7 @@ class BBCooker:
>> taskdata[depmc].add_provider(localdata[depmc],
>> self.recipecaches[depmc], l[3]) seen.add(k)
>> new = True
>> +
>> for mc in self.multiconfigs:
>> taskdata[mc].add_unresolved(localdata[mc],
>> self.recipecaches[mc])
>> @@ -676,7 +720,7 @@ class BBCooker:
>> @staticmethod
>> def add_mc_prefix(mc, pn):
>> if mc:
>> - return "multiconfig:%s:%s" % (mc, pn)
>> + return "mc:%s:%s" % (mc, pn)
>> return pn
>>
>> def buildDependTree(self, rq, taskdata):
>> @@ -875,6 +919,10 @@ class BBCooker:
>> os.unlink('package-depends.dot')
>> except FileNotFoundError:
>> pass
>> + try:
>> + os.unlink('recipe-depends.dot')
>> + except FileNotFoundError:
>> + pass
>>
>> with open('task-depends.dot', 'w') as f:
>> f.write("digraph depends {\n")
>> @@ -888,27 +936,6 @@ class BBCooker:
>> f.write("}\n")
>> logger.info("Task dependencies saved to 'task-depends.dot'")
>>
>> - with open('recipe-depends.dot', 'w') as f:
>> - f.write("digraph depends {\n")
>> - pndeps = {}
>> - for task in sorted(depgraph["tdepends"]):
>> - (pn, taskname) = task.rsplit(".", 1)
>> - if pn not in pndeps:
>> - pndeps[pn] = set()
>> - for dep in sorted(depgraph["tdepends"][task]):
>> - (deppn, deptaskname) = dep.rsplit(".", 1)
>> - pndeps[pn].add(deppn)
>> - for pn in sorted(pndeps):
>> - fn = depgraph["pn"][pn]["filename"]
>> - version = depgraph["pn"][pn]["version"]
>> - f.write('"%s" [label="%s\\n%s\\n%s"]\n' % (pn, pn,
>> version, fn))
>> - for dep in sorted(pndeps[pn]):
>> - if dep == pn:
>> - continue
>> - f.write('"%s" -> "%s"\n' % (pn, dep))
>> - f.write("}\n")
>> - logger.info("Flattened recipe dependencies saved to
>> 'recipe-depends.dot'") -
>> def show_appends_with_no_recipes(self):
>> # Determine which bbappends haven't been applied
>>
>> @@ -1191,8 +1218,8 @@ class BBCooker:
>> continue
>> elif regex == "":
>> parselog.debug(1, "BBFILE_PATTERN_%s is empty" %
>> c)
>> + cre = re.compile('^NULL$')
>> errors = False
>> - continue
>> else:
>> try:
>> cre = re.compile(regex)
>> @@ -1453,7 +1480,7 @@ class BBCooker:
>> ntargets = []
>> for target in runlist:
>> if target[0]:
>> - ntargets.append("multiconfig:%s:%s:%s" % (target[0],
>> target[1], target[2]))
>> + ntargets.append("mc:%s:%s:%s" % (target[0],
>> target[1], target[2])) ntargets.append("%s:%s" % (target[1],
>> target[2]))
>> for mc in self.multiconfigs:
>> @@ -1576,6 +1603,9 @@ class BBCooker:
>> for pkg in pkgs_to_build:
>> if pkg in ignore:
>> parselog.warning("Explicit target \"%s\" is in
>> ASSUME_PROVIDED, ignoring" % pkg)
>> + if pkg.startswith("multiconfig:"):
>> + pkgs_to_build.remove(pkg)
>> + pkgs_to_build.append(pkg.replace("multiconfig:",
>> "mc:"))
>> if 'world' in pkgs_to_build:
>> pkgs_to_build.remove('world')
>> @@ -1583,7 +1613,7 @@ class BBCooker:
>> bb.providers.buildWorldTargetList(self.recipecaches[mc],
>> task) for t in self.recipecaches[mc].world_target:
>> if mc:
>> - t = "multiconfig:" + mc + ":" + t
>> + t = "mc:" + mc + ":" + t
>> pkgs_to_build.append(t)
>>
>> if 'universe' in pkgs_to_build:
>> @@ -1602,7 +1632,7 @@ class BBCooker:
>> bb.debug(1, "Skipping %s for universe
>> tasks as task %s doesn't exist" % (t, task)) continue
>> if mc:
>> - t = "multiconfig:" + mc + ":" + t
>> + t = "mc:" + mc + ":" + t
>> pkgs_to_build.append(t)
>>
>> return pkgs_to_build
>> @@ -1615,9 +1645,11 @@ class BBCooker:
>>
>> def post_serve(self):
>> prserv.serv.auto_shutdown()
>> + if self.hashserv:
>> + self.hashserv.process.terminate()
>> + self.hashserv.process.join()
>> bb.event.fire(CookerExit(), self.data)
>>
>> -
>> def shutdown(self, force = False):
>> if force:
>> self.state = state.forceshutdown
>> @@ -1632,6 +1664,7 @@ class BBCooker:
>>
>> def reset(self):
>> self.initConfigurationData()
>> + self.handlePRServ()
>>
>> def clientComplete(self):
>> """Called when the client is done using the server"""
>> @@ -1865,35 +1898,6 @@ class ParsingFailure(Exception):
>> self.recipe = recipe
>> Exception.__init__(self, realexception, recipe)
>>
>> -class Feeder(multiprocessing.Process):
>> - def __init__(self, jobs, to_parsers, quit):
>> - self.quit = quit
>> - self.jobs = jobs
>> - self.to_parsers = to_parsers
>> - multiprocessing.Process.__init__(self)
>> -
>> - def run(self):
>> - while True:
>> - try:
>> - quit = self.quit.get_nowait()
>> - except queue.Empty:
>> - pass
>> - else:
>> - if quit == 'cancel':
>> - self.to_parsers.cancel_join_thread()
>> - break
>> -
>> - try:
>> - job = self.jobs.pop()
>> - except IndexError:
>> - break
>> -
>> - try:
>> - self.to_parsers.put(job, timeout=0.5)
>> - except queue.Full:
>> - self.jobs.insert(0, job)
>> - continue
>> -
>> class Parser(multiprocessing.Process):
>> def __init__(self, jobs, results, quit, init, profile):
>> self.jobs = jobs
>> @@ -1940,11 +1944,8 @@ class Parser(multiprocessing.Process):
>> result = pending.pop()
>> else:
>> try:
>> - job = self.jobs.get(timeout=0.25)
>> - except queue.Empty:
>> - continue
>> -
>> - if job is None:
>> + job = self.jobs.pop()
>> + except IndexError:
>> break
>> result = self.parse(*job)
>>
>> @@ -2028,14 +2029,15 @@ class CookerParser(object):
>> multiprocessing.util.Finalize(None,
>> bb.codeparser.parser_cache_save, exitpriority=1)
>> multiprocessing.util.Finalize(None, bb.fetch.fetcher_parse_save,
>> exitpriority=1)
>> - self.feeder_quit = multiprocessing.Queue(maxsize=1)
>> self.parser_quit =
>> multiprocessing.Queue(maxsize=self.num_processes)
>> - self.jobs =
>> multiprocessing.Queue(maxsize=self.num_processes) self.result_queue =
>> multiprocessing.Queue()
>> - self.feeder = Feeder(self.willparse, self.jobs,
>> self.feeder_quit)
>> - self.feeder.start()
>> +
>> + def chunkify(lst,n):
>> + return [lst[i::n] for i in range(n)]
>> + self.jobs = chunkify(self.willparse, self.num_processes)
>> +
>> for i in range(0, self.num_processes):
>> - parser = Parser(self.jobs, self.result_queue,
>> self.parser_quit, init, self.cooker.configuration.profile)
>> + parser = Parser(self.jobs[i], self.result_queue,
>> self.parser_quit, init, self.cooker.configuration.profile)
>> parser.start() self.process_names.append(parser.name)
>> self.processes.append(parser)
>> @@ -2056,17 +2058,20 @@ class CookerParser(object):
>> self.total)
>>
>> bb.event.fire(event, self.cfgdata)
>> - self.feeder_quit.put(None)
>> for process in self.processes:
>> self.parser_quit.put(None)
>> else:
>> - self.feeder_quit.put('cancel')
>> -
>> self.parser_quit.cancel_join_thread()
>> for process in self.processes:
>> self.parser_quit.put(None)
>>
>> - self.jobs.cancel_join_thread()
>> + # Cleanup the queue before call process.join(), otherwise
>> there might be
>> + # deadlocks.
>> + while True:
>> + try:
>> + self.result_queue.get(timeout=0.25)
>> + except queue.Empty:
>> + break
>>
>> for process in self.processes:
>> if force:
>> @@ -2074,7 +2079,6 @@ class CookerParser(object):
>> process.terminate()
>> else:
>> process.join()
>> - self.feeder.join()
>>
>> sync = threading.Thread(target=self.bb_cache.sync)
>> sync.start()
>> diff --git a/bitbake/lib/bb/cookerdata.py
>> b/bitbake/lib/bb/cookerdata.py index 5df66e6..472423f 100644
>> --- a/bitbake/lib/bb/cookerdata.py
>> +++ b/bitbake/lib/bb/cookerdata.py
>> @@ -1,6 +1,3 @@
>> -#!/usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # Copyright (C) 2003, 2004 Chris Larson
>> # Copyright (C) 2003, 2004 Phil Blundell
>> @@ -9,23 +6,14 @@
>> # Copyright (C) 2005 ROAD GmbH
>> # Copyright (C) 2006 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import logging
>> import os
>> import re
>> import sys
>> +import hashlib
>> from functools import wraps
>> import bb
>> from bb import data
>> @@ -134,6 +122,7 @@ class CookerConfiguration(object):
>> self.profile = False
>> self.nosetscene = False
>> self.setsceneonly = False
>> + self.skipsetscene = False
>> self.invalidate_stamp = False
>> self.dump_signatures = []
>> self.dry_run = False
>> @@ -279,12 +268,13 @@ class CookerDataBuilder(object):
>> self.mcdata = {}
>>
>> def parseBaseConfiguration(self):
>> + data_hash = hashlib.sha256()
>> try:
>> - bb.parse.init_parser(self.basedata)
>> self.data = self.parseConfigurationFiles(self.prefiles,
>> self.postfiles)
>> if self.data.getVar("BB_WORKERCONTEXT", False) is None:
>> bb.fetch.fetcher_init(self.data)
>> + bb.parse.init_parser(self.data)
>> bb.codeparser.parser_cache_init(self.data)
>>
>> bb.event.fire(bb.event.ConfigParsed(), self.data)
>> @@ -302,7 +292,7 @@ class CookerDataBuilder(object):
>> bb.event.fire(bb.event.ConfigParsed(), self.data)
>>
>> bb.parse.init_parser(self.data)
>> - self.data_hash = self.data.get_hash()
>> + data_hash.update(self.data.get_hash().encode('utf-8'))
>> self.mcdata[''] = self.data
>>
>> multiconfig = (self.data.getVar("BBMULTICONFIG") or
>> "").split() @@ -310,9 +300,11 @@ class CookerDataBuilder(object):
>> mcdata = self.parseConfigurationFiles(self.prefiles,
>> self.postfiles, config) bb.event.fire(bb.event.ConfigParsed(), mcdata)
>> self.mcdata[config] = mcdata
>> + data_hash.update(mcdata.get_hash().encode('utf-8'))
>> if multiconfig:
>> bb.event.fire(bb.event.MultiConfigParsed(self.mcdata),
>> self.data)
>> + self.data_hash = data_hash.hexdigest()
>> except (SyntaxError, bb.BBHandledException):
>> raise bb.BBHandledException
>> except bb.data_smart.ExpansionError as e:
>> @@ -354,14 +346,24 @@ class CookerDataBuilder(object):
>> data = parse_config_file(layerconf, data)
>>
>> layers = (data.getVar('BBLAYERS') or "").split()
>> + broken_layers = []
>>
>> data = bb.data.createCopy(data)
>> approved = bb.utils.approved_variables()
>> +
>> + # Check whether present layer directories exist
>> for layer in layers:
>> if not os.path.isdir(layer):
>> - parselog.critical("Layer directory '%s' does not
>> exist! "
>> - "Please check BBLAYERS in %s"
>> % (layer, layerconf))
>> - sys.exit(1)
>> + broken_layers.append(layer)
>> +
>> + if broken_layers:
>> + parselog.critical("The following layer directories
>> do not exist:")
>> + for layer in broken_layers:
>> + parselog.critical(" %s", layer)
>> + parselog.critical("Please check BBLAYERS in %s" %
>> (layerconf))
>> + sys.exit(1)
>> +
>> + for layer in layers:
>> parselog.debug(2, "Adding layer %s", layer)
>> if 'HOME' in approved and '~' in layer:
>> layer = os.path.expanduser(layer)
>> @@ -391,7 +393,11 @@ class CookerDataBuilder(object):
>> bb.fatal("BBFILES_DYNAMIC entries must be of the
>> form <collection name>:<filename pattern>, not:\n %s" % "\n
>> ".join(invalid)) layerseries =
>> set((data.getVar("LAYERSERIES_CORENAMES") or "").split())
>> + collections_tmp = collections[:]
>> for c in collections:
>> + collections_tmp.remove(c)
>> + if c in collections_tmp:
>> + bb.fatal("Found duplicated BBFILE_COLLECTIONS
>> '%s', check bblayers.conf or layer.conf to fix it." % c) compat =
>> set((data.getVar("LAYERSERIES_COMPAT_%s" % c) or "").split()) if
>> compat and not (compat & layerseries): bb.fatal("Layer %s is not
>> compatible with the core layer which only supports these series: %s
>> (layer is compatible with %s)" diff --git
>> a/bitbake/lib/bb/daemonize.py b/bitbake/lib/bb/daemonize.py index
>> c937675..f01e6ec 100644 --- a/bitbake/lib/bb/daemonize.py +++
>> b/bitbake/lib/bb/daemonize.py @@ -1,3 +1,7 @@
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> """
>> Python Daemonizing helper
>>
>> diff --git a/bitbake/lib/bb/data.py b/bitbake/lib/bb/data.py
>> index d66d98c..0d75d0c 100644
>> --- a/bitbake/lib/bb/data.py
>> +++ b/bitbake/lib/bb/data.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake 'Data' implementations
>>
>> @@ -22,18 +20,7 @@ the speed is more critical here.
>> # Copyright (C) 2003, 2004 Chris Larson
>> # Copyright (C) 2005 Holger Hans Peter Freyther
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> # Based on functions from the base bb module, Copyright 2003 Holger
>> Schurig
>> @@ -143,7 +130,7 @@ def emit_var(var, o=sys.__stdout__, d = init(),
>> all=False): if all:
>> oval = d.getVar(var, False)
>> val = d.getVar(var)
>> - except (KeyboardInterrupt, bb.build.FuncFailed):
>> + except (KeyboardInterrupt):
>> raise
>> except Exception as exc:
>> o.write('# expansion of %s threw %s: %s\n' % (var,
>> exc.__class__.__name__, str(exc))) @@ -322,8 +309,6 @@ def
>> build_dependencies(key, keys, shelldeps, varflagsexcl, d): if
>> varflags.get("python"): value = d.getVarFlag(key, "_content", False)
>> parser = bb.codeparser.PythonParser(key, logger)
>> - if value and "\t" in value:
>> - logger.warning("Variable %s contains tabs,
>> please remove these (%s)" % (key, d.getVar("FILE")))
>> parser.parse_python(value, filename=varflags.get("filename"),
>> lineno=varflags.get("lineno")) deps = deps | parser.references deps =
>> deps | (keys & parser.execs) @@ -437,8 +422,8 @@ def
>> generate_dependency_hash(tasklist, gendeps, lookupcache, whitelist,
>> fn): var = lookupcache[dep] if var is not None:
>> data = data + str(var)
>> - k = fn + "." + task
>> - basehash[k] = hashlib.md5(data.encode("utf-8")).hexdigest()
>> + k = fn + ":" + task
>> + basehash[k] =
>> hashlib.sha256(data.encode("utf-8")).hexdigest() taskdeps[task] =
>> alldeps
>> return taskdeps, basehash
>> diff --git a/bitbake/lib/bb/data_smart.py
>> b/bitbake/lib/bb/data_smart.py index 67af380..dd5c618 100644
>> --- a/bitbake/lib/bb/data_smart.py
>> +++ b/bitbake/lib/bb/data_smart.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake Smart Dictionary Implementation
>>
>> @@ -14,18 +12,8 @@ BitBake build tools.
>> # Copyright (C) 2005 Uli Luckas
>> # Copyright (C) 2005 ROAD GmbH
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. # Based on functions from the base bb module,
>> Copyright 2003 Holger Schurig
>> import copy, re, sys, traceback
>> @@ -39,10 +27,11 @@ from bb.COW import COWDictBase
>> logger = logging.getLogger("BitBake.Data")
>>
>> __setvar_keyword__ = ["_append", "_prepend", "_remove"]
>> -__setvar_regexp__ =
>> re.compile('(?P<base>.*?)(?P<keyword>_append|_prepend|_remove)(_(?P<add>[^A-Z]*))?$')
>> -__expand_var_regexp__ = re.compile(r"\${[^{}@\n\t :]+}")
>> +__setvar_regexp__ =
>> re.compile(r'(?P<base>.*?)(?P<keyword>_append|_prepend|_remove)(_(?P<add>[^A-Z]*))?$')
>> +__expand_var_regexp__ = re.compile(r"\${[a-zA-Z0-9\-_+./~]+?}")
>> __expand_python_regexp__ = re.compile(r"\${@.+?}")
>> -__whitespace_split__ = re.compile('(\s)') +__whitespace_split__ =
>> re.compile(r'(\s)') +__override_regexp__ = re.compile(r'[a-z0-9]+')
>> def infer_caller_details(loginfo, parent = False, varval = True):
>> """Save the caller the trouble of specifying everything."""
>> @@ -597,7 +586,7 @@ class DataSmart(MutableMapping):
>> # aka pay the cookie monster
>> override = var[var.rfind('_')+1:]
>> shortvar = var[:var.rfind('_')]
>> - while override and override.islower():
>> + while override and __override_regexp__.match(override):
>> if shortvar not in self.overridedata:
>> self.overridedata[shortvar] = []
>> if [var, override] not in self.overridedata[shortvar]:
>> @@ -1073,4 +1062,4 @@ class DataSmart(MutableMapping):
>> data.update({i:value})
>>
>> data_str = str([(k, data[k]) for k in sorted(data.keys())])
>> - return hashlib.md5(data_str.encode("utf-8")).hexdigest()
>> + return hashlib.sha256(data_str.encode("utf-8")).hexdigest()
>> diff --git a/bitbake/lib/bb/event.py b/bitbake/lib/bb/event.py
>> index 5b1b094..d44621e 100644
>> --- a/bitbake/lib/bb/event.py
>> +++ b/bitbake/lib/bb/event.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake 'Event' implementation
>>
>> @@ -9,18 +7,8 @@ BitBake build tools.
>>
>> # Copyright (C) 2003, 2004 Chris Larson
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import os, sys
>> import warnings
>> @@ -136,6 +124,7 @@ def fire_class_handlers(event, d):
>> ui_queue = []
>> @atexit.register
>> def print_ui_queue():
>> + global ui_queue
>> """If we're exiting before a UI has been spawned, display any
>> queued LogRecords to the console."""
>> logger = logging.getLogger("BitBake")
>> @@ -180,6 +169,7 @@ def print_ui_queue():
>> logger.removeHandler(stderr)
>> else:
>> logger.removeHandler(stdout)
>> + ui_queue = []
>>
>> def fire_ui_handlers(event, d):
>> global _thread_lock
>> @@ -414,23 +404,6 @@ class RecipeTaskPreProcess(RecipeEvent):
>> class RecipeParsed(RecipeEvent):
>> """ Recipe Parsing Complete """
>>
>> -class StampUpdate(Event):
>> - """Trigger for any adjustment of the stamp files to happen"""
>> -
>> - def __init__(self, targets, stampfns):
>> - self._targets = targets
>> - self._stampfns = stampfns
>> - Event.__init__(self)
>> -
>> - def getStampPrefix(self):
>> - return self._stampfns
>> -
>> - def getTargets(self):
>> - return self._targets
>> -
>> - stampPrefix = property(getStampPrefix)
>> - targets = property(getTargets)
>> -
>> class BuildBase(Event):
>> """Base class for bitbake build events"""
>>
>> diff --git a/bitbake/lib/bb/exceptions.py
>> b/bitbake/lib/bb/exceptions.py index cd71343..ecbad59 100644
>> --- a/bitbake/lib/bb/exceptions.py
>> +++ b/bitbake/lib/bb/exceptions.py
>> @@ -1,3 +1,6 @@
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>>
>> import inspect
>> import traceback
>> diff --git a/bitbake/lib/bb/fetch2/__init__.py
>> b/bitbake/lib/bb/fetch2/__init__.py index 572b71a..1f5f8f1 100644
>> --- a/bitbake/lib/bb/fetch2/__init__.py
>> +++ b/bitbake/lib/bb/fetch2/__init__.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake 'Fetch' implementations
>>
>> @@ -10,18 +8,7 @@ BitBake build tools.
>> # Copyright (C) 2003, 2004 Chris Larson
>> # Copyright (C) 2012 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> # Based on functions from the base bb module, Copyright 2003 Holger
>> Schurig
>> @@ -256,7 +243,7 @@ class URI(object):
>>
>> # Identify if the URI is relative or not
>> if urlp.scheme in self._relative_schemes and \
>> - re.compile("^\w+:(?!//)").match(uri):
>> + re.compile(r"^\w+:(?!//)").match(uri):
>> self.relative = True
>>
>> if not self.relative:
>> @@ -524,7 +511,7 @@ def fetcher_parse_save():
>> def fetcher_parse_done():
>> _checksum_cache.save_merge()
>>
>> -def fetcher_compare_revisions():
>> +def fetcher_compare_revisions(d):
>> """
>> Compare the revisions in the persistant cache with current
>> values and return true/false on whether they've changed.
>> @@ -777,7 +764,8 @@ def get_srcrev(d,
>> method_name='sortable_revision'): #
>> format = d.getVar('SRCREV_FORMAT')
>> if not format:
>> - raise FetchError("The SRCREV_FORMAT variable must be set
>> when multiple SCMs are used.")
>> + raise FetchError("The SRCREV_FORMAT variable must be set
>> when multiple SCMs are used.\n"\
>> + "The SCMs are:\n%s" % '\n'.join(scms))
>>
>> name_to_rev = {}
>> seenautoinc = False
>> @@ -855,10 +843,18 @@ def runfetchcmd(cmd, d, quiet=False,
>> cleanup=None, log=None, workdir=None): if val:
>> cmd = 'export ' + var + '=\"%s\"; %s' % (val, cmd)
>>
>> + # Ensure that a _PYTHON_SYSCONFIGDATA_NAME value set by a recipe
>> + # (for example via python3native.bbclass since warrior) is not
>> set for
>> + # host Python (otherwise tools like git-make-shallow will fail)
>> + cmd = 'unset _PYTHON_SYSCONFIGDATA_NAME; ' + cmd
>> +
>> # Disable pseudo as it may affect ssh, potentially causing it to
>> hang. cmd = 'export PSEUDO_DISABLED=1; ' + cmd
>>
>> - logger.debug(1, "Running %s", cmd)
>> + if workdir:
>> + logger.debug(1, "Running '%s' in %s" % (cmd, workdir))
>> + else:
>> + logger.debug(1, "Running %s", cmd)
>>
>> success = False
>> error_message = ""
>> @@ -894,7 +890,7 @@ def check_network_access(d, info, url):
>> log remote network access, and error if BB_NO_NETWORK is set or
>> the given URI is untrusted
>> """
>> - if d.getVar("BB_NO_NETWORK") == "1":
>> + if bb.utils.to_boolean(d.getVar("BB_NO_NETWORK")):
>> raise NetworkAccess(url, info)
>> elif not trusted_network(d, url):
>> raise UntrustedUrl(url, info)
>> @@ -966,7 +962,8 @@ def rename_bad_checksum(ud, suffix):
>>
>> new_localpath = "%s_bad-checksum_%s" % (ud.localpath, suffix)
>> bb.warn("Renaming %s to %s" % (ud.localpath, new_localpath))
>> - bb.utils.movefile(ud.localpath, new_localpath)
>> + if not bb.utils.movefile(ud.localpath, new_localpath):
>> + bb.warn("Renaming %s to %s failed, grep movefile in
>> log.do_fetch to see why" % (ud.localpath, new_localpath))
>>
>> def try_mirror_url(fetch, origud, ud, ld, check = False):
>> @@ -1027,7 +1024,7 @@ def try_mirror_url(fetch, origud, ud, ld, check
>> = False): raise
>>
>> except IOError as e:
>> - if e.errno in [os.errno.ESTALE]:
>> + if e.errno in [errno.ESTALE]:
>> logger.warning("Stale Error Observed %s." % ud.url)
>> return False
>> raise
>> @@ -1094,7 +1091,7 @@ def trusted_network(d, url):
>> BB_ALLOWED_NETWORKS is set globally or for a specific recipe.
>> Note: modifies SRC_URI & mirrors.
>> """
>> - if d.getVar('BB_NO_NETWORK') == "1":
>> + if bb.utils.to_boolean(d.getVar("BB_NO_NETWORK")):
>> return True
>>
>> pkgname = d.expand(d.getVar('PN', False))
>> @@ -1403,7 +1400,7 @@ class FetchMethod(object):
>> Fetch urls
>> Assumes localpath was called first
>> """
>> - raise NoMethodError(url)
>> + raise NoMethodError(urldata.url)
>>
>> def unpack(self, urldata, rootdir, data):
>> iterate = False
>> @@ -1469,7 +1466,7 @@ class FetchMethod(object):
>> else:
>> cmd = 'rpm2cpio.sh %s | cpio -id' % (file)
>> elif file.endswith('.deb') or file.endswith('.ipk'):
>> - output = subprocess.check_output('ar -t %s' % file,
>> preexec_fn=subprocess_setup, shell=True)
>> + output = subprocess.check_output(['ar', '-t', file],
>> preexec_fn=subprocess_setup) datafile = None
>> if output:
>> for line in output.decode().splitlines():
>> @@ -1547,7 +1544,7 @@ class FetchMethod(object):
>> Check the status of a URL
>> Assumes localpath was called first
>> """
>> - logger.info("URL %s could not be checked for status since no
>> method exists.", url)
>> + logger.info("URL %s could not be checked for status since no
>> method exists.", urldata.url) return True
>>
>> def latest_revision(self, ud, d, name):
>> @@ -1555,7 +1552,7 @@ class FetchMethod(object):
>> Look in the cache for the latest revision, if not present
>> ask the SCM. """
>> if not hasattr(self, "_latest_revision"):
>> - raise ParameterError("The fetcher for this URL does not
>> support _latest_revision", url)
>> + raise ParameterError("The fetcher for this URL does not
>> support _latest_revision", ud.url)
>> revs = bb.persist_data.persist('BB_URI_HEADREVS', d)
>> key = self.generate_revision_key(ud, d, name)
>> @@ -1638,7 +1635,7 @@ class Fetch(object):
>> urls = self.urls
>>
>> network = self.d.getVar("BB_NO_NETWORK")
>> - premirroronly = (self.d.getVar("BB_FETCH_PREMIRRORONLY") ==
>> "1")
>> + premirroronly =
>> bb.utils.to_boolean(self.d.getVar("BB_FETCH_PREMIRRORONLY"))
>> for u in urls:
>> ud = self.ud[u]
>> @@ -1716,7 +1713,7 @@ class Fetch(object):
>> update_stamp(ud, self.d)
>>
>> except IOError as e:
>> - if e.errno in [os.errno.ESTALE]:
>> + if e.errno in [errno.ESTALE]:
>> logger.error("Stale Error Observed %s." % u)
>> raise ChecksumError("Stale Error Detected")
>>
>> @@ -1786,7 +1783,7 @@ class Fetch(object):
>>
>> for url in urls:
>> if url not in self.ud:
>> - self.ud[url] = FetchData(url, d)
>> + self.ud[url] = FetchData(url, self.d)
>> ud = self.ud[url]
>> ud.setup_localpath(self.d)
>>
>> diff --git a/bitbake/lib/bb/fetch2/bzr.py
>> b/bitbake/lib/bb/fetch2/bzr.py index 658502f..c56d875 100644
>> --- a/bitbake/lib/bb/fetch2/bzr.py
>> +++ b/bitbake/lib/bb/fetch2/bzr.py
>> @@ -10,18 +10,8 @@ BitBake 'Fetch' implementation for bzr.
>> # BitBake build tools.
>> # Copyright (C) 2003, 2004 Chris Larson
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import os
>> import sys
>> diff --git a/bitbake/lib/bb/fetch2/clearcase.py
>> b/bitbake/lib/bb/fetch2/clearcase.py index 3a6573d..3dd93ad 100644
>> --- a/bitbake/lib/bb/fetch2/clearcase.py
>> +++ b/bitbake/lib/bb/fetch2/clearcase.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake 'Fetch' clearcase implementation
>>
>> @@ -47,18 +45,7 @@ User credentials:
>> """
>> # Copyright (C) 2014 Siemens AG
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>>
>> import os
>> @@ -67,6 +54,8 @@ import shutil
>> import bb
>> from bb.fetch2 import FetchMethod
>> from bb.fetch2 import FetchError
>> +from bb.fetch2 import MissingParameterError
>> +from bb.fetch2 import ParameterError
>> from bb.fetch2 import runfetchcmd
>> from bb.fetch2 import logger
>>
>> @@ -92,7 +81,7 @@ class ClearCase(FetchMethod):
>> if 'protocol' in ud.parm:
>> ud.proto = ud.parm['protocol']
>> if not ud.proto in ('http', 'https'):
>> - raise fetch2.ParameterError("Invalid protocol type",
>> ud.url)
>> + raise ParameterError("Invalid protocol type", ud.url)
>>
>> ud.vob = ''
>> if 'vob' in ud.parm:
>> diff --git a/bitbake/lib/bb/fetch2/cvs.py
>> b/bitbake/lib/bb/fetch2/cvs.py index 0e0a319..1b35ba4 100644
>> --- a/bitbake/lib/bb/fetch2/cvs.py
>> +++ b/bitbake/lib/bb/fetch2/cvs.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake 'Fetch' implementations
>>
>> @@ -10,20 +8,9 @@ BitBake build tools.
>>
>> # Copyright (C) 2003, 2004 Chris Larson
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. -#
>> -#Based on functions from the base bb module, Copyright 2003 Holger
>> Schurig +# Based on functions from the base bb module, Copyright 2003
>> Holger Schurig #
>>
>> import os
>> diff --git a/bitbake/lib/bb/fetch2/git.py
>> b/bitbake/lib/bb/fetch2/git.py index 59a2ee8..2d1d2ca 100644
>> --- a/bitbake/lib/bb/fetch2/git.py
>> +++ b/bitbake/lib/bb/fetch2/git.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake 'Fetch' git implementation
>>
>> @@ -55,20 +53,10 @@ Supported SRC_URI options are:
>>
>> """
>>
>> -#Copyright (C) 2005 Richard Purdie
>> +# Copyright (C) 2005 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import collections
>> import errno
>> @@ -199,7 +187,7 @@ class Git(FetchMethod):
>> depth_default = 1
>> ud.shallow_depths = collections.defaultdict(lambda:
>> depth_default)
>> - revs_default = d.getVar("BB_GIT_SHALLOW_REVS", True)
>> + revs_default = d.getVar("BB_GIT_SHALLOW_REVS")
>> ud.shallow_revs = []
>> ud.branches = {}
>> for pos, name in enumerate(ud.names):
>> @@ -318,7 +306,7 @@ class Git(FetchMethod):
>> def try_premirror(self, ud, d):
>> # If we don't do this, updating an existing checkout with
>> only premirrors # is not possible
>> - if d.getVar("BB_FETCH_PREMIRRORONLY") is not None:
>> + if bb.utils.to_boolean(d.getVar("BB_FETCH_PREMIRRORONLY")):
>> return True
>> if os.path.exists(ud.clonedir):
>> return False
>> @@ -476,6 +464,8 @@ class Git(FetchMethod):
>> if os.path.exists(destdir):
>> bb.utils.prunedir(destdir)
>>
>> + need_lfs = ud.parm.get("lfs", "1") == "1"
>> +
>> source_found = False
>> source_error = []
>>
>> @@ -503,6 +493,13 @@ class Git(FetchMethod):
>>
>> repourl = self._get_repo_url(ud)
>> runfetchcmd("%s remote set-url origin %s" % (ud.basecmd,
>> repourl), d, workdir=destdir) +
>> + if self._contains_lfs(ud, d, destdir):
>> + if need_lfs and not self._find_git_lfs(d):
>> + raise bb.fetch2.FetchError("Repository %s has LFS
>> content, install git-lfs on host to download (or set lfs=0 to ignore
>> it)" % (repourl))
>> + else:
>> + bb.note("Repository %s has LFS content but it is not
>> being fetched" % (repourl)) +
>> if not ud.nocheckout:
>> if subdir != "":
>> runfetchcmd("%s read-tree %s%s" % (ud.basecmd,
>> ud.revisions[ud.names[0]], readpathspec), d, @@ -522,9 +519,17 @@
>> class Git(FetchMethod): def clean(self, ud, d):
>> """ clean the git directory """
>>
>> - bb.utils.remove(ud.localpath, True)
>> - bb.utils.remove(ud.fullmirror)
>> - bb.utils.remove(ud.fullmirror + ".done")
>> + to_remove = [ud.localpath, ud.fullmirror, ud.fullmirror +
>> ".done"]
>> + # The localpath is a symlink to clonedir when it is cloned
>> from a
>> + # mirror, so remove both of them.
>> + if os.path.islink(ud.localpath):
>> + clonedir = os.path.realpath(ud.localpath)
>> + to_remove.append(clonedir)
>> +
>> + for r in to_remove:
>> + if os.path.exists(r):
>> + bb.note('Removing %s' % r)
>> + bb.utils.remove(r, True)
>>
>> def supports_srcrev(self):
>> return True
>> @@ -545,6 +550,27 @@ class Git(FetchMethod):
>> raise bb.fetch2.FetchError("The command '%s' gave output
>> with more then 1 line unexpectedly, output: '%s'" % (cmd, output))
>> return output.split()[0] != "0"
>> + def _contains_lfs(self, ud, d, wd):
>> + """
>> + Check if the repository has 'lfs' (large file) content
>> + """
>> + cmd = "%s grep lfs HEAD:.gitattributes | wc -l" % (
>> + ud.basecmd)
>> + try:
>> + output = runfetchcmd(cmd, d, quiet=True, workdir=wd)
>> + if int(output) > 0:
>> + return True
>> + except (bb.fetch2.FetchError,ValueError):
>> + pass
>> + return False
>> +
>> + def _find_git_lfs(self, d):
>> + """
>> + Return True if git-lfs can be found, False otherwise.
>> + """
>> + import shutil
>> + return shutil.which("git-lfs", path=d.getVar('PATH')) is not
>> None +
>> def _get_repo_url(self, ud):
>> """
>> Return the repository URL
>> @@ -615,7 +641,7 @@ class Git(FetchMethod):
>> """
>> pupver = ('', '')
>>
>> - tagregex = re.compile(d.getVar('UPSTREAM_CHECK_GITTAGREGEX')
>> or "(?P<pver>([0-9][\.|_]?)+)")
>> + tagregex = re.compile(d.getVar('UPSTREAM_CHECK_GITTAGREGEX')
>> or r"(?P<pver>([0-9][\.|_]?)+)") try:
>> output = self._lsremote(ud, d, "refs/tags/*")
>> except (bb.fetch2.FetchError, bb.fetch2.NetworkAccess) as e:
>> @@ -630,7 +656,7 @@ class Git(FetchMethod):
>>
>> tag_head = line.split("/")[-1]
>> # Ignore non-released branches
>> - m = re.search("(alpha|beta|rc|final)+", tag_head)
>> + m = re.search(r"(alpha|beta|rc|final)+", tag_head)
>> if m:
>> continue
>>
>> diff --git a/bitbake/lib/bb/fetch2/gitannex.py
>> b/bitbake/lib/bb/fetch2/gitannex.py index a9b69ca..1d497dc 100644
>> --- a/bitbake/lib/bb/fetch2/gitannex.py
>> +++ b/bitbake/lib/bb/fetch2/gitannex.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake 'Fetch' git annex implementation
>> """
>> @@ -7,18 +5,8 @@ BitBake 'Fetch' git annex implementation
>> # Copyright (C) 2014 Otavio Salvador
>> # Copyright (C) 2014 O.S. Systems Software LTDA.
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import os
>> import bb
>> diff --git a/bitbake/lib/bb/fetch2/gitsm.py
>> b/bitbake/lib/bb/fetch2/gitsm.py index 35729db..c622771 100644
>> --- a/bitbake/lib/bb/fetch2/gitsm.py
>> +++ b/bitbake/lib/bb/fetch2/gitsm.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake 'Fetch' git submodules implementation
>>
>> @@ -16,18 +14,8 @@ NOTE: Switching a SRC_URI from "git://" to
>> "gitsm://" requires a clean of your r
>> # Copyright (C) 2013 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import os
>> import bb
>> @@ -45,60 +33,97 @@ class GitSM(Git):
>> """
>> return ud.type in ['gitsm']
>>
>> - @staticmethod
>> - def parse_gitmodules(gitmodules):
>> - modules = {}
>> - module = ""
>> - for line in gitmodules.splitlines():
>> - if line.startswith('[submodule'):
>> - module = line.split('"')[1]
>> - modules[module] = {}
>> - elif module and line.strip().startswith('path'):
>> - path = line.split('=')[1].strip()
>> - modules[module]['path'] = path
>> - elif module and line.strip().startswith('url'):
>> - url = line.split('=')[1].strip()
>> - modules[module]['url'] = url
>> - return modules
>> -
>> - def update_submodules(self, ud, d):
>> + def process_submodules(self, ud, workdir, function, d):
>> + """
>> + Iterate over all of the submodules in this repository and
>> execute
>> + the 'function' for each of them.
>> + """
>> +
>> submodules = []
>> paths = {}
>> + revision = {}
>> uris = {}
>> - local_paths = {}
>> -
>> + subrevision = {}
>> +
>> + def parse_gitmodules(gitmodules):
>> + modules = {}
>> + module = ""
>> + for line in gitmodules.splitlines():
>> + if line.startswith('[submodule'):
>> + module = line.split('"')[1]
>> + modules[module] = {}
>> + elif module and line.strip().startswith('path'):
>> + path = line.split('=')[1].strip()
>> + modules[module]['path'] = path
>> + elif module and line.strip().startswith('url'):
>> + url = line.split('=')[1].strip()
>> + modules[module]['url'] = url
>> + return modules
>> +
>> + # Collect the defined submodules, and their attributes
>> for name in ud.names:
>> try:
>> - gitmodules = runfetchcmd("%s show %s:.gitmodules" %
>> (ud.basecmd, ud.revisions[name]), d, quiet=True, workdir=ud.clonedir)
>> + gitmodules = runfetchcmd("%s show %s:.gitmodules" %
>> (ud.basecmd, ud.revisions[name]), d, quiet=True, workdir=workdir)
>> except: # No submodules to update
>> continue
>>
>> - for m, md in self.parse_gitmodules(gitmodules).items():
>> + for m, md in parse_gitmodules(gitmodules).items():
>> + try:
>> + module_hash = runfetchcmd("%s ls-tree -z -d %s
>> %s" % (ud.basecmd, ud.revisions[name], md['path']), d, quiet=True,
>> workdir=workdir)
>> + except:
>> + # If the command fails, we don't have a valid
>> file to check. If it doesn't
>> + # fail -- it still might be a failure, see next
>> check...
>> + module_hash = ""
>> +
>> + if not module_hash:
>> + logger.debug(1, "submodule %s is defined, but is
>> not initialized in the repository. Skipping", m)
>> + continue
>> +
>> submodules.append(m)
>> paths[m] = md['path']
>> + revision[m] = ud.revisions[name]
>> uris[m] = md['url']
>> + subrevision[m] = module_hash.split()[2]
>> +
>> + # Convert relative to absolute uri based on parent
>> uri if uris[m].startswith('..'):
>> newud = copy.copy(ud)
>> - newud.path =
>> os.path.realpath(os.path.join(newud.path, md['url']))
>> + newud.path =
>> os.path.realpath(os.path.join(newud.path, uris[m])) uris[m] =
>> Git._get_repo_url(self, newud)
>> for module in submodules:
>> - module_hash = runfetchcmd("%s ls-tree -z -d %s %s" %
>> (ud.basecmd, ud.revisions[name], paths[module]), d, quiet=True,
>> workdir=ud.clonedir)
>> - module_hash = module_hash.split()[2]
>> + # Translate the module url into a SRC_URI
>> +
>> + if "://" in uris[module]:
>> + # Properly formated URL already
>> + proto = uris[module].split(':', 1)[0]
>> + url = uris[module].replace('%s:' % proto, 'gitsm:',
>> 1)
>> + else:
>> + if ":" in uris[module]:
>> + # Most likely an SSH style reference
>> + proto = "ssh"
>> + if ":/" in uris[module]:
>> + # Absolute reference, easy to convert..
>> + url = "gitsm://" +
>> uris[module].replace(':/', '/', 1)
>> + else:
>> + # Relative reference, no way to know if this
>> is right!
>> + logger.warning("Submodule included by %s
>> refers to relative ssh reference %s. References may fail if not
>> absolute." % (ud.url, uris[module]))
>> + url = "gitsm://" + uris[module].replace(':',
>> '/', 1)
>> + else:
>> + # This has to be a file reference
>> + proto = "file"
>> + url = "gitsm://" + uris[module]
>>
>> - # Build new SRC_URI
>> - proto = uris[module].split(':', 1)[0]
>> - url = uris[module].replace('%s:' % proto, 'gitsm:', 1)
>> url += ';protocol=%s' % proto
>> url += ";name=%s" % module
>> - url += ";bareclone=1;nocheckout=1;nobranch=1"
>> + url += ";subpath=%s" % module
>>
>> ld = d.createCopy()
>> # Not necessary to set SRC_URI, since we're passing the
>> URI to # Fetch.
>> #ld.setVar('SRC_URI', url)
>> - ld.setVar('SRCREV_%s' % module, module_hash)
>> + ld.setVar('SRCREV_%s' % module, subrevision[module])
>>
>> # Workaround for issues with SRCPV/SRCREV_FORMAT errors
>> # error refer to 'multiple' repositories. Only the
>> repository @@ -106,145 +131,85 @@ class GitSM(Git):
>> ld.setVar('SRCPV', d.getVar('SRCPV'))
>> ld.setVar('SRCREV_FORMAT', module)
>>
>> - newfetch = Fetch([url], ld, cache=False)
>> - newfetch.download()
>> - local_paths[module] = newfetch.localpath(url)
>> + function(ud, url, module, paths[module], ld)
>>
>> - # Correct the submodule references to the local download
>> version...
>> - runfetchcmd("%(basecmd)s config submodule.%(module)s.url
>> %(url)s" % {'basecmd': ud.basecmd, 'module': module, 'url' :
>> local_paths[module]}, d, workdir=ud.clonedir) -
>> - symlink_path = os.path.join(ud.clonedir, 'modules',
>> paths[module])
>> - if not os.path.exists(symlink_path):
>> - try:
>> - os.makedirs(os.path.dirname(symlink_path),
>> exist_ok=True)
>> - except OSError:
>> - pass
>> - os.symlink(local_paths[module], symlink_path)
>> -
>> - return True
>> + return submodules != []
>>
>> def need_update(self, ud, d):
>> - main_repo_needs_update = Git.need_update(self, ud, d)
>> -
>> - # First check that the main repository has enough history
>> fetched. If it doesn't, then we don't
>> - # even have the .gitmodules and gitlinks for the submodules
>> to attempt asking whether the
>> - # submodules' histories are recent enough.
>> - if main_repo_needs_update:
>> + if Git.need_update(self, ud, d):
>> return True
>>
>> - # Now check that the submodule histories are new enough. The
>> git-submodule command doesn't have
>> - # any clean interface for doing this aside from just
>> attempting the checkout (with network
>> - # fetched disabled).
>> - return not self.update_submodules(ud, d)
>> + try:
>> + # Check for the nugget dropped by the download operation
>> + known_srcrevs = runfetchcmd("%s config --get-all
>> bitbake.srcrev" % \
>> + (ud.basecmd), d, workdir=ud.clonedir)
>>
>> - def download(self, ud, d):
>> - Git.download(self, ud, d)
>> + if ud.revisions[ud.names[0]] not in
>> known_srcrevs.split():
>> + return True
>> + except bb.fetch2.FetchError:
>> + # No srcrev nuggets, so this is new and needs to be
>> updated
>> + return True
>>
>> - if not ud.shallow or ud.localpath != ud.fullshallow:
>> - self.update_submodules(ud, d)
>> + return False
>>
>> - def copy_submodules(self, submodules, ud, destdir, d):
>> - if ud.bareclone:
>> - repo_conf = destdir
>> - else:
>> - repo_conf = os.path.join(destdir, '.git')
>> + def download(self, ud, d):
>> + def download_submodule(ud, url, module, modpath, d):
>> + url += ";bareclone=1;nobranch=1"
>>
>> - if submodules and not os.path.exists(os.path.join(repo_conf,
>> 'modules')):
>> - os.mkdir(os.path.join(repo_conf, 'modules'))
>> + # Is the following still needed?
>> + #url += ";nocheckout=1"
>>
>> - for module, md in submodules.items():
>> - srcpath = os.path.join(ud.clonedir, 'modules',
>> md['path'])
>> - modpath = os.path.join(repo_conf, 'modules', md['path'])
>> + try:
>> + newfetch = Fetch([url], d, cache=False)
>> + newfetch.download()
>> + # Drop a nugget to add each of the srcrevs we've
>> fetched (used by need_update)
>> + runfetchcmd("%s config --add bitbake.srcrev %s" % \
>> + (ud.basecmd, ud.revisions[ud.names[0]]),
>> d, workdir=ud.clonedir)
>> + except Exception as e:
>> + logger.error('gitsm: submodule download failed: %s
>> %s' % (type(e).__name__, str(e)))
>> + raise
>>
>> - if os.path.exists(srcpath):
>> - if os.path.exists(os.path.join(srcpath, '.git')):
>> - srcpath = os.path.join(srcpath, '.git')
>> + Git.download(self, ud, d)
>> + self.process_submodules(ud, ud.clonedir, download_submodule,
>> d)
>> - target = modpath
>> - if os.path.exists(modpath):
>> - target = os.path.dirname(modpath)
>> + def unpack(self, ud, destdir, d):
>> + def unpack_submodules(ud, url, module, modpath, d):
>> + url += ";bareclone=1;nobranch=1"
>>
>> - os.makedirs(os.path.dirname(target), exist_ok=True)
>> - runfetchcmd("cp -fpLR %s %s" % (srcpath, target), d)
>> - elif os.path.exists(modpath):
>> - # Module already exists, likely unpacked from a
>> shallow mirror clone
>> - pass
>> + # Figure out where we clone over the bare submodules...
>> + if ud.bareclone:
>> + repo_conf = ud.destdir
>> else:
>> - # This is fatal, as we do NOT want git-submodule to
>> hit the network
>> - raise bb.fetch2.FetchError('Submodule %s does not
>> exist in %s or %s.' % (module, srcpath, modpath)) -
>> - def clone_shallow_local(self, ud, dest, d):
>> - super(GitSM, self).clone_shallow_local(ud, dest, d)
>> + repo_conf = os.path.join(ud.destdir, '.git')
>>
>> - # Copy over the submodules' fetched histories too.
>> - repo_conf = os.path.join(dest, '.git')
>> -
>> - submodules = []
>> - for name in ud.names:
>> try:
>> - gitmodules = runfetchcmd("%s show %s:.gitmodules" %
>> (ud.basecmd, ud.revision), d, quiet=True, workdir=dest)
>> - except:
>> - # No submodules to update
>> - continue
>> + newfetch = Fetch([url], d, cache=False)
>> +
>> newfetch.unpack(root=os.path.dirname(os.path.join(repo_conf,
>> 'modules', module)))
>> + except Exception as e:
>> + logger.error('gitsm: submodule unpack failed: %s %s'
>> % (type(e).__name__, str(e)))
>> + raise
>>
>> - submodules = self.parse_gitmodules(gitmodules)
>> - self.copy_submodules(submodules, ud, dest, d)
>> + local_path = newfetch.localpath(url)
>>
>> - def unpack(self, ud, destdir, d):
>> - Git.unpack(self, ud, destdir, d)
>> + # Correct the submodule references to the local download
>> version...
>> + runfetchcmd("%(basecmd)s config submodule.%(module)s.url
>> %(url)s" % {'basecmd': ud.basecmd, 'module': module, 'url' :
>> local_path}, d, workdir=ud.destdir)
>> - # Copy over the submodules' fetched histories too.
>> - if ud.bareclone:
>> - repo_conf = ud.destdir
>> - else:
>> - repo_conf = os.path.join(ud.destdir, '.git')
>> + if ud.shallow:
>> + runfetchcmd("%(basecmd)s config
>> submodule.%(module)s.shallow true" % {'basecmd': ud.basecmd,
>> 'module': module}, d, workdir=ud.destdir)
>> - update_submodules = False
>> - paths = {}
>> - uris = {}
>> - local_paths = {}
>> - for name in ud.names:
>> + # Ensure the submodule repository is NOT set to bare,
>> since we're checking it out... try:
>> - gitmodules = runfetchcmd("%s show HEAD:.gitmodules"
>> % (ud.basecmd), d, quiet=True, workdir=ud.destdir)
>> + runfetchcmd("%s config core.bare false" %
>> (ud.basecmd), d, quiet=True, workdir=os.path.join(repo_conf,
>> 'modules', module)) except:
>> - # No submodules to update
>> - continue
>> -
>> - submodules = self.parse_gitmodules(gitmodules)
>> - self.copy_submodules(submodules, ud, ud.destdir, d)
>> -
>> - submodules_queue = [(module, os.path.join(repo_conf,
>> 'modules', md['path'])) for module, md in submodules.items()]
>> - while len(submodules_queue) != 0:
>> - module, modpath = submodules_queue.pop()
>> -
>> - # add submodule children recursively
>> - try:
>> - gitmodules = runfetchcmd("%s show
>> HEAD:.gitmodules" % (ud.basecmd), d, quiet=True, workdir=modpath)
>> - for m, md in
>> self.parse_gitmodules(gitmodules).items():
>> - submodules_queue.append([m,
>> os.path.join(modpath, 'modules', md['path'])])
>> - except:
>> - # no children
>> - pass
>> -
>> + logger.error("Unable to set git config core.bare to
>> false for %s" % os.path.join(repo_conf, 'modules', module))
>> + raise
>>
>> - # There are submodules to update
>> - update_submodules = True
>> -
>> - # Determine (from the submodule) the correct url to
>> reference
>> - try:
>> - output = runfetchcmd("%(basecmd)s config
>> remote.origin.url" % {'basecmd': ud.basecmd}, d, workdir=modpath)
>> - except bb.fetch2.FetchError as e:
>> - # No remote url defined in this submodule
>> - continue
>> -
>> - local_paths[module] = output
>> -
>> - # Setup the local URL properly (like git submodule
>> init or sync would do...)
>> - runfetchcmd("%(basecmd)s config
>> submodule.%(module)s.url %(url)s" % {'basecmd': ud.basecmd, 'module':
>> module, 'url' : local_paths[module]}, d, workdir=ud.destdir)
>> + Git.unpack(self, ud, destdir, d)
>>
>> - # Ensure the submodule repository is NOT set to
>> bare, since we're checking it out...
>> - runfetchcmd("%s config core.bare false" %
>> (ud.basecmd), d, quiet=True, workdir=modpath)
>> + ret = self.process_submodules(ud, ud.destdir,
>> unpack_submodules, d)
>> - if update_submodules:
>> - # Run submodule update, this sets up the directories --
>> without touching the config
>> + if not ud.bareclone and ret:
>> + # All submodules should already be downloaded and
>> configured in the tree. This simply sets
>> + # up the configuration and checks out the files. The
>> main project config should remain
>> + # unmodified, and no download from the internet should
>> occur. runfetchcmd("%s submodule update --recursive --no-fetch" %
>> (ud.basecmd), d, quiet=True, workdir=ud.destdir) diff --git
>> a/bitbake/lib/bb/fetch2/hg.py b/bitbake/lib/bb/fetch2/hg.py index
>> 936d043..15d729e 100644 --- a/bitbake/lib/bb/fetch2/hg.py
>> +++ b/bitbake/lib/bb/fetch2/hg.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake 'Fetch' implementation for mercurial DRCS (hg).
>>
>> @@ -9,20 +7,10 @@ BitBake 'Fetch' implementation for mercurial DRCS
>> (hg). # Copyright (C) 2004 Marcin Juszkiewicz
>> # Copyright (C) 2007 Robert Schuster
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> # Based on functions from the base bb module, Copyright 2003 Holger
>> Schurig +#
>>
>> import os
>> import sys
>> @@ -99,7 +87,7 @@ class Hg(FetchMethod):
>> def try_premirror(self, ud, d):
>> # If we don't do this, updating an existing checkout with
>> only premirrors # is not possible
>> - if d.getVar("BB_FETCH_PREMIRRORONLY") is not None:
>> + if bb.utils.to_boolean(d.getVar("BB_FETCH_PREMIRRORONLY")):
>> return True
>> if os.path.exists(ud.moddir):
>> return False
>> diff --git a/bitbake/lib/bb/fetch2/local.py
>> b/bitbake/lib/bb/fetch2/local.py index a114ac1..01d9ff9 100644
>> --- a/bitbake/lib/bb/fetch2/local.py
>> +++ b/bitbake/lib/bb/fetch2/local.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake 'Fetch' implementations
>>
>> @@ -10,20 +8,10 @@ BitBake build tools.
>>
>> # Copyright (C) 2003, 2004 Chris Larson
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> # Based on functions from the base bb module, Copyright 2003 Holger
>> Schurig +#
>>
>> import os
>> import urllib.request, urllib.parse, urllib.error
>> diff --git a/bitbake/lib/bb/fetch2/npm.py
>> b/bitbake/lib/bb/fetch2/npm.py index 65bf5a3..9700e61 100644
>> --- a/bitbake/lib/bb/fetch2/npm.py
>> +++ b/bitbake/lib/bb/fetch2/npm.py
>> @@ -1,5 +1,6 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> """
>> BitBake 'Fetch' NPM implementation
>>
>> @@ -100,11 +101,19 @@ class Npm(FetchMethod):
>> return False
>> return True
>>
>> - def _runwget(self, ud, d, command, quiet):
>> - logger.debug(2, "Fetching %s using command '%s'" % (ud.url,
>> command))
>> - bb.fetch2.check_network_access(d, command, ud.url)
>> + def _runpack(self, ud, d, pkgfullname: str, quiet=False) -> str:
>> + """
>> + Runs npm pack on a full package name.
>> + Returns the filename of the downloaded package
>> + """
>> + bb.fetch2.check_network_access(d, pkgfullname, ud.registry)
>> dldir = d.getVar("DL_DIR")
>> - runfetchcmd(command, d, quiet, workdir=dldir)
>> + dldir = os.path.join(dldir, ud.prefixdir)
>> +
>> + command = "npm pack {} --registry {}".format(pkgfullname,
>> ud.registry)
>> + logger.debug(2, "Fetching {} using command '{}' in
>> {}".format(pkgfullname, command, dldir))
>> + filename = runfetchcmd(command, d, quiet, workdir=dldir)
>> + return filename.rstrip()
>>
>> def _unpackdep(self, ud, pkg, data, destdir, dldir, d):
>> file = data[pkg]['tgz']
>> @@ -150,20 +159,11 @@ class Npm(FetchMethod):
>> Parse the output of npm view --json; the last JSON result
>> is assumed to be the one that we're interested in.
>> '''
>> - pdata = None
>> - outdeps = {}
>> - datalines = []
>> - bracelevel = 0
>> - for line in output.splitlines():
>> - if bracelevel:
>> - datalines.append(line)
>> - elif '{' in line:
>> - datalines = []
>> - datalines.append(line)
>> - bracelevel = bracelevel + line.count('{') -
>> line.count('}')
>> - if datalines:
>> - pdata = json.loads('\n'.join(datalines))
>> - return pdata
>> + pdata = json.loads(output);
>> + try:
>> + return pdata[-1]
>> + except:
>> + return pdata
>>
>> def _getdependencies(self, pkg, data, version, d, ud,
>> optional=False, fetchedlist=None): if fetchedlist is None:
>> @@ -171,6 +171,9 @@ class Npm(FetchMethod):
>> pkgfullname = pkg
>> if version != '*' and not '/' in version:
>> pkgfullname += "@'%s'" % version
>> + if pkgfullname in fetchedlist:
>> + return
>> +
>> logger.debug(2, "Calling getdeps on %s" % pkg)
>> fetchcmd = "npm view %s --json --registry %s" %
>> (pkgfullname, ud.registry) output = runfetchcmd(fetchcmd, d, True)
>> @@ -190,15 +193,10 @@ class Npm(FetchMethod):
>> if (not blacklist and 'linux' not in pkg_os) or
>> '!linux' in pkg_os: logger.debug(2, "Skipping %s since it's
>> incompatible with Linux" % pkg) return
>> - #logger.debug(2, "Output URL is %s - %s - %s" %
>> (ud.basepath, ud.basename, ud.localfile))
>> - outputurl = pdata['dist']['tarball']
>> + filename = self._runpack(ud, d, pkgfullname)
>> data[pkg] = {}
>> - data[pkg]['tgz'] = os.path.basename(outputurl)
>> - if outputurl in fetchedlist:
>> - return
>> -
>> - self._runwget(ud, d, "%s --directory-prefix=%s %s" %
>> (self.basecmd, ud.prefixdir, outputurl), False)
>> - fetchedlist.append(outputurl)
>> + data[pkg]['tgz'] = filename
>> + fetchedlist.append(pkgfullname)
>>
>> dependencies = pdata.get('dependencies', {})
>> optionalDependencies = pdata.get('optionalDependencies', {})
>> @@ -225,17 +223,12 @@ class Npm(FetchMethod):
>> if obj == pkg:
>> self._getshrinkeddependencies(obj,
>> data['dependencies'][obj], data['dependencies'][obj]['version'], d,
>> ud, lockdown, manifest, False) return
>> - outputurl = "invalid"
>> - if ('resolved' not in data) or (not
>> data['resolved'].startswith('http://') and not
>> data['resolved'].startswith('https://')):
>> - # will be the case for ${PN}
>> - fetchcmd = "npm view %s@%s dist.tarball --registry %s" %
>> (pkg, version, ud.registry)
>> - logger.debug(2, "Found this matching URL: %s" %
>> str(fetchcmd))
>> - outputurl = runfetchcmd(fetchcmd, d, True)
>> - else:
>> - outputurl = data['resolved']
>> - self._runwget(ud, d, "%s --directory-prefix=%s %s" %
>> (self.basecmd, ud.prefixdir, outputurl), False) +
>> + pkgnameWithVersion = "{}@{}".format(pkg, version)
>> + logger.debug(2, "Get dependencies for
>> {}".format(pkgnameWithVersion))
>> + filename = self._runpack(ud, d, pkgnameWithVersion)
>> manifest[pkg] = {}
>> - manifest[pkg]['tgz'] = os.path.basename(outputurl).rstrip()
>> + manifest[pkg]['tgz'] = filename
>> manifest[pkg]['deps'] = {}
>>
>> if pkg in lockdown:
>> diff --git a/bitbake/lib/bb/fetch2/osc.py
>> b/bitbake/lib/bb/fetch2/osc.py index 6c60456..3e56715 100644
>> --- a/bitbake/lib/bb/fetch2/osc.py
>> +++ b/bitbake/lib/bb/fetch2/osc.py
>> @@ -1,5 +1,6 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> """
>> Bitbake "Fetch" implementation for osc (Opensuse build service
>> client). Based on the svn "Fetch" implementation.
>> diff --git a/bitbake/lib/bb/fetch2/perforce.py
>> b/bitbake/lib/bb/fetch2/perforce.py index 903a8e6..54d001e 100644
>> --- a/bitbake/lib/bb/fetch2/perforce.py
>> +++ b/bitbake/lib/bb/fetch2/perforce.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake 'Fetch' implementation for perforce
>>
>> @@ -8,18 +6,7 @@ BitBake 'Fetch' implementation for perforce
>> # Copyright (C) 2003, 2004 Chris Larson
>> # Copyright (C) 2016 Kodak Alaris, Inc.
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> # Based on functions from the base bb module, Copyright 2003 Holger
>> Schurig
>> diff --git a/bitbake/lib/bb/fetch2/repo.py
>> b/bitbake/lib/bb/fetch2/repo.py index 8c7e818..2bdbbd4 100644
>> --- a/bitbake/lib/bb/fetch2/repo.py
>> +++ b/bitbake/lib/bb/fetch2/repo.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake "Fetch" repo (git) implementation
>>
>> @@ -8,20 +6,10 @@ BitBake "Fetch" repo (git) implementation
>> # Copyright (C) 2009 Tom Rini <trini@embeddedalley.com>
>> #
>> # Based on git.py which is:
>> -#Copyright (C) 2005 Richard Purdie
>> +# Copyright (C) 2005 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import os
>> import bb
>> diff --git a/bitbake/lib/bb/fetch2/s3.py b/bitbake/lib/bb/fetch2/s3.py
>> index 1629288..ffca73c 100644
>> --- a/bitbake/lib/bb/fetch2/s3.py
>> +++ b/bitbake/lib/bb/fetch2/s3.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake 'Fetch' implementation for Amazon AWS S3.
>>
>> @@ -13,18 +11,7 @@ The aws tool must be correctly installed and
>> configured prior to use. # Based in part on bb.fetch2.wget:
>> # Copyright (C) 2003, 2004 Chris Larson
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> # Based on functions from the base bb module, Copyright 2003 Holger
>> Schurig
>> diff --git a/bitbake/lib/bb/fetch2/sftp.py
>> b/bitbake/lib/bb/fetch2/sftp.py index 81884a6..f87f292 100644
>> --- a/bitbake/lib/bb/fetch2/sftp.py
>> +++ b/bitbake/lib/bb/fetch2/sftp.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake SFTP Fetch implementation
>>
>> @@ -44,18 +42,7 @@ SRC_URI =
>> "sftp://user@host.example.com/dir/path.file.txt" # Based in part on
>> bb.fetch2.wget: # Copyright (C) 2003, 2004 Chris Larson
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> # Based on functions from the base bb module, Copyright 2003 Holger
>> Schurig
>> diff --git a/bitbake/lib/bb/fetch2/ssh.py
>> b/bitbake/lib/bb/fetch2/ssh.py index 6047ee4..f5be060 100644
>> --- a/bitbake/lib/bb/fetch2/ssh.py
>> +++ b/bitbake/lib/bb/fetch2/ssh.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> '''
>> BitBake 'Fetch' implementations
>>
>> @@ -29,18 +27,8 @@ IETF secsh internet draft:
>> # Copyright 2003 Holger Schurig
>> #
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import re, os
>> from bb.fetch2 import FetchMethod
>> diff --git a/bitbake/lib/bb/fetch2/svn.py
>> b/bitbake/lib/bb/fetch2/svn.py index ed70bcf..96d666b 100644
>> --- a/bitbake/lib/bb/fetch2/svn.py
>> +++ b/bitbake/lib/bb/fetch2/svn.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake 'Fetch' implementation for svn.
>>
>> @@ -8,18 +6,7 @@ BitBake 'Fetch' implementation for svn.
>> # Copyright (C) 2003, 2004 Chris Larson
>> # Copyright (C) 2004 Marcin Juszkiewicz
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> # Based on functions from the base bb module, Copyright 2003 Holger
>> Schurig
>> @@ -63,6 +50,9 @@ class Svn(FetchMethod):
>> relpath = self._strip_leading_slashes(ud.path)
>> ud.pkgdir = os.path.join(svndir, ud.host, relpath)
>> ud.moddir = os.path.join(ud.pkgdir, ud.module)
>> + # Protects the repository from concurrent updates, e.g. from
>> two
>> + # recipes fetching different revisions at the same time
>> + ud.svnlock = os.path.join(ud.pkgdir, "svn.lock")
>>
>> ud.setup_revisions(d)
>>
>> @@ -101,6 +91,13 @@ class Svn(FetchMethod):
>> svncmd = "%s log --limit 1 %s %s://%s/%s/" %
>> (ud.basecmd, " ".join(options), proto, svnroot, ud.module) else:
>> suffix = ""
>> +
>> + # externals may be either 'allowed' or 'nowarn', but not
>> both. Allowed
>> + # will not issue a warning, but will log to the debug
>> buffer what has likely
>> + # been downloaded by SVN.
>> + if not ("externals" in ud.parm and ud.parm["externals"]
>> == "allowed"):
>> + options.append("--ignore-externals")
>> +
>> if ud.revision:
>> options.append("-r %s" % ud.revision)
>> suffix = "@%s" % (ud.revision)
>> @@ -123,35 +120,52 @@ class Svn(FetchMethod):
>>
>> logger.debug(2, "Fetch: checking for module directory '" +
>> ud.moddir + "'")
>> - if os.access(os.path.join(ud.moddir, '.svn'), os.R_OK):
>> - svnupdatecmd = self._buildsvncommand(ud, d, "update")
>> - logger.info("Update " + ud.url)
>> - # We need to attempt to run svn upgrade first in case
>> its an older working format
>> - try:
>> - runfetchcmd(ud.basecmd + " upgrade", d,
>> workdir=ud.moddir)
>> - except FetchError:
>> - pass
>> - logger.debug(1, "Running %s", svnupdatecmd)
>> - bb.fetch2.check_network_access(d, svnupdatecmd, ud.url)
>> - runfetchcmd(svnupdatecmd, d, workdir=ud.moddir)
>> - else:
>> - svnfetchcmd = self._buildsvncommand(ud, d, "fetch")
>> - logger.info("Fetch " + ud.url)
>> - # check out sources there
>> - bb.utils.mkdirhier(ud.pkgdir)
>> - logger.debug(1, "Running %s", svnfetchcmd)
>> - bb.fetch2.check_network_access(d, svnfetchcmd, ud.url)
>> - runfetchcmd(svnfetchcmd, d, workdir=ud.pkgdir)
>> -
>> - scmdata = ud.parm.get("scmdata", "")
>> - if scmdata == "keep":
>> - tar_flags = ""
>> - else:
>> - tar_flags = "--exclude='.svn'"
>> + lf = bb.utils.lockfile(ud.svnlock)
>> +
>> + try:
>> + if os.access(os.path.join(ud.moddir, '.svn'), os.R_OK):
>> + svnupdatecmd = self._buildsvncommand(ud, d, "update")
>> + logger.info("Update " + ud.url)
>> + # We need to attempt to run svn upgrade first in
>> case its an older working format
>> + try:
>> + runfetchcmd(ud.basecmd + " upgrade", d,
>> workdir=ud.moddir)
>> + except FetchError:
>> + pass
>> + logger.debug(1, "Running %s", svnupdatecmd)
>> + bb.fetch2.check_network_access(d, svnupdatecmd,
>> ud.url)
>> + runfetchcmd(svnupdatecmd, d, workdir=ud.moddir)
>> + else:
>> + svnfetchcmd = self._buildsvncommand(ud, d, "fetch")
>> + logger.info("Fetch " + ud.url)
>> + # check out sources there
>> + bb.utils.mkdirhier(ud.pkgdir)
>> + logger.debug(1, "Running %s", svnfetchcmd)
>> + bb.fetch2.check_network_access(d, svnfetchcmd,
>> ud.url)
>> + runfetchcmd(svnfetchcmd, d, workdir=ud.pkgdir)
>> +
>> + if not ("externals" in ud.parm and ud.parm["externals"]
>> == "nowarn"):
>> + # Warn the user if this had externals (won't catch
>> them all)
>> + output = runfetchcmd("svn propget svn:externals ||
>> true", d, workdir=ud.moddir)
>> + if output:
>> + if "--ignore-externals" in svnfetchcmd.split():
>> + bb.warn("%s contains svn:externals." %
>> ud.url)
>> + bb.warn("These should be added to the recipe
>> SRC_URI as necessary.")
>> + bb.warn("svn fetch has ignored
>> externals:\n%s" % output)
>> + bb.warn("To disable this warning add
>> ';externals=nowarn' to the url.")
>> + else:
>> + bb.debug(1, "svn repository has
>> externals:\n%s" % output) +
>> + scmdata = ud.parm.get("scmdata", "")
>> + if scmdata == "keep":
>> + tar_flags = ""
>> + else:
>> + tar_flags = "--exclude='.svn'"
>>
>> - # tar them up to a defined filename
>> - runfetchcmd("tar %s -czf %s %s" % (tar_flags, ud.localpath,
>> ud.path_spec), d,
>> - cleanup=[ud.localpath], workdir=ud.pkgdir)
>> + # tar them up to a defined filename
>> + runfetchcmd("tar %s -czf %s %s" % (tar_flags,
>> ud.localpath, ud.path_spec), d,
>> + cleanup=[ud.localpath], workdir=ud.pkgdir)
>> + finally:
>> + bb.utils.unlockfile(lf)
>>
>> def clean(self, ud, d):
>> """ Clean SVN specific files and dirs """
>> diff --git a/bitbake/lib/bb/fetch2/wget.py
>> b/bitbake/lib/bb/fetch2/wget.py index 8f505b6..725586d 100644
>> --- a/bitbake/lib/bb/fetch2/wget.py
>> +++ b/bitbake/lib/bb/fetch2/wget.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake 'Fetch' implementations
>>
>> @@ -10,18 +8,7 @@ BitBake build tools.
>>
>> # Copyright (C) 2003, 2004 Chris Larson
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> # Based on functions from the base bb module, Copyright 2003 Holger
>> Schurig
>> @@ -33,11 +20,14 @@ import logging
>> import errno
>> import bb
>> import bb.progress
>> +import socket
>> +import http.client
>> import urllib.request, urllib.parse, urllib.error
>> from bb.fetch2 import FetchMethod
>> from bb.fetch2 import FetchError
>> from bb.fetch2 import logger
>> from bb.fetch2 import runfetchcmd
>> +from bb.fetch2 import FetchConnectionCache
>> from bb.utils import export_proxies
>> from bs4 import BeautifulSoup
>> from bs4 import SoupStrainer
>> @@ -132,10 +122,6 @@ class Wget(FetchMethod):
>> return True
>>
>> def checkstatus(self, fetch, ud, d, try_again=True):
>> - import urllib.request, urllib.error, urllib.parse, socket,
>> http.client
>> - from urllib.response import addinfourl
>> - from bb.fetch2 import FetchConnectionCache
>> -
>> class HTTPConnectionCache(http.client.HTTPConnection):
>> if fetch.connection_cache:
>> def connect(self):
>> @@ -168,7 +154,7 @@ class Wget(FetchMethod):
>> """
>> host = req.host
>> if not host:
>> - raise urlllib2.URLError('no host given')
>> + raise urllib.error.URLError('no host given')
>>
>> h = http_class(host, timeout=req.timeout) # will
>> parse host:port h.set_debuglevel(self._debuglevel)
>> @@ -185,7 +171,7 @@ class Wget(FetchMethod):
>> # request.
>>
>> # Don't close connection when connection_cache is
>> enabled,
>> - if fetch.connection_cache is None:
>> + if fetch.connection_cache is None:
>> headers["Connection"] = "close"
>> else:
>> headers["Connection"] = "Keep-Alive" # Works for
>> HTTP/1.0 @@ -252,7 +238,7 @@ class Wget(FetchMethod):
>> pass
>> closed = False
>>
>> - resp = addinfourl(fp_dummy(), r.msg,
>> req.get_full_url())
>> + resp = urllib.response.addinfourl(fp_dummy(), r.msg,
>> req.get_full_url()) resp.code = r.status
>> resp.msg = r.reason
>>
>> @@ -271,17 +257,18 @@ class Wget(FetchMethod):
>> fp.read()
>> fp.close()
>>
>> - newheaders = dict((k,v) for k,v in
>> list(req.headers.items())
>> - if k.lower() not in
>> ("content-length", "content-type"))
>> - return
>> self.parent.open(urllib.request.Request(req.get_full_url(),
>> -
>> headers=newheaders,
>> -
>> origin_req_host=req.origin_req_host,
>> -
>> unverifiable=True))
>> + if req.get_method() != 'GET':
>> + newheaders = dict((k, v) for k, v in
>> list(req.headers.items())
>> + if k.lower() not in
>> ("content-length", "content-type"))
>> + return
>> self.parent.open(urllib.request.Request(req.get_full_url(),
>> +
>> headers=newheaders,
>> +
>> origin_req_host=req.origin_req_host,
>> +
>> unverifiable=True))
>> - """
>> - Some servers (e.g. GitHub archives, hosted on Amazon S3)
>> return 403
>> - Forbidden when they actually mean 405 Method Not Allowed.
>> - """
>> + raise urllib.request.HTTPError(req, code, msg,
>> headers, None) +
>> + # Some servers (e.g. GitHub archives, hosted on Amazon
>> S3) return 403
>> + # Forbidden when they actually mean 405 Method Not
>> Allowed. http_error_403 = http_error_405
>>
>>
>> @@ -292,15 +279,15 @@ class Wget(FetchMethod):
>> """
>> def redirect_request(self, req, fp, code, msg, headers,
>> newurl): newreq =
>> urllib.request.HTTPRedirectHandler.redirect_request(self, req, fp,
>> code, msg, headers, newurl)
>> - newreq.get_method = lambda: req.get_method()
>> + newreq.get_method = req.get_method
>> return newreq
>> exported_proxies = export_proxies(d)
>>
>> handlers = [FixedHTTPRedirectHandler, HTTPMethodFallback]
>> - if export_proxies:
>> + if exported_proxies:
>> handlers.append(urllib.request.ProxyHandler())
>> handlers.append(CacheHTTPHandler())
>> - # XXX: Since Python 2.7.9 ssl cert validation is enabled by
>> default
>> + # Since Python 2.7.9 ssl cert validation is enabled by
>> default # see PEP-0476, this causes verification errors on some https
>> servers # so disable by default.
>> import ssl
>> @@ -319,19 +306,19 @@ class Wget(FetchMethod):
>> '''Adds Basic auth to http request, pass in
>> login:password as string''' import base64
>> encodeuser =
>> base64.b64encode(login_str.encode('utf-8')).decode("utf-8")
>> - authheader = "Basic %s" % encodeuser
>> + authheader = "Basic %s" % encodeuser
>> r.add_header("Authorization", authheader)
>>
>> - if ud.user:
>> - add_basic_auth(ud.user, r)
>> + if ud.user and ud.pswd:
>> + add_basic_auth(ud.user + ':' + ud.pswd, r)
>>
>> try:
>> - import netrc, urllib.parse
>> + import netrc
>> n = netrc.netrc()
>> login, unused, password =
>> n.authenticators(urllib.parse.urlparse(uri).hostname)
>> add_basic_auth("%s:%s" % (login, password), r) except (TypeError,
>> ImportError, IOError, netrc.NetrcParseError):
>> - pass
>> + pass
>>
>> with opener.open(r) as response:
>> pass
>> @@ -396,18 +383,14 @@ class Wget(FetchMethod):
>> (oldpn, oldpv, oldsuffix) = old
>> (newpn, newpv, newsuffix) = new
>>
>> - """
>> - Check for a new suffix type that we have never heard of
>> before
>> - """
>> - if (newsuffix):
>> + # Check for a new suffix type that we have never heard of
>> before
>> + if newsuffix:
>> m = self.suffix_regex_comp.search(newsuffix)
>> if not m:
>> bb.warn("%s has a possible unknown suffix: %s" %
>> (newpn, newsuffix)) return False
>>
>> - """
>> - Not our package so ignore it
>> - """
>> + # Not our package so ignore it
>> if oldpn != newpn:
>> return False
>>
>> @@ -473,15 +456,14 @@ class Wget(FetchMethod):
>>
>> return ""
>>
>> - def _check_latest_version_by_dir(self, dirver, package,
>> package_regex,
>> - current_version, ud, d):
>> + def _check_latest_version_by_dir(self, dirver, package,
>> package_regex, current_version, ud, d): """
>> - Scan every directory in order to get upstream version.
>> + Scan every directory in order to get upstream version.
>> """
>> version_dir = ['', '', '']
>> version = ['', '', '']
>>
>> - dirver_regex =
>> re.compile("(?P<pfx>\D*)(?P<ver>(\d+[\.\-_])+(\d+))")
>> + dirver_regex =
>> re.compile(r"(?P<pfx>\D*)(?P<ver>(\d+[\.\-_])+(\d+))") s =
>> dirver_regex.search(dirver) if s:
>> version_dir[1] = s.group('ver')
>> @@ -541,26 +523,26 @@ class Wget(FetchMethod):
>> gst-fluendo-mp3
>> """
>> # match most patterns which uses "-" as separator to version
>> digits
>> - pn_prefix1 = "[a-zA-Z][a-zA-Z0-9]*([-_][a-zA-Z]\w+)*\+?[-_]"
>> + pn_prefix1 = r"[a-zA-Z][a-zA-Z0-9]*([-_][a-zA-Z]\w+)*\+?[-_]"
>> # a loose pattern such as for unzip552.tar.gz
>> - pn_prefix2 = "[a-zA-Z]+"
>> + pn_prefix2 = r"[a-zA-Z]+"
>> # a loose pattern such as for 80325-quicky-0.4.tar.gz
>> - pn_prefix3 = "[0-9]+[-]?[a-zA-Z]+"
>> + pn_prefix3 = r"[0-9]+[-]?[a-zA-Z]+"
>> # Save the Package Name (pn) Regex for use later
>> - pn_regex = "(%s|%s|%s)" % (pn_prefix1, pn_prefix2,
>> pn_prefix3)
>> + pn_regex = r"(%s|%s|%s)" % (pn_prefix1, pn_prefix2,
>> pn_prefix3)
>> # match version
>> - pver_regex = "(([A-Z]*\d+[a-zA-Z]*[\.\-_]*)+)"
>> + pver_regex = r"(([A-Z]*\d+[a-zA-Z]*[\.\-_]*)+)"
>>
>> # match arch
>> parch_regex = "-source|_all_"
>>
>> # src.rpm extension was added only for rpm package. Can be
>> removed if the rpm # packaged will always be considered as having to
>> be manually upgraded
>> - psuffix_regex =
>> "(tar\.gz|tgz|tar\.bz2|zip|xz|tar\.lz|rpm|bz2|orig\.tar\.gz|tar\.xz|src\.tar\.gz|src\.tgz|svnr\d+\.tar\.bz2|stable\.tar\.gz|src\.rpm)"
>> + psuffix_regex =
>> r"(tar\.gz|tgz|tar\.bz2|zip|xz|tar\.lz|rpm|bz2|orig\.tar\.gz|tar\.xz|src\.tar\.gz|src\.tgz|svnr\d+\.tar\.bz2|stable\.tar\.gz|src\.rpm)"
>> # match name, version and archive type of a package
>> - package_regex_comp =
>> re.compile("(?P<name>%s?\.?v?)(?P<pver>%s)(?P<arch>%s)?[\.-](?P<type>%s$)"
>> + package_regex_comp =
>> re.compile(r"(?P<name>%s?\.?v?)(?P<pver>%s)(?P<arch>%s)?[\.-](?P<type>%s$)"
>> % (pn_regex, pver_regex, parch_regex, psuffix_regex))
>> self.suffix_regex_comp = re.compile(psuffix_regex)
>> @@ -572,7 +554,7 @@ class Wget(FetchMethod):
>> version = self._parse_path(package_regex_comp, package)
>> if version:
>> package_custom_regex_comp = re.compile(
>> -
>> "(?P<name>%s)(?P<pver>%s)(?P<arch>%s)?[\.-](?P<type>%s)" %
>> +
>> r"(?P<name>%s)(?P<pver>%s)(?P<arch>%s)?[\.-](?P<type>%s)" %
>> (re.escape(version[0]), pver_regex, parch_regex, psuffix_regex)) else:
>> package_custom_regex_comp = None
>> @@ -589,7 +571,7 @@ class Wget(FetchMethod):
>> current_version = ['', d.getVar('PV'), '']
>>
>> """possible to have no version in pkg name, such as
>> spectrum-fw"""
>> - if not re.search("\d+", package):
>> + if not re.search(r"\d+", package):
>> current_version[1] = re.sub('_', '.', current_version[1])
>> current_version[1] = re.sub('-', '.', current_version[1])
>> return (current_version[1], '')
>> @@ -607,13 +589,13 @@ class Wget(FetchMethod):
>>
>> # search for version matches on folders inside the path,
>> like: # "5.7" in
>> http://download.gnome.org/sources/${PN}/5.7/${PN}-${PV}.tar.gz
>> - dirver_regex =
>> re.compile("(?P<dirver>[^/]*(\d+\.)*\d+([-_]r\d+)*)/")
>> + dirver_regex =
>> re.compile(r"(?P<dirver>[^/]*(\d+\.)*\d+([-_]r\d+)*)/") m =
>> dirver_regex.search(path) if m:
>> pn = d.getVar('PN')
>> dirver = m.group('dirver')
>>
>> - dirver_pn_regex = re.compile("%s\d?" %
>> (re.escape(pn)))
>> + dirver_pn_regex = re.compile(r"%s\d?" %
>> (re.escape(pn))) if not dirver_pn_regex.search(dirver):
>> return (self._check_latest_version_by_dir(dirver,
>> package, package_regex, current_version, ud,
>> d), '') diff --git a/bitbake/lib/bb/main.py b/bitbake/lib/bb/main.py
>> index 732a315..af2880f 100755
>> --- a/bitbake/lib/bb/main.py
>> +++ b/bitbake/lib/bb/main.py
>> @@ -1,6 +1,3 @@
>> -#!/usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # Copyright (C) 2003, 2004 Chris Larson
>> # Copyright (C) 2003, 2004 Phil Blundell
>> @@ -9,18 +6,8 @@
>> # Copyright (C) 2005 ROAD GmbH
>> # Copyright (C) 2006 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import os
>> import sys
>> @@ -267,6 +254,11 @@ class
>> BitBakeConfigParameters(cookerdata.ConfigParameters): help="Do not
>> run any setscene tasks. sstate will be ignored and " "everything
>> needed, built.")
>> + parser.add_option("", "--skip-setscene", action="store_true",
>> + dest="skipsetscene", default=False,
>> + help="Skip setscene tasks if they would be
>> executed. Tasks previously "
>> + "restored from sstate will be kept,
>> unlike --no-setscene") +
>> parser.add_option("", "--setscene-only", action="store_true",
>> dest="setsceneonly", default=False,
>> help="Only run setscene tasks, don't run
>> any real tasks.") @@ -448,7 +440,7 @@ def setup_bitbake(configParams,
>> configuration, extrafeatures=None): else:
>> logger.info("Reconnecting to bitbake server...")
>> if not os.path.exists(sockname):
>> - print("Previous bitbake instance shutting
>> down?, waiting to retry...")
>> + logger.info("Previous bitbake instance
>> shutting down?, waiting to retry...") i = 0
>> lock = None
>> # Wait for 5s or until we can get the lock
>> @@ -460,12 +452,7 @@ def setup_bitbake(configParams, configuration,
>> extrafeatures=None): bb.utils.unlockfile(lock)
>> raise
>> bb.server.process.ProcessTimeout("Bitbake still shutting down as
>> socket exists but no lock?") if not configParams.server_only:
>> - try:
>> - server_connection =
>> bb.server.process.connectProcessServer(sockname, featureset)
>> - except EOFError:
>> - # The server may have been shutting down but
>> not closed the socket yet. If that happened,
>> - # ignore it.
>> - pass
>> + server_connection =
>> bb.server.process.connectProcessServer(sockname, featureset)
>> if server_connection or configParams.server_only:
>> break
>> @@ -475,12 +462,14 @@ def setup_bitbake(configParams, configuration,
>> extrafeatures=None): if not retries:
>> raise
>> retries -= 1
>> - if isinstance(e, (bb.server.process.ProcessTimeout,
>> BrokenPipeError)):
>> - logger.info("Retrying server connection...")
>> + tryno = 8 - retries
>> + if isinstance(e, (bb.server.process.ProcessTimeout,
>> BrokenPipeError, EOFError)):
>> + logger.info("Retrying server connection
>> (#%d)..." % tryno) else:
>> - logger.info("Retrying server connection... (%s)"
>> % traceback.format_exc())
>> + logger.info("Retrying server connection (#%d)...
>> (%s)" % (tryno, traceback.format_exc())) if not retries:
>> - bb.fatal("Unable to connect to bitbake server, or
>> start one")
>> + bb.fatal("Unable to connect to bitbake server, or
>> start one (server startup failures would be in
>> bitbake-cookerdaemon.log).")
>> + bb.event.print_ui_queue()
>> if retries < 5:
>> time.sleep(5)
>>
>> @@ -502,7 +491,7 @@ def setup_bitbake(configParams, configuration,
>> extrafeatures=None): def lockBitbake():
>> topdir = bb.cookerdata.findTopdir()
>> if not topdir:
>> - bb.error("Unable to find conf/bblayers.conf or
>> conf/bitbake.conf. BBAPTH is unset and/or not in a build directory?")
>> + bb.error("Unable to find conf/bblayers.conf or
>> conf/bitbake.conf. BBPATH is unset and/or not in a build directory?")
>> raise BBMainFatal lockfile = topdir + "/bitbake.lock"
>> return topdir, bb.utils.lockfile(lockfile, False, False)
>> diff --git a/bitbake/lib/bb/methodpool.py
>> b/bitbake/lib/bb/methodpool.py index 49aed33..51783ac 100644
>> --- a/bitbake/lib/bb/methodpool.py
>> +++ b/bitbake/lib/bb/methodpool.py
>> @@ -1,21 +1,8 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> -#
>> #
>> # Copyright (C) 2006 Holger Hans Peter Freyther
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from bb.utils import better_compile, better_exec
>>
>> diff --git a/bitbake/lib/bb/monitordisk.py
>> b/bitbake/lib/bb/monitordisk.py index 833cd3d..1a25b00 100644
>> --- a/bitbake/lib/bb/monitordisk.py
>> +++ b/bitbake/lib/bb/monitordisk.py
>> @@ -1,21 +1,8 @@
>> -#!/usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # Copyright (C) 2012 Robert Yang
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import os, logging, re, sys
>> import bb
>> @@ -28,16 +15,16 @@ def convertGMK(unit):
>>
>> """ Convert the space unit G, M, K, the unit is case-insensitive
>> """
>> - unitG = re.match('([1-9][0-9]*)[gG]\s?$', unit)
>> + unitG = re.match(r'([1-9][0-9]*)[gG]\s?$', unit)
>> if unitG:
>> return int(unitG.group(1)) * (1024 ** 3)
>> - unitM = re.match('([1-9][0-9]*)[mM]\s?$', unit)
>> + unitM = re.match(r'([1-9][0-9]*)[mM]\s?$', unit)
>> if unitM:
>> return int(unitM.group(1)) * (1024 ** 2)
>> - unitK = re.match('([1-9][0-9]*)[kK]\s?$', unit)
>> + unitK = re.match(r'([1-9][0-9]*)[kK]\s?$', unit)
>> if unitK:
>> return int(unitK.group(1)) * 1024
>> - unitN = re.match('([1-9][0-9]*)\s?$', unit)
>> + unitN = re.match(r'([1-9][0-9]*)\s?$', unit)
>> if unitN:
>> return int(unitN.group(1))
>> else:
>> @@ -83,7 +70,7 @@ def getDiskData(BBDirs, configuration):
>> for pathSpaceInode in BBDirs.split():
>> # The input format is: "dir,space,inode", dir is a must,
>> space # and inode are optional
>> - pathSpaceInodeRe = re.match('([^,]*),([^,]*),([^,]*),?(.*)',
>> pathSpaceInode)
>> + pathSpaceInodeRe =
>> re.match(r'([^,]*),([^,]*),([^,]*),?(.*)', pathSpaceInode) if not
>> pathSpaceInodeRe: printErr("Invalid value in BB_DISKMON_DIRS: %s" %
>> pathSpaceInode) return None
>> @@ -147,7 +134,7 @@ def getInterval(configuration):
>> else:
>> # The disk space or inode interval is optional, but it should
>> # have a correct value once it is specified
>> - intervalRe = re.match('([^,]*),?\s*(.*)', interval)
>> + intervalRe = re.match(r'([^,]*),?\s*(.*)', interval)
>> if intervalRe:
>> intervalSpace = intervalRe.group(1)
>> if intervalSpace:
>> diff --git a/bitbake/lib/bb/msg.py b/bitbake/lib/bb/msg.py
>> index 96f077e..6216eb3 100644
>> --- a/bitbake/lib/bb/msg.py
>> +++ b/bitbake/lib/bb/msg.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake 'msg' implementation
>>
>> @@ -9,18 +7,8 @@ Message handling infrastructure for bitbake
>>
>> # Copyright (C) 2006 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import sys
>> import copy
>> diff --git a/bitbake/lib/bb/namedtuple_with_abc.py
>> b/bitbake/lib/bb/namedtuple_with_abc.py index 32f2fc6..646aed6 100644
>> --- a/bitbake/lib/bb/namedtuple_with_abc.py
>> +++ b/bitbake/lib/bb/namedtuple_with_abc.py
>> @@ -1,6 +1,8 @@
>> #
>> http://code.activestate.com/recipes/577629-namedtupleabc-abstract-base-class-mix-in-for-named/
>> -#!/usr/bin/env python # Copyright (c) 2011 Jan Kaliszewski (zuo).
>> Available under the MIT License. +#
>> +# SPDX-License-Identifier: MIT
>> +#
>>
>> """
>> namedtuple_with_abc.py:
>> diff --git a/bitbake/lib/bb/parse/__init__.py
>> b/bitbake/lib/bb/parse/__init__.py index 5397d57..76e180b 100644
>> --- a/bitbake/lib/bb/parse/__init__.py
>> +++ b/bitbake/lib/bb/parse/__init__.py
>> @@ -9,20 +9,10 @@ File parsers for the BitBake build tools.
>> # Copyright (C) 2003, 2004 Chris Larson
>> # Copyright (C) 2003, 2004 Phil Blundell
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> # Based on functions from the base bb module, Copyright 2003 Holger
>> Schurig +#
>>
>> handlers = []
>>
>> diff --git a/bitbake/lib/bb/parse/ast.py b/bitbake/lib/bb/parse/ast.py
>> index 6d7c80b..f0911e6 100644
>> --- a/bitbake/lib/bb/parse/ast.py
>> +++ b/bitbake/lib/bb/parse/ast.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> AbstractSyntaxTree classes for the Bitbake language
>> """
>> @@ -8,19 +6,8 @@
>> # Copyright (C) 2003, 2004 Phil Blundell
>> # Copyright (C) 2009 Holger Hans Peter Freyther
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. -
>>
>> import re
>> import string
>> diff --git a/bitbake/lib/bb/parse/parse_py/BBHandler.py
>> b/bitbake/lib/bb/parse/parse_py/BBHandler.py index 01fc47e..6f7cf82
>> 100644 --- a/bitbake/lib/bb/parse/parse_py/BBHandler.py
>> +++ b/bitbake/lib/bb/parse/parse_py/BBHandler.py
>> @@ -1,6 +1,3 @@
>> -#!/usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> class for handling .bb files
>>
>> @@ -12,19 +9,8 @@
>> # Copyright (C) 2003, 2004 Chris Larson
>> # Copyright (C) 2003, 2004 Phil Blundell
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. -
>>
>> import re, bb, os
>> import logging
>> @@ -38,14 +24,15 @@ from .ConfHandler import include, init
>> # For compatibility
>> bb.deprecate_import(__name__, "bb.parse", ["vars_from_file"])
>>
>> -__func_start_regexp__ =
>> re.compile( r"(((?P<py>python)|(?P<fr>fakeroot))\s*)*(?P<func>[\w\.\-\+\{\}\$]+)?\s*\(\s*\)\s*{$" )
>> -__inherit_regexp__ = re.compile( r"inherit\s+(.+)" )
>> -__export_func_regexp__ = re.compile( r"EXPORT_FUNCTIONS\s+(.+)" )
>> -__addtask_regexp__ =
>> re.compile("addtask\s+(?P<func>\w+)\s*((before\s*(?P<before>((.*(?=after))|(.*))))|(after\s*(?P<after>((.*(?=before))|(.*)))))*")
>> -__deltask_regexp__ = re.compile("deltask\s+(?P<func>\w+)")
>> -__addhandler_regexp__ = re.compile( r"addhandler\s+(.+)" )
>> -__def_regexp__ = re.compile( r"def\s+(\w+).*:" )
>> -__python_func_regexp__ = re.compile( r"(\s+.*)|(^$)|(^#)" )
>> +__func_start_regexp__ =
>> re.compile(r"(((?P<py>python)|(?P<fr>fakeroot))\s*)*(?P<func>[\w\.\-\+\{\}\$]+)?\s*\(\s*\)\s*{$" )
>> +__inherit_regexp__ = re.compile(r"inherit\s+(.+)" )
>> +__export_func_regexp__ = re.compile(r"EXPORT_FUNCTIONS\s+(.+)" )
>> +__addtask_regexp__ =
>> re.compile(r"addtask\s+(?P<func>\w+)\s*((before\s*(?P<before>((.*(?=after))|(.*))))|(after\s*(?P<after>((.*(?=before))|(.*)))))*")
>> +__deltask_regexp__ =
>> re.compile(r"deltask\s+(?P<func>\w+)(?P<ignores>.*)")
>> +__addhandler_regexp__ = re.compile(r"addhandler\s+(.+)" )
>> +__def_regexp__ = re.compile(r"def\s+(\w+).*:" )
>> +__python_func_regexp__ = re.compile(r"(\s+.*)|(^$)|(^#)" )
>> +__python_tab_regexp__ = re.compile(r" *\t") __infunc__ = []
>> __inpython__ = False @@ -160,6 +147,16 @@ def handle(fn, d, include):
>> def feeder(lineno, s, fn, root, statements, eof=False): global
>> __func_start_regexp__, __inherit_regexp__, __export_func_regexp__,
>> __addtask_regexp__, __addhandler_regexp__, __def_regexp__,
>> __python_func_regexp__, __inpython__, __infunc__, __body__, bb,
>> __residue__, __classname__ +
>> + # Check tabs in python functions:
>> + # - def py_funcname(): covered by __inpython__
>> + # - python(): covered by '__anonymous' == __infunc__[0]
>> + # - python funcname(): covered by __infunc__[3]
>> + if __inpython__ or (__infunc__ and ('__anonymous' ==
>> __infunc__[0] or __infunc__[3])):
>> + tab = __python_tab_regexp__.match(s)
>> + if tab:
>> + bb.warn('python should use 4 spaces indentation, but
>> found tabs in %s, line %s' % (root, lineno)) +
>> if __infunc__:
>> if s == '}':
>> __body__.append('')
>> @@ -225,11 +222,27 @@ def feeder(lineno, s, fn, root, statements,
>> eof=False):
>> m = __addtask_regexp__.match(s)
>> if m:
>> + if len(m.group().split()) == 2:
>> + # Check and warn for "addtask task1 task2"
>> + m2 = re.match(r"addtask\s+(?P<func>\w+)(?P<ignores>.*)",
>> s)
>> + if m2 and m2.group('ignores'):
>> + logger.warning('addtask ignored: "%s"' %
>> m2.group('ignores')) +
>> + # Check and warn for "addtask task1 before task2 before
>> task3", the
>> + # similar to "after"
>> + taskexpression = s.split()
>> + for word in ('before', 'after'):
>> + if taskexpression.count(word) > 1:
>> + logger.warning("addtask contained multiple '%s'
>> keywords, only one is supported" % word) +
>> ast.handleAddTask(statements, fn, lineno, m)
>> return
>>
>> m = __deltask_regexp__.match(s)
>> if m:
>> + # Check and warn "for deltask task1 task2"
>> + if m.group('ignores'):
>> + logger.warning('deltask ignored: "%s"' %
>> m.group('ignores')) ast.handleDelTask(statements, fn, lineno, m)
>> return
>>
>> diff --git a/bitbake/lib/bb/parse/parse_py/ConfHandler.py
>> b/bitbake/lib/bb/parse/parse_py/ConfHandler.py index 9d3ebe1..2e84b91
>> 100644 --- a/bitbake/lib/bb/parse/parse_py/ConfHandler.py
>> +++ b/bitbake/lib/bb/parse/parse_py/ConfHandler.py
>> @@ -1,6 +1,3 @@
>> -#!/usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> class for handling configuration data files
>>
>> @@ -11,18 +8,8 @@
>> # Copyright (C) 2003, 2004 Chris Larson
>> # Copyright (C) 2003, 2004 Phil Blundell
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import errno
>> import re
>> @@ -147,7 +134,7 @@ def handle(fn, data, include):
>> continue
>> s = s.rstrip()
>> while s[-1] == '\\':
>> - s2 = f.readline().strip()
>> + s2 = f.readline().rstrip()
>> lineno = lineno + 1
>> if (not s2 or s2 and s2[0] != "#") and s[0] == "#" :
>> bb.fatal("There is a confusing multiline, partially
>> commented expression on line %s of file %s (%s).\nPlease clarify
>> whether this is all a comment or should be parsed." % (lineno, fn,
>> s)) diff --git a/bitbake/lib/bb/parse/parse_py/__init__.py
>> b/bitbake/lib/bb/parse/parse_py/__init__.py index 3e658d0..f508afa
>> 100644 --- a/bitbake/lib/bb/parse/parse_py/__init__.py +++
>> b/bitbake/lib/bb/parse/parse_py/__init__.py @@ -1,6 +1,3 @@
>> -#!/usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake Parsers
>>
>> @@ -11,20 +8,10 @@ File parsers for the BitBake build tools.
>> # Copyright (C) 2003, 2004 Chris Larson
>> # Copyright (C) 2003, 2004 Phil Blundell
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> # Based on functions from the base bb module, Copyright 2003 Holger
>> Schurig +#
>>
>> from __future__ import absolute_import
>> from . import ConfHandler
>> diff --git a/bitbake/lib/bb/persist_data.py
>> b/bitbake/lib/bb/persist_data.py index bef7018..de8f87a 100644
>> --- a/bitbake/lib/bb/persist_data.py
>> +++ b/bitbake/lib/bb/persist_data.py
>> @@ -8,18 +8,8 @@ currently, providing a key/value store accessed by
>> 'domain'. # Copyright (C) 2007 Richard Purdie
>> # Copyright (C) 2010 Chris Larson <chris_larson@mentor.com>
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import collections
>> import logging
>> @@ -29,6 +19,7 @@ import warnings
>> from bb.compat import total_ordering
>> from collections import Mapping
>> import sqlite3
>> +import contextlib
>>
>> sqlversion = sqlite3.sqlite_version_info
>> if sqlversion[0] < 3 or (sqlversion[0] == 3 and sqlversion[1] < 3):
>> @@ -36,84 +27,181 @@ if sqlversion[0] < 3 or (sqlversion[0] == 3 and
>> sqlversion[1] < 3):
>>
>> logger = logging.getLogger("BitBake.PersistData")
>> -if hasattr(sqlite3, 'enable_shared_cache'):
>> - try:
>> - sqlite3.enable_shared_cache(True)
>> - except sqlite3.OperationalError:
>> - pass
>> -
>>
>> @total_ordering
>> class SQLTable(collections.MutableMapping):
>> + class _Decorators(object):
>> + @staticmethod
>> + def retry(*, reconnect=True):
>> + """
>> + Decorator that restarts a function if a database locked
>> sqlite
>> + exception occurs. If reconnect is True, the database
>> connection
>> + will be closed and reopened each time a failure occurs
>> + """
>> + def retry_wrapper(f):
>> + def wrap_func(self, *args, **kwargs):
>> + # Reconnect if necessary
>> + if self.connection is None and reconnect:
>> + self.reconnect()
>> +
>> + count = 0
>> + while True:
>> + try:
>> + return f(self, *args, **kwargs)
>> + except sqlite3.OperationalError as exc:
>> + if count < 500 and ('is locked' in
>> str(exc) or 'locking protocol' in str(exc)):
>> + count = count + 1
>> + if reconnect:
>> + self.reconnect()
>> + continue
>> + raise
>> + return wrap_func
>> + return retry_wrapper
>> +
>> + @staticmethod
>> + def transaction(f):
>> + """
>> + Decorator that starts a database transaction and creates
>> a database
>> + cursor for performing queries. If no exception is
>> thrown, the
>> + database results are commited. If an exception occurs,
>> the database
>> + is rolled back. In all cases, the cursor is closed after
>> the
>> + function ends.
>> +
>> + Note that the cursor is passed as an extra argument to
>> the function
>> + after `self` and before any of the normal arguments
>> + """
>> + def wrap_func(self, *args, **kwargs):
>> + # Context manager will COMMIT the database on
>> success,
>> + # or ROLLBACK on an exception
>> + with self.connection:
>> + # Automatically close the cursor when done
>> + with
>> contextlib.closing(self.connection.cursor()) as cursor:
>> + return f(self, cursor, *args, **kwargs)
>> + return wrap_func
>> +
>> """Object representing a table/domain in the database"""
>> def __init__(self, cachefile, table):
>> self.cachefile = cachefile
>> self.table = table
>> - self.cursor = connect(self.cachefile)
>> -
>> - self._execute("CREATE TABLE IF NOT EXISTS %s(key TEXT, value
>> TEXT);"
>> - % table)
>> -
>> - def _execute(self, *query):
>> - """Execute a query, waiting to acquire a lock if necessary"""
>> - count = 0
>> - while True:
>> - try:
>> - return self.cursor.execute(*query)
>> - except sqlite3.OperationalError as exc:
>> - if 'database is locked' in str(exc) and count < 500:
>> - count = count + 1
>> +
>> + self.connection = None
>> + self._execute_single("CREATE TABLE IF NOT EXISTS %s(key TEXT
>> PRIMARY KEY NOT NULL, value TEXT);" % table) +
>> + @_Decorators.retry(reconnect=False)
>> + @_Decorators.transaction
>> + def _setup_database(self, cursor):
>> + cursor.execute("pragma synchronous = off;")
>> + # Enable WAL and keep the autocheckpoint length small (the
>> default is
>> + # usually 1000). Persistent caches are usually read-mostly,
>> so keeping
>> + # this short will keep readers running quickly
>> + cursor.execute("pragma journal_mode = WAL;")
>> + cursor.execute("pragma wal_autocheckpoint = 100;")
>> +
>> + def reconnect(self):
>> + if self.connection is not None:
>> + self.connection.close()
>> + self.connection = sqlite3.connect(self.cachefile, timeout=5)
>> + self.connection.text_factory = str
>> + self._setup_database()
>> +
>> + @_Decorators.retry()
>> + @_Decorators.transaction
>> + def _execute_single(self, cursor, *query):
>> + """
>> + Executes a single query and discards the results. This
>> correctly closes
>> + the database cursor when finished
>> + """
>> + cursor.execute(*query)
>> +
>> + @_Decorators.retry()
>> + def _row_iter(self, f, *query):
>> + """
>> + Helper function that returns a row iterator. Each time
>> __next__ is
>> + called on the iterator, the provided function is evaluated
>> to determine
>> + the return value
>> + """
>> + class CursorIter(object):
>> + def __init__(self, cursor):
>> + self.cursor = cursor
>> +
>> + def __iter__(self):
>> + return self
>> +
>> + def __next__(self):
>> + row = self.cursor.fetchone()
>> + if row is None:
>> self.cursor.close()
>> - self.cursor = connect(self.cachefile)
>> - continue
>> - raise
>> + raise StopIteration
>> + return f(row)
>> +
>> + def __enter__(self):
>> + return self
>> +
>> + def __exit__(self, typ, value, traceback):
>> + self.cursor.close()
>> + return False
>> +
>> + cursor = self.connection.cursor()
>> + try:
>> + cursor.execute(*query)
>> + return CursorIter(cursor)
>> + except:
>> + cursor.close()
>>
>> def __enter__(self):
>> - self.cursor.__enter__()
>> + self.connection.__enter__()
>> return self
>>
>> def __exit__(self, *excinfo):
>> - self.cursor.__exit__(*excinfo)
>> -
>> - def __getitem__(self, key):
>> - data = self._execute("SELECT * from %s where key=?;" %
>> - self.table, [key])
>> - for row in data:
>> + self.connection.__exit__(*excinfo)
>> +
>> + @_Decorators.retry()
>> + @_Decorators.transaction
>> + def __getitem__(self, cursor, key):
>> + cursor.execute("SELECT * from %s where key=?;" % self.table,
>> [key])
>> + row = cursor.fetchone()
>> + if row is not None:
>> return row[1]
>> raise KeyError(key)
>>
>> - def __delitem__(self, key):
>> + @_Decorators.retry()
>> + @_Decorators.transaction
>> + def __delitem__(self, cursor, key):
>> if key not in self:
>> raise KeyError(key)
>> - self._execute("DELETE from %s where key=?;" % self.table,
>> [key])
>> + cursor.execute("DELETE from %s where key=?;" % self.table,
>> [key])
>> - def __setitem__(self, key, value):
>> + @_Decorators.retry()
>> + @_Decorators.transaction
>> + def __setitem__(self, cursor, key, value):
>> if not isinstance(key, str):
>> raise TypeError('Only string keys are supported')
>> elif not isinstance(value, str):
>> raise TypeError('Only string values are supported')
>>
>> - data = self._execute("SELECT * from %s where key=?;" %
>> - self.table, [key])
>> - exists = len(list(data))
>> - if exists:
>> - self._execute("UPDATE %s SET value=? WHERE key=?;" %
>> self.table,
>> - [value, key])
>> + cursor.execute("SELECT * from %s where key=?;" % self.table,
>> [key])
>> + row = cursor.fetchone()
>> + if row is not None:
>> + cursor.execute("UPDATE %s SET value=? WHERE key=?;" %
>> self.table, [value, key]) else:
>> - self._execute("INSERT into %s(key, value) values
>> (?, ?);" %
>> - self.table, [key, value])
>> -
>> - def __contains__(self, key):
>> - return key in set(self)
>> -
>> - def __len__(self):
>> - data = self._execute("SELECT COUNT(key) FROM %s;" %
>> self.table)
>> - for row in data:
>> + cursor.execute("INSERT into %s(key, value) values
>> (?, ?);" % self.table, [key, value]) +
>> + @_Decorators.retry()
>> + @_Decorators.transaction
>> + def __contains__(self, cursor, key):
>> + cursor.execute('SELECT * from %s where key=?;' % self.table,
>> [key])
>> + return cursor.fetchone() is not None
>> +
>> + @_Decorators.retry()
>> + @_Decorators.transaction
>> + def __len__(self, cursor):
>> + cursor.execute("SELECT COUNT(key) FROM %s;" % self.table)
>> + row = cursor.fetchone()
>> + if row is not None:
>> return row[0]
>>
>> def __iter__(self):
>> - data = self._execute("SELECT key FROM %s;" % self.table)
>> - return (row[0] for row in data)
>> + return self._row_iter(lambda row: row[0], "SELECT key from
>> %s;" % self.table)
>> def __lt__(self, other):
>> if not isinstance(other, Mapping):
>> @@ -122,25 +210,27 @@ class SQLTable(collections.MutableMapping):
>> return len(self) < len(other)
>>
>> def get_by_pattern(self, pattern):
>> - data = self._execute("SELECT * FROM %s WHERE key LIKE ?;" %
>> - self.table, [pattern])
>> - return [row[1] for row in data]
>> + return self._row_iter(lambda row: row[1], "SELECT * FROM %s
>> WHERE key LIKE ?;" %
>> + self.table, [pattern])
>>
>> def values(self):
>> return list(self.itervalues())
>>
>> def itervalues(self):
>> - data = self._execute("SELECT value FROM %s;" % self.table)
>> - return (row[0] for row in data)
>> + return self._row_iter(lambda row: row[0], "SELECT value FROM
>> %s;" %
>> + self.table)
>>
>> def items(self):
>> return list(self.iteritems())
>>
>> def iteritems(self):
>> - return self._execute("SELECT * FROM %s;" % self.table)
>> + return self._row_iter(lambda row: (row[0], row[1]), "SELECT
>> * FROM %s;" %
>> + self.table)
>>
>> - def clear(self):
>> - self._execute("DELETE FROM %s;" % self.table)
>> + @_Decorators.retry()
>> + @_Decorators.transaction
>> + def clear(self, cursor):
>> + cursor.execute("DELETE FROM %s;" % self.table)
>>
>> def has_key(self, key):
>> return key in self
>> @@ -194,12 +284,6 @@ class PersistData(object):
>> """
>> del self.data[domain][key]
>>
>> -def connect(database):
>> - connection = sqlite3.connect(database, timeout=5,
>> isolation_level=None)
>> - connection.execute("pragma synchronous = off;")
>> - connection.text_factory = str
>> - return connection
>> -
>> def persist(domain, d):
>> """Convenience factory for SQLTable objects based upon
>> metadata""" import bb.utils
>> diff --git a/bitbake/lib/bb/process.py b/bitbake/lib/bb/process.py
>> index e69697c..2dc472a 100644
>> --- a/bitbake/lib/bb/process.py
>> +++ b/bitbake/lib/bb/process.py
>> @@ -1,3 +1,7 @@
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> import logging
>> import signal
>> import subprocess
>> diff --git a/bitbake/lib/bb/progress.py b/bitbake/lib/bb/progress.py
>> index f54d1c7..4022caa 100644
>> --- a/bitbake/lib/bb/progress.py
>> +++ b/bitbake/lib/bb/progress.py
>> @@ -4,18 +4,8 @@ BitBake progress handling code
>>
>> # Copyright (C) 2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import sys
>> import re
>> @@ -23,6 +13,7 @@ import time
>> import inspect
>> import bb.event
>> import bb.build
>> +from bb.build import StdoutNoopContextManager
>>
>> class ProgressHandler(object):
>> """
>> @@ -37,7 +28,14 @@ class ProgressHandler(object):
>> if outfile:
>> self._outfile = outfile
>> else:
>> - self._outfile = sys.stdout
>> + self._outfile = StdoutNoopContextManager()
>> +
>> + def __enter__(self):
>> + self._outfile.__enter__()
>> + return self
>> +
>> + def __exit__(self, *excinfo):
>> + self._outfile.__exit__(*excinfo)
>>
>> def _fire_progress(self, taskprogress, rate=None):
>> """Internal function to fire the progress event"""
>> @@ -157,6 +155,12 @@ class MultiStageProgressReporter(object):
>> self._stage_total = None
>> self._callers = []
>>
>> + def __enter__(self):
>> + return self
>> +
>> + def __exit__(self, *excinfo):
>> + pass
>> +
>> def _fire_progress(self, taskprogress):
>> bb.event.fire(bb.build.TaskProgress(taskprogress),
>> self._data)
>> diff --git a/bitbake/lib/bb/providers.py b/bitbake/lib/bb/providers.py
>> index c2aa98c..f80963c 100644
>> --- a/bitbake/lib/bb/providers.py
>> +++ b/bitbake/lib/bb/providers.py
>> @@ -1,5 +1,3 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # Copyright (C) 2003, 2004 Chris Larson
>> # Copyright (C) 2003, 2004 Phil Blundell
>> @@ -8,18 +6,8 @@
>> # Copyright (C) 2005 ROAD GmbH
>> # Copyright (C) 2006 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import re
>> import logging
>> @@ -129,7 +117,7 @@ def findPreferredProvider(pn, cfgData, dataCache,
>> pkg_pn = None, item = None): preferred_v =
>> cfgData.getVar("PREFERRED_VERSION")
>> if preferred_v:
>> - m = re.match('(\d+:)*(.*)(_.*)*', preferred_v)
>> + m = re.match(r'(\d+:)*(.*)(_.*)*', preferred_v)
>> if m:
>> if m.group(1):
>> preferred_e = m.group(1)[:-1]
>> @@ -384,7 +372,7 @@ def getRuntimeProviders(dataCache, rdepend):
>>
>> # Only search dynamic packages if we can't find anything in
>> other variables for pattern in dataCache.packages_dynamic:
>> - pattern = pattern.replace('+', "\+")
>> + pattern = pattern.replace(r'+', r"\+")
>> if pattern in regexp_cache:
>> regexp = regexp_cache[pattern]
>> else:
>> diff --git a/bitbake/lib/bb/pysh/builtin.py
>> b/bitbake/lib/bb/pysh/builtin.py deleted file mode 100644
>> index a8814dc..0000000
>> --- a/bitbake/lib/bb/pysh/builtin.py
>> +++ /dev/null
>> @@ -1,710 +0,0 @@
>> -# builtin.py - builtins and utilities definitions for pysh.
>> -#
>> -# Copyright 2007 Patrick Mezard
>> -#
>> -# This software may be used and distributed according to the terms
>> -# of the GNU General Public License, incorporated herein by
>> reference. -
>> -"""Builtin and internal utilities implementations.
>> -
>> -- Beware not to use python interpreter environment as if it were the
>> shell -environment. For instance, commands working directory must be
>> explicitely handled -through env['PWD'] instead of relying on python
>> working directory. -"""
>> -import errno
>> -import optparse
>> -import os
>> -import re
>> -import subprocess
>> -import sys
>> -import time
>> -
>> -def has_subprocess_bug():
>> - return getattr(subprocess, 'list2cmdline') and \
>> - ( subprocess.list2cmdline(['']) == '' or \
>> - subprocess.list2cmdline(['foo|bar']) == 'foo|bar')
>> -
>> -# Detect python bug 1634343: "subprocess swallows empty arguments
>> under win32" -#
>> <http://sourceforge.net/tracker/index.php?func=detail&aid=1634343&group_id=5470&atid=105470>
>> -# Also detect: "[ 1710802 ] subprocess must escape redirection
>> characters under win32" -#
>> <http://sourceforge.net/tracker/index.php?func=detail&aid=1710802&group_id=5470&atid=105470>
>> -if has_subprocess_bug():
>> - import subprocess_fix
>> - subprocess.list2cmdline = subprocess_fix.list2cmdline
>> -
>> -from sherrors import *
>> -
>> -class NonExitingParser(optparse.OptionParser):
>> - """OptionParser default behaviour upon error is to print the
>> error message and
>> - exit. Raise a utility error instead.
>> - """
>> - def error(self, msg):
>> - raise UtilityError(msg)
>> -
>> -#-------------------------------------------------------------------------------
>> -# set special builtin
>> -#-------------------------------------------------------------------------------
>> -OPT_SET = NonExitingParser(usage="set - set or unset options and
>> positional parameters") -OPT_SET.add_option( '-f',
>> action='store_true', dest='has_f', default=False,
>> - help='The shell shall disable pathname expansion.')
>> -OPT_SET.add_option('-e', action='store_true', dest='has_e',
>> default=False,
>> - help="""When this option is on, if a simple command fails for
>> any of the \
>> - reasons listed in Consequences of Shell Errors or returns an
>> exit status \
>> - value >0, and is not part of the compound list following a
>> while, until, \
>> - or if keyword, and is not a part of an AND or OR list, and is
>> not a \
>> - pipeline preceded by the ! reserved word, then the shell shall
>> immediately \
>> - exit.""")
>> -OPT_SET.add_option('-x', action='store_true', dest='has_x',
>> default=False,
>> - help="""The shell shall write to standard error a trace for each
>> command \
>> - after it expands the command and before it executes it. It is
>> unspecified \
>> - whether the command that turns tracing off is traced.""")
>> -
>> -def builtin_set(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n') -
>> - option, args = OPT_SET.parse_args(args)
>> - env = interp.get_env()
>> -
>> - if option.has_f:
>> - env.set_opt('-f')
>> - if option.has_e:
>> - env.set_opt('-e')
>> - if option.has_x:
>> - env.set_opt('-x')
>> - return 0
>> -
>> -#-------------------------------------------------------------------------------
>> -# shift special builtin
>> -#-------------------------------------------------------------------------------
>> -def builtin_shift(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> -
>> - params = interp.get_env().get_positional_args()
>> - if args:
>> - try:
>> - n = int(args[0])
>> - if n > len(params):
>> - raise ValueError()
>> - except ValueError:
>> - return 1
>> - else:
>> - n = 1
>> -
>> - params[:n] = []
>> - interp.get_env().set_positional_args(params)
>> - return 0
>> -
>> -#-------------------------------------------------------------------------------
>> -# export special builtin
>> -#-------------------------------------------------------------------------------
>> -OPT_EXPORT = NonExitingParser(usage="set - set or unset options and
>> positional parameters") -OPT_EXPORT.add_option('-p',
>> action='store_true', dest='has_p', default=False) -
>> -def builtin_export(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> -
>> - option, args = OPT_EXPORT.parse_args(args)
>> - if option.has_p:
>> - raise NotImplementedError()
>> -
>> - for arg in args:
>> - try:
>> - name, value = arg.split('=', 1)
>> - except ValueError:
>> - name, value = arg, None
>> - env = interp.get_env().export(name, value)
>> -
>> - return 0
>> -
>> -#-------------------------------------------------------------------------------
>> -# return special builtin
>> -#-------------------------------------------------------------------------------
>> -def builtin_return(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> - res = 0
>> - if args:
>> - try:
>> - res = int(args[0])
>> - except ValueError:
>> - res = 0
>> - if not 0<=res<=255:
>> - res = 0
>> -
>> - # BUG: should be last executed command exit code
>> - raise ReturnSignal(res)
>> -
>> -#-------------------------------------------------------------------------------
>> -# trap special builtin
>> -#-------------------------------------------------------------------------------
>> -def builtin_trap(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> - if len(args) < 2:
>> - stderr.write('trap: usage: trap [[arg] signal_spec ...]\n')
>> - return 2
>> -
>> - action = args[0]
>> - for sig in args[1:]:
>> - try:
>> - env.traps[sig] = action
>> - except Exception as e:
>> - stderr.write('trap: %s\n' % str(e))
>> - return 0
>> -
>> -#-------------------------------------------------------------------------------
>> -# unset special builtin
>> -#-------------------------------------------------------------------------------
>> -OPT_UNSET = NonExitingParser("unset - unset values and attributes of
>> variables and functions") -OPT_UNSET.add_option( '-f',
>> action='store_true', dest='has_f', default=False)
>> -OPT_UNSET.add_option( '-v', action='store_true', dest='has_v',
>> default=False) - -def builtin_unset(name, args, interp, env, stdin,
>> stdout, stderr, debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> -
>> - option, args = OPT_UNSET.parse_args(args)
>> -
>> - status = 0
>> - env = interp.get_env()
>> - for arg in args:
>> - try:
>> - if option.has_f:
>> - env.remove_function(arg)
>> - else:
>> - del env[arg]
>> - except KeyError:
>> - pass
>> - except VarAssignmentError:
>> - status = 1
>> -
>> - return status
>> -
>> -#-------------------------------------------------------------------------------
>> -# wait special builtin
>> -#-------------------------------------------------------------------------------
>> -def builtin_wait(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n') -
>> - return interp.wait([int(arg) for arg in args])
>> -
>> -#-------------------------------------------------------------------------------
>> -# cat utility
>> -#-------------------------------------------------------------------------------
>> -def utility_cat(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n') -
>> - if not args:
>> - args = ['-']
>> -
>> - status = 0
>> - for arg in args:
>> - if arg == '-':
>> - data = stdin.read()
>> - else:
>> - path = os.path.join(env['PWD'], arg)
>> - try:
>> - f = file(path, 'rb')
>> - try:
>> - data = f.read()
>> - finally:
>> - f.close()
>> - except IOError as e:
>> - if e.errno != errno.ENOENT:
>> - raise
>> - status = 1
>> - continue
>> - stdout.write(data)
>> - stdout.flush()
>> - return status
>> -
>> -#-------------------------------------------------------------------------------
>> -# cd utility
>> -#-------------------------------------------------------------------------------
>> -OPT_CD = NonExitingParser("cd - change the working directory")
>> -
>> -def utility_cd(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n') -
>> - option, args = OPT_CD.parse_args(args)
>> - env = interp.get_env()
>> -
>> - directory = None
>> - printdir = False
>> - if not args:
>> - home = env.get('HOME')
>> - if home:
>> - # Unspecified, do nothing
>> - return 0
>> - else:
>> - directory = home
>> - elif len(args)==1:
>> - directory = args[0]
>> - if directory=='-':
>> - if 'OLDPWD' not in env:
>> - raise UtilityError("OLDPWD not set")
>> - printdir = True
>> - directory = env['OLDPWD']
>> - else:
>> - raise UtilityError("too many arguments")
>> -
>> - curpath = None
>> - # Absolute directories will be handled correctly by the
>> os.path.join call.
>> - if not directory.startswith('.') and not
>> directory.startswith('..'):
>> - cdpaths = env.get('CDPATH', '.').split(';')
>> - for cdpath in cdpaths:
>> - p = os.path.join(cdpath, directory)
>> - if os.path.isdir(p):
>> - curpath = p
>> - break
>> -
>> - if curpath is None:
>> - curpath = directory
>> - curpath = os.path.join(env['PWD'], directory)
>> -
>> - env['OLDPWD'] = env['PWD']
>> - env['PWD'] = curpath
>> - if printdir:
>> - stdout.write('%s\n' % curpath)
>> - return 0
>> -
>> -#-------------------------------------------------------------------------------
>> -# colon utility
>> -#-------------------------------------------------------------------------------
>> -def utility_colon(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> - return 0
>> -
>> -#-------------------------------------------------------------------------------
>> -# echo utility
>> -#-------------------------------------------------------------------------------
>> -def utility_echo(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> -
>> - # Echo only takes arguments, no options. Use printf if you need
>> fancy stuff.
>> - output = ' '.join(args) + '\n'
>> - stdout.write(output)
>> - stdout.flush()
>> - return 0
>> -
>> -#-------------------------------------------------------------------------------
>> -# egrep utility
>> -#-------------------------------------------------------------------------------
>> -# egrep is usually a shell script.
>> -# Unfortunately, pysh does not support shell scripts *with
>> arguments* right now, -# so the redirection is implemented here,
>> assuming grep is available. -def utility_egrep(name, args, interp,
>> env, stdin, stdout, stderr, debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> -
>> - return run_command('grep', ['-E'] + args, interp, env, stdin,
>> stdout,
>> - stderr, debugflags)
>> -
>> -#-------------------------------------------------------------------------------
>> -# env utility
>> -#-------------------------------------------------------------------------------
>> -def utility_env(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> -
>> - if args and args[0]=='-i':
>> - raise NotImplementedError('env: -i option is not
>> implemented')
>> -
>> - i = 0
>> - for arg in args:
>> - if '=' not in arg:
>> - break
>> - # Update the current environment
>> - name, value = arg.split('=', 1)
>> - env[name] = value
>> - i += 1
>> -
>> - if args[i:]:
>> - # Find then execute the specified interpreter
>> - utility = env.find_in_path(args[i])
>> - if not utility:
>> - return 127
>> - args[i:i+1] = utility
>> - name = args[i]
>> - args = args[i+1:]
>> - try:
>> - return run_command(name, args, interp, env, stdin,
>> stdout, stderr,
>> - debugflags)
>> - except UtilityError:
>> - stderr.write('env: failed to execute %s' % '
>> '.join([name]+args))
>> - return 126
>> - else:
>> - for pair in env.get_variables().iteritems():
>> - stdout.write('%s=%s\n' % pair)
>> - return 0
>> -
>> -#-------------------------------------------------------------------------------
>> -# exit utility
>> -#-------------------------------------------------------------------------------
>> -def utility_exit(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> -
>> - res = None
>> - if args:
>> - try:
>> - res = int(args[0])
>> - except ValueError:
>> - res = None
>> - if not 0<=res<=255:
>> - res = None
>> -
>> - if res is None:
>> - # BUG: should be last executed command exit code
>> - res = 0
>> -
>> - raise ExitSignal(res)
>> -
>> -#-------------------------------------------------------------------------------
>> -# fgrep utility
>> -#-------------------------------------------------------------------------------
>> -# see egrep
>> -def utility_fgrep(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> -
>> - return run_command('grep', ['-F'] + args, interp, env, stdin,
>> stdout,
>> - stderr, debugflags)
>> -
>> -#-------------------------------------------------------------------------------
>> -# gunzip utility
>> -#-------------------------------------------------------------------------------
>> -# see egrep
>> -def utility_gunzip(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> -
>> - return run_command('gzip', ['-d'] + args, interp, env, stdin,
>> stdout,
>> - stderr, debugflags)
>> -
>> -#-------------------------------------------------------------------------------
>> -# kill utility
>> -#-------------------------------------------------------------------------------
>> -def utility_kill(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> -
>> - for arg in args:
>> - pid = int(arg)
>> - status = subprocess.call(['pskill', '/T', str(pid)],
>> - shell=True,
>> - stdout=subprocess.PIPE,
>> - stderr=subprocess.PIPE)
>> - # pskill is asynchronous, hence the stupid polling loop
>> - while 1:
>> - p = subprocess.Popen(['pslist', str(pid)],
>> - shell=True,
>> - stdout=subprocess.PIPE,
>> - stderr=subprocess.STDOUT)
>> - output = p.communicate()[0]
>> - if ('process %d was not' % pid) in output:
>> - break
>> - time.sleep(1)
>> - return status
>> -
>> -#-------------------------------------------------------------------------------
>> -# mkdir utility
>> -#-------------------------------------------------------------------------------
>> -OPT_MKDIR = NonExitingParser("mkdir - make directories.")
>> -OPT_MKDIR.add_option('-p', action='store_true', dest='has_p',
>> default=False) -
>> -def utility_mkdir(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> -
>> - # TODO: implement umask
>> - # TODO: implement proper utility error report
>> - option, args = OPT_MKDIR.parse_args(args)
>> - for arg in args:
>> - path = os.path.join(env['PWD'], arg)
>> - if option.has_p:
>> - try:
>> - os.makedirs(path)
>> - except IOError as e:
>> - if e.errno != errno.EEXIST:
>> - raise
>> - else:
>> - os.mkdir(path)
>> - return 0
>> -
>> -#-------------------------------------------------------------------------------
>> -# netstat utility
>> -#-------------------------------------------------------------------------------
>> -def utility_netstat(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - # Do you really expect me to implement netstat ?
>> - # This empty form is enough for Mercurial tests since it's
>> - # supposed to generate nothing upon success. Faking this test
>> - # is not a big deal either.
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> - return 0
>> -
>> -#-------------------------------------------------------------------------------
>> -# pwd utility
>> -#-------------------------------------------------------------------------------
>> -OPT_PWD = NonExitingParser("pwd - return working directory name")
>> -OPT_PWD.add_option('-L', action='store_true', dest='has_L',
>> default=True,
>> - help="""If the PWD environment variable contains an absolute
>> pathname of \
>> - the current directory that does not contain the filenames dot or
>> dot-dot, \
>> - pwd shall write this pathname to standard output. Otherwise, the
>> -L option \
>> - shall behave as the -P option.""")
>> -OPT_PWD.add_option('-P', action='store_true', dest='has_L',
>> default=False,
>> - help="""The absolute pathname written shall not contain
>> filenames that, in \
>> - the context of the pathname, refer to files of type symbolic
>> link.""") -
>> -def utility_pwd(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n') -
>> - option, args = OPT_PWD.parse_args(args)
>> - stdout.write('%s\n' % env['PWD'])
>> - return 0
>> -
>> -#-------------------------------------------------------------------------------
>> -# printf utility
>> -#-------------------------------------------------------------------------------
>> -RE_UNESCAPE = re.compile(r'(\\x[a-zA-Z0-9]{2}|\\[0-7]{1,3}|\\.)')
>> -
>> -def utility_printf(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> -
>> - def replace(m):
>> - assert m.group()
>> - g = m.group()[1:]
>> - if g.startswith('x'):
>> - return chr(int(g[1:], 16))
>> - if len(g) <= 3 and len([c for c in g if c in '01234567']) ==
>> len(g):
>> - # Yay, an octal number
>> - return chr(int(g, 8))
>> - return {
>> - 'a': '\a',
>> - 'b': '\b',
>> - 'f': '\f',
>> - 'n': '\n',
>> - 'r': '\r',
>> - 't': '\t',
>> - 'v': '\v',
>> - '\\': '\\',
>> - }.get(g)
>> -
>> - # Convert escape sequences
>> - format = re.sub(RE_UNESCAPE, replace, args[0])
>> - stdout.write(format % tuple(args[1:]))
>> - return 0
>> -
>> -#-------------------------------------------------------------------------------
>> -# true utility
>> -#-------------------------------------------------------------------------------
>> -def utility_true(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> - return 0
>> -
>> -#-------------------------------------------------------------------------------
>> -# sed utility
>> -#-------------------------------------------------------------------------------
>> -RE_SED = re.compile(r'^s(.).*\1[a-zA-Z]*$')
>> -
>> -# cygwin sed fails with some expressions when they do not end with a
>> single space. -# see unit tests for details. Interestingly, the same
>> expressions works perfectly -# in cygwin shell.
>> -def utility_sed(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> -
>> - # Scan pattern arguments and append a space if necessary
>> - for i in range(len(args)):
>> - if not RE_SED.search(args[i]):
>> - continue
>> - args[i] = args[i] + ' '
>> -
>> - return run_command(name, args, interp, env, stdin, stdout,
>> - stderr, debugflags)
>> -
>> -#-------------------------------------------------------------------------------
>> -# sleep utility
>> -#-------------------------------------------------------------------------------
>> -def utility_sleep(name, args, interp, env, stdin, stdout, stderr,
>> debugflags):
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> - time.sleep(int(args[0]))
>> - return 0
>> -
>> -#-------------------------------------------------------------------------------
>> -# sort utility
>> -#-------------------------------------------------------------------------------
>> -OPT_SORT = NonExitingParser("sort - sort, merge, or sequence check
>> text files") -
>> -def utility_sort(name, args, interp, env, stdin, stdout, stderr,
>> debugflags): -
>> - def sort(path):
>> - if path == '-':
>> - lines = stdin.readlines()
>> - else:
>> - try:
>> - f = file(path)
>> - try:
>> - lines = f.readlines()
>> - finally:
>> - f.close()
>> - except IOError as e:
>> - stderr.write(str(e) + '\n')
>> - return 1
>> -
>> - if lines and lines[-1][-1]!='\n':
>> - lines[-1] = lines[-1] + '\n'
>> - return lines
>> -
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n')
>> -
>> - option, args = OPT_SORT.parse_args(args)
>> - alllines = []
>> -
>> - if len(args)<=0:
>> - args += ['-']
>> -
>> - # Load all files lines
>> - curdir = os.getcwd()
>> - try:
>> - os.chdir(env['PWD'])
>> - for path in args:
>> - alllines += sort(path)
>> - finally:
>> - os.chdir(curdir)
>> -
>> - alllines.sort()
>> - for line in alllines:
>> - stdout.write(line)
>> - return 0
>> -
>> -#-------------------------------------------------------------------------------
>> -# hg utility
>> -#-------------------------------------------------------------------------------
>> -
>> -hgcommands = [
>> - 'add',
>> - 'addremove',
>> - 'commit', 'ci',
>> - 'debugrename',
>> - 'debugwalk',
>> - 'falabala', # Dummy command used in a mercurial test
>> - 'incoming',
>> - 'locate',
>> - 'pull',
>> - 'push',
>> - 'qinit',
>> - 'remove', 'rm',
>> - 'rename', 'mv',
>> - 'revert',
>> - 'showconfig',
>> - 'status', 'st',
>> - 'strip',
>> - ]
>> -
>> -def rewriteslashes(name, args):
>> - # Several hg commands output file paths, rewrite the separators
>> - if len(args) > 1 and name.lower().endswith('python') \
>> - and args[0].endswith('hg'):
>> - for cmd in hgcommands:
>> - if cmd in args[1:]:
>> - return True
>> -
>> - # svn output contains many paths with OS specific separators.
>> - # Normalize these to unix paths.
>> - base = os.path.basename(name)
>> - if base.startswith('svn'):
>> - return True
>> -
>> - return False
>> -
>> -def rewritehg(output):
>> - if not output:
>> - return output
>> - # Rewrite os specific messages
>> - output = output.replace(': The system cannot find the file
>> specified',
>> - ': No such file or directory')
>> - output = re.sub(': Access is denied.*$', ': Permission denied',
>> output)
>> - output = output.replace(': No connection could be made because
>> the target machine actively refused it',
>> - ': Connection refused')
>> - return output
>> -
>> -
>> -def run_command(name, args, interp, env, stdin, stdout,
>> - stderr, debugflags):
>> - # Execute the command
>> - if 'debug-utility' in debugflags:
>> - print interp.log(' '.join([name, str(args), interp['PWD']])
>> + '\n') -
>> - hgbin = interp.options().hgbinary
>> - ishg = hgbin and ('hg' in name or args and 'hg' in args[0])
>> - unixoutput = 'cygwin' in name or ishg
>> -
>> - exec_env = env.get_variables()
>> - try:
>> - # BUG: comparing file descriptor is clearly not a reliable
>> way to tell
>> - # whether they point on the same underlying object. But in
>> pysh limited
>> - # scope this is usually right, we do not expect complicated
>> redirections
>> - # besides usual 2>&1.
>> - # Still there is one case we have but cannot deal with is
>> when stdout
>> - # and stderr are redirected *by pysh caller*. This the
>> reason for the
>> - # --redirect pysh() option.
>> - # Now, we want to know they are the same because we
>> sometimes need to
>> - # transform the command output, mostly remove CR-LF to
>> ensure that
>> - # command output is unix-like. Cygwin utilies are a special
>> case because
>> - # they explicitely set their output streams to binary mode,
>> so we have
>> - # nothing to do. For all others commands, we have to guess
>> whether they
>> - # are sending text data, in which case the transformation
>> must be done.
>> - # Again, the NUL character test is unreliable but should be
>> enough for
>> - # hg tests.
>> - redirected = stdout.fileno()==stderr.fileno()
>> - if not redirected:
>> - p = subprocess.Popen([name] + args, cwd=env['PWD'],
>> env=exec_env,
>> - stdin=stdin, stdout=subprocess.PIPE,
>> stderr=subprocess.PIPE)
>> - else:
>> - p = subprocess.Popen([name] + args, cwd=env['PWD'],
>> env=exec_env,
>> - stdin=stdin, stdout=subprocess.PIPE,
>> stderr=subprocess.STDOUT)
>> - out, err = p.communicate()
>> - except WindowsError as e:
>> - raise UtilityError(str(e))
>> -
>> - if not unixoutput:
>> - def encode(s):
>> - if '\0' in s:
>> - return s
>> - return s.replace('\r\n', '\n')
>> - else:
>> - encode = lambda s: s
>> -
>> - if rewriteslashes(name, args):
>> - encode1_ = encode
>> - def encode(s):
>> - s = encode1_(s)
>> - s = s.replace('\\\\', '\\')
>> - s = s.replace('\\', '/')
>> - return s
>> -
>> - if ishg:
>> - encode2_ = encode
>> - def encode(s):
>> - return rewritehg(encode2_(s))
>> -
>> - stdout.write(encode(out))
>> - if not redirected:
>> - stderr.write(encode(err))
>> - return p.returncode
>> -
>> diff --git a/bitbake/lib/bb/pysh/interp.py
>> b/bitbake/lib/bb/pysh/interp.py deleted file mode 100644
>> index d14ecf3..0000000
>> --- a/bitbake/lib/bb/pysh/interp.py
>> +++ /dev/null
>> @@ -1,1367 +0,0 @@
>> -# interp.py - shell interpreter for pysh.
>> -#
>> -# Copyright 2007 Patrick Mezard
>> -#
>> -# This software may be used and distributed according to the terms
>> -# of the GNU General Public License, incorporated herein by
>> reference. -
>> -"""Implement the shell interpreter.
>> -
>> -Most references are made to "The Open Group Base Specifications
>> Issue 6".
>> -<http://www.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html>
>> -""" -# TODO: document the fact input streams must implement fileno()
>> so Popen will work correctly. -# it requires non-stdin stream to be
>> implemented as files. Still to be tested... -# DOC: pathsep is used
>> in PATH instead of ':'. Clearly, there are path syntax issues here.
>> -# TODO: stop command execution upon error. -# TODO: sort out the
>> filename/io_number mess. It should be possible to use filenames only.
>> -# TODO: review subshell implementation -# TODO: test environment
>> cloning for non-special builtins -# TODO: set -x should not rebuild
>> commands from tokens, assignments/redirections are lost -# TODO: unit
>> test for variable assignment -# TODO: test error management wrt error
>> type/utility type -# TODO: test for binary output everywhere
>> -# BUG: debug-parsing does not pass log file to PLY. Maybe a PLY
>> upgrade is necessary. -import base64
>> -import cPickle as pickle
>> -import errno
>> -import glob
>> -import os
>> -import re
>> -import subprocess
>> -import sys
>> -import tempfile
>> -
>> -try:
>> - s = set()
>> - del s
>> -except NameError:
>> - from Set import Set as set
>> -
>> -import builtin
>> -from sherrors import *
>> -import pyshlex
>> -import pyshyacc
>> -
>> -def mappend(func, *args, **kargs):
>> - """Like map but assume func returns a list. Returned lists are
>> merged into
>> - a single one.
>> - """
>> - return reduce(lambda a,b: a+b, map(func, *args, **kargs), [])
>> -
>> -class FileWrapper:
>> - """File object wrapper to ease debugging.
>> -
>> - Allow mode checking and implement file duplication through a
>> simple
>> - reference counting scheme. Not sure the latter is really useful
>> since
>> - only real file descriptors can be used.
>> - """
>> - def __init__(self, mode, file, close=True):
>> - if mode not in ('r', 'w', 'a'):
>> - raise IOError('invalid mode: %s' % mode)
>> - self._mode = mode
>> - self._close = close
>> - if isinstance(file, FileWrapper):
>> - if file._refcount[0] <= 0:
>> - raise IOError(0, 'Error')
>> - self._refcount = file._refcount
>> - self._refcount[0] += 1
>> - self._file = file._file
>> - else:
>> - self._refcount = [1]
>> - self._file = file
>> -
>> - def dup(self):
>> - return FileWrapper(self._mode, self, self._close)
>> -
>> - def fileno(self):
>> - """fileno() should be only necessary for input streams."""
>> - return self._file.fileno()
>> -
>> - def read(self, size=-1):
>> - if self._mode!='r':
>> - raise IOError(0, 'Error')
>> - return self._file.read(size)
>> -
>> - def readlines(self, *args, **kwargs):
>> - return self._file.readlines(*args, **kwargs)
>> -
>> - def write(self, s):
>> - if self._mode not in ('w', 'a'):
>> - raise IOError(0, 'Error')
>> - return self._file.write(s)
>> -
>> - def flush(self):
>> - self._file.flush()
>> -
>> - def close(self):
>> - if not self._refcount:
>> - return
>> - assert self._refcount[0] > 0
>> -
>> - self._refcount[0] -= 1
>> - if self._refcount[0] == 0:
>> - self._mode = 'c'
>> - if self._close:
>> - self._file.close()
>> - self._refcount = None
>> -
>> - def mode(self):
>> - return self._mode
>> -
>> - def __getattr__(self, name):
>> - if name == 'name':
>> - self.name = getattr(self._file, name)
>> - return self.name
>> - else:
>> - raise AttributeError(name)
>> -
>> - def __del__(self):
>> - self.close()
>> -
>> -
>> -def win32_open_devnull(mode):
>> - return open('NUL', mode)
>> -
>> -
>> -class Redirections:
>> - """Stores open files and their mapping to pseudo-sh file
>> descriptor.
>> - """
>> - # BUG: redirections are not handled correctly: 1>&3 2>&3 3>&4
>> does
>> - # not make 1 to redirect to 4
>> - def __init__(self, stdin=None, stdout=None, stderr=None):
>> - self._descriptors = {}
>> - if stdin is not None:
>> - self._add_descriptor(0, stdin)
>> - if stdout is not None:
>> - self._add_descriptor(1, stdout)
>> - if stderr is not None:
>> - self._add_descriptor(2, stderr)
>> -
>> - def add_here_document(self, interp, name, content,
>> io_number=None):
>> - if io_number is None:
>> - io_number = 0
>> -
>> - if name==pyshlex.unquote_wordtree(name):
>> - content = interp.expand_here_document(('TOKEN', content))
>> -
>> - # Write document content in a temporary file
>> - tmp = tempfile.TemporaryFile()
>> - try:
>> - tmp.write(content)
>> - tmp.flush()
>> - tmp.seek(0)
>> - self._add_descriptor(io_number, FileWrapper('r', tmp))
>> - except:
>> - tmp.close()
>> - raise
>> -
>> - def add(self, interp, op, filename, io_number=None):
>> - if op not in ('<', '>', '>|', '>>', '>&'):
>> - # TODO: add descriptor duplication and
>> here_documents
>> - raise RedirectionError('Unsupported redirection operator
>> "%s"' % op)
>> -
>> - if io_number is not None:
>> - io_number = int(io_number)
>> -
>> - if (op == '>&' and filename.isdigit()) or filename=='-':
>> - # No expansion for file descriptors, quote them if you
>> want a filename
>> - fullname = filename
>> - else:
>> - if filename.startswith('/'):
>> - # TODO: win32 kludge
>> - if filename=='/dev/null':
>> - fullname = 'NUL'
>> - else:
>> - # TODO: handle absolute pathnames, they are
>> unlikely to exist on the
>> - # current platform (win32 for instance).
>> - raise NotImplementedError()
>> - else:
>> - fullname = interp.expand_redirection(('TOKEN',
>> filename))
>> - if not fullname:
>> - raise RedirectionError('%s: ambiguous redirect'
>> % filename)
>> - # Build absolute path based on PWD
>> - fullname = os.path.join(interp.get_env()['PWD'],
>> fullname)
>> -
>> - if op=='<':
>> - return self._add_input_redirection(interp, fullname,
>> io_number)
>> - elif op in ('>', '>|'):
>> - clobber = ('>|'==op)
>> - return self._add_output_redirection(interp, fullname,
>> io_number, clobber)
>> - elif op=='>>':
>> - return self._add_output_appending(interp, fullname,
>> io_number)
>> - elif op=='>&':
>> - return self._dup_output_descriptor(fullname, io_number)
>> -
>> - def close(self):
>> - if self._descriptors is not None:
>> - for desc in self._descriptors.itervalues():
>> - desc.flush()
>> - desc.close()
>> - self._descriptors = None
>> -
>> - def stdin(self):
>> - return self._descriptors[0]
>> -
>> - def stdout(self):
>> - return self._descriptors[1]
>> -
>> - def stderr(self):
>> - return self._descriptors[2]
>> -
>> - def clone(self):
>> - clone = Redirections()
>> - for desc, fileobj in self._descriptors.iteritems():
>> - clone._descriptors[desc] = fileobj.dup()
>> - return clone
>> -
>> - def _add_output_redirection(self, interp, filename, io_number,
>> clobber):
>> - if io_number is None:
>> - # io_number default to standard output
>> - io_number = 1
>> -
>> - if not clobber and interp.get_env().has_opt('-C') and
>> os.path.isfile(filename):
>> - # File already exist in no-clobber mode, bail out
>> - raise RedirectionError('File "%s" already exists' %
>> filename)
>> -
>> - # Open and register
>> - self._add_file_descriptor(io_number, filename, 'w')
>> -
>> - def _add_output_appending(self, interp, filename, io_number):
>> - if io_number is None:
>> - io_number = 1
>> - self._add_file_descriptor(io_number, filename, 'a')
>> -
>> - def _add_input_redirection(self, interp, filename, io_number):
>> - if io_number is None:
>> - io_number = 0
>> - self._add_file_descriptor(io_number, filename, 'r')
>> -
>> - def _add_file_descriptor(self, io_number, filename, mode):
>> - try:
>> - if filename.startswith('/'):
>> - if filename=='/dev/null':
>> - f = win32_open_devnull(mode+'b')
>> - else:
>> - # TODO: handle absolute pathnames, they are
>> unlikely to exist on the
>> - # current platform (win32 for instance).
>> - raise NotImplementedError('cannot open absolute
>> path %s' % repr(filename))
>> - else:
>> - f = file(filename, mode+'b')
>> - except IOError as e:
>> - raise RedirectionError(str(e))
>> -
>> - wrapper = None
>> - try:
>> - wrapper = FileWrapper(mode, f)
>> - f = None
>> - self._add_descriptor(io_number, wrapper)
>> - except:
>> - if f: f.close()
>> - if wrapper: wrapper.close()
>> - raise
>> -
>> - def _dup_output_descriptor(self, source_fd, dest_fd):
>> - if source_fd is None:
>> - source_fd = 1
>> - self._dup_file_descriptor(source_fd, dest_fd, 'w')
>> -
>> - def _dup_file_descriptor(self, source_fd, dest_fd, mode):
>> - source_fd = int(source_fd)
>> - if source_fd not in self._descriptors:
>> - raise RedirectionError('"%s" is not a valid file
>> descriptor' % str(source_fd))
>> - source = self._descriptors[source_fd]
>> -
>> - if source.mode()!=mode:
>> - raise RedirectionError('Descriptor %s cannot be
>> duplicated in mode "%s"' % (str(source), mode))
>> -
>> - if dest_fd=='-':
>> - # Close the source descriptor
>> - del self._descriptors[source_fd]
>> - source.close()
>> - else:
>> - dest_fd = int(dest_fd)
>> - if dest_fd not in self._descriptors:
>> - raise RedirectionError('Cannot replace file
>> descriptor %s' % str(dest_fd))
>> -
>> - dest = self._descriptors[dest_fd]
>> - if dest.mode()!=mode:
>> - raise RedirectionError('Descriptor %s cannot be
>> cannot be redirected in mode "%s"' % (str(dest), mode))
>> -
>> - self._descriptors[dest_fd] = source.dup()
>> - dest.close()
>> -
>> - def _add_descriptor(self, io_number, file):
>> - io_number = int(io_number)
>> -
>> - if io_number in self._descriptors:
>> - # Close the current descriptor
>> - d = self._descriptors[io_number]
>> - del self._descriptors[io_number]
>> - d.close()
>> -
>> - self._descriptors[io_number] = file
>> -
>> - def __str__(self):
>> - names = [('%d=%r' % (k, getattr(v, 'name', None))) for k,v
>> - in self._descriptors.iteritems()]
>> - names = ','.join(names)
>> - return 'Redirections(%s)' % names
>> -
>> - def __del__(self):
>> - self.close()
>> -
>> -def cygwin_to_windows_path(path):
>> - """Turn /cygdrive/c/foo into c:/foo, or return path if it
>> - is not a cygwin path.
>> - """
>> - if not path.startswith('/cygdrive/'):
>> - return path
>> - path = path[len('/cygdrive/'):]
>> - path = path[:1] + ':' + path[1:]
>> - return path
>> -
>> -def win32_to_unix_path(path):
>> - if path is not None:
>> - path = path.replace('\\', '/')
>> - return path
>> -
>> -_RE_SHEBANG = re.compile(r'^\#!\s?([^\s]+)(?:\s([^\s]+))?')
>> -_SHEBANG_CMDS = {
>> - '/usr/bin/env': 'env',
>> - '/bin/sh': 'pysh',
>> - 'python': 'python',
>> -}
>> -
>> -def resolve_shebang(path, ignoreshell=False):
>> - """Return a list of arguments as shebang interpreter call or an
>> empty list
>> - if path does not refer to an executable script.
>> - See <http://www.opengroup.org/austin/docs/austin_51r2.txt>.
>> -
>> - ignoreshell - set to True to ignore sh shebangs. Return an empty
>> list instead.
>> - """
>> - try:
>> - f = file(path)
>> - try:
>> - # At most 80 characters in the first line
>> - header = f.read(80).splitlines()[0]
>> - finally:
>> - f.close()
>> -
>> - m = _RE_SHEBANG.search(header)
>> - if not m:
>> - return []
>> - cmd, arg = m.group(1,2)
>> - if os.path.isfile(cmd):
>> - # Keep this one, the hg script for instance contains a
>> weird windows
>> - # shebang referencing the current python install.
>> - cmdfile = os.path.basename(cmd).lower()
>> - if cmdfile == 'python.exe':
>> - cmd = 'python'
>> - pass
>> - elif cmd not in _SHEBANG_CMDS:
>> - raise CommandNotFound('Unknown interpreter "%s"
>> referenced in '\
>> - 'shebang' % header)
>> - cmd = _SHEBANG_CMDS.get(cmd)
>> - if cmd is None or (ignoreshell and cmd == 'pysh'):
>> - return []
>> - if arg is None:
>> - return [cmd, win32_to_unix_path(path)]
>> - return [cmd, arg, win32_to_unix_path(path)]
>> - except IOError as e:
>> - if e.errno!=errno.ENOENT and \
>> - (e.errno!=errno.EPERM and not os.path.isdir(path)): #
>> Opening a directory raises EPERM
>> - raise
>> - return []
>> -
>> -def win32_find_in_path(name, path):
>> - if isinstance(path, str):
>> - path = path.split(os.pathsep)
>> -
>> - exts = os.environ.get('PATHEXT', '').lower().split(os.pathsep)
>> - for p in path:
>> - p_name = os.path.join(p, name)
>> -
>> - prefix = resolve_shebang(p_name)
>> - if prefix:
>> - return prefix
>> -
>> - for ext in exts:
>> - p_name_ext = p_name + ext
>> - if os.path.exists(p_name_ext):
>> - return [win32_to_unix_path(p_name_ext)]
>> - return []
>> -
>> -class Traps(dict):
>> - def __setitem__(self, key, value):
>> - if key not in ('EXIT',):
>> - raise NotImplementedError()
>> - super(Traps, self).__setitem__(key, value)
>> -
>> -# IFS white spaces character class
>> -_IFS_WHITESPACES = (' ', '\t', '\n')
>> -
>> -class Environment:
>> - """Environment holds environment variables, export table,
>> function
>> - definitions and whatever is defined in 2.12 "Shell Execution
>> Environment",
>> - redirection excepted.
>> - """
>> - def __init__(self, pwd):
>> - self._opt = set() #Shell options
>> -
>> - self._functions = {}
>> - self._env = {'?': '0', '#': '0'}
>> - self._exported = set([
>> - 'HOME', 'IFS', 'PATH'
>> - ])
>> -
>> - # Set environment vars with side-effects
>> - self._ifs_ws = None # Set of IFS whitespace characters
>> - self._ifs_re = None # Regular expression used to split
>> between words using IFS classes
>> - self['IFS'] = ''.join(_IFS_WHITESPACES) #Default environment
>> values
>> - self['PWD'] = pwd
>> - self.traps = Traps()
>> -
>> - def clone(self, subshell=False):
>> - env = Environment(self['PWD'])
>> - env._opt = set(self._opt)
>> - for k,v in self.get_variables().iteritems():
>> - if k in self._exported:
>> - env.export(k,v)
>> - elif subshell:
>> - env[k] = v
>> -
>> - if subshell:
>> - env._functions = dict(self._functions)
>> -
>> - return env
>> -
>> - def __getitem__(self, key):
>> - if key in ('@', '*', '-', '$'):
>> - raise NotImplementedError('%s is not implemented' %
>> repr(key))
>> - return self._env[key]
>> -
>> - def get(self, key, defval=None):
>> - try:
>> - return self[key]
>> - except KeyError:
>> - return defval
>> -
>> - def __setitem__(self, key, value):
>> - if key=='IFS':
>> - # Update the whitespace/non-whitespace classes
>> - self._update_ifs(value)
>> - elif key=='PWD':
>> - pwd = os.path.abspath(value)
>> - if not os.path.isdir(pwd):
>> - raise VarAssignmentError('Invalid directory %s' %
>> value)
>> - value = pwd
>> - elif key in ('?', '!'):
>> - value = str(int(value))
>> - self._env[key] = value
>> -
>> - def __delitem__(self, key):
>> - if key in ('IFS', 'PWD', '?'):
>> - raise VarAssignmentError('%s cannot be unset' % key)
>> - del self._env[key]
>> -
>> - def __contains__(self, item):
>> - return item in self._env
>> -
>> - def set_positional_args(self, args):
>> - """Set the content of 'args' as positional argument from 1
>> to len(args).
>> - Return previous argument as a list of strings.
>> - """
>> - # Save and remove previous arguments
>> - prevargs = []
>> - for i in range(int(self._env['#'])):
>> - i = str(i+1)
>> - prevargs.append(self._env[i])
>> - del self._env[i]
>> - self._env['#'] = '0'
>> -
>> - #Set new ones
>> - for i,arg in enumerate(args):
>> - self._env[str(i+1)] = str(arg)
>> - self._env['#'] = str(len(args))
>> -
>> - return prevargs
>> -
>> - def get_positional_args(self):
>> - return [self._env[str(i+1)] for i in
>> range(int(self._env['#']))]
>> -
>> - def get_variables(self):
>> - return dict(self._env)
>> -
>> - def export(self, key, value=None):
>> - if value is not None:
>> - self[key] = value
>> - self._exported.add(key)
>> -
>> - def get_exported(self):
>> - return [(k,self._env.get(k)) for k in self._exported]
>> -
>> - def split_fields(self, word):
>> - if not self._ifs_ws or not word:
>> - return [word]
>> - return re.split(self._ifs_re, word)
>> -
>> - def _update_ifs(self, value):
>> - """Update the split_fields related variables when IFS
>> character set is
>> - changed.
>> - """
>> - # TODO: handle NULL IFS
>> -
>> - # Separate characters in whitespace and non-whitespace
>> - chars = set(value)
>> - ws = [c for c in chars if c in _IFS_WHITESPACES]
>> - nws = [c for c in chars if c not in _IFS_WHITESPACES]
>> -
>> - # Keep whitespaces in a string for left and right stripping
>> - self._ifs_ws = ''.join(ws)
>> -
>> - # Build a regexp to split fields
>> - trailing = '[' + ''.join([re.escape(c) for c in ws]) + ']'
>> - if nws:
>> - # First, the single non-whitespace occurence.
>> - nws = '[' + ''.join([re.escape(c) for c in nws]) + ']'
>> - nws = '(?:' + trailing + '*' + nws + trailing + '*' +
>> '|' + trailing + '+)'
>> - else:
>> - # Then mix all parts with quantifiers
>> - nws = trailing + '+'
>> - self._ifs_re = re.compile(nws)
>> -
>> - def has_opt(self, opt, val=None):
>> - return (opt, val) in self._opt
>> -
>> - def set_opt(self, opt, val=None):
>> - self._opt.add((opt, val))
>> -
>> - def find_in_path(self, name, pwd=False):
>> - path = self._env.get('PATH', '').split(os.pathsep)
>> - if pwd:
>> - path[:0] = [self['PWD']]
>> - if os.name == 'nt':
>> - return win32_find_in_path(name, self._env.get('PATH',
>> ''))
>> - else:
>> - raise NotImplementedError()
>> -
>> - def define_function(self, name, body):
>> - if not is_name(name):
>> - raise ShellSyntaxError('%s is not a valid function name'
>> % repr(name))
>> - self._functions[name] = body
>> -
>> - def remove_function(self, name):
>> - del self._functions[name]
>> -
>> - def is_function(self, name):
>> - return name in self._functions
>> -
>> - def get_function(self, name):
>> - return self._functions.get(name)
>> -
>> -
>> -name_charset =
>> 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_'
>> -name_charset = dict(zip(name_charset,name_charset))
>> -
>> -def match_name(s):
>> - """Return the length in characters of the longest prefix made of
>> name
>> - allowed characters in s.
>> - """
>> - for i,c in enumerate(s):
>> - if c not in name_charset:
>> - return s[:i]
>> - return s
>> -
>> -def is_name(s):
>> - return len([c for c in s if c not in name_charset])<=0
>> -
>> -def is_special_param(c):
>> - return len(c)==1 and c in ('@','*','#','?','-','$','!','0')
>> -
>> -def utility_not_implemented(name, *args, **kwargs):
>> - raise NotImplementedError('%s utility is not implemented' % name)
>> -
>> -
>> -class Utility:
>> - """Define utilities properties:
>> - func -- utility callable. See builtin module for utility samples.
>> - is_special -- see XCU 2.8.
>> - """
>> - def __init__(self, func, is_special=0):
>> - self.func = func
>> - self.is_special = bool(is_special)
>> -
>> -
>> -def encodeargs(args):
>> - def encodearg(s):
>> - lines = base64.encodestring(s)
>> - lines = [l.splitlines()[0] for l in lines]
>> - return ''.join(lines)
>> -
>> - s = pickle.dumps(args)
>> - return encodearg(s)
>> -
>> -def decodeargs(s):
>> - s = base64.decodestring(s)
>> - return pickle.loads(s)
>> -
>> -
>> -class GlobError(Exception):
>> - pass
>> -
>> -class Options:
>> - def __init__(self):
>> - # True if Mercurial operates with binary streams
>> - self.hgbinary = True
>> -
>> -class Interpreter:
>> - # Implementation is very basic: the execute() method just makes
>> a DFS on the
>> - # AST and execute nodes one by one. Nodes are tuple (name,obj)
>> where name
>> - # is a string identifier and obj the AST element returned by the
>> parser.
>> - #
>> - # Handler are named after the node identifiers.
>> - # TODO: check node names and remove the switch in execute with
>> some
>> - # dynamic getattr() call to find node handlers.
>> - """Shell interpreter.
>> -
>> - The following debugging flags can be passed:
>> - debug-parsing - enable PLY debugging.
>> - debug-tree - print the generated AST.
>> - debug-cmd - trace command execution before word expansion, plus
>> exit status.
>> - debug-utility - trace utility execution.
>> - """
>> -
>> - # List supported commands.
>> - COMMANDS = {
>> - 'cat': Utility(builtin.utility_cat,),
>> - 'cd': Utility(builtin.utility_cd,),
>> - ':': Utility(builtin.utility_colon,),
>> - 'echo': Utility(builtin.utility_echo),
>> - 'env': Utility(builtin.utility_env),
>> - 'exit': Utility(builtin.utility_exit),
>> - 'export': Utility(builtin.builtin_export,
>> is_special=1),
>> - 'egrep': Utility(builtin.utility_egrep),
>> - 'fgrep': Utility(builtin.utility_fgrep),
>> - 'gunzip': Utility(builtin.utility_gunzip),
>> - 'kill': Utility(builtin.utility_kill),
>> - 'mkdir': Utility(builtin.utility_mkdir),
>> - 'netstat': Utility(builtin.utility_netstat),
>> - 'printf': Utility(builtin.utility_printf),
>> - 'pwd': Utility(builtin.utility_pwd),
>> - 'return': Utility(builtin.builtin_return,
>> is_special=1),
>> - 'sed': Utility(builtin.utility_sed,),
>> - 'set': Utility(builtin.builtin_set,),
>> - 'shift': Utility(builtin.builtin_shift,),
>> - 'sleep': Utility(builtin.utility_sleep,),
>> - 'sort': Utility(builtin.utility_sort,),
>> - 'trap': Utility(builtin.builtin_trap,
>> is_special=1),
>> - 'true': Utility(builtin.utility_true),
>> - 'unset': Utility(builtin.builtin_unset,
>> is_special=1),
>> - 'wait': Utility(builtin.builtin_wait,
>> is_special=1),
>> - }
>> -
>> - def __init__(self, pwd, debugflags = [], env=None, redirs=None,
>> stdin=None,
>> - stdout=None, stderr=None, opts=Options()):
>> - self._env = env
>> - if self._env is None:
>> - self._env = Environment(pwd)
>> - self._children = {}
>> -
>> - self._redirs = redirs
>> - self._close_redirs = False
>> -
>> - if self._redirs is None:
>> - if stdin is None:
>> - stdin = sys.stdin
>> - if stdout is None:
>> - stdout = sys.stdout
>> - if stderr is None:
>> - stderr = sys.stderr
>> - stdin = FileWrapper('r', stdin, False)
>> - stdout = FileWrapper('w', stdout, False)
>> - stderr = FileWrapper('w', stderr, False)
>> - self._redirs = Redirections(stdin, stdout, stderr)
>> - self._close_redirs = True
>> -
>> - self._debugflags = list(debugflags)
>> - self._logfile = sys.stderr
>> - self._options = opts
>> -
>> - def close(self):
>> - """Must be called when the interpreter is no longer used."""
>> - script = self._env.traps.get('EXIT')
>> - if script:
>> - try:
>> - self.execute_script(script=script)
>> - except:
>> - pass
>> -
>> - if self._redirs is not None and self._close_redirs:
>> - self._redirs.close()
>> - self._redirs = None
>> -
>> - def log(self, s):
>> - self._logfile.write(s)
>> - self._logfile.flush()
>> -
>> - def __getitem__(self, key):
>> - return self._env[key]
>> -
>> - def __setitem__(self, key, value):
>> - self._env[key] = value
>> -
>> - def options(self):
>> - return self._options
>> -
>> - def redirect(self, redirs, ios):
>> - def add_redir(io):
>> - if isinstance(io, pyshyacc.IORedirect):
>> - redirs.add(self, io.op, io.filename, io.io_number)
>> - else:
>> - redirs.add_here_document(self, io.name, io.content,
>> io.io_number)
>> -
>> - map(add_redir, ios)
>> - return redirs
>> -
>> - def execute_script(self, script=None, ast=None, sourced=False,
>> - scriptpath=None):
>> - """If script is not None, parse the input. Otherwise takes
>> the supplied
>> - AST. Then execute the AST.
>> - Return the script exit status.
>> - """
>> - try:
>> - if scriptpath is not None:
>> - self._env['0'] = os.path.abspath(scriptpath)
>> -
>> - if script is not None:
>> - debug_parsing = ('debug-parsing' in
>> self._debugflags)
>> - cmds, script = pyshyacc.parse(script, True,
>> debug_parsing)
>> - if 'debug-tree' in self._debugflags:
>> - pyshyacc.print_commands(cmds, self._logfile)
>> - self._logfile.flush()
>> - else:
>> - cmds, script = ast, ''
>> -
>> - status = 0
>> - for cmd in cmds:
>> - try:
>> - status = self.execute(cmd)
>> - except ExitSignal as e:
>> - if sourced:
>> - raise
>> - status = int(e.args[0])
>> - return status
>> - except ShellError:
>> - self._env['?'] = 1
>> - raise
>> - if 'debug-utility' in self._debugflags or
>> 'debug-cmd' in self._debugflags:
>> - self.log('returncode ' + str(status)+ '\n')
>> - return status
>> - except CommandNotFound as e:
>> - print >>self._redirs.stderr, str(e)
>> - self._redirs.stderr.flush()
>> - # Command not found by non-interactive shell
>> - # return 127
>> - raise
>> - except RedirectionError as e:
>> - # TODO: should be handled depending on the utility status
>> - print >>self._redirs.stderr, str(e)
>> - self._redirs.stderr.flush()
>> - # Command not found by non-interactive shell
>> - # return 127
>> - raise
>> -
>> - def dotcommand(self, env, args):
>> - if len(args) < 1:
>> - raise ShellError('. expects at least one argument')
>> - path = args[0]
>> - if '/' not in path:
>> - found = env.find_in_path(args[0], True)
>> - if found:
>> - path = found[0]
>> - script = file(path).read()
>> - return self.execute_script(script=script, sourced=True)
>> -
>> - def execute(self, token, redirs=None):
>> - """Execute and AST subtree with supplied redirections
>> overriding default
>> - interpreter ones.
>> - Return the exit status.
>> - """
>> - if not token:
>> - return 0
>> -
>> - if redirs is None:
>> - redirs = self._redirs
>> -
>> - if isinstance(token, list):
>> - # Commands sequence
>> - res = 0
>> - for t in token:
>> - res = self.execute(t, redirs)
>> - return res
>> -
>> - type, value = token
>> - status = 0
>> - if type=='simple_command':
>> - redirs_copy = redirs.clone()
>> - try:
>> - # TODO: define and handle command return values
>> - # TODO: implement set -e
>> - status = self._execute_simple_command(value,
>> redirs_copy)
>> - finally:
>> - redirs_copy.close()
>> - elif type=='pipeline':
>> - status = self._execute_pipeline(value, redirs)
>> - elif type=='and_or':
>> - status = self._execute_and_or(value, redirs)
>> - elif type=='for_clause':
>> - status = self._execute_for_clause(value, redirs)
>> - elif type=='while_clause':
>> - status = self._execute_while_clause(value, redirs)
>> - elif type=='function_definition':
>> - status = self._execute_function_definition(value, redirs)
>> - elif type=='brace_group':
>> - status = self._execute_brace_group(value, redirs)
>> - elif type=='if_clause':
>> - status = self._execute_if_clause(value, redirs)
>> - elif type=='subshell':
>> - status = self.subshell(ast=value.cmds, redirs=redirs)
>> - elif type=='async':
>> - status = self._asynclist(value)
>> - elif type=='redirect_list':
>> - redirs_copy = self.redirect(redirs.clone(), value.redirs)
>> - try:
>> - status = self.execute(value.cmd, redirs_copy)
>> - finally:
>> - redirs_copy.close()
>> - else:
>> - raise NotImplementedError('Unsupported token type ' +
>> type) -
>> - if status < 0:
>> - status = 255
>> - return status
>> -
>> - def _execute_if_clause(self, if_clause, redirs):
>> - cond_status = self.execute(if_clause.cond, redirs)
>> - if cond_status==0:
>> - return self.execute(if_clause.if_cmds, redirs)
>> - else:
>> - return self.execute(if_clause.else_cmds, redirs)
>> -
>> - def _execute_brace_group(self, group, redirs):
>> - status = 0
>> - for cmd in group.cmds:
>> - status = self.execute(cmd, redirs)
>> - return status
>> -
>> - def _execute_function_definition(self, fundef, redirs):
>> - self._env.define_function(fundef.name, fundef.body)
>> - return 0
>> -
>> - def _execute_while_clause(self, while_clause, redirs):
>> - status = 0
>> - while 1:
>> - cond_status = 0
>> - for cond in while_clause.condition:
>> - cond_status = self.execute(cond, redirs)
>> -
>> - if cond_status:
>> - break
>> -
>> - for cmd in while_clause.cmds:
>> - status = self.execute(cmd, redirs)
>> -
>> - return status
>> -
>> - def _execute_for_clause(self, for_clause, redirs):
>> - if not is_name(for_clause.name):
>> - raise ShellSyntaxError('%s is not a valid name' %
>> repr(for_clause.name))
>> - items = mappend(self.expand_token, for_clause.items)
>> -
>> - status = 0
>> - for item in items:
>> - self._env[for_clause.name] = item
>> - for cmd in for_clause.cmds:
>> - status = self.execute(cmd, redirs)
>> - return status
>> -
>> - def _execute_and_or(self, or_and, redirs):
>> - res = self.execute(or_and.left, redirs)
>> - if (or_and.op=='&&' and res==0) or (or_and.op!='&&' and
>> res!=0):
>> - res = self.execute(or_and.right, redirs)
>> - return res
>> -
>> - def _execute_pipeline(self, pipeline, redirs):
>> - if len(pipeline.commands)==1:
>> - status = self.execute(pipeline.commands[0], redirs)
>> - else:
>> - # Execute all commands one after the other
>> - status = 0
>> - inpath, outpath = None, None
>> - try:
>> - # Commands inputs and outputs cannot really be
>> plugged as done
>> - # by a real shell. Run commands sequentially and
>> chain their
>> - # input/output throught temporary files.
>> - tmpfd, inpath = tempfile.mkstemp()
>> - os.close(tmpfd)
>> - tmpfd, outpath = tempfile.mkstemp()
>> - os.close(tmpfd)
>> -
>> - inpath = win32_to_unix_path(inpath)
>> - outpath = win32_to_unix_path(outpath)
>> -
>> - for i, cmd in enumerate(pipeline.commands):
>> - call_redirs = redirs.clone()
>> - try:
>> - if i!=0:
>> - call_redirs.add(self, '<', inpath)
>> - if i!=len(pipeline.commands)-1:
>> - call_redirs.add(self, '>', outpath)
>> -
>> - status = self.execute(cmd, call_redirs)
>> -
>> - # Chain inputs/outputs
>> - inpath, outpath = outpath, inpath
>> - finally:
>> - call_redirs.close()
>> - finally:
>> - if inpath: os.remove(inpath)
>> - if outpath: os.remove(outpath)
>> -
>> - if pipeline.reverse_status:
>> - status = int(not status)
>> - self._env['?'] = status
>> - return status
>> -
>> - def _execute_function(self, name, args, interp, env, stdin,
>> stdout, stderr, *others):
>> - assert interp is self
>> -
>> - func = env.get_function(name)
>> - #Set positional parameters
>> - prevargs = None
>> - try:
>> - prevargs = env.set_positional_args(args)
>> - try:
>> - redirs = Redirections(stdin.dup(), stdout.dup(),
>> stderr.dup())
>> - try:
>> - status = self.execute(func, redirs)
>> - finally:
>> - redirs.close()
>> - except ReturnSignal as e:
>> - status = int(e.args[0])
>> - env['?'] = status
>> - return status
>> - finally:
>> - #Reset positional parameters
>> - if prevargs is not None:
>> - env.set_positional_args(prevargs)
>> -
>> - def _execute_simple_command(self, token, redirs):
>> - """Can raise ReturnSignal when return builtin is called,
>> ExitSignal when
>> - exit is called, and other shell exceptions upon builtin
>> failures.
>> - """
>> - debug_command = 'debug-cmd' in self._debugflags
>> - if debug_command:
>> - self.log('word' + repr(token.words) + '\n')
>> - self.log('assigns' + repr(token.assigns) + '\n')
>> - self.log('redirs' + repr(token.redirs) + '\n')
>> -
>> - is_special = None
>> - env = self._env
>> -
>> - try:
>> - # Word expansion
>> - args = []
>> - for word in token.words:
>> - args += self.expand_token(word)
>> - if is_special is None and args:
>> - is_special = env.is_function(args[0]) or \
>> - (args[0] in self.COMMANDS and
>> self.COMMANDS[args[0]].is_special)
>> -
>> - if debug_command:
>> - self.log('_execute_simple_command' + str(args) +
>> '\n')
>> -
>> - if not args:
>> - # Redirections happen is a subshell
>> - redirs = redirs.clone()
>> - elif not is_special:
>> - env = self._env.clone()
>> -
>> - # Redirections
>> - self.redirect(redirs, token.redirs)
>> -
>> - # Variables assignments
>> - res = 0
>> - for type,(k,v) in token.assigns:
>> - status, expanded = self.expand_variable((k,v))
>> - if status is not None:
>> - res = status
>> - if args:
>> - env.export(k, expanded)
>> - else:
>> - env[k] = expanded
>> -
>> - if args and args[0] in ('.', 'source'):
>> - res = self.dotcommand(env, args[1:])
>> - elif args:
>> - if args[0] in self.COMMANDS:
>> - command = self.COMMANDS[args[0]]
>> - elif env.is_function(args[0]):
>> - command = Utility(self._execute_function,
>> is_special=True)
>> - else:
>> - if not '/' in args[0].replace('\\', '/'):
>> - cmd = env.find_in_path(args[0])
>> - if not cmd:
>> - # TODO: test error code on unknown
>> command => 127
>> - raise CommandNotFound('Unknown command:
>> "%s"' % args[0])
>> - else:
>> - # Handle commands like '/cygdrive/c/foo.bat'
>> - cmd = cygwin_to_windows_path(args[0])
>> - if not os.path.exists(cmd):
>> - raise CommandNotFound('%s: No such file
>> or directory' % args[0])
>> - shebang = resolve_shebang(cmd)
>> - if shebang:
>> - cmd = shebang
>> - else:
>> - cmd = [cmd]
>> - args[0:1] = cmd
>> - command = Utility(builtin.run_command)
>> -
>> - # Command execution
>> - if 'debug-cmd' in self._debugflags:
>> - self.log('redirections ' + str(redirs) + '\n')
>> -
>> - res = command.func(args[0], args[1:], self, env,
>> - redirs.stdin(), redirs.stdout(),
>> - redirs.stderr(), self._debugflags)
>> -
>> - if self._env.has_opt('-x'):
>> - # Trace command execution in shell environment
>> - # BUG: would be hard to reproduce a real shell
>> behaviour since
>> - # the AST is not annotated with source lines/tokens.
>> - self._redirs.stdout().write(' '.join(args))
>> -
>> - except ReturnSignal:
>> - raise
>> - except ShellError as e:
>> - if is_special or isinstance(e, (ExitSignal,
>> - ShellSyntaxError,
>> ExpansionError)):
>> - raise e
>> - self._redirs.stderr().write(str(e)+'\n')
>> - return 1
>> -
>> - return res
>> -
>> - def expand_token(self, word):
>> - """Expand a word as specified in [2.6 Word Expansions].
>> Return the list
>> - of expanded words.
>> - """
>> - status, wtrees = self._expand_word(word)
>> - return map(pyshlex.wordtree_as_string, wtrees)
>> -
>> - def expand_variable(self, word):
>> - """Return a status code (or None if no command expansion
>> occurred)
>> - and a single word.
>> - """
>> - status, wtrees = self._expand_word(word, pathname=False,
>> split=False)
>> - words = map(pyshlex.wordtree_as_string, wtrees)
>> - assert len(words)==1
>> - return status, words[0]
>> -
>> - def expand_here_document(self, word):
>> - """Return the expanded document as a single word. The here
>> document is
>> - assumed to be unquoted.
>> - """
>> - status, wtrees = self._expand_word(word, pathname=False,
>> - split=False,
>> here_document=True)
>> - words = map(pyshlex.wordtree_as_string, wtrees)
>> - assert len(words)==1
>> - return words[0]
>> -
>> - def expand_redirection(self, word):
>> - """Return a single word."""
>> - return self.expand_variable(word)[1]
>> -
>> - def get_env(self):
>> - return self._env
>> -
>> - def _expand_word(self, token, pathname=True, split=True,
>> here_document=False):
>> - wtree = pyshlex.make_wordtree(token[1],
>> here_document=here_document)
>> -
>> - # TODO: implement tilde expansion
>> - def expand(wtree):
>> - """Return a pseudo wordtree: the tree or its subelements
>> can be empty
>> - lists when no value result from the expansion.
>> - """
>> - status = None
>> - for part in wtree:
>> - if not isinstance(part, list):
>> - continue
>> - if part[0]in ("'", '\\'):
>> - continue
>> - elif part[0] in ('`', '$('):
>> - status, result = self._expand_command(part)
>> - part[:] = result
>> - elif part[0] in ('$', '${'):
>> - part[:] = self._expand_parameter(part,
>> wtree[0]=='"', split)
>> - elif part[0] in ('', '"'):
>> - status, result = expand(part)
>> - part[:] = result
>> - else:
>> - raise NotImplementedError('%s expansion is not
>> implemented'
>> - % part[0])
>> - # [] is returned when an expansion result in no-field,
>> - # like an empty $@
>> - wtree = [p for p in wtree if p != []]
>> - if len(wtree) < 3:
>> - return status, []
>> - return status, wtree
>> -
>> - status, wtree = expand(wtree)
>> - if len(wtree) == 0:
>> - return status, wtree
>> - wtree = pyshlex.normalize_wordtree(wtree)
>> -
>> - if split:
>> - wtrees = self._split_fields(wtree)
>> - else:
>> - wtrees = [wtree]
>> -
>> - if pathname:
>> - wtrees = mappend(self._expand_pathname, wtrees)
>> -
>> - wtrees = map(self._remove_quotes, wtrees)
>> - return status, wtrees
>> -
>> - def _expand_command(self, wtree):
>> - # BUG: there is something to do with backslashes and quoted
>> - # characters here
>> - command = pyshlex.wordtree_as_string(wtree[1:-1])
>> - status, output = self.subshell_output(command)
>> - return status, ['', output, '']
>> -
>> - def _expand_parameter(self, wtree, quoted=False, split=False):
>> - """Return a valid wtree or an empty list when no parameter
>> results."""
>> - # Get the parameter name
>> - # TODO: implement weird expansion rules with ':'
>> - name = pyshlex.wordtree_as_string(wtree[1:-1])
>> - if not is_name(name) and not is_special_param(name):
>> - raise ExpansionError('Bad substitution "%s"' % name)
>> - # TODO: implement special parameters
>> - if name in ('@', '*'):
>> - args = self._env.get_positional_args()
>> - if len(args) == 0:
>> - return []
>> - if len(args)<2:
>> - return ['', ''.join(args), '']
>> -
>> - sep = self._env.get('IFS', '')[:1]
>> - if split and quoted and name=='@':
>> - # Introduce a new token to tell the caller that
>> these parameters
>> - # cause a split as specified in 2.5.2
>> - return ['@'] + args + ['']
>> - else:
>> - return ['', sep.join(args), '']
>> -
>> - return ['', self._env.get(name, ''), '']
>> -
>> - def _split_fields(self, wtree):
>> - def is_empty(split):
>> - return split==['', '', '']
>> -
>> - def split_positional(quoted):
>> - # Return a list of wtree split according positional
>> parameters rules.
>> - # All remaining '@' groups are removed.
>> - assert quoted[0]=='"'
>> -
>> - splits = [[]]
>> - for part in quoted:
>> - if not isinstance(part, list) or part[0]!='@':
>> - splits[-1].append(part)
>> - else:
>> - # Empty or single argument list were dealt with
>> already
>> - assert len(part)>3
>> - # First argument must join with the beginning
>> part of the original word
>> - splits[-1].append(part[1])
>> - # Create double-quotes expressions for every
>> argument after the first
>> - for arg in part[2:-1]:
>> - splits[-1].append('"')
>> - splits.append(['"', arg])
>> - return splits
>> -
>> - # At this point, all expansions but pathnames have occured.
>> Only quoted
>> - # and positional sequences remain. Thus, all candidates for
>> field splitting
>> - # are in the tree root, or are positional splits ('@') and
>> lie in root
>> - # children.
>> - if not wtree or wtree[0] not in ('', '"'):
>> - # The whole token is quoted or empty, nothing to split
>> - return [wtree]
>> -
>> - if wtree[0]=='"':
>> - wtree = ['', wtree, '']
>> -
>> - result = [['', '']]
>> - for part in wtree[1:-1]:
>> - if isinstance(part, list):
>> - if part[0]=='"':
>> - splits = split_positional(part)
>> - if len(splits)<=1:
>> - result[-1] += [part, '']
>> - else:
>> - # Terminate the current split
>> - result[-1] += [splits[0], '']
>> - result += splits[1:-1]
>> - # Create a new split
>> - result += [['', splits[-1], '']]
>> - else:
>> - result[-1] += [part, '']
>> - else:
>> - splits = self._env.split_fields(part)
>> - if len(splits)<=1:
>> - # No split
>> - result[-1][-1] += part
>> - else:
>> - # Terminate the current resulting part and
>> create a new one
>> - result[-1][-1] += splits[0]
>> - result[-1].append('')
>> - result += [['', r, ''] for r in splits[1:-1]]
>> - result += [['', splits[-1]]]
>> - result[-1].append('')
>> -
>> - # Leading and trailing empty groups come from
>> leading/trailing blanks
>> - if result and is_empty(result[-1]):
>> - result[-1:] = []
>> - if result and is_empty(result[0]):
>> - result[:1] = []
>> - return result
>> -
>> - def _expand_pathname(self, wtree):
>> - """See [2.6.6 Pathname Expansion]."""
>> - if self._env.has_opt('-f'):
>> - return [wtree]
>> -
>> - # All expansions have been performed, only quoted sequences
>> should remain
>> - # in the tree. Generate the pattern by folding the tree,
>> escaping special
>> - # characters when appear quoted
>> - special_chars = '*?[]'
>> -
>> - def make_pattern(wtree):
>> - subpattern = []
>> - for part in wtree[1:-1]:
>> - if isinstance(part, list):
>> - part = make_pattern(part)
>> - elif wtree[0]!='':
>> - for c in part:
>> - # Meta-characters cannot be quoted
>> - if c in special_chars:
>> - raise GlobError()
>> - subpattern.append(part)
>> - return ''.join(subpattern)
>> -
>> - def pwd_glob(pattern):
>> - cwd = os.getcwd()
>> - os.chdir(self._env['PWD'])
>> - try:
>> - return glob.glob(pattern)
>> - finally:
>> - os.chdir(cwd)
>> -
>> - #TODO: check working directory issues here wrt relative
>> patterns
>> - try:
>> - pattern = make_pattern(wtree)
>> - paths = pwd_glob(pattern)
>> - except GlobError:
>> - # BUG: Meta-characters were found in quoted sequences.
>> The should
>> - # have been used literally but this is unsupported in
>> current glob module.
>> - # Instead we consider the whole tree must be used
>> literally and
>> - # therefore there is no point in globbing. This is wrong
>> when meta
>> - # characters are mixed with quoted meta in the same
>> pattern like:
>> - # < foo*"py*" >
>> - paths = []
>> -
>> - if not paths:
>> - return [wtree]
>> - return [['', path, ''] for path in paths]
>> -
>> - def _remove_quotes(self, wtree):
>> - """See [2.6.7 Quote Removal]."""
>> -
>> - def unquote(wtree):
>> - unquoted = []
>> - for part in wtree[1:-1]:
>> - if isinstance(part, list):
>> - part = unquote(part)
>> - unquoted.append(part)
>> - return ''.join(unquoted)
>> -
>> - return ['', unquote(wtree), '']
>> -
>> - def subshell(self, script=None, ast=None, redirs=None):
>> - """Execute the script or AST in a subshell, with inherited
>> redirections
>> - if redirs is not None.
>> - """
>> - if redirs:
>> - sub_redirs = redirs
>> - else:
>> - sub_redirs = redirs.clone()
>> -
>> - subshell = None
>> - try:
>> - subshell = Interpreter(None, self._debugflags,
>> self._env.clone(True),
>> - sub_redirs, opts=self._options)
>> - return subshell.execute_script(script, ast)
>> - finally:
>> - if not redirs: sub_redirs.close()
>> - if subshell: subshell.close()
>> -
>> - def subshell_output(self, script):
>> - """Execute the script in a subshell and return the captured
>> output."""
>> - # Create temporary file to capture subshell output
>> - tmpfd, tmppath = tempfile.mkstemp()
>> - try:
>> - tmpfile = os.fdopen(tmpfd, 'wb')
>> - stdout = FileWrapper('w', tmpfile)
>> -
>> - redirs = Redirections(self._redirs.stdin().dup(),
>> - stdout,
>> -
>> self._redirs.stderr().dup())
>> - try:
>> - status = self.subshell(script=script, redirs=redirs)
>> - finally:
>> - redirs.close()
>> - redirs = None
>> -
>> - # Extract subshell standard output
>> - tmpfile = open(tmppath, 'rb')
>> - try:
>> - output = tmpfile.read()
>> - return status, output.rstrip('\n')
>> - finally:
>> - tmpfile.close()
>> - finally:
>> - os.remove(tmppath)
>> -
>> - def _asynclist(self, cmd):
>> - args = (self._env.get_variables(), cmd)
>> - arg = encodeargs(args)
>> - assert len(args) < 30*1024
>> - cmd = ['pysh.bat', '--ast', '-c', arg]
>> - p = subprocess.Popen(cmd, cwd=self._env['PWD'])
>> - self._children[p.pid] = p
>> - self._env['!'] = p.pid
>> - return 0
>> -
>> - def wait(self, pids=None):
>> - if not pids:
>> - pids = self._children.keys()
>> -
>> - status = 127
>> - for pid in pids:
>> - if pid not in self._children:
>> - continue
>> - p = self._children.pop(pid)
>> - status = p.wait()
>> -
>> - return status
>> -
>> diff --git a/bitbake/lib/bb/pysh/lsprof.py
>> b/bitbake/lib/bb/pysh/lsprof.py deleted file mode 100644
>> index b1831c2..0000000
>> --- a/bitbake/lib/bb/pysh/lsprof.py
>> +++ /dev/null
>> @@ -1,116 +0,0 @@
>> -#! /usr/bin/env python
>> -
>> -import sys
>> -from _lsprof import Profiler, profiler_entry
>> -
>> -__all__ = ['profile', 'Stats']
>> -
>> -def profile(f, *args, **kwds):
>> - """XXX docstring"""
>> - p = Profiler()
>> - p.enable(subcalls=True, builtins=True)
>> - try:
>> - f(*args, **kwds)
>> - finally:
>> - p.disable()
>> - return Stats(p.getstats())
>> -
>> -
>> -class Stats(object):
>> - """XXX docstring"""
>> -
>> - def __init__(self, data):
>> - self.data = data
>> -
>> - def sort(self, crit="inlinetime"):
>> - """XXX docstring"""
>> - if crit not in profiler_entry.__dict__:
>> - raise ValueError("Can't sort by %s" % crit)
>> - self.data.sort(lambda b, a: cmp(getattr(a, crit),
>> - getattr(b, crit)))
>> - for e in self.data:
>> - if e.calls:
>> - e.calls.sort(lambda b, a: cmp(getattr(a, crit),
>> - getattr(b, crit)))
>> -
>> - def pprint(self, top=None, file=None, limit=None, climit=None):
>> - """XXX docstring"""
>> - if file is None:
>> - file = sys.stdout
>> - d = self.data
>> - if top is not None:
>> - d = d[:top]
>> - cols = "% 12s %12s %11.4f %11.4f %s\n"
>> - hcols = "% 12s %12s %12s %12s %s\n"
>> - cols2 = "+%12s %12s %11.4f %11.4f + %s\n"
>> - file.write(hcols % ("CallCount", "Recursive", "Total(ms)",
>> - "Inline(ms)", "module:lineno(function)"))
>> - count = 0
>> - for e in d:
>> - file.write(cols % (e.callcount, e.reccallcount,
>> e.totaltime,
>> - e.inlinetime, label(e.code)))
>> - count += 1
>> - if limit is not None and count == limit:
>> - return
>> - ccount = 0
>> - if e.calls:
>> - for se in e.calls:
>> - file.write(cols % ("+%s" % se.callcount,
>> se.reccallcount,
>> - se.totaltime, se.inlinetime,
>> - "+%s" % label(se.code)))
>> - count += 1
>> - ccount += 1
>> - if limit is not None and count == limit:
>> - return
>> - if climit is not None and ccount == climit:
>> - break
>> -
>> - def freeze(self):
>> - """Replace all references to code objects with string
>> - descriptions; this makes it possible to pickle the
>> instance.""" -
>> - # this code is probably rather ickier than it needs to be!
>> - for i in range(len(self.data)):
>> - e = self.data[i]
>> - if not isinstance(e.code, str):
>> - self.data[i] = type(e)((label(e.code),) + e[1:])
>> - if e.calls:
>> - for j in range(len(e.calls)):
>> - se = e.calls[j]
>> - if not isinstance(se.code, str):
>> - e.calls[j] = type(se)((label(se.code),) +
>> se[1:]) -
>> -_fn2mod = {}
>> -
>> -def label(code):
>> - if isinstance(code, str):
>> - return code
>> - try:
>> - mname = _fn2mod[code.co_filename]
>> - except KeyError:
>> - for k, v in sys.modules.items():
>> - if v is None:
>> - continue
>> - if not hasattr(v, '__file__'):
>> - continue
>> - if not isinstance(v.__file__, str):
>> - continue
>> - if v.__file__.startswith(code.co_filename):
>> - mname = _fn2mod[code.co_filename] = k
>> - break
>> - else:
>> - mname = _fn2mod[code.co_filename] =
>> '<%s>'%code.co_filename -
>> - return '%s:%d(%s)' % (mname, code.co_firstlineno, code.co_name)
>> -
>> -
>> -if __name__ == '__main__':
>> - import os
>> - sys.argv = sys.argv[1:]
>> - if not sys.argv:
>> - print >> sys.stderr, "usage: lsprof.py <script>
>> <arguments...>"
>> - sys.exit(2)
>> - sys.path.insert(0, os.path.abspath(os.path.dirname(sys.argv[0])))
>> - stats = profile(execfile, sys.argv[0], globals(), locals())
>> - stats.sort()
>> - stats.pprint()
>> diff --git a/bitbake/lib/bb/pysh/pysh.py b/bitbake/lib/bb/pysh/pysh.py
>> deleted file mode 100644
>> index b4e6145..0000000
>> --- a/bitbake/lib/bb/pysh/pysh.py
>> +++ /dev/null
>> @@ -1,167 +0,0 @@
>> -# pysh.py - command processing for pysh.
>> -#
>> -# Copyright 2007 Patrick Mezard
>> -#
>> -# This software may be used and distributed according to the terms
>> -# of the GNU General Public License, incorporated herein by
>> reference. -
>> -import optparse
>> -import os
>> -import sys
>> -
>> -import interp
>> -
>> -SH_OPT = optparse.OptionParser(prog='pysh', usage="%prog [OPTIONS]",
>> version='0.1') -SH_OPT.add_option('-c', action='store_true',
>> dest='command_string', default=None,
>> - help='A string that shall be interpreted by the shell as one or
>> more commands') -SH_OPT.add_option('--redirect-to',
>> dest='redirect_to', default=None,
>> - help='Redirect script commands stdout and stderr to the
>> specified file') -# See utility_command in builtin.py about the
>> reason for this flag. -SH_OPT.add_option('--redirected',
>> dest='redirected', action='store_true', default=False,
>> - help='Tell the interpreter that stdout and stderr are actually
>> the same objects, which is really stdout')
>> -SH_OPT.add_option('--debug-parsing', action='store_true',
>> dest='debug_parsing', default=False,
>> - help='Trace PLY execution')
>> -SH_OPT.add_option('--debug-tree', action='store_true',
>> dest='debug_tree', default=False,
>> - help='Display the generated syntax tree.')
>> -SH_OPT.add_option('--debug-cmd', action='store_true',
>> dest='debug_cmd', default=False,
>> - help='Trace command execution before parameters expansion and
>> exit status.') -SH_OPT.add_option('--debug-utility',
>> action='store_true', dest='debug_utility', default=False,
>> - help='Trace utility calls, after parameters expansions')
>> -SH_OPT.add_option('--ast', action='store_true', dest='ast',
>> default=False,
>> - help='Encoded commands to execute in a subprocess')
>> -SH_OPT.add_option('--profile', action='store_true', default=False,
>> - help='Profile pysh run')
>> -
>> -
>> -def split_args(args):
>> - # Separate shell arguments from command ones
>> - # Just stop at the first argument not starting with a dash. I
>> know, this is completely broken,
>> - # it ignores files starting with a dash or may take option
>> values for command file. This is not
>> - # supposed to happen for now
>> - command_index = len(args)
>> - for i,arg in enumerate(args):
>> - if not arg.startswith('-'):
>> - command_index = i
>> - break
>> -
>> - return args[:command_index], args[command_index:]
>> -
>> -
>> -def fixenv(env):
>> - path = env.get('PATH')
>> - if path is not None:
>> - parts = path.split(os.pathsep)
>> - # Remove Windows utilities from PATH, they are useless at
>> best and
>> - # some of them (find) may be confused with other utilities.
>> - parts = [p for p in parts if 'system32' not in p.lower()]
>> - env['PATH'] = os.pathsep.join(parts)
>> - if env.get('HOME') is None:
>> - # Several utilities, including cvsps, cannot work without
>> - # a defined HOME directory.
>> - env['HOME'] = os.path.expanduser('~')
>> - return env
>> -
>> -def _sh(cwd, shargs, cmdargs, options, debugflags=None, env=None):
>> - if os.environ.get('PYSH_TEXT') != '1':
>> - import msvcrt
>> - for fp in (sys.stdin, sys.stdout, sys.stderr):
>> - msvcrt.setmode(fp.fileno(), os.O_BINARY)
>> -
>> - hgbin = os.environ.get('PYSH_HGTEXT') != '1'
>> -
>> - if debugflags is None:
>> - debugflags = []
>> - if options.debug_parsing:
>> debugflags.append('debug-parsing')
>> - if options.debug_utility:
>> debugflags.append('debug-utility')
>> - if options.debug_cmd: debugflags.append('debug-cmd')
>> - if options.debug_tree: debugflags.append('debug-tree')
>> -
>> - if env is None:
>> - env = fixenv(dict(os.environ))
>> - if cwd is None:
>> - cwd = os.getcwd()
>> -
>> - if not cmdargs:
>> - # Nothing to do
>> - return 0
>> -
>> - ast = None
>> - command_file = None
>> - if options.command_string:
>> - input = cmdargs[0]
>> - if not options.ast:
>> - input += '\n'
>> - else:
>> - args, input = interp.decodeargs(input), None
>> - env, ast = args
>> - cwd = env.get('PWD', cwd)
>> - else:
>> - command_file = cmdargs[0]
>> - arguments = cmdargs[1:]
>> -
>> - prefix = interp.resolve_shebang(command_file,
>> ignoreshell=True)
>> - if prefix:
>> - input = ' '.join(prefix + [command_file] + arguments)
>> - else:
>> - # Read commands from file
>> - f = file(command_file)
>> - try:
>> - # Trailing newline to help the parser
>> - input = f.read() + '\n'
>> - finally:
>> - f.close()
>> -
>> - redirect = None
>> - try:
>> - if options.redirected:
>> - stdout = sys.stdout
>> - stderr = stdout
>> - elif options.redirect_to:
>> - redirect = open(options.redirect_to, 'wb')
>> - stdout = redirect
>> - stderr = redirect
>> - else:
>> - stdout = sys.stdout
>> - stderr = sys.stderr
>> -
>> - # TODO: set arguments to environment variables
>> - opts = interp.Options()
>> - opts.hgbinary = hgbin
>> - ip = interp.Interpreter(cwd, debugflags, stdout=stdout,
>> stderr=stderr,
>> - opts=opts)
>> - try:
>> - # Export given environment in shell object
>> - for k,v in env.iteritems():
>> - ip.get_env().export(k,v)
>> - return ip.execute_script(input, ast,
>> scriptpath=command_file)
>> - finally:
>> - ip.close()
>> - finally:
>> - if redirect is not None:
>> - redirect.close()
>> -
>> -def sh(cwd=None, args=None, debugflags=None, env=None):
>> - if args is None:
>> - args = sys.argv[1:]
>> - shargs, cmdargs = split_args(args)
>> - options, shargs = SH_OPT.parse_args(shargs)
>> -
>> - if options.profile:
>> - import lsprof
>> - p = lsprof.Profiler()
>> - p.enable(subcalls=True)
>> - try:
>> - return _sh(cwd, shargs, cmdargs, options, debugflags,
>> env)
>> - finally:
>> - p.disable()
>> - stats = lsprof.Stats(p.getstats())
>> - stats.sort()
>> - stats.pprint(top=10, file=sys.stderr, climit=5)
>> - else:
>> - return _sh(cwd, shargs, cmdargs, options, debugflags, env)
>> -
>> -def main():
>> - sys.exit(sh())
>> -
>> -if __name__=='__main__':
>> - main()
>> diff --git a/bitbake/lib/bb/pysh/pyshlex.py
>> b/bitbake/lib/bb/pysh/pyshlex.py index fbf094b..a42c294 100644
>> --- a/bitbake/lib/bb/pysh/pyshlex.py
>> +++ b/bitbake/lib/bb/pysh/pyshlex.py
>> @@ -13,11 +13,6 @@
>> # PLY in pull mode. It was designed to work incrementally and it
>> would not be # that hard to enable pull mode.
>> import re
>> -try:
>> - s = set()
>> - del s
>> -except NameError:
>> - from Set import Set as set
>>
>> from ply import lex
>> from bb.pysh.sherrors import *
>> diff --git a/bitbake/lib/bb/pysh/pyshyacc.py
>> b/bitbake/lib/bb/pysh/pyshyacc.py index ba4cefd..de565dc 100644
>> --- a/bitbake/lib/bb/pysh/pyshyacc.py
>> +++ b/bitbake/lib/bb/pysh/pyshyacc.py
>> @@ -636,13 +636,16 @@ def p_empty(p):
>> def p_error(p):
>> msg = []
>> w = msg.append
>> - w('%r\n' % p)
>> - w('followed by:\n')
>> - for i in range(5):
>> - n = yacc.token()
>> - if not n:
>> - break
>> - w(' %r\n' % n)
>> + if p:
>> + w('%r\n' % p)
>> + w('followed by:\n')
>> + for i in range(5):
>> + n = yacc.token()
>> + if not n:
>> + break
>> + w(' %r\n' % n)
>> + else:
>> + w('Unexpected EOF')
>> raise sherrors.ShellSyntaxError(''.join(msg))
>>
>> # Build the parser
>> diff --git a/bitbake/lib/bb/pysh/sherrors.py
>> b/bitbake/lib/bb/pysh/sherrors.py index 49d0533..3fe8e47 100644
>> --- a/bitbake/lib/bb/pysh/sherrors.py
>> +++ b/bitbake/lib/bb/pysh/sherrors.py
>> @@ -13,29 +13,3 @@ class ShellError(Exception):
>>
>> class ShellSyntaxError(ShellError):
>> pass
>> -
>> -class UtilityError(ShellError):
>> - """Raised upon utility syntax error (option or operand error)."""
>> - pass
>> -
>> -class ExpansionError(ShellError):
>> - pass
>> -
>> -class CommandNotFound(ShellError):
>> - """Specified command was not found."""
>> - pass
>> -
>> -class RedirectionError(ShellError):
>> - pass
>> -
>> -class VarAssignmentError(ShellError):
>> - """Variable assignment error."""
>> - pass
>> -
>> -class ExitSignal(ShellError):
>> - """Exit signal."""
>> - pass
>> -
>> -class ReturnSignal(ShellError):
>> - """Exit signal."""
>> - pass
>> diff --git a/bitbake/lib/bb/pysh/subprocess_fix.py
>> b/bitbake/lib/bb/pysh/subprocess_fix.py deleted file mode 100644
>> index 46eca22..0000000
>> --- a/bitbake/lib/bb/pysh/subprocess_fix.py
>> +++ /dev/null
>> @@ -1,77 +0,0 @@
>> -# subprocess - Subprocesses with accessible I/O streams
>> -#
>> -# For more information about this module, see PEP 324.
>> -#
>> -# This module should remain compatible with Python 2.2, see PEP 291.
>> -#
>> -# Copyright (c) 2003-2005 by Peter Astrand <astrand@lysator.liu.se>
>> -#
>> -# Licensed to PSF under a Contributor Agreement.
>> -# See http://www.python.org/2.4/license for licensing details.
>> -
>> -def list2cmdline(seq):
>> - """
>> - Translate a sequence of arguments into a command line
>> - string, using the same rules as the MS C runtime:
>> -
>> - 1) Arguments are delimited by white space, which is either a
>> - space or a tab.
>> -
>> - 2) A string surrounded by double quotation marks is
>> - interpreted as a single argument, regardless of white space
>> - contained within. A quoted string can be embedded in an
>> - argument.
>> -
>> - 3) A double quotation mark preceded by a backslash is
>> - interpreted as a literal double quotation mark.
>> -
>> - 4) Backslashes are interpreted literally, unless they
>> - immediately precede a double quotation mark.
>> -
>> - 5) If backslashes immediately precede a double quotation mark,
>> - every pair of backslashes is interpreted as a literal
>> - backslash. If the number of backslashes is odd, the last
>> - backslash escapes the next double quotation mark as
>> - described in rule 3.
>> - """
>> -
>> - # See
>> - #
>> http://msdn.microsoft.com/library/en-us/vccelng/htm/progs_12.asp
>> - result = []
>> - needquote = False
>> - for arg in seq:
>> - bs_buf = []
>> -
>> - # Add a space to separate this argument from the others
>> - if result:
>> - result.append(' ')
>> -
>> - needquote = (" " in arg) or ("\t" in arg) or ("|" in arg) or
>> arg == ""
>> - if needquote:
>> - result.append('"')
>> -
>> - for c in arg:
>> - if c == '\\':
>> - # Don't know if we need to double yet.
>> - bs_buf.append(c)
>> - elif c == '"':
>> - # Double backspaces.
>> - result.append('\\' * len(bs_buf)*2)
>> - bs_buf = []
>> - result.append('\\"')
>> - else:
>> - # Normal char
>> - if bs_buf:
>> - result.extend(bs_buf)
>> - bs_buf = []
>> - result.append(c)
>> -
>> - # Add remaining backspaces, if any.
>> - if bs_buf:
>> - result.extend(bs_buf)
>> -
>> - if needquote:
>> - result.extend(bs_buf)
>> - result.append('"')
>> -
>> - return ''.join(result)
>> diff --git a/bitbake/lib/bb/remotedata.py
>> b/bitbake/lib/bb/remotedata.py index 68ecffc..7391e1b 100644
>> --- a/bitbake/lib/bb/remotedata.py
>> +++ b/bitbake/lib/bb/remotedata.py
>> @@ -6,18 +6,8 @@ Provides support for using a datastore from the
>> bitbake client
>> # Copyright (C) 2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import bb.data
>>
>> diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
>> index 9ce06c4..1804943 100644
>> --- a/bitbake/lib/bb/runqueue.py
>> +++ b/bitbake/lib/bb/runqueue.py
>> @@ -1,6 +1,3 @@
>> -#!/usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake 'RunQueue' implementation
>>
>> @@ -9,18 +6,8 @@ Handles preparation and execution of a queue of tasks
>>
>> # Copyright (C) 2006-2007 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import copy
>> import os
>> @@ -37,11 +24,13 @@ from bb import monitordisk
>> import subprocess
>> import pickle
>> from multiprocessing import Process
>> +import shlex
>> +import pprint
>>
>> bblogger = logging.getLogger("BitBake")
>> logger = logging.getLogger("BitBake.RunQueue")
>>
>> -__find_md5__ =
>> re.compile( r'(?i)(?<![a-z0-9])[a-f0-9]{32}(?![a-z0-9])' )
>> +__find_sha256__ =
>> re.compile( r'(?i)(?<![a-z0-9])[a-f0-9]{64}(?![a-z0-9])' ) def
>> fn_from_tid(tid): return tid.rsplit(":", 1)[0]
>> @@ -49,17 +38,22 @@ def fn_from_tid(tid):
>> def taskname_from_tid(tid):
>> return tid.rsplit(":", 1)[1]
>>
>> +def mc_from_tid(tid):
>> + if tid.startswith('mc:'):
>> + return tid.split(':')[1]
>> + return ""
>> +
>> def split_tid(tid):
>> (mc, fn, taskname, _) = split_tid_mcfn(tid)
>> return (mc, fn, taskname)
>>
>> def split_tid_mcfn(tid):
>> - if tid.startswith('multiconfig:'):
>> + if tid.startswith('mc:'):
>> elems = tid.split(':')
>> mc = elems[1]
>> fn = ":".join(elems[2:-1])
>> taskname = elems[-1]
>> - mcfn = "multiconfig:" + mc + ":" + fn
>> + mcfn = "mc:" + mc + ":" + fn
>> else:
>> tid = tid.rsplit(":", 1)
>> mc = ""
>> @@ -71,9 +65,17 @@ def split_tid_mcfn(tid):
>>
>> def build_tid(mc, fn, taskname):
>> if mc:
>> - return "multiconfig:" + mc + ":" + fn + ":" + taskname
>> + return "mc:" + mc + ":" + fn + ":" + taskname
>> return fn + ":" + taskname
>>
>> +# Index used to pair up potentially matching multiconfig tasks
>> +# We match on PN, taskname and hash being equal
>> +def pending_hash_index(tid, rqdata):
>> + (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
>> + pn = rqdata.dataCaches[mc].pkg_fn[taskfn]
>> + h = rqdata.runtaskentries[tid].unihash
>> + return pn + ":" + "taskname" + h
>> +
>> class RunQueueStats:
>> """
>> Holds statistics on the tasks handled by the associated runQueue
>> @@ -109,8 +111,6 @@ class RunQueueStats:
>> # runQueue state machine
>> runQueuePrepare = 2
>> runQueueSceneInit = 3
>> -runQueueSceneRun = 4
>> -runQueueRunInit = 5
>> runQueueRunning = 6
>> runQueueFailed = 7
>> runQueueCleanUp = 8
>> @@ -133,7 +133,7 @@ class RunQueueScheduler(object):
>>
>> self.prio_map = [self.rqdata.runtaskentries.keys()]
>>
>> - self.buildable = []
>> + self.buildable = set()
>> self.skip_maxthread = {}
>> self.stamps = {}
>> for tid in self.rqdata.runtaskentries:
>> @@ -148,8 +148,11 @@ class RunQueueScheduler(object):
>> """
>> Return the id of the first task we find that is buildable
>> """
>> - self.buildable = [x for x in self.buildable if x not in
>> self.rq.runq_running]
>> - if not self.buildable:
>> + buildable = set(self.buildable)
>> + buildable.difference_update(self.rq.runq_running)
>> + buildable.difference_update(self.rq.holdoff_tasks)
>> + buildable.intersection_update(self.rq.tasks_covered |
>> self.rq.tasks_notcovered)
>> + if not buildable:
>> return None
>>
>> # Filter out tasks that have a max number of threads that
>> have been exceeded @@ -165,8 +168,8 @@ class
>> RunQueueScheduler(object): else:
>> skip_buildable[rtaskname] = 1
>>
>> - if len(self.buildable) == 1:
>> - tid = self.buildable[0]
>> + if len(buildable) == 1:
>> + tid = buildable.pop()
>> taskname = taskname_from_tid(tid)
>> if taskname in skip_buildable and
>> skip_buildable[taskname] >= int(self.skip_maxthread[taskname]):
>> return None @@ -181,7 +184,7 @@ class RunQueueScheduler(object):
>>
>> best = None
>> bestprio = None
>> - for tid in self.buildable:
>> + for tid in buildable:
>> taskname = taskname_from_tid(tid)
>> if taskname in skip_buildable and
>> skip_buildable[taskname] >= int(self.skip_maxthread[taskname]):
>> continue @@ -203,7 +206,12 @@ class RunQueueScheduler(object):
>> return self.next_buildable_task()
>>
>> def newbuildable(self, task):
>> - self.buildable.append(task)
>> + self.buildable.add(task)
>> + # Once tasks are running we don't need to worry about them
>> again
>> + self.buildable.difference_update(self.rq.runq_running)
>> +
>> + def removebuildable(self, task):
>> + self.buildable.remove(task)
>>
>> def describe_task(self, taskid):
>> result = 'ID %s' % taskid
>> @@ -346,6 +354,7 @@ class RunTaskEntry(object):
>> self.depends = set()
>> self.revdeps = set()
>> self.hash = None
>> + self.unihash = None
>> self.task = None
>> self.weight = 1
>>
>> @@ -385,6 +394,9 @@ class RunQueueData:
>> def get_task_hash(self, tid):
>> return self.runtaskentries[tid].hash
>>
>> + def get_task_unihash(self, tid):
>> + return self.runtaskentries[tid].unihash
>> +
>> def get_user_idstring(self, tid, task_name_suffix = ""):
>> return tid + task_name_suffix
>>
>> @@ -405,6 +417,9 @@ class RunQueueData:
>> explored_deps = {}
>> msgs = []
>>
>> + class TooManyLoops(Exception):
>> + pass
>> +
>> def chain_reorder(chain):
>> """
>> Reorder a dependency chain so the lowest task id is first
>> @@ -457,7 +472,7 @@ class RunQueueData:
>> msgs.append("\n")
>> if len(valid_chains) > 10:
>> msgs.append("Aborted dependency loops search
>> after 10 matches.\n")
>> - return msgs
>> + raise TooManyLoops
>> continue
>> scan = False
>> if revdep not in explored_deps:
>> @@ -476,8 +491,11 @@ class RunQueueData:
>>
>> explored_deps[tid] = total_deps
>>
>> - for task in tasks:
>> - find_chains(task, [])
>> + try:
>> + for task in tasks:
>> + find_chains(task, [])
>> + except TooManyLoops:
>> + pass
>>
>> return msgs
>>
>> @@ -833,6 +851,20 @@ class RunQueueData:
>> for depend in depends:
>> mark_active(depend, depth+1)
>>
>> + def invalidate_task(tid, error_nostamp):
>> + (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
>> + taskdep = self.dataCaches[mc].task_deps[taskfn]
>> + if fn + ":" + taskname not in taskData[mc].taskentries:
>> + logger.warning("Task %s does not exist, invalidating
>> this task will have no effect" % taskname)
>> + if 'nostamp' in taskdep and taskname in
>> taskdep['nostamp']:
>> + if error_nostamp:
>> + bb.fatal("Task %s is marked nostamp, cannot
>> invalidate this task" % taskname)
>> + else:
>> + bb.debug(1, "Task %s is marked nostamp, cannot
>> invalidate this task" % taskname)
>> + else:
>> + logger.verbose("Invalidate task %s, %s", taskname,
>> fn)
>> + bb.parse.siggen.invalidate_task(taskname,
>> self.dataCaches[mc], taskfn) +
>> self.target_tids = []
>> for (mc, target, task, fn) in self.targets:
>>
>> @@ -901,6 +933,8 @@ class RunQueueData:
>>
>> for tid in list(runall_tids):
>> mark_active(tid,1)
>> + if self.cooker.configuration.force:
>> + invalidate_task(tid, False)
>>
>> for tid in list(self.runtaskentries.keys()):
>> if tid not in runq_build:
>> @@ -922,6 +956,8 @@ class RunQueueData:
>>
>> for tid in list(runonly_tids):
>> mark_active(tid,1)
>> + if self.cooker.configuration.force:
>> + invalidate_task(tid, False)
>>
>> for tid in list(self.runtaskentries.keys()):
>> if tid not in runq_build:
>> @@ -1098,20 +1134,6 @@ class RunQueueData:
>> continue
>> self.runq_setscene_tids.append(tid)
>>
>> - def invalidate_task(tid, error_nostamp):
>> - (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
>> - taskdep = self.dataCaches[mc].task_deps[taskfn]
>> - if fn + ":" + taskname not in taskData[mc].taskentries:
>> - logger.warning("Task %s does not exist, invalidating
>> this task will have no effect" % taskname)
>> - if 'nostamp' in taskdep and taskname in
>> taskdep['nostamp']:
>> - if error_nostamp:
>> - bb.fatal("Task %s is marked nostamp, cannot
>> invalidate this task" % taskname)
>> - else:
>> - bb.debug(1, "Task %s is marked nostamp, cannot
>> invalidate this task" % taskname)
>> - else:
>> - logger.verbose("Invalidate task %s, %s", taskname,
>> fn)
>> - bb.parse.siggen.invalidate_task(taskname,
>> self.dataCaches[mc], taskfn) -
>> self.init_progress_reporter.next_stage()
>>
>> # Invalidate task if force mode active
>> @@ -1142,6 +1164,8 @@ class RunQueueData:
>>
>> self.init_progress_reporter.next_stage()
>>
>> + bb.parse.siggen.set_setscene_tasks(self.runq_setscene_tids)
>> +
>> # Iterate over the task list and call into the siggen code
>> dealtwith = set()
>> todeal = set(self.runtaskentries)
>> @@ -1150,18 +1174,20 @@ class RunQueueData:
>> if len(self.runtaskentries[tid].depends - dealtwith)
>> == 0: dealtwith.add(tid)
>> todeal.remove(tid)
>> - procdep = []
>> - for dep in self.runtaskentries[tid].depends:
>> - procdep.append(fn_from_tid(dep) + "." +
>> taskname_from_tid(dep))
>> - (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
>> - self.runtaskentries[tid].hash =
>> bb.parse.siggen.get_taskhash(taskfn, taskname, procdep,
>> self.dataCaches[mc])
>> - task = self.runtaskentries[tid].task
>> + self.prepare_task_hash(tid)
>>
>> bb.parse.siggen.writeout_file_checksum_cache()
>>
>> #self.dump_data()
>> return len(self.runtaskentries)
>>
>> + def prepare_task_hash(self, tid):
>> + procdep = []
>> + for dep in self.runtaskentries[tid].depends:
>> + procdep.append(dep)
>> + self.runtaskentries[tid].hash =
>> bb.parse.siggen.get_taskhash(tid, procdep,
>> self.dataCaches[mc_from_tid(tid)])
>> + self.runtaskentries[tid].unihash =
>> bb.parse.siggen.get_unihash(tid) +
>> def dump_data(self):
>> """
>> Dump some debug information on the internal data structures
>> @@ -1187,7 +1213,6 @@ class RunQueue:
>>
>> self.stamppolicy = cfgData.getVar("BB_STAMP_POLICY") or
>> "perfile" self.hashvalidate = cfgData.getVar("BB_HASHCHECK_FUNCTION")
>> or None
>> - self.setsceneverify =
>> cfgData.getVar("BB_SETSCENE_VERIFY_FUNCTION2") or None
>> self.depvalidate = cfgData.getVar("BB_SETSCENE_DEPVALID") or None
>> self.state = runQueuePrepare
>> @@ -1196,7 +1221,7 @@ class RunQueue:
>> # Invoked at regular time intervals via the bitbake
>> heartbeat event # while the build is running. We generate a unique
>> name for the handler # here, just in case that there ever is more
>> than one RunQueue instance,
>> - # start the handler when reaching runQueueSceneRun, and stop
>> it when
>> + # start the handler when reaching runQueueSceneInit, and
>> stop it when # done with the build.
>> self.dm = monitordisk.diskMonitor(cfgData)
>> self.dm_event_handler_name = '_bb_diskmonitor_' +
>> str(id(self)) @@ -1213,28 +1238,23 @@ class RunQueue:
>> if fakeroot:
>> magic = magic + "beef"
>> mcdata = self.cooker.databuilder.mcdata[mc]
>> - fakerootcmd = mcdata.getVar("FAKEROOTCMD")
>> + fakerootcmd = shlex.split(mcdata.getVar("FAKEROOTCMD"))
>> fakerootenv = (mcdata.getVar("FAKEROOTBASEENV") or
>> "").split() env = os.environ.copy()
>> for key, value in (var.split('=') for var in
>> fakerootenv): env[key] = value
>> - worker = subprocess.Popen([fakerootcmd,
>> "bitbake-worker", magic], stdout=subprocess.PIPE,
>> stdin=subprocess.PIPE, env=env)
>> + worker = subprocess.Popen(fakerootcmd +
>> ["bitbake-worker", magic], stdout=subprocess.PIPE,
>> stdin=subprocess.PIPE, env=env) else: worker =
>> subprocess.Popen(["bitbake-worker", magic], stdout=subprocess.PIPE,
>> stdin=subprocess.PIPE) bb.utils.nonblockingfd(worker.stdout)
>> workerpipe = runQueuePipe(worker.stdout, None, self.cfgData, self,
>> rqexec)
>> - runqhash = {}
>> - for tid in self.rqdata.runtaskentries:
>> - runqhash[tid] = self.rqdata.runtaskentries[tid].hash
>> -
>> workerdata = {
>> "taskdeps" : self.rqdata.dataCaches[mc].task_deps,
>> "fakerootenv" : self.rqdata.dataCaches[mc].fakerootenv,
>> "fakerootdirs" : self.rqdata.dataCaches[mc].fakerootdirs,
>> "fakerootnoenv" :
>> self.rqdata.dataCaches[mc].fakerootnoenv, "sigdata" :
>> bb.parse.siggen.get_taskdata(),
>> - "runq_hash" : runqhash,
>> "logdefaultdebug" : bb.msg.loggerDefaultDebugLevel,
>> "logdefaultverbose" : bb.msg.loggerDefaultVerbose,
>> "logdefaultverboselogs" : bb.msg.loggerVerboseLogs,
>> @@ -1243,6 +1263,7 @@ class RunQueue:
>> "buildname" : self.cfgData.getVar("BUILDNAME"),
>> "date" : self.cfgData.getVar("DATE"),
>> "time" : self.cfgData.getVar("TIME"),
>> + "hashservaddr" : self.cooker.hashservaddr,
>> }
>>
>> worker.stdin.write(b"<cookerconfig>" +
>> pickle.dumps(self.cooker.configuration) + b"</cookerconfig>") @@
>> -1376,6 +1397,31 @@ class RunQueue: cache[tid] = iscurrent
>> return iscurrent
>>
>> + def validate_hashes(self, tocheck, data, currentcount=0,
>> siginfo=False):
>> + valid = set()
>> + if self.hashvalidate:
>> + sq_data = {}
>> + sq_data['hash'] = {}
>> + sq_data['hashfn'] = {}
>> + sq_data['unihash'] = {}
>> + for tid in tocheck:
>> + (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
>> + sq_data['hash'][tid] =
>> self.rqdata.runtaskentries[tid].hash
>> + sq_data['hashfn'][tid] =
>> self.rqdata.dataCaches[mc].hashfn[taskfn]
>> + sq_data['unihash'][tid] =
>> self.rqdata.runtaskentries[tid].unihash +
>> + valid = self.validate_hash(sq_data, data, siginfo,
>> currentcount) +
>> + return valid
>> +
>> + def validate_hash(self, sq_data, d, siginfo, currentcount):
>> + locs = {"sq_data" : sq_data, "d" : d, "siginfo" : siginfo,
>> "currentcount" : currentcount} +
>> + # Metadata has **kwargs so args can be added, sq_data can
>> also gain new fields
>> + call = self.hashvalidate + "(sq_data, d, siginfo=siginfo,
>> currentcount=currentcount)" +
>> + return bb.utils.better_eval(call, locs)
>> +
>> def _execute_runqueue(self):
>> """
>> Run the tasks in a queue prepared by rqdata.prepare()
>> @@ -1386,7 +1432,6 @@ class RunQueue:
>> retval = True
>>
>> if self.state is runQueuePrepare:
>> - self.rqexe = RunQueueExecuteDummy(self)
>> # NOTE: if you add, remove or significantly refactor the
>> stages of this # process then you should recalculate the weightings
>> here. This is quite # easy to do - just change the next line
>> temporarily to pass debug=True as @@ -1400,15 +1445,23 @@ class
>> RunQueue: self.state = runQueueComplete
>> else:
>> self.state = runQueueSceneInit
>> - self.rqdata.init_progress_reporter.next_stage()
>> -
>> - # we are ready to run, emit dependency info to any
>> UI or class which
>> - # needs it
>> - depgraph = self.cooker.buildDependTree(self,
>> self.rqdata.taskData)
>> - self.rqdata.init_progress_reporter.next_stage()
>> - bb.event.fire(bb.event.DepTreeGenerated(depgraph),
>> self.cooker.data)
>> + bb.parse.siggen.save_unitaskhashes()
>>
>> if self.state is runQueueSceneInit:
>> + self.rqdata.init_progress_reporter.next_stage()
>> +
>> + # we are ready to run, emit dependency info to any UI
>> or class which
>> + # needs it
>> + depgraph = self.cooker.buildDependTree(self,
>> self.rqdata.taskData)
>> + self.rqdata.init_progress_reporter.next_stage()
>> + bb.event.fire(bb.event.DepTreeGenerated(depgraph),
>> self.cooker.data) +
>> + if not self.dm_event_handler_registered:
>> + res = bb.event.register(self.dm_event_handler_name,
>> + lambda x:
>> self.dm.check(self) if self.state in [runQueueRunning,
>> runQueueCleanUp] else False,
>> +
>> ('bb.event.HeartbeatEvent',))
>> + self.dm_event_handler_registered = True
>> +
>> dump = self.cooker.configuration.dump_signatures
>> if dump:
>> self.rqdata.init_progress_reporter.finish()
>> @@ -1418,29 +1471,23 @@ class RunQueue:
>> if 'printdiff' in dump:
>> self.write_diffscenetasks(invalidtasks)
>> self.state = runQueueComplete
>> - else:
>> - self.rqdata.init_progress_reporter.next_stage()
>> - self.start_worker()
>> - self.rqdata.init_progress_reporter.next_stage()
>> - self.rqexe = RunQueueExecuteScenequeue(self)
>> -
>> - if self.state is runQueueSceneRun:
>> - if not self.dm_event_handler_registered:
>> - res = bb.event.register(self.dm_event_handler_name,
>> - lambda x:
>> self.dm.check(self) if self.state in [runQueueSceneRun,
>> runQueueRunning, runQueueCleanUp] else False,
>> -
>> ('bb.event.HeartbeatEvent',))
>> - self.dm_event_handler_registered = True
>> - retval = self.rqexe.execute()
>>
>> - if self.state is runQueueRunInit:
>> - if self.cooker.configuration.setsceneonly:
>> - self.state = runQueueComplete
>> - else:
>> - # Just in case we didn't setscene
>> - self.rqdata.init_progress_reporter.finish()
>> - logger.info("Executing RunQueue Tasks")
>> - self.rqexe = RunQueueExecuteTasks(self)
>> - self.state = runQueueRunning
>> + if self.state is runQueueSceneInit:
>> + self.rqdata.init_progress_reporter.next_stage()
>> + self.start_worker()
>> + self.rqdata.init_progress_reporter.next_stage()
>> + self.rqexe = RunQueueExecute(self)
>> +
>> + # If we don't have any setscene functions, skip execution
>> + if len(self.rqdata.runq_setscene_tids) == 0:
>> + logger.info('No setscene tasks')
>> + for tid in self.rqdata.runtaskentries:
>> + if len(self.rqdata.runtaskentries[tid].depends)
>> == 0:
>> + self.rqexe.setbuildable(tid)
>> + self.rqexe.tasks_notcovered.add(tid)
>> + self.rqexe.sqdone = True
>> + logger.info('Executing Tasks')
>> + self.state = runQueueRunning
>>
>> if self.state is runQueueRunning:
>> retval = self.rqexe.execute()
>> @@ -1455,12 +1502,14 @@ class RunQueue:
>> self.dm_event_handler_registered = False
>>
>> if build_done and self.rqexe:
>> + bb.parse.siggen.save_unitaskhashes()
>> self.teardown_workers()
>> - if self.rqexe.stats.failed:
>> - logger.info("Tasks Summary: Attempted %d tasks of
>> which %d didn't need to be rerun and %d failed.",
>> self.rqexe.stats.completed + self.rqexe.stats.failed,
>> self.rqexe.stats.skipped, self.rqexe.stats.failed)
>> - else:
>> - # Let's avoid the word "failed" if nothing actually
>> did
>> - logger.info("Tasks Summary: Attempted %d tasks of
>> which %d didn't need to be rerun and all succeeded.",
>> self.rqexe.stats.completed, self.rqexe.stats.skipped)
>> + if self.rqexe:
>> + if self.rqexe.stats.failed:
>> + logger.info("Tasks Summary: Attempted %d tasks
>> of which %d didn't need to be rerun and %d failed.",
>> self.rqexe.stats.completed + self.rqexe.stats.failed,
>> self.rqexe.stats.skipped, self.rqexe.stats.failed)
>> + else:
>> + # Let's avoid the word "failed" if nothing
>> actually did
>> + logger.info("Tasks Summary: Attempted %d tasks
>> of which %d didn't need to be rerun and all succeeded.",
>> self.rqexe.stats.completed, self.rqexe.stats.skipped) if self.state
>> is runQueueFailed: raise
>> bb.runqueue.TaskFailure(self.rqexe.failed_tids) @@ -1543,15 +1592,8
>> @@ class RunQueue:
>> def print_diffscenetasks(self):
>>
>> - valid = []
>> - sq_hash = []
>> - sq_hashfn = []
>> - sq_fn = []
>> - sq_taskname = []
>> - sq_task = []
>> noexec = []
>> - stamppresent = []
>> - valid_new = set()
>> + tocheck = set()
>>
>> for tid in self.rqdata.runtaskentries:
>> (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
>> @@ -1561,21 +1603,9 @@ class RunQueue:
>> noexec.append(tid)
>> continue
>>
>> - sq_fn.append(fn)
>> -
>> sq_hashfn.append(self.rqdata.dataCaches[mc].hashfn[taskfn])
>> - sq_hash.append(self.rqdata.runtaskentries[tid].hash)
>> - sq_taskname.append(taskname)
>> - sq_task.append(tid)
>> - locs = { "sq_fn" : sq_fn, "sq_task" : sq_taskname,
>> "sq_hash" : sq_hash, "sq_hashfn" : sq_hashfn, "d" : self.cooker.data }
>> - try:
>> - call = self.hashvalidate + "(sq_fn, sq_task, sq_hash,
>> sq_hashfn, d, siginfo=True)"
>> - valid = bb.utils.better_eval(call, locs)
>> - # Handle version with no siginfo parameter
>> - except TypeError:
>> - call = self.hashvalidate + "(sq_fn, sq_task, sq_hash,
>> sq_hashfn, d)"
>> - valid = bb.utils.better_eval(call, locs)
>> - for v in valid:
>> - valid_new.add(sq_task[v])
>> + tocheck.add(tid)
>> +
>> + valid_new = self.validate_hashes(tocheck, self.cooker.data,
>> 0, True)
>> # Tasks which are both setscene and noexec never care about
>> dependencies # We therefore find tasks which are setscene and noexec
>> and mark their @@ -1634,7 +1664,7 @@ class RunQueue:
>> recout = []
>> if len(hashfiles) == 2:
>> out2 = bb.siggen.compare_sigfiles(hashfiles[hash1],
>> hashfiles[hash2], recursecb)
>> - recout.extend(list(' ' + l for l in out2))
>> + recout.extend(list(' ' + l for l in out2))
>> else:
>> recout.append("Unable to find matching sigdata for
>> %s with hashes %s or %s" % (key, hash1, hash2))
>> @@ -1655,10 +1685,11 @@ class RunQueue:
>> matches = {k : v for k, v in iter(matches.items()) if h
>> not in k} if matches:
>> latestmatch = sorted(matches.keys(), key=lambda f:
>> matches[f])[-1]
>> - prevh = __find_md5__.search(latestmatch).group(0)
>> + prevh = __find_sha256__.search(latestmatch).group(0)
>> output = bb.siggen.compare_sigfiles(latestmatch,
>> match, recursecb) bb.plain("\nTask %s:%s couldn't be used from the
>> cache because:\n We need hash %s, closest matching task was %s\n "
>> % (pn, taskname, h, prevh) + '\n '.join(output)) +
>> class RunQueueExecute:
>>
>> def __init__(self, rq):
>> @@ -1670,6 +1701,13 @@ class RunQueueExecute:
>> self.number_tasks =
>> int(self.cfgData.getVar("BB_NUMBER_THREADS") or 1) self.scheduler =
>> self.cfgData.getVar("BB_SCHEDULER") or "speed"
>> + self.sq_buildable = set()
>> + self.sq_running = set()
>> + self.sq_live = set()
>> +
>> + self.updated_taskhash_queue = []
>> + self.pending_migrations = set()
>> +
>> self.runq_buildable = set()
>> self.runq_running = set()
>> self.runq_complete = set()
>> @@ -1677,9 +1715,17 @@ class RunQueueExecute:
>> self.build_stamps = {}
>> self.build_stamps2 = []
>> self.failed_tids = []
>> + self.sq_deferred = {}
>>
>> self.stampcache = {}
>>
>> + self.holdoff_tasks = set()
>> + self.holdoff_need_update = True
>> + self.sqdone = False
>> +
>> + self.stats = RunQueueStats(len(self.rqdata.runtaskentries))
>> + self.sq_stats =
>> RunQueueStats(len(self.rqdata.runq_setscene_tids)) +
>> for mc in rq.worker:
>> rq.worker[mc].pipe.setrunqueueexec(self)
>> for mc in rq.fakeworker:
>> @@ -1688,6 +1734,34 @@ class RunQueueExecute:
>> if self.number_tasks <= 0:
>> bb.fatal("Invalid BB_NUMBER_THREADS %s" %
>> self.number_tasks)
>> + # List of setscene tasks which we've covered
>> + self.scenequeue_covered = set()
>> + # List of tasks which are covered (including setscene ones)
>> + self.tasks_covered = set()
>> + self.tasks_scenequeue_done = set()
>> + self.scenequeue_notcovered = set()
>> + self.tasks_notcovered = set()
>> + self.scenequeue_notneeded = set()
>> +
>> + # We can't skip specified target tasks which aren't setscene
>> tasks
>> + self.cantskip = set(self.rqdata.target_tids)
>> +
>> self.cantskip.difference_update(self.rqdata.runq_setscene_tids)
>> + self.cantskip.intersection_update(self.rqdata.runtaskentries)
>> +
>> + schedulers = self.get_schedulers()
>> + for scheduler in schedulers:
>> + if self.scheduler == scheduler.name:
>> + self.sched = scheduler(self, self.rqdata)
>> + logger.debug(1, "Using runqueue scheduler '%s'",
>> scheduler.name)
>> + break
>> + else:
>> + bb.fatal("Invalid scheduler '%s'. Available schedulers:
>> %s" %
>> + (self.scheduler, ", ".join(obj.name for obj in
>> schedulers))) +
>> + #if len(self.rqdata.runq_setscene_tids) > 0:
>> + self.sqdata = SQData()
>> + build_scenequeue_data(self.sqdata, self.rqdata, self.rq,
>> self.cooker, self.stampcache, self) +
>> def runqueue_process_waitpid(self, task, status):
>>
>> # self.build_stamps[pid] may not exist when use shared work
>> directory. @@ -1695,10 +1769,17 @@ class RunQueueExecute:
>> self.build_stamps2.remove(self.build_stamps[task])
>> del self.build_stamps[task]
>>
>> - if status != 0:
>> - self.task_fail(task, status)
>> + if task in self.sq_live:
>> + if status != 0:
>> + self.sq_task_fail(task, status)
>> + else:
>> + self.sq_task_complete(task)
>> + self.sq_live.remove(task)
>> else:
>> - self.task_complete(task)
>> + if status != 0:
>> + self.task_fail(task, status)
>> + else:
>> + self.task_complete(task)
>> return True
>>
>> def finish_now(self):
>> @@ -1727,8 +1808,9 @@ class RunQueueExecute:
>> def finish(self):
>> self.rq.state = runQueueCleanUp
>>
>> - if self.stats.active > 0:
>> - bb.event.fire(runQueueExitWait(self.stats.active),
>> self.cfgData)
>> + active = self.stats.active + self.sq_stats.active
>> + if active > 0:
>> + bb.event.fire(runQueueExitWait(active), self.cfgData)
>> self.rq.read_workers()
>> return self.rq.active_fds()
>>
>> @@ -1739,10 +1821,14 @@ class RunQueueExecute:
>> self.rq.state = runQueueComplete
>> return True
>>
>> - def check_dependencies(self, task, taskdeps, setscene = False):
>> + # Used by setscene only
>> + def check_dependencies(self, task, taskdeps):
>> if not self.rq.depvalidate:
>> return False
>>
>> + # Must not edit parent data
>> + taskdeps = set(taskdeps)
>> +
>> taskdata = {}
>> taskdeps.add(task)
>> for dep in taskdeps:
>> @@ -1755,121 +1841,10 @@ class RunQueueExecute:
>> return valid
>>
>> def can_start_task(self):
>> - can_start = self.stats.active < self.number_tasks
>> + active = self.stats.active + self.sq_stats.active
>> + can_start = active < self.number_tasks
>> return can_start
>>
>> -class RunQueueExecuteDummy(RunQueueExecute):
>> - def __init__(self, rq):
>> - self.rq = rq
>> - self.stats = RunQueueStats(0)
>> -
>> - def finish(self):
>> - self.rq.state = runQueueComplete
>> - return
>> -
>> -class RunQueueExecuteTasks(RunQueueExecute):
>> - def __init__(self, rq):
>> - RunQueueExecute.__init__(self, rq)
>> -
>> - self.stats = RunQueueStats(len(self.rqdata.runtaskentries))
>> -
>> - self.stampcache = {}
>> -
>> - initial_covered = self.rq.scenequeue_covered.copy()
>> -
>> - # Mark initial buildable tasks
>> - for tid in self.rqdata.runtaskentries:
>> - if len(self.rqdata.runtaskentries[tid].depends) == 0:
>> - self.runq_buildable.add(tid)
>> - if len(self.rqdata.runtaskentries[tid].revdeps) > 0 and
>> self.rqdata.runtaskentries[tid].revdeps.issubset(self.rq.scenequeue_covered):
>> - self.rq.scenequeue_covered.add(tid)
>> -
>> - found = True
>> - while found:
>> - found = False
>> - for tid in self.rqdata.runtaskentries:
>> - if tid in self.rq.scenequeue_covered:
>> - continue
>> - logger.debug(1, 'Considering %s: %s' % (tid,
>> str(self.rqdata.runtaskentries[tid].revdeps))) -
>> - if len(self.rqdata.runtaskentries[tid].revdeps) > 0
>> and
>> self.rqdata.runtaskentries[tid].revdeps.issubset(self.rq.scenequeue_covered):
>> - if tid in self.rq.scenequeue_notcovered:
>> - continue
>> - found = True
>> - self.rq.scenequeue_covered.add(tid)
>> -
>> - logger.debug(1, 'Skip list (pre setsceneverify) %s',
>> sorted(self.rq.scenequeue_covered)) -
>> - # Allow the metadata to elect for setscene tasks to run
>> anyway
>> - covered_remove = set()
>> - if self.rq.setsceneverify:
>> - invalidtasks = []
>> - tasknames = {}
>> - fns = {}
>> - for tid in self.rqdata.runtaskentries:
>> - (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
>> - taskdep =
>> self.rqdata.dataCaches[mc].task_deps[taskfn]
>> - fns[tid] = taskfn
>> - tasknames[tid] = taskname
>> - if 'noexec' in taskdep and taskname in
>> taskdep['noexec']:
>> - continue
>> - if self.rq.check_stamp_task(tid, taskname +
>> "_setscene", cache=self.stampcache):
>> - logger.debug(2, 'Setscene stamp current for task
>> %s', tid)
>> - continue
>> - if self.rq.check_stamp_task(tid, taskname, recurse =
>> True, cache=self.stampcache):
>> - logger.debug(2, 'Normal stamp current for task
>> %s', tid)
>> - continue
>> - invalidtasks.append(tid)
>> -
>> - call = self.rq.setsceneverify + "(covered, tasknames,
>> fns, d, invalidtasks=invalidtasks)"
>> - locs = { "covered" : self.rq.scenequeue_covered,
>> "tasknames" : tasknames, "fns" : fns, "d" : self.cooker.data,
>> "invalidtasks" : invalidtasks }
>> - covered_remove = bb.utils.better_eval(call, locs)
>> -
>> - def removecoveredtask(tid):
>> - (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
>> - taskname = taskname + '_setscene'
>> - bb.build.del_stamp(taskname, self.rqdata.dataCaches[mc],
>> taskfn)
>> - self.rq.scenequeue_covered.remove(tid)
>> -
>> - toremove = covered_remove | self.rq.scenequeue_notcovered
>> - for task in toremove:
>> - logger.debug(1, 'Not skipping task %s due to
>> setsceneverify', task)
>> - while toremove:
>> - covered_remove = []
>> - for task in toremove:
>> - if task in self.rq.scenequeue_covered:
>> - removecoveredtask(task)
>> - for deptask in
>> self.rqdata.runtaskentries[task].depends:
>> - if deptask not in self.rq.scenequeue_covered:
>> - continue
>> - if deptask in toremove or deptask in
>> covered_remove or deptask in initial_covered:
>> - continue
>> - logger.debug(1, 'Task %s depends on task %s so
>> not skipping' % (task, deptask))
>> - covered_remove.append(deptask)
>> - toremove = covered_remove
>> -
>> - logger.debug(1, 'Full skip list %s',
>> self.rq.scenequeue_covered) -
>> -
>> - for mc in self.rqdata.dataCaches:
>> - target_pairs = []
>> - for tid in self.rqdata.target_tids:
>> - (tidmc, fn, taskname, _) = split_tid_mcfn(tid)
>> - if tidmc == mc:
>> - target_pairs.append((fn, taskname))
>> -
>> - event.fire(bb.event.StampUpdate(target_pairs,
>> self.rqdata.dataCaches[mc].stamp), self.cfgData) -
>> - schedulers = self.get_schedulers()
>> - for scheduler in schedulers:
>> - if self.scheduler == scheduler.name:
>> - self.sched = scheduler(self, self.rqdata)
>> - logger.debug(1, "Using runqueue scheduler '%s'",
>> scheduler.name)
>> - break
>> - else:
>> - bb.fatal("Invalid scheduler '%s'. Available schedulers:
>> %s" %
>> - (self.scheduler, ", ".join(obj.name for obj in
>> schedulers))) -
>> def get_schedulers(self):
>> schedulers = set(obj for obj in globals().values()
>> if type(obj) is type and
>> @@ -1941,67 +1916,172 @@ class RunQueueExecuteTasks(RunQueueExecute):
>> self.stats.taskSkipped()
>> self.stats.taskCompleted()
>>
>> + def summarise_scenequeue_errors(self):
>> + err = False
>> + if not self.sqdone:
>> + logger.debug(1, 'We could skip tasks %s',
>> "\n".join(sorted(self.scenequeue_covered)))
>> + completeevent = sceneQueueComplete(self.sq_stats,
>> self.rq)
>> + bb.event.fire(completeevent, self.cfgData)
>> + if self.sq_deferred:
>> + logger.error("Scenequeue had deferred entries: %s" %
>> pprint.pformat(self.sq_deferred))
>> + err = True
>> + if self.updated_taskhash_queue:
>> + logger.error("Scenequeue had unprocessed changed
>> taskhash entries: %s" % pprint.pformat(self.updated_taskhash_queue))
>> + err = True
>> + if self.holdoff_tasks:
>> + logger.error("Scenequeue had holdoff tasks: %s" %
>> pprint.pformat(self.holdoff_tasks))
>> + err = True
>> +
>> + for tid in self.rqdata.runq_setscene_tids:
>> + if tid not in self.scenequeue_covered and tid not in
>> self.scenequeue_notcovered:
>> + err = True
>> + logger.error("Setscene Task %s was never marked as
>> covered or not covered" % tid)
>> + if tid not in self.sq_buildable:
>> + err = True
>> + logger.error("Setscene Task %s was never marked as
>> buildable" % tid)
>> + if tid not in self.sq_running:
>> + err = True
>> + logger.error("Setscene Task %s was never marked as
>> running" % tid) +
>> + for x in self.rqdata.runtaskentries:
>> + if x not in self.tasks_covered and x not in
>> self.tasks_notcovered:
>> + logger.error("Task %s was never moved from the
>> setscene queue" % x)
>> + err = True
>> + if x not in self.tasks_scenequeue_done:
>> + logger.error("Task %s was never processed by the
>> setscene code" % x)
>> + err = True
>> + if len(self.rqdata.runtaskentries[x].depends) == 0 and x
>> not in self.runq_buildable:
>> + logger.error("Task %s was never marked as buildable
>> by the setscene code" % x)
>> + err = True
>> + return err
>> +
>> +
>> def execute(self):
>> """
>> - Run the tasks in a queue prepared by rqdata.prepare()
>> + Run the tasks in a queue prepared by prepare_runqueue
>> """
>>
>> - if self.rqdata.setscenewhitelist is not None and not
>> self.rqdata.setscenewhitelist_checked:
>> - self.rqdata.setscenewhitelist_checked = True
>> -
>> - # Check tasks that are going to run against the whitelist
>> - def check_norun_task(tid, showerror=False):
>> - (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
>> - # Ignore covered tasks
>> - if tid in self.rq.scenequeue_covered:
>> - return False
>> - # Ignore stamped tasks
>> - if self.rq.check_stamp_task(tid, taskname,
>> cache=self.stampcache):
>> - return False
>> - # Ignore noexec tasks
>> - taskdep =
>> self.rqdata.dataCaches[mc].task_deps[taskfn]
>> - if 'noexec' in taskdep and taskname in
>> taskdep['noexec']:
>> - return False
>> + self.rq.read_workers()
>> + self.process_possible_migrations()
>>
>> - pn = self.rqdata.dataCaches[mc].pkg_fn[taskfn]
>> - if not check_setscene_enforce_whitelist(pn,
>> taskname, self.rqdata.setscenewhitelist):
>> - if showerror:
>> - if tid in self.rqdata.runq_setscene_tids:
>> - logger.error('Task %s.%s attempted to
>> execute unexpectedly and should have been setscened' % (pn, taskname))
>> + task = None
>> + if not self.sqdone and self.can_start_task():
>> + # Find the next setscene to run
>> + for nexttask in sorted(self.rqdata.runq_setscene_tids):
>> + if nexttask in self.sq_buildable and nexttask not in
>> self.sq_running and self.sqdata.stamps[nexttask] not in
>> self.build_stamps.values():
>> + if nexttask not in self.sqdata.unskippable and
>> len(self.sqdata.sq_revdeps[nexttask]) > 0 and
>> self.sqdata.sq_revdeps[nexttask].issubset(self.scenequeue_covered)
>> and self.check_dependencies(nexttask,
>> self.sqdata.sq_revdeps[nexttask]):
>> + if nexttask not in self.rqdata.target_tids:
>> + logger.debug(2, "Skipping setscene for
>> task %s" % nexttask)
>> + self.sq_task_skip(nexttask)
>> + self.scenequeue_notneeded.add(nexttask)
>> + if nexttask in self.sq_deferred:
>> + del self.sq_deferred[nexttask]
>> + return True
>> + # If covered tasks are running, need to wait for
>> them to complete
>> + for t in self.sqdata.sq_covered_tasks[nexttask]:
>> + if t in self.runq_running and t not in
>> self.runq_complete:
>> + continue
>> + if nexttask in self.sq_deferred:
>> + if self.sq_deferred[nexttask] not in
>> self.runq_complete:
>> + continue
>> + logger.debug(1, "Task %s no longer deferred"
>> % nexttask)
>> + del self.sq_deferred[nexttask]
>> + valid =
>> self.rq.validate_hashes(set([nexttask]), self.cooker.data, 0, False)
>> + if not valid:
>> + logger.debug(1, "%s didn't become valid,
>> skipping setscene" % nexttask)
>> + self.sq_task_failoutright(nexttask)
>> + return True
>> else:
>> - logger.error('Task %s.%s attempted to
>> execute unexpectedly' % (pn, taskname))
>> - return True
>> - return False
>> - # Look to see if any tasks that we think shouldn't run
>> are going to
>> - unexpected = False
>> - for tid in self.rqdata.runtaskentries:
>> - if check_norun_task(tid):
>> - unexpected = True
>> + self.sqdata.outrightfail.remove(nexttask)
>> + if nexttask in self.sqdata.outrightfail:
>> + logger.debug(2, 'No package found, so
>> skipping setscene task %s', nexttask)
>> + self.sq_task_failoutright(nexttask)
>> + return True
>> + if nexttask in self.sqdata.unskippable:
>> + logger.debug(2, "Setscene task %s is
>> unskippable" % nexttask)
>> + task = nexttask
>> break
>> - if unexpected:
>> - # Run through the tasks in the rough order they'd
>> have executed and print errors
>> - # (since the order can be useful - usually missing
>> sstate for the last few tasks
>> - # is the cause of the problem)
>> - task = self.sched.next()
>> - while task is not None:
>> - check_norun_task(task, showerror=True)
>> - self.task_skip(task, 'Setscene enforcement
>> check')
>> - task = self.sched.next()
>> + if task is not None:
>> + (mc, fn, taskname, taskfn) = split_tid_mcfn(task)
>> + taskname = taskname + "_setscene"
>> + if self.rq.check_stamp_task(task,
>> taskname_from_tid(task), recurse = True, cache=self.stampcache):
>> + logger.debug(2, 'Stamp for underlying task %s is
>> current, so skipping setscene variant', task)
>> + self.sq_task_failoutright(task)
>> + return True
>>
>> - self.rq.state = runQueueCleanUp
>> + if self.cooker.configuration.force:
>> + if task in self.rqdata.target_tids:
>> + self.sq_task_failoutright(task)
>> + return True
>> +
>> + if self.rq.check_stamp_task(task, taskname,
>> cache=self.stampcache):
>> + logger.debug(2, 'Setscene stamp current task %s, so
>> skip it and its dependencies', task)
>> + self.sq_task_skip(task)
>> return True
>>
>> - self.rq.read_workers()
>> + if self.cooker.configuration.skipsetscene:
>> + logger.debug(2, 'No setscene tasks should be
>> executed. Skipping %s', task)
>> + self.sq_task_failoutright(task)
>> + return True
>>
>> - if self.stats.total == 0:
>> - # nothing to do
>> - self.rq.state = runQueueCleanUp
>> + startevent = sceneQueueTaskStarted(task, self.sq_stats,
>> self.rq)
>> + bb.event.fire(startevent, self.cfgData)
>> +
>> + taskdepdata = self.sq_build_taskdepdata(task)
>> +
>> + taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn]
>> + taskhash = self.rqdata.get_task_hash(task)
>> + unihash = self.rqdata.get_task_unihash(task)
>> + if 'fakeroot' in taskdep and taskname in
>> taskdep['fakeroot'] and not self.cooker.configuration.dry_run:
>> + if not mc in self.rq.fakeworker:
>> + self.rq.start_fakeworker(self, mc)
>> +
>> self.rq.fakeworker[mc].process.stdin.write(b"<runtask>" +
>> pickle.dumps((taskfn, task, taskname, taskhash, unihash, True,
>> self.cooker.collection.get_file_appends(taskfn), taskdepdata, False))
>> + b"</runtask>")
>> + self.rq.fakeworker[mc].process.stdin.flush()
>> + else:
>> + self.rq.worker[mc].process.stdin.write(b"<runtask>"
>> + pickle.dumps((taskfn, task, taskname, taskhash, unihash, True,
>> self.cooker.collection.get_file_appends(taskfn), taskdepdata, False))
>> + b"</runtask>")
>> + self.rq.worker[mc].process.stdin.flush()
>>
>> - task = self.sched.next()
>> + self.build_stamps[task] = bb.build.stampfile(taskname,
>> self.rqdata.dataCaches[mc], taskfn, noextra=True)
>> + self.build_stamps2.append(self.build_stamps[task])
>> + self.sq_running.add(task)
>> + self.sq_live.add(task)
>> + self.sq_stats.taskActive()
>> + if self.can_start_task():
>> + return True
>> +
>> + self.update_holdofftasks()
>> +
>> + if not self.sq_live and not self.sqdone and not
>> self.sq_deferred and not self.updated_taskhash_queue and not
>> self.holdoff_tasks:
>> + logger.info("Setscene tasks completed")
>> +
>> + err = self.summarise_scenequeue_errors()
>> + if err:
>> + self.rq.state = runQueueFailed
>> + return True
>> +
>> + if self.cooker.configuration.setsceneonly:
>> + self.rq.state = runQueueComplete
>> + return True
>> + self.sqdone = True
>> +
>> + if self.stats.total == 0:
>> + # nothing to do
>> + self.rq.state = runQueueComplete
>> + return True
>> +
>> + if self.cooker.configuration.setsceneonly:
>> + task = None
>> + else:
>> + task = self.sched.next()
>> if task is not None:
>> (mc, fn, taskname, taskfn) = split_tid_mcfn(task)
>>
>> - if task in self.rq.scenequeue_covered:
>> + if self.rqdata.setscenewhitelist is not None:
>> + if self.check_setscenewhitelist(task):
>> + self.task_fail(task, "setscene whitelist")
>> + return True
>> +
>> + if task in self.tasks_covered:
>> logger.debug(2, "Setscene covered task %s", task)
>> self.task_skip(task, "covered")
>> return True
>> @@ -2030,6 +2110,8 @@ class RunQueueExecuteTasks(RunQueueExecute):
>> taskdepdata = self.build_taskdepdata(task)
>>
>> taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn]
>> + taskhash = self.rqdata.get_task_hash(task)
>> + unihash = self.rqdata.get_task_unihash(task)
>> if 'fakeroot' in taskdep and taskname in
>> taskdep['fakeroot'] and not (self.cooker.configuration.dry_run or
>> self.rqdata.setscene_enforce): if not mc in self.rq.fakeworker: try:
>> @@ -2039,10 +2121,10 @@ class RunQueueExecuteTasks(RunQueueExecute):
>> self.rq.state = runQueueFailed
>> self.stats.taskFailed()
>> return True
>> -
>> self.rq.fakeworker[mc].process.stdin.write(b"<runtask>" +
>> pickle.dumps((taskfn, task, taskname, False,
>> self.cooker.collection.get_file_appends(taskfn), taskdepdata,
>> self.rqdata.setscene_enforce)) + b"</runtask>")
>> +
>> self.rq.fakeworker[mc].process.stdin.write(b"<runtask>" +
>> pickle.dumps((taskfn, task, taskname, taskhash, unihash, False,
>> self.cooker.collection.get_file_appends(taskfn), taskdepdata,
>> self.rqdata.setscene_enforce)) + b"</runtask>")
>> self.rq.fakeworker[mc].process.stdin.flush() else:
>> - self.rq.worker[mc].process.stdin.write(b"<runtask>"
>> + pickle.dumps((taskfn, task, taskname, False,
>> self.cooker.collection.get_file_appends(taskfn), taskdepdata,
>> self.rqdata.setscene_enforce)) + b"</runtask>")
>> + self.rq.worker[mc].process.stdin.write(b"<runtask>"
>> + pickle.dumps((taskfn, task, taskname, taskhash, unihash, False,
>> self.cooker.collection.get_file_appends(taskfn), taskdepdata,
>> self.rqdata.setscene_enforce)) + b"</runtask>")
>> self.rq.worker[mc].process.stdin.flush() self.build_stamps[task] =
>> bb.build.stampfile(taskname, self.rqdata.dataCaches[mc], taskfn,
>> noextra=True) @@ -2052,30 +2134,58 @@ class
>> RunQueueExecuteTasks(RunQueueExecute): if self.can_start_task():
>> return True
>> - if self.stats.active > 0:
>> + if self.stats.active > 0 or self.sq_stats.active > 0:
>> self.rq.read_workers()
>> return self.rq.active_fds()
>>
>> + # No more tasks can be run. If we have deferred setscene
>> tasks we should run them.
>> + if self.sq_deferred:
>> + tid =
>> self.sq_deferred.pop(list(self.sq_deferred.keys())[0])
>> + logger.warning("Runqeueue deadlocked on deferred tasks,
>> forcing task %s" % tid)
>> + self.sq_task_failoutright(tid)
>> + return True
>> +
>> if len(self.failed_tids) != 0:
>> self.rq.state = runQueueFailed
>> return True
>>
>> # Sanity Checks
>> + err = self.summarise_scenequeue_errors()
>> for task in self.rqdata.runtaskentries:
>> if task not in self.runq_buildable:
>> logger.error("Task %s never buildable!", task)
>> - if task not in self.runq_running:
>> + err = True
>> + elif task not in self.runq_running:
>> logger.error("Task %s never ran!", task)
>> - if task not in self.runq_complete:
>> + err = True
>> + elif task not in self.runq_complete:
>> logger.error("Task %s never completed!", task)
>> - self.rq.state = runQueueComplete
>> + err = True
>> +
>> + if err:
>> + self.rq.state = runQueueFailed
>> + else:
>> + self.rq.state = runQueueComplete
>>
>> return True
>>
>> + def filtermcdeps(self, task, mc, deps):
>> + ret = set()
>> + for dep in deps:
>> + thismc = mc_from_tid(dep)
>> + if thismc != mc:
>> + continue
>> + ret.add(dep)
>> + return ret
>> +
>> + # We filter out multiconfig dependencies from taskdepdata we
>> pass to the tasks
>> + # as most code can't handle them
>> def build_taskdepdata(self, task):
>> taskdepdata = {}
>> - next = self.rqdata.runtaskentries[task].depends
>> + mc = mc_from_tid(task)
>> + next = self.rqdata.runtaskentries[task].depends.copy()
>> next.add(task)
>> + next = self.filtermcdeps(task, mc, next)
>> while next:
>> additional = []
>> for revdep in next:
>> @@ -2084,7 +2194,9 @@ class RunQueueExecuteTasks(RunQueueExecute):
>> deps = self.rqdata.runtaskentries[revdep].depends
>> provides =
>> self.rqdata.dataCaches[mc].fn_provides[taskfn] taskhash =
>> self.rqdata.runtaskentries[revdep].hash
>> - taskdepdata[revdep] = [pn, taskname, fn, deps,
>> provides, taskhash]
>> + unihash = self.rqdata.runtaskentries[revdep].unihash
>> + deps = self.filtermcdeps(task, mc, deps)
>> + taskdepdata[revdep] = [pn, taskname, fn, deps,
>> provides, taskhash, unihash] for revdep2 in deps:
>> if revdep2 not in taskdepdata:
>> additional.append(revdep2)
>> @@ -2093,269 +2205,197 @@ class RunQueueExecuteTasks(RunQueueExecute):
>> #bb.note("Task %s: " % task + str(taskdepdata).replace("],
>> ", "],\n")) return taskdepdata
>>
>> -class RunQueueExecuteScenequeue(RunQueueExecute):
>> - def __init__(self, rq):
>> - RunQueueExecute.__init__(self, rq)
>> -
>> - self.scenequeue_covered = set()
>> - self.scenequeue_notcovered = set()
>> - self.scenequeue_notneeded = set()
>> + def update_holdofftasks(self):
>>
>> - # If we don't have any setscene functions, skip this step
>> - if len(self.rqdata.runq_setscene_tids) == 0:
>> - rq.scenequeue_covered = set()
>> - rq.scenequeue_notcovered = set()
>> - rq.state = runQueueRunInit
>> + if not self.holdoff_need_update:
>> return
>>
>> - self.stats =
>> RunQueueStats(len(self.rqdata.runq_setscene_tids))
>> + notcovered = set(self.scenequeue_notcovered)
>> + notcovered |= self.cantskip
>> + for tid in self.scenequeue_notcovered:
>> + notcovered |= self.sqdata.sq_covered_tasks[tid]
>> + notcovered |=
>> self.sqdata.unskippable.difference(self.rqdata.runq_setscene_tids)
>> + notcovered.intersection_update(self.tasks_scenequeue_done)
>>
>> - sq_revdeps = {}
>> - sq_revdeps_new = {}
>> - sq_revdeps_squash = {}
>> - self.sq_harddeps = {}
>> - self.stamps = {}
>> -
>> - # We need to construct a dependency graph for the setscene
>> functions. Intermediate
>> - # dependencies between the setscene tasks only complicate
>> the code. This code
>> - # therefore aims to collapse the huge runqueue dependency
>> tree into a smaller one
>> - # only containing the setscene functions.
>> -
>> - self.rqdata.init_progress_reporter.next_stage()
>> -
>> - # First process the chains up to the first setscene task.
>> - endpoints = {}
>> - for tid in self.rqdata.runtaskentries:
>> - sq_revdeps[tid] =
>> copy.copy(self.rqdata.runtaskentries[tid].revdeps)
>> - sq_revdeps_new[tid] = set()
>> - if (len(sq_revdeps[tid]) == 0) and tid not in
>> self.rqdata.runq_setscene_tids:
>> - #bb.warn("Added endpoint %s" % (tid))
>> - endpoints[tid] = set()
>> + covered = set(self.scenequeue_covered)
>> + for tid in self.scenequeue_covered:
>> + covered |= self.sqdata.sq_covered_tasks[tid]
>> + covered.difference_update(notcovered)
>> + covered.intersection_update(self.tasks_scenequeue_done)
>>
>> - self.rqdata.init_progress_reporter.next_stage()
>> + for tid in notcovered | covered:
>> + if len(self.rqdata.runtaskentries[tid].depends) == 0:
>> + self.setbuildable(tid)
>> + elif
>> self.rqdata.runtaskentries[tid].depends.issubset(self.runq_complete):
>> + self.setbuildable(tid)
>>
>> - # Secondly process the chains between setscene tasks.
>> - for tid in self.rqdata.runq_setscene_tids:
>> - #bb.warn("Added endpoint 2 %s" % (tid))
>> - for dep in self.rqdata.runtaskentries[tid].depends:
>> - if tid in sq_revdeps[dep]:
>> - sq_revdeps[dep].remove(tid)
>> - if dep not in endpoints:
>> - endpoints[dep] = set()
>> - #bb.warn(" Added endpoint 3 %s" % (dep))
>> - endpoints[dep].add(tid)
>> -
>> - self.rqdata.init_progress_reporter.next_stage()
>> -
>> - def process_endpoints(endpoints):
>> - newendpoints = {}
>> - for point, task in endpoints.items():
>> - tasks = set()
>> - if task:
>> - tasks |= task
>> - if sq_revdeps_new[point]:
>> - tasks |= sq_revdeps_new[point]
>> - sq_revdeps_new[point] = set()
>> - if point in self.rqdata.runq_setscene_tids:
>> - sq_revdeps_new[point] = tasks
>> - tasks = set()
>> - continue
>> - for dep in self.rqdata.runtaskentries[point].depends:
>> - if point in sq_revdeps[dep]:
>> - sq_revdeps[dep].remove(point)
>> - if tasks:
>> - sq_revdeps_new[dep] |= tasks
>> - if len(sq_revdeps[dep]) == 0 and dep not in
>> self.rqdata.runq_setscene_tids:
>> - newendpoints[dep] = task
>> - if len(newendpoints) != 0:
>> - process_endpoints(newendpoints)
>> -
>> - process_endpoints(endpoints)
>> -
>> - self.rqdata.init_progress_reporter.next_stage()
>> -
>> - # Build a list of setscene tasks which are "unskippable"
>> - # These are direct endpoints referenced by the build
>> - endpoints2 = {}
>> - sq_revdeps2 = {}
>> - sq_revdeps_new2 = {}
>> - def process_endpoints2(endpoints):
>> - newendpoints = {}
>> - for point, task in endpoints.items():
>> - tasks = set([point])
>> - if task:
>> - tasks |= task
>> - if sq_revdeps_new2[point]:
>> - tasks |= sq_revdeps_new2[point]
>> - sq_revdeps_new2[point] = set()
>> - if point in self.rqdata.runq_setscene_tids:
>> - sq_revdeps_new2[point] = tasks
>> - for dep in self.rqdata.runtaskentries[point].depends:
>> - if point in sq_revdeps2[dep]:
>> - sq_revdeps2[dep].remove(point)
>> - if tasks:
>> - sq_revdeps_new2[dep] |= tasks
>> - if (len(sq_revdeps2[dep]) == 0 or
>> len(sq_revdeps_new2[dep]) != 0) and dep not in
>> self.rqdata.runq_setscene_tids:
>> - newendpoints[dep] = tasks
>> - if len(newendpoints) != 0:
>> - process_endpoints2(newendpoints)
>> - for tid in self.rqdata.runtaskentries:
>> - sq_revdeps2[tid] =
>> copy.copy(self.rqdata.runtaskentries[tid].revdeps)
>> - sq_revdeps_new2[tid] = set()
>> - if (len(sq_revdeps2[tid]) == 0) and tid not in
>> self.rqdata.runq_setscene_tids:
>> - endpoints2[tid] = set()
>> - process_endpoints2(endpoints2)
>> - self.unskippable = []
>> - for tid in self.rqdata.runq_setscene_tids:
>> - if sq_revdeps_new2[tid]:
>> - self.unskippable.append(tid)
>> + self.tasks_covered = covered
>> + self.tasks_notcovered = notcovered
>>
>> -
>> self.rqdata.init_progress_reporter.next_stage(len(self.rqdata.runtaskentries))
>> + self.holdoff_tasks = set()
>>
>> - for taskcounter, tid in
>> enumerate(self.rqdata.runtaskentries):
>> - if tid in self.rqdata.runq_setscene_tids:
>> - deps = set()
>> - for dep in sq_revdeps_new[tid]:
>> - deps.add(dep)
>> - sq_revdeps_squash[tid] = deps
>> - elif len(sq_revdeps_new[tid]) != 0:
>> - bb.msg.fatal("RunQueue", "Something went badly wrong
>> during scenequeue generation, aborting. Please report this problem.")
>> - self.rqdata.init_progress_reporter.update(taskcounter)
>> -
>> - self.rqdata.init_progress_reporter.next_stage()
>> -
>> - # Resolve setscene inter-task dependencies
>> - # e.g. do_sometask_setscene[depends] =
>> "targetname:do_someothertask_setscene"
>> - # Note that anything explicitly depended upon will have its
>> reverse dependencies removed to avoid circular dependencies for tid
>> in self.rqdata.runq_setscene_tids:
>> - (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
>> - realtid = tid + "_setscene"
>> - idepends =
>> self.rqdata.taskData[mc].taskentries[realtid].idepends
>> - self.stamps[tid] = bb.build.stampfile(taskname +
>> "_setscene", self.rqdata.dataCaches[mc], taskfn, noextra=True)
>> - for (depname, idependtask) in idepends:
>> -
>> - if depname not in
>> self.rqdata.taskData[mc].build_targets:
>> - continue
>> + if tid not in self.scenequeue_covered and tid not in
>> self.scenequeue_notcovered:
>> + self.holdoff_tasks.add(tid)
>>
>> - depfn =
>> self.rqdata.taskData[mc].build_targets[depname][0]
>> - if depfn is None:
>> - continue
>> - deptid = depfn + ":" +
>> idependtask.replace("_setscene", "")
>> - if deptid not in self.rqdata.runtaskentries:
>> - bb.msg.fatal("RunQueue", "Task %s depends
>> upon non-existent task %s:%s" % (realtid, depfn, idependtask)) -
>> - if not deptid in self.sq_harddeps:
>> - self.sq_harddeps[deptid] = set()
>> - self.sq_harddeps[deptid].add(tid)
>> -
>> - sq_revdeps_squash[tid].add(deptid)
>> - # Have to zero this to avoid circular
>> dependencies
>> - sq_revdeps_squash[deptid] = set()
>> -
>> - self.rqdata.init_progress_reporter.next_stage()
>> -
>> - for task in self.sq_harddeps:
>> - for dep in self.sq_harddeps[task]:
>> - sq_revdeps_squash[dep].add(task)
>> -
>> - self.rqdata.init_progress_reporter.next_stage()
>> -
>> - #for tid in sq_revdeps_squash:
>> - # for dep in sq_revdeps_squash[tid]:
>> - # data = data + "\n %s" % dep
>> - # bb.warn("Task %s_setscene: is %s " % (tid, data
>> -
>> - self.sq_deps = {}
>> - self.sq_revdeps = sq_revdeps_squash
>> - self.sq_revdeps2 = copy.deepcopy(self.sq_revdeps)
>> -
>> - for tid in self.sq_revdeps:
>> - self.sq_deps[tid] = set()
>> - for tid in self.sq_revdeps:
>> - for dep in self.sq_revdeps[tid]:
>> - self.sq_deps[dep].add(tid)
>> -
>> - self.rqdata.init_progress_reporter.next_stage()
>> -
>> - for tid in self.sq_revdeps:
>> - if len(self.sq_revdeps[tid]) == 0:
>> - self.runq_buildable.add(tid)
>> -
>> - self.rqdata.init_progress_reporter.finish()
>> -
>> - self.outrightfail = []
>> - if self.rq.hashvalidate:
>> - sq_hash = []
>> - sq_hashfn = []
>> - sq_fn = []
>> - sq_taskname = []
>> - sq_task = []
>> - noexec = []
>> - stamppresent = []
>> - for tid in self.sq_revdeps:
>> - (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
>> + for tid in self.holdoff_tasks.copy():
>> + for dep in self.sqdata.sq_covered_tasks[tid]:
>> + if dep not in self.runq_complete:
>> + self.holdoff_tasks.add(dep)
>>
>> - taskdep =
>> self.rqdata.dataCaches[mc].task_deps[taskfn]
>> + self.holdoff_need_update = False
>>
>> - if 'noexec' in taskdep and taskname in
>> taskdep['noexec']:
>> - noexec.append(tid)
>> - self.task_skip(tid)
>> - bb.build.make_stamp(taskname + "_setscene",
>> self.rqdata.dataCaches[mc], taskfn)
>> - continue
>> + def process_possible_migrations(self):
>>
>> - if self.rq.check_stamp_task(tid, taskname +
>> "_setscene", cache=self.stampcache):
>> - logger.debug(2, 'Setscene stamp current for task
>> %s', tid)
>> - stamppresent.append(tid)
>> - self.task_skip(tid)
>> - continue
>> + changed = set()
>> + for tid, unihash in self.updated_taskhash_queue.copy():
>> + if tid in self.runq_running and tid not in
>> self.runq_complete:
>> + continue
>>
>> - if self.rq.check_stamp_task(tid, taskname, recurse =
>> True, cache=self.stampcache):
>> - logger.debug(2, 'Normal stamp current for task
>> %s', tid)
>> - stamppresent.append(tid)
>> - self.task_skip(tid)
>> - continue
>> + self.updated_taskhash_queue.remove((tid, unihash))
>> +
>> + if unihash != self.rqdata.runtaskentries[tid].unihash:
>> + logger.info("Task %s unihash changed to %s" % (tid,
>> unihash))
>> + self.rqdata.runtaskentries[tid].unihash = unihash
>> + bb.parse.siggen.set_unihash(tid, unihash)
>> +
>> + # Work out all tasks which depend on this one
>> + total = set()
>> + next = set(self.rqdata.runtaskentries[tid].revdeps)
>> + while next:
>> + current = next.copy()
>> + total = total |next
>> + next = set()
>> + for ntid in current:
>> + next |=
>> self.rqdata.runtaskentries[ntid].revdeps
>> + next.difference_update(total)
>> +
>> + # Now iterate those tasks in dependency order to
>> regenerate their taskhash/unihash
>> + done = set()
>> + next = set(self.rqdata.runtaskentries[tid].revdeps)
>> + while next:
>> + current = next.copy()
>> + next = set()
>> + for tid in current:
>> + if not
>> self.rqdata.runtaskentries[tid].depends.isdisjoint(total):
>> + continue
>> + procdep = []
>> + for dep in
>> self.rqdata.runtaskentries[tid].depends:
>> + procdep.append(dep)
>> + orighash =
>> self.rqdata.runtaskentries[tid].hash
>> + self.rqdata.runtaskentries[tid].hash =
>> bb.parse.siggen.get_taskhash(tid, procdep,
>> self.rqdata.dataCaches[mc_from_tid(tid)])
>> + origuni =
>> self.rqdata.runtaskentries[tid].unihash
>> + self.rqdata.runtaskentries[tid].unihash =
>> bb.parse.siggen.get_unihash(tid)
>> + logger.debug(1, "Task %s hash changes:
>> %s->%s %s->%s" % (tid, orighash,
>> self.rqdata.runtaskentries[tid].hash, origuni,
>> self.rqdata.runtaskentries[tid].unihash))
>> + next |=
>> self.rqdata.runtaskentries[tid].revdeps
>> + changed.add(tid)
>> + total.remove(tid)
>> + next.intersection_update(total)
>> +
>> + if changed:
>> + for mc in self.rq.worker:
>> +
>> self.rq.worker[mc].process.stdin.write(b"<newtaskhashes>" +
>> pickle.dumps(bb.parse.siggen.get_taskhashes()) + b"</newtaskhashes>")
>> + for mc in self.rq.fakeworker:
>> +
>> self.rq.fakeworker[mc].process.stdin.write(b"<newtaskhashes>" +
>> pickle.dumps(bb.parse.siggen.get_taskhashes()) + b"</newtaskhashes>")
>> +
>> + logger.debug(1, pprint.pformat("Tasks changed:\n%s" %
>> (changed))) +
>> + for tid in changed:
>> + if tid not in self.rqdata.runq_setscene_tids:
>> + continue
>> + if tid in self.runq_running:
>> + continue
>> + if tid in self.scenequeue_covered:
>> + # Potentially risky, should we report this hash as a
>> match?
>> + logger.info("Already covered setscene for %s so
>> ignoring rehash" % (tid))
>> + continue
>> + if tid not in self.pending_migrations:
>> + self.pending_migrations.add(tid)
>> +
>> + for tid in self.pending_migrations.copy():
>> + valid = True
>> + # Check no tasks this covers are running
>> + for dep in self.sqdata.sq_covered_tasks[tid]:
>> + if dep in self.runq_running and dep not in
>> self.runq_complete:
>> + logger.debug(2, "Task %s is running which blocks
>> setscene for %s from running" % (dep, tid))
>> + valid = False
>> + break
>> + if not valid:
>> + continue
>>
>> - sq_fn.append(fn)
>> -
>> sq_hashfn.append(self.rqdata.dataCaches[mc].hashfn[taskfn])
>> - sq_hash.append(self.rqdata.runtaskentries[tid].hash)
>> - sq_taskname.append(taskname)
>> - sq_task.append(tid)
>> + self.pending_migrations.remove(tid)
>> + changed = True
>>
>> -
>> self.cooker.data.setVar("BB_SETSCENE_STAMPCURRENT_COUNT",
>> len(stamppresent))
>> + if tid in self.tasks_scenequeue_done:
>> + self.tasks_scenequeue_done.remove(tid)
>> + for dep in self.sqdata.sq_covered_tasks[tid]:
>> + if dep not in self.runq_complete:
>> + if dep in self.tasks_scenequeue_done and dep not
>> in self.sqdata.unskippable:
>> + self.tasks_scenequeue_done.remove(dep)
>> +
>> + if tid in self.sq_buildable:
>> + self.sq_buildable.remove(tid)
>> + if tid in self.sq_running:
>> + self.sq_running.remove(tid)
>> + if
>> self.sqdata.sq_revdeps[tid].issubset(self.scenequeue_covered |
>> self.scenequeue_notcovered):
>> + if tid not in self.sq_buildable:
>> + self.sq_buildable.add(tid)
>> + if len(self.sqdata.sq_revdeps[tid]) == 0:
>> + self.sq_buildable.add(tid)
>> +
>> + if tid in self.sqdata.outrightfail:
>> + self.sqdata.outrightfail.remove(tid)
>> + if tid in self.scenequeue_notcovered:
>> + self.scenequeue_notcovered.remove(tid)
>> + if tid in self.scenequeue_covered:
>> + self.scenequeue_covered.remove(tid)
>> + if tid in self.scenequeue_notneeded:
>> + self.scenequeue_notneeded.remove(tid)
>>
>> - call = self.rq.hashvalidate + "(sq_fn, sq_task, sq_hash,
>> sq_hashfn, d)"
>> - locs = { "sq_fn" : sq_fn, "sq_task" : sq_taskname,
>> "sq_hash" : sq_hash, "sq_hashfn" : sq_hashfn, "d" : self.cooker.data }
>> - valid = bb.utils.better_eval(call, locs)
>> + (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
>> + self.sqdata.stamps[tid] = bb.build.stampfile(taskname +
>> "_setscene", self.rqdata.dataCaches[mc], taskfn, noextra=True)
>> - self.cooker.data.delVar("BB_SETSCENE_STAMPCURRENT_COUNT")
>> + if tid in self.stampcache:
>> + del self.stampcache[tid]
>>
>> - valid_new = stamppresent
>> - for v in valid:
>> - valid_new.append(sq_task[v])
>> + if tid in self.build_stamps:
>> + del self.build_stamps[tid]
>>
>> - for tid in self.sq_revdeps:
>> - if tid not in valid_new and tid not in noexec:
>> - logger.debug(2, 'No package found, so skipping
>> setscene task %s', tid)
>> - self.outrightfail.append(tid)
>> + logger.info("Setscene task %s now valid and being rerun"
>> % tid)
>> + self.sqdone = False
>> + update_scenequeue_data([tid], self.sqdata, self.rqdata,
>> self.rq, self.cooker, self.stampcache, self)
>> - logger.info('Executing SetScene Tasks')
>> + if changed:
>> + self.holdoff_need_update = True
>>
>> - self.rq.state = runQueueSceneRun
>> + def scenequeue_updatecounters(self, task, fail=False):
>>
>> - def scenequeue_updatecounters(self, task, fail = False):
>> - for dep in self.sq_deps[task]:
>> - if fail and task in self.sq_harddeps and dep in
>> self.sq_harddeps[task]:
>> + for dep in sorted(self.sqdata.sq_deps[task]):
>> + if fail and task in self.sqdata.sq_harddeps and dep in
>> self.sqdata.sq_harddeps[task]: logger.debug(2, "%s was unavailable
>> and is a hard dependency of %s so skipping" % (task, dep))
>> - self.scenequeue_updatecounters(dep, fail)
>> + self.sq_task_failoutright(dep)
>> continue
>> - if task not in self.sq_revdeps2[dep]:
>> - # May already have been removed by the fail case
>> above
>> - continue
>> - self.sq_revdeps2[dep].remove(task)
>> - if len(self.sq_revdeps2[dep]) == 0:
>> - self.runq_buildable.add(dep)
>> + if
>> self.sqdata.sq_revdeps[dep].issubset(self.scenequeue_covered |
>> self.scenequeue_notcovered):
>> + if dep not in self.sq_buildable:
>> + self.sq_buildable.add(dep)
>>
>> - def task_completeoutright(self, task):
>> + next = set([task])
>> + while next:
>> + new = set()
>> + for t in sorted(next):
>> + self.tasks_scenequeue_done.add(t)
>> + # Look down the dependency chain for non-setscene
>> things which this task depends on
>> + # and mark as 'done'
>> + for dep in self.rqdata.runtaskentries[t].depends:
>> + if dep in self.rqdata.runq_setscene_tids or dep
>> in self.tasks_scenequeue_done:
>> + continue
>> + if
>> self.rqdata.runtaskentries[dep].revdeps.issubset(self.tasks_scenequeue_done):
>> + new.add(dep)
>> + next = new
>> +
>> + self.holdoff_need_update = True
>> +
>> + def sq_task_completeoutright(self, task):
>> """
>> Mark a task as completed
>> Look at the reverse dependencies and mark any task with
>> @@ -2366,7 +2406,7 @@ class
>> RunQueueExecuteScenequeue(RunQueueExecute):
>> self.scenequeue_covered.add(task) self.scenequeue_updatecounters(task)
>>
>> - def check_taskfail(self, task):
>> + def sq_check_taskfail(self, task):
>> if self.rqdata.setscenewhitelist is not None:
>> realtask = task.split('_setscene')[0]
>> (mc, fn, taskname, taskfn) = split_tid_mcfn(realtask)
>> @@ -2375,130 +2415,34 @@ class
>> RunQueueExecuteScenequeue(RunQueueExecute): logger.error('Task %s.%s
>> failed' % (pn, taskname + "_setscene")) self.rq.state =
>> runQueueCleanUp
>> - def task_complete(self, task):
>> - self.stats.taskCompleted()
>> - bb.event.fire(sceneQueueTaskCompleted(task, self.stats,
>> self.rq), self.cfgData)
>> - self.task_completeoutright(task)
>> + def sq_task_complete(self, task):
>> + self.sq_stats.taskCompleted()
>> + bb.event.fire(sceneQueueTaskCompleted(task, self.sq_stats,
>> self.rq), self.cfgData)
>> + self.sq_task_completeoutright(task)
>>
>> - def task_fail(self, task, result):
>> - self.stats.taskFailed()
>> - bb.event.fire(sceneQueueTaskFailed(task, self.stats, result,
>> self), self.cfgData)
>> + def sq_task_fail(self, task, result):
>> + self.sq_stats.taskFailed()
>> + bb.event.fire(sceneQueueTaskFailed(task, self.sq_stats,
>> result, self), self.cfgData) self.scenequeue_notcovered.add(task)
>> self.scenequeue_updatecounters(task, True)
>> - self.check_taskfail(task)
>> + self.sq_check_taskfail(task)
>>
>> - def task_failoutright(self, task):
>> - self.runq_running.add(task)
>> - self.runq_buildable.add(task)
>> - self.stats.taskSkipped()
>> - self.stats.taskCompleted()
>> + def sq_task_failoutright(self, task):
>> + self.sq_running.add(task)
>> + self.sq_buildable.add(task)
>> + self.sq_stats.taskSkipped()
>> + self.sq_stats.taskCompleted()
>> self.scenequeue_notcovered.add(task)
>> self.scenequeue_updatecounters(task, True)
>>
>> - def task_skip(self, task):
>> - self.runq_running.add(task)
>> - self.runq_buildable.add(task)
>> - self.task_completeoutright(task)
>> - self.stats.taskSkipped()
>> - self.stats.taskCompleted()
>> -
>> - def execute(self):
>> - """
>> - Run the tasks in a queue prepared by prepare_runqueue
>> - """
>> -
>> - self.rq.read_workers()
>> -
>> - task = None
>> - if self.can_start_task():
>> - # Find the next setscene to run
>> - for nexttask in self.rqdata.runq_setscene_tids:
>> - if nexttask in self.runq_buildable and nexttask not
>> in self.runq_running and self.stamps[nexttask] not in
>> self.build_stamps.values():
>> - if nexttask in self.unskippable:
>> - logger.debug(2, "Setscene task %s is
>> unskippable" % nexttask)
>> - if nexttask not in self.unskippable and
>> len(self.sq_revdeps[nexttask]) > 0 and
>> self.sq_revdeps[nexttask].issubset(self.scenequeue_covered) and
>> self.check_dependencies(nexttask, self.sq_revdeps[nexttask], True):
>> - fn = fn_from_tid(nexttask)
>> - foundtarget = False
>> -
>> - if nexttask in self.rqdata.target_tids:
>> - foundtarget = True
>> - if not foundtarget:
>> - logger.debug(2, "Skipping setscene for
>> task %s" % nexttask)
>> - self.task_skip(nexttask)
>> - self.scenequeue_notneeded.add(nexttask)
>> - return True
>> - if nexttask in self.outrightfail:
>> - self.task_failoutright(nexttask)
>> - return True
>> - task = nexttask
>> - break
>> - if task is not None:
>> - (mc, fn, taskname, taskfn) = split_tid_mcfn(task)
>> - taskname = taskname + "_setscene"
>> - if self.rq.check_stamp_task(task,
>> taskname_from_tid(task), recurse = True, cache=self.stampcache):
>> - logger.debug(2, 'Stamp for underlying task %s is
>> current, so skipping setscene variant', task)
>> - self.task_failoutright(task)
>> - return True
>> -
>> - if self.cooker.configuration.force:
>> - if task in self.rqdata.target_tids:
>> - self.task_failoutright(task)
>> - return True
>> -
>> - if self.rq.check_stamp_task(task, taskname,
>> cache=self.stampcache):
>> - logger.debug(2, 'Setscene stamp current task %s, so
>> skip it and its dependencies', task)
>> - self.task_skip(task)
>> - return True
>> -
>> - startevent = sceneQueueTaskStarted(task, self.stats,
>> self.rq)
>> - bb.event.fire(startevent, self.cfgData)
>> + def sq_task_skip(self, task):
>> + self.sq_running.add(task)
>> + self.sq_buildable.add(task)
>> + self.sq_task_completeoutright(task)
>> + self.sq_stats.taskSkipped()
>> + self.sq_stats.taskCompleted()
>>
>> - taskdepdata = self.build_taskdepdata(task)
>> -
>> - taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn]
>> - if 'fakeroot' in taskdep and taskname in
>> taskdep['fakeroot'] and not self.cooker.configuration.dry_run:
>> - if not mc in self.rq.fakeworker:
>> - self.rq.start_fakeworker(self, mc)
>> -
>> self.rq.fakeworker[mc].process.stdin.write(b"<runtask>" +
>> pickle.dumps((taskfn, task, taskname, True,
>> self.cooker.collection.get_file_appends(taskfn), taskdepdata, False))
>> + b"</runtask>")
>> - self.rq.fakeworker[mc].process.stdin.flush()
>> - else:
>> - self.rq.worker[mc].process.stdin.write(b"<runtask>"
>> + pickle.dumps((taskfn, task, taskname, True,
>> self.cooker.collection.get_file_appends(taskfn), taskdepdata, False))
>> + b"</runtask>")
>> - self.rq.worker[mc].process.stdin.flush()
>> -
>> - self.build_stamps[task] = bb.build.stampfile(taskname,
>> self.rqdata.dataCaches[mc], taskfn, noextra=True)
>> - self.build_stamps2.append(self.build_stamps[task])
>> - self.runq_running.add(task)
>> - self.stats.taskActive()
>> - if self.can_start_task():
>> - return True
>> -
>> - if self.stats.active > 0:
>> - self.rq.read_workers()
>> - return self.rq.active_fds()
>> -
>> - #for tid in self.sq_revdeps:
>> - # if tid not in self.runq_running:
>> - # buildable = tid in self.runq_buildable
>> - # revdeps = self.sq_revdeps[tid]
>> - # bb.warn("Found we didn't run %s %s %s" % (tid,
>> buildable, str(revdeps))) -
>> - self.rq.scenequeue_covered = self.scenequeue_covered
>> - self.rq.scenequeue_notcovered = self.scenequeue_notcovered
>> -
>> - logger.debug(1, 'We can skip tasks %s',
>> "\n".join(sorted(self.rq.scenequeue_covered))) -
>> - self.rq.state = runQueueRunInit
>> -
>> - completeevent = sceneQueueComplete(self.stats, self.rq)
>> - bb.event.fire(completeevent, self.cfgData)
>> -
>> - return True
>> -
>> - def runqueue_process_waitpid(self, task, status):
>> - RunQueueExecute.runqueue_process_waitpid(self, task, status)
>> -
>> -
>> - def build_taskdepdata(self, task):
>> + def sq_build_taskdepdata(self, task):
>> def getsetscenedeps(tid):
>> deps = set()
>> (mc, fn, taskname, _) = split_tid_mcfn(tid)
>> @@ -2526,7 +2470,8 @@ class
>> RunQueueExecuteScenequeue(RunQueueExecute): deps =
>> getsetscenedeps(revdep) provides =
>> self.rqdata.dataCaches[mc].fn_provides[taskfn] taskhash =
>> self.rqdata.runtaskentries[revdep].hash
>> - taskdepdata[revdep] = [pn, taskname, fn, deps,
>> provides, taskhash]
>> + unihash = self.rqdata.runtaskentries[revdep].unihash
>> + taskdepdata[revdep] = [pn, taskname, fn, deps,
>> provides, taskhash, unihash] for revdep2 in deps:
>> if revdep2 not in taskdepdata:
>> additional.append(revdep2)
>> @@ -2535,6 +2480,279 @@ class
>> RunQueueExecuteScenequeue(RunQueueExecute): #bb.note("Task %s: " %
>> task + str(taskdepdata).replace("], ", "],\n")) return taskdepdata
>>
>> + def check_setscenewhitelist(self, tid):
>> + # Check task that is going to run against the whitelist
>> + (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
>> + # Ignore covered tasks
>> + if tid in self.tasks_covered:
>> + return False
>> + # Ignore stamped tasks
>> + if self.rq.check_stamp_task(tid, taskname,
>> cache=self.stampcache):
>> + return False
>> + # Ignore noexec tasks
>> + taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn]
>> + if 'noexec' in taskdep and taskname in taskdep['noexec']:
>> + return False
>> +
>> + pn = self.rqdata.dataCaches[mc].pkg_fn[taskfn]
>> + if not check_setscene_enforce_whitelist(pn, taskname,
>> self.rqdata.setscenewhitelist):
>> + if tid in self.rqdata.runq_setscene_tids:
>> + msg = 'Task %s.%s attempted to execute unexpectedly
>> and should have been setscened' % (pn, taskname)
>> + else:
>> + msg = 'Task %s.%s attempted to execute unexpectedly'
>> % (pn, taskname)
>> + logger.error(msg + '\nThis is usually due to missing
>> setscene tasks. Those missing in this build were: %s' %
>> pprint.pformat(self.scenequeue_notcovered))
>> + return True
>> + return False
>> +
>> +class SQData(object):
>> + def __init__(self):
>> + # SceneQueue dependencies
>> + self.sq_deps = {}
>> + # SceneQueue reverse dependencies
>> + self.sq_revdeps = {}
>> + # Injected inter-setscene task dependencies
>> + self.sq_harddeps = {}
>> + # Cache of stamp files so duplicates can't run in parallel
>> + self.stamps = {}
>> + # Setscene tasks directly depended upon by the build
>> + self.unskippable = set()
>> + # List of setscene tasks which aren't present
>> + self.outrightfail = set()
>> + # A list of normal tasks a setscene task covers
>> + self.sq_covered_tasks = {}
>> +
>> +def build_scenequeue_data(sqdata, rqdata, rq, cooker, stampcache,
>> sqrq): +
>> + sq_revdeps = {}
>> + sq_revdeps_squash = {}
>> + sq_collated_deps = {}
>> +
>> + # We need to construct a dependency graph for the setscene
>> functions. Intermediate
>> + # dependencies between the setscene tasks only complicate the
>> code. This code
>> + # therefore aims to collapse the huge runqueue dependency tree
>> into a smaller one
>> + # only containing the setscene functions.
>> +
>> + rqdata.init_progress_reporter.next_stage()
>> +
>> + # First process the chains up to the first setscene task.
>> + endpoints = {}
>> + for tid in rqdata.runtaskentries:
>> + sq_revdeps[tid] =
>> copy.copy(rqdata.runtaskentries[tid].revdeps)
>> + sq_revdeps_squash[tid] = set()
>> + if (len(sq_revdeps[tid]) == 0) and tid not in
>> rqdata.runq_setscene_tids:
>> + #bb.warn("Added endpoint %s" % (tid))
>> + endpoints[tid] = set()
>> +
>> + rqdata.init_progress_reporter.next_stage()
>> +
>> + # Secondly process the chains between setscene tasks.
>> + for tid in rqdata.runq_setscene_tids:
>> + sq_collated_deps[tid] = set()
>> + #bb.warn("Added endpoint 2 %s" % (tid))
>> + for dep in rqdata.runtaskentries[tid].depends:
>> + if tid in sq_revdeps[dep]:
>> + sq_revdeps[dep].remove(tid)
>> + if dep not in endpoints:
>> + endpoints[dep] = set()
>> + #bb.warn(" Added endpoint 3 %s" % (dep))
>> + endpoints[dep].add(tid)
>> +
>> + rqdata.init_progress_reporter.next_stage()
>> +
>> + def process_endpoints(endpoints):
>> + newendpoints = {}
>> + for point, task in endpoints.items():
>> + tasks = set()
>> + if task:
>> + tasks |= task
>> + if sq_revdeps_squash[point]:
>> + tasks |= sq_revdeps_squash[point]
>> + if point not in rqdata.runq_setscene_tids:
>> + for t in tasks:
>> + sq_collated_deps[t].add(point)
>> + sq_revdeps_squash[point] = set()
>> + if point in rqdata.runq_setscene_tids:
>> + sq_revdeps_squash[point] = tasks
>> + tasks = set()
>> + continue
>> + for dep in rqdata.runtaskentries[point].depends:
>> + if point in sq_revdeps[dep]:
>> + sq_revdeps[dep].remove(point)
>> + if tasks:
>> + sq_revdeps_squash[dep] |= tasks
>> + if len(sq_revdeps[dep]) == 0 and dep not in
>> rqdata.runq_setscene_tids:
>> + newendpoints[dep] = task
>> + if len(newendpoints) != 0:
>> + process_endpoints(newendpoints)
>> +
>> + process_endpoints(endpoints)
>> +
>> + rqdata.init_progress_reporter.next_stage()
>> +
>> + # Build a list of tasks which are "unskippable"
>> + # These are direct endpoints referenced by the build upto and
>> including setscene tasks
>> + # Take the build endpoints (no revdeps) and find the sstate
>> tasks they depend upon
>> + new = True
>> + for tid in rqdata.runtaskentries:
>> + if len(rqdata.runtaskentries[tid].revdeps) == 0:
>> + sqdata.unskippable.add(tid)
>> + sqdata.unskippable |= sqrq.cantskip
>> + while new:
>> + new = False
>> + orig = sqdata.unskippable.copy()
>> + for tid in sorted(orig, reverse=True):
>> + if tid in rqdata.runq_setscene_tids:
>> + continue
>> + if len(rqdata.runtaskentries[tid].depends) == 0:
>> + # These are tasks which have no setscene tasks in
>> their chain, need to mark as directly buildable
>> + sqrq.setbuildable(tid)
>> + sqdata.unskippable |= rqdata.runtaskentries[tid].depends
>> + if sqdata.unskippable != orig:
>> + new = True
>> +
>> + sqrq.tasks_scenequeue_done |=
>> sqdata.unskippable.difference(rqdata.runq_setscene_tids) +
>> +
>> rqdata.init_progress_reporter.next_stage(len(rqdata.runtaskentries)) +
>> + # Sanity check all dependencies could be changed to setscene
>> task references
>> + for taskcounter, tid in enumerate(rqdata.runtaskentries):
>> + if tid in rqdata.runq_setscene_tids:
>> + pass
>> + elif len(sq_revdeps_squash[tid]) != 0:
>> + bb.msg.fatal("RunQueue", "Something went badly wrong
>> during scenequeue generation, aborting. Please report this problem.")
>> + else:
>> + del sq_revdeps_squash[tid]
>> + rqdata.init_progress_reporter.update(taskcounter)
>> +
>> + rqdata.init_progress_reporter.next_stage()
>> +
>> + # Resolve setscene inter-task dependencies
>> + # e.g. do_sometask_setscene[depends] =
>> "targetname:do_someothertask_setscene"
>> + # Note that anything explicitly depended upon will have its
>> reverse dependencies removed to avoid circular dependencies
>> + for tid in rqdata.runq_setscene_tids:
>> + (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
>> + realtid = tid + "_setscene"
>> + idepends = rqdata.taskData[mc].taskentries[realtid].idepends
>> + sqdata.stamps[tid] = bb.build.stampfile(taskname +
>> "_setscene", rqdata.dataCaches[mc], taskfn, noextra=True)
>> + for (depname, idependtask) in idepends:
>> +
>> + if depname not in rqdata.taskData[mc].build_targets:
>> + continue
>> +
>> + depfn = rqdata.taskData[mc].build_targets[depname][0]
>> + if depfn is None:
>> + continue
>> + deptid = depfn + ":" + idependtask.replace("_setscene",
>> "")
>> + if deptid not in rqdata.runtaskentries:
>> + bb.msg.fatal("RunQueue", "Task %s depends upon
>> non-existent task %s:%s" % (realtid, depfn, idependtask)) +
>> + if not deptid in sqdata.sq_harddeps:
>> + sqdata.sq_harddeps[deptid] = set()
>> + sqdata.sq_harddeps[deptid].add(tid)
>> +
>> + sq_revdeps_squash[tid].add(deptid)
>> + # Have to zero this to avoid circular dependencies
>> + sq_revdeps_squash[deptid] = set()
>> +
>> + rqdata.init_progress_reporter.next_stage()
>> +
>> + for task in sqdata.sq_harddeps:
>> + for dep in sqdata.sq_harddeps[task]:
>> + sq_revdeps_squash[dep].add(task)
>> +
>> + rqdata.init_progress_reporter.next_stage()
>> +
>> + #for tid in sq_revdeps_squash:
>> + # data = ""
>> + # for dep in sq_revdeps_squash[tid]:
>> + # data = data + "\n %s" % dep
>> + # bb.warn("Task %s_setscene: is %s " % (tid, data))
>> +
>> + sqdata.sq_revdeps = sq_revdeps_squash
>> + sqdata.sq_covered_tasks = sq_collated_deps
>> +
>> + # Build reverse version of revdeps to populate deps structure
>> + for tid in sqdata.sq_revdeps:
>> + sqdata.sq_deps[tid] = set()
>> + for tid in sqdata.sq_revdeps:
>> + for dep in sqdata.sq_revdeps[tid]:
>> + sqdata.sq_deps[dep].add(tid)
>> +
>> + rqdata.init_progress_reporter.next_stage()
>> +
>> + sqdata.multiconfigs = set()
>> + for tid in sqdata.sq_revdeps:
>> + sqdata.multiconfigs.add(mc_from_tid(tid))
>> + if len(sqdata.sq_revdeps[tid]) == 0:
>> + sqrq.sq_buildable.add(tid)
>> +
>> + rqdata.init_progress_reporter.finish()
>> +
>> + sqdata.noexec = set()
>> + sqdata.stamppresent = set()
>> + sqdata.valid = set()
>> +
>> + update_scenequeue_data(sqdata.sq_revdeps, sqdata, rqdata, rq,
>> cooker, stampcache, sqrq) +
>> +def update_scenequeue_data(tids, sqdata, rqdata, rq, cooker,
>> stampcache, sqrq): +
>> + tocheck = set()
>> +
>> + for tid in sorted(tids):
>> + if tid in sqdata.stamppresent:
>> + sqdata.stamppresent.remove(tid)
>> + if tid in sqdata.valid:
>> + sqdata.valid.remove(tid)
>> +
>> + (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
>> +
>> + taskdep = rqdata.dataCaches[mc].task_deps[taskfn]
>> +
>> + if 'noexec' in taskdep and taskname in taskdep['noexec']:
>> + sqdata.noexec.add(tid)
>> + sqrq.sq_task_skip(tid)
>> + bb.build.make_stamp(taskname + "_setscene",
>> rqdata.dataCaches[mc], taskfn)
>> + continue
>> +
>> + if rq.check_stamp_task(tid, taskname + "_setscene",
>> cache=stampcache):
>> + logger.debug(2, 'Setscene stamp current for task %s',
>> tid)
>> + sqdata.stamppresent.add(tid)
>> + sqrq.sq_task_skip(tid)
>> + continue
>> +
>> + if rq.check_stamp_task(tid, taskname, recurse = True,
>> cache=stampcache):
>> + logger.debug(2, 'Normal stamp current for task %s', tid)
>> + sqdata.stamppresent.add(tid)
>> + sqrq.sq_task_skip(tid)
>> + continue
>> +
>> + tocheck.add(tid)
>> +
>> + sqdata.valid |= rq.validate_hashes(tocheck, cooker.data,
>> len(sqdata.stamppresent), False) +
>> + sqdata.hashes = {}
>> + for mc in sorted(sqdata.multiconfigs):
>> + for tid in sorted(sqdata.sq_revdeps):
>> + if mc_from_tid(tid) != mc:
>> + continue
>> + if tid in sqdata.stamppresent:
>> + continue
>> + if tid in sqdata.valid:
>> + continue
>> + if tid in sqdata.noexec:
>> + continue
>> + if tid in sqrq.scenequeue_notcovered:
>> + continue
>> + sqdata.outrightfail.add(tid)
>> +
>> + h = pending_hash_index(tid, rqdata)
>> + if h not in sqdata.hashes:
>> + sqdata.hashes[h] = tid
>> + else:
>> + sqrq.sq_deferred[tid] = sqdata.hashes[h]
>> + bb.warn("Deferring %s after %s" % (tid,
>> sqdata.hashes[h])) +
>> +
>> class TaskFailure(Exception):
>> """
>> Exception raised when a task in a runqueue fails
>> @@ -2641,6 +2859,15 @@ class runQueueTaskSkipped(runQueueEvent):
>> runQueueEvent.__init__(self, task, stats, rq)
>> self.reason = reason
>>
>> +class taskUniHashUpdate(bb.event.Event):
>> + """
>> + Base runQueue event class
>> + """
>> + def __init__(self, task, unihash):
>> + self.taskid = task
>> + self.unihash = unihash
>> + bb.event.Event.__init__(self)
>> +
>> class runQueuePipe():
>> """
>> Abstraction for a pipe between a worker thread and the server
>> @@ -2683,6 +2910,8 @@ class runQueuePipe():
>> except ValueError as e:
>> bb.msg.fatal("RunQueue", "failed load pickle
>> '%s': '%s'" % (e, self.queue[7:index]))
>> bb.event.fire_from_worker(event, self.d)
>> + if isinstance(event, taskUniHashUpdate):
>> +
>> self.rqexec.updated_taskhash_queue.append((event.taskid,
>> event.unihash)) found = True self.queue = self.queue[index+8:]
>> index = self.queue.find(b"</event>")
>> diff --git a/bitbake/lib/bb/server/__init__.py
>> b/bitbake/lib/bb/server/__init__.py index 5a3fba9..b6f7513 100644
>> --- a/bitbake/lib/bb/server/__init__.py
>> +++ b/bitbake/lib/bb/server/__init__.py
>> @@ -5,17 +5,5 @@
>> # Copyright (C) 2006 - 2008 Richard Purdie
>> # Copyright (C) 2013 Alexandru Damian
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. -
>> -
>> diff --git a/bitbake/lib/bb/server/process.py
>> b/bitbake/lib/bb/server/process.py index 4e0d9c2..69aae62 100644
>> --- a/bitbake/lib/bb/server/process.py
>> +++ b/bitbake/lib/bb/server/process.py
>> @@ -3,18 +3,8 @@
>> #
>> # Copyright (C) 2010 Bob Foerster <robert@erafx.com>
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> """
>> This module implements a multiprocessing.Process based server
>> for bitbake. @@ -130,6 +120,7 @@ class
>> ProcessServer(multiprocessing.Process):
>> bb.utils.set_process_name("Cooker")
>> ready = []
>> + newconnections = []
>>
>> self.controllersock = False
>> fds = [self.sock]
>> @@ -138,37 +129,48 @@ class ProcessServer(multiprocessing.Process):
>> print("Entering server connection loop")
>>
>> def disconnect_client(self, fds):
>> - if not self.haveui:
>> - return
>> print("Disconnecting Client")
>> - fds.remove(self.controllersock)
>> - fds.remove(self.command_channel)
>> - bb.event.unregister_UIHhandler(self.event_handle, True)
>> - self.command_channel_reply.writer.close()
>> - self.event_writer.writer.close()
>> - del self.event_writer
>> - self.controllersock.close()
>> - self.controllersock = False
>> - self.haveui = False
>> - self.lastui = time.time()
>> - self.cooker.clientComplete()
>> - if self.timeout is None:
>> + if self.controllersock:
>> + fds.remove(self.controllersock)
>> + self.controllersock.close()
>> + self.controllersock = False
>> + if self.haveui:
>> + fds.remove(self.command_channel)
>> + bb.event.unregister_UIHhandler(self.event_handle,
>> True)
>> + self.command_channel_reply.writer.close()
>> + self.event_writer.writer.close()
>> + self.command_channel.close()
>> + self.command_channel = False
>> + del self.event_writer
>> + self.lastui = time.time()
>> + self.cooker.clientComplete()
>> + self.haveui = False
>> + ready = select.select(fds,[],[],0)[0]
>> + if newconnections:
>> + print("Starting new client")
>> + conn = newconnections.pop(-1)
>> + fds.append(conn)
>> + self.controllersock = conn
>> + elif self.timeout is None and not ready:
>> print("No timeout, exiting.")
>> self.quit = True
>>
>> while not self.quit:
>> if self.sock in ready:
>> - self.controllersock, address = self.sock.accept()
>> - if self.haveui:
>> - print("Dropping connection attempt as we have a
>> UI %s" % (str(ready)))
>> - self.controllersock.close()
>> - else:
>> - print("Accepting %s" % (str(ready)))
>> - fds.append(self.controllersock)
>> + while select.select([self.sock],[],[],0)[0]:
>> + controllersock, address = self.sock.accept()
>> + if self.controllersock:
>> + print("Queuing %s (%s)" % (str(ready),
>> str(newconnections)))
>> + newconnections.append(controllersock)
>> + else:
>> + print("Accepting %s (%s)" % (str(ready),
>> str(newconnections)))
>> + self.controllersock = controllersock
>> + fds.append(controllersock)
>> if self.controllersock in ready:
>> try:
>> - print("Connecting Client")
>> + print("Processing Client")
>> ui_fds = recvfds(self.controllersock, 3)
>> + print("Connecting Client")
>>
>> # Where to write events to
>> writer = ConnectionWriter(ui_fds[0])
>> @@ -239,6 +241,12 @@ class ProcessServer(multiprocessing.Process):
>> while not lock:
>> with bb.utils.timeout(3):
>> lock = bb.utils.lockfile(lockfile, shared=False,
>> retry=False, block=True)
>> + if lock:
>> + # We hold the lock so we can remove the file
>> (hide stale pid data)
>> + bb.utils.remove(lockfile)
>> + bb.utils.unlockfile(lock)
>> + return
>> +
>> if not lock:
>> # Some systems may not have lsof available
>> procs = None
>> @@ -259,10 +267,6 @@ class ProcessServer(multiprocessing.Process):
>> if procs:
>> msg += ":\n%s" % str(procs)
>> print(msg)
>> - return
>> - # We hold the lock so we can remove the file (hide stale pid
>> data)
>> - bb.utils.remove(lockfile)
>> - bb.utils.unlockfile(lock)
>>
>> def idle_commands(self, delay, fds=None):
>> nextsleep = delay
>> @@ -398,36 +402,45 @@ class BitBakeServer(object):
>> os.close(self.readypipein)
>>
>> ready = ConnectionReader(self.readypipe)
>> - r = ready.poll(30)
>> + r = ready.poll(5)
>> + if not r:
>> + bb.note("Bitbake server didn't start within 5 seconds,
>> waiting for 90")
>> + r = ready.poll(90)
>> if r:
>> try:
>> r = ready.get()
>> except EOFError:
>> # Trap the child exitting/closing the pipe and error
>> out r = None
>> - if not r or r != "ready":
>> + if not r or r[0] != "r":
>> ready.close()
>> - bb.error("Unable to start bitbake server")
>> + bb.error("Unable to start bitbake server (%s)" % str(r))
>> if os.path.exists(logfile):
>> logstart_re = re.compile(self.start_log_format %
>> ('([0-9]+)', '([0-9-]+ [0-9:.]+)')) started = False
>> lines = []
>> + lastlines = []
>> with open(logfile, "r") as f:
>> for line in f:
>> if started:
>> lines.append(line)
>> else:
>> + lastlines.append(line)
>> res = logstart_re.match(line.rstrip())
>> if res:
>> ldatetime =
>> datetime.datetime.strptime(res.group(2),
>> self.start_log_datetime_format) if ldatetime >= startdatetime:
>> started = True lines.append(line)
>> + if len(lastlines) > 60:
>> + lastlines = lastlines[-60:]
>> if lines:
>> - if len(lines) > 10:
>> - bb.error("Last 10 lines of server log for
>> this session (%s):\n%s" % (logfile, "".join(lines[-10:])))
>> + if len(lines) > 60:
>> + bb.error("Last 60 lines of server log for
>> this session (%s):\n%s" % (logfile, "".join(lines[-60:]))) else:
>> bb.error("Server log for this session
>> (%s):\n%s" % (logfile, "".join(lines)))
>> + elif lastlines:
>> + bb.error("Server didn't start, last 60
>> loglines (%s):\n%s" % (logfile, "".join(lastlines))) else:
>> bb.error("%s doesn't exist" % logfile)
>>
>> @@ -437,17 +450,24 @@ class BitBakeServer(object):
>>
>> def _startServer(self):
>> print(self.start_log_format % (os.getpid(),
>> datetime.datetime.now().strftime(self.start_log_datetime_format)))
>> + sys.stdout.flush()
>> +
>> server = ProcessServer(self.bitbake_lock, self.sock,
>> self.sockname)
>> self.configuration.setServerRegIdleCallback(server.register_idle_function)
>> os.close(self.readypipe) writer = ConnectionWriter(self.readypipein)
>> - self.cooker = bb.cooker.BBCooker(self.configuration,
>> self.featureset)
>> - writer.send("ready")
>> + try:
>> + self.cooker = bb.cooker.BBCooker(self.configuration,
>> self.featureset)
>> + except bb.BBHandledException:
>> + return None
>> + writer.send("r")
>> writer.close()
>> server.cooker = self.cooker
>> server.server_timeout = self.configuration.server_timeout
>> server.xmlrpcinterface = self.configuration.xmlrpcinterface
>> print("Started bitbake server pid %d" % os.getpid())
>> + sys.stdout.flush()
>> +
>> server.start()
>>
>> def connectProcessServer(sockname, featureset):
>> @@ -459,10 +479,20 @@ def connectProcessServer(sockname, featureset):
>> readfd = writefd = readfd1 = writefd1 = readfd2 = writefd2 = None
>> eq = command_chan_recv = command_chan = None
>>
>> + sock.settimeout(10)
>> +
>> try:
>> try:
>> os.chdir(os.path.dirname(sockname))
>> - sock.connect(os.path.basename(sockname))
>> + finished = False
>> + while not finished:
>> + try:
>> + sock.connect(os.path.basename(sockname))
>> + finished = True
>> + except IOError as e:
>> + if e.errno == errno.EWOULDBLOCK:
>> + pass
>> + raise
>> finally:
>> os.chdir(cwd)
>>
>> @@ -493,7 +523,8 @@ def connectProcessServer(sockname, featureset):
>> command_chan.close()
>> for i in [writefd, readfd1, writefd2]:
>> try:
>> - os.close(i)
>> + if i:
>> + os.close(i)
>> except OSError:
>> pass
>> sock.close()
>> diff --git a/bitbake/lib/bb/server/xmlrpcclient.py
>> b/bitbake/lib/bb/server/xmlrpcclient.py index 4661a9e..c054c3c 100644
>> --- a/bitbake/lib/bb/server/xmlrpcclient.py
>> +++ b/bitbake/lib/bb/server/xmlrpcclient.py
>> @@ -4,18 +4,8 @@
>> # Copyright (C) 2006 - 2007 Michael 'Mickey' Lauer
>> # Copyright (C) 2006 - 2008 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import os
>> import sys
>> diff --git a/bitbake/lib/bb/server/xmlrpcserver.py
>> b/bitbake/lib/bb/server/xmlrpcserver.py index 875b128..54fa32f 100644
>> --- a/bitbake/lib/bb/server/xmlrpcserver.py
>> +++ b/bitbake/lib/bb/server/xmlrpcserver.py
>> @@ -4,18 +4,8 @@
>> # Copyright (C) 2006 - 2007 Michael 'Mickey' Lauer
>> # Copyright (C) 2006 - 2008 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import os
>> import sys
>> diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py
>> index fdbb2a3..a4bb1ff 100644
>> --- a/bitbake/lib/bb/siggen.py
>> +++ b/bitbake/lib/bb/siggen.py
>> @@ -1,3 +1,7 @@
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> import hashlib
>> import logging
>> import os
>> @@ -8,6 +12,8 @@ import bb.data
>> import difflib
>> import simplediff
>> from bb.checksum import FileChecksumCache
>> +from bb import runqueue
>> +import hashserv
>>
>> logger = logging.getLogger('BitBake.SigGen')
>>
>> @@ -37,12 +43,18 @@ class SignatureGenerator(object):
>> self.runtaskdeps = {}
>> self.file_checksum_values = {}
>> self.taints = {}
>> + self.unitaskhashes = {}
>> + self.setscenetasks = {}
>>
>> def finalise(self, fn, d, varient):
>> return
>>
>> - def get_taskhash(self, fn, task, deps, dataCache):
>> - return "0"
>> + def get_unihash(self, tid):
>> + return self.taskhash[tid]
>> +
>> + def get_taskhash(self, tid, deps, dataCache):
>> + self.taskhash[tid] =
>> hashlib.sha256(tid.encode("utf-8")).hexdigest()
>> + return self.taskhash[tid]
>>
>> def writeout_file_checksum_cache(self):
>> """Write/update the file checksum cache onto disk"""
>> @@ -64,14 +76,25 @@ class SignatureGenerator(object):
>> return
>>
>> def get_taskdata(self):
>> - return (self.runtaskdeps, self.taskhash,
>> self.file_checksum_values, self.taints, self.basehash)
>> + return (self.runtaskdeps, self.taskhash,
>> self.file_checksum_values, self.taints, self.basehash,
>> self.unitaskhashes, self.setscenetasks) def set_taskdata(self, data):
>> - self.runtaskdeps, self.taskhash, self.file_checksum_values,
>> self.taints, self.basehash = data
>> + self.runtaskdeps, self.taskhash, self.file_checksum_values,
>> self.taints, self.basehash, self.unitaskhashes, self.setscenetasks =
>> data def reset(self, data):
>> self.__init__(data)
>>
>> + def get_taskhashes(self):
>> + return self.taskhash, self.unitaskhashes
>> +
>> + def set_taskhashes(self, hashes):
>> + self.taskhash, self.unitaskhashes = hashes
>> +
>> + def save_unitaskhashes(self):
>> + return
>> +
>> + def set_setscene_tasks(self, setscene_tasks):
>> + return
>>
>> class SignatureGeneratorBasic(SignatureGenerator):
>> """
>> @@ -87,7 +110,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
>> self.taints = {}
>> self.gendeps = {}
>> self.lookupcache = {}
>> - self.pkgnameextract = re.compile("(?P<fn>.*)\..*")
>> + self.setscenetasks = {}
>> self.basewhitelist =
>> set((data.getVar("BB_HASHBASE_WHITELIST") or "").split())
>> self.taskwhitelist = None self.init_rundepcheck(data)
>> @@ -98,6 +121,9 @@ class SignatureGeneratorBasic(SignatureGenerator):
>> else:
>> self.checksum_cache = None
>>
>> + self.unihash_cache = bb.cache.SimpleCache("1")
>> + self.unitaskhashes = self.unihash_cache.init_cache(data,
>> "bb_unihashes.dat", {}) +
>> def init_rundepcheck(self, data):
>> self.taskwhitelist = data.getVar("BB_HASHTASK_WHITELIST") or
>> None if self.taskwhitelist:
>> @@ -113,10 +139,16 @@ class
>> SignatureGeneratorBasic(SignatureGenerator): taskdeps, basehash =
>> bb.data.generate_dependency_hash(tasklist, gendeps, lookupcache,
>> self.basewhitelist, fn) for task in tasklist:
>> - k = fn + "." + task
>> - if not ignore_mismatch and k in self.basehash and
>> self.basehash[k] != basehash[k]:
>> - bb.error("When reparsing %s, the basehash value
>> changed from %s to %s. The metadata is not deterministic and this
>> needs to be fixed." % (k, self.basehash[k], basehash[k]))
>> - self.basehash[k] = basehash[k]
>> + tid = fn + ":" + task
>> + if not ignore_mismatch and tid in self.basehash and
>> self.basehash[tid] != basehash[tid]:
>> + bb.error("When reparsing %s, the basehash value
>> changed from %s to %s. The metadata is not deterministic and this
>> needs to be fixed." % (tid, self.basehash[tid], basehash[tid]))
>> + bb.error("The following commands may help:")
>> + cmd = "$ bitbake %s -c%s" % (d.getVar('PN'), task)
>> + # Make sure sigdata is dumped before run printdiff
>> + bb.error("%s -Snone" % cmd)
>> + bb.error("Then:")
>> + bb.error("%s -Sprintdiff\n" % cmd)
>> + self.basehash[tid] = basehash[tid]
>>
>> self.taskdeps[fn] = taskdeps
>> self.gendeps[fn] = gendeps
>> @@ -124,6 +156,9 @@ class SignatureGeneratorBasic(SignatureGenerator):
>>
>> return taskdeps
>>
>> + def set_setscene_tasks(self, setscene_tasks):
>> + self.setscenetasks = setscene_tasks
>> +
>> def finalise(self, fn, d, variant):
>>
>> mc = d.getVar("__BBMULTICONFIG", False) or ""
>> @@ -143,7 +178,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
>> # self.dump_sigtask(fn, task, d.getVar("STAMP"), False)
>>
>> for task in taskdeps:
>> - d.setVar("BB_BASEHASH_task-%s" % task, self.basehash[fn
>> + "." + task])
>> + d.setVar("BB_BASEHASH_task-%s" % task, self.basehash[fn
>> + ":" + task])
>> def rundep_check(self, fn, recipename, task, dep, depname,
>> dataCache): # Return True if we should keep the dependency, False to
>> drop it @@ -163,31 +198,26 @@ class
>> SignatureGeneratorBasic(SignatureGenerator): pass
>> return taint
>>
>> - def get_taskhash(self, fn, task, deps, dataCache):
>> + def get_taskhash(self, tid, deps, dataCache):
>>
>> - mc = ''
>> - if fn.startswith('multiconfig:'):
>> - mc = fn.split(':')[1]
>> - k = fn + "." + task
>> + (mc, _, task, fn) = bb.runqueue.split_tid_mcfn(tid)
>>
>> - data = dataCache.basetaskhash[k]
>> - self.basehash[k] = data
>> - self.runtaskdeps[k] = []
>> - self.file_checksum_values[k] = []
>> + data = dataCache.basetaskhash[tid]
>> + self.basehash[tid] = data
>> + self.runtaskdeps[tid] = []
>> + self.file_checksum_values[tid] = []
>> recipename = dataCache.pkg_fn[fn]
>> for dep in sorted(deps, key=clean_basepath):
>> - pkgname = self.pkgnameextract.search(dep).group('fn')
>> - if mc:
>> - depmc = pkgname.split(':')[1]
>> - if mc != depmc:
>> - continue
>> - depname = dataCache.pkg_fn[pkgname]
>> + (depmc, _, deptaskname, depfn) =
>> bb.runqueue.split_tid_mcfn(dep)
>> + if mc != depmc:
>> + continue
>> + depname = dataCache.pkg_fn[depfn]
>> if not self.rundep_check(fn, recipename, task, dep,
>> depname, dataCache): continue
>> if dep not in self.taskhash:
>> bb.fatal("%s is not in taskhash, caller isn't
>> calling in dependency order?" % dep)
>> - data = data + self.taskhash[dep]
>> - self.runtaskdeps[k].append(dep)
>> + data = data + self.get_unihash(dep)
>> + self.runtaskdeps[tid].append(dep)
>>
>> if task in dataCache.file_checksums[fn]:
>> if self.checksum_cache:
>> @@ -195,7 +225,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
>> else:
>> checksums =
>> bb.fetch2.get_file_checksums(dataCache.file_checksums[fn][task],
>> recipename) for (f,cs) in checksums:
>> - self.file_checksum_values[k].append((f,cs))
>> + self.file_checksum_values[tid].append((f,cs))
>> if cs:
>> data = data + cs
>>
>> @@ -205,16 +235,16 @@ class
>> SignatureGeneratorBasic(SignatureGenerator): import uuid
>> taint = str(uuid.uuid4())
>> data = data + taint
>> - self.taints[k] = "nostamp:" + taint
>> + self.taints[tid] = "nostamp:" + taint
>>
>> taint = self.read_taint(fn, task, dataCache.stamp[fn])
>> if taint:
>> data = data + taint
>> - self.taints[k] = taint
>> - logger.warning("%s is tainted from a forced run" % k)
>> + self.taints[tid] = taint
>> + logger.warning("%s is tainted from a forced run" % tid)
>>
>> - h = hashlib.md5(data.encode("utf-8")).hexdigest()
>> - self.taskhash[k] = h
>> + h = hashlib.sha256(data.encode("utf-8")).hexdigest()
>> + self.taskhash[tid] = h
>> #d.setVar("BB_TASKHASH_task-%s" % task, taskhash[task])
>> return h
>>
>> @@ -227,17 +257,20 @@ class
>> SignatureGeneratorBasic(SignatureGenerator):
>> bb.fetch2.fetcher_parse_save() bb.fetch2.fetcher_parse_done()
>>
>> + def save_unitaskhashes(self):
>> + self.unihash_cache.save(self.unitaskhashes)
>> +
>> def dump_sigtask(self, fn, task, stampbase, runtime):
>>
>> - k = fn + "." + task
>> + tid = fn + ":" + task
>> referencestamp = stampbase
>> if isinstance(runtime, str) and
>> runtime.startswith("customfile"): sigfile = stampbase
>> referencestamp = runtime[11:]
>> - elif runtime and k in self.taskhash:
>> - sigfile = stampbase + "." + task + ".sigdata" + "." +
>> self.taskhash[k]
>> + elif runtime and tid in self.taskhash:
>> + sigfile = stampbase + "." + task + ".sigdata" + "." +
>> self.get_unihash(tid) else:
>> - sigfile = stampbase + "." + task + ".sigbasedata" + "."
>> + self.basehash[k]
>> + sigfile = stampbase + "." + task + ".sigbasedata" + "."
>> + self.basehash[tid]
>> bb.utils.mkdirhier(os.path.dirname(sigfile))
>>
>> @@ -246,7 +279,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
>> data['basewhitelist'] = self.basewhitelist
>> data['taskwhitelist'] = self.taskwhitelist
>> data['taskdeps'] = self.taskdeps[fn][task]
>> - data['basehash'] = self.basehash[k]
>> + data['basehash'] = self.basehash[tid]
>> data['gendeps'] = {}
>> data['varvals'] = {}
>> data['varvals'][task] = self.lookupcache[fn][task]
>> @@ -256,30 +289,31 @@ class
>> SignatureGeneratorBasic(SignatureGenerator): data['gendeps'][dep] =
>> self.gendeps[fn][dep] data['varvals'][dep] = self.lookupcache[fn][dep]
>>
>> - if runtime and k in self.taskhash:
>> - data['runtaskdeps'] = self.runtaskdeps[k]
>> - data['file_checksum_values'] = [(os.path.basename(f),
>> cs) for f,cs in self.file_checksum_values[k]]
>> + if runtime and tid in self.taskhash:
>> + data['runtaskdeps'] = self.runtaskdeps[tid]
>> + data['file_checksum_values'] = [(os.path.basename(f),
>> cs) for f,cs in self.file_checksum_values[tid]] data['runtaskhashes']
>> = {} for dep in data['runtaskdeps']:
>> - data['runtaskhashes'][dep] = self.taskhash[dep]
>> - data['taskhash'] = self.taskhash[k]
>> + data['runtaskhashes'][dep] = self.get_unihash(dep)
>> + data['taskhash'] = self.taskhash[tid]
>> + data['unihash'] = self.get_unihash(tid)
>>
>> taint = self.read_taint(fn, task, referencestamp)
>> if taint:
>> data['taint'] = taint
>>
>> - if runtime and k in self.taints:
>> - if 'nostamp:' in self.taints[k]:
>> - data['taint'] = self.taints[k]
>> + if runtime and tid in self.taints:
>> + if 'nostamp:' in self.taints[tid]:
>> + data['taint'] = self.taints[tid]
>>
>> computed_basehash = calc_basehash(data)
>> - if computed_basehash != self.basehash[k]:
>> - bb.error("Basehash mismatch %s versus %s for %s" %
>> (computed_basehash, self.basehash[k], k))
>> - if runtime and k in self.taskhash:
>> + if computed_basehash != self.basehash[tid]:
>> + bb.error("Basehash mismatch %s versus %s for %s" %
>> (computed_basehash, self.basehash[tid], tid))
>> + if runtime and tid in self.taskhash:
>> computed_taskhash = calc_taskhash(data)
>> - if computed_taskhash != self.taskhash[k]:
>> - bb.error("Taskhash mismatch %s versus %s for %s" %
>> (computed_taskhash, self.taskhash[k], k))
>> - sigfile = sigfile.replace(self.taskhash[k],
>> computed_taskhash)
>> + if computed_taskhash != self.taskhash[tid]:
>> + bb.error("Taskhash mismatch %s versus %s for %s" %
>> (computed_taskhash, self.taskhash[tid], tid))
>> + sigfile = sigfile.replace(self.taskhash[tid],
>> computed_taskhash)
>> fd, tmpfile = tempfile.mkstemp(dir=os.path.dirname(sigfile),
>> prefix="sigtask.") try:
>> @@ -299,30 +333,34 @@ class
>> SignatureGeneratorBasic(SignatureGenerator): if fn in self.taskdeps:
>> for task in self.taskdeps[fn]:
>> tid = fn + ":" + task
>> - (mc, _, _) = bb.runqueue.split_tid(tid)
>> - k = fn + "." + task
>> - if k not in self.taskhash:
>> + mc = bb.runqueue.mc_from_tid(tid)
>> + if tid not in self.taskhash:
>> continue
>> - if dataCaches[mc].basetaskhash[k] !=
>> self.basehash[k]:
>> - bb.error("Bitbake's cached basehash does not
>> match the one we just generated (%s)!" % k)
>> - bb.error("The mismatched hashes were %s and %s"
>> % (dataCaches[mc].basetaskhash[k], self.basehash[k]))
>> + if dataCaches[mc].basetaskhash[tid] !=
>> self.basehash[tid]:
>> + bb.error("Bitbake's cached basehash does not
>> match the one we just generated (%s)!" % tid)
>> + bb.error("The mismatched hashes were %s and %s"
>> % (dataCaches[mc].basetaskhash[tid], self.basehash[tid]))
>> self.dump_sigtask(fn, task, dataCaches[mc].stamp[fn], True)
>> class SignatureGeneratorBasicHash(SignatureGeneratorBasic):
>> name = "basichash"
>>
>> + def get_stampfile_hash(self, tid):
>> + if tid in self.taskhash:
>> + return self.taskhash[tid]
>> +
>> + # If task is not in basehash, then error
>> + return self.basehash[tid]
>> +
>> def stampfile(self, stampbase, fn, taskname, extrainfo,
>> clean=False): if taskname != "do_setscene" and
>> taskname.endswith("_setscene"):
>> - k = fn + "." + taskname[:-9]
>> + tid = fn + ":" + taskname[:-9]
>> else:
>> - k = fn + "." + taskname
>> + tid = fn + ":" + taskname
>> if clean:
>> h = "*"
>> - elif k in self.taskhash:
>> - h = self.taskhash[k]
>> else:
>> - # If k is not in basehash, then error
>> - h = self.basehash[k]
>> + h = self.get_stampfile_hash(tid)
>> +
>> return ("%s.%s.%s.%s" % (stampbase, taskname, h,
>> extrainfo)).rstrip('.')
>> def stampcleanmask(self, stampbase, fn, taskname, extrainfo):
>> @@ -332,6 +370,172 @@ class
>> SignatureGeneratorBasicHash(SignatureGeneratorBasic):
>> bb.note("Tainting hash to force rebuild of task %s, %s" % (fn, task))
>> bb.build.write_taint(task, d, fn)
>> +class SignatureGeneratorUniHashMixIn(object):
>> + def get_taskdata(self):
>> + return (self.server, self.method) + super().get_taskdata()
>> +
>> + def set_taskdata(self, data):
>> + self.server, self.method = data[:2]
>> + super().set_taskdata(data[2:])
>> +
>> + def client(self):
>> + if getattr(self, '_client', None) is None:
>> + self._client = hashserv.create_client(self.server)
>> + return self._client
>> +
>> + def __get_task_unihash_key(self, tid):
>> + # TODO: The key only *needs* to be the taskhash, the tid is
>> just
>> + # convenient
>> + return '%s:%s' % (tid.rsplit("/", 1)[1], self.taskhash[tid])
>> +
>> + def get_stampfile_hash(self, tid):
>> + if tid in self.taskhash:
>> + # If a unique hash is reported, use it as the stampfile
>> hash. This
>> + # ensures that if a task won't be re-run if the taskhash
>> changes,
>> + # but it would result in the same output hash
>> + unihash =
>> self.unitaskhashes.get(self.__get_task_unihash_key(tid), None)
>> + if unihash is not None:
>> + return unihash
>> +
>> + return super().get_stampfile_hash(tid)
>> +
>> + def set_unihash(self, tid, unihash):
>> + self.unitaskhashes[self.__get_task_unihash_key(tid)] =
>> unihash +
>> + def get_unihash(self, tid):
>> + taskhash = self.taskhash[tid]
>> +
>> + # If its not a setscene task we can return
>> + if self.setscenetasks and tid not in self.setscenetasks:
>> + return taskhash
>> +
>> + key = self.__get_task_unihash_key(tid)
>> +
>> + # TODO: This cache can grow unbounded. It probably only
>> needs to keep
>> + # for each task
>> + unihash = self.unitaskhashes.get(key, None)
>> + if unihash is not None:
>> + return unihash
>> +
>> + # In the absence of being able to discover a unique hash
>> from the
>> + # server, make it be equivalent to the taskhash. The unique
>> "hash" only
>> + # really needs to be a unique string (not even necessarily a
>> hash), but
>> + # making it match the taskhash has a few advantages:
>> + #
>> + # 1) All of the sstate code that assumes hashes can be the
>> same
>> + # 2) It provides maximal compatibility with builders that
>> don't use
>> + # an equivalency server
>> + # 3) The value is easy for multiple independent builders to
>> derive the
>> + # same unique hash from the same input. This means that
>> if the
>> + # independent builders find the same taskhash, but it
>> isn't reported
>> + # to the server, there is a better chance that they will
>> agree on
>> + # the unique hash.
>> + unihash = taskhash
>> +
>> + try:
>> + data = self.client().get_unihash(self.method,
>> self.taskhash[tid])
>> + if data:
>> + unihash = data
>> + # A unique hash equal to the taskhash is not very
>> interesting,
>> + # so it is reported it at debug level 2. If they
>> differ, that
>> + # is much more interesting, so it is reported at
>> debug level 1
>> + bb.debug((1, 2)[unihash == taskhash], 'Found unihash
>> %s in place of %s for %s from %s' % (unihash, taskhash, tid,
>> self.server))
>> + else:
>> + bb.debug(2, 'No reported unihash for %s:%s from %s'
>> % (tid, taskhash, self.server))
>> + except hashserv.client.HashConnectionError as e:
>> + bb.warn('Error contacting Hash Equivalence Server %s:
>> %s' % (self.server, str(e))) +
>> + self.unitaskhashes[key] = unihash
>> + return unihash
>> +
>> + def report_unihash(self, path, task, d):
>> + import importlib
>> +
>> + taskhash = d.getVar('BB_TASKHASH')
>> + unihash = d.getVar('BB_UNIHASH')
>> + report_taskdata =
>> d.getVar('SSTATE_HASHEQUIV_REPORT_TASKDATA') == '1'
>> + tempdir = d.getVar('T')
>> + fn = d.getVar('BB_FILENAME')
>> + tid = fn + ':do_' + task
>> + key = tid.rsplit("/", 1)[1] + ':' + taskhash
>> +
>> + if self.setscenetasks and tid not in self.setscenetasks:
>> + return
>> +
>> + # Sanity checks
>> + cache_unihash = self.unitaskhashes.get(key, None)
>> + if cache_unihash is None:
>> + bb.fatal('%s not in unihash cache. Please report this
>> error' % key) +
>> + if cache_unihash != unihash:
>> + bb.fatal("Cache unihash %s doesn't match BB_UNIHASH %s"
>> % (cache_unihash, unihash)) +
>> + sigfile = None
>> + sigfile_name = "depsig.do_%s.%d" % (task, os.getpid())
>> + sigfile_link = "depsig.do_%s" % task
>> +
>> + try:
>> + sigfile = open(os.path.join(tempdir, sigfile_name),
>> 'w+b') +
>> + locs = {'path': path, 'sigfile': sigfile, 'task': task,
>> 'd': d} +
>> + if "." in self.method:
>> + (module, method) = self.method.rsplit('.', 1)
>> + locs['method'] =
>> getattr(importlib.import_module(module), method)
>> + outhash = bb.utils.better_eval('method(path,
>> sigfile, task, d)', locs)
>> + else:
>> + outhash = bb.utils.better_eval(self.method + '(path,
>> sigfile, task, d)', locs) +
>> + try:
>> + extra_data = {}
>> +
>> + owner = d.getVar('SSTATE_HASHEQUIV_OWNER')
>> + if owner:
>> + extra_data['owner'] = owner
>> +
>> + if report_taskdata:
>> + sigfile.seek(0)
>> +
>> + extra_data['PN'] = d.getVar('PN')
>> + extra_data['PV'] = d.getVar('PV')
>> + extra_data['PR'] = d.getVar('PR')
>> + extra_data['task'] = task
>> + extra_data['outhash_siginfo'] =
>> sigfile.read().decode('utf-8') +
>> + data = self.client().report_unihash(taskhash,
>> self.method, outhash, unihash, extra_data)
>> + new_unihash = data['unihash']
>> +
>> + if new_unihash != unihash:
>> + bb.debug(1, 'Task %s unihash changed %s -> %s by
>> server %s' % (taskhash, unihash, new_unihash, self.server))
>> + bb.event.fire(bb.runqueue.taskUniHashUpdate(fn +
>> ':do_' + task, new_unihash), d)
>> + else:
>> + bb.debug(1, 'Reported task %s as unihash %s to
>> %s' % (taskhash, unihash, self.server))
>> + except hashserv.client.HashConnectionError as e:
>> + bb.warn('Error contacting Hash Equivalence Server
>> %s: %s' % (self.server, str(e)))
>> + finally:
>> + if sigfile:
>> + sigfile.close()
>> +
>> + sigfile_link_path = os.path.join(tempdir,
>> sigfile_link)
>> + bb.utils.remove(sigfile_link_path)
>> +
>> + try:
>> + os.symlink(sigfile_name, sigfile_link_path)
>> + except OSError:
>> + pass
>> +
>> +
>> +#
>> +# Dummy class used for bitbake-selftest
>> +#
>> +class
>> SignatureGeneratorTestEquivHash(SignatureGeneratorUniHashMixIn,
>> SignatureGeneratorBasicHash):
>> + name = "TestEquivHash"
>> + def init_rundepcheck(self, data):
>> + super().init_rundepcheck(data)
>> + self.server = data.getVar('BB_HASHSERVE')
>> + self.method = "sstate_output_hash"
>> +
>> +
>> def dump_this_task(outfile, d):
>> import bb.parse
>> fn = d.getVar("BB_FILENAME")
>> @@ -392,13 +596,13 @@ def list_inline_diff(oldlist, newlist,
>> colors=None):
>> def clean_basepath(a):
>> mc = None
>> - if a.startswith("multiconfig:"):
>> + if a.startswith("mc:"):
>> _, mc, a = a.split(":", 2)
>> b = a.rsplit("/", 2)[1] + '/' + a.rsplit("/", 2)[2]
>> if a.startswith("virtual:"):
>> b = b + ":" + a.rsplit(":", 1)[0]
>> if mc:
>> - b = b + ":multiconfig:" + mc
>> + b = b + ":mc:" + mc
>> return b
>>
>> def clean_basepaths(a):
>> @@ -623,6 +827,10 @@ def compare_sigfiles(a, b, recursecb=None,
>> color=False, collapsed=False): a_taint = a_data.get('taint', None)
>> b_taint = b_data.get('taint', None)
>> if a_taint != b_taint:
>> + if a_taint and a_taint.startswith('nostamp:'):
>> + a_taint = a_taint.replace('nostamp:', 'nostamp(uuid4):')
>> + if b_taint and b_taint.startswith('nostamp:'):
>> + b_taint = b_taint.replace('nostamp:', 'nostamp(uuid4):')
>> output.append(color_format("{color_title}Taint (by
>> forced/invalidated task) changed{color_default} from %s to %s") %
>> (a_taint, b_taint)) return output
>> @@ -642,7 +850,7 @@ def calc_basehash(sigdata):
>> if val is not None:
>> basedata = basedata + str(val)
>>
>> - return hashlib.md5(basedata.encode("utf-8")).hexdigest()
>> + return hashlib.sha256(basedata.encode("utf-8")).hexdigest()
>>
>> def calc_taskhash(sigdata):
>> data = sigdata['basehash']
>> @@ -660,7 +868,7 @@ def calc_taskhash(sigdata):
>> else:
>> data = data + sigdata['taint']
>>
>> - return hashlib.md5(data.encode("utf-8")).hexdigest()
>> + return hashlib.sha256(data.encode("utf-8")).hexdigest()
>>
>>
>> def dump_sigfile(a):
>> @@ -695,7 +903,11 @@ def dump_sigfile(a):
>> output.append("Hash for dependent task %s is %s" % (dep,
>> a_data['runtaskhashes'][dep]))
>> if 'taint' in a_data:
>> - output.append("Tainted (by forced/invalidated task): %s" %
>> a_data['taint'])
>> + if a_data['taint'].startswith('nostamp:'):
>> + msg = a_data['taint'].replace('nostamp:',
>> 'nostamp(uuid4):')
>> + else:
>> + msg = a_data['taint']
>> + output.append("Tainted (by forced/invalidated task): %s" %
>> msg)
>> if 'task' in a_data:
>> computed_basehash = calc_basehash(a_data)
>> diff --git a/bitbake/lib/bb/taskdata.py b/bitbake/lib/bb/taskdata.py
>> index 94e822c..8c25e09 100644
>> --- a/bitbake/lib/bb/taskdata.py
>> +++ b/bitbake/lib/bb/taskdata.py
>> @@ -1,6 +1,3 @@
>> -#!/usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake 'TaskData' implementation
>>
>> @@ -10,18 +7,8 @@ Task data collection and handling
>>
>> # Copyright (C) 2006 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import logging
>> import re
>> @@ -93,7 +80,7 @@ class TaskData:
>> def add_mcdepends(task):
>> for dep in task_deps['mcdepends'][task].split():
>> if len(dep.split(':')) != 5:
>> - bb.msg.fatal("TaskData", "Error for %s:%s[%s],
>> multiconfig dependency %s does not contain exactly four ':'
>> characters.\n Task '%s' should be specified in the form
>> 'multiconfig:fromMC:toMC:packagename:task'" % (fn, task, 'mcdepends',
>> dep, 'mcdepends'))
>> + bb.msg.fatal("TaskData", "Error for %s:%s[%s],
>> multiconfig dependency %s does not contain exactly four ':'
>> characters.\n Task '%s' should be specified in the form
>> 'mc:fromMC:toMC:packagename:task'" % (fn, task, 'mcdepends', dep,
>> 'mcdepends')) if dep not in self.mcdepends:
>> self.mcdepends.append(dep) diff --git
>> a/bitbake/lib/bb/tests/codeparser.py
>> b/bitbake/lib/bb/tests/codeparser.py index e30e78c..826a2d2 100644
>> --- a/bitbake/lib/bb/tests/codeparser.py +++
>> b/bitbake/lib/bb/tests/codeparser.py @@ -1,23 +1,10 @@ -#
>> ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4;
>> indent-tabs-mode: nil -*- #
>> # BitBake Test for codeparser.py
>> #
>> # Copyright (C) 2010 Chris Larson
>> # Copyright (C) 2012 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>>
>> import unittest
>> @@ -123,6 +110,13 @@ ${D}${libdir}/pkgconfig/*.pc
>> self.parseExpression("sed -i -e 's:IP{:I${:g' $pc")
>> self.assertExecs(set(["sed"]))
>>
>> + def test_parameter_expansion_modifiers(self):
>> + # - and + are also valid modifiers for parameter expansion,
>> but are
>> + # valid characters in bitbake variable names, so are not
>> included here
>> + for i in ('=', ':-', ':=', '?', ':?', ':+', '#', '%', '##',
>> '%%'):
>> + name = "foo%sbar" % i
>> + self.parseExpression("${%s}" % name)
>> + self.assertNotIn(name, self.references)
>>
>> def test_until(self):
>> self.parseExpression("until false; do echo true; done")
>> diff --git a/bitbake/lib/bb/tests/cooker.py
>> b/bitbake/lib/bb/tests/cooker.py index 2b44236..090916e 100644
>> --- a/bitbake/lib/bb/tests/cooker.py
>> +++ b/bitbake/lib/bb/tests/cooker.py
>> @@ -1,20 +1,7 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # BitBake Tests for cooker.py
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>>
>> import unittest
>> diff --git a/bitbake/lib/bb/tests/cow.py b/bitbake/lib/bb/tests/cow.py
>> index d149d84..b4af4bb 100644
>> --- a/bitbake/lib/bb/tests/cow.py
>> +++ b/bitbake/lib/bb/tests/cow.py
>> @@ -1,22 +1,9 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # BitBake Tests for Copy-on-Write (cow.py)
>> #
>> -# Copyright 2006 Holger Freyther <freyther@handhelds.org>
>> -#
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# Copyright 2006 Holger Freyther
>> <freyther@handhelds.org> #
>>
>> import unittest
>> diff --git a/bitbake/lib/bb/tests/data.py
>> b/bitbake/lib/bb/tests/data.py index db3e201..3e49984 100644
>> --- a/bitbake/lib/bb/tests/data.py
>> +++ b/bitbake/lib/bb/tests/data.py
>> @@ -1,23 +1,10 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # BitBake Tests for the Data Store (data.py/data_smart.py)
>> #
>> # Copyright (C) 2010 Chris Larson
>> # Copyright (C) 2012 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>>
>> import unittest
>> @@ -394,6 +381,28 @@ class TestOverrides(unittest.TestCase):
>> self.d.setVar("OVERRIDES", "foo:bar:some_val")
>> self.assertEqual(self.d.getVar("TEST"), " testvalue5")
>>
>> + def test_append_and_override_1(self):
>> + self.d.setVar("TEST_append", "testvalue2")
>> + self.d.setVar("TEST_bar", "testvalue3")
>> + self.assertEqual(self.d.getVar("TEST"),
>> "testvalue3testvalue2") +
>> + def test_append_and_override_2(self):
>> + self.d.setVar("TEST_append_bar", "testvalue2")
>> + self.assertEqual(self.d.getVar("TEST"),
>> "testvaluetestvalue2") +
>> + def test_append_and_override_3(self):
>> + self.d.setVar("TEST_bar_append", "testvalue2")
>> + self.assertEqual(self.d.getVar("TEST"), "testvalue2")
>> +
>> + # Test an override with _<numeric> in it based on a real world
>> OE issue
>> + def test_underscore_override(self):
>> + self.d.setVar("TARGET_ARCH", "x86_64")
>> + self.d.setVar("PN", "test-${TARGET_ARCH}")
>> + self.d.setVar("VERSION", "1")
>> + self.d.setVar("VERSION_pn-test-${TARGET_ARCH}", "2")
>> + self.d.setVar("OVERRIDES", "pn-${PN}")
>> + bb.data.expandKeys(self.d)
>> + self.assertEqual(self.d.getVar("VERSION"), "2")
>>
>> class TestKeyExpansion(unittest.TestCase):
>> def setUp(self):
>> @@ -470,7 +479,7 @@ class TaskHash(unittest.TestCase):
>> tasklist, gendeps, lookupcache =
>> bb.data.generate_dependencies(d) taskdeps, basehash =
>> bb.data.generate_dependency_hash(tasklist, gendeps, lookupcache,
>> set(), "somefile") bb.warn(str(lookupcache))
>> - return basehash["somefile." + taskname]
>> + return basehash["somefile:" + taskname]
>>
>> d = bb.data.init()
>> d.setVar("__BBTASKS", ["mytask"])
>> diff --git a/bitbake/lib/bb/tests/event.py
>> b/bitbake/lib/bb/tests/event.py index d3a5f62..9229b63 100644
>> --- a/bitbake/lib/bb/tests/event.py
>> +++ b/bitbake/lib/bb/tests/event.py
>> @@ -1,22 +1,9 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # BitBake Tests for the Event implementation (event.py)
>> #
>> # Copyright (C) 2017 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>>
>> import unittest
>> @@ -574,14 +561,6 @@ class EventClassesTest(unittest.TestCase):
>> self.assertEqual(event.fn(1), callback(1))
>> self.assertEqual(event.pid, EventClassesTest._worker_pid)
>>
>> - def test_StampUpdate(self):
>> - targets = ["foo", "bar"]
>> - stampfns = [lambda:"foobar"]
>> - event = bb.event.StampUpdate(targets, stampfns)
>> - self.assertEqual(event.targets, targets)
>> - self.assertEqual(event.stampPrefix, stampfns)
>> - self.assertEqual(event.pid, EventClassesTest._worker_pid)
>> -
>> def test_BuildBase(self):
>> """ Test base class for bitbake build events """
>> name = "foo"
>> diff --git a/bitbake/lib/bb/tests/fetch.py
>> b/bitbake/lib/bb/tests/fetch.py index 6848095..a0b656b 100644
>> --- a/bitbake/lib/bb/tests/fetch.py
>> +++ b/bitbake/lib/bb/tests/fetch.py
>> @@ -1,22 +1,9 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # BitBake Tests for the Fetcher (fetch2/)
>> #
>> # Copyright (C) 2012 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>>
>> import unittest
>> @@ -893,12 +880,201 @@ class FetcherNetworkTest(FetcherTest):
>>
>> @skipIfNoNetwork()
>> def test_git_submodule(self):
>> - fetcher =
>> bb.fetch.Fetch(["gitsm://git.yoctoproject.org/git-submodule-test;rev=f12e57f2edf0aa534cf1616fa983d165a92b0842"],
>> self.d)
>> + # URL with ssh submodules
>> + url =
>> "gitsm://git.yoctoproject.org/git-submodule-test;branch=ssh-gitsm-tests;rev=049da4a6cb198d7c0302e9e8b243a1443cb809a7"
>> + # Original URL (comment this if you have ssh access to
>> git.yoctoproject.org)
>> + url =
>> "gitsm://git.yoctoproject.org/git-submodule-test;branch=master;rev=a2885dd7d25380d23627e7544b7bbb55014b16ee"
>> + fetcher = bb.fetch.Fetch([url], self.d)
>> + fetcher.download()
>> + # Previous cwd has been deleted
>> + os.chdir(os.path.dirname(self.unpackdir))
>> + fetcher.unpack(self.unpackdir)
>> +
>> + repo_path = os.path.join(self.tempdir, 'unpacked', 'git')
>> + self.assertTrue(os.path.exists(repo_path), msg='Unpacked
>> repository missing')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> 'bitbake')), msg='bitbake submodule missing')
>> + self.assertFalse(os.path.exists(os.path.join(repo_path,
>> 'na')), msg='uninitialized submodule present') +
>> + # Only when we're running the extended test with a
>> submodule's submodule, can we check this.
>> + if os.path.exists(os.path.join(repo_path,
>> 'bitbake-gitsm-test1')):
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> 'bitbake-gitsm-test1', 'bitbake')), msg='submodule of submodule
>> missing') +
>> + @skipIfNoNetwork()
>> + def test_git_submodule_dbus_broker(self):
>> + # The following external repositories have show failures in
>> fetch and unpack operations
>> + # We want to avoid regressions!
>> + url =
>> "gitsm://github.com/bus1/dbus-broker;protocol=git;rev=fc874afa0992d0c75ec25acb43d344679f0ee7d2"
>> + fetcher = bb.fetch.Fetch([url], self.d)
>> + fetcher.download()
>> + # Previous cwd has been deleted
>> + os.chdir(os.path.dirname(self.unpackdir))
>> + fetcher.unpack(self.unpackdir)
>> +
>> + repo_path = os.path.join(self.tempdir, 'unpacked', 'git')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> '.git/modules/subprojects/c-dvar/config')), msg='Missing submodule
>> config "subprojects/c-dvar"')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> '.git/modules/subprojects/c-list/config')), msg='Missing submodule
>> config "subprojects/c-list"')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> '.git/modules/subprojects/c-rbtree/config')), msg='Missing submodule
>> config "subprojects/c-rbtree"')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> '.git/modules/subprojects/c-sundry/config')), msg='Missing submodule
>> config "subprojects/c-sundry"')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> '.git/modules/subprojects/c-utf8/config')), msg='Missing submodule
>> config "subprojects/c-utf8"') +
>> + @skipIfNoNetwork()
>> + def test_git_submodule_CLI11(self):
>> + url =
>> "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=bd4dc911847d0cde7a6b41dfa626a85aab213baf"
>> + fetcher = bb.fetch.Fetch([url], self.d)
>> + fetcher.download()
>> + # Previous cwd has been deleted
>> + os.chdir(os.path.dirname(self.unpackdir))
>> + fetcher.unpack(self.unpackdir)
>> +
>> + repo_path = os.path.join(self.tempdir, 'unpacked', 'git')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> '.git/modules/extern/googletest/config')), msg='Missing submodule
>> config "extern/googletest"')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> '.git/modules/extern/json/config')), msg='Missing submodule config
>> "extern/json"')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> '.git/modules/extern/sanitizers/config')), msg='Missing submodule
>> config "extern/sanitizers"') +
>> + @skipIfNoNetwork()
>> + def test_git_submodule_update_CLI11(self):
>> + """ Prevent regression on update detection not finding
>> missing submodule, or modules without needed commits """
>> + url =
>> "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=cf6a99fa69aaefe477cc52e3ef4a7d2d7fa40714"
>> + fetcher = bb.fetch.Fetch([url], self.d)
>> + fetcher.download()
>> +
>> + # CLI11 that pulls in a newer nlohmann-json
>> + url =
>> "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=49ac989a9527ee9bb496de9ded7b4872c2e0e5ca"
>> + fetcher = bb.fetch.Fetch([url], self.d)
>> + fetcher.download()
>> + # Previous cwd has been deleted
>> + os.chdir(os.path.dirname(self.unpackdir))
>> + fetcher.unpack(self.unpackdir)
>> +
>> + repo_path = os.path.join(self.tempdir, 'unpacked', 'git')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> '.git/modules/extern/googletest/config')), msg='Missing submodule
>> config "extern/googletest"')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> '.git/modules/extern/json/config')), msg='Missing submodule config
>> "extern/json"')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> '.git/modules/extern/sanitizers/config')), msg='Missing submodule
>> config "extern/sanitizers"') +
>> + @skipIfNoNetwork()
>> + def test_git_submodule_aktualizr(self):
>> + url =
>> "gitsm://github.com/advancedtelematic/aktualizr;branch=master;protocol=git;rev=d00d1a04cc2366d1a5f143b84b9f507f8bd32c44"
>> + fetcher = bb.fetch.Fetch([url], self.d)
>> fetcher.download()
>> # Previous cwd has been deleted
>> os.chdir(os.path.dirname(self.unpackdir))
>> fetcher.unpack(self.unpackdir)
>>
>> + repo_path = os.path.join(self.tempdir, 'unpacked', 'git')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> '.git/modules/partial/extern/isotp-c/config')), msg='Missing
>> submodule config "partial/extern/isotp-c/config"')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> '.git/modules/partial/extern/isotp-c/modules/deps/bitfield-c/config')),
>> msg='Missing submodule config
>> "partial/extern/isotp-c/modules/deps/bitfield-c/config"')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> 'partial/extern/isotp-c/deps/bitfield-c/.git')), msg="Submodule of
>> submodule isotp-c did not unpack properly")
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> '.git/modules/tests/tuf-test-vectors/config')), msg='Missing
>> submodule config "tests/tuf-test-vectors/config"')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> '.git/modules/third_party/googletest/config')), msg='Missing
>> submodule config "third_party/googletest/config"')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> '.git/modules/third_party/HdrHistogram_c/config')), msg='Missing
>> submodule config "third_party/HdrHistogram_c/config"') +
>> + @skipIfNoNetwork()
>> + def test_git_submodule_iotedge(self):
>> + """ Prevent regression on deeply nested submodules not being
>> checked out properly, even though they were fetched. """ +
>> + # This repository also has submodules where the module
>> (name), path and url do not align
>> + url =
>> "gitsm://github.com/azure/iotedge.git;protocol=git;rev=d76e0316c6f324345d77c48a83ce836d09392699"
>> + fetcher = bb.fetch.Fetch([url], self.d)
>> + fetcher.download()
>> + # Previous cwd has been deleted
>> + os.chdir(os.path.dirname(self.unpackdir))
>> + fetcher.unpack(self.unpackdir)
>> +
>> + repo_path = os.path.join(self.tempdir, 'unpacked', 'git')
>> +
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared/README.md')),
>> msg='Missing submodule checkout')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared/testtools/ctest/README.md')),
>> msg='Missing submodule checkout')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared/testtools/testrunner/readme.md')),
>> msg='Missing submodule checkout')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared/testtools/umock-c/readme.md')),
>> msg='Missing submodule checkout')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared/testtools/umock-c/deps/ctest/README.md')),
>> msg='Missing submodule checkout')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared/testtools/umock-c/deps/testrunner/readme.md')),
>> msg='Missing submodule checkout')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/utpm/README.md')), msg='Missing
>> submodule checkout')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/utpm/deps/c-utility/README.md')),
>> msg='Missing submodule checkout')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/utpm/deps/c-utility/testtools/ctest/README.md')),
>> msg='Missing submodule checkout')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/utpm/deps/c-utility/testtools/testrunner/readme.md')),
>> msg='Missing submodule checkout')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/utpm/deps/c-utility/testtools/umock-c/readme.md')),
>> msg='Missing submodule checkout')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/utpm/deps/c-utility/testtools/umock-c/deps/ctest/README.md')),
>> msg='Missing submodule checkout')
>> + self.assertTrue(os.path.exists(os.path.join(repo_path,
>> 'edgelet/hsm-sys/azure-iot-hsm-c/deps/utpm/deps/c-utility/testtools/umock-c/deps/testrunner/readme.md')),
>> msg='Missing submodule checkout') + +class SVNTest(FetcherTest):
>> + def skipIfNoSvn():
>> + import shutil
>> + if not shutil.which("svn"):
>> + return unittest.skip("svn not installed, tests being
>> skipped") +
>> + if not shutil.which("svnadmin"):
>> + return unittest.skip("svnadmin not installed, tests
>> being skipped") +
>> + return lambda f: f
>> +
>> + @skipIfNoSvn()
>> + def setUp(self):
>> + """ Create a local repository """
>> +
>> + super(SVNTest, self).setUp()
>> +
>> + # Create something we can fetch
>> + src_dir = tempfile.mkdtemp(dir=self.tempdir,
>> + prefix='svnfetch_srcdir_')
>> + src_dir = os.path.abspath(src_dir)
>> + bb.process.run("echo readme > README.md", cwd=src_dir)
>> +
>> + # Store it in a local SVN repository
>> + repo_dir = tempfile.mkdtemp(dir=self.tempdir,
>> + prefix='svnfetch_localrepo_')
>> + repo_dir = os.path.abspath(repo_dir)
>> + bb.process.run("svnadmin create project", cwd=repo_dir)
>> +
>> + self.repo_url = "file://%s/project" % repo_dir
>> + bb.process.run("svn import --non-interactive -m 'Initial
>> import' %s %s/trunk" % (src_dir, self.repo_url),
>> + cwd=repo_dir)
>> +
>> + bb.process.run("svn co %s svnfetch_co" % self.repo_url,
>> cwd=self.tempdir)
>> + # Github will emulate SVN. Use this to check if we're
>> downloding...
>> + bb.process.run("svn propset svn:externals 'bitbake
>> http://github.com/openembedded/bitbake' .",
>> + cwd=os.path.join(self.tempdir, 'svnfetch_co',
>> 'trunk'))
>> + bb.process.run("svn commit --non-interactive -m 'Add
>> external'",
>> + cwd=os.path.join(self.tempdir, 'svnfetch_co',
>> 'trunk')) +
>> + self.src_dir = src_dir
>> + self.repo_dir = repo_dir
>> +
>> + @skipIfNoSvn()
>> + def tearDown(self):
>> + os.chdir(self.origdir)
>> + if os.environ.get("BB_TMPDIR_NOCLEAN") == "yes":
>> + print("Not cleaning up %s. Please remove manually." %
>> self.tempdir)
>> + else:
>> + bb.utils.prunedir(self.tempdir)
>> +
>> + @skipIfNoSvn()
>> + @skipIfNoNetwork()
>> + def test_noexternal_svn(self):
>> + # Always match the rev count from setUp (currently rev 2)
>> + url = "svn://%s;module=trunk;protocol=file;rev=2" %
>> self.repo_url.replace('file://', '')
>> + fetcher = bb.fetch.Fetch([url], self.d)
>> + fetcher.download()
>> + os.chdir(os.path.dirname(self.unpackdir))
>> + fetcher.unpack(self.unpackdir)
>> +
>> + self.assertTrue(os.path.exists(os.path.join(self.unpackdir,
>> 'trunk')), msg="Missing trunk")
>> + self.assertTrue(os.path.exists(os.path.join(self.unpackdir,
>> 'trunk', 'README.md')), msg="Missing contents")
>> + self.assertFalse(os.path.exists(os.path.join(self.unpackdir,
>> 'trunk/bitbake/trunk')), msg="External dir should NOT exist")
>> + self.assertFalse(os.path.exists(os.path.join(self.unpackdir,
>> 'trunk/bitbake/trunk', 'README')), msg="External README should NOT
>> exit") +
>> + @skipIfNoSvn()
>> + def test_external_svn(self):
>> + # Always match the rev count from setUp (currently rev 2)
>> + url =
>> "svn://%s;module=trunk;protocol=file;externals=allowed;rev=2" %
>> self.repo_url.replace('file://', '')
>> + fetcher = bb.fetch.Fetch([url], self.d)
>> + fetcher.download()
>> + os.chdir(os.path.dirname(self.unpackdir))
>> + fetcher.unpack(self.unpackdir)
>> +
>> + self.assertTrue(os.path.exists(os.path.join(self.unpackdir,
>> 'trunk')), msg="Missing trunk")
>> + self.assertTrue(os.path.exists(os.path.join(self.unpackdir,
>> 'trunk', 'README.md')), msg="Missing contents")
>> + self.assertTrue(os.path.exists(os.path.join(self.unpackdir,
>> 'trunk/bitbake/trunk')), msg="External dir should exist")
>> + self.assertTrue(os.path.exists(os.path.join(self.unpackdir,
>> 'trunk/bitbake/trunk', 'README')), msg="External README should exit")
>> class TrustedNetworksTest(FetcherTest):
>> def test_trusted_network(self):
>> @@ -1024,8 +1200,8 @@ class FetchLatestVersionTest(FetcherTest):
>> # packages with valid UPSTREAM_CHECK_URI and
>> UPSTREAM_CHECK_REGEX ("cups",
>> "http://www.cups.org/software/1.7.2/cups-1.7.2-source.tar.bz2",
>> "https://github.com/apple/cups/releases",
>> "(?P<name>cups\-)(?P<pver>((\d+[\.\-_]*)+))\-source\.tar\.gz") :
>> "2.0.0",
>> - ("db",
>> "http://download.oracle.com/berkeley-db/db-5.3.21.tar.gz",
>> "http://www.oracle.com/technetwork/products/berkeleydb/downloads/index-082944.html",
>> "http://download.oracle.com/otn/berkeley-db/(?P<name>db-)(?P<pver>((\d+[\.\-_]*)+))\.tar\.gz")
>> - : "6.1.19",
>> + ("db",
>> "http://download.oracle.com/berkeley-db/db-5.3.21.tar.gz",
>> "http://ftp.debian.org/debian/pool/main/d/db5.3/",
>> "(?P<name>db5\.3_)(?P<pver>\d+(\.\d+)+).+\.orig\.tar\.xz")
>> + : "5.3.10",
>> }
>>
>> @skipIfNoNetwork()
>> @@ -1280,7 +1456,7 @@ class GitShallowTest(FetcherTest):
>>
>> def fetch(self, uri=None):
>> if uri is None:
>> - uris = self.d.getVar('SRC_URI', True).split()
>> + uris = self.d.getVar('SRC_URI').split()
>> uri = uris[0]
>> d = self.d
>> else:
>> @@ -1312,6 +1488,7 @@ class GitShallowTest(FetcherTest):
>> # fetch and unpack, from the shallow tarball
>> bb.utils.remove(self.gitdir, recurse=True)
>> bb.utils.remove(ud.clonedir, recurse=True)
>> + bb.utils.remove(ud.clonedir.replace('gitsource',
>> 'gitsubmodule'), recurse=True)
>> # confirm that the unpacked repo is used when no git clone
>> or git # mirror tarball is available
>> @@ -1338,7 +1515,7 @@ class GitShallowTest(FetcherTest):
>>
>> srcrev = self.git('rev-parse HEAD', cwd=self.srcdir).strip()
>> self.d.setVar('SRCREV', srcrev)
>> - uri = self.d.getVar('SRC_URI', True).split()[0]
>> + uri = self.d.getVar('SRC_URI').split()[0]
>> uri = '%s;nobranch=1;bare=1' % uri
>>
>> self.fetch_shallow(uri)
>> @@ -1466,6 +1643,7 @@ class GitShallowTest(FetcherTest):
>> self.git('config --add remote.origin.url "%s"' % smdir,
>> cwd=smdir) self.git('config --add remote.origin.fetch
>> "+refs/heads/*:refs/remotes/origin/*"', cwd=smdir)
>> self.add_empty_file('asub', cwd=smdir)
>> + self.add_empty_file('bsub', cwd=smdir)
>>
>> self.git('submodule init', cwd=self.srcdir)
>> self.git('submodule add file://%s' % smdir, cwd=self.srcdir)
>> @@ -1475,10 +1653,16 @@ class GitShallowTest(FetcherTest):
>> uri = 'gitsm://%s;protocol=file;subdir=${S}' % self.srcdir
>> fetcher, ud = self.fetch_shallow(uri)
>>
>> + # Verify the main repository is shallow
>> self.assertRevCount(1)
>> - assert './.git/modules/' in bb.process.run('tar -tzf %s' %
>> os.path.join(self.dldir, ud.mirrortarballs[0]))[0] +
>> + # Verify the gitsubmodule directory is present
>> assert os.listdir(os.path.join(self.gitdir, 'gitsubmodule'))
>>
>> + # Verify the submodule is also shallow
>> + self.assertRevCount(1, cwd=os.path.join(self.gitdir,
>> 'gitsubmodule')) +
>> +
>> if any(os.path.exists(os.path.join(p, 'git-annex')) for p in
>> os.environ.get('PATH').split(':')): def test_shallow_annex(self):
>> self.add_empty_file('a')
>> @@ -1510,7 +1694,7 @@ class GitShallowTest(FetcherTest):
>> self.add_empty_file('f')
>> self.assertRevCount(7, cwd=self.srcdir)
>>
>> - uri = self.d.getVar('SRC_URI', True).split()[0]
>> + uri = self.d.getVar('SRC_URI').split()[0]
>> uri = '%s;branch=master,a_branch;name=master,a_branch' % uri
>>
>> self.d.setVar('BB_GIT_SHALLOW_DEPTH', '0')
>> @@ -1536,7 +1720,7 @@ class GitShallowTest(FetcherTest):
>> self.add_empty_file('f')
>> self.assertRevCount(7, cwd=self.srcdir)
>>
>> - uri = self.d.getVar('SRC_URI', True).split()[0]
>> + uri = self.d.getVar('SRC_URI').split()[0]
>> uri = '%s;branch=master,a_branch;name=master,a_branch' % uri
>>
>> self.d.setVar('BB_GIT_SHALLOW_DEPTH', '0')
>> @@ -1724,3 +1908,83 @@ class GitShallowTest(FetcherTest):
>>
>> dir = os.listdir(self.unpackdir + "/git/")
>> self.assertIn("fstests.doap", dir)
>> +
>> +class GitLfsTest(FetcherTest):
>> + def setUp(self):
>> + FetcherTest.setUp(self)
>> +
>> + self.gitdir = os.path.join(self.tempdir, 'git')
>> + self.srcdir = os.path.join(self.tempdir, 'gitsource')
>> +
>> + self.d.setVar('WORKDIR', self.tempdir)
>> + self.d.setVar('S', self.gitdir)
>> + self.d.delVar('PREMIRRORS')
>> + self.d.delVar('MIRRORS')
>> +
>> + self.d.setVar('SRCREV', '${AUTOREV}')
>> + self.d.setVar('AUTOREV', '${@bb.fetch2.get_autorev(d)}')
>> +
>> + bb.utils.mkdirhier(self.srcdir)
>> + self.git('init', cwd=self.srcdir)
>> + with open(os.path.join(self.srcdir, '.gitattributes'), 'wt')
>> as attrs:
>> + attrs.write('*.mp3 filter=lfs -text')
>> + self.git(['add', '.gitattributes'], cwd=self.srcdir)
>> + self.git(['commit', '-m', "attributes", '.gitattributes'],
>> cwd=self.srcdir) +
>> + def git(self, cmd, cwd=None):
>> + if isinstance(cmd, str):
>> + cmd = 'git ' + cmd
>> + else:
>> + cmd = ['git'] + cmd
>> + if cwd is None:
>> + cwd = self.gitdir
>> + return bb.process.run(cmd, cwd=cwd)[0]
>> +
>> + def fetch(self, uri=None):
>> + uris = self.d.getVar('SRC_URI').split()
>> + uri = uris[0]
>> + d = self.d
>> +
>> + fetcher = bb.fetch2.Fetch(uris, d)
>> + fetcher.download()
>> + ud = fetcher.ud[uri]
>> + return fetcher, ud
>> +
>> + def test_lfs_enabled(self):
>> + import shutil
>> +
>> + uri = 'git://%s;protocol=file;subdir=${S};lfs=1' %
>> self.srcdir
>> + self.d.setVar('SRC_URI', uri)
>> +
>> + fetcher, ud = self.fetch()
>> + self.assertIsNotNone(ud.method._find_git_lfs)
>> +
>> + # If git-lfs can be found, the unpack should be successful
>> + ud.method._find_git_lfs = lambda d: True
>> + shutil.rmtree(self.gitdir, ignore_errors=True)
>> + fetcher.unpack(self.d.getVar('WORKDIR'))
>> +
>> + # If git-lfs cannot be found, the unpack should throw an
>> error
>> + with self.assertRaises(bb.fetch2.FetchError):
>> + ud.method._find_git_lfs = lambda d: False
>> + shutil.rmtree(self.gitdir, ignore_errors=True)
>> + fetcher.unpack(self.d.getVar('WORKDIR'))
>> +
>> + def test_lfs_disabled(self):
>> + import shutil
>> +
>> + uri = 'git://%s;protocol=file;subdir=${S};lfs=0' %
>> self.srcdir
>> + self.d.setVar('SRC_URI', uri)
>> +
>> + fetcher, ud = self.fetch()
>> + self.assertIsNotNone(ud.method._find_git_lfs)
>> +
>> + # If git-lfs can be found, the unpack should be successful
>> + ud.method._find_git_lfs = lambda d: True
>> + shutil.rmtree(self.gitdir, ignore_errors=True)
>> + fetcher.unpack(self.d.getVar('WORKDIR'))
>> +
>> + # If git-lfs cannot be found, the unpack should be successful
>> + ud.method._find_git_lfs = lambda d: False
>> + shutil.rmtree(self.gitdir, ignore_errors=True)
>> + fetcher.unpack(self.d.getVar('WORKDIR'))
>> diff --git a/bitbake/lib/bb/tests/parse.py
>> b/bitbake/lib/bb/tests/parse.py index 1bc4740..9afd1b2 100644
>> --- a/bitbake/lib/bb/tests/parse.py
>> +++ b/bitbake/lib/bb/tests/parse.py
>> @@ -1,22 +1,9 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # BitBake Test for lib/bb/parse/
>> #
>> # Copyright (C) 2015 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>>
>> import unittest
>> @@ -187,3 +174,21 @@ python () {
>> self.assertEqual(d1.getVar("VAR_var"), "B")
>> self.assertEqual(d2.getVar("VAR_var"), None)
>>
>> + addtask_deltask = """
>> +addtask do_patch after do_foo after do_unpack before do_configure
>> before do_compile +addtask do_fetch do_patch
>> +
>> +deltask do_fetch do_patch
>> +"""
>> + def test_parse_addtask_deltask(self):
>> + import sys
>> + f = self.parsehelper(self.addtask_deltask)
>> + d = bb.parse.handle(f.name, self.d)['']
>> +
>> + stdout = sys.stdout.getvalue()
>> + self.assertTrue("addtask contained multiple 'before'
>> keywords" in stdout)
>> + self.assertTrue("addtask contained multiple 'after'
>> keywords" in stdout)
>> + self.assertTrue('addtask ignored: " do_patch"' in stdout)
>> + self.assertTrue('deltask ignored: " do_patch"' in stdout)
>> + #self.assertTrue('dependent task do_foo for do_patch does
>> not exist' in stdout) +
>> diff --git a/bitbake/lib/bb/tests/persist_data.py
>> b/bitbake/lib/bb/tests/persist_data.py new file mode 100644
>> index 0000000..f641b5a
>> --- /dev/null
>> +++ b/bitbake/lib/bb/tests/persist_data.py
>> @@ -0,0 +1,129 @@
>> +#
>> +# BitBake Test for lib/bb/persist_data/
>> +#
>> +# Copyright (C) 2018 Garmin Ltd.
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> +import unittest
>> +import bb.data
>> +import bb.persist_data
>> +import tempfile
>> +import threading
>> +
>> +class PersistDataTest(unittest.TestCase):
>> + def _create_data(self):
>> + return bb.persist_data.persist('TEST_PERSIST_DATA', self.d)
>> +
>> + def setUp(self):
>> + self.d = bb.data.init()
>> + self.tempdir = tempfile.TemporaryDirectory()
>> + self.d['PERSISTENT_DIR'] = self.tempdir.name
>> + self.data = self._create_data()
>> + self.items = {
>> + 'A1': '1',
>> + 'B1': '2',
>> + 'C2': '3'
>> + }
>> + self.stress_count = 10000
>> + self.thread_count = 5
>> +
>> + for k,v in self.items.items():
>> + self.data[k] = v
>> +
>> + def tearDown(self):
>> + self.tempdir.cleanup()
>> +
>> + def _iter_helper(self, seen, iterator):
>> + with iter(iterator):
>> + for v in iterator:
>> + self.assertTrue(v in seen)
>> + seen.remove(v)
>> + self.assertEqual(len(seen), 0, '%s not seen' % seen)
>> +
>> + def test_get(self):
>> + for k, v in self.items.items():
>> + self.assertEqual(self.data[k], v)
>> +
>> + self.assertIsNone(self.data.get('D'))
>> + with self.assertRaises(KeyError):
>> + self.data['D']
>> +
>> + def test_set(self):
>> + for k, v in self.items.items():
>> + self.data[k] += '-foo'
>> +
>> + for k, v in self.items.items():
>> + self.assertEqual(self.data[k], v + '-foo')
>> +
>> + def test_delete(self):
>> + self.data['D'] = '4'
>> + self.assertEqual(self.data['D'], '4')
>> + del self.data['D']
>> + self.assertIsNone(self.data.get('D'))
>> + with self.assertRaises(KeyError):
>> + self.data['D']
>> +
>> + def test_contains(self):
>> + for k in self.items:
>> + self.assertTrue(k in self.data)
>> + self.assertTrue(self.data.has_key(k))
>> + self.assertFalse('NotFound' in self.data)
>> + self.assertFalse(self.data.has_key('NotFound'))
>> +
>> + def test_len(self):
>> + self.assertEqual(len(self.data), len(self.items))
>> +
>> + def test_iter(self):
>> + self._iter_helper(set(self.items.keys()), self.data)
>> +
>> + def test_itervalues(self):
>> + self._iter_helper(set(self.items.values()),
>> self.data.itervalues()) +
>> + def test_iteritems(self):
>> + self._iter_helper(set(self.items.items()),
>> self.data.iteritems()) +
>> + def test_get_by_pattern(self):
>> + self._iter_helper({'1', '2'}, self.data.get_by_pattern('_1'))
>> +
>> + def _stress_read(self, data):
>> + for i in range(self.stress_count):
>> + for k in self.items:
>> + data[k]
>> +
>> + def _stress_write(self, data):
>> + for i in range(self.stress_count):
>> + for k, v in self.items.items():
>> + data[k] = v + str(i)
>> +
>> + def _validate_stress(self):
>> + for k, v in self.items.items():
>> + self.assertEqual(self.data[k], v + str(self.stress_count
>> - 1)) +
>> + def test_stress(self):
>> + self._stress_read(self.data)
>> + self._stress_write(self.data)
>> + self._validate_stress()
>> +
>> + def test_stress_threads(self):
>> + def read_thread():
>> + data = self._create_data()
>> + self._stress_read(data)
>> +
>> + def write_thread():
>> + data = self._create_data()
>> + self._stress_write(data)
>> +
>> + threads = []
>> + for i in range(self.thread_count):
>> + threads.append(threading.Thread(target=read_thread))
>> + threads.append(threading.Thread(target=write_thread))
>> +
>> + for t in threads:
>> + t.start()
>> + self._stress_read(self.data)
>> + for t in threads:
>> + t.join()
>> + self._validate_stress()
>> +
>> diff --git a/bitbake/lib/bb/tests/runqueue-tests/classes/base.bbclass
>> b/bitbake/lib/bb/tests/runqueue-tests/classes/base.bbclass new file
>> mode 100644 index 0000000..b57650d
>> --- /dev/null
>> +++ b/bitbake/lib/bb/tests/runqueue-tests/classes/base.bbclass
>> @@ -0,0 +1,262 @@
>> +SLOWTASKS ??= ""
>> +SSTATEVALID ??= ""
>> +
>> +def stamptask(d):
>> + import time
>> +
>> + thistask = d.expand("${PN}:${BB_CURRENTTASK}")
>> + stampname = d.expand("${TOPDIR}/%s.run" % thistask)
>> + with open(stampname, "a+") as f:
>> + f.write(d.getVar("BB_UNIHASH") + "\n")
>> +
>> + if d.getVar("BB_CURRENT_MC") != "default":
>> + thistask =
>> d.expand("${BB_CURRENT_MC}:${PN}:${BB_CURRENTTASK}")
>> + if thistask in d.getVar("SLOWTASKS").split():
>> + bb.note("Slowing task %s" % thistask)
>> + time.sleep(0.5)
>> + if d.getVar("BB_HASHSERVE"):
>> + task = d.getVar("BB_CURRENTTASK")
>> + if task in ['package', 'package_qa', 'packagedata',
>> 'package_write_ipk', 'package_write_rpm', 'populate_lic',
>> 'populate_sysroot']:
>> + bb.parse.siggen.report_unihash(os.getcwd(),
>> d.getVar("BB_CURRENTTASK"), d) +
>> + with open(d.expand("${TOPDIR}/task.log"), "a+") as f:
>> + f.write(thistask + "\n")
>> +
>> +
>> +def sstate_output_hash(path, sigfile, task, d):
>> + import hashlib
>> + h = hashlib.sha256()
>> + h.update(d.expand("${PN}:${BB_CURRENTTASK}").encode('utf-8'))
>> + return h.hexdigest()
>> +
>> +python do_fetch() {
>> + # fetch
>> + stamptask(d)
>> +}
>> +python do_unpack() {
>> + # unpack
>> + stamptask(d)
>> +}
>> +python do_patch() {
>> + # patch
>> + stamptask(d)
>> +}
>> +python do_populate_lic() {
>> + # populate_lic
>> + stamptask(d)
>> +}
>> +python do_prepare_recipe_sysroot() {
>> + # prepare_recipe_sysroot
>> + stamptask(d)
>> +}
>> +python do_configure() {
>> + # configure
>> + stamptask(d)
>> +}
>> +python do_compile() {
>> + # compile
>> + stamptask(d)
>> +}
>> +python do_install() {
>> + # install
>> + stamptask(d)
>> +}
>> +python do_populate_sysroot() {
>> + # populate_sysroot
>> + stamptask(d)
>> +}
>> +python do_package() {
>> + # package
>> + stamptask(d)
>> +}
>> +python do_package_write_ipk() {
>> + # package_write_ipk
>> + stamptask(d)
>> +}
>> +python do_package_write_rpm() {
>> + # package_write_rpm
>> + stamptask(d)
>> +}
>> +python do_packagedata() {
>> + # packagedata
>> + stamptask(d)
>> +}
>> +python do_package_qa() {
>> + # package_qa
>> + stamptask(d)
>> +}
>> +python do_build() {
>> + # build
>> + stamptask(d)
>> +}
>> +do_prepare_recipe_sysroot[deptask] = "do_populate_sysroot"
>> +do_package[deptask] += "do_packagedata"
>> +do_build[recrdeptask] += "do_deploy"
>> +do_build[recrdeptask] += "do_package_write_ipk"
>> +do_build[recrdeptask] += "do_package_write_rpm"
>> +do_package_qa[rdeptask] = "do_packagedata"
>> +do_populate_lic_deploy[recrdeptask] += "do_populate_lic do_deploy"
>> +
>> +DEBIANRDEP = "do_packagedata"
>> +oo_package_write_ipk[rdeptask] = "${DEBIANRDEP}"
>> +do_package_write_rpm[rdeptask] = "${DEBIANRDEP}"
>> +
>> +addtask fetch
>> +addtask unpack after do_fetch
>> +addtask patch after do_unpack
>> +addtask prepare_recipe_sysroot after do_patch
>> +addtask configure after do_prepare_recipe_sysroot
>> +addtask compile after do_configure
>> +addtask install after do_compile
>> +addtask populate_sysroot after do_install
>> +addtask package after do_install
>> +addtask package_write_ipk after do_packagedata do_package
>> +addtask package_write_rpm after do_packagedata do_package
>> +addtask packagedata after do_package
>> +addtask package_qa after do_package
>> +addtask build after do_package_qa do_package_write_rpm
>> do_package_write_ipk do_populate_sysroot +
>> +python do_package_setscene() {
>> + stamptask(d)
>> +}
>> +python do_package_qa_setscene() {
>> + stamptask(d)
>> +}
>> +python do_package_write_ipk_setscene() {
>> + stamptask(d)
>> +}
>> +python do_package_write_rpm_setscene() {
>> + stamptask(d)
>> +}
>> +python do_packagedata_setscene() {
>> + stamptask(d)
>> +}
>> +python do_populate_lic_setscene() {
>> + stamptask(d)
>> +}
>> +python do_populate_sysroot_setscene() {
>> + stamptask(d)
>> +}
>> +
>> +addtask package_setscene
>> +addtask package_qa_setscene
>> +addtask package_write_ipk_setscene
>> +addtask package_write_rpm_setscene
>> +addtask packagedata_setscene
>> +addtask populate_lic_setscene
>> +addtask populate_sysroot_setscene
>> +
>> +BB_SETSCENE_DEPVALID = "setscene_depvalid"
>> +
>> +def setscene_depvalid(task, taskdependees, notneeded, d, log=None):
>> + # taskdependees is a dict of tasks which depend on task, each
>> being a 3 item list of [PN, TASKNAME, FILENAME]
>> + # task is included in taskdependees too
>> + # Return - False - We need this dependency
>> + # - True - We can skip this dependency
>> + import re
>> +
>> + def logit(msg, log):
>> + if log is not None:
>> + log.append(msg)
>> + else:
>> + bb.debug(2, msg)
>> +
>> + logit("Considering setscene task: %s" %
>> (str(taskdependees[task])), log) +
>> + def isNativeCross(x):
>> + return x.endswith("-native") or "-cross-" in x or
>> "-crosssdk" in x or x.endswith("-cross") +
>> + # We only need to trigger populate_lic through direct
>> dependencies
>> + if taskdependees[task][1] == "do_populate_lic":
>> + return True
>> +
>> + # We only need to trigger packagedata through direct dependencies
>> + # but need to preserve packagedata on packagedata links
>> + if taskdependees[task][1] == "do_packagedata":
>> + for dep in taskdependees:
>> + if taskdependees[dep][1] == "do_packagedata":
>> + return False
>> + return True
>> +
>> + for dep in taskdependees:
>> + logit(" considering dependency: %s" %
>> (str(taskdependees[dep])), log)
>> + if task == dep:
>> + continue
>> + if dep in notneeded:
>> + continue
>> + # do_package_write_* and do_package doesn't need do_package
>> + if taskdependees[task][1] == "do_package" and
>> taskdependees[dep][1] in ['do_package', 'do_package_write_ipk',
>> 'do_package_write_rpm', 'do_packagedata', 'do_package_qa']:
>> + continue
>> + # do_package_write_* need do_populate_sysroot as they're
>> mainly postinstall dependencies
>> + if taskdependees[task][1] == "do_populate_sysroot" and
>> taskdependees[dep][1] in ['do_package_write_ipk',
>> 'do_package_write_rpm']:
>> + return False
>> + # do_package/packagedata/package_qa don't need
>> do_populate_sysroot
>> + if taskdependees[task][1] == "do_populate_sysroot" and
>> taskdependees[dep][1] in ['do_package', 'do_packagedata',
>> 'do_package_qa']:
>> + continue
>> + # Native/Cross packages don't exist and are noexec anyway
>> + if isNativeCross(taskdependees[dep][0]) and
>> taskdependees[dep][1] in ['do_package_write_ipk',
>> 'do_package_write_rpm', 'do_packagedata', 'do_package',
>> 'do_package_qa']:
>> + continue
>> +
>> + # This is due to the [depends] in useradd.bbclass
>> complicating matters
>> + # The logic *is* reversed here due to the way hard setscene
>> dependencies are injected
>> + if (taskdependees[task][1] == 'do_package' or
>> taskdependees[task][1] == 'do_populate_sysroot') and
>> taskdependees[dep][0].endswith(('shadow-native', 'shadow-sysroot',
>> 'base-passwd', 'pseudo-native')) and taskdependees[dep][1] ==
>> 'do_populate_sysroot':
>> + continue
>> +
>> + # Consider sysroot depending on sysroot tasks
>> + if taskdependees[task][1] == 'do_populate_sysroot' and
>> taskdependees[dep][1] == 'do_populate_sysroot':
>> + # Native/Cross populate_sysroot need their dependencies
>> + if isNativeCross(taskdependees[task][0]) and
>> isNativeCross(taskdependees[dep][0]):
>> + return False
>> + # Target populate_sysroot depended on by cross tools
>> need to be installed
>> + if isNativeCross(taskdependees[dep][0]):
>> + return False
>> + # Native/cross tools depended upon by target sysroot are
>> not needed
>> + # Add an exception for shadow-native as required by
>> useradd.bbclass
>> + if isNativeCross(taskdependees[task][0]) and
>> taskdependees[task][0] != 'shadow-native':
>> + continue
>> + # Target populate_sysroot need their dependencies
>> + return False
>> +
>> +
>> + if taskdependees[dep][1] == "do_populate_lic":
>> + continue
>> +
>> + # Safe fallthrough default
>> + logit(" Default setscene dependency fall through due to
>> dependency: %s" % (str(taskdependees[dep])), log)
>> + return False
>> + return True
>> +
>> +BB_HASHCHECK_FUNCTION = "sstate_checkhashes"
>> +
>> +def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0,
>> **kwargs): +
>> + found = set()
>> + missed = set()
>> +
>> + valid = d.getVar("SSTATEVALID").split()
>> +
>> + for tid in sorted(sq_data['hash']):
>> + n =
>> os.path.basename(bb.runqueue.fn_from_tid(tid)).split(".")[0] + ":do_"
>> + bb.runqueue.taskname_from_tid(tid)[3:]
>> + print(n)
>> + stampfile = d.expand("${TOPDIR}/%s.run" % n.replace("do_",
>> ""))
>> + if n in valid:
>> + bb.note("SState: Found valid sstate for %s" % n)
>> + found.add(tid)
>> + elif n + ":" + sq_data['hash'][tid] in valid:
>> + bb.note("SState: Found valid sstate for %s" % n)
>> + found.add(tid)
>> + elif os.path.exists(stampfile):
>> + with open(stampfile, "r") as f:
>> + hash = f.readline().strip()
>> + if hash == sq_data['hash'][tid]:
>> + bb.note("SState: Found valid sstate for %s (already
>> run)" % n)
>> + found.add(tid)
>> + else:
>> + bb.note("SState: sstate hash didn't match previous
>> run for %s (%s vs %s)" % (n, sq_data['hash'][tid], hash))
>> + missed.add(tid)
>> + else:
>> + missed.add(tid)
>> + bb.note("SState: Found no valid sstate for %s (%s)" %
>> (n, sq_data['hash'][tid])) +
>> + return found
>> +
>> diff --git
>> a/bitbake/lib/bb/tests/runqueue-tests/classes/image.bbclass
>> b/bitbake/lib/bb/tests/runqueue-tests/classes/image.bbclass new file
>> mode 100644 index 0000000..da9ff11 --- /dev/null
>> +++ b/bitbake/lib/bb/tests/runqueue-tests/classes/image.bbclass
>> @@ -0,0 +1,5 @@
>> +do_rootfs[recrdeptask] += "do_package_write_deb do_package_qa"
>> +do_rootfs[recrdeptask] += "do_package_write_ipk do_package_qa"
>> +do_rootfs[recrdeptask] += "do_package_write_rpm do_package_qa
>> +do_rootfs[recrdeptask] += "do_packagedata"
>> +do_rootfs[recrdeptask] += "do_populate_lic"
>> diff --git
>> a/bitbake/lib/bb/tests/runqueue-tests/classes/native.bbclass
>> b/bitbake/lib/bb/tests/runqueue-tests/classes/native.bbclass new file
>> mode 100644 index 0000000..7eaaee5 --- /dev/null
>> +++ b/bitbake/lib/bb/tests/runqueue-tests/classes/native.bbclass
>> @@ -0,0 +1,2 @@
>> +RECIPERDEPTASK = "do_populate_sysroot"
>> +do_populate_sysroot[rdeptask] = "${RECIPERDEPTASK}"
>> diff --git a/bitbake/lib/bb/tests/runqueue-tests/conf/bitbake.conf
>> b/bitbake/lib/bb/tests/runqueue-tests/conf/bitbake.conf new file mode
>> 100644 index 0000000..5e451fc
>> --- /dev/null
>> +++ b/bitbake/lib/bb/tests/runqueue-tests/conf/bitbake.conf
>> @@ -0,0 +1,16 @@
>> +CACHE = "${TOPDIR}/cache"
>> +THISDIR = "${@os.path.dirname(d.getVar('FILE'))}"
>> +COREBASE :=
>> "${@os.path.normpath(os.path.dirname(d.getVar('FILE')+'/../../'))}"
>> +BBFILES = "${COREBASE}/recipes/*.bb" +PROVIDES = "${PN}"
>> +PN = "${@bb.parse.vars_from_file(d.getVar('FILE', False),d)[0]}"
>> +PF = "${BB_CURRENT_MC}:${PN}"
>> +export PATH
>> +TMPDIR ??= "${TOPDIR}"
>> +STAMP = "${TMPDIR}/stamps/${PN}"
>> +T = "${TMPDIR}/workdir/${PN}/temp"
>> +BB_NUMBER_THREADS = "4"
>> +
>> +BB_HASHBASE_WHITELIST = "BB_CURRENT_MC BB_HASHSERVE TMPDIR TOPDIR
>> SLOWTASKS SSTATEVALID FILE" +
>> +include conf/multiconfig/${BB_CURRENT_MC}.conf
>> diff --git
>> a/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc1.conf
>> b/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc1.conf new
>> file mode 100644 index 0000000..ecf23e1 --- /dev/null
>> +++ b/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc1.conf
>> @@ -0,0 +1 @@
>> +TMPDIR = "${TOPDIR}/mc1/"
>> diff --git
>> a/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc2.conf
>> b/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc2.conf new
>> file mode 100644 index 0000000..eef338e --- /dev/null
>> +++ b/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc2.conf
>> @@ -0,0 +1 @@
>> +TMPDIR = "${TOPDIR}/mc2/"
>> diff --git a/bitbake/lib/bb/tests/runqueue-tests/recipes/a1.bb
>> b/bitbake/lib/bb/tests/runqueue-tests/recipes/a1.bb new file mode
>> 100644 index 0000000..e69de29
>> diff --git a/bitbake/lib/bb/tests/runqueue-tests/recipes/b1.bb
>> b/bitbake/lib/bb/tests/runqueue-tests/recipes/b1.bb new file mode
>> 100644 index 0000000..c0b288e
>> --- /dev/null
>> +++ b/bitbake/lib/bb/tests/runqueue-tests/recipes/b1.bb
>> @@ -0,0 +1 @@
>> +DEPENDS = "a1"
>> \ No newline at end of file
>> diff --git a/bitbake/lib/bb/tests/runqueue-tests/recipes/c1.bb
>> b/bitbake/lib/bb/tests/runqueue-tests/recipes/c1.bb new file mode
>> 100644 index 0000000..e69de29
>> diff --git a/bitbake/lib/bb/tests/runqueue-tests/recipes/d1.bb
>> b/bitbake/lib/bb/tests/runqueue-tests/recipes/d1.bb new file mode
>> 100644 index 0000000..5ba1975
>> --- /dev/null
>> +++ b/bitbake/lib/bb/tests/runqueue-tests/recipes/d1.bb
>> @@ -0,0 +1,3 @@
>> +DEPENDS = "a1"
>> +
>> +do_package_setscene[depends] = "a1:do_populate_sysroot_setscene"
>> diff --git a/bitbake/lib/bb/tests/runqueue-tests/recipes/e1.bb
>> b/bitbake/lib/bb/tests/runqueue-tests/recipes/e1.bb new file mode
>> 100644 index 0000000..1588bc8
>> --- /dev/null
>> +++ b/bitbake/lib/bb/tests/runqueue-tests/recipes/e1.bb
>> @@ -0,0 +1 @@
>> +DEPENDS = "b1"
>> \ No newline at end of file
>> diff --git a/bitbake/lib/bb/tests/runqueue.py
>> b/bitbake/lib/bb/tests/runqueue.py new file mode 100644
>> index 0000000..5e64391
>> --- /dev/null
>> +++ b/bitbake/lib/bb/tests/runqueue.py
>> @@ -0,0 +1,323 @@
>> +#
>> +# BitBake Tests for runqueue task processing
>> +#
>> +# Copyright (C) 2019 Richard Purdie
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> +import unittest
>> +import bb
>> +import os
>> +import tempfile
>> +import subprocess
>> +import sys
>> +import time
>> +
>> +#
>> +# TODO:
>> +# Add tests on task ordering (X happens before Y after Z)
>> +#
>> +
>> +class RunQueueTests(unittest.TestCase):
>> +
>> + alltasks = ['package', 'fetch', 'unpack', 'patch',
>> 'prepare_recipe_sysroot', 'configure',
>> + 'compile', 'install', 'packagedata', 'package_qa',
>> 'package_write_rpm', 'package_write_ipk',
>> + 'populate_sysroot', 'build']
>> + a1_sstatevalid = "a1:do_package a1:do_package_qa
>> a1:do_packagedata a1:do_package_write_ipk a1:do_package_write_rpm
>> a1:do_populate_lic a1:do_populate_sysroot"
>> + b1_sstatevalid = "b1:do_package b1:do_package_qa
>> b1:do_packagedata b1:do_package_write_ipk b1:do_package_write_rpm
>> b1:do_populate_lic b1:do_populate_sysroot" +
>> + def run_bitbakecmd(self, cmd, builddir, sstatevalid="",
>> slowtasks="", extraenv=None, cleanup=False):
>> + env = os.environ.copy()
>> + env["BBPATH"] =
>> os.path.realpath(os.path.join(os.path.dirname(__file__),
>> "runqueue-tests"))
>> + env["BB_ENV_EXTRAWHITE"] = "SSTATEVALID SLOWTASKS"
>> + env["SSTATEVALID"] = sstatevalid
>> + env["SLOWTASKS"] = slowtasks
>> + if extraenv:
>> + for k in extraenv:
>> + env[k] = extraenv[k]
>> + env["BB_ENV_EXTRAWHITE"] = env["BB_ENV_EXTRAWHITE"]
>> + " " + k
>> + try:
>> + output = subprocess.check_output(cmd, env=env,
>> stderr=subprocess.STDOUT,universal_newlines=True, cwd=builddir)
>> + print(output)
>> + except subprocess.CalledProcessError as e:
>> + self.fail("Command %s failed with %s" % (cmd, e.output))
>> + tasks = []
>> + tasklog = builddir + "/task.log"
>> + if os.path.exists(tasklog):
>> + with open(tasklog, "r") as f:
>> + tasks = [line.rstrip() for line in f]
>> + if cleanup:
>> + os.remove(tasklog)
>> + return tasks
>> +
>> + def test_no_setscenevalid(self):
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + cmd = ["bitbake", "a1"]
>> + sstatevalid = ""
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
>> + expected = ['a1:' + x for x in self.alltasks]
>> + self.assertEqual(set(tasks), set(expected))
>> +
>> + def test_single_setscenevalid(self):
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + cmd = ["bitbake", "a1"]
>> + sstatevalid = "a1:do_package"
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
>> + expected = ['a1:package_setscene', 'a1:fetch',
>> 'a1:unpack', 'a1:patch', 'a1:prepare_recipe_sysroot', 'a1:configure',
>> + 'a1:compile', 'a1:install',
>> 'a1:packagedata', 'a1:package_qa', 'a1:package_write_rpm',
>> 'a1:package_write_ipk',
>> + 'a1:populate_sysroot', 'a1:build']
>> + self.assertEqual(set(tasks), set(expected))
>> +
>> + def test_intermediate_setscenevalid(self):
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + cmd = ["bitbake", "a1"]
>> + sstatevalid = "a1:do_package a1:do_populate_sysroot"
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
>> + expected = ['a1:package_setscene', 'a1:packagedata',
>> 'a1:package_qa', 'a1:package_write_rpm', 'a1:package_write_ipk',
>> + 'a1:populate_sysroot_setscene', 'a1:build']
>> + self.assertEqual(set(tasks), set(expected))
>> +
>> + def test_intermediate_notcovered(self):
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + cmd = ["bitbake", "a1"]
>> + sstatevalid = "a1:do_package_qa a1:do_packagedata
>> a1:do_package_write_ipk a1:do_package_write_rpm a1:do_populate_lic
>> a1:do_populate_sysroot"
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
>> + expected = ['a1:package_write_ipk_setscene',
>> 'a1:package_write_rpm_setscene', 'a1:packagedata_setscene',
>> + 'a1:package_qa_setscene', 'a1:build',
>> 'a1:populate_sysroot_setscene']
>> + self.assertEqual(set(tasks), set(expected))
>> +
>> + def test_all_setscenevalid(self):
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + cmd = ["bitbake", "a1"]
>> + sstatevalid = self.a1_sstatevalid
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
>> + expected = ['a1:package_write_ipk_setscene',
>> 'a1:package_write_rpm_setscene', 'a1:packagedata_setscene',
>> + 'a1:package_qa_setscene', 'a1:build',
>> 'a1:populate_sysroot_setscene']
>> + self.assertEqual(set(tasks), set(expected))
>> +
>> + def test_no_settasks(self):
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + cmd = ["bitbake", "a1", "-c", "patch"]
>> + sstatevalid = self.a1_sstatevalid
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
>> + expected = ['a1:fetch', 'a1:unpack', 'a1:patch']
>> + self.assertEqual(set(tasks), set(expected))
>> +
>> + def test_mix_covered_notcovered(self):
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + cmd = ["bitbake", "a1:do_patch",
>> "a1:do_populate_sysroot"]
>> + sstatevalid = self.a1_sstatevalid
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
>> + expected = ['a1:fetch', 'a1:unpack', 'a1:patch',
>> 'a1:populate_sysroot_setscene']
>> + self.assertEqual(set(tasks), set(expected))
>> +
>> +
>> + # Test targets with intermediate setscene tasks alongside a
>> target with no intermediate setscene tasks
>> + def test_mixed_direct_tasks_setscene_tasks(self):
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + cmd = ["bitbake", "c1:do_patch", "a1"]
>> + sstatevalid = self.a1_sstatevalid
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
>> + expected = ['c1:fetch', 'c1:unpack', 'c1:patch',
>> 'a1:package_write_ipk_setscene', 'a1:package_write_rpm_setscene',
>> 'a1:packagedata_setscene',
>> + 'a1:package_qa_setscene', 'a1:build',
>> 'a1:populate_sysroot_setscene']
>> + self.assertEqual(set(tasks), set(expected))
>> +
>> + # This test slows down the execution of do_package_setscene
>> until after other real tasks have
>> + # started running which tests for a bug where tasks were being
>> lost from the buildable list of real
>> + # tasks if they weren't in tasks_covered or tasks_notcovered
>> + def test_slow_setscene(self):
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + cmd = ["bitbake", "a1"]
>> + sstatevalid = "a1:do_package"
>> + slowtasks = "a1:package_setscene"
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
>> slowtasks)
>> + expected = ['a1:package_setscene', 'a1:fetch',
>> 'a1:unpack', 'a1:patch', 'a1:prepare_recipe_sysroot', 'a1:configure',
>> + 'a1:compile', 'a1:install',
>> 'a1:packagedata', 'a1:package_qa', 'a1:package_write_rpm',
>> 'a1:package_write_ipk',
>> + 'a1:populate_sysroot', 'a1:build']
>> + self.assertEqual(set(tasks), set(expected))
>> +
>> + def test_setscenewhitelist(self):
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + cmd = ["bitbake", "a1"]
>> + extraenv = {
>> + "BB_SETSCENE_ENFORCE" : "1",
>> + "BB_SETSCENE_ENFORCE_WHITELIST" :
>> "a1:do_package_write_rpm a1:do_build"
>> + }
>> + sstatevalid = "a1:do_package a1:do_package_qa
>> a1:do_packagedata a1:do_package_write_ipk a1:do_populate_lic
>> a1:do_populate_sysroot"
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
>> extraenv=extraenv)
>> + expected = ['a1:packagedata_setscene',
>> 'a1:package_qa_setscene', 'a1:package_write_ipk_setscene',
>> + 'a1:populate_sysroot_setscene',
>> 'a1:package_setscene']
>> + self.assertEqual(set(tasks), set(expected))
>> +
>> + # Tests for problems with dependencies between setscene tasks
>> + def test_no_setscenevalid_harddeps(self):
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + cmd = ["bitbake", "d1"]
>> + sstatevalid = ""
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
>> + expected = ['a1:package', 'a1:fetch', 'a1:unpack',
>> 'a1:patch', 'a1:prepare_recipe_sysroot', 'a1:configure',
>> + 'a1:compile', 'a1:install',
>> 'a1:packagedata', 'a1:package_write_rpm', 'a1:package_write_ipk',
>> + 'a1:populate_sysroot', 'd1:package',
>> 'd1:fetch', 'd1:unpack', 'd1:patch', 'd1:prepare_recipe_sysroot',
>> 'd1:configure',
>> + 'd1:compile', 'd1:install',
>> 'd1:packagedata', 'd1:package_qa', 'd1:package_write_rpm',
>> 'd1:package_write_ipk',
>> + 'd1:populate_sysroot', 'd1:build']
>> + self.assertEqual(set(tasks), set(expected))
>> +
>> + def test_no_setscenevalid_withdeps(self):
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + cmd = ["bitbake", "b1"]
>> + sstatevalid = ""
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
>> + expected = ['a1:' + x for x in self.alltasks] + ['b1:' +
>> x for x in self.alltasks]
>> + expected.remove('a1:build')
>> + expected.remove('a1:package_qa')
>> + self.assertEqual(set(tasks), set(expected))
>> +
>> + def test_single_a1_setscenevalid_withdeps(self):
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + cmd = ["bitbake", "b1"]
>> + sstatevalid = "a1:do_package"
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
>> + expected = ['a1:package_setscene', 'a1:fetch',
>> 'a1:unpack', 'a1:patch', 'a1:prepare_recipe_sysroot', 'a1:configure',
>> + 'a1:compile', 'a1:install',
>> 'a1:packagedata', 'a1:package_write_rpm', 'a1:package_write_ipk',
>> + 'a1:populate_sysroot'] + ['b1:' + x for x in
>> self.alltasks]
>> + self.assertEqual(set(tasks), set(expected))
>> +
>> + def test_single_b1_setscenevalid_withdeps(self):
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + cmd = ["bitbake", "b1"]
>> + sstatevalid = "b1:do_package"
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
>> + expected = ['a1:package', 'a1:fetch', 'a1:unpack',
>> 'a1:patch', 'a1:prepare_recipe_sysroot', 'a1:configure',
>> + 'a1:compile', 'a1:install',
>> 'a1:packagedata', 'a1:package_write_rpm', 'a1:package_write_ipk',
>> + 'a1:populate_sysroot',
>> 'b1:package_setscene'] + ['b1:' + x for x in self.alltasks]
>> + expected.remove('b1:package')
>> + self.assertEqual(set(tasks), set(expected))
>> +
>> + def test_intermediate_setscenevalid_withdeps(self):
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + cmd = ["bitbake", "b1"]
>> + sstatevalid = "a1:do_package a1:do_populate_sysroot
>> b1:do_package"
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
>> + expected = ['a1:package_setscene', 'a1:packagedata',
>> 'a1:package_write_rpm', 'a1:package_write_ipk',
>> + 'a1:populate_sysroot_setscene',
>> 'b1:package_setscene'] + ['b1:' + x for x in self.alltasks]
>> + expected.remove('b1:package')
>> + self.assertEqual(set(tasks), set(expected))
>> +
>> + def test_all_setscenevalid_withdeps(self):
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + cmd = ["bitbake", "b1"]
>> + sstatevalid = self.a1_sstatevalid + " " +
>> self.b1_sstatevalid
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid)
>> + expected = ['a1:package_write_ipk_setscene',
>> 'a1:package_write_rpm_setscene', 'a1:packagedata_setscene',
>> + 'b1:build', 'a1:populate_sysroot_setscene',
>> 'b1:package_write_ipk_setscene', 'b1:package_write_rpm_setscene',
>> + 'b1:packagedata_setscene',
>> 'b1:package_qa_setscene', 'b1:populate_sysroot_setscene']
>> + self.assertEqual(set(tasks), set(expected))
>> +
>> + def test_multiconfig_setscene_optimise(self):
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + extraenv = {
>> + "BBMULTICONFIG" : "mc1 mc2",
>> + "BB_SIGNATURE_HANDLER" : "basic"
>> + }
>> + cmd = ["bitbake", "b1", "mc:mc1:b1", "mc:mc2:b1"]
>> + setscenetasks = ['package_write_ipk_setscene',
>> 'package_write_rpm_setscene', 'packagedata_setscene',
>> + 'populate_sysroot_setscene',
>> 'package_qa_setscene']
>> + sstatevalid = ""
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
>> extraenv=extraenv)
>> + expected = ['a1:' + x for x in self.alltasks] + ['b1:' +
>> x for x in self.alltasks] + \
>> + ['mc1:b1:' + x for x in setscenetasks] +
>> ['mc1:a1:' + x for x in setscenetasks] + \
>> + ['mc2:b1:' + x for x in setscenetasks] +
>> ['mc2:a1:' + x for x in setscenetasks] + \
>> + ['mc1:b1:build', 'mc2:b1:build']
>> + for x in ['mc1:a1:package_qa_setscene',
>> 'mc2:a1:package_qa_setscene', 'a1:build', 'a1:package_qa']:
>> + expected.remove(x)
>> + self.assertEqual(set(tasks), set(expected))
>> +
>> +
>> + @unittest.skipIf(sys.version_info < (3, 5, 0), 'Python 3.5 or
>> later required')
>> + def test_hashserv_single(self):
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + extraenv = {
>> + "BB_HASHSERVE" : "auto",
>> + "BB_SIGNATURE_HANDLER" : "TestEquivHash"
>> + }
>> + cmd = ["bitbake", "a1", "b1"]
>> + setscenetasks = ['package_write_ipk_setscene',
>> 'package_write_rpm_setscene', 'packagedata_setscene',
>> + 'populate_sysroot_setscene',
>> 'package_qa_setscene']
>> + sstatevalid = ""
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
>> extraenv=extraenv, cleanup=True)
>> + expected = ['a1:' + x for x in self.alltasks] + ['b1:' +
>> x for x in self.alltasks]
>> + self.assertEqual(set(tasks), set(expected))
>> + cmd = ["bitbake", "a1", "-c", "install", "-f"]
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
>> extraenv=extraenv, cleanup=True)
>> + expected = ['a1:install']
>> + self.assertEqual(set(tasks), set(expected))
>> + cmd = ["bitbake", "a1", "b1"]
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
>> extraenv=extraenv, cleanup=True)
>> + expected = ['a1:populate_sysroot', 'a1:package',
>> 'a1:package_write_rpm_setscene', 'a1:packagedata_setscene',
>> + 'a1:package_write_ipk_setscene',
>> 'a1:package_qa_setscene']
>> + self.assertEqual(set(tasks), set(expected))
>> +
>> + self.shutdown(tempdir)
>> +
>> + @unittest.skipIf(sys.version_info < (3, 5, 0), 'Python 3.5 or
>> later required')
>> + def test_hashserv_double(self):
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + extraenv = {
>> + "BB_HASHSERVE" : "auto",
>> + "BB_SIGNATURE_HANDLER" : "TestEquivHash"
>> + }
>> + cmd = ["bitbake", "a1", "b1", "e1"]
>> + setscenetasks = ['package_write_ipk_setscene',
>> 'package_write_rpm_setscene', 'packagedata_setscene',
>> + 'populate_sysroot_setscene',
>> 'package_qa_setscene']
>> + sstatevalid = ""
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
>> extraenv=extraenv, cleanup=True)
>> + expected = ['a1:' + x for x in self.alltasks] + ['b1:' +
>> x for x in self.alltasks] + ['e1:' + x for x in self.alltasks]
>> + self.assertEqual(set(tasks), set(expected))
>> + cmd = ["bitbake", "a1", "b1", "-c", "install", "-fn"]
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
>> extraenv=extraenv, cleanup=True)
>> + cmd = ["bitbake", "e1"]
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
>> extraenv=extraenv, cleanup=True)
>> + expected = ['a1:package', 'a1:install', 'b1:package',
>> 'b1:install', 'a1:populate_sysroot', 'b1:populate_sysroot',
>> + 'a1:package_write_ipk_setscene',
>> 'b1:packagedata_setscene', 'b1:package_write_rpm_setscene',
>> + 'a1:package_write_rpm_setscene',
>> 'b1:package_write_ipk_setscene', 'a1:packagedata_setscene']
>> + self.assertEqual(set(tasks), set(expected))
>> +
>> + self.shutdown(tempdir)
>> +
>> + @unittest.skipIf(sys.version_info < (3, 5, 0), 'Python 3.5 or
>> later required')
>> + def test_hashserv_multiple_setscene(self):
>> + # Runs e1:do_package_setscene twice
>> + with tempfile.TemporaryDirectory(prefix="runqueuetest") as
>> tempdir:
>> + extraenv = {
>> + "BB_HASHSERVE" : "auto",
>> + "BB_SIGNATURE_HANDLER" : "TestEquivHash"
>> + }
>> + cmd = ["bitbake", "a1", "b1", "e1"]
>> + setscenetasks = ['package_write_ipk_setscene',
>> 'package_write_rpm_setscene', 'packagedata_setscene',
>> + 'populate_sysroot_setscene',
>> 'package_qa_setscene']
>> + sstatevalid = ""
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
>> extraenv=extraenv, cleanup=True)
>> + expected = ['a1:' + x for x in self.alltasks] + ['b1:' +
>> x for x in self.alltasks] + ['e1:' + x for x in self.alltasks]
>> + self.assertEqual(set(tasks), set(expected))
>> + cmd = ["bitbake", "a1", "b1", "-c", "install", "-fn"]
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
>> extraenv=extraenv, cleanup=True)
>> + cmd = ["bitbake", "e1"]
>> + sstatevalid = "e1:do_package"
>> + tasks = self.run_bitbakecmd(cmd, tempdir, sstatevalid,
>> extraenv=extraenv, cleanup=True, slowtasks="a1:populate_sysroot
>> b1:populate_sysroot")
>> + expected = ['a1:package', 'a1:install', 'b1:package',
>> 'b1:install', 'a1:populate_sysroot', 'b1:populate_sysroot',
>> + 'a1:package_write_ipk_setscene',
>> 'b1:packagedata_setscene', 'b1:package_write_rpm_setscene',
>> + 'a1:package_write_rpm_setscene',
>> 'b1:package_write_ipk_setscene', 'a1:packagedata_setscene',
>> + 'e1:package_setscene']
>> + self.assertEqual(set(tasks), set(expected))
>> + for i in expected:
>> + self.assertEqual(tasks.count(i), 1, "%s not in task
>> list once" % i) +
>> + self.shutdown(tempdir)
>> +
>> + def shutdown(self, tempdir):
>> + # Wait for the hashserve socket to disappear else we'll see
>> races with the tempdir cleanup
>> + while os.path.exists(tempdir + "/hashserve.sock"):
>> + time.sleep(0.5)
>> +
>> +
>> diff --git a/bitbake/lib/bb/tests/utils.py
>> b/bitbake/lib/bb/tests/utils.py index 2f4ccf3..f4adf1d 100644
>> --- a/bitbake/lib/bb/tests/utils.py
>> +++ b/bitbake/lib/bb/tests/utils.py
>> @@ -1,22 +1,9 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # BitBake Tests for utils.py
>> #
>> # Copyright (C) 2012 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>>
>> import unittest
>> @@ -42,6 +29,14 @@ class VerCmpString(unittest.TestCase):
>> self.assertTrue(result < 0)
>> result = bb.utils.vercmp_string('1.1', '1.0+1.1-beta1')
>> self.assertTrue(result > 0)
>> + result = bb.utils.vercmp_string('1a', '1a1')
>> + self.assertTrue(result < 0)
>> + result = bb.utils.vercmp_string('1a1', '1a')
>> + self.assertTrue(result > 0)
>> + result = bb.utils.vercmp_string('1.', '1.1')
>> + self.assertTrue(result < 0)
>> + result = bb.utils.vercmp_string('1.1', '1.')
>> + self.assertTrue(result > 0)
>>
>> def test_explode_dep_versions(self):
>> correctresult = {"foo" : ["= 1.10"]}
>> diff --git a/bitbake/lib/bb/tinfoil.py b/bitbake/lib/bb/tinfoil.py
>> index 368264f..0a1b913 100644
>> --- a/bitbake/lib/bb/tinfoil.py
>> +++ b/bitbake/lib/bb/tinfoil.py
>> @@ -4,18 +4,8 @@
>> # Copyright (C) 2011 Mentor Graphics Corporation
>> # Copyright (C) 2006-2012 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import logging
>> import os
>> diff --git a/bitbake/lib/bb/ui/__init__.py
>> b/bitbake/lib/bb/ui/__init__.py index a4805ed..4b7ac36 100644
>> --- a/bitbake/lib/bb/ui/__init__.py
>> +++ b/bitbake/lib/bb/ui/__init__.py
>> @@ -3,15 +3,5 @@
>> #
>> # Copyright (C) 2006-2007 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. diff --git a/bitbake/lib/bb/ui/buildinfohelper.py
>> b/bitbake/lib/bb/ui/buildinfohelper.py index 31323d2..5cbca97 100644
>> --- a/bitbake/lib/bb/ui/buildinfohelper.py
>> +++ b/bitbake/lib/bb/ui/buildinfohelper.py
>> @@ -3,18 +3,8 @@
>> #
>> # Copyright (C) 2013 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import sys
>> import bb
>> @@ -656,6 +646,9 @@ class ORMWrapper(object):
>> Target_Installed_Package.objects.create(target =
>> target_obj, package = packagedict[p]['object'])
>> packagedeps_objs = []
>> + pattern_so = re.compile(r'.*\.so(\.\d*)?$')
>> + pattern_lib = re.compile(r'.*\-suffix(\d*)?$')
>> + pattern_ko = re.compile(r'^kernel-module-.*')
>> for p in packagedict:
>> for (px,deptype) in packagedict[p]['depends']:
>> if deptype == 'depends':
>> @@ -664,6 +657,13 @@ class ORMWrapper(object):
>> tdeptype = Package_Dependency.TYPE_TRECOMMENDS
>>
>> try:
>> + # Skip known non-package objects like libraries
>> and kernel modules
>> + if pattern_so.match(px) or pattern_lib.match(px):
>> + logger.info("Toaster does not add library
>> file dependencies to packages (%s,%s)", p, px)
>> + continue
>> + if pattern_ko.match(px):
>> + logger.info("Toaster does not add kernel
>> module dependencies to packages (%s,%s)", p, px)
>> + continue
>> packagedeps_objs.append(Package_Dependency(
>> package = packagedict[p]['object'],
>> depends_on = packagedict[px]['object'],
>> diff --git a/bitbake/lib/bb/ui/knotty.py b/bitbake/lib/bb/ui/knotty.py
>> index fa88e6c..35736ad 100644
>> --- a/bitbake/lib/bb/ui/knotty.py
>> +++ b/bitbake/lib/bb/ui/knotty.py
>> @@ -5,18 +5,8 @@
>> #
>> # Copyright (C) 2006-2012 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from __future__ import division
>>
>> @@ -222,6 +212,23 @@ class TerminalFilter(object):
>> sys.stdout.flush()
>> self.footer_present = False
>>
>> + def elapsed(self, sec):
>> + hrs = int(sec / 3600.0)
>> + sec -= hrs * 3600
>> + min = int(sec / 60.0)
>> + sec -= min * 60
>> + if hrs > 0:
>> + return "%dh%dm%ds" % (hrs, min, sec)
>> + elif min > 0:
>> + return "%dm%ds" % (min, sec)
>> + else:
>> + return "%ds" % (sec)
>> +
>> + def keepAlive(self, t):
>> + if not self.cuu:
>> + print("Bitbake still alive (%ds)" % t)
>> + sys.stdout.flush()
>> +
>> def updateFooter(self):
>> if not self.cuu:
>> return
>> @@ -258,7 +265,7 @@ class TerminalFilter(object):
>> else:
>> start_time = activetasks[t].get("starttime", None)
>> if start_time:
>> - tasks.append("%s - %ds (pid %s)" %
>> (activetasks[t]["title"], currenttime - start_time, t))
>> + tasks.append("%s - %s (pid %s)" %
>> (activetasks[t]["title"], self.elapsed(currenttime - start_time), t))
>> else: tasks.append("%s (pid %s)" % (activetasks[t]["title"], t))
>>
>> @@ -293,8 +300,8 @@ class TerminalFilter(object):
>> if start_time:
>> pbar.start_time = start_time
>> pbar.setmessage('%s:%s' % (tasknum,
>> pbar.msg.split(':', 1)[1]))
>> + pbar.setextra(rate)
>> if progress > -1:
>> - pbar.setextra(rate)
>> content = pbar.update(progress)
>> else:
>> content = pbar.update(1)
>> @@ -455,11 +462,17 @@ def main(server, eventHandler, params, tf =
>> TerminalFilter): warnings = 0
>> taskfailures = []
>>
>> + printinterval = 5000
>> + lastprint = time.time()
>> +
>> termfilter = tf(main, helper, console, errconsole, format,
>> params.options.quiet) atexit.register(termfilter.finish)
>>
>> while True:
>> try:
>> + if (lastprint + printinterval) <= time.time():
>> + termfilter.keepAlive(printinterval)
>> + printinterval += 5000
>> event = eventHandler.waitEvent(0)
>> if event is None:
>> if main.shutdown > 1:
>> @@ -488,6 +501,8 @@ def main(server, eventHandler, params, tf =
>> TerminalFilter): continue
>>
>> if isinstance(event, logging.LogRecord):
>> + lastprint = time.time()
>> + printinterval = 5000
>> if event.levelno >= format.ERROR:
>> errors = errors + 1
>> return_value = 1
>> @@ -645,7 +660,6 @@ def main(server, eventHandler, params, tf =
>> TerminalFilter): # ignore
>> if isinstance(event, (bb.event.BuildBase,
>> bb.event.MetadataEvent,
>> - bb.event.StampUpdate,
>> bb.event.ConfigParsed,
>> bb.event.MultiConfigParsed,
>> bb.event.RecipeParsed,
>> @@ -675,17 +689,27 @@ def main(server, eventHandler, params, tf =
>> TerminalFilter): if params.observe_only:
>> print("\nKeyboard Interrupt, exiting observer...")
>> main.shutdown = 2
>> - if not params.observe_only and main.shutdown == 1:
>> +
>> + def state_force_shutdown():
>> print("\nSecond Keyboard Interrupt, stopping...\n")
>> _, error = server.runCommand(["stateForceShutdown"])
>> if error:
>> logger.error("Unable to cleanly stop: %s" %
>> error) +
>> + if not params.observe_only and main.shutdown == 1:
>> + state_force_shutdown()
>> +
>> if not params.observe_only and main.shutdown == 0:
>> print("\nKeyboard Interrupt, closing down...\n")
>> interrupted = True
>> - _, error = server.runCommand(["stateShutdown"])
>> - if error:
>> - logger.error("Unable to cleanly shutdown: %s" %
>> error)
>> + # Capture the second KeyboardInterrupt during
>> stateShutdown is running
>> + try:
>> + _, error = server.runCommand(["stateShutdown"])
>> + if error:
>> + logger.error("Unable to cleanly shutdown:
>> %s" % error)
>> + except KeyboardInterrupt:
>> + state_force_shutdown()
>> +
>> main.shutdown = main.shutdown + 1
>> pass
>> except Exception as e:
>> diff --git a/bitbake/lib/bb/ui/ncurses.py
>> b/bitbake/lib/bb/ui/ncurses.py index 8690c52..c422732 100644
>> --- a/bitbake/lib/bb/ui/ncurses.py
>> +++ b/bitbake/lib/bb/ui/ncurses.py
>> @@ -6,18 +6,8 @@
>> # Copyright (C) 2006 Michael 'Mickey' Lauer
>> # Copyright (C) 2006-2007 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> """
>> We have the following windows:
>> diff --git a/bitbake/lib/bb/ui/taskexp.py
>> b/bitbake/lib/bb/ui/taskexp.py index 8305d70..50a943c 100644
>> --- a/bitbake/lib/bb/ui/taskexp.py
>> +++ b/bitbake/lib/bb/ui/taskexp.py
>> @@ -4,18 +4,8 @@
>> # Copyright (C) 2007 Ross Burton
>> # Copyright (C) 2007 - 2008 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import sys
>> import gi
>> diff --git a/bitbake/lib/bb/ui/toasterui.py
>> b/bitbake/lib/bb/ui/toasterui.py index 88cec37..51892c9 100644
>> --- a/bitbake/lib/bb/ui/toasterui.py
>> +++ b/bitbake/lib/bb/ui/toasterui.py
>> @@ -7,18 +7,8 @@
>> # Copyright (C) 2006-2012 Richard Purdie
>> # Copyright (C) 2013 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from __future__ import division
>> import time
>> diff --git a/bitbake/lib/bb/ui/uievent.py
>> b/bitbake/lib/bb/ui/uievent.py index 9542b91..fedb050 100644
>> --- a/bitbake/lib/bb/ui/uievent.py
>> +++ b/bitbake/lib/bb/ui/uievent.py
>> @@ -1,22 +1,9 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # Copyright (C) 2006 - 2007 Michael 'Mickey' Lauer
>> # Copyright (C) 2006 - 2007 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. -
>>
>> """
>> Use this class to fork off a thread to recieve event callbacks from
>> the bitbake diff --git a/bitbake/lib/bb/ui/uihelper.py
>> b/bitbake/lib/bb/ui/uihelper.py index 963c1ea..c8dd7df 100644
>> --- a/bitbake/lib/bb/ui/uihelper.py
>> +++ b/bitbake/lib/bb/ui/uihelper.py
>> @@ -1,21 +1,9 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> # Copyright (C) 2006 - 2007 Michael 'Mickey' Lauer
>> # Copyright (C) 2006 - 2007 Richard Purdie
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import bb.build
>> import time
>> @@ -52,7 +40,7 @@ class BBUIHelper:
>> self.running_pids.remove(event.pid)
>> self.failed_tasks.append( { 'title' : "%s %s" %
>> (event._package, event._task)}) self.needUpdate = True
>> - elif isinstance(event, bb.runqueue.runQueueTaskStarted) or
>> isinstance(event, bb.runqueue.sceneQueueTaskStarted):
>> + elif isinstance(event, bb.runqueue.runQueueTaskStarted):
>> self.tasknumber_current = event.stats.completed +
>> event.stats.active + event.stats.failed + 1 self.tasknumber_total =
>> event.stats.total self.needUpdate = True
>> diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
>> index 13bb5f2..d035949 100644
>> --- a/bitbake/lib/bb/utils.py
>> +++ b/bitbake/lib/bb/utils.py
>> @@ -1,23 +1,11 @@
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> """
>> BitBake Utility Functions
>> """
>>
>> # Copyright (C) 2004 Michael Lauer
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import re, fcntl, os, string, stat, shutil, time
>> import sys
>> @@ -121,6 +109,10 @@ def vercmp_part(a, b):
>> return -1
>> elif oa > ob:
>> return 1
>> + elif ca is None:
>> + return -1
>> + elif cb is None:
>> + return 1
>> elif ca < cb:
>> return -1
>> elif ca > cb:
>> @@ -402,7 +394,7 @@ def better_exec(code, context, text = None,
>> realfile = "<code>", pythonexception code = better_compile(code,
>> realfile, realfile) try:
>> exec(code, get_context(), context)
>> - except (bb.BBHandledException, bb.parse.SkipRecipe,
>> bb.build.FuncFailed, bb.data_smart.ExpansionError):
>> + except (bb.BBHandledException, bb.parse.SkipRecipe,
>> bb.data_smart.ExpansionError): # Error already shown so passthrough,
>> no need for traceback raise
>> except Exception as e:
>> @@ -685,7 +677,7 @@ def _check_unsafe_delete_path(path):
>> return True
>> return False
>>
>> -def remove(path, recurse=False):
>> +def remove(path, recurse=False, ionice=False):
>> """Equivalent to rm -f or rm -rf"""
>> if not path:
>> return
>> @@ -694,7 +686,10 @@ def remove(path, recurse=False):
>> if _check_unsafe_delete_path(path):
>> raise Exception('bb.utils.remove: called with
>> dangerous path "%s" and recurse=True, refusing to delete!' % path) #
>> shutil.rmtree(name) would be ideal but its too slow
>> - subprocess.check_call(['rm', '-rf'] + glob.glob(path))
>> + cmd = []
>> + if ionice:
>> + cmd = ['ionice', '-c', '3']
>> + subprocess.check_call(cmd + ['rm', '-rf'] + glob.glob(path))
>> return
>> for name in glob.glob(path):
>> try:
>> @@ -703,20 +698,12 @@ def remove(path, recurse=False):
>> if exc.errno != errno.ENOENT:
>> raise
>>
>> -def prunedir(topdir):
>> +def prunedir(topdir, ionice=False):
>> # Delete everything reachable from the directory named in
>> 'topdir'. # CAUTION: This is dangerous!
>> if _check_unsafe_delete_path(topdir):
>> raise Exception('bb.utils.prunedir: called with dangerous
>> path "%s", refusing to delete!' % topdir)
>> - for root, dirs, files in os.walk(topdir, topdown = False):
>> - for name in files:
>> - os.remove(os.path.join(root, name))
>> - for name in dirs:
>> - if os.path.islink(os.path.join(root, name)):
>> - os.remove(os.path.join(root, name))
>> - else:
>> - os.rmdir(os.path.join(root, name))
>> - os.rmdir(topdir)
>> + remove(topdir, recurse=True, ionice=ionice)
>>
>> #
>> # Could also use return re.compile("(%s)" % "|".join(map(re.escape,
>> suffixes))).sub(lambda mo: "", var) @@ -726,8 +713,8 @@ def
>> prune_suffix(var, suffixes, d): # See if var ends with any of the
>> suffixes listed and # remove it if found
>> for suffix in suffixes:
>> - if var.endswith(suffix):
>> - return var.replace(suffix, "")
>> + if suffix and var.endswith(suffix):
>> + return var[:-len(suffix)]
>> return var
>>
>> def mkdirhier(directory):
>> @@ -738,7 +725,7 @@ def mkdirhier(directory):
>> try:
>> os.makedirs(directory)
>> except OSError as e:
>> - if e.errno != errno.EEXIST:
>> + if e.errno != errno.EEXIST or not os.path.isdir(directory):
>> raise e
>>
>> def movefile(src, dest, newmtime = None, sstat = None):
>> @@ -796,7 +783,7 @@ def movefile(src, dest, newmtime = None, sstat =
>> None): os.rename(src, destpath)
>> renamefailed = 0
>> except Exception as e:
>> - if e[0] != errno.EXDEV:
>> + if e.errno != errno.EXDEV:
>> # Some random error.
>> print("movefile: Failed to move", src, "to", dest, e)
>> return None
>> @@ -1505,6 +1492,8 @@ def ioprio_set(who, cls, value):
>> NR_ioprio_set = 251
>> elif _unamearch[0] == "i" and _unamearch[2:3] == "86":
>> NR_ioprio_set = 289
>> + elif _unamearch == "aarch64":
>> + NR_ioprio_set = 30
>>
>> if NR_ioprio_set:
>> ioprio = value | (cls << IOPRIO_CLASS_SHIFT)
>> diff --git a/bitbake/lib/bblayers/__init__.py
>> b/bitbake/lib/bblayers/__init__.py index 3ad9513..4e7c09d 100644
>> --- a/bitbake/lib/bblayers/__init__.py
>> +++ b/bitbake/lib/bblayers/__init__.py
>> @@ -1,2 +1,6 @@
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> from pkgutil import extend_path
>> __path__ = extend_path(__path__, __name__)
>> diff --git a/bitbake/lib/bblayers/action.py
>> b/bitbake/lib/bblayers/action.py index a3f658f..d6459d6 100644
>> --- a/bitbake/lib/bblayers/action.py
>> +++ b/bitbake/lib/bblayers/action.py
>> @@ -1,3 +1,7 @@
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> import fnmatch
>> import logging
>> import os
>> diff --git a/bitbake/lib/bblayers/common.py
>> b/bitbake/lib/bblayers/common.py index 98515ce..c5657d3 100644
>> --- a/bitbake/lib/bblayers/common.py
>> +++ b/bitbake/lib/bblayers/common.py
>> @@ -1,3 +1,7 @@
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> import argparse
>> import logging
>> import os
>> diff --git a/bitbake/lib/bblayers/layerindex.py
>> b/bitbake/lib/bblayers/layerindex.py index 9f02a9d..57cd902 100644
>> --- a/bitbake/lib/bblayers/layerindex.py
>> +++ b/bitbake/lib/bblayers/layerindex.py
>> @@ -1,3 +1,7 @@
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> import layerindexlib
>>
>> import argparse
>> @@ -28,7 +32,7 @@ class LayerIndexPlugin(ActionPlugin):
>> layerdir = os.path.join(repodir, subdir)
>> if not os.path.exists(repodir):
>> if fetch_layer:
>> - result = subprocess.call('git clone %s %s' % (url,
>> repodir), shell = True)
>> + result = subprocess.call(['git', 'clone', url,
>> repodir]) if result:
>> logger.error("Failed to download %s" % url)
>> return None, None, None
>> diff --git a/bitbake/lib/bblayers/query.py
>> b/bitbake/lib/bblayers/query.py index 9294dfa..7db49c8 100644
>> --- a/bitbake/lib/bblayers/query.py
>> +++ b/bitbake/lib/bblayers/query.py
>> @@ -1,3 +1,7 @@
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> import collections
>> import fnmatch
>> import logging
>> @@ -42,7 +46,7 @@ layer, with the preferred version first. Note that
>> skipped recipes that are overlayed will also be listed, with a
>> " (skipped)" suffix. """
>>
>> - items_listed = self.list_recipes('Overlayed recipes', None,
>> True, args.same_version, args.filenames, True, None)
>> + items_listed = self.list_recipes('Overlayed recipes', None,
>> True, args.same_version, args.filenames, False, True, None, False,
>> None) # Check for overlayed .bbclass files
>> classes = collections.defaultdict(list)
>> @@ -108,9 +112,9 @@ skipped recipes will also be listed, with a
>> " (skipped)" suffix. title = 'Matching recipes:'
>> else:
>> title = 'Available recipes:'
>> - self.list_recipes(title, args.pnspec, False, False,
>> args.filenames, args.multiple, inheritlist)
>> + self.list_recipes(title, args.pnspec, False, False,
>> args.filenames, args.recipes_only, args.multiple, args.layer,
>> args.bare, inheritlist)
>> - def list_recipes(self, title, pnspec, show_overlayed_only,
>> show_same_ver_only, show_filenames, show_multi_provider_only,
>> inherits):
>> + def list_recipes(self, title, pnspec, show_overlayed_only,
>> show_same_ver_only, show_filenames, show_recipes_only,
>> show_multi_provider_only, selected_layer, bare, inherits): if
>> inherits: bbpath = str(self.tinfoil.config_data.getVar('BBPATH')) for
>> classname in inherits: @@ -140,24 +144,30 @@ skipped recipes will
>> also be listed, with a " (skipped)" suffix. preferred_versions[p] =
>> (ver, fn)
>> def print_item(f, pn, ver, layer, ispref):
>> - if f in skiplist:
>> - skipped = ' (skipped)'
>> - else:
>> - skipped = ''
>> - if show_filenames:
>> - if ispref:
>> - logger.plain("%s%s", f, skipped)
>> + if not selected_layer or layer == selected_layer:
>> + if not bare and f in skiplist:
>> + skipped = ' (skipped)'
>> else:
>> - logger.plain(" %s%s", f, skipped)
>> - else:
>> - if ispref:
>> - logger.plain("%s:", pn)
>> - logger.plain(" %s %s%s", layer.ljust(20), ver,
>> skipped)
>> + skipped = ''
>> + if show_filenames:
>> + if ispref:
>> + logger.plain("%s%s", f, skipped)
>> + else:
>> + logger.plain(" %s%s", f, skipped)
>> + elif show_recipes_only:
>> + if pn not in show_unique_pn:
>> + show_unique_pn.append(pn)
>> + logger.plain("%s%s", pn, skipped)
>> + else:
>> + if ispref:
>> + logger.plain("%s:", pn)
>> + logger.plain(" %s %s%s", layer.ljust(20), ver,
>> skipped)
>> global_inherit = (self.tinfoil.config_data.getVar('INHERIT')
>> or "").split() cls_re = re.compile('classes/')
>>
>> preffiles = []
>> + show_unique_pn = []
>> items_listed = False
>> for p in sorted(pkg_pn):
>> if pnspec:
>> @@ -489,8 +499,11 @@ NOTE: .bbappend files can impact the
>> dependencies.
>> parser_show_recipes = self.add_command(sp, 'show-recipes',
>> self.do_show_recipes) parser_show_recipes.add_argument('-f',
>> '--filenames', help='instead of the default formatting, list
>> filenames of higher priority recipes with the ones they overlay
>> indented underneath', action='store_true')
>> + parser_show_recipes.add_argument('-r', '--recipes-only',
>> help='instead of the default formatting, list recipes only',
>> action='store_true') parser_show_recipes.add_argument('-m',
>> '--multiple', help='only list where multiple recipes (in the same
>> layer or different layers) exist for the same recipe name',
>> action='store_true') parser_show_recipes.add_argument('-i',
>> '--inherits', help='only list recipes that inherit the named
>> class(es) - separate multiple classes using , (without spaces)',
>> metavar='CLASS', default='')
>> + parser_show_recipes.add_argument('-l', '--layer', help='only
>> list recipes from the selected layer', default='')
>> + parser_show_recipes.add_argument('-b', '--bare',
>> help='output just names without the "(skipped)" marker',
>> action='store_true') parser_show_recipes.add_argument('pnspec',
>> nargs='*', help='optional recipe name specification (wildcards
>> allowed, enclose in quotes to avoid shell expansion)')
>> parser_show_appends = self.add_command(sp, 'show-appends',
>> self.do_show_appends) diff --git a/bitbake/lib/bs4/dammit.py
>> b/bitbake/lib/bs4/dammit.py index 68d419f..805aa90 100644 ---
>> a/bitbake/lib/bs4/dammit.py +++ b/bitbake/lib/bs4/dammit.py @@ -45,9
>> +45,9 @@ except ImportError: pass
>>
>> xml_encoding_re = re.compile(
>> - '^<\?.*encoding=[\'"](.*?)[\'"].*\?>'.encode(), re.I)
>> + r'^<\?.*encoding=[\'"](.*?)[\'"].*\?>'.encode(), re.I)
>> html_meta_re = re.compile(
>> - '<\s*meta[^>]+charset\s*=\s*["\']?([^>]*?)[ /;\'">]'.encode(),
>> re.I)
>> + r'<\s*meta[^>]+charset\s*=\s*["\']?([^>]*?)[ /;\'">]'.encode(),
>> re.I)
>> class EntitySubstitution(object):
>>
>> @@ -80,11 +80,11 @@ class EntitySubstitution(object):
>> ">": "gt",
>> }
>>
>> - BARE_AMPERSAND_OR_BRACKET = re.compile("([<>]|"
>> -
>> "&(?!#\d+;|#x[0-9a-fA-F]+;|\w+;)"
>> - ")")
>> + BARE_AMPERSAND_OR_BRACKET = re.compile(r"([<>]|"
>> +
>> r"&(?!#\d+;|#x[0-9a-fA-F]+;|\w+;)"
>> + r")")
>>
>> - AMPERSAND_OR_BRACKET = re.compile("([<>&])")
>> + AMPERSAND_OR_BRACKET = re.compile(r"([<>&])")
>>
>> @classmethod
>> def _substitute_html_entity(cls, matchobj):
>> diff --git a/bitbake/lib/bs4/element.py b/bitbake/lib/bs4/element.py
>> index 0e62c2e..3775a60 100644
>> --- a/bitbake/lib/bs4/element.py
>> +++ b/bitbake/lib/bs4/element.py
>> @@ -1,7 +1,7 @@
>> __license__ = "MIT"
>>
>> from pdb import set_trace
>> -import collections
>> +import collections.abc
>> import re
>> import sys
>> import warnings
>> @@ -10,7 +10,7 @@ from bs4.dammit import EntitySubstitution
>> DEFAULT_OUTPUT_ENCODING = "utf-8"
>> PY3K = (sys.version_info[0] > 2)
>>
>> -whitespace_re = re.compile("\s+")
>> +whitespace_re = re.compile(r"\s+")
>>
>> def _alias(attr):
>> """Alias one attribute name to another for backward
>> compatibility""" @@ -67,7 +67,7 @@ class
>> ContentMetaAttributeValue(AttributeValueWithCharsetSubstitution): The
>> value of the 'content' attribute will be one of these objects. """
>>
>> - CHARSET_RE = re.compile("((^|;)\s*charset=)([^;]*)", re.M)
>> + CHARSET_RE = re.compile(r"((^|;)\s*charset=)([^;]*)", re.M)
>>
>> def __new__(cls, original_value):
>> match = cls.CHARSET_RE.search(original_value)
>> @@ -155,7 +155,7 @@ class PageElement(object):
>>
>> def format_string(self, s, formatter='minimal'):
>> """Format the given string using the given formatter."""
>> - if not isinstance(formatter, collections.Callable):
>> + if not isinstance(formatter, collections.abc.Callable):
>> formatter = self._formatter_for_name(formatter)
>> if formatter is None:
>> output = s
>> @@ -580,7 +580,7 @@ class PageElement(object):
>>
>> # Methods for supporting CSS selectors.
>>
>> - tag_name_re = re.compile('^[a-zA-Z0-9][-.a-zA-Z0-9:_]*$')
>> + tag_name_re = re.compile(r'^[a-zA-Z0-9][-.a-zA-Z0-9:_]*$')
>>
>> # /^([a-zA-Z0-9][-.a-zA-Z0-9:_]*)\[(\w+)([=~\|\^\$\*]?)=?"?([^\]"]*)"?\]$/
>> # \---------------------------/ \---/\-------------/
>> \-------/ @@ -1077,7 +1077,7 @@ class Tag(PageElement):
>>
>> # First off, turn a string formatter into a function. This
>> # will stop the lookup from happening over and over again.
>> - if not isinstance(formatter, collections.Callable):
>> + if not isinstance(formatter, collections.abc.Callable):
>> formatter = self._formatter_for_name(formatter)
>>
>> attrs = []
>> @@ -1181,7 +1181,7 @@ class Tag(PageElement):
>> """
>> # First off, turn a string formatter into a function. This
>> # will stop the lookup from happening over and over again.
>> - if not isinstance(formatter, collections.Callable):
>> + if not isinstance(formatter, collections.abc.Callable):
>> formatter = self._formatter_for_name(formatter)
>>
>> pretty_print = (indent_level is not None)
>> @@ -1364,7 +1364,7 @@ class Tag(PageElement):
>> if tag_name == '':
>> raise ValueError(
>> "A pseudo-class must be prefixed with a tag
>> name.")
>> - pseudo_attributes =
>> re.match('([a-zA-Z\d-]+)\(([a-zA-Z\d]+)\)', pseudo)
>> + pseudo_attributes =
>> re.match(r'([a-zA-Z\d-]+)\(([a-zA-Z\d]+)\)', pseudo) found = []
>> if pseudo_attributes is None:
>> pseudo_type = pseudo
>> @@ -1562,7 +1562,7 @@ class SoupStrainer(object):
>> def _normalize_search_value(self, value):
>> # Leave it alone if it's a Unicode string, a callable, a
>> # regular expression, a boolean, or None.
>> - if (isinstance(value, str) or isinstance(value,
>> collections.Callable) or hasattr(value, 'match')
>> + if (isinstance(value, str) or isinstance(value,
>> collections.abc.Callable) or hasattr(value, 'match') or
>> isinstance(value, bool) or value is None): return value
>>
>> @@ -1602,7 +1602,7 @@ class SoupStrainer(object):
>> markup = markup_name
>> markup_attrs = markup
>> call_function_with_tag_data = (
>> - isinstance(self.name, collections.Callable)
>> + isinstance(self.name, collections.abc.Callable)
>> and not isinstance(markup_name, Tag))
>>
>> if ((not self.name)
>> @@ -1688,7 +1688,7 @@ class SoupStrainer(object):
>> # True matches any non-None value.
>> return markup is not None
>>
>> - if isinstance(match_against, collections.Callable):
>> + if isinstance(match_against, collections.abc.Callable):
>> return match_against(markup)
>>
>> # Custom callables take the tag as an argument, but all
>> diff --git a/bitbake/lib/hashserv/__init__.py
>> b/bitbake/lib/hashserv/__init__.py new file mode 100644
>> index 0000000..c331862
>> --- /dev/null
>> +++ b/bitbake/lib/hashserv/__init__.py
>> @@ -0,0 +1,93 @@
>> +# Copyright (C) 2018-2019 Garmin Ltd.
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> +from contextlib import closing
>> +import re
>> +import sqlite3
>> +
>> +UNIX_PREFIX = "unix://"
>> +
>> +ADDR_TYPE_UNIX = 0
>> +ADDR_TYPE_TCP = 1
>> +
>> +
>> +def setup_database(database, sync=True):
>> + db = sqlite3.connect(database)
>> + db.row_factory = sqlite3.Row
>> +
>> + with closing(db.cursor()) as cursor:
>> + cursor.execute('''
>> + CREATE TABLE IF NOT EXISTS tasks_v2 (
>> + id INTEGER PRIMARY KEY AUTOINCREMENT,
>> + method TEXT NOT NULL,
>> + outhash TEXT NOT NULL,
>> + taskhash TEXT NOT NULL,
>> + unihash TEXT NOT NULL,
>> + created DATETIME,
>> +
>> + -- Optional fields
>> + owner TEXT,
>> + PN TEXT,
>> + PV TEXT,
>> + PR TEXT,
>> + task TEXT,
>> + outhash_siginfo TEXT,
>> +
>> + UNIQUE(method, outhash, taskhash)
>> + )
>> + ''')
>> + cursor.execute('PRAGMA journal_mode = WAL')
>> + cursor.execute('PRAGMA synchronous = %s' % ('NORMAL' if sync
>> else 'OFF')) +
>> + # Drop old indexes
>> + cursor.execute('DROP INDEX IF EXISTS taskhash_lookup')
>> + cursor.execute('DROP INDEX IF EXISTS outhash_lookup')
>> +
>> + # Create new indexes
>> + cursor.execute('CREATE INDEX IF NOT EXISTS
>> taskhash_lookup_v2 ON tasks_v2 (method, taskhash, created)')
>> + cursor.execute('CREATE INDEX IF NOT EXISTS outhash_lookup_v2
>> ON tasks_v2 (method, outhash)') +
>> + return db
>> +
>> +
>> +def parse_address(addr):
>> + if addr.startswith(UNIX_PREFIX):
>> + return (ADDR_TYPE_UNIX, (addr[len(UNIX_PREFIX):],))
>> + else:
>> + m = re.match(r'\[(?P<host>[^\]]*)\]:(?P<port>\d+)$', addr)
>> + if m is not None:
>> + host = m.group('host')
>> + port = m.group('port')
>> + else:
>> + host, port = addr.split(':')
>> +
>> + return (ADDR_TYPE_TCP, (host, int(port)))
>> +
>> +
>> +def create_server(addr, dbname, *, sync=True):
>> + from . import server
>> + db = setup_database(dbname, sync=sync)
>> + s = server.Server(db)
>> +
>> + (typ, a) = parse_address(addr)
>> + if typ == ADDR_TYPE_UNIX:
>> + s.start_unix_server(*a)
>> + else:
>> + s.start_tcp_server(*a)
>> +
>> + return s
>> +
>> +
>> +def create_client(addr):
>> + from . import client
>> + c = client.Client()
>> +
>> + (typ, a) = parse_address(addr)
>> + if typ == ADDR_TYPE_UNIX:
>> + c.connect_unix(*a)
>> + else:
>> + c.connect_tcp(*a)
>> +
>> + return c
>> diff --git a/bitbake/lib/hashserv/client.py
>> b/bitbake/lib/hashserv/client.py new file mode 100644
>> index 0000000..f659566
>> --- /dev/null
>> +++ b/bitbake/lib/hashserv/client.py
>> @@ -0,0 +1,157 @@
>> +# Copyright (C) 2019 Garmin Ltd.
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> +from contextlib import closing
>> +import json
>> +import logging
>> +import socket
>> +import os
>> +
>> +
>> +logger = logging.getLogger('hashserv.client')
>> +
>> +
>> +class HashConnectionError(Exception):
>> + pass
>> +
>> +
>> +class Client(object):
>> + MODE_NORMAL = 0
>> + MODE_GET_STREAM = 1
>> +
>> + def __init__(self):
>> + self._socket = None
>> + self.reader = None
>> + self.writer = None
>> + self.mode = self.MODE_NORMAL
>> +
>> + def connect_tcp(self, address, port):
>> + def connect_sock():
>> + s = socket.create_connection((address, port))
>> +
>> + s.setsockopt(socket.SOL_TCP, socket.TCP_NODELAY, 1)
>> + s.setsockopt(socket.SOL_TCP, socket.TCP_QUICKACK, 1)
>> + s.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
>> + return s
>> +
>> + self._connect_sock = connect_sock
>> +
>> + def connect_unix(self, path):
>> + def connect_sock():
>> + s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
>> + # AF_UNIX has path length issues so chdir here to
>> workaround
>> + cwd = os.getcwd()
>> + try:
>> + os.chdir(os.path.dirname(path))
>> + s.connect(os.path.basename(path))
>> + finally:
>> + os.chdir(cwd)
>> + return s
>> +
>> + self._connect_sock = connect_sock
>> +
>> + def connect(self):
>> + if self._socket is None:
>> + self._socket = self._connect_sock()
>> +
>> + self.reader = self._socket.makefile('r',
>> encoding='utf-8')
>> + self.writer = self._socket.makefile('w',
>> encoding='utf-8') +
>> + self.writer.write('OEHASHEQUIV 1.0\n\n')
>> + self.writer.flush()
>> +
>> + # Restore mode if the socket is being re-created
>> + cur_mode = self.mode
>> + self.mode = self.MODE_NORMAL
>> + self._set_mode(cur_mode)
>> +
>> + return self._socket
>> +
>> + def close(self):
>> + if self._socket is not None:
>> + self._socket.close()
>> + self._socket = None
>> + self.reader = None
>> + self.writer = None
>> +
>> + def _send_wrapper(self, proc):
>> + count = 0
>> + while True:
>> + try:
>> + self.connect()
>> + return proc()
>> + except (OSError, HashConnectionError,
>> json.JSONDecodeError, UnicodeDecodeError) as e:
>> + logger.warning('Error talking to server: %s' % e)
>> + if count >= 3:
>> + if not isinstance(e, HashConnectionError):
>> + raise HashConnectionError(str(e))
>> + raise e
>> + self.close()
>> + count += 1
>> +
>> + def send_message(self, msg):
>> + def proc():
>> + self.writer.write('%s\n' % json.dumps(msg))
>> + self.writer.flush()
>> +
>> + l = self.reader.readline()
>> + if not l:
>> + raise HashConnectionError('Connection closed')
>> +
>> + if not l.endswith('\n'):
>> + raise HashConnectionError('Bad message %r' % message)
>> +
>> + return json.loads(l)
>> +
>> + return self._send_wrapper(proc)
>> +
>> + def send_stream(self, msg):
>> + def proc():
>> + self.writer.write("%s\n" % msg)
>> + self.writer.flush()
>> + l = self.reader.readline()
>> + if not l:
>> + raise HashConnectionError('Connection closed')
>> + return l.rstrip()
>> +
>> + return self._send_wrapper(proc)
>> +
>> + def _set_mode(self, new_mode):
>> + if new_mode == self.MODE_NORMAL and self.mode ==
>> self.MODE_GET_STREAM:
>> + r = self.send_stream('END')
>> + if r != 'ok':
>> + raise HashConnectionError('Bad response from server
>> %r' % r)
>> + elif new_mode == self.MODE_GET_STREAM and self.mode ==
>> self.MODE_NORMAL:
>> + r = self.send_message({'get-stream': None})
>> + if r != 'ok':
>> + raise HashConnectionError('Bad response from server
>> %r' % r)
>> + elif new_mode != self.mode:
>> + raise Exception('Undefined mode transition %r -> %r' %
>> (self.mode, new_mode)) +
>> + self.mode = new_mode
>> +
>> + def get_unihash(self, method, taskhash):
>> + self._set_mode(self.MODE_GET_STREAM)
>> + r = self.send_stream('%s %s' % (method, taskhash))
>> + if not r:
>> + return None
>> + return r
>> +
>> + def report_unihash(self, taskhash, method, outhash, unihash,
>> extra={}):
>> + self._set_mode(self.MODE_NORMAL)
>> + m = extra.copy()
>> + m['taskhash'] = taskhash
>> + m['method'] = method
>> + m['outhash'] = outhash
>> + m['unihash'] = unihash
>> + return self.send_message({'report': m})
>> +
>> + def get_stats(self):
>> + self._set_mode(self.MODE_NORMAL)
>> + return self.send_message({'get-stats': None})
>> +
>> + def reset_stats(self):
>> + self._set_mode(self.MODE_NORMAL)
>> + return self.send_message({'reset-stats': None})
>> diff --git a/bitbake/lib/hashserv/server.py
>> b/bitbake/lib/hashserv/server.py new file mode 100644
>> index 0000000..0aff776
>> --- /dev/null
>> +++ b/bitbake/lib/hashserv/server.py
>> @@ -0,0 +1,414 @@
>> +# Copyright (C) 2019 Garmin Ltd.
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> +from contextlib import closing
>> +from datetime import datetime
>> +import asyncio
>> +import json
>> +import logging
>> +import math
>> +import os
>> +import signal
>> +import socket
>> +import time
>> +
>> +logger = logging.getLogger('hashserv.server')
>> +
>> +
>> +class Measurement(object):
>> + def __init__(self, sample):
>> + self.sample = sample
>> +
>> + def start(self):
>> + self.start_time = time.perf_counter()
>> +
>> + def end(self):
>> + self.sample.add(time.perf_counter() - self.start_time)
>> +
>> + def __enter__(self):
>> + self.start()
>> + return self
>> +
>> + def __exit__(self, *args, **kwargs):
>> + self.end()
>> +
>> +
>> +class Sample(object):
>> + def __init__(self, stats):
>> + self.stats = stats
>> + self.num_samples = 0
>> + self.elapsed = 0
>> +
>> + def measure(self):
>> + return Measurement(self)
>> +
>> + def __enter__(self):
>> + return self
>> +
>> + def __exit__(self, *args, **kwargs):
>> + self.end()
>> +
>> + def add(self, elapsed):
>> + self.num_samples += 1
>> + self.elapsed += elapsed
>> +
>> + def end(self):
>> + if self.num_samples:
>> + self.stats.add(self.elapsed)
>> + self.num_samples = 0
>> + self.elapsed = 0
>> +
>> +
>> +class Stats(object):
>> + def __init__(self):
>> + self.reset()
>> +
>> + def reset(self):
>> + self.num = 0
>> + self.total_time = 0
>> + self.max_time = 0
>> + self.m = 0
>> + self.s = 0
>> + self.current_elapsed = None
>> +
>> + def add(self, elapsed):
>> + self.num += 1
>> + if self.num == 1:
>> + self.m = elapsed
>> + self.s = 0
>> + else:
>> + last_m = self.m
>> + self.m = last_m + (elapsed - last_m) / self.num
>> + self.s = self.s + (elapsed - last_m) * (elapsed - self.m)
>> +
>> + self.total_time += elapsed
>> +
>> + if self.max_time < elapsed:
>> + self.max_time = elapsed
>> +
>> + def start_sample(self):
>> + return Sample(self)
>> +
>> + @property
>> + def average(self):
>> + if self.num == 0:
>> + return 0
>> + return self.total_time / self.num
>> +
>> + @property
>> + def stdev(self):
>> + if self.num <= 1:
>> + return 0
>> + return math.sqrt(self.s / (self.num - 1))
>> +
>> + def todict(self):
>> + return {k: getattr(self, k) for k in ('num', 'total_time',
>> 'max_time', 'average', 'stdev')} +
>> +
>> +class ServerClient(object):
>> + def __init__(self, reader, writer, db, request_stats):
>> + self.reader = reader
>> + self.writer = writer
>> + self.db = db
>> + self.request_stats = request_stats
>> +
>> + async def process_requests(self):
>> + try:
>> + self.addr = self.writer.get_extra_info('peername')
>> + logger.debug('Client %r connected' % (self.addr,))
>> +
>> + # Read protocol and version
>> + protocol = await self.reader.readline()
>> + if protocol is None:
>> + return
>> +
>> + (proto_name, proto_version) =
>> protocol.decode('utf-8').rstrip().split()
>> + if proto_name != 'OEHASHEQUIV' or proto_version != '1.0':
>> + return
>> +
>> + # Read headers. Currently, no headers are implemented,
>> so look for
>> + # an empty line to signal the end of the headers
>> + while True:
>> + line = await self.reader.readline()
>> + if line is None:
>> + return
>> +
>> + line = line.decode('utf-8').rstrip()
>> + if not line:
>> + break
>> +
>> + # Handle messages
>> + handlers = {
>> + 'get': self.handle_get,
>> + 'report': self.handle_report,
>> + 'get-stream': self.handle_get_stream,
>> + 'get-stats': self.handle_get_stats,
>> + 'reset-stats': self.handle_reset_stats,
>> + }
>> +
>> + while True:
>> + d = await self.read_message()
>> + if d is None:
>> + break
>> +
>> + for k in handlers.keys():
>> + if k in d:
>> + logger.debug('Handling %s' % k)
>> + if 'stream' in k:
>> + await handlers[k](d[k])
>> + else:
>> + with self.request_stats.start_sample()
>> as self.request_sample, \
>> + self.request_sample.measure():
>> + await handlers[k](d[k])
>> + break
>> + else:
>> + logger.warning("Unrecognized command %r" % d)
>> + break
>> +
>> + await self.writer.drain()
>> + finally:
>> + self.writer.close()
>> +
>> + def write_message(self, msg):
>> + self.writer.write(('%s\n' % json.dumps(msg)).encode('utf-8'))
>> +
>> + async def read_message(self):
>> + l = await self.reader.readline()
>> + if not l:
>> + return None
>> +
>> + try:
>> + message = l.decode('utf-8')
>> +
>> + if not message.endswith('\n'):
>> + return None
>> +
>> + return json.loads(message)
>> + except (json.JSONDecodeError, UnicodeDecodeError) as e:
>> + logger.error('Bad message from client: %r' % message)
>> + raise e
>> +
>> + async def handle_get(self, request):
>> + method = request['method']
>> + taskhash = request['taskhash']
>> +
>> + row = self.query_equivalent(method, taskhash)
>> + if row is not None:
>> + logger.debug('Found equivalent task %s -> %s',
>> (row['taskhash'], row['unihash']))
>> + d = {k: row[k] for k in ('taskhash', 'method',
>> 'unihash')} +
>> + self.write_message(d)
>> + else:
>> + self.write_message(None)
>> +
>> + async def handle_get_stream(self, request):
>> + self.write_message('ok')
>> +
>> + while True:
>> + l = await self.reader.readline()
>> + if not l:
>> + return
>> +
>> + try:
>> + # This inner loop is very sensitive and must be as
>> fast as
>> + # possible (which is why the request sample is
>> handled manually
>> + # instead of using 'with', and also why logging
>> statements are
>> + # commented out.
>> + self.request_sample =
>> self.request_stats.start_sample()
>> + request_measure = self.request_sample.measure()
>> + request_measure.start()
>> +
>> + l = l.decode('utf-8').rstrip()
>> + if l == 'END':
>> + self.writer.write('ok\n'.encode('utf-8'))
>> + return
>> +
>> + (method, taskhash) = l.split()
>> + #logger.debug('Looking up %s %s' % (method,
>> taskhash))
>> + row = self.query_equivalent(method, taskhash)
>> + if row is not None:
>> + msg = ('%s\n' % row['unihash']).encode('utf-8')
>> + #logger.debug('Found equivalent task %s -> %s',
>> (row['taskhash'], row['unihash']))
>> + else:
>> + msg = '\n'.encode('utf-8')
>> +
>> + self.writer.write(msg)
>> + finally:
>> + request_measure.end()
>> + self.request_sample.end()
>> +
>> + await self.writer.drain()
>> +
>> + async def handle_report(self, data):
>> + with closing(self.db.cursor()) as cursor:
>> + cursor.execute('''
>> + -- Find tasks with a matching outhash (that is,
>> tasks that
>> + -- are equivalent)
>> + SELECT taskhash, method, unihash FROM tasks_v2 WHERE
>> method=:method AND outhash=:outhash +
>> + -- If there is an exact match on the taskhash,
>> return it.
>> + -- Otherwise return the oldest matching outhash of
>> any
>> + -- taskhash
>> + ORDER BY CASE WHEN taskhash=:taskhash THEN 1 ELSE 2
>> END,
>> + created ASC
>> +
>> + -- Only return one row
>> + LIMIT 1
>> + ''', {k: data[k] for k in ('method', 'outhash',
>> 'taskhash')}) +
>> + row = cursor.fetchone()
>> +
>> + # If no matching outhash was found, or one *was* found
>> but it
>> + # wasn't an exact match on the taskhash, a new entry for
>> this
>> + # taskhash should be added
>> + if row is None or row['taskhash'] != data['taskhash']:
>> + # If a row matching the outhash was found, the
>> unihash for
>> + # the new taskhash should be the same as that one.
>> + # Otherwise the caller provided unihash is used.
>> + unihash = data['unihash']
>> + if row is not None:
>> + unihash = row['unihash']
>> +
>> + insert_data = {
>> + 'method': data['method'],
>> + 'outhash': data['outhash'],
>> + 'taskhash': data['taskhash'],
>> + 'unihash': unihash,
>> + 'created': datetime.now()
>> + }
>> +
>> + for k in ('owner', 'PN', 'PV', 'PR', 'task',
>> 'outhash_siginfo'):
>> + if k in data:
>> + insert_data[k] = data[k]
>> +
>> + cursor.execute('''INSERT INTO tasks_v2 (%s) VALUES
>> (%s)''' % (
>> + ', '.join(sorted(insert_data.keys())),
>> + ', '.join(':' + k for k in
>> sorted(insert_data.keys()))),
>> + insert_data)
>> +
>> + self.db.commit()
>> +
>> + logger.info('Adding taskhash %s with unihash %s',
>> + data['taskhash'], unihash)
>> +
>> + d = {
>> + 'taskhash': data['taskhash'],
>> + 'method': data['method'],
>> + 'unihash': unihash
>> + }
>> + else:
>> + d = {k: row[k] for k in ('taskhash', 'method',
>> 'unihash')} +
>> + self.write_message(d)
>> +
>> + async def handle_get_stats(self, request):
>> + d = {
>> + 'requests': self.request_stats.todict(),
>> + }
>> +
>> + self.write_message(d)
>> +
>> + async def handle_reset_stats(self, request):
>> + d = {
>> + 'requests': self.request_stats.todict(),
>> + }
>> +
>> + self.request_stats.reset()
>> + self.write_message(d)
>> +
>> + def query_equivalent(self, method, taskhash):
>> + # This is part of the inner loop and must be as fast as
>> possible
>> + try:
>> + cursor = self.db.cursor()
>> + cursor.execute('SELECT taskhash, method, unihash FROM
>> tasks_v2 WHERE method=:method AND taskhash=:taskhash ORDER BY created
>> ASC LIMIT 1',
>> + {'method': method, 'taskhash': taskhash})
>> + return cursor.fetchone()
>> + except:
>> + cursor.close()
>> +
>> +
>> +class Server(object):
>> + def __init__(self, db, loop=None):
>> + self.request_stats = Stats()
>> + self.db = db
>> +
>> + if loop is None:
>> + self.loop = asyncio.new_event_loop()
>> + self.close_loop = True
>> + else:
>> + self.loop = loop
>> + self.close_loop = False
>> +
>> + self._cleanup_socket = None
>> +
>> + def start_tcp_server(self, host, port):
>> + self.server = self.loop.run_until_complete(
>> + asyncio.start_server(self.handle_client, host, port,
>> loop=self.loop)
>> + )
>> +
>> + for s in self.server.sockets:
>> + logger.info('Listening on %r' % (s.getsockname(),))
>> + # Newer python does this automatically. Do it manually
>> here for
>> + # maximum compatibility
>> + s.setsockopt(socket.SOL_TCP, socket.TCP_NODELAY, 1)
>> + s.setsockopt(socket.SOL_TCP, socket.TCP_QUICKACK, 1)
>> +
>> + name = self.server.sockets[0].getsockname()
>> + if self.server.sockets[0].family == socket.AF_INET6:
>> + self.address = "[%s]:%d" % (name[0], name[1])
>> + else:
>> + self.address = "%s:%d" % (name[0], name[1])
>> +
>> + def start_unix_server(self, path):
>> + def cleanup():
>> + os.unlink(path)
>> +
>> + cwd = os.getcwd()
>> + try:
>> + # Work around path length limits in AF_UNIX
>> + os.chdir(os.path.dirname(path))
>> + self.server = self.loop.run_until_complete(
>> + asyncio.start_unix_server(self.handle_client,
>> os.path.basename(path), loop=self.loop)
>> + )
>> + finally:
>> + os.chdir(cwd)
>> +
>> + logger.info('Listening on %r' % path)
>> +
>> + self._cleanup_socket = cleanup
>> + self.address = "unix://%s" % os.path.abspath(path)
>> +
>> + async def handle_client(self, reader, writer):
>> + # writer.transport.set_write_buffer_limits(0)
>> + try:
>> + client = ServerClient(reader, writer, self.db,
>> self.request_stats)
>> + await client.process_requests()
>> + except Exception as e:
>> + import traceback
>> + logger.error('Error from client: %s' % str(e),
>> exc_info=True)
>> + traceback.print_exc()
>> + writer.close()
>> + logger.info('Client disconnected')
>> +
>> + def serve_forever(self):
>> + def signal_handler():
>> + self.loop.stop()
>> +
>> + self.loop.add_signal_handler(signal.SIGTERM, signal_handler)
>> +
>> + try:
>> + self.loop.run_forever()
>> + except KeyboardInterrupt:
>> + pass
>> +
>> + self.server.close()
>> + self.loop.run_until_complete(self.server.wait_closed())
>> + logger.info('Server shutting down')
>> +
>> + if self.close_loop:
>> + self.loop.close()
>> +
>> + if self._cleanup_socket is not None:
>> + self._cleanup_socket()
>> diff --git a/bitbake/lib/hashserv/tests.py
>> b/bitbake/lib/hashserv/tests.py new file mode 100644
>> index 0000000..a5472a9
>> --- /dev/null
>> +++ b/bitbake/lib/hashserv/tests.py
>> @@ -0,0 +1,142 @@
>> +#! /usr/bin/env python3
>> +#
>> +# Copyright (C) 2018-2019 Garmin Ltd.
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> +from . import create_server, create_client
>> +import hashlib
>> +import logging
>> +import multiprocessing
>> +import sys
>> +import tempfile
>> +import threading
>> +import unittest
>> +
>> +
>> +class TestHashEquivalenceServer(object):
>> + METHOD = 'TestMethod'
>> +
>> + def _run_server(self):
>> + # logging.basicConfig(level=logging.DEBUG,
>> filename='bbhashserv.log', filemode='w',
>> + # format='%(levelname)s
>> %(filename)s:%(lineno)d %(message)s')
>> + self.server.serve_forever()
>> +
>> + def setUp(self):
>> + if sys.version_info < (3, 5, 0):
>> + self.skipTest('Python 3.5 or later required')
>> +
>> + self.temp_dir =
>> tempfile.TemporaryDirectory(prefix='bb-hashserv')
>> + self.dbfile = os.path.join(self.temp_dir.name, 'db.sqlite')
>> +
>> + self.server = create_server(self.get_server_addr(),
>> self.dbfile)
>> + self.server_thread =
>> multiprocessing.Process(target=self._run_server)
>> + self.server_thread.start()
>> + self.client = create_client(self.server.address)
>> +
>> + def tearDown(self):
>> + # Shutdown server
>> + s = getattr(self, 'server', None)
>> + if s is not None:
>> + self.server_thread.terminate()
>> + self.server_thread.join()
>> + self.client.close()
>> + self.temp_dir.cleanup()
>> +
>> + def test_create_hash(self):
>> + # Simple test that hashes can be created
>> + taskhash = '35788efcb8dfb0a02659d81cf2bfd695fb30faf9'
>> + outhash =
>> '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5970796f'
>> + unihash = 'f46d3fbb439bd9b921095da657a4de906510d2cd'
>> +
>> + result = self.client.get_unihash(self.METHOD, taskhash)
>> + self.assertIsNone(result, msg='Found unexpected task, %r' %
>> result) +
>> + result = self.client.report_unihash(taskhash, self.METHOD,
>> outhash, unihash)
>> + self.assertEqual(result['unihash'], unihash, 'Server
>> returned bad unihash') +
>> + def test_create_equivalent(self):
>> + # Tests that a second reported task with the same outhash
>> will be
>> + # assigned the same unihash
>> + taskhash = '53b8dce672cb6d0c73170be43f540460bfc347b4'
>> + outhash =
>> '5a9cb1649625f0bf41fc7791b635cd9c2d7118c7f021ba87dcd03f72b67ce7a8'
>> + unihash = 'f37918cc02eb5a520b1aff86faacbc0a38124646'
>> +
>> + result = self.client.report_unihash(taskhash, self.METHOD,
>> outhash, unihash)
>> + self.assertEqual(result['unihash'], unihash, 'Server
>> returned bad unihash') +
>> + # Report a different task with the same outhash. The
>> returned unihash
>> + # should match the first task
>> + taskhash2 = '3bf6f1e89d26205aec90da04854fbdbf73afe6b4'
>> + unihash2 = 'af36b199320e611fbb16f1f277d3ee1d619ca58b'
>> + result = self.client.report_unihash(taskhash2, self.METHOD,
>> outhash, unihash2)
>> + self.assertEqual(result['unihash'], unihash, 'Server
>> returned bad unihash') +
>> + def test_duplicate_taskhash(self):
>> + # Tests that duplicate reports of the same taskhash with
>> different
>> + # outhash & unihash always return the unihash from the first
>> reported
>> + # taskhash
>> + taskhash = '8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a'
>> + outhash =
>> 'afe240a439959ce86f5e322f8c208e1fedefea9e813f2140c81af866cc9edf7e'
>> + unihash = '218e57509998197d570e2c98512d0105985dffc9'
>> + self.client.report_unihash(taskhash, self.METHOD, outhash,
>> unihash) +
>> + result = self.client.get_unihash(self.METHOD, taskhash)
>> + self.assertEqual(result, unihash)
>> +
>> + outhash2 =
>> '0904a7fe3dc712d9fd8a74a616ddca2a825a8ee97adf0bd3fc86082c7639914d'
>> + unihash2 = 'ae9a7d252735f0dafcdb10e2e02561ca3a47314c'
>> + self.client.report_unihash(taskhash, self.METHOD, outhash2,
>> unihash2) +
>> + result = self.client.get_unihash(self.METHOD, taskhash)
>> + self.assertEqual(result, unihash)
>> +
>> + outhash3 =
>> '77623a549b5b1a31e3732dfa8fe61d7ce5d44b3370f253c5360e136b852967b4'
>> + unihash3 = '9217a7d6398518e5dc002ed58f2cbbbc78696603'
>> + self.client.report_unihash(taskhash, self.METHOD, outhash3,
>> unihash3) +
>> + result = self.client.get_unihash(self.METHOD, taskhash)
>> + self.assertEqual(result, unihash)
>> +
>> + def test_stress(self):
>> + def query_server(failures):
>> + client = Client(self.server.address)
>> + try:
>> + for i in range(1000):
>> + taskhash = hashlib.sha256()
>> + taskhash.update(str(i).encode('utf-8'))
>> + taskhash = taskhash.hexdigest()
>> + result = client.get_unihash(self.METHOD,
>> taskhash)
>> + if result != taskhash:
>> + failures.append("taskhash mismatch: %s !=
>> %s" % (result, taskhash))
>> + finally:
>> + client.close()
>> +
>> + # Report hashes
>> + for i in range(1000):
>> + taskhash = hashlib.sha256()
>> + taskhash.update(str(i).encode('utf-8'))
>> + taskhash = taskhash.hexdigest()
>> + self.client.report_unihash(taskhash, self.METHOD,
>> taskhash, taskhash) +
>> + failures = []
>> + threads = [threading.Thread(target=query_server,
>> args=(failures,)) for t in range(100)] +
>> + for t in threads:
>> + t.start()
>> +
>> + for t in threads:
>> + t.join()
>> +
>> + self.assertFalse(failures)
>> +
>> +
>> +class TestHashEquivalenceUnixServer(TestHashEquivalenceServer,
>> unittest.TestCase):
>> + def get_server_addr(self):
>> + return "unix://" + os.path.join(self.temp_dir.name, 'sock')
>> +
>> +
>> +class TestHashEquivalenceTCPServer(TestHashEquivalenceServer,
>> unittest.TestCase):
>> + def get_server_addr(self):
>> + return "localhost:0"
>> diff --git a/bitbake/lib/layerindexlib/__init__.py
>> b/bitbake/lib/layerindexlib/__init__.py index cb79cb3..77196b4 100644
>> --- a/bitbake/lib/layerindexlib/__init__.py
>> +++ b/bitbake/lib/layerindexlib/__init__.py
>> @@ -1,17 +1,7 @@
>> # Copyright (C) 2016-2018 Wind River Systems, Inc.
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>> -# See the GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> -# along with this program; if not, write to the Free Software
>> -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
>> 02111-1307 USA
>> import datetime
>>
>> @@ -386,7 +376,7 @@ layerBranches set. If not, they are effectively
>> blank.''' invalid.append(name)
>>
>>
>> - def _resolve_dependencies(layerbranches, ignores,
>> dependencies, invalid):
>> + def _resolve_dependencies(layerbranches, ignores,
>> dependencies, invalid, processed=None): for layerbranch in
>> layerbranches: if ignores and layerbranch.layer.name in ignores:
>> continue
>> @@ -398,6 +388,13 @@ layerBranches set. If not, they are effectively
>> blank.''' if ignores and deplayerbranch.layer.name in ignores:
>> continue
>>
>> + # Since this is depth first, we need to know
>> what we're currently processing
>> + # in order to avoid infinite recursion on a loop.
>> + if processed and deplayerbranch.layer.name in
>> processed:
>> + # We have found a recursion...
>> + logger.warning('Circular layer dependency
>> found: %s -> %s' % (processed, deplayerbranch.layer.name))
>> + continue
>> +
>> # This little block is why we can't re-use the
>> LayerIndexObj version, # we must be able to satisfy each dependencies
>> across layer indexes and # use the layer index order for priority.
>> (r stands for replacement below) @@ -421,7 +418,17 @@ layerBranches
>> set. If not, they are effectively blank.'''
>> # New dependency, we need to resolve it now...
>> depth-first if deplayerbranch.layer.name not in dependencies:
>> - (dependencies, invalid) =
>> _resolve_dependencies([deplayerbranch], ignores, dependencies,
>> invalid)
>> + # Avoid recursion on this branch.
>> + # We copy so we don't end up polluting the
>> depth-first branch with other
>> + # branches. Duplication between individual
>> branches IS expected and
>> + # handled by 'dependencies' processing.
>> + if not processed:
>> + local_processed = []
>> + else:
>> + local_processed = processed.copy()
>> +
>> local_processed.append(deplayerbranch.layer.name) +
>> + (dependencies, invalid) =
>> _resolve_dependencies([deplayerbranch], ignores, dependencies,
>> invalid, local_processed) if deplayerbranch.layer.name not in
>> dependencies: dependencies[deplayerbranch.layer.name] =
>> [deplayerbranch, layerdependency] diff --git
>> a/bitbake/lib/layerindexlib/cooker.py
>> b/bitbake/lib/layerindexlib/cooker.py index 848f0e2..604a961 100644
>> --- a/bitbake/lib/layerindexlib/cooker.py +++
>> b/bitbake/lib/layerindexlib/cooker.py @@ -1,17 +1,7 @@
>> # Copyright (C) 2016-2018 Wind River Systems, Inc.
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>> -# See the GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> -# along with this program; if not, write to the Free Software
>> -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
>> 02111-1307 USA
>> import logging
>> import json
>> diff --git a/bitbake/lib/layerindexlib/plugin.py
>> b/bitbake/lib/layerindexlib/plugin.py index 92a2e97..7015a1a 100644
>> --- a/bitbake/lib/layerindexlib/plugin.py
>> +++ b/bitbake/lib/layerindexlib/plugin.py
>> @@ -1,18 +1,7 @@
>> # Copyright (C) 2016-2018 Wind River Systems, Inc.
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>> -# See the GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> -# along with this program; if not, write to the Free Software
>> -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
>> 02111-1307 USA -
>> # The file contains:
>> # LayerIndex exceptions
>> # Plugin base class
>> diff --git a/bitbake/lib/layerindexlib/restapi.py
>> b/bitbake/lib/layerindexlib/restapi.py index d08eb20..21fd144 100644
>> --- a/bitbake/lib/layerindexlib/restapi.py
>> +++ b/bitbake/lib/layerindexlib/restapi.py
>> @@ -1,17 +1,7 @@
>> # Copyright (C) 2016-2018 Wind River Systems, Inc.
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>> -# See the GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> -# along with this program; if not, write to the Free Software
>> -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
>> 02111-1307 USA
>> import logging
>> import json
>> diff --git a/bitbake/lib/layerindexlib/tests/common.py
>> b/bitbake/lib/layerindexlib/tests/common.py index 22a5458..077382f
>> 100644 --- a/bitbake/lib/layerindexlib/tests/common.py
>> +++ b/bitbake/lib/layerindexlib/tests/common.py
>> @@ -1,17 +1,7 @@
>> # Copyright (C) 2017-2018 Wind River Systems, Inc.
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>> -# See the GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> -# along with this program; if not, write to the Free Software
>> -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
>> 02111-1307 USA
>> import unittest
>> import tempfile
>> diff --git a/bitbake/lib/layerindexlib/tests/cooker.py
>> b/bitbake/lib/layerindexlib/tests/cooker.py index fdbf091..1fa102e
>> 100644 --- a/bitbake/lib/layerindexlib/tests/cooker.py
>> +++ b/bitbake/lib/layerindexlib/tests/cooker.py
>> @@ -1,17 +1,7 @@
>> # Copyright (C) 2018 Wind River Systems, Inc.
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>> -# See the GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> -# along with this program; if not, write to the Free Software
>> -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
>> 02111-1307 USA
>> import unittest
>> import tempfile
>> diff --git a/bitbake/lib/layerindexlib/tests/layerindexobj.py
>> b/bitbake/lib/layerindexlib/tests/layerindexobj.py index
>> e2fbb95..0c5ec88 100644 ---
>> a/bitbake/lib/layerindexlib/tests/layerindexobj.py +++
>> b/bitbake/lib/layerindexlib/tests/layerindexobj.py @@ -1,17 +1,7 @@
>> # Copyright (C) 2017-2018 Wind River Systems, Inc.
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>> -# See the GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> -# along with this program; if not, write to the Free Software
>> -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
>> 02111-1307 USA
>> import unittest
>> import tempfile
>> diff --git a/bitbake/lib/layerindexlib/tests/restapi.py
>> b/bitbake/lib/layerindexlib/tests/restapi.py index 5876695..6d8dc00
>> 100644 --- a/bitbake/lib/layerindexlib/tests/restapi.py
>> +++ b/bitbake/lib/layerindexlib/tests/restapi.py
>> @@ -1,17 +1,7 @@
>> # Copyright (C) 2017-2018 Wind River Systems, Inc.
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>> -# See the GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> -# along with this program; if not, write to the Free Software
>> -# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
>> 02111-1307 USA
>> import unittest
>> import tempfile
>> diff --git a/bitbake/lib/progressbar/__init__.py
>> b/bitbake/lib/progressbar/__init__.py index fbab744..c545a62 100644
>> --- a/bitbake/lib/progressbar/__init__.py
>> +++ b/bitbake/lib/progressbar/__init__.py
>> @@ -4,6 +4,8 @@
>> # progressbar - Text progress bar library for Python.
>> # Copyright (c) 2005 Nilton Volpato
>> #
>> +# SPDX-License-Identifier: LGPL-2.1-or-later OR BSD-3-Clause-Clear
>> +#
>> # This library is free software; you can redistribute it and/or
>> # modify it under the terms of the GNU Lesser General Public
>> # License as published by the Free Software Foundation; either
>> diff --git a/bitbake/lib/progressbar/compat.py
>> b/bitbake/lib/progressbar/compat.py index a39f4a1..9804e0b 100644
>> --- a/bitbake/lib/progressbar/compat.py
>> +++ b/bitbake/lib/progressbar/compat.py
>> @@ -3,6 +3,8 @@
>> # progressbar - Text progress bar library for Python.
>> # Copyright (c) 2005 Nilton Volpato
>> #
>> +# SPDX-License-Identifier: LGPL-2.1-or-later OR BSD-3-Clause-Clear
>> +#
>> # This library is free software; you can redistribute it and/or
>> # modify it under the terms of the GNU Lesser General Public
>> # License as published by the Free Software Foundation; either
>> diff --git a/bitbake/lib/progressbar/progressbar.py
>> b/bitbake/lib/progressbar/progressbar.py index 2873ad6..e2b6ba1 100644
>> --- a/bitbake/lib/progressbar/progressbar.py
>> +++ b/bitbake/lib/progressbar/progressbar.py
>> @@ -5,6 +5,8 @@
>> #
>> # (With some small changes after importing into BitBake)
>> #
>> +# SPDX-License-Identifier: LGPL-2.1-or-later OR BSD-3-Clause-Clear
>> +#
>> # This library is free software; you can redistribute it and/or
>> # modify it under the terms of the GNU Lesser General Public
>> # License as published by the Free Software Foundation; either
>> diff --git a/bitbake/lib/progressbar/widgets.py
>> b/bitbake/lib/progressbar/widgets.py index 77285ca..0772aa5 100644
>> --- a/bitbake/lib/progressbar/widgets.py
>> +++ b/bitbake/lib/progressbar/widgets.py
>> @@ -3,6 +3,8 @@
>> # progressbar - Text progress bar library for Python.
>> # Copyright (c) 2005 Nilton Volpato
>> #
>> +# SPDX-License-Identifier: LGPL-2.1-or-later OR BSD-3-Clause-Clear
>> +#
>> # This library is free software; you can redistribute it and/or
>> # modify it under the terms of the GNU Lesser General Public
>> # License as published by the Free Software Foundation; either
>> diff --git a/bitbake/lib/prserv/__init__.py
>> b/bitbake/lib/prserv/__init__.py index c3cb73a..9961040 100644
>> --- a/bitbake/lib/prserv/__init__.py
>> +++ b/bitbake/lib/prserv/__init__.py
>> @@ -1,3 +1,7 @@
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> __version__ = "1.0.0"
>>
>> import os, time
>> diff --git a/bitbake/lib/prserv/db.py b/bitbake/lib/prserv/db.py
>> index 495d09f..117d8c0 100644
>> --- a/bitbake/lib/prserv/db.py
>> +++ b/bitbake/lib/prserv/db.py
>> @@ -1,3 +1,7 @@
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> import logging
>> import os.path
>> import errno
>> @@ -253,7 +257,7 @@ class PRData(object):
>> self.connection=sqlite3.connect(self.filename,
>> isolation_level="EXCLUSIVE", check_same_thread = False)
>> self.connection.row_factory=sqlite3.Row
>> self.connection.execute("pragma synchronous = off;")
>> - self.connection.execute("PRAGMA journal_mode = WAL;")
>> + self.connection.execute("PRAGMA journal_mode = MEMORY;")
>> self._tables={}
>>
>> def disconnect(self):
>> diff --git a/bitbake/lib/prserv/serv.py b/bitbake/lib/prserv/serv.py
>> index 6a99728..be3acec 100644
>> --- a/bitbake/lib/prserv/serv.py
>> +++ b/bitbake/lib/prserv/serv.py
>> @@ -1,3 +1,7 @@
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> import os,sys,logging
>> import signal, time
>> from xmlrpc.server import SimpleXMLRPCServer,
>> SimpleXMLRPCRequestHandler diff --git a/bitbake/lib/pyinotify.py
>> b/bitbake/lib/pyinotify.py index 4eb03b0..1528a22 100644
>> --- a/bitbake/lib/pyinotify.py
>> +++ b/bitbake/lib/pyinotify.py
>> @@ -1,25 +1,9 @@
>> -#!/usr/bin/env python
>> -
>> +#
>> # pyinotify.py - python interface to inotify
>> # Copyright (c) 2005-2015 Sebastien Martini <seb@dbzteam.org>
>> #
>> -# Permission is hereby granted, free of charge, to any person
>> obtaining a copy -# of this software and associated documentation
>> files (the "Software"), to deal -# in the Software without
>> restriction, including without limitation the rights -# to use, copy,
>> modify, merge, publish, distribute, sublicense, and/or sell -# copies
>> of the Software, and to permit persons to whom the Software is -#
>> furnished to do so, subject to the following conditions: -#
>> -# The above copyright notice and this permission notice shall be
>> included in -# all copies or substantial portions of the Software.
>> +# SPDX-License-Identifier: MIT
>> #
>> -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
>> EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
>> MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND
>> NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT
>> HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY,
>> WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -#
>> OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
>> DEALINGS IN -# THE SOFTWARE. """
>> pyinotify
>>
>> diff --git a/bitbake/lib/toaster/bldcollector/admin.py
>> b/bitbake/lib/toaster/bldcollector/admin.py index 1f2e07f..feaa888
>> 100644 --- a/bitbake/lib/toaster/bldcollector/admin.py
>> +++ b/bitbake/lib/toaster/bldcollector/admin.py
>> @@ -1,3 +1,7 @@
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> from django.contrib import admin
>> from orm.models import BitbakeVersion, Release, ToasterSetting,
>> Layer_Version from django import forms
>> diff --git a/bitbake/lib/toaster/bldcollector/urls.py
>> b/bitbake/lib/toaster/bldcollector/urls.py index 888175d..8eb1e34
>> 100644 --- a/bitbake/lib/toaster/bldcollector/urls.py
>> +++ b/bitbake/lib/toaster/bldcollector/urls.py
>> @@ -3,19 +3,8 @@
>> #
>> # Copyright (C) 2014-2017 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. -
>>
>> from django.conf.urls import include, url
>>
>> diff --git a/bitbake/lib/toaster/bldcollector/views.py
>> b/bitbake/lib/toaster/bldcollector/views.py index f32fa4d..c708b41
>> 100644 --- a/bitbake/lib/toaster/bldcollector/views.py
>> +++ b/bitbake/lib/toaster/bldcollector/views.py
>> @@ -3,18 +3,8 @@
>> #
>> # Copyright (C) 2014 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from django.views.decorators.cache import cache_control
>> from django.core.urlresolvers import reverse
>> diff --git a/bitbake/lib/toaster/bldcontrol/admin.py
>> b/bitbake/lib/toaster/bldcontrol/admin.py index fcbe5f5..e85c30e
>> 100644 --- a/bitbake/lib/toaster/bldcontrol/admin.py
>> +++ b/bitbake/lib/toaster/bldcontrol/admin.py
>> @@ -1,3 +1,7 @@
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> from django.contrib import admin
>> from django.contrib.admin.filters import RelatedFieldListFilter
>> from .models import BuildEnvironment
>> diff --git a/bitbake/lib/toaster/bldcontrol/bbcontroller.py
>> b/bitbake/lib/toaster/bldcontrol/bbcontroller.py index
>> 5195600..301df18 100644 ---
>> a/bitbake/lib/toaster/bldcontrol/bbcontroller.py +++
>> b/bitbake/lib/toaster/bldcontrol/bbcontroller.py @@ -1,24 +1,10 @@
>> #
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> -#
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2014 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. -
>>
>> import os
>> import sys
>> diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
>> b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py index
>> 9490635..39ea736 100644 ---
>> a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py +++
>> b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py @@ -1,24
>> +1,10 @@ #
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> -#
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2014 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. -
>>
>> import os
>> import sys
>> diff --git
>> a/bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py
>> b/bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py
>> index 14298d9..fe2c4dc 100644 ---
>> a/bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py
>> +++
>> b/bitbake/lib/toaster/bldcontrol/management/commands/checksettings.py
>> @@ -1,3 +1,7 @@ +# +# SPDX-License-Identifier: GPL-2.0-only +#
>> +
>> from django.core.management.base import BaseCommand, CommandError
>> from django.db import transaction
>>
>> diff --git
>> a/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py
>> b/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py
>> index 6a55dd4..50ec409 100644 ---
>> a/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py +++
>> b/bitbake/lib/toaster/bldcontrol/management/commands/runbuilds.py @@
>> -1,3 +1,7 @@ +# +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> from django.core.management.base import BaseCommand
>> from django.db import transaction
>> from django.db.models import Q
>> diff --git a/bitbake/lib/toaster/bldcontrol/models.py
>> b/bitbake/lib/toaster/bldcontrol/models.py index 409614b..bcffcf5
>> 100644 --- a/bitbake/lib/toaster/bldcontrol/models.py
>> +++ b/bitbake/lib/toaster/bldcontrol/models.py
>> @@ -1,3 +1,7 @@
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> from __future__ import unicode_literals
>> from django.db import models
>> from django.core.validators import MaxValueValidator,
>> MinValueValidator diff --git
>> a/bitbake/lib/toaster/bldcontrol/views.py
>> b/bitbake/lib/toaster/bldcontrol/views.py index 60f00ef..286d88b
>> 100644 --- a/bitbake/lib/toaster/bldcontrol/views.py +++
>> b/bitbake/lib/toaster/bldcontrol/views.py @@ -1 +1,5 @@
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> # Create your views here.
>> diff --git a/bitbake/lib/toaster/manage.py
>> b/bitbake/lib/toaster/manage.py index 0c7ea50..ae32619 100755
>> --- a/bitbake/lib/toaster/manage.py
>> +++ b/bitbake/lib/toaster/manage.py
>> @@ -1,4 +1,8 @@
>> #!/usr/bin/env python3
>> +#
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> import os
>> import sys
>>
>> diff --git a/bitbake/lib/toaster/orm/fixtures/oe-core.xml
>> b/bitbake/lib/toaster/orm/fixtures/oe-core.xml index fec93ab..a723f5a
>> 100644 --- a/bitbake/lib/toaster/orm/fixtures/oe-core.xml
>> +++ b/bitbake/lib/toaster/orm/fixtures/oe-core.xml
>> @@ -8,9 +8,9 @@
>>
>> <!-- Bitbake versions which correspond to the metadata release -->
>> <object model="orm.bitbakeversion" pk="1">
>> - <field type="CharField" name="name">sumo</field>
>> + <field type="CharField" name="name">warrior</field>
>> <field type="CharField"
>> name="giturl">git://git.openembedded.org/bitbake</field>
>> - <field type="CharField" name="branch">1.38</field>
>> + <field type="CharField" name="branch">1.42</field>
>> </object>
>> <object model="orm.bitbakeversion" pk="2">
>> <field type="CharField" name="name">HEAD</field>
>> @@ -23,18 +23,18 @@
>> <field type="CharField" name="branch">master</field>
>> </object>
>> <object model="orm.bitbakeversion" pk="4">
>> - <field type="CharField" name="name">thud</field>
>> + <field type="CharField" name="name">zeus</field>
>> <field type="CharField"
>> name="giturl">git://git.openembedded.org/bitbake</field>
>> - <field type="CharField" name="branch">1.40</field>
>> + <field type="CharField" name="branch">1.44</field>
>> </object>
>>
>> <!-- Releases available -->
>> <object model="orm.release" pk="1">
>> - <field type="CharField" name="name">sumo</field>
>> - <field type="CharField" name="description">Openembedded
>> Sumo</field>
>> + <field type="CharField" name="name">warrior</field>
>> + <field type="CharField" name="description">Openembedded
>> Warrior</field> <field rel="ManyToOneRel" to="orm.bitbakeversion"
>> name="bitbake_version">1</field>
>> - <field type="CharField" name="branch_name">sumo</field>
>> - <field type="TextField" name="helptext">Toaster will run your
>> builds using the tip of the <a
>> href=\"http://cgit.openembedded.org/openembedded-core/log/?h=sumo\">OpenEmbedded
>> Sumo</a> branch.</field>
>> + <field type="CharField" name="branch_name">warrior</field>
>> + <field type="TextField" name="helptext">Toaster will run your
>> builds using the tip of the <a
>> href=\"http://cgit.openembedded.org/openembedded-core/log/?h=warrior\">OpenEmbedded
>> Warrior</a> branch.</field> </object> <object
>> model="orm.release" pk="2"> <field type="CharField"
>> name="name">local</field> @@ -51,11 +51,11 @@ <field type="TextField"
>> name="helptext">Toaster will run your builds using the tip of the
>> <a
>> href=\"http://cgit.openembedded.org/openembedded-core/log/\">OpenEmbedded
>> master</a> branch.</field> </object> <object
>> model="orm.release" pk="4">
>> - <field type="CharField" name="name">thud</field>
>> - <field type="CharField" name="description">Openembedded
>> Rocko</field>
>> - <field rel="ManyToOneRel" to="orm.bitbakeversion"
>> name="bitbake_version">1</field>
>> - <field type="CharField" name="branch_name">thud</field>
>> - <field type="TextField" name="helptext">Toaster will run your
>> builds using the tip of the <a
>> href=\"http://cgit.openembedded.org/openembedded-core/log/?h=thud\">OpenEmbedded
>> Thud</a> branch.</field>
>> + <field type="CharField" name="name">zeus</field>
>> + <field type="CharField" name="description">Openembedded
>> Zeus</field>
>> + <field rel="ManyToOneRel" to="orm.bitbakeversion"
>> name="bitbake_version">4</field>
>> + <field type="CharField" name="branch_name">zeus</field>
>> + <field type="TextField" name="helptext">Toaster will run your
>> builds using the tip of the <a
>> href=\"http://cgit.openembedded.org/openembedded-core/log/?h=zeus\">OpenEmbedded
>> Zeus</a> branch.</field> </object> <!-- Default layers for each
>> release --> diff --git a/bitbake/lib/toaster/orm/fixtures/poky.xml
>> b/bitbake/lib/toaster/orm/fixtures/poky.xml index fb9a771..7992383
>> 100644 --- a/bitbake/lib/toaster/orm/fixtures/poky.xml
>> +++ b/bitbake/lib/toaster/orm/fixtures/poky.xml
>> @@ -8,9 +8,9 @@
>>
>> <!-- Bitbake versions which correspond to the metadata release -->
>> <object model="orm.bitbakeversion" pk="1">
>> - <field type="CharField" name="name">sumo</field>
>> + <field type="CharField" name="name">warrior</field>
>> <field type="CharField"
>> name="giturl">git://git.yoctoproject.org/poky</field>
>> - <field type="CharField" name="branch">sumo</field>
>> + <field type="CharField" name="branch">warrior</field>
>> <field type="CharField" name="dirpath">bitbake</field>
>> </object>
>> <object model="orm.bitbakeversion" pk="2">
>> @@ -26,20 +26,20 @@
>> <field type="CharField" name="dirpath">bitbake</field>
>> </object>
>> <object model="orm.bitbakeversion" pk="4">
>> - <field type="CharField" name="name">thud</field>
>> + <field type="CharField" name="name">zeus</field>
>> <field type="CharField"
>> name="giturl">git://git.yoctoproject.org/poky</field>
>> - <field type="CharField" name="branch">thud</field>
>> + <field type="CharField" name="branch">zeus</field>
>> <field type="CharField" name="dirpath">bitbake</field>
>> </object>
>>
>>
>> <!-- Releases available -->
>> <object model="orm.release" pk="1">
>> - <field type="CharField" name="name">sumo</field>
>> - <field type="CharField" name="description">Yocto Project 2.5
>> "Sumo"</field>
>> + <field type="CharField" name="name">warrior</field>
>> + <field type="CharField" name="description">Yocto Project 2.7
>> "Warrior"</field> <field rel="ManyToOneRel" to="orm.bitbakeversion"
>> name="bitbake_version">1</field>
>> - <field type="CharField" name="branch_name">sumo</field>
>> - <field type="TextField" name="helptext">Toaster will run your
>> builds using the tip of the <a
>> href="http://git.yoctoproject.org/cgit/cgit.cgi/poky/log/?h=sumo">Yocto
>> Project Sumo branch</a>.</field>
>> + <field type="CharField" name="branch_name">warrior</field>
>> + <field type="TextField" name="helptext">Toaster will run your
>> builds using the tip of the <a
>> href="http://git.yoctoproject.org/cgit/cgit.cgi/poky/log/?h=warrior">Yocto
>> Project Warrior branch</a>.</field> </object> <object
>> model="orm.release" pk="2"> <field type="CharField"
>> name="name">local</field> @@ -56,11 +56,11 @@ <field type="TextField"
>> name="helptext">Toaster will run your builds using the tip of the
>> <a
>> href="http://git.yoctoproject.org/cgit/cgit.cgi/poky/log/">Yocto
>> Project Master branch</a>.</field> </object> <object
>> model="orm.release" pk="4">
>> - <field type="CharField" name="name">rocko</field>
>> - <field type="CharField" name="description">Yocto Project 2.6
>> "Thud"</field>
>> - <field rel="ManyToOneRel" to="orm.bitbakeversion"
>> name="bitbake_version">1</field>
>> - <field type="CharField" name="branch_name">thud</field>
>> - <field type="TextField" name="helptext">Toaster will run your
>> builds using the tip of the <a
>> href="http://git.yoctoproject.org/cgit/cgit.cgi/poky/log/?h=thud">Yocto
>> Project Thud branch</a>.</field>
>> + <field type="CharField" name="name">zeus</field>
>> + <field type="CharField" name="description">Yocto Project 3.0
>> "Zeus"</field>
>> + <field rel="ManyToOneRel" to="orm.bitbakeversion"
>> name="bitbake_version">4</field>
>> + <field type="CharField" name="branch_name">zeus</field>
>> + <field type="TextField" name="helptext">Toaster will run your
>> builds using the tip of the <a
>> href="http://git.yoctoproject.org/cgit/cgit.cgi/poky/log/?h=zeus">Yocto
>> Project Zeus branch</a>.</field> </object> <!-- Default project
>> layers for each release --> @@ -130,7 +130,7 @@
>> <field rel="ManyToOneRel" to="orm.layer" name="layer">1</field>
>> <field type="IntegerField" name="layer_source">0</field>
>> <field rel="ManyToOneRel" to="orm.release"
>> name="release">1</field>
>> - <field type="CharField" name="branch">sumo</field>
>> + <field type="CharField" name="branch">warrior</field>
>> <field type="CharField" name="dirpath">meta</field>
>> </object>
>> <object model="orm.layer_version" pk="2">
>> @@ -152,7 +152,7 @@
>> <field rel="ManyToOneRel" to="orm.layer" name="layer">1</field>
>> <field type="IntegerField" name="layer_source">0</field>
>> <field rel="ManyToOneRel" to="orm.release"
>> name="release">4</field>
>> - <field type="CharField" name="branch">rocko</field>
>> + <field type="CharField" name="branch">zeus</field>
>> <field type="CharField" name="dirpath">meta</field>
>> </object>
>>
>> @@ -168,7 +168,7 @@
>> <field rel="ManyToOneRel" to="orm.layer" name="layer">2</field>
>> <field type="IntegerField" name="layer_source">0</field>
>> <field rel="ManyToOneRel" to="orm.release"
>> name="release">1</field>
>> - <field type="CharField" name="branch">sumo</field>
>> + <field type="CharField" name="branch">warrior</field>
>> <field type="CharField" name="dirpath">meta-poky</field>
>> </object>
>> <object model="orm.layer_version" pk="6">
>> @@ -190,7 +190,7 @@
>> <field rel="ManyToOneRel" to="orm.layer" name="layer">2</field>
>> <field type="IntegerField" name="layer_source">0</field>
>> <field rel="ManyToOneRel" to="orm.release"
>> name="release">4</field>
>> - <field type="CharField" name="branch">rocko</field>
>> + <field type="CharField" name="branch">zeus</field>
>> <field type="CharField" name="dirpath">meta-poky</field>
>> </object>
>>
>> @@ -206,7 +206,7 @@
>> <field rel="ManyToOneRel" to="orm.layer" name="layer">3</field>
>> <field type="IntegerField" name="layer_source">0</field>
>> <field rel="ManyToOneRel" to="orm.release"
>> name="release">1</field>
>> - <field type="CharField" name="branch">sumo</field>
>> + <field type="CharField" name="branch">warrior</field>
>> <field type="CharField" name="dirpath">meta-yocto-bsp</field>
>> </object>
>> <object model="orm.layer_version" pk="10">
>> @@ -228,7 +228,7 @@
>> <field rel="ManyToOneRel" to="orm.layer" name="layer">3</field>
>> <field type="IntegerField" name="layer_source">0</field>
>> <field rel="ManyToOneRel" to="orm.release"
>> name="release">4</field>
>> - <field type="CharField" name="branch">rocko</field>
>> + <field type="CharField" name="branch">zeus</field>
>> <field type="CharField" name="dirpath">meta-yocto-bsp</field>
>> </object>
>> </django-objects>
>> diff --git a/bitbake/lib/toaster/orm/management/commands/lsupdates.py
>> b/bitbake/lib/toaster/orm/management/commands/lsupdates.py index
>> 66114ff..5b5abbb 100644 ---
>> a/bitbake/lib/toaster/orm/management/commands/lsupdates.py +++
>> b/bitbake/lib/toaster/orm/management/commands/lsupdates.py @@ -1,23
>> +1,10 @@ #
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> -#
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2016-2017 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from django.core.management.base import BaseCommand
>>
>> diff --git a/bitbake/lib/toaster/orm/models.py
>> b/bitbake/lib/toaster/orm/models.py index 7720290..bb6b5de 100644
>> --- a/bitbake/lib/toaster/orm/models.py
>> +++ b/bitbake/lib/toaster/orm/models.py
>> @@ -1,23 +1,10 @@
>> #
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> -#
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from __future__ import unicode_literals
>>
>> @@ -978,12 +965,12 @@ class TargetSDKFile(models.Model):
>> class Target_Image_File(models.Model):
>> # valid suffixes for image files produced by a build
>> SUFFIXES = {
>> - 'btrfs', 'cpio', 'cpio.gz', 'cpio.lz4', 'cpio.lzma',
>> 'cpio.xz',
>> - 'cramfs', 'elf', 'ext2', 'ext2.bz2', 'ext2.gz', 'ext2.lzma',
>> 'ext4',
>> - 'ext4.gz', 'ext3', 'ext3.gz', 'hdddirect', 'hddimg', 'iso',
>> 'jffs2',
>> - 'jffs2.sum', 'multiubi', 'qcow2', 'squashfs', 'squashfs-lzo',
>> + 'btrfs', 'container', 'cpio', 'cpio.gz', 'cpio.lz4',
>> 'cpio.lzma',
>> + 'cpio.xz', 'cramfs', 'ext2', 'ext2.bz2', 'ext2.gz',
>> 'ext2.lzma',
>> + 'ext3', 'ext3.gz', 'ext4', 'ext4.gz', 'f2fs', 'hddimg',
>> 'iso', 'jffs2',
>> + 'jffs2.sum', 'multiubi', 'squashfs', 'squashfs-lz4',
>> 'squashfs-lzo', 'squashfs-xz', 'tar', 'tar.bz2', 'tar.gz', 'tar.lz4',
>> 'tar.xz', 'ubi',
>> - 'ubifs', 'vdi', 'vmdk', 'wic', 'wic.bmap', 'wic.bz2',
>> 'wic.gz', 'wic.lzma'
>> + 'ubifs', 'wic', 'wic.bz2', 'wic.gz', 'wic.lzma'
>> }
>>
>> target = models.ForeignKey(Target)
>> diff --git a/bitbake/lib/toaster/tests/browser/selenium_helpers.py
>> b/bitbake/lib/toaster/tests/browser/selenium_helpers.py index
>> 08711e4..02d4f4b 100644 ---
>> a/bitbake/lib/toaster/tests/browser/selenium_helpers.py +++
>> b/bitbake/lib/toaster/tests/browser/selenium_helpers.py @@ -1,23
>> +1,10 @@ -#! /usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> # The Wait class and some of SeleniumDriverHelper and
>> SeleniumTestCase are # modified from Patchwork, released under the
>> same licence terms as Toaster: diff --git
>> a/bitbake/lib/toaster/tests/browser/selenium_helpers_base.py
>> b/bitbake/lib/toaster/tests/browser/selenium_helpers_base.py index
>> 156d639..6c94684 100644 ---
>> a/bitbake/lib/toaster/tests/browser/selenium_helpers_base.py +++
>> b/bitbake/lib/toaster/tests/browser/selenium_helpers_base.py @@ -1,23
>> +1,10 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> # The Wait class and some of SeleniumDriverHelper and
>> SeleniumTestCase are # modified from Patchwork, released under the
>> same licence terms as Toaster: diff --git
>> a/bitbake/lib/toaster/tests/browser/test_all_builds_page.py
>> b/bitbake/lib/toaster/tests/browser/test_all_builds_page.py index
>> b86f29b..fba627b 100644 ---
>> a/bitbake/lib/toaster/tests/browser/test_all_builds_page.py +++
>> b/bitbake/lib/toaster/tests/browser/test_all_builds_page.py @@ -1,23
>> +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import re
>>
>> diff --git
>> a/bitbake/lib/toaster/tests/browser/test_all_projects_page.py
>> b/bitbake/lib/toaster/tests/browser/test_all_projects_page.py index
>> 44da640..afd2d35 100644 ---
>> a/bitbake/lib/toaster/tests/browser/test_all_projects_page.py +++
>> b/bitbake/lib/toaster/tests/browser/test_all_projects_page.py @@
>> -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import re
>>
>> diff --git
>> a/bitbake/lib/toaster/tests/browser/test_builddashboard_page.py
>> b/bitbake/lib/toaster/tests/browser/test_builddashboard_page.py index
>> f8ccb54..d972aff 100644 ---
>> a/bitbake/lib/toaster/tests/browser/test_builddashboard_page.py +++
>> b/bitbake/lib/toaster/tests/browser/test_builddashboard_page.py @@
>> -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from django.core.urlresolvers import reverse
>> from django.utils import timezone
>> diff --git
>> a/bitbake/lib/toaster/tests/browser/test_builddashboard_page_artifacts.py
>> b/bitbake/lib/toaster/tests/browser/test_builddashboard_page_artifacts.py
>> index 1c627ad..e2623e8 100644 ---
>> a/bitbake/lib/toaster/tests/browser/test_builddashboard_page_artifacts.py
>> +++
>> b/bitbake/lib/toaster/tests/browser/test_builddashboard_page_artifacts.py
>> @@ -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et -#
>> -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3 #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from django.core.urlresolvers import reverse
>> from django.utils import timezone
>> diff --git
>> a/bitbake/lib/toaster/tests/browser/test_builddashboard_page_recipes.py
>> b/bitbake/lib/toaster/tests/browser/test_builddashboard_page_recipes.py
>> index ed18324..c542d45 100644 ---
>> a/bitbake/lib/toaster/tests/browser/test_builddashboard_page_recipes.py
>> +++
>> b/bitbake/lib/toaster/tests/browser/test_builddashboard_page_recipes.py
>> @@ -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et -#
>> -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3 #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from django.core.urlresolvers import reverse
>> from django.utils import timezone
>> diff --git
>> a/bitbake/lib/toaster/tests/browser/test_builddashboard_page_tasks.py
>> b/bitbake/lib/toaster/tests/browser/test_builddashboard_page_tasks.py
>> index da50f16..22acb47 100644 ---
>> a/bitbake/lib/toaster/tests/browser/test_builddashboard_page_tasks.py
>> +++
>> b/bitbake/lib/toaster/tests/browser/test_builddashboard_page_tasks.py
>> @@ -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et -#
>> -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3 #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from django.core.urlresolvers import reverse
>> from django.utils import timezone
>> diff --git a/bitbake/lib/toaster/tests/browser/test_js_unit_tests.py
>> b/bitbake/lib/toaster/tests/browser/test_js_unit_tests.py index
>> 3c0b962..e8b4295 100644 ---
>> a/bitbake/lib/toaster/tests/browser/test_js_unit_tests.py +++
>> b/bitbake/lib/toaster/tests/browser/test_js_unit_tests.py @@ -1,23
>> +1,11 @@ -#! /usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> """
>> Run the js unit tests
>> diff --git a/bitbake/lib/toaster/tests/browser/test_landing_page.py
>> b/bitbake/lib/toaster/tests/browser/test_landing_page.py index
>> 4d4cd66..0790198 100644 ---
>> a/bitbake/lib/toaster/tests/browser/test_landing_page.py +++
>> b/bitbake/lib/toaster/tests/browser/test_landing_page.py @@ -1,23
>> +1,11 @@ -#! /usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> -# Copyright (C) 2013-2016 Intel Corporation
>> -#
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from django.core.urlresolvers import reverse
>> from django.utils import timezone
>> diff --git
>> a/bitbake/lib/toaster/tests/browser/test_layerdetails_page.py
>> b/bitbake/lib/toaster/tests/browser/test_layerdetails_page.py index
>> f24fb09..f81e696 100644 ---
>> a/bitbake/lib/toaster/tests/browser/test_layerdetails_page.py +++
>> b/bitbake/lib/toaster/tests/browser/test_layerdetails_page.py @@
>> -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> -# Copyright (C) 2013-2016 Intel Corporation
>> -#
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from django.core.urlresolvers import reverse
>> from tests.browser.selenium_helpers import SeleniumTestCase
>> diff --git
>> a/bitbake/lib/toaster/tests/browser/test_most_recent_builds_states.py
>> b/bitbake/lib/toaster/tests/browser/test_most_recent_builds_states.py
>> index abc0b0b..15d25dc 100644 ---
>> a/bitbake/lib/toaster/tests/browser/test_most_recent_builds_states.py
>> +++
>> b/bitbake/lib/toaster/tests/browser/test_most_recent_builds_states.py
>> @@ -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et -#
>> -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3 #
>> # BitBake Toaster Implementation
>> #
>> -# Copyright (C) 2013-2016 Intel Corporation
>> -#
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from django.core.urlresolvers import reverse
>> from django.utils import timezone
>> diff --git
>> a/bitbake/lib/toaster/tests/browser/test_new_custom_image_page.py
>> b/bitbake/lib/toaster/tests/browser/test_new_custom_image_page.py
>> index ab5a8e6..0aa3b7a 100644 ---
>> a/bitbake/lib/toaster/tests/browser/test_new_custom_image_page.py +++
>> b/bitbake/lib/toaster/tests/browser/test_new_custom_image_page.py @@
>> -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from django.core.urlresolvers import reverse
>> from tests.browser.selenium_helpers import SeleniumTestCase
>> diff --git
>> a/bitbake/lib/toaster/tests/browser/test_new_project_page.py
>> b/bitbake/lib/toaster/tests/browser/test_new_project_page.py index
>> 77e5f15..8e56bb0 100644 ---
>> a/bitbake/lib/toaster/tests/browser/test_new_project_page.py +++
>> b/bitbake/lib/toaster/tests/browser/test_new_project_page.py @@ -1,23
>> +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from django.core.urlresolvers import reverse
>> from tests.browser.selenium_helpers import SeleniumTestCase
>> diff --git
>> a/bitbake/lib/toaster/tests/browser/test_project_builds_page.py
>> b/bitbake/lib/toaster/tests/browser/test_project_builds_page.py index
>> 9fe91ab..47fb10b 100644 ---
>> a/bitbake/lib/toaster/tests/browser/test_project_builds_page.py +++
>> b/bitbake/lib/toaster/tests/browser/test_project_builds_page.py @@
>> -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import re
>>
>> diff --git
>> a/bitbake/lib/toaster/tests/browser/test_project_config_page.py
>> b/bitbake/lib/toaster/tests/browser/test_project_config_page.py index
>> 0710084..2816eb9 100644 ---
>> a/bitbake/lib/toaster/tests/browser/test_project_config_page.py +++
>> b/bitbake/lib/toaster/tests/browser/test_project_config_page.py @@
>> -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import re
>>
>> diff --git a/bitbake/lib/toaster/tests/browser/test_project_page.py
>> b/bitbake/lib/toaster/tests/browser/test_project_page.py index
>> 0186463..8b5e1b6 100644 ---
>> a/bitbake/lib/toaster/tests/browser/test_project_page.py +++
>> b/bitbake/lib/toaster/tests/browser/test_project_page.py @@ -1,23
>> +1,11 @@ -#! /usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from django.core.urlresolvers import reverse
>> from django.utils import timezone
>> diff --git a/bitbake/lib/toaster/tests/browser/test_sample.py
>> b/bitbake/lib/toaster/tests/browser/test_sample.py index
>> 20ec53c..f4ad670 100644 ---
>> a/bitbake/lib/toaster/tests/browser/test_sample.py +++
>> b/bitbake/lib/toaster/tests/browser/test_sample.py @@ -1,23 +1,11 @@
>> -#! /usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> """
>> A small example test demonstrating the basics of writing a test with
>> diff --git a/bitbake/lib/toaster/tests/browser/test_task_page.py
>> b/bitbake/lib/toaster/tests/browser/test_task_page.py index
>> 690d116..26f3dca 100644 ---
>> a/bitbake/lib/toaster/tests/browser/test_task_page.py +++
>> b/bitbake/lib/toaster/tests/browser/test_task_page.py @@ -1,23 +1,11
>> @@ -#! /usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from django.core.urlresolvers import reverse
>> from django.utils import timezone
>> diff --git
>> a/bitbake/lib/toaster/tests/browser/test_toastertable_ui.py
>> b/bitbake/lib/toaster/tests/browser/test_toastertable_ui.py index
>> 53ddf30..ef78cbb 100644 ---
>> a/bitbake/lib/toaster/tests/browser/test_toastertable_ui.py +++
>> b/bitbake/lib/toaster/tests/browser/test_toastertable_ui.py @@ -1,23
>> +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013-2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from datetime import datetime
>>
>> diff --git a/bitbake/lib/toaster/tests/builds/buildtest.py
>> b/bitbake/lib/toaster/tests/builds/buildtest.py index
>> 5a56a11..872bbd3 100644 ---
>> a/bitbake/lib/toaster/tests/builds/buildtest.py +++
>> b/bitbake/lib/toaster/tests/builds/buildtest.py @@ -1,23 +1,11 @@
>> -#! /usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import os
>> import sys
>> diff --git a/bitbake/lib/toaster/tests/builds/test_core_image_min.py
>> b/bitbake/lib/toaster/tests/builds/test_core_image_min.py index
>> 586f4a8..44b6cbe 100644 ---
>> a/bitbake/lib/toaster/tests/builds/test_core_image_min.py +++
>> b/bitbake/lib/toaster/tests/builds/test_core_image_min.py @@ -1,24
>> +1,11 @@ -#! /usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. -
>> # Tests were part of openembedded-core oe selftest Authored by:
>> Lucian Musat # Ionut Chisanovici, Paul Eggleton and Cristian Iorga
>>
>> diff --git a/bitbake/lib/toaster/tests/commands/test_loaddata.py
>> b/bitbake/lib/toaster/tests/commands/test_loaddata.py index
>> 951f6ff..9e8d555 100644 ---
>> a/bitbake/lib/toaster/tests/commands/test_loaddata.py +++
>> b/bitbake/lib/toaster/tests/commands/test_loaddata.py @@ -1,23 +1,11
>> @@ -#! /usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from django.test import TestCase
>> from django.core import management
>> diff --git a/bitbake/lib/toaster/tests/commands/test_lsupdates.py
>> b/bitbake/lib/toaster/tests/commands/test_lsupdates.py index
>> 49897a4..3c4fbe0 100644 ---
>> a/bitbake/lib/toaster/tests/commands/test_lsupdates.py +++
>> b/bitbake/lib/toaster/tests/commands/test_lsupdates.py @@ -1,23 +1,11
>> @@ -#! /usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from django.test import TestCase
>> from django.core import management
>> diff --git a/bitbake/lib/toaster/tests/commands/test_runbuilds.py
>> b/bitbake/lib/toaster/tests/commands/test_runbuilds.py index
>> 3e63483..e223b95 100644 ---
>> a/bitbake/lib/toaster/tests/commands/test_runbuilds.py +++
>> b/bitbake/lib/toaster/tests/commands/test_runbuilds.py @@ -1,23 +1,11
>> @@ -#! /usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import os
>>
>> diff --git a/bitbake/lib/toaster/tests/db/test_db.py
>> b/bitbake/lib/toaster/tests/db/test_db.py index a0f5f6e..0410422
>> 100644 --- a/bitbake/lib/toaster/tests/db/test_db.py
>> +++ b/bitbake/lib/toaster/tests/db/test_db.py
>> @@ -2,6 +2,8 @@
>> #
>> # Copyright (c) 2016 Damien Lespiau
>> #
>> +# SPDX-License-Identifier: MIT
>> +#
>> # Permission is hereby granted, free of charge, to any person
>> obtaining a copy # of this software and associated documentation
>> files (the "Software"), to deal # in the Software without
>> restriction, including without limitation the rights diff --git
>> a/bitbake/lib/toaster/tests/eventreplay/__init__.py
>> b/bitbake/lib/toaster/tests/eventreplay/__init__.py index
>> 6956619..8ed6792 100644 ---
>> a/bitbake/lib/toaster/tests/eventreplay/__init__.py +++
>> b/bitbake/lib/toaster/tests/eventreplay/__init__.py @@ -1,23 +1,11 @@
>> -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> # Tests were part of openembedded-core oe selftest Authored by:
>> Lucian Musat # Ionut Chisanovici, Paul Eggleton and Cristian Iorga
>> diff --git
>> a/bitbake/lib/toaster/tests/functional/functional_helpers.py
>> b/bitbake/lib/toaster/tests/functional/functional_helpers.py index
>> 486078a..455c408 100644 ---
>> a/bitbake/lib/toaster/tests/functional/functional_helpers.py +++
>> b/bitbake/lib/toaster/tests/functional/functional_helpers.py @@ -1,23
>> +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster functional tests implementation
>> #
>> # Copyright (C) 2017 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import os
>> import logging
>> diff --git
>> a/bitbake/lib/toaster/tests/functional/test_functional_basic.py
>> b/bitbake/lib/toaster/tests/functional/test_functional_basic.py index
>> cfa2b0f..56c84fb 100644 ---
>> a/bitbake/lib/toaster/tests/functional/test_functional_basic.py +++
>> b/bitbake/lib/toaster/tests/functional/test_functional_basic.py @@
>> -1,23 +1,11 @@ -#! /usr/bin/env python -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster functional tests implementation
>> #
>> # Copyright (C) 2017 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import time
>> import re
>> diff --git a/bitbake/lib/toaster/tests/views/test_views.py
>> b/bitbake/lib/toaster/tests/views/test_views.py index
>> 1463077..68d9e9d 100644 ---
>> a/bitbake/lib/toaster/tests/views/test_views.py +++
>> b/bitbake/lib/toaster/tests/views/test_views.py @@ -1,23 +1,11 @@
>> -#! /usr/bin/env python
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#! /usr/bin/env python3
>> #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013-2015 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> """Test cases for Toaster GUI and ReST."""
>>
>> diff --git a/bitbake/lib/toaster/toastergui/api.py
>> b/bitbake/lib/toaster/toastergui/api.py index 564d595..8b49b3e 100644
>> --- a/bitbake/lib/toaster/toastergui/api.py
>> +++ b/bitbake/lib/toaster/toastergui/api.py
>> @@ -3,19 +3,8 @@
>> #
>> # Copyright (C) 2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. -
>> # Please run flake8 on this file before sending patches
>>
>> import os
>> diff --git a/bitbake/lib/toaster/toastergui/buildtables.py
>> b/bitbake/lib/toaster/toastergui/buildtables.py index
>> 755a7c2..327059d 100644 ---
>> a/bitbake/lib/toaster/toastergui/buildtables.py +++
>> b/bitbake/lib/toaster/toastergui/buildtables.py @@ -1,23 +1,10 @@
>> #
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> -#
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from orm.models import Build, Task, Target, Package
>> from django.db.models import Q, Sum
>> diff --git a/bitbake/lib/toaster/toastergui/static/js/importlayer.js
>> b/bitbake/lib/toaster/toastergui/static/js/importlayer.js index
>> 2964839..8e2032d 100644 ---
>> a/bitbake/lib/toaster/toastergui/static/js/importlayer.js +++
>> b/bitbake/lib/toaster/toastergui/static/js/importlayer.js @@ -17,11
>> +17,15 @@ function importLayerPageInit (ctx) { var
>> currentLayerDepSelection; var validLayerName = /^(\w|-)+$/;
>>
>> + /* Catch 'disable' race condition between type-ahead started and
>> "input change" */
>> + var typeAheadStarted = 0;
>> +
>> libtoaster.makeTypeahead(layerDepInput,
>> libtoaster.ctx.layersTypeAheadUrl,
>> { include_added: "true" }, function(item){
>> currentLayerDepSelection = item;
>> layerDepBtn.removeAttr("disabled");
>> + typeAheadStarted = 1;
>> });
>>
>> layerDepInput.on("typeahead:select", function(event, data){
>> @@ -34,7 +38,10 @@ function importLayerPageInit (ctx) {
>> // disable the "Add layer" button when the layer input typeahead
>> is empty // or not in the typeahead choices
>> layerDepInput.on("input change", function(){
>> - layerDepBtn.attr("disabled","disabled");
>> + if (0 == typeAheadStarted) {
>> + layerDepBtn.attr("disabled","disabled");
>> + }
>> + typeAheadStarted = 0;
>> });
>>
>> /* We automatically add "openembedded-core" layer for convenience
>> as a @@ -50,6 +57,7 @@ function importLayerPageInit (ctx) {
>> });
>>
>> layerDepBtn.click(function(){
>> + typeAheadStarted = 0;
>> if (currentLayerDepSelection == undefined)
>> return;
>>
>> @@ -77,7 +85,7 @@ function importLayerPageInit (ctx) {
>>
>> $("#layer-deps-list").append(newLayerDep);
>>
>> -
>> libtoaster.getLayerDepsForProject(currentLayerDepSelection.layerdetailurl,
>> +
>> libtoaster.getLayerDepsForProject(currentLayerDepSelection.xhrLayerUrl,
>> function (data){ /* These are the dependencies of the layer added as
>> a dependency */ if (data.list.length > 0) {
>> diff --git a/bitbake/lib/toaster/toastergui/tablefilter.py
>> b/bitbake/lib/toaster/toastergui/tablefilter.py index
>> 65454e1..ffef795 100644 ---
>> a/bitbake/lib/toaster/toastergui/tablefilter.py +++
>> b/bitbake/lib/toaster/toastergui/tablefilter.py @@ -1,23 +1,10 @@
>> #
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> -#
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2015 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from django.db.models import Q, Max, Min
>> from django.utils import dateparse, timezone
>> diff --git a/bitbake/lib/toaster/toastergui/tables.py
>> b/bitbake/lib/toaster/toastergui/tables.py index 9ff756b..b3ea222
>> 100644 --- a/bitbake/lib/toaster/toastergui/tables.py
>> +++ b/bitbake/lib/toaster/toastergui/tables.py
>> @@ -1,23 +1,10 @@
>> #
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> -#
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2015 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from toastergui.widgets import ToasterTable
>> from orm.models import Recipe, ProjectLayer, Layer_Version, Machine,
>> Project diff --git
>> a/bitbake/lib/toaster/toastergui/templatetags/field_values_filter.py
>> b/bitbake/lib/toaster/toastergui/templatetags/field_values_filter.py
>> index 5a73af7..eb48339 100644 ---
>> a/bitbake/lib/toaster/toastergui/templatetags/field_values_filter.py
>> +++
>> b/bitbake/lib/toaster/toastergui/templatetags/field_values_filter.py
>> @@ -1,3 +1,7 @@ +# +# SPDX-License-Identifier: GPL-2.0-only +#
>> +
>> from django import template
>>
>> register = template.Library()
>> diff --git
>> a/bitbake/lib/toaster/toastergui/templatetags/objects_to_dictionaries_filter.py
>> b/bitbake/lib/toaster/toastergui/templatetags/objects_to_dictionaries_filter.py
>> index 0dcc7d2..048d533 100644 ---
>> a/bitbake/lib/toaster/toastergui/templatetags/objects_to_dictionaries_filter.py
>> +++
>> b/bitbake/lib/toaster/toastergui/templatetags/objects_to_dictionaries_filter.py
>> @@ -1,3 +1,7 @@ +# +# SPDX-License-Identifier: GPL-2.0-only +#
>> +
>> from django import template
>> import json
>>
>> diff --git
>> a/bitbake/lib/toaster/toastergui/templatetags/project_url_tag.py
>> b/bitbake/lib/toaster/toastergui/templatetags/project_url_tag.py
>> index 04770ac..71e0925 100644 ---
>> a/bitbake/lib/toaster/toastergui/templatetags/project_url_tag.py +++
>> b/bitbake/lib/toaster/toastergui/templatetags/project_url_tag.py @@
>> -1,3 +1,7 @@ +# +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> from django import template
>> from django.core.urlresolvers import reverse
>>
>> diff --git
>> a/bitbake/lib/toaster/toastergui/templatetags/projecttags.py
>> b/bitbake/lib/toaster/toastergui/templatetags/projecttags.py index
>> b170a16..1dbab3b 100644 ---
>> a/bitbake/lib/toaster/toastergui/templatetags/projecttags.py +++
>> b/bitbake/lib/toaster/toastergui/templatetags/projecttags.py @@ -1,23
>> +1,10 @@ # -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> -#
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from datetime import datetime, timedelta
>> from os.path import relpath
>> diff --git a/bitbake/lib/toaster/toastergui/typeaheads.py
>> b/bitbake/lib/toaster/toastergui/typeaheads.py index 5aa0f8d..fd750ff
>> 100644 --- a/bitbake/lib/toaster/toastergui/typeaheads.py
>> +++ b/bitbake/lib/toaster/toastergui/typeaheads.py
>> @@ -3,18 +3,8 @@
>> #
>> # Copyright (C) 2015 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> import subprocess
>>
>> diff --git a/bitbake/lib/toaster/toastergui/urls.py
>> b/bitbake/lib/toaster/toastergui/urls.py index dc03e30..673d9ae 100644
>> --- a/bitbake/lib/toaster/toastergui/urls.py
>> +++ b/bitbake/lib/toaster/toastergui/urls.py
>> @@ -3,18 +3,8 @@
>> #
>> # Copyright (C) 2013-2017 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from django.conf.urls import include, url
>> from django.views.generic import RedirectView, TemplateView
>> diff --git a/bitbake/lib/toaster/toastergui/views.py
>> b/bitbake/lib/toaster/toastergui/views.py index c712b06..d7acaff
>> 100644 --- a/bitbake/lib/toaster/toastergui/views.py
>> +++ b/bitbake/lib/toaster/toastergui/views.py
>> @@ -1,24 +1,10 @@
>> #
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> -#
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA. -
>>
>> import re
>>
>> diff --git a/bitbake/lib/toaster/toastergui/widgets.py
>> b/bitbake/lib/toaster/toastergui/widgets.py index db5c3aa..645f458
>> 100644 --- a/bitbake/lib/toaster/toastergui/widgets.py
>> +++ b/bitbake/lib/toaster/toastergui/widgets.py
>> @@ -1,23 +1,10 @@
>> #
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> -#
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2015 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from django.views.generic import View, TemplateView
>> from django.views.decorators.cache import cache_control
>> diff --git
>> a/bitbake/lib/toaster/toastermain/management/commands/builddelete.py
>> b/bitbake/lib/toaster/toastermain/management/commands/builddelete.py
>> index bf69a8f..c2d773a 100644 ---
>> a/bitbake/lib/toaster/toastermain/management/commands/builddelete.py
>> +++
>> b/bitbake/lib/toaster/toastermain/management/commands/builddelete.py
>> @@ -1,3 +1,7 @@ +# +# SPDX-License-Identifier: GPL-2.0-only +#
>> +
>> from django.core.management.base import BaseCommand, CommandError
>> from django.core.exceptions import ObjectDoesNotExist
>> from orm.models import Build
>> diff --git
>> a/bitbake/lib/toaster/toastermain/management/commands/buildimport.py
>> b/bitbake/lib/toaster/toastermain/management/commands/buildimport.py
>> index 2d57ab5..408ad44 100644 ---
>> a/bitbake/lib/toaster/toastermain/management/commands/buildimport.py
>> +++
>> b/bitbake/lib/toaster/toastermain/management/commands/buildimport.py
>> @@ -1,23 +1,10 @@ # -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4;
>> c-basic-offset: 4; indent-tabs-mode: nil -*- -#
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2018 Wind River Systems
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> # buildimport: import a project for project specific configuration
>> #
>> diff --git
>> a/bitbake/lib/toaster/toastermain/management/commands/buildslist.py
>> b/bitbake/lib/toaster/toastermain/management/commands/buildslist.py
>> index 70b5812..1ed2022 100644 ---
>> a/bitbake/lib/toaster/toastermain/management/commands/buildslist.py
>> +++
>> b/bitbake/lib/toaster/toastermain/management/commands/buildslist.py
>> @@ -1,3 +1,7 @@ +# +# SPDX-License-Identifier: GPL-2.0-only +#
>> +
>> from django.core.management.base import BaseCommand, CommandError
>> from orm.models import Build
>> import os
>> diff --git
>> a/bitbake/lib/toaster/toastermain/management/commands/checksocket.py
>> b/bitbake/lib/toaster/toastermain/management/commands/checksocket.py
>> index 0399b86..811fd5d 100644 ---
>> a/bitbake/lib/toaster/toastermain/management/commands/checksocket.py
>> +++
>> b/bitbake/lib/toaster/toastermain/management/commands/checksocket.py
>> @@ -1,23 +1,11 @@ -#!/usr/bin/env python -# ex:ts=4:sw=4:sts=4:et -#
>> -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> +#!/usr/bin/env python3 #
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2015 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> -#
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> """Custom management command checksocket."""
>>
>> diff --git
>> a/bitbake/lib/toaster/toastermain/management/commands/perf.py
>> b/bitbake/lib/toaster/toastermain/management/commands/perf.py index
>> 6b450bb..7d629fb 100644 ---
>> a/bitbake/lib/toaster/toastermain/management/commands/perf.py +++
>> b/bitbake/lib/toaster/toastermain/management/commands/perf.py @@ -1,3
>> +1,7 @@ +# +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> from django.core.management.base import BaseCommand
>> from django.test.client import Client
>> import os, sys, re
>> diff --git a/bitbake/lib/toaster/toastermain/settings.py
>> b/bitbake/lib/toaster/toastermain/settings.py index 13541d3..74501fa
>> 100644 --- a/bitbake/lib/toaster/toastermain/settings.py
>> +++ b/bitbake/lib/toaster/toastermain/settings.py
>> @@ -1,23 +1,10 @@
>> #
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> -#
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> # Django settings for Toaster project.
>>
>> diff --git
>> a/bitbake/lib/toaster/toastermain/settings_production_example.py
>> b/bitbake/lib/toaster/toastermain/settings_production_example.py
>> index 61a2888..6cd0f52 100644 ---
>> a/bitbake/lib/toaster/toastermain/settings_production_example.py +++
>> b/bitbake/lib/toaster/toastermain/settings_production_example.py @@
>> -1,23 +1,10 @@ # -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> -#
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> # See Django documentation for more information about deployment
>> # https://docs.djangoproject.com/en/1.8/howto/deployment/checklist/
>> diff --git a/bitbake/lib/toaster/toastermain/settings_test.py
>> b/bitbake/lib/toaster/toastermain/settings_test.py index
>> a322711..6538d9e 100644 ---
>> a/bitbake/lib/toaster/toastermain/settings_test.py +++
>> b/bitbake/lib/toaster/toastermain/settings_test.py @@ -1,23 +1,10 @@
>> #
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> -#
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2016 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> # Django settings for Toaster project.
>>
>> diff --git a/bitbake/lib/toaster/toastermain/urls.py
>> b/bitbake/lib/toaster/toastermain/urls.py index e2fb0ae..ac77bc3
>> 100644 --- a/bitbake/lib/toaster/toastermain/urls.py
>> +++ b/bitbake/lib/toaster/toastermain/urls.py
>> @@ -1,23 +1,10 @@
>> #
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> -#
>> # BitBake Toaster Implementation
>> #
>> # Copyright (C) 2013 Intel Corporation
>> #
>> -# This program is free software; you can redistribute it and/or
>> modify -# it under the terms of the GNU General Public License
>> version 2 as -# published by the Free Software Foundation.
>> -#
>> -# This program is distributed in the hope that it will be useful,
>> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> -# GNU General Public License for more details.
>> +# SPDX-License-Identifier: GPL-2.0-only
>> #
>> -# You should have received a copy of the GNU General Public License
>> along -# with this program; if not, write to the Free Software
>> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA
>> 02110-1301 USA.
>> from django.conf.urls import include, url
>> from django.views.generic import RedirectView, TemplateView
>> diff --git a/bitbake/lib/toaster/toastermain/wsgi.py
>> b/bitbake/lib/toaster/toastermain/wsgi.py index 031b314..4c31283
>> 100644 --- a/bitbake/lib/toaster/toastermain/wsgi.py
>> +++ b/bitbake/lib/toaster/toastermain/wsgi.py
>> @@ -1,7 +1,8 @@
>> -"""
>> -# ex:ts=4:sw=4:sts=4:et
>> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
>> #
>> +# SPDX-License-Identifier: GPL-2.0-only
>> +#
>> +
>> +"""
>> WSGI config for Toaster project.
>>
>> This module contains the WSGI application used by Django's
>> development server diff --git a/doc/user_manual.md
>> b/doc/user_manual.md index c2657da..96aa2e3 100644
>> --- a/doc/user_manual.md
>> +++ b/doc/user_manual.md
>> @@ -135,8 +135,8 @@ DISTRO_ARCH ??= "armhf"
>> Then, call `bitbake` with image names, e.g.:
>>
>> ```
>> -bitbake multiconfig:qemuarm-buster:isar-image-base \
>> - multiconfig:qemuarm-buster:isar-image-debug
>> +bitbake mc:qemuarm-buster:isar-image-base \
>> + mc:qemuarm-buster:isar-image-debug
>> ```
>>
>> The following images are created:
>> @@ -169,14 +169,14 @@ The following command will produce
>> `isar-image-base` images for all targets:
>> ```
>> $ bitbake \
>> - multiconfig:qemuarm-stretch:isar-image-base \
>> - multiconfig:qemuarm-buster:isar-image-base \
>> - multiconfig:qemuarm64-stretch:isar-image-base \
>> - multiconfig:qemui386-stretch:isar-image-base \
>> - multiconfig:qemui386-buster:isar-image-base \
>> - multiconfig:qemuamd64-stretch:isar-image-base \
>> - multiconfig:qemuamd64-buster:isar-image-base \
>> - multiconfig:rpi-stretch:isar-image-base
>> + mc:qemuarm-stretch:isar-image-base \
>> + mc:qemuarm-buster:isar-image-base \
>> + mc:qemuarm64-stretch:isar-image-base \
>> + mc:qemui386-stretch:isar-image-base \
>> + mc:qemui386-buster:isar-image-base \
>> + mc:qemuamd64-stretch:isar-image-base \
>> + mc:qemuamd64-buster:isar-image-base \
>> + mc:rpi-stretch:isar-image-base
>> ```
>>
>> Created images are:
>> @@ -197,9 +197,9 @@ tmp/deploy/images/rpi/isar-image-base.rpi-sdimg
>> A bootable disk image is generated if you set IMAGE_TYPE to
>> 'wic-img'. Behind the scenes a tool called `wic` is used to assemble
>> the images. It is controlled by a `.wks` file which you can choose
>> with changing WKS_FILE. Some examples in the tree use that feature
>> already. ``` # Generate an image for the `i386` target architecture
>> - $ bitbake multiconfig:qemui386-buster:isar-image-base
>> + $ bitbake mc:qemui386-buster:isar-image-base
>> # Similarly, for the `amd64` target architecture, in this case EFI
>> - $ bitbake multiconfig:qemuamd64-buster:isar-image-base
>> + $ bitbake mc:qemuamd64-buster:isar-image-base
>> ```
>>
>> Variables may be used in `.wks.in` files; Isar will expand them and
>> generate a regular `.wks` file before generating the disk image using
>> `wic`. @@ -303,7 +303,7 @@ following variables define the default
>> configuration to build for:
>> - `DISTRO_ARCH` - The Debian architecture to build for (e.g.,
>> `armhf`).
>> If BitBake is called with multiconfig targets (e.g.,
>> -`multiconfig:qemuarm-buster:isar-image-base`), the following
>> variable defines +`mc:qemuarm-buster:isar-image-base`), the following
>> variable defines all supported configurations:
>>
>> - `BBMULTICONFIG` - The list of the complete configuration
>> definition files. @@ -687,7 +687,7 @@ Debian cross-compilation works
>> out of the box starting from Debian stretch distr Just like
>> OpenEmbedded, Isar supports a devshell target for all dpkg package
>> recipes. This target opens a terminal inside the buildchroot that
>> runs the package build. To invoke it, just call -`bitbake
>> multiconfig:${MACHINE}-${DISTRO}:<package_name> -c devshell`.
>> +`bitbake mc:${MACHINE}-${DISTRO}:<package_name> -c devshell`.
>>
>> ## Create an ISAR SDK root filesystem
>> @@ -705,7 +705,7 @@ target binary artifacts. Developer chroots to sdk
>> rootfs and develops applicatio ### Solution
>>
>> User manually triggers creation of SDK root filesystem for his
>> target platform by launching the task `do_populate_sdk` for target
>> image, f.e. -`bitbake -c do_populate_sdk
>> multiconfig:${MACHINE}-${DISTRO}:isar-image-base`. +`bitbake -c
>> do_populate_sdk mc:${MACHINE}-${DISTRO}:isar-image-base`. The
>> resulting SDK rootfs is archived into
>> `tmp/deploy/images/${MACHINE}/sdk-${DISTRO}-${DISTRO_ARCH}.tar.xz`.
>> It is additionally available for direct use under
>> `tmp/deploy/images/${MACHINE}/sdk-${DISTRO}-${DISTRO_ARCH}/`. @@
>> -717,7 +717,7 @@ One may chroot into the SDK and install required
>> target packages with the help o
>> - Trigger creation of SDK root filesystem
>>
>> ```
>> -bitbake -c do_populate_sdk multiconfig:qemuarm-buster:isar-image-base
>> +bitbake -c do_populate_sdk mc:qemuarm-buster:isar-image-base
>> ```
>>
>> - Mount the following directories in chroot by passing resulting
>> rootfs as an argument to the script `mount_chroot.sh`: @@ -801,7
>> +801,7 @@ BASE_REPO_KEY =
>> "file://<absolute_path_to_your_pub_key_file>"'
>> - Trigger creation of local apt caching Debian packages during
>> image generation.
>> ```
>> -bitbake -c cache_base_repo multiconfig:qemuarm-buster:isar-image-base
>> +bitbake -c cache_base_repo mc:qemuarm-buster:isar-image-base
>> ```
>>
>> - Set `ISAR_USE_CACHED_BASE_REPO` in `conf/local.conf`:
>> @@ -820,7 +820,7 @@ sudo rm -rf tmp
>> - Trigger again generation of image (now using local caching repo):
>>
>> ```
>> -bitbake multiconfig:qemuarm-buster:isar-image-base
>> +bitbake mc:qemuarm-buster:isar-image-base
>> ```
>>
>> ### Limitation
>> @@ -860,5 +860,5 @@ DISTRO_APT_SOURCES_append = "
>> conf/distro/docker-buster.list" And build the corresponding image
>> target:
>> ```
>> -bitbake multiconfig:qemuarm64-buster:isar-image-base
>> +bitbake mc:qemuarm64-buster:isar-image-base
>> ```
>> diff --git a/meta-isar/conf/conf-notes.txt
>> b/meta-isar/conf/conf-notes.txt index 74df97c..c557cd3 100644
>> --- a/meta-isar/conf/conf-notes.txt
>> +++ b/meta-isar/conf/conf-notes.txt
>> @@ -1,4 +1,4 @@
>> Common targets are:
>> - multiconfig:qemuarm-buster:isar-image-base
>> - multiconfig:qemuamd64-buster:isar-image-base
>> - multiconfig:rpi-stretch:isar-image-base
>> + mc:qemuarm-buster:isar-image-base
>> + mc:qemuamd64-buster:isar-image-base
>> + mc:rpi-stretch:isar-image-base
>> diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
>> index 713d1c2..a8556a6 100755
>> --- a/scripts/ci_build.sh
>> +++ b/scripts/ci_build.sh
>> @@ -21,40 +21,40 @@ BUILD_DIR=./build
>> BB_ARGS="-v"
>>
>> TARGETS_SET="\
>> - multiconfig:qemuarm-stretch:isar-image-base \
>> - multiconfig:qemuarm-buster:isar-image-base \
>> - multiconfig:qemuarm-bullseye:isar-image-base \
>> - multiconfig:qemuarm64-stretch:isar-image-base \
>> - multiconfig:qemui386-stretch:isar-image-base \
>> - multiconfig:qemui386-buster:isar-image-base \
>> - multiconfig:qemui386-bullseye:isar-image-base \
>> - multiconfig:qemuamd64-stretch:isar-image-base \
>> - multiconfig:qemuamd64-buster:isar-image-base \
>> - multiconfig:qemuamd64-buster-tgz:isar-image-base \
>> - multiconfig:qemuamd64-bullseye:isar-image-base \
>> - multiconfig:qemumipsel-stretch:isar-image-base \
>> - multiconfig:qemumipsel-buster:isar-image-base \
>> - multiconfig:qemumipsel-bullseye:isar-image-base \
>> - multiconfig:nand-ubi-demo-buster:isar-image-ubi \
>> - multiconfig:rpi-stretch:isar-image-base"
>> + mc:qemuarm-stretch:isar-image-base \
>> + mc:qemuarm-buster:isar-image-base \
>> + mc:qemuarm-bullseye:isar-image-base \
>> + mc:qemuarm64-stretch:isar-image-base \
>> + mc:qemui386-stretch:isar-image-base \
>> + mc:qemui386-buster:isar-image-base \
>> + mc:qemui386-bullseye:isar-image-base \
>> + mc:qemuamd64-stretch:isar-image-base \
>> + mc:qemuamd64-buster:isar-image-base \
>> + mc:qemuamd64-buster-tgz:isar-image-base \
>> + mc:qemuamd64-bullseye:isar-image-base \
>> + mc:qemumipsel-stretch:isar-image-base \
>> + mc:qemumipsel-buster:isar-image-base \
>> + mc:qemumipsel-bullseye:isar-image-base \
>> + mc:nand-ubi-demo-buster:isar-image-ubi \
>> + mc:rpi-stretch:isar-image-base"
>> # qemu-user-static of <= buster too old to build that
>> - # multiconfig:qemuarm64-buster:isar-image-base
>> - # multiconfig:qemuarm64-bullseye:isar-image-base
>> + # mc:qemuarm64-buster:isar-image-base
>> + # mc:qemuarm64-bullseye:isar-image-base
>>
>> CROSS_TARGETS_SET="\
>> - multiconfig:qemuarm-stretch:isar-image-base \
>> - multiconfig:qemuarm-buster:isar-image-base \
>> - multiconfig:qemuarm-bullseye:isar-image-base \
>> - multiconfig:qemuarm64-stretch:isar-image-base \
>> - multiconfig:qemuamd64-stretch:isar-image-base \
>> - multiconfig:de0-nano-soc-stretch:isar-image-base \
>> - multiconfig:rpi-stretch:isar-image-base"
>> + mc:qemuarm-stretch:isar-image-base \
>> + mc:qemuarm-buster:isar-image-base \
>> + mc:qemuarm-bullseye:isar-image-base \
>> + mc:qemuarm64-stretch:isar-image-base \
>> + mc:qemuamd64-stretch:isar-image-base \
>> + mc:de0-nano-soc-stretch:isar-image-base \
>> + mc:rpi-stretch:isar-image-base"
>>
>> REPRO_TARGETS_SET="\
>> - multiconfig:qemuarm-stretch:isar-image-base \
>> - multiconfig:qemuarm64-stretch:isar-image-base \
>> - multiconfig:qemuamd64-stretch:isar-image-base \
>> - multiconfig:qemuarm-buster:isar-image-base"
>> + mc:qemuarm-stretch:isar-image-base \
>> + mc:qemuarm64-stretch:isar-image-base \
>> + mc:qemuamd64-stretch:isar-image-base \
>> + mc:qemuarm-buster:isar-image-base"
>>
>>
>> show_help() {
>> @@ -162,7 +162,7 @@ sed -i -e 's/ISAR_CROSS_COMPILE ?=
>> "0"/ISAR_CROSS_COMPILE ?= "1"/g' conf/local.c bitbake $BB_ARGS
>> $CROSS_TARGETS_SET while [ -e bitbake.sock ]; do sleep 1; done
>> # In addition test SDK creation
>> -bitbake $BB_ARGS -c do_populate_sdk
>> multiconfig:qemuarm-stretch:isar-image-base +bitbake $BB_ARGS -c
>> do_populate_sdk mc:qemuarm-stretch:isar-image-base while [ -e
>> bitbake.sock ]; do sleep 1; done
>> if [ -z "$FAST_BUILD" ]; then
>> @@ -175,6 +175,6 @@ fi
>> cp -a "${ISARROOT}/meta/classes/dpkg-base.bbclass"
>> "${ISARROOT}/meta/classes/dpkg-base.bbclass.ci-backup" echo -e
>> "do_fetch_append() {\n\n}" >>
>> "${ISARROOT}/meta/classes/dpkg-base.bbclass" -bitbake $BB_ARGS
>> multiconfig:qemuamd64-stretch:isar-image-base +bitbake $BB_ARGS
>> mc:qemuamd64-stretch:isar-image-base
>> mv "${ISARROOT}/meta/classes/dpkg-base.bbclass.ci-backup"
>> "${ISARROOT}/meta/classes/dpkg-base.bbclass" diff --git
>> a/scripts/start_vm b/scripts/start_vm index 9f07a03..71b55f2 100755
>> --- a/scripts/start_vm
>> +++ b/scripts/start_vm
>> @@ -101,7 +101,7 @@ do
>> shift
>> done
>>
>> -eval $(bitbake -e multiconfig:qemu$ARCH-$DISTRO:isar-image-base |
>> grep "^DEPLOY_DIR_IMAGE=") +eval $(bitbake -e
>> mc:qemu$ARCH-$DISTRO:isar-image-base | grep "^DEPLOY_DIR_IMAGE=")
>> readonly IMAGE_DIR=$DEPLOY_DIR_IMAGE
>> readonly ISARROOT="$(dirname "$0")"/..
>> @@ -112,13 +112,13 @@ eval "$(egrep 'MACHINE_SERIAL' $MACHINE_CONF
>> |bb2sh)" readonly
>> CONFIG_CONF=$ISARROOT/meta-isar/conf/multiconfig/qemu$ARCH-$DISTRO.conf
>> eval "$(egrep 'QEMU_' $CONFIG_CONF |bb2sh)"
>> -eval $(bitbake -e multiconfig:qemu$ARCH-$DISTRO:isar-image-base |
>> grep "^IMAGE_TYPE=") +eval $(bitbake -e
>> mc:qemu$ARCH-$DISTRO:isar-image-base | grep "^IMAGE_TYPE=") case
>> "$IMAGE_TYPE" in ext4-img)
>> readonly
>> ROOTFS_IMAGE=isar-image-base-debian-$DISTRO-qemu$ARCH.ext4.img
>> - eval $(bitbake -e multiconfig:qemu$ARCH-$DISTRO:isar-image-base
>> | grep "^KERNEL_IMAGE=")
>> - eval $(bitbake -e multiconfig:qemu$ARCH-$DISTRO:isar-image-base
>> | grep "^INITRD_IMAGE=")
>> + eval $(bitbake -e mc:qemu$ARCH-$DISTRO:isar-image-base | grep
>> "^KERNEL_IMAGE=")
>> + eval $(bitbake -e mc:qemu$ARCH-$DISTRO:isar-image-base | grep
>> "^INITRD_IMAGE=") QKERNEL=$IMAGE_DIR/${KERNEL_IMAGE}
>> QINITRD=/dev/null
>> [ -n "$INITRD_IMAGE" ] && QINITRD=$IMAGE_DIR/${INITRD_IMAGE}
>> diff --git a/testsuite/build_test/build_test.py
>> b/testsuite/build_test/build_test.py index 4220d3a..7a55c2f 100644
>> --- a/testsuite/build_test/build_test.py
>> +++ b/testsuite/build_test/build_test.py
>> @@ -22,7 +22,7 @@ class BuildTest(Test):
>>
>> #isar_root = dirname(__file__) + '/..'
>> os.chdir(build_dir)
>> - cmdline = ['bitbake', 'multiconfig:qemu' + arch + '-' +
>> distro + ':isar-image-base']
>> + cmdline = ['bitbake', 'mc:qemu' + arch + '-' + distro +
>> ':isar-image-base'] p1 = subprocess32.run(cmdline)
>>
>> if p1.returncode:
>> diff --git a/testsuite/start_vm.py b/testsuite/start_vm.py
>> index 02a4b51..a3e32ac 100755
>> --- a/testsuite/start_vm.py
>> +++ b/testsuite/start_vm.py
>> @@ -10,7 +10,7 @@ import sys
>> import time
>>
>> def get_bitbake_env(arch, distro):
>> - multiconfig = 'multiconfig:qemu' + arch + '-' + distro +
>> ':isar-image-base'
>> + multiconfig = 'mc:qemu' + arch + '-' + distro +
>> ':isar-image-base' output = subprocess.check_output(['bitbake', '-e',
>> str(multiconfig)]) return output
>>
next prev parent reply other threads:[~2019-10-22 7:42 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-10-21 14:19 [PATCH 0/1] update bitbake " Cedric Hombourger
2019-10-21 14:19 ` [PATCH 1/1] bitbake: update " Cedric Hombourger
2019-10-22 7:36 ` Henning Schild
2019-10-22 7:41 ` Cedric Hombourger [this message]
2019-10-22 15:56 ` Baurzhan Ismagulov
2019-10-24 6:58 ` chombourger
2019-10-24 19:44 ` Baurzhan Ismagulov
2019-11-05 12:36 ` chombourger
2019-11-06 6:01 ` Baurzhan Ismagulov
2019-11-06 6:02 ` Cedric Hombourger
2019-11-08 8:05 ` [PATCH 0/1] update bitbake " Jan Kiszka
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=514fe94a-882a-9b77-b554-f79b255a89d3@mentor.com \
--to=cedric_hombourger@mentor.com \
--cc=henning.schild@siemens.com \
--cc=isar-users@googlegroups.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox