From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6750261011454885888 X-Received: by 2002:ac2:51d9:: with SMTP id u25mr18087784lfm.19.1571729789903; Tue, 22 Oct 2019 00:36:29 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a2e:8915:: with SMTP id d21ls1878337lji.0.gmail; Tue, 22 Oct 2019 00:36:28 -0700 (PDT) X-Google-Smtp-Source: APXvYqzeK5VjPTJ5JrLcYA2mTOrrcZB+WNgKlXIGc/T39fpD8LRzVIJwf4jQXo27+a2HR1snJ0ox X-Received: by 2002:a2e:86cd:: with SMTP id n13mr11173828ljj.252.1571729788244; Tue, 22 Oct 2019 00:36:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1571729788; cv=none; d=google.com; s=arc-20160816; b=m4kYghCVvDx1OvDpozZKnVKitcmyq3yb3c0lVf0xVpNZgXR5xlW/7NqObhjPccT4Iu x55zZVb3SVo9HuBPiOnznSUcPNHjpzMoC/Ee6kQXPgYfingS5yXygJ+A7h+MCfBAeSle qfErnLqi5PExav9LnzV2tTdKWQ24dZfgnYISNL5htl3W3DPFFOnSXQn5QFnP26PSJD3v 2tWId2a804Uc7e5kEu6w1d5GVJXEEtI1aKgeBpWhGc18WKenA9tWsSozgs6/1xtnMY7C R9mIBDWIbvr7mCHOjM9gXVi5fMM6IsanhCkUvZeg7t5emR0vES3G67uvK4Rc4x7/24qK R7og== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date; bh=Io7of93JaBda5vHY8EVrGkDgdGRztpoqWDipZ8YXfXE=; b=OCtbom7MYfU+6tcXj7r7kIfytYYTUSrYqKrHxzRuU1dSP8utP0ogV40YvBm5+qPU8a tOAbwKd9br9s9RWZwztjtlcyUGMttAxPay+7nXz8ODIbSvA+SdGUnXCvXYWGlo1dowPy Hjih5OAQ5AaxIOTfbW8MbYvoKvjK/CbUUMTiJMAFY+GJUqwL4h+wreaoxQgeMTWv3H7W d4+HaP7nxCQcuMyVSNRCdLRizpHvPrZaXiTLlfl5qAbki+dJCvqpaboFt0M38ZdvVpza yyAMrLexBXxpBgq6WFYEzLWxe/Rk7bVOE4c8ArR73T/ISLQZomMF9Xa0m0NaS4E5h+mU +y1Q== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 194.138.37.39 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from lizzard.sbs.de (lizzard.sbs.de. [194.138.37.39]) by gmr-mx.google.com with ESMTPS id h9si676108lfp.5.2019.10.22.00.36.27 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 22 Oct 2019 00:36:28 -0700 (PDT) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 194.138.37.39 as permitted sender) client-ip=194.138.37.39; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 194.138.37.39 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by lizzard.sbs.de (8.15.2/8.15.2) with ESMTPS id x9M7aQMF026168 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 22 Oct 2019 09:36:26 +0200 Received: from md1za8fc.ad001.siemens.net ([139.25.69.254]) by mail1.sbs.de (8.15.2/8.15.2) with ESMTP id x9M7aP4d025403; Tue, 22 Oct 2019 09:36:25 +0200 Date: Tue, 22 Oct 2019 09:36:09 +0200 From: Henning Schild To: Cedric Hombourger Cc: Subject: Re: [PATCH 1/1] bitbake: update to version 1.44.0 Message-ID: <20191022093609.5ed343a5@md1za8fc.ad001.siemens.net> In-Reply-To: <1571667598-657-2-git-send-email-Cedric_Hombourger@mentor.com> References: <1571667598-657-1-git-send-email-Cedric_Hombourger@mentor.com> <1571667598-657-2-git-send-email-Cedric_Hombourger@mentor.com> X-Mailer: Claws Mail 3.17.3 (GTK+ 2.24.32; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-TUID: uWeb+2DePSHB Hi Cedric, i would suggest a split between the actual bump and the following Isar-changes. But i do not care too much and both ways have their pros and cons. Henning Am Mon, 21 Oct 2019 16:19:58 +0200 schrieb Cedric Hombourger : > Found that the gitsm fetcher we ship is rather out-dated compared to > upstream and was failing to pull git sub-modules from repositories > such as azure-umqtt-c. >=20 > It shall be noted that bitbake has renamed its multiconfig targets: > the "multiconfig:" prefix was changed to "mc:". Updated our scripts > as well as documentation. >=20 > Origin: https://github.com/openembedded/bitbake.git > Commit 5d83d828cacb58ccb7c464e799c85fd2d2a50ccc (tag: 1.44.0) > Author: Richard Purdie > Date: Wed Oct 9 14:10:21 2019 +0100 >=20 > bitbake: Update to version 1.44.0 >=20 > Signed-off-by: Richard Purdie >=20 > Signed-off-by: Cedric Hombourger > --- > RECIPE-API-CHANGELOG.md | 16 + > bitbake/.gitattributes | 2 + > bitbake/HEADER | 19 - > bitbake/LICENSE | 12 +- > bitbake/{COPYING =3D> LICENSE.GPL-2.0-only} | 61 +- > bitbake/LICENSE.MIT | 25 + > bitbake/MANIFEST.in | 5 +- > bitbake/bin/bitbake | 16 +- > bitbake/bin/bitbake-diffsigs | 155 +- > bitbake/bin/bitbake-dumpsig | 95 +- > bitbake/bin/bitbake-hashclient | 170 ++ > bitbake/bin/bitbake-hashserv | 62 + > bitbake/bin/bitbake-layers | 12 +- > bitbake/bin/bitbake-prserv | 4 + > bitbake/bin/bitbake-selftest | 16 +- > bitbake/bin/bitbake-worker | 19 +- > bitbake/bin/bitdoc | 14 +- > bitbake/bin/git-make-shallow | 4 + > bitbake/bin/toaster | 12 +- > bitbake/bin/toaster-eventreplay | 17 +- > bitbake/classes/base.bbclass | 2 +- > bitbake/contrib/dump_cache.py | 2 - > .../bitbake-user-manual-execution.xml | 78 +- > .../bitbake-user-manual-fetching.xml | 69 +- > .../bitbake-user-manual-hello.xml | 28 +- > .../bitbake-user-manual-intro.xml | 18 +- > .../bitbake-user-manual-metadata.xml | 179 +- > .../bitbake-user-manual-ref-variables.xml | 422 ++--- > bitbake/doc/poky.ent | 8 - > bitbake/lib/bb/COW.py | 15 - > bitbake/lib/bb/__init__.py | 16 +- > bitbake/lib/bb/build.py | 151 +- > bitbake/lib/bb/cache.py | 109 +- > bitbake/lib/bb/cache_extra.py | 14 +- > bitbake/lib/bb/checksum.py | 12 +- > bitbake/lib/bb/codeparser.py | 13 +- > bitbake/lib/bb/command.py | 12 +- > bitbake/lib/bb/compat.py | 4 + > bitbake/lib/bb/cooker.py | 218 +-- > bitbake/lib/bb/cookerdata.py | 44 +- > bitbake/lib/bb/daemonize.py | 4 + > bitbake/lib/bb/data.py | 23 +- > bitbake/lib/bb/data_smart.py | 25 +- > bitbake/lib/bb/event.py | 33 +- > bitbake/lib/bb/exceptions.py | 3 + > bitbake/lib/bb/fetch2/__init__.py | 55 +- > bitbake/lib/bb/fetch2/bzr.py | 12 +- > bitbake/lib/bb/fetch2/clearcase.py | 19 +- > bitbake/lib/bb/fetch2/cvs.py | 17 +- > bitbake/lib/bb/fetch2/git.py | 68 +- > bitbake/lib/bb/fetch2/gitannex.py | 14 +- > bitbake/lib/bb/fetch2/gitsm.py | 281 ++- > bitbake/lib/bb/fetch2/hg.py | 18 +- > bitbake/lib/bb/fetch2/local.py | 16 +- > bitbake/lib/bb/fetch2/npm.py | 69 +- > bitbake/lib/bb/fetch2/osc.py | 5 +- > bitbake/lib/bb/fetch2/perforce.py | 15 +- > bitbake/lib/bb/fetch2/repo.py | 16 +- > bitbake/lib/bb/fetch2/s3.py | 15 +- > bitbake/lib/bb/fetch2/sftp.py | 15 +- > bitbake/lib/bb/fetch2/ssh.py | 14 +- > bitbake/lib/bb/fetch2/svn.py | 98 +- > bitbake/lib/bb/fetch2/wget.py | 104 +- > bitbake/lib/bb/main.py | 41 +- > bitbake/lib/bb/methodpool.py | 15 +- > bitbake/lib/bb/monitordisk.py | 27 +- > bitbake/lib/bb/msg.py | 14 +- > bitbake/lib/bb/namedtuple_with_abc.py | 4 +- > bitbake/lib/bb/parse/__init__.py | 14 +- > bitbake/lib/bb/parse/ast.py | 15 +- > bitbake/lib/bb/parse/parse_py/BBHandler.py | 59 +- > bitbake/lib/bb/parse/parse_py/ConfHandler.py | 17 +- > bitbake/lib/bb/parse/parse_py/__init__.py | 17 +- > bitbake/lib/bb/persist_data.py | 234 ++- > bitbake/lib/bb/process.py | 4 + > bitbake/lib/bb/progress.py | 28 +- > bitbake/lib/bb/providers.py | 18 +- > bitbake/lib/bb/pysh/builtin.py | 710 -------- > bitbake/lib/bb/pysh/interp.py | 1367 --------------- > bitbake/lib/bb/pysh/lsprof.py | 116 -- > bitbake/lib/bb/pysh/pysh.py | 167 -- > bitbake/lib/bb/pysh/pyshlex.py | 5 - > bitbake/lib/bb/pysh/pyshyacc.py | 17 +- > bitbake/lib/bb/pysh/sherrors.py | 26 - > bitbake/lib/bb/pysh/subprocess_fix.py | 77 - > bitbake/lib/bb/remotedata.py | 12 +- > bitbake/lib/bb/runqueue.py | 1523 > ++++++++++------- bitbake/lib/bb/server/__init__.py | > 14 +- bitbake/lib/bb/server/process.py | 123 +- > bitbake/lib/bb/server/xmlrpcclient.py | 12 +- > bitbake/lib/bb/server/xmlrpcserver.py | 12 +- > bitbake/lib/bb/siggen.py | 350 +++- > bitbake/lib/bb/taskdata.py | 17 +- > bitbake/lib/bb/tests/codeparser.py | 22 +- > bitbake/lib/bb/tests/cooker.py | 15 +- > bitbake/lib/bb/tests/cow.py | 17 +- > bitbake/lib/bb/tests/data.py | 39 +- > bitbake/lib/bb/tests/event.py | 23 +- > bitbake/lib/bb/tests/fetch.py | 308 +++- > bitbake/lib/bb/tests/parse.py | 33 +- > bitbake/lib/bb/tests/persist_data.py | 129 ++ > .../tests/runqueue-tests/classes/base.bbclass | 262 +++ > .../runqueue-tests/classes/image.bbclass | 5 + > .../runqueue-tests/classes/native.bbclass | 2 + > .../bb/tests/runqueue-tests/conf/bitbake.conf | 16 + > .../runqueue-tests/conf/multiconfig/mc1.conf | 1 + > .../runqueue-tests/conf/multiconfig/mc2.conf | 1 + > .../lib/bb/tests/runqueue-tests/recipes/a1.bb | 0 > .../lib/bb/tests/runqueue-tests/recipes/b1.bb | 1 + > .../lib/bb/tests/runqueue-tests/recipes/c1.bb | 0 > .../lib/bb/tests/runqueue-tests/recipes/d1.bb | 3 + > .../lib/bb/tests/runqueue-tests/recipes/e1.bb | 1 + > bitbake/lib/bb/tests/runqueue.py | 323 ++++ > bitbake/lib/bb/tests/utils.py | 23 +- > bitbake/lib/bb/tinfoil.py | 12 +- > bitbake/lib/bb/ui/__init__.py | 12 +- > bitbake/lib/bb/ui/buildinfohelper.py | 22 +- > bitbake/lib/bb/ui/knotty.py | 60 +- > bitbake/lib/bb/ui/ncurses.py | 12 +- > bitbake/lib/bb/ui/taskexp.py | 12 +- > bitbake/lib/bb/ui/toasterui.py | 12 +- > bitbake/lib/bb/ui/uievent.py | 15 +- > bitbake/lib/bb/ui/uihelper.py | 16 +- > bitbake/lib/bb/utils.py | 49 +- > bitbake/lib/bblayers/__init__.py | 4 + > bitbake/lib/bblayers/action.py | 4 + > bitbake/lib/bblayers/common.py | 4 + > bitbake/lib/bblayers/layerindex.py | 6 +- > bitbake/lib/bblayers/query.py | 43 +- > bitbake/lib/bs4/dammit.py | 12 +- > bitbake/lib/bs4/element.py | 22 +- > bitbake/lib/hashserv/__init__.py | 93 + > bitbake/lib/hashserv/client.py | 157 ++ > bitbake/lib/hashserv/server.py | 414 +++++ > bitbake/lib/hashserv/tests.py | 142 ++ > bitbake/lib/layerindexlib/__init__.py | 33 +- > bitbake/lib/layerindexlib/cooker.py | 12 +- > bitbake/lib/layerindexlib/plugin.py | 13 +- > bitbake/lib/layerindexlib/restapi.py | 12 +- > bitbake/lib/layerindexlib/tests/common.py | 12 +- > bitbake/lib/layerindexlib/tests/cooker.py | 12 +- > .../lib/layerindexlib/tests/layerindexobj.py | 12 +- > bitbake/lib/layerindexlib/tests/restapi.py | 12 +- > bitbake/lib/progressbar/__init__.py | 2 + > bitbake/lib/progressbar/compat.py | 2 + > bitbake/lib/progressbar/progressbar.py | 2 + > bitbake/lib/progressbar/widgets.py | 2 + > bitbake/lib/prserv/__init__.py | 4 + > bitbake/lib/prserv/db.py | 6 +- > bitbake/lib/prserv/serv.py | 4 + > bitbake/lib/pyinotify.py | 20 +- > bitbake/lib/toaster/bldcollector/admin.py | 4 + > bitbake/lib/toaster/bldcollector/urls.py | 13 +- > bitbake/lib/toaster/bldcollector/views.py | 12 +- > bitbake/lib/toaster/bldcontrol/admin.py | 4 + > .../lib/toaster/bldcontrol/bbcontroller.py | 16 +- > .../bldcontrol/localhostbecontroller.py | 16 +- > .../management/commands/checksettings.py | 4 + > .../management/commands/runbuilds.py | 4 + > bitbake/lib/toaster/bldcontrol/models.py | 4 + > bitbake/lib/toaster/bldcontrol/views.py | 4 + > bitbake/lib/toaster/manage.py | 4 + > bitbake/lib/toaster/orm/fixtures/oe-core.xml | 26 +- > bitbake/lib/toaster/orm/fixtures/poky.xml | 38 +- > .../orm/management/commands/lsupdates.py | 15 +- > bitbake/lib/toaster/orm/models.py | 25 +- > .../toaster/tests/browser/selenium_helpers.py | 17 +- > .../tests/browser/selenium_helpers_base.py | 17 +- > .../tests/browser/test_all_builds_page.py | 16 +- > .../tests/browser/test_all_projects_page.py | 16 +- > .../tests/browser/test_builddashboard_page.py | 16 +- > .../test_builddashboard_page_artifacts.py | 16 +- > .../test_builddashboard_page_recipes.py | 16 +- > .../browser/test_builddashboard_page_tasks.py | 16 +- > .../tests/browser/test_js_unit_tests.py | 16 +- > .../tests/browser/test_landing_page.py | 18 +- > .../tests/browser/test_layerdetails_page.py | 18 +- > .../browser/test_most_recent_builds_states.py | 18 +- > .../browser/test_new_custom_image_page.py | 16 +- > .../tests/browser/test_new_project_page.py | 16 +- > .../tests/browser/test_project_builds_page.py | 16 +- > .../tests/browser/test_project_config_page.py | 16 +- > .../tests/browser/test_project_page.py | 16 +- > .../lib/toaster/tests/browser/test_sample.py | 16 +- > .../toaster/tests/browser/test_task_page.py | 16 +- > .../tests/browser/test_toastertable_ui.py | 16 +- > bitbake/lib/toaster/tests/builds/buildtest.py | 16 +- > .../tests/builds/test_core_image_min.py | 17 +- > .../toaster/tests/commands/test_loaddata.py | 16 +- > .../toaster/tests/commands/test_lsupdates.py | 16 +- > .../toaster/tests/commands/test_runbuilds.py | 16 +- > bitbake/lib/toaster/tests/db/test_db.py | 2 + > .../lib/toaster/tests/eventreplay/__init__.py | 16 +- > .../tests/functional/functional_helpers.py | 16 +- > .../tests/functional/test_functional_basic.py | 16 +- > bitbake/lib/toaster/tests/views/test_views.py | 16 +- > bitbake/lib/toaster/toastergui/api.py | 13 +- > bitbake/lib/toaster/toastergui/buildtables.py | 15 +- > .../toastergui/static/js/importlayer.js | 12 +- > bitbake/lib/toaster/toastergui/tablefilter.py | 15 +- > bitbake/lib/toaster/toastergui/tables.py | 15 +- > .../templatetags/field_values_filter.py | 4 + > .../objects_to_dictionaries_filter.py | 4 + > .../templatetags/project_url_tag.py | 4 + > .../toastergui/templatetags/projecttags.py | 15 +- > bitbake/lib/toaster/toastergui/typeaheads.py | 12 +- > bitbake/lib/toaster/toastergui/urls.py | 12 +- > bitbake/lib/toaster/toastergui/views.py | 16 +- > bitbake/lib/toaster/toastergui/widgets.py | 15 +- > .../management/commands/builddelete.py | 4 + > .../management/commands/buildimport.py | 15 +- > .../management/commands/buildslist.py | 4 + > .../management/commands/checksocket.py | 16 +- > .../toastermain/management/commands/perf.py | 4 + > bitbake/lib/toaster/toastermain/settings.py | 15 +- > .../settings_production_example.py | 15 +- > .../lib/toaster/toastermain/settings_test.py | 15 +- > bitbake/lib/toaster/toastermain/urls.py | 15 +- > bitbake/lib/toaster/toastermain/wsgi.py | 7 +- > doc/user_manual.md | 38 +- > meta-isar/conf/conf-notes.txt | 6 +- > scripts/ci_build.sh | 62 +- > scripts/start_vm | 8 +- > testsuite/build_test/build_test.py | 2 +- > testsuite/start_vm.py | 2 +- > 225 files changed, 5375 insertions(+), 6320 deletions(-) > create mode 100644 bitbake/.gitattributes > delete mode 100644 bitbake/HEADER > rename bitbake/{COPYING =3D> LICENSE.GPL-2.0-only} (84%) > create mode 100644 bitbake/LICENSE.MIT > mode change 100755 =3D> 120000 bitbake/bin/bitbake-dumpsig > create mode 100755 bitbake/bin/bitbake-hashclient > create mode 100755 bitbake/bin/bitbake-hashserv > delete mode 100644 bitbake/lib/bb/pysh/builtin.py > delete mode 100644 bitbake/lib/bb/pysh/interp.py > delete mode 100644 bitbake/lib/bb/pysh/lsprof.py > delete mode 100644 bitbake/lib/bb/pysh/pysh.py > delete mode 100644 bitbake/lib/bb/pysh/subprocess_fix.py > create mode 100644 bitbake/lib/bb/tests/persist_data.py > create mode 100644 > bitbake/lib/bb/tests/runqueue-tests/classes/base.bbclass create mode > 100644 bitbake/lib/bb/tests/runqueue-tests/classes/image.bbclass > create mode 100644 > bitbake/lib/bb/tests/runqueue-tests/classes/native.bbclass create > mode 100644 bitbake/lib/bb/tests/runqueue-tests/conf/bitbake.conf > create mode 100644 > bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc1.conf create > mode 100644 > bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc2.conf create > mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/a1.bb create > mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/b1.bb create > mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/c1.bb create > mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/d1.bb create > mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/e1.bb create > mode 100644 bitbake/lib/bb/tests/runqueue.py create mode 100644 > bitbake/lib/hashserv/__init__.py create mode 100644 > bitbake/lib/hashserv/client.py create mode 100644 > bitbake/lib/hashserv/server.py create mode 100644 > bitbake/lib/hashserv/tests.py >=20 > diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md > index bbef1a3..81ec430 100644 > --- a/RECIPE-API-CHANGELOG.md > +++ b/RECIPE-API-CHANGELOG.md > @@ -176,3 +176,19 @@ Otherwise set a encrypted root password like > this: USERS +=3D "root" > USER_root[password] =3D > "$6$rounds=3D10000$RXeWrnFmkY$DtuS/OmsAS2cCEDo0BF5qQsizIrq6jPgXnwv3PHqREJ= eKd1sXdHX/ayQtuQWVDHe0KIO0/sVH8dvQm1KthF0d/" > ``` + > +### multiconfig build targets were renamed > + > +bitbake was upgraded to version 1.44.0 where "multiconfig" build > targets were +renamed "mc". As an example, builds for the > qemuarm-stretch machine should now +be done as follows: > + > +``` > +bitbake mc:qemuarm-stretch:isar-image-base > +``` > + > +The old syntax is no longer supported and will produce an error: > + > +``` > +bitbake multiconfig:qemuarm-stretch:isar-image-base > +``` > diff --git a/bitbake/.gitattributes b/bitbake/.gitattributes > new file mode 100644 > index 0000000..e4f8f62 > --- /dev/null > +++ b/bitbake/.gitattributes > @@ -0,0 +1,2 @@ > +*min.js binary > +*min.css binary > diff --git a/bitbake/HEADER b/bitbake/HEADER > deleted file mode 100644 > index 9859255..0000000 > --- a/bitbake/HEADER > +++ /dev/null > @@ -1,19 +0,0 @@ > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > -# > -# does.> -# Copyright (C) > -# > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > -# > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA. - > diff --git a/bitbake/LICENSE b/bitbake/LICENSE > index 7d4e5f4..8458042 100644 > --- a/bitbake/LICENSE > +++ b/bitbake/LICENSE > @@ -1,4 +1,13 @@ > -BitBake is licensed under the GNU General Public License version > 2.0. See COPYING for further details. +BitBake is licensed under the > GNU General Public License version 2.0. See +LICENSE.GPL-2.0-only for > further details. + > +Individual files contain the following style tags instead of the > full license text: + > + SPDX-License-Identifier: GPL-2.0-only > + > +This enables machine processing of license information based on the > SPDX +License Identifiers that are here available: > http://spdx.org/licenses/ + > =20 > The following external components are distributed with this software: > =20 > @@ -17,3 +26,4 @@ Foundation and individual contributors. > * Font Awesome fonts redistributed under the SIL Open Font License > 1.1=20 > * simplediff is distributed under the zlib license. > + > diff --git a/bitbake/COPYING b/bitbake/LICENSE.GPL-2.0-only > similarity index 84% > rename from bitbake/COPYING > rename to bitbake/LICENSE.GPL-2.0-only > index d511905..5db3c0a 100644 > --- a/bitbake/COPYING > +++ b/bitbake/LICENSE.GPL-2.0-only > @@ -279,61 +279,10 @@ POSSIBILITY OF SUCH DAMAGES. > =20 > END OF TERMS AND CONDITIONS > =20 > - How to Apply These Terms to Your New Programs > +Note: > +Individual files contain the following tag instead of the full > license text.=20 > - If you develop a new program, and you want it to be of the greatest > -possible use to the public, the best way to achieve this is to make > it -free software which everyone can redistribute and change under > these terms. > + SPDX-License-Identifier: GPL-2.0-only > =20 > - To do so, attach the following notices to the program. It is > safest -to attach them to the start of each source file to most > effectively -convey the exclusion of warranty; and each file should > have at least -the "copyright" line and a pointer to where the full > notice is found. - > - does.> > - Copyright (C) > - > - This program is free software; you can redistribute it and/or > modify > - it under the terms of the GNU General Public License as > published by > - the Free Software Foundation; either version 2 of the License, or > - (at your option) any later version. > - > - This program is distributed in the hope that it will be useful, > - but WITHOUT ANY WARRANTY; without even the implied warranty of > - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > - GNU General Public License for more details. > - > - You should have received a copy of the GNU General Public > License along > - with this program; if not, write to the Free Software > Foundation, Inc., > - 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. > - > -Also add information on how to contact you by electronic and paper > mail. - > -If the program is interactive, make it output a short notice like > this -when it starts in an interactive mode: > - > - Gnomovision version 69, Copyright (C) year name of author > - Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type > `show w'. > - This is free software, and you are welcome to redistribute it > - under certain conditions; type `show c' for details. > - > -The hypothetical commands `show w' and `show c' should show the > appropriate -parts of the General Public License. Of course, the > commands you use may -be called something other than `show w' and > `show c'; they could even be -mouse-clicks or menu items--whatever > suits your program. - > -You should also get your employer (if you work as a programmer) or > your -school, if any, to sign a "copyright disclaimer" for the > program, if -necessary. Here is a sample; alter the names: > - > - Yoyodyne, Inc., hereby disclaims all copyright interest in the > program > - `Gnomovision' (which makes passes at compilers) written by James > Hacker. - > - , 1 April 1989 > - Ty Coon, President of Vice > - > -This General Public License does not permit incorporating your > program into -proprietary programs. If your program is a subroutine > library, you may -consider it more useful to permit linking > proprietary applications with the -library. If this is what you want > to do, use the GNU Lesser General -Public License instead of this > License. +This enables machine processing of license information > based on the SPDX +License Identifiers that are here available: > http://spdx.org/licenses/ diff --git a/bitbake/LICENSE.MIT > b/bitbake/LICENSE.MIT new file mode 100644 > index 0000000..a6919eb > --- /dev/null > +++ b/bitbake/LICENSE.MIT > @@ -0,0 +1,25 @@ > +Permission is hereby granted, free of charge, to any person > obtaining a copy +of this software and associated documentation files > (the "Software"), to deal +in the Software without restriction, > including without limitation the rights +to use, copy, modify, merge, > publish, distribute, sublicense, and/or sell +copies of the Software, > and to permit persons to whom the Software is +furnished to do so, > subject to the following conditions: + > +The above copyright notice and this permission notice shall be > included in +all copies or substantial portions of the Software. > + > +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, > EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF > MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND > NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS > BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN > ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN > CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE > +SOFTWARE. + > +Note: > +Individual files contain the following tag instead of the full > license text. + > + SPDX-License-Identifier: MIT > + > +This enables machine processing of license information based on the > SPDX +License Identifiers that are here available: > http://spdx.org/licenses/ diff --git a/bitbake/MANIFEST.in > b/bitbake/MANIFEST.in index b197378..f24969a 100644 > --- a/bitbake/MANIFEST.in > +++ b/bitbake/MANIFEST.in > @@ -1,6 +1,8 @@ > -include COPYING > include ChangeLog > include AUTHORS > +include LICENSE > +include LICENSE.GPL-2.0-only > +include LICENSE.MIT > include contrib/* > include contrib/vim/*/* > include conf/* > @@ -8,4 +10,3 @@ include classes/* > include doc/* > include doc/manual/* > include ez_setup.py > -include HEADER > diff --git a/bitbake/bin/bitbake b/bitbake/bin/bitbake > index 57dec2a..66d08f8 100755 > --- a/bitbake/bin/bitbake > +++ b/bitbake/bin/bitbake > @@ -1,6 +1,4 @@ > #!/usr/bin/env python3 > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > # > # Copyright (C) 2003, 2004 Chris Larson > # Copyright (C) 2003, 2004 Phil Blundell > @@ -9,18 +7,8 @@ > # Copyright (C) 2005 ROAD GmbH > # Copyright (C) 2006 Richard Purdie > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > +# SPDX-License-Identifier: GPL-2.0-only > # > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA.=20 > import os > import sys > @@ -38,7 +26,7 @@ from bb.main import bitbake_main, > BitBakeConfigParameters, BBMainException if > sys.getfilesystemencoding() !=3D "utf-8": sys.exit("Please use a locale > setting which supports UTF-8 (such as LANG=3Den_US.UTF-8).\nPython > can't change the filesystem locale after loading so we need a UTF-8 > when Python starts or things won't work.") -__version__ =3D "1.40.0" > +__version__ =3D "1.44.0"=20 > if __name__ =3D=3D "__main__": > if __version__ !=3D bb.__version__: > diff --git a/bitbake/bin/bitbake-diffsigs > b/bitbake/bin/bitbake-diffsigs index 4e6bbdd..19420a2 100755 > --- a/bitbake/bin/bitbake-diffsigs > +++ b/bitbake/bin/bitbake-diffsigs > @@ -1,27 +1,16 @@ > #!/usr/bin/env python3 > =20 > -# bitbake-diffsigs > -# BitBake task signature data comparison utility > +# bitbake-diffsigs / bitbake-dumpsig > +# BitBake task signature data dump and comparison utility > # > # Copyright (C) 2012-2013, 2017 Intel Corporation > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > +# SPDX-License-Identifier: GPL-2.0-only > # > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA.=20 > import os > import sys > import warnings > -import fnmatch > import argparse > import logging > import pickle > @@ -32,7 +21,10 @@ import bb.tinfoil > import bb.siggen > import bb.msg > =20 > -logger =3D bb.msg.logger_create('bitbake-diffsigs') > +myname =3D os.path.basename(sys.argv[0]) > +logger =3D bb.msg.logger_create(myname) > + > +is_dump =3D myname =3D=3D 'bitbake-dumpsig' > =20 > def find_siginfo(tinfoil, pn, taskname, sigs=3DNone): > result =3D None > @@ -59,8 +51,8 @@ def find_siginfo(tinfoil, pn, taskname, sigs=3DNone): > sys.exit(2) > return result > =20 > -def find_compare_task(bbhandler, pn, taskname, sig1=3DNone, sig2=3DNone, > color=3DFalse): > - """ Find the most recent signature files for the specified > PN/task and compare them """ +def find_siginfo_task(bbhandler, pn, > taskname, sig1=3DNone, sig2=3DNone): > + """ Find the most recent signature files for the specified > PN/task """=20 > if not taskname.startswith('do_'): > taskname =3D 'do_%s' % taskname > @@ -79,73 +71,81 @@ def find_compare_task(bbhandler, pn, taskname, > sig1=3DNone, sig2=3DNone, color=3DFalse latestfiles =3D [sigfiles[sig1], > sigfiles[sig2]] else: > filedates =3D find_siginfo(bbhandler, pn, taskname) > - latestfiles =3D sorted(filedates.keys(), key=3Dlambda f: > filedates[f])[-3:] > + latestfiles =3D sorted(filedates.keys(), key=3Dlambda f: > filedates[f])[-2:] if not latestfiles: > logger.error('No sigdata files found matching %s %s' % > (pn, taskname)) sys.exit(1) > - elif len(latestfiles) < 2: > - logger.error('Only one matching sigdata file found for > the specified task (%s %s)' % (pn, taskname)) > - sys.exit(1) > =20 > - # Define recursion callback > - def recursecb(key, hash1, hash2): > - hashes =3D [hash1, hash2] > - hashfiles =3D find_siginfo(bbhandler, key, None, hashes) > - > - recout =3D [] > - if len(hashfiles) =3D=3D 0: > - recout.append("Unable to find matching sigdata for %s > with hashes %s or %s" % (key, hash1, hash2)) > - elif not hash1 in hashfiles: > - recout.append("Unable to find matching sigdata for %s > with hash %s" % (key, hash1)) > - elif not hash2 in hashfiles: > - recout.append("Unable to find matching sigdata for %s > with hash %s" % (key, hash2)) > - else: > - out2 =3D bb.siggen.compare_sigfiles(hashfiles[hash1], > hashfiles[hash2], recursecb, color=3Dcolor) > - for change in out2: > - for line in change.splitlines(): > - recout.append(' ' + line) > + return latestfiles > + > =20 > - return recout > +# Define recursion callback > +def recursecb(key, hash1, hash2): > + hashes =3D [hash1, hash2] > + hashfiles =3D find_siginfo(tinfoil, key, None, hashes) > =20 > - # Recurse into signature comparison > - logger.debug("Signature file (previous): %s" % latestfiles[-2]) > - logger.debug("Signature file (latest): %s" % latestfiles[-1]) > - output =3D bb.siggen.compare_sigfiles(latestfiles[-2], > latestfiles[-1], recursecb, color=3Dcolor) > - if output: > - print('\n'.join(output)) > - sys.exit(0) > + recout =3D [] > + if len(hashfiles) =3D=3D 0: > + recout.append("Unable to find matching sigdata for %s with > hashes %s or %s" % (key, hash1, hash2)) > + elif not hash1 in hashfiles: > + recout.append("Unable to find matching sigdata for %s with > hash %s" % (key, hash1)) > + elif not hash2 in hashfiles: > + recout.append("Unable to find matching sigdata for %s with > hash %s" % (key, hash2)) > + else: > + out2 =3D bb.siggen.compare_sigfiles(hashfiles[hash1], > hashfiles[hash2], recursecb, color=3Dcolor) > + for change in out2: > + for line in change.splitlines(): > + recout.append(' ' + line) > =20 > + return recout > =20 > =20 > parser =3D argparse.ArgumentParser( > - description=3D"Compares siginfo/sigdata files written out by > BitBake") > + description=3D("Dumps" if is_dump else "Compares") + " > siginfo/sigdata files written out by BitBake")=20 > -parser.add_argument('-d', '--debug', > +parser.add_argument('-D', '--debug', > help=3D'Enable debug output', > action=3D'store_true') > =20 > -parser.add_argument('--color', > - help=3D'Colorize output (where %(metavar)s is %(choices)s)', > - choices=3D['auto', 'always', 'never'], default=3D'auto', > metavar=3D'color') +if is_dump: > + parser.add_argument("-t", "--task", > + help=3D"find the signature data file for the last run of > the specified task", > + action=3D"store", dest=3D"taskargs", nargs=3D2, > metavar=3D('recipename', 'taskname')) + > + parser.add_argument("sigdatafile1", > + help=3D"Signature file to dump. Not used when using > -t/--task.", > + action=3D"store", nargs=3D'?', metavar=3D"sigdatafile") > +else: > + parser.add_argument('-c', '--color', > + help=3D'Colorize the output (where %(metavar)s is > %(choices)s)', > + choices=3D['auto', 'always', 'never'], default=3D'auto', > metavar=3D'color')=20 > -parser.add_argument("-t", "--task", > - help=3D"find the signature data files for last two runs of the > specified task and compare them", > - action=3D"store", dest=3D"taskargs", nargs=3D2, > metavar=3D('recipename', 'taskname')) > + parser.add_argument('-d', '--dump', > + help=3D'Dump the last signature data instead of comparing > (equivalent to using bitbake-dumpsig)', > + action=3D'store_true') > =20 > -parser.add_argument("-s", "--signature", > - help=3D"With -t/--task, specify the signatures to look for > instead of taking the last two", > - action=3D"store", dest=3D"sigargs", nargs=3D2, metavar=3D('froms= ig', > 'tosig')) > + parser.add_argument("-t", "--task", > + help=3D"find the signature data files for the last two > runs of the specified task and compare them", > + action=3D"store", dest=3D"taskargs", nargs=3D2, > metavar=3D('recipename', 'taskname'))=20 > -parser.add_argument("sigdatafile1", > - help=3D"First signature file to compare (or signature file to > dump, if second not specified). Not used when using -t/--task.", > - action=3D"store", nargs=3D'?') > + parser.add_argument("-s", "--signature", > + help=3D"With -t/--task, specify the signatures to look for > instead of taking the last two", > + action=3D"store", dest=3D"sigargs", nargs=3D2, > metavar=3D('fromsig', 'tosig'))=20 > -parser.add_argument("sigdatafile2", > - help=3D"Second signature file to compare", > - action=3D"store", nargs=3D'?') > + parser.add_argument("sigdatafile1", > + help=3D"First signature file to compare (or signature file > to dump, if second not specified). Not used when using -t/--task.", > + action=3D"store", nargs=3D'?') > =20 > + parser.add_argument("sigdatafile2", > + help=3D"Second signature file to compare", > + action=3D"store", nargs=3D'?') > =20 > options =3D parser.parse_args() > +if is_dump: > + options.color =3D 'never' > + options.dump =3D True > + options.sigdatafile2 =3D None > + options.sigargs =3D None > =20 > if options.debug: > logger.setLevel(logging.DEBUG) > @@ -155,17 +155,32 @@ color =3D (options.color =3D=3D 'always' or > (options.color =3D=3D 'auto' and sys.stdout.is if options.taskargs: > with bb.tinfoil.Tinfoil() as tinfoil: > tinfoil.prepare(config_only=3DTrue) > - if options.sigargs: > - find_compare_task(tinfoil, options.taskargs[0], > options.taskargs[1], options.sigargs[0], options.sigargs[1], > color=3Dcolor) > + if not options.dump and options.sigargs: > + files =3D find_siginfo_task(tinfoil, options.taskargs[0], > options.taskargs[1], options.sigargs[0], options.sigargs[1]) > + else: > + files =3D find_siginfo_task(tinfoil, options.taskargs[0], > options.taskargs[1]) + > + if options.dump: > + logger.debug("Signature file: %s" % files[-1]) > + output =3D bb.siggen.dump_sigfile(files[-1]) > else: > - find_compare_task(tinfoil, options.taskargs[0], > options.taskargs[1], color=3Dcolor) > + if len(files) < 2: > + logger.error('Only one matching sigdata file found > for the specified task (%s %s)' % (options.taskargs[0], > options.taskargs[1])) > + sys.exit(1) > + > + # Recurse into signature comparison > + logger.debug("Signature file (previous): %s" % files[-2]) > + logger.debug("Signature file (latest): %s" % files[-1]) > + output =3D bb.siggen.compare_sigfiles(files[-2], > files[-1], recursecb, color=3Dcolor) else: > if options.sigargs: > logger.error('-s/--signature can only be used together with > -t/--task') sys.exit(1) > try: > - if options.sigdatafile1 and options.sigdatafile2: > - output =3D > bb.siggen.compare_sigfiles(options.sigdatafile1, > options.sigdatafile2, color=3Dcolor) > + if not options.dump and options.sigdatafile1 and > options.sigdatafile2: > + with bb.tinfoil.Tinfoil() as tinfoil: > + tinfoil.prepare(config_only=3DTrue) > + output =3D > bb.siggen.compare_sigfiles(options.sigdatafile1, > options.sigdatafile2, recursecb, color=3Dcolor) elif > options.sigdatafile1: output =3D > bb.siggen.dump_sigfile(options.sigdatafile1) else: @@ -179,5 +194,5 > @@ else: logger.error('Invalid signature data - ensure you are > specifying sigdata/siginfo files') sys.exit(1) > =20 > - if output: > - print('\n'.join(output)) > +if output: > + print('\n'.join(output)) > diff --git a/bitbake/bin/bitbake-dumpsig b/bitbake/bin/bitbake-dumpsig > deleted file mode 100755 > index 95ebd93..0000000 > --- a/bitbake/bin/bitbake-dumpsig > +++ /dev/null > @@ -1,94 +0,0 @@ > -#!/usr/bin/env python3 > - > -# bitbake-dumpsig > -# BitBake task signature dump utility > -# > -# Copyright (C) 2013 Intel Corporation > -# > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > -# > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA. - > -import os > -import sys > -import warnings > -import optparse > -import logging > -import pickle > - > -sys.path.insert(0, > os.path.join(os.path.dirname(os.path.dirname(sys.argv[0])), 'lib')) - > -import bb.tinfoil > -import bb.siggen > -import bb.msg > - > -logger =3D bb.msg.logger_create('bitbake-dumpsig') > - > -def find_siginfo_task(bbhandler, pn, taskname): > - """ Find the most recent signature file for the specified > PN/task """ - > - if not hasattr(bb.siggen, 'find_siginfo'): > - logger.error('Metadata does not support finding signature > data files') > - sys.exit(1) > - > - if not taskname.startswith('do_'): > - taskname =3D 'do_%s' % taskname > - > - filedates =3D bb.siggen.find_siginfo(pn, taskname, None, > bbhandler.config_data) > - latestfiles =3D sorted(filedates.keys(), key=3Dlambda f: > filedates[f])[-1:] > - if not latestfiles: > - logger.error('No sigdata files found matching %s %s' % (pn, > taskname)) > - sys.exit(1) > - > - return latestfiles[0] > - > -parser =3D optparse.OptionParser( > - description =3D "Dumps siginfo/sigdata files written out by > BitBake", > - usage =3D """ > - %prog -t recipename taskname > - %prog sigdatafile""") > - > -parser.add_option("-D", "--debug", > - help =3D "enable debug", > - action =3D "store_true", dest=3D"debug", default =3D False) > - > -parser.add_option("-t", "--task", > - help =3D "find the signature data file for the specified task", > - action=3D"store", dest=3D"taskargs", nargs=3D2, > metavar=3D'recipename taskname') - > -options, args =3D parser.parse_args(sys.argv) > - > -if options.debug: > - logger.setLevel(logging.DEBUG) > - > -if options.taskargs: > - tinfoil =3D bb.tinfoil.Tinfoil() > - tinfoil.prepare(config_only =3D True) > - file =3D find_siginfo_task(tinfoil, options.taskargs[0], > options.taskargs[1]) > - logger.debug("Signature file: %s" % file) > -elif len(args) =3D=3D 1: > - parser.print_help() > - sys.exit(0) > -else: > - file =3D args[1] > - > -try: > - output =3D bb.siggen.dump_sigfile(file) > -except IOError as e: > - logger.error(str(e)) > - sys.exit(1) > -except (pickle.UnpicklingError, EOFError): > - logger.error('Invalid signature data - ensure you are specifying > a sigdata/siginfo file') > - sys.exit(1) > - > -if output: > - print('\n'.join(output)) > diff --git a/bitbake/bin/bitbake-dumpsig b/bitbake/bin/bitbake-dumpsig > new file mode 120000 > index 0000000..b1e8489 > --- /dev/null > +++ b/bitbake/bin/bitbake-dumpsig > @@ -0,0 +1 @@ > +bitbake-diffsigs > \ No newline at end of file > diff --git a/bitbake/bin/bitbake-hashclient > b/bitbake/bin/bitbake-hashclient new file mode 100755 > index 0000000..29ab65f > --- /dev/null > +++ b/bitbake/bin/bitbake-hashclient > @@ -0,0 +1,170 @@ > +#! /usr/bin/env python3 > +# > +# Copyright (C) 2019 Garmin Ltd. > +# > +# SPDX-License-Identifier: GPL-2.0-only > +# > + > +import argparse > +import hashlib > +import logging > +import os > +import pprint > +import sys > +import threading > +import time > + > +try: > + import tqdm > + ProgressBar =3D tqdm.tqdm > +except ImportError: > + class ProgressBar(object): > + def __init__(self, *args, **kwargs): > + pass > + > + def __enter__(self): > + return self > + > + def __exit__(self, *args, **kwargs): > + pass > + > + def update(self): > + pass > + > +sys.path.insert(0, > os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib')) + > +import hashserv > + > +DEFAULT_ADDRESS =3D 'unix://./hashserve.sock' > +METHOD =3D 'stress.test.method' > + > + > +def main(): > + def handle_stats(args, client): > + if args.reset: > + s =3D client.reset_stats() > + else: > + s =3D client.get_stats() > + pprint.pprint(s) > + return 0 > + > + def handle_stress(args, client): > + def thread_main(pbar, lock): > + nonlocal found_hashes > + nonlocal missed_hashes > + nonlocal max_time > + > + client =3D hashserv.create_client(args.address) > + > + for i in range(args.requests): > + taskhash =3D hashlib.sha256() > + taskhash.update(args.taskhash_seed.encode('utf-8')) > + taskhash.update(str(i).encode('utf-8')) > + > + start_time =3D time.perf_counter() > + l =3D client.get_unihash(METHOD, taskhash.hexdigest()) > + elapsed =3D time.perf_counter() - start_time > + > + with lock: > + if l: > + found_hashes +=3D 1 > + else: > + missed_hashes +=3D 1 > + > + max_time =3D max(elapsed, max_time) > + pbar.update() > + > + max_time =3D 0 > + found_hashes =3D 0 > + missed_hashes =3D 0 > + lock =3D threading.Lock() > + total_requests =3D args.clients * args.requests > + start_time =3D time.perf_counter() > + with ProgressBar(total=3Dtotal_requests) as pbar: > + threads =3D [threading.Thread(target=3Dthread_main, > args=3D(pbar, lock), daemon=3DFalse) for _ in range(args.clients)] > + for t in threads: > + t.start() > + > + for t in threads: > + t.join() > + > + elapsed =3D time.perf_counter() - start_time > + with lock: > + print("%d requests in %.1fs. %.1f requests per second" % > (total_requests, elapsed, total_requests / elapsed)) > + print("Average request time %.8fs" % (elapsed / > total_requests)) > + print("Max request time was %.8fs" % max_time) > + print("Found %d hashes, missed %d" % (found_hashes, > missed_hashes)) + > + if args.report: > + with ProgressBar(total=3Dargs.requests) as pbar: > + for i in range(args.requests): > + taskhash =3D hashlib.sha256() > + > taskhash.update(args.taskhash_seed.encode('utf-8')) > + taskhash.update(str(i).encode('utf-8')) > + > + outhash =3D hashlib.sha256() > + outhash.update(args.outhash_seed.encode('utf-8')) > + outhash.update(str(i).encode('utf-8')) > + > + client.report_unihash(taskhash.hexdigest(), > METHOD, outhash.hexdigest(), taskhash.hexdigest()) + > + with lock: > + pbar.update() > + > + parser =3D argparse.ArgumentParser(description=3D'Hash Equivalence > Client') > + parser.add_argument('--address', default=3DDEFAULT_ADDRESS, > help=3D'Server address (default "%(default)s")') > + parser.add_argument('--log', default=3D'WARNING', help=3D'Set > logging level') + > + subparsers =3D parser.add_subparsers() > + > + stats_parser =3D subparsers.add_parser('stats', help=3D'Show server > stats') > + stats_parser.add_argument('--reset', action=3D'store_true', > + help=3D'Reset server stats') > + stats_parser.set_defaults(func=3Dhandle_stats) > + > + stress_parser =3D subparsers.add_parser('stress', help=3D'Run stress > test') > + stress_parser.add_argument('--clients', type=3Dint, default=3D10, > + help=3D'Number of simultaneous clients') > + stress_parser.add_argument('--requests', type=3Dint, default=3D1000, > + help=3D'Number of requests each client > will perform') > + stress_parser.add_argument('--report', action=3D'store_true', > + help=3D'Report new hashes') > + stress_parser.add_argument('--taskhash-seed', default=3D'', > + help=3D'Include string in taskhash') > + stress_parser.add_argument('--outhash-seed', default=3D'', > + help=3D'Include string in outhash') > + stress_parser.set_defaults(func=3Dhandle_stress) > + > + args =3D parser.parse_args() > + > + logger =3D logging.getLogger('hashserv') > + > + level =3D getattr(logging, args.log.upper(), None) > + if not isinstance(level, int): > + raise ValueError('Invalid log level: %s' % args.log) > + > + logger.setLevel(level) > + console =3D logging.StreamHandler() > + console.setLevel(level) > + logger.addHandler(console) > + > + func =3D getattr(args, 'func', None) > + if func: > + client =3D hashserv.create_client(args.address) > + # Try to establish a connection to the server now to detect > failures > + # early > + client.connect() > + > + return func(args, client) > + > + return 0 > + > + > +if __name__ =3D=3D '__main__': > + try: > + ret =3D main() > + except Exception: > + ret =3D 1 > + import traceback > + traceback.print_exc() > + sys.exit(ret) > diff --git a/bitbake/bin/bitbake-hashserv > b/bitbake/bin/bitbake-hashserv new file mode 100755 > index 0000000..1bc1f91 > --- /dev/null > +++ b/bitbake/bin/bitbake-hashserv > @@ -0,0 +1,62 @@ > +#! /usr/bin/env python3 > +# > +# Copyright (C) 2018 Garmin Ltd. > +# > +# SPDX-License-Identifier: GPL-2.0-only > +# > + > +import os > +import sys > +import logging > +import argparse > +import sqlite3 > + > +sys.path.insert(0, > os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib')) + > +import hashserv > + > +VERSION =3D "1.0.0" > + > +DEFAULT_BIND =3D 'unix://./hashserve.sock' > + > + > +def main(): > + parser =3D argparse.ArgumentParser(description=3D'Hash Equivalence > Reference Server. Version=3D%s' % VERSION, > + epilog=3D'''The bind address is > the path to a unix domain socket if it is > + prefixed with > "unix://". Otherwise, it is an IP address > + and port in form > ADDRESS:PORT. To bind to all addresses, leave > + the ADDRESS empty, > e.g. "--bind :8686". To bind to a specific > + IPv6 address, enclose > the address in "[]", e.g. > + "--bind [::1]:8686"''' > + ) > + > + parser.add_argument('--bind', default=3DDEFAULT_BIND, help=3D'Bind > address (default "%(default)s")') > + parser.add_argument('--database', default=3D'./hashserv.db', > help=3D'Database file (default "%(default)s")') > + parser.add_argument('--log', default=3D'WARNING', help=3D'Set > logging level') + > + args =3D parser.parse_args() > + > + logger =3D logging.getLogger('hashserv') > + > + level =3D getattr(logging, args.log.upper(), None) > + if not isinstance(level, int): > + raise ValueError('Invalid log level: %s' % args.log) > + > + logger.setLevel(level) > + console =3D logging.StreamHandler() > + console.setLevel(level) > + logger.addHandler(console) > + > + server =3D hashserv.create_server(args.bind, args.database) > + server.serve_forever() > + return 0 > + > + > +if __name__ =3D=3D '__main__': > + try: > + ret =3D main() > + except Exception: > + ret =3D 1 > + import traceback > + traceback.print_exc() > + sys.exit(ret) > diff --git a/bitbake/bin/bitbake-layers b/bitbake/bin/bitbake-layers > index d184011..a884dc1 100755 > --- a/bitbake/bin/bitbake-layers > +++ b/bitbake/bin/bitbake-layers > @@ -7,18 +7,8 @@ > # Copyright (C) 2011 Mentor Graphics Corporation > # Copyright (C) 2011-2015 Intel Corporation > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > +# SPDX-License-Identifier: GPL-2.0-only > # > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA.=20 > import logging > import os > diff --git a/bitbake/bin/bitbake-prserv b/bitbake/bin/bitbake-prserv > index f38d2dd..1e9b6cb 100755 > --- a/bitbake/bin/bitbake-prserv > +++ b/bitbake/bin/bitbake-prserv > @@ -1,4 +1,8 @@ > #!/usr/bin/env python3 > +# > +# SPDX-License-Identifier: GPL-2.0-only > +# > + > import os > import sys,logging > import optparse > diff --git a/bitbake/bin/bitbake-selftest > b/bitbake/bin/bitbake-selftest index cfa7ac5..041a271 100755 > --- a/bitbake/bin/bitbake-selftest > +++ b/bitbake/bin/bitbake-selftest > @@ -2,18 +2,8 @@ > # > # Copyright (C) 2012 Richard Purdie > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > +# SPDX-License-Identifier: GPL-2.0-only > # > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA.=20 > import os > import sys, logging > @@ -22,6 +12,7 @@ sys.path.insert(0, > os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib import > unittest try: > import bb > + import hashserv > import layerindexlib > except RuntimeError as exc: > sys.exit(str(exc)) > @@ -33,7 +24,10 @@ tests =3D ["bb.tests.codeparser", > "bb.tests.event", > "bb.tests.fetch", > "bb.tests.parse", > + "bb.tests.persist_data", > + "bb.tests.runqueue", > "bb.tests.utils", > + "hashserv.tests", > "layerindexlib.tests.layerindexobj", > "layerindexlib.tests.restapi", > "layerindexlib.tests.cooker"] > diff --git a/bitbake/bin/bitbake-worker b/bitbake/bin/bitbake-worker > index e925054..6776cad 100755 > --- a/bitbake/bin/bitbake-worker > +++ b/bitbake/bin/bitbake-worker > @@ -1,4 +1,7 @@ > #!/usr/bin/env python3 > +# > +# SPDX-License-Identifier: GPL-2.0-only > +# > =20 > import os > import sys > @@ -136,7 +139,7 @@ def sigterm_handler(signum, frame): > os.killpg(0, signal.SIGTERM) > sys.exit() > =20 > -def fork_off_task(cfg, data, databuilder, workerdata, fn, task, > taskname, appends, taskdepdata, extraconfigdata, quieterrors=3DFalse, > dry_run_exec=3DFalse): +def fork_off_task(cfg, data, databuilder, > workerdata, fn, task, taskname, taskhash, unihash, appends, > taskdepdata, extraconfigdata, quieterrors=3DFalse, dry_run_exec=3DFalse): > # We need to setup the environment BEFORE the fork, since # a fork() > or exec*() activates PSEUDO... @@ -231,10 +234,13 @@ def > fork_off_task(cfg, data, databuilder, workerdata, fn, task, taskname, > append the_data.setVar(varname, value) > bb.parse.siggen.set_taskdata(workerdata["sigdata"]) > + if "newhashes" in workerdata: > + > bb.parse.siggen.set_taskhashes(workerdata["newhashes"]) ret =3D 0 > =20 > the_data =3D bb_cache.loadDataFull(fn, appends) > - the_data.setVar('BB_TASKHASH', > workerdata["runq_hash"][task]) > + the_data.setVar('BB_TASKHASH', taskhash) > + the_data.setVar('BB_UNIHASH', unihash) > =20 > bb.utils.set_process_name("%s:%s" % > (the_data.getVar("PN"), taskname.replace("do_", "")))=20 > @@ -373,6 +379,7 @@ class BitbakeWorker(object): > self.handle_item(b"cookerconfig", > self.handle_cookercfg) self.handle_item(b"extraconfigdata", > self.handle_extraconfigdata) self.handle_item(b"workerdata", > self.handle_workerdata) > + self.handle_item(b"newtaskhashes", > self.handle_newtaskhashes) self.handle_item(b"runtask", > self.handle_runtask) self.handle_item(b"finishnow", > self.handle_finishnow) self.handle_item(b"ping", self.handle_ping) > @@ -411,6 +418,10 @@ class BitbakeWorker(object): > bb.msg.loggerDefaultDomains =3D > self.workerdata["logdefaultdomain"] for mc in self.databuilder.mcdata: > self.databuilder.mcdata[mc].setVar("PRSERV_HOST", > self.workerdata["prhost"]) > + self.databuilder.mcdata[mc].setVar("BB_HASHSERVE", > self.workerdata["hashservaddr"]) + > + def handle_newtaskhashes(self, data): > + self.workerdata["newhashes"] =3D pickle.loads(data) > =20 > def handle_ping(self, _): > workerlog_write("Handling ping\n") > @@ -425,10 +436,10 @@ class BitbakeWorker(object): > sys.exit(0) > =20 > def handle_runtask(self, data): > - fn, task, taskname, quieterrors, appends, taskdepdata, > dry_run_exec =3D pickle.loads(data) > + fn, task, taskname, taskhash, unihash, quieterrors, appends, > taskdepdata, dry_run_exec =3D pickle.loads(data) > workerlog_write("Handling runtask %s %s %s\n" % (task, fn, taskname))=20 > - pid, pipein, pipeout =3D fork_off_task(self.cookercfg, > self.data, self.databuilder, self.workerdata, fn, task, taskname, > appends, taskdepdata, self.extraconfigdata, quieterrors, dry_run_exec) > + pid, pipein, pipeout =3D fork_off_task(self.cookercfg, > self.data, self.databuilder, self.workerdata, fn, task, taskname, > taskhash, unihash, appends, taskdepdata, self.extraconfigdata, > quieterrors, dry_run_exec) self.build_pids[pid] =3D task > self.build_pipes[pid] =3D runQueueWorkerPipe(pipein, pipeout) diff > --git a/bitbake/bin/bitdoc b/bitbake/bin/bitdoc index > 2744678..9bd02be 100755 --- a/bitbake/bin/bitdoc > +++ b/bitbake/bin/bitdoc > @@ -1,21 +1,9 @@ > #!/usr/bin/env python3 > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > # > # Copyright (C) 2005 Holger Hans Peter Freyther > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > +# SPDX-License-Identifier: GPL-2.0-only > # > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA.=20 > import optparse, os, sys > =20 > diff --git a/bitbake/bin/git-make-shallow > b/bitbake/bin/git-make-shallow index 296d3a3..57069f7 100755 > --- a/bitbake/bin/git-make-shallow > +++ b/bitbake/bin/git-make-shallow > @@ -1,4 +1,8 @@ > #!/usr/bin/env python3 > +# > +# SPDX-License-Identifier: GPL-2.0-only > +# > + > """git-make-shallow: make the current git repository shallow > =20 > Remove the history of the specified revisions, then optionally > filter the diff --git a/bitbake/bin/toaster b/bitbake/bin/toaster > index ecf66fa..c3472df 100755 > --- a/bitbake/bin/toaster > +++ b/bitbake/bin/toaster > @@ -3,19 +3,9 @@ > # toaster - shell script to start Toaster > =20 > # Copyright (C) 2013-2015 Intel Corp. > - > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License as > published by -# the Free Software Foundation; either version 2 of the > License, or -# (at your option) any later version. > # > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > +# SPDX-License-Identifier: GPL-2.0-or-later > # > -# You should have received a copy of the GNU General Public License > -# along with this program. If not, see http://www.gnu.org/licenses/. > =20 > HELP=3D" > Usage: source toaster start|stop [webport=3D] [noweb] > [nobuild] [toasterdir] diff --git a/bitbake/bin/toaster-eventreplay > b/bitbake/bin/toaster-eventreplay index 80967a0..8fa4ab7 100755 > --- a/bitbake/bin/toaster-eventreplay > +++ b/bitbake/bin/toaster-eventreplay > @@ -1,25 +1,12 @@ > #!/usr/bin/env python3 > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > # > # Copyright (C) 2014 Alex Damian > # > +# SPDX-License-Identifier: GPL-2.0-only > +# > # This file re-uses code spread throughout other Bitbake source > files. # As such, all other copyrights belong to their own right > holders. # > -# > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > -# > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA.=20 > """ > This command takes a filename as a single parameter. The filename is > read diff --git a/bitbake/classes/base.bbclass > b/bitbake/classes/base.bbclass index 71d9ec1..08441fe 100644 > --- a/bitbake/classes/base.bbclass > +++ b/bitbake/classes/base.bbclass > @@ -44,7 +44,7 @@ python do_showdata() { > # emit the metadata which isnt valid shell > for e in bb.data.keys(d): > if d.getVarFlag(e, 'python', False): > - bb.plain("\npython %s () {\n%s}" % (e, > d.getVar(e, True))) > + bb.plain("\npython %s () {\n%s}" % (e, > d.getVar(e))) } > =20 > addtask listtasks > diff --git a/bitbake/contrib/dump_cache.py > b/bitbake/contrib/dump_cache.py index 8963ca4..c6723cb 100755 > --- a/bitbake/contrib/dump_cache.py > +++ b/bitbake/contrib/dump_cache.py > @@ -1,6 +1,4 @@ > #!/usr/bin/env python3 > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > # > # Copyright (C) 2012, 2018 Wind River Systems, Inc. > # > diff --git > a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml > b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml > index f1caaec..46dafee 100644 --- > a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml > +++ > b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml > @@ -31,7 +31,7 @@ Prior to executing BitBake, you should take > advantage of available parallel thread execution on your build host > by setting the > - linkend=3D'var-BB_NUMBER_THREADS'>BB_NUMBER_THREADS<= /link> > + linkend=3D'var-bb-BB_NUMBER_THREADS'>BB_NUMBER_THREADS > variable in your project's local.conf > configuration file. > @@ -87,9 +87,9 @@ > > The layer.conf files are used to > construct key variables such as > - linkend=3D'var-BBPATH'>BBPATH > + linkend=3D'var-bb-BBPATH'>BBPATH and > - linkend=3D'var-BBFILES'>BBFILES. > + linkend=3D'var-bb-BBFILES'>BBFILES. > BBPATH is used to search for configuration and > class files under the conf and > classes @@ -117,19 +117,19 @@ > at certain variables, including: > > > - linkend=3D'var-BB_ENV_WHITELIST'>BB_ENV_WHITELIST > + linkend=3D'var-bb-BB_ENV_WHITELIST'>BB_ENV_WHITELIST= > > - linkend=3D'var-BB_ENV_EXTRAWHITE'>BB_ENV_EXTRAWHITE<= /link> > + linkend=3D'var-bb-BB_ENV_EXTRAWHITE'>BB_ENV_EXTRAWHITE > > - linkend=3D'var-BB_PRESERVE_ENV'>BB_PRESERVE_ENV > + linkend=3D'var-bb-BB_PRESERVE_ENV'>BB_PRESERVE_ENV > > - linkend=3D'var-BB_ORIGENV'>BB_ORIGENV > + linkend=3D'var-bb-BB_ORIGENV'>BB_ORIGENV > > - linkend=3D'var-BITBAKE_UI'>BITBAKE_UI > + linkend=3D'var-bb-BITBAKE_UI'>BITBAKE_UI > > The first four variables in this list relate to how > BitBake treats shell @@ -156,7 +156,7 @@ > BitBake first searches the current working directory for > an optional conf/bblayers.conf configuration > file. This file is expected to contain a > - linkend=3D'var-BBLAYERS'>BBLAYERS > + linkend=3D'var-bb-BBLAYERS'>BBLAYERS > variable that is a space-delimited list of 'layer' directories. > Recall that if BitBake cannot find a > bblayers.conf file, then it is assumed the user > has set the BBPATH @@ -166,10 +166,10 @@ > For each directory (layer) in this list, a > conf/layer.conf file is located and parsed with > the > - linkend=3D'var-LAYERDIR'>LAYERDIR > + linkend=3D'var-bb-LAYERDIR'>LAYERDIR > variable being set to the directory where the layer was found. The > idea is these files automatically set up > - linkend=3D'var-BBPATH'>BBPATH > + linkend=3D'var-bb-BBPATH'>BBPATH and other > variables correctly for a given build directory. > =20 > @@ -189,7 +189,7 @@ > depending on the environment variables previously > mentioned or set in the configuration files. > The > - "Variables > Glossary" > + "Variables > Glossary" chapter presents a full list of variables. > > =20 > @@ -204,7 +204,7 @@ > > The base.bbclass file is always > included. Other classes that are specified in the configuration using > the > - linkend=3D'var-INHERIT'>INHERIT > + linkend=3D'var-bb-INHERIT'>INHERIT variable > are also included. BitBake searches for class files in a > classes subdirectory under > @@ -270,7 +270,7 @@ > =20 > > During the configuration phase, BitBake will have set > - linkend=3D'var-BBFILES'>BBFILES. > + linkend=3D'var-bb-BBFILES'>BBFILES. BitBake > now uses it to construct a list of recipes to parse, along with any > append files (.bbappend) to apply. > @@ -292,7 +292,7 @@ > Any inherit statements cause BitBake to find and > then parse class files (.bbclass) > using > - linkend=3D'var-BBPATH'>BBPATH > + linkend=3D'var-bb-BBPATH'>BBPATH as the > search path. Finally, BitBake parses in order any append files found > in BBFILES. > @@ -303,8 +303,8 @@ > pieces of metadata. > For example, in bitbake.conf the > recipe name and version are used to set the variables > - PN and > - PV: > + PN > and > + PV: > > PN =3D "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', > False),d)[0] or 'defaultpkgname'}" PV =3D > "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[1] > or '1.0'}" @@ -336,7 +336,7 @@ recipe information. > The validity of this cache is determined by first > computing a checksum of the base configuration data (see > - linkend=3D'var-BB_HASHCONFIG_WHITELIST'>BB_HASHCONFIG_WHITELIST= ) > + linkend=3D'var-bb-BB_HASHCONFIG_WHITELIST'>BB_HASHCONFIG_WHITEL= IST) > and then checking if the checksum matches. If that checksum matches > what is in the cache and the recipe and class files have not changed, > Bitbake is able to use @@ -384,9 +384,9 @@ > the recipe can be known. > Each recipe's PROVIDES list is > created implicitly through the recipe's > - PN > variable > + PN > variable and explicitly through the recipe's > - linkend=3D'var-PROVIDES'>PROVIDES > + linkend=3D'var-bb-PROVIDES'>PROVIDES > variable, which is optional. > =20 > @@ -427,7 +427,7 @@ > PREFERRED_PROVIDER_virtual/kernel =3D "linux-yocto" > > The default > - linkend=3D'var-PREFERRED_PROVIDER'>PREFERRED_PROVIDER > + linkend=3D'var-bb-PREFERRED_PROVIDER'>PREFERRED_PROVIDER > is the provider with the same name as the target. Bitbake iterates > through each target it needs to build and resolves them and their > dependencies using this process. @@ -439,10 +439,10 @@ > BitBake defaults to the highest version of a provider. > Version comparisons are made using the same method as > Debian. You can use the > - linkend=3D'var-PREFERRED_VERSION'>PREFERRED_VERSION<= /link> > + linkend=3D'var-bb-PREFERRED_VERSION'>PREFERRED_VERSION > variable to specify a particular version. You can influence the order > by using the > - linkend=3D'var-DEFAULT_PREFERENCE'>DEFAULT_PREFERENCE > + linkend=3D'var-bb-DEFAULT_PREFERENCE'>DEFAULT_PREFERENCE > variable. > =20 > @@ -464,7 +464,7 @@ > BitBake defaults to selecting the most recent > version, unless otherwise specified. > If the recipe in question has a > - linkend=3D'var-DEFAULT_PREFERENCE'>DEFAULT_PREFERENCE > + linkend=3D'var-bb-DEFAULT_PREFERENCE'>DEFAULT_PREFERENCE > set lower than the other recipes (default is 0), then it will not be > selected. This allows the person or persons maintaining > @@ -475,9 +475,9 @@ > =20 > > If the first recipe is named > a_1.1.bb, then the > - PN > variable > + PN > variable will be set to =E2=80=9Ca=E2=80=9D, and the > - PV > + PV > variable will be set to 1.1. > > =20 > @@ -532,11 +532,11 @@ > > Dependencies are defined through several variables. > You can find information about variables BitBake uses in > - the Variables > Glossary > + the Variables > Glossary near the end of this manual. > At a basic level, it is sufficient to know that BitBake > uses the > - linkend=3D'var-DEPENDS'>DEPENDS and > - linkend=3D'var-RDEPENDS'>RDEPENDS variables > when > + linkend=3D'var-bb-DEPENDS'>DEPENDS and > + linkend=3D'var-bb-RDEPENDS'>RDEPENDS > variables when calculating dependencies. > =20 > @@ -560,7 +560,7 @@ > =20 > > The build now starts with BitBake forking off threads up > to the limit set in the > - linkend=3D'var-BB_NUMBER_THREADS'>BB_NUMBER_THREADS<= /link> > + linkend=3D'var-bb-BB_NUMBER_THREADS'>BB_NUMBER_THREADS > variable. BitBake continues to fork threads as long as there are > tasks ready to run, those tasks have all their dependencies met, and > the thread threshold has not been @@ -574,7 +574,7 @@ > =20 > > As each task completes, a timestamp is written to the > directory specified by the > - linkend=3D'var-STAMP'>STAMP variable. > + linkend=3D'var-bb-STAMP'>STAMP variable. On > subsequent runs, BitBake looks in the build directory within > tmp/stamps and does not rerun tasks that are > already completed unless a timestamp is found to be invalid. @@ > -618,7 +618,7 @@ > Tasks can be either a shell task or a Python task. > For shell tasks, BitBake writes a shell script to > - ${ linkend=3D'var-T'>T}/run.do_tasknam= e.pid > + ${ linkend=3D'var-bb-T'>T}/run.do_task= name.pid > and then executes the script. The generated shell script contains all > the exported variables, and the shell functions with all variables > expanded. @@ -645,10 +645,10 @@ > behavior: > > > - linkend=3D'var-BB_SCHEDULER'>BB_SCHEDULER > + linkend=3D'var-bb-BB_SCHEDULER'>BB_SCHEDULER > > - linkend=3D'var-BB_SCHEDULERS'>BB_SCHEDULERS > + linkend=3D'var-bb-BB_SCHEDULERS'>BB_SCHEDULERS > > It is possible to have functions run before and after a > task's main @@ -684,7 +684,7 @@ > The simplistic approach for excluding the working > directory is to set it to some fixed value and create the checksum > for the "run" script. BitBake goes one step better and uses the > - linkend=3D'var-BB_HASHBASE_WHITELIST'>BB_HASHBASE_WHITELIST > + linkend=3D'var-bb-BB_HASHBASE_WHITELIST'>BB_HASHBASE_WHITELIST<= /filename> > variable to define a list of variables that should never be included > when generating the signatures. > @@ -795,7 +795,7 @@ > This results in any metadata change that changes the > task hash, automatically causing the task to be run again. > This removes the need to bump > - PR > + PR > values, and changes to metadata automatically ripple > across the build. > =20 > @@ -884,7 +884,7 @@ > =20 > > BitBake first calls the function defined by the > - linkend=3D'var-BB_HASHCHECK_FUNCTION'>BB_HASHCHECK_FUNCTION > + linkend=3D'var-bb-BB_HASHCHECK_FUNCTION'>BB_HASHCHECK_FUNCTION<= /filename> > variable with a list of tasks and corresponding hashes it wants to > build. This function is designed to be fast and returns a list > @@ -908,7 +908,7 @@ > For example, it is pointless to obtain a compiler if you > already have the compiled binary. > To handle this, BitBake calls the > - linkend=3D'var-BB_SETSCENE_DEPVALID'>BB_SETSCENE_DEPVALID > + linkend=3D'var-bb-BB_SETSCENE_DEPVALID'>BB_SETSCENE_DEPVALID > function for each successful setscene task to know whether or not it > needs to obtain the dependencies of that task. > @@ -916,7 +916,7 @@ > > Finally, after all the setscene tasks have executed, > BitBake calls the function listed in > - linkend=3D'var-BB_SETSCENE_VERIFY_FUNCTION2'>BB_SETSCENE_VERIFY= _FUNCTION2 > + linkend=3D'var-bb-BB_SETSCENE_VERIFY_FUNCTION2'>BB_SETSCENE_VER= IFY_FUNCTION2 > with the list of tasks BitBake thinks has been "covered". The > metadata can then ensure that this list is correct and can inform > BitBake that it wants specific tasks to be run regardless diff --git > a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml > b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml > index 29ae486..6840408 100644 --- > a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml > +++ > b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml @@ > -44,7 +44,7 @@ This code sets up an instance of the > fetch class. The instance uses a space-separated list of URLs from the > - linkend=3D'var-SRC_URI'>SRC_URI > + linkend=3D'var-bb-SRC_URI'>SRC_URI variable > and then calls the download method to download > the files. > @@ -78,7 +78,7 @@ > Pre-mirror > Sites: BitBake first uses pre-mirrors to try and find > source files. These locations are defined using the > - linkend=3D'var-PREMIRRORS'>PREMIRRORS > + linkend=3D'var-bb-PREMIRRORS'>PREMIRRORS > variable. > Source URI: > @@ -88,7 +88,7 @@ > Mirror Sites: > If fetch failures occur, BitBake next uses > mirror locations as defined by the > - linkend=3D'var-MIRRORS'>MIRRORS > + linkend=3D'var-bb-MIRRORS'>MIRRORS variable. > > > @@ -144,7 +144,7 @@ > Any source files that are not local (i.e. > downloaded from the Internet) are placed into the > download directory, which is specified by the > - linkend=3D'var-DL_DIR'>DL_DIR > + linkend=3D'var-bb-DL_DIR'>DL_DIR variable. > > =20 > @@ -184,11 +184,11 @@ > =20 > > If > - linkend=3D'var-BB_STRICT_CHECKSUM'>BB_STRICT_CHECKSUM > + linkend=3D'var-bb-BB_STRICT_CHECKSUM'>BB_STRICT_CHECKSUM > is set, any download without a checksum triggers an error message. > The > - linkend=3D'var-BB_NO_NETWORK'>BB_NO_NETWORK > + linkend=3D'var-bb-BB_NO_NETWORK'>BB_NO_NETWORK > variable can be used to make any attempted network access a fatal > error, which is useful for checking that mirrors are complete as well > as other things. @@ -265,11 +265,11 @@ > The filename you specify within the URL can be > either an absolute or relative path to a file. > If the filename is relative, the contents of the > - linkend=3D'var-FILESPATH'>FILESPATH > + linkend=3D'var-bb-FILESPATH'>FILESPATH > variable is used in the same way PATH is used to > find executables. If the file cannot be found, it is assumed that it > is available in > - linkend=3D'var-DL_DIR'>DL_DIR > + linkend=3D'var-bb-DL_DIR'>DL_DIR by the > time the download() method is called. > =20 > @@ -304,7 +304,7 @@ > allows the name of the downloaded file to be > specified. Specifying the name of the downloaded file is useful > for avoiding collisions in > - linkend=3D'var-DL_DIR'>DL_DIR > + linkend=3D'var-bb-DL_DIR'>DL_DIR when > dealing with multiple files that have the same name. > =20 > @@ -355,7 +355,7 @@ > A special value of "now" causes the checkout > to be updated on every build. > > - linkend=3D'var-CVSDIR'>CVSDIR: > + linkend=3D'var-bb-CVSDIR'>CVSDIR: > Specifies where a temporary checkout is saved. The location is often > DL_DIR/cvs. > @@ -395,7 +395,7 @@ > "date": > Specifies a date. > If no "date" is specified, the > - linkend=3D'var-SRCDATE'>SRCDATE > + linkend=3D'var-bb-SRCDATE'>SRCDATE of the > configuration is used to checkout a specific date. The special value > of "now" causes the checkout to be updated on every build. > @@ -406,7 +406,7 @@ > to which the module is unpacked. > You are forcing the module into a special > directory relative to > - linkend=3D'var-CVSDIR'>CVSDIR. > + linkend=3D'var-bb-CVSDIR'>CVSDIR. > "rsh" > Used in conjunction with the "method" > parameter. @@ -448,7 +448,7 @@ > FETCHCMD_svn, which defaults > to "svn". > The fetcher's temporary working directory is set by > - linkend=3D'var-SVNDIR'>SVNDIR, > + linkend=3D'var-bb-SVNDIR'>SVNDIR, which is > usually DL_DIR/svn. > =20 > @@ -509,7 +509,7 @@ > source control system. > The fetcher works by creating a bare clone of the > remote into > - linkend=3D'var-GITDIR'>GITDIR, > + linkend=3D'var-bb-GITDIR'>GITDIR, which is > usually DL_DIR/git2. This bare clone is then > cloned into the work directory during the unpack stage when a > specific tree is checked out. @@ -588,6 +588,14 @@ > The name of the path in which to place the > checkout. By default, the path is git/. > > + "usehead": > + Enables local git:// > URLs to use the > + current branch HEAD as the revision for use > with > + AUTOREV. > + The "usehead" parameter implies no branch > and only works > + when the transfer protocol is > + file://. > + > > Here are some example URLs: > > @@ -604,7 +612,7 @@ > This fetcher submodule inherits from the > Git fetcher and > extends that fetcher's behavior by fetching a repository's submodules. > - linkend=3D'var-SRC_URI'>SRC_URI > + linkend=3D'var-bb-SRC_URI'>SRC_URI is > passed to the Git fetcher as described in the " linkend=3D'git-fetcher'>Git Fetcher > (git://)" section. @@ -639,9 +647,9 @@ > =20 > > To use this fetcher, make sure your recipe has proper > - linkend=3D'var-SRC_URI'>SRC_URI, > - linkend=3D'var-SRCREV'>SRCREV, and > - linkend=3D'var-PV'>PV settings. > + linkend=3D'var-bb-SRC_URI'>SRC_URI, > + linkend=3D'var-bb-SRCREV'>SRCREV, and > + linkend=3D'var-bb-PV'>PV settings. Here is > an example: > SRC_URI =3D > "ccrc://cc.example.org/ccrc;vob=3D/example_vob;module=3D/example_module" > @@ -726,15 +734,15 @@ FETCHCMD_p4, which defaults > to "p4". > The fetcher's temporary working directory is set by > - linkend=3D'var-P4DIR'>P4DIR, > + linkend=3D'var-bb-P4DIR'>P4DIR, which > defaults to "DL_DIR/p4". > =20 > > To use this fetcher, make sure your recipe has proper > - linkend=3D'var-SRC_URI'>SRC_URI, > - linkend=3D'var-SRCREV'>SRCREV, and > - linkend=3D'var-PV'>PV values. > + linkend=3D'var-bb-SRC_URI'>SRC_URI, > + linkend=3D'var-bb-SRCREV'>SRCREV, and > + linkend=3D'var-bb-PV'>PV values. The p4 > executable is able to use the config file defined by your system's > P4CONFIG environment variable in order to define > the Perforce server URL and port, username, and @@ -785,9 +793,9 @@ > google-repo source control > system. The fetcher works by initiating and syncing sources of the > repository into > - linkend=3D'var-REPODIR'>REPODIR, > + linkend=3D'var-bb-REPODIR'>REPODIR, which > is usually > - linkend=3D'var-DL_DIR'>DL_DIR/repo<= /filename>. > + linkend=3D'var-bb-DL_DIR'>DL_DIR/re= po. > =20 > > @@ -824,19 +832,22 @@ > Bazaar (bzr://) > > > - Trees using Git Annex > (gitannex://) > + Mercurial (hg://) > > > - Secure FTP (sftp://) > + npm (npm://) > > > - Secure Shell (ssh://) > + OSC (osc://) > > > - OSC (osc://) > + Secure FTP (sftp://) > > > - Mercurial (hg://) > + Secure Shell (ssh://) > + > + > + Trees using Git Annex > (gitannex://) > > No documentation currently exists for these lesser > used diff --git > a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.xml > b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.xml index > 9076f0f..39066e4 100644 --- > a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.xml +++ > b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.xml @@ > -194,7 +194,7 @@ When you run BitBake, it begins looking for > metadata files. The > - linkend=3D'var-BBPATH'>BBPATH > + linkend=3D'var-bb-BBPATH'>BBPATH variable > is what tells BitBake where to look for those files. > BBPATH is not set and you need to set it. > Without BBPATH, Bitbake cannot @@ -273,14 > +273,14 @@ some editor to create the bitbake.conf > so that it contains the following: > > - PN =3D > "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0] > or 'defaultpkgname'}" > + PN =3D > "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0] > or 'defaultpkgname'}" class=3D'monospaced'> > - TMPDIR =3D "${TOPDIR}/tmp" > - CACHE =3D "${TMPDIR}/cache" > - STAMP =3D > "${TMPDIR}/${PN}/stamps" > - T =3D "${TMPDIR}/${PN}/work" > - B =3D "${TMPDIR}/${PN}" > + TMPDIR =3D "${TOPDIR}/tmp" > + CACHE =3D "${TMPDIR}/cache" > + STAMP =3D > "${TMPDIR}/${PN}/stamps" > + T =3D "${TMPDIR}/${PN}/work" > + B =3D "${TMPDIR}/${PN}" > > > Without a value for PN, the > @@ -402,12 +402,12 @@ > Move to the conf directory and > create a layer.conf file that has the following: > > - BBPATH .=3D ":${LAYERDIR}" > + BBPATH .=3D ":${LAYERDIR}" > =20 > - BBFILES +=3D "${LAYERDIR}/*.bb" > + BBFILES +=3D > "${LAYERDIR}/*.bb"=20 > - linkend=3D'var-BBFILE_COLLECTIONS'>BBFILE_COLLECTIONS +=3D > "mylayer" > - linkend=3D'var-BBFILE_PATTERN'>BBFILE_PATTERN_mylayer :=3D > "^${LAYERDIR_RE}/" > + linkend=3D'var-bb-BBFILE_COLLECTIONS'>BBFILE_COLLECTIONS +=3D > "mylayer" > + linkend=3D'var-bb-BBFILE_PATTERN'>BBFILE_PATTERN_mylayer :=3D > "^${LAYERDIR_RE}/" For information on these > variables, click the links to go to the definitions in the > glossary. @@ -416,9 +416,9 @@ > a recipe file named > printhello.bb that has the following: > > - DESCRIPTION =3D "Prints > Hello World" > - PN =3D 'printhello' > - PV =3D '1' > + DESCRIPTION =3D "Prints > Hello World" > + PN =3D 'printhello' > + PV =3D '1' > =20 > python do_build() { > bb.plain("********************"); > diff --git > a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml > b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml index > f7d312a..8f2a960 100644 --- > a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml +++ > b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml @@ > -781,7 +781,7 @@ target, you must also enable BitBake to perform > multiple configuration builds. Enabling is accomplished by setting the > - linkend=3D'var-BBMULTICONFIG'>BBMULTICONFIG > + linkend=3D'var-bb-BBMULTICONFIG'>BBMULTICONFIG > variable in the local.conf configuration file. > As an example, suppose you had configuration > files @@ -791,7 +791,7 @@ > The following statement in the > local.conf file both enables > BitBake to perform multiple configuration builds > and > - specifies the two multiconfigs: > + specifies the two extra multiconfigs: > > BBMULTICONFIG =3D "target1 target2" > > @@ -803,13 +803,13 @@ > builds, use the following command form to start > the builds: > > - $ bitbake > [multiconfig:multiconfigname:]tar= get > [[[multiconfig:multiconfigname:]t= arget] ... ] > + $ bitbake > [mc:multiconfigname:]target > [[[mc:multiconfigname:]target] ... ] > > - Here is an example for two multiconfigs: > + Here is an example for two extra multiconfigs: > target1 and > target2: > > - $ bitbake multiconfig:target1:target > multiconfig:target2:target > + $ bitbake mc::target > mc:target1:target > mc:target2:target > > @@ -837,13 +837,13 @@ > build, you must declare the dependencies in the > recipe using the following statement form: > > - task_or_package[mcdepends] =3D > "multiconfig:from_multiconfig:to_= multiconfig:recipe_name:task_on_which_to_depend" > + task_or_package[mcdepends] =3D > "mc:from_multiconfig:to_multiconf= ig:recipe_name:task_o= n_which_to_depend" > To better show how to use this statement, consider an > example with two multiconfigs: > target1 and target2: > > - image_task[mcdepends] =3D > "multiconfig:target1:target2:image2:rootfs_task" > + image_task[mcdepends] =3D > "mc:target1:target2:image2:rootfs= _task" > In this example, the > from_multiconfig is > "target1" and @@ -859,7 +859,7 @@ > Once you set up this dependency, you can build the > "target1" multiconfig using a BitBake command as > follows: > - $ bitbake multiconfig:target1:image1 > + $ bitbake mc:target1:image1 > > This command executes all the tasks needed to > create image1 for the "target1" > @@ -875,7 +875,7 @@ > Consider this change to the statement in the > image1 recipe: > > - image_task[mcdepends] =3D > "multiconfig:target1:target2:image2:image_task" > + image_task[mcdepends] =3D > "mc:target1:target2:image2:image_= task" > In this case, BitBake must create > image2 for the > "target2" diff --git > a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml > b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml > index 2490f6e..421364c 100644 --- > a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml > +++ > b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml @@ > -61,6 +61,78 @@ =20 > +
> + Modifying Existing Variables > + > + > + Sometimes you need to modify existing variables. > + Following are some cases where you might find you > want to > + modify an existing variable: > + > + > + Customize a recipe that uses the variable. > + > + > + Change a variable's default value used in a > + *.bbclass file. > + > + > + Change the variable in a > *.bbappend > + file to override the variable in the > original recipe. > + > + > + Change the variable in a configuration file > so that the > + value overrides an existing configuration. > + > + > + > + > + > + Changing a variable value can sometimes depend on > how the > + value was originally assigned and also on the desired > + intent of the change. > + In particular, when you append a value to a variable > that > + has a default value, the resulting value might not > be what > + you expect. > + In this case, the value you provide might replace > the value > + rather than append to the default value. > + > + > + > + If after you have changed a variable's value and > something > + unexplained occurs, you can use BitBake to check the > actual > + value of the suspect variable. > + You can make these checks for both configuration and > recipe > + level changes: > + > + > + For configuration changes, use the following: > + > + $ bitbake -e > + > + This command displays variable values after > the > + configuration files (i.e. > local.conf, > + bblayers.conf, > + bitbake.conf and so > forth) have > + been parsed. > + > + Variables that are exported to the > environment are > + preceded by the string "export" in the > command's > + output. > + > + > + > + For recipe changes, use the following: > + > + $ bitbake recipe -e | grep VARIABLE=3D" > + > + This command checks to see if the variable > actually > + makes it into a specific recipe. > + > + > + > +
> + >
> Line Joining > =20 > @@ -297,9 +369,8 @@ > =20 > > These operators differ from the ":=3D", ".=3D", "=3D.", > "+=3D", and "=3D+" > - operators in that their effects are deferred > - until after parsing completes rather than being > immediately > - applied. > + operators in that their effects are applied at > variable > + expansion time rather than being immediately applied. > Here are some examples: > > B =3D "bval" > @@ -348,18 +419,22 @@ > FOO =3D "123 456 789 123456 123 456 123 456" > FOO_remove =3D "123" > FOO_remove =3D "456" > - FOO2 =3D "abc def ghi abcdef abc def abc def" > - FOO2_remove =3D "abc def" > + FOO2 =3D " abc def ghi abcdef abc def abc def def" > + FOO2_remove =3D " \ > + def \ > + abc \ > + ghi \ > + " > > The variable FOO becomes > - "  789 123456    " > + "  789 123456    " > and FOO2 becomes > - "  ghi abcdef    ". > + > "     jkl  abcdef   &nb= sp;  ". > =20 > > Like "_append" and "_prepend", "_remove" > - is deferred until after parsing completes. > + is applied at variable expansion time. > >
> =20 > @@ -503,7 +578,7 @@ > > =20 >
> - Unseting variables > + Unsetting variables > =20 > > It is possible to completely remove a variable or a > variable flag @@ -595,7 +670,7 @@ > =20 > > BitBake uses > - linkend=3D'var-OVERRIDES'>OVERRIDES > + linkend=3D'var-bb-OVERRIDES'>OVERRIDES to > control what variables are overridden after BitBake parses recipes > and configuration files. This section describes how you can use > @@ -705,7 +780,7 @@ > =20 > Internally, this is implemented by > prepending the task (e.g. "task-compile:") to the value of > - linkend=3D'var-OVERRIDES'>OVERRIDES > + linkend=3D'var-bb-OVERRIDES'>OVERRIDES for > the local datastore of the do_compile > task.=20 > @@ -724,17 +799,15 @@ > Key Expansion > =20 > > - Key expansion happens when the BitBake datastore is > finalized > - just before BitBake expands overrides. > + Key expansion happens when the BitBake datastore is > finalized. To better understand this, consider the following example: > > A${B} =3D "X" > B =3D "2" > A2 =3D "Y" > > - In this case, after all the parsing is complete, and > - before any overrides are handled, BitBake expands > - ${B} into "2". > + In this case, after all the parsing is complete, > + BitBake expands ${B} into "2". > This expansion causes A2, which > was set to "Y" before the expansion, to become "X". > > @@ -868,7 +941,7 @@ > =20 > > BitBake uses the > - linkend=3D'var-BBPATH'>BBPATH > + linkend=3D'var-bb-BBPATH'>BBPATH variable > to locate needed include and class files. Additionally, BitBake > searches the current directory for include and > require @@ -1086,7 +1159,7 @@ > > When creating a configuration file > (.conf), you can use the > - linkend=3D'var-INHERIT'>INHERIT > + linkend=3D'var-bb-INHERIT'>INHERIT > configuration directive to inherit a class. BitBake only supports > this directive when used within a configuration file. > @@ -1370,7 +1443,7 @@ > > > BitBake-style Python functions generate a > separate > - ${ linkend=3D'var-T'>T}/run.function-name.pid > + ${ linkend=3D'var-bb-T'>T}/run.function-name.pid > script that is executed to run the function, and also generate a log > file in > ${T}/log.function-name.pid > @@ -1773,7 +1846,7 @@ things exported or listed in its whitelist to > ensure that the build environment is reproducible and consistent. > You can prevent this "cleaning" by setting the > - linkend=3D'var-BB_PRESERVE_ENV'>BB_PRESERVE_ENV > + linkend=3D'var-bb-BB_PRESERVE_ENV'>BB_PRESERVE_ENV > variable. > Consequently, if you do want something to get passed > into the @@ -1783,9 +1856,9 @@ > Tell BitBake to load what you want from the > environment into the datastore. > You can do so through the > - linkend=3D'var-BB_ENV_WHITELIST'>BB_ENV_WHITELIST > + linkend=3D'var-bb-BB_ENV_WHITELIST'>BB_ENV_WHITELIST= > and > - linkend=3D'var-BB_ENV_EXTRAWHITE'>BB_ENV_EXTRAWHITE<= /link> > + linkend=3D'var-bb-BB_ENV_EXTRAWHITE'>BB_ENV_EXTRAWHITE > variables. For example, assume you want to prevent the build system > from accessing your $HOME/.ccache > @@ -1824,7 +1897,7 @@ > from the original execution environment. > Bitbake saves a copy of the original environment into > a special variable named > - linkend=3D'var-BB_ORIGENV'>BB_ORIGENV. > + linkend=3D'var-bb-BB_ORIGENV'>BB_ORIGENV. > =20 > > @@ -1883,7 +1956,7 @@ > [depends]= : > Controls inter-task dependencies. > See the > - linkend=3D'var-DEPENDS'>DEPENDS > + linkend=3D'var-bb-DEPENDS'>DEPENDS variable > and the "Inter-Task > Dependencies" section for more information. > @@ -1891,7 +1964,7 @@ > [deptask]= : > Controls task build-time dependencies. > See the > - linkend=3D'var-DEPENDS'>DEPENDS > + linkend=3D'var-bb-DEPENDS'>DEPENDS variable > and the "Build Dependencies" > section for more information. > @@ -1937,7 +2010,7 @@ > of cores but certain tasks need to be > rate-limited due to various kinds of resource constraints (e.g. to > avoid network throttling). number_threads works > similarly to the > - linkend=3D'var-BB_NUMBER_THREADS'>BB_NUMBER_THREADS<= /link> > + linkend=3D'var-bb-BB_NUMBER_THREADS'>BB_NUMBER_THREADS > variable but is task-specific.=20 > Set the value globally. > @@ -1971,9 +2044,9 @@ > [rdepends]: > Controls inter-task runtime dependencies. > See the > - linkend=3D'var-RDEPENDS'>RDEPENDS > + linkend=3D'var-bb-RDEPENDS'>RDEPENDS > variable, the > - linkend=3D'var-RRECOMMENDS'>RRECOMMENDS > + linkend=3D'var-bb-RRECOMMENDS'>RRECOMMENDS > variable, and the "Inter-Task > Dependencies" section for more information. > @@ -1981,9 +2054,9 @@ > [rdeptask]: > Controls task runtime dependencies. > See the > - linkend=3D'var-RDEPENDS'>RDEPENDS > + linkend=3D'var-bb-RDEPENDS'>RDEPENDS > variable, the > - linkend=3D'var-RRECOMMENDS'>RRECOMMENDS > + linkend=3D'var-bb-RRECOMMENDS'>RRECOMMENDS > variable, and the "Runtime > Dependencies" section for more information. > @@ -1996,9 +2069,9 @@ > [recrdeptask]: > Controls task recursive runtime dependencies. > See the > - linkend=3D'var-RDEPENDS'>RDEPENDS > + linkend=3D'var-bb-RDEPENDS'>RDEPENDS > variable, the > - linkend=3D'var-RRECOMMENDS'>RRECOMMENDS > + linkend=3D'var-bb-RRECOMMENDS'>RRECOMMENDS > variable, and the "Recursive > Dependencies" section for more information. > @@ -2127,7 +2200,7 @@ > Any given datastore only has one such event > executed against it, however. > If > - linkende=3D'var-BB_INVALIDCONF'>BB_INVALIDCONF > + linkende=3D'var-bb-BB_INVALIDCONF'>BB_INVALIDCONF > is set in the datastore by the event handler, the configuration is > reparsed and a new event triggered, allowing the metadata to update > configuration. @@ -2256,17 +2329,17 @@ > from a single recipe file multiple incarnations of that > recipe file where all incarnations are buildable. > These features are enabled through the > - linkend=3D'var-BBCLASSEXTEND'>BBCLASSEXTEND > + linkend=3D'var-bb-BBCLASSEXTEND'>BBCLASSEXTEND > and > - linkend=3D'var-BBVERSIONS'>BBVERSIONS > + linkend=3D'var-bb-BBVERSIONS'>BBVERSIONS > variables. > The mechanism for this class extension is extremely > specific to the implementation. > Usually, the recipe's > - linkend=3D'var-PROVIDES'>PROVIDES, > - linkend=3D'var-PN'>PN, and > - linkend=3D'var-DEPENDS'>DEPENDS > + linkend=3D'var-bb-PROVIDES'>PROVIDES, > + linkend=3D'var-bb-PN'>PN, and > + linkend=3D'var-bb-DEPENDS'>DEPENDS > variables would need to be modified by the extension class. For > specific examples, see the OE-Core native, > nativesdk, @@ -2287,7 +2360,7 @@ > project from a single recipe file. > You can also specify conditional metadata > (using the > - linkend=3D'var-OVERRIDES'>OVERRIDES > + linkend=3D'var-bb-OVERRIDES'>OVERRIDES > mechanism) for a single version, or an optionally named range of > versions. Here is an example: > @@ -2306,7 +2379,7 @@ > into overrides, but it is also made available > for the metadata to use in the variable that defines the base recipe > versions for use in file:// search paths > - ( linkend=3D'var-FILESPATH'>FILESPATH). > + ( linkend=3D'var-bb-FILESPATH'>FILESPATH). > > > @@ -2408,7 +2481,7 @@ > =20 > > BitBake uses the > - linkend=3D'var-DEPENDS'>DEPENDS > + linkend=3D'var-bb-DEPENDS'>DEPENDS variable > to manage build time dependencies. The [deptask] > varflag for tasks signifies the task of each > @@ -2429,9 +2502,9 @@ > =20 > > BitBake uses the > - linkend=3D'var-PACKAGES'>PACKAGES, > - linkend=3D'var-RDEPENDS'>RDEPENDS, and > - linkend=3D'var-RRECOMMENDS'>RRECOMMENDS > + linkend=3D'var-bb-PACKAGES'>PACKAGES, > + linkend=3D'var-bb-RDEPENDS'>RDEPENDS, and > + linkend=3D'var-bb-RRECOMMENDS'>RRECOMMENDS > variables to manage runtime dependencies. > =20 > @@ -2686,7 +2759,7 @@ > =20 > > These checksums are stored in > - linkend=3D'var-STAMP'>STAMP. > + linkend=3D'var-bb-STAMP'>STAMP. You can > examine the checksums using the following BitBake command: > $ bitbake-dumpsigs > @@ -2708,44 +2781,44 @@ > The following list describes related variables: > > > - linkend=3D'var-BB_HASHCHECK_FUNCTION'>BB_HASHCHECK_FUNCTION: > + linkend=3D'var-bb-BB_HASHCHECK_FUNCTION'>BB_HASHCHECK_FUNCTION<= /filename>: > Specifies the name of the function to call during the "setscene" part > of the task's execution in order to validate the list of task hashes. > > > - linkend=3D'var-BB_SETSCENE_DEPVALID'>BB_SETSCENE_DEPVALID: > + linkend=3D'var-bb-BB_SETSCENE_DEPVALID'>BB_SETSCENE_DEPVALID: > Specifies a function BitBake calls that determines whether BitBake > requires a setscene dependency to be met. > > > - linkend=3D'var-BB_SETSCENE_VERIFY_FUNCTION2'>BB_SETSCENE_VERIFY= _FUNCTION2: > + linkend=3D'var-bb-BB_SETSCENE_VERIFY_FUNCTION2'>BB_SETSCENE_VER= IFY_FUNCTION2: > Specifies a function to call that verifies the list of planned task > execution before the main task execution happens. > > > - linkend=3D'var-BB_STAMP_POLICY'>BB_STAMP_POLICY: > + linkend=3D'var-bb-BB_STAMP_POLICY'>BB_STAMP_POLICY: > Defines the mode for comparing timestamps of stamp files. > > - linkend=3D'var-BB_STAMP_WHITELIST'>BB_STAMP_WHITELIST: > + linkend=3D'var-bb-BB_STAMP_WHITELIST'>BB_STAMP_WHITELIST: > Lists stamp files that are looked at when the stamp policy is > "whitelist". > > - linkend=3D'var-BB_TASKHASH'>BB_TASKHASH: > + linkend=3D'var-bb-BB_TASKHASH'>BB_TASKHASH: > Within an executing task, this variable holds the hash of the task as > returned by the currently enabled signature generator. > > > - linkend=3D'var-STAMP'>STAMP: > + linkend=3D'var-bb-STAMP'>STAMP: The base > path to create stamp files. > > - linkend=3D'var-STAMPCLEAN'>STAMPCLEAN: > + linkend=3D'var-bb-STAMPCLEAN'>STAMPCLEAN: > Again, the base path to create stamp files but can use wildcards for > matching a range of files for clean operations. > diff --git > a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml > b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml > index a84b2bc..aca6741 100644 --- > a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml > +++ > b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml > @@ -3,7 +3,7 @@ [ %poky; ] > > - > + > =20 > Variables Glossary > =20 > @@ -34,29 +34,29 @@ > > > =20 > - > + > =20 > > - A > - B > - C > - D > - E > - F > - G > - H > - > + A > + B > + C > + D > + E > + F > + G > + H > + I > > > - L > - M > + L > + M > > - O > - P > + O > + P > > - R > - S > - T > + R > + S > + T > > > > @@ -65,13 +65,13 @@ > > > =20 > - A > + A > =20 > - id=3D'var-ASSUME_PROVIDED'>ASSUME_PROVIDED > + id=3D'var-bb-ASSUME_PROVIDED'>ASSUME_PROVIDED > > Lists recipe names > - ( linkend=3D'var-PN'>PN > + ( linkend=3D'var-bb-PN'>PN values) BitBake > does not attempt to build. Instead, BitBake assumes these recipes > have already been built. > @@ -91,9 +91,9 @@ > > =20 > =20 > - B > + B > =20 > - B > + B > > > The directory in which BitBake executes functions > @@ -102,7 +102,7 @@ > > > =20 > - id=3D'var-BB_ALLOWED_NETWORKS'>BB_ALLOWED_NETWORKS > + id=3D'var-bb-BB_ALLOWED_NETWORKS'>BB_ALLOWED_NETWORKS > > Specifies a space-delimited list of hosts that > the fetcher @@ -111,7 +111,7 @@ > > > This host list is only used if > - linkend=3D'var-BB_NO_NETWORK'>BB_NO_NETWORK > + linkend=3D'var-bb-BB_NO_NETWORK'>BB_NO_NETWORK > is either not set or set to "0". > > @@ -151,13 +151,13 @@ > > Using BB_ALLOWED_NETWORKS in > conjunction with > - linkend=3D'var-PREMIRRORS'>PREMIRRORS > + linkend=3D'var-bb-PREMIRRORS'>PREMIRRORS is > very useful. Adding the host you want to use to > PREMIRRORS results in the > source code being fetched from an allowed location and avoids raising > an error when a host that is not allowed is in a > - linkend=3D'var-SRC_URI'>SRC_URI > + linkend=3D'var-bb-SRC_URI'>SRC_URI > statement. This is because the fetcher does not attempt to use the > host listed in SRC_URI > after a @@ -167,7 +167,7 @@ > > > =20 > - id=3D'var-BB_CONSOLELOG'>BB_CONSOLELOG > + id=3D'var-bb-BB_CONSOLELOG'>BB_CONSOLELOG > > Specifies the path to a log file into which > BitBake's user @@ -176,7 +176,7 @@ > > > =20 > - id=3D'var-BB_CURRENTTASK'>BB_CURRENTTASK > + id=3D'var-bb-BB_CURRENTTASK'>BB_CURRENTTASK > > Contains the name of the currently running task. > @@ -186,7 +186,7 @@ > > > =20 > - id=3D'var-BB_DANGLINGAPPENDS_WARNONLY'>BB_DANGLINGAPPENDS_WARN= ONLY > + id=3D'var-bb-BB_DANGLINGAPPENDS_WARNONLY'>BB_DANGLINGAPPENDS_W= ARNONLY > > Defines how BitBake handles situations where an > append @@ -208,7 +208,7 @@ > > > =20 > - id=3D'var-BB_DEFAULT_TASK'>BB_DEFAULT_TASK > + id=3D'var-bb-BB_DEFAULT_TASK'>BB_DEFAULT_TASK > > The default task to use when none is specified > (e.g. @@ -219,7 +219,7 @@ > > > =20 > - id=3D'var-BB_DISKMON_DIRS'>BB_DISKMON_DIRS > + id=3D'var-bb-BB_DISKMON_DIRS'>BB_DISKMON_DIRS > > Monitors disk space and available inodes during > the build @@ -245,7 +245,7 @@ > build when a threshold is broken. > Subsequent warnings are issued as > defined by the > - linkend=3D'var-BB_DISKMON_WARNINTERVAL'>BB_DISKMON_WARNINTERVAL > variable, > + linkend=3D'var-bb-BB_DISKMON_WARNINTERVAL'>BB_DISKMON_WARNINTERVAL > variable, which must be defined.=20 > <dir> is: > @@ -275,7 +275,7 @@ > BB_DISKMON_DIRS =3D "ABORT,${TMPDIR},,100K" > > The first example works only if you also set > - the linkend=3D'var-BB_DISKMON_WARNINTERVAL'>BB_DISKMON_WARNINTERVAL= > variable. > + the linkend=3D'var-bb-BB_DISKMON_WARNINTERVAL'>BB_DISKMON_WARNINTER= VAL > variable. This example causes the build system to immediately abort > when either the disk space in ${TMPDIR} drops > below 1 Gbyte or the available free inodes drops below @@ -309,7 > +309,7 @@ > > =20 > - id=3D'var-BB_DISKMON_WARNINTERVAL'>BB_DISKMON_WARNINTERVAL > + id=3D'var-bb-BB_DISKMON_WARNINTERVAL'>BB_DISKMON_WARNINTERVAL<= /glossterm> > > Defines the disk space and free inode warning > intervals. @@ -319,7 +319,7 @@ > If you are going to use the > BB_DISKMON_WARNINTERVAL > variable, you must also use the > - linkend=3D'var-BB_DISKMON_DIRS'>BB_DISKMON_DIRS > variable > + linkend=3D'var-bb-BB_DISKMON_DIRS'>BB_DISKMON_DIRS > variable and define its action as "WARN". During the build, > subsequent warnings are issued each time disk space or number of free > inodes further reduces by @@ -374,7 +374,7 @@ > > > =20 > - id=3D'var-BB_ENV_WHITELIST'>BB_ENV_WHITELIST > + id=3D'var-bb-BB_ENV_WHITELIST'>BB_ENV_WHITELIST > > Specifies the internal whitelist of variables to > allow @@ -382,11 +382,11 @@ > datastore. > If the value of this variable is not specified > (which is the default), the following list is > used: > - linkend=3D'var-BBPATH'>BBPATH, > - linkend=3D'var-BB_PRESERVE_ENV'>BB_PRESERVE_ENV, > - linkend=3D'var-BB_ENV_WHITELIST'>BB_ENV_WHITELIST, > + linkend=3D'var-bb-BBPATH'>BBPATH, > + linkend=3D'var-bb-BB_PRESERVE_ENV'>BB_PRESERVE_ENV, > + linkend=3D'var-bb-BB_ENV_WHITELIST'>BB_ENV_WHITELIST= , > and > - linkend=3D'var-BB_ENV_EXTRAWHITE'>BB_ENV_EXTRAWHITE<= /link>. > + linkend=3D'var-bb-BB_ENV_EXTRAWHITE'>BB_ENV_EXTRAWHITE. > You must set this variable in the external environment > in order for it to work. > @@ -395,7 +395,7 @@ > > > =20 > - id=3D'var-BB_ENV_EXTRAWHITE'>BB_ENV_EXTRAWHITE > + id=3D'var-bb-BB_ENV_EXTRAWHITE'>BB_ENV_EXTRAWHITE > > Specifies an additional set of variables to > allow through @@ -403,7 +403,7 @@ > datastore. > This list of variables are on top of the > internal list set in > - linkend=3D'var-BB_ENV_WHITELIST'>BB_ENV_WHITELIST. > + linkend=3D'var-bb-BB_ENV_WHITELIST'>BB_ENV_WHITELIST= . > You must set this variable in the external > environment in order for it to work. > @@ -412,22 +412,22 @@ > > > =20 > - id=3D'var-BB_FETCH_PREMIRRORONLY'>BB_FETCH_PREMIRRORONLY > + id=3D'var-bb-BB_FETCH_PREMIRRORONLY'>BB_FETCH_PREMIRRORONLY > > When set to "1", causes BitBake's fetcher module > to only search > - linkend=3D'var-PREMIRRORS'>PREMIRRORS > + linkend=3D'var-bb-PREMIRRORS'>PREMIRRORS > for files. BitBake will not search the main > - linkend=3D'var-SRC_URI'>SRC_URI > + linkend=3D'var-bb-SRC_URI'>SRC_URI or > - linkend=3D'var-MIRRORS'>MIRRORS. > + linkend=3D'var-bb-MIRRORS'>MIRRORS. > > > =20 > - id=3D'var-BB_FILENAME'>BB_FILENAME > + id=3D'var-bb-BB_FILENAME'>BB_FILENAME > > Contains the filename of the recipe that owns > the currently @@ -440,12 +440,12 @@ > > > =20 > - id=3D'var-BB_GENERATE_MIRROR_TARBALLS'>BB_GENERATE_MIRROR_TARB= ALLS > + id=3D'var-bb-BB_GENERATE_MIRROR_TARBALLS'>BB_GENERATE_MIRROR_T= ARBALLS > > Causes tarballs of the Git repositories, > including the Git metadata, to be placed in the > - linkend=3D'var-DL_DIR'>DL_DIR > + linkend=3D'var-bb-DL_DIR'>DL_DIR directory. > Anyone wishing to create a source mirror would > want to enable this variable. > @@ -461,7 +461,7 @@ > > > =20 > - id=3D'var-BB_HASHCONFIG_WHITELIST'>BB_HASHCONFIG_WHITELIST > + id=3D'var-bb-BB_HASHCONFIG_WHITELIST'>BB_HASHCONFIG_WHITELIST<= /glossterm> > > Lists variables that are excluded from base > configuration @@ -485,7 +485,7 @@ > > > =20 > - id=3D'var-BB_HASHBASE_WHITELIST'>BB_HASHBASE_WHITELIST > + id=3D'var-bb-BB_HASHBASE_WHITELIST'>BB_HASHBASE_WHITELIST > > Lists variables that are excluded from checksum > and @@ -500,7 +500,7 @@ > > > =20 > - id=3D'var-BB_HASHCHECK_FUNCTION'>BB_HASHCHECK_FUNCTION > + id=3D'var-bb-BB_HASHCHECK_FUNCTION'>BB_HASHCHECK_FUNCTION > > Specifies the name of the function to call > during the @@ -524,7 +524,7 @@ > > > =20 > - id=3D'var-BB_INVALIDCONF'>BB_INVALIDCONF > + id=3D'var-bb-BB_INVALIDCONF'>BB_INVALIDCONF > > Used in combination with the > @@ -539,11 +539,11 @@ > > > =20 > - id=3D'var-BB_LOGFMT'>BB_LOGFMT > + id=3D'var-bb-BB_LOGFMT'>BB_LOGFMT > > Specifies the name of the log files saved into > - ${ linkend=3D'var-T'>T}. > + ${ linkend=3D'var-bb-T'>T}. > By default, the BB_LOGFMT variable is undefined > and the log file names get created using the following form: > @@ -556,7 +556,7 @@ > > > =20 > - id=3D'var-BB_NICE_LEVEL'>BB_NICE_LEVEL > + id=3D'var-bb-BB_NICE_LEVEL'>BB_NICE_LEVEL > > Allows BitBake to run at a specific priority > @@ -564,13 +564,13 @@ > System permissions usually mean that BitBake can > reduce its priority but not raise it again. > See > - linkend=3D'var-BB_TASK_NICE_LEVEL'>BB_TASK_NICE_LEVEL > + linkend=3D'var-bb-BB_TASK_NICE_LEVEL'>BB_TASK_NICE_LEVEL > for additional information. > > > =20 > - id=3D'var-BB_NO_NETWORK'>BB_NO_NETWORK > + id=3D'var-bb-BB_NO_NETWORK'>BB_NO_NETWORK > > Disables network access in the BitBake fetcher > modules. @@ -587,7 +587,7 @@ > > > =20 > - id=3D'var-BB_NUMBER_THREADS'>BB_NUMBER_THREADS > + id=3D'var-bb-BB_NUMBER_THREADS'>BB_NUMBER_THREADS > > The maximum number of tasks BitBake should run > in parallel @@ -599,7 +599,7 @@ > > > =20 > - id=3D'var-BB_NUMBER_PARSE_THREADS'>BB_NUMBER_PARSE_THREADS > + id=3D'var-bb-BB_NUMBER_PARSE_THREADS'>BB_NUMBER_PARSE_THREADS<= /glossterm> > > Sets the number of threads BitBake uses when > parsing. @@ -609,7 +609,7 @@ > > > =20 > - id=3D'var-BB_ORIGENV'>BB_ORIGENV > + id=3D'var-bb-BB_ORIGENV'>BB_ORIGENV > > Contains a copy of the original external > environment in @@ -625,7 +625,7 @@ > > > =20 > - id=3D'var-BB_PRESERVE_ENV'>BB_PRESERVE_ENV > + id=3D'var-bb-BB_PRESERVE_ENV'>BB_PRESERVE_ENV > > Disables whitelisting and instead allows all > variables @@ -639,12 +639,12 @@ > > > =20 > - id=3D'var-BB_RUNFMT'>BB_RUNFMT > + id=3D'var-bb-BB_RUNFMT'>BB_RUNFMT > > Specifies the name of the executable script files > (i.e. run files) saved into > - ${ linkend=3D'var-T'>T}. > + ${ linkend=3D'var-bb-T'>T}. > By default, the BB_RUNFMT variable is undefined > and the run file names get created using the following form: > @@ -657,7 +657,7 @@ > > > =20 > - id=3D'var-BB_RUNTASK'>BB_RUNTASK > + id=3D'var-bb-BB_RUNTASK'>BB_RUNTASK > > Contains the name of the currently executing > task. @@ -669,7 +669,7 @@ > > > =20 > - id=3D'var-BB_SCHEDULER'>BB_SCHEDULER > + id=3D'var-bb-BB_SCHEDULER'>BB_SCHEDULER > > Selects the name of the scheduler to use for the > @@ -695,7 +695,7 @@ > > > =20 > - id=3D'var-BB_SCHEDULERS'>BB_SCHEDULERS > + id=3D'var-bb-BB_SCHEDULERS'>BB_SCHEDULERS > > Defines custom schedulers to import. > @@ -705,13 +705,13 @@ > =20 > > For information how to select a scheduler, see > the > - linkend=3D'var-BB_SCHEDULER'>BB_SCHEDULER > + linkend=3D'var-bb-BB_SCHEDULER'>BB_SCHEDULER > variable. > > > =20 > - id=3D'var-BB_SETSCENE_DEPVALID'>BB_SETSCENE_DEPVALID > + id=3D'var-bb-BB_SETSCENE_DEPVALID'>BB_SETSCENE_DEPVALID > > Specifies a function BitBake calls that > determines @@ -731,7 +731,7 @@ > > > =20 > - id=3D'var-BB_SETSCENE_VERIFY_FUNCTION2'>BB_SETSCENE_VERIFY_FUN= CTION2 > + id=3D'var-bb-BB_SETSCENE_VERIFY_FUNCTION2'>BB_SETSCENE_VERIFY_= FUNCTION2 > > Specifies a function to call that verifies the > list of @@ -752,7 +752,7 @@ > > > =20 > - id=3D'var-BB_SIGNATURE_EXCLUDE_FLAGS'>BB_SIGNATURE_EXCLUDE_FLA= GS > + id=3D'var-bb-BB_SIGNATURE_EXCLUDE_FLAGS'>BB_SIGNATURE_EXCLUDE_= FLAGS > > Lists variable flags (varflags) > @@ -771,7 +771,7 @@ > > > =20 > - id=3D'var-BB_SIGNATURE_HANDLER'>BB_SIGNATURE_HANDLER > + id=3D'var-bb-BB_SIGNATURE_HANDLER'>BB_SIGNATURE_HANDLER > > Defines the name of the signature handler > BitBake uses. @@ -790,7 +790,7 @@ > > > =20 > - id=3D'var-BB_SRCREV_POLICY'>BB_SRCREV_POLICY > + id=3D'var-bb-BB_SRCREV_POLICY'>BB_SRCREV_POLICY > > Defines the behavior of the fetcher when it > interacts with @@ -817,7 +817,7 @@ > > > =20 > - id=3D'var-BB_STAMP_POLICY'>BB_STAMP_POLICY > + id=3D'var-bb-BB_STAMP_POLICY'>BB_STAMP_POLICY > > Defines the mode used for how timestamps of > stamp files @@ -836,7 +836,7 @@ > whitelist > - Identical to "full" mode except timestamp > comparisons are made for recipes listed > in the > - linkend=3D'var-BB_STAMP_WHITELIST'>BB_STAMP_WHITELIST > + linkend=3D'var-bb-BB_STAMP_WHITELIST'>BB_STAMP_WHITELIST > variable. > > @@ -848,19 +848,19 @@ > > > =20 > - id=3D'var-BB_STAMP_WHITELIST'>BB_STAMP_WHITELIST > + id=3D'var-bb-BB_STAMP_WHITELIST'>BB_STAMP_WHITELIST > > Lists files whose stamp file timestamps are > compared when the stamp policy mode is set to "whitelist". > For information on stamp policies, see the > - linkend=3D'var-BB_STAMP_POLICY'>BB_STAMP_POLICY > + linkend=3D'var-bb-BB_STAMP_POLICY'>BB_STAMP_POLICY > variable. > > > =20 > - id=3D'var-BB_STRICT_CHECKSUM'>BB_STRICT_CHECKSUM > + id=3D'var-bb-BB_STRICT_CHECKSUM'>BB_STRICT_CHECKSUM > > Sets a more strict checksum mechanism for > non-local URLs. @@ -871,7 +871,7 @@ > > > =20 > - id=3D'var-BB_TASK_IONICE_LEVEL'>BB_TASK_IONICE_LEVEL > + id=3D'var-bb-BB_TASK_IONICE_LEVEL'>BB_TASK_IONICE_LEVEL > > Allows adjustment of a task's Input/Output > priority. @@ -882,7 +882,7 @@ > variable to adjust the I/O priority of these > tasks. > This variable works similarly to the > - linkend=3D'var-BB_TASK_NICE_LEVEL'>BB_TASK_NICE_LEVEL > + linkend=3D'var-bb-BB_TASK_NICE_LEVEL'>BB_TASK_NICE_LEVEL > variable except with a task's I/O priorities. > > @@ -921,7 +921,7 @@ > > > =20 > - id=3D'var-BB_TASK_NICE_LEVEL'>BB_TASK_NICE_LEVEL > + id=3D'var-bb-BB_TASK_NICE_LEVEL'>BB_TASK_NICE_LEVEL > > Allows specific tasks to change their priority > @@ -940,7 +940,7 @@ > > > =20 > - id=3D'var-BB_TASKHASH'>BB_TASKHASH > + id=3D'var-bb-BB_TASKHASH'>BB_TASKHASH > > Within an executing task, this variable holds > the hash @@ -950,7 +950,7 @@ > > > =20 > - id=3D'var-BB_VERBOSE_LOGS'>BB_VERBOSE_LOGS > + id=3D'var-bb-BB_VERBOSE_LOGS'>BB_VERBOSE_LOGS > > Controls how verbose BitBake is during builds. > @@ -960,7 +960,7 @@ > > > =20 > - id=3D'var-BB_WORKERCONTEXT'>BB_WORKERCONTEXT > + id=3D'var-bb-BB_WORKERCONTEXT'>BB_WORKERCONTEXT > > Specifies if the current context is executing a > task. @@ -973,7 +973,7 @@ > > =20 > =20 > - id=3D'var-BBCLASSEXTEND'>BBCLASSEXTEND > + id=3D'var-bb-BBCLASSEXTEND'>BBCLASSEXTEND > > Allows you to extend a recipe so that it builds > variants @@ -1009,7 +1009,7 @@ > _class-native. > For example, to generate a native version of > a recipe, a > - linkend=3D'var-DEPENDS'>DEPENDS > + linkend=3D'var-bb-DEPENDS'>DEPENDS on "foo" > is rewritten to a DEPENDS on "foo-native". > > @@ -1028,7 +1028,7 @@ > > > =20 > - BBDEBUG > + id=3D'var-bb-BBDEBUG'>BBDEBUG > > Sets the BitBake debug output level to a > specific value @@ -1042,7 +1042,7 @@ > > > =20 > - id=3D'var-BBFILE_COLLECTIONS'>BBFILE_COLLECTIONS > + id=3D'var-bb-BBFILE_COLLECTIONS'>BBFILE_COLLECTIONS > Lists the names of configured layers. > These names are used to find the other > BBFILE_* @@ -1053,10 +1053,10 @@ > > > =20 > - id=3D'var-BBFILE_PATTERN'>BBFILE_PATTERN > + id=3D'var-bb-BBFILE_PATTERN'>BBFILE_PATTERN > Variable that expands to match files from > - linkend=3D'var-BBFILES'>BBFILES > + linkend=3D'var-bb-BBFILES'>BBFILES in a > particular layer. This variable is used in the > conf/layer.conf file and must be suffixed with > the name of the specific layer (e.g. @@ -1064,7 +1064,7 @@ > > > =20 > - id=3D'var-BBFILE_PRIORITY'>BBFILE_PRIORITY > + id=3D'var-bb-BBFILE_PRIORITY'>BBFILE_PRIORITY > Assigns the priority for recipe files in each > layer. This variable is useful in situations where the > same recipe appears in @@ -1074,7 +1074,7 @@ > letting you control the precedence for the > multiple layers. The precedence established through this variable > stands regardless of a recipe's version > - ( linkend=3D'var-PV'>PV variable). > + ( linkend=3D'var-bb-PV'>PV variable). For > example, a layer that has a recipe with a higher > PV value but for which the > BBFILE_PRIORITY is set to have a lower > precedence still has a lower precedence. @@ -1083,7 +1083,7 @@ > For example, the value 6 has a higher precedence than the value 5. If > not specified, the BBFILE_PRIORITY variable is > set based on layer dependencies (see the > - linkend=3D'var-LAYERDEPENDS'>LAYERDEPENDS variable for > + linkend=3D'var-bb-LAYERDEPENDS'>LAYERDEPENDS variable > for more information. The default priority, if unspecified > for a layer with no dependencies, is the lowest > defined priority + 1 @@ -1095,7 +1095,7 @@ > > > =20 > - BBFILES > + id=3D'var-bb-BBFILES'>BBFILES > > A space-separated list of recipe files BitBake > uses to @@ -1113,7 +1113,7 @@ > > > =20 > - id=3D'var-BBINCLUDED'>BBINCLUDED > + id=3D'var-bb-BBINCLUDED'>BBINCLUDED > > Contains a space-separated list of all of all > files that @@ -1123,7 +1123,7 @@ > > > =20 > - id=3D'var-BBINCLUDELOGS'>BBINCLUDELOGS > + id=3D'var-bb-BBINCLUDELOGS'>BBINCLUDELOGS > > If set to a value, enables printing the task log > when @@ -1132,11 +1132,11 @@ > > > =20 > - id=3D'var-BBINCLUDELOGS_LINES'>BBINCLUDELOGS_LINES > + id=3D'var-bb-BBINCLUDELOGS_LINES'>BBINCLUDELOGS_LINES > > If > - linkend=3D'var-BBINCLUDELOGS'>BBINCLUDELOGS > + linkend=3D'var-bb-BBINCLUDELOGS'>BBINCLUDELOGS > is set, specifies the maximum number of lines from the task log file > to print when reporting a failed task. If you do not set > BBINCLUDELOGS_LINES, @@ -1145,7 +1145,7 @@ > > > =20 > - BBLAYERS > + id=3D'var-bb-BBLAYERS'>BBLAYERS > Lists the layers to enable during the build. > This variable is defined in the > bblayers.conf configuration @@ -1166,7 +1166,7 @@ > > > =20 > - id=3D'var-BBLAYERS_FETCH_DIR'>BBLAYERS_FETCH_DIR > + id=3D'var-bb-BBLAYERS_FETCH_DIR'>BBLAYERS_FETCH_DIR > > Sets the base location where layers are stored. > @@ -1178,7 +1178,7 @@ > > > =20 > - BBMASK > + BBMASK > > > Prevents BitBake from processing recipes and > recipe @@ -1236,7 +1236,7 @@ > > > =20 > - id=3D'var-BBMULTICONFIG'>BBMULTICONFIG > + id=3D'var-bb-BBMULTICONFIG'>BBMULTICONFIG > BBMULTICONFIG[doc] =3D "Enables BitBake to perform > multiple configuration builds and lists each separate configuration > (multiconfig)." @@ -1275,7 +1275,7 @@ > > > =20 > - BBPATH > + BBPATH > > > Used by BitBake to locate class > @@ -1302,7 +1302,7 @@ > > > =20 > - BBSERVER > + id=3D'var-bb-BBSERVER'>BBSERVER > > Points to the server that runs memory-resident > BitBake. @@ -1312,7 +1312,7 @@ > > > =20 > - id=3D'var-BBTARGETS'>BBTARGETS > + id=3D'var-bb-BBTARGETS'>BBTARGETS > > Allows you to use a configuration file to add to > the list @@ -1321,14 +1321,14 @@ > > > =20 > - id=3D'var-BBVERSIONS'>BBVERSIONS > + id=3D'var-bb-BBVERSIONS'>BBVERSIONS > > Allows a single recipe to build multiple > versions of a project from a single recipe file. > You also able to specify conditional metadata > using the > - linkend=3D'var-OVERRIDES'>OVERRIDES > + linkend=3D'var-bb-OVERRIDES'>OVERRIDES > mechanism for a single version or for an optionally named range of > versions. > @@ -1342,7 +1342,7 @@ > > > =20 > - id=3D'var-BITBAKE_UI'>BITBAKE_UI > + id=3D'var-bb-BITBAKE_UI'>BITBAKE_UI > > Used to specify the UI module to use when > running BitBake. @@ -1356,7 +1356,7 @@ > > > =20 > - id=3D'var-BUILDNAME'>BUILDNAME > + id=3D'var-bb-BUILDNAME'>BUILDNAME > > A name assigned to the build. > @@ -1366,7 +1366,7 @@ > > > =20 > - BZRDIR > + BZRDIR > > > The directory in which files checked out of a > Bazaar @@ -1377,9 +1377,9 @@ > =20 > > =20 > - C > + C > =20 > - CACHE > + CACHE > > > Specifies the directory BitBake uses to store a > cache @@ -1389,7 +1389,7 @@ > > > =20 > - CVSDIR > + CVSDIR > > > The directory in which files checked out under > the @@ -1400,9 +1400,9 @@ > =20 > > =20 > - D > + D > =20 > - id=3D'var-DEFAULT_PREFERENCE'>DEFAULT_PREFERENCE > + id=3D'var-bb-DEFAULT_PREFERENCE'>DEFAULT_PREFERENCE > > Specifies a weak bias for recipe selection > priority. @@ -1413,20 +1413,20 @@ > piece of software. > Using the variable in this way causes the stable > version of the recipe to build by default in the absence of > - linkend=3D'var-PREFERRED_VERSION'>PREFERRED_VERSION > + linkend=3D'var-bb-PREFERRED_VERSION'>PREFERRED_VERSION > being used to build the development version. > > The bias provided by > DEFAULT_PREFERENCE is weak and is overridden by > - linkend=3D'var-BBFILE_PRIORITY'>BBFILE_PRIORITY > + linkend=3D'var-bb-BBFILE_PRIORITY'>BBFILE_PRIORITY if > that variable is different between two layers that contain different > versions of the same recipe. > > > =20 > - DEPENDS > + id=3D'var-bb-DEPENDS'>DEPENDS > > Lists a recipe's build-time dependencies > @@ -1451,13 +1451,13 @@ > =20 > > For information on runtime dependencies, see the > - linkend=3D'var-RDEPENDS'>RDEPENDS > + linkend=3D'var-bb-RDEPENDS'>RDEPENDS > variable. > > > =20 > - id=3D'var-DESCRIPTION'>DESCRIPTION > + id=3D'var-bb-DESCRIPTION'>DESCRIPTION > > A long description for the recipe. > @@ -1465,7 +1465,7 @@ > > > =20 > - DL_DIR > + DL_DIR > > > The central download directory used by the build > process to @@ -1474,7 +1474,7 @@ > suitable for mirroring for everything except Git > repositories. > If you want tarballs of Git repositories, use the > - linkend=3D'var-BB_GENERATE_MIRROR_TARBALLS'>BB_GENERATE_MIRROR_= TARBALLS > + linkend=3D'var-bb-BB_GENERATE_MIRROR_TARBALLS'>BB_GENERATE_MIRR= OR_TARBALLS > variable. > > @@ -1482,9 +1482,9 @@ > > > =20 > - E > + E > =20 > - id=3D'var-EXCLUDE_FROM_WORLD'>EXCLUDE_FROM_WORLD > + id=3D'var-bb-EXCLUDE_FROM_WORLD'>EXCLUDE_FROM_WORLD > > Directs BitBake to exclude a recipe from world > builds (i.e. @@ -1512,9 +1512,9 @@ > =20 > > =20 > - F > + F > =20 > - FAKEROOT > + id=3D'var-bb-FAKEROOT'>FAKEROOT > > Contains the command to use when running a > shell script @@ -1527,19 +1527,19 @@ > > > =20 > - id=3D'var-FAKEROOTBASEENV'>FAKEROOTBASEENV > + id=3D'var-bb-FAKEROOTBASEENV'>FAKEROOTBASEENV > > Lists environment variables to set when > executing the command defined by > - linkend=3D'var-FAKEROOTCMD'>FAKEROOTCMD > + linkend=3D'var-bb-FAKEROOTCMD'>FAKEROOTCMD > that starts the bitbake-worker process in the fakeroot environment. > > > > =20 > - id=3D'var-FAKEROOTCMD'>FAKEROOTCMD > + id=3D'var-bb-FAKEROOTCMD'>FAKEROOTCMD > > Contains the command that starts the > bitbake-worker @@ -1548,7 +1548,7 @@ > > > =20 > - id=3D'var-FAKEROOTDIRS'>FAKEROOTDIRS > + id=3D'var-bb-FAKEROOTDIRS'>FAKEROOTDIRS > > Lists directories to create before running a > task in @@ -1557,33 +1557,33 @@ > > > =20 > - id=3D'var-FAKEROOTENV'>FAKEROOTENV > + id=3D'var-bb-FAKEROOTENV'>FAKEROOTENV > > Lists environment variables to set when running > a task in the fakeroot environment. > For additional information on environment > variables and the fakeroot environment, see the > - linkend=3D'var-FAKEROOTBASEENV'>FAKEROOTBASEENV > + linkend=3D'var-bb-FAKEROOTBASEENV'>FAKEROOTBASEENV > variable. > > > =20 > - id=3D'var-FAKEROOTNOENV'>FAKEROOTNOENV > + id=3D'var-bb-FAKEROOTNOENV'>FAKEROOTNOENV > > Lists environment variables to set when running > a task that is not in the fakeroot environment. > For additional information on environment > variables and the fakeroot environment, see the > - linkend=3D'var-FAKEROOTENV'>FAKEROOTENV > + linkend=3D'var-bb-FAKEROOTENV'>FAKEROOTENV > variable. > > > =20 > - FETCHCMD > + id=3D'var-bb-FETCHCMD'>FETCHCMD > > Defines the command the BitBake fetcher module > @@ -1595,7 +1595,7 @@ > > > =20 > - FILE > + FILE > > > Points at the current file. > @@ -1607,7 +1607,7 @@ > > > =20 > - id=3D'var-FILESPATH'>FILESPATH > + id=3D'var-bb-FILESPATH'>FILESPATH > > Specifies directories BitBake uses when > searching for @@ -1625,9 +1625,9 @@ > > =20 > =20 > - G > + G > =20 > - GITDIR > + GITDIR > > > The directory in which a local copy of a Git > repository @@ -1639,9 +1639,9 @@ > > =20 > =20 > - H > + H > =20 > - HGDIR > + HGDIR > > > The directory in which files checked out of a > Mercurial @@ -1650,7 +1650,7 @@ > > > =20 > - HOMEPAGE > + id=3D'var-bb-HOMEPAGE'>HOMEPAGE > Website where more information about the > software the recipe is building can be found. > @@ -1659,9 +1659,9 @@ > =20 > > =20 > - I > + I > =20 > - INHERIT > + id=3D'var-bb-INHERIT'>INHERIT > > Causes the named class or classes to be > inherited globally. @@ -1691,15 +1691,15 @@ > > --> =20 > =20 > - L > + L > =20 > - id=3D'var-LAYERDEPENDS'>LAYERDEPENDS > + id=3D'var-bb-LAYERDEPENDS'>LAYERDEPENDS > Lists the layers, separated by spaces, upon which > this recipe depends. Optionally, you can specify a specific layer > version for a dependency by adding it to the end of the layer name > with a colon, (e.g. "anotherlayer:3" to be compared against > - linkend=3D'var-LAYERVERSION'>LAYERVERSION_anotherlayer > + linkend=3D'var-bb-LAYERVERSION'>LAYERVERSION<= filename>_anotherlayer > in this case). BitBake produces an error if any dependency is missing > or the version numbers do not match exactly (if specified). > @@ -1710,7 +1710,7 @@ > > > =20 > - LAYERDIR > + id=3D'var-bb-LAYERDIR'>LAYERDIR > When used inside the > layer.conf configuration file, this variable > provides the path of the current layer. @@ -1719,22 +1719,22 @@ > > > =20 > - id=3D'var-LAYERDIR_RE'>LAYERDIR_RE > + id=3D'var-bb-LAYERDIR_RE'>LAYERDIR_RE > When used inside the > layer.conf configuration file, this variable > provides the path of the current layer, escaped for use in a regular > expression > - ( linkend=3D'var-BBFILE_PATTERN'>BBFILE_PATTERN= ). > + ( linkend=3D'var-bb-BBFILE_PATTERN'>BBFILE_PATTERN). > This variable is not available outside of > layer.conf and references are expanded > immediately when parsing of the file completes. > =20 > - id=3D'var-LAYERVERSION'>LAYERVERSION > + id=3D'var-bb-LAYERVERSION'>LAYERVERSION > Optionally specifies the version of a layer as a > single number. You can use this variable within > - linkend=3D'var-LAYERDEPENDS'>LAYERDEPENDS > + linkend=3D'var-bb-LAYERDEPENDS'>LAYERDEPENDS > for another layer in order to depend on a specific version of the > layer. > @@ -1744,7 +1744,7 @@ > > > =20 > - LICENSE > + id=3D'var-bb-LICENSE'>LICENSE > > The list of source licenses for the recipe. > @@ -1754,9 +1754,9 @@ > =20 > > =20 > - M > + M > =20 > - MIRRORS > + id=3D'var-bb-MIRRORS'>MIRRORS > > Specifies additional paths from which BitBake > gets source code. @@ -1764,14 +1764,14 @@ > tries the local download directory. > If that location fails, the build system tries > locations defined by > - linkend=3D'var-PREMIRRORS'>PREMIRRORS, > + linkend=3D'var-bb-PREMIRRORS'>PREMIRRORS, > the upstream source, and then locations specified by > MIRRORS in that order. > > > =20 > - id=3D'var-MULTI_PROVIDER_WHITELIST'>MULTI_PROVIDER_WHITELIST > + id=3D'var-bb-MULTI_PROVIDER_WHITELIST'>MULTI_PROVIDER_WHITELIS= T > > Allows you to suppress BitBake warnings caused > when @@ -1804,9 +1804,9 @@ > > --> =20 > =20 > - O > + O > =20 > - id=3D'var-OVERRIDES'>OVERRIDES > + id=3D'var-bb-OVERRIDES'>OVERRIDES > > BitBake uses OVERRIDES to > control @@ -1829,9 +1829,9 @@ > > > =20 > - P > + P > =20 > - P4DIR > + P4DIR > > > The directory in which a local copy of a > Perforce depot @@ -1840,14 +1840,14 @@ > > > =20 > - PACKAGES > + id=3D'var-bb-PACKAGES'>PACKAGES > The list of packages the recipe creates. > > > > =20 > - id=3D'var-PACKAGES_DYNAMIC'>PACKAGES_DYNAMIC > + id=3D'var-bb-PACKAGES_DYNAMIC'>PACKAGES_DYNAMIC > > A promise that your recipe satisfies runtime > dependencies @@ -1856,7 +1856,7 @@ > does not actually satisfy the dependencies, it > only states that they should be satisfied. > For example, if a hard, runtime dependency > - ( linkend=3D'var-RDEPENDS'>RDEPENDS) > + ( linkend=3D'var-bb-RDEPENDS'>RDEPENDS) of > another package is satisfied during the build through the > PACKAGES_DYNAMIC variable, but a package with > the module name is never actually @@ -1865,7 +1865,7 @@ > > > =20 > - PE > + PE > > > The epoch of the recipe. > @@ -1877,7 +1877,7 @@ > > > =20 > - id=3D'var-PERSISTENT_DIR'>PERSISTENT_DIR > + id=3D'var-bb-PERSISTENT_DIR'>PERSISTENT_DIR > > Specifies the directory BitBake uses to store > data that @@ -1889,7 +1889,7 @@ > > > =20 > - PF > + PF > > > Specifies the recipe or package name and > includes all version and revision @@ -1899,27 +1899,27 @@ > > > =20 > - PN > + PN > > The recipe name. > > > =20 > - PR > + PR > > The revision of the recipe. > > > > =20 > - id=3D'var-PREFERRED_PROVIDER'>PREFERRED_PROVIDER > + id=3D'var-bb-PREFERRED_PROVIDER'>PREFERRED_PROVIDER > > Determines which recipe should be given > preference when multiple recipes provide the same item. > You should always suffix the variable with the > name of the provided item, and you should set it to the > - linkend=3D'var-PN'>PN > + linkend=3D'var-bb-PN'>PN of the recipe to > which you want to give precedence. Some examples: > > @@ -1931,14 +1931,14 @@ > > > =20 > - id=3D'var-PREFERRED_PROVIDERS'>PREFERRED_PROVIDERS > + id=3D'var-bb-PREFERRED_PROVIDERS'>PREFERRED_PROVIDERS > > Determines which recipe should be given > preference for cases where multiple recipes provide the same item. > Functionally, > PREFERRED_PROVIDERS is > identical to > - linkend=3D'var-PREFERRED_PROVIDER'>PREFERRED_PROVIDER. > + linkend=3D'var-bb-PREFERRED_PROVIDER'>PREFERRED_PROVIDER. > However, the PREFERRED_PROVIDERS variable lets > you define preferences for multiple situations using the following > form: @@ -1954,15 +1954,15 @@ > > > =20 > - id=3D'var-PREFERRED_VERSION'>PREFERRED_VERSION > + id=3D'var-bb-PREFERRED_VERSION'>PREFERRED_VERSION > > If there are multiple versions of recipes > available, this variable determines which recipe should be given > preference. You must always suffix the variable with the > - linkend=3D'var-PN'>PN > + linkend=3D'var-bb-PN'>PN you want to > select, and you should set > - linkend=3D'var-PV'>PV > + linkend=3D'var-bb-PV'>PV accordingly for > precedence. > =20 > @@ -1989,7 +1989,7 @@ > > > =20 > - id=3D'var-PREMIRRORS'>PREMIRRORS > + id=3D'var-bb-PREMIRRORS'>PREMIRRORS > > Specifies additional paths from which BitBake > gets source code. @@ -1998,7 +1998,7 @@ > If that location fails, the build system tries > locations defined by PREMIRRORS, the upstream > source, and then locations specified by > - linkend=3D'var-MIRRORS'>MIRRORS > + linkend=3D'var-bb-MIRRORS'>MIRRORS in that > order. > =20 > @@ -2022,20 +2022,20 @@ > > > =20 > - PROVIDES > + id=3D'var-bb-PROVIDES'>PROVIDES > > A list of aliases by which a particular recipe > can be known. > By default, a recipe's own > - linkend=3D'var-PN'>PN > + linkend=3D'var-bb-PN'>PN is implicitly already in its > PROVIDES list. > If a recipe uses PROVIDES, > the additional aliases are synonyms for the recipe and can > be useful satisfying dependencies of other > recipes during the build as specified by > - linkend=3D'var-DEPENDS'>DEPENDS. > + linkend=3D'var-bb-DEPENDS'>DEPENDS. > =20 > > @@ -2059,7 +2059,7 @@ > virtual target in PROVIDES. > Recipes that depend on the functionality in > question can include the virtual target in > - linkend=3D'var-DEPENDS'>DEPENDS > + linkend=3D'var-bb-DEPENDS'>DEPENDS to leave > the choice of provider open. > =20 > @@ -2072,11 +2072,11 @@ > > > =20 > - id=3D'var-PRSERV_HOST'>PRSERV_HOST > + id=3D'var-bb-PRSERV_HOST'>PRSERV_HOST > > The network based > - linkend=3D'var-PR'>PR > + linkend=3D'var-bb-PR'>PR service host and > port. > =20 > @@ -2094,7 +2094,7 @@ > > > =20 > - PV > + PV > > The version of the recipe. > > @@ -2108,9 +2108,9 @@ > > --> =20 > =20 > - R > + R > =20 > - RDEPENDS > + id=3D'var-bb-RDEPENDS'>RDEPENDS > > Lists a package's runtime dependencies (i.e. > other packages) @@ -2165,13 +2165,13 @@ > =20 > > For information on build-time dependencies, see > the > - linkend=3D'var-DEPENDS'>DEPENDS > + linkend=3D'var-bb-DEPENDS'>DEPENDS variable. > > > > =20 > - REPODIR > + id=3D'var-bb-REPODIR'>REPODIR > > The directory in which a local copy of a > @@ -2181,14 +2181,14 @@ > > > =20 > - id=3D'var-RPROVIDES'>RPROVIDES > + id=3D'var-bb-RPROVIDES'>RPROVIDES > > A list of package name aliases that a package > also provides. These aliases are useful for satisfying runtime > dependencies of other packages both during the build and on the target > (as specified by > - linkend=3D'var-RDEPENDS'>RDEPENDS). > + linkend=3D'var-bb-RDEPENDS'>RDEPENDS). > > As with all package-controlling variables, you > must always @@ -2201,7 +2201,7 @@ > > > =20 > - id=3D'var-RRECOMMENDS'>RRECOMMENDS > + id=3D'var-bb-RRECOMMENDS'>RRECOMMENDS > > A list of packages that extends the usability of > a package @@ -2210,7 +2210,7 @@ > packages in order to successfully build, but > needs them for the extended usability. > To specify runtime dependencies for packages, > see the > - linkend=3D'var-RDEPENDS'>RDEPENDS > + linkend=3D'var-bb-RDEPENDS'>RDEPENDS variable. > > =20 > @@ -2243,15 +2243,15 @@ > =20 > > =20 > - S > + S > =20 > - SECTION > + id=3D'var-bb-SECTION'>SECTION > The section in which packages should be > categorized. > > =20 > - SRC_URI > + id=3D'var-bb-SRC_URI'>SRC_URI > > The list of source files - local or remote. > @@ -2272,7 +2272,7 @@ > the metadata, > from the local machine. > The path is relative to the > - linkend=3D'var-FILESPATH'>FILESPATH > + linkend=3D'var-bb-FILESPATH'>FILESPATH > variable. > bzr:// - > Fetches files from a Bazaar revision control > repository. @@ -2322,7 +2322,7 @@ > > =20 > - SRCDATE > + id=3D'var-bb-SRCDATE'>SRCDATE > > The date of the source code used to build the > package. @@ -2331,7 +2331,7 @@ > > > =20 > - SRCREV > + SRCREV > > > The revision of the source code used to build > the package. @@ -2344,13 +2344,13 @@ > > > =20 > - id=3D'var-SRCREV_FORMAT'>SRCREV_FORMAT > + id=3D'var-bb-SRCREV_FORMAT'>SRCREV_FORMAT > > Helps construct valid > - linkend=3D'var-SRCREV'>SRCREV > + linkend=3D'var-bb-SRCREV'>SRCREV values > when multiple source controlled URLs are used in > - linkend=3D'var-SRC_URI'>SRC_URI. > + linkend=3D'var-bb-SRC_URI'>SRC_URI. > =20 > > @@ -2371,7 +2371,7 @@ > > > =20 > - STAMP > + STAMP > > > Specifies the base path used to create recipe > stamp files. @@ -2381,12 +2381,12 @@ > > > =20 > - id=3D'var-STAMPCLEAN'>STAMPCLEAN > + id=3D'var-bb-STAMPCLEAN'>STAMPCLEAN > > Specifies the base path used to create recipe > stamp files. Unlike the > - linkend=3D'var-STAMP'>STAMP > + linkend=3D'var-bb-STAMP'>STAMP variable, > STAMPCLEAN can contain wildcards to match the > range of files a clean operation should remove. > @@ -2396,7 +2396,7 @@ > > > =20 > - SUMMARY > + id=3D'var-bb-SUMMARY'>SUMMARY > > A short summary for the recipe, which is 72 > characters or less. @@ -2404,7 +2404,7 @@ > > > =20 > - SVNDIR > + SVNDIR > > > The directory in which files checked out of a > Subversion @@ -2415,9 +2415,9 @@ > =20 > > =20 > - T > + T > =20 > - T > + T > > Points to a directory were BitBake places > temporary files, which consist mostly of task > logs and @@ -2426,7 +2426,7 @@ > > > =20 > - TOPDIR > + TOPDIR > > > Points to the build directory. > diff --git a/bitbake/doc/poky.ent b/bitbake/doc/poky.ent > index c032e14..85d9c83 100644 > --- a/bitbake/doc/poky.ent > +++ b/bitbake/doc/poky.ent > @@ -17,13 +17,6 @@ > > > "http://developer.berlios.de/projects/bitbake/"> - ECLIPSE_MAIN_URL "http://www.eclipse.org/downloads"> - ECLIPSE_DL_URL "http://download.eclipse.org"> - ECLIPSE_DL_PLUGIN_URL > "&YOCTO_DL_URL;/releases/eclipse-plugin/&DISTRO;"> - ECLIPSE_UPDATES_URL "&ECLIPSE_DL_URL;/tm/updates/3.3"> - ECLIPSE_INDIGO_URL "&ECLIPSE_DL_URL;/releases/indigo"> - ECLIPSE_JUNO_URL "&ECLIPSE_DL_URL;/releases/juno"> - ECLIPSE_INDIGO_CDT_URL "&ECLIPSE_DL_URL;tools/cdt/releases/indigo"> > YOCTO_SOURCES_URL "&YOCTO_HOME_URL;/sources/"> YOCTO_AB_PORT_URL "&YOCTO_AB_URL;:8010"> @@ -31,7 +24,6 @@ YOCTO_POKY_URL "&YOCTO_DL_URL;/releases/poky/"> YOCTO_RELEASE_DL_URL "&YOCTO_DL_URL;/releases/yocto/yocto-&DISTRO;"> > > - "&YOCTO_RELEASE_DL_URL;/eclipse-plugin/indigo;"> YOCTO_ADTINSTALLER_DL_URL "&YOCTO_RELEASE_DL_URL;/adt_installer"> > "&YOCTO_RELEASE_DL_URL;/&YOCTO_POKY;.tar.bz2"> YOCTO_MACHINES_DL_URL "&YOCTO_RELEASE_DL_URL;/machines"> diff --git > a/bitbake/lib/bb/COW.py b/bitbake/lib/bb/COW.py index > 7817473..d26e981 100644 --- a/bitbake/lib/bb/COW.py +++ > b/bitbake/lib/bb/COW.py @@ -1,23 +1,8 @@ -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > # > # This is a copy on write dictionary and set which abuses classes to > try and be nice and fast. # > # Copyright (C) 2006 Tim Ansell > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > -# > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA. -# > #Please Note: > # Be careful when using mutable types (ie Dict and Lists) - > operations involving these are SLOW. # Assign a file to __warn__ to > get warnings about slow operations. diff --git > a/bitbake/lib/bb/__init__.py b/bitbake/lib/bb/__init__.py index > 4bc47c8..c144311 100644 --- a/bitbake/lib/bb/__init__.py > +++ b/bitbake/lib/bb/__init__.py > @@ -1,5 +1,3 @@ > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > # > # BitBake Build System Python Library > # > @@ -8,20 +6,10 @@ > # > # Based on Gentoo's portage.py. > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > +# SPDX-License-Identifier: GPL-2.0-only > # > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA.=20 > -__version__ =3D "1.40.0" > +__version__ =3D "1.44.0" > =20 > import sys > if sys.version_info < (3, 4, 0): > diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py > index 3e2a94e..30a2ba2 100644 > --- a/bitbake/lib/bb/build.py > +++ b/bitbake/lib/bb/build.py > @@ -1,5 +1,3 @@ > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > # > # BitBake 'Build' implementation > # > @@ -10,18 +8,7 @@ > # > # Based on Gentoo's portage.py. > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > -# > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only > # > # Based on functions from the base bb module, Copyright 2003 Holger > Schurig=20 > @@ -67,23 +54,6 @@ else: > builtins['bb'] =3D bb > builtins['os'] =3D os > =20 > -class FuncFailed(Exception): > - def __init__(self, name =3D None, logfile =3D None): > - self.logfile =3D logfile > - self.name =3D name > - if name: > - self.msg =3D 'Function failed: %s' % name > - else: > - self.msg =3D "Function failed" > - > - def __str__(self): > - if self.logfile and os.path.exists(self.logfile): > - msg =3D ("%s (log file is located at %s)" % > - (self.msg, self.logfile)) > - else: > - msg =3D self.msg > - return msg > - > class TaskBase(event.Event): > """Base class for task events""" > =20 > @@ -176,15 +146,33 @@ class LogTee(object): > =20 > def __repr__(self): > return ''.format(self.name) > + > def flush(self): > self.outfile.flush() > =20 > -# > -# pythonexception allows the python exceptions generated to be raised > -# as the real exceptions (not FuncFailed) and without a backtrace at > the -# origin of the failure. > -# > -def exec_func(func, d, dirs =3D None, pythonexception=3DFalse): > + > +class StdoutNoopContextManager: > + """ > + This class acts like sys.stdout, but adds noop __enter__ and > __exit__ methods. > + """ > + def __enter__(self): > + return sys.stdout > + > + def __exit__(self, *exc_info): > + pass > + > + def write(self, string): > + return sys.stdout.write(string) > + > + def flush(self): > + sys.stdout.flush() > + > + @property > + def name(self): > + return sys.stdout.name > + > + > +def exec_func(func, d, dirs =3D None): > """Execute a BB 'function'""" > =20 > try: > @@ -256,7 +244,7 @@ def exec_func(func, d, dirs =3D None, > pythonexception=3DFalse):=20 > with bb.utils.fileslocked(lockfiles): > if ispython: > - exec_func_python(func, d, runfile, cwd=3Dadir, > pythonexception=3Dpythonexception) > + exec_func_python(func, d, runfile, cwd=3Dadir) > else: > exec_func_shell(func, d, runfile, cwd=3Dadir) > =20 > @@ -276,7 +264,7 @@ _functionfmt =3D """ > {function}(d) > """ > logformatter =3D bb.msg.BBLogFormatter("%(levelname)s: %(message)s") > -def exec_func_python(func, d, runfile, cwd=3DNone, > pythonexception=3DFalse): +def exec_func_python(func, d, runfile, > cwd=3DNone): """Execute a python BB 'function'""" > =20 > code =3D _functionfmt.format(function=3Dfunc) > @@ -301,13 +289,7 @@ def exec_func_python(func, d, runfile, cwd=3DNone, > pythonexception=3DFalse): bb.methodpool.insert_method(func, text, fn, > lineno - 1)=20 > comp =3D utils.better_compile(code, func, "exec_python_func() > autogenerated") > - utils.better_exec(comp, {"d": d}, code, "exec_python_func() > autogenerated", pythonexception=3Dpythonexception) > - except (bb.parse.SkipRecipe, bb.build.FuncFailed): > - raise > - except: > - if pythonexception: > - raise > - raise FuncFailed(func, None) > + utils.better_exec(comp, {"d": d}, code, "exec_python_func() > autogenerated") finally: > bb.debug(2, "Python function %s finished" % func) > =20 > @@ -335,6 +317,42 @@ trap 'bb_exit_handler' 0 > set -e > ''' > =20 > +def create_progress_handler(func, progress, logfile, d): > + if progress =3D=3D 'percent': > + # Use default regex > + return bb.progress.BasicProgressHandler(d, outfile=3Dlogfile) > + elif progress.startswith('percent:'): > + # Use specified regex > + return bb.progress.BasicProgressHandler(d, > regex=3Dprogress.split(':', 1)[1], outfile=3Dlogfile) > + elif progress.startswith('outof:'): > + # Use specified regex > + return bb.progress.OutOfProgressHandler(d, > regex=3Dprogress.split(':', 1)[1], outfile=3Dlogfile) > + elif progress.startswith("custom:"): > + # Use a custom progress handler that was injected via > OE_EXTRA_IMPORTS or __builtins__ > + import functools > + from types import ModuleType > + > + parts =3D progress.split(":", 2) > + _, cls, otherargs =3D parts[0], parts[1], (parts[2] or None) > if parts[2:] else None > + if cls: > + def resolve(x, y): > + if not x: > + return None > + if isinstance(x, ModuleType): > + return getattr(x, y, None) > + return x.get(y) > + cls_obj =3D functools.reduce(resolve, cls.split("."), > bb.utils._context) > + if not cls_obj: > + # Fall-back on __builtins__ > + cls_obj =3D functools.reduce(lambda x, y: x.get(y), > cls.split("."), __builtins__) > + if cls_obj: > + return cls_obj(d, outfile=3Dlogfile, > otherargs=3Dotherargs) > + bb.warn('%s: unknown custom progress handler in task > progress varflag value "%s", ignoring' % (func, cls)) > + else: > + bb.warn('%s: invalid task progress varflag value "%s", > ignoring' % (func, progress)) + > + return logfile > + > def exec_func_shell(func, d, runfile, cwd=3DNone): > """Execute a shell function from the metadata > =20 > @@ -372,23 +390,13 @@ exit $ret > cmd =3D [fakerootcmd, runfile] > =20 > if bb.msg.loggerDefaultVerbose: > - logfile =3D LogTee(logger, sys.stdout) > + logfile =3D LogTee(logger, StdoutNoopContextManager()) > else: > - logfile =3D sys.stdout > + logfile =3D StdoutNoopContextManager() > =20 > progress =3D d.getVarFlag(func, 'progress') > if progress: > - if progress =3D=3D 'percent': > - # Use default regex > - logfile =3D bb.progress.BasicProgressHandler(d, > outfile=3Dlogfile) > - elif progress.startswith('percent:'): > - # Use specified regex > - logfile =3D bb.progress.BasicProgressHandler(d, > regex=3Dprogress.split(':', 1)[1], outfile=3Dlogfile) > - elif progress.startswith('outof:'): > - # Use specified regex > - logfile =3D bb.progress.OutOfProgressHandler(d, > regex=3Dprogress.split(':', 1)[1], outfile=3Dlogfile) > - else: > - bb.warn('%s: invalid task progress varflag value "%s", > ignoring' % (func, progress)) > + logfile =3D create_progress_handler(func, progress, logfile, d) > =20 > fifobuffer =3D bytearray() > def readfifo(data): > @@ -407,6 +415,8 @@ exit $ret > bb.plain(value) > elif cmd =3D=3D 'bbnote': > bb.note(value) > + elif cmd =3D=3D 'bbverbnote': > + bb.verbnote(value) > elif cmd =3D=3D 'bbwarn': > bb.warn(value) > elif cmd =3D=3D 'bberror': > @@ -436,13 +446,8 @@ exit $ret > with open(fifopath, 'r+b', buffering=3D0) as fifo: > try: > bb.debug(2, "Executing shell function %s" % func) > - > - try: > - with open(os.devnull, 'r+') as stdin: > - bb.process.run(cmd, shell=3DFalse, stdin=3Dstdin, > log=3Dlogfile, extrafiles=3D[(fifo,readfifo)]) > - except bb.process.CmdError: > - logfn =3D d.getVar('BB_LOGFILE') > - raise FuncFailed(func, logfn) > + with open(os.devnull, 'r+') as stdin, logfile: > + bb.process.run(cmd, shell=3DFalse, stdin=3Dstdin, > log=3Dlogfile, extrafiles=3D[(fifo,readfifo)]) finally: > os.unlink(fifopath) > =20 > @@ -570,9 +575,6 @@ def _exec_task(fn, task, d, quieterr): > event.fire(TaskStarted(task, logfn, flags, localdata), > localdata) except (bb.BBHandledException, SystemExit): > return 1 > - except FuncFailed as exc: > - logger.error(str(exc)) > - return 1 > =20 > try: > for func in (prefuncs or '').split(): > @@ -580,7 +582,10 @@ def _exec_task(fn, task, d, quieterr): > exec_func(task, localdata) > for func in (postfuncs or '').split(): > exec_func(func, localdata) > - except FuncFailed as exc: > + except bb.BBHandledException: > + event.fire(TaskFailed(task, logfn, localdata, True), > localdata) > + return 1 > + except Exception as exc: > if quieterr: > event.fire(TaskFailedSilent(task, logfn, localdata), > localdata) else: > @@ -588,9 +593,6 @@ def _exec_task(fn, task, d, quieterr): > logger.error(str(exc)) > event.fire(TaskFailed(task, logfn, localdata, > errprinted), localdata) return 1 > - except bb.BBHandledException: > - event.fire(TaskFailed(task, logfn, localdata, True), > localdata) > - return 1 > finally: > sys.stdout.flush() > sys.stderr.flush() > @@ -814,6 +816,9 @@ def add_tasks(tasklist, d): > task_deps['parents'][task] =3D [] > if 'deps' in flags: > for dep in flags['deps']: > + # Check and warn for "addtask task after foo" while > foo does not exist > + #if not dep in tasklist: > + # bb.warn('%s: dependent task %s for %s does not > exist' % (d.getVar('PN'), dep, task)) dep =3D d.expand(dep) > task_deps['parents'][task].append(dep) > =20 > diff --git a/bitbake/lib/bb/cache.py b/bitbake/lib/bb/cache.py > index 258d679..b6f7da5 100644 > --- a/bitbake/lib/bb/cache.py > +++ b/bitbake/lib/bb/cache.py > @@ -1,5 +1,3 @@ > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > # > # BitBake Cache implementation > # > @@ -15,18 +13,8 @@ > # Copyright (C) 2005 Holger Hans Peter Freyther > # Copyright (C) 2005 ROAD GmbH > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > +# SPDX-License-Identifier: GPL-2.0-only > # > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA.=20 > import os > import sys > @@ -95,21 +83,21 @@ class CoreRecipeInfo(RecipeInfoCommon): > self.appends =3D self.listvar('__BBAPPEND', metadata) > self.nocache =3D self.getvar('BB_DONT_CACHE', metadata) > =20 > + self.provides =3D self.depvar('PROVIDES', metadata) > + self.rprovides =3D self.depvar('RPROVIDES', metadata) > + self.pn =3D self.getvar('PN', metadata) or > bb.parse.vars_from_file(filename,metadata)[0] > + self.packages =3D self.listvar('PACKAGES', metadata) > + if not self.packages: > + self.packages.append(self.pn) > + self.packages_dynamic =3D self.listvar('PACKAGES_DYNAMIC', > metadata) + > self.skipreason =3D self.getvar('__SKIPPED', metadata) > if self.skipreason: > - self.pn =3D self.getvar('PN', metadata) or > bb.parse.BBHandler.vars_from_file(filename,metadata)[0] self.skipped > =3D True > - self.provides =3D self.depvar('PROVIDES', metadata) > - self.rprovides =3D self.depvar('RPROVIDES', metadata) > return > =20 > self.tasks =3D metadata.getVar('__BBTASKS', False) > =20 > - self.pn =3D self.getvar('PN', metadata) > - self.packages =3D self.listvar('PACKAGES', metadata) > - if not self.packages: > - self.packages.append(self.pn) > - > self.basetaskhashes =3D self.taskvar('BB_BASEHASH', > self.tasks, metadata) self.hashfilename =3D > self.getvar('BB_HASHFILENAME', metadata)=20 > @@ -125,11 +113,8 @@ class CoreRecipeInfo(RecipeInfoCommon): > self.stampclean =3D self.getvar('STAMPCLEAN', metadata) > self.stamp_extrainfo =3D self.flaglist('stamp-extra-info', > self.tasks, metadata) self.file_checksums =3D > self.flaglist('file-checksums', self.tasks, metadata, True) > - self.packages_dynamic =3D self.listvar('PACKAGES_DYNAMIC', > metadata) self.depends =3D self.depvar('DEPENDS', metadata) > - self.provides =3D self.depvar('PROVIDES', metadata) > self.rdepends =3D self.depvar('RDEPENDS', metadata) > - self.rprovides =3D self.depvar('RPROVIDES', metadata) > self.rrecommends =3D self.depvar('RRECOMMENDS', metadata) > self.rprovides_pkg =3D self.pkgvar('RPROVIDES', > self.packages, metadata) self.rdepends_pkg =3D > self.pkgvar('RDEPENDS', self.packages, metadata) @@ -235,7 +220,7 @@ > class CoreRecipeInfo(RecipeInfoCommon):=20 > cachedata.hashfn[fn] =3D self.hashfilename > for task, taskhash in self.basetaskhashes.items(): > - identifier =3D '%s.%s' % (fn, task) > + identifier =3D '%s:%s' % (fn, task) > cachedata.basetaskhash[identifier] =3D taskhash > =20 > cachedata.inherits[fn] =3D self.inherits > @@ -249,7 +234,7 @@ def virtualfn2realfn(virtualfn): > Convert a virtual file name to a real one + the associated > subclass keyword """ > mc =3D "" > - if virtualfn.startswith('multiconfig:'): > + if virtualfn.startswith('mc:'): > elems =3D virtualfn.split(':') > mc =3D elems[1] > virtualfn =3D ":".join(elems[2:]) > @@ -270,7 +255,7 @@ def realfn2virtual(realfn, cls, mc): > if cls: > realfn =3D "virtual:" + cls + ":" + realfn > if mc: > - realfn =3D "multiconfig:" + mc + ":" + realfn > + realfn =3D "mc:" + mc + ":" + realfn > return realfn > =20 > def variant2virtual(realfn, variant): > @@ -279,11 +264,11 @@ def variant2virtual(realfn, variant): > """ > if variant =3D=3D "": > return realfn > - if variant.startswith("multiconfig:"): > + if variant.startswith("mc:"): > elems =3D variant.split(":") > if elems[2]: > - return "multiconfig:" + elems[1] + ":virtual:" + > ":".join(elems[2:]) + ":" + realfn > - return "multiconfig:" + elems[1] + ":" + realfn > + return "mc:" + elems[1] + ":virtual:" + > ":".join(elems[2:]) + ":" + realfn > + return "mc:" + elems[1] + ":" + realfn > return "virtual:" + variant + ":" + realfn > =20 > def parse_recipe(bb_data, bbfile, appends, mc=3D''): > @@ -361,7 +346,7 @@ class NoCache(object): > bb_data =3D self.databuilder.mcdata[mc].createCopy() > newstores =3D parse_recipe(bb_data, bbfile, appends, mc) > for ns in newstores: > - datastores["multiconfig:%s:%s" % (mc, ns)] =3D > newstores[ns] > + datastores["mc:%s:%s" % (mc, ns)] =3D newstores[ns] > =20 > return datastores > =20 > @@ -411,6 +396,15 @@ class Cache(NoCache): > else: > logger.debug(1, "Cache file %s not found, building..." % > self.cachefile)=20 > + # We don't use the symlink, its just for debugging > convinience > + symlink =3D os.path.join(self.cachedir, "bb_cache.dat") > + if os.path.exists(symlink): > + bb.utils.remove(symlink) > + try: > + os.symlink(os.path.basename(self.cachefile), symlink) > + except OSError: > + pass > + > def load_cachefile(self): > cachesize =3D 0 > previous_progress =3D 0 > @@ -889,3 +883,56 @@ class MultiProcessCache(object): > p.dump([data, self.__class__.CACHE_VERSION]) > =20 > bb.utils.unlockfile(glf) > + > + > +class SimpleCache(object): > + """ > + BitBake multi-process cache implementation > + > + Used by the codeparser & file checksum caches > + """ > + > + def __init__(self, version): > + self.cachefile =3D None > + self.cachedata =3D None > + self.cacheversion =3D version > + > + def init_cache(self, d, cache_file_name=3DNone, defaultdata=3DNone): > + cachedir =3D (d.getVar("PERSISTENT_DIR") or > + d.getVar("CACHE")) > + if not cachedir: > + return defaultdata > + > + bb.utils.mkdirhier(cachedir) > + self.cachefile =3D os.path.join(cachedir, > + cache_file_name or > self.__class__.cache_file_name) > + logger.debug(1, "Using cache in '%s'", self.cachefile) > + > + glf =3D bb.utils.lockfile(self.cachefile + ".lock") > + > + try: > + with open(self.cachefile, "rb") as f: > + p =3D pickle.Unpickler(f) > + data, version =3D p.load() > + except: > + bb.utils.unlockfile(glf) > + return defaultdata > + > + bb.utils.unlockfile(glf) > + > + if version !=3D self.cacheversion: > + return defaultdata > + > + return data > + > + def save(self, data): > + if not self.cachefile: > + return > + > + glf =3D bb.utils.lockfile(self.cachefile + ".lock") > + > + with open(self.cachefile, "wb") as f: > + p =3D pickle.Pickler(f, -1) > + p.dump([data, self.cacheversion]) > + > + bb.utils.unlockfile(glf) > diff --git a/bitbake/lib/bb/cache_extra.py > b/bitbake/lib/bb/cache_extra.py index 83f4959..bf4226d 100644 > --- a/bitbake/lib/bb/cache_extra.py > +++ b/bitbake/lib/bb/cache_extra.py > @@ -1,5 +1,3 @@ > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > # > # Extra RecipeInfo will be all defined in this file. Currently, > # Only Hob (Image Creator) Requests some extra fields. So > @@ -12,18 +10,8 @@ > =20 > # Copyright (C) 2011, Intel Corporation. All rights reserved. > =20 > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > +# SPDX-License-Identifier: GPL-2.0-only > # > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA.=20 > from bb.cache import RecipeInfoCommon > =20 > diff --git a/bitbake/lib/bb/checksum.py b/bitbake/lib/bb/checksum.py > index 4e1598f..5bc8a8f 100644 > --- a/bitbake/lib/bb/checksum.py > +++ b/bitbake/lib/bb/checksum.py > @@ -2,18 +2,8 @@ > # > # Copyright (C) 2012 Intel Corporation > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > +# SPDX-License-Identifier: GPL-2.0-only > # > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA.=20 > import glob > import operator > diff --git a/bitbake/lib/bb/codeparser.py > b/bitbake/lib/bb/codeparser.py index ddd1b97..fd2c473 100644 > --- a/bitbake/lib/bb/codeparser.py > +++ b/bitbake/lib/bb/codeparser.py > @@ -1,3 +1,7 @@ > +# > +# SPDX-License-Identifier: GPL-2.0-only > +# > + > """ > BitBake code parser > =20 > @@ -33,7 +37,7 @@ from bb.cache import MultiProcessCache > logger =3D logging.getLogger('BitBake.CodeParser') > =20 > def bbhash(s): > - return hashlib.md5(s.encode("utf-8")).hexdigest() > + return hashlib.sha256(s.encode("utf-8")).hexdigest() > =20 > def check_indent(codestr): > """If the code is indented, add a top level piece of code to > 'remove' the indentation""" @@ -140,7 +144,7 @@ class > CodeParserCache(MultiProcessCache): # so that an existing cache gets > invalidated. Additionally you'll need # to increment > __cache_version__ in cache.py in order to ensure that old # recipe > caches don't trigger "Taskhash mismatch" errors. > - CACHE_VERSION =3D 10 > + CACHE_VERSION =3D 11 > =20 > def __init__(self): > MultiProcessCache.__init__(self) > @@ -368,8 +372,9 @@ class ShellParser(): > def _parse_shell(self, value): > try: > tokens, _ =3D pyshyacc.parse(value, eof=3DTrue, debug=3DFals= e) > - except pyshlex.NeedMore: > - raise sherrors.ShellSyntaxError("Unexpected EOF") > + except Exception: > + bb.error('Error during parse shell code, the last 5 > lines are:\n%s' % '\n'.join(value.split('\n')[-5:])) > + raise > =20 > self.process_tokens(tokens) > =20 > diff --git a/bitbake/lib/bb/command.py b/bitbake/lib/bb/command.py > index 6c966e3..378f389 100644 > --- a/bitbake/lib/bb/command.py > +++ b/bitbake/lib/bb/command.py > @@ -6,18 +6,8 @@ Provide an interface to interact with the bitbake > server through 'commands'=20 > # Copyright (C) 2006-2007 Richard Purdie > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > +# SPDX-License-Identifier: GPL-2.0-only > # > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA.=20 > """ > The bitbake server takes 'commands' from its UI/commandline. > diff --git a/bitbake/lib/bb/compat.py b/bitbake/lib/bb/compat.py > index de1923d..4935668 100644 > --- a/bitbake/lib/bb/compat.py > +++ b/bitbake/lib/bb/compat.py > @@ -1,3 +1,7 @@ > +# > +# SPDX-License-Identifier: GPL-2.0-only > +# > + > """Code pulled from future python versions, here for compatibility""" > =20 > from collections import MutableMapping, KeysView, ValuesView, > ItemsView, OrderedDict diff --git a/bitbake/lib/bb/cooker.py > b/bitbake/lib/bb/cooker.py index 16681ba..20ef04d 100644 > --- a/bitbake/lib/bb/cooker.py > +++ b/bitbake/lib/bb/cooker.py > @@ -1,6 +1,3 @@ > -#!/usr/bin/env python > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > # > # Copyright (C) 2003, 2004 Chris Larson > # Copyright (C) 2003, 2004 Phil Blundell > @@ -9,19 +6,8 @@ > # Copyright (C) 2005 ROAD GmbH > # Copyright (C) 2006 - 2007 Richard Purdie > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > +# SPDX-License-Identifier: GPL-2.0-only > # > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA. - > =20 > import sys, os, glob, os.path, re, time > import atexit > @@ -45,6 +31,7 @@ import pyinotify > import json > import pickle > import codecs > +import hashserv > =20 > logger =3D logging.getLogger("BitBake") > collectlog =3D logging.getLogger("BitBake.Collection") > @@ -175,27 +162,45 @@ class BBCooker: > =20 > self.configuration =3D configuration > =20 > + bb.debug(1, "BBCooker starting %s" % time.time()) > + sys.stdout.flush() > + > self.configwatcher =3D pyinotify.WatchManager() > + bb.debug(1, "BBCooker pyinotify1 %s" % time.time()) > + sys.stdout.flush() > + > self.configwatcher.bbseen =3D [] > self.configwatcher.bbwatchedfiles =3D [] > self.confignotifier =3D pyinotify.Notifier(self.configwatcher, > self.config_notifications) > + bb.debug(1, "BBCooker pyinotify2 %s" % time.time()) > + sys.stdout.flush() > self.watchmask =3D pyinotify.IN_CLOSE_WRITE | > pyinotify.IN_CREATE | pyinotify.IN_DELETE | \ > pyinotify.IN_DELETE_SELF | pyinotify.IN_MODIFY | > pyinotify.IN_MOVE_SELF | \ pyinotify.IN_MOVED_FROM | > pyinotify.IN_MOVED_TO self.watcher =3D pyinotify.WatchManager() > + bb.debug(1, "BBCooker pyinotify3 %s" % time.time()) > + sys.stdout.flush() > self.watcher.bbseen =3D [] > self.watcher.bbwatchedfiles =3D [] > self.notifier =3D pyinotify.Notifier(self.watcher, > self.notifications)=20 > + bb.debug(1, "BBCooker pyinotify complete %s" % time.time()) > + sys.stdout.flush() > + > # If being called by something like tinfoil, we need to > clean cached data # which may now be invalid > bb.parse.clear_cache() > bb.parse.BBHandler.cached_statements =3D {} > =20 > self.ui_cmdline =3D None > + self.hashserv =3D None > + self.hashservaddr =3D None > =20 > self.initConfigurationData() > =20 > + bb.debug(1, "BBCooker parsed base configuration %s" % > time.time()) > + sys.stdout.flush() > + > # we log all events to a file if so directed > if self.configuration.writeeventlog: > # register the log file writer as UI Handler > @@ -233,6 +238,9 @@ class BBCooker: > # Let SIGHUP exit as SIGTERM > signal.signal(signal.SIGHUP, self.sigterm_exception) > =20 > + bb.debug(1, "BBCooker startup complete %s" % time.time()) > + sys.stdout.flush() > + > def process_inotify_updates(self): > for n in [self.confignotifier, self.notifier]: > if n.check_events(timeout=3D0): > @@ -367,13 +375,12 @@ class BBCooker: > # Copy of the data store which has been expanded. > # Used for firing events and accessing variables where > expansion needs to be accounted for # > - bb.parse.init_parser(self.data) > - > if CookerFeatures.BASEDATASTORE_TRACKING in self.featureset: > self.disableDataTracking() > =20 > - self.data.renameVar("__depends", "__base_depends") > - self.add_filewatch(self.data.getVar("__base_depends", > False), self.configwatcher) > + for mc in self.databuilder.mcdata.values(): > + mc.renameVar("__depends", "__base_depends") > + self.add_filewatch(mc.getVar("__base_depends", False), > self.configwatcher)=20 > self.baseconfig_valid =3D True > self.parsecache_valid =3D False > @@ -385,6 +392,22 @@ class BBCooker: > except prserv.serv.PRServiceConfigError as e: > bb.fatal("Unable to start PR Server, exitting") > =20 > + if self.data.getVar("BB_HASHSERVE") =3D=3D "auto": > + # Create a new hash server bound to a unix domain socket > + if not self.hashserv: > + dbfile =3D (self.data.getVar("PERSISTENT_DIR") or > self.data.getVar("CACHE")) + "/hashserv.db" > + self.hashservaddr =3D "unix://%s/hashserve.sock" % > self.data.getVar("TOPDIR") > + self.hashserv =3D > hashserv.create_server(self.hashservaddr, dbfile, sync=3DFalse) > + self.hashserv.process =3D > multiprocessing.Process(target=3Dself.hashserv.serve_forever) > + self.hashserv.process.start() > + self.data.setVar("BB_HASHSERVE", self.hashservaddr) > + self.databuilder.origdata.setVar("BB_HASHSERVE", > self.hashservaddr) > + self.databuilder.data.setVar("BB_HASHSERVE", > self.hashservaddr) > + for mc in self.databuilder.mcdata: > + self.databuilder.mcdata[mc].setVar("BB_HASHSERVE", > self.hashservaddr) + > + bb.parse.init_parser(self.data) > + > def enableDataTracking(self): > self.configuration.tracking =3D True > if hasattr(self, "data"): > @@ -488,6 +511,7 @@ class BBCooker: > """ > fn =3D None > envdata =3D None > + mc =3D '' > if not pkgs_to_build: > pkgs_to_build =3D [] > =20 > @@ -496,6 +520,12 @@ class BBCooker: > self.enableDataTracking() > self.reset() > =20 > + def mc_base(p): > + if p.startswith('mc:'): > + s =3D p.split(':') > + if len(s) =3D=3D 2: > + return s[1] > + return None > =20 > if buildfile: > # Parse the configuration here. We need to do it > explicitly here since @@ -506,18 +536,16 @@ class BBCooker: > fn =3D self.matchFile(fn) > fn =3D bb.cache.realfn2virtual(fn, cls, mc) > elif len(pkgs_to_build) =3D=3D 1: > - ignore =3D self.data.getVar("ASSUME_PROVIDED") or "" > - if pkgs_to_build[0] in set(ignore.split()): > - bb.fatal("%s is in ASSUME_PROVIDED" % > pkgs_to_build[0]) > + mc =3D mc_base(pkgs_to_build[0]) > + if not mc: > + ignore =3D self.data.getVar("ASSUME_PROVIDED") or "" > + if pkgs_to_build[0] in set(ignore.split()): > + bb.fatal("%s is in ASSUME_PROVIDED" % > pkgs_to_build[0])=20 > - taskdata, runlist =3D self.buildTaskData(pkgs_to_build, > None, self.configuration.abort, allowincomplete=3DTrue) > + taskdata, runlist =3D > self.buildTaskData(pkgs_to_build, None, self.configuration.abort, > allowincomplete=3DTrue)=20 > - mc =3D runlist[0][0] > - fn =3D runlist[0][3] > - else: > - envdata =3D self.data > - data.expandKeys(envdata) > - parse.ast.runAnonFuncs(envdata) > + mc =3D runlist[0][0] > + fn =3D runlist[0][3] > =20 > if fn: > try: > @@ -526,6 +554,12 @@ class BBCooker: > except Exception as e: > parselog.exception("Unable to read %s", fn) > raise > + else: > + if not mc in self.databuilder.mcdata: > + bb.fatal('Not multiconfig named "%s" found' % mc) > + envdata =3D self.databuilder.mcdata[mc] > + data.expandKeys(envdata) > + parse.ast.runAnonFuncs(envdata) > =20 > # Display history > with closing(StringIO()) as env: > @@ -565,10 +599,10 @@ class BBCooker: > wildcard =3D False > =20 > # Wild card expansion: > - # Replace string such as "multiconfig:*:bash" > - # into "multiconfig:A:bash multiconfig:B:bash bash" > + # Replace string such as "mc:*:bash" > + # into "mc:A:bash mc:B:bash bash" > for k in targetlist: > - if k.startswith("multiconfig:"): > + if k.startswith("mc:"): > if wildcard: > bb.fatal('multiconfig conflict') > if k.split(":")[1] =3D=3D "*": > @@ -601,7 +635,7 @@ class BBCooker: > runlist =3D [] > for k in fulltargetlist: > mc =3D "" > - if k.startswith("multiconfig:"): > + if k.startswith("mc:"): > mc =3D k.split(":")[1] > k =3D ":".join(k.split(":")[2:]) > ktask =3D task > @@ -620,13 +654,22 @@ class BBCooker: > runlist.append([mc, k, ktask, fn]) > bb.event.fire(bb.event.TreeDataPreparationProgress(current, > len(fulltargetlist)), self.data)=20 > - mcdeps =3D taskdata[mc].get_mcdepends() > + havemc =3D False > + for mc in self.multiconfigs: > + if taskdata[mc].get_mcdepends(): > + havemc =3D True > + > # No need to do check providers if there are no mcdeps or > not an mc build > - if mcdeps and mc: > - # Make sure we can provide the multiconfig dependency > + if havemc or len(self.multiconfigs) > 1: > seen =3D set() > new =3D True > + # Make sure we can provide the multiconfig dependency > while new: > + mcdeps =3D set() > + # Add unresolved first, so we can get multiconfig > indirect dependencies on time > + for mc in self.multiconfigs: > + taskdata[mc].add_unresolved(localdata[mc], > self.recipecaches[mc]) > + mcdeps |=3D set(taskdata[mc].get_mcdepends()) > new =3D False > for mc in self.multiconfigs: > for k in mcdeps: > @@ -641,6 +684,7 @@ class BBCooker: > taskdata[depmc].add_provider(localdata[depmc= ], > self.recipecaches[depmc], l[3]) seen.add(k) > new =3D True > + > for mc in self.multiconfigs: > taskdata[mc].add_unresolved(localdata[mc], > self.recipecaches[mc])=20 > @@ -676,7 +720,7 @@ class BBCooker: > @staticmethod > def add_mc_prefix(mc, pn): > if mc: > - return "multiconfig:%s:%s" % (mc, pn) > + return "mc:%s:%s" % (mc, pn) > return pn > =20 > def buildDependTree(self, rq, taskdata): > @@ -875,6 +919,10 @@ class BBCooker: > os.unlink('package-depends.dot') > except FileNotFoundError: > pass > + try: > + os.unlink('recipe-depends.dot') > + except FileNotFoundError: > + pass > =20 > with open('task-depends.dot', 'w') as f: > f.write("digraph depends {\n") > @@ -888,27 +936,6 @@ class BBCooker: > f.write("}\n") > logger.info("Task dependencies saved to 'task-depends.dot'") > =20 > - with open('recipe-depends.dot', 'w') as f: > - f.write("digraph depends {\n") > - pndeps =3D {} > - for task in sorted(depgraph["tdepends"]): > - (pn, taskname) =3D task.rsplit(".", 1) > - if pn not in pndeps: > - pndeps[pn] =3D set() > - for dep in sorted(depgraph["tdepends"][task]): > - (deppn, deptaskname) =3D dep.rsplit(".", 1) > - pndeps[pn].add(deppn) > - for pn in sorted(pndeps): > - fn =3D depgraph["pn"][pn]["filename"] > - version =3D depgraph["pn"][pn]["version"] > - f.write('"%s" [label=3D"%s\\n%s\\n%s"]\n' % (pn, pn, > version, fn)) > - for dep in sorted(pndeps[pn]): > - if dep =3D=3D pn: > - continue > - f.write('"%s" -> "%s"\n' % (pn, dep)) > - f.write("}\n") > - logger.info("Flattened recipe dependencies saved to > 'recipe-depends.dot'") - > def show_appends_with_no_recipes(self): > # Determine which bbappends haven't been applied > =20 > @@ -1191,8 +1218,8 @@ class BBCooker: > continue > elif regex =3D=3D "": > parselog.debug(1, "BBFILE_PATTERN_%s is empty" % > c) > + cre =3D re.compile('^NULL$') > errors =3D False > - continue > else: > try: > cre =3D re.compile(regex) > @@ -1453,7 +1480,7 @@ class BBCooker: > ntargets =3D [] > for target in runlist: > if target[0]: > - ntargets.append("multiconfig:%s:%s:%s" % (target[0], > target[1], target[2])) > + ntargets.append("mc:%s:%s:%s" % (target[0], > target[1], target[2])) ntargets.append("%s:%s" % (target[1], > target[2]))=20 > for mc in self.multiconfigs: > @@ -1576,6 +1603,9 @@ class BBCooker: > for pkg in pkgs_to_build: > if pkg in ignore: > parselog.warning("Explicit target \"%s\" is in > ASSUME_PROVIDED, ignoring" % pkg) > + if pkg.startswith("multiconfig:"): > + pkgs_to_build.remove(pkg) > + pkgs_to_build.append(pkg.replace("multiconfig:", > "mc:"))=20 > if 'world' in pkgs_to_build: > pkgs_to_build.remove('world') > @@ -1583,7 +1613,7 @@ class BBCooker: > bb.providers.buildWorldTargetList(self.recipecaches[mc], > task) for t in self.recipecaches[mc].world_target: > if mc: > - t =3D "multiconfig:" + mc + ":" + t > + t =3D "mc:" + mc + ":" + t > pkgs_to_build.append(t) > =20 > if 'universe' in pkgs_to_build: > @@ -1602,7 +1632,7 @@ class BBCooker: > bb.debug(1, "Skipping %s for universe > tasks as task %s doesn't exist" % (t, task)) continue > if mc: > - t =3D "multiconfig:" + mc + ":" + t > + t =3D "mc:" + mc + ":" + t > pkgs_to_build.append(t) > =20 > return pkgs_to_build > @@ -1615,9 +1645,11 @@ class BBCooker: > =20 > def post_serve(self): > prserv.serv.auto_shutdown() > + if self.hashserv: > + self.hashserv.process.terminate() > + self.hashserv.process.join() > bb.event.fire(CookerExit(), self.data) > =20 > - > def shutdown(self, force =3D False): > if force: > self.state =3D state.forceshutdown > @@ -1632,6 +1664,7 @@ class BBCooker: > =20 > def reset(self): > self.initConfigurationData() > + self.handlePRServ() > =20 > def clientComplete(self): > """Called when the client is done using the server""" > @@ -1865,35 +1898,6 @@ class ParsingFailure(Exception): > self.recipe =3D recipe > Exception.__init__(self, realexception, recipe) > =20 > -class Feeder(multiprocessing.Process): > - def __init__(self, jobs, to_parsers, quit): > - self.quit =3D quit > - self.jobs =3D jobs > - self.to_parsers =3D to_parsers > - multiprocessing.Process.__init__(self) > - > - def run(self): > - while True: > - try: > - quit =3D self.quit.get_nowait() > - except queue.Empty: > - pass > - else: > - if quit =3D=3D 'cancel': > - self.to_parsers.cancel_join_thread() > - break > - > - try: > - job =3D self.jobs.pop() > - except IndexError: > - break > - > - try: > - self.to_parsers.put(job, timeout=3D0.5) > - except queue.Full: > - self.jobs.insert(0, job) > - continue > - > class Parser(multiprocessing.Process): > def __init__(self, jobs, results, quit, init, profile): > self.jobs =3D jobs > @@ -1940,11 +1944,8 @@ class Parser(multiprocessing.Process): > result =3D pending.pop() > else: > try: > - job =3D self.jobs.get(timeout=3D0.25) > - except queue.Empty: > - continue > - > - if job is None: > + job =3D self.jobs.pop() > + except IndexError: > break > result =3D self.parse(*job) > =20 > @@ -2028,14 +2029,15 @@ class CookerParser(object): > multiprocessing.util.Finalize(None, > bb.codeparser.parser_cache_save, exitpriority=3D1) > multiprocessing.util.Finalize(None, bb.fetch.fetcher_parse_save, > exitpriority=3D1)=20 > - self.feeder_quit =3D multiprocessing.Queue(maxsize=3D1) > self.parser_quit =3D > multiprocessing.Queue(maxsize=3Dself.num_processes) > - self.jobs =3D > multiprocessing.Queue(maxsize=3Dself.num_processes) self.result_queue =3D > multiprocessing.Queue() > - self.feeder =3D Feeder(self.willparse, self.jobs, > self.feeder_quit) > - self.feeder.start() > + > + def chunkify(lst,n): > + return [lst[i::n] for i in range(n)] > + self.jobs =3D chunkify(self.willparse, self.num_processes) > + > for i in range(0, self.num_processes): > - parser =3D Parser(self.jobs, self.result_queue, > self.parser_quit, init, self.cooker.configuration.profile) > + parser =3D Parser(self.jobs[i], self.result_queue, > self.parser_quit, init, self.cooker.configuration.profile) > parser.start() self.process_names.append(parser.name) > self.processes.append(parser) > @@ -2056,17 +2058,20 @@ class CookerParser(object): > self.total) > =20 > bb.event.fire(event, self.cfgdata) > - self.feeder_quit.put(None) > for process in self.processes: > self.parser_quit.put(None) > else: > - self.feeder_quit.put('cancel') > - > self.parser_quit.cancel_join_thread() > for process in self.processes: > self.parser_quit.put(None) > =20 > - self.jobs.cancel_join_thread() > + # Cleanup the queue before call process.join(), otherwise > there might be > + # deadlocks. > + while True: > + try: > + self.result_queue.get(timeout=3D0.25) > + except queue.Empty: > + break > =20 > for process in self.processes: > if force: > @@ -2074,7 +2079,6 @@ class CookerParser(object): > process.terminate() > else: > process.join() > - self.feeder.join() > =20 > sync =3D threading.Thread(target=3Dself.bb_cache.sync) > sync.start() > diff --git a/bitbake/lib/bb/cookerdata.py > b/bitbake/lib/bb/cookerdata.py index 5df66e6..472423f 100644 > --- a/bitbake/lib/bb/cookerdata.py > +++ b/bitbake/lib/bb/cookerdata.py > @@ -1,6 +1,3 @@ > -#!/usr/bin/env python > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > # > # Copyright (C) 2003, 2004 Chris Larson > # Copyright (C) 2003, 2004 Phil Blundell > @@ -9,23 +6,14 @@ > # Copyright (C) 2005 ROAD GmbH > # Copyright (C) 2006 Richard Purdie > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > +# SPDX-License-Identifier: GPL-2.0-only > # > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA.=20 > import logging > import os > import re > import sys > +import hashlib > from functools import wraps > import bb > from bb import data > @@ -134,6 +122,7 @@ class CookerConfiguration(object): > self.profile =3D False > self.nosetscene =3D False > self.setsceneonly =3D False > + self.skipsetscene =3D False > self.invalidate_stamp =3D False > self.dump_signatures =3D [] > self.dry_run =3D False > @@ -279,12 +268,13 @@ class CookerDataBuilder(object): > self.mcdata =3D {} > =20 > def parseBaseConfiguration(self): > + data_hash =3D hashlib.sha256() > try: > - bb.parse.init_parser(self.basedata) > self.data =3D self.parseConfigurationFiles(self.prefiles, > self.postfiles)=20 > if self.data.getVar("BB_WORKERCONTEXT", False) is None: > bb.fetch.fetcher_init(self.data) > + bb.parse.init_parser(self.data) > bb.codeparser.parser_cache_init(self.data) > =20 > bb.event.fire(bb.event.ConfigParsed(), self.data) > @@ -302,7 +292,7 @@ class CookerDataBuilder(object): > bb.event.fire(bb.event.ConfigParsed(), self.data) > =20 > bb.parse.init_parser(self.data) > - self.data_hash =3D self.data.get_hash() > + data_hash.update(self.data.get_hash().encode('utf-8')) > self.mcdata[''] =3D self.data > =20 > multiconfig =3D (self.data.getVar("BBMULTICONFIG") or > "").split() @@ -310,9 +300,11 @@ class CookerDataBuilder(object): > mcdata =3D self.parseConfigurationFiles(self.prefiles, > self.postfiles, config) bb.event.fire(bb.event.ConfigParsed(), mcdata) > self.mcdata[config] =3D mcdata > + data_hash.update(mcdata.get_hash().encode('utf-8')) > if multiconfig: > bb.event.fire(bb.event.MultiConfigParsed(self.mcdata), > self.data)=20 > + self.data_hash =3D data_hash.hexdigest() > except (SyntaxError, bb.BBHandledException): > raise bb.BBHandledException > except bb.data_smart.ExpansionError as e: > @@ -354,14 +346,24 @@ class CookerDataBuilder(object): > data =3D parse_config_file(layerconf, data) > =20 > layers =3D (data.getVar('BBLAYERS') or "").split() > + broken_layers =3D [] > =20 > data =3D bb.data.createCopy(data) > approved =3D bb.utils.approved_variables() > + > + # Check whether present layer directories exist > for layer in layers: > if not os.path.isdir(layer): > - parselog.critical("Layer directory '%s' does not > exist! " > - "Please check BBLAYERS in %s" > % (layer, layerconf)) > - sys.exit(1) > + broken_layers.append(layer) > + > + if broken_layers: > + parselog.critical("The following layer directories > do not exist:") > + for layer in broken_layers: > + parselog.critical(" %s", layer) > + parselog.critical("Please check BBLAYERS in %s" % > (layerconf)) > + sys.exit(1) > + > + for layer in layers: > parselog.debug(2, "Adding layer %s", layer) > if 'HOME' in approved and '~' in layer: > layer =3D os.path.expanduser(layer) > @@ -391,7 +393,11 @@ class CookerDataBuilder(object): > bb.fatal("BBFILES_DYNAMIC entries must be of the > form :, not:\n %s" % "\n > ".join(invalid)) layerseries =3D > set((data.getVar("LAYERSERIES_CORENAMES") or "").split()) > + collections_tmp =3D collections[:] > for c in collections: > + collections_tmp.remove(c) > + if c in collections_tmp: > + bb.fatal("Found duplicated BBFILE_COLLECTIONS > '%s', check bblayers.conf or layer.conf to fix it." % c) compat =3D > set((data.getVar("LAYERSERIES_COMPAT_%s" % c) or "").split()) if > compat and not (compat & layerseries): bb.fatal("Layer %s is not > compatible with the core layer which only supports these series: %s > (layer is compatible with %s)" diff --git > a/bitbake/lib/bb/daemonize.py b/bitbake/lib/bb/daemonize.py index > c937675..f01e6ec 100644 --- a/bitbake/lib/bb/daemonize.py +++ > b/bitbake/lib/bb/daemonize.py @@ -1,3 +1,7 @@ > +# > +# SPDX-License-Identifier: GPL-2.0-only > +# > + > """ > Python Daemonizing helper > =20 > diff --git a/bitbake/lib/bb/data.py b/bitbake/lib/bb/data.py > index d66d98c..0d75d0c 100644 > --- a/bitbake/lib/bb/data.py > +++ b/bitbake/lib/bb/data.py > @@ -1,5 +1,3 @@ > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > """ > BitBake 'Data' implementations > =20 > @@ -22,18 +20,7 @@ the speed is more critical here. > # Copyright (C) 2003, 2004 Chris Larson > # Copyright (C) 2005 Holger Hans Peter Freyther > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > -# > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only > # > # Based on functions from the base bb module, Copyright 2003 Holger > Schurig=20 > @@ -143,7 +130,7 @@ def emit_var(var, o=3Dsys.__stdout__, d =3D init(), > all=3DFalse): if all: > oval =3D d.getVar(var, False) > val =3D d.getVar(var) > - except (KeyboardInterrupt, bb.build.FuncFailed): > + except (KeyboardInterrupt): > raise > except Exception as exc: > o.write('# expansion of %s threw %s: %s\n' % (var, > exc.__class__.__name__, str(exc))) @@ -322,8 +309,6 @@ def > build_dependencies(key, keys, shelldeps, varflagsexcl, d): if > varflags.get("python"): value =3D d.getVarFlag(key, "_content", False) > parser =3D bb.codeparser.PythonParser(key, logger) > - if value and "\t" in value: > - logger.warning("Variable %s contains tabs, > please remove these (%s)" % (key, d.getVar("FILE"))) > parser.parse_python(value, filename=3Dvarflags.get("filename"), > lineno=3Dvarflags.get("lineno")) deps =3D deps | parser.references deps = =3D > deps | (keys & parser.execs) @@ -437,8 +422,8 @@ def > generate_dependency_hash(tasklist, gendeps, lookupcache, whitelist, > fn): var =3D lookupcache[dep] if var is not None: > data =3D data + str(var) > - k =3D fn + "." + task > - basehash[k] =3D hashlib.md5(data.encode("utf-8")).hexdigest() > + k =3D fn + ":" + task > + basehash[k] =3D > hashlib.sha256(data.encode("utf-8")).hexdigest() taskdeps[task] =3D > alldeps=20 > return taskdeps, basehash > diff --git a/bitbake/lib/bb/data_smart.py > b/bitbake/lib/bb/data_smart.py index 67af380..dd5c618 100644 > --- a/bitbake/lib/bb/data_smart.py > +++ b/bitbake/lib/bb/data_smart.py > @@ -1,5 +1,3 @@ > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > """ > BitBake Smart Dictionary Implementation > =20 > @@ -14,18 +12,8 @@ BitBake build tools. > # Copyright (C) 2005 Uli Luckas > # Copyright (C) 2005 ROAD GmbH > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > +# SPDX-License-Identifier: GPL-2.0-only > # > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA. # Based on functions from the base bb module, > Copyright 2003 Holger Schurig=20 > import copy, re, sys, traceback > @@ -39,10 +27,11 @@ from bb.COW import COWDictBase > logger =3D logging.getLogger("BitBake.Data") > =20 > __setvar_keyword__ =3D ["_append", "_prepend", "_remove"] > -__setvar_regexp__ =3D > re.compile('(?P.*?)(?P_append|_prepend|_remove)(_(?P[= ^A-Z]*))?$') > -__expand_var_regexp__ =3D re.compile(r"\${[^{}@\n\t :]+}") > +__setvar_regexp__ =3D > re.compile(r'(?P.*?)(?P_append|_prepend|_remove)(_(?P= [^A-Z]*))?$') > +__expand_var_regexp__ =3D re.compile(r"\${[a-zA-Z0-9\-_+./~]+?}") > __expand_python_regexp__ =3D re.compile(r"\${@.+?}") > -__whitespace_split__ =3D re.compile('(\s)') +__whitespace_split__ =3D > re.compile(r'(\s)') +__override_regexp__ =3D re.compile(r'[a-z0-9]+')=20 > def infer_caller_details(loginfo, parent =3D False, varval =3D True): > """Save the caller the trouble of specifying everything.""" > @@ -597,7 +586,7 @@ class DataSmart(MutableMapping): > # aka pay the cookie monster > override =3D var[var.rfind('_')+1:] > shortvar =3D var[:var.rfind('_')] > - while override and override.islower(): > + while override and __override_regexp__.match(override): > if shortvar not in self.overridedata: > self.overridedata[shortvar] =3D [] > if [var, override] not in self.overridedata[shortvar]: > @@ -1073,4 +1062,4 @@ class DataSmart(MutableMapping): > data.update({i:value}) > =20 > data_str =3D str([(k, data[k]) for k in sorted(data.keys())]) > - return hashlib.md5(data_str.encode("utf-8")).hexdigest() > + return hashlib.sha256(data_str.encode("utf-8")).hexdigest() > diff --git a/bitbake/lib/bb/event.py b/bitbake/lib/bb/event.py > index 5b1b094..d44621e 100644 > --- a/bitbake/lib/bb/event.py > +++ b/bitbake/lib/bb/event.py > @@ -1,5 +1,3 @@ > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > """ > BitBake 'Event' implementation > =20 > @@ -9,18 +7,8 @@ BitBake build tools. > =20 > # Copyright (C) 2003, 2004 Chris Larson > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > +# SPDX-License-Identifier: GPL-2.0-only > # > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA.=20 > import os, sys > import warnings > @@ -136,6 +124,7 @@ def fire_class_handlers(event, d): > ui_queue =3D [] > @atexit.register > def print_ui_queue(): > + global ui_queue > """If we're exiting before a UI has been spawned, display any > queued LogRecords to the console.""" > logger =3D logging.getLogger("BitBake") > @@ -180,6 +169,7 @@ def print_ui_queue(): > logger.removeHandler(stderr) > else: > logger.removeHandler(stdout) > + ui_queue =3D [] > =20 > def fire_ui_handlers(event, d): > global _thread_lock > @@ -414,23 +404,6 @@ class RecipeTaskPreProcess(RecipeEvent): > class RecipeParsed(RecipeEvent): > """ Recipe Parsing Complete """ > =20 > -class StampUpdate(Event): > - """Trigger for any adjustment of the stamp files to happen""" > - > - def __init__(self, targets, stampfns): > - self._targets =3D targets > - self._stampfns =3D stampfns > - Event.__init__(self) > - > - def getStampPrefix(self): > - return self._stampfns > - > - def getTargets(self): > - return self._targets > - > - stampPrefix =3D property(getStampPrefix) > - targets =3D property(getTargets) > - > class BuildBase(Event): > """Base class for bitbake build events""" > =20 > diff --git a/bitbake/lib/bb/exceptions.py > b/bitbake/lib/bb/exceptions.py index cd71343..ecbad59 100644 > --- a/bitbake/lib/bb/exceptions.py > +++ b/bitbake/lib/bb/exceptions.py > @@ -1,3 +1,6 @@ > +# > +# SPDX-License-Identifier: GPL-2.0-only > +# > =20 > import inspect > import traceback > diff --git a/bitbake/lib/bb/fetch2/__init__.py > b/bitbake/lib/bb/fetch2/__init__.py index 572b71a..1f5f8f1 100644 > --- a/bitbake/lib/bb/fetch2/__init__.py > +++ b/bitbake/lib/bb/fetch2/__init__.py > @@ -1,5 +1,3 @@ > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > """ > BitBake 'Fetch' implementations > =20 > @@ -10,18 +8,7 @@ BitBake build tools. > # Copyright (C) 2003, 2004 Chris Larson > # Copyright (C) 2012 Intel Corporation > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > -# > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only > # > # Based on functions from the base bb module, Copyright 2003 Holger > Schurig=20 > @@ -256,7 +243,7 @@ class URI(object): > =20 > # Identify if the URI is relative or not > if urlp.scheme in self._relative_schemes and \ > - re.compile("^\w+:(?!//)").match(uri): > + re.compile(r"^\w+:(?!//)").match(uri): > self.relative =3D True > =20 > if not self.relative: > @@ -524,7 +511,7 @@ def fetcher_parse_save(): > def fetcher_parse_done(): > _checksum_cache.save_merge() > =20 > -def fetcher_compare_revisions(): > +def fetcher_compare_revisions(d): > """ > Compare the revisions in the persistant cache with current > values and return true/false on whether they've changed. > @@ -777,7 +764,8 @@ def get_srcrev(d, > method_name=3D'sortable_revision'): # > format =3D d.getVar('SRCREV_FORMAT') > if not format: > - raise FetchError("The SRCREV_FORMAT variable must be set > when multiple SCMs are used.") > + raise FetchError("The SRCREV_FORMAT variable must be set > when multiple SCMs are used.\n"\ > + "The SCMs are:\n%s" % '\n'.join(scms)) > =20 > name_to_rev =3D {} > seenautoinc =3D False > @@ -855,10 +843,18 @@ def runfetchcmd(cmd, d, quiet=3DFalse, > cleanup=3DNone, log=3DNone, workdir=3DNone): if val: > cmd =3D 'export ' + var + '=3D\"%s\"; %s' % (val, cmd) > =20 > + # Ensure that a _PYTHON_SYSCONFIGDATA_NAME value set by a recipe > + # (for example via python3native.bbclass since warrior) is not > set for > + # host Python (otherwise tools like git-make-shallow will fail) > + cmd =3D 'unset _PYTHON_SYSCONFIGDATA_NAME; ' + cmd > + > # Disable pseudo as it may affect ssh, potentially causing it to > hang. cmd =3D 'export PSEUDO_DISABLED=3D1; ' + cmd > =20 > - logger.debug(1, "Running %s", cmd) > + if workdir: > + logger.debug(1, "Running '%s' in %s" % (cmd, workdir)) > + else: > + logger.debug(1, "Running %s", cmd) > =20 > success =3D False > error_message =3D "" > @@ -894,7 +890,7 @@ def check_network_access(d, info, url): > log remote network access, and error if BB_NO_NETWORK is set or > the given URI is untrusted > """ > - if d.getVar("BB_NO_NETWORK") =3D=3D "1": > + if bb.utils.to_boolean(d.getVar("BB_NO_NETWORK")): > raise NetworkAccess(url, info) > elif not trusted_network(d, url): > raise UntrustedUrl(url, info) > @@ -966,7 +962,8 @@ def rename_bad_checksum(ud, suffix): > =20 > new_localpath =3D "%s_bad-checksum_%s" % (ud.localpath, suffix) > bb.warn("Renaming %s to %s" % (ud.localpath, new_localpath)) > - bb.utils.movefile(ud.localpath, new_localpath) > + if not bb.utils.movefile(ud.localpath, new_localpath): > + bb.warn("Renaming %s to %s failed, grep movefile in > log.do_fetch to see why" % (ud.localpath, new_localpath))=20 > =20 > def try_mirror_url(fetch, origud, ud, ld, check =3D False): > @@ -1027,7 +1024,7 @@ def try_mirror_url(fetch, origud, ud, ld, check > =3D False): raise > =20 > except IOError as e: > - if e.errno in [os.errno.ESTALE]: > + if e.errno in [errno.ESTALE]: > logger.warning("Stale Error Observed %s." % ud.url) > return False > raise > @@ -1094,7 +1091,7 @@ def trusted_network(d, url): > BB_ALLOWED_NETWORKS is set globally or for a specific recipe. > Note: modifies SRC_URI & mirrors. > """ > - if d.getVar('BB_NO_NETWORK') =3D=3D "1": > + if bb.utils.to_boolean(d.getVar("BB_NO_NETWORK")): > return True > =20 > pkgname =3D d.expand(d.getVar('PN', False)) > @@ -1403,7 +1400,7 @@ class FetchMethod(object): > Fetch urls > Assumes localpath was called first > """ > - raise NoMethodError(url) > + raise NoMethodError(urldata.url) > =20 > def unpack(self, urldata, rootdir, data): > iterate =3D False > @@ -1469,7 +1466,7 @@ class FetchMethod(object): > else: > cmd =3D 'rpm2cpio.sh %s | cpio -id' % (file) > elif file.endswith('.deb') or file.endswith('.ipk'): > - output =3D subprocess.check_output('ar -t %s' % file, > preexec_fn=3Dsubprocess_setup, shell=3DTrue) > + output =3D subprocess.check_output(['ar', '-t', file], > preexec_fn=3Dsubprocess_setup) datafile =3D None > if output: > for line in output.decode().splitlines(): > @@ -1547,7 +1544,7 @@ class FetchMethod(object): > Check the status of a URL > Assumes localpath was called first > """ > - logger.info("URL %s could not be checked for status since no > method exists.", url) > + logger.info("URL %s could not be checked for status since no > method exists.", urldata.url) return True > =20 > def latest_revision(self, ud, d, name): > @@ -1555,7 +1552,7 @@ class FetchMethod(object): > Look in the cache for the latest revision, if not present > ask the SCM. """ > if not hasattr(self, "_latest_revision"): > - raise ParameterError("The fetcher for this URL does not > support _latest_revision", url) > + raise ParameterError("The fetcher for this URL does not > support _latest_revision", ud.url)=20 > revs =3D bb.persist_data.persist('BB_URI_HEADREVS', d) > key =3D self.generate_revision_key(ud, d, name) > @@ -1638,7 +1635,7 @@ class Fetch(object): > urls =3D self.urls > =20 > network =3D self.d.getVar("BB_NO_NETWORK") > - premirroronly =3D (self.d.getVar("BB_FETCH_PREMIRRORONLY") =3D=3D > "1") > + premirroronly =3D > bb.utils.to_boolean(self.d.getVar("BB_FETCH_PREMIRRORONLY"))=20 > for u in urls: > ud =3D self.ud[u] > @@ -1716,7 +1713,7 @@ class Fetch(object): > update_stamp(ud, self.d) > =20 > except IOError as e: > - if e.errno in [os.errno.ESTALE]: > + if e.errno in [errno.ESTALE]: > logger.error("Stale Error Observed %s." % u) > raise ChecksumError("Stale Error Detected") > =20 > @@ -1786,7 +1783,7 @@ class Fetch(object): > =20 > for url in urls: > if url not in self.ud: > - self.ud[url] =3D FetchData(url, d) > + self.ud[url] =3D FetchData(url, self.d) > ud =3D self.ud[url] > ud.setup_localpath(self.d) > =20 > diff --git a/bitbake/lib/bb/fetch2/bzr.py > b/bitbake/lib/bb/fetch2/bzr.py index 658502f..c56d875 100644 > --- a/bitbake/lib/bb/fetch2/bzr.py > +++ b/bitbake/lib/bb/fetch2/bzr.py > @@ -10,18 +10,8 @@ BitBake 'Fetch' implementation for bzr. > # BitBake build tools. > # Copyright (C) 2003, 2004 Chris Larson > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > +# SPDX-License-Identifier: GPL-2.0-only > # > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA.=20 > import os > import sys > diff --git a/bitbake/lib/bb/fetch2/clearcase.py > b/bitbake/lib/bb/fetch2/clearcase.py index 3a6573d..3dd93ad 100644 > --- a/bitbake/lib/bb/fetch2/clearcase.py > +++ b/bitbake/lib/bb/fetch2/clearcase.py > @@ -1,5 +1,3 @@ > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > """ > BitBake 'Fetch' clearcase implementation > =20 > @@ -47,18 +45,7 @@ User credentials: > """ > # Copyright (C) 2014 Siemens AG > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > -# > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only > # > =20 > import os > @@ -67,6 +54,8 @@ import shutil > import bb > from bb.fetch2 import FetchMethod > from bb.fetch2 import FetchError > +from bb.fetch2 import MissingParameterError > +from bb.fetch2 import ParameterError > from bb.fetch2 import runfetchcmd > from bb.fetch2 import logger > =20 > @@ -92,7 +81,7 @@ class ClearCase(FetchMethod): > if 'protocol' in ud.parm: > ud.proto =3D ud.parm['protocol'] > if not ud.proto in ('http', 'https'): > - raise fetch2.ParameterError("Invalid protocol type", > ud.url) > + raise ParameterError("Invalid protocol type", ud.url) > =20 > ud.vob =3D '' > if 'vob' in ud.parm: > diff --git a/bitbake/lib/bb/fetch2/cvs.py > b/bitbake/lib/bb/fetch2/cvs.py index 0e0a319..1b35ba4 100644 > --- a/bitbake/lib/bb/fetch2/cvs.py > +++ b/bitbake/lib/bb/fetch2/cvs.py > @@ -1,5 +1,3 @@ > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > """ > BitBake 'Fetch' implementations > =20 > @@ -10,20 +8,9 @@ BitBake build tools. > =20 > # Copyright (C) 2003, 2004 Chris Larson > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > +# SPDX-License-Identifier: GPL-2.0-only > # > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA. -# > -#Based on functions from the base bb module, Copyright 2003 Holger > Schurig +# Based on functions from the base bb module, Copyright 2003 > Holger Schurig # > =20 > import os > diff --git a/bitbake/lib/bb/fetch2/git.py > b/bitbake/lib/bb/fetch2/git.py index 59a2ee8..2d1d2ca 100644 > --- a/bitbake/lib/bb/fetch2/git.py > +++ b/bitbake/lib/bb/fetch2/git.py > @@ -1,5 +1,3 @@ > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > """ > BitBake 'Fetch' git implementation > =20 > @@ -55,20 +53,10 @@ Supported SRC_URI options are: > =20 > """ > =20 > -#Copyright (C) 2005 Richard Purdie > +# Copyright (C) 2005 Richard Purdie > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > +# SPDX-License-Identifier: GPL-2.0-only > # > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA.=20 > import collections > import errno > @@ -199,7 +187,7 @@ class Git(FetchMethod): > depth_default =3D 1 > ud.shallow_depths =3D collections.defaultdict(lambda: > depth_default)=20 > - revs_default =3D d.getVar("BB_GIT_SHALLOW_REVS", True) > + revs_default =3D d.getVar("BB_GIT_SHALLOW_REVS") > ud.shallow_revs =3D [] > ud.branches =3D {} > for pos, name in enumerate(ud.names): > @@ -318,7 +306,7 @@ class Git(FetchMethod): > def try_premirror(self, ud, d): > # If we don't do this, updating an existing checkout with > only premirrors # is not possible > - if d.getVar("BB_FETCH_PREMIRRORONLY") is not None: > + if bb.utils.to_boolean(d.getVar("BB_FETCH_PREMIRRORONLY")): > return True > if os.path.exists(ud.clonedir): > return False > @@ -476,6 +464,8 @@ class Git(FetchMethod): > if os.path.exists(destdir): > bb.utils.prunedir(destdir) > =20 > + need_lfs =3D ud.parm.get("lfs", "1") =3D=3D "1" > + > source_found =3D False > source_error =3D [] > =20 > @@ -503,6 +493,13 @@ class Git(FetchMethod): > =20 > repourl =3D self._get_repo_url(ud) > runfetchcmd("%s remote set-url origin %s" % (ud.basecmd, > repourl), d, workdir=3Ddestdir) + > + if self._contains_lfs(ud, d, destdir): > + if need_lfs and not self._find_git_lfs(d): > + raise bb.fetch2.FetchError("Repository %s has LFS > content, install git-lfs on host to download (or set lfs=3D0 to ignore > it)" % (repourl)) > + else: > + bb.note("Repository %s has LFS content but it is not > being fetched" % (repourl)) + > if not ud.nocheckout: > if subdir !=3D "": > runfetchcmd("%s read-tree %s%s" % (ud.basecmd, > ud.revisions[ud.names[0]], readpathspec), d, @@ -522,9 +519,17 @@ > class Git(FetchMethod): def clean(self, ud, d): > """ clean the git directory """ > =20 > - bb.utils.remove(ud.localpath, True) > - bb.utils.remove(ud.fullmirror) > - bb.utils.remove(ud.fullmirror + ".done") > + to_remove =3D [ud.localpath, ud.fullmirror, ud.fullmirror + > ".done"] > + # The localpath is a symlink to clonedir when it is cloned > from a > + # mirror, so remove both of them. > + if os.path.islink(ud.localpath): > + clonedir =3D os.path.realpath(ud.localpath) > + to_remove.append(clonedir) > + > + for r in to_remove: > + if os.path.exists(r): > + bb.note('Removing %s' % r) > + bb.utils.remove(r, True) > =20 > def supports_srcrev(self): > return True > @@ -545,6 +550,27 @@ class Git(FetchMethod): > raise bb.fetch2.FetchError("The command '%s' gave output > with more then 1 line unexpectedly, output: '%s'" % (cmd, output)) > return output.split()[0] !=3D "0"=20 > + def _contains_lfs(self, ud, d, wd): > + """ > + Check if the repository has 'lfs' (large file) content > + """ > + cmd =3D "%s grep lfs HEAD:.gitattributes | wc -l" % ( > + ud.basecmd) > + try: > + output =3D runfetchcmd(cmd, d, quiet=3DTrue, workdir=3Dwd) > + if int(output) > 0: > + return True > + except (bb.fetch2.FetchError,ValueError): > + pass > + return False > + > + def _find_git_lfs(self, d): > + """ > + Return True if git-lfs can be found, False otherwise. > + """ > + import shutil > + return shutil.which("git-lfs", path=3Dd.getVar('PATH')) is not > None + > def _get_repo_url(self, ud): > """ > Return the repository URL > @@ -615,7 +641,7 @@ class Git(FetchMethod): > """ > pupver =3D ('', '') > =20 > - tagregex =3D re.compile(d.getVar('UPSTREAM_CHECK_GITTAGREGEX') > or "(?P([0-9][\.|_]?)+)") > + tagregex =3D re.compile(d.getVar('UPSTREAM_CHECK_GITTAGREGEX') > or r"(?P([0-9][\.|_]?)+)") try: > output =3D self._lsremote(ud, d, "refs/tags/*") > except (bb.fetch2.FetchError, bb.fetch2.NetworkAccess) as e: > @@ -630,7 +656,7 @@ class Git(FetchMethod): > =20 > tag_head =3D line.split("/")[-1] > # Ignore non-released branches > - m =3D re.search("(alpha|beta|rc|final)+", tag_head) > + m =3D re.search(r"(alpha|beta|rc|final)+", tag_head) > if m: > continue > =20 > diff --git a/bitbake/lib/bb/fetch2/gitannex.py > b/bitbake/lib/bb/fetch2/gitannex.py index a9b69ca..1d497dc 100644 > --- a/bitbake/lib/bb/fetch2/gitannex.py > +++ b/bitbake/lib/bb/fetch2/gitannex.py > @@ -1,5 +1,3 @@ > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > """ > BitBake 'Fetch' git annex implementation > """ > @@ -7,18 +5,8 @@ BitBake 'Fetch' git annex implementation > # Copyright (C) 2014 Otavio Salvador > # Copyright (C) 2014 O.S. Systems Software LTDA. > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > +# SPDX-License-Identifier: GPL-2.0-only > # > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA.=20 > import os > import bb > diff --git a/bitbake/lib/bb/fetch2/gitsm.py > b/bitbake/lib/bb/fetch2/gitsm.py index 35729db..c622771 100644 > --- a/bitbake/lib/bb/fetch2/gitsm.py > +++ b/bitbake/lib/bb/fetch2/gitsm.py > @@ -1,5 +1,3 @@ > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > """ > BitBake 'Fetch' git submodules implementation > =20 > @@ -16,18 +14,8 @@ NOTE: Switching a SRC_URI from "git://" to > "gitsm://" requires a clean of your r=20 > # Copyright (C) 2013 Richard Purdie > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > +# SPDX-License-Identifier: GPL-2.0-only > # > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA.=20 > import os > import bb > @@ -45,60 +33,97 @@ class GitSM(Git): > """ > return ud.type in ['gitsm'] > =20 > - @staticmethod > - def parse_gitmodules(gitmodules): > - modules =3D {} > - module =3D "" > - for line in gitmodules.splitlines(): > - if line.startswith('[submodule'): > - module =3D line.split('"')[1] > - modules[module] =3D {} > - elif module and line.strip().startswith('path'): > - path =3D line.split('=3D')[1].strip() > - modules[module]['path'] =3D path > - elif module and line.strip().startswith('url'): > - url =3D line.split('=3D')[1].strip() > - modules[module]['url'] =3D url > - return modules > - > - def update_submodules(self, ud, d): > + def process_submodules(self, ud, workdir, function, d): > + """ > + Iterate over all of the submodules in this repository and > execute > + the 'function' for each of them. > + """ > + > submodules =3D [] > paths =3D {} > + revision =3D {} > uris =3D {} > - local_paths =3D {} > - > + subrevision =3D {} > + > + def parse_gitmodules(gitmodules): > + modules =3D {} > + module =3D "" > + for line in gitmodules.splitlines(): > + if line.startswith('[submodule'): > + module =3D line.split('"')[1] > + modules[module] =3D {} > + elif module and line.strip().startswith('path'): > + path =3D line.split('=3D')[1].strip() > + modules[module]['path'] =3D path > + elif module and line.strip().startswith('url'): > + url =3D line.split('=3D')[1].strip() > + modules[module]['url'] =3D url > + return modules > + > + # Collect the defined submodules, and their attributes > for name in ud.names: > try: > - gitmodules =3D runfetchcmd("%s show %s:.gitmodules" % > (ud.basecmd, ud.revisions[name]), d, quiet=3DTrue, workdir=3Dud.clonedir) > + gitmodules =3D runfetchcmd("%s show %s:.gitmodules" % > (ud.basecmd, ud.revisions[name]), d, quiet=3DTrue, workdir=3Dworkdir) > except: # No submodules to update > continue > =20 > - for m, md in self.parse_gitmodules(gitmodules).items(): > + for m, md in parse_gitmodules(gitmodules).items(): > + try: > + module_hash =3D runfetchcmd("%s ls-tree -z -d %s > %s" % (ud.basecmd, ud.revisions[name], md['path']), d, quiet=3DTrue, > workdir=3Dworkdir) > + except: > + # If the command fails, we don't have a valid > file to check. If it doesn't > + # fail -- it still might be a failure, see next > check... > + module_hash =3D "" > + > + if not module_hash: > + logger.debug(1, "submodule %s is defined, but is > not initialized in the repository. Skipping", m) > + continue > + > submodules.append(m) > paths[m] =3D md['path'] > + revision[m] =3D ud.revisions[name] > uris[m] =3D md['url'] > + subrevision[m] =3D module_hash.split()[2] > + > + # Convert relative to absolute uri based on parent > uri if uris[m].startswith('..'): > newud =3D copy.copy(ud) > - newud.path =3D > os.path.realpath(os.path.join(newud.path, md['url'])) > + newud.path =3D > os.path.realpath(os.path.join(newud.path, uris[m])) uris[m] =3D > Git._get_repo_url(self, newud)=20 > for module in submodules: > - module_hash =3D runfetchcmd("%s ls-tree -z -d %s %s" % > (ud.basecmd, ud.revisions[name], paths[module]), d, quiet=3DTrue, > workdir=3Dud.clonedir) > - module_hash =3D module_hash.split()[2] > + # Translate the module url into a SRC_URI > + > + if "://" in uris[module]: > + # Properly formated URL already > + proto =3D uris[module].split(':', 1)[0] > + url =3D uris[module].replace('%s:' % proto, 'gitsm:', > 1) > + else: > + if ":" in uris[module]: > + # Most likely an SSH style reference > + proto =3D "ssh" > + if ":/" in uris[module]: > + # Absolute reference, easy to convert.. > + url =3D "gitsm://" + > uris[module].replace(':/', '/', 1) > + else: > + # Relative reference, no way to know if this > is right! > + logger.warning("Submodule included by %s > refers to relative ssh reference %s. References may fail if not > absolute." % (ud.url, uris[module])) > + url =3D "gitsm://" + uris[module].replace(':', > '/', 1) > + else: > + # This has to be a file reference > + proto =3D "file" > + url =3D "gitsm://" + uris[module] > =20 > - # Build new SRC_URI > - proto =3D uris[module].split(':', 1)[0] > - url =3D uris[module].replace('%s:' % proto, 'gitsm:', 1) > url +=3D ';protocol=3D%s' % proto > url +=3D ";name=3D%s" % module > - url +=3D ";bareclone=3D1;nocheckout=3D1;nobranch=3D1" > + url +=3D ";subpath=3D%s" % module > =20 > ld =3D d.createCopy() > # Not necessary to set SRC_URI, since we're passing the > URI to # Fetch. > #ld.setVar('SRC_URI', url) > - ld.setVar('SRCREV_%s' % module, module_hash) > + ld.setVar('SRCREV_%s' % module, subrevision[module]) > =20 > # Workaround for issues with SRCPV/SRCREV_FORMAT errors > # error refer to 'multiple' repositories. Only the > repository @@ -106,145 +131,85 @@ class GitSM(Git): > ld.setVar('SRCPV', d.getVar('SRCPV')) > ld.setVar('SRCREV_FORMAT', module) > =20 > - newfetch =3D Fetch([url], ld, cache=3DFalse) > - newfetch.download() > - local_paths[module] =3D newfetch.localpath(url) > + function(ud, url, module, paths[module], ld) > =20 > - # Correct the submodule references to the local download > version... > - runfetchcmd("%(basecmd)s config submodule.%(module)s.url > %(url)s" % {'basecmd': ud.basecmd, 'module': module, 'url' : > local_paths[module]}, d, workdir=3Dud.clonedir) - > - symlink_path =3D os.path.join(ud.clonedir, 'modules', > paths[module]) > - if not os.path.exists(symlink_path): > - try: > - os.makedirs(os.path.dirname(symlink_path), > exist_ok=3DTrue) > - except OSError: > - pass > - os.symlink(local_paths[module], symlink_path) > - > - return True > + return submodules !=3D [] > =20 > def need_update(self, ud, d): > - main_repo_needs_update =3D Git.need_update(self, ud, d) > - > - # First check that the main repository has enough history > fetched. If it doesn't, then we don't > - # even have the .gitmodules and gitlinks for the submodules > to attempt asking whether the > - # submodules' histories are recent enough. > - if main_repo_needs_update: > + if Git.need_update(self, ud, d): > return True > =20 > - # Now check that the submodule histories are new enough. The > git-submodule command doesn't have > - # any clean interface for doing this aside from just > attempting the checkout (with network > - # fetched disabled). > - return not self.update_submodules(ud, d) > + try: > + # Check for the nugget dropped by the download operation > + known_srcrevs =3D runfetchcmd("%s config --get-all > bitbake.srcrev" % \ > + (ud.basecmd), d, workdir=3Dud.clonedir) > =20 > - def download(self, ud, d): > - Git.download(self, ud, d) > + if ud.revisions[ud.names[0]] not in > known_srcrevs.split(): > + return True > + except bb.fetch2.FetchError: > + # No srcrev nuggets, so this is new and needs to be > updated > + return True > =20 > - if not ud.shallow or ud.localpath !=3D ud.fullshallow: > - self.update_submodules(ud, d) > + return False > =20 > - def copy_submodules(self, submodules, ud, destdir, d): > - if ud.bareclone: > - repo_conf =3D destdir > - else: > - repo_conf =3D os.path.join(destdir, '.git') > + def download(self, ud, d): > + def download_submodule(ud, url, module, modpath, d): > + url +=3D ";bareclone=3D1;nobranch=3D1" > =20 > - if submodules and not os.path.exists(os.path.join(repo_conf, > 'modules')): > - os.mkdir(os.path.join(repo_conf, 'modules')) > + # Is the following still needed? > + #url +=3D ";nocheckout=3D1" > =20 > - for module, md in submodules.items(): > - srcpath =3D os.path.join(ud.clonedir, 'modules', > md['path']) > - modpath =3D os.path.join(repo_conf, 'modules', md['path']) > + try: > + newfetch =3D Fetch([url], d, cache=3DFalse) > + newfetch.download() > + # Drop a nugget to add each of the srcrevs we've > fetched (used by need_update) > + runfetchcmd("%s config --add bitbake.srcrev %s" % \ > + (ud.basecmd, ud.revisions[ud.names[0]]), > d, workdir=3Dud.clonedir) > + except Exception as e: > + logger.error('gitsm: submodule download failed: %s > %s' % (type(e).__name__, str(e))) > + raise > =20 > - if os.path.exists(srcpath): > - if os.path.exists(os.path.join(srcpath, '.git')): > - srcpath =3D os.path.join(srcpath, '.git') > + Git.download(self, ud, d) > + self.process_submodules(ud, ud.clonedir, download_submodule, > d)=20 > - target =3D modpath > - if os.path.exists(modpath): > - target =3D os.path.dirname(modpath) > + def unpack(self, ud, destdir, d): > + def unpack_submodules(ud, url, module, modpath, d): > + url +=3D ";bareclone=3D1;nobranch=3D1" > =20 > - os.makedirs(os.path.dirname(target), exist_ok=3DTrue) > - runfetchcmd("cp -fpLR %s %s" % (srcpath, target), d) > - elif os.path.exists(modpath): > - # Module already exists, likely unpacked from a > shallow mirror clone > - pass > + # Figure out where we clone over the bare submodules... > + if ud.bareclone: > + repo_conf =3D ud.destdir > else: > - # This is fatal, as we do NOT want git-submodule to > hit the network > - raise bb.fetch2.FetchError('Submodule %s does not > exist in %s or %s.' % (module, srcpath, modpath)) - > - def clone_shallow_local(self, ud, dest, d): > - super(GitSM, self).clone_shallow_local(ud, dest, d) > + repo_conf =3D os.path.join(ud.destdir, '.git') > =20 > - # Copy over the submodules' fetched histories too. > - repo_conf =3D os.path.join(dest, '.git') > - > - submodules =3D [] > - for name in ud.names: > try: > - gitmodules =3D runfetchcmd("%s show %s:.gitmodules" % > (ud.basecmd, ud.revision), d, quiet=3DTrue, workdir=3Ddest) > - except: > - # No submodules to update > - continue > + newfetch =3D Fetch([url], d, cache=3DFalse) > + > newfetch.unpack(root=3Dos.path.dirname(os.path.join(repo_conf, > 'modules', module))) > + except Exception as e: > + logger.error('gitsm: submodule unpack failed: %s %s' > % (type(e).__name__, str(e))) > + raise > =20 > - submodules =3D self.parse_gitmodules(gitmodules) > - self.copy_submodules(submodules, ud, dest, d) > + local_path =3D newfetch.localpath(url) > =20 > - def unpack(self, ud, destdir, d): > - Git.unpack(self, ud, destdir, d) > + # Correct the submodule references to the local download > version... > + runfetchcmd("%(basecmd)s config submodule.%(module)s.url > %(url)s" % {'basecmd': ud.basecmd, 'module': module, 'url' : > local_path}, d, workdir=3Dud.destdir)=20 > - # Copy over the submodules' fetched histories too. > - if ud.bareclone: > - repo_conf =3D ud.destdir > - else: > - repo_conf =3D os.path.join(ud.destdir, '.git') > + if ud.shallow: > + runfetchcmd("%(basecmd)s config > submodule.%(module)s.shallow true" % {'basecmd': ud.basecmd, > 'module': module}, d, workdir=3Dud.destdir)=20 > - update_submodules =3D False > - paths =3D {} > - uris =3D {} > - local_paths =3D {} > - for name in ud.names: > + # Ensure the submodule repository is NOT set to bare, > since we're checking it out... try: > - gitmodules =3D runfetchcmd("%s show HEAD:.gitmodules" > % (ud.basecmd), d, quiet=3DTrue, workdir=3Dud.destdir) > + runfetchcmd("%s config core.bare false" % > (ud.basecmd), d, quiet=3DTrue, workdir=3Dos.path.join(repo_conf, > 'modules', module)) except: > - # No submodules to update > - continue > - > - submodules =3D self.parse_gitmodules(gitmodules) > - self.copy_submodules(submodules, ud, ud.destdir, d) > - > - submodules_queue =3D [(module, os.path.join(repo_conf, > 'modules', md['path'])) for module, md in submodules.items()] > - while len(submodules_queue) !=3D 0: > - module, modpath =3D submodules_queue.pop() > - > - # add submodule children recursively > - try: > - gitmodules =3D runfetchcmd("%s show > HEAD:.gitmodules" % (ud.basecmd), d, quiet=3DTrue, workdir=3Dmodpath) > - for m, md in > self.parse_gitmodules(gitmodules).items(): > - submodules_queue.append([m, > os.path.join(modpath, 'modules', md['path'])]) > - except: > - # no children > - pass > - > + logger.error("Unable to set git config core.bare to > false for %s" % os.path.join(repo_conf, 'modules', module)) > + raise > =20 > - # There are submodules to update > - update_submodules =3D True > - > - # Determine (from the submodule) the correct url to > reference > - try: > - output =3D runfetchcmd("%(basecmd)s config > remote.origin.url" % {'basecmd': ud.basecmd}, d, workdir=3Dmodpath) > - except bb.fetch2.FetchError as e: > - # No remote url defined in this submodule > - continue > - > - local_paths[module] =3D output > - > - # Setup the local URL properly (like git submodule > init or sync would do...) > - runfetchcmd("%(basecmd)s config > submodule.%(module)s.url %(url)s" % {'basecmd': ud.basecmd, 'module': > module, 'url' : local_paths[module]}, d, workdir=3Dud.destdir) > + Git.unpack(self, ud, destdir, d) > =20 > - # Ensure the submodule repository is NOT set to > bare, since we're checking it out... > - runfetchcmd("%s config core.bare false" % > (ud.basecmd), d, quiet=3DTrue, workdir=3Dmodpath) > + ret =3D self.process_submodules(ud, ud.destdir, > unpack_submodules, d)=20 > - if update_submodules: > - # Run submodule update, this sets up the directories -- > without touching the config > + if not ud.bareclone and ret: > + # All submodules should already be downloaded and > configured in the tree. This simply sets > + # up the configuration and checks out the files. The > main project config should remain > + # unmodified, and no download from the internet should > occur. runfetchcmd("%s submodule update --recursive --no-fetch" % > (ud.basecmd), d, quiet=3DTrue, workdir=3Dud.destdir) diff --git > a/bitbake/lib/bb/fetch2/hg.py b/bitbake/lib/bb/fetch2/hg.py index > 936d043..15d729e 100644 --- a/bitbake/lib/bb/fetch2/hg.py > +++ b/bitbake/lib/bb/fetch2/hg.py > @@ -1,5 +1,3 @@ > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > """ > BitBake 'Fetch' implementation for mercurial DRCS (hg). > =20 > @@ -9,20 +7,10 @@ BitBake 'Fetch' implementation for mercurial DRCS > (hg). # Copyright (C) 2004 Marcin Juszkiewicz > # Copyright (C) 2007 Robert Schuster > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > -# > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only > # > # Based on functions from the base bb module, Copyright 2003 Holger > Schurig +# > =20 > import os > import sys > @@ -99,7 +87,7 @@ class Hg(FetchMethod): > def try_premirror(self, ud, d): > # If we don't do this, updating an existing checkout with > only premirrors # is not possible > - if d.getVar("BB_FETCH_PREMIRRORONLY") is not None: > + if bb.utils.to_boolean(d.getVar("BB_FETCH_PREMIRRORONLY")): > return True > if os.path.exists(ud.moddir): > return False > diff --git a/bitbake/lib/bb/fetch2/local.py > b/bitbake/lib/bb/fetch2/local.py index a114ac1..01d9ff9 100644 > --- a/bitbake/lib/bb/fetch2/local.py > +++ b/bitbake/lib/bb/fetch2/local.py > @@ -1,5 +1,3 @@ > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > """ > BitBake 'Fetch' implementations > =20 > @@ -10,20 +8,10 @@ BitBake build tools. > =20 > # Copyright (C) 2003, 2004 Chris Larson > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > -# > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only > # > # Based on functions from the base bb module, Copyright 2003 Holger > Schurig +# > =20 > import os > import urllib.request, urllib.parse, urllib.error > diff --git a/bitbake/lib/bb/fetch2/npm.py > b/bitbake/lib/bb/fetch2/npm.py index 65bf5a3..9700e61 100644 > --- a/bitbake/lib/bb/fetch2/npm.py > +++ b/bitbake/lib/bb/fetch2/npm.py > @@ -1,5 +1,6 @@ > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > +# > +# SPDX-License-Identifier: GPL-2.0-only > +# > """ > BitBake 'Fetch' NPM implementation > =20 > @@ -100,11 +101,19 @@ class Npm(FetchMethod): > return False > return True > =20 > - def _runwget(self, ud, d, command, quiet): > - logger.debug(2, "Fetching %s using command '%s'" % (ud.url, > command)) > - bb.fetch2.check_network_access(d, command, ud.url) > + def _runpack(self, ud, d, pkgfullname: str, quiet=3DFalse) -> str: > + """ > + Runs npm pack on a full package name. > + Returns the filename of the downloaded package > + """ > + bb.fetch2.check_network_access(d, pkgfullname, ud.registry) > dldir =3D d.getVar("DL_DIR") > - runfetchcmd(command, d, quiet, workdir=3Ddldir) > + dldir =3D os.path.join(dldir, ud.prefixdir) > + > + command =3D "npm pack {} --registry {}".format(pkgfullname, > ud.registry) > + logger.debug(2, "Fetching {} using command '{}' in > {}".format(pkgfullname, command, dldir)) > + filename =3D runfetchcmd(command, d, quiet, workdir=3Ddldir) > + return filename.rstrip() > =20 > def _unpackdep(self, ud, pkg, data, destdir, dldir, d): > file =3D data[pkg]['tgz'] > @@ -150,20 +159,11 @@ class Npm(FetchMethod): > Parse the output of npm view --json; the last JSON result > is assumed to be the one that we're interested in. > ''' > - pdata =3D None > - outdeps =3D {} > - datalines =3D [] > - bracelevel =3D 0 > - for line in output.splitlines(): > - if bracelevel: > - datalines.append(line) > - elif '{' in line: > - datalines =3D [] > - datalines.append(line) > - bracelevel =3D bracelevel + line.count('{') - > line.count('}') > - if datalines: > - pdata =3D json.loads('\n'.join(datalines)) > - return pdata > + pdata =3D json.loads(output); > + try: > + return pdata[-1] > + except: > + return pdata > =20 > def _getdependencies(self, pkg, data, version, d, ud, > optional=3DFalse, fetchedlist=3DNone): if fetchedlist is None: > @@ -171,6 +171,9 @@ class Npm(FetchMethod): > pkgfullname =3D pkg > if version !=3D '*' and not '/' in version: > pkgfullname +=3D "@'%s'" % version > + if pkgfullname in fetchedlist: > + return > + > logger.debug(2, "Calling getdeps on %s" % pkg) > fetchcmd =3D "npm view %s --json --registry %s" % > (pkgfullname, ud.registry) output =3D runfetchcmd(fetchcmd, d, True) > @@ -190,15 +193,10 @@ class Npm(FetchMethod): > if (not blacklist and 'linux' not in pkg_os) or > '!linux' in pkg_os: logger.debug(2, "Skipping %s since it's > incompatible with Linux" % pkg) return > - #logger.debug(2, "Output URL is %s - %s - %s" % > (ud.basepath, ud.basename, ud.localfile)) > - outputurl =3D pdata['dist']['tarball'] > + filename =3D self._runpack(ud, d, pkgfullname) > data[pkg] =3D {} > - data[pkg]['tgz'] =3D os.path.basename(outputurl) > - if outputurl in fetchedlist: > - return > - > - self._runwget(ud, d, "%s --directory-prefix=3D%s %s" % > (self.basecmd, ud.prefixdir, outputurl), False) > - fetchedlist.append(outputurl) > + data[pkg]['tgz'] =3D filename > + fetchedlist.append(pkgfullname) > =20 > dependencies =3D pdata.get('dependencies', {}) > optionalDependencies =3D pdata.get('optionalDependencies', {}) > @@ -225,17 +223,12 @@ class Npm(FetchMethod): > if obj =3D=3D pkg: > self._getshrinkeddependencies(obj, > data['dependencies'][obj], data['dependencies'][obj]['version'], d, > ud, lockdown, manifest, False) return > - outputurl =3D "invalid" > - if ('resolved' not in data) or (not > data['resolved'].startswith('http://') and not > data['resolved'].startswith('https://')): > - # will be the case for ${PN} > - fetchcmd =3D "npm view %s@%s dist.tarball --registry %s" % > (pkg, version, ud.registry) > - logger.debug(2, "Found this matching URL: %s" % > str(fetchcmd)) > - outputurl =3D runfetchcmd(fetchcmd, d, True) > - else: > - outputurl =3D data['resolved'] > - self._runwget(ud, d, "%s --directory-prefix=3D%s %s" % > (self.basecmd, ud.prefixdir, outputurl), False) + > + pkgnameWithVersion =3D "{}@{}".format(pkg, version) > + logger.debug(2, "Get dependencies for > {}".format(pkgnameWithVersion)) > + filename =3D self._runpack(ud, d, pkgnameWithVersion) > manifest[pkg] =3D {} > - manifest[pkg]['tgz'] =3D os.path.basename(outputurl).rstrip() > + manifest[pkg]['tgz'] =3D filename > manifest[pkg]['deps'] =3D {} > =20 > if pkg in lockdown: > diff --git a/bitbake/lib/bb/fetch2/osc.py > b/bitbake/lib/bb/fetch2/osc.py index 6c60456..3e56715 100644 > --- a/bitbake/lib/bb/fetch2/osc.py > +++ b/bitbake/lib/bb/fetch2/osc.py > @@ -1,5 +1,6 @@ > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > +# > +# SPDX-License-Identifier: GPL-2.0-only > +# > """ > Bitbake "Fetch" implementation for osc (Opensuse build service > client). Based on the svn "Fetch" implementation. > diff --git a/bitbake/lib/bb/fetch2/perforce.py > b/bitbake/lib/bb/fetch2/perforce.py index 903a8e6..54d001e 100644 > --- a/bitbake/lib/bb/fetch2/perforce.py > +++ b/bitbake/lib/bb/fetch2/perforce.py > @@ -1,5 +1,3 @@ > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > """ > BitBake 'Fetch' implementation for perforce > =20 > @@ -8,18 +6,7 @@ BitBake 'Fetch' implementation for perforce > # Copyright (C) 2003, 2004 Chris Larson > # Copyright (C) 2016 Kodak Alaris, Inc. > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > -# > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only > # > # Based on functions from the base bb module, Copyright 2003 Holger > Schurig=20 > diff --git a/bitbake/lib/bb/fetch2/repo.py > b/bitbake/lib/bb/fetch2/repo.py index 8c7e818..2bdbbd4 100644 > --- a/bitbake/lib/bb/fetch2/repo.py > +++ b/bitbake/lib/bb/fetch2/repo.py > @@ -1,5 +1,3 @@ > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > """ > BitBake "Fetch" repo (git) implementation > =20 > @@ -8,20 +6,10 @@ BitBake "Fetch" repo (git) implementation > # Copyright (C) 2009 Tom Rini > # > # Based on git.py which is: > -#Copyright (C) 2005 Richard Purdie > +# Copyright (C) 2005 Richard Purdie > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > +# SPDX-License-Identifier: GPL-2.0-only > # > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA.=20 > import os > import bb > diff --git a/bitbake/lib/bb/fetch2/s3.py b/bitbake/lib/bb/fetch2/s3.py > index 1629288..ffca73c 100644 > --- a/bitbake/lib/bb/fetch2/s3.py > +++ b/bitbake/lib/bb/fetch2/s3.py > @@ -1,5 +1,3 @@ > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > """ > BitBake 'Fetch' implementation for Amazon AWS S3. > =20 > @@ -13,18 +11,7 @@ The aws tool must be correctly installed and > configured prior to use. # Based in part on bb.fetch2.wget: > # Copyright (C) 2003, 2004 Chris Larson > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > -# > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only > # > # Based on functions from the base bb module, Copyright 2003 Holger > Schurig=20 > diff --git a/bitbake/lib/bb/fetch2/sftp.py > b/bitbake/lib/bb/fetch2/sftp.py index 81884a6..f87f292 100644 > --- a/bitbake/lib/bb/fetch2/sftp.py > +++ b/bitbake/lib/bb/fetch2/sftp.py > @@ -1,5 +1,3 @@ > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > """ > BitBake SFTP Fetch implementation > =20 > @@ -44,18 +42,7 @@ SRC_URI =3D > "sftp://user@host.example.com/dir/path.file.txt" # Based in part on > bb.fetch2.wget: # Copyright (C) 2003, 2004 Chris Larson > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > -# > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only > # > # Based on functions from the base bb module, Copyright 2003 Holger > Schurig=20 > diff --git a/bitbake/lib/bb/fetch2/ssh.py > b/bitbake/lib/bb/fetch2/ssh.py index 6047ee4..f5be060 100644 > --- a/bitbake/lib/bb/fetch2/ssh.py > +++ b/bitbake/lib/bb/fetch2/ssh.py > @@ -1,5 +1,3 @@ > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > ''' > BitBake 'Fetch' implementations > =20 > @@ -29,18 +27,8 @@ IETF secsh internet draft: > # Copyright 2003 Holger Schurig > # > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > +# SPDX-License-Identifier: GPL-2.0-only > # > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA.=20 > import re, os > from bb.fetch2 import FetchMethod > diff --git a/bitbake/lib/bb/fetch2/svn.py > b/bitbake/lib/bb/fetch2/svn.py index ed70bcf..96d666b 100644 > --- a/bitbake/lib/bb/fetch2/svn.py > +++ b/bitbake/lib/bb/fetch2/svn.py > @@ -1,5 +1,3 @@ > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > """ > BitBake 'Fetch' implementation for svn. > =20 > @@ -8,18 +6,7 @@ BitBake 'Fetch' implementation for svn. > # Copyright (C) 2003, 2004 Chris Larson > # Copyright (C) 2004 Marcin Juszkiewicz > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > -# > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only > # > # Based on functions from the base bb module, Copyright 2003 Holger > Schurig=20 > @@ -63,6 +50,9 @@ class Svn(FetchMethod): > relpath =3D self._strip_leading_slashes(ud.path) > ud.pkgdir =3D os.path.join(svndir, ud.host, relpath) > ud.moddir =3D os.path.join(ud.pkgdir, ud.module) > + # Protects the repository from concurrent updates, e.g. from > two > + # recipes fetching different revisions at the same time > + ud.svnlock =3D os.path.join(ud.pkgdir, "svn.lock") > =20 > ud.setup_revisions(d) > =20 > @@ -101,6 +91,13 @@ class Svn(FetchMethod): > svncmd =3D "%s log --limit 1 %s %s://%s/%s/" % > (ud.basecmd, " ".join(options), proto, svnroot, ud.module) else: > suffix =3D "" > + > + # externals may be either 'allowed' or 'nowarn', but not > both. Allowed > + # will not issue a warning, but will log to the debug > buffer what has likely > + # been downloaded by SVN. > + if not ("externals" in ud.parm and ud.parm["externals"] > =3D=3D "allowed"): > + options.append("--ignore-externals") > + > if ud.revision: > options.append("-r %s" % ud.revision) > suffix =3D "@%s" % (ud.revision) > @@ -123,35 +120,52 @@ class Svn(FetchMethod): > =20 > logger.debug(2, "Fetch: checking for module directory '" + > ud.moddir + "'")=20 > - if os.access(os.path.join(ud.moddir, '.svn'), os.R_OK): > - svnupdatecmd =3D self._buildsvncommand(ud, d, "update") > - logger.info("Update " + ud.url) > - # We need to attempt to run svn upgrade first in case > its an older working format > - try: > - runfetchcmd(ud.basecmd + " upgrade", d, > workdir=3Dud.moddir) > - except FetchError: > - pass > - logger.debug(1, "Running %s", svnupdatecmd) > - bb.fetch2.check_network_access(d, svnupdatecmd, ud.url) > - runfetchcmd(svnupdatecmd, d, workdir=3Dud.moddir) > - else: > - svnfetchcmd =3D self._buildsvncommand(ud, d, "fetch") > - logger.info("Fetch " + ud.url) > - # check out sources there > - bb.utils.mkdirhier(ud.pkgdir) > - logger.debug(1, "Running %s", svnfetchcmd) > - bb.fetch2.check_network_access(d, svnfetchcmd, ud.url) > - runfetchcmd(svnfetchcmd, d, workdir=3Dud.pkgdir) > - > - scmdata =3D ud.parm.get("scmdata", "") > - if scmdata =3D=3D "keep": > - tar_flags =3D "" > - else: > - tar_flags =3D "--exclude=3D'.svn'" > + lf =3D bb.utils.lockfile(ud.svnlock) > + > + try: > + if os.access(os.path.join(ud.moddir, '.svn'), os.R_OK): > + svnupdatecmd =3D self._buildsvncommand(ud, d, "update") > + logger.info("Update " + ud.url) > + # We need to attempt to run svn upgrade first in > case its an older working format > + try: > + runfetchcmd(ud.basecmd + " upgrade", d, > workdir=3Dud.moddir) > + except FetchError: > + pass > + logger.debug(1, "Running %s", svnupdatecmd) > + bb.fetch2.check_network_access(d, svnupdatecmd, > ud.url) > + runfetchcmd(svnupdatecmd, d, workdir=3Dud.moddir) > + else: > + svnfetchcmd =3D self._buildsvncommand(ud, d, "fetch") > + logger.info("Fetch " + ud.url) > + # check out sources there > + bb.utils.mkdirhier(ud.pkgdir) > + logger.debug(1, "Running %s", svnfetchcmd) > + bb.fetch2.check_network_access(d, svnfetchcmd, > ud.url) > + runfetchcmd(svnfetchcmd, d, workdir=3Dud.pkgdir) > + > + if not ("externals" in ud.parm and ud.parm["externals"] > =3D=3D "nowarn"): > + # Warn the user if this had externals (won't catch > them all) > + output =3D runfetchcmd("svn propget svn:externals || > true", d, workdir=3Dud.moddir) > + if output: > + if "--ignore-externals" in svnfetchcmd.split(): > + bb.warn("%s contains svn:externals." % > ud.url) > + bb.warn("These should be added to the recipe > SRC_URI as necessary.") > + bb.warn("svn fetch has ignored > externals:\n%s" % output) > + bb.warn("To disable this warning add > ';externals=3Dnowarn' to the url.") > + else: > + bb.debug(1, "svn repository has > externals:\n%s" % output) + > + scmdata =3D ud.parm.get("scmdata", "") > + if scmdata =3D=3D "keep": > + tar_flags =3D "" > + else: > + tar_flags =3D "--exclude=3D'.svn'" > =20 > - # tar them up to a defined filename > - runfetchcmd("tar %s -czf %s %s" % (tar_flags, ud.localpath, > ud.path_spec), d, > - cleanup=3D[ud.localpath], workdir=3Dud.pkgdir) > + # tar them up to a defined filename > + runfetchcmd("tar %s -czf %s %s" % (tar_flags, > ud.localpath, ud.path_spec), d, > + cleanup=3D[ud.localpath], workdir=3Dud.pkgdir) > + finally: > + bb.utils.unlockfile(lf) > =20 > def clean(self, ud, d): > """ Clean SVN specific files and dirs """ > diff --git a/bitbake/lib/bb/fetch2/wget.py > b/bitbake/lib/bb/fetch2/wget.py index 8f505b6..725586d 100644 > --- a/bitbake/lib/bb/fetch2/wget.py > +++ b/bitbake/lib/bb/fetch2/wget.py > @@ -1,5 +1,3 @@ > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > """ > BitBake 'Fetch' implementations > =20 > @@ -10,18 +8,7 @@ BitBake build tools. > =20 > # Copyright (C) 2003, 2004 Chris Larson > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > -# > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only > # > # Based on functions from the base bb module, Copyright 2003 Holger > Schurig=20 > @@ -33,11 +20,14 @@ import logging > import errno > import bb > import bb.progress > +import socket > +import http.client > import urllib.request, urllib.parse, urllib.error > from bb.fetch2 import FetchMethod > from bb.fetch2 import FetchError > from bb.fetch2 import logger > from bb.fetch2 import runfetchcmd > +from bb.fetch2 import FetchConnectionCache > from bb.utils import export_proxies > from bs4 import BeautifulSoup > from bs4 import SoupStrainer > @@ -132,10 +122,6 @@ class Wget(FetchMethod): > return True > =20 > def checkstatus(self, fetch, ud, d, try_again=3DTrue): > - import urllib.request, urllib.error, urllib.parse, socket, > http.client > - from urllib.response import addinfourl > - from bb.fetch2 import FetchConnectionCache > - > class HTTPConnectionCache(http.client.HTTPConnection): > if fetch.connection_cache: > def connect(self): > @@ -168,7 +154,7 @@ class Wget(FetchMethod): > """ > host =3D req.host > if not host: > - raise urlllib2.URLError('no host given') > + raise urllib.error.URLError('no host given') > =20 > h =3D http_class(host, timeout=3Dreq.timeout) # will > parse host:port h.set_debuglevel(self._debuglevel) > @@ -185,7 +171,7 @@ class Wget(FetchMethod): > # request. > =20 > # Don't close connection when connection_cache is > enabled, > - if fetch.connection_cache is None:=20 > + if fetch.connection_cache is None: > headers["Connection"] =3D "close" > else: > headers["Connection"] =3D "Keep-Alive" # Works for > HTTP/1.0 @@ -252,7 +238,7 @@ class Wget(FetchMethod): > pass > closed =3D False > =20 > - resp =3D addinfourl(fp_dummy(), r.msg, > req.get_full_url()) > + resp =3D urllib.response.addinfourl(fp_dummy(), r.msg, > req.get_full_url()) resp.code =3D r.status > resp.msg =3D r.reason > =20 > @@ -271,17 +257,18 @@ class Wget(FetchMethod): > fp.read() > fp.close() > =20 > - newheaders =3D dict((k,v) for k,v in > list(req.headers.items()) > - if k.lower() not in > ("content-length", "content-type")) > - return > self.parent.open(urllib.request.Request(req.get_full_url(), > - > headers=3Dnewheaders, > - > origin_req_host=3Dreq.origin_req_host, > - > unverifiable=3DTrue)) > + if req.get_method() !=3D 'GET': > + newheaders =3D dict((k, v) for k, v in > list(req.headers.items()) > + if k.lower() not in > ("content-length", "content-type")) > + return > self.parent.open(urllib.request.Request(req.get_full_url(), > + > headers=3Dnewheaders, > + > origin_req_host=3Dreq.origin_req_host, > + > unverifiable=3DTrue))=20 > - """ > - Some servers (e.g. GitHub archives, hosted on Amazon S3) > return 403 > - Forbidden when they actually mean 405 Method Not Allowed. > - """ > + raise urllib.request.HTTPError(req, code, msg, > headers, None) + > + # Some servers (e.g. GitHub archives, hosted on Amazon > S3) return 403 > + # Forbidden when they actually mean 405 Method Not > Allowed. http_error_403 =3D http_error_405 > =20 > =20 > @@ -292,15 +279,15 @@ class Wget(FetchMethod): > """ > def redirect_request(self, req, fp, code, msg, headers, > newurl): newreq =3D > urllib.request.HTTPRedirectHandler.redirect_request(self, req, fp, > code, msg, headers, newurl) > - newreq.get_method =3D lambda: req.get_method() > + newreq.get_method =3D req.get_method > return newreq > exported_proxies =3D export_proxies(d) > =20 > handlers =3D [FixedHTTPRedirectHandler, HTTPMethodFallback] > - if export_proxies: > + if exported_proxies: > handlers.append(urllib.request.ProxyHandler()) > handlers.append(CacheHTTPHandler()) > - # XXX: Since Python 2.7.9 ssl cert validation is enabled by > default > + # Since Python 2.7.9 ssl cert validation is enabled by > default # see PEP-0476, this causes verification errors on some https > servers # so disable by default. > import ssl > @@ -319,19 +306,19 @@ class Wget(FetchMethod): > '''Adds Basic auth to http request, pass in > login:password as string''' import base64 > encodeuser =3D > base64.b64encode(login_str.encode('utf-8')).decode("utf-8") > - authheader =3D "Basic %s" % encodeuser > + authheader =3D "Basic %s" % encodeuser > r.add_header("Authorization", authheader) > =20 > - if ud.user: > - add_basic_auth(ud.user, r) > + if ud.user and ud.pswd: > + add_basic_auth(ud.user + ':' + ud.pswd, r) > =20 > try: > - import netrc, urllib.parse > + import netrc > n =3D netrc.netrc() > login, unused, password =3D > n.authenticators(urllib.parse.urlparse(uri).hostname) > add_basic_auth("%s:%s" % (login, password), r) except (TypeError, > ImportError, IOError, netrc.NetrcParseError): > - pass > + pass > =20 > with opener.open(r) as response: > pass > @@ -396,18 +383,14 @@ class Wget(FetchMethod): > (oldpn, oldpv, oldsuffix) =3D old > (newpn, newpv, newsuffix) =3D new > =20 > - """ > - Check for a new suffix type that we have never heard of > before > - """ > - if (newsuffix): > + # Check for a new suffix type that we have never heard of > before > + if newsuffix: > m =3D self.suffix_regex_comp.search(newsuffix) > if not m: > bb.warn("%s has a possible unknown suffix: %s" % > (newpn, newsuffix)) return False > =20 > - """ > - Not our package so ignore it > - """ > + # Not our package so ignore it > if oldpn !=3D newpn: > return False > =20 > @@ -473,15 +456,14 @@ class Wget(FetchMethod): > =20 > return "" > =20 > - def _check_latest_version_by_dir(self, dirver, package, > package_regex, > - current_version, ud, d): > + def _check_latest_version_by_dir(self, dirver, package, > package_regex, current_version, ud, d): """ > - Scan every directory in order to get upstream version. > + Scan every directory in order to get upstream version. > """ > version_dir =3D ['', '', ''] > version =3D ['', '', ''] > =20 > - dirver_regex =3D > re.compile("(?P\D*)(?P(\d+[\.\-_])+(\d+))") > + dirver_regex =3D > re.compile(r"(?P\D*)(?P(\d+[\.\-_])+(\d+))") s =3D > dirver_regex.search(dirver) if s: > version_dir[1] =3D s.group('ver') > @@ -541,26 +523,26 @@ class Wget(FetchMethod): > gst-fluendo-mp3 > """ > # match most patterns which uses "-" as separator to version > digits > - pn_prefix1 =3D "[a-zA-Z][a-zA-Z0-9]*([-_][a-zA-Z]\w+)*\+?[-_]" > + pn_prefix1 =3D r"[a-zA-Z][a-zA-Z0-9]*([-_][a-zA-Z]\w+)*\+?[-_]" > # a loose pattern such as for unzip552.tar.gz > - pn_prefix2 =3D "[a-zA-Z]+" > + pn_prefix2 =3D r"[a-zA-Z]+" > # a loose pattern such as for 80325-quicky-0.4.tar.gz > - pn_prefix3 =3D "[0-9]+[-]?[a-zA-Z]+" > + pn_prefix3 =3D r"[0-9]+[-]?[a-zA-Z]+" > # Save the Package Name (pn) Regex for use later > - pn_regex =3D "(%s|%s|%s)" % (pn_prefix1, pn_prefix2, > pn_prefix3) > + pn_regex =3D r"(%s|%s|%s)" % (pn_prefix1, pn_prefix2, > pn_prefix3)=20 > # match version > - pver_regex =3D "(([A-Z]*\d+[a-zA-Z]*[\.\-_]*)+)" > + pver_regex =3D r"(([A-Z]*\d+[a-zA-Z]*[\.\-_]*)+)" > =20 > # match arch > parch_regex =3D "-source|_all_" > =20 > # src.rpm extension was added only for rpm package. Can be > removed if the rpm # packaged will always be considered as having to > be manually upgraded > - psuffix_regex =3D > "(tar\.gz|tgz|tar\.bz2|zip|xz|tar\.lz|rpm|bz2|orig\.tar\.gz|tar\.xz|src\.= tar\.gz|src\.tgz|svnr\d+\.tar\.bz2|stable\.tar\.gz|src\.rpm)" > + psuffix_regex =3D > r"(tar\.gz|tgz|tar\.bz2|zip|xz|tar\.lz|rpm|bz2|orig\.tar\.gz|tar\.xz|src\= .tar\.gz|src\.tgz|svnr\d+\.tar\.bz2|stable\.tar\.gz|src\.rpm)"=20 > # match name, version and archive type of a package > - package_regex_comp =3D > re.compile("(?P%s?\.?v?)(?P%s)(?P%s)?[\.-](?P%s$)" > + package_regex_comp =3D > re.compile(r"(?P%s?\.?v?)(?P%s)(?P%s)?[\.-](?P%s$= )" > % (pn_regex, pver_regex, parch_regex, psuffix_regex)) > self.suffix_regex_comp =3D re.compile(psuffix_regex)=20 > @@ -572,7 +554,7 @@ class Wget(FetchMethod): > version =3D self._parse_path(package_regex_comp, package) > if version: > package_custom_regex_comp =3D re.compile( > - > "(?P%s)(?P%s)(?P%s)?[\.-](?P%s)" % > + > r"(?P%s)(?P%s)(?P%s)?[\.-](?P%s)" % > (re.escape(version[0]), pver_regex, parch_regex, psuffix_regex)) else: > package_custom_regex_comp =3D None > @@ -589,7 +571,7 @@ class Wget(FetchMethod): > current_version =3D ['', d.getVar('PV'), ''] > =20 > """possible to have no version in pkg name, such as > spectrum-fw""" > - if not re.search("\d+", package): > + if not re.search(r"\d+", package): > current_version[1] =3D re.sub('_', '.', current_version[1]) > current_version[1] =3D re.sub('-', '.', current_version[1]) > return (current_version[1], '') > @@ -607,13 +589,13 @@ class Wget(FetchMethod): > =20 > # search for version matches on folders inside the path, > like: # "5.7" in > http://download.gnome.org/sources/${PN}/5.7/${PN}-${PV}.tar.gz > - dirver_regex =3D > re.compile("(?P[^/]*(\d+\.)*\d+([-_]r\d+)*)/") > + dirver_regex =3D > re.compile(r"(?P[^/]*(\d+\.)*\d+([-_]r\d+)*)/") m =3D > dirver_regex.search(path) if m: > pn =3D d.getVar('PN') > dirver =3D m.group('dirver') > =20 > - dirver_pn_regex =3D re.compile("%s\d?" % > (re.escape(pn))) > + dirver_pn_regex =3D re.compile(r"%s\d?" % > (re.escape(pn))) if not dirver_pn_regex.search(dirver): > return (self._check_latest_version_by_dir(dirver, > package, package_regex, current_version, ud, > d), '') diff --git a/bitbake/lib/bb/main.py b/bitbake/lib/bb/main.py > index 732a315..af2880f 100755 > --- a/bitbake/lib/bb/main.py > +++ b/bitbake/lib/bb/main.py > @@ -1,6 +1,3 @@ > -#!/usr/bin/env python > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > # > # Copyright (C) 2003, 2004 Chris Larson > # Copyright (C) 2003, 2004 Phil Blundell > @@ -9,18 +6,8 @@ > # Copyright (C) 2005 ROAD GmbH > # Copyright (C) 2006 Richard Purdie > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > +# SPDX-License-Identifier: GPL-2.0-only > # > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA.=20 > import os > import sys > @@ -267,6 +254,11 @@ class > BitBakeConfigParameters(cookerdata.ConfigParameters): help=3D"Do not > run any setscene tasks. sstate will be ignored and " "everything > needed, built.")=20 > + parser.add_option("", "--skip-setscene", action=3D"store_true", > + dest=3D"skipsetscene", default=3DFalse, > + help=3D"Skip setscene tasks if they would be > executed. Tasks previously " > + "restored from sstate will be kept, > unlike --no-setscene") + > parser.add_option("", "--setscene-only", action=3D"store_true", > dest=3D"setsceneonly", default=3DFalse, > help=3D"Only run setscene tasks, don't run > any real tasks.") @@ -448,7 +440,7 @@ def setup_bitbake(configParams, > configuration, extrafeatures=3DNone): else: > logger.info("Reconnecting to bitbake server...") > if not os.path.exists(sockname): > - print("Previous bitbake instance shutting > down?, waiting to retry...") > + logger.info("Previous bitbake instance > shutting down?, waiting to retry...") i =3D 0 > lock =3D None > # Wait for 5s or until we can get the lock > @@ -460,12 +452,7 @@ def setup_bitbake(configParams, configuration, > extrafeatures=3DNone): bb.utils.unlockfile(lock) > raise > bb.server.process.ProcessTimeout("Bitbake still shutting down as > socket exists but no lock?") if not configParams.server_only: > - try: > - server_connection =3D > bb.server.process.connectProcessServer(sockname, featureset) > - except EOFError: > - # The server may have been shutting down but > not closed the socket yet. If that happened, > - # ignore it. > - pass > + server_connection =3D > bb.server.process.connectProcessServer(sockname, featureset)=20 > if server_connection or configParams.server_only: > break > @@ -475,12 +462,14 @@ def setup_bitbake(configParams, configuration, > extrafeatures=3DNone): if not retries: > raise > retries -=3D 1 > - if isinstance(e, (bb.server.process.ProcessTimeout, > BrokenPipeError)): > - logger.info("Retrying server connection...") > + tryno =3D 8 - retries > + if isinstance(e, (bb.server.process.ProcessTimeout, > BrokenPipeError, EOFError)): > + logger.info("Retrying server connection > (#%d)..." % tryno) else: > - logger.info("Retrying server connection... (%s)" > % traceback.format_exc()) > + logger.info("Retrying server connection (#%d)... > (%s)" % (tryno, traceback.format_exc())) if not retries: > - bb.fatal("Unable to connect to bitbake server, or > start one") > + bb.fatal("Unable to connect to bitbake server, or > start one (server startup failures would be in > bitbake-cookerdaemon.log).") > + bb.event.print_ui_queue() > if retries < 5: > time.sleep(5) > =20 > @@ -502,7 +491,7 @@ def setup_bitbake(configParams, configuration, > extrafeatures=3DNone): def lockBitbake(): > topdir =3D bb.cookerdata.findTopdir() > if not topdir: > - bb.error("Unable to find conf/bblayers.conf or > conf/bitbake.conf. BBAPTH is unset and/or not in a build directory?") > + bb.error("Unable to find conf/bblayers.conf or > conf/bitbake.conf. BBPATH is unset and/or not in a build directory?") > raise BBMainFatal lockfile =3D topdir + "/bitbake.lock" > return topdir, bb.utils.lockfile(lockfile, False, False) > diff --git a/bitbake/lib/bb/methodpool.py > b/bitbake/lib/bb/methodpool.py index 49aed33..51783ac 100644 > --- a/bitbake/lib/bb/methodpool.py > +++ b/bitbake/lib/bb/methodpool.py > @@ -1,21 +1,8 @@ > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > -# > # > # Copyright (C) 2006 Holger Hans Peter Freyther > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > -# > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > +# SPDX-License-Identifier: GPL-2.0-only > # > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA.=20 > from bb.utils import better_compile, better_exec > =20 > diff --git a/bitbake/lib/bb/monitordisk.py > b/bitbake/lib/bb/monitordisk.py index 833cd3d..1a25b00 100644 > --- a/bitbake/lib/bb/monitordisk.py > +++ b/bitbake/lib/bb/monitordisk.py > @@ -1,21 +1,8 @@ > -#!/usr/bin/env python > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > # > # Copyright (C) 2012 Robert Yang > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > +# SPDX-License-Identifier: GPL-2.0-only > # > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA.=20 > import os, logging, re, sys > import bb > @@ -28,16 +15,16 @@ def convertGMK(unit): > =20 > """ Convert the space unit G, M, K, the unit is case-insensitive > """=20 > - unitG =3D re.match('([1-9][0-9]*)[gG]\s?$', unit) > + unitG =3D re.match(r'([1-9][0-9]*)[gG]\s?$', unit) > if unitG: > return int(unitG.group(1)) * (1024 ** 3) > - unitM =3D re.match('([1-9][0-9]*)[mM]\s?$', unit) > + unitM =3D re.match(r'([1-9][0-9]*)[mM]\s?$', unit) > if unitM: > return int(unitM.group(1)) * (1024 ** 2) > - unitK =3D re.match('([1-9][0-9]*)[kK]\s?$', unit) > + unitK =3D re.match(r'([1-9][0-9]*)[kK]\s?$', unit) > if unitK: > return int(unitK.group(1)) * 1024 > - unitN =3D re.match('([1-9][0-9]*)\s?$', unit) > + unitN =3D re.match(r'([1-9][0-9]*)\s?$', unit) > if unitN: > return int(unitN.group(1)) > else: > @@ -83,7 +70,7 @@ def getDiskData(BBDirs, configuration): > for pathSpaceInode in BBDirs.split(): > # The input format is: "dir,space,inode", dir is a must, > space # and inode are optional > - pathSpaceInodeRe =3D re.match('([^,]*),([^,]*),([^,]*),?(.*)', > pathSpaceInode) > + pathSpaceInodeRe =3D > re.match(r'([^,]*),([^,]*),([^,]*),?(.*)', pathSpaceInode) if not > pathSpaceInodeRe: printErr("Invalid value in BB_DISKMON_DIRS: %s" % > pathSpaceInode) return None > @@ -147,7 +134,7 @@ def getInterval(configuration): > else: > # The disk space or inode interval is optional, but it should > # have a correct value once it is specified > - intervalRe =3D re.match('([^,]*),?\s*(.*)', interval) > + intervalRe =3D re.match(r'([^,]*),?\s*(.*)', interval) > if intervalRe: > intervalSpace =3D intervalRe.group(1) > if intervalSpace: > diff --git a/bitbake/lib/bb/msg.py b/bitbake/lib/bb/msg.py > index 96f077e..6216eb3 100644 > --- a/bitbake/lib/bb/msg.py > +++ b/bitbake/lib/bb/msg.py > @@ -1,5 +1,3 @@ > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > """ > BitBake 'msg' implementation > =20 > @@ -9,18 +7,8 @@ Message handling infrastructure for bitbake > =20 > # Copyright (C) 2006 Richard Purdie > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > +# SPDX-License-Identifier: GPL-2.0-only > # > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA.=20 > import sys > import copy > diff --git a/bitbake/lib/bb/namedtuple_with_abc.py > b/bitbake/lib/bb/namedtuple_with_abc.py index 32f2fc6..646aed6 100644 > --- a/bitbake/lib/bb/namedtuple_with_abc.py > +++ b/bitbake/lib/bb/namedtuple_with_abc.py > @@ -1,6 +1,8 @@ > # > http://code.activestate.com/recipes/577629-namedtupleabc-abstract-base-cl= ass-mix-in-for-named/ > -#!/usr/bin/env python # Copyright (c) 2011 Jan Kaliszewski (zuo). > Available under the MIT License. +# > +# SPDX-License-Identifier: MIT > +# > =20 > """ > namedtuple_with_abc.py: > diff --git a/bitbake/lib/bb/parse/__init__.py > b/bitbake/lib/bb/parse/__init__.py index 5397d57..76e180b 100644 > --- a/bitbake/lib/bb/parse/__init__.py > +++ b/bitbake/lib/bb/parse/__init__.py > @@ -9,20 +9,10 @@ File parsers for the BitBake build tools. > # Copyright (C) 2003, 2004 Chris Larson > # Copyright (C) 2003, 2004 Phil Blundell > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > -# > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only > # > # Based on functions from the base bb module, Copyright 2003 Holger > Schurig +# > =20 > handlers =3D [] > =20 > diff --git a/bitbake/lib/bb/parse/ast.py b/bitbake/lib/bb/parse/ast.py > index 6d7c80b..f0911e6 100644 > --- a/bitbake/lib/bb/parse/ast.py > +++ b/bitbake/lib/bb/parse/ast.py > @@ -1,5 +1,3 @@ > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > """ > AbstractSyntaxTree classes for the Bitbake language > """ > @@ -8,19 +6,8 @@ > # Copyright (C) 2003, 2004 Phil Blundell > # Copyright (C) 2009 Holger Hans Peter Freyther > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > +# SPDX-License-Identifier: GPL-2.0-only > # > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA. - > =20 > import re > import string > diff --git a/bitbake/lib/bb/parse/parse_py/BBHandler.py > b/bitbake/lib/bb/parse/parse_py/BBHandler.py index 01fc47e..6f7cf82 > 100644 --- a/bitbake/lib/bb/parse/parse_py/BBHandler.py > +++ b/bitbake/lib/bb/parse/parse_py/BBHandler.py > @@ -1,6 +1,3 @@ > -#!/usr/bin/env python > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > """ > class for handling .bb files > =20 > @@ -12,19 +9,8 @@ > # Copyright (C) 2003, 2004 Chris Larson > # Copyright (C) 2003, 2004 Phil Blundell > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > +# SPDX-License-Identifier: GPL-2.0-only > # > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA. - > =20 > import re, bb, os > import logging > @@ -38,14 +24,15 @@ from .ConfHandler import include, init > # For compatibility > bb.deprecate_import(__name__, "bb.parse", ["vars_from_file"]) > =20 > -__func_start_regexp__ =3D > re.compile( r"(((?Ppython)|(?Pfakeroot))\s*)*(?P[\w\.\-\+\{= \}\$]+)?\s*\(\s*\)\s*{$" ) > -__inherit_regexp__ =3D re.compile( r"inherit\s+(.+)" ) > -__export_func_regexp__ =3D re.compile( r"EXPORT_FUNCTIONS\s+(.+)" ) > -__addtask_regexp__ =3D > re.compile("addtask\s+(?P\w+)\s*((before\s*(?P((.*(?=3Dafte= r))|(.*))))|(after\s*(?P((.*(?=3Dbefore))|(.*)))))*") > -__deltask_regexp__ =3D re.compile("deltask\s+(?P\w+)") > -__addhandler_regexp__ =3D re.compile( r"addhandler\s+(.+)" ) > -__def_regexp__ =3D re.compile( r"def\s+(\w+).*:" ) > -__python_func_regexp__ =3D re.compile( r"(\s+.*)|(^$)|(^#)" ) > +__func_start_regexp__ =3D > re.compile(r"(((?Ppython)|(?Pfakeroot))\s*)*(?P[\w\.\-\+\{\= }\$]+)?\s*\(\s*\)\s*{$" ) > +__inherit_regexp__ =3D re.compile(r"inherit\s+(.+)" ) > +__export_func_regexp__ =3D re.compile(r"EXPORT_FUNCTIONS\s+(.+)" ) > +__addtask_regexp__ =3D > re.compile(r"addtask\s+(?P\w+)\s*((before\s*(?P((.*(?=3Daft= er))|(.*))))|(after\s*(?P((.*(?=3Dbefore))|(.*)))))*") > +__deltask_regexp__ =3D > re.compile(r"deltask\s+(?P\w+)(?P.*)") > +__addhandler_regexp__ =3D re.compile(r"addhandler\s+(.+)" ) > +__def_regexp__ =3D re.compile(r"def\s+(\w+).*:" ) > +__python_func_regexp__ =3D re.compile(r"(\s+.*)|(^$)|(^#)" ) > +__python_tab_regexp__ =3D re.compile(r" *\t") __infunc__ =3D [] > __inpython__ =3D False @@ -160,6 +147,16 @@ def handle(fn, d, include): > def feeder(lineno, s, fn, root, statements, eof=3DFalse): global > __func_start_regexp__, __inherit_regexp__, __export_func_regexp__, > __addtask_regexp__, __addhandler_regexp__, __def_regexp__, > __python_func_regexp__, __inpython__, __infunc__, __body__, bb, > __residue__, __classname__ + > + # Check tabs in python functions: > + # - def py_funcname(): covered by __inpython__ > + # - python(): covered by '__anonymous' =3D=3D __infunc__[0] > + # - python funcname(): covered by __infunc__[3] > + if __inpython__ or (__infunc__ and ('__anonymous' =3D=3D > __infunc__[0] or __infunc__[3])): > + tab =3D __python_tab_regexp__.match(s) > + if tab: > + bb.warn('python should use 4 spaces indentation, but > found tabs in %s, line %s' % (root, lineno)) + > if __infunc__: > if s =3D=3D '}': > __body__.append('') > @@ -225,11 +222,27 @@ def feeder(lineno, s, fn, root, statements, > eof=3DFalse):=20 > m =3D __addtask_regexp__.match(s) > if m: > + if len(m.group().split()) =3D=3D 2: > + # Check and warn for "addtask task1 task2" > + m2 =3D re.match(r"addtask\s+(?P\w+)(?P.*)", > s) > + if m2 and m2.group('ignores'): > + logger.warning('addtask ignored: "%s"' % > m2.group('ignores')) + > + # Check and warn for "addtask task1 before task2 before > task3", the > + # similar to "after" > + taskexpression =3D s.split() > + for word in ('before', 'after'): > + if taskexpression.count(word) > 1: > + logger.warning("addtask contained multiple '%s' > keywords, only one is supported" % word) + > ast.handleAddTask(statements, fn, lineno, m) > return > =20 > m =3D __deltask_regexp__.match(s) > if m: > + # Check and warn "for deltask task1 task2" > + if m.group('ignores'): > + logger.warning('deltask ignored: "%s"' % > m.group('ignores')) ast.handleDelTask(statements, fn, lineno, m) > return > =20 > diff --git a/bitbake/lib/bb/parse/parse_py/ConfHandler.py > b/bitbake/lib/bb/parse/parse_py/ConfHandler.py index 9d3ebe1..2e84b91 > 100644 --- a/bitbake/lib/bb/parse/parse_py/ConfHandler.py > +++ b/bitbake/lib/bb/parse/parse_py/ConfHandler.py > @@ -1,6 +1,3 @@ > -#!/usr/bin/env python > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > """ > class for handling configuration data files > =20 > @@ -11,18 +8,8 @@ > # Copyright (C) 2003, 2004 Chris Larson > # Copyright (C) 2003, 2004 Phil Blundell > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > +# SPDX-License-Identifier: GPL-2.0-only > # > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA.=20 > import errno > import re > @@ -147,7 +134,7 @@ def handle(fn, data, include): > continue > s =3D s.rstrip() > while s[-1] =3D=3D '\\': > - s2 =3D f.readline().strip() > + s2 =3D f.readline().rstrip() > lineno =3D lineno + 1 > if (not s2 or s2 and s2[0] !=3D "#") and s[0] =3D=3D "#" : > bb.fatal("There is a confusing multiline, partially > commented expression on line %s of file %s (%s).\nPlease clarify > whether this is all a comment or should be parsed." % (lineno, fn, > s)) diff --git a/bitbake/lib/bb/parse/parse_py/__init__.py > b/bitbake/lib/bb/parse/parse_py/__init__.py index 3e658d0..f508afa > 100644 --- a/bitbake/lib/bb/parse/parse_py/__init__.py +++ > b/bitbake/lib/bb/parse/parse_py/__init__.py @@ -1,6 +1,3 @@ > -#!/usr/bin/env python -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > """ > BitBake Parsers > =20 > @@ -11,20 +8,10 @@ File parsers for the BitBake build tools. > # Copyright (C) 2003, 2004 Chris Larson > # Copyright (C) 2003, 2004 Phil Blundell > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > -# > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only > # > # Based on functions from the base bb module, Copyright 2003 Holger > Schurig +# > =20 > from __future__ import absolute_import > from . import ConfHandler > diff --git a/bitbake/lib/bb/persist_data.py > b/bitbake/lib/bb/persist_data.py index bef7018..de8f87a 100644 > --- a/bitbake/lib/bb/persist_data.py > +++ b/bitbake/lib/bb/persist_data.py > @@ -8,18 +8,8 @@ currently, providing a key/value store accessed by > 'domain'. # Copyright (C) 2007 Richard Purdie > # Copyright (C) 2010 Chris Larson > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > +# SPDX-License-Identifier: GPL-2.0-only > # > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA.=20 > import collections > import logging > @@ -29,6 +19,7 @@ import warnings > from bb.compat import total_ordering > from collections import Mapping > import sqlite3 > +import contextlib > =20 > sqlversion =3D sqlite3.sqlite_version_info > if sqlversion[0] < 3 or (sqlversion[0] =3D=3D 3 and sqlversion[1] < 3): > @@ -36,84 +27,181 @@ if sqlversion[0] < 3 or (sqlversion[0] =3D=3D 3 and > sqlversion[1] < 3):=20 > =20 > logger =3D logging.getLogger("BitBake.PersistData") > -if hasattr(sqlite3, 'enable_shared_cache'): > - try: > - sqlite3.enable_shared_cache(True) > - except sqlite3.OperationalError: > - pass > - > =20 > @total_ordering > class SQLTable(collections.MutableMapping): > + class _Decorators(object): > + @staticmethod > + def retry(*, reconnect=3DTrue): > + """ > + Decorator that restarts a function if a database locked > sqlite > + exception occurs. If reconnect is True, the database > connection > + will be closed and reopened each time a failure occurs > + """ > + def retry_wrapper(f): > + def wrap_func(self, *args, **kwargs): > + # Reconnect if necessary > + if self.connection is None and reconnect: > + self.reconnect() > + > + count =3D 0 > + while True: > + try: > + return f(self, *args, **kwargs) > + except sqlite3.OperationalError as exc: > + if count < 500 and ('is locked' in > str(exc) or 'locking protocol' in str(exc)): > + count =3D count + 1 > + if reconnect: > + self.reconnect() > + continue > + raise > + return wrap_func > + return retry_wrapper > + > + @staticmethod > + def transaction(f): > + """ > + Decorator that starts a database transaction and creates > a database > + cursor for performing queries. If no exception is > thrown, the > + database results are commited. If an exception occurs, > the database > + is rolled back. In all cases, the cursor is closed after > the > + function ends. > + > + Note that the cursor is passed as an extra argument to > the function > + after `self` and before any of the normal arguments > + """ > + def wrap_func(self, *args, **kwargs): > + # Context manager will COMMIT the database on > success, > + # or ROLLBACK on an exception > + with self.connection: > + # Automatically close the cursor when done > + with > contextlib.closing(self.connection.cursor()) as cursor: > + return f(self, cursor, *args, **kwargs) > + return wrap_func > + > """Object representing a table/domain in the database""" > def __init__(self, cachefile, table): > self.cachefile =3D cachefile > self.table =3D table > - self.cursor =3D connect(self.cachefile) > - > - self._execute("CREATE TABLE IF NOT EXISTS %s(key TEXT, value > TEXT);" > - % table) > - > - def _execute(self, *query): > - """Execute a query, waiting to acquire a lock if necessary""" > - count =3D 0 > - while True: > - try: > - return self.cursor.execute(*query) > - except sqlite3.OperationalError as exc: > - if 'database is locked' in str(exc) and count < 500: > - count =3D count + 1 > + > + self.connection =3D None > + self._execute_single("CREATE TABLE IF NOT EXISTS %s(key TEXT > PRIMARY KEY NOT NULL, value TEXT);" % table) + > + @_Decorators.retry(reconnect=3DFalse) > + @_Decorators.transaction > + def _setup_database(self, cursor): > + cursor.execute("pragma synchronous =3D off;") > + # Enable WAL and keep the autocheckpoint length small (the > default is > + # usually 1000). Persistent caches are usually read-mostly, > so keeping > + # this short will keep readers running quickly > + cursor.execute("pragma journal_mode =3D WAL;") > + cursor.execute("pragma wal_autocheckpoint =3D 100;") > + > + def reconnect(self): > + if self.connection is not None: > + self.connection.close() > + self.connection =3D sqlite3.connect(self.cachefile, timeout=3D5) > + self.connection.text_factory =3D str > + self._setup_database() > + > + @_Decorators.retry() > + @_Decorators.transaction > + def _execute_single(self, cursor, *query): > + """ > + Executes a single query and discards the results. This > correctly closes > + the database cursor when finished > + """ > + cursor.execute(*query) > + > + @_Decorators.retry() > + def _row_iter(self, f, *query): > + """ > + Helper function that returns a row iterator. Each time > __next__ is > + called on the iterator, the provided function is evaluated > to determine > + the return value > + """ > + class CursorIter(object): > + def __init__(self, cursor): > + self.cursor =3D cursor > + > + def __iter__(self): > + return self > + > + def __next__(self): > + row =3D self.cursor.fetchone() > + if row is None: > self.cursor.close() > - self.cursor =3D connect(self.cachefile) > - continue > - raise > + raise StopIteration > + return f(row) > + > + def __enter__(self): > + return self > + > + def __exit__(self, typ, value, traceback): > + self.cursor.close() > + return False > + > + cursor =3D self.connection.cursor() > + try: > + cursor.execute(*query) > + return CursorIter(cursor) > + except: > + cursor.close() > =20 > def __enter__(self): > - self.cursor.__enter__() > + self.connection.__enter__() > return self > =20 > def __exit__(self, *excinfo): > - self.cursor.__exit__(*excinfo) > - > - def __getitem__(self, key): > - data =3D self._execute("SELECT * from %s where key=3D?;" % > - self.table, [key]) > - for row in data: > + self.connection.__exit__(*excinfo) > + > + @_Decorators.retry() > + @_Decorators.transaction > + def __getitem__(self, cursor, key): > + cursor.execute("SELECT * from %s where key=3D?;" % self.table, > [key]) > + row =3D cursor.fetchone() > + if row is not None: > return row[1] > raise KeyError(key) > =20 > - def __delitem__(self, key): > + @_Decorators.retry() > + @_Decorators.transaction > + def __delitem__(self, cursor, key): > if key not in self: > raise KeyError(key) > - self._execute("DELETE from %s where key=3D?;" % self.table, > [key]) > + cursor.execute("DELETE from %s where key=3D?;" % self.table, > [key])=20 > - def __setitem__(self, key, value): > + @_Decorators.retry() > + @_Decorators.transaction > + def __setitem__(self, cursor, key, value): > if not isinstance(key, str): > raise TypeError('Only string keys are supported') > elif not isinstance(value, str): > raise TypeError('Only string values are supported') > =20 > - data =3D self._execute("SELECT * from %s where key=3D?;" % > - self.table, [key]) > - exists =3D len(list(data)) > - if exists: > - self._execute("UPDATE %s SET value=3D? WHERE key=3D?;" % > self.table, > - [value, key]) > + cursor.execute("SELECT * from %s where key=3D?;" % self.table, > [key]) > + row =3D cursor.fetchone() > + if row is not None: > + cursor.execute("UPDATE %s SET value=3D? WHERE key=3D?;" % > self.table, [value, key]) else: > - self._execute("INSERT into %s(key, value) values > (?, ?);" % > - self.table, [key, value]) > - > - def __contains__(self, key): > - return key in set(self) > - > - def __len__(self): > - data =3D self._execute("SELECT COUNT(key) FROM %s;" % > self.table) > - for row in data: > + cursor.execute("INSERT into %s(key, value) values > (?, ?);" % self.table, [key, value]) + > + @_Decorators.retry() > + @_Decorators.transaction > + def __contains__(self, cursor, key): > + cursor.execute('SELECT * from %s where key=3D?;' % self.table, > [key]) > + return cursor.fetchone() is not None > + > + @_Decorators.retry() > + @_Decorators.transaction > + def __len__(self, cursor): > + cursor.execute("SELECT COUNT(key) FROM %s;" % self.table) > + row =3D cursor.fetchone() > + if row is not None: > return row[0] > =20 > def __iter__(self): > - data =3D self._execute("SELECT key FROM %s;" % self.table) > - return (row[0] for row in data) > + return self._row_iter(lambda row: row[0], "SELECT key from > %s;" % self.table)=20 > def __lt__(self, other): > if not isinstance(other, Mapping): > @@ -122,25 +210,27 @@ class SQLTable(collections.MutableMapping): > return len(self) < len(other) > =20 > def get_by_pattern(self, pattern): > - data =3D self._execute("SELECT * FROM %s WHERE key LIKE ?;" % > - self.table, [pattern]) > - return [row[1] for row in data] > + return self._row_iter(lambda row: row[1], "SELECT * FROM %s > WHERE key LIKE ?;" % > + self.table, [pattern]) > =20 > def values(self): > return list(self.itervalues()) > =20 > def itervalues(self): > - data =3D self._execute("SELECT value FROM %s;" % self.table) > - return (row[0] for row in data) > + return self._row_iter(lambda row: row[0], "SELECT value FROM > %s;" % > + self.table) > =20 > def items(self): > return list(self.iteritems()) > =20 > def iteritems(self): > - return self._execute("SELECT * FROM %s;" % self.table) > + return self._row_iter(lambda row: (row[0], row[1]), "SELECT > * FROM %s;" % > + self.table) > =20 > - def clear(self): > - self._execute("DELETE FROM %s;" % self.table) > + @_Decorators.retry() > + @_Decorators.transaction > + def clear(self, cursor): > + cursor.execute("DELETE FROM %s;" % self.table) > =20 > def has_key(self, key): > return key in self > @@ -194,12 +284,6 @@ class PersistData(object): > """ > del self.data[domain][key] > =20 > -def connect(database): > - connection =3D sqlite3.connect(database, timeout=3D5, > isolation_level=3DNone) > - connection.execute("pragma synchronous =3D off;") > - connection.text_factory =3D str > - return connection > - > def persist(domain, d): > """Convenience factory for SQLTable objects based upon > metadata""" import bb.utils > diff --git a/bitbake/lib/bb/process.py b/bitbake/lib/bb/process.py > index e69697c..2dc472a 100644 > --- a/bitbake/lib/bb/process.py > +++ b/bitbake/lib/bb/process.py > @@ -1,3 +1,7 @@ > +# > +# SPDX-License-Identifier: GPL-2.0-only > +# > + > import logging > import signal > import subprocess > diff --git a/bitbake/lib/bb/progress.py b/bitbake/lib/bb/progress.py > index f54d1c7..4022caa 100644 > --- a/bitbake/lib/bb/progress.py > +++ b/bitbake/lib/bb/progress.py > @@ -4,18 +4,8 @@ BitBake progress handling code > =20 > # Copyright (C) 2016 Intel Corporation > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > +# SPDX-License-Identifier: GPL-2.0-only > # > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA.=20 > import sys > import re > @@ -23,6 +13,7 @@ import time > import inspect > import bb.event > import bb.build > +from bb.build import StdoutNoopContextManager > =20 > class ProgressHandler(object): > """ > @@ -37,7 +28,14 @@ class ProgressHandler(object): > if outfile: > self._outfile =3D outfile > else: > - self._outfile =3D sys.stdout > + self._outfile =3D StdoutNoopContextManager() > + > + def __enter__(self): > + self._outfile.__enter__() > + return self > + > + def __exit__(self, *excinfo): > + self._outfile.__exit__(*excinfo) > =20 > def _fire_progress(self, taskprogress, rate=3DNone): > """Internal function to fire the progress event""" > @@ -157,6 +155,12 @@ class MultiStageProgressReporter(object): > self._stage_total =3D None > self._callers =3D [] > =20 > + def __enter__(self): > + return self > + > + def __exit__(self, *excinfo): > + pass > + > def _fire_progress(self, taskprogress): > bb.event.fire(bb.build.TaskProgress(taskprogress), > self._data)=20 > diff --git a/bitbake/lib/bb/providers.py b/bitbake/lib/bb/providers.py > index c2aa98c..f80963c 100644 > --- a/bitbake/lib/bb/providers.py > +++ b/bitbake/lib/bb/providers.py > @@ -1,5 +1,3 @@ > -# ex:ts=3D4:sw=3D4:sts=3D4:et > -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- > # > # Copyright (C) 2003, 2004 Chris Larson > # Copyright (C) 2003, 2004 Phil Blundell > @@ -8,18 +6,8 @@ > # Copyright (C) 2005 ROAD GmbH > # Copyright (C) 2006 Richard Purdie > # > -# This program is free software; you can redistribute it and/or > modify -# it under the terms of the GNU General Public License > version 2 as -# published by the Free Software Foundation. > +# SPDX-License-Identifier: GPL-2.0-only > # > -# This program is distributed in the hope that it will be useful, > -# but WITHOUT ANY WARRANTY; without even the implied warranty of > -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > -# GNU General Public License for more details. > -# > -# You should have received a copy of the GNU General Public License > along -# with this program; if not, write to the Free Software > Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA > 02110-1301 USA.=20 > import re > import logging > @@ -129,7 +117,7 @@ def findPreferredProvider(pn, cfgData, dataCache, > pkg_pn =3D None, item =3D None): preferred_v =3D > cfgData.getVar("PREFERRED_VERSION")=20 > if preferred_v: > - m =3D re.match('(\d+:)*(.*)(_.*)*', preferred_v) > + m =3D re.match(r'(\d+:)*(.*)(_.*)*', preferred_v) > if m: > if m.group(1): > preferred_e =3D m.group(1)[:-1] > @@ -384,7 +372,7 @@ def getRuntimeProviders(dataCache, rdepend): > =20 > # Only search dynamic packages if we can't find anything in > other variables for pattern in dataCache.packages_dynamic: > - pattern =3D pattern.replace('+', "\+") > + pattern =3D pattern.replace(r'+', r"\+") > if pattern in regexp_cache: > regexp =3D regexp_cache[pattern] > else: > diff --git a/bitbake/lib/bb/pysh/builtin.py > b/bitbake/lib/bb/pysh/builtin.py deleted file mode 100644 > index a8814dc..0000000 > --- a/bitbake/lib/bb/pysh/builtin.py > +++ /dev/null > @@ -1,710 +0,0 @@ > -# builtin.py - builtins and utilities definitions for pysh. > -# > -# Copyright 2007 Patrick Mezard > -# > -# This software may be used and distributed according to the terms > -# of the GNU General Public License, incorporated herein by > reference. - > -"""Builtin and internal utilities implementations. > - > -- Beware not to use python interpreter environment as if it were the > shell -environment. For instance, commands working directory must be > explicitely handled -through env['PWD'] instead of relying on python > working directory. -""" > -import errno > -import optparse > -import os > -import re > -import subprocess > -import sys > -import time > - > -def has_subprocess_bug(): > - return getattr(subprocess, 'list2cmdline') and \ > - ( subprocess.list2cmdline(['']) =3D=3D '' or \ > - subprocess.list2cmdline(['foo|bar']) =3D=3D 'foo|bar') > - =20 > -# Detect python bug 1634343: "subprocess swallows empty arguments > under win32" -# > > -# Also detect: "[ 1710802 ] subprocess must escape redirection > characters under win32" -# > > -if has_subprocess_bug(): > - import subprocess_fix > - subprocess.list2cmdline =3D subprocess_fix.list2cmdline > - > -from sherrors import * > - > -class NonExitingParser(optparse.OptionParser): > - """OptionParser default behaviour upon error is to print the > error message and > - exit. Raise a utility error instead. > - """ > - def error(self, msg): > - raise UtilityError(msg) > - > -#-----------------------------------------------------------------------= -------- =20 > -# set special builtin > -#-----------------------------------------------------------------------= -------- =20 > -OPT_SET =3D NonExitingParser(usage=3D"set - set or unset options and > positional parameters") -OPT_SET.add_option( '-f', > action=3D'store_true', dest=3D'has_f', default=3DFalse, > - help=3D'The shell shall disable pathname expansion.') > -OPT_SET.add_option('-e', action=3D'store_true', dest=3D'has_e', > default=3DFalse, > - help=3D"""When this option is on, if a simple command fails for > any of the \ > - reasons listed in Consequences of Shell Errors or returns an > exit status \ > - value >0, and is not part of the compound list following a > while, until, \ > - or if keyword, and is not a part of an AND or OR list, and is > not a \ > - pipeline preceded by the ! reserved word, then the shell shall > immediately \ > - exit.""") > -OPT_SET.add_option('-x', action=3D'store_true', dest=3D'has_x', > default=3DFalse, > - help=3D"""The shell shall write to standard error a trace for each > command \ > - after it expands the command and before it executes it. It is > unspecified \ > - whether the command that turns tracing off is traced.""") > - > -def builtin_set(name, args, interp, env, stdin, stdout, stderr, > debugflags): > - if 'debug-utility' in debugflags: > - print interp.log(' '.join([name, str(args), interp['PWD']]) > + '\n') - > - option, args =3D OPT_SET.parse_args(args) > - env =3D interp.get_env() > - =20 > - if option.has_f: > - env.set_opt('-f') > - if option.has_e: > - env.set_opt('-e') > - if option.has_x: > - env.set_opt('-x') > - return 0 > - =20 > -#-----------------------------------------------------------------------= -------- =20 > -# shift special builtin > -#-----------------------------------------------------------------------= -------- =20 > -def builtin_shift(name, args, interp, env, stdin, stdout, stderr, > debugflags): > - if 'debug-utility' in debugflags: > - print interp.log(' '.join([name, str(args), interp['PWD']]) > + '\n') > - =20 > - params =3D interp.get_env().get_positional_args() > - if args: > - try: > - n =3D int(args[0]) > - if n > len(params): > - raise ValueError() > - except ValueError: > - return 1 > - else: > - n =3D 1 > - =20 > - params[:n] =3D [] > - interp.get_env().set_positional_args(params) > - return 0 > - =20 > -#-----------------------------------------------------------------------= -------- =20 > -# export special builtin > -#-----------------------------------------------------------------------= -------- =20 > -OPT_EXPORT =3D NonExitingParser(usage=3D"set - set or unset options and > positional parameters") -OPT_EXPORT.add_option('-p', > action=3D'store_true', dest=3D'has_p', default=3DFalse) - > -def builtin_export(name, args, interp, env, stdin, stdout, stderr, > debugflags): > - if 'debug-utility' in debugflags: > - print interp.log(' '.join([name, str(args), interp['PWD']]) > + '\n') > - =20 > - option, args =3D OPT_EXPORT.parse_args(args) > - if option.has_p: > - raise NotImplementedError() > - =20 > - for arg in args: > - try: > - name, value =3D arg.split('=3D', 1) > - except ValueError: > - name, value =3D arg, None > - env =3D interp.get_env().export(name, value) > - =20 > - return 0 > - =20 > -#-----------------------------------------------------------------------= -------- =20 > -# return special builtin > -#-----------------------------------------------------------------------= -------- =20 > -def builtin_return(name, args, interp, env, stdin, stdout, stderr, > debugflags): > - if 'debug-utility' in debugflags: > - print interp.log(' '.join([name, str(args), interp['PWD']]) > + '\n') > - res =3D 0 > - if args: > - try: > - res =3D int(args[0]) > - except ValueError: > - res =3D 0 > - if not 0<=3Dres<=3D255: > - res =3D 0 > - =20 > - # BUG: should be last executed command exit code =20 > - raise ReturnSignal(res) > - > -#-----------------------------------------------------------------------= -------- =20 > -# trap special builtin > -#-----------------------------------------------------------------------= -------- =20 > -def builtin_trap(name, args, interp, env, stdin, stdout, stderr, > debugflags): > - if 'debug-utility' in debugflags: > - print interp.log(' '.join([name, str(args), interp['PWD']]) > + '\n') > - if len(args) < 2: > - stderr.write('trap: usage: trap [[arg] signal_spec ...]\n') > - return 2 > - > - action =3D args[0] > - for sig in args[1:]: > - try: > - env.traps[sig] =3D action > - except Exception as e: > - stderr.write('trap: %s\n' % str(e)) > - return 0 > - > -#-----------------------------------------------------------------------= -------- =20 > -# unset special builtin > -#-----------------------------------------------------------------------= -------- > -OPT_UNSET =3D NonExitingParser("unset - unset values and attributes of > variables and functions") -OPT_UNSET.add_option( '-f', > action=3D'store_true', dest=3D'has_f', default=3DFalse) > -OPT_UNSET.add_option( '-v', action=3D'store_true', dest=3D'has_v', > default=3DFalse) - -def builtin_unset(name, args, interp, env, stdin, > stdout, stderr, debugflags): > - if 'debug-utility' in debugflags: > - print interp.log(' '.join([name, str(args), interp['PWD']]) > + '\n') > - =20 > - option, args =3D OPT_UNSET.parse_args(args) > - =20 > - status =3D 0 > - env =3D interp.get_env() > - for arg in args: =20 > - try: > - if option.has_f: > - env.remove_function(arg) > - else: > - del env[arg] > - except KeyError: > - pass > - except VarAssignmentError: > - status =3D 1 > - =20 > - return status > - > -#-----------------------------------------------------------------------= -------- =20 > -# wait special builtin > -#-----------------------------------------------------------------------= -------- =20 > -def builtin_wait(name, args, interp, env, stdin, stdout, stderr, > debugflags): > - if 'debug-utility' in debugflags: > - print interp.log(' '.join([name, str(args), interp['PWD']]) > + '\n') - > - return interp.wait([int(arg) for arg in args]) > - > -#-----------------------------------------------------------------------= -------- =20 > -# cat utility > -#-----------------------------------------------------------------------= -------- > -def utility_cat(name, args, interp, env, stdin, stdout, stderr, > debugflags): > - if 'debug-utility' in debugflags: > - print interp.log(' '.join([name, str(args), interp['PWD']]) > + '\n') - > - if not args: > - args =3D ['-'] > - > - status =3D 0 > - for arg in args: > - if arg =3D=3D '-': > - data =3D stdin.read() > - else: > - path =3D os.path.join(env['PWD'], arg) > - try: > - f =3D file(path, 'rb') > - try: > - data =3D f.read() > - finally: > - f.close() > - except IOError as e: > - if e.errno !=3D errno.ENOENT: > - raise > - status =3D 1 > - continue > - stdout.write(data) > - stdout.flush() > - return status > - =20 > -#-----------------------------------------------------------------------= -------- =20 > -# cd utility > -#-----------------------------------------------------------------------= -------- =20 > -OPT_CD =3D NonExitingParser("cd - change the working directory") > - > -def utility_cd(name, args, interp, env, stdin, stdout, stderr, > debugflags): > - if 'debug-utility' in debugflags: > - print interp.log(' '.join([name, str(args), interp['PWD']]) > + '\n') - > - option, args =3D OPT_CD.parse_args(args) > - env =3D interp.get_env() > - =20 > - directory =3D None > - printdir =3D False > - if not args: > - home =3D env.get('HOME') > - if home: > - # Unspecified, do nothing > - return 0 > - else: > - directory =3D home > - elif len(args)=3D=3D1: > - directory =3D args[0] > - if directory=3D=3D'-': > - if 'OLDPWD' not in env: > - raise UtilityError("OLDPWD not set") > - printdir =3D True > - directory =3D env['OLDPWD'] > - else: > - raise UtilityError("too many arguments") > - =20 > - curpath =3D None > - # Absolute directories will be handled correctly by the > os.path.join call. > - if not directory.startswith('.') and not > directory.startswith('..'): > - cdpaths =3D env.get('CDPATH', '.').split(';') > - for cdpath in cdpaths: > - p =3D os.path.join(cdpath, directory) > - if os.path.isdir(p): > - curpath =3D p > - break > - =20 > - if curpath is None: > - curpath =3D directory > - curpath =3D os.path.join(env['PWD'], directory) > - > - env['OLDPWD'] =3D env['PWD'] > - env['PWD'] =3D curpath > - if printdir: > - stdout.write('%s\n' % curpath) > - return 0 > - > -#-----------------------------------------------------------------------= -------- =20 > -# colon utility > -#-----------------------------------------------------------------------= -------- =20 > -def utility_colon(name, args, interp, env, stdin, stdout, stderr, > debugflags): > - if 'debug-utility' in debugflags: > - print interp.log(' '.join([name, str(args), interp['PWD']]) > + '\n') > - return 0 > - =20 > -#-----------------------------------------------------------------------= -------- =20 > -# echo utility > -#-----------------------------------------------------------------------= -------- =20 > -def utility_echo(name, args, interp, env, stdin, stdout, stderr, > debugflags): > - if 'debug-utility' in debugflags: > - print interp.log(' '.join([name, str(args), interp['PWD']]) > + '\n') > - =20 > - # Echo only takes arguments, no options. Use printf if you need > fancy stuff. > - output =3D ' '.join(args) + '\n' > - stdout.write(output) > - stdout.flush() > - return 0 > - =20 > -#-----------------------------------------------------------------------= -------- =20 > -# egrep utility > -#-----------------------------------------------------------------------= -------- > -# egrep is usually a shell script. > -# Unfortunately, pysh does not support shell scripts *with > arguments* right now, -# so the redirection is implemented here, > assuming grep is available. -def utility_egrep(name, args, interp, > env, stdin, stdout, stderr, debugflags): > - if 'debug-utility' in debugflags: > - print interp.log(' '.join([name, str(args), interp['PWD']]) > + '\n') > - =20 > - return run_command('grep', ['-E'] + args, interp, env, stdin, > stdout,=20 > - stderr, debugflags) > - =20 > -#-----------------------------------------------------------------------= -------- =20 > -# env utility > -#-----------------------------------------------------------------------= -------- =20 > -def utility_env(name, args, interp, env, stdin, stdout, stderr, > debugflags): > - if 'debug-utility' in debugflags: > - print interp.log(' '.join([name, str(args), interp['PWD']]) > + '\n') > - =20 > - if args and args[0]=3D=3D'-i': > - raise NotImplementedError('env: -i option is not > implemented') > - =20 > - i =3D 0 > - for arg in args: > - if '=3D' not in arg: > - break > - # Update the current environment > - name, value =3D arg.split('=3D', 1) > - env[name] =3D value > - i +=3D 1 > - =20 > - if args[i:]: > - # Find then execute the specified interpreter > - utility =3D env.find_in_path(args[i]) > - if not utility: > - return 127 > - args[i:i+1] =3D utility > - name =3D args[i] > - args =3D args[i+1:] > - try: > - return run_command(name, args, interp, env, stdin, > stdout, stderr,=20 > - debugflags) > - except UtilityError: > - stderr.write('env: failed to execute %s' % ' > '.join([name]+args)) > - return 126 =20 > - else: > - for pair in env.get_variables().iteritems(): > - stdout.write('%s=3D%s\n' % pair) > - return 0 > - =20 > -#-----------------------------------------------------------------------= -------- =20 > -# exit utility > -#-----------------------------------------------------------------------= -------- > -def utility_exit(name, args, interp, env, stdin, stdout, stderr, > debugflags): > - if 'debug-utility' in debugflags: > - print interp.log(' '.join([name, str(args), interp['PWD']]) > + '\n') > - =20 > - res =3D None > - if args: > - try: > - res =3D int(args[0]) > - except ValueError: > - res =3D None > - if not 0<=3Dres<=3D255: > - res =3D None > - =20 > - if res is None: > - # BUG: should be last executed command exit code > - res =3D 0 > - =20 > - raise ExitSignal(res) > - > -#-----------------------------------------------------------------------= -------- =20 > -# fgrep utility > -#-----------------------------------------------------------------------= -------- > -# see egrep > -def utility_fgrep(name, args, interp, env, stdin, stdout, stderr, > debugflags): > - if 'debug-utility' in debugflags: > - print interp.log(' '.join([name, str(args), interp['PWD']]) > + '\n') > - =20 > - return run_command('grep', ['-F'] + args, interp, env, stdin, > stdout,=20 > - stderr, debugflags) > - > -#-----------------------------------------------------------------------= -------- =20 > -# gunzip utility > -#-----------------------------------------------------------------------= -------- > -# see egrep > -def utility_gunzip(name, args, interp, env, stdin, stdout, stderr, > debugflags): > - if 'debug-utility' in debugflags: > - print interp.log(' '.join([name, str(args), interp['PWD']]) > + '\n') > - =20 > - return run_command('gzip', ['-d'] + args, interp, env, stdin, > stdout,=20 > - stderr, debugflags) > - =20 > -#-----------------------------------------------------------------------= -------- =20 > -# kill utility > -#-----------------------------------------------------------------------= -------- > -def utility_kill(name, args, interp, env, stdin, stdout, stderr, > debugflags): > - if 'debug-utility' in debugflags: > - print interp.log(' '.join([name, str(args), interp['PWD']]) > + '\n') > - =20 > - for arg in args: > - pid =3D int(arg) > - status =3D subprocess.call(['pskill', '/T', str(pid)], > - shell=3DTrue, > - stdout=3Dsubprocess.PIPE, > - stderr=3Dsubprocess.PIPE) > - # pskill is asynchronous, hence the stupid polling loop > - while 1: > - p =3D subprocess.Popen(['pslist', str(pid)], > - shell=3DTrue, > - stdout=3Dsubprocess.PIPE, > - stderr=3Dsubprocess.STDOUT) > - output =3D p.communicate()[0] > - if ('process %d was not' % pid) in output: > - break > - time.sleep(1) > - return status > - =20 > -#-----------------------------------------------------------------------= -------- =20 > -# mkdir utility > -#-----------------------------------------------------------------------= -------- > -OPT_MKDIR =3D NonExitingParser("mkdir - make directories.") > -OPT_MKDIR.add_option('-p', action=3D'store_true', dest=3D'has_p', > default=3DFalse) - > -def utility_mkdir(name, args, interp, env, stdin, stdout, stderr, > debugflags): > - if 'debug-utility' in debugflags: > - print interp.log(' '.join([name, str(args), interp['PWD']]) > + '\n') > - =20 > - # TODO: implement umask > - # TODO: implement proper utility error report > - option, args =3D OPT_MKDIR.parse_args(args) > - for arg in args: > - path =3D os.path.join(env['PWD'], arg) > - if option.has_p: > - try: > - os.makedirs(path) > - except IOError as e: > - if e.errno !=3D errno.EEXIST: > - raise > - else: =20 > - os.mkdir(path) > - return 0 > - > -#-----------------------------------------------------------------------= -------- =20 > -# netstat utility > -#-----------------------------------------------------------------------= -------- > -def utility_netstat(name, args, interp, env, stdin, stdout, stderr, > debugflags): > - # Do you really expect me to implement netstat ? > - # This empty form is enough for Mercurial tests since it's > - # supposed to generate nothing upon success. Faking this test > - # is not a big deal either. > - if 'debug-utility' in debugflags: > - print interp.log(' '.join([name, str(args), interp['PWD']]) > + '\n') > - return 0 > - =20 > -#-----------------------------------------------------------------------= -------- =20 > -# pwd utility > -#-----------------------------------------------------------------------= -------- =20 > -OPT_PWD =3D NonExitingParser("pwd - return working directory name") > -OPT_PWD.add_option('-L', action=3D'store_true', dest=3D'has_L', > default=3DTrue, > - help=3D"""If the PWD environment variable contains an absolute > pathname of \ > - the current directory that does not contain the filenames dot or > dot-dot, \ > - pwd shall write this pathname to standard output. Otherwise, the > -L option \ > - shall behave as the -P option.""") > -OPT_PWD.add_option('-P', action=3D'store_true', dest=3D'has_L', > default=3DFalse, > - help=3D"""The absolute pathname written shall not contain > filenames that, in \ > - the context of the pathname, refer to files of type symbolic > link.""") - > -def utility_pwd(name, args, interp, env, stdin, stdout, stderr, > debugflags): > - if 'debug-utility' in debugflags: > - print interp.log(' '.join([name, str(args), interp['PWD']]) > + '\n') - > - option, args =3D OPT_PWD.parse_args(args) =20 > - stdout.write('%s\n' % env['PWD']) > - return 0 > - =20 > -#-----------------------------------------------------------------------= -------- =20 > -# printf utility > -#-----------------------------------------------------------------------= -------- > -RE_UNESCAPE =3D re.compile(r'(\\x[a-zA-Z0-9]{2}|\\[0-7]{1,3}|\\.)') > - > -def utility_printf(name, args, interp, env, stdin, stdout, stderr, > debugflags): > - if 'debug-utility' in debugflags: > - print interp.log(' '.join([name, str(args), interp['PWD']]) > + '\n') > - =20 > - def replace(m): > - assert m.group() > - g =3D m.group()[1:] > - if g.startswith('x'): > - return chr(int(g[1:], 16)) > - if len(g) <=3D 3 and len([c for c in g if c in '01234567']) =3D= =3D > len(g): > - # Yay, an octal number > - return chr(int(g, 8)) > - return { > - 'a': '\a', > - 'b': '\b', > - 'f': '\f', > - 'n': '\n', > - 'r': '\r', > - 't': '\t', > - 'v': '\v', > - '\\': '\\', > - }.get(g) > - =20 > - # Convert escape sequences > - format =3D re.sub(RE_UNESCAPE, replace, args[0]) > - stdout.write(format % tuple(args[1:])) > - return 0 > - =20 > -#-----------------------------------------------------------------------= -------- =20 > -# true utility > -#-----------------------------------------------------------------------= -------- > -def utility_true(name, args, interp, env, stdin, stdout, stderr, > debugflags): > - if 'debug-utility' in debugflags: > - print interp.log(' '.join([name, str(args), interp['PWD']]) > + '\n') > - return 0 > - > -#-----------------------------------------------------------------------= -------- =20 > -# sed utility > -#-----------------------------------------------------------------------= -------- > -RE_SED =3D re.compile(r'^s(.).*\1[a-zA-Z]*$') > - > -# cygwin sed fails with some expressions when they do not end with a > single space. -# see unit tests for details. Interestingly, the same > expressions works perfectly -# in cygwin shell. > -def utility_sed(name, args, interp, env, stdin, stdout, stderr, > debugflags): > - if 'debug-utility' in debugflags: > - print interp.log(' '.join([name, str(args), interp['PWD']]) > + '\n') > - =20 > - # Scan pattern arguments and append a space if necessary > - for i in range(len(args)): > - if not RE_SED.search(args[i]): > - continue > - args[i] =3D args[i] + ' ' > - > - return run_command(name, args, interp, env, stdin, stdout,=20 > - stderr, debugflags) > - > -#-----------------------------------------------------------------------= -------- =20 > -# sleep utility > -#-----------------------------------------------------------------------= -------- > -def utility_sleep(name, args, interp, env, stdin, stdout, stderr, > debugflags): > - if 'debug-utility' in debugflags: > - print interp.log(' '.join([name, str(args), interp['PWD']]) > + '\n') > - time.sleep(int(args[0])) > - return 0 > - =20 > -#-----------------------------------------------------------------------= -------- =20 > -# sort utility > -#-----------------------------------------------------------------------= -------- > -OPT_SORT =3D NonExitingParser("sort - sort, merge, or sequence check > text files") - > -def utility_sort(name, args, interp, env, stdin, stdout, stderr, > debugflags): - > - def sort(path): > - if path =3D=3D '-': > - lines =3D stdin.readlines() > - else: > - try: > - f =3D file(path) > - try: > - lines =3D f.readlines() > - finally: > - f.close() > - except IOError as e: > - stderr.write(str(e) + '\n') > - return 1 > - =20 > - if lines and lines[-1][-1]!=3D'\n': > - lines[-1] =3D lines[-1] + '\n' > - return lines > - =20 > - if 'debug-utility' in debugflags: > - print interp.log(' '.join([name, str(args), interp['PWD']]) > + '\n') > - =20 > - option, args =3D OPT_SORT.parse_args(args) > - alllines =3D [] > - =20 > - if len(args)<=3D0: > - args +=3D ['-'] > - =20 > - # Load all files lines > - curdir =3D os.getcwd() > - try: > - os.chdir(env['PWD']) > - for path in args: > - alllines +=3D sort(path) > - finally: > - os.chdir(curdir) > - =20 > - alllines.sort() > - for line in alllines: > - stdout.write(line) > - return 0 > - =20 > -#-----------------------------------------------------------------------= -------- > -# hg utility > -#-----------------------------------------------------------------------= -------- > - > -hgcommands =3D [ > - 'add', > - 'addremove', > - 'commit', 'ci', > - 'debugrename', > - 'debugwalk', > - 'falabala', # Dummy command used in a mercurial test > - 'incoming', > - 'locate', > - 'pull', > - 'push', > - 'qinit', > - 'remove', 'rm', > - 'rename', 'mv', > - 'revert', =20 > - 'showconfig', > - 'status', 'st', > - 'strip', > - ] > - > -def rewriteslashes(name, args): > - # Several hg commands output file paths, rewrite the separators > - if len(args) > 1 and name.lower().endswith('python') \ > - and args[0].endswith('hg'): > - for cmd in hgcommands: > - if cmd in args[1:]: > - return True > - =20 > - # svn output contains many paths with OS specific separators. > - # Normalize these to unix paths. > - base =3D os.path.basename(name) > - if base.startswith('svn'): > - return True > - =20 > - return False > - > -def rewritehg(output): > - if not output: > - return output > - # Rewrite os specific messages > - output =3D output.replace(': The system cannot find the file > specified', > - ': No such file or directory') > - output =3D re.sub(': Access is denied.*$', ': Permission denied', > output) > - output =3D output.replace(': No connection could be made because > the target machine actively refused it', > - ': Connection refused') > - return output > - =20 > - > -def run_command(name, args, interp, env, stdin, stdout, > - stderr, debugflags): > - # Execute the command > - if 'debug-utility' in debugflags: > - print interp.log(' '.join([name, str(args), interp['PWD']]) > + '\n') - > - hgbin =3D interp.options().hgbinary > - ishg =3D hgbin and ('hg' in name or args and 'hg' in args[0]) > - unixoutput =3D 'cygwin' in name or ishg > - =20 > - exec_env =3D env.get_variables() =20 > - try: > - # BUG: comparing file descriptor is clearly not a reliable > way to tell > - # whether they point on the same underlying object. But in > pysh limited > - # scope this is usually right, we do not expect complicated > redirections > - # besides usual 2>&1. > - # Still there is one case we have but cannot deal with is > when stdout > - # and stderr are redirected *by pysh caller*. This the > reason for the > - # --redirect pysh() option. > - # Now, we want to know they are the same because we > sometimes need to=20 > - # transform the command output, mostly remove CR-LF to > ensure that > - # command output is unix-like. Cygwin utilies are a special > case because > - # they explicitely set their output streams to binary mode, > so we have > - # nothing to do. For all others commands, we have to guess > whether they > - # are sending text data, in which case the transformation > must be done. > - # Again, the NUL character test is unreliable but should be > enough for > - # hg tests. > - redirected =3D stdout.fileno()=3D=3Dstderr.fileno() > - if not redirected: > - p =3D subprocess.Popen([name] + args, cwd=3Denv['PWD'], > env=3Dexec_env,=20 > - stdin=3Dstdin, stdout=3Dsubprocess.PIPE, > stderr=3Dsubprocess.PIPE) > - else: > - p =3D subprocess.Popen([name] + args, cwd=3Denv['PWD'], > env=3Dexec_env,=20 > - stdin=3Dstdin, stdout=3Dsubprocess.PIPE, > stderr=3Dsubprocess.STDOUT) > - out, err =3D p.communicate() > - except WindowsError as e: > - raise UtilityError(str(e)) > - > - if not unixoutput: > - def encode(s): > - if '\0' in s: > - return s > - return s.replace('\r\n', '\n') > - else: > - encode =3D lambda s: s > - > - if rewriteslashes(name, args): > - encode1_ =3D encode > - def encode(s): > - s =3D encode1_(s) > - s =3D s.replace('\\\\', '\\') > - s =3D s.replace('\\', '/') > - return s > - > - if ishg: > - encode2_ =3D encode > - def encode(s): > - return rewritehg(encode2_(s)) > - =20 > - stdout.write(encode(out)) > - if not redirected: > - stderr.write(encode(err)) > - return p.returncode > - =20 > diff --git a/bitbake/lib/bb/pysh/interp.py > b/bitbake/lib/bb/pysh/interp.py deleted file mode 100644 > index d14ecf3..0000000 > --- a/bitbake/lib/bb/pysh/interp.py > +++ /dev/null > @@ -1,1367 +0,0 @@ > -# interp.py - shell interpreter for pysh. > -# > -# Copyright 2007 Patrick Mezard > -# > -# This software may be used and distributed according to the terms > -# of the GNU General Public License, incorporated herein by > reference. - > -"""Implement the shell interpreter. > - > -Most references are made to "The Open Group Base Specifications > Issue 6". > - > -""" -# TODO: document the fact input streams must implement fileno() > so Popen will work correctly. -# it requires non-stdin stream to be > implemented as files. Still to be tested... -# DOC: pathsep is used > in PATH instead of ':'. Clearly, there are path syntax issues here. > -# TODO: stop command execution upon error. -# TODO: sort out the > filename/io_number mess. It should be possible to use filenames only. > -# TODO: review subshell implementation -# TODO: test environment > cloning for non-special builtins -# TODO: set -x should not rebuild > commands from tokens, assignments/redirections are lost -# TODO: unit > test for variable assignment -# TODO: test error management wrt error > type/utility type -# TODO: test for binary output everywhere > -# BUG: debug-parsing does not pass log file to PLY. Maybe a PLY > upgrade is necessary. -import base64 > -import cPickle as pickle > -import errno > -import glob > -import os > -import re > -import subprocess > -import sys > -import tempfile > - > -try: > - s =3D set() > - del s > -except NameError: > - from Set import Set as set > - > -import builtin > -from sherrors import * > -import pyshlex > -import pyshyacc > - > -def mappend(func, *args, **kargs): > - """Like map but assume func returns a list. Returned lists are > merged into > - a single one. > - """ > - return reduce(lambda a,b: a+b, map(func, *args, **kargs), []) > - > -class FileWrapper: > - """File object wrapper to ease debugging. > - =20 > - Allow mode checking and implement file duplication through a > simple=20 > - reference counting scheme. Not sure the latter is really useful > since > - only real file descriptors can be used. > - """ > - def __init__(self, mode, file, close=3DTrue): > - if mode not in ('r', 'w', 'a'): > - raise IOError('invalid mode: %s' % mode) > - self._mode =3D mode > - self._close =3D close > - if isinstance(file, FileWrapper): > - if file._refcount[0] <=3D 0: > - raise IOError(0, 'Error') > - self._refcount =3D file._refcount > - self._refcount[0] +=3D 1 > - self._file =3D file._file > - else: > - self._refcount =3D [1] > - self._file =3D file > - =20 > - def dup(self): > - return FileWrapper(self._mode, self, self._close) > - =20 > - def fileno(self): > - """fileno() should be only necessary for input streams.""" > - return self._file.fileno() > - =20 > - def read(self, size=3D-1): > - if self._mode!=3D'r': > - raise IOError(0, 'Error') > - return self._file.read(size) > - =20 > - def readlines(self, *args, **kwargs): > - return self._file.readlines(*args, **kwargs) > - =20 > - def write(self, s): > - if self._mode not in ('w', 'a'): > - raise IOError(0, 'Error') > - return self._file.write(s) > - =20 > - def flush(self): > - self._file.flush() > - =20 > - def close(self): =20 > - if not self._refcount: > - return > - assert self._refcount[0] > 0 > - =20 > - self._refcount[0] -=3D 1 =20 > - if self._refcount[0] =3D=3D 0: > - self._mode =3D 'c' > - if self._close: > - self._file.close() > - self._refcount =3D None > - =20 > - def mode(self): > - return self._mode > - > - def __getattr__(self, name): > - if name =3D=3D 'name': > - self.name =3D getattr(self._file, name) > - return self.name > - else: > - raise AttributeError(name) > - > - def __del__(self): > - self.close() > - =20 > - =20 > -def win32_open_devnull(mode): > - return open('NUL', mode) > - =20 > - =20 > -class Redirections: > - """Stores open files and their mapping to pseudo-sh file > descriptor. > - """ > - # BUG: redirections are not handled correctly: 1>&3 2>&3 3>&4 > does=20 > - # not make 1 to redirect to 4 > - def __init__(self, stdin=3DNone, stdout=3DNone, stderr=3DNone): > - self._descriptors =3D {} > - if stdin is not None: > - self._add_descriptor(0, stdin) > - if stdout is not None: > - self._add_descriptor(1, stdout) > - if stderr is not None: > - self._add_descriptor(2, stderr) > - =20 > - def add_here_document(self, interp, name, content, > io_number=3DNone): > - if io_number is None: > - io_number =3D 0 > - =20 > - if name=3D=3Dpyshlex.unquote_wordtree(name): > - content =3D interp.expand_here_document(('TOKEN', content)) > - =20 > - # Write document content in a temporary file > - tmp =3D tempfile.TemporaryFile() > - try: > - tmp.write(content) > - tmp.flush() > - tmp.seek(0) > - self._add_descriptor(io_number, FileWrapper('r', tmp)) > - except: > - tmp.close() > - raise =20 > - =20 > - def add(self, interp, op, filename, io_number=3DNone): > - if op not in ('<', '>', '>|', '>>', '>&'): > - # TODO: add descriptor duplication and > here_documents =20 > - raise RedirectionError('Unsupported redirection operator > "%s"' % op) > - =20 > - if io_number is not None: > - io_number =3D int(io_number) > - =20 > - if (op =3D=3D '>&' and filename.isdigit()) or filename=3D=3D'-': > - # No expansion for file descriptors, quote them if you > want a filename > - fullname =3D filename > - else: > - if filename.startswith('/'): > - # TODO: win32 kludge > - if filename=3D=3D'/dev/null': > - fullname =3D 'NUL' > - else: > - # TODO: handle absolute pathnames, they are > unlikely to exist on the > - # current platform (win32 for instance). > - raise NotImplementedError() > - else: > - fullname =3D interp.expand_redirection(('TOKEN', > filename)) > - if not fullname: > - raise RedirectionError('%s: ambiguous redirect' > % filename) > - # Build absolute path based on PWD > - fullname =3D os.path.join(interp.get_env()['PWD'], > fullname) > - =20 > - if op=3D=3D'<': > - return self._add_input_redirection(interp, fullname, > io_number) > - elif op in ('>', '>|'): > - clobber =3D ('>|'=3D=3Dop) > - return self._add_output_redirection(interp, fullname, > io_number, clobber) > - elif op=3D=3D'>>': > - return self._add_output_appending(interp, fullname, > io_number) > - elif op=3D=3D'>&': > - return self._dup_output_descriptor(fullname, io_number) > - =20 > - def close(self): > - if self._descriptors is not None: > - for desc in self._descriptors.itervalues(): > - desc.flush() > - desc.close() > - self._descriptors =3D None > - =20 > - def stdin(self): > - return self._descriptors[0] > - =20 > - def stdout(self): > - return self._descriptors[1]=20 > - =20 > - def stderr(self): > - return self._descriptors[2]=20 > - =20 > - def clone(self): > - clone =3D Redirections() > - for desc, fileobj in self._descriptors.iteritems(): > - clone._descriptors[desc] =3D fileobj.dup() > - return clone > - =20 > - def _add_output_redirection(self, interp, filename, io_number, > clobber): =20 > - if io_number is None: > - # io_number default to standard output > - io_number =3D 1 > - =20 > - if not clobber and interp.get_env().has_opt('-C') and > os.path.isfile(filename): > - # File already exist in no-clobber mode, bail out > - raise RedirectionError('File "%s" already exists' % > filename) > - =20 > - # Open and register > - self._add_file_descriptor(io_number, filename, 'w') > - =20 > - def _add_output_appending(self, interp, filename, io_number): =20 > - if io_number is None: > - io_number =3D 1 > - self._add_file_descriptor(io_number, filename, 'a') > - =20 > - def _add_input_redirection(self, interp, filename, io_number): > - if io_number is None: > - io_number =3D 0 > - self._add_file_descriptor(io_number, filename, 'r') > - =20 > - def _add_file_descriptor(self, io_number, filename, mode): =20 > - try: =20 > - if filename.startswith('/'): > - if filename=3D=3D'/dev/null': > - f =3D win32_open_devnull(mode+'b') > - else: > - # TODO: handle absolute pathnames, they are > unlikely to exist on the > - # current platform (win32 for instance). > - raise NotImplementedError('cannot open absolute > path %s' % repr(filename)) > - else: > - f =3D file(filename, mode+'b') > - except IOError as e: > - raise RedirectionError(str(e)) > - =20 > - wrapper =3D None > - try: > - wrapper =3D FileWrapper(mode, f) > - f =3D None > - self._add_descriptor(io_number, wrapper) > - except: > - if f: f.close() > - if wrapper: wrapper.close() > - raise > - =20 > - def _dup_output_descriptor(self, source_fd, dest_fd): > - if source_fd is None: > - source_fd =3D 1 > - self._dup_file_descriptor(source_fd, dest_fd, 'w') > - =20 > - def _dup_file_descriptor(self, source_fd, dest_fd, mode): > - source_fd =3D int(source_fd) > - if source_fd not in self._descriptors: > - raise RedirectionError('"%s" is not a valid file > descriptor' % str(source_fd)) > - source =3D self._descriptors[source_fd] > - =20 > - if source.mode()!=3Dmode: > - raise RedirectionError('Descriptor %s cannot be > duplicated in mode "%s"' % (str(source), mode)) > - =20 > - if dest_fd=3D=3D'-': > - # Close the source descriptor > - del self._descriptors[source_fd] > - source.close() > - else: > - dest_fd =3D int(dest_fd) > - if dest_fd not in self._descriptors: > - raise RedirectionError('Cannot replace file > descriptor %s' % str(dest_fd)) > - =20 > - dest =3D self._descriptors[dest_fd] > - if dest.mode()!=3Dmode: > - raise RedirectionError('Descriptor %s cannot be > cannot be redirected in mode "%s"' % (str(dest), mode)) > - =20 > - self._descriptors[dest_fd] =3D source.dup() > - dest.close() =20 > - =20 > - def _add_descriptor(self, io_number, file): > - io_number =3D int(io_number) > - =20 > - if io_number in self._descriptors: > - # Close the current descriptor > - d =3D self._descriptors[io_number] > - del self._descriptors[io_number] > - d.close() > - =20 > - self._descriptors[io_number] =3D file > - > - def __str__(self): > - names =3D [('%d=3D%r' % (k, getattr(v, 'name', None))) for k,v > - in self._descriptors.iteritems()] > - names =3D ','.join(names) > - return 'Redirections(%s)' % names > - > - def __del__(self): > - self.close() > - =20 > -def cygwin_to_windows_path(path): > - """Turn /cygdrive/c/foo into c:/foo, or return path if it > - is not a cygwin path. > - """ > - if not path.startswith('/cygdrive/'): > - return path > - path =3D path[len('/cygdrive/'):] > - path =3D path[:1] + ':' + path[1:] > - return path > - =20 > -def win32_to_unix_path(path): > - if path is not None: > - path =3D path.replace('\\', '/') > - return path =20 > - =20 > -_RE_SHEBANG =3D re.compile(r'^\#!\s?([^\s]+)(?:\s([^\s]+))?') > -_SHEBANG_CMDS =3D { > - '/usr/bin/env': 'env', > - '/bin/sh': 'pysh', > - 'python': 'python', > -} > - =20 > -def resolve_shebang(path, ignoreshell=3DFalse): > - """Return a list of arguments as shebang interpreter call or an > empty list > - if path does not refer to an executable script. > - See . > - =20 > - ignoreshell - set to True to ignore sh shebangs. Return an empty > list instead. > - """ > - try: > - f =3D file(path) > - try: > - # At most 80 characters in the first line > - header =3D f.read(80).splitlines()[0] > - finally: > - f.close() > - =20 > - m =3D _RE_SHEBANG.search(header) > - if not m: > - return [] > - cmd, arg =3D m.group(1,2) > - if os.path.isfile(cmd): > - # Keep this one, the hg script for instance contains a > weird windows > - # shebang referencing the current python install. > - cmdfile =3D os.path.basename(cmd).lower() > - if cmdfile =3D=3D 'python.exe': > - cmd =3D 'python' > - pass > - elif cmd not in _SHEBANG_CMDS: > - raise CommandNotFound('Unknown interpreter "%s" > referenced in '\ > - 'shebang' % header) > - cmd =3D _SHEBANG_CMDS.get(cmd) > - if cmd is None or (ignoreshell and cmd =3D=3D 'pysh'): > - return [] > - if arg is None: > - return [cmd, win32_to_unix_path(path)] > - return [cmd, arg, win32_to_unix_path(path)] > - except IOError as e: > - if e.errno!=3Derrno.ENOENT and \ > - (e.errno!=3Derrno.EPERM and not os.path.isdir(path)): # > Opening a directory raises EPERM > - raise > - return [] > - =20 > -def win32_find_in_path(name, path): > - if isinstance(path, str): > - path =3D path.split(os.pathsep) > - =20 > - exts =3D os.environ.get('PATHEXT', '').lower().split(os.pathsep) > - for p in path: > - p_name =3D os.path.join(p, name) > - =20 > - prefix =3D resolve_shebang(p_name) > - if prefix: > - return prefix > - =20 > - for ext in exts: =20 > - p_name_ext =3D p_name + ext > - if os.path.exists(p_name_ext): > - return [win32_to_unix_path(p_name_ext)] > - return [] > - > -class Traps(dict): > - def __setitem__(self, key, value): > - if key not in ('EXIT',): > - raise NotImplementedError() > - super(Traps, self).__setitem__(key, value) > - > -# IFS white spaces character class > -_IFS_WHITESPACES =3D (' ', '\t', '\n') > - > -class Environment: > - """Environment holds environment variables, export table, > function=20 > - definitions and whatever is defined in 2.12 "Shell Execution > Environment", > - redirection excepted. > - """ > - def __init__(self, pwd): > - self._opt =3D set() #Shell options > - =20 > - self._functions =3D {} =20 > - self._env =3D {'?': '0', '#': '0'} > - self._exported =3D set([ > - 'HOME', 'IFS', 'PATH' > - ]) > - =20 > - # Set environment vars with side-effects > - self._ifs_ws =3D None # Set of IFS whitespace characters > - self._ifs_re =3D None # Regular expression used to split > between words using IFS classes > - self['IFS'] =3D ''.join(_IFS_WHITESPACES) #Default environment > values > - self['PWD'] =3D pwd > - self.traps =3D Traps() > - =20 > - def clone(self, subshell=3DFalse): > - env =3D Environment(self['PWD']) > - env._opt =3D set(self._opt) > - for k,v in self.get_variables().iteritems(): > - if k in self._exported: > - env.export(k,v) > - elif subshell: > - env[k] =3D v > - =20 > - if subshell: > - env._functions =3D dict(self._functions) > - =20 > - return env =20 > - =20 > - def __getitem__(self, key): > - if key in ('@', '*', '-', '$'): > - raise NotImplementedError('%s is not implemented' % > repr(key)) > - return self._env[key] > - =20 > - def get(self, key, defval=3DNone): > - try: > - return self[key] > - except KeyError: > - return defval > - =20 > - def __setitem__(self, key, value): > - if key=3D=3D'IFS': > - # Update the whitespace/non-whitespace classes > - self._update_ifs(value) > - elif key=3D=3D'PWD': > - pwd =3D os.path.abspath(value) > - if not os.path.isdir(pwd): > - raise VarAssignmentError('Invalid directory %s' % > value) > - value =3D pwd > - elif key in ('?', '!'): > - value =3D str(int(value)) > - self._env[key] =3D value > - =20 > - def __delitem__(self, key): > - if key in ('IFS', 'PWD', '?'): > - raise VarAssignmentError('%s cannot be unset' % key) > - del self._env[key] > - > - def __contains__(self, item): > - return item in self._env > - =20 > - def set_positional_args(self, args): > - """Set the content of 'args' as positional argument from 1 > to len(args). > - Return previous argument as a list of strings. > - """ > - # Save and remove previous arguments > - prevargs =3D [] =20 > - for i in range(int(self._env['#'])): > - i =3D str(i+1) > - prevargs.append(self._env[i]) > - del self._env[i] > - self._env['#'] =3D '0' > - =20 > - #Set new ones > - for i,arg in enumerate(args): > - self._env[str(i+1)] =3D str(arg) > - self._env['#'] =3D str(len(args)) > - =20 > - return prevargs > - =20 > - def get_positional_args(self): > - return [self._env[str(i+1)] for i in > range(int(self._env['#']))] > - =20 > - def get_variables(self): > - return dict(self._env) > - =20 > - def export(self, key, value=3DNone): > - if value is not None: > - self[key] =3D value > - self._exported.add(key) > - =20 > - def get_exported(self): > - return [(k,self._env.get(k)) for k in self._exported] > - =20 > - def split_fields(self, word): > - if not self._ifs_ws or not word: > - return [word] > - return re.split(self._ifs_re, word) > - =20 > - def _update_ifs(self, value): > - """Update the split_fields related variables when IFS > character set is > - changed. > - """ > - # TODO: handle NULL IFS > - =20 > - # Separate characters in whitespace and non-whitespace > - chars =3D set(value) > - ws =3D [c for c in chars if c in _IFS_WHITESPACES] > - nws =3D [c for c in chars if c not in _IFS_WHITESPACES] > - =20 > - # Keep whitespaces in a string for left and right stripping > - self._ifs_ws =3D ''.join(ws) > - =20 > - # Build a regexp to split fields > - trailing =3D '[' + ''.join([re.escape(c) for c in ws]) + ']' > - if nws: > - # First, the single non-whitespace occurence. > - nws =3D '[' + ''.join([re.escape(c) for c in nws]) + ']' > - nws =3D '(?:' + trailing + '*' + nws + trailing + '*' + > '|' + trailing + '+)' > - else: > - # Then mix all parts with quantifiers > - nws =3D trailing + '+' > - self._ifs_re =3D re.compile(nws) > - =20 > - def has_opt(self, opt, val=3DNone): > - return (opt, val) in self._opt > - =20 > - def set_opt(self, opt, val=3DNone): > - self._opt.add((opt, val)) > - =20 > - def find_in_path(self, name, pwd=3DFalse): > - path =3D self._env.get('PATH', '').split(os.pathsep) > - if pwd: > - path[:0] =3D [self['PWD']] > - if os.name =3D=3D 'nt': > - return win32_find_in_path(name, self._env.get('PATH', > '')) > - else: > - raise NotImplementedError() > - =20 > - def define_function(self, name, body): > - if not is_name(name): > - raise ShellSyntaxError('%s is not a valid function name' > % repr(name)) > - self._functions[name] =3D body > - =20 > - def remove_function(self, name): > - del self._functions[name] > - =20 > - def is_function(self, name): > - return name in self._functions > - =20 > - def get_function(self, name): > - return self._functions.get(name) > - =20 > - =20 > -name_charset =3D > 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_' > -name_charset =3D dict(zip(name_charset,name_charset)) > - =20 > -def match_name(s): > - """Return the length in characters of the longest prefix made of > name > - allowed characters in s. > - """ > - for i,c in enumerate(s): > - if c not in name_charset: > - return s[:i] > - return s > - =20 > -def is_name(s): > - return len([c for c in s if c not in name_charset])<=3D0 > - =20 > -def is_special_param(c): > - return len(c)=3D=3D1 and c in ('@','*','#','?','-','$','!','0') > - =20 > -def utility_not_implemented(name, *args, **kwargs): > - raise NotImplementedError('%s utility is not implemented' % name) > - =20 > - > -class Utility: > - """Define utilities properties: > - func -- utility callable. See builtin module for utility samples. > - is_special -- see XCU 2.8. > - """ > - def __init__(self, func, is_special=3D0): > - self.func =3D func > - self.is_special =3D bool(is_special) > - > - > -def encodeargs(args): > - def encodearg(s): > - lines =3D base64.encodestring(s) > - lines =3D [l.splitlines()[0] for l in lines] > - return ''.join(lines) > - > - s =3D pickle.dumps(args) > - return encodearg(s) > - > -def decodeargs(s): > - s =3D base64.decodestring(s) > - return pickle.loads(s) > - =20 > - > -class GlobError(Exception): > - pass > - > -class Options: > - def __init__(self): > - # True if Mercurial operates with binary streams > - self.hgbinary =3D True > - > -class Interpreter: > - # Implementation is very basic: the execute() method just makes > a DFS on the > - # AST and execute nodes one by one. Nodes are tuple (name,obj) > where name > - # is a string identifier and obj the AST element returned by the > parser. > - # > - # Handler are named after the node identifiers. > - # TODO: check node names and remove the switch in execute with > some > - # dynamic getattr() call to find node handlers. > - """Shell interpreter. > - =20 > - The following debugging flags can be passed: > - debug-parsing - enable PLY debugging. > - debug-tree - print the generated AST. > - debug-cmd - trace command execution before word expansion, plus > exit status. > - debug-utility - trace utility execution. > - """ > - =20 > - # List supported commands. > - COMMANDS =3D { > - 'cat': Utility(builtin.utility_cat,), > - 'cd': Utility(builtin.utility_cd,), > - ':': Utility(builtin.utility_colon,), > - 'echo': Utility(builtin.utility_echo), > - 'env': Utility(builtin.utility_env), > - 'exit': Utility(builtin.utility_exit), > - 'export': Utility(builtin.builtin_export, > is_special=3D1), > - 'egrep': Utility(builtin.utility_egrep), > - 'fgrep': Utility(builtin.utility_fgrep), > - 'gunzip': Utility(builtin.utility_gunzip), > - 'kill': Utility(builtin.utility_kill), > - 'mkdir': Utility(builtin.utility_mkdir), > - 'netstat': Utility(builtin.utility_netstat), > - 'printf': Utility(builtin.utility_printf), > - 'pwd': Utility(builtin.utility_pwd), > - 'return': Utility(builtin.builtin_return, > is_special=3D1), > - 'sed': Utility(builtin.utility_sed,), > - 'set': Utility(builtin.builtin_set,), > - 'shift': Utility(builtin.builtin_shift,), > - 'sleep': Utility(builtin.utility_sleep,), > - 'sort': Utility(builtin.utility_sort,), > - 'trap': Utility(builtin.builtin_trap, > is_special=3D1), > - 'true': Utility(builtin.utility_true), > - 'unset': Utility(builtin.builtin_unset, > is_special=3D1), > - 'wait': Utility(builtin.builtin_wait, > is_special=3D1), > - } > - =20 > - def __init__(self, pwd, debugflags =3D [], env=3DNone, redirs=3DNone, > stdin=3DNone, > - stdout=3DNone, stderr=3DNone, opts=3DOptions()): > - self._env =3D env > - if self._env is None: > - self._env =3D Environment(pwd) > - self._children =3D {} > - =20 > - self._redirs =3D redirs > - self._close_redirs =3D False > - =20 > - if self._redirs is None: > - if stdin is None: > - stdin =3D sys.stdin > - if stdout is None: > - stdout =3D sys.stdout > - if stderr is None: > - stderr =3D sys.stderr > - stdin =3D FileWrapper('r', stdin, False) > - stdout =3D FileWrapper('w', stdout, False) > - stderr =3D FileWrapper('w', stderr, False) > - self._redirs =3D Redirections(stdin, stdout, stderr) > - self._close_redirs =3D True > - =20 > - self._debugflags =3D list(debugflags) > - self._logfile =3D sys.stderr > - self._options =3D opts > - =20 > - def close(self): > - """Must be called when the interpreter is no longer used.""" > - script =3D self._env.traps.get('EXIT') > - if script: > - try: > - self.execute_script(script=3Dscript) > - except: > - pass > - > - if self._redirs is not None and self._close_redirs: > - self._redirs.close() > - self._redirs =3D None > - =20 > - def log(self, s): > - self._logfile.write(s) > - self._logfile.flush() > - =20 > - def __getitem__(self, key): > - return self._env[key] > - =20 > - def __setitem__(self, key, value): > - self._env[key] =3D value > - > - def options(self): > - return self._options > - > - def redirect(self, redirs, ios): > - def add_redir(io): > - if isinstance(io, pyshyacc.IORedirect): > - redirs.add(self, io.op, io.filename, io.io_number) > - else: > - redirs.add_here_document(self, io.name, io.content, > io.io_number) > - =20 > - map(add_redir, ios) > - return redirs > - =20 > - def execute_script(self, script=3DNone, ast=3DNone, sourced=3DFalse, > - scriptpath=3DNone): > - """If script is not None, parse the input. Otherwise takes > the supplied > - AST. Then execute the AST. > - Return the script exit status. > - """ > - try: > - if scriptpath is not None: > - self._env['0'] =3D os.path.abspath(scriptpath) > - > - if script is not None: > - debug_parsing =3D ('debug-parsing' in > self._debugflags) =20 > - cmds, script =3D pyshyacc.parse(script, True, > debug_parsing) > - if 'debug-tree' in self._debugflags: > - pyshyacc.print_commands(cmds, self._logfile) > - self._logfile.flush() > - else: > - cmds, script =3D ast, '' =20 > - =20 > - status =3D 0 > - for cmd in cmds: > - try: > - status =3D self.execute(cmd) > - except ExitSignal as e: > - if sourced: > - raise > - status =3D int(e.args[0]) > - return status > - except ShellError: > - self._env['?'] =3D 1 > - raise > - if 'debug-utility' in self._debugflags or > 'debug-cmd' in self._debugflags: > - self.log('returncode ' + str(status)+ '\n') > - return status > - except CommandNotFound as e: > - print >>self._redirs.stderr, str(e) > - self._redirs.stderr.flush() > - # Command not found by non-interactive shell > - # return 127 > - raise > - except RedirectionError as e: > - # TODO: should be handled depending on the utility status > - print >>self._redirs.stderr, str(e) > - self._redirs.stderr.flush() > - # Command not found by non-interactive shell > - # return 127 > - raise > - > - def dotcommand(self, env, args): > - if len(args) < 1: > - raise ShellError('. expects at least one argument') > - path =3D args[0] > - if '/' not in path: > - found =3D env.find_in_path(args[0], True) > - if found: > - path =3D found[0] > - script =3D file(path).read() > - return self.execute_script(script=3Dscript, sourced=3DTrue) > - > - def execute(self, token, redirs=3DNone): > - """Execute and AST subtree with supplied redirections > overriding default > - interpreter ones. > - Return the exit status. > - """ > - if not token: > - return 0 > - =20 > - if redirs is None: > - redirs =3D self._redirs > - =20 > - if isinstance(token, list): > - # Commands sequence > - res =3D 0 > - for t in token: > - res =3D self.execute(t, redirs) > - return res > - > - type, value =3D token > - status =3D 0 > - if type=3D=3D'simple_command': > - redirs_copy =3D redirs.clone() > - try: > - # TODO: define and handle command return values > - # TODO: implement set -e > - status =3D self._execute_simple_command(value, > redirs_copy) > - finally: > - redirs_copy.close() > - elif type=3D=3D'pipeline': > - status =3D self._execute_pipeline(value, redirs) > - elif type=3D=3D'and_or': > - status =3D self._execute_and_or(value, redirs) > - elif type=3D=3D'for_clause': > - status =3D self._execute_for_clause(value, redirs) > - elif type=3D=3D'while_clause': > - status =3D self._execute_while_clause(value, redirs) > - elif type=3D=3D'function_definition': > - status =3D self._execute_function_definition(value, redirs) > - elif type=3D=3D'brace_group': > - status =3D self._execute_brace_group(value, redirs) > - elif type=3D=3D'if_clause': > - status =3D self._execute_if_clause(value, redirs) > - elif type=3D=3D'subshell': > - status =3D self.subshell(ast=3Dvalue.cmds, redirs=3Dredirs) > - elif type=3D=3D'async': > - status =3D self._asynclist(value) > - elif type=3D=3D'redirect_list': > - redirs_copy =3D self.redirect(redirs.clone(), value.redirs) > - try: > - status =3D self.execute(value.cmd, redirs_copy) > - finally: > - redirs_copy.close() > - else: > - raise NotImplementedError('Unsupported token type ' + > type) - > - if status < 0: > - status =3D 255 > - return status > - =20 > - def _execute_if_clause(self, if_clause, redirs): > - cond_status =3D self.execute(if_clause.cond, redirs) > - if cond_status=3D=3D0: > - return self.execute(if_clause.if_cmds, redirs) > - else: > - return self.execute(if_clause.else_cmds, redirs) > - =20 > - def _execute_brace_group(self, group, redirs): > - status =3D 0 > - for cmd in group.cmds: > - status =3D self.execute(cmd, redirs) > - return status > - =20 > - def _execute_function_definition(self, fundef, redirs): > - self._env.define_function(fundef.name, fundef.body) > - return 0 > - =20 > - def _execute_while_clause(self, while_clause, redirs): > - status =3D 0 > - while 1: > - cond_status =3D 0 > - for cond in while_clause.condition: > - cond_status =3D self.execute(cond, redirs) > - =20 > - if cond_status: > - break > - =20 > - for cmd in while_clause.cmds: > - status =3D self.execute(cmd, redirs) > - =20 > - return status > - =20 > - def _execute_for_clause(self, for_clause, redirs): > - if not is_name(for_clause.name): > - raise ShellSyntaxError('%s is not a valid name' % > repr(for_clause.name)) > - items =3D mappend(self.expand_token, for_clause.items) > - =20 > - status =3D 0 =20 > - for item in items: > - self._env[for_clause.name] =3D item > - for cmd in for_clause.cmds: > - status =3D self.execute(cmd, redirs) > - return status > - =20 > - def _execute_and_or(self, or_and, redirs): > - res =3D self.execute(or_and.left, redirs) =20 > - if (or_and.op=3D=3D'&&' and res=3D=3D0) or (or_and.op!=3D'&&' and > res!=3D0): > - res =3D self.execute(or_and.right, redirs) > - return res > - =20 > - def _execute_pipeline(self, pipeline, redirs): =20 > - if len(pipeline.commands)=3D=3D1: > - status =3D self.execute(pipeline.commands[0], redirs) > - else: > - # Execute all commands one after the other > - status =3D 0 > - inpath, outpath =3D None, None > - try: > - # Commands inputs and outputs cannot really be > plugged as done > - # by a real shell. Run commands sequentially and > chain their > - # input/output throught temporary files. > - tmpfd, inpath =3D tempfile.mkstemp() > - os.close(tmpfd) > - tmpfd, outpath =3D tempfile.mkstemp() > - os.close(tmpfd) > - =20 > - inpath =3D win32_to_unix_path(inpath) > - outpath =3D win32_to_unix_path(outpath) > - =20 > - for i, cmd in enumerate(pipeline.commands): > - call_redirs =3D redirs.clone() > - try: > - if i!=3D0: > - call_redirs.add(self, '<', inpath) > - if i!=3Dlen(pipeline.commands)-1: > - call_redirs.add(self, '>', outpath) > - =20 > - status =3D self.execute(cmd, call_redirs) > - =20 > - # Chain inputs/outputs > - inpath, outpath =3D outpath, inpath > - finally: > - call_redirs.close() =20 > - finally: > - if inpath: os.remove(inpath) > - if outpath: os.remove(outpath) > - =20 > - if pipeline.reverse_status: > - status =3D int(not status) > - self._env['?'] =3D status > - return status > - =20 > - def _execute_function(self, name, args, interp, env, stdin, > stdout, stderr, *others): > - assert interp is self > - =20 > - func =3D env.get_function(name) > - #Set positional parameters > - prevargs =3D None > - try: > - prevargs =3D env.set_positional_args(args) > - try: > - redirs =3D Redirections(stdin.dup(), stdout.dup(), > stderr.dup()) > - try: > - status =3D self.execute(func, redirs) > - finally: > - redirs.close() > - except ReturnSignal as e: > - status =3D int(e.args[0]) > - env['?'] =3D status > - return status > - finally: > - #Reset positional parameters > - if prevargs is not None: > - env.set_positional_args(prevargs) > - =20 > - def _execute_simple_command(self, token, redirs): > - """Can raise ReturnSignal when return builtin is called, > ExitSignal when > - exit is called, and other shell exceptions upon builtin > failures. > - """ > - debug_command =3D 'debug-cmd' in self._debugflags > - if debug_command: > - self.log('word' + repr(token.words) + '\n') > - self.log('assigns' + repr(token.assigns) + '\n') > - self.log('redirs' + repr(token.redirs) + '\n') > - =20 > - is_special =3D None > - env =3D self._env > - =20 > - try: > - # Word expansion > - args =3D [] > - for word in token.words: =20 > - args +=3D self.expand_token(word) > - if is_special is None and args: > - is_special =3D env.is_function(args[0]) or \ > - (args[0] in self.COMMANDS and > self.COMMANDS[args[0]].is_special) > - =20 > - if debug_command: > - self.log('_execute_simple_command' + str(args) + > '\n') > - =20 > - if not args: > - # Redirections happen is a subshell > - redirs =3D redirs.clone() > - elif not is_special: > - env =3D self._env.clone() > - =20 > - # Redirections > - self.redirect(redirs, token.redirs) > - =20 > - # Variables assignments > - res =3D 0 > - for type,(k,v) in token.assigns: > - status, expanded =3D self.expand_variable((k,v)) > - if status is not None: > - res =3D status > - if args: > - env.export(k, expanded) > - else: > - env[k] =3D expanded > - =20 > - if args and args[0] in ('.', 'source'): > - res =3D self.dotcommand(env, args[1:]) > - elif args: > - if args[0] in self.COMMANDS: > - command =3D self.COMMANDS[args[0]] > - elif env.is_function(args[0]): > - command =3D Utility(self._execute_function, > is_special=3DTrue) > - else: > - if not '/' in args[0].replace('\\', '/'): > - cmd =3D env.find_in_path(args[0]) > - if not cmd: > - # TODO: test error code on unknown > command =3D> 127 > - raise CommandNotFound('Unknown command: > "%s"' % args[0]) > - else: > - # Handle commands like '/cygdrive/c/foo.bat' > - cmd =3D cygwin_to_windows_path(args[0]) > - if not os.path.exists(cmd): > - raise CommandNotFound('%s: No such file > or directory' % args[0]) > - shebang =3D resolve_shebang(cmd) > - if shebang: > - cmd =3D shebang > - else: > - cmd =3D [cmd] > - args[0:1] =3D cmd > - command =3D Utility(builtin.run_command) > - =20 > - # Command execution > - if 'debug-cmd' in self._debugflags: > - self.log('redirections ' + str(redirs) + '\n') > - =20 > - res =3D command.func(args[0], args[1:], self, env, > - redirs.stdin(), redirs.stdout(),=20 > - redirs.stderr(), self._debugflags) > - =20 > - if self._env.has_opt('-x'): > - # Trace command execution in shell environment > - # BUG: would be hard to reproduce a real shell > behaviour since > - # the AST is not annotated with source lines/tokens. > - self._redirs.stdout().write(' '.join(args)) > - =20 > - except ReturnSignal: > - raise > - except ShellError as e: > - if is_special or isinstance(e, (ExitSignal, > - ShellSyntaxError, > ExpansionError)): > - raise e > - self._redirs.stderr().write(str(e)+'\n') > - return 1 > - > - return res > - > - def expand_token(self, word): > - """Expand a word as specified in [2.6 Word Expansions]. > Return the list > - of expanded words. > - """ > - status, wtrees =3D self._expand_word(word) > - return map(pyshlex.wordtree_as_string, wtrees) > - =20 > - def expand_variable(self, word): > - """Return a status code (or None if no command expansion > occurred) > - and a single word. > - """ > - status, wtrees =3D self._expand_word(word, pathname=3DFalse, > split=3DFalse) > - words =3D map(pyshlex.wordtree_as_string, wtrees) > - assert len(words)=3D=3D1 > - return status, words[0] > - =20 > - def expand_here_document(self, word): > - """Return the expanded document as a single word. The here > document is=20 > - assumed to be unquoted. > - """ > - status, wtrees =3D self._expand_word(word, pathname=3DFalse, > - split=3DFalse, > here_document=3DTrue) > - words =3D map(pyshlex.wordtree_as_string, wtrees) > - assert len(words)=3D=3D1 > - return words[0] > - =20 > - def expand_redirection(self, word): > - """Return a single word.""" > - return self.expand_variable(word)[1] > - =20 > - def get_env(self): > - return self._env > - =20 > - def _expand_word(self, token, pathname=3DTrue, split=3DTrue, > here_document=3DFalse): > - wtree =3D pyshlex.make_wordtree(token[1], > here_document=3Dhere_document) > - =20 > - # TODO: implement tilde expansion > - def expand(wtree): > - """Return a pseudo wordtree: the tree or its subelements > can be empty > - lists when no value result from the expansion. > - """ > - status =3D None > - for part in wtree: > - if not isinstance(part, list): > - continue > - if part[0]in ("'", '\\'): > - continue > - elif part[0] in ('`', '$('): > - status, result =3D self._expand_command(part) > - part[:] =3D result > - elif part[0] in ('$', '${'): > - part[:] =3D self._expand_parameter(part, > wtree[0]=3D=3D'"', split) > - elif part[0] in ('', '"'): > - status, result =3D expand(part) > - part[:] =3D result > - else: > - raise NotImplementedError('%s expansion is not > implemented' > - % part[0]) > - # [] is returned when an expansion result in no-field, > - # like an empty $@ > - wtree =3D [p for p in wtree if p !=3D []] > - if len(wtree) < 3: > - return status, [] > - return status, wtree > - =20 > - status, wtree =3D expand(wtree) > - if len(wtree) =3D=3D 0: > - return status, wtree > - wtree =3D pyshlex.normalize_wordtree(wtree) > - =20 > - if split: > - wtrees =3D self._split_fields(wtree) > - else: > - wtrees =3D [wtree] > - =20 > - if pathname: > - wtrees =3D mappend(self._expand_pathname, wtrees) > - =20 > - wtrees =3D map(self._remove_quotes, wtrees) > - return status, wtrees > - =20 > - def _expand_command(self, wtree): > - # BUG: there is something to do with backslashes and quoted > - # characters here > - command =3D pyshlex.wordtree_as_string(wtree[1:-1]) > - status, output =3D self.subshell_output(command) > - return status, ['', output, ''] > - =20 > - def _expand_parameter(self, wtree, quoted=3DFalse, split=3DFalse): > - """Return a valid wtree or an empty list when no parameter > results.""" > - # Get the parameter name > - # TODO: implement weird expansion rules with ':' > - name =3D pyshlex.wordtree_as_string(wtree[1:-1]) > - if not is_name(name) and not is_special_param(name): > - raise ExpansionError('Bad substitution "%s"' % name) > - # TODO: implement special parameters > - if name in ('@', '*'): > - args =3D self._env.get_positional_args() > - if len(args) =3D=3D 0: > - return [] > - if len(args)<2: > - return ['', ''.join(args), ''] > - =20 > - sep =3D self._env.get('IFS', '')[:1] > - if split and quoted and name=3D=3D'@': > - # Introduce a new token to tell the caller that > these parameters > - # cause a split as specified in 2.5.2 > - return ['@'] + args + [''] > - else: > - return ['', sep.join(args), ''] =20 > - =20 > - return ['', self._env.get(name, ''), ''] > - =20 > - def _split_fields(self, wtree): > - def is_empty(split): > - return split=3D=3D['', '', ''] > - =20 > - def split_positional(quoted): > - # Return a list of wtree split according positional > parameters rules. > - # All remaining '@' groups are removed. > - assert quoted[0]=3D=3D'"' > - =20 > - splits =3D [[]] > - for part in quoted: > - if not isinstance(part, list) or part[0]!=3D'@': > - splits[-1].append(part) > - else: > - # Empty or single argument list were dealt with > already > - assert len(part)>3 > - # First argument must join with the beginning > part of the original word > - splits[-1].append(part[1]) > - # Create double-quotes expressions for every > argument after the first > - for arg in part[2:-1]: > - splits[-1].append('"') > - splits.append(['"', arg]) > - return splits > - =20 > - # At this point, all expansions but pathnames have occured. > Only quoted > - # and positional sequences remain. Thus, all candidates for > field splitting=20 > - # are in the tree root, or are positional splits ('@') and > lie in root > - # children. > - if not wtree or wtree[0] not in ('', '"'): > - # The whole token is quoted or empty, nothing to split > - return [wtree] > - =20 > - if wtree[0]=3D=3D'"': > - wtree =3D ['', wtree, ''] > - =20 > - result =3D [['', '']] > - for part in wtree[1:-1]: > - if isinstance(part, list): > - if part[0]=3D=3D'"': > - splits =3D split_positional(part) > - if len(splits)<=3D1: > - result[-1] +=3D [part, ''] > - else: > - # Terminate the current split > - result[-1] +=3D [splits[0], ''] > - result +=3D splits[1:-1] > - # Create a new split > - result +=3D [['', splits[-1], '']] > - else: > - result[-1] +=3D [part, ''] > - else: > - splits =3D self._env.split_fields(part) > - if len(splits)<=3D1: > - # No split > - result[-1][-1] +=3D part > - else: > - # Terminate the current resulting part and > create a new one > - result[-1][-1] +=3D splits[0] > - result[-1].append('') > - result +=3D [['', r, ''] for r in splits[1:-1]] > - result +=3D [['', splits[-1]]] > - result[-1].append('') > - =20 > - # Leading and trailing empty groups come from > leading/trailing blanks > - if result and is_empty(result[-1]): > - result[-1:] =3D [] > - if result and is_empty(result[0]): > - result[:1] =3D [] > - return result > - =20 > - def _expand_pathname(self, wtree): > - """See [2.6.6 Pathname Expansion].""" > - if self._env.has_opt('-f'): > - return [wtree] > - =20 > - # All expansions have been performed, only quoted sequences > should remain > - # in the tree. Generate the pattern by folding the tree, > escaping special > - # characters when appear quoted > - special_chars =3D '*?[]' > - =20 > - def make_pattern(wtree): > - subpattern =3D [] > - for part in wtree[1:-1]: > - if isinstance(part, list): > - part =3D make_pattern(part) > - elif wtree[0]!=3D'': > - for c in part: > - # Meta-characters cannot be quoted > - if c in special_chars: > - raise GlobError() > - subpattern.append(part) > - return ''.join(subpattern) > - =20 > - def pwd_glob(pattern): > - cwd =3D os.getcwd() > - os.chdir(self._env['PWD']) > - try: > - return glob.glob(pattern)=20 > - finally: > - os.chdir(cwd) =20 > - =20 > - #TODO: check working directory issues here wrt relative > patterns > - try: > - pattern =3D make_pattern(wtree) > - paths =3D pwd_glob(pattern) > - except GlobError: > - # BUG: Meta-characters were found in quoted sequences. > The should=20 > - # have been used literally but this is unsupported in > current glob module. > - # Instead we consider the whole tree must be used > literally and > - # therefore there is no point in globbing. This is wrong > when meta > - # characters are mixed with quoted meta in the same > pattern like: > - # < foo*"py*" > > - paths =3D [] > - =20 > - if not paths: > - return [wtree] > - return [['', path, ''] for path in paths] > - =20 > - def _remove_quotes(self, wtree): > - """See [2.6.7 Quote Removal].""" > - =20 > - def unquote(wtree): > - unquoted =3D [] > - for part in wtree[1:-1]: > - if isinstance(part, list): > - part =3D unquote(part) > - unquoted.append(part) > - return ''.join(unquoted) > - =20 > - return ['', unquote(wtree), ''] > - =20 > - def subshell(self, script=3DNone, ast=3DNone, redirs=3DNone): > - """Execute the script or AST in a subshell, with inherited > redirections > - if redirs is not None. > - """ > - if redirs: > - sub_redirs =3D redirs > - else: > - sub_redirs =3D redirs.clone() > - =20 > - subshell =3D None =20 > - try: > - subshell =3D Interpreter(None, self._debugflags, > self._env.clone(True), > - sub_redirs, opts=3Dself._options) > - return subshell.execute_script(script, ast) > - finally: > - if not redirs: sub_redirs.close() > - if subshell: subshell.close() > - =20 > - def subshell_output(self, script): > - """Execute the script in a subshell and return the captured > output.""" =20 > - # Create temporary file to capture subshell output > - tmpfd, tmppath =3D tempfile.mkstemp() > - try: > - tmpfile =3D os.fdopen(tmpfd, 'wb') > - stdout =3D FileWrapper('w', tmpfile) > - =20 > - redirs =3D Redirections(self._redirs.stdin().dup(), > - stdout, > - > self._redirs.stderr().dup()) =20 > - try: > - status =3D self.subshell(script=3Dscript, redirs=3Dredir= s) > - finally: > - redirs.close() > - redirs =3D None > - =20 > - # Extract subshell standard output > - tmpfile =3D open(tmppath, 'rb') > - try: > - output =3D tmpfile.read() > - return status, output.rstrip('\n') > - finally: > - tmpfile.close() > - finally: > - os.remove(tmppath) > - > - def _asynclist(self, cmd): > - args =3D (self._env.get_variables(), cmd) > - arg =3D encodeargs(args) > - assert len(args) < 30*1024 > - cmd =3D ['pysh.bat', '--ast', '-c', arg] > - p =3D subprocess.Popen(cmd, cwd=3Dself._env['PWD']) > - self._children[p.pid] =3D p > - self._env['!'] =3D p.pid > - return 0 > - > - def wait(self, pids=3DNone): > - if not pids: > - pids =3D self._children.keys() > - > - status =3D 127 > - for pid in pids: > - if pid not in self._children: > - continue > - p =3D self._children.pop(pid) > - status =3D p.wait() > - > - return status > - > diff --git a/bitbake/lib/bb/pysh/lsprof.py > b/bitbake/lib/bb/pysh/lsprof.py deleted file mode 100644 > index b1831c2..0000000 > --- a/bitbake/lib/bb/pysh/lsprof.py > +++ /dev/null > @@ -1,116 +0,0 @@ > -#! /usr/bin/env python > - > -import sys > -from _lsprof import Profiler, profiler_entry > - > -__all__ =3D ['profile', 'Stats'] > - > -def profile(f, *args, **kwds): > - """XXX docstring""" > - p =3D Profiler() > - p.enable(subcalls=3DTrue, builtins=3DTrue) > - try: > - f(*args, **kwds) > - finally: > - p.disable() > - return Stats(p.getstats()) > - > - > -class Stats(object): > - """XXX docstring""" > - > - def __init__(self, data): > - self.data =3D data > - > - def sort(self, crit=3D"inlinetime"): > - """XXX docstring""" > - if crit not in profiler_entry.__dict__: > - raise ValueError("Can't sort by %s" % crit) > - self.data.sort(lambda b, a: cmp(getattr(a, crit), > - getattr(b, crit))) > - for e in self.data: > - if e.calls: > - e.calls.sort(lambda b, a: cmp(getattr(a, crit), > - getattr(b, crit))) > - > - def pprint(self, top=3DNone, file=3DNone, limit=3DNone, climit=3DNon= e): > - """XXX docstring""" > - if file is None: > - file =3D sys.stdout > - d =3D self.data > - if top is not None: > - d =3D d[:top] > - cols =3D "% 12s %12s %11.4f %11.4f %s\n" > - hcols =3D "% 12s %12s %12s %12s %s\n" > - cols2 =3D "+%12s %12s %11.4f %11.4f + %s\n" > - file.write(hcols % ("CallCount", "Recursive", "Total(ms)", > - "Inline(ms)", "module:lineno(function)")) > - count =3D 0 > - for e in d: > - file.write(cols % (e.callcount, e.reccallcount, > e.totaltime, > - e.inlinetime, label(e.code))) > - count +=3D 1 > - if limit is not None and count =3D=3D limit: > - return > - ccount =3D 0 > - if e.calls: > - for se in e.calls: > - file.write(cols % ("+%s" % se.callcount, > se.reccallcount, > - se.totaltime, se.inlinetime, > - "+%s" % label(se.code))) > - count +=3D 1 > - ccount +=3D 1 > - if limit is not None and count =3D=3D limit: > - return > - if climit is not None and ccount =3D=3D climit: > - break > - > - def freeze(self): > - """Replace all references to code objects with string > - descriptions; this makes it possible to pickle the > instance.""" - > - # this code is probably rather ickier than it needs to be! > - for i in range(len(self.data)): > - e =3D self.data[i] > - if not isinstance(e.code, str): > - self.data[i] =3D type(e)((label(e.code),) + e[1:]) > - if e.calls: > - for j in range(len(e.calls)): > - se =3D e.calls[j] > - if not isinstance(se.code, str): > - e.calls[j] =3D type(se)((label(se.code),) + > se[1:]) - > -_fn2mod =3D {} > - > -def label(code): > - if isinstance(code, str): > - return code > - try: > - mname =3D _fn2mod[code.co_filename] > - except KeyError: > - for k, v in sys.modules.items(): > - if v is None: > - continue > - if not hasattr(v, '__file__'): > - continue > - if not isinstance(v.__file__, str): > - continue > - if v.__file__.startswith(code.co_filename): > - mname =3D _fn2mod[code.co_filename] =3D k > - break > - else: > - mname =3D _fn2mod[code.co_filename] =3D > '<%s>'%code.co_filename - > - return '%s:%d(%s)' % (mname, code.co_firstlineno, code.co_name) > - > - > -if __name__ =3D=3D '__main__': > - import os > - sys.argv =3D sys.argv[1:] > - if not sys.argv: > - print >> sys.stderr, "usage: lsprof.py