From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6750261011454885888 X-Received: by 2002:a25:d604:: with SMTP id n4mr1363424ybg.460.1571730148151; Tue, 22 Oct 2019 00:42:28 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a25:68cd:: with SMTP id d196ls2393817ybc.13.gmail; Tue, 22 Oct 2019 00:42:27 -0700 (PDT) X-Google-Smtp-Source: APXvYqylLHIFrH2ab8bItv7jfwTHdQUbmDrweM5i2leO06dWDChbFdWh6D8s6SjwxkIikCJPfRth X-Received: by 2002:a25:8308:: with SMTP id s8mr1515678ybk.130.1571730146495; Tue, 22 Oct 2019 00:42:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1571730146; cv=none; d=google.com; s=arc-20160816; b=lJe9kLYBdsBhVSeQ6bDhVK/8DDLUudDFGx/JQtWEHCBjNSSo7spyqM5c36qwEIrafd yNBUjcqvGbG1Zc8dPtsakstDircjWTIqHm2ixiqv64AimHu9h/wIHZHw417eL9aPPzq5 HMkbsGOrgxANLYePJ9qEbv03/1idMcOFmDkUdx9mh+AdBhhD2mnCf4wUesO3EHB06r2h fGdWm/mNLslSupD8sHNccdUFtKZchsIBmUgUGUuwjjmfF1JZ5qFcxzB5ry8DqQGk2NR8 EUB7L9NEHmapVUj3eEXwFI+Su5x/HoF1JjCr8WtK+KTHnL0lzXSQwTA6aJiJPVXt4xyg 9myA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-language:content-transfer-encoding:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject :ironport-sdr:ironport-sdr; bh=ksmV1nzo3d9MF8colknPz+PFvLdMe2y4XcaW7dS1xQk=; b=gm9+YSo3/eb8Ep/wg7N+Ldlaoa9mTZV3eMAXa84gM11hG3m/Ks4kE8CV4gBs/MoKt4 BNsHkxHfu68L1xNIgcX5I4PpLQL479SZzLArVuFmgwUlEytR44eSDa5T79L7MQGM5fQg sORPtRtc0YmhQvR/KCgNLzBcC2Kp4sVBhvgFFGyj0ILQ9DSwsJv0MGiZxX8husyPc0f1 u17GzzCVcvHPDzFIOBZWmn120SHcaTc1mErnWeW/Kko8oKkGv0/9B6yz7kuB1a8XfF+5 Oiruql0irRUVBoKsIyIwzygrrpIh+vE8RiDD8pt3FK1qPWik0mH3UaqF8XwZ0vrmxTtP tDCw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of cedric_hombourger@mentor.com designates 68.232.137.180 as permitted sender) smtp.mailfrom=Cedric_Hombourger@mentor.com Return-Path: Received: from esa3.mentor.iphmx.com (esa3.mentor.iphmx.com. [68.232.137.180]) by gmr-mx.google.com with ESMTPS id u129si753732ywc.1.2019.10.22.00.42.24 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 22 Oct 2019 00:42:26 -0700 (PDT) Received-SPF: pass (google.com: domain of cedric_hombourger@mentor.com designates 68.232.137.180 as permitted sender) client-ip=68.232.137.180; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of cedric_hombourger@mentor.com designates 68.232.137.180 as permitted sender) smtp.mailfrom=Cedric_Hombourger@mentor.com IronPort-SDR: QTck59O66pkbHExwRcrT9En6BjBdGyY9IhNwFDtNJs/QjarQtt0StdmrOukIvtKuI0FjNP+vcS aZ0DMw4M85snGUOMFQqMtMl9F5uQU4bfQ8SBj0SVv45kKjbQYlTHCOQcZ0djaYsIdoyvDUVhQv crJVPwIgzBaQ960RBfrbshPexQArC4rDHsDRcUhO5vFglvFcsn03+hZX4vPoXvU/OQRzwgTWHt S5ghSqR9yvcWNRQY0rTzNMxZVm9NGgXqsvboTmveMX6p27p+g33eZuzycX+1/9i9bq4DkjxGPL tJg= X-IronPort-AV: E=Sophos;i="5.67,326,1566892800"; d="scan'208";a="42463009" Received: from orw-gwy-01-in.mentorg.com ([192.94.38.165]) by esa3.mentor.iphmx.com with ESMTP; 21 Oct 2019 23:42:20 -0800 IronPort-SDR: pgYBH4nYgG4R7xJv+PRmwgGz7QJq3Zxk16ziENHDRU8SA/Kd3GYV6SMnXcLR4lVPentGXCarE7 p7AHiRQT2HBhfWS9Cey+UkbHRWbana31UsaqLnB/WXEi8Rya/o7xxZyjdhJ+QYgjnJ6IJ7w17a 7tBx2M7EU/gvJgMuTXEySO7z8XTaOXtyHzBShwSQh1HqTDksM38pTC5os2uVZT3/JrP9BYfKJ7 3khr0965CGNjpxQFfkx5kPyeWDXVqIK5HD3/70dlwF1lri4S4C0nasXKripiu5GmnMwrIhj/iW MZw= Subject: Re: [PATCH 1/1] bitbake: update to version 1.44.0 To: Henning Schild CC: References: <1571667598-657-1-git-send-email-Cedric_Hombourger@mentor.com> <1571667598-657-2-git-send-email-Cedric_Hombourger@mentor.com> <20191022093609.5ed343a5@md1za8fc.ad001.siemens.net> From: Cedric Hombourger Message-ID: <514fe94a-882a-9b77-b554-f79b255a89d3@mentor.com> Date: Tue, 22 Oct 2019 09:41:59 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: <20191022093609.5ed343a5@md1za8fc.ad001.siemens.net> Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US Return-Path: cedric_hombourger@mentor.com X-Originating-IP: [137.202.0.90] X-ClientProxiedBy: SVR-IES-MBX-09.mgc.mentorg.com (139.181.222.9) To svr-ies-mbx-02.mgc.mentorg.com (139.181.222.2) X-TUID: 6o4qJeKyk0Zx Hi Henning On 10/22/2019 9:36 AM, Henning Schild wrote: > Hi Cedric, > > i would suggest a split between the actual bump and the following > Isar-changes. But i do not care too much and both ways have their pros > and cons. I have debated (with myself :)) and I agree that there are pros and cons with each approach. I ended up submitting a single commit at the end of the day to support people needing to "git bisect". I have otherwise no issues resubmitting the changes if we prefer to have the bitbake upgrade clearly separated from the Isar changes (either way is fine with me - @maintainers just let me know what your preference is!) > Henning > > Am Mon, 21 Oct 2019 16:19:58 +0200 > schrieb Cedric Hombourger : > >> Found that the gitsm fetcher we ship is rather out-dated compared to >> upstream and was failing to pull git sub-modules from repositories >> such as azure-umqtt-c. >> >> It shall be noted that bitbake has renamed its multiconfig targets: >> the "multiconfig:" prefix was changed to "mc:". Updated our scripts >> as well as documentation. >> >> Origin: https://github.com/openembedded/bitbake.git >> Commit 5d83d828cacb58ccb7c464e799c85fd2d2a50ccc (tag: 1.44.0) >> Author: Richard Purdie >> Date: Wed Oct 9 14:10:21 2019 +0100 >> >> bitbake: Update to version 1.44.0 >> >> Signed-off-by: Richard Purdie >> >> Signed-off-by: Cedric Hombourger >> --- >> RECIPE-API-CHANGELOG.md | 16 + >> bitbake/.gitattributes | 2 + >> bitbake/HEADER | 19 - >> bitbake/LICENSE | 12 +- >> bitbake/{COPYING => LICENSE.GPL-2.0-only} | 61 +- >> bitbake/LICENSE.MIT | 25 + >> bitbake/MANIFEST.in | 5 +- >> bitbake/bin/bitbake | 16 +- >> bitbake/bin/bitbake-diffsigs | 155 +- >> bitbake/bin/bitbake-dumpsig | 95 +- >> bitbake/bin/bitbake-hashclient | 170 ++ >> bitbake/bin/bitbake-hashserv | 62 + >> bitbake/bin/bitbake-layers | 12 +- >> bitbake/bin/bitbake-prserv | 4 + >> bitbake/bin/bitbake-selftest | 16 +- >> bitbake/bin/bitbake-worker | 19 +- >> bitbake/bin/bitdoc | 14 +- >> bitbake/bin/git-make-shallow | 4 + >> bitbake/bin/toaster | 12 +- >> bitbake/bin/toaster-eventreplay | 17 +- >> bitbake/classes/base.bbclass | 2 +- >> bitbake/contrib/dump_cache.py | 2 - >> .../bitbake-user-manual-execution.xml | 78 +- >> .../bitbake-user-manual-fetching.xml | 69 +- >> .../bitbake-user-manual-hello.xml | 28 +- >> .../bitbake-user-manual-intro.xml | 18 +- >> .../bitbake-user-manual-metadata.xml | 179 +- >> .../bitbake-user-manual-ref-variables.xml | 422 ++--- >> bitbake/doc/poky.ent | 8 - >> bitbake/lib/bb/COW.py | 15 - >> bitbake/lib/bb/__init__.py | 16 +- >> bitbake/lib/bb/build.py | 151 +- >> bitbake/lib/bb/cache.py | 109 +- >> bitbake/lib/bb/cache_extra.py | 14 +- >> bitbake/lib/bb/checksum.py | 12 +- >> bitbake/lib/bb/codeparser.py | 13 +- >> bitbake/lib/bb/command.py | 12 +- >> bitbake/lib/bb/compat.py | 4 + >> bitbake/lib/bb/cooker.py | 218 +-- >> bitbake/lib/bb/cookerdata.py | 44 +- >> bitbake/lib/bb/daemonize.py | 4 + >> bitbake/lib/bb/data.py | 23 +- >> bitbake/lib/bb/data_smart.py | 25 +- >> bitbake/lib/bb/event.py | 33 +- >> bitbake/lib/bb/exceptions.py | 3 + >> bitbake/lib/bb/fetch2/__init__.py | 55 +- >> bitbake/lib/bb/fetch2/bzr.py | 12 +- >> bitbake/lib/bb/fetch2/clearcase.py | 19 +- >> bitbake/lib/bb/fetch2/cvs.py | 17 +- >> bitbake/lib/bb/fetch2/git.py | 68 +- >> bitbake/lib/bb/fetch2/gitannex.py | 14 +- >> bitbake/lib/bb/fetch2/gitsm.py | 281 ++- >> bitbake/lib/bb/fetch2/hg.py | 18 +- >> bitbake/lib/bb/fetch2/local.py | 16 +- >> bitbake/lib/bb/fetch2/npm.py | 69 +- >> bitbake/lib/bb/fetch2/osc.py | 5 +- >> bitbake/lib/bb/fetch2/perforce.py | 15 +- >> bitbake/lib/bb/fetch2/repo.py | 16 +- >> bitbake/lib/bb/fetch2/s3.py | 15 +- >> bitbake/lib/bb/fetch2/sftp.py | 15 +- >> bitbake/lib/bb/fetch2/ssh.py | 14 +- >> bitbake/lib/bb/fetch2/svn.py | 98 +- >> bitbake/lib/bb/fetch2/wget.py | 104 +- >> bitbake/lib/bb/main.py | 41 +- >> bitbake/lib/bb/methodpool.py | 15 +- >> bitbake/lib/bb/monitordisk.py | 27 +- >> bitbake/lib/bb/msg.py | 14 +- >> bitbake/lib/bb/namedtuple_with_abc.py | 4 +- >> bitbake/lib/bb/parse/__init__.py | 14 +- >> bitbake/lib/bb/parse/ast.py | 15 +- >> bitbake/lib/bb/parse/parse_py/BBHandler.py | 59 +- >> bitbake/lib/bb/parse/parse_py/ConfHandler.py | 17 +- >> bitbake/lib/bb/parse/parse_py/__init__.py | 17 +- >> bitbake/lib/bb/persist_data.py | 234 ++- >> bitbake/lib/bb/process.py | 4 + >> bitbake/lib/bb/progress.py | 28 +- >> bitbake/lib/bb/providers.py | 18 +- >> bitbake/lib/bb/pysh/builtin.py | 710 -------- >> bitbake/lib/bb/pysh/interp.py | 1367 --------------- >> bitbake/lib/bb/pysh/lsprof.py | 116 -- >> bitbake/lib/bb/pysh/pysh.py | 167 -- >> bitbake/lib/bb/pysh/pyshlex.py | 5 - >> bitbake/lib/bb/pysh/pyshyacc.py | 17 +- >> bitbake/lib/bb/pysh/sherrors.py | 26 - >> bitbake/lib/bb/pysh/subprocess_fix.py | 77 - >> bitbake/lib/bb/remotedata.py | 12 +- >> bitbake/lib/bb/runqueue.py | 1523 >> ++++++++++------- bitbake/lib/bb/server/__init__.py | >> 14 +- bitbake/lib/bb/server/process.py | 123 +- >> bitbake/lib/bb/server/xmlrpcclient.py | 12 +- >> bitbake/lib/bb/server/xmlrpcserver.py | 12 +- >> bitbake/lib/bb/siggen.py | 350 +++- >> bitbake/lib/bb/taskdata.py | 17 +- >> bitbake/lib/bb/tests/codeparser.py | 22 +- >> bitbake/lib/bb/tests/cooker.py | 15 +- >> bitbake/lib/bb/tests/cow.py | 17 +- >> bitbake/lib/bb/tests/data.py | 39 +- >> bitbake/lib/bb/tests/event.py | 23 +- >> bitbake/lib/bb/tests/fetch.py | 308 +++- >> bitbake/lib/bb/tests/parse.py | 33 +- >> bitbake/lib/bb/tests/persist_data.py | 129 ++ >> .../tests/runqueue-tests/classes/base.bbclass | 262 +++ >> .../runqueue-tests/classes/image.bbclass | 5 + >> .../runqueue-tests/classes/native.bbclass | 2 + >> .../bb/tests/runqueue-tests/conf/bitbake.conf | 16 + >> .../runqueue-tests/conf/multiconfig/mc1.conf | 1 + >> .../runqueue-tests/conf/multiconfig/mc2.conf | 1 + >> .../lib/bb/tests/runqueue-tests/recipes/a1.bb | 0 >> .../lib/bb/tests/runqueue-tests/recipes/b1.bb | 1 + >> .../lib/bb/tests/runqueue-tests/recipes/c1.bb | 0 >> .../lib/bb/tests/runqueue-tests/recipes/d1.bb | 3 + >> .../lib/bb/tests/runqueue-tests/recipes/e1.bb | 1 + >> bitbake/lib/bb/tests/runqueue.py | 323 ++++ >> bitbake/lib/bb/tests/utils.py | 23 +- >> bitbake/lib/bb/tinfoil.py | 12 +- >> bitbake/lib/bb/ui/__init__.py | 12 +- >> bitbake/lib/bb/ui/buildinfohelper.py | 22 +- >> bitbake/lib/bb/ui/knotty.py | 60 +- >> bitbake/lib/bb/ui/ncurses.py | 12 +- >> bitbake/lib/bb/ui/taskexp.py | 12 +- >> bitbake/lib/bb/ui/toasterui.py | 12 +- >> bitbake/lib/bb/ui/uievent.py | 15 +- >> bitbake/lib/bb/ui/uihelper.py | 16 +- >> bitbake/lib/bb/utils.py | 49 +- >> bitbake/lib/bblayers/__init__.py | 4 + >> bitbake/lib/bblayers/action.py | 4 + >> bitbake/lib/bblayers/common.py | 4 + >> bitbake/lib/bblayers/layerindex.py | 6 +- >> bitbake/lib/bblayers/query.py | 43 +- >> bitbake/lib/bs4/dammit.py | 12 +- >> bitbake/lib/bs4/element.py | 22 +- >> bitbake/lib/hashserv/__init__.py | 93 + >> bitbake/lib/hashserv/client.py | 157 ++ >> bitbake/lib/hashserv/server.py | 414 +++++ >> bitbake/lib/hashserv/tests.py | 142 ++ >> bitbake/lib/layerindexlib/__init__.py | 33 +- >> bitbake/lib/layerindexlib/cooker.py | 12 +- >> bitbake/lib/layerindexlib/plugin.py | 13 +- >> bitbake/lib/layerindexlib/restapi.py | 12 +- >> bitbake/lib/layerindexlib/tests/common.py | 12 +- >> bitbake/lib/layerindexlib/tests/cooker.py | 12 +- >> .../lib/layerindexlib/tests/layerindexobj.py | 12 +- >> bitbake/lib/layerindexlib/tests/restapi.py | 12 +- >> bitbake/lib/progressbar/__init__.py | 2 + >> bitbake/lib/progressbar/compat.py | 2 + >> bitbake/lib/progressbar/progressbar.py | 2 + >> bitbake/lib/progressbar/widgets.py | 2 + >> bitbake/lib/prserv/__init__.py | 4 + >> bitbake/lib/prserv/db.py | 6 +- >> bitbake/lib/prserv/serv.py | 4 + >> bitbake/lib/pyinotify.py | 20 +- >> bitbake/lib/toaster/bldcollector/admin.py | 4 + >> bitbake/lib/toaster/bldcollector/urls.py | 13 +- >> bitbake/lib/toaster/bldcollector/views.py | 12 +- >> bitbake/lib/toaster/bldcontrol/admin.py | 4 + >> .../lib/toaster/bldcontrol/bbcontroller.py | 16 +- >> .../bldcontrol/localhostbecontroller.py | 16 +- >> .../management/commands/checksettings.py | 4 + >> .../management/commands/runbuilds.py | 4 + >> bitbake/lib/toaster/bldcontrol/models.py | 4 + >> bitbake/lib/toaster/bldcontrol/views.py | 4 + >> bitbake/lib/toaster/manage.py | 4 + >> bitbake/lib/toaster/orm/fixtures/oe-core.xml | 26 +- >> bitbake/lib/toaster/orm/fixtures/poky.xml | 38 +- >> .../orm/management/commands/lsupdates.py | 15 +- >> bitbake/lib/toaster/orm/models.py | 25 +- >> .../toaster/tests/browser/selenium_helpers.py | 17 +- >> .../tests/browser/selenium_helpers_base.py | 17 +- >> .../tests/browser/test_all_builds_page.py | 16 +- >> .../tests/browser/test_all_projects_page.py | 16 +- >> .../tests/browser/test_builddashboard_page.py | 16 +- >> .../test_builddashboard_page_artifacts.py | 16 +- >> .../test_builddashboard_page_recipes.py | 16 +- >> .../browser/test_builddashboard_page_tasks.py | 16 +- >> .../tests/browser/test_js_unit_tests.py | 16 +- >> .../tests/browser/test_landing_page.py | 18 +- >> .../tests/browser/test_layerdetails_page.py | 18 +- >> .../browser/test_most_recent_builds_states.py | 18 +- >> .../browser/test_new_custom_image_page.py | 16 +- >> .../tests/browser/test_new_project_page.py | 16 +- >> .../tests/browser/test_project_builds_page.py | 16 +- >> .../tests/browser/test_project_config_page.py | 16 +- >> .../tests/browser/test_project_page.py | 16 +- >> .../lib/toaster/tests/browser/test_sample.py | 16 +- >> .../toaster/tests/browser/test_task_page.py | 16 +- >> .../tests/browser/test_toastertable_ui.py | 16 +- >> bitbake/lib/toaster/tests/builds/buildtest.py | 16 +- >> .../tests/builds/test_core_image_min.py | 17 +- >> .../toaster/tests/commands/test_loaddata.py | 16 +- >> .../toaster/tests/commands/test_lsupdates.py | 16 +- >> .../toaster/tests/commands/test_runbuilds.py | 16 +- >> bitbake/lib/toaster/tests/db/test_db.py | 2 + >> .../lib/toaster/tests/eventreplay/__init__.py | 16 +- >> .../tests/functional/functional_helpers.py | 16 +- >> .../tests/functional/test_functional_basic.py | 16 +- >> bitbake/lib/toaster/tests/views/test_views.py | 16 +- >> bitbake/lib/toaster/toastergui/api.py | 13 +- >> bitbake/lib/toaster/toastergui/buildtables.py | 15 +- >> .../toastergui/static/js/importlayer.js | 12 +- >> bitbake/lib/toaster/toastergui/tablefilter.py | 15 +- >> bitbake/lib/toaster/toastergui/tables.py | 15 +- >> .../templatetags/field_values_filter.py | 4 + >> .../objects_to_dictionaries_filter.py | 4 + >> .../templatetags/project_url_tag.py | 4 + >> .../toastergui/templatetags/projecttags.py | 15 +- >> bitbake/lib/toaster/toastergui/typeaheads.py | 12 +- >> bitbake/lib/toaster/toastergui/urls.py | 12 +- >> bitbake/lib/toaster/toastergui/views.py | 16 +- >> bitbake/lib/toaster/toastergui/widgets.py | 15 +- >> .../management/commands/builddelete.py | 4 + >> .../management/commands/buildimport.py | 15 +- >> .../management/commands/buildslist.py | 4 + >> .../management/commands/checksocket.py | 16 +- >> .../toastermain/management/commands/perf.py | 4 + >> bitbake/lib/toaster/toastermain/settings.py | 15 +- >> .../settings_production_example.py | 15 +- >> .../lib/toaster/toastermain/settings_test.py | 15 +- >> bitbake/lib/toaster/toastermain/urls.py | 15 +- >> bitbake/lib/toaster/toastermain/wsgi.py | 7 +- >> doc/user_manual.md | 38 +- >> meta-isar/conf/conf-notes.txt | 6 +- >> scripts/ci_build.sh | 62 +- >> scripts/start_vm | 8 +- >> testsuite/build_test/build_test.py | 2 +- >> testsuite/start_vm.py | 2 +- >> 225 files changed, 5375 insertions(+), 6320 deletions(-) >> create mode 100644 bitbake/.gitattributes >> delete mode 100644 bitbake/HEADER >> rename bitbake/{COPYING => LICENSE.GPL-2.0-only} (84%) >> create mode 100644 bitbake/LICENSE.MIT >> mode change 100755 => 120000 bitbake/bin/bitbake-dumpsig >> create mode 100755 bitbake/bin/bitbake-hashclient >> create mode 100755 bitbake/bin/bitbake-hashserv >> delete mode 100644 bitbake/lib/bb/pysh/builtin.py >> delete mode 100644 bitbake/lib/bb/pysh/interp.py >> delete mode 100644 bitbake/lib/bb/pysh/lsprof.py >> delete mode 100644 bitbake/lib/bb/pysh/pysh.py >> delete mode 100644 bitbake/lib/bb/pysh/subprocess_fix.py >> create mode 100644 bitbake/lib/bb/tests/persist_data.py >> create mode 100644 >> bitbake/lib/bb/tests/runqueue-tests/classes/base.bbclass create mode >> 100644 bitbake/lib/bb/tests/runqueue-tests/classes/image.bbclass >> create mode 100644 >> bitbake/lib/bb/tests/runqueue-tests/classes/native.bbclass create >> mode 100644 bitbake/lib/bb/tests/runqueue-tests/conf/bitbake.conf >> create mode 100644 >> bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc1.conf create >> mode 100644 >> bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc2.conf create >> mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/a1.bb create >> mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/b1.bb create >> mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/c1.bb create >> mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/d1.bb create >> mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/e1.bb create >> mode 100644 bitbake/lib/bb/tests/runqueue.py create mode 100644 >> bitbake/lib/hashserv/__init__.py create mode 100644 >> bitbake/lib/hashserv/client.py create mode 100644 >> bitbake/lib/hashserv/server.py create mode 100644 >> bitbake/lib/hashserv/tests.py >> >> diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md >> index bbef1a3..81ec430 100644 >> --- a/RECIPE-API-CHANGELOG.md >> +++ b/RECIPE-API-CHANGELOG.md >> @@ -176,3 +176,19 @@ Otherwise set a encrypted root password like >> this: USERS += "root" >> USER_root[password] = >> "$6$rounds=10000$RXeWrnFmkY$DtuS/OmsAS2cCEDo0BF5qQsizIrq6jPgXnwv3PHqREJeKd1sXdHX/ayQtuQWVDHe0KIO0/sVH8dvQm1KthF0d/" >> ``` + >> +### multiconfig build targets were renamed >> + >> +bitbake was upgraded to version 1.44.0 where "multiconfig" build >> targets were +renamed "mc". As an example, builds for the >> qemuarm-stretch machine should now +be done as follows: >> + >> +``` >> +bitbake mc:qemuarm-stretch:isar-image-base >> +``` >> + >> +The old syntax is no longer supported and will produce an error: >> + >> +``` >> +bitbake multiconfig:qemuarm-stretch:isar-image-base >> +``` >> diff --git a/bitbake/.gitattributes b/bitbake/.gitattributes >> new file mode 100644 >> index 0000000..e4f8f62 >> --- /dev/null >> +++ b/bitbake/.gitattributes >> @@ -0,0 +1,2 @@ >> +*min.js binary >> +*min.css binary >> diff --git a/bitbake/HEADER b/bitbake/HEADER >> deleted file mode 100644 >> index 9859255..0000000 >> --- a/bitbake/HEADER >> +++ /dev/null >> @@ -1,19 +0,0 @@ >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> -# >> -# > does.> -# Copyright (C) >> -# >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> -# >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. - >> diff --git a/bitbake/LICENSE b/bitbake/LICENSE >> index 7d4e5f4..8458042 100644 >> --- a/bitbake/LICENSE >> +++ b/bitbake/LICENSE >> @@ -1,4 +1,13 @@ >> -BitBake is licensed under the GNU General Public License version >> 2.0. See COPYING for further details. +BitBake is licensed under the >> GNU General Public License version 2.0. See +LICENSE.GPL-2.0-only for >> further details. + >> +Individual files contain the following style tags instead of the >> full license text: + >> + SPDX-License-Identifier: GPL-2.0-only >> + >> +This enables machine processing of license information based on the >> SPDX +License Identifiers that are here available: >> http://spdx.org/licenses/ + >> >> The following external components are distributed with this software: >> >> @@ -17,3 +26,4 @@ Foundation and individual contributors. >> * Font Awesome fonts redistributed under the SIL Open Font License >> 1.1 >> * simplediff is distributed under the zlib license. >> + >> diff --git a/bitbake/COPYING b/bitbake/LICENSE.GPL-2.0-only >> similarity index 84% >> rename from bitbake/COPYING >> rename to bitbake/LICENSE.GPL-2.0-only >> index d511905..5db3c0a 100644 >> --- a/bitbake/COPYING >> +++ b/bitbake/LICENSE.GPL-2.0-only >> @@ -279,61 +279,10 @@ POSSIBILITY OF SUCH DAMAGES. >> >> END OF TERMS AND CONDITIONS >> >> - How to Apply These Terms to Your New Programs >> +Note: >> +Individual files contain the following tag instead of the full >> license text. >> - If you develop a new program, and you want it to be of the greatest >> -possible use to the public, the best way to achieve this is to make >> it -free software which everyone can redistribute and change under >> these terms. >> + SPDX-License-Identifier: GPL-2.0-only >> >> - To do so, attach the following notices to the program. It is >> safest -to attach them to the start of each source file to most >> effectively -convey the exclusion of warranty; and each file should >> have at least -the "copyright" line and a pointer to where the full >> notice is found. - >> - > does.> >> - Copyright (C) >> - >> - This program is free software; you can redistribute it and/or >> modify >> - it under the terms of the GNU General Public License as >> published by >> - the Free Software Foundation; either version 2 of the License, or >> - (at your option) any later version. >> - >> - This program is distributed in the hope that it will be useful, >> - but WITHOUT ANY WARRANTY; without even the implied warranty of >> - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> - GNU General Public License for more details. >> - >> - You should have received a copy of the GNU General Public >> License along >> - with this program; if not, write to the Free Software >> Foundation, Inc., >> - 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. >> - >> -Also add information on how to contact you by electronic and paper >> mail. - >> -If the program is interactive, make it output a short notice like >> this -when it starts in an interactive mode: >> - >> - Gnomovision version 69, Copyright (C) year name of author >> - Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type >> `show w'. >> - This is free software, and you are welcome to redistribute it >> - under certain conditions; type `show c' for details. >> - >> -The hypothetical commands `show w' and `show c' should show the >> appropriate -parts of the General Public License. Of course, the >> commands you use may -be called something other than `show w' and >> `show c'; they could even be -mouse-clicks or menu items--whatever >> suits your program. - >> -You should also get your employer (if you work as a programmer) or >> your -school, if any, to sign a "copyright disclaimer" for the >> program, if -necessary. Here is a sample; alter the names: >> - >> - Yoyodyne, Inc., hereby disclaims all copyright interest in the >> program >> - `Gnomovision' (which makes passes at compilers) written by James >> Hacker. - >> - , 1 April 1989 >> - Ty Coon, President of Vice >> - >> -This General Public License does not permit incorporating your >> program into -proprietary programs. If your program is a subroutine >> library, you may -consider it more useful to permit linking >> proprietary applications with the -library. If this is what you want >> to do, use the GNU Lesser General -Public License instead of this >> License. +This enables machine processing of license information >> based on the SPDX +License Identifiers that are here available: >> http://spdx.org/licenses/ diff --git a/bitbake/LICENSE.MIT >> b/bitbake/LICENSE.MIT new file mode 100644 >> index 0000000..a6919eb >> --- /dev/null >> +++ b/bitbake/LICENSE.MIT >> @@ -0,0 +1,25 @@ >> +Permission is hereby granted, free of charge, to any person >> obtaining a copy +of this software and associated documentation files >> (the "Software"), to deal +in the Software without restriction, >> including without limitation the rights +to use, copy, modify, merge, >> publish, distribute, sublicense, and/or sell +copies of the Software, >> and to permit persons to whom the Software is +furnished to do so, >> subject to the following conditions: + >> +The above copyright notice and this permission notice shall be >> included in +all copies or substantial portions of the Software. >> + >> +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, >> EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF >> MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND >> NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS >> BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN >> ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN >> CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE >> +SOFTWARE. + >> +Note: >> +Individual files contain the following tag instead of the full >> license text. + >> + SPDX-License-Identifier: MIT >> + >> +This enables machine processing of license information based on the >> SPDX +License Identifiers that are here available: >> http://spdx.org/licenses/ diff --git a/bitbake/MANIFEST.in >> b/bitbake/MANIFEST.in index b197378..f24969a 100644 >> --- a/bitbake/MANIFEST.in >> +++ b/bitbake/MANIFEST.in >> @@ -1,6 +1,8 @@ >> -include COPYING >> include ChangeLog >> include AUTHORS >> +include LICENSE >> +include LICENSE.GPL-2.0-only >> +include LICENSE.MIT >> include contrib/* >> include contrib/vim/*/* >> include conf/* >> @@ -8,4 +10,3 @@ include classes/* >> include doc/* >> include doc/manual/* >> include ez_setup.py >> -include HEADER >> diff --git a/bitbake/bin/bitbake b/bitbake/bin/bitbake >> index 57dec2a..66d08f8 100755 >> --- a/bitbake/bin/bitbake >> +++ b/bitbake/bin/bitbake >> @@ -1,6 +1,4 @@ >> #!/usr/bin/env python3 >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> # >> # Copyright (C) 2003, 2004 Chris Larson >> # Copyright (C) 2003, 2004 Phil Blundell >> @@ -9,18 +7,8 @@ >> # Copyright (C) 2005 ROAD GmbH >> # Copyright (C) 2006 Richard Purdie >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> +# SPDX-License-Identifier: GPL-2.0-only >> # >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. >> import os >> import sys >> @@ -38,7 +26,7 @@ from bb.main import bitbake_main, >> BitBakeConfigParameters, BBMainException if >> sys.getfilesystemencoding() != "utf-8": sys.exit("Please use a locale >> setting which supports UTF-8 (such as LANG=en_US.UTF-8).\nPython >> can't change the filesystem locale after loading so we need a UTF-8 >> when Python starts or things won't work.") -__version__ = "1.40.0" >> +__version__ = "1.44.0" >> if __name__ == "__main__": >> if __version__ != bb.__version__: >> diff --git a/bitbake/bin/bitbake-diffsigs >> b/bitbake/bin/bitbake-diffsigs index 4e6bbdd..19420a2 100755 >> --- a/bitbake/bin/bitbake-diffsigs >> +++ b/bitbake/bin/bitbake-diffsigs >> @@ -1,27 +1,16 @@ >> #!/usr/bin/env python3 >> >> -# bitbake-diffsigs >> -# BitBake task signature data comparison utility >> +# bitbake-diffsigs / bitbake-dumpsig >> +# BitBake task signature data dump and comparison utility >> # >> # Copyright (C) 2012-2013, 2017 Intel Corporation >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> +# SPDX-License-Identifier: GPL-2.0-only >> # >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. >> import os >> import sys >> import warnings >> -import fnmatch >> import argparse >> import logging >> import pickle >> @@ -32,7 +21,10 @@ import bb.tinfoil >> import bb.siggen >> import bb.msg >> >> -logger = bb.msg.logger_create('bitbake-diffsigs') >> +myname = os.path.basename(sys.argv[0]) >> +logger = bb.msg.logger_create(myname) >> + >> +is_dump = myname == 'bitbake-dumpsig' >> >> def find_siginfo(tinfoil, pn, taskname, sigs=None): >> result = None >> @@ -59,8 +51,8 @@ def find_siginfo(tinfoil, pn, taskname, sigs=None): >> sys.exit(2) >> return result >> >> -def find_compare_task(bbhandler, pn, taskname, sig1=None, sig2=None, >> color=False): >> - """ Find the most recent signature files for the specified >> PN/task and compare them """ +def find_siginfo_task(bbhandler, pn, >> taskname, sig1=None, sig2=None): >> + """ Find the most recent signature files for the specified >> PN/task """ >> if not taskname.startswith('do_'): >> taskname = 'do_%s' % taskname >> @@ -79,73 +71,81 @@ def find_compare_task(bbhandler, pn, taskname, >> sig1=None, sig2=None, color=False latestfiles = [sigfiles[sig1], >> sigfiles[sig2]] else: >> filedates = find_siginfo(bbhandler, pn, taskname) >> - latestfiles = sorted(filedates.keys(), key=lambda f: >> filedates[f])[-3:] >> + latestfiles = sorted(filedates.keys(), key=lambda f: >> filedates[f])[-2:] if not latestfiles: >> logger.error('No sigdata files found matching %s %s' % >> (pn, taskname)) sys.exit(1) >> - elif len(latestfiles) < 2: >> - logger.error('Only one matching sigdata file found for >> the specified task (%s %s)' % (pn, taskname)) >> - sys.exit(1) >> >> - # Define recursion callback >> - def recursecb(key, hash1, hash2): >> - hashes = [hash1, hash2] >> - hashfiles = find_siginfo(bbhandler, key, None, hashes) >> - >> - recout = [] >> - if len(hashfiles) == 0: >> - recout.append("Unable to find matching sigdata for %s >> with hashes %s or %s" % (key, hash1, hash2)) >> - elif not hash1 in hashfiles: >> - recout.append("Unable to find matching sigdata for %s >> with hash %s" % (key, hash1)) >> - elif not hash2 in hashfiles: >> - recout.append("Unable to find matching sigdata for %s >> with hash %s" % (key, hash2)) >> - else: >> - out2 = bb.siggen.compare_sigfiles(hashfiles[hash1], >> hashfiles[hash2], recursecb, color=color) >> - for change in out2: >> - for line in change.splitlines(): >> - recout.append(' ' + line) >> + return latestfiles >> + >> >> - return recout >> +# Define recursion callback >> +def recursecb(key, hash1, hash2): >> + hashes = [hash1, hash2] >> + hashfiles = find_siginfo(tinfoil, key, None, hashes) >> >> - # Recurse into signature comparison >> - logger.debug("Signature file (previous): %s" % latestfiles[-2]) >> - logger.debug("Signature file (latest): %s" % latestfiles[-1]) >> - output = bb.siggen.compare_sigfiles(latestfiles[-2], >> latestfiles[-1], recursecb, color=color) >> - if output: >> - print('\n'.join(output)) >> - sys.exit(0) >> + recout = [] >> + if len(hashfiles) == 0: >> + recout.append("Unable to find matching sigdata for %s with >> hashes %s or %s" % (key, hash1, hash2)) >> + elif not hash1 in hashfiles: >> + recout.append("Unable to find matching sigdata for %s with >> hash %s" % (key, hash1)) >> + elif not hash2 in hashfiles: >> + recout.append("Unable to find matching sigdata for %s with >> hash %s" % (key, hash2)) >> + else: >> + out2 = bb.siggen.compare_sigfiles(hashfiles[hash1], >> hashfiles[hash2], recursecb, color=color) >> + for change in out2: >> + for line in change.splitlines(): >> + recout.append(' ' + line) >> >> + return recout >> >> >> parser = argparse.ArgumentParser( >> - description="Compares siginfo/sigdata files written out by >> BitBake") >> + description=("Dumps" if is_dump else "Compares") + " >> siginfo/sigdata files written out by BitBake") >> -parser.add_argument('-d', '--debug', >> +parser.add_argument('-D', '--debug', >> help='Enable debug output', >> action='store_true') >> >> -parser.add_argument('--color', >> - help='Colorize output (where %(metavar)s is %(choices)s)', >> - choices=['auto', 'always', 'never'], default='auto', >> metavar='color') +if is_dump: >> + parser.add_argument("-t", "--task", >> + help="find the signature data file for the last run of >> the specified task", >> + action="store", dest="taskargs", nargs=2, >> metavar=('recipename', 'taskname')) + >> + parser.add_argument("sigdatafile1", >> + help="Signature file to dump. Not used when using >> -t/--task.", >> + action="store", nargs='?', metavar="sigdatafile") >> +else: >> + parser.add_argument('-c', '--color', >> + help='Colorize the output (where %(metavar)s is >> %(choices)s)', >> + choices=['auto', 'always', 'never'], default='auto', >> metavar='color') >> -parser.add_argument("-t", "--task", >> - help="find the signature data files for last two runs of the >> specified task and compare them", >> - action="store", dest="taskargs", nargs=2, >> metavar=('recipename', 'taskname')) >> + parser.add_argument('-d', '--dump', >> + help='Dump the last signature data instead of comparing >> (equivalent to using bitbake-dumpsig)', >> + action='store_true') >> >> -parser.add_argument("-s", "--signature", >> - help="With -t/--task, specify the signatures to look for >> instead of taking the last two", >> - action="store", dest="sigargs", nargs=2, metavar=('fromsig', >> 'tosig')) >> + parser.add_argument("-t", "--task", >> + help="find the signature data files for the last two >> runs of the specified task and compare them", >> + action="store", dest="taskargs", nargs=2, >> metavar=('recipename', 'taskname')) >> -parser.add_argument("sigdatafile1", >> - help="First signature file to compare (or signature file to >> dump, if second not specified). Not used when using -t/--task.", >> - action="store", nargs='?') >> + parser.add_argument("-s", "--signature", >> + help="With -t/--task, specify the signatures to look for >> instead of taking the last two", >> + action="store", dest="sigargs", nargs=2, >> metavar=('fromsig', 'tosig')) >> -parser.add_argument("sigdatafile2", >> - help="Second signature file to compare", >> - action="store", nargs='?') >> + parser.add_argument("sigdatafile1", >> + help="First signature file to compare (or signature file >> to dump, if second not specified). Not used when using -t/--task.", >> + action="store", nargs='?') >> >> + parser.add_argument("sigdatafile2", >> + help="Second signature file to compare", >> + action="store", nargs='?') >> >> options = parser.parse_args() >> +if is_dump: >> + options.color = 'never' >> + options.dump = True >> + options.sigdatafile2 = None >> + options.sigargs = None >> >> if options.debug: >> logger.setLevel(logging.DEBUG) >> @@ -155,17 +155,32 @@ color = (options.color == 'always' or >> (options.color == 'auto' and sys.stdout.is if options.taskargs: >> with bb.tinfoil.Tinfoil() as tinfoil: >> tinfoil.prepare(config_only=True) >> - if options.sigargs: >> - find_compare_task(tinfoil, options.taskargs[0], >> options.taskargs[1], options.sigargs[0], options.sigargs[1], >> color=color) >> + if not options.dump and options.sigargs: >> + files = find_siginfo_task(tinfoil, options.taskargs[0], >> options.taskargs[1], options.sigargs[0], options.sigargs[1]) >> + else: >> + files = find_siginfo_task(tinfoil, options.taskargs[0], >> options.taskargs[1]) + >> + if options.dump: >> + logger.debug("Signature file: %s" % files[-1]) >> + output = bb.siggen.dump_sigfile(files[-1]) >> else: >> - find_compare_task(tinfoil, options.taskargs[0], >> options.taskargs[1], color=color) >> + if len(files) < 2: >> + logger.error('Only one matching sigdata file found >> for the specified task (%s %s)' % (options.taskargs[0], >> options.taskargs[1])) >> + sys.exit(1) >> + >> + # Recurse into signature comparison >> + logger.debug("Signature file (previous): %s" % files[-2]) >> + logger.debug("Signature file (latest): %s" % files[-1]) >> + output = bb.siggen.compare_sigfiles(files[-2], >> files[-1], recursecb, color=color) else: >> if options.sigargs: >> logger.error('-s/--signature can only be used together with >> -t/--task') sys.exit(1) >> try: >> - if options.sigdatafile1 and options.sigdatafile2: >> - output = >> bb.siggen.compare_sigfiles(options.sigdatafile1, >> options.sigdatafile2, color=color) >> + if not options.dump and options.sigdatafile1 and >> options.sigdatafile2: >> + with bb.tinfoil.Tinfoil() as tinfoil: >> + tinfoil.prepare(config_only=True) >> + output = >> bb.siggen.compare_sigfiles(options.sigdatafile1, >> options.sigdatafile2, recursecb, color=color) elif >> options.sigdatafile1: output = >> bb.siggen.dump_sigfile(options.sigdatafile1) else: @@ -179,5 +194,5 >> @@ else: logger.error('Invalid signature data - ensure you are >> specifying sigdata/siginfo files') sys.exit(1) >> >> - if output: >> - print('\n'.join(output)) >> +if output: >> + print('\n'.join(output)) >> diff --git a/bitbake/bin/bitbake-dumpsig b/bitbake/bin/bitbake-dumpsig >> deleted file mode 100755 >> index 95ebd93..0000000 >> --- a/bitbake/bin/bitbake-dumpsig >> +++ /dev/null >> @@ -1,94 +0,0 @@ >> -#!/usr/bin/env python3 >> - >> -# bitbake-dumpsig >> -# BitBake task signature dump utility >> -# >> -# Copyright (C) 2013 Intel Corporation >> -# >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> -# >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. - >> -import os >> -import sys >> -import warnings >> -import optparse >> -import logging >> -import pickle >> - >> -sys.path.insert(0, >> os.path.join(os.path.dirname(os.path.dirname(sys.argv[0])), 'lib')) - >> -import bb.tinfoil >> -import bb.siggen >> -import bb.msg >> - >> -logger = bb.msg.logger_create('bitbake-dumpsig') >> - >> -def find_siginfo_task(bbhandler, pn, taskname): >> - """ Find the most recent signature file for the specified >> PN/task """ - >> - if not hasattr(bb.siggen, 'find_siginfo'): >> - logger.error('Metadata does not support finding signature >> data files') >> - sys.exit(1) >> - >> - if not taskname.startswith('do_'): >> - taskname = 'do_%s' % taskname >> - >> - filedates = bb.siggen.find_siginfo(pn, taskname, None, >> bbhandler.config_data) >> - latestfiles = sorted(filedates.keys(), key=lambda f: >> filedates[f])[-1:] >> - if not latestfiles: >> - logger.error('No sigdata files found matching %s %s' % (pn, >> taskname)) >> - sys.exit(1) >> - >> - return latestfiles[0] >> - >> -parser = optparse.OptionParser( >> - description = "Dumps siginfo/sigdata files written out by >> BitBake", >> - usage = """ >> - %prog -t recipename taskname >> - %prog sigdatafile""") >> - >> -parser.add_option("-D", "--debug", >> - help = "enable debug", >> - action = "store_true", dest="debug", default = False) >> - >> -parser.add_option("-t", "--task", >> - help = "find the signature data file for the specified task", >> - action="store", dest="taskargs", nargs=2, >> metavar='recipename taskname') - >> -options, args = parser.parse_args(sys.argv) >> - >> -if options.debug: >> - logger.setLevel(logging.DEBUG) >> - >> -if options.taskargs: >> - tinfoil = bb.tinfoil.Tinfoil() >> - tinfoil.prepare(config_only = True) >> - file = find_siginfo_task(tinfoil, options.taskargs[0], >> options.taskargs[1]) >> - logger.debug("Signature file: %s" % file) >> -elif len(args) == 1: >> - parser.print_help() >> - sys.exit(0) >> -else: >> - file = args[1] >> - >> -try: >> - output = bb.siggen.dump_sigfile(file) >> -except IOError as e: >> - logger.error(str(e)) >> - sys.exit(1) >> -except (pickle.UnpicklingError, EOFError): >> - logger.error('Invalid signature data - ensure you are specifying >> a sigdata/siginfo file') >> - sys.exit(1) >> - >> -if output: >> - print('\n'.join(output)) >> diff --git a/bitbake/bin/bitbake-dumpsig b/bitbake/bin/bitbake-dumpsig >> new file mode 120000 >> index 0000000..b1e8489 >> --- /dev/null >> +++ b/bitbake/bin/bitbake-dumpsig >> @@ -0,0 +1 @@ >> +bitbake-diffsigs >> \ No newline at end of file >> diff --git a/bitbake/bin/bitbake-hashclient >> b/bitbake/bin/bitbake-hashclient new file mode 100755 >> index 0000000..29ab65f >> --- /dev/null >> +++ b/bitbake/bin/bitbake-hashclient >> @@ -0,0 +1,170 @@ >> +#! /usr/bin/env python3 >> +# >> +# Copyright (C) 2019 Garmin Ltd. >> +# >> +# SPDX-License-Identifier: GPL-2.0-only >> +# >> + >> +import argparse >> +import hashlib >> +import logging >> +import os >> +import pprint >> +import sys >> +import threading >> +import time >> + >> +try: >> + import tqdm >> + ProgressBar = tqdm.tqdm >> +except ImportError: >> + class ProgressBar(object): >> + def __init__(self, *args, **kwargs): >> + pass >> + >> + def __enter__(self): >> + return self >> + >> + def __exit__(self, *args, **kwargs): >> + pass >> + >> + def update(self): >> + pass >> + >> +sys.path.insert(0, >> os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib')) + >> +import hashserv >> + >> +DEFAULT_ADDRESS = 'unix://./hashserve.sock' >> +METHOD = 'stress.test.method' >> + >> + >> +def main(): >> + def handle_stats(args, client): >> + if args.reset: >> + s = client.reset_stats() >> + else: >> + s = client.get_stats() >> + pprint.pprint(s) >> + return 0 >> + >> + def handle_stress(args, client): >> + def thread_main(pbar, lock): >> + nonlocal found_hashes >> + nonlocal missed_hashes >> + nonlocal max_time >> + >> + client = hashserv.create_client(args.address) >> + >> + for i in range(args.requests): >> + taskhash = hashlib.sha256() >> + taskhash.update(args.taskhash_seed.encode('utf-8')) >> + taskhash.update(str(i).encode('utf-8')) >> + >> + start_time = time.perf_counter() >> + l = client.get_unihash(METHOD, taskhash.hexdigest()) >> + elapsed = time.perf_counter() - start_time >> + >> + with lock: >> + if l: >> + found_hashes += 1 >> + else: >> + missed_hashes += 1 >> + >> + max_time = max(elapsed, max_time) >> + pbar.update() >> + >> + max_time = 0 >> + found_hashes = 0 >> + missed_hashes = 0 >> + lock = threading.Lock() >> + total_requests = args.clients * args.requests >> + start_time = time.perf_counter() >> + with ProgressBar(total=total_requests) as pbar: >> + threads = [threading.Thread(target=thread_main, >> args=(pbar, lock), daemon=False) for _ in range(args.clients)] >> + for t in threads: >> + t.start() >> + >> + for t in threads: >> + t.join() >> + >> + elapsed = time.perf_counter() - start_time >> + with lock: >> + print("%d requests in %.1fs. %.1f requests per second" % >> (total_requests, elapsed, total_requests / elapsed)) >> + print("Average request time %.8fs" % (elapsed / >> total_requests)) >> + print("Max request time was %.8fs" % max_time) >> + print("Found %d hashes, missed %d" % (found_hashes, >> missed_hashes)) + >> + if args.report: >> + with ProgressBar(total=args.requests) as pbar: >> + for i in range(args.requests): >> + taskhash = hashlib.sha256() >> + >> taskhash.update(args.taskhash_seed.encode('utf-8')) >> + taskhash.update(str(i).encode('utf-8')) >> + >> + outhash = hashlib.sha256() >> + outhash.update(args.outhash_seed.encode('utf-8')) >> + outhash.update(str(i).encode('utf-8')) >> + >> + client.report_unihash(taskhash.hexdigest(), >> METHOD, outhash.hexdigest(), taskhash.hexdigest()) + >> + with lock: >> + pbar.update() >> + >> + parser = argparse.ArgumentParser(description='Hash Equivalence >> Client') >> + parser.add_argument('--address', default=DEFAULT_ADDRESS, >> help='Server address (default "%(default)s")') >> + parser.add_argument('--log', default='WARNING', help='Set >> logging level') + >> + subparsers = parser.add_subparsers() >> + >> + stats_parser = subparsers.add_parser('stats', help='Show server >> stats') >> + stats_parser.add_argument('--reset', action='store_true', >> + help='Reset server stats') >> + stats_parser.set_defaults(func=handle_stats) >> + >> + stress_parser = subparsers.add_parser('stress', help='Run stress >> test') >> + stress_parser.add_argument('--clients', type=int, default=10, >> + help='Number of simultaneous clients') >> + stress_parser.add_argument('--requests', type=int, default=1000, >> + help='Number of requests each client >> will perform') >> + stress_parser.add_argument('--report', action='store_true', >> + help='Report new hashes') >> + stress_parser.add_argument('--taskhash-seed', default='', >> + help='Include string in taskhash') >> + stress_parser.add_argument('--outhash-seed', default='', >> + help='Include string in outhash') >> + stress_parser.set_defaults(func=handle_stress) >> + >> + args = parser.parse_args() >> + >> + logger = logging.getLogger('hashserv') >> + >> + level = getattr(logging, args.log.upper(), None) >> + if not isinstance(level, int): >> + raise ValueError('Invalid log level: %s' % args.log) >> + >> + logger.setLevel(level) >> + console = logging.StreamHandler() >> + console.setLevel(level) >> + logger.addHandler(console) >> + >> + func = getattr(args, 'func', None) >> + if func: >> + client = hashserv.create_client(args.address) >> + # Try to establish a connection to the server now to detect >> failures >> + # early >> + client.connect() >> + >> + return func(args, client) >> + >> + return 0 >> + >> + >> +if __name__ == '__main__': >> + try: >> + ret = main() >> + except Exception: >> + ret = 1 >> + import traceback >> + traceback.print_exc() >> + sys.exit(ret) >> diff --git a/bitbake/bin/bitbake-hashserv >> b/bitbake/bin/bitbake-hashserv new file mode 100755 >> index 0000000..1bc1f91 >> --- /dev/null >> +++ b/bitbake/bin/bitbake-hashserv >> @@ -0,0 +1,62 @@ >> +#! /usr/bin/env python3 >> +# >> +# Copyright (C) 2018 Garmin Ltd. >> +# >> +# SPDX-License-Identifier: GPL-2.0-only >> +# >> + >> +import os >> +import sys >> +import logging >> +import argparse >> +import sqlite3 >> + >> +sys.path.insert(0, >> os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib')) + >> +import hashserv >> + >> +VERSION = "1.0.0" >> + >> +DEFAULT_BIND = 'unix://./hashserve.sock' >> + >> + >> +def main(): >> + parser = argparse.ArgumentParser(description='Hash Equivalence >> Reference Server. Version=%s' % VERSION, >> + epilog='''The bind address is >> the path to a unix domain socket if it is >> + prefixed with >> "unix://". Otherwise, it is an IP address >> + and port in form >> ADDRESS:PORT. To bind to all addresses, leave >> + the ADDRESS empty, >> e.g. "--bind :8686". To bind to a specific >> + IPv6 address, enclose >> the address in "[]", e.g. >> + "--bind [::1]:8686"''' >> + ) >> + >> + parser.add_argument('--bind', default=DEFAULT_BIND, help='Bind >> address (default "%(default)s")') >> + parser.add_argument('--database', default='./hashserv.db', >> help='Database file (default "%(default)s")') >> + parser.add_argument('--log', default='WARNING', help='Set >> logging level') + >> + args = parser.parse_args() >> + >> + logger = logging.getLogger('hashserv') >> + >> + level = getattr(logging, args.log.upper(), None) >> + if not isinstance(level, int): >> + raise ValueError('Invalid log level: %s' % args.log) >> + >> + logger.setLevel(level) >> + console = logging.StreamHandler() >> + console.setLevel(level) >> + logger.addHandler(console) >> + >> + server = hashserv.create_server(args.bind, args.database) >> + server.serve_forever() >> + return 0 >> + >> + >> +if __name__ == '__main__': >> + try: >> + ret = main() >> + except Exception: >> + ret = 1 >> + import traceback >> + traceback.print_exc() >> + sys.exit(ret) >> diff --git a/bitbake/bin/bitbake-layers b/bitbake/bin/bitbake-layers >> index d184011..a884dc1 100755 >> --- a/bitbake/bin/bitbake-layers >> +++ b/bitbake/bin/bitbake-layers >> @@ -7,18 +7,8 @@ >> # Copyright (C) 2011 Mentor Graphics Corporation >> # Copyright (C) 2011-2015 Intel Corporation >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> +# SPDX-License-Identifier: GPL-2.0-only >> # >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. >> import logging >> import os >> diff --git a/bitbake/bin/bitbake-prserv b/bitbake/bin/bitbake-prserv >> index f38d2dd..1e9b6cb 100755 >> --- a/bitbake/bin/bitbake-prserv >> +++ b/bitbake/bin/bitbake-prserv >> @@ -1,4 +1,8 @@ >> #!/usr/bin/env python3 >> +# >> +# SPDX-License-Identifier: GPL-2.0-only >> +# >> + >> import os >> import sys,logging >> import optparse >> diff --git a/bitbake/bin/bitbake-selftest >> b/bitbake/bin/bitbake-selftest index cfa7ac5..041a271 100755 >> --- a/bitbake/bin/bitbake-selftest >> +++ b/bitbake/bin/bitbake-selftest >> @@ -2,18 +2,8 @@ >> # >> # Copyright (C) 2012 Richard Purdie >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> +# SPDX-License-Identifier: GPL-2.0-only >> # >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. >> import os >> import sys, logging >> @@ -22,6 +12,7 @@ sys.path.insert(0, >> os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib import >> unittest try: >> import bb >> + import hashserv >> import layerindexlib >> except RuntimeError as exc: >> sys.exit(str(exc)) >> @@ -33,7 +24,10 @@ tests = ["bb.tests.codeparser", >> "bb.tests.event", >> "bb.tests.fetch", >> "bb.tests.parse", >> + "bb.tests.persist_data", >> + "bb.tests.runqueue", >> "bb.tests.utils", >> + "hashserv.tests", >> "layerindexlib.tests.layerindexobj", >> "layerindexlib.tests.restapi", >> "layerindexlib.tests.cooker"] >> diff --git a/bitbake/bin/bitbake-worker b/bitbake/bin/bitbake-worker >> index e925054..6776cad 100755 >> --- a/bitbake/bin/bitbake-worker >> +++ b/bitbake/bin/bitbake-worker >> @@ -1,4 +1,7 @@ >> #!/usr/bin/env python3 >> +# >> +# SPDX-License-Identifier: GPL-2.0-only >> +# >> >> import os >> import sys >> @@ -136,7 +139,7 @@ def sigterm_handler(signum, frame): >> os.killpg(0, signal.SIGTERM) >> sys.exit() >> >> -def fork_off_task(cfg, data, databuilder, workerdata, fn, task, >> taskname, appends, taskdepdata, extraconfigdata, quieterrors=False, >> dry_run_exec=False): +def fork_off_task(cfg, data, databuilder, >> workerdata, fn, task, taskname, taskhash, unihash, appends, >> taskdepdata, extraconfigdata, quieterrors=False, dry_run_exec=False): >> # We need to setup the environment BEFORE the fork, since # a fork() >> or exec*() activates PSEUDO... @@ -231,10 +234,13 @@ def >> fork_off_task(cfg, data, databuilder, workerdata, fn, task, taskname, >> append the_data.setVar(varname, value) >> bb.parse.siggen.set_taskdata(workerdata["sigdata"]) >> + if "newhashes" in workerdata: >> + >> bb.parse.siggen.set_taskhashes(workerdata["newhashes"]) ret = 0 >> >> the_data = bb_cache.loadDataFull(fn, appends) >> - the_data.setVar('BB_TASKHASH', >> workerdata["runq_hash"][task]) >> + the_data.setVar('BB_TASKHASH', taskhash) >> + the_data.setVar('BB_UNIHASH', unihash) >> >> bb.utils.set_process_name("%s:%s" % >> (the_data.getVar("PN"), taskname.replace("do_", ""))) >> @@ -373,6 +379,7 @@ class BitbakeWorker(object): >> self.handle_item(b"cookerconfig", >> self.handle_cookercfg) self.handle_item(b"extraconfigdata", >> self.handle_extraconfigdata) self.handle_item(b"workerdata", >> self.handle_workerdata) >> + self.handle_item(b"newtaskhashes", >> self.handle_newtaskhashes) self.handle_item(b"runtask", >> self.handle_runtask) self.handle_item(b"finishnow", >> self.handle_finishnow) self.handle_item(b"ping", self.handle_ping) >> @@ -411,6 +418,10 @@ class BitbakeWorker(object): >> bb.msg.loggerDefaultDomains = >> self.workerdata["logdefaultdomain"] for mc in self.databuilder.mcdata: >> self.databuilder.mcdata[mc].setVar("PRSERV_HOST", >> self.workerdata["prhost"]) >> + self.databuilder.mcdata[mc].setVar("BB_HASHSERVE", >> self.workerdata["hashservaddr"]) + >> + def handle_newtaskhashes(self, data): >> + self.workerdata["newhashes"] = pickle.loads(data) >> >> def handle_ping(self, _): >> workerlog_write("Handling ping\n") >> @@ -425,10 +436,10 @@ class BitbakeWorker(object): >> sys.exit(0) >> >> def handle_runtask(self, data): >> - fn, task, taskname, quieterrors, appends, taskdepdata, >> dry_run_exec = pickle.loads(data) >> + fn, task, taskname, taskhash, unihash, quieterrors, appends, >> taskdepdata, dry_run_exec = pickle.loads(data) >> workerlog_write("Handling runtask %s %s %s\n" % (task, fn, taskname)) >> - pid, pipein, pipeout = fork_off_task(self.cookercfg, >> self.data, self.databuilder, self.workerdata, fn, task, taskname, >> appends, taskdepdata, self.extraconfigdata, quieterrors, dry_run_exec) >> + pid, pipein, pipeout = fork_off_task(self.cookercfg, >> self.data, self.databuilder, self.workerdata, fn, task, taskname, >> taskhash, unihash, appends, taskdepdata, self.extraconfigdata, >> quieterrors, dry_run_exec) self.build_pids[pid] = task >> self.build_pipes[pid] = runQueueWorkerPipe(pipein, pipeout) diff >> --git a/bitbake/bin/bitdoc b/bitbake/bin/bitdoc index >> 2744678..9bd02be 100755 --- a/bitbake/bin/bitdoc >> +++ b/bitbake/bin/bitdoc >> @@ -1,21 +1,9 @@ >> #!/usr/bin/env python3 >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> # >> # Copyright (C) 2005 Holger Hans Peter Freyther >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> +# SPDX-License-Identifier: GPL-2.0-only >> # >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. >> import optparse, os, sys >> >> diff --git a/bitbake/bin/git-make-shallow >> b/bitbake/bin/git-make-shallow index 296d3a3..57069f7 100755 >> --- a/bitbake/bin/git-make-shallow >> +++ b/bitbake/bin/git-make-shallow >> @@ -1,4 +1,8 @@ >> #!/usr/bin/env python3 >> +# >> +# SPDX-License-Identifier: GPL-2.0-only >> +# >> + >> """git-make-shallow: make the current git repository shallow >> >> Remove the history of the specified revisions, then optionally >> filter the diff --git a/bitbake/bin/toaster b/bitbake/bin/toaster >> index ecf66fa..c3472df 100755 >> --- a/bitbake/bin/toaster >> +++ b/bitbake/bin/toaster >> @@ -3,19 +3,9 @@ >> # toaster - shell script to start Toaster >> >> # Copyright (C) 2013-2015 Intel Corp. >> - >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License as >> published by -# the Free Software Foundation; either version 2 of the >> License, or -# (at your option) any later version. >> # >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> +# SPDX-License-Identifier: GPL-2.0-or-later >> # >> -# You should have received a copy of the GNU General Public License >> -# along with this program. If not, see http://www.gnu.org/licenses/. >> >> HELP=" >> Usage: source toaster start|stop [webport=] [noweb] >> [nobuild] [toasterdir] diff --git a/bitbake/bin/toaster-eventreplay >> b/bitbake/bin/toaster-eventreplay index 80967a0..8fa4ab7 100755 >> --- a/bitbake/bin/toaster-eventreplay >> +++ b/bitbake/bin/toaster-eventreplay >> @@ -1,25 +1,12 @@ >> #!/usr/bin/env python3 >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> # >> # Copyright (C) 2014 Alex Damian >> # >> +# SPDX-License-Identifier: GPL-2.0-only >> +# >> # This file re-uses code spread throughout other Bitbake source >> files. # As such, all other copyrights belong to their own right >> holders. # >> -# >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> -# >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. >> """ >> This command takes a filename as a single parameter. The filename is >> read diff --git a/bitbake/classes/base.bbclass >> b/bitbake/classes/base.bbclass index 71d9ec1..08441fe 100644 >> --- a/bitbake/classes/base.bbclass >> +++ b/bitbake/classes/base.bbclass >> @@ -44,7 +44,7 @@ python do_showdata() { >> # emit the metadata which isnt valid shell >> for e in bb.data.keys(d): >> if d.getVarFlag(e, 'python', False): >> - bb.plain("\npython %s () {\n%s}" % (e, >> d.getVar(e, True))) >> + bb.plain("\npython %s () {\n%s}" % (e, >> d.getVar(e))) } >> >> addtask listtasks >> diff --git a/bitbake/contrib/dump_cache.py >> b/bitbake/contrib/dump_cache.py index 8963ca4..c6723cb 100755 >> --- a/bitbake/contrib/dump_cache.py >> +++ b/bitbake/contrib/dump_cache.py >> @@ -1,6 +1,4 @@ >> #!/usr/bin/env python3 >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> # >> # Copyright (C) 2012, 2018 Wind River Systems, Inc. >> # >> diff --git >> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml >> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml >> index f1caaec..46dafee 100644 --- >> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml >> +++ >> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml >> @@ -31,7 +31,7 @@ Prior to executing BitBake, you should take >> advantage of available parallel thread execution on your build host >> by setting the >> - > linkend='var-BB_NUMBER_THREADS'>BB_NUMBER_THREADS >> + > linkend='var-bb-BB_NUMBER_THREADS'>BB_NUMBER_THREADS >> variable in your project's local.conf >> configuration file. >> @@ -87,9 +87,9 @@ >> >> The layer.conf files are used to >> construct key variables such as >> - > linkend='var-BBPATH'>BBPATH >> + > linkend='var-bb-BBPATH'>BBPATH and >> - > linkend='var-BBFILES'>BBFILES. >> + > linkend='var-bb-BBFILES'>BBFILES. >> BBPATH is used to search for configuration and >> class files under the conf and >> classes @@ -117,19 +117,19 @@ >> at certain variables, including: >> >> >> - > linkend='var-BB_ENV_WHITELIST'>BB_ENV_WHITELIST >> + > linkend='var-bb-BB_ENV_WHITELIST'>BB_ENV_WHITELIST >> >> - > linkend='var-BB_ENV_EXTRAWHITE'>BB_ENV_EXTRAWHITE >> + > linkend='var-bb-BB_ENV_EXTRAWHITE'>BB_ENV_EXTRAWHITE >> >> - > linkend='var-BB_PRESERVE_ENV'>BB_PRESERVE_ENV >> + > linkend='var-bb-BB_PRESERVE_ENV'>BB_PRESERVE_ENV >> >> - > linkend='var-BB_ORIGENV'>BB_ORIGENV >> + > linkend='var-bb-BB_ORIGENV'>BB_ORIGENV >> >> - > linkend='var-BITBAKE_UI'>BITBAKE_UI >> + > linkend='var-bb-BITBAKE_UI'>BITBAKE_UI >> >> The first four variables in this list relate to how >> BitBake treats shell @@ -156,7 +156,7 @@ >> BitBake first searches the current working directory for >> an optional conf/bblayers.conf configuration >> file. This file is expected to contain a >> - > linkend='var-BBLAYERS'>BBLAYERS >> + > linkend='var-bb-BBLAYERS'>BBLAYERS >> variable that is a space-delimited list of 'layer' directories. >> Recall that if BitBake cannot find a >> bblayers.conf file, then it is assumed the user >> has set the BBPATH @@ -166,10 +166,10 @@ >> For each directory (layer) in this list, a >> conf/layer.conf file is located and parsed with >> the >> - > linkend='var-LAYERDIR'>LAYERDIR >> + > linkend='var-bb-LAYERDIR'>LAYERDIR >> variable being set to the directory where the layer was found. The >> idea is these files automatically set up >> - > linkend='var-BBPATH'>BBPATH >> + > linkend='var-bb-BBPATH'>BBPATH and other >> variables correctly for a given build directory. >> >> @@ -189,7 +189,7 @@ >> depending on the environment variables previously >> mentioned or set in the configuration files. >> The >> - "Variables >> Glossary" >> + "Variables >> Glossary" chapter presents a full list of variables. >> >> >> @@ -204,7 +204,7 @@ >> >> The base.bbclass file is always >> included. Other classes that are specified in the configuration using >> the >> - > linkend='var-INHERIT'>INHERIT >> + > linkend='var-bb-INHERIT'>INHERIT variable >> are also included. BitBake searches for class files in a >> classes subdirectory under >> @@ -270,7 +270,7 @@ >> >> >> During the configuration phase, BitBake will have set >> - > linkend='var-BBFILES'>BBFILES. >> + > linkend='var-bb-BBFILES'>BBFILES. BitBake >> now uses it to construct a list of recipes to parse, along with any >> append files (.bbappend) to apply. >> @@ -292,7 +292,7 @@ >> Any inherit statements cause BitBake to find and >> then parse class files (.bbclass) >> using >> - > linkend='var-BBPATH'>BBPATH >> + > linkend='var-bb-BBPATH'>BBPATH as the >> search path. Finally, BitBake parses in order any append files found >> in BBFILES. >> @@ -303,8 +303,8 @@ >> pieces of metadata. >> For example, in bitbake.conf the >> recipe name and version are used to set the variables >> - PN and >> - PV: >> + PN >> and >> + PV: >> >> PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', >> False),d)[0] or 'defaultpkgname'}" PV = >> "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[1] >> or '1.0'}" @@ -336,7 +336,7 @@ recipe information. >> The validity of this cache is determined by first >> computing a checksum of the base configuration data (see >> - > linkend='var-BB_HASHCONFIG_WHITELIST'>BB_HASHCONFIG_WHITELIST) >> + > linkend='var-bb-BB_HASHCONFIG_WHITELIST'>BB_HASHCONFIG_WHITELIST) >> and then checking if the checksum matches. If that checksum matches >> what is in the cache and the recipe and class files have not changed, >> Bitbake is able to use @@ -384,9 +384,9 @@ >> the recipe can be known. >> Each recipe's PROVIDES list is >> created implicitly through the recipe's >> - PN >> variable >> + PN >> variable and explicitly through the recipe's >> - > linkend='var-PROVIDES'>PROVIDES >> + > linkend='var-bb-PROVIDES'>PROVIDES >> variable, which is optional. >> >> @@ -427,7 +427,7 @@ >> PREFERRED_PROVIDER_virtual/kernel = "linux-yocto" >> >> The default >> - > linkend='var-PREFERRED_PROVIDER'>PREFERRED_PROVIDER >> + > linkend='var-bb-PREFERRED_PROVIDER'>PREFERRED_PROVIDER >> is the provider with the same name as the target. Bitbake iterates >> through each target it needs to build and resolves them and their >> dependencies using this process. @@ -439,10 +439,10 @@ >> BitBake defaults to the highest version of a provider. >> Version comparisons are made using the same method as >> Debian. You can use the >> - > linkend='var-PREFERRED_VERSION'>PREFERRED_VERSION >> + > linkend='var-bb-PREFERRED_VERSION'>PREFERRED_VERSION >> variable to specify a particular version. You can influence the order >> by using the >> - > linkend='var-DEFAULT_PREFERENCE'>DEFAULT_PREFERENCE >> + > linkend='var-bb-DEFAULT_PREFERENCE'>DEFAULT_PREFERENCE >> variable. >> >> @@ -464,7 +464,7 @@ >> BitBake defaults to selecting the most recent >> version, unless otherwise specified. >> If the recipe in question has a >> - > linkend='var-DEFAULT_PREFERENCE'>DEFAULT_PREFERENCE >> + > linkend='var-bb-DEFAULT_PREFERENCE'>DEFAULT_PREFERENCE >> set lower than the other recipes (default is 0), then it will not be >> selected. This allows the person or persons maintaining >> @@ -475,9 +475,9 @@ >> >> >> If the first recipe is named >> a_1.1.bb, then the >> - PN >> variable >> + PN >> variable will be set to “a”, and the >> - PV >> + PV >> variable will be set to 1.1. >> >> >> @@ -532,11 +532,11 @@ >> >> Dependencies are defined through several variables. >> You can find information about variables BitBake uses in >> - the Variables >> Glossary >> + the Variables >> Glossary near the end of this manual. >> At a basic level, it is sufficient to know that BitBake >> uses the >> - > linkend='var-DEPENDS'>DEPENDS and >> - > linkend='var-RDEPENDS'>RDEPENDS variables >> when >> + > linkend='var-bb-DEPENDS'>DEPENDS and >> + > linkend='var-bb-RDEPENDS'>RDEPENDS >> variables when calculating dependencies. >> >> @@ -560,7 +560,7 @@ >> >> >> The build now starts with BitBake forking off threads up >> to the limit set in the >> - > linkend='var-BB_NUMBER_THREADS'>BB_NUMBER_THREADS >> + > linkend='var-bb-BB_NUMBER_THREADS'>BB_NUMBER_THREADS >> variable. BitBake continues to fork threads as long as there are >> tasks ready to run, those tasks have all their dependencies met, and >> the thread threshold has not been @@ -574,7 +574,7 @@ >> >> >> As each task completes, a timestamp is written to the >> directory specified by the >> - > linkend='var-STAMP'>STAMP variable. >> + > linkend='var-bb-STAMP'>STAMP variable. On >> subsequent runs, BitBake looks in the build directory within >> tmp/stamps and does not rerun tasks that are >> already completed unless a timestamp is found to be invalid. @@ >> -618,7 +618,7 @@ >> Tasks can be either a shell task or a Python task. >> For shell tasks, BitBake writes a shell script to >> - ${> linkend='var-T'>T}/run.do_taskname.pid >> + ${> linkend='var-bb-T'>T}/run.do_taskname.pid >> and then executes the script. The generated shell script contains all >> the exported variables, and the shell functions with all variables >> expanded. @@ -645,10 +645,10 @@ >> behavior: >> >> >> - > linkend='var-BB_SCHEDULER'>BB_SCHEDULER >> + > linkend='var-bb-BB_SCHEDULER'>BB_SCHEDULER >> >> - > linkend='var-BB_SCHEDULERS'>BB_SCHEDULERS >> + > linkend='var-bb-BB_SCHEDULERS'>BB_SCHEDULERS >> >> It is possible to have functions run before and after a >> task's main @@ -684,7 +684,7 @@ >> The simplistic approach for excluding the working >> directory is to set it to some fixed value and create the checksum >> for the "run" script. BitBake goes one step better and uses the >> - > linkend='var-BB_HASHBASE_WHITELIST'>BB_HASHBASE_WHITELIST >> + > linkend='var-bb-BB_HASHBASE_WHITELIST'>BB_HASHBASE_WHITELIST >> variable to define a list of variables that should never be included >> when generating the signatures. >> @@ -795,7 +795,7 @@ >> This results in any metadata change that changes the >> task hash, automatically causing the task to be run again. >> This removes the need to bump >> - PR >> + PR >> values, and changes to metadata automatically ripple >> across the build. >> >> @@ -884,7 +884,7 @@ >> >> >> BitBake first calls the function defined by the >> - > linkend='var-BB_HASHCHECK_FUNCTION'>BB_HASHCHECK_FUNCTION >> + > linkend='var-bb-BB_HASHCHECK_FUNCTION'>BB_HASHCHECK_FUNCTION >> variable with a list of tasks and corresponding hashes it wants to >> build. This function is designed to be fast and returns a list >> @@ -908,7 +908,7 @@ >> For example, it is pointless to obtain a compiler if you >> already have the compiled binary. >> To handle this, BitBake calls the >> - > linkend='var-BB_SETSCENE_DEPVALID'>BB_SETSCENE_DEPVALID >> + > linkend='var-bb-BB_SETSCENE_DEPVALID'>BB_SETSCENE_DEPVALID >> function for each successful setscene task to know whether or not it >> needs to obtain the dependencies of that task. >> @@ -916,7 +916,7 @@ >> >> Finally, after all the setscene tasks have executed, >> BitBake calls the function listed in >> - > linkend='var-BB_SETSCENE_VERIFY_FUNCTION2'>BB_SETSCENE_VERIFY_FUNCTION2 >> + > linkend='var-bb-BB_SETSCENE_VERIFY_FUNCTION2'>BB_SETSCENE_VERIFY_FUNCTION2 >> with the list of tasks BitBake thinks has been "covered". The >> metadata can then ensure that this list is correct and can inform >> BitBake that it wants specific tasks to be run regardless diff --git >> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml >> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml >> index 29ae486..6840408 100644 --- >> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml >> +++ >> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml @@ >> -44,7 +44,7 @@ This code sets up an instance of the >> fetch class. The instance uses a space-separated list of URLs from the >> - > linkend='var-SRC_URI'>SRC_URI >> + > linkend='var-bb-SRC_URI'>SRC_URI variable >> and then calls the download method to download >> the files. >> @@ -78,7 +78,7 @@ >> Pre-mirror >> Sites: BitBake first uses pre-mirrors to try and find >> source files. These locations are defined using the >> - > linkend='var-PREMIRRORS'>PREMIRRORS >> + > linkend='var-bb-PREMIRRORS'>PREMIRRORS >> variable. >> Source URI: >> @@ -88,7 +88,7 @@ >> Mirror Sites: >> If fetch failures occur, BitBake next uses >> mirror locations as defined by the >> - > linkend='var-MIRRORS'>MIRRORS >> + > linkend='var-bb-MIRRORS'>MIRRORS variable. >> >> >> @@ -144,7 +144,7 @@ >> Any source files that are not local (i.e. >> downloaded from the Internet) are placed into the >> download directory, which is specified by the >> - > linkend='var-DL_DIR'>DL_DIR >> + > linkend='var-bb-DL_DIR'>DL_DIR variable. >> >> >> @@ -184,11 +184,11 @@ >> >> >> If >> - > linkend='var-BB_STRICT_CHECKSUM'>BB_STRICT_CHECKSUM >> + > linkend='var-bb-BB_STRICT_CHECKSUM'>BB_STRICT_CHECKSUM >> is set, any download without a checksum triggers an error message. >> The >> - > linkend='var-BB_NO_NETWORK'>BB_NO_NETWORK >> + > linkend='var-bb-BB_NO_NETWORK'>BB_NO_NETWORK >> variable can be used to make any attempted network access a fatal >> error, which is useful for checking that mirrors are complete as well >> as other things. @@ -265,11 +265,11 @@ >> The filename you specify within the URL can be >> either an absolute or relative path to a file. >> If the filename is relative, the contents of the >> - > linkend='var-FILESPATH'>FILESPATH >> + > linkend='var-bb-FILESPATH'>FILESPATH >> variable is used in the same way PATH is used to >> find executables. If the file cannot be found, it is assumed that it >> is available in >> - > linkend='var-DL_DIR'>DL_DIR >> + > linkend='var-bb-DL_DIR'>DL_DIR by the >> time the download() method is called. >> >> @@ -304,7 +304,7 @@ >> allows the name of the downloaded file to be >> specified. Specifying the name of the downloaded file is useful >> for avoiding collisions in >> - > linkend='var-DL_DIR'>DL_DIR >> + > linkend='var-bb-DL_DIR'>DL_DIR when >> dealing with multiple files that have the same name. >> >> @@ -355,7 +355,7 @@ >> A special value of "now" causes the checkout >> to be updated on every build. >> >> - > linkend='var-CVSDIR'>CVSDIR: >> + > linkend='var-bb-CVSDIR'>CVSDIR: >> Specifies where a temporary checkout is saved. The location is often >> DL_DIR/cvs. >> @@ -395,7 +395,7 @@ >> "date": >> Specifies a date. >> If no "date" is specified, the >> - > linkend='var-SRCDATE'>SRCDATE >> + > linkend='var-bb-SRCDATE'>SRCDATE of the >> configuration is used to checkout a specific date. The special value >> of "now" causes the checkout to be updated on every build. >> @@ -406,7 +406,7 @@ >> to which the module is unpacked. >> You are forcing the module into a special >> directory relative to >> - > linkend='var-CVSDIR'>CVSDIR. >> + > linkend='var-bb-CVSDIR'>CVSDIR. >> "rsh" >> Used in conjunction with the "method" >> parameter. @@ -448,7 +448,7 @@ >> FETCHCMD_svn, which defaults >> to "svn". >> The fetcher's temporary working directory is set by >> - > linkend='var-SVNDIR'>SVNDIR, >> + > linkend='var-bb-SVNDIR'>SVNDIR, which is >> usually DL_DIR/svn. >> >> @@ -509,7 +509,7 @@ >> source control system. >> The fetcher works by creating a bare clone of the >> remote into >> - > linkend='var-GITDIR'>GITDIR, >> + > linkend='var-bb-GITDIR'>GITDIR, which is >> usually DL_DIR/git2. This bare clone is then >> cloned into the work directory during the unpack stage when a >> specific tree is checked out. @@ -588,6 +588,14 @@ >> The name of the path in which to place the >> checkout. By default, the path is git/. >> >> + "usehead": >> + Enables local git:// >> URLs to use the >> + current branch HEAD as the revision for use >> with >> + AUTOREV. >> + The "usehead" parameter implies no branch >> and only works >> + when the transfer protocol is >> + file://. >> + >> >> Here are some example URLs: >> >> @@ -604,7 +612,7 @@ >> This fetcher submodule inherits from the >> Git fetcher and >> extends that fetcher's behavior by fetching a repository's submodules. >> - > linkend='var-SRC_URI'>SRC_URI >> + > linkend='var-bb-SRC_URI'>SRC_URI is >> passed to the Git fetcher as described in the "> linkend='git-fetcher'>Git Fetcher >> (git://)" section. @@ -639,9 +647,9 @@ >> >> >> To use this fetcher, make sure your recipe has proper >> - > linkend='var-SRC_URI'>SRC_URI, >> - > linkend='var-SRCREV'>SRCREV, and >> - > linkend='var-PV'>PV settings. >> + > linkend='var-bb-SRC_URI'>SRC_URI, >> + > linkend='var-bb-SRCREV'>SRCREV, and >> + > linkend='var-bb-PV'>PV settings. Here is >> an example: >> SRC_URI = >> "ccrc://cc.example.org/ccrc;vob=/example_vob;module=/example_module" >> @@ -726,15 +734,15 @@ FETCHCMD_p4, which defaults >> to "p4". >> The fetcher's temporary working directory is set by >> - > linkend='var-P4DIR'>P4DIR, >> + > linkend='var-bb-P4DIR'>P4DIR, which >> defaults to "DL_DIR/p4". >> >> >> To use this fetcher, make sure your recipe has proper >> - > linkend='var-SRC_URI'>SRC_URI, >> - > linkend='var-SRCREV'>SRCREV, and >> - > linkend='var-PV'>PV values. >> + > linkend='var-bb-SRC_URI'>SRC_URI, >> + > linkend='var-bb-SRCREV'>SRCREV, and >> + > linkend='var-bb-PV'>PV values. The p4 >> executable is able to use the config file defined by your system's >> P4CONFIG environment variable in order to define >> the Perforce server URL and port, username, and @@ -785,9 +793,9 @@ >> google-repo source control >> system. The fetcher works by initiating and syncing sources of the >> repository into >> - > linkend='var-REPODIR'>REPODIR, >> + > linkend='var-bb-REPODIR'>REPODIR, which >> is usually >> - > linkend='var-DL_DIR'>DL_DIR/repo. >> + > linkend='var-bb-DL_DIR'>DL_DIR/repo. >> >> >> @@ -824,19 +832,22 @@ >> Bazaar (bzr://) >> >> >> - Trees using Git Annex >> (gitannex://) >> + Mercurial (hg://) >> >> >> - Secure FTP (sftp://) >> + npm (npm://) >> >> >> - Secure Shell (ssh://) >> + OSC (osc://) >> >> >> - OSC (osc://) >> + Secure FTP (sftp://) >> >> >> - Mercurial (hg://) >> + Secure Shell (ssh://) >> + >> + >> + Trees using Git Annex >> (gitannex://) >> >> No documentation currently exists for these lesser >> used diff --git >> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.xml >> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.xml index >> 9076f0f..39066e4 100644 --- >> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.xml +++ >> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.xml @@ >> -194,7 +194,7 @@ When you run BitBake, it begins looking for >> metadata files. The >> - > linkend='var-BBPATH'>BBPATH >> + > linkend='var-bb-BBPATH'>BBPATH variable >> is what tells BitBake where to look for those files. >> BBPATH is not set and you need to set it. >> Without BBPATH, Bitbake cannot @@ -273,14 >> +273,14 @@ some editor to create the bitbake.conf >> so that it contains the following: >> >> - PN = >> "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0] >> or 'defaultpkgname'}" >> + PN = >> "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0] >> or 'defaultpkgname'}" > class='monospaced'> >> - TMPDIR = "${TOPDIR}/tmp" >> - CACHE = "${TMPDIR}/cache" >> - STAMP = >> "${TMPDIR}/${PN}/stamps" >> - T = "${TMPDIR}/${PN}/work" >> - B = "${TMPDIR}/${PN}" >> + TMPDIR = "${TOPDIR}/tmp" >> + CACHE = "${TMPDIR}/cache" >> + STAMP = >> "${TMPDIR}/${PN}/stamps" >> + T = "${TMPDIR}/${PN}/work" >> + B = "${TMPDIR}/${PN}" >> >> >> Without a value for PN, the >> @@ -402,12 +402,12 @@ >> Move to the conf directory and >> create a layer.conf file that has the following: >> >> - BBPATH .= ":${LAYERDIR}" >> + BBPATH .= ":${LAYERDIR}" >> >> - BBFILES += "${LAYERDIR}/*.bb" >> + BBFILES += >> "${LAYERDIR}/*.bb" >> - > linkend='var-BBFILE_COLLECTIONS'>BBFILE_COLLECTIONS += >> "mylayer" >> - > linkend='var-BBFILE_PATTERN'>BBFILE_PATTERN_mylayer := >> "^${LAYERDIR_RE}/" >> + > linkend='var-bb-BBFILE_COLLECTIONS'>BBFILE_COLLECTIONS += >> "mylayer" >> + > linkend='var-bb-BBFILE_PATTERN'>BBFILE_PATTERN_mylayer := >> "^${LAYERDIR_RE}/" For information on these >> variables, click the links to go to the definitions in the >> glossary. @@ -416,9 +416,9 @@ >> a recipe file named >> printhello.bb that has the following: >> >> - DESCRIPTION = "Prints >> Hello World" >> - PN = 'printhello' >> - PV = '1' >> + DESCRIPTION = "Prints >> Hello World" >> + PN = 'printhello' >> + PV = '1' >> >> python do_build() { >> bb.plain("********************"); >> diff --git >> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml >> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml index >> f7d312a..8f2a960 100644 --- >> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml +++ >> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml @@ >> -781,7 +781,7 @@ target, you must also enable BitBake to perform >> multiple configuration builds. Enabling is accomplished by setting the >> - > linkend='var-BBMULTICONFIG'>BBMULTICONFIG >> + > linkend='var-bb-BBMULTICONFIG'>BBMULTICONFIG >> variable in the local.conf configuration file. >> As an example, suppose you had configuration >> files @@ -791,7 +791,7 @@ >> The following statement in the >> local.conf file both enables >> BitBake to perform multiple configuration builds >> and >> - specifies the two multiconfigs: >> + specifies the two extra multiconfigs: >> >> BBMULTICONFIG = "target1 target2" >> >> @@ -803,13 +803,13 @@ >> builds, use the following command form to start >> the builds: >> >> - $ bitbake >> [multiconfig:multiconfigname:]target >> [[[multiconfig:multiconfigname:]target] ... ] >> + $ bitbake >> [mc:multiconfigname:]target >> [[[mc:multiconfigname:]target] ... ] >> >> - Here is an example for two multiconfigs: >> + Here is an example for two extra multiconfigs: >> target1 and >> target2: >> >> - $ bitbake multiconfig:target1:target >> multiconfig:target2:target >> + $ bitbake mc::target >> mc:target1:target >> mc:target2:target >> >> @@ -837,13 +837,13 @@ >> build, you must declare the dependencies in the >> recipe using the following statement form: >> >> - task_or_package[mcdepends] = >> "multiconfig:from_multiconfig:to_multiconfig:recipe_name:task_on_which_to_depend" >> + task_or_package[mcdepends] = >> "mc:from_multiconfig:to_multiconfig:recipe_name:task_on_which_to_depend" >> To better show how to use this statement, consider an >> example with two multiconfigs: >> target1 and target2: >> >> - image_task[mcdepends] = >> "multiconfig:target1:target2:image2:rootfs_task" >> + image_task[mcdepends] = >> "mc:target1:target2:image2:rootfs_task" >> In this example, the >> from_multiconfig is >> "target1" and @@ -859,7 +859,7 @@ >> Once you set up this dependency, you can build the >> "target1" multiconfig using a BitBake command as >> follows: >> - $ bitbake multiconfig:target1:image1 >> + $ bitbake mc:target1:image1 >> >> This command executes all the tasks needed to >> create image1 for the "target1" >> @@ -875,7 +875,7 @@ >> Consider this change to the statement in the >> image1 recipe: >> >> - image_task[mcdepends] = >> "multiconfig:target1:target2:image2:image_task" >> + image_task[mcdepends] = >> "mc:target1:target2:image2:image_task" >> In this case, BitBake must create >> image2 for the >> "target2" diff --git >> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml >> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml >> index 2490f6e..421364c 100644 --- >> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml >> +++ >> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml @@ >> -61,6 +61,78 @@ >> +
>> + Modifying Existing Variables >> + >> + >> + Sometimes you need to modify existing variables. >> + Following are some cases where you might find you >> want to >> + modify an existing variable: >> + >> + >> + Customize a recipe that uses the variable. >> + >> + >> + Change a variable's default value used in a >> + *.bbclass file. >> + >> + >> + Change the variable in a >> *.bbappend >> + file to override the variable in the >> original recipe. >> + >> + >> + Change the variable in a configuration file >> so that the >> + value overrides an existing configuration. >> + >> + >> + >> + >> + >> + Changing a variable value can sometimes depend on >> how the >> + value was originally assigned and also on the desired >> + intent of the change. >> + In particular, when you append a value to a variable >> that >> + has a default value, the resulting value might not >> be what >> + you expect. >> + In this case, the value you provide might replace >> the value >> + rather than append to the default value. >> + >> + >> + >> + If after you have changed a variable's value and >> something >> + unexplained occurs, you can use BitBake to check the >> actual >> + value of the suspect variable. >> + You can make these checks for both configuration and >> recipe >> + level changes: >> + >> + >> + For configuration changes, use the following: >> + >> + $ bitbake -e >> + >> + This command displays variable values after >> the >> + configuration files (i.e. >> local.conf, >> + bblayers.conf, >> + bitbake.conf and so >> forth) have >> + been parsed. >> + >> + Variables that are exported to the >> environment are >> + preceded by the string "export" in the >> command's >> + output. >> + >> + >> + >> + For recipe changes, use the following: >> + >> + $ bitbake recipe -e | grep VARIABLE=" >> + >> + This command checks to see if the variable >> actually >> + makes it into a specific recipe. >> + >> + >> + >> +
>> + >>
>> Line Joining >> >> @@ -297,9 +369,8 @@ >> >> >> These operators differ from the ":=", ".=", "=.", >> "+=", and "=+" >> - operators in that their effects are deferred >> - until after parsing completes rather than being >> immediately >> - applied. >> + operators in that their effects are applied at >> variable >> + expansion time rather than being immediately applied. >> Here are some examples: >> >> B = "bval" >> @@ -348,18 +419,22 @@ >> FOO = "123 456 789 123456 123 456 123 456" >> FOO_remove = "123" >> FOO_remove = "456" >> - FOO2 = "abc def ghi abcdef abc def abc def" >> - FOO2_remove = "abc def" >> + FOO2 = " abc def ghi abcdef abc def abc def def" >> + FOO2_remove = " \ >> + def \ >> + abc \ >> + ghi \ >> + " >> >> The variable FOO becomes >> - "  789 123456    " >> + "  789 123456    " >> and FOO2 becomes >> - "  ghi abcdef    ". >> + >> "     jkl  abcdef      ". >> >> >> Like "_append" and "_prepend", "_remove" >> - is deferred until after parsing completes. >> + is applied at variable expansion time. >> >>
>> >> @@ -503,7 +578,7 @@ >> >> >>
>> - Unseting variables >> + Unsetting variables >> >> >> It is possible to completely remove a variable or a >> variable flag @@ -595,7 +670,7 @@ >> >> >> BitBake uses >> - > linkend='var-OVERRIDES'>OVERRIDES >> + > linkend='var-bb-OVERRIDES'>OVERRIDES to >> control what variables are overridden after BitBake parses recipes >> and configuration files. This section describes how you can use >> @@ -705,7 +780,7 @@ >> >> Internally, this is implemented by >> prepending the task (e.g. "task-compile:") to the value of >> - > linkend='var-OVERRIDES'>OVERRIDES >> + > linkend='var-bb-OVERRIDES'>OVERRIDES for >> the local datastore of the do_compile >> task. >> @@ -724,17 +799,15 @@ >> Key Expansion >> >> >> - Key expansion happens when the BitBake datastore is >> finalized >> - just before BitBake expands overrides. >> + Key expansion happens when the BitBake datastore is >> finalized. To better understand this, consider the following example: >> >> A${B} = "X" >> B = "2" >> A2 = "Y" >> >> - In this case, after all the parsing is complete, and >> - before any overrides are handled, BitBake expands >> - ${B} into "2". >> + In this case, after all the parsing is complete, >> + BitBake expands ${B} into "2". >> This expansion causes A2, which >> was set to "Y" before the expansion, to become "X". >> >> @@ -868,7 +941,7 @@ >> >> >> BitBake uses the >> - > linkend='var-BBPATH'>BBPATH >> + > linkend='var-bb-BBPATH'>BBPATH variable >> to locate needed include and class files. Additionally, BitBake >> searches the current directory for include and >> require @@ -1086,7 +1159,7 @@ >> >> When creating a configuration file >> (.conf), you can use the >> - > linkend='var-INHERIT'>INHERIT >> + > linkend='var-bb-INHERIT'>INHERIT >> configuration directive to inherit a class. BitBake only supports >> this directive when used within a configuration file. >> @@ -1370,7 +1443,7 @@ >> >> >> BitBake-style Python functions generate a >> separate >> - ${> linkend='var-T'>T}/run.function-name.pid >> + ${> linkend='var-bb-T'>T}/run.function-name.pid >> script that is executed to run the function, and also generate a log >> file in >> ${T}/log.function-name.pid >> @@ -1773,7 +1846,7 @@ things exported or listed in its whitelist to >> ensure that the build environment is reproducible and consistent. >> You can prevent this "cleaning" by setting the >> - > linkend='var-BB_PRESERVE_ENV'>BB_PRESERVE_ENV >> + > linkend='var-bb-BB_PRESERVE_ENV'>BB_PRESERVE_ENV >> variable. >> Consequently, if you do want something to get passed >> into the @@ -1783,9 +1856,9 @@ >> Tell BitBake to load what you want from the >> environment into the datastore. >> You can do so through the >> - > linkend='var-BB_ENV_WHITELIST'>BB_ENV_WHITELIST >> + > linkend='var-bb-BB_ENV_WHITELIST'>BB_ENV_WHITELIST >> and >> - > linkend='var-BB_ENV_EXTRAWHITE'>BB_ENV_EXTRAWHITE >> + > linkend='var-bb-BB_ENV_EXTRAWHITE'>BB_ENV_EXTRAWHITE >> variables. For example, assume you want to prevent the build system >> from accessing your $HOME/.ccache >> @@ -1824,7 +1897,7 @@ >> from the original execution environment. >> Bitbake saves a copy of the original environment into >> a special variable named >> - > linkend='var-BB_ORIGENV'>BB_ORIGENV. >> + > linkend='var-bb-BB_ORIGENV'>BB_ORIGENV. >> >> >> @@ -1883,7 +1956,7 @@ >> [depends]: >> Controls inter-task dependencies. >> See the >> - > linkend='var-DEPENDS'>DEPENDS >> + > linkend='var-bb-DEPENDS'>DEPENDS variable >> and the "Inter-Task >> Dependencies" section for more information. >> @@ -1891,7 +1964,7 @@ >> [deptask]: >> Controls task build-time dependencies. >> See the >> - > linkend='var-DEPENDS'>DEPENDS >> + > linkend='var-bb-DEPENDS'>DEPENDS variable >> and the "Build Dependencies" >> section for more information. >> @@ -1937,7 +2010,7 @@ >> of cores but certain tasks need to be >> rate-limited due to various kinds of resource constraints (e.g. to >> avoid network throttling). number_threads works >> similarly to the >> - > linkend='var-BB_NUMBER_THREADS'>BB_NUMBER_THREADS >> + > linkend='var-bb-BB_NUMBER_THREADS'>BB_NUMBER_THREADS >> variable but is task-specific. >> Set the value globally. >> @@ -1971,9 +2044,9 @@ >> [rdepends]: >> Controls inter-task runtime dependencies. >> See the >> - > linkend='var-RDEPENDS'>RDEPENDS >> + > linkend='var-bb-RDEPENDS'>RDEPENDS >> variable, the >> - > linkend='var-RRECOMMENDS'>RRECOMMENDS >> + > linkend='var-bb-RRECOMMENDS'>RRECOMMENDS >> variable, and the "Inter-Task >> Dependencies" section for more information. >> @@ -1981,9 +2054,9 @@ >> [rdeptask]: >> Controls task runtime dependencies. >> See the >> - > linkend='var-RDEPENDS'>RDEPENDS >> + > linkend='var-bb-RDEPENDS'>RDEPENDS >> variable, the >> - > linkend='var-RRECOMMENDS'>RRECOMMENDS >> + > linkend='var-bb-RRECOMMENDS'>RRECOMMENDS >> variable, and the "Runtime >> Dependencies" section for more information. >> @@ -1996,9 +2069,9 @@ >> [recrdeptask]: >> Controls task recursive runtime dependencies. >> See the >> - > linkend='var-RDEPENDS'>RDEPENDS >> + > linkend='var-bb-RDEPENDS'>RDEPENDS >> variable, the >> - > linkend='var-RRECOMMENDS'>RRECOMMENDS >> + > linkend='var-bb-RRECOMMENDS'>RRECOMMENDS >> variable, and the "Recursive >> Dependencies" section for more information. >> @@ -2127,7 +2200,7 @@ >> Any given datastore only has one such event >> executed against it, however. >> If >> - > linkende='var-BB_INVALIDCONF'>BB_INVALIDCONF >> + > linkende='var-bb-BB_INVALIDCONF'>BB_INVALIDCONF >> is set in the datastore by the event handler, the configuration is >> reparsed and a new event triggered, allowing the metadata to update >> configuration. @@ -2256,17 +2329,17 @@ >> from a single recipe file multiple incarnations of that >> recipe file where all incarnations are buildable. >> These features are enabled through the >> - > linkend='var-BBCLASSEXTEND'>BBCLASSEXTEND >> + > linkend='var-bb-BBCLASSEXTEND'>BBCLASSEXTEND >> and >> - > linkend='var-BBVERSIONS'>BBVERSIONS >> + > linkend='var-bb-BBVERSIONS'>BBVERSIONS >> variables. >> The mechanism for this class extension is extremely >> specific to the implementation. >> Usually, the recipe's >> - > linkend='var-PROVIDES'>PROVIDES, >> - > linkend='var-PN'>PN, and >> - > linkend='var-DEPENDS'>DEPENDS >> + > linkend='var-bb-PROVIDES'>PROVIDES, >> + > linkend='var-bb-PN'>PN, and >> + > linkend='var-bb-DEPENDS'>DEPENDS >> variables would need to be modified by the extension class. For >> specific examples, see the OE-Core native, >> nativesdk, @@ -2287,7 +2360,7 @@ >> project from a single recipe file. >> You can also specify conditional metadata >> (using the >> - > linkend='var-OVERRIDES'>OVERRIDES >> + > linkend='var-bb-OVERRIDES'>OVERRIDES >> mechanism) for a single version, or an optionally named range of >> versions. Here is an example: >> @@ -2306,7 +2379,7 @@ >> into overrides, but it is also made available >> for the metadata to use in the variable that defines the base recipe >> versions for use in file:// search paths >> - (> linkend='var-FILESPATH'>FILESPATH). >> + (> linkend='var-bb-FILESPATH'>FILESPATH). >> >> >> @@ -2408,7 +2481,7 @@ >> >> >> BitBake uses the >> - > linkend='var-DEPENDS'>DEPENDS >> + > linkend='var-bb-DEPENDS'>DEPENDS variable >> to manage build time dependencies. The [deptask] >> varflag for tasks signifies the task of each >> @@ -2429,9 +2502,9 @@ >> >> >> BitBake uses the >> - > linkend='var-PACKAGES'>PACKAGES, >> - > linkend='var-RDEPENDS'>RDEPENDS, and >> - > linkend='var-RRECOMMENDS'>RRECOMMENDS >> + > linkend='var-bb-PACKAGES'>PACKAGES, >> + > linkend='var-bb-RDEPENDS'>RDEPENDS, and >> + > linkend='var-bb-RRECOMMENDS'>RRECOMMENDS >> variables to manage runtime dependencies. >> >> @@ -2686,7 +2759,7 @@ >> >> >> These checksums are stored in >> - > linkend='var-STAMP'>STAMP. >> + > linkend='var-bb-STAMP'>STAMP. You can >> examine the checksums using the following BitBake command: >> $ bitbake-dumpsigs >> @@ -2708,44 +2781,44 @@ >> The following list describes related variables: >> >> >> - > linkend='var-BB_HASHCHECK_FUNCTION'>BB_HASHCHECK_FUNCTION: >> + > linkend='var-bb-BB_HASHCHECK_FUNCTION'>BB_HASHCHECK_FUNCTION: >> Specifies the name of the function to call during the "setscene" part >> of the task's execution in order to validate the list of task hashes. >> >> >> - > linkend='var-BB_SETSCENE_DEPVALID'>BB_SETSCENE_DEPVALID: >> + > linkend='var-bb-BB_SETSCENE_DEPVALID'>BB_SETSCENE_DEPVALID: >> Specifies a function BitBake calls that determines whether BitBake >> requires a setscene dependency to be met. >> >> >> - > linkend='var-BB_SETSCENE_VERIFY_FUNCTION2'>BB_SETSCENE_VERIFY_FUNCTION2: >> + > linkend='var-bb-BB_SETSCENE_VERIFY_FUNCTION2'>BB_SETSCENE_VERIFY_FUNCTION2: >> Specifies a function to call that verifies the list of planned task >> execution before the main task execution happens. >> >> >> - > linkend='var-BB_STAMP_POLICY'>BB_STAMP_POLICY: >> + > linkend='var-bb-BB_STAMP_POLICY'>BB_STAMP_POLICY: >> Defines the mode for comparing timestamps of stamp files. >> >> - > linkend='var-BB_STAMP_WHITELIST'>BB_STAMP_WHITELIST: >> + > linkend='var-bb-BB_STAMP_WHITELIST'>BB_STAMP_WHITELIST: >> Lists stamp files that are looked at when the stamp policy is >> "whitelist". >> >> - > linkend='var-BB_TASKHASH'>BB_TASKHASH: >> + > linkend='var-bb-BB_TASKHASH'>BB_TASKHASH: >> Within an executing task, this variable holds the hash of the task as >> returned by the currently enabled signature generator. >> >> >> - > linkend='var-STAMP'>STAMP: >> + > linkend='var-bb-STAMP'>STAMP: The base >> path to create stamp files. >> >> - > linkend='var-STAMPCLEAN'>STAMPCLEAN: >> + > linkend='var-bb-STAMPCLEAN'>STAMPCLEAN: >> Again, the base path to create stamp files but can use wildcards for >> matching a range of files for clean operations. >> diff --git >> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml >> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml >> index a84b2bc..aca6741 100644 --- >> a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml >> +++ >> b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml >> @@ -3,7 +3,7 @@ [ %poky; ] > >> - >> + >> >> Variables Glossary >> >> @@ -34,29 +34,29 @@ >> >> >> >> - >> + >> >> >> - A >> - B >> - C >> - D >> - E >> - F >> - G >> - H >> - >> + A >> + B >> + C >> + D >> + E >> + F >> + G >> + H >> + I >> >> >> - L >> - M >> + L >> + M >> >> - O >> - P >> + O >> + P >> >> - R >> - S >> - T >> + R >> + S >> + T >> >> >> >> @@ -65,13 +65,13 @@ >> >> >> >> - A >> + A >> >> - > id='var-ASSUME_PROVIDED'>ASSUME_PROVIDED >> + > id='var-bb-ASSUME_PROVIDED'>ASSUME_PROVIDED >> >> Lists recipe names >> - (> linkend='var-PN'>PN >> + (> linkend='var-bb-PN'>PN values) BitBake >> does not attempt to build. Instead, BitBake assumes these recipes >> have already been built. >> @@ -91,9 +91,9 @@ >> >> >> >> - B >> + B >> >> - B >> + B >> >> >> The directory in which BitBake executes functions >> @@ -102,7 +102,7 @@ >> >> >> >> - > id='var-BB_ALLOWED_NETWORKS'>BB_ALLOWED_NETWORKS >> + > id='var-bb-BB_ALLOWED_NETWORKS'>BB_ALLOWED_NETWORKS >> >> Specifies a space-delimited list of hosts that >> the fetcher @@ -111,7 +111,7 @@ >> >> >> This host list is only used if >> - > linkend='var-BB_NO_NETWORK'>BB_NO_NETWORK >> + > linkend='var-bb-BB_NO_NETWORK'>BB_NO_NETWORK >> is either not set or set to "0". >> >> @@ -151,13 +151,13 @@ >> >> Using BB_ALLOWED_NETWORKS in >> conjunction with >> - > linkend='var-PREMIRRORS'>PREMIRRORS >> + > linkend='var-bb-PREMIRRORS'>PREMIRRORS is >> very useful. Adding the host you want to use to >> PREMIRRORS results in the >> source code being fetched from an allowed location and avoids raising >> an error when a host that is not allowed is in a >> - > linkend='var-SRC_URI'>SRC_URI >> + > linkend='var-bb-SRC_URI'>SRC_URI >> statement. This is because the fetcher does not attempt to use the >> host listed in SRC_URI >> after a @@ -167,7 +167,7 @@ >> >> >> >> - > id='var-BB_CONSOLELOG'>BB_CONSOLELOG >> + > id='var-bb-BB_CONSOLELOG'>BB_CONSOLELOG >> >> Specifies the path to a log file into which >> BitBake's user @@ -176,7 +176,7 @@ >> >> >> >> - > id='var-BB_CURRENTTASK'>BB_CURRENTTASK >> + > id='var-bb-BB_CURRENTTASK'>BB_CURRENTTASK >> >> Contains the name of the currently running task. >> @@ -186,7 +186,7 @@ >> >> >> >> - > id='var-BB_DANGLINGAPPENDS_WARNONLY'>BB_DANGLINGAPPENDS_WARNONLY >> + > id='var-bb-BB_DANGLINGAPPENDS_WARNONLY'>BB_DANGLINGAPPENDS_WARNONLY >> >> Defines how BitBake handles situations where an >> append @@ -208,7 +208,7 @@ >> >> >> >> - > id='var-BB_DEFAULT_TASK'>BB_DEFAULT_TASK >> + > id='var-bb-BB_DEFAULT_TASK'>BB_DEFAULT_TASK >> >> The default task to use when none is specified >> (e.g. @@ -219,7 +219,7 @@ >> >> >> >> - > id='var-BB_DISKMON_DIRS'>BB_DISKMON_DIRS >> + > id='var-bb-BB_DISKMON_DIRS'>BB_DISKMON_DIRS >> >> Monitors disk space and available inodes during >> the build @@ -245,7 +245,7 @@ >> build when a threshold is broken. >> Subsequent warnings are issued as >> defined by the >> - > linkend='var-BB_DISKMON_WARNINTERVAL'>BB_DISKMON_WARNINTERVAL >> variable, >> + > linkend='var-bb-BB_DISKMON_WARNINTERVAL'>BB_DISKMON_WARNINTERVAL >> variable, which must be defined. >> <dir> is: >> @@ -275,7 +275,7 @@ >> BB_DISKMON_DIRS = "ABORT,${TMPDIR},,100K" >> >> The first example works only if you also set >> - the > linkend='var-BB_DISKMON_WARNINTERVAL'>BB_DISKMON_WARNINTERVAL >> variable. >> + the > linkend='var-bb-BB_DISKMON_WARNINTERVAL'>BB_DISKMON_WARNINTERVAL >> variable. This example causes the build system to immediately abort >> when either the disk space in ${TMPDIR} drops >> below 1 Gbyte or the available free inodes drops below @@ -309,7 >> +309,7 @@ >> >> >> - > id='var-BB_DISKMON_WARNINTERVAL'>BB_DISKMON_WARNINTERVAL >> + > id='var-bb-BB_DISKMON_WARNINTERVAL'>BB_DISKMON_WARNINTERVAL >> >> Defines the disk space and free inode warning >> intervals. @@ -319,7 +319,7 @@ >> If you are going to use the >> BB_DISKMON_WARNINTERVAL >> variable, you must also use the >> - > linkend='var-BB_DISKMON_DIRS'>BB_DISKMON_DIRS >> variable >> + > linkend='var-bb-BB_DISKMON_DIRS'>BB_DISKMON_DIRS >> variable and define its action as "WARN". During the build, >> subsequent warnings are issued each time disk space or number of free >> inodes further reduces by @@ -374,7 +374,7 @@ >> >> >> >> - > id='var-BB_ENV_WHITELIST'>BB_ENV_WHITELIST >> + > id='var-bb-BB_ENV_WHITELIST'>BB_ENV_WHITELIST >> >> Specifies the internal whitelist of variables to >> allow @@ -382,11 +382,11 @@ >> datastore. >> If the value of this variable is not specified >> (which is the default), the following list is >> used: >> - > linkend='var-BBPATH'>BBPATH, >> - > linkend='var-BB_PRESERVE_ENV'>BB_PRESERVE_ENV, >> - > linkend='var-BB_ENV_WHITELIST'>BB_ENV_WHITELIST, >> + > linkend='var-bb-BBPATH'>BBPATH, >> + > linkend='var-bb-BB_PRESERVE_ENV'>BB_PRESERVE_ENV, >> + > linkend='var-bb-BB_ENV_WHITELIST'>BB_ENV_WHITELIST, >> and >> - > linkend='var-BB_ENV_EXTRAWHITE'>BB_ENV_EXTRAWHITE. >> + > linkend='var-bb-BB_ENV_EXTRAWHITE'>BB_ENV_EXTRAWHITE. >> You must set this variable in the external environment >> in order for it to work. >> @@ -395,7 +395,7 @@ >> >> >> >> - > id='var-BB_ENV_EXTRAWHITE'>BB_ENV_EXTRAWHITE >> + > id='var-bb-BB_ENV_EXTRAWHITE'>BB_ENV_EXTRAWHITE >> >> Specifies an additional set of variables to >> allow through @@ -403,7 +403,7 @@ >> datastore. >> This list of variables are on top of the >> internal list set in >> - > linkend='var-BB_ENV_WHITELIST'>BB_ENV_WHITELIST. >> + > linkend='var-bb-BB_ENV_WHITELIST'>BB_ENV_WHITELIST. >> You must set this variable in the external >> environment in order for it to work. >> @@ -412,22 +412,22 @@ >> >> >> >> - > id='var-BB_FETCH_PREMIRRORONLY'>BB_FETCH_PREMIRRORONLY >> + > id='var-bb-BB_FETCH_PREMIRRORONLY'>BB_FETCH_PREMIRRORONLY >> >> When set to "1", causes BitBake's fetcher module >> to only search >> - > linkend='var-PREMIRRORS'>PREMIRRORS >> + > linkend='var-bb-PREMIRRORS'>PREMIRRORS >> for files. BitBake will not search the main >> - > linkend='var-SRC_URI'>SRC_URI >> + > linkend='var-bb-SRC_URI'>SRC_URI or >> - > linkend='var-MIRRORS'>MIRRORS. >> + > linkend='var-bb-MIRRORS'>MIRRORS. >> >> >> >> - > id='var-BB_FILENAME'>BB_FILENAME >> + > id='var-bb-BB_FILENAME'>BB_FILENAME >> >> Contains the filename of the recipe that owns >> the currently @@ -440,12 +440,12 @@ >> >> >> >> - > id='var-BB_GENERATE_MIRROR_TARBALLS'>BB_GENERATE_MIRROR_TARBALLS >> + > id='var-bb-BB_GENERATE_MIRROR_TARBALLS'>BB_GENERATE_MIRROR_TARBALLS >> >> Causes tarballs of the Git repositories, >> including the Git metadata, to be placed in the >> - > linkend='var-DL_DIR'>DL_DIR >> + > linkend='var-bb-DL_DIR'>DL_DIR directory. >> Anyone wishing to create a source mirror would >> want to enable this variable. >> @@ -461,7 +461,7 @@ >> >> >> >> - > id='var-BB_HASHCONFIG_WHITELIST'>BB_HASHCONFIG_WHITELIST >> + > id='var-bb-BB_HASHCONFIG_WHITELIST'>BB_HASHCONFIG_WHITELIST >> >> Lists variables that are excluded from base >> configuration @@ -485,7 +485,7 @@ >> >> >> >> - > id='var-BB_HASHBASE_WHITELIST'>BB_HASHBASE_WHITELIST >> + > id='var-bb-BB_HASHBASE_WHITELIST'>BB_HASHBASE_WHITELIST >> >> Lists variables that are excluded from checksum >> and @@ -500,7 +500,7 @@ >> >> >> >> - > id='var-BB_HASHCHECK_FUNCTION'>BB_HASHCHECK_FUNCTION >> + > id='var-bb-BB_HASHCHECK_FUNCTION'>BB_HASHCHECK_FUNCTION >> >> Specifies the name of the function to call >> during the @@ -524,7 +524,7 @@ >> >> >> >> - > id='var-BB_INVALIDCONF'>BB_INVALIDCONF >> + > id='var-bb-BB_INVALIDCONF'>BB_INVALIDCONF >> >> Used in combination with the >> @@ -539,11 +539,11 @@ >> >> >> >> - > id='var-BB_LOGFMT'>BB_LOGFMT >> + > id='var-bb-BB_LOGFMT'>BB_LOGFMT >> >> Specifies the name of the log files saved into >> - ${> linkend='var-T'>T}. >> + ${> linkend='var-bb-T'>T}. >> By default, the BB_LOGFMT variable is undefined >> and the log file names get created using the following form: >> @@ -556,7 +556,7 @@ >> >> >> >> - > id='var-BB_NICE_LEVEL'>BB_NICE_LEVEL >> + > id='var-bb-BB_NICE_LEVEL'>BB_NICE_LEVEL >> >> Allows BitBake to run at a specific priority >> @@ -564,13 +564,13 @@ >> System permissions usually mean that BitBake can >> reduce its priority but not raise it again. >> See >> - > linkend='var-BB_TASK_NICE_LEVEL'>BB_TASK_NICE_LEVEL >> + > linkend='var-bb-BB_TASK_NICE_LEVEL'>BB_TASK_NICE_LEVEL >> for additional information. >> >> >> >> - > id='var-BB_NO_NETWORK'>BB_NO_NETWORK >> + > id='var-bb-BB_NO_NETWORK'>BB_NO_NETWORK >> >> Disables network access in the BitBake fetcher >> modules. @@ -587,7 +587,7 @@ >> >> >> >> - > id='var-BB_NUMBER_THREADS'>BB_NUMBER_THREADS >> + > id='var-bb-BB_NUMBER_THREADS'>BB_NUMBER_THREADS >> >> The maximum number of tasks BitBake should run >> in parallel @@ -599,7 +599,7 @@ >> >> >> >> - > id='var-BB_NUMBER_PARSE_THREADS'>BB_NUMBER_PARSE_THREADS >> + > id='var-bb-BB_NUMBER_PARSE_THREADS'>BB_NUMBER_PARSE_THREADS >> >> Sets the number of threads BitBake uses when >> parsing. @@ -609,7 +609,7 @@ >> >> >> >> - > id='var-BB_ORIGENV'>BB_ORIGENV >> + > id='var-bb-BB_ORIGENV'>BB_ORIGENV >> >> Contains a copy of the original external >> environment in @@ -625,7 +625,7 @@ >> >> >> >> - > id='var-BB_PRESERVE_ENV'>BB_PRESERVE_ENV >> + > id='var-bb-BB_PRESERVE_ENV'>BB_PRESERVE_ENV >> >> Disables whitelisting and instead allows all >> variables @@ -639,12 +639,12 @@ >> >> >> >> - > id='var-BB_RUNFMT'>BB_RUNFMT >> + > id='var-bb-BB_RUNFMT'>BB_RUNFMT >> >> Specifies the name of the executable script files >> (i.e. run files) saved into >> - ${> linkend='var-T'>T}. >> + ${> linkend='var-bb-T'>T}. >> By default, the BB_RUNFMT variable is undefined >> and the run file names get created using the following form: >> @@ -657,7 +657,7 @@ >> >> >> >> - > id='var-BB_RUNTASK'>BB_RUNTASK >> + > id='var-bb-BB_RUNTASK'>BB_RUNTASK >> >> Contains the name of the currently executing >> task. @@ -669,7 +669,7 @@ >> >> >> >> - > id='var-BB_SCHEDULER'>BB_SCHEDULER >> + > id='var-bb-BB_SCHEDULER'>BB_SCHEDULER >> >> Selects the name of the scheduler to use for the >> @@ -695,7 +695,7 @@ >> >> >> >> - > id='var-BB_SCHEDULERS'>BB_SCHEDULERS >> + > id='var-bb-BB_SCHEDULERS'>BB_SCHEDULERS >> >> Defines custom schedulers to import. >> @@ -705,13 +705,13 @@ >> >> >> For information how to select a scheduler, see >> the >> - > linkend='var-BB_SCHEDULER'>BB_SCHEDULER >> + > linkend='var-bb-BB_SCHEDULER'>BB_SCHEDULER >> variable. >> >> >> >> - > id='var-BB_SETSCENE_DEPVALID'>BB_SETSCENE_DEPVALID >> + > id='var-bb-BB_SETSCENE_DEPVALID'>BB_SETSCENE_DEPVALID >> >> Specifies a function BitBake calls that >> determines @@ -731,7 +731,7 @@ >> >> >> >> - > id='var-BB_SETSCENE_VERIFY_FUNCTION2'>BB_SETSCENE_VERIFY_FUNCTION2 >> + > id='var-bb-BB_SETSCENE_VERIFY_FUNCTION2'>BB_SETSCENE_VERIFY_FUNCTION2 >> >> Specifies a function to call that verifies the >> list of @@ -752,7 +752,7 @@ >> >> >> >> - > id='var-BB_SIGNATURE_EXCLUDE_FLAGS'>BB_SIGNATURE_EXCLUDE_FLAGS >> + > id='var-bb-BB_SIGNATURE_EXCLUDE_FLAGS'>BB_SIGNATURE_EXCLUDE_FLAGS >> >> Lists variable flags (varflags) >> @@ -771,7 +771,7 @@ >> >> >> >> - > id='var-BB_SIGNATURE_HANDLER'>BB_SIGNATURE_HANDLER >> + > id='var-bb-BB_SIGNATURE_HANDLER'>BB_SIGNATURE_HANDLER >> >> Defines the name of the signature handler >> BitBake uses. @@ -790,7 +790,7 @@ >> >> >> >> - > id='var-BB_SRCREV_POLICY'>BB_SRCREV_POLICY >> + > id='var-bb-BB_SRCREV_POLICY'>BB_SRCREV_POLICY >> >> Defines the behavior of the fetcher when it >> interacts with @@ -817,7 +817,7 @@ >> >> >> >> - > id='var-BB_STAMP_POLICY'>BB_STAMP_POLICY >> + > id='var-bb-BB_STAMP_POLICY'>BB_STAMP_POLICY >> >> Defines the mode used for how timestamps of >> stamp files @@ -836,7 +836,7 @@ >> whitelist >> - Identical to "full" mode except timestamp >> comparisons are made for recipes listed >> in the >> - > linkend='var-BB_STAMP_WHITELIST'>BB_STAMP_WHITELIST >> + > linkend='var-bb-BB_STAMP_WHITELIST'>BB_STAMP_WHITELIST >> variable. >> >> @@ -848,19 +848,19 @@ >> >> >> >> - > id='var-BB_STAMP_WHITELIST'>BB_STAMP_WHITELIST >> + > id='var-bb-BB_STAMP_WHITELIST'>BB_STAMP_WHITELIST >> >> Lists files whose stamp file timestamps are >> compared when the stamp policy mode is set to "whitelist". >> For information on stamp policies, see the >> - > linkend='var-BB_STAMP_POLICY'>BB_STAMP_POLICY >> + > linkend='var-bb-BB_STAMP_POLICY'>BB_STAMP_POLICY >> variable. >> >> >> >> - > id='var-BB_STRICT_CHECKSUM'>BB_STRICT_CHECKSUM >> + > id='var-bb-BB_STRICT_CHECKSUM'>BB_STRICT_CHECKSUM >> >> Sets a more strict checksum mechanism for >> non-local URLs. @@ -871,7 +871,7 @@ >> >> >> >> - > id='var-BB_TASK_IONICE_LEVEL'>BB_TASK_IONICE_LEVEL >> + > id='var-bb-BB_TASK_IONICE_LEVEL'>BB_TASK_IONICE_LEVEL >> >> Allows adjustment of a task's Input/Output >> priority. @@ -882,7 +882,7 @@ >> variable to adjust the I/O priority of these >> tasks. >> This variable works similarly to the >> - > linkend='var-BB_TASK_NICE_LEVEL'>BB_TASK_NICE_LEVEL >> + > linkend='var-bb-BB_TASK_NICE_LEVEL'>BB_TASK_NICE_LEVEL >> variable except with a task's I/O priorities. >> >> @@ -921,7 +921,7 @@ >> >> >> >> - > id='var-BB_TASK_NICE_LEVEL'>BB_TASK_NICE_LEVEL >> + > id='var-bb-BB_TASK_NICE_LEVEL'>BB_TASK_NICE_LEVEL >> >> Allows specific tasks to change their priority >> @@ -940,7 +940,7 @@ >> >> >> >> - > id='var-BB_TASKHASH'>BB_TASKHASH >> + > id='var-bb-BB_TASKHASH'>BB_TASKHASH >> >> Within an executing task, this variable holds >> the hash @@ -950,7 +950,7 @@ >> >> >> >> - > id='var-BB_VERBOSE_LOGS'>BB_VERBOSE_LOGS >> + > id='var-bb-BB_VERBOSE_LOGS'>BB_VERBOSE_LOGS >> >> Controls how verbose BitBake is during builds. >> @@ -960,7 +960,7 @@ >> >> >> >> - > id='var-BB_WORKERCONTEXT'>BB_WORKERCONTEXT >> + > id='var-bb-BB_WORKERCONTEXT'>BB_WORKERCONTEXT >> >> Specifies if the current context is executing a >> task. @@ -973,7 +973,7 @@ >> >> >> >> - > id='var-BBCLASSEXTEND'>BBCLASSEXTEND >> + > id='var-bb-BBCLASSEXTEND'>BBCLASSEXTEND >> >> Allows you to extend a recipe so that it builds >> variants @@ -1009,7 +1009,7 @@ >> _class-native. >> For example, to generate a native version of >> a recipe, a >> - > linkend='var-DEPENDS'>DEPENDS >> + > linkend='var-bb-DEPENDS'>DEPENDS on "foo" >> is rewritten to a DEPENDS on "foo-native". >> >> @@ -1028,7 +1028,7 @@ >> >> >> >> - BBDEBUG >> + > id='var-bb-BBDEBUG'>BBDEBUG >> >> Sets the BitBake debug output level to a >> specific value @@ -1042,7 +1042,7 @@ >> >> >> >> - > id='var-BBFILE_COLLECTIONS'>BBFILE_COLLECTIONS >> + > id='var-bb-BBFILE_COLLECTIONS'>BBFILE_COLLECTIONS >> Lists the names of configured layers. >> These names are used to find the other >> BBFILE_* @@ -1053,10 +1053,10 @@ >> >> >> >> - > id='var-BBFILE_PATTERN'>BBFILE_PATTERN >> + > id='var-bb-BBFILE_PATTERN'>BBFILE_PATTERN >> Variable that expands to match files from >> - > linkend='var-BBFILES'>BBFILES >> + > linkend='var-bb-BBFILES'>BBFILES in a >> particular layer. This variable is used in the >> conf/layer.conf file and must be suffixed with >> the name of the specific layer (e.g. @@ -1064,7 +1064,7 @@ >> >> >> >> - > id='var-BBFILE_PRIORITY'>BBFILE_PRIORITY >> + > id='var-bb-BBFILE_PRIORITY'>BBFILE_PRIORITY >> Assigns the priority for recipe files in each >> layer. This variable is useful in situations where the >> same recipe appears in @@ -1074,7 +1074,7 @@ >> letting you control the precedence for the >> multiple layers. The precedence established through this variable >> stands regardless of a recipe's version >> - (> linkend='var-PV'>PV variable). >> + (> linkend='var-bb-PV'>PV variable). For >> example, a layer that has a recipe with a higher >> PV value but for which the >> BBFILE_PRIORITY is set to have a lower >> precedence still has a lower precedence. @@ -1083,7 +1083,7 @@ >> For example, the value 6 has a higher precedence than the value 5. If >> not specified, the BBFILE_PRIORITY variable is >> set based on layer dependencies (see the >> - > linkend='var-LAYERDEPENDS'>LAYERDEPENDS variable for >> + > linkend='var-bb-LAYERDEPENDS'>LAYERDEPENDS variable >> for more information. The default priority, if unspecified >> for a layer with no dependencies, is the lowest >> defined priority + 1 @@ -1095,7 +1095,7 @@ >> >> >> >> - BBFILES >> + > id='var-bb-BBFILES'>BBFILES >> >> A space-separated list of recipe files BitBake >> uses to @@ -1113,7 +1113,7 @@ >> >> >> >> - > id='var-BBINCLUDED'>BBINCLUDED >> + > id='var-bb-BBINCLUDED'>BBINCLUDED >> >> Contains a space-separated list of all of all >> files that @@ -1123,7 +1123,7 @@ >> >> >> >> - > id='var-BBINCLUDELOGS'>BBINCLUDELOGS >> + > id='var-bb-BBINCLUDELOGS'>BBINCLUDELOGS >> >> If set to a value, enables printing the task log >> when @@ -1132,11 +1132,11 @@ >> >> >> >> - > id='var-BBINCLUDELOGS_LINES'>BBINCLUDELOGS_LINES >> + > id='var-bb-BBINCLUDELOGS_LINES'>BBINCLUDELOGS_LINES >> >> If >> - > linkend='var-BBINCLUDELOGS'>BBINCLUDELOGS >> + > linkend='var-bb-BBINCLUDELOGS'>BBINCLUDELOGS >> is set, specifies the maximum number of lines from the task log file >> to print when reporting a failed task. If you do not set >> BBINCLUDELOGS_LINES, @@ -1145,7 +1145,7 @@ >> >> >> >> - BBLAYERS >> + > id='var-bb-BBLAYERS'>BBLAYERS >> Lists the layers to enable during the build. >> This variable is defined in the >> bblayers.conf configuration @@ -1166,7 +1166,7 @@ >> >> >> >> - > id='var-BBLAYERS_FETCH_DIR'>BBLAYERS_FETCH_DIR >> + > id='var-bb-BBLAYERS_FETCH_DIR'>BBLAYERS_FETCH_DIR >> >> Sets the base location where layers are stored. >> @@ -1178,7 +1178,7 @@ >> >> >> >> - BBMASK >> + BBMASK >> >> >> Prevents BitBake from processing recipes and >> recipe @@ -1236,7 +1236,7 @@ >> >> >> >> - > id='var-BBMULTICONFIG'>BBMULTICONFIG >> + > id='var-bb-BBMULTICONFIG'>BBMULTICONFIG >> BBMULTICONFIG[doc] = "Enables BitBake to perform >> multiple configuration builds and lists each separate configuration >> (multiconfig)." @@ -1275,7 +1275,7 @@ >> >> >> >> - BBPATH >> + BBPATH >> >> >> Used by BitBake to locate class >> @@ -1302,7 +1302,7 @@ >> >> >> >> - BBSERVER >> + > id='var-bb-BBSERVER'>BBSERVER >> >> Points to the server that runs memory-resident >> BitBake. @@ -1312,7 +1312,7 @@ >> >> >> >> - > id='var-BBTARGETS'>BBTARGETS >> + > id='var-bb-BBTARGETS'>BBTARGETS >> >> Allows you to use a configuration file to add to >> the list @@ -1321,14 +1321,14 @@ >> >> >> >> - > id='var-BBVERSIONS'>BBVERSIONS >> + > id='var-bb-BBVERSIONS'>BBVERSIONS >> >> Allows a single recipe to build multiple >> versions of a project from a single recipe file. >> You also able to specify conditional metadata >> using the >> - > linkend='var-OVERRIDES'>OVERRIDES >> + > linkend='var-bb-OVERRIDES'>OVERRIDES >> mechanism for a single version or for an optionally named range of >> versions. >> @@ -1342,7 +1342,7 @@ >> >> >> >> - > id='var-BITBAKE_UI'>BITBAKE_UI >> + > id='var-bb-BITBAKE_UI'>BITBAKE_UI >> >> Used to specify the UI module to use when >> running BitBake. @@ -1356,7 +1356,7 @@ >> >> >> >> - > id='var-BUILDNAME'>BUILDNAME >> + > id='var-bb-BUILDNAME'>BUILDNAME >> >> A name assigned to the build. >> @@ -1366,7 +1366,7 @@ >> >> >> >> - BZRDIR >> + BZRDIR >> >> >> The directory in which files checked out of a >> Bazaar @@ -1377,9 +1377,9 @@ >> >> >> >> - C >> + C >> >> - CACHE >> + CACHE >> >> >> Specifies the directory BitBake uses to store a >> cache @@ -1389,7 +1389,7 @@ >> >> >> >> - CVSDIR >> + CVSDIR >> >> >> The directory in which files checked out under >> the @@ -1400,9 +1400,9 @@ >> >> >> >> - D >> + D >> >> - > id='var-DEFAULT_PREFERENCE'>DEFAULT_PREFERENCE >> + > id='var-bb-DEFAULT_PREFERENCE'>DEFAULT_PREFERENCE >> >> Specifies a weak bias for recipe selection >> priority. @@ -1413,20 +1413,20 @@ >> piece of software. >> Using the variable in this way causes the stable >> version of the recipe to build by default in the absence of >> - > linkend='var-PREFERRED_VERSION'>PREFERRED_VERSION >> + > linkend='var-bb-PREFERRED_VERSION'>PREFERRED_VERSION >> being used to build the development version. >> >> The bias provided by >> DEFAULT_PREFERENCE is weak and is overridden by >> - > linkend='var-BBFILE_PRIORITY'>BBFILE_PRIORITY >> + > linkend='var-bb-BBFILE_PRIORITY'>BBFILE_PRIORITY if >> that variable is different between two layers that contain different >> versions of the same recipe. >> >> >> >> - DEPENDS >> + > id='var-bb-DEPENDS'>DEPENDS >> >> Lists a recipe's build-time dependencies >> @@ -1451,13 +1451,13 @@ >> >> >> For information on runtime dependencies, see the >> - > linkend='var-RDEPENDS'>RDEPENDS >> + > linkend='var-bb-RDEPENDS'>RDEPENDS >> variable. >> >> >> >> - > id='var-DESCRIPTION'>DESCRIPTION >> + > id='var-bb-DESCRIPTION'>DESCRIPTION >> >> A long description for the recipe. >> @@ -1465,7 +1465,7 @@ >> >> >> >> - DL_DIR >> + DL_DIR >> >> >> The central download directory used by the build >> process to @@ -1474,7 +1474,7 @@ >> suitable for mirroring for everything except Git >> repositories. >> If you want tarballs of Git repositories, use the >> - > linkend='var-BB_GENERATE_MIRROR_TARBALLS'>BB_GENERATE_MIRROR_TARBALLS >> + > linkend='var-bb-BB_GENERATE_MIRROR_TARBALLS'>BB_GENERATE_MIRROR_TARBALLS >> variable. >> >> @@ -1482,9 +1482,9 @@ >> >> >> >> - E >> + E >> >> - > id='var-EXCLUDE_FROM_WORLD'>EXCLUDE_FROM_WORLD >> + > id='var-bb-EXCLUDE_FROM_WORLD'>EXCLUDE_FROM_WORLD >> >> Directs BitBake to exclude a recipe from world >> builds (i.e. @@ -1512,9 +1512,9 @@ >> >> >> >> - F >> + F >> >> - FAKEROOT >> + > id='var-bb-FAKEROOT'>FAKEROOT >> >> Contains the command to use when running a >> shell script @@ -1527,19 +1527,19 @@ >> >> >> >> - > id='var-FAKEROOTBASEENV'>FAKEROOTBASEENV >> + > id='var-bb-FAKEROOTBASEENV'>FAKEROOTBASEENV >> >> Lists environment variables to set when >> executing the command defined by >> - > linkend='var-FAKEROOTCMD'>FAKEROOTCMD >> + > linkend='var-bb-FAKEROOTCMD'>FAKEROOTCMD >> that starts the bitbake-worker process in the fakeroot environment. >> >> >> >> >> - > id='var-FAKEROOTCMD'>FAKEROOTCMD >> + > id='var-bb-FAKEROOTCMD'>FAKEROOTCMD >> >> Contains the command that starts the >> bitbake-worker @@ -1548,7 +1548,7 @@ >> >> >> >> - > id='var-FAKEROOTDIRS'>FAKEROOTDIRS >> + > id='var-bb-FAKEROOTDIRS'>FAKEROOTDIRS >> >> Lists directories to create before running a >> task in @@ -1557,33 +1557,33 @@ >> >> >> >> - > id='var-FAKEROOTENV'>FAKEROOTENV >> + > id='var-bb-FAKEROOTENV'>FAKEROOTENV >> >> Lists environment variables to set when running >> a task in the fakeroot environment. >> For additional information on environment >> variables and the fakeroot environment, see the >> - > linkend='var-FAKEROOTBASEENV'>FAKEROOTBASEENV >> + > linkend='var-bb-FAKEROOTBASEENV'>FAKEROOTBASEENV >> variable. >> >> >> >> - > id='var-FAKEROOTNOENV'>FAKEROOTNOENV >> + > id='var-bb-FAKEROOTNOENV'>FAKEROOTNOENV >> >> Lists environment variables to set when running >> a task that is not in the fakeroot environment. >> For additional information on environment >> variables and the fakeroot environment, see the >> - > linkend='var-FAKEROOTENV'>FAKEROOTENV >> + > linkend='var-bb-FAKEROOTENV'>FAKEROOTENV >> variable. >> >> >> >> - FETCHCMD >> + > id='var-bb-FETCHCMD'>FETCHCMD >> >> Defines the command the BitBake fetcher module >> @@ -1595,7 +1595,7 @@ >> >> >> >> - FILE >> + FILE >> >> >> Points at the current file. >> @@ -1607,7 +1607,7 @@ >> >> >> >> - > id='var-FILESPATH'>FILESPATH >> + > id='var-bb-FILESPATH'>FILESPATH >> >> Specifies directories BitBake uses when >> searching for @@ -1625,9 +1625,9 @@ >> >> >> >> - G >> + G >> >> - GITDIR >> + GITDIR >> >> >> The directory in which a local copy of a Git >> repository @@ -1639,9 +1639,9 @@ >> >> >> >> - H >> + H >> >> - HGDIR >> + HGDIR >> >> >> The directory in which files checked out of a >> Mercurial @@ -1650,7 +1650,7 @@ >> >> >> >> - HOMEPAGE >> + > id='var-bb-HOMEPAGE'>HOMEPAGE >> Website where more information about the >> software the recipe is building can be found. >> @@ -1659,9 +1659,9 @@ >> >> >> >> - I >> + I >> >> - INHERIT >> + > id='var-bb-INHERIT'>INHERIT >> >> Causes the named class or classes to be >> inherited globally. @@ -1691,15 +1691,15 @@ >> >> --> >> >> - L >> + L >> >> - > id='var-LAYERDEPENDS'>LAYERDEPENDS >> + > id='var-bb-LAYERDEPENDS'>LAYERDEPENDS >> Lists the layers, separated by spaces, upon which >> this recipe depends. Optionally, you can specify a specific layer >> version for a dependency by adding it to the end of the layer name >> with a colon, (e.g. "anotherlayer:3" to be compared against >> - > linkend='var-LAYERVERSION'>LAYERVERSION_anotherlayer >> + > linkend='var-bb-LAYERVERSION'>LAYERVERSION_anotherlayer >> in this case). BitBake produces an error if any dependency is missing >> or the version numbers do not match exactly (if specified). >> @@ -1710,7 +1710,7 @@ >> >> >> >> - LAYERDIR >> + > id='var-bb-LAYERDIR'>LAYERDIR >> When used inside the >> layer.conf configuration file, this variable >> provides the path of the current layer. @@ -1719,22 +1719,22 @@ >> >> >> >> - > id='var-LAYERDIR_RE'>LAYERDIR_RE >> + > id='var-bb-LAYERDIR_RE'>LAYERDIR_RE >> When used inside the >> layer.conf configuration file, this variable >> provides the path of the current layer, escaped for use in a regular >> expression >> - (> linkend='var-BBFILE_PATTERN'>BBFILE_PATTERN). >> + (> linkend='var-bb-BBFILE_PATTERN'>BBFILE_PATTERN). >> This variable is not available outside of >> layer.conf and references are expanded >> immediately when parsing of the file completes. >> >> - > id='var-LAYERVERSION'>LAYERVERSION >> + > id='var-bb-LAYERVERSION'>LAYERVERSION >> Optionally specifies the version of a layer as a >> single number. You can use this variable within >> - > linkend='var-LAYERDEPENDS'>LAYERDEPENDS >> + > linkend='var-bb-LAYERDEPENDS'>LAYERDEPENDS >> for another layer in order to depend on a specific version of the >> layer. >> @@ -1744,7 +1744,7 @@ >> >> >> >> - LICENSE >> + > id='var-bb-LICENSE'>LICENSE >> >> The list of source licenses for the recipe. >> @@ -1754,9 +1754,9 @@ >> >> >> >> - M >> + M >> >> - MIRRORS >> + > id='var-bb-MIRRORS'>MIRRORS >> >> Specifies additional paths from which BitBake >> gets source code. @@ -1764,14 +1764,14 @@ >> tries the local download directory. >> If that location fails, the build system tries >> locations defined by >> - > linkend='var-PREMIRRORS'>PREMIRRORS, >> + > linkend='var-bb-PREMIRRORS'>PREMIRRORS, >> the upstream source, and then locations specified by >> MIRRORS in that order. >> >> >> >> - > id='var-MULTI_PROVIDER_WHITELIST'>MULTI_PROVIDER_WHITELIST >> + > id='var-bb-MULTI_PROVIDER_WHITELIST'>MULTI_PROVIDER_WHITELIST >> >> Allows you to suppress BitBake warnings caused >> when @@ -1804,9 +1804,9 @@ >> >> --> >> >> - O >> + O >> >> - > id='var-OVERRIDES'>OVERRIDES >> + > id='var-bb-OVERRIDES'>OVERRIDES >> >> BitBake uses OVERRIDES to >> control @@ -1829,9 +1829,9 @@ >> >> >> >> - P >> + P >> >> - P4DIR >> + P4DIR >> >> >> The directory in which a local copy of a >> Perforce depot @@ -1840,14 +1840,14 @@ >> >> >> >> - PACKAGES >> + > id='var-bb-PACKAGES'>PACKAGES >> The list of packages the recipe creates. >> >> >> >> >> - > id='var-PACKAGES_DYNAMIC'>PACKAGES_DYNAMIC >> + > id='var-bb-PACKAGES_DYNAMIC'>PACKAGES_DYNAMIC >> >> A promise that your recipe satisfies runtime >> dependencies @@ -1856,7 +1856,7 @@ >> does not actually satisfy the dependencies, it >> only states that they should be satisfied. >> For example, if a hard, runtime dependency >> - (> linkend='var-RDEPENDS'>RDEPENDS) >> + (> linkend='var-bb-RDEPENDS'>RDEPENDS) of >> another package is satisfied during the build through the >> PACKAGES_DYNAMIC variable, but a package with >> the module name is never actually @@ -1865,7 +1865,7 @@ >> >> >> >> - PE >> + PE >> >> >> The epoch of the recipe. >> @@ -1877,7 +1877,7 @@ >> >> >> >> - > id='var-PERSISTENT_DIR'>PERSISTENT_DIR >> + > id='var-bb-PERSISTENT_DIR'>PERSISTENT_DIR >> >> Specifies the directory BitBake uses to store >> data that @@ -1889,7 +1889,7 @@ >> >> >> >> - PF >> + PF >> >> >> Specifies the recipe or package name and >> includes all version and revision @@ -1899,27 +1899,27 @@ >> >> >> >> - PN >> + PN >> >> The recipe name. >> >> >> >> - PR >> + PR >> >> The revision of the recipe. >> >> >> >> >> - > id='var-PREFERRED_PROVIDER'>PREFERRED_PROVIDER >> + > id='var-bb-PREFERRED_PROVIDER'>PREFERRED_PROVIDER >> >> Determines which recipe should be given >> preference when multiple recipes provide the same item. >> You should always suffix the variable with the >> name of the provided item, and you should set it to the >> - > linkend='var-PN'>PN >> + > linkend='var-bb-PN'>PN of the recipe to >> which you want to give precedence. Some examples: >> >> @@ -1931,14 +1931,14 @@ >> >> >> >> - > id='var-PREFERRED_PROVIDERS'>PREFERRED_PROVIDERS >> + > id='var-bb-PREFERRED_PROVIDERS'>PREFERRED_PROVIDERS >> >> Determines which recipe should be given >> preference for cases where multiple recipes provide the same item. >> Functionally, >> PREFERRED_PROVIDERS is >> identical to >> - > linkend='var-PREFERRED_PROVIDER'>PREFERRED_PROVIDER. >> + > linkend='var-bb-PREFERRED_PROVIDER'>PREFERRED_PROVIDER. >> However, the PREFERRED_PROVIDERS variable lets >> you define preferences for multiple situations using the following >> form: @@ -1954,15 +1954,15 @@ >> >> >> >> - > id='var-PREFERRED_VERSION'>PREFERRED_VERSION >> + > id='var-bb-PREFERRED_VERSION'>PREFERRED_VERSION >> >> If there are multiple versions of recipes >> available, this variable determines which recipe should be given >> preference. You must always suffix the variable with the >> - > linkend='var-PN'>PN >> + > linkend='var-bb-PN'>PN you want to >> select, and you should set >> - > linkend='var-PV'>PV >> + > linkend='var-bb-PV'>PV accordingly for >> precedence. >> >> @@ -1989,7 +1989,7 @@ >> >> >> >> - > id='var-PREMIRRORS'>PREMIRRORS >> + > id='var-bb-PREMIRRORS'>PREMIRRORS >> >> Specifies additional paths from which BitBake >> gets source code. @@ -1998,7 +1998,7 @@ >> If that location fails, the build system tries >> locations defined by PREMIRRORS, the upstream >> source, and then locations specified by >> - > linkend='var-MIRRORS'>MIRRORS >> + > linkend='var-bb-MIRRORS'>MIRRORS in that >> order. >> >> @@ -2022,20 +2022,20 @@ >> >> >> >> - PROVIDES >> + > id='var-bb-PROVIDES'>PROVIDES >> >> A list of aliases by which a particular recipe >> can be known. >> By default, a recipe's own >> - > linkend='var-PN'>PN >> + > linkend='var-bb-PN'>PN is implicitly already in its >> PROVIDES list. >> If a recipe uses PROVIDES, >> the additional aliases are synonyms for the recipe and can >> be useful satisfying dependencies of other >> recipes during the build as specified by >> - > linkend='var-DEPENDS'>DEPENDS. >> + > linkend='var-bb-DEPENDS'>DEPENDS. >> >> >> @@ -2059,7 +2059,7 @@ >> virtual target in PROVIDES. >> Recipes that depend on the functionality in >> question can include the virtual target in >> - > linkend='var-DEPENDS'>DEPENDS >> + > linkend='var-bb-DEPENDS'>DEPENDS to leave >> the choice of provider open. >> >> @@ -2072,11 +2072,11 @@ >> >> >> >> - > id='var-PRSERV_HOST'>PRSERV_HOST >> + > id='var-bb-PRSERV_HOST'>PRSERV_HOST >> >> The network based >> - > linkend='var-PR'>PR >> + > linkend='var-bb-PR'>PR service host and >> port. >> >> @@ -2094,7 +2094,7 @@ >> >> >> >> - PV >> + PV >> >> The version of the recipe. >> >> @@ -2108,9 +2108,9 @@ >> >> --> >> >> - R >> + R >> >> - RDEPENDS >> + > id='var-bb-RDEPENDS'>RDEPENDS >> >> Lists a package's runtime dependencies (i.e. >> other packages) @@ -2165,13 +2165,13 @@ >> >> >> For information on build-time dependencies, see >> the >> - > linkend='var-DEPENDS'>DEPENDS >> + > linkend='var-bb-DEPENDS'>DEPENDS variable. >> >> >> >> >> - REPODIR >> + > id='var-bb-REPODIR'>REPODIR >> >> The directory in which a local copy of a >> @@ -2181,14 +2181,14 @@ >> >> >> >> - > id='var-RPROVIDES'>RPROVIDES >> + > id='var-bb-RPROVIDES'>RPROVIDES >> >> A list of package name aliases that a package >> also provides. These aliases are useful for satisfying runtime >> dependencies of other packages both during the build and on the target >> (as specified by >> - > linkend='var-RDEPENDS'>RDEPENDS). >> + > linkend='var-bb-RDEPENDS'>RDEPENDS). >> >> As with all package-controlling variables, you >> must always @@ -2201,7 +2201,7 @@ >> >> >> >> - > id='var-RRECOMMENDS'>RRECOMMENDS >> + > id='var-bb-RRECOMMENDS'>RRECOMMENDS >> >> A list of packages that extends the usability of >> a package @@ -2210,7 +2210,7 @@ >> packages in order to successfully build, but >> needs them for the extended usability. >> To specify runtime dependencies for packages, >> see the >> - > linkend='var-RDEPENDS'>RDEPENDS >> + > linkend='var-bb-RDEPENDS'>RDEPENDS variable. >> >> >> @@ -2243,15 +2243,15 @@ >> >> >> >> - S >> + S >> >> - SECTION >> + > id='var-bb-SECTION'>SECTION >> The section in which packages should be >> categorized. >> >> >> - SRC_URI >> + > id='var-bb-SRC_URI'>SRC_URI >> >> The list of source files - local or remote. >> @@ -2272,7 +2272,7 @@ >> the metadata, >> from the local machine. >> The path is relative to the >> - > linkend='var-FILESPATH'>FILESPATH >> + > linkend='var-bb-FILESPATH'>FILESPATH >> variable. >> bzr:// - >> Fetches files from a Bazaar revision control >> repository. @@ -2322,7 +2322,7 @@ >> >> >> - SRCDATE >> + > id='var-bb-SRCDATE'>SRCDATE >> >> The date of the source code used to build the >> package. @@ -2331,7 +2331,7 @@ >> >> >> >> - SRCREV >> + SRCREV >> >> >> The revision of the source code used to build >> the package. @@ -2344,13 +2344,13 @@ >> >> >> >> - > id='var-SRCREV_FORMAT'>SRCREV_FORMAT >> + > id='var-bb-SRCREV_FORMAT'>SRCREV_FORMAT >> >> Helps construct valid >> - > linkend='var-SRCREV'>SRCREV >> + > linkend='var-bb-SRCREV'>SRCREV values >> when multiple source controlled URLs are used in >> - > linkend='var-SRC_URI'>SRC_URI. >> + > linkend='var-bb-SRC_URI'>SRC_URI. >> >> >> @@ -2371,7 +2371,7 @@ >> >> >> >> - STAMP >> + STAMP >> >> >> Specifies the base path used to create recipe >> stamp files. @@ -2381,12 +2381,12 @@ >> >> >> >> - > id='var-STAMPCLEAN'>STAMPCLEAN >> + > id='var-bb-STAMPCLEAN'>STAMPCLEAN >> >> Specifies the base path used to create recipe >> stamp files. Unlike the >> - > linkend='var-STAMP'>STAMP >> + > linkend='var-bb-STAMP'>STAMP variable, >> STAMPCLEAN can contain wildcards to match the >> range of files a clean operation should remove. >> @@ -2396,7 +2396,7 @@ >> >> >> >> - SUMMARY >> + > id='var-bb-SUMMARY'>SUMMARY >> >> A short summary for the recipe, which is 72 >> characters or less. @@ -2404,7 +2404,7 @@ >> >> >> >> - SVNDIR >> + SVNDIR >> >> >> The directory in which files checked out of a >> Subversion @@ -2415,9 +2415,9 @@ >> >> >> >> - T >> + T >> >> - T >> + T >> >> Points to a directory were BitBake places >> temporary files, which consist mostly of task >> logs and @@ -2426,7 +2426,7 @@ >> >> >> >> - TOPDIR >> + TOPDIR >> >> >> Points to the build directory. >> diff --git a/bitbake/doc/poky.ent b/bitbake/doc/poky.ent >> index c032e14..85d9c83 100644 >> --- a/bitbake/doc/poky.ent >> +++ b/bitbake/doc/poky.ent >> @@ -17,13 +17,6 @@ >> >> >> > "http://developer.berlios.de/projects/bitbake/"> -> ECLIPSE_MAIN_URL "http://www.eclipse.org/downloads"> -> ECLIPSE_DL_URL "http://download.eclipse.org"> -> ECLIPSE_DL_PLUGIN_URL >> "&YOCTO_DL_URL;/releases/eclipse-plugin/&DISTRO;"> -> ECLIPSE_UPDATES_URL "&ECLIPSE_DL_URL;/tm/updates/3.3"> -> ECLIPSE_INDIGO_URL "&ECLIPSE_DL_URL;/releases/indigo"> -> ECLIPSE_JUNO_URL "&ECLIPSE_DL_URL;/releases/juno"> -> ECLIPSE_INDIGO_CDT_URL "&ECLIPSE_DL_URL;tools/cdt/releases/indigo"> >> > YOCTO_SOURCES_URL "&YOCTO_HOME_URL;/sources/"> > YOCTO_AB_PORT_URL "&YOCTO_AB_URL;:8010"> @@ -31,7 +24,6 @@ > YOCTO_POKY_URL "&YOCTO_DL_URL;/releases/poky/"> > YOCTO_RELEASE_DL_URL "&YOCTO_DL_URL;/releases/yocto/yocto-&DISTRO;"> >> >> -> "&YOCTO_RELEASE_DL_URL;/eclipse-plugin/indigo;"> > YOCTO_ADTINSTALLER_DL_URL "&YOCTO_RELEASE_DL_URL;/adt_installer"> >> > "&YOCTO_RELEASE_DL_URL;/&YOCTO_POKY;.tar.bz2"> > YOCTO_MACHINES_DL_URL "&YOCTO_RELEASE_DL_URL;/machines"> diff --git >> a/bitbake/lib/bb/COW.py b/bitbake/lib/bb/COW.py index >> 7817473..d26e981 100644 --- a/bitbake/lib/bb/COW.py +++ >> b/bitbake/lib/bb/COW.py @@ -1,23 +1,8 @@ -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> # >> # This is a copy on write dictionary and set which abuses classes to >> try and be nice and fast. # >> # Copyright (C) 2006 Tim Ansell >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> -# >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. -# >> #Please Note: >> # Be careful when using mutable types (ie Dict and Lists) - >> operations involving these are SLOW. # Assign a file to __warn__ to >> get warnings about slow operations. diff --git >> a/bitbake/lib/bb/__init__.py b/bitbake/lib/bb/__init__.py index >> 4bc47c8..c144311 100644 --- a/bitbake/lib/bb/__init__.py >> +++ b/bitbake/lib/bb/__init__.py >> @@ -1,5 +1,3 @@ >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> # >> # BitBake Build System Python Library >> # >> @@ -8,20 +6,10 @@ >> # >> # Based on Gentoo's portage.py. >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> +# SPDX-License-Identifier: GPL-2.0-only >> # >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. >> -__version__ = "1.40.0" >> +__version__ = "1.44.0" >> >> import sys >> if sys.version_info < (3, 4, 0): >> diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py >> index 3e2a94e..30a2ba2 100644 >> --- a/bitbake/lib/bb/build.py >> +++ b/bitbake/lib/bb/build.py >> @@ -1,5 +1,3 @@ >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> # >> # BitBake 'Build' implementation >> # >> @@ -10,18 +8,7 @@ >> # >> # Based on Gentoo's portage.py. >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> -# >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only >> # >> # Based on functions from the base bb module, Copyright 2003 Holger >> Schurig >> @@ -67,23 +54,6 @@ else: >> builtins['bb'] = bb >> builtins['os'] = os >> >> -class FuncFailed(Exception): >> - def __init__(self, name = None, logfile = None): >> - self.logfile = logfile >> - self.name = name >> - if name: >> - self.msg = 'Function failed: %s' % name >> - else: >> - self.msg = "Function failed" >> - >> - def __str__(self): >> - if self.logfile and os.path.exists(self.logfile): >> - msg = ("%s (log file is located at %s)" % >> - (self.msg, self.logfile)) >> - else: >> - msg = self.msg >> - return msg >> - >> class TaskBase(event.Event): >> """Base class for task events""" >> >> @@ -176,15 +146,33 @@ class LogTee(object): >> >> def __repr__(self): >> return ''.format(self.name) >> + >> def flush(self): >> self.outfile.flush() >> >> -# >> -# pythonexception allows the python exceptions generated to be raised >> -# as the real exceptions (not FuncFailed) and without a backtrace at >> the -# origin of the failure. >> -# >> -def exec_func(func, d, dirs = None, pythonexception=False): >> + >> +class StdoutNoopContextManager: >> + """ >> + This class acts like sys.stdout, but adds noop __enter__ and >> __exit__ methods. >> + """ >> + def __enter__(self): >> + return sys.stdout >> + >> + def __exit__(self, *exc_info): >> + pass >> + >> + def write(self, string): >> + return sys.stdout.write(string) >> + >> + def flush(self): >> + sys.stdout.flush() >> + >> + @property >> + def name(self): >> + return sys.stdout.name >> + >> + >> +def exec_func(func, d, dirs = None): >> """Execute a BB 'function'""" >> >> try: >> @@ -256,7 +244,7 @@ def exec_func(func, d, dirs = None, >> pythonexception=False): >> with bb.utils.fileslocked(lockfiles): >> if ispython: >> - exec_func_python(func, d, runfile, cwd=adir, >> pythonexception=pythonexception) >> + exec_func_python(func, d, runfile, cwd=adir) >> else: >> exec_func_shell(func, d, runfile, cwd=adir) >> >> @@ -276,7 +264,7 @@ _functionfmt = """ >> {function}(d) >> """ >> logformatter = bb.msg.BBLogFormatter("%(levelname)s: %(message)s") >> -def exec_func_python(func, d, runfile, cwd=None, >> pythonexception=False): +def exec_func_python(func, d, runfile, >> cwd=None): """Execute a python BB 'function'""" >> >> code = _functionfmt.format(function=func) >> @@ -301,13 +289,7 @@ def exec_func_python(func, d, runfile, cwd=None, >> pythonexception=False): bb.methodpool.insert_method(func, text, fn, >> lineno - 1) >> comp = utils.better_compile(code, func, "exec_python_func() >> autogenerated") >> - utils.better_exec(comp, {"d": d}, code, "exec_python_func() >> autogenerated", pythonexception=pythonexception) >> - except (bb.parse.SkipRecipe, bb.build.FuncFailed): >> - raise >> - except: >> - if pythonexception: >> - raise >> - raise FuncFailed(func, None) >> + utils.better_exec(comp, {"d": d}, code, "exec_python_func() >> autogenerated") finally: >> bb.debug(2, "Python function %s finished" % func) >> >> @@ -335,6 +317,42 @@ trap 'bb_exit_handler' 0 >> set -e >> ''' >> >> +def create_progress_handler(func, progress, logfile, d): >> + if progress == 'percent': >> + # Use default regex >> + return bb.progress.BasicProgressHandler(d, outfile=logfile) >> + elif progress.startswith('percent:'): >> + # Use specified regex >> + return bb.progress.BasicProgressHandler(d, >> regex=progress.split(':', 1)[1], outfile=logfile) >> + elif progress.startswith('outof:'): >> + # Use specified regex >> + return bb.progress.OutOfProgressHandler(d, >> regex=progress.split(':', 1)[1], outfile=logfile) >> + elif progress.startswith("custom:"): >> + # Use a custom progress handler that was injected via >> OE_EXTRA_IMPORTS or __builtins__ >> + import functools >> + from types import ModuleType >> + >> + parts = progress.split(":", 2) >> + _, cls, otherargs = parts[0], parts[1], (parts[2] or None) >> if parts[2:] else None >> + if cls: >> + def resolve(x, y): >> + if not x: >> + return None >> + if isinstance(x, ModuleType): >> + return getattr(x, y, None) >> + return x.get(y) >> + cls_obj = functools.reduce(resolve, cls.split("."), >> bb.utils._context) >> + if not cls_obj: >> + # Fall-back on __builtins__ >> + cls_obj = functools.reduce(lambda x, y: x.get(y), >> cls.split("."), __builtins__) >> + if cls_obj: >> + return cls_obj(d, outfile=logfile, >> otherargs=otherargs) >> + bb.warn('%s: unknown custom progress handler in task >> progress varflag value "%s", ignoring' % (func, cls)) >> + else: >> + bb.warn('%s: invalid task progress varflag value "%s", >> ignoring' % (func, progress)) + >> + return logfile >> + >> def exec_func_shell(func, d, runfile, cwd=None): >> """Execute a shell function from the metadata >> >> @@ -372,23 +390,13 @@ exit $ret >> cmd = [fakerootcmd, runfile] >> >> if bb.msg.loggerDefaultVerbose: >> - logfile = LogTee(logger, sys.stdout) >> + logfile = LogTee(logger, StdoutNoopContextManager()) >> else: >> - logfile = sys.stdout >> + logfile = StdoutNoopContextManager() >> >> progress = d.getVarFlag(func, 'progress') >> if progress: >> - if progress == 'percent': >> - # Use default regex >> - logfile = bb.progress.BasicProgressHandler(d, >> outfile=logfile) >> - elif progress.startswith('percent:'): >> - # Use specified regex >> - logfile = bb.progress.BasicProgressHandler(d, >> regex=progress.split(':', 1)[1], outfile=logfile) >> - elif progress.startswith('outof:'): >> - # Use specified regex >> - logfile = bb.progress.OutOfProgressHandler(d, >> regex=progress.split(':', 1)[1], outfile=logfile) >> - else: >> - bb.warn('%s: invalid task progress varflag value "%s", >> ignoring' % (func, progress)) >> + logfile = create_progress_handler(func, progress, logfile, d) >> >> fifobuffer = bytearray() >> def readfifo(data): >> @@ -407,6 +415,8 @@ exit $ret >> bb.plain(value) >> elif cmd == 'bbnote': >> bb.note(value) >> + elif cmd == 'bbverbnote': >> + bb.verbnote(value) >> elif cmd == 'bbwarn': >> bb.warn(value) >> elif cmd == 'bberror': >> @@ -436,13 +446,8 @@ exit $ret >> with open(fifopath, 'r+b', buffering=0) as fifo: >> try: >> bb.debug(2, "Executing shell function %s" % func) >> - >> - try: >> - with open(os.devnull, 'r+') as stdin: >> - bb.process.run(cmd, shell=False, stdin=stdin, >> log=logfile, extrafiles=[(fifo,readfifo)]) >> - except bb.process.CmdError: >> - logfn = d.getVar('BB_LOGFILE') >> - raise FuncFailed(func, logfn) >> + with open(os.devnull, 'r+') as stdin, logfile: >> + bb.process.run(cmd, shell=False, stdin=stdin, >> log=logfile, extrafiles=[(fifo,readfifo)]) finally: >> os.unlink(fifopath) >> >> @@ -570,9 +575,6 @@ def _exec_task(fn, task, d, quieterr): >> event.fire(TaskStarted(task, logfn, flags, localdata), >> localdata) except (bb.BBHandledException, SystemExit): >> return 1 >> - except FuncFailed as exc: >> - logger.error(str(exc)) >> - return 1 >> >> try: >> for func in (prefuncs or '').split(): >> @@ -580,7 +582,10 @@ def _exec_task(fn, task, d, quieterr): >> exec_func(task, localdata) >> for func in (postfuncs or '').split(): >> exec_func(func, localdata) >> - except FuncFailed as exc: >> + except bb.BBHandledException: >> + event.fire(TaskFailed(task, logfn, localdata, True), >> localdata) >> + return 1 >> + except Exception as exc: >> if quieterr: >> event.fire(TaskFailedSilent(task, logfn, localdata), >> localdata) else: >> @@ -588,9 +593,6 @@ def _exec_task(fn, task, d, quieterr): >> logger.error(str(exc)) >> event.fire(TaskFailed(task, logfn, localdata, >> errprinted), localdata) return 1 >> - except bb.BBHandledException: >> - event.fire(TaskFailed(task, logfn, localdata, True), >> localdata) >> - return 1 >> finally: >> sys.stdout.flush() >> sys.stderr.flush() >> @@ -814,6 +816,9 @@ def add_tasks(tasklist, d): >> task_deps['parents'][task] = [] >> if 'deps' in flags: >> for dep in flags['deps']: >> + # Check and warn for "addtask task after foo" while >> foo does not exist >> + #if not dep in tasklist: >> + # bb.warn('%s: dependent task %s for %s does not >> exist' % (d.getVar('PN'), dep, task)) dep = d.expand(dep) >> task_deps['parents'][task].append(dep) >> >> diff --git a/bitbake/lib/bb/cache.py b/bitbake/lib/bb/cache.py >> index 258d679..b6f7da5 100644 >> --- a/bitbake/lib/bb/cache.py >> +++ b/bitbake/lib/bb/cache.py >> @@ -1,5 +1,3 @@ >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> # >> # BitBake Cache implementation >> # >> @@ -15,18 +13,8 @@ >> # Copyright (C) 2005 Holger Hans Peter Freyther >> # Copyright (C) 2005 ROAD GmbH >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> +# SPDX-License-Identifier: GPL-2.0-only >> # >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. >> import os >> import sys >> @@ -95,21 +83,21 @@ class CoreRecipeInfo(RecipeInfoCommon): >> self.appends = self.listvar('__BBAPPEND', metadata) >> self.nocache = self.getvar('BB_DONT_CACHE', metadata) >> >> + self.provides = self.depvar('PROVIDES', metadata) >> + self.rprovides = self.depvar('RPROVIDES', metadata) >> + self.pn = self.getvar('PN', metadata) or >> bb.parse.vars_from_file(filename,metadata)[0] >> + self.packages = self.listvar('PACKAGES', metadata) >> + if not self.packages: >> + self.packages.append(self.pn) >> + self.packages_dynamic = self.listvar('PACKAGES_DYNAMIC', >> metadata) + >> self.skipreason = self.getvar('__SKIPPED', metadata) >> if self.skipreason: >> - self.pn = self.getvar('PN', metadata) or >> bb.parse.BBHandler.vars_from_file(filename,metadata)[0] self.skipped >> = True >> - self.provides = self.depvar('PROVIDES', metadata) >> - self.rprovides = self.depvar('RPROVIDES', metadata) >> return >> >> self.tasks = metadata.getVar('__BBTASKS', False) >> >> - self.pn = self.getvar('PN', metadata) >> - self.packages = self.listvar('PACKAGES', metadata) >> - if not self.packages: >> - self.packages.append(self.pn) >> - >> self.basetaskhashes = self.taskvar('BB_BASEHASH', >> self.tasks, metadata) self.hashfilename = >> self.getvar('BB_HASHFILENAME', metadata) >> @@ -125,11 +113,8 @@ class CoreRecipeInfo(RecipeInfoCommon): >> self.stampclean = self.getvar('STAMPCLEAN', metadata) >> self.stamp_extrainfo = self.flaglist('stamp-extra-info', >> self.tasks, metadata) self.file_checksums = >> self.flaglist('file-checksums', self.tasks, metadata, True) >> - self.packages_dynamic = self.listvar('PACKAGES_DYNAMIC', >> metadata) self.depends = self.depvar('DEPENDS', metadata) >> - self.provides = self.depvar('PROVIDES', metadata) >> self.rdepends = self.depvar('RDEPENDS', metadata) >> - self.rprovides = self.depvar('RPROVIDES', metadata) >> self.rrecommends = self.depvar('RRECOMMENDS', metadata) >> self.rprovides_pkg = self.pkgvar('RPROVIDES', >> self.packages, metadata) self.rdepends_pkg = >> self.pkgvar('RDEPENDS', self.packages, metadata) @@ -235,7 +220,7 @@ >> class CoreRecipeInfo(RecipeInfoCommon): >> cachedata.hashfn[fn] = self.hashfilename >> for task, taskhash in self.basetaskhashes.items(): >> - identifier = '%s.%s' % (fn, task) >> + identifier = '%s:%s' % (fn, task) >> cachedata.basetaskhash[identifier] = taskhash >> >> cachedata.inherits[fn] = self.inherits >> @@ -249,7 +234,7 @@ def virtualfn2realfn(virtualfn): >> Convert a virtual file name to a real one + the associated >> subclass keyword """ >> mc = "" >> - if virtualfn.startswith('multiconfig:'): >> + if virtualfn.startswith('mc:'): >> elems = virtualfn.split(':') >> mc = elems[1] >> virtualfn = ":".join(elems[2:]) >> @@ -270,7 +255,7 @@ def realfn2virtual(realfn, cls, mc): >> if cls: >> realfn = "virtual:" + cls + ":" + realfn >> if mc: >> - realfn = "multiconfig:" + mc + ":" + realfn >> + realfn = "mc:" + mc + ":" + realfn >> return realfn >> >> def variant2virtual(realfn, variant): >> @@ -279,11 +264,11 @@ def variant2virtual(realfn, variant): >> """ >> if variant == "": >> return realfn >> - if variant.startswith("multiconfig:"): >> + if variant.startswith("mc:"): >> elems = variant.split(":") >> if elems[2]: >> - return "multiconfig:" + elems[1] + ":virtual:" + >> ":".join(elems[2:]) + ":" + realfn >> - return "multiconfig:" + elems[1] + ":" + realfn >> + return "mc:" + elems[1] + ":virtual:" + >> ":".join(elems[2:]) + ":" + realfn >> + return "mc:" + elems[1] + ":" + realfn >> return "virtual:" + variant + ":" + realfn >> >> def parse_recipe(bb_data, bbfile, appends, mc=''): >> @@ -361,7 +346,7 @@ class NoCache(object): >> bb_data = self.databuilder.mcdata[mc].createCopy() >> newstores = parse_recipe(bb_data, bbfile, appends, mc) >> for ns in newstores: >> - datastores["multiconfig:%s:%s" % (mc, ns)] = >> newstores[ns] >> + datastores["mc:%s:%s" % (mc, ns)] = newstores[ns] >> >> return datastores >> >> @@ -411,6 +396,15 @@ class Cache(NoCache): >> else: >> logger.debug(1, "Cache file %s not found, building..." % >> self.cachefile) >> + # We don't use the symlink, its just for debugging >> convinience >> + symlink = os.path.join(self.cachedir, "bb_cache.dat") >> + if os.path.exists(symlink): >> + bb.utils.remove(symlink) >> + try: >> + os.symlink(os.path.basename(self.cachefile), symlink) >> + except OSError: >> + pass >> + >> def load_cachefile(self): >> cachesize = 0 >> previous_progress = 0 >> @@ -889,3 +883,56 @@ class MultiProcessCache(object): >> p.dump([data, self.__class__.CACHE_VERSION]) >> >> bb.utils.unlockfile(glf) >> + >> + >> +class SimpleCache(object): >> + """ >> + BitBake multi-process cache implementation >> + >> + Used by the codeparser & file checksum caches >> + """ >> + >> + def __init__(self, version): >> + self.cachefile = None >> + self.cachedata = None >> + self.cacheversion = version >> + >> + def init_cache(self, d, cache_file_name=None, defaultdata=None): >> + cachedir = (d.getVar("PERSISTENT_DIR") or >> + d.getVar("CACHE")) >> + if not cachedir: >> + return defaultdata >> + >> + bb.utils.mkdirhier(cachedir) >> + self.cachefile = os.path.join(cachedir, >> + cache_file_name or >> self.__class__.cache_file_name) >> + logger.debug(1, "Using cache in '%s'", self.cachefile) >> + >> + glf = bb.utils.lockfile(self.cachefile + ".lock") >> + >> + try: >> + with open(self.cachefile, "rb") as f: >> + p = pickle.Unpickler(f) >> + data, version = p.load() >> + except: >> + bb.utils.unlockfile(glf) >> + return defaultdata >> + >> + bb.utils.unlockfile(glf) >> + >> + if version != self.cacheversion: >> + return defaultdata >> + >> + return data >> + >> + def save(self, data): >> + if not self.cachefile: >> + return >> + >> + glf = bb.utils.lockfile(self.cachefile + ".lock") >> + >> + with open(self.cachefile, "wb") as f: >> + p = pickle.Pickler(f, -1) >> + p.dump([data, self.cacheversion]) >> + >> + bb.utils.unlockfile(glf) >> diff --git a/bitbake/lib/bb/cache_extra.py >> b/bitbake/lib/bb/cache_extra.py index 83f4959..bf4226d 100644 >> --- a/bitbake/lib/bb/cache_extra.py >> +++ b/bitbake/lib/bb/cache_extra.py >> @@ -1,5 +1,3 @@ >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> # >> # Extra RecipeInfo will be all defined in this file. Currently, >> # Only Hob (Image Creator) Requests some extra fields. So >> @@ -12,18 +10,8 @@ >> >> # Copyright (C) 2011, Intel Corporation. All rights reserved. >> >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> +# SPDX-License-Identifier: GPL-2.0-only >> # >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. >> from bb.cache import RecipeInfoCommon >> >> diff --git a/bitbake/lib/bb/checksum.py b/bitbake/lib/bb/checksum.py >> index 4e1598f..5bc8a8f 100644 >> --- a/bitbake/lib/bb/checksum.py >> +++ b/bitbake/lib/bb/checksum.py >> @@ -2,18 +2,8 @@ >> # >> # Copyright (C) 2012 Intel Corporation >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> +# SPDX-License-Identifier: GPL-2.0-only >> # >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. >> import glob >> import operator >> diff --git a/bitbake/lib/bb/codeparser.py >> b/bitbake/lib/bb/codeparser.py index ddd1b97..fd2c473 100644 >> --- a/bitbake/lib/bb/codeparser.py >> +++ b/bitbake/lib/bb/codeparser.py >> @@ -1,3 +1,7 @@ >> +# >> +# SPDX-License-Identifier: GPL-2.0-only >> +# >> + >> """ >> BitBake code parser >> >> @@ -33,7 +37,7 @@ from bb.cache import MultiProcessCache >> logger = logging.getLogger('BitBake.CodeParser') >> >> def bbhash(s): >> - return hashlib.md5(s.encode("utf-8")).hexdigest() >> + return hashlib.sha256(s.encode("utf-8")).hexdigest() >> >> def check_indent(codestr): >> """If the code is indented, add a top level piece of code to >> 'remove' the indentation""" @@ -140,7 +144,7 @@ class >> CodeParserCache(MultiProcessCache): # so that an existing cache gets >> invalidated. Additionally you'll need # to increment >> __cache_version__ in cache.py in order to ensure that old # recipe >> caches don't trigger "Taskhash mismatch" errors. >> - CACHE_VERSION = 10 >> + CACHE_VERSION = 11 >> >> def __init__(self): >> MultiProcessCache.__init__(self) >> @@ -368,8 +372,9 @@ class ShellParser(): >> def _parse_shell(self, value): >> try: >> tokens, _ = pyshyacc.parse(value, eof=True, debug=False) >> - except pyshlex.NeedMore: >> - raise sherrors.ShellSyntaxError("Unexpected EOF") >> + except Exception: >> + bb.error('Error during parse shell code, the last 5 >> lines are:\n%s' % '\n'.join(value.split('\n')[-5:])) >> + raise >> >> self.process_tokens(tokens) >> >> diff --git a/bitbake/lib/bb/command.py b/bitbake/lib/bb/command.py >> index 6c966e3..378f389 100644 >> --- a/bitbake/lib/bb/command.py >> +++ b/bitbake/lib/bb/command.py >> @@ -6,18 +6,8 @@ Provide an interface to interact with the bitbake >> server through 'commands' >> # Copyright (C) 2006-2007 Richard Purdie >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> +# SPDX-License-Identifier: GPL-2.0-only >> # >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. >> """ >> The bitbake server takes 'commands' from its UI/commandline. >> diff --git a/bitbake/lib/bb/compat.py b/bitbake/lib/bb/compat.py >> index de1923d..4935668 100644 >> --- a/bitbake/lib/bb/compat.py >> +++ b/bitbake/lib/bb/compat.py >> @@ -1,3 +1,7 @@ >> +# >> +# SPDX-License-Identifier: GPL-2.0-only >> +# >> + >> """Code pulled from future python versions, here for compatibility""" >> >> from collections import MutableMapping, KeysView, ValuesView, >> ItemsView, OrderedDict diff --git a/bitbake/lib/bb/cooker.py >> b/bitbake/lib/bb/cooker.py index 16681ba..20ef04d 100644 >> --- a/bitbake/lib/bb/cooker.py >> +++ b/bitbake/lib/bb/cooker.py >> @@ -1,6 +1,3 @@ >> -#!/usr/bin/env python >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> # >> # Copyright (C) 2003, 2004 Chris Larson >> # Copyright (C) 2003, 2004 Phil Blundell >> @@ -9,19 +6,8 @@ >> # Copyright (C) 2005 ROAD GmbH >> # Copyright (C) 2006 - 2007 Richard Purdie >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> +# SPDX-License-Identifier: GPL-2.0-only >> # >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. - >> >> import sys, os, glob, os.path, re, time >> import atexit >> @@ -45,6 +31,7 @@ import pyinotify >> import json >> import pickle >> import codecs >> +import hashserv >> >> logger = logging.getLogger("BitBake") >> collectlog = logging.getLogger("BitBake.Collection") >> @@ -175,27 +162,45 @@ class BBCooker: >> >> self.configuration = configuration >> >> + bb.debug(1, "BBCooker starting %s" % time.time()) >> + sys.stdout.flush() >> + >> self.configwatcher = pyinotify.WatchManager() >> + bb.debug(1, "BBCooker pyinotify1 %s" % time.time()) >> + sys.stdout.flush() >> + >> self.configwatcher.bbseen = [] >> self.configwatcher.bbwatchedfiles = [] >> self.confignotifier = pyinotify.Notifier(self.configwatcher, >> self.config_notifications) >> + bb.debug(1, "BBCooker pyinotify2 %s" % time.time()) >> + sys.stdout.flush() >> self.watchmask = pyinotify.IN_CLOSE_WRITE | >> pyinotify.IN_CREATE | pyinotify.IN_DELETE | \ >> pyinotify.IN_DELETE_SELF | pyinotify.IN_MODIFY | >> pyinotify.IN_MOVE_SELF | \ pyinotify.IN_MOVED_FROM | >> pyinotify.IN_MOVED_TO self.watcher = pyinotify.WatchManager() >> + bb.debug(1, "BBCooker pyinotify3 %s" % time.time()) >> + sys.stdout.flush() >> self.watcher.bbseen = [] >> self.watcher.bbwatchedfiles = [] >> self.notifier = pyinotify.Notifier(self.watcher, >> self.notifications) >> + bb.debug(1, "BBCooker pyinotify complete %s" % time.time()) >> + sys.stdout.flush() >> + >> # If being called by something like tinfoil, we need to >> clean cached data # which may now be invalid >> bb.parse.clear_cache() >> bb.parse.BBHandler.cached_statements = {} >> >> self.ui_cmdline = None >> + self.hashserv = None >> + self.hashservaddr = None >> >> self.initConfigurationData() >> >> + bb.debug(1, "BBCooker parsed base configuration %s" % >> time.time()) >> + sys.stdout.flush() >> + >> # we log all events to a file if so directed >> if self.configuration.writeeventlog: >> # register the log file writer as UI Handler >> @@ -233,6 +238,9 @@ class BBCooker: >> # Let SIGHUP exit as SIGTERM >> signal.signal(signal.SIGHUP, self.sigterm_exception) >> >> + bb.debug(1, "BBCooker startup complete %s" % time.time()) >> + sys.stdout.flush() >> + >> def process_inotify_updates(self): >> for n in [self.confignotifier, self.notifier]: >> if n.check_events(timeout=0): >> @@ -367,13 +375,12 @@ class BBCooker: >> # Copy of the data store which has been expanded. >> # Used for firing events and accessing variables where >> expansion needs to be accounted for # >> - bb.parse.init_parser(self.data) >> - >> if CookerFeatures.BASEDATASTORE_TRACKING in self.featureset: >> self.disableDataTracking() >> >> - self.data.renameVar("__depends", "__base_depends") >> - self.add_filewatch(self.data.getVar("__base_depends", >> False), self.configwatcher) >> + for mc in self.databuilder.mcdata.values(): >> + mc.renameVar("__depends", "__base_depends") >> + self.add_filewatch(mc.getVar("__base_depends", False), >> self.configwatcher) >> self.baseconfig_valid = True >> self.parsecache_valid = False >> @@ -385,6 +392,22 @@ class BBCooker: >> except prserv.serv.PRServiceConfigError as e: >> bb.fatal("Unable to start PR Server, exitting") >> >> + if self.data.getVar("BB_HASHSERVE") == "auto": >> + # Create a new hash server bound to a unix domain socket >> + if not self.hashserv: >> + dbfile = (self.data.getVar("PERSISTENT_DIR") or >> self.data.getVar("CACHE")) + "/hashserv.db" >> + self.hashservaddr = "unix://%s/hashserve.sock" % >> self.data.getVar("TOPDIR") >> + self.hashserv = >> hashserv.create_server(self.hashservaddr, dbfile, sync=False) >> + self.hashserv.process = >> multiprocessing.Process(target=self.hashserv.serve_forever) >> + self.hashserv.process.start() >> + self.data.setVar("BB_HASHSERVE", self.hashservaddr) >> + self.databuilder.origdata.setVar("BB_HASHSERVE", >> self.hashservaddr) >> + self.databuilder.data.setVar("BB_HASHSERVE", >> self.hashservaddr) >> + for mc in self.databuilder.mcdata: >> + self.databuilder.mcdata[mc].setVar("BB_HASHSERVE", >> self.hashservaddr) + >> + bb.parse.init_parser(self.data) >> + >> def enableDataTracking(self): >> self.configuration.tracking = True >> if hasattr(self, "data"): >> @@ -488,6 +511,7 @@ class BBCooker: >> """ >> fn = None >> envdata = None >> + mc = '' >> if not pkgs_to_build: >> pkgs_to_build = [] >> >> @@ -496,6 +520,12 @@ class BBCooker: >> self.enableDataTracking() >> self.reset() >> >> + def mc_base(p): >> + if p.startswith('mc:'): >> + s = p.split(':') >> + if len(s) == 2: >> + return s[1] >> + return None >> >> if buildfile: >> # Parse the configuration here. We need to do it >> explicitly here since @@ -506,18 +536,16 @@ class BBCooker: >> fn = self.matchFile(fn) >> fn = bb.cache.realfn2virtual(fn, cls, mc) >> elif len(pkgs_to_build) == 1: >> - ignore = self.data.getVar("ASSUME_PROVIDED") or "" >> - if pkgs_to_build[0] in set(ignore.split()): >> - bb.fatal("%s is in ASSUME_PROVIDED" % >> pkgs_to_build[0]) >> + mc = mc_base(pkgs_to_build[0]) >> + if not mc: >> + ignore = self.data.getVar("ASSUME_PROVIDED") or "" >> + if pkgs_to_build[0] in set(ignore.split()): >> + bb.fatal("%s is in ASSUME_PROVIDED" % >> pkgs_to_build[0]) >> - taskdata, runlist = self.buildTaskData(pkgs_to_build, >> None, self.configuration.abort, allowincomplete=True) >> + taskdata, runlist = >> self.buildTaskData(pkgs_to_build, None, self.configuration.abort, >> allowincomplete=True) >> - mc = runlist[0][0] >> - fn = runlist[0][3] >> - else: >> - envdata = self.data >> - data.expandKeys(envdata) >> - parse.ast.runAnonFuncs(envdata) >> + mc = runlist[0][0] >> + fn = runlist[0][3] >> >> if fn: >> try: >> @@ -526,6 +554,12 @@ class BBCooker: >> except Exception as e: >> parselog.exception("Unable to read %s", fn) >> raise >> + else: >> + if not mc in self.databuilder.mcdata: >> + bb.fatal('Not multiconfig named "%s" found' % mc) >> + envdata = self.databuilder.mcdata[mc] >> + data.expandKeys(envdata) >> + parse.ast.runAnonFuncs(envdata) >> >> # Display history >> with closing(StringIO()) as env: >> @@ -565,10 +599,10 @@ class BBCooker: >> wildcard = False >> >> # Wild card expansion: >> - # Replace string such as "multiconfig:*:bash" >> - # into "multiconfig:A:bash multiconfig:B:bash bash" >> + # Replace string such as "mc:*:bash" >> + # into "mc:A:bash mc:B:bash bash" >> for k in targetlist: >> - if k.startswith("multiconfig:"): >> + if k.startswith("mc:"): >> if wildcard: >> bb.fatal('multiconfig conflict') >> if k.split(":")[1] == "*": >> @@ -601,7 +635,7 @@ class BBCooker: >> runlist = [] >> for k in fulltargetlist: >> mc = "" >> - if k.startswith("multiconfig:"): >> + if k.startswith("mc:"): >> mc = k.split(":")[1] >> k = ":".join(k.split(":")[2:]) >> ktask = task >> @@ -620,13 +654,22 @@ class BBCooker: >> runlist.append([mc, k, ktask, fn]) >> bb.event.fire(bb.event.TreeDataPreparationProgress(current, >> len(fulltargetlist)), self.data) >> - mcdeps = taskdata[mc].get_mcdepends() >> + havemc = False >> + for mc in self.multiconfigs: >> + if taskdata[mc].get_mcdepends(): >> + havemc = True >> + >> # No need to do check providers if there are no mcdeps or >> not an mc build >> - if mcdeps and mc: >> - # Make sure we can provide the multiconfig dependency >> + if havemc or len(self.multiconfigs) > 1: >> seen = set() >> new = True >> + # Make sure we can provide the multiconfig dependency >> while new: >> + mcdeps = set() >> + # Add unresolved first, so we can get multiconfig >> indirect dependencies on time >> + for mc in self.multiconfigs: >> + taskdata[mc].add_unresolved(localdata[mc], >> self.recipecaches[mc]) >> + mcdeps |= set(taskdata[mc].get_mcdepends()) >> new = False >> for mc in self.multiconfigs: >> for k in mcdeps: >> @@ -641,6 +684,7 @@ class BBCooker: >> taskdata[depmc].add_provider(localdata[depmc], >> self.recipecaches[depmc], l[3]) seen.add(k) >> new = True >> + >> for mc in self.multiconfigs: >> taskdata[mc].add_unresolved(localdata[mc], >> self.recipecaches[mc]) >> @@ -676,7 +720,7 @@ class BBCooker: >> @staticmethod >> def add_mc_prefix(mc, pn): >> if mc: >> - return "multiconfig:%s:%s" % (mc, pn) >> + return "mc:%s:%s" % (mc, pn) >> return pn >> >> def buildDependTree(self, rq, taskdata): >> @@ -875,6 +919,10 @@ class BBCooker: >> os.unlink('package-depends.dot') >> except FileNotFoundError: >> pass >> + try: >> + os.unlink('recipe-depends.dot') >> + except FileNotFoundError: >> + pass >> >> with open('task-depends.dot', 'w') as f: >> f.write("digraph depends {\n") >> @@ -888,27 +936,6 @@ class BBCooker: >> f.write("}\n") >> logger.info("Task dependencies saved to 'task-depends.dot'") >> >> - with open('recipe-depends.dot', 'w') as f: >> - f.write("digraph depends {\n") >> - pndeps = {} >> - for task in sorted(depgraph["tdepends"]): >> - (pn, taskname) = task.rsplit(".", 1) >> - if pn not in pndeps: >> - pndeps[pn] = set() >> - for dep in sorted(depgraph["tdepends"][task]): >> - (deppn, deptaskname) = dep.rsplit(".", 1) >> - pndeps[pn].add(deppn) >> - for pn in sorted(pndeps): >> - fn = depgraph["pn"][pn]["filename"] >> - version = depgraph["pn"][pn]["version"] >> - f.write('"%s" [label="%s\\n%s\\n%s"]\n' % (pn, pn, >> version, fn)) >> - for dep in sorted(pndeps[pn]): >> - if dep == pn: >> - continue >> - f.write('"%s" -> "%s"\n' % (pn, dep)) >> - f.write("}\n") >> - logger.info("Flattened recipe dependencies saved to >> 'recipe-depends.dot'") - >> def show_appends_with_no_recipes(self): >> # Determine which bbappends haven't been applied >> >> @@ -1191,8 +1218,8 @@ class BBCooker: >> continue >> elif regex == "": >> parselog.debug(1, "BBFILE_PATTERN_%s is empty" % >> c) >> + cre = re.compile('^NULL$') >> errors = False >> - continue >> else: >> try: >> cre = re.compile(regex) >> @@ -1453,7 +1480,7 @@ class BBCooker: >> ntargets = [] >> for target in runlist: >> if target[0]: >> - ntargets.append("multiconfig:%s:%s:%s" % (target[0], >> target[1], target[2])) >> + ntargets.append("mc:%s:%s:%s" % (target[0], >> target[1], target[2])) ntargets.append("%s:%s" % (target[1], >> target[2])) >> for mc in self.multiconfigs: >> @@ -1576,6 +1603,9 @@ class BBCooker: >> for pkg in pkgs_to_build: >> if pkg in ignore: >> parselog.warning("Explicit target \"%s\" is in >> ASSUME_PROVIDED, ignoring" % pkg) >> + if pkg.startswith("multiconfig:"): >> + pkgs_to_build.remove(pkg) >> + pkgs_to_build.append(pkg.replace("multiconfig:", >> "mc:")) >> if 'world' in pkgs_to_build: >> pkgs_to_build.remove('world') >> @@ -1583,7 +1613,7 @@ class BBCooker: >> bb.providers.buildWorldTargetList(self.recipecaches[mc], >> task) for t in self.recipecaches[mc].world_target: >> if mc: >> - t = "multiconfig:" + mc + ":" + t >> + t = "mc:" + mc + ":" + t >> pkgs_to_build.append(t) >> >> if 'universe' in pkgs_to_build: >> @@ -1602,7 +1632,7 @@ class BBCooker: >> bb.debug(1, "Skipping %s for universe >> tasks as task %s doesn't exist" % (t, task)) continue >> if mc: >> - t = "multiconfig:" + mc + ":" + t >> + t = "mc:" + mc + ":" + t >> pkgs_to_build.append(t) >> >> return pkgs_to_build >> @@ -1615,9 +1645,11 @@ class BBCooker: >> >> def post_serve(self): >> prserv.serv.auto_shutdown() >> + if self.hashserv: >> + self.hashserv.process.terminate() >> + self.hashserv.process.join() >> bb.event.fire(CookerExit(), self.data) >> >> - >> def shutdown(self, force = False): >> if force: >> self.state = state.forceshutdown >> @@ -1632,6 +1664,7 @@ class BBCooker: >> >> def reset(self): >> self.initConfigurationData() >> + self.handlePRServ() >> >> def clientComplete(self): >> """Called when the client is done using the server""" >> @@ -1865,35 +1898,6 @@ class ParsingFailure(Exception): >> self.recipe = recipe >> Exception.__init__(self, realexception, recipe) >> >> -class Feeder(multiprocessing.Process): >> - def __init__(self, jobs, to_parsers, quit): >> - self.quit = quit >> - self.jobs = jobs >> - self.to_parsers = to_parsers >> - multiprocessing.Process.__init__(self) >> - >> - def run(self): >> - while True: >> - try: >> - quit = self.quit.get_nowait() >> - except queue.Empty: >> - pass >> - else: >> - if quit == 'cancel': >> - self.to_parsers.cancel_join_thread() >> - break >> - >> - try: >> - job = self.jobs.pop() >> - except IndexError: >> - break >> - >> - try: >> - self.to_parsers.put(job, timeout=0.5) >> - except queue.Full: >> - self.jobs.insert(0, job) >> - continue >> - >> class Parser(multiprocessing.Process): >> def __init__(self, jobs, results, quit, init, profile): >> self.jobs = jobs >> @@ -1940,11 +1944,8 @@ class Parser(multiprocessing.Process): >> result = pending.pop() >> else: >> try: >> - job = self.jobs.get(timeout=0.25) >> - except queue.Empty: >> - continue >> - >> - if job is None: >> + job = self.jobs.pop() >> + except IndexError: >> break >> result = self.parse(*job) >> >> @@ -2028,14 +2029,15 @@ class CookerParser(object): >> multiprocessing.util.Finalize(None, >> bb.codeparser.parser_cache_save, exitpriority=1) >> multiprocessing.util.Finalize(None, bb.fetch.fetcher_parse_save, >> exitpriority=1) >> - self.feeder_quit = multiprocessing.Queue(maxsize=1) >> self.parser_quit = >> multiprocessing.Queue(maxsize=self.num_processes) >> - self.jobs = >> multiprocessing.Queue(maxsize=self.num_processes) self.result_queue = >> multiprocessing.Queue() >> - self.feeder = Feeder(self.willparse, self.jobs, >> self.feeder_quit) >> - self.feeder.start() >> + >> + def chunkify(lst,n): >> + return [lst[i::n] for i in range(n)] >> + self.jobs = chunkify(self.willparse, self.num_processes) >> + >> for i in range(0, self.num_processes): >> - parser = Parser(self.jobs, self.result_queue, >> self.parser_quit, init, self.cooker.configuration.profile) >> + parser = Parser(self.jobs[i], self.result_queue, >> self.parser_quit, init, self.cooker.configuration.profile) >> parser.start() self.process_names.append(parser.name) >> self.processes.append(parser) >> @@ -2056,17 +2058,20 @@ class CookerParser(object): >> self.total) >> >> bb.event.fire(event, self.cfgdata) >> - self.feeder_quit.put(None) >> for process in self.processes: >> self.parser_quit.put(None) >> else: >> - self.feeder_quit.put('cancel') >> - >> self.parser_quit.cancel_join_thread() >> for process in self.processes: >> self.parser_quit.put(None) >> >> - self.jobs.cancel_join_thread() >> + # Cleanup the queue before call process.join(), otherwise >> there might be >> + # deadlocks. >> + while True: >> + try: >> + self.result_queue.get(timeout=0.25) >> + except queue.Empty: >> + break >> >> for process in self.processes: >> if force: >> @@ -2074,7 +2079,6 @@ class CookerParser(object): >> process.terminate() >> else: >> process.join() >> - self.feeder.join() >> >> sync = threading.Thread(target=self.bb_cache.sync) >> sync.start() >> diff --git a/bitbake/lib/bb/cookerdata.py >> b/bitbake/lib/bb/cookerdata.py index 5df66e6..472423f 100644 >> --- a/bitbake/lib/bb/cookerdata.py >> +++ b/bitbake/lib/bb/cookerdata.py >> @@ -1,6 +1,3 @@ >> -#!/usr/bin/env python >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> # >> # Copyright (C) 2003, 2004 Chris Larson >> # Copyright (C) 2003, 2004 Phil Blundell >> @@ -9,23 +6,14 @@ >> # Copyright (C) 2005 ROAD GmbH >> # Copyright (C) 2006 Richard Purdie >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> +# SPDX-License-Identifier: GPL-2.0-only >> # >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. >> import logging >> import os >> import re >> import sys >> +import hashlib >> from functools import wraps >> import bb >> from bb import data >> @@ -134,6 +122,7 @@ class CookerConfiguration(object): >> self.profile = False >> self.nosetscene = False >> self.setsceneonly = False >> + self.skipsetscene = False >> self.invalidate_stamp = False >> self.dump_signatures = [] >> self.dry_run = False >> @@ -279,12 +268,13 @@ class CookerDataBuilder(object): >> self.mcdata = {} >> >> def parseBaseConfiguration(self): >> + data_hash = hashlib.sha256() >> try: >> - bb.parse.init_parser(self.basedata) >> self.data = self.parseConfigurationFiles(self.prefiles, >> self.postfiles) >> if self.data.getVar("BB_WORKERCONTEXT", False) is None: >> bb.fetch.fetcher_init(self.data) >> + bb.parse.init_parser(self.data) >> bb.codeparser.parser_cache_init(self.data) >> >> bb.event.fire(bb.event.ConfigParsed(), self.data) >> @@ -302,7 +292,7 @@ class CookerDataBuilder(object): >> bb.event.fire(bb.event.ConfigParsed(), self.data) >> >> bb.parse.init_parser(self.data) >> - self.data_hash = self.data.get_hash() >> + data_hash.update(self.data.get_hash().encode('utf-8')) >> self.mcdata[''] = self.data >> >> multiconfig = (self.data.getVar("BBMULTICONFIG") or >> "").split() @@ -310,9 +300,11 @@ class CookerDataBuilder(object): >> mcdata = self.parseConfigurationFiles(self.prefiles, >> self.postfiles, config) bb.event.fire(bb.event.ConfigParsed(), mcdata) >> self.mcdata[config] = mcdata >> + data_hash.update(mcdata.get_hash().encode('utf-8')) >> if multiconfig: >> bb.event.fire(bb.event.MultiConfigParsed(self.mcdata), >> self.data) >> + self.data_hash = data_hash.hexdigest() >> except (SyntaxError, bb.BBHandledException): >> raise bb.BBHandledException >> except bb.data_smart.ExpansionError as e: >> @@ -354,14 +346,24 @@ class CookerDataBuilder(object): >> data = parse_config_file(layerconf, data) >> >> layers = (data.getVar('BBLAYERS') or "").split() >> + broken_layers = [] >> >> data = bb.data.createCopy(data) >> approved = bb.utils.approved_variables() >> + >> + # Check whether present layer directories exist >> for layer in layers: >> if not os.path.isdir(layer): >> - parselog.critical("Layer directory '%s' does not >> exist! " >> - "Please check BBLAYERS in %s" >> % (layer, layerconf)) >> - sys.exit(1) >> + broken_layers.append(layer) >> + >> + if broken_layers: >> + parselog.critical("The following layer directories >> do not exist:") >> + for layer in broken_layers: >> + parselog.critical(" %s", layer) >> + parselog.critical("Please check BBLAYERS in %s" % >> (layerconf)) >> + sys.exit(1) >> + >> + for layer in layers: >> parselog.debug(2, "Adding layer %s", layer) >> if 'HOME' in approved and '~' in layer: >> layer = os.path.expanduser(layer) >> @@ -391,7 +393,11 @@ class CookerDataBuilder(object): >> bb.fatal("BBFILES_DYNAMIC entries must be of the >> form :, not:\n %s" % "\n >> ".join(invalid)) layerseries = >> set((data.getVar("LAYERSERIES_CORENAMES") or "").split()) >> + collections_tmp = collections[:] >> for c in collections: >> + collections_tmp.remove(c) >> + if c in collections_tmp: >> + bb.fatal("Found duplicated BBFILE_COLLECTIONS >> '%s', check bblayers.conf or layer.conf to fix it." % c) compat = >> set((data.getVar("LAYERSERIES_COMPAT_%s" % c) or "").split()) if >> compat and not (compat & layerseries): bb.fatal("Layer %s is not >> compatible with the core layer which only supports these series: %s >> (layer is compatible with %s)" diff --git >> a/bitbake/lib/bb/daemonize.py b/bitbake/lib/bb/daemonize.py index >> c937675..f01e6ec 100644 --- a/bitbake/lib/bb/daemonize.py +++ >> b/bitbake/lib/bb/daemonize.py @@ -1,3 +1,7 @@ >> +# >> +# SPDX-License-Identifier: GPL-2.0-only >> +# >> + >> """ >> Python Daemonizing helper >> >> diff --git a/bitbake/lib/bb/data.py b/bitbake/lib/bb/data.py >> index d66d98c..0d75d0c 100644 >> --- a/bitbake/lib/bb/data.py >> +++ b/bitbake/lib/bb/data.py >> @@ -1,5 +1,3 @@ >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> """ >> BitBake 'Data' implementations >> >> @@ -22,18 +20,7 @@ the speed is more critical here. >> # Copyright (C) 2003, 2004 Chris Larson >> # Copyright (C) 2005 Holger Hans Peter Freyther >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> -# >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only >> # >> # Based on functions from the base bb module, Copyright 2003 Holger >> Schurig >> @@ -143,7 +130,7 @@ def emit_var(var, o=sys.__stdout__, d = init(), >> all=False): if all: >> oval = d.getVar(var, False) >> val = d.getVar(var) >> - except (KeyboardInterrupt, bb.build.FuncFailed): >> + except (KeyboardInterrupt): >> raise >> except Exception as exc: >> o.write('# expansion of %s threw %s: %s\n' % (var, >> exc.__class__.__name__, str(exc))) @@ -322,8 +309,6 @@ def >> build_dependencies(key, keys, shelldeps, varflagsexcl, d): if >> varflags.get("python"): value = d.getVarFlag(key, "_content", False) >> parser = bb.codeparser.PythonParser(key, logger) >> - if value and "\t" in value: >> - logger.warning("Variable %s contains tabs, >> please remove these (%s)" % (key, d.getVar("FILE"))) >> parser.parse_python(value, filename=varflags.get("filename"), >> lineno=varflags.get("lineno")) deps = deps | parser.references deps = >> deps | (keys & parser.execs) @@ -437,8 +422,8 @@ def >> generate_dependency_hash(tasklist, gendeps, lookupcache, whitelist, >> fn): var = lookupcache[dep] if var is not None: >> data = data + str(var) >> - k = fn + "." + task >> - basehash[k] = hashlib.md5(data.encode("utf-8")).hexdigest() >> + k = fn + ":" + task >> + basehash[k] = >> hashlib.sha256(data.encode("utf-8")).hexdigest() taskdeps[task] = >> alldeps >> return taskdeps, basehash >> diff --git a/bitbake/lib/bb/data_smart.py >> b/bitbake/lib/bb/data_smart.py index 67af380..dd5c618 100644 >> --- a/bitbake/lib/bb/data_smart.py >> +++ b/bitbake/lib/bb/data_smart.py >> @@ -1,5 +1,3 @@ >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> """ >> BitBake Smart Dictionary Implementation >> >> @@ -14,18 +12,8 @@ BitBake build tools. >> # Copyright (C) 2005 Uli Luckas >> # Copyright (C) 2005 ROAD GmbH >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> +# SPDX-License-Identifier: GPL-2.0-only >> # >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. # Based on functions from the base bb module, >> Copyright 2003 Holger Schurig >> import copy, re, sys, traceback >> @@ -39,10 +27,11 @@ from bb.COW import COWDictBase >> logger = logging.getLogger("BitBake.Data") >> >> __setvar_keyword__ = ["_append", "_prepend", "_remove"] >> -__setvar_regexp__ = >> re.compile('(?P.*?)(?P_append|_prepend|_remove)(_(?P[^A-Z]*))?$') >> -__expand_var_regexp__ = re.compile(r"\${[^{}@\n\t :]+}") >> +__setvar_regexp__ = >> re.compile(r'(?P.*?)(?P_append|_prepend|_remove)(_(?P[^A-Z]*))?$') >> +__expand_var_regexp__ = re.compile(r"\${[a-zA-Z0-9\-_+./~]+?}") >> __expand_python_regexp__ = re.compile(r"\${@.+?}") >> -__whitespace_split__ = re.compile('(\s)') +__whitespace_split__ = >> re.compile(r'(\s)') +__override_regexp__ = re.compile(r'[a-z0-9]+') >> def infer_caller_details(loginfo, parent = False, varval = True): >> """Save the caller the trouble of specifying everything.""" >> @@ -597,7 +586,7 @@ class DataSmart(MutableMapping): >> # aka pay the cookie monster >> override = var[var.rfind('_')+1:] >> shortvar = var[:var.rfind('_')] >> - while override and override.islower(): >> + while override and __override_regexp__.match(override): >> if shortvar not in self.overridedata: >> self.overridedata[shortvar] = [] >> if [var, override] not in self.overridedata[shortvar]: >> @@ -1073,4 +1062,4 @@ class DataSmart(MutableMapping): >> data.update({i:value}) >> >> data_str = str([(k, data[k]) for k in sorted(data.keys())]) >> - return hashlib.md5(data_str.encode("utf-8")).hexdigest() >> + return hashlib.sha256(data_str.encode("utf-8")).hexdigest() >> diff --git a/bitbake/lib/bb/event.py b/bitbake/lib/bb/event.py >> index 5b1b094..d44621e 100644 >> --- a/bitbake/lib/bb/event.py >> +++ b/bitbake/lib/bb/event.py >> @@ -1,5 +1,3 @@ >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> """ >> BitBake 'Event' implementation >> >> @@ -9,18 +7,8 @@ BitBake build tools. >> >> # Copyright (C) 2003, 2004 Chris Larson >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> +# SPDX-License-Identifier: GPL-2.0-only >> # >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. >> import os, sys >> import warnings >> @@ -136,6 +124,7 @@ def fire_class_handlers(event, d): >> ui_queue = [] >> @atexit.register >> def print_ui_queue(): >> + global ui_queue >> """If we're exiting before a UI has been spawned, display any >> queued LogRecords to the console.""" >> logger = logging.getLogger("BitBake") >> @@ -180,6 +169,7 @@ def print_ui_queue(): >> logger.removeHandler(stderr) >> else: >> logger.removeHandler(stdout) >> + ui_queue = [] >> >> def fire_ui_handlers(event, d): >> global _thread_lock >> @@ -414,23 +404,6 @@ class RecipeTaskPreProcess(RecipeEvent): >> class RecipeParsed(RecipeEvent): >> """ Recipe Parsing Complete """ >> >> -class StampUpdate(Event): >> - """Trigger for any adjustment of the stamp files to happen""" >> - >> - def __init__(self, targets, stampfns): >> - self._targets = targets >> - self._stampfns = stampfns >> - Event.__init__(self) >> - >> - def getStampPrefix(self): >> - return self._stampfns >> - >> - def getTargets(self): >> - return self._targets >> - >> - stampPrefix = property(getStampPrefix) >> - targets = property(getTargets) >> - >> class BuildBase(Event): >> """Base class for bitbake build events""" >> >> diff --git a/bitbake/lib/bb/exceptions.py >> b/bitbake/lib/bb/exceptions.py index cd71343..ecbad59 100644 >> --- a/bitbake/lib/bb/exceptions.py >> +++ b/bitbake/lib/bb/exceptions.py >> @@ -1,3 +1,6 @@ >> +# >> +# SPDX-License-Identifier: GPL-2.0-only >> +# >> >> import inspect >> import traceback >> diff --git a/bitbake/lib/bb/fetch2/__init__.py >> b/bitbake/lib/bb/fetch2/__init__.py index 572b71a..1f5f8f1 100644 >> --- a/bitbake/lib/bb/fetch2/__init__.py >> +++ b/bitbake/lib/bb/fetch2/__init__.py >> @@ -1,5 +1,3 @@ >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> """ >> BitBake 'Fetch' implementations >> >> @@ -10,18 +8,7 @@ BitBake build tools. >> # Copyright (C) 2003, 2004 Chris Larson >> # Copyright (C) 2012 Intel Corporation >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> -# >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only >> # >> # Based on functions from the base bb module, Copyright 2003 Holger >> Schurig >> @@ -256,7 +243,7 @@ class URI(object): >> >> # Identify if the URI is relative or not >> if urlp.scheme in self._relative_schemes and \ >> - re.compile("^\w+:(?!//)").match(uri): >> + re.compile(r"^\w+:(?!//)").match(uri): >> self.relative = True >> >> if not self.relative: >> @@ -524,7 +511,7 @@ def fetcher_parse_save(): >> def fetcher_parse_done(): >> _checksum_cache.save_merge() >> >> -def fetcher_compare_revisions(): >> +def fetcher_compare_revisions(d): >> """ >> Compare the revisions in the persistant cache with current >> values and return true/false on whether they've changed. >> @@ -777,7 +764,8 @@ def get_srcrev(d, >> method_name='sortable_revision'): # >> format = d.getVar('SRCREV_FORMAT') >> if not format: >> - raise FetchError("The SRCREV_FORMAT variable must be set >> when multiple SCMs are used.") >> + raise FetchError("The SRCREV_FORMAT variable must be set >> when multiple SCMs are used.\n"\ >> + "The SCMs are:\n%s" % '\n'.join(scms)) >> >> name_to_rev = {} >> seenautoinc = False >> @@ -855,10 +843,18 @@ def runfetchcmd(cmd, d, quiet=False, >> cleanup=None, log=None, workdir=None): if val: >> cmd = 'export ' + var + '=\"%s\"; %s' % (val, cmd) >> >> + # Ensure that a _PYTHON_SYSCONFIGDATA_NAME value set by a recipe >> + # (for example via python3native.bbclass since warrior) is not >> set for >> + # host Python (otherwise tools like git-make-shallow will fail) >> + cmd = 'unset _PYTHON_SYSCONFIGDATA_NAME; ' + cmd >> + >> # Disable pseudo as it may affect ssh, potentially causing it to >> hang. cmd = 'export PSEUDO_DISABLED=1; ' + cmd >> >> - logger.debug(1, "Running %s", cmd) >> + if workdir: >> + logger.debug(1, "Running '%s' in %s" % (cmd, workdir)) >> + else: >> + logger.debug(1, "Running %s", cmd) >> >> success = False >> error_message = "" >> @@ -894,7 +890,7 @@ def check_network_access(d, info, url): >> log remote network access, and error if BB_NO_NETWORK is set or >> the given URI is untrusted >> """ >> - if d.getVar("BB_NO_NETWORK") == "1": >> + if bb.utils.to_boolean(d.getVar("BB_NO_NETWORK")): >> raise NetworkAccess(url, info) >> elif not trusted_network(d, url): >> raise UntrustedUrl(url, info) >> @@ -966,7 +962,8 @@ def rename_bad_checksum(ud, suffix): >> >> new_localpath = "%s_bad-checksum_%s" % (ud.localpath, suffix) >> bb.warn("Renaming %s to %s" % (ud.localpath, new_localpath)) >> - bb.utils.movefile(ud.localpath, new_localpath) >> + if not bb.utils.movefile(ud.localpath, new_localpath): >> + bb.warn("Renaming %s to %s failed, grep movefile in >> log.do_fetch to see why" % (ud.localpath, new_localpath)) >> >> def try_mirror_url(fetch, origud, ud, ld, check = False): >> @@ -1027,7 +1024,7 @@ def try_mirror_url(fetch, origud, ud, ld, check >> = False): raise >> >> except IOError as e: >> - if e.errno in [os.errno.ESTALE]: >> + if e.errno in [errno.ESTALE]: >> logger.warning("Stale Error Observed %s." % ud.url) >> return False >> raise >> @@ -1094,7 +1091,7 @@ def trusted_network(d, url): >> BB_ALLOWED_NETWORKS is set globally or for a specific recipe. >> Note: modifies SRC_URI & mirrors. >> """ >> - if d.getVar('BB_NO_NETWORK') == "1": >> + if bb.utils.to_boolean(d.getVar("BB_NO_NETWORK")): >> return True >> >> pkgname = d.expand(d.getVar('PN', False)) >> @@ -1403,7 +1400,7 @@ class FetchMethod(object): >> Fetch urls >> Assumes localpath was called first >> """ >> - raise NoMethodError(url) >> + raise NoMethodError(urldata.url) >> >> def unpack(self, urldata, rootdir, data): >> iterate = False >> @@ -1469,7 +1466,7 @@ class FetchMethod(object): >> else: >> cmd = 'rpm2cpio.sh %s | cpio -id' % (file) >> elif file.endswith('.deb') or file.endswith('.ipk'): >> - output = subprocess.check_output('ar -t %s' % file, >> preexec_fn=subprocess_setup, shell=True) >> + output = subprocess.check_output(['ar', '-t', file], >> preexec_fn=subprocess_setup) datafile = None >> if output: >> for line in output.decode().splitlines(): >> @@ -1547,7 +1544,7 @@ class FetchMethod(object): >> Check the status of a URL >> Assumes localpath was called first >> """ >> - logger.info("URL %s could not be checked for status since no >> method exists.", url) >> + logger.info("URL %s could not be checked for status since no >> method exists.", urldata.url) return True >> >> def latest_revision(self, ud, d, name): >> @@ -1555,7 +1552,7 @@ class FetchMethod(object): >> Look in the cache for the latest revision, if not present >> ask the SCM. """ >> if not hasattr(self, "_latest_revision"): >> - raise ParameterError("The fetcher for this URL does not >> support _latest_revision", url) >> + raise ParameterError("The fetcher for this URL does not >> support _latest_revision", ud.url) >> revs = bb.persist_data.persist('BB_URI_HEADREVS', d) >> key = self.generate_revision_key(ud, d, name) >> @@ -1638,7 +1635,7 @@ class Fetch(object): >> urls = self.urls >> >> network = self.d.getVar("BB_NO_NETWORK") >> - premirroronly = (self.d.getVar("BB_FETCH_PREMIRRORONLY") == >> "1") >> + premirroronly = >> bb.utils.to_boolean(self.d.getVar("BB_FETCH_PREMIRRORONLY")) >> for u in urls: >> ud = self.ud[u] >> @@ -1716,7 +1713,7 @@ class Fetch(object): >> update_stamp(ud, self.d) >> >> except IOError as e: >> - if e.errno in [os.errno.ESTALE]: >> + if e.errno in [errno.ESTALE]: >> logger.error("Stale Error Observed %s." % u) >> raise ChecksumError("Stale Error Detected") >> >> @@ -1786,7 +1783,7 @@ class Fetch(object): >> >> for url in urls: >> if url not in self.ud: >> - self.ud[url] = FetchData(url, d) >> + self.ud[url] = FetchData(url, self.d) >> ud = self.ud[url] >> ud.setup_localpath(self.d) >> >> diff --git a/bitbake/lib/bb/fetch2/bzr.py >> b/bitbake/lib/bb/fetch2/bzr.py index 658502f..c56d875 100644 >> --- a/bitbake/lib/bb/fetch2/bzr.py >> +++ b/bitbake/lib/bb/fetch2/bzr.py >> @@ -10,18 +10,8 @@ BitBake 'Fetch' implementation for bzr. >> # BitBake build tools. >> # Copyright (C) 2003, 2004 Chris Larson >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> +# SPDX-License-Identifier: GPL-2.0-only >> # >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. >> import os >> import sys >> diff --git a/bitbake/lib/bb/fetch2/clearcase.py >> b/bitbake/lib/bb/fetch2/clearcase.py index 3a6573d..3dd93ad 100644 >> --- a/bitbake/lib/bb/fetch2/clearcase.py >> +++ b/bitbake/lib/bb/fetch2/clearcase.py >> @@ -1,5 +1,3 @@ >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> """ >> BitBake 'Fetch' clearcase implementation >> >> @@ -47,18 +45,7 @@ User credentials: >> """ >> # Copyright (C) 2014 Siemens AG >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> -# >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only >> # >> >> import os >> @@ -67,6 +54,8 @@ import shutil >> import bb >> from bb.fetch2 import FetchMethod >> from bb.fetch2 import FetchError >> +from bb.fetch2 import MissingParameterError >> +from bb.fetch2 import ParameterError >> from bb.fetch2 import runfetchcmd >> from bb.fetch2 import logger >> >> @@ -92,7 +81,7 @@ class ClearCase(FetchMethod): >> if 'protocol' in ud.parm: >> ud.proto = ud.parm['protocol'] >> if not ud.proto in ('http', 'https'): >> - raise fetch2.ParameterError("Invalid protocol type", >> ud.url) >> + raise ParameterError("Invalid protocol type", ud.url) >> >> ud.vob = '' >> if 'vob' in ud.parm: >> diff --git a/bitbake/lib/bb/fetch2/cvs.py >> b/bitbake/lib/bb/fetch2/cvs.py index 0e0a319..1b35ba4 100644 >> --- a/bitbake/lib/bb/fetch2/cvs.py >> +++ b/bitbake/lib/bb/fetch2/cvs.py >> @@ -1,5 +1,3 @@ >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> """ >> BitBake 'Fetch' implementations >> >> @@ -10,20 +8,9 @@ BitBake build tools. >> >> # Copyright (C) 2003, 2004 Chris Larson >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> +# SPDX-License-Identifier: GPL-2.0-only >> # >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. -# >> -#Based on functions from the base bb module, Copyright 2003 Holger >> Schurig +# Based on functions from the base bb module, Copyright 2003 >> Holger Schurig # >> >> import os >> diff --git a/bitbake/lib/bb/fetch2/git.py >> b/bitbake/lib/bb/fetch2/git.py index 59a2ee8..2d1d2ca 100644 >> --- a/bitbake/lib/bb/fetch2/git.py >> +++ b/bitbake/lib/bb/fetch2/git.py >> @@ -1,5 +1,3 @@ >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> """ >> BitBake 'Fetch' git implementation >> >> @@ -55,20 +53,10 @@ Supported SRC_URI options are: >> >> """ >> >> -#Copyright (C) 2005 Richard Purdie >> +# Copyright (C) 2005 Richard Purdie >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> +# SPDX-License-Identifier: GPL-2.0-only >> # >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. >> import collections >> import errno >> @@ -199,7 +187,7 @@ class Git(FetchMethod): >> depth_default = 1 >> ud.shallow_depths = collections.defaultdict(lambda: >> depth_default) >> - revs_default = d.getVar("BB_GIT_SHALLOW_REVS", True) >> + revs_default = d.getVar("BB_GIT_SHALLOW_REVS") >> ud.shallow_revs = [] >> ud.branches = {} >> for pos, name in enumerate(ud.names): >> @@ -318,7 +306,7 @@ class Git(FetchMethod): >> def try_premirror(self, ud, d): >> # If we don't do this, updating an existing checkout with >> only premirrors # is not possible >> - if d.getVar("BB_FETCH_PREMIRRORONLY") is not None: >> + if bb.utils.to_boolean(d.getVar("BB_FETCH_PREMIRRORONLY")): >> return True >> if os.path.exists(ud.clonedir): >> return False >> @@ -476,6 +464,8 @@ class Git(FetchMethod): >> if os.path.exists(destdir): >> bb.utils.prunedir(destdir) >> >> + need_lfs = ud.parm.get("lfs", "1") == "1" >> + >> source_found = False >> source_error = [] >> >> @@ -503,6 +493,13 @@ class Git(FetchMethod): >> >> repourl = self._get_repo_url(ud) >> runfetchcmd("%s remote set-url origin %s" % (ud.basecmd, >> repourl), d, workdir=destdir) + >> + if self._contains_lfs(ud, d, destdir): >> + if need_lfs and not self._find_git_lfs(d): >> + raise bb.fetch2.FetchError("Repository %s has LFS >> content, install git-lfs on host to download (or set lfs=0 to ignore >> it)" % (repourl)) >> + else: >> + bb.note("Repository %s has LFS content but it is not >> being fetched" % (repourl)) + >> if not ud.nocheckout: >> if subdir != "": >> runfetchcmd("%s read-tree %s%s" % (ud.basecmd, >> ud.revisions[ud.names[0]], readpathspec), d, @@ -522,9 +519,17 @@ >> class Git(FetchMethod): def clean(self, ud, d): >> """ clean the git directory """ >> >> - bb.utils.remove(ud.localpath, True) >> - bb.utils.remove(ud.fullmirror) >> - bb.utils.remove(ud.fullmirror + ".done") >> + to_remove = [ud.localpath, ud.fullmirror, ud.fullmirror + >> ".done"] >> + # The localpath is a symlink to clonedir when it is cloned >> from a >> + # mirror, so remove both of them. >> + if os.path.islink(ud.localpath): >> + clonedir = os.path.realpath(ud.localpath) >> + to_remove.append(clonedir) >> + >> + for r in to_remove: >> + if os.path.exists(r): >> + bb.note('Removing %s' % r) >> + bb.utils.remove(r, True) >> >> def supports_srcrev(self): >> return True >> @@ -545,6 +550,27 @@ class Git(FetchMethod): >> raise bb.fetch2.FetchError("The command '%s' gave output >> with more then 1 line unexpectedly, output: '%s'" % (cmd, output)) >> return output.split()[0] != "0" >> + def _contains_lfs(self, ud, d, wd): >> + """ >> + Check if the repository has 'lfs' (large file) content >> + """ >> + cmd = "%s grep lfs HEAD:.gitattributes | wc -l" % ( >> + ud.basecmd) >> + try: >> + output = runfetchcmd(cmd, d, quiet=True, workdir=wd) >> + if int(output) > 0: >> + return True >> + except (bb.fetch2.FetchError,ValueError): >> + pass >> + return False >> + >> + def _find_git_lfs(self, d): >> + """ >> + Return True if git-lfs can be found, False otherwise. >> + """ >> + import shutil >> + return shutil.which("git-lfs", path=d.getVar('PATH')) is not >> None + >> def _get_repo_url(self, ud): >> """ >> Return the repository URL >> @@ -615,7 +641,7 @@ class Git(FetchMethod): >> """ >> pupver = ('', '') >> >> - tagregex = re.compile(d.getVar('UPSTREAM_CHECK_GITTAGREGEX') >> or "(?P([0-9][\.|_]?)+)") >> + tagregex = re.compile(d.getVar('UPSTREAM_CHECK_GITTAGREGEX') >> or r"(?P([0-9][\.|_]?)+)") try: >> output = self._lsremote(ud, d, "refs/tags/*") >> except (bb.fetch2.FetchError, bb.fetch2.NetworkAccess) as e: >> @@ -630,7 +656,7 @@ class Git(FetchMethod): >> >> tag_head = line.split("/")[-1] >> # Ignore non-released branches >> - m = re.search("(alpha|beta|rc|final)+", tag_head) >> + m = re.search(r"(alpha|beta|rc|final)+", tag_head) >> if m: >> continue >> >> diff --git a/bitbake/lib/bb/fetch2/gitannex.py >> b/bitbake/lib/bb/fetch2/gitannex.py index a9b69ca..1d497dc 100644 >> --- a/bitbake/lib/bb/fetch2/gitannex.py >> +++ b/bitbake/lib/bb/fetch2/gitannex.py >> @@ -1,5 +1,3 @@ >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> """ >> BitBake 'Fetch' git annex implementation >> """ >> @@ -7,18 +5,8 @@ BitBake 'Fetch' git annex implementation >> # Copyright (C) 2014 Otavio Salvador >> # Copyright (C) 2014 O.S. Systems Software LTDA. >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> +# SPDX-License-Identifier: GPL-2.0-only >> # >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. >> import os >> import bb >> diff --git a/bitbake/lib/bb/fetch2/gitsm.py >> b/bitbake/lib/bb/fetch2/gitsm.py index 35729db..c622771 100644 >> --- a/bitbake/lib/bb/fetch2/gitsm.py >> +++ b/bitbake/lib/bb/fetch2/gitsm.py >> @@ -1,5 +1,3 @@ >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> """ >> BitBake 'Fetch' git submodules implementation >> >> @@ -16,18 +14,8 @@ NOTE: Switching a SRC_URI from "git://" to >> "gitsm://" requires a clean of your r >> # Copyright (C) 2013 Richard Purdie >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> +# SPDX-License-Identifier: GPL-2.0-only >> # >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. >> import os >> import bb >> @@ -45,60 +33,97 @@ class GitSM(Git): >> """ >> return ud.type in ['gitsm'] >> >> - @staticmethod >> - def parse_gitmodules(gitmodules): >> - modules = {} >> - module = "" >> - for line in gitmodules.splitlines(): >> - if line.startswith('[submodule'): >> - module = line.split('"')[1] >> - modules[module] = {} >> - elif module and line.strip().startswith('path'): >> - path = line.split('=')[1].strip() >> - modules[module]['path'] = path >> - elif module and line.strip().startswith('url'): >> - url = line.split('=')[1].strip() >> - modules[module]['url'] = url >> - return modules >> - >> - def update_submodules(self, ud, d): >> + def process_submodules(self, ud, workdir, function, d): >> + """ >> + Iterate over all of the submodules in this repository and >> execute >> + the 'function' for each of them. >> + """ >> + >> submodules = [] >> paths = {} >> + revision = {} >> uris = {} >> - local_paths = {} >> - >> + subrevision = {} >> + >> + def parse_gitmodules(gitmodules): >> + modules = {} >> + module = "" >> + for line in gitmodules.splitlines(): >> + if line.startswith('[submodule'): >> + module = line.split('"')[1] >> + modules[module] = {} >> + elif module and line.strip().startswith('path'): >> + path = line.split('=')[1].strip() >> + modules[module]['path'] = path >> + elif module and line.strip().startswith('url'): >> + url = line.split('=')[1].strip() >> + modules[module]['url'] = url >> + return modules >> + >> + # Collect the defined submodules, and their attributes >> for name in ud.names: >> try: >> - gitmodules = runfetchcmd("%s show %s:.gitmodules" % >> (ud.basecmd, ud.revisions[name]), d, quiet=True, workdir=ud.clonedir) >> + gitmodules = runfetchcmd("%s show %s:.gitmodules" % >> (ud.basecmd, ud.revisions[name]), d, quiet=True, workdir=workdir) >> except: # No submodules to update >> continue >> >> - for m, md in self.parse_gitmodules(gitmodules).items(): >> + for m, md in parse_gitmodules(gitmodules).items(): >> + try: >> + module_hash = runfetchcmd("%s ls-tree -z -d %s >> %s" % (ud.basecmd, ud.revisions[name], md['path']), d, quiet=True, >> workdir=workdir) >> + except: >> + # If the command fails, we don't have a valid >> file to check. If it doesn't >> + # fail -- it still might be a failure, see next >> check... >> + module_hash = "" >> + >> + if not module_hash: >> + logger.debug(1, "submodule %s is defined, but is >> not initialized in the repository. Skipping", m) >> + continue >> + >> submodules.append(m) >> paths[m] = md['path'] >> + revision[m] = ud.revisions[name] >> uris[m] = md['url'] >> + subrevision[m] = module_hash.split()[2] >> + >> + # Convert relative to absolute uri based on parent >> uri if uris[m].startswith('..'): >> newud = copy.copy(ud) >> - newud.path = >> os.path.realpath(os.path.join(newud.path, md['url'])) >> + newud.path = >> os.path.realpath(os.path.join(newud.path, uris[m])) uris[m] = >> Git._get_repo_url(self, newud) >> for module in submodules: >> - module_hash = runfetchcmd("%s ls-tree -z -d %s %s" % >> (ud.basecmd, ud.revisions[name], paths[module]), d, quiet=True, >> workdir=ud.clonedir) >> - module_hash = module_hash.split()[2] >> + # Translate the module url into a SRC_URI >> + >> + if "://" in uris[module]: >> + # Properly formated URL already >> + proto = uris[module].split(':', 1)[0] >> + url = uris[module].replace('%s:' % proto, 'gitsm:', >> 1) >> + else: >> + if ":" in uris[module]: >> + # Most likely an SSH style reference >> + proto = "ssh" >> + if ":/" in uris[module]: >> + # Absolute reference, easy to convert.. >> + url = "gitsm://" + >> uris[module].replace(':/', '/', 1) >> + else: >> + # Relative reference, no way to know if this >> is right! >> + logger.warning("Submodule included by %s >> refers to relative ssh reference %s. References may fail if not >> absolute." % (ud.url, uris[module])) >> + url = "gitsm://" + uris[module].replace(':', >> '/', 1) >> + else: >> + # This has to be a file reference >> + proto = "file" >> + url = "gitsm://" + uris[module] >> >> - # Build new SRC_URI >> - proto = uris[module].split(':', 1)[0] >> - url = uris[module].replace('%s:' % proto, 'gitsm:', 1) >> url += ';protocol=%s' % proto >> url += ";name=%s" % module >> - url += ";bareclone=1;nocheckout=1;nobranch=1" >> + url += ";subpath=%s" % module >> >> ld = d.createCopy() >> # Not necessary to set SRC_URI, since we're passing the >> URI to # Fetch. >> #ld.setVar('SRC_URI', url) >> - ld.setVar('SRCREV_%s' % module, module_hash) >> + ld.setVar('SRCREV_%s' % module, subrevision[module]) >> >> # Workaround for issues with SRCPV/SRCREV_FORMAT errors >> # error refer to 'multiple' repositories. Only the >> repository @@ -106,145 +131,85 @@ class GitSM(Git): >> ld.setVar('SRCPV', d.getVar('SRCPV')) >> ld.setVar('SRCREV_FORMAT', module) >> >> - newfetch = Fetch([url], ld, cache=False) >> - newfetch.download() >> - local_paths[module] = newfetch.localpath(url) >> + function(ud, url, module, paths[module], ld) >> >> - # Correct the submodule references to the local download >> version... >> - runfetchcmd("%(basecmd)s config submodule.%(module)s.url >> %(url)s" % {'basecmd': ud.basecmd, 'module': module, 'url' : >> local_paths[module]}, d, workdir=ud.clonedir) - >> - symlink_path = os.path.join(ud.clonedir, 'modules', >> paths[module]) >> - if not os.path.exists(symlink_path): >> - try: >> - os.makedirs(os.path.dirname(symlink_path), >> exist_ok=True) >> - except OSError: >> - pass >> - os.symlink(local_paths[module], symlink_path) >> - >> - return True >> + return submodules != [] >> >> def need_update(self, ud, d): >> - main_repo_needs_update = Git.need_update(self, ud, d) >> - >> - # First check that the main repository has enough history >> fetched. If it doesn't, then we don't >> - # even have the .gitmodules and gitlinks for the submodules >> to attempt asking whether the >> - # submodules' histories are recent enough. >> - if main_repo_needs_update: >> + if Git.need_update(self, ud, d): >> return True >> >> - # Now check that the submodule histories are new enough. The >> git-submodule command doesn't have >> - # any clean interface for doing this aside from just >> attempting the checkout (with network >> - # fetched disabled). >> - return not self.update_submodules(ud, d) >> + try: >> + # Check for the nugget dropped by the download operation >> + known_srcrevs = runfetchcmd("%s config --get-all >> bitbake.srcrev" % \ >> + (ud.basecmd), d, workdir=ud.clonedir) >> >> - def download(self, ud, d): >> - Git.download(self, ud, d) >> + if ud.revisions[ud.names[0]] not in >> known_srcrevs.split(): >> + return True >> + except bb.fetch2.FetchError: >> + # No srcrev nuggets, so this is new and needs to be >> updated >> + return True >> >> - if not ud.shallow or ud.localpath != ud.fullshallow: >> - self.update_submodules(ud, d) >> + return False >> >> - def copy_submodules(self, submodules, ud, destdir, d): >> - if ud.bareclone: >> - repo_conf = destdir >> - else: >> - repo_conf = os.path.join(destdir, '.git') >> + def download(self, ud, d): >> + def download_submodule(ud, url, module, modpath, d): >> + url += ";bareclone=1;nobranch=1" >> >> - if submodules and not os.path.exists(os.path.join(repo_conf, >> 'modules')): >> - os.mkdir(os.path.join(repo_conf, 'modules')) >> + # Is the following still needed? >> + #url += ";nocheckout=1" >> >> - for module, md in submodules.items(): >> - srcpath = os.path.join(ud.clonedir, 'modules', >> md['path']) >> - modpath = os.path.join(repo_conf, 'modules', md['path']) >> + try: >> + newfetch = Fetch([url], d, cache=False) >> + newfetch.download() >> + # Drop a nugget to add each of the srcrevs we've >> fetched (used by need_update) >> + runfetchcmd("%s config --add bitbake.srcrev %s" % \ >> + (ud.basecmd, ud.revisions[ud.names[0]]), >> d, workdir=ud.clonedir) >> + except Exception as e: >> + logger.error('gitsm: submodule download failed: %s >> %s' % (type(e).__name__, str(e))) >> + raise >> >> - if os.path.exists(srcpath): >> - if os.path.exists(os.path.join(srcpath, '.git')): >> - srcpath = os.path.join(srcpath, '.git') >> + Git.download(self, ud, d) >> + self.process_submodules(ud, ud.clonedir, download_submodule, >> d) >> - target = modpath >> - if os.path.exists(modpath): >> - target = os.path.dirname(modpath) >> + def unpack(self, ud, destdir, d): >> + def unpack_submodules(ud, url, module, modpath, d): >> + url += ";bareclone=1;nobranch=1" >> >> - os.makedirs(os.path.dirname(target), exist_ok=True) >> - runfetchcmd("cp -fpLR %s %s" % (srcpath, target), d) >> - elif os.path.exists(modpath): >> - # Module already exists, likely unpacked from a >> shallow mirror clone >> - pass >> + # Figure out where we clone over the bare submodules... >> + if ud.bareclone: >> + repo_conf = ud.destdir >> else: >> - # This is fatal, as we do NOT want git-submodule to >> hit the network >> - raise bb.fetch2.FetchError('Submodule %s does not >> exist in %s or %s.' % (module, srcpath, modpath)) - >> - def clone_shallow_local(self, ud, dest, d): >> - super(GitSM, self).clone_shallow_local(ud, dest, d) >> + repo_conf = os.path.join(ud.destdir, '.git') >> >> - # Copy over the submodules' fetched histories too. >> - repo_conf = os.path.join(dest, '.git') >> - >> - submodules = [] >> - for name in ud.names: >> try: >> - gitmodules = runfetchcmd("%s show %s:.gitmodules" % >> (ud.basecmd, ud.revision), d, quiet=True, workdir=dest) >> - except: >> - # No submodules to update >> - continue >> + newfetch = Fetch([url], d, cache=False) >> + >> newfetch.unpack(root=os.path.dirname(os.path.join(repo_conf, >> 'modules', module))) >> + except Exception as e: >> + logger.error('gitsm: submodule unpack failed: %s %s' >> % (type(e).__name__, str(e))) >> + raise >> >> - submodules = self.parse_gitmodules(gitmodules) >> - self.copy_submodules(submodules, ud, dest, d) >> + local_path = newfetch.localpath(url) >> >> - def unpack(self, ud, destdir, d): >> - Git.unpack(self, ud, destdir, d) >> + # Correct the submodule references to the local download >> version... >> + runfetchcmd("%(basecmd)s config submodule.%(module)s.url >> %(url)s" % {'basecmd': ud.basecmd, 'module': module, 'url' : >> local_path}, d, workdir=ud.destdir) >> - # Copy over the submodules' fetched histories too. >> - if ud.bareclone: >> - repo_conf = ud.destdir >> - else: >> - repo_conf = os.path.join(ud.destdir, '.git') >> + if ud.shallow: >> + runfetchcmd("%(basecmd)s config >> submodule.%(module)s.shallow true" % {'basecmd': ud.basecmd, >> 'module': module}, d, workdir=ud.destdir) >> - update_submodules = False >> - paths = {} >> - uris = {} >> - local_paths = {} >> - for name in ud.names: >> + # Ensure the submodule repository is NOT set to bare, >> since we're checking it out... try: >> - gitmodules = runfetchcmd("%s show HEAD:.gitmodules" >> % (ud.basecmd), d, quiet=True, workdir=ud.destdir) >> + runfetchcmd("%s config core.bare false" % >> (ud.basecmd), d, quiet=True, workdir=os.path.join(repo_conf, >> 'modules', module)) except: >> - # No submodules to update >> - continue >> - >> - submodules = self.parse_gitmodules(gitmodules) >> - self.copy_submodules(submodules, ud, ud.destdir, d) >> - >> - submodules_queue = [(module, os.path.join(repo_conf, >> 'modules', md['path'])) for module, md in submodules.items()] >> - while len(submodules_queue) != 0: >> - module, modpath = submodules_queue.pop() >> - >> - # add submodule children recursively >> - try: >> - gitmodules = runfetchcmd("%s show >> HEAD:.gitmodules" % (ud.basecmd), d, quiet=True, workdir=modpath) >> - for m, md in >> self.parse_gitmodules(gitmodules).items(): >> - submodules_queue.append([m, >> os.path.join(modpath, 'modules', md['path'])]) >> - except: >> - # no children >> - pass >> - >> + logger.error("Unable to set git config core.bare to >> false for %s" % os.path.join(repo_conf, 'modules', module)) >> + raise >> >> - # There are submodules to update >> - update_submodules = True >> - >> - # Determine (from the submodule) the correct url to >> reference >> - try: >> - output = runfetchcmd("%(basecmd)s config >> remote.origin.url" % {'basecmd': ud.basecmd}, d, workdir=modpath) >> - except bb.fetch2.FetchError as e: >> - # No remote url defined in this submodule >> - continue >> - >> - local_paths[module] = output >> - >> - # Setup the local URL properly (like git submodule >> init or sync would do...) >> - runfetchcmd("%(basecmd)s config >> submodule.%(module)s.url %(url)s" % {'basecmd': ud.basecmd, 'module': >> module, 'url' : local_paths[module]}, d, workdir=ud.destdir) >> + Git.unpack(self, ud, destdir, d) >> >> - # Ensure the submodule repository is NOT set to >> bare, since we're checking it out... >> - runfetchcmd("%s config core.bare false" % >> (ud.basecmd), d, quiet=True, workdir=modpath) >> + ret = self.process_submodules(ud, ud.destdir, >> unpack_submodules, d) >> - if update_submodules: >> - # Run submodule update, this sets up the directories -- >> without touching the config >> + if not ud.bareclone and ret: >> + # All submodules should already be downloaded and >> configured in the tree. This simply sets >> + # up the configuration and checks out the files. The >> main project config should remain >> + # unmodified, and no download from the internet should >> occur. runfetchcmd("%s submodule update --recursive --no-fetch" % >> (ud.basecmd), d, quiet=True, workdir=ud.destdir) diff --git >> a/bitbake/lib/bb/fetch2/hg.py b/bitbake/lib/bb/fetch2/hg.py index >> 936d043..15d729e 100644 --- a/bitbake/lib/bb/fetch2/hg.py >> +++ b/bitbake/lib/bb/fetch2/hg.py >> @@ -1,5 +1,3 @@ >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> """ >> BitBake 'Fetch' implementation for mercurial DRCS (hg). >> >> @@ -9,20 +7,10 @@ BitBake 'Fetch' implementation for mercurial DRCS >> (hg). # Copyright (C) 2004 Marcin Juszkiewicz >> # Copyright (C) 2007 Robert Schuster >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> -# >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only >> # >> # Based on functions from the base bb module, Copyright 2003 Holger >> Schurig +# >> >> import os >> import sys >> @@ -99,7 +87,7 @@ class Hg(FetchMethod): >> def try_premirror(self, ud, d): >> # If we don't do this, updating an existing checkout with >> only premirrors # is not possible >> - if d.getVar("BB_FETCH_PREMIRRORONLY") is not None: >> + if bb.utils.to_boolean(d.getVar("BB_FETCH_PREMIRRORONLY")): >> return True >> if os.path.exists(ud.moddir): >> return False >> diff --git a/bitbake/lib/bb/fetch2/local.py >> b/bitbake/lib/bb/fetch2/local.py index a114ac1..01d9ff9 100644 >> --- a/bitbake/lib/bb/fetch2/local.py >> +++ b/bitbake/lib/bb/fetch2/local.py >> @@ -1,5 +1,3 @@ >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> """ >> BitBake 'Fetch' implementations >> >> @@ -10,20 +8,10 @@ BitBake build tools. >> >> # Copyright (C) 2003, 2004 Chris Larson >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> -# >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only >> # >> # Based on functions from the base bb module, Copyright 2003 Holger >> Schurig +# >> >> import os >> import urllib.request, urllib.parse, urllib.error >> diff --git a/bitbake/lib/bb/fetch2/npm.py >> b/bitbake/lib/bb/fetch2/npm.py index 65bf5a3..9700e61 100644 >> --- a/bitbake/lib/bb/fetch2/npm.py >> +++ b/bitbake/lib/bb/fetch2/npm.py >> @@ -1,5 +1,6 @@ >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> +# >> +# SPDX-License-Identifier: GPL-2.0-only >> +# >> """ >> BitBake 'Fetch' NPM implementation >> >> @@ -100,11 +101,19 @@ class Npm(FetchMethod): >> return False >> return True >> >> - def _runwget(self, ud, d, command, quiet): >> - logger.debug(2, "Fetching %s using command '%s'" % (ud.url, >> command)) >> - bb.fetch2.check_network_access(d, command, ud.url) >> + def _runpack(self, ud, d, pkgfullname: str, quiet=False) -> str: >> + """ >> + Runs npm pack on a full package name. >> + Returns the filename of the downloaded package >> + """ >> + bb.fetch2.check_network_access(d, pkgfullname, ud.registry) >> dldir = d.getVar("DL_DIR") >> - runfetchcmd(command, d, quiet, workdir=dldir) >> + dldir = os.path.join(dldir, ud.prefixdir) >> + >> + command = "npm pack {} --registry {}".format(pkgfullname, >> ud.registry) >> + logger.debug(2, "Fetching {} using command '{}' in >> {}".format(pkgfullname, command, dldir)) >> + filename = runfetchcmd(command, d, quiet, workdir=dldir) >> + return filename.rstrip() >> >> def _unpackdep(self, ud, pkg, data, destdir, dldir, d): >> file = data[pkg]['tgz'] >> @@ -150,20 +159,11 @@ class Npm(FetchMethod): >> Parse the output of npm view --json; the last JSON result >> is assumed to be the one that we're interested in. >> ''' >> - pdata = None >> - outdeps = {} >> - datalines = [] >> - bracelevel = 0 >> - for line in output.splitlines(): >> - if bracelevel: >> - datalines.append(line) >> - elif '{' in line: >> - datalines = [] >> - datalines.append(line) >> - bracelevel = bracelevel + line.count('{') - >> line.count('}') >> - if datalines: >> - pdata = json.loads('\n'.join(datalines)) >> - return pdata >> + pdata = json.loads(output); >> + try: >> + return pdata[-1] >> + except: >> + return pdata >> >> def _getdependencies(self, pkg, data, version, d, ud, >> optional=False, fetchedlist=None): if fetchedlist is None: >> @@ -171,6 +171,9 @@ class Npm(FetchMethod): >> pkgfullname = pkg >> if version != '*' and not '/' in version: >> pkgfullname += "@'%s'" % version >> + if pkgfullname in fetchedlist: >> + return >> + >> logger.debug(2, "Calling getdeps on %s" % pkg) >> fetchcmd = "npm view %s --json --registry %s" % >> (pkgfullname, ud.registry) output = runfetchcmd(fetchcmd, d, True) >> @@ -190,15 +193,10 @@ class Npm(FetchMethod): >> if (not blacklist and 'linux' not in pkg_os) or >> '!linux' in pkg_os: logger.debug(2, "Skipping %s since it's >> incompatible with Linux" % pkg) return >> - #logger.debug(2, "Output URL is %s - %s - %s" % >> (ud.basepath, ud.basename, ud.localfile)) >> - outputurl = pdata['dist']['tarball'] >> + filename = self._runpack(ud, d, pkgfullname) >> data[pkg] = {} >> - data[pkg]['tgz'] = os.path.basename(outputurl) >> - if outputurl in fetchedlist: >> - return >> - >> - self._runwget(ud, d, "%s --directory-prefix=%s %s" % >> (self.basecmd, ud.prefixdir, outputurl), False) >> - fetchedlist.append(outputurl) >> + data[pkg]['tgz'] = filename >> + fetchedlist.append(pkgfullname) >> >> dependencies = pdata.get('dependencies', {}) >> optionalDependencies = pdata.get('optionalDependencies', {}) >> @@ -225,17 +223,12 @@ class Npm(FetchMethod): >> if obj == pkg: >> self._getshrinkeddependencies(obj, >> data['dependencies'][obj], data['dependencies'][obj]['version'], d, >> ud, lockdown, manifest, False) return >> - outputurl = "invalid" >> - if ('resolved' not in data) or (not >> data['resolved'].startswith('http://') and not >> data['resolved'].startswith('https://')): >> - # will be the case for ${PN} >> - fetchcmd = "npm view %s@%s dist.tarball --registry %s" % >> (pkg, version, ud.registry) >> - logger.debug(2, "Found this matching URL: %s" % >> str(fetchcmd)) >> - outputurl = runfetchcmd(fetchcmd, d, True) >> - else: >> - outputurl = data['resolved'] >> - self._runwget(ud, d, "%s --directory-prefix=%s %s" % >> (self.basecmd, ud.prefixdir, outputurl), False) + >> + pkgnameWithVersion = "{}@{}".format(pkg, version) >> + logger.debug(2, "Get dependencies for >> {}".format(pkgnameWithVersion)) >> + filename = self._runpack(ud, d, pkgnameWithVersion) >> manifest[pkg] = {} >> - manifest[pkg]['tgz'] = os.path.basename(outputurl).rstrip() >> + manifest[pkg]['tgz'] = filename >> manifest[pkg]['deps'] = {} >> >> if pkg in lockdown: >> diff --git a/bitbake/lib/bb/fetch2/osc.py >> b/bitbake/lib/bb/fetch2/osc.py index 6c60456..3e56715 100644 >> --- a/bitbake/lib/bb/fetch2/osc.py >> +++ b/bitbake/lib/bb/fetch2/osc.py >> @@ -1,5 +1,6 @@ >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> +# >> +# SPDX-License-Identifier: GPL-2.0-only >> +# >> """ >> Bitbake "Fetch" implementation for osc (Opensuse build service >> client). Based on the svn "Fetch" implementation. >> diff --git a/bitbake/lib/bb/fetch2/perforce.py >> b/bitbake/lib/bb/fetch2/perforce.py index 903a8e6..54d001e 100644 >> --- a/bitbake/lib/bb/fetch2/perforce.py >> +++ b/bitbake/lib/bb/fetch2/perforce.py >> @@ -1,5 +1,3 @@ >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> """ >> BitBake 'Fetch' implementation for perforce >> >> @@ -8,18 +6,7 @@ BitBake 'Fetch' implementation for perforce >> # Copyright (C) 2003, 2004 Chris Larson >> # Copyright (C) 2016 Kodak Alaris, Inc. >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> -# >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only >> # >> # Based on functions from the base bb module, Copyright 2003 Holger >> Schurig >> diff --git a/bitbake/lib/bb/fetch2/repo.py >> b/bitbake/lib/bb/fetch2/repo.py index 8c7e818..2bdbbd4 100644 >> --- a/bitbake/lib/bb/fetch2/repo.py >> +++ b/bitbake/lib/bb/fetch2/repo.py >> @@ -1,5 +1,3 @@ >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> """ >> BitBake "Fetch" repo (git) implementation >> >> @@ -8,20 +6,10 @@ BitBake "Fetch" repo (git) implementation >> # Copyright (C) 2009 Tom Rini >> # >> # Based on git.py which is: >> -#Copyright (C) 2005 Richard Purdie >> +# Copyright (C) 2005 Richard Purdie >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> +# SPDX-License-Identifier: GPL-2.0-only >> # >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. >> import os >> import bb >> diff --git a/bitbake/lib/bb/fetch2/s3.py b/bitbake/lib/bb/fetch2/s3.py >> index 1629288..ffca73c 100644 >> --- a/bitbake/lib/bb/fetch2/s3.py >> +++ b/bitbake/lib/bb/fetch2/s3.py >> @@ -1,5 +1,3 @@ >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> """ >> BitBake 'Fetch' implementation for Amazon AWS S3. >> >> @@ -13,18 +11,7 @@ The aws tool must be correctly installed and >> configured prior to use. # Based in part on bb.fetch2.wget: >> # Copyright (C) 2003, 2004 Chris Larson >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> -# >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only >> # >> # Based on functions from the base bb module, Copyright 2003 Holger >> Schurig >> diff --git a/bitbake/lib/bb/fetch2/sftp.py >> b/bitbake/lib/bb/fetch2/sftp.py index 81884a6..f87f292 100644 >> --- a/bitbake/lib/bb/fetch2/sftp.py >> +++ b/bitbake/lib/bb/fetch2/sftp.py >> @@ -1,5 +1,3 @@ >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> """ >> BitBake SFTP Fetch implementation >> >> @@ -44,18 +42,7 @@ SRC_URI = >> "sftp://user@host.example.com/dir/path.file.txt" # Based in part on >> bb.fetch2.wget: # Copyright (C) 2003, 2004 Chris Larson >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> -# >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only >> # >> # Based on functions from the base bb module, Copyright 2003 Holger >> Schurig >> diff --git a/bitbake/lib/bb/fetch2/ssh.py >> b/bitbake/lib/bb/fetch2/ssh.py index 6047ee4..f5be060 100644 >> --- a/bitbake/lib/bb/fetch2/ssh.py >> +++ b/bitbake/lib/bb/fetch2/ssh.py >> @@ -1,5 +1,3 @@ >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> ''' >> BitBake 'Fetch' implementations >> >> @@ -29,18 +27,8 @@ IETF secsh internet draft: >> # Copyright 2003 Holger Schurig >> # >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> +# SPDX-License-Identifier: GPL-2.0-only >> # >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. >> import re, os >> from bb.fetch2 import FetchMethod >> diff --git a/bitbake/lib/bb/fetch2/svn.py >> b/bitbake/lib/bb/fetch2/svn.py index ed70bcf..96d666b 100644 >> --- a/bitbake/lib/bb/fetch2/svn.py >> +++ b/bitbake/lib/bb/fetch2/svn.py >> @@ -1,5 +1,3 @@ >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> """ >> BitBake 'Fetch' implementation for svn. >> >> @@ -8,18 +6,7 @@ BitBake 'Fetch' implementation for svn. >> # Copyright (C) 2003, 2004 Chris Larson >> # Copyright (C) 2004 Marcin Juszkiewicz >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> -# >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only >> # >> # Based on functions from the base bb module, Copyright 2003 Holger >> Schurig >> @@ -63,6 +50,9 @@ class Svn(FetchMethod): >> relpath = self._strip_leading_slashes(ud.path) >> ud.pkgdir = os.path.join(svndir, ud.host, relpath) >> ud.moddir = os.path.join(ud.pkgdir, ud.module) >> + # Protects the repository from concurrent updates, e.g. from >> two >> + # recipes fetching different revisions at the same time >> + ud.svnlock = os.path.join(ud.pkgdir, "svn.lock") >> >> ud.setup_revisions(d) >> >> @@ -101,6 +91,13 @@ class Svn(FetchMethod): >> svncmd = "%s log --limit 1 %s %s://%s/%s/" % >> (ud.basecmd, " ".join(options), proto, svnroot, ud.module) else: >> suffix = "" >> + >> + # externals may be either 'allowed' or 'nowarn', but not >> both. Allowed >> + # will not issue a warning, but will log to the debug >> buffer what has likely >> + # been downloaded by SVN. >> + if not ("externals" in ud.parm and ud.parm["externals"] >> == "allowed"): >> + options.append("--ignore-externals") >> + >> if ud.revision: >> options.append("-r %s" % ud.revision) >> suffix = "@%s" % (ud.revision) >> @@ -123,35 +120,52 @@ class Svn(FetchMethod): >> >> logger.debug(2, "Fetch: checking for module directory '" + >> ud.moddir + "'") >> - if os.access(os.path.join(ud.moddir, '.svn'), os.R_OK): >> - svnupdatecmd = self._buildsvncommand(ud, d, "update") >> - logger.info("Update " + ud.url) >> - # We need to attempt to run svn upgrade first in case >> its an older working format >> - try: >> - runfetchcmd(ud.basecmd + " upgrade", d, >> workdir=ud.moddir) >> - except FetchError: >> - pass >> - logger.debug(1, "Running %s", svnupdatecmd) >> - bb.fetch2.check_network_access(d, svnupdatecmd, ud.url) >> - runfetchcmd(svnupdatecmd, d, workdir=ud.moddir) >> - else: >> - svnfetchcmd = self._buildsvncommand(ud, d, "fetch") >> - logger.info("Fetch " + ud.url) >> - # check out sources there >> - bb.utils.mkdirhier(ud.pkgdir) >> - logger.debug(1, "Running %s", svnfetchcmd) >> - bb.fetch2.check_network_access(d, svnfetchcmd, ud.url) >> - runfetchcmd(svnfetchcmd, d, workdir=ud.pkgdir) >> - >> - scmdata = ud.parm.get("scmdata", "") >> - if scmdata == "keep": >> - tar_flags = "" >> - else: >> - tar_flags = "--exclude='.svn'" >> + lf = bb.utils.lockfile(ud.svnlock) >> + >> + try: >> + if os.access(os.path.join(ud.moddir, '.svn'), os.R_OK): >> + svnupdatecmd = self._buildsvncommand(ud, d, "update") >> + logger.info("Update " + ud.url) >> + # We need to attempt to run svn upgrade first in >> case its an older working format >> + try: >> + runfetchcmd(ud.basecmd + " upgrade", d, >> workdir=ud.moddir) >> + except FetchError: >> + pass >> + logger.debug(1, "Running %s", svnupdatecmd) >> + bb.fetch2.check_network_access(d, svnupdatecmd, >> ud.url) >> + runfetchcmd(svnupdatecmd, d, workdir=ud.moddir) >> + else: >> + svnfetchcmd = self._buildsvncommand(ud, d, "fetch") >> + logger.info("Fetch " + ud.url) >> + # check out sources there >> + bb.utils.mkdirhier(ud.pkgdir) >> + logger.debug(1, "Running %s", svnfetchcmd) >> + bb.fetch2.check_network_access(d, svnfetchcmd, >> ud.url) >> + runfetchcmd(svnfetchcmd, d, workdir=ud.pkgdir) >> + >> + if not ("externals" in ud.parm and ud.parm["externals"] >> == "nowarn"): >> + # Warn the user if this had externals (won't catch >> them all) >> + output = runfetchcmd("svn propget svn:externals || >> true", d, workdir=ud.moddir) >> + if output: >> + if "--ignore-externals" in svnfetchcmd.split(): >> + bb.warn("%s contains svn:externals." % >> ud.url) >> + bb.warn("These should be added to the recipe >> SRC_URI as necessary.") >> + bb.warn("svn fetch has ignored >> externals:\n%s" % output) >> + bb.warn("To disable this warning add >> ';externals=nowarn' to the url.") >> + else: >> + bb.debug(1, "svn repository has >> externals:\n%s" % output) + >> + scmdata = ud.parm.get("scmdata", "") >> + if scmdata == "keep": >> + tar_flags = "" >> + else: >> + tar_flags = "--exclude='.svn'" >> >> - # tar them up to a defined filename >> - runfetchcmd("tar %s -czf %s %s" % (tar_flags, ud.localpath, >> ud.path_spec), d, >> - cleanup=[ud.localpath], workdir=ud.pkgdir) >> + # tar them up to a defined filename >> + runfetchcmd("tar %s -czf %s %s" % (tar_flags, >> ud.localpath, ud.path_spec), d, >> + cleanup=[ud.localpath], workdir=ud.pkgdir) >> + finally: >> + bb.utils.unlockfile(lf) >> >> def clean(self, ud, d): >> """ Clean SVN specific files and dirs """ >> diff --git a/bitbake/lib/bb/fetch2/wget.py >> b/bitbake/lib/bb/fetch2/wget.py index 8f505b6..725586d 100644 >> --- a/bitbake/lib/bb/fetch2/wget.py >> +++ b/bitbake/lib/bb/fetch2/wget.py >> @@ -1,5 +1,3 @@ >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> """ >> BitBake 'Fetch' implementations >> >> @@ -10,18 +8,7 @@ BitBake build tools. >> >> # Copyright (C) 2003, 2004 Chris Larson >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> -# >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only >> # >> # Based on functions from the base bb module, Copyright 2003 Holger >> Schurig >> @@ -33,11 +20,14 @@ import logging >> import errno >> import bb >> import bb.progress >> +import socket >> +import http.client >> import urllib.request, urllib.parse, urllib.error >> from bb.fetch2 import FetchMethod >> from bb.fetch2 import FetchError >> from bb.fetch2 import logger >> from bb.fetch2 import runfetchcmd >> +from bb.fetch2 import FetchConnectionCache >> from bb.utils import export_proxies >> from bs4 import BeautifulSoup >> from bs4 import SoupStrainer >> @@ -132,10 +122,6 @@ class Wget(FetchMethod): >> return True >> >> def checkstatus(self, fetch, ud, d, try_again=True): >> - import urllib.request, urllib.error, urllib.parse, socket, >> http.client >> - from urllib.response import addinfourl >> - from bb.fetch2 import FetchConnectionCache >> - >> class HTTPConnectionCache(http.client.HTTPConnection): >> if fetch.connection_cache: >> def connect(self): >> @@ -168,7 +154,7 @@ class Wget(FetchMethod): >> """ >> host = req.host >> if not host: >> - raise urlllib2.URLError('no host given') >> + raise urllib.error.URLError('no host given') >> >> h = http_class(host, timeout=req.timeout) # will >> parse host:port h.set_debuglevel(self._debuglevel) >> @@ -185,7 +171,7 @@ class Wget(FetchMethod): >> # request. >> >> # Don't close connection when connection_cache is >> enabled, >> - if fetch.connection_cache is None: >> + if fetch.connection_cache is None: >> headers["Connection"] = "close" >> else: >> headers["Connection"] = "Keep-Alive" # Works for >> HTTP/1.0 @@ -252,7 +238,7 @@ class Wget(FetchMethod): >> pass >> closed = False >> >> - resp = addinfourl(fp_dummy(), r.msg, >> req.get_full_url()) >> + resp = urllib.response.addinfourl(fp_dummy(), r.msg, >> req.get_full_url()) resp.code = r.status >> resp.msg = r.reason >> >> @@ -271,17 +257,18 @@ class Wget(FetchMethod): >> fp.read() >> fp.close() >> >> - newheaders = dict((k,v) for k,v in >> list(req.headers.items()) >> - if k.lower() not in >> ("content-length", "content-type")) >> - return >> self.parent.open(urllib.request.Request(req.get_full_url(), >> - >> headers=newheaders, >> - >> origin_req_host=req.origin_req_host, >> - >> unverifiable=True)) >> + if req.get_method() != 'GET': >> + newheaders = dict((k, v) for k, v in >> list(req.headers.items()) >> + if k.lower() not in >> ("content-length", "content-type")) >> + return >> self.parent.open(urllib.request.Request(req.get_full_url(), >> + >> headers=newheaders, >> + >> origin_req_host=req.origin_req_host, >> + >> unverifiable=True)) >> - """ >> - Some servers (e.g. GitHub archives, hosted on Amazon S3) >> return 403 >> - Forbidden when they actually mean 405 Method Not Allowed. >> - """ >> + raise urllib.request.HTTPError(req, code, msg, >> headers, None) + >> + # Some servers (e.g. GitHub archives, hosted on Amazon >> S3) return 403 >> + # Forbidden when they actually mean 405 Method Not >> Allowed. http_error_403 = http_error_405 >> >> >> @@ -292,15 +279,15 @@ class Wget(FetchMethod): >> """ >> def redirect_request(self, req, fp, code, msg, headers, >> newurl): newreq = >> urllib.request.HTTPRedirectHandler.redirect_request(self, req, fp, >> code, msg, headers, newurl) >> - newreq.get_method = lambda: req.get_method() >> + newreq.get_method = req.get_method >> return newreq >> exported_proxies = export_proxies(d) >> >> handlers = [FixedHTTPRedirectHandler, HTTPMethodFallback] >> - if export_proxies: >> + if exported_proxies: >> handlers.append(urllib.request.ProxyHandler()) >> handlers.append(CacheHTTPHandler()) >> - # XXX: Since Python 2.7.9 ssl cert validation is enabled by >> default >> + # Since Python 2.7.9 ssl cert validation is enabled by >> default # see PEP-0476, this causes verification errors on some https >> servers # so disable by default. >> import ssl >> @@ -319,19 +306,19 @@ class Wget(FetchMethod): >> '''Adds Basic auth to http request, pass in >> login:password as string''' import base64 >> encodeuser = >> base64.b64encode(login_str.encode('utf-8')).decode("utf-8") >> - authheader = "Basic %s" % encodeuser >> + authheader = "Basic %s" % encodeuser >> r.add_header("Authorization", authheader) >> >> - if ud.user: >> - add_basic_auth(ud.user, r) >> + if ud.user and ud.pswd: >> + add_basic_auth(ud.user + ':' + ud.pswd, r) >> >> try: >> - import netrc, urllib.parse >> + import netrc >> n = netrc.netrc() >> login, unused, password = >> n.authenticators(urllib.parse.urlparse(uri).hostname) >> add_basic_auth("%s:%s" % (login, password), r) except (TypeError, >> ImportError, IOError, netrc.NetrcParseError): >> - pass >> + pass >> >> with opener.open(r) as response: >> pass >> @@ -396,18 +383,14 @@ class Wget(FetchMethod): >> (oldpn, oldpv, oldsuffix) = old >> (newpn, newpv, newsuffix) = new >> >> - """ >> - Check for a new suffix type that we have never heard of >> before >> - """ >> - if (newsuffix): >> + # Check for a new suffix type that we have never heard of >> before >> + if newsuffix: >> m = self.suffix_regex_comp.search(newsuffix) >> if not m: >> bb.warn("%s has a possible unknown suffix: %s" % >> (newpn, newsuffix)) return False >> >> - """ >> - Not our package so ignore it >> - """ >> + # Not our package so ignore it >> if oldpn != newpn: >> return False >> >> @@ -473,15 +456,14 @@ class Wget(FetchMethod): >> >> return "" >> >> - def _check_latest_version_by_dir(self, dirver, package, >> package_regex, >> - current_version, ud, d): >> + def _check_latest_version_by_dir(self, dirver, package, >> package_regex, current_version, ud, d): """ >> - Scan every directory in order to get upstream version. >> + Scan every directory in order to get upstream version. >> """ >> version_dir = ['', '', ''] >> version = ['', '', ''] >> >> - dirver_regex = >> re.compile("(?P\D*)(?P(\d+[\.\-_])+(\d+))") >> + dirver_regex = >> re.compile(r"(?P\D*)(?P(\d+[\.\-_])+(\d+))") s = >> dirver_regex.search(dirver) if s: >> version_dir[1] = s.group('ver') >> @@ -541,26 +523,26 @@ class Wget(FetchMethod): >> gst-fluendo-mp3 >> """ >> # match most patterns which uses "-" as separator to version >> digits >> - pn_prefix1 = "[a-zA-Z][a-zA-Z0-9]*([-_][a-zA-Z]\w+)*\+?[-_]" >> + pn_prefix1 = r"[a-zA-Z][a-zA-Z0-9]*([-_][a-zA-Z]\w+)*\+?[-_]" >> # a loose pattern such as for unzip552.tar.gz >> - pn_prefix2 = "[a-zA-Z]+" >> + pn_prefix2 = r"[a-zA-Z]+" >> # a loose pattern such as for 80325-quicky-0.4.tar.gz >> - pn_prefix3 = "[0-9]+[-]?[a-zA-Z]+" >> + pn_prefix3 = r"[0-9]+[-]?[a-zA-Z]+" >> # Save the Package Name (pn) Regex for use later >> - pn_regex = "(%s|%s|%s)" % (pn_prefix1, pn_prefix2, >> pn_prefix3) >> + pn_regex = r"(%s|%s|%s)" % (pn_prefix1, pn_prefix2, >> pn_prefix3) >> # match version >> - pver_regex = "(([A-Z]*\d+[a-zA-Z]*[\.\-_]*)+)" >> + pver_regex = r"(([A-Z]*\d+[a-zA-Z]*[\.\-_]*)+)" >> >> # match arch >> parch_regex = "-source|_all_" >> >> # src.rpm extension was added only for rpm package. Can be >> removed if the rpm # packaged will always be considered as having to >> be manually upgraded >> - psuffix_regex = >> "(tar\.gz|tgz|tar\.bz2|zip|xz|tar\.lz|rpm|bz2|orig\.tar\.gz|tar\.xz|src\.tar\.gz|src\.tgz|svnr\d+\.tar\.bz2|stable\.tar\.gz|src\.rpm)" >> + psuffix_regex = >> r"(tar\.gz|tgz|tar\.bz2|zip|xz|tar\.lz|rpm|bz2|orig\.tar\.gz|tar\.xz|src\.tar\.gz|src\.tgz|svnr\d+\.tar\.bz2|stable\.tar\.gz|src\.rpm)" >> # match name, version and archive type of a package >> - package_regex_comp = >> re.compile("(?P%s?\.?v?)(?P%s)(?P%s)?[\.-](?P%s$)" >> + package_regex_comp = >> re.compile(r"(?P%s?\.?v?)(?P%s)(?P%s)?[\.-](?P%s$)" >> % (pn_regex, pver_regex, parch_regex, psuffix_regex)) >> self.suffix_regex_comp = re.compile(psuffix_regex) >> @@ -572,7 +554,7 @@ class Wget(FetchMethod): >> version = self._parse_path(package_regex_comp, package) >> if version: >> package_custom_regex_comp = re.compile( >> - >> "(?P%s)(?P%s)(?P%s)?[\.-](?P%s)" % >> + >> r"(?P%s)(?P%s)(?P%s)?[\.-](?P%s)" % >> (re.escape(version[0]), pver_regex, parch_regex, psuffix_regex)) else: >> package_custom_regex_comp = None >> @@ -589,7 +571,7 @@ class Wget(FetchMethod): >> current_version = ['', d.getVar('PV'), ''] >> >> """possible to have no version in pkg name, such as >> spectrum-fw""" >> - if not re.search("\d+", package): >> + if not re.search(r"\d+", package): >> current_version[1] = re.sub('_', '.', current_version[1]) >> current_version[1] = re.sub('-', '.', current_version[1]) >> return (current_version[1], '') >> @@ -607,13 +589,13 @@ class Wget(FetchMethod): >> >> # search for version matches on folders inside the path, >> like: # "5.7" in >> http://download.gnome.org/sources/${PN}/5.7/${PN}-${PV}.tar.gz >> - dirver_regex = >> re.compile("(?P[^/]*(\d+\.)*\d+([-_]r\d+)*)/") >> + dirver_regex = >> re.compile(r"(?P[^/]*(\d+\.)*\d+([-_]r\d+)*)/") m = >> dirver_regex.search(path) if m: >> pn = d.getVar('PN') >> dirver = m.group('dirver') >> >> - dirver_pn_regex = re.compile("%s\d?" % >> (re.escape(pn))) >> + dirver_pn_regex = re.compile(r"%s\d?" % >> (re.escape(pn))) if not dirver_pn_regex.search(dirver): >> return (self._check_latest_version_by_dir(dirver, >> package, package_regex, current_version, ud, >> d), '') diff --git a/bitbake/lib/bb/main.py b/bitbake/lib/bb/main.py >> index 732a315..af2880f 100755 >> --- a/bitbake/lib/bb/main.py >> +++ b/bitbake/lib/bb/main.py >> @@ -1,6 +1,3 @@ >> -#!/usr/bin/env python >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> # >> # Copyright (C) 2003, 2004 Chris Larson >> # Copyright (C) 2003, 2004 Phil Blundell >> @@ -9,18 +6,8 @@ >> # Copyright (C) 2005 ROAD GmbH >> # Copyright (C) 2006 Richard Purdie >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> +# SPDX-License-Identifier: GPL-2.0-only >> # >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. >> import os >> import sys >> @@ -267,6 +254,11 @@ class >> BitBakeConfigParameters(cookerdata.ConfigParameters): help="Do not >> run any setscene tasks. sstate will be ignored and " "everything >> needed, built.") >> + parser.add_option("", "--skip-setscene", action="store_true", >> + dest="skipsetscene", default=False, >> + help="Skip setscene tasks if they would be >> executed. Tasks previously " >> + "restored from sstate will be kept, >> unlike --no-setscene") + >> parser.add_option("", "--setscene-only", action="store_true", >> dest="setsceneonly", default=False, >> help="Only run setscene tasks, don't run >> any real tasks.") @@ -448,7 +440,7 @@ def setup_bitbake(configParams, >> configuration, extrafeatures=None): else: >> logger.info("Reconnecting to bitbake server...") >> if not os.path.exists(sockname): >> - print("Previous bitbake instance shutting >> down?, waiting to retry...") >> + logger.info("Previous bitbake instance >> shutting down?, waiting to retry...") i = 0 >> lock = None >> # Wait for 5s or until we can get the lock >> @@ -460,12 +452,7 @@ def setup_bitbake(configParams, configuration, >> extrafeatures=None): bb.utils.unlockfile(lock) >> raise >> bb.server.process.ProcessTimeout("Bitbake still shutting down as >> socket exists but no lock?") if not configParams.server_only: >> - try: >> - server_connection = >> bb.server.process.connectProcessServer(sockname, featureset) >> - except EOFError: >> - # The server may have been shutting down but >> not closed the socket yet. If that happened, >> - # ignore it. >> - pass >> + server_connection = >> bb.server.process.connectProcessServer(sockname, featureset) >> if server_connection or configParams.server_only: >> break >> @@ -475,12 +462,14 @@ def setup_bitbake(configParams, configuration, >> extrafeatures=None): if not retries: >> raise >> retries -= 1 >> - if isinstance(e, (bb.server.process.ProcessTimeout, >> BrokenPipeError)): >> - logger.info("Retrying server connection...") >> + tryno = 8 - retries >> + if isinstance(e, (bb.server.process.ProcessTimeout, >> BrokenPipeError, EOFError)): >> + logger.info("Retrying server connection >> (#%d)..." % tryno) else: >> - logger.info("Retrying server connection... (%s)" >> % traceback.format_exc()) >> + logger.info("Retrying server connection (#%d)... >> (%s)" % (tryno, traceback.format_exc())) if not retries: >> - bb.fatal("Unable to connect to bitbake server, or >> start one") >> + bb.fatal("Unable to connect to bitbake server, or >> start one (server startup failures would be in >> bitbake-cookerdaemon.log).") >> + bb.event.print_ui_queue() >> if retries < 5: >> time.sleep(5) >> >> @@ -502,7 +491,7 @@ def setup_bitbake(configParams, configuration, >> extrafeatures=None): def lockBitbake(): >> topdir = bb.cookerdata.findTopdir() >> if not topdir: >> - bb.error("Unable to find conf/bblayers.conf or >> conf/bitbake.conf. BBAPTH is unset and/or not in a build directory?") >> + bb.error("Unable to find conf/bblayers.conf or >> conf/bitbake.conf. BBPATH is unset and/or not in a build directory?") >> raise BBMainFatal lockfile = topdir + "/bitbake.lock" >> return topdir, bb.utils.lockfile(lockfile, False, False) >> diff --git a/bitbake/lib/bb/methodpool.py >> b/bitbake/lib/bb/methodpool.py index 49aed33..51783ac 100644 >> --- a/bitbake/lib/bb/methodpool.py >> +++ b/bitbake/lib/bb/methodpool.py >> @@ -1,21 +1,8 @@ >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> -# >> # >> # Copyright (C) 2006 Holger Hans Peter Freyther >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> -# >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> +# SPDX-License-Identifier: GPL-2.0-only >> # >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. >> from bb.utils import better_compile, better_exec >> >> diff --git a/bitbake/lib/bb/monitordisk.py >> b/bitbake/lib/bb/monitordisk.py index 833cd3d..1a25b00 100644 >> --- a/bitbake/lib/bb/monitordisk.py >> +++ b/bitbake/lib/bb/monitordisk.py >> @@ -1,21 +1,8 @@ >> -#!/usr/bin/env python >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> # >> # Copyright (C) 2012 Robert Yang >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> +# SPDX-License-Identifier: GPL-2.0-only >> # >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. >> import os, logging, re, sys >> import bb >> @@ -28,16 +15,16 @@ def convertGMK(unit): >> >> """ Convert the space unit G, M, K, the unit is case-insensitive >> """ >> - unitG = re.match('([1-9][0-9]*)[gG]\s?$', unit) >> + unitG = re.match(r'([1-9][0-9]*)[gG]\s?$', unit) >> if unitG: >> return int(unitG.group(1)) * (1024 ** 3) >> - unitM = re.match('([1-9][0-9]*)[mM]\s?$', unit) >> + unitM = re.match(r'([1-9][0-9]*)[mM]\s?$', unit) >> if unitM: >> return int(unitM.group(1)) * (1024 ** 2) >> - unitK = re.match('([1-9][0-9]*)[kK]\s?$', unit) >> + unitK = re.match(r'([1-9][0-9]*)[kK]\s?$', unit) >> if unitK: >> return int(unitK.group(1)) * 1024 >> - unitN = re.match('([1-9][0-9]*)\s?$', unit) >> + unitN = re.match(r'([1-9][0-9]*)\s?$', unit) >> if unitN: >> return int(unitN.group(1)) >> else: >> @@ -83,7 +70,7 @@ def getDiskData(BBDirs, configuration): >> for pathSpaceInode in BBDirs.split(): >> # The input format is: "dir,space,inode", dir is a must, >> space # and inode are optional >> - pathSpaceInodeRe = re.match('([^,]*),([^,]*),([^,]*),?(.*)', >> pathSpaceInode) >> + pathSpaceInodeRe = >> re.match(r'([^,]*),([^,]*),([^,]*),?(.*)', pathSpaceInode) if not >> pathSpaceInodeRe: printErr("Invalid value in BB_DISKMON_DIRS: %s" % >> pathSpaceInode) return None >> @@ -147,7 +134,7 @@ def getInterval(configuration): >> else: >> # The disk space or inode interval is optional, but it should >> # have a correct value once it is specified >> - intervalRe = re.match('([^,]*),?\s*(.*)', interval) >> + intervalRe = re.match(r'([^,]*),?\s*(.*)', interval) >> if intervalRe: >> intervalSpace = intervalRe.group(1) >> if intervalSpace: >> diff --git a/bitbake/lib/bb/msg.py b/bitbake/lib/bb/msg.py >> index 96f077e..6216eb3 100644 >> --- a/bitbake/lib/bb/msg.py >> +++ b/bitbake/lib/bb/msg.py >> @@ -1,5 +1,3 @@ >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> """ >> BitBake 'msg' implementation >> >> @@ -9,18 +7,8 @@ Message handling infrastructure for bitbake >> >> # Copyright (C) 2006 Richard Purdie >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> +# SPDX-License-Identifier: GPL-2.0-only >> # >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. >> import sys >> import copy >> diff --git a/bitbake/lib/bb/namedtuple_with_abc.py >> b/bitbake/lib/bb/namedtuple_with_abc.py index 32f2fc6..646aed6 100644 >> --- a/bitbake/lib/bb/namedtuple_with_abc.py >> +++ b/bitbake/lib/bb/namedtuple_with_abc.py >> @@ -1,6 +1,8 @@ >> # >> http://code.activestate.com/recipes/577629-namedtupleabc-abstract-base-class-mix-in-for-named/ >> -#!/usr/bin/env python # Copyright (c) 2011 Jan Kaliszewski (zuo). >> Available under the MIT License. +# >> +# SPDX-License-Identifier: MIT >> +# >> >> """ >> namedtuple_with_abc.py: >> diff --git a/bitbake/lib/bb/parse/__init__.py >> b/bitbake/lib/bb/parse/__init__.py index 5397d57..76e180b 100644 >> --- a/bitbake/lib/bb/parse/__init__.py >> +++ b/bitbake/lib/bb/parse/__init__.py >> @@ -9,20 +9,10 @@ File parsers for the BitBake build tools. >> # Copyright (C) 2003, 2004 Chris Larson >> # Copyright (C) 2003, 2004 Phil Blundell >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> -# >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only >> # >> # Based on functions from the base bb module, Copyright 2003 Holger >> Schurig +# >> >> handlers = [] >> >> diff --git a/bitbake/lib/bb/parse/ast.py b/bitbake/lib/bb/parse/ast.py >> index 6d7c80b..f0911e6 100644 >> --- a/bitbake/lib/bb/parse/ast.py >> +++ b/bitbake/lib/bb/parse/ast.py >> @@ -1,5 +1,3 @@ >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> """ >> AbstractSyntaxTree classes for the Bitbake language >> """ >> @@ -8,19 +6,8 @@ >> # Copyright (C) 2003, 2004 Phil Blundell >> # Copyright (C) 2009 Holger Hans Peter Freyther >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> +# SPDX-License-Identifier: GPL-2.0-only >> # >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. - >> >> import re >> import string >> diff --git a/bitbake/lib/bb/parse/parse_py/BBHandler.py >> b/bitbake/lib/bb/parse/parse_py/BBHandler.py index 01fc47e..6f7cf82 >> 100644 --- a/bitbake/lib/bb/parse/parse_py/BBHandler.py >> +++ b/bitbake/lib/bb/parse/parse_py/BBHandler.py >> @@ -1,6 +1,3 @@ >> -#!/usr/bin/env python >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> """ >> class for handling .bb files >> >> @@ -12,19 +9,8 @@ >> # Copyright (C) 2003, 2004 Chris Larson >> # Copyright (C) 2003, 2004 Phil Blundell >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> +# SPDX-License-Identifier: GPL-2.0-only >> # >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. - >> >> import re, bb, os >> import logging >> @@ -38,14 +24,15 @@ from .ConfHandler import include, init >> # For compatibility >> bb.deprecate_import(__name__, "bb.parse", ["vars_from_file"]) >> >> -__func_start_regexp__ = >> re.compile( r"(((?Ppython)|(?Pfakeroot))\s*)*(?P[\w\.\-\+\{\}\$]+)?\s*\(\s*\)\s*{$" ) >> -__inherit_regexp__ = re.compile( r"inherit\s+(.+)" ) >> -__export_func_regexp__ = re.compile( r"EXPORT_FUNCTIONS\s+(.+)" ) >> -__addtask_regexp__ = >> re.compile("addtask\s+(?P\w+)\s*((before\s*(?P((.*(?=after))|(.*))))|(after\s*(?P((.*(?=before))|(.*)))))*") >> -__deltask_regexp__ = re.compile("deltask\s+(?P\w+)") >> -__addhandler_regexp__ = re.compile( r"addhandler\s+(.+)" ) >> -__def_regexp__ = re.compile( r"def\s+(\w+).*:" ) >> -__python_func_regexp__ = re.compile( r"(\s+.*)|(^$)|(^#)" ) >> +__func_start_regexp__ = >> re.compile(r"(((?Ppython)|(?Pfakeroot))\s*)*(?P[\w\.\-\+\{\}\$]+)?\s*\(\s*\)\s*{$" ) >> +__inherit_regexp__ = re.compile(r"inherit\s+(.+)" ) >> +__export_func_regexp__ = re.compile(r"EXPORT_FUNCTIONS\s+(.+)" ) >> +__addtask_regexp__ = >> re.compile(r"addtask\s+(?P\w+)\s*((before\s*(?P((.*(?=after))|(.*))))|(after\s*(?P((.*(?=before))|(.*)))))*") >> +__deltask_regexp__ = >> re.compile(r"deltask\s+(?P\w+)(?P.*)") >> +__addhandler_regexp__ = re.compile(r"addhandler\s+(.+)" ) >> +__def_regexp__ = re.compile(r"def\s+(\w+).*:" ) >> +__python_func_regexp__ = re.compile(r"(\s+.*)|(^$)|(^#)" ) >> +__python_tab_regexp__ = re.compile(r" *\t") __infunc__ = [] >> __inpython__ = False @@ -160,6 +147,16 @@ def handle(fn, d, include): >> def feeder(lineno, s, fn, root, statements, eof=False): global >> __func_start_regexp__, __inherit_regexp__, __export_func_regexp__, >> __addtask_regexp__, __addhandler_regexp__, __def_regexp__, >> __python_func_regexp__, __inpython__, __infunc__, __body__, bb, >> __residue__, __classname__ + >> + # Check tabs in python functions: >> + # - def py_funcname(): covered by __inpython__ >> + # - python(): covered by '__anonymous' == __infunc__[0] >> + # - python funcname(): covered by __infunc__[3] >> + if __inpython__ or (__infunc__ and ('__anonymous' == >> __infunc__[0] or __infunc__[3])): >> + tab = __python_tab_regexp__.match(s) >> + if tab: >> + bb.warn('python should use 4 spaces indentation, but >> found tabs in %s, line %s' % (root, lineno)) + >> if __infunc__: >> if s == '}': >> __body__.append('') >> @@ -225,11 +222,27 @@ def feeder(lineno, s, fn, root, statements, >> eof=False): >> m = __addtask_regexp__.match(s) >> if m: >> + if len(m.group().split()) == 2: >> + # Check and warn for "addtask task1 task2" >> + m2 = re.match(r"addtask\s+(?P\w+)(?P.*)", >> s) >> + if m2 and m2.group('ignores'): >> + logger.warning('addtask ignored: "%s"' % >> m2.group('ignores')) + >> + # Check and warn for "addtask task1 before task2 before >> task3", the >> + # similar to "after" >> + taskexpression = s.split() >> + for word in ('before', 'after'): >> + if taskexpression.count(word) > 1: >> + logger.warning("addtask contained multiple '%s' >> keywords, only one is supported" % word) + >> ast.handleAddTask(statements, fn, lineno, m) >> return >> >> m = __deltask_regexp__.match(s) >> if m: >> + # Check and warn "for deltask task1 task2" >> + if m.group('ignores'): >> + logger.warning('deltask ignored: "%s"' % >> m.group('ignores')) ast.handleDelTask(statements, fn, lineno, m) >> return >> >> diff --git a/bitbake/lib/bb/parse/parse_py/ConfHandler.py >> b/bitbake/lib/bb/parse/parse_py/ConfHandler.py index 9d3ebe1..2e84b91 >> 100644 --- a/bitbake/lib/bb/parse/parse_py/ConfHandler.py >> +++ b/bitbake/lib/bb/parse/parse_py/ConfHandler.py >> @@ -1,6 +1,3 @@ >> -#!/usr/bin/env python >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> """ >> class for handling configuration data files >> >> @@ -11,18 +8,8 @@ >> # Copyright (C) 2003, 2004 Chris Larson >> # Copyright (C) 2003, 2004 Phil Blundell >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> +# SPDX-License-Identifier: GPL-2.0-only >> # >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. >> import errno >> import re >> @@ -147,7 +134,7 @@ def handle(fn, data, include): >> continue >> s = s.rstrip() >> while s[-1] == '\\': >> - s2 = f.readline().strip() >> + s2 = f.readline().rstrip() >> lineno = lineno + 1 >> if (not s2 or s2 and s2[0] != "#") and s[0] == "#" : >> bb.fatal("There is a confusing multiline, partially >> commented expression on line %s of file %s (%s).\nPlease clarify >> whether this is all a comment or should be parsed." % (lineno, fn, >> s)) diff --git a/bitbake/lib/bb/parse/parse_py/__init__.py >> b/bitbake/lib/bb/parse/parse_py/__init__.py index 3e658d0..f508afa >> 100644 --- a/bitbake/lib/bb/parse/parse_py/__init__.py +++ >> b/bitbake/lib/bb/parse/parse_py/__init__.py @@ -1,6 +1,3 @@ >> -#!/usr/bin/env python -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> """ >> BitBake Parsers >> >> @@ -11,20 +8,10 @@ File parsers for the BitBake build tools. >> # Copyright (C) 2003, 2004 Chris Larson >> # Copyright (C) 2003, 2004 Phil Blundell >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> -# >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only >> # >> # Based on functions from the base bb module, Copyright 2003 Holger >> Schurig +# >> >> from __future__ import absolute_import >> from . import ConfHandler >> diff --git a/bitbake/lib/bb/persist_data.py >> b/bitbake/lib/bb/persist_data.py index bef7018..de8f87a 100644 >> --- a/bitbake/lib/bb/persist_data.py >> +++ b/bitbake/lib/bb/persist_data.py >> @@ -8,18 +8,8 @@ currently, providing a key/value store accessed by >> 'domain'. # Copyright (C) 2007 Richard Purdie >> # Copyright (C) 2010 Chris Larson >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> +# SPDX-License-Identifier: GPL-2.0-only >> # >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. >> import collections >> import logging >> @@ -29,6 +19,7 @@ import warnings >> from bb.compat import total_ordering >> from collections import Mapping >> import sqlite3 >> +import contextlib >> >> sqlversion = sqlite3.sqlite_version_info >> if sqlversion[0] < 3 or (sqlversion[0] == 3 and sqlversion[1] < 3): >> @@ -36,84 +27,181 @@ if sqlversion[0] < 3 or (sqlversion[0] == 3 and >> sqlversion[1] < 3): >> >> logger = logging.getLogger("BitBake.PersistData") >> -if hasattr(sqlite3, 'enable_shared_cache'): >> - try: >> - sqlite3.enable_shared_cache(True) >> - except sqlite3.OperationalError: >> - pass >> - >> >> @total_ordering >> class SQLTable(collections.MutableMapping): >> + class _Decorators(object): >> + @staticmethod >> + def retry(*, reconnect=True): >> + """ >> + Decorator that restarts a function if a database locked >> sqlite >> + exception occurs. If reconnect is True, the database >> connection >> + will be closed and reopened each time a failure occurs >> + """ >> + def retry_wrapper(f): >> + def wrap_func(self, *args, **kwargs): >> + # Reconnect if necessary >> + if self.connection is None and reconnect: >> + self.reconnect() >> + >> + count = 0 >> + while True: >> + try: >> + return f(self, *args, **kwargs) >> + except sqlite3.OperationalError as exc: >> + if count < 500 and ('is locked' in >> str(exc) or 'locking protocol' in str(exc)): >> + count = count + 1 >> + if reconnect: >> + self.reconnect() >> + continue >> + raise >> + return wrap_func >> + return retry_wrapper >> + >> + @staticmethod >> + def transaction(f): >> + """ >> + Decorator that starts a database transaction and creates >> a database >> + cursor for performing queries. If no exception is >> thrown, the >> + database results are commited. If an exception occurs, >> the database >> + is rolled back. In all cases, the cursor is closed after >> the >> + function ends. >> + >> + Note that the cursor is passed as an extra argument to >> the function >> + after `self` and before any of the normal arguments >> + """ >> + def wrap_func(self, *args, **kwargs): >> + # Context manager will COMMIT the database on >> success, >> + # or ROLLBACK on an exception >> + with self.connection: >> + # Automatically close the cursor when done >> + with >> contextlib.closing(self.connection.cursor()) as cursor: >> + return f(self, cursor, *args, **kwargs) >> + return wrap_func >> + >> """Object representing a table/domain in the database""" >> def __init__(self, cachefile, table): >> self.cachefile = cachefile >> self.table = table >> - self.cursor = connect(self.cachefile) >> - >> - self._execute("CREATE TABLE IF NOT EXISTS %s(key TEXT, value >> TEXT);" >> - % table) >> - >> - def _execute(self, *query): >> - """Execute a query, waiting to acquire a lock if necessary""" >> - count = 0 >> - while True: >> - try: >> - return self.cursor.execute(*query) >> - except sqlite3.OperationalError as exc: >> - if 'database is locked' in str(exc) and count < 500: >> - count = count + 1 >> + >> + self.connection = None >> + self._execute_single("CREATE TABLE IF NOT EXISTS %s(key TEXT >> PRIMARY KEY NOT NULL, value TEXT);" % table) + >> + @_Decorators.retry(reconnect=False) >> + @_Decorators.transaction >> + def _setup_database(self, cursor): >> + cursor.execute("pragma synchronous = off;") >> + # Enable WAL and keep the autocheckpoint length small (the >> default is >> + # usually 1000). Persistent caches are usually read-mostly, >> so keeping >> + # this short will keep readers running quickly >> + cursor.execute("pragma journal_mode = WAL;") >> + cursor.execute("pragma wal_autocheckpoint = 100;") >> + >> + def reconnect(self): >> + if self.connection is not None: >> + self.connection.close() >> + self.connection = sqlite3.connect(self.cachefile, timeout=5) >> + self.connection.text_factory = str >> + self._setup_database() >> + >> + @_Decorators.retry() >> + @_Decorators.transaction >> + def _execute_single(self, cursor, *query): >> + """ >> + Executes a single query and discards the results. This >> correctly closes >> + the database cursor when finished >> + """ >> + cursor.execute(*query) >> + >> + @_Decorators.retry() >> + def _row_iter(self, f, *query): >> + """ >> + Helper function that returns a row iterator. Each time >> __next__ is >> + called on the iterator, the provided function is evaluated >> to determine >> + the return value >> + """ >> + class CursorIter(object): >> + def __init__(self, cursor): >> + self.cursor = cursor >> + >> + def __iter__(self): >> + return self >> + >> + def __next__(self): >> + row = self.cursor.fetchone() >> + if row is None: >> self.cursor.close() >> - self.cursor = connect(self.cachefile) >> - continue >> - raise >> + raise StopIteration >> + return f(row) >> + >> + def __enter__(self): >> + return self >> + >> + def __exit__(self, typ, value, traceback): >> + self.cursor.close() >> + return False >> + >> + cursor = self.connection.cursor() >> + try: >> + cursor.execute(*query) >> + return CursorIter(cursor) >> + except: >> + cursor.close() >> >> def __enter__(self): >> - self.cursor.__enter__() >> + self.connection.__enter__() >> return self >> >> def __exit__(self, *excinfo): >> - self.cursor.__exit__(*excinfo) >> - >> - def __getitem__(self, key): >> - data = self._execute("SELECT * from %s where key=?;" % >> - self.table, [key]) >> - for row in data: >> + self.connection.__exit__(*excinfo) >> + >> + @_Decorators.retry() >> + @_Decorators.transaction >> + def __getitem__(self, cursor, key): >> + cursor.execute("SELECT * from %s where key=?;" % self.table, >> [key]) >> + row = cursor.fetchone() >> + if row is not None: >> return row[1] >> raise KeyError(key) >> >> - def __delitem__(self, key): >> + @_Decorators.retry() >> + @_Decorators.transaction >> + def __delitem__(self, cursor, key): >> if key not in self: >> raise KeyError(key) >> - self._execute("DELETE from %s where key=?;" % self.table, >> [key]) >> + cursor.execute("DELETE from %s where key=?;" % self.table, >> [key]) >> - def __setitem__(self, key, value): >> + @_Decorators.retry() >> + @_Decorators.transaction >> + def __setitem__(self, cursor, key, value): >> if not isinstance(key, str): >> raise TypeError('Only string keys are supported') >> elif not isinstance(value, str): >> raise TypeError('Only string values are supported') >> >> - data = self._execute("SELECT * from %s where key=?;" % >> - self.table, [key]) >> - exists = len(list(data)) >> - if exists: >> - self._execute("UPDATE %s SET value=? WHERE key=?;" % >> self.table, >> - [value, key]) >> + cursor.execute("SELECT * from %s where key=?;" % self.table, >> [key]) >> + row = cursor.fetchone() >> + if row is not None: >> + cursor.execute("UPDATE %s SET value=? WHERE key=?;" % >> self.table, [value, key]) else: >> - self._execute("INSERT into %s(key, value) values >> (?, ?);" % >> - self.table, [key, value]) >> - >> - def __contains__(self, key): >> - return key in set(self) >> - >> - def __len__(self): >> - data = self._execute("SELECT COUNT(key) FROM %s;" % >> self.table) >> - for row in data: >> + cursor.execute("INSERT into %s(key, value) values >> (?, ?);" % self.table, [key, value]) + >> + @_Decorators.retry() >> + @_Decorators.transaction >> + def __contains__(self, cursor, key): >> + cursor.execute('SELECT * from %s where key=?;' % self.table, >> [key]) >> + return cursor.fetchone() is not None >> + >> + @_Decorators.retry() >> + @_Decorators.transaction >> + def __len__(self, cursor): >> + cursor.execute("SELECT COUNT(key) FROM %s;" % self.table) >> + row = cursor.fetchone() >> + if row is not None: >> return row[0] >> >> def __iter__(self): >> - data = self._execute("SELECT key FROM %s;" % self.table) >> - return (row[0] for row in data) >> + return self._row_iter(lambda row: row[0], "SELECT key from >> %s;" % self.table) >> def __lt__(self, other): >> if not isinstance(other, Mapping): >> @@ -122,25 +210,27 @@ class SQLTable(collections.MutableMapping): >> return len(self) < len(other) >> >> def get_by_pattern(self, pattern): >> - data = self._execute("SELECT * FROM %s WHERE key LIKE ?;" % >> - self.table, [pattern]) >> - return [row[1] for row in data] >> + return self._row_iter(lambda row: row[1], "SELECT * FROM %s >> WHERE key LIKE ?;" % >> + self.table, [pattern]) >> >> def values(self): >> return list(self.itervalues()) >> >> def itervalues(self): >> - data = self._execute("SELECT value FROM %s;" % self.table) >> - return (row[0] for row in data) >> + return self._row_iter(lambda row: row[0], "SELECT value FROM >> %s;" % >> + self.table) >> >> def items(self): >> return list(self.iteritems()) >> >> def iteritems(self): >> - return self._execute("SELECT * FROM %s;" % self.table) >> + return self._row_iter(lambda row: (row[0], row[1]), "SELECT >> * FROM %s;" % >> + self.table) >> >> - def clear(self): >> - self._execute("DELETE FROM %s;" % self.table) >> + @_Decorators.retry() >> + @_Decorators.transaction >> + def clear(self, cursor): >> + cursor.execute("DELETE FROM %s;" % self.table) >> >> def has_key(self, key): >> return key in self >> @@ -194,12 +284,6 @@ class PersistData(object): >> """ >> del self.data[domain][key] >> >> -def connect(database): >> - connection = sqlite3.connect(database, timeout=5, >> isolation_level=None) >> - connection.execute("pragma synchronous = off;") >> - connection.text_factory = str >> - return connection >> - >> def persist(domain, d): >> """Convenience factory for SQLTable objects based upon >> metadata""" import bb.utils >> diff --git a/bitbake/lib/bb/process.py b/bitbake/lib/bb/process.py >> index e69697c..2dc472a 100644 >> --- a/bitbake/lib/bb/process.py >> +++ b/bitbake/lib/bb/process.py >> @@ -1,3 +1,7 @@ >> +# >> +# SPDX-License-Identifier: GPL-2.0-only >> +# >> + >> import logging >> import signal >> import subprocess >> diff --git a/bitbake/lib/bb/progress.py b/bitbake/lib/bb/progress.py >> index f54d1c7..4022caa 100644 >> --- a/bitbake/lib/bb/progress.py >> +++ b/bitbake/lib/bb/progress.py >> @@ -4,18 +4,8 @@ BitBake progress handling code >> >> # Copyright (C) 2016 Intel Corporation >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> +# SPDX-License-Identifier: GPL-2.0-only >> # >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. >> import sys >> import re >> @@ -23,6 +13,7 @@ import time >> import inspect >> import bb.event >> import bb.build >> +from bb.build import StdoutNoopContextManager >> >> class ProgressHandler(object): >> """ >> @@ -37,7 +28,14 @@ class ProgressHandler(object): >> if outfile: >> self._outfile = outfile >> else: >> - self._outfile = sys.stdout >> + self._outfile = StdoutNoopContextManager() >> + >> + def __enter__(self): >> + self._outfile.__enter__() >> + return self >> + >> + def __exit__(self, *excinfo): >> + self._outfile.__exit__(*excinfo) >> >> def _fire_progress(self, taskprogress, rate=None): >> """Internal function to fire the progress event""" >> @@ -157,6 +155,12 @@ class MultiStageProgressReporter(object): >> self._stage_total = None >> self._callers = [] >> >> + def __enter__(self): >> + return self >> + >> + def __exit__(self, *excinfo): >> + pass >> + >> def _fire_progress(self, taskprogress): >> bb.event.fire(bb.build.TaskProgress(taskprogress), >> self._data) >> diff --git a/bitbake/lib/bb/providers.py b/bitbake/lib/bb/providers.py >> index c2aa98c..f80963c 100644 >> --- a/bitbake/lib/bb/providers.py >> +++ b/bitbake/lib/bb/providers.py >> @@ -1,5 +1,3 @@ >> -# ex:ts=4:sw=4:sts=4:et >> -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- >> # >> # Copyright (C) 2003, 2004 Chris Larson >> # Copyright (C) 2003, 2004 Phil Blundell >> @@ -8,18 +6,8 @@ >> # Copyright (C) 2005 ROAD GmbH >> # Copyright (C) 2006 Richard Purdie >> # >> -# This program is free software; you can redistribute it and/or >> modify -# it under the terms of the GNU General Public License >> version 2 as -# published by the Free Software Foundation. >> +# SPDX-License-Identifier: GPL-2.0-only >> # >> -# This program is distributed in the hope that it will be useful, >> -# but WITHOUT ANY WARRANTY; without even the implied warranty of >> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >> -# GNU General Public License for more details. >> -# >> -# You should have received a copy of the GNU General Public License >> along -# with this program; if not, write to the Free Software >> Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA >> 02110-1301 USA. >> import re >> import logging >> @@ -129,7 +117,7 @@ def findPreferredProvider(pn, cfgData, dataCache, >> pkg_pn = None, item = None): preferred_v = >> cfgData.getVar("PREFERRED_VERSION") >> if preferred_v: >> - m = re.match('(\d+:)*(.*)(_.*)*', preferred_v) >> + m = re.match(r'(\d+:)*(.*)(_.*)*', preferred_v) >> if m: >> if m.group(1): >> preferred_e = m.group(1)[:-1] >> @@ -384,7 +372,7 @@ def getRuntimeProviders(dataCache, rdepend): >> >> # Only search dynamic packages if we can't find anything in >> other variables for pattern in dataCache.packages_dynamic: >> - pattern = pattern.replace('+', "\+") >> + pattern = pattern.replace(r'+', r"\+") >> if pattern in regexp_cache: >> regexp = regexp_cache[pattern] >> else: >> diff --git a/bitbake/lib/bb/pysh/builtin.py >> b/bitbake/lib/bb/pysh/builtin.py deleted file mode 100644 >> index a8814dc..0000000 >> --- a/bitbake/lib/bb/pysh/builtin.py >> +++ /dev/null >> @@ -1,710 +0,0 @@ >> -# builtin.py - builtins and utilities definitions for pysh. >> -# >> -# Copyright 2007 Patrick Mezard >> -# >> -# This software may be used and distributed according to the terms >> -# of the GNU General Public License, incorporated herein by >> reference. - >> -"""Builtin and internal utilities implementations. >> - >> -- Beware not to use python interpreter environment as if it were the >> shell -environment. For instance, commands working directory must be >> explicitely handled -through env['PWD'] instead of relying on python >> working directory. -""" >> -import errno >> -import optparse >> -import os >> -import re >> -import subprocess >> -import sys >> -import time >> - >> -def has_subprocess_bug(): >> - return getattr(subprocess, 'list2cmdline') and \ >> - ( subprocess.list2cmdline(['']) == '' or \ >> - subprocess.list2cmdline(['foo|bar']) == 'foo|bar') >> - >> -# Detect python bug 1634343: "subprocess swallows empty arguments >> under win32" -# >> >> -# Also detect: "[ 1710802 ] subprocess must escape redirection >> characters under win32" -# >> >> -if has_subprocess_bug(): >> - import subprocess_fix >> - subprocess.list2cmdline = subprocess_fix.list2cmdline >> - >> -from sherrors import * >> - >> -class NonExitingParser(optparse.OptionParser): >> - """OptionParser default behaviour upon error is to print the >> error message and >> - exit. Raise a utility error instead. >> - """ >> - def error(self, msg): >> - raise UtilityError(msg) >> - >> -#------------------------------------------------------------------------------- >> -# set special builtin >> -#------------------------------------------------------------------------------- >> -OPT_SET = NonExitingParser(usage="set - set or unset options and >> positional parameters") -OPT_SET.add_option( '-f', >> action='store_true', dest='has_f', default=False, >> - help='The shell shall disable pathname expansion.') >> -OPT_SET.add_option('-e', action='store_true', dest='has_e', >> default=False, >> - help="""When this option is on, if a simple command fails for >> any of the \ >> - reasons listed in Consequences of Shell Errors or returns an >> exit status \ >> - value >0, and is not part of the compound list following a >> while, until, \ >> - or if keyword, and is not a part of an AND or OR list, and is >> not a \ >> - pipeline preceded by the ! reserved word, then the shell shall >> immediately \ >> - exit.""") >> -OPT_SET.add_option('-x', action='store_true', dest='has_x', >> default=False, >> - help="""The shell shall write to standard error a trace for each >> command \ >> - after it expands the command and before it executes it. It is >> unspecified \ >> - whether the command that turns tracing off is traced.""") >> - >> -def builtin_set(name, args, interp, env, stdin, stdout, stderr, >> debugflags): >> - if 'debug-utility' in debugflags: >> - print interp.log(' '.join([name, str(args), interp['PWD']]) >> + '\n') - >> - option, args = OPT_SET.parse_args(args) >> - env = interp.get_env() >> - >> - if option.has_f: >> - env.set_opt('-f') >> - if option.has_e: >> - env.set_opt('-e') >> - if option.has_x: >> - env.set_opt('-x') >> - return 0 >> - >> -#------------------------------------------------------------------------------- >> -# shift special builtin >> -#------------------------------------------------------------------------------- >> -def builtin_shift(name, args, interp, env, stdin, stdout, stderr, >> debugflags): >> - if 'debug-utility' in debugflags: >> - print interp.log(' '.join([name, str(args), interp['PWD']]) >> + '\n') >> - >> - params = interp.get_env().get_positional_args() >> - if args: >> - try: >> - n = int(args[0]) >> - if n > len(params): >> - raise ValueError() >> - except ValueError: >> - return 1 >> - else: >> - n = 1 >> - >> - params[:n] = [] >> - interp.get_env().set_positional_args(params) >> - return 0 >> - >> -#------------------------------------------------------------------------------- >> -# export special builtin >> -#------------------------------------------------------------------------------- >> -OPT_EXPORT = NonExitingParser(usage="set - set or unset options and >> positional parameters") -OPT_EXPORT.add_option('-p', >> action='store_true', dest='has_p', default=False) - >> -def builtin_export(name, args, interp, env, stdin, stdout, stderr, >> debugflags): >> - if 'debug-utility' in debugflags: >> - print interp.log(' '.join([name, str(args), interp['PWD']]) >> + '\n') >> - >> - option, args = OPT_EXPORT.parse_args(args) >> - if option.has_p: >> - raise NotImplementedError() >> - >> - for arg in args: >> - try: >> - name, value = arg.split('=', 1) >> - except ValueError: >> - name, value = arg, None >> - env = interp.get_env().export(name, value) >> - >> - return 0 >> - >> -#------------------------------------------------------------------------------- >> -# return special builtin >> -#------------------------------------------------------------------------------- >> -def builtin_return(name, args, interp, env, stdin, stdout, stderr, >> debugflags): >> - if 'debug-utility' in debugflags: >> - print interp.log(' '.join([name, str(args), interp['PWD']]) >> + '\n') >> - res = 0 >> - if args: >> - try: >> - res = int(args[0]) >> - except ValueError: >> - res = 0 >> - if not 0<=res<=255: >> - res = 0 >> - >> - # BUG: should be last executed command exit code >> - raise ReturnSignal(res) >> - >> -#------------------------------------------------------------------------------- >> -# trap special builtin >> -#------------------------------------------------------------------------------- >> -def builtin_trap(name, args, interp, env, stdin, stdout, stderr, >> debugflags): >> - if 'debug-utility' in debugflags: >> - print interp.log(' '.join([name, str(args), interp['PWD']]) >> + '\n') >> - if len(args) < 2: >> - stderr.write('trap: usage: trap [[arg] signal_spec ...]\n') >> - return 2 >> - >> - action = args[0] >> - for sig in args[1:]: >> - try: >> - env.traps[sig] = action >> - except Exception as e: >> - stderr.write('trap: %s\n' % str(e)) >> - return 0 >> - >> -#------------------------------------------------------------------------------- >> -# unset special builtin >> -#------------------------------------------------------------------------------- >> -OPT_UNSET = NonExitingParser("unset - unset values and attributes of >> variables and functions") -OPT_UNSET.add_option( '-f', >> action='store_true', dest='has_f', default=False) >> -OPT_UNSET.add_option( '-v', action='store_true', dest='has_v', >> default=False) - -def builtin_unset(name, args, interp, env, stdin, >> stdout, stderr, debugflags): >> - if 'debug-utility' in debugflags: >> - print interp.log(' '.join([name, str(args), interp['PWD']]) >> + '\n') >> - >> - option, args = OPT_UNSET.parse_args(args) >> - >> - status = 0 >> - env = interp.get_env() >> - for arg in args: >> - try: >> - if option.has_f: >> - env.remove_function(arg) >> - else: >> - del env[arg] >> - except KeyError: >> - pass >> - except VarAssignmentError: >> - status = 1 >> - >> - return status >> - >> -#------------------------------------------------------------------------------- >> -# wait special builtin >> -#------------------------------------------------------------------------------- >> -def builtin_wait(name, args, interp, env, stdin, stdout, stderr, >> debugflags): >> - if 'debug-utility' in debugflags: >> - print interp.log(' '.join([name, str(args), interp['PWD']]) >> + '\n') - >> - return interp.wait([int(arg) for arg in args]) >> - >> -#------------------------------------------------------------------------------- >> -# cat utility >> -#------------------------------------------------------------------------------- >> -def utility_cat(name, args, interp, env, stdin, stdout, stderr, >> debugflags): >> - if 'debug-utility' in debugflags: >> - print interp.log(' '.join([name, str(args), interp['PWD']]) >> + '\n') - >> - if not args: >> - args = ['-'] >> - >> - status = 0 >> - for arg in args: >> - if arg == '-': >> - data = stdin.read() >> - else: >> - path = os.path.join(env['PWD'], arg) >> - try: >> - f = file(path, 'rb') >> - try: >> - data = f.read() >> - finally: >> - f.close() >> - except IOError as e: >> - if e.errno != errno.ENOENT: >> - raise >> - status = 1 >> - continue >> - stdout.write(data) >> - stdout.flush() >> - return status >> - >> -#------------------------------------------------------------------------------- >> -# cd utility >> -#------------------------------------------------------------------------------- >> -OPT_CD = NonExitingParser("cd - change the working directory") >> - >> -def utility_cd(name, args, interp, env, stdin, stdout, stderr, >> debugflags): >> - if 'debug-utility' in debugflags: >> - print interp.log(' '.join([name, str(args), interp['PWD']]) >> + '\n') - >> - option, args = OPT_CD.parse_args(args) >> - env = interp.get_env() >> - >> - directory = None >> - printdir = False >> - if not args: >> - home = env.get('HOME') >> - if home: >> - # Unspecified, do nothing >> - return 0 >> - else: >> - directory = home >> - elif len(args)==1: >> - directory = args[0] >> - if directory=='-': >> - if 'OLDPWD' not in env: >> - raise UtilityError("OLDPWD not set") >> - printdir = True >> - directory = env['OLDPWD'] >> - else: >> - raise UtilityError("too many arguments") >> - >> - curpath = None >> - # Absolute directories will be handled correctly by the >> os.path.join call. >> - if not directory.startswith('.') and not >> directory.startswith('..'): >> - cdpaths = env.get('CDPATH', '.').split(';') >> - for cdpath in cdpaths: >> - p = os.path.join(cdpath, directory) >> - if os.path.isdir(p): >> - curpath = p >> - break >> - >> - if curpath is None: >> - curpath = directory >> - curpath = os.path.join(env['PWD'], directory) >> - >> - env['OLDPWD'] = env['PWD'] >> - env['PWD'] = curpath >> - if printdir: >> - stdout.write('%s\n' % curpath) >> - return 0 >> - >> -#------------------------------------------------------------------------------- >> -# colon utility >> -#------------------------------------------------------------------------------- >> -def utility_colon(name, args, interp, env, stdin, stdout, stderr, >> debugflags): >> - if 'debug-utility' in debugflags: >> - print interp.log(' '.join([name, str(args), interp['PWD']]) >> + '\n') >> - return 0 >> - >> -#------------------------------------------------------------------------------- >> -# echo utility >> -#------------------------------------------------------------------------------- >> -def utility_echo(name, args, interp, env, stdin, stdout, stderr, >> debugflags): >> - if 'debug-utility' in debugflags: >> - print interp.log(' '.join([name, str(args), interp['PWD']]) >> + '\n') >> - >> - # Echo only takes arguments, no options. Use printf if you need >> fancy stuff. >> - output = ' '.join(args) + '\n' >> - stdout.write(output) >> - stdout.flush() >> - return 0 >> - >> -#------------------------------------------------------------------------------- >> -# egrep utility >> -#------------------------------------------------------------------------------- >> -# egrep is usually a shell script. >> -# Unfortunately, pysh does not support shell scripts *with >> arguments* right now, -# so the redirection is implemented here, >> assuming grep is available. -def utility_egrep(name, args, interp, >> env, stdin, stdout, stderr, debugflags): >> - if 'debug-utility' in debugflags: >> - print interp.log(' '.join([name, str(args), interp['PWD']]) >> + '\n') >> - >> - return run_command('grep', ['-E'] + args, interp, env, stdin, >> stdout, >> - stderr, debugflags) >> - >> -#------------------------------------------------------------------------------- >> -# env utility >> -#------------------------------------------------------------------------------- >> -def utility_env(name, args, interp, env, stdin, stdout, stderr, >> debugflags): >> - if 'debug-utility' in debugflags: >> - print interp.log(' '.join([name, str(args), interp['PWD']]) >> + '\n') >> - >> - if args and args[0]=='-i': >> - raise NotImplementedError('env: -i option is not >> implemented') >> - >> - i = 0 >> - for arg in args: >> - if '=' not in arg: >> - break >> - # Update the current environment >> - name, value = arg.split('=', 1) >> - env[name] = value >> - i += 1 >> - >> - if args[i:]: >> - # Find then execute the specified interpreter >> - utility = env.find_in_path(args[i]) >> - if not utility: >> - return 127 >> - args[i:i+1] = utility >> - name = args[i] >> - args = args[i+1:] >> - try: >> - return run_command(name, args, interp, env, stdin, >> stdout, stderr, >> - debugflags) >> - except UtilityError: >> - stderr.write('env: failed to execute %s' % ' >> '.join([name]+args)) >> - return 126 >> - else: >> - for pair in env.get_variables().iteritems(): >> - stdout.write('%s=%s\n' % pair) >> - return 0 >> - >> -#------------------------------------------------------------------------------- >> -# exit utility >> -#------------------------------------------------------------------------------- >> -def utility_exit(name, args, interp, env, stdin, stdout, stderr, >> debugflags): >> - if 'debug-utility' in debugflags: >> - print interp.log(' '.join([name, str(args), interp['PWD']]) >> + '\n') >> - >> - res = None >> - if args: >> - try: >> - res = int(args[0]) >> - except ValueError: >> - res = None >> - if not 0<=res<=255: >> - res = None >> - >> - if res is None: >> - # BUG: should be last executed command exit code >> - res = 0 >> - >> - raise ExitSignal(res) >> - >> -#------------------------------------------------------------------------------- >> -# fgrep utility >> -#------------------------------------------------------------------------------- >> -# see egrep >> -def utility_fgrep(name, args, interp, env, stdin, stdout, stderr, >> debugflags): >> - if 'debug-utility' in debugflags: >> - print interp.log(' '.join([name, str(args), interp['PWD']]) >> + '\n') >> - >> - return run_command('grep', ['-F'] + args, interp, env, stdin, >> stdout, >> - stderr, debugflags) >> - >> -#------------------------------------------------------------------------------- >> -# gunzip utility >> -#------------------------------------------------------------------------------- >> -# see egrep >> -def utility_gunzip(name, args, interp, env, stdin, stdout, stderr, >> debugflags): >> - if 'debug-utility' in debugflags: >> - print interp.log(' '.join([name, str(args), interp['PWD']]) >> + '\n') >> - >> - return run_command('gzip', ['-d'] + args, interp, env, stdin, >> stdout, >> - stderr, debugflags) >> - >> -#------------------------------------------------------------------------------- >> -# kill utility >> -#------------------------------------------------------------------------------- >> -def utility_kill(name, args, interp, env, stdin, stdout, stderr, >> debugflags): >> - if 'debug-utility' in debugflags: >> - print interp.log(' '.join([name, str(args), interp['PWD']]) >> + '\n') >> - >> - for arg in args: >> - pid = int(arg) >> - status = subprocess.call(['pskill', '/T', str(pid)], >> - shell=True, >> - stdout=subprocess.PIPE, >> - stderr=subprocess.PIPE) >> - # pskill is asynchronous, hence the stupid polling loop >> - while 1: >> - p = subprocess.Popen(['pslist', str(pid)], >> - shell=True, >> - stdout=subprocess.PIPE, >> - stderr=subprocess.STDOUT) >> - output = p.communicate()[0] >> - if ('process %d was not' % pid) in output: >> - break >> - time.sleep(1) >> - return status >> - >> -#------------------------------------------------------------------------------- >> -# mkdir utility >> -#------------------------------------------------------------------------------- >> -OPT_MKDIR = NonExitingParser("mkdir - make directories.") >> -OPT_MKDIR.add_option('-p', action='store_true', dest='has_p', >> default=False) - >> -def utility_mkdir(name, args, interp, env, stdin, stdout, stderr, >> debugflags): >> - if 'debug-utility' in debugflags: >> - print interp.log(' '.join([name, str(args), interp['PWD']]) >> + '\n') >> - >> - # TODO: implement umask >> - # TODO: implement proper utility error report >> - option, args = OPT_MKDIR.parse_args(args) >> - for arg in args: >> - path = os.path.join(env['PWD'], arg) >> - if option.has_p: >> - try: >> - os.makedirs(path) >> - except IOError as e: >> - if e.errno != errno.EEXIST: >> - raise >> - else: >> - os.mkdir(path) >> - return 0 >> - >> -#------------------------------------------------------------------------------- >> -# netstat utility >> -#------------------------------------------------------------------------------- >> -def utility_netstat(name, args, interp, env, stdin, stdout, stderr, >> debugflags): >> - # Do you really expect me to implement netstat ? >> - # This empty form is enough for Mercurial tests since it's >> - # supposed to generate nothing upon success. Faking this test >> - # is not a big deal either. >> - if 'debug-utility' in debugflags: >> - print interp.log(' '.join([name, str(args), interp['PWD']]) >> + '\n') >> - return 0 >> - >> -#------------------------------------------------------------------------------- >> -# pwd utility >> -#------------------------------------------------------------------------------- >> -OPT_PWD = NonExitingParser("pwd - return working directory name") >> -OPT_PWD.add_option('-L', action='store_true', dest='has_L', >> default=True, >> - help="""If the PWD environment variable contains an absolute >> pathname of \ >> - the current directory that does not contain the filenames dot or >> dot-dot, \ >> - pwd shall write this pathname to standard output. Otherwise, the >> -L option \ >> - shall behave as the -P option.""") >> -OPT_PWD.add_option('-P', action='store_true', dest='has_L', >> default=False, >> - help="""The absolute pathname written shall not contain >> filenames that, in \ >> - the context of the pathname, refer to files of type symbolic >> link.""") - >> -def utility_pwd(name, args, interp, env, stdin, stdout, stderr, >> debugflags): >> - if 'debug-utility' in debugflags: >> - print interp.log(' '.join([name, str(args), interp['PWD']]) >> + '\n') - >> - option, args = OPT_PWD.parse_args(args) >> - stdout.write('%s\n' % env['PWD']) >> - return 0 >> - >> -#------------------------------------------------------------------------------- >> -# printf utility >> -#------------------------------------------------------------------------------- >> -RE_UNESCAPE = re.compile(r'(\\x[a-zA-Z0-9]{2}|\\[0-7]{1,3}|\\.)') >> - >> -def utility_printf(name, args, interp, env, stdin, stdout, stderr, >> debugflags): >> - if 'debug-utility' in debugflags: >> - print interp.log(' '.join([name, str(args), interp['PWD']]) >> + '\n') >> - >> - def replace(m): >> - assert m.group() >> - g = m.group()[1:] >> - if g.startswith('x'): >> - return chr(int(g[1:], 16)) >> - if len(g) <= 3 and len([c for c in g if c in '01234567']) == >> len(g): >> - # Yay, an octal number >> - return chr(int(g, 8)) >> - return { >> - 'a': '\a', >> - 'b': '\b', >> - 'f': '\f', >> - 'n': '\n', >> - 'r': '\r', >> - 't': '\t', >> - 'v': '\v', >> - '\\': '\\', >> - }.get(g) >> - >> - # Convert escape sequences >> - format = re.sub(RE_UNESCAPE, replace, args[0]) >> - stdout.write(format % tuple(args[1:])) >> - return 0 >> - >> -#------------------------------------------------------------------------------- >> -# true utility >> -#------------------------------------------------------------------------------- >> -def utility_true(name, args, interp, env, stdin, stdout, stderr, >> debugflags): >> - if 'debug-utility' in debugflags: >> - print interp.log(' '.join([name, str(args), interp['PWD']]) >> + '\n') >> - return 0 >> - >> -#------------------------------------------------------------------------------- >> -# sed utility >> -#------------------------------------------------------------------------------- >> -RE_SED = re.compile(r'^s(.).*\1[a-zA-Z]*$') >> - >> -# cygwin sed fails with some expressions when they do not end with a >> single space. -# see unit tests for details. Interestingly, the same >> expressions works perfectly -# in cygwin shell. >> -def utility_sed(name, args, interp, env, stdin, stdout, stderr, >> debugflags): >> - if 'debug-utility' in debugflags: >> - print interp.log(' '.join([name, str(args), interp['PWD']]) >> + '\n') >> - >> - # Scan pattern arguments and append a space if necessary >> - for i in range(len(args)): >> - if not RE_SED.search(args[i]): >> - continue >> - args[i] = args[i] + ' ' >> - >> - return run_command(name, args, interp, env, stdin, stdout, >> - stderr, debugflags) >> - >> -#------------------------------------------------------------------------------- >> -# sleep utility >> -#------------------------------------------------------------------------------- >> -def utility_sleep(name, args, interp, env, stdin, stdout, stderr, >> debugflags): >> - if 'debug-utility' in debugflags: >> - print interp.log(' '.join([name, str(args), interp['PWD']]) >> + '\n') >> - time.sleep(int(args[0])) >> - return 0 >> - >> -#------------------------------------------------------------------------------- >> -# sort utility >> -#------------------------------------------------------------------------------- >> -OPT_SORT = NonExitingParser("sort - sort, merge, or sequence check >> text files") - >> -def utility_sort(name, args, interp, env, stdin, stdout, stderr, >> debugflags): - >> - def sort(path): >> - if path == '-': >> - lines = stdin.readlines() >> - else: >> - try: >> - f = file(path) >> - try: >> - lines = f.readlines() >> - finally: >> - f.close() >> - except IOError as e: >> - stderr.write(str(e) + '\n') >> - return 1 >> - >> - if lines and lines[-1][-1]!='\n': >> - lines[-1] = lines[-1] + '\n' >> - return lines >> - >> - if 'debug-utility' in debugflags: >> - print interp.log(' '.join([name, str(args), interp['PWD']]) >> + '\n') >> - >> - option, args = OPT_SORT.parse_args(args) >> - alllines = [] >> - >> - if len(args)<=0: >> - args += ['-'] >> - >> - # Load all files lines >> - curdir = os.getcwd() >> - try: >> - os.chdir(env['PWD']) >> - for path in args: >> - alllines += sort(path) >> - finally: >> - os.chdir(curdir) >> - >> - alllines.sort() >> - for line in alllines: >> - stdout.write(line) >> - return 0 >> - >> -#------------------------------------------------------------------------------- >> -# hg utility >> -#------------------------------------------------------------------------------- >> - >> -hgcommands = [ >> - 'add', >> - 'addremove', >> - 'commit', 'ci', >> - 'debugrename', >> - 'debugwalk', >> - 'falabala', # Dummy command used in a mercurial test >> - 'incoming', >> - 'locate', >> - 'pull', >> - 'push', >> - 'qinit', >> - 'remove', 'rm', >> - 'rename', 'mv', >> - 'revert', >> - 'showconfig', >> - 'status', 'st', >> - 'strip', >> - ] >> - >> -def rewriteslashes(name, args): >> - # Several hg commands output file paths, rewrite the separators >> - if len(args) > 1 and name.lower().endswith('python') \ >> - and args[0].endswith('hg'): >> - for cmd in hgcommands: >> - if cmd in args[1:]: >> - return True >> - >> - # svn output contains many paths with OS specific separators. >> - # Normalize these to unix paths. >> - base = os.path.basename(name) >> - if base.startswith('svn'): >> - return True >> - >> - return False >> - >> -def rewritehg(output): >> - if not output: >> - return output >> - # Rewrite os specific messages >> - output = output.replace(': The system cannot find the file >> specified', >> - ': No such file or directory') >> - output = re.sub(': Access is denied.*$', ': Permission denied', >> output) >> - output = output.replace(': No connection could be made because >> the target machine actively refused it', >> - ': Connection refused') >> - return output >> - >> - >> -def run_command(name, args, interp, env, stdin, stdout, >> - stderr, debugflags): >> - # Execute the command >> - if 'debug-utility' in debugflags: >> - print interp.log(' '.join([name, str(args), interp['PWD']]) >> + '\n') - >> - hgbin = interp.options().hgbinary >> - ishg = hgbin and ('hg' in name or args and 'hg' in args[0]) >> - unixoutput = 'cygwin' in name or ishg >> - >> - exec_env = env.get_variables() >> - try: >> - # BUG: comparing file descriptor is clearly not a reliable >> way to tell >> - # whether they point on the same underlying object. But in >> pysh limited >> - # scope this is usually right, we do not expect complicated >> redirections >> - # besides usual 2>&1. >> - # Still there is one case we have but cannot deal with is >> when stdout >> - # and stderr are redirected *by pysh caller*. This the >> reason for the >> - # --redirect pysh() option. >> - # Now, we want to know they are the same because we >> sometimes need to >> - # transform the command output, mostly remove CR-LF to >> ensure that >> - # command output is unix-like. Cygwin utilies are a special >> case because >> - # they explicitely set their output streams to binary mode, >> so we have >> - # nothing to do. For all others commands, we have to guess >> whether they >> - # are sending text data, in which case the transformation >> must be done. >> - # Again, the NUL character test is unreliable but should be >> enough for >> - # hg tests. >> - redirected = stdout.fileno()==stderr.fileno() >> - if not redirected: >> - p = subprocess.Popen([name] + args, cwd=env['PWD'], >> env=exec_env, >> - stdin=stdin, stdout=subprocess.PIPE, >> stderr=subprocess.PIPE) >> - else: >> - p = subprocess.Popen([name] + args, cwd=env['PWD'], >> env=exec_env, >> - stdin=stdin, stdout=subprocess.PIPE, >> stderr=subprocess.STDOUT) >> - out, err = p.communicate() >> - except WindowsError as e: >> - raise UtilityError(str(e)) >> - >> - if not unixoutput: >> - def encode(s): >> - if '\0' in s: >> - return s >> - return s.replace('\r\n', '\n') >> - else: >> - encode = lambda s: s >> - >> - if rewriteslashes(name, args): >> - encode1_ = encode >> - def encode(s): >> - s = encode1_(s) >> - s = s.replace('\\\\', '\\') >> - s = s.replace('\\', '/') >> - return s >> - >> - if ishg: >> - encode2_ = encode >> - def encode(s): >> - return rewritehg(encode2_(s)) >> - >> - stdout.write(encode(out)) >> - if not redirected: >> - stderr.write(encode(err)) >> - return p.returncode >> - >> diff --git a/bitbake/lib/bb/pysh/interp.py >> b/bitbake/lib/bb/pysh/interp.py deleted file mode 100644 >> index d14ecf3..0000000 >> --- a/bitbake/lib/bb/pysh/interp.py >> +++ /dev/null >> @@ -1,1367 +0,0 @@ >> -# interp.py - shell interpreter for pysh. >> -# >> -# Copyright 2007 Patrick Mezard >> -# >> -# This software may be used and distributed according to the terms >> -# of the GNU General Public License, incorporated herein by >> reference. - >> -"""Implement the shell interpreter. >> - >> -Most references are made to "The Open Group Base Specifications >> Issue 6". >> - >> -""" -# TODO: document the fact input streams must implement fileno() >> so Popen will work correctly. -# it requires non-stdin stream to be >> implemented as files. Still to be tested... -# DOC: pathsep is used >> in PATH instead of ':'. Clearly, there are path syntax issues here. >> -# TODO: stop command execution upon error. -# TODO: sort out the >> filename/io_number mess. It should be possible to use filenames only. >> -# TODO: review subshell implementation -# TODO: test environment >> cloning for non-special builtins -# TODO: set -x should not rebuild >> commands from tokens, assignments/redirections are lost -# TODO: unit >> test for variable assignment -# TODO: test error management wrt error >> type/utility type -# TODO: test for binary output everywhere >> -# BUG: debug-parsing does not pass log file to PLY. Maybe a PLY >> upgrade is necessary. -import base64 >> -import cPickle as pickle >> -import errno >> -import glob >> -import os >> -import re >> -import subprocess >> -import sys >> -import tempfile >> - >> -try: >> - s = set() >> - del s >> -except NameError: >> - from Set import Set as set >> - >> -import builtin >> -from sherrors import * >> -import pyshlex >> -import pyshyacc >> - >> -def mappend(func, *args, **kargs): >> - """Like map but assume func returns a list. Returned lists are >> merged into >> - a single one. >> - """ >> - return reduce(lambda a,b: a+b, map(func, *args, **kargs), []) >> - >> -class FileWrapper: >> - """File object wrapper to ease debugging. >> - >> - Allow mode checking and implement file duplication through a >> simple >> - reference counting scheme. Not sure the latter is really useful >> since >> - only real file descriptors can be used. >> - """ >> - def __init__(self, mode, file, close=True): >> - if mode not in ('r', 'w', 'a'): >> - raise IOError('invalid mode: %s' % mode) >> - self._mode = mode >> - self._close = close >> - if isinstance(file, FileWrapper): >> - if file._refcount[0] <= 0: >> - raise IOError(0, 'Error') >> - self._refcount = file._refcount >> - self._refcount[0] += 1 >> - self._file = file._file >> - else: >> - self._refcount = [1] >> - self._file = file >> - >> - def dup(self): >> - return FileWrapper(self._mode, self, self._close) >> - >> - def fileno(self): >> - """fileno() should be only necessary for input streams.""" >> - return self._file.fileno() >> - >> - def read(self, size=-1): >> - if self._mode!='r': >> - raise IOError(0, 'Error') >> - return self._file.read(size) >> - >> - def readlines(self, *args, **kwargs): >> - return self._file.readlines(*args, **kwargs) >> - >> - def write(self, s): >> - if self._mode not in ('w', 'a'): >> - raise IOError(0, 'Error') >> - return self._file.write(s) >> - >> - def flush(self): >> - self._file.flush() >> - >> - def close(self): >> - if not self._refcount: >> - return >> - assert self._refcount[0] > 0 >> - >> - self._refcount[0] -= 1 >> - if self._refcount[0] == 0: >> - self._mode = 'c' >> - if self._close: >> - self._file.close() >> - self._refcount = None >> - >> - def mode(self): >> - return self._mode >> - >> - def __getattr__(self, name): >> - if name == 'name': >> - self.name = getattr(self._file, name) >> - return self.name >> - else: >> - raise AttributeError(name) >> - >> - def __del__(self): >> - self.close() >> - >> - >> -def win32_open_devnull(mode): >> - return open('NUL', mode) >> - >> - >> -class Redirections: >> - """Stores open files and their mapping to pseudo-sh file >> descriptor. >> - """ >> - # BUG: redirections are not handled correctly: 1>&3 2>&3 3>&4 >> does >> - # not make 1 to redirect to 4 >> - def __init__(self, stdin=None, stdout=None, stderr=None): >> - self._descriptors = {} >> - if stdin is not None: >> - self._add_descriptor(0, stdin) >> - if stdout is not None: >> - self._add_descriptor(1, stdout) >> - if stderr is not None: >> - self._add_descriptor(2, stderr) >> - >> - def add_here_document(self, interp, name, content, >> io_number=None): >> - if io_number is None: >> - io_number = 0 >> - >> - if name==pyshlex.unquote_wordtree(name): >> - content = interp.expand_here_document(('TOKEN', content)) >> - >> - # Write document content in a temporary file >> - tmp = tempfile.TemporaryFile() >> - try: >> - tmp.write(content) >> - tmp.flush() >> - tmp.seek(0) >> - self._add_descriptor(io_number, FileWrapper('r', tmp)) >> - except: >> - tmp.close() >> - raise >> - >> - def add(self, interp, op, filename, io_number=None): >> - if op not in ('<', '>', '>|', '>>', '>&'): >> - # TODO: add descriptor duplication and >> here_documents >> - raise RedirectionError('Unsupported redirection operator >> "%s"' % op) >> - >> - if io_number is not None: >> - io_number = int(io_number) >> - >> - if (op == '>&' and filename.isdigit()) or filename=='-': >> - # No expansion for file descriptors, quote them if you >> want a filename >> - fullname = filename >> - else: >> - if filename.startswith('/'): >> - # TODO: win32 kludge >> - if filename=='/dev/null': >> - fullname = 'NUL' >> - else: >> - # TODO: handle absolute pathnames, they are >> unlikely to exist on the >> - # current platform (win32 for instance). >> - raise NotImplementedError() >> - else: >> - fullname = interp.expand_redirection(('TOKEN', >> filename)) >> - if not fullname: >> - raise RedirectionError('%s: ambiguous redirect' >> % filename) >> - # Build absolute path based on PWD >> - fullname = os.path.join(interp.get_env()['PWD'], >> fullname) >> - >> - if op=='<': >> - return self._add_input_redirection(interp, fullname, >> io_number) >> - elif op in ('>', '>|'): >> - clobber = ('>|'==op) >> - return self._add_output_redirection(interp, fullname, >> io_number, clobber) >> - elif op=='>>': >> - return self._add_output_appending(interp, fullname, >> io_number) >> - elif op=='>&': >> - return self._dup_output_descriptor(fullname, io_number) >> - >> - def close(self): >> - if self._descriptors is not None: >> - for desc in self._descriptors.itervalues(): >> - desc.flush() >> - desc.close() >> - self._descriptors = None >> - >> - def stdin(self): >> - return self._descriptors[0] >> - >> - def stdout(self): >> - return self._descriptors[1] >> - >> - def stderr(self): >> - return self._descriptors[2] >> - >> - def clone(self): >> - clone = Redirections() >> - for desc, fileobj in self._descriptors.iteritems(): >> - clone._descriptors[desc] = fileobj.dup() >> - return clone >> - >> - def _add_output_redirection(self, interp, filename, io_number, >> clobber): >> - if io_number is None: >> - # io_number default to standard output >> - io_number = 1 >> - >> - if not clobber and interp.get_env().has_opt('-C') and >> os.path.isfile(filename): >> - # File already exist in no-clobber mode, bail out >> - raise RedirectionError('File "%s" already exists' % >> filename) >> - >> - # Open and register >> - self._add_file_descriptor(io_number, filename, 'w') >> - >> - def _add_output_appending(self, interp, filename, io_number): >> - if io_number is None: >> - io_number = 1 >> - self._add_file_descriptor(io_number, filename, 'a') >> - >> - def _add_input_redirection(self, interp, filename, io_number): >> - if io_number is None: >> - io_number = 0 >> - self._add_file_descriptor(io_number, filename, 'r') >> - >> - def _add_file_descriptor(self, io_number, filename, mode): >> - try: >> - if filename.startswith('/'): >> - if filename=='/dev/null': >> - f = win32_open_devnull(mode+'b') >> - else: >> - # TODO: handle absolute pathnames, they are >> unlikely to exist on the >> - # current platform (win32 for instance). >> - raise NotImplementedError('cannot open absolute >> path %s' % repr(filename)) >> - else: >> - f = file(filename, mode+'b') >> - except IOError as e: >> - raise RedirectionError(str(e)) >> - >> - wrapper = None >> - try: >> - wrapper = FileWrapper(mode, f) >> - f = None >> - self._add_descriptor(io_number, wrapper) >> - except: >> - if f: f.close() >> - if wrapper: wrapper.close() >> - raise >> - >> - def _dup_output_descriptor(self, source_fd, dest_fd): >> - if source_fd is None: >> - source_fd = 1 >> - self._dup_file_descriptor(source_fd, dest_fd, 'w') >> - >> - def _dup_file_descriptor(self, source_fd, dest_fd, mode): >> - source_fd = int(source_fd) >> - if source_fd not in self._descriptors: >> - raise RedirectionError('"%s" is not a valid file >> descriptor' % str(source_fd)) >> - source = self._descriptors[source_fd] >> - >> - if source.mode()!=mode: >> - raise RedirectionError('Descriptor %s cannot be >> duplicated in mode "%s"' % (str(source), mode)) >> - >> - if dest_fd=='-': >> - # Close the source descriptor >> - del self._descriptors[source_fd] >> - source.close() >> - else: >> - dest_fd = int(dest_fd) >> - if dest_fd not in self._descriptors: >> - raise RedirectionError('Cannot replace file >> descriptor %s' % str(dest_fd)) >> - >> - dest = self._descriptors[dest_fd] >> - if dest.mode()!=mode: >> - raise RedirectionError('Descriptor %s cannot be >> cannot be redirected in mode "%s"' % (str(dest), mode)) >> - >> - self._descriptors[dest_fd] = source.dup() >> - dest.close() >> - >> - def _add_descriptor(self, io_number, file): >> - io_number = int(io_number) >> - >> - if io_number in self._descriptors: >> - # Close the current descriptor >> - d = self._descriptors[io_number] >> - del self._descriptors[io_number] >> - d.close() >> - >> - self._descriptors[io_number] = file >> - >> - def __str__(self): >> - names = [('%d=%r' % (k, getattr(v, 'name', None))) for k,v >> - in self._descriptors.iteritems()] >> - names = ','.join(names) >> - return 'Redirections(%s)' % names >> - >> - def __del__(self): >> - self.close() >> - >> -def cygwin_to_windows_path(path): >> - """Turn /cygdrive/c/foo into c:/foo, or return path if it >> - is not a cygwin path. >> - """ >> - if not path.startswith('/cygdrive/'): >> - return path >> - path = path[len('/cygdrive/'):] >> - path = path[:1] + ':' + path[1:] >> - return path >> - >> -def win32_to_unix_path(path): >> - if path is not None: >> - path = path.replace('\\', '/') >> - return path >> - >> -_RE_SHEBANG = re.compile(r'^\#!\s?([^\s]+)(?:\s([^\s]+))?') >> -_SHEBANG_CMDS = { >> - '/usr/bin/env': 'env', >> - '/bin/sh': 'pysh', >> - 'python': 'python', >> -} >> - >> -def resolve_shebang(path, ignoreshell=False): >> - """Return a list of arguments as shebang interpreter call or an >> empty list >> - if path does not refer to an executable script. >> - See . >> - >> - ignoreshell - set to True to ignore sh shebangs. Return an empty >> list instead. >> - """ >> - try: >> - f = file(path) >> - try: >> - # At most 80 characters in the first line >> - header = f.read(80).splitlines()[0] >> - finally: >> - f.close() >> - >> - m = _RE_SHEBANG.search(header) >> - if not m: >> - return [] >> - cmd, arg = m.group(1,2) >> - if os.path.isfile(cmd): >> - # Keep this one, the hg script for instance contains a >> weird windows >> - # shebang referencing the current python install. >> - cmdfile = os.path.basename(cmd).lower() >> - if cmdfile == 'python.exe': >> - cmd = 'python' >> - pass >> - elif cmd not in _SHEBANG_CMDS: >> - raise CommandNotFound('Unknown interpreter "%s" >> referenced in '\ >> - 'shebang' % header) >> - cmd = _SHEBANG_CMDS.get(cmd) >> - if cmd is None or (ignoreshell and cmd == 'pysh'): >> - return [] >> - if arg is None: >> - return [cmd, win32_to_unix_path(path)] >> - return [cmd, arg, win32_to_unix_path(path)] >> - except IOError as e: >> - if e.errno!=errno.ENOENT and \ >> - (e.errno!=errno.EPERM and not os.path.isdir(path)): # >> Opening a directory raises EPERM >> - raise >> - return [] >> - >> -def win32_find_in_path(name, path): >> - if isinstance(path, str): >> - path = path.split(os.pathsep) >> - >> - exts = os.environ.get('PATHEXT', '').lower().split(os.pathsep) >> - for p in path: >> - p_name = os.path.join(p, name) >> - >> - prefix = resolve_shebang(p_name) >> - if prefix: >> - return prefix >> - >> - for ext in exts: >> - p_name_ext = p_name + ext >> - if os.path.exists(p_name_ext): >> - return [win32_to_unix_path(p_name_ext)] >> - return [] >> - >> -class Traps(dict): >> - def __setitem__(self, key, value): >> - if key not in ('EXIT',): >> - raise NotImplementedError() >> - super(Traps, self).__setitem__(key, value) >> - >> -# IFS white spaces character class >> -_IFS_WHITESPACES = (' ', '\t', '\n') >> - >> -class Environment: >> - """Environment holds environment variables, export table, >> function >> - definitions and whatever is defined in 2.12 "Shell Execution >> Environment", >> - redirection excepted. >> - """ >> - def __init__(self, pwd): >> - self._opt = set() #Shell options >> - >> - self._functions = {} >> - self._env = {'?': '0', '#': '0'} >> - self._exported = set([ >> - 'HOME', 'IFS', 'PATH' >> - ]) >> - >> - # Set environment vars with side-effects >> - self._ifs_ws = None # Set of IFS whitespace characters >> - self._ifs_re = None # Regular expression used to split >> between words using IFS classes >> - self['IFS'] = ''.join(_IFS_WHITESPACES) #Default environment >> values >> - self['PWD'] = pwd >> - self.traps = Traps() >> - >> - def clone(self, subshell=False): >> - env = Environment(self['PWD']) >> - env._opt = set(self._opt) >> - for k,v in self.get_variables().iteritems(): >> - if k in self._exported: >> - env.export(k,v) >> - elif subshell: >> - env[k] = v >> - >> - if subshell: >> - env._functions = dict(self._functions) >> - >> - return env >> - >> - def __getitem__(self, key): >> - if key in ('@', '*', '-', '$'): >> - raise NotImplementedError('%s is not implemented' % >> repr(key)) >> - return self._env[key] >> - >> - def get(self, key, defval=None): >> - try: >> - return self[key] >> - except KeyError: >> - return defval >> - >> - def __setitem__(self, key, value): >> - if key=='IFS': >> - # Update the whitespace/non-whitespace classes >> - self._update_ifs(value) >> - elif key=='PWD': >> - pwd = os.path.abspath(value) >> - if not os.path.isdir(pwd): >> - raise VarAssignmentError('Invalid directory %s' % >> value) >> - value = pwd >> - elif key in ('?', '!'): >> - value = str(int(value)) >> - self._env[key] = value >> - >> - def __delitem__(self, key): >> - if key in ('IFS', 'PWD', '?'): >> - raise VarAssignmentError('%s cannot be unset' % key) >> - del self._env[key] >> - >> - def __contains__(self, item): >> - return item in self._env >> - >> - def set_positional_args(self, args): >> - """Set the content of 'args' as positional argument from 1 >> to len(args). >> - Return previous argument as a list of strings. >> - """ >> - # Save and remove previous arguments >> - prevargs = [] >> - for i in range(int(self._env['#'])): >> - i = str(i+1) >> - prevargs.append(self._env[i]) >> - del self._env[i] >> - self._env['#'] = '0' >> - >> - #Set new ones >> - for i,arg in enumerate(args): >> - self._env[str(i+1)] = str(arg) >> - self._env['#'] = str(len(args)) >> - >> - return prevargs >> - >> - def get_positional_args(self): >> - return [self._env[str(i+1)] for i in >> range(int(self._env['#']))] >> - >> - def get_variables(self): >> - return dict(self._env) >> - >> - def export(self, key, value=None): >> - if value is not None: >> - self[key] = value >> - self._exported.add(key) >> - >> - def get_exported(self): >> - return [(k,self._env.get(k)) for k in self._exported] >> - >> - def split_fields(self, word): >> - if not self._ifs_ws or not word: >> - return [word] >> - return re.split(self._ifs_re, word) >> - >> - def _update_ifs(self, value): >> - """Update the split_fields related variables when IFS >> character set is >> - changed. >> - """ >> - # TODO: handle NULL IFS >> - >> - # Separate characters in whitespace and non-whitespace >> - chars = set(value) >> - ws = [c for c in chars if c in _IFS_WHITESPACES] >> - nws = [c for c in chars if c not in _IFS_WHITESPACES] >> - >> - # Keep whitespaces in a string for left and right stripping >> - self._ifs_ws = ''.join(ws) >> - >> - # Build a regexp to split fields >> - trailing = '[' + ''.join([re.escape(c) for c in ws]) + ']' >> - if nws: >> - # First, the single non-whitespace occurence. >> - nws = '[' + ''.join([re.escape(c) for c in nws]) + ']' >> - nws = '(?:' + trailing + '*' + nws + trailing + '*' + >> '|' + trailing + '+)' >> - else: >> - # Then mix all parts with quantifiers >> - nws = trailing + '+' >> - self._ifs_re = re.compile(nws) >> - >> - def has_opt(self, opt, val=None): >> - return (opt, val) in self._opt >> - >> - def set_opt(self, opt, val=None): >> - self._opt.add((opt, val)) >> - >> - def find_in_path(self, name, pwd=False): >> - path = self._env.get('PATH', '').split(os.pathsep) >> - if pwd: >> - path[:0] = [self['PWD']] >> - if os.name == 'nt': >> - return win32_find_in_path(name, self._env.get('PATH', >> '')) >> - else: >> - raise NotImplementedError() >> - >> - def define_function(self, name, body): >> - if not is_name(name): >> - raise ShellSyntaxError('%s is not a valid function name' >> % repr(name)) >> - self._functions[name] = body >> - >> - def remove_function(self, name): >> - del self._functions[name] >> - >> - def is_function(self, name): >> - return name in self._functions >> - >> - def get_function(self, name): >> - return self._functions.get(name) >> - >> - >> -name_charset = >> 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_' >> -name_charset = dict(zip(name_charset,name_charset)) >> - >> -def match_name(s): >> - """Return the length in characters of the longest prefix made of >> name >> - allowed characters in s. >> - """ >> - for i,c in enumerate(s): >> - if c not in name_charset: >> - return s[:i] >> - return s >> - >> -def is_name(s): >> - return len([c for c in s if c not in name_charset])<=0 >> - >> -def is_special_param(c): >> - return len(c)==1 and c in ('@','*','#','?','-','$','!','0') >> - >> -def utility_not_implemented(name, *args, **kwargs): >> - raise NotImplementedError('%s utility is not implemented' % name) >> - >> - >> -class Utility: >> - """Define utilities properties: >> - func -- utility callable. See builtin module for utility samples. >> - is_special -- see XCU 2.8. >> - """ >> - def __init__(self, func, is_special=0): >> - self.func = func >> - self.is_special = bool(is_special) >> - >> - >> -def encodeargs(args): >> - def encodearg(s): >> - lines = base64.encodestring(s) >> - lines = [l.splitlines()[0] for l in lines] >> - return ''.join(lines) >> - >> - s = pickle.dumps(args) >> - return encodearg(s) >> - >> -def decodeargs(s): >> - s = base64.decodestring(s) >> - return pickle.loads(s) >> - >> - >> -class GlobError(Exception): >> - pass >> - >> -class Options: >> - def __init__(self): >> - # True if Mercurial operates with binary streams >> - self.hgbinary = True >> - >> -class Interpreter: >> - # Implementation is very basic: the execute() method just makes >> a DFS on the >> - # AST and execute nodes one by one. Nodes are tuple (name,obj) >> where name >> - # is a string identifier and obj the AST element returned by the >> parser. >> - # >> - # Handler are named after the node identifiers. >> - # TODO: check node names and remove the switch in execute with >> some >> - # dynamic getattr() call to find node handlers. >> - """Shell interpreter. >> - >> - The following debugging flags can be passed: >> - debug-parsing - enable PLY debugging. >> - debug-tree - print the generated AST. >> - debug-cmd - trace command execution before word expansion, plus >> exit status. >> - debug-utility - trace utility execution. >> - """ >> - >> - # List supported commands. >> - COMMANDS = { >> - 'cat': Utility(builtin.utility_cat,), >> - 'cd': Utility(builtin.utility_cd,), >> - ':': Utility(builtin.utility_colon,), >> - 'echo': Utility(builtin.utility_echo), >> - 'env': Utility(builtin.utility_env), >> - 'exit': Utility(builtin.utility_exit), >> - 'export': Utility(builtin.builtin_export, >> is_special=1), >> - 'egrep': Utility(builtin.utility_egrep), >> - 'fgrep': Utility(builtin.utility_fgrep), >> - 'gunzip': Utility(builtin.utility_gunzip), >> - 'kill': Utility(builtin.utility_kill), >> - 'mkdir': Utility(builtin.utility_mkdir), >> - 'netstat': Utility(builtin.utility_netstat), >> - 'printf': Utility(builtin.utility_printf), >> - 'pwd': Utility(builtin.utility_pwd), >> - 'return': Utility(builtin.builtin_return, >> is_special=1), >> - 'sed': Utility(builtin.utility_sed,), >> - 'set': Utility(builtin.builtin_set,), >> - 'shift': Utility(builtin.builtin_shift,), >> - 'sleep': Utility(builtin.utility_sleep,), >> - 'sort': Utility(builtin.utility_sort,), >> - 'trap': Utility(builtin.builtin_trap, >> is_special=1), >> - 'true': Utility(builtin.utility_true), >> - 'unset': Utility(builtin.builtin_unset, >> is_special=1), >> - 'wait': Utility(builtin.builtin_wait, >> is_special=1), >> - } >> - >> - def __init__(self, pwd, debugflags = [], env=None, redirs=None, >> stdin=None, >> - stdout=None, stderr=None, opts=Options()): >> - self._env = env >> - if self._env is None: >> - self._env = Environment(pwd) >> - self._children = {} >> - >> - self._redirs = redirs >> - self._close_redirs = False >> - >> - if self._redirs is None: >> - if stdin is None: >> - stdin = sys.stdin >> - if stdout is None: >> - stdout = sys.stdout >> - if stderr is None: >> - stderr = sys.stderr >> - stdin = FileWrapper('r', stdin, False) >> - stdout = FileWrapper('w', stdout, False) >> - stderr = FileWrapper('w', stderr, False) >> - self._redirs = Redirections(stdin, stdout, stderr) >> - self._close_redirs = True >> - >> - self._debugflags = list(debugflags) >> - self._logfile = sys.stderr >> - self._options = opts >> - >> - def close(self): >> - """Must be called when the interpreter is no longer used.""" >> - script = self._env.traps.get('EXIT') >> - if script: >> - try: >> - self.execute_script(script=script) >> - except: >> - pass >> - >> - if self._redirs is not None and self._close_redirs: >> - self._redirs.close() >> - self._redirs = None >> - >> - def log(self, s): >> - self._logfile.write(s) >> - self._logfile.flush() >> - >> - def __getitem__(self, key): >> - return self._env[key] >> - >> - def __setitem__(self, key, value): >> - self._env[key] = value >> - >> - def options(self): >> - return self._options >> - >> - def redirect(self, redirs, ios): >> - def add_redir(io): >> - if isinstance(io, pyshyacc.IORedirect): >> - redirs.add(self, io.op, io.filename, io.io_number) >> - else: >> - redirs.add_here_document(self, io.name, io.content, >> io.io_number) >> - >> - map(add_redir, ios) >> - return redirs >> - >> - def execute_script(self, script=None, ast=None, sourced=False, >> - scriptpath=None): >> - """If script is not None, parse the input. Otherwise takes >> the supplied >> - AST. Then execute the AST. >> - Return the script exit status. >> - """ >> - try: >> - if scriptpath is not None: >> - self._env['0'] = os.path.abspath(scriptpath) >> - >> - if script is not None: >> - debug_parsing = ('debug-parsing' in >> self._debugflags) >> - cmds, script = pyshyacc.parse(script, True, >> debug_parsing) >> - if 'debug-tree' in self._debugflags: >> - pyshyacc.print_commands(cmds, self._logfile) >> - self._logfile.flush() >> - else: >> - cmds, script = ast, '' >> - >> - status = 0 >> - for cmd in cmds: >> - try: >> - status = self.execute(cmd) >> - except ExitSignal as e: >> - if sourced: >> - raise >> - status = int(e.args[0]) >> - return status >> - except ShellError: >> - self._env['?'] = 1 >> - raise >> - if 'debug-utility' in self._debugflags or >> 'debug-cmd' in self._debugflags: >> - self.log('returncode ' + str(status)+ '\n') >> - return status >> - except CommandNotFound as e: >> - print >>self._redirs.stderr, str(e) >> - self._redirs.stderr.flush() >> - # Command not found by non-interactive shell >> - # return 127 >> - raise >> - except RedirectionError as e: >> - # TODO: should be handled depending on the utility status >> - print >>self._redirs.stderr, str(e) >> - self._redirs.stderr.flush() >> - # Command not found by non-interactive shell >> - # return 127 >> - raise >> - >> - def dotcommand(self, env, args): >> - if len(args) < 1: >> - raise ShellError('. expects at least one argument') >> - path = args[0] >> - if '/' not in path: >> - found = env.find_in_path(args[0], True) >> - if found: >> - path = found[0] >> - script = file(path).read() >> - return self.execute_script(script=script, sourced=True) >> - >> - def execute(self, token, redirs=None): >> - """Execute and AST subtree with supplied redirections >> overriding default >> - interpreter ones. >> - Return the exit status. >> - """ >> - if not token: >> - return 0 >> - >> - if redirs is None: >> - redirs = self._redirs >> - >> - if isinstance(token, list): >> - # Commands sequence >> - res = 0 >> - for t in token: >> - res = self.execute(t, redirs) >> - return res >> - >> - type, value = token >> - status = 0 >> - if type=='simple_command': >> - redirs_copy = redirs.clone() >> - try: >> - # TODO: define and handle command return values >> - # TODO: implement set -e >> - status = self._execute_simple_command(value, >> redirs_copy) >> - finally: >> - redirs_copy.close() >> - elif type=='pipeline': >> - status = self._execute_pipeline(value, redirs) >> - elif type=='and_or': >> - status = self._execute_and_or(value, redirs) >> - elif type=='for_clause': >> - status = self._execute_for_clause(value, redirs) >> - elif type=='while_clause': >> - status = self._execute_while_clause(value, redirs) >> - elif type=='function_definition': >> - status = self._execute_function_definition(value, redirs) >> - elif type=='brace_group': >> - status = self._execute_brace_group(value, redirs) >> - elif type=='if_clause': >> - status = self._execute_if_clause(value, redirs) >> - elif type=='subshell': >> - status = self.subshell(ast=value.cmds, redirs=redirs) >> - elif type=='async': >> - status = self._asynclist(value) >> - elif type=='redirect_list': >> - redirs_copy = self.redirect(redirs.clone(), value.redirs) >> - try: >> - status = self.execute(value.cmd, redirs_copy) >> - finally: >> - redirs_copy.close() >> - else: >> - raise NotImplementedError('Unsupported token type ' + >> type) - >> - if status < 0: >> - status = 255 >> - return status >> - >> - def _execute_if_clause(self, if_clause, redirs): >> - cond_status = self.execute(if_clause.cond, redirs) >> - if cond_status==0: >> - return self.execute(if_clause.if_cmds, redirs) >> - else: >> - return self.execute(if_clause.else_cmds, redirs) >> - >> - def _execute_brace_group(self, group, redirs): >> - status = 0 >> - for cmd in group.cmds: >> - status = self.execute(cmd, redirs) >> - return status >> - >> - def _execute_function_definition(self, fundef, redirs): >> - self._env.define_function(fundef.name, fundef.body) >> - return 0 >> - >> - def _execute_while_clause(self, while_clause, redirs): >> - status = 0 >> - while 1: >> - cond_status = 0 >> - for cond in while_clause.condition: >> - cond_status = self.execute(cond, redirs) >> - >> - if cond_status: >> - break >> - >> - for cmd in while_clause.cmds: >> - status = self.execute(cmd, redirs) >> - >> - return status >> - >> - def _execute_for_clause(self, for_clause, redirs): >> - if not is_name(for_clause.name): >> - raise ShellSyntaxError('%s is not a valid name' % >> repr(for_clause.name)) >> - items = mappend(self.expand_token, for_clause.items) >> - >> - status = 0 >> - for item in items: >> - self._env[for_clause.name] = item >> - for cmd in for_clause.cmds: >> - status = self.execute(cmd, redirs) >> - return status >> - >> - def _execute_and_or(self, or_and, redirs): >> - res = self.execute(or_and.left, redirs) >> - if (or_and.op=='&&' and res==0) or (or_and.op!='&&' and >> res!=0): >> - res = self.execute(or_and.right, redirs) >> - return res >> - >> - def _execute_pipeline(self, pipeline, redirs): >> - if len(pipeline.commands)==1: >> - status = self.execute(pipeline.commands[0], redirs) >> - else: >> - # Execute all commands one after the other >> - status = 0 >> - inpath, outpath = None, None >> - try: >> - # Commands inputs and outputs cannot really be >> plugged as done >> - # by a real shell. Run commands sequentially and >> chain their >> - # input/output throught temporary files. >> - tmpfd, inpath = tempfile.mkstemp() >> - os.close(tmpfd) >> - tmpfd, outpath = tempfile.mkstemp() >> - os.close(tmpfd) >> - >> - inpath = win32_to_unix_path(inpath) >> - outpath = win32_to_unix_path(outpath) >> - >> - for i, cmd in enumerate(pipeline.commands): >> - call_redirs = redirs.clone() >> - try: >> - if i!=0: >> - call_redirs.add(self, '<', inpath) >> - if i!=len(pipeline.commands)-1: >> - call_redirs.add(self, '>', outpath) >> - >> - status = self.execute(cmd, call_redirs) >> - >> - # Chain inputs/outputs >> - inpath, outpath = outpath, inpath >> - finally: >> - call_redirs.close() >> - finally: >> - if inpath: os.remove(inpath) >> - if outpath: os.remove(outpath) >> - >> - if pipeline.reverse_status: >> - status = int(not status) >> - self._env['?'] = status >> - return status >> - >> - def _execute_function(self, name, args, interp, env, stdin, >> stdout, stderr, *others): >> - assert interp is self >> - >> - func = env.get_function(name) >> - #Set positional parameters >> - prevargs = None >> - try: >> - prevargs = env.set_positional_args(args) >> - try: >> - redirs = Redirections(stdin.dup(), stdout.dup(), >> stderr.dup()) >> - try: >> - status = self.execute(func, redirs) >> - finally: >> - redirs.close() >> - except ReturnSignal as e: >> - status = int(e.args[0]) >> - env['?'] = status >> - return status >> - finally: >> - #Reset positional parameters >> - if prevargs is not None: >> - env.set_positional_args(prevargs) >> - >> - def _execute_simple_command(self, token, redirs): >> - """Can raise ReturnSignal when return builtin is called, >> ExitSignal when >> - exit is called, and other shell exceptions upon builtin >> failures. >> - """ >> - debug_command = 'debug-cmd' in self._debugflags >> - if debug_command: >> - self.log('word' + repr(token.words) + '\n') >> - self.log('assigns' + repr(token.assigns) + '\n') >> - self.log('redirs' + repr(token.redirs) + '\n') >> - >> - is_special = None >> - env = self._env >> - >> - try: >> - # Word expansion >> - args = [] >> - for word in token.words: >> - args += self.expand_token(word) >> - if is_special is None and args: >> - is_special = env.is_function(args[0]) or \ >> - (args[0] in self.COMMANDS and >> self.COMMANDS[args[0]].is_special) >> - >> - if debug_command: >> - self.log('_execute_simple_command' + str(args) + >> '\n') >> - >> - if not args: >> - # Redirections happen is a subshell >> - redirs = redirs.clone() >> - elif not is_special: >> - env = self._env.clone() >> - >> - # Redirections >> - self.redirect(redirs, token.redirs) >> - >> - # Variables assignments >> - res = 0 >> - for type,(k,v) in token.assigns: >> - status, expanded = self.expand_variable((k,v)) >> - if status is not None: >> - res = status >> - if args: >> - env.export(k, expanded) >> - else: >> - env[k] = expanded >> - >> - if args and args[0] in ('.', 'source'): >> - res = self.dotcommand(env, args[1:]) >> - elif args: >> - if args[0] in self.COMMANDS: >> - command = self.COMMANDS[args[0]] >> - elif env.is_function(args[0]): >> - command = Utility(self._execute_function, >> is_special=True) >> - else: >> - if not '/' in args[0].replace('\\', '/'): >> - cmd = env.find_in_path(args[0]) >> - if not cmd: >> - # TODO: test error code on unknown >> command => 127 >> - raise CommandNotFound('Unknown command: >> "%s"' % args[0]) >> - else: >> - # Handle commands like '/cygdrive/c/foo.bat' >> - cmd = cygwin_to_windows_path(args[0]) >> - if not os.path.exists(cmd): >> - raise CommandNotFound('%s: No such file >> or directory' % args[0]) >> - shebang = resolve_shebang(cmd) >> - if shebang: >> - cmd = shebang >> - else: >> - cmd = [cmd] >> - args[0:1] = cmd >> - command = Utility(builtin.run_command) >> - >> - # Command execution >> - if 'debug-cmd' in self._debugflags: >> - self.log('redirections ' + str(redirs) + '\n') >> - >> - res = command.func(args[0], args[1:], self, env, >> - redirs.stdin(), redirs.stdout(), >> - redirs.stderr(), self._debugflags) >> - >> - if self._env.has_opt('-x'): >> - # Trace command execution in shell environment >> - # BUG: would be hard to reproduce a real shell >> behaviour since >> - # the AST is not annotated with source lines/tokens. >> - self._redirs.stdout().write(' '.join(args)) >> - >> - except ReturnSignal: >> - raise >> - except ShellError as e: >> - if is_special or isinstance(e, (ExitSignal, >> - ShellSyntaxError, >> ExpansionError)): >> - raise e >> - self._redirs.stderr().write(str(e)+'\n') >> - return 1 >> - >> - return res >> - >> - def expand_token(self, word): >> - """Expand a word as specified in [2.6 Word Expansions]. >> Return the list >> - of expanded words. >> - """ >> - status, wtrees = self._expand_word(word) >> - return map(pyshlex.wordtree_as_string, wtrees) >> - >> - def expand_variable(self, word): >> - """Return a status code (or None if no command expansion >> occurred) >> - and a single word. >> - """ >> - status, wtrees = self._expand_word(word, pathname=False, >> split=False) >> - words = map(pyshlex.wordtree_as_string, wtrees) >> - assert len(words)==1 >> - return status, words[0] >> - >> - def expand_here_document(self, word): >> - """Return the expanded document as a single word. The here >> document is >> - assumed to be unquoted. >> - """ >> - status, wtrees = self._expand_word(word, pathname=False, >> - split=False, >> here_document=True) >> - words = map(pyshlex.wordtree_as_string, wtrees) >> - assert len(words)==1 >> - return words[0] >> - >> - def expand_redirection(self, word): >> - """Return a single word.""" >> - return self.expand_variable(word)[1] >> - >> - def get_env(self): >> - return self._env >> - >> - def _expand_word(self, token, pathname=True, split=True, >> here_document=False): >> - wtree = pyshlex.make_wordtree(token[1], >> here_document=here_document) >> - >> - # TODO: implement tilde expansion >> - def expand(wtree): >> - """Return a pseudo wordtree: the tree or its subelements >> can be empty >> - lists when no value result from the expansion. >> - """ >> - status = None >> - for part in wtree: >> - if not isinstance(part, list): >> - continue >> - if part[0]in ("'", '\\'): >> - continue >> - elif part[0] in ('`', '$('): >> - status, result = self._expand_command(part) >> - part[:] = result >> - elif part[0] in ('$', '${'): >> - part[:] = self._expand_parameter(part, >> wtree[0]=='"', split) >> - elif part[0] in ('', '"'): >> - status, result = expand(part) >> - part[:] = result >> - else: >> - raise NotImplementedError('%s expansion is not >> implemented' >> - % part[0]) >> - # [] is returned when an expansion result in no-field, >> - # like an empty $@ >> - wtree = [p for p in wtree if p != []] >> - if len(wtree) < 3: >> - return status, [] >> - return status, wtree >> - >> - status, wtree = expand(wtree) >> - if len(wtree) == 0: >> - return status, wtree >> - wtree = pyshlex.normalize_wordtree(wtree) >> - >> - if split: >> - wtrees = self._split_fields(wtree) >> - else: >> - wtrees = [wtree] >> - >> - if pathname: >> - wtrees = mappend(self._expand_pathname, wtrees) >> - >> - wtrees = map(self._remove_quotes, wtrees) >> - return status, wtrees >> - >> - def _expand_command(self, wtree): >> - # BUG: there is something to do with backslashes and quoted >> - # characters here >> - command = pyshlex.wordtree_as_string(wtree[1:-1]) >> - status, output = self.subshell_output(command) >> - return status, ['', output, ''] >> - >> - def _expand_parameter(self, wtree, quoted=False, split=False): >> - """Return a valid wtree or an empty list when no parameter >> results.""" >> - # Get the parameter name >> - # TODO: implement weird expansion rules with ':' >> - name = pyshlex.wordtree_as_string(wtree[1:-1]) >> - if not is_name(name) and not is_special_param(name): >> - raise ExpansionError('Bad substitution "%s"' % name) >> - # TODO: implement special parameters >> - if name in ('@', '*'): >> - args = self._env.get_positional_args() >> - if len(args) == 0: >> - return [] >> - if len(args)<2: >> - return ['', ''.join(args), ''] >> - >> - sep = self._env.get('IFS', '')[:1] >> - if split and quoted and name=='@': >> - # Introduce a new token to tell the caller that >> these parameters >> - # cause a split as specified in 2.5.2 >> - return ['@'] + args + [''] >> - else: >> - return ['', sep.join(args), ''] >> - >> - return ['', self._env.get(name, ''), ''] >> - >> - def _split_fields(self, wtree): >> - def is_empty(split): >> - return split==['', '', ''] >> - >> - def split_positional(quoted): >> - # Return a list of wtree split according positional >> parameters rules. >> - # All remaining '@' groups are removed. >> - assert quoted[0]=='"' >> - >> - splits = [[]] >> - for part in quoted: >> - if not isinstance(part, list) or part[0]!='@': >> - splits[-1].append(part) >> - else: >> - # Empty or single argument list were dealt with >> already >> - assert len(part)>3 >> - # First argument must join with the beginning >> part of the original word >> - splits[-1].append(part[1]) >> - # Create double-quotes expressions for every >> argument after the first >> - for arg in part[2:-1]: >> - splits[-1].append('"') >> - splits.append(['"', arg]) >> - return splits >> - >> - # At this point, all expansions but pathnames have occured. >> Only quoted >> - # and positional sequences remain. Thus, all candidates for >> field splitting >> - # are in the tree root, or are positional splits ('@') and >> lie in root >> - # children. >> - if not wtree or wtree[0] not in ('', '"'): >> - # The whole token is quoted or empty, nothing to split >> - return [wtree] >> - >> - if wtree[0]=='"': >> - wtree = ['', wtree, ''] >> - >> - result = [['', '']] >> - for part in wtree[1:-1]: >> - if isinstance(part, list): >> - if part[0]=='"': >> - splits = split_positional(part) >> - if len(splits)<=1: >> - result[-1] += [part, ''] >> - else: >> - # Terminate the current split >> - result[-1] += [splits[0], ''] >> - result += splits[1:-1] >> - # Create a new split >> - result += [['', splits[-1], '']] >> - else: >> - result[-1] += [part, ''] >> - else: >> - splits = self._env.split_fields(part) >> - if len(splits)<=1: >> - # No split >> - result[-1][-1] += part >> - else: >> - # Terminate the current resulting part and >> create a new one >> - result[-1][-1] += splits[0] >> - result[-1].append('') >> - result += [['', r, ''] for r in splits[1:-1]] >> - result += [['', splits[-1]]] >> - result[-1].append('') >> - >> - # Leading and trailing empty groups come from >> leading/trailing blanks >> - if result and is_empty(result[-1]): >> - result[-1:] = [] >> - if result and is_empty(result[0]): >> - result[:1] = [] >> - return result >> - >> - def _expand_pathname(self, wtree): >> - """See [2.6.6 Pathname Expansion].""" >> - if self._env.has_opt('-f'): >> - return [wtree] >> - >> - # All expansions have been performed, only quoted sequences >> should remain >> - # in the tree. Generate the pattern by folding the tree, >> escaping special >> - # characters when appear quoted >> - special_chars = '*?[]' >> - >> - def make_pattern(wtree): >> - subpattern = [] >> - for part in wtree[1:-1]: >> - if isinstance(part, list): >> - part = make_pattern(part) >> - elif wtree[0]!='': >> - for c in part: >> - # Meta-characters cannot be quoted >> - if c in special_chars: >> - raise GlobError() >> - subpattern.append(part) >> - return ''.join(subpattern) >> - >> - def pwd_glob(pattern): >> - cwd = os.getcwd() >> - os.chdir(self._env['PWD']) >> - try: >> - return glob.glob(pattern) >> - finally: >> - os.chdir(cwd) >> - >> - #TODO: check working directory issues here wrt relative >> patterns >> - try: >> - pattern = make_pattern(wtree) >> - paths = pwd_glob(pattern) >> - except GlobError: >> - # BUG: Meta-characters were found in quoted sequences. >> The should >> - # have been used literally but this is unsupported in >> current glob module. >> - # Instead we consider the whole tree must be used >> literally and >> - # therefore there is no point in globbing. This is wrong >> when meta >> - # characters are mixed with quoted meta in the same >> pattern like: >> - # < foo*"py*" > >> - paths = [] >> - >> - if not paths: >> - return [wtree] >> - return [['', path, ''] for path in paths] >> - >> - def _remove_quotes(self, wtree): >> - """See [2.6.7 Quote Removal].""" >> - >> - def unquote(wtree): >> - unquoted = [] >> - for part in wtree[1:-1]: >> - if isinstance(part, list): >> - part = unquote(part) >> - unquoted.append(part) >> - return ''.join(unquoted) >> - >> - return ['', unquote(wtree), ''] >> - >> - def subshell(self, script=None, ast=None, redirs=None): >> - """Execute the script or AST in a subshell, with inherited >> redirections >> - if redirs is not None. >> - """ >> - if redirs: >> - sub_redirs = redirs >> - else: >> - sub_redirs = redirs.clone() >> - >> - subshell = None >> - try: >> - subshell = Interpreter(None, self._debugflags, >> self._env.clone(True), >> - sub_redirs, opts=self._options) >> - return subshell.execute_script(script, ast) >> - finally: >> - if not redirs: sub_redirs.close() >> - if subshell: subshell.close() >> - >> - def subshell_output(self, script): >> - """Execute the script in a subshell and return the captured >> output.""" >> - # Create temporary file to capture subshell output >> - tmpfd, tmppath = tempfile.mkstemp() >> - try: >> - tmpfile = os.fdopen(tmpfd, 'wb') >> - stdout = FileWrapper('w', tmpfile) >> - >> - redirs = Redirections(self._redirs.stdin().dup(), >> - stdout, >> - >> self._redirs.stderr().dup()) >> - try: >> - status = self.subshell(script=script, redirs=redirs) >> - finally: >> - redirs.close() >> - redirs = None >> - >> - # Extract subshell standard output >> - tmpfile = open(tmppath, 'rb') >> - try: >> - output = tmpfile.read() >> - return status, output.rstrip('\n') >> - finally: >> - tmpfile.close() >> - finally: >> - os.remove(tmppath) >> - >> - def _asynclist(self, cmd): >> - args = (self._env.get_variables(), cmd) >> - arg = encodeargs(args) >> - assert len(args) < 30*1024 >> - cmd = ['pysh.bat', '--ast', '-c', arg] >> - p = subprocess.Popen(cmd, cwd=self._env['PWD']) >> - self._children[p.pid] = p >> - self._env['!'] = p.pid >> - return 0 >> - >> - def wait(self, pids=None): >> - if not pids: >> - pids = self._children.keys() >> - >> - status = 127 >> - for pid in pids: >> - if pid not in self._children: >> - continue >> - p = self._children.pop(pid) >> - status = p.wait() >> - >> - return status >> - >> diff --git a/bitbake/lib/bb/pysh/lsprof.py >> b/bitbake/lib/bb/pysh/lsprof.py deleted file mode 100644 >> index b1831c2..0000000 >> --- a/bitbake/lib/bb/pysh/lsprof.py >> +++ /dev/null >> @@ -1,116 +0,0 @@ >> -#! /usr/bin/env python >> - >> -import sys >> -from _lsprof import Profiler, profiler_entry >> - >> -__all__ = ['profile', 'Stats'] >> - >> -def profile(f, *args, **kwds): >> - """XXX docstring""" >> - p = Profiler() >> - p.enable(subcalls=True, builtins=True) >> - try: >> - f(*args, **kwds) >> - finally: >> - p.disable() >> - return Stats(p.getstats()) >> - >> - >> -class Stats(object): >> - """XXX docstring""" >> - >> - def __init__(self, data): >> - self.data = data >> - >> - def sort(self, crit="inlinetime"): >> - """XXX docstring""" >> - if crit not in profiler_entry.__dict__: >> - raise ValueError("Can't sort by %s" % crit) >> - self.data.sort(lambda b, a: cmp(getattr(a, crit), >> - getattr(b, crit))) >> - for e in self.data: >> - if e.calls: >> - e.calls.sort(lambda b, a: cmp(getattr(a, crit), >> - getattr(b, crit))) >> - >> - def pprint(self, top=None, file=None, limit=None, climit=None): >> - """XXX docstring""" >> - if file is None: >> - file = sys.stdout >> - d = self.data >> - if top is not None: >> - d = d[:top] >> - cols = "% 12s %12s %11.4f %11.4f %s\n" >> - hcols = "% 12s %12s %12s %12s %s\n" >> - cols2 = "+%12s %12s %11.4f %11.4f + %s\n" >> - file.write(hcols % ("CallCount", "Recursive", "Total(ms)", >> - "Inline(ms)", "module:lineno(function)")) >> - count = 0 >> - for e in d: >> - file.write(cols % (e.callcount, e.reccallcount, >> e.totaltime, >> - e.inlinetime, label(e.code))) >> - count += 1 >> - if limit is not None and count == limit: >> - return >> - ccount = 0 >> - if e.calls: >> - for se in e.calls: >> - file.write(cols % ("+%s" % se.callcount, >> se.reccallcount, >> - se.totaltime, se.inlinetime, >> - "+%s" % label(se.code))) >> - count += 1 >> - ccount += 1 >> - if limit is not None and count == limit: >> - return >> - if climit is not None and ccount == climit: >> - break >> - >> - def freeze(self): >> - """Replace all references to code objects with string >> - descriptions; this makes it possible to pickle the >> instance.""" - >> - # this code is probably rather ickier than it needs to be! >> - for i in range(len(self.data)): >> - e = self.data[i] >> - if not isinstance(e.code, str): >> - self.data[i] = type(e)((label(e.code),) + e[1:]) >> - if e.calls: >> - for j in range(len(e.calls)): >> - se = e.calls[j] >> - if not isinstance(se.code, str): >> - e.calls[j] = type(se)((label(se.code),) + >> se[1:]) - >> -_fn2mod = {} >> - >> -def label(code): >> - if isinstance(code, str): >> - return code >> - try: >> - mname = _fn2mod[code.co_filename] >> - except KeyError: >> - for k, v in sys.modules.items(): >> - if v is None: >> - continue >> - if not hasattr(v, '__file__'): >> - continue >> - if not isinstance(v.__file__, str): >> - continue >> - if v.__file__.startswith(code.co_filename): >> - mname = _fn2mod[code.co_filename] = k >> - break >> - else: >> - mname = _fn2mod[code.co_filename] = >> '<%s>'%code.co_filename - >> - return '%s:%d(%s)' % (mname, code.co_firstlineno, code.co_name) >> - >> - >> -if __name__ == '__main__': >> - import os >> - sys.argv = sys.argv[1:] >> - if not sys.argv: >> - print >> sys.stderr, "usage: lsprof.py