From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6750261011454885888 X-Received: by 2002:a63:1904:: with SMTP id z4mr26292432pgl.413.1571667658410; Mon, 21 Oct 2019 07:20:58 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a63:4458:: with SMTP id t24ls3689665pgk.4.gmail; Mon, 21 Oct 2019 07:20:57 -0700 (PDT) X-Google-Smtp-Source: APXvYqyLDc1d0jukfgDlNvD/IbMrwd2rLONEYLrwWlkBZCbsUq9wPQ1LbbseR4fccSj/ccLtpKgT X-Received: by 2002:a63:2c9:: with SMTP id 192mr23664393pgc.315.1571667656911; Mon, 21 Oct 2019 07:20:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1571667656; cv=none; d=google.com; s=arc-20160816; b=uaB/cl/wauVXUoaTcJgpdZ3iUH7EbiYh2JfX41f2GTccRkZK+42rIXowHAn6Y6dHvc XKv+csf5gfRbYfHnAv6LOS2i1snHxOCQ2vCuudqz7AxqfsnhHX7zOOPi9RlAp6Mz4EFc PmVHaHKoQwmM6IbJzppfolSckO2zZNEFNPZgLE/edDvKSEEI4huUIL8h3lCYsIRhAugP q7dVHbiQin4Zhu48UuFJMX0ai3WFraIQqY4/EriNtzUV+KjRBHVoxN4XzvCGrNDteof/ Ixg9xEORPIhXOUE0DP3gO6jjQ51MCQ4kRGXof5fPGXLbdry8h0nY1MogNq9Iu/R33Js8 ydZA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:ironport-sdr:ironport-sdr; bh=AMhlpc5rwfN0vdjQ6i6rHEyfNhj4yUZ/k8Y/JS9qoog=; b=wjjJenB330NzCdV3mGaoBYMI8DFbtvGYrO6VslU9lgyHdAugEelFUqhdJ1bw5HeeH+ ighLQY5Lb1btiZJcp7eoPrlQ6VHMfO+2FxFowS2jHsTimYtoPjHtkHVbuSWscl8XwFZr WSOO+ylTtbfk07jRxlB/y0iUbyePuPHjX/1K4fRTr8ls+3viEy/bXM1bLjMtGMdZRjUy FDM6aD33PqatBhskzVK+8SxwaNMcHZcn0eZtsUdtnU0NodKPG/KLVSjZpEndqrQlKWmH UNsTLfL3Q0qX7fseTN+8OFc39DgAIi/DGJT3ltimbfg3R52Q7zwqnQhrZ7xkaA/6C4HX RbdQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of cedric_hombourger@mentor.com designates 68.232.129.153 as permitted sender) smtp.mailfrom=Cedric_Hombourger@mentor.com Return-Path: Received: from esa1.mentor.iphmx.com (esa1.mentor.iphmx.com. [68.232.129.153]) by gmr-mx.google.com with ESMTPS id b1si722440pjw.1.2019.10.21.07.20.55 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 21 Oct 2019 07:20:56 -0700 (PDT) Received-SPF: pass (google.com: domain of cedric_hombourger@mentor.com designates 68.232.129.153 as permitted sender) client-ip=68.232.129.153; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of cedric_hombourger@mentor.com designates 68.232.129.153 as permitted sender) smtp.mailfrom=Cedric_Hombourger@mentor.com IronPort-SDR: 4XAk/alrMx4Ka0xKQ7WQ5kZxjk0lYedIKtXtxQkZ2JcNG0He0AHzU8Xclfw1BBzNHDmuxZHkEs 8hhp3T3srwKLPhUz0dkc+I+ArnLu4rVpUUt94LdWbghN9oll9WiHcNBK7n9dskoO5Kkh0S3fYF Ug+iQu1jkbzxcN8JktYFcSJbsTBw99AbSxIWrXzRPIbgyWgyEiIHB86K6ZMA2tNKglSIzvjZGp ZA1hAVx9YhdYpcugM2YD/PTxTfcb8mJ3+sOd8leeW7VxVhDwI5S6p1IUH45uYWLrAbgB+buhpz Cdg= X-IronPort-AV: E=Sophos;i="5.67,323,1566892800"; d="scan'208";a="44273404" Received: from orw-gwy-01-in.mentorg.com ([192.94.38.165]) by esa1.mentor.iphmx.com with ESMTP; 21 Oct 2019 06:20:46 -0800 IronPort-SDR: 9WcYbd3Mw0iNLrnjRrS2XY0LM5X8aqG4vXV4+pcksDhO8QIsDa/KX9/4qpjPug9tyfjAx35bTz /C/hwM0day4UMaFBEkn0a7lyCYyzEtM8xIebqQ7GO4qhWky7h6Mpvp3ioXfAH3USY8pEeaJcuq Dax9y2Gm5Lg5Z/rxg25kOHF63ToHp8LREqQAoNzs+6j7BLcDfSvMIla7ZiZD1vYJVSbZTbXrnr r2hlgw7iRiaMD+Q/aJ9pdbLRjSriM3Pm/XRdYo3rnqyujtgOAAqgyQWKY61sV/6sGHm92f3V5b oUY= From: Cedric Hombourger To: CC: Cedric Hombourger Subject: [PATCH 1/1] bitbake: update to version 1.44.0 Date: Mon, 21 Oct 2019 16:19:58 +0200 Message-ID: <1571667598-657-2-git-send-email-Cedric_Hombourger@mentor.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1571667598-657-1-git-send-email-Cedric_Hombourger@mentor.com> References: <1571667598-657-1-git-send-email-Cedric_Hombourger@mentor.com> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit Return-Path: Cedric_Hombourger@mentor.com X-Originating-IP: [137.202.0.90] X-ClientProxiedBy: SVR-IES-MBX-03.mgc.mentorg.com (139.181.222.3) To svr-ies-mbx-02.mgc.mentorg.com (139.181.222.2) X-TUID: j2MF3TA4SFof Found that the gitsm fetcher we ship is rather out-dated compared to upstream and was failing to pull git sub-modules from repositories such as azure-umqtt-c. It shall be noted that bitbake has renamed its multiconfig targets: the "multiconfig:" prefix was changed to "mc:". Updated our scripts as well as documentation. Origin: https://github.com/openembedded/bitbake.git Commit 5d83d828cacb58ccb7c464e799c85fd2d2a50ccc (tag: 1.44.0) Author: Richard Purdie Date: Wed Oct 9 14:10:21 2019 +0100 bitbake: Update to version 1.44.0 Signed-off-by: Richard Purdie Signed-off-by: Cedric Hombourger --- RECIPE-API-CHANGELOG.md | 16 + bitbake/.gitattributes | 2 + bitbake/HEADER | 19 - bitbake/LICENSE | 12 +- bitbake/{COPYING => LICENSE.GPL-2.0-only} | 61 +- bitbake/LICENSE.MIT | 25 + bitbake/MANIFEST.in | 5 +- bitbake/bin/bitbake | 16 +- bitbake/bin/bitbake-diffsigs | 155 +- bitbake/bin/bitbake-dumpsig | 95 +- bitbake/bin/bitbake-hashclient | 170 ++ bitbake/bin/bitbake-hashserv | 62 + bitbake/bin/bitbake-layers | 12 +- bitbake/bin/bitbake-prserv | 4 + bitbake/bin/bitbake-selftest | 16 +- bitbake/bin/bitbake-worker | 19 +- bitbake/bin/bitdoc | 14 +- bitbake/bin/git-make-shallow | 4 + bitbake/bin/toaster | 12 +- bitbake/bin/toaster-eventreplay | 17 +- bitbake/classes/base.bbclass | 2 +- bitbake/contrib/dump_cache.py | 2 - .../bitbake-user-manual-execution.xml | 78 +- .../bitbake-user-manual-fetching.xml | 69 +- .../bitbake-user-manual-hello.xml | 28 +- .../bitbake-user-manual-intro.xml | 18 +- .../bitbake-user-manual-metadata.xml | 179 +- .../bitbake-user-manual-ref-variables.xml | 422 ++--- bitbake/doc/poky.ent | 8 - bitbake/lib/bb/COW.py | 15 - bitbake/lib/bb/__init__.py | 16 +- bitbake/lib/bb/build.py | 151 +- bitbake/lib/bb/cache.py | 109 +- bitbake/lib/bb/cache_extra.py | 14 +- bitbake/lib/bb/checksum.py | 12 +- bitbake/lib/bb/codeparser.py | 13 +- bitbake/lib/bb/command.py | 12 +- bitbake/lib/bb/compat.py | 4 + bitbake/lib/bb/cooker.py | 218 +-- bitbake/lib/bb/cookerdata.py | 44 +- bitbake/lib/bb/daemonize.py | 4 + bitbake/lib/bb/data.py | 23 +- bitbake/lib/bb/data_smart.py | 25 +- bitbake/lib/bb/event.py | 33 +- bitbake/lib/bb/exceptions.py | 3 + bitbake/lib/bb/fetch2/__init__.py | 55 +- bitbake/lib/bb/fetch2/bzr.py | 12 +- bitbake/lib/bb/fetch2/clearcase.py | 19 +- bitbake/lib/bb/fetch2/cvs.py | 17 +- bitbake/lib/bb/fetch2/git.py | 68 +- bitbake/lib/bb/fetch2/gitannex.py | 14 +- bitbake/lib/bb/fetch2/gitsm.py | 281 ++- bitbake/lib/bb/fetch2/hg.py | 18 +- bitbake/lib/bb/fetch2/local.py | 16 +- bitbake/lib/bb/fetch2/npm.py | 69 +- bitbake/lib/bb/fetch2/osc.py | 5 +- bitbake/lib/bb/fetch2/perforce.py | 15 +- bitbake/lib/bb/fetch2/repo.py | 16 +- bitbake/lib/bb/fetch2/s3.py | 15 +- bitbake/lib/bb/fetch2/sftp.py | 15 +- bitbake/lib/bb/fetch2/ssh.py | 14 +- bitbake/lib/bb/fetch2/svn.py | 98 +- bitbake/lib/bb/fetch2/wget.py | 104 +- bitbake/lib/bb/main.py | 41 +- bitbake/lib/bb/methodpool.py | 15 +- bitbake/lib/bb/monitordisk.py | 27 +- bitbake/lib/bb/msg.py | 14 +- bitbake/lib/bb/namedtuple_with_abc.py | 4 +- bitbake/lib/bb/parse/__init__.py | 14 +- bitbake/lib/bb/parse/ast.py | 15 +- bitbake/lib/bb/parse/parse_py/BBHandler.py | 59 +- bitbake/lib/bb/parse/parse_py/ConfHandler.py | 17 +- bitbake/lib/bb/parse/parse_py/__init__.py | 17 +- bitbake/lib/bb/persist_data.py | 234 ++- bitbake/lib/bb/process.py | 4 + bitbake/lib/bb/progress.py | 28 +- bitbake/lib/bb/providers.py | 18 +- bitbake/lib/bb/pysh/builtin.py | 710 -------- bitbake/lib/bb/pysh/interp.py | 1367 --------------- bitbake/lib/bb/pysh/lsprof.py | 116 -- bitbake/lib/bb/pysh/pysh.py | 167 -- bitbake/lib/bb/pysh/pyshlex.py | 5 - bitbake/lib/bb/pysh/pyshyacc.py | 17 +- bitbake/lib/bb/pysh/sherrors.py | 26 - bitbake/lib/bb/pysh/subprocess_fix.py | 77 - bitbake/lib/bb/remotedata.py | 12 +- bitbake/lib/bb/runqueue.py | 1523 ++++++++++------- bitbake/lib/bb/server/__init__.py | 14 +- bitbake/lib/bb/server/process.py | 123 +- bitbake/lib/bb/server/xmlrpcclient.py | 12 +- bitbake/lib/bb/server/xmlrpcserver.py | 12 +- bitbake/lib/bb/siggen.py | 350 +++- bitbake/lib/bb/taskdata.py | 17 +- bitbake/lib/bb/tests/codeparser.py | 22 +- bitbake/lib/bb/tests/cooker.py | 15 +- bitbake/lib/bb/tests/cow.py | 17 +- bitbake/lib/bb/tests/data.py | 39 +- bitbake/lib/bb/tests/event.py | 23 +- bitbake/lib/bb/tests/fetch.py | 308 +++- bitbake/lib/bb/tests/parse.py | 33 +- bitbake/lib/bb/tests/persist_data.py | 129 ++ .../tests/runqueue-tests/classes/base.bbclass | 262 +++ .../runqueue-tests/classes/image.bbclass | 5 + .../runqueue-tests/classes/native.bbclass | 2 + .../bb/tests/runqueue-tests/conf/bitbake.conf | 16 + .../runqueue-tests/conf/multiconfig/mc1.conf | 1 + .../runqueue-tests/conf/multiconfig/mc2.conf | 1 + .../lib/bb/tests/runqueue-tests/recipes/a1.bb | 0 .../lib/bb/tests/runqueue-tests/recipes/b1.bb | 1 + .../lib/bb/tests/runqueue-tests/recipes/c1.bb | 0 .../lib/bb/tests/runqueue-tests/recipes/d1.bb | 3 + .../lib/bb/tests/runqueue-tests/recipes/e1.bb | 1 + bitbake/lib/bb/tests/runqueue.py | 323 ++++ bitbake/lib/bb/tests/utils.py | 23 +- bitbake/lib/bb/tinfoil.py | 12 +- bitbake/lib/bb/ui/__init__.py | 12 +- bitbake/lib/bb/ui/buildinfohelper.py | 22 +- bitbake/lib/bb/ui/knotty.py | 60 +- bitbake/lib/bb/ui/ncurses.py | 12 +- bitbake/lib/bb/ui/taskexp.py | 12 +- bitbake/lib/bb/ui/toasterui.py | 12 +- bitbake/lib/bb/ui/uievent.py | 15 +- bitbake/lib/bb/ui/uihelper.py | 16 +- bitbake/lib/bb/utils.py | 49 +- bitbake/lib/bblayers/__init__.py | 4 + bitbake/lib/bblayers/action.py | 4 + bitbake/lib/bblayers/common.py | 4 + bitbake/lib/bblayers/layerindex.py | 6 +- bitbake/lib/bblayers/query.py | 43 +- bitbake/lib/bs4/dammit.py | 12 +- bitbake/lib/bs4/element.py | 22 +- bitbake/lib/hashserv/__init__.py | 93 + bitbake/lib/hashserv/client.py | 157 ++ bitbake/lib/hashserv/server.py | 414 +++++ bitbake/lib/hashserv/tests.py | 142 ++ bitbake/lib/layerindexlib/__init__.py | 33 +- bitbake/lib/layerindexlib/cooker.py | 12 +- bitbake/lib/layerindexlib/plugin.py | 13 +- bitbake/lib/layerindexlib/restapi.py | 12 +- bitbake/lib/layerindexlib/tests/common.py | 12 +- bitbake/lib/layerindexlib/tests/cooker.py | 12 +- .../lib/layerindexlib/tests/layerindexobj.py | 12 +- bitbake/lib/layerindexlib/tests/restapi.py | 12 +- bitbake/lib/progressbar/__init__.py | 2 + bitbake/lib/progressbar/compat.py | 2 + bitbake/lib/progressbar/progressbar.py | 2 + bitbake/lib/progressbar/widgets.py | 2 + bitbake/lib/prserv/__init__.py | 4 + bitbake/lib/prserv/db.py | 6 +- bitbake/lib/prserv/serv.py | 4 + bitbake/lib/pyinotify.py | 20 +- bitbake/lib/toaster/bldcollector/admin.py | 4 + bitbake/lib/toaster/bldcollector/urls.py | 13 +- bitbake/lib/toaster/bldcollector/views.py | 12 +- bitbake/lib/toaster/bldcontrol/admin.py | 4 + .../lib/toaster/bldcontrol/bbcontroller.py | 16 +- .../bldcontrol/localhostbecontroller.py | 16 +- .../management/commands/checksettings.py | 4 + .../management/commands/runbuilds.py | 4 + bitbake/lib/toaster/bldcontrol/models.py | 4 + bitbake/lib/toaster/bldcontrol/views.py | 4 + bitbake/lib/toaster/manage.py | 4 + bitbake/lib/toaster/orm/fixtures/oe-core.xml | 26 +- bitbake/lib/toaster/orm/fixtures/poky.xml | 38 +- .../orm/management/commands/lsupdates.py | 15 +- bitbake/lib/toaster/orm/models.py | 25 +- .../toaster/tests/browser/selenium_helpers.py | 17 +- .../tests/browser/selenium_helpers_base.py | 17 +- .../tests/browser/test_all_builds_page.py | 16 +- .../tests/browser/test_all_projects_page.py | 16 +- .../tests/browser/test_builddashboard_page.py | 16 +- .../test_builddashboard_page_artifacts.py | 16 +- .../test_builddashboard_page_recipes.py | 16 +- .../browser/test_builddashboard_page_tasks.py | 16 +- .../tests/browser/test_js_unit_tests.py | 16 +- .../tests/browser/test_landing_page.py | 18 +- .../tests/browser/test_layerdetails_page.py | 18 +- .../browser/test_most_recent_builds_states.py | 18 +- .../browser/test_new_custom_image_page.py | 16 +- .../tests/browser/test_new_project_page.py | 16 +- .../tests/browser/test_project_builds_page.py | 16 +- .../tests/browser/test_project_config_page.py | 16 +- .../tests/browser/test_project_page.py | 16 +- .../lib/toaster/tests/browser/test_sample.py | 16 +- .../toaster/tests/browser/test_task_page.py | 16 +- .../tests/browser/test_toastertable_ui.py | 16 +- bitbake/lib/toaster/tests/builds/buildtest.py | 16 +- .../tests/builds/test_core_image_min.py | 17 +- .../toaster/tests/commands/test_loaddata.py | 16 +- .../toaster/tests/commands/test_lsupdates.py | 16 +- .../toaster/tests/commands/test_runbuilds.py | 16 +- bitbake/lib/toaster/tests/db/test_db.py | 2 + .../lib/toaster/tests/eventreplay/__init__.py | 16 +- .../tests/functional/functional_helpers.py | 16 +- .../tests/functional/test_functional_basic.py | 16 +- bitbake/lib/toaster/tests/views/test_views.py | 16 +- bitbake/lib/toaster/toastergui/api.py | 13 +- bitbake/lib/toaster/toastergui/buildtables.py | 15 +- .../toastergui/static/js/importlayer.js | 12 +- bitbake/lib/toaster/toastergui/tablefilter.py | 15 +- bitbake/lib/toaster/toastergui/tables.py | 15 +- .../templatetags/field_values_filter.py | 4 + .../objects_to_dictionaries_filter.py | 4 + .../templatetags/project_url_tag.py | 4 + .../toastergui/templatetags/projecttags.py | 15 +- bitbake/lib/toaster/toastergui/typeaheads.py | 12 +- bitbake/lib/toaster/toastergui/urls.py | 12 +- bitbake/lib/toaster/toastergui/views.py | 16 +- bitbake/lib/toaster/toastergui/widgets.py | 15 +- .../management/commands/builddelete.py | 4 + .../management/commands/buildimport.py | 15 +- .../management/commands/buildslist.py | 4 + .../management/commands/checksocket.py | 16 +- .../toastermain/management/commands/perf.py | 4 + bitbake/lib/toaster/toastermain/settings.py | 15 +- .../settings_production_example.py | 15 +- .../lib/toaster/toastermain/settings_test.py | 15 +- bitbake/lib/toaster/toastermain/urls.py | 15 +- bitbake/lib/toaster/toastermain/wsgi.py | 7 +- doc/user_manual.md | 38 +- meta-isar/conf/conf-notes.txt | 6 +- scripts/ci_build.sh | 62 +- scripts/start_vm | 8 +- testsuite/build_test/build_test.py | 2 +- testsuite/start_vm.py | 2 +- 225 files changed, 5375 insertions(+), 6320 deletions(-) create mode 100644 bitbake/.gitattributes delete mode 100644 bitbake/HEADER rename bitbake/{COPYING => LICENSE.GPL-2.0-only} (84%) create mode 100644 bitbake/LICENSE.MIT mode change 100755 => 120000 bitbake/bin/bitbake-dumpsig create mode 100755 bitbake/bin/bitbake-hashclient create mode 100755 bitbake/bin/bitbake-hashserv delete mode 100644 bitbake/lib/bb/pysh/builtin.py delete mode 100644 bitbake/lib/bb/pysh/interp.py delete mode 100644 bitbake/lib/bb/pysh/lsprof.py delete mode 100644 bitbake/lib/bb/pysh/pysh.py delete mode 100644 bitbake/lib/bb/pysh/subprocess_fix.py create mode 100644 bitbake/lib/bb/tests/persist_data.py create mode 100644 bitbake/lib/bb/tests/runqueue-tests/classes/base.bbclass create mode 100644 bitbake/lib/bb/tests/runqueue-tests/classes/image.bbclass create mode 100644 bitbake/lib/bb/tests/runqueue-tests/classes/native.bbclass create mode 100644 bitbake/lib/bb/tests/runqueue-tests/conf/bitbake.conf create mode 100644 bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc1.conf create mode 100644 bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc2.conf create mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/a1.bb create mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/b1.bb create mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/c1.bb create mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/d1.bb create mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/e1.bb create mode 100644 bitbake/lib/bb/tests/runqueue.py create mode 100644 bitbake/lib/hashserv/__init__.py create mode 100644 bitbake/lib/hashserv/client.py create mode 100644 bitbake/lib/hashserv/server.py create mode 100644 bitbake/lib/hashserv/tests.py diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md index bbef1a3..81ec430 100644 --- a/RECIPE-API-CHANGELOG.md +++ b/RECIPE-API-CHANGELOG.md @@ -176,3 +176,19 @@ Otherwise set a encrypted root password like this: USERS += "root" USER_root[password] = "$6$rounds=10000$RXeWrnFmkY$DtuS/OmsAS2cCEDo0BF5qQsizIrq6jPgXnwv3PHqREJeKd1sXdHX/ayQtuQWVDHe0KIO0/sVH8dvQm1KthF0d/" ``` + +### multiconfig build targets were renamed + +bitbake was upgraded to version 1.44.0 where "multiconfig" build targets were +renamed "mc". As an example, builds for the qemuarm-stretch machine should now +be done as follows: + +``` +bitbake mc:qemuarm-stretch:isar-image-base +``` + +The old syntax is no longer supported and will produce an error: + +``` +bitbake multiconfig:qemuarm-stretch:isar-image-base +``` diff --git a/bitbake/.gitattributes b/bitbake/.gitattributes new file mode 100644 index 0000000..e4f8f62 --- /dev/null +++ b/bitbake/.gitattributes @@ -0,0 +1,2 @@ +*min.js binary +*min.css binary diff --git a/bitbake/HEADER b/bitbake/HEADER deleted file mode 100644 index 9859255..0000000 --- a/bitbake/HEADER +++ /dev/null @@ -1,19 +0,0 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- -# -# -# Copyright (C) -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - diff --git a/bitbake/LICENSE b/bitbake/LICENSE index 7d4e5f4..8458042 100644 --- a/bitbake/LICENSE +++ b/bitbake/LICENSE @@ -1,4 +1,13 @@ -BitBake is licensed under the GNU General Public License version 2.0. See COPYING for further details. +BitBake is licensed under the GNU General Public License version 2.0. See +LICENSE.GPL-2.0-only for further details. + +Individual files contain the following style tags instead of the full license text: + + SPDX-License-Identifier: GPL-2.0-only + +This enables machine processing of license information based on the SPDX +License Identifiers that are here available: http://spdx.org/licenses/ + The following external components are distributed with this software: @@ -17,3 +26,4 @@ Foundation and individual contributors. * Font Awesome fonts redistributed under the SIL Open Font License 1.1 * simplediff is distributed under the zlib license. + diff --git a/bitbake/COPYING b/bitbake/LICENSE.GPL-2.0-only similarity index 84% rename from bitbake/COPYING rename to bitbake/LICENSE.GPL-2.0-only index d511905..5db3c0a 100644 --- a/bitbake/COPYING +++ b/bitbake/LICENSE.GPL-2.0-only @@ -279,61 +279,10 @@ POSSIBILITY OF SUCH DAMAGES. END OF TERMS AND CONDITIONS - How to Apply These Terms to Your New Programs +Note: +Individual files contain the following tag instead of the full license text. - If you develop a new program, and you want it to be of the greatest -possible use to the public, the best way to achieve this is to make it -free software which everyone can redistribute and change under these terms. + SPDX-License-Identifier: GPL-2.0-only - To do so, attach the following notices to the program. It is safest -to attach them to the start of each source file to most effectively -convey the exclusion of warranty; and each file should have at least -the "copyright" line and a pointer to where the full notice is found. - - - Copyright (C) - - This program is free software; you can redistribute it and/or modify - it under the terms of the GNU General Public License as published by - the Free Software Foundation; either version 2 of the License, or - (at your option) any later version. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License along - with this program; if not, write to the Free Software Foundation, Inc., - 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -Also add information on how to contact you by electronic and paper mail. - -If the program is interactive, make it output a short notice like this -when it starts in an interactive mode: - - Gnomovision version 69, Copyright (C) year name of author - Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. - This is free software, and you are welcome to redistribute it - under certain conditions; type `show c' for details. - -The hypothetical commands `show w' and `show c' should show the appropriate -parts of the General Public License. Of course, the commands you use may -be called something other than `show w' and `show c'; they could even be -mouse-clicks or menu items--whatever suits your program. - -You should also get your employer (if you work as a programmer) or your -school, if any, to sign a "copyright disclaimer" for the program, if -necessary. Here is a sample; alter the names: - - Yoyodyne, Inc., hereby disclaims all copyright interest in the program - `Gnomovision' (which makes passes at compilers) written by James Hacker. - - , 1 April 1989 - Ty Coon, President of Vice - -This General Public License does not permit incorporating your program into -proprietary programs. If your program is a subroutine library, you may -consider it more useful to permit linking proprietary applications with the -library. If this is what you want to do, use the GNU Lesser General -Public License instead of this License. +This enables machine processing of license information based on the SPDX +License Identifiers that are here available: http://spdx.org/licenses/ diff --git a/bitbake/LICENSE.MIT b/bitbake/LICENSE.MIT new file mode 100644 index 0000000..a6919eb --- /dev/null +++ b/bitbake/LICENSE.MIT @@ -0,0 +1,25 @@ +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +Note: +Individual files contain the following tag instead of the full license text. + + SPDX-License-Identifier: MIT + +This enables machine processing of license information based on the SPDX +License Identifiers that are here available: http://spdx.org/licenses/ diff --git a/bitbake/MANIFEST.in b/bitbake/MANIFEST.in index b197378..f24969a 100644 --- a/bitbake/MANIFEST.in +++ b/bitbake/MANIFEST.in @@ -1,6 +1,8 @@ -include COPYING include ChangeLog include AUTHORS +include LICENSE +include LICENSE.GPL-2.0-only +include LICENSE.MIT include contrib/* include contrib/vim/*/* include conf/* @@ -8,4 +10,3 @@ include classes/* include doc/* include doc/manual/* include ez_setup.py -include HEADER diff --git a/bitbake/bin/bitbake b/bitbake/bin/bitbake index 57dec2a..66d08f8 100755 --- a/bitbake/bin/bitbake +++ b/bitbake/bin/bitbake @@ -1,6 +1,4 @@ #!/usr/bin/env python3 -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- # # Copyright (C) 2003, 2004 Chris Larson # Copyright (C) 2003, 2004 Phil Blundell @@ -9,18 +7,8 @@ # Copyright (C) 2005 ROAD GmbH # Copyright (C) 2006 Richard Purdie # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. +# SPDX-License-Identifier: GPL-2.0-only # -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. import os import sys @@ -38,7 +26,7 @@ from bb.main import bitbake_main, BitBakeConfigParameters, BBMainException if sys.getfilesystemencoding() != "utf-8": sys.exit("Please use a locale setting which supports UTF-8 (such as LANG=en_US.UTF-8).\nPython can't change the filesystem locale after loading so we need a UTF-8 when Python starts or things won't work.") -__version__ = "1.40.0" +__version__ = "1.44.0" if __name__ == "__main__": if __version__ != bb.__version__: diff --git a/bitbake/bin/bitbake-diffsigs b/bitbake/bin/bitbake-diffsigs index 4e6bbdd..19420a2 100755 --- a/bitbake/bin/bitbake-diffsigs +++ b/bitbake/bin/bitbake-diffsigs @@ -1,27 +1,16 @@ #!/usr/bin/env python3 -# bitbake-diffsigs -# BitBake task signature data comparison utility +# bitbake-diffsigs / bitbake-dumpsig +# BitBake task signature data dump and comparison utility # # Copyright (C) 2012-2013, 2017 Intel Corporation # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. +# SPDX-License-Identifier: GPL-2.0-only # -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. import os import sys import warnings -import fnmatch import argparse import logging import pickle @@ -32,7 +21,10 @@ import bb.tinfoil import bb.siggen import bb.msg -logger = bb.msg.logger_create('bitbake-diffsigs') +myname = os.path.basename(sys.argv[0]) +logger = bb.msg.logger_create(myname) + +is_dump = myname == 'bitbake-dumpsig' def find_siginfo(tinfoil, pn, taskname, sigs=None): result = None @@ -59,8 +51,8 @@ def find_siginfo(tinfoil, pn, taskname, sigs=None): sys.exit(2) return result -def find_compare_task(bbhandler, pn, taskname, sig1=None, sig2=None, color=False): - """ Find the most recent signature files for the specified PN/task and compare them """ +def find_siginfo_task(bbhandler, pn, taskname, sig1=None, sig2=None): + """ Find the most recent signature files for the specified PN/task """ if not taskname.startswith('do_'): taskname = 'do_%s' % taskname @@ -79,73 +71,81 @@ def find_compare_task(bbhandler, pn, taskname, sig1=None, sig2=None, color=False latestfiles = [sigfiles[sig1], sigfiles[sig2]] else: filedates = find_siginfo(bbhandler, pn, taskname) - latestfiles = sorted(filedates.keys(), key=lambda f: filedates[f])[-3:] + latestfiles = sorted(filedates.keys(), key=lambda f: filedates[f])[-2:] if not latestfiles: logger.error('No sigdata files found matching %s %s' % (pn, taskname)) sys.exit(1) - elif len(latestfiles) < 2: - logger.error('Only one matching sigdata file found for the specified task (%s %s)' % (pn, taskname)) - sys.exit(1) - # Define recursion callback - def recursecb(key, hash1, hash2): - hashes = [hash1, hash2] - hashfiles = find_siginfo(bbhandler, key, None, hashes) - - recout = [] - if len(hashfiles) == 0: - recout.append("Unable to find matching sigdata for %s with hashes %s or %s" % (key, hash1, hash2)) - elif not hash1 in hashfiles: - recout.append("Unable to find matching sigdata for %s with hash %s" % (key, hash1)) - elif not hash2 in hashfiles: - recout.append("Unable to find matching sigdata for %s with hash %s" % (key, hash2)) - else: - out2 = bb.siggen.compare_sigfiles(hashfiles[hash1], hashfiles[hash2], recursecb, color=color) - for change in out2: - for line in change.splitlines(): - recout.append(' ' + line) + return latestfiles + - return recout +# Define recursion callback +def recursecb(key, hash1, hash2): + hashes = [hash1, hash2] + hashfiles = find_siginfo(tinfoil, key, None, hashes) - # Recurse into signature comparison - logger.debug("Signature file (previous): %s" % latestfiles[-2]) - logger.debug("Signature file (latest): %s" % latestfiles[-1]) - output = bb.siggen.compare_sigfiles(latestfiles[-2], latestfiles[-1], recursecb, color=color) - if output: - print('\n'.join(output)) - sys.exit(0) + recout = [] + if len(hashfiles) == 0: + recout.append("Unable to find matching sigdata for %s with hashes %s or %s" % (key, hash1, hash2)) + elif not hash1 in hashfiles: + recout.append("Unable to find matching sigdata for %s with hash %s" % (key, hash1)) + elif not hash2 in hashfiles: + recout.append("Unable to find matching sigdata for %s with hash %s" % (key, hash2)) + else: + out2 = bb.siggen.compare_sigfiles(hashfiles[hash1], hashfiles[hash2], recursecb, color=color) + for change in out2: + for line in change.splitlines(): + recout.append(' ' + line) + return recout parser = argparse.ArgumentParser( - description="Compares siginfo/sigdata files written out by BitBake") + description=("Dumps" if is_dump else "Compares") + " siginfo/sigdata files written out by BitBake") -parser.add_argument('-d', '--debug', +parser.add_argument('-D', '--debug', help='Enable debug output', action='store_true') -parser.add_argument('--color', - help='Colorize output (where %(metavar)s is %(choices)s)', - choices=['auto', 'always', 'never'], default='auto', metavar='color') +if is_dump: + parser.add_argument("-t", "--task", + help="find the signature data file for the last run of the specified task", + action="store", dest="taskargs", nargs=2, metavar=('recipename', 'taskname')) + + parser.add_argument("sigdatafile1", + help="Signature file to dump. Not used when using -t/--task.", + action="store", nargs='?', metavar="sigdatafile") +else: + parser.add_argument('-c', '--color', + help='Colorize the output (where %(metavar)s is %(choices)s)', + choices=['auto', 'always', 'never'], default='auto', metavar='color') -parser.add_argument("-t", "--task", - help="find the signature data files for last two runs of the specified task and compare them", - action="store", dest="taskargs", nargs=2, metavar=('recipename', 'taskname')) + parser.add_argument('-d', '--dump', + help='Dump the last signature data instead of comparing (equivalent to using bitbake-dumpsig)', + action='store_true') -parser.add_argument("-s", "--signature", - help="With -t/--task, specify the signatures to look for instead of taking the last two", - action="store", dest="sigargs", nargs=2, metavar=('fromsig', 'tosig')) + parser.add_argument("-t", "--task", + help="find the signature data files for the last two runs of the specified task and compare them", + action="store", dest="taskargs", nargs=2, metavar=('recipename', 'taskname')) -parser.add_argument("sigdatafile1", - help="First signature file to compare (or signature file to dump, if second not specified). Not used when using -t/--task.", - action="store", nargs='?') + parser.add_argument("-s", "--signature", + help="With -t/--task, specify the signatures to look for instead of taking the last two", + action="store", dest="sigargs", nargs=2, metavar=('fromsig', 'tosig')) -parser.add_argument("sigdatafile2", - help="Second signature file to compare", - action="store", nargs='?') + parser.add_argument("sigdatafile1", + help="First signature file to compare (or signature file to dump, if second not specified). Not used when using -t/--task.", + action="store", nargs='?') + parser.add_argument("sigdatafile2", + help="Second signature file to compare", + action="store", nargs='?') options = parser.parse_args() +if is_dump: + options.color = 'never' + options.dump = True + options.sigdatafile2 = None + options.sigargs = None if options.debug: logger.setLevel(logging.DEBUG) @@ -155,17 +155,32 @@ color = (options.color == 'always' or (options.color == 'auto' and sys.stdout.is if options.taskargs: with bb.tinfoil.Tinfoil() as tinfoil: tinfoil.prepare(config_only=True) - if options.sigargs: - find_compare_task(tinfoil, options.taskargs[0], options.taskargs[1], options.sigargs[0], options.sigargs[1], color=color) + if not options.dump and options.sigargs: + files = find_siginfo_task(tinfoil, options.taskargs[0], options.taskargs[1], options.sigargs[0], options.sigargs[1]) + else: + files = find_siginfo_task(tinfoil, options.taskargs[0], options.taskargs[1]) + + if options.dump: + logger.debug("Signature file: %s" % files[-1]) + output = bb.siggen.dump_sigfile(files[-1]) else: - find_compare_task(tinfoil, options.taskargs[0], options.taskargs[1], color=color) + if len(files) < 2: + logger.error('Only one matching sigdata file found for the specified task (%s %s)' % (options.taskargs[0], options.taskargs[1])) + sys.exit(1) + + # Recurse into signature comparison + logger.debug("Signature file (previous): %s" % files[-2]) + logger.debug("Signature file (latest): %s" % files[-1]) + output = bb.siggen.compare_sigfiles(files[-2], files[-1], recursecb, color=color) else: if options.sigargs: logger.error('-s/--signature can only be used together with -t/--task') sys.exit(1) try: - if options.sigdatafile1 and options.sigdatafile2: - output = bb.siggen.compare_sigfiles(options.sigdatafile1, options.sigdatafile2, color=color) + if not options.dump and options.sigdatafile1 and options.sigdatafile2: + with bb.tinfoil.Tinfoil() as tinfoil: + tinfoil.prepare(config_only=True) + output = bb.siggen.compare_sigfiles(options.sigdatafile1, options.sigdatafile2, recursecb, color=color) elif options.sigdatafile1: output = bb.siggen.dump_sigfile(options.sigdatafile1) else: @@ -179,5 +194,5 @@ else: logger.error('Invalid signature data - ensure you are specifying sigdata/siginfo files') sys.exit(1) - if output: - print('\n'.join(output)) +if output: + print('\n'.join(output)) diff --git a/bitbake/bin/bitbake-dumpsig b/bitbake/bin/bitbake-dumpsig deleted file mode 100755 index 95ebd93..0000000 --- a/bitbake/bin/bitbake-dumpsig +++ /dev/null @@ -1,94 +0,0 @@ -#!/usr/bin/env python3 - -# bitbake-dumpsig -# BitBake task signature dump utility -# -# Copyright (C) 2013 Intel Corporation -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - -import os -import sys -import warnings -import optparse -import logging -import pickle - -sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(sys.argv[0])), 'lib')) - -import bb.tinfoil -import bb.siggen -import bb.msg - -logger = bb.msg.logger_create('bitbake-dumpsig') - -def find_siginfo_task(bbhandler, pn, taskname): - """ Find the most recent signature file for the specified PN/task """ - - if not hasattr(bb.siggen, 'find_siginfo'): - logger.error('Metadata does not support finding signature data files') - sys.exit(1) - - if not taskname.startswith('do_'): - taskname = 'do_%s' % taskname - - filedates = bb.siggen.find_siginfo(pn, taskname, None, bbhandler.config_data) - latestfiles = sorted(filedates.keys(), key=lambda f: filedates[f])[-1:] - if not latestfiles: - logger.error('No sigdata files found matching %s %s' % (pn, taskname)) - sys.exit(1) - - return latestfiles[0] - -parser = optparse.OptionParser( - description = "Dumps siginfo/sigdata files written out by BitBake", - usage = """ - %prog -t recipename taskname - %prog sigdatafile""") - -parser.add_option("-D", "--debug", - help = "enable debug", - action = "store_true", dest="debug", default = False) - -parser.add_option("-t", "--task", - help = "find the signature data file for the specified task", - action="store", dest="taskargs", nargs=2, metavar='recipename taskname') - -options, args = parser.parse_args(sys.argv) - -if options.debug: - logger.setLevel(logging.DEBUG) - -if options.taskargs: - tinfoil = bb.tinfoil.Tinfoil() - tinfoil.prepare(config_only = True) - file = find_siginfo_task(tinfoil, options.taskargs[0], options.taskargs[1]) - logger.debug("Signature file: %s" % file) -elif len(args) == 1: - parser.print_help() - sys.exit(0) -else: - file = args[1] - -try: - output = bb.siggen.dump_sigfile(file) -except IOError as e: - logger.error(str(e)) - sys.exit(1) -except (pickle.UnpicklingError, EOFError): - logger.error('Invalid signature data - ensure you are specifying a sigdata/siginfo file') - sys.exit(1) - -if output: - print('\n'.join(output)) diff --git a/bitbake/bin/bitbake-dumpsig b/bitbake/bin/bitbake-dumpsig new file mode 120000 index 0000000..b1e8489 --- /dev/null +++ b/bitbake/bin/bitbake-dumpsig @@ -0,0 +1 @@ +bitbake-diffsigs \ No newline at end of file diff --git a/bitbake/bin/bitbake-hashclient b/bitbake/bin/bitbake-hashclient new file mode 100755 index 0000000..29ab65f --- /dev/null +++ b/bitbake/bin/bitbake-hashclient @@ -0,0 +1,170 @@ +#! /usr/bin/env python3 +# +# Copyright (C) 2019 Garmin Ltd. +# +# SPDX-License-Identifier: GPL-2.0-only +# + +import argparse +import hashlib +import logging +import os +import pprint +import sys +import threading +import time + +try: + import tqdm + ProgressBar = tqdm.tqdm +except ImportError: + class ProgressBar(object): + def __init__(self, *args, **kwargs): + pass + + def __enter__(self): + return self + + def __exit__(self, *args, **kwargs): + pass + + def update(self): + pass + +sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib')) + +import hashserv + +DEFAULT_ADDRESS = 'unix://./hashserve.sock' +METHOD = 'stress.test.method' + + +def main(): + def handle_stats(args, client): + if args.reset: + s = client.reset_stats() + else: + s = client.get_stats() + pprint.pprint(s) + return 0 + + def handle_stress(args, client): + def thread_main(pbar, lock): + nonlocal found_hashes + nonlocal missed_hashes + nonlocal max_time + + client = hashserv.create_client(args.address) + + for i in range(args.requests): + taskhash = hashlib.sha256() + taskhash.update(args.taskhash_seed.encode('utf-8')) + taskhash.update(str(i).encode('utf-8')) + + start_time = time.perf_counter() + l = client.get_unihash(METHOD, taskhash.hexdigest()) + elapsed = time.perf_counter() - start_time + + with lock: + if l: + found_hashes += 1 + else: + missed_hashes += 1 + + max_time = max(elapsed, max_time) + pbar.update() + + max_time = 0 + found_hashes = 0 + missed_hashes = 0 + lock = threading.Lock() + total_requests = args.clients * args.requests + start_time = time.perf_counter() + with ProgressBar(total=total_requests) as pbar: + threads = [threading.Thread(target=thread_main, args=(pbar, lock), daemon=False) for _ in range(args.clients)] + for t in threads: + t.start() + + for t in threads: + t.join() + + elapsed = time.perf_counter() - start_time + with lock: + print("%d requests in %.1fs. %.1f requests per second" % (total_requests, elapsed, total_requests / elapsed)) + print("Average request time %.8fs" % (elapsed / total_requests)) + print("Max request time was %.8fs" % max_time) + print("Found %d hashes, missed %d" % (found_hashes, missed_hashes)) + + if args.report: + with ProgressBar(total=args.requests) as pbar: + for i in range(args.requests): + taskhash = hashlib.sha256() + taskhash.update(args.taskhash_seed.encode('utf-8')) + taskhash.update(str(i).encode('utf-8')) + + outhash = hashlib.sha256() + outhash.update(args.outhash_seed.encode('utf-8')) + outhash.update(str(i).encode('utf-8')) + + client.report_unihash(taskhash.hexdigest(), METHOD, outhash.hexdigest(), taskhash.hexdigest()) + + with lock: + pbar.update() + + parser = argparse.ArgumentParser(description='Hash Equivalence Client') + parser.add_argument('--address', default=DEFAULT_ADDRESS, help='Server address (default "%(default)s")') + parser.add_argument('--log', default='WARNING', help='Set logging level') + + subparsers = parser.add_subparsers() + + stats_parser = subparsers.add_parser('stats', help='Show server stats') + stats_parser.add_argument('--reset', action='store_true', + help='Reset server stats') + stats_parser.set_defaults(func=handle_stats) + + stress_parser = subparsers.add_parser('stress', help='Run stress test') + stress_parser.add_argument('--clients', type=int, default=10, + help='Number of simultaneous clients') + stress_parser.add_argument('--requests', type=int, default=1000, + help='Number of requests each client will perform') + stress_parser.add_argument('--report', action='store_true', + help='Report new hashes') + stress_parser.add_argument('--taskhash-seed', default='', + help='Include string in taskhash') + stress_parser.add_argument('--outhash-seed', default='', + help='Include string in outhash') + stress_parser.set_defaults(func=handle_stress) + + args = parser.parse_args() + + logger = logging.getLogger('hashserv') + + level = getattr(logging, args.log.upper(), None) + if not isinstance(level, int): + raise ValueError('Invalid log level: %s' % args.log) + + logger.setLevel(level) + console = logging.StreamHandler() + console.setLevel(level) + logger.addHandler(console) + + func = getattr(args, 'func', None) + if func: + client = hashserv.create_client(args.address) + # Try to establish a connection to the server now to detect failures + # early + client.connect() + + return func(args, client) + + return 0 + + +if __name__ == '__main__': + try: + ret = main() + except Exception: + ret = 1 + import traceback + traceback.print_exc() + sys.exit(ret) diff --git a/bitbake/bin/bitbake-hashserv b/bitbake/bin/bitbake-hashserv new file mode 100755 index 0000000..1bc1f91 --- /dev/null +++ b/bitbake/bin/bitbake-hashserv @@ -0,0 +1,62 @@ +#! /usr/bin/env python3 +# +# Copyright (C) 2018 Garmin Ltd. +# +# SPDX-License-Identifier: GPL-2.0-only +# + +import os +import sys +import logging +import argparse +import sqlite3 + +sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib')) + +import hashserv + +VERSION = "1.0.0" + +DEFAULT_BIND = 'unix://./hashserve.sock' + + +def main(): + parser = argparse.ArgumentParser(description='Hash Equivalence Reference Server. Version=%s' % VERSION, + epilog='''The bind address is the path to a unix domain socket if it is + prefixed with "unix://". Otherwise, it is an IP address + and port in form ADDRESS:PORT. To bind to all addresses, leave + the ADDRESS empty, e.g. "--bind :8686". To bind to a specific + IPv6 address, enclose the address in "[]", e.g. + "--bind [::1]:8686"''' + ) + + parser.add_argument('--bind', default=DEFAULT_BIND, help='Bind address (default "%(default)s")') + parser.add_argument('--database', default='./hashserv.db', help='Database file (default "%(default)s")') + parser.add_argument('--log', default='WARNING', help='Set logging level') + + args = parser.parse_args() + + logger = logging.getLogger('hashserv') + + level = getattr(logging, args.log.upper(), None) + if not isinstance(level, int): + raise ValueError('Invalid log level: %s' % args.log) + + logger.setLevel(level) + console = logging.StreamHandler() + console.setLevel(level) + logger.addHandler(console) + + server = hashserv.create_server(args.bind, args.database) + server.serve_forever() + return 0 + + +if __name__ == '__main__': + try: + ret = main() + except Exception: + ret = 1 + import traceback + traceback.print_exc() + sys.exit(ret) diff --git a/bitbake/bin/bitbake-layers b/bitbake/bin/bitbake-layers index d184011..a884dc1 100755 --- a/bitbake/bin/bitbake-layers +++ b/bitbake/bin/bitbake-layers @@ -7,18 +7,8 @@ # Copyright (C) 2011 Mentor Graphics Corporation # Copyright (C) 2011-2015 Intel Corporation # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. +# SPDX-License-Identifier: GPL-2.0-only # -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. import logging import os diff --git a/bitbake/bin/bitbake-prserv b/bitbake/bin/bitbake-prserv index f38d2dd..1e9b6cb 100755 --- a/bitbake/bin/bitbake-prserv +++ b/bitbake/bin/bitbake-prserv @@ -1,4 +1,8 @@ #!/usr/bin/env python3 +# +# SPDX-License-Identifier: GPL-2.0-only +# + import os import sys,logging import optparse diff --git a/bitbake/bin/bitbake-selftest b/bitbake/bin/bitbake-selftest index cfa7ac5..041a271 100755 --- a/bitbake/bin/bitbake-selftest +++ b/bitbake/bin/bitbake-selftest @@ -2,18 +2,8 @@ # # Copyright (C) 2012 Richard Purdie # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. +# SPDX-License-Identifier: GPL-2.0-only # -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. import os import sys, logging @@ -22,6 +12,7 @@ sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__)), 'lib import unittest try: import bb + import hashserv import layerindexlib except RuntimeError as exc: sys.exit(str(exc)) @@ -33,7 +24,10 @@ tests = ["bb.tests.codeparser", "bb.tests.event", "bb.tests.fetch", "bb.tests.parse", + "bb.tests.persist_data", + "bb.tests.runqueue", "bb.tests.utils", + "hashserv.tests", "layerindexlib.tests.layerindexobj", "layerindexlib.tests.restapi", "layerindexlib.tests.cooker"] diff --git a/bitbake/bin/bitbake-worker b/bitbake/bin/bitbake-worker index e925054..6776cad 100755 --- a/bitbake/bin/bitbake-worker +++ b/bitbake/bin/bitbake-worker @@ -1,4 +1,7 @@ #!/usr/bin/env python3 +# +# SPDX-License-Identifier: GPL-2.0-only +# import os import sys @@ -136,7 +139,7 @@ def sigterm_handler(signum, frame): os.killpg(0, signal.SIGTERM) sys.exit() -def fork_off_task(cfg, data, databuilder, workerdata, fn, task, taskname, appends, taskdepdata, extraconfigdata, quieterrors=False, dry_run_exec=False): +def fork_off_task(cfg, data, databuilder, workerdata, fn, task, taskname, taskhash, unihash, appends, taskdepdata, extraconfigdata, quieterrors=False, dry_run_exec=False): # We need to setup the environment BEFORE the fork, since # a fork() or exec*() activates PSEUDO... @@ -231,10 +234,13 @@ def fork_off_task(cfg, data, databuilder, workerdata, fn, task, taskname, append the_data.setVar(varname, value) bb.parse.siggen.set_taskdata(workerdata["sigdata"]) + if "newhashes" in workerdata: + bb.parse.siggen.set_taskhashes(workerdata["newhashes"]) ret = 0 the_data = bb_cache.loadDataFull(fn, appends) - the_data.setVar('BB_TASKHASH', workerdata["runq_hash"][task]) + the_data.setVar('BB_TASKHASH', taskhash) + the_data.setVar('BB_UNIHASH', unihash) bb.utils.set_process_name("%s:%s" % (the_data.getVar("PN"), taskname.replace("do_", ""))) @@ -373,6 +379,7 @@ class BitbakeWorker(object): self.handle_item(b"cookerconfig", self.handle_cookercfg) self.handle_item(b"extraconfigdata", self.handle_extraconfigdata) self.handle_item(b"workerdata", self.handle_workerdata) + self.handle_item(b"newtaskhashes", self.handle_newtaskhashes) self.handle_item(b"runtask", self.handle_runtask) self.handle_item(b"finishnow", self.handle_finishnow) self.handle_item(b"ping", self.handle_ping) @@ -411,6 +418,10 @@ class BitbakeWorker(object): bb.msg.loggerDefaultDomains = self.workerdata["logdefaultdomain"] for mc in self.databuilder.mcdata: self.databuilder.mcdata[mc].setVar("PRSERV_HOST", self.workerdata["prhost"]) + self.databuilder.mcdata[mc].setVar("BB_HASHSERVE", self.workerdata["hashservaddr"]) + + def handle_newtaskhashes(self, data): + self.workerdata["newhashes"] = pickle.loads(data) def handle_ping(self, _): workerlog_write("Handling ping\n") @@ -425,10 +436,10 @@ class BitbakeWorker(object): sys.exit(0) def handle_runtask(self, data): - fn, task, taskname, quieterrors, appends, taskdepdata, dry_run_exec = pickle.loads(data) + fn, task, taskname, taskhash, unihash, quieterrors, appends, taskdepdata, dry_run_exec = pickle.loads(data) workerlog_write("Handling runtask %s %s %s\n" % (task, fn, taskname)) - pid, pipein, pipeout = fork_off_task(self.cookercfg, self.data, self.databuilder, self.workerdata, fn, task, taskname, appends, taskdepdata, self.extraconfigdata, quieterrors, dry_run_exec) + pid, pipein, pipeout = fork_off_task(self.cookercfg, self.data, self.databuilder, self.workerdata, fn, task, taskname, taskhash, unihash, appends, taskdepdata, self.extraconfigdata, quieterrors, dry_run_exec) self.build_pids[pid] = task self.build_pipes[pid] = runQueueWorkerPipe(pipein, pipeout) diff --git a/bitbake/bin/bitdoc b/bitbake/bin/bitdoc index 2744678..9bd02be 100755 --- a/bitbake/bin/bitdoc +++ b/bitbake/bin/bitdoc @@ -1,21 +1,9 @@ #!/usr/bin/env python3 -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- # # Copyright (C) 2005 Holger Hans Peter Freyther # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. +# SPDX-License-Identifier: GPL-2.0-only # -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. import optparse, os, sys diff --git a/bitbake/bin/git-make-shallow b/bitbake/bin/git-make-shallow index 296d3a3..57069f7 100755 --- a/bitbake/bin/git-make-shallow +++ b/bitbake/bin/git-make-shallow @@ -1,4 +1,8 @@ #!/usr/bin/env python3 +# +# SPDX-License-Identifier: GPL-2.0-only +# + """git-make-shallow: make the current git repository shallow Remove the history of the specified revisions, then optionally filter the diff --git a/bitbake/bin/toaster b/bitbake/bin/toaster index ecf66fa..c3472df 100755 --- a/bitbake/bin/toaster +++ b/bitbake/bin/toaster @@ -3,19 +3,9 @@ # toaster - shell script to start Toaster # Copyright (C) 2013-2015 Intel Corp. - -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation; either version 2 of the License, or -# (at your option) any later version. # -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. +# SPDX-License-Identifier: GPL-2.0-or-later # -# You should have received a copy of the GNU General Public License -# along with this program. If not, see http://www.gnu.org/licenses/. HELP=" Usage: source toaster start|stop [webport=] [noweb] [nobuild] [toasterdir] diff --git a/bitbake/bin/toaster-eventreplay b/bitbake/bin/toaster-eventreplay index 80967a0..8fa4ab7 100755 --- a/bitbake/bin/toaster-eventreplay +++ b/bitbake/bin/toaster-eventreplay @@ -1,25 +1,12 @@ #!/usr/bin/env python3 -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- # # Copyright (C) 2014 Alex Damian # +# SPDX-License-Identifier: GPL-2.0-only +# # This file re-uses code spread throughout other Bitbake source files. # As such, all other copyrights belong to their own right holders. # -# -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. """ This command takes a filename as a single parameter. The filename is read diff --git a/bitbake/classes/base.bbclass b/bitbake/classes/base.bbclass index 71d9ec1..08441fe 100644 --- a/bitbake/classes/base.bbclass +++ b/bitbake/classes/base.bbclass @@ -44,7 +44,7 @@ python do_showdata() { # emit the metadata which isnt valid shell for e in bb.data.keys(d): if d.getVarFlag(e, 'python', False): - bb.plain("\npython %s () {\n%s}" % (e, d.getVar(e, True))) + bb.plain("\npython %s () {\n%s}" % (e, d.getVar(e))) } addtask listtasks diff --git a/bitbake/contrib/dump_cache.py b/bitbake/contrib/dump_cache.py index 8963ca4..c6723cb 100755 --- a/bitbake/contrib/dump_cache.py +++ b/bitbake/contrib/dump_cache.py @@ -1,6 +1,4 @@ #!/usr/bin/env python3 -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- # # Copyright (C) 2012, 2018 Wind River Systems, Inc. # diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml index f1caaec..46dafee 100644 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml @@ -31,7 +31,7 @@ Prior to executing BitBake, you should take advantage of available parallel thread execution on your build host by setting the - BB_NUMBER_THREADS + BB_NUMBER_THREADS variable in your project's local.conf configuration file. @@ -87,9 +87,9 @@ The layer.conf files are used to construct key variables such as - BBPATH + BBPATH and - BBFILES. + BBFILES. BBPATH is used to search for configuration and class files under the conf and classes @@ -117,19 +117,19 @@ at certain variables, including: - BB_ENV_WHITELIST + BB_ENV_WHITELIST - BB_ENV_EXTRAWHITE + BB_ENV_EXTRAWHITE - BB_PRESERVE_ENV + BB_PRESERVE_ENV - BB_ORIGENV + BB_ORIGENV - BITBAKE_UI + BITBAKE_UI The first four variables in this list relate to how BitBake treats shell @@ -156,7 +156,7 @@ BitBake first searches the current working directory for an optional conf/bblayers.conf configuration file. This file is expected to contain a - BBLAYERS + BBLAYERS variable that is a space-delimited list of 'layer' directories. Recall that if BitBake cannot find a bblayers.conf file, then it is assumed the user has set the BBPATH @@ -166,10 +166,10 @@ For each directory (layer) in this list, a conf/layer.conf file is located and parsed with the - LAYERDIR + LAYERDIR variable being set to the directory where the layer was found. The idea is these files automatically set up - BBPATH + BBPATH and other variables correctly for a given build directory. @@ -189,7 +189,7 @@ depending on the environment variables previously mentioned or set in the configuration files. The - "Variables Glossary" + "Variables Glossary" chapter presents a full list of variables. @@ -204,7 +204,7 @@ The base.bbclass file is always included. Other classes that are specified in the configuration using the - INHERIT + INHERIT variable are also included. BitBake searches for class files in a classes subdirectory under @@ -270,7 +270,7 @@ During the configuration phase, BitBake will have set - BBFILES. + BBFILES. BitBake now uses it to construct a list of recipes to parse, along with any append files (.bbappend) to apply. @@ -292,7 +292,7 @@ Any inherit statements cause BitBake to find and then parse class files (.bbclass) using - BBPATH + BBPATH as the search path. Finally, BitBake parses in order any append files found in BBFILES. @@ -303,8 +303,8 @@ pieces of metadata. For example, in bitbake.conf the recipe name and version are used to set the variables - PN and - PV: + PN and + PV: PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}" PV = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[1] or '1.0'}" @@ -336,7 +336,7 @@ recipe information. The validity of this cache is determined by first computing a checksum of the base configuration data (see - BB_HASHCONFIG_WHITELIST) + BB_HASHCONFIG_WHITELIST) and then checking if the checksum matches. If that checksum matches what is in the cache and the recipe and class files have not changed, Bitbake is able to use @@ -384,9 +384,9 @@ the recipe can be known. Each recipe's PROVIDES list is created implicitly through the recipe's - PN variable + PN variable and explicitly through the recipe's - PROVIDES + PROVIDES variable, which is optional. @@ -427,7 +427,7 @@ PREFERRED_PROVIDER_virtual/kernel = "linux-yocto" The default - PREFERRED_PROVIDER + PREFERRED_PROVIDER is the provider with the same name as the target. Bitbake iterates through each target it needs to build and resolves them and their dependencies using this process. @@ -439,10 +439,10 @@ BitBake defaults to the highest version of a provider. Version comparisons are made using the same method as Debian. You can use the - PREFERRED_VERSION + PREFERRED_VERSION variable to specify a particular version. You can influence the order by using the - DEFAULT_PREFERENCE + DEFAULT_PREFERENCE variable. @@ -464,7 +464,7 @@ BitBake defaults to selecting the most recent version, unless otherwise specified. If the recipe in question has a - DEFAULT_PREFERENCE + DEFAULT_PREFERENCE set lower than the other recipes (default is 0), then it will not be selected. This allows the person or persons maintaining @@ -475,9 +475,9 @@ If the first recipe is named a_1.1.bb, then the - PN variable + PN variable will be set to “a”, and the - PV + PV variable will be set to 1.1. @@ -532,11 +532,11 @@ Dependencies are defined through several variables. You can find information about variables BitBake uses in - the Variables Glossary + the Variables Glossary near the end of this manual. At a basic level, it is sufficient to know that BitBake uses the - DEPENDS and - RDEPENDS variables when + DEPENDS and + RDEPENDS variables when calculating dependencies. @@ -560,7 +560,7 @@ The build now starts with BitBake forking off threads up to the limit set in the - BB_NUMBER_THREADS + BB_NUMBER_THREADS variable. BitBake continues to fork threads as long as there are tasks ready to run, those tasks have all their dependencies met, and the thread threshold has not been @@ -574,7 +574,7 @@ As each task completes, a timestamp is written to the directory specified by the - STAMP variable. + STAMP variable. On subsequent runs, BitBake looks in the build directory within tmp/stamps and does not rerun tasks that are already completed unless a timestamp is found to be invalid. @@ -618,7 +618,7 @@ Tasks can be either a shell task or a Python task. For shell tasks, BitBake writes a shell script to - ${T}/run.do_taskname.pid + ${T}/run.do_taskname.pid and then executes the script. The generated shell script contains all the exported variables, and the shell functions with all variables expanded. @@ -645,10 +645,10 @@ behavior: - BB_SCHEDULER + BB_SCHEDULER - BB_SCHEDULERS + BB_SCHEDULERS It is possible to have functions run before and after a task's main @@ -684,7 +684,7 @@ The simplistic approach for excluding the working directory is to set it to some fixed value and create the checksum for the "run" script. BitBake goes one step better and uses the - BB_HASHBASE_WHITELIST + BB_HASHBASE_WHITELIST variable to define a list of variables that should never be included when generating the signatures. @@ -795,7 +795,7 @@ This results in any metadata change that changes the task hash, automatically causing the task to be run again. This removes the need to bump - PR + PR values, and changes to metadata automatically ripple across the build. @@ -884,7 +884,7 @@ BitBake first calls the function defined by the - BB_HASHCHECK_FUNCTION + BB_HASHCHECK_FUNCTION variable with a list of tasks and corresponding hashes it wants to build. This function is designed to be fast and returns a list @@ -908,7 +908,7 @@ For example, it is pointless to obtain a compiler if you already have the compiled binary. To handle this, BitBake calls the - BB_SETSCENE_DEPVALID + BB_SETSCENE_DEPVALID function for each successful setscene task to know whether or not it needs to obtain the dependencies of that task. @@ -916,7 +916,7 @@ Finally, after all the setscene tasks have executed, BitBake calls the function listed in - BB_SETSCENE_VERIFY_FUNCTION2 + BB_SETSCENE_VERIFY_FUNCTION2 with the list of tasks BitBake thinks has been "covered". The metadata can then ensure that this list is correct and can inform BitBake that it wants specific tasks to be run regardless diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml index 29ae486..6840408 100644 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml @@ -44,7 +44,7 @@ This code sets up an instance of the fetch class. The instance uses a space-separated list of URLs from the - SRC_URI + SRC_URI variable and then calls the download method to download the files. @@ -78,7 +78,7 @@ Pre-mirror Sites: BitBake first uses pre-mirrors to try and find source files. These locations are defined using the - PREMIRRORS + PREMIRRORS variable. Source URI: @@ -88,7 +88,7 @@ Mirror Sites: If fetch failures occur, BitBake next uses mirror locations as defined by the - MIRRORS + MIRRORS variable. @@ -144,7 +144,7 @@ Any source files that are not local (i.e. downloaded from the Internet) are placed into the download directory, which is specified by the - DL_DIR + DL_DIR variable. @@ -184,11 +184,11 @@ If - BB_STRICT_CHECKSUM + BB_STRICT_CHECKSUM is set, any download without a checksum triggers an error message. The - BB_NO_NETWORK + BB_NO_NETWORK variable can be used to make any attempted network access a fatal error, which is useful for checking that mirrors are complete as well as other things. @@ -265,11 +265,11 @@ The filename you specify within the URL can be either an absolute or relative path to a file. If the filename is relative, the contents of the - FILESPATH + FILESPATH variable is used in the same way PATH is used to find executables. If the file cannot be found, it is assumed that it is available in - DL_DIR + DL_DIR by the time the download() method is called. @@ -304,7 +304,7 @@ allows the name of the downloaded file to be specified. Specifying the name of the downloaded file is useful for avoiding collisions in - DL_DIR + DL_DIR when dealing with multiple files that have the same name. @@ -355,7 +355,7 @@ A special value of "now" causes the checkout to be updated on every build. - CVSDIR: + CVSDIR: Specifies where a temporary checkout is saved. The location is often DL_DIR/cvs. @@ -395,7 +395,7 @@ "date": Specifies a date. If no "date" is specified, the - SRCDATE + SRCDATE of the configuration is used to checkout a specific date. The special value of "now" causes the checkout to be updated on every build. @@ -406,7 +406,7 @@ to which the module is unpacked. You are forcing the module into a special directory relative to - CVSDIR. + CVSDIR. "rsh" Used in conjunction with the "method" parameter. @@ -448,7 +448,7 @@ FETCHCMD_svn, which defaults to "svn". The fetcher's temporary working directory is set by - SVNDIR, + SVNDIR, which is usually DL_DIR/svn. @@ -509,7 +509,7 @@ source control system. The fetcher works by creating a bare clone of the remote into - GITDIR, + GITDIR, which is usually DL_DIR/git2. This bare clone is then cloned into the work directory during the unpack stage when a specific tree is checked out. @@ -588,6 +588,14 @@ The name of the path in which to place the checkout. By default, the path is git/. + "usehead": + Enables local git:// URLs to use the + current branch HEAD as the revision for use with + AUTOREV. + The "usehead" parameter implies no branch and only works + when the transfer protocol is + file://. + Here are some example URLs: @@ -604,7 +612,7 @@ This fetcher submodule inherits from the Git fetcher and extends that fetcher's behavior by fetching a repository's submodules. - SRC_URI + SRC_URI is passed to the Git fetcher as described in the "Git Fetcher (git://)" section. @@ -639,9 +647,9 @@ To use this fetcher, make sure your recipe has proper - SRC_URI, - SRCREV, and - PV settings. + SRC_URI, + SRCREV, and + PV settings. Here is an example: SRC_URI = "ccrc://cc.example.org/ccrc;vob=/example_vob;module=/example_module" @@ -726,15 +734,15 @@ FETCHCMD_p4, which defaults to "p4". The fetcher's temporary working directory is set by - P4DIR, + P4DIR, which defaults to "DL_DIR/p4". To use this fetcher, make sure your recipe has proper - SRC_URI, - SRCREV, and - PV values. + SRC_URI, + SRCREV, and + PV values. The p4 executable is able to use the config file defined by your system's P4CONFIG environment variable in order to define the Perforce server URL and port, username, and @@ -785,9 +793,9 @@ google-repo source control system. The fetcher works by initiating and syncing sources of the repository into - REPODIR, + REPODIR, which is usually - DL_DIR/repo. + DL_DIR/repo. @@ -824,19 +832,22 @@ Bazaar (bzr://) - Trees using Git Annex (gitannex://) + Mercurial (hg://) - Secure FTP (sftp://) + npm (npm://) - Secure Shell (ssh://) + OSC (osc://) - OSC (osc://) + Secure FTP (sftp://) - Mercurial (hg://) + Secure Shell (ssh://) + + + Trees using Git Annex (gitannex://) No documentation currently exists for these lesser used diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.xml b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.xml index 9076f0f..39066e4 100644 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.xml +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.xml @@ -194,7 +194,7 @@ When you run BitBake, it begins looking for metadata files. The - BBPATH + BBPATH variable is what tells BitBake where to look for those files. BBPATH is not set and you need to set it. Without BBPATH, Bitbake cannot @@ -273,14 +273,14 @@ some editor to create the bitbake.conf so that it contains the following: - PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}" + PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}" - TMPDIR = "${TOPDIR}/tmp" - CACHE = "${TMPDIR}/cache" - STAMP = "${TMPDIR}/${PN}/stamps" - T = "${TMPDIR}/${PN}/work" - B = "${TMPDIR}/${PN}" + TMPDIR = "${TOPDIR}/tmp" + CACHE = "${TMPDIR}/cache" + STAMP = "${TMPDIR}/${PN}/stamps" + T = "${TMPDIR}/${PN}/work" + B = "${TMPDIR}/${PN}" Without a value for PN, the @@ -402,12 +402,12 @@ Move to the conf directory and create a layer.conf file that has the following: - BBPATH .= ":${LAYERDIR}" + BBPATH .= ":${LAYERDIR}" - BBFILES += "${LAYERDIR}/*.bb" + BBFILES += "${LAYERDIR}/*.bb" - BBFILE_COLLECTIONS += "mylayer" - BBFILE_PATTERN_mylayer := "^${LAYERDIR_RE}/" + BBFILE_COLLECTIONS += "mylayer" + BBFILE_PATTERN_mylayer := "^${LAYERDIR_RE}/" For information on these variables, click the links to go to the definitions in the glossary. @@ -416,9 +416,9 @@ a recipe file named printhello.bb that has the following: - DESCRIPTION = "Prints Hello World" - PN = 'printhello' - PV = '1' + DESCRIPTION = "Prints Hello World" + PN = 'printhello' + PV = '1' python do_build() { bb.plain("********************"); diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml index f7d312a..8f2a960 100644 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml @@ -781,7 +781,7 @@ target, you must also enable BitBake to perform multiple configuration builds. Enabling is accomplished by setting the - BBMULTICONFIG + BBMULTICONFIG variable in the local.conf configuration file. As an example, suppose you had configuration files @@ -791,7 +791,7 @@ The following statement in the local.conf file both enables BitBake to perform multiple configuration builds and - specifies the two multiconfigs: + specifies the two extra multiconfigs: BBMULTICONFIG = "target1 target2" @@ -803,13 +803,13 @@ builds, use the following command form to start the builds: - $ bitbake [multiconfig:multiconfigname:]target [[[multiconfig:multiconfigname:]target] ... ] + $ bitbake [mc:multiconfigname:]target [[[mc:multiconfigname:]target] ... ] - Here is an example for two multiconfigs: + Here is an example for two extra multiconfigs: target1 and target2: - $ bitbake multiconfig:target1:target multiconfig:target2:target + $ bitbake mc::target mc:target1:target mc:target2:target @@ -837,13 +837,13 @@ build, you must declare the dependencies in the recipe using the following statement form: - task_or_package[mcdepends] = "multiconfig:from_multiconfig:to_multiconfig:recipe_name:task_on_which_to_depend" + task_or_package[mcdepends] = "mc:from_multiconfig:to_multiconfig:recipe_name:task_on_which_to_depend" To better show how to use this statement, consider an example with two multiconfigs: target1 and target2: - image_task[mcdepends] = "multiconfig:target1:target2:image2:rootfs_task" + image_task[mcdepends] = "mc:target1:target2:image2:rootfs_task" In this example, the from_multiconfig is "target1" and @@ -859,7 +859,7 @@ Once you set up this dependency, you can build the "target1" multiconfig using a BitBake command as follows: - $ bitbake multiconfig:target1:image1 + $ bitbake mc:target1:image1 This command executes all the tasks needed to create image1 for the "target1" @@ -875,7 +875,7 @@ Consider this change to the statement in the image1 recipe: - image_task[mcdepends] = "multiconfig:target1:target2:image2:image_task" + image_task[mcdepends] = "mc:target1:target2:image2:image_task" In this case, BitBake must create image2 for the "target2" diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml index 2490f6e..421364c 100644 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml @@ -61,6 +61,78 @@ +
+ Modifying Existing Variables + + + Sometimes you need to modify existing variables. + Following are some cases where you might find you want to + modify an existing variable: + + + Customize a recipe that uses the variable. + + + Change a variable's default value used in a + *.bbclass file. + + + Change the variable in a *.bbappend + file to override the variable in the original recipe. + + + Change the variable in a configuration file so that the + value overrides an existing configuration. + + + + + + Changing a variable value can sometimes depend on how the + value was originally assigned and also on the desired + intent of the change. + In particular, when you append a value to a variable that + has a default value, the resulting value might not be what + you expect. + In this case, the value you provide might replace the value + rather than append to the default value. + + + + If after you have changed a variable's value and something + unexplained occurs, you can use BitBake to check the actual + value of the suspect variable. + You can make these checks for both configuration and recipe + level changes: + + + For configuration changes, use the following: + + $ bitbake -e + + This command displays variable values after the + configuration files (i.e. local.conf, + bblayers.conf, + bitbake.conf and so forth) have + been parsed. + + Variables that are exported to the environment are + preceded by the string "export" in the command's + output. + + + + For recipe changes, use the following: + + $ bitbake recipe -e | grep VARIABLE=" + + This command checks to see if the variable actually + makes it into a specific recipe. + + + +
+
Line Joining @@ -297,9 +369,8 @@ These operators differ from the ":=", ".=", "=.", "+=", and "=+" - operators in that their effects are deferred - until after parsing completes rather than being immediately - applied. + operators in that their effects are applied at variable + expansion time rather than being immediately applied. Here are some examples: B = "bval" @@ -348,18 +419,22 @@ FOO = "123 456 789 123456 123 456 123 456" FOO_remove = "123" FOO_remove = "456" - FOO2 = "abc def ghi abcdef abc def abc def" - FOO2_remove = "abc def" + FOO2 = " abc def ghi abcdef abc def abc def def" + FOO2_remove = " \ + def \ + abc \ + ghi \ + " The variable FOO becomes - "  789 123456    " + "  789 123456    " and FOO2 becomes - "  ghi abcdef    ". + "     jkl  abcdef      ". Like "_append" and "_prepend", "_remove" - is deferred until after parsing completes. + is applied at variable expansion time.
@@ -503,7 +578,7 @@
- Unseting variables + Unsetting variables It is possible to completely remove a variable or a variable flag @@ -595,7 +670,7 @@ BitBake uses - OVERRIDES + OVERRIDES to control what variables are overridden after BitBake parses recipes and configuration files. This section describes how you can use @@ -705,7 +780,7 @@ Internally, this is implemented by prepending the task (e.g. "task-compile:") to the value of - OVERRIDES + OVERRIDES for the local datastore of the do_compile task. @@ -724,17 +799,15 @@ Key Expansion - Key expansion happens when the BitBake datastore is finalized - just before BitBake expands overrides. + Key expansion happens when the BitBake datastore is finalized. To better understand this, consider the following example: A${B} = "X" B = "2" A2 = "Y" - In this case, after all the parsing is complete, and - before any overrides are handled, BitBake expands - ${B} into "2". + In this case, after all the parsing is complete, + BitBake expands ${B} into "2". This expansion causes A2, which was set to "Y" before the expansion, to become "X". @@ -868,7 +941,7 @@ BitBake uses the - BBPATH + BBPATH variable to locate needed include and class files. Additionally, BitBake searches the current directory for include and require @@ -1086,7 +1159,7 @@ When creating a configuration file (.conf), you can use the - INHERIT + INHERIT configuration directive to inherit a class. BitBake only supports this directive when used within a configuration file. @@ -1370,7 +1443,7 @@ BitBake-style Python functions generate a separate - ${T}/run.function-name.pid + ${T}/run.function-name.pid script that is executed to run the function, and also generate a log file in ${T}/log.function-name.pid @@ -1773,7 +1846,7 @@ things exported or listed in its whitelist to ensure that the build environment is reproducible and consistent. You can prevent this "cleaning" by setting the - BB_PRESERVE_ENV + BB_PRESERVE_ENV variable. Consequently, if you do want something to get passed into the @@ -1783,9 +1856,9 @@ Tell BitBake to load what you want from the environment into the datastore. You can do so through the - BB_ENV_WHITELIST + BB_ENV_WHITELIST and - BB_ENV_EXTRAWHITE + BB_ENV_EXTRAWHITE variables. For example, assume you want to prevent the build system from accessing your $HOME/.ccache @@ -1824,7 +1897,7 @@ from the original execution environment. Bitbake saves a copy of the original environment into a special variable named - BB_ORIGENV. + BB_ORIGENV. @@ -1883,7 +1956,7 @@ [depends]: Controls inter-task dependencies. See the - DEPENDS + DEPENDS variable and the "Inter-Task Dependencies" section for more information. @@ -1891,7 +1964,7 @@ [deptask]: Controls task build-time dependencies. See the - DEPENDS + DEPENDS variable and the "Build Dependencies" section for more information. @@ -1937,7 +2010,7 @@ of cores but certain tasks need to be rate-limited due to various kinds of resource constraints (e.g. to avoid network throttling). number_threads works similarly to the - BB_NUMBER_THREADS + BB_NUMBER_THREADS variable but is task-specific. Set the value globally. @@ -1971,9 +2044,9 @@ [rdepends]: Controls inter-task runtime dependencies. See the - RDEPENDS + RDEPENDS variable, the - RRECOMMENDS + RRECOMMENDS variable, and the "Inter-Task Dependencies" section for more information. @@ -1981,9 +2054,9 @@ [rdeptask]: Controls task runtime dependencies. See the - RDEPENDS + RDEPENDS variable, the - RRECOMMENDS + RRECOMMENDS variable, and the "Runtime Dependencies" section for more information. @@ -1996,9 +2069,9 @@ [recrdeptask]: Controls task recursive runtime dependencies. See the - RDEPENDS + RDEPENDS variable, the - RRECOMMENDS + RRECOMMENDS variable, and the "Recursive Dependencies" section for more information. @@ -2127,7 +2200,7 @@ Any given datastore only has one such event executed against it, however. If - BB_INVALIDCONF + BB_INVALIDCONF is set in the datastore by the event handler, the configuration is reparsed and a new event triggered, allowing the metadata to update configuration. @@ -2256,17 +2329,17 @@ from a single recipe file multiple incarnations of that recipe file where all incarnations are buildable. These features are enabled through the - BBCLASSEXTEND + BBCLASSEXTEND and - BBVERSIONS + BBVERSIONS variables. The mechanism for this class extension is extremely specific to the implementation. Usually, the recipe's - PROVIDES, - PN, and - DEPENDS + PROVIDES, + PN, and + DEPENDS variables would need to be modified by the extension class. For specific examples, see the OE-Core native, nativesdk, @@ -2287,7 +2360,7 @@ project from a single recipe file. You can also specify conditional metadata (using the - OVERRIDES + OVERRIDES mechanism) for a single version, or an optionally named range of versions. Here is an example: @@ -2306,7 +2379,7 @@ into overrides, but it is also made available for the metadata to use in the variable that defines the base recipe versions for use in file:// search paths - (FILESPATH). + (FILESPATH). @@ -2408,7 +2481,7 @@ BitBake uses the - DEPENDS + DEPENDS variable to manage build time dependencies. The [deptask] varflag for tasks signifies the task of each @@ -2429,9 +2502,9 @@ BitBake uses the - PACKAGES, - RDEPENDS, and - RRECOMMENDS + PACKAGES, + RDEPENDS, and + RRECOMMENDS variables to manage runtime dependencies. @@ -2686,7 +2759,7 @@ These checksums are stored in - STAMP. + STAMP. You can examine the checksums using the following BitBake command: $ bitbake-dumpsigs @@ -2708,44 +2781,44 @@ The following list describes related variables: - BB_HASHCHECK_FUNCTION: + BB_HASHCHECK_FUNCTION: Specifies the name of the function to call during the "setscene" part of the task's execution in order to validate the list of task hashes. - BB_SETSCENE_DEPVALID: + BB_SETSCENE_DEPVALID: Specifies a function BitBake calls that determines whether BitBake requires a setscene dependency to be met. - BB_SETSCENE_VERIFY_FUNCTION2: + BB_SETSCENE_VERIFY_FUNCTION2: Specifies a function to call that verifies the list of planned task execution before the main task execution happens. - BB_STAMP_POLICY: + BB_STAMP_POLICY: Defines the mode for comparing timestamps of stamp files. - BB_STAMP_WHITELIST: + BB_STAMP_WHITELIST: Lists stamp files that are looked at when the stamp policy is "whitelist". - BB_TASKHASH: + BB_TASKHASH: Within an executing task, this variable holds the hash of the task as returned by the currently enabled signature generator. - STAMP: + STAMP: The base path to create stamp files. - STAMPCLEAN: + STAMPCLEAN: Again, the base path to create stamp files but can use wildcards for matching a range of files for clean operations. diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml index a84b2bc..aca6741 100644 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml @@ -3,7 +3,7 @@ [ %poky; ] > - + Variables Glossary @@ -34,29 +34,29 @@ - + - A - B - C - D - E - F - G - H - + A + B + C + D + E + F + G + H + I - L - M + L + M - O - P + O + P - R - S - T + R + S + T @@ -65,13 +65,13 @@ - A + A - ASSUME_PROVIDED + ASSUME_PROVIDED Lists recipe names - (PN + (PN values) BitBake does not attempt to build. Instead, BitBake assumes these recipes have already been built. @@ -91,9 +91,9 @@ - B + B - B + B The directory in which BitBake executes functions @@ -102,7 +102,7 @@ - BB_ALLOWED_NETWORKS + BB_ALLOWED_NETWORKS Specifies a space-delimited list of hosts that the fetcher @@ -111,7 +111,7 @@ This host list is only used if - BB_NO_NETWORK + BB_NO_NETWORK is either not set or set to "0". @@ -151,13 +151,13 @@ Using BB_ALLOWED_NETWORKS in conjunction with - PREMIRRORS + PREMIRRORS is very useful. Adding the host you want to use to PREMIRRORS results in the source code being fetched from an allowed location and avoids raising an error when a host that is not allowed is in a - SRC_URI + SRC_URI statement. This is because the fetcher does not attempt to use the host listed in SRC_URI after a @@ -167,7 +167,7 @@ - BB_CONSOLELOG + BB_CONSOLELOG Specifies the path to a log file into which BitBake's user @@ -176,7 +176,7 @@ - BB_CURRENTTASK + BB_CURRENTTASK Contains the name of the currently running task. @@ -186,7 +186,7 @@ - BB_DANGLINGAPPENDS_WARNONLY + BB_DANGLINGAPPENDS_WARNONLY Defines how BitBake handles situations where an append @@ -208,7 +208,7 @@ - BB_DEFAULT_TASK + BB_DEFAULT_TASK The default task to use when none is specified (e.g. @@ -219,7 +219,7 @@ - BB_DISKMON_DIRS + BB_DISKMON_DIRS Monitors disk space and available inodes during the build @@ -245,7 +245,7 @@ build when a threshold is broken. Subsequent warnings are issued as defined by the - BB_DISKMON_WARNINTERVAL variable, + BB_DISKMON_WARNINTERVAL variable, which must be defined. <dir> is: @@ -275,7 +275,7 @@ BB_DISKMON_DIRS = "ABORT,${TMPDIR},,100K" The first example works only if you also set - the BB_DISKMON_WARNINTERVAL variable. + the BB_DISKMON_WARNINTERVAL variable. This example causes the build system to immediately abort when either the disk space in ${TMPDIR} drops below 1 Gbyte or the available free inodes drops below @@ -309,7 +309,7 @@ - BB_DISKMON_WARNINTERVAL + BB_DISKMON_WARNINTERVAL Defines the disk space and free inode warning intervals. @@ -319,7 +319,7 @@ If you are going to use the BB_DISKMON_WARNINTERVAL variable, you must also use the - BB_DISKMON_DIRS variable + BB_DISKMON_DIRS variable and define its action as "WARN". During the build, subsequent warnings are issued each time disk space or number of free inodes further reduces by @@ -374,7 +374,7 @@ - BB_ENV_WHITELIST + BB_ENV_WHITELIST Specifies the internal whitelist of variables to allow @@ -382,11 +382,11 @@ datastore. If the value of this variable is not specified (which is the default), the following list is used: - BBPATH, - BB_PRESERVE_ENV, - BB_ENV_WHITELIST, + BBPATH, + BB_PRESERVE_ENV, + BB_ENV_WHITELIST, and - BB_ENV_EXTRAWHITE. + BB_ENV_EXTRAWHITE. You must set this variable in the external environment in order for it to work. @@ -395,7 +395,7 @@ - BB_ENV_EXTRAWHITE + BB_ENV_EXTRAWHITE Specifies an additional set of variables to allow through @@ -403,7 +403,7 @@ datastore. This list of variables are on top of the internal list set in - BB_ENV_WHITELIST. + BB_ENV_WHITELIST. You must set this variable in the external environment in order for it to work. @@ -412,22 +412,22 @@ - BB_FETCH_PREMIRRORONLY + BB_FETCH_PREMIRRORONLY When set to "1", causes BitBake's fetcher module to only search - PREMIRRORS + PREMIRRORS for files. BitBake will not search the main - SRC_URI + SRC_URI or - MIRRORS. + MIRRORS. - BB_FILENAME + BB_FILENAME Contains the filename of the recipe that owns the currently @@ -440,12 +440,12 @@ - BB_GENERATE_MIRROR_TARBALLS + BB_GENERATE_MIRROR_TARBALLS Causes tarballs of the Git repositories, including the Git metadata, to be placed in the - DL_DIR + DL_DIR directory. Anyone wishing to create a source mirror would want to enable this variable. @@ -461,7 +461,7 @@ - BB_HASHCONFIG_WHITELIST + BB_HASHCONFIG_WHITELIST Lists variables that are excluded from base configuration @@ -485,7 +485,7 @@ - BB_HASHBASE_WHITELIST + BB_HASHBASE_WHITELIST Lists variables that are excluded from checksum and @@ -500,7 +500,7 @@ - BB_HASHCHECK_FUNCTION + BB_HASHCHECK_FUNCTION Specifies the name of the function to call during the @@ -524,7 +524,7 @@ - BB_INVALIDCONF + BB_INVALIDCONF Used in combination with the @@ -539,11 +539,11 @@ - BB_LOGFMT + BB_LOGFMT Specifies the name of the log files saved into - ${T}. + ${T}. By default, the BB_LOGFMT variable is undefined and the log file names get created using the following form: @@ -556,7 +556,7 @@ - BB_NICE_LEVEL + BB_NICE_LEVEL Allows BitBake to run at a specific priority @@ -564,13 +564,13 @@ System permissions usually mean that BitBake can reduce its priority but not raise it again. See - BB_TASK_NICE_LEVEL + BB_TASK_NICE_LEVEL for additional information. - BB_NO_NETWORK + BB_NO_NETWORK Disables network access in the BitBake fetcher modules. @@ -587,7 +587,7 @@ - BB_NUMBER_THREADS + BB_NUMBER_THREADS The maximum number of tasks BitBake should run in parallel @@ -599,7 +599,7 @@ - BB_NUMBER_PARSE_THREADS + BB_NUMBER_PARSE_THREADS Sets the number of threads BitBake uses when parsing. @@ -609,7 +609,7 @@ - BB_ORIGENV + BB_ORIGENV Contains a copy of the original external environment in @@ -625,7 +625,7 @@ - BB_PRESERVE_ENV + BB_PRESERVE_ENV Disables whitelisting and instead allows all variables @@ -639,12 +639,12 @@ - BB_RUNFMT + BB_RUNFMT Specifies the name of the executable script files (i.e. run files) saved into - ${T}. + ${T}. By default, the BB_RUNFMT variable is undefined and the run file names get created using the following form: @@ -657,7 +657,7 @@ - BB_RUNTASK + BB_RUNTASK Contains the name of the currently executing task. @@ -669,7 +669,7 @@ - BB_SCHEDULER + BB_SCHEDULER Selects the name of the scheduler to use for the @@ -695,7 +695,7 @@ - BB_SCHEDULERS + BB_SCHEDULERS Defines custom schedulers to import. @@ -705,13 +705,13 @@ For information how to select a scheduler, see the - BB_SCHEDULER + BB_SCHEDULER variable. - BB_SETSCENE_DEPVALID + BB_SETSCENE_DEPVALID Specifies a function BitBake calls that determines @@ -731,7 +731,7 @@ - BB_SETSCENE_VERIFY_FUNCTION2 + BB_SETSCENE_VERIFY_FUNCTION2 Specifies a function to call that verifies the list of @@ -752,7 +752,7 @@ - BB_SIGNATURE_EXCLUDE_FLAGS + BB_SIGNATURE_EXCLUDE_FLAGS Lists variable flags (varflags) @@ -771,7 +771,7 @@ - BB_SIGNATURE_HANDLER + BB_SIGNATURE_HANDLER Defines the name of the signature handler BitBake uses. @@ -790,7 +790,7 @@ - BB_SRCREV_POLICY + BB_SRCREV_POLICY Defines the behavior of the fetcher when it interacts with @@ -817,7 +817,7 @@ - BB_STAMP_POLICY + BB_STAMP_POLICY Defines the mode used for how timestamps of stamp files @@ -836,7 +836,7 @@ whitelist - Identical to "full" mode except timestamp comparisons are made for recipes listed in the - BB_STAMP_WHITELIST + BB_STAMP_WHITELIST variable. @@ -848,19 +848,19 @@ - BB_STAMP_WHITELIST + BB_STAMP_WHITELIST Lists files whose stamp file timestamps are compared when the stamp policy mode is set to "whitelist". For information on stamp policies, see the - BB_STAMP_POLICY + BB_STAMP_POLICY variable. - BB_STRICT_CHECKSUM + BB_STRICT_CHECKSUM Sets a more strict checksum mechanism for non-local URLs. @@ -871,7 +871,7 @@ - BB_TASK_IONICE_LEVEL + BB_TASK_IONICE_LEVEL Allows adjustment of a task's Input/Output priority. @@ -882,7 +882,7 @@ variable to adjust the I/O priority of these tasks. This variable works similarly to the - BB_TASK_NICE_LEVEL + BB_TASK_NICE_LEVEL variable except with a task's I/O priorities. @@ -921,7 +921,7 @@ - BB_TASK_NICE_LEVEL + BB_TASK_NICE_LEVEL Allows specific tasks to change their priority @@ -940,7 +940,7 @@ - BB_TASKHASH + BB_TASKHASH Within an executing task, this variable holds the hash @@ -950,7 +950,7 @@ - BB_VERBOSE_LOGS + BB_VERBOSE_LOGS Controls how verbose BitBake is during builds. @@ -960,7 +960,7 @@ - BB_WORKERCONTEXT + BB_WORKERCONTEXT Specifies if the current context is executing a task. @@ -973,7 +973,7 @@ - BBCLASSEXTEND + BBCLASSEXTEND Allows you to extend a recipe so that it builds variants @@ -1009,7 +1009,7 @@ _class-native. For example, to generate a native version of a recipe, a - DEPENDS + DEPENDS on "foo" is rewritten to a DEPENDS on "foo-native". @@ -1028,7 +1028,7 @@ - BBDEBUG + BBDEBUG Sets the BitBake debug output level to a specific value @@ -1042,7 +1042,7 @@ - BBFILE_COLLECTIONS + BBFILE_COLLECTIONS Lists the names of configured layers. These names are used to find the other BBFILE_* @@ -1053,10 +1053,10 @@ - BBFILE_PATTERN + BBFILE_PATTERN Variable that expands to match files from - BBFILES + BBFILES in a particular layer. This variable is used in the conf/layer.conf file and must be suffixed with the name of the specific layer (e.g. @@ -1064,7 +1064,7 @@ - BBFILE_PRIORITY + BBFILE_PRIORITY Assigns the priority for recipe files in each layer. This variable is useful in situations where the same recipe appears in @@ -1074,7 +1074,7 @@ letting you control the precedence for the multiple layers. The precedence established through this variable stands regardless of a recipe's version - (PV variable). + (PV variable). For example, a layer that has a recipe with a higher PV value but for which the BBFILE_PRIORITY is set to have a lower precedence still has a lower precedence. @@ -1083,7 +1083,7 @@ For example, the value 6 has a higher precedence than the value 5. If not specified, the BBFILE_PRIORITY variable is set based on layer dependencies (see the - LAYERDEPENDS variable for + LAYERDEPENDS variable for more information. The default priority, if unspecified for a layer with no dependencies, is the lowest defined priority + 1 @@ -1095,7 +1095,7 @@ - BBFILES + BBFILES A space-separated list of recipe files BitBake uses to @@ -1113,7 +1113,7 @@ - BBINCLUDED + BBINCLUDED Contains a space-separated list of all of all files that @@ -1123,7 +1123,7 @@ - BBINCLUDELOGS + BBINCLUDELOGS If set to a value, enables printing the task log when @@ -1132,11 +1132,11 @@ - BBINCLUDELOGS_LINES + BBINCLUDELOGS_LINES If - BBINCLUDELOGS + BBINCLUDELOGS is set, specifies the maximum number of lines from the task log file to print when reporting a failed task. If you do not set BBINCLUDELOGS_LINES, @@ -1145,7 +1145,7 @@ - BBLAYERS + BBLAYERS Lists the layers to enable during the build. This variable is defined in the bblayers.conf configuration @@ -1166,7 +1166,7 @@ - BBLAYERS_FETCH_DIR + BBLAYERS_FETCH_DIR Sets the base location where layers are stored. @@ -1178,7 +1178,7 @@ - BBMASK + BBMASK Prevents BitBake from processing recipes and recipe @@ -1236,7 +1236,7 @@ - BBMULTICONFIG + BBMULTICONFIG BBMULTICONFIG[doc] = "Enables BitBake to perform multiple configuration builds and lists each separate configuration (multiconfig)." @@ -1275,7 +1275,7 @@ - BBPATH + BBPATH Used by BitBake to locate class @@ -1302,7 +1302,7 @@ - BBSERVER + BBSERVER Points to the server that runs memory-resident BitBake. @@ -1312,7 +1312,7 @@ - BBTARGETS + BBTARGETS Allows you to use a configuration file to add to the list @@ -1321,14 +1321,14 @@ - BBVERSIONS + BBVERSIONS Allows a single recipe to build multiple versions of a project from a single recipe file. You also able to specify conditional metadata using the - OVERRIDES + OVERRIDES mechanism for a single version or for an optionally named range of versions. @@ -1342,7 +1342,7 @@ - BITBAKE_UI + BITBAKE_UI Used to specify the UI module to use when running BitBake. @@ -1356,7 +1356,7 @@ - BUILDNAME + BUILDNAME A name assigned to the build. @@ -1366,7 +1366,7 @@ - BZRDIR + BZRDIR The directory in which files checked out of a Bazaar @@ -1377,9 +1377,9 @@ - C + C - CACHE + CACHE Specifies the directory BitBake uses to store a cache @@ -1389,7 +1389,7 @@ - CVSDIR + CVSDIR The directory in which files checked out under the @@ -1400,9 +1400,9 @@ - D + D - DEFAULT_PREFERENCE + DEFAULT_PREFERENCE Specifies a weak bias for recipe selection priority. @@ -1413,20 +1413,20 @@ piece of software. Using the variable in this way causes the stable version of the recipe to build by default in the absence of - PREFERRED_VERSION + PREFERRED_VERSION being used to build the development version. The bias provided by DEFAULT_PREFERENCE is weak and is overridden by - BBFILE_PRIORITY + BBFILE_PRIORITY if that variable is different between two layers that contain different versions of the same recipe. - DEPENDS + DEPENDS Lists a recipe's build-time dependencies @@ -1451,13 +1451,13 @@ For information on runtime dependencies, see the - RDEPENDS + RDEPENDS variable. - DESCRIPTION + DESCRIPTION A long description for the recipe. @@ -1465,7 +1465,7 @@ - DL_DIR + DL_DIR The central download directory used by the build process to @@ -1474,7 +1474,7 @@ suitable for mirroring for everything except Git repositories. If you want tarballs of Git repositories, use the - BB_GENERATE_MIRROR_TARBALLS + BB_GENERATE_MIRROR_TARBALLS variable. @@ -1482,9 +1482,9 @@ - E + E - EXCLUDE_FROM_WORLD + EXCLUDE_FROM_WORLD Directs BitBake to exclude a recipe from world builds (i.e. @@ -1512,9 +1512,9 @@ - F + F - FAKEROOT + FAKEROOT Contains the command to use when running a shell script @@ -1527,19 +1527,19 @@ - FAKEROOTBASEENV + FAKEROOTBASEENV Lists environment variables to set when executing the command defined by - FAKEROOTCMD + FAKEROOTCMD that starts the bitbake-worker process in the fakeroot environment. - FAKEROOTCMD + FAKEROOTCMD Contains the command that starts the bitbake-worker @@ -1548,7 +1548,7 @@ - FAKEROOTDIRS + FAKEROOTDIRS Lists directories to create before running a task in @@ -1557,33 +1557,33 @@ - FAKEROOTENV + FAKEROOTENV Lists environment variables to set when running a task in the fakeroot environment. For additional information on environment variables and the fakeroot environment, see the - FAKEROOTBASEENV + FAKEROOTBASEENV variable. - FAKEROOTNOENV + FAKEROOTNOENV Lists environment variables to set when running a task that is not in the fakeroot environment. For additional information on environment variables and the fakeroot environment, see the - FAKEROOTENV + FAKEROOTENV variable. - FETCHCMD + FETCHCMD Defines the command the BitBake fetcher module @@ -1595,7 +1595,7 @@ - FILE + FILE Points at the current file. @@ -1607,7 +1607,7 @@ - FILESPATH + FILESPATH Specifies directories BitBake uses when searching for @@ -1625,9 +1625,9 @@ - G + G - GITDIR + GITDIR The directory in which a local copy of a Git repository @@ -1639,9 +1639,9 @@ - H + H - HGDIR + HGDIR The directory in which files checked out of a Mercurial @@ -1650,7 +1650,7 @@ - HOMEPAGE + HOMEPAGE Website where more information about the software the recipe is building can be found. @@ -1659,9 +1659,9 @@ - I + I - INHERIT + INHERIT Causes the named class or classes to be inherited globally. @@ -1691,15 +1691,15 @@ --> - L + L - LAYERDEPENDS + LAYERDEPENDS Lists the layers, separated by spaces, upon which this recipe depends. Optionally, you can specify a specific layer version for a dependency by adding it to the end of the layer name with a colon, (e.g. "anotherlayer:3" to be compared against - LAYERVERSION_anotherlayer + LAYERVERSION_anotherlayer in this case). BitBake produces an error if any dependency is missing or the version numbers do not match exactly (if specified). @@ -1710,7 +1710,7 @@ - LAYERDIR + LAYERDIR When used inside the layer.conf configuration file, this variable provides the path of the current layer. @@ -1719,22 +1719,22 @@ - LAYERDIR_RE + LAYERDIR_RE When used inside the layer.conf configuration file, this variable provides the path of the current layer, escaped for use in a regular expression - (BBFILE_PATTERN). + (BBFILE_PATTERN). This variable is not available outside of layer.conf and references are expanded immediately when parsing of the file completes. - LAYERVERSION + LAYERVERSION Optionally specifies the version of a layer as a single number. You can use this variable within - LAYERDEPENDS + LAYERDEPENDS for another layer in order to depend on a specific version of the layer. @@ -1744,7 +1744,7 @@ - LICENSE + LICENSE The list of source licenses for the recipe. @@ -1754,9 +1754,9 @@ - M + M - MIRRORS + MIRRORS Specifies additional paths from which BitBake gets source code. @@ -1764,14 +1764,14 @@ tries the local download directory. If that location fails, the build system tries locations defined by - PREMIRRORS, + PREMIRRORS, the upstream source, and then locations specified by MIRRORS in that order. - MULTI_PROVIDER_WHITELIST + MULTI_PROVIDER_WHITELIST Allows you to suppress BitBake warnings caused when @@ -1804,9 +1804,9 @@ --> - O + O - OVERRIDES + OVERRIDES BitBake uses OVERRIDES to control @@ -1829,9 +1829,9 @@ - P + P - P4DIR + P4DIR The directory in which a local copy of a Perforce depot @@ -1840,14 +1840,14 @@ - PACKAGES + PACKAGES The list of packages the recipe creates. - PACKAGES_DYNAMIC + PACKAGES_DYNAMIC A promise that your recipe satisfies runtime dependencies @@ -1856,7 +1856,7 @@ does not actually satisfy the dependencies, it only states that they should be satisfied. For example, if a hard, runtime dependency - (RDEPENDS) + (RDEPENDS) of another package is satisfied during the build through the PACKAGES_DYNAMIC variable, but a package with the module name is never actually @@ -1865,7 +1865,7 @@ - PE + PE The epoch of the recipe. @@ -1877,7 +1877,7 @@ - PERSISTENT_DIR + PERSISTENT_DIR Specifies the directory BitBake uses to store data that @@ -1889,7 +1889,7 @@ - PF + PF Specifies the recipe or package name and includes all version and revision @@ -1899,27 +1899,27 @@ - PN + PN The recipe name. - PR + PR The revision of the recipe. - PREFERRED_PROVIDER + PREFERRED_PROVIDER Determines which recipe should be given preference when multiple recipes provide the same item. You should always suffix the variable with the name of the provided item, and you should set it to the - PN + PN of the recipe to which you want to give precedence. Some examples: @@ -1931,14 +1931,14 @@ - PREFERRED_PROVIDERS + PREFERRED_PROVIDERS Determines which recipe should be given preference for cases where multiple recipes provide the same item. Functionally, PREFERRED_PROVIDERS is identical to - PREFERRED_PROVIDER. + PREFERRED_PROVIDER. However, the PREFERRED_PROVIDERS variable lets you define preferences for multiple situations using the following form: @@ -1954,15 +1954,15 @@ - PREFERRED_VERSION + PREFERRED_VERSION If there are multiple versions of recipes available, this variable determines which recipe should be given preference. You must always suffix the variable with the - PN + PN you want to select, and you should set - PV + PV accordingly for precedence. @@ -1989,7 +1989,7 @@ - PREMIRRORS + PREMIRRORS Specifies additional paths from which BitBake gets source code. @@ -1998,7 +1998,7 @@ If that location fails, the build system tries locations defined by PREMIRRORS, the upstream source, and then locations specified by - MIRRORS + MIRRORS in that order. @@ -2022,20 +2022,20 @@ - PROVIDES + PROVIDES A list of aliases by which a particular recipe can be known. By default, a recipe's own - PN + PN is implicitly already in its PROVIDES list. If a recipe uses PROVIDES, the additional aliases are synonyms for the recipe and can be useful satisfying dependencies of other recipes during the build as specified by - DEPENDS. + DEPENDS. @@ -2059,7 +2059,7 @@ virtual target in PROVIDES. Recipes that depend on the functionality in question can include the virtual target in - DEPENDS + DEPENDS to leave the choice of provider open. @@ -2072,11 +2072,11 @@ - PRSERV_HOST + PRSERV_HOST The network based - PR + PR service host and port. @@ -2094,7 +2094,7 @@ - PV + PV The version of the recipe. @@ -2108,9 +2108,9 @@ --> - R + R - RDEPENDS + RDEPENDS Lists a package's runtime dependencies (i.e. other packages) @@ -2165,13 +2165,13 @@ For information on build-time dependencies, see the - DEPENDS + DEPENDS variable. - REPODIR + REPODIR The directory in which a local copy of a @@ -2181,14 +2181,14 @@ - RPROVIDES + RPROVIDES A list of package name aliases that a package also provides. These aliases are useful for satisfying runtime dependencies of other packages both during the build and on the target (as specified by - RDEPENDS). + RDEPENDS). As with all package-controlling variables, you must always @@ -2201,7 +2201,7 @@ - RRECOMMENDS + RRECOMMENDS A list of packages that extends the usability of a package @@ -2210,7 +2210,7 @@ packages in order to successfully build, but needs them for the extended usability. To specify runtime dependencies for packages, see the - RDEPENDS + RDEPENDS variable. @@ -2243,15 +2243,15 @@ - S + S - SECTION + SECTION The section in which packages should be categorized. - SRC_URI + SRC_URI The list of source files - local or remote. @@ -2272,7 +2272,7 @@ the metadata, from the local machine. The path is relative to the - FILESPATH + FILESPATH variable. bzr:// - Fetches files from a Bazaar revision control repository. @@ -2322,7 +2322,7 @@ - SRCDATE + SRCDATE The date of the source code used to build the package. @@ -2331,7 +2331,7 @@ - SRCREV + SRCREV The revision of the source code used to build the package. @@ -2344,13 +2344,13 @@ - SRCREV_FORMAT + SRCREV_FORMAT Helps construct valid - SRCREV + SRCREV values when multiple source controlled URLs are used in - SRC_URI. + SRC_URI. @@ -2371,7 +2371,7 @@ - STAMP + STAMP Specifies the base path used to create recipe stamp files. @@ -2381,12 +2381,12 @@ - STAMPCLEAN + STAMPCLEAN Specifies the base path used to create recipe stamp files. Unlike the - STAMP + STAMP variable, STAMPCLEAN can contain wildcards to match the range of files a clean operation should remove. @@ -2396,7 +2396,7 @@ - SUMMARY + SUMMARY A short summary for the recipe, which is 72 characters or less. @@ -2404,7 +2404,7 @@ - SVNDIR + SVNDIR The directory in which files checked out of a Subversion @@ -2415,9 +2415,9 @@ - T + T - T + T Points to a directory were BitBake places temporary files, which consist mostly of task logs and @@ -2426,7 +2426,7 @@ - TOPDIR + TOPDIR Points to the build directory. diff --git a/bitbake/doc/poky.ent b/bitbake/doc/poky.ent index c032e14..85d9c83 100644 --- a/bitbake/doc/poky.ent +++ b/bitbake/doc/poky.ent @@ -17,13 +17,6 @@ - - - - - - - @@ -31,7 +24,6 @@ - diff --git a/bitbake/lib/bb/COW.py b/bitbake/lib/bb/COW.py index 7817473..d26e981 100644 --- a/bitbake/lib/bb/COW.py +++ b/bitbake/lib/bb/COW.py @@ -1,23 +1,8 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- # # This is a copy on write dictionary and set which abuses classes to try and be nice and fast. # # Copyright (C) 2006 Tim Ansell # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. -# #Please Note: # Be careful when using mutable types (ie Dict and Lists) - operations involving these are SLOW. # Assign a file to __warn__ to get warnings about slow operations. diff --git a/bitbake/lib/bb/__init__.py b/bitbake/lib/bb/__init__.py index 4bc47c8..c144311 100644 --- a/bitbake/lib/bb/__init__.py +++ b/bitbake/lib/bb/__init__.py @@ -1,5 +1,3 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- # # BitBake Build System Python Library # @@ -8,20 +6,10 @@ # # Based on Gentoo's portage.py. # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. +# SPDX-License-Identifier: GPL-2.0-only # -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. -__version__ = "1.40.0" +__version__ = "1.44.0" import sys if sys.version_info < (3, 4, 0): diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py index 3e2a94e..30a2ba2 100644 --- a/bitbake/lib/bb/build.py +++ b/bitbake/lib/bb/build.py @@ -1,5 +1,3 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- # # BitBake 'Build' implementation # @@ -10,18 +8,7 @@ # # Based on Gentoo's portage.py. # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only # # Based on functions from the base bb module, Copyright 2003 Holger Schurig @@ -67,23 +54,6 @@ else: builtins['bb'] = bb builtins['os'] = os -class FuncFailed(Exception): - def __init__(self, name = None, logfile = None): - self.logfile = logfile - self.name = name - if name: - self.msg = 'Function failed: %s' % name - else: - self.msg = "Function failed" - - def __str__(self): - if self.logfile and os.path.exists(self.logfile): - msg = ("%s (log file is located at %s)" % - (self.msg, self.logfile)) - else: - msg = self.msg - return msg - class TaskBase(event.Event): """Base class for task events""" @@ -176,15 +146,33 @@ class LogTee(object): def __repr__(self): return ''.format(self.name) + def flush(self): self.outfile.flush() -# -# pythonexception allows the python exceptions generated to be raised -# as the real exceptions (not FuncFailed) and without a backtrace at the -# origin of the failure. -# -def exec_func(func, d, dirs = None, pythonexception=False): + +class StdoutNoopContextManager: + """ + This class acts like sys.stdout, but adds noop __enter__ and __exit__ methods. + """ + def __enter__(self): + return sys.stdout + + def __exit__(self, *exc_info): + pass + + def write(self, string): + return sys.stdout.write(string) + + def flush(self): + sys.stdout.flush() + + @property + def name(self): + return sys.stdout.name + + +def exec_func(func, d, dirs = None): """Execute a BB 'function'""" try: @@ -256,7 +244,7 @@ def exec_func(func, d, dirs = None, pythonexception=False): with bb.utils.fileslocked(lockfiles): if ispython: - exec_func_python(func, d, runfile, cwd=adir, pythonexception=pythonexception) + exec_func_python(func, d, runfile, cwd=adir) else: exec_func_shell(func, d, runfile, cwd=adir) @@ -276,7 +264,7 @@ _functionfmt = """ {function}(d) """ logformatter = bb.msg.BBLogFormatter("%(levelname)s: %(message)s") -def exec_func_python(func, d, runfile, cwd=None, pythonexception=False): +def exec_func_python(func, d, runfile, cwd=None): """Execute a python BB 'function'""" code = _functionfmt.format(function=func) @@ -301,13 +289,7 @@ def exec_func_python(func, d, runfile, cwd=None, pythonexception=False): bb.methodpool.insert_method(func, text, fn, lineno - 1) comp = utils.better_compile(code, func, "exec_python_func() autogenerated") - utils.better_exec(comp, {"d": d}, code, "exec_python_func() autogenerated", pythonexception=pythonexception) - except (bb.parse.SkipRecipe, bb.build.FuncFailed): - raise - except: - if pythonexception: - raise - raise FuncFailed(func, None) + utils.better_exec(comp, {"d": d}, code, "exec_python_func() autogenerated") finally: bb.debug(2, "Python function %s finished" % func) @@ -335,6 +317,42 @@ trap 'bb_exit_handler' 0 set -e ''' +def create_progress_handler(func, progress, logfile, d): + if progress == 'percent': + # Use default regex + return bb.progress.BasicProgressHandler(d, outfile=logfile) + elif progress.startswith('percent:'): + # Use specified regex + return bb.progress.BasicProgressHandler(d, regex=progress.split(':', 1)[1], outfile=logfile) + elif progress.startswith('outof:'): + # Use specified regex + return bb.progress.OutOfProgressHandler(d, regex=progress.split(':', 1)[1], outfile=logfile) + elif progress.startswith("custom:"): + # Use a custom progress handler that was injected via OE_EXTRA_IMPORTS or __builtins__ + import functools + from types import ModuleType + + parts = progress.split(":", 2) + _, cls, otherargs = parts[0], parts[1], (parts[2] or None) if parts[2:] else None + if cls: + def resolve(x, y): + if not x: + return None + if isinstance(x, ModuleType): + return getattr(x, y, None) + return x.get(y) + cls_obj = functools.reduce(resolve, cls.split("."), bb.utils._context) + if not cls_obj: + # Fall-back on __builtins__ + cls_obj = functools.reduce(lambda x, y: x.get(y), cls.split("."), __builtins__) + if cls_obj: + return cls_obj(d, outfile=logfile, otherargs=otherargs) + bb.warn('%s: unknown custom progress handler in task progress varflag value "%s", ignoring' % (func, cls)) + else: + bb.warn('%s: invalid task progress varflag value "%s", ignoring' % (func, progress)) + + return logfile + def exec_func_shell(func, d, runfile, cwd=None): """Execute a shell function from the metadata @@ -372,23 +390,13 @@ exit $ret cmd = [fakerootcmd, runfile] if bb.msg.loggerDefaultVerbose: - logfile = LogTee(logger, sys.stdout) + logfile = LogTee(logger, StdoutNoopContextManager()) else: - logfile = sys.stdout + logfile = StdoutNoopContextManager() progress = d.getVarFlag(func, 'progress') if progress: - if progress == 'percent': - # Use default regex - logfile = bb.progress.BasicProgressHandler(d, outfile=logfile) - elif progress.startswith('percent:'): - # Use specified regex - logfile = bb.progress.BasicProgressHandler(d, regex=progress.split(':', 1)[1], outfile=logfile) - elif progress.startswith('outof:'): - # Use specified regex - logfile = bb.progress.OutOfProgressHandler(d, regex=progress.split(':', 1)[1], outfile=logfile) - else: - bb.warn('%s: invalid task progress varflag value "%s", ignoring' % (func, progress)) + logfile = create_progress_handler(func, progress, logfile, d) fifobuffer = bytearray() def readfifo(data): @@ -407,6 +415,8 @@ exit $ret bb.plain(value) elif cmd == 'bbnote': bb.note(value) + elif cmd == 'bbverbnote': + bb.verbnote(value) elif cmd == 'bbwarn': bb.warn(value) elif cmd == 'bberror': @@ -436,13 +446,8 @@ exit $ret with open(fifopath, 'r+b', buffering=0) as fifo: try: bb.debug(2, "Executing shell function %s" % func) - - try: - with open(os.devnull, 'r+') as stdin: - bb.process.run(cmd, shell=False, stdin=stdin, log=logfile, extrafiles=[(fifo,readfifo)]) - except bb.process.CmdError: - logfn = d.getVar('BB_LOGFILE') - raise FuncFailed(func, logfn) + with open(os.devnull, 'r+') as stdin, logfile: + bb.process.run(cmd, shell=False, stdin=stdin, log=logfile, extrafiles=[(fifo,readfifo)]) finally: os.unlink(fifopath) @@ -570,9 +575,6 @@ def _exec_task(fn, task, d, quieterr): event.fire(TaskStarted(task, logfn, flags, localdata), localdata) except (bb.BBHandledException, SystemExit): return 1 - except FuncFailed as exc: - logger.error(str(exc)) - return 1 try: for func in (prefuncs or '').split(): @@ -580,7 +582,10 @@ def _exec_task(fn, task, d, quieterr): exec_func(task, localdata) for func in (postfuncs or '').split(): exec_func(func, localdata) - except FuncFailed as exc: + except bb.BBHandledException: + event.fire(TaskFailed(task, logfn, localdata, True), localdata) + return 1 + except Exception as exc: if quieterr: event.fire(TaskFailedSilent(task, logfn, localdata), localdata) else: @@ -588,9 +593,6 @@ def _exec_task(fn, task, d, quieterr): logger.error(str(exc)) event.fire(TaskFailed(task, logfn, localdata, errprinted), localdata) return 1 - except bb.BBHandledException: - event.fire(TaskFailed(task, logfn, localdata, True), localdata) - return 1 finally: sys.stdout.flush() sys.stderr.flush() @@ -814,6 +816,9 @@ def add_tasks(tasklist, d): task_deps['parents'][task] = [] if 'deps' in flags: for dep in flags['deps']: + # Check and warn for "addtask task after foo" while foo does not exist + #if not dep in tasklist: + # bb.warn('%s: dependent task %s for %s does not exist' % (d.getVar('PN'), dep, task)) dep = d.expand(dep) task_deps['parents'][task].append(dep) diff --git a/bitbake/lib/bb/cache.py b/bitbake/lib/bb/cache.py index 258d679..b6f7da5 100644 --- a/bitbake/lib/bb/cache.py +++ b/bitbake/lib/bb/cache.py @@ -1,5 +1,3 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- # # BitBake Cache implementation # @@ -15,18 +13,8 @@ # Copyright (C) 2005 Holger Hans Peter Freyther # Copyright (C) 2005 ROAD GmbH # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. +# SPDX-License-Identifier: GPL-2.0-only # -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. import os import sys @@ -95,21 +83,21 @@ class CoreRecipeInfo(RecipeInfoCommon): self.appends = self.listvar('__BBAPPEND', metadata) self.nocache = self.getvar('BB_DONT_CACHE', metadata) + self.provides = self.depvar('PROVIDES', metadata) + self.rprovides = self.depvar('RPROVIDES', metadata) + self.pn = self.getvar('PN', metadata) or bb.parse.vars_from_file(filename,metadata)[0] + self.packages = self.listvar('PACKAGES', metadata) + if not self.packages: + self.packages.append(self.pn) + self.packages_dynamic = self.listvar('PACKAGES_DYNAMIC', metadata) + self.skipreason = self.getvar('__SKIPPED', metadata) if self.skipreason: - self.pn = self.getvar('PN', metadata) or bb.parse.BBHandler.vars_from_file(filename,metadata)[0] self.skipped = True - self.provides = self.depvar('PROVIDES', metadata) - self.rprovides = self.depvar('RPROVIDES', metadata) return self.tasks = metadata.getVar('__BBTASKS', False) - self.pn = self.getvar('PN', metadata) - self.packages = self.listvar('PACKAGES', metadata) - if not self.packages: - self.packages.append(self.pn) - self.basetaskhashes = self.taskvar('BB_BASEHASH', self.tasks, metadata) self.hashfilename = self.getvar('BB_HASHFILENAME', metadata) @@ -125,11 +113,8 @@ class CoreRecipeInfo(RecipeInfoCommon): self.stampclean = self.getvar('STAMPCLEAN', metadata) self.stamp_extrainfo = self.flaglist('stamp-extra-info', self.tasks, metadata) self.file_checksums = self.flaglist('file-checksums', self.tasks, metadata, True) - self.packages_dynamic = self.listvar('PACKAGES_DYNAMIC', metadata) self.depends = self.depvar('DEPENDS', metadata) - self.provides = self.depvar('PROVIDES', metadata) self.rdepends = self.depvar('RDEPENDS', metadata) - self.rprovides = self.depvar('RPROVIDES', metadata) self.rrecommends = self.depvar('RRECOMMENDS', metadata) self.rprovides_pkg = self.pkgvar('RPROVIDES', self.packages, metadata) self.rdepends_pkg = self.pkgvar('RDEPENDS', self.packages, metadata) @@ -235,7 +220,7 @@ class CoreRecipeInfo(RecipeInfoCommon): cachedata.hashfn[fn] = self.hashfilename for task, taskhash in self.basetaskhashes.items(): - identifier = '%s.%s' % (fn, task) + identifier = '%s:%s' % (fn, task) cachedata.basetaskhash[identifier] = taskhash cachedata.inherits[fn] = self.inherits @@ -249,7 +234,7 @@ def virtualfn2realfn(virtualfn): Convert a virtual file name to a real one + the associated subclass keyword """ mc = "" - if virtualfn.startswith('multiconfig:'): + if virtualfn.startswith('mc:'): elems = virtualfn.split(':') mc = elems[1] virtualfn = ":".join(elems[2:]) @@ -270,7 +255,7 @@ def realfn2virtual(realfn, cls, mc): if cls: realfn = "virtual:" + cls + ":" + realfn if mc: - realfn = "multiconfig:" + mc + ":" + realfn + realfn = "mc:" + mc + ":" + realfn return realfn def variant2virtual(realfn, variant): @@ -279,11 +264,11 @@ def variant2virtual(realfn, variant): """ if variant == "": return realfn - if variant.startswith("multiconfig:"): + if variant.startswith("mc:"): elems = variant.split(":") if elems[2]: - return "multiconfig:" + elems[1] + ":virtual:" + ":".join(elems[2:]) + ":" + realfn - return "multiconfig:" + elems[1] + ":" + realfn + return "mc:" + elems[1] + ":virtual:" + ":".join(elems[2:]) + ":" + realfn + return "mc:" + elems[1] + ":" + realfn return "virtual:" + variant + ":" + realfn def parse_recipe(bb_data, bbfile, appends, mc=''): @@ -361,7 +346,7 @@ class NoCache(object): bb_data = self.databuilder.mcdata[mc].createCopy() newstores = parse_recipe(bb_data, bbfile, appends, mc) for ns in newstores: - datastores["multiconfig:%s:%s" % (mc, ns)] = newstores[ns] + datastores["mc:%s:%s" % (mc, ns)] = newstores[ns] return datastores @@ -411,6 +396,15 @@ class Cache(NoCache): else: logger.debug(1, "Cache file %s not found, building..." % self.cachefile) + # We don't use the symlink, its just for debugging convinience + symlink = os.path.join(self.cachedir, "bb_cache.dat") + if os.path.exists(symlink): + bb.utils.remove(symlink) + try: + os.symlink(os.path.basename(self.cachefile), symlink) + except OSError: + pass + def load_cachefile(self): cachesize = 0 previous_progress = 0 @@ -889,3 +883,56 @@ class MultiProcessCache(object): p.dump([data, self.__class__.CACHE_VERSION]) bb.utils.unlockfile(glf) + + +class SimpleCache(object): + """ + BitBake multi-process cache implementation + + Used by the codeparser & file checksum caches + """ + + def __init__(self, version): + self.cachefile = None + self.cachedata = None + self.cacheversion = version + + def init_cache(self, d, cache_file_name=None, defaultdata=None): + cachedir = (d.getVar("PERSISTENT_DIR") or + d.getVar("CACHE")) + if not cachedir: + return defaultdata + + bb.utils.mkdirhier(cachedir) + self.cachefile = os.path.join(cachedir, + cache_file_name or self.__class__.cache_file_name) + logger.debug(1, "Using cache in '%s'", self.cachefile) + + glf = bb.utils.lockfile(self.cachefile + ".lock") + + try: + with open(self.cachefile, "rb") as f: + p = pickle.Unpickler(f) + data, version = p.load() + except: + bb.utils.unlockfile(glf) + return defaultdata + + bb.utils.unlockfile(glf) + + if version != self.cacheversion: + return defaultdata + + return data + + def save(self, data): + if not self.cachefile: + return + + glf = bb.utils.lockfile(self.cachefile + ".lock") + + with open(self.cachefile, "wb") as f: + p = pickle.Pickler(f, -1) + p.dump([data, self.cacheversion]) + + bb.utils.unlockfile(glf) diff --git a/bitbake/lib/bb/cache_extra.py b/bitbake/lib/bb/cache_extra.py index 83f4959..bf4226d 100644 --- a/bitbake/lib/bb/cache_extra.py +++ b/bitbake/lib/bb/cache_extra.py @@ -1,5 +1,3 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- # # Extra RecipeInfo will be all defined in this file. Currently, # Only Hob (Image Creator) Requests some extra fields. So @@ -12,18 +10,8 @@ # Copyright (C) 2011, Intel Corporation. All rights reserved. -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. +# SPDX-License-Identifier: GPL-2.0-only # -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. from bb.cache import RecipeInfoCommon diff --git a/bitbake/lib/bb/checksum.py b/bitbake/lib/bb/checksum.py index 4e1598f..5bc8a8f 100644 --- a/bitbake/lib/bb/checksum.py +++ b/bitbake/lib/bb/checksum.py @@ -2,18 +2,8 @@ # # Copyright (C) 2012 Intel Corporation # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. +# SPDX-License-Identifier: GPL-2.0-only # -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. import glob import operator diff --git a/bitbake/lib/bb/codeparser.py b/bitbake/lib/bb/codeparser.py index ddd1b97..fd2c473 100644 --- a/bitbake/lib/bb/codeparser.py +++ b/bitbake/lib/bb/codeparser.py @@ -1,3 +1,7 @@ +# +# SPDX-License-Identifier: GPL-2.0-only +# + """ BitBake code parser @@ -33,7 +37,7 @@ from bb.cache import MultiProcessCache logger = logging.getLogger('BitBake.CodeParser') def bbhash(s): - return hashlib.md5(s.encode("utf-8")).hexdigest() + return hashlib.sha256(s.encode("utf-8")).hexdigest() def check_indent(codestr): """If the code is indented, add a top level piece of code to 'remove' the indentation""" @@ -140,7 +144,7 @@ class CodeParserCache(MultiProcessCache): # so that an existing cache gets invalidated. Additionally you'll need # to increment __cache_version__ in cache.py in order to ensure that old # recipe caches don't trigger "Taskhash mismatch" errors. - CACHE_VERSION = 10 + CACHE_VERSION = 11 def __init__(self): MultiProcessCache.__init__(self) @@ -368,8 +372,9 @@ class ShellParser(): def _parse_shell(self, value): try: tokens, _ = pyshyacc.parse(value, eof=True, debug=False) - except pyshlex.NeedMore: - raise sherrors.ShellSyntaxError("Unexpected EOF") + except Exception: + bb.error('Error during parse shell code, the last 5 lines are:\n%s' % '\n'.join(value.split('\n')[-5:])) + raise self.process_tokens(tokens) diff --git a/bitbake/lib/bb/command.py b/bitbake/lib/bb/command.py index 6c966e3..378f389 100644 --- a/bitbake/lib/bb/command.py +++ b/bitbake/lib/bb/command.py @@ -6,18 +6,8 @@ Provide an interface to interact with the bitbake server through 'commands' # Copyright (C) 2006-2007 Richard Purdie # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. +# SPDX-License-Identifier: GPL-2.0-only # -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. """ The bitbake server takes 'commands' from its UI/commandline. diff --git a/bitbake/lib/bb/compat.py b/bitbake/lib/bb/compat.py index de1923d..4935668 100644 --- a/bitbake/lib/bb/compat.py +++ b/bitbake/lib/bb/compat.py @@ -1,3 +1,7 @@ +# +# SPDX-License-Identifier: GPL-2.0-only +# + """Code pulled from future python versions, here for compatibility""" from collections import MutableMapping, KeysView, ValuesView, ItemsView, OrderedDict diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py index 16681ba..20ef04d 100644 --- a/bitbake/lib/bb/cooker.py +++ b/bitbake/lib/bb/cooker.py @@ -1,6 +1,3 @@ -#!/usr/bin/env python -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- # # Copyright (C) 2003, 2004 Chris Larson # Copyright (C) 2003, 2004 Phil Blundell @@ -9,19 +6,8 @@ # Copyright (C) 2005 ROAD GmbH # Copyright (C) 2006 - 2007 Richard Purdie # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. +# SPDX-License-Identifier: GPL-2.0-only # -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - import sys, os, glob, os.path, re, time import atexit @@ -45,6 +31,7 @@ import pyinotify import json import pickle import codecs +import hashserv logger = logging.getLogger("BitBake") collectlog = logging.getLogger("BitBake.Collection") @@ -175,27 +162,45 @@ class BBCooker: self.configuration = configuration + bb.debug(1, "BBCooker starting %s" % time.time()) + sys.stdout.flush() + self.configwatcher = pyinotify.WatchManager() + bb.debug(1, "BBCooker pyinotify1 %s" % time.time()) + sys.stdout.flush() + self.configwatcher.bbseen = [] self.configwatcher.bbwatchedfiles = [] self.confignotifier = pyinotify.Notifier(self.configwatcher, self.config_notifications) + bb.debug(1, "BBCooker pyinotify2 %s" % time.time()) + sys.stdout.flush() self.watchmask = pyinotify.IN_CLOSE_WRITE | pyinotify.IN_CREATE | pyinotify.IN_DELETE | \ pyinotify.IN_DELETE_SELF | pyinotify.IN_MODIFY | pyinotify.IN_MOVE_SELF | \ pyinotify.IN_MOVED_FROM | pyinotify.IN_MOVED_TO self.watcher = pyinotify.WatchManager() + bb.debug(1, "BBCooker pyinotify3 %s" % time.time()) + sys.stdout.flush() self.watcher.bbseen = [] self.watcher.bbwatchedfiles = [] self.notifier = pyinotify.Notifier(self.watcher, self.notifications) + bb.debug(1, "BBCooker pyinotify complete %s" % time.time()) + sys.stdout.flush() + # If being called by something like tinfoil, we need to clean cached data # which may now be invalid bb.parse.clear_cache() bb.parse.BBHandler.cached_statements = {} self.ui_cmdline = None + self.hashserv = None + self.hashservaddr = None self.initConfigurationData() + bb.debug(1, "BBCooker parsed base configuration %s" % time.time()) + sys.stdout.flush() + # we log all events to a file if so directed if self.configuration.writeeventlog: # register the log file writer as UI Handler @@ -233,6 +238,9 @@ class BBCooker: # Let SIGHUP exit as SIGTERM signal.signal(signal.SIGHUP, self.sigterm_exception) + bb.debug(1, "BBCooker startup complete %s" % time.time()) + sys.stdout.flush() + def process_inotify_updates(self): for n in [self.confignotifier, self.notifier]: if n.check_events(timeout=0): @@ -367,13 +375,12 @@ class BBCooker: # Copy of the data store which has been expanded. # Used for firing events and accessing variables where expansion needs to be accounted for # - bb.parse.init_parser(self.data) - if CookerFeatures.BASEDATASTORE_TRACKING in self.featureset: self.disableDataTracking() - self.data.renameVar("__depends", "__base_depends") - self.add_filewatch(self.data.getVar("__base_depends", False), self.configwatcher) + for mc in self.databuilder.mcdata.values(): + mc.renameVar("__depends", "__base_depends") + self.add_filewatch(mc.getVar("__base_depends", False), self.configwatcher) self.baseconfig_valid = True self.parsecache_valid = False @@ -385,6 +392,22 @@ class BBCooker: except prserv.serv.PRServiceConfigError as e: bb.fatal("Unable to start PR Server, exitting") + if self.data.getVar("BB_HASHSERVE") == "auto": + # Create a new hash server bound to a unix domain socket + if not self.hashserv: + dbfile = (self.data.getVar("PERSISTENT_DIR") or self.data.getVar("CACHE")) + "/hashserv.db" + self.hashservaddr = "unix://%s/hashserve.sock" % self.data.getVar("TOPDIR") + self.hashserv = hashserv.create_server(self.hashservaddr, dbfile, sync=False) + self.hashserv.process = multiprocessing.Process(target=self.hashserv.serve_forever) + self.hashserv.process.start() + self.data.setVar("BB_HASHSERVE", self.hashservaddr) + self.databuilder.origdata.setVar("BB_HASHSERVE", self.hashservaddr) + self.databuilder.data.setVar("BB_HASHSERVE", self.hashservaddr) + for mc in self.databuilder.mcdata: + self.databuilder.mcdata[mc].setVar("BB_HASHSERVE", self.hashservaddr) + + bb.parse.init_parser(self.data) + def enableDataTracking(self): self.configuration.tracking = True if hasattr(self, "data"): @@ -488,6 +511,7 @@ class BBCooker: """ fn = None envdata = None + mc = '' if not pkgs_to_build: pkgs_to_build = [] @@ -496,6 +520,12 @@ class BBCooker: self.enableDataTracking() self.reset() + def mc_base(p): + if p.startswith('mc:'): + s = p.split(':') + if len(s) == 2: + return s[1] + return None if buildfile: # Parse the configuration here. We need to do it explicitly here since @@ -506,18 +536,16 @@ class BBCooker: fn = self.matchFile(fn) fn = bb.cache.realfn2virtual(fn, cls, mc) elif len(pkgs_to_build) == 1: - ignore = self.data.getVar("ASSUME_PROVIDED") or "" - if pkgs_to_build[0] in set(ignore.split()): - bb.fatal("%s is in ASSUME_PROVIDED" % pkgs_to_build[0]) + mc = mc_base(pkgs_to_build[0]) + if not mc: + ignore = self.data.getVar("ASSUME_PROVIDED") or "" + if pkgs_to_build[0] in set(ignore.split()): + bb.fatal("%s is in ASSUME_PROVIDED" % pkgs_to_build[0]) - taskdata, runlist = self.buildTaskData(pkgs_to_build, None, self.configuration.abort, allowincomplete=True) + taskdata, runlist = self.buildTaskData(pkgs_to_build, None, self.configuration.abort, allowincomplete=True) - mc = runlist[0][0] - fn = runlist[0][3] - else: - envdata = self.data - data.expandKeys(envdata) - parse.ast.runAnonFuncs(envdata) + mc = runlist[0][0] + fn = runlist[0][3] if fn: try: @@ -526,6 +554,12 @@ class BBCooker: except Exception as e: parselog.exception("Unable to read %s", fn) raise + else: + if not mc in self.databuilder.mcdata: + bb.fatal('Not multiconfig named "%s" found' % mc) + envdata = self.databuilder.mcdata[mc] + data.expandKeys(envdata) + parse.ast.runAnonFuncs(envdata) # Display history with closing(StringIO()) as env: @@ -565,10 +599,10 @@ class BBCooker: wildcard = False # Wild card expansion: - # Replace string such as "multiconfig:*:bash" - # into "multiconfig:A:bash multiconfig:B:bash bash" + # Replace string such as "mc:*:bash" + # into "mc:A:bash mc:B:bash bash" for k in targetlist: - if k.startswith("multiconfig:"): + if k.startswith("mc:"): if wildcard: bb.fatal('multiconfig conflict') if k.split(":")[1] == "*": @@ -601,7 +635,7 @@ class BBCooker: runlist = [] for k in fulltargetlist: mc = "" - if k.startswith("multiconfig:"): + if k.startswith("mc:"): mc = k.split(":")[1] k = ":".join(k.split(":")[2:]) ktask = task @@ -620,13 +654,22 @@ class BBCooker: runlist.append([mc, k, ktask, fn]) bb.event.fire(bb.event.TreeDataPreparationProgress(current, len(fulltargetlist)), self.data) - mcdeps = taskdata[mc].get_mcdepends() + havemc = False + for mc in self.multiconfigs: + if taskdata[mc].get_mcdepends(): + havemc = True + # No need to do check providers if there are no mcdeps or not an mc build - if mcdeps and mc: - # Make sure we can provide the multiconfig dependency + if havemc or len(self.multiconfigs) > 1: seen = set() new = True + # Make sure we can provide the multiconfig dependency while new: + mcdeps = set() + # Add unresolved first, so we can get multiconfig indirect dependencies on time + for mc in self.multiconfigs: + taskdata[mc].add_unresolved(localdata[mc], self.recipecaches[mc]) + mcdeps |= set(taskdata[mc].get_mcdepends()) new = False for mc in self.multiconfigs: for k in mcdeps: @@ -641,6 +684,7 @@ class BBCooker: taskdata[depmc].add_provider(localdata[depmc], self.recipecaches[depmc], l[3]) seen.add(k) new = True + for mc in self.multiconfigs: taskdata[mc].add_unresolved(localdata[mc], self.recipecaches[mc]) @@ -676,7 +720,7 @@ class BBCooker: @staticmethod def add_mc_prefix(mc, pn): if mc: - return "multiconfig:%s:%s" % (mc, pn) + return "mc:%s:%s" % (mc, pn) return pn def buildDependTree(self, rq, taskdata): @@ -875,6 +919,10 @@ class BBCooker: os.unlink('package-depends.dot') except FileNotFoundError: pass + try: + os.unlink('recipe-depends.dot') + except FileNotFoundError: + pass with open('task-depends.dot', 'w') as f: f.write("digraph depends {\n") @@ -888,27 +936,6 @@ class BBCooker: f.write("}\n") logger.info("Task dependencies saved to 'task-depends.dot'") - with open('recipe-depends.dot', 'w') as f: - f.write("digraph depends {\n") - pndeps = {} - for task in sorted(depgraph["tdepends"]): - (pn, taskname) = task.rsplit(".", 1) - if pn not in pndeps: - pndeps[pn] = set() - for dep in sorted(depgraph["tdepends"][task]): - (deppn, deptaskname) = dep.rsplit(".", 1) - pndeps[pn].add(deppn) - for pn in sorted(pndeps): - fn = depgraph["pn"][pn]["filename"] - version = depgraph["pn"][pn]["version"] - f.write('"%s" [label="%s\\n%s\\n%s"]\n' % (pn, pn, version, fn)) - for dep in sorted(pndeps[pn]): - if dep == pn: - continue - f.write('"%s" -> "%s"\n' % (pn, dep)) - f.write("}\n") - logger.info("Flattened recipe dependencies saved to 'recipe-depends.dot'") - def show_appends_with_no_recipes(self): # Determine which bbappends haven't been applied @@ -1191,8 +1218,8 @@ class BBCooker: continue elif regex == "": parselog.debug(1, "BBFILE_PATTERN_%s is empty" % c) + cre = re.compile('^NULL$') errors = False - continue else: try: cre = re.compile(regex) @@ -1453,7 +1480,7 @@ class BBCooker: ntargets = [] for target in runlist: if target[0]: - ntargets.append("multiconfig:%s:%s:%s" % (target[0], target[1], target[2])) + ntargets.append("mc:%s:%s:%s" % (target[0], target[1], target[2])) ntargets.append("%s:%s" % (target[1], target[2])) for mc in self.multiconfigs: @@ -1576,6 +1603,9 @@ class BBCooker: for pkg in pkgs_to_build: if pkg in ignore: parselog.warning("Explicit target \"%s\" is in ASSUME_PROVIDED, ignoring" % pkg) + if pkg.startswith("multiconfig:"): + pkgs_to_build.remove(pkg) + pkgs_to_build.append(pkg.replace("multiconfig:", "mc:")) if 'world' in pkgs_to_build: pkgs_to_build.remove('world') @@ -1583,7 +1613,7 @@ class BBCooker: bb.providers.buildWorldTargetList(self.recipecaches[mc], task) for t in self.recipecaches[mc].world_target: if mc: - t = "multiconfig:" + mc + ":" + t + t = "mc:" + mc + ":" + t pkgs_to_build.append(t) if 'universe' in pkgs_to_build: @@ -1602,7 +1632,7 @@ class BBCooker: bb.debug(1, "Skipping %s for universe tasks as task %s doesn't exist" % (t, task)) continue if mc: - t = "multiconfig:" + mc + ":" + t + t = "mc:" + mc + ":" + t pkgs_to_build.append(t) return pkgs_to_build @@ -1615,9 +1645,11 @@ class BBCooker: def post_serve(self): prserv.serv.auto_shutdown() + if self.hashserv: + self.hashserv.process.terminate() + self.hashserv.process.join() bb.event.fire(CookerExit(), self.data) - def shutdown(self, force = False): if force: self.state = state.forceshutdown @@ -1632,6 +1664,7 @@ class BBCooker: def reset(self): self.initConfigurationData() + self.handlePRServ() def clientComplete(self): """Called when the client is done using the server""" @@ -1865,35 +1898,6 @@ class ParsingFailure(Exception): self.recipe = recipe Exception.__init__(self, realexception, recipe) -class Feeder(multiprocessing.Process): - def __init__(self, jobs, to_parsers, quit): - self.quit = quit - self.jobs = jobs - self.to_parsers = to_parsers - multiprocessing.Process.__init__(self) - - def run(self): - while True: - try: - quit = self.quit.get_nowait() - except queue.Empty: - pass - else: - if quit == 'cancel': - self.to_parsers.cancel_join_thread() - break - - try: - job = self.jobs.pop() - except IndexError: - break - - try: - self.to_parsers.put(job, timeout=0.5) - except queue.Full: - self.jobs.insert(0, job) - continue - class Parser(multiprocessing.Process): def __init__(self, jobs, results, quit, init, profile): self.jobs = jobs @@ -1940,11 +1944,8 @@ class Parser(multiprocessing.Process): result = pending.pop() else: try: - job = self.jobs.get(timeout=0.25) - except queue.Empty: - continue - - if job is None: + job = self.jobs.pop() + except IndexError: break result = self.parse(*job) @@ -2028,14 +2029,15 @@ class CookerParser(object): multiprocessing.util.Finalize(None, bb.codeparser.parser_cache_save, exitpriority=1) multiprocessing.util.Finalize(None, bb.fetch.fetcher_parse_save, exitpriority=1) - self.feeder_quit = multiprocessing.Queue(maxsize=1) self.parser_quit = multiprocessing.Queue(maxsize=self.num_processes) - self.jobs = multiprocessing.Queue(maxsize=self.num_processes) self.result_queue = multiprocessing.Queue() - self.feeder = Feeder(self.willparse, self.jobs, self.feeder_quit) - self.feeder.start() + + def chunkify(lst,n): + return [lst[i::n] for i in range(n)] + self.jobs = chunkify(self.willparse, self.num_processes) + for i in range(0, self.num_processes): - parser = Parser(self.jobs, self.result_queue, self.parser_quit, init, self.cooker.configuration.profile) + parser = Parser(self.jobs[i], self.result_queue, self.parser_quit, init, self.cooker.configuration.profile) parser.start() self.process_names.append(parser.name) self.processes.append(parser) @@ -2056,17 +2058,20 @@ class CookerParser(object): self.total) bb.event.fire(event, self.cfgdata) - self.feeder_quit.put(None) for process in self.processes: self.parser_quit.put(None) else: - self.feeder_quit.put('cancel') - self.parser_quit.cancel_join_thread() for process in self.processes: self.parser_quit.put(None) - self.jobs.cancel_join_thread() + # Cleanup the queue before call process.join(), otherwise there might be + # deadlocks. + while True: + try: + self.result_queue.get(timeout=0.25) + except queue.Empty: + break for process in self.processes: if force: @@ -2074,7 +2079,6 @@ class CookerParser(object): process.terminate() else: process.join() - self.feeder.join() sync = threading.Thread(target=self.bb_cache.sync) sync.start() diff --git a/bitbake/lib/bb/cookerdata.py b/bitbake/lib/bb/cookerdata.py index 5df66e6..472423f 100644 --- a/bitbake/lib/bb/cookerdata.py +++ b/bitbake/lib/bb/cookerdata.py @@ -1,6 +1,3 @@ -#!/usr/bin/env python -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- # # Copyright (C) 2003, 2004 Chris Larson # Copyright (C) 2003, 2004 Phil Blundell @@ -9,23 +6,14 @@ # Copyright (C) 2005 ROAD GmbH # Copyright (C) 2006 Richard Purdie # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. +# SPDX-License-Identifier: GPL-2.0-only # -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. import logging import os import re import sys +import hashlib from functools import wraps import bb from bb import data @@ -134,6 +122,7 @@ class CookerConfiguration(object): self.profile = False self.nosetscene = False self.setsceneonly = False + self.skipsetscene = False self.invalidate_stamp = False self.dump_signatures = [] self.dry_run = False @@ -279,12 +268,13 @@ class CookerDataBuilder(object): self.mcdata = {} def parseBaseConfiguration(self): + data_hash = hashlib.sha256() try: - bb.parse.init_parser(self.basedata) self.data = self.parseConfigurationFiles(self.prefiles, self.postfiles) if self.data.getVar("BB_WORKERCONTEXT", False) is None: bb.fetch.fetcher_init(self.data) + bb.parse.init_parser(self.data) bb.codeparser.parser_cache_init(self.data) bb.event.fire(bb.event.ConfigParsed(), self.data) @@ -302,7 +292,7 @@ class CookerDataBuilder(object): bb.event.fire(bb.event.ConfigParsed(), self.data) bb.parse.init_parser(self.data) - self.data_hash = self.data.get_hash() + data_hash.update(self.data.get_hash().encode('utf-8')) self.mcdata[''] = self.data multiconfig = (self.data.getVar("BBMULTICONFIG") or "").split() @@ -310,9 +300,11 @@ class CookerDataBuilder(object): mcdata = self.parseConfigurationFiles(self.prefiles, self.postfiles, config) bb.event.fire(bb.event.ConfigParsed(), mcdata) self.mcdata[config] = mcdata + data_hash.update(mcdata.get_hash().encode('utf-8')) if multiconfig: bb.event.fire(bb.event.MultiConfigParsed(self.mcdata), self.data) + self.data_hash = data_hash.hexdigest() except (SyntaxError, bb.BBHandledException): raise bb.BBHandledException except bb.data_smart.ExpansionError as e: @@ -354,14 +346,24 @@ class CookerDataBuilder(object): data = parse_config_file(layerconf, data) layers = (data.getVar('BBLAYERS') or "").split() + broken_layers = [] data = bb.data.createCopy(data) approved = bb.utils.approved_variables() + + # Check whether present layer directories exist for layer in layers: if not os.path.isdir(layer): - parselog.critical("Layer directory '%s' does not exist! " - "Please check BBLAYERS in %s" % (layer, layerconf)) - sys.exit(1) + broken_layers.append(layer) + + if broken_layers: + parselog.critical("The following layer directories do not exist:") + for layer in broken_layers: + parselog.critical(" %s", layer) + parselog.critical("Please check BBLAYERS in %s" % (layerconf)) + sys.exit(1) + + for layer in layers: parselog.debug(2, "Adding layer %s", layer) if 'HOME' in approved and '~' in layer: layer = os.path.expanduser(layer) @@ -391,7 +393,11 @@ class CookerDataBuilder(object): bb.fatal("BBFILES_DYNAMIC entries must be of the form :, not:\n %s" % "\n ".join(invalid)) layerseries = set((data.getVar("LAYERSERIES_CORENAMES") or "").split()) + collections_tmp = collections[:] for c in collections: + collections_tmp.remove(c) + if c in collections_tmp: + bb.fatal("Found duplicated BBFILE_COLLECTIONS '%s', check bblayers.conf or layer.conf to fix it." % c) compat = set((data.getVar("LAYERSERIES_COMPAT_%s" % c) or "").split()) if compat and not (compat & layerseries): bb.fatal("Layer %s is not compatible with the core layer which only supports these series: %s (layer is compatible with %s)" diff --git a/bitbake/lib/bb/daemonize.py b/bitbake/lib/bb/daemonize.py index c937675..f01e6ec 100644 --- a/bitbake/lib/bb/daemonize.py +++ b/bitbake/lib/bb/daemonize.py @@ -1,3 +1,7 @@ +# +# SPDX-License-Identifier: GPL-2.0-only +# + """ Python Daemonizing helper diff --git a/bitbake/lib/bb/data.py b/bitbake/lib/bb/data.py index d66d98c..0d75d0c 100644 --- a/bitbake/lib/bb/data.py +++ b/bitbake/lib/bb/data.py @@ -1,5 +1,3 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- """ BitBake 'Data' implementations @@ -22,18 +20,7 @@ the speed is more critical here. # Copyright (C) 2003, 2004 Chris Larson # Copyright (C) 2005 Holger Hans Peter Freyther # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only # # Based on functions from the base bb module, Copyright 2003 Holger Schurig @@ -143,7 +130,7 @@ def emit_var(var, o=sys.__stdout__, d = init(), all=False): if all: oval = d.getVar(var, False) val = d.getVar(var) - except (KeyboardInterrupt, bb.build.FuncFailed): + except (KeyboardInterrupt): raise except Exception as exc: o.write('# expansion of %s threw %s: %s\n' % (var, exc.__class__.__name__, str(exc))) @@ -322,8 +309,6 @@ def build_dependencies(key, keys, shelldeps, varflagsexcl, d): if varflags.get("python"): value = d.getVarFlag(key, "_content", False) parser = bb.codeparser.PythonParser(key, logger) - if value and "\t" in value: - logger.warning("Variable %s contains tabs, please remove these (%s)" % (key, d.getVar("FILE"))) parser.parse_python(value, filename=varflags.get("filename"), lineno=varflags.get("lineno")) deps = deps | parser.references deps = deps | (keys & parser.execs) @@ -437,8 +422,8 @@ def generate_dependency_hash(tasklist, gendeps, lookupcache, whitelist, fn): var = lookupcache[dep] if var is not None: data = data + str(var) - k = fn + "." + task - basehash[k] = hashlib.md5(data.encode("utf-8")).hexdigest() + k = fn + ":" + task + basehash[k] = hashlib.sha256(data.encode("utf-8")).hexdigest() taskdeps[task] = alldeps return taskdeps, basehash diff --git a/bitbake/lib/bb/data_smart.py b/bitbake/lib/bb/data_smart.py index 67af380..dd5c618 100644 --- a/bitbake/lib/bb/data_smart.py +++ b/bitbake/lib/bb/data_smart.py @@ -1,5 +1,3 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- """ BitBake Smart Dictionary Implementation @@ -14,18 +12,8 @@ BitBake build tools. # Copyright (C) 2005 Uli Luckas # Copyright (C) 2005 ROAD GmbH # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. +# SPDX-License-Identifier: GPL-2.0-only # -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. # Based on functions from the base bb module, Copyright 2003 Holger Schurig import copy, re, sys, traceback @@ -39,10 +27,11 @@ from bb.COW import COWDictBase logger = logging.getLogger("BitBake.Data") __setvar_keyword__ = ["_append", "_prepend", "_remove"] -__setvar_regexp__ = re.compile('(?P.*?)(?P_append|_prepend|_remove)(_(?P[^A-Z]*))?$') -__expand_var_regexp__ = re.compile(r"\${[^{}@\n\t :]+}") +__setvar_regexp__ = re.compile(r'(?P.*?)(?P_append|_prepend|_remove)(_(?P[^A-Z]*))?$') +__expand_var_regexp__ = re.compile(r"\${[a-zA-Z0-9\-_+./~]+?}") __expand_python_regexp__ = re.compile(r"\${@.+?}") -__whitespace_split__ = re.compile('(\s)') +__whitespace_split__ = re.compile(r'(\s)') +__override_regexp__ = re.compile(r'[a-z0-9]+') def infer_caller_details(loginfo, parent = False, varval = True): """Save the caller the trouble of specifying everything.""" @@ -597,7 +586,7 @@ class DataSmart(MutableMapping): # aka pay the cookie monster override = var[var.rfind('_')+1:] shortvar = var[:var.rfind('_')] - while override and override.islower(): + while override and __override_regexp__.match(override): if shortvar not in self.overridedata: self.overridedata[shortvar] = [] if [var, override] not in self.overridedata[shortvar]: @@ -1073,4 +1062,4 @@ class DataSmart(MutableMapping): data.update({i:value}) data_str = str([(k, data[k]) for k in sorted(data.keys())]) - return hashlib.md5(data_str.encode("utf-8")).hexdigest() + return hashlib.sha256(data_str.encode("utf-8")).hexdigest() diff --git a/bitbake/lib/bb/event.py b/bitbake/lib/bb/event.py index 5b1b094..d44621e 100644 --- a/bitbake/lib/bb/event.py +++ b/bitbake/lib/bb/event.py @@ -1,5 +1,3 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- """ BitBake 'Event' implementation @@ -9,18 +7,8 @@ BitBake build tools. # Copyright (C) 2003, 2004 Chris Larson # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. +# SPDX-License-Identifier: GPL-2.0-only # -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. import os, sys import warnings @@ -136,6 +124,7 @@ def fire_class_handlers(event, d): ui_queue = [] @atexit.register def print_ui_queue(): + global ui_queue """If we're exiting before a UI has been spawned, display any queued LogRecords to the console.""" logger = logging.getLogger("BitBake") @@ -180,6 +169,7 @@ def print_ui_queue(): logger.removeHandler(stderr) else: logger.removeHandler(stdout) + ui_queue = [] def fire_ui_handlers(event, d): global _thread_lock @@ -414,23 +404,6 @@ class RecipeTaskPreProcess(RecipeEvent): class RecipeParsed(RecipeEvent): """ Recipe Parsing Complete """ -class StampUpdate(Event): - """Trigger for any adjustment of the stamp files to happen""" - - def __init__(self, targets, stampfns): - self._targets = targets - self._stampfns = stampfns - Event.__init__(self) - - def getStampPrefix(self): - return self._stampfns - - def getTargets(self): - return self._targets - - stampPrefix = property(getStampPrefix) - targets = property(getTargets) - class BuildBase(Event): """Base class for bitbake build events""" diff --git a/bitbake/lib/bb/exceptions.py b/bitbake/lib/bb/exceptions.py index cd71343..ecbad59 100644 --- a/bitbake/lib/bb/exceptions.py +++ b/bitbake/lib/bb/exceptions.py @@ -1,3 +1,6 @@ +# +# SPDX-License-Identifier: GPL-2.0-only +# import inspect import traceback diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py index 572b71a..1f5f8f1 100644 --- a/bitbake/lib/bb/fetch2/__init__.py +++ b/bitbake/lib/bb/fetch2/__init__.py @@ -1,5 +1,3 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- """ BitBake 'Fetch' implementations @@ -10,18 +8,7 @@ BitBake build tools. # Copyright (C) 2003, 2004 Chris Larson # Copyright (C) 2012 Intel Corporation # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only # # Based on functions from the base bb module, Copyright 2003 Holger Schurig @@ -256,7 +243,7 @@ class URI(object): # Identify if the URI is relative or not if urlp.scheme in self._relative_schemes and \ - re.compile("^\w+:(?!//)").match(uri): + re.compile(r"^\w+:(?!//)").match(uri): self.relative = True if not self.relative: @@ -524,7 +511,7 @@ def fetcher_parse_save(): def fetcher_parse_done(): _checksum_cache.save_merge() -def fetcher_compare_revisions(): +def fetcher_compare_revisions(d): """ Compare the revisions in the persistant cache with current values and return true/false on whether they've changed. @@ -777,7 +764,8 @@ def get_srcrev(d, method_name='sortable_revision'): # format = d.getVar('SRCREV_FORMAT') if not format: - raise FetchError("The SRCREV_FORMAT variable must be set when multiple SCMs are used.") + raise FetchError("The SRCREV_FORMAT variable must be set when multiple SCMs are used.\n"\ + "The SCMs are:\n%s" % '\n'.join(scms)) name_to_rev = {} seenautoinc = False @@ -855,10 +843,18 @@ def runfetchcmd(cmd, d, quiet=False, cleanup=None, log=None, workdir=None): if val: cmd = 'export ' + var + '=\"%s\"; %s' % (val, cmd) + # Ensure that a _PYTHON_SYSCONFIGDATA_NAME value set by a recipe + # (for example via python3native.bbclass since warrior) is not set for + # host Python (otherwise tools like git-make-shallow will fail) + cmd = 'unset _PYTHON_SYSCONFIGDATA_NAME; ' + cmd + # Disable pseudo as it may affect ssh, potentially causing it to hang. cmd = 'export PSEUDO_DISABLED=1; ' + cmd - logger.debug(1, "Running %s", cmd) + if workdir: + logger.debug(1, "Running '%s' in %s" % (cmd, workdir)) + else: + logger.debug(1, "Running %s", cmd) success = False error_message = "" @@ -894,7 +890,7 @@ def check_network_access(d, info, url): log remote network access, and error if BB_NO_NETWORK is set or the given URI is untrusted """ - if d.getVar("BB_NO_NETWORK") == "1": + if bb.utils.to_boolean(d.getVar("BB_NO_NETWORK")): raise NetworkAccess(url, info) elif not trusted_network(d, url): raise UntrustedUrl(url, info) @@ -966,7 +962,8 @@ def rename_bad_checksum(ud, suffix): new_localpath = "%s_bad-checksum_%s" % (ud.localpath, suffix) bb.warn("Renaming %s to %s" % (ud.localpath, new_localpath)) - bb.utils.movefile(ud.localpath, new_localpath) + if not bb.utils.movefile(ud.localpath, new_localpath): + bb.warn("Renaming %s to %s failed, grep movefile in log.do_fetch to see why" % (ud.localpath, new_localpath)) def try_mirror_url(fetch, origud, ud, ld, check = False): @@ -1027,7 +1024,7 @@ def try_mirror_url(fetch, origud, ud, ld, check = False): raise except IOError as e: - if e.errno in [os.errno.ESTALE]: + if e.errno in [errno.ESTALE]: logger.warning("Stale Error Observed %s." % ud.url) return False raise @@ -1094,7 +1091,7 @@ def trusted_network(d, url): BB_ALLOWED_NETWORKS is set globally or for a specific recipe. Note: modifies SRC_URI & mirrors. """ - if d.getVar('BB_NO_NETWORK') == "1": + if bb.utils.to_boolean(d.getVar("BB_NO_NETWORK")): return True pkgname = d.expand(d.getVar('PN', False)) @@ -1403,7 +1400,7 @@ class FetchMethod(object): Fetch urls Assumes localpath was called first """ - raise NoMethodError(url) + raise NoMethodError(urldata.url) def unpack(self, urldata, rootdir, data): iterate = False @@ -1469,7 +1466,7 @@ class FetchMethod(object): else: cmd = 'rpm2cpio.sh %s | cpio -id' % (file) elif file.endswith('.deb') or file.endswith('.ipk'): - output = subprocess.check_output('ar -t %s' % file, preexec_fn=subprocess_setup, shell=True) + output = subprocess.check_output(['ar', '-t', file], preexec_fn=subprocess_setup) datafile = None if output: for line in output.decode().splitlines(): @@ -1547,7 +1544,7 @@ class FetchMethod(object): Check the status of a URL Assumes localpath was called first """ - logger.info("URL %s could not be checked for status since no method exists.", url) + logger.info("URL %s could not be checked for status since no method exists.", urldata.url) return True def latest_revision(self, ud, d, name): @@ -1555,7 +1552,7 @@ class FetchMethod(object): Look in the cache for the latest revision, if not present ask the SCM. """ if not hasattr(self, "_latest_revision"): - raise ParameterError("The fetcher for this URL does not support _latest_revision", url) + raise ParameterError("The fetcher for this URL does not support _latest_revision", ud.url) revs = bb.persist_data.persist('BB_URI_HEADREVS', d) key = self.generate_revision_key(ud, d, name) @@ -1638,7 +1635,7 @@ class Fetch(object): urls = self.urls network = self.d.getVar("BB_NO_NETWORK") - premirroronly = (self.d.getVar("BB_FETCH_PREMIRRORONLY") == "1") + premirroronly = bb.utils.to_boolean(self.d.getVar("BB_FETCH_PREMIRRORONLY")) for u in urls: ud = self.ud[u] @@ -1716,7 +1713,7 @@ class Fetch(object): update_stamp(ud, self.d) except IOError as e: - if e.errno in [os.errno.ESTALE]: + if e.errno in [errno.ESTALE]: logger.error("Stale Error Observed %s." % u) raise ChecksumError("Stale Error Detected") @@ -1786,7 +1783,7 @@ class Fetch(object): for url in urls: if url not in self.ud: - self.ud[url] = FetchData(url, d) + self.ud[url] = FetchData(url, self.d) ud = self.ud[url] ud.setup_localpath(self.d) diff --git a/bitbake/lib/bb/fetch2/bzr.py b/bitbake/lib/bb/fetch2/bzr.py index 658502f..c56d875 100644 --- a/bitbake/lib/bb/fetch2/bzr.py +++ b/bitbake/lib/bb/fetch2/bzr.py @@ -10,18 +10,8 @@ BitBake 'Fetch' implementation for bzr. # BitBake build tools. # Copyright (C) 2003, 2004 Chris Larson # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. +# SPDX-License-Identifier: GPL-2.0-only # -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. import os import sys diff --git a/bitbake/lib/bb/fetch2/clearcase.py b/bitbake/lib/bb/fetch2/clearcase.py index 3a6573d..3dd93ad 100644 --- a/bitbake/lib/bb/fetch2/clearcase.py +++ b/bitbake/lib/bb/fetch2/clearcase.py @@ -1,5 +1,3 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- """ BitBake 'Fetch' clearcase implementation @@ -47,18 +45,7 @@ User credentials: """ # Copyright (C) 2014 Siemens AG # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only # import os @@ -67,6 +54,8 @@ import shutil import bb from bb.fetch2 import FetchMethod from bb.fetch2 import FetchError +from bb.fetch2 import MissingParameterError +from bb.fetch2 import ParameterError from bb.fetch2 import runfetchcmd from bb.fetch2 import logger @@ -92,7 +81,7 @@ class ClearCase(FetchMethod): if 'protocol' in ud.parm: ud.proto = ud.parm['protocol'] if not ud.proto in ('http', 'https'): - raise fetch2.ParameterError("Invalid protocol type", ud.url) + raise ParameterError("Invalid protocol type", ud.url) ud.vob = '' if 'vob' in ud.parm: diff --git a/bitbake/lib/bb/fetch2/cvs.py b/bitbake/lib/bb/fetch2/cvs.py index 0e0a319..1b35ba4 100644 --- a/bitbake/lib/bb/fetch2/cvs.py +++ b/bitbake/lib/bb/fetch2/cvs.py @@ -1,5 +1,3 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- """ BitBake 'Fetch' implementations @@ -10,20 +8,9 @@ BitBake build tools. # Copyright (C) 2003, 2004 Chris Larson # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. +# SPDX-License-Identifier: GPL-2.0-only # -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. -# -#Based on functions from the base bb module, Copyright 2003 Holger Schurig +# Based on functions from the base bb module, Copyright 2003 Holger Schurig # import os diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py index 59a2ee8..2d1d2ca 100644 --- a/bitbake/lib/bb/fetch2/git.py +++ b/bitbake/lib/bb/fetch2/git.py @@ -1,5 +1,3 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- """ BitBake 'Fetch' git implementation @@ -55,20 +53,10 @@ Supported SRC_URI options are: """ -#Copyright (C) 2005 Richard Purdie +# Copyright (C) 2005 Richard Purdie # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. +# SPDX-License-Identifier: GPL-2.0-only # -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. import collections import errno @@ -199,7 +187,7 @@ class Git(FetchMethod): depth_default = 1 ud.shallow_depths = collections.defaultdict(lambda: depth_default) - revs_default = d.getVar("BB_GIT_SHALLOW_REVS", True) + revs_default = d.getVar("BB_GIT_SHALLOW_REVS") ud.shallow_revs = [] ud.branches = {} for pos, name in enumerate(ud.names): @@ -318,7 +306,7 @@ class Git(FetchMethod): def try_premirror(self, ud, d): # If we don't do this, updating an existing checkout with only premirrors # is not possible - if d.getVar("BB_FETCH_PREMIRRORONLY") is not None: + if bb.utils.to_boolean(d.getVar("BB_FETCH_PREMIRRORONLY")): return True if os.path.exists(ud.clonedir): return False @@ -476,6 +464,8 @@ class Git(FetchMethod): if os.path.exists(destdir): bb.utils.prunedir(destdir) + need_lfs = ud.parm.get("lfs", "1") == "1" + source_found = False source_error = [] @@ -503,6 +493,13 @@ class Git(FetchMethod): repourl = self._get_repo_url(ud) runfetchcmd("%s remote set-url origin %s" % (ud.basecmd, repourl), d, workdir=destdir) + + if self._contains_lfs(ud, d, destdir): + if need_lfs and not self._find_git_lfs(d): + raise bb.fetch2.FetchError("Repository %s has LFS content, install git-lfs on host to download (or set lfs=0 to ignore it)" % (repourl)) + else: + bb.note("Repository %s has LFS content but it is not being fetched" % (repourl)) + if not ud.nocheckout: if subdir != "": runfetchcmd("%s read-tree %s%s" % (ud.basecmd, ud.revisions[ud.names[0]], readpathspec), d, @@ -522,9 +519,17 @@ class Git(FetchMethod): def clean(self, ud, d): """ clean the git directory """ - bb.utils.remove(ud.localpath, True) - bb.utils.remove(ud.fullmirror) - bb.utils.remove(ud.fullmirror + ".done") + to_remove = [ud.localpath, ud.fullmirror, ud.fullmirror + ".done"] + # The localpath is a symlink to clonedir when it is cloned from a + # mirror, so remove both of them. + if os.path.islink(ud.localpath): + clonedir = os.path.realpath(ud.localpath) + to_remove.append(clonedir) + + for r in to_remove: + if os.path.exists(r): + bb.note('Removing %s' % r) + bb.utils.remove(r, True) def supports_srcrev(self): return True @@ -545,6 +550,27 @@ class Git(FetchMethod): raise bb.fetch2.FetchError("The command '%s' gave output with more then 1 line unexpectedly, output: '%s'" % (cmd, output)) return output.split()[0] != "0" + def _contains_lfs(self, ud, d, wd): + """ + Check if the repository has 'lfs' (large file) content + """ + cmd = "%s grep lfs HEAD:.gitattributes | wc -l" % ( + ud.basecmd) + try: + output = runfetchcmd(cmd, d, quiet=True, workdir=wd) + if int(output) > 0: + return True + except (bb.fetch2.FetchError,ValueError): + pass + return False + + def _find_git_lfs(self, d): + """ + Return True if git-lfs can be found, False otherwise. + """ + import shutil + return shutil.which("git-lfs", path=d.getVar('PATH')) is not None + def _get_repo_url(self, ud): """ Return the repository URL @@ -615,7 +641,7 @@ class Git(FetchMethod): """ pupver = ('', '') - tagregex = re.compile(d.getVar('UPSTREAM_CHECK_GITTAGREGEX') or "(?P([0-9][\.|_]?)+)") + tagregex = re.compile(d.getVar('UPSTREAM_CHECK_GITTAGREGEX') or r"(?P([0-9][\.|_]?)+)") try: output = self._lsremote(ud, d, "refs/tags/*") except (bb.fetch2.FetchError, bb.fetch2.NetworkAccess) as e: @@ -630,7 +656,7 @@ class Git(FetchMethod): tag_head = line.split("/")[-1] # Ignore non-released branches - m = re.search("(alpha|beta|rc|final)+", tag_head) + m = re.search(r"(alpha|beta|rc|final)+", tag_head) if m: continue diff --git a/bitbake/lib/bb/fetch2/gitannex.py b/bitbake/lib/bb/fetch2/gitannex.py index a9b69ca..1d497dc 100644 --- a/bitbake/lib/bb/fetch2/gitannex.py +++ b/bitbake/lib/bb/fetch2/gitannex.py @@ -1,5 +1,3 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- """ BitBake 'Fetch' git annex implementation """ @@ -7,18 +5,8 @@ BitBake 'Fetch' git annex implementation # Copyright (C) 2014 Otavio Salvador # Copyright (C) 2014 O.S. Systems Software LTDA. # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. +# SPDX-License-Identifier: GPL-2.0-only # -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. import os import bb diff --git a/bitbake/lib/bb/fetch2/gitsm.py b/bitbake/lib/bb/fetch2/gitsm.py index 35729db..c622771 100644 --- a/bitbake/lib/bb/fetch2/gitsm.py +++ b/bitbake/lib/bb/fetch2/gitsm.py @@ -1,5 +1,3 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- """ BitBake 'Fetch' git submodules implementation @@ -16,18 +14,8 @@ NOTE: Switching a SRC_URI from "git://" to "gitsm://" requires a clean of your r # Copyright (C) 2013 Richard Purdie # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. +# SPDX-License-Identifier: GPL-2.0-only # -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. import os import bb @@ -45,60 +33,97 @@ class GitSM(Git): """ return ud.type in ['gitsm'] - @staticmethod - def parse_gitmodules(gitmodules): - modules = {} - module = "" - for line in gitmodules.splitlines(): - if line.startswith('[submodule'): - module = line.split('"')[1] - modules[module] = {} - elif module and line.strip().startswith('path'): - path = line.split('=')[1].strip() - modules[module]['path'] = path - elif module and line.strip().startswith('url'): - url = line.split('=')[1].strip() - modules[module]['url'] = url - return modules - - def update_submodules(self, ud, d): + def process_submodules(self, ud, workdir, function, d): + """ + Iterate over all of the submodules in this repository and execute + the 'function' for each of them. + """ + submodules = [] paths = {} + revision = {} uris = {} - local_paths = {} - + subrevision = {} + + def parse_gitmodules(gitmodules): + modules = {} + module = "" + for line in gitmodules.splitlines(): + if line.startswith('[submodule'): + module = line.split('"')[1] + modules[module] = {} + elif module and line.strip().startswith('path'): + path = line.split('=')[1].strip() + modules[module]['path'] = path + elif module and line.strip().startswith('url'): + url = line.split('=')[1].strip() + modules[module]['url'] = url + return modules + + # Collect the defined submodules, and their attributes for name in ud.names: try: - gitmodules = runfetchcmd("%s show %s:.gitmodules" % (ud.basecmd, ud.revisions[name]), d, quiet=True, workdir=ud.clonedir) + gitmodules = runfetchcmd("%s show %s:.gitmodules" % (ud.basecmd, ud.revisions[name]), d, quiet=True, workdir=workdir) except: # No submodules to update continue - for m, md in self.parse_gitmodules(gitmodules).items(): + for m, md in parse_gitmodules(gitmodules).items(): + try: + module_hash = runfetchcmd("%s ls-tree -z -d %s %s" % (ud.basecmd, ud.revisions[name], md['path']), d, quiet=True, workdir=workdir) + except: + # If the command fails, we don't have a valid file to check. If it doesn't + # fail -- it still might be a failure, see next check... + module_hash = "" + + if not module_hash: + logger.debug(1, "submodule %s is defined, but is not initialized in the repository. Skipping", m) + continue + submodules.append(m) paths[m] = md['path'] + revision[m] = ud.revisions[name] uris[m] = md['url'] + subrevision[m] = module_hash.split()[2] + + # Convert relative to absolute uri based on parent uri if uris[m].startswith('..'): newud = copy.copy(ud) - newud.path = os.path.realpath(os.path.join(newud.path, md['url'])) + newud.path = os.path.realpath(os.path.join(newud.path, uris[m])) uris[m] = Git._get_repo_url(self, newud) for module in submodules: - module_hash = runfetchcmd("%s ls-tree -z -d %s %s" % (ud.basecmd, ud.revisions[name], paths[module]), d, quiet=True, workdir=ud.clonedir) - module_hash = module_hash.split()[2] + # Translate the module url into a SRC_URI + + if "://" in uris[module]: + # Properly formated URL already + proto = uris[module].split(':', 1)[0] + url = uris[module].replace('%s:' % proto, 'gitsm:', 1) + else: + if ":" in uris[module]: + # Most likely an SSH style reference + proto = "ssh" + if ":/" in uris[module]: + # Absolute reference, easy to convert.. + url = "gitsm://" + uris[module].replace(':/', '/', 1) + else: + # Relative reference, no way to know if this is right! + logger.warning("Submodule included by %s refers to relative ssh reference %s. References may fail if not absolute." % (ud.url, uris[module])) + url = "gitsm://" + uris[module].replace(':', '/', 1) + else: + # This has to be a file reference + proto = "file" + url = "gitsm://" + uris[module] - # Build new SRC_URI - proto = uris[module].split(':', 1)[0] - url = uris[module].replace('%s:' % proto, 'gitsm:', 1) url += ';protocol=%s' % proto url += ";name=%s" % module - url += ";bareclone=1;nocheckout=1;nobranch=1" + url += ";subpath=%s" % module ld = d.createCopy() # Not necessary to set SRC_URI, since we're passing the URI to # Fetch. #ld.setVar('SRC_URI', url) - ld.setVar('SRCREV_%s' % module, module_hash) + ld.setVar('SRCREV_%s' % module, subrevision[module]) # Workaround for issues with SRCPV/SRCREV_FORMAT errors # error refer to 'multiple' repositories. Only the repository @@ -106,145 +131,85 @@ class GitSM(Git): ld.setVar('SRCPV', d.getVar('SRCPV')) ld.setVar('SRCREV_FORMAT', module) - newfetch = Fetch([url], ld, cache=False) - newfetch.download() - local_paths[module] = newfetch.localpath(url) + function(ud, url, module, paths[module], ld) - # Correct the submodule references to the local download version... - runfetchcmd("%(basecmd)s config submodule.%(module)s.url %(url)s" % {'basecmd': ud.basecmd, 'module': module, 'url' : local_paths[module]}, d, workdir=ud.clonedir) - - symlink_path = os.path.join(ud.clonedir, 'modules', paths[module]) - if not os.path.exists(symlink_path): - try: - os.makedirs(os.path.dirname(symlink_path), exist_ok=True) - except OSError: - pass - os.symlink(local_paths[module], symlink_path) - - return True + return submodules != [] def need_update(self, ud, d): - main_repo_needs_update = Git.need_update(self, ud, d) - - # First check that the main repository has enough history fetched. If it doesn't, then we don't - # even have the .gitmodules and gitlinks for the submodules to attempt asking whether the - # submodules' histories are recent enough. - if main_repo_needs_update: + if Git.need_update(self, ud, d): return True - # Now check that the submodule histories are new enough. The git-submodule command doesn't have - # any clean interface for doing this aside from just attempting the checkout (with network - # fetched disabled). - return not self.update_submodules(ud, d) + try: + # Check for the nugget dropped by the download operation + known_srcrevs = runfetchcmd("%s config --get-all bitbake.srcrev" % \ + (ud.basecmd), d, workdir=ud.clonedir) - def download(self, ud, d): - Git.download(self, ud, d) + if ud.revisions[ud.names[0]] not in known_srcrevs.split(): + return True + except bb.fetch2.FetchError: + # No srcrev nuggets, so this is new and needs to be updated + return True - if not ud.shallow or ud.localpath != ud.fullshallow: - self.update_submodules(ud, d) + return False - def copy_submodules(self, submodules, ud, destdir, d): - if ud.bareclone: - repo_conf = destdir - else: - repo_conf = os.path.join(destdir, '.git') + def download(self, ud, d): + def download_submodule(ud, url, module, modpath, d): + url += ";bareclone=1;nobranch=1" - if submodules and not os.path.exists(os.path.join(repo_conf, 'modules')): - os.mkdir(os.path.join(repo_conf, 'modules')) + # Is the following still needed? + #url += ";nocheckout=1" - for module, md in submodules.items(): - srcpath = os.path.join(ud.clonedir, 'modules', md['path']) - modpath = os.path.join(repo_conf, 'modules', md['path']) + try: + newfetch = Fetch([url], d, cache=False) + newfetch.download() + # Drop a nugget to add each of the srcrevs we've fetched (used by need_update) + runfetchcmd("%s config --add bitbake.srcrev %s" % \ + (ud.basecmd, ud.revisions[ud.names[0]]), d, workdir=ud.clonedir) + except Exception as e: + logger.error('gitsm: submodule download failed: %s %s' % (type(e).__name__, str(e))) + raise - if os.path.exists(srcpath): - if os.path.exists(os.path.join(srcpath, '.git')): - srcpath = os.path.join(srcpath, '.git') + Git.download(self, ud, d) + self.process_submodules(ud, ud.clonedir, download_submodule, d) - target = modpath - if os.path.exists(modpath): - target = os.path.dirname(modpath) + def unpack(self, ud, destdir, d): + def unpack_submodules(ud, url, module, modpath, d): + url += ";bareclone=1;nobranch=1" - os.makedirs(os.path.dirname(target), exist_ok=True) - runfetchcmd("cp -fpLR %s %s" % (srcpath, target), d) - elif os.path.exists(modpath): - # Module already exists, likely unpacked from a shallow mirror clone - pass + # Figure out where we clone over the bare submodules... + if ud.bareclone: + repo_conf = ud.destdir else: - # This is fatal, as we do NOT want git-submodule to hit the network - raise bb.fetch2.FetchError('Submodule %s does not exist in %s or %s.' % (module, srcpath, modpath)) - - def clone_shallow_local(self, ud, dest, d): - super(GitSM, self).clone_shallow_local(ud, dest, d) + repo_conf = os.path.join(ud.destdir, '.git') - # Copy over the submodules' fetched histories too. - repo_conf = os.path.join(dest, '.git') - - submodules = [] - for name in ud.names: try: - gitmodules = runfetchcmd("%s show %s:.gitmodules" % (ud.basecmd, ud.revision), d, quiet=True, workdir=dest) - except: - # No submodules to update - continue + newfetch = Fetch([url], d, cache=False) + newfetch.unpack(root=os.path.dirname(os.path.join(repo_conf, 'modules', module))) + except Exception as e: + logger.error('gitsm: submodule unpack failed: %s %s' % (type(e).__name__, str(e))) + raise - submodules = self.parse_gitmodules(gitmodules) - self.copy_submodules(submodules, ud, dest, d) + local_path = newfetch.localpath(url) - def unpack(self, ud, destdir, d): - Git.unpack(self, ud, destdir, d) + # Correct the submodule references to the local download version... + runfetchcmd("%(basecmd)s config submodule.%(module)s.url %(url)s" % {'basecmd': ud.basecmd, 'module': module, 'url' : local_path}, d, workdir=ud.destdir) - # Copy over the submodules' fetched histories too. - if ud.bareclone: - repo_conf = ud.destdir - else: - repo_conf = os.path.join(ud.destdir, '.git') + if ud.shallow: + runfetchcmd("%(basecmd)s config submodule.%(module)s.shallow true" % {'basecmd': ud.basecmd, 'module': module}, d, workdir=ud.destdir) - update_submodules = False - paths = {} - uris = {} - local_paths = {} - for name in ud.names: + # Ensure the submodule repository is NOT set to bare, since we're checking it out... try: - gitmodules = runfetchcmd("%s show HEAD:.gitmodules" % (ud.basecmd), d, quiet=True, workdir=ud.destdir) + runfetchcmd("%s config core.bare false" % (ud.basecmd), d, quiet=True, workdir=os.path.join(repo_conf, 'modules', module)) except: - # No submodules to update - continue - - submodules = self.parse_gitmodules(gitmodules) - self.copy_submodules(submodules, ud, ud.destdir, d) - - submodules_queue = [(module, os.path.join(repo_conf, 'modules', md['path'])) for module, md in submodules.items()] - while len(submodules_queue) != 0: - module, modpath = submodules_queue.pop() - - # add submodule children recursively - try: - gitmodules = runfetchcmd("%s show HEAD:.gitmodules" % (ud.basecmd), d, quiet=True, workdir=modpath) - for m, md in self.parse_gitmodules(gitmodules).items(): - submodules_queue.append([m, os.path.join(modpath, 'modules', md['path'])]) - except: - # no children - pass - + logger.error("Unable to set git config core.bare to false for %s" % os.path.join(repo_conf, 'modules', module)) + raise - # There are submodules to update - update_submodules = True - - # Determine (from the submodule) the correct url to reference - try: - output = runfetchcmd("%(basecmd)s config remote.origin.url" % {'basecmd': ud.basecmd}, d, workdir=modpath) - except bb.fetch2.FetchError as e: - # No remote url defined in this submodule - continue - - local_paths[module] = output - - # Setup the local URL properly (like git submodule init or sync would do...) - runfetchcmd("%(basecmd)s config submodule.%(module)s.url %(url)s" % {'basecmd': ud.basecmd, 'module': module, 'url' : local_paths[module]}, d, workdir=ud.destdir) + Git.unpack(self, ud, destdir, d) - # Ensure the submodule repository is NOT set to bare, since we're checking it out... - runfetchcmd("%s config core.bare false" % (ud.basecmd), d, quiet=True, workdir=modpath) + ret = self.process_submodules(ud, ud.destdir, unpack_submodules, d) - if update_submodules: - # Run submodule update, this sets up the directories -- without touching the config + if not ud.bareclone and ret: + # All submodules should already be downloaded and configured in the tree. This simply sets + # up the configuration and checks out the files. The main project config should remain + # unmodified, and no download from the internet should occur. runfetchcmd("%s submodule update --recursive --no-fetch" % (ud.basecmd), d, quiet=True, workdir=ud.destdir) diff --git a/bitbake/lib/bb/fetch2/hg.py b/bitbake/lib/bb/fetch2/hg.py index 936d043..15d729e 100644 --- a/bitbake/lib/bb/fetch2/hg.py +++ b/bitbake/lib/bb/fetch2/hg.py @@ -1,5 +1,3 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- """ BitBake 'Fetch' implementation for mercurial DRCS (hg). @@ -9,20 +7,10 @@ BitBake 'Fetch' implementation for mercurial DRCS (hg). # Copyright (C) 2004 Marcin Juszkiewicz # Copyright (C) 2007 Robert Schuster # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only # # Based on functions from the base bb module, Copyright 2003 Holger Schurig +# import os import sys @@ -99,7 +87,7 @@ class Hg(FetchMethod): def try_premirror(self, ud, d): # If we don't do this, updating an existing checkout with only premirrors # is not possible - if d.getVar("BB_FETCH_PREMIRRORONLY") is not None: + if bb.utils.to_boolean(d.getVar("BB_FETCH_PREMIRRORONLY")): return True if os.path.exists(ud.moddir): return False diff --git a/bitbake/lib/bb/fetch2/local.py b/bitbake/lib/bb/fetch2/local.py index a114ac1..01d9ff9 100644 --- a/bitbake/lib/bb/fetch2/local.py +++ b/bitbake/lib/bb/fetch2/local.py @@ -1,5 +1,3 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- """ BitBake 'Fetch' implementations @@ -10,20 +8,10 @@ BitBake build tools. # Copyright (C) 2003, 2004 Chris Larson # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only # # Based on functions from the base bb module, Copyright 2003 Holger Schurig +# import os import urllib.request, urllib.parse, urllib.error diff --git a/bitbake/lib/bb/fetch2/npm.py b/bitbake/lib/bb/fetch2/npm.py index 65bf5a3..9700e61 100644 --- a/bitbake/lib/bb/fetch2/npm.py +++ b/bitbake/lib/bb/fetch2/npm.py @@ -1,5 +1,6 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- +# +# SPDX-License-Identifier: GPL-2.0-only +# """ BitBake 'Fetch' NPM implementation @@ -100,11 +101,19 @@ class Npm(FetchMethod): return False return True - def _runwget(self, ud, d, command, quiet): - logger.debug(2, "Fetching %s using command '%s'" % (ud.url, command)) - bb.fetch2.check_network_access(d, command, ud.url) + def _runpack(self, ud, d, pkgfullname: str, quiet=False) -> str: + """ + Runs npm pack on a full package name. + Returns the filename of the downloaded package + """ + bb.fetch2.check_network_access(d, pkgfullname, ud.registry) dldir = d.getVar("DL_DIR") - runfetchcmd(command, d, quiet, workdir=dldir) + dldir = os.path.join(dldir, ud.prefixdir) + + command = "npm pack {} --registry {}".format(pkgfullname, ud.registry) + logger.debug(2, "Fetching {} using command '{}' in {}".format(pkgfullname, command, dldir)) + filename = runfetchcmd(command, d, quiet, workdir=dldir) + return filename.rstrip() def _unpackdep(self, ud, pkg, data, destdir, dldir, d): file = data[pkg]['tgz'] @@ -150,20 +159,11 @@ class Npm(FetchMethod): Parse the output of npm view --json; the last JSON result is assumed to be the one that we're interested in. ''' - pdata = None - outdeps = {} - datalines = [] - bracelevel = 0 - for line in output.splitlines(): - if bracelevel: - datalines.append(line) - elif '{' in line: - datalines = [] - datalines.append(line) - bracelevel = bracelevel + line.count('{') - line.count('}') - if datalines: - pdata = json.loads('\n'.join(datalines)) - return pdata + pdata = json.loads(output); + try: + return pdata[-1] + except: + return pdata def _getdependencies(self, pkg, data, version, d, ud, optional=False, fetchedlist=None): if fetchedlist is None: @@ -171,6 +171,9 @@ class Npm(FetchMethod): pkgfullname = pkg if version != '*' and not '/' in version: pkgfullname += "@'%s'" % version + if pkgfullname in fetchedlist: + return + logger.debug(2, "Calling getdeps on %s" % pkg) fetchcmd = "npm view %s --json --registry %s" % (pkgfullname, ud.registry) output = runfetchcmd(fetchcmd, d, True) @@ -190,15 +193,10 @@ class Npm(FetchMethod): if (not blacklist and 'linux' not in pkg_os) or '!linux' in pkg_os: logger.debug(2, "Skipping %s since it's incompatible with Linux" % pkg) return - #logger.debug(2, "Output URL is %s - %s - %s" % (ud.basepath, ud.basename, ud.localfile)) - outputurl = pdata['dist']['tarball'] + filename = self._runpack(ud, d, pkgfullname) data[pkg] = {} - data[pkg]['tgz'] = os.path.basename(outputurl) - if outputurl in fetchedlist: - return - - self._runwget(ud, d, "%s --directory-prefix=%s %s" % (self.basecmd, ud.prefixdir, outputurl), False) - fetchedlist.append(outputurl) + data[pkg]['tgz'] = filename + fetchedlist.append(pkgfullname) dependencies = pdata.get('dependencies', {}) optionalDependencies = pdata.get('optionalDependencies', {}) @@ -225,17 +223,12 @@ class Npm(FetchMethod): if obj == pkg: self._getshrinkeddependencies(obj, data['dependencies'][obj], data['dependencies'][obj]['version'], d, ud, lockdown, manifest, False) return - outputurl = "invalid" - if ('resolved' not in data) or (not data['resolved'].startswith('http://') and not data['resolved'].startswith('https://')): - # will be the case for ${PN} - fetchcmd = "npm view %s@%s dist.tarball --registry %s" % (pkg, version, ud.registry) - logger.debug(2, "Found this matching URL: %s" % str(fetchcmd)) - outputurl = runfetchcmd(fetchcmd, d, True) - else: - outputurl = data['resolved'] - self._runwget(ud, d, "%s --directory-prefix=%s %s" % (self.basecmd, ud.prefixdir, outputurl), False) + + pkgnameWithVersion = "{}@{}".format(pkg, version) + logger.debug(2, "Get dependencies for {}".format(pkgnameWithVersion)) + filename = self._runpack(ud, d, pkgnameWithVersion) manifest[pkg] = {} - manifest[pkg]['tgz'] = os.path.basename(outputurl).rstrip() + manifest[pkg]['tgz'] = filename manifest[pkg]['deps'] = {} if pkg in lockdown: diff --git a/bitbake/lib/bb/fetch2/osc.py b/bitbake/lib/bb/fetch2/osc.py index 6c60456..3e56715 100644 --- a/bitbake/lib/bb/fetch2/osc.py +++ b/bitbake/lib/bb/fetch2/osc.py @@ -1,5 +1,6 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- +# +# SPDX-License-Identifier: GPL-2.0-only +# """ Bitbake "Fetch" implementation for osc (Opensuse build service client). Based on the svn "Fetch" implementation. diff --git a/bitbake/lib/bb/fetch2/perforce.py b/bitbake/lib/bb/fetch2/perforce.py index 903a8e6..54d001e 100644 --- a/bitbake/lib/bb/fetch2/perforce.py +++ b/bitbake/lib/bb/fetch2/perforce.py @@ -1,5 +1,3 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- """ BitBake 'Fetch' implementation for perforce @@ -8,18 +6,7 @@ BitBake 'Fetch' implementation for perforce # Copyright (C) 2003, 2004 Chris Larson # Copyright (C) 2016 Kodak Alaris, Inc. # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only # # Based on functions from the base bb module, Copyright 2003 Holger Schurig diff --git a/bitbake/lib/bb/fetch2/repo.py b/bitbake/lib/bb/fetch2/repo.py index 8c7e818..2bdbbd4 100644 --- a/bitbake/lib/bb/fetch2/repo.py +++ b/bitbake/lib/bb/fetch2/repo.py @@ -1,5 +1,3 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- """ BitBake "Fetch" repo (git) implementation @@ -8,20 +6,10 @@ BitBake "Fetch" repo (git) implementation # Copyright (C) 2009 Tom Rini # # Based on git.py which is: -#Copyright (C) 2005 Richard Purdie +# Copyright (C) 2005 Richard Purdie # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. +# SPDX-License-Identifier: GPL-2.0-only # -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. import os import bb diff --git a/bitbake/lib/bb/fetch2/s3.py b/bitbake/lib/bb/fetch2/s3.py index 1629288..ffca73c 100644 --- a/bitbake/lib/bb/fetch2/s3.py +++ b/bitbake/lib/bb/fetch2/s3.py @@ -1,5 +1,3 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- """ BitBake 'Fetch' implementation for Amazon AWS S3. @@ -13,18 +11,7 @@ The aws tool must be correctly installed and configured prior to use. # Based in part on bb.fetch2.wget: # Copyright (C) 2003, 2004 Chris Larson # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only # # Based on functions from the base bb module, Copyright 2003 Holger Schurig diff --git a/bitbake/lib/bb/fetch2/sftp.py b/bitbake/lib/bb/fetch2/sftp.py index 81884a6..f87f292 100644 --- a/bitbake/lib/bb/fetch2/sftp.py +++ b/bitbake/lib/bb/fetch2/sftp.py @@ -1,5 +1,3 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- """ BitBake SFTP Fetch implementation @@ -44,18 +42,7 @@ SRC_URI = "sftp://user@host.example.com/dir/path.file.txt" # Based in part on bb.fetch2.wget: # Copyright (C) 2003, 2004 Chris Larson # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only # # Based on functions from the base bb module, Copyright 2003 Holger Schurig diff --git a/bitbake/lib/bb/fetch2/ssh.py b/bitbake/lib/bb/fetch2/ssh.py index 6047ee4..f5be060 100644 --- a/bitbake/lib/bb/fetch2/ssh.py +++ b/bitbake/lib/bb/fetch2/ssh.py @@ -1,5 +1,3 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- ''' BitBake 'Fetch' implementations @@ -29,18 +27,8 @@ IETF secsh internet draft: # Copyright 2003 Holger Schurig # # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. +# SPDX-License-Identifier: GPL-2.0-only # -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. import re, os from bb.fetch2 import FetchMethod diff --git a/bitbake/lib/bb/fetch2/svn.py b/bitbake/lib/bb/fetch2/svn.py index ed70bcf..96d666b 100644 --- a/bitbake/lib/bb/fetch2/svn.py +++ b/bitbake/lib/bb/fetch2/svn.py @@ -1,5 +1,3 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- """ BitBake 'Fetch' implementation for svn. @@ -8,18 +6,7 @@ BitBake 'Fetch' implementation for svn. # Copyright (C) 2003, 2004 Chris Larson # Copyright (C) 2004 Marcin Juszkiewicz # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only # # Based on functions from the base bb module, Copyright 2003 Holger Schurig @@ -63,6 +50,9 @@ class Svn(FetchMethod): relpath = self._strip_leading_slashes(ud.path) ud.pkgdir = os.path.join(svndir, ud.host, relpath) ud.moddir = os.path.join(ud.pkgdir, ud.module) + # Protects the repository from concurrent updates, e.g. from two + # recipes fetching different revisions at the same time + ud.svnlock = os.path.join(ud.pkgdir, "svn.lock") ud.setup_revisions(d) @@ -101,6 +91,13 @@ class Svn(FetchMethod): svncmd = "%s log --limit 1 %s %s://%s/%s/" % (ud.basecmd, " ".join(options), proto, svnroot, ud.module) else: suffix = "" + + # externals may be either 'allowed' or 'nowarn', but not both. Allowed + # will not issue a warning, but will log to the debug buffer what has likely + # been downloaded by SVN. + if not ("externals" in ud.parm and ud.parm["externals"] == "allowed"): + options.append("--ignore-externals") + if ud.revision: options.append("-r %s" % ud.revision) suffix = "@%s" % (ud.revision) @@ -123,35 +120,52 @@ class Svn(FetchMethod): logger.debug(2, "Fetch: checking for module directory '" + ud.moddir + "'") - if os.access(os.path.join(ud.moddir, '.svn'), os.R_OK): - svnupdatecmd = self._buildsvncommand(ud, d, "update") - logger.info("Update " + ud.url) - # We need to attempt to run svn upgrade first in case its an older working format - try: - runfetchcmd(ud.basecmd + " upgrade", d, workdir=ud.moddir) - except FetchError: - pass - logger.debug(1, "Running %s", svnupdatecmd) - bb.fetch2.check_network_access(d, svnupdatecmd, ud.url) - runfetchcmd(svnupdatecmd, d, workdir=ud.moddir) - else: - svnfetchcmd = self._buildsvncommand(ud, d, "fetch") - logger.info("Fetch " + ud.url) - # check out sources there - bb.utils.mkdirhier(ud.pkgdir) - logger.debug(1, "Running %s", svnfetchcmd) - bb.fetch2.check_network_access(d, svnfetchcmd, ud.url) - runfetchcmd(svnfetchcmd, d, workdir=ud.pkgdir) - - scmdata = ud.parm.get("scmdata", "") - if scmdata == "keep": - tar_flags = "" - else: - tar_flags = "--exclude='.svn'" + lf = bb.utils.lockfile(ud.svnlock) + + try: + if os.access(os.path.join(ud.moddir, '.svn'), os.R_OK): + svnupdatecmd = self._buildsvncommand(ud, d, "update") + logger.info("Update " + ud.url) + # We need to attempt to run svn upgrade first in case its an older working format + try: + runfetchcmd(ud.basecmd + " upgrade", d, workdir=ud.moddir) + except FetchError: + pass + logger.debug(1, "Running %s", svnupdatecmd) + bb.fetch2.check_network_access(d, svnupdatecmd, ud.url) + runfetchcmd(svnupdatecmd, d, workdir=ud.moddir) + else: + svnfetchcmd = self._buildsvncommand(ud, d, "fetch") + logger.info("Fetch " + ud.url) + # check out sources there + bb.utils.mkdirhier(ud.pkgdir) + logger.debug(1, "Running %s", svnfetchcmd) + bb.fetch2.check_network_access(d, svnfetchcmd, ud.url) + runfetchcmd(svnfetchcmd, d, workdir=ud.pkgdir) + + if not ("externals" in ud.parm and ud.parm["externals"] == "nowarn"): + # Warn the user if this had externals (won't catch them all) + output = runfetchcmd("svn propget svn:externals || true", d, workdir=ud.moddir) + if output: + if "--ignore-externals" in svnfetchcmd.split(): + bb.warn("%s contains svn:externals." % ud.url) + bb.warn("These should be added to the recipe SRC_URI as necessary.") + bb.warn("svn fetch has ignored externals:\n%s" % output) + bb.warn("To disable this warning add ';externals=nowarn' to the url.") + else: + bb.debug(1, "svn repository has externals:\n%s" % output) + + scmdata = ud.parm.get("scmdata", "") + if scmdata == "keep": + tar_flags = "" + else: + tar_flags = "--exclude='.svn'" - # tar them up to a defined filename - runfetchcmd("tar %s -czf %s %s" % (tar_flags, ud.localpath, ud.path_spec), d, - cleanup=[ud.localpath], workdir=ud.pkgdir) + # tar them up to a defined filename + runfetchcmd("tar %s -czf %s %s" % (tar_flags, ud.localpath, ud.path_spec), d, + cleanup=[ud.localpath], workdir=ud.pkgdir) + finally: + bb.utils.unlockfile(lf) def clean(self, ud, d): """ Clean SVN specific files and dirs """ diff --git a/bitbake/lib/bb/fetch2/wget.py b/bitbake/lib/bb/fetch2/wget.py index 8f505b6..725586d 100644 --- a/bitbake/lib/bb/fetch2/wget.py +++ b/bitbake/lib/bb/fetch2/wget.py @@ -1,5 +1,3 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- """ BitBake 'Fetch' implementations @@ -10,18 +8,7 @@ BitBake build tools. # Copyright (C) 2003, 2004 Chris Larson # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only # # Based on functions from the base bb module, Copyright 2003 Holger Schurig @@ -33,11 +20,14 @@ import logging import errno import bb import bb.progress +import socket +import http.client import urllib.request, urllib.parse, urllib.error from bb.fetch2 import FetchMethod from bb.fetch2 import FetchError from bb.fetch2 import logger from bb.fetch2 import runfetchcmd +from bb.fetch2 import FetchConnectionCache from bb.utils import export_proxies from bs4 import BeautifulSoup from bs4 import SoupStrainer @@ -132,10 +122,6 @@ class Wget(FetchMethod): return True def checkstatus(self, fetch, ud, d, try_again=True): - import urllib.request, urllib.error, urllib.parse, socket, http.client - from urllib.response import addinfourl - from bb.fetch2 import FetchConnectionCache - class HTTPConnectionCache(http.client.HTTPConnection): if fetch.connection_cache: def connect(self): @@ -168,7 +154,7 @@ class Wget(FetchMethod): """ host = req.host if not host: - raise urlllib2.URLError('no host given') + raise urllib.error.URLError('no host given') h = http_class(host, timeout=req.timeout) # will parse host:port h.set_debuglevel(self._debuglevel) @@ -185,7 +171,7 @@ class Wget(FetchMethod): # request. # Don't close connection when connection_cache is enabled, - if fetch.connection_cache is None: + if fetch.connection_cache is None: headers["Connection"] = "close" else: headers["Connection"] = "Keep-Alive" # Works for HTTP/1.0 @@ -252,7 +238,7 @@ class Wget(FetchMethod): pass closed = False - resp = addinfourl(fp_dummy(), r.msg, req.get_full_url()) + resp = urllib.response.addinfourl(fp_dummy(), r.msg, req.get_full_url()) resp.code = r.status resp.msg = r.reason @@ -271,17 +257,18 @@ class Wget(FetchMethod): fp.read() fp.close() - newheaders = dict((k,v) for k,v in list(req.headers.items()) - if k.lower() not in ("content-length", "content-type")) - return self.parent.open(urllib.request.Request(req.get_full_url(), - headers=newheaders, - origin_req_host=req.origin_req_host, - unverifiable=True)) + if req.get_method() != 'GET': + newheaders = dict((k, v) for k, v in list(req.headers.items()) + if k.lower() not in ("content-length", "content-type")) + return self.parent.open(urllib.request.Request(req.get_full_url(), + headers=newheaders, + origin_req_host=req.origin_req_host, + unverifiable=True)) - """ - Some servers (e.g. GitHub archives, hosted on Amazon S3) return 403 - Forbidden when they actually mean 405 Method Not Allowed. - """ + raise urllib.request.HTTPError(req, code, msg, headers, None) + + # Some servers (e.g. GitHub archives, hosted on Amazon S3) return 403 + # Forbidden when they actually mean 405 Method Not Allowed. http_error_403 = http_error_405 @@ -292,15 +279,15 @@ class Wget(FetchMethod): """ def redirect_request(self, req, fp, code, msg, headers, newurl): newreq = urllib.request.HTTPRedirectHandler.redirect_request(self, req, fp, code, msg, headers, newurl) - newreq.get_method = lambda: req.get_method() + newreq.get_method = req.get_method return newreq exported_proxies = export_proxies(d) handlers = [FixedHTTPRedirectHandler, HTTPMethodFallback] - if export_proxies: + if exported_proxies: handlers.append(urllib.request.ProxyHandler()) handlers.append(CacheHTTPHandler()) - # XXX: Since Python 2.7.9 ssl cert validation is enabled by default + # Since Python 2.7.9 ssl cert validation is enabled by default # see PEP-0476, this causes verification errors on some https servers # so disable by default. import ssl @@ -319,19 +306,19 @@ class Wget(FetchMethod): '''Adds Basic auth to http request, pass in login:password as string''' import base64 encodeuser = base64.b64encode(login_str.encode('utf-8')).decode("utf-8") - authheader = "Basic %s" % encodeuser + authheader = "Basic %s" % encodeuser r.add_header("Authorization", authheader) - if ud.user: - add_basic_auth(ud.user, r) + if ud.user and ud.pswd: + add_basic_auth(ud.user + ':' + ud.pswd, r) try: - import netrc, urllib.parse + import netrc n = netrc.netrc() login, unused, password = n.authenticators(urllib.parse.urlparse(uri).hostname) add_basic_auth("%s:%s" % (login, password), r) except (TypeError, ImportError, IOError, netrc.NetrcParseError): - pass + pass with opener.open(r) as response: pass @@ -396,18 +383,14 @@ class Wget(FetchMethod): (oldpn, oldpv, oldsuffix) = old (newpn, newpv, newsuffix) = new - """ - Check for a new suffix type that we have never heard of before - """ - if (newsuffix): + # Check for a new suffix type that we have never heard of before + if newsuffix: m = self.suffix_regex_comp.search(newsuffix) if not m: bb.warn("%s has a possible unknown suffix: %s" % (newpn, newsuffix)) return False - """ - Not our package so ignore it - """ + # Not our package so ignore it if oldpn != newpn: return False @@ -473,15 +456,14 @@ class Wget(FetchMethod): return "" - def _check_latest_version_by_dir(self, dirver, package, package_regex, - current_version, ud, d): + def _check_latest_version_by_dir(self, dirver, package, package_regex, current_version, ud, d): """ - Scan every directory in order to get upstream version. + Scan every directory in order to get upstream version. """ version_dir = ['', '', ''] version = ['', '', ''] - dirver_regex = re.compile("(?P\D*)(?P(\d+[\.\-_])+(\d+))") + dirver_regex = re.compile(r"(?P\D*)(?P(\d+[\.\-_])+(\d+))") s = dirver_regex.search(dirver) if s: version_dir[1] = s.group('ver') @@ -541,26 +523,26 @@ class Wget(FetchMethod): gst-fluendo-mp3 """ # match most patterns which uses "-" as separator to version digits - pn_prefix1 = "[a-zA-Z][a-zA-Z0-9]*([-_][a-zA-Z]\w+)*\+?[-_]" + pn_prefix1 = r"[a-zA-Z][a-zA-Z0-9]*([-_][a-zA-Z]\w+)*\+?[-_]" # a loose pattern such as for unzip552.tar.gz - pn_prefix2 = "[a-zA-Z]+" + pn_prefix2 = r"[a-zA-Z]+" # a loose pattern such as for 80325-quicky-0.4.tar.gz - pn_prefix3 = "[0-9]+[-]?[a-zA-Z]+" + pn_prefix3 = r"[0-9]+[-]?[a-zA-Z]+" # Save the Package Name (pn) Regex for use later - pn_regex = "(%s|%s|%s)" % (pn_prefix1, pn_prefix2, pn_prefix3) + pn_regex = r"(%s|%s|%s)" % (pn_prefix1, pn_prefix2, pn_prefix3) # match version - pver_regex = "(([A-Z]*\d+[a-zA-Z]*[\.\-_]*)+)" + pver_regex = r"(([A-Z]*\d+[a-zA-Z]*[\.\-_]*)+)" # match arch parch_regex = "-source|_all_" # src.rpm extension was added only for rpm package. Can be removed if the rpm # packaged will always be considered as having to be manually upgraded - psuffix_regex = "(tar\.gz|tgz|tar\.bz2|zip|xz|tar\.lz|rpm|bz2|orig\.tar\.gz|tar\.xz|src\.tar\.gz|src\.tgz|svnr\d+\.tar\.bz2|stable\.tar\.gz|src\.rpm)" + psuffix_regex = r"(tar\.gz|tgz|tar\.bz2|zip|xz|tar\.lz|rpm|bz2|orig\.tar\.gz|tar\.xz|src\.tar\.gz|src\.tgz|svnr\d+\.tar\.bz2|stable\.tar\.gz|src\.rpm)" # match name, version and archive type of a package - package_regex_comp = re.compile("(?P%s?\.?v?)(?P%s)(?P%s)?[\.-](?P%s$)" + package_regex_comp = re.compile(r"(?P%s?\.?v?)(?P%s)(?P%s)?[\.-](?P%s$)" % (pn_regex, pver_regex, parch_regex, psuffix_regex)) self.suffix_regex_comp = re.compile(psuffix_regex) @@ -572,7 +554,7 @@ class Wget(FetchMethod): version = self._parse_path(package_regex_comp, package) if version: package_custom_regex_comp = re.compile( - "(?P%s)(?P%s)(?P%s)?[\.-](?P%s)" % + r"(?P%s)(?P%s)(?P%s)?[\.-](?P%s)" % (re.escape(version[0]), pver_regex, parch_regex, psuffix_regex)) else: package_custom_regex_comp = None @@ -589,7 +571,7 @@ class Wget(FetchMethod): current_version = ['', d.getVar('PV'), ''] """possible to have no version in pkg name, such as spectrum-fw""" - if not re.search("\d+", package): + if not re.search(r"\d+", package): current_version[1] = re.sub('_', '.', current_version[1]) current_version[1] = re.sub('-', '.', current_version[1]) return (current_version[1], '') @@ -607,13 +589,13 @@ class Wget(FetchMethod): # search for version matches on folders inside the path, like: # "5.7" in http://download.gnome.org/sources/${PN}/5.7/${PN}-${PV}.tar.gz - dirver_regex = re.compile("(?P[^/]*(\d+\.)*\d+([-_]r\d+)*)/") + dirver_regex = re.compile(r"(?P[^/]*(\d+\.)*\d+([-_]r\d+)*)/") m = dirver_regex.search(path) if m: pn = d.getVar('PN') dirver = m.group('dirver') - dirver_pn_regex = re.compile("%s\d?" % (re.escape(pn))) + dirver_pn_regex = re.compile(r"%s\d?" % (re.escape(pn))) if not dirver_pn_regex.search(dirver): return (self._check_latest_version_by_dir(dirver, package, package_regex, current_version, ud, d), '') diff --git a/bitbake/lib/bb/main.py b/bitbake/lib/bb/main.py index 732a315..af2880f 100755 --- a/bitbake/lib/bb/main.py +++ b/bitbake/lib/bb/main.py @@ -1,6 +1,3 @@ -#!/usr/bin/env python -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- # # Copyright (C) 2003, 2004 Chris Larson # Copyright (C) 2003, 2004 Phil Blundell @@ -9,18 +6,8 @@ # Copyright (C) 2005 ROAD GmbH # Copyright (C) 2006 Richard Purdie # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. +# SPDX-License-Identifier: GPL-2.0-only # -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. import os import sys @@ -267,6 +254,11 @@ class BitBakeConfigParameters(cookerdata.ConfigParameters): help="Do not run any setscene tasks. sstate will be ignored and " "everything needed, built.") + parser.add_option("", "--skip-setscene", action="store_true", + dest="skipsetscene", default=False, + help="Skip setscene tasks if they would be executed. Tasks previously " + "restored from sstate will be kept, unlike --no-setscene") + parser.add_option("", "--setscene-only", action="store_true", dest="setsceneonly", default=False, help="Only run setscene tasks, don't run any real tasks.") @@ -448,7 +440,7 @@ def setup_bitbake(configParams, configuration, extrafeatures=None): else: logger.info("Reconnecting to bitbake server...") if not os.path.exists(sockname): - print("Previous bitbake instance shutting down?, waiting to retry...") + logger.info("Previous bitbake instance shutting down?, waiting to retry...") i = 0 lock = None # Wait for 5s or until we can get the lock @@ -460,12 +452,7 @@ def setup_bitbake(configParams, configuration, extrafeatures=None): bb.utils.unlockfile(lock) raise bb.server.process.ProcessTimeout("Bitbake still shutting down as socket exists but no lock?") if not configParams.server_only: - try: - server_connection = bb.server.process.connectProcessServer(sockname, featureset) - except EOFError: - # The server may have been shutting down but not closed the socket yet. If that happened, - # ignore it. - pass + server_connection = bb.server.process.connectProcessServer(sockname, featureset) if server_connection or configParams.server_only: break @@ -475,12 +462,14 @@ def setup_bitbake(configParams, configuration, extrafeatures=None): if not retries: raise retries -= 1 - if isinstance(e, (bb.server.process.ProcessTimeout, BrokenPipeError)): - logger.info("Retrying server connection...") + tryno = 8 - retries + if isinstance(e, (bb.server.process.ProcessTimeout, BrokenPipeError, EOFError)): + logger.info("Retrying server connection (#%d)..." % tryno) else: - logger.info("Retrying server connection... (%s)" % traceback.format_exc()) + logger.info("Retrying server connection (#%d)... (%s)" % (tryno, traceback.format_exc())) if not retries: - bb.fatal("Unable to connect to bitbake server, or start one") + bb.fatal("Unable to connect to bitbake server, or start one (server startup failures would be in bitbake-cookerdaemon.log).") + bb.event.print_ui_queue() if retries < 5: time.sleep(5) @@ -502,7 +491,7 @@ def setup_bitbake(configParams, configuration, extrafeatures=None): def lockBitbake(): topdir = bb.cookerdata.findTopdir() if not topdir: - bb.error("Unable to find conf/bblayers.conf or conf/bitbake.conf. BBAPTH is unset and/or not in a build directory?") + bb.error("Unable to find conf/bblayers.conf or conf/bitbake.conf. BBPATH is unset and/or not in a build directory?") raise BBMainFatal lockfile = topdir + "/bitbake.lock" return topdir, bb.utils.lockfile(lockfile, False, False) diff --git a/bitbake/lib/bb/methodpool.py b/bitbake/lib/bb/methodpool.py index 49aed33..51783ac 100644 --- a/bitbake/lib/bb/methodpool.py +++ b/bitbake/lib/bb/methodpool.py @@ -1,21 +1,8 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- -# # # Copyright (C) 2006 Holger Hans Peter Freyther # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. +# SPDX-License-Identifier: GPL-2.0-only # -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. from bb.utils import better_compile, better_exec diff --git a/bitbake/lib/bb/monitordisk.py b/bitbake/lib/bb/monitordisk.py index 833cd3d..1a25b00 100644 --- a/bitbake/lib/bb/monitordisk.py +++ b/bitbake/lib/bb/monitordisk.py @@ -1,21 +1,8 @@ -#!/usr/bin/env python -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- # # Copyright (C) 2012 Robert Yang # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. +# SPDX-License-Identifier: GPL-2.0-only # -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. import os, logging, re, sys import bb @@ -28,16 +15,16 @@ def convertGMK(unit): """ Convert the space unit G, M, K, the unit is case-insensitive """ - unitG = re.match('([1-9][0-9]*)[gG]\s?$', unit) + unitG = re.match(r'([1-9][0-9]*)[gG]\s?$', unit) if unitG: return int(unitG.group(1)) * (1024 ** 3) - unitM = re.match('([1-9][0-9]*)[mM]\s?$', unit) + unitM = re.match(r'([1-9][0-9]*)[mM]\s?$', unit) if unitM: return int(unitM.group(1)) * (1024 ** 2) - unitK = re.match('([1-9][0-9]*)[kK]\s?$', unit) + unitK = re.match(r'([1-9][0-9]*)[kK]\s?$', unit) if unitK: return int(unitK.group(1)) * 1024 - unitN = re.match('([1-9][0-9]*)\s?$', unit) + unitN = re.match(r'([1-9][0-9]*)\s?$', unit) if unitN: return int(unitN.group(1)) else: @@ -83,7 +70,7 @@ def getDiskData(BBDirs, configuration): for pathSpaceInode in BBDirs.split(): # The input format is: "dir,space,inode", dir is a must, space # and inode are optional - pathSpaceInodeRe = re.match('([^,]*),([^,]*),([^,]*),?(.*)', pathSpaceInode) + pathSpaceInodeRe = re.match(r'([^,]*),([^,]*),([^,]*),?(.*)', pathSpaceInode) if not pathSpaceInodeRe: printErr("Invalid value in BB_DISKMON_DIRS: %s" % pathSpaceInode) return None @@ -147,7 +134,7 @@ def getInterval(configuration): else: # The disk space or inode interval is optional, but it should # have a correct value once it is specified - intervalRe = re.match('([^,]*),?\s*(.*)', interval) + intervalRe = re.match(r'([^,]*),?\s*(.*)', interval) if intervalRe: intervalSpace = intervalRe.group(1) if intervalSpace: diff --git a/bitbake/lib/bb/msg.py b/bitbake/lib/bb/msg.py index 96f077e..6216eb3 100644 --- a/bitbake/lib/bb/msg.py +++ b/bitbake/lib/bb/msg.py @@ -1,5 +1,3 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- """ BitBake 'msg' implementation @@ -9,18 +7,8 @@ Message handling infrastructure for bitbake # Copyright (C) 2006 Richard Purdie # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. +# SPDX-License-Identifier: GPL-2.0-only # -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. import sys import copy diff --git a/bitbake/lib/bb/namedtuple_with_abc.py b/bitbake/lib/bb/namedtuple_with_abc.py index 32f2fc6..646aed6 100644 --- a/bitbake/lib/bb/namedtuple_with_abc.py +++ b/bitbake/lib/bb/namedtuple_with_abc.py @@ -1,6 +1,8 @@ # http://code.activestate.com/recipes/577629-namedtupleabc-abstract-base-class-mix-in-for-named/ -#!/usr/bin/env python # Copyright (c) 2011 Jan Kaliszewski (zuo). Available under the MIT License. +# +# SPDX-License-Identifier: MIT +# """ namedtuple_with_abc.py: diff --git a/bitbake/lib/bb/parse/__init__.py b/bitbake/lib/bb/parse/__init__.py index 5397d57..76e180b 100644 --- a/bitbake/lib/bb/parse/__init__.py +++ b/bitbake/lib/bb/parse/__init__.py @@ -9,20 +9,10 @@ File parsers for the BitBake build tools. # Copyright (C) 2003, 2004 Chris Larson # Copyright (C) 2003, 2004 Phil Blundell # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only # # Based on functions from the base bb module, Copyright 2003 Holger Schurig +# handlers = [] diff --git a/bitbake/lib/bb/parse/ast.py b/bitbake/lib/bb/parse/ast.py index 6d7c80b..f0911e6 100644 --- a/bitbake/lib/bb/parse/ast.py +++ b/bitbake/lib/bb/parse/ast.py @@ -1,5 +1,3 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- """ AbstractSyntaxTree classes for the Bitbake language """ @@ -8,19 +6,8 @@ # Copyright (C) 2003, 2004 Phil Blundell # Copyright (C) 2009 Holger Hans Peter Freyther # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. +# SPDX-License-Identifier: GPL-2.0-only # -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - import re import string diff --git a/bitbake/lib/bb/parse/parse_py/BBHandler.py b/bitbake/lib/bb/parse/parse_py/BBHandler.py index 01fc47e..6f7cf82 100644 --- a/bitbake/lib/bb/parse/parse_py/BBHandler.py +++ b/bitbake/lib/bb/parse/parse_py/BBHandler.py @@ -1,6 +1,3 @@ -#!/usr/bin/env python -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- """ class for handling .bb files @@ -12,19 +9,8 @@ # Copyright (C) 2003, 2004 Chris Larson # Copyright (C) 2003, 2004 Phil Blundell # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. +# SPDX-License-Identifier: GPL-2.0-only # -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. - import re, bb, os import logging @@ -38,14 +24,15 @@ from .ConfHandler import include, init # For compatibility bb.deprecate_import(__name__, "bb.parse", ["vars_from_file"]) -__func_start_regexp__ = re.compile( r"(((?Ppython)|(?Pfakeroot))\s*)*(?P[\w\.\-\+\{\}\$]+)?\s*\(\s*\)\s*{$" ) -__inherit_regexp__ = re.compile( r"inherit\s+(.+)" ) -__export_func_regexp__ = re.compile( r"EXPORT_FUNCTIONS\s+(.+)" ) -__addtask_regexp__ = re.compile("addtask\s+(?P\w+)\s*((before\s*(?P((.*(?=after))|(.*))))|(after\s*(?P((.*(?=before))|(.*)))))*") -__deltask_regexp__ = re.compile("deltask\s+(?P\w+)") -__addhandler_regexp__ = re.compile( r"addhandler\s+(.+)" ) -__def_regexp__ = re.compile( r"def\s+(\w+).*:" ) -__python_func_regexp__ = re.compile( r"(\s+.*)|(^$)|(^#)" ) +__func_start_regexp__ = re.compile(r"(((?Ppython)|(?Pfakeroot))\s*)*(?P[\w\.\-\+\{\}\$]+)?\s*\(\s*\)\s*{$" ) +__inherit_regexp__ = re.compile(r"inherit\s+(.+)" ) +__export_func_regexp__ = re.compile(r"EXPORT_FUNCTIONS\s+(.+)" ) +__addtask_regexp__ = re.compile(r"addtask\s+(?P\w+)\s*((before\s*(?P((.*(?=after))|(.*))))|(after\s*(?P((.*(?=before))|(.*)))))*") +__deltask_regexp__ = re.compile(r"deltask\s+(?P\w+)(?P.*)") +__addhandler_regexp__ = re.compile(r"addhandler\s+(.+)" ) +__def_regexp__ = re.compile(r"def\s+(\w+).*:" ) +__python_func_regexp__ = re.compile(r"(\s+.*)|(^$)|(^#)" ) +__python_tab_regexp__ = re.compile(r" *\t") __infunc__ = [] __inpython__ = False @@ -160,6 +147,16 @@ def handle(fn, d, include): def feeder(lineno, s, fn, root, statements, eof=False): global __func_start_regexp__, __inherit_regexp__, __export_func_regexp__, __addtask_regexp__, __addhandler_regexp__, __def_regexp__, __python_func_regexp__, __inpython__, __infunc__, __body__, bb, __residue__, __classname__ + + # Check tabs in python functions: + # - def py_funcname(): covered by __inpython__ + # - python(): covered by '__anonymous' == __infunc__[0] + # - python funcname(): covered by __infunc__[3] + if __inpython__ or (__infunc__ and ('__anonymous' == __infunc__[0] or __infunc__[3])): + tab = __python_tab_regexp__.match(s) + if tab: + bb.warn('python should use 4 spaces indentation, but found tabs in %s, line %s' % (root, lineno)) + if __infunc__: if s == '}': __body__.append('') @@ -225,11 +222,27 @@ def feeder(lineno, s, fn, root, statements, eof=False): m = __addtask_regexp__.match(s) if m: + if len(m.group().split()) == 2: + # Check and warn for "addtask task1 task2" + m2 = re.match(r"addtask\s+(?P\w+)(?P.*)", s) + if m2 and m2.group('ignores'): + logger.warning('addtask ignored: "%s"' % m2.group('ignores')) + + # Check and warn for "addtask task1 before task2 before task3", the + # similar to "after" + taskexpression = s.split() + for word in ('before', 'after'): + if taskexpression.count(word) > 1: + logger.warning("addtask contained multiple '%s' keywords, only one is supported" % word) + ast.handleAddTask(statements, fn, lineno, m) return m = __deltask_regexp__.match(s) if m: + # Check and warn "for deltask task1 task2" + if m.group('ignores'): + logger.warning('deltask ignored: "%s"' % m.group('ignores')) ast.handleDelTask(statements, fn, lineno, m) return diff --git a/bitbake/lib/bb/parse/parse_py/ConfHandler.py b/bitbake/lib/bb/parse/parse_py/ConfHandler.py index 9d3ebe1..2e84b91 100644 --- a/bitbake/lib/bb/parse/parse_py/ConfHandler.py +++ b/bitbake/lib/bb/parse/parse_py/ConfHandler.py @@ -1,6 +1,3 @@ -#!/usr/bin/env python -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- """ class for handling configuration data files @@ -11,18 +8,8 @@ # Copyright (C) 2003, 2004 Chris Larson # Copyright (C) 2003, 2004 Phil Blundell # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. +# SPDX-License-Identifier: GPL-2.0-only # -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. import errno import re @@ -147,7 +134,7 @@ def handle(fn, data, include): continue s = s.rstrip() while s[-1] == '\\': - s2 = f.readline().strip() + s2 = f.readline().rstrip() lineno = lineno + 1 if (not s2 or s2 and s2[0] != "#") and s[0] == "#" : bb.fatal("There is a confusing multiline, partially commented expression on line %s of file %s (%s).\nPlease clarify whether this is all a comment or should be parsed." % (lineno, fn, s)) diff --git a/bitbake/lib/bb/parse/parse_py/__init__.py b/bitbake/lib/bb/parse/parse_py/__init__.py index 3e658d0..f508afa 100644 --- a/bitbake/lib/bb/parse/parse_py/__init__.py +++ b/bitbake/lib/bb/parse/parse_py/__init__.py @@ -1,6 +1,3 @@ -#!/usr/bin/env python -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- """ BitBake Parsers @@ -11,20 +8,10 @@ File parsers for the BitBake build tools. # Copyright (C) 2003, 2004 Chris Larson # Copyright (C) 2003, 2004 Phil Blundell # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# SPDX-License-Identifier: GPL-2.0-only # # Based on functions from the base bb module, Copyright 2003 Holger Schurig +# from __future__ import absolute_import from . import ConfHandler diff --git a/bitbake/lib/bb/persist_data.py b/bitbake/lib/bb/persist_data.py index bef7018..de8f87a 100644 --- a/bitbake/lib/bb/persist_data.py +++ b/bitbake/lib/bb/persist_data.py @@ -8,18 +8,8 @@ currently, providing a key/value store accessed by 'domain'. # Copyright (C) 2007 Richard Purdie # Copyright (C) 2010 Chris Larson # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. +# SPDX-License-Identifier: GPL-2.0-only # -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. import collections import logging @@ -29,6 +19,7 @@ import warnings from bb.compat import total_ordering from collections import Mapping import sqlite3 +import contextlib sqlversion = sqlite3.sqlite_version_info if sqlversion[0] < 3 or (sqlversion[0] == 3 and sqlversion[1] < 3): @@ -36,84 +27,181 @@ if sqlversion[0] < 3 or (sqlversion[0] == 3 and sqlversion[1] < 3): logger = logging.getLogger("BitBake.PersistData") -if hasattr(sqlite3, 'enable_shared_cache'): - try: - sqlite3.enable_shared_cache(True) - except sqlite3.OperationalError: - pass - @total_ordering class SQLTable(collections.MutableMapping): + class _Decorators(object): + @staticmethod + def retry(*, reconnect=True): + """ + Decorator that restarts a function if a database locked sqlite + exception occurs. If reconnect is True, the database connection + will be closed and reopened each time a failure occurs + """ + def retry_wrapper(f): + def wrap_func(self, *args, **kwargs): + # Reconnect if necessary + if self.connection is None and reconnect: + self.reconnect() + + count = 0 + while True: + try: + return f(self, *args, **kwargs) + except sqlite3.OperationalError as exc: + if count < 500 and ('is locked' in str(exc) or 'locking protocol' in str(exc)): + count = count + 1 + if reconnect: + self.reconnect() + continue + raise + return wrap_func + return retry_wrapper + + @staticmethod + def transaction(f): + """ + Decorator that starts a database transaction and creates a database + cursor for performing queries. If no exception is thrown, the + database results are commited. If an exception occurs, the database + is rolled back. In all cases, the cursor is closed after the + function ends. + + Note that the cursor is passed as an extra argument to the function + after `self` and before any of the normal arguments + """ + def wrap_func(self, *args, **kwargs): + # Context manager will COMMIT the database on success, + # or ROLLBACK on an exception + with self.connection: + # Automatically close the cursor when done + with contextlib.closing(self.connection.cursor()) as cursor: + return f(self, cursor, *args, **kwargs) + return wrap_func + """Object representing a table/domain in the database""" def __init__(self, cachefile, table): self.cachefile = cachefile self.table = table - self.cursor = connect(self.cachefile) - - self._execute("CREATE TABLE IF NOT EXISTS %s(key TEXT, value TEXT);" - % table) - - def _execute(self, *query): - """Execute a query, waiting to acquire a lock if necessary""" - count = 0 - while True: - try: - return self.cursor.execute(*query) - except sqlite3.OperationalError as exc: - if 'database is locked' in str(exc) and count < 500: - count = count + 1 + + self.connection = None + self._execute_single("CREATE TABLE IF NOT EXISTS %s(key TEXT PRIMARY KEY NOT NULL, value TEXT);" % table) + + @_Decorators.retry(reconnect=False) + @_Decorators.transaction + def _setup_database(self, cursor): + cursor.execute("pragma synchronous = off;") + # Enable WAL and keep the autocheckpoint length small (the default is + # usually 1000). Persistent caches are usually read-mostly, so keeping + # this short will keep readers running quickly + cursor.execute("pragma journal_mode = WAL;") + cursor.execute("pragma wal_autocheckpoint = 100;") + + def reconnect(self): + if self.connection is not None: + self.connection.close() + self.connection = sqlite3.connect(self.cachefile, timeout=5) + self.connection.text_factory = str + self._setup_database() + + @_Decorators.retry() + @_Decorators.transaction + def _execute_single(self, cursor, *query): + """ + Executes a single query and discards the results. This correctly closes + the database cursor when finished + """ + cursor.execute(*query) + + @_Decorators.retry() + def _row_iter(self, f, *query): + """ + Helper function that returns a row iterator. Each time __next__ is + called on the iterator, the provided function is evaluated to determine + the return value + """ + class CursorIter(object): + def __init__(self, cursor): + self.cursor = cursor + + def __iter__(self): + return self + + def __next__(self): + row = self.cursor.fetchone() + if row is None: self.cursor.close() - self.cursor = connect(self.cachefile) - continue - raise + raise StopIteration + return f(row) + + def __enter__(self): + return self + + def __exit__(self, typ, value, traceback): + self.cursor.close() + return False + + cursor = self.connection.cursor() + try: + cursor.execute(*query) + return CursorIter(cursor) + except: + cursor.close() def __enter__(self): - self.cursor.__enter__() + self.connection.__enter__() return self def __exit__(self, *excinfo): - self.cursor.__exit__(*excinfo) - - def __getitem__(self, key): - data = self._execute("SELECT * from %s where key=?;" % - self.table, [key]) - for row in data: + self.connection.__exit__(*excinfo) + + @_Decorators.retry() + @_Decorators.transaction + def __getitem__(self, cursor, key): + cursor.execute("SELECT * from %s where key=?;" % self.table, [key]) + row = cursor.fetchone() + if row is not None: return row[1] raise KeyError(key) - def __delitem__(self, key): + @_Decorators.retry() + @_Decorators.transaction + def __delitem__(self, cursor, key): if key not in self: raise KeyError(key) - self._execute("DELETE from %s where key=?;" % self.table, [key]) + cursor.execute("DELETE from %s where key=?;" % self.table, [key]) - def __setitem__(self, key, value): + @_Decorators.retry() + @_Decorators.transaction + def __setitem__(self, cursor, key, value): if not isinstance(key, str): raise TypeError('Only string keys are supported') elif not isinstance(value, str): raise TypeError('Only string values are supported') - data = self._execute("SELECT * from %s where key=?;" % - self.table, [key]) - exists = len(list(data)) - if exists: - self._execute("UPDATE %s SET value=? WHERE key=?;" % self.table, - [value, key]) + cursor.execute("SELECT * from %s where key=?;" % self.table, [key]) + row = cursor.fetchone() + if row is not None: + cursor.execute("UPDATE %s SET value=? WHERE key=?;" % self.table, [value, key]) else: - self._execute("INSERT into %s(key, value) values (?, ?);" % - self.table, [key, value]) - - def __contains__(self, key): - return key in set(self) - - def __len__(self): - data = self._execute("SELECT COUNT(key) FROM %s;" % self.table) - for row in data: + cursor.execute("INSERT into %s(key, value) values (?, ?);" % self.table, [key, value]) + + @_Decorators.retry() + @_Decorators.transaction + def __contains__(self, cursor, key): + cursor.execute('SELECT * from %s where key=?;' % self.table, [key]) + return cursor.fetchone() is not None + + @_Decorators.retry() + @_Decorators.transaction + def __len__(self, cursor): + cursor.execute("SELECT COUNT(key) FROM %s;" % self.table) + row = cursor.fetchone() + if row is not None: return row[0] def __iter__(self): - data = self._execute("SELECT key FROM %s;" % self.table) - return (row[0] for row in data) + return self._row_iter(lambda row: row[0], "SELECT key from %s;" % self.table) def __lt__(self, other): if not isinstance(other, Mapping): @@ -122,25 +210,27 @@ class SQLTable(collections.MutableMapping): return len(self) < len(other) def get_by_pattern(self, pattern): - data = self._execute("SELECT * FROM %s WHERE key LIKE ?;" % - self.table, [pattern]) - return [row[1] for row in data] + return self._row_iter(lambda row: row[1], "SELECT * FROM %s WHERE key LIKE ?;" % + self.table, [pattern]) def values(self): return list(self.itervalues()) def itervalues(self): - data = self._execute("SELECT value FROM %s;" % self.table) - return (row[0] for row in data) + return self._row_iter(lambda row: row[0], "SELECT value FROM %s;" % + self.table) def items(self): return list(self.iteritems()) def iteritems(self): - return self._execute("SELECT * FROM %s;" % self.table) + return self._row_iter(lambda row: (row[0], row[1]), "SELECT * FROM %s;" % + self.table) - def clear(self): - self._execute("DELETE FROM %s;" % self.table) + @_Decorators.retry() + @_Decorators.transaction + def clear(self, cursor): + cursor.execute("DELETE FROM %s;" % self.table) def has_key(self, key): return key in self @@ -194,12 +284,6 @@ class PersistData(object): """ del self.data[domain][key] -def connect(database): - connection = sqlite3.connect(database, timeout=5, isolation_level=None) - connection.execute("pragma synchronous = off;") - connection.text_factory = str - return connection - def persist(domain, d): """Convenience factory for SQLTable objects based upon metadata""" import bb.utils diff --git a/bitbake/lib/bb/process.py b/bitbake/lib/bb/process.py index e69697c..2dc472a 100644 --- a/bitbake/lib/bb/process.py +++ b/bitbake/lib/bb/process.py @@ -1,3 +1,7 @@ +# +# SPDX-License-Identifier: GPL-2.0-only +# + import logging import signal import subprocess diff --git a/bitbake/lib/bb/progress.py b/bitbake/lib/bb/progress.py index f54d1c7..4022caa 100644 --- a/bitbake/lib/bb/progress.py +++ b/bitbake/lib/bb/progress.py @@ -4,18 +4,8 @@ BitBake progress handling code # Copyright (C) 2016 Intel Corporation # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. +# SPDX-License-Identifier: GPL-2.0-only # -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. import sys import re @@ -23,6 +13,7 @@ import time import inspect import bb.event import bb.build +from bb.build import StdoutNoopContextManager class ProgressHandler(object): """ @@ -37,7 +28,14 @@ class ProgressHandler(object): if outfile: self._outfile = outfile else: - self._outfile = sys.stdout + self._outfile = StdoutNoopContextManager() + + def __enter__(self): + self._outfile.__enter__() + return self + + def __exit__(self, *excinfo): + self._outfile.__exit__(*excinfo) def _fire_progress(self, taskprogress, rate=None): """Internal function to fire the progress event""" @@ -157,6 +155,12 @@ class MultiStageProgressReporter(object): self._stage_total = None self._callers = [] + def __enter__(self): + return self + + def __exit__(self, *excinfo): + pass + def _fire_progress(self, taskprogress): bb.event.fire(bb.build.TaskProgress(taskprogress), self._data) diff --git a/bitbake/lib/bb/providers.py b/bitbake/lib/bb/providers.py index c2aa98c..f80963c 100644 --- a/bitbake/lib/bb/providers.py +++ b/bitbake/lib/bb/providers.py @@ -1,5 +1,3 @@ -# ex:ts=4:sw=4:sts=4:et -# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*- # # Copyright (C) 2003, 2004 Chris Larson # Copyright (C) 2003, 2004 Phil Blundell @@ -8,18 +6,8 @@ # Copyright (C) 2005 ROAD GmbH # Copyright (C) 2006 Richard Purdie # -# This program is free software; you can redistribute it and/or modify -# it under the terms of the GNU General Public License version 2 as -# published by the Free Software Foundation. +# SPDX-License-Identifier: GPL-2.0-only # -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. -# -# You should have received a copy of the GNU General Public License along -# with this program; if not, write to the Free Software Foundation, Inc., -# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. import re import logging @@ -129,7 +117,7 @@ def findPreferredProvider(pn, cfgData, dataCache, pkg_pn = None, item = None): preferred_v = cfgData.getVar("PREFERRED_VERSION") if preferred_v: - m = re.match('(\d+:)*(.*)(_.*)*', preferred_v) + m = re.match(r'(\d+:)*(.*)(_.*)*', preferred_v) if m: if m.group(1): preferred_e = m.group(1)[:-1] @@ -384,7 +372,7 @@ def getRuntimeProviders(dataCache, rdepend): # Only search dynamic packages if we can't find anything in other variables for pattern in dataCache.packages_dynamic: - pattern = pattern.replace('+', "\+") + pattern = pattern.replace(r'+', r"\+") if pattern in regexp_cache: regexp = regexp_cache[pattern] else: diff --git a/bitbake/lib/bb/pysh/builtin.py b/bitbake/lib/bb/pysh/builtin.py deleted file mode 100644 index a8814dc..0000000 --- a/bitbake/lib/bb/pysh/builtin.py +++ /dev/null @@ -1,710 +0,0 @@ -# builtin.py - builtins and utilities definitions for pysh. -# -# Copyright 2007 Patrick Mezard -# -# This software may be used and distributed according to the terms -# of the GNU General Public License, incorporated herein by reference. - -"""Builtin and internal utilities implementations. - -- Beware not to use python interpreter environment as if it were the shell -environment. For instance, commands working directory must be explicitely handled -through env['PWD'] instead of relying on python working directory. -""" -import errno -import optparse -import os -import re -import subprocess -import sys -import time - -def has_subprocess_bug(): - return getattr(subprocess, 'list2cmdline') and \ - ( subprocess.list2cmdline(['']) == '' or \ - subprocess.list2cmdline(['foo|bar']) == 'foo|bar') - -# Detect python bug 1634343: "subprocess swallows empty arguments under win32" -# -# Also detect: "[ 1710802 ] subprocess must escape redirection characters under win32" -# -if has_subprocess_bug(): - import subprocess_fix - subprocess.list2cmdline = subprocess_fix.list2cmdline - -from sherrors import * - -class NonExitingParser(optparse.OptionParser): - """OptionParser default behaviour upon error is to print the error message and - exit. Raise a utility error instead. - """ - def error(self, msg): - raise UtilityError(msg) - -#------------------------------------------------------------------------------- -# set special builtin -#------------------------------------------------------------------------------- -OPT_SET = NonExitingParser(usage="set - set or unset options and positional parameters") -OPT_SET.add_option( '-f', action='store_true', dest='has_f', default=False, - help='The shell shall disable pathname expansion.') -OPT_SET.add_option('-e', action='store_true', dest='has_e', default=False, - help="""When this option is on, if a simple command fails for any of the \ - reasons listed in Consequences of Shell Errors or returns an exit status \ - value >0, and is not part of the compound list following a while, until, \ - or if keyword, and is not a part of an AND or OR list, and is not a \ - pipeline preceded by the ! reserved word, then the shell shall immediately \ - exit.""") -OPT_SET.add_option('-x', action='store_true', dest='has_x', default=False, - help="""The shell shall write to standard error a trace for each command \ - after it expands the command and before it executes it. It is unspecified \ - whether the command that turns tracing off is traced.""") - -def builtin_set(name, args, interp, env, stdin, stdout, stderr, debugflags): - if 'debug-utility' in debugflags: - print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n') - - option, args = OPT_SET.parse_args(args) - env = interp.get_env() - - if option.has_f: - env.set_opt('-f') - if option.has_e: - env.set_opt('-e') - if option.has_x: - env.set_opt('-x') - return 0 - -#------------------------------------------------------------------------------- -# shift special builtin -#------------------------------------------------------------------------------- -def builtin_shift(name, args, interp, env, stdin, stdout, stderr, debugflags): - if 'debug-utility' in debugflags: - print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n') - - params = interp.get_env().get_positional_args() - if args: - try: - n = int(args[0]) - if n > len(params): - raise ValueError() - except ValueError: - return 1 - else: - n = 1 - - params[:n] = [] - interp.get_env().set_positional_args(params) - return 0 - -#------------------------------------------------------------------------------- -# export special builtin -#------------------------------------------------------------------------------- -OPT_EXPORT = NonExitingParser(usage="set - set or unset options and positional parameters") -OPT_EXPORT.add_option('-p', action='store_true', dest='has_p', default=False) - -def builtin_export(name, args, interp, env, stdin, stdout, stderr, debugflags): - if 'debug-utility' in debugflags: - print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n') - - option, args = OPT_EXPORT.parse_args(args) - if option.has_p: - raise NotImplementedError() - - for arg in args: - try: - name, value = arg.split('=', 1) - except ValueError: - name, value = arg, None - env = interp.get_env().export(name, value) - - return 0 - -#------------------------------------------------------------------------------- -# return special builtin -#------------------------------------------------------------------------------- -def builtin_return(name, args, interp, env, stdin, stdout, stderr, debugflags): - if 'debug-utility' in debugflags: - print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n') - res = 0 - if args: - try: - res = int(args[0]) - except ValueError: - res = 0 - if not 0<=res<=255: - res = 0 - - # BUG: should be last executed command exit code - raise ReturnSignal(res) - -#------------------------------------------------------------------------------- -# trap special builtin -#------------------------------------------------------------------------------- -def builtin_trap(name, args, interp, env, stdin, stdout, stderr, debugflags): - if 'debug-utility' in debugflags: - print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n') - if len(args) < 2: - stderr.write('trap: usage: trap [[arg] signal_spec ...]\n') - return 2 - - action = args[0] - for sig in args[1:]: - try: - env.traps[sig] = action - except Exception as e: - stderr.write('trap: %s\n' % str(e)) - return 0 - -#------------------------------------------------------------------------------- -# unset special builtin -#------------------------------------------------------------------------------- -OPT_UNSET = NonExitingParser("unset - unset values and attributes of variables and functions") -OPT_UNSET.add_option( '-f', action='store_true', dest='has_f', default=False) -OPT_UNSET.add_option( '-v', action='store_true', dest='has_v', default=False) - -def builtin_unset(name, args, interp, env, stdin, stdout, stderr, debugflags): - if 'debug-utility' in debugflags: - print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n') - - option, args = OPT_UNSET.parse_args(args) - - status = 0 - env = interp.get_env() - for arg in args: - try: - if option.has_f: - env.remove_function(arg) - else: - del env[arg] - except KeyError: - pass - except VarAssignmentError: - status = 1 - - return status - -#------------------------------------------------------------------------------- -# wait special builtin -#------------------------------------------------------------------------------- -def builtin_wait(name, args, interp, env, stdin, stdout, stderr, debugflags): - if 'debug-utility' in debugflags: - print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n') - - return interp.wait([int(arg) for arg in args]) - -#------------------------------------------------------------------------------- -# cat utility -#------------------------------------------------------------------------------- -def utility_cat(name, args, interp, env, stdin, stdout, stderr, debugflags): - if 'debug-utility' in debugflags: - print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n') - - if not args: - args = ['-'] - - status = 0 - for arg in args: - if arg == '-': - data = stdin.read() - else: - path = os.path.join(env['PWD'], arg) - try: - f = file(path, 'rb') - try: - data = f.read() - finally: - f.close() - except IOError as e: - if e.errno != errno.ENOENT: - raise - status = 1 - continue - stdout.write(data) - stdout.flush() - return status - -#------------------------------------------------------------------------------- -# cd utility -#------------------------------------------------------------------------------- -OPT_CD = NonExitingParser("cd - change the working directory") - -def utility_cd(name, args, interp, env, stdin, stdout, stderr, debugflags): - if 'debug-utility' in debugflags: - print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n') - - option, args = OPT_CD.parse_args(args) - env = interp.get_env() - - directory = None - printdir = False - if not args: - home = env.get('HOME') - if home: - # Unspecified, do nothing - return 0 - else: - directory = home - elif len(args)==1: - directory = args[0] - if directory=='-': - if 'OLDPWD' not in env: - raise UtilityError("OLDPWD not set") - printdir = True - directory = env['OLDPWD'] - else: - raise UtilityError("too many arguments") - - curpath = None - # Absolute directories will be handled correctly by the os.path.join call. - if not directory.startswith('.') and not directory.startswith('..'): - cdpaths = env.get('CDPATH', '.').split(';') - for cdpath in cdpaths: - p = os.path.join(cdpath, directory) - if os.path.isdir(p): - curpath = p - break - - if curpath is None: - curpath = directory - curpath = os.path.join(env['PWD'], directory) - - env['OLDPWD'] = env['PWD'] - env['PWD'] = curpath - if printdir: - stdout.write('%s\n' % curpath) - return 0 - -#------------------------------------------------------------------------------- -# colon utility -#------------------------------------------------------------------------------- -def utility_colon(name, args, interp, env, stdin, stdout, stderr, debugflags): - if 'debug-utility' in debugflags: - print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n') - return 0 - -#------------------------------------------------------------------------------- -# echo utility -#------------------------------------------------------------------------------- -def utility_echo(name, args, interp, env, stdin, stdout, stderr, debugflags): - if 'debug-utility' in debugflags: - print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n') - - # Echo only takes arguments, no options. Use printf if you need fancy stuff. - output = ' '.join(args) + '\n' - stdout.write(output) - stdout.flush() - return 0 - -#------------------------------------------------------------------------------- -# egrep utility -#------------------------------------------------------------------------------- -# egrep is usually a shell script. -# Unfortunately, pysh does not support shell scripts *with arguments* right now, -# so the redirection is implemented here, assuming grep is available. -def utility_egrep(name, args, interp, env, stdin, stdout, stderr, debugflags): - if 'debug-utility' in debugflags: - print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n') - - return run_command('grep', ['-E'] + args, interp, env, stdin, stdout, - stderr, debugflags) - -#------------------------------------------------------------------------------- -# env utility -#------------------------------------------------------------------------------- -def utility_env(name, args, interp, env, stdin, stdout, stderr, debugflags): - if 'debug-utility' in debugflags: - print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n') - - if args and args[0]=='-i': - raise NotImplementedError('env: -i option is not implemented') - - i = 0 - for arg in args: - if '=' not in arg: - break - # Update the current environment - name, value = arg.split('=', 1) - env[name] = value - i += 1 - - if args[i:]: - # Find then execute the specified interpreter - utility = env.find_in_path(args[i]) - if not utility: - return 127 - args[i:i+1] = utility - name = args[i] - args = args[i+1:] - try: - return run_command(name, args, interp, env, stdin, stdout, stderr, - debugflags) - except UtilityError: - stderr.write('env: failed to execute %s' % ' '.join([name]+args)) - return 126 - else: - for pair in env.get_variables().iteritems(): - stdout.write('%s=%s\n' % pair) - return 0 - -#------------------------------------------------------------------------------- -# exit utility -#------------------------------------------------------------------------------- -def utility_exit(name, args, interp, env, stdin, stdout, stderr, debugflags): - if 'debug-utility' in debugflags: - print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n') - - res = None - if args: - try: - res = int(args[0]) - except ValueError: - res = None - if not 0<=res<=255: - res = None - - if res is None: - # BUG: should be last executed command exit code - res = 0 - - raise ExitSignal(res) - -#------------------------------------------------------------------------------- -# fgrep utility -#------------------------------------------------------------------------------- -# see egrep -def utility_fgrep(name, args, interp, env, stdin, stdout, stderr, debugflags): - if 'debug-utility' in debugflags: - print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n') - - return run_command('grep', ['-F'] + args, interp, env, stdin, stdout, - stderr, debugflags) - -#------------------------------------------------------------------------------- -# gunzip utility -#------------------------------------------------------------------------------- -# see egrep -def utility_gunzip(name, args, interp, env, stdin, stdout, stderr, debugflags): - if 'debug-utility' in debugflags: - print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n') - - return run_command('gzip', ['-d'] + args, interp, env, stdin, stdout, - stderr, debugflags) - -#------------------------------------------------------------------------------- -# kill utility -#------------------------------------------------------------------------------- -def utility_kill(name, args, interp, env, stdin, stdout, stderr, debugflags): - if 'debug-utility' in debugflags: - print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n') - - for arg in args: - pid = int(arg) - status = subprocess.call(['pskill', '/T', str(pid)], - shell=True, - stdout=subprocess.PIPE, - stderr=subprocess.PIPE) - # pskill is asynchronous, hence the stupid polling loop - while 1: - p = subprocess.Popen(['pslist', str(pid)], - shell=True, - stdout=subprocess.PIPE, - stderr=subprocess.STDOUT) - output = p.communicate()[0] - if ('process %d was not' % pid) in output: - break - time.sleep(1) - return status - -#------------------------------------------------------------------------------- -# mkdir utility -#------------------------------------------------------------------------------- -OPT_MKDIR = NonExitingParser("mkdir - make directories.") -OPT_MKDIR.add_option('-p', action='store_true', dest='has_p', default=False) - -def utility_mkdir(name, args, interp, env, stdin, stdout, stderr, debugflags): - if 'debug-utility' in debugflags: - print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n') - - # TODO: implement umask - # TODO: implement proper utility error report - option, args = OPT_MKDIR.parse_args(args) - for arg in args: - path = os.path.join(env['PWD'], arg) - if option.has_p: - try: - os.makedirs(path) - except IOError as e: - if e.errno != errno.EEXIST: - raise - else: - os.mkdir(path) - return 0 - -#------------------------------------------------------------------------------- -# netstat utility -#------------------------------------------------------------------------------- -def utility_netstat(name, args, interp, env, stdin, stdout, stderr, debugflags): - # Do you really expect me to implement netstat ? - # This empty form is enough for Mercurial tests since it's - # supposed to generate nothing upon success. Faking this test - # is not a big deal either. - if 'debug-utility' in debugflags: - print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n') - return 0 - -#------------------------------------------------------------------------------- -# pwd utility -#------------------------------------------------------------------------------- -OPT_PWD = NonExitingParser("pwd - return working directory name") -OPT_PWD.add_option('-L', action='store_true', dest='has_L', default=True, - help="""If the PWD environment variable contains an absolute pathname of \ - the current directory that does not contain the filenames dot or dot-dot, \ - pwd shall write this pathname to standard output. Otherwise, the -L option \ - shall behave as the -P option.""") -OPT_PWD.add_option('-P', action='store_true', dest='has_L', default=False, - help="""The absolute pathname written shall not contain filenames that, in \ - the context of the pathname, refer to files of type symbolic link.""") - -def utility_pwd(name, args, interp, env, stdin, stdout, stderr, debugflags): - if 'debug-utility' in debugflags: - print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n') - - option, args = OPT_PWD.parse_args(args) - stdout.write('%s\n' % env['PWD']) - return 0 - -#------------------------------------------------------------------------------- -# printf utility -#------------------------------------------------------------------------------- -RE_UNESCAPE = re.compile(r'(\\x[a-zA-Z0-9]{2}|\\[0-7]{1,3}|\\.)') - -def utility_printf(name, args, interp, env, stdin, stdout, stderr, debugflags): - if 'debug-utility' in debugflags: - print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n') - - def replace(m): - assert m.group() - g = m.group()[1:] - if g.startswith('x'): - return chr(int(g[1:], 16)) - if len(g) <= 3 and len([c for c in g if c in '01234567']) == len(g): - # Yay, an octal number - return chr(int(g, 8)) - return { - 'a': '\a', - 'b': '\b', - 'f': '\f', - 'n': '\n', - 'r': '\r', - 't': '\t', - 'v': '\v', - '\\': '\\', - }.get(g) - - # Convert escape sequences - format = re.sub(RE_UNESCAPE, replace, args[0]) - stdout.write(format % tuple(args[1:])) - return 0 - -#------------------------------------------------------------------------------- -# true utility -#------------------------------------------------------------------------------- -def utility_true(name, args, interp, env, stdin, stdout, stderr, debugflags): - if 'debug-utility' in debugflags: - print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n') - return 0 - -#------------------------------------------------------------------------------- -# sed utility -#------------------------------------------------------------------------------- -RE_SED = re.compile(r'^s(.).*\1[a-zA-Z]*$') - -# cygwin sed fails with some expressions when they do not end with a single space. -# see unit tests for details. Interestingly, the same expressions works perfectly -# in cygwin shell. -def utility_sed(name, args, interp, env, stdin, stdout, stderr, debugflags): - if 'debug-utility' in debugflags: - print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n') - - # Scan pattern arguments and append a space if necessary - for i in range(len(args)): - if not RE_SED.search(args[i]): - continue - args[i] = args[i] + ' ' - - return run_command(name, args, interp, env, stdin, stdout, - stderr, debugflags) - -#------------------------------------------------------------------------------- -# sleep utility -#------------------------------------------------------------------------------- -def utility_sleep(name, args, interp, env, stdin, stdout, stderr, debugflags): - if 'debug-utility' in debugflags: - print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n') - time.sleep(int(args[0])) - return 0 - -#------------------------------------------------------------------------------- -# sort utility -#------------------------------------------------------------------------------- -OPT_SORT = NonExitingParser("sort - sort, merge, or sequence check text files") - -def utility_sort(name, args, interp, env, stdin, stdout, stderr, debugflags): - - def sort(path): - if path == '-': - lines = stdin.readlines() - else: - try: - f = file(path) - try: - lines = f.readlines() - finally: - f.close() - except IOError as e: - stderr.write(str(e) + '\n') - return 1 - - if lines and lines[-1][-1]!='\n': - lines[-1] = lines[-1] + '\n' - return lines - - if 'debug-utility' in debugflags: - print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n') - - option, args = OPT_SORT.parse_args(args) - alllines = [] - - if len(args)<=0: - args += ['-'] - - # Load all files lines - curdir = os.getcwd() - try: - os.chdir(env['PWD']) - for path in args: - alllines += sort(path) - finally: - os.chdir(curdir) - - alllines.sort() - for line in alllines: - stdout.write(line) - return 0 - -#------------------------------------------------------------------------------- -# hg utility -#------------------------------------------------------------------------------- - -hgcommands = [ - 'add', - 'addremove', - 'commit', 'ci', - 'debugrename', - 'debugwalk', - 'falabala', # Dummy command used in a mercurial test - 'incoming', - 'locate', - 'pull', - 'push', - 'qinit', - 'remove', 'rm', - 'rename', 'mv', - 'revert', - 'showconfig', - 'status', 'st', - 'strip', - ] - -def rewriteslashes(name, args): - # Several hg commands output file paths, rewrite the separators - if len(args) > 1 and name.lower().endswith('python') \ - and args[0].endswith('hg'): - for cmd in hgcommands: - if cmd in args[1:]: - return True - - # svn output contains many paths with OS specific separators. - # Normalize these to unix paths. - base = os.path.basename(name) - if base.startswith('svn'): - return True - - return False - -def rewritehg(output): - if not output: - return output - # Rewrite os specific messages - output = output.replace(': The system cannot find the file specified', - ': No such file or directory') - output = re.sub(': Access is denied.*$', ': Permission denied', output) - output = output.replace(': No connection could be made because the target machine actively refused it', - ': Connection refused') - return output - - -def run_command(name, args, interp, env, stdin, stdout, - stderr, debugflags): - # Execute the command - if 'debug-utility' in debugflags: - print interp.log(' '.join([name, str(args), interp['PWD']]) + '\n') - - hgbin = interp.options().hgbinary - ishg = hgbin and ('hg' in name or args and 'hg' in args[0]) - unixoutput = 'cygwin' in name or ishg - - exec_env = env.get_variables() - try: - # BUG: comparing file descriptor is clearly not a reliable way to tell - # whether they point on the same underlying object. But in pysh limited - # scope this is usually right, we do not expect complicated redirections - # besides usual 2>&1. - # Still there is one case we have but cannot deal with is when stdout - # and stderr are redirected *by pysh caller*. This the reason for the - # --redirect pysh() option. - # Now, we want to know they are the same because we sometimes need to - # transform the command output, mostly remove CR-LF to ensure that - # command output is unix-like. Cygwin utilies are a special case because - # they explicitely set their output streams to binary mode, so we have - # nothing to do. For all others commands, we have to guess whether they - # are sending text data, in which case the transformation must be done. - # Again, the NUL character test is unreliable but should be enough for - # hg tests. - redirected = stdout.fileno()==stderr.fileno() - if not redirected: - p = subprocess.Popen([name] + args, cwd=env['PWD'], env=exec_env, - stdin=stdin, stdout=subprocess.PIPE, stderr=subprocess.PIPE) - else: - p = subprocess.Popen([name] + args, cwd=env['PWD'], env=exec_env, - stdin=stdin, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) - out, err = p.communicate() - except WindowsError as e: - raise UtilityError(str(e)) - - if not unixoutput: - def encode(s): - if '\0' in s: - return s - return s.replace('\r\n', '\n') - else: - encode = lambda s: s - - if rewriteslashes(name, args): - encode1_ = encode - def encode(s): - s = encode1_(s) - s = s.replace('\\\\', '\\') - s = s.replace('\\', '/') - return s - - if ishg: - encode2_ = encode - def encode(s): - return rewritehg(encode2_(s)) - - stdout.write(encode(out)) - if not redirected: - stderr.write(encode(err)) - return p.returncode - diff --git a/bitbake/lib/bb/pysh/interp.py b/bitbake/lib/bb/pysh/interp.py deleted file mode 100644 index d14ecf3..0000000 --- a/bitbake/lib/bb/pysh/interp.py +++ /dev/null @@ -1,1367 +0,0 @@ -# interp.py - shell interpreter for pysh. -# -# Copyright 2007 Patrick Mezard -# -# This software may be used and distributed according to the terms -# of the GNU General Public License, incorporated herein by reference. - -"""Implement the shell interpreter. - -Most references are made to "The Open Group Base Specifications Issue 6". - -""" -# TODO: document the fact input streams must implement fileno() so Popen will work correctly. -# it requires non-stdin stream to be implemented as files. Still to be tested... -# DOC: pathsep is used in PATH instead of ':'. Clearly, there are path syntax issues here. -# TODO: stop command execution upon error. -# TODO: sort out the filename/io_number mess. It should be possible to use filenames only. -# TODO: review subshell implementation -# TODO: test environment cloning for non-special builtins -# TODO: set -x should not rebuild commands from tokens, assignments/redirections are lost -# TODO: unit test for variable assignment -# TODO: test error management wrt error type/utility type -# TODO: test for binary output everywhere -# BUG: debug-parsing does not pass log file to PLY. Maybe a PLY upgrade is necessary. -import base64 -import cPickle as pickle -import errno -import glob -import os -import re -import subprocess -import sys -import tempfile - -try: - s = set() - del s -except NameError: - from Set import Set as set - -import builtin -from sherrors import * -import pyshlex -import pyshyacc - -def mappend(func, *args, **kargs): - """Like map but assume func returns a list. Returned lists are merged into - a single one. - """ - return reduce(lambda a,b: a+b, map(func, *args, **kargs), []) - -class FileWrapper: - """File object wrapper to ease debugging. - - Allow mode checking and implement file duplication through a simple - reference counting scheme. Not sure the latter is really useful since - only real file descriptors can be used. - """ - def __init__(self, mode, file, close=True): - if mode not in ('r', 'w', 'a'): - raise IOError('invalid mode: %s' % mode) - self._mode = mode - self._close = close - if isinstance(file, FileWrapper): - if file._refcount[0] <= 0: - raise IOError(0, 'Error') - self._refcount = file._refcount - self._refcount[0] += 1 - self._file = file._file - else: - self._refcount = [1] - self._file = file - - def dup(self): - return FileWrapper(self._mode, self, self._close) - - def fileno(self): - """fileno() should be only necessary for input streams.""" - return self._file.fileno() - - def read(self, size=-1): - if self._mode!='r': - raise IOError(0, 'Error') - return self._file.read(size) - - def readlines(self, *args, **kwargs): - return self._file.readlines(*args, **kwargs) - - def write(self, s): - if self._mode not in ('w', 'a'): - raise IOError(0, 'Error') - return self._file.write(s) - - def flush(self): - self._file.flush() - - def close(self): - if not self._refcount: - return - assert self._refcount[0] > 0 - - self._refcount[0] -= 1 - if self._refcount[0] == 0: - self._mode = 'c' - if self._close: - self._file.close() - self._refcount = None - - def mode(self): - return self._mode - - def __getattr__(self, name): - if name == 'name': - self.name = getattr(self._file, name) - return self.name - else: - raise AttributeError(name) - - def __del__(self): - self.close() - - -def win32_open_devnull(mode): - return open('NUL', mode) - - -class Redirections: - """Stores open files and their mapping to pseudo-sh file descriptor. - """ - # BUG: redirections are not handled correctly: 1>&3 2>&3 3>&4 does - # not make 1 to redirect to 4 - def __init__(self, stdin=None, stdout=None, stderr=None): - self._descriptors = {} - if stdin is not None: - self._add_descriptor(0, stdin) - if stdout is not None: - self._add_descriptor(1, stdout) - if stderr is not None: - self._add_descriptor(2, stderr) - - def add_here_document(self, interp, name, content, io_number=None): - if io_number is None: - io_number = 0 - - if name==pyshlex.unquote_wordtree(name): - content = interp.expand_here_document(('TOKEN', content)) - - # Write document content in a temporary file - tmp = tempfile.TemporaryFile() - try: - tmp.write(content) - tmp.flush() - tmp.seek(0) - self._add_descriptor(io_number, FileWrapper('r', tmp)) - except: - tmp.close() - raise - - def add(self, interp, op, filename, io_number=None): - if op not in ('<', '>', '>|', '>>', '>&'): - # TODO: add descriptor duplication and here_documents - raise RedirectionError('Unsupported redirection operator "%s"' % op) - - if io_number is not None: - io_number = int(io_number) - - if (op == '>&' and filename.isdigit()) or filename=='-': - # No expansion for file descriptors, quote them if you want a filename - fullname = filename - else: - if filename.startswith('/'): - # TODO: win32 kludge - if filename=='/dev/null': - fullname = 'NUL' - else: - # TODO: handle absolute pathnames, they are unlikely to exist on the - # current platform (win32 for instance). - raise NotImplementedError() - else: - fullname = interp.expand_redirection(('TOKEN', filename)) - if not fullname: - raise RedirectionError('%s: ambiguous redirect' % filename) - # Build absolute path based on PWD - fullname = os.path.join(interp.get_env()['PWD'], fullname) - - if op=='<': - return self._add_input_redirection(interp, fullname, io_number) - elif op in ('>', '>|'): - clobber = ('>|'==op) - return self._add_output_redirection(interp, fullname, io_number, clobber) - elif op=='>>': - return self._add_output_appending(interp, fullname, io_number) - elif op=='>&': - return self._dup_output_descriptor(fullname, io_number) - - def close(self): - if self._descriptors is not None: - for desc in self._descriptors.itervalues(): - desc.flush() - desc.close() - self._descriptors = None - - def stdin(self): - return self._descriptors[0] - - def stdout(self): - return self._descriptors[1] - - def stderr(self): - return self._descriptors[2] - - def clone(self): - clone = Redirections() - for desc, fileobj in self._descriptors.iteritems(): - clone._descriptors[desc] = fileobj.dup() - return clone - - def _add_output_redirection(self, interp, filename, io_number, clobber): - if io_number is None: - # io_number default to standard output - io_number = 1 - - if not clobber and interp.get_env().has_opt('-C') and os.path.isfile(filename): - # File already exist in no-clobber mode, bail out - raise RedirectionError('File "%s" already exists' % filename) - - # Open and register - self._add_file_descriptor(io_number, filename, 'w') - - def _add_output_appending(self, interp, filename, io_number): - if io_number is None: - io_number = 1 - self._add_file_descriptor(io_number, filename, 'a') - - def _add_input_redirection(self, interp, filename, io_number): - if io_number is None: - io_number = 0 - self._add_file_descriptor(io_number, filename, 'r') - - def _add_file_descriptor(self, io_number, filename, mode): - try: - if filename.startswith('/'): - if filename=='/dev/null': - f = win32_open_devnull(mode+'b') - else: - # TODO: handle absolute pathnames, they are unlikely to exist on the - # current platform (win32 for instance). - raise NotImplementedError('cannot open absolute path %s' % repr(filename)) - else: - f = file(filename, mode+'b') - except IOError as e: - raise RedirectionError(str(e)) - - wrapper = None - try: - wrapper = FileWrapper(mode, f) - f = None - self._add_descriptor(io_number, wrapper) - except: - if f: f.close() - if wrapper: wrapper.close() - raise - - def _dup_output_descriptor(self, source_fd, dest_fd): - if source_fd is None: - source_fd = 1 - self._dup_file_descriptor(source_fd, dest_fd, 'w') - - def _dup_file_descriptor(self, source_fd, dest_fd, mode): - source_fd = int(source_fd) - if source_fd not in self._descriptors: - raise RedirectionError('"%s" is not a valid file descriptor' % str(source_fd)) - source = self._descriptors[source_fd] - - if source.mode()!=mode: - raise RedirectionError('Descriptor %s cannot be duplicated in mode "%s"' % (str(source), mode)) - - if dest_fd=='-': - # Close the source descriptor - del self._descriptors[source_fd] - source.close() - else: - dest_fd = int(dest_fd) - if dest_fd not in self._descriptors: - raise RedirectionError('Cannot replace file descriptor %s' % str(dest_fd)) - - dest = self._descriptors[dest_fd] - if dest.mode()!=mode: - raise RedirectionError('Descriptor %s cannot be cannot be redirected in mode "%s"' % (str(dest), mode)) - - self._descriptors[dest_fd] = source.dup() - dest.close() - - def _add_descriptor(self, io_number, file): - io_number = int(io_number) - - if io_number in self._descriptors: - # Close the current descriptor - d = self._descriptors[io_number] - del self._descriptors[io_number] - d.close() - - self._descriptors[io_number] = file - - def __str__(self): - names = [('%d=%r' % (k, getattr(v, 'name', None))) for k,v - in self._descriptors.iteritems()] - names = ','.join(names) - return 'Redirections(%s)' % names - - def __del__(self): - self.close() - -def cygwin_to_windows_path(path): - """Turn /cygdrive/c/foo into c:/foo, or return path if it - is not a cygwin path. - """ - if not path.startswith('/cygdrive/'): - return path - path = path[len('/cygdrive/'):] - path = path[:1] + ':' + path[1:] - return path - -def win32_to_unix_path(path): - if path is not None: - path = path.replace('\\', '/') - return path - -_RE_SHEBANG = re.compile(r'^\#!\s?([^\s]+)(?:\s([^\s]+))?') -_SHEBANG_CMDS = { - '/usr/bin/env': 'env', - '/bin/sh': 'pysh', - 'python': 'python', -} - -def resolve_shebang(path, ignoreshell=False): - """Return a list of arguments as shebang interpreter call or an empty list - if path does not refer to an executable script. - See . - - ignoreshell - set to True to ignore sh shebangs. Return an empty list instead. - """ - try: - f = file(path) - try: - # At most 80 characters in the first line - header = f.read(80).splitlines()[0] - finally: - f.close() - - m = _RE_SHEBANG.search(header) - if not m: - return [] - cmd, arg = m.group(1,2) - if os.path.isfile(cmd): - # Keep this one, the hg script for instance contains a weird windows - # shebang referencing the current python install. - cmdfile = os.path.basename(cmd).lower() - if cmdfile == 'python.exe': - cmd = 'python' - pass - elif cmd not in _SHEBANG_CMDS: - raise CommandNotFound('Unknown interpreter "%s" referenced in '\ - 'shebang' % header) - cmd = _SHEBANG_CMDS.get(cmd) - if cmd is None or (ignoreshell and cmd == 'pysh'): - return [] - if arg is None: - return [cmd, win32_to_unix_path(path)] - return [cmd, arg, win32_to_unix_path(path)] - except IOError as e: - if e.errno!=errno.ENOENT and \ - (e.errno!=errno.EPERM and not os.path.isdir(path)): # Opening a directory raises EPERM - raise - return [] - -def win32_find_in_path(name, path): - if isinstance(path, str): - path = path.split(os.pathsep) - - exts = os.environ.get('PATHEXT', '').lower().split(os.pathsep) - for p in path: - p_name = os.path.join(p, name) - - prefix = resolve_shebang(p_name) - if prefix: - return prefix - - for ext in exts: - p_name_ext = p_name + ext - if os.path.exists(p_name_ext): - return [win32_to_unix_path(p_name_ext)] - return [] - -class Traps(dict): - def __setitem__(self, key, value): - if key not in ('EXIT',): - raise NotImplementedError() - super(Traps, self).__setitem__(key, value) - -# IFS white spaces character class -_IFS_WHITESPACES = (' ', '\t', '\n') - -class Environment: - """Environment holds environment variables, export table, function - definitions and whatever is defined in 2.12 "Shell Execution Environment", - redirection excepted. - """ - def __init__(self, pwd): - self._opt = set() #Shell options - - self._functions = {} - self._env = {'?': '0', '#': '0'} - self._exported = set([ - 'HOME', 'IFS', 'PATH' - ]) - - # Set environment vars with side-effects - self._ifs_ws = None # Set of IFS whitespace characters - self._ifs_re = None # Regular expression used to split between words using IFS classes - self['IFS'] = ''.join(_IFS_WHITESPACES) #Default environment values - self['PWD'] = pwd - self.traps = Traps() - - def clone(self, subshell=False): - env = Environment(self['PWD']) - env._opt = set(self._opt) - for k,v in self.get_variables().iteritems(): - if k in self._exported: - env.export(k,v) - elif subshell: - env[k] = v - - if subshell: - env._functions = dict(self._functions) - - return env - - def __getitem__(self, key): - if key in ('@', '*', '-', '$'): - raise NotImplementedError('%s is not implemented' % repr(key)) - return self._env[key] - - def get(self, key, defval=None): - try: - return self[key] - except KeyError: - return defval - - def __setitem__(self, key, value): - if key=='IFS': - # Update the whitespace/non-whitespace classes - self._update_ifs(value) - elif key=='PWD': - pwd = os.path.abspath(value) - if not os.path.isdir(pwd): - raise VarAssignmentError('Invalid directory %s' % value) - value = pwd - elif key in ('?', '!'): - value = str(int(value)) - self._env[key] = value - - def __delitem__(self, key): - if key in ('IFS', 'PWD', '?'): - raise VarAssignmentError('%s cannot be unset' % key) - del self._env[key] - - def __contains__(self, item): - return item in self._env - - def set_positional_args(self, args): - """Set the content of 'args' as positional argument from 1 to len(args). - Return previous argument as a list of strings. - """ - # Save and remove previous arguments - prevargs = [] - for i in range(int(self._env['#'])): - i = str(i+1) - prevargs.append(self._env[i]) - del self._env[i] - self._env['#'] = '0' - - #Set new ones - for i,arg in enumerate(args): - self._env[str(i+1)] = str(arg) - self._env['#'] = str(len(args)) - - return prevargs - - def get_positional_args(self): - return [self._env[str(i+1)] for i in range(int(self._env['#']))] - - def get_variables(self): - return dict(self._env) - - def export(self, key, value=None): - if value is not None: - self[key] = value - self._exported.add(key) - - def get_exported(self): - return [(k,self._env.get(k)) for k in self._exported] - - def split_fields(self, word): - if not self._ifs_ws or not word: - return [word] - return re.split(self._ifs_re, word) - - def _update_ifs(self, value): - """Update the split_fields related variables when IFS character set is - changed. - """ - # TODO: handle NULL IFS - - # Separate characters in whitespace and non-whitespace - chars = set(value) - ws = [c for c in chars if c in _IFS_WHITESPACES] - nws = [c for c in chars if c not in _IFS_WHITESPACES] - - # Keep whitespaces in a string for left and right stripping - self._ifs_ws = ''.join(ws) - - # Build a regexp to split fields - trailing = '[' + ''.join([re.escape(c) for c in ws]) + ']' - if nws: - # First, the single non-whitespace occurence. - nws = '[' + ''.join([re.escape(c) for c in nws]) + ']' - nws = '(?:' + trailing + '*' + nws + trailing + '*' + '|' + trailing + '+)' - else: - # Then mix all parts with quantifiers - nws = trailing + '+' - self._ifs_re = re.compile(nws) - - def has_opt(self, opt, val=None): - return (opt, val) in self._opt - - def set_opt(self, opt, val=None): - self._opt.add((opt, val)) - - def find_in_path(self, name, pwd=False): - path = self._env.get('PATH', '').split(os.pathsep) - if pwd: - path[:0] = [self['PWD']] - if os.name == 'nt': - return win32_find_in_path(name, self._env.get('PATH', '')) - else: - raise NotImplementedError() - - def define_function(self, name, body): - if not is_name(name): - raise ShellSyntaxError('%s is not a valid function name' % repr(name)) - self._functions[name] = body - - def remove_function(self, name): - del self._functions[name] - - def is_function(self, name): - return name in self._functions - - def get_function(self, name): - return self._functions.get(name) - - -name_charset = 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_' -name_charset = dict(zip(name_charset,name_charset)) - -def match_name(s): - """Return the length in characters of the longest prefix made of name - allowed characters in s. - """ - for i,c in enumerate(s): - if c not in name_charset: - return s[:i] - return s - -def is_name(s): - return len([c for c in s if c not in name_charset])<=0 - -def is_special_param(c): - return len(c)==1 and c in ('@','*','#','?','-','$','!','0') - -def utility_not_implemented(name, *args, **kwargs): - raise NotImplementedError('%s utility is not implemented' % name) - - -class Utility: - """Define utilities properties: - func -- utility callable. See builtin module for utility samples. - is_special -- see XCU 2.8. - """ - def __init__(self, func, is_special=0): - self.func = func - self.is_special = bool(is_special) - - -def encodeargs(args): - def encodearg(s): - lines = base64.encodestring(s) - lines = [l.splitlines()[0] for l in lines] - return ''.join(lines) - - s = pickle.dumps(args) - return encodearg(s) - -def decodeargs(s): - s = base64.decodestring(s) - return pickle.loads(s) - - -class GlobError(Exception): - pass - -class Options: - def __init__(self): - # True if Mercurial operates with binary streams - self.hgbinary = True - -class Interpreter: - # Implementation is very basic: the execute() method just makes a DFS on the - # AST and execute nodes one by one. Nodes are tuple (name,obj) where name - # is a string identifier and obj the AST element returned by the parser. - # - # Handler are named after the node identifiers. - # TODO: check node names and remove the switch in execute with some - # dynamic getattr() call to find node handlers. - """Shell interpreter. - - The following debugging flags can be passed: - debug-parsing - enable PLY debugging. - debug-tree - print the generated AST. - debug-cmd - trace command execution before word expansion, plus exit status. - debug-utility - trace utility execution. - """ - - # List supported commands. - COMMANDS = { - 'cat': Utility(builtin.utility_cat,), - 'cd': Utility(builtin.utility_cd,), - ':': Utility(builtin.utility_colon,), - 'echo': Utility(builtin.utility_echo), - 'env': Utility(builtin.utility_env), - 'exit': Utility(builtin.utility_exit), - 'export': Utility(builtin.builtin_export, is_special=1), - 'egrep': Utility(builtin.utility_egrep), - 'fgrep': Utility(builtin.utility_fgrep), - 'gunzip': Utility(builtin.utility_gunzip), - 'kill': Utility(builtin.utility_kill), - 'mkdir': Utility(builtin.utility_mkdir), - 'netstat': Utility(builtin.utility_netstat), - 'printf': Utility(builtin.utility_printf), - 'pwd': Utility(builtin.utility_pwd), - 'return': Utility(builtin.builtin_return, is_special=1), - 'sed': Utility(builtin.utility_sed,), - 'set': Utility(builtin.builtin_set,), - 'shift': Utility(builtin.builtin_shift,), - 'sleep': Utility(builtin.utility_sleep,), - 'sort': Utility(builtin.utility_sort,), - 'trap': Utility(builtin.builtin_trap, is_special=1), - 'true': Utility(builtin.utility_true), - 'unset': Utility(builtin.builtin_unset, is_special=1), - 'wait': Utility(builtin.builtin_wait, is_special=1), - } - - def __init__(self, pwd, debugflags = [], env=None, redirs=None, stdin=None, - stdout=None, stderr=None, opts=Options()): - self._env = env - if self._env is None: - self._env = Environment(pwd) - self._children = {} - - self._redirs = redirs - self._close_redirs = False - - if self._redirs is None: - if stdin is None: - stdin = sys.stdin - if stdout is None: - stdout = sys.stdout - if stderr is None: - stderr = sys.stderr - stdin = FileWrapper('r', stdin, False) - stdout = FileWrapper('w', stdout, False) - stderr = FileWrapper('w', stderr, False) - self._redirs = Redirections(stdin, stdout, stderr) - self._close_redirs = True - - self._debugflags = list(debugflags) - self._logfile = sys.stderr - self._options = opts - - def close(self): - """Must be called when the interpreter is no longer used.""" - script = self._env.traps.get('EXIT') - if script: - try: - self.execute_script(script=script) - except: - pass - - if self._redirs is not None and self._close_redirs: - self._redirs.close() - self._redirs = None - - def log(self, s): - self._logfile.write(s) - self._logfile.flush() - - def __getitem__(self, key): - return self._env[key] - - def __setitem__(self, key, value): - self._env[key] = value - - def options(self): - return self._options - - def redirect(self, redirs, ios): - def add_redir(io): - if isinstance(io, pyshyacc.IORedirect): - redirs.add(self, io.op, io.filename, io.io_number) - else: - redirs.add_here_document(self, io.name, io.content, io.io_number) - - map(add_redir, ios) - return redirs - - def execute_script(self, script=None, ast=None, sourced=False, - scriptpath=None): - """If script is not None, parse the input. Otherwise takes the supplied - AST. Then execute the AST. - Return the script exit status. - """ - try: - if scriptpath is not None: - self._env['0'] = os.path.abspath(scriptpath) - - if script is not None: - debug_parsing = ('debug-parsing' in self._debugflags) - cmds, script = pyshyacc.parse(script, True, debug_parsing) - if 'debug-tree' in self._debugflags: - pyshyacc.print_commands(cmds, self._logfile) - self._logfile.flush() - else: - cmds, script = ast, '' - - status = 0 - for cmd in cmds: - try: - status = self.execute(cmd) - except ExitSignal as e: - if sourced: - raise - status = int(e.args[0]) - return status - except ShellError: - self._env['?'] = 1 - raise - if 'debug-utility' in self._debugflags or 'debug-cmd' in self._debugflags: - self.log('returncode ' + str(status)+ '\n') - return status - except CommandNotFound as e: - print >>self._redirs.stderr, str(e) - self._redirs.stderr.flush() - # Command not found by non-interactive shell - # return 127 - raise - except RedirectionError as e: - # TODO: should be handled depending on the utility status - print >>self._redirs.stderr, str(e) - self._redirs.stderr.flush() - # Command not found by non-interactive shell - # return 127 - raise - - def dotcommand(self, env, args): - if len(args) < 1: - raise ShellError('. expects at least one argument') - path = args[0] - if '/' not in path: - found = env.find_in_path(args[0], True) - if found: - path = found[0] - script = file(path).read() - return self.execute_script(script=script, sourced=True) - - def execute(self, token, redirs=None): - """Execute and AST subtree with supplied redirections overriding default - interpreter ones. - Return the exit status. - """ - if not token: - return 0 - - if redirs is None: - redirs = self._redirs - - if isinstance(token, list): - # Commands sequence - res = 0 - for t in token: - res = self.execute(t, redirs) - return res - - type, value = token - status = 0 - if type=='simple_command': - redirs_copy = redirs.clone() - try: - # TODO: define and handle command return values - # TODO: implement set -e - status = self._execute_simple_command(value, redirs_copy) - finally: - redirs_copy.close() - elif type=='pipeline': - status = self._execute_pipeline(value, redirs) - elif type=='and_or': - status = self._execute_and_or(value, redirs) - elif type=='for_clause': - status = self._execute_for_clause(value, redirs) - elif type=='while_clause': - status = self._execute_while_clause(value, redirs) - elif type=='function_definition': - status = self._execute_function_definition(value, redirs) - elif type=='brace_group': - status = self._execute_brace_group(value, redirs) - elif type=='if_clause': - status = self._execute_if_clause(value, redirs) - elif type=='subshell': - status = self.subshell(ast=value.cmds, redirs=redirs) - elif type=='async': - status = self._asynclist(value) - elif type=='redirect_list': - redirs_copy = self.redirect(redirs.clone(), value.redirs) - try: - status = self.execute(value.cmd, redirs_copy) - finally: - redirs_copy.close() - else: - raise NotImplementedError('Unsupported token type ' + type) - - if status < 0: - status = 255 - return status - - def _execute_if_clause(self, if_clause, redirs): - cond_status = self.execute(if_clause.cond, redirs) - if cond_status==0: - return self.execute(if_clause.if_cmds, redirs) - else: - return self.execute(if_clause.else_cmds, redirs) - - def _execute_brace_group(self, group, redirs): - status = 0 - for cmd in group.cmds: - status = self.execute(cmd, redirs) - return status - - def _execute_function_definition(self, fundef, redirs): - self._env.define_function(fundef.name, fundef.body) - return 0 - - def _execute_while_clause(self, while_clause, redirs): - status = 0 - while 1: - cond_status = 0 - for cond in while_clause.condition: - cond_status = self.execute(cond, redirs) - - if cond_status: - break - - for cmd in while_clause.cmds: - status = self.execute(cmd, redirs) - - return status - - def _execute_for_clause(self, for_clause, redirs): - if not is_name(for_clause.name): - raise ShellSyntaxError('%s is not a valid name' % repr(for_clause.name)) - items = mappend(self.expand_token, for_clause.items) - - status = 0 - for item in items: - self._env[for_clause.name] = item - for cmd in for_clause.cmds: - status = self.execute(cmd, redirs) - return status - - def _execute_and_or(self, or_and, redirs): - res = self.execute(or_and.left, redirs) - if (or_and.op=='&&' and res==0) or (or_and.op!='&&' and res!=0): - res = self.execute(or_and.right, redirs) - return res - - def _execute_pipeline(self, pipeline, redirs): - if len(pipeline.commands)==1: - status = self.execute(pipeline.commands[0], redirs) - else: - # Execute all commands one after the other - status = 0 - inpath, outpath = None, None - try: - # Commands inputs and outputs cannot really be plugged as done - # by a real shell. Run commands sequentially and chain their - # input/output throught temporary files. - tmpfd, inpath = tempfile.mkstemp() - os.close(tmpfd) - tmpfd, outpath = tempfile.mkstemp() - os.close(tmpfd) - - inpath = win32_to_unix_path(inpath) - outpath = win32_to_unix_path(outpath) - - for i, cmd in enumerate(pipeline.commands): - call_redirs = redirs.clone() - try: - if i!=0: - call_redirs.add(self, '<', inpath) - if i!=len(pipeline.commands)-1: - call_redirs.add(self, '>', outpath) - - status = self.execute(cmd, call_redirs) - - # Chain inputs/outputs - inpath, outpath = outpath, inpath - finally: - call_redirs.close() - finally: - if inpath: os.remove(inpath) - if outpath: os.remove(outpath) - - if pipeline.reverse_status: - status = int(not status) - self._env['?'] = status - return status - - def _execute_function(self, name, args, interp, env, stdin, stdout, stderr, *others): - assert interp is self - - func = env.get_function(name) - #Set positional parameters - prevargs = None - try: - prevargs = env.set_positional_args(args) - try: - redirs = Redirections(stdin.dup(), stdout.dup(), stderr.dup()) - try: - status = self.execute(func, redirs) - finally: - redirs.close() - except ReturnSignal as e: - status = int(e.args[0]) - env['?'] = status - return status - finally: - #Reset positional parameters - if prevargs is not None: - env.set_positional_args(prevargs) - - def _execute_simple_command(self, token, redirs): - """Can raise ReturnSignal when return builtin is called, ExitSignal when - exit is called, and other shell exceptions upon builtin failures. - """ - debug_command = 'debug-cmd' in self._debugflags - if debug_command: - self.log('word' + repr(token.words) + '\n') - self.log('assigns' + repr(token.assigns) + '\n') - self.log('redirs' + repr(token.redirs) + '\n') - - is_special = None - env = self._env - - try: - # Word expansion - args = [] - for word in token.words: - args += self.expand_token(word) - if is_special is None and args: - is_special = env.is_function(args[0]) or \ - (args[0] in self.COMMANDS and self.COMMANDS[args[0]].is_special) - - if debug_command: - self.log('_execute_simple_command' + str(args) + '\n') - - if not args: - # Redirections happen is a subshell - redirs = redirs.clone() - elif not is_special: - env = self._env.clone() - - # Redirections - self.redirect(redirs, token.redirs) - - # Variables assignments - res = 0 - for type,(k,v) in token.assigns: - status, expanded = self.expand_variable((k,v)) - if status is not None: - res = status - if args: - env.export(k, expanded) - else: - env[k] = expanded - - if args and args[0] in ('.', 'source'): - res = self.dotcommand(env, args[1:]) - elif args: - if args[0] in self.COMMANDS: - command = self.COMMANDS[args[0]] - elif env.is_function(args[0]): - command = Utility(self._execute_function, is_special=True) - else: - if not '/' in args[0].replace('\\', '/'): - cmd = env.find_in_path(args[0]) - if not cmd: - # TODO: test error code on unknown command => 127 - raise CommandNotFound('Unknown command: "%s"' % args[0]) - else: - # Handle commands like '/cygdrive/c/foo.bat' - cmd = cygwin_to_windows_path(args[0]) - if not os.path.exists(cmd): - raise CommandNotFound('%s: No such file or directory' % args[0]) - shebang = resolve_shebang(cmd) - if shebang: - cmd = shebang - else: - cmd = [cmd] - args[0:1] = cmd - command = Utility(builtin.run_command) - - # Command execution - if 'debug-cmd' in self._debugflags: - self.log('redirections ' + str(redirs) + '\n') - - res = command.func(args[0], args[1:], self, env, - redirs.stdin(), redirs.stdout(), - redirs.stderr(), self._debugflags) - - if self._env.has_opt('-x'): - # Trace command execution in shell environment - # BUG: would be hard to reproduce a real shell behaviour since - # the AST is not annotated with source lines/tokens. - self._redirs.stdout().write(' '.join(args)) - - except ReturnSignal: - raise - except ShellError as e: - if is_special or isinstance(e, (ExitSignal, - ShellSyntaxError, ExpansionError)): - raise e - self._redirs.stderr().write(str(e)+'\n') - return 1 - - return res - - def expand_token(self, word): - """Expand a word as specified in [2.6 Word Expansions]. Return the list - of expanded words. - """ - status, wtrees = self._expand_word(word) - return map(pyshlex.wordtree_as_string, wtrees) - - def expand_variable(self, word): - """Return a status code (or None if no command expansion occurred) - and a single word. - """ - status, wtrees = self._expand_word(word, pathname=False, split=False) - words = map(pyshlex.wordtree_as_string, wtrees) - assert len(words)==1 - return status, words[0] - - def expand_here_document(self, word): - """Return the expanded document as a single word. The here document is - assumed to be unquoted. - """ - status, wtrees = self._expand_word(word, pathname=False, - split=False, here_document=True) - words = map(pyshlex.wordtree_as_string, wtrees) - assert len(words)==1 - return words[0] - - def expand_redirection(self, word): - """Return a single word.""" - return self.expand_variable(word)[1] - - def get_env(self): - return self._env - - def _expand_word(self, token, pathname=True, split=True, here_document=False): - wtree = pyshlex.make_wordtree(token[1], here_document=here_document) - - # TODO: implement tilde expansion - def expand(wtree): - """Return a pseudo wordtree: the tree or its subelements can be empty - lists when no value result from the expansion. - """ - status = None - for part in wtree: - if not isinstance(part, list): - continue - if part[0]in ("'", '\\'): - continue - elif part[0] in ('`', '$('): - status, result = self._expand_command(part) - part[:] = result - elif part[0] in ('$', '${'): - part[:] = self._expand_parameter(part, wtree[0]=='"', split) - elif part[0] in ('', '"'): - status, result = expand(part) - part[:] = result - else: - raise NotImplementedError('%s expansion is not implemented' - % part[0]) - # [] is returned when an expansion result in no-field, - # like an empty $@ - wtree = [p for p in wtree if p != []] - if len(wtree) < 3: - return status, [] - return status, wtree - - status, wtree = expand(wtree) - if len(wtree) == 0: - return status, wtree - wtree = pyshlex.normalize_wordtree(wtree) - - if split: - wtrees = self._split_fields(wtree) - else: - wtrees = [wtree] - - if pathname: - wtrees = mappend(self._expand_pathname, wtrees) - - wtrees = map(self._remove_quotes, wtrees) - return status, wtrees - - def _expand_command(self, wtree): - # BUG: there is something to do with backslashes and quoted - # characters here - command = pyshlex.wordtree_as_string(wtree[1:-1]) - status, output = self.subshell_output(command) - return status, ['', output, ''] - - def _expand_parameter(self, wtree, quoted=False, split=False): - """Return a valid wtree or an empty list when no parameter results.""" - # Get the parameter name - # TODO: implement weird expansion rules with ':' - name = pyshlex.wordtree_as_string(wtree[1:-1]) - if not is_name(name) and not is_special_param(name): - raise ExpansionError('Bad substitution "%s"' % name) - # TODO: implement special parameters - if name in ('@', '*'): - args = self._env.get_positional_args() - if len(args) == 0: - return [] - if len(args)<2: - return ['', ''.join(args), ''] - - sep = self._env.get('IFS', '')[:1] - if split and quoted and name=='@': - # Introduce a new token to tell the caller that these parameters - # cause a split as specified in 2.5.2 - return ['@'] + args + [''] - else: - return ['', sep.join(args), ''] - - return ['', self._env.get(name, ''), ''] - - def _split_fields(self, wtree): - def is_empty(split): - return split==['', '', ''] - - def split_positional(quoted): - # Return a list of wtree split according positional parameters rules. - # All remaining '@' groups are removed. - assert quoted[0]=='"' - - splits = [[]] - for part in quoted: - if not isinstance(part, list) or part[0]!='@': - splits[-1].append(part) - else: - # Empty or single argument list were dealt with already - assert len(part)>3 - # First argument must join with the beginning part of the original word - splits[-1].append(part[1]) - # Create double-quotes expressions for every argument after the first - for arg in part[2:-1]: - splits[-1].append('"') - splits.append(['"', arg]) - return splits - - # At this point, all expansions but pathnames have occured. Only quoted - # and positional sequences remain. Thus, all candidates for field splitting - # are in the tree root, or are positional splits ('@') and lie in root - # children. - if not wtree or wtree[0] not in ('', '"'): - # The whole token is quoted or empty, nothing to split - return [wtree] - - if wtree[0]=='"': - wtree = ['', wtree, ''] - - result = [['', '']] - for part in wtree[1:-1]: - if isinstance(part, list): - if part[0]=='"': - splits = split_positional(part) - if len(splits)<=1: - result[-1] += [part, ''] - else: - # Terminate the current split - result[-1] += [splits[0], ''] - result += splits[1:-1] - # Create a new split - result += [['', splits[-1], '']] - else: - result[-1] += [part, ''] - else: - splits = self._env.split_fields(part) - if len(splits)<=1: - # No split - result[-1][-1] += part - else: - # Terminate the current resulting part and create a new one - result[-1][-1] += splits[0] - result[-1].append('') - result += [['', r, ''] for r in splits[1:-1]] - result += [['', splits[-1]]] - result[-1].append('') - - # Leading and trailing empty groups come from leading/trailing blanks - if result and is_empty(result[-1]): - result[-1:] = [] - if result and is_empty(result[0]): - result[:1] = [] - return result - - def _expand_pathname(self, wtree): - """See [2.6.6 Pathname Expansion].""" - if self._env.has_opt('-f'): - return [wtree] - - # All expansions have been performed, only quoted sequences should remain - # in the tree. Generate the pattern by folding the tree, escaping special - # characters when appear quoted - special_chars = '*?[]' - - def make_pattern(wtree): - subpattern = [] - for part in wtree[1:-1]: - if isinstance(part, list): - part = make_pattern(part) - elif wtree[0]!='': - for c in part: - # Meta-characters cannot be quoted - if c in special_chars: - raise GlobError() - subpattern.append(part) - return ''.join(subpattern) - - def pwd_glob(pattern): - cwd = os.getcwd() - os.chdir(self._env['PWD']) - try: - return glob.glob(pattern) - finally: - os.chdir(cwd) - - #TODO: check working directory issues here wrt relative patterns - try: - pattern = make_pattern(wtree) - paths = pwd_glob(pattern) - except GlobError: - # BUG: Meta-characters were found in quoted sequences. The should - # have been used literally but this is unsupported in current glob module. - # Instead we consider the whole tree must be used literally and - # therefore there is no point in globbing. This is wrong when meta - # characters are mixed with quoted meta in the same pattern like: - # < foo*"py*" > - paths = [] - - if not paths: - return [wtree] - return [['', path, ''] for path in paths] - - def _remove_quotes(self, wtree): - """See [2.6.7 Quote Removal].""" - - def unquote(wtree): - unquoted = [] - for part in wtree[1:-1]: - if isinstance(part, list): - part = unquote(part) - unquoted.append(part) - return ''.join(unquoted) - - return ['', unquote(wtree), ''] - - def subshell(self, script=None, ast=None, redirs=None): - """Execute the script or AST in a subshell, with inherited redirections - if redirs is not None. - """ - if redirs: - sub_redirs = redirs - else: - sub_redirs = redirs.clone() - - subshell = None - try: - subshell = Interpreter(None, self._debugflags, self._env.clone(True), - sub_redirs, opts=self._options) - return subshell.execute_script(script, ast) - finally: - if not redirs: sub_redirs.close() - if subshell: subshell.close() - - def subshell_output(self, script): - """Execute the script in a subshell and return the captured output.""" - # Create temporary file to capture subshell output - tmpfd, tmppath = tempfile.mkstemp() - try: - tmpfile = os.fdopen(tmpfd, 'wb') - stdout = FileWrapper('w', tmpfile) - - redirs = Redirections(self._redirs.stdin().dup(), - stdout, - self._redirs.stderr().dup()) - try: - status = self.subshell(script=script, redirs=redirs) - finally: - redirs.close() - redirs = None - - # Extract subshell standard output - tmpfile = open(tmppath, 'rb') - try: - output = tmpfile.read() - return status, output.rstrip('\n') - finally: - tmpfile.close() - finally: - os.remove(tmppath) - - def _asynclist(self, cmd): - args = (self._env.get_variables(), cmd) - arg = encodeargs(args) - assert len(args) < 30*1024 - cmd = ['pysh.bat', '--ast', '-c', arg] - p = subprocess.Popen(cmd, cwd=self._env['PWD']) - self._children[p.pid] = p - self._env['!'] = p.pid - return 0 - - def wait(self, pids=None): - if not pids: - pids = self._children.keys() - - status = 127 - for pid in pids: - if pid not in self._children: - continue - p = self._children.pop(pid) - status = p.wait() - - return status - diff --git a/bitbake/lib/bb/pysh/lsprof.py b/bitbake/lib/bb/pysh/lsprof.py deleted file mode 100644 index b1831c2..0000000 --- a/bitbake/lib/bb/pysh/lsprof.py +++ /dev/null @@ -1,116 +0,0 @@ -#! /usr/bin/env python - -import sys -from _lsprof import Profiler, profiler_entry - -__all__ = ['profile', 'Stats'] - -def profile(f, *args, **kwds): - """XXX docstring""" - p = Profiler() - p.enable(subcalls=True, builtins=True) - try: - f(*args, **kwds) - finally: - p.disable() - return Stats(p.getstats()) - - -class Stats(object): - """XXX docstring""" - - def __init__(self, data): - self.data = data - - def sort(self, crit="inlinetime"): - """XXX docstring""" - if crit not in profiler_entry.__dict__: - raise ValueError("Can't sort by %s" % crit) - self.data.sort(lambda b, a: cmp(getattr(a, crit), - getattr(b, crit))) - for e in self.data: - if e.calls: - e.calls.sort(lambda b, a: cmp(getattr(a, crit), - getattr(b, crit))) - - def pprint(self, top=None, file=None, limit=None, climit=None): - """XXX docstring""" - if file is None: - file = sys.stdout - d = self.data - if top is not None: - d = d[:top] - cols = "% 12s %12s %11.4f %11.4f %s\n" - hcols = "% 12s %12s %12s %12s %s\n" - cols2 = "+%12s %12s %11.4f %11.4f + %s\n" - file.write(hcols % ("CallCount", "Recursive", "Total(ms)", - "Inline(ms)", "module:lineno(function)")) - count = 0 - for e in d: - file.write(cols % (e.callcount, e.reccallcount, e.totaltime, - e.inlinetime, label(e.code))) - count += 1 - if limit is not None and count == limit: - return - ccount = 0 - if e.calls: - for se in e.calls: - file.write(cols % ("+%s" % se.callcount, se.reccallcount, - se.totaltime, se.inlinetime, - "+%s" % label(se.code))) - count += 1 - ccount += 1 - if limit is not None and count == limit: - return - if climit is not None and ccount == climit: - break - - def freeze(self): - """Replace all references to code objects with string - descriptions; this makes it possible to pickle the instance.""" - - # this code is probably rather ickier than it needs to be! - for i in range(len(self.data)): - e = self.data[i] - if not isinstance(e.code, str): - self.data[i] = type(e)((label(e.code),) + e[1:]) - if e.calls: - for j in range(len(e.calls)): - se = e.calls[j] - if not isinstance(se.code, str): - e.calls[j] = type(se)((label(se.code),) + se[1:]) - -_fn2mod = {} - -def label(code): - if isinstance(code, str): - return code - try: - mname = _fn2mod[code.co_filename] - except KeyError: - for k, v in sys.modules.items(): - if v is None: - continue - if not hasattr(v, '__file__'): - continue - if not isinstance(v.__file__, str): - continue - if v.__file__.startswith(code.co_filename): - mname = _fn2mod[code.co_filename] = k - break - else: - mname = _fn2mod[code.co_filename] = '<%s>'%code.co_filename - - return '%s:%d(%s)' % (mname, code.co_firstlineno, code.co_name) - - -if __name__ == '__main__': - import os - sys.argv = sys.argv[1:] - if not sys.argv: - print >> sys.stderr, "usage: lsprof.py