From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7059705400961007616 X-Received: by 2002:a05:600c:3494:: with SMTP id a20mr1433158wmq.129.1643715752335; Tue, 01 Feb 2022 03:42:32 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a1c:a78a:: with SMTP id q132ls1105613wme.0.gmail; Tue, 01 Feb 2022 03:42:30 -0800 (PST) X-Google-Smtp-Source: ABdhPJxcV+K7fnk49WiD9oIuOwBGWh8Y/cTx/FC6w6jVVnwUXfmvcC7AcD+i0KpO7iGRmt7JnIE9 X-Received: by 2002:adf:a44b:: with SMTP id e11mr22202558wra.227.1643715749314; Tue, 01 Feb 2022 03:42:29 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1643715749; cv=none; d=google.com; s=arc-20160816; b=Xxkk2DlTnZHuB/60YnGYXNapnbIb6WVT80VFX8eKRX+a6NjBKYU1OQMMMKyATwMUot MtHUT3sT90Ni8LvKfz2bty8mEc0WAdZ5aLF4mC3oVr2W3qepfVD36+abkKwA1il+jN/V strhJ/LsdfMbIS/12bTEu5EZE688pi/hwuDUz9w3NHw7LCG525U90/SFFgmulJXcyw8P jYZlDQvr0EF5xZiutLUFW6oS9uXC3oNk67dy8LO3gASjA/l1F/R+zgE8exLsetMPtHlR Se8vvzFcgkhWO3IN3C5Mp1lKapLC5ARygky2oPdocGcWWn5K0n1ZhaVKNK6blHybK4lr QyTA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from; bh=kNNRtSnKp6dwu3slTjyTmS41Fv9QQArRds/if/dW8uY=; b=PBbiY2bA5QjDBRAuuYb2sMmLyRFOg+Zy4029LpE4uaMQ6NSkuNVeSXYIkt0Yg8WsOd mJWUW+knXmfGllEIduchX/9IYkpYZlDFc5F4HNvGm4h0LIjNtreb7wczgcION1rtGNbd rz1V8qzvkwFZ3fy3pGgJEuYwSIapM8Lvr2ioilnYnbVICJvXdfckMKGyeLFUroRSh366 RaGIx5MkHPWWrkdlYIPdgo9B/BiUnbj4qLt4WkotRrgAZiCLt7FIpZnCl425UwWhA/Uu ff2KpKndDoMt1IBrCBj6/4IKyZPQC0/FQjetr/C3f0/lsoWSie3L6HY/mfCx0vkXnXDP paZQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id h16si143130wml.0.2022.02.01.03.42.28 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Tue, 01 Feb 2022 03:42:29 -0800 (PST) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from baighyz.m.ilbers.de (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPSA id 211BgIVp018237 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 1 Feb 2022 12:42:23 +0100 From: Uladzimir Bely To: isar-users@googlegroups.com Subject: [PATCH 1/1] bitbake: Update to 1.50.4 release Date: Tue, 1 Feb 2022 12:42:18 +0100 Message-Id: <20220201114218.24817-2-ubely@ilbers.de> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20220201114218.24817-1-ubely@ilbers.de> References: <20220201114218.24817-1-ubely@ilbers.de> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-TUID: G+IT3TRZSPuS Update bitbake to the latest release in 1.50 branch. This release is tagged to the commit ID 0fe1a9e2d2e33f80d807cefc7a23df4a5f760c74 in the bitbake upstream. The version in bin/bitbake is shown as 1.50.0. Signed-off-by: Uladzimir Bely --- bitbake/.gitignore | 1 + bitbake/README | 2 +- bitbake/bin/bitbake | 2 +- bitbake/bin/bitbake-hashclient | 3 - bitbake/bin/bitbake-hashserv | 10 +- bitbake/bin/bitbake-layers | 2 - bitbake/bin/bitbake-selftest | 2 + bitbake/bin/bitbake-server | 54 + bitbake/bin/bitbake-worker | 44 +- bitbake/bin/bitdoc | 519 --- bitbake/contrib/bbparse-torture.py | 89 + bitbake/contrib/hashserv/Dockerfile | 19 + bitbake/contrib/vim/LICENSE.txt | 18 + bitbake/contrib/vim/ftdetect/bitbake.vim | 4 +- bitbake/contrib/vim/ftplugin/bitbake.vim | 15 +- bitbake/contrib/vim/plugin/newbb.vim | 14 +- bitbake/contrib/vim/plugin/newbbappend.vim | 46 + bitbake/contrib/vim/syntax/bitbake.vim | 16 +- bitbake/doc/.gitignore | 1 + bitbake/doc/Makefile | 108 +- bitbake/doc/README | 50 +- bitbake/doc/_templates/breadcrumbs.html | 14 + bitbake/doc/_templates/layout.html | 7 + .../bitbake-user-manual-customization.xsl | 29 - .../bitbake-user-manual-execution.rst | 734 +++++ .../bitbake-user-manual-execution.xml | 1029 ------ .../bitbake-user-manual-fetching.rst | 689 ++++ .../bitbake-user-manual-fetching.xml | 868 ----- .../bitbake-user-manual-hello.rst | 415 +++ .../bitbake-user-manual-hello.xml | 513 --- .../bitbake-user-manual-intro.rst | 651 ++++ .../bitbake-user-manual-intro.xml | 891 ----- .../bitbake-user-manual-metadata.rst | 1980 ++++++++++++ .../bitbake-user-manual-metadata.xml | 2862 ----------------- .../bitbake-user-manual-ref-variables.rst | 1405 ++++++++ .../bitbake-user-manual-ref-variables.xml | 2476 -------------- .../bitbake-user-manual-style.css | 984 ------ .../bitbake-user-manual.xml | 88 - bitbake/doc/bitbake-user-manual/html.css | 281 -- bitbake/doc/conf.py | 101 + bitbake/doc/genindex.rst | 3 + bitbake/doc/index.rst | 38 + bitbake/doc/poky.ent | 51 - bitbake/doc/releases.rst | 130 + bitbake/doc/sphinx-static/switchers.js | 233 ++ bitbake/doc/sphinx-static/theme_overrides.css | 162 + bitbake/doc/template/Vera.xml | 1 - bitbake/doc/template/VeraMoBd.xml | 1 - bitbake/doc/template/VeraMono.xml | 1 - bitbake/doc/template/component.title.xsl | 39 - bitbake/doc/template/db-pdf.xsl | 64 - bitbake/doc/template/division.title.xsl | 25 - bitbake/doc/template/fop-config.xml | 58 - .../doc/template/formal.object.heading.xsl | 21 - bitbake/doc/template/gloss-permalinks.xsl | 14 - bitbake/doc/template/permalinks.xsl | 25 - bitbake/doc/template/section.title.xsl | 55 - bitbake/doc/template/titlepage.templates.xml | 1259 -------- bitbake/doc/tools/docbook-to-pdf | 51 - bitbake/lib/bb/COW.py | 150 +- bitbake/lib/bb/__init__.py | 71 +- bitbake/lib/bb/build.py | 174 +- bitbake/lib/bb/cache.py | 249 +- bitbake/lib/bb/codeparser.py | 6 +- bitbake/lib/bb/command.py | 71 +- bitbake/lib/bb/compat.py | 10 - bitbake/lib/bb/cooker.py | 433 ++- bitbake/lib/bb/cookerdata.py | 74 +- bitbake/lib/bb/daemonize.py | 2 + bitbake/lib/bb/data.py | 6 + bitbake/lib/bb/data_smart.py | 20 +- bitbake/lib/bb/event.py | 61 +- bitbake/lib/bb/fetch2/__init__.py | 99 +- bitbake/lib/bb/fetch2/az.py | 93 + bitbake/lib/bb/fetch2/bzr.py | 8 +- bitbake/lib/bb/fetch2/clearcase.py | 2 +- bitbake/lib/bb/fetch2/cvs.py | 24 +- bitbake/lib/bb/fetch2/git.py | 106 +- bitbake/lib/bb/fetch2/gitsm.py | 68 +- bitbake/lib/bb/fetch2/hg.py | 16 +- bitbake/lib/bb/fetch2/local.py | 19 +- bitbake/lib/bb/fetch2/npmsw.py | 4 + bitbake/lib/bb/fetch2/osc.py | 9 +- bitbake/lib/bb/fetch2/perforce.py | 94 +- bitbake/lib/bb/fetch2/repo.py | 2 +- bitbake/lib/bb/fetch2/ssh.py | 7 +- bitbake/lib/bb/fetch2/svn.py | 8 +- bitbake/lib/bb/fetch2/wget.py | 32 +- bitbake/lib/bb/main.py | 355 +- bitbake/lib/bb/monitordisk.py | 4 +- bitbake/lib/bb/msg.py | 9 +- bitbake/lib/bb/namedtuple_with_abc.py | 14 +- bitbake/lib/bb/parse/__init__.py | 2 +- bitbake/lib/bb/parse/ast.py | 21 +- bitbake/lib/bb/parse/parse_py/BBHandler.py | 15 +- bitbake/lib/bb/parse/parse_py/ConfHandler.py | 4 +- bitbake/lib/bb/persist_data.py | 10 +- bitbake/lib/bb/process.py | 7 +- bitbake/lib/bb/progress.py | 60 +- bitbake/lib/bb/providers.py | 108 +- bitbake/lib/bb/runqueue.py | 337 +- bitbake/lib/bb/server/process.py | 273 +- bitbake/lib/bb/siggen.py | 125 +- bitbake/lib/bb/taskdata.py | 51 +- bitbake/lib/bb/tests/codeparser.py | 4 +- bitbake/lib/bb/tests/color.py | 95 + bitbake/lib/bb/tests/cooker.py | 2 +- bitbake/lib/bb/tests/cow.py | 218 +- bitbake/lib/bb/tests/data.py | 1 + bitbake/lib/bb/tests/event.py | 17 +- .../linux/utils/util-linux/v2.23/index.html | 90 +- .../linux/utils/util-linux/v2.24/index.html | 86 +- .../linux/utils/util-linux/v2.25/index.html | 92 +- .../linux/utils/util-linux/v2.26/index.html | 84 +- .../linux/utils/util-linux/v2.27/index.html | 70 +- .../linux/utils/util-linux/v2.28/index.html | 84 +- .../linux/utils/util-linux/v2.29/index.html | 84 +- .../linux/utils/util-linux/v2.30/index.html | 84 +- .../linux/utils/util-linux/v2.31/index.html | 70 +- .../linux/utils/util-linux/v2.32/index.html | 70 +- .../linux/utils/util-linux/v2.33/index.html | 84 +- .../linux/utils/util-linux/v2.34/index.html | 56 +- .../linux/utils/util-linux/v2.35/index.html | 36 +- .../fetch-testdata/releases/eglibc/index.html | 42 +- .../releases/gnu-config/index.html | 18 +- bitbake/lib/bb/tests/fetch.py | 237 +- bitbake/lib/bb/tests/parse.py | 6 +- .../bb/tests/runqueue-tests/conf/bitbake.conf | 3 +- .../runqueue-tests/conf/multiconfig/mc-1.conf | 2 + .../runqueue-tests/conf/multiconfig/mc1.conf | 1 - .../runqueue-tests/conf/multiconfig/mc2.conf | 1 - .../runqueue-tests/conf/multiconfig/mc_2.conf | 2 + .../lib/bb/tests/runqueue-tests/recipes/f1.bb | 1 + .../recipes/fails-mc/fails-mc1.bb | 5 + .../recipes/fails-mc/fails-mc2.bb | 4 + bitbake/lib/bb/tests/runqueue.py | 59 +- bitbake/lib/bb/tests/siggen.py | 91 + bitbake/lib/bb/tinfoil.py | 75 +- bitbake/lib/bb/ui/buildinfohelper.py | 10 +- bitbake/lib/bb/ui/knotty.py | 27 +- bitbake/lib/bb/ui/ncurses.py | 2 + bitbake/lib/bb/ui/taskexp.py | 7 +- bitbake/lib/bb/ui/toasterui.py | 6 +- bitbake/lib/bb/ui/uievent.py | 6 +- bitbake/lib/bb/ui/uihelper.py | 4 +- bitbake/lib/bb/utils.py | 118 +- bitbake/lib/bblayers/action.py | 26 +- bitbake/lib/bblayers/layerindex.py | 2 +- bitbake/lib/bblayers/query.py | 18 +- bitbake/lib/bs4/testing.py | 6 +- bitbake/lib/hashserv/__init__.py | 52 +- bitbake/lib/hashserv/client.py | 258 +- bitbake/lib/hashserv/server.py | 232 +- bitbake/lib/hashserv/tests.py | 223 +- bitbake/lib/layerindexlib/__init__.py | 61 +- bitbake/lib/layerindexlib/cooker.py | 11 +- bitbake/lib/layerindexlib/restapi.py | 38 +- bitbake/lib/layerindexlib/tests/cooker.py | 4 +- bitbake/lib/layerindexlib/tests/restapi.py | 20 +- bitbake/lib/ply/lex.py | 6 +- bitbake/lib/ply/yacc.py | 2 +- bitbake/lib/toaster/orm/fixtures/oe-core.xml | 12 +- bitbake/lib/toaster/orm/fixtures/poky.xml | 18 +- .../orm/management/commands/lsupdates.py | 2 +- .../tests/functional/functional_helpers.py | 8 +- .../toaster/toastergui/templates/base.html | 2 +- .../toastergui/templates/configvars.html | 2 +- .../toaster/toastergui/templates/landing.html | 8 +- .../templates/landing_not_managed.html | 2 +- .../toaster/toastergui/templates/project.html | 2 +- .../templates/project_specific.html | 2 +- .../toastergui/templates/projectconf.html | 8 +- .../toastermain/management/commands/perf.py | 68 +- 173 files changed, 11143 insertions(+), 14584 deletions(-) create mode 100755 bitbake/bin/bitbake-server delete mode 100755 bitbake/bin/bitdoc create mode 100755 bitbake/contrib/bbparse-torture.py create mode 100644 bitbake/contrib/hashserv/Dockerfile create mode 100644 bitbake/contrib/vim/LICENSE.txt mode change 100755 =3D> 100644 bitbake/contrib/vim/plugin/newbb.vim create mode 100644 bitbake/contrib/vim/plugin/newbbappend.vim create mode 100644 bitbake/doc/.gitignore create mode 100644 bitbake/doc/_templates/breadcrumbs.html create mode 100644 bitbake/doc/_templates/layout.html delete mode 100644 bitbake/doc/bitbake-user-manual/bitbake-user-manual-cus= tomization.xsl create mode 100644 bitbake/doc/bitbake-user-manual/bitbake-user-manual-exe= cution.rst delete mode 100644 bitbake/doc/bitbake-user-manual/bitbake-user-manual-exe= cution.xml create mode 100644 bitbake/doc/bitbake-user-manual/bitbake-user-manual-fet= ching.rst delete mode 100644 bitbake/doc/bitbake-user-manual/bitbake-user-manual-fet= ching.xml create mode 100644 bitbake/doc/bitbake-user-manual/bitbake-user-manual-hel= lo.rst delete mode 100644 bitbake/doc/bitbake-user-manual/bitbake-user-manual-hel= lo.xml create mode 100644 bitbake/doc/bitbake-user-manual/bitbake-user-manual-int= ro.rst delete mode 100644 bitbake/doc/bitbake-user-manual/bitbake-user-manual-int= ro.xml create mode 100644 bitbake/doc/bitbake-user-manual/bitbake-user-manual-met= adata.rst delete mode 100644 bitbake/doc/bitbake-user-manual/bitbake-user-manual-met= adata.xml create mode 100644 bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref= -variables.rst delete mode 100644 bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref= -variables.xml delete mode 100644 bitbake/doc/bitbake-user-manual/bitbake-user-manual-sty= le.css delete mode 100644 bitbake/doc/bitbake-user-manual/bitbake-user-manual.xml delete mode 100644 bitbake/doc/bitbake-user-manual/html.css create mode 100644 bitbake/doc/conf.py create mode 100644 bitbake/doc/genindex.rst create mode 100644 bitbake/doc/index.rst delete mode 100644 bitbake/doc/poky.ent create mode 100644 bitbake/doc/releases.rst create mode 100644 bitbake/doc/sphinx-static/switchers.js create mode 100644 bitbake/doc/sphinx-static/theme_overrides.css delete mode 100644 bitbake/doc/template/Vera.xml delete mode 100644 bitbake/doc/template/VeraMoBd.xml delete mode 100644 bitbake/doc/template/VeraMono.xml delete mode 100644 bitbake/doc/template/component.title.xsl delete mode 100644 bitbake/doc/template/db-pdf.xsl delete mode 100644 bitbake/doc/template/division.title.xsl delete mode 100644 bitbake/doc/template/fop-config.xml delete mode 100644 bitbake/doc/template/formal.object.heading.xsl delete mode 100644 bitbake/doc/template/gloss-permalinks.xsl delete mode 100644 bitbake/doc/template/permalinks.xsl delete mode 100644 bitbake/doc/template/section.title.xsl delete mode 100644 bitbake/doc/template/titlepage.templates.xml delete mode 100755 bitbake/doc/tools/docbook-to-pdf delete mode 100644 bitbake/lib/bb/compat.py create mode 100644 bitbake/lib/bb/fetch2/az.py create mode 100644 bitbake/lib/bb/tests/color.py create mode 100644 bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc= -1.conf delete mode 100644 bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc= 1.conf delete mode 100644 bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc= 2.conf create mode 100644 bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc= _2.conf create mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/f1.bb create mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/fails-mc/fa= ils-mc1.bb create mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/fails-mc/fa= ils-mc2.bb create mode 100644 bitbake/lib/bb/tests/siggen.py diff --git a/bitbake/.gitignore b/bitbake/.gitignore index 8ca4b5d1..fbf9eb20 100644 --- a/bitbake/.gitignore +++ b/bitbake/.gitignore @@ -15,3 +15,4 @@ doc/bitbake-user-manual/bitbake-user-manual.tgz lib/toaster/contrib/tts/backlog.txt lib/toaster/contrib/tts/log/* lib/toaster/contrib/tts/.cache/* +lib/bb/tests/runqueue-tests/bitbake-cookerdaemon.log diff --git a/bitbake/README b/bitbake/README index 479c3765..96e6007e 100644 --- a/bitbake/README +++ b/bitbake/README @@ -11,7 +11,7 @@ For information about Bitbake, see the OpenEmbedded websi= te: =20 Bitbake plain documentation can be found under the doc directory or its in= tegrated html version at the Yocto Project website: - http://yoctoproject.org/documentation + https://docs.yoctoproject.org =20 Contributing ------------ diff --git a/bitbake/bin/bitbake b/bitbake/bin/bitbake index 6c73710c..71275508 100755 --- a/bitbake/bin/bitbake +++ b/bitbake/bin/bitbake @@ -26,7 +26,7 @@ from bb.main import bitbake_main, BitBakeConfigParameters= , BBMainException if sys.getfilesystemencoding() !=3D "utf-8": sys.exit("Please use a locale setting which supports UTF-8 (such as LA= NG=3Den_US.UTF-8).\nPython can't change the filesystem locale after loading= so we need a UTF-8 when Python starts or things won't work.") =20 -__version__ =3D "1.46.0" +__version__ =3D "1.50.0" =20 if __name__ =3D=3D "__main__": if __version__ !=3D bb.__version__: diff --git a/bitbake/bin/bitbake-hashclient b/bitbake/bin/bitbake-hashclient index 29ab65f1..a8929021 100755 --- a/bitbake/bin/bitbake-hashclient +++ b/bitbake/bin/bitbake-hashclient @@ -151,9 +151,6 @@ def main(): func =3D getattr(args, 'func', None) if func: client =3D hashserv.create_client(args.address) - # Try to establish a connection to the server now to detect failur= es - # early - client.connect() =20 return func(args, client) =20 diff --git a/bitbake/bin/bitbake-hashserv b/bitbake/bin/bitbake-hashserv index 1bc1f91f..153f65a3 100755 --- a/bitbake/bin/bitbake-hashserv +++ b/bitbake/bin/bitbake-hashserv @@ -30,9 +30,11 @@ def main(): "--bind [::1]:8686"''' ) =20 - parser.add_argument('--bind', default=3DDEFAULT_BIND, help=3D'Bind add= ress (default "%(default)s")') - parser.add_argument('--database', default=3D'./hashserv.db', help=3D'D= atabase file (default "%(default)s")') - parser.add_argument('--log', default=3D'WARNING', help=3D'Set logging = level') + parser.add_argument('-b', '--bind', default=3DDEFAULT_BIND, help=3D'Bi= nd address (default "%(default)s")') + parser.add_argument('-d', '--database', default=3D'./hashserv.db', hel= p=3D'Database file (default "%(default)s")') + parser.add_argument('-l', '--log', default=3D'WARNING', help=3D'Set lo= gging level') + parser.add_argument('-u', '--upstream', help=3D'Upstream hashserv to p= ull hashes from') + parser.add_argument('-r', '--read-only', action=3D'store_true', help= =3D'Disallow write operations from clients') =20 args =3D parser.parse_args() =20 @@ -47,7 +49,7 @@ def main(): console.setLevel(level) logger.addHandler(console) =20 - server =3D hashserv.create_server(args.bind, args.database) + server =3D hashserv.create_server(args.bind, args.database, upstream= =3Dargs.upstream, read_only=3Dargs.read_only) server.serve_forever() return 0 =20 diff --git a/bitbake/bin/bitbake-layers b/bitbake/bin/bitbake-layers index 149f1b1a..ff085d67 100755 --- a/bitbake/bin/bitbake-layers +++ b/bitbake/bin/bitbake-layers @@ -14,7 +14,6 @@ import logging import os import sys import argparse -import signal =20 bindir =3D os.path.dirname(__file__) topdir =3D os.path.dirname(bindir) @@ -26,7 +25,6 @@ import bb.msg logger =3D bb.msg.logger_create('bitbake-layers', sys.stdout) =20 def main(): - signal.signal(signal.SIGPIPE, signal.SIG_DFL) parser =3D argparse.ArgumentParser( description=3D"BitBake layers utility", epilog=3D"Use %(prog)s --help to get help on a specif= ic command", diff --git a/bitbake/bin/bitbake-selftest b/bitbake/bin/bitbake-selftest index 041a2719..6c073741 100755 --- a/bitbake/bin/bitbake-selftest +++ b/bitbake/bin/bitbake-selftest @@ -18,6 +18,7 @@ except RuntimeError as exc: sys.exit(str(exc)) =20 tests =3D ["bb.tests.codeparser", + "bb.tests.color", "bb.tests.cooker", "bb.tests.cow", "bb.tests.data", @@ -26,6 +27,7 @@ tests =3D ["bb.tests.codeparser", "bb.tests.parse", "bb.tests.persist_data", "bb.tests.runqueue", + "bb.tests.siggen", "bb.tests.utils", "hashserv.tests", "layerindexlib.tests.layerindexobj", diff --git a/bitbake/bin/bitbake-server b/bitbake/bin/bitbake-server new file mode 100755 index 00000000..65796be7 --- /dev/null +++ b/bitbake/bin/bitbake-server @@ -0,0 +1,54 @@ +#!/usr/bin/env python3 +# +# SPDX-License-Identifier: GPL-2.0-only +# +# Copyright (C) 2020 Richard Purdie +# + +import os +import sys +import warnings +import logging +sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(sys.argv[0= ])), 'lib')) + +if sys.getfilesystemencoding() !=3D "utf-8": + sys.exit("Please use a locale setting which supports UTF-8 (such as LA= NG=3Den_US.UTF-8).\nPython can't change the filesystem locale after loading= so we need a UTF-8 when Python starts or things won't work.") + +# Users shouldn't be running this code directly +if len(sys.argv) !=3D 10 or not sys.argv[1].startswith("decafbad"): + print("bitbake-server is meant for internal execution by bitbake itsel= f, please don't use it standalone.") + sys.exit(1) + +import bb.server.process + +lockfd =3D int(sys.argv[2]) +readypipeinfd =3D int(sys.argv[3]) +logfile =3D sys.argv[4] +lockname =3D sys.argv[5] +sockname =3D sys.argv[6] +timeout =3D float(sys.argv[7]) +xmlrpcinterface =3D (sys.argv[8], int(sys.argv[9])) +if xmlrpcinterface[0] =3D=3D "None": + xmlrpcinterface =3D (None, xmlrpcinterface[1]) +if timeout =3D=3D "None": + timeout =3D None + +# Replace standard fds with our own +with open('/dev/null', 'r') as si: + os.dup2(si.fileno(), sys.stdin.fileno()) + +so =3D open(logfile, 'a+') +os.dup2(so.fileno(), sys.stdout.fileno()) +os.dup2(so.fileno(), sys.stderr.fileno()) + +# Have stdout and stderr be the same so log output matches chronologically +# and there aren't two seperate buffers +sys.stderr =3D sys.stdout + +logger =3D logging.getLogger("BitBake") +# Ensure logging messages get sent to the UI as events +handler =3D bb.event.LogHandler() +logger.addHandler(handler) + +bb.server.process.execServer(lockfd, readypipeinfd, lockname, sockname, ti= meout, xmlrpcinterface) + diff --git a/bitbake/bin/bitbake-worker b/bitbake/bin/bitbake-worker index 97cc0fd6..4318ce61 100755 --- a/bitbake/bin/bitbake-worker +++ b/bitbake/bin/bitbake-worker @@ -16,6 +16,8 @@ import signal import pickle import traceback import queue +import shlex +import subprocess from multiprocessing import Lock from threading import Thread =20 @@ -118,7 +120,9 @@ def worker_child_fire(event, d): data =3D b"" + pickle.dumps(event) + b"" try: worker_pipe_lock.acquire() - worker_pipe.write(data) + while(len(data)): + written =3D worker_pipe.write(data) + data =3D data[written:] worker_pipe_lock.release() except IOError: sigterm_handler(None, None) @@ -143,21 +147,27 @@ def fork_off_task(cfg, data, databuilder, workerdata,= fn, task, taskname, taskha # a fork() or exec*() activates PSEUDO... =20 envbackup =3D {} + fakeroot =3D False fakeenv =3D {} umask =3D None =20 taskdep =3D workerdata["taskdeps"][fn] if 'umask' in taskdep and taskname in taskdep['umask']: + umask =3D taskdep['umask'][taskname] + elif workerdata["umask"]: + umask =3D workerdata["umask"] + if umask: # umask might come in as a number or text string.. try: - umask =3D int(taskdep['umask'][taskname],8) + umask =3D int(umask, 8) except TypeError: - umask =3D taskdep['umask'][taskname] + pass =20 dry_run =3D cfg.dry_run or dry_run_exec =20 # We can't use the fakeroot environment in a dry run as it possibly ha= sn't been built if 'fakeroot' in taskdep and taskname in taskdep['fakeroot'] and not d= ry_run: + fakeroot =3D True envvars =3D (workerdata["fakerootenv"][fn] or "").split() for key, value in (var.split('=3D') for var in envvars): envbackup[key] =3D os.environ.get(key) @@ -167,7 +177,7 @@ def fork_off_task(cfg, data, databuilder, workerdata, f= n, task, taskname, taskha fakedirs =3D (workerdata["fakerootdirs"][fn] or "").split() for p in fakedirs: bb.utils.mkdirhier(p) - logger.debug(2, 'Running %s:%s under fakeroot, fakedirs: %s' % + logger.debug2('Running %s:%s under fakeroot, fakedirs: %s' % (fn, taskname, ', '.join(fakedirs))) else: envvars =3D (workerdata["fakerootnoenv"][fn] or "").split() @@ -276,7 +286,13 @@ def fork_off_task(cfg, data, databuilder, workerdata, = fn, task, taskname, taskha try: if dry_run: return 0 - return bb.build.exec_task(fn, taskname, the_data, cfg.prof= ile) + try: + ret =3D bb.build.exec_task(fn, taskname, the_data, cfg= .profile) + finally: + if fakeroot: + fakerootcmd =3D shlex.split(the_data.getVar("FAKER= OOTCMD")) + subprocess.run(fakerootcmd + ['-S'], check=3DTrue,= stdout=3Dsubprocess.PIPE) + return ret except: os._exit(1) if not profiling: @@ -321,7 +337,9 @@ class runQueueWorkerPipe(): end =3D len(self.queue) index =3D self.queue.find(b"") while index !=3D -1: - worker_fire_prepickled(self.queue[:index+8]) + msg =3D self.queue[:index+8] + assert msg.startswith(b"") and msg.count(b"") = =3D=3D 1 + worker_fire_prepickled(msg) self.queue =3D self.queue[index+8:] index =3D self.queue.find(b"") return (end > start) @@ -413,9 +431,9 @@ class BitbakeWorker(object): =20 def handle_workerdata(self, data): self.workerdata =3D pickle.loads(data) + bb.build.verboseShellLogging =3D self.workerdata["build_verbose_sh= ell"] + bb.build.verboseStdoutLogging =3D self.workerdata["build_verbose_s= tdout"] bb.msg.loggerDefaultLogLevel =3D self.workerdata["logdefaultlevel"] - bb.msg.loggerDefaultVerbose =3D self.workerdata["logdefaultverbose= "] - bb.msg.loggerVerboseLogs =3D self.workerdata["logdefaultverboselog= s"] bb.msg.loggerDefaultDomains =3D self.workerdata["logdefaultdomain"] for mc in self.databuilder.mcdata: self.databuilder.mcdata[mc].setVar("PRSERV_HOST", self.workerd= ata["prhost"]) @@ -505,9 +523,11 @@ except BaseException as e: import traceback sys.stderr.write(traceback.format_exc()) sys.stderr.write(str(e)) +finally: + worker_thread_exit =3D True + worker_thread.join() =20 -worker_thread_exit =3D True -worker_thread.join() - -workerlog_write("exitting") +workerlog_write("exiting") +if not normalexit: + sys.exit(1) sys.exit(0) diff --git a/bitbake/bin/bitdoc b/bitbake/bin/bitdoc deleted file mode 100755 index 9bd02be6..00000000 --- a/bitbake/bin/bitdoc +++ /dev/null @@ -1,519 +0,0 @@ -#!/usr/bin/env python3 -# -# Copyright (C) 2005 Holger Hans Peter Freyther -# -# SPDX-License-Identifier: GPL-2.0-only -# - -import optparse, os, sys - -# bitbake -sys.path.append(os.path.join(os.path.dirname(os.path.dirname(__file__), 'l= ib')) -import bb -import bb.parse -from string import split, join - -__version__ =3D "0.0.2" - -class HTMLFormatter: - """ - Simple class to help to generate some sort of HTML files. It is - quite inferior solution compared to docbook, gtkdoc, doxygen but it - should work for now. - We've a global introduction site (index.html) and then one site for - the list of keys (alphabetical sorted) and one for the list of groups, - one site for each key with links to the relations and groups. - - index.html - all_keys.html - all_groups.html - groupNAME.html - keyNAME.html - """ - - def replace(self, text, *pairs): - """ - From pydoc... almost identical at least - """ - while pairs: - (a, b) =3D pairs[0] - text =3D join(split(text, a), b) - pairs =3D pairs[1:] - return text - def escape(self, text): - """ - Escape string to be conform HTML - """ - return self.replace(text,=20 - ('&', '&'),=20 - ('<', '<' ), - ('>', '>' ) ) - def createNavigator(self): - """ - Create the navgiator - """ - return """ - - - - -
HomeGroupsKeys
-""" - - def relatedKeys(self, item): - """ - Create HTML to link to foreign keys - """ - - if len(item.related()) =3D=3D 0: - return "" - - txt =3D "

See also:
" - txts =3D [] - for it in item.related(): - txts.append("""%(it)s""" % vars= () ) - - return txt + ",".join(txts) - - def groups(self, item): - """ - Create HTML to link to related groups - """ - - if len(item.groups()) =3D=3D 0: - return "" - - - txt =3D "

See also:
" - txts =3D [] - for group in item.groups(): - txts.append( """%s """ % (group, = group) ) - - return txt + ",".join(txts) - - - def createKeySite(self, item): - """ - Create a site for a key. It contains the header/navigator, a headi= ng, - the description, links to related keys and to the groups. - """ - - return """ -Key %s - - -%s -

%s

- -
-

Synopsis

-

-%s -

-
- -
-

Related Keys

-

-%s -

-
- -
-

Groups

-

-%s -

-
- - - -""" % (item.name(), self.createNavigator(), item.name(),=20 - self.escape(item.description()), self.relatedKeys(item), self.g= roups(item)) - - def createGroupsSite(self, doc): - """ - Create the Group Overview site - """ - - groups =3D "" - sorted_groups =3D sorted(doc.groups()) - for group in sorted_groups: - groups +=3D """%s
""" % (group= , group) - - return """ -Group overview - - -%s -

Available Groups

-%s - -""" % (self.createNavigator(), groups) - - def createIndex(self): - """ - Create the index file - """ - - return """ -Bitbake Documentation - - -%s -

Documentation Entrance

-All available groups
-All available keys
- -""" % self.createNavigator() - - def createKeysSite(self, doc): - """ - Create Overview of all avilable keys - """ - keys =3D "" - sorted_keys =3D sorted(doc.doc_keys()) - for key in sorted_keys: - keys +=3D """%s
""" % (key, key) - - return """ -Key overview - - -%s -

Available Keys

-%s - -""" % (self.createNavigator(), keys) - - def createGroupSite(self, gr, items, _description =3D None): - """ - Create a site for a group: - Group the name of the group, items contain the name of the keys - inside this group - """ - groups =3D "" - description =3D "" - - # create a section with the group descriptions - if _description: - description +=3D "

" % gr - description +=3D _description - - items.sort(lambda x, y:cmp(x.name(), y.name())) - for group in items: - groups +=3D """%s
""" % (group.n= ame(), group.name()) - - return """ -Group %s - - -%s -%s -
-

Keys in Group %s

-
-%s
-
-
- -""" % (gr, self.createNavigator(), description, gr, groups) - - - - def createCSS(self): - """ - Create the CSS file - """ - return """.synopsis, .classsynopsis -{ - background: #eeeeee; - border: solid 1px #aaaaaa; - padding: 0.5em; -} -.programlisting -{ - background: #eeeeff; - border: solid 1px #aaaaff; - padding: 0.5em; -} -.variablelist -{ - padding: 4px; - margin-left: 3em; -} -.variablelist td:first-child -{ - vertical-align: top; -} -table.navigation -{ - background: #ffeeee; - border: solid 1px #ffaaaa; - margin-top: 0.5em; - margin-bottom: 0.5em; -} -.navigation a -{ - color: #770000; -} -.navigation a:visited -{ - color: #550000; -} -.navigation .title -{ - font-size: 200%; -} -div.refnamediv -{ - margin-top: 2em; -} -div.gallery-float -{ - float: left; - padding: 10px; -} -div.gallery-float img -{ - border-style: none; -} -div.gallery-spacer -{ - clear: both; -} -a -{ - text-decoration: none; -} -a:hover -{ - text-decoration: underline; - color: #FF0000; -} -""" - - - -class DocumentationItem: - """ - A class to hold information about a configuration - item. It contains the key name, description, a list of related names, - and the group this item is contained in. - """ - - def __init__(self): - self._groups =3D [] - self._related =3D [] - self._name =3D "" - self._desc =3D "" - - def groups(self): - return self._groups - - def name(self): - return self._name - - def description(self): - return self._desc - - def related(self): - return self._related - - def setName(self, name): - self._name =3D name - - def setDescription(self, desc): - self._desc =3D desc - - def addGroup(self, group): - self._groups.append(group) - - def addRelation(self, relation): - self._related.append(relation) - - def sort(self): - self._related.sort() - self._groups.sort() - - -class Documentation: - """ - Holds the documentation... with mappings from key to items... - """ - - def __init__(self): - self.__keys =3D {} - self.__groups =3D {} - - def insert_doc_item(self, item): - """ - Insert the Doc Item into the internal list - of representation - """ - item.sort() - self.__keys[item.name()] =3D item - - for group in item.groups(): - if not group in self.__groups: - self.__groups[group] =3D [] - self.__groups[group].append(item) - self.__groups[group].sort() - - - def doc_item(self, key): - """ - Return the DocumentationInstance describing the key - """ - try: - return self.__keys[key] - except KeyError: - return None - - def doc_keys(self): - """ - Return the documented KEYS (names) - """ - return self.__keys.keys() - - def groups(self): - """ - Return the names of available groups - """ - return self.__groups.keys() - - def group_content(self, group_name): - """ - Return a list of keys/names that are in a specefic - group or the empty list - """ - try: - return self.__groups[group_name] - except KeyError: - return [] - - -def parse_cmdline(args): - """ - Parse the CMD line and return the result as a n-tuple - """ - - parser =3D optparse.OptionParser( version =3D "Bitbake Documentation T= ool Core version %s, %%prog version %s" % (bb.__version__, __version__)) - usage =3D """%prog [options] - -Create a set of html pages (documentation) for a bitbake.conf.... -""" - - # Add the needed options - parser.add_option( "-c", "--config", help =3D "Use the specified confi= guration file as source", - action =3D "store", dest =3D "config", default =3D = os.path.join("conf", "documentation.conf") ) - - parser.add_option( "-o", "--output", help =3D "Output directory for ht= ml files", - action =3D "store", dest =3D "output", default =3D = "html/" ) - - parser.add_option( "-D", "--debug", help =3D "Increase the debug leve= l", - action =3D "count", dest =3D "debug", default =3D 0= ) - - parser.add_option( "-v", "--verbose", help =3D "output more chit-char = to the terminal", - action =3D "store_true", dest =3D "verbose", defaul= t =3D False ) - - options, args =3D parser.parse_args( sys.argv ) -=20 - bb.msg.init_msgconfig(options.verbose, options.debug) - - return options.config, options.output - -def main(): - """ - The main Method - """ - - (config_file, output_dir) =3D parse_cmdline( sys.argv ) - - # right to let us load the file now - try: - documentation =3D bb.parse.handle( config_file, bb.data.init() ) - except IOError: - bb.fatal( "Unable to open %s" % config_file ) - except bb.parse.ParseError: - bb.fatal( "Unable to parse %s" % config_file ) - - if isinstance(documentation, dict): - documentation =3D documentation[""] - - # Assuming we've the file loaded now, we will initialize the 'tree' - doc =3D Documentation() - - # defined states - state_begin =3D 0 - state_see =3D 1 - state_group =3D 2 - - for key in bb.data.keys(documentation): - data =3D documentation.getVarFlag(key, "doc", False) - if not data: - continue - - # The Documentation now starts - doc_ins =3D DocumentationItem() - doc_ins.setName(key) - - - tokens =3D data.split(' ') - state =3D state_begin - string=3D "" - for token in tokens: - token =3D token.strip(',') - - if not state =3D=3D state_see and token =3D=3D "@see": - state =3D state_see - continue - elif not state =3D=3D state_group and token =3D=3D "@group": - state =3D state_group - continue - - if state =3D=3D state_begin: - string +=3D " %s" % token - elif state =3D=3D state_see: - doc_ins.addRelation(token) - elif state =3D=3D state_group: - doc_ins.addGroup(token) - - # set the description - doc_ins.setDescription(string) - doc.insert_doc_item(doc_ins) - - # let us create the HTML now - bb.utils.mkdirhier(output_dir) - os.chdir(output_dir) - - # Let us create the sites now. We do it in the following order - # Start with the index.html. It will point to sites explaining all - # keys and groups - html_slave =3D HTMLFormatter() - - f =3D file('style.css', 'w') - print >> f, html_slave.createCSS() - - f =3D file('index.html', 'w') - print >> f, html_slave.createIndex() - - f =3D file('all_groups.html', 'w') - print >> f, html_slave.createGroupsSite(doc) - - f =3D file('all_keys.html', 'w') - print >> f, html_slave.createKeysSite(doc) - - # now for each group create the site - for group in doc.groups(): - f =3D file('group%s.html' % group, 'w') - print >> f, html_slave.createGroupSite(group, doc.group_content(gr= oup)) - - # now for the keys - for key in doc.doc_keys(): - f =3D file('key%s.html' % doc.doc_item(key).name(), 'w') - print >> f, html_slave.createKeySite(doc.doc_item(key)) - - -if __name__ =3D=3D "__main__": - main() diff --git a/bitbake/contrib/bbparse-torture.py b/bitbake/contrib/bbparse-t= orture.py new file mode 100755 index 00000000..c25d547b --- /dev/null +++ b/bitbake/contrib/bbparse-torture.py @@ -0,0 +1,89 @@ +#! /usr/bin/env python3 +# +# Copyright (C) 2020 Joshua Watt +# +# SPDX-License-Identifier: MIT + +import argparse +import os +import random +import shutil +import signal +import subprocess +import sys +import time + + +def try_unlink(path): + try: + os.unlink(path) + except: + pass + + +def main(): + def cleanup(): + shutil.rmtree("tmp/cache", ignore_errors=3DTrue) + try_unlink("bitbake-cookerdaemon.log") + try_unlink("bitbake.sock") + try_unlink("bitbake.lock") + + parser =3D argparse.ArgumentParser( + description=3D"Bitbake parser torture test", + epilog=3D""" + A torture test for bitbake's parser. Repeatedly interrupts parsing= until + bitbake decides to deadlock. + """, + ) + + args =3D parser.parse_args() + + if not "BUILDDIR" in os.environ: + print( + "'BUILDDIR' not found in the environment. Did you initialize t= he build environment?" + ) + return 1 + + os.chdir(os.environ["BUILDDIR"]) + + run_num =3D 0 + while True: + if run_num % 100 =3D=3D 0: + print("Calibrating wait time...") + cleanup() + + start_time =3D time.monotonic() + r =3D subprocess.run(["bitbake", "-p"]) + max_wait_time =3D time.monotonic() - start_time + + if r.returncode !=3D 0: + print("Calibration run exited with %d" % r.returncode) + return 1 + + print("Maximum wait time is %f seconds" % max_wait_time) + + run_num +=3D 1 + wait_time =3D random.random() * max_wait_time + + print("Run #%d" % run_num) + print("Will sleep for %f seconds" % wait_time) + + cleanup() + with subprocess.Popen(["bitbake", "-p"]) as proc: + time.sleep(wait_time) + proc.send_signal(signal.SIGINT) + try: + proc.wait(45) + except subprocess.TimeoutExpired: + print("Run #%d: Waited too long. Possible deadlock!" % run= _num) + proc.wait() + return 1 + + if proc.returncode =3D=3D 0: + print("Exited successfully. Timeout too long?") + else: + print("Exited with %d" % proc.returncode) + + +if __name__ =3D=3D "__main__": + sys.exit(main()) diff --git a/bitbake/contrib/hashserv/Dockerfile b/bitbake/contrib/hashserv= /Dockerfile new file mode 100644 index 00000000..d6fc728f --- /dev/null +++ b/bitbake/contrib/hashserv/Dockerfile @@ -0,0 +1,19 @@ +# SPDX-License-Identifier: MIT +# +# Copyright (c) 2021 Joshua Watt +#=20 +# Dockerfile to build a bitbake hash equivalence server container +# +# From the root of the bitbake repository, run: +# +# docker build -f contrib/hashserv/Dockerfile . +# + +FROM alpine:3.13.1 + +RUN apk add --no-cache python3 + +COPY bin/bitbake-hashserv /opt/bbhashserv/bin/ +COPY lib/hashserv /opt/bbhashserv/lib/hashserv/ + +ENTRYPOINT ["/opt/bbhashserv/bin/bitbake-hashserv"] diff --git a/bitbake/contrib/vim/LICENSE.txt b/bitbake/contrib/vim/LICENSE.= txt new file mode 100644 index 00000000..c7d91502 --- /dev/null +++ b/bitbake/contrib/vim/LICENSE.txt @@ -0,0 +1,18 @@ +The MIT License (MIT) + +Permission is hereby granted, free of charge, to any person obtaining a co= py of +this software and associated documentation files (the "Software"), to deal= in +the Software without restriction, including without limitation the rights = to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell cop= ies of +the Software, and to permit persons to whom the Software is furnished to d= o so, +subject to the following conditions: + +The above copyright notice and this permission notice shall be included in= all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, F= ITNESS +FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHOR= S OR +COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHE= THER +IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. diff --git a/bitbake/contrib/vim/ftdetect/bitbake.vim b/bitbake/contrib/vim= /ftdetect/bitbake.vim index 200f8ae4..09fc4dc7 100644 --- a/bitbake/contrib/vim/ftdetect/bitbake.vim +++ b/bitbake/contrib/vim/ftdetect/bitbake.vim @@ -6,12 +6,12 @@ " " This sets up the syntax highlighting for BitBake files, like .bb, .bbcla= ss and .inc =20 -if &compatible || version < 600 +if &compatible || version < 600 || exists("b:loaded_bitbake_plugin") finish endif =20 " .bb, .bbappend and .bbclass -au BufNewFile,BufRead *.{bb,bbappend,bbclass} set filetype=3Dbitbake +au BufNewFile,BufRead *.{bb,bbappend,bbclass} set filetype=3Dbitbake =20 " .inc au BufNewFile,BufRead *.inc set filetype=3Dbitbake diff --git a/bitbake/contrib/vim/ftplugin/bitbake.vim b/bitbake/contrib/vim= /ftplugin/bitbake.vim index db0d7531..9e8d3e13 100644 --- a/bitbake/contrib/vim/ftplugin/bitbake.vim +++ b/bitbake/contrib/vim/ftplugin/bitbake.vim @@ -1,2 +1,13 @@ -set sts=3D4 sw=3D4 et -set cms=3D#%s +" Only do this when not done yet for this buffer +if exists("b:did_ftplugin") + finish +endif + +" Don't load another plugin for this buffer +let b:did_ftplugin =3D 1 + +let b:undo_ftplugin =3D "setl cms< sts< sw< et< sua<" + +setlocal commentstring=3D#\ %s +setlocal softtabstop=3D4 shiftwidth=3D4 expandtab +setlocal suffixesadd+=3D.bb,.bbclass diff --git a/bitbake/contrib/vim/plugin/newbb.vim b/bitbake/contrib/vim/plu= gin/newbb.vim old mode 100755 new mode 100644 index 874e3380..3a420273 --- a/bitbake/contrib/vim/plugin/newbb.vim +++ b/bitbake/contrib/vim/plugin/newbb.vim @@ -10,7 +10,7 @@ " " Will try to use git to find the user name and email =20 -if &compatible || v:version < 600 +if &compatible || v:version < 600 || exists("b:loaded_bitbake_plugin") finish endif =20 @@ -25,7 +25,7 @@ endfun fun! GetUserEmail() let l:user_email =3D system("git config --get user.email") if v:shell_error - return "unknow@user.org" + return "unknown@user.org" else return substitute(l:user_email, "\n", "", "") endfun @@ -41,6 +41,10 @@ fun! BBHeader() endfun =20 fun! NewBBTemplate() + if line2byte(line('$') + 1) !=3D -1 + return + endif + let l:paste =3D &paste set nopaste =20=20=20=20=20 @@ -48,7 +52,7 @@ fun! NewBBTemplate() call BBHeader() =20 " New the bb template - put =3D'DESCRIPTION =3D \"\"' + put =3D'SUMMARY =3D \"\"' put =3D'HOMEPAGE =3D \"\"' put =3D'LICENSE =3D \"\"'=20 put =3D'SECTION =3D \"\"' @@ -58,7 +62,7 @@ fun! NewBBTemplate() =20 " Go to the first place to edit 0 - /^DESCRIPTION =3D/ + /^SUMMARY =3D/ exec "normal 2f\"" =20 if paste =3D=3D 1 @@ -76,7 +80,7 @@ if v:progname =3D~ "vimdiff" endif =20 augroup NewBB - au BufNewFile *.bb + au BufNewFile,BufReadPost *.bb \ if g:bb_create_on_empty | \ call NewBBTemplate() | \ endif diff --git a/bitbake/contrib/vim/plugin/newbbappend.vim b/bitbake/contrib/v= im/plugin/newbbappend.vim new file mode 100644 index 00000000..e04174cf --- /dev/null +++ b/bitbake/contrib/vim/plugin/newbbappend.vim @@ -0,0 +1,46 @@ +" Vim plugin file +" Purpose: Create a template for new bbappend file +" Author: Joshua Watt +" Copyright: Copyright (C) 2017 Joshua Watt +" +" This file is licensed under the MIT license, see COPYING.MIT in +" this source distribution for the terms. +" + +if &compatible || v:version < 600 || exists("b:loaded_bitbake_plugin") + finish +endif + +fun! NewBBAppendTemplate() + if line2byte(line('$') + 1) !=3D -1 + return + endif + + let l:paste =3D &paste + set nopaste + + " New bbappend template + 0 put =3D'FILESEXTRAPATHS_prepend :=3D \"${THISDIR}/${PN}:\"' + 2 + + if paste =3D=3D 1 + set paste + endif +endfun + +if !exists("g:bb_create_on_empty") + let g:bb_create_on_empty =3D 1 +endif + +" disable in case of vimdiff +if v:progname =3D~ "vimdiff" + let g:bb_create_on_empty =3D 0 +endif + +augroup NewBBAppend + au BufNewFile,BufReadPost *.bbappend + \ if g:bb_create_on_empty | + \ call NewBBAppendTemplate() | + \ endif +augroup END + diff --git a/bitbake/contrib/vim/syntax/bitbake.vim b/bitbake/contrib/vim/s= yntax/bitbake.vim index fb55f910..f964621a 100644 --- a/bitbake/contrib/vim/syntax/bitbake.vim +++ b/bitbake/contrib/vim/syntax/bitbake.vim @@ -12,7 +12,7 @@ " " It's an entirely new type, just has specific syntax in shell and python = code =20 -if &compatible || v:version < 600 +if &compatible || v:version < 600 || exists("b:loaded_bitbake_plugin") finish endif if exists("b:current_syntax") @@ -58,8 +58,8 @@ syn match bbVarValue ".*$" contained contains= =3DbbString,bbVarDeref,bbV syn region bbVarPyValue start=3D+${@+ skip=3D+\\$+ end=3D+}+ conta= ined contains=3D@python =20 " Vars metadata flags -syn match bbVarFlagDef "^\([a-zA-Z0-9\-_\.]\+\)\(\[[a-zA-Z0-9\-_\= .]\+\]\)\@=3D" contains=3DbbIdentifier nextgroup=3DbbVarFlagFlag -syn region bbVarFlagFlag matchgroup=3DbbArrayBrackets start=3D"\[" = end=3D"\]\s*\(=3D\|+=3D\|=3D+\|?=3D\)\@=3D" contained contains=3DbbIdentifi= er nextgroup=3DbbVarEq +syn match bbVarFlagDef "^\([a-zA-Z0-9\-_\.]\+\)\(\[[a-zA-Z0-9\-_\= .+]\+\]\)\@=3D" contains=3DbbIdentifier nextgroup=3DbbVarFlagFlag +syn region bbVarFlagFlag matchgroup=3DbbArrayBrackets start=3D"\[" = end=3D"\]\s*\(:=3D\|=3D\|.=3D\|=3D.|+=3D\|=3D+\|?=3D\)\@=3D" contained cont= ains=3DbbIdentifier nextgroup=3DbbVarEq =20 " Includes and requires syn keyword bbInclude inherit include require contained=20 @@ -67,15 +67,15 @@ syn match bbIncludeRest ".*$" contained contain= s=3DbbString,bbVarDeref syn match bbIncludeLine "^\(inherit\|include\|require\)\s\+" conta= ins=3DbbInclude nextgroup=3DbbIncludeRest =20 " Add taks and similar -syn keyword bbStatement addtask addhandler after before EXPORT_FUN= CTIONS contained +syn keyword bbStatement addtask deltask addhandler after before EX= PORT_FUNCTIONS contained syn match bbStatementRest ".*$" skipwhite contained contains=3DbbSta= tement -syn match bbStatementLine "^\(addtask\|addhandler\|after\|before\|EX= PORT_FUNCTIONS\)\s\+" contains=3DbbStatement nextgroup=3DbbStatementRest +syn match bbStatementLine "^\(addtask\|deltask\|addhandler\|after\|b= efore\|EXPORT_FUNCTIONS\)\s\+" contains=3DbbStatement nextgroup=3DbbStateme= ntRest =20 " OE Important Functions syn keyword bbOEFunctions do_fetch do_unpack do_patch do_configure d= o_compile do_stage do_install do_package contained =20 " Generic Functions -syn match bbFunction "\h[0-9A-Za-z_-]*" display contained conta= ins=3DbbOEFunctions +syn match bbFunction "\h[0-9A-Za-z_\-\.]*" display contained co= ntains=3DbbOEFunctions =20 " BitBake shell metadata syn include @shell syntax/sh.vim @@ -83,7 +83,7 @@ if exists("b:current_syntax") unlet b:current_syntax endif syn keyword bbShFakeRootFlag fakeroot contained -syn match bbShFuncDef "^\(fakeroot\s*\)\?\([0-9A-Za-z_${}-]\+\)\= (python\)\@/dev/null 2>&1; then echo 1; else e= cho 0; fi),0) +$(error "The '$(SPHINXBUILD)' command was not found. Make sure you have Sp= hinx installed") endif =20 -## -# These URI should be rewritten by your distribution's xml catalog to -# match your localy installed XSL stylesheets. -XSL_BASE_URI =3D http://docbook.sourceforge.net/release/xsl/current -XSL_XHTML_URI =3D $(XSL_BASE_URI)/xhtml/docbook.xsl +# Put it first so that "make" without argument is like "make help". +help: + @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) =20 -all: $(ALLPREQ) +.PHONY: help Makefile clean publish =20 -pdf: -ifeq ($(DOC),bitbake-user-manual) - @echo " " - @echo "********** Building."$(DOC) - @echo " " - cd $(DOC); ../tools/docbook-to-pdf $(DOC).xml ../template; cd .. -endif - -html: -ifeq ($(DOC),bitbake-user-manual) -# See http://www.sagehill.net/docbookxsl/HtmlOutput.html - @echo " " - @echo "******** Building "$(DOC) - @echo " " - cd $(DOC); xsltproc $(XSLTOPTS) -o $(DOC).html $(DOC)-customization.xsl $= (DOC).xml; cd .. -endif - -tarball: html - @echo " " - @echo "******** Creating Tarball of document files" - @echo " " - cd $(DOC); tar -cvzf $(DOC).tgz $(TARFILES); cd .. - -validate: - cd $(DOC); xmllint --postvalid --xinclude --noout $(DOC).xml; cd .. - -publish: - @if test -f $(DOC)/$(DOC).html; \ - then \ - echo " "; \ - echo "******** Publishing "$(DOC)".html"; \ - echo " "; \ - scp -r $(MANUALS) $(STYLESHEET) docs.yp:/var/www/www.yoctoproj= ect.org-docs/$(VER)/$(DOC); \ - cd $(DOC); scp -r $(FIGURES) docs.yp:/var/www/www.yoctoproject= .org-docs/$(VER)/$(DOC); \ - else \ - echo " "; \ - echo $(DOC)".html missing. Generate the file first then try aga= in."; \ - echo " "; \ - fi +publish: Makefile html singlehtml + rm -rf $(BUILDDIR)/$(DESTDIR)/ + mkdir -p $(BUILDDIR)/$(DESTDIR)/ + cp -r $(BUILDDIR)/html/* $(BUILDDIR)/$(DESTDIR)/ + cp $(BUILDDIR)/singlehtml/index.html $(BUILDDIR)/$(DESTDIR)/singleindex.h= tml + sed -i -e 's@index.html#@singleindex.html#@g' $(BUILDDIR)/$(DESTDIR)/sing= leindex.html =20 clean: - rm -rf $(MANUALS); rm $(DOC)/$(DOC).tgz; + @rm -rf $(BUILDDIR) + +# Catch-all target: route all unknown targets to Sphinx using the new +# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). +%: Makefile + @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) diff --git a/bitbake/doc/README b/bitbake/doc/README index 303cf8ee..cbb5215d 100644 --- a/bitbake/doc/README +++ b/bitbake/doc/README @@ -15,25 +15,41 @@ Each folder is self-contained regarding content and fig= ures. If you want to find HTML versions of the BitBake manuals on the web,=20 go to http://www.openembedded.org/wiki/Documentation.=20 =20 -Makefile -=3D=3D=3D=3D=3D=3D=3D=3D +Sphinx +=3D=3D=3D=3D=3D=3D =20 -The Makefile processes manual directories to create HTML, PDF, -tarballs, etc. Details on how the Makefile work are documented -inside the Makefile. See that file for more information. +The BitBake documentation was migrated from the original DocBook +format to Sphinx based documentation for the Yocto Project 3.2 +release. =20 -To build a manual, you run the make command and pass it the name -of the folder containing the manual's contents.=20 -For example, the following command run from the documentation directory=20 -creates an HTML and a PDF version of the BitBake User Manual. -The DOC variable specifies the manual you are making: +Additional information related to the Sphinx migration, and guidelines +for developers willing to contribute to the BitBake documentation can +be found in the Yocto Project Documentation README file: =20 - $ make DOC=3Dbitbake-user-manual +https://git.yoctoproject.org/cgit/cgit.cgi/yocto-docs/tree/documentation/R= EADME =20 -template -=3D=3D=3D=3D=3D=3D=3D=3D -Contains various templates, fonts, and some old PNG files. +How to build the Yocto Project documentation +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 -tools -=3D=3D=3D=3D=3D -Contains a tool to convert the DocBook files to PDF format. +Sphinx is written in Python. While it might work with Python2, for +obvious reasons, we will only support building the BitBake +documentation with Python3. + +Sphinx might be available in your Linux distro packages repositories, +however it is not recommend using distro packages, as they might be +old versions, especially if you are using an LTS version of your +distro. The recommended method to install Sphinx and all required +dependencies is to use the Python Package Index (pip). + +To install all required packages run: + + $ pip3 install sphinx sphinx_rtd_theme pyyaml + +To build the documentation locally, run: + + $ cd documentation + $ make html + +The resulting HTML index page will be _build/html/index.html, and you +can browse your own copy of the locally generated documentation with +your browser. diff --git a/bitbake/doc/_templates/breadcrumbs.html b/bitbake/doc/_templat= es/breadcrumbs.html new file mode 100644 index 00000000..eb6244b7 --- /dev/null +++ b/bitbake/doc/_templates/breadcrumbs.html @@ -0,0 +1,14 @@ +{% extends "!breadcrumbs.html" %} + +{% block breadcrumbs %} +
  • + {{ doctype or 'single' = }} + {{ release }} +
    • +
  • »
    • + {% for doc in parents %} +
  • {{ doc.title }} »
    • + {% endfor %} +
  • {{ title }}
    • +{% endblock %} + diff --git a/bitbake/doc/_templates/layout.html b/bitbake/doc/_templates/la= yout.html new file mode 100644 index 00000000..308d5c7a --- /dev/null +++ b/bitbake/doc/_templates/layout.html @@ -0,0 +1,7 @@ +{% extends "!layout.html" %} + +{% block extrabody %} +
    +
    +{% endblock %} + diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-customizat= ion.xsl b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-customization= .xsl deleted file mode 100644 index 5985ea78..00000000 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-customization.xsl +++ /dev/null @@ -1,29 +0,0 @@ - - - - - - - - - - - - - - - - - - - A - - - - diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.= rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst new file mode 100644 index 00000000..56abf773 --- /dev/null +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst @@ -0,0 +1,734 @@ +.. SPDX-License-Identifier: CC-BY-2.5 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D +Execution +=3D=3D=3D=3D=3D=3D=3D=3D=3D + +| + +The primary purpose for running BitBake is to produce some kind of +output such as a single installable package, a kernel, a software +development kit, or even a full, board-specific bootable Linux image, +complete with bootloader, kernel, and root filesystem. Of course, you +can execute the ``bitbake`` command with options that cause it to +execute single tasks, compile single recipe files, capture or clear +data, or simply return information about the execution environment. + +This chapter describes BitBake's execution process from start to finish +when you use it to create an image. The execution process is launched +using the following command form: :: + + $ bitbake target + +For information on +the BitBake command and its options, see ":ref:`The BitBake Command +`" section. + +.. note:: + + Prior to executing BitBake, you should take advantage of available + parallel thread execution on your build host by setting the + :term:`BB_NUMBER_THREADS` variable in + your project's ``local.conf`` configuration file. + + A common method to determine this value for your build host is to run + the following: :: + + $ grep processor /proc/cpuinfo + + This command returns + the number of processors, which takes into account hyper-threading. + Thus, a quad-core build host with hyper-threading most likely shows + eight processors, which is the value you would then assign to + ``BB_NUMBER_THREADS``. + + A possibly simpler solution is that some Linux distributions (e.g. + Debian and Ubuntu) provide the ``ncpus`` command. + +Parsing the Base Configuration Metadata +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +The first thing BitBake does is parse base configuration metadata. Base +configuration metadata consists of your project's ``bblayers.conf`` file +to determine what layers BitBake needs to recognize, all necessary +``layer.conf`` files (one from each layer), and ``bitbake.conf``. The +data itself is of various types: + +- **Recipes:** Details about particular pieces of software. + +- **Class Data:** An abstraction of common build information (e.g. how to + build a Linux kernel). + +- **Configuration Data:** Machine-specific settings, policy decisions, + and so forth. Configuration data acts as the glue to bind everything + together. + +The ``layer.conf`` files are used to construct key variables such as +:term:`BBPATH` and :term:`BBFILES`. +``BBPATH`` is used to search for configuration and class files under the +``conf`` and ``classes`` directories, respectively. ``BBFILES`` is used +to locate both recipe and recipe append files (``.bb`` and +``.bbappend``). If there is no ``bblayers.conf`` file, it is assumed the +user has set the ``BBPATH`` and ``BBFILES`` directly in the environment. + +Next, the ``bitbake.conf`` file is located using the ``BBPATH`` variable +that was just constructed. The ``bitbake.conf`` file may also include +other configuration files using the ``include`` or ``require`` +directives. + +Prior to parsing configuration files, BitBake looks at certain +variables, including: + +- :term:`BB_ENV_WHITELIST` +- :term:`BB_ENV_EXTRAWHITE` +- :term:`BB_PRESERVE_ENV` +- :term:`BB_ORIGENV` +- :term:`BITBAKE_UI` + +The first four variables in this list relate to how BitBake treats shell +environment variables during task execution. By default, BitBake cleans +the environment variables and provides tight control over the shell +execution environment. However, through the use of these first four +variables, you can apply your control regarding the environment +variables allowed to be used by BitBake in the shell during execution of +tasks. See the +":ref:`bitbake-user-manual/bitbake-user-manual-metadata:Passing Informatio= n Into the Build Task Environment`" +section and the information about these variables in the variable +glossary for more information on how they work and on how to use them. + +The base configuration metadata is global and therefore affects all +recipes and tasks that are executed. + +BitBake first searches the current working directory for an optional +``conf/bblayers.conf`` configuration file. This file is expected to +contain a :term:`BBLAYERS` variable that is a +space-delimited list of 'layer' directories. Recall that if BitBake +cannot find a ``bblayers.conf`` file, then it is assumed the user has +set the ``BBPATH`` and ``BBFILES`` variables directly in the +environment. + +For each directory (layer) in this list, a ``conf/layer.conf`` file is +located and parsed with the :term:`LAYERDIR` variable +being set to the directory where the layer was found. The idea is these +files automatically set up :term:`BBPATH` and other +variables correctly for a given build directory. + +BitBake then expects to find the ``conf/bitbake.conf`` file somewhere in +the user-specified ``BBPATH``. That configuration file generally has +include directives to pull in any other metadata such as files specific +to the architecture, the machine, the local environment, and so forth. + +Only variable definitions and include directives are allowed in BitBake +``.conf`` files. Some variables directly influence BitBake's behavior. +These variables might have been set from the environment depending on +the environment variables previously mentioned or set in the +configuration files. The ":ref:`bitbake-user-manual/bitbake-user-manual-re= f-variables:Variables Glossary`" +chapter presents a full list of +variables. + +After parsing configuration files, BitBake uses its rudimentary +inheritance mechanism, which is through class files, to inherit some +standard classes. BitBake parses a class when the inherit directive +responsible for getting that class is encountered. + +The ``base.bbclass`` file is always included. Other classes that are +specified in the configuration using the +:term:`INHERIT` variable are also included. BitBake +searches for class files in a ``classes`` subdirectory under the paths +in ``BBPATH`` in the same way as configuration files. + +A good way to get an idea of the configuration files and the class files +used in your execution environment is to run the following BitBake +command: :: + + $ bitbake -e > mybb.log + +Examining the top of the ``mybb.log`` +shows you the many configuration files and class files used in your +execution environment. + +.. note:: + + You need to be aware of how BitBake parses curly braces. If a recipe + uses a closing curly brace within the function and the character has + no leading spaces, BitBake produces a parsing error. If you use a + pair of curly braces in a shell function, the closing curly brace + must not be located at the start of the line without leading spaces. + + Here is an example that causes BitBake to produce a parsing error: :: + + fakeroot create_shar() { + cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh + usage() + { + echo "test" + ###### The following "}" at the start of the line causes a parsi= ng error ###### + } + EOF + } + + Writing the recipe this way avoids the error: + fakeroot create_shar() { + cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh + usage() + { + echo "test" + ###### The following "}" with a leading space at the start of the= line avoids the error ###### + } + EOF + } + +Locating and Parsing Recipes +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D + +During the configuration phase, BitBake will have set +:term:`BBFILES`. BitBake now uses it to construct a +list of recipes to parse, along with any append files (``.bbappend``) to +apply. ``BBFILES`` is a space-separated list of available files and +supports wildcards. An example would be: :: + + BBFILES =3D "/path/to/bbfiles/*.bb /path/to/appends/*.bbappend" + +BitBake parses each +recipe and append file located with ``BBFILES`` and stores the values of +various variables into the datastore. + +.. note:: + + Append files are applied in the order they are encountered in BBFILES. + +For each file, a fresh copy of the base configuration is made, then the +recipe is parsed line by line. Any inherit statements cause BitBake to +find and then parse class files (``.bbclass``) using +:term:`BBPATH` as the search path. Finally, BitBake +parses in order any append files found in ``BBFILES``. + +One common convention is to use the recipe filename to define pieces of +metadata. For example, in ``bitbake.conf`` the recipe name and version +are used to set the variables :term:`PN` and +:term:`PV`: :: + + PN =3D "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)= [0] or 'defaultpkgname'}" + PV =3D "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)= [1] or '1.0'}" + +In this example, a recipe called "something_1.2.3.bb" would set +``PN`` to "something" and ``PV`` to "1.2.3". + +By the time parsing is complete for a recipe, BitBake has a list of +tasks that the recipe defines and a set of data consisting of keys and +values as well as dependency information about the tasks. + +BitBake does not need all of this information. It only needs a small +subset of the information to make decisions about the recipe. +Consequently, BitBake caches the values in which it is interested and +does not store the rest of the information. Experience has shown it is +faster to re-parse the metadata than to try and write it out to the disk +and then reload it. + +Where possible, subsequent BitBake commands reuse this cache of recipe +information. The validity of this cache is determined by first computing +a checksum of the base configuration data (see +:term:`BB_HASHCONFIG_WHITELIST`) and +then checking if the checksum matches. If that checksum matches what is +in the cache and the recipe and class files have not changed, BitBake is +able to use the cache. BitBake then reloads the cached information about +the recipe instead of reparsing it from scratch. + +Recipe file collections exist to allow the user to have multiple +repositories of ``.bb`` files that contain the same exact package. For +example, one could easily use them to make one's own local copy of an +upstream repository, but with custom modifications that one does not +want upstream. Here is an example: :: + + BBFILES =3D "/stuff/openembedded/*/*.bb /stuff/openembedded.modified/*/*= .bb" + BBFILE_COLLECTIONS =3D "upstream local" + BBFILE_PATTERN_upstream =3D "^/stuff/openembedded/" + BBFILE_PATTERN_local =3D "^/stuff/openembedded.modified/" + BBFILE_PRIORITY_upstream =3D "5" + BBFILE_PRIORITY_local =3D "10" + +.. note:: + + The layers mechanism is now the preferred method of collecting code. + While the collections code remains, its main use is to set layer + priorities and to deal with overlap (conflicts) between layers. + +.. _bb-bitbake-providers: + +Providers +=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Assuming BitBake has been instructed to execute a target and that all +the recipe files have been parsed, BitBake starts to figure out how to +build the target. BitBake looks through the ``PROVIDES`` list for each +of the recipes. A ``PROVIDES`` list is the list of names by which the +recipe can be known. Each recipe's ``PROVIDES`` list is created +implicitly through the recipe's :term:`PN` variable and +explicitly through the recipe's :term:`PROVIDES` +variable, which is optional. + +When a recipe uses ``PROVIDES``, that recipe's functionality can be +found under an alternative name or names other than the implicit ``PN`` +name. As an example, suppose a recipe named ``keyboard_1.0.bb`` +contained the following: :: + + PROVIDES +=3D "fullkeyboard" + +The ``PROVIDES`` +list for this recipe becomes "keyboard", which is implicit, and +"fullkeyboard", which is explicit. Consequently, the functionality found +in ``keyboard_1.0.bb`` can be found under two different names. + +.. _bb-bitbake-preferences: + +Preferences +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +The ``PROVIDES`` list is only part of the solution for figuring out a +target's recipes. Because targets might have multiple providers, BitBake +needs to prioritize providers by determining provider preferences. + +A common example in which a target has multiple providers is +"virtual/kernel", which is on the ``PROVIDES`` list for each kernel +recipe. Each machine often selects the best kernel provider by using a +line similar to the following in the machine configuration file: :: + + PREFERRED_PROVIDER_virtual/kernel =3D "linux-yocto" + +The default :term:`PREFERRED_PROVIDER` is the provider +with the same name as the target. BitBake iterates through each target +it needs to build and resolves them and their dependencies using this +process. + +Understanding how providers are chosen is made complicated by the fact +that multiple versions might exist for a given provider. BitBake +defaults to the highest version of a provider. Version comparisons are +made using the same method as Debian. You can use the +:term:`PREFERRED_VERSION` variable to +specify a particular version. You can influence the order by using the +:term:`DEFAULT_PREFERENCE` variable. + +By default, files have a preference of "0". Setting +``DEFAULT_PREFERENCE`` to "-1" makes the recipe unlikely to be used +unless it is explicitly referenced. Setting ``DEFAULT_PREFERENCE`` to +"1" makes it likely the recipe is used. ``PREFERRED_VERSION`` overrides +any ``DEFAULT_PREFERENCE`` setting. ``DEFAULT_PREFERENCE`` is often used +to mark newer and more experimental recipe versions until they have +undergone sufficient testing to be considered stable. + +When there are multiple "versions" of a given recipe, BitBake defaults +to selecting the most recent version, unless otherwise specified. If the +recipe in question has a +:term:`DEFAULT_PREFERENCE` set lower than +the other recipes (default is 0), then it will not be selected. This +allows the person or persons maintaining the repository of recipe files +to specify their preference for the default selected version. +Additionally, the user can specify their preferred version. + +If the first recipe is named ``a_1.1.bb``, then the +:term:`PN` variable will be set to "a", and the +:term:`PV` variable will be set to 1.1. + +Thus, if a recipe named ``a_1.2.bb`` exists, BitBake will choose 1.2 by +default. However, if you define the following variable in a ``.conf`` +file that BitBake parses, you can change that preference: :: + + PREFERRED_VERSION_a =3D "1.1" + +.. note:: + + It is common for a recipe to provide two versions -- a stable, + numbered (and preferred) version, and a version that is automatically + checked out from a source code repository that is considered more + "bleeding edge" but can be selected only explicitly. + + For example, in the OpenEmbedded codebase, there is a standard, + versioned recipe file for BusyBox, ``busybox_1.22.1.bb``, but there + is also a Git-based version, ``busybox_git.bb``, which explicitly + contains the line :: + + DEFAULT_PREFERENCE =3D "-1" + + to ensure that the + numbered, stable version is always preferred unless the developer + selects otherwise. + +.. _bb-bitbake-dependencies: + +Dependencies +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Each target BitBake builds consists of multiple tasks such as ``fetch``, +``unpack``, ``patch``, ``configure``, and ``compile``. For best +performance on multi-core systems, BitBake considers each task as an +independent entity with its own set of dependencies. + +Dependencies are defined through several variables. You can find +information about variables BitBake uses in the +:doc:`bitbake-user-manual-ref-variables` near the end of this manual. At a +basic level, it is sufficient to know that BitBake uses the +:term:`DEPENDS` and +:term:`RDEPENDS` variables when calculating +dependencies. + +For more information on how BitBake handles dependencies, see the +:ref:`bitbake-user-manual/bitbake-user-manual-metadata:Dependencies` +section. + +.. _ref-bitbake-tasklist: + +The Task List +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Based on the generated list of providers and the dependency information, +BitBake can now calculate exactly what tasks it needs to run and in what +order it needs to run them. The +:ref:`bitbake-user-manual/bitbake-user-manual-execution:executing tasks` +section has more information on how BitBake chooses which task to +execute next. + +The build now starts with BitBake forking off threads up to the limit +set in the :term:`BB_NUMBER_THREADS` +variable. BitBake continues to fork threads as long as there are tasks +ready to run, those tasks have all their dependencies met, and the +thread threshold has not been exceeded. + +It is worth noting that you can greatly speed up the build time by +properly setting the ``BB_NUMBER_THREADS`` variable. + +As each task completes, a timestamp is written to the directory +specified by the :term:`STAMP` variable. On subsequent +runs, BitBake looks in the build directory within ``tmp/stamps`` and +does not rerun tasks that are already completed unless a timestamp is +found to be invalid. Currently, invalid timestamps are only considered +on a per recipe file basis. So, for example, if the configure stamp has +a timestamp greater than the compile timestamp for a given target, then +the compile task would rerun. Running the compile task again, however, +has no effect on other providers that depend on that target. + +The exact format of the stamps is partly configurable. In modern +versions of BitBake, a hash is appended to the stamp so that if the +configuration changes, the stamp becomes invalid and the task is +automatically rerun. This hash, or signature used, is governed by the +signature policy that is configured (see the +:ref:`bitbake-user-manual/bitbake-user-manual-execution:checksums (signatu= res)` +section for information). It is also +possible to append extra metadata to the stamp using the +``[stamp-extra-info]`` task flag. For example, OpenEmbedded uses this +flag to make some tasks machine-specific. + +.. note:: + + Some tasks are marked as "nostamp" tasks. No timestamp file is + created when these tasks are run. Consequently, "nostamp" tasks are + always rerun. + +For more information on tasks, see the +:ref:`bitbake-user-manual/bitbake-user-manual-metadata:tasks` section. + +Executing Tasks +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Tasks can be either a shell task or a Python task. For shell tasks, +BitBake writes a shell script to +``${``\ :term:`T`\ ``}/run.do_taskname.pid`` and then +executes the script. The generated shell script contains all the +exported variables, and the shell functions with all variables expanded. +Output from the shell script goes to the file +``${T}/log.do_taskname.pid``. Looking at the expanded shell functions in +the run file and the output in the log files is a useful debugging +technique. + +For Python tasks, BitBake executes the task internally and logs +information to the controlling terminal. Future versions of BitBake will +write the functions to files similar to the way shell tasks are handled. +Logging will be handled in a way similar to shell tasks as well. + +The order in which BitBake runs the tasks is controlled by its task +scheduler. It is possible to configure the scheduler and define custom +implementations for specific use cases. For more information, see these +variables that control the behavior: + +- :term:`BB_SCHEDULER` + +- :term:`BB_SCHEDULERS` + +It is possible to have functions run before and after a task's main +function. This is done using the ``[prefuncs]`` and ``[postfuncs]`` +flags of the task that lists the functions to run. + +.. _checksums: + +Checksums (Signatures) +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +A checksum is a unique signature of a task's inputs. The signature of a +task can be used to determine if a task needs to be run. Because it is a +change in a task's inputs that triggers running the task, BitBake needs +to detect all the inputs to a given task. For shell tasks, this turns +out to be fairly easy because BitBake generates a "run" shell script for +each task and it is possible to create a checksum that gives you a good +idea of when the task's data changes. + +To complicate the problem, some things should not be included in the +checksum. First, there is the actual specific build path of a given task +- the working directory. It does not matter if the working directory +changes because it should not affect the output for target packages. The +simplistic approach for excluding the working directory is to set it to +some fixed value and create the checksum for the "run" script. BitBake +goes one step better and uses the +:term:`BB_HASHBASE_WHITELIST` variable +to define a list of variables that should never be included when +generating the signatures. + +Another problem results from the "run" scripts containing functions that +might or might not get called. The incremental build solution contains +code that figures out dependencies between shell functions. This code is +used to prune the "run" scripts down to the minimum set, thereby +alleviating this problem and making the "run" scripts much more readable +as a bonus. + +So far we have solutions for shell scripts. What about Python tasks? The +same approach applies even though these tasks are more difficult. The +process needs to figure out what variables a Python function accesses +and what functions it calls. Again, the incremental build solution +contains code that first figures out the variable and function +dependencies, and then creates a checksum for the data used as the input +to the task. + +Like the working directory case, situations exist where dependencies +should be ignored. For these cases, you can instruct the build process +to ignore a dependency by using a line like the following: :: + + PACKAGE_ARCHS[vardepsexclude] =3D "MACHINE" + +This example ensures that the +``PACKAGE_ARCHS`` variable does not depend on the value of ``MACHINE``, +even if it does reference it. + +Equally, there are cases where we need to add dependencies BitBake is +not able to find. You can accomplish this by using a line like the +following: :: + + PACKAGE_ARCHS[vardeps] =3D "MACHINE" + +This example explicitly +adds the ``MACHINE`` variable as a dependency for ``PACKAGE_ARCHS``. + +Consider a case with in-line Python, for example, where BitBake is not +able to figure out dependencies. When running in debug mode (i.e. using +``-DDD``), BitBake produces output when it discovers something for which +it cannot figure out dependencies. + +Thus far, this section has limited discussion to the direct inputs into +a task. Information based on direct inputs is referred to as the +"basehash" in the code. However, there is still the question of a task's +indirect inputs - the things that were already built and present in the +build directory. The checksum (or signature) for a particular task needs +to add the hashes of all the tasks on which the particular task depends. +Choosing which dependencies to add is a policy decision. However, the +effect is to generate a master checksum that combines the basehash and +the hashes of the task's dependencies. + +At the code level, there are a variety of ways both the basehash and the +dependent task hashes can be influenced. Within the BitBake +configuration file, we can give BitBake some extra information to help +it construct the basehash. The following statement effectively results +in a list of global variable dependency excludes - variables never +included in any checksum. This example uses variables from OpenEmbedded +to help illustrate the concept: :: + + BB_HASHBASE_WHITELIST ?=3D "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_= DIR \ + SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL \ + USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_= HOST \ + PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \ + CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SD= KPKGSUFFIX" + +The previous example excludes the work directory, which is part of +``TMPDIR``. + +The rules for deciding which hashes of dependent tasks to include +through dependency chains are more complex and are generally +accomplished with a Python function. The code in +``meta/lib/oe/sstatesig.py`` shows two examples of this and also +illustrates how you can insert your own policy into the system if so +desired. This file defines the two basic signature generators +OpenEmbedded-Core uses: "OEBasic" and "OEBasicHash". By default, there +is a dummy "noop" signature handler enabled in BitBake. This means that +behavior is unchanged from previous versions. ``OE-Core`` uses the +"OEBasicHash" signature handler by default through this setting in the +``bitbake.conf`` file: :: + + BB_SIGNATURE_HANDLER ?=3D "OEBasicHash" + +The "OEBasicHash" ``BB_SIGNATURE_HANDLER`` is the same as the "OEBasic" +version but adds the task hash to the stamp files. This results in any +metadata change that changes the task hash, automatically causing the +task to be run again. This removes the need to bump +:term:`PR` values, and changes to metadata automatically +ripple across the build. + +It is also worth noting that the end result of these signature +generators is to make some dependency and hash information available to +the build. This information includes: + +- ``BB_BASEHASH_task-``\ *taskname*: The base hashes for each task in the + recipe. + +- ``BB_BASEHASH_``\ *filename:taskname*: The base hashes for each + dependent task. + +- ``BBHASHDEPS_``\ *filename:taskname*: The task dependencies for + each task. + +- ``BB_TASKHASH``: The hash of the currently running task. + +It is worth noting that BitBake's "-S" option lets you debug BitBake's +processing of signatures. The options passed to -S allow different +debugging modes to be used, either using BitBake's own debug functions +or possibly those defined in the metadata/signature handler itself. The +simplest parameter to pass is "none", which causes a set of signature +information to be written out into ``STAMPS_DIR`` corresponding to the +targets specified. The other currently available parameter is +"printdiff", which causes BitBake to try to establish the closest +signature match it can (e.g. in the sstate cache) and then run +``bitbake-diffsigs`` over the matches to determine the stamps and delta +where these two stamp trees diverge. + +.. note:: + + It is likely that future versions of BitBake will provide other + signature handlers triggered through additional "-S" parameters. + +You can find more information on checksum metadata in the +:ref:`bitbake-user-manual/bitbake-user-manual-metadata:task checksums and = setscene` +section. + +Setscene +=3D=3D=3D=3D=3D=3D=3D=3D + +The setscene process enables BitBake to handle "pre-built" artifacts. +The ability to handle and reuse these artifacts allows BitBake the +luxury of not having to build something from scratch every time. +Instead, BitBake can use, when possible, existing build artifacts. + +BitBake needs to have reliable data indicating whether or not an +artifact is compatible. Signatures, described in the previous section, +provide an ideal way of representing whether an artifact is compatible. +If a signature is the same, an object can be reused. + +If an object can be reused, the problem then becomes how to replace a +given task or set of tasks with the pre-built artifact. BitBake solves +the problem with the "setscene" process. + +When BitBake is asked to build a given target, before building anything, +it first asks whether cached information is available for any of the +targets it's building, or any of the intermediate targets. If cached +information is available, BitBake uses this information instead of +running the main tasks. + +BitBake first calls the function defined by the +:term:`BB_HASHCHECK_FUNCTION` variable +with a list of tasks and corresponding hashes it wants to build. This +function is designed to be fast and returns a list of the tasks for +which it believes in can obtain artifacts. + +Next, for each of the tasks that were returned as possibilities, BitBake +executes a setscene version of the task that the possible artifact +covers. Setscene versions of a task have the string "_setscene" appended +to the task name. So, for example, the task with the name ``xxx`` has a +setscene task named ``xxx_setscene``. The setscene version of the task +executes and provides the necessary artifacts returning either success +or failure. + +As previously mentioned, an artifact can cover more than one task. For +example, it is pointless to obtain a compiler if you already have the +compiled binary. To handle this, BitBake calls the +:term:`BB_SETSCENE_DEPVALID` function for +each successful setscene task to know whether or not it needs to obtain +the dependencies of that task. + +Finally, after all the setscene tasks have executed, BitBake calls the +function listed in +:term:`BB_SETSCENE_VERIFY_FUNCTION2` +with the list of tasks BitBake thinks has been "covered". The metadata +can then ensure that this list is correct and can inform BitBake that it +wants specific tasks to be run regardless of the setscene result. + +You can find more information on setscene metadata in the +:ref:`bitbake-user-manual/bitbake-user-manual-metadata:task checksums and = setscene` +section. + +Logging +=3D=3D=3D=3D=3D=3D=3D + +In addition to the standard command line option to control how verbose +builds are when execute, bitbake also supports user defined +configuration of the `Python +logging `__ facilities +through the :term:`BB_LOGCONFIG` variable. This +variable defines a json or yaml `logging +configuration `__ +that will be intelligently merged into the default configuration. The +logging configuration is merged using the following rules: + +- The user defined configuration will completely replace the default + configuration if top level key ``bitbake_merge`` is set to the value + ``False``. In this case, all other rules are ignored. + +- The user configuration must have a top level ``version`` which must + match the value of the default configuration. + +- Any keys defined in the ``handlers``, ``formatters``, or ``filters``, + will be merged into the same section in the default configuration, + with the user specified keys taking replacing a default one if there + is a conflict. In practice, this means that if both the default + configuration and user configuration specify a handler named + ``myhandler``, the user defined one will replace the default. To + prevent the user from inadvertently replacing a default handler, + formatter, or filter, all of the default ones are named with a prefix + of "``BitBake.``" + +- If a logger is defined by the user with the key ``bitbake_merge`` set + to ``False``, that logger will be completely replaced by user + configuration. In this case, no other rules will apply to that + logger. + +- All user defined ``filter`` and ``handlers`` properties for a given + logger will be merged with corresponding properties from the default + logger. For example, if the user configuration adds a filter called + ``myFilter`` to the ``BitBake.SigGen``, and the default configuration + adds a filter called ``BitBake.defaultFilter``, both filters will be + applied to the logger + +As an example, consider the following user logging configuration file +which logs all Hash Equivalence related messages of VERBOSE or higher to +a file called ``hashequiv.log`` :: + + { + "version": 1, + "handlers": { + "autobuilderlog": { + "class": "logging.FileHandler", + "formatter": "logfileFormatter", + "level": "DEBUG", + "filename": "hashequiv.log", + "mode": "w" + } + }, + "formatters": { + "logfileFormatter": { + "format": "%(name)s: %(levelname)s: %(message)s" + } + }, + "loggers": { + "BitBake.SigGen.HashEquiv": { + "level": "VERBOSE", + "handlers": ["autobuilderlog"] + }, + "BitBake.RunQueue.HashEquiv": { + "level": "VERBOSE", + "handlers": ["autobuilderlog"] + } + } + } diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.= xml b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml deleted file mode 100644 index e4251dff..00000000 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml +++ /dev/null @@ -1,1029 +0,0 @@ - - - - Execution - - - The primary purpose for running BitBake is to produce some kind - of output such as a single installable package, a kernel, a softwa= re - development kit, or even a full, board-specific bootable Linux ima= ge, - complete with bootloader, kernel, and root filesystem. - Of course, you can execute the bitbake - command with options that cause it to execute single tasks, - compile single recipe files, capture or clear data, or simply - return information about the execution environment. - - - - This chapter describes BitBake's execution process from start - to finish when you use it to create an image. - The execution process is launched using the following command - form: - - $ bitbake target - - For information on the BitBake command and its options, - see - "The BitBake Command= " - section. - - - Prior to executing BitBake, you should take advantage of a= vailable - parallel thread execution on your build host by setting the - BB_NU= MBER_THREADS - variable in your project's local.conf - configuration file. - - - - A common method to determine this value for your build hos= t is to run - the following: - - $ grep processor /proc/cpuinfo - - This command returns the number of processors, which takes= into - account hyper-threading. - Thus, a quad-core build host with hyper-threading most lik= ely - shows eight processors, which is the value you would then = assign to - BB_NUMBER_THREADS. - - - - A possibly simpler solution is that some Linux distributio= ns - (e.g. Debian and Ubuntu) provide the ncpus command. - - - - -
    - Parsing the Base Configuration Metadata - - - The first thing BitBake does is parse base configuration - metadata. - Base configuration metadata consists of your project's - bblayers.conf file to determine what - layers BitBake needs to recognize, all necessary - layer.conf files (one from each layer), - and bitbake.conf. - The data itself is of various types: - - Recipes: - Details about particular pieces of software. - - Class Data: - An abstraction of common build information - (e.g. how to build a Linux kernel). - - Configuration Data: - Machine-specific settings, policy decisions, - and so forth. - Configuration data acts as the glue to bind everything - together. - - - - - The layer.conf files are used to - construct key variables such as - BBPATH - and - BBFILES<= /link>. - BBPATH is used to search for - configuration and class files under the - conf and classes - directories, respectively. - BBFILES is used to locate both recipe - and recipe append files - (.bb and .bbappend). - If there is no bblayers.conf file, - it is assumed the user has set the BBPATH - and BBFILES directly in the environment. - - - - Next, the bitbake.conf file is located - using the BBPATH variable that was - just constructed. - The bitbake.conf file may also include ot= her - configuration files using the - include or - require directives. - - - - Prior to parsing configuration files, BitBake looks - at certain variables, including: - - - BB= _ENV_WHITELIST - - - B= B_ENV_EXTRAWHITE - - - BB_= PRESERVE_ENV - - - BB_ORIGE= NV - - - BITBAKE_= UI - - - The first four variables in this list relate to how BitBake tr= eats shell - environment variables during task execution. - By default, BitBake cleans the environment variables and provi= des tight - control over the shell execution environment. - However, through the use of these first four variables, you can - apply your control regarding the - environment variables allowed to be used by BitBake in the she= ll - during execution of tasks. - See the - "Passing Information Into the Build Task Environment" - section and the information about these variables in the - variable glossary for more information on how they work and - on how to use them. - - - - The base configuration metadata is global - and therefore affects all recipes and tasks that are executed. - - - - BitBake first searches the current working directory for an - optional conf/bblayers.conf configuration= file. - This file is expected to contain a - BBLAYERS - variable that is a space-delimited list of 'layer' directories. - Recall that if BitBake cannot find a bblayers.conf - file, then it is assumed the user has set the BBPATH= - and BBFILES variables directly in the env= ironment. - - - - For each directory (layer) in this list, a conf/laye= r.conf - file is located and parsed with the - LAYERDIR - variable being set to the directory where the layer was found. - The idea is these files automatically set up - BBPATH - and other variables correctly for a given build directory. - - - - BitBake then expects to find the conf/bitbake.conf - file somewhere in the user-specified BBPATH. - That configuration file generally has include directives to pu= ll - in any other metadata such as files specific to the architectu= re, - the machine, the local environment, and so forth. - - - - Only variable definitions and include directives are allowed - in BitBake .conf files. - Some variables directly influence BitBake's behavior. - These variables might have been set from the environment - depending on the environment variables previously - mentioned or set in the configuration files. - The - "Variables Glossary" - chapter presents a full list of variables. - - - - After parsing configuration files, BitBake uses its rudimentary - inheritance mechanism, which is through class files, to inherit - some standard classes. - BitBake parses a class when the inherit directive responsible - for getting that class is encountered. - - - - The base.bbclass file is always included. - Other classes that are specified in the configuration using the - INHERIT<= /link> - variable are also included. - BitBake searches for class files in a - classes subdirectory under - the paths in BBPATH in the same way as - configuration files. - - - - A good way to get an idea of the configuration files and - the class files used in your execution environment is to - run the following BitBake command: - - $ bitbake -e > mybb.log - - Examining the top of the mybb.log - shows you the many configuration files and class files - used in your execution environment. - - - - - You need to be aware of how BitBake parses curly braces. - If a recipe uses a closing curly brace within the function= and - the character has no leading spaces, BitBake produces a pa= rsing - error. - If you use a pair of curly braces in a shell function, the - closing curly brace must not be located at the start of th= e line - without leading spaces. - - - - Here is an example that causes BitBake to produce a parsing - error: - - fakeroot create_shar() { - cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh - usage() - { - echo "test" - ###### The following "}" at the start of the line causes a parsing = error ###### - } - EOF - } - - Writing the recipe this way avoids the error: - - fakeroot create_shar() { - cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh - usage() - { - echo "test" - ######The following "}" with a leading space at the start of the li= ne avoids the error ###### - } - EOF - } - - - -
    - -
    - Locating and Parsing Recipes - - - During the configuration phase, BitBake will have set - BBFILES<= /link>. - BitBake now uses it to construct a list of recipes to parse, - along with any append files (.bbappend) - to apply. - BBFILES is a space-separated list of - available files and supports wildcards. - An example would be: - - BBFILES =3D "/path/to/bbfiles/*.bb /path/to/appends/*.bbappend" - - BitBake parses each recipe and append file located - with BBFILES and stores the values of - various variables into the datastore. - - Append files are applied in the order they are encountered= in - BBFILES. - - For each file, a fresh copy of the base configuration is - made, then the recipe is parsed line by line. - Any inherit statements cause BitBake to find and - then parse class files (.bbclass) - using - BBPATH - as the search path. - Finally, BitBake parses in order any append files found in - BBFILES. - - - - One common convention is to use the recipe filename to define - pieces of metadata. - For example, in bitbake.conf the recipe - name and version are used to set the variables - PN and - PV: - - PN =3D "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),= d)[0] or 'defaultpkgname'}" - PV =3D "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),= d)[1] or '1.0'}" - - In this example, a recipe called "something_1.2.3.bb" would set - PN to "something" and - PV to "1.2.3". - - - - By the time parsing is complete for a recipe, BitBake - has a list of tasks that the recipe defines and a set of - data consisting of keys and values as well as - dependency information about the tasks. - - - - BitBake does not need all of this information. - It only needs a small subset of the information to make - decisions about the recipe. - Consequently, BitBake caches the values in which it is - interested and does not store the rest of the information. - Experience has shown it is faster to re-parse the metadata tha= n to - try and write it out to the disk and then reload it. - - - - Where possible, subsequent BitBake commands reuse this cache of - recipe information. - The validity of this cache is determined by first computing a - checksum of the base configuration data (see - BB_= HASHCONFIG_WHITELIST) - and then checking if the checksum matches. - If that checksum matches what is in the cache and the recipe - and class files have not changed, BitBake is able to use - the cache. - BitBake then reloads the cached information about the recipe - instead of reparsing it from scratch. - - - - Recipe file collections exist to allow the user to - have multiple repositories of - .bb files that contain the same - exact package. - For example, one could easily use them to make one's - own local copy of an upstream repository, but with - custom modifications that one does not want upstream. - Here is an example: - - BBFILES =3D "/stuff/openembedded/*/*.bb /stuff/openembedded.modified/*= /*.bb" - BBFILE_COLLECTIONS =3D "upstream local" - BBFILE_PATTERN_upstream =3D "^/stuff/openembedded/" - BBFILE_PATTERN_local =3D "^/stuff/openembedded.modified/" - BBFILE_PRIORITY_upstream =3D "5" - BBFILE_PRIORITY_local =3D "10" - - - The layers mechanism is now the preferred method of collec= ting - code. - While the collections code remains, its main use is to set= layer - priorities and to deal with overlap (conflicts) between la= yers. - - -
    - -
    - Providers - - - Assuming BitBake has been instructed to execute a target - and that all the recipe files have been parsed, BitBake - starts to figure out how to build the target. - BitBake looks through the PROVIDES list - for each of the recipes. - A PROVIDES list is the list of names by w= hich - the recipe can be known. - Each recipe's PROVIDES list is created - implicitly through the recipe's - PN var= iable - and explicitly through the recipe's - PROVIDES - variable, which is optional. - - - - When a recipe uses PROVIDES, that recipe's - functionality can be found under an alternative name or names = other - than the implicit PN name. - As an example, suppose a recipe named keyboard_1.0.b= b - contained the following: - - PROVIDES +=3D "fullkeyboard" - - The PROVIDES list for this recipe becomes - "keyboard", which is implicit, and "fullkeyboard", which is ex= plicit. - Consequently, the functionality found in - keyboard_1.0.bb can be found under two - different names. - -
    - -
    - Preferences - - - The PROVIDES list is only part of the sol= ution - for figuring out a target's recipes. - Because targets might have multiple providers, BitBake needs - to prioritize providers by determining provider preferences. - - - - A common example in which a target has multiple providers - is "virtual/kernel", which is on the - PROVIDES list for each kernel recipe. - Each machine often selects the best kernel provider by using a - line similar to the following in the machine configuration fil= e: - - PREFERRED_PROVIDER_virtual/kernel =3D "linux-yocto" - - The default - PREFERRE= D_PROVIDER - is the provider with the same name as the target. - BitBake iterates through each target it needs to build and - resolves them and their dependencies using this process. - - - - Understanding how providers are chosen is made complicated by = the fact - that multiple versions might exist for a given provider. - BitBake defaults to the highest version of a provider. - Version comparisons are made using the same method as Debian. - You can use the - PREFERRED= _VERSION - variable to specify a particular version. - You can influence the order by using the - DEFAULT_= PREFERENCE - variable. - - - - By default, files have a preference of "0". - Setting DEFAULT_PREFERENCE to "-1" makes = the - recipe unlikely to be used unless it is explicitly referenced. - Setting DEFAULT_PREFERENCE to "1" makes it - likely the recipe is used. - PREFERRED_VERSION overrides any - DEFAULT_PREFERENCE setting. - DEFAULT_PREFERENCE is often used to mark = newer - and more experimental recipe versions until they have undergone - sufficient testing to be considered stable. - - - - When there are multiple =E2=80=9Cversions=E2=80=9D of a given = recipe, - BitBake defaults to selecting the most recent - version, unless otherwise specified. - If the recipe in question has a - DEFAULT_= PREFERENCE - set lower than the other recipes (default is 0), then - it will not be selected. - This allows the person or persons maintaining - the repository of recipe files to specify - their preference for the default selected version. - Additionally, the user can specify their preferred version. - - - - If the first recipe is named a_1.1.bb, th= en the - PN var= iable - will be set to =E2=80=9Ca=E2=80=9D, and the - PV - variable will be set to 1.1. - - - - Thus, if a recipe named a_1.2.bb exists, = BitBake - will choose 1.2 by default. - However, if you define the following variable in a - .conf file that BitBake parses, you - can change that preference: - - PREFERRED_VERSION_a =3D "1.1" - - - - - - It is common for a recipe to provide two versions -- a sta= ble, - numbered (and preferred) version, and a version that is - automatically checked out from a source code repository th= at - is considered more "bleeding edge" but can be selected only - explicitly. - - - - For example, in the OpenEmbedded codebase, there is a stan= dard, - versioned recipe file for BusyBox, - busybox_1.22.1.bb, - but there is also a Git-based version, - busybox_git.bb, which explicitly cont= ains the line - - DEFAULT_PREFERENCE =3D "-1" - - to ensure that the numbered, stable version is always pref= erred - unless the developer selects otherwise. - - -
    - -
    - Dependencies - - - Each target BitBake builds consists of multiple tasks such as - fetch, unpack, - patch, configure, - and compile. - For best performance on multi-core systems, BitBake considers = each - task as an independent - entity with its own set of dependencies. - - - - Dependencies are defined through several variables. - You can find information about variables BitBake uses in - the Variables Glossary= - near the end of this manual. - At a basic level, it is sufficient to know that BitBake uses t= he - DEPENDS<= /link> and - RDEPENDS variables when - calculating dependencies. - - - - For more information on how BitBake handles dependencies, see = the - "Dependencies" section. - -
    - -
    - The Task List - - - Based on the generated list of providers and the dependency in= formation, - BitBake can now calculate exactly what tasks it needs to run a= nd in what - order it needs to run them. - The - "Executing Tasks" sec= tion has more - information on how BitBake chooses which task to execute next. - - - - The build now starts with BitBake forking off threads up to th= e limit set in the - BB_NUMBER= _THREADS - variable. - BitBake continues to fork threads as long as there are tasks r= eady to run, - those tasks have all their dependencies met, and the thread th= reshold has not been - exceeded. - - - - It is worth noting that you can greatly speed up the build tim= e by properly setting - the BB_NUMBER_THREADS variable. - - - - As each task completes, a timestamp is written to the director= y specified by the - STAMP variable. - On subsequent runs, BitBake looks in the build directory within - tmp/stamps and does not rerun - tasks that are already completed unless a timestamp is found t= o be invalid. - Currently, invalid timestamps are only considered on a per - recipe file basis. - So, for example, if the configure stamp has a timestamp greate= r than the - compile timestamp for a given target, then the compile task wo= uld rerun. - Running the compile task again, however, has no effect on othe= r providers - that depend on that target. - - - - The exact format of the stamps is partly configurable. - In modern versions of BitBake, a hash is appended to the - stamp so that if the configuration changes, the stamp becomes - invalid and the task is automatically rerun. - This hash, or signature used, is governed by the signature pol= icy - that is configured (see the - "Checksums (Signatures)" - section for information). - It is also possible to append extra metadata to the stamp using - the [stamp-extra-info] task flag. - For example, OpenEmbedded uses this flag to make some tasks ma= chine-specific. - - - - Some tasks are marked as "nostamp" tasks. - No timestamp file is created when these tasks are run. - Consequently, "nostamp" tasks are always rerun. - - - - For more information on tasks, see the - "Tasks" section. - -
    - -
    - Executing Tasks - - - Tasks can be either a shell task or a Python task. - For shell tasks, BitBake writes a shell script to - ${T<= /filename>}/run.do_taskname.pid= - and then executes the script. - The generated shell script contains all the exported variables, - and the shell functions with all variables expanded. - Output from the shell script goes to the file - ${T}/log.do_taskname.pid<= /filename>. - Looking at the expanded shell functions in the run file and - the output in the log files is a useful debugging technique. - - - - For Python tasks, BitBake executes the task internally and logs - information to the controlling terminal. - Future versions of BitBake will write the functions to files - similar to the way shell tasks are handled. - Logging will be handled in a way similar to shell tasks as wel= l. - - - - The order in which BitBake runs the tasks is controlled by its - task scheduler. - It is possible to configure the scheduler and define custom - implementations for specific use cases. - For more information, see these variables that control the - behavior: - - - BB_SCH= EDULER - - - BB_SC= HEDULERS - - - It is possible to have functions run before and after a task's= main - function. - This is done using the [prefuncs] - and [postfuncs] flags of the task - that lists the functions to run. - -
    - -
    - Checksums (Signatures) - - - A checksum is a unique signature of a task's inputs. - The signature of a task can be used to determine if a task - needs to be run. - Because it is a change in a task's inputs that triggers running - the task, BitBake needs to detect all the inputs to a given ta= sk. - For shell tasks, this turns out to be fairly easy because - BitBake generates a "run" shell script for each task and - it is possible to create a checksum that gives you a good idea= of when - the task's data changes. - - - - To complicate the problem, some things should not be included = in - the checksum. - First, there is the actual specific build path of a given task= - - the working directory. - It does not matter if the working directory changes because it= should not - affect the output for target packages. - The simplistic approach for excluding the working directory is= to set - it to some fixed value and create the checksum for the "run" s= cript. - BitBake goes one step better and uses the - BB_HA= SHBASE_WHITELIST - variable to define a list of variables that should never be in= cluded - when generating the signatures. - - - - Another problem results from the "run" scripts containing func= tions that - might or might not get called. - The incremental build solution contains code that figures out = dependencies - between shell functions. - This code is used to prune the "run" scripts down to the minim= um set, - thereby alleviating this problem and making the "run" scripts = much more - readable as a bonus. - - - - So far we have solutions for shell scripts. - What about Python tasks? - The same approach applies even though these tasks are more dif= ficult. - The process needs to figure out what variables a Python functi= on accesses - and what functions it calls. - Again, the incremental build solution contains code that first= figures out - the variable and function dependencies, and then creates a che= cksum for the data - used as the input to the task. - - - - Like the working directory case, situations exist where depend= encies - should be ignored. - For these cases, you can instruct the build process to ignore = a dependency - by using a line like the following: - - PACKAGE_ARCHS[vardepsexclude] =3D "MACHINE" - - This example ensures that the PACKAGE_ARCHS variable does not - depend on the value of MACHINE, even if i= t does reference it. - - - - Equally, there are cases where we need to add dependencies Bit= Bake - is not able to find. - You can accomplish this by using a line like the following: - - PACKAGE_ARCHS[vardeps] =3D "MACHINE" - - This example explicitly adds the MACHINE = variable as a - dependency for PACKAGE_ARCHS. - - - - Consider a case with in-line Python, for example, where BitBak= e is not - able to figure out dependencies. - When running in debug mode (i.e. using -DDD), BitBake - produces output when it discovers something for which it canno= t figure out - dependencies. - - - - Thus far, this section has limited discussion to the direct in= puts into a task. - Information based on direct inputs is referred to as the "base= hash" in the - code. - However, there is still the question of a task's indirect inpu= ts - the - things that were already built and present in the build direct= ory. - The checksum (or signature) for a particular task needs to add= the hashes - of all the tasks on which the particular task depends. - Choosing which dependencies to add is a policy decision. - However, the effect is to generate a master checksum that comb= ines the basehash - and the hashes of the task's dependencies. - - - - At the code level, there are a variety of ways both the baseha= sh and the - dependent task hashes can be influenced. - Within the BitBake configuration file, we can give BitBake som= e extra information - to help it construct the basehash. - The following statement effectively results in a list of globa= l variable - dependency excludes - variables never included in any checksum. - This example uses variables from OpenEmbedded to help illustra= te - the concept: - - BB_HASHBASE_WHITELIST ?=3D "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH D= L_DIR \ - SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHEL= L TERM \ - USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSER= V_HOST \ - PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \ - CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH = SDKPKGSUFFIX" - - The previous example excludes the work directory, which is par= t of - TMPDIR. - - - - The rules for deciding which hashes of dependent tasks to incl= ude through - dependency chains are more complex and are generally accomplis= hed with a - Python function. - The code in meta/lib/oe/sstatesig.py show= s two examples - of this and also illustrates how you can insert your own polic= y into the system - if so desired. - This file defines the two basic signature generators OpenEmbed= ded-Core - uses: "OEBasic" and "OEBasicHash". - By default, there is a dummy "noop" signature handler enabled = in BitBake. - This means that behavior is unchanged from previous versions. - OE-Core uses the "OEBasicHash" signature = handler by default - through this setting in the bitbake.conf = file: - - BB_SIGNATURE_HANDLER ?=3D "OEBasicHash" - - The "OEBasicHash" BB_SIGNATURE_HANDLER is= the same as the - "OEBasic" version but adds the task hash to the stamp files. - This results in any metadata change that changes the task hash= , automatically - causing the task to be run again. - This removes the need to bump - PR - values, and changes to metadata automatically ripple across th= e build. - - - - It is also worth noting that the end result of these signature= generators is to - make some dependency and hash information available to the bui= ld. - This information includes: - - BB_BASEHASH_task-taskname: - The base hashes for each task in the recipe. - - BB_BASEHASH_filename:taskname: - The base hashes for each dependent task. - - BBHASHDEPS_filename:taskname: - The task dependencies for each task. - - BB_TASKHASH: - The hash of the currently running task. - - - - - - It is worth noting that BitBake's "-S" option lets you - debug BitBake's processing of signatures. - The options passed to -S allow different debugging modes - to be used, either using BitBake's own debug functions - or possibly those defined in the metadata/signature handler - itself. - The simplest parameter to pass is "none", which causes a - set of signature information to be written out into - STAMPS_DIR - corresponding to the targets specified. - The other currently available parameter is "printdiff", - which causes BitBake to try to establish the closest - signature match it can (e.g. in the sstate cache) and then - run bitbake-diffsigs over the matches - to determine the stamps and delta where these two - stamp trees diverge. - - It is likely that future versions of BitBake will - provide other signature handlers triggered through - additional "-S" parameters. - - - - - You can find more information on checksum metadata in the - "Task Checksums = and Setscene" - section. - -
    - -
    - Setscene - - - The setscene process enables BitBake to handle "pre-built" art= ifacts. - The ability to handle and reuse these artifacts allows BitBake - the luxury of not having to build something from scratch every= time. - Instead, BitBake can use, when possible, existing build artifa= cts. - - - - BitBake needs to have reliable data indicating whether or not = an - artifact is compatible. - Signatures, described in the previous section, provide an ideal - way of representing whether an artifact is compatible. - If a signature is the same, an object can be reused. - - - - If an object can be reused, the problem then becomes how to - replace a given task or set of tasks with the pre-built artifa= ct. - BitBake solves the problem with the "setscene" process. - - - - When BitBake is asked to build a given target, before building= anything, - it first asks whether cached information is available for any = of the - targets it's building, or any of the intermediate targets. - If cached information is available, BitBake uses this informat= ion instead of - running the main tasks. - - - - BitBake first calls the function defined by the - BB_HA= SHCHECK_FUNCTION - variable with a list of tasks and corresponding - hashes it wants to build. - This function is designed to be fast and returns a list - of the tasks for which it believes in can obtain artifacts. - - - - Next, for each of the tasks that were returned as possibilitie= s, - BitBake executes a setscene version of the task that the possi= ble - artifact covers. - Setscene versions of a task have the string "_setscene" append= ed to the - task name. - So, for example, the task with the name xxx has - a setscene task named xxx_setscene. - The setscene version of the task executes and provides the nec= essary - artifacts returning either success or failure. - - - - As previously mentioned, an artifact can cover more than one t= ask. - For example, it is pointless to obtain a compiler if you - already have the compiled binary. - To handle this, BitBake calls the - BB_SET= SCENE_DEPVALID - function for each successful setscene task to know whether or = not it needs - to obtain the dependencies of that task. - - - - Finally, after all the setscene tasks have executed, BitBake c= alls the - function listed in - BB_SETSCENE_VERIFY_FUNCTION2 - with the list of tasks BitBake thinks has been "covered". - The metadata can then ensure that this list is correct and can - inform BitBake that it wants specific tasks to be run regardle= ss - of the setscene result. - - - - You can find more information on setscene metadata in the - "Task Checksums = and Setscene" - section. - -
    - -
    - Logging - - In addition to the standard command line option to control how - verbose builds are when execute, bitbake also supports user de= fined - configuration of the - = Python logging - facilities through the - BB_LOGCONFIG - variable. This variable defines a json or yaml - logging configuration - that will be intelligently merged into the default configurati= on. - The logging configuration is merged using the following rules: - - - The user defined configuration will completely replace= the default - configuration if top level key - bitbake_merge is set to the value - False. In this case, all other ru= les - are ignored. - - - The user configuration must have a top level - version which must match the valu= e of - the default configuration. - - - Any keys defined in the handlers, - formatters, or filters<= /filename>, - will be merged into the same section in the default - configuration, with the user specified keys taking - replacing a default one if there is a conflict. In - practice, this means that if both the default configur= ation - and user configuration specify a handler named - myhandler, the user defined one w= ill - replace the default. To prevent the user from inadvert= ently - replacing a default handler, formatter, or filter, all= of - the default ones are named with a prefix of - "BitBake." - - - If a logger is defined by the user with the key - bitbake_merge set to - False, that logger will be comple= tely - replaced by user configuration. In this case, no other - rules will apply to that logger. - - - All user defined filter and - handlers properties for a given l= ogger - will be merged with corresponding properties from the - default logger. For example, if the user configuration= adds - a filter called myFilter to the - BitBake.SigGen, and the default - configuration adds a filter called - BitBake.defaultFilter, both filte= rs - will be applied to the logger - - - - - - As an example, consider the following user logging configurati= on - file which logs all Hash Equivalence related messages of VERBO= SE or - higher to a file called hashequiv.log - - { - "version": 1, - "handlers": { - "autobuilderlog": { - "class": "logging.FileHandler", - "formatter": "logfileFormatter", - "level": "DEBUG", - "filename": "hashequiv.log", - "mode": "w" - } - }, - "formatters": { - "logfileFormatter": { - "format": "%(name)s: %(levelname)s: %(message)s" - } - }, - "loggers": { - "BitBake.SigGen.HashEquiv": { - "level": "VERBOSE", - "handlers": ["autobuilderlog"] - }, - "BitBake.RunQueue.HashEquiv": { - "level": "VERBOSE", - "handlers": ["autobuilderlog"] - } - } - } - - -
    -     diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.r= st b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst new file mode 100644 index 00000000..bd6cc0ef --- /dev/null +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst @@ -0,0 +1,689 @@ +.. SPDX-License-Identifier: CC-BY-2.5 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +File Download Support +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +| + +BitBake's fetch module is a standalone piece of library code that deals +with the intricacies of downloading source code and files from remote +systems. Fetching source code is one of the cornerstones of building +software. As such, this module forms an important part of BitBake. + +The current fetch module is called "fetch2" and refers to the fact that +it is the second major version of the API. The original version is +obsolete and has been removed from the codebase. Thus, in all cases, +"fetch" refers to "fetch2" in this manual. + +The Download (Fetch) +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +BitBake takes several steps when fetching source code or files. The +fetcher codebase deals with two distinct processes in order: obtaining +the files from somewhere (cached or otherwise) and then unpacking those +files into a specific location and perhaps in a specific way. Getting +and unpacking the files is often optionally followed by patching. +Patching, however, is not covered by this module. + +The code to execute the first part of this process, a fetch, looks +something like the following: :: + + src_uri =3D (d.getVar('SRC_URI') or "").split() + fetcher =3D bb.fetch2.Fetch(src_uri, d) + fetcher.download() + +This code sets up an instance of the fetch class. The instance uses a +space-separated list of URLs from the :term:`SRC_URI` +variable and then calls the ``download`` method to download the files. + +The instantiation of the fetch class is usually followed by: :: + + rootdir =3D l.getVar('WORKDIR') + fetcher.unpack(rootdir) + +This code unpacks the downloaded files to the specified by ``WORKDIR``. + +.. note:: + + For convenience, the naming in these examples matches the variables + used by OpenEmbedded. If you want to see the above code in action, + examine the OpenEmbedded class file ``base.bbclass`` + . + +The ``SRC_URI`` and ``WORKDIR`` variables are not hardcoded into the +fetcher, since those fetcher methods can be (and are) called with +different variable names. In OpenEmbedded for example, the shared state +(sstate) code uses the fetch module to fetch the sstate files. + +When the ``download()`` method is called, BitBake tries to resolve the +URLs by looking for source files in a specific search order: + +- *Pre-mirror Sites:* BitBake first uses pre-mirrors to try and find + source files. These locations are defined using the + :term:`PREMIRRORS` variable. + +- *Source URI:* If pre-mirrors fail, BitBake uses the original URL (e.g + from ``SRC_URI``). + +- *Mirror Sites:* If fetch failures occur, BitBake next uses mirror + locations as defined by the :term:`MIRRORS` variable. + +For each URL passed to the fetcher, the fetcher calls the submodule that +handles that particular URL type. This behavior can be the source of +some confusion when you are providing URLs for the ``SRC_URI`` variable. +Consider the following two URLs: :: + + http://git.yoctoproject.org/git/poky;protocol=3Dgit + git://git.yoctoproject.org/git/poky;protocol=3Dhttp + +In the former case, the URL is passed to the ``wget`` fetcher, which does = not +understand "git". Therefore, the latter case is the correct form since the= Git +fetcher does know how to use HTTP as a transport. + +Here are some examples that show commonly used mirror definitions: :: + + PREMIRRORS ?=3D "\ + bzr://.*/.\* http://somemirror.org/sources/ \\n \ + cvs://.*/.\* http://somemirror.org/sources/ \\n \ + git://.*/.\* http://somemirror.org/sources/ \\n \ + hg://.*/.\* http://somemirror.org/sources/ \\n \ + osc://.*/.\* http://somemirror.org/sources/ \\n \ + p4://.*/.\* http://somemirror.org/sources/ \\n \ + svn://.*/.\* http://somemirror.org/sources/ \\n" + + MIRRORS =3D+ "\ + ftp://.*/.\* http://somemirror.org/sources/ \\n \ + http://.*/.\* http://somemirror.org/sources/ \\n \ + https://.*/.\* http://somemirror.org/sources/ \\n" + +It is useful to note that BitBake +supports cross-URLs. It is possible to mirror a Git repository on an +HTTP server as a tarball. This is what the ``git://`` mapping in the +previous example does. + +Since network accesses are slow, BitBake maintains a cache of files +downloaded from the network. Any source files that are not local (i.e. +downloaded from the Internet) are placed into the download directory, +which is specified by the :term:`DL_DIR` variable. + +File integrity is of key importance for reproducing builds. For +non-local archive downloads, the fetcher code can verify SHA-256 and MD5 +checksums to ensure the archives have been downloaded correctly. You can +specify these checksums by using the ``SRC_URI`` variable with the +appropriate varflags as follows: :: + + SRC_URI[md5sum] =3D "value" + SRC_URI[sha256sum] =3D "value" + +You can also specify the checksums as +parameters on the ``SRC_URI`` as shown below: :: + + SRC_URI =3D "http://example.com/foobar.tar.bz2;md5sum=3D4a8e0f237e961fd7= 785d19d07fdb994d" + +If multiple URIs exist, you can specify the checksums either directly as +in the previous example, or you can name the URLs. The following syntax +shows how you name the URIs: :: + + SRC_URI =3D "http://example.com/foobar.tar.bz2;name=3Dfoo" + SRC_URI[foo.md5sum] =3D 4a8e0f237e961fd7785d19d07fdb994d + +After a file has been downloaded and +has had its checksum checked, a ".done" stamp is placed in ``DL_DIR``. +BitBake uses this stamp during subsequent builds to avoid downloading or +comparing a checksum for the file again. + +.. note:: + + It is assumed that local storage is safe from data corruption. If + this were not the case, there would be bigger issues to worry about. + +If :term:`BB_STRICT_CHECKSUM` is set, any +download without a checksum triggers an error message. The +:term:`BB_NO_NETWORK` variable can be used to +make any attempted network access a fatal error, which is useful for +checking that mirrors are complete as well as other things. + +.. _bb-the-unpack: + +The Unpack +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +The unpack process usually immediately follows the download. For all +URLs except Git URLs, BitBake uses the common ``unpack`` method. + +A number of parameters exist that you can specify within the URL to +govern the behavior of the unpack stage: + +- *unpack:* Controls whether the URL components are unpacked. If set to + "1", which is the default, the components are unpacked. If set to + "0", the unpack stage leaves the file alone. This parameter is useful + when you want an archive to be copied in and not be unpacked. + +- *dos:* Applies to ``.zip`` and ``.jar`` files and specifies whether + to use DOS line ending conversion on text files. + +- *basepath:* Instructs the unpack stage to strip the specified + directories from the source path when unpacking. + +- *subdir:* Unpacks the specific URL to the specified subdirectory + within the root directory. + +The unpack call automatically decompresses and extracts files with ".Z", +".z", ".gz", ".xz", ".zip", ".jar", ".ipk", ".rpm". ".srpm", ".deb" and +".bz2" extensions as well as various combinations of tarball extensions. + +As mentioned, the Git fetcher has its own unpack method that is +optimized to work with Git trees. Basically, this method works by +cloning the tree into the final directory. The process is completed +using references so that there is only one central copy of the Git +metadata needed. + +.. _bb-fetchers: + +Fetchers +=3D=3D=3D=3D=3D=3D=3D=3D + +As mentioned earlier, the URL prefix determines which fetcher submodule +BitBake uses. Each submodule can support different URL parameters, which +are described in the following sections. + +.. _local-file-fetcher: + +Local file fetcher (``file://``) +-------------------------------- + +This submodule handles URLs that begin with ``file://``. The filename +you specify within the URL can be either an absolute or relative path to +a file. If the filename is relative, the contents of the +:term:`FILESPATH` variable is used in the same way +``PATH`` is used to find executables. If the file cannot be found, it is +assumed that it is available in :term:`DL_DIR` by the +time the ``download()`` method is called. + +If you specify a directory, the entire directory is unpacked. + +Here are a couple of example URLs, the first relative and the second +absolute: :: + + SRC_URI =3D "file://relativefile.patch" + SRC_URI =3D "file:///Users/ich/very_important_software" + +.. _http-ftp-fetcher: + +HTTP/FTP wget fetcher (``http://``, ``ftp://``, ``https://``) +------------------------------------------------------------- + +This fetcher obtains files from web and FTP servers. Internally, the +fetcher uses the wget utility. + +The executable and parameters used are specified by the +``FETCHCMD_wget`` variable, which defaults to sensible values. The +fetcher supports a parameter "downloadfilename" that allows the name of +the downloaded file to be specified. Specifying the name of the +downloaded file is useful for avoiding collisions in +:term:`DL_DIR` when dealing with multiple files that +have the same name. + +Some example URLs are as follows: :: + + SRC_URI =3D "http://oe.handhelds.org/not_there.aac" + SRC_URI =3D "ftp://oe.handhelds.org/not_there_as_well.aac" + SRC_URI =3D "ftp://you@oe.handhelds.org/home/you/secret.plan" + +.. note:: + + Because URL parameters are delimited by semi-colons, this can + introduce ambiguity when parsing URLs that also contain semi-colons, + for example: + :: + + SRC_URI =3D "http://abc123.org/git/?p=3Dgcc/gcc.git;a=3Dsnapsho= t;h=3Da5dd47" + + + Such URLs should should be modified by replacing semi-colons with '&' + characters: + :: + + SRC_URI =3D "http://abc123.org/git/?p=3Dgcc/gcc.git&a=3Dsnapsho= t&h=3Da5dd47" + + + In most cases this should work. Treating semi-colons and '&' in + queries identically is recommended by the World Wide Web Consortium + (W3C). Note that due to the nature of the URL, you may have to + specify the name of the downloaded file as well: + :: + + SRC_URI =3D "http://abc123.org/git/?p=3Dgcc/gcc.git&a=3Dsnapsho= t&h=3Da5dd47;downloadfilename=3Dmyfile.bz2" + + +.. _cvs-fetcher: + +CVS fetcher (``(cvs://``) +------------------------- + +This submodule handles checking out files from the CVS version control +system. You can configure it using a number of different variables: + +- :term:`FETCHCMD_cvs `: The name of the executable to use when= running + the ``cvs`` command. This name is usually "cvs". + +- :term:`SRCDATE`: The date to use when fetching the CVS source code. A + special value of "now" causes the checkout to be updated on every + build. + +- :term:`CVSDIR`: Specifies where a temporary + checkout is saved. The location is often ``DL_DIR/cvs``. + +- CVS_PROXY_HOST: The name to use as a "proxy=3D" parameter to the + ``cvs`` command. + +- CVS_PROXY_PORT: The port number to use as a "proxyport=3D" + parameter to the ``cvs`` command. + +As well as the standard username and password URL syntax, you can also +configure the fetcher with various URL parameters: + +The supported parameters are as follows: + +- *"method":* The protocol over which to communicate with the CVS + server. By default, this protocol is "pserver". If "method" is set to + "ext", BitBake examines the "rsh" parameter and sets ``CVS_RSH``. You + can use "dir" for local directories. + +- *"module":* Specifies the module to check out. You must supply this + parameter. + +- *"tag":* Describes which CVS TAG should be used for the checkout. By + default, the TAG is empty. + +- *"date":* Specifies a date. If no "date" is specified, the + :term:`SRCDATE` of the configuration is used to + checkout a specific date. The special value of "now" causes the + checkout to be updated on every build. + +- *"localdir":* Used to rename the module. Effectively, you are + renaming the output directory to which the module is unpacked. You + are forcing the module into a special directory relative to + :term:`CVSDIR`. + +- *"rsh":* Used in conjunction with the "method" parameter. + +- *"scmdata":* Causes the CVS metadata to be maintained in the tarball + the fetcher creates when set to "keep". The tarball is expanded into + the work directory. By default, the CVS metadata is removed. + +- *"fullpath":* Controls whether the resulting checkout is at the + module level, which is the default, or is at deeper paths. + +- *"norecurse":* Causes the fetcher to only checkout the specified + directory with no recurse into any subdirectories. + +- *"port":* The port to which the CVS server connects. + +Some example URLs are as follows: :: + + SRC_URI =3D "cvs://CVSROOT;module=3Dmymodule;tag=3Dsome-version;method= =3Dext" + SRC_URI =3D "cvs://CVSROOT;module=3Dmymodule;date=3D20060126;localdir= =3Dusethat" + +.. _svn-fetcher: + +Subversion (SVN) Fetcher (``svn://``) +------------------------------------- + +This fetcher submodule fetches code from the Subversion source control +system. The executable used is specified by ``FETCHCMD_svn``, which +defaults to "svn". The fetcher's temporary working directory is set by +:term:`SVNDIR`, which is usually ``DL_DIR/svn``. + +The supported parameters are as follows: + +- *"module":* The name of the svn module to checkout. You must provide + this parameter. You can think of this parameter as the top-level + directory of the repository data you want. + +- *"path_spec":* A specific directory in which to checkout the + specified svn module. + +- *"protocol":* The protocol to use, which defaults to "svn". If + "protocol" is set to "svn+ssh", the "ssh" parameter is also used. + +- *"rev":* The revision of the source code to checkout. + +- *"scmdata":* Causes the ".svn" directories to be available during + compile-time when set to "keep". By default, these directories are + removed. + +- *"ssh":* An optional parameter used when "protocol" is set to + "svn+ssh". You can use this parameter to specify the ssh program used + by svn. + +- *"transportuser":* When required, sets the username for the + transport. By default, this parameter is empty. The transport + username is different than the username used in the main URL, which + is passed to the subversion command. + +Following are three examples using svn: :: + + SRC_URI =3D "svn://myrepos/proj1;module=3Dvip;protocol=3Dhttp;rev=3D667" + SRC_URI =3D "svn://myrepos/proj1;module=3Dopie;protocol=3Dsvn+ssh" + SRC_URI =3D "svn://myrepos/proj1;module=3Dtrunk;protocol=3Dhttp;path_sp= ec=3D${MY_DIR}/proj1" + +.. _git-fetcher: + +Git Fetcher (``git://``) +------------------------ + +This fetcher submodule fetches code from the Git source control system. +The fetcher works by creating a bare clone of the remote into +:term:`GITDIR`, which is usually ``DL_DIR/git2``. This +bare clone is then cloned into the work directory during the unpack +stage when a specific tree is checked out. This is done using alternates +and by reference to minimize the amount of duplicate data on the disk +and make the unpack process fast. The executable used can be set with +``FETCHCMD_git``. + +This fetcher supports the following parameters: + +- *"protocol":* The protocol used to fetch the files. The default is + "git" when a hostname is set. If a hostname is not set, the Git + protocol is "file". You can also use "http", "https", "ssh" and + "rsync". + +- *"nocheckout":* Tells the fetcher to not checkout source code when + unpacking when set to "1". Set this option for the URL where there is + a custom routine to checkout code. The default is "0". + +- *"rebaseable":* Indicates that the upstream Git repository can be + rebased. You should set this parameter to "1" if revisions can become + detached from branches. In this case, the source mirror tarball is + done per revision, which has a loss of efficiency. Rebasing the + upstream Git repository could cause the current revision to disappear + from the upstream repository. This option reminds the fetcher to + preserve the local cache carefully for future use. The default value + for this parameter is "0". + +- *"nobranch":* Tells the fetcher to not check the SHA validation for + the branch when set to "1". The default is "0". Set this option for + the recipe that refers to the commit that is valid for a tag instead + of the branch. + +- *"bareclone":* Tells the fetcher to clone a bare clone into the + destination directory without checking out a working tree. Only the + raw Git metadata is provided. This parameter implies the "nocheckout" + parameter as well. + +- *"branch":* The branch(es) of the Git tree to clone. If unset, this + is assumed to be "master". The number of branch parameters much match + the number of name parameters. + +- *"rev":* The revision to use for the checkout. The default is + "master". + +- *"tag":* Specifies a tag to use for the checkout. To correctly + resolve tags, BitBake must access the network. For that reason, tags + are often not used. As far as Git is concerned, the "tag" parameter + behaves effectively the same as the "rev" parameter. + +- *"subpath":* Limits the checkout to a specific subpath of the tree. + By default, the whole tree is checked out. + +- *"destsuffix":* The name of the path in which to place the checkout. + By default, the path is ``git/``. + +- *"usehead":* Enables local ``git://`` URLs to use the current branch + HEAD as the revision for use with ``AUTOREV``. The "usehead" + parameter implies no branch and only works when the transfer protocol + is ``file://``. + +Here are some example URLs: :: + + SRC_URI =3D "git://git.oe.handhelds.org/git/vip.git;tag=3Dversion-1" + SRC_URI =3D "git://git.oe.handhelds.org/git/vip.git;protocol=3Dhttp" + +.. note:: + + Specifying passwords directly in ``git://`` urls is not supported. + There are several reasons: ``SRC_URI`` is often written out to logs and + other places, and that could easily leak passwords; it is also all too + easy to share metadata without removing passwords. SSH keys, ``~/.netrc= `` + and ``~/.ssh/config`` files can be used as alternatives. + + +.. _gitsm-fetcher: + +Git Submodule Fetcher (``gitsm://``) +------------------------------------ + +This fetcher submodule inherits from the :ref:`Git +fetcher` and extends that fetcher's behavior by fetching a +repository's submodules. :term:`SRC_URI` is passed to the Git fetcher as +described in the :ref:`bitbake-user-manual/bitbake-user-manual-fetching:git +fetcher (\`\`git://\`\`)` section. + +.. note:: + + You must clean a recipe when switching between '``git://``' and + '``gitsm://``' URLs. + + The Git Submodules fetcher is not a complete fetcher implementation. + The fetcher has known issues where it does not use the normal source + mirroring infrastructure properly. Further, the submodule sources it + fetches are not visible to the licensing and source archiving + infrastructures. + +.. _clearcase-fetcher: + +ClearCase Fetcher (``ccrc://``) +------------------------------- + +This fetcher submodule fetches code from a +`ClearCase `__ +repository. + +To use this fetcher, make sure your recipe has proper +:term:`SRC_URI`, :term:`SRCREV`, and +:term:`PV` settings. Here is an example: :: + + SRC_URI =3D "ccrc://cc.example.org/ccrc;vob=3D/example_vob;module=3D/ex= ample_module" + SRCREV =3D "EXAMPLE_CLEARCASE_TAG" + PV =3D "${@d.getVar("SRCREV", False).replace("/", "+")}" + +The fetcher uses the ``rcleartool`` or +``cleartool`` remote client, depending on which one is available. + +Following are options for the ``SRC_URI`` statement: + +- *vob*: The name, which must include the prepending "/" character, + of the ClearCase VOB. This option is required. + +- *module*: The module, which must include the prepending "/" + character, in the selected VOB. + + .. note:: + + The module and vob options are combined to create the load rule in t= he + view config spec. As an example, consider the vob and module values = from + the SRC_URI statement at the start of this section. Combining those = values + results in the following: :: + + load /example_vob/example_module + +- *proto*: The protocol, which can be either ``http`` or ``https``. + +By default, the fetcher creates a configuration specification. If you +want this specification written to an area other than the default, use +the ``CCASE_CUSTOM_CONFIG_SPEC`` variable in your recipe to define where +the specification is written. + +.. note:: + + the SRCREV loses its functionality if you specify this variable. Howeve= r, + SRCREV is still used to label the archive after a fetch even though it = does + not define what is fetched. + +Here are a couple of other behaviors worth mentioning: + +- When using ``cleartool``, the login of ``cleartool`` is handled by + the system. The login require no special steps. + +- In order to use ``rcleartool`` with authenticated users, an + "rcleartool login" is necessary before using the fetcher. + +.. _perforce-fetcher: + +Perforce Fetcher (``p4://``) +---------------------------- + +This fetcher submodule fetches code from the +`Perforce `__ source control system. The +executable used is specified by ``FETCHCMD_p4``, which defaults to "p4". +The fetcher's temporary working directory is set by +:term:`P4DIR`, which defaults to "DL_DIR/p4". +The fetcher does not make use of a perforce client, instead it +relies on ``p4 files`` to retrieve a list of +files and ``p4 print`` to transfer the content +of those files locally. + +To use this fetcher, make sure your recipe has proper +:term:`SRC_URI`, :term:`SRCREV`, and +:term:`PV` values. The p4 executable is able to use the +config file defined by your system's ``P4CONFIG`` environment variable +in order to define the Perforce server URL and port, username, and +password if you do not wish to keep those values in a recipe itself. If +you choose not to use ``P4CONFIG``, or to explicitly set variables that +``P4CONFIG`` can contain, you can specify the ``P4PORT`` value, which is +the server's URL and port number, and you can specify a username and +password directly in your recipe within ``SRC_URI``. + +Here is an example that relies on ``P4CONFIG`` to specify the server URL +and port, username, and password, and fetches the Head Revision: :: + + SRC_URI =3D "p4://example-depot/main/source/..." + SRCREV =3D "${AUTOREV}" + PV =3D "p4-${SRCPV}" + S =3D "${WORKDIR}/p4" + +Here is an example that specifies the server URL and port, username, and +password, and fetches a Revision based on a Label: :: + + P4PORT =3D "tcp:p4server.example.net:1666" + SRC_URI =3D "p4://user:passwd@example-depot/main/source/..." + SRCREV =3D "release-1.0" + PV =3D "p4-${SRCPV}" + S =3D "${WORKDIR}/p4" + +.. note:: + + You should always set S to "${WORKDIR}/p4" in your recipe. + +By default, the fetcher strips the depot location from the local file path= s. In +the above example, the content of ``example-depot/main/source/`` will be p= laced +in ``${WORKDIR}/p4``. For situations where preserving parts of the remote= depot +paths locally is desirable, the fetcher supports two parameters: + +- *"module":* + The top-level depot location or directory to fetch. The value of this + parameter can also point to a single file within the depot, in which c= ase + the local file path will include the module path. +- *"remotepath":* + When used with the value "``keep``", the fetcher will mirror the full = depot + paths locally for the specified location, even in combination with the + ``module`` parameter. + +Here is an example use of the the ``module`` parameter: :: + + SRC_URI =3D "p4://user:passwd@example-depot/main;module=3Dsource/..." + +In this case, the content of the top-level directory ``source/`` will be f= etched +to ``${P4DIR}``, including the directory itself. The top-level directory = will +be accesible at ``${P4DIR}/source/``. + +Here is an example use of the the ``remotepath`` parameter: :: + + SRC_URI =3D "p4://user:passwd@example-depot/main;module=3Dsource/...;re= motepath=3Dkeep" + +In this case, the content of the top-level directory ``source/`` will be f= etched +to ``${P4DIR}``, but the complete depot paths will be mirrored locally. The +top-level directory will be accessible at +``${P4DIR}/example-depot/main/source/``. + +.. _repo-fetcher: + +Repo Fetcher (``repo://``) +-------------------------- + +This fetcher submodule fetches code from ``google-repo`` source control +system. The fetcher works by initiating and syncing sources of the +repository into :term:`REPODIR`, which is usually +``${DL_DIR}/repo``. + +This fetcher supports the following parameters: + +- *"protocol":* Protocol to fetch the repository manifest (default: + git). + +- *"branch":* Branch or tag of repository to get (default: master). + +- *"manifest":* Name of the manifest file (default: ``default.xml``). + +Here are some example URLs: :: + + SRC_URI =3D "repo://REPOROOT;protocol=3Dgit;branch=3Dsome_branch;manife= st=3Dmy_manifest.xml" + SRC_URI =3D "repo://REPOROOT;protocol=3Dfile;branch=3Dsome_branch;manif= est=3Dmy_manifest.xml" + +.. _az-fetcher: + +Az Fetcher (``az://``) +-------------------------- + +This submodule fetches data from an +`Azure Storage account `_= _ , +it inherits its functionality from the HTTP wget fetcher, but modifies its +behavior to accomodate the usage of a +`Shared Access Signature (SAS) `__ +for non-public data. + +Such functionality is set by the variable: + +- :term:`AZ_SAS`: The Azure Storage Shared Access Signature provides secu= re + delegate access to resources, if this variable is set, the Az Fetcher w= ill + use it when fetching artifacts from the cloud. + +You can specify the AZ_SAS variable as shown below: :: + + AZ_SAS =3D "se=3D2021-01-01&sp=3Dr&sv=3D2018-11-09&sr=3Dc&skoid=3D&sig=3D" + +Here is an example URL: :: + + SRC_URI =3D "az://.blob.core.windows.net//" + +It can also be used when setting mirrors definitions using the :term:`PREM= IRRORS` variable. + +Other Fetchers +-------------- + +Fetch submodules also exist for the following: + +- Bazaar (``bzr://``) + +- Mercurial (``hg://``) + +- npm (``npm://``) + +- OSC (``osc://``) + +- Secure FTP (``sftp://``) + +- Secure Shell (``ssh://``) + +- Trees using Git Annex (``gitannex://``) + +No documentation currently exists for these lesser used fetcher +submodules. However, you might find the code helpful and readable. + +Auto Revisions +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +We need to document ``AUTOREV`` and ``SRCREV_FORMAT`` here. diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.x= ml b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml deleted file mode 100644 index d1bfc233..00000000 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml +++ /dev/null @@ -1,868 +0,0 @@ - - - -File Download Support - - - BitBake's fetch module is a standalone piece of library code - that deals with the intricacies of downloading source code - and files from remote systems. - Fetching source code is one of the cornerstones of building softwa= re. - As such, this module forms an important part of BitBake. - - - - The current fetch module is called "fetch2" and refers to the - fact that it is the second major version of the API. - The original version is obsolete and has been removed from the cod= ebase. - Thus, in all cases, "fetch" refers to "fetch2" in this - manual. - - -
    - The Download (Fetch) - - - BitBake takes several steps when fetching source code or files. - The fetcher codebase deals with two distinct processes in orde= r: - obtaining the files from somewhere (cached or otherwise) - and then unpacking those files into a specific location and - perhaps in a specific way. - Getting and unpacking the files is often optionally followed - by patching. - Patching, however, is not covered by this module. - - - - The code to execute the first part of this process, a fetch, - looks something like the following: - - src_uri =3D (d.getVar('SRC_URI') or "").split() - fetcher =3D bb.fetch2.Fetch(src_uri, d) - fetcher.download() - - This code sets up an instance of the fetch class. - The instance uses a space-separated list of URLs from the - SRC_URI<= /link> - variable and then calls the download - method to download the files. - - - - The instantiation of the fetch class is usually followed by: - - rootdir =3D l.getVar('WORKDIR') - fetcher.unpack(rootdir) - - This code unpacks the downloaded files to the - specified by WORKDIR. - - For convenience, the naming in these examples matches - the variables used by OpenEmbedded. - If you want to see the above code in action, examine - the OpenEmbedded class file base.bbclass. - - The SRC_URI and WORKDIR - variables are not hardcoded into the fetcher, since those fetc= her - methods can be (and are) called with different variable names. - In OpenEmbedded for example, the shared state (sstate) code us= es - the fetch module to fetch the sstate files. - - - - When the download() method is called, - BitBake tries to resolve the URLs by looking for source files - in a specific search order: - - Pre-mirror Sites: - BitBake first uses pre-mirrors to try and find source = files. - These locations are defined using the - PREMIRRO= RS - variable. - - Source URI: - If pre-mirrors fail, BitBake uses the original URL (e.= g from - SRC_URI). - - Mirror Sites: - If fetch failures occur, BitBake next uses mirror loca= tions as - defined by the - MIRRORS - variable. - - - - - - For each URL passed to the fetcher, the fetcher - calls the submodule that handles that particular URL type. - This behavior can be the source of some confusion when you - are providing URLs for the SRC_URI - variable. - Consider the following two URLs: - - http://git.yoctoproject.org/git/poky;protocol=3Dgit - git://git.yoctoproject.org/git/poky;protocol=3Dhttp - - In the former case, the URL is passed to the - wget fetcher, which does not - understand "git". - Therefore, the latter case is the correct form since the - Git fetcher does know how to use HTTP as a transport. - - - - Here are some examples that show commonly used mirror - definitions: - - PREMIRRORS ?=3D "\ - bzr://.*/.* http://somemirror.org/sources/ \n \ - cvs://.*/.* http://somemirror.org/sources/ \n \ - git://.*/.* http://somemirror.org/sources/ \n \ - hg://.*/.* http://somemirror.org/sources/ \n \ - osc://.*/.* http://somemirror.org/sources/ \n \ - p4://.*/.* http://somemirror.org/sources/ \n \ - svn://.*/.* http://somemirror.org/sources/ \n" - - MIRRORS =3D+ "\ - ftp://.*/.* http://somemirror.org/sources/ \n \ - http://.*/.* http://somemirror.org/sources/ \n \ - https://.*/.* http://somemirror.org/sources/ \n" - - It is useful to note that BitBake supports - cross-URLs. - It is possible to mirror a Git repository on an HTTP - server as a tarball. - This is what the git:// mapping in - the previous example does. - - - - Since network accesses are slow, BitBake maintains a - cache of files downloaded from the network. - Any source files that are not local (i.e. - downloaded from the Internet) are placed into the download - directory, which is specified by the - DL_DIR - variable. - - - - File integrity is of key importance for reproducing builds. - For non-local archive downloads, the fetcher code can verify - SHA-256 and MD5 checksums to ensure the archives have been - downloaded correctly. - You can specify these checksums by using the - SRC_URI variable with the appropriate - varflags as follows: - - SRC_URI[md5sum] =3D "value" - SRC_URI[sha256sum] =3D "value" - - You can also specify the checksums as parameters on the - SRC_URI as shown below: - - SRC_URI =3D "http://example.com/foobar.tar.bz2;md5sum=3D4a8e0f237e961= fd7785d19d07fdb994d" - - If multiple URIs exist, you can specify the checksums either - directly as in the previous example, or you can name the URLs. - The following syntax shows how you name the URIs: - - SRC_URI =3D "http://example.com/foobar.tar.bz2;name=3Dfoo" - SRC_URI[foo.md5sum] =3D 4a8e0f237e961fd7785d19d07fdb994d - - After a file has been downloaded and has had its checksum chec= ked, - a ".done" stamp is placed in DL_DIR. - BitBake uses this stamp during subsequent builds to avoid - downloading or comparing a checksum for the file again. - - It is assumed that local storage is safe from data corrupt= ion. - If this were not the case, there would be bigger issues to= worry about. - - - - - If - BB_STRIC= T_CHECKSUM - is set, any download without a checksum triggers an - error message. - The - BB_NO_NETWORK= - variable can be used to make any attempted network access a fa= tal - error, which is useful for checking that mirrors are complete - as well as other things. - -
    - -
    - The Unpack - - - The unpack process usually immediately follows the download. - For all URLs except Git URLs, BitBake uses the common - unpack method. - - - - A number of parameters exist that you can specify within the - URL to govern the behavior of the unpack stage: - - unpack: - Controls whether the URL components are unpacked. - If set to "1", which is the default, the components - are unpacked. - If set to "0", the unpack stage leaves the file alone. - This parameter is useful when you want an archive to be - copied in and not be unpacked. - - dos: - Applies to .zip and - .jar files and specifies whether = to - use DOS line ending conversion on text files. - - basepath: - Instructs the unpack stage to strip the specified - directories from the source path when unpacking. - - subdir: - Unpacks the specific URL to the specified subdirectory - within the root directory. - - - The unpack call automatically decompresses and extracts files - with ".Z", ".z", ".gz", ".xz", ".zip", ".jar", ".ipk", ".rpm". - ".srpm", ".deb" and ".bz2" extensions as well as various combi= nations - of tarball extensions. - - - - As mentioned, the Git fetcher has its own unpack method that - is optimized to work with Git trees. - Basically, this method works by cloning the tree into the final - directory. - The process is completed using references so that there is - only one central copy of the Git metadata needed. - -
    - -
    - Fetchers - - - As mentioned earlier, the URL prefix determines which - fetcher submodule BitBake uses. - Each submodule can support different URL parameters, - which are described in the following sections. - - -
    - Local file fetcher (<filename>file://</filename>)</titl= e> - - <para> - This submodule handles URLs that begin with - <filename>file://</filename>. - The filename you specify within the URL can be - either an absolute or relative path to a file. - If the filename is relative, the contents of the - <link linkend=3D'var-bb-FILESPATH'><filename>FILESPATH</fi= lename></link> - variable is used in the same way - <filename>PATH</filename> is used to find executables. - If the file cannot be found, it is assumed that it is avai= lable in - <link linkend=3D'var-bb-DL_DIR'><filename>DL_DIR</filename= ></link> - by the time the <filename>download()</filename> method is = called. - </para> - - <para> - If you specify a directory, the entire directory is - unpacked. - </para> - - <para> - Here are a couple of example URLs, the first relative and - the second absolute: - <literallayout class=3D'monospaced'> - SRC_URI =3D "file://relativefile.patch" - SRC_URI =3D "file:///Users/ich/very_important_software" - </literallayout> - </para> - </section> - - <section id=3D'http-ftp-fetcher'> - <title>HTTP/FTP wget fetcher (<filename>http://</filename>, <f= ilename>ftp://</filename>, <filename>https://</filename>) - - - This fetcher obtains files from web and FTP servers. - Internally, the fetcher uses the wget utility. - - - - The executable and parameters used are specified by the - FETCHCMD_wget variable, which defaults - to sensible values. - The fetcher supports a parameter "downloadfilename" that - allows the name of the downloaded file to be specified. - Specifying the name of the downloaded file is useful - for avoiding collisions in - DL_DIR - when dealing with multiple files that have the same name. - - - - Some example URLs are as follows: - - SRC_URI =3D "http://oe.handhelds.org/not_there.aac" - SRC_URI =3D "ftp://oe.handhelds.org/not_there_as_well.aac" - SRC_URI =3D "ftp://you@oe.handhelds.org/home/you/secret.plan" - - - - Because URL parameters are delimited by semi-colons, this c= an - introduce ambiguity when parsing URLs that also contain sem= i-colons, - for example: - - SRC_URI =3D "http://abc123.org/git/?p=3Dgcc/gcc.git;a=3Dsnapshot;h=3D= a5dd47" - - Such URLs should should be modified by replacing semi-colon= s with '&' characters: - - SRC_URI =3D "http://abc123.org/git/?p=3Dgcc/gcc.git&a=3Dsnapshot&= amp;h=3Da5dd47" - - In most cases this should work. Treating semi-colons and '= &' in queries - identically is recommended by the World Wide Web Consortiu= m (W3C). - Note that due to the nature of the URL, you may have to sp= ecify the name - of the downloaded file as well: - - SRC_URI =3D "http://abc123.org/git/?p=3Dgcc/gcc.git&a=3Dsnapshot&= amp;h=3Da5dd47;downloadfilename=3Dmyfile.bz2" - - -
    - -
    - CVS fetcher (<filename>(cvs://</filename>) - - - This submodule handles checking out files from the - CVS version control system. - You can configure it using a number of different variables: - - FETCHCMD_cvs: - The name of the executable to use when running - the cvs command. - This name is usually "cvs". - - SRCDATE= : - The date to use when fetching the CVS source code. - A special value of "now" causes the checkout to - be updated on every build. - - CVSDIR: - Specifies where a temporary checkout is saved. - The location is often DL_DIR/cvs. - - CVS_PROXY_HOST: - The name to use as a "proxy=3D" parameter to the - cvs command. - - CVS_PROXY_PORT: - The port number to use as a "proxyport=3D" paramet= er to - the cvs command. - - - As well as the standard username and password URL syntax, - you can also configure the fetcher with various URL parame= ters: - - - - The supported parameters are as follows: - - "method": - The protocol over which to communicate with the CVS - server. - By default, this protocol is "pserver". - If "method" is set to "ext", BitBake examines the - "rsh" parameter and sets CVS_RSH. - You can use "dir" for local directories. - - "module": - Specifies the module to check out. - You must supply this parameter. - - "tag": - Describes which CVS TAG should be used for - the checkout. - By default, the TAG is empty. - - "date": - Specifies a date. - If no "date" is specified, the - SRCDATE= - of the configuration is used to checkout a specifi= c date. - The special value of "now" causes the checkout to = be - updated on every build. - - "localdir": - Used to rename the module. - Effectively, you are renaming the output directory - to which the module is unpacked. - You are forcing the module into a special - directory relative to - CVSDIR. - - "rsh" - Used in conjunction with the "method" parameter. - - "scmdata": - Causes the CVS metadata to be maintained in the ta= rball - the fetcher creates when set to "keep". - The tarball is expanded into the work directory. - By default, the CVS metadata is removed. - - "fullpath": - Controls whether the resulting checkout is at the - module level, which is the default, or is at deeper - paths. - - "norecurse": - Causes the fetcher to only checkout the specified - directory with no recurse into any subdirectories. - - "port": - The port to which the CVS server connects. - - - Some example URLs are as follows: - - SRC_URI =3D "cvs://CVSROOT;module=3Dmymodule;tag=3Dsome-version;metho= d=3Dext" - SRC_URI =3D "cvs://CVSROOT;module=3Dmymodule;date=3D20060126;localdir= =3Dusethat" - - -
    - -
    - Subversion (SVN) Fetcher (<filename>svn://</filename>)<= /title> - - <para> - This fetcher submodule fetches code from the - Subversion source control system. - The executable used is specified by - <filename>FETCHCMD_svn</filename>, which defaults - to "svn". - The fetcher's temporary working directory is set by - <link linkend=3D'var-bb-SVNDIR'><filename>SVNDIR</filename= ></link>, - which is usually <filename>DL_DIR/svn</filename>. - </para> - - <para> - The supported parameters are as follows: - <itemizedlist> - <listitem><para><emphasis>"module":</emphasis> - The name of the svn module to checkout. - You must provide this parameter. - You can think of this parameter as the top-level - directory of the repository data you want. - </para></listitem> - <listitem><para><emphasis>"path_spec":</emphasis> - A specific directory in which to checkout the - specified svn module. - </para></listitem> - <listitem><para><emphasis>"protocol":</emphasis> - The protocol to use, which defaults to "svn". - If "protocol" is set to "svn+ssh", the "ssh" - parameter is also used. - </para></listitem> - <listitem><para><emphasis>"rev":</emphasis> - The revision of the source code to checkout. - </para></listitem> - <listitem><para><emphasis>"scmdata":</emphasis> - Causes the =E2=80=9C.svn=E2=80=9D directories to b= e available during - compile-time when set to "keep". - By default, these directories are removed. - </para></listitem> - <listitem><para><emphasis>"ssh":</emphasis> - An optional parameter used when "protocol" is set - to "svn+ssh". - You can use this parameter to specify the ssh - program used by svn. - </para></listitem> - <listitem><para><emphasis>"transportuser":</emphasis> - When required, sets the username for the transport. - By default, this parameter is empty. - The transport username is different than the usern= ame - used in the main URL, which is passed to the subve= rsion - command. - </para></listitem> - </itemizedlist> - Following are three examples using svn: - <literallayout class=3D'monospaced'> - SRC_URI =3D "svn://myrepos/proj1;module=3Dvip;protocol=3Dhttp;rev=3D6= 67" - SRC_URI =3D "svn://myrepos/proj1;module=3Dopie;protocol=3Dsvn+ssh" - SRC_URI =3D "svn://myrepos/proj1;module=3Dtrunk;protocol=3Dhttp;path_= spec=3D${MY_DIR}/proj1" - </literallayout> - </para> - </section> - - <section id=3D'git-fetcher'> - <title>Git Fetcher (<filename>git://</filename>) - - - This fetcher submodule fetches code from the Git - source control system. - The fetcher works by creating a bare clone of the - remote into - GITDIR, - which is usually DL_DIR/git2. - This bare clone is then cloned into the work directory dur= ing the - unpack stage when a specific tree is checked out. - This is done using alternates and by reference to - minimize the amount of duplicate data on the disk and - make the unpack process fast. - The executable used can be set with - FETCHCMD_git. - - - - This fetcher supports the following parameters: - - "protocol": - The protocol used to fetch the files. - The default is "git" when a hostname is set. - If a hostname is not set, the Git protocol is "fil= e". - You can also use "http", "https", "ssh" and "rsync= ". - - "nocheckout": - Tells the fetcher to not checkout source code when - unpacking when set to "1". - Set this option for the URL where there is a custom - routine to checkout code. - The default is "0". - - "rebaseable": - Indicates that the upstream Git repository can be = rebased. - You should set this parameter to "1" if - revisions can become detached from branches. - In this case, the source mirror tarball is done per - revision, which has a loss of efficiency. - Rebasing the upstream Git repository could cause t= he - current revision to disappear from the upstream re= pository. - This option reminds the fetcher to preserve the lo= cal cache - carefully for future use. - The default value for this parameter is "0". - - "nobranch": - Tells the fetcher to not check the SHA validation - for the branch when set to "1". - The default is "0". - Set this option for the recipe that refers to - the commit that is valid for a tag instead of - the branch. - - "bareclone": - Tells the fetcher to clone a bare clone into the - destination directory without checking out a worki= ng tree. - Only the raw Git metadata is provided. - This parameter implies the "nocheckout" parameter = as well. - - "branch": - The branch(es) of the Git tree to clone. - If unset, this is assumed to be "master". - The number of branch parameters much match the num= ber of - name parameters. - - "rev": - The revision to use for the checkout. - The default is "master". - - "tag": - Specifies a tag to use for the checkout. - To correctly resolve tags, BitBake must access the - network. - For that reason, tags are often not used. - As far as Git is concerned, the "tag" parameter be= haves - effectively the same as the "rev" parameter. - - "subpath": - Limits the checkout to a specific subpath of the t= ree. - By default, the whole tree is checked out. - - "destsuffix": - The name of the path in which to place the checkou= t. - By default, the path is git/. - - "usehead": - Enables local git:// URLs to = use the - current branch HEAD as the revision for use with - AUTOREV. - The "usehead" parameter implies no branch and only= works - when the transfer protocol is - file://. - - - Here are some example URLs: - - SRC_URI =3D "git://git.oe.handhelds.org/git/vip.git;tag=3Dversion-1" - SRC_URI =3D "git://git.oe.handhelds.org/git/vip.git;protocol=3Dhttp" - - -
    - -
    - Git Submodule Fetcher (<filename>gitsm://</filename>)</= title> - - <para> - This fetcher submodule inherits from the - <link linkend=3D'git-fetcher'>Git fetcher</link> and exten= ds - that fetcher's behavior by fetching a repository's submodu= les. - <link linkend=3D'var-bb-SRC_URI'><filename>SRC_URI</filena= me></link> - is passed to the Git fetcher as described in the - "<link linkend=3D'git-fetcher'>Git Fetcher (<filename>git:= //</filename>)</link>" - section. - <note> - <title>Notes and Warnings - - You must clean a recipe when switching between - 'git://' and - 'gitsm://' URLs. - - - - The Git Submodules fetcher is not a complete fetch= er - implementation. - The fetcher has known issues where it does not use= the - normal source mirroring infrastructure properly. F= urther, - the submodule sources it fetches are not visible t= o the - licensing and source archiving infrastructures. - - - -
    - -
    - ClearCase Fetcher (<filename>ccrc://</filename>) - - - This fetcher submodule fetches code from a - ClearCase - repository. - - - - To use this fetcher, make sure your recipe has proper - SRC_URI, - SRCREV, and - PV= settings. - Here is an example: - - SRC_URI =3D "ccrc://cc.example.org/ccrc;vob=3D/example_vob;module=3D/= example_module" - SRCREV =3D "EXAMPLE_CLEARCASE_TAG" - PV =3D "${@d.getVar("SRCREV", False).replace("/", "+")}" - - The fetcher uses the rcleartool or - cleartool remote client, depending on - which one is available. - - - - Following are options for the SRC_URI - statement: - - vob: - The name, which must include the - prepending "/" character, of the ClearCase VOB. - This option is required. - - module<= /emphasis>: - The module, which must include the - prepending "/" character, in the selected VOB. - - The module and = vob - options are combined to create the l= oad rule in - the view config spec. - As an example, consider the vob and - module values from the - SRC_URI statement at the = start of this section. - Combining those values results in the followin= g: - - load /example_vob/example_module - - - - proto: - The protocol, which can be either http or - https. - - - - - - By default, the fetcher creates a configuration specificat= ion. - If you want this specification written to an area other th= an the default, - use the CCASE_CUSTOM_CONFIG_SPEC vari= able - in your recipe to define where the specification is writte= n. - - the SRCREV loses its functionalit= y if you - specify this variable. - However, SRCREV is still used to = label the - archive after a fetch even though it does not define w= hat is - fetched. - - - - - Here are a couple of other behaviors worth mentioning: - - - When using cleartool, the log= in of - cleartool is handled by the s= ystem. - The login require no special steps. - - - In order to use rcleartool wi= th authenticated - users, an "rcleartool login" is necessary before u= sing the fetcher. - - - -
    - -
    - Perforce Fetcher (<filename>p4://</filename>) - - - This fetcher submodule fetches code from the - Perforce - source control system. - The executable used is specified by - FETCHCMD_p4, which defaults - to "p4". - The fetcher's temporary working directory is set by - P4DIR<= /link>, - which defaults to "DL_DIR/p4". - - - - To use this fetcher, make sure your recipe has proper - SRC_URI, - SRCREV, and - PV= values. - The p4 executable is able to use the config file defined b= y your - system's P4CONFIG environment variabl= e in - order to define the Perforce server URL and port, username= , and - password if you do not wish to keep those values in a reci= pe - itself. - If you choose not to use P4CONFIG, - or to explicitly set variables that P4CONFIG - can contain, you can specify the P4PORT value, - which is the server's URL and port number, and you can - specify a username and password directly in your recipe wi= thin - SRC_URI. - - - - Here is an example that relies on P4CONFIG - to specify the server URL and port, username, and password= , and - fetches the Head Revision: - - SRC_URI =3D "p4://example-depot/main/source/..." - SRCREV =3D "${AUTOREV}" - PV =3D "p4-${SRCPV}" - S =3D "${WORKDIR}/p4" - - - - - Here is an example that specifies the server URL and port, - username, and password, and fetches a Revision based on a = Label: - - P4PORT =3D "tcp:p4server.example.net:1666" - SRC_URI =3D "p4://user:passwd@example-depot/main/source/..." - SRCREV =3D "release-1.0" - PV =3D "p4-${SRCPV}" - S =3D "${WORKDIR}/p4" - - - You should always set S - to "${WORKDIR}/p4" in your recipe. - - -
    - -
    - Repo Fetcher (<filename>repo://</filename>) - - - This fetcher submodule fetches code from - google-repo source control system. - The fetcher works by initiating and syncing sources of the - repository into - REPODIR, - which is usually - DL_DIR/repo. - - - - This fetcher supports the following parameters: - - - "protocol": - Protocol to fetch the repository manifest (default= : git). - - - "branch": - Branch or tag of repository to get (default: maste= r). - - - "manifest": - Name of the manifest file (default: defa= ult.xml). - - - Here are some example URLs: - - SRC_URI =3D "repo://REPOROOT;protocol=3Dgit;branch=3Dsome_branch;manif= est=3Dmy_manifest.xml" - SRC_URI =3D "repo://REPOROOT;protocol=3Dfile;branch=3Dsome_branch;mani= fest=3Dmy_manifest.xml" - - -
    - -
    - Other Fetchers - - - Fetch submodules also exist for the following: - - - Bazaar (bzr://) - - - Mercurial (hg://) - - - npm (npm://) - - - OSC (osc://) - - - Secure FTP (sftp://) - - - Secure Shell (ssh://) - - - Trees using Git Annex (gitannex://) - - - No documentation currently exists for these lesser used - fetcher submodules. - However, you might find the code helpful and readable. - -
    -
    - -
    - Auto Revisions - - - We need to document AUTOREV and - SRCREV_FORMAT here. - -
    -     diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.rst = b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.rst new file mode 100644 index 00000000..e3fd3215 --- /dev/null +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.rst @@ -0,0 +1,415 @@ +.. SPDX-License-Identifier: CC-BY-2.5 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Hello World Example +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +BitBake Hello World +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +The simplest example commonly used to demonstrate any new programming +language or tool is the "`Hello +World `__" example. +This appendix demonstrates, in tutorial form, Hello World within the +context of BitBake. The tutorial describes how to create a new project +and the applicable metadata files necessary to allow BitBake to build +it. + +Obtaining BitBake +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +See the :ref:`bitbake-user-manual/bitbake-user-manual-hello:obtaining bitb= ake` section for +information on how to obtain BitBake. Once you have the source code on +your machine, the BitBake directory appears as follows: :: + + $ ls -al + total 100 + drwxrwxr-x. 9 wmat wmat 4096 Jan 31 13:44 . + drwxrwxr-x. 3 wmat wmat 4096 Feb 4 10:45 .. + -rw-rw-r--. 1 wmat wmat 365 Nov 26 04:55 AUTHORS + drwxrwxr-x. 2 wmat wmat 4096 Nov 26 04:55 bin + drwxrwxr-x. 4 wmat wmat 4096 Jan 31 13:44 build + -rw-rw-r--. 1 wmat wmat 16501 Nov 26 04:55 ChangeLog + drwxrwxr-x. 2 wmat wmat 4096 Nov 26 04:55 classes + drwxrwxr-x. 2 wmat wmat 4096 Nov 26 04:55 conf + drwxrwxr-x. 3 wmat wmat 4096 Nov 26 04:55 contrib + -rw-rw-r--. 1 wmat wmat 17987 Nov 26 04:55 COPYING + drwxrwxr-x. 3 wmat wmat 4096 Nov 26 04:55 doc + -rw-rw-r--. 1 wmat wmat 69 Nov 26 04:55 .gitignore + -rw-rw-r--. 1 wmat wmat 849 Nov 26 04:55 HEADER + drwxrwxr-x. 5 wmat wmat 4096 Jan 31 13:44 lib + -rw-rw-r--. 1 wmat wmat 195 Nov 26 04:55 MANIFEST.in + -rw-rw-r--. 1 wmat wmat 2887 Nov 26 04:55 TODO + +At this point, you should have BitBake cloned to a directory that +matches the previous listing except for dates and user names. + +Setting Up the BitBake Environment +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +First, you need to be sure that you can run BitBake. Set your working +directory to where your local BitBake files are and run the following +command: :: + + $ ./bin/bitbake --version + BitBake Build Tool Core version 1.23.0, bitbake version 1.23.0 + +The console output tells you what version +you are running. + +The recommended method to run BitBake is from a directory of your +choice. To be able to run BitBake from any directory, you need to add +the executable binary to your binary to your shell's environment +``PATH`` variable. First, look at your current ``PATH`` variable by +entering the following: :: + + $ echo $PATH + +Next, add the directory location +for the BitBake binary to the ``PATH``. Here is an example that adds the +``/home/scott-lenovo/bitbake/bin`` directory to the front of the +``PATH`` variable: :: + + $ export PATH=3D/home/scott-lenovo/bitbake/bin:$PATH + +You should now be able to enter the ``bitbake`` command from the command +line while working from any directory. + +The Hello World Example +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +The overall goal of this exercise is to build a complete "Hello World" +example utilizing task and layer concepts. Because this is how modern +projects such as OpenEmbedded and the Yocto Project utilize BitBake, the +example provides an excellent starting point for understanding BitBake. + +To help you understand how to use BitBake to build targets, the example +starts with nothing but the ``bitbake`` command, which causes BitBake to +fail and report problems. The example progresses by adding pieces to the +build to eventually conclude with a working, minimal "Hello World" +example. + +While every attempt is made to explain what is happening during the +example, the descriptions cannot cover everything. You can find further +information throughout this manual. Also, you can actively participate +in the :oe_lists:`/g/bitbake-devel` +discussion mailing list about the BitBake build tool. + +.. note:: + + This example was inspired by and drew heavily from + `Mailing List post - The BitBake equivalent of "Hello, World!" + `_. + +As stated earlier, the goal of this example is to eventually compile +"Hello World". However, it is unknown what BitBake needs and what you +have to provide in order to achieve that goal. Recall that BitBake +utilizes three types of metadata files: +:ref:`bitbake-user-manual/bitbake-user-manual-intro:configuration files`, +:ref:`bitbake-user-manual/bitbake-user-manual-intro:classes`, and +:ref:`bitbake-user-manual/bitbake-user-manual-intro:recipes`. +But where do they go? How does BitBake find +them? BitBake's error messaging helps you answer these types of +questions and helps you better understand exactly what is going on. + +Following is the complete "Hello World" example. + +#. **Create a Project Directory:** First, set up a directory for the + "Hello World" project. Here is how you can do so in your home + directory: :: + + $ mkdir ~/hello + $ cd ~/hello + + This is the directory that + BitBake will use to do all of its work. You can use this directory + to keep all the metafiles needed by BitBake. Having a project + directory is a good way to isolate your project. + +#. **Run BitBake:** At this point, you have nothing but a project + directory. Run the ``bitbake`` command and see what it does: :: + + $ bitbake + The BBPATH variable is not set and bitbake did not + find a conf/bblayers.conf file in the expected location. + Maybe you accidentally invoked bitbake from the wrong directory? + DEBUG: Removed the following variables from the environment: + GNOME_DESKTOP_SESSION_ID, XDG_CURRENT_DESKTOP, + GNOME_KEYRING_CONTROL, DISPLAY, SSH_AGENT_PID, LANG, no_proxy, + XDG_SESSION_PATH, XAUTHORITY, SESSION_MANAGER, SHLVL, + MANDATORY_PATH, COMPIZ_CONFIG_PROFILE, WINDOWID, EDITOR, + GPG_AGENT_INFO, SSH_AUTH_SOCK, GDMSESSION, GNOME_KEYRING_PID, + XDG_SEAT_PATH, XDG_CONFIG_DIRS, LESSOPEN, DBUS_SESSION_BUS_ADDRESS, + _, XDG_SESSION_COOKIE, DESKTOP_SESSION, LESSCLOSE, DEFAULTS_PATH, + UBUNTU_MENUPROXY, OLDPWD, XDG_DATA_DIRS, COLORTERM, LS_COLORS + + The majority of this output is specific to environment variables that + are not directly relevant to BitBake. However, the very first + message regarding the ``BBPATH`` variable and the + ``conf/bblayers.conf`` file is relevant. + + When you run BitBake, it begins looking for metadata files. The + :term:`BBPATH` variable is what tells BitBake where + to look for those files. ``BBPATH`` is not set and you need to set + it. Without ``BBPATH``, BitBake cannot find any configuration files + (``.conf``) or recipe files (``.bb``) at all. BitBake also cannot + find the ``bitbake.conf`` file. + +#. **Setting BBPATH:** For this example, you can set ``BBPATH`` in + the same manner that you set ``PATH`` earlier in the appendix. You + should realize, though, that it is much more flexible to set the + ``BBPATH`` variable up in a configuration file for each project. + + From your shell, enter the following commands to set and export the + ``BBPATH`` variable: :: + + $ BBPATH=3D"projectdirectory" + $ export BBPATH + + Use your actual project directory in the command. BitBake uses that + directory to find the metadata it needs for your project. + + .. note:: + + When specifying your project directory, do not use the tilde + ("~") character as BitBake does not expand that character as the + shell would. + +#. **Run BitBake:** Now that you have ``BBPATH`` defined, run the + ``bitbake`` command again: :: + + $ bitbake + ERROR: Traceback (most recent call last): + File "/home/scott-lenovo/bitbake/lib/bb/cookerdata.py", line 163,= in wrapped + return func(fn, *args) + File "/home/scott-lenovo/bitbake/lib/bb/cookerdata.py", line 173,= in parse_config_file + return bb.parse.handle(fn, data, include) + File "/home/scott-lenovo/bitbake/lib/bb/parse/__init__.py", line = 99, in handle + return h['handle'](fn, data, include) + File "/home/scott-lenovo/bitbake/lib/bb/parse/parse_py/ConfHandle= r.py", line 120, in handle + abs_fn =3D resolve_file(fn, data) + File "/home/scott-lenovo/bitbake/lib/bb/parse/__init__.py", line = 117, in resolve_file + raise IOError("file %s not found in %s" % (fn, bbpath)) + IOError: file conf/bitbake.conf not found in /home/scott-lenovo/hel= lo + + ERROR: Unable to parse conf/bitbake.conf: file conf/bitbake.conf no= t found in /home/scott-lenovo/hello + + This sample output shows that BitBake could not find the + ``conf/bitbake.conf`` file in the project directory. This file is + the first thing BitBake must find in order to build a target. And, + since the project directory for this example is empty, you need to + provide a ``conf/bitbake.conf`` file. + +#. **Creating conf/bitbake.conf:** The ``conf/bitbake.conf`` includes + a number of configuration variables BitBake uses for metadata and + recipe files. For this example, you need to create the file in your + project directory and define some key BitBake variables. For more + information on the ``bitbake.conf`` file, see + http://git.openembedded.org/bitbake/tree/conf/bitbake.conf. + + Use the following commands to create the ``conf`` directory in the + project directory: :: + + $ mkdir conf + + From within the ``conf`` directory, + use some editor to create the ``bitbake.conf`` so that it contains + the following: :: + + PN =3D "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', Fals= e),d)[0] or 'defaultpkgname'}" + + TMPDIR =3D "${TOPDIR}/tmp" + CACHE =3D "${TMPDIR}/cache" + STAMP =3D "${TMPDIR}/${PN}/stamps" + T =3D "${TMPDIR}/${PN}/work" + B =3D "${TMPDIR}/${PN}" + + .. note:: + + Without a value for PN , the variables STAMP , T , and B , prevent = more + than one recipe from working. You can fix this by either setting PN= to + have a value similar to what OpenEmbedded and BitBake use in the de= fault + bitbake.conf file (see previous example). Or, by manually updating = each + recipe to set PN . You will also need to include PN as part of the = STAMP + , T , and B variable definitions in the local.conf file. + + The ``TMPDIR`` variable establishes a directory that BitBake uses + for build output and intermediate files other than the cached + information used by the + :ref:`bitbake-user-manual/bitbake-user-manual-execution:setscene` + process. Here, the ``TMPDIR`` directory is set to ``hello/tmp``. + + .. tip:: + + You can always safely delete the tmp directory in order to rebuild a + BitBake target. The build process creates the directory for you whe= n you + run BitBake. + + For information about each of the other variables defined in this + example, check :term:`PN`, :term:`TOPDIR`, :term:`CACHE`, :term:`STAMP= `, + :term:`T` or :term:`B` to take you to the definitions in the + glossary. + +#. **Run BitBake:** After making sure that the ``conf/bitbake.conf`` file + exists, you can run the ``bitbake`` command again: :: + + $ bitbake + ERROR: Traceback (most recent call last): + File "/home/scott-lenovo/bitbake/lib/bb/cookerdata.py", line 163,= in wrapped + return func(fn, *args) + File "/home/scott-lenovo/bitbake/lib/bb/cookerdata.py", line 177,= in _inherit + bb.parse.BBHandler.inherit(bbclass, "configuration INHERITs", 0= , data) + File "/home/scott-lenovo/bitbake/lib/bb/parse/parse_py/BBHandler.= py", line 92, in inherit + include(fn, file, lineno, d, "inherit") + File "/home/scott-lenovo/bitbake/lib/bb/parse/parse_py/ConfHandle= r.py", line 100, in include + raise ParseError("Could not %(error_out)s file %(fn)s" % vars()= , oldfn, lineno) + ParseError: ParseError in configuration INHERITs: Could not inherit= file classes/base.bbclass + + ERROR: Unable to parse base: ParseError in configuration INHERITs: = Could not inherit file classes/base.bbclass + + In the sample output, + BitBake could not find the ``classes/base.bbclass`` file. You need + to create that file next. + +#. **Creating classes/base.bbclass:** BitBake uses class files to + provide common code and functionality. The minimally required class + for BitBake is the ``classes/base.bbclass`` file. The ``base`` class + is implicitly inherited by every recipe. BitBake looks for the class + in the ``classes`` directory of the project (i.e ``hello/classes`` + in this example). + + Create the ``classes`` directory as follows: :: + + $ cd $HOME/hello + $ mkdir classes + + Move to the ``classes`` directory and then create the + ``base.bbclass`` file by inserting this single line: addtask build + The minimal task that BitBake runs is the ``do_build`` task. This is + all the example needs in order to build the project. Of course, the + ``base.bbclass`` can have much more depending on which build + environments BitBake is supporting. + +#. **Run BitBake:** After making sure that the ``classes/base.bbclass`` + file exists, you can run the ``bitbake`` command again: :: + + $ bitbake + Nothing to do. Use 'bitbake world' to build everything, or run 'bit= bake --help' for usage information. + + BitBake is finally reporting + no errors. However, you can see that it really does not have + anything to do. You need to create a recipe that gives BitBake + something to do. + +#. **Creating a Layer:** While it is not really necessary for such a + small example, it is good practice to create a layer in which to + keep your code separate from the general metadata used by BitBake. + Thus, this example creates and uses a layer called "mylayer". + + .. note:: + + You can find additional information on layers in the + ":ref:`bitbake-user-manual/bitbake-user-manual-intro:Layers`" secti= on. + + Minimally, you need a recipe file and a layer configuration file in + your layer. The configuration file needs to be in the ``conf`` + directory inside the layer. Use these commands to set up the layer + and the ``conf`` directory: :: + + $ cd $HOME + $ mkdir mylayer + $ cd mylayer + $ mkdir conf + + Move to the ``conf`` directory and create a ``layer.conf`` file that h= as the + following: :: + + BBPATH .=3D ":${LAYERDIR}" + BBFILES +=3D "${LAYERDIR}/\*.bb" + BBFILE_COLLECTIONS +=3D "mylayer" + `BBFILE_PATTERN_mylayer :=3D "^${LAYERDIR_RE}/" + + For information on these variables, click on :term:`BBFILES`, + :term:`LAYERDIR`, :term:`BBFILE_COLLECTIONS` or :term:`BBFILE_PATTERN_= mylayer ` + to go to the definitions in the glossary. + + You need to create the recipe file next. Inside your layer at the + top-level, use an editor and create a recipe file named + ``printhello.bb`` that has the following: :: + + DESCRIPTION =3D "Prints Hello World" + PN =3D 'printhello' + PV =3D '1' + + python do_build() { + bb.plain("********************"); + bb.plain("* *"); + bb.plain("* Hello, World! *"); + bb.plain("* *"); + bb.plain("********************"); + } + + The recipe file simply provides + a description of the recipe, the name, version, and the ``do_build`` + task, which prints out "Hello World" to the console. For more + information on :term:`DESCRIPTION`, :term:`PN` or :term:`PV` + follow the links to the glossary. + +#. **Run BitBake With a Target:** Now that a BitBake target exists, run + the command and provide that target: :: + + $ cd $HOME/hello + $ bitbake printhello + ERROR: no recipe files to build, check your BBPATH and BBFILES? + + Summary: There was 1 ERROR message shown, returning a non-zero exit = code. + + We have created the layer with the recipe and + the layer configuration file but it still seems that BitBake cannot + find the recipe. BitBake needs a ``conf/bblayers.conf`` that lists + the layers for the project. Without this file, BitBake cannot find + the recipe. + +#. **Creating conf/bblayers.conf:** BitBake uses the + ``conf/bblayers.conf`` file to locate layers needed for the project. + This file must reside in the ``conf`` directory of the project (i.e. + ``hello/conf`` for this example). + + Set your working directory to the ``hello/conf`` directory and then + create the ``bblayers.conf`` file so that it contains the following: :: + + BBLAYERS ?=3D " \ + /home//mylayer \ + " + + You need to provide your own information for ``you`` in the file. + +#. **Run BitBake With a Target:** Now that you have supplied the + ``bblayers.conf`` file, run the ``bitbake`` command and provide the + target: :: + + $ bitbake printhello + Parsing recipes: 100% |############################################= ######################################| + Time: 00:00:00 + Parsing of 1 .bb files complete (0 cached, 1 parsed). 1 targets, 0 = skipped, 0 masked, 0 errors. + NOTE: Resolving any missing task queue dependencies + NOTE: Preparing RunQueue + NOTE: Executing RunQueue Tasks + ******************** + * * + * Hello, World! * + * * + ******************** + NOTE: Tasks Summary: Attempted 1 tasks of which 0 didn't need to be= rerun and all succeeded. + + .. note:: + + After the first execution, re-running bitbake printhello again will= not + result in a BitBake run that prints the same console output. The re= ason + for this is that the first time the printhello.bb recipe's do_build= task + executes successfully, BitBake writes a stamp file for the task. Th= us, + the next time you attempt to run the task using that same bitbake + command, BitBake notices the stamp and therefore determines that th= e task + does not need to be re-run. If you delete the tmp directory or run + bitbake -c clean printhello and then re-run the build, the "Hello, + World!" message will be printed again. diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.xml = b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.xml deleted file mode 100644 index 11eb36aa..00000000 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.xml +++ /dev/null @@ -1,513 +0,0 @@ - - - - Hello World Example - -
    - BitBake Hello World - - - The simplest example commonly used to demonstrate any new - programming language or tool is the - "Hello World" - example. - This appendix demonstrates, in tutorial form, Hello - World within the context of BitBake. - The tutorial describes how to create a new project - and the applicable metadata files necessary to allow - BitBake to build it. - -
    - -
    - Obtaining BitBake - - - See the - "Obtaining BitBake" - section for information on how to obtain BitBake. - Once you have the source code on your machine, the BitBake dir= ectory - appears as follows: - - $ ls -al - total 100 - drwxrwxr-x. 9 wmat wmat 4096 Jan 31 13:44 . - drwxrwxr-x. 3 wmat wmat 4096 Feb 4 10:45 .. - -rw-rw-r--. 1 wmat wmat 365 Nov 26 04:55 AUTHORS - drwxrwxr-x. 2 wmat wmat 4096 Nov 26 04:55 bin - drwxrwxr-x. 4 wmat wmat 4096 Jan 31 13:44 build - -rw-rw-r--. 1 wmat wmat 16501 Nov 26 04:55 ChangeLog - drwxrwxr-x. 2 wmat wmat 4096 Nov 26 04:55 classes - drwxrwxr-x. 2 wmat wmat 4096 Nov 26 04:55 conf - drwxrwxr-x. 3 wmat wmat 4096 Nov 26 04:55 contrib - -rw-rw-r--. 1 wmat wmat 17987 Nov 26 04:55 COPYING - drwxrwxr-x. 3 wmat wmat 4096 Nov 26 04:55 doc - -rw-rw-r--. 1 wmat wmat 69 Nov 26 04:55 .gitignore - -rw-rw-r--. 1 wmat wmat 849 Nov 26 04:55 HEADER - drwxrwxr-x. 5 wmat wmat 4096 Jan 31 13:44 lib - -rw-rw-r--. 1 wmat wmat 195 Nov 26 04:55 MANIFEST.in - -rw-rw-r--. 1 wmat wmat 2887 Nov 26 04:55 TODO - - - - - At this point, you should have BitBake cloned to - a directory that matches the previous listing except for - dates and user names. - -
    - -
    - Setting Up the BitBake Environment - - - First, you need to be sure that you can run BitBake. - Set your working directory to where your local BitBake - files are and run the following command: - - $ ./bin/bitbake --version - BitBake Build Tool Core version 1.23.0, bitbake version 1.23.0 - - The console output tells you what version you are running. - - - - The recommended method to run BitBake is from a directory of y= our - choice. - To be able to run BitBake from any directory, you need to add = the - executable binary to your binary to your shell's environment - PATH variable. - First, look at your current PATH variable - by entering the following: - - $ echo $PATH - - Next, add the directory location for the BitBake binary to the - PATH. - Here is an example that adds the - /home/scott-lenovo/bitbake/bin directory - to the front of the PATH variable: - - $ export PATH=3D/home/scott-lenovo/bitbake/bin:$PATH - - You should now be able to enter the bitbake - command from the command line while working from any directory. - -
    - -
    - The Hello World Example - - - The overall goal of this exercise is to build a - complete "Hello World" example utilizing task and layer - concepts. - Because this is how modern projects such as OpenEmbedded and - the Yocto Project utilize BitBake, the example - provides an excellent starting point for understanding - BitBake. - - - - To help you understand how to use BitBake to build targets, - the example starts with nothing but the bitbake - command, which causes BitBake to fail and report problems. - The example progresses by adding pieces to the build to - eventually conclude with a working, minimal "Hello World" - example. - - - - While every attempt is made to explain what is happening during - the example, the descriptions cannot cover everything. - You can find further information throughout this manual. - Also, you can actively participate in the - - discussion mailing list about the BitBake build tool. - - - - This example was inspired by and drew heavily from - Mailing List post - The BitBake equivalent of "Hello, Wor= ld!". - - - - As stated earlier, the goal of this example - is to eventually compile "Hello World". - However, it is unknown what BitBake needs and what you have - to provide in order to achieve that goal. - Recall that BitBake utilizes three types of metadata files: - Configuration Files, - Classes, and - Recipes. - But where do they go? - How does BitBake find them? - BitBake's error messaging helps you answer these types of ques= tions - and helps you better understand exactly what is going on. - - - - Following is the complete "Hello World" example. - - - - Create a Project Directory: - First, set up a directory for the "Hello World" project. - Here is how you can do so in your home directory: - - $ mkdir ~/hello - $ cd ~/hello - - This is the directory that BitBake will use to do all of - its work. - You can use this directory to keep all the metafiles needed - by BitBake. - Having a project directory is a good way to isolate your - project. - - Run BitBake: - At this point, you have nothing but a project directory. - Run the bitbake command and see what - it does: - - $ bitbake - The BBPATH variable is not set and bitbake did not - find a conf/bblayers.conf file in the expected location. - Maybe you accidentally invoked bitbake from the wrong directory? - DEBUG: Removed the following variables from the environment: - GNOME_DESKTOP_SESSION_ID, XDG_CURRENT_DESKTOP, - GNOME_KEYRING_CONTROL, DISPLAY, SSH_AGENT_PID, LANG, no_proxy, - XDG_SESSION_PATH, XAUTHORITY, SESSION_MANAGER, SHLVL, - MANDATORY_PATH, COMPIZ_CONFIG_PROFILE, WINDOWID, EDITOR, - GPG_AGENT_INFO, SSH_AUTH_SOCK, GDMSESSION, GNOME_KEYRING_PID, - XDG_SEAT_PATH, XDG_CONFIG_DIRS, LESSOPEN, DBUS_SESSION_BUS_ADDRESS, - _, XDG_SESSION_COOKIE, DESKTOP_SESSION, LESSCLOSE, DEFAULTS_PATH, - UBUNTU_MENUPROXY, OLDPWD, XDG_DATA_DIRS, COLORTERM, LS_COLORS - - The majority of this output is specific to environment var= iables - that are not directly relevant to BitBake. - However, the very first message regarding the - BBPATH variable and the - conf/bblayers.conf file - is relevant. - - When you run BitBake, it begins looking for metadata files. - The - BBPATH - variable is what tells BitBake where to look for those fil= es. - BBPATH is not set and you need to set= it. - Without BBPATH, BitBake cannot - find any configuration files (.conf) - or recipe files (.bb) at all. - BitBake also cannot find the bitbake.conf - file. - - Setting BBPATH:= - For this example, you can set BBPATH - in the same manner that you set PATH - earlier in the appendix. - You should realize, though, that it is much more flexible = to set the - BBPATH variable up in a configuration - file for each project. - From your shell, enter the following commands to set= and - export the BBPATH variable: - - $ BBPATH=3D"projectdirectory" - $ export BBPATH - - Use your actual project directory in the command. - BitBake uses that directory to find the metadata it needs = for - your project. - - When specifying your project directory, do not use the - tilde ("~") character as BitBake does not expand that = character - as the shell would. - - - Run BitBake: - Now that you have BBPATH defined, run - the bitbake command again: - - $ bitbake - ERROR: Traceback (most recent call last): - File "/home/scott-lenovo/bitbake/lib/bb/cookerdata.py", line 163, i= n wrapped - return func(fn, *args) - File "/home/scott-lenovo/bitbake/lib/bb/cookerdata.py", line 173, i= n parse_config_file - return bb.parse.handle(fn, data, include) - File "/home/scott-lenovo/bitbake/lib/bb/parse/__init__.py", line 99= , in handle - return h['handle'](fn, data, include) - File "/home/scott-lenovo/bitbake/lib/bb/parse/parse_py/ConfHandler.= py", line 120, in handle - abs_fn =3D resolve_file(fn, data) - File "/home/scott-lenovo/bitbake/lib/bb/parse/__init__.py", line 11= 7, in resolve_file - raise IOError("file %s not found in %s" % (fn, bbpath)) - IOError: file conf/bitbake.conf not found in /home/scott-lenovo/hello - - ERROR: Unable to parse conf/bitbake.conf: file conf/bitbake.conf not = found in /home/scott-lenovo/hello - - This sample output shows that BitBake could not find the - conf/bitbake.conf file in the project - directory. - This file is the first thing BitBake must find in order - to build a target. - And, since the project directory for this example is - empty, you need to provide a conf/bitbake.conf - file. - - Creating conf/bitbake.conf= : - The conf/bitbake.conf includes a numb= er of - configuration variables BitBake uses for metadata and reci= pe - files. - For this example, you need to create the file in your proj= ect directory - and define some key BitBake variables. - For more information on the bitbake.conf file, - see - . - - Use the following commands to create the c= onf - directory in the project directory: - - $ mkdir conf - - From within the conf directory, use - some editor to create the bitbake.conf - so that it contains the following: - - PN =3D "${@bb.parse.BBHandler.var= s_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}" - - - TMPDIR =3D "${TOPDIR}/tmp" - CACHE =3D "${TMPDIR}/cache" - STAMP =3D "${TMPDIR}/${PN}/st= amps" - T =3D "${TMPDIR}/${PN}/work" - B =3D "${TMPDIR}/${PN}" - - - Without a value for PN, the - variables STAMP, - T, and B, - prevent more than one recipe from working. You can fix - this by either setting PN to have - a value similar to what OpenEmbedded and BitBake use - in the default bitbake.conf file - (see previous example). Or, by manually updating each - recipe to set PN. You will also - need to include PN as part of the - STAMP, T, and - B variable definitions in the - local.conf file. - - The TMPDIR variable establishes a dir= ectory - that BitBake uses for build output and intermediate files = other - than the cached information used by the - Setscene process. - Here, the TMPDIR directory is set to - hello/tmp. - Tip - You can always safely delete the tmp - directory in order to rebuild a BitBake target. - The build process creates the directory for you - when you run BitBake. - - For information about each of the other variables de= fined in this - example, click on the links to take you to the definitions= in - the glossary. - - Run BitBake: - After making sure that the conf/bitbake.conf - file exists, you can run the bitbake - command again: - - $ bitbake - ERROR: Traceback (most recent call last): - File "/home/scott-lenovo/bitbake/lib/bb/cookerdata.py", line 163, i= n wrapped - return func(fn, *args) - File "/home/scott-lenovo/bitbake/lib/bb/cookerdata.py", line 177, i= n _inherit - bb.parse.BBHandler.inherit(bbclass, "configuration INHERITs", 0, = data) - File "/home/scott-lenovo/bitbake/lib/bb/parse/parse_py/BBHandler.py= ", line 92, in inherit - include(fn, file, lineno, d, "inherit") - File "/home/scott-lenovo/bitbake/lib/bb/parse/parse_py/ConfHandler.= py", line 100, in include - raise ParseError("Could not %(error_out)s file %(fn)s" % vars(), = oldfn, lineno) - ParseError: ParseError in configuration INHERITs: Could not inherit f= ile classes/base.bbclass - - ERROR: Unable to parse base: ParseError in configuration INHERITs: Co= uld not inherit file classes/base.bbclass - - In the sample output, BitBake could not find the - classes/base.bbclass file. - You need to create that file next. - - Creating classes/base.bbcl= ass: - BitBake uses class files to provide common code and functi= onality. - The minimally required class for BitBake is the - classes/base.bbclass file. - The base class is implicitly inherite= d by - every recipe. - BitBake looks for the class in the classes - directory of the project (i.e hello/classes - in this example). - - Create the classes directory as= follows: - - $ cd $HOME/hello - $ mkdir classes - - Move to the classes directory and then - create the base.bbclass file by inser= ting - this single line: - - addtask build - - The minimal task that BitBake runs is the - do_build task. - This is all the example needs in order to build the projec= t. - Of course, the base.bbclass can have = much - more depending on which build environments BitBake is - supporting. - - Run BitBake: - After making sure that the classes/base.bbclass<= /filename> - file exists, you can run the bitbake - command again: - - $ bitbake - Nothing to do. Use 'bitbake world' to build everything, or run 'bitb= ake --help' for usage information. - - BitBake is finally reporting no errors. - However, you can see that it really does not have anything - to do. - You need to create a recipe that gives BitBake something t= o do. - - Creating a Layer: - While it is not really necessary for such a small example, - it is good practice to create a layer in which to keep your - code separate from the general metadata used by BitBake. - Thus, this example creates and uses a layer called "mylaye= r". - - You can find additional information on layers in the - "Layers" section. - - - Minimally, you need a recipe file and a layer config= uration - file in your layer. - The configuration file needs to be in the conf - directory inside the layer. - Use these commands to set up the layer and the c= onf - directory: - - $ cd $HOME - $ mkdir mylayer - $ cd mylayer - $ mkdir conf - - Move to the conf directory and create= a - layer.conf file that has the followin= g: - - BBPATH .=3D ":${LAYERDIR}" - - BBFILES +=3D "${LAYERDIR}/*.b= b" - - BBFILE_COLLECTIONS= +=3D "mylayer" - BBFILE_PATTERN_mylayer= :=3D "^${LAYERDIR_RE}/" - - For information on these variables, click the links - to go to the definitions in the glossary. - You need to create the recipe file next. - Inside your layer at the top-level, use an editor and crea= te - a recipe file named printhello.bb that - has the following: - - DESCRIPTION =3D "Prints H= ello World" - PN =3D 'printhello' - PV =3D '1' - - python do_build() { - bb.plain("********************"); - bb.plain("* *"); - bb.plain("* Hello, World! *"); - bb.plain("* *"); - bb.plain("********************"); - } - - The recipe file simply provides a description of the - recipe, the name, version, and the do_build - task, which prints out "Hello World" to the console. - For more information on these variables, follow the links - to the glossary. - - Run BitBake With a Target: - Now that a BitBake target exists, run the command and prov= ide - that target: - - $ cd $HOME/hello - $ bitbake printhello - ERROR: no recipe files to build, check your BBPATH and BBFILES? - - Summary: There was 1 ERROR message shown, returning a non-zero exit c= ode. - - We have created the layer with the recipe and the layer - configuration file but it still seems that BitBake cannot - find the recipe. - BitBake needs a conf/bblayers.conf th= at - lists the layers for the project. - Without this file, BitBake cannot find the recipe. - - Creating conf/bblayers.con= f: - BitBake uses the conf/bblayers.conf f= ile - to locate layers needed for the project. - This file must reside in the conf dir= ectory - of the project (i.e. hello/conf for t= his - example). - Set your working directory to the hello/co= nf - directory and then create the bblayers.conf - file so that it contains the following: - - BBLAYERS ?=3D " \ - /home/<you>/mylayer \ - " - - You need to provide your own information for - you in the file. - - Run BitBake With a Target: - Now that you have supplied the bblayers.conf - file, run the bitbake command and pro= vide - the target: - - $ bitbake printhello - Parsing recipes: 100% |##############################################= ####################################| - Time: 00:00:00 - Parsing of 1 .bb files complete (0 cached, 1 parsed). 1 targets, 0 sk= ipped, 0 masked, 0 errors. - NOTE: Resolving any missing task queue dependencies - NOTE: Preparing RunQueue - NOTE: Executing RunQueue Tasks - ******************** - * * - * Hello, World! * - * * - ******************** - NOTE: Tasks Summary: Attempted 1 tasks of which 0 didn't need to be r= erun and all succeeded. - - BitBake finds the printhello recipe a= nd - successfully runs the task. - - After the first execution, re-running - bitbake printhello again will not - result in a BitBake run that prints the same console - output. - The reason for this is that the first time the - printhello.bb recipe's - do_build task executes - successfully, BitBake writes a stamp file for the task. - Thus, the next time you attempt to run the task - using that same bitbake command, - BitBake notices the stamp and therefore determines - that the task does not need to be re-run. - If you delete the tmp directory - or run bitbake -c clean printhello - and then re-run the build, the "Hello, World!" message= will - be printed again. - - - -
    -     diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.rst = b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.rst new file mode 100644 index 00000000..6f9d3929 --- /dev/null +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.rst @@ -0,0 +1,651 @@ +.. SPDX-License-Identifier: CC-BY-2.5 + +=3D=3D=3D=3D=3D=3D=3D=3D +Overview +=3D=3D=3D=3D=3D=3D=3D=3D + +| + +Welcome to the BitBake User Manual. This manual provides information on +the BitBake tool. The information attempts to be as independent as +possible regarding systems that use BitBake, such as OpenEmbedded and +the Yocto Project. In some cases, scenarios or examples within the +context of a build system are used in the manual to help with +understanding. For these cases, the manual clearly states the context. + +.. _intro: + +Introduction +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Fundamentally, BitBake is a generic task execution engine that allows +shell and Python tasks to be run efficiently and in parallel while +working within complex inter-task dependency constraints. One of +BitBake's main users, OpenEmbedded, takes this core and builds embedded +Linux software stacks using a task-oriented approach. + +Conceptually, BitBake is similar to GNU Make in some regards but has +significant differences: + +- BitBake executes tasks according to provided metadata that builds up + the tasks. Metadata is stored in recipe (``.bb``) and related recipe + "append" (``.bbappend``) files, configuration (``.conf``) and + underlying include (``.inc``) files, and in class (``.bbclass``) + files. The metadata provides BitBake with instructions on what tasks + to run and the dependencies between those tasks. + +- BitBake includes a fetcher library for obtaining source code from + various places such as local files, source control systems, or + websites. + +- The instructions for each unit to be built (e.g. a piece of software) + are known as "recipe" files and contain all the information about the + unit (dependencies, source file locations, checksums, description and + so on). + +- BitBake includes a client/server abstraction and can be used from a + command line or used as a service over XML-RPC and has several + different user interfaces. + +History and Goals +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +BitBake was originally a part of the OpenEmbedded project. It was +inspired by the Portage package management system used by the Gentoo +Linux distribution. On December 7, 2004, OpenEmbedded project team +member Chris Larson split the project into two distinct pieces: + +- BitBake, a generic task executor + +- OpenEmbedded, a metadata set utilized by BitBake + +Today, BitBake is the primary basis of the +`OpenEmbedded `__ project, which is being +used to build and maintain Linux distributions such as the `Angstrom +Distribution `__, and which is +also being used as the build tool for Linux projects such as the `Yocto +Project `__. + +Prior to BitBake, no other build tool adequately met the needs of an +aspiring embedded Linux distribution. All of the build systems used by +traditional desktop Linux distributions lacked important functionality, +and none of the ad hoc Buildroot-based systems, prevalent in the +embedded space, were scalable or maintainable. + +Some important original goals for BitBake were: + +- Handle cross-compilation. + +- Handle inter-package dependencies (build time on target architecture, + build time on native architecture, and runtime). + +- Support running any number of tasks within a given package, + including, but not limited to, fetching upstream sources, unpacking + them, patching them, configuring them, and so forth. + +- Be Linux distribution agnostic for both build and target systems. + +- Be architecture agnostic. + +- Support multiple build and target operating systems (e.g. Cygwin, the + BSDs, and so forth). + +- Be self-contained, rather than tightly integrated into the build + machine's root filesystem. + +- Handle conditional metadata on the target architecture, operating + system, distribution, and machine. + +- Be easy to use the tools to supply local metadata and packages + against which to operate. + +- Be easy to use BitBake to collaborate between multiple projects for + their builds. + +- Provide an inheritance mechanism to share common metadata between + many packages. + +Over time it became apparent that some further requirements were +necessary: + +- Handle variants of a base recipe (e.g. native, sdk, and multilib). + +- Split metadata into layers and allow layers to enhance or override + other layers. + +- Allow representation of a given set of input variables to a task as a + checksum. Based on that checksum, allow acceleration of builds with + prebuilt components. + +BitBake satisfies all the original requirements and many more with +extensions being made to the basic functionality to reflect the +additional requirements. Flexibility and power have always been the +priorities. BitBake is highly extensible and supports embedded Python +code and execution of any arbitrary tasks. + +.. _Concepts: + +Concepts +=3D=3D=3D=3D=3D=3D=3D=3D + +BitBake is a program written in the Python language. At the highest +level, BitBake interprets metadata, decides what tasks are required to +run, and executes those tasks. Similar to GNU Make, BitBake controls how +software is built. GNU Make achieves its control through "makefiles", +while BitBake uses "recipes". + +BitBake extends the capabilities of a simple tool like GNU Make by +allowing for the definition of much more complex tasks, such as +assembling entire embedded Linux distributions. + +The remainder of this section introduces several concepts that should be +understood in order to better leverage the power of BitBake. + +Recipes +------- + +BitBake Recipes, which are denoted by the file extension ``.bb``, are +the most basic metadata files. These recipe files provide BitBake with +the following: + +- Descriptive information about the package (author, homepage, license, + and so on) + +- The version of the recipe + +- Existing dependencies (both build and runtime dependencies) + +- Where the source code resides and how to fetch it + +- Whether the source code requires any patches, where to find them, and + how to apply them + +- How to configure and compile the source code + +- How to assemble the generated artifacts into one or more installable + packages + +- Where on the target machine to install the package or packages + created + +Within the context of BitBake, or any project utilizing BitBake as its +build system, files with the ``.bb`` extension are referred to as +recipes. + +.. note:: + + The term "package" is also commonly used to describe recipes. + However, since the same word is used to describe packaged output from + a project, it is best to maintain a single descriptive term - + "recipes". Put another way, a single "recipe" file is quite capable + of generating a number of related but separately installable + "packages". In fact, that ability is fairly common. + +Configuration Files +------------------- + +Configuration files, which are denoted by the ``.conf`` extension, +define various configuration variables that govern the project's build +process. These files fall into several areas that define machine +configuration, distribution configuration, possible compiler tuning, +general common configuration, and user configuration. The main +configuration file is the sample ``bitbake.conf`` file, which is located +within the BitBake source tree ``conf`` directory. + +Classes +------- + +Class files, which are denoted by the ``.bbclass`` extension, contain +information that is useful to share between metadata files. The BitBake +source tree currently comes with one class metadata file called +``base.bbclass``. You can find this file in the ``classes`` directory. +The ``base.bbclass`` class files is special since it is always included +automatically for all recipes and classes. This class contains +definitions for standard basic tasks such as fetching, unpacking, +configuring (empty by default), compiling (runs any Makefile present), +installing (empty by default) and packaging (empty by default). These +tasks are often overridden or extended by other classes added during the +project development process. + +Layers +------ + +Layers allow you to isolate different types of customizations from each +other. While you might find it tempting to keep everything in one layer +when working on a single project, the more modular your metadata, the +easier it is to cope with future changes. + +To illustrate how you can use layers to keep things modular, consider +customizations you might make to support a specific target machine. +These types of customizations typically reside in a special layer, +rather than a general layer, called a Board Support Package (BSP) layer. +Furthermore, the machine customizations should be isolated from recipes +and metadata that support a new GUI environment, for example. This +situation gives you a couple of layers: one for the machine +configurations and one for the GUI environment. It is important to +understand, however, that the BSP layer can still make machine-specific +additions to recipes within the GUI environment layer without polluting +the GUI layer itself with those machine-specific changes. You can +accomplish this through a recipe that is a BitBake append +(``.bbappend``) file. + +.. _append-bbappend-files: + +Append Files +------------ + +Append files, which are files that have the ``.bbappend`` file +extension, extend or override information in an existing recipe file. + +BitBake expects every append file to have a corresponding recipe file. +Furthermore, the append file and corresponding recipe file must use the +same root filename. The filenames can differ only in the file type +suffix used (e.g. ``formfactor_0.0.bb`` and +``formfactor_0.0.bbappend``). + +Information in append files extends or overrides the information in the +underlying, similarly-named recipe files. + +When you name an append file, you can use the "``%``" wildcard character +to allow for matching recipe names. For example, suppose you have an +append file named as follows: :: + + busybox_1.21.%.bbappend + +That append file +would match any ``busybox_1.21.``\ x\ ``.bb`` version of the recipe. So, +the append file would match the following recipe names: :: + + busybox_1.21.1.bb + busybox_1.21.2.bb + busybox_1.21.3.bb + +.. note:: + + The use of the " % " character is limited in that it only works directl= y in + front of the .bbappend portion of the append file's name. You cannot us= e the + wildcard character in any other location of the name. + +If the ``busybox`` recipe was updated to ``busybox_1.3.0.bb``, the +append name would not match. However, if you named the append file +``busybox_1.%.bbappend``, then you would have a match. + +In the most general case, you could name the append file something as +simple as ``busybox_%.bbappend`` to be entirely version independent. + +Obtaining BitBake +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +You can obtain BitBake several different ways: + +- **Cloning BitBake:** Using Git to clone the BitBake source code + repository is the recommended method for obtaining BitBake. Cloning + the repository makes it easy to get bug fixes and have access to + stable branches and the master branch. Once you have cloned BitBake, + you should use the latest stable branch for development since the + master branch is for BitBake development and might contain less + stable changes. + + You usually need a version of BitBake that matches the metadata you + are using. The metadata is generally backwards compatible but not + forward compatible. + + Here is an example that clones the BitBake repository: :: + + $ git clone git://git.openembedded.org/bitbake + + This command clones the BitBake + Git repository into a directory called ``bitbake``. Alternatively, + you can designate a directory after the ``git clone`` command if you + want to call the new directory something other than ``bitbake``. Here + is an example that names the directory ``bbdev``: :: + + $ git clone git://git.openembedded.org/bitbake bbdev + +- **Installation using your Distribution Package Management System:** + This method is not recommended because the BitBake version that is + provided by your distribution, in most cases, is several releases + behind a snapshot of the BitBake repository. + +- **Taking a snapshot of BitBake:** Downloading a snapshot of BitBake + from the source code repository gives you access to a known branch or + release of BitBake. + + .. note:: + + Cloning the Git repository, as described earlier, is the preferred + method for getting BitBake. Cloning the repository makes it easier + to update as patches are added to the stable branches. + + The following example downloads a snapshot of BitBake version 1.17.0: :: + + $ wget http://git.openembedded.org/bitbake/snapshot/bitbake-1.17.0.ta= r.gz + $ tar zxpvf bitbake-1.17.0.tar.gz + + After extraction of the tarball using + the tar utility, you have a directory entitled ``bitbake-1.17.0``. + +- **Using the BitBake that Comes With Your Build Checkout:** A final + possibility for getting a copy of BitBake is that it already comes + with your checkout of a larger BitBake-based build system, such as + Poky. Rather than manually checking out individual layers and gluing + them together yourself, you can check out an entire build system. The + checkout will already include a version of BitBake that has been + thoroughly tested for compatibility with the other components. For + information on how to check out a particular BitBake-based build + system, consult that build system's supporting documentation. + +.. _bitbake-user-manual-command: + +The BitBake Command +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +The ``bitbake`` command is the primary interface to the BitBake tool. +This section presents the BitBake command syntax and provides several +execution examples. + +Usage and syntax +---------------- + +Following is the usage and syntax for BitBake: :: + + $ bitbake -h + Usage: bitbake [options] [recipename/target recipe:do_task ...] + + Executes the specified task (default is 'build') for a given set of= target recipes (.bb files). + It is assumed there is a conf/bblayers.conf available in cwd or in = BBPATH which + will provide the layer, BBFILES and other configuration information. + + Options: + --version show program's version number and exit + -h, --help show this help message and exit + -b BUILDFILE, --buildfile=3DBUILDFILE + Execute tasks from a specific .bb recipe direct= ly. + WARNING: Does not handle any dependencies from = other + recipes. + -k, --continue Continue as much as possible after an error. Wh= ile the + target that failed and anything depending on it= cannot + be built, as much as possible will be built bef= ore + stopping. + -f, --force Force the specified targets/task to run (invali= dating + any existing stamp file). + -c CMD, --cmd=3DCMD Specify the task to execute. The exact options + available depend on the metadata. Some examples= might + be 'compile' or 'populate_sysroot' or 'listtask= s' may + give a list of the tasks available. + -C INVALIDATE_STAMP, --clear-stamp=3DINVALIDATE_STAMP + Invalidate the stamp for the specified task suc= h as + 'compile' and then run the default task for the + specified target(s). + -r PREFILE, --read=3DPREFILE + Read the specified file before bitbake.conf. + -R POSTFILE, --postread=3DPOSTFILE + Read the specified file after bitbake.conf. + -v, --verbose Enable tracing of shell tasks (with 'set -x'). = Also + print bb.note(...) messages to stdout (in addit= ion to + writing them to ${T}/log.do_<task>). + -D, --debug Increase the debug level. You can specify this = more + than once. -D sets the debug level to 1, where = only + bb.debug(1, ...) messages are printed to stdout= ; -DD + sets the debug level to 2, where both bb.debug(= 1, ...) + and bb.debug(2, ...) messages are printed; etc. + Without -D, no debug messages are printed. Note= that + -D only affects output to stdout. All debug mes= sages + are written to ${T}/log.do_taskname, regardless= of the + debug level. + -q, --quiet Output less log message data to the terminal. Y= ou can + specify this more than once. + -n, --dry-run Don't execute, just go through the motions. + -S SIGNATURE_HANDLER, --dump-signatures=3DSIGNATURE_HANDLER + Dump out the signature construction information= , with + no task execution. The SIGNATURE_HANDLER parame= ter is + passed to the handler. Two common values are no= ne and + printdiff but the handler may define more/less.= none + means only dump the signature, printdiff means = compare + the dumped signature with the cached one. + -p, --parse-only Quit after parsing the BB recipes. + -s, --show-versions Show current and preferred versions of all reci= pes. + -e, --environment Show the global or per-recipe environment compl= ete + with information about where variables were + set/changed. + -g, --graphviz Save dependency tree information for the specif= ied + targets in the dot syntax. + -I EXTRA_ASSUME_PROVIDED, --ignore-deps=3DEXTRA_ASSUME_PROVIDED + Assume these dependencies don't exist and are a= lready + provided (equivalent to ASSUME_PROVIDED). Usefu= l to + make dependency graphs more appealing + -l DEBUG_DOMAINS, --log-domains=3DDEBUG_DOMAINS + Show debug logging for the specified logging do= mains + -P, --profile Profile the command and save reports. + -u UI, --ui=3DUI The user interface to use (knotty, ncurses or= taskexp + - default knotty). + --token=3DXMLRPCTOKEN Specify the connection token to be used when + connecting to a remote server. + --revisions-changed Set the exit code depending on whether upstream + floating revisions have changed or not. + --server-only Run bitbake without a UI, only starting a server + (cooker) process. + -B BIND, --bind=3DBIND The name/address for the bitbake xmlrpc serve= r to bind + to. + -T SERVER_TIMEOUT, --idle-timeout=3DSERVER_TIMEOUT + Set timeout to unload bitbake server due to + inactivity, set to -1 means no unload, default: + Environment variable BB_SERVER_TIMEOUT. + --no-setscene Do not run any setscene tasks. sstate will be i= gnored + and everything needed, built. + --setscene-only Only run setscene tasks, don't run any real tas= ks. + --remote-server=3DREMOTE_SERVER + Connect to the specified server. + -m, --kill-server Terminate any running bitbake server. + --observe-only Connect to a server as an observing-only client. + --status-only Check the status of the remote bitbake server. + -w WRITEEVENTLOG, --write-log=3DWRITEEVENTLOG + Writes the event log of the build to a bitbake = event + json file. Use '' (empty string) to assign the = name + automatically. + --runall=3DRUNALL Run the specified task for any recipe in the = taskgraph + of the specified target (even if it wouldn't ot= herwise + have run). + --runonly=3DRUNONLY Run only the specified task within the taskgr= aph of + the specified targets (and any task dependencie= s those + tasks may have). + +.. _bitbake-examples: + +Examples +-------- + +This section presents some examples showing how to use BitBake. + +.. _example-executing-a-task-against-a-single-recipe: + +Executing a Task Against a Single Recipe +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Executing tasks for a single recipe file is relatively simple. You +specify the file in question, and BitBake parses it and executes the +specified task. If you do not specify a task, BitBake executes the +default task, which is "build". BitBake obeys inter-task dependencies +when doing so. + +The following command runs the build task, which is the default task, on +the ``foo_1.0.bb`` recipe file: :: + + $ bitbake -b foo_1.0.bb + +The following command runs the clean task on the ``foo.bb`` recipe file: :: + + $ bitbake -b foo.bb -c clean + +.. note:: + + The "-b" option explicitly does not handle recipe dependencies. Other + than for debugging purposes, it is instead recommended that you use + the syntax presented in the next section. + +Executing Tasks Against a Set of Recipe Files +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +There are a number of additional complexities introduced when one wants +to manage multiple ``.bb`` files. Clearly there needs to be a way to +tell BitBake what files are available and, of those, which you want to +execute. There also needs to be a way for each recipe to express its +dependencies, both for build-time and runtime. There must be a way for +you to express recipe preferences when multiple recipes provide the same +functionality, or when there are multiple versions of a recipe. + +The ``bitbake`` command, when not using "--buildfile" or "-b" only +accepts a "PROVIDES". You cannot provide anything else. By default, a +recipe file generally "PROVIDES" its "packagename" as shown in the +following example: :: + + $ bitbake foo + +This next example "PROVIDES" the +package name and also uses the "-c" option to tell BitBake to just +execute the ``do_clean`` task: :: + + $ bitbake -c clean foo + +Executing a List of Task and Recipe Combinations +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The BitBake command line supports specifying different tasks for +individual targets when you specify multiple targets. For example, +suppose you had two targets (or recipes) ``myfirstrecipe`` and +``mysecondrecipe`` and you needed BitBake to run ``taskA`` for the first +recipe and ``taskB`` for the second recipe: :: + + $ bitbake myfirstrecipe:do_taskA mysecondrecipe:do_taskB + +Generating Dependency Graphs +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +BitBake is able to generate dependency graphs using the ``dot`` syntax. +You can convert these graphs into images using the ``dot`` tool from +`Graphviz `__. + +When you generate a dependency graph, BitBake writes two files to the +current working directory: + +- ``task-depends.dot``: Shows dependencies between tasks. These + dependencies match BitBake's internal task execution list. + +- ``pn-buildlist``: Shows a simple list of targets that are to be + built. + +To stop depending on common depends, use the "-I" depend option and +BitBake omits them from the graph. Leaving this information out can +produce more readable graphs. This way, you can remove from the graph +``DEPENDS`` from inherited classes such as ``base.bbclass``. + +Here are two examples that create dependency graphs. The second example +omits depends common in OpenEmbedded from the graph: :: + + $ bitbake -g foo + + $ bitbake -g -I virtual/kernel -I eglibc foo + +Executing a Multiple Configuration Build +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +BitBake is able to build multiple images or packages using a single +command where the different targets require different configurations +(multiple configuration builds). Each target, in this scenario, is +referred to as a "multiconfig". + +To accomplish a multiple configuration build, you must define each +target's configuration separately using a parallel configuration file in +the build directory. The location for these multiconfig configuration +files is specific. They must reside in the current build directory in a +sub-directory of ``conf`` named ``multiconfig``. Following is an example +for two separate targets: + +.. image:: figures/bb_multiconfig_files.png + :align: center + +The reason for this required file hierarchy is because the ``BBPATH`` +variable is not constructed until the layers are parsed. Consequently, +using the configuration file as a pre-configuration file is not possible +unless it is located in the current working directory. + +Minimally, each configuration file must define the machine and the +temporary directory BitBake uses for the build. Suggested practice +dictates that you do not overlap the temporary directories used during +the builds. + +Aside from separate configuration files for each target, you must also +enable BitBake to perform multiple configuration builds. Enabling is +accomplished by setting the +:term:`BBMULTICONFIG` variable in the +``local.conf`` configuration file. As an example, suppose you had +configuration files for ``target1`` and ``target2`` defined in the build +directory. The following statement in the ``local.conf`` file both +enables BitBake to perform multiple configuration builds and specifies +the two extra multiconfigs: :: + + BBMULTICONFIG =3D "target1 target2" + +Once the target configuration files are in place and BitBake has been +enabled to perform multiple configuration builds, use the following +command form to start the builds: :: + + $ bitbake [mc:multiconfigname:]target [[[mc:multiconfigname:]target] ...= ] + +Here is an example for two extra multiconfigs: ``target1`` and ``target2``= : :: + + $ bitbake mc::target mc:target1:target mc:target2:target + +.. _bb-enabling-multiple-configuration-build-dependencies: + +Enabling Multiple Configuration Build Dependencies +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Sometimes dependencies can exist between targets (multiconfigs) in a +multiple configuration build. For example, suppose that in order to +build an image for a particular architecture, the root filesystem of +another build for a different architecture needs to exist. In other +words, the image for the first multiconfig depends on the root +filesystem of the second multiconfig. This dependency is essentially +that the task in the recipe that builds one multiconfig is dependent on +the completion of the task in the recipe that builds another +multiconfig. + +To enable dependencies in a multiple configuration build, you must +declare the dependencies in the recipe using the following statement +form: :: + + task_or_package[mcdepends] =3D "mc:from_multiconfig:to_multiconfig:recip= e_name:task_on_which_to_depend" + +To better show how to use this statement, consider an example with two +multiconfigs: ``target1`` and ``target2``: :: + + image_task[mcdepends] =3D "mc:target1:target2:image2:rootfs_task" + +In this example, the +``from_multiconfig`` is "target1" and the ``to_multiconfig`` is "target2".= The +task on which the image whose recipe contains image_task depends on the +completion of the rootfs_task used to build out image2, which is +associated with the "target2" multiconfig. + +Once you set up this dependency, you can build the "target1" multiconfig +using a BitBake command as follows: :: + + $ bitbake mc:target1:image1 + +This command executes all the tasks needed to create ``image1`` for the "t= arget1" +multiconfig. Because of the dependency, BitBake also executes through +the ``rootfs_task`` for the "target2" multiconfig build. + +Having a recipe depend on the root filesystem of another build might not +seem that useful. Consider this change to the statement in the image1 +recipe: :: + + image_task[mcdepends] =3D "mc:target1:target2:image2:image_task" + +In this case, BitBake must create ``image2`` for the "target2" build since +the "target1" build depends on it. + +Because "target1" and "target2" are enabled for multiple configuration +builds and have separate configuration files, BitBake places the +artifacts for each build in the respective temporary build directories. diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml = b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml deleted file mode 100644 index 995c2fa7..00000000 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.xml +++ /dev/null @@ -1,891 +0,0 @@ - - - - Overview - - - Welcome to the BitBake User Manual. - This manual provides information on the BitBake tool. - The information attempts to be as independent as possible regarding - systems that use BitBake, such as OpenEmbedded and the - Yocto Project. - In some cases, scenarios or examples within the context of - a build system are used in the manual to help with understanding. - For these cases, the manual clearly states the context. - - -
    - Introduction - - - Fundamentally, BitBake is a generic task execution - engine that allows shell and Python tasks to be run - efficiently and in parallel while working within - complex inter-task dependency constraints. - One of BitBake's main users, OpenEmbedded, takes this core - and builds embedded Linux software stacks using - a task-oriented approach. - - - - Conceptually, BitBake is similar to GNU Make in - some regards but has significant differences: - - - BitBake executes tasks according to provided - metadata that builds up the tasks. - Metadata is stored in recipe (.bb) - and related recipe "append" (.bbappend) - files, configuration (.conf) and - underlying include (.inc) files, = and - in class (.bbclass) files. - The metadata provides - BitBake with instructions on what tasks to run and - the dependencies between those tasks. - - - BitBake includes a fetcher library for obtaining source - code from various places such as local files, source c= ontrol - systems, or websites. - - - The instructions for each unit to be built (e.g. a pie= ce - of software) are known as "recipe" files and - contain all the information about the unit - (dependencies, source file locations, checksums, descr= iption - and so on). - - - BitBake includes a client/server abstraction and can - be used from a command line or used as a service over - XML-RPC and has several different user interfaces. - - - -
    - -
    - History and Goals - - - BitBake was originally a part of the OpenEmbedded project. - It was inspired by the Portage package management system - used by the Gentoo Linux distribution. - On December 7, 2004, OpenEmbedded project team member - Chris Larson split the project into two distinct pieces: - - BitBake, a generic task executor - OpenEmbedded, a metadata set utilized by - BitBake - - Today, BitBake is the primary basis of the - OpenEmbedded - project, which is being used to build and maintain Linux - distributions such as the - Angstrom = Distribution, - and which is also being used as the build tool for Linux proje= cts - such as the - Yocto Project. - - - - Prior to BitBake, no other build tool adequately met the needs= of - an aspiring embedded Linux distribution. - All of the build systems used by traditional desktop Linux - distributions lacked important functionality, and none of the - ad hoc Buildroot-based systems, prevalent in the - embedded space, were scalable or maintainable. - - - - Some important original goals for BitBake were: - - - Handle cross-compilation. - - - Handle inter-package dependencies (build time on - target architecture, build time on native - architecture, and runtime). - - - Support running any number of tasks within a given - package, including, but not limited to, fetching - upstream sources, unpacking them, patching them, - configuring them, and so forth. - - - Be Linux distribution agnostic for both build and - target systems. - - - Be architecture agnostic. - - - Support multiple build and target operating systems - (e.g. Cygwin, the BSDs, and so forth). - - - Be self-contained, rather than tightly - integrated into the build machine's root - filesystem. - - - Handle conditional metadata on the target architecture, - operating system, distribution, and machine. - - - Be easy to use the tools to supply local metadata and = packages - against which to operate. - - - Be easy to use BitBake to collaborate between multiple - projects for their builds. - - - Provide an inheritance mechanism to share - common metadata between many packages. - - - Over time it became apparent that some further requirements - were necessary: - - - Handle variants of a base recipe (e.g. native, sdk, - and multilib). - - - Split metadata into layers and allow layers - to enhance or override other layers. - - - Allow representation of a given set of input variables - to a task as a checksum. - Based on that checksum, allow acceleration of builds - with prebuilt components. - - - BitBake satisfies all the original requirements and many more - with extensions being made to the basic functionality to - reflect the additional requirements. - Flexibility and power have always been the priorities. - BitBake is highly extensible and supports embedded Python code= and - execution of any arbitrary tasks. - -
    - -
    - Concepts - - - BitBake is a program written in the Python language. - At the highest level, BitBake interprets metadata, decides - what tasks are required to run, and executes those tasks. - Similar to GNU Make, BitBake controls how software is - built. - GNU Make achieves its control through "makefiles", while - BitBake uses "recipes". - - - - BitBake extends the capabilities of a simple - tool like GNU Make by allowing for the definition of much more - complex tasks, such as assembling entire embedded Linux - distributions. - - - - The remainder of this section introduces several concepts - that should be understood in order to better leverage - the power of BitBake. - - -
    - Recipes - - - BitBake Recipes, which are denoted by the file extension - .bb, are the most basic metadata file= s. - These recipe files provide BitBake with the following: - - Descriptive information about the - package (author, homepage, license, and so on) - The version of the recipe - Existing dependencies (both build - and runtime dependencies) - Where the source code resides and - how to fetch it - Whether the source code requires - any patches, where to find them, and how to apply - them - How to configure and compile the - source code - How to assemble the generated artifact= s into - one or more installable packages - Where on the target machine to install= the - package or packages created - - - - - Within the context of BitBake, or any project utilizing Bi= tBake - as its build system, files with the .bb - extension are referred to as recipes. - - The term "package" is also commonly used to describe r= ecipes. - However, since the same word is used to describe packa= ged - output from a project, it is best to maintain a single - descriptive term - "recipes". - Put another way, a single "recipe" file is quite capab= le - of generating a number of related but separately insta= llable - "packages". - In fact, that ability is fairly common. - - -
    - -
    - Configuration Files - - - Configuration files, which are denoted by the - .conf extension, define - various configuration variables that govern the project's = build - process. - These files fall into several areas that define - machine configuration, distribution configuration, - possible compiler tuning, general common - configuration, and user configuration. - The main configuration file is the sample - bitbake.conf file, which is - located within the BitBake source tree - conf directory. - -
    - -
    - Classes - - - Class files, which are denoted by the - .bbclass extension, contain - information that is useful to share between metadata files. - The BitBake source tree currently comes with one class met= adata file - called base.bbclass. - You can find this file in the - classes directory. - The base.bbclass class files is speci= al since it - is always included automatically for all recipes - and classes. - This class contains definitions for standard basic tasks s= uch - as fetching, unpacking, configuring (empty by default), - compiling (runs any Makefile present), installing (empty by - default) and packaging (empty by default). - These tasks are often overridden or extended by other clas= ses - added during the project development process. - -
    - -
    - Layers - - - Layers allow you to isolate different types of - customizations from each other. - While you might find it tempting to keep everything in one= layer - when working on a single project, the more modular - your metadata, the easier it is to cope with future change= s. - - - - To illustrate how you can use layers to keep things modula= r, - consider customizations you might make to support a specif= ic target machine. - These types of customizations typically reside in a specia= l layer, - rather than a general layer, called a Board Sup= port Package (BSP) - layer. - Furthermore, the machine customizations should be isolated= from - recipes and metadata that support a new GUI environment, f= or - example. - This situation gives you a couple of layers: one for the m= achine - configurations and one for the GUI environment. - It is important to understand, however, that the BSP layer= can still - make machine-specific additions to recipes within - the GUI environment layer without polluting the GUI layer = itself - with those machine-specific changes. - You can accomplish this through a recipe that is a BitBake= append - (.bbappend) file. - -
    - -
    - Append Files - - - Append files, which are files that have the - .bbappend file extension, extend or - override information in an existing recipe file. - - - - BitBake expects every append file to have a corresponding = recipe file. - Furthermore, the append file and corresponding recipe file - must use the same root filename. - The filenames can differ only in the file type suffix used - (e.g. formfactor_0.0.bb and - formfactor_0.0.bbappend). - - - - Information in append files extends or - overrides the information in the underlying, - similarly-named recipe files. - - - - When you name an append file, you can use the - "%" wildcard character to allow for m= atching - recipe names. - For example, suppose you have an append file named - as follows: - - busybox_1.21.%.bbappend - - That append file would match any busybox_1.21.x.bb - version of the recipe. - So, the append file would match the following recipe names: - - busybox_1.21.1.bb - busybox_1.21.2.bb - busybox_1.21.3.bb - - Important - The use of the "%" character - is limited in that it only works directly in front of = the - .bbappend portion of the append f= ile's - name. - You cannot use the wildcard character in any other - location of the name. - - If the busybox recipe was updated to - busybox_1.3.0.bb, the append name wou= ld not - match. - However, if you named the append file - busybox_1.%.bbappend, then you would = have a match. - - - - In the most general case, you could name the append file s= omething as - simple as busybox_%.bbappend to be en= tirely - version independent. - -
    -
    - -
    - Obtaining BitBake - - - You can obtain BitBake several different ways: - - Cloning BitBake: - Using Git to clone the BitBake source code repository - is the recommended method for obtaining BitBake. - Cloning the repository makes it easy to get bug fixes - and have access to stable branches and the master - branch. - Once you have cloned BitBake, you should use - the latest stable - branch for development since the master branch is for - BitBake development and might contain less stable chan= ges. - - You usually need a version of BitBake - that matches the metadata you are using. - The metadata is generally backwards compatible but - not forward compatible. - Here is an example that clones the BitBake repos= itory: - - $ git clone git://git.openembedded.org/bitbake - - This command clones the BitBake Git repository into a - directory called bitbake. - Alternatively, you can - designate a directory after the - git clone command - if you want to call the new directory something - other than bitbake. - Here is an example that names the directory - bbdev: - - $ git clone git://git.openembedded.org/bitbake bbdev - - Installation using your Distribu= tion - Package Management System: - This method is not - recommended because the BitBake version that is - provided by your distribution, in most cases, - is several - releases behind a snapshot of the BitBake repository. - - Taking a snapshot of BitBake: - Downloading a snapshot of BitBake from the - source code repository gives you access to a known - branch or release of BitBake. - - Cloning the Git repository, as described earlier, - is the preferred method for getting BitBake. - Cloning the repository makes it easier to update = as - patches are added to the stable branches. - - The following example downloads a snapshot of - BitBake version 1.17.0: - - $ wget http://git.openembedded.org/bitbake/snapshot/bitbake-1.17.0.ta= r.gz - $ tar zxpvf bitbake-1.17.0.tar.gz - - After extraction of the tarball using the tar utility, - you have a directory entitled - bitbake-1.17.0. - - Using the BitBake that Comes Wit= h Your - Build Checkout: - A final possibility for getting a copy of BitBake is t= hat it - already comes with your checkout of a larger BitBake-b= ased build - system, such as Poky. - Rather than manually checking out individual layers and - gluing them together yourself, you can check - out an entire build system. - The checkout will already include a version of BitBake= that - has been thoroughly tested for compatibility with the = other - components. - For information on how to check out a particular BitBa= ke-based - build system, consult that build system's supporting d= ocumentation. - - - -
    - -
    - The BitBake Command - - - The bitbake command is the primary interf= ace - to the BitBake tool. - This section presents the BitBake command syntax and provides - several execution examples. - - -
    - Usage and syntax - - - Following is the usage and syntax for BitBake: - - $ bitbake -h - Usage: bitbake [options] [recipename/target recipe:do_task ...] - - Executes the specified task (default is 'build') for a given set = of target recipes (.bb files). - It is assumed there is a conf/bblayers.conf available in cwd or i= n BBPATH which - will provide the layer, BBFILES and other configuration informati= on. - - Options: - --version show program's version number and exit - -h, --help show this help message and exit - -b BUILDFILE, --buildfile=3DBUILDFILE - Execute tasks from a specific .bb recipe dire= ctly. - WARNING: Does not handle any dependencies fro= m other - recipes. - -k, --continue Continue as much as possible after an error. = While the - target that failed and anything depending on = it cannot - be built, as much as possible will be built b= efore - stopping. - -f, --force Force the specified targets/task to run (inva= lidating - any existing stamp file). - -c CMD, --cmd=3DCMD Specify the task to execute. The exact opti= ons - available depend on the metadata. Some exampl= es might - be 'compile' or 'populate_sysroot' or 'listta= sks' may - give a list of the tasks available. - -C INVALIDATE_STAMP, --clear-stamp=3DINVALIDATE_STAMP - Invalidate the stamp for the specified task s= uch as - 'compile' and then run the default task for t= he - specified target(s). - -r PREFILE, --read=3DPREFILE - Read the specified file before bitbake.conf. - -R POSTFILE, --postread=3DPOSTFILE - Read the specified file after bitbake.conf. - -v, --verbose Enable tracing of shell tasks (with 'set -x')= . Also - print bb.note(...) messages to stdout (in add= ition to - writing them to ${T}/log.do_<task>). - -D, --debug Increase the debug level. You can specify thi= s more - than once. -D sets the debug level to 1, wher= e only - bb.debug(1, ...) messages are printed to stdo= ut; -DD - sets the debug level to 2, where both bb.debu= g(1, ...) - and bb.debug(2, ...) messages are printed; et= c. - Without -D, no debug messages are printed. No= te that - -D only affects output to stdout. All debug m= essages - are written to ${T}/log.do_taskname, regardle= ss of the - debug level. - -q, --quiet Output less log message data to the terminal.= You can - specify this more than once. - -n, --dry-run Don't execute, just go through the motions. - -S SIGNATURE_HANDLER, --dump-signatures=3DSIGNATURE_HANDLER - Dump out the signature construction informati= on, with - no task execution. The SIGNATURE_HANDLER para= meter is - passed to the handler. Two common values are = none and - printdiff but the handler may define more/les= s. none - means only dump the signature, printdiff mean= s compare - the dumped signature with the cached one. - -p, --parse-only Quit after parsing the BB recipes. - -s, --show-versions Show current and preferred versions of all re= cipes. - -e, --environment Show the global or per-recipe environment com= plete - with information about where variables were - set/changed. - -g, --graphviz Save dependency tree information for the spec= ified - targets in the dot syntax. - -I EXTRA_ASSUME_PROVIDED, --ignore-deps=3DEXTRA_ASSUME_PROVIDED - Assume these dependencies don't exist and are= already - provided (equivalent to ASSUME_PROVIDED). Use= ful to - make dependency graphs more appealing - -l DEBUG_DOMAINS, --log-domains=3DDEBUG_DOMAINS - Show debug logging for the specified logging = domains - -P, --profile Profile the command and save reports. - -u UI, --ui=3DUI The user interface to use (knotty, ncurses = or taskexp - - default knotty). - --token=3DXMLRPCTOKEN Specify the connection token to be used when - connecting to a remote server. - --revisions-changed Set the exit code depending on whether upstre= am - floating revisions have changed or not. - --server-only Run bitbake without a UI, only starting a ser= ver - (cooker) process. - -B BIND, --bind=3DBIND The name/address for the bitbake xmlrpc ser= ver to bind - to. - -T SERVER_TIMEOUT, --idle-timeout=3DSERVER_TIMEOUT - Set timeout to unload bitbake server due to - inactivity, set to -1 means no unload, defaul= t: - Environment variable BB_SERVER_TIMEOUT. - --no-setscene Do not run any setscene tasks. sstate will be= ignored - and everything needed, built. - --setscene-only Only run setscene tasks, don't run any real t= asks. - --remote-server=3DREMOTE_SERVER - Connect to the specified server. - -m, --kill-server Terminate any running bitbake server. - --observe-only Connect to a server as an observing-only clie= nt. - --status-only Check the status of the remote bitbake server. - -w WRITEEVENTLOG, --write-log=3DWRITEEVENTLOG - Writes the event log of the build to a bitbak= e event - json file. Use '' (empty string) to assign th= e name - automatically. - --runall=3DRUNALL Run the specified task for any recipe in th= e taskgraph - of the specified target (even if it wouldn't = otherwise - have run). - --runonly=3DRUNONLY Run only the specified task within the task= graph of - the specified targets (and any task dependenc= ies those - tasks may have). - - -
    - -
    - Examples - - - This section presents some examples showing how to use Bit= Bake. - - -
    - Executing a Task Against a Single Recipe - - - Executing tasks for a single recipe file is relatively= simple. - You specify the file in question, and BitBake parses - it and executes the specified task. - If you do not specify a task, BitBake executes the def= ault - task, which is "build=E2=80=9D. - BitBake obeys inter-task dependencies when doing - so. - - - - The following command runs the build task, which is - the default task, on the foo_1.0.bb - recipe file: - - $ bitbake -b foo_1.0.bb - - The following command runs the clean task on the - foo.bb recipe file: - - $ bitbake -b foo.bb -c clean - - - The "-b" option explicitly does not handle recipe - dependencies. - Other than for debugging purposes, it is instead - recommended that you use the syntax presented in t= he - next section. - - -
    - -
    - Executing Tasks Against a Set of Recipe Files</titl= e> - - <para> - There are a number of additional complexities introduc= ed - when one wants to manage multiple <filename>.bb</filen= ame> - files. - Clearly there needs to be a way to tell BitBake what - files are available and, of those, which you - want to execute. - There also needs to be a way for each recipe - to express its dependencies, both for build-time and - runtime. - There must be a way for you to express recipe preferen= ces - when multiple recipes provide the same functionality, = or when - there are multiple versions of a recipe. - </para> - - <para> - The <filename>bitbake</filename> command, when not usi= ng - "--buildfile" or "-b" only accepts a "PROVIDES". - You cannot provide anything else. - By default, a recipe file generally "PROVIDES" its - "packagename" as shown in the following example: - <literallayout class=3D'monospaced'> - $ bitbake foo - </literallayout> - This next example "PROVIDES" the package name and also= uses - the "-c" option to tell BitBake to just execute the - <filename>do_clean</filename> task: - <literallayout class=3D'monospaced'> - $ bitbake -c clean foo - </literallayout> - </para> - </section> - - <section id=3D'executing-a-list-of-task-and-recipe-combination= s'> - <title>Executing a List of Task and Recipe Combinations</t= itle> - - <para> - The BitBake command line supports specifying different - tasks for individual targets when you specify multiple - targets. - For example, suppose you had two targets (or recipes) - <filename>myfirstrecipe</filename> and - <filename>mysecondrecipe</filename> and you needed - BitBake to run <filename>taskA</filename> for the first - recipe and <filename>taskB</filename> for the second - recipe: - <literallayout class=3D'monospaced'> - $ bitbake myfirstrecipe:do_taskA mysecondrecipe:do_taskB - </literallayout> - </para> - </section> - - <section id=3D'generating-dependency-graphs'> - <title>Generating Dependency Graphs - - - BitBake is able to generate dependency graphs using - the dot syntax. - You can convert these graphs into images using the - dot tool from - Graphviz. - - - - When you generate a dependency graph, BitBake writes t= wo files - to the current working directory: - - - task-depends.dot: - Shows dependencies between tasks. - These dependencies match BitBake's internal ta= sk execution list. - - - pn-buildlist: - Shows a simple list of targets that are to be = built. - - - - - - To stop depending on common depends, use the "-I" depe= nd - option and BitBake omits them from the graph. - Leaving this information out can produce more readable= graphs. - This way, you can remove from the graph - DEPENDS from inherited classes - such as base.bbclass. - - - - Here are two examples that create dependency graphs. - The second example omits depends common in OpenEmbedde= d from - the graph: - - $ bitbake -g foo - - $ bitbake -g -I virtual/kernel -I eglibc foo - - -
    - -
    - Executing a Multiple Configuration Build - - - BitBake is able to build multiple images or packages - using a single command where the different targets - require different configurations (multiple configurati= on - builds). - Each target, in this scenario, is referred to as a - "multiconfig". - - - - To accomplish a multiple configuration build, you must - define each target's configuration separately using - a parallel configuration file in the build directory. - The location for these multiconfig configuration files - is specific. - They must reside in the current build directory in - a sub-directory of conf named - multiconfig. - Following is an example for two separate targets: - - - - - The reason for this required file hierarchy - is because the BBPATH variable - is not constructed until the layers are parsed. - Consequently, using the configuration file as a - pre-configuration file is not possible unless it is - located in the current working directory. - - - - Minimally, each configuration file must define the - machine and the temporary directory BitBake uses - for the build. - Suggested practice dictates that you do not - overlap the temporary directories used during the - builds. - - - - Aside from separate configuration files for each - target, you must also enable BitBake to perform multip= le - configuration builds. - Enabling is accomplished by setting the - BBMUL= TICONFIG - variable in the local.conf - configuration file. - As an example, suppose you had configuration files - for target1 and - target2 defined in the build - directory. - The following statement in the - local.conf file both enables - BitBake to perform multiple configuration builds and - specifies the two extra multiconfigs: - - BBMULTICONFIG =3D "target1 target2" - - - - - Once the target configuration files are in place and - BitBake has been enabled to perform multiple configura= tion - builds, use the following command form to start the - builds: - - $ bitbake [mc:multiconfigname:]target [[[mc:multiconfigname:]target] ... ] - - Here is an example for two extra multiconfigs: - target1 and - target2: - - $ bitbake mc::target mc:target1:target mc:target2:target - - -
    - -
    - Enabling Multiple Configuration Build Dependencies<= /title> - - <para> - Sometimes dependencies can exist between targets - (multiconfigs) in a multiple configuration build. - For example, suppose that in order to build an image - for a particular architecture, the root filesystem of - another build for a different architecture needs to - exist. - In other words, the image for the first multiconfig de= pends - on the root filesystem of the second multiconfig. - This dependency is essentially that the task in the re= cipe - that builds one multiconfig is dependent on the - completion of the task in the recipe that builds - another multiconfig. - </para> - - <para> - To enable dependencies in a multiple configuration - build, you must declare the dependencies in the recipe - using the following statement form: - <literallayout class=3D'monospaced'> - <replaceable>task_or_package</replaceable>[mcdepends] =3D "mc:<replac= eable>from_multiconfig</replaceable>:<replaceable>to_multiconfig</replaceab= le>:<replaceable>recipe_name</replaceable>:<replaceable>task_on_which_to_de= pend</replaceable>" - </literallayout> - To better show how to use this statement, consider an - example with two multiconfigs: <filename>target1</file= name> - and <filename>target2</filename>: - <literallayout class=3D'monospaced'> - <replaceable>image_task</replaceable>[mcdepends] =3D "mc:target1:targ= et2:<replaceable>image2</replaceable>:<replaceable>rootfs_task</replaceable= >" - </literallayout> - In this example, the - <replaceable>from_multiconfig</replaceable> is "target= 1" and - the <replaceable>to_multiconfig</replaceable> is "targ= et2". - The task on which the image whose recipe contains - <replaceable>image_task</replaceable> depends on the - completion of the <replaceable>rootfs_task</replaceabl= e> - used to build out <replaceable>image2</replaceable>, w= hich - is associated with the "target2" multiconfig. - </para> - - <para> - Once you set up this dependency, you can build the - "target1" multiconfig using a BitBake command as follow= s: - <literallayout class=3D'monospaced'> - $ bitbake mc:target1:<replaceable>image1</replaceable> - </literallayout> - This command executes all the tasks needed to create - <replaceable>image1</replaceable> for the "target1" - multiconfig. - Because of the dependency, BitBake also executes through - the <replaceable>rootfs_task</replaceable> for the "tar= get2" - multiconfig build. - </para> - - <para> - Having a recipe depend on the root filesystem of another - build might not seem that useful. - Consider this change to the statement in the - <replaceable>image1</replaceable> recipe: - <literallayout class=3D'monospaced'> - <replaceable>image_task</replaceable>[mcdepends] =3D "mc:target1:targ= et2:<replaceable>image2</replaceable>:<replaceable>image_task</replaceable>" - </literallayout> - In this case, BitBake must create - <replaceable>image2</replaceable> for the "target2" - build since the "target1" build depends on it. - </para> - - <para> - Because "target1" and "target2" are enabled for multiple - configuration builds and have separate configuration - files, BitBake places the artifacts for each build in t= he - respective temporary build directories. - </para> - </section> - </section> - </section> -</chapter> diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.r= st b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst new file mode 100644 index 00000000..d4190c26 --- /dev/null +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst @@ -0,0 +1,1980 @@ +.. SPDX-License-Identifier: CC-BY-2.5 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Syntax and Operators +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +| + +BitBake files have their own syntax. The syntax has similarities to +several other languages but also has some unique features. This section +describes the available syntax and operators as well as provides +examples. + +Basic Syntax +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +This section provides some basic syntax examples. + +Basic Variable Setting +---------------------- + +The following example sets ``VARIABLE`` to "value". This assignment +occurs immediately as the statement is parsed. It is a "hard" +assignment. :: + + VARIABLE =3D "value" + +As expected, if you include leading or +trailing spaces as part of an assignment, the spaces are retained: :: + + VARIABLE =3D " value" + VARIABLE =3D "value " + +Setting ``VARIABLE`` to "" sets +it to an empty string, while setting the variable to " " sets it to a +blank space (i.e. these are not the same values). :: + + VARIABLE =3D "" + VARIABLE =3D " " + +You can use single quotes instead of double quotes when setting a +variable's value. Doing so allows you to use values that contain the +double quote character: :: + + VARIABLE =3D 'I have a " in my value' + +.. note:: + + Unlike in Bourne shells, single quotes work identically to double + quotes in all other ways. They do not suppress variable expansions. + +Modifying Existing Variables +---------------------------- + +Sometimes you need to modify existing variables. Following are some +cases where you might find you want to modify an existing variable: + +- Customize a recipe that uses the variable. + +- Change a variable's default value used in a ``*.bbclass`` file. + +- Change the variable in a ``*.bbappend`` file to override the variable + in the original recipe. + +- Change the variable in a configuration file so that the value + overrides an existing configuration. + +Changing a variable value can sometimes depend on how the value was +originally assigned and also on the desired intent of the change. In +particular, when you append a value to a variable that has a default +value, the resulting value might not be what you expect. In this case, +the value you provide might replace the value rather than append to the +default value. + +If after you have changed a variable's value and something unexplained +occurs, you can use BitBake to check the actual value of the suspect +variable. You can make these checks for both configuration and recipe +level changes: + +- For configuration changes, use the following: :: + + $ bitbake -e + + This + command displays variable values after the configuration files (i.e. + ``local.conf``, ``bblayers.conf``, ``bitbake.conf`` and so forth) + have been parsed. + + .. note:: + + Variables that are exported to the environment are preceded by the + string "export" in the command's output. + +- For recipe changes, use the following: :: + + $ bitbake recipe -e \| grep VARIABLE=3D" + + This command checks to see if the variable actually makes + it into a specific recipe. + +Line Joining +------------ + +Outside of :ref:`functions <bitbake-user-manual/bitbake-user-manual-metada= ta:functions>`, +BitBake joins any line ending in +a backslash character ("\") with the following line before parsing +statements. The most common use for the "\" character is to split +variable assignments over multiple lines, as in the following example: :: + + FOO =3D "bar \ + baz \ + qaz" + +Both the "\" character and the newline +character that follow it are removed when joining lines. Thus, no +newline characters end up in the value of ``FOO``. + +Consider this additional example where the two assignments both assign +"barbaz" to ``FOO``: :: + + FOO =3D "barbaz" + FOO =3D "bar\ + baz" + +.. note:: + + BitBake does not interpret escape sequences like "\n" in variable + values. For these to have an effect, the value must be passed to some + utility that interprets escape sequences, such as + ``printf`` or ``echo -n``. + +Variable Expansion +------------------ + +Variables can reference the contents of other variables using a syntax +that is similar to variable expansion in Bourne shells. The following +assignments result in A containing "aval" and B evaluating to +"preavalpost". :: + + A =3D "aval" + B =3D "pre${A}post" + +.. note:: + + Unlike in Bourne shells, the curly braces are mandatory: Only ``${FOO}`= ` and not + ``$FOO`` is recognized as an expansion of ``FOO``. + +The "=3D" operator does not immediately expand variable references in the +right-hand side. Instead, expansion is deferred until the variable +assigned to is actually used. The result depends on the current values +of the referenced variables. The following example should clarify this +behavior: :: + + A =3D "${B} baz" + B =3D "${C} bar" + C =3D "foo" + *At this point, ${A} equals "foo bar baz"* + C =3D "qux" + *At this point, ${A} equals "qux bar baz"* + B =3D "norf" + *At this point, ${A} equals "norf baz"\* + +Contrast this behavior with the +:ref:`bitbake-user-manual/bitbake-user-manual-metadata:immediate variable +expansion (:=3D)` operator. + +If the variable expansion syntax is used on a variable that does not +exist, the string is kept as is. For example, given the following +assignment, ``BAR`` expands to the literal string "${FOO}" as long as +``FOO`` does not exist. :: + + BAR =3D "${FOO}" + +Setting a default value (?=3D) +---------------------------- + +You can use the "?=3D" operator to achieve a "softer" assignment for a +variable. This type of assignment allows you to define a variable if it +is undefined when the statement is parsed, but to leave the value alone +if the variable has a value. Here is an example: :: + + A ?=3D "aval" + +If ``A`` is +set at the time this statement is parsed, the variable retains its +value. However, if ``A`` is not set, the variable is set to "aval". + +.. note:: + + This assignment is immediate. Consequently, if multiple "?=3D" + assignments to a single variable exist, the first of those ends up + getting used. + +Setting a weak default value (??=3D) +---------------------------------- + +It is possible to use a "weaker" assignment than in the previous section +by using the "??=3D" operator. This assignment behaves identical to "?=3D" +except that the assignment is made at the end of the parsing process +rather than immediately. Consequently, when multiple "??=3D" assignments +exist, the last one is used. Also, any "=3D" or "?=3D" assignment will +override the value set with "??=3D". Here is an example: :: + + A ??=3D "somevalue" + A ??=3D "someothervalue" + +If ``A`` is set before the above statements are +parsed, the variable retains its value. If ``A`` is not set, the +variable is set to "someothervalue". + +Again, this assignment is a "lazy" or "weak" assignment because it does +not occur until the end of the parsing process. + +Immediate variable expansion (:=3D) +--------------------------------- + +The ":=3D" operator results in a variable's contents being expanded +immediately, rather than when the variable is actually used: :: + + T =3D "123" + A :=3D "test ${T}" + T =3D "456" + B :=3D "${T} ${C}" + C =3D "cval" + C :=3D "${C}append" + +In this example, ``A`` contains "test 123", even though the final value +of ``T`` is "456". The variable ``B`` will end up containing "456 +cvalappend". This is because references to undefined variables are +preserved as is during (immediate)expansion. This is in contrast to GNU +Make, where undefined variables expand to nothing. The variable ``C`` +contains "cvalappend" since ``${C}`` immediately expands to "cval". + +.. _appending-and-prepending: + +Appending (+=3D) and prepending (=3D+) With Spaces +---------------------------------------------- + +Appending and prepending values is common and can be accomplished using +the "+=3D" and "=3D+" operators. These operators insert a space between the +current value and prepended or appended value. + +These operators take immediate effect during parsing. Here are some +examples: :: + + B =3D "bval" + B +=3D "additionaldata" + C =3D "cval" + C =3D+ "test" + +The variable ``B`` contains "bval additionaldata" and ``C`` contains "test +cval". + +.. _appending-and-prepending-without-spaces: + +Appending (.=3D) and Prepending (=3D.) Without Spaces +------------------------------------------------- + +If you want to append or prepend values without an inserted space, use +the ".=3D" and "=3D." operators. + +These operators take immediate effect during parsing. Here are some +examples: :: + + B =3D "bval" + B .=3D "additionaldata" + C =3D "cval" + C =3D. "test" + +The variable ``B`` contains "bvaladditionaldata" and ``C`` contains +"testcval". + +Appending and Prepending (Override Style Syntax) +------------------------------------------------ + +You can also append and prepend a variable's value using an override +style syntax. When you use this syntax, no spaces are inserted. + +These operators differ from the ":=3D", ".=3D", "=3D.", "+=3D", and "=3D+" +operators in that their effects are applied at variable expansion time +rather than being immediately applied. Here are some examples: :: + + B =3D "bval" + B_append =3D " additional data" + C =3D "cval" + C_prepend =3D "additional data " + D =3D "dval" + D_append =3D "additional data" + +The variable ``B`` +becomes "bval additional data" and ``C`` becomes "additional data cval". +The variable ``D`` becomes "dvaladditional data". + +.. note:: + + You must control all spacing when you use the override syntax. + +It is also possible to append and prepend to shell functions and +BitBake-style Python functions. See the ":ref:`bitbake-user-manual/bitbake= -user-manual-metadata:shell functions`" and ":ref:`bitbake-user-manual/bitb= ake-user-manual-metadata:bitbake-style python functions`" +sections for examples. + +.. _removing-override-style-syntax: + +Removal (Override Style Syntax) +------------------------------- + +You can remove values from lists using the removal override style +syntax. Specifying a value for removal causes all occurrences of that +value to be removed from the variable. + +When you use this syntax, BitBake expects one or more strings. +Surrounding spaces and spacing are preserved. Here is an example: :: + + FOO =3D "123 456 789 123456 123 456 123 456" + FOO_remove =3D "123" + FOO_remove =3D "456" + FOO2 =3D " abc def ghi abcdef abc def abc def def" + FOO2_remove =3D "\ + def \ + abc \ + ghi \ + " + +The variable ``FOO`` becomes +" 789 123456 " and ``FOO2`` becomes " abcdef ". + +Like "_append" and "_prepend", "_remove" is applied at variable +expansion time. + +Override Style Operation Advantages +----------------------------------- + +An advantage of the override style operations "_append", "_prepend", and +"_remove" as compared to the "+=3D" and "=3D+" operators is that the +override style operators provide guaranteed operations. For example, +consider a class ``foo.bbclass`` that needs to add the value "val" to +the variable ``FOO``, and a recipe that uses ``foo.bbclass`` as follows: :: + + inherit foo + FOO =3D "initial" + +If ``foo.bbclass`` uses the "+=3D" operator, +as follows, then the final value of ``FOO`` will be "initial", which is +not what is desired: :: + + FOO +=3D "val" + +If, on the other hand, ``foo.bbclass`` +uses the "_append" operator, then the final value of ``FOO`` will be +"initial val", as intended: :: + + FOO_append =3D " val" + +.. note:: + + It is never necessary to use "+=3D" together with "_append". The follow= ing + sequence of assignments appends "barbaz" to FOO: :: + + FOO_append =3D "bar" + FOO_append =3D "baz" + + + The only effect of changing the second assignment in the previous + example to use "+=3D" would be to add a space before "baz" in the + appended value (due to how the "+=3D" operator works). + +Another advantage of the override style operations is that you can +combine them with other overrides as described in the +":ref:`bitbake-user-manual/bitbake-user-manual-metadata:conditional syntax= (overrides)`" section. + +Variable Flag Syntax +-------------------- + +Variable flags are BitBake's implementation of variable properties or +attributes. It is a way of tagging extra information onto a variable. +You can find more out about variable flags in general in the +":ref:`bitbake-user-manual/bitbake-user-manual-metadata:variable flags`" s= ection. + +You can define, append, and prepend values to variable flags. All the +standard syntax operations previously mentioned work for variable flags +except for override style syntax (i.e. "_prepend", "_append", and +"_remove"). + +Here are some examples showing how to set variable flags: :: + + FOO[a] =3D "abc" + FOO[b] =3D "123" + FOO[a] +=3D "456" + +The variable ``FOO`` has two flags: +``[a]`` and ``[b]``. The flags are immediately set to "abc" and "123", +respectively. The ``[a]`` flag becomes "abc 456". + +No need exists to pre-define variable flags. You can simply start using +them. One extremely common application is to attach some brief +documentation to a BitBake variable as follows: :: + + CACHE[doc] =3D "The directory holding the cache of the metadata." + +Inline Python Variable Expansion +-------------------------------- + +You can use inline Python variable expansion to set variables. Here is +an example: :: + + DATE =3D "${@time.strftime('%Y%m%d',time.gmtime())}" + +This example results in the ``DATE`` variable being set to the current dat= e. + +Probably the most common use of this feature is to extract the value of +variables from BitBake's internal data dictionary, ``d``. The following +lines select the values of a package name and its version number, +respectively: :: + + PN =3D "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)= [0] or 'defaultpkgname'}" + PV =3D "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)= [1] or '1.0'}" + +.. note:: + + Inline Python expressions work just like variable expansions insofar as= the + "=3D" and ":=3D" operators are concerned. Given the following assignmen= t, foo() + is called each time FOO is expanded: :: + + FOO =3D "${@foo()}" + + Contrast this with the following immediate assignment, where foo() is o= nly + called once, while the assignment is parsed: :: + + FOO :=3D "${@foo()}" + +For a different way to set variables with Python code during parsing, +see the +":ref:`bitbake-user-manual/bitbake-user-manual-metadata:anonymous python f= unctions`" section. + +Unsetting variables +------------------- + +It is possible to completely remove a variable or a variable flag from +BitBake's internal data dictionary by using the "unset" keyword. Here is +an example: :: + + unset DATE + unset do_fetch[noexec] + +These two statements remove the ``DATE`` and the ``do_fetch[noexec]`` flag. + +Providing Pathnames +------------------- + +When specifying pathnames for use with BitBake, do not use the tilde +("~") character as a shortcut for your home directory. Doing so might +cause BitBake to not recognize the path since BitBake does not expand +this character in the same way a shell would. + +Instead, provide a fuller path as the following example illustrates: :: + + BBLAYERS ?=3D " \ + /home/scott-lenovo/LayerA \ + " + +Exporting Variables to the Environment +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +You can export variables to the environment of running tasks by using +the ``export`` keyword. For example, in the following example, the +``do_foo`` task prints "value from the environment" when run: :: + + export ENV_VARIABLE + ENV_VARIABLE =3D "value from the environment" + + do_foo() { + bbplain "$ENV_VARIABLE" + } + +.. note:: + + BitBake does not expand ``$ENV_VARIABLE`` in this case because it lacks= the + obligatory ``{}`` . Rather, ``$ENV_VARIABLE`` is expanded by the shell. + +It does not matter whether ``export ENV_VARIABLE`` appears before or +after assignments to ``ENV_VARIABLE``. + +It is also possible to combine ``export`` with setting a value for the +variable. Here is an example: :: + + export ENV_VARIABLE =3D "variable-value" + +In the output of ``bitbake -e``, variables that are exported to the +environment are preceded by "export". + +Among the variables commonly exported to the environment are ``CC`` and +``CFLAGS``, which are picked up by many build systems. + +Conditional Syntax (Overrides) +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D + +BitBake uses :term:`OVERRIDES` to control what +variables are overridden after BitBake parses recipes and configuration +files. This section describes how you can use ``OVERRIDES`` as +conditional metadata, talks about key expansion in relationship to +``OVERRIDES``, and provides some examples to help with understanding. + +Conditional Metadata +-------------------- + +You can use ``OVERRIDES`` to conditionally select a specific version of +a variable and to conditionally append or prepend the value of a +variable. + +.. note:: + + Overrides can only use lower-case characters. Additionally, + underscores are not permitted in override names as they are used to + separate overrides from each other and from the variable name. + +- *Selecting a Variable:* The ``OVERRIDES`` variable is a + colon-character-separated list that contains items for which you want + to satisfy conditions. Thus, if you have a variable that is + conditional on "arm", and "arm" is in ``OVERRIDES``, then the + "arm"-specific version of the variable is used rather than the + non-conditional version. Here is an example: :: + + OVERRIDES =3D "architecture:os:machine" + TEST =3D "default" + TEST_os =3D "osspecific" + TEST_nooverride =3D "othercondvalue" + + In this example, the ``OVERRIDES`` + variable lists three overrides: "architecture", "os", and "machine". + The variable ``TEST`` by itself has a default value of "default". You + select the os-specific version of the ``TEST`` variable by appending + the "os" override to the variable (i.e. ``TEST_os``). + + To better understand this, consider a practical example that assumes + an OpenEmbedded metadata-based Linux kernel recipe file. The + following lines from the recipe file first set the kernel branch + variable ``KBRANCH`` to a default value, then conditionally override + that value based on the architecture of the build: :: + + KBRANCH =3D "standard/base" + KBRANCH_qemuarm =3D "standard/arm-versatile-926ejs" + KBRANCH_qemumips =3D "standard/mti-malta32" + KBRANCH_qemuppc =3D "standard/qemuppc" + KBRANCH_qemux86 =3D "standard/common-pc/base" + KBRANCH_qemux86-64 =3D "standard/common-pc-64/base" + KBRANCH_qemumips64 =3D "standard/mti-malta64" + +- *Appending and Prepending:* BitBake also supports append and prepend + operations to variable values based on whether a specific item is + listed in ``OVERRIDES``. Here is an example: :: + + DEPENDS =3D "glibc ncurses" + OVERRIDES =3D "machine:local" + DEPENDS_append_machine =3D "libmad" + + In this example, ``DEPENDS`` becomes "glibc ncurses libmad". + + Again, using an OpenEmbedded metadata-based kernel recipe file as an + example, the following lines will conditionally append to the + ``KERNEL_FEATURES`` variable based on the architecture: :: + + KERNEL_FEATURES_append =3D " ${KERNEL_EXTRA_FEATURES}" + KERNEL_FEATURES_append_qemux86=3D" cfg/sound.scc cfg/paravirt_kvm.sc= c" + KERNEL_FEATURES_append_qemux86-64=3D" cfg/sound.scc cfg/paravirt_kvm= .scc" + +- *Setting a Variable for a Single Task:* BitBake supports setting a + variable just for the duration of a single task. Here is an example: :: + + FOO_task-configure =3D "val 1" + FOO_task-compile =3D "val 2" + + In the + previous example, ``FOO`` has the value "val 1" while the + ``do_configure`` task is executed, and the value "val 2" while the + ``do_compile`` task is executed. + + Internally, this is implemented by prepending the task (e.g. + "task-compile:") to the value of + :term:`OVERRIDES` for the local datastore of the + ``do_compile`` task. + + You can also use this syntax with other combinations (e.g. + "``_prepend``") as shown in the following example: :: + + EXTRA_OEMAKE_prepend_task-compile =3D "${PARALLEL_MAKE} " + +Key Expansion +------------- + +Key expansion happens when the BitBake datastore is finalized. To better +understand this, consider the following example: :: + + A${B} =3D "X" + B =3D "2" + A2 =3D "Y" + +In this case, after all the parsing is complete, BitBake expands +``${B}`` into "2". This expansion causes ``A2``, which was set to "Y" +before the expansion, to become "X". + +.. _variable-interaction-worked-examples: + +Examples +-------- + +Despite the previous explanations that show the different forms of +variable definitions, it can be hard to work out exactly what happens +when variable operators, conditional overrides, and unconditional +overrides are combined. This section presents some common scenarios +along with explanations for variable interactions that typically confuse +users. + +There is often confusion concerning the order in which overrides and +various "append" operators take effect. Recall that an append or prepend +operation using "_append" and "_prepend" does not result in an immediate +assignment as would "+=3D", ".=3D", "=3D+", or "=3D.". Consider the follow= ing +example: :: + + OVERRIDES =3D "foo" + A =3D "Z" + A_foo_append =3D "X" + +For this case, +``A`` is unconditionally set to "Z" and "X" is unconditionally and +immediately appended to the variable ``A_foo``. Because overrides have +not been applied yet, ``A_foo`` is set to "X" due to the append and +``A`` simply equals "Z". + +Applying overrides, however, changes things. Since "foo" is listed in +``OVERRIDES``, the conditional variable ``A`` is replaced with the "foo" +version, which is equal to "X". So effectively, ``A_foo`` replaces +``A``. + +This next example changes the order of the override and the append: :: + + OVERRIDES =3D "foo" + A =3D "Z" + A_append_foo =3D "X" + +For this case, before +overrides are handled, ``A`` is set to "Z" and ``A_append_foo`` is set +to "X". Once the override for "foo" is applied, however, ``A`` gets +appended with "X". Consequently, ``A`` becomes "ZX". Notice that spaces +are not appended. + +This next example has the order of the appends and overrides reversed +back as in the first example: :: + + OVERRIDES =3D "foo" + A =3D "Y" + A_foo_append =3D "Z" + A_foo_append =3D "X" + +For this case, before any overrides are resolved, +``A`` is set to "Y" using an immediate assignment. After this immediate +assignment, ``A_foo`` is set to "Z", and then further appended with "X" +leaving the variable set to "ZX". Finally, applying the override for +"foo" results in the conditional variable ``A`` becoming "ZX" (i.e. +``A`` is replaced with ``A_foo``). + +This final example mixes in some varying operators: :: + + A =3D "1" + A_append =3D "2" + A_append =3D "3" + A +=3D "4" + A .=3D "5" + +For this case, the type of append +operators are affecting the order of assignments as BitBake passes +through the code multiple times. Initially, ``A`` is set to "1 45" +because of the three statements that use immediate operators. After +these assignments are made, BitBake applies the "_append" operations. +Those operations result in ``A`` becoming "1 4523". + +Sharing Functionality +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +BitBake allows for metadata sharing through include files (``.inc``) and +class files (``.bbclass``). For example, suppose you have a piece of +common functionality such as a task definition that you want to share +between more than one recipe. In this case, creating a ``.bbclass`` file +that contains the common functionality and then using the ``inherit`` +directive in your recipes to inherit the class would be a common way to +share the task. + +This section presents the mechanisms BitBake provides to allow you to +share functionality between recipes. Specifically, the mechanisms +include ``include``, ``inherit``, ``INHERIT``, and ``require`` +directives. + +Locating Include and Class Files +-------------------------------- + +BitBake uses the :term:`BBPATH` variable to locate +needed include and class files. Additionally, BitBake searches the +current directory for ``include`` and ``require`` directives. + +.. note:: + + The BBPATH variable is analogous to the environment variable PATH . + +In order for include and class files to be found by BitBake, they need +to be located in a "classes" subdirectory that can be found in +``BBPATH``. + +``inherit`` Directive +--------------------- + +When writing a recipe or class file, you can use the ``inherit`` +directive to inherit the functionality of a class (``.bbclass``). +BitBake only supports this directive when used within recipe and class +files (i.e. ``.bb`` and ``.bbclass``). + +The ``inherit`` directive is a rudimentary means of specifying +functionality contained in class files that your recipes require. For +example, you can easily abstract out the tasks involved in building a +package that uses Autoconf and Automake and put those tasks into a class +file and then have your recipe inherit that class file. + +As an example, your recipes could use the following directive to inherit +an ``autotools.bbclass`` file. The class file would contain common +functionality for using Autotools that could be shared across recipes: :: + + inherit autotools + +In this case, BitBake would search for the directory +``classes/autotools.bbclass`` in ``BBPATH``. + +.. note:: + + You can override any values and functions of the inherited class + within your recipe by doing so after the "inherit" statement. + +If you want to use the directive to inherit multiple classes, separate +them with spaces. The following example shows how to inherit both the +``buildhistory`` and ``rm_work`` classes: :: + + inherit buildhistory rm_work + +An advantage with the inherit directive as compared to both the +:ref:`include <bitbake-user-manual/bitbake-user-manual-metadata:\`\`includ= e\`\` directive>` and :ref:`require <bitbake-user-manual/bitbake-user-manua= l-metadata:\`\`require\`\` directive>` +directives is that you can inherit class files conditionally. You can +accomplish this by using a variable expression after the ``inherit`` +statement. Here is an example: :: + + inherit ${VARNAME} + +If ``VARNAME`` is +going to be set, it needs to be set before the ``inherit`` statement is +parsed. One way to achieve a conditional inherit in this case is to use +overrides: :: + + VARIABLE =3D "" + VARIABLE_someoverride =3D "myclass" + +Another method is by using anonymous Python. Here is an example: :: + + python () { + if condition =3D=3D value: + d.setVar('VARIABLE', 'myclass') + else: + d.setVar('VARIABLE', '') + } + +Alternatively, you could use an in-line Python expression in the +following form: :: + + inherit ${@'classname' if condition else ''} + inherit ${@functionname(params)} + +In all cases, if the expression evaluates to an +empty string, the statement does not trigger a syntax error because it +becomes a no-op. + +``include`` Directive +--------------------- + +BitBake understands the ``include`` directive. This directive causes +BitBake to parse whatever file you specify, and to insert that file at +that location. The directive is much like its equivalent in Make except +that if the path specified on the include line is a relative path, +BitBake locates the first file it can find within ``BBPATH``. + +The include directive is a more generic method of including +functionality as compared to the :ref:`inherit <bitbake-user-manual/bitbak= e-user-manual-metadata:\`\`inherit\`\` directive>` +directive, which is restricted to class (i.e. ``.bbclass``) files. The +include directive is applicable for any other kind of shared or +encapsulated functionality or configuration that does not suit a +``.bbclass`` file. + +As an example, suppose you needed a recipe to include some self-test +definitions: :: + + include test_defs.inc + +.. note:: + + The include directive does not produce an error when the file cannot be + found. Consequently, it is recommended that if the file you are includ= ing is + expected to exist, you should use :ref:`require <require-inclusion>` in= stead + of include . Doing so makes sure that an error is produced if the file = cannot + be found. + +.. _require-inclusion: + +``require`` Directive +--------------------- + +BitBake understands the ``require`` directive. This directive behaves +just like the ``include`` directive with the exception that BitBake +raises a parsing error if the file to be included cannot be found. Thus, +any file you require is inserted into the file that is being parsed at +the location of the directive. + +The require directive, like the include directive previously described, +is a more generic method of including functionality as compared to the +:ref:`inherit <bitbake-user-manual/bitbake-user-manual-metadata:\`\`inheri= t\`\` directive>` directive, which is restricted to class +(i.e. ``.bbclass``) files. The require directive is applicable for any +other kind of shared or encapsulated functionality or configuration that +does not suit a ``.bbclass`` file. + +Similar to how BitBake handles :ref:`include <bitbake-user-manual/bitbake-= user-manual-metadata:\`\`include\`\` directive>`, if +the path specified on the require line is a relative path, BitBake +locates the first file it can find within ``BBPATH``. + +As an example, suppose you have two versions of a recipe (e.g. +``foo_1.2.2.bb`` and ``foo_2.0.0.bb``) where each version contains some +identical functionality that could be shared. You could create an +include file named ``foo.inc`` that contains the common definitions +needed to build "foo". You need to be sure ``foo.inc`` is located in the +same directory as your two recipe files as well. Once these conditions +are set up, you can share the functionality using a ``require`` +directive from within each recipe: :: + + require foo.inc + +``INHERIT`` Configuration Directive +----------------------------------- + +When creating a configuration file (``.conf``), you can use the +:term:`INHERIT` configuration directive to inherit a +class. BitBake only supports this directive when used within a +configuration file. + +As an example, suppose you needed to inherit a class file called +``abc.bbclass`` from a configuration file as follows: :: + + INHERIT +=3D "abc" + +This configuration directive causes the named class to be inherited at +the point of the directive during parsing. As with the ``inherit`` +directive, the ``.bbclass`` file must be located in a "classes" +subdirectory in one of the directories specified in ``BBPATH``. + +.. note:: + + Because .conf files are parsed first during BitBake's execution, using + INHERIT to inherit a class effectively inherits the class globally (i.e= . for + all recipes). + +If you want to use the directive to inherit multiple classes, you can +provide them on the same line in the ``local.conf`` file. Use spaces to +separate the classes. The following example shows how to inherit both +the ``autotools`` and ``pkgconfig`` classes: :: + + INHERIT +=3D "autotools pkgconfig" + +Functions +=3D=3D=3D=3D=3D=3D=3D=3D=3D + +As with most languages, functions are the building blocks that are used +to build up operations into tasks. BitBake supports these types of +functions: + +- *Shell Functions:* Functions written in shell script and executed + either directly as functions, tasks, or both. They can also be called + by other shell functions. + +- *BitBake-Style Python Functions:* Functions written in Python and + executed by BitBake or other Python functions using + ``bb.build.exec_func()``. + +- *Python Functions:* Functions written in Python and executed by + Python. + +- *Anonymous Python Functions:* Python functions executed automatically + during parsing. + +Regardless of the type of function, you can only define them in class +(``.bbclass``) and recipe (``.bb`` or ``.inc``) files. + +Shell Functions +--------------- + +Functions written in shell script and executed either directly as +functions, tasks, or both. They can also be called by other shell +functions. Here is an example shell function definition: :: + + some_function () { + echo "Hello World" + } + +When you create these types of functions in +your recipe or class files, you need to follow the shell programming +rules. The scripts are executed by ``/bin/sh``, which may not be a bash +shell but might be something such as ``dash``. You should not use +Bash-specific script (bashisms). + +Overrides and override-style operators like ``_append`` and ``_prepend`` +can also be applied to shell functions. Most commonly, this application +would be used in a ``.bbappend`` file to modify functions in the main +recipe. It can also be used to modify functions inherited from classes. + +As an example, consider the following: :: + + do_foo() { + bbplain first + fn + } + + fn_prepend() { + bbplain second + } + + fn() { + bbplain third + } + + do_foo_append() { + bbplain fourth + } + +Running ``do_foo`` prints the following: :: + + recipename do_foo: first + recipename do_foo: second + recipename do_foo: third + recipename do_foo: fourth + +.. note:: + + Overrides and override-style operators can be applied to any shell + function, not just :ref:`tasks <bitbake-user-manual/bitbake-user-manual= -metadata:tasks>`. + +You can use the ``bitbake -e``=C2=A0recipename command to view the final +assembled function after all overrides have been applied. + +BitBake-Style Python Functions +------------------------------ + +These functions are written in Python and executed by BitBake or other +Python functions using ``bb.build.exec_func()``. + +An example BitBake function is: :: + + python some_python_function () { + d.setVar("TEXT", "Hello World") + print d.getVar("TEXT") + } + +Because the +Python "bb" and "os" modules are already imported, you do not need to +import these modules. Also in these types of functions, the datastore +("d") is a global variable and is always automatically available. + +.. note:: + + Variable expressions (e.g. ``${X}`` ) are no longer expanded within Py= thon + functions. This behavior is intentional in order to allow you to freely= set + variable values to expandable expressions without having them expanded + prematurely. If you do wish to expand a variable within a Python functi= on, + use ``d.getVar("X")`` . Or, for more complicated expressions, use ``d.e= xpand()``. + +Similar to shell functions, you can also apply overrides and +override-style operators to BitBake-style Python functions. + +As an example, consider the following: :: + + python do_foo_prepend() { + bb.plain("first") + } + + python do_foo() { + bb.plain("second") + } + + python do_foo_append() { + bb.plain("third") + } + +Running ``do_foo`` prints the following: :: + + recipename do_foo: first + recipename do_foo: second + recipename do_foo: third + +You can use the ``bitbake -e``=C2=A0recipename command to view +the final assembled function after all overrides have been applied. + +Python Functions +---------------- + +These functions are written in Python and are executed by other Python +code. Examples of Python functions are utility functions that you intend +to call from in-line Python or from within other Python functions. Here +is an example: :: + + def get_depends(d): + if d.getVar('SOMECONDITION'): + return "dependencywithcond" + else: + return "dependency" + + SOMECONDITION =3D "1" + DEPENDS =3D "${@get_depends(d)}" + +This would result in ``DEPENDS`` containing ``dependencywithcond``. + +Here are some things to know about Python functions: + +- Python functions can take parameters. + +- The BitBake datastore is not automatically available. Consequently, + you must pass it in as a parameter to the function. + +- The "bb" and "os" Python modules are automatically available. You do + not need to import them. + +BitBake-Style Python Functions Versus Python Functions +------------------------------------------------------ + +Following are some important differences between BitBake-style Python +functions and regular Python functions defined with "def": + +- Only BitBake-style Python functions can be :ref:`tasks <bitbake-user-ma= nual/bitbake-user-manual-metadata:tasks>`. + +- Overrides and override-style operators can only be applied to + BitBake-style Python functions. + +- Only regular Python functions can take arguments and return values. + +- :ref:`Variable flags <bitbake-user-manual/bitbake-user-manual-metadata:= variable flags>` such as + ``[dirs]``, ``[cleandirs]``, and ``[lockfiles]`` can be used on BitBake= -style + Python functions, but not on regular Python functions. + +- BitBake-style Python functions generate a separate + ``${``\ :term:`T`\ ``}/run.``\ function-name\ ``.``\ pid + script that is executed to run the function, and also generate a log + file in ``${T}/log.``\ function-name\ ``.``\ pid if they are executed + as tasks. + + Regular Python functions execute "inline" and do not generate any + files in ``${T}``. + +- Regular Python functions are called with the usual Python syntax. + BitBake-style Python functions are usually tasks and are called + directly by BitBake, but can also be called manually from Python code + by using the ``bb.build.exec_func()`` function. Here is an example: :: + + bb.build.exec_func("my_bitbake_style_function", d) + + .. note:: + + ``bb.build.exec_func()`` can also be used to run shell functions fro= m Python + code. If you want to run a shell function before a Python function w= ithin + the same task, then you can use a parent helper Python function that + starts by running the shell function with ``bb.build.exec_func()`` a= nd then + runs the Python code. + + To detect errors from functions executed with + ``bb.build.exec_func()``, you can catch the ``bb.build.FuncFailed`` + exception. + + .. note:: + + Functions in metadata (recipes and classes) should not themselves ra= ise + ``bb.build.FuncFailed``. Rather, ``bb.build.FuncFailed`` should be v= iewed as a + general indicator that the called function failed by raising an + exception. For example, an exception raised by ``bb.fatal()`` will b= e caught + inside ``bb.build.exec_func()``, and a ``bb.build.FuncFailed`` will = be raised in + response. + +Due to their simplicity, you should prefer regular Python functions over +BitBake-style Python functions unless you need a feature specific to +BitBake-style Python functions. Regular Python functions in metadata are +a more recent invention than BitBake-style Python functions, and older +code tends to use ``bb.build.exec_func()`` more often. + +Anonymous Python Functions +-------------------------- + +Sometimes it is useful to set variables or perform other operations +programmatically during parsing. To do this, you can define special +Python functions, called anonymous Python functions, that run at the end +of parsing. For example, the following conditionally sets a variable +based on the value of another variable: :: + + python () { + if d.getVar('SOMEVAR') =3D=3D 'value': + d.setVar('ANOTHERVAR', 'value2') + } + +An equivalent way to mark a function as an anonymous function is to give it +the name "__anonymous", rather than no name. + +Anonymous Python functions always run at the end of parsing, regardless +of where they are defined. If a recipe contains many anonymous +functions, they run in the same order as they are defined within the +recipe. As an example, consider the following snippet: :: + + python () { + d.setVar('FOO', 'foo 2') + } + + FOO =3D "foo 1" + + python () { + d.appendVar('BAR',' bar 2') + } + + BAR =3D "bar 1" + +The previous example is conceptually +equivalent to the following snippet: :: + + FOO =3D "foo 1" + BAR =3D "bar 1" + FOO =3D "foo 2" + BAR +=3D "bar 2" + +``FOO`` ends up with the value "foo 2", and +``BAR`` with the value "bar 1 bar 2". Just as in the second snippet, the +values set for the variables within the anonymous functions become +available to tasks, which always run after parsing. + +Overrides and override-style operators such as "``_append``" are applied +before anonymous functions run. In the following example, ``FOO`` ends +up with the value "foo from anonymous": :: + + FOO =3D "foo" + FOO_append =3D " from outside" + + python () { + d.setVar("FOO", "foo from anonymous") + } + +For methods +you can use with anonymous Python functions, see the +":ref:`bitbake-user-manual/bitbake-user-manual-metadata:functions you can = call from within python`" +section. For a different method to run Python code during parsing, see +the ":ref:`bitbake-user-manual/bitbake-user-manual-metadata:inline python = variable expansion`" section. + +Flexible Inheritance for Class Functions +---------------------------------------- + +Through coding techniques and the use of ``EXPORT_FUNCTIONS``, BitBake +supports exporting a function from a class such that the class function +appears as the default implementation of the function, but can still be +called if a recipe inheriting the class needs to define its own version +of the function. + +To understand the benefits of this feature, consider the basic scenario +where a class defines a task function and your recipe inherits the +class. In this basic scenario, your recipe inherits the task function as +defined in the class. If desired, your recipe can add to the start and +end of the function by using the "_prepend" or "_append" operations +respectively, or it can redefine the function completely. However, if it +redefines the function, there is no means for it to call the class +version of the function. ``EXPORT_FUNCTIONS`` provides a mechanism that +enables the recipe's version of the function to call the original +version of the function. + +To make use of this technique, you need the following things in place: + +- The class needs to define the function as follows: :: + + classname_functionname + + For example, if you have a class file + ``bar.bbclass`` and a function named ``do_foo``, the class must + define the function as follows: :: + + bar_do_foo + +- The class needs to contain the ``EXPORT_FUNCTIONS`` statement as + follows: :: + + EXPORT_FUNCTIONS functionname + + For example, continuing with + the same example, the statement in the ``bar.bbclass`` would be as + follows: :: + + EXPORT_FUNCTIONS do_foo + +- You need to call the function appropriately from within your recipe. + Continuing with the same example, if your recipe needs to call the + class version of the function, it should call ``bar_do_foo``. + Assuming ``do_foo`` was a shell function and ``EXPORT_FUNCTIONS`` was + used as above, the recipe's function could conditionally call the + class version of the function as follows: :: + + do_foo() { + if [ somecondition ] ; then + bar_do_foo + else + # Do something else + fi + } + + To call your modified version of the function as defined in your recipe, + call it as ``do_foo``. + +With these conditions met, your single recipe can freely choose between +the original function as defined in the class file and the modified +function in your recipe. If you do not set up these conditions, you are +limited to using one function or the other. + +Tasks +=3D=3D=3D=3D=3D + +Tasks are BitBake execution units that make up the steps that BitBake +can run for a given recipe. Tasks are only supported in recipes and +classes (i.e. in ``.bb`` files and files included or inherited from +``.bb`` files). By convention, tasks have names that start with "do\_". + +Promoting a Function to a Task +------------------------------ + +Tasks are either :ref:`shell functions <bitbake-user-manual/bitbake-user-m= anual-metadata:shell functions>` or +:ref:`BitBake-style Python functions <bitbake-user-manual/bitbake-user-man= ual-metadata:bitbake-style python functions>` +that have been promoted to tasks by using the ``addtask`` command. The +``addtask`` command can also optionally describe dependencies between +the task and other tasks. Here is an example that shows how to define a +task and declare some dependencies: :: + + python do_printdate () { + import time + print time.strftime('%Y%m%d', time.gmtime()) + } + addtask printdate after do_fetch before do_build + +The first argument to ``addtask`` is the name +of the function to promote to a task. If the name does not start with +"do\_", "do\_" is implicitly added, which enforces the convention that all +task names start with "do\_". + +In the previous example, the ``do_printdate`` task becomes a dependency +of the ``do_build`` task, which is the default task (i.e. the task run +by the ``bitbake`` command unless another task is specified explicitly). +Additionally, the ``do_printdate`` task becomes dependent upon the +``do_fetch`` task. Running the ``do_build`` task results in the +``do_printdate`` task running first. + +.. note:: + + If you try out the previous example, you might see that the + ``do_printdate`` + task is only run the first time you build the recipe with the + ``bitbake`` + command. This is because BitBake considers the task "up-to-date" + after that initial run. If you want to force the task to always be + rerun for experimentation purposes, you can make BitBake always + consider the task "out-of-date" by using the + :ref:`[nostamp] <bitbake-user-manual/bitbake-user-manual-metadata:Varia= ble Flags>` + variable flag, as follows: :: + + do_printdate[nostamp] =3D "1" + + You can also explicitly run the task and provide the + -f option as follows: :: + + $ bitbake recipe -c printdate -f + + When manually selecting a task to run with the bitbake ``recipe + -c task`` command, you can omit the "do\_" prefix as part of the task + name. + +You might wonder about the practical effects of using ``addtask`` +without specifying any dependencies as is done in the following example: :: + + addtask printdate + +In this example, assuming dependencies have not been +added through some other means, the only way to run the task is by +explicitly selecting it with ``bitbake``=C2=A0recipe=C2=A0``-c printdate``= . You +can use the ``do_listtasks`` task to list all tasks defined in a recipe +as shown in the following example: :: + + $ bitbake recipe -c listtasks + +For more information on task dependencies, see the +":ref:`bitbake-user-manual/bitbake-user-manual-execution:dependencies`" se= ction. + +See the ":ref:`bitbake-user-manual/bitbake-user-manual-metadata:variable f= lags`" section for information +on variable flags you can use with tasks. + +.. note:: + + While it's infrequent, it's possible to define multiple tasks as + dependencies when calling ``addtask``. For example, here's a snippet + from the OpenEmbedded class file ``package_tar.bbclass``:: + + addtask package_write_tar before do_build after do_packagedata do_pac= kage + + Note how the ``package_write_tar`` task has to wait until both of + ``do_packagedata`` and ``do_package`` complete. + +Deleting a Task +--------------- + +As well as being able to add tasks, you can delete them. Simply use the +``deltask`` command to delete a task. For example, to delete the example +task used in the previous sections, you would use: :: + + deltask printdate + +If you delete a task using the ``deltask`` command and the task has +dependencies, the dependencies are not reconnected. For example, suppose +you have three tasks named ``do_a``, ``do_b``, and ``do_c``. +Furthermore, ``do_c`` is dependent on ``do_b``, which in turn is +dependent on ``do_a``. Given this scenario, if you use ``deltask`` to +delete ``do_b``, the implicit dependency relationship between ``do_c`` +and ``do_a`` through ``do_b`` no longer exists, and ``do_c`` +dependencies are not updated to include ``do_a``. Thus, ``do_c`` is free +to run before ``do_a``. + +If you want dependencies such as these to remain intact, use the +``[noexec]`` varflag to disable the task instead of using the +``deltask`` command to delete it: :: + + do_b[noexec] =3D "1" + +Passing Information Into the Build Task Environment +--------------------------------------------------- + +When running a task, BitBake tightly controls the shell execution +environment of the build tasks to make sure unwanted contamination from +the build machine cannot influence the build. + +.. note:: + + By default, BitBake cleans the environment to include only those + things exported or listed in its whitelist to ensure that the build + environment is reproducible and consistent. You can prevent this + "cleaning" by setting the :term:`BB_PRESERVE_ENV` variable. + +Consequently, if you do want something to get passed into the build task +environment, you must take these two steps: + +#. Tell BitBake to load what you want from the environment into the + datastore. You can do so through the + :term:`BB_ENV_WHITELIST` and + :term:`BB_ENV_EXTRAWHITE` variables. For + example, assume you want to prevent the build system from accessing + your ``$HOME/.ccache`` directory. The following command "whitelists" + the environment variable ``CCACHE_DIR`` causing BitBake to allow that + variable into the datastore: :: + + export BB_ENV_EXTRAWHITE=3D"$BB_ENV_EXTRAWHITE CCACHE_DIR" + +#. Tell BitBake to export what you have loaded into the datastore to the + task environment of every running task. Loading something from the + environment into the datastore (previous step) only makes it + available in the datastore. To export it to the task environment of + every running task, use a command similar to the following in your + local configuration file ``local.conf`` or your distribution + configuration file: :: + + export CCACHE_DIR + + .. note:: + + A side effect of the previous steps is that BitBake records the + variable as a dependency of the build process in things like the + setscene checksums. If doing so results in unnecessary rebuilds of + tasks, you can whitelist the variable so that the setscene code + ignores the dependency when it creates checksums. + +Sometimes, it is useful to be able to obtain information from the +original execution environment. BitBake saves a copy of the original +environment into a special variable named :term:`BB_ORIGENV`. + +The ``BB_ORIGENV`` variable returns a datastore object that can be +queried using the standard datastore operators such as +``getVar(, False)``. The datastore object is useful, for example, to +find the original ``DISPLAY`` variable. Here is an example: :: + + origenv =3D d.getVar("BB_ORIGENV", False) + bar =3D origenv.getVar("BAR", False) + +The previous example returns ``BAR`` from the original execution +environment. + +Variable Flags +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Variable flags (varflags) help control a task's functionality and +dependencies. BitBake reads and writes varflags to the datastore using +the following command forms: :: + + variable =3D d.getVarFlags("variable") + self.d.setVarFlags("FOO", {"func": True}) + +When working with varflags, the same syntax, with the exception of +overrides, applies. In other words, you can set, append, and prepend +varflags just like variables. See the +":ref:`bitbake-user-manual/bitbake-user-manual-metadata:variable flag synt= ax`" section for details. + +BitBake has a defined set of varflags available for recipes and classes. +Tasks support a number of these flags which control various +functionality of the task: + +- ``[cleandirs]``: Empty directories that should be created before + the task runs. Directories that already exist are removed and + recreated to empty them. + +- ``[depends]``: Controls inter-task dependencies. See the + :term:`DEPENDS` variable and the + ":ref:`bitbake-user-manual/bitbake-user-manual-metadata:inter-task + dependencies`" section for more information. + +- ``[deptask]``: Controls task build-time dependencies. See the + :term:`DEPENDS` variable and the ":ref:`bitbake-user-manual/bitbake-use= r-manual-metadata:build dependencies`" section for more information. + +- ``[dirs]``: Directories that should be created before the task + runs. Directories that already exist are left as is. The last + directory listed is used as the current working directory for the + task. + +- ``[lockfiles]``: Specifies one or more lockfiles to lock while the + task executes. Only one task may hold a lockfile, and any task that + attempts to lock an already locked file will block until the lock is + released. You can use this variable flag to accomplish mutual + exclusion. + +- ``[noexec]``: When set to "1", marks the task as being empty, with + no execution required. You can use the ``[noexec]`` flag to set up + tasks as dependency placeholders, or to disable tasks defined + elsewhere that are not needed in a particular recipe. + +- ``[nostamp]``: When set to "1", tells BitBake to not generate a + stamp file for a task, which implies the task should always be + executed. + + .. caution:: + + Any task that depends (possibly indirectly) on a ``[nostamp]`` task = will + always be executed as well. This can cause unnecessary rebuilding if= you + are not careful. + +- ``[number_threads]``: Limits tasks to a specific number of + simultaneous threads during execution. This varflag is useful when + your build host has a large number of cores but certain tasks need to + be rate-limited due to various kinds of resource constraints (e.g. to + avoid network throttling). ``number_threads`` works similarly to the + :term:`BB_NUMBER_THREADS` variable but is task-specific. + + Set the value globally. For example, the following makes sure the + ``do_fetch`` task uses no more than two simultaneous execution + threads: do_fetch[number_threads] =3D "2" + + .. warning:: + + - Setting the varflag in individual recipes rather than globally + can result in unpredictable behavior. + + - Setting the varflag to a value greater than the value used in + the ``BB_NUMBER_THREADS`` variable causes ``number_threads`` to + have no effect. + +- ``[postfuncs]``: List of functions to call after the completion of + the task. + +- ``[prefuncs]``: List of functions to call before the task executes. + +- ``[rdepends]``: Controls inter-task runtime dependencies. See the + :term:`RDEPENDS` variable, the + :term:`RRECOMMENDS` variable, and the + ":ref:`bitbake-user-manual/bitbake-user-manual-metadata:inter-task depe= ndencies`" section for + more information. + +- ``[rdeptask]``: Controls task runtime dependencies. See the + :term:`RDEPENDS` variable, the + :term:`RRECOMMENDS` variable, and the + ":ref:`bitbake-user-manual/bitbake-user-manual-metadata:runtime depende= ncies`" section for more + information. + +- ``[recideptask]``: When set in conjunction with ``recrdeptask``, + specifies a task that should be inspected for additional + dependencies. + +- ``[recrdeptask]``: Controls task recursive runtime dependencies. + See the :term:`RDEPENDS` variable, the + :term:`RRECOMMENDS` variable, and the + ":ref:`bitbake-user-manual/bitbake-user-manual-metadata:recursive depen= dencies`" section for + more information. + +- ``[stamp-extra-info]``: Extra stamp information to append to the + task's stamp. As an example, OpenEmbedded uses this flag to allow + machine-specific tasks. + +- ``[umask]``: The umask to run the task under. + +Several varflags are useful for controlling how signatures are +calculated for variables. For more information on this process, see the +":ref:`bitbake-user-manual/bitbake-user-manual-execution:checksums (signat= ures)`" section. + +- ``[vardeps]``: Specifies a space-separated list of additional + variables to add to a variable's dependencies for the purposes of + calculating its signature. Adding variables to this list is useful, + for example, when a function refers to a variable in a manner that + does not allow BitBake to automatically determine that the variable + is referred to. + +- ``[vardepsexclude]``: Specifies a space-separated list of variables + that should be excluded from a variable's dependencies for the + purposes of calculating its signature. + +- ``[vardepvalue]``: If set, instructs BitBake to ignore the actual + value of the variable and instead use the specified value when + calculating the variable's signature. + +- ``[vardepvalueexclude]``: Specifies a pipe-separated list of + strings to exclude from the variable's value when calculating the + variable's signature. + +Events +=3D=3D=3D=3D=3D=3D + +BitBake allows installation of event handlers within recipe and class +files. Events are triggered at certain points during operation, such as +the beginning of operation against a given recipe (i.e. ``*.bb``), the +start of a given task, a task failure, a task success, and so forth. The +intent is to make it easy to do things like email notification on build +failures. + +Following is an example event handler that prints the name of the event +and the content of the ``FILE`` variable: :: + + addhandler myclass_eventhandler + python myclass_eventhandler() { + from bb.event import getName + print("The name of the Event is %s" % getName(e)) + print("The file we run for is %s" % d.getVar('FILE')) + } + myclass_eventhandler[eventmask] =3D "bb.event.BuildStarted + bb.event.BuildCompleted" + +In the previous example, an eventmask has been +set so that the handler only sees the "BuildStarted" and +"BuildCompleted" events. This event handler gets called every time an +event matching the eventmask is triggered. A global variable "e" is +defined, which represents the current event. With the ``getName(e)`` +method, you can get the name of the triggered event. The global +datastore is available as "d". In legacy code, you might see "e.data" +used to get the datastore. However, realize that "e.data" is deprecated +and you should use "d" going forward. + +The context of the datastore is appropriate to the event in question. +For example, "BuildStarted" and "BuildCompleted" events run before any +tasks are executed so would be in the global configuration datastore +namespace. No recipe-specific metadata exists in that namespace. The +"BuildStarted" and "BuildCompleted" events also run in the main +cooker/server process rather than any worker context. Thus, any changes +made to the datastore would be seen by other cooker/server events within +the current build but not seen outside of that build or in any worker +context. Task events run in the actual tasks in question consequently +have recipe-specific and task-specific contents. These events run in the +worker context and are discarded at the end of task execution. + +During a standard build, the following common events might occur. The +following events are the most common kinds of events that most metadata +might have an interest in viewing: + +- ``bb.event.ConfigParsed()``: Fired when the base configuration; which + consists of ``bitbake.conf``, ``base.bbclass`` and any global + ``INHERIT`` statements; has been parsed. You can see multiple such + events when each of the workers parse the base configuration or if + the server changes configuration and reparses. Any given datastore + only has one such event executed against it, however. If + :term:`BB_INVALIDCONF` is set in the datastore by the event + handler, the configuration is reparsed and a new event triggered, + allowing the metadata to update configuration. + +- ``bb.event.HeartbeatEvent()``: Fires at regular time intervals of one + second. You can configure the interval time using the + ``BB_HEARTBEAT_EVENT`` variable. The event's "time" attribute is the + ``time.time()`` value when the event is triggered. This event is + useful for activities such as system state monitoring. + +- ``bb.event.ParseStarted()``: Fired when BitBake is about to start + parsing recipes. This event's "total" attribute represents the number + of recipes BitBake plans to parse. + +- ``bb.event.ParseProgress()``: Fired as parsing progresses. This + event's "current" attribute is the number of recipes parsed as well + as the "total" attribute. + +- ``bb.event.ParseCompleted()``: Fired when parsing is complete. This + event's "cached", "parsed", "skipped", "virtuals", "masked", and + "errors" attributes provide statistics for the parsing results. + +- ``bb.event.BuildStarted()``: Fired when a new build starts. BitBake + fires multiple "BuildStarted" events (one per configuration) when + multiple configuration (multiconfig) is enabled. + +- ``bb.build.TaskStarted()``: Fired when a task starts. This event's + "taskfile" attribute points to the recipe from which the task + originates. The "taskname" attribute, which is the task's name, + includes the ``do_`` prefix, and the "logfile" attribute point to + where the task's output is stored. Finally, the "time" attribute is + the task's execution start time. + +- ``bb.build.TaskInvalid()``: Fired if BitBake tries to execute a task + that does not exist. + +- ``bb.build.TaskFailedSilent()``: Fired for setscene tasks that fail + and should not be presented to the user verbosely. + +- ``bb.build.TaskFailed()``: Fired for normal tasks that fail. + +- ``bb.build.TaskSucceeded()``: Fired when a task successfully + completes. + +- ``bb.event.BuildCompleted()``: Fired when a build finishes. + +- ``bb.cooker.CookerExit()``: Fired when the BitBake server/cooker + shuts down. This event is usually only seen by the UIs as a sign they + should also shutdown. + +This next list of example events occur based on specific requests to the +server. These events are often used to communicate larger pieces of +information from the BitBake server to other parts of BitBake such as +user interfaces: + +- ``bb.event.TreeDataPreparationStarted()`` +- ``bb.event.TreeDataPreparationProgress()`` +- ``bb.event.TreeDataPreparationCompleted()`` +- ``bb.event.DepTreeGenerated()`` +- ``bb.event.CoreBaseFilesFound()`` +- ``bb.event.ConfigFilePathFound()`` +- ``bb.event.FilesMatchingFound()`` +- ``bb.event.ConfigFilesFound()`` +- ``bb.event.TargetsTreeGenerated()`` + +.. _variants-class-extension-mechanism: + +Variants - Class Extension Mechanism +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +BitBake supports two features that facilitate creating from a single +recipe file multiple incarnations of that recipe file where all +incarnations are buildable. These features are enabled through the +:term:`BBCLASSEXTEND` and :term:`BBVERSIONS` variables. + +.. note:: + + The mechanism for this class extension is extremely specific to the + implementation. Usually, the recipe's :term:`PROVIDES` , :term:`PN` , a= nd + :term:`DEPENDS` variables would need to be modified by the extension + class. For specific examples, see the OE-Core native , nativesdk , and + multilib classes. + +- ``BBCLASSEXTEND``: This variable is a space separated list of + classes used to "extend" the recipe for each variant. Here is an + example that results in a second incarnation of the current recipe + being available. This second incarnation will have the "native" class + inherited. :: + + BBCLASSEXTEND =3D "native" + +- ``BBVERSIONS``: This variable allows a single recipe to build + multiple versions of a project from a single recipe file. You can + also specify conditional metadata (using the + :term:`OVERRIDES` mechanism) for a single + version, or an optionally named range of versions. Here is an + example: :: + + BBVERSIONS =3D "1.0 2.0 git" + SRC_URI_git =3D "git://someurl/somepath.git" + + BBVERSIONS =3D "1.0.[0-6]:1.0.0+ 1.0.[7-9]:1.0.7+" + SRC_URI_append_1.0.7+ =3D "file://some_patch_which_the_new_versions_= need.patch;patch=3D1" + + The name of the range defaults to the original version of the recipe. F= or + example, in OpenEmbedded, the recipe file ``foo_1.0.0+.bb`` creates a d= efault + name range of ``1.0.0+``. This is useful because the range name is not = only + placed into overrides, but it is also made available for the metadata t= o use + in the variable that defines the base recipe versions for use in ``file= ://`` + search paths (:term:`FILESPATH`). + +Dependencies +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +To allow for efficient parallel processing, BitBake handles dependencies +at the task level. Dependencies can exist both between tasks within a +single recipe and between tasks in different recipes. Following are +examples of each: + +- For tasks within a single recipe, a recipe's ``do_configure`` task + might need to complete before its ``do_compile`` task can run. + +- For tasks in different recipes, one recipe's ``do_configure`` task + might require another recipe's ``do_populate_sysroot`` task to finish + first such that the libraries and headers provided by the other + recipe are available. + +This section describes several ways to declare dependencies. Remember, +even though dependencies are declared in different ways, they are all +simply dependencies between tasks. + +.. _dependencies-internal-to-the-bb-file: + +Dependencies Internal to the ``.bb`` File +----------------------------------------- + +BitBake uses the ``addtask`` directive to manage dependencies that are +internal to a given recipe file. You can use the ``addtask`` directive +to indicate when a task is dependent on other tasks or when other tasks +depend on that recipe. Here is an example: :: + + addtask printdate after do_fetch before do_build + +In this example, the ``do_printdate`` task +depends on the completion of the ``do_fetch`` task, and the ``do_build`` +task depends on the completion of the ``do_printdate`` task. + +.. note:: + + For a task to run, it must be a direct or indirect dependency of some + other task that is scheduled to run. + + For illustration, here are some examples: + + - The directive ``addtask mytask before do_configure`` causes + ``do_mytask`` to run before ``do_configure`` runs. Be aware that + ``do_mytask`` still only runs if its :ref:`input + checksum <bitbake-user-manual/bitbake-user-manual-execution:checksum= s (signatures)>` has changed since the last time it was + run. Changes to the input checksum of ``do_mytask`` also + indirectly cause ``do_configure`` to run. + + - The directive ``addtask mytask after do_configure`` by itself + never causes ``do_mytask`` to run. ``do_mytask`` can still be run + manually as follows: :: + + $ bitbake recipe -c mytask + + Declaring ``do_mytask`` as a dependency of some other task that is + scheduled to run also causes it to run. Regardless, the task runs af= ter + ``do_configure``. + +Build Dependencies +------------------ + +BitBake uses the :term:`DEPENDS` variable to manage +build time dependencies. The ``[deptask]`` varflag for tasks signifies +the task of each item listed in ``DEPENDS`` that must complete before +that task can be executed. Here is an example: :: + + do_configure[deptask] =3D "do_populate_sysroot" + +In this example, the ``do_populate_sysroot`` task +of each item in ``DEPENDS`` must complete before ``do_configure`` can +execute. + +Runtime Dependencies +-------------------- + +BitBake uses the :term:`PACKAGES`, :term:`RDEPENDS`, and :term:`RRECOMMEND= S` +variables to manage runtime dependencies. + +The ``PACKAGES`` variable lists runtime packages. Each of those packages +can have ``RDEPENDS`` and ``RRECOMMENDS`` runtime dependencies. The +``[rdeptask]`` flag for tasks is used to signify the task of each item +runtime dependency which must have completed before that task can be +executed. :: + + do_package_qa[rdeptask] =3D "do_packagedata" + +In the previous +example, the ``do_packagedata`` task of each item in ``RDEPENDS`` must +have completed before ``do_package_qa`` can execute. +Although ``RDEPENDS`` contains entries from the +runtime dependency namespace, BitBake knows how to map them back +to the build-time dependency namespace, in which the tasks are defined. + +Recursive Dependencies +---------------------- + +BitBake uses the ``[recrdeptask]`` flag to manage recursive task +dependencies. BitBake looks through the build-time and runtime +dependencies of the current recipe, looks through the task's inter-task +dependencies, and then adds dependencies for the listed task. Once +BitBake has accomplished this, it recursively works through the +dependencies of those tasks. Iterative passes continue until all +dependencies are discovered and added. + +The ``[recrdeptask]`` flag is most commonly used in high-level recipes +that need to wait for some task to finish "globally". For example, +``image.bbclass`` has the following: :: + + do_rootfs[recrdeptask] +=3D "do_packagedata" + +This statement says that the ``do_packagedata`` task of +the current recipe and all recipes reachable (by way of dependencies) +from the image recipe must run before the ``do_rootfs`` task can run. + +BitBake allows a task to recursively depend on itself by +referencing itself in the task list: :: + + do_a[recrdeptask] =3D "do_a do_b" + +In the same way as before, this means that the ``do_a`` +and ``do_b`` tasks of the current recipe and all +recipes reachable (by way of dependencies) from the recipe +must run before the ``do_a`` task can run. In this +case BitBake will ignore the current recipe's ``do_a`` +task circular dependency on itself. + +Inter-Task Dependencies +----------------------- + +BitBake uses the ``[depends]`` flag in a more generic form to manage +inter-task dependencies. This more generic form allows for +inter-dependency checks for specific tasks rather than checks for the +data in ``DEPENDS``. Here is an example: :: + + do_patch[depends] =3D "quilt-native:do_populate_sysroot" + +In this example, the ``do_populate_sysroot`` task of the target ``quilt-na= tive`` +must have completed before the ``do_patch`` task can execute. + +The ``[rdepends]`` flag works in a similar way but takes targets in the +runtime namespace instead of the build-time dependency namespace. + +Functions You Can Call From Within Python +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +BitBake provides many functions you can call from within Python +functions. This section lists the most commonly used functions, and +mentions where to find others. + +Functions for Accessing Datastore Variables +------------------------------------------- + +It is often necessary to access variables in the BitBake datastore using +Python functions. The BitBake datastore has an API that allows you this +access. Here is a list of available operations: + +.. list-table:: + :widths: auto + :header-rows: 1 + + * - *Operation* + - *Description* + * - ``d.getVar("X", expand)`` + - Returns the value of variable "X". Using "expand=3DTrue" expands the + value. Returns "None" if the variable "X" does not exist. + * - ``d.setVar("X", "value")`` + - Sets the variable "X" to "value" + * - ``d.appendVar("X", "value")`` + - Adds "value" to the end of the variable "X". Acts like ``d.setVar("= X", + "value")`` if the variable "X" does not exist. + * - ``d.prependVar("X", "value")`` + - Adds "value" to the start of the variable "X". Acts like + ``d.setVar("X","value")`` if the variable "X" does not exist. + * - ``d.delVar("X")`` + - Deletes the variable "X" from the datastore. Does nothing if the va= riable + "X" does not exist. + * - ``d.renameVar("X", "Y")`` + - Renames the variable "X" to "Y". Does nothing if the variable "X" d= oes + not exist. + * - ``d.getVarFlag("X", flag, expand)`` + - Returns the value of variable "X". Using "expand=3DTrue" expands the + value. Returns "None" if either the variable "X" or the named flag = does + not exist. + * - ``d.setVarFlag("X", flag, "value")`` + - Sets the named flag for variable "X" to "value". + * - ``d.appendVarFlag("X", flag, "value")`` + - Appends "value" to the named flag on the variable "X". Acts like + ``d.setVarFlag("X", flag, "value")`` if the named flag does not exi= st. + * - ``d.prependVarFlag("X", flag, "value")`` + - Prepends "value" to the named flag on the variable "X". Acts like + ``d.setVarFlag("X", flag, "value")`` if the named flag does not exi= st. + * - ``d.delVarFlag("X", flag)`` + - Deletes the named flag on the variable "X" from the datastore. + * - ``d.setVarFlags("X", flagsdict)`` + - Sets the flags specified in the ``flagsdict()`` + parameter. ``setVarFlags`` does not clear previous flags. Think of = this + operation as ``addVarFlags``. + * - ``d.getVarFlags("X")`` + - Returns a ``flagsdict`` of the flags for the variable "X". Returns = "None" + if the variable "X" does not exist. + * - ``d.delVarFlags("X")`` + - Deletes all the flags for the variable "X". Does nothing if the var= iable + "X" does not exist. + * - ``d.expand(expression)`` + - Expands variable references in the specified string + expression. References to variables that do not exist are left as i= s. For + example, ``d.expand("foo ${X}")`` expands to the literal string "foo + ${X}" if the variable "X" does not exist. + +Other Functions +--------------- + +You can find many other functions that can be called from Python by +looking at the source code of the ``bb`` module, which is in +``bitbake/lib/bb``. For example, ``bitbake/lib/bb/utils.py`` includes +the commonly used functions ``bb.utils.contains()`` and +``bb.utils.mkdirhier()``, which come with docstrings. + +Task Checksums and Setscene +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D + +BitBake uses checksums (or signatures) along with the setscene to +determine if a task needs to be run. This section describes the process. +To help understand how BitBake does this, the section assumes an +OpenEmbedded metadata-based example. + +These checksums are stored in :term:`STAMP`. You can +examine the checksums using the following BitBake command: :: + + $ bitbake-dumpsigs + +This command returns the signature data in a readable +format that allows you to examine the inputs used when the OpenEmbedded +build system generates signatures. For example, using +``bitbake-dumpsigs`` allows you to examine the ``do_compile`` task's +"sigdata" for a C application (e.g. ``bash``). Running the command also +reveals that the "CC" variable is part of the inputs that are hashed. +Any changes to this variable would invalidate the stamp and cause the +``do_compile`` task to run. + +The following list describes related variables: + +- :term:`BB_HASHCHECK_FUNCTION`: + Specifies the name of the function to call during the "setscene" part + of the task's execution in order to validate the list of task hashes. + +- :term:`BB_SETSCENE_DEPVALID`: + Specifies a function BitBake calls that determines whether BitBake + requires a setscene dependency to be met. + +- :term:`BB_SETSCENE_VERIFY_FUNCTION2`: + Specifies a function to call that verifies the list of planned task + execution before the main task execution happens. + +- :term:`BB_STAMP_POLICY`: Defines the mode + for comparing timestamps of stamp files. + +- :term:`BB_STAMP_WHITELIST`: Lists stamp + files that are looked at when the stamp policy is "whitelist". + +- :term:`BB_TASKHASH`: Within an executing task, + this variable holds the hash of the task as returned by the currently + enabled signature generator. + +- :term:`STAMP`: The base path to create stamp files. + +- :term:`STAMPCLEAN`: Again, the base path to + create stamp files but can use wildcards for matching a range of + files for clean operations. + +Wildcard Support in Variables +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D + +Support for wildcard use in variables varies depending on the context in +which it is used. For example, some variables and file names allow +limited use of wildcards through the "``%``" and "``*``" characters. +Other variables or names support Python's +`glob <https://docs.python.org/3/library/glob.html>`_ syntax, +`fnmatch <https://docs.python.org/3/library/fnmatch.html#module-fnmatch>`_ +syntax, or +`Regular Expression (re) <https://docs.python.org/3/library/re.html>`_ +syntax. + +For variables that have wildcard suport, the documentation describes +which form of wildcard, its use, and its limitations. diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.x= ml b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml deleted file mode 100644 index 0ca53216..00000000 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml +++ /dev/null @@ -1,2862 +0,0 @@ -<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" -"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"> - -<chapter id=3D"bitbake-user-manual-metadata"> - <title>Syntax and Operators - - - BitBake files have their own syntax. - The syntax has similarities to several - other languages but also has some unique features. - This section describes the available syntax and operators - as well as provides examples. - - -
    - Basic Syntax - - - This section provides some basic syntax examples. - - -
    - Basic Variable Setting - - - The following example sets VARIABLE to - "value". - This assignment occurs immediately as the statement is par= sed. - It is a "hard" assignment. - - VARIABLE =3D "value" - - As expected, if you include leading or trailing spaces as = part of - an assignment, the spaces are retained: - - VARIABLE =3D " value" - VARIABLE =3D "value " - - Setting VARIABLE to "" sets it to an = empty string, - while setting the variable to " " sets it to a blank space - (i.e. these are not the same values). - - VARIABLE =3D "" - VARIABLE =3D " " - - - - - You can use single quotes instead of double quotes - when setting a variable's value. - Doing so allows you to use values that contain the double - quote character: - - VARIABLE =3D 'I have a " in my value' - - - Unlike in Bourne shells, single quotes work identically - to double quotes in all other ways. - They do not suppress variable expansions. - - -
    - -
    - Modifying Existing Variables - - - Sometimes you need to modify existing variables. - Following are some cases where you might find you want to - modify an existing variable: - - - Customize a recipe that uses the variable. - - - Change a variable's default value used in a - *.bbclass file. - - - Change the variable in a *.bbappend - file to override the variable in the original reci= pe. - - - Change the variable in a configuration file so tha= t the - value overrides an existing configuration. - - - - - - Changing a variable value can sometimes depend on how the - value was originally assigned and also on the desired - intent of the change. - In particular, when you append a value to a variable that - has a default value, the resulting value might not be what - you expect. - In this case, the value you provide might replace the value - rather than append to the default value. - - - - If after you have changed a variable's value and something - unexplained occurs, you can use BitBake to check the actual - value of the suspect variable. - You can make these checks for both configuration and recipe - level changes: - - - For configuration changes, use the following: - - $ bitbake -e - - This command displays variable values after the - configuration files (i.e. local.conf, - bblayers.conf, - bitbake.conf and so forth) ha= ve - been parsed. - - Variables that are exported to the environment= are - preceded by the string "export" in the command= 's - output. - - - - For recipe changes, use the following: - - $ bitbake recipe -e | grep VARIABLE=3D" - - This command checks to see if the variable actually - makes it into a specific recipe. - - - -
    - -
    - Line Joining - - - Outside of - functions, BitBake joins - any line ending in a backslash character ("\") - with the following line before parsing statements. - The most common use for the "\" character is to split vari= able - assignments over multiple lines, as in the following examp= le: - - FOO =3D "bar \ - baz \ - qaz" - - Both the "\" character and the newline character - that follow it are removed when joining lines. - Thus, no newline characters end up in the value of - FOO. - - - - Consider this additional example where the two - assignments both assign "barbaz" to - FOO: - - FOO =3D "barbaz" - - FOO =3D "bar\ - baz" - - - BitBake does not interpret escape sequences like - "\n" in variable values. - For these to have an effect, the value must be passed - to some utility that interprets escape sequences, - such as printf or - echo -n. - - -
    - -
    - Variable Expansion - - - Variables can reference the contents of other variables - using a syntax that is similar to variable expansion in - Bourne shells. - The following assignments - result in A containing "aval" and B evaluating to "preaval= post". - - A =3D "aval" - B =3D "pre${A}post" - - - Unlike in Bourne shells, the curly braces are mandator= y: - Only ${FOO} and not - $FOO is recognized as an expansio= n of - FOO. - - The "=3D" operator does not immediately expand variable - references in the right-hand side. - Instead, expansion is deferred until the variable assigned= to - is actually used. - The result depends on the current values of the referenced - variables. - The following example should clarify this behavior: - - A =3D "${B} baz" - B =3D "${C} bar" - C =3D "foo" - *At this point, ${A} equals "foo bar baz"* - C =3D "qux" - *At this point, ${A} equals "qux bar baz"* - B =3D "norf" - *At this point, ${A} equals "norf baz"* - - Contrast this behavior with the - immediate v= ariable expansion - operator (i.e. ":=3D"). - - - - If the variable expansion syntax is used on a variable that - does not exist, the string is kept as is. - For example, given the following assignment, - BAR expands to the literal string - "${FOO}" as long as FOO does not exis= t. - - BAR =3D "${FOO}" - - -
    - -
    - Setting a default value (?=3D) - - - You can use the "?=3D" operator to achieve a "softer" assi= gnment - for a variable. - This type of assignment allows you to define a variable if= it - is undefined when the statement is parsed, but to leave the - value alone if the variable has a value. - Here is an example: - - A ?=3D "aval" - - If A is set at the time this statemen= t is parsed, - the variable retains its value. - However, if A is not set, - the variable is set to "aval". - - This assignment is immediate. - Consequently, if multiple "?=3D" assignments - to a single variable exist, the first of those ends up= getting - used. - - -
    - -
    - Setting a weak default value (??=3D) - - - It is possible to use a "weaker" assignment than in the - previous section by using the "??=3D" operator. - This assignment behaves identical to "?=3D" except that the - assignment is made at the end of the parsing process rather - than immediately. - Consequently, when multiple "??=3D" assignments exist, the= last - one is used. - Also, any "=3D" or "?=3D" assignment will override the val= ue set with - "??=3D". - Here is an example: - - A ??=3D "somevalue" - A ??=3D "someothervalue" - - If A is set before the above statemen= ts are parsed, - the variable retains its value. - If A is not set, - the variable is set to "someothervalue". - - - - Again, this assignment is a "lazy" or "weak" assignment - because it does not occur until the end - of the parsing process. - -
    - -
    - Immediate variable expansion (:=3D) - - - The ":=3D" operator results in a variable's - contents being expanded immediately, - rather than when the variable is actually used: - - T =3D "123" - A :=3D "test ${T}" - T =3D "456" - B :=3D "${T} ${C}" - C =3D "cval" - C :=3D "${C}append" - - In this example, A contains - "test 123", even though the final value of T - is "456". - The variable B will end up containing= "456 cvalappend". - This is because references to undefined variables are pres= erved as is - during (immediate)expansion. This is in contrast to GNU Ma= ke, where undefined - variables expand to nothing. - The variable C - contains "cvalappend" since ${C} imme= diately - expands to "cval". - -
    - -
    - Appending (+=3D) and prepending (=3D+) With Spaces</tit= le> - - <para> - Appending and prepending values is common and can be accom= plished - using the "+=3D" and "=3D+" operators. - These operators insert a space between the current - value and prepended or appended value. - </para> - - <para> - These operators take immediate effect during parsing. - Here are some examples: - <literallayout class=3D'monospaced'> - B =3D "bval" - B +=3D "additionaldata" - C =3D "cval" - C =3D+ "test" - </literallayout> - The variable <filename>B</filename> contains - "bval additionaldata" and <filename>C</filename> - contains "test cval". - </para> - </section> - - <section id=3D'appending-and-prepending-without-spaces'> - <title>Appending (.=3D) and Prepending (=3D.) Without Spaces</= title> - - <para> - If you want to append or prepend values without an - inserted space, use the ".=3D" and "=3D." operators. - </para> - - <para> - These operators take immediate effect during parsing. - Here are some examples: - <literallayout class=3D'monospaced'> - B =3D "bval" - B .=3D "additionaldata" - C =3D "cval" - C =3D. "test" - </literallayout> - The variable <filename>B</filename> contains - "bvaladditionaldata" and - <filename>C</filename> contains "testcval". - </para> - </section> - - <section id=3D'appending-and-prepending-override-style-syntax'> - <title>Appending and Prepending (Override Style Syntax) - - - You can also append and prepend a variable's value - using an override style syntax. - When you use this syntax, no spaces are inserted. - - - - These operators differ from the ":=3D", ".=3D", "=3D.", "+= =3D", and "=3D+" - operators in that their effects are applied at variable - expansion time rather than being immediately applied. - Here are some examples: - - B =3D "bval" - B_append =3D " additional data" - C =3D "cval" - C_prepend =3D "additional data " - D =3D "dval" - D_append =3D "additional data" - - The variable B becomes - "bval additional data" and C becomes - "additional data cval". - The variable D becomes - "dvaladditional data". - - You must control all spacing when you use the - override syntax. - - - - - It is also possible to append and prepend to shell - functions and BitBake-style Python functions. - See the - "Shell Functions"= and - "BitBake-= Style Python Functions - sections for examples. - -
    - -
    - Removal (Override Style Syntax) - - - You can remove values from lists using the removal - override style syntax. - Specifying a value for removal causes all occurrences of t= hat - value to be removed from the variable. - - - - When you use this syntax, BitBake expects one or more stri= ngs. - Surrounding spaces and spacing are preserved. - Here is an example: - - FOO =3D "123 456 789 123456 123 456 123 456" - FOO_remove =3D "123" - FOO_remove =3D "456" - FOO2 =3D " abc def ghi abcdef abc def abc def def" - FOO2_remove =3D " \ - def \ - abc \ - ghi \ - " - - The variable FOO becomes - "  789 123456    " - and FOO2 becomes - "     abcdef   &nb= sp;  ". - - - - Like "_append" and "_prepend", "_remove" - is applied at variable expansion time. - -
    - -
    - Override Style Operation Advantages - - - An advantage of the override style operations - "_append", "_prepend", and "_remove" as compared to the - "+=3D" and "=3D+" operators is that the override style - operators provide guaranteed operations. - For example, consider a class foo.bbclass - that needs to add the value "val" to the variable - FOO, and a recipe that uses - foo.bbclass as follows: - - inherit foo - - FOO =3D "initial" - - If foo.bbclass uses the "+=3D" operat= or, - as follows, then the final value of FOO - will be "initial", which is not what is desired: - - FOO +=3D "val" - - If, on the other hand, foo.bbclass - uses the "_append" operator, then the final value of - FOO will be "initial val", as intende= d: - - FOO_append =3D " val" - - - It is never necessary to use "+=3D" together with "_ap= pend". - The following sequence of assignments appends "barbaz"= to - FOO: - - FOO_append =3D "bar" - FOO_append =3D "baz" - - The only effect of changing the second assignment in t= he - previous example to use "+=3D" would be to add a space= before - "baz" in the appended value (due to how the "+=3D" ope= rator - works). - - Another advantage of the override style operations is that - you can combine them with other overrides as described in = the - "Conditiona= l Syntax (Overrides)" - section. - -
    - -
    - Variable Flag Syntax - - - Variable flags are BitBake's implementation of variable pr= operties - or attributes. - It is a way of tagging extra information onto a variable. - You can find more out about variable flags in general in t= he - "Variable Flags" - section. - - - - You can define, append, and prepend values to variable fla= gs. - All the standard syntax operations previously mentioned wo= rk - for variable flags except for override style syntax - (i.e. "_prepend", "_append", and "_remove"). - - - - Here are some examples showing how to set variable flags: - - FOO[a] =3D "abc" - FOO[b] =3D "123" - FOO[a] +=3D "456" - - The variable FOO has two flags: - [a] and [b]. - The flags are immediately set to "abc" and "123", respecti= vely. - The [a] flag becomes "abc 456". - - - - No need exists to pre-define variable flags. - You can simply start using them. - One extremely common application - is to attach some brief documentation to a BitBake variabl= e as - follows: - - CACHE[doc] =3D "The directory holding the cache of the metadata." - - -
    - -
    - Inline Python Variable Expansion - - - You can use inline Python variable expansion to - set variables. - Here is an example: - - DATE =3D "${@time.strftime('%Y%m%d',time.gmtime())}" - - This example results in the DATE - variable being set to the current date. - - - - Probably the most common use of this feature is to extract - the value of variables from BitBake's internal data dictio= nary, - d. - The following lines select the values of a package name - and its version number, respectively: - - PN =3D "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),= d)[0] or 'defaultpkgname'}" - PV =3D "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),= d)[1] or '1.0'}" - - - Inline Python expressions work just like variable expa= nsions - insofar as the "=3D" and ":=3D" operators are concerne= d. - Given the following assignment, foo() - is called each time FOO is expand= ed: - - FOO =3D "${@foo()}" - - Contrast this with the following immediate assignment,= where - foo() is only called once, while = the - assignment is parsed: - - FOO :=3D "${@foo()}" - - - For a different way to set variables with Python code duri= ng - parsing, see the - "Anonymous Py= thon Functions" - section. - -
    - -
    - Unsetting variables - - - It is possible to completely remove a variable or a variab= le flag - from BitBake's internal data dictionary by using the "unse= t" keyword. - Here is an example: - - unset DATE - unset do_fetch[noexec] - - These two statements remove the DATE = and the - do_fetch[noexec] flag. - - -
    - -
    - Providing Pathnames - - - When specifying pathnames for use with BitBake, - do not use the tilde ("~") character as a shortcut - for your home directory. - Doing so might cause BitBake to not recognize the - path since BitBake does not expand this character in - the same way a shell would. - - - - Instead, provide a fuller path as the following - example illustrates: - - BBLAYERS ?=3D " \ - /home/scott-lenovo/LayerA \ - " - - -
    -
    - -
    - Exporting Variables to the Environment - - - You can export variables to the environment of running - tasks by using the export keyword. - For example, in the following example, the - do_foo task prints "value from - the environment" when run: - - export ENV_VARIABLE - ENV_VARIABLE =3D "value from the environment" - - do_foo() { - bbplain "$ENV_VARIABLE" - } - - - BitBake does not expand $ENV_VARIABLE - in this case because it lacks the obligatory - {}. - Rather, $ENV_VARIABLE is expanded - by the shell. - - It does not matter whether - export ENV_VARIABLE appears before or - after assignments to ENV_VARIABLE. - - - - It is also possible to combine export - with setting a value for the variable. - Here is an example: - - export ENV_VARIABLE =3D "variable-value" - - In the output of bitbake -e, variables - that are exported to the environment are preceded by "export". - - - - Among the variables commonly exported to the environment - are CC and CFLAGS, - which are picked up by many build systems. - -
    - -
    - Conditional Syntax (Overrides) - - - BitBake uses - OVERRIDES - to control what variables are overridden after BitBake - parses recipes and configuration files. - This section describes how you can use - OVERRIDES as conditional metadata, - talks about key expansion in relationship to - OVERRIDES, and provides some examples - to help with understanding. - - -
    - Conditional Metadata - - - You can use OVERRIDES to conditionall= y select - a specific version of a variable and to conditionally - append or prepend the value of a variable. - - Overrides can only use lower-case characters. - Additionally, underscores are not permitted in overrid= e names - as they are used to separate overrides from each other= and - from the variable name. - - - Selecting a Variable: - The OVERRIDES variable is - a colon-character-separated list that contains ite= ms - for which you want to satisfy conditions. - Thus, if you have a variable that is conditional o= n =E2=80=9Carm=E2=80=9D, and =E2=80=9Carm=E2=80=9D - is in OVERRIDES, then the =E2= =80=9Carm=E2=80=9D-specific - version of the variable is used rather than the no= n-conditional - version. - Here is an example: - - OVERRIDES =3D "architecture:os:machine" - TEST =3D "default" - TEST_os =3D "osspecific" - TEST_nooverride =3D "othercondvalue" - - In this example, the OVERRIDES - variable lists three overrides: - "architecture", "os", and "machine". - The variable TEST by itself h= as a default - value of "default". - You select the os-specific version of the TEST - variable by appending the "os" override to the var= iable - (i.e.TEST_os). - - - - To better understand this, consider a practica= l example - that assumes an OpenEmbedded metadata-based Li= nux - kernel recipe file. - The following lines from the recipe file first= set - the kernel branch variable KBRANCH - to a default value, then conditionally overrid= e that - value based on the architecture of the build: - - KBRANCH =3D "standard/base" - KBRANCH_qemuarm =3D "standard/arm-versatile-926ejs" - KBRANCH_qemumips =3D "standard/mti-malta32" - KBRANCH_qemuppc =3D "standard/qemuppc" - KBRANCH_qemux86 =3D "standard/common-pc/base" - KBRANCH_qemux86-64 =3D "standard/common-pc-64/base" - KBRANCH_qemumips64 =3D "standard/mti-malta64" - - - Appending and Prepending: - BitBake also supports append and prepend operation= s to - variable values based on whether a specific item is - listed in OVERRIDES. - Here is an example: - - DEPENDS =3D "glibc ncurses" - OVERRIDES =3D "machine:local" - DEPENDS_append_machine =3D " libmad" - - In this example, DEPENDS beco= mes - "glibc ncurses libmad". - - - - Again, using an OpenEmbedded metadata-based - kernel recipe file as an example, the - following lines will conditionally append to t= he - KERNEL_FEATURES variable = based - on the architecture: - - KERNEL_FEATURES_append =3D " ${KERNEL_EXTRA_FEATURES}" - KERNEL_FEATURES_append_qemux86=3D" cfg/sound.scc cfg/paravirt_kvm.scc" - KERNEL_FEATURES_append_qemux86-64=3D" cfg/sound.scc cfg/paravirt_kvm.= scc" - - - Setting a Variable for a Sin= gle Task: - BitBake supports setting a variable just for the - duration of a single task. - Here is an example: - - FOO_task-configure =3D "val 1" - FOO_task-compile =3D "val 2" - - In the previous example, FOO - has the value "val 1" while the - do_configure task is executed, - and the value "val 2" while the - do_compile task is executed. - - - Internally, this is implemented by prepending - the task (e.g. "task-compile:") to the value of - OVERR= IDES - for the local datastore of the do_compil= e - task. - - You can also use this syntax with other comb= inations - (e.g. "_prepend") as shown in= the - following example: - - EXTRA_OEMAKE_prepend_task-compile =3D "${PARALLEL_MAKE} " - - - - -
    - -
    - Key Expansion - - - Key expansion happens when the BitBake datastore is finali= zed. - To better understand this, consider the following example: - - A${B} =3D "X" - B =3D "2" - A2 =3D "Y" - - In this case, after all the parsing is complete, - BitBake expands ${B} into "2". - This expansion causes A2, which was - set to "Y" before the expansion, to become "X". - -
    - -
    - Examples - - - Despite the previous explanations that show the different = forms of - variable definitions, it can be hard to work - out exactly what happens when variable operators, conditio= nal - overrides, and unconditional overrides are combined. - This section presents some common scenarios along - with explanations for variable interactions that - typically confuse users. - - - - There is often confusion concerning the order in which - overrides and various "append" operators take effect. - Recall that an append or prepend operation using "_append" - and "_prepend" does not result in an immediate assignment - as would "+=3D", ".=3D", "=3D+", or "=3D.". - Consider the following example: - - OVERRIDES =3D "foo" - A =3D "Z" - A_foo_append =3D "X" - - For this case, A is - unconditionally set to "Z" and "X" is - unconditionally and immediately appended to the variable - A_foo. - Because overrides have not been applied yet, - A_foo is set to "X" due to the append - and A simply equals "Z". - - - - Applying overrides, however, changes things. - Since "foo" is listed in OVERRIDES, - the conditional variable A is replaced - with the "foo" version, which is equal to "X". - So effectively, A_foo replaces A. - - - - This next example changes the order of the override and - the append: - - OVERRIDES =3D "foo" - A =3D "Z" - A_append_foo =3D "X" - - For this case, before overrides are handled, - A is set to "Z" and A_appen= d_foo - is set to "X". - Once the override for "foo" is applied, however, - A gets appended with "X". - Consequently, A becomes "ZX". - Notice that spaces are not appended. - - - - This next example has the order of the appends and overrid= es reversed - back as in the first example: - - OVERRIDES =3D "foo" - A =3D "Y" - A_foo_append =3D "Z" - A_foo_append =3D "X" - - For this case, before any overrides are resolved, - A is set to "Y" using an immediate as= signment. - After this immediate assignment, A_foo is set - to "Z", and then further appended with - "X" leaving the variable set to "ZX". - Finally, applying the override for "foo" results in the co= nditional - variable A becoming "ZX" (i.e. - A is replaced with A_foo). - - - - This final example mixes in some varying operators: - - A =3D "1" - A_append =3D "2" - A_append =3D "3" - A +=3D "4" - A .=3D "5" - - For this case, the type of append operators are affecting = the - order of assignments as BitBake passes through the code - multiple times. - Initially, A is set to "1 45" because - of the three statements that use immediate operators. - After these assignments are made, BitBake applies the - "_append" operations. - Those operations result in A becoming= "1 4523". - -
    -
    - -
    - Sharing Functionality - - - BitBake allows for metadata sharing through include files - (.inc) and class files - (.bbclass). - For example, suppose you have a piece of common functionality - such as a task definition that you want to share between - more than one recipe. - In this case, creating a .bbclass - file that contains the common functionality and then using - the inherit directive in your recipes to - inherit the class would be a common way to share the task. - - - - This section presents the mechanisms BitBake provides to - allow you to share functionality between recipes. - Specifically, the mechanisms include include, - inherit, INHERIT, and - require directives. - - -
    - Locating Include and Class Files - - - BitBake uses the - BBPATH - variable to locate needed include and class files. - Additionally, BitBake searches the current directory for - include and require - directives. - - The BBPATH variable is analogous = to - the environment variable PATH. - - - - - In order for include and class files to be found by BitBak= e, - they need to be located in a "classes" subdirectory that c= an - be found in BBPATH. - -
    - -
    - <filename>inherit</filename> Directive - - - When writing a recipe or class file, you can use the - inherit directive to inherit the - functionality of a class (.bbclass). - BitBake only supports this directive when used within reci= pe - and class files (i.e. .bb and - .bbclass). - - - - The inherit directive is a rudimentary - means of specifying functionality contained in class files - that your recipes require. - For example, you can easily abstract out the tasks involve= d in - building a package that uses Autoconf and Automake and put - those tasks into a class file and then have your recipe - inherit that class file. - - - - As an example, your recipes could use the following direct= ive - to inherit an autotools.bbclass file. - The class file would contain common functionality for using - Autotools that could be shared across recipes: - - inherit autotools - - In this case, BitBake would search for the directory - classes/autotools.bbclass - in BBPATH. - - You can override any values and functions of the - inherited class within your recipe by doing so - after the "inherit" statement. - - If you want to use the directive to inherit - multiple classes, separate them with spaces. - The following example shows how to inherit both the - buildhistory and rm_work - classes: - - inherit buildhistory rm_work - - - - - An advantage with the inherit directive as compared to both - the - include and - require directi= ves - is that you can inherit class files conditionally. - You can accomplish this by using a variable expression - after the inherit statement. - Here is an example: - - inherit ${VARNAME} - - If VARNAME is going to be set, it nee= ds - to be set before the inherit statement - is parsed. - One way to achieve a conditional inherit in this case is t= o use - overrides: - - VARIABLE =3D "" - VARIABLE_someoverride =3D "myclass" - - - - - Another method is by using anonymous Python. - Here is an example: - - python () { - if condition =3D=3D value: - d.setVar('VARIABLE', 'myclass') - else: - d.setVar('VARIABLE', '') - } - - - - - Alternatively, you could use an in-line Python expression - in the following form: - - inherit ${@'classname' if condition else ''} - inherit ${@functionname(params)} - - In all cases, if the expression evaluates to an empty - string, the statement does not trigger a syntax error - because it becomes a no-op. - -
    - -
    - <filename>include</filename> Directive - - - BitBake understands the include - directive. - This directive causes BitBake to parse whatever file you s= pecify, - and to insert that file at that location. - The directive is much like its equivalent in Make except - that if the path specified on the include line is a relati= ve - path, BitBake locates the first file it can find - within BBPATH. - - - - The include directive is a more generic method of including - functionality as compared to the - inherit directi= ve, - which is restricted to class (i.e. .bbclass) - files. - The include directive is applicable for any other kind of - shared or encapsulated functionality or configuration that - does not suit a .bbclass file. - - - - As an example, suppose you needed a recipe to include some - self-test definitions: - - include test_defs.inc - - - The include directive does not - produce an error when the file cannot be found. - Consequently, it is recommended that if the file you - are including is expected to exist, you should use - require<= /filename> - instead of include. - Doing so makes sure that an error is produced if the - file cannot be found. - - -
    - -
    - <filename>require</filename> Directive - - - BitBake understands the require - directive. - This directive behaves just like the - include directive with the exception = that - BitBake raises a parsing error if the file to be included = cannot - be found. - Thus, any file you require is inserted into the file that = is - being parsed at the location of the directive. - - - - The require directive, like the include directive previous= ly - described, is a more generic method of including - functionality as compared to the - inherit directi= ve, - which is restricted to class (i.e. .bbclass) - files. - The require directive is applicable for any other kind of - shared or encapsulated functionality or configuration that - does not suit a .bbclass file. - - - - Similar to how BitBake handles - include, - if the path specified - on the require line is a relative path, BitBake locates - the first file it can find within BBPATH. - - - - As an example, suppose you have two versions of a recipe - (e.g. foo_1.2.2.bb and - foo_2.0.0.bb) where - each version contains some identical functionality that co= uld be - shared. - You could create an include file named foo.inc - that contains the common definitions needed to build "foo". - You need to be sure foo.inc is locate= d in the - same directory as your two recipe files as well. - Once these conditions are set up, you can share the functi= onality - using a require directive from within= each - recipe: - - require foo.inc - - -
    - -
    - <filename>INHERIT</filename> Configuration Directive</t= itle> - - <para> - When creating a configuration file (<filename>.conf</filen= ame>), - you can use the - <link linkend=3D'var-bb-INHERIT'><filename>INHERIT</filena= me></link> - configuration directive to inherit a class. - BitBake only supports this directive when used within - a configuration file. - </para> - - <para> - As an example, suppose you needed to inherit a class - file called <filename>abc.bbclass</filename> from a - configuration file as follows: - <literallayout class=3D'monospaced'> - INHERIT +=3D "abc" - </literallayout> - This configuration directive causes the named - class to be inherited at the point of the directive - during parsing. - As with the <filename>inherit</filename> directive, the - <filename>.bbclass</filename> file must be located in a - "classes" subdirectory in one of the directories specified - in <filename>BBPATH</filename>. - <note> - Because <filename>.conf</filename> files are parsed - first during BitBake's execution, using - <filename>INHERIT</filename> to inherit a class effect= ively - inherits the class globally (i.e. for all recipes). - </note> - If you want to use the directive to inherit - multiple classes, you can provide them on the same line in= the - <filename>local.conf</filename> file. - Use spaces to separate the classes. - The following example shows how to inherit both the - <filename>autotools</filename> and <filename>pkgconfig</fi= lename> - classes: - <literallayout class=3D'monospaced'> - INHERIT +=3D "autotools pkgconfig" - </literallayout> - </para> - </section> - </section> - - <section id=3D'functions'> - <title>Functions - - - As with most languages, functions are the building blocks that - are used to build up operations into tasks. - BitBake supports these types of functions: - - Shell Functions: - Functions written in shell script and executed either - directly as functions, tasks, or both. - They can also be called by other shell functions. - - BitBake-Style Python Functions:<= /emphasis> - Functions written in Python and executed by BitBake or= other - Python functions using bb.build.exec_func()<= /filename>. - - Python Functions: - Functions written in Python and executed by Python. - - Anonymous Python Functions: - Python functions executed automatically during - parsing. - - - Regardless of the type of function, you can only - define them in class (.bbclass) - and recipe (.bb or .inc) - files. - - -
    - Shell Functions - - - Functions written in shell script and executed either - directly as functions, tasks, or both. - They can also be called by other shell functions. - Here is an example shell function definition: - - some_function () { - echo "Hello World" - } - - When you create these types of functions in your recipe - or class files, you need to follow the shell programming - rules. - The scripts are executed by /bin/sh, - which may not be a bash shell but might be something - such as dash. - You should not use Bash-specific script (bashisms). - - - - Overrides and override-style operators like - _append and - _prepend can also be applied to - shell functions. - Most commonly, this application would be used in a - .bbappend file to modify functions in - the main recipe. - It can also be used to modify functions inherited from - classes. - - - - As an example, consider the following: - - do_foo() { - bbplain first - fn - } - - fn_prepend() { - bbplain second - } - - fn() { - bbplain third - } - - do_foo_append() { - bbplain fourth - } - - Running do_foo - prints the following: - - recipename do_foo: first - recipename do_foo: second - recipename do_foo: third - recipename do_foo: fourth - - - Overrides and override-style operators can - be applied to any shell function, not just - tasks. - - You can use the bitbake -e recipename - command to view the final assembled function - after all overrides have been applied. - -
    - -
    - BitBake-Style Python Functions - - - These functions are written in Python and executed by - BitBake or other Python functions using - bb.build.exec_func(). - - - - An example BitBake function is: - - python some_python_function () { - d.setVar("TEXT", "Hello World") - print d.getVar("TEXT") - } - - Because the Python "bb" and "os" modules are already - imported, you do not need to import these modules. - Also in these types of functions, the datastore ("d") - is a global variable and is always automatically - available. - - Variable expressions (e.g. ${X}) - are no longer expanded within Python functions. - This behavior is intentional in order to allow you - to freely set variable values to expandable expressions - without having them expanded prematurely. - If you do wish to expand a variable within a Python - function, use d.getVar("X"). - Or, for more complicated expressions, use - d.expand(). - - - - - Similar to shell functions, you can also apply overrides - and override-style operators to BitBake-style Python - functions. - - - - As an example, consider the following: - - python do_foo_prepend() { - bb.plain("first") - } - - python do_foo() { - bb.plain("second") - } - - python do_foo_append() { - bb.plain("third") - } - - Running do_foo prints - the following: - - recipename do_foo: first - recipename do_foo: second - recipename do_foo: third - - You can use the bitbake -e recipename - command to view the final assembled function - after all overrides have been applied. - -
    - -
    - Python Functions - - - These functions are written in Python and are executed by - other Python code. - Examples of Python functions are utility functions - that you intend to call from in-line Python or - from within other Python functions. - Here is an example: - - def get_depends(d): - if d.getVar('SOMECONDITION'): - return "dependencywithcond" - else: - return "dependency" - SOMECONDITION =3D "1" - DEPENDS =3D "${@get_depends(d)}" - - This would result in DEPENDS - containing dependencywithcond. - - - - Here are some things to know about Python functions: - - Python functions can take parameters. - - The BitBake datastore is not - automatically available. - Consequently, you must pass it in as a - parameter to the function. - - The "bb" and "os" Python modules are - automatically available. - You do not need to import them. - - - -
    - -
    - BitBake-Style Python Functions Versus Python Functions<= /title> - - <para> - Following are some important differences between - BitBake-style Python functions and regular Python - functions defined with "def": - <itemizedlist> - <listitem><para> - Only BitBake-style Python functions can be - <link linkend=3D'tasks'>tasks</link>. - </para></listitem> - <listitem><para> - Overrides and override-style operators can only - be applied to BitBake-style Python functions. - </para></listitem> - <listitem><para> - Only regular Python functions can take arguments - and return values. - </para></listitem> - <listitem><para> - <link linkend=3D'variable-flags'>Variable flags</l= ink> - such as <filename>[dirs]</filename>, - <filename>[cleandirs]</filename>, and - <filename>[lockfiles]</filename> can be used - on BitBake-style Python functions, but not on - regular Python functions. - </para></listitem> - <listitem><para> - BitBake-style Python functions generate a separate - <filename>${</filename><link linkend=3D'var-bb-T'>= <filename>T</filename></link><filename>}/run.</filename><replaceable>functi= on-name</replaceable><filename>.</filename><replaceable>pid</replaceable> - script that is executed to run the function, and a= lso - generate a log file in - <filename>${T}/log.</filename><replaceable>functio= n-name</replaceable><filename>.</filename><replaceable>pid</replaceable> - if they are executed as tasks.</para> - - <para> - Regular Python functions execute "inline" and do n= ot - generate any files in <filename>${T}</filename>. - </para></listitem> - <listitem><para> - Regular Python functions are called with the usual - Python syntax. - BitBake-style Python functions are usually tasks a= nd - are called directly by BitBake, but can also be ca= lled - manually from Python code by using the - <filename>bb.build.exec_func()</filename> function. - Here is an example: - <literallayout class=3D'monospaced'> - bb.build.exec_func("my_bitbake_style_function", d) - </literallayout> - <note> - <filename>bb.build.exec_func()</filename> can = also - be used to run shell functions from Python cod= e. - If you want to run a shell function before a P= ython - function within the same task, then you can us= e a - parent helper Python function that starts by r= unning - the shell function with - <filename>bb.build.exec_func()</filename> and = then - runs the Python code. - </note></para> - - <para>To detect errors from functions executed with - <filename>bb.build.exec_func()</filename>, you - can catch the <filename>bb.build.FuncFailed</filen= ame> - exception. - <note> - Functions in metadata (recipes and classes) sh= ould - not themselves raise - <filename>bb.build.FuncFailed</filename>. - Rather, <filename>bb.build.FuncFailed</filenam= e> - should be viewed as a general indicator that t= he - called function failed by raising an exception. - For example, an exception raised by - <filename>bb.fatal()</filename> will be caught= inside - <filename>bb.build.exec_func()</filename>, and= a - <filename>bb.build.FuncFailed</filename> will = be raised - in response. - </note> - </para></listitem> - </itemizedlist> - </para> - - <para> - Due to their simplicity, you should prefer regular Python = functions - over BitBake-style Python functions unless you need a feat= ure specific - to BitBake-style Python functions. - Regular Python functions in metadata are a more recent inv= ention than - BitBake-style Python functions, and older code tends to use - <filename>bb.build.exec_func()</filename> more often. - </para> - </section> - - <section id=3D'anonymous-python-functions'> - <title>Anonymous Python Functions - - - Sometimes it is useful to set variables or perform - other operations programmatically during parsing. - To do this, you can define special Python functions, - called anonymous Python functions, that run at the - end of parsing. - For example, the following conditionally sets a variable - based on the value of another variable: - - python () { - if d.getVar('SOMEVAR') =3D=3D 'value': - d.setVar('ANOTHERVAR', 'value2') - } - - An equivalent way to mark a function as an anonymous - function is to give it the name "__anonymous", rather - than no name. - - - - Anonymous Python functions always run at the end - of parsing, regardless of where they are defined. - If a recipe contains many anonymous functions, they - run in the same order as they are defined within the - recipe. - As an example, consider the following snippet: - - python () { - d.setVar('FOO', 'foo 2') - } - - FOO =3D "foo 1" - - python () { - d.appendVar('BAR', ' bar 2') - } - - BAR =3D "bar 1" - - The previous example is conceptually equivalent to the - following snippet: - - FOO =3D "foo 1" - BAR =3D "bar 1" - FOO =3D "foo 2" - BAR +=3D "bar 2" - - FOO ends up with the value "foo 2", - and BAR with the value "bar 1 bar 2". - Just as in the second snippet, the values set for the - variables within the anonymous functions become available - to tasks, which always run after parsing. - - - - Overrides and override-style operators such as - "_append" are applied before - anonymous functions run. - In the following example, FOO ends - up with the value "foo from anonymous": - - FOO =3D "foo" - FOO_append =3D " from outside" - - python () { - d.setVar("FOO", "foo from anonymous") - } - - For methods you can use with anonymous Python functions, - see the - "Functions You Can Call From Within Python" - section. - For a different method to run Python code during parsing, - see the - "Inline= Python Variable Expansion" - section. - -
    - -
    - Flexible Inheritance for Class Functions - - - Through coding techniques and the use of - EXPORT_FUNCTIONS, BitBake supports - exporting a function from a class such that the - class function appears as the default implementation - of the function, but can still be called if a recipe - inheriting the class needs to define its own version of - the function. - - - - To understand the benefits of this feature, consider - the basic scenario where a class defines a task function - and your recipe inherits the class. - In this basic scenario, your recipe inherits the task - function as defined in the class. - If desired, your recipe can add to the start and end of the - function by using the "_prepend" or "_append" operations - respectively, or it can redefine the function completely. - However, if it redefines the function, there is - no means for it to call the class version of the function. - EXPORT_FUNCTIONS provides a mechanism - that enables the recipe's version of the function to call - the original version of the function. - - - - To make use of this technique, you need the following - things in place: - - - The class needs to define the function as follows: - - classname_functionname - - For example, if you have a class file - bar.bbclass and a function na= med - do_foo, the class must define= the function - as follows: - - bar_do_foo - - - - The class needs to contain the EXPORT_FU= NCTIONS - statement as follows: - - EXPORT_FUNCTIONS functionname - - For example, continuing with the same example, the - statement in the bar.bbclass = would be - as follows: - - EXPORT_FUNCTIONS do_foo - - - - You need to call the function appropriately from w= ithin your - recipe. - Continuing with the same example, if your recipe - needs to call the class version of the function, - it should call bar_do_foo. - Assuming do_foo was a shell f= unction - and EXPORT_FUNCTIONS was used= as above, - the recipe's function could conditionally call the - class version of the function as follows: - - do_foo() { - if [ somecondition ] ; then - bar_do_foo - else - # Do something else - fi - } - - To call your modified version of the function as d= efined - in your recipe, call it as do_foo. - - - With these conditions met, your single recipe - can freely choose between the original function - as defined in the class file and the modified function in = your recipe. - If you do not set up these conditions, you are limited to = using one function - or the other. - -
    -
    - -
    - Tasks - - - Tasks are BitBake execution units that make up the - steps that BitBake can run for a given recipe. - Tasks are only supported in recipes and classes - (i.e. in .bb files and files - included or inherited from .bb - files). - By convention, tasks have names that start with "do_". - - -
    - Promoting a Function to a Task - - - Tasks are either - shell functions or - BitBake-s= tyle Python functions - that have been promoted to tasks by using the - addtask command. - The addtask command can also - optionally describe dependencies between the - task and other tasks. - Here is an example that shows how to define a task - and declare some dependencies: - - python do_printdate () { - import time - print time.strftime('%Y%m%d', time.gmtime()) - } - addtask printdate after do_fetch before do_build - - The first argument to addtask - is the name of the function to promote to - a task. - If the name does not start with "do_", "do_" is - implicitly added, which enforces the convention that - all task names start with "do_". - - - - In the previous example, the - do_printdate task becomes a - dependency of the do_build - task, which is the default task (i.e. the task run by - the bitbake command unless - another task is specified explicitly). - Additionally, the do_printdate - task becomes dependent upon the - do_fetch task. - Running the do_build task - results in the do_printdate - task running first. - - If you try out the previous example, you might see that - the do_printdate task is only run - the first time you build the recipe with - the bitbake command. - This is because BitBake considers the task "up-to-date" - after that initial run. - If you want to force the task to always be rerun for - experimentation purposes, you can make BitBake always - consider the task "out-of-date" by using the - [nostamp] - variable flag, as follows: - - do_printdate[nostamp] =3D "1" - - You can also explicitly run the task and provide the - -f option as follows: - - $ bitbake recipe -c printdate -f - - When manually selecting a task to run with the - bitbake recipe<= /replaceable> -c task - command, you can omit the "do_" prefix as part of the - task name. - - - - - You might wonder about the practical effects of using - addtask without specifying any - dependencies as is done in the following example: - - addtask printdate - - In this example, assuming dependencies have not been - added through some other means, the only way to run - the task is by explicitly selecting it with - bitbake recipe -c printdate. - You can use the - do_listtasks task to list all tasks - defined in a recipe as shown in the following example: - - $ bitbake recipe -c listtasks - - For more information on task dependencies, see the - "Dependencies" - section. - - - - See the - "Variable Flags" - section for information on variable flags you can use with - tasks. - -
    - -
    - Deleting a Task - - - As well as being able to add tasks, you can delete them. - Simply use the deltask command to - delete a task. - For example, to delete the example task used in the previo= us - sections, you would use: - - deltask printdate - - If you delete a task using the deltask - command and the task has dependencies, the dependencies are - not reconnected. - For example, suppose you have three tasks named - do_a, do_b, and - do_c. - Furthermore, do_c is dependent on - do_b, which in turn is dependent on - do_a. - Given this scenario, if you use deltask - to delete do_b, the implicit dependen= cy - relationship between do_c and - do_a through do_b - no longer exists, and do_c dependenci= es - are not updated to include do_a. - Thus, do_c is free to run before - do_a. - - - - If you want dependencies such as these to remain intact, u= se - the [noexec] varflag to disable the t= ask - instead of using the deltask command = to - delete it: - - do_b[noexec] =3D "1" - - -
    - -
    - Passing Information Into the Build Task Environment</ti= tle> - - <para> - When running a task, BitBake tightly controls the shell ex= ecution - environment of the build tasks to make - sure unwanted contamination from the build machine cannot - influence the build. - <note> - By default, BitBake cleans the environment to include = only those - things exported or listed in its whitelist to ensure t= hat the build - environment is reproducible and consistent. - You can prevent this "cleaning" by setting the - <link linkend=3D'var-bb-BB_PRESERVE_ENV'><filename>BB_= PRESERVE_ENV</filename></link> - variable. - </note> - Consequently, if you do want something to get passed into = the - build task environment, you must take these two steps: - <orderedlist> - <listitem><para> - Tell BitBake to load what you want from the enviro= nment - into the datastore. - You can do so through the - <link linkend=3D'var-bb-BB_ENV_WHITELIST'><filenam= e>BB_ENV_WHITELIST</filename></link> - and - <link linkend=3D'var-bb-BB_ENV_EXTRAWHITE'><filena= me>BB_ENV_EXTRAWHITE</filename></link> - variables. - For example, assume you want to prevent the build = system from - accessing your <filename>$HOME/.ccache</filename> - directory. - The following command "whitelists" the environment= variable - <filename>CCACHE_DIR</filename> causing BitBake to= allow that - variable into the datastore: - <literallayout class=3D'monospaced'> - export BB_ENV_EXTRAWHITE=3D"$BB_ENV_EXTRAWHITE CCACHE_DIR" - </literallayout></para></listitem> - <listitem><para> - Tell BitBake to export what you have loaded into t= he - datastore to the task environment of every running= task. - Loading something from the environment into the da= tastore - (previous step) only makes it available in the dat= astore. - To export it to the task environment of every runn= ing task, - use a command similar to the following in your loc= al configuration - file <filename>local.conf</filename> or your - distribution configuration file: - <literallayout class=3D'monospaced'> - export CCACHE_DIR - </literallayout> - <note> - A side effect of the previous steps is that Bi= tBake - records the variable as a dependency of the bu= ild process - in things like the setscene checksums. - If doing so results in unnecessary rebuilds of= tasks, you can - whitelist the variable so that the setscene co= de - ignores the dependency when it creates checksu= ms. - </note></para></listitem> - </orderedlist> - </para> - - <para> - Sometimes, it is useful to be able to obtain information - from the original execution environment. - BitBake saves a copy of the original environment into - a special variable named - <link linkend=3D'var-bb-BB_ORIGENV'><filename>BB_ORIGENV</= filename></link>. - </para> - - <para> - The <filename>BB_ORIGENV</filename> variable returns a dat= astore - object that can be queried using the standard datastore op= erators - such as <filename>getVar(, False)</filename>. - The datastore object is useful, for example, to find the o= riginal - <filename>DISPLAY</filename> variable. - Here is an example: - <literallayout class=3D'monospaced'> - origenv =3D d.getVar("BB_ORIGENV", False) - bar =3D origenv.getVar("BAR", False) - </literallayout> - The previous example returns <filename>BAR</filename> from= the original - execution environment. - </para> - </section> - </section> - - <section id=3D'variable-flags'> - <title>Variable Flags - - - Variable flags (varflags) help control a task's functionality - and dependencies. - BitBake reads and writes varflags to the datastore using the f= ollowing - command forms: - - variable =3D d.getVarFlags("v= ariable") - self.d.setVarFlags("FOO", {"func": True}) - - - - - When working with varflags, the same syntax, with the exceptio= n of - overrides, applies. - In other words, you can set, append, and prepend varflags just= like - variables. - See the - "Variable Flag Syntax" - section for details. - - - - BitBake has a defined set of varflags available for recipes and - classes. - Tasks support a number of these flags which control various - functionality of the task: - - [cleandirs]= : - Empty directories that should be created before the - task runs. - Directories that already exist are removed and recreat= ed - to empty them. - - [depends]:<= /emphasis> - Controls inter-task dependencies. - See the - DEPENDS - variable and the - "Inter-Task = Dependencies" - section for more information. - - [deptask]:<= /emphasis> - Controls task build-time dependencies. - See the - DEPENDS - variable and the - "Build Dependenci= es" - section for more information. - - [dirs]: - Directories that should be created before the task run= s. - Directories that already exist are left as is. - The last directory listed is used as the - current working directory for the task. - - [lockfiles]= : - Specifies one or more lockfiles to lock while the task - executes. - Only one task may hold a lockfile, and any task that - attempts to lock an already locked file will block un= til - the lock is released. - You can use this variable flag to accomplish mutual - exclusion. - - [noexec]: - When set to "1", marks the task as being empty, with - no execution required. - You can use the [noexec] flag to = set up - tasks as dependency placeholders, or to disable tasks = defined - elsewhere that are not needed in a particular recipe. - - [nostamp]:<= /emphasis> - When set to "1", tells BitBake to not generate a stamp - file for a task, which implies the task should always - be executed. - Caution - Any task that depends (possibly indirectly) on a - [nostamp] task will always be - executed as well. - This can cause unnecessary rebuilding if you are - not careful. - - - [number_threads]: - Limits tasks to a specific number of simultaneous thre= ads - during execution. - This varflag is useful when your build host has a larg= e number - of cores but certain tasks need to be rate-limited due= to various - kinds of resource constraints (e.g. to avoid network t= hrottling). - number_threads works similarly to= the - B= B_NUMBER_THREADS - variable but is task-specific. - - Set the value globally. - For example, the following makes sure the - do_fetch task uses no more than t= wo - simultaneous execution threads: - - do_fetch[number_threads] =3D "2" - - Warnings - - - Setting the varflag in individual recipes = rather - than globally can result in unpredictable = behavior. - - - Setting the varflag to a value greater tha= n the - value used in the BB_NUMBER_THRE= ADS - variable causes number_threads - to have no effect. - - - - - [postfuncs]= : - List of functions to call after the completion of the = task. - - [prefuncs]:= - List of functions to call before the task executes. - - [rdepends]:= - Controls inter-task runtime dependencies. - See the - RDEPENDS - variable, the - RRECOMM= ENDS - variable, and the - "Inter-Task = Dependencies" - section for more information. - - [rdeptask]:= - Controls task runtime dependencies. - See the - RDEPENDS - variable, the - RRECOMM= ENDS - variable, and the - "Runtime Depend= encies" - section for more information. - - [recideptask]: - When set in conjunction with - recrdeptask, specifies a task that - should be inspected for additional dependencies. - - [recrdeptask]: - Controls task recursive runtime dependencies. - See the - RDEPENDS - variable, the - RRECOMM= ENDS - variable, and the - "Recursive De= pendencies" - section for more information. - - [stamp-extra-info]: - Extra stamp information to append to the task's stamp. - As an example, OpenEmbedded uses this flag to allow - machine-specific tasks. - - [umask]: - The umask to run the task under. - - - - - - Several varflags are useful for controlling how signatures are - calculated for variables. - For more information on this process, see the - "Checksums (Signatures)" - section. - - [vardeps]:<= /emphasis> - Specifies a space-separated list of additional - variables to add to a variable's dependencies - for the purposes of calculating its signature. - Adding variables to this list is useful, for example, = when - a function refers to a variable in a manner that - does not allow BitBake to automatically determine - that the variable is referred to. - - [vardepsexclude]: - Specifies a space-separated list of variables - that should be excluded from a variable's dependencies - for the purposes of calculating its signature. - - [vardepvalue]: - If set, instructs BitBake to ignore the actual - value of the variable and instead use the specified - value when calculating the variable's signature. - - [vardepvalueexclude]: - Specifies a pipe-separated list of strings to exclude - from the variable's value when calculating the - variable's signature. - - - -
    - -
    - Events - - - BitBake allows installation of event handlers within recipe - and class files. - Events are triggered at certain points during operation, such - as the beginning of operation against a given recipe - (i.e. *.bb), the start of a given task, - a task failure, a task success, and so forth. - The intent is to make it easy to do things like email - notification on build failures. - - - - Following is an example event handler that prints the name - of the event and the content of the - FILE variable: - - addhandler myclass_eventhandler - python myclass_eventhandler() { - from bb.event import getName - print("The name of the Event is %s" % getName(e)) - print("The file we run for is %s" % d.getVar('FILE')) - } - myclass_eventhandler[eventmask] =3D "bb.event.BuildStarted bb.event.B= uildCompleted" - - In the previous example, an eventmask has been set so that - the handler only sees the "BuildStarted" and "BuildCompleted" - events. - This event handler gets called every time an event matching - the eventmask is triggered. - A global variable "e" is defined, which represents the current - event. - With the getName(e) method, you can get - the name of the triggered event. - The global datastore is available as "d". - In legacy code, you might see "e.data" used to get the datasto= re. - However, realize that "e.data" is deprecated and you should use - "d" going forward. - - - - The context of the datastore is appropriate to the event - in question. - For example, "BuildStarted" and "BuildCompleted" events run - before any tasks are executed so would be in the global - configuration datastore namespace. - No recipe-specific metadata exists in that namespace. - The "BuildStarted" and "BuildCompleted" events also run in - the main cooker/server process rather than any worker context. - Thus, any changes made to the datastore would be seen by other - cooker/server events within the current build but not seen - outside of that build or in any worker context. - Task events run in the actual tasks in question consequently - have recipe-specific and task-specific contents. - These events run in the worker context and are discarded at - the end of task execution. - - - - During a standard build, the following common events might - occur. - The following events are the most common kinds of events that - most metadata might have an interest in viewing: - - - bb.event.ConfigParsed(): - Fired when the base configuration; which consists of - bitbake.conf, - base.bbclass and any global - INHERIT statements; has been pars= ed. - You can see multiple such events when each of the - workers parse the base configuration or if the server - changes configuration and reparses. - Any given datastore only has one such event executed - against it, however. - If - BB_= INVALIDCONF - is set in the datastore by the event handler, the - configuration is reparsed and a new event triggered, - allowing the metadata to update configuration. - - - bb.event.HeartbeatEvent(): - Fires at regular time intervals of one second. - You can configure the interval time using the - BB_HEARTBEAT_EVENT variable. - The event's "time" attribute is the - time.time() value when the - event is triggered. - This event is useful for activities such as - system state monitoring. - - - bb.event.ParseStarted(): - Fired when BitBake is about to start parsing recipes. - This event's "total" attribute represents the number of - recipes BitBake plans to parse. - - - bb.event.ParseProgress(): - Fired as parsing progresses. - This event's "current" attribute is the number of - recipes parsed as well as the "total" attribute. - - - bb.event.ParseCompleted(): - Fired when parsing is complete. - This event's "cached", "parsed", "skipped", "virtuals", - "masked", and "errors" attributes provide statistics - for the parsing results. - - - bb.event.BuildStarted(): - Fired when a new build starts. - BitBake fires multiple "BuildStarted" events (one per = configuration) - when multiple configuration (multiconfig) is enabled. - - - bb.build.TaskStarted(): - Fired when a task starts. - This event's "taskfile" attribute points to the recipe - from which the task originates. - The "taskname" attribute, which is the task's name, - includes the do_ prefix, and the - "logfile" attribute point to where the task's output is - stored. - Finally, the "time" attribute is the task's execution = start - time. - - - bb.build.TaskInvalid(): - Fired if BitBake tries to execute a task that does not= exist. - - - bb.build.TaskFailedSilent(): - Fired for setscene tasks that fail and should not be - presented to the user verbosely. - - - bb.build.TaskFailed(): - Fired for normal tasks that fail. - - - bb.build.TaskSucceeded(): - Fired when a task successfully completes. - - - bb.event.BuildCompleted(): - Fired when a build finishes. - - - bb.cooker.CookerExit(): - Fired when the BitBake server/cooker shuts down. - This event is usually only seen by the UIs as a - sign they should also shutdown. - - - - - - This next list of example events occur based on specific - requests to the server. - These events are often used to communicate larger pieces of - information from the BitBake server to other parts of - BitBake such as user interfaces: - - - bb.event.TreeDataPreparationStarted() - - - bb.event.TreeDataPreparationProgress() - - - bb.event.TreeDataPreparationCompleted() - - - bb.event.DepTreeGenerated() - - - bb.event.CoreBaseFilesFound() - - - bb.event.ConfigFilePathFound() - - - bb.event.FilesMatchingFound() - - - bb.event.ConfigFilesFound() - - - bb.event.TargetsTreeGenerated() - - - -
    - -
    - Variants - Class Extension Mechanism - - - BitBake supports two features that facilitate creating - from a single recipe file multiple incarnations of that - recipe file where all incarnations are buildable. - These features are enabled through the - BBCLASSEXTEND= - and - BBVERSIONS - variables. - - The mechanism for this class extension is extremely - specific to the implementation. - Usually, the recipe's - PROVIDES, - PN= , and - DEPENDS - variables would need to be modified by the extension class. - For specific examples, see the OE-Core - native, nativesdk, - and multilib classes. - - - BBCLASSEXTEND: - This variable is a space separated list of classes use= d to "extend" the - recipe for each variant. - Here is an example that results in a second incarnatio= n of the current - recipe being available. - This second incarnation will have the "native" class i= nherited. - - BBCLASSEXTEND =3D "native" - - BBVERSIONS:= - This variable allows a single recipe to build multiple= versions of a - project from a single recipe file. - You can also specify conditional metadata - (using the - OVERRIDES= - mechanism) for a single version, or an optionally name= d range of versions. - Here is an example: - - BBVERSIONS =3D "1.0 2.0 git" - SRC_URI_git =3D "git://someurl/somepath.git" - - BBVERSIONS =3D "1.0.[0-6]:1.0.0+ \ 1.0.[7-9]:1.0.7+" - SRC_URI_append_1.0.7+ =3D "file://some_patch_which_the_new_versions_n= eed.patch;patch=3D1" - - The name of the range defaults to the original version= of the - recipe. - For example, in OpenEmbedded, the recipe file - foo_1.0.0+.bb creates a default n= ame range - of 1.0.0+. - This is useful because the range name is not only plac= ed - into overrides, but it is also made available for the = metadata to use - in the variable that defines the base recipe versions = for use in - file:// search paths - (FILESPAT= H). - - - -
    - -
    - Dependencies - - - To allow for efficient parallel processing, BitBake handles - dependencies at the task level. - Dependencies can exist both between tasks within a single reci= pe - and between tasks in different recipes. - Following are examples of each: - - For tasks within a single recipe, a - recipe's do_configure - task might need to complete before its - do_compile task can run. - - For tasks in different recipes, one - recipe's do_configure - task might require another recipe's - do_populate_sysroot - task to finish first such that the libraries and heade= rs - provided by the other recipe are available. - - - - - - This section describes several ways to declare dependencies. - Remember, even though dependencies are declared in different = ways, they - are all simply dependencies between tasks. - - -
    - Dependencies Internal to the <filename>.bb</filename> F= ile - - - BitBake uses the addtask directive - to manage dependencies that are internal to a given recipe - file. - You can use the addtask directive to - indicate when a task is dependent on other tasks or when - other tasks depend on that recipe. - Here is an example: - - addtask printdate after do_fetch before do_build - - In this example, the do_printdate - task depends on the completion of the - do_fetch task, and the - do_build task depends on the - completion of the do_printdate - task. - - For a task to run, it must be a direct or indirect - dependency of some other task that is scheduled to - run. - - For illustration, here are some examples: - - - The directive - addtask mytask before do_configure - causes do_mytask to run b= efore - do_configure runs. - Be aware that do_mytask s= till only - runs if its input = checksum - has changed since the last time it was run. - Changes to the input checksum of - do_mytask also indirectly= cause - do_configure to run. - - - The directive - addtask mytask after do_configure - by itself never causes do_mytask - to run. - do_mytask can still be ru= n manually - as follows: - - $ bitbake recipe -c mytask - - Declaring do_mytask as a = dependency - of some other task that is scheduled to run al= so causes - it to run. - Regardless, the task runs after - do_configure. - - - - -
    - -
    - Build Dependencies - - - BitBake uses the - DEPENDS - variable to manage build time dependencies. - The [deptask] varflag for tasks - signifies the task of each - item listed in DEPENDS that must - complete before that task can be executed. - Here is an example: - - do_configure[deptask] =3D "do_populate_sysroot" - - In this example, the do_populate_sysroot - task of each item in DEPENDS must com= plete before - do_configure can execute. - -
    - -
    - Runtime Dependencies - - - BitBake uses the - PACKAGES, - RDEPENDS, and - RRECOMMENDS= - variables to manage runtime dependencies. - - - - The PACKAGES variable lists runtime - packages. - Each of those packages can have RDEPENDS and - RRECOMMENDS runtime dependencies. - The [rdeptask] flag for tasks is used= to - signify the task of each - item runtime dependency which must have completed before t= hat - task can be executed. - - do_package_qa[rdeptask] =3D "do_packagedata" - - In the previous example, the do_packagedata - task of each item in RDEPENDS must ha= ve - completed before do_package_qa can ex= ecute. - Although RDEPENDS contains entries fr= om the - runtime dependency namespace, BitBake knows how to map the= m back - to the build-time dependency namespace, in which the tasks= are defined. - -
    - -
    - Recursive Dependencies - - - BitBake uses the [recrdeptask] flag t= o manage - recursive task dependencies. - BitBake looks through the build-time and runtime - dependencies of the current recipe, looks through - the task's inter-task - dependencies, and then adds dependencies for the - listed task. - Once BitBake has accomplished this, it recursively works t= hrough - the dependencies of those tasks. - Iterative passes continue until all dependencies are disco= vered - and added. - - - - The [recrdeptask] flag is most common= ly - used in high-level - recipes that need to wait for some task to finish "globall= y". - For example, image.bbclass has the fo= llowing: - - do_rootfs[recrdeptask] +=3D "do_packagedata" - - This statement says that the do_packagedata - task of the current recipe and all recipes reachable - (by way of dependencies) from the - image recipe must run before the do_rootfs - task can run. - - - - BitBake allows a task to recursively depend on itself by - referencing itself in the task list: - - do_a[recrdeptask] =3D "do_a do_b" - - In the same way as before, this means that the d= o_a - and do_b tasks of the current recipe = and all - recipes reachable (by way of dependencies) from the recipe - must run before the do_a task can run= . In this - case BitBake will ignore the current recipe's do= _a - task circular dependency on itself. - -
    - -
    - Inter-Task Dependencies - - - BitBake uses the [depends] - flag in a more generic form - to manage inter-task dependencies. - This more generic form allows for inter-dependency - checks for specific tasks rather than checks for - the data in DEPENDS. - Here is an example: - - do_patch[depends] =3D "quilt-native:do_populate_sysroot" - - In this example, the do_populate_sysroot - task of the target quilt-native - must have completed before the - do_patch task can execute. - - - - The [rdepends] flag works in a similar - way but takes targets - in the runtime namespace instead of the build-time depende= ncy - namespace. - -
    -
    - -
    - Functions You Can Call From Within Python - - - BitBake provides many functions you can call from - within Python functions. - This section lists the most commonly used functions, - and mentions where to find others. - - -
    - Functions for Accessing Datastore Variables - - - It is often necessary to access variables in the - BitBake datastore using Python functions. - The BitBake datastore has an API that allows you this - access. - Here is a list of available operations: - - - - - - - - - - Operation<= /emphasis> - Descriptio= n - - - - - d.getVar("= X", expand) - Returns the value of= variable "X". - Using "expand=3DTrue" expands the valu= e. - Returns "None" if the variable "X" doe= s not exist. - - - d.setVar("= X", "value") - Sets the variable "X= " to "value". - - - d.appendVa= r("X", "value") - Adds "value" to the = end of the variable "X". - Acts like d.setVar("X", "val= ue") - if the variable "X" does not exist. - - - d.prependV= ar("X", "value") - Adds "value" to the = start of the variable "X". - Acts like d.setVar("X", "val= ue") - if the variable "X" does not exist. - - - d.delVar("= X") - Deletes the variable= "X" from the datastore. - Does nothing if the variable "X" does = not exist. - - - d.renameVa= r("X", "Y") - Renames the variable= "X" to "Y". - Does nothing if the variable "X" does = not exist. - - - d.getVarFl= ag("X", flag, expand) - Returns the value of= variable "X". - Using "expand=3DTrue" expands the valu= e. - Returns "None" if either the variable = "X" or the named flag - does not exist. - - - d.setVarFl= ag("X", flag, "value") - Sets the named flag = for variable "X" to "value". - - - d.appendVa= rFlag("X", flag, "value") - Appends "value" to t= he named flag on the - variable "X". - Acts like d.setVarFlag("X", = flag, "value") - if the named flag does not exist. - - - d.prependV= arFlag("X", flag, "value") - Prepends "value" to = the named flag on - the variable "X". - Acts like d.setVarFlag("X", = flag, "value") - if the named flag does not exist. - - - d.delVarFl= ag("X", flag) - Deletes the named fl= ag on the variable - "X" from the datastore. - - - d.setVarFl= ags("X", flagsdict) - Sets the flags speci= fied in - the flagsdict() p= arameter. - setVarFlags does = not clear previous flags. - Think of this operation as a= ddVarFlags. - - - d.getVarFl= ags("X") - Returns a = flagsdict - of the flags for the variable "X". - Returns "None" if the variable "X" doe= s not exist. - - - d.delVarFl= ags("X") - Deletes all the flag= s for the variable "X". - Does nothing if the variable "X" does = not exist. - - - d.expand(e= xpression) - Expands variable ref= erences in the specified - string expression. - References to variables that do not ex= ist are left as is. - For example, d.expand("foo $= {X}") - expands to the literal string "foo ${X= }" if the - variable "X" does not exist. - - - - - -
    - -
    - Other Functions - - - You can find many other functions that can be called - from Python by looking at the source code of the - bb module, which is in - bitbake/lib/bb. - For example, - bitbake/lib/bb/utils.py includes - the commonly used functions - bb.utils.contains() and - bb.utils.mkdirhier(), which come - with docstrings. - -
    -
    - -
    - Task Checksums and Setscene - - - BitBake uses checksums (or signatures) along with the setscene - to determine if a task needs to be run. - This section describes the process. - To help understand how BitBake does this, the section assumes = an - OpenEmbedded metadata-based example. - - - - These checksums are stored in - STAMP. - You can examine the checksums using the following BitBake comm= and: - - $ bitbake-dumpsigs - - This command returns the signature data in a readable format - that allows you to examine the inputs used when the - OpenEmbedded build system generates signatures. - For example, using bitbake-dumpsigs - allows you to examine the do_compile - task's =E2=80=9Csigdata=E2=80=9D for a C application (e.g. - bash). - Running the command also reveals that the =E2=80=9CCC=E2=80=9D= variable is part of - the inputs that are hashed. - Any changes to this variable would invalidate the stamp and - cause the do_compile task to run. - - - - The following list describes related variables: - - - BB_HASHCHECK_FUNCTION: - Specifies the name of the function to call during - the "setscene" part of the task's execution in order - to validate the list of task hashes. - - - BB_SETSCENE_DEPVALID: - Specifies a function BitBake calls that determines - whether BitBake requires a setscene dependency to - be met. - - - = BB_SETSCENE_VERIFY_FUNCTION2: - Specifies a function to call that verifies the list of - planned task execution before the main task execution - happens. - - - BB_= STAMP_POLICY: - Defines the mode for comparing timestamps of stamp fil= es. - - - = BB_STAMP_WHITELIST: - Lists stamp files that are looked at when the stamp po= licy - is "whitelist". - - - BB_TASK= HASH: - Within an executing task, this variable holds the hash - of the task as returned by the currently enabled - signature generator. - - - STAMP: - The base path to create stamp files. - - - STAMPCLE= AN: - Again, the base path to create stamp files but can use= wildcards - for matching a range of files for clean operations. - - - -
    - -
    - Wildcard Support in Variables - - - Support for wildcard use in variables varies depending on the - context in which it is used. - For example, some variables and file names allow limited use of - wildcards through the "%" and - "*" characters. - Other variables or names support Python's - glob - syntax, - fnmatch - syntax, or - Regular Expression (re) - syntax. - - - - For variables that have wildcard suport, the - documentation describes which form of wildcard, its - use, and its limitations. - -
    - - diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variab= les.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables= .rst new file mode 100644 index 00000000..489fa15f --- /dev/null +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst @@ -0,0 +1,1405 @@ +.. SPDX-License-Identifier: CC-BY-2.5 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Variables Glossary +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +| + +This chapter lists common variables used by BitBake and gives an +overview of their function and contents. + +.. note:: + + Following are some points regarding the variables listed in this + glossary: + + - The variables listed in this glossary are specific to BitBake. + Consequently, the descriptions are limited to that context. + + - Also, variables exist in other systems that use BitBake (e.g. The + Yocto Project and OpenEmbedded) that have names identical to those + found in this glossary. For such cases, the variables in those + systems extend the functionality of the variable as it is + described here in this glossary. + + - Finally, there are variables mentioned in this glossary that do + not appear in the BitBake glossary. These other variables are + variables used in systems that use BitBake. + +.. glossary:: + + :term:`ASSUME_PROVIDED` + Lists recipe names (:term:`PN` values) BitBake does not + attempt to build. Instead, BitBake assumes these recipes have already + been built. + + In OpenEmbedded-Core, ``ASSUME_PROVIDED`` mostly specifies native + tools that should not be built. An example is ``git-native``, which + when specified allows for the Git binary from the host to be used + rather than building ``git-native``. + + :term:`AZ_SAS` + Azure Storage Shared Access Signature, when using the + :ref:`Azure Storage fetcher ` + This variable can be defined to be used by the fetcher to authentica= te + and gain access to non-public artifacts. + :: + + AZ_SAS =3D ""se=3D2021-01-01&sp=3Dr&sv=3D2018-11-09&sr=3Dc&skoid= =3D&sig=3D"" + + For more information see Microsoft's Azure Storage documentation at + https://docs.microsoft.com/en-us/azure/storage/common/storage-sas-ov= erview + + + :term:`B` + The directory in which BitBake executes functions during a recipe's + build process. + + :term:`BB_ALLOWED_NETWORKS` + Specifies a space-delimited list of hosts that the fetcher is allowed + to use to obtain the required source code. Following are + considerations surrounding this variable: + + - This host list is only used if + :term:`BB_NO_NETWORK` is either not set or + set to "0". + + - Limited support for the "``*``" wildcard character for matching + against the beginning of host names exists. For example, the + following setting matches ``git.gnu.org``, ``ftp.gnu.org``, and + ``foo.git.gnu.org``. :: + + BB_ALLOWED_NETWORKS =3D "\*.gnu.org" + + .. important:: + + The use of the "``*``" character only works at the beginning of + a host name and it must be isolated from the remainder of the + host name. You cannot use the wildcard character in any other + location of the name or combined with the front part of the + name. + + For example, ``*.foo.bar`` is supported, while ``*aa.foo.bar`` + is not. + + - Mirrors not in the host list are skipped and logged in debug. + + - Attempts to access networks not in the host list cause a failure. + + Using ``BB_ALLOWED_NETWORKS`` in conjunction with + :term:`PREMIRRORS` is very useful. Adding the + host you want to use to ``PREMIRRORS`` results in the source code + being fetched from an allowed location and avoids raising an error + when a host that is not allowed is in a + :term:`SRC_URI` statement. This is because the + fetcher does not attempt to use the host listed in ``SRC_URI`` after + a successful fetch from the ``PREMIRRORS`` occurs. + + :term:`BB_CONSOLELOG` + Specifies the path to a log file into which BitBake's user interface + writes output during the build. + + :term:`BB_CURRENTTASK` + Contains the name of the currently running task. The name does not + include the ``do_`` prefix. + + :term:`BB_DANGLINGAPPENDS_WARNONLY` + Defines how BitBake handles situations where an append file + (``.bbappend``) has no corresponding recipe file (``.bb``). This + condition often occurs when layers get out of sync (e.g. ``oe-core`` + bumps a recipe version and the old recipe no longer exists and the + other layer has not been updated to the new version of the recipe + yet). + + The default fatal behavior is safest because it is the sane reaction + given something is out of sync. It is important to realize when your + changes are no longer being applied. + + :term:`BB_DEFAULT_TASK` + The default task to use when none is specified (e.g. with the ``-c`` + command line option). The task name specified should not include the + ``do_`` prefix. + + :term:`BB_DEFAULT_UMASK` + The default umask to apply to tasks if specified and no task specific + umask flag is set. + + :term:`BB_DISKMON_DIRS` + Monitors disk space and available inodes during the build and allows + you to control the build based on these parameters. + + Disk space monitoring is disabled by default. When setting this + variable, use the following form: :: + + BB_DISKMON_DIRS =3D ",, [...]" + + where: + + is: + ABORT: Immediately abort the build when + a threshold is broken. + STOPTASKS: Stop the build after the currently + executing tasks have finished when + a threshold is broken. + WARN: Issue a warning but continue the + build when a threshold is broken. + Subsequent warnings are issued as + defined by the + BB_DISKMON_WARNINTERVAL variable, + which must be defined. + + is: + Any directory you choose. You can specify one or + more directories to monitor by separating the + groupings with a space. If two directories are + on the same device, only the first directory + is monitored. + + is: + Either the minimum available disk space, + the minimum number of free inodes, or + both. You must specify at least one. To + omit one or the other, simply omit the value. + Specify the threshold using G, M, K for Gbytes, + Mbytes, and Kbytes, respectively. If you do + not specify G, M, or K, Kbytes is assumed by + default. Do not use GB, MB, or KB. + + Here are some examples: :: + + BB_DISKMON_DIRS =3D "ABORT,${TMPDIR},1G,100K WARN,${SSTATE_DIR},1= G,100K" + BB_DISKMON_DIRS =3D "STOPTASKS,${TMPDIR},1G" + BB_DISKMON_DIRS =3D "ABORT,${TMPDIR},,100K" + + The first example works only if you also set the + :term:`BB_DISKMON_WARNINTERVAL` + variable. This example causes the build system to immediately abort + when either the disk space in ``${TMPDIR}`` drops below 1 Gbyte or + the available free inodes drops below 100 Kbytes. Because two + directories are provided with the variable, the build system also + issues a warning when the disk space in the ``${SSTATE_DIR}`` + directory drops below 1 Gbyte or the number of free inodes drops + below 100 Kbytes. Subsequent warnings are issued during intervals as + defined by the ``BB_DISKMON_WARNINTERVAL`` variable. + + The second example stops the build after all currently executing + tasks complete when the minimum disk space in the ``${TMPDIR}`` + directory drops below 1 Gbyte. No disk monitoring occurs for the free + inodes in this case. + + The final example immediately aborts the build when the number of + free inodes in the ``${TMPDIR}`` directory drops below 100 Kbytes. No + disk space monitoring for the directory itself occurs in this case. + + :term:`BB_DISKMON_WARNINTERVAL` + Defines the disk space and free inode warning intervals. + + If you are going to use the ``BB_DISKMON_WARNINTERVAL`` variable, you + must also use the :term:`BB_DISKMON_DIRS` + variable and define its action as "WARN". During the build, + subsequent warnings are issued each time disk space or number of free + inodes further reduces by the respective interval. + + If you do not provide a ``BB_DISKMON_WARNINTERVAL`` variable and you + do use ``BB_DISKMON_DIRS`` with the "WARN" action, the disk + monitoring interval defaults to the following: + BB_DISKMON_WARNINTERVAL =3D "50M,5K" + + When specifying the variable in your configuration file, use the + following form: :: + + BB_DISKMON_WARNINTERVAL =3D "," + + where: + + is: + An interval of memory expressed in either + G, M, or K for Gbytes, Mbytes, or Kbytes, + respectively. You cannot use GB, MB, or KB. + + is: + An interval of free inodes expressed in either + G, M, or K for Gbytes, Mbytes, or Kbytes, + respectively. You cannot use GB, MB, or KB. + + Here is an example: :: + + BB_DISKMON_DIRS =3D "WARN,${SSTATE_DIR},1G,100K" + BB_DISKMON_WARNINTERVAL =3D "50M,5K" + + These variables cause BitBake to + issue subsequent warnings each time the available disk space further + reduces by 50 Mbytes or the number of free inodes further reduces by + 5 Kbytes in the ``${SSTATE_DIR}`` directory. Subsequent warnings + based on the interval occur each time a respective interval is + reached beyond the initial warning (i.e. 1 Gbytes and 100 Kbytes). + + :term:`BB_ENV_WHITELIST` + Specifies the internal whitelist of variables to allow through from + the external environment into BitBake's datastore. If the value of + this variable is not specified (which is the default), the following + list is used: :term:`BBPATH`, :term:`BB_PRESERVE_ENV`, + :term:`BB_ENV_WHITELIST`, and :term:`BB_ENV_EXTRAWHITE`. + + .. note:: + + You must set this variable in the external environment in order + for it to work. + + :term:`BB_ENV_EXTRAWHITE` + Specifies an additional set of variables to allow through (whitelist) + from the external environment into BitBake's datastore. This list of + variables are on top of the internal list set in + :term:`BB_ENV_WHITELIST`. + + .. note:: + + You must set this variable in the external environment in order + for it to work. + + :term:`BB_FETCH_PREMIRRORONLY` + When set to "1", causes BitBake's fetcher module to only search + :term:`PREMIRRORS` for files. BitBake will not + search the main :term:`SRC_URI` or + :term:`MIRRORS`. + + :term:`BB_FILENAME` + Contains the filename of the recipe that owns the currently running + task. For example, if the ``do_fetch`` task that resides in the + ``my-recipe.bb`` is executing, the ``BB_FILENAME`` variable contains + "/foo/path/my-recipe.bb". + + :term:`BB_GENERATE_MIRROR_TARBALLS` + Causes tarballs of the Git repositories, including the Git metadata, + to be placed in the :term:`DL_DIR` directory. Anyone + wishing to create a source mirror would want to enable this variable. + + For performance reasons, creating and placing tarballs of the Git + repositories is not the default action by BitBake. :: + + BB_GENERATE_MIRROR_TARBALLS =3D "1" + + :term:`BB_HASHCONFIG_WHITELIST` + Lists variables that are excluded from base configuration checksum, + which is used to determine if the cache can be reused. + + One of the ways BitBake determines whether to re-parse the main + metadata is through checksums of the variables in the datastore of + the base configuration data. There are variables that you typically + want to exclude when checking whether or not to re-parse and thus + rebuild the cache. As an example, you would usually exclude ``TIME`` + and ``DATE`` because these variables are always changing. If you did + not exclude them, BitBake would never reuse the cache. + + :term:`BB_HASHBASE_WHITELIST` + Lists variables that are excluded from checksum and dependency data. + Variables that are excluded can therefore change without affecting + the checksum mechanism. A common example would be the variable for + the path of the build. BitBake's output should not (and usually does + not) depend on the directory in which it was built. + + :term:`BB_HASHCHECK_FUNCTION` + Specifies the name of the function to call during the "setscene" part + of the task's execution in order to validate the list of task hashes. + The function returns the list of setscene tasks that should be + executed. + + At this point in the execution of the code, the objective is to + quickly verify if a given setscene function is likely to work or not. + It's easier to check the list of setscene functions in one pass than + to call many individual tasks. The returned list need not be + completely accurate. A given setscene task can still later fail. + However, the more accurate the data returned, the more efficient the + build will be. + + :term:`BB_INVALIDCONF` + Used in combination with the ``ConfigParsed`` event to trigger + re-parsing the base metadata (i.e. all the recipes). The + ``ConfigParsed`` event can set the variable to trigger the re-parse. + You must be careful to avoid recursive loops with this functionality. + + :term:`BB_LOGCONFIG` + Specifies the name of a config file that contains the user logging + configuration. See + :ref:`bitbake-user-manual/bitbake-user-manual-execution:logging` + for additional information + + :term:`BB_LOGFMT` + Specifies the name of the log files saved into + ``${``\ :term:`T`\ ``}``. By default, the ``BB_LOGFMT`` + variable is undefined and the log file names get created using the + following form: :: + + log.{task}.{pid} + + If you want to force log files to take a specific name, you can set = this + variable in a configuration file. + + :term:`BB_NICE_LEVEL` + Allows BitBake to run at a specific priority (i.e. nice level). + System permissions usually mean that BitBake can reduce its priority + but not raise it again. See :term:`BB_TASK_NICE_LEVEL` for + additional information. + + :term:`BB_NO_NETWORK` + Disables network access in the BitBake fetcher modules. With this + access disabled, any command that attempts to access the network + becomes an error. + + Disabling network access is useful for testing source mirrors, + running builds when not connected to the Internet, and when operating + in certain kinds of firewall environments. + + :term:`BB_NUMBER_THREADS` + The maximum number of tasks BitBake should run in parallel at any one + time. If your host development system supports multiple cores, a good + rule of thumb is to set this variable to twice the number of cores. + + :term:`BB_NUMBER_PARSE_THREADS` + Sets the number of threads BitBake uses when parsing. By default, the + number of threads is equal to the number of cores on the system. + + :term:`BB_ORIGENV` + Contains a copy of the original external environment in which BitBake + was run. The copy is taken before any whitelisted variable values are + filtered into BitBake's datastore. + + .. note:: + + The contents of this variable is a datastore object that can be + queried using the normal datastore operations. + + :term:`BB_PRESERVE_ENV` + Disables whitelisting and instead allows all variables through from + the external environment into BitBake's datastore. + + .. note:: + + You must set this variable in the external environment in order + for it to work. + + :term:`BB_RUNFMT` + Specifies the name of the executable script files (i.e. run files) + saved into ``${``\ :term:`T`\ ``}``. By default, the + ``BB_RUNFMT`` variable is undefined and the run file names get + created using the following form: :: + + run.{task}.{pid} + + If you want to force run files to take a specific name, you can set = this + variable in a configuration file. + + :term:`BB_RUNTASK` + Contains the name of the currently executing task. The value includes + the "do\_" prefix. For example, if the currently executing task is + ``do_config``, the value is "do_config". + + :term:`BB_SCHEDULER` + Selects the name of the scheduler to use for the scheduling of + BitBake tasks. Three options exist: + + - *basic* - The basic framework from which everything derives. Using + this option causes tasks to be ordered numerically as they are + parsed. + + - *speed* - Executes tasks first that have more tasks depending on + them. The "speed" option is the default. + + - *completion* - Causes the scheduler to try to complete a given + recipe once its build has started. + + :term:`BB_SCHEDULERS` + Defines custom schedulers to import. Custom schedulers need to be + derived from the ``RunQueueScheduler`` class. + + For information how to select a scheduler, see the + :term:`BB_SCHEDULER` variable. + + :term:`BB_SETSCENE_DEPVALID` + Specifies a function BitBake calls that determines whether BitBake + requires a setscene dependency to be met. + + When running a setscene task, BitBake needs to know which + dependencies of that setscene task also need to be run. Whether + dependencies also need to be run is highly dependent on the metadata. + The function specified by this variable returns a "True" or "False" + depending on whether the dependency needs to be met. + + :term:`BB_SETSCENE_VERIFY_FUNCTION2` + Specifies a function to call that verifies the list of planned task + execution before the main task execution happens. The function is + called once BitBake has a list of setscene tasks that have run and + either succeeded or failed. + + The function allows for a task list check to see if they make sense. + Even if BitBake was planning to skip a task, the returned value of + the function can force BitBake to run the task, which is necessary + under certain metadata defined circumstances. + + :term:`BB_SIGNATURE_EXCLUDE_FLAGS` + Lists variable flags (varflags) that can be safely excluded from + checksum and dependency data for keys in the datastore. When + generating checksum or dependency data for keys in the datastore, the + flags set against that key are normally included in the checksum. + + For more information on varflags, see the + ":ref:`bitbake-user-manual/bitbake-user-manual-metadata:variable fla= gs`" + section. + + :term:`BB_SIGNATURE_HANDLER` + Defines the name of the signature handler BitBake uses. The signature + handler defines the way stamp files are created and handled, if and + how the signature is incorporated into the stamps, and how the + signature itself is generated. + + A new signature handler can be added by injecting a class derived + from the ``SignatureGenerator`` class into the global namespace. + + :term:`BB_SRCREV_POLICY` + Defines the behavior of the fetcher when it interacts with source + control systems and dynamic source revisions. The + ``BB_SRCREV_POLICY`` variable is useful when working without a + network. + + The variable can be set using one of two policies: + + - *cache* - Retains the value the system obtained previously rather + than querying the source control system each time. + + - *clear* - Queries the source controls system every time. With this + policy, there is no cache. The "clear" policy is the default. + + :term:`BB_STAMP_POLICY` + Defines the mode used for how timestamps of stamp files are compared. + You can set the variable to one of the following modes: + + - *perfile* - Timestamp comparisons are only made between timestamps + of a specific recipe. This is the default mode. + + - *full* - Timestamp comparisons are made for all dependencies. + + - *whitelist* - Identical to "full" mode except timestamp + comparisons are made for recipes listed in the + :term:`BB_STAMP_WHITELIST` variable. + + .. note:: + + Stamp policies are largely obsolete with the introduction of + setscene tasks. + + :term:`BB_STAMP_WHITELIST` + Lists files whose stamp file timestamps are compared when the stamp + policy mode is set to "whitelist". For information on stamp policies, + see the :term:`BB_STAMP_POLICY` variable. + + :term:`BB_STRICT_CHECKSUM` + Sets a more strict checksum mechanism for non-local URLs. Setting + this variable to a value causes BitBake to report an error if it + encounters a non-local URL that does not have at least one checksum + specified. + + :term:`BB_TASK_IONICE_LEVEL` + Allows adjustment of a task's Input/Output priority. During + Autobuilder testing, random failures can occur for tasks due to I/O + starvation. These failures occur during various QEMU runtime + timeouts. You can use the ``BB_TASK_IONICE_LEVEL`` variable to adjust + the I/O priority of these tasks. + + .. note:: + + This variable works similarly to the :term:`BB_TASK_NICE_LEVEL` + variable except with a task's I/O priorities. + + Set the variable as follows: :: + + BB_TASK_IONICE_LEVEL =3D "class.prio" + + For *class*, the default value is "2", which is a best effort. You c= an use + "1" for realtime and "3" for idle. If you want to use realtime, you + must have superuser privileges. + + For *prio*, you can use any value from "0", which is the highest + priority, to "7", which is the lowest. The default value is "4". You + do not need any special privileges to use this range of priority + values. + + .. note:: + + In order for your I/O priority settings to take effect, you need = the + Completely Fair Queuing (CFQ) Scheduler selected for the backing = block + device. To select the scheduler, use the following command form w= here + device is the device (e.g. sda, sdb, and so forth): :: + + $ sudo sh -c "echo cfq > /sys/block/device/queu/scheduler" + + :term:`BB_TASK_NICE_LEVEL` + Allows specific tasks to change their priority (i.e. nice level). + + You can use this variable in combination with task overrides to raise + or lower priorities of specific tasks. For example, on the `Yocto + Project `__ autobuilder, QEMU emulation + in images is given a higher priority as compared to build tasks to + ensure that images do not suffer timeouts on loaded systems. + + :term:`BB_TASKHASH` + Within an executing task, this variable holds the hash of the task as + returned by the currently enabled signature generator. + + :term:`BB_VERBOSE_LOGS` + Controls how verbose BitBake is during builds. If set, shell scripts + echo commands and shell script output appears on standard out + (stdout). + + :term:`BB_WORKERCONTEXT` + Specifies if the current context is executing a task. BitBake sets + this variable to "1" when a task is being executed. The value is not + set when the task is in server context during parsing or event + handling. + + :term:`BBCLASSEXTEND` + Allows you to extend a recipe so that it builds variants of the + software. Some examples of these variants for recipes from the + OpenEmbedded-Core metadata are "natives" such as ``quilt-native``, + which is a copy of Quilt built to run on the build system; "crosses" + such as ``gcc-cross``, which is a compiler built to run on the build + machine but produces binaries that run on the target ``MACHINE``; + "nativesdk", which targets the SDK machine instead of ``MACHINE``; + and "mulitlibs" in the form "``multilib:``\ multilib_name". + + To build a different variant of the recipe with a minimal amount of + code, it usually is as simple as adding the variable to your recipe. + Here are two examples. The "native" variants are from the + OpenEmbedded-Core metadata: :: + + BBCLASSEXTEND =3D+ "native nativesdk" + BBCLASSEXTEND =3D+ "multilib:multilib_name" + + .. note:: + + Internally, the ``BBCLASSEXTEND`` mechanism generates recipe + variants by rewriting variable values and applying overrides such + as ``_class-native``. For example, to generate a native version of + a recipe, a :term:`DEPENDS` on "foo" is + rewritten to a ``DEPENDS`` on "foo-native". + + Even when using ``BBCLASSEXTEND``, the recipe is only parsed once. + Parsing once adds some limitations. For example, it is not + possible to include a different file depending on the variant, + since ``include`` statements are processed when the recipe is + parsed. + + :term:`BBDEBUG` + Sets the BitBake debug output level to a specific value as + incremented by the ``-D`` command line option. + + .. note:: + + You must set this variable in the external environment in order + for it to work. + + :term:`BBFILE_COLLECTIONS` + Lists the names of configured layers. These names are used to find + the other ``BBFILE_*`` variables. Typically, each layer appends its + name to this variable in its ``conf/layer.conf`` file. + + :term:`BBFILE_PATTERN` + Variable that expands to match files from + :term:`BBFILES` in a particular layer. This + variable is used in the ``conf/layer.conf`` file and must be suffixed + with the name of the specific layer (e.g. + ``BBFILE_PATTERN_emenlow``). + + :term:`BBFILE_PRIORITY` + Assigns the priority for recipe files in each layer. + + This variable is useful in situations where the same recipe appears + in more than one layer. Setting this variable allows you to + prioritize a layer against other layers that contain the same recipe + - effectively letting you control the precedence for the multiple + layers. The precedence established through this variable stands + regardless of a recipe's version (:term:`PV` variable). + For example, a layer that has a recipe with a higher ``PV`` value but + for which the ``BBFILE_PRIORITY`` is set to have a lower precedence + still has a lower precedence. + + A larger value for the ``BBFILE_PRIORITY`` variable results in a + higher precedence. For example, the value 6 has a higher precedence + than the value 5. If not specified, the ``BBFILE_PRIORITY`` variable + is set based on layer dependencies (see the ``LAYERDEPENDS`` variable + for more information. The default priority, if unspecified for a + layer with no dependencies, is the lowest defined priority + 1 (or 1 + if no priorities are defined). + + .. tip:: + + You can use the command bitbake-layers show-layers to list all + configured layers along with their priorities. + + :term:`BBFILES` + A space-separated list of recipe files BitBake uses to build + software. + + When specifying recipe files, you can pattern match using Python's + `glob `_ syntax. + For details on the syntax, see the documentation by following the + previous link. + + :term:`BBFILES_DYNAMIC` + Activates content depending on presence of identified layers. You + identify the layers by the collections that the layers define. + + Use the ``BBFILES_DYNAMIC`` variable to avoid ``.bbappend`` files wh= ose + corresponding ``.bb`` file is in a layer that attempts to modify oth= er + layers through ``.bbappend`` but does not want to introduce a hard + dependency on those other layers. + + Additionally you can prefix the rule with "!" to add ``.bbappend`` a= nd + ``.bb`` files in case a layer is not present. Use this avoid hard + dependency on those other layers. + + Use the following form for ``BBFILES_DYNAMIC``: :: + + collection_name:filename_pattern + + The following example identifies two collection names and two filena= me + patterns: :: + + BBFILES_DYNAMIC +=3D "\ + clang-layer:${LAYERDIR}/bbappends/meta-clang/*/*/*.bbappend \ + core:${LAYERDIR}/bbappends/openembedded-core/meta/*/*/*.bbapp= end \ + " + + When the collection name is prefixed with "!" it will add the file p= attern in case + the layer is absent: :: + + BBFILES_DYNAMIC +=3D "\ + !clang-layer:${LAYERDIR}/backfill/meta-clang/*/*/*.bb \ + " + + This next example shows an error message that occurs because invalid + entries are found, which cause parsing to abort: :: + + ERROR: BBFILES_DYNAMIC entries must be of the form {!}:, not: + /work/my-layer/bbappends/meta-security-isafw/*/*/*.bbappend + /work/my-layer/bbappends/openembedded-core/meta/*/*/*.bbappend + + :term:`BBINCLUDED` + Contains a space-separated list of all of all files that BitBake's + parser included during parsing of the current file. + + :term:`BBINCLUDELOGS` + If set to a value, enables printing the task log when reporting a + failed task. + + :term:`BBINCLUDELOGS_LINES` + If :term:`BBINCLUDELOGS` is set, specifies + the maximum number of lines from the task log file to print when + reporting a failed task. If you do not set ``BBINCLUDELOGS_LINES``, + the entire log is printed. + + :term:`BBLAYERS` + Lists the layers to enable during the build. This variable is defined + in the ``bblayers.conf`` configuration file in the build directory. + Here is an example: :: + + BBLAYERS =3D " \ + /home/scottrif/poky/meta \ + /home/scottrif/poky/meta-yocto \ + /home/scottrif/poky/meta-yocto-bsp \ + /home/scottrif/poky/meta-mykernel \ + " + + This example enables four layers, one of which is a custom, user-def= ined + layer named ``meta-mykernel``. + + :term:`BBLAYERS_FETCH_DIR` + Sets the base location where layers are stored. This setting is used + in conjunction with ``bitbake-layers layerindex-fetch`` and tells + ``bitbake-layers`` where to place the fetched layers. + + :term:`BBMASK` + Prevents BitBake from processing recipes and recipe append files. + + You can use the ``BBMASK`` variable to "hide" these ``.bb`` and + ``.bbappend`` files. BitBake ignores any recipe or recipe append + files that match any of the expressions. It is as if BitBake does not + see them at all. Consequently, matching files are not parsed or + otherwise used by BitBake. + + The values you provide are passed to Python's regular expression + compiler. Consequently, the syntax follows Python's Regular + Expression (re) syntax. The expressions are compared against the full + paths to the files. For complete syntax information, see Python's + documentation at http://docs.python.org/3/library/re.html. + + The following example uses a complete regular expression to tell + BitBake to ignore all recipe and recipe append files in the + ``meta-ti/recipes-misc/`` directory: :: + + BBMASK =3D "meta-ti/recipes-misc/" + + If you want to mask out multiple directories or recipes, you can + specify multiple regular expression fragments. This next example + masks out multiple directories and individual recipes: :: + + BBMASK +=3D "/meta-ti/recipes-misc/ meta-ti/recipes-ti/packagegro= up/" + BBMASK +=3D "/meta-oe/recipes-support/" + BBMASK +=3D "/meta-foo/.*/openldap" + BBMASK +=3D "opencv.*\.bbappend" + BBMASK +=3D "lzma" + + .. note:: + + When specifying a directory name, use the trailing slash character + to ensure you match just that directory name. + + :term:`BBMULTICONFIG` + Enables BitBake to perform multiple configuration builds and lists + each separate configuration (multiconfig). You can use this variable + to cause BitBake to build multiple targets where each target has a + separate configuration. Define ``BBMULTICONFIG`` in your + ``conf/local.conf`` configuration file. + + As an example, the following line specifies three multiconfigs, each + having a separate configuration file: :: + + BBMULTIFONFIG =3D "configA configB configC" + + Each configuration file you use must reside in the + build directory within a directory named ``conf/multiconfig`` (e.g. + build_directory\ ``/conf/multiconfig/configA.conf``). + + For information on how to use ``BBMULTICONFIG`` in an environment + that supports building targets with multiple configurations, see the + ":ref:`bitbake-user-manual/bitbake-user-manual-intro:executing a mul= tiple configuration build`" + section. + + :term:`BBPATH` + Used by BitBake to locate class (``.bbclass``) and configuration + (``.conf``) files. This variable is analogous to the ``PATH`` + variable. + + If you run BitBake from a directory outside of the build directory, + you must be sure to set ``BBPATH`` to point to the build directory. + Set the variable as you would any environment variable and then run + BitBake: :: + + $ BBPATH=3D"build_directory" + $ export BBPATH + $ bitbake target + + :term:`BBSERVER` + Points to the server that runs memory-resident BitBake. The variable + is only used when you employ memory-resident BitBake. + + :term:`BBTARGETS` + Allows you to use a configuration file to add to the list of + command-line target recipes you want to build. + + :term:`BBVERSIONS` + Allows a single recipe to build multiple versions of a project from a + single recipe file. You also able to specify conditional metadata + using the :term:`OVERRIDES` mechanism for a + single version or for an optionally named range of versions. + + For more information on ``BBVERSIONS``, see the + ":ref:`bitbake-user-manual/bitbake-user-manual-metadata:variants - c= lass extension mechanism`" + section. + + :term:`BITBAKE_UI` + Used to specify the UI module to use when running BitBake. Using this + variable is equivalent to using the ``-u`` command-line option. + + .. note:: + + You must set this variable in the external environment in order + for it to work. + + :term:`BUILDNAME` + A name assigned to the build. The name defaults to a datetime stamp + of when the build was started but can be defined by the metadata. + + :term:`BZRDIR` + The directory in which files checked out of a Bazaar system are + stored. + + :term:`CACHE` + Specifies the directory BitBake uses to store a cache of the metadata + so it does not need to be parsed every time BitBake is started. + + :term:`CVSDIR` + The directory in which files checked out under the CVS system are + stored. + + :term:`DEFAULT_PREFERENCE` + Specifies a weak bias for recipe selection priority. + + The most common usage of this is variable is to set it to "-1" within + a recipe for a development version of a piece of software. Using the + variable in this way causes the stable version of the recipe to build + by default in the absence of ``PREFERRED_VERSION`` being used to + build the development version. + + .. note:: + + The bias provided by DEFAULT_PREFERENCE is weak and is overridden= by + :term:`BBFILE_PRIORITY` if that variable is different between two + layers that contain different versions of the same recipe. + + :term:`DEPENDS` + Lists a recipe's build-time dependencies (i.e. other recipe files). + + Consider this simple example for two recipes named "a" and "b" that + produce similarly named packages. In this example, the ``DEPENDS`` + statement appears in the "a" recipe: :: + + DEPENDS =3D "b" + + Here, the dependency is such that the ``do_configure`` task for reci= pe "a" + depends on the ``do_populate_sysroot`` task of recipe "b". This means + anything that recipe "b" puts into sysroot is available when recipe = "a" is + configuring itself. + + For information on runtime dependencies, see the :term:`RDEPENDS` + variable. + + :term:`DESCRIPTION` + A long description for the recipe. + + :term:`DL_DIR` + The central download directory used by the build process to store + downloads. By default, ``DL_DIR`` gets files suitable for mirroring = for + everything except Git repositories. If you want tarballs of Git + repositories, use the :term:`BB_GENERATE_MIRROR_TARBALLS` variable. + + :term:`EXCLUDE_FROM_WORLD` + Directs BitBake to exclude a recipe from world builds (i.e. + ``bitbake world``). During world builds, BitBake locates, parses and + builds all recipes found in every layer exposed in the + ``bblayers.conf`` configuration file. + + To exclude a recipe from a world build using this variable, set the + variable to "1" in the recipe. + + .. note:: + + Recipes added to ``EXCLUDE_FROM_WORLD`` may still be built during= a world + build in order to satisfy dependencies of other recipes. Adding a + recipe to ``EXCLUDE_FROM_WORLD`` only ensures that the recipe is = not + explicitly added to the list of build targets in a world build. + + :term:`FAKEROOT` + Contains the command to use when running a shell script in a fakeroot + environment. The ``FAKEROOT`` variable is obsolete and has been + replaced by the other ``FAKEROOT*`` variables. See these entries in + the glossary for more information. + + :term:`FAKEROOTBASEENV` + Lists environment variables to set when executing the command defined + by :term:`FAKEROOTCMD` that starts the + bitbake-worker process in the fakeroot environment. + + :term:`FAKEROOTCMD` + Contains the command that starts the bitbake-worker process in the + fakeroot environment. + + :term:`FAKEROOTDIRS` + Lists directories to create before running a task in the fakeroot + environment. + + :term:`FAKEROOTENV` + Lists environment variables to set when running a task in the + fakeroot environment. For additional information on environment + variables and the fakeroot environment, see the + :term:`FAKEROOTBASEENV` variable. + + :term:`FAKEROOTNOENV` + Lists environment variables to set when running a task that is not in + the fakeroot environment. For additional information on environment + variables and the fakeroot environment, see the + :term:`FAKEROOTENV` variable. + + :term:`FETCHCMD` + Defines the command the BitBake fetcher module executes when running + fetch operations. You need to use an override suffix when you use the + variable (e.g. ``FETCHCMD_git`` or ``FETCHCMD_svn``). + + :term:`FILE` + Points at the current file. BitBake sets this variable during the + parsing process to identify the file being parsed. BitBake also sets + this variable when a recipe is being executed to identify the recipe + file. + + :term:`FILESPATH` + Specifies directories BitBake uses when searching for patches and + files. The "local" fetcher module uses these directories when + handling ``file://`` URLs. The variable behaves like a shell ``PATH`` + environment variable. The value is a colon-separated list of + directories that are searched left-to-right in order. + + :term:`GITDIR` + The directory in which a local copy of a Git repository is stored + when it is cloned. + + :term:`HGDIR` + The directory in which files checked out of a Mercurial system are + stored. + + :term:`HOMEPAGE` + Website where more information about the software the recipe is + building can be found. + + :term:`INHERIT` + Causes the named class or classes to be inherited globally. Anonymous + functions in the class or classes are not executed for the base + configuration and in each individual recipe. The OpenEmbedded build + system ignores changes to ``INHERIT`` in individual recipes. + + For more information on ``INHERIT``, see the + ":ref:`bitbake-user-manual/bitbake-user-manual-metadata:\`\`inherit\= `\` configuration directive`" + section. + + :term:`LAYERDEPENDS` + Lists the layers, separated by spaces, upon which this recipe + depends. Optionally, you can specify a specific layer version for a + dependency by adding it to the end of the layer name with a colon, + (e.g. "anotherlayer:3" to be compared against + :term:`LAYERVERSION`\ ``_anotherlayer`` in + this case). BitBake produces an error if any dependency is missing or + the version numbers do not match exactly (if specified). + + You use this variable in the ``conf/layer.conf`` file. You must also + use the specific layer name as a suffix to the variable (e.g. + ``LAYERDEPENDS_mylayer``). + + :term:`LAYERDIR` + When used inside the ``layer.conf`` configuration file, this variable + provides the path of the current layer. This variable is not + available outside of ``layer.conf`` and references are expanded + immediately when parsing of the file completes. + + :term:`LAYERDIR_RE` + When used inside the ``layer.conf`` configuration file, this variable + provides the path of the current layer, escaped for use in a regular + expression (:term:`BBFILE_PATTERN`). This + variable is not available outside of ``layer.conf`` and references + are expanded immediately when parsing of the file completes. + + :term:`LAYERVERSION` + Optionally specifies the version of a layer as a single number. You + can use this variable within + :term:`LAYERDEPENDS` for another layer in + order to depend on a specific version of the layer. + + You use this variable in the ``conf/layer.conf`` file. You must also + use the specific layer name as a suffix to the variable (e.g. + ``LAYERDEPENDS_mylayer``). + + :term:`LICENSE` + The list of source licenses for the recipe. + + :term:`MIRRORS` + Specifies additional paths from which BitBake gets source code. When + the build system searches for source code, it first tries the local + download directory. If that location fails, the build system tries + locations defined by :term:`PREMIRRORS`, the + upstream source, and then locations specified by ``MIRRORS`` in that + order. + + :term:`MULTI_PROVIDER_WHITELIST` + Allows you to suppress BitBake warnings caused when building two + separate recipes that provide the same output. + + BitBake normally issues a warning when building two different recipes + where each provides the same output. This scenario is usually + something the user does not want. However, cases do exist where it + makes sense, particularly in the ``virtual/*`` namespace. You can use + this variable to suppress BitBake's warnings. + + To use the variable, list provider names (e.g. recipe names, + ``virtual/kernel``, and so forth). + + :term:`OVERRIDES` + BitBake uses ``OVERRIDES`` to control what variables are overridden + after BitBake parses recipes and configuration files. + + Following is a simple example that uses an overrides list based on + machine architectures: OVERRIDES =3D "arm:x86:mips:powerpc" You can + find information on how to use ``OVERRIDES`` in the + ":ref:`bitbake-user-manual/bitbake-user-manual-metadata:conditional = syntax + (overrides)`" section. + + :term:`P4DIR` + The directory in which a local copy of a Perforce depot is stored + when it is fetched. + + :term:`PACKAGES` + The list of packages the recipe creates. + + :term:`PACKAGES_DYNAMIC` + A promise that your recipe satisfies runtime dependencies for + optional modules that are found in other recipes. + ``PACKAGES_DYNAMIC`` does not actually satisfy the dependencies, it + only states that they should be satisfied. For example, if a hard, + runtime dependency (:term:`RDEPENDS`) of another + package is satisfied during the build through the + ``PACKAGES_DYNAMIC`` variable, but a package with the module name is + never actually produced, then the other package will be broken. + + :term:`PE` + The epoch of the recipe. By default, this variable is unset. The + variable is used to make upgrades possible when the versioning scheme + changes in some backwards incompatible way. + + :term:`PERSISTENT_DIR` + Specifies the directory BitBake uses to store data that should be + preserved between builds. In particular, the data stored is the data + that uses BitBake's persistent data API and the data used by the PR + Server and PR Service. + + :term:`PF` + Specifies the recipe or package name and includes all version and + revision numbers (i.e. ``eglibc-2.13-r20+svnr15508/`` and + ``bash-4.2-r1/``). + + :term:`PN` + The recipe name. + + :term:`PR` + The revision of the recipe. + + :term:`PREFERRED_PROVIDER` + Determines which recipe should be given preference when multiple + recipes provide the same item. You should always suffix the variable + with the name of the provided item, and you should set it to the + :term:`PN` of the recipe to which you want to give + precedence. Some examples: :: + + PREFERRED_PROVIDER_virtual/kernel ?=3D "linux-yocto" + PREFERRED_PROVIDER_virtual/xserver =3D "xserver-xf86" + PREFERRED_PROVIDER_virtual/libgl ?=3D "mesa" + + :term:`PREFERRED_PROVIDERS` + Determines which recipe should be given preference for cases where + multiple recipes provide the same item. Functionally, + ``PREFERRED_PROVIDERS`` is identical to + :term:`PREFERRED_PROVIDER`. However, the ``PREFERRED_PROVIDERS`` var= iable + lets you define preferences for multiple situations using the follow= ing + form: :: + + PREFERRED_PROVIDERS =3D "xxx:yyy aaa:bbb ..." + + This form is a convenient replacement for the following: :: + + PREFERRED_PROVIDER_xxx =3D "yyy" + PREFERRED_PROVIDER_aaa =3D "bbb" + + :term:`PREFERRED_VERSION` + If there are multiple versions of a recipe available, this variable + determines which version should be given preference. You must always + suffix the variable with the :term:`PN` you want to + select, and you should set :term:`PV` accordingly for + precedence. + + The ``PREFERRED_VERSION`` variable supports limited wildcard use + through the "``%``" character. You can use the character to match any + number of characters, which can be useful when specifying versions + that contain long revision numbers that potentially change. Here are + two examples: :: + + PREFERRED_VERSION_python =3D "2.7.3" + PREFERRED_VERSION_linux-yocto =3D "4.12%" + + .. important:: + + The use of the " % " character is limited in that it only works a= t the + end of the string. You cannot use the wildcard character in any o= ther + location of the string. + + If a recipe with the specified version is not available, a warning + message will be shown. See :term:`REQUIRED_VERSION` if you want this + to be an error instead. + + :term:`PREMIRRORS` + Specifies additional paths from which BitBake gets source code. When + the build system searches for source code, it first tries the local + download directory. If that location fails, the build system tries + locations defined by ``PREMIRRORS``, the upstream source, and then + locations specified by :term:`MIRRORS` in that order. + + Typically, you would add a specific server for the build system to + attempt before any others by adding something like the following to + your configuration: :: + + PREMIRRORS_prepend =3D "\ + git://.*/.* http://www.yoctoproject.org/sources/ \n \ + ftp://.*/.* http://www.yoctoproject.org/sources/ \n \ + http://.*/.* http://www.yoctoproject.org/sources/ \n \ + https://.*/.* http://www.yoctoproject.org/sources/ \n" + + These changes cause the build system to intercept Git, FTP, HTTP, and + HTTPS requests and direct them to the ``http://`` sources mirror. Yo= u can + use ``file://`` URLs to point to local directories or network shares= as + well. + + :term:`PROVIDES` + A list of aliases by which a particular recipe can be known. By + default, a recipe's own ``PN`` is implicitly already in its + ``PROVIDES`` list. If a recipe uses ``PROVIDES``, the additional + aliases are synonyms for the recipe and can be useful satisfying + dependencies of other recipes during the build as specified by + ``DEPENDS``. + + Consider the following example ``PROVIDES`` statement from a recipe + file ``libav_0.8.11.bb``: :: + + PROVIDES +=3D "libpostproc" + + The ``PROVIDES`` statement results in the "libav" recipe also being = known + as "libpostproc". + + In addition to providing recipes under alternate names, the + ``PROVIDES`` mechanism is also used to implement virtual targets. A + virtual target is a name that corresponds to some particular + functionality (e.g. a Linux kernel). Recipes that provide the + functionality in question list the virtual target in ``PROVIDES``. + Recipes that depend on the functionality in question can include the + virtual target in :term:`DEPENDS` to leave the + choice of provider open. + + Conventionally, virtual targets have names on the form + "virtual/function" (e.g. "virtual/kernel"). The slash is simply part + of the name and has no syntactical significance. + + :term:`PRSERV_HOST` + The network based :term:`PR` service host and port. + + Following is an example of how the ``PRSERV_HOST`` variable is set: = :: + + PRSERV_HOST =3D "localhost:0" + + You must set the variable if you want to automatically start a local= PR + service. You can set ``PRSERV_HOST`` to other values to use a remote= PR + service. + + :term:`PV` + The version of the recipe. + + :term:`RDEPENDS` + Lists a package's runtime dependencies (i.e. other packages) that + must be installed in order for the built package to run correctly. If + a package in this list cannot be found during the build, you will get + a build error. + + Because the ``RDEPENDS`` variable applies to packages being built, + you should always use the variable in a form with an attached package + name. For example, suppose you are building a development package + that depends on the ``perl`` package. In this case, you would use the + following ``RDEPENDS`` statement: :: + + RDEPENDS_${PN}-dev +=3D "perl" + + In the example, the development package depends on the ``perl`` pack= age. + Thus, the ``RDEPENDS`` variable has the ``${PN}-dev`` package name a= s part + of the variable. + + BitBake supports specifying versioned dependencies. Although the + syntax varies depending on the packaging format, BitBake hides these + differences from you. Here is the general syntax to specify versions + with the ``RDEPENDS`` variable: :: + + RDEPENDS_${PN} =3D "package (operator version)" + + For ``operator``, you can specify the following: :: + + =3D + < + > + <=3D + >=3D + + For example, the following sets up a dependency on version 1.2 or + greater of the package ``foo``: :: + + RDEPENDS_${PN} =3D "foo (>=3D 1.2)" + + For information on build-time dependencies, see the :term:`DEPENDS` + variable. + + :term:`REPODIR` + The directory in which a local copy of a ``google-repo`` directory is + stored when it is synced. + + :term:`REQUIRED_VERSION` + If there are multiple versions of a recipe available, this variable + determines which version should be given preference. ``REQUIRED_VERS= ION`` + works in exactly the same manner as :term:`PREFERRED_VERSION`, except + that if the specified version is not available then an error message + is shown and the build fails immediately. + + If both ``REQUIRED_VERSION`` and ``PREFERRED_VERSION`` are set for + the same recipe, the ``REQUIRED_VERSION`` value applies. + + :term:`RPROVIDES` + A list of package name aliases that a package also provides. These + aliases are useful for satisfying runtime dependencies of other + packages both during the build and on the target (as specified by + ``RDEPENDS``). + + As with all package-controlling variables, you must always use the + variable in conjunction with a package name override. Here is an + example: :: + + RPROVIDES_${PN} =3D "widget-abi-2" + + :term:`RRECOMMENDS` + A list of packages that extends the usability of a package being + built. The package being built does not depend on this list of + packages in order to successfully build, but needs them for the + extended usability. To specify runtime dependencies for packages, see + the ``RDEPENDS`` variable. + + BitBake supports specifying versioned recommends. Although the syntax + varies depending on the packaging format, BitBake hides these + differences from you. Here is the general syntax to specify versions + with the ``RRECOMMENDS`` variable: :: + + RRECOMMENDS_${PN} =3D "package (operator version)" + + For ``operator``, you can specify the following: :: + + =3D + < + > + <=3D + >=3D + + For example, the following sets up a recommend on version + 1.2 or greater of the package ``foo``: :: + + RRECOMMENDS_${PN} =3D "foo (>=3D 1.2)" + + :term:`SECTION` + The section in which packages should be categorized. + + :term:`SRC_URI` + The list of source files - local or remote. This variable tells + BitBake which bits to pull for the build and how to pull them. For + example, if the recipe or append file needs to fetch a single tarball + from the Internet, the recipe or append file uses a ``SRC_URI`` entry + that specifies that tarball. On the other hand, if the recipe or + append file needs to fetch a tarball and include a custom file, the + recipe or append file needs an ``SRC_URI`` variable that specifies + all those sources. + + The following list explains the available URI protocols: + + - ``file://`` : Fetches files, which are usually files shipped + with the metadata, from the local machine. The path is relative to + the :term:`FILESPATH` variable. + + - ``bzr://`` : Fetches files from a Bazaar revision control + repository. + + - ``git://`` : Fetches files from a Git revision control + repository. + + - ``osc://`` : Fetches files from an OSC (OpenSUSE Build service) + revision control repository. + + - ``repo://`` : Fetches files from a repo (Git) repository. + + - ``http://`` : Fetches files from the Internet using HTTP. + + - ``https://`` : Fetches files from the Internet using HTTPS. + + - ``ftp://`` : Fetches files from the Internet using FTP. + + - ``cvs://`` : Fetches files from a CVS revision control + repository. + + - ``hg://`` : Fetches files from a Mercurial (``hg``) revision + control repository. + + - ``p4://`` : Fetches files from a Perforce (``p4``) revision + control repository. + + - ``ssh://`` : Fetches files from a secure shell. + + - ``svn://`` : Fetches files from a Subversion (``svn``) revision + control repository. + + - ``az://`` : Fetches files from an Azure Storage account using HTT= PS. + + Here are some additional options worth mentioning: + + - ``unpack`` : Controls whether or not to unpack the file if it is + an archive. The default action is to unpack the file. + + - ``subdir`` : Places the file (or extracts its contents) into the + specified subdirectory. This option is useful for unusual tarballs + or other archives that do not have their files already in a + subdirectory within the archive. + + - ``name`` : Specifies a name to be used for association with + ``SRC_URI`` checksums when you have more than one file specified + in ``SRC_URI``. + + - ``downloadfilename`` : Specifies the filename used when storing + the downloaded file. + + :term:`SRCDATE` + The date of the source code used to build the package. This variable + applies only if the source was fetched from a Source Code Manager + (SCM). + + :term:`SRCREV` + The revision of the source code used to build the package. This + variable applies only when using Subversion, Git, Mercurial and + Bazaar. If you want to build a fixed revision and you want to avoid + performing a query on the remote repository every time BitBake parses + your recipe, you should specify a ``SRCREV`` that is a full revision + identifier and not just a tag. + + :term:`SRCREV_FORMAT` + Helps construct valid :term:`SRCREV` values when + multiple source controlled URLs are used in + :term:`SRC_URI`. + + The system needs help constructing these values under these + circumstances. Each component in the ``SRC_URI`` is assigned a name + and these are referenced in the ``SRCREV_FORMAT`` variable. Consider + an example with URLs named "machine" and "meta". In this case, + ``SRCREV_FORMAT`` could look like "machine_meta" and those names + would have the SCM versions substituted into each position. Only one + ``AUTOINC`` placeholder is added and if needed. And, this placeholder + is placed at the start of the returned string. + + :term:`STAMP` + Specifies the base path used to create recipe stamp files. The path + to an actual stamp file is constructed by evaluating this string and + then appending additional information. + + :term:`STAMPCLEAN` + Specifies the base path used to create recipe stamp files. Unlike the + :term:`STAMP` variable, ``STAMPCLEAN`` can contain + wildcards to match the range of files a clean operation should + remove. BitBake uses a clean operation to remove any other stamps it + should be removing when creating a new stamp. + + :term:`SUMMARY` + A short summary for the recipe, which is 72 characters or less. + + :term:`SVNDIR` + The directory in which files checked out of a Subversion system are + stored. + + :term:`T` + Points to a directory were BitBake places temporary files, which + consist mostly of task logs and scripts, when building a particular + recipe. + + :term:`TOPDIR` + Points to the build directory. BitBake automatically sets this + variable. diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variab= les.xml b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables= .xml deleted file mode 100644 index c4bd1f25..00000000 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml +++ /dev/null @@ -1,2476 +0,0 @@ - %poky; ] > - - - - -Variables Glossary - - - This chapter lists common variables used by BitBake and gives an overv= iew - of their function and contents. - - - - Following are some points regarding the variables listed in this gloss= ary: - - The variables listed in this glossary - are specific to BitBake. - Consequently, the descriptions are limited to that context. - - Also, variables exist in other systems that use Bi= tBake - (e.g. The Yocto Project and OpenEmbedded) that have names iden= tical - to those found in this glossary. - For such cases, the variables in those systems extend the - functionality of the variable as it is described here in - this glossary. - - Finally, there are variables mentioned in this - glossary that do not appear in the BitBake glossary. - These other variables are variables used in systems that use - BitBake. - - - - - - - - A - B - C - D - E - F - G - H - I - - - L - M - - O - P - - R - S - T - - - - - - - - - A - - ASSUME_PROVID= ED - - - Lists recipe names - (PN - values) BitBake does not attempt to build. - Instead, BitBake assumes these recipes have already be= en - built. - - - - In OpenEmbedded-Core, ASSUME_PROVIDED - mostly specifies native tools that should not be built. - An example is git-native, which - when specified allows for the Git binary from the host= to - be used rather than building - git-native. - - - - - - - - B - - B - - - The directory in which BitBake executes functions - during a recipe's build process. - - - - - BB_ALLOWE= D_NETWORKS - - - Specifies a space-delimited list of hosts that the fet= cher - is allowed to use to obtain the required source code. - Following are considerations surrounding this variable: - - - This host list is only used if - BB_NO_NETWORK - is either not set or set to "0". - - - Limited support for the "*" - wildcard character for matching against the - beginning of host names exists. - For example, the following setting matches - git.gnu.org, - ftp.gnu.org, and - foo.git.gnu.org. - - BB_ALLOWED_NETWORKS =3D "*.gnu.org" - - Important - The use of the "*" - character only works at the beginning of - a host name and it must be isolated from - the remainder of the host name. - You cannot use the wildcard character in a= ny - other location of the name or combined with - the front part of the name. - - For example, - *.foo.bar is supporte= d, - while *aa.foo.bar is = not. - - - - - Mirrors not in the host list are skipped and - logged in debug. - - - Attempts to access networks not in the host li= st - cause a failure. - - - Using BB_ALLOWED_NETWORKS in - conjunction with - PREMIRRO= RS - is very useful. - Adding the host you want to use to - PREMIRRORS results in the source = code - being fetched from an allowed location and avoids rais= ing - an error when a host that is not allowed is in a - SRC_URI - statement. - This is because the fetcher does not attempt to use the - host listed in SRC_URI after a - successful fetch from the - PREMIRRORS occurs. - - - - - BB_CONSOLELOG - - - Specifies the path to a log file into which BitBake's = user - interface writes output during the build. - - - - - BB_CURRENTTASK= - - - Contains the name of the currently running task. - The name does not include the - do_ prefix. - - - - - B= B_DANGLINGAPPENDS_WARNONLY - - - Defines how BitBake handles situations where an append - file (.bbappend) has no - corresponding recipe file (.bb). - This condition often occurs when layers get out of sync - (e.g. oe-core bumps a - recipe version and the old recipe no longer exists and= the - other layer has not been updated to the new version - of the recipe yet). - - - - The default fatal behavior is safest because it is - the sane reaction given something is out of sync. - It is important to realize when your changes are no lo= nger - being applied. - - - - - BB_DEFAULT_TA= SK - - - The default task to use when none is specified (e.g. - with the -c command line option). - The task name specified should not include the - do_ prefix. - - - - - BB_DISKMON_DI= RS - - - Monitors disk space and available inodes during the bu= ild - and allows you to control the build based on these - parameters. - - - - Disk space monitoring is disabled by default. - When setting this variable, use the following form: - - BB_DISKMON_DIRS =3D "<action>,<dir>,<threshold> [..= .]" - - where: - - <action> is: - ABORT: Immediately abort the build when - a threshold is broken. - STOPTASKS: Stop the build after the currently - executing tasks have finished when - a threshold is broken. - WARN: Issue a warning but continue the - build when a threshold is broken. - Subsequent warnings are issued as - defined by the - BB_= DISKMON_WARNINTERVAL variable, - which must be defined. - - <dir> is: - Any directory you choose. You can specify one or - more directories to monitor by separating the - groupings with a space. If two directories are - on the same device, only the first directory - is monitored. - - <threshold> is: - Either the minimum available disk space, - the minimum number of free inodes, or - both. You must specify at least one. To - omit one or the other, simply omit the value. - Specify the threshold using G, M, K for Gbytes, - Mbytes, and Kbytes, respectively. If you do - not specify G, M, or K, Kbytes is assumed by - default. Do not use GB, MB, or KB. - - - - - Here are some examples: - - BB_DISKMON_DIRS =3D "ABORT,${TMPDIR},1G,100K WARN,${SSTATE_DIR},1G,10= 0K" - BB_DISKMON_DIRS =3D "STOPTASKS,${TMPDIR},1G" - BB_DISKMON_DIRS =3D "ABORT,${TMPDIR},,100K" - - The first example works only if you also set - the <= filename>BB_DISKMON_WARNINTERVAL variable. - This example causes the build system to immediately - abort when either the disk space in ${TMPDIR= } drops - below 1 Gbyte or the available free inodes drops below - 100 Kbytes. - Because two directories are provided with the variable= , the - build system also issues a - warning when the disk space in the - ${SSTATE_DIR} directory drops - below 1 Gbyte or the number of free inodes drops - below 100 Kbytes. - Subsequent warnings are issued during intervals as - defined by the BB_DISKMON_WARNINTERVAL - variable. - - - - The second example stops the build after all currently - executing tasks complete when the minimum disk space - in the ${TMPDIR} - directory drops below 1 Gbyte. - No disk monitoring occurs for the free inodes in this = case. - - - - The final example immediately aborts the build when the - number of free inodes in the ${TMPDIR} directory - drops below 100 Kbytes. - No disk space monitoring for the directory itself occu= rs - in this case. - - - - - BB_DI= SKMON_WARNINTERVAL - - - Defines the disk space and free inode warning interval= s. - - - - If you are going to use the - BB_DISKMON_WARNINTERVAL variable,= you must - also use the - BB_= DISKMON_DIRS variable - and define its action as "WARN". - During the build, subsequent warnings are issued each = time - disk space or number of free inodes further reduces by - the respective interval. - - - - If you do not provide a BB_DISKMON_WARNINTER= VAL - variable and you do use BB_DISKMON_DIRS with - the "WARN" action, the disk monitoring interval defaul= ts to - the following: - - BB_DISKMON_WARNINTERVAL =3D "50M,5K" - - - - - When specifying the variable in your configuration fil= e, - use the following form: - - BB_DISKMON_WARNINTERVAL =3D "<disk_space_interval>,<disk_ino= de_interval>" - - where: - - <disk_space_interval> is: - An interval of memory expressed in either - G, M, or K for Gbytes, Mbytes, or Kbytes, - respectively. You cannot use GB, MB, or KB. - - <disk_inode_interval> is: - An interval of free inodes expressed in either - G, M, or K for Gbytes, Mbytes, or Kbytes, - respectively. You cannot use GB, MB, or KB. - - - - - Here is an example: - - BB_DISKMON_DIRS =3D "WARN,${SSTATE_DIR},1G,100K" - BB_DISKMON_WARNINTERVAL =3D "50M,5K" - - These variables cause BitBake to - issue subsequent warnings each time the available - disk space further reduces by 50 Mbytes or the number - of free inodes further reduces by 5 Kbytes in the - ${SSTATE_DIR} directory. - Subsequent warnings based on the interval occur each t= ime - a respective interval is reached beyond the initial wa= rning - (i.e. 1 Gbytes and 100 Kbytes). - - - - - BB_ENV_WHITE= LIST - - - Specifies the internal whitelist of variables to allow - through from the external environment into BitBake's - datastore. - If the value of this variable is not specified - (which is the default), the following list is used: - BBPATH, - BB_= PRESERVE_ENV, - BB= _ENV_WHITELIST, - and - B= B_ENV_EXTRAWHITE. - - You must set this variable in the external environ= ment - in order for it to work. - - - - - - BB_ENV_EXTR= AWHITE - - - Specifies an additional set of variables to allow thro= ugh - (whitelist) from the external environment into BitBake= 's - datastore. - This list of variables are on top of the internal list - set in - BB= _ENV_WHITELIST. - - You must set this variable in the external - environment in order for it to work. - - - - - - BB_FET= CH_PREMIRRORONLY - - - When set to "1", causes BitBake's fetcher module to on= ly - search - PREMIRRO= RS - for files. - BitBake will not search the main - SRC_URI - or - MIRRORS. - - - - - BB_FILENAME - - - Contains the filename of the recipe that owns the curr= ently - running task. - For example, if the do_fetch task= that - resides in the my-recipe.bb is - executing, the BB_FILENAME variab= le - contains "/foo/path/my-recipe.bb". - - - - - B= B_GENERATE_MIRROR_TARBALLS - - - Causes tarballs of the Git repositories, including the - Git metadata, to be placed in the - DL_DIR - directory. - Anyone wishing to create a source mirror would want to - enable this variable. - - - - For performance reasons, creating and placing tarballs= of - the Git repositories is not the default action by BitB= ake. - - BB_GENERATE_MIRROR_TARBALLS =3D "1" - - - - - - BB_HA= SHCONFIG_WHITELIST - - - Lists variables that are excluded from base configurat= ion - checksum, which is used to determine if the cache can - be reused. - - - - One of the ways BitBake determines whether to re-parse= the - main metadata is through checksums of the variables in= the - datastore of the base configuration data. - There are variables that you typically want to exclude= when - checking whether or not to re-parse and thus rebuild t= he - cache. - As an example, you would usually exclude - TIME and DATE - because these variables are always changing. - If you did not exclude them, BitBake would never reuse= the - cache. - - - - - BB_HASH= BASE_WHITELIST - - - Lists variables that are excluded from checksum and - dependency data. - Variables that are excluded can therefore change witho= ut - affecting the checksum mechanism. - A common example would be the variable for the path of - the build. - BitBake's output should not (and usually does not) dep= end - on the directory in which it was built. - - - - - BB_HASH= CHECK_FUNCTION - - - Specifies the name of the function to call during the - "setscene" part of the task's execution in order to - validate the list of task hashes. - The function returns the list of setscene tasks that s= hould - be executed. - - - - At this point in the execution of the code, the object= ive - is to quickly verify if a given setscene function is l= ikely - to work or not. - It's easier to check the list of setscene functions in - one pass than to call many individual tasks. - The returned list need not be completely accurate. - A given setscene task can still later fail. - However, the more accurate the data returned, the more - efficient the build will be. - - - - - BB_INVALIDCONF= - - - Used in combination with the - ConfigParsed event to trigger - re-parsing the base metadata (i.e. all the - recipes). - The ConfigParsed event can set the - variable to trigger the re-parse. - You must be careful to avoid recursive loops with this - functionality. - - - - - BB_LOGCONFIG - - - Specifies the name of a config file that contains the = user - logging configuration. See - Logging for additional - information - - - - - BB_LOGFMT - - - Specifies the name of the log files saved into - ${T}. - By default, the BB_LOGFMT variable - is undefined and the log file names get created using = the - following form: - - log.{task}.{pid} - - If you want to force log files to take a specific name, - you can set this variable in a configuration file. - - - - - BB_NICE_LEVEL - - - Allows BitBake to run at a specific priority - (i.e. nice level). - System permissions usually mean that BitBake can reduc= e its - priority but not raise it again. - See - = BB_TASK_NICE_LEVEL - for additional information. - - - - - BB_NO_NETWORK - - - Disables network access in the BitBake fetcher modules. - With this access disabled, any command that attempts to - access the network becomes an error. - - - - Disabling network access is useful for testing source - mirrors, running builds when not connected to the Inte= rnet, - and when operating in certain kinds of firewall - environments. - - - - - BB_NUMBER_T= HREADS - - - The maximum number of tasks BitBake should run in para= llel - at any one time. - If your host development system supports multiple core= s, - a good rule of thumb is to set this variable to twice = the - number of cores. - - - - - BB_NU= MBER_PARSE_THREADS - - - Sets the number of threads BitBake uses when parsing. - By default, the number of threads is equal to the numb= er - of cores on the system. - - - - - BB_ORIGENV - - - Contains a copy of the original external environment in - which BitBake was run. - The copy is taken before any whitelisted variable valu= es - are filtered into BitBake's datastore. - - The contents of this variable is a datastore object - that can be queried using the normal datastore - operations. - - - - - - BB_PRESERVE_E= NV - - - Disables whitelisting and instead allows all variables - through from the external environment into BitBake's - datastore. - - You must set this variable in the external - environment in order for it to work. - - - - - - BB_RUNFMT - - - Specifies the name of the executable script files - (i.e. run files) saved into - ${T}. - By default, the BB_RUNFMT variable - is undefined and the run file names get created using = the - following form: - - run.{task}.{pid} - - If you want to force run files to take a specific name, - you can set this variable in a configuration file. - - - - - BB_RUNTASK - - - Contains the name of the currently executing task. - The value includes the "do_" prefix. - For example, if the currently executing task is - do_config, the value is - "do_config". - - - - - BB_SCHEDULER - - - Selects the name of the scheduler to use for the - scheduling of BitBake tasks. - Three options exist: - - basic - - The basic framework from which everything deri= ves. - Using this option causes tasks to be ordered - numerically as they are parsed. - - speed - - Executes tasks first that have more tasks - depending on them. - The "speed" option is the default. - - completion - - Causes the scheduler to try to complete a given - recipe once its build has started. - - - - - - - BB_SCHEDULERS - - - Defines custom schedulers to import. - Custom schedulers need to be derived from the - RunQueueScheduler class. - - - - For information how to select a scheduler, see the - BB_SCH= EDULER - variable. - - - - - BB_SETSC= ENE_DEPVALID - - - Specifies a function BitBake calls that determines - whether BitBake requires a setscene dependency to be m= et. - - - - When running a setscene task, BitBake needs to - know which dependencies of that setscene task also need - to be run. - Whether dependencies also need to be run is highly - dependent on the metadata. - The function specified by this variable returns a - "True" or "False" depending on whether the dependency = needs - to be met. - - - - - = BB_SETSCENE_VERIFY_FUNCTION2 - - - Specifies a function to call that verifies the list of - planned task execution before the main task execution - happens. - The function is called once BitBake has a list of sets= cene - tasks that have run and either succeeded or failed. - - - - The function allows for a task list check to see if th= ey - make sense. - Even if BitBake was planning to skip a task, the - returned value of the function can force BitBake to run - the task, which is necessary under certain metadata - defined circumstances. - - - - - BB= _SIGNATURE_EXCLUDE_FLAGS - - - Lists variable flags (varflags) - that can be safely excluded from checksum - and dependency data for keys in the datastore. - When generating checksum or dependency data for keys i= n the - datastore, the flags set against that key are normally - included in the checksum. - - - - For more information on varflags, see the - "Variable Flags" - section. - - - - - BB_SIGNA= TURE_HANDLER - - - Defines the name of the signature handler BitBake uses. - The signature handler defines the way stamp files are - created and handled, if and how the signature is - incorporated into the stamps, and how the signature - itself is generated. - - - - A new signature handler can be added by injecting a cl= ass - derived from the - SignatureGenerator class into the - global namespace. - - - - - BB_SRCREV_PO= LICY - - - Defines the behavior of the fetcher when it interacts = with - source control systems and dynamic source revisions. - The BB_SRCREV_POLICY variable is - useful when working without a network. - - - - The variable can be set using one of two policies: - - cache - - Retains the value the system obtained previous= ly - rather than querying the source control system - each time. - - clear - - Queries the source controls system every time. - With this policy, there is no cache. - The "clear" policy is the default. - - - - - - - BB_STAMP_POLI= CY - - - Defines the mode used for how timestamps of stamp files - are compared. - You can set the variable to one of the following modes: - - perfile - - Timestamp comparisons are only made - between timestamps of a specific recipe. - This is the default mode. - - full - - Timestamp comparisons are made for all - dependencies. - - whitelist - - Identical to "full" mode except timestamp - comparisons are made for recipes listed in the - BB_STAMP_WHITELIST - variable. - - - - Stamp policies are largely obsolete with the - introduction of setscene tasks. - - - - - - BB_STAMP_W= HITELIST - - - Lists files whose stamp file timestamps are compared w= hen - the stamp policy mode is set to "whitelist". - For information on stamp policies, see the - BB_= STAMP_POLICY - variable. - - - - - BB_STRICT_= CHECKSUM - - - Sets a more strict checksum mechanism for non-local UR= Ls. - Setting this variable to a value causes BitBake - to report an error if it encounters a non-local URL - that does not have at least one checksum specified. - - - - - BB_TASK_= IONICE_LEVEL - - - Allows adjustment of a task's Input/Output priority. - During Autobuilder testing, random failures can occur - for tasks due to I/O starvation. - These failures occur during various QEMU runtime timeo= uts. - You can use the BB_TASK_IONICE_LEVEL - variable to adjust the I/O priority of these tasks. - - This variable works similarly to the - BB_TASK_NICE_LEVEL - variable except with a task's I/O priorities. - - - - - Set the variable as follows: - - BB_TASK_IONICE_LEVEL =3D "class.prio" - - For class, the default valu= e is - "2", which is a best effort. - You can use "1" for realtime and "3" for idle. - If you want to use realtime, you must have superuser - privileges. - - - - For prio, you can use any - value from "0", which is the highest priority, to "7", - which is the lowest. - The default value is "4". - You do not need any special privileges to use this ran= ge - of priority values. - - In order for your I/O priority settings to take ef= fect, - you need the Completely Fair Queuing (CFQ) Schedul= er - selected for the backing block device. - To select the scheduler, use the following command= form - where device is the dev= ice - (e.g. sda, sdb, and so forth): - - $ sudo sh -c =E2=80=9Cecho cfq > /sys/block/device/queu/scheduler - - - - - - - BB_TASK_NI= CE_LEVEL - - - Allows specific tasks to change their priority - (i.e. nice level). - - - - You can use this variable in combination with task - overrides to raise or lower priorities of specific tas= ks. - For example, on the - Yocto Proje= ct - autobuilder, QEMU emulation in images is given a higher - priority as compared to build tasks to ensure that ima= ges - do not suffer timeouts on loaded systems. - - - - - BB_TASKHASH - - - Within an executing task, this variable holds the hash - of the task as returned by the currently enabled - signature generator. - - - - - BB_VERBOSE_LO= GS - - - Controls how verbose BitBake is during builds. - If set, shell scripts echo commands and shell script o= utput - appears on standard out (stdout). - - - - - BB_WORKERCON= TEXT - - - Specifies if the current context is executing a task. - BitBake sets this variable to "1" when a task is - being executed. - The value is not set when the task is in server context - during parsing or event handling. - - - - - - BBCLASSEXTEND - - - Allows you to extend a recipe so that it builds varian= ts - of the software. - Some examples of these variants for recipes from the - OpenEmbedded-Core metadata are "natives" such as - quilt-native, which is a copy of - Quilt built to run on the build system; "crosses" such - as gcc-cross, which is a compiler - built to run on the build machine but produces binaries - that run on the target MACHINE; - "nativesdk", which targets the SDK machine instead of - MACHINE; and "mulitlibs" in the f= orm - "multilib:multilib_n= ame". - - - - To build a different variant of the recipe with a mini= mal - amount of code, it usually is as simple as adding the - variable to your recipe. - Here are two examples. - The "native" variants are from the OpenEmbedded-Core - metadata: - - BBCLASSEXTEND =3D+ "native nativesdk" - BBCLASSEXTEND =3D+ "multilib:multilib_name" - - - - Internally, the BBCLASSEXTEND - mechanism generates recipe variants by rewriting - variable values and applying overrides such as - _class-native. - For example, to generate a native version of a rec= ipe, - a - DEPENDS= - on "foo" is rewritten to a DEPENDS - on "foo-native". - - - - Even when using BBCLASSEXTEND= , the - recipe is only parsed once. - Parsing once adds some limitations. - For example, it is not possible to - include a different file depending on the variant, - since include statements are - processed when the recipe is parsed. - - - - - - - BBDEBUG - - - Sets the BitBake debug output level to a specific value - as incremented by the -D command = line - option. - - You must set this variable in the external environ= ment - in order for it to work. - - - - - - BBFILE_COL= LECTIONS - - Lists the names of configured layers. - These names are used to find the other BBFIL= E_* - variables. - Typically, each layer appends its name to this variabl= e in its - conf/layer.conf file. - - - - - BBFILE_PATTERN= - - Variable that expands to match files from - BBFILES - in a particular layer. - This variable is used in the conf/layer.conf= file and must - be suffixed with the name of the specific layer (e.g. - BBFILE_PATTERN_emenlow). - - - - BBFILE_PRIORI= TY - - Assigns the priority for recipe files in each layer.= - This variable is useful in situations where the same= recipe appears in - more than one layer. - Setting this variable allows you to prioritize a - layer against other layers that contain the same recip= e - effectively - letting you control the precedence for the multiple la= yers. - The precedence established through this variable stand= s regardless of a - recipe's version - (PV variable). - For example, a layer that has a recipe with a higher <= filename>PV value but for - which the BBFILE_PRIORITY is set = to have a lower precedence still has a - lower precedence. - A larger value for the BBFILE_PRIORITY variable results in a higher - precedence. - For example, the value 6 has a higher precedence than = the value 5. - If not specified, the BBFILE_PRIORITY variable is set based on layer - dependencies (see the - LAYERD= EPENDS variable for - more information. - The default priority, if unspecified - for a layer with no dependencies, is the lowest define= d priority + 1 - (or 1 if no priorities are defined). - - You can use the command bitbake-layers show-= layers to list - all configured layers along with their priorities. - - - - - BBFILES - - - A space-separated list of recipe files BitBake uses to - build software. - - - - When specifying recipe files, you can pattern match us= ing - Python's - glob - syntax. - For details on the syntax, see the documentation by - following the previous link. - - - - - BBINCLUDED - - - Contains a space-separated list of all of all files th= at - BitBake's parser included during parsing of the current - file. - - - - - BBINCLUDELOGS - - - If set to a value, enables printing the task log when - reporting a failed task. - - - - - BBINCLUDE= LOGS_LINES - - - If - BBINC= LUDELOGS - is set, specifies the maximum number of lines from the - task log file to print when reporting a failed task. - If you do not set BBINCLUDELOGS_LINES, - the entire log is printed. - - - - - BBLAYERS - - Lists the layers to enable during the build. - This variable is defined in the bblayers.con= f configuration - file in the build directory. - Here is an example: - - BBLAYERS =3D " \ - /home/scottrif/poky/meta \ - /home/scottrif/poky/meta-yocto \ - /home/scottrif/poky/meta-yocto-bsp \ - /home/scottrif/poky/meta-mykernel \ - " - - - This example enables four layers, one of which is a cu= stom, user-defined layer - named meta-mykernel. - - - - - BBLAYERS_F= ETCH_DIR - - - Sets the base location where layers are stored. - This setting is used in conjunction with - bitbake-layers layerindex-fetch a= nd - tells bitbake-layers where to pla= ce - the fetched layers. - - - - - BBMASK - - - Prevents BitBake from processing recipes and recipe - append files. - - - - You can use the BBMASK variable - to "hide" these .bb and - .bbappend files. - BitBake ignores any recipe or recipe append files that - match any of the expressions. - It is as if BitBake does not see them at all. - Consequently, matching files are not parsed or otherwi= se - used by BitBake. - - - - The values you provide are passed to Python's regular - expression compiler. - Consequently, the syntax follows Python's Regular - Expression (re) syntax. - The expressions are compared against the full paths to - the files. - For complete syntax information, see Python's - documentation at - . - - - - The following example uses a complete regular expressi= on - to tell BitBake to ignore all recipe and recipe append - files in the meta-ti/recipes-misc/ - directory: - - BBMASK =3D "meta-ti/recipes-misc/" - - If you want to mask out multiple directories or recipe= s, - you can specify multiple regular expression fragments. - This next example masks out multiple directories and - individual recipes: - - BBMASK +=3D "/meta-ti/recipes-misc/ meta-ti/recipes-ti/packagegroup/" - BBMASK +=3D "/meta-oe/recipes-support/" - BBMASK +=3D "/meta-foo/.*/openldap" - BBMASK +=3D "opencv.*\.bbappend" - BBMASK +=3D "lzma" - - - When specifying a directory name, use the trailing - slash character to ensure you match just that dire= ctory - name. - - - - - - BBMULTICONFIG - - BBMULTICONFIG[doc] =3D "Enables BitBake to perform multipl= e configuration builds and lists each separate configuration (multiconfig)." - - - - - Enables BitBake to perform multiple configuration buil= ds - and lists each separate configuration (multiconfig). - You can use this variable to cause BitBake to build - multiple targets where each target has a separate - configuration. - Define BBMULTICONFIG in your - conf/local.conf configuration fil= e. - - - - As an example, the following line specifies three - multiconfigs, each having a separate configuration fil= e: - - BBMULTIFONFIG =3D "configA configB configC" - - Each configuration file you use must reside in the - build directory within a directory named - conf/multiconfig (e.g. - build_directory/c= onf/multiconfig/configA.conf). - - - - For information on how to use - BBMULTICONFIG in an environment t= hat - supports building targets with multiple configurations, - see the - "Executing a Multiple Configuration Build" - section. - - - - - BBPATH - - - Used by BitBake to locate class - (.bbclass) and configuration - (.conf) files. - This variable is analogous to the - PATH variable. - - - - If you run BitBake from a directory outside of the - build directory, - you must be sure to set - BBPATH to point to the - build directory. - Set the variable as you would any environment variable - and then run BitBake: - - $ BBPATH=3D"build_directory" - $ export BBPATH - $ bitbake target - - - - - - BBSERVER - - - Points to the server that runs memory-resident BitBake. - The variable is only used when you employ memory-resid= ent - BitBake. - - - - - BBTARGETS - - - Allows you to use a configuration file to add to the l= ist - of command-line target recipes you want to build. - - - - - BBVERSIONS - - - Allows a single recipe to build multiple versions of a - project from a single recipe file. - You also able to specify conditional metadata - using the - OVERRIDES= - mechanism for a single version or for an optionally na= med - range of versions. - - - - For more information on BBVERSIONS, - see the - "= Variants - Class Extension Mechanism" - section. - - - - - BITBAKE_UI - - - Used to specify the UI module to use when running BitB= ake. - Using this variable is equivalent to using the - -u command-line option. - - You must set this variable in the external environ= ment - in order for it to work. - - - - - - BUILDNAME - - - A name assigned to the build. - The name defaults to a datetime stamp of when the buil= d was - started but can be defined by the metadata. - - - - - BZRDIR - - - The directory in which files checked out of a Bazaar - system are stored. - - - - - - - C - - CACHE - - - Specifies the directory BitBake uses to store a cache - of the metadata so it does not need to be parsed every - time BitBake is started. - - - - - CVSDIR - - - The directory in which files checked out under the - CVS system are stored. - - - - - - - D - - DEFAULT_PR= EFERENCE - - - Specifies a weak bias for recipe selection priority. - - - The most common usage of this is variable is to set - it to "-1" within a recipe for a development version o= f a - piece of software. - Using the variable in this way causes the stable versi= on - of the recipe to build by default in the absence of - P= REFERRED_VERSION - being used to build the development version. - - - The bias provided by DEFAULT_PREFERENCE - is weak and is overridden by - BBF= ILE_PRIORITY - if that variable is different between two layers - that contain different versions of the same recipe. - - - - - DEPENDS - - - Lists a recipe's build-time dependencies - (i.e. other recipe files). - - - - Consider this simple example for two recipes named "a"= and - "b" that produce similarly named packages. - In this example, the DEPENDS - statement appears in the "a" recipe: - - DEPENDS =3D "b" - - Here, the dependency is such that the - do_configure task for recipe "a" - depends on the do_populate_sysroot - task of recipe "b". - This means anything that recipe "b" puts into sysroot - is available when recipe "a" is configuring itself. - - - - For information on runtime dependencies, see the - RDEPENDS - variable. - - - - - DESCRIPTION - - - A long description for the recipe. - - - - - DL_DIR - - - The central download directory used by the build proce= ss to - store downloads. - By default, DL_DIR gets files - suitable for mirroring for everything except Git - repositories. - If you want tarballs of Git repositories, use the - <= filename>BB_GENERATE_MIRROR_TARBALLS - variable. - - - - - - - E - - EXCLUDE_FR= OM_WORLD - - - Directs BitBake to exclude a recipe from world builds = (i.e. - bitbake world). - During world builds, BitBake locates, parses and build= s all - recipes found in every layer exposed in the - bblayers.conf configuration file. - - - - To exclude a recipe from a world build using this vari= able, - set the variable to "1" in the recipe. - - - - Recipes added to EXCLUDE_FROM_WORLD - may still be built during a world build in order to sa= tisfy - dependencies of other recipes. - Adding a recipe to EXCLUDE_FROM_WORLD - only ensures that the recipe is not explicitly added - to the list of build targets in a world build. - - - - - - - F - - FAKEROOT - - - Contains the command to use when running a shell scri= pt - in a fakeroot environment. - The FAKEROOT variable is obsolete - and has been replaced by the other - FAKEROOT* variables. - See these entries in the glossary for more informatio= n. - - - - - FAKEROOTBASEE= NV - - - Lists environment variables to set when executing - the command defined by - FAKERO= OTCMD - that starts the bitbake-worker process - in the fakeroot environment. - - - - - FAKEROOTCMD - - - Contains the command that starts the bitbake-worker - process in the fakeroot environment. - - - - - FAKEROOTDIRS - - - Lists directories to create before running a task in - the fakeroot environment. - - - - - FAKEROOTENV - - - Lists environment variables to set when running a task - in the fakeroot environment. - For additional information on environment variables a= nd - the fakeroot environment, see the - FA= KEROOTBASEENV - variable. - - - - - FAKEROOTNOENV - - - Lists environment variables to set when running a task - that is not in the fakeroot environment. - For additional information on environment variables a= nd - the fakeroot environment, see the - FAKERO= OTENV - variable. - - - - - FETCHCMD - - - Defines the command the BitBake fetcher module - executes when running fetch operations. - You need to use an override suffix when you use the - variable (e.g. FETCHCMD_git - or FETCHCMD_svn). - - - - - FILE - - - Points at the current file. - BitBake sets this variable during the parsing process - to identify the file being parsed. - BitBake also sets this variable when a recipe is being - executed to identify the recipe file. - - - - - FILESPATH - - - Specifies directories BitBake uses when searching for - patches and files. - The "local" fetcher module uses these directories when - handling file:// URLs. - The variable behaves like a shell PATH - environment variable. - The value is a colon-separated list of directories that - are searched left-to-right in order. - - - - - - - - G - - GITDIR - - - The directory in which a local copy of a Git repository - is stored when it is cloned. - - - - - - - - H - - HGDIR - - - The directory in which files checked out of a Mercurial - system are stored. - - - - - HOMEPAGE - - Website where more information about the software th= e recipe is building - can be found. - - - - - - I - - INHERIT - - - Causes the named class or classes to be inherited glob= ally. - Anonymous functions in the class or classes - are not executed for the - base configuration and in each individual recipe. - The OpenEmbedded build system ignores changes to - INHERIT in individual recipes. - - - - For more information on INHERIT, = see - the - "INHERIT Configuration Directive" - section. - - - - - - - - - L - - LAYERDEPENDS - - Lists the layers, separated by spaces, upon which th= is recipe depends. - Optionally, you can specify a specific layer version f= or a dependency - by adding it to the end of the layer name with a colon= , (e.g. "anotherlayer:3" - to be compared against - LAYERV= ERSION_anotherlayer - in this case). - BitBake produces an error if any dependency is missing= or - the version numbers do not match exactly (if specified= ). - - You use this variable in the conf/layer.conf= file. - You must also use the specific layer name as a suffix - to the variable (e.g. LAYERDEPENDS_mylayer). - - - - LAYERDIR - - When used inside the layer.conf= configuration - file, this variable provides the path of the current l= ayer. - This variable is not available outside of la= yer.conf - and references are expanded immediately when parsing o= f the file completes. - - - - LAYERDIR_RE - - When used inside the layer.conf= configuration - file, this variable provides the path of the current l= ayer, - escaped for use in a regular expression - (BBF= ILE_PATTERN). - This variable is not available outside of la= yer.conf - and references are expanded immediately when parsing o= f the file completes. - - - - LAYERVERSION - - Optionally specifies the version of a layer as a sin= gle number. - You can use this variable within - LAYERD= EPENDS - for another layer in order to depend on a specific ver= sion - of the layer. - - You use this variable in the conf/layer.conf= file. - You must also use the specific layer name as a suffix - to the variable (e.g. LAYERDEPENDS_mylayer). - - - - LICENSE - - - The list of source licenses for the recipe. - - - - - - - M - - MIRRORS - - - Specifies additional paths from which BitBake gets sou= rce code. - When the build system searches for source code, it fir= st - tries the local download directory. - If that location fails, the build system tries locatio= ns - defined by - PREMIRRO= RS, - the upstream source, and then locations specified by - MIRRORS in that order. - - - - - MULT= I_PROVIDER_WHITELIST - - - Allows you to suppress BitBake warnings caused when - building two separate recipes that provide the same - output. - - - - BitBake normally issues a warning when building two - different recipes where each provides the same output. - This scenario is usually something the user does not - want. - However, cases do exist where it makes sense, particul= arly - in the virtual/* namespace. - You can use this variable to suppress BitBake's warnin= gs. - - - - To use the variable, list provider names (e.g. - recipe names, virtual/kernel, - and so forth). - - - - - - - - - O - - OVERRIDES - - - BitBake uses OVERRIDES to control - what variables are overridden after BitBake parses - recipes and configuration files. - - - - Following is a simple example that uses an overrides - list based on machine architectures: - - OVERRIDES =3D "arm:x86:mips:powerpc" - - You can find information on how to use - OVERRIDES in the - "Condit= ional Syntax (Overrides)" - section. - - - - - - P - - P4DIR - - - The directory in which a local copy of a Perforce depot - is stored when it is fetched. - - - - - PACKAGES - - The list of packages the recipe creates. - - - - - PACKAGES_DYN= AMIC - - - A promise that your recipe satisfies runtime dependenc= ies - for optional modules that are found in other recipes. - PACKAGES_DYNAMIC - does not actually satisfy the dependencies, it only st= ates that - they should be satisfied. - For example, if a hard, runtime dependency - (RDEPENDS<= /filename>) - of another package is satisfied during the build - through the PACKAGES_DYNAMIC - variable, but a package with the module name is never = actually - produced, then the other package will be broken. - - - - - PE - - - The epoch of the recipe. - By default, this variable is unset. - The variable is used to make upgrades possible when the - versioning scheme changes in some backwards incompatib= le - way. - - - - - PERSISTENT_DIR= - - - Specifies the directory BitBake uses to store data that - should be preserved between builds. - In particular, the data stored is the data that uses - BitBake's persistent data API and the data used by the - PR Server and PR Service. - - - - - PF - - - Specifies the recipe or package name and includes all = version and revision - numbers (i.e. eglibc-2.13-r20+svnr15508/ and - bash-4.2-r1/). - - - - - PN - - The recipe name. - - - - PR - - The revision of the recipe. - - - - - PREFERRED_= PROVIDER - - - Determines which recipe should be given preference when - multiple recipes provide the same item. - You should always suffix the variable with the name of= the - provided item, and you should set it to the - PN - of the recipe to which you want to give precedence. - Some examples: - - PREFERRED_PROVIDER_virtual/kernel ?=3D "linux-yocto" - PREFERRED_PROVIDER_virtual/xserver =3D "xserver-xf86" - PREFERRED_PROVIDER_virtual/libgl ?=3D "mesa" - - - - - - PREFERRED= _PROVIDERS - - - Determines which recipe should be given preference for - cases where multiple recipes provide the same item. - Functionally, - PREFERRED_PROVIDERS is identical = to - = PREFERRED_PROVIDER. - However, the PREFERRED_PROVIDERS - variable lets you define preferences for multiple - situations using the following form: - - PREFERRED_PROVIDERS =3D "xxx:yyy aaa:bbb ..." - - This form is a convenient replacement for the followin= g: - - PREFERRED_PROVIDER_xxx =3D "yyy" - PREFERRED_PROVIDER_aaa =3D "bbb" - - - - - - PREFERRED_V= ERSION - - - If there are multiple versions of recipes available, t= his - variable determines which recipe should be given prefe= rence. - You must always suffix the variable with the - PN - you want to select, and you should set - PV - accordingly for precedence. - - - - The PREFERRED_VERSION variable - supports limited wildcard use through the - "%" character. - You can use the character to match any number of - characters, which can be useful when specifying versio= ns - that contain long revision numbers that potentially ch= ange. - Here are two examples: - - PREFERRED_VERSION_python =3D "2.7.3" - PREFERRED_VERSION_linux-yocto =3D "4.12%" - - Important - The use of the "%" character - is limited in that it only works at the end of the - string. - You cannot use the wildcard character in any other - location of the string. - - - - - - PREMIRRORS - - - Specifies additional paths from which BitBake gets sou= rce code. - When the build system searches for source code, it fir= st - tries the local download directory. - If that location fails, the build system tries locatio= ns - defined by PREMIRRORS, the upstre= am - source, and then locations specified by - MIRRORS - in that order. - - - - Typically, you would add a specific server for the - build system to attempt before any others by adding - something like the following to your configuration: - - PREMIRRORS_prepend =3D "\ - git://.*/.* http://www.yoctoproject.org/sources/ \n \ - ftp://.*/.* http://www.yoctoproject.org/sources/ \n \ - http://.*/.* http://www.yoctoproject.org/sources/ \n \ - https://.*/.* http://www.yoctoproject.org/sources/ \n" - - These changes cause the build system to intercept - Git, FTP, HTTP, and HTTPS requests and direct them to - the http:// sources mirror. - You can use file:// URLs to point - to local directories or network shares as well. - - - - - PROVIDES - - - A list of aliases by which a particular recipe can be - known. - By default, a recipe's own - PN - is implicitly already in its PROVIDES - list. - If a recipe uses PROVIDES, the - additional aliases are synonyms for the recipe and can - be useful satisfying dependencies of other recipes dur= ing - the build as specified by - DEPENDS. - - - - Consider the following example - PROVIDES statement from a recipe - file libav_0.8.11.bb: - - PROVIDES +=3D "libpostproc" - - The PROVIDES statement results in - the "libav" recipe also being known as "libpostproc". - - - - In addition to providing recipes under alternate names, - the PROVIDES mechanism is also us= ed - to implement virtual targets. - A virtual target is a name that corresponds to some - particular functionality (e.g. a Linux kernel). - Recipes that provide the functionality in question lis= t the - virtual target in PROVIDES. - Recipes that depend on the functionality in question c= an - include the virtual target in - DEPENDS - to leave the choice of provider open. - - - - Conventionally, virtual targets have names on the form - "virtual/function" (e.g. "virtual/kernel"). - The slash is simply part of the name and has no - syntactical significance. - - - - - PRSERV_HOST - - - The network based - PR - service host and port. - - - - Following is an example of how the PRSERV_HO= ST variable is - set: - - PRSERV_HOST =3D "localhost:0" - - You must set the variable if you want to automatically - start a local PR service. - You can set PRSERV_HOST to other - values to use a remote PR service. - - - - - PV - - The version of the recipe. - - - - - - - - - R - - RDEPENDS - - - Lists a package's runtime dependencies (i.e. other pac= kages) - that must be installed in order for the built package = to run - correctly. - If a package in this list cannot be found during the b= uild, - you will get a build error. - - - - Because the RDEPENDS variable app= lies - to packages being built, you should always use the var= iable - in a form with an attached package name. - For example, suppose you are building a development pa= ckage - that depends on the perl package. - In this case, you would use the following - RDEPENDS statement: - - RDEPENDS_${PN}-dev +=3D "perl" - - In the example, the development package depends on - the perl package. - Thus, the RDEPENDS variable has t= he - ${PN}-dev package name as part of= the - variable. - - - - BitBake supports specifying versioned dependencies. - Although the syntax varies depending on the packaging - format, BitBake hides these differences from you. - Here is the general syntax to specify versions with - the RDEPENDS variable: - - RDEPENDS_${PN} =3D "package (= operator version)" - - For operator, you can specify the - following: - - =3D - < - > - <=3D - >=3D - - For example, the following sets up a dependency on ver= sion - 1.2 or greater of the package foo: - - RDEPENDS_${PN} =3D "foo (>=3D 1.2)" - - - - - For information on build-time dependencies, see the - DEPENDS - variable. - - - - - REPODIR - - - The directory in which a local copy of a - google-repo directory is stored - when it is synced. - - - - - RPROVIDES - - - A list of package name aliases that a package also pro= vides. - These aliases are useful for satisfying runtime depend= encies - of other packages both during the build and on the tar= get - (as specified by - RDEPENDS). - - - As with all package-controlling variables, you must alw= ays - use the variable in conjunction with a package name ove= rride. - Here is an example: - - RPROVIDES_${PN} =3D "widget-abi-2" - - - - - - RRECOMMENDS - - - A list of packages that extends the usability of a pac= kage - being built. - The package being built does not depend on this list of - packages in order to successfully build, but needs the= m for - the extended usability. - To specify runtime dependencies for packages, see the - RDEPENDS - variable. - - - - BitBake supports specifying versioned recommends. - Although the syntax varies depending on the packaging - format, BitBake hides these differences from you. - Here is the general syntax to specify versions with - the RRECOMMENDS variable: - - RRECOMMENDS_${PN} =3D "package (operator version)" - - For operator, you can specify the - following: - - =3D - < - > - <=3D - >=3D - - For example, the following sets up a recommend on vers= ion - 1.2 or greater of the package foo: - - RRECOMMENDS_${PN} =3D "foo (>=3D 1.2)" - - - - - - - - S - - SECTION - - The section in which packages should be categorized.= - - - - SRC_URI - - - The list of source files - local or remote. - This variable tells BitBake which bits - to pull for the build and how to pull them. - For example, if the recipe or append file needs to - fetch a single tarball from the Internet, the recipe or - append file uses a SRC_URI - entry that specifies that tarball. - On the other hand, if the recipe or append file needs = to - fetch a tarball and include a custom file, the recipe = or - append file needs an SRC_URI vari= able - that specifies all those sources. - The following list explains the available URI protoc= ols: - - file:// - - Fetches files, which are usually files shipped= with - the metadata, - from the local machine. - The path is relative to the - F= ILESPATH - variable. - bzr:// - Fetches files from a - Bazaar revision control repository. - git:// - Fetches files from a - Git revision control repository. - osc:// - Fetches files from - an OSC (OpenSUSE Build service) revision contr= ol repository. - repo:// - Fetches files from - a repo (Git) repository. - http:// - Fetches files from - the Internet using HTTP. - https:// - Fetches files - from the Internet using HTTPS. - ftp:// - Fetches files - from the Internet using FTP. - cvs:// - Fetches files from - a CVS revision control repository. - hg:// - Fetches files from - a Mercurial (hg) revision= control repository. - p4:// - Fetches files from - a Perforce (p4) revision = control repository. - ssh:// - Fetches files from - a secure shell. - svn:// - Fetches files from - a Subversion (svn) revisi= on control repository. - - - Here are some additional options worth mentioning: - - unpack - Controls - whether or not to unpack the file if it is an = archive. - The default action is to unpack the file. - subdir - Places the file - (or extracts its contents) into the specified - subdirectory. - This option is useful for unusual tarballs or = other archives that - do not have their files already in a subdirect= ory within the archive. - - name - Specifies a - name to be used for association with SRC_URI checksums - when you have more than one file specified in = SRC_URI. - - downloadfilena= me - Specifies - the filename used when storing the downloaded = file. - - - - - - SRCDATE - - - The date of the source code used to build the package. - This variable applies only if the source was fetched f= rom a Source Code Manager (SCM). - - - - - SRCREV - - - The revision of the source code used to build the pack= age. - This variable applies only when using Subversion, Git,= Mercurial and Bazaar. - If you want to build a fixed revision and you want - to avoid performing a query on the remote repository e= very time - BitBake parses your recipe, you should specify a SRCREV that is a - full revision identifier and not just a tag. - - - - - SRCREV_FORMAT - - - Helps construct valid - SRCREV - values when multiple source controlled URLs are used in - SRC_URI. - - - - The system needs help constructing these values under = these - circumstances. - Each component in the SRC_URI - is assigned a name and these are referenced - in the SRCREV_FORMAT variable. - Consider an example with URLs named "machine" and "met= a". - In this case, SRCREV_FORMAT could= look - like "machine_meta" and those names would have the SCM - versions substituted into each position. - Only one AUTOINC placeholder is a= dded - and if needed. - And, this placeholder is placed at the start of the - returned string. - - - - - STAMP - - - Specifies the base path used to create recipe stamp fi= les. - The path to an actual stamp file is constructed by eva= luating this - string and then appending additional information. - - - - - STAMPCLEAN - - - Specifies the base path used to create recipe stamp fi= les. - Unlike the - STAMP - variable, STAMPCLEAN can contain - wildcards to match the range of files a clean operation - should remove. - BitBake uses a clean operation to remove any other sta= mps - it should be removing when creating a new stamp. - - - - - SUMMARY - - - A short summary for the recipe, which is 72 characters= or less. - - - - - SVNDIR - - - The directory in which files checked out of a Subversi= on - system are stored. - - - - - - - T - - T - - Points to a directory were BitBake places - temporary files, which consist mostly of task logs and - scripts, when building a particular recipe. - - - - - TOPDIR - - - Points to the build directory. - BitBake automatically sets this variable. - - - - - - - - - - - - diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-style.css = b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-style.css deleted file mode 100644 index 65da2a4e..00000000 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-style.css +++ /dev/null @@ -1,984 +0,0 @@ -/* - Generic XHTML / DocBook XHTML CSS Stylesheet. - - Browser wrangling and typographic design by - Oyvind Kolas / pippin@gimp.org - - Customised for Poky by - Matthew Allum / mallum@o-hand.com - - Thanks to: - Liam R. E. Quin - William Skaggs - Jakub Steiner - - Structure - --------- - - The stylesheet is divided into the following sections: - - Positioning - Margins, paddings, width, font-size, clearing. - Decorations - Borders, style - Colors - Colors - Graphics - Graphical backgrounds - Nasty IE tweaks - Workarounds needed to make it work in internet explorer, - currently makes the stylesheet non validating, but up until - this point it is validating. - Mozilla extensions - Transparency for footer - Rounded corners on boxes - -*/ - - - /*************** / - / Positioning / -/ ***************/ - -body { - font-family: Verdana, Sans, sans-serif; - - min-width: 640px; - width: 80%; - margin: 0em auto; - padding: 2em 5em 5em 5em; - color: #333; -} - -h1,h2,h3,h4,h5,h6,h7 { - font-family: Arial, Sans; - color: #00557D; - clear: both; -} - -h1 { - font-size: 2em; - text-align: left; - padding: 0em 0em 0em 0em; - margin: 2em 0em 0em 0em; -} - -h2.subtitle { - margin: 0.10em 0em 3.0em 0em; - padding: 0em 0em 0em 0em; - font-size: 1.8em; - padding-left: 20%; - font-weight: normal; - font-style: italic; -} - -h2 { - margin: 2em 0em 0.66em 0em; - padding: 0.5em 0em 0em 0em; - font-size: 1.5em; - font-weight: bold; -} - -h3.subtitle { - margin: 0em 0em 1em 0em; - padding: 0em 0em 0em 0em; - font-size: 142.14%; - text-align: right; -} - -h3 { - margin: 1em 0em 0.5em 0em; - padding: 1em 0em 0em 0em; - font-size: 140%; - font-weight: bold; -} - -h4 { - margin: 1em 0em 0.5em 0em; - padding: 1em 0em 0em 0em; - font-size: 120%; - font-weight: bold; -} - -h5 { - margin: 1em 0em 0.5em 0em; - padding: 1em 0em 0em 0em; - font-size: 110%; - font-weight: bold; -} - -h6 { - margin: 1em 0em 0em 0em; - padding: 1em 0em 0em 0em; - font-size: 110%; - font-weight: bold; -} - -.authorgroup { - background-color: transparent; - background-repeat: no-repeat; - padding-top: 256px; - background-image: url("figures/bitbake-title.png"); - background-position: left top; - margin-top: -256px; - padding-right: 50px; - margin-left: 0px; - text-align: right; - width: 740px; -} - -h3.author { - margin: 0em 0me 0em 0em; - padding: 0em 0em 0em 0em; - font-weight: normal; - font-size: 100%; - color: #333; - clear: both; -} - -.author tt.email { - font-size: 66%; -} - -.titlepage hr { - width: 0em; - clear: both; -} - -.revhistory { - padding-top: 2em; - clear: both; -} - -.toc, -.list-of-tables, -.list-of-examples, -.list-of-figures { - padding: 1.33em 0em 2.5em 0em; - color: #00557D; -} - -.toc p, -.list-of-tables p, -.list-of-figures p, -.list-of-examples p { - padding: 0em 0em 0em 0em; - padding: 0em 0em 0.3em; - margin: 1.5em 0em 0em 0em; -} - -.toc p b, -.list-of-tables p b, -.list-of-figures p b, -.list-of-examples p b{ - font-size: 100.0%; - font-weight: bold; -} - -.toc dl, -.list-of-tables dl, -.list-of-figures dl, -.list-of-examples dl { - margin: 0em 0em 0.5em 0em; - padding: 0em 0em 0em 0em; -} - -.toc dt { - margin: 0em 0em 0em 0em; - padding: 0em 0em 0em 0em; -} - -.toc dd { - margin: 0em 0em 0em 2.6em; - padding: 0em 0em 0em 0em; -} - -div.glossary dl, -div.variablelist dl { -} - -.glossary dl dt, -.variablelist dl dt, -.variablelist dl dt span.term { - font-weight: normal; - width: 20em; - text-align: right; -} - -.variablelist dl dt { - margin-top: 0.5em; -} - -.glossary dl dd, -.variablelist dl dd { - margin-top: -1em; - margin-left: 25.5em; -} - -.glossary dd p, -.variablelist dd p { - margin-top: 0em; - margin-bottom: 1em; -} - - -div.calloutlist table td { - padding: 0em 0em 0em 0em; - margin: 0em 0em 0em 0em; -} - -div.calloutlist table td p { - margin-top: 0em; - margin-bottom: 1em; -} - -div p.copyright { - text-align: left; -} - -div.legalnotice p.legalnotice-title { - margin-bottom: 0em; -} - -p { - line-height: 1.5em; - margin-top: 0em; - -} - -dl { - padding-top: 0em; -} - -hr { - border: solid 1px; -} - - -.mediaobject, -.mediaobjectco { - text-align: center; -} - -img { - border: none; -} - -ul { - padding: 0em 0em 0em 1.5em; -} - -ul li { - padding: 0em 0em 0em 0em; -} - -ul li p { - text-align: left; -} - -table { - width :100%; -} - -th { - padding: 0.25em; - text-align: left; - font-weight: normal; - vertical-align: top; -} - -td { - padding: 0.25em; - vertical-align: top; -} - -p a[id] { - margin: 0px; - padding: 0px; - display: inline; - background-image: none; -} - -a { - text-decoration: underline; - color: #444; -} - -pre { - overflow: auto; -} - -a:hover { - text-decoration: underline; - /*font-weight: bold;*/ -} - -/* This style defines how the permalink character - appears by itself and when hovered over with - the mouse. */ - -[alt=3D'Permalink'] { color: #eee; } -[alt=3D'Permalink']:hover { color: black; } - - -div.informalfigure, -div.informalexample, -div.informaltable, -div.figure, -div.table, -div.example { - margin: 1em 0em; - padding: 1em; - page-break-inside: avoid; -} - - -div.informalfigure p.title b, -div.informalexample p.title b, -div.informaltable p.title b, -div.figure p.title b, -div.example p.title b, -div.table p.title b{ - padding-top: 0em; - margin-top: 0em; - font-size: 100%; - font-weight: normal; -} - -.mediaobject .caption, -.mediaobject .caption p { - text-align: center; - font-size: 80%; - padding-top: 0.5em; - padding-bottom: 0.5em; -} - -.epigraph { - padding-left: 55%; - margin-bottom: 1em; -} - -.epigraph p { - text-align: left; -} - -.epigraph .quote { - font-style: italic; -} -.epigraph .attribution { - font-style: normal; - text-align: right; -} - -span.application { - font-style: italic; -} - -.programlisting { - font-family: monospace; - font-size: 80%; - white-space: pre; - margin: 1.33em 0em; - padding: 1.33em; -} - -.tip, -.warning, -.caution, -.note { - margin-top: 1em; - margin-bottom: 1em; - -} - -/* force full width of table within div */ -.tip table, -.warning table, -.caution table, -.note table { - border: none; - width: 100%; -} - - -.tip table th, -.warning table th, -.caution table th, -.note table th { - padding: 0.8em 0.0em 0.0em 0.0em; - margin : 0em 0em 0em 0em; -} - -.tip p, -.warning p, -.caution p, -.note p { - margin-top: 0.5em; - margin-bottom: 0.5em; - padding-right: 1em; - text-align: left; -} - -.acronym { - text-transform: uppercase; -} - -b.keycap, -.keycap { - padding: 0.09em 0.3em; - margin: 0em; -} - -.itemizedlist li { - clear: none; -} - -.filename { - font-size: medium; - font-family: Courier, monospace; -} - - -div.navheader, div.heading{ - position: absolute; - left: 0em; - top: 0em; - width: 100%; - background-color: #cdf; - width: 100%; -} - -div.navfooter, div.footing{ - position: fixed; - left: 0em; - bottom: 0em; - background-color: #eee; - width: 100%; -} - - -div.navheader td, -div.navfooter td { - font-size: 66%; -} - -div.navheader table th { - /*font-family: Georgia, Times, serif;*/ - /*font-size: x-large;*/ - font-size: 80%; -} - -div.navheader table { - border-left: 0em; - border-right: 0em; - border-top: 0em; - width: 100%; -} - -div.navfooter table { - border-left: 0em; - border-right: 0em; - border-bottom: 0em; - width: 100%; -} - -div.navheader table td a, -div.navfooter table td a { - color: #777; - text-decoration: none; -} - -/* normal text in the footer */ -div.navfooter table td { - color: black; -} - -div.navheader table td a:visited, -div.navfooter table td a:visited { - color: #444; -} - - -/* links in header and footer */ -div.navheader table td a:hover, -div.navfooter table td a:hover { - text-decoration: underline; - background-color: transparent; - color: #33a; -} - -div.navheader hr, -div.navfooter hr { - display: none; -} - - -.qandaset tr.question td p { - margin: 0em 0em 1em 0em; - padding: 0em 0em 0em 0em; -} - -.qandaset tr.answer td p { - margin: 0em 0em 1em 0em; - padding: 0em 0em 0em 0em; -} -.answer td { - padding-bottom: 1.5em; -} - -.emphasis { - font-weight: bold; -} - - - /************* / - / decorations / -/ *************/ - -.titlepage { -} - -.part .title { -} - -.subtitle { - border: none; -} - -/* -h1 { - border: none; -} - -h2 { - border-top: solid 0.2em; - border-bottom: solid 0.06em; -} - -h3 { - border-top: 0em; - border-bottom: solid 0.06em; -} - -h4 { - border: 0em; - border-bottom: solid 0.06em; -} - -h5 { - border: 0em; -} -*/ - -.programlisting { - border: solid 1px; -} - -div.figure, -div.table, -div.informalfigure, -div.informaltable, -div.informalexample, -div.example { - border: 1px solid; -} - - - -.tip, -.warning, -.caution, -.note { - border: 1px solid; -} - -.tip table th, -.warning table th, -.caution table th, -.note table th { - border-bottom: 1px solid; -} - -.question td { - border-top: 1px solid black; -} - -.answer { -} - - -b.keycap, -.keycap { - border: 1px solid; -} - - -div.navheader, div.heading{ - border-bottom: 1px solid; -} - - -div.navfooter, div.footing{ - border-top: 1px solid; -} - - /********* / - / colors / -/ *********/ - -body { - color: #333; - background: white; -} - -a { - background: transparent; -} - -a:hover { - background-color: #dedede; -} - - -h1, -h2, -h3, -h4, -h5, -h6, -h7, -h8 { - background-color: transparent; -} - -hr { - border-color: #aaa; -} - - -.tip, .warning, .caution, .note { - border-color: #fff; -} - - -.tip table th, -.warning table th, -.caution table th, -.note table th { - border-bottom-color: #fff; -} - - -.warning { - background-color: #f0f0f2; -} - -.caution { - background-color: #f0f0f2; -} - -.tip { - background-color: #f0f0f2; -} - -.note { - background-color: #f0f0f2; -} - -.glossary dl dt, -.variablelist dl dt, -.variablelist dl dt span.term { - color: #044; -} - -div.figure, -div.table, -div.example, -div.informalfigure, -div.informaltable, -div.informalexample { - border-color: #aaa; -} - -pre.programlisting { - color: black; - background-color: #fff; - border-color: #aaa; - border-width: 2px; -} - -.guimenu, -.guilabel, -.guimenuitem { - background-color: #eee; -} - - -b.keycap, -.keycap { - background-color: #eee; - border-color: #999; -} - - -div.navheader { - border-color: black; -} - - -div.navfooter { - border-color: black; -} - - - /*********** / - / graphics / -/ ***********/ - -/* -body { - background-image: url("images/body_bg.jpg"); - background-attachment: fixed; -} - -.navheader, -.note, -.tip { - background-image: url("images/note_bg.jpg"); - background-attachment: fixed; -} - -.warning, -.caution { - background-image: url("images/warning_bg.jpg"); - background-attachment: fixed; -} - -.figure, -.informalfigure, -.example, -.informalexample, -.table, -.informaltable { - background-image: url("images/figure_bg.jpg"); - background-attachment: fixed; -} - -*/ -h1, -h2, -h3, -h4, -h5, -h6, -h7{ -} - -/* -Example of how to stick an image as part of the title. - -div.article .titlepage .title -{ - background-image: url("figures/white-on-black.png"); - background-position: center; - background-repeat: repeat-x; -} -*/ - -div.preface .titlepage .title, -div.colophon .title, -div.chapter .titlepage .title, -div.article .titlepage .title -{ -} - -div.section div.section .titlepage .title, -div.sect2 .titlepage .title { - background: none; -} - - -h1.title { - background-color: transparent; - background-repeat: no-repeat; - height: 256px; - text-indent: -9000px; - overflow:hidden; -} - -h2.subtitle { - background-color: transparent; - text-indent: -9000px; - overflow:hidden; - width: 0px; - display: none; -} - - /*************************************** / - / pippin.gimp.org specific alterations / -/ ***************************************/ - -/* -div.heading, div.navheader { - color: #777; - font-size: 80%; - padding: 0; - margin: 0; - text-align: left; - position: absolute; - top: 0px; - left: 0px; - width: 100%; - height: 50px; - background: url('/gfx/heading_bg.png') transparent; - background-repeat: repeat-x; - background-attachment: fixed; - border: none; -} - -div.heading a { - color: #444; -} - -div.footing, div.navfooter { - border: none; - color: #ddd; - font-size: 80%; - text-align:right; - - width: 100%; - padding-top: 10px; - position: absolute; - bottom: 0px; - left: 0px; - - background: url('/gfx/footing_bg.png') transparent; -} -*/ - - - - /****************** / - / nasty ie tweaks / -/ ******************/ - -/* -div.heading, div.navheader { - width:expression(document.body.clientWidth + "px"); -} - -div.footing, div.navfooter { - width:expression(document.body.clientWidth + "px"); - margin-left:expression("-5em"); -} -body { - padding:expression("4em 5em 0em 5em"); -} -*/ - - /**************************************** / - / mozilla vendor specific css extensions / -/ ****************************************/ -/* -div.navfooter, div.footing{ - -moz-opacity: 0.8em; -} - -div.figure, -div.table, -div.informalfigure, -div.informaltable, -div.informalexample, -div.example, -.tip, -.warning, -.caution, -.note { - -moz-border-radius: 0.5em; -} - -b.keycap, -.keycap { - -moz-border-radius: 0.3em; -} -*/ - -table tr td table tr td { - display: none; -} - - -hr { - display: none; -} - -table { - border: 0em; -} - - .photo { - float: right; - margin-left: 1.5em; - margin-bottom: 1.5em; - margin-top: 0em; - max-width: 17em; - border: 1px solid gray; - padding: 3px; - background: white; -} - .seperator { - padding-top: 2em; - clear: both; - } - - #validators { - margin-top: 5em; - text-align: right; - color: #777; - } - @media print { - body { - font-size: 8pt; - } - .noprint { - display: none; - } - } - - -.tip, -.note { - background: #f0f0f2; - color: #333; - padding: 20px; - margin: 20px; -} - -.tip h3, -.note h3 { - padding: 0em; - margin: 0em; - font-size: 2em; - font-weight: bold; - color: #333; -} - -.tip a, -.note a { - color: #333; - text-decoration: underline; -} - -.footnote { - font-size: small; - color: #333; -} - -/* Changes the announcement text */ -.tip h3, -.warning h3, -.caution h3, -.note h3 { - font-size:large; - color: #00557D; -} diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual.xml b/bitb= ake/doc/bitbake-user-manual/bitbake-user-manual.xml deleted file mode 100644 index d793265c..00000000 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual.xml +++ /dev/null @@ -1,88 +0,0 @@ - - - - - - - - - - - - - BitBake User Manual - - - - - Richard Purdie, Chris Larson, and = Phil Blundell - - BitBake Community - - bitbake-devel@lists.openembedded.org - - - - - - - 2004-2018 - Richard Purdie - Chris Larson - and Phil Blundell - - - - - This work is licensed under the Creative Commons Attributi= on License. - To view a copy of this license, visit - http://creativecommons.org/licenses/by/2.5/ - or send a letter to Creative Commons, 444 Castro Street, - Suite 900, Mountain View, California 94041, USA. - - - - - - - - - - - - - - - - - diff --git a/bitbake/doc/bitbake-user-manual/html.css b/bitbake/doc/bitbake= -user-manual/html.css deleted file mode 100644 index 6eedfd31..00000000 --- a/bitbake/doc/bitbake-user-manual/html.css +++ /dev/null @@ -1,281 +0,0 @@ -/* Feuille de style DocBook du projet Traduc.org */ -/* DocBook CSS stylesheet of the Traduc.org project */ - -/* (c) Jean-Philippe Gu=E9rard - 14 ao=FBt 2004 */ -/* (c) Jean-Philippe Gu=E9rard - 14 August 2004 */ - -/* Cette feuille de style est libre, vous pouvez la */ -/* redistribuer et la modifier selon les termes de la Licence */ -/* Art Libre. Vous trouverez un exemplaire de cette Licence sur */ -/* http://tigreraye.org/Petit-guide-du-traducteur.html#licence-art-libre */ - -/* This work of art is free, you can redistribute it and/or */ -/* modify it according to terms of the Free Art license. You */ -/* will find a specimen of this license on the Copyleft */ -/* Attitude web site: http://artlibre.org as well as on other */ -/* sites. */ -/* Please note that the French version of this licence as shown */ -/* on http://tigreraye.org/Petit-guide-du-traducteur.html#licence-art-libr= e */ -/* is only official licence of this document. The English */ -/* is only provided to help you understand this licence. */ - -/* La derni=E8re version de cette feuille de style est toujours */ -/* disponible sur=A0: http://tigreraye.org/style.css */ -/* Elle est =E9galement disponible sur=A0: */ -/* http://www.traduc.org/docs/HOWTO/lecture/style.css */ - -/* The latest version of this stylesheet is available from: */ -/* http://tigreraye.org/style.css */ -/* It is also available on: */ -/* http://www.traduc.org/docs/HOWTO/lecture/style.css */ - -/* N'h=E9sitez pas =E0 envoyer vos commentaires et corrections =E0 */ -/* Jean-Philippe Gu=E9rard */ - -/* Please send feedback and bug reports to */ -/* Jean-Philippe Gu=E9rard */ - -/* $Id: style.css,v 1.14 2004/09/10 20:12:09 fevrier Exp fevrier $ */ - -/* Pr=E9sentation g=E9n=E9rale du document */ -/* Overall document presentation */ - -body { - /* - font-family: Apolline, "URW Palladio L", Garamond, jGaramond, - "Bitstream Cyberbit", "Palatino Linotype", serif; - */ - margin: 7%; - background-color: white; -} - -/* Taille du texte */ -/* Text size */ - -* { font-size: 100%; } - -/* Gestion des textes mis en relief imbriqu=E9s */ -/* Embedded emphasis */ - -em { font-style: italic; } -em em { font-style: normal; } -em em em { font-style: italic; } - -/* Titres */ -/* Titles */ - -h1 { font-size: 200%; font-weight: 900; } -h2 { font-size: 160%; font-weight: 900; } -h3 { font-size: 130%; font-weight: bold; } -h4 { font-size: 115%; font-weight: bold; } -h5 { font-size: 108%; font-weight: bold; } -h6 { font-weight: bold; } - -/* Nom de famille en petites majuscules (uniquement en fran=E7ais) */ -/* Last names in small caps (for French only) */ - -*[class~=3D"surname"]:lang(fr) { font-variant: small-caps; } - -/* Blocs de citation */ -/* Quotation blocs */ - -div[class~=3D"blockquote"] { - border: solid 2px #AAA; - padding: 5px; - margin: 5px; -} - -div[class~=3D"blockquote"] > table { - border: none; -} - -/* Blocs lit=E9raux=A0: fond gris clair */ -/* Literal blocs: light gray background */ - -*[class~=3D"literallayout"] { - background: #f0f0f0; - padding: 5px; - margin: 5px; -} - -/* Programmes et captures texte=A0: fond bleu clair */ -/* Listing and text screen snapshots: light blue background */ - -*[class~=3D"programlisting"], *[class~=3D"screen"] { - background: #f0f0ff; - padding: 5px; - margin: 5px; -} - -/* Les textes =E0 remplacer sont surlign=E9s en vert p=E2le */ -/* Replaceable text in highlighted in pale green */ - -*[class~=3D"replaceable"] {=20 - background-color: #98fb98; - font-style: normal; } - -/* Tables=A0: fonds gris clair & bords simples */ -/* Tables: light gray background and solid borders */ - -*[class~=3D"table"] *[class~=3D"title"] { width:100%; border: 0px; } - -table { - border: 1px solid #aaa; - border-collapse: collapse; - padding: 2px; - margin: 5px; -} - -/* Listes simples en style table */ -/* Simples lists in table presentation */ - -table[class~=3D"simplelist"] { - background-color: #F0F0F0; - margin: 5px; - border: solid 1px #AAA; -} - -table[class~=3D"simplelist"] td { - border: solid 1px #AAA; -} - -/* Les tables */ -/* Tables */ - -*[class~=3D"table"] table { - background-color: #F0F0F0; - border: solid 1px #AAA; -} -*[class~=3D"informaltable"] table { background-color: #F0F0F0; } - -th,td { - vertical-align: baseline; - text-align: left; - padding: 0.1em 0.3em; - empty-cells: show;=20 -} - -/* Alignement des colonnes */ -/* Colunms alignment */ - -td[align=3Dcenter] , th[align=3Dcenter] { text-align: center; } -td[align=3Dright] , th[align=3Dright] { text-align: right; } -td[align=3Dleft] , th[align=3Dleft] { text-align: left; } -td[align=3Djustify] , th[align=3Djustify] { text-align: justify; } - -/* Pas de marge autour des images */ -/* No inside margins for images */ - -img { border: 0; } - -/* Les liens ne sont pas soulign=E9s */ -/* No underlines for links */ - -:link , :visited , :active { text-decoration: none; } - -/* Prudence=A0: cadre jaune et fond jaune clair */ -/* Caution: yellow border and light yellow background */ - -*[class~=3D"caution"] { - border: solid 2px yellow; - background-color: #ffffe0; - padding: 1em 6px 1em ; - margin: 5px; -} - -*[class~=3D"caution"] th { - vertical-align: middle -} - -*[class~=3D"caution"] table { - background-color: #ffffe0; - border: none; -} - -/* Note importante=A0: cadre jaune et fond jaune clair */ -/* Important: yellow border and light yellow background */ - -*[class~=3D"important"] { - border: solid 2px yellow; - background-color: #ffffe0; - padding: 1em 6px 1em; - margin: 5px; -} - -*[class~=3D"important"] th { - vertical-align: middle -} - -*[class~=3D"important"] table { - background-color: #ffffe0; - border: none; -} - -/* Mise en =E9vidence=A0: texte l=E9g=E8rement plus grand */ -/* Highlights: slightly larger texts */ - -*[class~=3D"highlights"] { - font-size: 110%; -} - -/* Note=A0: cadre bleu et fond bleu clair */ -/* Notes: blue border and light blue background */ - -*[class~=3D"note"] { - border: solid 2px #7099C5; - background-color: #f0f0ff; - padding: 1em 6px 1em ; - margin: 5px; -} - -*[class~=3D"note"] th { - vertical-align: middle -} - -*[class~=3D"note"] table { - background-color: #f0f0ff; - border: none; -} - -/* Astuce=A0: cadre vert et fond vert clair */ -/* Tip: green border and light green background */ - -*[class~=3D"tip"] { - border: solid 2px #00ff00; - background-color: #f0ffff; - padding: 1em 6px 1em ; - margin: 5px; -} - -*[class~=3D"tip"] th { - vertical-align: middle; -} - -*[class~=3D"tip"] table { - background-color: #f0ffff; - border: none; -} - -/* Avertissement=A0: cadre rouge et fond rouge clair */ -/* Warning: red border and light red background */ - -*[class~=3D"warning"] { - border: solid 2px #ff0000; - background-color: #fff0f0;=20 - padding: 1em 6px 1em ; - margin: 5px; -} - -*[class~=3D"warning"] th { - vertical-align: middle; -} -=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 - -*[class~=3D"warning"] table { - background-color: #fff0f0; - border: none; -} - -/* Fin */ -/* The End */ - diff --git a/bitbake/doc/conf.py b/bitbake/doc/conf.py new file mode 100644 index 00000000..fc2ee081 --- /dev/null +++ b/bitbake/doc/conf.py @@ -0,0 +1,101 @@ +# Configuration file for the Sphinx documentation builder. +# +# This file only contains a selection of the most common options. For a fu= ll +# list see the documentation: +# https://www.sphinx-doc.org/en/master/usage/configuration.html + +# -- Path setup ----------------------------------------------------------= ---- + +# If extensions (or modules to document with autodoc) are in another direc= tory, +# add these directories to sys.path here. If the directory is relative to = the +# documentation root, use os.path.abspath to make it absolute, like shown = here. +# +# import os +# import sys +# sys.path.insert(0, os.path.abspath('.')) + +import sys +import datetime + +current_version =3D "dev" + +# String used in sidebar +version =3D 'Version: ' + current_version +if current_version =3D=3D 'dev': + version =3D 'Version: Current Development' +# Version seen in documentation_options.js and hence in js switchers code +release =3D current_version + +# -- Project information -------------------------------------------------= ---- + +project =3D 'Bitbake' +copyright =3D '2004-%s, Richard Purdie, Chris Larson, and Phil Blundell' \ + % datetime.datetime.now().year +author =3D 'Richard Purdie, Chris Larson, and Phil Blundell' + +# external links and substitutions +extlinks =3D { + 'yocto_docs': ('https://docs.yoctoproject.org%s', None), + 'oe_lists': ('https://lists.openembedded.org%s', None), +} + +# -- General configuration -----------------------------------------------= ---- + +# Add any Sphinx extension module names here, as strings. They can be +# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom +# ones. +extensions =3D [ + 'sphinx.ext.autosectionlabel', + 'sphinx.ext.extlinks', +] +autosectionlabel_prefix_document =3D True + +# Add any paths that contain templates here, relative to this directory. +templates_path =3D ['_templates'] + +# List of patterns, relative to source directory, that match files and +# directories to ignore when looking for source files. +# This pattern also affects html_static_path and html_extra_path. +exclude_patterns =3D ['_build', 'Thumbs.db', '.DS_Store'] + +# master document name. The default changed from contents to index. so bet= ter +# set it ourselves. +master_doc =3D 'index' + +# create substitution for project configuration variables +rst_prolog =3D """ +.. |project_name| replace:: %s +.. |copyright| replace:: %s +.. |author| replace:: %s +""" % (project, copyright, author) + +# -- Options for HTML output ---------------------------------------------= ---- + +# The theme to use for HTML and HTML Help pages. See the documentation for +# a list of builtin themes. +# +try: + import sphinx_rtd_theme + html_theme =3D 'sphinx_rtd_theme' +except ImportError: + sys.stderr.write("The Sphinx sphinx_rtd_theme HTML theme was not found= .\ + \nPlease make sure to install the sphinx_rtd_theme python package.\n") + sys.exit(1) + +# Add any paths that contain custom static files (such as style sheets) he= re, +# relative to this directory. They are copied after the builtin static fil= es, +# so a file named "default.css" will overwrite the builtin "default.css". +html_static_path =3D ['sphinx-static'] + +# Add customm CSS and JS files +html_css_files =3D ['theme_overrides.css'] +html_js_files =3D ['switchers.js'] + +# Hide 'Created using Sphinx' text +html_show_sphinx =3D False + +# Add 'Last updated' on each page +html_last_updated_fmt =3D '%b %d, %Y' + +# Remove the trailing 'dot' in section numbers +html_secnumber_suffix =3D " " diff --git a/bitbake/doc/genindex.rst b/bitbake/doc/genindex.rst new file mode 100644 index 00000000..a4af06f6 --- /dev/null +++ b/bitbake/doc/genindex.rst @@ -0,0 +1,3 @@ +=3D=3D=3D=3D=3D +Index +=3D=3D=3D=3D=3D diff --git a/bitbake/doc/index.rst b/bitbake/doc/index.rst new file mode 100644 index 00000000..3ff8b158 --- /dev/null +++ b/bitbake/doc/index.rst @@ -0,0 +1,38 @@ +.. SPDX-License-Identifier: CC-BY-2.5 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +BitBake User Manual +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +| + +.. toctree:: + :caption: Table of Contents + :numbered: + + bitbake-user-manual/bitbake-user-manual-intro + bitbake-user-manual/bitbake-user-manual-execution + bitbake-user-manual/bitbake-user-manual-metadata + bitbake-user-manual/bitbake-user-manual-fetching + bitbake-user-manual/bitbake-user-manual-ref-variables + bitbake-user-manual/bitbake-user-manual-hello + +.. toctree:: + :maxdepth: 1 + :hidden: + + genindex + releases + +---- + +.. include:: + +| BitBake Community +| Copyright |copy| |copyright| +| + +This work is licensed under the Creative Commons Attribution License. To = view a +copy of this license, visit http://creativecommons.org/licenses/by/2.5/ or= send +a letter to Creative Commons, 444 Castro Street, Suite 900, Mountain View, +California 94041, USA. diff --git a/bitbake/doc/poky.ent b/bitbake/doc/poky.ent deleted file mode 100644 index 85d9c83b..00000000 --- a/bitbake/doc/poky.ent +++ /dev/null @@ -1,51 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/bitbake/doc/releases.rst b/bitbake/doc/releases.rst new file mode 100644 index 00000000..d68d7159 --- /dev/null +++ b/bitbake/doc/releases.rst @@ -0,0 +1,130 @@ +.. SPDX-License-Identifier: CC-BY-2.5 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + Current Release Manuals +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +**************************** +3.1 'dunfell' Release Series +**************************** + +- :yocto_docs:`3.1 BitBake User Manual ` +- :yocto_docs:`3.1.1 BitBake User Manual ` +- :yocto_docs:`3.1.2 BitBake User Manual ` + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D + Previous Release Manuals +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D + +************************* +3.0 'zeus' Release Series +************************* + +- :yocto_docs:`3.0 BitBake User Manual ` +- :yocto_docs:`3.0.1 BitBake User Manual ` +- :yocto_docs:`3.0.2 BitBake User Manual ` +- :yocto_docs:`3.0.3 BitBake User Manual ` + +**************************** +2.7 'warrior' Release Series +**************************** + +- :yocto_docs:`2.7 BitBake User Manual ` +- :yocto_docs:`2.7.1 BitBake User Manual ` +- :yocto_docs:`2.7.2 BitBake User Manual ` +- :yocto_docs:`2.7.3 BitBake User Manual ` +- :yocto_docs:`2.7.4 BitBake User Manual ` + +************************* +2.6 'thud' Release Series +************************* + +- :yocto_docs:`2.6 BitBake User Manual ` +- :yocto_docs:`2.6.1 BitBake User Manual ` +- :yocto_docs:`2.6.2 BitBake User Manual ` +- :yocto_docs:`2.6.3 BitBake User Manual ` +- :yocto_docs:`2.6.4 BitBake User Manual ` + +************************* +2.5 'sumo' Release Series +************************* + +- :yocto_docs:`2.5 BitBake User Manual ` +- :yocto_docs:`2.5.1 BitBake User Manual ` +- :yocto_docs:`2.5.2 BitBake User Manual ` +- :yocto_docs:`2.5.3 BitBake User Manual ` + +************************** +2.4 'rocko' Release Series +************************** + +- :yocto_docs:`2.4 BitBake User Manual ` +- :yocto_docs:`2.4.1 BitBake User Manual ` +- :yocto_docs:`2.4.2 BitBake User Manual ` +- :yocto_docs:`2.4.3 BitBake User Manual ` +- :yocto_docs:`2.4.4 BitBake User Manual ` + +************************* +2.3 'pyro' Release Series +************************* + +- :yocto_docs:`2.3 BitBake User Manual ` +- :yocto_docs:`2.3.1 BitBake User Manual ` +- :yocto_docs:`2.3.2 BitBake User Manual ` +- :yocto_docs:`2.3.3 BitBake User Manual ` +- :yocto_docs:`2.3.4 BitBake User Manual ` + +************************** +2.2 'morty' Release Series +************************** + +- :yocto_docs:`2.2 BitBake User Manual ` +- :yocto_docs:`2.2.1 BitBake User Manual ` +- :yocto_docs:`2.2.2 BitBake User Manual ` +- :yocto_docs:`2.2.3 BitBake User Manual ` + +**************************** +2.1 'krogoth' Release Series +**************************** + +- :yocto_docs:`2.1 BitBake User Manual ` +- :yocto_docs:`2.1.1 BitBake User Manual ` +- :yocto_docs:`2.1.2 BitBake User Manual ` +- :yocto_docs:`2.1.3 BitBake User Manual ` + +*************************** +2.0 'jethro' Release Series +*************************** + +- :yocto_docs:`1.9 BitBake User Manual ` +- :yocto_docs:`2.0 BitBake User Manual ` +- :yocto_docs:`2.0.1 BitBake User Manual ` +- :yocto_docs:`2.0.2 BitBake User Manual ` +- :yocto_docs:`2.0.3 BitBake User Manual ` + +************************* +1.8 'fido' Release Series +************************* + +- :yocto_docs:`1.8 BitBake User Manual ` +- :yocto_docs:`1.8.1 BitBake User Manual ` +- :yocto_docs:`1.8.2 BitBake User Manual ` + +************************** +1.7 'dizzy' Release Series +************************** + +- :yocto_docs:`1.7 BitBake User Manual ` +- :yocto_docs:`1.7.1 BitBake User Manual ` +- :yocto_docs:`1.7.2 BitBake User Manual ` +- :yocto_docs:`1.7.3 BitBake User Manual ` + +************************** +1.6 'daisy' Release Series +************************** + +- :yocto_docs:`1.6 BitBake User Manual ` +- :yocto_docs:`1.6.1 BitBake User Manual ` +- :yocto_docs:`1.6.2 BitBake User Manual ` +- :yocto_docs:`1.6.3 BitBake User Manual ` + diff --git a/bitbake/doc/sphinx-static/switchers.js b/bitbake/doc/sphinx-st= atic/switchers.js new file mode 100644 index 00000000..32113cfa --- /dev/null +++ b/bitbake/doc/sphinx-static/switchers.js @@ -0,0 +1,233 @@ +(function() { + 'use strict'; + + var all_versions =3D { + 'dev': 'dev (3.2)', + '3.1.2': '3.1.2', + '3.0.3': '3.0.3', + '2.7.4': '2.7.4', + }; + + var all_doctypes =3D { + 'single': 'Individual Webpages', + 'mega': "All-in-one 'Mega' Manual", + }; + + // Simple version comparision + // Return 1 if a > b + // Return -1 if a < b + // Return 0 if a =3D=3D b + function ver_compare(a, b) { + if (a =3D=3D "dev") { + return 1; + } + + if (a =3D=3D=3D b) { + return 0; + } + + var a_components =3D a.split("."); + var b_components =3D b.split("."); + + var len =3D Math.min(a_components.length, b_components.length); + + // loop while the components are equal + for (var i =3D 0; i < len; i++) { + // A bigger than B + if (parseInt(a_components[i]) > parseInt(b_components[i])) { + return 1; + } + + // B bigger than A + if (parseInt(a_components[i]) < parseInt(b_components[i])) { + return -1; + } + } + + // If one's a prefix of the other, the longer one is greater. + if (a_components.length > b_components.length) { + return 1; + } + + if (a_components.length < b_components.length) { + return -1; + } + + // Otherwise they are the same. + return 0; + } + + function build_version_select(current_series, current_version) { + var buf =3D [''); + return buf.join(''); + } + + function build_doctype_select(current_doctype) { + var buf =3D [''); + return buf.join(''); + } + + function navigate_to_first_existing(urls) { + // Navigate to the first existing URL in urls. + var url =3D urls.shift(); + + // Web browsers won't redirect file:// urls to file urls using ajax but + // its useful for local testing + if (url.startsWith("file://")) { + window.location.href =3D url; + return; + } + + if (urls.length =3D=3D 0) { + window.location.href =3D url; + return; + } + $.ajax({ + url: url, + success: function() { + window.location.href =3D url; + }, + error: function() { + navigate_to_first_existing(urls); + } + }); + } + + function get_docroot_url() { + var url =3D window.location.href; + var root =3D DOCUMENTATION_OPTIONS.URL_ROOT; + + var urlarray =3D url.split('/'); + // Trim off anything after '/' + urlarray.pop(); + var depth =3D (root.match(/\.\.\//g) || []).length; + for (var i =3D 0; i < depth; i++) { + urlarray.pop(); + } + + return urlarray.join('/') + '/'; + } + + function on_version_switch() { + var selected_version =3D $(this).children('option:selected').attr('val= ue'); + var url =3D window.location.href; + var current_version =3D DOCUMENTATION_OPTIONS.VERSION; + var docroot =3D get_docroot_url() + + var new_versionpath =3D selected_version + '/'; + if (selected_version =3D=3D "dev") + new_versionpath =3D ''; + + // dev versions have no version prefix + if (current_version =3D=3D "dev") { + var new_url =3D docroot + new_versionpath + url.replace(docroot, "= "); + var fallback_url =3D docroot + new_versionpath; + } else { + var new_url =3D url.replace('/' + current_version + '/', '/' + new= _versionpath); + var fallback_url =3D new_url.replace(url.replace(docroot, ""), ""); + } + + console.log(get_docroot_url()) + console.log(url + " to url " + new_url); + console.log(url + " to fallback " + fallback_url); + + if (new_url !=3D url) { + navigate_to_first_existing([ + new_url, + fallback_url, + 'https://www.yoctoproject.org/docs/', + ]); + } + } + + function on_doctype_switch() { + var selected_doctype =3D $(this).children('option:selected').attr('val= ue'); + var url =3D window.location.href; + if (selected_doctype =3D=3D 'mega') { + var docroot =3D get_docroot_url() + var current_version =3D DOCUMENTATION_OPTIONS.VERSION; + // Assume manuals before 3.2 are using old docbook mega-manual + if (ver_compare(current_version, "3.2") < 0) { + var new_url =3D docroot + "mega-manual/mega-manual.html"; + } else { + var new_url =3D docroot + "singleindex.html"; + } + } else { + var new_url =3D url.replace("singleindex.html", "index.html") + } + + if (new_url !=3D url) { + navigate_to_first_existing([ + new_url, + 'https://www.yoctoproject.org/docs/', + ]); + } + } + + // Returns the current doctype based upon the url + function doctype_segment_from_url(url) { + if (url.includes("singleindex") || url.includes("mega-manual")) + return "mega"; + return "single"; + } + + $(document).ready(function() { + var release =3D DOCUMENTATION_OPTIONS.VERSION; + var current_doctype =3D doctype_segment_from_url(window.location.href); + var current_series =3D release.substr(0, 3); + var version_select =3D build_version_select(current_series, release); + + $('.version_switcher_placeholder').html(version_select); + $('.version_switcher_placeholder select').bind('change', on_version_sw= itch); + + var doctype_select =3D build_doctype_select(current_doctype); + + $('.doctype_switcher_placeholder').html(doctype_select); + $('.doctype_switcher_placeholder select').bind('change', on_doctype_sw= itch); + + if (ver_compare(release, "3.1") < 0) { + $('#outdated-warning').html('Version ' + release + ' of the project = is now considered obsolete, please select and use a more recent version'); + $('#outdated-warning').css('padding', '.5em'); + } else if (release !=3D "dev") { + $.each(all_versions, function(version, title) { + var series =3D version.substr(0, 3); + if (series =3D=3D current_series && version !=3D release) { + $('#outdated-warning').html('This document is for outdated versi= on ' + release + ', you should select the latest release version in this se= ries, ' + version + '.'); + $('#outdated-warning').css('padding', '.5em'); + } + }); + } + }); +})(); diff --git a/bitbake/doc/sphinx-static/theme_overrides.css b/bitbake/doc/sp= hinx-static/theme_overrides.css new file mode 100644 index 00000000..e362677a --- /dev/null +++ b/bitbake/doc/sphinx-static/theme_overrides.css @@ -0,0 +1,162 @@ +/* + SPDX-License-Identifier: CC-BY-2.0-UK +*/ + +body { + font-family: Verdana, Sans, sans-serif; + margin: 0em auto; + color: #333; +} + +h1,h2,h3,h4,h5,h6,h7 { + font-family: Arial, Sans; + color: #00557D; + clear: both; +} + +h1 { + font-size: 2em; + text-align: left; + padding: 0em 0em 0em 0em; + margin: 2em 0em 0em 0em; +} + +h2.subtitle { + margin: 0.10em 0em 3.0em 0em; + padding: 0em 0em 0em 0em; + font-size: 1.8em; + padding-left: 20%; + font-weight: normal; + font-style: italic; +} + +h2 { + margin: 2em 0em 0.66em 0em; + padding: 0.5em 0em 0em 0em; + font-size: 1.5em; + font-weight: bold; +} + +h3.subtitle { + margin: 0em 0em 1em 0em; + padding: 0em 0em 0em 0em; + font-size: 142.14%; + text-align: right; +} + +h3 { + margin: 1em 0em 0.5em 0em; + padding: 1em 0em 0em 0em; + font-size: 140%; + font-weight: bold; +} + +h4 { + margin: 1em 0em 0.5em 0em; + padding: 1em 0em 0em 0em; + font-size: 120%; + font-weight: bold; +} + +h5 { + margin: 1em 0em 0.5em 0em; + padding: 1em 0em 0em 0em; + font-size: 110%; + font-weight: bold; +} + +h6 { + margin: 1em 0em 0em 0em; + padding: 1em 0em 0em 0em; + font-size: 110%; + font-weight: bold; +} + +em { + font-weight: bold; +} + +.pre { + font-size: medium; + font-family: Courier, monospace; +} + +.wy-nav-content a { + text-decoration: underline; + color: #444; + background: transparent; +} + +.wy-nav-content a:hover { + text-decoration: underline; + background-color: #dedede; +} + +.wy-nav-content a:visited { + color: #444; +} + +[alt=3D'Permalink'] { color: #eee; } +[alt=3D'Permalink']:hover { color: black; } + +@media screen { + /* content column + * + * RTD theme's default is 800px as max width for the content, but we h= ave + * tables with tons of columns, which need the full width of the view-= port. + */ + + .wy-nav-content{max-width: none; } + + /* inline literal: drop the borderbox, padding and red color */ + code, .rst-content tt, .rst-content code { + color: inherit; + border: none; + padding: unset; + background: inherit; + font-size: 85%; + } + + .rst-content tt.literal,.rst-content tt.literal,.rst-content code.lite= ral { + color: inherit; + } + + /* Admonition should be gray, not blue or green */ + .rst-content .note .admonition-title, + .rst-content .tip .admonition-title, + .rst-content .warning .admonition-title, + .rst-content .caution .admonition-title, + .rst-content .important .admonition-title { + background: #f0f0f2; + color: #00557D; + + } + + .rst-content .note, + .rst-content .tip, + .rst-content .important, + .rst-content .warning, + .rst-content .caution { + background: #f0f0f2; + } + + /* Remove the icon in front of note/tip element, and before the logo */ + .icon-home:before, .rst-content .admonition-title:before { + display: none + } + + /* a custom informalexample container is used in some doc */ + .informalexample { + border: 1px solid; + border-color: #aaa; + margin: 1em 0em; + padding: 1em; + page-break-inside: avoid; + } + + /* Remove the blue background in the top left corner, around the logo = */ + .wy-side-nav-search { + background: inherit; + } + +} diff --git a/bitbake/doc/template/Vera.xml b/bitbake/doc/template/Vera.xml deleted file mode 100644 index 3c82043e..00000000 --- a/bitbake/doc/template/Vera.xml +++ /dev/null @@ -1 +0,0 @@ -BitstreamVeraSans729546928-235-183-2351287928= 3200TYPE0CIDFontType20<= bf gi=3D"139" ue=3D"169" us=3D"169"/>= <= bf gi=3D"123" ue=3D"244" us=3D"244"/>= <= bf gi=3D"0" ue=3D"8228" us=3D"8228"/>= <= wx w=3D"317"/><= wx w=3D"950"/><= wx w=3D"500"/><= wx w=3D"336"/><= wx w=3D"636"/><= wx w=3D"636"/><= wx w=3D"837"/>= = = = = = = = = = = = = = = = = = = = = = <= pair kern=3D"31" kpx2=3D"180"/>= <= pair kern=3D"-17" kpx2=3D"55"/><= pair kern=3D"-67" kpx2=3D"186"/><= pair kern=3D"-17" kpx2=3D"111"/>= <= pair kern=3D"-21" kpx2=3D"124"/>= = = <= pair kern=3D"27" kpx2=3D"16"/><= pair kern=3D"-54" kpx2=3D"234"/>= <= pair kern=3D"-63" kpx2=3D"57"/>= = = = = = <= pair kern=3D"-17" kpx2=3D"254"/>= = <= pair kern=3D"-40" kpx2=3D"16"/><= pair kern=3D"-44" kpx2=3D"127"/>= = = <= pair kern=3D"-17" kpx2=3D"84"/>= <= pair kern=3D"-67" kpx2=3D"235"/> \ No newline at end of file diff --git a/bitbake/doc/template/VeraMoBd.xml b/bitbake/doc/template/VeraM= oBd.xml deleted file mode 100644 index 9b33107a..00000000 --- a/bitbake/doc/template/VeraMoBd.xml +++ /dev/null @@ -1 +0,0 @@ -BitstreamVeraSansMono-BoldBitstream Vera Sans Mono BoldBitstream Vera Sa= ns Mono729546759-240-19-2356059283400TYPE0<= multibyte-extras>CIDFontType20<= bf gi=3D"202" ue=3D"203" us=3D"203"/>= <= bf gi=3D"0" ue=3D"8213" us=3D"8213"/>= <= bf gi=3D"0" ue=3D"8707" us=3D"8707"/>= <= bf gi=3D"0" ue=3D"8801" us=3D"8801"/>= \ No newline at end of file diff --git a/bitbake/doc/template/VeraMono.xml b/bitbake/doc/template/VeraM= ono.xml deleted file mode 100644 index 3a0a8665..00000000 --- a/bitbake/doc/template/VeraMono.xml +++ /dev/null @@ -1 +0,0 @@ -BitstreamVeraSansMono-RomanBitstream Vera Sans MonoBitstream Vera Sans M= ono729546759-240-4<= bottom>-2356059283400TYPE0CIDFontType20<= bf gi=3D"145" ue=3D"216" us=3D"216"/>= <= bf gi=3D"0" ue=3D"8707" us=3D"8707"/>= <= bf gi=3D"0" ue=3D"8801" us=3D"8801"/>= \ No newline at end of file diff --git a/bitbake/doc/template/component.title.xsl b/bitbake/doc/templat= e/component.title.xsl deleted file mode 100644 index faef0432..00000000 --- a/bitbake/doc/template/component.title.xsl +++ /dev/null @@ -1,39 +0,0 @@ - -=20=20 - - -=20=20=20=20 - - - - - - 6 - 5 - 4 - 3 - 2 - 1 - - - - title - - - - - - - - - - - - - - - diff --git a/bitbake/doc/template/db-pdf.xsl b/bitbake/doc/template/db-pdf.= xsl deleted file mode 100644 index 3dd065a5..00000000 --- a/bitbake/doc/template/db-pdf.xsl +++ /dev/null @@ -1,64 +0,0 @@ - - -=20=20 - - - - =20 - - - - - - - - - - - - 1 10 1 - - =20 - - - - 0.5pt - solid - #cccccc - - - - - - 0.5pt - solid - #cccccc - - - - - #cccccc - - - - #cccccc - - -=20 - - - - - - - - - - diff --git a/bitbake/doc/template/division.title.xsl b/bitbake/doc/template= /division.title.xsl deleted file mode 100644 index 9c843bc7..00000000 --- a/bitbake/doc/template/division.title.xsl +++ /dev/null @@ -1,25 +0,0 @@ - -=20=20 - - -=20=20=20=20 -

    - title - - - - - - - - - - -

    -     -     - diff --git a/bitbake/doc/template/fop-config.xml b/bitbake/doc/template/fop= -config.xml deleted file mode 100644 index 09cc5ca0..00000000 --- a/bitbake/doc/template/fop-config.xml +++ /dev/null @@ -1,58 +0,0 @@ - - - - true - - - true - - =09 - ../template - ../template -=20 - - - - -=20 - - =20 -=20 - -=20 - - - - - - - - - - - - - - - - - -=20=20=20=20=20=20=20=20 - - - - - - diff --git a/bitbake/doc/template/formal.object.heading.xsl b/bitbake/doc/t= emplate/formal.object.heading.xsl deleted file mode 100644 index 4f3900d1..00000000 --- a/bitbake/doc/template/formal.object.heading.xsl +++ /dev/null @@ -1,21 +0,0 @@ - -=20=20 - - - - - - - -

    - - - - -

    -     -     \ No newline at end of file diff --git a/bitbake/doc/template/gloss-permalinks.xsl b/bitbake/doc/templa= te/gloss-permalinks.xsl deleted file mode 100644 index 6bf58116..00000000 --- a/bitbake/doc/template/gloss-permalinks.xsl +++ /dev/null @@ -1,14 +0,0 @@ -=0D -=0D - =0D - =0D - =0D - =0D - =0D - =0D - =0D - =0D -=0D diff --git a/bitbake/doc/template/permalinks.xsl b/bitbake/doc/template/per= malinks.xsl deleted file mode 100644 index d2a1c145..00000000 --- a/bitbake/doc/template/permalinks.xsl +++ /dev/null @@ -1,25 +0,0 @@ - - - - - =C2=B6 - - - - - - - - - - - - - - - - - - diff --git a/bitbake/doc/template/section.title.xsl b/bitbake/doc/template/= section.title.xsl deleted file mode 100644 index 5c6ff9a9..00000000 --- a/bitbake/doc/template/section.title.xsl +++ /dev/null @@ -1,55 +0,0 @@ - - - - - - - - 1 - 2 - 3 - 4 - 5 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/bitbake/doc/template/titlepage.templates.xml b/bitbake/doc/tem= plate/titlepage.templates.xml deleted file mode 100644 index 38ec11a4..00000000 --- a/bitbake/doc/template/titlepage.templates.xml +++ /dev/null @@ -1,1259 +0,0 @@ - - - - - - - - - - - - -]> - - - - - - - - - - - - - - - <subtitle param:node=3D"ancestor-or-self::article[1]" - keep-with-next=3D"always" - font-size=3D"&hsize3;" - font-weight=3D"bold" - space-after=3D"0.8em"/> - - <corpauthor space-before=3D"0.5em" - font-size=3D"&hsize3;"/> - <authorgroup space-before=3D"0.5em" - font-size=3D"&hsize2;"/> - <author space-before=3D"0.5em" - font-size=3D"&hsize2;" - space-after=3D"0.8em"/> - - <email font-size=3D"&hsize2;"/> - - <othercredit space-before=3D"0.5em"/> - <releaseinfo space-before=3D"0.5em"/> - <copyright space-before=3D"0.5em"/> - <legalnotice text-align=3D"start" - margin-left=3D"0.5in" - margin-right=3D"0.5in" - font-family=3D"{$body.fontset}"/> - <pubdate space-before=3D"0.5em"/> - <para></para> - <revision space-before=3D"0.5em"/> - <revhistory space-before=3D"0.5em"/> - <abstract space-before=3D"0.5em" - text-align=3D"start" - margin-left=3D"0.5in" - margin-right=3D"0.5in" - font-family=3D"{$body.fontset}"/> - - <para></para> - </t:titlepage-content> - - <t:titlepage-content t:side=3D"verso"> - </t:titlepage-content> - - <t:titlepage-separator> - </t:titlepage-separator> - - <t:titlepage-before t:side=3D"recto"> - </t:titlepage-before> - - <t:titlepage-before t:side=3D"verso"> - </t:titlepage-before> -</t:titlepage> - -<!-- =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --> - -<t:titlepage t:element=3D"set" t:wrapper=3D"fo:block"> - <t:titlepage-content t:side=3D"recto"> - <title - t:named-template=3D"division.title" - param:node=3D"ancestor-or-self::set[1]" - text-align=3D"center" - font-size=3D"&hsize5;" - space-before=3D"&hsize5space;" - font-weight=3D"bold" - font-family=3D"{$title.fontset}"/> - <subtitle - font-family=3D"{$title.fontset}" - text-align=3D"center"/> - <corpauthor/> - <authorgroup/> - <author/> - <othercredit/> - <releaseinfo/> - <copyright/> - <legalnotice/> - <pubdate/> - <revision/> - <revhistory/> - <abstract/> - </t:titlepage-content> - - <t:titlepage-content t:side=3D"verso"> - </t:titlepage-content> - - <t:titlepage-separator> - </t:titlepage-separator> - - <t:titlepage-before t:side=3D"recto"> - </t:titlepage-before> - - <t:titlepage-before t:side=3D"verso"> - </t:titlepage-before> -</t:titlepage> - -<!-- =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --> - - <t:titlepage t:element=3D"book" t:wrapper=3D"fo:block"> - <t:titlepage-content t:side=3D"recto"> - - <mediaobject/> - -<!-- - -# If you leave this block of code in then the text title in the -# <title>BitBake User Manual statement of the -# bitbake-user-manual.xml file is rendered on the title page below the -# image. Commenting it out gets it out of there yet allows it -# to be retained in the tab text for the HTML version of the -# manual. - - ---> - <subtitle - text-align=3D"center" - font-size=3D"&hsize4;" - space-before=3D"&hsize4space;" - font-family=3D"{$title.fontset}"/> - <corpauthor font-size=3D"&hsize3;" - keep-with-next=3D"always" - space-before=3D"2in"/> - <authorgroup space-before=3D"2in"/> - <author font-size=3D"&hsize3;" - space-before=3D"&hsize2space;" - keep-with-next=3D"always"/> - </t:titlepage-content> - - <t:titlepage-content t:side=3D"verso"> -<!-- -# If you leave this block of code in then the text title in the -# <title>BitBake User Manual statement of the -# bitbake-user-manual.xml file is rendered on the title page below the -# image. Commenting it out gets it out of there yet allows it -# to be retained in the tab text for the HTML version of the -# manual. - - ---> - <corpauthor/> - <authorgroup t:named-template=3D"verso.authorgroup"/> - <author/> - <othercredit/> - <pubdate space-before=3D"1em"/> - <copyright/> - <abstract/> - <legalnotice font-size=3D"8pt"/> - </t:titlepage-content> - - <t:titlepage-separator> - <fo:block break-after=3D"page"/> - </t:titlepage-separator> - - <t:titlepage-before t:side=3D"recto"> - </t:titlepage-before> - - <t:titlepage-before t:side=3D"verso"> - <fo:block break-after=3D"page"/> - </t:titlepage-before> -</t:titlepage> - -<!-- =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --> - -<t:titlepage t:element=3D"part" t:wrapper=3D"fo:block"> - <t:titlepage-content t:side=3D"recto"> - <title - t:named-template=3D"division.title" - param:node=3D"ancestor-or-self::part[1]" - text-align=3D"center" - font-size=3D"&hsize5;" - space-before=3D"&hsize5space;" - font-weight=3D"bold" - font-family=3D"{$title.fontset}"/> - <subtitle - text-align=3D"center" - font-size=3D"&hsize4;" - space-before=3D"&hsize4space;" - font-weight=3D'bold' - font-style=3D'italic' - font-family=3D"{$title.fontset}"/> - </t:titlepage-content> - - <t:titlepage-content t:side=3D"verso"> - </t:titlepage-content> - - <t:titlepage-separator> - </t:titlepage-separator> - - <t:titlepage-before t:side=3D"recto"> - </t:titlepage-before> - - <t:titlepage-before t:side=3D"verso"> - </t:titlepage-before> -</t:titlepage> - -<t:titlepage t:element=3D"partintro" t:wrapper=3D"fo:block"> - <t:titlepage-content t:side=3D"recto"> - <title - text-align=3D"center" - font-size=3D"&hsize5;" - font-weight=3D"bold" - space-before=3D"1em" - font-family=3D"{$title.fontset}"/> - <subtitle - text-align=3D"center" - font-size=3D"&hsize2;" - font-weight=3D"bold" - font-style=3D"italic" - font-family=3D"{$title.fontset}"/> - <corpauthor/> - <authorgroup/> - <author/> - <othercredit/> - <releaseinfo/> - <copyright/> - <legalnotice/> - <pubdate/> - <revision/> - <revhistory/> - <abstract/> - </t:titlepage-content> - - <t:titlepage-content t:side=3D"verso"> - </t:titlepage-content> - - <t:titlepage-separator> - </t:titlepage-separator> - - <t:titlepage-before t:side=3D"recto"> - </t:titlepage-before> - - <t:titlepage-before t:side=3D"verso"> - </t:titlepage-before> -</t:titlepage> - -<!-- =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --> - -<t:titlepage t:element=3D"reference" t:wrapper=3D"fo:block"> - <t:titlepage-content t:side=3D"recto"> - <title - t:named-template=3D"division.title" - param:node=3D"ancestor-or-self::reference[1]" - text-align=3D"center" - font-size=3D"&hsize5;" - space-before=3D"&hsize5space;" - font-weight=3D"bold" - font-family=3D"{$title.fontset}"/> - <subtitle - font-family=3D"{$title.fontset}" - text-align=3D"center"/> - <corpauthor/> - <authorgroup/> - <author/> - <othercredit/> - <releaseinfo/> - <copyright/> - <legalnotice/> - <pubdate/> - <revision/> - <revhistory/> - <abstract/> - </t:titlepage-content> - - <t:titlepage-content t:side=3D"verso"> - </t:titlepage-content> - - <t:titlepage-separator> - </t:titlepage-separator> - - <t:titlepage-before t:side=3D"recto"> - </t:titlepage-before> - - <t:titlepage-before t:side=3D"verso"> - </t:titlepage-before> -</t:titlepage> - -<!-- =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --> - -<t:titlepage t:element=3D"refsynopsisdiv" t:wrapper=3D"fo:block"> - <t:titlepage-content t:side=3D"recto"> - <title - font-family=3D"{$title.fontset}"/> - </t:titlepage-content> - - <t:titlepage-content t:side=3D"verso"> - </t:titlepage-content> - - <t:titlepage-separator> - </t:titlepage-separator> - - <t:titlepage-before t:side=3D"recto"> - </t:titlepage-before> - - <t:titlepage-before t:side=3D"verso"> - </t:titlepage-before> -</t:titlepage> - -<!-- =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --> - -<t:titlepage t:element=3D"refsection" t:wrapper=3D"fo:block"> - <t:titlepage-content t:side=3D"recto"> - <title - font-family=3D"{$title.fontset}"/> - </t:titlepage-content> - - <t:titlepage-content t:side=3D"verso"> - </t:titlepage-content> - - <t:titlepage-separator> - </t:titlepage-separator> - - <t:titlepage-before t:side=3D"recto"> - </t:titlepage-before> - - <t:titlepage-before t:side=3D"verso"> - </t:titlepage-before> -</t:titlepage> - -<!-- =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --> - -<t:titlepage t:element=3D"refsect1" t:wrapper=3D"fo:block"> - <t:titlepage-content t:side=3D"recto"> - <title - font-family=3D"{$title.fontset}"/> - </t:titlepage-content> - - <t:titlepage-content t:side=3D"verso"> - </t:titlepage-content> - - <t:titlepage-separator> - </t:titlepage-separator> - - <t:titlepage-before t:side=3D"recto"> - </t:titlepage-before> - - <t:titlepage-before t:side=3D"verso"> - </t:titlepage-before> -</t:titlepage> - -<!-- =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --> - -<t:titlepage t:element=3D"refsect2" t:wrapper=3D"fo:block"> - <t:titlepage-content t:side=3D"recto"> - <title - font-family=3D"{$title.fontset}"/> - </t:titlepage-content> - - <t:titlepage-content t:side=3D"verso"> - </t:titlepage-content> - - <t:titlepage-separator> - </t:titlepage-separator> - - <t:titlepage-before t:side=3D"recto"> - </t:titlepage-before> - - <t:titlepage-before t:side=3D"verso"> - </t:titlepage-before> -</t:titlepage> - -<!-- =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --> - -<t:titlepage t:element=3D"refsect3" t:wrapper=3D"fo:block"> - <t:titlepage-content t:side=3D"recto"> - <title - font-family=3D"{$title.fontset}"/> - </t:titlepage-content> - - <t:titlepage-content t:side=3D"verso"> - </t:titlepage-content> - - <t:titlepage-separator> - </t:titlepage-separator> - - <t:titlepage-before t:side=3D"recto"> - </t:titlepage-before> - - <t:titlepage-before t:side=3D"verso"> - </t:titlepage-before> -</t:titlepage> - -<!-- =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --> - - <t:titlepage t:element=3D"dedication" t:wrapper=3D"fo:block"> - <t:titlepage-content t:side=3D"recto"> - <title - t:force=3D"1" - t:named-template=3D"component.title" - param:node=3D"ancestor-or-self::dedication[1]" - margin-left=3D"{$title.margin.left}" - font-size=3D"&hsize5;" - font-family=3D"{$title.fontset}" - font-weight=3D"bold"/> - <subtitle - font-family=3D"{$title.fontset}"/> - </t:titlepage-content> - - <t:titlepage-content t:side=3D"verso"> - </t:titlepage-content> - - <t:titlepage-separator> - </t:titlepage-separator> - - <t:titlepage-before t:side=3D"recto"> - </t:titlepage-before> - - <t:titlepage-before t:side=3D"verso"> - </t:titlepage-before> -</t:titlepage> - -<!-- =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --> - - <t:titlepage t:element=3D"preface" t:wrapper=3D"fo:block"> - <t:titlepage-content t:side=3D"recto"> - <title - t:force=3D"1" - t:named-template=3D"component.title" - param:node=3D"ancestor-or-self::preface[1]" - margin-left=3D"{$title.margin.left}" - font-size=3D"&hsize5;" - font-family=3D"{$title.fontset}" - font-weight=3D"bold"/> - <subtitle - font-family=3D"{$title.fontset}"/> - <corpauthor/> - <authorgroup/> - <author/> - <othercredit/> - <releaseinfo/> - <copyright/> - <legalnotice/> - <pubdate/> - <revision/> - <revhistory/> - <abstract/> - </t:titlepage-content> - - <t:titlepage-content t:side=3D"verso"> - </t:titlepage-content> - - <t:titlepage-separator> - </t:titlepage-separator> - - <t:titlepage-before t:side=3D"recto"> - </t:titlepage-before> - - <t:titlepage-before t:side=3D"verso"> - </t:titlepage-before> -</t:titlepage> - -<!-- =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --> - - <t:titlepage t:element=3D"chapter" t:wrapper=3D"fo:block" - font-family=3D"{$title.fontset}"> - <t:titlepage-content t:side=3D"recto" margin-left=3D"{$title.margin.le= ft}"> - <title t:named-template=3D"component.title" - param:node=3D"ancestor-or-self::chapter[1]" - font-size=3D"&hsize5;" - font-weight=3D"bold"/> - - <subtitle space-before=3D"0.5em" - font-style=3D"italic" - font-size=3D"&hsize2;" - font-weight=3D"bold"/> - - <corpauthor space-before=3D"0.5em" - space-after=3D"0.5em" - font-size=3D"&hsize2;"/> - - <authorgroup space-before=3D"0.5em" - space-after=3D"0.5em" - font-size=3D"&hsize2;"/> - - <author space-before=3D"0.5em" - space-after=3D"0.5em" - font-size=3D"&hsize2;"/> - - <othercredit/> - <releaseinfo/> - <copyright/> - <legalnotice/> - <pubdate/> - <revision/> - <revhistory/> - <abstract/> - </t:titlepage-content> - - <t:titlepage-content t:side=3D"verso"> - </t:titlepage-content> - - <t:titlepage-separator> - </t:titlepage-separator> - - <t:titlepage-before t:side=3D"recto"> - </t:titlepage-before> - - <t:titlepage-before t:side=3D"verso"> - </t:titlepage-before> -</t:titlepage> - -<!-- =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --> - - <t:titlepage t:element=3D"appendix" t:wrapper=3D"fo:block"> - <t:titlepage-content t:side=3D"recto"> - <title - t:named-template=3D"component.title" - param:node=3D"ancestor-or-self::appendix[1]" - margin-left=3D"{$title.margin.left}" - font-size=3D"&hsize5;" - font-weight=3D"bold" - font-family=3D"{$title.fontset}"/> - <subtitle - font-family=3D"{$title.fontset}"/> - <corpauthor/> - <authorgroup/> - <author/> - <othercredit/> - <releaseinfo/> - <copyright/> - <legalnotice/> - <pubdate/> - <revision/> - <revhistory/> - <abstract/> - </t:titlepage-content> - - <t:titlepage-content t:side=3D"verso"> - </t:titlepage-content> - - <t:titlepage-separator> - </t:titlepage-separator> - - <t:titlepage-before t:side=3D"recto"> - </t:titlepage-before> - - <t:titlepage-before t:side=3D"verso"> - </t:titlepage-before> -</t:titlepage> - -<!-- =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --> - -<t:titlepage t:element=3D"section" t:wrapper=3D"fo:block"> - <t:titlepage-content t:side=3D"recto"> - <title - margin-left=3D"{$title.margin.left}" - font-family=3D"{$title.fontset}"/> - <subtitle - font-family=3D"{$title.fontset}"/> - <corpauthor/> - <authorgroup/> - <author/> - <othercredit/> - <releaseinfo/> - <copyright/> - <legalnotice/> - <pubdate/> - <revision/> - <revhistory/> - <abstract/> - </t:titlepage-content> - - <t:titlepage-content t:side=3D"verso"> - </t:titlepage-content> - - <t:titlepage-separator> - </t:titlepage-separator> - - <t:titlepage-before t:side=3D"recto"> - </t:titlepage-before> - - <t:titlepage-before t:side=3D"verso"> - </t:titlepage-before> -</t:titlepage> - -<t:titlepage t:element=3D"sect1" t:wrapper=3D"fo:block"> - <t:titlepage-content t:side=3D"recto"> - <title - margin-left=3D"{$title.margin.left}" - font-family=3D"{$title.fontset}"/> - <subtitle - font-family=3D"{$title.fontset}"/> - <corpauthor/> - <authorgroup/> - <author/> - <othercredit/> - <releaseinfo/> - <copyright/> - <legalnotice/> - <pubdate/> - <revision/> - <revhistory/> - <abstract/> - </t:titlepage-content> - - <t:titlepage-content t:side=3D"verso"> - </t:titlepage-content> - - <t:titlepage-separator> - </t:titlepage-separator> - - <t:titlepage-before t:side=3D"recto"> - </t:titlepage-before> - - <t:titlepage-before t:side=3D"verso"> - </t:titlepage-before> -</t:titlepage> - -<t:titlepage t:element=3D"sect2" t:wrapper=3D"fo:block"> - <t:titlepage-content t:side=3D"recto"> - <title - margin-left=3D"{$title.margin.left}" - font-family=3D"{$title.fontset}"/> - <subtitle - font-family=3D"{$title.fontset}"/> - <corpauthor/> - <authorgroup/> - <author/> - <othercredit/> - <releaseinfo/> - <copyright/> - <legalnotice/> - <pubdate/> - <revision/> - <revhistory/> - <abstract/> - </t:titlepage-content> - - <t:titlepage-content t:side=3D"verso"> - </t:titlepage-content> - - <t:titlepage-separator> - </t:titlepage-separator> - - <t:titlepage-before t:side=3D"recto"> - </t:titlepage-before> - - <t:titlepage-before t:side=3D"verso"> - </t:titlepage-before> -</t:titlepage> - -<t:titlepage t:element=3D"sect3" t:wrapper=3D"fo:block"> - <t:titlepage-content t:side=3D"recto"> - <title - margin-left=3D"{$title.margin.left}" - font-family=3D"{$title.fontset}"/> - <subtitle - font-family=3D"{$title.fontset}"/> - <corpauthor/> - <authorgroup/> - <author/> - <othercredit/> - <releaseinfo/> - <copyright/> - <legalnotice/> - <pubdate/> - <revision/> - <revhistory/> - <abstract/> - </t:titlepage-content> - - <t:titlepage-content t:side=3D"verso"> - </t:titlepage-content> - - <t:titlepage-separator> - </t:titlepage-separator> - - <t:titlepage-before t:side=3D"recto"> - </t:titlepage-before> - - <t:titlepage-before t:side=3D"verso"> - </t:titlepage-before> -</t:titlepage> - -<t:titlepage t:element=3D"sect4" t:wrapper=3D"fo:block"> - <t:titlepage-content t:side=3D"recto"> - <title - margin-left=3D"{$title.margin.left}" - font-family=3D"{$title.fontset}"/> - <subtitle - font-family=3D"{$title.fontset}"/> - <corpauthor/> - <authorgroup/> - <author/> - <othercredit/> - <releaseinfo/> - <copyright/> - <legalnotice/> - <pubdate/> - <revision/> - <revhistory/> - <abstract/> - </t:titlepage-content> - - <t:titlepage-content t:side=3D"verso"> - </t:titlepage-content> - - <t:titlepage-separator> - </t:titlepage-separator> - - <t:titlepage-before t:side=3D"recto"> - </t:titlepage-before> - - <t:titlepage-before t:side=3D"verso"> - </t:titlepage-before> -</t:titlepage> - -<t:titlepage t:element=3D"sect5" t:wrapper=3D"fo:block"> - <t:titlepage-content t:side=3D"recto"> - <title - margin-left=3D"{$title.margin.left}" - font-family=3D"{$title.fontset}"/> - <subtitle - font-family=3D"{$title.fontset}"/> - <corpauthor/> - <authorgroup/> - <author/> - <othercredit/> - <releaseinfo/> - <copyright/> - <legalnotice/> - <pubdate/> - <revision/> - <revhistory/> - <abstract/> - </t:titlepage-content> - - <t:titlepage-content t:side=3D"verso"> - </t:titlepage-content> - - <t:titlepage-separator> - </t:titlepage-separator> - - <t:titlepage-before t:side=3D"recto"> - </t:titlepage-before> - - <t:titlepage-before t:side=3D"verso"> - </t:titlepage-before> -</t:titlepage> - -<t:titlepage t:element=3D"simplesect" t:wrapper=3D"fo:block"> - <t:titlepage-content t:side=3D"recto"> - <title - margin-left=3D"{$title.margin.left}" - font-family=3D"{$title.fontset}"/> - <subtitle - font-family=3D"{$title.fontset}"/> - <corpauthor/> - <authorgroup/> - <author/> - <othercredit/> - <releaseinfo/> - <copyright/> - <legalnotice/> - <pubdate/> - <revision/> - <revhistory/> - <abstract/> - </t:titlepage-content> - - <t:titlepage-content t:side=3D"verso"> - </t:titlepage-content> - - <t:titlepage-separator> - </t:titlepage-separator> - - <t:titlepage-before t:side=3D"recto"> - </t:titlepage-before> - - <t:titlepage-before t:side=3D"verso"> - </t:titlepage-before> -</t:titlepage> - -<!-- =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --> - - <t:titlepage t:element=3D"bibliography" t:wrapper=3D"fo:block"> - <t:titlepage-content t:side=3D"recto"> - <title - t:force=3D"1" - t:named-template=3D"component.title" - param:node=3D"ancestor-or-self::bibliography[1]" - margin-left=3D"{$title.margin.left}" - font-size=3D"&hsize5;" - font-family=3D"{$title.fontset}" - font-weight=3D"bold"/> - <subtitle - font-family=3D"{$title.fontset}"/> - </t:titlepage-content> - - <t:titlepage-content t:side=3D"verso"> - </t:titlepage-content> - - <t:titlepage-separator> - </t:titlepage-separator> - - <t:titlepage-before t:side=3D"recto"> - </t:titlepage-before> - - <t:titlepage-before t:side=3D"verso"> - </t:titlepage-before> - </t:titlepage> - -<!-- =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --> - - <t:titlepage t:element=3D"bibliodiv" t:wrapper=3D"fo:block"> - <t:titlepage-content t:side=3D"recto"> - <title t:named-template=3D"component.title" - param:node=3D"ancestor-or-self::bibliodiv[1]" - margin-left=3D"{$title.margin.left}" - font-size=3D"&hsize4;" - font-family=3D"{$title.fontset}" - font-weight=3D"bold"/> - <subtitle - font-family=3D"{$title.fontset}"/> - </t:titlepage-content> - - <t:titlepage-content t:side=3D"verso"> - </t:titlepage-content> - - <t:titlepage-separator> - </t:titlepage-separator> - - <t:titlepage-before t:side=3D"recto"> - </t:titlepage-before> - - <t:titlepage-before t:side=3D"verso"> - </t:titlepage-before> - </t:titlepage> - -<!-- =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --> - - <t:titlepage t:element=3D"glossary" t:wrapper=3D"fo:block"> - <t:titlepage-content t:side=3D"recto"> - <title - t:force=3D"1" - t:named-template=3D"component.title" - param:node=3D"ancestor-or-self::glossary[1]" - margin-left=3D"{$title.margin.left}" - font-size=3D"&hsize5;" - font-family=3D"{$title.fontset}" - font-weight=3D"bold"/> - <subtitle - font-family=3D"{$title.fontset}"/> - </t:titlepage-content> - - <t:titlepage-content t:side=3D"verso"> - </t:titlepage-content> - - <t:titlepage-separator> - </t:titlepage-separator> - - <t:titlepage-before t:side=3D"recto"> - </t:titlepage-before> - - <t:titlepage-before t:side=3D"verso"> - </t:titlepage-before> - </t:titlepage> - -<!-- =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --> - - <t:titlepage t:element=3D"glossdiv" t:wrapper=3D"fo:block"> - <t:titlepage-content t:side=3D"recto"> - <title t:named-template=3D"component.title" - param:node=3D"ancestor-or-self::glossdiv[1]" - margin-left=3D"{$title.margin.left}" - font-size=3D"&hsize4;" - font-family=3D"{$title.fontset}" - font-weight=3D"bold"/> - <subtitle - font-family=3D"{$title.fontset}"/> - </t:titlepage-content> - - <t:titlepage-content t:side=3D"verso"> - </t:titlepage-content> - - <t:titlepage-separator> - </t:titlepage-separator> - - <t:titlepage-before t:side=3D"recto"> - </t:titlepage-before> - - <t:titlepage-before t:side=3D"verso"> - </t:titlepage-before> - </t:titlepage> - -<!-- =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --> - - <t:titlepage t:element=3D"index" t:wrapper=3D"fo:block"> - <t:titlepage-content t:side=3D"recto"> - <title - t:force=3D"1" - t:named-template=3D"component.title" - param:node=3D"ancestor-or-self::index[1]" - param:pagewide=3D"1" - margin-left=3D"0pt" - font-size=3D"&hsize5;" - font-family=3D"{$title.fontset}" - font-weight=3D"bold"/> - <subtitle - font-family=3D"{$title.fontset}"/> - </t:titlepage-content> - - <t:titlepage-content t:side=3D"verso"> - </t:titlepage-content> - - <t:titlepage-separator> - </t:titlepage-separator> - - <t:titlepage-before t:side=3D"recto"> - </t:titlepage-before> - - <t:titlepage-before t:side=3D"verso"> - </t:titlepage-before> - </t:titlepage> - -<!-- =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --> - - <!-- The indexdiv.title template is used so that manual and --> - <!-- automatically generated indexdiv titles get the same --> - <!-- formatting. --> - - <t:titlepage t:element=3D"indexdiv" t:wrapper=3D"fo:block"> - <t:titlepage-content t:side=3D"recto"> - <title t:force=3D"1" - t:named-template=3D"indexdiv.title" - param:title=3D"title"/> - <subtitle - font-family=3D"{$title.fontset}"/> - </t:titlepage-content> - - <t:titlepage-content t:side=3D"verso"> - </t:titlepage-content> - - <t:titlepage-separator> - </t:titlepage-separator> - - <t:titlepage-before t:side=3D"recto"> - </t:titlepage-before> - - <t:titlepage-before t:side=3D"verso"> - </t:titlepage-before> - </t:titlepage> - -<!-- =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --> - - <t:titlepage t:element=3D"setindex" t:wrapper=3D"fo:block"> - <t:titlepage-content t:side=3D"recto"> - <title - t:force=3D"1" - t:named-template=3D"component.title" - param:node=3D"ancestor-or-self::setindex[1]" - param:pagewide=3D"1" - margin-left=3D"0pt" - font-size=3D"&hsize5;" - font-family=3D"{$title.fontset}" - font-weight=3D"bold"/> - <subtitle - font-family=3D"{$title.fontset}"/> - </t:titlepage-content> - - <t:titlepage-content t:side=3D"verso"> - </t:titlepage-content> - - <t:titlepage-separator> - </t:titlepage-separator> - - <t:titlepage-before t:side=3D"recto"> - </t:titlepage-before> - - <t:titlepage-before t:side=3D"verso"> - </t:titlepage-before> - </t:titlepage> - -<!-- =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --> - - <t:titlepage t:element=3D"colophon" t:wrapper=3D"fo:block"> - <t:titlepage-content t:side=3D"recto"> - <title - t:force=3D"1" - t:named-template=3D"component.title" - param:node=3D"ancestor-or-self::colophon[1]" - margin-left=3D"{$title.margin.left}" - font-size=3D"&hsize5;" - font-family=3D"{$title.fontset}" - font-weight=3D"bold"/> - <subtitle - font-family=3D"{$title.fontset}"/> - </t:titlepage-content> - - <t:titlepage-content t:side=3D"verso"> - </t:titlepage-content> - - <t:titlepage-separator> - </t:titlepage-separator> - - <t:titlepage-before t:side=3D"recto"> - </t:titlepage-before> - - <t:titlepage-before t:side=3D"verso"> - </t:titlepage-before> -</t:titlepage> - -<!-- =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --> - - <t:titlepage t:element=3D"table.of.contents" t:wrapper=3D"fo:block"> - <t:titlepage-content t:side=3D"recto"> - <title - t:force=3D"1" - t:named-template=3D"gentext" - param:key=3D"'TableofContents'" - space-before.minimum=3D"1em" - space-before.optimum=3D"1.5em" - space-before.maximum=3D"2em" - space-after=3D"0.5em" - margin-left=3D"{$title.margin.left}" - font-size=3D"&hsize3;" - font-weight=3D"bold" - font-family=3D"{$title.fontset}"/> - </t:titlepage-content> - - <t:titlepage-content t:side=3D"verso"> - </t:titlepage-content> - - <t:titlepage-separator> - </t:titlepage-separator> - - <t:titlepage-before t:side=3D"recto"> - </t:titlepage-before> - - <t:titlepage-before t:side=3D"verso"> - </t:titlepage-before> - </t:titlepage> - - <t:titlepage t:element=3D"list.of.tables" t:wrapper=3D"fo:block"> - <t:titlepage-content t:side=3D"recto"> - <title - t:force=3D"1" - t:named-template=3D"gentext" - param:key=3D"'ListofTables'" - space-before.minimum=3D"1em" - space-before.optimum=3D"1.5em" - space-before.maximum=3D"2em" - space-after=3D"0.5em" - margin-left=3D"{$title.margin.left}" - font-size=3D"&hsize3;" - font-weight=3D"bold" - font-family=3D"{$title.fontset}"/> - </t:titlepage-content> - - <t:titlepage-content t:side=3D"verso"> - </t:titlepage-content> - - <t:titlepage-separator> - </t:titlepage-separator> - - <t:titlepage-before t:side=3D"recto"> - </t:titlepage-before> - - <t:titlepage-before t:side=3D"verso"> - </t:titlepage-before> - </t:titlepage> - - <t:titlepage t:element=3D"list.of.figures" t:wrapper=3D"fo:block"> - <t:titlepage-content t:side=3D"recto"> - <title - t:force=3D"1" - t:named-template=3D"gentext" - param:key=3D"'ListofFigures'" - space-before.minimum=3D"1em" - space-before.optimum=3D"1.5em" - space-before.maximum=3D"2em" - space-after=3D"0.5em" - margin-left=3D"{$title.margin.left}" - font-size=3D"&hsize3;" - font-weight=3D"bold" - font-family=3D"{$title.fontset}"/> - </t:titlepage-content> - - <t:titlepage-content t:side=3D"verso"> - </t:titlepage-content> - - <t:titlepage-separator> - </t:titlepage-separator> - - <t:titlepage-before t:side=3D"recto"> - </t:titlepage-before> - - <t:titlepage-before t:side=3D"verso"> - </t:titlepage-before> - </t:titlepage> - - <t:titlepage t:element=3D"list.of.examples" t:wrapper=3D"fo:block"> - <t:titlepage-content t:side=3D"recto"> - <title - t:force=3D"1" - t:named-template=3D"gentext" - param:key=3D"'ListofExamples'" - space-before.minimum=3D"1em" - space-before.optimum=3D"1.5em" - space-before.maximum=3D"2em" - space-after=3D"0.5em" - margin-left=3D"{$title.margin.left}" - font-size=3D"&hsize3;" - font-weight=3D"bold" - font-family=3D"{$title.fontset}"/> - </t:titlepage-content> - - <t:titlepage-content t:side=3D"verso"> - </t:titlepage-content> - - <t:titlepage-separator> - </t:titlepage-separator> - - <t:titlepage-before t:side=3D"recto"> - </t:titlepage-before> - - <t:titlepage-before t:side=3D"verso"> - </t:titlepage-before> - </t:titlepage> - - <t:titlepage t:element=3D"list.of.equations" t:wrapper=3D"fo:block"> - <t:titlepage-content t:side=3D"recto"> - <title - t:force=3D"1" - t:named-template=3D"gentext" - param:key=3D"'ListofEquations'" - space-before.minimum=3D"1em" - space-before.optimum=3D"1.5em" - space-before.maximum=3D"2em" - space-after=3D"0.5em" - margin-left=3D"{$title.margin.left}" - font-size=3D"&hsize3;" - font-weight=3D"bold" - font-family=3D"{$title.fontset}"/> - </t:titlepage-content> - - <t:titlepage-content t:side=3D"verso"> - </t:titlepage-content> - - <t:titlepage-separator> - </t:titlepage-separator> - - <t:titlepage-before t:side=3D"recto"> - </t:titlepage-before> - - <t:titlepage-before t:side=3D"verso"> - </t:titlepage-before> - </t:titlepage> - - <t:titlepage t:element=3D"list.of.procedures" t:wrapper=3D"fo:block"> - <t:titlepage-content t:side=3D"recto"> - <title - t:force=3D"1" - t:named-template=3D"gentext" - param:key=3D"'ListofProcedures'" - space-before.minimum=3D"1em" - space-before.optimum=3D"1.5em" - space-before.maximum=3D"2em" - space-after=3D"0.5em" - margin-left=3D"{$title.margin.left}" - font-size=3D"&hsize3;" - font-weight=3D"bold" - font-family=3D"{$title.fontset}"/> - </t:titlepage-content> - - <t:titlepage-content t:side=3D"verso"> - </t:titlepage-content> - - <t:titlepage-separator> - </t:titlepage-separator> - - <t:titlepage-before t:side=3D"recto"> - </t:titlepage-before> - - <t:titlepage-before t:side=3D"verso"> - </t:titlepage-before> - </t:titlepage> - - <t:titlepage t:element=3D"list.of.unknowns" t:wrapper=3D"fo:block"> - <t:titlepage-content t:side=3D"recto"> - <title - t:force=3D"1" - t:named-template=3D"gentext" - param:key=3D"'ListofUnknown'" - space-before.minimum=3D"1em" - space-before.optimum=3D"1.5em" - space-before.maximum=3D"2em" - space-after=3D"0.5em" - margin-left=3D"{$title.margin.left}" - font-size=3D"&hsize3;" - font-weight=3D"bold" - font-family=3D"{$title.fontset}"/> - </t:titlepage-content> - - <t:titlepage-content t:side=3D"verso"> - </t:titlepage-content> - - <t:titlepage-separator> - </t:titlepage-separator> - - <t:titlepage-before t:side=3D"recto"> - </t:titlepage-before> - - <t:titlepage-before t:side=3D"verso"> - </t:titlepage-before> - </t:titlepage> - -<!-- =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --> - -</t:templates> diff --git a/bitbake/doc/tools/docbook-to-pdf b/bitbake/doc/tools/docbook-t= o-pdf deleted file mode 100755 index 558ded9e..00000000 --- a/bitbake/doc/tools/docbook-to-pdf +++ /dev/null @@ -1,51 +0,0 @@ -#!/bin/sh -=09 -if [ -z "$1" -o -z "$2" ]; then - echo "usage: [-v] $0 <docbook file> <templatedir>" - echo - echo "*NOTE* you need xsltproc, fop and nwalsh docbook stylesheets"=20 - echo " installed for this to work!" - echo - exit 0 -fi - -FO=3D`echo $1 | sed s/.xml/.fo/` || exit 1 -PDF=3D`echo $1 | sed s/.xml/.pdf/` || exit 1 -TEMPLATEDIR=3D$2 - -## -# These URI should be rewritten by your distribution's xml catalog to -# match your localy installed XSL stylesheets. -XSL_BASE_URI=3D"http://docbook.sourceforge.net/release/xsl/current" - -# Creates a temporary XSL stylesheet based on titlepage.xsl -xsltproc -o /tmp/titlepage.xsl \ - --xinclude \ - $XSL_BASE_URI/template/titlepage.xsl \ - $TEMPLATEDIR/titlepage.templates.xml || exit 1 - -# Creates the file needed for FOP -xsltproc --xinclude \ - --stringparam hyphenate false \ - --stringparam formal.title.placement "figure after" \ - --stringparam ulink.show 1 \ - --stringparam body.font.master 9 \ - --stringparam title.font.master 11 \ - --stringparam draft.watermark.image "$TEMPLATEDIR/draft.png" \ - --stringparam chapter.autolabel 1 \ - --stringparam appendix.autolabel A \ - --stringparam section.autolabel 1 \ - --stringparam section.label.includes.component.label 1 \ - --output $FO \ - $TEMPLATEDIR/db-pdf.xsl \ - $1 || exit 1 - -# Invokes the Java version of FOP. Uses the additional configuration file= common/fop-config.xml -fop -c $TEMPLATEDIR/fop-config.xml -fo $FO -pdf $PDF || exit 1 - -rm -f $FO -rm -f /tmp/titlepage.xsl - -echo -echo " #### Success! $PDF ready. ####" -echo diff --git a/bitbake/lib/bb/COW.py b/bitbake/lib/bb/COW.py index bc20ce38..23c22b65 100644 --- a/bitbake/lib/bb/COW.py +++ b/bitbake/lib/bb/COW.py @@ -3,13 +3,14 @@ # # Copyright (C) 2006 Tim Ansell # -#Please Note: +# Please Note: # Be careful when using mutable types (ie Dict and Lists) - operations inv= olving these are SLOW. # Assign a file to __warn__ to get warnings about slow operations. # =20 =20 import copy + ImmutableTypes =3D ( bool, complex, @@ -22,9 +23,11 @@ ImmutableTypes =3D ( =20 MUTABLE =3D "__mutable__" =20 + class COWMeta(type): pass =20 + class COWDictMeta(COWMeta): __warn__ =3D False __hasmutable__ =3D False @@ -33,12 +36,15 @@ class COWDictMeta(COWMeta): def __str__(cls): # FIXME: I have magic numbers! return "<COWDict Level: %i Current Keys: %i>" % (cls.__count__, le= n(cls.__dict__) - 3) + __repr__ =3D __str__ =20 def cow(cls): class C(cls): __count__ =3D cls.__count__ + 1 + return C + copy =3D cow __call__ =3D cow =20 @@ -70,8 +76,9 @@ class COWDictMeta(COWMeta): return value =20 __getmarker__ =3D [] + def __getreadonly__(cls, key, default=3D__getmarker__): - """\ + """ Get a value (even if mutable) which you promise not to change. """ return cls.__getitem__(key, default, True) @@ -138,24 +145,29 @@ class COWDictMeta(COWMeta): =20 def iterkeys(cls): return cls.iter("keys") + def itervalues(cls, readonly=3DFalse): if not cls.__warn__ is False and cls.__hasmutable__ and readonly i= s False: - print("Warning: If you arn't going to change any of the values= call with True.", file=3Dcls.__warn__) + print("Warning: If you aren't going to change any of the value= s call with True.", file=3Dcls.__warn__) return cls.iter("values", readonly) + def iteritems(cls, readonly=3DFalse): if not cls.__warn__ is False and cls.__hasmutable__ and readonly i= s False: - print("Warning: If you arn't going to change any of the values= call with True.", file=3Dcls.__warn__) + print("Warning: If you aren't going to change any of the value= s call with True.", file=3Dcls.__warn__) return cls.iter("items", readonly) =20 + class COWSetMeta(COWDictMeta): def __str__(cls): # FIXME: I have magic numbers! - return "<COWSet Level: %i Current Keys: %i>" % (cls.__count__, len= (cls.__dict__) -3) + return "<COWSet Level: %i Current Keys: %i>" % (cls.__count__, len= (cls.__dict__) - 3) + __repr__ =3D __str__ =20 def cow(cls): class C(cls): __count__ =3D cls.__count__ + 1 + return C =20 def add(cls, value): @@ -173,131 +185,11 @@ class COWSetMeta(COWDictMeta): def iteritems(cls): raise TypeError("sets don't have 'items'") =20 + # These are the actual classes you use! -class COWDictBase(object, metaclass =3D COWDictMeta): +class COWDictBase(metaclass=3DCOWDictMeta): __count__ =3D 0 =20 -class COWSetBase(object, metaclass =3D COWSetMeta): - __count__ =3D 0 =20 -if __name__ =3D=3D "__main__": - import sys - COWDictBase.__warn__ =3D sys.stderr - a =3D COWDictBase() - print("a", a) - - a['a'] =3D 'a' - a['b'] =3D 'b' - a['dict'] =3D {} - - b =3D a.copy() - print("b", b) - b['c'] =3D 'b' - - print() - - print("a", a) - for x in a.iteritems(): - print(x) - print("--") - print("b", b) - for x in b.iteritems(): - print(x) - print() - - b['dict']['a'] =3D 'b' - b['a'] =3D 'c' - - print("a", a) - for x in a.iteritems(): - print(x) - print("--") - print("b", b) - for x in b.iteritems(): - print(x) - print() - - try: - b['dict2'] - except KeyError as e: - print("Okay!") - - a['set'] =3D COWSetBase() - a['set'].add("o1") - a['set'].add("o1") - a['set'].add("o2") - - print("a", a) - for x in a['set'].itervalues(): - print(x) - print("--") - print("b", b) - for x in b['set'].itervalues(): - print(x) - print() - - b['set'].add('o3') - - print("a", a) - for x in a['set'].itervalues(): - print(x) - print("--") - print("b", b) - for x in b['set'].itervalues(): - print(x) - print() - - a['set2'] =3D set() - a['set2'].add("o1") - a['set2'].add("o1") - a['set2'].add("o2") - - print("a", a) - for x in a.iteritems(): - print(x) - print("--") - print("b", b) - for x in b.iteritems(readonly=3DTrue): - print(x) - print() - - del b['b'] - try: - print(b['b']) - except KeyError: - print("Yay! deleted key raises error") - - if 'b' in b: - print("Boo!") - else: - print("Yay - has_key with delete works!") - - print("a", a) - for x in a.iteritems(): - print(x) - print("--") - print("b", b) - for x in b.iteritems(readonly=3DTrue): - print(x) - print() - - b.__revertitem__('b') - - print("a", a) - for x in a.iteritems(): - print(x) - print("--") - print("b", b) - for x in b.iteritems(readonly=3DTrue): - print(x) - print() - - b.__revertitem__('dict') - print("a", a) - for x in a.iteritems(): - print(x) - print("--") - print("b", b) - for x in b.iteritems(readonly=3DTrue): - print(x) - print() +class COWSetBase(metaclass=3DCOWSetMeta): + __count__ =3D 0 diff --git a/bitbake/lib/bb/__init__.py b/bitbake/lib/bb/__init__.py index b96466e6..9f61dae1 100644 --- a/bitbake/lib/bb/__init__.py +++ b/bitbake/lib/bb/__init__.py @@ -9,7 +9,7 @@ # SPDX-License-Identifier: GPL-2.0-only # =20 -__version__ =3D "1.46.0" +__version__ =3D "1.50.0" =20 import sys if sys.version_info < (3, 5, 0): @@ -21,8 +21,8 @@ class BBHandledException(Exception): The big dilemma for generic bitbake code is what information to give t= he user when an exception occurs. Any exception inheriting this base exception= class has already provided information to the user via some 'fired' message = type such as - an explicitly fired event using bb.fire, or a bb.error message. If bit= bake=20 - encounters an exception derived from this class, no backtrace or other= information=20 + an explicitly fired event using bb.fire, or a bb.error message. If bit= bake + encounters an exception derived from this class, no backtrace or other= information will be given to the user, its assumed the earlier event provided the = relevant information. """ pass @@ -35,19 +35,30 @@ class NullHandler(logging.Handler): def emit(self, record): pass =20 -Logger =3D logging.getLoggerClass() -class BBLogger(Logger): - def __init__(self, name): +class BBLoggerMixin(object): + def __init__(self, *args, **kwargs): + # Does nothing to allow calling super() from derived classes + pass + + def setup_bblogger(self, name): if name.split(".")[0] =3D=3D "BitBake": - self.debug =3D self.bbdebug - Logger.__init__(self, name) + self.debug =3D self._debug_helper + + def _debug_helper(self, *args, **kwargs): + return self.bbdebug(1, *args, **kwargs) + + def debug2(self, *args, **kwargs): + return self.bbdebug(2, *args, **kwargs) + + def debug3(self, *args, **kwargs): + return self.bbdebug(3, *args, **kwargs) =20 def bbdebug(self, level, msg, *args, **kwargs): loglevel =3D logging.DEBUG - level + 1 if not bb.event.worker_pid: if self.name in bb.msg.loggerDefaultDomains and loglevel > (bb= .msg.loggerDefaultDomains[self.name]): return - if loglevel > bb.msg.loggerDefaultLogLevel: + if loglevel < bb.msg.loggerDefaultLogLevel: return return self.log(loglevel, msg, *args, **kwargs) =20 @@ -60,16 +71,56 @@ class BBLogger(Logger): def verbnote(self, msg, *args, **kwargs): return self.log(logging.INFO + 2, msg, *args, **kwargs) =20 +Logger =3D logging.getLoggerClass() +class BBLogger(Logger, BBLoggerMixin): + def __init__(self, name, *args, **kwargs): + self.setup_bblogger(name) + super().__init__(name, *args, **kwargs) =20 logging.raiseExceptions =3D False logging.setLoggerClass(BBLogger) =20 +class BBLoggerAdapter(logging.LoggerAdapter, BBLoggerMixin): + def __init__(self, logger, *args, **kwargs): + self.setup_bblogger(logger.name) + super().__init__(logger, *args, **kwargs) + + if sys.version_info < (3, 6): + # These properties were added in Python 3.6. Add them in older ver= sions + # for compatibility + @property + def manager(self): + return self.logger.manager + + @manager.setter + def manager(self, value): + self.logger.manager =3D value + + @property + def name(self): + return self.logger.name + + def __repr__(self): + logger =3D self.logger + level =3D logger.getLevelName(logger.getEffectiveLevel()) + return '<%s %s (%s)>' % (self.__class__.__name__, logger.name,= level) + +logging.LoggerAdapter =3D BBLoggerAdapter + logger =3D logging.getLogger("BitBake") logger.addHandler(NullHandler()) logger.setLevel(logging.DEBUG - 2) =20 mainlogger =3D logging.getLogger("BitBake.Main") =20 +class PrefixLoggerAdapter(logging.LoggerAdapter): + def __init__(self, prefix, logger): + super().__init__(logger, {}) + self.__msg_prefix =3D prefix + + def process(self, msg, kwargs): + return "%s%s" %(self.__msg_prefix, msg), kwargs + # This has to be imported after the setLoggerClass, as the import of bb.msg # can result in construction of the various loggers. import bb.msg @@ -86,7 +137,7 @@ def debug(lvl, *args): mainlogger.warning("Passed invalid debug level '%s' to bb.debug", = lvl) args =3D (lvl,) + args lvl =3D 1 - mainlogger.debug(lvl, ''.join(args)) + mainlogger.bbdebug(lvl, ''.join(args)) =20 def note(*args): mainlogger.info(''.join(args)) diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py index 23b6ee45..163572be 100644 --- a/bitbake/lib/bb/build.py +++ b/bitbake/lib/bb/build.py @@ -16,7 +16,9 @@ import os import sys import logging import glob +import itertools import time +import re import stat import bb import bb.msg @@ -27,6 +29,9 @@ from bb import data, event, utils bblogger =3D logging.getLogger('BitBake') logger =3D logging.getLogger('BitBake.Build') =20 +verboseShellLogging =3D False +verboseStdoutLogging =3D False + __mtime_cache =3D {} =20 def cached_mtime_noerror(f): @@ -293,6 +298,10 @@ def exec_func_python(func, d, runfile, cwd=3DNone): comp =3D utils.better_compile(code, func, "exec_python_func() auto= generated") utils.better_exec(comp, {"d": d}, code, "exec_python_func() autoge= nerated") finally: + # We want any stdout/stderr to be printed before any other log mes= sages to make debugging + # more accurate. In some cases we seem to lose stdout/stderr entir= ely in logging tests without this. + sys.stdout.flush() + sys.stderr.flush() bb.debug(2, "Python function %s finished" % func) =20 if cwd and olddir: @@ -303,20 +312,60 @@ def exec_func_python(func, d, runfile, cwd=3DNone): =20 def shell_trap_code(): return '''#!/bin/sh\n +__BITBAKE_LAST_LINE=3D0 + # Emit a useful diagnostic if something fails: -bb_exit_handler() { +bb_sh_exit_handler() { + ret=3D$? + if [ "$ret" !=3D 0 ]; then + echo "WARNING: exit code $ret from a shell command." + fi + exit $ret +} + +bb_bash_exit_handler() { ret=3D$? - case $ret in - 0) ;; - *) case $BASH_VERSION in - "") echo "WARNING: exit code $ret from a shell command.";; - *) echo "WARNING: ${BASH_SOURCE[0]}:${BASH_LINENO[0]} exit $ret f= rom '$BASH_COMMAND'";; - esac - exit $ret - esac + { set +x; } > /dev/null + trap "" DEBUG + if [ "$ret" !=3D 0 ]; then + echo "WARNING: ${BASH_SOURCE[0]}:${__BITBAKE_LAST_LINE} exit $ret = from '$1'" + + echo "WARNING: Backtrace (BB generated script): " + for i in $(seq 1 $((${#FUNCNAME[@]} - 1))); do + if [ "$i" -eq 1 ]; then + echo -e "\t#$((i)): ${FUNCNAME[$i]}, ${BASH_SOURCE[$((i-1)= )]}, line ${__BITBAKE_LAST_LINE}" + else + echo -e "\t#$((i)): ${FUNCNAME[$i]}, ${BASH_SOURCE[$((i-1)= )]}, line ${BASH_LINENO[$((i-1))]}" + fi + done + fi + exit $ret } -trap 'bb_exit_handler' 0 -set -e + +bb_bash_debug_handler() { + local line=3D${BASH_LINENO[0]} + # For some reason the DEBUG trap trips with lineno=3D1 when scripts ex= it; ignore it + if [ "$line" -eq 1 ]; then + return + fi + + # Track the line number of commands as they execute. This is so we can= have access to the failing line number + # in the EXIT trap. See http://gnu-bash.2382.n7.nabble.com/trap-echo-q= uot-trap-exit-on-LINENO-quot-EXIT-gt-wrong-linenumber-td3666.html + if [ "${FUNCNAME[1]}" !=3D "bb_bash_exit_handler" ]; then + __BITBAKE_LAST_LINE=3D$line + fi +} + +case $BASH_VERSION in +"") trap 'bb_sh_exit_handler' 0 + set -e + ;; +*) trap 'bb_bash_exit_handler "$BASH_COMMAND"' 0 + trap '{ bb_bash_debug_handler; } 2>/dev/null' DEBUG + set -e + shopt -s extdebug + ;; +esac ''' =20 def create_progress_handler(func, progress, logfile, d): @@ -346,7 +395,7 @@ def create_progress_handler(func, progress, logfile, d): cls_obj =3D functools.reduce(resolve, cls.split("."), bb.utils= ._context) if not cls_obj: # Fall-back on __builtins__ - cls_obj =3D functools.reduce(lambda x, y: x.get(y), cls.sp= lit("."), __builtins__) + cls_obj =3D functools.reduce(resolve, cls.split("."), __bu= iltins__) if cls_obj: return cls_obj(d, outfile=3Dlogfile, otherargs=3Dotherargs) bb.warn('%s: unknown custom progress handler in task progress = varflag value "%s", ignoring' % (func, cls)) @@ -371,7 +420,7 @@ def exec_func_shell(func, d, runfile, cwd=3DNone): =20 bb.data.emit_func(func, script, d) =20 - if bb.msg.loggerVerboseLogs: + if verboseShellLogging or bb.utils.to_boolean(d.getVar("BB_VERBOSE= _LOGS", False)): script.write("set -x\n") if cwd: script.write("cd '%s'\n" % cwd) @@ -391,14 +440,20 @@ exit $ret if fakerootcmd: cmd =3D [fakerootcmd, runfile] =20 - if bb.msg.loggerDefaultVerbose: + if verboseStdoutLogging: logfile =3D LogTee(logger, StdoutNoopContextManager()) else: logfile =3D StdoutNoopContextManager() =20 progress =3D d.getVarFlag(func, 'progress') if progress: - logfile =3D create_progress_handler(func, progress, logfile, d) + try: + logfile =3D create_progress_handler(func, progress, logfile, d) + except: + from traceback import format_exc + logger.error("Failed to create progress handler") + logger.error(format_exc()) + raise =20 fifobuffer =3D bytearray() def readfifo(data): @@ -450,6 +505,62 @@ exit $ret bb.debug(2, "Executing shell function %s" % func) with open(os.devnull, 'r+') as stdin, logfile: bb.process.run(cmd, shell=3DFalse, stdin=3Dstdin, log=3Dlo= gfile, extrafiles=3D[(fifo,readfifo)]) + except bb.process.ExecutionError as exe: + # Find the backtrace that the shell trap generated + backtrace_marker_regex =3D re.compile(r"WARNING: Backtrace \(B= B generated script\)") + stdout_lines =3D (exe.stdout or "").split("\n") + backtrace_start_line =3D None + for i, line in enumerate(reversed(stdout_lines)): + if backtrace_marker_regex.search(line): + backtrace_start_line =3D len(stdout_lines) - i + break + + # Read the backtrace frames, starting at the location we just = found + backtrace_entry_regex =3D re.compile(r"#(?P<frameno>\d+): (?P<= funcname>[^\s]+), (?P<file>.+?), line (" + r"?P<lineno>\d+)") + backtrace_frames =3D [] + if backtrace_start_line: + for line in itertools.islice(stdout_lines, backtrace_start= _line, None): + match =3D backtrace_entry_regex.search(line) + if match: + backtrace_frames.append(match.groupdict()) + + with open(runfile, "r") as script: + script_lines =3D [line.rstrip() for line in script.readlin= es()] + + # For each backtrace frame, search backwards in the script (fr= om the line number called out by the frame), + # to find the comment that emit_vars injected when it wrote th= e script. This will give us the metadata + # filename (e.g. .bb or .bbclass) and line number where the sh= ell function was originally defined. + script_metadata_comment_regex =3D re.compile(r"# line: (?P<lin= eno>\d+), file: (?P<file>.+)") + better_frames =3D [] + # Skip the very last frame since it's just the call to the she= ll task in the body of the script + for frame in backtrace_frames[:-1]: + # Check whether the frame corresponds to a function define= d in the script vs external script. + if os.path.samefile(frame["file"], runfile): + # Search backwards from the frame lineno to locate the= comment that BB injected + i =3D int(frame["lineno"]) - 1 + while i >=3D 0: + match =3D script_metadata_comment_regex.match(scri= pt_lines[i]) + if match: + # Calculate the relative line in the function = itself + relative_line_in_function =3D int(frame["linen= o"]) - i - 2 + # Calculate line in the function as declared i= n the metadata + metadata_function_line =3D relative_line_in_fu= nction + int(match["lineno"]) + better_frames.append("#{frameno}: {funcname}, = {file}, line {lineno}".format( + frameno=3Dframe["frameno"], + funcname=3Dframe["funcname"], + file=3Dmatch["file"], + lineno=3Dmetadata_function_line + )) + break + i -=3D 1 + else: + better_frames.append("#{frameno}: {funcname}, {file}, = line {lineno}".format(**frame)) + + if better_frames: + better_frames =3D ("\t{0}".format(frame) for frame in bett= er_frames) + exe.extra_message =3D "\nBacktrace (metadata-relative loca= tions):\n{0}".format("\n".join(better_frames)) + raise finally: os.unlink(fifopath) =20 @@ -476,7 +587,7 @@ def _exec_task(fn, task, d, quieterr): logger.error("No such task: %s" % task) return 1 =20 - logger.debug(1, "Executing task %s", task) + logger.debug("Executing task %s", task) =20 localdata =3D _task_data(fn, task, d) tempdir =3D localdata.getVar('T') @@ -489,7 +600,7 @@ def _exec_task(fn, task, d, quieterr): curnice =3D os.nice(0) nice =3D int(nice) - curnice newnice =3D os.nice(nice) - logger.debug(1, "Renice to %s " % newnice) + logger.debug("Renice to %s " % newnice) ionice =3D localdata.getVar("BB_TASK_IONICE_LEVEL") if ionice: try: @@ -587,12 +698,16 @@ def _exec_task(fn, task, d, quieterr): except bb.BBHandledException: event.fire(TaskFailed(task, fn, logfn, localdata, True), local= data) return 1 - except Exception as exc: + except (Exception, SystemExit) as exc: if quieterr: event.fire(TaskFailedSilent(task, fn, logfn, localdata), l= ocaldata) else: errprinted =3D errchk.triggered - logger.error(str(exc)) + # If the output is already on stdout, we've printed the in= formation in the + # logs once already so don't duplicate + if verboseStdoutLogging: + errprinted =3D True + logger.error(repr(exc)) event.fire(TaskFailed(task, fn, logfn, localdata, errprint= ed), localdata) return 1 finally: @@ -613,7 +728,7 @@ def _exec_task(fn, task, d, quieterr): =20 logfile.close() if os.path.exists(logfn) and os.path.getsize(logfn) =3D=3D 0: - logger.debug(2, "Zero size logfn %s, removing", logfn) + logger.debug2("Zero size logfn %s, removing", logfn) bb.utils.remove(logfn) bb.utils.remove(loglink) event.fire(TaskSucceeded(task, fn, logfn, localdata), localdata) @@ -747,6 +862,23 @@ def make_stamp(task, d, file_name =3D None): file_name =3D d.getVar('BB_FILENAME') bb.parse.siggen.dump_sigtask(file_name, task, stampbase, True) =20 +def find_stale_stamps(task, d, file_name=3DNone): + current =3D stamp_internal(task, d, file_name) + current2 =3D stamp_internal(task + "_setscene", d, file_name) + cleanmask =3D stamp_cleanmask_internal(task, d, file_name) + found =3D [] + for mask in cleanmask: + for name in glob.glob(mask): + if "sigdata" in name or "sigbasedata" in name: + continue + if name.endswith('.taint'): + continue + if name =3D=3D current or name =3D=3D current2: + continue + logger.debug2("Stampfile %s does not match %s or %s" % (name, = current, current2)) + found.append(name) + return found + def del_stamp(task, d, file_name =3D None): """ Removes a stamp for a given task @@ -901,6 +1033,8 @@ def tasksbetween(task_start, task_end, d): def follow_chain(task, endtask, chain=3DNone): if not chain: chain =3D [] + if task in chain: + bb.fatal("Circular task dependencies as %s depends on itself v= ia the chain %s" % (task, " -> ".join(chain))) chain.append(task) for othertask in tasks: if othertask =3D=3D task: diff --git a/bitbake/lib/bb/cache.py b/bitbake/lib/bb/cache.py index d1be8361..27eb2717 100644 --- a/bitbake/lib/bb/cache.py +++ b/bitbake/lib/bb/cache.py @@ -19,16 +19,20 @@ import os import logging import pickle -from collections import defaultdict +from collections import defaultdict, Mapping import bb.utils +from bb import PrefixLoggerAdapter import re =20 logger =3D logging.getLogger("BitBake.Cache") =20 -__cache_version__ =3D "152" +__cache_version__ =3D "154" =20 -def getCacheFile(path, filename, data_hash): - return os.path.join(path, filename + "." + data_hash) +def getCacheFile(path, filename, mc, data_hash): + mcspec =3D '' + if mc: + mcspec =3D ".%s" % mc + return os.path.join(path, filename + mcspec + "." + data_hash) =20 # RecipeInfoCommon defines common data retrieving methods # from meta data for caches. CoreRecipeInfo as well as other @@ -90,6 +94,7 @@ class CoreRecipeInfo(RecipeInfoCommon): if not self.packages: self.packages.append(self.pn) self.packages_dynamic =3D self.listvar('PACKAGES_DYNAMIC', metadat= a) + self.rprovides_pkg =3D self.pkgvar('RPROVIDES', self.packages, met= adata) =20 self.skipreason =3D self.getvar('__SKIPPED', metadata) if self.skipreason: @@ -116,12 +121,12 @@ class CoreRecipeInfo(RecipeInfoCommon): self.depends =3D self.depvar('DEPENDS', metadata) self.rdepends =3D self.depvar('RDEPENDS', metadata) self.rrecommends =3D self.depvar('RRECOMMENDS', metadata) - self.rprovides_pkg =3D self.pkgvar('RPROVIDES', self.packages, = metadata) self.rdepends_pkg =3D self.pkgvar('RDEPENDS', self.packages, m= etadata) self.rrecommends_pkg =3D self.pkgvar('RRECOMMENDS', self.packages= , metadata) self.inherits =3D self.getvar('__inherit_cache', metadata,= expand=3DFalse) self.fakerootenv =3D self.getvar('FAKEROOTENV', metadata) self.fakerootdirs =3D self.getvar('FAKEROOTDIRS', metadata) + self.fakerootlogs =3D self.getvar('FAKEROOTLOGS', metadata) self.fakerootnoenv =3D self.getvar('FAKEROOTNOENV', metadata) self.extradepsfunc =3D self.getvar('calculate_extra_depends', m= etadata) =20 @@ -159,6 +164,7 @@ class CoreRecipeInfo(RecipeInfoCommon): cachedata.fakerootenv =3D {} cachedata.fakerootnoenv =3D {} cachedata.fakerootdirs =3D {} + cachedata.fakerootlogs =3D {} cachedata.extradepsfunc =3D {} =20 def add_cacheData(self, cachedata, fn): @@ -211,7 +217,7 @@ class CoreRecipeInfo(RecipeInfoCommon): if not self.not_world: cachedata.possible_world.append(fn) #else: - # logger.debug(2, "EXCLUDE FROM WORLD: %s", fn) + # logger.debug2("EXCLUDE FROM WORLD: %s", fn) =20 # create a collection of all targets for sanity checking # tasks, such as upstream versions, license, and tools for @@ -227,6 +233,7 @@ class CoreRecipeInfo(RecipeInfoCommon): cachedata.fakerootenv[fn] =3D self.fakerootenv cachedata.fakerootnoenv[fn] =3D self.fakerootnoenv cachedata.fakerootdirs[fn] =3D self.fakerootdirs + cachedata.fakerootlogs[fn] =3D self.fakerootlogs cachedata.extradepsfunc[fn] =3D self.extradepsfunc =20 def virtualfn2realfn(virtualfn): @@ -234,7 +241,7 @@ def virtualfn2realfn(virtualfn): Convert a virtual file name to a real one + the associated subclass ke= yword """ mc =3D "" - if virtualfn.startswith('mc:'): + if virtualfn.startswith('mc:') and virtualfn.count(':') >=3D 2: elems =3D virtualfn.split(':') mc =3D elems[1] virtualfn =3D ":".join(elems[2:]) @@ -264,7 +271,7 @@ def variant2virtual(realfn, variant): """ if variant =3D=3D "": return realfn - if variant.startswith("mc:"): + if variant.startswith("mc:") and variant.count(':') >=3D 2: elems =3D variant.split(":") if elems[2]: return "mc:" + elems[1] + ":virtual:" + ":".join(elems[2:]) + = ":" + realfn @@ -319,12 +326,12 @@ class NoCache(object): Return a complete set of data for fn. To do this, we need to parse the file. """ - logger.debug(1, "Parsing %s (full)" % virtualfn) + logger.debug("Parsing %s (full)" % virtualfn) (fn, virtual, mc) =3D virtualfn2realfn(virtualfn) bb_data =3D self.load_bbfile(virtualfn, appends, virtonly=3DTrue) return bb_data[virtual] =20 - def load_bbfile(self, bbfile, appends, virtonly =3D False): + def load_bbfile(self, bbfile, appends, virtonly =3D False, mc=3DNone): """ Load and parse one .bb build file Return the data and whether parsing resulted in the file being ski= pped @@ -337,6 +344,10 @@ class NoCache(object): datastores =3D parse_recipe(bb_data, bbfile, appends, mc) return datastores =20 + if mc is not None: + bb_data =3D self.databuilder.mcdata[mc].createCopy() + return parse_recipe(bb_data, bbfile, appends, mc) + bb_data =3D self.data.createCopy() datastores =3D parse_recipe(bb_data, bbfile, appends) =20 @@ -354,14 +365,15 @@ class Cache(NoCache): """ BitBake Cache implementation """ - - def __init__(self, databuilder, data_hash, caches_array): + def __init__(self, databuilder, mc, data_hash, caches_array): super().__init__(databuilder) data =3D databuilder.data =20 # Pass caches_array information into Cache Constructor # It will be used later for deciding whether we # need extra cache file dump/load support + self.mc =3D mc + self.logger =3D PrefixLoggerAdapter("Cache: %s: " % (mc if mc else= "default"), logger) self.caches_array =3D caches_array self.cachedir =3D data.getVar("CACHE") self.clean =3D set() @@ -374,31 +386,47 @@ class Cache(NoCache): =20 if self.cachedir in [None, '']: self.has_cache =3D False - logger.info("Not using a cache. " - "Set CACHE =3D <directory> to enable.") + self.logger.info("Not using a cache. " + "Set CACHE =3D <directory> to enable.") return =20 self.has_cache =3D True - self.cachefile =3D getCacheFile(self.cachedir, "bb_cache.dat", sel= f.data_hash) =20 - logger.debug(1, "Cache dir: %s", self.cachedir) + def getCacheFile(self, cachefile): + return getCacheFile(self.cachedir, cachefile, self.mc, self.data_h= ash) + + def prepare_cache(self, progress): + if not self.has_cache: + return 0 + + loaded =3D 0 + + self.cachefile =3D self.getCacheFile("bb_cache.dat") + + self.logger.debug("Cache dir: %s", self.cachedir) bb.utils.mkdirhier(self.cachedir) =20 cache_ok =3D True if self.caches_array: for cache_class in self.caches_array: - cachefile =3D getCacheFile(self.cachedir, cache_class.cach= efile, self.data_hash) - cache_ok =3D cache_ok and os.path.exists(cachefile) + cachefile =3D self.getCacheFile(cache_class.cachefile) + cache_exists =3D os.path.exists(cachefile) + self.logger.debug2("Checking if %s exists: %r", cachefile,= cache_exists) + cache_ok =3D cache_ok and cache_exists cache_class.init_cacheData(self) if cache_ok: - self.load_cachefile() + loaded =3D self.load_cachefile(progress) elif os.path.isfile(self.cachefile): - logger.info("Out of date cache found, rebuilding...") + self.logger.info("Out of date cache found, rebuilding...") else: - logger.debug(1, "Cache file %s not found, building..." % self.= cachefile) + self.logger.debug("Cache file %s not found, building..." % sel= f.cachefile) =20 # We don't use the symlink, its just for debugging convinience - symlink =3D os.path.join(self.cachedir, "bb_cache.dat") + if self.mc: + symlink =3D os.path.join(self.cachedir, "bb_cache.dat.%s" % se= lf.mc) + else: + symlink =3D os.path.join(self.cachedir, "bb_cache.dat") + if os.path.exists(symlink): bb.utils.remove(symlink) try: @@ -406,22 +434,29 @@ class Cache(NoCache): except OSError: pass =20 - def load_cachefile(self): - cachesize =3D 0 - previous_progress =3D 0 - previous_percent =3D 0 + return loaded =20 - # Calculate the correct cachesize of all those cache files + def cachesize(self): + if not self.has_cache: + return 0 + + cachesize =3D 0 for cache_class in self.caches_array: - cachefile =3D getCacheFile(self.cachedir, cache_class.cachefil= e, self.data_hash) - with open(cachefile, "rb") as cachefile: - cachesize +=3D os.fstat(cachefile.fileno()).st_size + cachefile =3D self.getCacheFile(cache_class.cachefile) + try: + with open(cachefile, "rb") as cachefile: + cachesize +=3D os.fstat(cachefile.fileno()).st_size + except FileNotFoundError: + pass =20 - bb.event.fire(bb.event.CacheLoadStarted(cachesize), self.data) + return cachesize + + def load_cachefile(self, progress): + previous_progress =3D 0 =20 for cache_class in self.caches_array: - cachefile =3D getCacheFile(self.cachedir, cache_class.cachefil= e, self.data_hash) - logger.debug(1, 'Loading cache file: %s' % cachefile) + cachefile =3D self.getCacheFile(cache_class.cachefile) + self.logger.debug('Loading cache file: %s' % cachefile) with open(cachefile, "rb") as cachefile: pickled =3D pickle.Unpickler(cachefile) # Check cache version information @@ -429,15 +464,15 @@ class Cache(NoCache): cache_ver =3D pickled.load() bitbake_ver =3D pickled.load() except Exception: - logger.info('Invalid cache, rebuilding...') - return + self.logger.info('Invalid cache, rebuilding...') + return 0 =20 if cache_ver !=3D __cache_version__: - logger.info('Cache version mismatch, rebuilding...') - return + self.logger.info('Cache version mismatch, rebuilding..= .') + return 0 elif bitbake_ver !=3D bb.__version__: - logger.info('Bitbake version mismatch, rebuilding...') - return + self.logger.info('Bitbake version mismatch, rebuilding= ...') + return 0 =20 # Load the rest of the cache file current_progress =3D 0 @@ -460,29 +495,17 @@ class Cache(NoCache): self.depends_cache[key] =3D [value] # only fire events on even percentage boundaries current_progress =3D cachefile.tell() + previous_progr= ess - if current_progress > cachesize: - # we might have calculated incorrect total size be= cause a file - # might've been written out just after we checked = its size - cachesize =3D current_progress - current_percent =3D 100 * current_progress / cachesize - if current_percent > previous_percent: - previous_percent =3D current_percent - bb.event.fire(bb.event.CacheLoadProgress(current_p= rogress, cachesize), - self.data) + progress(cachefile.tell() + previous_progress) =20 previous_progress +=3D current_progress =20 - # Note: depends cache number is corresponding to the parsing file = numbers. - # The same file has several caches, still regarded as one item in = the cache - bb.event.fire(bb.event.CacheLoadCompleted(cachesize, - len(self.depends_cache)), - self.data) + return len(self.depends_cache) =20 def parse(self, filename, appends): """Parse the specified filename, returning the recipe information"= "" - logger.debug(1, "Parsing %s", filename) + self.logger.debug("Parsing %s", filename) infos =3D [] - datastores =3D self.load_bbfile(filename, appends) + datastores =3D self.load_bbfile(filename, appends, mc=3Dself.mc) depends =3D [] variants =3D [] # Process the "real" fn last so we can store variants list @@ -534,7 +557,7 @@ class Cache(NoCache): cached, infos =3D self.load(fn, appends) for virtualfn, info_array in infos: if info_array[0].skipped: - logger.debug(1, "Skipping %s: %s", virtualfn, info_array[0= ].skipreason) + self.logger.debug("Skipping %s: %s", virtualfn, info_array= [0].skipreason) skipped +=3D 1 else: self.add_info(virtualfn, info_array, cacheData, not cached) @@ -570,21 +593,21 @@ class Cache(NoCache): =20 # File isn't in depends_cache if not fn in self.depends_cache: - logger.debug(2, "Cache: %s is not cached", fn) + self.logger.debug2("%s is not cached", fn) return False =20 mtime =3D bb.parse.cached_mtime_noerror(fn) =20 # Check file still exists if mtime =3D=3D 0: - logger.debug(2, "Cache: %s no longer exists", fn) + self.logger.debug2("%s no longer exists", fn) self.remove(fn) return False =20 info_array =3D self.depends_cache[fn] # Check the file's timestamp if mtime !=3D info_array[0].timestamp: - logger.debug(2, "Cache: %s changed", fn) + self.logger.debug2("%s changed", fn) self.remove(fn) return False =20 @@ -595,14 +618,14 @@ class Cache(NoCache): fmtime =3D bb.parse.cached_mtime_noerror(f) # Check if file still exists if old_mtime !=3D 0 and fmtime =3D=3D 0: - logger.debug(2, "Cache: %s's dependency %s was removed= ", - fn, f) + self.logger.debug2("%s's dependency %s was removed", + fn, f) self.remove(fn) return False =20 if (fmtime !=3D old_mtime): - logger.debug(2, "Cache: %s's dependency %s changed", - fn, f) + self.logger.debug2("%s's dependency %s changed", + fn, f) self.remove(fn) return False =20 @@ -614,18 +637,18 @@ class Cache(NoCache): # Have to be careful about spaces and colons in filenames flist =3D self.filelist_regex.split(fl) for f in flist: - if not f or "*" in f: + if not f: continue f, exist =3D f.split(":") if (exist =3D=3D "True" and not os.path.exists(f)) or = (exist =3D=3D "False" and os.path.exists(f)): - logger.debug(2, "Cache: %s's file checksum list fi= le %s changed", - fn, f) + self.logger.debug2("%s's file checksum list file %= s changed", + fn, f) self.remove(fn) return False =20 - if appends !=3D info_array[0].appends: - logger.debug(2, "Cache: appends for %s changed", fn) - logger.debug(2, "%s to %s" % (str(appends), str(info_array[0].= appends))) + if tuple(appends) !=3D tuple(info_array[0].appends): + self.logger.debug2("appends for %s changed", fn) + self.logger.debug2("%s to %s" % (str(appends), str(info_array[= 0].appends))) self.remove(fn) return False =20 @@ -634,10 +657,10 @@ class Cache(NoCache): virtualfn =3D variant2virtual(fn, cls) self.clean.add(virtualfn) if virtualfn not in self.depends_cache: - logger.debug(2, "Cache: %s is not cached", virtualfn) + self.logger.debug2("%s is not cached", virtualfn) invalid =3D True elif len(self.depends_cache[virtualfn]) !=3D len(self.caches_a= rray): - logger.debug(2, "Cache: Extra caches missing for %s?" % vi= rtualfn) + self.logger.debug2("Extra caches missing for %s?" % virtua= lfn) invalid =3D True =20 # If any one of the variants is not present, mark as invalid for a= ll @@ -645,10 +668,10 @@ class Cache(NoCache): for cls in info_array[0].variants: virtualfn =3D variant2virtual(fn, cls) if virtualfn in self.clean: - logger.debug(2, "Cache: Removing %s from cache", virtu= alfn) + self.logger.debug2("Removing %s from cache", virtualfn) self.clean.remove(virtualfn) if fn in self.clean: - logger.debug(2, "Cache: Marking %s as not clean", fn) + self.logger.debug2("Marking %s as not clean", fn) self.clean.remove(fn) return False =20 @@ -661,10 +684,10 @@ class Cache(NoCache): Called from the parser in error cases """ if fn in self.depends_cache: - logger.debug(1, "Removing %s from cache", fn) + self.logger.debug("Removing %s from cache", fn) del self.depends_cache[fn] if fn in self.clean: - logger.debug(1, "Marking %s as unclean", fn) + self.logger.debug("Marking %s as unclean", fn) self.clean.remove(fn) =20 def sync(self): @@ -677,12 +700,13 @@ class Cache(NoCache): return =20 if self.cacheclean: - logger.debug(2, "Cache is clean, not saving.") + self.logger.debug2("Cache is clean, not saving.") return =20 for cache_class in self.caches_array: cache_class_name =3D cache_class.__name__ - cachefile =3D getCacheFile(self.cachedir, cache_class.cachefil= e, self.data_hash) + cachefile =3D self.getCacheFile(cache_class.cachefile) + self.logger.debug2("Writing %s", cachefile) with open(cachefile, "wb") as f: p =3D pickle.Pickler(f, pickle.HIGHEST_PROTOCOL) p.dump(__cache_version__) @@ -701,8 +725,18 @@ class Cache(NoCache): return bb.parse.cached_mtime_noerror(cachefile) =20 def add_info(self, filename, info_array, cacheData, parsed=3DNone, wat= cher=3DNone): + if self.mc is not None: + (fn, cls, mc) =3D virtualfn2realfn(filename) + if mc: + self.logger.error("Unexpected multiconfig %s", filename) + return + + vfn =3D realfn2virtual(fn, cls, self.mc) + else: + vfn =3D filename + if isinstance(info_array[0], CoreRecipeInfo) and (not info_array[0= ].skipped): - cacheData.add_from_recipeinfo(filename, info_array) + cacheData.add_from_recipeinfo(vfn, info_array) =20 if watcher: watcher(info_array[0].file_depends) @@ -727,6 +761,61 @@ class Cache(NoCache): info_array.append(cache_class(realfn, data)) self.add_info(file_name, info_array, cacheData, parsed) =20 +class MulticonfigCache(Mapping): + def __init__(self, databuilder, data_hash, caches_array): + def progress(p): + nonlocal current_progress + nonlocal previous_progress + nonlocal previous_percent + nonlocal cachesize + + current_progress =3D previous_progress + p + + if current_progress > cachesize: + # we might have calculated incorrect total size because a = file + # might've been written out just after we checked its size + cachesize =3D current_progress + current_percent =3D 100 * current_progress / cachesize + if current_percent > previous_percent: + previous_percent =3D current_percent + bb.event.fire(bb.event.CacheLoadProgress(current_progress,= cachesize), + databuilder.data) + + + cachesize =3D 0 + current_progress =3D 0 + previous_progress =3D 0 + previous_percent =3D 0 + self.__caches =3D {} + + for mc, mcdata in databuilder.mcdata.items(): + self.__caches[mc] =3D Cache(databuilder, mc, data_hash, caches= _array) + + cachesize +=3D self.__caches[mc].cachesize() + + bb.event.fire(bb.event.CacheLoadStarted(cachesize), databuilder.da= ta) + loaded =3D 0 + + for c in self.__caches.values(): + loaded +=3D c.prepare_cache(progress) + previous_progress =3D current_progress + + # Note: depends cache number is corresponding to the parsing file = numbers. + # The same file has several caches, still regarded as one item in = the cache + bb.event.fire(bb.event.CacheLoadCompleted(cachesize, loaded), data= builder.data) + + def __len__(self): + return len(self.__caches) + + def __getitem__(self, key): + return self.__caches[key] + + def __contains__(self, key): + return key in self.__caches + + def __iter__(self): + for k in self.__caches: + yield k =20 def init(cooker): """ @@ -793,7 +882,7 @@ class MultiProcessCache(object): bb.utils.mkdirhier(cachedir) self.cachefile =3D os.path.join(cachedir, cache_file_name or self.__class__.ca= che_file_name) - logger.debug(1, "Using cache in '%s'", self.cachefile) + logger.debug("Using cache in '%s'", self.cachefile) =20 glf =3D bb.utils.lockfile(self.cachefile + ".lock") =20 @@ -899,7 +988,7 @@ class SimpleCache(object): bb.utils.mkdirhier(cachedir) self.cachefile =3D os.path.join(cachedir, cache_file_name or self.__class__.ca= che_file_name) - logger.debug(1, "Using cache in '%s'", self.cachefile) + logger.debug("Using cache in '%s'", self.cachefile) =20 glf =3D bb.utils.lockfile(self.cachefile + ".lock") =20 diff --git a/bitbake/lib/bb/codeparser.py b/bitbake/lib/bb/codeparser.py index 25a7ac69..0cec452c 100644 --- a/bitbake/lib/bb/codeparser.py +++ b/bitbake/lib/bb/codeparser.py @@ -212,9 +212,9 @@ class PythonParser(): funcstr =3D codegen.to_source(func) argstr =3D codegen.to_source(arg) except TypeError: - self.log.debug(2, 'Failed to convert function and argument to = source form') + self.log.debug2('Failed to convert function and argument to so= urce form') else: - self.log.debug(1, self.unhandled_message % (funcstr, argstr)) + self.log.debug(self.unhandled_message % (funcstr, argstr)) =20 def visit_Call(self, node): name =3D self.called_node_name(node.func) @@ -450,7 +450,7 @@ class ShellParser(): =20 cmd =3D word[1] if cmd.startswith("$"): - self.log.debug(1, self.unhandled_template % cmd) + self.log.debug(self.unhandled_template % cmd) elif cmd =3D=3D "eval": command =3D " ".join(word for _, word in words[1:]) self._parse_shell(command) diff --git a/bitbake/lib/bb/command.py b/bitbake/lib/bb/command.py index 6abf3866..dd77cdd6 100644 --- a/bitbake/lib/bb/command.py +++ b/bitbake/lib/bb/command.py @@ -54,13 +54,20 @@ class Command: self.cooker =3D cooker self.cmds_sync =3D CommandsSync() self.cmds_async =3D CommandsAsync() - self.remotedatastores =3D bb.remotedata.RemoteDatastores(cooker) + self.remotedatastores =3D None =20 # FIXME Add lock for this self.currentAsyncCommand =3D None =20 def runCommand(self, commandline, ro_only =3D False): command =3D commandline.pop(0) + + # Ensure cooker is ready for commands + if command !=3D "updateConfig" and command !=3D "setFeatures": + self.cooker.init_configdata() + if not self.remotedatastores: + self.remotedatastores =3D bb.remotedata.RemoteDatastores(s= elf.cooker) + if hasattr(CommandsSync, command): # Can run synchronous commands straight away command_method =3D getattr(self.cmds_sync, command) @@ -74,8 +81,12 @@ class Command: result =3D command_method(self, commandline) except CommandError as exc: return None, exc.args[0] - except (Exception, SystemExit): + except (Exception, SystemExit) as exc: import traceback + if isinstance(exc, bb.BBHandledException): + # We need to start returning real exceptions here. Unt= il we do, we can't + # tell if an exception is an instance of bb.BBHandledE= xception + return None, "bb.BBHandledException()\n" + traceback.f= ormat_exc() return None, traceback.format_exc() else: return result, None @@ -84,7 +95,7 @@ class Command: if command not in CommandsAsync.__dict__: return None, "No such command" self.currentAsyncCommand =3D (command, commandline) - self.cooker.configuration.server_register_idlecallback(self.cooker= .runCommands, self.cooker) + self.cooker.idleCallBackRegister(self.cooker.runCommands, self.coo= ker) return True, None =20 def runAsyncCommand(self): @@ -136,13 +147,8 @@ class Command: self.cooker.finishcommand() =20 def reset(self): - self.remotedatastores =3D bb.remotedata.RemoteDatastores(self.cook= er) - -def split_mc_pn(pn): - if pn.startswith("multiconfig:"): - _, mc, pn =3D pn.split(":", 2) - return (mc, pn) - return ('', pn) + if self.remotedatastores: + self.remotedatastores =3D bb.remotedata.RemoteDatastores(self.c= ooker) =20 class CommandsSync: """ @@ -232,7 +238,11 @@ class CommandsSync: =20 def matchFile(self, command, params): fMatch =3D params[0] - return command.cooker.matchFile(fMatch) + try: + mc =3D params[0] + except IndexError: + mc =3D '' + return command.cooker.matchFile(fMatch, mc) matchFile.needconfig =3D False =20 def getUIHandlerNum(self, command, params): @@ -395,22 +405,38 @@ class CommandsSync: def getSkippedRecipes(self, command, params): # Return list sorted by reverse priority order import bb.cache - skipdict =3D OrderedDict(sorted(command.cooker.skiplist.items(), - key=3Dlambda x: (-command.cooker.col= lection.calc_bbfile_priority(bb.cache.virtualfn2realfn(x[0])[0]), x[0]))) + def sortkey(x): + vfn, _ =3D x + realfn, _, mc =3D bb.cache.virtualfn2realfn(vfn) + return (-command.cooker.collections[mc].calc_bbfile_priority(r= ealfn)[0], vfn) + + skipdict =3D OrderedDict(sorted(command.cooker.skiplist.items(), k= ey=3Dsortkey)) return list(skipdict.items()) getSkippedRecipes.readonly =3D True =20 def getOverlayedRecipes(self, command, params): - return list(command.cooker.collection.overlayed.items()) + try: + mc =3D params[0] + except IndexError: + mc =3D '' + return list(command.cooker.collections[mc].overlayed.items()) getOverlayedRecipes.readonly =3D True =20 def getFileAppends(self, command, params): fn =3D params[0] - return command.cooker.collection.get_file_appends(fn) + try: + mc =3D params[1] + except IndexError: + mc =3D '' + return command.cooker.collections[mc].get_file_appends(fn) getFileAppends.readonly =3D True =20 def getAllAppends(self, command, params): - return command.cooker.collection.bbappends + try: + mc =3D params[0] + except IndexError: + mc =3D '' + return command.cooker.collections[mc].bbappends getAllAppends.readonly =3D True =20 def findProviders(self, command, params): @@ -422,7 +448,7 @@ class CommandsSync: findProviders.readonly =3D True =20 def findBestProvider(self, command, params): - (mc, pn) =3D split_mc_pn(params[0]) + (mc, pn) =3D bb.runqueue.split_mc(params[0]) return command.cooker.findBestProvider(pn, mc) findBestProvider.readonly =3D True =20 @@ -496,6 +522,7 @@ class CommandsSync: for the recipe. """ fn =3D params[0] + mc =3D bb.runqueue.mc_from_tid(fn) appends =3D params[1] appendlist =3D params[2] if len(params) > 3: @@ -507,7 +534,7 @@ class CommandsSync: if appendlist is not None: appendfiles =3D appendlist else: - appendfiles =3D command.cooker.collection.get_file_appends= (fn) + appendfiles =3D command.cooker.collections[mc].get_file_ap= pends(fn) else: appendfiles =3D [] # We are calling bb.cache locally here rather than on the server, @@ -517,7 +544,7 @@ class CommandsSync: if config_data: # We have to use a different function here if we're passing in= a datastore # NOTE: we took a copy above, so we don't do it here again - envdata =3D bb.cache.parse_recipe(config_data, fn, appendfiles= )[''] + envdata =3D bb.cache.parse_recipe(config_data, fn, appendfiles= , mc)[''] else: # Use the standard path parser =3D bb.cache.NoCache(command.cooker.databuilder) @@ -708,10 +735,10 @@ class CommandsAsync: """ Find signature info files via the signature generator """ - pn =3D params[0] + (mc, pn) =3D bb.runqueue.split_mc(params[0]) taskname =3D params[1] sigs =3D params[2] - res =3D bb.siggen.find_siginfo(pn, taskname, sigs, command.cooker.= data) - bb.event.fire(bb.event.FindSigInfoResult(res), command.cooker.data) + res =3D bb.siggen.find_siginfo(pn, taskname, sigs, command.cooker.= databuilder.mcdata[mc]) + bb.event.fire(bb.event.FindSigInfoResult(res), command.cooker.data= builder.mcdata[mc]) command.finishAsyncCommand() findSigInfo.needcache =3D False diff --git a/bitbake/lib/bb/compat.py b/bitbake/lib/bb/compat.py deleted file mode 100644 index 49356681..00000000 --- a/bitbake/lib/bb/compat.py +++ /dev/null @@ -1,10 +0,0 @@ -# -# SPDX-License-Identifier: GPL-2.0-only -# - -"""Code pulled from future python versions, here for compatibility""" - -from collections import MutableMapping, KeysView, ValuesView, ItemsView, O= rderedDict -from functools import total_ordering - - diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py index d90bd394..89f1fad0 100644 --- a/bitbake/lib/bb/cooker.py +++ b/bitbake/lib/bb/cooker.py @@ -73,7 +73,9 @@ class SkippedPackage: self.pn =3D info.pn self.skipreason =3D info.skipreason self.provides =3D info.provides - self.rprovides =3D info.rprovides + self.rprovides =3D info.packages + info.rprovides + for package in info.packages: + self.rprovides +=3D info.rprovides_pkg[package] elif reason: self.skipreason =3D reason =20 @@ -148,15 +150,18 @@ class BBCooker: Manages one bitbake build run """ =20 - def __init__(self, configuration, featureSet=3DNone): + def __init__(self, featureSet=3DNone, idleCallBackRegister=3DNone): self.recipecaches =3D None + self.eventlog =3D None self.skiplist =3D {} self.featureset =3D CookerFeatures() if featureSet: for f in featureSet: self.featureset.setFeature(f) =20 - self.configuration =3D configuration + self.configuration =3D bb.cookerdata.CookerConfiguration() + + self.idleCallBackRegister =3D idleCallBackRegister =20 bb.debug(1, "BBCooker starting %s" % time.time()) sys.stdout.flush() @@ -192,25 +197,13 @@ class BBCooker: self.hashserv =3D None self.hashservaddr =3D None =20 - self.initConfigurationData() - - bb.debug(1, "BBCooker parsed base configuration %s" % time.time()) - sys.stdout.flush() - - # we log all events to a file if so directed - if self.configuration.writeeventlog: - # register the log file writer as UI Handler - writer =3D EventWriter(self, self.configuration.writeeventlog) - EventLogWriteHandler =3D namedtuple('EventLogWriteHandler', ['= event']) - bb.event.register_UIHhandler(EventLogWriteHandler(writer)) - self.inotify_modified_files =3D [] =20 def _process_inotify_updates(server, cooker, abort): cooker.process_inotify_updates() return 1.0 =20 - self.configuration.server_register_idlecallback(_process_inotify_u= pdates, self) + self.idleCallBackRegister(_process_inotify_updates, self) =20 # TOSTOP must not be set or our children will hang when they output try: @@ -237,6 +230,13 @@ class BBCooker: bb.debug(1, "BBCooker startup complete %s" % time.time()) sys.stdout.flush() =20 + def init_configdata(self): + if not hasattr(self, "data"): + self.initConfigurationData() + bb.debug(1, "BBCooker parsed base configuration %s" % time.tim= e()) + sys.stdout.flush() + self.handlePRServ() + def process_inotify_updates(self): for n in [self.confignotifier, self.notifier]: if n.check_events(timeout=3D0): @@ -322,7 +322,7 @@ class BBCooker: for feature in features: self.featureset.setFeature(feature) bb.debug(1, "Features set %s (was %s)" % (original_featureset, lis= t(self.featureset))) - if (original_featureset !=3D list(self.featureset)) and self.state= !=3D state.error: + if (original_featureset !=3D list(self.featureset)) and self.state= !=3D state.error and hasattr(self, "data"): self.reset() =20 def initConfigurationData(self): @@ -354,7 +354,7 @@ class BBCooker: self.caches_array.append(getattr(module, cache_name)) except ImportError as exc: logger.critical("Unable to import extra RecipeInfo '%s' fr= om '%s': %s" % (cache_name, module_name, exc)) - sys.exit("FATAL: Failed to import extra cache class '%s'."= % cache_name) + raise bb.BBHandledException() =20 self.databuilder =3D bb.cookerdata.CookerDataBuilder(self.configur= ation, False) self.databuilder.parseBaseConfiguration() @@ -382,14 +382,19 @@ class BBCooker: try: self.prhost =3D prserv.serv.auto_start(self.data) except prserv.serv.PRServiceConfigError as e: - bb.fatal("Unable to start PR Server, exitting") + bb.fatal("Unable to start PR Server, exiting") =20 if self.data.getVar("BB_HASHSERVE") =3D=3D "auto": # Create a new hash server bound to a unix domain socket if not self.hashserv: dbfile =3D (self.data.getVar("PERSISTENT_DIR") or self.dat= a.getVar("CACHE")) + "/hashserv.db" self.hashservaddr =3D "unix://%s/hashserve.sock" % self.da= ta.getVar("TOPDIR") - self.hashserv =3D hashserv.create_server(self.hashservaddr= , dbfile, sync=3DFalse) + self.hashserv =3D hashserv.create_server( + self.hashservaddr, + dbfile, + sync=3DFalse, + upstream=3Dself.data.getVar("BB_HASHSERVE_UPSTREAM") o= r None, + ) self.hashserv.process =3D multiprocessing.Process(target= =3Dself.hashserv.serve_forever) self.hashserv.process.start() self.data.setVar("BB_HASHSERVE", self.hashservaddr) @@ -411,10 +416,7 @@ class BBCooker: self.data.disableTracking() =20 def parseConfiguration(self): - # Set log file verbosity - verboselogs =3D bb.utils.to_boolean(self.data.getVar("BB_VERBOSE_L= OGS", False)) - if verboselogs: - bb.msg.loggerVerboseLogs =3D True + self.updateCacheSync() =20 # Change nice level if we're asked to nice =3D self.data.getVar("BB_NICE_LEVEL") @@ -446,27 +448,52 @@ class BBCooker: continue except AttributeError: pass - logger.debug(1, "Marking as dirty due to '%s' option chang= e to '%s'" % (o, options[o])) + logger.debug("Marking as dirty due to '%s' option change t= o '%s'" % (o, options[o])) print("Marking as dirty due to '%s' option change to '%s'"= % (o, options[o])) clean =3D False - setattr(self.configuration, o, options[o]) + if hasattr(self.configuration, o): + setattr(self.configuration, o, options[o]) + + if self.configuration.writeeventlog: + if self.eventlog and self.eventlog[0] !=3D self.configuration.= writeeventlog: + bb.event.unregister_UIHhandler(self.eventlog[1]) + if not self.eventlog or self.eventlog[0] !=3D self.configurati= on.writeeventlog: + # we log all events to a file if so directed + # register the log file writer as UI Handler + writer =3D EventWriter(self, self.configuration.writeevent= log) + EventLogWriteHandler =3D namedtuple('EventLogWriteHandler'= , ['event']) + self.eventlog =3D (self.configuration.writeeventlog, bb.ev= ent.register_UIHhandler(EventLogWriteHandler(writer))) + + bb.msg.loggerDefaultLogLevel =3D self.configuration.default_loglev= el + bb.msg.loggerDefaultDomains =3D self.configuration.debug_domains + + if hasattr(self, "data"): + origenv =3D bb.data.init() + for k in environment: + origenv.setVar(k, environment[k]) + self.data.setVar("BB_ORIGENV", origenv) + for k in bb.utils.approved_variables(): if k in environment and k not in self.configuration.env: - logger.debug(1, "Updating new environment variable %s to %= s" % (k, environment[k])) + logger.debug("Updating new environment variable %s to %s" = % (k, environment[k])) self.configuration.env[k] =3D environment[k] clean =3D False if k in self.configuration.env and k not in environment: - logger.debug(1, "Updating environment variable %s (deleted= )" % (k)) + logger.debug("Updating environment variable %s (deleted)" = % (k)) del self.configuration.env[k] clean =3D False if k not in self.configuration.env and k not in environment: continue if environment[k] !=3D self.configuration.env[k]: - logger.debug(1, "Updating environment variable %s from %s = to %s" % (k, self.configuration.env[k], environment[k])) + logger.debug("Updating environment variable %s from %s to = %s" % (k, self.configuration.env[k], environment[k])) self.configuration.env[k] =3D environment[k] clean =3D False + + # Now update all the variables not in the datastore to match + self.configuration.env =3D environment + if not clean: - logger.debug(1, "Base environment change, triggering reparse") + logger.debug("Base environment change, triggering reparse") self.reset() =20 def runCommands(self, server, data, abort): @@ -480,22 +507,30 @@ class BBCooker: =20 def showVersions(self): =20 - (latest_versions, preferred_versions) =3D self.findProviders() + (latest_versions, preferred_versions, required) =3D self.findProvi= ders() =20 - logger.plain("%-35s %25s %25s", "Recipe Name", "Latest Version", "= Preferred Version") - logger.plain("%-35s %25s %25s\n", "=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D", "=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D", "=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D") + logger.plain("%-35s %25s %25s %25s", "Recipe Name", "Latest Versio= n", "Preferred Version", "Required Version") + logger.plain("%-35s %25s %25s %25s\n", "=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D", "=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D", "=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D", "=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D") =20 for p in sorted(self.recipecaches[''].pkg_pn): - pref =3D preferred_versions[p] + preferred =3D preferred_versions[p] latest =3D latest_versions[p] + requiredstr =3D "" + preferredstr =3D "" + if required[p]: + if preferred[0] is not None: + requiredstr =3D preferred[0][0] + ":" + preferred[0][1= ] + '-' + preferred[0][2] + else: + bb.fatal("REQUIRED_VERSION of package %s not available= " % p) + else: + preferredstr =3D preferred[0][0] + ":" + preferred[0][1] += '-' + preferred[0][2] =20 - prefstr =3D pref[0][0] + ":" + pref[0][1] + '-' + pref[0][2] lateststr =3D latest[0][0] + ":" + latest[0][1] + "-" + latest= [0][2] =20 - if pref =3D=3D latest: - prefstr =3D "" + if preferred =3D=3D latest: + preferredstr =3D "" =20 - logger.plain("%-35s %25s %25s", p, lateststr, prefstr) + logger.plain("%-35s %25s %25s %25s", p, lateststr, preferredst= r, requiredstr) =20 def showEnvironment(self, buildfile=3DNone, pkgs_to_build=3DNone): """ @@ -525,7 +560,7 @@ class BBCooker: self.parseConfiguration() =20 fn, cls, mc =3D bb.cache.virtualfn2realfn(buildfile) - fn =3D self.matchFile(fn) + fn =3D self.matchFile(fn, mc) fn =3D bb.cache.realfn2virtual(fn, cls, mc) elif len(pkgs_to_build) =3D=3D 1: mc =3D mc_base(pkgs_to_build[0]) @@ -541,8 +576,8 @@ class BBCooker: =20 if fn: try: - bb_cache =3D bb.cache.Cache(self.databuilder, self.data_ha= sh, self.caches_array) - envdata =3D bb_cache.loadDataFull(fn, self.collection.get_= file_appends(fn)) + bb_caches =3D bb.cache.MulticonfigCache(self.databuilder, = self.data_hash, self.caches_array) + envdata =3D bb_caches[mc].loadDataFull(fn, self.collection= s[mc].get_file_appends(fn)) except Exception as e: parselog.exception("Unable to read %s", fn) raise @@ -594,7 +629,7 @@ class BBCooker: # Replace string such as "mc:*:bash" # into "mc:A:bash mc:B:bash bash" for k in targetlist: - if k.startswith("mc:"): + if k.startswith("mc:") and k.count(':') >=3D 2: if wildcard: bb.fatal('multiconfig conflict') if k.split(":")[1] =3D=3D "*": @@ -626,8 +661,9 @@ class BBCooker: current =3D 0 runlist =3D [] for k in fulltargetlist: + origk =3D k mc =3D "" - if k.startswith("mc:"): + if k.startswith("mc:") and k.count(':') >=3D 2: mc =3D k.split(":")[1] k =3D ":".join(k.split(":")[2:]) ktask =3D task @@ -635,6 +671,10 @@ class BBCooker: k2 =3D k.split(":do_") k =3D k2[0] ktask =3D k2[1] + + if mc not in self.multiconfigs: + bb.fatal("Multiconfig dependency %s depends on nonexisten= t multiconfig configuration named %s" % (origk, mc)) + taskdata[mc].add_provider(localdata[mc], self.recipecaches[mc]= , k) current +=3D 1 if not ktask.startswith("do_"): @@ -670,9 +710,9 @@ class BBCooker: l =3D k.split(':') depmc =3D l[2] if depmc not in self.multiconfigs: - bb.fatal("Multiconfig dependency %s depends on= nonexistent mc configuration %s" % (k,depmc)) + bb.fatal("Multiconfig dependency %s depends on= nonexistent multiconfig configuration named configuration %s" % (k,depmc)) else: - logger.debug(1, "Adding providers for multicon= fig dependency %s" % l[3]) + logger.debug("Adding providers for multiconfig= dependency %s" % l[3]) taskdata[depmc].add_provider(localdata[depmc],= self.recipecaches[depmc], l[3]) seen.add(k) new =3D True @@ -929,26 +969,33 @@ class BBCooker: logger.info("Task dependencies saved to 'task-depends.dot'") =20 def show_appends_with_no_recipes(self): + appends_without_recipes =3D {} # Determine which bbappends haven't been applied - - # First get list of recipes, including skipped - recipefns =3D list(self.recipecaches[''].pkg_fn.keys()) - recipefns.extend(self.skiplist.keys()) - - # Work out list of bbappends that have been applied - applied_appends =3D [] - for fn in recipefns: - applied_appends.extend(self.collection.get_file_appends(fn)) - - appends_without_recipes =3D [] - for _, appendfn in self.collection.bbappends: - if not appendfn in applied_appends: - appends_without_recipes.append(appendfn) - - if appends_without_recipes: - msg =3D 'No recipes available for:\n %s' % '\n '.join(append= s_without_recipes) - warn_only =3D self.data.getVar("BB_DANGLINGAPPENDS_WARNONLY", \ - False) or "no" + for mc in self.multiconfigs: + # First get list of recipes, including skipped + recipefns =3D list(self.recipecaches[mc].pkg_fn.keys()) + recipefns.extend(self.skiplist.keys()) + + # Work out list of bbappends that have been applied + applied_appends =3D [] + for fn in recipefns: + applied_appends.extend(self.collections[mc].get_file_appen= ds(fn)) + + appends_without_recipes[mc] =3D [] + for _, appendfn in self.collections[mc].bbappends: + if not appendfn in applied_appends: + appends_without_recipes[mc].append(appendfn) + + msgs =3D [] + for mc in sorted(appends_without_recipes.keys()): + if appends_without_recipes[mc]: + msgs.append('No recipes in %s available for:\n %s' % (mc = if mc else 'default', + '\= n '.join(appends_without_recipes[mc]))) + + if msgs: + msg =3D "\n".join(msgs) + warn_only =3D self.databuilder.mcdata[mc].getVar("BB_DANGLINGA= PPENDS_WARNONLY", \ + False) or "no" if warn_only.lower() in ("1", "yes", "true"): bb.warn(msg) else: @@ -1029,10 +1076,16 @@ class BBCooker: if pn in self.recipecaches[mc].providers: filenames =3D self.recipecaches[mc].providers[pn] eligible, foundUnique =3D bb.providers.filterProviders(filenam= es, pn, self.databuilder.mcdata[mc], self.recipecaches[mc]) - filename =3D eligible[0] + if eligible is not None: + filename =3D eligible[0] + else: + filename =3D None return None, None, None, filename elif pn in self.recipecaches[mc].pkg_pn: - return bb.providers.findBestProvider(pn, self.databuilder.mcda= ta[mc], self.recipecaches[mc], self.recipecaches[mc].pkg_pn) + (latest, latest_f, preferred_ver, preferred_file, required) = =3D bb.providers.findBestProvider(pn, self.databuilder.mcdata[mc], self.rec= ipecaches[mc], self.recipecaches[mc].pkg_pn) + if required and preferred_file is None: + return None, None, None, None + return (latest, latest_f, preferred_ver, preferred_file) else: return None, None, None, None =20 @@ -1097,7 +1150,7 @@ class BBCooker: from bb import shell except ImportError: parselog.exception("Interactive mode not available") - sys.exit(1) + raise bb.BBHandledException() else: shell.start( self ) =20 @@ -1249,15 +1302,15 @@ class BBCooker: if siggen_cache: bb.parse.siggen.checksum_cache.mtime_cache.clear() =20 - def matchFiles(self, bf): + def matchFiles(self, bf, mc=3D''): """ Find the .bb files which match the expression in 'buildfile'. """ if bf.startswith("/") or bf.startswith("../"): bf =3D os.path.abspath(bf) =20 - self.collection =3D CookerCollectFiles(self.bbfile_config_prioriti= es) - filelist, masked, searchdirs =3D self.collection.collect_bbfiles(s= elf.data, self.data) + self.collections =3D {mc: CookerCollectFiles(self.bbfile_config_pr= iorities, mc)} + filelist, masked, searchdirs =3D self.collections[mc].collect_bbfi= les(self.databuilder.mcdata[mc], self.databuilder.mcdata[mc]) try: os.stat(bf) bf =3D os.path.abspath(bf) @@ -1270,12 +1323,12 @@ class BBCooker: matches.append(f) return matches =20 - def matchFile(self, buildfile): + def matchFile(self, buildfile, mc=3D''): """ Find the .bb file which matches the expression in 'buildfile'. Raise an error if multiple files """ - matches =3D self.matchFiles(buildfile) + matches =3D self.matchFiles(buildfile, mc) if len(matches) !=3D 1: if matches: msg =3D "Unable to match '%s' to a specific recipe file - = %s matches found:" % (buildfile, len(matches)) @@ -1316,14 +1369,14 @@ class BBCooker: task =3D "do_%s" % task =20 fn, cls, mc =3D bb.cache.virtualfn2realfn(buildfile) - fn =3D self.matchFile(fn) + fn =3D self.matchFile(fn, mc) =20 self.buildSetVars() self.reset_mtime_caches() =20 - bb_cache =3D bb.cache.Cache(self.databuilder, self.data_hash, self= .caches_array) + bb_caches =3D bb.cache.MulticonfigCache(self.databuilder, self.dat= a_hash, self.caches_array) =20 - infos =3D bb_cache.parse(fn, self.collection.get_file_appends(fn)) + infos =3D bb_caches[mc].parse(fn, self.collections[mc].get_file_ap= pends(fn)) infos =3D dict(infos) =20 fn =3D bb.cache.realfn2virtual(fn, cls, mc) @@ -1411,7 +1464,7 @@ class BBCooker: return True return retval =20 - self.configuration.server_register_idlecallback(buildFileIdle, rq) + self.idleCallBackRegister(buildFileIdle, rq) =20 def buildTargets(self, targets, task): """ @@ -1482,7 +1535,7 @@ class BBCooker: if 'universe' in targets: rq.rqdata.warn_multi_bb =3D True =20 - self.configuration.server_register_idlecallback(buildTargetsIdle, = rq) + self.idleCallBackRegister(buildTargetsIdle, rq) =20 =20 def getAllKeysWithFlags(self, flaglist): @@ -1521,7 +1574,7 @@ class BBCooker: self.inotify_modified_files =3D [] =20 if not self.baseconfig_valid: - logger.debug(1, "Reloading base configuration data") + logger.debug("Reloading base configuration data") self.initConfigurationData() self.handlePRServ() =20 @@ -1533,6 +1586,7 @@ class BBCooker: if self.state in (state.shutdown, state.forceshutdown, state.error= ): if hasattr(self.parser, 'shutdown'): self.parser.shutdown(clean=3DFalse, force =3D True) + self.parser.final_cleanup() raise bb.BBHandledException() =20 if self.state !=3D state.parsing: @@ -1552,14 +1606,24 @@ class BBCooker: for dep in self.configuration.extra_assume_provided: self.recipecaches[mc].ignored_dependencies.add(dep) =20 - self.collection =3D CookerCollectFiles(self.bbfile_config_prio= rities) - (filelist, masked, searchdirs) =3D self.collection.collect_bbf= iles(self.data, self.data) + self.collections =3D {} + + mcfilelist =3D {} + total_masked =3D 0 + searchdirs =3D set() + for mc in self.multiconfigs: + self.collections[mc] =3D CookerCollectFiles(self.bbfile_co= nfig_priorities, mc) + (filelist, masked, search) =3D self.collections[mc].collec= t_bbfiles(self.databuilder.mcdata[mc], self.databuilder.mcdata[mc]) + + mcfilelist[mc] =3D filelist + total_masked +=3D masked + searchdirs |=3D set(search) =20 # Add inotify watches for directories searched for bb/bbappend= files for dirent in searchdirs: self.add_filewatch([[dirent]], dirs=3DTrue) =20 - self.parser =3D CookerParser(self, filelist, masked) + self.parser =3D CookerParser(self, mcfilelist, total_masked) self.parsecache_valid =3D True =20 self.state =3D state.parsing @@ -1571,7 +1635,7 @@ class BBCooker: self.show_appends_with_no_recipes() self.handlePrefProviders() for mc in self.multiconfigs: - self.recipecaches[mc].bbfile_priority =3D self.collection.= collection_priorities(self.recipecaches[mc].pkg_fn, self.data) + self.recipecaches[mc].bbfile_priority =3D self.collections= [mc].collection_priorities(self.recipecaches[mc].pkg_fn, self.parser.mcfile= list[mc], self.data) self.state =3D state.running =20 # Send an event listing all stamps reachable after parsing @@ -1630,17 +1694,16 @@ class BBCooker: return pkgs_to_build =20 def pre_serve(self): - # We now are in our own process so we can call this here. - # PRServ exits if its parent process exits - self.handlePRServ() return =20 def post_serve(self): + self.shutdown(force=3DTrue) prserv.serv.auto_shutdown() if self.hashserv: self.hashserv.process.terminate() self.hashserv.process.join() - bb.event.fire(CookerExit(), self.data) + if hasattr(self, "data"): + bb.event.fire(CookerExit(), self.data) =20 def shutdown(self, force =3D False): if force: @@ -1650,6 +1713,7 @@ class BBCooker: =20 if self.parser: self.parser.shutdown(clean=3Dnot force, force=3Dforce) + self.parser.final_cleanup() =20 def finishcommand(self): self.state =3D state.initial @@ -1663,8 +1727,9 @@ class BBCooker: self.finishcommand() self.extraconfigdata =3D {} self.command.reset() - self.databuilder.reset() - self.data =3D self.databuilder.data + if hasattr(self, "data"): + self.databuilder.reset() + self.data =3D self.databuilder.data self.parsecache_valid =3D False self.baseconfig_valid =3D False =20 @@ -1679,21 +1744,19 @@ class CookerExit(bb.event.Event): =20 =20 class CookerCollectFiles(object): - def __init__(self, priorities): + def __init__(self, priorities, mc=3D''): + self.mc =3D mc self.bbappends =3D [] # Priorities is a list of tupples, with the second element as the = pattern. # We need to sort the list with the longest pattern first, and so = on to # the shortest. This allows nested layers to be properly evaluate= d. self.bbfile_config_priorities =3D sorted(priorities, key=3Dlambda = tup: tup[1], reverse=3DTrue) =20 - def calc_bbfile_priority( self, filename, matched =3D None ): + def calc_bbfile_priority(self, filename): for _, _, regex, pri in self.bbfile_config_priorities: if regex.match(filename): - if matched is not None: - if not regex in matched: - matched.add(regex) - return pri - return 0 + return pri, regex + return 0, None =20 def get_bbfiles(self): """Get list of default .bb files by reading out the current direct= ory""" @@ -1723,10 +1786,10 @@ class CookerCollectFiles(object): collectlog.debug(1, "collecting .bb files") =20 files =3D (config.getVar( "BBFILES") or "").split() - config.setVar("BBFILES", " ".join(files)) =20 # Sort files by priority - files.sort( key=3Dlambda fileitem: self.calc_bbfile_priority(filei= tem) ) + files.sort( key=3Dlambda fileitem: self.calc_bbfile_priority(filei= tem)[0] ) + config.setVar("BBFILES_PRIORITIZED", " ".join(files)) =20 if not len(files): files =3D self.get_bbfiles() @@ -1846,43 +1909,67 @@ class CookerCollectFiles(object): (bbappend, filename) =3D b if (bbappend =3D=3D f) or ('%' in bbappend and bbappend.starts= with(f[:bbappend.index('%')])): filelist.append(filename) - return filelist + return tuple(filelist) =20 - def collection_priorities(self, pkgfns, d): + def collection_priorities(self, pkgfns, fns, d): + # Return the priorities of the entries in pkgfns + # Also check that all the regexes in self.bbfile_config_priorities= are used + # (but to do that we need to ensure skipped recipes aren't counted= , nor + # collections in BBFILE_PATTERN_IGNORE_EMPTY) =20 priorities =3D {} + seen =3D set() + matched =3D set() + + matched_regex =3D set() + unmatched_regex =3D set() + for _, _, regex, _ in self.bbfile_config_priorities: + unmatched_regex.add(regex) =20 # Calculate priorities for each file - matched =3D set() for p in pkgfns: realfn, cls, mc =3D bb.cache.virtualfn2realfn(p) - priorities[p] =3D self.calc_bbfile_priority(realfn, matched) - - unmatched =3D set() - for _, _, regex, pri in self.bbfile_config_priorities: - if not regex in matched: - unmatched.add(regex) - - # Don't show the warning if the BBFILE_PATTERN did match .bbappend= files - def find_bbappend_match(regex): + priorities[p], regex =3D self.calc_bbfile_priority(realfn) + if regex in unmatched_regex: + matched_regex.add(regex) + unmatched_regex.remove(regex) + seen.add(realfn) + if regex: + matched.add(realfn) + + if unmatched_regex: + # Account for bbappend files for b in self.bbappends: (bbfile, append) =3D b - if regex.match(append): - # If the bbappend is matched by already "matched set",= return False - for matched_regex in matched: - if matched_regex.match(append): - return False - return True - return False + seen.add(append) =20 - for unmatch in unmatched.copy(): - if find_bbappend_match(unmatch): - unmatched.remove(unmatch) + # Account for skipped recipes + seen.update(fns) + + seen.difference_update(matched) + + def already_matched(fn): + for regex in matched_regex: + if regex.match(fn): + return True + return False + + for unmatch in unmatched_regex.copy(): + for fn in seen: + if unmatch.match(fn): + # If the bbappend or file was already matched by a= nother regex, skip it + # e.g. for a layer within a layer, the outer regex= could match, the inner + # regex may match nothing and we should warn about= that + if already_matched(fn): + continue + unmatched_regex.remove(unmatch) + break =20 for collection, pattern, regex, _ in self.bbfile_config_priorities: - if regex in unmatched: + if regex in unmatched_regex: if d.getVar('BBFILE_PATTERN_IGNORE_EMPTY_%s' % collection)= !=3D '1': - collectlog.warning("No bb files matched BBFILE_PATTERN= _%s '%s'" % (collection, pattern)) + collectlog.warning("No bb files in %s matched BBFILE_P= ATTERN_%s '%s'" % (self.mc if self.mc else 'default', + = collection, pattern)) =20 return priorities =20 @@ -1931,7 +2018,8 @@ class Parser(multiprocessing.Process): except queue.Empty: pass else: - self.results.cancel_join_thread() + self.results.close() + self.results.join_thread() break =20 if pending: @@ -1940,6 +2028,8 @@ class Parser(multiprocessing.Process): try: job =3D self.jobs.pop() except IndexError: + self.results.close() + self.results.join_thread() break result =3D self.parse(*job) # Clear the siggen cache after parsing to control memory u= sage, its huge @@ -1949,7 +2039,7 @@ class Parser(multiprocessing.Process): except queue.Full: pending.append(result) =20 - def parse(self, filename, appends): + def parse(self, mc, cache, filename, appends): try: origfilter =3D bb.event.LogHandler.filter # Record the filename we're parsing into any events generated @@ -1963,7 +2053,7 @@ class Parser(multiprocessing.Process): bb.event.set_class_handlers(self.handlers.copy()) bb.event.LogHandler.filter =3D parse_filter =20 - return True, self.bb_cache.parse(filename, appends) + return True, mc, cache.parse(filename, appends) except Exception as exc: tb =3D sys.exc_info()[2] exc.recipe =3D filename @@ -1978,8 +2068,8 @@ class Parser(multiprocessing.Process): bb.event.LogHandler.filter =3D origfilter =20 class CookerParser(object): - def __init__(self, cooker, filelist, masked): - self.filelist =3D filelist + def __init__(self, cooker, mcfilelist, masked): + self.mcfilelist =3D mcfilelist self.cooker =3D cooker self.cfgdata =3D cooker.data self.cfghash =3D cooker.data_hash @@ -1993,28 +2083,31 @@ class CookerParser(object): =20 self.skipped =3D 0 self.virtuals =3D 0 - self.total =3D len(filelist) =20 self.current =3D 0 self.process_names =3D [] =20 - self.bb_cache =3D bb.cache.Cache(self.cfgbuilder, self.cfghash, co= oker.caches_array) - self.fromcache =3D [] - self.willparse =3D [] - for filename in self.filelist: - appends =3D self.cooker.collection.get_file_appends(filename) - if not self.bb_cache.cacheValid(filename, appends): - self.willparse.append((filename, appends)) - else: - self.fromcache.append((filename, appends)) - self.toparse =3D self.total - len(self.fromcache) + self.bb_caches =3D bb.cache.MulticonfigCache(self.cfgbuilder, self= .cfghash, cooker.caches_array) + self.fromcache =3D set() + self.willparse =3D set() + for mc in self.cooker.multiconfigs: + for filename in self.mcfilelist[mc]: + appends =3D self.cooker.collections[mc].get_file_appends(f= ilename) + if not self.bb_caches[mc].cacheValid(filename, appends): + self.willparse.add((mc, self.bb_caches[mc], filename, = appends)) + else: + self.fromcache.add((mc, self.bb_caches[mc], filename, = appends)) + + self.total =3D len(self.fromcache) + len(self.willparse) + self.toparse =3D len(self.willparse) self.progress_chunk =3D int(max(self.toparse / 100, 1)) =20 self.num_processes =3D min(int(self.cfgdata.getVar("BB_NUMBER_PARS= E_THREADS") or - multiprocessing.cpu_count()), len(self.wi= llparse)) + multiprocessing.cpu_count()), self.topars= e) =20 self.start() self.haveshutdown =3D False + self.syncthread =3D None =20 def start(self): self.results =3D self.load_cached() @@ -2022,7 +2115,9 @@ class CookerParser(object): if self.toparse: bb.event.fire(bb.event.ParseStarted(self.toparse), self.cfgdat= a) def init(): - Parser.bb_cache =3D self.bb_cache + signal.signal(signal.SIGTERM, signal.SIG_DFL) + signal.signal(signal.SIGHUP, signal.SIG_DFL) + signal.signal(signal.SIGINT, signal.SIG_IGN) bb.utils.set_process_name(multiprocessing.current_process(= ).name) multiprocessing.util.Finalize(None, bb.codeparser.parser_c= ache_save, exitpriority=3D1) multiprocessing.util.Finalize(None, bb.fetch.fetcher_parse= _save, exitpriority=3D1) @@ -2032,7 +2127,7 @@ class CookerParser(object): =20 def chunkify(lst,n): return [lst[i::n] for i in range(n)] - self.jobs =3D chunkify(self.willparse, self.num_processes) + self.jobs =3D chunkify(list(self.willparse), self.num_processe= s) =20 for i in range(0, self.num_processes): parser =3D Parser(self.jobs[i], self.result_queue, self.pa= rser_quit, init, self.cooker.configuration.profile) @@ -2056,12 +2151,9 @@ class CookerParser(object): self.total) =20 bb.event.fire(event, self.cfgdata) - for process in self.processes: - self.parser_quit.put(None) - else: - self.parser_quit.cancel_join_thread() - for process in self.processes: - self.parser_quit.put(None) + + for process in self.processes: + self.parser_quit.put(None) =20 # Cleanup the queue before call process.join(), otherwise there mi= ght be # deadlocks. @@ -2078,9 +2170,17 @@ class CookerParser(object): else: process.join() =20 - sync =3D threading.Thread(target=3Dself.bb_cache.sync) + self.parser_quit.close() + # Allow data left in the cancel queue to be discarded + self.parser_quit.cancel_join_thread() + + def sync_caches(): + for c in self.bb_caches.values(): + c.sync() + + sync =3D threading.Thread(target=3Dsync_caches, name=3D"SyncThread= ") + self.syncthread =3D sync sync.start() - multiprocessing.util.Finalize(None, sync.join, exitpriority=3D-100) bb.codeparser.parser_cache_savemerge() bb.fetch.fetcher_parse_done() if self.cooker.configuration.profile: @@ -2094,10 +2194,14 @@ class CookerParser(object): bb.utils.process_profilelog(profiles, pout =3D pout) print("Processed parsing statistics saved to %s" % (pout)) =20 + def final_cleanup(self): + if self.syncthread: + self.syncthread.join() + def load_cached(self): - for filename, appends in self.fromcache: - cached, infos =3D self.bb_cache.load(filename, appends) - yield not cached, infos + for mc, cache, filename, appends in self.fromcache: + cached, infos =3D cache.load(filename, appends) + yield not cached, mc, infos =20 def parse_generator(self): while True: @@ -2119,25 +2223,25 @@ class CookerParser(object): result =3D [] parsed =3D None try: - parsed, result =3D next(self.results) + parsed, mc, result =3D next(self.results) except StopIteration: self.shutdown() return False except bb.BBHandledException as exc: self.error +=3D 1 logger.error('Failed to parse recipe: %s' % exc.recipe) - self.shutdown(clean=3DFalse) + self.shutdown(clean=3DFalse, force=3DTrue) return False except ParsingFailure as exc: self.error +=3D 1 logger.error('Unable to parse %s: %s' % (exc.recipe, bb.exceptions.to_string(exc.realexceptio= n))) - self.shutdown(clean=3DFalse) + self.shutdown(clean=3DFalse, force=3DTrue) return False except bb.parse.ParseError as exc: self.error +=3D 1 logger.error(str(exc)) - self.shutdown(clean=3DFalse) + self.shutdown(clean=3DFalse, force=3DTrue) return False except bb.data_smart.ExpansionError as exc: self.error +=3D 1 @@ -2146,7 +2250,7 @@ class CookerParser(object): tb =3D list(itertools.dropwhile(lambda e: e.filename.startswit= h(bbdir), exc.traceback)) logger.error('ExpansionError during parsing %s', value.recipe, exc_info=3D(etype, value, tb)) - self.shutdown(clean=3DFalse) + self.shutdown(clean=3DFalse, force=3DTrue) return False except Exception as exc: self.error +=3D 1 @@ -2158,7 +2262,7 @@ class CookerParser(object): # Most likely, an exception occurred during raising an exc= eption import traceback logger.error('Exception during parse: %s' % traceback.form= at_exc()) - self.shutdown(clean=3DFalse) + self.shutdown(clean=3DFalse, force=3DTrue) return False =20 self.current +=3D 1 @@ -2175,13 +2279,16 @@ class CookerParser(object): if info_array[0].skipped: self.skipped +=3D 1 self.cooker.skiplist[virtualfn] =3D SkippedPackage(info_ar= ray[0]) - (fn, cls, mc) =3D bb.cache.virtualfn2realfn(virtualfn) - self.bb_cache.add_info(virtualfn, info_array, self.cooker.reci= pecaches[mc], + self.bb_caches[mc].add_info(virtualfn, info_array, self.cooker= .recipecaches[mc], parsed=3Dparsed, watcher =3D self.= cooker.add_filewatch) return True =20 def reparse(self, filename): - infos =3D self.bb_cache.parse(filename, self.cooker.collection.get= _file_appends(filename)) - for vfn, info_array in infos: - (fn, cls, mc) =3D bb.cache.virtualfn2realfn(vfn) - self.cooker.recipecaches[mc].add_from_recipeinfo(vfn, info_arr= ay) + to_reparse =3D set() + for mc in self.cooker.multiconfigs: + to_reparse.add((mc, filename, self.cooker.collections[mc].get_= file_appends(filename))) + + for mc, filename, appends in to_reparse: + infos =3D self.bb_caches[mc].parse(filename, appends) + for vfn, info_array in infos: + self.cooker.recipecaches[mc].add_from_recipeinfo(vfn, info= _array) diff --git a/bitbake/lib/bb/cookerdata.py b/bitbake/lib/bb/cookerdata.py index 472423fd..ba657c03 100644 --- a/bitbake/lib/bb/cookerdata.py +++ b/bitbake/lib/bb/cookerdata.py @@ -23,8 +23,8 @@ logger =3D logging.getLogger("BitBake") parselog =3D logging.getLogger("BitBake.Parsing") =20 class ConfigParameters(object): - def __init__(self, argv=3Dsys.argv): - self.options, targets =3D self.parseCommandLine(argv) + def __init__(self, argv=3DNone): + self.options, targets =3D self.parseCommandLine(argv or sys.argv) self.environment =3D self.parseEnvironment() =20 self.options.pkgs_to_build =3D targets or [] @@ -58,11 +58,18 @@ class ConfigParameters(object): def updateToServer(self, server, environment): options =3D {} for o in ["abort", "force", "invalidate_stamp", - "verbose", "debug", "dry_run", "dump_signatures", - "debug_domains", "extra_assume_provided", "profile", - "prefile", "postfile", "server_timeout"]: + "dry_run", "dump_signatures", + "extra_assume_provided", "profile", + "prefile", "postfile", "server_timeout", + "nosetscene", "setsceneonly", "skipsetscene", + "runall", "runonly", "writeeventlog"]: options[o] =3D getattr(self.options, o) =20 + options['build_verbose_shell'] =3D self.options.verbose + options['build_verbose_stdout'] =3D self.options.verbose + options['default_loglevel'] =3D bb.msg.loggerDefaultLogLevel + options['debug_domains'] =3D bb.msg.loggerDefaultDomains + ret, error =3D server.runCommand(["updateConfig", options, environ= ment, sys.argv]) if error: raise Exception("Unable to update the server configuration wit= h local parameters: %s" % error) @@ -111,11 +118,11 @@ class CookerConfiguration(object): """ =20 def __init__(self): - self.debug_domains =3D [] + self.debug_domains =3D bb.msg.loggerDefaultDomains + self.default_loglevel =3D bb.msg.loggerDefaultLogLevel self.extra_assume_provided =3D [] self.prefile =3D [] self.postfile =3D [] - self.debug =3D 0 self.cmd =3D None self.abort =3D True self.force =3D False @@ -125,34 +132,21 @@ class CookerConfiguration(object): self.skipsetscene =3D False self.invalidate_stamp =3D False self.dump_signatures =3D [] + self.build_verbose_shell =3D False + self.build_verbose_stdout =3D False self.dry_run =3D False self.tracking =3D False - self.xmlrpcinterface =3D [] - self.server_timeout =3D None self.writeeventlog =3D False - self.server_only =3D False self.limited_deps =3D False self.runall =3D [] self.runonly =3D [] =20 self.env =3D {} =20 - def setConfigParameters(self, parameters): - for key in self.__dict__.keys(): - if key in parameters.options.__dict__: - setattr(self, key, parameters.options.__dict__[key]) - self.env =3D parameters.environment.copy() - - def setServerRegIdleCallback(self, srcb): - self.server_register_idlecallback =3D srcb - def __getstate__(self): state =3D {} for key in self.__dict__.keys(): - if key =3D=3D "server_register_idlecallback": - state[key] =3D None - else: - state[key] =3D getattr(self, key) + state[key] =3D getattr(self, key) return state =20 def __setstate__(self,state): @@ -170,7 +164,7 @@ def catch_parse_error(func): import traceback parselog.critical(traceback.format_exc()) parselog.critical("Unable to parse %s: %s" % (fn, exc)) - sys.exit(1) + raise bb.BBHandledException() except bb.data_smart.ExpansionError as exc: import traceback =20 @@ -182,10 +176,10 @@ def catch_parse_error(func): if not fn.startswith(bbdir): break parselog.critical("Unable to parse %s" % fn, exc_info=3D(exc_c= lass, exc, tb)) - sys.exit(1) + raise bb.BBHandledException() except bb.parse.ParseError as exc: parselog.critical(str(exc)) - sys.exit(1) + raise bb.BBHandledException() return wrapped =20 @catch_parse_error @@ -215,7 +209,7 @@ def findConfigFile(configfile, data): return None =20 # -# We search for a conf/bblayers.conf under an entry in BBPATH or in cwd wo= rking=20 +# We search for a conf/bblayers.conf under an entry in BBPATH or in cwd wo= rking # up to /. If that fails, we search for a conf/bitbake.conf in BBPATH. # =20 @@ -297,6 +291,8 @@ class CookerDataBuilder(object): =20 multiconfig =3D (self.data.getVar("BBMULTICONFIG") or "").spli= t() for config in multiconfig: + if config[0].isdigit(): + bb.fatal("Multiconfig name '%s' is invalid as multicon= figs cannot start with a digit" % config) mcdata =3D self.parseConfigurationFiles(self.prefiles, sel= f.postfiles, config) bb.event.fire(bb.event.ConfigParsed(), mcdata) self.mcdata[config] =3D mcdata @@ -306,13 +302,13 @@ class CookerDataBuilder(object): =20 self.data_hash =3D data_hash.hexdigest() except (SyntaxError, bb.BBHandledException): - raise bb.BBHandledException + raise bb.BBHandledException() except bb.data_smart.ExpansionError as e: logger.error(str(e)) - raise bb.BBHandledException + raise bb.BBHandledException() except Exception: logger.exception("Error parsing configuration files") - raise bb.BBHandledException + raise bb.BBHandledException() =20 # Create a copy so we can reset at a later date when UIs disconnect self.origdata =3D self.data @@ -348,6 +344,9 @@ class CookerDataBuilder(object): layers =3D (data.getVar('BBLAYERS') or "").split() broken_layers =3D [] =20 + if not layers: + bb.fatal("The bblayers.conf file doesn't contain any BBLAY= ERS definition") + data =3D bb.data.createCopy(data) approved =3D bb.utils.approved_variables() =20 @@ -361,7 +360,7 @@ class CookerDataBuilder(object): for layer in broken_layers: parselog.critical(" %s", layer) parselog.critical("Please check BBLAYERS in %s" % (layerco= nf)) - sys.exit(1) + raise bb.BBHandledException() =20 for layer in layers: parselog.debug(2, "Adding layer %s", layer) @@ -387,10 +386,13 @@ class CookerDataBuilder(object): invalid.append(entry) continue l, f =3D parts - if l in collections: + invert =3D l[0] =3D=3D "!" + if invert: + l =3D l[1:] + if (l in collections and not invert) or (l not in collecti= ons and invert): data.appendVar("BBFILES", " " + f) if invalid: - bb.fatal("BBFILES_DYNAMIC entries must be of the form <col= lection name>:<filename pattern>, not:\n %s" % "\n ".join(invalid)) + bb.fatal("BBFILES_DYNAMIC entries must be of the form {!}<= collection name>:<filename pattern>, not:\n %s" % "\n ".join(invalid)) =20 layerseries =3D set((data.getVar("LAYERSERIES_CORENAMES") or "= ").split()) collections_tmp =3D collections[:] @@ -399,6 +401,8 @@ class CookerDataBuilder(object): if c in collections_tmp: bb.fatal("Found duplicated BBFILE_COLLECTIONS '%s', ch= eck bblayers.conf or layer.conf to fix it." % c) compat =3D set((data.getVar("LAYERSERIES_COMPAT_%s" % c) o= r "").split()) + if compat and not layerseries: + bb.fatal("No core layer found to work with layer '%s'.= Missing entry in bblayers.conf?" % c) if compat and not (compat & layerseries): bb.fatal("Layer %s is not compatible with the core lay= er which only supports these series: %s (layer is compatible with %s)" % (c, " ".join(layerseries), " ".join(compat= ))) @@ -430,9 +434,9 @@ class CookerDataBuilder(object): handlerfn =3D data.getVarFlag(var, "filename", False) if not handlerfn: parselog.critical("Undefined event handler function '%s'" = % var) - sys.exit(1) + raise bb.BBHandledException() handlerln =3D int(data.getVarFlag(var, "lineno", False)) - bb.event.register(var, data.getVar(var, False), (data.getVarF= lag(var, "eventmask") or "").split(), handlerfn, handlerln) + bb.event.register(var, data.getVar(var, False), (data.getVarF= lag(var, "eventmask") or "").split(), handlerfn, handlerln, data) =20 data.setVar('BBINCLUDED',bb.parse.get_file_depends(data)) =20 diff --git a/bitbake/lib/bb/daemonize.py b/bitbake/lib/bb/daemonize.py index f01e6ec7..c187fcfc 100644 --- a/bitbake/lib/bb/daemonize.py +++ b/bitbake/lib/bb/daemonize.py @@ -14,6 +14,8 @@ import sys import io import traceback =20 +import bb + def createDaemon(function, logfile): """ Detach a process from the controlling terminal and run it in the diff --git a/bitbake/lib/bb/data.py b/bitbake/lib/bb/data.py index b0683c51..97022853 100644 --- a/bitbake/lib/bb/data.py +++ b/bitbake/lib/bb/data.py @@ -161,6 +161,12 @@ def emit_var(var, o=3Dsys.__stdout__, d =3D init(), al= l=3DFalse): return True =20 if func: + # Write a comment indicating where the shell function came from (l= ine number and filename) to make it easier + # for the user to diagnose task failures. This comment is also use= d by build.py to determine the metadata + # location of shell functions. + o.write("# line: {0}, file: {1}\n".format( + d.getVarFlag(var, "lineno", False), + d.getVarFlag(var, "filename", False))) # NOTE: should probably check for unbalanced {} within the var val =3D val.rstrip('\n') o.write("%s() {\n%s\n}\n" % (varExpanded, val)) diff --git a/bitbake/lib/bb/data_smart.py b/bitbake/lib/bb/data_smart.py index 7f1b6dcb..8291ca65 100644 --- a/bitbake/lib/bb/data_smart.py +++ b/bitbake/lib/bb/data_smart.py @@ -28,7 +28,7 @@ logger =3D logging.getLogger("BitBake.Data") =20 __setvar_keyword__ =3D ["_append", "_prepend", "_remove"] __setvar_regexp__ =3D re.compile(r'(?P<base>.*?)(?P<keyword>_append|_prepe= nd|_remove)(_(?P<add>[^A-Z]*))?$') -__expand_var_regexp__ =3D re.compile(r"\${[a-zA-Z0-9\-_+./~]+?}") +__expand_var_regexp__ =3D re.compile(r"\${[a-zA-Z0-9\-_+./~:]+?}") __expand_python_regexp__ =3D re.compile(r"\${@.+?}") __whitespace_split__ =3D re.compile(r'(\s)') __override_regexp__ =3D re.compile(r'[a-z0-9]+') @@ -189,7 +189,7 @@ class IncludeHistory(object): if self.current.parent: self.current =3D self.current.parent else: - bb.warn("Include log: Tried to finish '%s' at top level." % fi= lename) + bb.warn("Include log: Tried to finish '%s' at top level." % se= lf.filename) return False =20 def emit(self, o, level =3D 0): @@ -411,6 +411,8 @@ class DataSmart(MutableMapping): raise except bb.parse.SkipRecipe: raise + except bb.BBHandledException: + raise except Exception as exc: tb =3D sys.exc_info()[2] raise ExpansionError(varname, s, exc).with_traceback(tb) f= rom exc @@ -481,6 +483,7 @@ class DataSmart(MutableMapping): =20 def setVar(self, var, value, **loginfo): #print("var=3D" + str(var) + " val=3D" + str(value)) + var =3D var.replace(":", "_") self.expand_cache =3D {} parsing=3DFalse if 'parsing' in loginfo: @@ -589,6 +592,8 @@ class DataSmart(MutableMapping): """ Rename the variable key to newkey """ + key =3D key.replace(":", "_") + newkey =3D newkey.replace(":", "_") if key =3D=3D newkey: bb.warn("Calling renameVar with equivalent keys (%s) is invali= d" % key) return @@ -637,6 +642,7 @@ class DataSmart(MutableMapping): self.setVar(var + "_prepend", value, ignore=3DTrue, parsing=3DTrue) =20 def delVar(self, var, **loginfo): + var =3D var.replace(":", "_") self.expand_cache =3D {} =20 loginfo['detail'] =3D "" @@ -664,6 +670,7 @@ class DataSmart(MutableMapping): override =3D None =20 def setVarFlag(self, var, flag, value, **loginfo): + var =3D var.replace(":", "_") self.expand_cache =3D {} =20 if 'op' not in loginfo: @@ -687,6 +694,7 @@ class DataSmart(MutableMapping): self.dict["__exportlist"]["_content"].add(var) =20 def getVarFlag(self, var, flag, expand=3DTrue, noweakdefault=3DFalse, = parsing=3DFalse, retparser=3DFalse): + var =3D var.replace(":", "_") if flag =3D=3D "_content": cachename =3D var else: @@ -814,6 +822,7 @@ class DataSmart(MutableMapping): return value =20 def delVarFlag(self, var, flag, **loginfo): + var =3D var.replace(":", "_") self.expand_cache =3D {} =20 local_var, _ =3D self._findVar(var) @@ -831,6 +840,7 @@ class DataSmart(MutableMapping): del self.dict[var][flag] =20 def appendVarFlag(self, var, flag, value, **loginfo): + var =3D var.replace(":", "_") loginfo['op'] =3D 'append' loginfo['flag'] =3D flag self.varhistory.record(**loginfo) @@ -838,6 +848,7 @@ class DataSmart(MutableMapping): self.setVarFlag(var, flag, newvalue, ignore=3DTrue) =20 def prependVarFlag(self, var, flag, value, **loginfo): + var =3D var.replace(":", "_") loginfo['op'] =3D 'prepend' loginfo['flag'] =3D flag self.varhistory.record(**loginfo) @@ -845,6 +856,7 @@ class DataSmart(MutableMapping): self.setVarFlag(var, flag, newvalue, ignore=3DTrue) =20 def setVarFlags(self, var, flags, **loginfo): + var =3D var.replace(":", "_") self.expand_cache =3D {} infer_caller_details(loginfo) if not var in self.dict: @@ -859,6 +871,7 @@ class DataSmart(MutableMapping): self.dict[var][i] =3D flags[i] =20 def getVarFlags(self, var, expand =3D False, internalflags=3DFalse): + var =3D var.replace(":", "_") local_var, _ =3D self._findVar(var) flags =3D {} =20 @@ -875,6 +888,7 @@ class DataSmart(MutableMapping): =20 =20 def delVarFlags(self, var, **loginfo): + var =3D var.replace(":", "_") self.expand_cache =3D {} if not var in self.dict: self._makeShadowCopy(var) @@ -1005,7 +1019,7 @@ class DataSmart(MutableMapping): else: data.update({key:value}) =20 - varflags =3D d.getVarFlags(key, internalflags =3D True) + varflags =3D d.getVarFlags(key, internalflags =3D True, expand= =3D["vardepvalue"]) if not varflags: continue for f in varflags: diff --git a/bitbake/lib/bb/event.py b/bitbake/lib/bb/event.py index d1359f01..0454c753 100644 --- a/bitbake/lib/bb/event.py +++ b/bitbake/lib/bb/event.py @@ -10,17 +10,17 @@ BitBake build tools. # SPDX-License-Identifier: GPL-2.0-only # =20 -import sys -import pickle -import logging -import atexit -import traceback import ast +import atexit +import collections +import logging +import pickle +import sys import threading +import traceback =20 -import bb.utils -import bb.compat import bb.exceptions +import bb.utils =20 # This is the pid for which we should generate the event. This is set when # the runqueue forks off. @@ -56,7 +56,7 @@ def set_class_handlers(h): _handlers =3D h =20 def clean_class_handlers(): - return bb.compat.OrderedDict() + return collections.OrderedDict() =20 # Internal _handlers =3D clean_class_handlers() @@ -118,6 +118,8 @@ def fire_class_handlers(event, d): if _eventfilter: if not _eventfilter(name, handler, event, d): continue + if d is not None and not name in (d.getVar("__BBHANDLERS_MC") = or set()): + continue execute_handler(name, handler, event, d) =20 ui_queue =3D [] @@ -227,11 +229,19 @@ def fire_from_worker(event, d): fire_ui_handlers(event, d) =20 noop =3D lambda _: None -def register(name, handler, mask=3DNone, filename=3DNone, lineno=3DNone): +def register(name, handler, mask=3DNone, filename=3DNone, lineno=3DNone, d= ata=3DNone): """Register an Event handler""" =20 + if data is not None and data.getVar("BB_CURRENT_MC"): + mc =3D data.getVar("BB_CURRENT_MC") + name =3D '%s%s' % (mc.replace('-', '_'), name) + # already registered if name in _handlers: + if data is not None: + bbhands_mc =3D (data.getVar("__BBHANDLERS_MC") or set()) + bbhands_mc.add(name) + data.setVar("__BBHANDLERS_MC", bbhands_mc) return AlreadyRegistered =20 if handler is not None: @@ -268,10 +278,20 @@ def register(name, handler, mask=3DNone, filename=3DN= one, lineno=3DNone): _event_handler_map[m] =3D {} _event_handler_map[m][name] =3D True =20 + if data is not None: + bbhands_mc =3D (data.getVar("__BBHANDLERS_MC") or set()) + bbhands_mc.add(name) + data.setVar("__BBHANDLERS_MC", bbhands_mc) + return Registered =20 -def remove(name, handler): +def remove(name, handler, data=3DNone): """Remove an Event handler""" + if data is not None: + if data.getVar("BB_CURRENT_MC"): + mc =3D data.getVar("BB_CURRENT_MC") + name =3D '%s%s' % (mc.replace('-', '_'), name) + _handlers.pop(name) if name in _catchall_handlers: _catchall_handlers.pop(name) @@ -279,6 +299,12 @@ def remove(name, handler): if name in _event_handler_map[event]: _event_handler_map[event].pop(name) =20 + if data is not None: + bbhands_mc =3D (data.getVar("__BBHANDLERS_MC") or set()) + if name in bbhands_mc: + bbhands_mc.remove(name) + data.setVar("__BBHANDLERS_MC", bbhands_mc) + def get_handlers(): return _handlers =20 @@ -389,6 +415,10 @@ class RecipeEvent(Event): class RecipePreFinalise(RecipeEvent): """ Recipe Parsing Complete but not yet finalised""" =20 +class RecipePostKeyExpansion(RecipeEvent): + """ Recipe Parsing Complete but not yet finalised""" + + class RecipeTaskPreProcess(RecipeEvent): """ Recipe Tasks about to be finalised @@ -640,6 +670,17 @@ class ReachableStamps(Event): Event.__init__(self) self.stamps =3D stamps =20 +class StaleSetSceneTasks(Event): + """ + An event listing setscene tasks which are 'stale' and will + be rerun. The metadata may use to clean up stale data. + tasks is a mapping of tasks and matching stale stamps. + """ + + def __init__(self, tasks): + Event.__init__(self) + self.tasks =3D tasks + class FilesMatchingFound(Event): """ Event when a list of files matching the supplied pattern has diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__in= it__.py index eb112f06..dbf8b50e 100644 --- a/bitbake/lib/bb/fetch2/__init__.py +++ b/bitbake/lib/bb/fetch2/__init__.py @@ -290,7 +290,7 @@ class URI(object): =20 def _param_str_split(self, string, elmdelim, kvdelim=3D"=3D"): ret =3D collections.OrderedDict() - for k, v in [x.split(kvdelim, 1) for x in string.split(elmdelim)]: + for k, v in [x.split(kvdelim, 1) for x in string.split(elmdelim) i= f x]: ret[k] =3D v return ret =20 @@ -428,7 +428,7 @@ def uri_replace(ud, uri_find, uri_replace, replacements= , d, mirrortarball=3DNone): uri_decoded =3D list(decodeurl(ud.url)) uri_find_decoded =3D list(decodeurl(uri_find)) uri_replace_decoded =3D list(decodeurl(uri_replace)) - logger.debug(2, "For url %s comparing %s to %s" % (uri_decoded, uri_fi= nd_decoded, uri_replace_decoded)) + logger.debug2("For url %s comparing %s to %s" % (uri_decoded, uri_find= _decoded, uri_replace_decoded)) result_decoded =3D ['', '', '', '', '', {}] for loc, i in enumerate(uri_find_decoded): result_decoded[loc] =3D uri_decoded[loc] @@ -474,7 +474,7 @@ def uri_replace(ud, uri_find, uri_replace, replacements= , d, mirrortarball=3DNone): result =3D encodeurl(result_decoded) if result =3D=3D ud.url: return None - logger.debug(2, "For url %s returning %s" % (ud.url, result)) + logger.debug2("For url %s returning %s" % (ud.url, result)) return result =20 methods =3D [] @@ -499,9 +499,9 @@ def fetcher_init(d): # When to drop SCM head revisions controlled by user policy srcrev_policy =3D d.getVar('BB_SRCREV_POLICY') or "clear" if srcrev_policy =3D=3D "cache": - logger.debug(1, "Keeping SRCREV cache due to cache policy of: %s",= srcrev_policy) + logger.debug("Keeping SRCREV cache due to cache policy of: %s", sr= crev_policy) elif srcrev_policy =3D=3D "clear": - logger.debug(1, "Clearing SRCREV cache due to cache policy of: %s"= , srcrev_policy) + logger.debug("Clearing SRCREV cache due to cache policy of: %s", s= rcrev_policy) revs.clear() else: raise FetchError("Invalid SRCREV cache policy of: %s" % srcrev_pol= icy) @@ -562,6 +562,9 @@ def verify_checksum(ud, d, precomputed=3D{}): =20 checksum_expected =3D getattr(ud, "%s_expected" % checksum_id) =20 + if checksum_expected =3D=3D '': + checksum_expected =3D None + return { "id": checksum_id, "name": checksum_name, @@ -612,7 +615,7 @@ def verify_checksum(ud, d, precomputed=3D{}): =20 for ci in checksum_infos: if ci["expected"] and ci["expected"] !=3D ci["data"]: - messages.append("File: '%s' has %s checksum %s when %s was " \ + messages.append("File: '%s' has %s checksum '%s' when '%s' was= " \ "expected" % (ud.localpath, ci["id"], ci["data= "], ci["expected"])) bad_checksum =3D ci["data"] =20 @@ -853,18 +856,13 @@ def runfetchcmd(cmd, d, quiet=3DFalse, cleanup=3DNone= , log=3DNone, workdir=3DNone): if val: cmd =3D 'export ' + var + '=3D\"%s\"; %s' % (val, cmd) =20 - # Ensure that a _PYTHON_SYSCONFIGDATA_NAME value set by a recipe - # (for example via python3native.bbclass since warrior) is not set for - # host Python (otherwise tools like git-make-shallow will fail) - cmd =3D 'unset _PYTHON_SYSCONFIGDATA_NAME; ' + cmd - # Disable pseudo as it may affect ssh, potentially causing it to hang. cmd =3D 'export PSEUDO_DISABLED=3D1; ' + cmd =20 if workdir: - logger.debug(1, "Running '%s' in %s" % (cmd, workdir)) + logger.debug("Running '%s' in %s" % (cmd, workdir)) else: - logger.debug(1, "Running %s", cmd) + logger.debug("Running %s", cmd) =20 success =3D False error_message =3D "" @@ -873,7 +871,7 @@ def runfetchcmd(cmd, d, quiet=3DFalse, cleanup=3DNone, = log=3DNone, workdir=3DNone): (output, errors) =3D bb.process.run(cmd, log=3Dlog, shell=3DTrue, = stderr=3Dsubprocess.PIPE, cwd=3Dworkdir) success =3D True except bb.process.NotFoundError as e: - error_message =3D "Fetch command %s" % (e.command) + error_message =3D "Fetch command %s not found" % (e.command) except bb.process.ExecutionError as e: if e.stdout: output =3D "output:\n%s\n%s" % (e.stdout, e.stderr) @@ -905,7 +903,7 @@ def check_network_access(d, info, url): elif not trusted_network(d, url): raise UntrustedUrl(url, info) else: - logger.debug(1, "Fetcher accessed the network with the command %s"= % info) + logger.debug("Fetcher accessed the network with the command %s" % = info) =20 def build_mirroruris(origud, mirrors, ld): uris =3D [] @@ -931,7 +929,7 @@ def build_mirroruris(origud, mirrors, ld): continue =20 if not trusted_network(ld, newuri): - logger.debug(1, "Mirror %s not in the list of trusted = networks, skipping" % (newuri)) + logger.debug("Mirror %s not in the list of trusted net= works, skipping" % (newuri)) continue =20 # Create a local copy of the mirrors minus the current line @@ -944,8 +942,8 @@ def build_mirroruris(origud, mirrors, ld): newud =3D FetchData(newuri, ld) newud.setup_localpath(ld) except bb.fetch2.BBFetchException as e: - logger.debug(1, "Mirror fetch failure for url %s (orig= inal url: %s)" % (newuri, origud.url)) - logger.debug(1, str(e)) + logger.debug("Mirror fetch failure for url %s (origina= l url: %s)" % (newuri, origud.url)) + logger.debug(str(e)) try: # setup_localpath of file:// urls may fail, we sho= uld still see # if mirrors of the url exist @@ -1048,8 +1046,8 @@ def try_mirror_url(fetch, origud, ud, ld, check =3D F= alse): elif isinstance(e, NoChecksumError): raise else: - logger.debug(1, "Mirror fetch failure for url %s (original url= : %s)" % (ud.url, origud.url)) - logger.debug(1, str(e)) + logger.debug("Mirror fetch failure for url %s (original url: %= s)" % (ud.url, origud.url)) + logger.debug(str(e)) try: ud.method.clean(ud, ld) except UnboundLocalError: @@ -1195,8 +1193,6 @@ def get_checksum_file_list(d): paths =3D ud.method.localpaths(ud, d) for f in paths: pth =3D ud.decodedurl - if '*' in pth: - f =3D os.path.join(os.path.abspath(f), pth) if f.startswith(dl_dir): # The local fetcher's behaviour is to return a path un= der DL_DIR if it couldn't find the file anywhere else if os.path.exists(f): @@ -1250,7 +1246,7 @@ class FetchData(object): =20 if checksum_name in self.parm: checksum_expected =3D self.parm[checksum_name] - elif self.type not in ["http", "https", "ftp", "ftps", "sftp",= "s3"]: + elif self.type not in ["http", "https", "ftp", "ftps", "sftp",= "s3", "az"]: checksum_expected =3D None else: checksum_expected =3D d.getVarFlag("SRC_URI", checksum_nam= e) @@ -1365,9 +1361,6 @@ class FetchMethod(object): # We cannot compute checksums for directories if os.path.isdir(urldata.localpath): return False - if urldata.localpath.find("*") !=3D -1: - return False - return True =20 def recommends_checksum(self, urldata): @@ -1430,11 +1423,6 @@ class FetchMethod(object): iterate =3D False file =3D urldata.localpath =20 - # Localpath can't deal with 'dir/*' entries, so it converts them t= o '.', - # but it must be corrected back for local files copying - if urldata.basename =3D=3D '*' and file.endswith('/.'): - file =3D '%s/%s' % (file.rstrip('/.'), urldata.path) - try: unpack =3D bb.utils.to_boolean(urldata.parm.get('unpack'), Tru= e) except ValueError as exc: @@ -1471,6 +1459,10 @@ class FetchMethod(object): cmd =3D '7z x -so %s | tar x --no-same-owner -f -' % file elif file.endswith('.7z'): cmd =3D '7za x -y %s 1>/dev/null' % file + elif file.endswith('.tzst') or file.endswith('.tar.zst'): + cmd =3D 'zstd --decompress --stdout %s | tar x --no-same-o= wner -f -' % file + elif file.endswith('.zst'): + cmd =3D 'zstd --decompress --stdout %s > %s' % (file, efil= e) elif file.endswith('.zip') or file.endswith('.jar'): try: dos =3D bb.utils.to_boolean(urldata.parm.get('dos'), F= alse) @@ -1530,7 +1522,7 @@ class FetchMethod(object): if urlpath.find("/") !=3D -1: destdir =3D urlpath.rsplit("/", 1)[0] + '/' bb.utils.mkdirhier("%s/%s" % (unpackdir, destdir)) - cmd =3D 'cp -fpPRH %s %s' % (file, destdir) + cmd =3D 'cp -fpPRH "%s" "%s"' % (file, destdir) =20 if not cmd: return @@ -1613,10 +1605,15 @@ class FetchMethod(object): """ if os.path.exists(ud.localpath): return True - if ud.localpath.find("*") !=3D -1: - return True return False =20 + def implicit_urldata(self, ud, d): + """ + Get a list of FetchData objects for any implicit URLs that will al= so + be downloaded when we fetch the given URL. + """ + return [] + class Fetch(object): def __init__(self, urls, d, cache =3D True, localonly =3D False, conne= ction_cache =3D None): if localonly and cache: @@ -1694,7 +1691,7 @@ class Fetch(object): if m.verify_donestamp(ud, self.d) and not m.need_update(ud= , self.d): done =3D True elif m.try_premirror(ud, self.d): - logger.debug(1, "Trying PREMIRRORS") + logger.debug("Trying PREMIRRORS") mirrors =3D mirror_from_string(self.d.getVar('PREMIRRO= RS')) done =3D m.try_mirrors(self, ud, self.d, mirrors) if done: @@ -1704,7 +1701,7 @@ class Fetch(object): m.update_donestamp(ud, self.d) except ChecksumError as e: logger.warning("Checksum failure encountered w= ith premirror download of %s - will attempt other sources." % u) - logger.debug(1, str(e)) + logger.debug(str(e)) done =3D False =20 if premirroronly: @@ -1716,7 +1713,7 @@ class Fetch(object): try: if not trusted_network(self.d, ud.url): raise UntrustedUrl(ud.url) - logger.debug(1, "Trying Upstream") + logger.debug("Trying Upstream") m.download(ud, self.d) if hasattr(m, "build_mirror_data"): m.build_mirror_data(ud, self.d) @@ -1731,19 +1728,19 @@ class Fetch(object): except BBFetchException as e: if isinstance(e, ChecksumError): logger.warning("Checksum failure encountered w= ith download of %s - will attempt other sources if available" % u) - logger.debug(1, str(e)) + logger.debug(str(e)) if os.path.exists(ud.localpath): rename_bad_checksum(ud, e.checksum) elif isinstance(e, NoChecksumError): raise else: logger.warning('Failed to fetch URL %s, attemp= ting MIRRORS if available' % u) - logger.debug(1, str(e)) + logger.debug(str(e)) firsterr =3D e # Remove any incomplete fetch if not verified_stamp: m.clean(ud, self.d) - logger.debug(1, "Trying MIRRORS") + logger.debug("Trying MIRRORS") mirrors =3D mirror_from_string(self.d.getVar('MIRR= ORS')) done =3D m.try_mirrors(self, ud, self.d, mirrors) =20 @@ -1780,7 +1777,7 @@ class Fetch(object): ud =3D self.ud[u] ud.setup_localpath(self.d) m =3D ud.method - logger.debug(1, "Testing URL %s", u) + logger.debug("Testing URL %s", u) # First try checking uri, u, from PREMIRRORS mirrors =3D mirror_from_string(self.d.getVar('PREMIRRORS')) ret =3D m.try_mirrors(self, ud, self.d, mirrors, True) @@ -1842,6 +1839,24 @@ class Fetch(object): if ud.lockfile: bb.utils.unlockfile(lf) =20 + def expanded_urldata(self, urls=3DNone): + """ + Get an expanded list of FetchData objects covering both the given + URLS and any additional implicit URLs that are added automatically= by + the appropriate FetchMethod. + """ + + if not urls: + urls =3D self.urls + + urldata =3D [] + for url in urls: + ud =3D self.ud[url] + urldata.append(ud) + urldata +=3D ud.method.implicit_urldata(ud, self.d) + + return urldata + class FetchConnectionCache(object): """ A class which represents an container for socket connections. @@ -1896,6 +1911,7 @@ from . import repo from . import clearcase from . import npm from . import npmsw +from . import az =20 methods.append(local.Local()) methods.append(wget.Wget()) @@ -1915,3 +1931,4 @@ methods.append(repo.Repo()) methods.append(clearcase.ClearCase()) methods.append(npm.Npm()) methods.append(npmsw.NpmShrinkWrap()) +methods.append(az.Az()) diff --git a/bitbake/lib/bb/fetch2/az.py b/bitbake/lib/bb/fetch2/az.py new file mode 100644 index 00000000..3ccc594c --- /dev/null +++ b/bitbake/lib/bb/fetch2/az.py @@ -0,0 +1,93 @@ +""" +BitBake 'Fetch' Azure Storage implementation + +""" + +# Copyright (C) 2021 Alejandro Hernandez Samaniego +# +# Based on bb.fetch2.wget: +# Copyright (C) 2003, 2004 Chris Larson +# +# SPDX-License-Identifier: GPL-2.0-only +# +# Based on functions from the base bb module, Copyright 2003 Holger Schurig + +import shlex +import os +import bb +from bb.fetch2 import FetchError +from bb.fetch2 import logger +from bb.fetch2.wget import Wget + + +class Az(Wget): + + def supports(self, ud, d): + """ + Check to see if a given url can be fetched from Azure Storage + """ + return ud.type in ['az'] + + + def checkstatus(self, fetch, ud, d, try_again=3DTrue): + + # checkstatus discards parameters either way, we need to do this b= efore adding the SAS + ud.url =3D ud.url.replace('az://','https://').split(';')[0] + + az_sas =3D d.getVar('AZ_SAS') + if az_sas and az_sas not in ud.url: + ud.url +=3D az_sas + + return Wget.checkstatus(self, fetch, ud, d, try_again) + + # Override download method, include retries + def download(self, ud, d, retries=3D3): + """Fetch urls""" + + # If were reaching the account transaction limit we might be refus= ed a connection, + # retrying allows us to avoid false negatives since the limit chan= ges over time + fetchcmd =3D self.basecmd + ' --retry-connrefused --waitretry=3D5' + + # We need to provide a localpath to avoid wget using the SAS + # ud.localfile either has the downloadfilename or ud.path + localpath =3D os.path.join(d.getVar("DL_DIR"), ud.localfile) + bb.utils.mkdirhier(os.path.dirname(localpath)) + fetchcmd +=3D " -O %s" % shlex.quote(localpath) + + + if ud.user and ud.pswd: + fetchcmd +=3D " --user=3D%s --password=3D%s --auth-no-challeng= e" % (ud.user, ud.pswd) + + # Check if a Shared Access Signature was given and use it + az_sas =3D d.getVar('AZ_SAS') + + if az_sas: + azuri =3D '%s%s%s%s' % ('https://', ud.host, ud.path, az_sas) + else: + azuri =3D '%s%s%s' % ('https://', ud.host, ud.path) + + if os.path.exists(ud.localpath): + # file exists, but we didnt complete it.. trying again. + fetchcmd +=3D d.expand(" -c -P ${DL_DIR} '%s'" % azuri) + else: + fetchcmd +=3D d.expand(" -P ${DL_DIR} '%s'" % azuri) + + try: + self._runwget(ud, d, fetchcmd, False) + except FetchError as e: + # Azure fails on handshake sometimes when using wget after som= e stress, producing a + # FetchError from the fetcher, if the artifact exists retyring= should succeed + if 'Unable to establish SSL connection' in str(e): + logger.debug2('Unable to establish SSL connection: Retries= remaining: %s, Retrying...' % retries) + self.download(ud, d, retries -1) + + # Sanity check since wget can pretend it succeed when it didn't + # Also, this used to happen if sourceforge sent us to the mirror p= age + if not os.path.exists(ud.localpath): + raise FetchError("The fetch command returned success for url %= s but %s doesn't exist?!" % (azuri, ud.localpath), azuri) + + if os.path.getsize(ud.localpath) =3D=3D 0: + os.remove(ud.localpath) + raise FetchError("The fetch of %s resulted in a zero size file= ?! Deleting and failing since this isn't right." % (azuri), azuri) + + return True diff --git a/bitbake/lib/bb/fetch2/bzr.py b/bitbake/lib/bb/fetch2/bzr.py index 566ace9f..fc558f50 100644 --- a/bitbake/lib/bb/fetch2/bzr.py +++ b/bitbake/lib/bb/fetch2/bzr.py @@ -74,16 +74,16 @@ class Bzr(FetchMethod): =20 if os.access(os.path.join(ud.pkgdir, os.path.basename(ud.pkgdir), = '.bzr'), os.R_OK): bzrcmd =3D self._buildbzrcommand(ud, d, "update") - logger.debug(1, "BZR Update %s", ud.url) + logger.debug("BZR Update %s", ud.url) bb.fetch2.check_network_access(d, bzrcmd, ud.url) runfetchcmd(bzrcmd, d, workdir=3Dos.path.join(ud.pkgdir, os.pa= th.basename(ud.path))) else: bb.utils.remove(os.path.join(ud.pkgdir, os.path.basename(ud.pk= gdir)), True) bzrcmd =3D self._buildbzrcommand(ud, d, "fetch") bb.fetch2.check_network_access(d, bzrcmd, ud.url) - logger.debug(1, "BZR Checkout %s", ud.url) + logger.debug("BZR Checkout %s", ud.url) bb.utils.mkdirhier(ud.pkgdir) - logger.debug(1, "Running %s", bzrcmd) + logger.debug("Running %s", bzrcmd) runfetchcmd(bzrcmd, d, workdir=3Dud.pkgdir) =20 scmdata =3D ud.parm.get("scmdata", "") @@ -109,7 +109,7 @@ class Bzr(FetchMethod): """ Return the latest upstream revision number """ - logger.debug(2, "BZR fetcher hitting network for %s", ud.url) + logger.debug2("BZR fetcher hitting network for %s", ud.url) =20 bb.fetch2.check_network_access(d, self._buildbzrcommand(ud, d, "re= vno"), ud.url) =20 diff --git a/bitbake/lib/bb/fetch2/clearcase.py b/bitbake/lib/bb/fetch2/cle= arcase.py index 49d7ae1b..1a9c8637 100644 --- a/bitbake/lib/bb/fetch2/clearcase.py +++ b/bitbake/lib/bb/fetch2/clearcase.py @@ -70,7 +70,7 @@ class ClearCase(FetchMethod): return ud.type in ['ccrc'] =20 def debug(self, msg): - logger.debug(1, "ClearCase: %s", msg) + logger.debug("ClearCase: %s", msg) =20 def urldata_init(self, ud, d): """ diff --git a/bitbake/lib/bb/fetch2/cvs.py b/bitbake/lib/bb/fetch2/cvs.py index 29123a48..01de5ff4 100644 --- a/bitbake/lib/bb/fetch2/cvs.py +++ b/bitbake/lib/bb/fetch2/cvs.py @@ -51,6 +51,10 @@ class Cvs(FetchMethod): =20 ud.localfile =3D d.expand('%s_%s_%s_%s%s%s.tar.gz' % (ud.module.re= place('/', '.'), ud.host, ud.tag, ud.date, norecurse, fullpath)) =20 + pkg =3D d.getVar('PN') + cvsdir =3D d.getVar("CVSDIR") or (d.getVar("DL_DIR") + "/cvs") + ud.pkgdir =3D os.path.join(cvsdir, pkg) + def need_update(self, ud, d): if (ud.date =3D=3D "now"): return True @@ -105,11 +109,8 @@ class Cvs(FetchMethod): cvsupdatecmd =3D "CVS_RSH=3D\"%s\" %s" % (cvs_rsh, cvsupdatecm= d) =20 # create module directory - logger.debug(2, "Fetch: checking for module directory") - pkg =3D d.getVar('PN') - cvsdir =3D d.getVar("CVSDIR") or (d.getVar("DL_DIR") + "/cvs") - pkgdir =3D os.path.join(cvsdir, pkg) - moddir =3D os.path.join(pkgdir, localdir) + logger.debug2("Fetch: checking for module directory") + moddir =3D os.path.join(ud.pkgdir, localdir) workdir =3D None if os.access(os.path.join(moddir, 'CVS'), os.R_OK): logger.info("Update " + ud.url) @@ -120,9 +121,9 @@ class Cvs(FetchMethod): else: logger.info("Fetch " + ud.url) # check out sources there - bb.utils.mkdirhier(pkgdir) - workdir =3D pkgdir - logger.debug(1, "Running %s", cvscmd) + bb.utils.mkdirhier(ud.pkgdir) + workdir =3D ud.pkgdir + logger.debug("Running %s", cvscmd) bb.fetch2.check_network_access(d, cvscmd, ud.url) cmd =3D cvscmd =20 @@ -140,7 +141,7 @@ class Cvs(FetchMethod): # tar them up to a defined filename workdir =3D None if 'fullpath' in ud.parm: - workdir =3D pkgdir + workdir =3D ud.pkgdir cmd =3D "tar %s -czf %s %s" % (tar_flags, ud.localpath, locald= ir) else: workdir =3D os.path.dirname(os.path.realpath(moddir)) @@ -151,9 +152,6 @@ class Cvs(FetchMethod): def clean(self, ud, d): """ Clean CVS Files and tarballs """ =20 - pkg =3D d.getVar('PN') - pkgdir =3D os.path.join(d.getVar("CVSDIR"), pkg) - - bb.utils.remove(pkgdir, True) + bb.utils.remove(ud.pkgdir, True) bb.utils.remove(ud.localpath) =20 diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py index 2ce9395f..d17e2f07 100644 --- a/bitbake/lib/bb/fetch2/git.py +++ b/bitbake/lib/bb/fetch2/git.py @@ -63,10 +63,12 @@ import errno import fnmatch import os import re +import shlex import subprocess import tempfile import bb import bb.progress +from contextlib import contextmanager from bb.fetch2 import FetchMethod from bb.fetch2 import runfetchcmd from bb.fetch2 import logger @@ -140,6 +142,10 @@ class Git(FetchMethod): ud.proto =3D 'file' else: ud.proto =3D "git" + if ud.host =3D=3D "github.com" and ud.proto =3D=3D "git": + # github stopped supporting git protocol + # https://github.blog/2021-09-01-improving-git-protocol-securi= ty-github/#no-more-unauthenticated-git + ud.proto =3D "https" =20 if not ud.proto in ('git', 'file', 'ssh', 'http', 'https', 'rsync'= ): raise bb.fetch2.ParameterError("Invalid protocol type", ud.url) @@ -219,7 +225,12 @@ class Git(FetchMethod): ud.shallow =3D False =20 if ud.usehead: - ud.unresolvedrev['default'] =3D 'HEAD' + # When usehead is set let's associate 'HEAD' with the unresolv= ed + # rev of this repository. This will get resolved into a revisi= on + # later. If an actual revision happens to have also been provi= ded + # then this setting will be overridden. + for name in ud.names: + ud.unresolvedrev[name] =3D 'HEAD' =20 ud.basecmd =3D d.getVar("FETCHCMD_git") or "git -c core.fsyncobjec= tfiles=3D0" =20 @@ -236,7 +247,7 @@ class Git(FetchMethod): ud.unresolvedrev[name] =3D ud.revisions[name] ud.revisions[name] =3D self.latest_revision(ud, d, name) =20 - gitsrcname =3D '%s%s' % (ud.host.replace(':', '.'), ud.path.replac= e('/', '.').replace('*', '.')) + gitsrcname =3D '%s%s' % (ud.host.replace(':', '.'), ud.path.replac= e('/', '.').replace('*', '.').replace(' ','_')) if gitsrcname.startswith('.'): gitsrcname =3D gitsrcname[1:] =20 @@ -342,7 +353,7 @@ class Git(FetchMethod): # We do this since git will use a "-l" option automatically fo= r local urls where possible if repourl.startswith("file://"): repourl =3D repourl[7:] - clone_cmd =3D "LANG=3DC %s clone --bare --mirror %s %s --progr= ess" % (ud.basecmd, repourl, ud.clonedir) + clone_cmd =3D "LANG=3DC %s clone --bare --mirror %s %s --progr= ess" % (ud.basecmd, shlex.quote(repourl), ud.clonedir) if ud.proto.lower() !=3D 'file': bb.fetch2.check_network_access(d, clone_cmd, ud.url) progresshandler =3D GitProgressHandler(d) @@ -354,8 +365,8 @@ class Git(FetchMethod): if "origin" in output: runfetchcmd("%s remote rm origin" % ud.basecmd, d, workdir= =3Dud.clonedir) =20 - runfetchcmd("%s remote add --mirror=3Dfetch origin %s" % (ud.b= asecmd, repourl), d, workdir=3Dud.clonedir) - fetch_cmd =3D "LANG=3DC %s fetch -f --progress %s refs/*:refs/= *" % (ud.basecmd, repourl) + runfetchcmd("%s remote add --mirror=3Dfetch origin %s" % (ud.b= asecmd, shlex.quote(repourl)), d, workdir=3Dud.clonedir) + fetch_cmd =3D "LANG=3DC %s fetch -f --progress %s refs/*:refs/= *" % (ud.basecmd, shlex.quote(repourl)) if ud.proto.lower() !=3D 'file': bb.fetch2.check_network_access(d, fetch_cmd, ud.url) progresshandler =3D GitProgressHandler(d) @@ -378,7 +389,50 @@ class Git(FetchMethod): if missing_rev: raise bb.fetch2.FetchError("Unable to find revision %s eve= n from upstream" % missing_rev) =20 + if self._contains_lfs(ud, d, ud.clonedir) and self._need_lfs(ud): + # Unpack temporary working copy, use it to run 'git checkout' = to force pre-fetching + # of all LFS blobs needed at the the srcrev. + # + # It would be nice to just do this inline here by running 'git= -lfs fetch' + # on the bare clonedir, but that operation requires a working = copy on some + # releases of Git LFS. + tmpdir =3D tempfile.mkdtemp(dir=3Dd.getVar('DL_DIR')) + try: + # Do the checkout. This implicitly involves a Git LFS fetc= h. + Git.unpack(self, ud, tmpdir, d) + + # Scoop up a copy of any stuff that Git LFS downloaded. Me= rge them into + # the bare clonedir. + # + # As this procedure is invoked repeatedly on incremental f= etches as + # a recipe's SRCREV is bumped throughout its lifetime, thi= s will + # result in a gradual accumulation of LFS blobs in <ud.clo= nedir>/lfs + # corresponding to all the blobs reachable from the differ= ent revs + # fetched across time. + # + # Only do this if the unpack resulted in a .git/lfs direct= ory being + # created; this only happens if at least one blob needed t= o be + # downloaded. + if os.path.exists(os.path.join(tmpdir, "git", ".git", "lfs= ")): + runfetchcmd("tar -cf - lfs | tar -xf - -C %s" % ud.clo= nedir, d, workdir=3D"%s/git/.git" % tmpdir) + finally: + bb.utils.remove(tmpdir, recurse=3DTrue) + def build_mirror_data(self, ud, d): + + # Create as a temp file and move atomically into position to avoid= races + @contextmanager + def create_atomic(filename): + fd, tfile =3D tempfile.mkstemp(dir=3Dos.path.dirname(filename)) + try: + yield tfile + umask =3D os.umask(0o666) + os.umask(umask) + os.chmod(tfile, (0o666 & ~umask)) + os.rename(tfile, filename) + finally: + os.close(fd) + if ud.shallow and ud.write_shallow_tarballs: if not os.path.exists(ud.fullshallow): if os.path.islink(ud.fullshallow): @@ -389,7 +443,8 @@ class Git(FetchMethod): self.clone_shallow_local(ud, shallowclone, d) =20 logger.info("Creating tarball of git repository") - runfetchcmd("tar -czf %s ." % ud.fullshallow, d, workd= ir=3Dshallowclone) + with create_atomic(ud.fullshallow) as tfile: + runfetchcmd("tar -czf %s ." % tfile, d, workdir=3D= shallowclone) runfetchcmd("touch %s.done" % ud.fullshallow, d) finally: bb.utils.remove(tempdir, recurse=3DTrue) @@ -398,7 +453,8 @@ class Git(FetchMethod): os.unlink(ud.fullmirror) =20 logger.info("Creating tarball of git repository") - runfetchcmd("tar -czf %s ." % ud.fullmirror, d, workdir=3Dud.c= lonedir) + with create_atomic(ud.fullmirror) as tfile: + runfetchcmd("tar -czf %s ." % tfile, d, workdir=3Dud.clone= dir) runfetchcmd("touch %s.done" % ud.fullmirror, d) =20 def clone_shallow_local(self, ud, dest, d): @@ -473,7 +529,10 @@ class Git(FetchMethod): if os.path.exists(destdir): bb.utils.prunedir(destdir) =20 - need_lfs =3D ud.parm.get("lfs", "1") =3D=3D "1" + need_lfs =3D self._need_lfs(ud) + + if not need_lfs: + ud.basecmd =3D "GIT_LFS_SKIP_SMUDGE=3D1 " + ud.basecmd =20 source_found =3D False source_error =3D [] @@ -501,12 +560,12 @@ class Git(FetchMethod): raise bb.fetch2.UnpackError("No up to date source found: " + "= ; ".join(source_error), ud.url) =20 repourl =3D self._get_repo_url(ud) - runfetchcmd("%s remote set-url origin %s" % (ud.basecmd, repourl),= d, workdir=3Ddestdir) + runfetchcmd("%s remote set-url origin %s" % (ud.basecmd, shlex.quo= te(repourl)), d, workdir=3Ddestdir) =20 if self._contains_lfs(ud, d, destdir): if need_lfs and not self._find_git_lfs(d): raise bb.fetch2.FetchError("Repository %s has LFS content,= install git-lfs on host to download (or set lfs=3D0 to ignore it)" % (repo= url)) - else: + elif not need_lfs: bb.note("Repository %s has LFS content but it is not being= fetched" % (repourl)) =20 if not ud.nocheckout: @@ -559,12 +618,28 @@ class Git(FetchMethod): raise bb.fetch2.FetchError("The command '%s' gave output with = more then 1 line unexpectedly, output: '%s'" % (cmd, output)) return output.split()[0] !=3D "0" =20 + def _need_lfs(self, ud): + return ud.parm.get("lfs", "1") =3D=3D "1" + def _contains_lfs(self, ud, d, wd): """ Check if the repository has 'lfs' (large file) content """ - cmd =3D "%s grep lfs HEAD:.gitattributes | wc -l" % ( - ud.basecmd) + + if not ud.nobranch: + branchname =3D ud.branches[ud.names[0]] + else: + branchname =3D "master" + + # The bare clonedir doesn't use the remote names; it has the branc= h immediately. + if wd =3D=3D ud.clonedir: + refname =3D ud.branches[ud.names[0]] + else: + refname =3D "origin/%s" % ud.branches[ud.names[0]] + + cmd =3D "%s grep lfs %s:.gitattributes | wc -l" % ( + ud.basecmd, refname) + try: output =3D runfetchcmd(cmd, d, quiet=3DTrue, workdir=3Dwd) if int(output) > 0: @@ -584,6 +659,11 @@ class Git(FetchMethod): """ Return the repository URL """ + # Note that we do not support passwords directly in the git urls. = There are several + # reasons. SRC_URI can be written out to things like buildhistory = and people don't + # want to leak passwords like that. Its also all too easy to share= metadata without=20 + # removing the password. ssh keys, ~/.netrc and ~/.ssh/config file= s can be used as + # alternatives so we will not take patches adding password support= here. if ud.user: username =3D ud.user + '@' else: @@ -614,7 +694,7 @@ class Git(FetchMethod): try: repourl =3D self._get_repo_url(ud) cmd =3D "%s ls-remote %s %s" % \ - (ud.basecmd, repourl, search) + (ud.basecmd, shlex.quote(repourl), search) if ud.proto.lower() !=3D 'file': bb.fetch2.check_network_access(d, cmd, repourl) output =3D runfetchcmd(cmd, d, True) diff --git a/bitbake/lib/bb/fetch2/gitsm.py b/bitbake/lib/bb/fetch2/gitsm.py index e7083001..a4527bf3 100644 --- a/bitbake/lib/bb/fetch2/gitsm.py +++ b/bitbake/lib/bb/fetch2/gitsm.py @@ -78,7 +78,7 @@ class GitSM(Git): module_hash =3D "" =20 if not module_hash: - logger.debug(1, "submodule %s is defined, but is not i= nitialized in the repository. Skipping", m) + logger.debug("submodule %s is defined, but is not init= ialized in the repository. Skipping", m) continue =20 submodules.append(m) @@ -143,12 +143,43 @@ class GitSM(Git): try: # Check for the nugget dropped by the download operation known_srcrevs =3D runfetchcmd("%s config --get-all bitbake.src= rev" % \ - (ud.basecmd), d, workdir=3Dud.clonedir) + (ud.basecmd), d, workdir=3Dud.clon= edir) =20 - if ud.revisions[ud.names[0]] not in known_srcrevs.split(): - return True + if ud.revisions[ud.names[0]] in known_srcrevs.split(): + return False except bb.fetch2.FetchError: - # No srcrev nuggets, so this is new and needs to be updated + pass + + need_update_list =3D [] + def need_update_submodule(ud, url, module, modpath, workdir, d): + url +=3D ";bareclone=3D1;nobranch=3D1" + + try: + newfetch =3D Fetch([url], d, cache=3DFalse) + new_ud =3D newfetch.ud[url] + if new_ud.method.need_update(new_ud, d): + need_update_list.append(modpath) + except Exception as e: + logger.error('gitsm: submodule update check failed: %s %s'= % (type(e).__name__, str(e))) + need_update_result =3D True + + # If we're using a shallow mirror tarball it needs to be unpacked + # temporarily so that we can examine the .gitmodules file + if ud.shallow and os.path.exists(ud.fullshallow) and not os.path.e= xists(ud.clonedir): + tmpdir =3D tempfile.mkdtemp(dir=3Dd.getVar("DL_DIR")) + runfetchcmd("tar -xzf %s" % ud.fullshallow, d, workdir=3Dtmpdi= r) + self.process_submodules(ud, tmpdir, need_update_submodule, d) + shutil.rmtree(tmpdir) + else: + self.process_submodules(ud, ud.clonedir, need_update_submodule= , d) + if len(need_update_list) =3D=3D 0: + # We already have the required commits of all submodules. = Drop + # a nugget so we don't need to check again. + runfetchcmd("%s config --add bitbake.srcrev %s" % \ + (ud.basecmd, ud.revisions[ud.names[0]]), d, wo= rkdir=3Dud.clonedir) + + if len(need_update_list) > 0: + logger.debug('gitsm: Submodules requiring update: %s' % (' '.j= oin(need_update_list))) return True =20 return False @@ -163,9 +194,6 @@ class GitSM(Git): try: newfetch =3D Fetch([url], d, cache=3DFalse) newfetch.download() - # Drop a nugget to add each of the srcrevs we've fetched (= used by need_update) - runfetchcmd("%s config --add bitbake.srcrev %s" % \ - (ud.basecmd, ud.revisions[ud.names[0]]), d, wo= rkdir=3Dworkdir) except Exception as e: logger.error('gitsm: submodule download failed: %s %s' % (= type(e).__name__, str(e))) raise @@ -181,6 +209,9 @@ class GitSM(Git): shutil.rmtree(tmpdir) else: self.process_submodules(ud, ud.clonedir, download_submodule, d) + # Drop a nugget for the srcrev we've fetched (used by need_upd= ate) + runfetchcmd("%s config --add bitbake.srcrev %s" % \ + (ud.basecmd, ud.revisions[ud.names[0]]), d, workdi= r=3Dud.clonedir) =20 def unpack(self, ud, destdir, d): def unpack_submodules(ud, url, module, modpath, workdir, d): @@ -223,3 +254,24 @@ class GitSM(Git): # up the configuration and checks out the files. The main pro= ject config should remain # unmodified, and no download from the internet should occur. runfetchcmd("%s submodule update --recursive --no-fetch" % (ud= .basecmd), d, quiet=3DTrue, workdir=3Dud.destdir) + + def implicit_urldata(self, ud, d): + import shutil, subprocess, tempfile + + urldata =3D [] + def add_submodule(ud, url, module, modpath, workdir, d): + url +=3D ";bareclone=3D1;nobranch=3D1" + newfetch =3D Fetch([url], d, cache=3DFalse) + urldata.extend(newfetch.expanded_urldata()) + + # If we're using a shallow mirror tarball it needs to be unpacked + # temporarily so that we can examine the .gitmodules file + if ud.shallow and os.path.exists(ud.fullshallow) and ud.method.nee= d_update(ud, d): + tmpdir =3D tempfile.mkdtemp(dir=3Dd.getVar("DL_DIR")) + subprocess.check_call("tar -xzf %s" % ud.fullshallow, cwd=3Dtm= pdir, shell=3DTrue) + self.process_submodules(ud, tmpdir, add_submodule, d) + shutil.rmtree(tmpdir) + else: + self.process_submodules(ud, ud.clonedir, add_submodule, d) + + return urldata diff --git a/bitbake/lib/bb/fetch2/hg.py b/bitbake/lib/bb/fetch2/hg.py index 8f503701..063e1300 100644 --- a/bitbake/lib/bb/fetch2/hg.py +++ b/bitbake/lib/bb/fetch2/hg.py @@ -150,7 +150,7 @@ class Hg(FetchMethod): def download(self, ud, d): """Fetch url""" =20 - logger.debug(2, "Fetch: checking for module directory '" + ud.modd= ir + "'") + logger.debug2("Fetch: checking for module directory '" + ud.moddir= + "'") =20 # If the checkout doesn't exist and the mirror tarball does, extra= ct it if not os.path.exists(ud.pkgdir) and os.path.exists(ud.fullmirror): @@ -160,7 +160,7 @@ class Hg(FetchMethod): if os.access(os.path.join(ud.moddir, '.hg'), os.R_OK): # Found the source, check whether need pull updatecmd =3D self._buildhgcommand(ud, d, "update") - logger.debug(1, "Running %s", updatecmd) + logger.debug("Running %s", updatecmd) try: runfetchcmd(updatecmd, d, workdir=3Dud.moddir) except bb.fetch2.FetchError: @@ -168,7 +168,7 @@ class Hg(FetchMethod): pullcmd =3D self._buildhgcommand(ud, d, "pull") logger.info("Pulling " + ud.url) # update sources there - logger.debug(1, "Running %s", pullcmd) + logger.debug("Running %s", pullcmd) bb.fetch2.check_network_access(d, pullcmd, ud.url) runfetchcmd(pullcmd, d, workdir=3Dud.moddir) try: @@ -183,14 +183,14 @@ class Hg(FetchMethod): logger.info("Fetch " + ud.url) # check out sources there bb.utils.mkdirhier(ud.pkgdir) - logger.debug(1, "Running %s", fetchcmd) + logger.debug("Running %s", fetchcmd) bb.fetch2.check_network_access(d, fetchcmd, ud.url) runfetchcmd(fetchcmd, d, workdir=3Dud.pkgdir) =20 # Even when we clone (fetch), we still need to update as hg's clone # won't checkout the specified revision if its on a branch updatecmd =3D self._buildhgcommand(ud, d, "update") - logger.debug(1, "Running %s", updatecmd) + logger.debug("Running %s", updatecmd) runfetchcmd(updatecmd, d, workdir=3Dud.moddir) =20 def clean(self, ud, d): @@ -247,9 +247,9 @@ class Hg(FetchMethod): if scmdata !=3D "nokeep": proto =3D ud.parm.get('protocol', 'http') if not os.access(os.path.join(codir, '.hg'), os.R_OK): - logger.debug(2, "Unpack: creating new hg repository in '" = + codir + "'") + logger.debug2("Unpack: creating new hg repository in '" + = codir + "'") runfetchcmd("%s init %s" % (ud.basecmd, codir), d) - logger.debug(2, "Unpack: updating source in '" + codir + "'") + logger.debug2("Unpack: updating source in '" + codir + "'") if ud.user and ud.pswd: runfetchcmd("%s --config auth.default.prefix=3D* --config = auth.default.username=3D%s --config auth.default.password=3D%s --config \"a= uth.default.schemes=3D%s\" pull %s" % (ud.basecmd, ud.user, ud.pswd, proto,= ud.moddir), d, workdir=3Dcodir) else: @@ -259,5 +259,5 @@ class Hg(FetchMethod): else: runfetchcmd("%s up -C %s" % (ud.basecmd, revflag), d, work= dir=3Dcodir) else: - logger.debug(2, "Unpack: extracting source to '" + codir + "'") + logger.debug2("Unpack: extracting source to '" + codir + "'") runfetchcmd("%s archive -t files %s %s" % (ud.basecmd, revflag= , codir), d, workdir=3Dud.moddir) diff --git a/bitbake/lib/bb/fetch2/local.py b/bitbake/lib/bb/fetch2/local.py index 01d9ff9f..e7d1c8c5 100644 --- a/bitbake/lib/bb/fetch2/local.py +++ b/bitbake/lib/bb/fetch2/local.py @@ -17,7 +17,7 @@ import os import urllib.request, urllib.parse, urllib.error import bb import bb.utils -from bb.fetch2 import FetchMethod, FetchError +from bb.fetch2 import FetchMethod, FetchError, ParameterError from bb.fetch2 import logger =20 class Local(FetchMethod): @@ -33,6 +33,8 @@ class Local(FetchMethod): ud.basename =3D os.path.basename(ud.decodedurl) ud.basepath =3D ud.decodedurl ud.needdonestamp =3D False + if "*" in ud.decodedurl: + raise bb.fetch2.ParameterError("file:// urls using globbing ar= e no longer supported. Please place the files in a directory and reference = that instead.", ud.url) return =20 def localpath(self, urldata, d): @@ -52,26 +54,18 @@ class Local(FetchMethod): return [path] filespath =3D d.getVar('FILESPATH') if filespath: - logger.debug(2, "Searching for %s in paths:\n %s" % (path, = "\n ".join(filespath.split(":")))) + logger.debug2("Searching for %s in paths:\n %s" % (path, "\= n ".join(filespath.split(":")))) newpath, hist =3D bb.utils.which(filespath, path, history=3DTr= ue) searched.extend(hist) - if (not newpath or not os.path.exists(newpath)) and path.find("*")= !=3D -1: - # For expressions using '*', best we can do is take the first = directory in FILESPATH that exists - newpath, hist =3D bb.utils.which(filespath, ".", history=3DTru= e) - searched.extend(hist) - logger.debug(2, "Searching for %s in path: %s" % (path, newpat= h)) - return searched if not os.path.exists(newpath): dldirfile =3D os.path.join(d.getVar("DL_DIR"), path) - logger.debug(2, "Defaulting to %s for %s" % (dldirfile, path)) + logger.debug2("Defaulting to %s for %s" % (dldirfile, path)) bb.utils.mkdirhier(os.path.dirname(dldirfile)) searched.append(dldirfile) return searched return searched =20 def need_update(self, ud, d): - if ud.url.find("*") !=3D -1: - return False if os.path.exists(ud.localpath): return False return True @@ -95,9 +89,6 @@ class Local(FetchMethod): """ Check the status of the url """ - if urldata.localpath.find("*") !=3D -1: - logger.info("URL %s looks like a glob and was therefore not ch= ecked.", urldata.url) - return True if os.path.exists(urldata.localpath): return True return False diff --git a/bitbake/lib/bb/fetch2/npmsw.py b/bitbake/lib/bb/fetch2/npmsw.py index 0c3511d8..fdecbc6d 100644 --- a/bitbake/lib/bb/fetch2/npmsw.py +++ b/bitbake/lib/bb/fetch2/npmsw.py @@ -29,6 +29,8 @@ from bb.fetch2.npm import npm_integrity from bb.fetch2.npm import npm_localfile from bb.fetch2.npm import npm_unpack from bb.utils import is_semver +from bb.utils import lockfile +from bb.utils import unlockfile =20 def foreach_dependencies(shrinkwrap, callback=3DNone, dev=3DFalse): """ @@ -187,7 +189,9 @@ class NpmShrinkWrap(FetchMethod): proxy_ud =3D ud.proxy.ud[proxy_url] proxy_d =3D ud.proxy.d proxy_ud.setup_localpath(proxy_d) + lf =3D lockfile(proxy_ud.lockfile) returns.append(handle(proxy_ud.method, proxy_ud, proxy_d)) + unlockfile(lf) return returns =20 def verify_donestamp(self, ud, d): diff --git a/bitbake/lib/bb/fetch2/osc.py b/bitbake/lib/bb/fetch2/osc.py index 8f091efd..d9ce4439 100644 --- a/bitbake/lib/bb/fetch2/osc.py +++ b/bitbake/lib/bb/fetch2/osc.py @@ -8,12 +8,15 @@ Based on the svn "Fetch" implementation. """ =20 import logging +import os import bb from bb.fetch2 import FetchMethod from bb.fetch2 import FetchError from bb.fetch2 import MissingParameterError from bb.fetch2 import runfetchcmd =20 +logger =3D logging.getLogger(__name__) + class Osc(FetchMethod): """Class to fetch a module or modules from Opensuse build server repositories.""" @@ -81,13 +84,13 @@ class Osc(FetchMethod): Fetch url """ =20 - logger.debug(2, "Fetch: checking for module directory '" + ud.modd= ir + "'") + logger.debug2("Fetch: checking for module directory '" + ud.moddir= + "'") =20 if os.access(os.path.join(d.getVar('OSCDIR'), ud.path, ud.module),= os.R_OK): oscupdatecmd =3D self._buildosccommand(ud, d, "update") logger.info("Update "+ ud.url) # update sources there - logger.debug(1, "Running %s", oscupdatecmd) + logger.debug("Running %s", oscupdatecmd) bb.fetch2.check_network_access(d, oscupdatecmd, ud.url) runfetchcmd(oscupdatecmd, d, workdir=3Dud.moddir) else: @@ -95,7 +98,7 @@ class Osc(FetchMethod): logger.info("Fetch " + ud.url) # check out sources there bb.utils.mkdirhier(ud.pkgdir) - logger.debug(1, "Running %s", oscfetchcmd) + logger.debug("Running %s", oscfetchcmd) bb.fetch2.check_network_access(d, oscfetchcmd, ud.url) runfetchcmd(oscfetchcmd, d, workdir=3Dud.pkgdir) =20 diff --git a/bitbake/lib/bb/fetch2/perforce.py b/bitbake/lib/bb/fetch2/perf= orce.py index f57c2a4f..e2a41a4a 100644 --- a/bitbake/lib/bb/fetch2/perforce.py +++ b/bitbake/lib/bb/fetch2/perforce.py @@ -1,6 +1,20 @@ """ BitBake 'Fetch' implementation for perforce =20 +Supported SRC_URI options are: + +- module + The top-level location to fetch while preserving the remote paths + + The value of module can point to either a directory or a file. The resu= lt, + in both cases, is that the fetcher will preserve all file paths starting + from the module path. That is, the top-level directory in the module va= lue + will also be the top-level directory in P4DIR. + +- remotepath + If the value "keep" is given, the full depot location of each file is + preserved in P4DIR. This option overrides the effect of the module opti= on. + """ =20 # Copyright (C) 2003, 2004 Chris Larson @@ -17,6 +31,36 @@ from bb.fetch2 import FetchError from bb.fetch2 import logger from bb.fetch2 import runfetchcmd =20 +class PerforceProgressHandler (bb.progress.BasicProgressHandler): + """ + Implements basic progress information for perforce, based on the numbe= r of + files to be downloaded. + + The p4 print command will print one line per file, therefore it can be= used + to "count" the number of files already completed and give an indicatio= n of + the progress. + """ + def __init__(self, d, num_files): + self._num_files =3D num_files + self._count =3D 0 + super(PerforceProgressHandler, self).__init__(d) + + # Send an initial progress event so the bar gets shown + self._fire_progress(-1) + + def write(self, string): + self._count =3D self._count + 1 + + percent =3D int(100.0 * float(self._count) / float(self._num_files= )) + + # In case something goes wrong, we try to preserve our sanity + if percent > 100: + percent =3D 100 + + self.update(percent) + + super(PerforceProgressHandler, self).write(string) + class Perforce(FetchMethod): """ Class to fetch from perforce repositories """ def supports(self, ud, d): @@ -46,31 +90,51 @@ class Perforce(FetchMethod): p4port =3D d.getVar('P4PORT') =20 if p4port: - logger.debug(1, 'Using recipe provided P4PORT: %s' % p4port) + logger.debug('Using recipe provided P4PORT: %s' % p4port) ud.host =3D p4port else: - logger.debug(1, 'Trying to use P4CONFIG to automatically set P= 4PORT...') + logger.debug('Trying to use P4CONFIG to automatically set P4PO= RT...') ud.usingp4config =3D True p4cmd =3D '%s info | grep "Server address"' % ud.basecmd bb.fetch2.check_network_access(d, p4cmd, ud.url) ud.host =3D runfetchcmd(p4cmd, d, True) ud.host =3D ud.host.split(': ')[1].strip() - logger.debug(1, 'Determined P4PORT to be: %s' % ud.host) + logger.debug('Determined P4PORT to be: %s' % ud.host) if not ud.host: raise FetchError('Could not determine P4PORT from P4CONFIG= ') -=09 + + # Fetcher options + ud.module =3D ud.parm.get('module') + ud.keepremotepath =3D (ud.parm.get('remotepath', '') =3D=3D 'keep') + if ud.path.find('/...') >=3D 0: ud.pathisdir =3D True else: ud.pathisdir =3D False =20 + # Avoid using the "/..." syntax in SRC_URI when a module value is = given + if ud.pathisdir and ud.module: + raise FetchError('SRC_URI depot path cannot not end in /... wh= en a module value is given') + cleanedpath =3D ud.path.replace('/...', '').replace('/', '.') cleanedhost =3D ud.host.replace(':', '.') + + cleanedmodule =3D "" + # Merge the path and module into the final depot location + if ud.module: + if ud.module.find('/') =3D=3D 0: + raise FetchError('module cannot begin with /') + ud.path =3D os.path.join(ud.path, ud.module) + + # Append the module path to the local pkg name + cleanedmodule =3D ud.module.replace('/...', '').replace('/', '= .') + cleanedpath +=3D '--%s' % cleanedmodule + ud.pkgdir =3D os.path.join(ud.dldir, cleanedhost, cleanedpath) =20 ud.setup_revisions(d) =20 - ud.localfile =3D d.expand('%s_%s_%s.tar.gz' % (cleanedhost, cleane= dpath, ud.revision)) + ud.localfile =3D d.expand('%s_%s_%s_%s.tar.gz' % (cleanedhost, cle= anedpath, cleandedmodule, ud.revision)) =20 def _buildp4command(self, ud, d, command, depot_filename=3DNone): """ @@ -95,10 +159,20 @@ class Perforce(FetchMethod): pathnrev =3D '%s' % (ud.path) =20 if depot_filename: - if ud.pathisdir: # Remove leading path to obtain filename + if ud.keepremotepath: + # preserve everything, remove the leading // + filename =3D depot_filename.lstrip('/') + elif ud.module: + # remove everything up to the module path + modulepath =3D ud.module.rstrip('/...') + filename =3D depot_filename[depot_filename.rfind(modulepat= h):] + elif ud.pathisdir: + # Remove leading (visible) path to obtain the filepath filename =3D depot_filename[len(ud.path)-1:] else: + # Remove everything, except the filename filename =3D depot_filename[depot_filename.rfind('/'):] + filename =3D filename[:filename.find('#')] # Remove trailing '= #rev' =20 if command =3D=3D 'changes': @@ -134,7 +208,7 @@ class Perforce(FetchMethod): for filename in p4fileslist: item =3D filename.split(' - ') lastaction =3D item[1].split() - logger.debug(1, 'File: %s Last Action: %s' % (item[0], lastact= ion[0])) + logger.debug('File: %s Last Action: %s' % (item[0], lastaction= [0])) if lastaction[0] =3D=3D 'delete': continue filelist.append(item[0]) @@ -150,10 +224,12 @@ class Perforce(FetchMethod): bb.utils.remove(ud.pkgdir, True) bb.utils.mkdirhier(ud.pkgdir) =20 + progresshandler =3D PerforceProgressHandler(d, len(filelist)) + for afile in filelist: p4fetchcmd =3D self._buildp4command(ud, d, 'print', afile) bb.fetch2.check_network_access(d, p4fetchcmd, ud.url) - runfetchcmd(p4fetchcmd, d, workdir=3Dud.pkgdir) + runfetchcmd(p4fetchcmd, d, workdir=3Dud.pkgdir, log=3Dprogress= handler) =20 runfetchcmd('tar -czf %s p4' % (ud.localpath), d, cleanup=3D[ud.lo= calpath], workdir=3Dud.pkgdir) =20 @@ -179,7 +255,7 @@ class Perforce(FetchMethod): raise FetchError('Could not determine the latest perforce chan= gelist') =20 tipcset =3D tip.split(' ')[1] - logger.debug(1, 'p4 tip found to be changelist %s' % tipcset) + logger.debug('p4 tip found to be changelist %s' % tipcset) return tipcset =20 def sortable_revision(self, ud, d, name): diff --git a/bitbake/lib/bb/fetch2/repo.py b/bitbake/lib/bb/fetch2/repo.py index 2bdbbd40..fa4cb814 100644 --- a/bitbake/lib/bb/fetch2/repo.py +++ b/bitbake/lib/bb/fetch2/repo.py @@ -47,7 +47,7 @@ class Repo(FetchMethod): """Fetch url""" =20 if os.access(os.path.join(d.getVar("DL_DIR"), ud.localfile), os.R_= OK): - logger.debug(1, "%s already exists (or was stashed). Skipping = repo init / sync.", ud.localpath) + logger.debug("%s already exists (or was stashed). Skipping rep= o init / sync.", ud.localpath) return =20 repodir =3D d.getVar("REPODIR") or (d.getVar("DL_DIR") + "/repo") diff --git a/bitbake/lib/bb/fetch2/ssh.py b/bitbake/lib/bb/fetch2/ssh.py index 5e982ecf..2c8557e1 100644 --- a/bitbake/lib/bb/fetch2/ssh.py +++ b/bitbake/lib/bb/fetch2/ssh.py @@ -31,8 +31,7 @@ IETF secsh internet draft: # =20 import re, os -from bb.fetch2 import FetchMethod -from bb.fetch2 import runfetchcmd +from bb.fetch2 import check_network_access, FetchMethod, ParameterError, r= unfetchcmd =20 =20 __pattern__ =3D re.compile(r''' @@ -65,7 +64,7 @@ class SSH(FetchMethod): =20 def urldata_init(self, urldata, d): if 'protocol' in urldata.parm and urldata.parm['protocol'] =3D=3D = 'git': - raise bb.fetch2.ParameterError( + raise ParameterError( "Invalid protocol - if you wish to fetch from a git " + "repository using ssh, you need to use " + "git:// prefix with protocol=3Dssh", urldata.url) @@ -105,7 +104,7 @@ class SSH(FetchMethod): dldir ) =20 - bb.fetch2.check_network_access(d, cmd, urldata.url) + check_network_access(d, cmd, urldata.url) =20 runfetchcmd(cmd, d) =20 diff --git a/bitbake/lib/bb/fetch2/svn.py b/bitbake/lib/bb/fetch2/svn.py index 971a5add..80102b44 100644 --- a/bitbake/lib/bb/fetch2/svn.py +++ b/bitbake/lib/bb/fetch2/svn.py @@ -86,7 +86,7 @@ class Svn(FetchMethod): if command =3D=3D "info": svncmd =3D "%s info %s %s://%s/%s/" % (ud.basecmd, " ".join(op= tions), proto, svnroot, ud.module) elif command =3D=3D "log1": - svncmd =3D "%s log --limit 1 %s %s://%s/%s/" % (ud.basecmd, " = ".join(options), proto, svnroot, ud.module) + svncmd =3D "%s log --limit 1 --quiet %s %s://%s/%s/" % (ud.bas= ecmd, " ".join(options), proto, svnroot, ud.module) else: suffix =3D "" =20 @@ -116,7 +116,7 @@ class Svn(FetchMethod): def download(self, ud, d): """Fetch url""" =20 - logger.debug(2, "Fetch: checking for module directory '" + ud.modd= ir + "'") + logger.debug2("Fetch: checking for module directory '" + ud.moddir= + "'") =20 lf =3D bb.utils.lockfile(ud.svnlock) =20 @@ -129,7 +129,7 @@ class Svn(FetchMethod): runfetchcmd(ud.basecmd + " upgrade", d, workdir=3Dud.m= oddir) except FetchError: pass - logger.debug(1, "Running %s", svncmd) + logger.debug("Running %s", svncmd) bb.fetch2.check_network_access(d, svncmd, ud.url) runfetchcmd(svncmd, d, workdir=3Dud.moddir) else: @@ -137,7 +137,7 @@ class Svn(FetchMethod): logger.info("Fetch " + ud.url) # check out sources there bb.utils.mkdirhier(ud.pkgdir) - logger.debug(1, "Running %s", svncmd) + logger.debug("Running %s", svncmd) bb.fetch2.check_network_access(d, svncmd, ud.url) runfetchcmd(svncmd, d, workdir=3Dud.pkgdir) =20 diff --git a/bitbake/lib/bb/fetch2/wget.py b/bitbake/lib/bb/fetch2/wget.py index f7d1de26..784df70c 100644 --- a/bitbake/lib/bb/fetch2/wget.py +++ b/bitbake/lib/bb/fetch2/wget.py @@ -52,6 +52,12 @@ class WgetProgressHandler(bb.progress.LineFilterProgress= Handler): =20 =20 class Wget(FetchMethod): + + # CDNs like CloudFlare may do a 'browser integrity test' which can fail + # with the standard wget/urllib User-Agent, so pretend to be a modern + # browser. + user_agent =3D "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko= /20100101 Firefox/84.0" + """Class to fetch urls via 'wget'""" def supports(self, ud, d): """ @@ -82,7 +88,7 @@ class Wget(FetchMethod): =20 progresshandler =3D WgetProgressHandler(d) =20 - logger.debug(2, "Fetching %s using command '%s'" % (ud.url, comman= d)) + logger.debug2("Fetching %s using command '%s'" % (ud.url, command)) bb.fetch2.check_network_access(d, command, ud.url) runfetchcmd(command + ' --progress=3Ddot -v', d, quiet, log=3Dprog= resshandler, workdir=3Dworkdir) =20 @@ -208,10 +214,7 @@ class Wget(FetchMethod): fetch.connection_cache.remove_connection(h.host, h= .port) raise urllib.error.URLError(err) else: - try: - r =3D h.getresponse(buffering=3DTrue) - except TypeError: # buffering kw not supported - r =3D h.getresponse() + r =3D h.getresponse() =20 # Pick apart the HTTPResponse object to get the addinfourl # object initialized properly. @@ -300,7 +303,7 @@ class Wget(FetchMethod): # Some servers (FusionForge, as used on Alioth) require that t= he # optional Accept header is set. r.add_header("Accept", "*/*") - r.add_header("User-Agent", "Mozilla/5.0 (X11; U; Linux i686; e= n-US; rv:1.9.2.12) Gecko/20101027 Ubuntu/9.10 (karmic) Firefox/3.6.12") + r.add_header("User-Agent", self.user_agent) def add_basic_auth(login_str, request): '''Adds Basic auth to http request, pass in login:password= as string''' import base64 @@ -323,11 +326,19 @@ class Wget(FetchMethod): pass except urllib.error.URLError as e: if try_again: - logger.debug(2, "checkstatus: trying again") + logger.debug2("checkstatus: trying again") + return self.checkstatus(fetch, ud, d, False) + else: + # debug for now to avoid spamming the logs in e.g. remote = sstate searches + logger.debug2("checkstatus() urlopen failed: %s" % e) + return False + except ConnectionResetError as e: + if try_again: + logger.debug2("checkstatus: trying again") return self.checkstatus(fetch, ud, d, False) else: # debug for now to avoid spamming the logs in e.g. remote = sstate searches - logger.debug(2, "checkstatus() urlopen failed: %s" % e) + logger.debug2("checkstatus() urlopen failed: %s" % e) return False return True =20 @@ -404,9 +415,8 @@ class Wget(FetchMethod): """ f =3D tempfile.NamedTemporaryFile() with tempfile.TemporaryDirectory(prefix=3D"wget-index-") as workdi= r, tempfile.NamedTemporaryFile(dir=3Dworkdir, prefix=3D"wget-listing-") as = f: - agent =3D "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.12= ) Gecko/20101027 Ubuntu/9.10 (karmic) Firefox/3.6.12" fetchcmd =3D self.basecmd - fetchcmd +=3D " -O " + f.name + " --user-agent=3D'" + agent + = "' '" + uri + "'" + fetchcmd +=3D " -O " + f.name + " --user-agent=3D'" + self.use= r_agent + "' '" + uri + "'" try: self._runwget(ud, d, fetchcmd, True, workdir=3Dworkdir) fetchresult =3D f.read() @@ -462,7 +472,7 @@ class Wget(FetchMethod): version_dir =3D ['', '', ''] version =3D ['', '', ''] =20 - dirver_regex =3D re.compile(r"(?P<pfx>\D*)(?P<ver>(\d+[\.\-_])+(\d= +))") + dirver_regex =3D re.compile(r"(?P<pfx>\D*)(?P<ver>(\d+[\.\-_])*(\d= +))") s =3D dirver_regex.search(dirver) if s: version_dir[1] =3D s.group('ver') diff --git a/bitbake/lib/bb/main.py b/bitbake/lib/bb/main.py index af2880f8..06bad495 100755 --- a/bitbake/lib/bb/main.py +++ b/bitbake/lib/bb/main.py @@ -119,178 +119,181 @@ warnings.filterwarnings("ignore", category=3DImport= Warning) warnings.filterwarnings("ignore", category=3DDeprecationWarning, module=3D= "<string>$") warnings.filterwarnings("ignore", message=3D"With-statements now directly = support multiple context managers") =20 -class BitBakeConfigParameters(cookerdata.ConfigParameters): =20 - def parseCommandLine(self, argv=3Dsys.argv): - parser =3D optparse.OptionParser( - formatter=3DBitbakeHelpFormatter(), - version=3D"BitBake Build Tool Core version %s" % bb.__version_= _, - usage=3D"""%prog [options] [recipename/target recipe:do_task .= ..] +def create_bitbake_parser(): + parser =3D optparse.OptionParser( + formatter=3DBitbakeHelpFormatter(), + version=3D"BitBake Build Tool Core version %s" % bb.__version__, + usage=3D"""%prog [options] [recipename/target recipe:do_task ...] =20 Executes the specified task (default is 'build') for a given set of ta= rget recipes (.bb files). It is assumed there is a conf/bblayers.conf available in cwd or in BBP= ATH which will provide the layer, BBFILES and other configuration information.""= ") =20 - parser.add_option("-b", "--buildfile", action=3D"store", dest=3D"b= uildfile", default=3DNone, - help=3D"Execute tasks from a specific .bb recipe= directly. WARNING: Does " - "not handle any dependencies from other rec= ipes.") - - parser.add_option("-k", "--continue", action=3D"store_false", dest= =3D"abort", default=3DTrue, - help=3D"Continue as much as possible after an er= ror. While the target that " - "failed and anything depending on it cannot= be built, as much as " - "possible will be built before stopping.") - - parser.add_option("-f", "--force", action=3D"store_true", dest=3D"= force", default=3DFalse, - help=3D"Force the specified targets/task to run = (invalidating any " - "existing stamp file).") - - parser.add_option("-c", "--cmd", action=3D"store", dest=3D"cmd", - help=3D"Specify the task to execute. The exact o= ptions available " - "depend on the metadata. Some examples migh= t be 'compile'" - " or 'populate_sysroot' or 'listtasks' may = give a list of " - "the tasks available.") - - parser.add_option("-C", "--clear-stamp", action=3D"store", dest=3D= "invalidate_stamp", - help=3D"Invalidate the stamp for the specified t= ask such as 'compile' " - "and then run the default task for the spec= ified target(s).") - - parser.add_option("-r", "--read", action=3D"append", dest=3D"prefi= le", default=3D[], - help=3D"Read the specified file before bitbake.c= onf.") - - parser.add_option("-R", "--postread", action=3D"append", dest=3D"p= ostfile", default=3D[], - help=3D"Read the specified file after bitbake.co= nf.") - - parser.add_option("-v", "--verbose", action=3D"store_true", dest= =3D"verbose", default=3DFalse, - help=3D"Enable tracing of shell tasks (with 'set= -x'). " - "Also print bb.note(...) messages to stdout= (in " - "addition to writing them to ${T}/log.do_<t= ask>).") - - parser.add_option("-D", "--debug", action=3D"count", dest=3D"debug= ", default=3D0, - help=3D"Increase the debug level. You can specif= y this " - "more than once. -D sets the debug level to= 1, " - "where only bb.debug(1, ...) messages are p= rinted " - "to stdout; -DD sets the debug level to 2, = where " - "both bb.debug(1, ...) and bb.debug(2, ...)= " - "messages are printed; etc. Without -D, no = debug " - "messages are printed. Note that -D only af= fects " - "output to stdout. All debug messages are w= ritten " - "to ${T}/log.do_taskname, regardless of the= debug " - "level.") - - parser.add_option("-q", "--quiet", action=3D"count", dest=3D"quiet= ", default=3D0, - help=3D"Output less log message data to the term= inal. You can specify this more than once.") - - parser.add_option("-n", "--dry-run", action=3D"store_true", dest= =3D"dry_run", default=3DFalse, - help=3D"Don't execute, just go through the motio= ns.") - - parser.add_option("-S", "--dump-signatures", action=3D"append", de= st=3D"dump_signatures", - default=3D[], metavar=3D"SIGNATURE_HANDLER", - help=3D"Dump out the signature construction info= rmation, with no task " - "execution. The SIGNATURE_HANDLER parameter= is passed to the " - "handler. Two common values are none and pr= intdiff but the handler " - "may define more/less. none means only dump= the signature, printdiff" - " means compare the dumped signature with t= he cached one.") - - parser.add_option("-p", "--parse-only", action=3D"store_true", - dest=3D"parse_only", default=3DFalse, - help=3D"Quit after parsing the BB recipes.") - - parser.add_option("-s", "--show-versions", action=3D"store_true", - dest=3D"show_versions", default=3DFalse, - help=3D"Show current and preferred versions of a= ll recipes.") - - parser.add_option("-e", "--environment", action=3D"store_true", - dest=3D"show_environment", default=3DFalse, - help=3D"Show the global or per-recipe environmen= t complete with information" - " about where variables were set/changed.") - - parser.add_option("-g", "--graphviz", action=3D"store_true", dest= =3D"dot_graph", default=3DFalse, - help=3D"Save dependency tree information for the= specified " - "targets in the dot syntax.") - - parser.add_option("-I", "--ignore-deps", action=3D"append", - dest=3D"extra_assume_provided", default=3D[], - help=3D"Assume these dependencies don't exist an= d are already provided " - "(equivalent to ASSUME_PROVIDED). Useful to= make dependency " - "graphs more appealing") - - parser.add_option("-l", "--log-domains", action=3D"append", dest= =3D"debug_domains", default=3D[], - help=3D"Show debug logging for the specified log= ging domains") - - parser.add_option("-P", "--profile", action=3D"store_true", dest= =3D"profile", default=3DFalse, - help=3D"Profile the command and save reports.") - - # @CHOICES@ is substituted out by BitbakeHelpFormatter above - parser.add_option("-u", "--ui", action=3D"store", dest=3D"ui", - default=3Dos.environ.get('BITBAKE_UI', 'knotty'), - help=3D"The user interface to use (@CHOICES@ - d= efault %default).") - - parser.add_option("", "--token", action=3D"store", dest=3D"xmlrpct= oken", - default=3Dos.environ.get("BBTOKEN"), - help=3D"Specify the connection token to be used = when connecting " - "to a remote server.") - - parser.add_option("", "--revisions-changed", action=3D"store_true", - dest=3D"revisions_changed", default=3DFalse, - help=3D"Set the exit code depending on whether u= pstream floating " - "revisions have changed or not.") - - parser.add_option("", "--server-only", action=3D"store_true", - dest=3D"server_only", default=3DFalse, - help=3D"Run bitbake without a UI, only starting = a server " - "(cooker) process.") - - parser.add_option("-B", "--bind", action=3D"store", dest=3D"bind",= default=3DFalse, - help=3D"The name/address for the bitbake xmlrpc = server to bind to.") - - parser.add_option("-T", "--idle-timeout", type=3Dfloat, dest=3D"se= rver_timeout", - default=3Dos.getenv("BB_SERVER_TIMEOUT"), - help=3D"Set timeout to unload bitbake server due= to inactivity, " - "set to -1 means no unload, " - "default: Environment variable BB_SERVER_T= IMEOUT.") - - parser.add_option("", "--no-setscene", action=3D"store_true", - dest=3D"nosetscene", default=3DFalse, - help=3D"Do not run any setscene tasks. sstate wi= ll be ignored and " - "everything needed, built.") - - parser.add_option("", "--skip-setscene", action=3D"store_true", - dest=3D"skipsetscene", default=3DFalse, - help=3D"Skip setscene tasks if they would be exe= cuted. Tasks previously " - "restored from sstate will be kept, unlike = --no-setscene") - - parser.add_option("", "--setscene-only", action=3D"store_true", - dest=3D"setsceneonly", default=3DFalse, - help=3D"Only run setscene tasks, don't run any r= eal tasks.") - - parser.add_option("", "--remote-server", action=3D"store", dest=3D= "remote_server", - default=3Dos.environ.get("BBSERVER"), - help=3D"Connect to the specified server.") - - parser.add_option("-m", "--kill-server", action=3D"store_true", - dest=3D"kill_server", default=3DFalse, - help=3D"Terminate any running bitbake server.") - - parser.add_option("", "--observe-only", action=3D"store_true", - dest=3D"observe_only", default=3DFalse, - help=3D"Connect to a server as an observing-only= client.") - - parser.add_option("", "--status-only", action=3D"store_true", - dest=3D"status_only", default=3DFalse, - help=3D"Check the status of the remote bitbake s= erver.") - - parser.add_option("-w", "--write-log", action=3D"store", dest=3D"w= riteeventlog", - default=3Dos.environ.get("BBEVENTLOG"), - help=3D"Writes the event log of the build to a b= itbake event json file. " - "Use '' (empty string) to assign the name a= utomatically.") - - parser.add_option("", "--runall", action=3D"append", dest=3D"runal= l", - help=3D"Run the specified task for any recipe in= the taskgraph of the specified target (even if it wouldn't otherwise have = run).") - - parser.add_option("", "--runonly", action=3D"append", dest=3D"runo= nly", - help=3D"Run only the specified task within the t= askgraph of the specified targets (and any task dependencies those tasks ma= y have).") + parser.add_option("-b", "--buildfile", action=3D"store", dest=3D"build= file", default=3DNone, + help=3D"Execute tasks from a specific .bb recipe dir= ectly. WARNING: Does " + "not handle any dependencies from other recipes= .") + + parser.add_option("-k", "--continue", action=3D"store_false", dest=3D"= abort", default=3DTrue, + help=3D"Continue as much as possible after an error.= While the target that " + "failed and anything depending on it cannot be = built, as much as " + "possible will be built before stopping.") + + parser.add_option("-f", "--force", action=3D"store_true", dest=3D"forc= e", default=3DFalse, + help=3D"Force the specified targets/task to run (inv= alidating any " + "existing stamp file).") + + parser.add_option("-c", "--cmd", action=3D"store", dest=3D"cmd", + help=3D"Specify the task to execute. The exact optio= ns available " + "depend on the metadata. Some examples might be= 'compile'" + " or 'populate_sysroot' or 'listtasks' may give= a list of " + "the tasks available.") + + parser.add_option("-C", "--clear-stamp", action=3D"store", dest=3D"inv= alidate_stamp", + help=3D"Invalidate the stamp for the specified task = such as 'compile' " + "and then run the default task for the specifie= d target(s).") + + parser.add_option("-r", "--read", action=3D"append", dest=3D"prefile",= default=3D[], + help=3D"Read the specified file before bitbake.conf.= ") + + parser.add_option("-R", "--postread", action=3D"append", dest=3D"postf= ile", default=3D[], + help=3D"Read the specified file after bitbake.conf.") + + parser.add_option("-v", "--verbose", action=3D"store_true", dest=3D"ve= rbose", default=3DFalse, + help=3D"Enable tracing of shell tasks (with 'set -x'= ). " + "Also print bb.note(...) messages to stdout (in= " + "addition to writing them to ${T}/log.do_<task>= ).") + + parser.add_option("-D", "--debug", action=3D"count", dest=3D"debug", d= efault=3D0, + help=3D"Increase the debug level. You can specify th= is " + "more than once. -D sets the debug level to 1, " + "where only bb.debug(1, ...) messages are print= ed " + "to stdout; -DD sets the debug level to 2, wher= e " + "both bb.debug(1, ...) and bb.debug(2, ...) " + "messages are printed; etc. Without -D, no debu= g " + "messages are printed. Note that -D only affect= s " + "output to stdout. All debug messages are writt= en " + "to ${T}/log.do_taskname, regardless of the deb= ug " + "level.") + + parser.add_option("-q", "--quiet", action=3D"count", dest=3D"quiet", d= efault=3D0, + help=3D"Output less log message data to the terminal= . You can specify this more than once.") + + parser.add_option("-n", "--dry-run", action=3D"store_true", dest=3D"dr= y_run", default=3DFalse, + help=3D"Don't execute, just go through the motions.") + + parser.add_option("-S", "--dump-signatures", action=3D"append", dest= =3D"dump_signatures", + default=3D[], metavar=3D"SIGNATURE_HANDLER", + help=3D"Dump out the signature construction informat= ion, with no task " + "execution. The SIGNATURE_HANDLER parameter is = passed to the " + "handler. Two common values are none and printd= iff but the handler " + "may define more/less. none means only dump the= signature, printdiff" + " means compare the dumped signature with the c= ached one.") + + parser.add_option("-p", "--parse-only", action=3D"store_true", + dest=3D"parse_only", default=3DFalse, + help=3D"Quit after parsing the BB recipes.") + + parser.add_option("-s", "--show-versions", action=3D"store_true", + dest=3D"show_versions", default=3DFalse, + help=3D"Show current and preferred versions of all r= ecipes.") + + parser.add_option("-e", "--environment", action=3D"store_true", + dest=3D"show_environment", default=3DFalse, + help=3D"Show the global or per-recipe environment co= mplete with information" + " about where variables were set/changed.") + + parser.add_option("-g", "--graphviz", action=3D"store_true", dest=3D"d= ot_graph", default=3DFalse, + help=3D"Save dependency tree information for the spe= cified " + "targets in the dot syntax.") + + parser.add_option("-I", "--ignore-deps", action=3D"append", + dest=3D"extra_assume_provided", default=3D[], + help=3D"Assume these dependencies don't exist and ar= e already provided " + "(equivalent to ASSUME_PROVIDED). Useful to mak= e dependency " + "graphs more appealing") + + parser.add_option("-l", "--log-domains", action=3D"append", dest=3D"de= bug_domains", default=3D[], + help=3D"Show debug logging for the specified logging= domains") + + parser.add_option("-P", "--profile", action=3D"store_true", dest=3D"pr= ofile", default=3DFalse, + help=3D"Profile the command and save reports.") + + # @CHOICES@ is substituted out by BitbakeHelpFormatter above + parser.add_option("-u", "--ui", action=3D"store", dest=3D"ui", + default=3Dos.environ.get('BITBAKE_UI', 'knotty'), + help=3D"The user interface to use (@CHOICES@ - defau= lt %default).") + + parser.add_option("", "--token", action=3D"store", dest=3D"xmlrpctoken= ", + default=3Dos.environ.get("BBTOKEN"), + help=3D"Specify the connection token to be used when= connecting " + "to a remote server.") + + parser.add_option("", "--revisions-changed", action=3D"store_true", + dest=3D"revisions_changed", default=3DFalse, + help=3D"Set the exit code depending on whether upstr= eam floating " + "revisions have changed or not.") + + parser.add_option("", "--server-only", action=3D"store_true", + dest=3D"server_only", default=3DFalse, + help=3D"Run bitbake without a UI, only starting a se= rver " + "(cooker) process.") + + parser.add_option("-B", "--bind", action=3D"store", dest=3D"bind", def= ault=3DFalse, + help=3D"The name/address for the bitbake xmlrpc serv= er to bind to.") + + parser.add_option("-T", "--idle-timeout", type=3Dfloat, dest=3D"server= _timeout", + default=3Dos.getenv("BB_SERVER_TIMEOUT"), + help=3D"Set timeout to unload bitbake server due to = inactivity, " + "set to -1 means no unload, " + "default: Environment variable BB_SERVER_TIMEOU= T.") + + parser.add_option("", "--no-setscene", action=3D"store_true", + dest=3D"nosetscene", default=3DFalse, + help=3D"Do not run any setscene tasks. sstate will b= e ignored and " + "everything needed, built.") + + parser.add_option("", "--skip-setscene", action=3D"store_true", + dest=3D"skipsetscene", default=3DFalse, + help=3D"Skip setscene tasks if they would be execute= d. Tasks previously " + "restored from sstate will be kept, unlike --no= -setscene") + + parser.add_option("", "--setscene-only", action=3D"store_true", + dest=3D"setsceneonly", default=3DFalse, + help=3D"Only run setscene tasks, don't run any real = tasks.") + + parser.add_option("", "--remote-server", action=3D"store", dest=3D"rem= ote_server", + default=3Dos.environ.get("BBSERVER"), + help=3D"Connect to the specified server.") + + parser.add_option("-m", "--kill-server", action=3D"store_true", + dest=3D"kill_server", default=3DFalse, + help=3D"Terminate any running bitbake server.") + + parser.add_option("", "--observe-only", action=3D"store_true", + dest=3D"observe_only", default=3DFalse, + help=3D"Connect to a server as an observing-only cli= ent.") + + parser.add_option("", "--status-only", action=3D"store_true", + dest=3D"status_only", default=3DFalse, + help=3D"Check the status of the remote bitbake serve= r.") + + parser.add_option("-w", "--write-log", action=3D"store", dest=3D"write= eventlog", + default=3Dos.environ.get("BBEVENTLOG"), + help=3D"Writes the event log of the build to a bitba= ke event json file. " + "Use '' (empty string) to assign the name autom= atically.") + + parser.add_option("", "--runall", action=3D"append", dest=3D"runall", + help=3D"Run the specified task for any recipe in the= taskgraph of the specified target (even if it wouldn't otherwise have run)= .") + + parser.add_option("", "--runonly", action=3D"append", dest=3D"runonly", + help=3D"Run only the specified task within the taskg= raph of the specified targets (and any task dependencies those tasks may ha= ve).") + return parser =20 =20 +class BitBakeConfigParameters(cookerdata.ConfigParameters): + def parseCommandLine(self, argv=3Dsys.argv): + parser =3D create_bitbake_parser() options, targets =3D parser.parse_args(argv) =20 if options.quiet and options.verbose: @@ -344,8 +347,6 @@ def bitbake_main(configParams, configuration): except: pass =20 - configuration.setConfigParameters(configParams) - if configParams.server_only and configParams.remote_server: raise BBMainException("FATAL: The '--server-only' option confl= icts with %s.\n" % ("the BBSERVER environment variable" if = "BBSERVER" in os.environ \ @@ -357,13 +358,13 @@ def bitbake_main(configParams, configuration): =20 if "BBDEBUG" in os.environ: level =3D int(os.environ["BBDEBUG"]) - if level > configuration.debug: - configuration.debug =3D level + if level > configParams.debug: + configParams.debug =3D level =20 - bb.msg.init_msgconfig(configParams.verbose, configuration.debug, - configuration.debug_domains) + bb.msg.init_msgconfig(configParams.verbose, configParams.debug, + configParams.debug_domains) =20 - server_connection, ui_module =3D setup_bitbake(configParams, configura= tion) + server_connection, ui_module =3D setup_bitbake(configParams) # No server connection if server_connection is None: if configParams.status_only: @@ -390,7 +391,7 @@ def bitbake_main(configParams, configuration): =20 return 1 =20 -def setup_bitbake(configParams, configuration, extrafeatures=3DNone): +def setup_bitbake(configParams, extrafeatures=3DNone): # Ensure logging messages get sent to the UI as events handler =3D bb.event.LogHandler() if not configParams.status_only: @@ -431,11 +432,11 @@ def setup_bitbake(configParams, configuration, extraf= eatures=3DNone): logger.info("bitbake server is not running.") lock.close() return None, None - # we start a server with a given configuration + # we start a server with a given featureset logger.info("Starting bitbake server...") # Clear the event queue since we already displayed mes= sages bb.event.ui_queue =3D [] - server =3D bb.server.process.BitBakeServer(lock, sockn= ame, configuration, featureset) + server =3D bb.server.process.BitBakeServer(lock, sockn= ame, featureset, configParams.server_timeout, configParams.xmlrpcinterface) =20 else: logger.info("Reconnecting to bitbake server...") @@ -458,15 +459,17 @@ def setup_bitbake(configParams, configuration, extraf= eatures=3DNone): break except BBMainFatal: raise - except (Exception, bb.server.process.ProcessTimeout) as e: + except (Exception, bb.server.process.ProcessTimeout, SystemExi= t) as e: + # SystemExit does not inherit from the Exception class, ne= eds to be included explicitly if not retries: raise retries -=3D 1 tryno =3D 8 - retries - if isinstance(e, (bb.server.process.ProcessTimeout, Broken= PipeError, EOFError)): + if isinstance(e, (bb.server.process.ProcessTimeout, Broken= PipeError, EOFError, SystemExit)): logger.info("Retrying server connection (#%d)..." % tr= yno) else: logger.info("Retrying server connection (#%d)... (%s)"= % (tryno, traceback.format_exc())) + if not retries: bb.fatal("Unable to connect to bitbake server, or start on= e (server startup failures would be in bitbake-cookerdaemon.log).") bb.event.print_ui_queue() diff --git a/bitbake/lib/bb/monitordisk.py b/bitbake/lib/bb/monitordisk.py index e7c07264..98f2109e 100644 --- a/bitbake/lib/bb/monitordisk.py +++ b/bitbake/lib/bb/monitordisk.py @@ -59,7 +59,7 @@ def getMountedDev(path): pass return None =20 -def getDiskData(BBDirs, configuration): +def getDiskData(BBDirs): =20 """Prepare disk data for disk space monitor""" =20 @@ -168,7 +168,7 @@ class diskMonitor: =20 BBDirs =3D configuration.getVar("BB_DISKMON_DIRS") or None if BBDirs: - self.devDict =3D getDiskData(BBDirs, configuration) + self.devDict =3D getDiskData(BBDirs) if self.devDict: self.spaceInterval, self.inodeInterval =3D getInterval(con= figuration) if self.spaceInterval and self.inodeInterval: diff --git a/bitbake/lib/bb/msg.py b/bitbake/lib/bb/msg.py index 2d88c4e7..291b38ff 100644 --- a/bitbake/lib/bb/msg.py +++ b/bitbake/lib/bb/msg.py @@ -14,6 +14,7 @@ import sys import copy import logging import logging.config +import os from itertools import groupby import bb import bb.event @@ -146,18 +147,12 @@ class LogFilterLTLevel(logging.Filter): # =20 loggerDefaultLogLevel =3D BBLogFormatter.NOTE -loggerDefaultVerbose =3D False -loggerVerboseLogs =3D False loggerDefaultDomains =3D {} =20 def init_msgconfig(verbose, debug, debug_domains=3DNone): """ Set default verbosity and debug levels config the logger """ - bb.msg.loggerDefaultVerbose =3D verbose - if verbose: - bb.msg.loggerVerboseLogs =3D True - if debug: bb.msg.loggerDefaultLogLevel =3D BBLogFormatter.DEBUG - debug + 1 elif verbose: @@ -283,7 +278,7 @@ def setLoggingConfig(defaultconfig, userconfigfile=3DNo= ne): with open(os.path.normpath(userconfigfile), 'r') as f: if userconfigfile.endswith('.yml') or userconfigfile.endswith(= '.yaml'): import yaml - userconfig =3D yaml.load(f) + userconfig =3D yaml.safe_load(f) elif userconfigfile.endswith('.json') or userconfigfile.endswi= th('.cfg'): import json userconfig =3D json.load(f) diff --git a/bitbake/lib/bb/namedtuple_with_abc.py b/bitbake/lib/bb/namedtu= ple_with_abc.py index 646aed6f..e46dbf08 100644 --- a/bitbake/lib/bb/namedtuple_with_abc.py +++ b/bitbake/lib/bb/namedtuple_with_abc.py @@ -61,17 +61,9 @@ class _NamedTupleABCMeta(ABCMeta): return ABCMeta.__new__(mcls, name, bases, namespace) =20 =20 -exec( - # Python 2.x metaclass declaration syntax - """class _NamedTupleABC(object): - '''The abstract base class + mix-in for named tuples.''' - __metaclass__ =3D _NamedTupleABCMeta - _fields =3D abstractproperty()""" if version_info[0] < 3 else - # Python 3.x metaclass declaration syntax - """class _NamedTupleABC(metaclass=3D_NamedTupleABCMeta): - '''The abstract base class + mix-in for named tuples.''' - _fields =3D abstractproperty()""" -) +class _NamedTupleABC(metaclass=3D_NamedTupleABCMeta): + '''The abstract base class + mix-in for named tuples.''' + _fields =3D abstractproperty() =20 =20 _namedtuple.abc =3D _NamedTupleABC diff --git a/bitbake/lib/bb/parse/__init__.py b/bitbake/lib/bb/parse/__init= __.py index 76e180b4..c01807ba 100644 --- a/bitbake/lib/bb/parse/__init__.py +++ b/bitbake/lib/bb/parse/__init__.py @@ -71,7 +71,7 @@ def update_mtime(f): =20 def update_cache(f): if f in __mtime_cache: - logger.debug(1, "Updating mtime cache for %s" % f) + logger.debug("Updating mtime cache for %s" % f) update_mtime(f) =20 def clear_cache(): diff --git a/bitbake/lib/bb/parse/ast.py b/bitbake/lib/bb/parse/ast.py index eb8cfa21..db2bdc35 100644 --- a/bitbake/lib/bb/parse/ast.py +++ b/bitbake/lib/bb/parse/ast.py @@ -34,7 +34,7 @@ class IncludeNode(AstNode): Include the file and evaluate the statements """ s =3D data.expand(self.what_file) - logger.debug(2, "CONF %s:%s: including %s", self.filename, self.li= neno, s) + logger.debug2("CONF %s:%s: including %s", self.filename, self.line= no, s) =20 # TODO: Cache those includes... maybe not here though if self.force: @@ -97,6 +97,7 @@ class DataNode(AstNode): def eval(self, data): groupd =3D self.groupd key =3D groupd["var"] + key =3D key.replace(":", "_") loginfo =3D { 'variable': key, 'file': self.filename, @@ -207,6 +208,7 @@ class ExportFuncsNode(AstNode): def eval(self, data): =20 for func in self.n: + func =3D func.replace(":", "_") calledfunc =3D self.classname + "_" + func =20 if data.getVar(func, False) and not data.getVarFlag(func, 'exp= ort_func', False): @@ -244,12 +246,14 @@ class AddTaskNode(AstNode): bb.build.addtask(self.func, self.before, self.after, data) =20 class DelTaskNode(AstNode): - def __init__(self, filename, lineno, func): + def __init__(self, filename, lineno, tasks): AstNode.__init__(self, filename, lineno) - self.func =3D func + self.tasks =3D tasks =20 def eval(self, data): - bb.build.deltask(self.func, data) + tasks =3D data.expand(self.tasks).split() + for task in tasks: + bb.build.deltask(task, data) =20 class BBHandlerNode(AstNode): def __init__(self, filename, lineno, fns): @@ -305,7 +309,7 @@ def handleAddTask(statements, filename, lineno, m): statements.append(AddTaskNode(filename, lineno, func, before, after)) =20 def handleDelTask(statements, filename, lineno, m): - func =3D m.group("func") + func =3D m.group(1) if func is None: return =20 @@ -333,11 +337,14 @@ def finalize(fn, d, variant =3D None): if not handlerfn: bb.fatal("Undefined event handler function '%s'" % var) handlerln =3D int(d.getVarFlag(var, "lineno", False)) - bb.event.register(var, d.getVar(var, False), (d.getVarFlag(var= , "eventmask") or "").split(), handlerfn, handlerln) + bb.event.register(var, d.getVar(var, False), (d.getVarFlag(var= , "eventmask") or "").split(), handlerfn, handlerln, data=3Dd) =20 bb.event.fire(bb.event.RecipePreFinalise(fn), d) =20 bb.data.expandKeys(d) + + bb.event.fire(bb.event.RecipePostKeyExpansion(fn), d) + runAnonFuncs(d) =20 tasklist =3D d.getVar('__BBTASKS', False) or [] @@ -371,7 +378,7 @@ def _create_variants(datastores, names, function, onlyf= inalise): def multi_finalize(fn, d): appends =3D (d.getVar("__BBAPPEND") or "").split() for append in appends: - logger.debug(1, "Appending .bbappend file %s to %s", append, fn) + logger.debug("Appending .bbappend file %s to %s", append, fn) bb.parse.BBHandler.handle(append, d, True) =20 onlyfinalise =3D d.getVar("__ONLYFINALISE", False) diff --git a/bitbake/lib/bb/parse/parse_py/BBHandler.py b/bitbake/lib/bb/pa= rse/parse_py/BBHandler.py index 6e216eff..152ef6ab 100644 --- a/bitbake/lib/bb/parse/parse_py/BBHandler.py +++ b/bitbake/lib/bb/parse/parse_py/BBHandler.py @@ -13,7 +13,7 @@ # =20 import re, bb, os -import bb.build, bb.utils +import bb.build, bb.utils, bb.data_smart =20 from . import ConfHandler from .. import resolve_file, ast, logger, ParseError @@ -22,11 +22,11 @@ from .ConfHandler import include, init # For compatibility bb.deprecate_import(__name__, "bb.parse", ["vars_from_file"]) =20 -__func_start_regexp__ =3D re.compile(r"(((?P<py>python)|(?P<fr>fakeroot= ))\s*)*(?P<func>[\w\.\-\+\{\}\$]+)?\s*\(\s*\)\s*{$" ) +__func_start_regexp__ =3D re.compile(r"(((?P<py>python(?=3D(\s|\()))|(?= P<fr>fakeroot(?=3D\s)))\s*)*(?P<func>[\w\.\-\+\{\}\$:]+)?\s*\(\s*\)\s*{$" ) __inherit_regexp__ =3D re.compile(r"inherit\s+(.+)" ) __export_func_regexp__ =3D re.compile(r"EXPORT_FUNCTIONS\s+(.+)" ) __addtask_regexp__ =3D re.compile(r"addtask\s+(?P<func>\w+)\s*((befo= re\s*(?P<before>((.*(?=3Dafter))|(.*))))|(after\s*(?P<after>((.*(?=3Dbefore= ))|(.*)))))*") -__deltask_regexp__ =3D re.compile(r"deltask\s+(?P<func>\w+)(?P<ignor= es>.*)") +__deltask_regexp__ =3D re.compile(r"deltask\s+(.+)") __addhandler_regexp__ =3D re.compile(r"addhandler\s+(.+)" ) __def_regexp__ =3D re.compile(r"def\s+(\w+).*:" ) __python_func_regexp__ =3D re.compile(r"(\s+.*)|(^$)|(^#)" ) @@ -60,7 +60,7 @@ def inherit(files, fn, lineno, d): file =3D abs_fn =20 if not file in __inherit_cache: - logger.debug(1, "Inheriting %s (from %s:%d)" % (file, fn, line= no)) + logger.debug("Inheriting %s (from %s:%d)" % (file, fn, lineno)) __inherit_cache.append( file ) d.setVar('__inherit_cache', __inherit_cache) include(fn, file, lineno, d, "inherit") @@ -233,14 +233,15 @@ def feeder(lineno, s, fn, root, statements, eof=3DFal= se): if taskexpression.count(word) > 1: logger.warning("addtask contained multiple '%s' keywords, = only one is supported" % word) =20 + # Check and warn for having task with exprssion as part of task na= me + for te in taskexpression: + if any( ( "%s_" % keyword ) in te for keyword in bb.data_smart= .__setvar_keyword__ ): + raise ParseError("Task name '%s' contains a keyword which = is not recommended/supported.\nPlease rename the task not to include the ke= yword.\n%s" % (te, ("\n".join(map(str, bb.data_smart.__setvar_keyword__))))= , fn) ast.handleAddTask(statements, fn, lineno, m) return =20 m =3D __deltask_regexp__.match(s) if m: - # Check and warn "for deltask task1 task2" - if m.group('ignores'): - logger.warning('deltask ignored: "%s"' % m.group('ignores')) ast.handleDelTask(statements, fn, lineno, m) return =20 diff --git a/bitbake/lib/bb/parse/parse_py/ConfHandler.py b/bitbake/lib/bb/= parse/parse_py/ConfHandler.py index af64d344..0834fe3f 100644 --- a/bitbake/lib/bb/parse/parse_py/ConfHandler.py +++ b/bitbake/lib/bb/parse/parse_py/ConfHandler.py @@ -20,7 +20,7 @@ from bb.parse import ParseError, resolve_file, ast, logge= r, handle __config_regexp__ =3D re.compile( r""" ^ (?P<exp>export\s+)? - (?P<var>[a-zA-Z0-9\-_+.${}/~]+?) + (?P<var>[a-zA-Z0-9\-_+.${}/~:]+?) (\[(?P<flag>[a-zA-Z0-9\-_+.]+)\])? =20 \s* ( @@ -95,7 +95,7 @@ def include_single_file(parentfn, fn, lineno, data, error= _out): if exc.errno =3D=3D errno.ENOENT: if error_out: raise ParseError("Could not %s file %s" % (error_out, fn),= parentfn, lineno) - logger.debug(2, "CONF file '%s' not found", fn) + logger.debug2("CONF file '%s' not found", fn) else: if error_out: raise ParseError("Could not %s file %s: %s" % (error_out, = fn, exc.strerror), parentfn, lineno) diff --git a/bitbake/lib/bb/persist_data.py b/bitbake/lib/bb/persist_data.py index 7357ab2d..c6a209fb 100644 --- a/bitbake/lib/bb/persist_data.py +++ b/bitbake/lib/bb/persist_data.py @@ -12,14 +12,14 @@ currently, providing a key/value store accessed by 'dom= ain'. # =20 import collections +import contextlib +import functools import logging import os.path +import sqlite3 import sys import warnings -from bb.compat import total_ordering from collections import Mapping -import sqlite3 -import contextlib =20 sqlversion =3D sqlite3.sqlite_version_info if sqlversion[0] < 3 or (sqlversion[0] =3D=3D 3 and sqlversion[1] < 3): @@ -28,7 +28,7 @@ if sqlversion[0] < 3 or (sqlversion[0] =3D=3D 3 and sqlve= rsion[1] < 3): =20 logger =3D logging.getLogger("BitBake.PersistData") =20 -@total_ordering +@functools.total_ordering class SQLTable(collections.MutableMapping): class _Decorators(object): @staticmethod @@ -248,7 +248,7 @@ class PersistData(object): stacklevel=3D2) =20 self.data =3D persist(d) - logger.debug(1, "Using '%s' as the persistent data cache", + logger.debug("Using '%s' as the persistent data cache", self.data.filename) =20 def addDomain(self, domain): diff --git a/bitbake/lib/bb/process.py b/bitbake/lib/bb/process.py index 2dc472a8..d5a1775f 100644 --- a/bitbake/lib/bb/process.py +++ b/bitbake/lib/bb/process.py @@ -7,6 +7,7 @@ import signal import subprocess import errno import select +import bb =20 logger =3D logging.getLogger('BitBake.Process') =20 @@ -41,6 +42,7 @@ class ExecutionError(CmdError): self.exitcode =3D exitcode self.stdout =3D stdout self.stderr =3D stderr + self.extra_message =3D None =20 def __str__(self): message =3D "" @@ -51,7 +53,7 @@ class ExecutionError(CmdError): if message: message =3D ":\n" + message return (CmdError.__str__(self) + - " with exit code %s" % self.exitcode + message) + " with exit code %s" % self.exitcode + message + (self.ext= ra_message or "")) =20 class Popen(subprocess.Popen): defaults =3D { @@ -179,5 +181,8 @@ def run(cmd, input=3DNone, log=3DNone, extrafiles=3DNon= e, **options): stderr =3D stderr.decode("utf-8") =20 if pipe.returncode !=3D 0: + if log: + # Don't duplicate the output in the exception if logging it + raise ExecutionError(cmd, pipe.returncode, None, None) raise ExecutionError(cmd, pipe.returncode, stdout, stderr) return stdout, stderr diff --git a/bitbake/lib/bb/progress.py b/bitbake/lib/bb/progress.py index 9c755b7f..d051ba01 100644 --- a/bitbake/lib/bb/progress.py +++ b/bitbake/lib/bb/progress.py @@ -14,7 +14,27 @@ import bb.event import bb.build from bb.build import StdoutNoopContextManager =20 -class ProgressHandler(object): + +# from https://stackoverflow.com/a/14693789/221061 +ANSI_ESCAPE_REGEX =3D re.compile(r'\x1B\[[0-?]*[ -/]*[@-~]') + + +def filter_color(string): + """ + Filter ANSI escape codes out of |string|, return new string + """ + return ANSI_ESCAPE_REGEX.sub('', string) + + +def filter_color_n(string): + """ + Filter ANSI escape codes out of |string|, returns tuple of + (new string, # of ANSI codes removed) + """ + return ANSI_ESCAPE_REGEX.subn('', string) + + +class ProgressHandler: """ Base class that can pretend to be a file object well enough to be used to build objects to intercept console output and determine the @@ -55,6 +75,7 @@ class ProgressHandler(object): self._lastevent =3D ts self._progress =3D progress =20 + class LineFilterProgressHandler(ProgressHandler): """ A ProgressHandler variant that provides the ability to filter out @@ -66,7 +87,7 @@ class LineFilterProgressHandler(ProgressHandler): """ def __init__(self, d, outfile=3DNone): self._linebuffer =3D '' - super(LineFilterProgressHandler, self).__init__(d, outfile) + super().__init__(d, outfile) =20 def write(self, string): self._linebuffer +=3D string @@ -80,41 +101,44 @@ class LineFilterProgressHandler(ProgressHandler): lbreakpos =3D line.rfind('\r') + 1 if lbreakpos: line =3D line[lbreakpos:] - if self.writeline(line): - super(LineFilterProgressHandler, self).write(line) + if self.writeline(filter_color(line)): + super().write(line) =20 def writeline(self, line): return True =20 + class BasicProgressHandler(ProgressHandler): def __init__(self, d, regex=3Dr'(\d+)%', outfile=3DNone): - super(BasicProgressHandler, self).__init__(d, outfile) + super().__init__(d, outfile) self._regex =3D re.compile(regex) # Send an initial progress event so the bar gets shown self._fire_progress(0) =20 def write(self, string): - percs =3D self._regex.findall(string) + percs =3D self._regex.findall(filter_color(string)) if percs: progress =3D int(percs[-1]) self.update(progress) - super(BasicProgressHandler, self).write(string) + super().write(string) + =20 class OutOfProgressHandler(ProgressHandler): def __init__(self, d, regex, outfile=3DNone): - super(OutOfProgressHandler, self).__init__(d, outfile) + super().__init__(d, outfile) self._regex =3D re.compile(regex) # Send an initial progress event so the bar gets shown self._fire_progress(0) =20 def write(self, string): - nums =3D self._regex.findall(string) + nums =3D self._regex.findall(filter_color(string)) if nums: progress =3D (float(nums[-1][0]) / float(nums[-1][1])) * 100 self.update(progress) - super(OutOfProgressHandler, self).write(string) + super().write(string) + =20 -class MultiStageProgressReporter(object): +class MultiStageProgressReporter: """ Class which allows reporting progress without the caller having to know where they are in the overall sequence. Useful @@ -199,6 +223,7 @@ class MultiStageProgressReporter(object): value is considered to be out of stage_total, otherwise it should be a percentage value from 0 to 100. """ + progress =3D None if self._stage_total: stage_progress =3D (float(stage_progress) / self._stage_total)= * 100 if self._stage < 0: @@ -207,9 +232,10 @@ class MultiStageProgressReporter(object): progress =3D self._base_progress + (stage_progress * self._sta= ge_weights[self._stage]) else: progress =3D self._base_progress - if progress > 100: - progress =3D 100 - self._fire_progress(progress) + if progress: + if progress > 100: + progress =3D 100 + self._fire_progress(progress) =20 def finish(self): if self._finished: @@ -230,6 +256,7 @@ class MultiStageProgressReporter(object): out.append('Up to finish: %d' % stage_weight) bb.warn('Stage times:\n %s' % '\n '.join(out)) =20 + class MultiStageProcessProgressReporter(MultiStageProgressReporter): """ Version of MultiStageProgressReporter intended for use with @@ -238,7 +265,7 @@ class MultiStageProcessProgressReporter(MultiStageProgr= essReporter): def __init__(self, d, processname, stage_weights, debug=3DFalse): self._processname =3D processname self._started =3D False - MultiStageProgressReporter.__init__(self, d, stage_weights, debug) + super().__init__(d, stage_weights, debug) =20 def start(self): if not self._started: @@ -255,13 +282,14 @@ class MultiStageProcessProgressReporter(MultiStagePro= gressReporter): MultiStageProgressReporter.finish(self) bb.event.fire(bb.event.ProcessFinished(self._processname), self._d= ata) =20 + class DummyMultiStageProcessProgressReporter(MultiStageProgressReporter): """ MultiStageProcessProgressReporter that takes the calls and does nothing with them (to avoid a bunch of "if progress_reporter:" checks) """ def __init__(self): - MultiStageProcessProgressReporter.__init__(self, "", None, []) + super().__init__(None, []) =20 def _fire_progress(self, taskprogress, rate=3DNone): pass diff --git a/bitbake/lib/bb/providers.py b/bitbake/lib/bb/providers.py index 81459c36..3ec11a40 100644 --- a/bitbake/lib/bb/providers.py +++ b/bitbake/lib/bb/providers.py @@ -38,16 +38,17 @@ def findProviders(cfgData, dataCache, pkg_pn =3D None): localdata =3D data.createCopy(cfgData) bb.data.expandKeys(localdata) =20 + required =3D {} preferred_versions =3D {} latest_versions =3D {} =20 for pn in pkg_pn: - (last_ver, last_file, pref_ver, pref_file) =3D findBestProvider(pn= , localdata, dataCache, pkg_pn) + (last_ver, last_file, pref_ver, pref_file, req) =3D findBestProvid= er(pn, localdata, dataCache, pkg_pn) preferred_versions[pn] =3D (pref_ver, pref_file) latest_versions[pn] =3D (last_ver, last_file) + required[pn] =3D req =20 - return (latest_versions, preferred_versions) - + return (latest_versions, preferred_versions, required) =20 def allProviders(dataCache): """ @@ -59,7 +60,6 @@ def allProviders(dataCache): all_providers[pn].append((ver, fn)) return all_providers =20 - def sortPriorities(pn, dataCache, pkg_pn =3D None): """ Reorder pkg_pn by file priority and default preference @@ -87,6 +87,21 @@ def sortPriorities(pn, dataCache, pkg_pn =3D None): =20 return tmp_pn =20 +def versionVariableMatch(cfgData, keyword, pn): + """ + Return the value of the <keyword>_VERSION variable if set. + """ + + # pn can contain '_', e.g. gcc-cross-x86_64 and an override cannot + # hence we do this manually rather than use OVERRIDES + ver =3D cfgData.getVar("%s_VERSION_pn-%s" % (keyword, pn)) + if not ver: + ver =3D cfgData.getVar("%s_VERSION_%s" % (keyword, pn)) + if not ver: + ver =3D cfgData.getVar("%s_VERSION" % keyword) + + return ver + def preferredVersionMatch(pe, pv, pr, preferred_e, preferred_v, preferred_= r): """ Check if the version pe,pv,pr is the preferred one. @@ -102,19 +117,28 @@ def preferredVersionMatch(pe, pv, pr, preferred_e, pr= eferred_v, preferred_r): =20 def findPreferredProvider(pn, cfgData, dataCache, pkg_pn =3D None, item = =3D None): """ - Find the first provider in pkg_pn with a PREFERRED_VERSION set. + Find the first provider in pkg_pn with REQUIRED_VERSION or PREFERRED_V= ERSION set. """ =20 preferred_file =3D None preferred_ver =3D None + required =3D False =20 - # pn can contain '_', e.g. gcc-cross-x86_64 and an override cannot - # hence we do this manually rather than use OVERRIDES - preferred_v =3D cfgData.getVar("PREFERRED_VERSION_pn-%s" % pn) - if not preferred_v: - preferred_v =3D cfgData.getVar("PREFERRED_VERSION_%s" % pn) - if not preferred_v: - preferred_v =3D cfgData.getVar("PREFERRED_VERSION") + required_v =3D versionVariableMatch(cfgData, "REQUIRED", pn) + preferred_v =3D versionVariableMatch(cfgData, "PREFERRED", pn) + + itemstr =3D "" + if item: + itemstr =3D " (for item %s)" % item + + if required_v is not None: + if preferred_v is not None: + logger.warn("REQUIRED_VERSION and PREFERRED_VERSION for packag= e %s%s are both set using REQUIRED_VERSION %s", pn, itemstr, required_v) + else: + logger.debug("REQUIRED_VERSION is set for package %s%s", pn, i= temstr) + # REQUIRED_VERSION always takes precedence over PREFERRED_VERSION + preferred_v =3D required_v + required =3D True =20 if preferred_v: m =3D re.match(r'(\d+:)*(.*)(_.*)*', preferred_v) @@ -147,11 +171,9 @@ def findPreferredProvider(pn, cfgData, dataCache, pkg_= pn =3D None, item =3D None): pv_str =3D preferred_v if not (preferred_e is None): pv_str =3D '%s:%s' % (preferred_e, pv_str) - itemstr =3D "" - if item: - itemstr =3D " (for item %s)" % item if preferred_file is None: - logger.info("preferred version %s of %s not available%s", pv_s= tr, pn, itemstr) + if not required: + logger.warn("preferred version %s of %s not available%s", = pv_str, pn, itemstr) available_vers =3D [] for file_set in pkg_pn: for f in file_set: @@ -163,12 +185,16 @@ def findPreferredProvider(pn, cfgData, dataCache, pkg= _pn =3D None, item =3D None): available_vers.append(ver_str) if available_vers: available_vers.sort() - logger.info("versions of %s available: %s", pn, ' '.join(a= vailable_vers)) + logger.warn("versions of %s available: %s", pn, ' '.join(a= vailable_vers)) + if required: + logger.error("required version %s of %s not available%s", = pv_str, pn, itemstr) else: - logger.debug(1, "selecting %s as PREFERRED_VERSION %s of packa= ge %s%s", preferred_file, pv_str, pn, itemstr) - - return (preferred_ver, preferred_file) + if required: + logger.debug("selecting %s as REQUIRED_VERSION %s of packa= ge %s%s", preferred_file, pv_str, pn, itemstr) + else: + logger.debug("selecting %s as PREFERRED_VERSION %s of pack= age %s%s", preferred_file, pv_str, pn, itemstr) =20 + return (preferred_ver, preferred_file, required) =20 def findLatestProvider(pn, cfgData, dataCache, file_set): """ @@ -189,7 +215,6 @@ def findLatestProvider(pn, cfgData, dataCache, file_set= ): =20 return (latest, latest_f) =20 - def findBestProvider(pn, cfgData, dataCache, pkg_pn =3D None, item =3D Non= e): """ If there is a PREFERRED_VERSION, find the highest-priority bbfile @@ -198,17 +223,16 @@ def findBestProvider(pn, cfgData, dataCache, pkg_pn = =3D None, item =3D None): """ =20 sortpkg_pn =3D sortPriorities(pn, dataCache, pkg_pn) - # Find the highest priority provider with a PREFERRED_VERSION set - (preferred_ver, preferred_file) =3D findPreferredProvider(pn, cfgData,= dataCache, sortpkg_pn, item) + # Find the highest priority provider with a REQUIRED_VERSION or PREFER= RED_VERSION set + (preferred_ver, preferred_file, required) =3D findPreferredProvider(pn= , cfgData, dataCache, sortpkg_pn, item) # Find the latest version of the highest priority provider (latest, latest_f) =3D findLatestProvider(pn, cfgData, dataCache, sort= pkg_pn[0]) =20 - if preferred_file is None: + if not required and preferred_file is None: preferred_file =3D latest_f preferred_ver =3D latest =20 - return (latest, latest_f, preferred_ver, preferred_file) - + return (latest, latest_f, preferred_ver, preferred_file, required) =20 def _filterProviders(providers, item, cfgData, dataCache): """ @@ -232,12 +256,15 @@ def _filterProviders(providers, item, cfgData, dataCa= che): pkg_pn[pn] =3D [] pkg_pn[pn].append(p) =20 - logger.debug(1, "providers for %s are: %s", item, list(sorted(pkg_pn.k= eys()))) + logger.debug("providers for %s are: %s", item, list(sorted(pkg_pn.keys= ()))) =20 - # First add PREFERRED_VERSIONS + # First add REQUIRED_VERSIONS or PREFERRED_VERSIONS for pn in sorted(pkg_pn): sortpkg_pn[pn] =3D sortPriorities(pn, dataCache, pkg_pn) - preferred_versions[pn] =3D findPreferredProvider(pn, cfgData, data= Cache, sortpkg_pn[pn], item) + preferred_ver, preferred_file, required =3D findPreferredProvider(= pn, cfgData, dataCache, sortpkg_pn[pn], item) + if required and preferred_file is None: + return eligible + preferred_versions[pn] =3D (preferred_ver, preferred_file) if preferred_versions[pn][1]: eligible.append(preferred_versions[pn][1]) =20 @@ -248,9 +275,8 @@ def _filterProviders(providers, item, cfgData, dataCach= e): preferred_versions[pn] =3D findLatestProvider(pn, cfgData, dataCac= he, sortpkg_pn[pn][0]) eligible.append(preferred_versions[pn][1]) =20 - if len(eligible) =3D=3D 0: - logger.error("no eligible providers for %s", item) - return 0 + if not eligible: + return eligible =20 # If pn =3D=3D item, give it a slight default preference # This means PREFERRED_PROVIDER_foobar defaults to foobar if available @@ -266,7 +292,6 @@ def _filterProviders(providers, item, cfgData, dataCach= e): =20 return eligible =20 - def filterProviders(providers, item, cfgData, dataCache): """ Take a list of providers and filter/reorder according to the @@ -291,7 +316,7 @@ def filterProviders(providers, item, cfgData, dataCache= ): foundUnique =3D True break =20 - logger.debug(1, "sorted providers for %s are: %s", item, eligible) + logger.debug("sorted providers for %s are: %s", item, eligible) =20 return eligible, foundUnique =20 @@ -333,7 +358,7 @@ def filterProvidersRunTime(providers, item, cfgData, da= taCache): provides =3D dataCache.pn_provides[pn] for provide in provides: prefervar =3D cfgData.getVar('PREFERRED_PROVIDER_%s' % pro= vide) - #logger.debug(1, "checking PREFERRED_PROVIDER_%s (value %s= ) against %s", provide, prefervar, pns.keys()) + #logger.debug("checking PREFERRED_PROVIDER_%s (value %s) a= gainst %s", provide, prefervar, pns.keys()) if prefervar in pns and pns[prefervar] not in preferred: var =3D "PREFERRED_PROVIDER_%s =3D %s" % (provide, pre= fervar) logger.verbose("selecting %s to satisfy runtime %s due= to %s", prefervar, item, var) @@ -349,7 +374,7 @@ def filterProvidersRunTime(providers, item, cfgData, da= taCache): if numberPreferred > 1: logger.error("Trying to resolve runtime dependency %s resulted in = conflicting PREFERRED_PROVIDER entries being found.\nThe providers found we= re: %s\nThe PREFERRED_PROVIDER entries resulting in this conflict were: %s.= You could set PREFERRED_RPROVIDER_%s" % (item, preferred, preferred_vars, = item)) =20 - logger.debug(1, "sorted runtime providers for %s are: %s", item, eligi= ble) + logger.debug("sorted runtime providers for %s are: %s", item, eligible) =20 return eligible, numberPreferred =20 @@ -384,11 +409,10 @@ def getRuntimeProviders(dataCache, rdepend): regexp_cache[pattern] =3D regexp if regexp.match(rdepend): rproviders +=3D dataCache.packages_dynamic[pattern] - logger.debug(1, "Assuming %s is a dynamic package, but it may = not exist" % rdepend) + logger.debug("Assuming %s is a dynamic package, but it may not= exist" % rdepend) =20 return rproviders =20 - def buildWorldTargetList(dataCache, task=3DNone): """ Build package list for "bitbake world" @@ -396,22 +420,22 @@ def buildWorldTargetList(dataCache, task=3DNone): if dataCache.world_target: return =20 - logger.debug(1, "collating packages for \"world\"") + logger.debug("collating packages for \"world\"") for f in dataCache.possible_world: terminal =3D True pn =3D dataCache.pkg_fn[f] if task and task not in dataCache.task_deps[f]['tasks']: - logger.debug(2, "World build skipping %s as task %s doesn't ex= ist", f, task) + logger.debug2("World build skipping %s as task %s doesn't exis= t", f, task) terminal =3D False =20 for p in dataCache.pn_provides[pn]: if p.startswith('virtual/'): - logger.debug(2, "World build skipping %s due to %s provide= r starting with virtual/", f, p) + logger.debug2("World build skipping %s due to %s provider = starting with virtual/", f, p) terminal =3D False break for pf in dataCache.providers[p]: if dataCache.pkg_fn[pf] !=3D pn: - logger.debug(2, "World build skipping %s due to both u= s and %s providing %s", f, pf, p) + logger.debug2("World build skipping %s due to both us = and %s providing %s", f, pf, p) terminal =3D False break if terminal: diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py index 30cab537..10511a09 100644 --- a/bitbake/lib/bb/runqueue.py +++ b/bitbake/lib/bb/runqueue.py @@ -38,7 +38,7 @@ def taskname_from_tid(tid): return tid.rsplit(":", 1)[1] =20 def mc_from_tid(tid): - if tid.startswith('mc:'): + if tid.startswith('mc:') and tid.count(':') >=3D 2: return tid.split(':')[1] return "" =20 @@ -46,8 +46,14 @@ def split_tid(tid): (mc, fn, taskname, _) =3D split_tid_mcfn(tid) return (mc, fn, taskname) =20 +def split_mc(n): + if n.startswith("mc:") and n.count(':') >=3D 2: + _, mc, n =3D n.split(":", 2) + return (mc, n) + return ('', n) + def split_tid_mcfn(tid): - if tid.startswith('mc:'): + if tid.startswith('mc:') and tid.count(':') >=3D 2: elems =3D tid.split(':') mc =3D elems[1] fn =3D ":".join(elems[2:-1]) @@ -79,15 +85,19 @@ class RunQueueStats: """ Holds statistics on the tasks handled by the associated runQueue """ - def __init__(self, total): + def __init__(self, total, setscene_total): self.completed =3D 0 self.skipped =3D 0 self.failed =3D 0 self.active =3D 0 + self.setscene_active =3D 0 + self.setscene_covered =3D 0 + self.setscene_notcovered =3D 0 + self.setscene_total =3D setscene_total self.total =3D total =20 def copy(self): - obj =3D self.__class__(self.total) + obj =3D self.__class__(self.total, self.setscene_total) obj.__dict__.update(self.__dict__) return obj =20 @@ -106,6 +116,13 @@ class RunQueueStats: def taskActive(self): self.active =3D self.active + 1 =20 + def updateCovered(self, covered, notcovered): + self.setscene_covered =3D covered + self.setscene_notcovered =3D notcovered + + def updateActiveSetscene(self, active): + self.setscene_active =3D active + # These values indicate the next step due to be run in the # runQueue state machine runQueuePrepare =3D 2 @@ -370,7 +387,7 @@ class RunQueueData: =20 self.stampwhitelist =3D cfgData.getVar("BB_STAMP_WHITELIST") or "" self.multi_provider_whitelist =3D (cfgData.getVar("MULTI_PROVIDER_= WHITELIST") or "").split() - self.setscenewhitelist =3D get_setscene_enforce_whitelist(cfgData) + self.setscenewhitelist =3D get_setscene_enforce_whitelist(cfgData,= targets) self.setscenewhitelist_checked =3D False self.setscene_enforce =3D (cfgData.getVar('BB_SETSCENE_ENFORCE') = =3D=3D "1") self.init_progress_reporter =3D bb.progress.DummyMultiStageProcess= ProgressReporter() @@ -538,8 +555,8 @@ class RunQueueData: for tid in self.runtaskentries: if task_done[tid] is False or deps_left[tid] !=3D 0: problem_tasks.append(tid) - logger.debug(2, "Task %s is not buildable", tid) - logger.debug(2, "(Complete marker was %s and the remaining= dependency count was %s)\n", task_done[tid], deps_left[tid]) + logger.debug2("Task %s is not buildable", tid) + logger.debug2("(Complete marker was %s and the remaining d= ependency count was %s)\n", task_done[tid], deps_left[tid]) self.runtaskentries[tid].weight =3D weight[tid] =20 if problem_tasks: @@ -637,7 +654,7 @@ class RunQueueData: (mc, fn, taskname, taskfn) =3D split_tid_mcfn(tid) #runtid =3D build_tid(mc, fn, taskname) =20 - #logger.debug(2, "Processing %s,%s:%s", mc, fn, taskname) + #logger.debug2("Processing %s,%s:%s", mc, fn, taskname) =20 depends =3D set() task_deps =3D self.dataCaches[mc].task_deps[taskfn] @@ -1184,17 +1201,18 @@ class RunQueueData: return len(self.runtaskentries) =20 def prepare_task_hash(self, tid): - bb.parse.siggen.prep_taskhash(tid, self.runtaskentries[tid].depend= s, self.dataCaches[mc_from_tid(tid)]) - self.runtaskentries[tid].hash =3D bb.parse.siggen.get_taskhash(tid= , self.runtaskentries[tid].depends, self.dataCaches[mc_from_tid(tid)]) + dc =3D bb.parse.siggen.get_data_caches(self.dataCaches, mc_from_ti= d(tid)) + bb.parse.siggen.prep_taskhash(tid, self.runtaskentries[tid].depend= s, dc) + self.runtaskentries[tid].hash =3D bb.parse.siggen.get_taskhash(tid= , self.runtaskentries[tid].depends, dc) self.runtaskentries[tid].unihash =3D bb.parse.siggen.get_unihash(t= id) =20 def dump_data(self): """ Dump some debug information on the internal data structures """ - logger.debug(3, "run_tasks:") + logger.debug3("run_tasks:") for tid in self.runtaskentries: - logger.debug(3, " %s: %s Deps %s RevDeps %s", tid, + logger.debug3(" %s: %s Deps %s RevDeps %s", tid, self.runtaskentries[tid].weight, self.runtaskentries[tid].depends, self.runtaskentries[tid].revdeps) @@ -1231,10 +1249,11 @@ class RunQueue: self.fakeworker =3D {} =20 def _start_worker(self, mc, fakeroot =3D False, rqexec =3D None): - logger.debug(1, "Starting bitbake-worker") + logger.debug("Starting bitbake-worker") magic =3D "decafbad" if self.cooker.configuration.profile: magic =3D "decafbadbad" + fakerootlogs =3D None if fakeroot: magic =3D magic + "beef" mcdata =3D self.cooker.databuilder.mcdata[mc] @@ -1244,10 +1263,11 @@ class RunQueue: for key, value in (var.split('=3D') for var in fakerootenv): env[key] =3D value worker =3D subprocess.Popen(fakerootcmd + ["bitbake-worker", m= agic], stdout=3Dsubprocess.PIPE, stdin=3Dsubprocess.PIPE, env=3Denv) + fakerootlogs =3D self.rqdata.dataCaches[mc].fakerootlogs else: worker =3D subprocess.Popen(["bitbake-worker", magic], stdout= =3Dsubprocess.PIPE, stdin=3Dsubprocess.PIPE) bb.utils.nonblockingfd(worker.stdout) - workerpipe =3D runQueuePipe(worker.stdout, None, self.cfgData, sel= f, rqexec) + workerpipe =3D runQueuePipe(worker.stdout, None, self.cfgData, sel= f, rqexec, fakerootlogs=3Dfakerootlogs) =20 workerdata =3D { "taskdeps" : self.rqdata.dataCaches[mc].task_deps, @@ -1256,14 +1276,15 @@ class RunQueue: "fakerootnoenv" : self.rqdata.dataCaches[mc].fakerootnoenv, "sigdata" : bb.parse.siggen.get_taskdata(), "logdefaultlevel" : bb.msg.loggerDefaultLogLevel, - "logdefaultverbose" : bb.msg.loggerDefaultVerbose, - "logdefaultverboselogs" : bb.msg.loggerVerboseLogs, + "build_verbose_shell" : self.cooker.configuration.build_verbos= e_shell, + "build_verbose_stdout" : self.cooker.configuration.build_verbo= se_stdout, "logdefaultdomain" : bb.msg.loggerDefaultDomains, "prhost" : self.cooker.prhost, "buildname" : self.cfgData.getVar("BUILDNAME"), "date" : self.cfgData.getVar("DATE"), "time" : self.cfgData.getVar("TIME"), "hashservaddr" : self.cooker.hashservaddr, + "umask" : self.cfgData.getVar("BB_DEFAULT_UMASK"), } =20 worker.stdin.write(b"<cookerconfig>" + pickle.dumps(self.cooker.co= nfiguration) + b"</cookerconfig>") @@ -1276,7 +1297,7 @@ class RunQueue: def _teardown_worker(self, worker): if not worker: return - logger.debug(1, "Teardown for bitbake-worker") + logger.debug("Teardown for bitbake-worker") try: worker.process.stdin.write(b"<quit></quit>") worker.process.stdin.flush() @@ -1349,12 +1370,12 @@ class RunQueue: =20 # If the stamp is missing, it's not current if not os.access(stampfile, os.F_OK): - logger.debug(2, "Stampfile %s not available", stampfile) + logger.debug2("Stampfile %s not available", stampfile) return False # If it's a 'nostamp' task, it's not current taskdep =3D self.rqdata.dataCaches[mc].task_deps[taskfn] if 'nostamp' in taskdep and taskname in taskdep['nostamp']: - logger.debug(2, "%s.%s is nostamp\n", fn, taskname) + logger.debug2("%s.%s is nostamp\n", fn, taskname) return False =20 if taskname !=3D "do_setscene" and taskname.endswith("_setscene"): @@ -1378,18 +1399,18 @@ class RunQueue: continue if fn =3D=3D fn2 or (fulldeptree and fn2 not in stampwhite= list): if not t2: - logger.debug(2, 'Stampfile %s does not exist', sta= mpfile2) + logger.debug2('Stampfile %s does not exist', stamp= file2) iscurrent =3D False break if t1 < t2: - logger.debug(2, 'Stampfile %s < %s', stampfile, st= ampfile2) + logger.debug2('Stampfile %s < %s', stampfile, stam= pfile2) iscurrent =3D False break if recurse and iscurrent: if dep in cache: iscurrent =3D cache[dep] if not iscurrent: - logger.debug(2, 'Stampfile for dependency = %s:%s invalid (cached)' % (fn2, taskname2)) + logger.debug2('Stampfile for dependency %s= :%s invalid (cached)' % (fn2, taskname2)) else: iscurrent =3D self.check_stamp_task(dep, recur= se=3DTrue, cache=3Dcache) cache[dep] =3D iscurrent @@ -1459,7 +1480,7 @@ class RunQueue: if not self.dm_event_handler_registered: res =3D bb.event.register(self.dm_event_handler_name, lambda x: self.dm.check(self) if = self.state in [runQueueRunning, runQueueCleanUp] else False, - ('bb.event.HeartbeatEvent',)) + ('bb.event.HeartbeatEvent',), dat= a=3Dself.cfgData) self.dm_event_handler_registered =3D True =20 dump =3D self.cooker.configuration.dump_signatures @@ -1498,7 +1519,7 @@ class RunQueue: build_done =3D self.state is runQueueComplete or self.state is run= QueueFailed =20 if build_done and self.dm_event_handler_registered: - bb.event.remove(self.dm_event_handler_name, None) + bb.event.remove(self.dm_event_handler_name, None, data=3Dself.= cfgData) self.dm_event_handler_registered =3D False =20 if build_done and self.rqexe: @@ -1557,7 +1578,8 @@ class RunQueue: =20 def rq_dump_sigfn(self, fn, options): bb_cache =3D bb.cache.NoCache(self.cooker.databuilder) - the_data =3D bb_cache.loadDataFull(fn, self.cooker.collection.get_= file_appends(fn)) + mc =3D bb.runqueue.mc_from_tid(fn) + the_data =3D bb_cache.loadDataFull(fn, self.cooker.collections[mc]= .get_file_appends(fn)) siggen =3D bb.parse.siggen dataCaches =3D self.rqdata.dataCaches siggen.dump_sigfn(fn, dataCaches, options) @@ -1724,8 +1746,7 @@ class RunQueueExecute: self.holdoff_need_update =3D True self.sqdone =3D False =20 - self.stats =3D RunQueueStats(len(self.rqdata.runtaskentries)) - self.sq_stats =3D RunQueueStats(len(self.rqdata.runq_setscene_tids= )) + self.stats =3D RunQueueStats(len(self.rqdata.runtaskentries), len(= self.rqdata.runq_setscene_tids)) =20 for mc in rq.worker: rq.worker[mc].pipe.setrunqueueexec(self) @@ -1753,7 +1774,7 @@ class RunQueueExecute: for scheduler in schedulers: if self.scheduler =3D=3D scheduler.name: self.sched =3D scheduler(self, self.rqdata) - logger.debug(1, "Using runqueue scheduler '%s'", scheduler= .name) + logger.debug("Using runqueue scheduler '%s'", scheduler.na= me) break else: bb.fatal("Invalid scheduler '%s'. Available schedulers: %s" % @@ -1763,7 +1784,7 @@ class RunQueueExecute: self.sqdata =3D SQData() build_scenequeue_data(self.sqdata, self.rqdata, self.rq, self.cook= er, self.stampcache, self) =20 - def runqueue_process_waitpid(self, task, status): + def runqueue_process_waitpid(self, task, status, fakerootlog=3DNone): =20 # self.build_stamps[pid] may not exist when use shared work direct= ory. if task in self.build_stamps: @@ -1776,9 +1797,10 @@ class RunQueueExecute: else: self.sq_task_complete(task) self.sq_live.remove(task) + self.stats.updateActiveSetscene(len(self.sq_live)) else: if status !=3D 0: - self.task_fail(task, status) + self.task_fail(task, status, fakerootlog=3Dfakerootlog) else: self.task_complete(task) return True @@ -1809,7 +1831,7 @@ class RunQueueExecute: def finish(self): self.rq.state =3D runQueueCleanUp =20 - active =3D self.stats.active + self.sq_stats.active + active =3D self.stats.active + len(self.sq_live) if active > 0: bb.event.fire(runQueueExitWait(active), self.cfgData) self.rq.read_workers() @@ -1842,7 +1864,7 @@ class RunQueueExecute: return valid =20 def can_start_task(self): - active =3D self.stats.active + self.sq_stats.active + active =3D self.stats.active + len(self.sq_live) can_start =3D active < self.number_tasks return can_start =20 @@ -1891,7 +1913,13 @@ class RunQueueExecute: break if alldeps: self.setbuildable(revdep) - logger.debug(1, "Marking task %s as buildable", revdep) + logger.debug("Marking task %s as buildable", revdep) + + for t in self.sq_deferred.copy(): + if self.sq_deferred[t] =3D=3D task: + logger.debug2("Deferred task %s now buildable" % t) + del self.sq_deferred[t] + update_scenequeue_data([t], self.sqdata, self.rqdata, self= .rq, self.cooker, self.stampcache, self, summary=3DFalse) =20 def task_complete(self, task): self.stats.taskCompleted() @@ -1899,14 +1927,31 @@ class RunQueueExecute: self.task_completeoutright(task) self.runq_tasksrun.add(task) =20 - def task_fail(self, task, exitcode): + def task_fail(self, task, exitcode, fakerootlog=3DNone): """ Called when a task has failed Updates the state engine with the failure """ self.stats.taskFailed() self.failed_tids.append(task) - bb.event.fire(runQueueTaskFailed(task, self.stats, exitcode, self.= rq), self.cfgData) + + fakeroot_log =3D "" + if fakerootlog and os.path.exists(fakerootlog): + with open(fakerootlog) as fakeroot_log_file: + fakeroot_failed =3D False + for line in reversed(fakeroot_log_file.readlines()): + for fakeroot_error in ['mismatch', 'error', 'fatal']: + if fakeroot_error in line.lower(): + fakeroot_failed =3D True + if 'doing new pid setup and server start' in line: + break + fakeroot_log =3D line + fakeroot_log + + if not fakeroot_failed: + fakeroot_log =3D None + + bb.event.fire(runQueueTaskFailed(task, self.stats, exitcode, self.= rq, fakeroot_log=3Dfakeroot_log), self.cfgData) + if self.rqdata.taskData[''].abort: self.rq.state =3D runQueueCleanUp =20 @@ -1921,8 +1966,8 @@ class RunQueueExecute: def summarise_scenequeue_errors(self): err =3D False if not self.sqdone: - logger.debug(1, 'We could skip tasks %s', "\n".join(sorted(sel= f.scenequeue_covered))) - completeevent =3D sceneQueueComplete(self.sq_stats, self.rq) + logger.debug('We could skip tasks %s', "\n".join(sorted(self.s= cenequeue_covered))) + completeevent =3D sceneQueueComplete(self.stats, self.rq) bb.event.fire(completeevent, self.cfgData) if self.sq_deferred: logger.error("Scenequeue had deferred entries: %s" % pprint.pf= ormat(self.sq_deferred)) @@ -1934,6 +1979,10 @@ class RunQueueExecute: logger.error("Scenequeue had holdoff tasks: %s" % pprint.pform= at(self.holdoff_tasks)) err =3D True =20 + for tid in self.scenequeue_covered.intersection(self.scenequeue_no= tcovered): + # No task should end up in both covered and uncovered, that is= a bug. + logger.error("Setscene task %s in both covered and notcovered.= " % tid) + for tid in self.rqdata.runq_setscene_tids: if tid not in self.scenequeue_covered and tid not in self.scen= equeue_notcovered: err =3D True @@ -1978,7 +2027,7 @@ class RunQueueExecute: if nexttask in self.sq_buildable and nexttask not in self.= sq_running and self.sqdata.stamps[nexttask] not in self.build_stamps.values= (): if nexttask not in self.sqdata.unskippable and len(sel= f.sqdata.sq_revdeps[nexttask]) > 0 and self.sqdata.sq_revdeps[nexttask].iss= ubset(self.scenequeue_covered) and self.check_dependencies(nexttask, self.s= qdata.sq_revdeps[nexttask]): if nexttask not in self.rqdata.target_tids: - logger.debug(2, "Skipping setscene for task %s= " % nexttask) + logger.debug2("Skipping setscene for task %s" = % nexttask) self.sq_task_skip(nexttask) self.scenequeue_notneeded.add(nexttask) if nexttask in self.sq_deferred: @@ -1991,28 +2040,26 @@ class RunQueueExecute: if nexttask in self.sq_deferred: if self.sq_deferred[nexttask] not in self.runq_com= plete: continue - logger.debug(1, "Task %s no longer deferred" % nex= ttask) + logger.debug("Task %s no longer deferred" % nextta= sk) del self.sq_deferred[nexttask] valid =3D self.rq.validate_hashes(set([nexttask]),= self.cooker.data, 0, False, summary=3DFalse) if not valid: - logger.debug(1, "%s didn't become valid, skipp= ing setscene" % nexttask) + logger.debug("%s didn't become valid, skipping= setscene" % nexttask) self.sq_task_failoutright(nexttask) return True - else: - self.sqdata.outrightfail.remove(nexttask) if nexttask in self.sqdata.outrightfail: - logger.debug(2, 'No package found, so skipping set= scene task %s', nexttask) + logger.debug2('No package found, so skipping setsc= ene task %s', nexttask) self.sq_task_failoutright(nexttask) return True if nexttask in self.sqdata.unskippable: - logger.debug(2, "Setscene task %s is unskippable" = % nexttask) + logger.debug2("Setscene task %s is unskippable" % = nexttask) task =3D nexttask break if task is not None: (mc, fn, taskname, taskfn) =3D split_tid_mcfn(task) taskname =3D taskname + "_setscene" if self.rq.check_stamp_task(task, taskname_from_tid(task), rec= urse =3D True, cache=3Dself.stampcache): - logger.debug(2, 'Stamp for underlying task %s is current, = so skipping setscene variant', task) + logger.debug2('Stamp for underlying task %s is current, so= skipping setscene variant', task) self.sq_task_failoutright(task) return True =20 @@ -2022,16 +2069,16 @@ class RunQueueExecute: return True =20 if self.rq.check_stamp_task(task, taskname, cache=3Dself.stamp= cache): - logger.debug(2, 'Setscene stamp current task %s, so skip i= t and its dependencies', task) + logger.debug2('Setscene stamp current task %s, so skip it = and its dependencies', task) self.sq_task_skip(task) return True =20 if self.cooker.configuration.skipsetscene: - logger.debug(2, 'No setscene tasks should be executed. Ski= pping %s', task) + logger.debug2('No setscene tasks should be executed. Skipp= ing %s', task) self.sq_task_failoutright(task) return True =20 - startevent =3D sceneQueueTaskStarted(task, self.sq_stats, self= .rq) + startevent =3D sceneQueueTaskStarted(task, self.stats, self.rq) bb.event.fire(startevent, self.cfgData) =20 taskdepdata =3D self.sq_build_taskdepdata(task) @@ -2042,17 +2089,17 @@ class RunQueueExecute: if 'fakeroot' in taskdep and taskname in taskdep['fakeroot'] a= nd not self.cooker.configuration.dry_run: if not mc in self.rq.fakeworker: self.rq.start_fakeworker(self, mc) - self.rq.fakeworker[mc].process.stdin.write(b"<runtask>" + = pickle.dumps((taskfn, task, taskname, taskhash, unihash, True, self.cooker.= collection.get_file_appends(taskfn), taskdepdata, False)) + b"</runtask>") + self.rq.fakeworker[mc].process.stdin.write(b"<runtask>" + = pickle.dumps((taskfn, task, taskname, taskhash, unihash, True, self.cooker.= collections[mc].get_file_appends(taskfn), taskdepdata, False)) + b"</runtas= k>") self.rq.fakeworker[mc].process.stdin.flush() else: - self.rq.worker[mc].process.stdin.write(b"<runtask>" + pick= le.dumps((taskfn, task, taskname, taskhash, unihash, True, self.cooker.coll= ection.get_file_appends(taskfn), taskdepdata, False)) + b"</runtask>") + self.rq.worker[mc].process.stdin.write(b"<runtask>" + pick= le.dumps((taskfn, task, taskname, taskhash, unihash, True, self.cooker.coll= ections[mc].get_file_appends(taskfn), taskdepdata, False)) + b"</runtask>") self.rq.worker[mc].process.stdin.flush() =20 self.build_stamps[task] =3D bb.build.stampfile(taskname, self.= rqdata.dataCaches[mc], taskfn, noextra=3DTrue) self.build_stamps2.append(self.build_stamps[task]) self.sq_running.add(task) self.sq_live.add(task) - self.sq_stats.taskActive() + self.stats.updateActiveSetscene(len(self.sq_live)) if self.can_start_task(): return True =20 @@ -2089,12 +2136,12 @@ class RunQueueExecute: return True =20 if task in self.tasks_covered: - logger.debug(2, "Setscene covered task %s", task) + logger.debug2("Setscene covered task %s", task) self.task_skip(task, "covered") return True =20 if self.rq.check_stamp_task(task, taskname, cache=3Dself.stamp= cache): - logger.debug(2, "Stamp current task %s", task) + logger.debug2("Stamp current task %s", task) =20 self.task_skip(task, "existing") self.runq_tasksrun.add(task) @@ -2129,10 +2176,10 @@ class RunQueueExecute: self.rq.state =3D runQueueFailed self.stats.taskFailed() return True - self.rq.fakeworker[mc].process.stdin.write(b"<runtask>" + = pickle.dumps((taskfn, task, taskname, taskhash, unihash, False, self.cooker= .collection.get_file_appends(taskfn), taskdepdata, self.rqdata.setscene_enf= orce)) + b"</runtask>") + self.rq.fakeworker[mc].process.stdin.write(b"<runtask>" + = pickle.dumps((taskfn, task, taskname, taskhash, unihash, False, self.cooker= .collections[mc].get_file_appends(taskfn), taskdepdata, self.rqdata.setscen= e_enforce)) + b"</runtask>") self.rq.fakeworker[mc].process.stdin.flush() else: - self.rq.worker[mc].process.stdin.write(b"<runtask>" + pick= le.dumps((taskfn, task, taskname, taskhash, unihash, False, self.cooker.col= lection.get_file_appends(taskfn), taskdepdata, self.rqdata.setscene_enforce= )) + b"</runtask>") + self.rq.worker[mc].process.stdin.write(b"<runtask>" + pick= le.dumps((taskfn, task, taskname, taskhash, unihash, False, self.cooker.col= lections[mc].get_file_appends(taskfn), taskdepdata, self.rqdata.setscene_en= force)) + b"</runtask>") self.rq.worker[mc].process.stdin.flush() =20 self.build_stamps[task] =3D bb.build.stampfile(taskname, self.= rqdata.dataCaches[mc], taskfn, noextra=3DTrue) @@ -2142,7 +2189,7 @@ class RunQueueExecute: if self.can_start_task(): return True =20 - if self.stats.active > 0 or self.sq_stats.active > 0: + if self.stats.active > 0 or len(self.sq_live) > 0: self.rq.read_workers() return self.rq.active_fds() =20 @@ -2150,7 +2197,8 @@ class RunQueueExecute: if self.sq_deferred: tid =3D self.sq_deferred.pop(list(self.sq_deferred.keys())[0]) logger.warning("Runqeueue deadlocked on deferred tasks, forcin= g task %s" % tid) - self.sq_task_failoutright(tid) + if tid not in self.runq_complete: + self.sq_task_failoutright(tid) return True =20 if len(self.failed_tids) !=3D 0: @@ -2264,10 +2312,16 @@ class RunQueueExecute: self.updated_taskhash_queue.remove((tid, unihash)) =20 if unihash !=3D self.rqdata.runtaskentries[tid].unihash: - hashequiv_logger.verbose("Task %s unihash changed to %s" %= (tid, unihash)) - self.rqdata.runtaskentries[tid].unihash =3D unihash - bb.parse.siggen.set_unihash(tid, unihash) - toprocess.add(tid) + # Make sure we rehash any other tasks with the same task h= ash that we're deferred against. + torehash =3D [tid] + for deftid in self.sq_deferred: + if self.sq_deferred[deftid] =3D=3D tid: + torehash.append(deftid) + for hashtid in torehash: + hashequiv_logger.verbose("Task %s unihash changed to %= s" % (hashtid, unihash)) + self.rqdata.runtaskentries[hashtid].unihash =3D unihash + bb.parse.siggen.set_unihash(hashtid, unihash) + toprocess.add(hashtid) =20 # Work out all tasks which depend upon these total =3D set() @@ -2298,7 +2352,8 @@ class RunQueueExecute: if len(self.rqdata.runtaskentries[p].depends) and not self= .rqdata.runtaskentries[tid].depends.isdisjoint(total): continue orighash =3D self.rqdata.runtaskentries[tid].hash - newhash =3D bb.parse.siggen.get_taskhash(tid, self.rqdata.= runtaskentries[tid].depends, self.rqdata.dataCaches[mc_from_tid(tid)]) + dc =3D bb.parse.siggen.get_data_caches(self.rqdata.dataCac= hes, mc_from_tid(tid)) + newhash =3D bb.parse.siggen.get_taskhash(tid, self.rqdata.= runtaskentries[tid].depends, dc) origuni =3D self.rqdata.runtaskentries[tid].unihash newuni =3D bb.parse.siggen.get_unihash(tid) # FIXME, need to check it can come from sstate at all for = determinism? @@ -2313,7 +2368,7 @@ class RunQueueExecute: remapped =3D True =20 if not remapped: - #logger.debug(1, "Task %s hash changes: %s->%s %s->%s"= % (tid, orighash, newhash, origuni, newuni)) + #logger.debug("Task %s hash changes: %s->%s %s->%s" % = (tid, orighash, newhash, origuni, newuni)) self.rqdata.runtaskentries[tid].hash =3D newhash self.rqdata.runtaskentries[tid].unihash =3D newuni changed.add(tid) @@ -2328,7 +2383,7 @@ class RunQueueExecute: for mc in self.rq.fakeworker: self.rq.fakeworker[mc].process.stdin.write(b"<newtaskhashe= s>" + pickle.dumps(bb.parse.siggen.get_taskhashes()) + b"</newtaskhashes>") =20 - hashequiv_logger.debug(1, pprint.pformat("Tasks changed:\n%s" = % (changed))) + hashequiv_logger.debug(pprint.pformat("Tasks changed:\n%s" % (= changed))) =20 for tid in changed: if tid not in self.rqdata.runq_setscene_tids: @@ -2347,7 +2402,7 @@ class RunQueueExecute: # Check no tasks this covers are running for dep in self.sqdata.sq_covered_tasks[tid]: if dep in self.runq_running and dep not in self.runq_compl= ete: - hashequiv_logger.debug(2, "Task %s is running which bl= ocks setscene for %s from running" % (dep, tid)) + hashequiv_logger.debug2("Task %s is running which bloc= ks setscene for %s from running" % (dep, tid)) valid =3D False break if not valid: @@ -2406,6 +2461,11 @@ class RunQueueExecute: =20 if update_tasks: self.sqdone =3D False + for tid in [t[0] for t in update_tasks]: + h =3D pending_hash_index(tid, self.rqdata) + if h in self.sqdata.hashes and tid !=3D self.sqdata.hashes= [h]: + self.sq_deferred[tid] =3D self.sqdata.hashes[h] + bb.note("Deferring %s after %s" % (tid, self.sqdata.ha= shes[h])) update_scenequeue_data([t[0] for t in update_tasks], self.sqda= ta, self.rqdata, self.rq, self.cooker, self.stampcache, self, summary=3DFal= se) =20 for (tid, harddepfail, origvalid) in update_tasks: @@ -2415,13 +2475,20 @@ class RunQueueExecute: self.sq_task_failoutright(tid) =20 if changed: + self.stats.updateCovered(len(self.scenequeue_covered), len(sel= f.scenequeue_notcovered)) self.holdoff_need_update =3D True =20 def scenequeue_updatecounters(self, task, fail=3DFalse): =20 for dep in sorted(self.sqdata.sq_deps[task]): if fail and task in self.sqdata.sq_harddeps and dep in self.sq= data.sq_harddeps[task]: - logger.debug(2, "%s was unavailable and is a hard dependen= cy of %s so skipping" % (task, dep)) + if dep in self.scenequeue_covered or dep in self.scenequeu= e_notcovered: + # dependency could be already processed, e.g. noexec s= etscene task + continue + noexec, stamppresent =3D check_setscene_stamps(dep, self.r= qdata, self.rq, self.stampcache) + if noexec or stamppresent: + continue + logger.debug2("%s was unavailable and is a hard dependency= of %s so skipping" % (task, dep)) self.sq_task_failoutright(dep) continue if self.sqdata.sq_revdeps[dep].issubset(self.scenequeue_covere= d | self.scenequeue_notcovered): @@ -2442,6 +2509,7 @@ class RunQueueExecute: new.add(dep) next =3D new =20 + self.stats.updateCovered(len(self.scenequeue_covered), len(self.sc= enequeue_notcovered)) self.holdoff_need_update =3D True =20 def sq_task_completeoutright(self, task): @@ -2451,7 +2519,7 @@ class RunQueueExecute: completed dependencies as buildable """ =20 - logger.debug(1, 'Found task %s which could be accelerated', task) + logger.debug('Found task %s which could be accelerated', task) self.scenequeue_covered.add(task) self.scenequeue_updatecounters(task) =20 @@ -2465,13 +2533,11 @@ class RunQueueExecute: self.rq.state =3D runQueueCleanUp =20 def sq_task_complete(self, task): - self.sq_stats.taskCompleted() - bb.event.fire(sceneQueueTaskCompleted(task, self.sq_stats, self.rq= ), self.cfgData) + bb.event.fire(sceneQueueTaskCompleted(task, self.stats, self.rq), = self.cfgData) self.sq_task_completeoutright(task) =20 def sq_task_fail(self, task, result): - self.sq_stats.taskFailed() - bb.event.fire(sceneQueueTaskFailed(task, self.sq_stats, result, se= lf), self.cfgData) + bb.event.fire(sceneQueueTaskFailed(task, self.stats, result, self)= , self.cfgData) self.scenequeue_notcovered.add(task) self.scenequeue_updatecounters(task, True) self.sq_check_taskfail(task) @@ -2479,8 +2545,6 @@ class RunQueueExecute: def sq_task_failoutright(self, task): self.sq_running.add(task) self.sq_buildable.add(task) - self.sq_stats.taskSkipped() - self.sq_stats.taskCompleted() self.scenequeue_notcovered.add(task) self.scenequeue_updatecounters(task, True) =20 @@ -2488,8 +2552,6 @@ class RunQueueExecute: self.sq_running.add(task) self.sq_buildable.add(task) self.sq_task_completeoutright(task) - self.sq_stats.taskSkipped() - self.sq_stats.taskCompleted() =20 def sq_build_taskdepdata(self, task): def getsetscenedeps(tid): @@ -2743,8 +2805,55 @@ def build_scenequeue_data(sqdata, rqdata, rq, cooker= , stampcache, sqrq): sqdata.stamppresent =3D set() sqdata.valid =3D set() =20 + sqdata.hashes =3D {} + sqrq.sq_deferred =3D {} + for mc in sorted(sqdata.multiconfigs): + for tid in sorted(sqdata.sq_revdeps): + if mc_from_tid(tid) !=3D mc: + continue + h =3D pending_hash_index(tid, rqdata) + if h not in sqdata.hashes: + sqdata.hashes[h] =3D tid + else: + sqrq.sq_deferred[tid] =3D sqdata.hashes[h] + bb.note("Deferring %s after %s" % (tid, sqdata.hashes[h])) + update_scenequeue_data(sqdata.sq_revdeps, sqdata, rqdata, rq, cooker, = stampcache, sqrq, summary=3DTrue) =20 + # Compute a list of 'stale' sstate tasks where the current hash does n= ot match the one + # in any stamp files. Pass the list out to metadata as an event. + found =3D {} + for tid in rqdata.runq_setscene_tids: + (mc, fn, taskname, taskfn) =3D split_tid_mcfn(tid) + stamps =3D bb.build.find_stale_stamps(taskname, rqdata.dataCaches[= mc], taskfn) + if stamps: + if mc not in found: + found[mc] =3D {} + found[mc][tid] =3D stamps + for mc in found: + event =3D bb.event.StaleSetSceneTasks(found[mc]) + bb.event.fire(event, cooker.databuilder.mcdata[mc]) + +def check_setscene_stamps(tid, rqdata, rq, stampcache, noexecstamp=3DFalse= ): + + (mc, fn, taskname, taskfn) =3D split_tid_mcfn(tid) + + taskdep =3D rqdata.dataCaches[mc].task_deps[taskfn] + + if 'noexec' in taskdep and taskname in taskdep['noexec']: + bb.build.make_stamp(taskname + "_setscene", rqdata.dataCaches[mc],= taskfn) + return True, False + + if rq.check_stamp_task(tid, taskname + "_setscene", cache=3Dstampcache= ): + logger.debug2('Setscene stamp current for task %s', tid) + return False, True + + if rq.check_stamp_task(tid, taskname, recurse =3D True, cache=3Dstampc= ache): + logger.debug2('Normal stamp current for task %s', tid) + return False, True + + return False, False + def update_scenequeue_data(tids, sqdata, rqdata, rq, cooker, stampcache, s= qrq, summary=3DTrue): =20 tocheck =3D set() @@ -2754,25 +2863,17 @@ def update_scenequeue_data(tids, sqdata, rqdata, rq= , cooker, stampcache, sqrq, s sqdata.stamppresent.remove(tid) if tid in sqdata.valid: sqdata.valid.remove(tid) + if tid in sqdata.outrightfail: + sqdata.outrightfail.remove(tid) =20 - (mc, fn, taskname, taskfn) =3D split_tid_mcfn(tid) - - taskdep =3D rqdata.dataCaches[mc].task_deps[taskfn] + noexec, stamppresent =3D check_setscene_stamps(tid, rqdata, rq, st= ampcache, noexecstamp=3DTrue) =20 - if 'noexec' in taskdep and taskname in taskdep['noexec']: + if noexec: sqdata.noexec.add(tid) sqrq.sq_task_skip(tid) - bb.build.make_stamp(taskname + "_setscene", rqdata.dataCaches[= mc], taskfn) continue =20 - if rq.check_stamp_task(tid, taskname + "_setscene", cache=3Dstampc= ache): - logger.debug(2, 'Setscene stamp current for task %s', tid) - sqdata.stamppresent.add(tid) - sqrq.sq_task_skip(tid) - continue - - if rq.check_stamp_task(tid, taskname, recurse =3D True, cache=3Dst= ampcache): - logger.debug(2, 'Normal stamp current for task %s', tid) + if stamppresent: sqdata.stamppresent.add(tid) sqrq.sq_task_skip(tid) continue @@ -2781,28 +2882,20 @@ def update_scenequeue_data(tids, sqdata, rqdata, rq= , cooker, stampcache, sqrq, s =20 sqdata.valid |=3D rq.validate_hashes(tocheck, cooker.data, len(sqdata.= stamppresent), False, summary=3Dsummary) =20 - sqdata.hashes =3D {} - for mc in sorted(sqdata.multiconfigs): - for tid in sorted(sqdata.sq_revdeps): - if mc_from_tid(tid) !=3D mc: - continue - if tid in sqdata.stamppresent: - continue - if tid in sqdata.valid: - continue - if tid in sqdata.noexec: - continue - if tid in sqrq.scenequeue_notcovered: - continue - sqdata.outrightfail.add(tid) - - h =3D pending_hash_index(tid, rqdata) - if h not in sqdata.hashes: - sqdata.hashes[h] =3D tid - else: - sqrq.sq_deferred[tid] =3D sqdata.hashes[h] - bb.note("Deferring %s after %s" % (tid, sqdata.hashes[h])) - + for tid in tids: + if tid in sqdata.stamppresent: + continue + if tid in sqdata.valid: + continue + if tid in sqdata.noexec: + continue + if tid in sqrq.scenequeue_covered: + continue + if tid in sqrq.scenequeue_notcovered: + continue + if tid in sqrq.sq_deferred: + continue + sqdata.outrightfail.add(tid) =20 class TaskFailure(Exception): """ @@ -2866,12 +2959,16 @@ class runQueueTaskFailed(runQueueEvent): """ Event notifying a task failed """ - def __init__(self, task, stats, exitcode, rq): + def __init__(self, task, stats, exitcode, rq, fakeroot_log=3DNone): runQueueEvent.__init__(self, task, stats, rq) self.exitcode =3D exitcode + self.fakeroot_log =3D fakeroot_log =20 def __str__(self): - return "Task (%s) failed with exit code '%s'" % (self.taskstring, = self.exitcode) + if self.fakeroot_log: + return "Task (%s) failed with exit code '%s' \nPseudo log:\n%s= " % (self.taskstring, self.exitcode, self.fakeroot_log) + else: + return "Task (%s) failed with exit code '%s'" % (self.taskstri= ng, self.exitcode) =20 class sceneQueueTaskFailed(sceneQueueEvent): """ @@ -2923,7 +3020,7 @@ class runQueuePipe(): """ Abstraction for a pipe between a worker thread and the server """ - def __init__(self, pipein, pipeout, d, rq, rqexec): + def __init__(self, pipein, pipeout, d, rq, rqexec, fakerootlogs=3DNone= ): self.input =3D pipein if pipeout: pipeout.close() @@ -2932,6 +3029,7 @@ class runQueuePipe(): self.d =3D d self.rq =3D rq self.rqexec =3D rqexec + self.fakerootlogs =3D fakerootlogs =20 def setrunqueueexec(self, rqexec): self.rqexec =3D rqexec @@ -2977,7 +3075,11 @@ class runQueuePipe(): task, status =3D pickle.loads(self.queue[10:index]) except (ValueError, pickle.UnpicklingError, AttributeError= , IndexError) as e: bb.msg.fatal("RunQueue", "failed load pickle '%s': '%s= '" % (e, self.queue[10:index])) - self.rqexec.runqueue_process_waitpid(task, status) + (_, _, _, taskfn) =3D split_tid_mcfn(task) + fakerootlog =3D None + if self.fakerootlogs and taskfn and taskfn in self.fakeroo= tlogs: + fakerootlog =3D self.fakerootlogs[taskfn] + self.rqexec.runqueue_process_waitpid(task, status, fakeroo= tlog=3Dfakerootlog) found =3D True self.queue =3D self.queue[index+11:] index =3D self.queue.find(b"</exitcode>") @@ -2990,16 +3092,15 @@ class runQueuePipe(): print("Warning, worker left partial message: %s" % self.queue) self.input.close() =20 -def get_setscene_enforce_whitelist(d): +def get_setscene_enforce_whitelist(d, targets): if d.getVar('BB_SETSCENE_ENFORCE') !=3D '1': return None whitelist =3D (d.getVar("BB_SETSCENE_ENFORCE_WHITELIST") or "").split() outlist =3D [] for item in whitelist[:]: if item.startswith('%:'): - for target in sys.argv[1:]: - if not target.startswith('-'): - outlist.append(target.split(':')[0] + ':' + item.split= (':')[1]) + for (mc, target, task, fn) in targets: + outlist.append(target + ':' + item.split(':')[1]) else: outlist.append(item) return outlist diff --git a/bitbake/lib/bb/server/process.py b/bitbake/lib/bb/server/proce= ss.py index 9ec79f5b..07bb785a 100644 --- a/bitbake/lib/bb/server/process.py +++ b/bitbake/lib/bb/server/process.py @@ -25,6 +25,7 @@ import subprocess import errno import re import datetime +import pickle import bb.server.xmlrpcserver from bb import daemonize from multiprocessing import queues @@ -34,12 +35,15 @@ logger =3D logging.getLogger('BitBake') class ProcessTimeout(SystemExit): pass =20 -class ProcessServer(multiprocessing.Process): +def serverlog(msg): + print(str(os.getpid()) + " " + datetime.datetime.now().strftime('%H:%= M:%S.%f') + " " + msg) + sys.stdout.flush() + +class ProcessServer(): profile_filename =3D "profile.log" profile_processed_filename =3D "profile.log.processed" =20 - def __init__(self, lock, sock, sockname): - multiprocessing.Process.__init__(self) + def __init__(self, lock, lockname, sock, sockname, server_timeout, xml= rpcinterface): self.command_channel =3D False self.command_channel_reply =3D False self.quit =3D False @@ -47,6 +51,7 @@ class ProcessServer(multiprocessing.Process): self.next_heartbeat =3D time.time() =20 self.event_handle =3D None + self.hadanyui =3D False self.haveui =3D False self.maxuiwait =3D 30 self.xmlrpc =3D False @@ -54,9 +59,14 @@ class ProcessServer(multiprocessing.Process): self._idlefuns =3D {} =20 self.bitbake_lock =3D lock + self.bitbake_lock_name =3D lockname self.sock =3D sock self.sockname =3D sockname =20 + self.server_timeout =3D server_timeout + self.timeout =3D self.server_timeout + self.xmlrpcinterface =3D xmlrpcinterface + def register_idle_function(self, function, data): """Register a function to be called while the server is idle""" assert hasattr(function, '__call__') @@ -67,22 +77,7 @@ class ProcessServer(multiprocessing.Process): if self.xmlrpcinterface[0]: self.xmlrpc =3D bb.server.xmlrpcserver.BitBakeXMLRPCServer(sel= f.xmlrpcinterface, self.cooker, self) =20 - print("Bitbake XMLRPC server address: %s, server port: %s" % (= self.xmlrpc.host, self.xmlrpc.port)) - - heartbeat_event =3D self.cooker.data.getVar('BB_HEARTBEAT_EVENT') - if heartbeat_event: - try: - self.heartbeat_seconds =3D float(heartbeat_event) - except: - bb.warn('Ignoring invalid BB_HEARTBEAT_EVENT=3D%s, must be= a float specifying seconds.' % heartbeat_event) - - self.timeout =3D self.server_timeout or self.cooker.data.getVar('B= B_SERVER_TIMEOUT') - try: - if self.timeout: - self.timeout =3D float(self.timeout) - except: - bb.warn('Ignoring invalid BB_SERVER_TIMEOUT=3D%s, must be a fl= oat specifying seconds.' % self.timeout) - + serverlog("Bitbake XMLRPC server address: %s, server port: %s"= % (self.xmlrpc.host, self.xmlrpc.port)) =20 try: self.bitbake_lock.seek(0) @@ -93,7 +88,7 @@ class ProcessServer(multiprocessing.Process): self.bitbake_lock.write("%s\n" % (os.getpid())) self.bitbake_lock.flush() except Exception as e: - print("Error writing to lock file: %s" % str(e)) + serverlog("Error writing to lock file: %s" % str(e)) pass =20 if self.cooker.configuration.profile: @@ -107,7 +102,7 @@ class ProcessServer(multiprocessing.Process): =20 prof.dump_stats("profile.log") bb.utils.process_profilelog("profile.log") - print("Raw profiling information saved to profile.log and proc= essed statistics to profile.log.processed") + serverlog("Raw profiling information saved to profile.log and = processed statistics to profile.log.processed") =20 else: ret =3D self.main() @@ -126,10 +121,11 @@ class ProcessServer(multiprocessing.Process): fds =3D [self.sock] if self.xmlrpc: fds.append(self.xmlrpc) - print("Entering server connection loop") + seendata =3D False + serverlog("Entering server connection loop") =20 def disconnect_client(self, fds): - print("Disconnecting Client") + serverlog("Disconnecting Client") if self.controllersock: fds.remove(self.controllersock) self.controllersock.close() @@ -147,12 +143,12 @@ class ProcessServer(multiprocessing.Process): self.haveui =3D False ready =3D select.select(fds,[],[],0)[0] if newconnections: - print("Starting new client") + serverlog("Starting new client") conn =3D newconnections.pop(-1) fds.append(conn) self.controllersock =3D conn - elif self.timeout is None and not ready: - print("No timeout, exiting.") + elif not self.timeout and not ready: + serverlog("No timeout, exiting.") self.quit =3D True =20 self.lastui =3D time.time() @@ -161,17 +157,17 @@ class ProcessServer(multiprocessing.Process): while select.select([self.sock],[],[],0)[0]: controllersock, address =3D self.sock.accept() if self.controllersock: - print("Queuing %s (%s)" % (str(ready), str(newconn= ections))) + serverlog("Queuing %s (%s)" % (str(ready), str(new= connections))) newconnections.append(controllersock) else: - print("Accepting %s (%s)" % (str(ready), str(newco= nnections))) + serverlog("Accepting %s (%s)" % (str(ready), str(n= ewconnections))) self.controllersock =3D controllersock fds.append(controllersock) if self.controllersock in ready: try: - print("Processing Client") + serverlog("Processing Client") ui_fds =3D recvfds(self.controllersock, 3) - print("Connecting Client") + serverlog("Connecting Client") =20 # Where to write events to writer =3D ConnectionWriter(ui_fds[0]) @@ -188,20 +184,21 @@ class ProcessServer(multiprocessing.Process): self.command_channel_reply =3D writer =20 self.haveui =3D True + self.hadanyui =3D True =20 except (EOFError, OSError): disconnect_client(self, fds) =20 if not self.timeout =3D=3D -1.0 and not self.haveui and self.t= imeout and \ (self.lastui + self.timeout) < time.time(): - print("Server timeout, exiting.") + serverlog("Server timeout, exiting.") self.quit =3D True =20 # If we don't see a UI connection within maxuiwait, its unlike= ly we're going to see # one. We have had issue with processes hanging indefinitely s= o timing out UI-less # servers is useful. - if not self.haveui and not self.timeout and (self.lastui + sel= f.maxuiwait) < time.time(): - print("No UI connection within max timeout, exiting to avo= id infinite loop.") + if not self.hadanyui and not self.xmlrpc and not self.timeout = and (self.lastui + self.maxuiwait) < time.time(): + serverlog("No UI connection within max timeout, exiting to= avoid infinite loop.") self.quit =3D True =20 if self.command_channel in ready: @@ -216,17 +213,38 @@ class ProcessServer(multiprocessing.Process): self.quit =3D True continue try: - print("Running command %s" % command) + serverlog("Running command %s" % command) self.command_channel_reply.send(self.cooker.command.ru= nCommand(command)) + serverlog("Command Completed") except Exception as e: + serverlog('Exception in server main event loop running = command %s (%s)' % (command, str(e))) logger.exception('Exception in server main event loop r= unning command %s (%s)' % (command, str(e))) =20 if self.xmlrpc in ready: self.xmlrpc.handle_requests() =20 + if not seendata and hasattr(self.cooker, "data"): + heartbeat_event =3D self.cooker.data.getVar('BB_HEARTBEAT_= EVENT') + if heartbeat_event: + try: + self.heartbeat_seconds =3D float(heartbeat_event) + except: + bb.warn('Ignoring invalid BB_HEARTBEAT_EVENT=3D%s,= must be a float specifying seconds.' % heartbeat_event) + + self.timeout =3D self.server_timeout or self.cooker.data.g= etVar('BB_SERVER_TIMEOUT') + try: + if self.timeout: + self.timeout =3D float(self.timeout) + except: + bb.warn('Ignoring invalid BB_SERVER_TIMEOUT=3D%s, must= be a float specifying seconds.' % self.timeout) + seendata =3D True + ready =3D self.idle_commands(.1, fds) =20 - print("Exiting") + if len(threading.enumerate()) !=3D 1: + serverlog("More than one thread left?: " + str(threading.enume= rate())) + + serverlog("Exiting") # Remove the socket file so we don't get any more connections to a= void races try: os.unlink(self.sockname) @@ -243,41 +261,73 @@ class ProcessServer(multiprocessing.Process): =20 self.cooker.post_serve() =20 + # Flush logs before we release the lock + sys.stdout.flush() + sys.stderr.flush() + # Finally release the lockfile but warn about other processes hold= ing it open lock =3D self.bitbake_lock - lockfile =3D lock.name + lockfile =3D self.bitbake_lock_name + + def get_lock_contents(lockfile): + try: + with open(lockfile, "r") as f: + return f.readlines() + except FileNotFoundError: + return None + + lockcontents =3D get_lock_contents(lockfile) + serverlog("Original lockfile contents: " + str(lockcontents)) + lock.close() lock =3D None =20 while not lock: - with bb.utils.timeout(3): - lock =3D bb.utils.lockfile(lockfile, shared=3DFalse, retry= =3DFalse, block=3DTrue) - if lock: - # We hold the lock so we can remove the file (hide sta= le pid data) - # via unlockfile. - bb.utils.unlockfile(lock) - return - + i =3D 0 + lock =3D None + while not lock and i < 30: + lock =3D bb.utils.lockfile(lockfile, shared=3DFalse, retry= =3DFalse, block=3DFalse) if not lock: - # Some systems may not have lsof available - procs =3D None + newlockcontents =3D get_lock_contents(lockfile) + if newlockcontents !=3D lockcontents: + # A new server was started, the lockfile contents = changed, we can exit + serverlog("Lockfile now contains different content= s, exiting: " + str(newlockcontents)) + return + time.sleep(0.1) + i +=3D 1 + if lock: + # We hold the lock so we can remove the file (hide stale p= id data) + # via unlockfile. + bb.utils.unlockfile(lock) + serverlog("Exiting as we could obtain the lock") + return + + if not lock: + # Some systems may not have lsof available + procs =3D None + try: + procs =3D subprocess.check_output(["lsof", '-w', lockf= ile], stderr=3Dsubprocess.STDOUT) + except subprocess.CalledProcessError: + # File was deleted? + continue + except OSError as e: + if e.errno !=3D errno.ENOENT: + raise + if procs is None: + # Fall back to fuser if lsof is unavailable try: - procs =3D subprocess.check_output(["lsof", '-w', l= ockfile], stderr=3Dsubprocess.STDOUT) + procs =3D subprocess.check_output(["fuser", '-v', = lockfile], stderr=3Dsubprocess.STDOUT) + except subprocess.CalledProcessError: + # File was deleted? + continue except OSError as e: if e.errno !=3D errno.ENOENT: raise - if procs is None: - # Fall back to fuser if lsof is unavailable - try: - procs =3D subprocess.check_output(["fuser", '-= v', lockfile], stderr=3Dsubprocess.STDOUT) - except OSError as e: - if e.errno !=3D errno.ENOENT: - raise - - msg =3D "Delaying shutdown due to active processes whi= ch appear to be holding bitbake.lock" - if procs: - msg +=3D ":\n%s" % str(procs) - print(msg) + + msg =3D "Delaying shutdown due to active processes which a= ppear to be holding bitbake.lock" + if procs: + msg +=3D ":\n%s" % str(procs.decode("utf-8")) + serverlog(msg) =20 def idle_commands(self, delay, fds=3DNone): nextsleep =3D delay @@ -315,8 +365,14 @@ class ProcessServer(multiprocessing.Process): self.next_heartbeat +=3D self.heartbeat_seconds if self.next_heartbeat <=3D now: self.next_heartbeat =3D now + self.heartbeat_seconds - heartbeat =3D bb.event.HeartbeatEvent(now) - bb.event.fire(heartbeat, self.cooker.data) + if hasattr(self.cooker, "data"): + heartbeat =3D bb.event.HeartbeatEvent(now) + try: + bb.event.fire(heartbeat, self.cooker.data) + except Exception as exc: + if not isinstance(exc, bb.BBHandledException): + logger.exception('Running heartbeat function') + self.quit =3D True if nextsleep and now + nextsleep > self.next_heartbeat: # Shorten timeout so that we we wake up in time for # the heartbeat. @@ -345,7 +401,12 @@ class ServerCommunicator(): logger.info("No reply from server in 30s") if not self.recv.poll(30): raise ProcessTimeout("Timeout while waiting for a reply fr= om the bitbake server (60s)") - return self.recv.get() + ret, exc =3D self.recv.get() + # Should probably turn all exceptions in exc back into exceptions? + # For now, at least handle BBHandledException + if exc and ("BBHandledException" in exc or "SystemExit" in exc): + raise bb.BBHandledException() + return ret, exc =20 def updateFeatureSet(self, featureset): _, error =3D self.runCommand(["setFeatures", featureset]) @@ -378,39 +439,26 @@ class BitBakeProcessServerConnection(object): self.connection.recv.close() return =20 +start_log_format =3D '--- Starting bitbake server pid %s at %s ---' +start_log_datetime_format =3D '%Y-%m-%d %H:%M:%S.%f' + class BitBakeServer(object): - start_log_format =3D '--- Starting bitbake server pid %s at %s ---' - start_log_datetime_format =3D '%Y-%m-%d %H:%M:%S.%f' =20 - def __init__(self, lock, sockname, configuration, featureset): + def __init__(self, lock, sockname, featureset, server_timeout, xmlrpci= nterface): =20 - self.configuration =3D configuration + self.server_timeout =3D server_timeout + self.xmlrpcinterface =3D xmlrpcinterface self.featureset =3D featureset self.sockname =3D sockname self.bitbake_lock =3D lock self.readypipe, self.readypipein =3D os.pipe() =20 - # Create server control socket - if os.path.exists(sockname): - os.unlink(sockname) - # Place the log in the builddirectory alongside the lock file logfile =3D os.path.join(os.path.dirname(self.bitbake_lock.name), = "bitbake-cookerdaemon.log") + self.logfile =3D logfile =20 - self.sock =3D socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) - # AF_UNIX has path length issues so chdir here to workaround - cwd =3D os.getcwd() - try: - os.chdir(os.path.dirname(sockname)) - self.sock.bind(os.path.basename(sockname)) - finally: - os.chdir(cwd) - self.sock.listen(1) - - os.set_inheritable(self.sock.fileno(), True) startdatetime =3D datetime.datetime.now() bb.daemonize.createDaemon(self._startServer, logfile) - self.sock.close() self.bitbake_lock.close() os.close(self.readypipein) =20 @@ -423,13 +471,13 @@ class BitBakeServer(object): try: r =3D ready.get() except EOFError: - # Trap the child exitting/closing the pipe and error out + # Trap the child exiting/closing the pipe and error out r =3D None if not r or r[0] !=3D "r": ready.close() bb.error("Unable to start bitbake server (%s)" % str(r)) if os.path.exists(logfile): - logstart_re =3D re.compile(self.start_log_format % ('([0-9= ]+)', '([0-9-]+ [0-9:.]+)')) + logstart_re =3D re.compile(start_log_format % ('([0-9]+)',= '([0-9-]+ [0-9:.]+)')) started =3D False lines =3D [] lastlines =3D [] @@ -439,9 +487,9 @@ class BitBakeServer(object): lines.append(line) else: lastlines.append(line) - res =3D logstart_re.match(line.rstrip()) + res =3D logstart_re.search(line.rstrip()) if res: - ldatetime =3D datetime.datetime.strptime(r= es.group(2), self.start_log_datetime_format) + ldatetime =3D datetime.datetime.strptime(r= es.group(2), start_log_datetime_format) if ldatetime >=3D startdatetime: started =3D True lines.append(line) @@ -462,26 +510,53 @@ class BitBakeServer(object): ready.close() =20 def _startServer(self): - print(self.start_log_format % (os.getpid(), datetime.datetime.now(= ).strftime(self.start_log_datetime_format))) - sys.stdout.flush() - - server =3D ProcessServer(self.bitbake_lock, self.sock, self.sockna= me) - self.configuration.setServerRegIdleCallback(server.register_idle_f= unction) os.close(self.readypipe) - writer =3D ConnectionWriter(self.readypipein) + os.set_inheritable(self.bitbake_lock.fileno(), True) + os.set_inheritable(self.readypipein, True) + serverscript =3D os.path.realpath(os.path.dirname(__file__) + "/..= /../../bin/bitbake-server") + os.execl(sys.executable, "bitbake-server", serverscript, "decafbad= ", str(self.bitbake_lock.fileno()), str(self.readypipein), self.logfile, se= lf.bitbake_lock.name, self.sockname, str(self.server_timeout or 0), str(se= lf.xmlrpcinterface[0]), str(self.xmlrpcinterface[1])) + +def execServer(lockfd, readypipeinfd, lockname, sockname, server_timeout, = xmlrpcinterface): + + import bb.cookerdata + import bb.cooker + + serverlog(start_log_format % (os.getpid(), datetime.datetime.now().str= ftime(start_log_datetime_format))) + + try: + bitbake_lock =3D os.fdopen(lockfd, "w") + + # Create server control socket + if os.path.exists(sockname): + os.unlink(sockname) + + sock =3D socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + # AF_UNIX has path length issues so chdir here to workaround + cwd =3D os.getcwd() try: - self.cooker =3D bb.cooker.BBCooker(self.configuration, self.fe= atureset) + os.chdir(os.path.dirname(sockname)) + sock.bind(os.path.basename(sockname)) + finally: + os.chdir(cwd) + sock.listen(1) + + server =3D ProcessServer(bitbake_lock, lockname, sock, sockname, s= erver_timeout, xmlrpcinterface) + writer =3D ConnectionWriter(readypipeinfd) + try: + featureset =3D [] + cooker =3D bb.cooker.BBCooker(featureset, server.register_idle= _function) except bb.BBHandledException: return None writer.send("r") writer.close() - server.cooker =3D self.cooker - server.server_timeout =3D self.configuration.server_timeout - server.xmlrpcinterface =3D self.configuration.xmlrpcinterface - print("Started bitbake server pid %d" % os.getpid()) - sys.stdout.flush() + server.cooker =3D cooker + serverlog("Started bitbake server pid %d" % os.getpid()) =20 - server.start() + server.run() + finally: + # Flush any ,essages/errors to the logfile before exit + sys.stdout.flush() + sys.stderr.flush() =20 def connectProcessServer(sockname, featureset): # Connect to socket diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py index 94d1762d..0d88c6ec 100644 --- a/bitbake/lib/bb/siggen.py +++ b/bitbake/lib/bb/siggen.py @@ -39,6 +39,11 @@ class SignatureGenerator(object): """ name =3D "noop" =20 + # If the derived class supports multiconfig datacaches, set this to Tr= ue + # The default is False for backward compatibility with derived signatu= re + # generators that do not understand multiconfig caches + supports_multiconfig_datacaches =3D False + def __init__(self, data): self.basehash =3D {} self.taskhash =3D {} @@ -59,10 +64,10 @@ class SignatureGenerator(object): def get_unihash(self, tid): return self.taskhash[tid] =20 - def prep_taskhash(self, tid, deps, dataCache): + def prep_taskhash(self, tid, deps, dataCaches): return =20 - def get_taskhash(self, tid, deps, dataCache): + def get_taskhash(self, tid, deps, dataCaches): self.taskhash[tid] =3D hashlib.sha256(tid.encode("utf-8")).hexdige= st() return self.taskhash[tid] =20 @@ -106,6 +111,38 @@ class SignatureGenerator(object): def set_setscene_tasks(self, setscene_tasks): return =20 + @classmethod + def get_data_caches(cls, dataCaches, mc): + """ + This function returns the datacaches that should be passed to sign= ature + generator functions. If the signature generator supports multiconf= ig + caches, the entire dictionary of data caches is sent, otherwise a + special proxy is sent that support both index access to all + multiconfigs, and also direct access for the default multiconfig. + + The proxy class allows code in this class itself to always use + multiconfig aware code (to ease maintenance), but derived classes = that + are unaware of multiconfig data caches can still access the default + multiconfig as expected. + + Do not override this function in derived classes; it will be remov= ed in + the future when support for multiconfig data caches is mandatory + """ + class DataCacheProxy(object): + def __init__(self): + pass + + def __getitem__(self, key): + return dataCaches[key] + + def __getattr__(self, name): + return getattr(dataCaches[mc], name) + + if cls.supports_multiconfig_datacaches: + return dataCaches + + return DataCacheProxy() + class SignatureGeneratorBasic(SignatureGenerator): """ """ @@ -201,7 +238,7 @@ class SignatureGeneratorBasic(SignatureGenerator): self.lookupcache =3D {} self.taskdeps =3D {} =20 - def rundep_check(self, fn, recipename, task, dep, depname, dataCache): + def rundep_check(self, fn, recipename, task, dep, depname, dataCaches): # Return True if we should keep the dependency, False to drop it # We only manipulate the dependencies for packages not in the whit= elist if self.twl and not self.twl.search(recipename): @@ -219,37 +256,40 @@ class SignatureGeneratorBasic(SignatureGenerator): pass return taint =20 - def prep_taskhash(self, tid, deps, dataCache): + def prep_taskhash(self, tid, deps, dataCaches): =20 (mc, _, task, fn) =3D bb.runqueue.split_tid_mcfn(tid) =20 - self.basehash[tid] =3D dataCache.basetaskhash[tid] + self.basehash[tid] =3D dataCaches[mc].basetaskhash[tid] self.runtaskdeps[tid] =3D [] self.file_checksum_values[tid] =3D [] - recipename =3D dataCache.pkg_fn[fn] + recipename =3D dataCaches[mc].pkg_fn[fn] =20 self.tidtopn[tid] =3D recipename =20 for dep in sorted(deps, key=3Dclean_basepath): - (depmc, _, deptaskname, depfn) =3D bb.runqueue.split_tid_mcfn(= dep) - if mc !=3D depmc: + (depmc, _, _, depmcfn) =3D bb.runqueue.split_tid_mcfn(dep) + depname =3D dataCaches[depmc].pkg_fn[depmcfn] + if not self.supports_multiconfig_datacaches and mc !=3D depmc: + # If the signature generator doesn't understand multiconfig + # data caches, any dependency not in the same multiconfig = must + # be skipped for backward compatibility continue - depname =3D dataCache.pkg_fn[depfn] - if not self.rundep_check(fn, recipename, task, dep, depname, d= ataCache): + if not self.rundep_check(fn, recipename, task, dep, depname, d= ataCaches): continue if dep not in self.taskhash: bb.fatal("%s is not in taskhash, caller isn't calling in d= ependency order?" % dep) self.runtaskdeps[tid].append(dep) =20 - if task in dataCache.file_checksums[fn]: + if task in dataCaches[mc].file_checksums[fn]: if self.checksum_cache: - checksums =3D self.checksum_cache.get_checksums(dataCache.= file_checksums[fn][task], recipename, self.localdirsexclude) + checksums =3D self.checksum_cache.get_checksums(dataCaches= [mc].file_checksums[fn][task], recipename, self.localdirsexclude) else: - checksums =3D bb.fetch2.get_file_checksums(dataCache.file_= checksums[fn][task], recipename, self.localdirsexclude) + checksums =3D bb.fetch2.get_file_checksums(dataCaches[mc].= file_checksums[fn][task], recipename, self.localdirsexclude) for (f,cs) in checksums: self.file_checksum_values[tid].append((f,cs)) =20 - taskdep =3D dataCache.task_deps[fn] + taskdep =3D dataCaches[mc].task_deps[fn] if 'nostamp' in taskdep and task in taskdep['nostamp']: # Nostamp tasks need an implicit taint so that they force any = dependent tasks to run if tid in self.taints and self.taints[tid].startswith("nostamp= :"): @@ -260,24 +300,18 @@ class SignatureGeneratorBasic(SignatureGenerator): taint =3D str(uuid.uuid4()) self.taints[tid] =3D "nostamp:" + taint =20 - taint =3D self.read_taint(fn, task, dataCache.stamp[fn]) + taint =3D self.read_taint(fn, task, dataCaches[mc].stamp[fn]) if taint: self.taints[tid] =3D taint logger.warning("%s is tainted from a forced run" % tid) =20 return =20 - def get_taskhash(self, tid, deps, dataCache): + def get_taskhash(self, tid, deps, dataCaches): =20 data =3D self.basehash[tid] for dep in self.runtaskdeps[tid]: - if dep in self.unihash: - if self.unihash[dep] is None: - data =3D data + self.taskhash[dep] - else: - data =3D data + self.unihash[dep] - else: - data =3D data + self.get_unihash(dep) + data =3D data + self.get_unihash(dep) =20 for (f, cs) in self.file_checksum_values[tid]: if cs: @@ -318,7 +352,8 @@ class SignatureGeneratorBasic(SignatureGenerator): else: sigfile =3D stampbase + "." + task + ".sigbasedata" + "." + se= lf.basehash[tid] =20 - bb.utils.mkdirhier(os.path.dirname(sigfile)) + with bb.utils.umask(0o002): + bb.utils.mkdirhier(os.path.dirname(sigfile)) =20 data =3D {} data['task'] =3D task @@ -506,7 +541,7 @@ class SignatureGeneratorUniHashMixIn(object): # is much more interesting, so it is reported at debug lev= el 1 hashequiv_logger.debug((1, 2)[unihash =3D=3D taskhash], 'F= ound unihash %s in place of %s for %s from %s' % (unihash, taskhash, tid, s= elf.server)) else: - hashequiv_logger.debug(2, 'No reported unihash for %s:%s f= rom %s' % (tid, taskhash, self.server)) + hashequiv_logger.debug2('No reported unihash for %s:%s fro= m %s' % (tid, taskhash, self.server)) except hashserv.client.HashConnectionError as e: bb.warn('Error contacting Hash Equivalence Server %s: %s' % (s= elf.server, str(e))) =20 @@ -580,12 +615,12 @@ class SignatureGeneratorUniHashMixIn(object): new_unihash =3D data['unihash'] =20 if new_unihash !=3D unihash: - hashequiv_logger.debug(1, 'Task %s unihash changed %s = -> %s by server %s' % (taskhash, unihash, new_unihash, self.server)) + hashequiv_logger.debug('Task %s unihash changed %s -> = %s by server %s' % (taskhash, unihash, new_unihash, self.server)) bb.event.fire(bb.runqueue.taskUniHashUpdate(fn + ':do_= ' + task, new_unihash), d) self.set_unihash(tid, new_unihash) d.setVar('BB_UNIHASH', new_unihash) else: - hashequiv_logger.debug(1, 'Reported task %s as unihash= %s to %s' % (taskhash, unihash, self.server)) + hashequiv_logger.debug('Reported task %s as unihash %s= to %s' % (taskhash, unihash, self.server)) except hashserv.client.HashConnectionError as e: bb.warn('Error contacting Hash Equivalence Server %s: %s' = % (self.server, str(e))) finally: @@ -641,6 +676,12 @@ class SignatureGeneratorTestEquivHash(SignatureGenerat= orUniHashMixIn, SignatureG self.server =3D data.getVar('BB_HASHSERVE') self.method =3D "sstate_output_hash" =20 +# +# Dummy class used for bitbake-selftest +# +class SignatureGeneratorTestMulticonfigDepends(SignatureGeneratorBasicHash= ): + name =3D "TestMulticonfigDepends" + supports_multiconfig_datacaches =3D True =20 def dump_this_task(outfile, d): import bb.parse @@ -700,16 +741,26 @@ def list_inline_diff(oldlist, newlist, colors=3DNone): ret.append(item) return '[%s]' % (', '.join(ret)) =20 -def clean_basepath(a): - mc =3D None - if a.startswith("mc:"): - _, mc, a =3D a.split(":", 2) - b =3D a.rsplit("/", 2)[1] + '/' + a.rsplit("/", 2)[2] - if a.startswith("virtual:"): - b =3D b + ":" + a.rsplit(":", 1)[0] - if mc: - b =3D b + ":mc:" + mc - return b +def clean_basepath(basepath): + basepath, dir, recipe_task =3D basepath.rsplit("/", 2) + cleaned =3D dir + '/' + recipe_task + + if basepath[0] =3D=3D '/': + return cleaned + + if basepath.startswith("mc:") and basepath.count(':') >=3D 2: + mc, mc_name, basepath =3D basepath.split(":", 2) + mc_suffix =3D ':mc:' + mc_name + else: + mc_suffix =3D '' + + # mc stuff now removed from basepath. Whatever was next, if present wi= ll be the first + # suffix. ':/', recipe path start, marks the end of this. Something li= ke + # 'virtual:a[:b[:c]]:/path...' (b and c being optional) + if basepath[0] !=3D '/': + cleaned +=3D ':' + basepath.split(':/', 1)[0] + + return cleaned + mc_suffix =20 def clean_basepaths(a): b =3D {} diff --git a/bitbake/lib/bb/taskdata.py b/bitbake/lib/bb/taskdata.py index d13a1249..47bad6d1 100644 --- a/bitbake/lib/bb/taskdata.py +++ b/bitbake/lib/bb/taskdata.py @@ -21,8 +21,13 @@ def re_match_strings(target, strings): Whether or not the string 'target' matches any one string of the strings which can be regular expression string """ - return any(name =3D=3D target or re.match(name, target) - for name in strings) + for name in strings: + if name.startswith("^") or name.endswith("$"): + if re.match(name, target): + return True + elif name =3D=3D target: + return True + return False =20 class TaskEntry: def __init__(self): @@ -126,7 +131,7 @@ class TaskData: for depend in dataCache.deps[fn]: dependids.add(depend) self.depids[fn] =3D list(dependids) - logger.debug(2, "Added dependencies %s for %s", str(dataCache.= deps[fn]), fn) + logger.debug2("Added dependencies %s for %s", str(dataCache.de= ps[fn]), fn) =20 # Work out runtime dependencies if not fn in self.rdepids: @@ -144,9 +149,9 @@ class TaskData: rreclist.append(rdepend) rdependids.add(rdepend) if rdependlist: - logger.debug(2, "Added runtime dependencies %s for %s", st= r(rdependlist), fn) + logger.debug2("Added runtime dependencies %s for %s", str(= rdependlist), fn) if rreclist: - logger.debug(2, "Added runtime recommendations %s for %s",= str(rreclist), fn) + logger.debug2("Added runtime recommendations %s for %s", s= tr(rreclist), fn) self.rdepids[fn] =3D list(rdependids) =20 for dep in self.depids[fn]: @@ -373,7 +378,7 @@ class TaskData: for fn in eligible: if fn in self.failed_fns: continue - logger.debug(2, "adding %s to satisfy %s", fn, item) + logger.debug2("adding %s to satisfy %s", fn, item) self.add_build_target(fn, item) self.add_tasks(fn, dataCache) =20 @@ -426,7 +431,7 @@ class TaskData: for fn in eligible: if fn in self.failed_fns: continue - logger.debug(2, "adding '%s' to satisfy runtime '%s'", fn, ite= m) + logger.debug2("adding '%s' to satisfy runtime '%s'", fn, item) self.add_runtime_target(fn, item) self.add_tasks(fn, dataCache) =20 @@ -441,7 +446,7 @@ class TaskData: return if not missing_list: missing_list =3D [] - logger.debug(1, "File '%s' is unbuildable, removing...", fn) + logger.debug("File '%s' is unbuildable, removing...", fn) self.failed_fns.append(fn) for target in self.build_targets: if fn in self.build_targets[target]: @@ -521,7 +526,7 @@ class TaskData: added =3D added + 1 except (bb.providers.NoRProvider, bb.providers.MultipleRPr= ovider): self.remove_runtarget(target) - logger.debug(1, "Resolved " + str(added) + " extra dependencie= s") + logger.debug("Resolved " + str(added) + " extra dependencies") if added =3D=3D 0: break # self.dump_data() @@ -544,38 +549,38 @@ class TaskData: """ Dump some debug information on the internal data structures """ - logger.debug(3, "build_names:") - logger.debug(3, ", ".join(self.build_targets)) + logger.debug3("build_names:") + logger.debug3(", ".join(self.build_targets)) =20 - logger.debug(3, "run_names:") - logger.debug(3, ", ".join(self.run_targets)) + logger.debug3("run_names:") + logger.debug3(", ".join(self.run_targets)) =20 - logger.debug(3, "build_targets:") + logger.debug3("build_targets:") for target in self.build_targets: targets =3D "None" if target in self.build_targets: targets =3D self.build_targets[target] - logger.debug(3, " %s: %s", target, targets) + logger.debug3(" %s: %s", target, targets) =20 - logger.debug(3, "run_targets:") + logger.debug3("run_targets:") for target in self.run_targets: targets =3D "None" if target in self.run_targets: targets =3D self.run_targets[target] - logger.debug(3, " %s: %s", target, targets) + logger.debug3(" %s: %s", target, targets) =20 - logger.debug(3, "tasks:") + logger.debug3("tasks:") for tid in self.taskentries: - logger.debug(3, " %s: %s %s %s", + logger.debug3(" %s: %s %s %s", tid, self.taskentries[tid].idepends, self.taskentries[tid].irdepends, self.taskentries[tid].tdepends) =20 - logger.debug(3, "dependency ids (per fn):") + logger.debug3("dependency ids (per fn):") for fn in self.depids: - logger.debug(3, " %s: %s", fn, self.depids[fn]) + logger.debug3(" %s: %s", fn, self.depids[fn]) =20 - logger.debug(3, "runtime dependency ids (per fn):") + logger.debug3("runtime dependency ids (per fn):") for fn in self.rdepids: - logger.debug(3, " %s: %s", fn, self.rdepids[fn]) + logger.debug3(" %s: %s", fn, self.rdepids[fn]) diff --git a/bitbake/lib/bb/tests/codeparser.py b/bitbake/lib/bb/tests/code= parser.py index 826a2d2f..f4852047 100644 --- a/bitbake/lib/bb/tests/codeparser.py +++ b/bitbake/lib/bb/tests/codeparser.py @@ -111,9 +111,9 @@ ${D}${libdir}/pkgconfig/*.pc self.assertExecs(set(["sed"])) =20 def test_parameter_expansion_modifiers(self): - # - and + are also valid modifiers for parameter expansion, but are + # -,+ and : are also valid modifiers for parameter expansion, but = are # valid characters in bitbake variable names, so are not included = here - for i in ('=3D', ':-', ':=3D', '?', ':?', ':+', '#', '%', '##', '%= %'): + for i in ('=3D', '?', '#', '%', '##', '%%'): name =3D "foo%sbar" % i self.parseExpression("${%s}" % name) self.assertNotIn(name, self.references) diff --git a/bitbake/lib/bb/tests/color.py b/bitbake/lib/bb/tests/color.py new file mode 100644 index 00000000..88dd2780 --- /dev/null +++ b/bitbake/lib/bb/tests/color.py @@ -0,0 +1,95 @@ +# +# BitBake Test for ANSI color code filtering +# +# Copyright (C) 2020 Agilent Technologies, Inc. +# Author: Chris Laplante <chris.laplante@agilent.com> +# +# SPDX-License-Identifier: MIT +# + +import unittest +import bb.progress +import bb.data +import bb.event +from bb.progress import filter_color, filter_color_n +import io +import re + + +class ProgressWatcher: + def __init__(self): + self._reports =3D [] + + def handle_event(self, event): + self._reports.append((event.progress, event.rate)) + + def reports(self): + return self._reports + + +class ColorCodeTests(unittest.TestCase): + def setUp(self): + self.d =3D bb.data.init() + self._progress_watcher =3D ProgressWatcher() + bb.event.register("bb.build.TaskProgress", self._progress_watcher.= handle_event, data=3Dself.d) + + def tearDown(self): + bb.event.remove("bb.build.TaskProgress", None) + + def test_filter_color(self): + input_string =3D "=1B[01;35m=1B[K~~~~~~~~~~~~^~~~~~~~=1B[m=1B[K" + filtered =3D filter_color(input_string) + self.assertEqual(filtered, "~~~~~~~~~~~~^~~~~~~~") + + def test_filter_color_n(self): + input_string =3D "=1B[01;35m=1B[K~~~~~~~~~~~~^~~~~~~~=1B[m=1B[K" + filtered, code_count =3D filter_color_n(input_string) + self.assertEqual(filtered, "~~~~~~~~~~~~^~~~~~~~") + self.assertEqual(code_count, 4) + + def test_LineFilterProgressHandler_color_filtering(self): + class CustomProgressHandler(bb.progress.LineFilterProgressHandler): + PROGRESS_REGEX =3D re.compile(r"Progress: (?P<progress>\d+)%") + + def writeline(self, line): + match =3D self.PROGRESS_REGEX.match(line) + if match: + self.update(int(match.group("progress"))) + return False + return True + + buffer =3D io.StringIO() + handler =3D CustomProgressHandler(self.d, buffer) + handler.write("Program output!\n") + handler.write("More output!\n") + handler.write("Progress: =1B[01;35m=1B[K10=1B[m=1B[K%\n") # 10% + handler.write("Even more\n") + handler.write("=1B[01;35m=1B[KProgress: 50=1B[m=1B[K%\n") # 50% + handler.write("=1B[01;35m=1B[KProgress: 60=1B[m=1B[K%\n") # 60% + handler.write("Pro=1B[01;35m=1B[Kgress: =1B[m=1B[K100%\n") # 100% + + expected =3D [(10, None), (50, None), (60, None), (100, None)] + self.assertEqual(self._progress_watcher.reports(), expected) + + self.assertEqual(buffer.getvalue(), "Program output!\nMore output!= \nEven more\n") + + def test_BasicProgressHandler_color_filtering(self): + buffer =3D io.StringIO() + handler =3D bb.progress.BasicProgressHandler(self.d, outfile=3Dbuf= fer) + handler.write("=1B[01;35m=1B[K1=1B[m=1B[K%\n") # 1% + handler.write("=1B[01;35m=1B[K2=1B[m=1B[K%\n") # 2% + handler.write("=1B[01;35m=1B[K10=1B[m=1B[K%\n") # 10% + handler.write("=1B[01;35m=1B[K100=1B[m=1B[K%\n") # 100% + + expected =3D [(0, None), (1, None), (2, None), (10, None), (100, N= one)] + self.assertListEqual(self._progress_watcher.reports(), expected) + + def test_OutOfProgressHandler_color_filtering(self): + buffer =3D io.StringIO() + handler =3D bb.progress.OutOfProgressHandler(self.d, r'(\d+) of (\= d+)', outfile=3Dbuffer) + handler.write("=1B[01;35m=1B[KText text 1 of=1B[m=1B[K 5") # 1/5 + handler.write("=1B[01;35m=1B[KText text 3 of=1B[m=1B[K 5") # 3/5 + handler.write("=1B[01;35m=1B[KText text 5 of=1B[m=1B[K 5") # 5/5 + + expected =3D [(0, None), (20.0, None), (60.0, None), (100.0, None)] + self.assertListEqual(self._progress_watcher.reports(), expected) diff --git a/bitbake/lib/bb/tests/cooker.py b/bitbake/lib/bb/tests/cooker.py index 74c903f0..c82d4b7b 100644 --- a/bitbake/lib/bb/tests/cooker.py +++ b/bitbake/lib/bb/tests/cooker.py @@ -60,7 +60,7 @@ class CookerTest(unittest.TestCase): log_handler =3D LogHandler() logger.addHandler(log_handler) collection =3D bb.cooker.CookerCollectFiles(bbfile_config_prioriti= es) - collection.collection_priorities(pkgfns, self.d) + collection.collection_priorities(pkgfns, pkgfns, self.d) logger.removeHandler(log_handler) =20 # Should be empty (no generated messages) diff --git a/bitbake/lib/bb/tests/cow.py b/bitbake/lib/bb/tests/cow.py index bf6e79fc..75142649 100644 --- a/bitbake/lib/bb/tests/cow.py +++ b/bitbake/lib/bb/tests/cow.py @@ -4,9 +4,17 @@ # SPDX-License-Identifier: GPL-2.0-only # # Copyright 2006 Holger Freyther <freyther@handhelds.org> +# Copyright (C) 2020 Agilent Technologies, Inc. # =20 +import io +import re +import sys import unittest +import contextlib +import collections + +from bb.COW import COWDictBase, COWSetBase, COWDictMeta, COWSetMeta =20 =20 class COWTestCase(unittest.TestCase): @@ -14,11 +22,61 @@ class COWTestCase(unittest.TestCase): Test case for the COW module from mithro """ =20 + def setUp(self): + self._track_warnings =3D False + self._warning_file =3D io.StringIO() + self._unhandled_warnings =3D collections.deque() + COWDictBase.__warn__ =3D self._warning_file + + def tearDown(self): + COWDictBase.__warn__ =3D sys.stderr + if self._track_warnings: + self._checkAllWarningsRead() + + def trackWarnings(self): + self._track_warnings =3D True + + def _collectWarnings(self): + self._warning_file.seek(0) + for warning in self._warning_file: + self._unhandled_warnings.append(warning.rstrip("\n")) + self._warning_file.truncate(0) + self._warning_file.seek(0) + + def _checkAllWarningsRead(self): + self._collectWarnings() + self.assertSequenceEqual(self._unhandled_warnings, []) + + @contextlib.contextmanager + def checkReportsWarning(self, expected_warning): + self._checkAllWarningsRead() + yield + self._collectWarnings() + warning =3D self._unhandled_warnings.popleft() + self.assertEqual(warning, expected_warning) + + def checkStrOutput(self, obj, expected_levels, expected_keys): + if obj.__class__ is COWDictMeta: + expected_class_name =3D "COWDict" + elif obj.__class__ is COWSetMeta: + expected_class_name =3D "COWSet" + else: + self.fail("obj is of unknown type {0}".format(type(obj))) + s =3D str(obj) + regex =3D re.compile(r"<(\w+) Level: (\d+) Current Keys: (\d+)>") + match =3D regex.match(s) + self.assertIsNotNone(match, "bad str output: '{0}'".format(s)) + class_name =3D match.group(1) + self.assertEqual(class_name, expected_class_name) + levels =3D int(match.group(2)) + self.assertEqual(levels, expected_levels, "wrong # levels in str: = '{0}'".format(s)) + keys =3D int(match.group(3)) + self.assertEqual(keys, expected_keys, "wrong # keys in str: '{0}'"= .format(s)) + def testGetSet(self): """ Test and set """ - from bb.COW import COWDictBase a =3D COWDictBase.copy() =20 self.assertEqual(False, 'a' in a) @@ -27,16 +85,14 @@ class COWTestCase(unittest.TestCase): a['b'] =3D 'b' self.assertEqual(True, 'a' in a) self.assertEqual(True, 'b' in a) - self.assertEqual('a', a['a'] ) - self.assertEqual('b', a['b'] ) + self.assertEqual('a', a['a']) + self.assertEqual('b', a['b']) =20 def testCopyCopy(self): """ Test the copy of copies """ =20 - from bb.COW import COWDictBase - # create two COW dict 'instances' b =3D COWDictBase.copy() c =3D COWDictBase.copy() @@ -94,30 +150,168 @@ class COWTestCase(unittest.TestCase): self.assertEqual(False, 'e' in b_2) =20 def testCow(self): - from bb.COW import COWDictBase + self.trackWarnings() + c =3D COWDictBase.copy() c['123'] =3D 1027 c['other'] =3D 4711 - c['d'] =3D { 'abc' : 10, 'bcd' : 20 } + c['d'] =3D {'abc': 10, 'bcd': 20} =20 copy =3D c.copy() =20 self.assertEqual(1027, c['123']) self.assertEqual(4711, c['other']) - self.assertEqual({'abc':10, 'bcd':20}, c['d']) + self.assertEqual({'abc': 10, 'bcd': 20}, c['d']) self.assertEqual(1027, copy['123']) self.assertEqual(4711, copy['other']) - self.assertEqual({'abc':10, 'bcd':20}, copy['d']) + with self.checkReportsWarning("Warning: Doing a copy because d is = a mutable type."): + self.assertEqual({'abc': 10, 'bcd': 20}, copy['d']) =20 # cow it now copy['123'] =3D 1028 copy['other'] =3D 4712 copy['d']['abc'] =3D 20 =20 - self.assertEqual(1027, c['123']) self.assertEqual(4711, c['other']) - self.assertEqual({'abc':10, 'bcd':20}, c['d']) + self.assertEqual({'abc': 10, 'bcd': 20}, c['d']) self.assertEqual(1028, copy['123']) self.assertEqual(4712, copy['other']) - self.assertEqual({'abc':20, 'bcd':20}, copy['d']) + self.assertEqual({'abc': 20, 'bcd': 20}, copy['d']) + + def testOriginalTestSuite(self): + # This test suite is a port of the original one from COW.py + self.trackWarnings() + + a =3D COWDictBase.copy() + self.checkStrOutput(a, 1, 0) + + a['a'] =3D 'a' + a['b'] =3D 'b' + a['dict'] =3D {} + self.checkStrOutput(a, 1, 4) # 4th member is dict__mutable__ + + b =3D a.copy() + self.checkStrOutput(b, 2, 0) + b['c'] =3D 'b' + self.checkStrOutput(b, 2, 1) + + with self.checkReportsWarning("Warning: If you aren't going to cha= nge any of the values call with True."): + self.assertListEqual(list(a.iteritems()), + [('a', 'a'), + ('b', 'b'), + ('dict', {}) + ]) + + with self.checkReportsWarning("Warning: If you aren't going to cha= nge any of the values call with True."): + b_gen =3D b.iteritems() + self.assertTupleEqual(next(b_gen), ('a', 'a')) + self.assertTupleEqual(next(b_gen), ('b', 'b')) + self.assertTupleEqual(next(b_gen), ('c', 'b')) + with self.checkReportsWarning("Warning: Doing a copy because dict = is a mutable type."): + self.assertTupleEqual(next(b_gen), ('dict', {})) + with self.assertRaises(StopIteration): + next(b_gen) + + b['dict']['a'] =3D 'b' + b['a'] =3D 'c' + + self.checkStrOutput(a, 1, 4) + self.checkStrOutput(b, 2, 3) + + with self.checkReportsWarning("Warning: If you aren't going to cha= nge any of the values call with True."): + self.assertListEqual(list(a.iteritems()), + [('a', 'a'), + ('b', 'b'), + ('dict', {}) + ]) + + with self.checkReportsWarning("Warning: If you aren't going to cha= nge any of the values call with True."): + b_gen =3D b.iteritems() + self.assertTupleEqual(next(b_gen), ('a', 'c')) + self.assertTupleEqual(next(b_gen), ('b', 'b')) + self.assertTupleEqual(next(b_gen), ('c', 'b')) + self.assertTupleEqual(next(b_gen), ('dict', {'a': 'b'})) + with self.assertRaises(StopIteration): + next(b_gen) + + with self.assertRaises(KeyError): + print(b["dict2"]) + + a['set'] =3D COWSetBase() + a['set'].add("o1") + a['set'].add("o1") + a['set'].add("o2") + self.assertSetEqual(set(a['set'].itervalues()), {"o1", "o2"}) + self.assertSetEqual(set(b['set'].itervalues()), {"o1", "o2"}) + + b['set'].add('o3') + self.assertSetEqual(set(a['set'].itervalues()), {"o1", "o2"}) + self.assertSetEqual(set(b['set'].itervalues()), {"o1", "o2", "o3"}) + + a['set2'] =3D set() + a['set2'].add("o1") + a['set2'].add("o1") + a['set2'].add("o2") + + # We don't expect 'a' to change anymore + def check_a(): + with self.checkReportsWarning("Warning: If you aren't going to= change any of the values call with True."): + a_gen =3D a.iteritems() + self.assertTupleEqual(next(a_gen), ('a', 'a')) + self.assertTupleEqual(next(a_gen), ('b', 'b')) + self.assertTupleEqual(next(a_gen), ('dict', {})) + self.assertTupleEqual(next(a_gen), ('set2', {'o1', 'o2'})) + a_sub_set =3D next(a_gen) + self.assertEqual(a_sub_set[0], 'set') + self.checkStrOutput(a_sub_set[1], 1, 2) + self.assertSetEqual(set(a_sub_set[1].itervalues()), {'o1', 'o2= '}) + + check_a() + + b_gen =3D b.iteritems(readonly=3DTrue) + self.assertTupleEqual(next(b_gen), ('a', 'c')) + self.assertTupleEqual(next(b_gen), ('b', 'b')) + self.assertTupleEqual(next(b_gen), ('c', 'b')) + self.assertTupleEqual(next(b_gen), ('dict', {'a': 'b'})) + self.assertTupleEqual(next(b_gen), ('set2', {'o1', 'o2'})) + b_sub_set =3D next(b_gen) + self.assertEqual(b_sub_set[0], 'set') + self.checkStrOutput(b_sub_set[1], 2, 1) + self.assertSetEqual(set(b_sub_set[1].itervalues()), {'o1', 'o2', '= o3'}) + + del b['b'] + with self.assertRaises(KeyError): + print(b['b']) + self.assertFalse('b' in b) + + check_a() + + b.__revertitem__('b') + check_a() + self.assertEqual(b['b'], 'b') + self.assertTrue('b' in b) + + b.__revertitem__('dict') + check_a() + + b_gen =3D b.iteritems(readonly=3DTrue) + self.assertTupleEqual(next(b_gen), ('a', 'c')) + self.assertTupleEqual(next(b_gen), ('b', 'b')) + self.assertTupleEqual(next(b_gen), ('c', 'b')) + self.assertTupleEqual(next(b_gen), ('dict', {})) + self.assertTupleEqual(next(b_gen), ('set2', {'o1', 'o2'})) + b_sub_set =3D next(b_gen) + self.assertEqual(b_sub_set[0], 'set') + self.checkStrOutput(b_sub_set[1], 2, 1) + self.assertSetEqual(set(b_sub_set[1].itervalues()), {'o1', 'o2', '= o3'}) + + self.checkStrOutput(a, 1, 6) + self.checkStrOutput(b, 2, 3) + + def testSetMethods(self): + s =3D COWSetBase() + with self.assertRaises(TypeError): + print(s.iteritems()) + with self.assertRaises(TypeError): + print(s.iterkeys()) diff --git a/bitbake/lib/bb/tests/data.py b/bitbake/lib/bb/tests/data.py index 5f195047..1d4a64b1 100644 --- a/bitbake/lib/bb/tests/data.py +++ b/bitbake/lib/bb/tests/data.py @@ -12,6 +12,7 @@ import bb import bb.data import bb.parse import logging +import os =20 class LogRecord(): def __enter__(self): diff --git a/bitbake/lib/bb/tests/event.py b/bitbake/lib/bb/tests/event.py index 9229b63d..9ca7e9bc 100644 --- a/bitbake/lib/bb/tests/event.py +++ b/bitbake/lib/bb/tests/event.py @@ -6,17 +6,18 @@ # SPDX-License-Identifier: GPL-2.0-only # =20 -import unittest -import bb -import logging -import bb.compat -import bb.event +import collections import importlib +import logging +import pickle import threading import time -import pickle +import unittest from unittest.mock import Mock from unittest.mock import call + +import bb +import bb.event from bb.msg import BBLogFormatter =20 =20 @@ -75,7 +76,7 @@ class EventHandlingTest(unittest.TestCase): =20 def _create_test_handlers(self): """ Method used to create a test handler ordered dictionary """ - test_handlers =3D bb.compat.OrderedDict() + test_handlers =3D collections.OrderedDict() test_handlers["handler1"] =3D self._test_process.handler1 test_handlers["handler2"] =3D self._test_process.handler2 return test_handlers @@ -96,7 +97,7 @@ class EventHandlingTest(unittest.TestCase): =20 def test_clean_class_handlers(self): """ Test clean_class_handlers method """ - cleanDict =3D bb.compat.OrderedDict() + cleanDict =3D collections.OrderedDict() self.assertEqual(cleanDict, bb.event.clean_class_handlers()) =20 diff --git a/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux= /v2.23/index.html b/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/uti= l-linux/v2.23/index.html index a251c981..b3d9244b 100644 --- a/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux/v2.23/= index.html +++ b/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux/v2.23/= index.html @@ -1,45 +1,45 @@ -<html> -<head><title>Index of /pub/linux/utils/util-linux/v2.23/ - -

    Index of /pub/linux/utils/util-linux/v2.23/

    ../
-libblkid-docs/                             =
        31-Jul-2013 12:35       -
-libmount-docs/                             =
        31-Jul-2013 12:39       -
-sha256sums.asc                             =
        12-May-2017 10:53    2942
-util-linux-2.23-rc1.tar.bz2   =
                     22-Mar-2013 12:48      5M
-util-linux-2.23-rc1.tar.gz     =
                    22-Mar-2013 12:48      7M
-util-linux-2.23-rc1.tar.sign =
                      22-Mar-2013 12:48     836
-util-linux-2.23-rc1.tar.xz     =
                    22-Mar-2013 12:48      3M
-util-linux-2.23-rc2.tar.bz2   =
                     10-Apr-2013 22:14      5M
-util-linux-2.23-rc2.tar.gz     =
                    10-Apr-2013 22:14      7M
-util-linux-2.23-rc2.tar.sign =
                      10-Apr-2013 22:14     836
-util-linux-2.23-rc2.tar.xz     =
                    10-Apr-2013 22:14      3M
-util-linux-2.23.1.tar.bz2       =
                   28-May-2013 09:57      5M
-util-linux-2.23.1.tar.gz         =
                  28-May-2013 09:57      7M
-util-linux-2.23.1.tar.sign     =
                    28-May-2013 09:57     836
-util-linux-2.23.1.tar.xz         =
                  28-May-2013 09:57      3M
-util-linux-2.23.2.tar.bz2       =
                   31-Jul-2013 12:40      5M
-util-linux-2.23.2.tar.gz         =
                  31-Jul-2013 12:40      7M
-util-linux-2.23.2.tar.sign     =
                    31-Jul-2013 12:40     836
-util-linux-2.23.2.tar.xz         =
                  31-Jul-2013 12:40      3M
-util-linux-2.23.tar.bz2           =
                 25-Apr-2013 10:48      5M
-util-linux-2.23.tar.gz             =
                25-Apr-2013 10:48      7M
-util-linux-2.23.tar.sign         =
                  25-Apr-2013 10:48     836
-util-linux-2.23.tar.xz             =
                25-Apr-2013 10:48      3M
-v2.23-ChangeLog                           =
         25-Apr-2013 10:48     19K
-v2.23-ChangeLog.sign                 =
              25-Apr-2013 10:48     836
-v2.23-ReleaseNotes                     =
            25-Apr-2013 10:48     53K
-v2.23-ReleaseNotes.sign           =
                 25-Apr-2013 10:48     836
-v2.23-rc1-ChangeLog                   =
             22-Mar-2013 12:48    361K
-v2.23-rc1-ChangeLog.sign         =
                  22-Mar-2013 12:48     836
-v2.23-rc2-ChangeLog                   =
             10-Apr-2013 22:14     80K
-v2.23-rc2-ChangeLog.sign         =
                  10-Apr-2013 22:14     836
-v2.23.1-ChangeLog                       =
           28-May-2013 09:57     13K
-v2.23.1-ChangeLog.sign             =
                28-May-2013 09:57     836
-v2.23.1-ReleaseNotes                 =
              28-May-2013 09:58    1448
-v2.23.1-ReleaseNotes.sign       =
                   28-May-2013 09:58     836
-v2.23.2-ChangeLog                       =
           31-Jul-2013 12:40     23K
-v2.23.2-ChangeLog.sign             =
                31-Jul-2013 12:40     836
-v2.23.2-ReleaseNotes                 =
              31-Jul-2013 12:40    2582
-v2.23.2-ReleaseNotes.sign       =
                   31-Jul-2013 12:40     836
-
    - +=0D +Index of /pub/linux/utils/util-linux/v2.23/=0D +=0D +

    Index of /pub/linux/utils/util-linux/v2.23/

    ../=0D
+libblkid-docs/                             =
        31-Jul-2013 12:35       -=0D
+libmount-docs/                             =
        31-Jul-2013 12:39       -=0D
+sha256sums.asc                             =
        12-May-2017 10:53    2942=0D
+util-linux-2.23-rc1.tar.bz2   =
                     22-Mar-2013 12:48      5M=0D
+util-linux-2.23-rc1.tar.gz     =
                    22-Mar-2013 12:48      7M=0D
+util-linux-2.23-rc1.tar.sign =
                      22-Mar-2013 12:48     836=0D
+util-linux-2.23-rc1.tar.xz     =
                    22-Mar-2013 12:48      3M=0D
+util-linux-2.23-rc2.tar.bz2   =
                     10-Apr-2013 22:14      5M=0D
+util-linux-2.23-rc2.tar.gz     =
                    10-Apr-2013 22:14      7M=0D
+util-linux-2.23-rc2.tar.sign =
                      10-Apr-2013 22:14     836=0D
+util-linux-2.23-rc2.tar.xz     =
                    10-Apr-2013 22:14      3M=0D
+util-linux-2.23.1.tar.bz2       =
                   28-May-2013 09:57      5M=0D
+util-linux-2.23.1.tar.gz         =
                  28-May-2013 09:57      7M=0D
+util-linux-2.23.1.tar.sign     =
                    28-May-2013 09:57     836=0D
+util-linux-2.23.1.tar.xz         =
                  28-May-2013 09:57      3M=0D
+util-linux-2.23.2.tar.bz2       =
                   31-Jul-2013 12:40      5M=0D
+util-linux-2.23.2.tar.gz         =
                  31-Jul-2013 12:40      7M=0D
+util-linux-2.23.2.tar.sign     =
                    31-Jul-2013 12:40     836=0D
+util-linux-2.23.2.tar.xz         =
                  31-Jul-2013 12:40      3M=0D
+util-linux-2.23.tar.bz2           =
                 25-Apr-2013 10:48      5M=0D
+util-linux-2.23.tar.gz             =
                25-Apr-2013 10:48      7M=0D
+util-linux-2.23.tar.sign         =
                  25-Apr-2013 10:48     836=0D
+util-linux-2.23.tar.xz             =
                25-Apr-2013 10:48      3M=0D
+v2.23-ChangeLog                           =
         25-Apr-2013 10:48     19K=0D
+v2.23-ChangeLog.sign                 =
              25-Apr-2013 10:48     836=0D
+v2.23-ReleaseNotes                     =
            25-Apr-2013 10:48     53K=0D
+v2.23-ReleaseNotes.sign           =
                 25-Apr-2013 10:48     836=0D
+v2.23-rc1-ChangeLog                   =
             22-Mar-2013 12:48    361K=0D
+v2.23-rc1-ChangeLog.sign         =
                  22-Mar-2013 12:48     836=0D
+v2.23-rc2-ChangeLog                   =
             10-Apr-2013 22:14     80K=0D
+v2.23-rc2-ChangeLog.sign         =
                  10-Apr-2013 22:14     836=0D
+v2.23.1-ChangeLog                       =
           28-May-2013 09:57     13K=0D
+v2.23.1-ChangeLog.sign             =
                28-May-2013 09:57     836=0D
+v2.23.1-ReleaseNotes                 =
              28-May-2013 09:58    1448=0D
+v2.23.1-ReleaseNotes.sign       =
                   28-May-2013 09:58     836=0D
+v2.23.2-ChangeLog                       =
           31-Jul-2013 12:40     23K=0D
+v2.23.2-ChangeLog.sign             =
                31-Jul-2013 12:40     836=0D
+v2.23.2-ReleaseNotes                 =
              31-Jul-2013 12:40    2582=0D
+v2.23.2-ReleaseNotes.sign       =
                   31-Jul-2013 12:40     836=0D
+
    =0D +=0D diff --git a/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux= /v2.24/index.html b/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/uti= l-linux/v2.24/index.html index 486791d6..4afb4625 100644 --- a/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux/v2.24/= index.html +++ b/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux/v2.24/= index.html @@ -1,43 +1,43 @@ - -Index of /pub/linux/utils/util-linux/v2.24/ - -

    Index of /pub/linux/utils/util-linux/v2.24/

    ../
-libblkid-docs/                             =
        24-Apr-2014 10:15       -
-libmount-docs/                             =
        24-Apr-2014 10:17       -
-sha256sums.asc                             =
        12-May-2017 10:53    2758
-util-linux-2.24-rc1.tar.bz2   =
                     27-Sep-2013 12:54      5M
-util-linux-2.24-rc1.tar.gz     =
                    27-Sep-2013 12:54      7M
-util-linux-2.24-rc1.tar.sign =
                      27-Sep-2013 12:54     836
-util-linux-2.24-rc1.tar.xz     =
                    27-Sep-2013 12:54      3M
-util-linux-2.24-rc2.tar.bz2   =
                     11-Oct-2013 11:37      5M
-util-linux-2.24-rc2.tar.gz     =
                    11-Oct-2013 11:37      7M
-util-linux-2.24-rc2.tar.sign =
                      11-Oct-2013 11:37     836
-util-linux-2.24-rc2.tar.xz     =
                    11-Oct-2013 11:37      3M
-util-linux-2.24.1.tar.gz         =
                  20-Jan-2014 13:33      7M
-util-linux-2.24.1.tar.sign     =
                    20-Jan-2014 13:33     819
-util-linux-2.24.1.tar.xz         =
                  20-Jan-2014 13:33      3M
-util-linux-2.24.2.tar.gz         =
                  24-Apr-2014 10:17      7M
-util-linux-2.24.2.tar.sign     =
                    24-Apr-2014 10:17     819
-util-linux-2.24.2.tar.xz         =
                  24-Apr-2014 10:17      3M
-util-linux-2.24.tar.bz2           =
                 21-Oct-2013 13:49      5M
-util-linux-2.24.tar.gz             =
                21-Oct-2013 13:49      7M
-util-linux-2.24.tar.sign         =
                  21-Oct-2013 13:49     836
-util-linux-2.24.tar.xz             =
                21-Oct-2013 13:49      3M
-v2.24-ChangeLog                           =
         21-Oct-2013 13:49     22K
-v2.24-ChangeLog.sign                 =
              21-Oct-2013 13:49     836
-v2.24-ReleaseNotes                     =
            21-Oct-2013 13:49     44K
-v2.24-ReleaseNotes.sign           =
                 21-Oct-2013 13:49     836
-v2.24-rc1-ChangeLog                   =
             27-Sep-2013 12:54    292K
-v2.24-rc1-ChangeLog.sign         =
                  27-Sep-2013 12:54     836
-v2.24-rc2-ChangeLog                   =
             11-Oct-2013 11:37     42K
-v2.24-rc2-ChangeLog.sign         =
                  11-Oct-2013 11:37     836
-v2.24.1-ChangeLog                       =
           20-Jan-2014 13:33     38K
-v2.24.1-ChangeLog.sign             =
                20-Jan-2014 13:33     819
-v2.24.1-ReleaseNotes                 =
              20-Jan-2014 13:33    4449
-v2.24.1-ReleaseNotes.sign       =
                   20-Jan-2014 13:33     819
-v2.24.2-ChangeLog                       =
           24-Apr-2014 10:17     47K
-v2.24.2-ChangeLog.sign             =
                24-Apr-2014 10:17     819
-v2.24.2-ReleaseNotes                 =
              24-Apr-2014 10:18    5748
-v2.24.2-ReleaseNotes.sign       =
                   24-Apr-2014 10:18     819
-
    - +=0D +Index of /pub/linux/utils/util-linux/v2.24/=0D +=0D +

    Index of /pub/linux/utils/util-linux/v2.24/

    ../=0D
+libblkid-docs/                             =
        24-Apr-2014 10:15       -=0D
+libmount-docs/                             =
        24-Apr-2014 10:17       -=0D
+sha256sums.asc                             =
        12-May-2017 10:53    2758=0D
+util-linux-2.24-rc1.tar.bz2   =
                     27-Sep-2013 12:54      5M=0D
+util-linux-2.24-rc1.tar.gz     =
                    27-Sep-2013 12:54      7M=0D
+util-linux-2.24-rc1.tar.sign =
                      27-Sep-2013 12:54     836=0D
+util-linux-2.24-rc1.tar.xz     =
                    27-Sep-2013 12:54      3M=0D
+util-linux-2.24-rc2.tar.bz2   =
                     11-Oct-2013 11:37      5M=0D
+util-linux-2.24-rc2.tar.gz     =
                    11-Oct-2013 11:37      7M=0D
+util-linux-2.24-rc2.tar.sign =
                      11-Oct-2013 11:37     836=0D
+util-linux-2.24-rc2.tar.xz     =
                    11-Oct-2013 11:37      3M=0D
+util-linux-2.24.1.tar.gz         =
                  20-Jan-2014 13:33      7M=0D
+util-linux-2.24.1.tar.sign     =
                    20-Jan-2014 13:33     819=0D
+util-linux-2.24.1.tar.xz         =
                  20-Jan-2014 13:33      3M=0D
+util-linux-2.24.2.tar.gz         =
                  24-Apr-2014 10:17      7M=0D
+util-linux-2.24.2.tar.sign     =
                    24-Apr-2014 10:17     819=0D
+util-linux-2.24.2.tar.xz         =
                  24-Apr-2014 10:17      3M=0D
+util-linux-2.24.tar.bz2           =
                 21-Oct-2013 13:49      5M=0D
+util-linux-2.24.tar.gz             =
                21-Oct-2013 13:49      7M=0D
+util-linux-2.24.tar.sign         =
                  21-Oct-2013 13:49     836=0D
+util-linux-2.24.tar.xz             =
                21-Oct-2013 13:49      3M=0D
+v2.24-ChangeLog                           =
         21-Oct-2013 13:49     22K=0D
+v2.24-ChangeLog.sign                 =
              21-Oct-2013 13:49     836=0D
+v2.24-ReleaseNotes                     =
            21-Oct-2013 13:49     44K=0D
+v2.24-ReleaseNotes.sign           =
                 21-Oct-2013 13:49     836=0D
+v2.24-rc1-ChangeLog                   =
             27-Sep-2013 12:54    292K=0D
+v2.24-rc1-ChangeLog.sign         =
                  27-Sep-2013 12:54     836=0D
+v2.24-rc2-ChangeLog                   =
             11-Oct-2013 11:37     42K=0D
+v2.24-rc2-ChangeLog.sign         =
                  11-Oct-2013 11:37     836=0D
+v2.24.1-ChangeLog                       =
           20-Jan-2014 13:33     38K=0D
+v2.24.1-ChangeLog.sign             =
                20-Jan-2014 13:33     819=0D
+v2.24.1-ReleaseNotes                 =
              20-Jan-2014 13:33    4449=0D
+v2.24.1-ReleaseNotes.sign       =
                   20-Jan-2014 13:33     819=0D
+v2.24.2-ChangeLog                       =
           24-Apr-2014 10:17     47K=0D
+v2.24.2-ChangeLog.sign             =
                24-Apr-2014 10:17     819=0D
+v2.24.2-ReleaseNotes                 =
              24-Apr-2014 10:18    5748=0D
+v2.24.2-ReleaseNotes.sign       =
                   24-Apr-2014 10:18     819=0D
+
    =0D +=0D diff --git a/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux= /v2.25/index.html b/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/uti= l-linux/v2.25/index.html index b3fdadc8..9516c3b1 100644 --- a/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux/v2.25/= index.html +++ b/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux/v2.25/= index.html @@ -1,46 +1,46 @@ - -Index of /pub/linux/utils/util-linux/v2.25/ - -

    Index of /pub/linux/utils/util-linux/v2.25/

    ../
-libblkid-docs/                             =
        24-Oct-2014 13:05       -
-libmount-docs/                             =
        24-Oct-2014 13:06       -
-libsmartcols-docs/                     =
            24-Oct-2014 13:08       -
-sha256sums.asc                             =
        12-May-2017 10:53    2758
-util-linux-2.25-rc1.tar.gz     =
                    18-Jun-2014 13:33      7M
-util-linux-2.25-rc1.tar.sign =
                      18-Jun-2014 13:33     819
-util-linux-2.25-rc1.tar.xz     =
                    18-Jun-2014 13:33      3M
-util-linux-2.25-rc2.tar.gz     =
                    02-Jul-2014 10:02      7M
-util-linux-2.25-rc2.tar.sign =
                      02-Jul-2014 10:02     819
-util-linux-2.25-rc2.tar.xz     =
                    02-Jul-2014 10:02      3M
-util-linux-2.25.1-rc1.tar.gz =
                      27-Aug-2014 13:18      8M
-util-linux-2.25.1-rc1.tar.sign<=
/a>                     27-Aug-2014 13:18     819
-util-linux-2.25.1-rc1.tar.xz =
                      27-Aug-2014 13:18      4M
-util-linux-2.25.1.tar.gz         =
                  03-Sep-2014 10:41      8M
-util-linux-2.25.1.tar.sign     =
                    03-Sep-2014 10:41     819
-util-linux-2.25.1.tar.xz         =
                  03-Sep-2014 10:41      4M
-util-linux-2.25.2.tar.gz         =
                  24-Oct-2014 13:08      8M
-util-linux-2.25.2.tar.sign     =
                    24-Oct-2014 13:08     819
-util-linux-2.25.2.tar.xz         =
                  24-Oct-2014 13:08      4M
-util-linux-2.25.tar.gz             =
                22-Jul-2014 09:50      8M
-util-linux-2.25.tar.sign         =
                  22-Jul-2014 09:50     819
-util-linux-2.25.tar.xz             =
                22-Jul-2014 09:50      4M
-v2.25-ChangeLog                           =
         22-Jul-2014 09:50     41K
-v2.25-ChangeLog.sign                 =
              22-Jul-2014 09:50     819
-v2.25-ReleaseNotes                     =
            22-Jul-2014 09:50     61K
-v2.25-ReleaseNotes.sign           =
                 22-Jul-2014 09:50     819
-v2.25-rc1-ChangeLog                   =
             18-Jun-2014 13:33    489K
-v2.25-rc1-ChangeLog.sign         =
                  18-Jun-2014 13:33     819
-v2.25-rc2-ChangeLog                   =
             02-Jul-2014 10:02     27K
-v2.25-rc2-ChangeLog.sign         =
                  02-Jul-2014 10:02     819
-v2.25.1-ChangeLog                       =
           03-Sep-2014 10:41    5816
-v2.25.1-ChangeLog.sign             =
                03-Sep-2014 10:41     819
-v2.25.1-ReleaseNotes                 =
              03-Sep-2014 10:52    3220
-v2.25.1-ReleaseNotes.sign       =
                   03-Sep-2014 10:52     819
-v2.25.1-rc1-ChangeLog               =
               27-Aug-2014 13:18     22K
-v2.25.1-rc1-ChangeLog.sign     =
                    27-Aug-2014 13:18     819
-v2.25.2-ChangeLog                       =
           24-Oct-2014 13:08     26K
-v2.25.2-ChangeLog.sign             =
                24-Oct-2014 13:08     819
-v2.25.2-ReleaseNotes                 =
              24-Oct-2014 13:08    3016
-v2.25.2-ReleaseNotes.sign       =
                   24-Oct-2014 13:08     819
-
    - +=0D +Index of /pub/linux/utils/util-linux/v2.25/=0D +=0D +

    Index of /pub/linux/utils/util-linux/v2.25/

    ../=0D
+libblkid-docs/                             =
        24-Oct-2014 13:05       -=0D
+libmount-docs/                             =
        24-Oct-2014 13:06       -=0D
+libsmartcols-docs/                     =
            24-Oct-2014 13:08       -=0D
+sha256sums.asc                             =
        12-May-2017 10:53    2758=0D
+util-linux-2.25-rc1.tar.gz     =
                    18-Jun-2014 13:33      7M=0D
+util-linux-2.25-rc1.tar.sign =
                      18-Jun-2014 13:33     819=0D
+util-linux-2.25-rc1.tar.xz     =
                    18-Jun-2014 13:33      3M=0D
+util-linux-2.25-rc2.tar.gz     =
                    02-Jul-2014 10:02      7M=0D
+util-linux-2.25-rc2.tar.sign =
                      02-Jul-2014 10:02     819=0D
+util-linux-2.25-rc2.tar.xz     =
                    02-Jul-2014 10:02      3M=0D
+util-linux-2.25.1-rc1.tar.gz =
                      27-Aug-2014 13:18      8M=0D
+util-linux-2.25.1-rc1.tar.sign<=
/a>                     27-Aug-2014 13:18     819=0D
+util-linux-2.25.1-rc1.tar.xz =
                      27-Aug-2014 13:18      4M=0D
+util-linux-2.25.1.tar.gz         =
                  03-Sep-2014 10:41      8M=0D
+util-linux-2.25.1.tar.sign     =
                    03-Sep-2014 10:41     819=0D
+util-linux-2.25.1.tar.xz         =
                  03-Sep-2014 10:41      4M=0D
+util-linux-2.25.2.tar.gz         =
                  24-Oct-2014 13:08      8M=0D
+util-linux-2.25.2.tar.sign     =
                    24-Oct-2014 13:08     819=0D
+util-linux-2.25.2.tar.xz         =
                  24-Oct-2014 13:08      4M=0D
+util-linux-2.25.tar.gz             =
                22-Jul-2014 09:50      8M=0D
+util-linux-2.25.tar.sign         =
                  22-Jul-2014 09:50     819=0D
+util-linux-2.25.tar.xz             =
                22-Jul-2014 09:50      4M=0D
+v2.25-ChangeLog                           =
         22-Jul-2014 09:50     41K=0D
+v2.25-ChangeLog.sign                 =
              22-Jul-2014 09:50     819=0D
+v2.25-ReleaseNotes                     =
            22-Jul-2014 09:50     61K=0D
+v2.25-ReleaseNotes.sign           =
                 22-Jul-2014 09:50     819=0D
+v2.25-rc1-ChangeLog                   =
             18-Jun-2014 13:33    489K=0D
+v2.25-rc1-ChangeLog.sign         =
                  18-Jun-2014 13:33     819=0D
+v2.25-rc2-ChangeLog                   =
             02-Jul-2014 10:02     27K=0D
+v2.25-rc2-ChangeLog.sign         =
                  02-Jul-2014 10:02     819=0D
+v2.25.1-ChangeLog                       =
           03-Sep-2014 10:41    5816=0D
+v2.25.1-ChangeLog.sign             =
                03-Sep-2014 10:41     819=0D
+v2.25.1-ReleaseNotes                 =
              03-Sep-2014 10:52    3220=0D
+v2.25.1-ReleaseNotes.sign       =
                   03-Sep-2014 10:52     819=0D
+v2.25.1-rc1-ChangeLog               =
               27-Aug-2014 13:18     22K=0D
+v2.25.1-rc1-ChangeLog.sign     =
                    27-Aug-2014 13:18     819=0D
+v2.25.2-ChangeLog                       =
           24-Oct-2014 13:08     26K=0D
+v2.25.2-ChangeLog.sign             =
                24-Oct-2014 13:08     819=0D
+v2.25.2-ReleaseNotes                 =
              24-Oct-2014 13:08    3016=0D
+v2.25.2-ReleaseNotes.sign       =
                   24-Oct-2014 13:08     819=0D
+
    =0D +=0D diff --git a/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux= /v2.26/index.html b/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/uti= l-linux/v2.26/index.html index cd15132e..b9914895 100644 --- a/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux/v2.26/= index.html +++ b/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux/v2.26/= index.html @@ -1,42 +1,42 @@ - -Index of /pub/linux/utils/util-linux/v2.26/ - -

    Index of /pub/linux/utils/util-linux/v2.26/

    ../
-libblkid-docs/                             =
        30-Apr-2015 10:38       -
-libfdisk-docs/                             =
        30-Apr-2015 10:40       -
-libmount-docs/                             =
        30-Apr-2015 10:42       -
-libsmartcols-docs/                     =
            30-Apr-2015 10:43       -
-sha256sums.asc                             =
        12-May-2017 10:54    2480
-util-linux-2.26-rc1.tar.gz     =
                    14-Jan-2015 13:14      8M
-util-linux-2.26-rc1.tar.sign =
                      14-Jan-2015 13:14     819
-util-linux-2.26-rc1.tar.xz     =
                    14-Jan-2015 13:14      4M
-util-linux-2.26-rc2.tar.gz     =
                    04-Feb-2015 11:49      8M
-util-linux-2.26-rc2.tar.sign =
                      04-Feb-2015 11:49     819
-util-linux-2.26-rc2.tar.xz     =
                    04-Feb-2015 11:49      4M
-util-linux-2.26.1.tar.gz         =
                  13-Mar-2015 14:23      8M
-util-linux-2.26.1.tar.sign     =
                    13-Mar-2015 14:23     819
-util-linux-2.26.1.tar.xz         =
                  13-Mar-2015 14:23      4M
-util-linux-2.26.2.tar.gz         =
                  30-Apr-2015 10:44      8M
-util-linux-2.26.2.tar.sign     =
                    30-Apr-2015 10:44     819
-util-linux-2.26.2.tar.xz         =
                  30-Apr-2015 10:44      4M
-util-linux-2.26.tar.gz             =
                19-Feb-2015 12:47      8M
-util-linux-2.26.tar.sign         =
                  19-Feb-2015 12:47     819
-util-linux-2.26.tar.xz             =
                19-Feb-2015 12:47      4M
-v2.26-ChangeLog                           =
         19-Feb-2015 12:47     30K
-v2.26-ChangeLog.sign                 =
              19-Feb-2015 12:47     819
-v2.26-ReleaseNotes                     =
            19-Feb-2015 12:47     51K
-v2.26-ReleaseNotes.sign           =
                 19-Feb-2015 12:47     819
-v2.26-rc1-ChangeLog                   =
             14-Jan-2015 13:14    360K
-v2.26-rc1-ChangeLog.sign         =
                  14-Jan-2015 13:14     819
-v2.26-rc2-ChangeLog                   =
             04-Feb-2015 11:50     51K
-v2.26-rc2-ChangeLog.sign         =
                  04-Feb-2015 11:50     819
-v2.26.1-ChangeLog                       =
           13-Mar-2015 14:23     32K
-v2.26.1-ChangeLog.sign             =
                13-Mar-2015 14:23     819
-v2.26.1-ReleaseNotes                 =
              13-Mar-2015 14:23    2944
-v2.26.1-ReleaseNotes.sign       =
                   13-Mar-2015 14:23     819
-v2.26.2-ChangeLog                       =
           30-Apr-2015 10:44     58K
-v2.26.2-ChangeLog.sign             =
                30-Apr-2015 10:44     819
-v2.26.2-ReleaseNotes                 =
              30-Apr-2015 10:44    5834
-v2.26.2-ReleaseNotes.sign       =
                   30-Apr-2015 10:44     819
-
    - +=0D +Index of /pub/linux/utils/util-linux/v2.26/=0D +=0D +

    Index of /pub/linux/utils/util-linux/v2.26/

    ../=0D
+libblkid-docs/                             =
        30-Apr-2015 10:38       -=0D
+libfdisk-docs/                             =
        30-Apr-2015 10:40       -=0D
+libmount-docs/                             =
        30-Apr-2015 10:42       -=0D
+libsmartcols-docs/                     =
            30-Apr-2015 10:43       -=0D
+sha256sums.asc                             =
        12-May-2017 10:54    2480=0D
+util-linux-2.26-rc1.tar.gz     =
                    14-Jan-2015 13:14      8M=0D
+util-linux-2.26-rc1.tar.sign =
                      14-Jan-2015 13:14     819=0D
+util-linux-2.26-rc1.tar.xz     =
                    14-Jan-2015 13:14      4M=0D
+util-linux-2.26-rc2.tar.gz     =
                    04-Feb-2015 11:49      8M=0D
+util-linux-2.26-rc2.tar.sign =
                      04-Feb-2015 11:49     819=0D
+util-linux-2.26-rc2.tar.xz     =
                    04-Feb-2015 11:49      4M=0D
+util-linux-2.26.1.tar.gz         =
                  13-Mar-2015 14:23      8M=0D
+util-linux-2.26.1.tar.sign     =
                    13-Mar-2015 14:23     819=0D
+util-linux-2.26.1.tar.xz         =
                  13-Mar-2015 14:23      4M=0D
+util-linux-2.26.2.tar.gz         =
                  30-Apr-2015 10:44      8M=0D
+util-linux-2.26.2.tar.sign     =
                    30-Apr-2015 10:44     819=0D
+util-linux-2.26.2.tar.xz         =
                  30-Apr-2015 10:44      4M=0D
+util-linux-2.26.tar.gz             =
                19-Feb-2015 12:47      8M=0D
+util-linux-2.26.tar.sign         =
                  19-Feb-2015 12:47     819=0D
+util-linux-2.26.tar.xz             =
                19-Feb-2015 12:47      4M=0D
+v2.26-ChangeLog                           =
         19-Feb-2015 12:47     30K=0D
+v2.26-ChangeLog.sign                 =
              19-Feb-2015 12:47     819=0D
+v2.26-ReleaseNotes                     =
            19-Feb-2015 12:47     51K=0D
+v2.26-ReleaseNotes.sign           =
                 19-Feb-2015 12:47     819=0D
+v2.26-rc1-ChangeLog                   =
             14-Jan-2015 13:14    360K=0D
+v2.26-rc1-ChangeLog.sign         =
                  14-Jan-2015 13:14     819=0D
+v2.26-rc2-ChangeLog                   =
             04-Feb-2015 11:50     51K=0D
+v2.26-rc2-ChangeLog.sign         =
                  04-Feb-2015 11:50     819=0D
+v2.26.1-ChangeLog                       =
           13-Mar-2015 14:23     32K=0D
+v2.26.1-ChangeLog.sign             =
                13-Mar-2015 14:23     819=0D
+v2.26.1-ReleaseNotes                 =
              13-Mar-2015 14:23    2944=0D
+v2.26.1-ReleaseNotes.sign       =
                   13-Mar-2015 14:23     819=0D
+v2.26.2-ChangeLog                       =
           30-Apr-2015 10:44     58K=0D
+v2.26.2-ChangeLog.sign             =
                30-Apr-2015 10:44     819=0D
+v2.26.2-ReleaseNotes                 =
              30-Apr-2015 10:44    5834=0D
+v2.26.2-ReleaseNotes.sign       =
                   30-Apr-2015 10:44     819=0D
+
    =0D +=0D diff --git a/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux= /v2.27/index.html b/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/uti= l-linux/v2.27/index.html index b7b21606..14eb3683 100644 --- a/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux/v2.27/= index.html +++ b/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux/v2.27/= index.html @@ -1,35 +1,35 @@ - -Index of /pub/linux/utils/util-linux/v2.27/ - -

    Index of /pub/linux/utils/util-linux/v2.27/

    ../
-libblkid-docs/                             =
        02-Nov-2015 11:01       -
-libfdisk-docs/                             =
        02-Nov-2015 11:03       -
-libmount-docs/                             =
        02-Nov-2015 11:04       -
-libsmartcols-docs/                     =
            02-Nov-2015 11:06       -
-sha256sums.asc                             =
        12-May-2017 10:54    2127
-util-linux-2.27-rc1.tar.gz     =
                    31-Jul-2015 11:01      8M
-util-linux-2.27-rc1.tar.sign =
                      31-Jul-2015 11:01     819
-util-linux-2.27-rc1.tar.xz     =
                    31-Jul-2015 11:01      4M
-util-linux-2.27-rc2.tar.gz     =
                    24-Aug-2015 11:04      8M
-util-linux-2.27-rc2.tar.sign =
                      24-Aug-2015 11:04     819
-util-linux-2.27-rc2.tar.xz     =
                    24-Aug-2015 11:04      4M
-util-linux-2.27.1.tar.gz         =
                  02-Nov-2015 11:06      8M
-util-linux-2.27.1.tar.sign     =
                    02-Nov-2015 11:06     819
-util-linux-2.27.1.tar.xz         =
                  02-Nov-2015 11:06      4M
-util-linux-2.27.tar.gz             =
                07-Sep-2015 08:17      8M
-util-linux-2.27.tar.sign         =
                  07-Sep-2015 08:17     819
-util-linux-2.27.tar.xz             =
                07-Sep-2015 08:17      4M
-v2.27-ChangeLog                           =
         07-Sep-2015 08:17     21K
-v2.27-ChangeLog.sign                 =
              07-Sep-2015 08:17     819
-v2.27-ReleaseNotes                     =
            07-Sep-2015 08:17     35K
-v2.27-ReleaseNotes.sign           =
                 07-Sep-2015 08:17     819
-v2.27-rc1-ChangeLog                   =
             31-Jul-2015 11:01    263K
-v2.27-rc1-ChangeLog.sign         =
                  31-Jul-2015 11:01     819
-v2.27-rc2-ChangeLog                   =
             24-Aug-2015 11:04     38K
-v2.27-rc2-ChangeLog.sign         =
                  24-Aug-2015 11:04     819
-v2.27.1-ChangeLog                       =
           02-Nov-2015 11:06     18K
-v2.27.1-ChangeLog.sign             =
                02-Nov-2015 11:06     819
-v2.27.1-ReleaseNotes                 =
              02-Nov-2015 11:06    2107
-v2.27.1-ReleaseNotes.sign       =
                   02-Nov-2015 11:06     819
-
    - +=0D +Index of /pub/linux/utils/util-linux/v2.27/=0D +=0D +

    Index of /pub/linux/utils/util-linux/v2.27/

    ../=0D
+libblkid-docs/                             =
        02-Nov-2015 11:01       -=0D
+libfdisk-docs/                             =
        02-Nov-2015 11:03       -=0D
+libmount-docs/                             =
        02-Nov-2015 11:04       -=0D
+libsmartcols-docs/                     =
            02-Nov-2015 11:06       -=0D
+sha256sums.asc                             =
        12-May-2017 10:54    2127=0D
+util-linux-2.27-rc1.tar.gz     =
                    31-Jul-2015 11:01      8M=0D
+util-linux-2.27-rc1.tar.sign =
                      31-Jul-2015 11:01     819=0D
+util-linux-2.27-rc1.tar.xz     =
                    31-Jul-2015 11:01      4M=0D
+util-linux-2.27-rc2.tar.gz     =
                    24-Aug-2015 11:04      8M=0D
+util-linux-2.27-rc2.tar.sign =
                      24-Aug-2015 11:04     819=0D
+util-linux-2.27-rc2.tar.xz     =
                    24-Aug-2015 11:04      4M=0D
+util-linux-2.27.1.tar.gz         =
                  02-Nov-2015 11:06      8M=0D
+util-linux-2.27.1.tar.sign     =
                    02-Nov-2015 11:06     819=0D
+util-linux-2.27.1.tar.xz         =
                  02-Nov-2015 11:06      4M=0D
+util-linux-2.27.tar.gz             =
                07-Sep-2015 08:17      8M=0D
+util-linux-2.27.tar.sign         =
                  07-Sep-2015 08:17     819=0D
+util-linux-2.27.tar.xz             =
                07-Sep-2015 08:17      4M=0D
+v2.27-ChangeLog                           =
         07-Sep-2015 08:17     21K=0D
+v2.27-ChangeLog.sign                 =
              07-Sep-2015 08:17     819=0D
+v2.27-ReleaseNotes                     =
            07-Sep-2015 08:17     35K=0D
+v2.27-ReleaseNotes.sign           =
                 07-Sep-2015 08:17     819=0D
+v2.27-rc1-ChangeLog                   =
             31-Jul-2015 11:01    263K=0D
+v2.27-rc1-ChangeLog.sign         =
                  31-Jul-2015 11:01     819=0D
+v2.27-rc2-ChangeLog                   =
             24-Aug-2015 11:04     38K=0D
+v2.27-rc2-ChangeLog.sign         =
                  24-Aug-2015 11:04     819=0D
+v2.27.1-ChangeLog                       =
           02-Nov-2015 11:06     18K=0D
+v2.27.1-ChangeLog.sign             =
                02-Nov-2015 11:06     819=0D
+v2.27.1-ReleaseNotes                 =
              02-Nov-2015 11:06    2107=0D
+v2.27.1-ReleaseNotes.sign       =
                   02-Nov-2015 11:06     819=0D
+
    =0D +=0D diff --git a/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux= /v2.28/index.html b/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/uti= l-linux/v2.28/index.html index f2370edf..4bba6b47 100644 --- a/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux/v2.28/= index.html +++ b/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux/v2.28/= index.html @@ -1,42 +1,42 @@ - -Index of /pub/linux/utils/util-linux/v2.28/ - -

    Index of /pub/linux/utils/util-linux/v2.28/

    ../
-libblkid-docs/                             =
        07-Sep-2016 12:00       -
-libfdisk-docs/                             =
        07-Sep-2016 12:02       -
-libmount-docs/                             =
        07-Sep-2016 12:04       -
-libsmartcols-docs/                     =
            07-Sep-2016 12:06       -
-sha256sums.asc                             =
        12-May-2017 10:55    2480
-util-linux-2.28-rc1.tar.gz     =
                    11-Mar-2016 11:45      8M
-util-linux-2.28-rc1.tar.sign =
                      11-Mar-2016 11:45     819
-util-linux-2.28-rc1.tar.xz     =
                    11-Mar-2016 11:45      4M
-util-linux-2.28-rc2.tar.gz     =
                    29-Mar-2016 09:04      8M
-util-linux-2.28-rc2.tar.sign =
                      29-Mar-2016 09:04     819
-util-linux-2.28-rc2.tar.xz     =
                    29-Mar-2016 09:04      4M
-util-linux-2.28.1.tar.gz         =
                  11-Aug-2016 10:09      9M
-util-linux-2.28.1.tar.sign     =
                    11-Aug-2016 10:09     819
-util-linux-2.28.1.tar.xz         =
                  11-Aug-2016 10:09      4M
-util-linux-2.28.2.tar.gz         =
                  07-Sep-2016 12:06      9M
-util-linux-2.28.2.tar.sign     =
                    07-Sep-2016 12:06     819
-util-linux-2.28.2.tar.xz         =
                  07-Sep-2016 12:06      4M
-util-linux-2.28.tar.gz             =
                12-Apr-2016 11:26      8M
-util-linux-2.28.tar.sign         =
                  12-Apr-2016 11:26     819
-util-linux-2.28.tar.xz             =
                12-Apr-2016 11:26      4M
-v2.28-ChangeLog                           =
         12-Apr-2016 11:26     13K
-v2.28-ChangeLog.sign                 =
              12-Apr-2016 11:26     819
-v2.28-ReleaseNotes                     =
            12-Apr-2016 11:26     33K
-v2.28-ReleaseNotes.sign           =
                 12-Apr-2016 11:26     819
-v2.28-rc1-ChangeLog                   =
             11-Mar-2016 11:45    269K
-v2.28-rc1-ChangeLog.sign         =
                  11-Mar-2016 11:45     819
-v2.28-rc2-ChangeLog                   =
             29-Mar-2016 09:04     52K
-v2.28-rc2-ChangeLog.sign         =
                  29-Mar-2016 09:04     819
-v2.28.1-ChangeLog                       =
           11-Aug-2016 10:09     37K
-v2.28.1-ChangeLog.sign             =
                11-Aug-2016 10:09     819
-v2.28.1-ReleaseNotes                 =
              11-Aug-2016 10:09    3748
-v2.28.1-ReleaseNotes.sign       =
                   11-Aug-2016 10:09     819
-v2.28.2-ChangeLog                       =
           07-Sep-2016 12:06    8900
-v2.28.2-ChangeLog.sign             =
                07-Sep-2016 12:06     819
-v2.28.2-ReleaseNotes                 =
              07-Sep-2016 12:06    1161
-v2.28.2-ReleaseNotes.sign       =
                   07-Sep-2016 12:06     819
-
    - +=0D +Index of /pub/linux/utils/util-linux/v2.28/=0D +=0D +

    Index of /pub/linux/utils/util-linux/v2.28/

    ../=0D
+libblkid-docs/                             =
        07-Sep-2016 12:00       -=0D
+libfdisk-docs/                             =
        07-Sep-2016 12:02       -=0D
+libmount-docs/                             =
        07-Sep-2016 12:04       -=0D
+libsmartcols-docs/                     =
            07-Sep-2016 12:06       -=0D
+sha256sums.asc                             =
        12-May-2017 10:55    2480=0D
+util-linux-2.28-rc1.tar.gz     =
                    11-Mar-2016 11:45      8M=0D
+util-linux-2.28-rc1.tar.sign =
                      11-Mar-2016 11:45     819=0D
+util-linux-2.28-rc1.tar.xz     =
                    11-Mar-2016 11:45      4M=0D
+util-linux-2.28-rc2.tar.gz     =
                    29-Mar-2016 09:04      8M=0D
+util-linux-2.28-rc2.tar.sign =
                      29-Mar-2016 09:04     819=0D
+util-linux-2.28-rc2.tar.xz     =
                    29-Mar-2016 09:04      4M=0D
+util-linux-2.28.1.tar.gz         =
                  11-Aug-2016 10:09      9M=0D
+util-linux-2.28.1.tar.sign     =
                    11-Aug-2016 10:09     819=0D
+util-linux-2.28.1.tar.xz         =
                  11-Aug-2016 10:09      4M=0D
+util-linux-2.28.2.tar.gz         =
                  07-Sep-2016 12:06      9M=0D
+util-linux-2.28.2.tar.sign     =
                    07-Sep-2016 12:06     819=0D
+util-linux-2.28.2.tar.xz         =
                  07-Sep-2016 12:06      4M=0D
+util-linux-2.28.tar.gz             =
                12-Apr-2016 11:26      8M=0D
+util-linux-2.28.tar.sign         =
                  12-Apr-2016 11:26     819=0D
+util-linux-2.28.tar.xz             =
                12-Apr-2016 11:26      4M=0D
+v2.28-ChangeLog                           =
         12-Apr-2016 11:26     13K=0D
+v2.28-ChangeLog.sign                 =
              12-Apr-2016 11:26     819=0D
+v2.28-ReleaseNotes                     =
            12-Apr-2016 11:26     33K=0D
+v2.28-ReleaseNotes.sign           =
                 12-Apr-2016 11:26     819=0D
+v2.28-rc1-ChangeLog                   =
             11-Mar-2016 11:45    269K=0D
+v2.28-rc1-ChangeLog.sign         =
                  11-Mar-2016 11:45     819=0D
+v2.28-rc2-ChangeLog                   =
             29-Mar-2016 09:04     52K=0D
+v2.28-rc2-ChangeLog.sign         =
                  29-Mar-2016 09:04     819=0D
+v2.28.1-ChangeLog                       =
           11-Aug-2016 10:09     37K=0D
+v2.28.1-ChangeLog.sign             =
                11-Aug-2016 10:09     819=0D
+v2.28.1-ReleaseNotes                 =
              11-Aug-2016 10:09    3748=0D
+v2.28.1-ReleaseNotes.sign       =
                   11-Aug-2016 10:09     819=0D
+v2.28.2-ChangeLog                       =
           07-Sep-2016 12:06    8900=0D
+v2.28.2-ChangeLog.sign             =
                07-Sep-2016 12:06     819=0D
+v2.28.2-ReleaseNotes                 =
              07-Sep-2016 12:06    1161=0D
+v2.28.2-ReleaseNotes.sign       =
                   07-Sep-2016 12:06     819=0D
+
    =0D +=0D diff --git a/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux= /v2.29/index.html b/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/uti= l-linux/v2.29/index.html index 07a8e51c..916a255f 100644 --- a/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux/v2.29/= index.html +++ b/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux/v2.29/= index.html @@ -1,42 +1,42 @@ - -Index of /pub/linux/utils/util-linux/v2.29/ - -

    Index of /pub/linux/utils/util-linux/v2.29/

    ../
-libblkid-docs/                             =
        22-Feb-2017 15:20       -
-libfdisk-docs/                             =
        22-Feb-2017 15:22       -
-libmount-docs/                             =
        22-Feb-2017 15:24       -
-libsmartcols-docs/                     =
            22-Feb-2017 15:26       -
-sha256sums.asc                             =
        12-May-2017 10:55    2480
-util-linux-2.29-rc1.tar.gz     =
                    30-Sep-2016 09:55      9M
-util-linux-2.29-rc1.tar.sign =
                      30-Sep-2016 09:55     819
-util-linux-2.29-rc1.tar.xz     =
                    30-Sep-2016 09:55      4M
-util-linux-2.29-rc2.tar.gz     =
                    19-Oct-2016 13:13      9M
-util-linux-2.29-rc2.tar.sign =
                      19-Oct-2016 13:13     819
-util-linux-2.29-rc2.tar.xz     =
                    19-Oct-2016 13:13      4M
-util-linux-2.29.1.tar.gz         =
                  20-Jan-2017 14:02      9M
-util-linux-2.29.1.tar.sign     =
                    20-Jan-2017 14:02     819
-util-linux-2.29.1.tar.xz         =
                  20-Jan-2017 14:02      4M
-util-linux-2.29.2.tar.gz         =
                  22-Feb-2017 15:26      9M
-util-linux-2.29.2.tar.sign     =
                    22-Feb-2017 15:26     819
-util-linux-2.29.2.tar.xz         =
                  22-Feb-2017 15:26      4M
-util-linux-2.29.tar.gz             =
                08-Nov-2016 11:23      9M
-util-linux-2.29.tar.sign         =
                  08-Nov-2016 11:23     819
-util-linux-2.29.tar.xz             =
                08-Nov-2016 11:23      4M
-v2.29-ChangeLog                           =
         08-Nov-2016 11:23     28K
-v2.29-ChangeLog.sign                 =
              08-Nov-2016 11:23     819
-v2.29-ReleaseNotes                     =
            08-Nov-2016 11:24     26K
-v2.29-ReleaseNotes.sign           =
                 08-Nov-2016 11:24     819
-v2.29-rc1-ChangeLog                   =
             30-Sep-2016 09:55    219K
-v2.29-rc1-ChangeLog.sign         =
                  30-Sep-2016 09:55     819
-v2.29-rc2-ChangeLog                   =
             19-Oct-2016 13:13     19K
-v2.29-rc2-ChangeLog.sign         =
                  19-Oct-2016 13:13     819
-v2.29.1-ChangeLog                       =
           20-Jan-2017 14:02     47K
-v2.29.1-ChangeLog.sign             =
                20-Jan-2017 14:02     819
-v2.29.1-ReleaseNotes                 =
              20-Jan-2017 14:02    5067
-v2.29.1-ReleaseNotes.sign       =
                   20-Jan-2017 14:02     819
-v2.29.2-ChangeLog                       =
           22-Feb-2017 15:26     14K
-v2.29.2-ChangeLog.sign             =
                22-Feb-2017 15:26     819
-v2.29.2-ReleaseNotes                 =
              22-Feb-2017 15:26    2012
-v2.29.2-ReleaseNotes.sign       =
                   22-Feb-2017 15:26     819
-
    - +=0D +Index of /pub/linux/utils/util-linux/v2.29/=0D +=0D +

    Index of /pub/linux/utils/util-linux/v2.29/

    ../=0D
+libblkid-docs/                             =
        22-Feb-2017 15:20       -=0D
+libfdisk-docs/                             =
        22-Feb-2017 15:22       -=0D
+libmount-docs/                             =
        22-Feb-2017 15:24       -=0D
+libsmartcols-docs/                     =
            22-Feb-2017 15:26       -=0D
+sha256sums.asc                             =
        12-May-2017 10:55    2480=0D
+util-linux-2.29-rc1.tar.gz     =
                    30-Sep-2016 09:55      9M=0D
+util-linux-2.29-rc1.tar.sign =
                      30-Sep-2016 09:55     819=0D
+util-linux-2.29-rc1.tar.xz     =
                    30-Sep-2016 09:55      4M=0D
+util-linux-2.29-rc2.tar.gz     =
                    19-Oct-2016 13:13      9M=0D
+util-linux-2.29-rc2.tar.sign =
                      19-Oct-2016 13:13     819=0D
+util-linux-2.29-rc2.tar.xz     =
                    19-Oct-2016 13:13      4M=0D
+util-linux-2.29.1.tar.gz         =
                  20-Jan-2017 14:02      9M=0D
+util-linux-2.29.1.tar.sign     =
                    20-Jan-2017 14:02     819=0D
+util-linux-2.29.1.tar.xz         =
                  20-Jan-2017 14:02      4M=0D
+util-linux-2.29.2.tar.gz         =
                  22-Feb-2017 15:26      9M=0D
+util-linux-2.29.2.tar.sign     =
                    22-Feb-2017 15:26     819=0D
+util-linux-2.29.2.tar.xz         =
                  22-Feb-2017 15:26      4M=0D
+util-linux-2.29.tar.gz             =
                08-Nov-2016 11:23      9M=0D
+util-linux-2.29.tar.sign         =
                  08-Nov-2016 11:23     819=0D
+util-linux-2.29.tar.xz             =
                08-Nov-2016 11:23      4M=0D
+v2.29-ChangeLog                           =
         08-Nov-2016 11:23     28K=0D
+v2.29-ChangeLog.sign                 =
              08-Nov-2016 11:23     819=0D
+v2.29-ReleaseNotes                     =
            08-Nov-2016 11:24     26K=0D
+v2.29-ReleaseNotes.sign           =
                 08-Nov-2016 11:24     819=0D
+v2.29-rc1-ChangeLog                   =
             30-Sep-2016 09:55    219K=0D
+v2.29-rc1-ChangeLog.sign         =
                  30-Sep-2016 09:55     819=0D
+v2.29-rc2-ChangeLog                   =
             19-Oct-2016 13:13     19K=0D
+v2.29-rc2-ChangeLog.sign         =
                  19-Oct-2016 13:13     819=0D
+v2.29.1-ChangeLog                       =
           20-Jan-2017 14:02     47K=0D
+v2.29.1-ChangeLog.sign             =
                20-Jan-2017 14:02     819=0D
+v2.29.1-ReleaseNotes                 =
              20-Jan-2017 14:02    5067=0D
+v2.29.1-ReleaseNotes.sign       =
                   20-Jan-2017 14:02     819=0D
+v2.29.2-ChangeLog                       =
           22-Feb-2017 15:26     14K=0D
+v2.29.2-ChangeLog.sign             =
                22-Feb-2017 15:26     819=0D
+v2.29.2-ReleaseNotes                 =
              22-Feb-2017 15:26    2012=0D
+v2.29.2-ReleaseNotes.sign       =
                   22-Feb-2017 15:26     819=0D
+
    =0D +=0D diff --git a/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux= /v2.30/index.html b/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/uti= l-linux/v2.30/index.html index 6f17f06e..0441bc0d 100644 --- a/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux/v2.30/= index.html +++ b/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux/v2.30/= index.html @@ -1,42 +1,42 @@ - -Index of /pub/linux/utils/util-linux/v2.30/ - -

    Index of /pub/linux/utils/util-linux/v2.30/

    ../
-libblkid-docs/                             =
        21-Sep-2017 09:49       -
-libfdisk-docs/                             =
        21-Sep-2017 09:50       -
-libmount-docs/                             =
        21-Sep-2017 09:50       -
-libsmartcols-docs/                     =
            21-Sep-2017 09:51       -
-sha256sums.asc                             =
        21-Sep-2017 09:52    2480
-util-linux-2.30-rc1.tar.gz     =
                    12-May-2017 12:02      9M
-util-linux-2.30-rc1.tar.sign =
                      12-May-2017 12:02     819
-util-linux-2.30-rc1.tar.xz     =
                    12-May-2017 12:02      4M
-util-linux-2.30-rc2.tar.gz     =
                    23-May-2017 10:42      9M
-util-linux-2.30-rc2.tar.sign =
                      23-May-2017 10:42     819
-util-linux-2.30-rc2.tar.xz     =
                    23-May-2017 10:42      4M
-util-linux-2.30.1.tar.gz         =
                  20-Jul-2017 09:33      9M
-util-linux-2.30.1.tar.sign     =
                    20-Jul-2017 09:33     819
-util-linux-2.30.1.tar.xz         =
                  20-Jul-2017 09:33      4M
-util-linux-2.30.2.tar.gz         =
                  21-Sep-2017 09:51      9M
-util-linux-2.30.2.tar.sign     =
                    21-Sep-2017 09:51     833
-util-linux-2.30.2.tar.xz         =
                  21-Sep-2017 09:51      4M
-util-linux-2.30.tar.gz             =
                02-Jun-2017 10:44      9M
-util-linux-2.30.tar.sign         =
                  02-Jun-2017 10:44     819
-util-linux-2.30.tar.xz             =
                02-Jun-2017 10:44      4M
-v2.30-ChangeLog                           =
         02-Jun-2017 10:44     28K
-v2.30-ChangeLog.sign                 =
              02-Jun-2017 10:44     819
-v2.30-ReleaseNotes                     =
            02-Jun-2017 10:44     34K
-v2.30-ReleaseNotes.sign           =
                 02-Jun-2017 10:44     819
-v2.30-rc1-ChangeLog                   =
             23-May-2017 10:42    318K
-v2.30-rc1-ChangeLog.sign         =
                  23-May-2017 10:42     819
-v2.30-rc2-ChangeLog                   =
             23-May-2017 10:42     19K
-v2.30-rc2-ChangeLog.sign         =
                  23-May-2017 10:42     819
-v2.30.1-ChangeLog                       =
           20-Jul-2017 09:33     20K
-v2.30.1-ChangeLog.sign             =
                20-Jul-2017 09:33     819
-v2.30.1-ReleaseNotes                 =
              20-Jul-2017 09:33    1901
-v2.30.1-ReleaseNotes.sign       =
                   20-Jul-2017 09:33     819
-v2.30.2-ChangeLog                       =
           21-Sep-2017 09:51     13K
-v2.30.2-ChangeLog.sign             =
                21-Sep-2017 09:51     833
-v2.30.2-ReleaseNotes                 =
              21-Sep-2017 09:51    1604
-v2.30.2-ReleaseNotes.sign       =
                   21-Sep-2017 09:51     833
-
    - +=0D +Index of /pub/linux/utils/util-linux/v2.30/=0D +=0D +

    Index of /pub/linux/utils/util-linux/v2.30/

    ../=0D
+libblkid-docs/                             =
        21-Sep-2017 09:49       -=0D
+libfdisk-docs/                             =
        21-Sep-2017 09:50       -=0D
+libmount-docs/                             =
        21-Sep-2017 09:50       -=0D
+libsmartcols-docs/                     =
            21-Sep-2017 09:51       -=0D
+sha256sums.asc                             =
        21-Sep-2017 09:52    2480=0D
+util-linux-2.30-rc1.tar.gz     =
                    12-May-2017 12:02      9M=0D
+util-linux-2.30-rc1.tar.sign =
                      12-May-2017 12:02     819=0D
+util-linux-2.30-rc1.tar.xz     =
                    12-May-2017 12:02      4M=0D
+util-linux-2.30-rc2.tar.gz     =
                    23-May-2017 10:42      9M=0D
+util-linux-2.30-rc2.tar.sign =
                      23-May-2017 10:42     819=0D
+util-linux-2.30-rc2.tar.xz     =
                    23-May-2017 10:42      4M=0D
+util-linux-2.30.1.tar.gz         =
                  20-Jul-2017 09:33      9M=0D
+util-linux-2.30.1.tar.sign     =
                    20-Jul-2017 09:33     819=0D
+util-linux-2.30.1.tar.xz         =
                  20-Jul-2017 09:33      4M=0D
+util-linux-2.30.2.tar.gz         =
                  21-Sep-2017 09:51      9M=0D
+util-linux-2.30.2.tar.sign     =
                    21-Sep-2017 09:51     833=0D
+util-linux-2.30.2.tar.xz         =
                  21-Sep-2017 09:51      4M=0D
+util-linux-2.30.tar.gz             =
                02-Jun-2017 10:44      9M=0D
+util-linux-2.30.tar.sign         =
                  02-Jun-2017 10:44     819=0D
+util-linux-2.30.tar.xz             =
                02-Jun-2017 10:44      4M=0D
+v2.30-ChangeLog                           =
         02-Jun-2017 10:44     28K=0D
+v2.30-ChangeLog.sign                 =
              02-Jun-2017 10:44     819=0D
+v2.30-ReleaseNotes                     =
            02-Jun-2017 10:44     34K=0D
+v2.30-ReleaseNotes.sign           =
                 02-Jun-2017 10:44     819=0D
+v2.30-rc1-ChangeLog                   =
             23-May-2017 10:42    318K=0D
+v2.30-rc1-ChangeLog.sign         =
                  23-May-2017 10:42     819=0D
+v2.30-rc2-ChangeLog                   =
             23-May-2017 10:42     19K=0D
+v2.30-rc2-ChangeLog.sign         =
                  23-May-2017 10:42     819=0D
+v2.30.1-ChangeLog                       =
           20-Jul-2017 09:33     20K=0D
+v2.30.1-ChangeLog.sign             =
                20-Jul-2017 09:33     819=0D
+v2.30.1-ReleaseNotes                 =
              20-Jul-2017 09:33    1901=0D
+v2.30.1-ReleaseNotes.sign       =
                   20-Jul-2017 09:33     819=0D
+v2.30.2-ChangeLog                       =
           21-Sep-2017 09:51     13K=0D
+v2.30.2-ChangeLog.sign             =
                21-Sep-2017 09:51     833=0D
+v2.30.2-ReleaseNotes                 =
              21-Sep-2017 09:51    1604=0D
+v2.30.2-ReleaseNotes.sign       =
                   21-Sep-2017 09:51     833=0D
+
    =0D +=0D diff --git a/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux= /v2.31/index.html b/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/uti= l-linux/v2.31/index.html index e3c7dd5c..097e4e0e 100644 --- a/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux/v2.31/= index.html +++ b/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux/v2.31/= index.html @@ -1,35 +1,35 @@ - -Index of /pub/linux/utils/util-linux/v2.31/ - -

    Index of /pub/linux/utils/util-linux/v2.31/

    ../
-libblkid-docs/                             =
        19-Dec-2017 15:16       -
-libfdisk-docs/                             =
        19-Dec-2017 15:17       -
-libmount-docs/                             =
        19-Dec-2017 15:17       -
-libsmartcols-docs/                     =
            19-Dec-2017 15:17       -
-sha256sums.asc                             =
        19-Dec-2017 15:20    2127
-util-linux-2.31-rc1.tar.gz     =
                    22-Sep-2017 10:39      9M
-util-linux-2.31-rc1.tar.sign =
                      22-Sep-2017 10:39     833
-util-linux-2.31-rc1.tar.xz     =
                    22-Sep-2017 10:39      4M
-util-linux-2.31-rc2.tar.gz     =
                    03-Oct-2017 16:03      9M
-util-linux-2.31-rc2.tar.sign =
                      03-Oct-2017 16:03     833
-util-linux-2.31-rc2.tar.xz     =
                    03-Oct-2017 16:03      4M
-util-linux-2.31.1.tar.gz         =
                  19-Dec-2017 15:18      9M
-util-linux-2.31.1.tar.sign     =
                    19-Dec-2017 15:18     833
-util-linux-2.31.1.tar.xz         =
                  19-Dec-2017 15:18      4M
-util-linux-2.31.tar.gz             =
                19-Oct-2017 11:27      9M
-util-linux-2.31.tar.sign         =
                  19-Oct-2017 11:27     833
-util-linux-2.31.tar.xz             =
                19-Oct-2017 11:27      4M
-v2.31-ChangeLog                           =
         19-Oct-2017 11:27     15K
-v2.31-ChangeLog.sign                 =
              19-Oct-2017 11:27     833
-v2.31-ReleaseNotes                     =
            19-Oct-2017 11:27     31K
-v2.31-ReleaseNotes.sign           =
                 19-Oct-2017 11:27     833
-v2.31-rc1-ChangeLog                   =
             22-Sep-2017 10:39    290K
-v2.31-rc1-ChangeLog.sign         =
                  22-Sep-2017 10:39     833
-v2.31-rc2-ChangeLog                   =
             03-Oct-2017 16:03     12K
-v2.31-rc2-ChangeLog.sign         =
                  03-Oct-2017 16:03     833
-v2.31.1-ChangeLog                       =
           19-Dec-2017 15:18     27K
-v2.31.1-ChangeLog.sign             =
                19-Dec-2017 15:18     833
-v2.31.1-ReleaseNotes                 =
              19-Dec-2017 15:18    3175
-v2.31.1-ReleaseNotes.sign       =
                   19-Dec-2017 15:18     833
-
    - +=0D +Index of /pub/linux/utils/util-linux/v2.31/=0D +=0D +

    Index of /pub/linux/utils/util-linux/v2.31/

    ../=0D
+libblkid-docs/                             =
        19-Dec-2017 15:16       -=0D
+libfdisk-docs/                             =
        19-Dec-2017 15:17       -=0D
+libmount-docs/                             =
        19-Dec-2017 15:17       -=0D
+libsmartcols-docs/                     =
            19-Dec-2017 15:17       -=0D
+sha256sums.asc                             =
        19-Dec-2017 15:20    2127=0D
+util-linux-2.31-rc1.tar.gz     =
                    22-Sep-2017 10:39      9M=0D
+util-linux-2.31-rc1.tar.sign =
                      22-Sep-2017 10:39     833=0D
+util-linux-2.31-rc1.tar.xz     =
                    22-Sep-2017 10:39      4M=0D
+util-linux-2.31-rc2.tar.gz     =
                    03-Oct-2017 16:03      9M=0D
+util-linux-2.31-rc2.tar.sign =
                      03-Oct-2017 16:03     833=0D
+util-linux-2.31-rc2.tar.xz     =
                    03-Oct-2017 16:03      4M=0D
+util-linux-2.31.1.tar.gz         =
                  19-Dec-2017 15:18      9M=0D
+util-linux-2.31.1.tar.sign     =
                    19-Dec-2017 15:18     833=0D
+util-linux-2.31.1.tar.xz         =
                  19-Dec-2017 15:18      4M=0D
+util-linux-2.31.tar.gz             =
                19-Oct-2017 11:27      9M=0D
+util-linux-2.31.tar.sign         =
                  19-Oct-2017 11:27     833=0D
+util-linux-2.31.tar.xz             =
                19-Oct-2017 11:27      4M=0D
+v2.31-ChangeLog                           =
         19-Oct-2017 11:27     15K=0D
+v2.31-ChangeLog.sign                 =
              19-Oct-2017 11:27     833=0D
+v2.31-ReleaseNotes                     =
            19-Oct-2017 11:27     31K=0D
+v2.31-ReleaseNotes.sign           =
                 19-Oct-2017 11:27     833=0D
+v2.31-rc1-ChangeLog                   =
             22-Sep-2017 10:39    290K=0D
+v2.31-rc1-ChangeLog.sign         =
                  22-Sep-2017 10:39     833=0D
+v2.31-rc2-ChangeLog                   =
             03-Oct-2017 16:03     12K=0D
+v2.31-rc2-ChangeLog.sign         =
                  03-Oct-2017 16:03     833=0D
+v2.31.1-ChangeLog                       =
           19-Dec-2017 15:18     27K=0D
+v2.31.1-ChangeLog.sign             =
                19-Dec-2017 15:18     833=0D
+v2.31.1-ReleaseNotes                 =
              19-Dec-2017 15:18    3175=0D
+v2.31.1-ReleaseNotes.sign       =
                   19-Dec-2017 15:18     833=0D
+
    =0D +=0D diff --git a/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux= /v2.32/index.html b/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/uti= l-linux/v2.32/index.html index 2955ace0..d373e981 100644 --- a/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux/v2.32/= index.html +++ b/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux/v2.32/= index.html @@ -1,35 +1,35 @@ - -Index of /pub/linux/utils/util-linux/v2.32/ - -

    Index of /pub/linux/utils/util-linux/v2.32/

    ../
-libblkid-docs/                             =
        16-Jul-2018 11:27       -
-libfdisk-docs/                             =
        16-Jul-2018 11:28       -
-libmount-docs/                             =
        16-Jul-2018 11:28       -
-libsmartcols-docs/                     =
            16-Jul-2018 11:28       -
-sha256sums.asc                             =
        16-Jul-2018 11:30    2127
-util-linux-2.32-rc1.tar.gz     =
                    13-Feb-2018 12:25      9M
-util-linux-2.32-rc1.tar.sign =
                      13-Feb-2018 12:25     833
-util-linux-2.32-rc1.tar.xz     =
                    13-Feb-2018 12:25      4M
-util-linux-2.32-rc2.tar.gz     =
                    01-Mar-2018 13:38      9M
-util-linux-2.32-rc2.tar.sign =
                      01-Mar-2018 13:38     833
-util-linux-2.32-rc2.tar.xz     =
                    01-Mar-2018 13:38      4M
-util-linux-2.32.1.tar.gz         =
                  16-Jul-2018 11:29      9M
-util-linux-2.32.1.tar.sign     =
                    16-Jul-2018 11:29     833
-util-linux-2.32.1.tar.xz         =
                  16-Jul-2018 11:29      4M
-util-linux-2.32.tar.gz             =
                21-Mar-2018 14:49      9M
-util-linux-2.32.tar.sign         =
                  21-Mar-2018 14:49     833
-util-linux-2.32.tar.xz             =
                21-Mar-2018 14:49      4M
-v2.32-ChangeLog                           =
         21-Mar-2018 14:49     36K
-v2.32-ChangeLog.sign                 =
              21-Mar-2018 14:49     833
-v2.32-ReleaseNotes                     =
            21-Mar-2018 14:49     21K
-v2.32-ReleaseNotes.sign           =
                 21-Mar-2018 14:49     833
-v2.32-rc1-ChangeLog                   =
             13-Feb-2018 12:25    174K
-v2.32-rc1-ChangeLog.sign         =
                  13-Feb-2018 12:25     833
-v2.32-rc2-ChangeLog                   =
             01-Mar-2018 13:38     21K
-v2.32-rc2-ChangeLog.sign         =
                  01-Mar-2018 13:38     833
-v2.32.1-ChangeLog                       =
           16-Jul-2018 11:29     31K
-v2.32.1-ChangeLog.sign             =
                16-Jul-2018 11:29     833
-v2.32.1-ReleaseNotes                 =
              16-Jul-2018 11:29    3425
-v2.32.1-ReleaseNotes.sign       =
                   16-Jul-2018 11:29     833
-
    - +=0D +Index of /pub/linux/utils/util-linux/v2.32/=0D +=0D +

    Index of /pub/linux/utils/util-linux/v2.32/

    ../=0D
+libblkid-docs/                             =
        16-Jul-2018 11:27       -=0D
+libfdisk-docs/                             =
        16-Jul-2018 11:28       -=0D
+libmount-docs/                             =
        16-Jul-2018 11:28       -=0D
+libsmartcols-docs/                     =
            16-Jul-2018 11:28       -=0D
+sha256sums.asc                             =
        16-Jul-2018 11:30    2127=0D
+util-linux-2.32-rc1.tar.gz     =
                    13-Feb-2018 12:25      9M=0D
+util-linux-2.32-rc1.tar.sign =
                      13-Feb-2018 12:25     833=0D
+util-linux-2.32-rc1.tar.xz     =
                    13-Feb-2018 12:25      4M=0D
+util-linux-2.32-rc2.tar.gz     =
                    01-Mar-2018 13:38      9M=0D
+util-linux-2.32-rc2.tar.sign =
                      01-Mar-2018 13:38     833=0D
+util-linux-2.32-rc2.tar.xz     =
                    01-Mar-2018 13:38      4M=0D
+util-linux-2.32.1.tar.gz         =
                  16-Jul-2018 11:29      9M=0D
+util-linux-2.32.1.tar.sign     =
                    16-Jul-2018 11:29     833=0D
+util-linux-2.32.1.tar.xz         =
                  16-Jul-2018 11:29      4M=0D
+util-linux-2.32.tar.gz             =
                21-Mar-2018 14:49      9M=0D
+util-linux-2.32.tar.sign         =
                  21-Mar-2018 14:49     833=0D
+util-linux-2.32.tar.xz             =
                21-Mar-2018 14:49      4M=0D
+v2.32-ChangeLog                           =
         21-Mar-2018 14:49     36K=0D
+v2.32-ChangeLog.sign                 =
              21-Mar-2018 14:49     833=0D
+v2.32-ReleaseNotes                     =
            21-Mar-2018 14:49     21K=0D
+v2.32-ReleaseNotes.sign           =
                 21-Mar-2018 14:49     833=0D
+v2.32-rc1-ChangeLog                   =
             13-Feb-2018 12:25    174K=0D
+v2.32-rc1-ChangeLog.sign         =
                  13-Feb-2018 12:25     833=0D
+v2.32-rc2-ChangeLog                   =
             01-Mar-2018 13:38     21K=0D
+v2.32-rc2-ChangeLog.sign         =
                  01-Mar-2018 13:38     833=0D
+v2.32.1-ChangeLog                       =
           16-Jul-2018 11:29     31K=0D
+v2.32.1-ChangeLog.sign             =
                16-Jul-2018 11:29     833=0D
+v2.32.1-ReleaseNotes                 =
              16-Jul-2018 11:29    3425=0D
+v2.32.1-ReleaseNotes.sign       =
                   16-Jul-2018 11:29     833=0D
+
    =0D +=0D diff --git a/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux= /v2.33/index.html b/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/uti= l-linux/v2.33/index.html index 2c051750..54953054 100644 --- a/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux/v2.33/= index.html +++ b/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux/v2.33/= index.html @@ -1,42 +1,42 @@ - -Index of /pub/linux/utils/util-linux/v2.33/ - -

    Index of /pub/linux/utils/util-linux/v2.33/

    ../
-libblkid-docs/                             =
        09-Apr-2019 13:55       -
-libfdisk-docs/                             =
        09-Apr-2019 13:56       -
-libmount-docs/                             =
        09-Apr-2019 13:56       -
-libsmartcols-docs/                     =
            09-Apr-2019 13:56       -
-sha256sums.asc                             =
        09-Apr-2019 14:01    2480
-util-linux-2.33-rc1.tar.gz     =
                    25-Sep-2018 10:34      9M
-util-linux-2.33-rc1.tar.sign =
                      25-Sep-2018 10:34     833
-util-linux-2.33-rc1.tar.xz     =
                    25-Sep-2018 10:34      4M
-util-linux-2.33-rc2.tar.gz     =
                    19-Oct-2018 11:44      9M
-util-linux-2.33-rc2.tar.sign =
                      19-Oct-2018 11:44     833
-util-linux-2.33-rc2.tar.xz     =
                    19-Oct-2018 11:44      4M
-util-linux-2.33.1.tar.gz         =
                  09-Jan-2019 10:28      9M
-util-linux-2.33.1.tar.sign     =
                    09-Jan-2019 10:28     833
-util-linux-2.33.1.tar.xz         =
                  09-Jan-2019 10:28      4M
-util-linux-2.33.2.tar.gz         =
                  09-Apr-2019 13:57     10M
-util-linux-2.33.2.tar.sign     =
                    09-Apr-2019 13:57     833
-util-linux-2.33.2.tar.xz         =
                  09-Apr-2019 13:57      4M
-util-linux-2.33.tar.gz             =
                06-Nov-2018 11:25      9M
-util-linux-2.33.tar.sign         =
                  06-Nov-2018 11:25     833
-util-linux-2.33.tar.xz             =
                06-Nov-2018 11:25      4M
-v2.33-ChangeLog                           =
         06-Nov-2018 11:25    7977
-v2.33-ChangeLog.sign                 =
              06-Nov-2018 11:25     833
-v2.33-ReleaseNotes                     =
            06-Nov-2018 11:25     27K
-v2.33-ReleaseNotes.sign           =
                 06-Nov-2018 11:25     833
-v2.33-rc1-ChangeLog                   =
             25-Sep-2018 10:34    210K
-v2.33-rc1-ChangeLog.sign         =
                  25-Sep-2018 10:34     833
-v2.33-rc2-ChangeLog                   =
             19-Oct-2018 11:44     18K
-v2.33-rc2-ChangeLog.sign         =
                  19-Oct-2018 11:44     833
-v2.33.1-ChangeLog                       =
           09-Jan-2019 10:28     17K
-v2.33.1-ChangeLog.sign             =
                09-Jan-2019 10:28     833
-v2.33.1-ReleaseNotes                 =
              09-Jan-2019 10:28    1899
-v2.33.1-ReleaseNotes.sign       =
                   09-Jan-2019 10:28     833
-v2.33.2-ChangeLog                       =
           09-Apr-2019 13:57     21K
-v2.33.2-ChangeLog.sign             =
                09-Apr-2019 13:57     833
-v2.33.2-ReleaseNotes                 =
              09-Apr-2019 13:57    2566
-v2.33.2-ReleaseNotes.sign       =
                   09-Apr-2019 13:57     833
-
    - +=0D +Index of /pub/linux/utils/util-linux/v2.33/=0D +=0D +

    Index of /pub/linux/utils/util-linux/v2.33/

    ../=0D
+libblkid-docs/                             =
        09-Apr-2019 13:55       -=0D
+libfdisk-docs/                             =
        09-Apr-2019 13:56       -=0D
+libmount-docs/                             =
        09-Apr-2019 13:56       -=0D
+libsmartcols-docs/                     =
            09-Apr-2019 13:56       -=0D
+sha256sums.asc                             =
        09-Apr-2019 14:01    2480=0D
+util-linux-2.33-rc1.tar.gz     =
                    25-Sep-2018 10:34      9M=0D
+util-linux-2.33-rc1.tar.sign =
                      25-Sep-2018 10:34     833=0D
+util-linux-2.33-rc1.tar.xz     =
                    25-Sep-2018 10:34      4M=0D
+util-linux-2.33-rc2.tar.gz     =
                    19-Oct-2018 11:44      9M=0D
+util-linux-2.33-rc2.tar.sign =
                      19-Oct-2018 11:44     833=0D
+util-linux-2.33-rc2.tar.xz     =
                    19-Oct-2018 11:44      4M=0D
+util-linux-2.33.1.tar.gz         =
                  09-Jan-2019 10:28      9M=0D
+util-linux-2.33.1.tar.sign     =
                    09-Jan-2019 10:28     833=0D
+util-linux-2.33.1.tar.xz         =
                  09-Jan-2019 10:28      4M=0D
+util-linux-2.33.2.tar.gz         =
                  09-Apr-2019 13:57     10M=0D
+util-linux-2.33.2.tar.sign     =
                    09-Apr-2019 13:57     833=0D
+util-linux-2.33.2.tar.xz         =
                  09-Apr-2019 13:57      4M=0D
+util-linux-2.33.tar.gz             =
                06-Nov-2018 11:25      9M=0D
+util-linux-2.33.tar.sign         =
                  06-Nov-2018 11:25     833=0D
+util-linux-2.33.tar.xz             =
                06-Nov-2018 11:25      4M=0D
+v2.33-ChangeLog                           =
         06-Nov-2018 11:25    7977=0D
+v2.33-ChangeLog.sign                 =
              06-Nov-2018 11:25     833=0D
+v2.33-ReleaseNotes                     =
            06-Nov-2018 11:25     27K=0D
+v2.33-ReleaseNotes.sign           =
                 06-Nov-2018 11:25     833=0D
+v2.33-rc1-ChangeLog                   =
             25-Sep-2018 10:34    210K=0D
+v2.33-rc1-ChangeLog.sign         =
                  25-Sep-2018 10:34     833=0D
+v2.33-rc2-ChangeLog                   =
             19-Oct-2018 11:44     18K=0D
+v2.33-rc2-ChangeLog.sign         =
                  19-Oct-2018 11:44     833=0D
+v2.33.1-ChangeLog                       =
           09-Jan-2019 10:28     17K=0D
+v2.33.1-ChangeLog.sign             =
                09-Jan-2019 10:28     833=0D
+v2.33.1-ReleaseNotes                 =
              09-Jan-2019 10:28    1899=0D
+v2.33.1-ReleaseNotes.sign       =
                   09-Jan-2019 10:28     833=0D
+v2.33.2-ChangeLog                       =
           09-Apr-2019 13:57     21K=0D
+v2.33.2-ChangeLog.sign             =
                09-Apr-2019 13:57     833=0D
+v2.33.2-ReleaseNotes                 =
              09-Apr-2019 13:57    2566=0D
+v2.33.2-ReleaseNotes.sign       =
                   09-Apr-2019 13:57     833=0D
+
    =0D +=0D diff --git a/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux= /v2.34/index.html b/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/uti= l-linux/v2.34/index.html index ac6c34f6..bd9d9c87 100644 --- a/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux/v2.34/= index.html +++ b/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux/v2.34/= index.html @@ -1,28 +1,28 @@ - -Index of /pub/linux/utils/util-linux/v2.34/ - -

    Index of /pub/linux/utils/util-linux/v2.34/

    ../
-libblkid-docs/                             =
        14-Jun-2019 10:45       -
-libfdisk-docs/                             =
        14-Jun-2019 10:45       -
-libmount-docs/                             =
        14-Jun-2019 10:45       -
-libsmartcols-docs/                     =
            14-Jun-2019 10:46       -
-sha256sums.asc                             =
        14-Jun-2019 10:51    1774
-util-linux-2.34-rc1.tar.gz     =
                    30-Apr-2019 10:24     10M
-util-linux-2.34-rc1.tar.sign =
                      30-Apr-2019 10:24     833
-util-linux-2.34-rc1.tar.xz     =
                    30-Apr-2019 10:24      5M
-util-linux-2.34-rc2.tar.gz     =
                    30-May-2019 10:24     10M
-util-linux-2.34-rc2.tar.sign =
                      30-May-2019 10:24     833
-util-linux-2.34-rc2.tar.xz     =
                    30-May-2019 10:24      5M
-util-linux-2.34.tar.gz             =
                14-Jun-2019 10:46     10M
-util-linux-2.34.tar.sign         =
                  14-Jun-2019 10:46     833
-util-linux-2.34.tar.xz             =
                14-Jun-2019 10:46      5M
-v2.34-ChangeLog                           =
         14-Jun-2019 10:46     14K
-v2.34-ChangeLog.sign                 =
              14-Jun-2019 10:46     833
-v2.34-ReleaseNotes                     =
            14-Jun-2019 10:46     27K
-v2.34-ReleaseNotes.sign           =
                 14-Jun-2019 10:46     833
-v2.34-rc1-ChangeLog                   =
             30-Apr-2019 10:24    167K
-v2.34-rc1-ChangeLog.sign         =
                  30-Apr-2019 10:24     833
-v2.34-rc2-ChangeLog                   =
             30-May-2019 10:24     57K
-v2.34-rc2-ChangeLog.sign         =
                  30-May-2019 10:24     833
-
    - +=0D +Index of /pub/linux/utils/util-linux/v2.34/=0D +=0D +

    Index of /pub/linux/utils/util-linux/v2.34/

    ../=0D
+libblkid-docs/                             =
        14-Jun-2019 10:45       -=0D
+libfdisk-docs/                             =
        14-Jun-2019 10:45       -=0D
+libmount-docs/                             =
        14-Jun-2019 10:45       -=0D
+libsmartcols-docs/                     =
            14-Jun-2019 10:46       -=0D
+sha256sums.asc                             =
        14-Jun-2019 10:51    1774=0D
+util-linux-2.34-rc1.tar.gz     =
                    30-Apr-2019 10:24     10M=0D
+util-linux-2.34-rc1.tar.sign =
                      30-Apr-2019 10:24     833=0D
+util-linux-2.34-rc1.tar.xz     =
                    30-Apr-2019 10:24      5M=0D
+util-linux-2.34-rc2.tar.gz     =
                    30-May-2019 10:24     10M=0D
+util-linux-2.34-rc2.tar.sign =
                      30-May-2019 10:24     833=0D
+util-linux-2.34-rc2.tar.xz     =
                    30-May-2019 10:24      5M=0D
+util-linux-2.34.tar.gz             =
                14-Jun-2019 10:46     10M=0D
+util-linux-2.34.tar.sign         =
                  14-Jun-2019 10:46     833=0D
+util-linux-2.34.tar.xz             =
                14-Jun-2019 10:46      5M=0D
+v2.34-ChangeLog                           =
         14-Jun-2019 10:46     14K=0D
+v2.34-ChangeLog.sign                 =
              14-Jun-2019 10:46     833=0D
+v2.34-ReleaseNotes                     =
            14-Jun-2019 10:46     27K=0D
+v2.34-ReleaseNotes.sign           =
                 14-Jun-2019 10:46     833=0D
+v2.34-rc1-ChangeLog                   =
             30-Apr-2019 10:24    167K=0D
+v2.34-rc1-ChangeLog.sign         =
                  30-Apr-2019 10:24     833=0D
+v2.34-rc2-ChangeLog                   =
             30-May-2019 10:24     57K=0D
+v2.34-rc2-ChangeLog.sign         =
                  30-May-2019 10:24     833=0D
+
    =0D +=0D diff --git a/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux= /v2.35/index.html b/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/uti= l-linux/v2.35/index.html index 66dc0c59..aa714d39 100644 --- a/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux/v2.35/= index.html +++ b/bitbake/lib/bb/tests/fetch-testdata/pub/linux/utils/util-linux/v2.35/= index.html @@ -1,18 +1,18 @@ - -Index of /pub/linux/utils/util-linux/v2.35/ - -

    Index of /pub/linux/utils/util-linux/v2.35/

    ../
-libblkid-docs/                             =
        11-Dec-2019 10:04       -
-libfdisk-docs/                             =
        11-Dec-2019 10:05       -
-libmount-docs/                             =
        11-Dec-2019 10:05       -
-libsmartcols-docs/                     =
            11-Dec-2019 10:05       -
-sha256sums.asc                             =
        11-Dec-2019 10:11    1242
-util-linux-2.35-rc1.tar.gz     =
                    11-Dec-2019 10:06     10M
-util-linux-2.35-rc1.tar.sign =
                      11-Dec-2019 10:06     833
-util-linux-2.35-rc1.tar.xz     =
                    11-Dec-2019 10:06      5M
-v2.35-ReleaseNotes                     =
            11-Dec-2019 10:06     21K
-v2.35-ReleaseNotes.sign           =
                 11-Dec-2019 10:06     833
-v2.35-rc1-ChangeLog                   =
             11-Dec-2019 10:06    228K
-v2.35-rc1-ChangeLog.sign         =
                  11-Dec-2019 10:06     833
-
    - +=0D +Index of /pub/linux/utils/util-linux/v2.35/=0D +=0D +

    Index of /pub/linux/utils/util-linux/v2.35/

    ../=0D
+libblkid-docs/                             =
        11-Dec-2019 10:04       -=0D
+libfdisk-docs/                             =
        11-Dec-2019 10:05       -=0D
+libmount-docs/                             =
        11-Dec-2019 10:05       -=0D
+libsmartcols-docs/                     =
            11-Dec-2019 10:05       -=0D
+sha256sums.asc                             =
        11-Dec-2019 10:11    1242=0D
+util-linux-2.35-rc1.tar.gz     =
                    11-Dec-2019 10:06     10M=0D
+util-linux-2.35-rc1.tar.sign =
                      11-Dec-2019 10:06     833=0D
+util-linux-2.35-rc1.tar.xz     =
                    11-Dec-2019 10:06      5M=0D
+v2.35-ReleaseNotes                     =
            11-Dec-2019 10:06     21K=0D
+v2.35-ReleaseNotes.sign           =
                 11-Dec-2019 10:06     833=0D
+v2.35-rc1-ChangeLog                   =
             11-Dec-2019 10:06    228K=0D
+v2.35-rc1-ChangeLog.sign         =
                  11-Dec-2019 10:06     833=0D
+
    =0D +=0D diff --git a/bitbake/lib/bb/tests/fetch-testdata/releases/eglibc/index.html= b/bitbake/lib/bb/tests/fetch-testdata/releases/eglibc/index.html index ca97c87f..b2679402 100644 --- a/bitbake/lib/bb/tests/fetch-testdata/releases/eglibc/index.html +++ b/bitbake/lib/bb/tests/fetch-testdata/releases/eglibc/index.html @@ -1,21 +1,21 @@ - -Index of /releases/eglibc/ - -

    Index of /releases/eglibc/

    ../
-eglibc-2.16-svnr21224.tar.bz2                      17-Oct-2012 18:01            17310656
-eglibc-2.16-svnr21224.tar.bz=
2.md5                  17-Oct-2012 21:53                  64
-eglibc-2.16-svnr21224.tar.b=
z2.sha1                 17-Oct-2012 21:53                  72
-eglibc-2.17-svnr22064.tar.bz2                      04-Jan-2013 05:44            17565519
-eglibc-2.17-svnr22064.tar.bz=
2.asc                  04-Jan-2013 05:45                 302
-eglibc-2.17-svnr22064.tar.bz=
2.md5                  04-Jan-2013 05:44                  64
-eglibc-2.17-svnr22064.tar.b=
z2.sha1                 04-Jan-2013 05:44                  72
-eglibc-2.18-svnr23787.tar.bz2                      21-Aug-2013 05:36            17862773
-eglibc-2.18-svnr23787.tar.bz=
2.asc                  21-Aug-2013 05:36                 302
-eglibc-2.18-svnr23787.tar.bz=
2.md5                  21-Aug-2013 05:36                  64
-eglibc-2.18-svnr23787.tar.b=
z2.sha1                 21-Aug-2013 05:36                  72
-eglibc-2.19-svnr25243.tar.bz2                      08-Feb-2014 10:06            18873620
-eglibc-2.19-svnr25243.tar.bz=
2.asc                  08-Feb-2014 10:06                 285
-eglibc-2.19-svnr25243.tar.bz=
2.md5                  08-Feb-2014 10:06                  64
-eglibc-2.19-svnr25243.tar.b=
z2.sha1                 08-Feb-2014 10:06                  72
-
    - +=0D +Index of /releases/eglibc/=0D +=0D +

    Index of /releases/eglibc/

    ../=0D
+eglibc-2.16-svnr21224.tar.bz2                      17-Oct-2012 18:01            17310656=0D
+eglibc-2.16-svnr21224.tar.bz=
2.md5                  17-Oct-2012 21:53                  64=0D
+eglibc-2.16-svnr21224.tar.b=
z2.sha1                 17-Oct-2012 21:53                  72=0D
+eglibc-2.17-svnr22064.tar.bz2                      04-Jan-2013 05:44            17565519=0D
+eglibc-2.17-svnr22064.tar.bz=
2.asc                  04-Jan-2013 05:45                 302=0D
+eglibc-2.17-svnr22064.tar.bz=
2.md5                  04-Jan-2013 05:44                  64=0D
+eglibc-2.17-svnr22064.tar.b=
z2.sha1                 04-Jan-2013 05:44                  72=0D
+eglibc-2.18-svnr23787.tar.bz2                      21-Aug-2013 05:36            17862773=0D
+eglibc-2.18-svnr23787.tar.bz=
2.asc                  21-Aug-2013 05:36                 302=0D
+eglibc-2.18-svnr23787.tar.bz=
2.md5                  21-Aug-2013 05:36                  64=0D
+eglibc-2.18-svnr23787.tar.b=
z2.sha1                 21-Aug-2013 05:36                  72=0D
+eglibc-2.19-svnr25243.tar.bz2                      08-Feb-2014 10:06            18873620=0D
+eglibc-2.19-svnr25243.tar.bz=
2.asc                  08-Feb-2014 10:06                 285=0D
+eglibc-2.19-svnr25243.tar.bz=
2.md5                  08-Feb-2014 10:06                  64=0D
+eglibc-2.19-svnr25243.tar.b=
z2.sha1                 08-Feb-2014 10:06                  72=0D
+
    =0D +=0D diff --git a/bitbake/lib/bb/tests/fetch-testdata/releases/gnu-config/index.= html b/bitbake/lib/bb/tests/fetch-testdata/releases/gnu-config/index.html index bf85a50b..051aa481 100644 --- a/bitbake/lib/bb/tests/fetch-testdata/releases/gnu-config/index.html +++ b/bitbake/lib/bb/tests/fetch-testdata/releases/gnu-config/index.html @@ -1,9 +1,9 @@ - -Index of /releases/gnu-config/ - -

    Index of /releases/gnu-config/

    ../
-SHA256SUM                                       =
   03-Oct-2012 17:23                 190
-gnu-config-20120814.tar.bz2   =
                     18-Sep-2012 09:28               43026
-gnu-config-yocto-20111111.tgz                      08-May-2012 21:11               48762
-
    - +=0D +Index of /releases/gnu-config/=0D +=0D +

    Index of /releases/gnu-config/

    ../=0D
+SHA256SUM                                       =
   03-Oct-2012 17:23                 190=0D
+gnu-config-20120814.tar.bz2   =
                     18-Sep-2012 09:28               43026=0D
+gnu-config-yocto-20111111.tgz                      08-May-2012 21:11               48762=0D
+
    =0D +=0D diff --git a/bitbake/lib/bb/tests/fetch.py b/bitbake/lib/bb/tests/fetch.py index 29c96b2b..f5d557e8 100644 --- a/bitbake/lib/bb/tests/fetch.py +++ b/bitbake/lib/bb/tests/fetch.py @@ -87,6 +87,25 @@ class URITest(unittest.TestCase): }, 'relative': False }, + # Check that trailing semicolons are handled correctly + "http://www.example.org/index.html?qparam1=3Dqvalue1;param2=3Dvalu= e2;" : { + 'uri': 'http://www.example.org/index.html?qparam1=3Dqvalue1;pa= ram2=3Dvalue2', + 'scheme': 'http', + 'hostname': 'www.example.org', + 'port': None, + 'hostport': 'www.example.org', + 'path': '/index.html', + 'userinfo': '', + 'username': '', + 'password': '', + 'params': { + 'param2': 'value2' + }, + 'query': { + 'qparam1': 'qvalue1' + }, + 'relative': False + }, "http://www.example.com:8080/index.html" : { 'uri': 'http://www.example.com:8080/index.html', 'scheme': 'http', @@ -223,6 +242,21 @@ class URITest(unittest.TestCase): 'query': {}, 'relative': False }, + "git://tfs-example.org:22/tfs/example%20path/example.git": { + 'uri': 'git://tfs-example.org:22/tfs/example%20path/example.gi= t', + 'scheme': 'git', + 'hostname': 'tfs-example.org', + 'port': 22, + 'hostport': 'tfs-example.org:22', + 'path': '/tfs/example path/example.git', + 'userinfo': '', + 'userinfo': '', + 'username': '', + 'password': '', + 'params': {}, + 'query': {}, + 'relative': False + }, "http://somesite.net;someparam=3D1": { 'uri': 'http://somesite.net;someparam=3D1', 'scheme': 'http', @@ -356,6 +390,7 @@ class FetcherTest(unittest.TestCase): if os.environ.get("BB_TMPDIR_NOCLEAN") =3D=3D "yes": print("Not cleaning up %s. Please remove manually." % self.tem= pdir) else: + bb.process.run('chmod u+rw -R %s' % self.tempdir) bb.utils.prunedir(self.tempdir) =20 class MirrorUriTest(FetcherTest): @@ -584,6 +619,7 @@ class FetcherLocalTest(FetcherTest): touch(os.path.join(self.localsrcdir, 'dir', 'd')) os.makedirs(os.path.join(self.localsrcdir, 'dir', 'subdir')) touch(os.path.join(self.localsrcdir, 'dir', 'subdir', 'e')) + touch(os.path.join(self.localsrcdir, r'backslash\x2dsystemd-unit.d= evice')) self.d.setVar("FILESPATH", self.localsrcdir) =20 def fetchUnpack(self, uris): @@ -601,9 +637,13 @@ class FetcherLocalTest(FetcherTest): tree =3D self.fetchUnpack(['file://a', 'file://dir/c']) self.assertEqual(tree, ['a', 'dir/c']) =20 + def test_local_backslash(self): + tree =3D self.fetchUnpack([r'file://backslash\x2dsystemd-unit.devi= ce']) + self.assertEqual(tree, [r'backslash\x2dsystemd-unit.device']) + def test_local_wildcard(self): - tree =3D self.fetchUnpack(['file://a', 'file://dir/*']) - self.assertEqual(tree, ['a', 'dir/c', 'dir/d', 'dir/subdir/e']) + with self.assertRaises(bb.fetch2.ParameterError): + tree =3D self.fetchUnpack(['file://a', 'file://dir/*']) =20 def test_local_dir(self): tree =3D self.fetchUnpack(['file://a', 'file://dir']) @@ -634,6 +674,62 @@ class FetcherLocalTest(FetcherTest): with self.assertRaises(bb.fetch2.UnpackError): self.fetchUnpack(['file://a;subdir=3D/bin/sh']) =20 + def test_local_gitfetch_usehead(self): + # Create dummy local Git repo + src_dir =3D tempfile.mkdtemp(dir=3Dself.tempdir, + prefix=3D'gitfetch_localusehead_') + src_dir =3D os.path.abspath(src_dir) + bb.process.run("git init", cwd=3Dsrc_dir) + bb.process.run("git config user.email 'you@example.com'", cwd=3Dsr= c_dir) + bb.process.run("git config user.name 'Your Name'", cwd=3Dsrc_dir) + bb.process.run("git commit --allow-empty -m'Dummy commit'", + cwd=3Dsrc_dir) + # Use other branch than master + bb.process.run("git checkout -b my-devel", cwd=3Dsrc_dir) + bb.process.run("git commit --allow-empty -m'Dummy commit 2'", + cwd=3Dsrc_dir) + stdout =3D bb.process.run("git rev-parse HEAD", cwd=3Dsrc_dir) + orig_rev =3D stdout[0].strip() + + # Fetch and check revision + self.d.setVar("SRCREV", "AUTOINC") + url =3D "git://" + src_dir + ";protocol=3Dfile;usehead=3D1" + fetcher =3D bb.fetch.Fetch([url], self.d) + fetcher.download() + fetcher.unpack(self.unpackdir) + stdout =3D bb.process.run("git rev-parse HEAD", + cwd=3Dos.path.join(self.unpackdir, 'git')) + unpack_rev =3D stdout[0].strip() + self.assertEqual(orig_rev, unpack_rev) + + def test_local_gitfetch_usehead_withname(self): + # Create dummy local Git repo + src_dir =3D tempfile.mkdtemp(dir=3Dself.tempdir, + prefix=3D'gitfetch_localusehead_') + src_dir =3D os.path.abspath(src_dir) + bb.process.run("git init", cwd=3Dsrc_dir) + bb.process.run("git config user.email 'you@example.com'", cwd=3Dsr= c_dir) + bb.process.run("git config user.name 'Your Name'", cwd=3Dsrc_dir) + bb.process.run("git commit --allow-empty -m'Dummy commit'", + cwd=3Dsrc_dir) + # Use other branch than master + bb.process.run("git checkout -b my-devel", cwd=3Dsrc_dir) + bb.process.run("git commit --allow-empty -m'Dummy commit 2'", + cwd=3Dsrc_dir) + stdout =3D bb.process.run("git rev-parse HEAD", cwd=3Dsrc_dir) + orig_rev =3D stdout[0].strip() + + # Fetch and check revision + self.d.setVar("SRCREV", "AUTOINC") + url =3D "git://" + src_dir + ";protocol=3Dfile;usehead=3D1;name=3D= newName" + fetcher =3D bb.fetch.Fetch([url], self.d) + fetcher.download() + fetcher.unpack(self.unpackdir) + stdout =3D bb.process.run("git rev-parse HEAD", + cwd=3Dos.path.join(self.unpackdir, 'git')) + unpack_rev =3D stdout[0].strip() + self.assertEqual(orig_rev, unpack_rev) + class FetcherNoNetworkTest(FetcherTest): def setUp(self): super().setUp() @@ -824,35 +920,21 @@ class FetcherNetworkTest(FetcherTest): self.assertRaises(bb.fetch.FetchError, self.gitfetcher, url1, url2) =20 @skipIfNoNetwork() - def test_gitfetch_localusehead(self): - # Create dummy local Git repo - src_dir =3D tempfile.mkdtemp(dir=3Dself.tempdir, - prefix=3D'gitfetch_localusehead_') - src_dir =3D os.path.abspath(src_dir) - bb.process.run("git init", cwd=3Dsrc_dir) - bb.process.run("git commit --allow-empty -m'Dummy commit'", - cwd=3Dsrc_dir) - # Use other branch than master - bb.process.run("git checkout -b my-devel", cwd=3Dsrc_dir) - bb.process.run("git commit --allow-empty -m'Dummy commit 2'", - cwd=3Dsrc_dir) - stdout =3D bb.process.run("git rev-parse HEAD", cwd=3Dsrc_dir) - orig_rev =3D stdout[0].strip() - - # Fetch and check revision - self.d.setVar("SRCREV", "AUTOINC") - url =3D "git://" + src_dir + ";protocol=3Dfile;usehead=3D1" - fetcher =3D bb.fetch.Fetch([url], self.d) - fetcher.download() - fetcher.unpack(self.unpackdir) - stdout =3D bb.process.run("git rev-parse HEAD", - cwd=3Dos.path.join(self.unpackdir, 'git')) - unpack_rev =3D stdout[0].strip() - self.assertEqual(orig_rev, unpack_rev) + def test_gitfetch_usehead(self): + # Since self.gitfetcher() sets SRCREV we expect this to override + # `usehead=3D1' and instead fetch the specified SRCREV. See + # test_local_gitfetch_usehead() for a positive use of the usehead + # feature. + url =3D "git://git.openembedded.org/bitbake;usehead=3D1" + self.assertRaises(bb.fetch.ParameterError, self.gitfetcher, url, u= rl) =20 @skipIfNoNetwork() - def test_gitfetch_remoteusehead(self): - url =3D "git://git.openembedded.org/bitbake;usehead=3D1" + def test_gitfetch_usehead_withname(self): + # Since self.gitfetcher() sets SRCREV we expect this to override + # `usehead=3D1' and instead fetch the specified SRCREV. See + # test_local_gitfetch_usehead() for a positive use of the usehead + # feature. + url =3D "git://git.openembedded.org/bitbake;usehead=3D1;name=3Dnew= Name" self.assertRaises(bb.fetch.ParameterError, self.gitfetcher, url, u= rl) =20 @skipIfNoNetwork() @@ -903,7 +985,7 @@ class FetcherNetworkTest(FetcherTest): def test_git_submodule_dbus_broker(self): # The following external repositories have show failures in fetch = and unpack operations # We want to avoid regressions! - url =3D "gitsm://github.com/bus1/dbus-broker;protocol=3Dgit;rev=3D= fc874afa0992d0c75ec25acb43d344679f0ee7d2" + url =3D "gitsm://github.com/bus1/dbus-broker;protocol=3Dgit;rev=3D= fc874afa0992d0c75ec25acb43d344679f0ee7d2;branch=3Dmain" fetcher =3D bb.fetch.Fetch([url], self.d) fetcher.download() # Previous cwd has been deleted @@ -919,7 +1001,7 @@ class FetcherNetworkTest(FetcherTest): =20 @skipIfNoNetwork() def test_git_submodule_CLI11(self): - url =3D "gitsm://github.com/CLIUtils/CLI11;protocol=3Dgit;rev=3Dbd= 4dc911847d0cde7a6b41dfa626a85aab213baf" + url =3D "gitsm://github.com/CLIUtils/CLI11;protocol=3Dgit;rev=3Dbd= 4dc911847d0cde7a6b41dfa626a85aab213baf;branch=3Dmain" fetcher =3D bb.fetch.Fetch([url], self.d) fetcher.download() # Previous cwd has been deleted @@ -934,12 +1016,12 @@ class FetcherNetworkTest(FetcherTest): @skipIfNoNetwork() def test_git_submodule_update_CLI11(self): """ Prevent regression on update detection not finding missing sub= module, or modules without needed commits """ - url =3D "gitsm://github.com/CLIUtils/CLI11;protocol=3Dgit;rev=3Dcf= 6a99fa69aaefe477cc52e3ef4a7d2d7fa40714" + url =3D "gitsm://github.com/CLIUtils/CLI11;protocol=3Dgit;rev=3Dcf= 6a99fa69aaefe477cc52e3ef4a7d2d7fa40714;branch=3Dmain" fetcher =3D bb.fetch.Fetch([url], self.d) fetcher.download() =20 # CLI11 that pulls in a newer nlohmann-json - url =3D "gitsm://github.com/CLIUtils/CLI11;protocol=3Dgit;rev=3D49= ac989a9527ee9bb496de9ded7b4872c2e0e5ca" + url =3D "gitsm://github.com/CLIUtils/CLI11;protocol=3Dgit;rev=3D49= ac989a9527ee9bb496de9ded7b4872c2e0e5ca;branch=3Dmain" fetcher =3D bb.fetch.Fetch([url], self.d) fetcher.download() # Previous cwd has been deleted @@ -1156,10 +1238,11 @@ class FetchLatestVersionTest(FetcherTest): ("mtd-utils", "git://git.yoctoproject.org/mtd-utils.git", "ca39eb1= d98e736109c64ff9c1aa2a6ecca222d8f", "") : "1.5.0", # version pattern "pkg_name-X.Y" - ("presentproto", "git://anongit.freedesktop.org/git/xorg/proto/pre= sentproto", "24f3a56e541b0a9e6c6ee76081f441221a120ef9", "") + # mirror of git://anongit.freedesktop.org/git/xorg/proto/presentpr= oto since network issues interfered with testing + ("presentproto", "git://git.yoctoproject.org/bbfetchtests-presentp= roto", "24f3a56e541b0a9e6c6ee76081f441221a120ef9", "") : "1.0", # version pattern "pkg_name-vX.Y.Z" - ("dtc", "git://git.qemu.org/dtc.git", "65cc4d2748a2c2e6f27f1cf39e0= 7a5dbabd80ebf", "") + ("dtc", "git://git.yoctoproject.org/bbfetchtests-dtc.git", "65cc4d= 2748a2c2e6f27f1cf39e07a5dbabd80ebf", "") : "1.4.0", # combination version pattern ("sysprof", "git://gitlab.gnome.org/GNOME/sysprof.git;protocol=3Dh= ttps", "cd44ee6644c3641507fb53b8a2a69137f2971219", "") @@ -1170,7 +1253,8 @@ class FetchLatestVersionTest(FetcherTest): ("mobile-broadband-provider-info", "git://gitlab.gnome.org/GNOME/m= obile-broadband-provider-info.git;protocol=3Dhttps", "4ed19e11c2975105b71b9= 56440acdb25d46a347d", "") : "20120614", # packages with a valid UPSTREAM_CHECK_GITTAGREGEX - ("xf86-video-omap", "git://anongit.freedesktop.org/xorg/driver/xf8= 6-video-omap", "ae0394e687f1a77e966cf72f895da91840dffb8f", "(?P(\d+\.= (\d\.?)*))") + # mirror of git://anongit.freedesktop.org/xorg/driver/xf86= -video-omap since network issues interfered with testing + ("xf86-video-omap", "git://git.yoctoproject.org/bbfetchtests-xf86-= video-omap", "ae0394e687f1a77e966cf72f895da91840dffb8f", "(?P(\d+\.(\= d\.?)*))") : "0.4.3", ("build-appliance-image", "git://git.yoctoproject.org/poky", "b37d= d451a52622d5b570183a81583cc34c2ff555", "(?P(([0-9][\.|_]?)+[0-9]))") : "11.0.0", @@ -1262,13 +1346,11 @@ class FetchLatestVersionTest(FetcherTest): =20 =20 class FetchCheckStatusTest(FetcherTest): - test_wget_uris =3D ["http://www.cups.org/software/1.7.2/cups-1.7.2-sou= rce.tar.bz2", - "http://www.cups.org/", - "http://downloads.yoctoproject.org/releases/sato/sat= o-engine-0.1.tar.gz", + test_wget_uris =3D ["http://downloads.yoctoproject.org/releases/sato/s= ato-engine-0.1.tar.gz", "http://downloads.yoctoproject.org/releases/sato/sat= o-engine-0.2.tar.gz", "http://downloads.yoctoproject.org/releases/sato/sat= o-engine-0.3.tar.gz", "https://yoctoproject.org/", - "https://yoctoproject.org/documentation", + "https://docs.yoctoproject.org", "http://downloads.yoctoproject.org/releases/opkg/opk= g-0.1.7.tar.gz", "http://downloads.yoctoproject.org/releases/opkg/opk= g-0.3.0.tar.gz", "ftp://sourceware.org/pub/libffi/libffi-1.20.tar.gz", @@ -1313,6 +1395,8 @@ class GitMakeShallowTest(FetcherTest): self.gitdir =3D os.path.join(self.tempdir, 'gitshallow') bb.utils.mkdirhier(self.gitdir) bb.process.run('git init', cwd=3Dself.gitdir) + bb.process.run('git config user.email "you@example.com"', cwd=3Dse= lf.gitdir) + bb.process.run('git config user.name "Your Name"', cwd=3Dself.gitd= ir) =20 def assertRefs(self, expected_refs): actual_refs =3D self.git(['for-each-ref', '--format=3D%(refname)']= ).splitlines() @@ -1436,6 +1520,8 @@ class GitShallowTest(FetcherTest): =20 bb.utils.mkdirhier(self.srcdir) self.git('init', cwd=3Dself.srcdir) + self.git('config user.email "you@example.com"', cwd=3Dself.srcdir) + self.git('config user.name "Your Name"', cwd=3Dself.srcdir) self.d.setVar('WORKDIR', self.tempdir) self.d.setVar('S', self.gitdir) self.d.delVar('PREMIRRORS') @@ -1517,6 +1603,7 @@ class GitShallowTest(FetcherTest): =20 # fetch and unpack, from the shallow tarball bb.utils.remove(self.gitdir, recurse=3DTrue) + bb.process.run('chmod u+w -R "%s"' % ud.clonedir) bb.utils.remove(ud.clonedir, recurse=3DTrue) bb.utils.remove(ud.clonedir.replace('gitsource', 'gitsubmodule'), = recurse=3DTrue) =20 @@ -1669,6 +1756,8 @@ class GitShallowTest(FetcherTest): smdir =3D os.path.join(self.tempdir, 'gitsubmodule') bb.utils.mkdirhier(smdir) self.git('init', cwd=3Dsmdir) + self.git('config user.email "you@example.com"', cwd=3Dsmdir) + self.git('config user.name "Your Name"', cwd=3Dsmdir) # Make this look like it was cloned from a remote... self.git('config --add remote.origin.url "%s"' % smdir, cwd=3Dsmdi= r) self.git('config --add remote.origin.fetch "+refs/heads/*:refs/rem= otes/origin/*"', cwd=3Dsmdir) @@ -1699,6 +1788,8 @@ class GitShallowTest(FetcherTest): smdir =3D os.path.join(self.tempdir, 'gitsubmodule') bb.utils.mkdirhier(smdir) self.git('init', cwd=3Dsmdir) + self.git('config user.email "you@example.com"', cwd=3Dsmdir) + self.git('config user.name "Your Name"', cwd=3Dsmdir) # Make this look like it was cloned from a remote... self.git('config --add remote.origin.url "%s"' % smdir, cwd=3Dsmdi= r) self.git('config --add remote.origin.fetch "+refs/heads/*:refs/rem= otes/origin/*"', cwd=3Dsmdir) @@ -1741,8 +1832,8 @@ class GitShallowTest(FetcherTest): self.git('annex init', cwd=3Dself.srcdir) open(os.path.join(self.srcdir, 'c'), 'w').close() self.git('annex add c', cwd=3Dself.srcdir) - self.git('commit -m annex-c -a', cwd=3Dself.srcdir) - bb.process.run('chmod u+w -R %s' % os.path.join(self.srcdir, '= .git', 'annex')) + self.git('commit --author "Foo Bar " -m annex-c -a', = cwd=3Dself.srcdir) + bb.process.run('chmod u+w -R %s' % self.srcdir) =20 uri =3D 'gitannex://%s;protocol=3Dfile;subdir=3D${S}' % self.s= rcdir fetcher, ud =3D self.fetch_shallow(uri) @@ -2017,6 +2108,8 @@ class GitLfsTest(FetcherTest): =20 bb.utils.mkdirhier(self.srcdir) self.git('init', cwd=3Dself.srcdir) + self.git('config user.email "you@example.com"', cwd=3Dself.srcdir) + self.git('config user.name "Your Name"', cwd=3Dself.srcdir) with open(os.path.join(self.srcdir, '.gitattributes'), 'wt') as at= trs: attrs.write('*.mp3 filter=3Dlfs -text') self.git(['add', '.gitattributes'], cwd=3Dself.srcdir) @@ -2031,13 +2124,14 @@ class GitLfsTest(FetcherTest): cwd =3D self.gitdir return bb.process.run(cmd, cwd=3Dcwd)[0] =20 - def fetch(self, uri=3DNone): + def fetch(self, uri=3DNone, download=3DTrue): uris =3D self.d.getVar('SRC_URI').split() uri =3D uris[0] d =3D self.d =20 fetcher =3D bb.fetch2.Fetch(uris, d) - fetcher.download() + if download: + fetcher.download() ud =3D fetcher.ud[uri] return fetcher, ud =20 @@ -2047,16 +2141,21 @@ class GitLfsTest(FetcherTest): uri =3D 'git://%s;protocol=3Dfile;subdir=3D${S};lfs=3D1' % self.sr= cdir self.d.setVar('SRC_URI', uri) =20 - fetcher, ud =3D self.fetch() + # Careful: suppress initial attempt at downloading until + # we know whether git-lfs is installed. + fetcher, ud =3D self.fetch(uri=3DNone, download=3DFalse) self.assertIsNotNone(ud.method._find_git_lfs) =20 - # If git-lfs can be found, the unpack should be successful - ud.method._find_git_lfs =3D lambda d: True - shutil.rmtree(self.gitdir, ignore_errors=3DTrue) - fetcher.unpack(self.d.getVar('WORKDIR')) + # If git-lfs can be found, the unpack should be successful. Only + # attempt this with the real live copy of git-lfs installed. + if ud.method._find_git_lfs(self.d): + fetcher.download() + shutil.rmtree(self.gitdir, ignore_errors=3DTrue) + fetcher.unpack(self.d.getVar('WORKDIR')) =20 # If git-lfs cannot be found, the unpack should throw an error with self.assertRaises(bb.fetch2.FetchError): + fetcher.download() ud.method._find_git_lfs =3D lambda d: False shutil.rmtree(self.gitdir, ignore_errors=3DTrue) fetcher.unpack(self.d.getVar('WORKDIR')) @@ -2067,10 +2166,16 @@ class GitLfsTest(FetcherTest): uri =3D 'git://%s;protocol=3Dfile;subdir=3D${S};lfs=3D0' % self.sr= cdir self.d.setVar('SRC_URI', uri) =20 + # In contrast to test_lfs_enabled(), allow the implicit download + # done by self.fetch() to occur here. The point of this test case + # is to verify that the fetcher can survive even if the source + # repository has Git LFS usage configured. fetcher, ud =3D self.fetch() self.assertIsNotNone(ud.method._find_git_lfs) =20 - # If git-lfs can be found, the unpack should be successful + # If git-lfs can be found, the unpack should be successful. A + # live copy of git-lfs is not required for this case, so + # unconditionally forge its presence. ud.method._find_git_lfs =3D lambda d: True shutil.rmtree(self.gitdir, ignore_errors=3DTrue) fetcher.unpack(self.d.getVar('WORKDIR')) @@ -2080,6 +2185,38 @@ class GitLfsTest(FetcherTest): shutil.rmtree(self.gitdir, ignore_errors=3DTrue) fetcher.unpack(self.d.getVar('WORKDIR')) =20 +class GitURLWithSpacesTest(FetcherTest): + test_git_urls =3D { + "git://tfs-example.org:22/tfs/example%20path/example.git" : { + 'url': 'git://tfs-example.org:22/tfs/example%20path/example.gi= t', + 'gitsrcname': 'tfs-example.org.22.tfs.example_path.example.git= ', + 'path': '/tfs/example path/example.git' + }, + "git://tfs-example.org:22/tfs/example%20path/example%20repo.git" := { + 'url': 'git://tfs-example.org:22/tfs/example%20path/example%20= repo.git', + 'gitsrcname': 'tfs-example.org.22.tfs.example_path.example_rep= o.git', + 'path': '/tfs/example path/example repo.git' + } + } + + def test_urls(self): + + # Set fake SRCREV to stop git fetcher from trying to contact non-e= xistent git repo + self.d.setVar('SRCREV', '82ea737a0b42a8b53e11c9cde141e9e9c0bd8c40') + + for test_git_url, ref in self.test_git_urls.items(): + + fetcher =3D bb.fetch.Fetch([test_git_url], self.d) + ud =3D fetcher.ud[fetcher.urls[0]] + + self.assertEqual(ud.url, ref['url']) + self.assertEqual(ud.path, ref['path']) + self.assertEqual(ud.localfile, os.path.join(self.dldir, "git2"= , ref['gitsrcname'])) + self.assertEqual(ud.localpath, os.path.join(self.dldir, "git2"= , ref['gitsrcname'])) + self.assertEqual(ud.lockfile, os.path.join(self.dldir, "git2",= ref['gitsrcname'] + '.lock')) + self.assertEqual(ud.clonedir, os.path.join(self.dldir, "git2",= ref['gitsrcname'])) + self.assertEqual(ud.fullmirror, os.path.join(self.dldir, "git2= _" + ref['gitsrcname'] + '.tar.gz')) + class NPMTest(FetcherTest): def skipIfNoNpm(): import shutil diff --git a/bitbake/lib/bb/tests/parse.py b/bitbake/lib/bb/tests/parse.py index 9afd1b20..9e21e184 100644 --- a/bitbake/lib/bb/tests/parse.py +++ b/bitbake/lib/bb/tests/parse.py @@ -178,7 +178,10 @@ python () { addtask do_patch after do_foo after do_unpack before do_configure before d= o_compile addtask do_fetch do_patch =20 -deltask do_fetch do_patch +MYVAR =3D "do_patch" +EMPTYVAR =3D "" +deltask do_fetch ${MYVAR} ${EMPTYVAR} +deltask ${EMPTYVAR} """ def test_parse_addtask_deltask(self): import sys @@ -189,6 +192,5 @@ deltask do_fetch do_patch self.assertTrue("addtask contained multiple 'before' keywords" in = stdout) self.assertTrue("addtask contained multiple 'after' keywords" in s= tdout) self.assertTrue('addtask ignored: " do_patch"' in stdout) - self.assertTrue('deltask ignored: " do_patch"' in stdout) #self.assertTrue('dependent task do_foo for do_patch does not exis= t' in stdout) =20 diff --git a/bitbake/lib/bb/tests/runqueue-tests/conf/bitbake.conf b/bitbak= e/lib/bb/tests/runqueue-tests/conf/bitbake.conf index 5e451fc2..efebf001 100644 --- a/bitbake/lib/bb/tests/runqueue-tests/conf/bitbake.conf +++ b/bitbake/lib/bb/tests/runqueue-tests/conf/bitbake.conf @@ -1,7 +1,8 @@ CACHE =3D "${TOPDIR}/cache" THISDIR =3D "${@os.path.dirname(d.getVar('FILE'))}" COREBASE :=3D "${@os.path.normpath(os.path.dirname(d.getVar('FILE')+'/../.= ./'))}" -BBFILES =3D "${COREBASE}/recipes/*.bb" +EXTRA_BBFILES ?=3D "" +BBFILES =3D "${COREBASE}/recipes/*.bb ${EXTRA_BBFILES}" PROVIDES =3D "${PN}" PN =3D "${@bb.parse.vars_from_file(d.getVar('FILE', False),d)[0]}" PF =3D "${BB_CURRENT_MC}:${PN}" diff --git a/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc-1.conf= b/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc-1.conf new file mode 100644 index 00000000..f34b8dcc --- /dev/null +++ b/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc-1.conf @@ -0,0 +1,2 @@ +TMPDIR =3D "${TOPDIR}/mc1/" +BBMASK +=3D "recipes/fails-mc/fails-mc1.bb" diff --git a/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc1.conf = b/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc1.conf deleted file mode 100644 index ecf23e1c..00000000 --- a/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc1.conf +++ /dev/null @@ -1 +0,0 @@ -TMPDIR =3D "${TOPDIR}/mc1/" diff --git a/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc2.conf = b/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc2.conf deleted file mode 100644 index eef338e4..00000000 --- a/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc2.conf +++ /dev/null @@ -1 +0,0 @@ -TMPDIR =3D "${TOPDIR}/mc2/" diff --git a/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc_2.conf= b/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc_2.conf new file mode 100644 index 00000000..c3360fc5 --- /dev/null +++ b/bitbake/lib/bb/tests/runqueue-tests/conf/multiconfig/mc_2.conf @@ -0,0 +1,2 @@ +TMPDIR =3D "${TOPDIR}/mc2/" +BBMASK +=3D "recipes/fails-mc/fails-mc2.bb" diff --git a/bitbake/lib/bb/tests/runqueue-tests/recipes/f1.bb b/bitbake/li= b/bb/tests/runqueue-tests/recipes/f1.bb new file mode 100644 index 00000000..7b8fc592 --- /dev/null +++ b/bitbake/lib/bb/tests/runqueue-tests/recipes/f1.bb @@ -0,0 +1 @@ +do_install[mcdepends] =3D "mc:mc-1:mc_2:a1:do_build" diff --git a/bitbake/lib/bb/tests/runqueue-tests/recipes/fails-mc/fails-mc1= .bb b/bitbake/lib/bb/tests/runqueue-tests/recipes/fails-mc/fails-mc1.bb new file mode 100644 index 00000000..eed69c80 --- /dev/null +++ b/bitbake/lib/bb/tests/runqueue-tests/recipes/fails-mc/fails-mc1.bb @@ -0,0 +1,5 @@ +python () { + if d.getVar("BB_CURRENT_MC") =3D=3D "mc-1": + bb.fatal("Multiconfig is mc-1") +} + diff --git a/bitbake/lib/bb/tests/runqueue-tests/recipes/fails-mc/fails-mc2= .bb b/bitbake/lib/bb/tests/runqueue-tests/recipes/fails-mc/fails-mc2.bb new file mode 100644 index 00000000..3c172ef9 --- /dev/null +++ b/bitbake/lib/bb/tests/runqueue-tests/recipes/fails-mc/fails-mc2.bb @@ -0,0 +1,4 @@ +python () { + if d.getVar("BB_CURRENT_MC") =3D=3D "mc_2": + bb.fatal("Multiconfig is mc_2") +} diff --git a/bitbake/lib/bb/tests/runqueue.py b/bitbake/lib/bb/tests/runque= ue.py index 4ba12a07..4f335b8f 100644 --- a/bitbake/lib/bb/tests/runqueue.py +++ b/bitbake/lib/bb/tests/runqueue.py @@ -216,22 +216,67 @@ class RunQueueTests(unittest.TestCase): def test_multiconfig_setscene_optimise(self): with tempfile.TemporaryDirectory(prefix=3D"runqueuetest") as tempd= ir: extraenv =3D { - "BBMULTICONFIG" : "mc1 mc2", + "BBMULTICONFIG" : "mc-1 mc_2", "BB_SIGNATURE_HANDLER" : "basic" } - cmd =3D ["bitbake", "b1", "mc:mc1:b1", "mc:mc2:b1"] + cmd =3D ["bitbake", "b1", "mc:mc-1:b1", "mc:mc_2:b1"] setscenetasks =3D ['package_write_ipk_setscene', 'package_writ= e_rpm_setscene', 'packagedata_setscene', 'populate_sysroot_setscene', 'package_qa_sets= cene'] sstatevalid =3D "" tasks =3D self.run_bitbakecmd(cmd, tempdir, sstatevalid, extra= env=3Dextraenv) expected =3D ['a1:' + x for x in self.alltasks] + ['b1:' + x f= or x in self.alltasks] + \ - ['mc1:b1:' + x for x in setscenetasks] + ['mc1:a1:'= + x for x in setscenetasks] + \ - ['mc2:b1:' + x for x in setscenetasks] + ['mc2:a1:'= + x for x in setscenetasks] + \ - ['mc1:b1:build', 'mc2:b1:build'] - for x in ['mc1:a1:package_qa_setscene', 'mc2:a1:package_qa_set= scene', 'a1:build', 'a1:package_qa']: + ['mc-1:b1:' + x for x in setscenetasks] + ['mc-1:a1= :' + x for x in setscenetasks] + \ + ['mc_2:b1:' + x for x in setscenetasks] + ['mc_2:a1= :' + x for x in setscenetasks] + \ + ['mc-1:b1:build', 'mc_2:b1:build'] + for x in ['mc-1:a1:package_qa_setscene', 'mc_2:a1:package_qa_s= etscene', 'a1:build', 'a1:package_qa']: expected.remove(x) self.assertEqual(set(tasks), set(expected)) =20 + def test_multiconfig_bbmask(self): + # This test validates that multiconfigs can independently mask off + # recipes they do not want with BBMASK. It works by having recipes + # that will fail to parse for mc-1 and mc_2, then making each mult= iconfig + # build the one that does parse. This ensures that the recipes are= in + # each multiconfigs BBFILES, but each is masking only the one that + # doesn't parse + with tempfile.TemporaryDirectory(prefix=3D"runqueuetest") as tempd= ir: + extraenv =3D { + "BBMULTICONFIG" : "mc-1 mc_2", + "BB_SIGNATURE_HANDLER" : "basic", + "EXTRA_BBFILES": "${COREBASE}/recipes/fails-mc/*.bb", + } + cmd =3D ["bitbake", "mc:mc-1:fails-mc2", "mc:mc_2:fails-mc1"] + self.run_bitbakecmd(cmd, tempdir, "", extraenv=3Dextraenv) + + def test_multiconfig_mcdepends(self): + with tempfile.TemporaryDirectory(prefix=3D"runqueuetest") as tempd= ir: + extraenv =3D { + "BBMULTICONFIG" : "mc-1 mc_2", + "BB_SIGNATURE_HANDLER" : "TestMulticonfigDepends", + "EXTRA_BBFILES": "${COREBASE}/recipes/fails-mc/*.bb", + } + tasks =3D self.run_bitbakecmd(["bitbake", "mc:mc-1:f1"], tempd= ir, "", extraenv=3Dextraenv, cleanup=3DTrue) + expected =3D ["mc-1:f1:%s" % t for t in self.alltasks] + \ + ["mc_2:a1:%s" % t for t in self.alltasks] + self.assertEqual(set(tasks), set(expected)) + + # A rebuild does nothing + tasks =3D self.run_bitbakecmd(["bitbake", "mc:mc-1:f1"], tempd= ir, "", extraenv=3Dextraenv, cleanup=3DTrue) + self.assertEqual(set(tasks), set()) + + # Test that a signature change in the dependent task causes + # mcdepends to rebuild + tasks =3D self.run_bitbakecmd(["bitbake", "mc:mc_2:a1", "-c", = "compile", "-f"], tempdir, "", extraenv=3Dextraenv, cleanup=3DTrue) + expected =3D ["mc_2:a1:compile"] + self.assertEqual(set(tasks), set(expected)) + + rerun_tasks =3D self.alltasks[:] + for x in ("fetch", "unpack", "patch", "prepare_recipe_sysroot"= , "configure", "compile"): + rerun_tasks.remove(x) + tasks =3D self.run_bitbakecmd(["bitbake", "mc:mc-1:f1"], tempd= ir, "", extraenv=3Dextraenv, cleanup=3DTrue) + expected =3D ["mc-1:f1:%s" % t for t in rerun_tasks] + \ + ["mc_2:a1:%s" % t for t in rerun_tasks] + self.assertEqual(set(tasks), set(expected)) =20 @unittest.skipIf(sys.version_info < (3, 5, 0), 'Python 3.5 or later re= quired') def test_hashserv_single(self): @@ -316,7 +361,7 @@ class RunQueueTests(unittest.TestCase): =20 def shutdown(self, tempdir): # Wait for the hashserve socket to disappear else we'll see races = with the tempdir cleanup - while os.path.exists(tempdir + "/hashserve.sock"): + while (os.path.exists(tempdir + "/hashserve.sock") or os.path.exis= ts(tempdir + "cache/hashserv.db-wal")): time.sleep(0.5) =20 =20 diff --git a/bitbake/lib/bb/tests/siggen.py b/bitbake/lib/bb/tests/siggen.py new file mode 100644 index 00000000..c21ab4e4 --- /dev/null +++ b/bitbake/lib/bb/tests/siggen.py @@ -0,0 +1,91 @@ +# +# BitBake Test for lib/bb/siggen.py +# +# Copyright (C) 2020 Jean-Fran=C3=A7ois Dagenais +# +# SPDX-License-Identifier: GPL-2.0-only +# + +import unittest +import logging +import bb +import time + +logger =3D logging.getLogger('BitBake.TestSiggen') + +import bb.siggen + +class SiggenTest(unittest.TestCase): + + def test_clean_basepath_simple_target_basepath(self): + basepath =3D '/full/path/to/poky/meta/recipes-whatever/helloworld/= helloworld_1.2.3.bb:do_sometask' + expected_cleaned =3D 'helloworld/helloworld_1.2.3.bb:do_sometask' + + actual_cleaned =3D bb.siggen.clean_basepath(basepath) + + self.assertEqual(actual_cleaned, expected_cleaned) + + def test_clean_basepath_basic_virtual_basepath(self): + basepath =3D 'virtual:something:/full/path/to/poky/meta/recipes-wh= atever/helloworld/helloworld_1.2.3.bb:do_sometask' + expected_cleaned =3D 'helloworld/helloworld_1.2.3.bb:do_sometask:v= irtual:something' + + actual_cleaned =3D bb.siggen.clean_basepath(basepath) + + self.assertEqual(actual_cleaned, expected_cleaned) + + def test_clean_basepath_mc_basepath(self): + basepath =3D 'mc:somemachine:/full/path/to/poky/meta/recipes-whate= ver/helloworld/helloworld_1.2.3.bb:do_sometask' + expected_cleaned =3D 'helloworld/helloworld_1.2.3.bb:do_sometask:m= c:somemachine' + + actual_cleaned =3D bb.siggen.clean_basepath(basepath) + + self.assertEqual(actual_cleaned, expected_cleaned) + + def test_clean_basepath_virtual_long_prefix_basepath(self): + basepath =3D 'virtual:something:A:B:C:/full/path/to/poky/meta/reci= pes-whatever/helloworld/helloworld_1.2.3.bb:do_sometask' + expected_cleaned =3D 'helloworld/helloworld_1.2.3.bb:do_sometask:v= irtual:something:A:B:C' + + actual_cleaned =3D bb.siggen.clean_basepath(basepath) + + self.assertEqual(actual_cleaned, expected_cleaned) + + def test_clean_basepath_mc_virtual_basepath(self): + basepath =3D 'mc:somemachine:virtual:something:/full/path/to/poky/= meta/recipes-whatever/helloworld/helloworld_1.2.3.bb:do_sometask' + expected_cleaned =3D 'helloworld/helloworld_1.2.3.bb:do_sometask:v= irtual:something:mc:somemachine' + + actual_cleaned =3D bb.siggen.clean_basepath(basepath) + + self.assertEqual(actual_cleaned, expected_cleaned) + + def test_clean_basepath_mc_virtual_long_prefix_basepath(self): + basepath =3D 'mc:X:virtual:something:C:B:A:/full/path/to/poky/meta= /recipes-whatever/helloworld/helloworld_1.2.3.bb:do_sometask' + expected_cleaned =3D 'helloworld/helloworld_1.2.3.bb:do_sometask:v= irtual:something:C:B:A:mc:X' + + actual_cleaned =3D bb.siggen.clean_basepath(basepath) + + self.assertEqual(actual_cleaned, expected_cleaned) + + + # def test_clean_basepath_performance(self): + # input_basepaths =3D [ + # 'mc:X:/full/path/to/poky/meta/recipes-whatever/helloworld/he= lloworld_1.2.3.bb:do_sometask', + # 'mc:X:virtual:something:C:B:A:/full/path/to/poky/meta/recipe= s-whatever/helloworld/helloworld_1.2.3.bb:do_sometask', + # 'virtual:something:C:B:A:/different/path/to/poky/meta/recipe= s-whatever/helloworld/helloworld_1.2.3.bb:do_sometask', + # 'virtual:something:A:/full/path/to/poky/meta/recipes-whateve= r/helloworld/helloworld_1.2.3.bb:do_sometask', + # '/this/is/most/common/input/recipes-whatever/helloworld/hell= oworld_1.2.3.bb:do_sometask', + # '/and/should/be/tested/with/recipes-whatever/helloworld/hell= oworld_1.2.3.bb:do_sometask', + # '/more/weight/recipes-whatever/helloworld/helloworld_1.2.3.b= b:do_sometask', + # ] + + # time_start =3D time.time() + + # i =3D 2000000 + # while i >=3D 0: + # for basepath in input_basepaths: + # bb.siggen.clean_basepath(basepath) + # i -=3D 1 + + # elapsed =3D time.time() - time_start + # print('{} ({}s)'.format(self.id(), round(elapsed, 3))) + + # self.assertTrue(False) diff --git a/bitbake/lib/bb/tinfoil.py b/bitbake/lib/bb/tinfoil.py index 8c9b6b8c..796a98f0 100644 --- a/bitbake/lib/bb/tinfoil.py +++ b/bitbake/lib/bb/tinfoil.py @@ -22,7 +22,6 @@ import bb.taskdata import bb.utils import bb.command import bb.remotedata -from bb.cookerdata import CookerConfiguration from bb.main import setup_bitbake, BitBakeConfigParameters import bb.fetch2 =20 @@ -117,15 +116,16 @@ class TinfoilCookerAdapter: =20 class TinfoilCookerCollectionAdapter: """ cooker.collection adapter """ - def __init__(self, tinfoil): + def __init__(self, tinfoil, mc=3D''): self.tinfoil =3D tinfoil + self.mc =3D mc def get_file_appends(self, fn): - return self.tinfoil.get_file_appends(fn) + return self.tinfoil.get_file_appends(fn, self.mc) def __getattr__(self, name): if name =3D=3D 'overlayed': - return self.tinfoil.get_overlayed_recipes() + return self.tinfoil.get_overlayed_recipes(self.mc) elif name =3D=3D 'bbappends': - return self.tinfoil.run_command('getAllAppends') + return self.tinfoil.run_command('getAllAppends', self.mc) else: raise AttributeError("%s instance has no attribute '%s'" %= (self.__class__.__name__, name)) =20 @@ -185,10 +185,11 @@ class TinfoilCookerAdapter: =20 def __init__(self, tinfoil): self.tinfoil =3D tinfoil - self.collection =3D self.TinfoilCookerCollectionAdapter(tinfoil) + self.multiconfigs =3D [''] + (tinfoil.config_data.getVar('BBMULTIC= ONFIG') or '').split() + self.collections =3D {} self.recipecaches =3D {} - self.recipecaches[''] =3D self.TinfoilRecipeCacheAdapter(tinfoil) - for mc in (tinfoil.config_data.getVar('BBMULTICONFIG') or '').spli= t(): + for mc in self.multiconfigs: + self.collections[mc] =3D self.TinfoilCookerCollectionAdapter(t= infoil, mc) self.recipecaches[mc] =3D self.TinfoilRecipeCacheAdapter(tinfo= il, mc) self._cache =3D {} def __getattr__(self, name): @@ -379,18 +380,13 @@ class Tinfoil: if not config_params: config_params =3D TinfoilConfigParameters(config_only=3Dconfig= _only, quiet=3Dquiet) =20 - cookerconfig =3D CookerConfiguration() - cookerconfig.setConfigParameters(config_params) - if not config_only: # Disable local loggers because the UI module is going to set = up its own for handler in self.localhandlers: self.logger.handlers.remove(handler) self.localhandlers =3D [] =20 - self.server_connection, ui_module =3D setup_bitbake(config_params, - cookerconfig, - extrafeatures) + self.server_connection, ui_module =3D setup_bitbake(config_params,= extrafeatures) =20 self.ui_module =3D ui_module =20 @@ -444,7 +440,7 @@ class Tinfoil: to initialise Tinfoil and use it with config_only=3DTrue first and then conditionally call this function to parse recipes later. """ - config_params =3D TinfoilConfigParameters(config_only=3DFalse) + config_params =3D TinfoilConfigParameters(config_only=3DFalse, qui= et=3Dself.quiet) self.run_actions(config_params) self.recipes_parsed =3D True =20 @@ -465,7 +461,16 @@ class Tinfoil: commandline =3D [command] if params: commandline.extend(params) - result =3D self.server_connection.connection.runCommand(commandlin= e) + try: + result =3D self.server_connection.connection.runCommand(comman= dline) + finally: + while True: + event =3D self.wait_event() + if not event: + break + if isinstance(event, logging.LogRecord): + if event.taskpid =3D=3D 0 or event.levelno > logging.I= NFO: + self.logger.handle(event) if result[1]: raise TinfoilCommandFailed(result[1]) return result[0] @@ -492,11 +497,11 @@ class Tinfoil: raise Exception('Not connected to server (did you call .prepar= e()?)') return self.server_connection.events.waitEvent(timeout) =20 - def get_overlayed_recipes(self): + def get_overlayed_recipes(self, mc=3D''): """ Find recipes which are overlayed (i.e. where recipes exist in mult= iple layers) """ - return defaultdict(list, self.run_command('getOverlayedRecipes')) + return defaultdict(list, self.run_command('getOverlayedRecipes', m= c)) =20 def get_skipped_recipes(self): """ @@ -534,11 +539,11 @@ class Tinfoil: raise bb.providers.NoProvider('Unable to find any recipe f= ile matching "%s"' % pn) return best[3] =20 - def get_file_appends(self, fn): + def get_file_appends(self, fn, mc=3D''): """ Find the bbappends for a recipe file """ - return self.run_command('getFileAppends', fn) + return self.run_command('getFileAppends', fn, mc) =20 def all_recipes(self, mc=3D'', sort=3DTrue): """ @@ -736,7 +741,7 @@ class Tinfoil: continue if helper.eventHandler(event): if isinstance(event, bb.build.TaskFailedSi= lent): - logger.warning("Logfile for failed set= scene task is %s" % event.logfile) + self.logger.warning("Logfile for faile= d setscene task is %s" % event.logfile) elif isinstance(event, bb.build.TaskFailed= ): bb.ui.knotty.print_event_log(event, in= cludelogs, loglines, termfilter) continue @@ -810,18 +815,22 @@ class Tinfoil: prepare() has been called, or use a with... block when you create the tinfoil object which will ensure that it gets called. """ - if self.server_connection: - self.run_command('clientComplete') - _server_connections.remove(self.server_connection) - bb.event.ui_queue =3D [] - self.server_connection.terminate() - self.server_connection =3D None - - # Restore logging handlers to how it looked when we started - if self.oldhandlers: - for handler in self.logger.handlers: - if handler not in self.oldhandlers: - self.logger.handlers.remove(handler) + try: + if self.server_connection: + try: + self.run_command('clientComplete') + finally: + _server_connections.remove(self.server_connection) + bb.event.ui_queue =3D [] + self.server_connection.terminate() + self.server_connection =3D None + + finally: + # Restore logging handlers to how it looked when we started + if self.oldhandlers: + for handler in self.logger.handlers: + if handler not in self.oldhandlers: + self.logger.handlers.remove(handler) =20 def _reconvert_type(self, obj, origtypename): """ diff --git a/bitbake/lib/bb/ui/buildinfohelper.py b/bitbake/lib/bb/ui/build= infohelper.py index 82c62e33..43aa5928 100644 --- a/bitbake/lib/bb/ui/buildinfohelper.py +++ b/bitbake/lib/bb/ui/buildinfohelper.py @@ -148,14 +148,14 @@ class ORMWrapper(object): buildrequest =3D None if brbe is not None: # Toaster-triggered build - logger.debug(1, "buildinfohelper: brbe is %s" % brbe) + logger.debug("buildinfohelper: brbe is %s" % brbe) br, _ =3D brbe.split(":") buildrequest =3D BuildRequest.objects.get(pk=3Dbr) prj =3D buildrequest.project else: # CLI build prj =3D Project.objects.get_or_create_default_project() - logger.debug(1, "buildinfohelper: project is not specified, de= faulting to %s" % prj) + logger.debug("buildinfohelper: project is not specified, defau= lting to %s" % prj) =20 if buildrequest is not None: # reuse existing Build object @@ -171,7 +171,7 @@ class ORMWrapper(object): completed_on=3Dnow, build_name=3D'') =20 - logger.debug(1, "buildinfohelper: build is created %s" % build) + logger.debug("buildinfohelper: build is created %s" % build) =20 if buildrequest is not None: buildrequest.build =3D build @@ -906,7 +906,7 @@ class BuildInfoHelper(object): =20 self.project =3D None =20 - logger.debug(1, "buildinfohelper: Build info helper inited %s" % v= ars(self)) + logger.debug("buildinfohelper: Build info helper inited %s" % vars= (self)) =20 =20 ################### @@ -1620,7 +1620,7 @@ class BuildInfoHelper(object): # if we have a backlog of events, do our best to save them here if len(self.internal_state['backlog']): tempevent =3D self.internal_state['backlog'].pop() - logger.debug(1, "buildinfohelper: Saving stored event %s " + logger.debug("buildinfohelper: Saving stored event %s " % tempevent) self.store_log_event(tempevent,cli_backlog) else: diff --git a/bitbake/lib/bb/ui/knotty.py b/bitbake/lib/bb/ui/knotty.py index 87e873d6..04285eac 100644 --- a/bitbake/lib/bb/ui/knotty.py +++ b/bitbake/lib/bb/ui/knotty.py @@ -144,7 +144,7 @@ class TerminalFilter(object): pass if not cr: try: - cr =3D (env['LINES'], env['COLUMNS']) + cr =3D (os.environ['LINES'], os.environ['COLUMNS']) except: cr =3D (25, 80) return cr @@ -380,14 +380,27 @@ _evt_list =3D [ "bb.runqueue.runQueueExitWait", "bb.e= vent.LogExecTTY", "logging.Lo "bb.event.BuildBase", "bb.build.TaskStarted", "bb.build.Task= Succeeded", "bb.build.TaskFailedSilent", "bb.build.TaskProgress", "bb.event.ProcessStarted", "bb.even= t.ProcessProgress", "bb.event.ProcessFinished"] =20 +def drain_events_errorhandling(eventHandler): + # We don't have logging setup, we do need to show any events we see be= fore exiting + event =3D True + logger =3D bb.msg.logger_create('bitbake', sys.stdout) + while event: + event =3D eventHandler.waitEvent(0) + if isinstance(event, logging.LogRecord): + logger.handle(event) + def main(server, eventHandler, params, tf =3D TerminalFilter): =20 - if not params.observe_only: - params.updateToServer(server, os.environ.copy()) + try: + if not params.observe_only: + params.updateToServer(server, os.environ.copy()) =20 - includelogs, loglines, consolelogfile, logconfigfile =3D _log_settings= _from_server(server, params.observe_only) + includelogs, loglines, consolelogfile, logconfigfile =3D _log_sett= ings_from_server(server, params.observe_only) =20 - loglevel, _ =3D bb.msg.constructLogOptions() + loglevel, _ =3D bb.msg.constructLogOptions() + except bb.BBHandledException: + drain_events_errorhandling(eventHandler) + return 1 =20 if params.options.quiet =3D=3D 0: console_loglevel =3D loglevel @@ -679,7 +692,7 @@ def main(server, eventHandler, params, tf =3D TerminalF= ilter): if not parseprogress: continue parseprogress.finish() - pasreprogress =3D None + parseprogress =3D None if params.options.quiet =3D=3D 0: print(("Parsing of %d .bb files complete (%d cached, %= d parsed). %d targets, %d skipped, %d masked, %d errors." % ( event.total, event.cached, event.parsed, event= .virtuals, event.skipped, event.masked, event.errors))) @@ -732,7 +745,7 @@ def main(server, eventHandler, params, tf =3D TerminalF= ilter): continue =20 if isinstance(event, bb.runqueue.sceneQueueTaskStarted): - logger.info("Running setscene task %d of %d (%s)" % (event= .stats.completed + event.stats.active + event.stats.failed + 1, event.stats= .total, event.taskstring)) + logger.info("Running setscene task %d of %d (%s)" % (event= .stats.setscene_covered + event.stats.setscene_active + event.stats.setscen= e_notcovered + 1, event.stats.setscene_total, event.taskstring)) continue =20 if isinstance(event, bb.runqueue.runQueueTaskStarted): diff --git a/bitbake/lib/bb/ui/ncurses.py b/bitbake/lib/bb/ui/ncurses.py index da4fbeab..cf1c876a 100644 --- a/bitbake/lib/bb/ui/ncurses.py +++ b/bitbake/lib/bb/ui/ncurses.py @@ -48,6 +48,8 @@ import bb import xmlrpc.client from bb.ui import uihelper =20 +logger =3D logging.getLogger(__name__) + parsespin =3D itertools.cycle( r'|/-\\' ) =20 X =3D 0 diff --git a/bitbake/lib/bb/ui/taskexp.py b/bitbake/lib/bb/ui/taskexp.py index 05e32338..2b246710 100644 --- a/bitbake/lib/bb/ui/taskexp.py +++ b/bitbake/lib/bb/ui/taskexp.py @@ -58,7 +58,12 @@ class PackageReverseDepView(Gtk.TreeView): self.current =3D None self.filter_model =3D model.filter_new() self.filter_model.set_visible_func(self._filter) - self.sort_model =3D self.filter_model.sort_new_with_model() + # The introspected API was fixed but we can't rely on a pygobject = that hides this. + # https://gitlab.gnome.org/GNOME/pygobject/-/commit/9cdbc56fbac4db= 2de78dc080934b8f0a7efc892a + if hasattr(Gtk.TreeModelSort, "new_with_model"): + self.sort_model =3D Gtk.TreeModelSort.new_with_model(self.filt= er_model) + else: + self.sort_model =3D self.filter_model.sort_new_with_model() self.sort_model.set_sort_column_id(COL_DEP_PARENT, Gtk.SortType.AS= CENDING) self.set_model(self.sort_model) self.append_column(Gtk.TreeViewColumn(label, Gtk.CellRendererText(= ), text=3DCOL_DEP_PARENT)) diff --git a/bitbake/lib/bb/ui/toasterui.py b/bitbake/lib/bb/ui/toasterui.py index 9260f5d9..ec5bd4f1 100644 --- a/bitbake/lib/bb/ui/toasterui.py +++ b/bitbake/lib/bb/ui/toasterui.py @@ -131,6 +131,10 @@ def main(server, eventHandler, params): =20 helper =3D uihelper.BBUIHelper() =20 + if not params.observe_only: + params.updateToServer(server, os.environ.copy()) + params.updateFromServer(server) + # TODO don't use log output to determine when bitbake has started # # WARNING: this log handler cannot be removed, as localhostbecontroller @@ -162,8 +166,6 @@ def main(server, eventHandler, params): logger.warning("buildstats is not enabled. Please enable INHERIT += =3D \"buildstats\" to generate build statistics.") =20 if not params.observe_only: - params.updateFromServer(server) - params.updateToServer(server, os.environ.copy()) cmdline =3D params.parseActions() if not cmdline: print("Nothing to do. Use 'bitbake world' to build everything= , or run 'bitbake --help' for usage information.") diff --git a/bitbake/lib/bb/ui/uievent.py b/bitbake/lib/bb/ui/uievent.py index 13d0d4a0..8607d052 100644 --- a/bitbake/lib/bb/ui/uievent.py +++ b/bitbake/lib/bb/ui/uievent.py @@ -11,9 +11,13 @@ server and queue them for the UI to process. This proces= s must be used to avoid client/server deadlocks. """ =20 -import socket, threading, pickle, collections +import collections, logging, pickle, socket, threading from xmlrpc.server import SimpleXMLRPCServer, SimpleXMLRPCRequestHandler =20 +import bb + +logger =3D logging.getLogger(__name__) + class BBUIEventQueue: def __init__(self, BBServer, clientinfo=3D("localhost, 0")): =20 diff --git a/bitbake/lib/bb/ui/uihelper.py b/bitbake/lib/bb/ui/uihelper.py index 48d808ae..52fdae3f 100644 --- a/bitbake/lib/bb/ui/uihelper.py +++ b/bitbake/lib/bb/ui/uihelper.py @@ -49,8 +49,8 @@ class BBUIHelper: tid =3D event._fn + ":" + event._task removetid(event.pid, tid) self.failed_tasks.append( { 'title' : "%s %s" % (event._packag= e, event._task)}) - elif isinstance(event, bb.runqueue.runQueueTaskStarted): - self.tasknumber_current =3D event.stats.completed + event.stat= s.active + event.stats.failed + 1 + elif isinstance(event, bb.runqueue.runQueueTaskStarted) or isinsta= nce(event, bb.runqueue.sceneQueueTaskStarted): + self.tasknumber_current =3D event.stats.completed + event.stat= s.active + event.stats.failed + event.stats.setscene_active + 1 self.tasknumber_total =3D event.stats.total self.needUpdate =3D True elif isinstance(event, bb.build.TaskProgress): diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py index 5f5767c1..b282d09a 100644 --- a/bitbake/lib/bb/utils.py +++ b/bitbake/lib/bb/utils.py @@ -129,6 +129,7 @@ def vercmp(ta, tb): return r =20 def vercmp_string(a, b): + """ Split version strings and compare them """ ta =3D split_version(a) tb =3D split_version(b) return vercmp(ta, tb) @@ -247,6 +248,12 @@ def explode_dep_versions2(s, *, sort=3DTrue): return r =20 def explode_dep_versions(s): + """ + Take an RDEPENDS style string of format: + "DEPEND1 (optional version) DEPEND2 (optional version) ..." + skip null value and items appeared in dependancy string multiple times + and return a dictionary of dependencies and versions. + """ r =3D explode_dep_versions2(s) for d in r: if not r[d]: @@ -402,8 +409,8 @@ def better_exec(code, context, text =3D None, realfile = =3D "", pythonexception (t, value, tb) =3D sys.exc_info() try: _print_exception(t, value, tb, realfile, text, context) - except Exception as e: - logger.error("Exception handler error: %s" % str(e)) + except Exception as e2: + logger.error("Exception handler error: %s" % str(e2)) =20 e =3D bb.BBHandledException(e) raise e @@ -433,20 +440,6 @@ def fileslocked(files): for lock in locks: bb.utils.unlockfile(lock) =20 -@contextmanager -def timeout(seconds): - def timeout_handler(signum, frame): - pass - - original_handler =3D signal.signal(signal.SIGALRM, timeout_handler) - - try: - signal.alarm(seconds) - yield - finally: - signal.alarm(0) - signal.signal(signal.SIGALRM, original_handler) - def lockfile(name, shared=3DFalse, retry=3DTrue, block=3DFalse): """ Use the specified file as a lock file, return when the lock has @@ -580,7 +573,6 @@ def preserved_envvars_exported(): 'PATH', 'PWD', 'SHELL', - 'TERM', 'USER', 'LC_ALL', 'BBSERVER', @@ -617,7 +609,7 @@ def filter_environment(good_vars): os.environ["LC_ALL"] =3D "en_US.UTF-8" =20 if removed_vars: - logger.debug(1, "Removed the following variables from the environm= ent: %s", ", ".join(removed_vars.keys())) + logger.debug("Removed the following variables from the environment= : %s", ", ".join(removed_vars.keys())) =20 return removed_vars =20 @@ -707,7 +699,7 @@ def remove(path, recurse=3DFalse, ionice=3DFalse): raise =20 def prunedir(topdir, ionice=3DFalse): - # Delete everything reachable from the directory named in 'topdir'. + """ Delete everything reachable from the directory named in 'topdir'. = """ # CAUTION: This is dangerous! if _check_unsafe_delete_path(topdir): raise Exception('bb.utils.prunedir: called with dangerous path "%s= ", refusing to delete!' % topdir) @@ -718,8 +710,10 @@ def prunedir(topdir, ionice=3DFalse): # but thats possibly insane and suffixes is probably going to be small # def prune_suffix(var, suffixes, d): - # See if var ends with any of the suffixes listed and - # remove it if found + """=20 + See if var ends with any of the suffixes listed and + remove it if found=20 + """ for suffix in suffixes: if suffix and var.endswith(suffix): return var[:-len(suffix)] @@ -959,7 +953,22 @@ def which(path, item, direction =3D 0, history =3D Fal= se, executable=3DFalse): return "", hist return "" =20 +@contextmanager +def umask(new_mask): + """ + Context manager to set the umask to a specific mask, and restore it af= terwards. + """ + current_mask =3D os.umask(new_mask) + try: + yield + finally: + os.umask(current_mask) + def to_boolean(string, default=3DNone): + """=20 + Check input string and return boolean value True/False/None + depending upon the checks=20 + """ if not string: return default =20 @@ -1003,6 +1012,23 @@ def contains(variable, checkvalues, truevalue, false= value, d): return falsevalue =20 def contains_any(variable, checkvalues, truevalue, falsevalue, d): + """Check if a variable contains any values specified. + + Arguments: + + variable -- the variable name. This will be fetched and expanded (using + d.getVar(variable)) and then split into a set(). + + checkvalues -- if this is a string it is split on whitespace into a se= t(), + otherwise coerced directly into a set(). + + truevalue -- the value to return if checkvalues is a subset of variabl= e. + + falsevalue -- the value to return if variable is empty or if checkvalu= es is + not a subset of variable. + + d -- the data store. + """ val =3D d.getVar(variable) if not val: return falsevalue @@ -1086,21 +1112,20 @@ def process_profilelog(fn, pout =3D None): # Either call with a list of filenames and set pout or a filename and = optionally pout. if not pout: pout =3D fn + '.processed' - pout =3D open(pout, 'w') -=20=20=20 - import pstats - if isinstance(fn, list): - p =3D pstats.Stats(*fn, stream=3Dpout) - else: - p =3D pstats.Stats(fn, stream=3Dpout) - p.sort_stats('time') - p.print_stats() - p.print_callers() - p.sort_stats('cumulative') - p.print_stats() =20 - pout.flush() - pout.close()=20=20 + with open(pout, 'w') as pout: + import pstats + if isinstance(fn, list): + p =3D pstats.Stats(*fn, stream=3Dpout) + else: + p =3D pstats.Stats(fn, stream=3Dpout) + p.sort_stats('time') + p.print_stats() + p.print_callers() + p.sort_stats('cumulative') + p.print_stats() + + pout.flush() =20 # # Was present to work around multiprocessing pool bugs in python < 2.7.3 @@ -1473,14 +1498,20 @@ def edit_bblayers_conf(bblayers_conf, add, remove, = edit_cb=3DNone): =20 return (notadded, notremoved) =20 - -def get_file_layer(filename, d): - """Determine the collection (as defined by a layer's layer.conf file) = containing the specified file""" +def get_collection_res(d): collections =3D (d.getVar('BBFILE_COLLECTIONS') or '').split() collection_res =3D {} for collection in collections: collection_res[collection] =3D d.getVar('BBFILE_PATTERN_%s' % coll= ection) or '' =20 + return collection_res + + +def get_file_layer(filename, d, collection_res=3D{}): + """Determine the collection (as defined by a layer's layer.conf file) = containing the specified file""" + if not collection_res: + collection_res =3D get_collection_res(d) + def path_to_layer(path): # Use longest path so we handle nested layers matchlen =3D 0 @@ -1492,12 +1523,13 @@ def get_file_layer(filename, d): return match =20 result =3D None - bbfiles =3D (d.getVar('BBFILES') or '').split() + bbfiles =3D (d.getVar('BBFILES_PRIORITIZED') or '').split() bbfilesmatch =3D False for bbfilesentry in bbfiles: - if fnmatch.fnmatch(filename, bbfilesentry): + if fnmatch.fnmatchcase(filename, bbfilesentry): bbfilesmatch =3D True result =3D path_to_layer(bbfilesentry) + break =20 if not bbfilesmatch: # Probably a bbclass @@ -1558,8 +1590,8 @@ def set_process_name(name): except: pass =20 -# export common proxies variables from datastore to environment def export_proxies(d): + """ export common proxies variables from datastore to environment """ import os =20 variables =3D ['http_proxy', 'HTTP_PROXY', 'https_proxy', 'HTTPS_PROXY= ', @@ -1581,12 +1613,12 @@ def export_proxies(d): =20 def load_plugins(logger, plugins, pluginpath): def load_plugin(name): - logger.debug(1, 'Loading plugin %s' % name) + logger.debug('Loading plugin %s' % name) spec =3D importlib.machinery.PathFinder.find_spec(name, path=3D[pl= uginpath] ) if spec: return spec.loader.load_module() =20 - logger.debug(1, 'Loading plugins from %s...' % pluginpath) + logger.debug('Loading plugins from %s...' % pluginpath) =20 expanded =3D (glob.glob(os.path.join(pluginpath, '*' + ext)) for ext in python_extensions) diff --git a/bitbake/lib/bblayers/action.py b/bitbake/lib/bblayers/action.py index d6459d66..f05f5d33 100644 --- a/bitbake/lib/bblayers/action.py +++ b/bitbake/lib/bblayers/action.py @@ -50,10 +50,10 @@ class ActionPlugin(LayerPlugin): if not (args.force or notadded): try: self.tinfoil.run_command('parseConfiguration') - except bb.tinfoil.TinfoilUIException: + except (bb.tinfoil.TinfoilUIException, bb.BBHandledExcepti= on): # Restore the back up copy of bblayers.conf shutil.copy2(backup, bblayers_conf) - bb.fatal("Parse failure with the specified layer added= ") + bb.fatal("Parse failure with the specified layer added= , aborting.") else: for item in notadded: sys.stderr.write("Specified layer %s is already in= BBLAYERS\n" % item) @@ -143,11 +143,12 @@ build results (as the layer priority order has effect= ively changed). =20 applied_appends =3D [] for layer in layers: - overlayed =3D [] - for f in self.tinfoil.cooker.collection.overlayed.keys(): - for of in self.tinfoil.cooker.collection.overlayed[f]: - if of.startswith(layer): - overlayed.append(of) + overlayed =3D set() + for mc in self.tinfoil.cooker.multiconfigs: + for f in self.tinfoil.cooker.collections[mc].overlayed.key= s(): + for of in self.tinfoil.cooker.collections[mc].overlaye= d[f]: + if of.startswith(layer): + overlayed.add(of) =20 logger.plain('Copying files from %s...' % layer ) for root, dirs, files in os.walk(layer): @@ -174,14 +175,21 @@ build results (as the layer priority order has effect= ively changed). logger.warning('Overwriting file %s', = fdest) bb.utils.copyfile(f1full, fdest) if ext =3D=3D '.bb': - for append in self.tinfoil.cooker.collecti= on.get_file_appends(f1full): + appends =3D set() + for mc in self.tinfoil.cooker.multiconfigs: + appends |=3D set(self.tinfoil.cooker.c= ollections[mc].get_file_appends(f1full)) + for append in appends: if layer_path_match(append): logger.plain(' Applying append %s= to %s' % (append, fdest)) self.apply_append(append, fdest) applied_appends.append(append) =20 # Take care of when some layers are excluded and yet we have inclu= ded bbappends for those recipes - for b in self.tinfoil.cooker.collection.bbappends: + bbappends =3D set() + for mc in self.tinfoil.cooker.multiconfigs: + bbappends |=3D set(self.tinfoil.cooker.collections[mc].bbappen= ds) + + for b in bbappends: (recipename, appendname) =3D b if appendname not in applied_appends: first_append =3D None diff --git a/bitbake/lib/bblayers/layerindex.py b/bitbake/lib/bblayers/laye= rindex.py index 95b67a66..b2f27b21 100644 --- a/bitbake/lib/bblayers/layerindex.py +++ b/bitbake/lib/bblayers/layerindex.py @@ -79,7 +79,7 @@ class LayerIndexPlugin(ActionPlugin): branches =3D [args.branch] else: branches =3D (self.tinfoil.config_data.getVar('LAYERSERIES_COR= ENAMES') or 'master').split() - logger.debug(1, 'Trying branches: %s' % branches) + logger.debug('Trying branches: %s' % branches) =20 ignore_layers =3D [] if args.ignore: diff --git a/bitbake/lib/bblayers/query.py b/bitbake/lib/bblayers/query.py index e2cc3105..947422a7 100644 --- a/bitbake/lib/bblayers/query.py +++ b/bitbake/lib/bblayers/query.py @@ -21,6 +21,10 @@ def plugin_init(plugins): =20 =20 class QueryPlugin(LayerPlugin): + def __init__(self): + super(QueryPlugin, self).__init__() + self.collection_res =3D {} + def do_show_layers(self, args): """show current configured layers.""" logger.plain("%s %s %s" % ("layer".ljust(20), "path".ljust(40), = "priority")) @@ -124,7 +128,7 @@ skipped recipes will also be listed, with a " (skipped)= " suffix. sys.exit(1) =20 pkg_pn =3D self.tinfoil.cooker.recipecaches[mc].pkg_pn - (latest_versions, preferred_versions) =3D self.tinfoil.find_provid= ers(mc) + (latest_versions, preferred_versions, required_versions) =3D self.= tinfoil.find_providers(mc) allproviders =3D self.tinfoil.get_all_providers(mc) =20 # Ensure we list skipped recipes @@ -222,7 +226,6 @@ skipped recipes will also be listed, with a " (skipped)= " suffix. multilayer =3D True if prov[0] !=3D pref[0]: same_ver =3D False - if (multilayer or not show_overlayed_only) and (same_v= er or not show_same_ver_only): if not items_listed: logger.plain('=3D=3D=3D %s =3D=3D=3D' % title) @@ -243,8 +246,13 @@ skipped recipes will also be listed, with a " (skipped= )" suffix. else: return '?' =20 + def get_collection_res(self): + if not self.collection_res: + self.collection_res =3D bb.utils.get_collection_res(self.tinfo= il.config_data) + return self.collection_res + def get_file_layerdir(self, filename): - layer =3D bb.utils.get_file_layer(filename, self.tinfoil.config_da= ta) + layer =3D bb.utils.get_file_layer(filename, self.tinfoil.config_da= ta, self.get_collection_res()) return self.bbfile_collections.get(layer, None) =20 def remove_layer_prefix(self, f): @@ -320,12 +328,12 @@ Lists recipes with the bbappends that apply to them a= s subitems. def get_appends_for_files(self, filenames): appended, notappended =3D [], [] for filename in filenames: - _, cls, _ =3D bb.cache.virtualfn2realfn(filename) + _, cls, mc =3D bb.cache.virtualfn2realfn(filename) if cls: continue =20 basename =3D os.path.basename(filename) - appends =3D self.tinfoil.cooker.collection.get_file_appends(ba= sename) + appends =3D self.tinfoil.cooker.collections[mc].get_file_appen= ds(basename) if appends: appended.append((basename, list(appends))) else: diff --git a/bitbake/lib/bs4/testing.py b/bitbake/lib/bs4/testing.py index 953bca8e..6584ecf3 100644 --- a/bitbake/lib/bs4/testing.py +++ b/bitbake/lib/bs4/testing.py @@ -15,7 +15,7 @@ from bs4.element import ( SoupStrainer, ) =20 -from bs4.builder import HTMLParserTreeBuilder +from bs4.builder._htmlparser import HTMLParserTreeBuilder default_builder =3D HTMLParserTreeBuilder =20 =20 @@ -56,7 +56,7 @@ class SoupTest(unittest.TestCase): self.assertEqual(earlier, e.previous_element) earlier =3D e =20 -class HTMLTreeBuilderSmokeTest(object): +class HTMLTreeBuilderSmokeTest(SoupTest): =20 """A basic test of a treebuilder's competence. =20 @@ -541,7 +541,7 @@ Hello, world! data.a['foo'] =3D 'bar' self.assertEqual('text', data.a.decode()) =20 -class XMLTreeBuilderSmokeTest(object): +class XMLTreeBuilderSmokeTest(SoupTest): =20 def test_pickle_and_unpickle_identity(self): # Pickling a tree, then unpickling it, yields a tree identical diff --git a/bitbake/lib/hashserv/__init__.py b/bitbake/lib/hashserv/__init= __.py index f95e8f43..5f2e101e 100644 --- a/bitbake/lib/hashserv/__init__.py +++ b/bitbake/lib/hashserv/__init__.py @@ -3,6 +3,7 @@ # SPDX-License-Identifier: GPL-2.0-only # =20 +import asyncio from contextlib import closing import re import sqlite3 @@ -21,6 +22,24 @@ ADDR_TYPE_TCP =3D 1 # is necessary DEFAULT_MAX_CHUNK =3D 32 * 1024 =20 +TABLE_DEFINITION =3D ( + ("method", "TEXT NOT NULL"), + ("outhash", "TEXT NOT NULL"), + ("taskhash", "TEXT NOT NULL"), + ("unihash", "TEXT NOT NULL"), + ("created", "DATETIME"), + + # Optional fields + ("owner", "TEXT"), + ("PN", "TEXT"), + ("PV", "TEXT"), + ("PR", "TEXT"), + ("task", "TEXT"), + ("outhash_siginfo", "TEXT"), +) + +TABLE_COLUMNS =3D tuple(name for name, _ in TABLE_DEFINITION) + def setup_database(database, sync=3DTrue): db =3D sqlite3.connect(database) db.row_factory =3D sqlite3.Row @@ -29,23 +48,10 @@ def setup_database(database, sync=3DTrue): cursor.execute(''' CREATE TABLE IF NOT EXISTS tasks_v2 ( id INTEGER PRIMARY KEY AUTOINCREMENT, - method TEXT NOT NULL, - outhash TEXT NOT NULL, - taskhash TEXT NOT NULL, - unihash TEXT NOT NULL, - created DATETIME, - - -- Optional fields - owner TEXT, - PN TEXT, - PV TEXT, - PR TEXT, - task TEXT, - outhash_siginfo TEXT, - + %s UNIQUE(method, outhash, taskhash) ) - ''') + ''' % " ".join("%s %s," % (name, typ) for name, typ in TABLE_D= EFINITION)) cursor.execute('PRAGMA journal_mode =3D WAL') cursor.execute('PRAGMA synchronous =3D %s' % ('NORMAL' if sync els= e 'OFF')) =20 @@ -88,10 +94,10 @@ def chunkify(msg, max_chunk): yield "\n" =20 =20 -def create_server(addr, dbname, *, sync=3DTrue): +def create_server(addr, dbname, *, sync=3DTrue, upstream=3DNone, read_only= =3DFalse): from . import server db =3D setup_database(dbname, sync=3Dsync) - s =3D server.Server(db) + s =3D server.Server(db, upstream=3Dupstream, read_only=3Dread_only) =20 (typ, a) =3D parse_address(addr) if typ =3D=3D ADDR_TYPE_UNIX: @@ -113,3 +119,15 @@ def create_client(addr): c.connect_tcp(*a) =20 return c + +async def create_async_client(addr): + from . import client + c =3D client.AsyncClient() + + (typ, a) =3D parse_address(addr) + if typ =3D=3D ADDR_TYPE_UNIX: + await c.connect_unix(*a) + else: + await c.connect_tcp(*a) + + return c diff --git a/bitbake/lib/hashserv/client.py b/bitbake/lib/hashserv/client.py index a29af836..e05c1eb5 100644 --- a/bitbake/lib/hashserv/client.py +++ b/bitbake/lib/hashserv/client.py @@ -3,189 +3,231 @@ # SPDX-License-Identifier: GPL-2.0-only # =20 +import asyncio import json import logging import socket import os -from . import chunkify, DEFAULT_MAX_CHUNK +from . import chunkify, DEFAULT_MAX_CHUNK, create_async_client =20 =20 -logger =3D logging.getLogger('hashserv.client') +logger =3D logging.getLogger("hashserv.client") =20 =20 class HashConnectionError(Exception): pass =20 =20 -class Client(object): +class AsyncClient(object): MODE_NORMAL =3D 0 MODE_GET_STREAM =3D 1 =20 def __init__(self): - self._socket =3D None self.reader =3D None self.writer =3D None self.mode =3D self.MODE_NORMAL self.max_chunk =3D DEFAULT_MAX_CHUNK =20 - def connect_tcp(self, address, port): - def connect_sock(): - s =3D socket.create_connection((address, port)) - - s.setsockopt(socket.SOL_TCP, socket.TCP_NODELAY, 1) - s.setsockopt(socket.SOL_TCP, socket.TCP_QUICKACK, 1) - s.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1) - return s + async def connect_tcp(self, address, port): + async def connect_sock(): + return await asyncio.open_connection(address, port) =20 self._connect_sock =3D connect_sock =20 - def connect_unix(self, path): - def connect_sock(): - s =3D socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) - # AF_UNIX has path length issues so chdir here to workaround - cwd =3D os.getcwd() - try: - os.chdir(os.path.dirname(path)) - s.connect(os.path.basename(path)) - finally: - os.chdir(cwd) - return s + async def connect_unix(self, path): + async def connect_sock(): + return await asyncio.open_unix_connection(path) =20 self._connect_sock =3D connect_sock =20 - def connect(self): - if self._socket is None: - self._socket =3D self._connect_sock() - - self.reader =3D self._socket.makefile('r', encoding=3D'utf-8') - self.writer =3D self._socket.makefile('w', encoding=3D'utf-8') + async def connect(self): + if self.reader is None or self.writer is None: + (self.reader, self.writer) =3D await self._connect_sock() =20 - self.writer.write('OEHASHEQUIV 1.1\n\n') - self.writer.flush() + self.writer.write("OEHASHEQUIV 1.1\n\n".encode("utf-8")) + await self.writer.drain() =20 - # Restore mode if the socket is being re-created cur_mode =3D self.mode self.mode =3D self.MODE_NORMAL - self._set_mode(cur_mode) + await self._set_mode(cur_mode) =20 - return self._socket + async def close(self): + self.reader =3D None =20 - def close(self): - if self._socket is not None: - self._socket.close() - self._socket =3D None - self.reader =3D None + if self.writer is not None: + self.writer.close() self.writer =3D None =20 - def _send_wrapper(self, proc): + async def _send_wrapper(self, proc): count =3D 0 while True: try: - self.connect() - return proc() - except (OSError, HashConnectionError, json.JSONDecodeError, Un= icodeDecodeError) as e: - logger.warning('Error talking to server: %s' % e) + await self.connect() + return await proc() + except ( + OSError, + HashConnectionError, + json.JSONDecodeError, + UnicodeDecodeError, + ) as e: + logger.warning("Error talking to server: %s" % e) if count >=3D 3: if not isinstance(e, HashConnectionError): raise HashConnectionError(str(e)) raise e - self.close() + await self.close() count +=3D 1 =20 - def send_message(self, msg): - def get_line(): - line =3D self.reader.readline() + async def send_message(self, msg): + async def get_line(): + line =3D await self.reader.readline() if not line: - raise HashConnectionError('Connection closed') + raise HashConnectionError("Connection closed") + + line =3D line.decode("utf-8") =20 - if not line.endswith('\n'): - raise HashConnectionError('Bad message %r' % message) + if not line.endswith("\n"): + raise HashConnectionError("Bad message %r" % message) =20 return line =20 - def proc(): + async def proc(): for c in chunkify(json.dumps(msg), self.max_chunk): - self.writer.write(c) - self.writer.flush() + self.writer.write(c.encode("utf-8")) + await self.writer.drain() =20 - l =3D get_line() + l =3D await get_line() =20 m =3D json.loads(l) - if 'chunk-stream' in m: + if m and "chunk-stream" in m: lines =3D [] while True: - l =3D get_line().rstrip('\n') + l =3D (await get_line()).rstrip("\n") if not l: break lines.append(l) =20 - m =3D json.loads(''.join(lines)) + m =3D json.loads("".join(lines)) =20 return m =20 - return self._send_wrapper(proc) + return await self._send_wrapper(proc) =20 - def send_stream(self, msg): - def proc(): - self.writer.write("%s\n" % msg) - self.writer.flush() - l =3D self.reader.readline() + async def send_stream(self, msg): + async def proc(): + self.writer.write(("%s\n" % msg).encode("utf-8")) + await self.writer.drain() + l =3D await self.reader.readline() if not l: - raise HashConnectionError('Connection closed') - return l.rstrip() + raise HashConnectionError("Connection closed") + return l.decode("utf-8").rstrip() =20 - return self._send_wrapper(proc) + return await self._send_wrapper(proc) =20 - def _set_mode(self, new_mode): + async def _set_mode(self, new_mode): if new_mode =3D=3D self.MODE_NORMAL and self.mode =3D=3D self.MODE= _GET_STREAM: - r =3D self.send_stream('END') - if r !=3D 'ok': - raise HashConnectionError('Bad response from server %r' % = r) + r =3D await self.send_stream("END") + if r !=3D "ok": + raise HashConnectionError("Bad response from server %r" % = r) elif new_mode =3D=3D self.MODE_GET_STREAM and self.mode =3D=3D sel= f.MODE_NORMAL: - r =3D self.send_message({'get-stream': None}) - if r !=3D 'ok': - raise HashConnectionError('Bad response from server %r' % = r) + r =3D await self.send_message({"get-stream": None}) + if r !=3D "ok": + raise HashConnectionError("Bad response from server %r" % = r) elif new_mode !=3D self.mode: - raise Exception('Undefined mode transition %r -> %r' % (self.m= ode, new_mode)) + raise Exception( + "Undefined mode transition %r -> %r" % (self.mode, new_mod= e) + ) =20 self.mode =3D new_mode =20 - def get_unihash(self, method, taskhash): - self._set_mode(self.MODE_GET_STREAM) - r =3D self.send_stream('%s %s' % (method, taskhash)) + async def get_unihash(self, method, taskhash): + await self._set_mode(self.MODE_GET_STREAM) + r =3D await self.send_stream("%s %s" % (method, taskhash)) if not r: return None return r =20 - def report_unihash(self, taskhash, method, outhash, unihash, extra=3D{= }): - self._set_mode(self.MODE_NORMAL) + async def report_unihash(self, taskhash, method, outhash, unihash, ext= ra=3D{}): + await self._set_mode(self.MODE_NORMAL) m =3D extra.copy() - m['taskhash'] =3D taskhash - m['method'] =3D method - m['outhash'] =3D outhash - m['unihash'] =3D unihash - return self.send_message({'report': m}) - - def report_unihash_equiv(self, taskhash, method, unihash, extra=3D{}): - self._set_mode(self.MODE_NORMAL) + m["taskhash"] =3D taskhash + m["method"] =3D method + m["outhash"] =3D outhash + m["unihash"] =3D unihash + return await self.send_message({"report": m}) + + async def report_unihash_equiv(self, taskhash, method, unihash, extra= =3D{}): + await self._set_mode(self.MODE_NORMAL) m =3D extra.copy() - m['taskhash'] =3D taskhash - m['method'] =3D method - m['unihash'] =3D unihash - return self.send_message({'report-equiv': m}) - - def get_taskhash(self, method, taskhash, all_properties=3DFalse): - self._set_mode(self.MODE_NORMAL) - return self.send_message({'get': { - 'taskhash': taskhash, - 'method': method, - 'all': all_properties - }}) - - def get_stats(self): - self._set_mode(self.MODE_NORMAL) - return self.send_message({'get-stats': None}) - - def reset_stats(self): - self._set_mode(self.MODE_NORMAL) - return self.send_message({'reset-stats': None}) + m["taskhash"] =3D taskhash + m["method"] =3D method + m["unihash"] =3D unihash + return await self.send_message({"report-equiv": m}) + + async def get_taskhash(self, method, taskhash, all_properties=3DFalse): + await self._set_mode(self.MODE_NORMAL) + return await self.send_message( + {"get": {"taskhash": taskhash, "method": method, "all": all_pr= operties}} + ) + + async def get_outhash(self, method, outhash, taskhash): + await self._set_mode(self.MODE_NORMAL) + return await self.send_message( + {"get-outhash": {"outhash": outhash, "taskhash": taskhash, "me= thod": method}} + ) + + async def get_stats(self): + await self._set_mode(self.MODE_NORMAL) + return await self.send_message({"get-stats": None}) + + async def reset_stats(self): + await self._set_mode(self.MODE_NORMAL) + return await self.send_message({"reset-stats": None}) + + async def backfill_wait(self): + await self._set_mode(self.MODE_NORMAL) + return (await self.send_message({"backfill-wait": None}))["tasks"] + + +class Client(object): + def __init__(self): + self.client =3D AsyncClient() + self.loop =3D asyncio.new_event_loop() + + for call in ( + "connect_tcp", + "close", + "get_unihash", + "report_unihash", + "report_unihash_equiv", + "get_taskhash", + "get_stats", + "reset_stats", + "backfill_wait", + ): + downcall =3D getattr(self.client, call) + setattr(self, call, self._get_downcall_wrapper(downcall)) + + def _get_downcall_wrapper(self, downcall): + def wrapper(*args, **kwargs): + return self.loop.run_until_complete(downcall(*args, **kwargs)) + + return wrapper + + def connect_unix(self, path): + # AF_UNIX has path length issues so chdir here to workaround + cwd =3D os.getcwd() + try: + os.chdir(os.path.dirname(path)) + self.loop.run_until_complete(self.client.connect_unix(os.path.= basename(path))) + self.loop.run_until_complete(self.client.connect()) + finally: + os.chdir(cwd) + + @property + def max_chunk(self): + return self.client.max_chunk + + @max_chunk.setter + def max_chunk(self, value): + self.client.max_chunk =3D value diff --git a/bitbake/lib/hashserv/server.py b/bitbake/lib/hashserv/server.py index 81050715..a0dc0c17 100644 --- a/bitbake/lib/hashserv/server.py +++ b/bitbake/lib/hashserv/server.py @@ -3,7 +3,7 @@ # SPDX-License-Identifier: GPL-2.0-only # =20 -from contextlib import closing +from contextlib import closing, contextmanager from datetime import datetime import asyncio import json @@ -12,8 +12,9 @@ import math import os import signal import socket +import sys import time -from . import chunkify, DEFAULT_MAX_CHUNK +from . import chunkify, DEFAULT_MAX_CHUNK, create_async_client, TABLE_COLU= MNS =20 logger =3D logging.getLogger('hashserv.server') =20 @@ -111,29 +112,95 @@ class Stats(object): class ClientError(Exception): pass =20 +class ServerError(Exception): + pass + +def insert_task(cursor, data, ignore=3DFalse): + keys =3D sorted(data.keys()) + query =3D '''INSERT%s INTO tasks_v2 (%s) VALUES (%s)''' % ( + " OR IGNORE" if ignore else "", + ', '.join(keys), + ', '.join(':' + k for k in keys)) + cursor.execute(query, data) + +async def copy_from_upstream(client, db, method, taskhash): + d =3D await client.get_taskhash(method, taskhash, True) + if d is not None: + # Filter out unknown columns + d =3D {k: v for k, v in d.items() if k in TABLE_COLUMNS} + keys =3D sorted(d.keys()) + + with closing(db.cursor()) as cursor: + insert_task(cursor, d) + db.commit() + + return d + +async def copy_outhash_from_upstream(client, db, method, outhash, taskhash= ): + d =3D await client.get_outhash(method, outhash, taskhash) + if d is not None: + # Filter out unknown columns + d =3D {k: v for k, v in d.items() if k in TABLE_COLUMNS} + keys =3D sorted(d.keys()) + + with closing(db.cursor()) as cursor: + insert_task(cursor, d) + db.commit() + + return d + class ServerClient(object): FAST_QUERY =3D 'SELECT taskhash, method, unihash FROM tasks_v2 WHERE m= ethod=3D:method AND taskhash=3D:taskhash ORDER BY created ASC LIMIT 1' ALL_QUERY =3D 'SELECT * FROM tasks_v2 WHERE m= ethod=3D:method AND taskhash=3D:taskhash ORDER BY created ASC LIMIT 1' - - def __init__(self, reader, writer, db, request_stats): + OUTHASH_QUERY =3D ''' + -- Find tasks with a matching outhash (that is, tasks that + -- are equivalent) + SELECT * FROM tasks_v2 WHERE method=3D:method AND outhash=3D:outha= sh + + -- If there is an exact match on the taskhash, return it. + -- Otherwise return the oldest matching outhash of any + -- taskhash + ORDER BY CASE WHEN taskhash=3D:taskhash THEN 1 ELSE 2 END, + created ASC + + -- Only return one row + LIMIT 1 + ''' + + def __init__(self, reader, writer, db, request_stats, backfill_queue, = upstream, read_only): self.reader =3D reader self.writer =3D writer self.db =3D db self.request_stats =3D request_stats self.max_chunk =3D DEFAULT_MAX_CHUNK + self.backfill_queue =3D backfill_queue + self.upstream =3D upstream =20 self.handlers =3D { 'get': self.handle_get, - 'report': self.handle_report, - 'report-equiv': self.handle_equivreport, + 'get-outhash': self.handle_get_outhash, 'get-stream': self.handle_get_stream, 'get-stats': self.handle_get_stats, - 'reset-stats': self.handle_reset_stats, 'chunk-stream': self.handle_chunk, } =20 + if not read_only: + self.handlers.update({ + 'report': self.handle_report, + 'report-equiv': self.handle_equivreport, + 'reset-stats': self.handle_reset_stats, + 'backfill-wait': self.handle_backfill_wait, + }) + async def process_requests(self): + if self.upstream is not None: + self.upstream_client =3D await create_async_client(self.upstre= am) + else: + self.upstream_client =3D None + try: + + self.addr =3D self.writer.get_extra_info('peername') logger.debug('Client %r connected' % (self.addr,)) =20 @@ -171,6 +238,9 @@ class ServerClient(object): except ClientError as e: logger.error(str(e)) finally: + if self.upstream_client is not None: + await self.upstream_client.close() + self.writer.close() =20 async def dispatch_message(self, msg): @@ -239,15 +309,34 @@ class ServerClient(object): if row is not None: logger.debug('Found equivalent task %s -> %s', (row['taskhash'= ], row['unihash'])) d =3D {k: row[k] for k in row.keys()} + elif self.upstream_client is not None: + d =3D await copy_from_upstream(self.upstream_client, self.db, = method, taskhash) + else: + d =3D None =20 - self.write_message(d) + self.write_message(d) + + async def handle_get_outhash(self, request): + with closing(self.db.cursor()) as cursor: + cursor.execute(self.OUTHASH_QUERY, + {k: request[k] for k in ('method', 'outhash', '= taskhash')}) + + row =3D cursor.fetchone() + + if row is not None: + logger.debug('Found equivalent outhash %s -> %s', (row['outhas= h'], row['unihash'])) + d =3D {k: row[k] for k in row.keys()} else: - self.write_message(None) + d =3D None + + self.write_message(d) =20 async def handle_get_stream(self, request): self.write_message('ok') =20 while True: + upstream =3D None + l =3D await self.reader.readline() if not l: return @@ -272,6 +361,12 @@ class ServerClient(object): if row is not None: msg =3D ('%s\n' % row['unihash']).encode('utf-8') #logger.debug('Found equivalent task %s -> %s', (row['= taskhash'], row['unihash'])) + elif self.upstream_client is not None: + upstream =3D await self.upstream_client.get_unihash(me= thod, taskhash) + if upstream: + msg =3D ("%s\n" % upstream).encode("utf-8") + else: + msg =3D "\n".encode("utf-8") else: msg =3D '\n'.encode('utf-8') =20 @@ -282,25 +377,26 @@ class ServerClient(object): =20 await self.writer.drain() =20 + # Post to the backfill queue after writing the result to minim= ize + # the turn around time on a request + if upstream is not None: + await self.backfill_queue.put((method, taskhash)) + async def handle_report(self, data): with closing(self.db.cursor()) as cursor: - cursor.execute(''' - -- Find tasks with a matching outhash (that is, tasks that - -- are equivalent) - SELECT taskhash, method, unihash FROM tasks_v2 WHERE metho= d=3D:method AND outhash=3D:outhash - - -- If there is an exact match on the taskhash, return it. - -- Otherwise return the oldest matching outhash of any - -- taskhash - ORDER BY CASE WHEN taskhash=3D:taskhash THEN 1 ELSE 2 END, - created ASC - - -- Only return one row - LIMIT 1 - ''', {k: data[k] for k in ('method', 'outhash', 'taskhash'= )}) + cursor.execute(self.OUTHASH_QUERY, + {k: data[k] for k in ('method', 'outhash', 'tas= khash')}) =20 row =3D cursor.fetchone() =20 + if row is None and self.upstream_client: + # Try upstream + row =3D await copy_outhash_from_upstream(self.upstream_cli= ent, + self.db, + data['method'], + data['outhash'], + data['taskhash']) + # If no matching outhash was found, or one *was* found but it # wasn't an exact match on the taskhash, a new entry for this # taskhash should be added @@ -324,11 +420,7 @@ class ServerClient(object): if k in data: insert_data[k] =3D data[k] =20 - cursor.execute('''INSERT INTO tasks_v2 (%s) VALUES (%s)'''= % ( - ', '.join(sorted(insert_data.keys())), - ', '.join(':' + k for k in sorted(insert_data.keys()))= ), - insert_data) - + insert_task(cursor, insert_data) self.db.commit() =20 logger.info('Adding taskhash %s with unihash %s', @@ -358,11 +450,7 @@ class ServerClient(object): if k in data: insert_data[k] =3D data[k] =20 - cursor.execute('''INSERT OR IGNORE INTO tasks_v2 (%s) VALUES (= %s)''' % ( - ', '.join(sorted(insert_data.keys())), - ', '.join(':' + k for k in sorted(insert_data.keys()))), - insert_data) - + insert_task(cursor, insert_data, ignore=3DTrue) self.db.commit() =20 # Fetch the unihash that will be reported for the taskhash. If= the @@ -394,6 +482,13 @@ class ServerClient(object): self.request_stats.reset() self.write_message(d) =20 + async def handle_backfill_wait(self, request): + d =3D { + 'tasks': self.backfill_queue.qsize(), + } + await self.backfill_queue.join() + self.write_message(d) + def query_equivalent(self, method, taskhash, query): # This is part of the inner loop and must be as fast as possible try: @@ -405,7 +500,10 @@ class ServerClient(object): =20 =20 class Server(object): - def __init__(self, db, loop=3DNone): + def __init__(self, db, loop=3DNone, upstream=3DNone, read_only=3DFalse= ): + if upstream and read_only: + raise ServerError("Read-only hashserv cannot pull from an upst= ream server") + self.request_stats =3D Stats() self.db =3D db =20 @@ -416,6 +514,9 @@ class Server(object): self.loop =3D loop self.close_loop =3D False =20 + self.upstream =3D upstream + self.read_only =3D read_only + self._cleanup_socket =3D None =20 def start_tcp_server(self, host, port): @@ -458,7 +559,7 @@ class Server(object): async def handle_client(self, reader, writer): # writer.transport.set_write_buffer_limits(0) try: - client =3D ServerClient(reader, writer, self.db, self.request_= stats) + client =3D ServerClient(reader, writer, self.db, self.request_= stats, self.backfill_queue, self.upstream, self.read_only) await client.process_requests() except Exception as e: import traceback @@ -467,23 +568,60 @@ class Server(object): writer.close() logger.info('Client disconnected') =20 + @contextmanager + def _backfill_worker(self): + async def backfill_worker_task(): + client =3D await create_async_client(self.upstream) + try: + while True: + item =3D await self.backfill_queue.get() + if item is None: + self.backfill_queue.task_done() + break + method, taskhash =3D item + await copy_from_upstream(client, self.db, method, task= hash) + self.backfill_queue.task_done() + finally: + await client.close() + + async def join_worker(worker): + await self.backfill_queue.put(None) + await worker + + if self.upstream is not None: + worker =3D asyncio.ensure_future(backfill_worker_task()) + try: + yield + finally: + self.loop.run_until_complete(join_worker(worker)) + else: + yield + def serve_forever(self): def signal_handler(): self.loop.stop() =20 - self.loop.add_signal_handler(signal.SIGTERM, signal_handler) - + asyncio.set_event_loop(self.loop) try: - self.loop.run_forever() - except KeyboardInterrupt: - pass + self.backfill_queue =3D asyncio.Queue() =20 - self.server.close() - self.loop.run_until_complete(self.server.wait_closed()) - logger.info('Server shutting down') + self.loop.add_signal_handler(signal.SIGTERM, signal_handler) =20 - if self.close_loop: - self.loop.close() + with self._backfill_worker(): + try: + self.loop.run_forever() + except KeyboardInterrupt: + pass + + self.server.close() + + self.loop.run_until_complete(self.server.wait_closed()) + logger.info('Server shutting down') + finally: + if self.close_loop: + if sys.version_info >=3D (3, 6): + self.loop.run_until_complete(self.loop.shutdown_asyncg= ens()) + self.loop.close() =20 - if self._cleanup_socket is not None: - self._cleanup_socket() + if self._cleanup_socket is not None: + self._cleanup_socket() diff --git a/bitbake/lib/hashserv/tests.py b/bitbake/lib/hashserv/tests.py index 6e862950..1a696481 100644 --- a/bitbake/lib/hashserv/tests.py +++ b/bitbake/lib/hashserv/tests.py @@ -6,52 +6,79 @@ # =20 from . import create_server, create_client +from .client import HashConnectionError import hashlib import logging import multiprocessing +import os import sys import tempfile import threading import unittest +import socket =20 +def _run_server(server, idx): + # logging.basicConfig(level=3Dlogging.DEBUG, filename=3D'bbhashserv.lo= g', filemode=3D'w', + # format=3D'%(levelname)s %(filename)s:%(lineno)d = %(message)s') + sys.stdout =3D open('bbhashserv-%d.log' % idx, 'w') + sys.stderr =3D sys.stdout + server.serve_forever() =20 -class TestHashEquivalenceServer(object): + +class HashEquivalenceTestSetup(object): METHOD =3D 'TestMethod' =20 - def _run_server(self): - # logging.basicConfig(level=3Dlogging.DEBUG, filename=3D'bbhashser= v.log', filemode=3D'w', - # format=3D'%(levelname)s %(filename)s:%(linen= o)d %(message)s') - self.server.serve_forever() + server_index =3D 0 + + def start_server(self, dbpath=3DNone, upstream=3DNone, read_only=3DFal= se): + self.server_index +=3D 1 + if dbpath is None: + dbpath =3D os.path.join(self.temp_dir.name, "db%d.sqlite" % se= lf.server_index) + + def cleanup_thread(thread): + thread.terminate() + thread.join() + + server =3D create_server(self.get_server_addr(self.server_index), + dbpath, + upstream=3Dupstream, + read_only=3Dread_only) + server.dbpath =3D dbpath + + server.thread =3D multiprocessing.Process(target=3D_run_server, ar= gs=3D(server, self.server_index)) + server.thread.start() + self.addCleanup(cleanup_thread, server.thread) + + def cleanup_client(client): + client.close() + + client =3D create_client(server.address) + self.addCleanup(cleanup_client, client) + + return (client, server) =20 def setUp(self): if sys.version_info < (3, 5, 0): self.skipTest('Python 3.5 or later required') =20 self.temp_dir =3D tempfile.TemporaryDirectory(prefix=3D'bb-hashser= v') - self.dbfile =3D os.path.join(self.temp_dir.name, 'db.sqlite') - - self.server =3D create_server(self.get_server_addr(), self.dbfile) - self.server_thread =3D multiprocessing.Process(target=3Dself._run_= server) - self.server_thread.start() - self.client =3D create_client(self.server.address) - - def tearDown(self): - # Shutdown server - s =3D getattr(self, 'server', None) - if s is not None: - self.server_thread.terminate() - self.server_thread.join() - self.client.close() - self.temp_dir.cleanup() + self.addCleanup(self.temp_dir.cleanup) + + (self.client, self.server) =3D self.start_server() + + def assertClientGetHash(self, client, taskhash, unihash): + result =3D client.get_unihash(self.METHOD, taskhash) + self.assertEqual(result, unihash) =20 + +class HashEquivalenceCommonTests(object): def test_create_hash(self): # Simple test that hashes can be created taskhash =3D '35788efcb8dfb0a02659d81cf2bfd695fb30faf9' outhash =3D '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98f= d7e5970796f' unihash =3D 'f46d3fbb439bd9b921095da657a4de906510d2cd' =20 - result =3D self.client.get_unihash(self.METHOD, taskhash) - self.assertIsNone(result, msg=3D'Found unexpected task, %r' % resu= lt) + self.assertClientGetHash(self.client, taskhash, None) =20 result =3D self.client.report_unihash(taskhash, self.METHOD, outha= sh, unihash) self.assertEqual(result['unihash'], unihash, 'Server returned bad = unihash') @@ -82,22 +109,19 @@ class TestHashEquivalenceServer(object): unihash =3D '218e57509998197d570e2c98512d0105985dffc9' self.client.report_unihash(taskhash, self.METHOD, outhash, unihash) =20 - result =3D self.client.get_unihash(self.METHOD, taskhash) - self.assertEqual(result, unihash) + self.assertClientGetHash(self.client, taskhash, unihash) =20 outhash2 =3D '0904a7fe3dc712d9fd8a74a616ddca2a825a8ee97adf0bd3fc86= 082c7639914d' unihash2 =3D 'ae9a7d252735f0dafcdb10e2e02561ca3a47314c' self.client.report_unihash(taskhash, self.METHOD, outhash2, unihas= h2) =20 - result =3D self.client.get_unihash(self.METHOD, taskhash) - self.assertEqual(result, unihash) + self.assertClientGetHash(self.client, taskhash, unihash) =20 outhash3 =3D '77623a549b5b1a31e3732dfa8fe61d7ce5d44b3370f253c5360e= 136b852967b4' unihash3 =3D '9217a7d6398518e5dc002ed58f2cbbbc78696603' self.client.report_unihash(taskhash, self.METHOD, outhash3, unihas= h3) =20 - result =3D self.client.get_unihash(self.METHOD, taskhash) - self.assertEqual(result, unihash) + self.assertClientGetHash(self.client, taskhash, unihash) =20 def test_huge_message(self): # Simple test that hashes can be created @@ -105,8 +129,7 @@ class TestHashEquivalenceServer(object): outhash =3D '3c979c3db45c569f51ab7626a4651074be3a9d11a84b1db076f5b= 14f7d39db44' unihash =3D '90e9bc1d1f094c51824adca7f8ea79a048d68824' =20 - result =3D self.client.get_unihash(self.METHOD, taskhash) - self.assertIsNone(result, msg=3D'Found unexpected task, %r' % resu= lt) + self.assertClientGetHash(self.client, taskhash, None) =20 siginfo =3D "0" * (self.client.max_chunk * 4) =20 @@ -154,12 +177,140 @@ class TestHashEquivalenceServer(object): =20 self.assertFalse(failures) =20 + def test_upstream_server(self): + # Tests upstream server support. This is done by creating two serv= ers + # that share a database file. The downstream server has it upstream + # set to the test server, whereas the side server doesn't. This al= lows + # verification that the hash requests are being proxied to the ups= tream + # server by verifying that they appear on the downstream client, b= ut not + # the side client. It also verifies that the results are pulled in= to + # the downstream database by checking that the downstream and side= servers + # match after the downstream is done waiting for all backfill tasks + (down_client, down_server) =3D self.start_server(upstream=3Dself.s= erver.address) + (side_client, side_server) =3D self.start_server(dbpath=3Ddown_ser= ver.dbpath) + + def check_hash(taskhash, unihash, old_sidehash): + nonlocal down_client + nonlocal side_client + + # check upstream server + self.assertClientGetHash(self.client, taskhash, unihash) + + # Hash should *not* be present on the side server + self.assertClientGetHash(side_client, taskhash, old_sidehash) + + # Hash should be present on the downstream server, since it + # will defer to the upstream server. This will trigger + # the backfill in the downstream server + self.assertClientGetHash(down_client, taskhash, unihash) + + # After waiting for the downstream client to finish backfillin= g the + # task from the upstream server, it should appear in the side = server + # since the database is populated + down_client.backfill_wait() + self.assertClientGetHash(side_client, taskhash, unihash) + + # Basic report + taskhash =3D '8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a' + outhash =3D 'afe240a439959ce86f5e322f8c208e1fedefea9e813f2140c81af= 866cc9edf7e' + unihash =3D '218e57509998197d570e2c98512d0105985dffc9' + self.client.report_unihash(taskhash, self.METHOD, outhash, unihash) + + check_hash(taskhash, unihash, None) + + # Duplicated taskhash with multiple output hashes and unihashes. + # All servers should agree with the originally reported hash + outhash2 =3D '0904a7fe3dc712d9fd8a74a616ddca2a825a8ee97adf0bd3fc86= 082c7639914d' + unihash2 =3D 'ae9a7d252735f0dafcdb10e2e02561ca3a47314c' + self.client.report_unihash(taskhash, self.METHOD, outhash2, unihas= h2) + + check_hash(taskhash, unihash, unihash) + + # Report an equivalent task. The sideload will originally report + # no unihash until backfilled + taskhash3 =3D "044c2ec8aaf480685a00ff6ff49e6162e6ad34e1" + unihash3 =3D "def64766090d28f627e816454ed46894bb3aab36" + self.client.report_unihash(taskhash3, self.METHOD, outhash, unihas= h3) + + check_hash(taskhash3, unihash, None) + + # Test that reporting a unihash in the downstream client isn't + # propagating to the upstream server + taskhash4 =3D "e3da00593d6a7fb435c7e2114976c59c5fd6d561" + outhash4 =3D "1cf8713e645f491eb9c959d20b5cae1c47133a292626dda9b107= 09857cbe688a" + unihash4 =3D "3b5d3d83f07f259e9086fcb422c855286e18a57d" + down_client.report_unihash(taskhash4, self.METHOD, outhash4, uniha= sh4) + down_client.backfill_wait() =20 -class TestHashEquivalenceUnixServer(TestHashEquivalenceServer, unittest.Te= stCase): - def get_server_addr(self): - return "unix://" + os.path.join(self.temp_dir.name, 'sock') + self.assertClientGetHash(down_client, taskhash4, unihash4) + self.assertClientGetHash(side_client, taskhash4, unihash4) + self.assertClientGetHash(self.client, taskhash4, None) + + # Test that reporting a unihash in the downstream is able to find a + # match which was previously reported to the upstream server + taskhash5 =3D '35788efcb8dfb0a02659d81cf2bfd695fb30faf9' + outhash5 =3D '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98= fd7e5970796f' + unihash5 =3D 'f46d3fbb439bd9b921095da657a4de906510d2cd' + result =3D self.client.report_unihash(taskhash5, self.METHOD, outh= ash5, unihash5) + + taskhash6 =3D '35788efcb8dfb0a02659d81cf2bfd695fb30fafa' + unihash6 =3D 'f46d3fbb439bd9b921095da657a4de906510d2ce' + result =3D down_client.report_unihash(taskhash6, self.METHOD, outh= ash5, unihash6) + self.assertEqual(result['unihash'], unihash5, 'Server failed to co= py unihash from upstream') + + def test_ro_server(self): + (ro_client, ro_server) =3D self.start_server(dbpath=3Dself.server.= dbpath, read_only=3DTrue) + + # Report a hash via the read-write server + taskhash =3D '35788efcb8dfb0a02659d81cf2bfd695fb30faf9' + outhash =3D '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98f= d7e5970796f' + unihash =3D 'f46d3fbb439bd9b921095da657a4de906510d2cd' + + result =3D self.client.report_unihash(taskhash, self.METHOD, outha= sh, unihash) + self.assertEqual(result['unihash'], unihash, 'Server returned bad = unihash') + + # Check the hash via the read-only server + self.assertClientGetHash(ro_client, taskhash, unihash) + + # Ensure that reporting via the read-only server fails + taskhash2 =3D 'c665584ee6817aa99edfc77a44dd853828279370' + outhash2 =3D '3c979c3db45c569f51ab7626a4651074be3a9d11a84b1db076f5= b14f7d39db44' + unihash2 =3D '90e9bc1d1f094c51824adca7f8ea79a048d68824' + + with self.assertRaises(HashConnectionError): + ro_client.report_unihash(taskhash2, self.METHOD, outhash2, uni= hash2) + + # Ensure that the database was not modified + self.assertClientGetHash(self.client, taskhash2, None) + + +class TestHashEquivalenceUnixServer(HashEquivalenceTestSetup, HashEquivale= nceCommonTests, unittest.TestCase): + def get_server_addr(self, server_idx): + return "unix://" + os.path.join(self.temp_dir.name, 'sock%d' % ser= ver_idx) + + +class TestHashEquivalenceUnixServerLongPath(HashEquivalenceTestSetup, unit= test.TestCase): + DEEP_DIRECTORY =3D "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa= aaaaa/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb/ccccccccccc= cccccccccccccccccccccccccccccccc" + def get_server_addr(self, server_idx): + os.makedirs(os.path.join(self.temp_dir.name, self.DEEP_DIRECTORY),= exist_ok=3DTrue) + return "unix://" + os.path.join(self.temp_dir.name, self.DEEP_DIRE= CTORY, 'sock%d' % server_idx) + + + def test_long_sock_path(self): + # Simple test that hashes can be created + taskhash =3D '35788efcb8dfb0a02659d81cf2bfd695fb30faf9' + outhash =3D '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98f= d7e5970796f' + unihash =3D 'f46d3fbb439bd9b921095da657a4de906510d2cd' + + self.assertClientGetHash(self.client, taskhash, None) + + result =3D self.client.report_unihash(taskhash, self.METHOD, outha= sh, unihash) + self.assertEqual(result['unihash'], unihash, 'Server returned bad = unihash') =20 =20 -class TestHashEquivalenceTCPServer(TestHashEquivalenceServer, unittest.Tes= tCase): - def get_server_addr(self): - return "localhost:0" +class TestHashEquivalenceTCPServer(HashEquivalenceTestSetup, HashEquivalen= ceCommonTests, unittest.TestCase): + def get_server_addr(self, server_idx): + # Some hosts cause asyncio module to misbehave, when IPv6 is not e= nabled. + # If IPv6 is enabled, it should be safe to use localhost directly,= in general + # case it is more reliable to resolve the IP address explicitly. + return socket.gethostbyname("localhost") + ":0" diff --git a/bitbake/lib/layerindexlib/__init__.py b/bitbake/lib/layerindex= lib/__init__.py index 77196b40..9ca127b9 100644 --- a/bitbake/lib/layerindexlib/__init__.py +++ b/bitbake/lib/layerindexlib/__init__.py @@ -7,6 +7,7 @@ import datetime =20 import logging import imp +import os =20 from collections import OrderedDict from layerindexlib.plugin import LayerIndexPluginUrlError @@ -70,7 +71,7 @@ class LayerIndex(): =20 if self.__class__ !=3D newIndex.__class__ or \ other.__class__ !=3D newIndex.__class__: - raise TypeException("Can not add different types.") + raise TypeError("Can not add different types.") =20 for indexEnt in self.indexes: newIndex.indexes.append(indexEnt) @@ -93,7 +94,7 @@ class LayerIndex(): if not param: continue item =3D param.split('=3D', 1) - logger.debug(1, item) + logger.debug(item) param_dict[item[0]] =3D item[1] =20 return param_dict @@ -122,7 +123,7 @@ class LayerIndex(): up =3D urlparse(url) =20 if username: - logger.debug(1, "Configuring authentication for %s..." % url) + logger.debug("Configuring authentication for %s..." % url) password_mgr =3D urllib.request.HTTPPasswordMgrWithDefaultReal= m() password_mgr.add_password(None, "%s://%s" % (up.scheme, up.net= loc), username, password) handler =3D urllib.request.HTTPBasicAuthHandler(password_mgr) @@ -132,20 +133,20 @@ class LayerIndex(): =20 urllib.request.install_opener(opener) =20 - logger.debug(1, "Fetching %s (%s)..." % (url, ["without authentica= tion", "with authentication"][bool(username)])) + logger.debug("Fetching %s (%s)..." % (url, ["without authenticatio= n", "with authentication"][bool(username)])) =20 try: res =3D urlopen(Request(url, headers=3D{'User-Agent': 'Mozilla= /5.0 (bitbake/lib/layerindex)'}, unverifiable=3DTrue)) except urllib.error.HTTPError as e: - logger.debug(1, "HTTP Error: %s: %s" % (e.code, e.reason)) - logger.debug(1, " Requested: %s" % (url)) - logger.debug(1, " Actual: %s" % (e.geturl())) + logger.debug("HTTP Error: %s: %s" % (e.code, e.reason)) + logger.debug(" Requested: %s" % (url)) + logger.debug(" Actual: %s" % (e.geturl())) =20 if e.code =3D=3D 404: - logger.debug(1, "Request not found.") + logger.debug("Request not found.") raise LayerIndexFetchError(url, e) else: - logger.debug(1, "Headers:\n%s" % (e.headers)) + logger.debug("Headers:\n%s" % (e.headers)) raise LayerIndexFetchError(url, e) except OSError as e: error =3D 0 @@ -169,7 +170,7 @@ class LayerIndex(): raise LayerIndexFetchError(url, "Unable to fetch OSError e= xception: %s" % e) =20 finally: - logger.debug(1, "...fetching %s (%s), done." % (url, ["without= authentication", "with authentication"][bool(username)])) + logger.debug("...fetching %s (%s), done." % (url, ["without au= thentication", "with authentication"][bool(username)])) =20 return res =20 @@ -204,14 +205,14 @@ The format of the indexURI: if reload: self.indexes =3D [] =20 - logger.debug(1, 'Loading: %s' % indexURI) + logger.debug('Loading: %s' % indexURI) =20 if not self.plugins: raise LayerIndexException("No LayerIndex Plugins available") =20 for plugin in self.plugins: # Check if the plugin was initialized - logger.debug(1, 'Trying %s' % plugin.__class__) + logger.debug('Trying %s' % plugin.__class__) if not hasattr(plugin, 'type') or not plugin.type: continue try: @@ -219,11 +220,11 @@ The format of the indexURI: indexEnt =3D plugin.load_index(indexURI, load) break except LayerIndexPluginUrlError as e: - logger.debug(1, "%s doesn't support %s" % (plugin.type, e.= url)) + logger.debug("%s doesn't support %s" % (plugin.type, e.url= )) except NotImplementedError: pass else: - logger.debug(1, "No plugins support %s" % indexURI) + logger.debug("No plugins support %s" % indexURI) raise LayerIndexException("No plugins support %s" % indexURI) =20 # Mark CONFIG data as something we've added... @@ -254,20 +255,20 @@ will write out the individual elements split by layer= and related components. =20 for plugin in self.plugins: # Check if the plugin was initialized - logger.debug(1, 'Trying %s' % plugin.__class__) + logger.debug('Trying %s' % plugin.__class__) if not hasattr(plugin, 'type') or not plugin.type: continue try: plugin.store_index(indexURI, index) break except LayerIndexPluginUrlError as e: - logger.debug(1, "%s doesn't support %s" % (plugin.type, e.= url)) + logger.debug("%s doesn't support %s" % (plugin.type, e.url= )) except NotImplementedError: - logger.debug(1, "Store not implemented in %s" % plugin.typ= e) + logger.debug("Store not implemented in %s" % plugin.type) pass else: - logger.debug(1, "No plugins support %s" % url) - raise LayerIndexException("No plugins support %s" % url) + logger.debug("No plugins support %s" % indexURI) + raise LayerIndexException("No plugins support %s" % indexURI) =20 =20 def is_empty(self): @@ -291,7 +292,7 @@ layerBranches set. If not, they are effectively blank.= ''' the default configuration until the first vcs_url/branch match.= ''' =20 for index in self.indexes: - logger.debug(1, ' searching %s' % index.config['DESCRIPTION']) + logger.debug(' searching %s' % index.config['DESCRIPTION']) layerBranch =3D index.find_vcs_url(vcs_url, [branch]) if layerBranch: return layerBranch @@ -303,7 +304,7 @@ layerBranches set. If not, they are effectively blank.= ''' If a branch has not been specified, we will iterate over the br= anches in the default configuration until the first collection/branch mat= ch.''' =20 - logger.debug(1, 'find_collection: %s (%s) %s' % (collection, versi= on, branch)) + logger.debug('find_collection: %s (%s) %s' % (collection, version,= branch)) =20 if branch: branches =3D [branch] @@ -311,12 +312,12 @@ layerBranches set. If not, they are effectively blan= k.''' branches =3D None =20 for index in self.indexes: - logger.debug(1, ' searching %s' % index.config['DESCRIPTION']) + logger.debug(' searching %s' % index.config['DESCRIPTION']) layerBranch =3D index.find_collection(collection, version, bra= nches) if layerBranch: return layerBranch else: - logger.debug(1, 'Collection %s (%s) not found for branch (%s)'= % (collection, version, branch)) + logger.debug('Collection %s (%s) not found for branch (%s)' % = (collection, version, branch)) return None =20 def find_layerbranch(self, name, branch=3DNone): @@ -407,7 +408,7 @@ layerBranches set. If not, they are effectively blank.= ''' version=3Ddeplayerbranch.ver= sion ) if rdeplayerbranch !=3D deplayerbranch: - logger.debug(1, 'Replaced %s:%s:%s with %s= :%s:%s' % \ + logger.debug('Replaced %s:%s:%s with %s:%s= :%s' % \ (deplayerbranch.index.config['DESCRI= PTION'], deplayerbranch.branch.name, deplayerbranch.layer.name, @@ -657,7 +658,7 @@ class LayerIndexObj(): if obj.id in self._index[indexname]: if self._index[indexname][obj.id] =3D=3D obj: continue - raise LayerIndexError('Conflict adding object %s(%s) to in= dex' % (indexname, obj.id)) + raise LayerIndexException('Conflict adding object %s(%s) t= o index' % (indexname, obj.id)) self._index[indexname][obj.id] =3D obj =20 def add_raw_element(self, indexname, objtype, rawobjs): @@ -842,11 +843,11 @@ class LayerIndexObj(): =20 def _resolve_dependencies(layerbranches, ignores, dependencies, in= valid): for layerbranch in layerbranches: - if ignores and layerBranch.layer.name in ignores: + if ignores and layerbranch.layer.name in ignores: continue =20 - for layerdependency in layerbranch.index.layerDependencies= _layerBranchId[layerBranch.id]: - deplayerbranch =3D layerDependency.dependency_layerBra= nch + for layerdependency in layerbranch.index.layerDependencies= _layerBranchId[layerbranch.id]: + deplayerbranch =3D layerdependency.dependency_layerBra= nch =20 if ignores and deplayerbranch.layer.name in ignores: continue @@ -1120,7 +1121,7 @@ class LayerBranch(LayerIndexItemObj): @property def branch(self): try: - logger.debug(1, "Get branch object from branches[%s]" % (self.= branch_id)) + logger.debug("Get branch object from branches[%s]" % (self.bra= nch_id)) return self.index.branches[self.branch_id] except KeyError: raise AttributeError('Unable to find branches in index to map = branch_id %s' % self.branch_id) @@ -1148,7 +1149,7 @@ class LayerBranch(LayerIndexItemObj): =20 @actual_branch.setter def actual_branch(self, value): - logger.debug(1, "Set actual_branch to %s .. name is %s" % (value, = self.branch.name)) + logger.debug("Set actual_branch to %s .. name is %s" % (value, sel= f.branch.name)) if value !=3D self.branch.name: self._setattr('actual_branch', value, prop=3DFalse) else: diff --git a/bitbake/lib/layerindexlib/cooker.py b/bitbake/lib/layerindexli= b/cooker.py index 65b23d08..2de6e5fa 100644 --- a/bitbake/lib/layerindexlib/cooker.py +++ b/bitbake/lib/layerindexlib/cooker.py @@ -4,6 +4,7 @@ # =20 import logging +import os =20 from collections import defaultdict =20 @@ -73,7 +74,7 @@ class CookerPlugin(layerindexlib.plugin.IndexPlugin): d =3D self.layerindex.data =20 if not branches: - raise LayerIndexFetchError("No branches specified for _load_bb= layers!") + raise layerindexlib.LayerIndexFetchError("No branches specifie= d for _load_bblayers!") =20 index =3D layerindexlib.LayerIndexObj() =20 @@ -172,7 +173,7 @@ class CookerPlugin(layerindexlib.plugin.IndexPlugin): else: branches =3D ['HEAD'] =20 - logger.debug(1, "Loading cooker data branches %s" % branches) + logger.debug("Loading cooker data branches %s" % branches) =20 index =3D self._load_bblayers(branches=3Dbranches) =20 @@ -202,7 +203,7 @@ class CookerPlugin(layerindexlib.plugin.IndexPlugin): try: depDict =3D bb.utils.explode_dep_versions2(deps) except bb.utils.VersionStringException as vse: - bb.fatal('Error parsing LAYERDEPENDS_%s: %s' % (c,= str(vse))) + bb.fatal('Error parsing LAYERDEPENDS_%s: %s' % (co= llection, str(vse))) =20 for dep, oplist in list(depDict.items()): # We need to search ourselves, so use the _ versio= n... @@ -219,7 +220,7 @@ class CookerPlugin(layerindexlib.plugin.IndexPlugin): required=3Drequired, layerbranch= =3DlayerBranchId, dependency=3DdepLayerBranch.layer_= id) =20 - logger.debug(1, '%s requires %s' % (layerDependenc= y.layer.name, layerDependency.dependency.name)) + logger.debug('%s requires %s' % (layerDependency.l= ayer.name, layerDependency.dependency.name)) index.add_element("layerDependencies", [layerDepen= dency]) =20 return layerDependencyId @@ -268,7 +269,7 @@ class CookerPlugin(layerindexlib.plugin.IndexPlugin): =20 layer =3D bb.utils.get_file_layer(realfn[0], self.conf= ig_data) =20 - depBranchId =3D collection_layerbranch[layer] + depBranchId =3D collection[layer] =20 recipeId +=3D 1 recipe =3D layerindexlib.Recipe(index, None) diff --git a/bitbake/lib/layerindexlib/restapi.py b/bitbake/lib/layerindexl= ib/restapi.py index 21fd1441..26a1c967 100644 --- a/bitbake/lib/layerindexlib/restapi.py +++ b/bitbake/lib/layerindexlib/restapi.py @@ -5,9 +5,13 @@ =20 import logging import json +import os + from urllib.parse import unquote from urllib.parse import urlparse =20 +import bb + import layerindexlib import layerindexlib.plugin =20 @@ -78,7 +82,7 @@ class RestApiPlugin(layerindexlib.plugin.IndexPlugin): =20 =20 def load_cache(path, index, branches=3D[]): - logger.debug(1, 'Loading json file %s' % path) + logger.debug('Loading json file %s' % path) with open(path, 'rt', encoding=3D'utf-8') as f: pindex =3D json.load(f) =20 @@ -98,7 +102,7 @@ class RestApiPlugin(layerindexlib.plugin.IndexPlugin): if newpBranch: index.add_raw_element('branches', layerindexlib.Branch, ne= wpBranch) else: - logger.debug(1, 'No matching branches (%s) in index file(s= )' % branches) + logger.debug('No matching branches (%s) in index file(s)' = % branches) # No matching branches.. return nothing... return =20 @@ -116,7 +120,7 @@ class RestApiPlugin(layerindexlib.plugin.IndexPlugin): load_cache(up.path, index, branches) return index =20 - logger.debug(1, 'Loading from dir %s...' % (up.path)) + logger.debug('Loading from dir %s...' % (up.path)) for (dirpath, _, filenames) in os.walk(up.path): for filename in filenames: if not filename.endswith('.json'): @@ -140,7 +144,7 @@ class RestApiPlugin(layerindexlib.plugin.IndexPlugin): def _get_json_response(apiurl=3DNone, username=3DNone, password=3D= None, retry=3DTrue): assert apiurl is not None =20 - logger.debug(1, "fetching %s" % apiurl) + logger.debug("fetching %s" % apiurl) =20 up =3D urlparse(apiurl) =20 @@ -159,11 +163,11 @@ class RestApiPlugin(layerindexlib.plugin.IndexPlugin): parsed =3D json.loads(res.read().decode('utf-8')) except ConnectionResetError: if retry: - logger.debug(1, "%s: Connection reset by peer. Retryi= ng..." % url) + logger.debug("%s: Connection reset by peer. Retrying.= .." % url) parsed =3D _get_json_response(apiurl=3Dup_stripped.get= url(), username=3Dusername, password=3Dpassword, retry=3DFalse) - logger.debug(1, "%s: retry successful.") + logger.debug("%s: retry successful.") else: - raise LayerIndexFetchError('%s: Connection reset by pe= er. Is there a firewall blocking your connection?' % apiurl) + raise layerindexlib.LayerIndexFetchError('%s: Connecti= on reset by peer. Is there a firewall blocking your connection?' % apiurl) =20 return parsed =20 @@ -203,25 +207,25 @@ class RestApiPlugin(layerindexlib.plugin.IndexPlugin): if "*" not in branches: filter =3D "?filter=3Dname:%s" % "OR".join(branches) =20 - logger.debug(1, "Loading %s from %s" % (branches, index.apilinks['= branches'])) + logger.debug("Loading %s from %s" % (branches, index.apilinks['bra= nches'])) =20 # The link won't include username/password, so pull it from the or= iginal url pindex['branches'] =3D _get_json_response(index.apilinks['branches= '] + filter, username=3Dup.username= , password=3Dup.password) if not pindex['branches']: - logger.debug(1, "No valid branches (%s) found at url %s." % (b= ranch, url)) + logger.debug("No valid branches (%s) found at url %s." % (bran= ch, url)) return index index.add_raw_element("branches", layerindexlib.Branch, pindex['br= anches']) =20 # Load all of the layerItems (these can not be easily filtered) - logger.debug(1, "Loading %s from %s" % ('layerItems', index.apilin= ks['layerItems'])) + logger.debug("Loading %s from %s" % ('layerItems', index.apilinks[= 'layerItems'])) =20 =20 # The link won't include username/password, so pull it from the or= iginal url pindex['layerItems'] =3D _get_json_response(index.apilinks['layerI= tems'], username=3Dup.username, = password=3Dup.password) if not pindex['layerItems']: - logger.debug(1, "No layers were found at url %s." % (url)) + logger.debug("No layers were found at url %s." % (url)) return index index.add_raw_element("layerItems", layerindexlib.LayerItem, pinde= x['layerItems']) =20 @@ -231,13 +235,13 @@ class RestApiPlugin(layerindexlib.plugin.IndexPlugin): for branch in index.branches: filter =3D "?filter=3Dbranch__name:%s" % index.branches[branch= ].name =20 - logger.debug(1, "Loading %s from %s" % ('layerBranches', index= .apilinks['layerBranches'])) + logger.debug("Loading %s from %s" % ('layerBranches', index.ap= ilinks['layerBranches'])) =20 # The link won't include username/password, so pull it from th= e original url pindex['layerBranches'] =3D _get_json_response(index.apilinks[= 'layerBranches'] + filter, username=3Dup.username, = password=3Dup.password) if not pindex['layerBranches']: - logger.debug(1, "No valid layer branches (%s) found at url= %s." % (branches or "*", url)) + logger.debug("No valid layer branches (%s) found at url %s= ." % (branches or "*", url)) return index index.add_raw_element("layerBranches", layerindexlib.LayerBran= ch, pindex['layerBranches']) =20 @@ -252,7 +256,7 @@ class RestApiPlugin(layerindexlib.plugin.IndexPlugin): ("distros", layerindexlib.Distro)]: if lName not in load: continue - logger.debug(1, "Loading %s from %s" % (lName, index.apili= nks[lName])) + logger.debug("Loading %s from %s" % (lName, index.apilinks= [lName])) =20 # The link won't include username/password, so pull it fro= m the original url pindex[lName] =3D _get_json_response(index.apilinks[lName]= + filter, @@ -279,7 +283,7 @@ class RestApiPlugin(layerindexlib.plugin.IndexPlugin): if up.scheme !=3D 'file': raise layerindexlib.plugin.LayerIndexPluginUrlError(self.type,= url) =20 - logger.debug(1, "Storing to %s..." % up.path) + logger.debug("Storing to %s..." % up.path) =20 try: layerbranches =3D index.layerBranches @@ -295,12 +299,12 @@ class RestApiPlugin(layerindexlib.plugin.IndexPlugin): if getattr(index, objects)[obj].layerbranch_id =3D=3D = layerbranchid: filtered.append(getattr(index, objects)[obj]._data) except AttributeError: - logger.debug(1, 'No obj.layerbranch_id: %s' % objects) + logger.debug('No obj.layerbranch_id: %s' % objects) # No simple filter method, just include it... try: filtered.append(getattr(index, objects)[obj]._data) except AttributeError: - logger.debug(1, 'No obj._data: %s %s' % (objects, = type(obj))) + logger.debug('No obj._data: %s %s' % (objects, typ= e(obj))) filtered.append(obj) return filtered =20 diff --git a/bitbake/lib/layerindexlib/tests/cooker.py b/bitbake/lib/layeri= ndexlib/tests/cooker.py index 1d0685e0..5ddf89aa 100644 --- a/bitbake/lib/layerindexlib/tests/cooker.py +++ b/bitbake/lib/layerindexlib/tests/cooker.py @@ -72,7 +72,7 @@ class LayerIndexCookerTest(LayersTest): =20 def test_find_collection(self): def _check(collection, expected): - self.logger.debug(1, "Looking for collection %s..." % collecti= on) + self.logger.debug("Looking for collection %s..." % collection) result =3D self.layerindex.find_collection(collection) if expected: self.assertIsNotNone(result, msg=3D"Did not find %s when i= t shouldn't be there" % collection) @@ -91,7 +91,7 @@ class LayerIndexCookerTest(LayersTest): =20 def test_find_layerbranch(self): def _check(name, expected): - self.logger.debug(1, "Looking for layerbranch %s..." % name) + self.logger.debug("Looking for layerbranch %s..." % name) result =3D self.layerindex.find_layerbranch(name) if expected: self.assertIsNotNone(result, msg=3D"Did not find %s when i= t shouldn't be there" % collection) diff --git a/bitbake/lib/layerindexlib/tests/restapi.py b/bitbake/lib/layer= indexlib/tests/restapi.py index e5ccafe5..33b5c1c4 100644 --- a/bitbake/lib/layerindexlib/tests/restapi.py +++ b/bitbake/lib/layerindexlib/tests/restapi.py @@ -57,11 +57,11 @@ class LayerIndexWebRestApiTest(LayersTest): type in self.layerindex.indexes[0].config['local']: continue for id in getattr(self.layerindex.indexes[0], type): - self.logger.debug(1, "type %s" % (type)) + self.logger.debug("type %s" % (type)) =20 self.assertTrue(id in getattr(reload.indexes[0], type), ms= g=3D"Id number not in reloaded index") =20 - self.logger.debug(1, "%s ? %s" % (getattr(self.layerindex.= indexes[0], type)[id], getattr(reload.indexes[0], type)[id])) + self.logger.debug("%s ? %s" % (getattr(self.layerindex.ind= exes[0], type)[id], getattr(reload.indexes[0], type)[id])) =20 self.assertEqual(getattr(self.layerindex.indexes[0], type)= [id], getattr(reload.indexes[0], type)[id], msg=3D"Reloaded contents differ= ent") =20 @@ -80,11 +80,11 @@ class LayerIndexWebRestApiTest(LayersTest): type in self.layerindex.indexes[0].config['local']: continue for id in getattr(self.layerindex.indexes[0] ,type): - self.logger.debug(1, "type %s" % (type)) + self.logger.debug("type %s" % (type)) =20 self.assertTrue(id in getattr(reload.indexes[0], type), ms= g=3D"Id number missing from reloaded data") =20 - self.logger.debug(1, "%s ? %s" % (getattr(self.layerindex.= indexes[0] ,type)[id], getattr(reload.indexes[0], type)[id])) + self.logger.debug("%s ? %s" % (getattr(self.layerindex.ind= exes[0] ,type)[id], getattr(reload.indexes[0], type)[id])) =20 self.assertEqual(getattr(self.layerindex.indexes[0] ,type)= [id], getattr(reload.indexes[0], type)[id], msg=3D"reloaded data does not m= atch original") =20 @@ -111,14 +111,14 @@ class LayerIndexWebRestApiTest(LayersTest): if dep.layer.name =3D=3D 'meta-python': break else: - self.logger.debug(1, "meta-python was not found") - self.assetTrue(False) + self.logger.debug("meta-python was not found") + raise self.failureException =20 # Only check the first element... break else: # Empty list, this is bad. - self.logger.debug(1, "Empty list of dependencies") + self.logger.debug("Empty list of dependencies") self.assertIsNotNone(first, msg=3D"Empty list of dependencies") =20 # Last dep should be the requested item @@ -128,7 +128,7 @@ class LayerIndexWebRestApiTest(LayersTest): @skipIfNoNetwork() def test_find_collection(self): def _check(collection, expected): - self.logger.debug(1, "Looking for collection %s..." % collecti= on) + self.logger.debug("Looking for collection %s..." % collection) result =3D self.layerindex.find_collection(collection) if expected: self.assertIsNotNone(result, msg=3D"Did not find %s when i= t should be there" % collection) @@ -148,11 +148,11 @@ class LayerIndexWebRestApiTest(LayersTest): @skipIfNoNetwork() def test_find_layerbranch(self): def _check(name, expected): - self.logger.debug(1, "Looking for layerbranch %s..." % name) + self.logger.debug("Looking for layerbranch %s..." % name) =20 for index in self.layerindex.indexes: for layerbranchid in index.layerBranches: - self.logger.debug(1, "Present: %s" % index.layerBranch= es[layerbranchid].layer.name) + self.logger.debug("Present: %s" % index.layerBranches[= layerbranchid].layer.name) result =3D self.layerindex.find_layerbranch(name) if expected: self.assertIsNotNone(result, msg=3D"Did not find %s when i= t should be there" % collection) diff --git a/bitbake/lib/ply/lex.py b/bitbake/lib/ply/lex.py index 267ec100..182f2e83 100644 --- a/bitbake/lib/ply/lex.py +++ b/bitbake/lib/ply/lex.py @@ -705,11 +705,7 @@ class LexerReflect(object): =20 # Sort the functions by line number for f in self.funcsym.values(): - if sys.version_info[0] < 3: - f.sort(lambda x,y: cmp(func_code(x[1]).co_firstlineno,func= _code(y[1]).co_firstlineno)) - else: - # Python 3.0 - f.sort(key=3Dlambda x: func_code(x[1]).co_firstlineno) + f.sort(key=3Dlambda x: func_code(x[1]).co_firstlineno) =20 # Sort the strings by regular expression length for s in self.strsym.values(): diff --git a/bitbake/lib/ply/yacc.py b/bitbake/lib/ply/yacc.py index 561784f2..46e7dc96 100644 --- a/bitbake/lib/ply/yacc.py +++ b/bitbake/lib/ply/yacc.py @@ -1205,7 +1205,7 @@ class Production(object): =20 # Precompute the list of productions immediately following. Hack.= Remove later try: - p.lr_after =3D Prodnames[p.prod[n+1]] + p.lr_after =3D self.Prodnames[p.prod[n+1]] except (IndexError,KeyError): p.lr_after =3D [] try: diff --git a/bitbake/lib/toaster/orm/fixtures/oe-core.xml b/bitbake/lib/toa= ster/orm/fixtures/oe-core.xml index fd93f4d8..026d9486 100644 --- a/bitbake/lib/toaster/orm/fixtures/oe-core.xml +++ b/bitbake/lib/toaster/orm/fixtures/oe-core.xml @@ -23,9 +23,9 @@ master - zeus + gatesgarth git://git.openembedded.org/b= itbake - 1.44 + 1.48 =20 @@ -51,11 +51,11 @@ Toaster will run your buil= ds using the tip of the <a href=3D\"http://cgit.openembedded.org/openemb= edded-core/log/\">OpenEmbedded master</a> branch. - zeus - Openembedded Zeus + gatesgarth + Openembedded Gatesgarth= 4 - zeus - Toaster will run your buil= ds using the tip of the <a href=3D\"http://cgit.openembedded.org/openemb= edded-core/log/?h=3Dzeus\">OpenEmbedded Zeus</a> branch. + gatesgarth + Toaster will run your buil= ds using the tip of the <a href=3D\"http://cgit.openembedded.org/openemb= edded-core/log/?h=3Dgatesgarth\">OpenEmbedded Gatesgarth</a> branc= h. =20 diff --git a/bitbake/lib/toaster/orm/fixtures/poky.xml b/bitbake/lib/toaste= r/orm/fixtures/poky.xml index 902bc88a..a468a54c 100644 --- a/bitbake/lib/toaster/orm/fixtures/poky.xml +++ b/bitbake/lib/toaster/orm/fixtures/poky.xml @@ -26,9 +26,9 @@ bitbake - zeus + gatesgarth git://git.yoctoproject.org/p= oky - zeus + gatesgarth bitbake =20 @@ -56,11 +56,11 @@ Toaster will run your buil= ds using the tip of the <a href=3D"http://git.yoctoproject.org/cgit/cgit= .cgi/poky/log/">Yocto Project Master branch</a>. - zeus - Yocto Project 3.0 "Zeus= " + gatesgarth + Yocto Project 3.2 "Gate= sgarth" 4 - zeus - Toaster will run your buil= ds using the tip of the <a href=3D"http://git.yoctoproject.org/cgit/cgit= .cgi/poky/log/?h=3Dzeus">Yocto Project Zeus branch</a>. + gatesgarth + Toaster will run your buil= ds using the tip of the <a href=3D"http://git.yoctoproject.org/cgit/cgit= .cgi/poky/log/?h=3Dgatesgarth">Yocto Project Gatesgarth branch</a>= . =20 @@ -152,7 +152,7 @@ 1 0 4 - zeus + gatesgarth meta =20 @@ -190,7 +190,7 @@ 2 0 4 - zeus + gatesgarth meta-poky =20 @@ -228,7 +228,7 @@ 3 0 4 - zeus + gatesgarth meta-yocto-bsp diff --git a/bitbake/lib/toaster/orm/management/commands/lsupdates.py b/bit= bake/lib/toaster/orm/management/commands/lsupdates.py index a4dbcaa7..2fbd7be3 100644 --- a/bitbake/lib/toaster/orm/management/commands/lsupdates.py +++ b/bitbake/lib/toaster/orm/management/commands/lsupdates.py @@ -68,7 +68,7 @@ class Command(BaseCommand): (what, pec)) sys.stdout.flush() - if int(pec) is 100: + if int(pec) =3D=3D 100: sys.stdout.write("\n") sys.stdout.flush() =20 diff --git a/bitbake/lib/toaster/tests/functional/functional_helpers.py b/b= itbake/lib/toaster/tests/functional/functional_helpers.py index 455c408e..5c4ea717 100644 --- a/bitbake/lib/toaster/tests/functional/functional_helpers.py +++ b/bitbake/lib/toaster/tests/functional/functional_helpers.py @@ -75,7 +75,7 @@ class SeleniumFunctionalTestCase(SeleniumTestCaseBase): try: table_element =3D self.get_table_element(table_id) element =3D table_element.find_element_by_link_text(link_text) - except NoSuchElementException as e: + except self.NoSuchElementException: print('no element found') raise return element @@ -86,7 +86,7 @@ class SeleniumFunctionalTestCase(SeleniumTestCaseBase): element_xpath =3D "//*[@id=3D'" + table_id + "']" try: element =3D self.driver.find_element_by_xpath(element_xpat= h) - except NoSuchElementException as e: + except self.NoSuchElementException: raise return element row =3D coordinate[0] @@ -96,7 +96,7 @@ class SeleniumFunctionalTestCase(SeleniumTestCaseBase): element_xpath =3D "//*[@id=3D'" + table_id + "']/tbody/tr[" + = str(row) + "]" try: element =3D self.driver.find_element_by_xpath(element_xpat= h) - except NoSuchElementException as e: + except self.NoSuchElementException: return False return element #now we are looking for an element with specified X and Y @@ -105,6 +105,6 @@ class SeleniumFunctionalTestCase(SeleniumTestCaseBase): element_xpath =3D "//*[@id=3D'" + table_id + "']/tbody/tr[" + str(= row) + "]/td[" + str(column) + "]" try: element =3D self.driver.find_element_by_xpath(element_xpath) - except NoSuchElementException as e: + except self.NoSuchElementException: return False return element diff --git a/bitbake/lib/toaster/toastergui/templates/base.html b/bitbake/l= ib/toaster/toastergui/templates/base.html index 4f720648..9e19cc33 100644 --- a/bitbake/lib/toaster/toastergui/templates/base.html +++ b/bitbake/lib/toaster/toastergui/templates/base.html @@ -123,7 +123,7 @@ {% endif %} {% endif %}
  • - + Documentation diff --git a/bitbake/lib/toaster/toastergui/templates/configvars.html b/bit= bake/lib/toaster/toastergui/templates/configvars.html index ca2e1eab..33fef931 100644 --- a/bitbake/lib/toaster/toastergui/templates/configvars.html +++ b/bitbake/lib/toaster/toastergui/templates/configvars.html @@ -66,7 +66,7 @@ {% if variable.description %} {{variable.description}} - + {% endif %} diff --git a/bitbake/lib/toaster/toastergui/templates/landing.html b/bitbak= e/lib/toaster/toastergui/templates/landing.html index 70c7359f..bfaaf6fc 100644 --- a/bitbake/lib/toaster/toastergui/templates/landing.html +++ b/bitbake/lib/toaster/toastergui/templates/landing.html @@ -12,10 +12,10 @@

    This is Toaster

    =20 -

    A web interface to OpenEmbedded and BitBake, the Yocto Project build system.

    +

    A web interface to OpenEmbedded and BitBake, the Yocto Project build system.

    =20

    - + Toaster is ready to capture your command line builds

    @@ -33,7 +33,7 @@ Toaster has no layer information. Without layer informat= ion, you cannot run builds. To generate layer information you can:
    diff --git a/bitbake/lib/toaster/toastergui/templates/project_specific.html= b/bitbake/lib/toaster/toastergui/templates/project_specific.html index f625d18b..42725c0d 100644 --- a/bitbake/lib/toaster/toastergui/templates/project_specific.html +++ b/bitbake/lib/toaster/toastergui/templates/project_specific.html @@ -137,7 +137,7 @@ diff --git a/bitbake/lib/toaster/toastergui/templates/projectconf.html b/bi= tbake/lib/toaster/toastergui/templates/projectconf.html index fb20b26f..bd49f1f5 100644 --- a/bitbake/lib/toaster/toastergui/templates/projectconf.html +++ b/bitbake/lib/toaster/toastergui/templates/projectconf.html @@ -201,12 +201,12 @@

    Toaster cannot set any variables that impact 1) the configura= tion of the build servers, or 2) where artifacts produced by the build are stored. Such var= iables include:

    - BB_DISKMON_DIRS - BB_NUMBER_THREA= DS + BB_DISKMON_DIR= S + BB_NUMBER_TH= READS CVS_PROXY_HOST CVS_PROXY_PORT - PARALLEL_MAKE - TMPDIR

    + PARALLEL_MAKE + TMPDIR

    Plus the following standard shell environment variables:

    http_proxy ftp_proxy https_pr= oxy all_proxy

    diff --git a/bitbake/lib/toaster/toastermain/management/commands/perf.py b/= bitbake/lib/toaster/toastermain/management/commands/perf.py index 7d629fb3..5c41c5b2 100644 --- a/bitbake/lib/toaster/toastermain/management/commands/perf.py +++ b/bitbake/lib/toaster/toastermain/management/commands/perf.py @@ -17,46 +17,46 @@ class Command(BaseCommand): help =3D "Test the response time for all toaster urls" =20 def handle(self, *args, **options): - root_urlconf =3D __import__(settings.ROOT_URLCONF) - patterns =3D root_urlconf.urls.urlpatterns - global full_url - for pat in patterns: - if pat.__class__.__name__ =3D=3D 'RegexURLResolver': - url_root_res =3D str(pat).split('^')[1].replace('>', '') - if 'gui' in url_root_res: - for url_patt in pat.url_patterns: - full_url =3D self.get_full_url(url_patt, url_root_res) - info =3D self.url_info(full_url) - status_code =3D info[0] - load_time =3D info[1] - print('Trying \'' + full_url + '\', ' + str(status_code) + ', ' + str(l= oad_time)) + root_urlconf =3D __import__(settings.ROOT_URLCONF) + patterns =3D root_urlconf.urls.urlpatterns + global full_url + for pat in patterns: + if pat.__class__.__name__ =3D=3D 'RegexURLResolver': + url_root_res =3D str(pat).split('^')[1].replace('>', '') + if 'gui' in url_root_res: + for url_patt in pat.url_patterns: + full_url =3D self.get_full_url(url_patt, url_root_res) + info =3D self.url_info(full_url) + status_code =3D info[0] + load_time =3D info[1] + print('Trying \'' + full_url + '\', ' + str(status_cod= e) + ', ' + str(load_time)) =20 def get_full_url(self, url_patt, url_root_res): - full_url =3D str(url_patt).split('^')[1].replace('$>', '').replace('(?P(?:/[', '/bin/busybox').replace('.*', '') - full_url =3D str(url_root_res + full_url) - full_url =3D re.sub('\(\?P<.*?>\\\d\+\)', '1', full_url) - full_url =3D 'http://localhost:8000/' + full_url - return full_url + full_url =3D str(url_patt).split('^')[1].replace('$>', '').replace= ('(?P(?:/[', '/bin/busybox').replace('.*', '') + full_url =3D str(url_root_res + full_url) + full_url =3D re.sub('\(\?P<.*?>\\\d\+\)', '1', full_url) + full_url =3D 'http://localhost:8000/' + full_url + return full_url =20 def url_info(self, full_url): - client =3D Client() - info =3D [] - try: - resp =3D client.get(full_url, follow =3D True) - except Exception as e_status_code: + client =3D Client() + info =3D [] + try: + resp =3D client.get(full_url, follow =3D True) + except Exception as e_status_code: self.error('Url: %s, error: %s' % (full_url, e_status_code)) resp =3D type('object', (), {'status_code':0, 'content': str(e= _status_code)}) - status_code =3D resp.status_code - info.append(status_code) - try: - req =3D requests.get(full_url) - except Exception as e_load_time: + status_code =3D resp.status_code + info.append(status_code) + try: + req =3D requests.get(full_url) + except Exception as e_load_time: self.error('Url: %s, error: %s' % (full_url, e_load_time)) - load_time =3D req.elapsed - info.append(load_time) - return info + load_time =3D req.elapsed + info.append(load_time) + return info =20 def error(self, *args): - for arg in args: - print(arg, end=3D' ', file=3Dsys.stderr) - print(file=3Dsys.stderr) + for arg in args: + print(arg, end=3D' ', file=3Dsys.stderr) + print(file=3Dsys.stderr) --=20 2.20.1