From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7128378901610364928 X-Received: by 2002:a92:da8e:0:b0:2de:2733:a389 with SMTP id u14-20020a92da8e000000b002de2733a389mr2862075iln.316.1659705047988; Fri, 05 Aug 2022 06:10:47 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a02:8664:0:b0:342:7a46:3e8c with SMTP id e91-20020a028664000000b003427a463e8cls840247jai.8.-pod-prod-gmail; Fri, 05 Aug 2022 06:10:47 -0700 (PDT) X-Google-Smtp-Source: AA6agR5iq4tpbExJFBoUlkNoJv7WEZEtmkXo8WEIhNwgElt8OWpQR4BXD6nkLOdeznmWJ0LaXKlm X-Received: by 2002:a05:6638:2605:b0:341:5898:eb1f with SMTP id m5-20020a056638260500b003415898eb1fmr3145739jat.304.1659705047291; Fri, 05 Aug 2022 06:10:47 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1659705047; cv=none; d=google.com; s=arc-20160816; b=F2Bu5vzBbsIPkRT91jGacxkVg8jWAQU1+lwELBzuTg/fIoMqd6GLrLpNK69aeth0Dv AwyefI9BMgL/P5YL6hg0N4ebNNOP0QQnl2Pi6a84In8ZcvVOkrbhqAhFEVxftUIADDzw shj88QEfEc5ARf+RACeiPUFQdXI7KoyZd5mcsKhlcjEzH5CvpBWWcGEHDzjYprfPhOVi k3O6HnfvUPGA43sazNa7B5Gk79vkKNhzoTTaSp6jiT60eHKC0Jea6FIsRnsiQkLr4ftk V9ccGo2itSliwSINDo2Tf8bZZ9EQT7jnnpv7dhzy+nZpAztJBsBBKNcRJmmOK5W4tvyk BusA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:date:subject:cc:to:from; bh=8rA7GeC4L3kOolJojUMrwq5H+gXpr3lTcVcLHkrJ5vc=; b=zceGaihc7QhqyvyUzN7Huk/kthuptVAxtNQ1Z/TQymFgPPwVZJYk8HmDDOM9NukZB3 1miG/AU2i6xjQM6zoiQ2LO+KYPcL02+5GbohdU5kUPf78Y2jWJ0YMjAPYWWoC2wFbLt9 yuov+bcPkiLlRMQ+xknuA1weM2wAVmOkRSoYBfK6vYG5XkBiE4Dy5ZSA6Z7lgq/DDDxy EJKfz5ah/5nyE/0urT9SViQlJ6yK0W+MvEMVEi8S8oqU29kUaMUq/oJhZCRCma/pUyZC J0XpcSPG+1/s8uRn6HUV20atX7syG3a89PhnJQhusEmLf74exjVOOStRVpYFvdOukkKx mN2w== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id l10-20020a05660227ca00b0067898a22fbfsi194253ios.3.2022.08.05.06.10.46 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Fri, 05 Aug 2022 06:10:47 -0700 (PDT) Received-SPF: pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of amikan@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=amikan@ilbers.de Received: from alena-nb.promwad.com (mm-183-76-214-37.mgts.dynamic.pppoe.byfly.by [37.214.76.183] (may be forged)) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 275DAhHk009685 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 5 Aug 2022 15:10:44 +0200 From: Anton Mikanovich To: isar-users@googlegroups.com Cc: Anton Mikanovich Subject: [PATCH v4 00/21] Migrate to Bitbake 2.0 Date: Fri, 5 Aug 2022 16:10:14 +0300 Message-Id: <20220805131035.22844-1-amikan@ilbers.de> X-Mailer: git-send-email 2.17.1 X-Spam-Status: No, score=-0.5 required=5.0 tests=ALL_TRUSTED,URI_NOVOWEL autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: NRMgVNa9w+ik This patchset moves Isar to use Bitbake 2.0 branch. scripts/contrib/convert-overrides.py is the script for overrides update and it needs some testing on downstream layers. Usage: $ ./scripts/contrib/convert-overrides.py meta-customlayer Changes since v3: - Rebase on next - Add network flag for do_containerize - Update imagetypes variables in override script - Update RECIPE-API-CHANGELOG Changes since v2: - Fixed building of stretch targets Changes since v1: - Rebased on top of sbuild - Resolved 'metadata is not deterministic' issues - Update sstate class and maintenance script - Update OE classes - Adopt CI configuration Anton Mikanovich (21): meta: change deprecated parse calls scripts/contrib: Add override conversion script scripts/contrib: configure override conversion script meta-isar: set default branch names meta: remove non recommended syntax bitbake: Update to Bitbake 2.0.1 doc: require zstd tool meta: update bitbake variables bitbake.conf: align hash vars with openembedded meta: mark network and sudo tasks meta: update overrides syntax sstate: update bbclass bitbake.conf: declare default XZ and ZSTD options Revert "devshell: Use different termination test to avoid warnings" meta: align with OE-core libraries update Revert "Revert "devshell: Use different termination test to avoid warnings"" CI: Adopt tests to syntax change isar-sstate: adopt sstate maintenance script Revert "bitbake: Make 3.6.0 the minimum python version" Revert "utils/ply: Change md5 usages to work on FIPS enabled hosts" RECIPE-API-CHANGELOG: Add tips after bitbake version update RECIPE-API-CHANGELOG.md | 14 + bitbake/README | 21 +- bitbake/bin/bitbake | 4 +- bitbake/bin/bitbake-diffsigs | 5 +- bitbake/bin/bitbake-getvar | 50 ++ bitbake/bin/bitbake-hashclient | 2 + bitbake/bin/bitbake-hashserv | 2 + bitbake/bin/bitbake-layers | 2 + bitbake/bin/bitbake-prserv | 6 +- bitbake/bin/bitbake-selftest | 3 + bitbake/bin/bitbake-server | 3 +- bitbake/bin/bitbake-worker | 21 +- bitbake/bin/git-make-shallow | 2 + bitbake/bin/toaster | 6 +- bitbake/bin/toaster-eventreplay | 2 + bitbake/conf/bitbake.conf | 6 +- bitbake/contrib/hashserv/Dockerfile | 6 +- bitbake/contrib/prserv/Dockerfile | 62 ++ bitbake/contrib/vim/plugin/newbbappend.vim | 2 +- bitbake/contrib/vim/syntax/bitbake.vim | 11 +- bitbake/doc/Makefile | 2 +- bitbake/doc/README | 6 +- .../bitbake-user-manual-execution.rst | 112 ++-- .../bitbake-user-manual-fetching.rst | 151 +++-- .../bitbake-user-manual-hello.rst | 60 +- .../bitbake-user-manual-intro.rst | 64 +- .../bitbake-user-manual-metadata.rst | 446 +++++++------- .../bitbake-user-manual-ref-variables.rst | 583 ++++++++++-------- bitbake/doc/releases.rst | 84 ++- bitbake/lib/bb/__init__.py | 15 +- bitbake/lib/bb/asyncrpc/__init__.py | 31 + bitbake/lib/bb/asyncrpc/client.py | 172 ++++++ bitbake/lib/bb/asyncrpc/serv.py | 286 +++++++++ bitbake/lib/bb/build.py | 90 +-- bitbake/lib/bb/cache.py | 35 +- bitbake/lib/bb/checksum.py | 22 +- bitbake/lib/bb/codeparser.py | 20 +- bitbake/lib/bb/command.py | 36 +- bitbake/lib/bb/compress/_pipecompress.py | 194 ++++++ bitbake/lib/bb/compress/lz4.py | 17 + bitbake/lib/bb/compress/zstd.py | 28 + bitbake/lib/bb/cooker.py | 293 +++++---- bitbake/lib/bb/cookerdata.py | 47 +- bitbake/lib/bb/daemonize.py | 42 +- bitbake/lib/bb/data.py | 57 +- bitbake/lib/bb/data_smart.py | 213 ++++--- bitbake/lib/bb/event.py | 6 +- bitbake/lib/bb/fetch2/README | 57 ++ bitbake/lib/bb/fetch2/__init__.py | 136 ++-- bitbake/lib/bb/fetch2/crate.py | 136 ++++ bitbake/lib/bb/fetch2/git.py | 54 +- bitbake/lib/bb/fetch2/gitsm.py | 20 +- bitbake/lib/bb/fetch2/npm.py | 50 +- bitbake/lib/bb/fetch2/npmsw.py | 33 +- bitbake/lib/bb/fetch2/osc.py | 2 +- bitbake/lib/bb/fetch2/s3.py | 41 +- bitbake/lib/bb/fetch2/ssh.py | 49 +- bitbake/lib/bb/fetch2/svn.py | 10 +- bitbake/lib/bb/fetch2/wget.py | 161 +++-- bitbake/lib/bb/main.py | 9 +- bitbake/lib/bb/monitordisk.py | 17 +- bitbake/lib/bb/msg.py | 32 +- bitbake/lib/bb/parse/__init__.py | 2 + bitbake/lib/bb/parse/ast.py | 14 +- bitbake/lib/bb/parse/parse_py/BBHandler.py | 3 - bitbake/lib/bb/parse/parse_py/ConfHandler.py | 5 +- bitbake/lib/bb/persist_data.py | 54 +- bitbake/lib/bb/process.py | 2 +- bitbake/lib/bb/progress.py | 9 +- bitbake/lib/bb/providers.py | 14 +- bitbake/lib/bb/runqueue.py | 179 +++--- bitbake/lib/bb/server/process.py | 44 +- bitbake/lib/bb/server/xmlrpcserver.py | 1 + bitbake/lib/bb/siggen.py | 182 ++++-- bitbake/lib/bb/taskdata.py | 14 +- bitbake/lib/bb/tests/codeparser.py | 28 +- bitbake/lib/bb/tests/compression.py | 98 +++ bitbake/lib/bb/tests/data.py | 95 +-- .../debian/pool/main/m/minicom/index.html | 59 ++ bitbake/lib/bb/tests/fetch.py | 529 ++++++++++------ bitbake/lib/bb/tests/parse.py | 20 +- .../bb/tests/runqueue-tests/conf/bitbake.conf | 2 +- bitbake/lib/bb/tests/runqueue.py | 52 +- bitbake/lib/bb/tests/utils.py | 20 +- bitbake/lib/bb/tinfoil.py | 12 +- bitbake/lib/bb/ui/buildinfohelper.py | 83 +-- bitbake/lib/bb/ui/knotty.py | 100 +-- bitbake/lib/bb/ui/taskexp.py | 5 + bitbake/lib/bb/ui/uievent.py | 6 +- bitbake/lib/bb/ui/uihelper.py | 4 +- bitbake/lib/bb/utils.py | 116 +++- bitbake/lib/bblayers/action.py | 2 +- bitbake/lib/bblayers/layerindex.py | 16 +- bitbake/lib/bblayers/query.py | 8 +- bitbake/lib/codegen.py | 6 + bitbake/lib/hashserv/__init__.py | 66 +- bitbake/lib/hashserv/client.py | 152 +---- bitbake/lib/hashserv/server.py | 549 ++++++++--------- bitbake/lib/hashserv/tests.py | 161 ++++- bitbake/lib/layerindexlib/__init__.py | 9 +- bitbake/lib/layerindexlib/cooker.py | 2 +- bitbake/lib/layerindexlib/restapi.py | 4 +- bitbake/lib/layerindexlib/tests/restapi.py | 2 +- bitbake/lib/prserv/client.py | 48 ++ bitbake/lib/prserv/db.py | 65 +- bitbake/lib/prserv/serv.py | 540 ++++++---------- bitbake/lib/pyinotify.py | 44 +- .../bldcontrol/localhostbecontroller.py | 4 +- .../management/commands/runbuilds.py | 83 ++- .../migrations/0008_models_bigautofield.py | 48 ++ .../lib/toaster/orm/fixtures/gen_fixtures.py | 445 +++++++++++++ bitbake/lib/toaster/orm/fixtures/oe-core.xml | 48 +- bitbake/lib/toaster/orm/fixtures/poky.xml | 118 ++-- bitbake/lib/toaster/orm/fixtures/settings.xml | 2 +- .../orm/management/commands/lsupdates.py | 14 +- .../migrations/0020_models_bigautofield.py | 173 ++++++ bitbake/lib/toaster/orm/models.py | 5 +- .../toaster/toastergui/templates/base.html | 2 +- .../toastergui/templates/configvars.html | 2 +- .../toaster/toastergui/templates/landing.html | 6 +- .../templates/landing_not_managed.html | 34 - .../toastergui/templates/layerdetails.html | 2 +- .../templates/package_detail_base.html | 2 +- .../toaster/toastergui/templates/project.html | 2 +- .../templates/project_specific.html | 2 +- .../toastergui/templates/projectconf.html | 34 +- bitbake/lib/toaster/toastergui/views.py | 22 +- .../management/commands/buildimport.py | 2 +- bitbake/lib/toaster/toastermain/settings.py | 3 + bitbake/toaster-requirements.txt | 2 +- doc/user_manual.md | 3 +- meta-isar/classes/rpi-sdimg.bbclass | 4 +- meta-isar/classes/ubi-ubifs-img.bbclass | 6 +- meta-isar/conf/distro/raspios-bullseye.conf | 8 +- meta-isar/conf/distro/ubuntu-focal.conf | 4 +- meta-isar/conf/local.conf.sample | 8 +- meta-isar/conf/machine/de0-nano-soc.conf | 4 +- meta-isar/conf/machine/hikey.conf | 2 +- meta-isar/conf/machine/phyboard-mira.conf | 2 +- meta-isar/conf/machine/qemuamd64.conf | 6 +- meta-isar/conf/machine/qemuarm64.conf | 2 +- meta-isar/conf/machine/qemuriscv64.conf | 2 +- meta-isar/conf/machine/stm32mp15x.conf | 4 +- .../conf/multiconfig/qemuamd64-bullseye.conf | 2 +- .../conf/multiconfig/qemuarm-bookworm.conf | 2 +- meta-isar/recipes-app/cowsay/cowsay_git.bb | 2 +- .../recipes-app/hello-isar/hello-isar.bb | 4 +- meta-isar/recipes-app/libhello/libhello.bb | 4 +- .../bootconfig-rpi/bootconfig-rpi.bb | 6 +- .../optee-os/optee-os-stm32mp15x_3.11.0.bb | 2 +- .../linux/linux-cip_4.4.166-cip29.bb | 2 +- .../linux/linux-mainline_5.4.70.bb | 4 +- .../linux/linux-phy_5.10.76-phy4.bb | 2 +- meta/classes/base.bbclass | 1 + meta/classes/dpkg-base.bbclass | 9 +- meta/classes/dpkg-gbp.bbclass | 2 +- meta/classes/dpkg-prebuilt.bbclass | 2 +- meta/classes/dpkg.bbclass | 2 +- meta/classes/image-locales-extension.bbclass | 2 + meta/classes/image-tools-extension.bbclass | 1 + meta/classes/image.bbclass | 38 +- meta/classes/imagetypes.bbclass | 59 +- meta/classes/imagetypes_container.bbclass | 5 +- meta/classes/imagetypes_vm.bbclass | 10 +- meta/classes/imagetypes_wic.bbclass | 5 +- meta/classes/rootfs.bbclass | 7 +- meta/classes/sdk.bbclass | 32 +- meta/classes/sstate.bbclass | 247 ++++---- meta/conf/bitbake.conf | 66 +- meta/conf/distro/debian-common.conf | 16 +- meta/conf/distro/debian-stretch.conf | 4 +- meta/lib/oe/gpg_sign.py | 33 +- meta/lib/oe/maketype.py | 7 +- meta/lib/oe/patch.py | 51 +- meta/lib/oe/path.py | 77 +++ meta/lib/oe/sstatesig.py | 118 ++-- meta/lib/oe/terminal.py | 33 +- meta/lib/oe/utils.py | 65 +- meta/recipes-bsp/barebox/barebox.inc | 2 +- .../libubootenv/libubootenv_0.3-3.bb | 2 +- meta/recipes-bsp/optee-os/optee-os-custom.inc | 2 +- .../trusted-firmware-a-custom.inc | 2 +- meta/recipes-bsp/u-boot/u-boot-custom.inc | 2 +- .../isar-bootstrap/isar-bootstrap.inc | 14 +- .../buildchroot/buildchroot-host.bb | 2 +- .../buildchroot/buildchroot.inc | 6 +- .../sbuild-chroot/sbuild-chroot.inc | 2 +- meta/recipes-kernel/kselftest/kselftest.inc | 6 +- meta/recipes-kernel/linux-module/module.inc | 2 +- meta/recipes-kernel/linux/linux-custom.inc | 10 +- scripts/contrib/convert-overrides.py | 144 +++++ scripts/isar-buildenv-internal | 4 +- scripts/isar-sstate | 17 +- testsuite/cibuilder.py | 8 +- testsuite/citest.py | 2 +- 195 files changed, 6555 insertions(+), 3279 deletions(-) create mode 100755 bitbake/bin/bitbake-getvar create mode 100644 bitbake/contrib/prserv/Dockerfile create mode 100644 bitbake/lib/bb/asyncrpc/__init__.py create mode 100644 bitbake/lib/bb/asyncrpc/client.py create mode 100644 bitbake/lib/bb/asyncrpc/serv.py create mode 100644 bitbake/lib/bb/compress/_pipecompress.py create mode 100644 bitbake/lib/bb/compress/lz4.py create mode 100644 bitbake/lib/bb/compress/zstd.py create mode 100644 bitbake/lib/bb/fetch2/README create mode 100644 bitbake/lib/bb/fetch2/crate.py create mode 100644 bitbake/lib/bb/tests/compression.py create mode 100644 bitbake/lib/bb/tests/fetch-testdata/debian/pool/main/m/minicom/index.html create mode 100644 bitbake/lib/prserv/client.py create mode 100644 bitbake/lib/toaster/bldcontrol/migrations/0008_models_bigautofield.py create mode 100755 bitbake/lib/toaster/orm/fixtures/gen_fixtures.py create mode 100644 bitbake/lib/toaster/orm/migrations/0020_models_bigautofield.py delete mode 100644 bitbake/lib/toaster/toastergui/templates/landing_not_managed.html create mode 100644 scripts/contrib/convert-overrides.py -- 2.17.1