From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Mon, 09 Nov 2026 10:19:47 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-qt1-f185.google.com (mail-qt1-f185.google.com [209.85.160.185]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 6A99Jj17026157 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 9 Nov 2026 10:19:46 +0100 Received: by mail-qt1-f185.google.com with SMTP id d75a77b69052e-544019b1831sf14491171cf.0 for ; Mon, 09 Nov 2026 01:19:46 -0800 (PST) ARC-Seal: i=3; a=rsa-sha256; t=1794215980; cv=pass; d=google.com; s=arc-20260327; b=BouMDH8bRlUDGOdJR3Ctb3SoPPErsu0vXofrCg4tvToUOGV/X/HylRXA5ha0nYR6Xt 9+Hlx+aWIc9Rlg5mg9cpnoeUc8me23D+RPtL+tSBwo+0eG8y7XiB7x1AoflKcuBjG1Jl 7sYLLBSa61UcR1zqOxCyHSyLDSAq2d2W2sTWdpbpK78n8yvygAEbJNCoJKMjHmqD/WUs 3LoRW9w3K+q4ZDJbreTE9uaiXi+N0hBZRorHw4fBzKZp0blLffsHFxiEq6jwwhaCO3es DKnv7US9Y+YeNoanoPbLoTPsYOWEhDcmzRs1IRsEZJ6chVaLpsUqkZtXm8Ry1CESZ0kJ Az4Q== ARC-Message-Signature: i=3; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:mime-version :content-transfer-encoding:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=go0O3LhBPODqIwOuFwJmfZyhfxmswrdel7+1l87nL8k=; fh=H06QSS2ehSMkYrIxUGCZ0QLS5Uw8idmCRpXslajkahw=; b=B1UroQMbBDCD48OzJgmz8n6n+TQhjPtsiI5rZ/TDoNAOkzIGvwZ1SmbXtYXyXmPZRU 1lWcUS/ibshz4mHX8W5E0FJ3prEy6d+Rh/Ip0XNmLiZhTQRd0WMe4ZRfjLz/gEP86esj 2WWoiaxtLsAwjJ7C/sqkskIkNt6uY/ets693unaZOwG98o17srUQd0VvA3GLO9/1r6Gk MCARPb0uszUNtX2QBeuUM+YvswZLa9zdpxQVBjA4pO1ZmbD6ypQ57FJRaoEY1fWnOWfu FYOwEviToFbY33ANFf9q/ZiCS2Ht3k16ritqwieMAQnuxGQ5YxHvci6+qZ5i3+bZQu62 +pog==; darn=ilbers.de ARC-Authentication-Results: i=3; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=FUN8WYfb; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of clara.kowalsky@siemens.com designates 2a01:111:f403:c201::3 as permitted sender) smtp.mailfrom=clara.kowalsky@siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20260707; t=1794215979; x=1794820779; darn=ilbers.de; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:mime-version :content-transfer-encoding:content-type:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=go0O3LhBPODqIwOuFwJmfZyhfxmswrdel7+1l87nL8k=; b=eTcygKSEQXbrMPRK/0Ag5hoa7OFbzuuhyxSxhJkOoR69utNU9HFNNUPplk96jhKuOh tZrK5/Ou9TOcg1fASvAlbbriFi36veSK9IGTKSl3QT3HpMGnrVNgYSayvAMqwTRVVvYS hvRVx5mUmlToB2cLgsfPGW1hpA3XDNfaZEFd2D7U525ej+HMfp3ynKAl6yJbQtkeGHZ8 5R5v13ZpjJwTh4W3Jec0codHREiT53Qz9bbxGLESohR2uHgt0Mhms5lW/btuJExFtYnN qAC9W505AceQzgq6Nzf+CIbXZvY572VxmL0y67T+isItrONK+QWiZ9O7Jonu22y91JUi hcWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1794215979; x=1794820779; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence:reply-to :x-original-authentication-results:x-original-sender:mime-version :content-transfer-encoding:content-type:references:in-reply-to :message-id:date:subject:cc:to:from:x-beenthere:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to:content-type; bh=go0O3LhBPODqIwOuFwJmfZyhfxmswrdel7+1l87nL8k=; b=RK1o87r3OcLpnBX1/InuV3eAk4OH0BiW6Ep4u+mDlPFgsSLrcREK3gdEB9R9EDzquh 0PvaUxagajjly0uArYjvPFH7OsSrn7NIbBt7hyyZqFQfiQjJwVgPxNskF6fYBSWZiT54 5uWuwJDUQm0Alewmxtecn2iDGFxgC8eBW4rcPFgAmyNW/M2Pe/T1Uxw/3qOTtAETfZN2 LF0R/bG+TH41XXq7fYTgRZb1Ob5JvV9Ol3N4EdteEfKYXo0FjwNykmFK93vJkneneS/W kL1p2iowEhRN9wGdT5AZ9jjN852KPpnADKyuyQkteSNlXikNfaSV+VcGCsAo7w8UP0Hy v7Cw== X-Forwarded-Encrypted: i=3; AL+hErEzvzG7EJhH5Ecu3C8gD4Xct6TCgRQCZzCBlRGhgP2LZr80OZrV/xX/9GK+6r8HlZ23GdQx@ilbers.de X-Gm-Message-State: APuU4Z/WEWIoEZTqcaTZ44HW8lSg8yWS3hVeyrIsW+Nl0wcRy0nBvudJ SyeRecr/oPJLsDQdl8OpbqP5WiWxvElbC71hGfgy0bQldE+uP0NHx5DS X-Received: by 2002:a05:622a:5588:b0:533:7a4c:d8e9 with SMTP id d75a77b69052e-552a4cc7e62mr45622461cf.37.1794215978520; Mon, 09 Nov 2026 01:19:38 -0800 (PST) X-BeenThere: isar-users@googlegroups.com; h="ATskLdf1UXYb2149+mDFsZEEcbWygd0WR0kUXpd6zVbvw5KeZg==" Received: by 2002:a05:6214:2589:b0:914:1182:d90e with SMTP id 6a1803df08f44-91eccb3713bls35188576d6.1.-pod-prod-03-us; Mon, 09 Nov 2026 01:19:37 -0800 (PST) X-Received: by 2002:a05:6102:512c:b0:7d2:d45e:5c9d with SMTP id ada2fe7eead31-7d2db0e27d2mr663159137.13.1794215976808; Mon, 09 Nov 2026 01:19:36 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1794215976; cv=pass; d=google.com; s=arc-20260327; b=Hr74bfo++q4FBZFso7tt3eeEXwkNdO743MRNRA10lTW/hBrCpk1FdtQuvwJRdUhesr tUFVAj2VbB8DBGyQ+cUtlHRjI8noapo9zcq+RptcS4anVeUPwGMwPwi0iz5Q60FS1/H5 j3ZtW8ZTdFJwvOD1ITAXyK+WgVne5TwUiueD4NaRTzwdBCPRuAe3NmKHrHM9UdO33RkY eKyjK9LvQpcqc3idFHPUwIRVhStWqjoqJE3++85fMkZC6qasaGoOjzta8iFlicLTlPR5 Dp7D/k6Qpqolxrt4Q3YMDH5tSYjfns9HKis4J/jWHLgarxZZI8aEFZgIaqbQOZYHs8/5 86pw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=mime-version:content-transfer-encoding:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=ODJ0ZYuPAilOAyIgo4ZWu/UQ6yKRLrzrSIq4t8GFbKo=; fh=AqT8zD6jUs4YP+XkSwmnbRYu3wrat9Vyg/xvm1MGQCM=; b=eShR2ASGH4HuSN2ibUak9yrKngawqd/5xo2UuUTPGk/0T+ykRT8J9xc029Ka5HDujD NgPPUHBgLeJPNOfKfOGgtdeZ3NUfHLCF+rIBOEi0cmxizLGAp4Xzj3iNY23XtFlXBe9L H/oylsNoH/lueqXPfhj/4/BdLOMZOr1P/ZRR/IErYieWcJA1qUjnZCpZ+LMHCDxK8+ef zpGF547iurj5gpX9/GLxR5/3pg+nwwg/FQrMxyU36F1V5ChR58+Z25Ihz/LusM3DUT/o 11uDsKiP35RLu3ECce4ffugJD+7fhzni59ASuvXiF8F8krrx5xcS+r4IzB1D7AipJjZz F8BA==; dara=google.com ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=FUN8WYfb; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of clara.kowalsky@siemens.com designates 2a01:111:f403:c201::3 as permitted sender) smtp.mailfrom=clara.kowalsky@siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: from AS8PR04CU009.outbound.protection.outlook.com (mail-westeuropeazlp170110003.outbound.protection.outlook.com. [2a01:111:f403:c201::3]) by gmr-mx.google.com with ESMTPS id a1e0cc1a2514c-99c0292009csi92776241.1.2026.11.09.01.19.34 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Nov 2026 01:19:36 -0800 (PST) Received-SPF: pass (google.com: domain of clara.kowalsky@siemens.com designates 2a01:111:f403:c201::3 as permitted sender) client-ip=2a01:111:f403:c201::3; ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=f7822B1i620WFGhx0qfkCLklQeb5e/X5Ojs5OOG6oIZZKtAZgcB0CfP8H73rg+jbziq8ldi/dMi9tuiEPL3mffn2AXBEaVG/tPACJDFpx15ABxmbQIgmBXk4LeY1Qu8KLdRS3lT4WW7QYvKerQJj/sm+7iigBxZd04jO4fzd/nLESxmflyxH/JAOsWqyUOsqAs6ilEaEJIFKmepC/GSEdxgMkfRR5mVdkRU2RupJVXS69wIqkrne+Uew+KJ5cbDYCBfs4ipQEuPxHIteNWrQh2OAVCl9oOjRaeYo1/lW0DVvb4eJd9UhihImJ+UqfoiNV2nYt1lY/Pp1n/3CuhjZ1g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ODJ0ZYuPAilOAyIgo4ZWu/UQ6yKRLrzrSIq4t8GFbKo=; b=p1Vo+eVSms3F7y79N/kUrFUt/dRO1wkl+wzezqc3IWaDQq6EalMt7NoQnwQn2DMwJNMDbdYSXPvqvvFEmydKTr/MMJP/cOQvHENnvFuvSWTyrhao3iP2h9IsnqkJM+Ph44kXe8hWak5A+eQ35pQtnxjd9813Zeapouu/UT52SGn9qhmLRDlaCLCAp4EmMB1bbdFaMrccJAv82oHt1Te32d0jFMFuI9K6dA9+yyFZURmkkZKrVE35lXI1NsxIWPHMLg3yA86ZJfffEZR5ehCMhZGjNBDyFBBtdeuWFt4ocJ1GtUjSylp41tqWXOYQOqK/RF+QdrfzyZPkKpctwYpLmA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=siemens.com; dmarc=pass action=none header.from=siemens.com; dkim=pass header.d=siemens.com; arc=none Received: from AM0PR10MB9645.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:740::12) by DU0PR10MB7554.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:10:426::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.615.7; Mon, 9 Nov 2026 09:18:54 +0000 Received: from AM0PR10MB9645.EURPRD10.PROD.OUTLOOK.COM ([fe80::527e:14a7:bd3e:86be]) by AM0PR10MB9645.EURPRD10.PROD.OUTLOOK.COM ([fe80::527e:14a7:bd3e:86be%6]) with mapi id 15.21.0615.007; Mon, 9 Nov 2026 09:18:54 +0000 From: "'Clara Kowalsky' via isar-users" To: isar-users@googlegroups.com Cc: Clara Kowalsky Subject: [PATCH 8/8] bitbake: update to v2.20.0 Date: Mon, 9 Nov 2026 10:18:01 +0100 Message-ID: <20261109091802.202521-9-clara.kowalsky@siemens.com> X-Mailer: git-send-email 2.56.0 In-Reply-To: <20261109091802.202521-1-clara.kowalsky@siemens.com> References: <20261109091802.202521-1-clara.kowalsky@siemens.com> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: FR4P281CA0130.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:b9::18) To AM0PR10MB9645.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:740::12) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AM0PR10MB9645:EE_|DU0PR10MB7554:EE_ X-MS-Office365-Filtering-Correlation-Id: c29c303d-1dd7-4f46-27fc-08df3e427695 X-MS-Exchange-AtpMessageProperties: SA X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|69100299015|366016|376014|3109299003|7049299003|23010399003|1800799024|39142699007|31052699007|260925021911599003|260925022911599003|260925021311599003|261009223327099003|5023799004|11063799006|56012099006|261021133027099003|261021123027099003|261021135927099003|261021130227099003|261021121427099003|261022113327099003|261022115727099003|261022075727099003|261022105027099003|261013131602099003|261009221227099003|261009220027099003|261009221727099003|261009223027099003|261026130827099003|261026140227099003|261026135527099003|261026121527099003|261028144727099003|261102130727099003|261102124527099003|261102125427099003|261102123227099003|10067099003|12006099003|8126099003|21046099003|10086099003|10136099003|25016099003|21026099003|55112099003|20052099010|18092099006|18002099003|22082099003|17002099007|16056099006|10063799003|9063799003|29003799003|6133799003|13003099007|4133799003|3023799007|15866825006|191535003; X-Microsoft-Antispam-Message-Info: HHc2Av906Gi+WoQ/FiXFEulCGjfyH1YudPhhmyB9kELRnKM0SNjh/rRTfxRv9tRND78xp/nDC6PBUTqvY4v0ap9As0g8rXUCQnKfcVb8DOEnmD/maNFjkR2P0g/fvhCqAUiHqTeSMO0aSG6ScXysarF/qxy+/Y18dQ9q9nGvT55rnCP5hy4bRbzZB5jU3acb7yE0Tj1755gPdoU+eFdHz1G5psO4mOunJUK2J1Z8Jct1LgYRCs5VoIZ0vnk4GkO8+Q7682Ieia7+0Lt6UqopVSr0k4XksbC7yK836U+mCl9h6tGNupDv4/PUlNB1eH/TcPNfcpTnr9UF6lU4CNVVjmqmjGQXeN4CkjtJhpSpXl9AOcgMWuYugJEjfzza6bEtw7iMV0WIEIPiP6hQDLC2T9wtQ/uRDBFJ/ca4kTUby2hqOmFdzAcjdSPRkSGkVkILhQJK7apDhIcFBFRYRy0hFlLlPOr/KMmVG57PPxiRew5lOgrYP8pDq242YwTMV3NEgu66+671kHUZvi4BPbN8/YHyvcIyyhXbKnRufA1gv7B5uvmfCOfeyYmdatUVZedrqUQjYRKyL9enRQERiFdoDuwPG6axmkboo671C8o8/qn1TP1uCIK9CNFvSQ+AIhhNPE/PEW1O2HddRPyfJYSOeVOyuI9RpwfoE0jfG0ZRlKb9C9TnY9VXGruOBl1EUhJV X-Forefront-Antispam-Report: =?utf-8?B?Q0lQOjI1NS4yNTUuMjU1LjI1NTtDVFJZOjtMQU5HOmVuO1NDTDoxO1NSVjo7?= =?utf-8?B?SVBWOk5MSTtTRlY6TlNQTTtIOkFNMFBSMTBNQjk2NDUuRVVSUFJEMTAuUFJP?= =?utf-8?B?RC5PVVRMT09LLkNPTTtQVFI6O0NBVDpOT05FO1NGUzooMTMyMzAwNDApKDY5?= =?utf-8?B?MTAwMjk5MDE1KSgzNjYwMTYpKDM3NjAxNCkoMzEwOTI5OTAwMykoNzA0OTI5?= =?utf-8?B?OTAwMykoMjMwMTAzOTkwMDMpKDE4MDA3OTkwMjQpKDM5MTQyNjk5MDA3KSgz?= =?utf-8?B?MTA1MjY5OTAwNykoMjYwOTI1MDIxOTExNTk5MDAzKSgyNjA5MjUwMjI5MTE1?= =?utf-8?B?OTkwMDMpKDI2MDkyNTAyMTMxMTU5OTAwMykoMjYxMDA5MjIzMzI3MDk5MDAz?= =?utf-8?B?KSg1MDIzNzk5MDA0KSgxMTA2Mzc5OTAwNikoNTYwMTIwOTkwMDYpKDI2MTAy?= =?utf-8?B?MTEzMzAyNzA5OTAwMykoMjYxMDIxMTIzMDI3MDk5MDAzKSgyNjEwMjExMzU5?= =?utf-8?B?MjcwOTkwMDMpKDI2MTAyMTEzMDIyNzA5OTAwMykoMjYxMDIxMTIxNDI3MDk5?= =?utf-8?B?MDAzKSgyNjEwMjIxMTMzMjcwOTkwMDMpKDI2MTAyMjExNTcyNzA5OTAwMyko?= =?utf-8?B?MjYxMDIyMDc1NzI3MDk5MDAzKSgyNjEwMjIxMDUwMjcwOTkwMDMpKDI2MTAx?= =?utf-8?B?MzEzMTYwMjA5OTAwMykoMjYxMDA5MjIxMjI3MDk5MDAzKSgyNjEwMDkyMjAw?= =?utf-8?B?MjcwOTkwMDMpKDI2MTAwOTIyMTcyNzA5OTAwMykoMjYxMDA5MjIzMDI3MDk5?= =?utf-8?B?MDAzKSgyNjEwMjYxMzA4MjcwOTkwMDMpKDI2MTAyNjE0MDIyNzA5OTAwMyko?= =?utf-8?B?MjYxMDI2MTM1NTI3MDk5MDAzKSgyNjEwMjYxMjE1MjcwOTkwMDMpKDI2MTAy?= =?utf-8?B?ODE0NDcyNzA5OTAwMykoMjYxMTAyMTMwNzI3MDk5MDAzKSgyNjExMDIxMjQ1?= =?utf-8?B?MjcwOTkwMDMpKDI2MTEwMjEyNTQyNzA5OTAwMykoMjYxMTAyMTIzMjI3MDk5?= =?utf-8?B?MDAzKSgxMDA2NzA5OTAwMykoMTIwMDYwOTkwMDMpKDgxMjYwOTkwMDMpKDIx?= =?utf-8?B?MDQ2MDk5MDAzKSgxMDA4NjA5OTAwMykoMTAxMzYwOTkwMDMpKDI1MDE2MDk5?= =?utf-8?B?MDAzKSgyMTAyNjA5OTAwMykoNTUxMTIwOTkwMDMpKDIwMDUyMDk5MDEwKSgx?= =?utf-8?B?ODA5MjA5OTAwNikoMTgwMDIwOTkwMDMpKDIyMDgyMDk5MDAzKSgxNzAwMjA5?= =?utf-8?B?OTAwNykoMTYwNTYwOTkwMDYpKDEwMDYzNzk5MDAzKSg5MDYzNzk5MDAzKSgy?= =?utf-8?B?OTAwMzc5OTAwMykoNjEzMzc5OTAwMykoMTMwMDMwOTkwMDcpKDQxMzM3OTkw?= =?utf-8?B?MDMpKDMwMjM3OTkwMDcpKDE1ODY2ODI1MDA2KSgxOTE1MzUwMDMpO0RJUjpP?= =?utf-8?Q?UT;SFP:1101;?= X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?a0lJVnVkYWdTTFBmamhERHZGV2RNa044UlRweGNRamdIMUVvSUY1N3dMN3pZ?= =?utf-8?B?VGJWZ1NpQVVDMWZuY2dpZGUwN3Nibkhhek15TXlMZlRoMlRMY3FFS3g5VFhz?= =?utf-8?B?ajJ6N1k0ZE55WjVwUTZQaWRMZVhiTFdNRHBuZkxCT2UzOFd3WlI0cXlXN0lV?= =?utf-8?B?bkcySHlEd3ZyVFczVTVGYVpXZVpVZ1c1K3dsclVFcGF1OU1DaU9sZVdCdldQ?= =?utf-8?B?Y2ZHUDk1ejJ4cnZlam85M2RsSW04THNtVHVpNVIxZUxKSTZKQ0pQdENQN1Bl?= =?utf-8?B?aDVaeklZTzBvRmtrcktuVGpRczlGUUdrMFNDVUt6bnBHTDk3NCt4RWxTVTJH?= =?utf-8?B?R2VJWkJzUExDRVhjNmVOdkt5Vk5IckloRSt6OTVqWURCay8vYVc2ZmZWMys1?= =?utf-8?B?T25nR3BlUFlUVVJKazdtR0UwVXhWU3BhZjFFWU5OZ1duTUk5SGlzNzNrN0Fm?= =?utf-8?B?a1lrOCtXelU4NzdBN2wxODArRUxPVjRRZ1M2Z1VYNllqYlEwMmFlRjRreFI0?= =?utf-8?B?WTN0eDdxRWwzZmlPd0ZtcWFTNm5EWjc1WXk1alNuQkxpNS9GcUs1emZjU2ln?= =?utf-8?B?NDE0Zy9FcVBiaXpFTW41bDZ5UGpraWkrb0V0b284RkdRNFBCdkZOQTNUcFNL?= =?utf-8?B?YmJ4dUt1empIYW1CN2JXRjFhbEw4THBsLzBBcGw5bWoybUFlblpYMFVXVG5Q?= =?utf-8?B?cWFFSUdCLzJLTWNXYWZPb3B2RlNxdFZ3b3MwbGw2SVhiVUtTZ2hmdUxGWVNK?= =?utf-8?B?NmZEVmlzalZMSm8yRkxmdUZoV1k1eDhRa0I0bERrZGs0YXNwR2pVb0ZXa1N6?= =?utf-8?B?eklGQmJGWDRyeUNGSzZubFBJTXFaWUVVNHloanpzMWpSTDR4MjZ2UExxRENW?= =?utf-8?B?S0F5T21JQWxnU1ZSa01yVTFaOGxWQ09ld296ZDJNcHVEVzd1UTlSbm5Eby9q?= =?utf-8?B?aUw2cnpFT0tlYnFXdmVsMkx4NXlKM3Vxd3lZYWRnRGIyWWNVUEtjOEhINWZI?= =?utf-8?B?N3htZkxmNUNmajRDYXhvTEpCdE5JMjJlaDVRSFk0ckxhbmt0WGF0cmdNUGZt?= =?utf-8?B?cmtDZXV2SVNwNGVvVUpVNVpTbHdMa05BSUF2YXB1Rnp4cVVOS2tXVGxlVkRV?= =?utf-8?B?MlV1cktxTXgyRXRNUkN6eGZibWdjVVpDS2h1aGVlemNkTUpNNGluaWJZRGh1?= =?utf-8?B?bldDOEgzR21Gbjl6Zis1Z05sTEdIdFFTWU0rV0dJSitRZVpCQWZZUW0yQVF2?= =?utf-8?B?Umt2YmpRYnJrcllYbEdERUJYMXpXMGJzZldrVEkySDlwclJVcHhrUUhxNVly?= =?utf-8?B?L1ZUK3pGUGNvbmNqRzg0UlgxK1RVdW5XMUc4VEVHKysvTVV4bDRHNWoxSVVV?= =?utf-8?B?Z1ZrN1llWUFBNjNHWVlqQ1VJTlVGR1BFbWQ0WG14cGdHU21wTm1SYlJTK1hY?= =?utf-8?B?OXdzczJDaHRubGZlWmtsSW9WWVkrWVY3Y0d5R2JDN1pjNlRlclJQWSs0d3pZ?= =?utf-8?B?d21NUjdXbiszS0JpTkpQL29sVEw4UGI4enhtZ2l1VGNYTnFUVklOM1ZwSlFq?= =?utf-8?B?RlRWVlcwNkIrNzd3UnkxVWh2RjJXbkVjcjQ3anByWEMyQlFxcWRha2VVQ3Q4?= =?utf-8?B?YlRzWTc2UGJtKzJjK05zYmNCVWV2c21oSVhiZ0U0RzdSemFuaUVpaHVydFdO?= =?utf-8?B?TzhCYU9Hc2JoR0p5T1FJaTd5Q29uZkR6UjlMWmNvMG5DdWx6T3RxcnU0RkYx?= =?utf-8?B?Y3Z5dy9WNW92Tng0WGNaeTVCdWExYU9yNi9uN3pkSjBQa2F1ajgzaWhSZENY?= =?utf-8?B?aldXMTJ6ZG8xQnpZSFVmU2ZPeklCVVp0Q3g1UnFqMjB2b0xFSjZDbUZwd1BO?= =?utf-8?B?QXhlZ01KSlR0TmF3MllIRWg0T0wrUDVWZHFSMzZqQXVGWk13cHRiVFNmSUhU?= =?utf-8?B?OENGSkZBS2ZjRlZud3MxS29LSHJFd3dsS0tTdW9MNDhYRTVLL1hqUTNmMnhE?= =?utf-8?B?OFlTaGxCc21tZ29MNXE0UFhwbEE0dHh2Tnp4QkJhSUt3aWU0ZVpkU0N0QXpq?= =?utf-8?B?bHV1QmV3ZlV1aDVhdlRFZktrVko1MldnYk9WeW5Hc3NXL3hVTkFoR2NhWU1G?= =?utf-8?B?YmhCblp6bmM0S0VHSERQS0Z4Y09Ya2ljMXBOdlR5dGhpOERhVXA0NGJCanN0?= =?utf-8?B?REx3SDB5VUZ5WklsMnF5b3ZrSnBGbEJFNkZQSUFxUXlOMjlwZThYWFduWU40?= =?utf-8?B?bkR3eHBwL0dLVUdCUmRQc3llNWdmeDRhYktJdXluclJhTThOdXIrNHl5TVdt?= =?utf-8?B?Tkx2MnRrRzdobzhRWjhVY2tDa2dva0ZCYWxRUkJGU0FGQmVvamIzQXRzWWti?= =?utf-8?Q?3vX07gfdprr67IME=3D?= X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-Network-Message-Id: c29c303d-1dd7-4f46-27fc-08df3e427695 X-MS-Exchange-CrossTenant-AuthSource: AM0PR10MB9645.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Nov 2026 09:18:53.9751 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: /ggjromjMhGgeLAivxiEXCtHfJ+Z+cfg8uprrU9X15zH8l0HEQsWxZN7/3q5aegEkvkAaZdpMAS8Nh2s1MwxRU3AdB214WSHWfyQDydInf4= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU0PR10MB7554 X-Original-Sender: clara.kowalsky@siemens.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=FUN8WYfb; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of clara.kowalsky@siemens.com designates 2a01:111:f403:c201::3 as permitted sender) smtp.mailfrom=clara.kowalsky@siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com X-Original-From: Clara Kowalsky Reply-To: Clara Kowalsky Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-TUID: bpW8LhxnAs3I Update BitBake from v2.8.0 to v2.20.0. This is upstream commit 634f8dede6b9d71b5a6691499fd05a8678b46eae, tagged yocto-6.1. Drop Isar's lowered Python version check and retain upstream's Python 3.9 minimum. WIC now runs with the build host's Python, so imaging Debian buster no longer requires this workaround. In gitignore, set /build* so Git does not ignore BitBake's new bs4/builder directory. Signed-off-by: Clara Kowalsky --- .gitignore | 2 +- bitbake/.b4-config | 4 + bitbake/.gitignore | 2 + bitbake/AGENTS.md | 55 + bitbake/README | 2 +- bitbake/TODO | 65 +- bitbake/bin/bitbake | 4 +- bitbake/bin/bitbake-config-build | 1 + bitbake/bin/bitbake-diffsigs | 16 +- bitbake/bin/bitbake-getvar | 21 +- bitbake/bin/bitbake-hashclient | 115 +- bitbake/bin/bitbake-hashserv | 17 +- bitbake/bin/bitbake-layers | 46 +- bitbake/bin/bitbake-prserv | 34 +- bitbake/bin/bitbake-selftest | 13 +- bitbake/bin/bitbake-server | 14 +- bitbake/bin/bitbake-setup | 1481 ++++++ bitbake/bin/bitbake-worker | 42 +- bitbake/bin/git-make-shallow | 175 - bitbake/bin/toaster | 23 +- bitbake/bin/toaster-eventreplay | 7 + bitbake/conf/bitbake.conf | 2 +- bitbake/contrib/b4-wrapper-bitbake.py | 40 + bitbake/contrib/hashserv/Dockerfile | 5 +- bitbake/contrib/prserv/Dockerfile | 3 +- bitbake/contrib/vim/LICENSE.txt | 18 - bitbake/contrib/vim/ftdetect/bitbake.vim | 24 - bitbake/contrib/vim/ftplugin/bitbake.vim | 13 - bitbake/contrib/vim/indent/bitbake.vim | 343 -- bitbake/contrib/vim/plugin/newbb.vim | 88 - bitbake/contrib/vim/plugin/newbbappend.vim | 46 - bitbake/contrib/vim/syntax/bitbake.vim | 131 - .../oe-nodistro-blacksail.conf.json | 48 + .../oe-nodistro-master.conf.json | 48 + .../oe-nodistro-whinlatter.conf.json | 56 + .../oe-nodistro-wrynose.conf.json | 48 + .../configurations/poky-blacksail.conf.json | 74 + .../configurations/poky-master.conf.json | 74 + .../configurations/poky-whinlatter.conf.json | 80 + .../configurations/poky-wrynose.conf.json | 74 + bitbake/doc/.gitignore | 3 + bitbake/doc/Makefile | 2 +- bitbake/doc/README | 2 +- .../bitbake-user-manual-environment-setup.rst | 1332 +++++ .../bitbake-user-manual-execution.rst | 35 +- .../bitbake-user-manual-fetching.rst | 305 +- .../bitbake-user-manual-intro.rst | 267 +- .../bitbake-user-manual-library-functions.rst | 59 + .../bitbake-user-manual-metadata.rst | 462 +- .../bitbake-user-manual-ref-variables.rst | 464 +- bitbake/doc/conf.py | 41 +- bitbake/doc/index.rst | 2 + bitbake/doc/releases.rst | 174 - bitbake/doc/setversions.py | 374 ++ bitbake/doc/sphinx-static/switchers.js | 233 - bitbake/doc/sphinx-static/switchers.js.in | 249 + bitbake/doc/sphinx-static/theme_overrides.css | 17 +- bitbake/lib/bb/COW.py | 10 +- bitbake/lib/bb/__init__.py | 158 +- .../_vendor/beautifulsoup4.LICENSE} | 11 +- bitbake/lib/bb/_vendor/bs4/__init__.py | 1177 +++++ bitbake/lib/bb/_vendor/bs4/_deprecation.py | 80 + bitbake/lib/bb/_vendor/bs4/_typing.py | 205 + bitbake/lib/bb/_vendor/bs4/_warnings.py | 98 + .../lib/bb/_vendor/bs4/builder/__init__.py | 848 ++++ .../lib/bb/_vendor/bs4/builder/_html5lib.py | 611 +++ .../lib/bb/_vendor/bs4/builder/_htmlparser.py | 517 ++ bitbake/lib/bb/_vendor/bs4/builder/_lxml.py | 501 ++ bitbake/lib/bb/_vendor/bs4/css.py | 330 ++ bitbake/lib/bb/_vendor/bs4/dammit.py | 1525 ++++++ bitbake/lib/bb/_vendor/bs4/diagnose.py | 268 + bitbake/lib/bb/_vendor/bs4/element.py | 3395 +++++++++++++ bitbake/lib/bb/_vendor/bs4/exceptions.py | 28 + bitbake/lib/bb/_vendor/bs4/filter.py | 764 +++ bitbake/lib/bb/_vendor/bs4/formatter.py | 276 ++ bitbake/lib/bb/_vendor/bs4/py.typed | 0 bitbake/lib/bb/_vendor/ply.pyi | 1 + bitbake/lib/{ =3D> bb/_vendor}/ply/__init__.py | 1 + bitbake/lib/bb/_vendor/ply/cpp.py | 918 ++++ bitbake/lib/bb/_vendor/ply/ctokens.py | 133 + bitbake/lib/bb/_vendor/ply/lex.py | 901 ++++ bitbake/lib/{ =3D> bb/_vendor}/ply/yacc.py | 2238 +++++---- bitbake/lib/bb/_vendor/ply/ygen.py | 74 + bitbake/lib/bb/_vendor/progressbar.pyi | 1 + .../{ =3D> bb/_vendor}/progressbar/LICENSE.txt | 0 .../{ =3D> bb/_vendor}/progressbar/__init__.py | 4 +- .../{ =3D> bb/_vendor}/progressbar/compat.py | 0 .../_vendor}/progressbar/progressbar.py | 37 +- .../{ =3D> bb/_vendor}/progressbar/widgets.py | 14 +- bitbake/lib/bb/_vendor/simplediff.pyi | 1 + .../lib/{ =3D> bb/_vendor}/simplediff/LICENSE | 0 .../{ =3D> bb/_vendor}/simplediff/__init__.py | 2 +- bitbake/lib/bb/_vendor/tomli/LICENSE | 21 + bitbake/lib/bb/_vendor/tomli/__init__.py | 8 + bitbake/lib/bb/_vendor/tomli/_parser.py | 793 +++ bitbake/lib/bb/_vendor/tomli/_re.py | 119 + bitbake/lib/bb/_vendor/tomli/_types.py | 10 + bitbake/lib/bb/_vendor/tomli/py.typed | 1 + .../lib/bb/_vendor/typing_extensions.LICENSE | 279 ++ bitbake/lib/bb/_vendor/typing_extensions.py | 4317 +++++++++++++++++ bitbake/lib/bb/_vendor/typing_extensions.pyi | 1 + bitbake/lib/bb/acl.py | 2 - bitbake/lib/bb/asyncrpc/__init__.py | 3 +- bitbake/lib/bb/asyncrpc/client.py | 162 +- bitbake/lib/bb/asyncrpc/serv.py | 58 +- bitbake/lib/bb/asyncrpc/taskgroup.py | 52 + bitbake/lib/bb/build.py | 29 +- bitbake/lib/bb/cache.py | 37 +- bitbake/lib/bb/checksum.py | 25 + bitbake/lib/bb/codeparser.py | 102 +- bitbake/lib/bb/command.py | 50 +- bitbake/lib/bb/compress/lz4.py | 4 +- bitbake/lib/bb/cooker.py | 319 +- bitbake/lib/bb/cookerdata.py | 40 +- bitbake/lib/bb/daemonize.py | 3 + bitbake/lib/bb/data.py | 19 +- bitbake/lib/bb/data_smart.py | 157 +- bitbake/lib/bb/event.py | 29 +- bitbake/lib/bb/{fetch2 =3D> fetch}/README | 0 bitbake/lib/bb/{fetch2 =3D> fetch}/__init__.py | 526 +- bitbake/lib/bb/{fetch2 =3D> fetch}/az.py | 23 +- bitbake/lib/bb/{fetch2 =3D> fetch}/clearcase.py | 63 +- bitbake/lib/bb/fetch/crate.py | 198 + bitbake/lib/bb/{fetch2 =3D> fetch}/gcp.py | 45 +- bitbake/lib/bb/{fetch2 =3D> fetch}/git.py | 674 +-- bitbake/lib/bb/{fetch2 =3D> fetch}/gitannex.py | 22 +- bitbake/lib/bb/{fetch2 =3D> fetch}/gitsm.py | 144 +- bitbake/lib/bb/fetch/gomod.py | 270 ++ bitbake/lib/bb/{fetch2 =3D> fetch}/hg.py | 65 +- bitbake/lib/bb/{fetch2 =3D> fetch}/local.py | 15 +- bitbake/lib/bb/{fetch2 =3D> fetch}/npm.py | 169 +- bitbake/lib/bb/{fetch2 =3D> fetch}/npmsw.py | 120 +- bitbake/lib/bb/{fetch2 =3D> fetch}/perforce.py | 46 +- bitbake/lib/bb/{fetch2 =3D> fetch}/repo.py | 26 +- bitbake/lib/bb/{fetch2 =3D> fetch}/s3.py | 21 +- bitbake/lib/bb/{fetch2 =3D> fetch}/sftp.py | 23 +- bitbake/lib/bb/{fetch2 =3D> fetch}/ssh.py | 33 +- bitbake/lib/bb/{fetch2 =3D> fetch}/svn.py | 88 +- bitbake/lib/bb/{fetch2 =3D> fetch}/wget.py | 316 +- bitbake/lib/bb/fetch2.py | 19 + bitbake/lib/bb/fetch2/bzr.py | 128 - bitbake/lib/bb/fetch2/crate.py | 141 - bitbake/lib/bb/fetch2/cvs.py | 157 - bitbake/lib/bb/fetch2/osc.py | 165 - bitbake/lib/bb/filter.py | 142 + bitbake/lib/bb/main.py | 8 +- bitbake/lib/bb/parse/__init__.py | 49 +- bitbake/lib/bb/parse/ast.py | 201 +- bitbake/lib/bb/parse/parse_py/BBHandler.py | 53 +- bitbake/lib/bb/parse/parse_py/ConfHandler.py | 24 +- bitbake/lib/bb/persist_data.py | 271 -- bitbake/lib/bb/pysh/pyshlex.py | 2 +- bitbake/lib/bb/pysh/pyshyacc.py | 2 +- bitbake/lib/bb/runqueue.py | 487 +- bitbake/lib/bb/server/process.py | 86 +- bitbake/lib/bb/server/xmlrpcserver.py | 11 +- bitbake/lib/bb/siggen.py | 102 +- bitbake/lib/bb/taskdata.py | 8 +- bitbake/lib/bb/tests/codeparser.py | 40 + bitbake/lib/bb/tests/compression.py | 4 +- bitbake/lib/bb/tests/cooker.py | 342 +- bitbake/lib/bb/tests/data.py | 207 +- .../software/miniupnp/download.php | 3528 ++++++++++++++ bitbake/lib/bb/tests/fetch.py | 2583 +++++++--- bitbake/lib/bb/tests/fetch_import.py | 86 + bitbake/lib/bb/tests/filter.py | 88 + .../bb/tests/parse-tests/classes/base.bbclass | 5 + .../classes/recipe-file-class.bbclass | 2 + .../bb/tests/parse-tests/conf/bitbake.conf | 15 + bitbake/lib/bb/tests/parse.py | 413 +- bitbake/lib/bb/tests/persist_data.py | 129 - .../tests/runqueue-tests/classes/base.bbclass | 4 +- .../lib/bb/tests/runqueue-tests/recipes/g1.bb | 2 + .../lib/bb/tests/runqueue-tests/recipes/h1.bb | 0 bitbake/lib/bb/tests/runqueue.py | 20 +- bitbake/lib/bb/tests/setup.py | 1089 +++++ bitbake/lib/bb/tests/siggen.py | 47 + bitbake/lib/bb/tests/support/httpserver.py | 4 +- bitbake/lib/bb/tests/utils.py | 19 + bitbake/lib/bb/tinfoil.py | 227 +- bitbake/lib/bb/ui/buildinfohelper.py | 5 +- bitbake/lib/bb/ui/knotty.py | 203 +- bitbake/lib/bb/ui/ncurses.py | 2 +- bitbake/lib/bb/ui/taskexp_ncurses.py | 3 + bitbake/lib/bb/ui/teamcity.py | 14 +- bitbake/lib/bb/ui/uihelper.py | 12 +- bitbake/lib/bb/utils.py | 867 +++- bitbake/lib/bblayers/action.py | 4 +- bitbake/lib/bblayers/layerindex.py | 12 +- bitbake/lib/bblayers/query.py | 37 +- bitbake/lib/bs4/AUTHORS.txt | 43 - bitbake/lib/bs4/NEWS.txt | 1066 ---- bitbake/lib/bs4/__init__.py | 468 -- bitbake/lib/bs4/builder/__init__.py | 324 -- bitbake/lib/bs4/builder/_html5lib.py | 338 -- bitbake/lib/bs4/builder/_htmlparser.py | 262 - bitbake/lib/bs4/builder/_lxml.py | 248 - bitbake/lib/bs4/dammit.py | 832 ---- bitbake/lib/bs4/diagnose.py | 216 - bitbake/lib/bs4/element.py | 1724 ------- bitbake/lib/bs4/testing.py | 686 --- bitbake/lib/bs4/tests/__init__.py | 1 - .../lib/bs4/tests/test_builder_registry.py | 147 - bitbake/lib/bs4/tests/test_docs.py | 32 - bitbake/lib/bs4/tests/test_html5lib.py | 98 - bitbake/lib/bs4/tests/test_htmlparser.py | 31 - bitbake/lib/bs4/tests/test_lxml.py | 70 - bitbake/lib/bs4/tests/test_soup.py | 479 -- bitbake/lib/bs4/tests/test_tree.py | 2004 -------- bitbake/lib/codegen.py | 564 --- bitbake/lib/hashserv/__init__.py | 15 +- bitbake/lib/hashserv/client.py | 423 +- bitbake/lib/hashserv/server.py | 257 +- bitbake/lib/hashserv/sqlite.py | 17 + bitbake/lib/hashserv/tests.py | 589 ++- bitbake/lib/layerindexlib/__init__.py | 6 +- bitbake/lib/layerindexlib/cooker.py | 16 +- bitbake/lib/layerindexlib/restapi.py | 2 +- bitbake/lib/ply/lex.py | 1054 ---- bitbake/lib/prserv/__init__.py | 97 +- bitbake/lib/prserv/client.py | 15 +- bitbake/lib/prserv/db.py | 452 +- bitbake/lib/prserv/serv.py | 140 +- bitbake/lib/prserv/tests.py | 388 ++ bitbake/lib/pyinotify.py | 2326 --------- bitbake/lib/toaster/bldcollector/admin.py | 8 +- bitbake/lib/toaster/bldcontrol/admin.py | 2 +- .../bldcontrol/localhostbecontroller.py | 175 +- .../toaster/orm/fixtures/check_fixtures.py | 76 + .../lib/toaster/orm/fixtures/gen_fixtures.py | 137 +- bitbake/lib/toaster/orm/fixtures/oe-core.xml | 72 +- bitbake/lib/toaster/orm/fixtures/poky.xml | 136 +- .../orm/management/commands/lsupdates.py | 29 +- bitbake/lib/toaster/orm/models.py | 3 +- .../tests/browser/selenium_helpers_base.py | 75 +- .../tests/browser/test_all_builds_page.py | 7 +- .../tests/browser/test_all_projects_page.py | 12 +- .../tests/browser/test_builddashboard_page.py | 2 +- .../tests/browser/test_landing_page.py | 14 +- .../tests/browser/test_layerdetails_page.py | 65 +- .../browser/test_new_custom_image_page.py | 2 +- .../tests/browser/test_new_project_page.py | 12 +- bitbake/lib/toaster/tests/builds/buildtest.py | 2 +- .../tests/functional/functional_helpers.py | 88 + .../functional/test_create_new_project.py | 121 +- .../tests/functional/test_functional_basic.py | 200 +- .../tests/functional/test_project_config.py | 99 +- .../tests/functional/test_project_page.py | 417 +- .../test_project_page_tab_config.py | 313 +- bitbake/lib/toaster/tests/functional/utils.py | 21 +- .../tests/toaster-tests-requirements.txt | 4 +- bitbake/lib/toaster/tests/views/test_views.py | 2 +- .../toastergui/templates/layerdetails.html | 8 +- .../toastergui/templatetags/projecttags.py | 1 - bitbake/lib/toaster/toastergui/views.py | 18 +- bitbake/lib/toaster/toastermain/settings.py | 5 +- bitbake/packaging-pypi/bitbake-setup/BUILD.md | 67 + bitbake/packaging-pypi/bitbake-setup/LICENSE | 23 + .../packaging-pypi/bitbake-setup/README.md | 42 + .../bitbake-setup/package-bitbake-setup.py | 79 + .../bitbake-setup/pyproject.toml | 92 + bitbake/pyproject.toml | 16 + .../setup-schema/bitbake-setup.schema.json | 211 + bitbake/setup-schema/layers.schema.json | 107 + bitbake/toaster-requirements.txt | 2 +- bitbake/vendor.txt | 6 + bitbake/vendor/licenses/ply/LICENSE | 32 + ...0010-lib-bs4-Avoid-soupsieve-warning.patch | 40 + .../patches/bs4-remove-double-imports.patch | 23 + ...mplement-basic-task-progress-support.patch | 125 + ...ial-pass-of-SPDX-license-headers-to-.patch | 79 + ...bar-accept-value-over-initial-maxval.patch | 54 + ...self._fd_console-to-use-for-self._ha.patch | 53 + ...ke-bars-show-correctly-with-maxval-0.patch | 31 + ...ty-Allow-mixing-log-messages-and-pro.patch | 31 + 275 files changed, 41536 insertions(+), 21231 deletions(-) create mode 100644 bitbake/.b4-config create mode 100644 bitbake/AGENTS.md create mode 120000 bitbake/bin/bitbake-config-build create mode 100755 bitbake/bin/bitbake-setup delete mode 100755 bitbake/bin/git-make-shallow create mode 100755 bitbake/contrib/b4-wrapper-bitbake.py delete mode 100644 bitbake/contrib/vim/LICENSE.txt delete mode 100644 bitbake/contrib/vim/ftdetect/bitbake.vim delete mode 100644 bitbake/contrib/vim/ftplugin/bitbake.vim delete mode 100644 bitbake/contrib/vim/indent/bitbake.vim delete mode 100644 bitbake/contrib/vim/plugin/newbb.vim delete mode 100644 bitbake/contrib/vim/plugin/newbbappend.vim delete mode 100644 bitbake/contrib/vim/syntax/bitbake.vim create mode 100644 bitbake/default-registry/configurations/oe-nodistro-bla= cksail.conf.json create mode 100644 bitbake/default-registry/configurations/oe-nodistro-mas= ter.conf.json create mode 100644 bitbake/default-registry/configurations/oe-nodistro-whi= nlatter.conf.json create mode 100644 bitbake/default-registry/configurations/oe-nodistro-wry= nose.conf.json create mode 100644 bitbake/default-registry/configurations/poky-blacksail.= conf.json create mode 100644 bitbake/default-registry/configurations/poky-master.con= f.json create mode 100644 bitbake/default-registry/configurations/poky-whinlatter= .conf.json create mode 100644 bitbake/default-registry/configurations/poky-wrynose.co= nf.json create mode 100644 bitbake/doc/bitbake-user-manual/bitbake-user-manual-env= ironment-setup.rst create mode 100644 bitbake/doc/bitbake-user-manual/bitbake-user-manual-lib= rary-functions.rst delete mode 100644 bitbake/doc/releases.rst create mode 100755 bitbake/doc/setversions.py delete mode 100644 bitbake/doc/sphinx-static/switchers.js create mode 100644 bitbake/doc/sphinx-static/switchers.js.in rename bitbake/lib/{bs4/COPYING.txt =3D> bb/_vendor/beautifulsoup4.LICENSE= } (80%) create mode 100644 bitbake/lib/bb/_vendor/bs4/__init__.py create mode 100644 bitbake/lib/bb/_vendor/bs4/_deprecation.py create mode 100644 bitbake/lib/bb/_vendor/bs4/_typing.py create mode 100644 bitbake/lib/bb/_vendor/bs4/_warnings.py create mode 100644 bitbake/lib/bb/_vendor/bs4/builder/__init__.py create mode 100644 bitbake/lib/bb/_vendor/bs4/builder/_html5lib.py create mode 100644 bitbake/lib/bb/_vendor/bs4/builder/_htmlparser.py create mode 100644 bitbake/lib/bb/_vendor/bs4/builder/_lxml.py create mode 100644 bitbake/lib/bb/_vendor/bs4/css.py create mode 100644 bitbake/lib/bb/_vendor/bs4/dammit.py create mode 100644 bitbake/lib/bb/_vendor/bs4/diagnose.py create mode 100644 bitbake/lib/bb/_vendor/bs4/element.py create mode 100644 bitbake/lib/bb/_vendor/bs4/exceptions.py create mode 100644 bitbake/lib/bb/_vendor/bs4/filter.py create mode 100644 bitbake/lib/bb/_vendor/bs4/formatter.py create mode 100644 bitbake/lib/bb/_vendor/bs4/py.typed create mode 100644 bitbake/lib/bb/_vendor/ply.pyi rename bitbake/lib/{ =3D> bb/_vendor}/ply/__init__.py (80%) create mode 100644 bitbake/lib/bb/_vendor/ply/cpp.py create mode 100644 bitbake/lib/bb/_vendor/ply/ctokens.py create mode 100644 bitbake/lib/bb/_vendor/ply/lex.py rename bitbake/lib/{ =3D> bb/_vendor}/ply/yacc.py (60%) create mode 100644 bitbake/lib/bb/_vendor/ply/ygen.py create mode 100644 bitbake/lib/bb/_vendor/progressbar.pyi rename bitbake/lib/{ =3D> bb/_vendor}/progressbar/LICENSE.txt (100%) rename bitbake/lib/{ =3D> bb/_vendor}/progressbar/__init__.py (96%) rename bitbake/lib/{ =3D> bb/_vendor}/progressbar/compat.py (100%) rename bitbake/lib/{ =3D> bb/_vendor}/progressbar/progressbar.py (88%) rename bitbake/lib/{ =3D> bb/_vendor}/progressbar/widgets.py (97%) create mode 100644 bitbake/lib/bb/_vendor/simplediff.pyi rename bitbake/lib/{ =3D> bb/_vendor}/simplediff/LICENSE (100%) rename bitbake/lib/{ =3D> bb/_vendor}/simplediff/__init__.py (99%) create mode 100644 bitbake/lib/bb/_vendor/tomli/LICENSE create mode 100644 bitbake/lib/bb/_vendor/tomli/__init__.py create mode 100644 bitbake/lib/bb/_vendor/tomli/_parser.py create mode 100644 bitbake/lib/bb/_vendor/tomli/_re.py create mode 100644 bitbake/lib/bb/_vendor/tomli/_types.py create mode 100644 bitbake/lib/bb/_vendor/tomli/py.typed create mode 100644 bitbake/lib/bb/_vendor/typing_extensions.LICENSE create mode 100644 bitbake/lib/bb/_vendor/typing_extensions.py create mode 100644 bitbake/lib/bb/_vendor/typing_extensions.pyi create mode 100644 bitbake/lib/bb/asyncrpc/taskgroup.py rename bitbake/lib/bb/{fetch2 =3D> fetch}/README (100%) rename bitbake/lib/bb/{fetch2 =3D> fetch}/__init__.py (84%) rename bitbake/lib/bb/{fetch2 =3D> fetch}/az.py (78%) rename bitbake/lib/bb/{fetch2 =3D> fetch}/clearcase.py (84%) create mode 100644 bitbake/lib/bb/fetch/crate.py rename bitbake/lib/bb/{fetch2 =3D> fetch}/gcp.py (62%) rename bitbake/lib/bb/{fetch2 =3D> fetch}/git.py (52%) rename bitbake/lib/bb/{fetch2 =3D> fetch}/gitannex.py (65%) rename bitbake/lib/bb/{fetch2 =3D> fetch}/gitsm.py (63%) create mode 100644 bitbake/lib/bb/fetch/gomod.py rename bitbake/lib/bb/{fetch2 =3D> fetch}/hg.py (72%) rename bitbake/lib/bb/{fetch2 =3D> fetch}/local.py (83%) rename bitbake/lib/bb/{fetch2 =3D> fetch}/npm.py (62%) rename bitbake/lib/bb/{fetch2 =3D> fetch}/npmsw.py (70%) rename bitbake/lib/bb/{fetch2 =3D> fetch}/perforce.py (87%) rename bitbake/lib/bb/{fetch2 =3D> fetch}/repo.py (68%) rename bitbake/lib/bb/{fetch2 =3D> fetch}/s3.py (87%) rename bitbake/lib/bb/{fetch2 =3D> fetch}/sftp.py (85%) rename bitbake/lib/bb/{fetch2 =3D> fetch}/ssh.py (85%) rename bitbake/lib/bb/{fetch2 =3D> fetch}/svn.py (65%) rename bitbake/lib/bb/{fetch2 =3D> fetch}/wget.py (68%) create mode 100644 bitbake/lib/bb/fetch2.py delete mode 100644 bitbake/lib/bb/fetch2/bzr.py delete mode 100644 bitbake/lib/bb/fetch2/crate.py delete mode 100644 bitbake/lib/bb/fetch2/cvs.py delete mode 100644 bitbake/lib/bb/fetch2/osc.py create mode 100644 bitbake/lib/bb/filter.py delete mode 100644 bitbake/lib/bb/persist_data.py create mode 100644 bitbake/lib/bb/tests/fetch-testdata/software/miniupnp/d= ownload.php create mode 100644 bitbake/lib/bb/tests/fetch_import.py create mode 100644 bitbake/lib/bb/tests/filter.py create mode 100644 bitbake/lib/bb/tests/parse-tests/classes/base.bbclass create mode 100644 bitbake/lib/bb/tests/parse-tests/classes/recipe-file-cl= ass.bbclass create mode 100644 bitbake/lib/bb/tests/parse-tests/conf/bitbake.conf delete mode 100644 bitbake/lib/bb/tests/persist_data.py create mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/g1.bb create mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/h1.bb create mode 100644 bitbake/lib/bb/tests/setup.py delete mode 100644 bitbake/lib/bs4/AUTHORS.txt delete mode 100644 bitbake/lib/bs4/NEWS.txt delete mode 100644 bitbake/lib/bs4/__init__.py delete mode 100644 bitbake/lib/bs4/builder/__init__.py delete mode 100644 bitbake/lib/bs4/builder/_html5lib.py delete mode 100644 bitbake/lib/bs4/builder/_htmlparser.py delete mode 100644 bitbake/lib/bs4/builder/_lxml.py delete mode 100644 bitbake/lib/bs4/dammit.py delete mode 100644 bitbake/lib/bs4/diagnose.py delete mode 100644 bitbake/lib/bs4/element.py delete mode 100644 bitbake/lib/bs4/testing.py delete mode 100644 bitbake/lib/bs4/tests/__init__.py delete mode 100644 bitbake/lib/bs4/tests/test_builder_registry.py delete mode 100644 bitbake/lib/bs4/tests/test_docs.py delete mode 100644 bitbake/lib/bs4/tests/test_html5lib.py delete mode 100644 bitbake/lib/bs4/tests/test_htmlparser.py delete mode 100644 bitbake/lib/bs4/tests/test_lxml.py delete mode 100644 bitbake/lib/bs4/tests/test_soup.py delete mode 100644 bitbake/lib/bs4/tests/test_tree.py delete mode 100644 bitbake/lib/codegen.py delete mode 100644 bitbake/lib/ply/lex.py create mode 100644 bitbake/lib/prserv/tests.py delete mode 100644 bitbake/lib/pyinotify.py create mode 100755 bitbake/lib/toaster/orm/fixtures/check_fixtures.py create mode 100644 bitbake/packaging-pypi/bitbake-setup/BUILD.md create mode 100644 bitbake/packaging-pypi/bitbake-setup/LICENSE create mode 100644 bitbake/packaging-pypi/bitbake-setup/README.md create mode 100755 bitbake/packaging-pypi/bitbake-setup/package-bitbake-se= tup.py create mode 100644 bitbake/packaging-pypi/bitbake-setup/pyproject.toml create mode 100644 bitbake/pyproject.toml create mode 100644 bitbake/setup-schema/bitbake-setup.schema.json create mode 100644 bitbake/setup-schema/layers.schema.json create mode 100644 bitbake/vendor.txt create mode 100644 bitbake/vendor/licenses/ply/LICENSE create mode 100644 bitbake/vendor/patches/bs4-0010-lib-bs4-Avoid-soupsieve= -warning.patch create mode 100644 bitbake/vendor/patches/bs4-remove-double-imports.patch create mode 100644 bitbake/vendor/patches/progressbar-0001-lib-implement-b= asic-task-progress-support.patch create mode 100644 bitbake/vendor/patches/progressbar-0002-bitbake-Add-ini= tial-pass-of-SPDX-license-headers-to-.patch create mode 100644 bitbake/vendor/patches/progressbar-0003-bitbake-progres= sbar-accept-value-over-initial-maxval.patch create mode 100644 bitbake/vendor/patches/progressbar-0004-progressbar-Add= -self._fd_console-to-use-for-self._ha.patch create mode 100644 bitbake/vendor/patches/progressbar-0005-progressbar-Mak= e-bars-show-correctly-with-maxval-0.patch create mode 100644 bitbake/vendor/patches/progressbar-0006-progressbar-kno= tty-Allow-mixing-log-messages-and-pro.patch diff --git a/.gitignore b/.gitignore index b5f6149e..f397ff51 100644 --- a/.gitignore +++ b/.gitignore @@ -4,4 +4,4 @@ __pycache__ .config.yaml* .kas_shell_history .isar_test_shell_history -build* +/build* diff --git a/bitbake/.b4-config b/bitbake/.b4-config new file mode 100644 index 00000000..047f0b94 --- /dev/null +++ b/bitbake/.b4-config @@ -0,0 +1,4 @@ +[b4] + send-series-to =3D bitbake-devel@lists.openembedded.org + send-auto-cc-cmd =3D ./contrib/b4-wrapper-bitbake.py send-auto-cc-cmd + prep-pre-flight-checks =3D disable-needs-checking diff --git a/bitbake/.gitignore b/bitbake/.gitignore index fbf9eb20..23cb9688 100644 --- a/bitbake/.gitignore +++ b/bitbake/.gitignore @@ -3,6 +3,7 @@ *~ pyshtables.py /doc/manual/html/ +**/bitbake-builds/ /build/ /bin/bitbakec *.swp @@ -16,3 +17,4 @@ lib/toaster/contrib/tts/backlog.txt lib/toaster/contrib/tts/log/* lib/toaster/contrib/tts/.cache/* lib/bb/tests/runqueue-tests/bitbake-cookerdaemon.log +packaging-workspace/ diff --git a/bitbake/AGENTS.md b/bitbake/AGENTS.md new file mode 100644 index 00000000..833eca7f --- /dev/null +++ b/bitbake/AGENTS.md @@ -0,0 +1,55 @@ +# AGENTS.md =E2=80=94 BitBake + +Guidance for AI agents working in the `bitbake` repository. + +## Making changes + +- No web PR workflow. The project uses **patches on mailing lists** + (bitbake-devel@lists.openembedded.org), tracked via patchwork. Don't ass= ume + GitHub PR conventions apply. +- One Git commit per logical change (e.g., one commit per recipe added/upg= raded). +- Commit message format: + - Summary line prefixed with the recipe name or short file path, + e.g. `openssl: upgrade 3.2.0 -> 3.2.1` or `classes/rootfs-postcommands= : fix typo`. + Check `git log --oneline ` for the prefix convention already in = use. + - Body explains what/why/how, and how it was tested. + - Prefer explaining *why* a change was made over restating what's + already visible in the diff, unless the diff is long/hard to follow + or the reason for the change isn't known =E2=80=94 in those cases, sum= marize + what changed instead. + - Be succinct: use as few sentences and lines as possible while still + covering what/why/how. Cut restatements, filler, and any detail a + reviewer wouldn't need (e.g. exact test-run counts, step-by-step + narration). One short paragraph is usually enough; only add a second + if a distinct point (e.g. testing) needs it. + - The most relevant information about the change should be presented fir= st. + - Reference bugs as `[YOCTO #1234]` when applicable. + - Use `Reported-by`, `Suggested-by`, `Tested-by`, `Reviewed-by`, `Cc` ta= gs + where relevant. + +### AI-generated code + +- **Mandatory**: any AI-generated code/commit must add an `AI-Generated:` + tag line in the commit message, placed *before* `Signed-off-by` (if pres= ent), + naming the tool used, e.g.: + + ``` + component: Add the ability to ... + + AI-Generated: Uses GitHub Copilot + ``` + +- Also add a code comment or other in-patch indication that the code is + AI-generated (recommended, not just the commit tag). +- **Mandatory** Do not automatically add a `Signed-off-by`. This tag must + always be written by the human contributor, never by the AI tool =E2=80= =94 the + DCO/Developer's Statement of Origin still applies in full. +- Contributor is responsible for confirming the AI tool's terms don't + conflict with the project's open source license/IP policy, and that + any third-party copyrighted material in the AI output is properly + licensed before contributing it. See the Linux Foundation generative AI + guidance: https://www.linuxfoundation.org/legal/generative-ai. +- `Co-authored-by` should be omitted in favor of the `AI-generated` tag. + +When in doubt about a convention, grep `git log --oneline ` in the +target repo for precedent before inventing a new style. diff --git a/bitbake/README b/bitbake/README index e9f4c858..39e4a978 100644 --- a/bitbake/README +++ b/bitbake/README @@ -13,7 +13,7 @@ Bitbake plain documentation can be found under the doc di= rectory or its integrat html version at the Yocto Project website: https://docs.yoctoproject.org =20 -Bitbake requires Python version 3.8 or newer. +Bitbake requires Python version 3.9 or newer. =20 Contributing ------------ diff --git a/bitbake/TODO b/bitbake/TODO index 54d091fb..4e0bd9a7 100644 --- a/bitbake/TODO +++ b/bitbake/TODO @@ -1,36 +1,31 @@ -- Reimplement the interactive mode as a proper ui -- Continue dropping fatal/SystemExit/sys.exit usage in favor of raising - appropriate exceptions -- Continue pylint / pyflakes / pychecker / pep8 fixups + +- Interfaces + - Add an interactive mode to knotty (similar to the interactive mode we = dropped) + - Fix up or drop the ncurses interface? Currently too limited to be usef= ul. + - Concerns about whether teamcity interface works. No testing. + +- Improve exception handling: + - dropping fatal/SystemExit/sys.exit usage in favor of raising + appropriate exceptions + - Audit bb.fatal usage - these should all be able to be replaced with + exceptions + +- Continue pylint / pyflakes / pychecker / pep8 fixups - focus on real iss= ues + or performance issues, not cosmetic ones. + - Drop os.system usage in favor of direct subprocess usage or a subprocess wrapper -- Kill the execution of 'tee' for the task log file in build.py -- Fix up the exception handling - - - Kill exec_task's catch of FuncFailed, instead catch it in the other - callers of exec_task/exec_func - - What exactly is the purpose of the "EventException"? I can see using = an - exception like that, *perhaps*, to abstract away exceptions raised by - event handlers, but it has no place in bb.build.exec_task =20 - BUG: if you chmod 000 local.conf, it silently doesn't parse it, when it should really fail, so the user can fix the problem. =20 -- Audit bb.fatal usage - these should all be able to be replaced with - exceptions -- Figure out how to handle the ncurses UI. Should some of our logging - formatting stuff be made common to all of bitbake, or perhaps all UIs vi= a - the UIHelper? =20 Long term, high impact: =20 - - Change override application to actually *move* it over -- so the origi= nal - override specific version of the variable goes away, rather than stick= ing - around as a duplicate. - - Change the behavior when a variable is referenced and is unset. Today= , it - evaluates to ${FOO} and then shell has a chance to expand it, but this= is - far from ideal. We had considered evaluating it to the empty string, = but - that has other potential problems. Frans Meulenbroeks has proposed ju= st + - Consider changing the behavior when a variable is referenced and is un= set + Today, it evaluates to ${FOO} and then shell has a chance to expand it= , but + this is far from ideal. We had considered evaluating it to the empty = string, + but that has other potential problems. Frans Meulenbroeks has propose= d just erroring when this occurs, as we can always define default values for = the variables in bitbake.conf. This seems reasonable. My only concern wi= th that is the case where you want to reference a shell variable with odd @@ -38,25 +33,3 @@ Long term, high impact: expansion rather than normal $. To handle that case, we'd really need= a way to escape / disable bitbake variable expansion, \${} perhaps. =20 -Uncertain: - - - Leverage the python 2.6 multiprocessing module - - - Worker processes for bb.cooker - - Server / UI processes - - - Create a bitbake configuration class which is utilized by the library,= not - just bin/bitbake. This class should be responsible for extracting - configuration parameters from the metadata for bitbake internal use, a= s well - as pulling specific items like BBDEBUG, and importing settings from an - optparse options object. - - - Python version bits - - - Utilize the new string formatting where appropriate - - Do we need to take into account the bytes literals changes? - - Do we have any file-like objects that would benefit from using the "= io" - module? - - Do we want to leverage the abstract base classes in collections? - - Aside: Set methods now accept multiple iterables - diff --git a/bitbake/bin/bitbake b/bitbake/bin/bitbake index f494eaa1..71bc3791 100755 --- a/bitbake/bin/bitbake +++ b/bitbake/bin/bitbake @@ -27,11 +27,11 @@ from bb.main import bitbake_main, BitBakeConfigParamete= rs, BBMainException =20 bb.utils.check_system_locale() =20 -__version__ =3D "2.8.0" +__version__ =3D "2.20.0" =20 if __name__ =3D=3D "__main__": if __version__ !=3D bb.__version__: - sys.exit("Bitbake core version and program version mismatch!") + sys.exit("Bitbake core library version and program version mismatc= h!") try: sys.exit(bitbake_main(BitBakeConfigParameters(sys.argv), cookerdata.CookerConfiguration())) diff --git a/bitbake/bin/bitbake-config-build b/bitbake/bin/bitbake-config-= build new file mode 120000 index 00000000..11e6df80 --- /dev/null +++ b/bitbake/bin/bitbake-config-build @@ -0,0 +1 @@ +bitbake-layers \ No newline at end of file diff --git a/bitbake/bin/bitbake-diffsigs b/bitbake/bin/bitbake-diffsigs index 8202c786..99660d5c 100755 --- a/bitbake/bin/bitbake-diffsigs +++ b/bitbake/bin/bitbake-diffsigs @@ -22,6 +22,13 @@ sys.path.insert(0, os.path.join(os.path.dirname(os.path.= dirname(sys.argv[0])), ' import bb.tinfoil import bb.siggen import bb.msg +from bb import __version__ as libbb_version + +__version__ =3D "2.20.0" + +if __version__ !=3D libbb_version: + sys.exit("Bitbake core library version and program version mismatch!") + =20 myname =3D os.path.basename(sys.argv[0]) logger =3D bb.msg.logger_create(myname) @@ -72,16 +79,17 @@ def find_siginfo_task(bbhandler, pn, taskname, sig1=3DN= one, sig2=3DNone): elif sig2 not in sigfiles: logger.error('No sigdata files found matching %s %s with signa= ture %s' % (pn, taskname, sig2)) sys.exit(1) + + latestfiles =3D [sigfiles[sig1]['path'], sigfiles[sig2]['path']] else: sigfiles =3D find_siginfo(bbhandler, pn, taskname) latestsigs =3D sorted(sigfiles.keys(), key=3Dlambda h: sigfiles[h]= ['time'])[-2:] if not latestsigs: logger.error('No sigdata files found matching %s %s' % (pn, ta= skname)) sys.exit(1) - sig1 =3D latestsigs[0] - sig2 =3D latestsigs[1] - - latestfiles =3D [sigfiles[sig1]['path'], sigfiles[sig2]['path']] + latestfiles =3D [sigfiles[latestsigs[0]]['path']] + if len(latestsigs) > 1: + latestfiles.append(sigfiles[latestsigs[1]]['path']) =20 return latestfiles =20 diff --git a/bitbake/bin/bitbake-getvar b/bitbake/bin/bitbake-getvar index 8901f99a..7f4d3745 100755 --- a/bitbake/bin/bitbake-getvar +++ b/bitbake/bin/bitbake-getvar @@ -10,15 +10,23 @@ import io import os import sys import warnings +import logging warnings.simplefilter("default") =20 bindir =3D os.path.dirname(__file__) topdir =3D os.path.dirname(bindir) sys.path[0:0] =3D [os.path.join(topdir, 'lib')] =20 +import bb.providers import bb.tinfoil +from bb import __version__ as libbb_version + +__version__ =3D "2.20.0" =20 if __name__ =3D=3D "__main__": + if __version__ !=3D libbb_version: + sys.exit("Bitbake core library version and program version mismatc= h!") + parser =3D argparse.ArgumentParser(description=3D"Bitbake Query Variab= le") parser.add_argument("variable", help=3D"variable name to query") parser.add_argument("-r", "--recipe", help=3D"Recipe name to query", d= efault=3DNone, required=3DFalse) @@ -37,13 +45,22 @@ if __name__ =3D=3D "__main__": sys.exit("--flag only makes sense with --value") =20 quiet =3D args.quiet or args.value + if quiet: + logger =3D logging.getLogger("BitBake") + logger.setLevel(logging.WARNING) + with bb.tinfoil.Tinfoil(tracking=3DTrue, setup_logging=3Dnot quiet) as= tinfoil: if args.recipe: tinfoil.prepare(quiet=3D3 if quiet else 2) - d =3D tinfoil.parse_recipe(args.recipe) + try: + d =3D tinfoil.parse_recipe(args.recipe) + except bb.providers.NoProvider as e: + sys.exit(str(e)) else: tinfoil.prepare(quiet=3D2, config_only=3DTrue) - d =3D tinfoil.config_data + # Expand keys and run anonymous functions to get identical res= ult to + # "bitbake -e" + d =3D tinfoil.finalizeData() =20 value =3D None if args.flag: diff --git a/bitbake/bin/bitbake-hashclient b/bitbake/bin/bitbake-hashclien= t index 610787ed..ce5fd173 100755 --- a/bitbake/bin/bitbake-hashclient +++ b/bitbake/bin/bitbake-hashclient @@ -16,8 +16,12 @@ import time import warnings import netrc import json +import statistics +import textwrap warnings.simplefilter("default") =20 +__version__ =3D "2.20.0" + try: import tqdm ProgressBar =3D tqdm.tqdm @@ -39,6 +43,12 @@ sys.path.insert(0, os.path.join(os.path.dirname(os.path.= dirname(__file__)), 'lib =20 import hashserv import bb.asyncrpc +from bb import __version__ as libbb_version + +if __version__ !=3D libbb_version: + sys.exit("Bitbake core library version and program version mismatch!") +if __version__ !=3D hashserv.__version__: + sys.exit("Bitbake hashserv library version and program version mismatc= h!") =20 DEFAULT_ADDRESS =3D 'unix://./hashserve.sock' METHOD =3D 'stress.test.method' @@ -81,6 +91,7 @@ def main(): nonlocal found_hashes nonlocal missed_hashes nonlocal max_time + nonlocal times =20 with hashserv.create_client(args.address) as client: for i in range(args.requests): @@ -98,29 +109,41 @@ def main(): else: missed_hashes +=3D 1 =20 - max_time =3D max(elapsed, max_time) + times.append(elapsed) pbar.update() =20 max_time =3D 0 found_hashes =3D 0 missed_hashes =3D 0 lock =3D threading.Lock() - total_requests =3D args.clients * args.requests + times =3D [] start_time =3D time.perf_counter() - with ProgressBar(total=3Dtotal_requests) as pbar: + with ProgressBar(total=3Dargs.clients * args.requests) as pbar: threads =3D [threading.Thread(target=3Dthread_main, args=3D(pb= ar, lock), daemon=3DFalse) for _ in range(args.clients)] for t in threads: t.start() =20 for t in threads: t.join() + total_elapsed =3D time.perf_counter() - start_time =20 - elapsed =3D time.perf_counter() - start_time with lock: - print("%d requests in %.1fs. %.1f requests per second" % (tota= l_requests, elapsed, total_requests / elapsed)) - print("Average request time %.8fs" % (elapsed / total_requests= )) - print("Max request time was %.8fs" % max_time) - print("Found %d hashes, missed %d" % (found_hashes, missed_has= hes)) + mean =3D statistics.mean(times) + median =3D statistics.median(times) + stddev =3D statistics.pstdev(times) + + print(f"Number of clients: {args.clients}") + print(f"Requests per client: {args.requests}") + print(f"Number of requests: {len(times)}") + print(f"Total elapsed time: {total_elapsed:.3f}s") + print(f"Total request rate: {len(times)/total_elapsed:.3f} r= eq/s") + print(f"Average request time: {mean:.3f}s") + print(f"Median request time: {median:.3f}s") + print(f"Request time std dev: {stddev:.3f}s") + print(f"Maximum request time: {max(times):.3f}s") + print(f"Minimum request time: {min(times):.3f}s") + print(f"Hashes found: {found_hashes}") + print(f"Hashes missed: {missed_hashes}") =20 if args.report: with ProgressBar(total=3Dargs.requests) as pbar: @@ -212,6 +235,27 @@ def main(): print("New hashes marked: %d" % result["count"]) return 0 =20 + def handle_gc_mark_stream(args, client): + stdin =3D (l.strip() for l in sys.stdin) + marked_hashes =3D 0 + + try: + result =3D client.gc_mark_batch(args.mark, stdin) + marked_hashes =3D result["count"] + except ConnectionError: + logger.warning( + "Server doesn't seem to support `gc-mark-stream`. Sending = " + "hashes sequentially using `gc-mark` API." + ) + for line in stdin: + pairs =3D line.split() + condition =3D dict(zip(pairs[::2], pairs[1::2])) + result =3D client.gc_mark(args.mark, condition) + marked_hashes +=3D result["count"] + + print("New hashes marked: %d" % marked_hashes) + return 0 + def handle_gc_sweep(args, client): result =3D client.gc_sweep(args.mark) print("Removed %d rows" % result["count"]) @@ -225,7 +269,45 @@ def main(): print("true" if result else "false") return 0 =20 - parser =3D argparse.ArgumentParser(description=3D'Hash Equivalence Cli= ent') + def handle_ping(args, client): + times =3D [] + for i in range(1, args.count + 1): + if not args.quiet: + print(f"Ping {i} of {args.count}... ", end=3D"") + start_time =3D time.perf_counter() + client.ping() + elapsed =3D time.perf_counter() - start_time + times.append(elapsed) + if not args.quiet: + print(f"{elapsed:.3f}s") + + mean =3D statistics.mean(times) + median =3D statistics.median(times) + std_dev =3D statistics.pstdev(times) + + if not args.quiet: + print("------------------------") + print(f"Number of pings: {len(times)}") + print(f"Average round trip time: {mean:.3f}s") + print(f"Median round trip time: {median:.3f}s") + print(f"Round trip time std dev: {std_dev:.3f}s") + print(f"Min time is: {min(times):.3f}s") + print(f"Max time is: {max(times):.3f}s") + return 0 + + parser =3D argparse.ArgumentParser( + formatter_class=3Dargparse.RawDescriptionHelpFormatter, + description=3D'Hash Equivalence Client', + epilog=3Dtextwrap.dedent( + """ + Possible ADDRESS options are: + unix://PATH Connect to UNIX domain socket at PATH + ws://HOST[:PORT] Connect to websocket at HOST:PORT (def= ault port is 80) + wss://HOST[:PORT] Connect to secure websocket at HOST:PO= RT (default port is 443) + HOST:PORT Connect to TCP server at HOST:PORT + """ + ), + ) parser.add_argument('--address', default=3DDEFAULT_ADDRESS, help=3D'Se= rver address (default "%(default)s")') parser.add_argument('--log', default=3D'WARNING', help=3D'Set logging = level') parser.add_argument('--login', '-l', metavar=3D"USERNAME", help=3D"Aut= henticate as USERNAME") @@ -313,6 +395,16 @@ def main(): help=3D"Keep entries in table where KEY =3D= =3D VALUE") gc_mark_parser.set_defaults(func=3Dhandle_gc_mark) =20 + gc_mark_parser_stream =3D subparsers.add_parser( + 'gc-mark-stream', + help=3D( + "Mark multiple hashes to be retained for garbage collection. I= nput should be provided via stdin, " + "with each line formatted as key-value pairs separated by spac= es, for example 'column1 foo column2 bar'." + ) + ) + gc_mark_parser_stream.add_argument("mark", help=3D"Mark for this garba= ge collection operation") + gc_mark_parser_stream.set_defaults(func=3Dhandle_gc_mark_stream) + gc_sweep_parser =3D subparsers.add_parser('gc-sweep', help=3D"Perform = garbage collection and delete any entries that are not marked") gc_sweep_parser.add_argument("mark", help=3D"Mark for this garbage col= lection operation") gc_sweep_parser.set_defaults(func=3Dhandle_gc_sweep) @@ -322,6 +414,11 @@ def main(): unihash_exists_parser.add_argument("unihash", help=3D"Unihash to check= ") unihash_exists_parser.set_defaults(func=3Dhandle_unihash_exists) =20 + ping_parser =3D subparsers.add_parser('ping', help=3D"Ping server") + ping_parser.add_argument("-n", "--count", type=3Dint, help=3D"Number o= f pings. Default is %(default)s", default=3D10) + ping_parser.add_argument("-q", "--quiet", action=3D"store_true", help= =3D"Don't print each ping; only print results") + ping_parser.set_defaults(func=3Dhandle_ping) + args =3D parser.parse_args() =20 logger =3D logging.getLogger('hashserv') diff --git a/bitbake/bin/bitbake-hashserv b/bitbake/bin/bitbake-hashserv index 4bfb7abf..3e68b6a2 100755 --- a/bitbake/bin/bitbake-hashserv +++ b/bitbake/bin/bitbake-hashserv @@ -12,6 +12,8 @@ import argparse import sqlite3 import warnings =20 +__version__ =3D "2.20.0" + warnings.simplefilter("default") =20 sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__))= , "lib")) @@ -19,14 +21,15 @@ sys.path.insert(0, os.path.join(os.path.dirname(os.path= .dirname(__file__)), "lib import hashserv from hashserv.server import DEFAULT_ANON_PERMS =20 -VERSION =3D "1.0.0" +if __version__ !=3D hashserv.__version__: + sys.exit("Bitbake library hashserv version and program version mismatc= h!") =20 DEFAULT_BIND =3D "unix://./hashserve.sock" =20 =20 def main(): parser =3D argparse.ArgumentParser( - description=3D"Hash Equivalence Reference Server. Version=3D%s" % = VERSION, + description=3D"Hash Equivalence Reference Server. Version=3D%s" % = __version__, formatter_class=3Dargparse.RawTextHelpFormatter, epilog=3D""" The bind address may take one of the following formats: @@ -125,6 +128,11 @@ The following permissions are supported by the server: default=3Dos.environ.get("HASHSERVER_ADMIN_PASSWORD", None), help=3D"Create default admin user with password ADMIN_PASSWORD ($H= ASHSERVER_ADMIN_PASSWORD)", ) + parser.add_argument( + "--reuseport", + action=3D"store_true", + help=3D"Enable SO_REUSEPORT, allowing multiple servers to bind to = the same port for load balancing", + ) =20 args =3D parser.parse_args() =20 @@ -132,7 +140,9 @@ The following permissions are supported by the server: =20 level =3D getattr(logging, args.log.upper(), None) if not isinstance(level, int): - raise ValueError("Invalid log level: %s (Try ERROR/WARNING/INFO/DE= BUG)" % args.log) + raise ValueError( + "Invalid log level: %s (Try ERROR/WARNING/INFO/DEBUG)" % args.= log + ) =20 logger.setLevel(level) console =3D logging.StreamHandler() @@ -155,6 +165,7 @@ The following permissions are supported by the server: anon_perms=3Danon_perms, admin_username=3Dargs.admin_user, admin_password=3Dargs.admin_password, + reuseport=3Dargs.reuseport, ) server.serve_forever() return 0 diff --git a/bitbake/bin/bitbake-layers b/bitbake/bin/bitbake-layers index d4b1d1aa..c1eafb22 100755 --- a/bitbake/bin/bitbake-layers +++ b/bitbake/bin/bitbake-layers @@ -17,25 +17,38 @@ import argparse import warnings warnings.simplefilter("default") =20 +__version__ =3D "2.20.0" + bindir =3D os.path.dirname(__file__) +toolname =3D os.path.basename(__file__).split(".")[0] topdir =3D os.path.dirname(bindir) sys.path[0:0] =3D [os.path.join(topdir, 'lib')] =20 import bb.tinfoil +import bb.cooker import bb.msg +from bb import __version__ as libbb_version + +if __version__ !=3D libbb_version: + sys.exit("Bitbake library version and program version mismatch!") =20 -logger =3D bb.msg.logger_create('bitbake-layers', sys.stdout) +logger =3D bb.msg.logger_create(toolname, sys.stdout) =20 def main(): parser =3D argparse.ArgumentParser( - description=3D"BitBake layers utility", + description=3D{"bitbake-layers":"BitBake layers utility", "bitbake= -config-build":"BitBake build configuration utility"}[toolname], epilog=3D"Use %(prog)s --help to get help on a specif= ic command", add_help=3DFalse) parser.add_argument('-d', '--debug', help=3D'Enable debug output', act= ion=3D'store_true') parser.add_argument('-q', '--quiet', help=3D'Print only errors', actio= n=3D'store_true') - parser.add_argument('-F', '--force', help=3D'Force add without recipe = parse verification', action=3D'store_true') + if toolname =3D=3D "bitbake-layers": + parser.add_argument('-F', '--force', help=3D'Forced execution: can= be specified multiple times. -F will force add without recipe parse verifi= cation and -FF will additionally force the run without layer parsing.', act= ion=3D'count') parser.add_argument('--color', choices=3D['auto', 'always', 'never'], = default=3D'auto', help=3D'Colorize output (where %(metavar)s is %(choices)s= )', metavar=3D'COLOR') =20 + # Set the default for --force here rather than above so that it is alw= ays + # defined even if the option is not available. + parser.set_defaults(force=3D0) + global_args, unparsed_args =3D parser.parse_known_args() =20 # Help is added here rather than via add_help=3DTrue, as we don't want= it to @@ -57,18 +70,27 @@ def main(): level=3Dlogger.getEffectiveLevel()) =20 plugins =3D [] - tinfoil =3D bb.tinfoil.Tinfoil(tracking=3DTrue) - tinfoil.logger.setLevel(logger.getEffectiveLevel()) - try: - tinfoil.prepare(True) - for path in ([topdir] + - tinfoil.config_data.getVar('BBPATH').split(':')): - pluginpath =3D os.path.join(path, 'lib', 'bblayers') + with bb.tinfoil.Tinfoil(tracking=3DTrue) as tinfoil: + tinfoil.logger.setLevel(logger.getEffectiveLevel()) + + if global_args.force > 1: + bbpaths =3D [] + else: + if toolname =3D=3D "bitbake-config-build": + extra_features =3D [bb.cooker.CookerFeatures.SKIP_FRAGMENT= S] + else: + extra_features =3D None + tinfoil.prepare(True, extra_features=3Dextra_features) + bbpaths =3D tinfoil.config_data.getVar('BBPATH').split(':') + + for path in ([topdir] + bbpaths): + pluginbasepath =3D {"bitbake-layers":'bblayers', 'bitbake-conf= ig-build':'bbconfigbuild'}[toolname] + pluginpath =3D os.path.join(path, 'lib', pluginbasepath) bb.utils.load_plugins(logger, plugins, pluginpath) =20 registered =3D False for plugin in plugins: - if hasattr(plugin, 'tinfoil_init'): + if hasattr(plugin, 'tinfoil_init') and global_args.force <=3D = 1: plugin.tinfoil_init(tinfoil) if hasattr(plugin, 'register_commands'): registered =3D True @@ -86,8 +108,6 @@ def main(): tinfoil.config_data.enableTracking() =20 return args.func(args) - finally: - tinfoil.shutdown() =20 =20 if __name__ =3D=3D "__main__": diff --git a/bitbake/bin/bitbake-prserv b/bitbake/bin/bitbake-prserv index ad0a0694..8ba246c0 100755 --- a/bitbake/bin/bitbake-prserv +++ b/bitbake/bin/bitbake-prserv @@ -11,19 +11,30 @@ import argparse import warnings warnings.simplefilter("default") =20 +__version__ =3D "2.20.0" + sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__))= , "lib")) =20 import prserv import prserv.serv - -VERSION =3D "1.1.0" +=20 +if __name__ =3D=3D "__main__": + if __version__ !=3D prserv.__version__: + sys.exit("Bitbake prserv library version and program version misma= tch!") =20 PRHOST_DEFAULT=3D"0.0.0.0" PRPORT_DEFAULT=3D8585 =20 +def init_logger(logfile, loglevel): + numeric_level =3D getattr(logging, loglevel.upper(), None) + if not isinstance(numeric_level, int): + raise ValueError("Invalid log level: %s" % loglevel) + FORMAT =3D "%(asctime)-15s %(message)s" + logging.basicConfig(level=3Dnumeric_level, filename=3Dlogfile, format= =3DFORMAT) + def main(): parser =3D argparse.ArgumentParser( - description=3D"BitBake PR Server. Version=3D%s" % VERSION, + description=3D"BitBake PR Server. Version=3D%s" % __version__, formatter_class=3Dargparse.RawTextHelpFormatter) =20 parser.add_argument( @@ -70,12 +81,25 @@ def main(): action=3D"store_true", help=3D"open database in read-only mode", ) + parser.add_argument( + "-u", + "--upstream", + default=3Dos.environ.get("PRSERV_UPSTREAM", None), + help=3D"Upstream PR service (host:port)", + ) =20 args =3D parser.parse_args() - prserv.init_logger(os.path.abspath(args.log), args.loglevel) + init_logger(os.path.abspath(args.log), args.loglevel) =20 if args.start: - ret=3Dprserv.serv.start_daemon(args.file, args.host, args.port, os= .path.abspath(args.log), args.read_only) + ret=3Dprserv.serv.start_daemon( + args.file, + args.host, + args.port, + os.path.abspath(args.log), + args.read_only, + args.upstream + ) elif args.stop: ret=3Dprserv.serv.stop_daemon(args.host, args.port) else: diff --git a/bitbake/bin/bitbake-selftest b/bitbake/bin/bitbake-selftest index f25f23b1..855abb1f 100755 --- a/bitbake/bin/bitbake-selftest +++ b/bitbake/bin/bitbake-selftest @@ -11,14 +11,20 @@ import warnings warnings.simplefilter("default") sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(__file__))= , 'lib')) =20 +__version__ =3D "2.20.0" + import unittest try: import bb import hashserv + import prserv import layerindexlib except RuntimeError as exc: sys.exit(str(exc)) =20 +if __version__ !=3D bb.__version__: + sys.exit("Bitbake core library version and program version mismatch!") + tests =3D ["bb.tests.codeparser", "bb.tests.color", "bb.tests.cooker", @@ -26,13 +32,16 @@ tests =3D ["bb.tests.codeparser", "bb.tests.data", "bb.tests.event", "bb.tests.fetch", + "bb.tests.fetch_import", "bb.tests.parse", - "bb.tests.persist_data", "bb.tests.runqueue", + "bb.tests.setup", "bb.tests.siggen", "bb.tests.utils", "bb.tests.compression", + "bb.tests.filter", "hashserv.tests", + "prserv.tests", "layerindexlib.tests.layerindexobj", "layerindexlib.tests.restapi", "layerindexlib.tests.cooker"] @@ -64,6 +73,8 @@ ENV_HELP =3D """\ Environment variables: BB_SKIP_NETTESTS set to 'yes' in order to skip tests using network connection + BB_SKIP_PYPI_TESTS set to 'no' to run PyPI packaging tests + (default: yes/skip) BB_TMPDIR_NOCLEAN set to 'yes' to preserve test tmp directories """ =20 diff --git a/bitbake/bin/bitbake-server b/bitbake/bin/bitbake-server index 454a3919..6157fb33 100755 --- a/bitbake/bin/bitbake-server +++ b/bitbake/bin/bitbake-server @@ -12,6 +12,8 @@ warnings.simplefilter("default") import logging sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(sys.argv[0= ])), 'lib')) =20 +__version__ =3D "2.20.0" + import bb =20 bb.utils.check_system_locale() @@ -23,13 +25,17 @@ if len(sys.argv) !=3D 11 or not sys.argv[1].startswith(= "decafbad"): =20 import bb.server.process =20 +if __name__ =3D=3D "__main__": + if __version__ !=3D bb.__version__: + sys.exit("Bitbake core library version and program version mismatc= h!") + lockfd =3D int(sys.argv[2]) readypipeinfd =3D int(sys.argv[3]) logfile =3D sys.argv[4] lockname =3D sys.argv[5] sockname =3D sys.argv[6] timeout =3D float(sys.argv[7]) -profile =3D bool(int(sys.argv[8])) +profile =3D sys.argv[8] xmlrpcinterface =3D (sys.argv[9], int(sys.argv[10])) if xmlrpcinterface[0] =3D=3D "None": xmlrpcinterface =3D (None, xmlrpcinterface[1]) @@ -38,9 +44,9 @@ if xmlrpcinterface[0] =3D=3D "None": with open('/dev/null', 'r') as si: os.dup2(si.fileno(), sys.stdin.fileno()) =20 -so =3D open(logfile, 'a+') -os.dup2(so.fileno(), sys.stdout.fileno()) -os.dup2(so.fileno(), sys.stderr.fileno()) +with open(logfile, 'a+') as so: + os.dup2(so.fileno(), sys.stdout.fileno()) + os.dup2(so.fileno(), sys.stderr.fileno()) =20 # Have stdout and stderr be the same so log output matches chronologically # and there aren't two seperate buffers diff --git a/bitbake/bin/bitbake-setup b/bitbake/bin/bitbake-setup new file mode 100755 index 00000000..e50117ea --- /dev/null +++ b/bitbake/bin/bitbake-setup @@ -0,0 +1,1481 @@ +#!/usr/bin/env python3 + +# +# SPDX-License-Identifier: GPL-2.0-only +# + +import argparse +import configparser +import copy +import datetime +import functools +import glob +import json +import logging +import os +import shutil +import signal +import string +import subprocess +import sys +import textwrap +import time + +__version__ =3D "2.20.0" + +bindir =3D os.path.abspath(os.path.dirname(__file__)) +sys.path[0:0] =3D [os.path.join(os.path.dirname(bindir), 'lib')] + +import bb.msg # noqa: E402 +import bb.fetch # noqa: E402 +import bb.process # noqa: E402 +from bb import __version__ as libbb_version # noqa: E402 + +if __name__ =3D=3D "__main__": + if __version__ !=3D libbb_version: + sys.exit("Bitbake core library version and program version mismatc= h!") + +logger =3D bb.msg.logger_create('bitbake-setup', sys.stdout) + +# These settings can only be set in the global context +GLOBAL_ONLY_SETTINGS =3D ( + "top-dir-prefix", + "top-dir-name", +) + +def color_enabled() -> bool: + """ + Our logger has a BBLogFormatter formatter which holds whether color is + enabled or not. Return this value. + """ + return logger.handlers[0].formatter.color_enabled + +def get_diff_color_param() -> str: + return "--color=3Dalways" if color_enabled() else "--color=3Dnever" + +def print_configs(prompt: str, choices: list[str], descriptions: list[str]= =3D []): + """ + Helper function to print a list of choices and align the output. + Each option name is made bold to stand out, unless color is not enable= d in + our logger. + """ + if not prompt.endswith(':'): + prompt +=3D ":" + logger.plain(prompt) + + if not descriptions: + descriptions =3D ["" for _ in choices] + + # maximum size of all choices, for alignment + cmax =3D max([len(c) for c in choices]) + 1 + + for n, c in enumerate(choices): + msg =3D f"{n + 1}. " + if color_enabled(): + # make it bold + msg +=3D "\033[1m" + msg +=3D f"{c:<{cmax}}" + if color_enabled(): + msg +=3D "\033[0m" + msg +=3D f" {descriptions[n]}" + logger.plain(msg) + +# If bitbake is from a release tarball or somewhere like pypi where +# updates may not be straightforward, prefer to use the git repo as the +# default registry +def get_default_registry(): + internal_registry =3D os.path.normpath(os.path.dirname(__file__) + "/.= ./default-registry") + git_registry =3D "git://git.openembedded.org/bitbake;protocol=3Dhttps;= branch=3Dmaster;rev=3Dmaster" + if os.path.exists(os.path.dirname(__file__) + "/../.git"): + return internal_registry + else: + return git_registry + +def cache_dir(top_dir): + return os.path.join(top_dir, '.bitbake-setup-cache') + +def init_bb_cache(top_dir, settings, args): + dldir =3D settings["default"]["dl-dir"] + bb_cachedir =3D os.path.join(cache_dir(top_dir), 'bitbake-cache') + os.makedirs(bb_cachedir, exist_ok=3DTrue) + + d =3D bb.data.init() + d.setVar("DL_DIR", dldir) + d.setVar("BB_CACHEDIR", bb_cachedir) + d.setVar("__BBSRCREV_SEEN", "1") + if args.no_network: + d.setVar("BB_SRCREV_POLICY", "cache") + bb.fetch.fetcher_init(d) + return d + +def save_bb_cache(): + bb.fetch.fetcher_parse_save() + bb.fetch.fetcher_parse_done() + +def get_config_name(config): + suffix =3D '.conf.json' + config_file =3D os.path.basename(config) + if config_file.endswith(suffix): + return config_file[:-len(suffix)] + else: + raise Exception("Config file {} does not end with {}, please renam= e the file.".format(config, suffix)) + +def write_upstream_config(config_dir, config_data): + with open(os.path.join(config_dir, "config-upstream.json"),'w') as s: + json.dump(config_data, s, sort_keys=3DTrue, indent=3D4) + +def write_sources_fixed_revisions(config_dir, layer_dir, config_data): + json_path =3D os.path.join(config_dir, "sources-fixed-revisions.json") + json_link =3D os.path.join(layer_dir, "sources-fixed-revisions.json") + sources =3D {} + sources['sources'] =3D config_data + with open(os.path.join(config_dir, "sources-fixed-revisions.json"),'w'= ) as s: + json.dump(sources, s, sort_keys=3DTrue, indent=3D4) + if not os.path.lexists(json_link): + os.symlink(os.path.relpath(json_path ,layer_dir), json_link) + +def commit_config(config_dir): + bb.process.run(['git', '-C', config_dir, 'add', '.']) + bb.process.run(['git', '-C', config_dir, 'commit', '--allow-empty', '-= -no-verify', '-a', '-m', 'Configuration at ' + time.asctime()]) + +def _write_layer_list(dest, repodirs): + layers =3D [] + for r in repodirs: + for root, _, files in os.walk(os.path.join(dest,r)): + if os.path.basename(root) =3D=3D 'conf' and 'layer.conf' in fi= les: + layers.append(os.path.relpath(os.path.dirname(root), dest)= ) + layers_f =3D os.path.join(dest, ".oe-layers.json") + with open(layers_f, 'w') as f: + json.dump({"version":"1.0","layers":layers}, f, sort_keys=3DTrue, = indent=3D4) + +def add_unique_timestamp_to_path(path): + timestamp =3D time.strftime("%Y%m%d%H%M%S") + path_unique =3D "{}.{}".format(path, timestamp) + if os.path.exists(path_unique): + import itertools + for i in itertools.count(start=3D1): + path_unique =3D "{}.{}.{}".format(path, timestamp, i) + if not os.path.exists(path_unique): + break + return path_unique + +def _get_remotes(r_remote): + remotes =3D [] + + if 'remotes' not in r_remote and 'uri' not in r_remote: + raise Exception("Expected key(s): 'remotes', 'uri'") + + if 'remotes' in r_remote: + for remote in r_remote['remotes']: + remotes.append(r_remote['remotes'][remote]['uri']) + + if 'uri' in r_remote: + remotes.append(r_remote['uri']) + + return remotes + +def checkout_layers(layers, confdir, layerdir, d, rebase_conflicts_strateg= y=3D'abort'): + def _checkout_git_remote(r_remote, repodir, layers_fixed_revisions): + rev =3D r_remote['rev'] + branch =3D r_remote.get('branch', None) + + remotes =3D _get_remotes(r_remote) + + for remote in remotes: + prot,host,path,user,pswd,params =3D bb.fetch.decodeurl(remote) + fetchuri =3D bb.fetch.encodeurl(('git',host,path,user,pswd,par= ams)) + logger.plain(" {}".format(r_name)) + if branch: + src_uri =3D f"{fetchuri};protocol=3D{prot};rev=3D{rev};bra= nch=3D{branch};destsuffix=3D{repodir}" + else: + src_uri =3D f"{fetchuri};protocol=3D{prot};rev=3D{rev};nob= ranch=3D1;destsuffix=3D{repodir}" + fetcher =3D bb.fetch.Fetch([src_uri], d) + repodir_path =3D os.path.join(layerdir, repodir) + try: + do_fetch(fetcher, layerdir) + except (bb.fetch.LocalModificationsError, bb.fetch.RebaseError= ) as e: + if rebase_conflicts_strategy !=3D 'backup': + e.msg +=3D ("\nUse 'bitbake-setup update --rebase-conf= licts-strategy=3Dbackup'" + " to automatically back up the directory and= re-clone from upstream," + " or use 'bitbake-setup init -L %s /path/to/= local/checkout'" + " to work with a local checkout instead." % = r_name) + raise + backup_path =3D add_unique_timestamp_to_path(repodir_path = + '-backup') + logger.warning( + "%s\n" + "Renaming %s to %s to preserve your work, then re-clon= ing from upstream.", + e, repodir_path, backup_path) + os.rename(repodir_path, backup_path) + fetcher.unpack(layerdir) + urldata =3D fetcher.ud[src_uri] + revision =3D urldata.revision + layers_fixed_revisions[r_name]['git-remote']['rev'] =3D revisi= on + + def _symlink_local(src, dst): + logger.plain("Making a symbolic link {} pointing to {}".format(dst= , src)) + os.symlink(src, dst) + + layers_fixed_revisions =3D copy.deepcopy(layers) + repodirs =3D [] + oesetupbuild =3D None + logger.plain("Fetching layer/tool repositories into {}".format(layerdi= r)) + for r_name in layers: + r_data =3D layers[r_name] + repodir =3D r_data.get("path", r_name) + repodirs.append(repodir) + + r_remote =3D r_data.get('git-remote') + r_local =3D r_data.get('local') + if r_remote and r_local: + raise Exception("Source {} contains both git-remote and local = properties.".format(r_name)) + + repodir_path =3D os.path.join(layerdir, repodir) + if os.path.lexists(repodir_path): + if os.path.islink(repodir_path): + os.remove(repodir_path) + elif r_local: + backup_path =3D add_unique_timestamp_to_path(repodir_path = + '-backup') + logger.warning("""Source {} in {} contains local modificat= ions. Renaming to {} to preserve them. +For local development work it is recommended to clone the needed layers se= parately and re-initialize using -L option: +bitbake-setup init -L {} /path/to/repo/checkout""".format( + r_name, repodir_path, backup_path, r_name)) + os.rename(repodir_path, backup_path) + + if r_remote: + _checkout_git_remote(r_remote, repodir, layers_fixed_revisions= ) + if r_local: + _symlink_local(os.path.expanduser(r_local["path"]), repodir_pa= th) + + if os.path.exists(os.path.join(layerdir, repodir, 'scripts/oe-setu= p-build')): + oesetupbuild =3D os.path.join(layerdir, repodir, 'scripts/oe-s= etup-build') + oeinitbuildenvdir =3D os.path.join(layerdir, repodir) + + logger.plain(" ") + _write_layer_list(layerdir, repodirs) + + if oesetupbuild: + links =3D {'setup-build': oesetupbuild, 'oe-scripts': os.path.dirn= ame(oesetupbuild), 'oe-init-build-env-dir': oeinitbuildenvdir} + for link,item in links.items(): + symlink =3D os.path.join(layerdir, link) + if os.path.lexists(symlink): + os.remove(symlink) + os.symlink(os.path.relpath(item,layerdir),symlink) + + return layers_fixed_revisions + +def setup_bitbake_build(bitbake_config, layerdir, setupdir, thisdir, updat= e_bb_conf, init_vscode=3DFalse): + def _setup_build_conf(layers, filerelative_layers, build_conf_dir): + os.makedirs(build_conf_dir) + layers_s =3D [] + + for layer in layers: + layer =3D os.path.join(layerdir, layer) + layers_s.append(" {} \\".format(layer)) + + for layer in filerelative_layers: + if thisdir: + layer =3D os.path.join(thisdir, layer) + else: + raise Exception("Configuration is using bb-layers-file-rel= ative to specify " \ + "a layer path relative to itself. This can be done only " = \ + "when the configuration is specified by its path on local = " \ + "disk, not when it's in a registry or is fetched over http= .") + layers_s.append(" {} \\".format(layer)) + + layers_s =3D "\n".join(layers_s) + bblayers_conf =3D """BBLAYERS ?=3D " \\ +{} + " +""".format(layers_s) + with open(os.path.join(build_conf_dir, "bblayers.conf"), 'w') as f= : + f.write(bblayers_conf) + + local_conf =3D """# +# This file is intended for local configuration tweaks. +# +# If you would like to publish and share changes made to this file, +# it is recommended to put them into a distro config, or to create +# layer fragments from changes made here. +# +""" + with open(os.path.join(build_conf_dir, "local.conf"), 'w') as f: + f.write(local_conf) + + with open(os.path.join(build_conf_dir, "templateconf.cfg"), 'w') a= s f: + f.write("") + + with open(os.path.join(build_conf_dir, "conf-summary.txt"), 'w') a= s f: + f.write(bitbake_config["description"] + "\n") + + with open(os.path.join(build_conf_dir, "conf-notes.txt"), 'w') as = f: + notes =3D bitbake_config.get("notes") + if isinstance(notes, list): + notes =3D "\n".join(notes) + f.write(notes + "\n" if notes else "") + + def _make_init_build_env(builddir, oeinitbuildenvdir): + builddir =3D os.path.realpath(builddir) + cmd =3D 'cd {}\nset {}\n. ./oe-init-build-env\nset --\n'.format(oe= initbuildenvdir, builddir) + initbuild_in_builddir =3D os.path.join(builddir, 'init-build-env') + + with open(initbuild_in_builddir, 'w') as f: + f.write("# init-build-env wrapper created by bitbake-setup\n") + f.write(cmd + '\n') + + def _prepend_passthrough_to_init_build_env(builddir): + env =3D bitbake_config.get("bb-env-passthrough-additions") + if not env: + return + + initbuild_in_builddir =3D os.path.join(builddir, 'init-build-env') + with open(initbuild_in_builddir) as f: + content =3D f.read() + + joined =3D " \\\n".join(env) + env =3D "export BB_ENV_PASSTHROUGH_ADDITIONS=3D\" \\\n" + env +=3D "${BB_ENV_PASSTHROUGH_ADDITIONS} \\\n" + env +=3D joined + env +=3D '"' + + with open(initbuild_in_builddir, 'w') as f: + f.write("# environment passthrough added by bitbake-setup\n") + f.write(env + '\n') + f.write('\n') + f.write(content) + + bitbake_builddir =3D os.path.join(setupdir, "build") + logger.plain("Setting up bitbake configuration in\n {}\n".format(bi= tbake_builddir)) + + template =3D bitbake_config.get("oe-template") + layers =3D bitbake_config.get("bb-layers") + if not template and not layers: + logger.error("Bitbake configuration does not contain a reference t= o an OpenEmbedded build template via 'oe-template' or a list of layers via = 'bb-layers'; please use oe-setup-build, oe-init-build-env or another mechan= ism manually to complete the setup.") + return + oesetupbuild =3D os.path.join(layerdir, 'setup-build') + if template and not os.path.exists(oesetupbuild): + raise Exception("Cannot complete setting up a bitbake build direct= ory from OpenEmbedded template '{}' as oe-setup-build was not found in any = layers; please use oe-init-build-env manually.".format(template)) + + bitbake_confdir =3D os.path.join(bitbake_builddir, 'conf') + backup_bitbake_confdir =3D add_unique_timestamp_to_path(os.path.join(b= itbake_builddir, 'conf-backup')) + upstream_bitbake_confdir =3D add_unique_timestamp_to_path(os.path.join= (bitbake_builddir, 'conf-upstream')) + + if os.path.exists(bitbake_confdir): + os.rename(bitbake_confdir, backup_bitbake_confdir) + + if layers: + filerelative_layers =3D bitbake_config.get("bb-layers-file-relativ= e") or [] + _setup_build_conf(layers, filerelative_layers, bitbake_confdir) + + if template: + bb.process.run([oesetupbuild, "setup", "-c", template, "-b", bitba= ke_builddir, "--no-shell"]) + else: + oeinitbuildenvdir =3D os.path.join(layerdir, 'oe-init-build-env-di= r') + if not os.path.exists(os.path.join(oeinitbuildenvdir, "oe-init-bui= ld-env")): + logger.error("Could not find oe-init-build-env in any of the l= ayers; please use another mechanism to initialize the bitbake environment") + return + _make_init_build_env(bitbake_builddir, os.path.realpath(oeinitbuil= denvdir)) + + _prepend_passthrough_to_init_build_env(bitbake_builddir) + + siteconf_symlink =3D os.path.join(bitbake_confdir, "site.conf") + siteconf =3D os.path.normpath(os.path.join(setupdir, '..', "site.conf"= )) + if os.path.lexists(siteconf_symlink): + os.remove(siteconf_symlink) + os.symlink(os.path.relpath(siteconf, bitbake_confdir), siteconf_symlin= k) + + + init_script =3D os.path.join(bitbake_builddir, "init-build-env") + workspace_file =3D os.path.join(setupdir, "bitbake.code-workspace") + shell =3D "bash" + # oe-fragments entries can be strings or dicts with 'name'/'descriptio= n' keys + oe_fragments =3D [f["name"] if isinstance(f, dict) else f for f in bit= bake_config.get("oe-fragments", [])] + fragments =3D oe_fragments + sorted(bitbake_config.get("oe-fragment-ch= oices",{}).values()) + if fragments: + bb.process.run("{} -c '. {} && bitbake-config-build enable-fragmen= t {}'".format(shell, init_script, " ".join(fragments))) + + if os.path.exists(backup_bitbake_confdir): + conf_diff =3D get_diff(backup_bitbake_confdir, bitbake_confdir) + if not conf_diff: + logger.plain('New bitbake configuration from upstream is the s= ame as the current one, no need to update it.') + shutil.rmtree(bitbake_confdir) + os.rename(backup_bitbake_confdir, bitbake_confdir) + if init_vscode or os.path.exists(workspace_file): + configure_vscode(setupdir, layerdir, bitbake_builddir, ini= t_script) + return + + logger.plain('Upstream bitbake configuration changes were found:') + logger.plain(conf_diff) + + if update_bb_conf =3D=3D 'prompt': + y_or_n =3D input('Apply these changes to the current configura= tion? (y/N): ') + if y_or_n !=3D 'y': + update_bb_conf =3D 'no' + + if update_bb_conf =3D=3D 'no': + logger.plain('Ignoring upstream bitbake configuration changes'= ) + logger.plain(f'Leaving the upstream configuration in {upstream= _bitbake_confdir}') + os.rename(bitbake_confdir, upstream_bitbake_confdir) + os.rename(backup_bitbake_confdir, bitbake_confdir) + if init_vscode or os.path.exists(workspace_file): + configure_vscode(setupdir, layerdir, bitbake_builddir, ini= t_script) + return + + logger.plain('Applying upstream bitbake configuration changes') + logger.plain(f'Leaving the previous configuration in {backup_bitba= ke_confdir}') + + fragment_note =3D "Run 'bitbake-config-build enable-fragment ' to enable additional fragments or replace built-in ones (e.g. machi= ne/ or distro/ to change MACHINE or DISTRO)." + + readme_extra =3D "" + if init_vscode: + readme_extra =3D "\nTo edit the code in VSCode, open the workspace= : code {}\n".format(workspace_file) + + readme =3D """{}\n\nAdditional information is in {} and {}\n +Source the environment using '. {}' to run builds from the command line.\n +{}\n +The bitbake configuration files (local.conf, bblayers.conf and more) can b= e found in {}/conf +{}""".format( + bitbake_config["description"], + os.path.join(bitbake_builddir,'conf/conf-summary.txt'), + os.path.join(bitbake_builddir,'conf/conf-notes.txt'), + init_script, + fragment_note, + bitbake_builddir, + readme_extra + ) + readme_file =3D os.path.join(bitbake_builddir, "README") + with open(readme_file, 'w') as f: + f.write(readme) + + logger.plain("This bitbake configuration provides:\n {}\n".format(b= itbake_config["description"])) + logger.plain("Usage instructions and additional information are in\n = {}\n".format(readme_file)) + logger.plain("To run builds, source the environment using\n . {}\n"= .format(init_script)) + logger.plain("{}\n".format(fragment_note)) + logger.plain("The bitbake configuration files (local.conf, bblayers.co= nf and more) can be found in\n {}/conf\n".format(bitbake_builddir)) + if init_vscode: + logger.plain("To edit the code in VSCode, open the workspace:\n = code {}\n".format(workspace_file)) + + if init_vscode or os.path.exists(workspace_file): + configure_vscode(setupdir, layerdir, bitbake_builddir, init_script= ) + +def get_registry_config(registry_path, id): + for root, _, files in os.walk(registry_path): + for file in files: + if file.endswith('.conf.json') and id =3D=3D get_config_name(f= ile): + return os.path.join(root, file) + raise Exception("Unable to find {} in available configurations; use 'l= ist' sub-command to see what is available".format(id)) + +def merge_overrides_into_sources(sources, overrides): + layers =3D copy.deepcopy(sources) + for k,v in overrides.items(): + if k in layers: + layers[k] =3D v + return layers + +def update_build(config, confdir, setupdir, layerdir, d, update_bb_conf=3D= "prompt", init_vscode=3DFalse, rebase_conflicts_strategy=3D'abort'): + layer_config =3D merge_overrides_into_sources(config["data"]["sources"= ], config["source-overrides"]["sources"]) + sources_fixed_revisions =3D checkout_layers(layer_config, confdir, lay= erdir, d, rebase_conflicts_strategy=3Drebase_conflicts_strategy) + bitbake_config =3D config["bitbake-config"] + thisdir =3D os.path.dirname(config["path"]) if config["type"] =3D=3D '= local' else None + setup_bitbake_build(bitbake_config, layerdir, setupdir, thisdir, updat= e_bb_conf, init_vscode) + write_sources_fixed_revisions(confdir, layerdir, sources_fixed_revisio= ns) + commit_config(confdir) + +def int_input(allowed_values, prompt=3D''): + n =3D None + while n is None: + try: + n =3D int(input(prompt)) + except ValueError: + prompt =3D 'Not a valid number, please try again: ' + continue + if n not in allowed_values: + prompt =3D 'Number {} not one of {}, please try again: '.forma= t(n, allowed_values) + n =3D None + return n + +def flatten_bitbake_configs(configs): + def merge_configs(c1,c2): + c_merged =3D {} + for k,v in c2.items(): + if k not in c1.keys(): + c_merged[k] =3D v + for k,v in c1.items(): + if k not in c2.keys(): + c_merged[k] =3D v + else: + c_merged[k] =3D c1[k] + c2[k] + del c_merged['configurations'] + return c_merged + + flattened_configs =3D [] + for c in configs: + if 'configurations' not in c: + flattened_configs.append(c) + else: + for sub_c in flatten_bitbake_configs(c['configurations']): + flattened_configs.append(merge_configs(c, sub_c)) + return flattened_configs + +def choose_bitbake_config(configs, parameters, non_interactive): + flattened_configs =3D flatten_bitbake_configs(configs) + configs_dict =3D {i["name"]:i for i in flattened_configs} + + if parameters: + config_id =3D parameters[0] + if config_id not in configs_dict: + raise Exception("Bitbake configuration {} not found; replace w= ith one of {}".format(config_id, configs_dict)) + return configs_dict[config_id] + + enumerated_configs =3D list(enumerate(flattened_configs, 1)) + if len(enumerated_configs) =3D=3D 1: + only_config =3D flattened_configs[0] + logger.plain("\nSelecting the only available bitbake configuration= {}".format(only_config["name"])) + return only_config + + if non_interactive: + raise Exception("Unable to choose from bitbake configurations in n= on-interactive mode: {}".format(configs_dict)) + + logger.plain("") + print_configs("Available bitbake configurations", + [c["name"] for c in flattened_configs], + [c["description"] for c in flattened_configs]) + config_n =3D int_input([i[0] for i in enumerated_configs], + "\nPlease select one of the above bitbake configu= rations by its number: ") - 1 + return flattened_configs[config_n] + +def choose_config(configs, non_interactive): + not_expired_configs =3D [k for k in sorted(configs.keys()) if not has_= expired(configs[k].get("expires", None))] + if len(not_expired_configs) =3D=3D 1: + only_config =3D not_expired_configs[0] + logger.plain("\nSelecting the only available configuration {}\n".f= ormat(only_config)) + return only_config + + if non_interactive: + raise Exception("Unable to choose from configurations in non-inter= active mode: {}".format(not_expired_configs)) + + descs =3D [] + for c in not_expired_configs: + d =3D configs[c]["description"] + expiry_date =3D configs[c].get("expires", None) + if expiry_date: + d +=3D f" (supported until {expiry_date})" + descs.append(d) + + logger.plain("") + print_configs("Available Configuration Templates", + [c for c in not_expired_configs], + descs) + config_n =3D int_input([i[0] for i in list(enumerate(not_expired_confi= gs, 1))], + "\nPlease select one of the above configurations = by its number: ") - 1 + return not_expired_configs[config_n] + +def choose_fragments(possibilities, parameters, non_interactive, skip_sele= ction): + choices =3D {} + for k,v in possibilities.items(): + if skip_selection and k in skip_selection: + logger.info("Skipping a selection of {}, as requested on comma= nd line. The resulting bitbake configuration may require further manual adj= ustments.".format(k)) + continue + + # options can be a list of strings or a list of dicts + options =3D v["options"] + if len(options) > 0 and isinstance(v["options"][0], str): + options =3D [{"name": o, "description": ""} for o in v["option= s"]] + + choice =3D [o["name"] for o in options if o["name"] in parameters] + if len(choice) > 1: + raise Exception("Options specified on command line do not allo= w a single selection " + f"from possibilities {[o['name'] for o in opti= ons]}, please " + f"remove one or more from {parameters}") + if len(choice) =3D=3D 1: + choices[k] =3D choice[0] + continue + + if non_interactive: + raise Exception(f"Unable to choose from options in non-interac= tive mode: {[o['name'] for o in options]}") + + logger.plain("") + print_configs(v["description"], + [o['name'] for o in options], + [o['description'] for o in options]) + options_enumerated =3D list(enumerate(options, 1)) + option_n =3D int_input([i[0] for i in options_enumerated], + "\nPlease select one of the above options by = its number: ") - 1 + choices[k] =3D options_enumerated[option_n][1]["name"] + return choices + +def obtain_config(top_dir, registry, args, source_overrides, d): + if args.config: + config_id =3D args.config[0] + config_parameters =3D args.config[1:] + if os.path.exists(config_id): + config_id =3D os.path.abspath(config_id) + logger.info("Reading configuration from local file\n {}".fo= rmat(config_id)) + upstream_config =3D {'type':'local', + 'path':config_id, + 'name':get_config_name(config_id), + 'data':json.load(open(config_id)) + } + elif config_id.startswith("http://") or config_id.startswith("http= s://"): + logger.info("Reading configuration from network URI\n {}".f= ormat(config_id)) + import urllib.request + try: + with urllib.request.urlopen(config_id) as f: + json_data =3D json.load(f) + upstream_config =3D {'type':'network','uri':config_id,'nam= e':get_config_name(config_id),'data':json_data} + except json.JSONDecodeError as e: + raise Exception ("Invalid JSON from {}. Are you pointing t= o an HTML page? {}".format(config_id, e)) from e + else: + logger.info("Looking up config {} in configuration registry".f= ormat(config_id)) + registry_path =3D update_registry(registry, cache_dir(top_dir)= , d) + registry_configs =3D list_registry(registry_path, with_expired= =3DTrue) + if config_id not in registry_configs: + raise Exception("Config {} not found in configuration regi= stry, re-run 'init' without parameters to choose from available configurati= ons.".format(config_id)) + upstream_config =3D {'type':'registry','registry':registry,'na= me':config_id,'data':json.load(open(get_registry_config(registry_path,confi= g_id)))} + expiry_date =3D upstream_config['data'].get("expires", None) + if has_expired(expiry_date): + logger.warning("This configuration is no longer supported afte= r {}. Please consider changing to a supported configuration.".format(expiry= _date)) + else: + registry_path =3D update_registry(registry, cache_dir(top_dir), d) + registry_configs =3D list_registry(registry_path, with_expired=3DT= rue) + config_id =3D choose_config(registry_configs, args.non_interactive= ) + config_parameters =3D [] + upstream_config =3D {'type':'registry','registry':registry,'name':= config_id,'data':json.load(open(get_registry_config(registry_path,config_id= )))} + + upstream_config['bitbake-config'] =3D choose_bitbake_config(upstream_c= onfig['data']['bitbake-setup']['configurations'], config_parameters, args.n= on_interactive) + upstream_config['bitbake-config']['oe-fragment-choices'] =3D choose_fr= agments(upstream_config['bitbake-config'].get('oe-fragments-one-of',{}), co= nfig_parameters[1:], args.non_interactive, args.skip_selection) + upstream_config['non-interactive-cmdline-options'] =3D [config_id, ups= tream_config['bitbake-config']['name']] + sorted(upstream_config['bitbake-c= onfig']['oe-fragment-choices'].values()) + upstream_config['source-overrides'] =3D source_overrides + upstream_config['skip-selection'] =3D args.skip_selection + return upstream_config + +def obtain_overrides(args): + overrides =3D {'sources':{}} + if args.source_overrides: + overrides =3D json.load(open(args.source_overrides)) + overrides_dir =3D os.path.dirname(os.path.abspath(args.source_over= rides)) + for s,v in overrides['sources'].items(): + local =3D v.get('local') + if local: + path =3D os.path.expanduser(local['path']) + if not os.path.isabs(path): + overrides['sources'][s]['local']['path'] =3D os.path.j= oin(overrides_dir, path) + + for local_name, local_path in args.use_local_source: + overrides['sources'][local_name] =3D {'local':{'path':os.path.absp= ath(os.path.expanduser(local_path))}} + + return overrides + +def configure_vscode(setupdir, layerdir, builddir, init_script): + """ + Configure the VSCode environment by creating or updating a workspace f= ile. + + Create or update a bitbake.code-workspace file with folders for the la= yers and build/conf. + Managed folders are regenerated; user-added folders are kept. Settings= are merged, with + managed keys (bitbake.*, python extra paths) always overwritten. + """ + logger.debug("configure_vscode: setupdir=3D{}, layerdir=3D{}, builddir= =3D{}, init_script=3D{}".format( + setupdir, layerdir, builddir, init_script)) + + # Get git repository directories + git_repos =3D [] + if os.path.exists(layerdir): + for entry in os.listdir(layerdir): + entry_path =3D os.path.join(layerdir, entry) + if os.path.isdir(entry_path) and not os.path.islink(entry_path= ): + # Check if it's a git repository + if os.path.exists(os.path.join(entry_path, '.git')): + git_repos.append(entry) + logger.debug("configure_vscode: found {} git repos: {}".format(len(git= _repos), git_repos)) + + conf_path =3D os.path.relpath(os.path.join(builddir, "conf"), setupdir= ) + repo_paths =3D [os.path.relpath(os.path.join(layerdir, repo), setupdir= ) for repo in git_repos] + logger.debug("configure_vscode: conf_path=3D{}, repo_paths=3D{}".forma= t(conf_path, repo_paths)) + + # Load existing workspace + workspace_file =3D os.path.join(setupdir, "bitbake.code-workspace") + workspace =3D { + "extensions": { + "recommendations": [ + "yocto-project.yocto-bitbake" + ] + } + } + if os.path.exists(workspace_file): + logger.debug("configure_vscode: loading existing workspace file: {= }".format(workspace_file)) + try: + with open(workspace_file, 'r') as f: + workspace =3D json.load(f) + logger.debug("configure_vscode: loaded workspace with {} folde= rs, {} settings".format( + len(workspace.get("folders", [])), len(workspace.get("sett= ings", {})))) + except (json.JSONDecodeError, OSError) as e: + logger.error( + "Unable to read existing workspace file {}: {}. Skipping u= pdate.".format( + workspace_file, str(e) + ) + ) + return + else: + logger.debug("configure_vscode: creating new workspace file: {}".f= ormat(workspace_file)) + + # Update folders + existing_folders =3D workspace.get("folders", []) + new_folders =3D [{"name": "conf", "path": conf_path}] + for rp in repo_paths: + repo_name =3D os.path.basename(rp) + new_folders.append({"name": repo_name, "path": rp}) + # Keep any user-added folders that are not managed + managed_paths =3D {f["path"] for f in new_folders} + for f in existing_folders: + if f["path"] not in managed_paths: + new_folders.append(f) + logger.debug("configure_vscode: keeping user-added folder: {}"= .format(f["path"])) + workspace["folders"] =3D new_folders + logger.debug("configure_vscode: updated workspace with {} folders".for= mat(len(new_folders))) + + # Build Python extra paths for each layer - only check top level of ea= ch repo + extra_paths =3D [] + subdirs_to_check =3D ['lib', 'scripts'] + for repo in git_repos: + repo_path_abs =3D os.path.join(layerdir, repo) + for subdir in subdirs_to_check: + sub_path =3D os.path.join(repo_path_abs, subdir) + if os.path.isdir(sub_path): + extra_paths.append(sub_path) + + # Update settings + existing_settings =3D workspace.get("settings", {}) + new_settings =3D { + "bitbake.disableConfigModification": True, + "bitbake.pathToBitbakeFolder": os.path.join(layerdir, "bitbake"), + "bitbake.pathToBuildFolder": builddir, + "bitbake.pathToEnvScript": init_script, + "bitbake.workingDirectory": builddir, + "files.associations": { + "*.conf": "bitbake", + "*.inc": "bitbake" + }, + "files.exclude": { + "**/.git/**": True + }, + "search.exclude": { + "**/.git/**": True, + "**/logs/**": True + }, + "files.watcherExclude": { + "**/.git/**": True, + "**/logs/**": True + }, + "python.analysis.exclude": [ + "**/.git/**", + "**/logs/**" + ], + "python.autoComplete.extraPaths": extra_paths, + "python.analysis.extraPaths": extra_paths + } + + # Merge settings: add missing, always update bitbake paths and python = extra paths + for key, value in new_settings.items(): + if key not in existing_settings: + existing_settings[key] =3D value + elif key.startswith("bitbake.") or key in [ + "python.autoComplete.extraPaths", + "python.analysis.extraPaths", + ]: + # Always replace - these are managed/machine-generated setting= s + existing_settings[key] =3D value + elif key in [ + "files.associations", + "files.exclude", + "search.exclude", + "files.watcherExclude", + "python.analysis.exclude", + ]: + # For dicts and lists, merge new values in without removing us= er additions + if isinstance(value, dict): + if not isinstance(existing_settings[key], dict): + existing_settings[key] =3D {} + for k, v in value.items(): + if k not in existing_settings[key]: + existing_settings[key][k] =3D v + elif isinstance(value, list): + if not isinstance(existing_settings[key], list): + existing_settings[key] =3D [] + for item in value: + if item not in existing_settings[key]: + existing_settings[key].append(item) + + workspace["settings"] =3D existing_settings + logger.debug("configure_vscode: merged settings, total {} keys".format= (len(existing_settings))) + + with open(workspace_file, 'w') as f: + json.dump(workspace, f, indent=3D4) + logger.debug("configure_vscode: wrote workspace file: {}".format(works= pace_file)) + +def init_config(top_dir, settings, args): + create_siteconf(top_dir, args.non_interactive, settings) + + d =3D init_bb_cache(top_dir, settings, args) + + def handle_task_progress(event, d): + rate =3D event.rate if event.rate else '' + progress =3D event.progress if event.progress > 0 else 0 + logger.handlers[0].terminator =3D '\r' + logger.plain("{}% {} ".format(progress, rate)) + logger.handlers[0].terminator =3D '\n' + + source_overrides =3D obtain_overrides(args) + upstream_config =3D obtain_config(top_dir, settings["default"]["regist= ry"], args, source_overrides, d) + logger.info("Run 'bitbake-setup init --non-interactive {}' to select t= his configuration non-interactively.\n".format(" ".join(upstream_config['no= n-interactive-cmdline-options']))) + + if args.setup_dir_name: + setup_dir_name =3D args.setup_dir_name + else: + setup_dir_name =3D "{}-{}".format(upstream_config['name']," ".join= (upstream_config['non-interactive-cmdline-options'][1:]).replace(" ","-").r= eplace("/","_")) + if 'setup-dir-name' in upstream_config['bitbake-config']: + mapping =3D { + k: v.partition("/")[2].replace(" ", "-").replace("/", = "_") + for k, v in upstream_config['bitbake-config']['oe-frag= ment-choices'].items() + } + config_setup_dir_name =3D string.Template(upstream_config['bit= bake-config']['setup-dir-name']).substitute(mapping) + config_setup_dir =3D os.path.join(top_dir, config_setup_dir_na= me) + if os.path.exists(config_setup_dir): + logger.info("Setup directory {} (as suggested by configura= tion) already exists, using the full name instead.\n".format(config_setup_d= ir)) + elif settings['default']['use-full-setup-dir-name'] !=3D 'no': + logger.info("Using the full setup directory name instead o= f {} suggested by configuration, as set in the settings.\n".format(config_s= etup_dir)) + else: + setup_dir_name =3D config_setup_dir_name + + if not args.non_interactive: + n =3D input(f"Enter setup directory name [{setup_dir_name}]: "= ) + if n: + setup_dir_name =3D n + + setupdir =3D os.path.join(os.path.abspath(top_dir), setup_dir_name) + if os.path.exists(os.path.join(setupdir, "layers")): + logger.info(f"Setup already initialized in:\n {setupdir}\nUse '= bitbake-setup status' to check if it needs to be updated, or 'bitbake-setup= update' to perform the update.\nIf you would like to start over and re-ini= tialize in this directory, remove it, and run 'bitbake-setup init' again.") + return + + logger.plain("Initializing a setup directory in\n {}".format(setupd= ir)) + if not args.non_interactive: + y_or_n =3D input('Continue? (y/N): ') + if y_or_n !=3D 'y': + exit() + logger.plain("") + + os.makedirs(setupdir, exist_ok=3DTrue) + + confdir =3D os.path.join(setupdir, "config") + layerdir =3D os.path.join(setupdir, "layers") + + os.makedirs(confdir) + os.makedirs(layerdir) + + bb.process.run(["git", "-C", confdir, "init", "-b", "main"]) + # Make sure commiting doesn't fail if no default git user is configure= d on the machine + bb.process.run(["git", "-C", confdir, "config", "user.name", "bitbake-= setup"]) + bb.process.run(["git", "-C", confdir, "config", "user.email", "bitbake= -setup@not.set"]) + bb.process.run(["git", "-C", confdir, "commit", "--no-verify", "--allo= w-empty", "-m", "Initial commit"]) + + bb.event.register("bb.build.TaskProgress", handle_task_progress, data= =3Dd) + + write_upstream_config(confdir, upstream_config) + update_build(upstream_config, confdir, setupdir, layerdir, d, update_b= b_conf=3D"yes", init_vscode=3Dargs.init_vscode) + + bb.event.remove("bb.build.TaskProgress", None) + +def get_diff(file1, file2): + try: + bb.process.run(["diff", get_diff_color_param(), "-uNr", file1, fil= e2]) + except bb.process.ExecutionError as e: + if e.exitcode =3D=3D 1: + return e.stdout + else: + raise e + return None + +def are_layers_changed(layers, layerdir, d): + def _is_git_remote_changed(r_remote, repodir): + from bb.fetch.git import git_hash_re + + rev =3D r_remote['rev'] + branch =3D r_remote.get('branch', None) + + rev_parse_result =3D bb.process.run(['git', '-C', os.path.join(lay= erdir, repodir), "rev-parse", "HEAD"]) + local_revision =3D rev_parse_result[0].strip() + if git_hash_re.match(rev): + if rev !=3D local_revision: + logger.info('Layer repository checked out into {} is at re= vision {} but should be at {}'.format(os.path.join(layerdir, repodir),local= _revision, rev)) + return True + return False + + remotes =3D _get_remotes(r_remote) + changed =3D False + for remote in remotes: + type,host,path,user,pswd,params =3D bb.fetch.decodeurl(remote) + fetchuri =3D bb.fetch.encodeurl(('git',host,path,user,pswd,par= ams)) + if branch: + fetcher =3D bb.fetch.FetchData("{};protocol=3D{};rev=3D{};= branch=3D{};destsuffix=3D{}".format(fetchuri,type,rev,branch,repodir), d) + else: + fetcher =3D bb.fetch.FetchData("{};protocol=3D{};rev=3D{};= nobranch=3D1;destsuffix=3D{}".format(fetchuri,type,rev,repodir), d) + upstream_revision =3D fetcher.method.latest_revision(fetcher, = d, 'default') + if upstream_revision !=3D local_revision: + changed =3D True + logger.info('Layer repository {} checked out into {} updat= ed revision {} from {} to {}'.format(remote, os.path.join(layerdir, repodir= ), rev, local_revision, upstream_revision)) + return changed + + changed =3D False + for r_name in layers: + r_data =3D layers[r_name] + repodir =3D r_data.get("path", r_name) + + git_remote =3D r_data.get('git-remote') + if git_remote: + changed =3D changed | _is_git_remote_changed(git_remote, repod= ir) + + return changed + +def build_status(top_dir, settings, args, d, update=3DFalse): + setupdir =3D os.path.abspath(args.setup_dir) + + confdir =3D os.path.join(setupdir, "config") + layerdir =3D os.path.join(setupdir, "layers") + + current_upstream_config =3D json.load(open(os.path.join(confdir, "conf= ig-upstream.json"))) + + args.config =3D current_upstream_config['non-interactive-cmdline-optio= ns'] + args.non_interactive =3D True + args.skip_selection =3D current_upstream_config['skip-selection'] + source_overrides =3D current_upstream_config["source-overrides"] + registry =3D current_upstream_config.get("registry") + new_upstream_config =3D obtain_config(top_dir, registry, args, source_= overrides, d) + + write_upstream_config(confdir, new_upstream_config) + config_diff =3D bb.process.run(["git", "-C" , confdir, "diff", get_dif= f_color_param()])[0] + + if config_diff: + logger.plain('\nConfiguration in {} has changed:\n{}'.format(setup= dir, config_diff)) + if update: + update_build(new_upstream_config, confdir, setupdir, layerdir,= d, + update_bb_conf=3Dargs.update_bb_conf, rebase_conf= licts_strategy=3Dargs.rebase_conflicts_strategy) + else: + bb.process.run(["git", "-C", confdir, "restore", "config-upstr= eam.json"]) + return + + layer_config =3D merge_overrides_into_sources(current_upstream_config[= "data"]["sources"], current_upstream_config["source-overrides"]["sources"]) + if are_layers_changed(layer_config, layerdir, d): + if update: + update_build(current_upstream_config, confdir, setupdir, layer= dir, + d, update_bb_conf=3Dargs.update_bb_conf, rebase_c= onflicts_strategy=3Dargs.rebase_conflicts_strategy) + return + + logger.plain("\nConfiguration in {} has not changed.".format(setupdir)= ) + +def build_update(top_dir, settings, args, d): + build_status(top_dir, settings, args, d, update=3DTrue) + +def do_fetch(fetcher, dir): + # git fetcher simply dumps git output to stdout; in bitbake context th= at is redirected to temp/log.do_fetch + # and we need to set up smth similar here + fetchlogdir =3D os.path.join(dir, 'logs') + os.makedirs(fetchlogdir, exist_ok=3DTrue) + fetchlog =3D os.path.join(fetchlogdir, 'fetch_log.{}'.format(datetime.= datetime.now().strftime("%Y%m%d%H%M%S"))) + with open(fetchlog, 'a') as f: + oldstdout =3D sys.stdout + sys.stdout =3D f + try: + fetcher.download() + fetcher.unpack_update(dir) + except: + if os.path.exists(fetchlog) and os.path.getsize(fetchlog) > 0: + logger.error("Check {} for additional information that may= help diagnoze the issue.".format(fetchlog)) + raise + finally: + sys.stdout =3D oldstdout + +def update_registry(registry, cachedir, d): + registrydir =3D 'configurations' + if registry.startswith("."): + full_registrydir =3D os.path.join(os.getcwd(), registry, registryd= ir) + elif registry.startswith("/"): + full_registrydir =3D os.path.join(registry, registrydir) + else: + full_registrydir =3D os.path.join(cachedir, registrydir) + logger.info("Fetching configuration registry\n {}\ninto\n {}= ".format(registry, full_registrydir)) + fetcher =3D bb.fetch.Fetch(["{};destsuffix=3D{}".format(registry, = registrydir)], d) + do_fetch(fetcher, cachedir) + return full_registrydir + +def has_expired(expiry_date): + if expiry_date: + return datetime.datetime.now() > datetime.datetime.fromisoformat(e= xpiry_date) + return False + +def list_registry(registry_path, with_expired): + json_data =3D {} + + for root, _, files in os.walk(registry_path): + for file in files: + if file.endswith('.conf.json'): + config_name =3D get_config_name(file) + config_data =3D json.load(open(os.path.join(root, file))) + config_desc =3D config_data["description"] + expiry_date =3D config_data.get("expires", None) + if expiry_date: + if with_expired or not has_expired(expiry_date): + json_data[config_name] =3D {"description": config_= desc, "expires": expiry_date} + else: + json_data[config_name] =3D {"description": config_desc= } + return json_data + +def list_configs(settings, args): + import tempfile + top_dir =3D tempfile.mkdtemp(prefix=3D"bitbake-setup-list-") + settings['default']['dl-dir'] =3D os.path.join(top_dir, '.bitbake-setu= p-downloads') + d =3D init_bb_cache(top_dir, settings, args) + registry_path =3D update_registry(settings["default"]["registry"], cac= he_dir(top_dir), d) + json_data =3D list_registry(registry_path, args.with_expired) + shutil.rmtree(top_dir) + + logger.plain("Available configurations:") + for config_name, config_data in sorted(json_data.items()): + expiry_date =3D config_data.get("expires", None) + config_desc =3D config_data["description"] + if expiry_date: + if args.with_expired or not has_expired(expiry_date): + logger.plain("{}\t{} (supported until {})".format(config_na= me, config_desc, expiry_date)) + else: + logger.plain("{}\t{}".format(config_name, config_desc)) + logger.plain("\nRun 'init' with one of the above configuration identif= iers to set up a build.") + + if args.write_json: + with open(args.write_json, 'w') as f: + json.dump(json_data, f, sort_keys=3DTrue, indent=3D4) + logger.plain("Available configurations written into {}".format(arg= s.write_json)) + +def install_buildtools(top_dir, settings, args, d): + buildtools_install_dir =3D os.path.join(args.setup_dir, 'buildtools') + if os.path.exists(buildtools_install_dir): + if not args.force: + logger.plain("Buildtools are already installed in {}.".format(= buildtools_install_dir)) + env_scripts =3D glob.glob(os.path.join(buildtools_install_dir,= 'environment-setup-*')) + if env_scripts: + logger.plain("If you wish to use them, you need to source = the environment setup script e.g.") + for s in env_scripts: + logger.plain("$ . {}".format(s)) + logger.plain("You can also re-run bitbake-setup install-buildt= ools with --force option to force a reinstallation.") + return + shutil.rmtree(buildtools_install_dir) + + install_buildtools_script =3D os.path.join(args.setup_dir, 'layers/oe-= scripts/install-buildtools') + + buildtools_config =3D {} + upstream_config_path =3D os.path.join(args.setup_dir, 'config', 'confi= g-upstream.json') + try: + with open(upstream_config_path) as f: + upstream_config =3D json.load(f) + buildtools_config =3D upstream_config.get('bitbake-config', {}).ge= t('install-buildtools', {}) + except FileNotFoundError: + pass + except json.JSONDecodeError as e: + logger.error("Failed to parse %s: %s" % (upstream_config_path, e)) + sys.exit(1) + + url =3D args.url if args.url is not None else buildtools_config.get('u= rl') + sha256 =3D args.sha256 if args.sha256 is not None else buildtools_conf= ig.get('sha256sum') + + buildtools_download_dir =3D add_unique_timestamp_to_path( + os.path.join(args.setup_dir, 'buildtools-downloads/buildtools')) + + if args.default or (not url and not sha256): + # No config or --default: fall back to install-buildtools defaults + logger.plain("Using install-buildtools script defaults") + subprocess.check_call([install_buildtools_script, + '-d', buildtools_install_dir, + '--downloads-directory', buildtools_downloa= d_dir]) + return + + if not url: + logger.error("No buildtools URL configured. " + "Set 'url' in the install-buildtools config section o= r pass --url.") + sys.exit(1) + + if not sha256: + logger.error("No sha256sum configured for buildtools installer. " + "Add 'sha256sum' to the install-buildtools config sec= tion or pass --sha256.\n" + "You can obtain it with: sha256sum ") + sys.exit(1) + + # Download using bb.fetch with sha256 validation + src_uri =3D "{};sha256sum=3D{}".format(url, sha256) + + logger.plain("Fetching buildtools installer using bitbake fetcher") + logger.plain(" {}".format(url)) + + fetcher =3D bb.fetch.Fetch([src_uri], d) + try: + fetcher.download() + except bb.fetch.ChecksumError as e: + logger.error("Checksum mismatch for buildtools installer. " + "Verify the URL and downloaded file.\n%s" % str(e)) + sys.exit(1) + except bb.fetch.FetchError as e: + logger.error("Failed to download buildtools installer: %s" % str(e= )) + sys.exit(1) + + # Unpack into the standard buildtools download location + os.makedirs(buildtools_download_dir, exist_ok=3DTrue) + fetcher.unpack(buildtools_download_dir) + + filename =3D os.path.basename(url.rstrip('/')) + matches =3D glob.glob(os.path.join(buildtools_download_dir, '**', file= name), recursive=3DTrue) + if not matches: + logger.error("Unpacked file not found: %s" % filename) + sys.exit(1) + local_path =3D matches[0] + + logger.plain("Buildtools installer will be installed into {}".format( + buildtools_install_dir)) + + cmd =3D [ + install_buildtools_script, + '--local-file', local_path, + '-d', buildtools_install_dir, + ] + + try: + subprocess.check_call(cmd) + except subprocess.CalledProcessError as e: + logger.error("install-buildtools failed with exit code %d" % e.ret= urncode) + sys.exit(1) + +def create_siteconf(top_dir, non_interactive, settings): + siteconfpath =3D os.path.join(top_dir, 'site.conf') + if os.path.exists(siteconfpath): + logger.info('A site.conf file already exists. Please remove it if = you would like to replace it with a default one') + else: + logger.plain(f'{top_dir} looks like a new top directory. If you wo= uld like to use a different directory, answer "n" below and either:') + logger.plain('\t1) Change the default bitbake-setup settings:') + logger.plain('\t\tbitbake-setup settings set default top-dir-prefi= x ') + logger.plain('\t\tbitbake-setup settings set default top-dir-name = ') + logger.plain('\t2) Pass one or more options on the command line to= change the top level directory in that invocation only:') + logger.plain('\t\tbitbake-setup --setting default top-dir-prefix <= PATH> ...') + logger.plain('\t\tbitbake-setup --setting default top-dir-name ...') + logger.plain('') + + logger.plain('A common site.conf file will be created, please chec= k it is correct before running builds\n {}\n'.format(siteconfpath)) + if not non_interactive: + y_or_n =3D input('Proceed? (y/N): ') + if y_or_n !=3D 'y': + exit() + + os.makedirs(top_dir, exist_ok=3DTrue) + with open(siteconfpath, 'w') as siteconffile: + sstate_settings =3D textwrap.dedent( + """ + # + # Where to place shared-state files + # + # BitBake has the capability to accelerate builds based on= previously built output. + # This is done using "shared state" files which can be tho= ught of as cache objects + # and this option determines where those files are placed. + # + # You can wipe out TMPDIR leaving this directory intact an= d the build would regenerate + # from these files if no changes were made to the configur= ation. If changes were made + # to the configuration, only shared state files where the = state was still valid would + # be used (done using checksums). + SSTATE_DIR ?=3D "{sstate_dir}" + # + # Hash Equivalence database location + # + # Hash equivalence improves reuse of sstate by detecting w= hen a given sstate + # artifact can be reused as equivalent, even if the curren= t task hash doesn't + # match the one that generated the artifact. This variable= controls where the + # Hash Equivalence database ("hashserv.db") is stored and = can be shared between + # concurrent builds. + BB_HASHSERVE_DB_DIR ?=3D "${{SSTATE_DIR}}" + """.format(sstate_dir=3Dos.path.join(top_dir, ".sstate-cac= he")) + ) + siteconffile.write( + textwrap.dedent( + """\ + # This file is intended for build host-specific bitbak= e settings + + # Where to place downloads + # + # During a first build the system will download many d= ifferent source code + # tarballs from various upstream projects. This can ta= ke a while, particularly + # if your network connection is slow. These are all st= ored in DL_DIR. When + # wiping and rebuilding you can preserve this director= y to speed up this part of + # subsequent builds. This directory is safe to share b= etween multiple builds on + # the same machine too. + DL_DIR ?=3D "{dl_dir}" + """.format( + dl_dir=3Dsettings["default"]["dl-dir"], + ) + ) + (sstate_settings if settings["default"]["common-sstate= "] =3D=3D 'yes' else "") + ) + + +def topdir_settings_path(top_dir): + return os.path.join(top_dir, 'settings.conf') + +def global_settings_path(args): + return os.path.abspath(args.global_settings) if args.global_settings e= lse os.path.join(os.path.expanduser('~'), '.config', 'bitbake-setup', 'sett= ings.conf') + +def load_settings(settings_path): + settings =3D configparser.ConfigParser() + if os.path.exists(settings_path): + logger.info('Loading settings from {}'.format(settings_path)) + settings.read_file(open(settings_path)) + + for section in settings.sections(): + for key, value in settings[section].items(): + settings[section][key] =3D value.strip('\"\'') + + return settings + +def change_setting(top_dir, args): + if vars(args)['global']: + settings_path =3D global_settings_path(args) + elif args.setting in GLOBAL_ONLY_SETTINGS: + logger.info(f"{args.setting} can only be set in the global config;= '--global' is implied") + settings_path =3D global_settings_path(args) + else: + settings_path =3D topdir_settings_path(top_dir) + settings =3D load_settings(settings_path) + + if args.subcommand =3D=3D 'set': + if args.section not in settings.keys(): + settings[args.section] =3D {} + settings[args.section][args.setting] =3D args.value + logger.plain(f"From section '{args.section}' the setting '{args.se= tting}' was changed to '{args.value}'") + if args.subcommand =3D=3D 'unset': + if args.section in settings.keys() and args.setting in settings[ar= gs.section].keys(): + del settings[args.section][args.setting] + logger.plain(f"From section '{args.section}' the setting '{arg= s.setting}' has been removed") + + os.makedirs(os.path.dirname(settings_path), exist_ok=3DTrue) + with open(settings_path, 'w') as settingsfile: + settings.write(settingsfile) + logger.info(f"Settings written to {settings_path}") + +def list_settings(all_settings): + for section, section_settings in all_settings.items(): + for key, value in section_settings.items(): + logger.plain("{} {} {}".format(section, key, value)) + +def settings_func(top_dir, all_settings, args): + if args.subcommand =3D=3D 'list': + list_settings(all_settings) + elif args.subcommand =3D=3D 'set' or args.subcommand =3D=3D 'unset': + change_setting(top_dir, args) + +def get_setup_dir_via_bbpath(): + bbpath =3D os.environ.get('BBPATH') + if bbpath: + bitbake_dir =3D os.path.normpath(bbpath.split(':')[0]) + if os.path.exists(os.path.join(bitbake_dir,'init-build-env')): + setup_dir =3D os.path.dirname(bitbake_dir) + return setup_dir + return None + +def get_top_dir(args, settings): + setup_dir_via_bbpath =3D get_setup_dir_via_bbpath() + if setup_dir_via_bbpath: + top_dir =3D os.path.dirname(setup_dir_via_bbpath) + if os.path.exists(cache_dir(top_dir)): + return top_dir + + if hasattr(args, 'setup_dir'): + top_dir =3D os.path.dirname(os.path.normpath(args.setup_dir)) + return top_dir + + top_dir_prefix =3D settings['default']['top-dir-prefix'] + top_dir_name =3D settings['default']['top-dir-name'] + return os.path.join(top_dir_prefix, top_dir_name) + +def merge_settings(builtin_settings, global_settings, topdir_settings, cmd= line_settings): + all_settings =3D builtin_settings + + for s in (global_settings, topdir_settings): + for section, section_settings in s.items(): + for setting, value in section_settings.items(): + if section not in all_settings.keys(): + all_settings[section] =3D {} + all_settings[section][setting] =3D value + + for (section, setting, value) in cmdline_settings: + if section not in all_settings.keys(): + all_settings[section] =3D {} + all_settings[section][setting] =3D value + + return all_settings + +def sigint_handler(sig, frame, func, top_dir): + logger.plain('\nShutting down...') + if isinstance(top_dir, str) and os.path.exists(top_dir): + if func in [init_config, build_update]: + logger.warning(f'{top_dir} may contain an incomplete setup!') + elif func =3D=3D install_buildtools: + logger.warning(f'{top_dir} may contain an incomplete buildtool= s installation!') + exit() + +def main(): + def add_setup_dir_arg(parser): + setup_dir =3D get_setup_dir_via_bbpath() + if setup_dir: + parser.add_argument('--setup-dir', default=3Dsetup_dir, help= =3D"Path to the setup, default is %(default)s via BBPATH") + else: + parser.add_argument('--setup-dir', required=3DTrue, help=3D"Pa= th to the setup") + + parser =3D argparse.ArgumentParser( + description=3D"BitBake setup utility. Run with 'init' argument to = get started.", + epilog=3D"Use %(prog)s --help to get help on a specif= ic command" + ) + parser.add_argument('-d', '--debug', help=3D'Enable debug output', act= ion=3D'store_true') + parser.add_argument('-q', '--quiet', help=3D'Print only errors', actio= n=3D'store_true') + parser.add_argument('--version', action=3D'version', version=3D'%(prog= )s ' + __version__) + parser.add_argument('--color', choices=3D['auto', 'always', 'never'], = default=3D'auto', help=3D'Colorize output (where %(metavar)s is %(choices)s= )', metavar=3D'COLOR') + parser.add_argument('--no-network', action=3D'store_true', help=3D'Do = not check whether configuration repositories and layer repositories have be= en updated; use only the local cache.') + parser.add_argument('--global-settings', action=3D'store', metavar=3D'= PATH', help=3D'Path to the global settings file.') + parser.add_argument('--setting', default=3D[], action=3D'append', dest= =3D'cmdline_settings', + nargs=3D3, metavar=3D('SECTION', 'SETTING', 'VALUE= '), + help=3D'Modify a setting (for this bitbake-setup i= nvocation only), for example "--setting default top-dir-prefix /path/to/top= /dir".') + + subparsers =3D parser.add_subparsers() + + parser_list =3D subparsers.add_parser('list', help=3D'List available c= onfigurations') + parser_list.add_argument('--with-expired', action=3D'store_true', help= =3D'List also configurations that are no longer supported due to reaching t= heir end-of-life dates.') + parser_list.add_argument('--write-json', action=3D'store', help=3D'Wri= te available configurations into a json file so they can be programmaticall= y processed.') + parser_list.set_defaults(func=3Dlist_configs) + + parser_init =3D subparsers.add_parser('init', help=3D'Select a configu= ration and initialize a setup from it') + parser_init.add_argument('config', nargs=3D'*', help=3D"path/URL/id to= a configuration file (use 'list' command to get available ids), followed b= y configuration options. Bitbake-setup will ask to choose from available ch= oices if command line doesn't completely specify them.") + parser_init.add_argument('--non-interactive', action=3D'store_true', h= elp=3D'Do not ask to interactively choose from available options; if bitbak= e-setup cannot make a decision it will stop with a failure.') + parser_init.add_argument('--source-overrides', action=3D'store', help= =3D'Override sources information (repositories/revisions) with values from = a local json file.') + parser_init.add_argument('--setup-dir-name', action=3D'store', help=3D= 'A custom setup directory name under the top directory.') + parser_init.add_argument('--skip-selection', action=3D'append', help= =3D'Do not select and set an option/fragment from available choices; the re= sulting bitbake configuration may be incomplete.') + parser_init.add_argument('-L', '--use-local-source', default=3D[], act= ion=3D'append', nargs=3D2, metavar=3D('SOURCE_NAME', 'PATH'), + help=3D'Symlink local source into a build, instead= of getting it as prescribed by a configuration (useful for local developme= nt).') + parser_init.add_argument('--init-vscode', action=3Dargparse.BooleanOpt= ionalAction, default=3Dbool(shutil.which('code')), + help=3D'Generate VSCode workspace configuration (d= efault: %(default)s)') + parser_init.set_defaults(func=3Dinit_config) + + parser_status =3D subparsers.add_parser('status', help=3D'Check if the= setup needs to be synchronized with configuration') + add_setup_dir_arg(parser_status) + parser_status.set_defaults(func=3Dbuild_status) + + parser_update =3D subparsers.add_parser('update', help=3D'Update a set= up to be in sync with configuration') + add_setup_dir_arg(parser_update) + parser_update.add_argument('--update-bb-conf', choices=3D['prompt', 'y= es', 'no'], default=3D'prompt', help=3D'Update bitbake configuration files = (bblayers.conf, local.conf) (default: prompt)') + parser_update.add_argument('--rebase-conflicts-strategy', choices=3D['= abort', 'backup'], default=3D'abort', + help=3D"What to do when a layer repository has loc= al modifications that prevent " + "an in-place update: 'abort' (default) aborts= with an error message; " + "'backup' renames the directory to a timestam= ped backup and re-clones from upstream.") + parser_update.set_defaults(func=3Dbuild_update) + + parser_install_buildtools =3D subparsers.add_parser('install-buildtool= s', help=3D'Install buildtools which can help fulfil missing or incorrect d= ependencies on the host machine') + add_setup_dir_arg(parser_install_buildtools) + parser_install_buildtools.add_argument('--force', action=3D'store_true= ', help=3D'Force a reinstall of buildtools over the previous installation.'= ) + parser_install_buildtools.add_argument('--default', action=3D'store_tr= ue', help=3D'Use install-buildtools script defaults, ignoring any config.') + parser_install_buildtools.add_argument('--url', help=3D'Full URL to th= e buildtools SDK installer. Overrides config value.') + parser_install_buildtools.add_argument('--sha256', help=3D'SHA256 chec= ksum of the buildtools installer. Overrides config value.') + parser_install_buildtools.set_defaults(func=3Dinstall_buildtools) + + parser_settings_arg_global =3D argparse.ArgumentParser(add_help=3DFals= e) + parser_settings_arg_global.add_argument('--global', action=3D'store_tr= ue', help=3D"Modify the setting in a global settings file, rather than one = specific to a top directory") + + parser_settings =3D subparsers.add_parser('settings', + help=3D'List current settings,= or set or unset a setting in a settings file (e.g. the default prefix and = name of the top directory, the location of configuration registry, download= s directory and other settings specific to a top directory)') + parser_settings.set_defaults(func=3Dsettings_func) + + subparser_settings =3D parser_settings.add_subparsers(dest=3D"subcomma= nd", required=3DTrue, help=3D"The action to perform on the settings file") + + subparser_settings.add_parser('list', help=3D"List all settings with t= heir values") + + parser_settings_set =3D subparser_settings.add_parser('set', parents= =3D[parser_settings_arg_global], + help=3D"In a Secti= on, set a setting to a certain value") + parser_settings_set.add_argument("section", metavar=3D"
", hel= p=3D"Section in a settings file, typically 'default'") + parser_settings_set.add_argument("setting", metavar=3D"", hel= p=3D"Name of a setting") + parser_settings_set.add_argument("value", metavar=3D"", help=3D= "The setting value") + + parser_settings_unset =3D subparser_settings.add_parser('unset', paren= ts=3D[parser_settings_arg_global], + help=3D"Unset a = setting, e.g. 'bitbake-setup settings unset default registry' would revert = to the registry setting in a global settings file") + parser_settings_unset.add_argument("section", metavar=3D"
", h= elp=3D"Section in a settings file, typically 'default'") + parser_settings_unset.add_argument("setting", metavar=3D"", h= elp=3D"The setting to remove") + + args =3D parser.parse_args() + + if args.debug: + logger.setLevel(logging.DEBUG) + elif args.quiet: + logger.setLevel(logging.ERROR) + + # Need to re-run logger_create with color argument + # (will be the same logger since it has the same name) + bb.msg.logger_create('bitbake-setup', output=3Dsys.stdout, + color=3Dargs.color, + level=3Dlogger.getEffectiveLevel()) + + if 'func' in args: + if hasattr(args, 'setup_dir'): + if not os.path.exists(os.path.join(args.setup_dir,'build', 'in= it-build-env')): + logger.error("Not a valid setup directory: build/init-buil= d-env does not exist in {}".format(args.setup_dir)) + return + + if not hasattr(args, 'non_interactive'): + args.non_interactive =3D True + + builtin_settings =3D {} + builtin_settings['default'] =3D { + 'top-dir-prefix':os.getcwd(), + 'top-dir-name':'bitbake-builds', + 'registry':get_default_registry(), + 'use-full-setup-dir-name':'no', + 'common-sstate':'yes', + } + + global_settings =3D load_settings(global_settings_path(args)) + top_dir =3D get_top_dir(args, merge_settings(builtin_settings, glo= bal_settings, {}, args.cmdline_settings)) + + # register handler now to pass top_dir + _handler =3D functools.partial(sigint_handler, + func=3Dargs.func, + top_dir=3Dos.path.abspath(top_dir)) + signal.signal(signal.SIGINT, _handler) + + # This cannot be set with the rest of the builtin settings as top_= dir needs to be determined first + builtin_settings['default']['dl-dir'] =3D os.path.join(top_dir, '.= bitbake-setup-downloads') + + topdir_settings =3D load_settings(topdir_settings_path(top_dir)) + all_settings =3D merge_settings(builtin_settings, global_settings,= topdir_settings, args.cmdline_settings) + + if args.func =3D=3D settings_func: + settings_func(top_dir, all_settings, args) + return + if args.func =3D=3D list_configs: + list_configs(all_settings, args) + return + + logger.info('Bitbake-setup is using {} as top directory.'.format(t= op_dir)) + + try: + if args.func =3D=3D init_config: + init_config(top_dir, all_settings, args) + else: + d =3D init_bb_cache(top_dir, all_settings, args) + args.func(top_dir, all_settings, args, d) + + save_bb_cache() + except (SystemExit, KeyboardInterrupt): + raise + except Exception as e: + if args.debug: + raise + logger.error(str(e)) + sys.exit(1) + else: + parser.print_help() + +if __name__ =3D=3D '__main__': + main() diff --git a/bitbake/bin/bitbake-worker b/bitbake/bin/bitbake-worker index e8073f2a..61484a5c 100755 --- a/bitbake/bin/bitbake-worker +++ b/bitbake/bin/bitbake-worker @@ -9,8 +9,9 @@ import os import sys import warnings warnings.simplefilter("default") +warnings.filterwarnings("ignore", category=3DDeprecationWarning, message= =3D".*use.of.fork.*may.lead.to.deadlocks.in.the.child.*") sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(sys.argv[0= ])), 'lib')) -from bb import fetch2 +from bb import fetch import logging import bb import select @@ -21,9 +22,21 @@ import traceback import queue import shlex import subprocess -from multiprocessing import Lock +import fcntl from threading import Thread =20 +__version__ =3D "2.20.0" +=20 +if __name__ =3D=3D "__main__": + if __version__ !=3D bb.__version__: + sys.exit("Bitbake core library version and program version mismatc= h!") + +Lock =3D bb.multiprocessing.Lock + +# Remove when we have a minimum of python 3.10 +if not hasattr(fcntl, 'F_SETPIPE_SZ'): + fcntl.F_SETPIPE_SZ =3D 1031 + bb.utils.check_system_locale() =20 # Users shouldn't be running this code directly @@ -44,7 +57,6 @@ if sys.argv[1].startswith("decafbadbad"): # updates to log files for use with tail try: if sys.stdout.name =3D=3D '': - import fcntl fl =3D fcntl.fcntl(sys.stdout.fileno(), fcntl.F_GETFL) fl |=3D os.O_SYNC=20 fcntl.fcntl(sys.stdout.fileno(), fcntl.F_SETFL, fl) @@ -56,6 +68,12 @@ logger =3D logging.getLogger("BitBake") =20 worker_pipe =3D sys.stdout.fileno() bb.utils.nonblockingfd(worker_pipe) +# Try to make the pipe buffers larger as it is much more efficient. If we = can't +# e.g. out of buffer space (/proc/sys/fs/pipe-user-pages-soft) then just p= ass over. +try: + fcntl.fcntl(worker_pipe, fcntl.F_SETPIPE_SZ, 512 * 1024) +except: + pass # Need to guard against multiprocessing being used in child processes # and multiple processes trying to write to the parent at the same time worker_pipe_lock =3D None @@ -105,7 +123,7 @@ def worker_flush(worker_queue): if not worker_queue.empty(): worker_queue_int.extend(worker_queue.get()) written =3D os.write(worker_pipe, worker_queue_int) - worker_queue_int =3D worker_queue_int[written:] + del worker_queue_int[0:written] except (IOError, OSError) as e: if e.errno !=3D errno.EAGAIN and e.errno !=3D errno.EPIPE: raise @@ -171,11 +189,8 @@ def fork_off_task(cfg, data, databuilder, workerdata, = extraconfigdata, runtask): elif workerdata["umask"]: umask =3D workerdata["umask"] if umask: - # umask might come in as a number or text string.. - try: - umask =3D int(umask, 8) - except TypeError: - pass + # Convert to a python numeric value as it could be a string + umask =3D bb.utils.to_filemode(umask) =20 dry_run =3D cfg.dry_run or runtask['dry_run'] =20 @@ -278,9 +293,12 @@ def fork_off_task(cfg, data, databuilder, workerdata, = extraconfigdata, runtask): bb.utils.disable_network(uid, gid) else: logger.debug("Skipping disable network for %s sinc= e %s is not a local uid." % (taskname, uid)) + bb.utils.landlock_restrict_network() =20 - # exported_vars() returns a generator which *cannot* be pa= ssed to os.environ.update()=20 - # successfully. We also need to unset anything from the en= vironment which shouldn't be there=20 + # exported_vars() expands each value, so it must be called= before + # empty_environment() below: an expansion that shells out = needs the + # environment bitbake was started with. We also need to un= set + # anything from the environment which shouldn't be there exports =3D bb.data.exported_vars(the_data) =20 bb.utils.empty_environment() @@ -357,7 +375,7 @@ class runQueueWorkerPipe(): def read(self): start =3D len(self.queue) try: - self.queue.extend(self.input.read(102400) or b"") + self.queue.extend(self.input.read(512*1024) or b"") except (OSError, IOError) as e: if e.errno !=3D errno.EAGAIN: raise diff --git a/bitbake/bin/git-make-shallow b/bitbake/bin/git-make-shallow deleted file mode 100755 index 9de557c1..00000000 --- a/bitbake/bin/git-make-shallow +++ /dev/null @@ -1,175 +0,0 @@ -#!/usr/bin/env python3 -# -# Copyright BitBake Contributors -# -# SPDX-License-Identifier: GPL-2.0-only -# - -"""git-make-shallow: make the current git repository shallow - -Remove the history of the specified revisions, then optionally filter the -available refs to those specified. -""" - -import argparse -import collections -import errno -import itertools -import os -import subprocess -import sys -import warnings -warnings.simplefilter("default") - -version =3D 1.0 - - -git_cmd =3D ['git', '-c', 'safe.bareRepository=3Dall'] - -def main(): - if sys.version_info < (3, 4, 0): - sys.exit('Python 3.4 or greater is required') - - git_dir =3D check_output(git_cmd + ['rev-parse', '--git-dir']).rstrip(= ) - shallow_file =3D os.path.join(git_dir, 'shallow') - if os.path.exists(shallow_file): - try: - check_output(git_cmd + ['fetch', '--unshallow']) - except subprocess.CalledProcessError: - try: - os.unlink(shallow_file) - except OSError as exc: - if exc.errno !=3D errno.ENOENT: - raise - - args =3D process_args() - revs =3D check_output(git_cmd + ['rev-list'] + args.revisions).splitli= nes() - - make_shallow(shallow_file, args.revisions, args.refs) - - ref_revs =3D check_output(git_cmd + ['rev-list'] + args.refs).splitlin= es() - remaining_history =3D set(revs) & set(ref_revs) - for rev in remaining_history: - if check_output(git_cmd + ['rev-parse', '{}^@'.format(rev)]): - sys.exit('Error: %s was not made shallow' % rev) - - filter_refs(args.refs) - - if args.shrink: - shrink_repo(git_dir) - subprocess.check_call(git_cmd + ['fsck', '--unreachable']) - - -def process_args(): - # TODO: add argument to automatically keep local-only refs, since they - # can't be easily restored with a git fetch. - parser =3D argparse.ArgumentParser(description=3D'Remove the history o= f the specified revisions, then optionally filter the available refs to tho= se specified.') - parser.add_argument('--ref', '-r', metavar=3D'REF', action=3D'append',= dest=3D'refs', help=3D'remove all but the specified refs (cumulative)') - parser.add_argument('--shrink', '-s', action=3D'store_true', help=3D's= hrink the git repository by repacking and pruning') - parser.add_argument('revisions', metavar=3D'REVISION', nargs=3D'+', he= lp=3D'a git revision/commit') - if len(sys.argv) < 2: - parser.print_help() - sys.exit(2) - - args =3D parser.parse_args() - - if args.refs: - args.refs =3D check_output(git_cmd + ['rev-parse', '--symbolic-ful= l-name'] + args.refs).splitlines() - else: - args.refs =3D get_all_refs(lambda r, t, tt: t =3D=3D 'commit' or t= t =3D=3D 'commit') - - args.refs =3D list(filter(lambda r: not r.endswith('/HEAD'), args.refs= )) - args.revisions =3D check_output(git_cmd + ['rev-parse'] + ['%s^{}' % i= for i in args.revisions]).splitlines() - return args - - -def check_output(cmd, input=3DNone): - return subprocess.check_output(cmd, universal_newlines=3DTrue, input= =3Dinput) - - -def make_shallow(shallow_file, revisions, refs): - """Remove the history of the specified revisions.""" - for rev in follow_history_intersections(revisions, refs): - print("Processing %s" % rev) - with open(shallow_file, 'a') as f: - f.write(rev + '\n') - - -def get_all_refs(ref_filter=3DNone): - """Return all the existing refs in this repository, optionally filteri= ng the refs.""" - ref_output =3D check_output(git_cmd + ['for-each-ref', '--format=3D%(r= efname)\t%(objecttype)\t%(*objecttype)']) - ref_split =3D [tuple(iter_extend(l.rsplit('\t'), 3)) for l in ref_outp= ut.splitlines()] - if ref_filter: - ref_split =3D (e for e in ref_split if ref_filter(*e)) - refs =3D [r[0] for r in ref_split] - return refs - - -def iter_extend(iterable, length, obj=3DNone): - """Ensure that iterable is the specified length by extending with obj.= """ - return itertools.islice(itertools.chain(iterable, itertools.repeat(obj= )), length) - - -def filter_refs(refs): - """Remove all but the specified refs from the git repository.""" - all_refs =3D get_all_refs() - to_remove =3D set(all_refs) - set(refs) - if to_remove: - check_output(['xargs', '-0', '-n', '1'] + git_cmd + ['update-ref',= '-d', '--no-deref'], - input=3D''.join(l + '\0' for l in to_remove)) - - -def follow_history_intersections(revisions, refs): - """Determine all the points where the history of the specified revisio= ns intersects the specified refs.""" - queue =3D collections.deque(revisions) - seen =3D set() - - for rev in iter_except(queue.popleft, IndexError): - if rev in seen: - continue - - parents =3D check_output(git_cmd + ['rev-parse', '%s^@' % rev]).sp= litlines() - - yield rev - seen.add(rev) - - if not parents: - continue - - check_refs =3D check_output(git_cmd + ['merge-base', '--independen= t'] + sorted(refs)).splitlines() - for parent in parents: - for ref in check_refs: - print("Checking %s vs %s" % (parent, ref)) - try: - merge_base =3D check_output(git_cmd + ['merge-base', p= arent, ref]).rstrip() - except subprocess.CalledProcessError: - continue - else: - queue.append(merge_base) - - -def iter_except(func, exception, start=3DNone): - """Yield a function repeatedly until it raises an exception.""" - try: - if start is not None: - yield start() - while True: - yield func() - except exception: - pass - - -def shrink_repo(git_dir): - """Shrink the newly shallow repository, removing the unreachable objec= ts.""" - subprocess.check_call(git_cmd + ['reflog', 'expire', '--expire-unreach= able=3Dnow', '--all']) - subprocess.check_call(git_cmd + ['repack', '-ad']) - try: - os.unlink(os.path.join(git_dir, 'objects', 'info', 'alternates')) - except OSError as exc: - if exc.errno !=3D errno.ENOENT: - raise - subprocess.check_call(git_cmd + ['prune', '--expire', 'now']) - - -if __name__ =3D=3D '__main__': - main() diff --git a/bitbake/bin/toaster b/bitbake/bin/toaster index f002c8c1..16a64bab 100755 --- a/bitbake/bin/toaster +++ b/bitbake/bin/toaster @@ -164,14 +164,20 @@ fi # in the local layers that currently make using an arbitrary # toasterconf.json difficult. =20 -. $OE_ROOT/.templateconf -if [ -n "$TEMPLATECONF" ]; then - if [ ! -d "$TEMPLATECONF" ]; then - # Allow TEMPLATECONF=3Dmeta-xyz/conf as a shortcut - if [ -d "$OE_ROOT/$TEMPLATECONF" ]; then - TEMPLATECONF=3D"$OE_ROOT/$TEMPLATECONF" +# BBSETUP no longer supports .templateconf +if [ -f $OE_ROOT/.templateconf ] ; then=20 + . $OE_ROOT/.templateconf + if [ -n "$TEMPLATECONF" ]; then + if [ ! -d "$TEMPLATECONF" ]; then + # Allow TEMPLATECONF=3Dmeta-xyz/conf as a shortcut + if [ -d "$OE_ROOT/$TEMPLATECONF" ]; then + TEMPLATECONF=3D"$OE_ROOT/$TEMPLATECONF" + fi fi fi +else + # Force poky + TEMPLATECONF=3D"layers/meta-yocto/meta-poky/conf/templates/default" fi =20 unset OE_ROOT @@ -230,7 +236,7 @@ for param in $*; do manage_cmd=3D"$manage_cmd $param" else echo "$HELP" - exit 1 + return 1 fi ;; =20 @@ -255,7 +261,8 @@ fi # 2) the build dir (in build) # 3) the sqlite db if that is being used. # 4) pid's we need to clean up on exit/shutdown -export TOASTER_DIR=3D$TOASTERDIR +# By default we move this all into the Toaster's parent build directory +export TOASTER_DIR=3D$BUILDDIR export BB_ENV_PASSTHROUGH_ADDITIONS=3D"$BB_ENV_PASSTHROUGH_ADDITIONS TOAST= ER_DIR" =20 # Determine the action. If specified by arguments, fine, if not, toggle it diff --git a/bitbake/bin/toaster-eventreplay b/bitbake/bin/toaster-eventrep= lay index 74a31932..742c2b78 100755 --- a/bitbake/bin/toaster-eventreplay +++ b/bitbake/bin/toaster-eventreplay @@ -31,6 +31,13 @@ sys.path.insert(0, join(dirname(dirname(abspath(__file__= ))), 'lib')) import bb.cooker from bb.ui import toasterui from bb.ui import eventreplay +from bb import __version__ as libbb_version + +__version__ =3D "2.20.0" + +if __name__ =3D=3D "__main__": + if __version__ !=3D libbb_version: + sys.exit("Bitbake core library version and program version mismatc= h!") =20 def main(argv): with open(argv[-1]) as eventfile: diff --git a/bitbake/conf/bitbake.conf b/bitbake/conf/bitbake.conf index f5a5a333..a8cd5c07 100644 --- a/bitbake/conf/bitbake.conf +++ b/bitbake/conf/bitbake.conf @@ -43,4 +43,4 @@ T =3D "${WORKDIR}/temp" TARGET_ARCH =3D "${BUILD_ARCH}" TMPDIR =3D "${TOPDIR}/tmp" WORKDIR =3D "${TMPDIR}/work/${PF}" -GITPKGV =3D "${@bb.fetch2.get_srcrev(d, 'gitpkgv_revision')}" +GITPKGV =3D "${@bb.fetch.get_srcrev(d, 'gitpkgv_revision')}" diff --git a/bitbake/contrib/b4-wrapper-bitbake.py b/bitbake/contrib/b4-wra= pper-bitbake.py new file mode 100755 index 00000000..87dff2c3 --- /dev/null +++ b/bitbake/contrib/b4-wrapper-bitbake.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python3 +# +# Copyright OpenEmbedded Contributors +# +# SPDX-License-Identifier: MIT +# +# This script is to be called by b4: +# - through b4.send-auto-cc-cmd with "send-auto-cc-cmd" as first argument, +# +# When send-auto-cc-cmd is passed: +# +# This returns the list of Cc recipients for a patch. +# +# This script takes as stdin a patch. + +import subprocess +import sys + +cmd =3D sys.argv[1] +if cmd !=3D "send-auto-cc-cmd": + sys.exit(-1) + +patch =3D sys.stdin.read() + +if subprocess.call(["which", "lsdiff"], stdout=3Dsubprocess.DEVNULL) !=3D = 0: + print("lsdiff missing from host, please install patchutils") + sys.exit(-1) + +files =3D subprocess.check_output(["lsdiff", "--strip-match=3D1", "--strip= =3D1", "--include=3Ddoc/*"], + input=3Dpatch, text=3DTrue) +if len(files): + print("docs@lists.yoctoproject.org") +else: +# Handle patches made with --no-prefix + files =3D subprocess.check_output(["lsdiff", "--include=3Ddoc/*"], + input=3Dpatch, text=3DTrue) + if len(files): + print("docs@lists.yoctoproject.org") + +sys.exit(0) diff --git a/bitbake/contrib/hashserv/Dockerfile b/bitbake/contrib/hashserv= /Dockerfile index 74b4a3be..5c8b8dd1 100644 --- a/bitbake/contrib/hashserv/Dockerfile +++ b/bitbake/contrib/hashserv/Dockerfile @@ -9,14 +9,13 @@ # docker build -f contrib/hashserv/Dockerfile . # =20 -FROM alpine:3.13.1 +FROM alpine:3.23 =20 -RUN apk add --no-cache python3 +RUN apk add --no-cache python3 py3-websockets libgcc =20 COPY bin/bitbake-hashserv /opt/bbhashserv/bin/ COPY lib/hashserv /opt/bbhashserv/lib/hashserv/ COPY lib/bb /opt/bbhashserv/lib/bb/ -COPY lib/codegen.py /opt/bbhashserv/lib/codegen.py COPY lib/ply /opt/bbhashserv/lib/ply/ COPY lib/bs4 /opt/bbhashserv/lib/bs4/ =20 diff --git a/bitbake/contrib/prserv/Dockerfile b/bitbake/contrib/prserv/Doc= kerfile index 9585fe3f..694b9b9f 100644 --- a/bitbake/contrib/prserv/Dockerfile +++ b/bitbake/contrib/prserv/Dockerfile @@ -28,14 +28,13 @@ # prserv # =20 -FROM alpine:3.14.4 +FROM alpine:3.23 =20 RUN apk add --no-cache python3 =20 COPY bin/bitbake-prserv /opt/bbprserv/bin/ COPY lib/prserv /opt/bbprserv/lib/prserv/ COPY lib/bb /opt/bbprserv/lib/bb/ -COPY lib/codegen.py /opt/bbprserv/lib/codegen.py COPY lib/ply /opt/bbprserv/lib/ply/ COPY lib/bs4 /opt/bbprserv/lib/bs4/ =20 diff --git a/bitbake/contrib/vim/LICENSE.txt b/bitbake/contrib/vim/LICENSE.= txt deleted file mode 100644 index c7d91502..00000000 --- a/bitbake/contrib/vim/LICENSE.txt +++ /dev/null @@ -1,18 +0,0 @@ -The MIT License (MIT) - -Permission is hereby granted, free of charge, to any person obtaining a co= py of -this software and associated documentation files (the "Software"), to deal= in -the Software without restriction, including without limitation the rights = to -use, copy, modify, merge, publish, distribute, sublicense, and/or sell cop= ies of -the Software, and to permit persons to whom the Software is furnished to d= o so, -subject to the following conditions: - -The above copyright notice and this permission notice shall be included in= all -copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, F= ITNESS -FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHOR= S OR -COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHE= THER -IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN -CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. diff --git a/bitbake/contrib/vim/ftdetect/bitbake.vim b/bitbake/contrib/vim= /ftdetect/bitbake.vim deleted file mode 100644 index 09fc4dc7..00000000 --- a/bitbake/contrib/vim/ftdetect/bitbake.vim +++ /dev/null @@ -1,24 +0,0 @@ -" Vim filetype detection file -" Language: BitBake -" Author: Ricardo Salveti -" Copyright: Copyright (C) 2008 Ricardo Salveti -" Licence: You may redistribute this under the same terms as Vim itse= lf -" -" This sets up the syntax highlighting for BitBake files, like .bb, .bbcla= ss and .inc - -if &compatible || version < 600 || exists("b:loaded_bitbake_plugin") - finish -endif - -" .bb, .bbappend and .bbclass -au BufNewFile,BufRead *.{bb,bbappend,bbclass} set filetype=3Dbitbake - -" .inc -au BufNewFile,BufRead *.inc set filetype=3Dbitbake - -" .conf -au BufNewFile,BufRead *.conf - \ if (match(expand("%:p:h"), "conf") > 0) | - \ set filetype=3Dbitbake | - \ endif - diff --git a/bitbake/contrib/vim/ftplugin/bitbake.vim b/bitbake/contrib/vim= /ftplugin/bitbake.vim deleted file mode 100644 index 9e8d3e13..00000000 --- a/bitbake/contrib/vim/ftplugin/bitbake.vim +++ /dev/null @@ -1,13 +0,0 @@ -" Only do this when not done yet for this buffer -if exists("b:did_ftplugin") - finish -endif - -" Don't load another plugin for this buffer -let b:did_ftplugin =3D 1 - -let b:undo_ftplugin =3D "setl cms< sts< sw< et< sua<" - -setlocal commentstring=3D#\ %s -setlocal softtabstop=3D4 shiftwidth=3D4 expandtab -setlocal suffixesadd+=3D.bb,.bbclass diff --git a/bitbake/contrib/vim/indent/bitbake.vim b/bitbake/contrib/vim/i= ndent/bitbake.vim deleted file mode 100644 index 7ee9d699..00000000 --- a/bitbake/contrib/vim/indent/bitbake.vim +++ /dev/null @@ -1,343 +0,0 @@ -" Vim indent file -" Language: BitBake -" Copyright: Copyright (C) 2019 Agilent Technologies, Inc. -" Maintainer: Chris Laplante -" License: You may redistribute this under the same terms as = Vim itself - - -if exists("b:did_indent") - finish -endif - -if exists("*BitbakeIndent") - finish -endif - -runtime! indent/sh.vim -unlet b:did_indent - -setlocal indentexpr=3DBitbakeIndent(v:lnum) -setlocal autoindent nolisp - -function s:is_bb_python_func_def(lnum) - let stack =3D synstack(a:lnum, 1) - if len(stack) =3D=3D 0 - return 0 - endif - - let top =3D synIDattr(stack[0], "name") - echo top - - return synIDattr(stack[0], "name") =3D=3D "bbPyFuncDef" -endfunction - -"""" begin modified from indent/python.vim, upstream commit 7a9bd7c1e0ce1b= af5a02daf36eeae3638aa315c7 -"""" This copied code is licensed the same as Vim itself. -setlocal indentkeys+=3D<:>,=3Delif,=3Dexcept - -let s:keepcpo=3D &cpo -set cpo&vim - -let s:maxoff =3D 50 " maximum number of lines to look backwards for () - -function! GetBBPythonIndent(lnum) - - " If this line is explicitly joined: If the previous line was also joine= d, - " line it up with that one, otherwise add two 'shiftwidth' - if getline(a:lnum - 1) =3D~ '\\$' - if a:lnum > 1 && getline(a:lnum - 2) =3D~ '\\$' - return indent(a:lnum - 1) - endif - return indent(a:lnum - 1) + (exists("g:pyindent_continue") ? eval(g:py= indent_continue) : (shiftwidth() * 2)) - endif - - " If the start of the line is in a string don't change the indent. - if has('syntax_items') - \ && synIDattr(synID(a:lnum, 1, 1), "name") =3D~ "String$" - return -1 - endif - - " Search backwards for the previous non-empty line. - let plnum =3D prevnonblank(v:lnum - 1) - - if plnum =3D=3D 0 - " This is the first non-empty line, use zero indent. - return 0 - endif - - call cursor(plnum, 1) - - " Identing inside parentheses can be very slow, regardless of the search= pair() - " timeout, so let the user disable this feature if he doesn't need it - let disable_parentheses_indenting =3D get(g:, "pyindent_disable_parenthe= ses_indenting", 0) - - if disable_parentheses_indenting =3D=3D 1 - let plindent =3D indent(plnum) - let plnumstart =3D plnum - else - " searchpair() can be slow sometimes, limit the time to 150 msec or wh= at is - " put in g:pyindent_searchpair_timeout - let searchpair_stopline =3D 0 - let searchpair_timeout =3D get(g:, 'pyindent_searchpair_timeout', 150) - - " If the previous line is inside parenthesis, use the indent of the st= arting - " line. - " Trick: use the non-existing "dummy" variable to break out of the loo= p when - " going too far back. - let parlnum =3D searchpair('(\|{\|\[', '', ')\|}\|\]', 'nbW', - \ "line('.') < " . (plnum - s:maxoff) . " ? dummy :" - \ . " synIDattr(synID(line('.'), col('.'), 1), 'name')" - \ . " =3D~ '\\(Comment\\|Todo\\|String\\)$'", - \ searchpair_stopline, searchpair_timeout) - if parlnum > 0 - " We may have found the opening brace of a BitBake Python task, e.g.= 'python do_task {' - " If so, ignore it here - it will be handled later. - if s:is_bb_python_func_def(parlnum) - let parlnum =3D 0 - let plindent =3D indent(plnum) - let plnumstart =3D plnum - else - let plindent =3D indent(parlnum) - let plnumstart =3D parlnum - endif - else - let plindent =3D indent(plnum) - let plnumstart =3D plnum - endif - - " When inside parenthesis: If at the first line below the parenthesis = add - " two 'shiftwidth', otherwise same as previous line. - " i =3D (a - " + b - " + c) - call cursor(a:lnum, 1) - let p =3D searchpair('(\|{\|\[', '', ')\|}\|\]', 'bW', - \ "line('.') < " . (a:lnum - s:maxoff) . " ? dummy :" - \ . " synIDattr(synID(line('.'), col('.'), 1), 'name')" - \ . " =3D~ '\\(Comment\\|Todo\\|String\\)$'", - \ searchpair_stopline, searchpair_timeout) - if p > 0 - if s:is_bb_python_func_def(p) - " Handle first non-empty line inside a BB Python task - if p =3D=3D plnum - return shiftwidth() - endif - - " Handle the user actually trying to close a BitBake Python task - let line =3D getline(a:lnum) - if line =3D~ '^\s*}' - return -2 - endif - - " Otherwise ignore the brace - let p =3D 0 - else - if p =3D=3D plnum - " When the start is inside parenthesis, only indent one 'shift= width'. - let pp =3D searchpair('(\|{\|\[', '', ')\|}\|\]', 'bW', - \ "line('.') < " . (a:lnum - s:maxoff) . " ? dummy :" - \ . " synIDattr(synID(line('.'), col('.'), 1), 'name')" - \ . " =3D~ '\\(Comment\\|Todo\\|String\\)$'", - \ searchpair_stopline, searchpair_timeout) - if pp > 0 - return indent(plnum) + (exists("g:pyindent_nested_paren") ? = eval(g:pyindent_nested_paren) : shiftwidth()) - endif - return indent(plnum) + (exists("g:pyindent_open_paren") ? eval= (g:pyindent_open_paren) : (shiftwidth() * 2)) - endif - if plnumstart =3D=3D p - return indent(plnum) - endif - return plindent - endif - endif - - endif - - - " Get the line and remove a trailing comment. - " Use syntax highlighting attributes when possible. - let pline =3D getline(plnum) - let pline_len =3D strlen(pline) - if has('syntax_items') - " If the last character in the line is a comment, do a binary search f= or - " the start of the comment. synID() is slow, a linear search would ta= ke - " too long on a long line. - if synIDattr(synID(plnum, pline_len, 1), "name") =3D~ "\\(Comment\\|To= do\\)$" - let min =3D 1 - let max =3D pline_len - while min < max - let col =3D (min + max) / 2 - if synIDattr(synID(plnum, col, 1), "name") =3D~ "\\(Comment\\|Todo\\)$" - let max =3D col - else - let min =3D col + 1 - endif - endwhile - let pline =3D strpart(pline, 0, min - 1) - endif - else - let col =3D 0 - while col < pline_len - if pline[col] =3D=3D '#' - let pline =3D strpart(pline, 0, col) - break - endif - let col =3D col + 1 - endwhile - endif - - " If the previous line ended with a colon, indent this line - if pline =3D~ ':\s*$' - return plindent + shiftwidth() - endif - - " If the previous line was a stop-execution statement... - " TODO: utilize this logic to deindent when ending a bbPyDefRegion - if getline(plnum) =3D~ '^\s*\(break\|continue\|raise\|return\|pass\|bb\.= fatal\)\>' - " See if the user has already dedented - if indent(a:lnum) > indent(plnum) - shiftwidth() - " If not, recommend one dedent - return indent(plnum) - shiftwidth() - endif - " Otherwise, trust the user - return -1 - endif - - " If the current line begins with a keyword that lines up with "try" - if getline(a:lnum) =3D~ '^\s*\(except\|finally\)\>' - let lnum =3D a:lnum - 1 - while lnum >=3D 1 - if getline(lnum) =3D~ '^\s*\(try\|except\)\>' - let ind =3D indent(lnum) - if ind >=3D indent(a:lnum) - return -1 " indent is already less than this - endif - return ind " line up with previous try or except - endif - let lnum =3D lnum - 1 - endwhile - return -1 " no matching "try"! - endif - - " If the current line begins with a header keyword, dedent - if getline(a:lnum) =3D~ '^\s*\(elif\|else\)\>' - - " Unless the previous line was a one-liner - if getline(plnumstart) =3D~ '^\s*\(for\|if\|try\)\>' - return plindent - endif - - " Or the user has already dedented - if indent(a:lnum) <=3D plindent - shiftwidth() - return -1 - endif - - return plindent - shiftwidth() - endif - - " When after a () construct we probably want to go back to the start lin= e. - " a =3D (b - " + c) - " here - if parlnum > 0 - return plindent - endif - - return -1 - -endfunction - -let &cpo =3D s:keepcpo -unlet s:keepcpo - -""" end of stuff from indent/python.vim - - -let b:did_indent =3D 1 -setlocal indentkeys+=3D0\" - - -function! BitbakeIndent(lnum) - if !has('syntax_items') - return -1 - endif - - let stack =3D synstack(a:lnum, 1) - if len(stack) =3D=3D 0 - return -1 - endif - - let name =3D synIDattr(stack[0], "name") - - " TODO: support different styles of indentation for assignments. For n= ow, - " we only support like this: - " VAR =3D " \ - " value1 \ - " value2 \ - " " - " - " i.e. each value indented by shiftwidth(), with the final quote " com= pletely unindented. - if name =3D=3D "bbVarValue" - " Quote handling is tricky. kernel.bbclass has this line for insta= nce: - " EXTRA_OEMAKE =3D " HOSTCC=3D"${BUILD_CC} ${BUILD_CFLAGS} ${B= UILD_LDFLAGS}" " HOSTCPP=3D"${BUILD_CPP}"" - " Instead of trying to handle crazy cases like that, just assume t= hat a - " double-quote on a line by itself (following an assignment) means= the - " user is closing the assignment, and de-dent. - if getline(a:lnum) =3D~ '^\s*"$' - return 0 - endif - - let prevstack =3D synstack(a:lnum - 1, 1) - if len(prevstack) =3D=3D 0 - return -1 - endif - - let prevname =3D synIDattr(prevstack[0], "name") - - " Only indent if there was actually a continuation character on - " the previous line, to avoid misleading indentation. - let prevlinelastchar =3D synIDattr(synID(a:lnum - 1, col([a:lnum -= 1, "$"]) - 1, 1), "name") - let prev_continued =3D prevlinelastchar =3D=3D "bbContinue" - - " Did the previous line introduce an assignment? - if index(["bbVarDef", "bbVarFlagDef"], prevname) !=3D -1 - if prev_continued - return shiftwidth() - endif - endif - - if !prev_continued - return 0 - endif - - " Autoindent can take it from here - return -1 - endif - - if index(["bbPyDefRegion", "bbPyFuncRegion"], name) !=3D -1 - let ret =3D GetBBPythonIndent(a:lnum) - " Should normally always be indented by at least one shiftwidth; b= ut allow - " return of -1 (defer to autoindent) or -2 (force indent to 0) - if ret =3D=3D 0 - return shiftwidth() - elseif ret =3D=3D -2 - return 0 - endif - return ret - endif - - " TODO: GetShIndent doesn't detect tasks prepended with 'fakeroot' - " Need to submit a patch upstream to Vim to provide an extension point= . - " Unlike the Python indenter, the Sh indenter is way too large to copy= and - " modify here. - if name =3D=3D "bbShFuncRegion" - return GetShIndent() - endif - - " TODO: - " + heuristics for de-denting out of a bbPyDefRegion? e.g. when the = user - " types an obvious BB keyword like addhandler or addtask, or sta= rts - " writing a shell task. Maybe too hard to implement... - - return -1 -endfunction diff --git a/bitbake/contrib/vim/plugin/newbb.vim b/bitbake/contrib/vim/plu= gin/newbb.vim deleted file mode 100644 index 3a420273..00000000 --- a/bitbake/contrib/vim/plugin/newbb.vim +++ /dev/null @@ -1,88 +0,0 @@ -" Vim plugin file -" Purpose: Create a template for new bb files -" Author: Ricardo Salveti -" Copyright: Copyright (C) 2008 Ricardo Salveti -" -" This file is licensed under the MIT license, see COPYING.MIT in -" this source distribution for the terms. -" -" Based on the gentoo-syntax package -" -" Will try to use git to find the user name and email - -if &compatible || v:version < 600 || exists("b:loaded_bitbake_plugin") - finish -endif - -fun! GetUserName() - let l:user_name =3D system("git config --get user.name") - if v:shell_error - return "Unknown User" - else - return substitute(l:user_name, "\n", "", "") -endfun - -fun! GetUserEmail() - let l:user_email =3D system("git config --get user.email") - if v:shell_error - return "unknown@user.org" - else - return substitute(l:user_email, "\n", "", "") -endfun - -fun! BBHeader() - let l:current_year =3D strftime("%Y") - let l:user_name =3D GetUserName() - let l:user_email =3D GetUserEmail() - 0 put =3D'# Copyright (C) ' . l:current_year . - \ ' ' . l:user_name . ' <' . l:user_email . '>' - put =3D'# Released under the MIT license (see COPYING.MIT for the term= s)' - $ -endfun - -fun! NewBBTemplate() - if line2byte(line('$') + 1) !=3D -1 - return - endif - - let l:paste =3D &paste - set nopaste - =20 - " Get the header - call BBHeader() - - " New the bb template - put =3D'SUMMARY =3D \"\"' - put =3D'HOMEPAGE =3D \"\"' - put =3D'LICENSE =3D \"\"'=20 - put =3D'SECTION =3D \"\"' - put =3D'DEPENDS =3D \"\"' - put =3D'' - put =3D'SRC_URI =3D \"\"' - - " Go to the first place to edit - 0 - /^SUMMARY =3D/ - exec "normal 2f\"" - - if paste =3D=3D 1 - set paste - endif -endfun - -if !exists("g:bb_create_on_empty") - let g:bb_create_on_empty =3D 1 -endif - -" disable in case of vimdiff -if v:progname =3D~ "vimdiff" - let g:bb_create_on_empty =3D 0 -endif - -augroup NewBB - au BufNewFile,BufReadPost *.bb - \ if g:bb_create_on_empty | - \ call NewBBTemplate() | - \ endif -augroup END - diff --git a/bitbake/contrib/vim/plugin/newbbappend.vim b/bitbake/contrib/v= im/plugin/newbbappend.vim deleted file mode 100644 index 3f65f79c..00000000 --- a/bitbake/contrib/vim/plugin/newbbappend.vim +++ /dev/null @@ -1,46 +0,0 @@ -" Vim plugin file -" Purpose: Create a template for new bbappend file -" Author: Joshua Watt -" Copyright: Copyright (C) 2017 Joshua Watt -" -" This file is licensed under the MIT license, see COPYING.MIT in -" this source distribution for the terms. -" - -if &compatible || v:version < 600 || exists("b:loaded_bitbake_plugin") - finish -endif - -fun! NewBBAppendTemplate() - if line2byte(line('$') + 1) !=3D -1 - return - endif - - let l:paste =3D &paste - set nopaste - - " New bbappend template - 0 put =3D'FILESEXTRAPATHS:prepend :=3D \"${THISDIR}/${PN}:\"' - 2 - - if paste =3D=3D 1 - set paste - endif -endfun - -if !exists("g:bb_create_on_empty") - let g:bb_create_on_empty =3D 1 -endif - -" disable in case of vimdiff -if v:progname =3D~ "vimdiff" - let g:bb_create_on_empty =3D 0 -endif - -augroup NewBBAppend - au BufNewFile,BufReadPost *.bbappend - \ if g:bb_create_on_empty | - \ call NewBBAppendTemplate() | - \ endif -augroup END - diff --git a/bitbake/contrib/vim/syntax/bitbake.vim b/bitbake/contrib/vim/s= yntax/bitbake.vim deleted file mode 100644 index 8f39b8f9..00000000 --- a/bitbake/contrib/vim/syntax/bitbake.vim +++ /dev/null @@ -1,131 +0,0 @@ -" Vim syntax file -" Language: BitBake bb/bbclasses/inc -" Author: Chris Larson -" Ricardo Salveti -" Copyright: Copyright (C) 2004 Chris Larson -" Copyright (C) 2008 Ricardo Salveti -" -" This file is licensed under the MIT license, see COPYING.MIT in -" this source distribution for the terms. -" -" Syntax highlighting for bb, bbclasses and inc files. -" -" It's an entirely new type, just has specific syntax in shell and python = code - -if &compatible || v:version < 600 || exists("b:loaded_bitbake_plugin") - finish -endif -if exists("b:current_syntax") - finish -endif - -syn include @python syntax/python.vim -if exists("b:current_syntax") - unlet b:current_syntax -endif - -" BitBake syntax - -" Matching case -syn case match - -" Indicates the error when nothing is matched -syn match bbUnmatched "." - -" Comments -syn cluster bbCommentGroup contains=3DbbTodo,@Spell -syn keyword bbTodo COMBAK FIXME TODO XXX contained -syn match bbComment "#.*$" contains=3D@bbCommentGroup - -" String helpers -syn match bbQuote +['"]+ contained=20 -syn match bbDelimiter "[(){}=3D]" contained -syn match bbArrayBrackets "[\[\]]" contained - -" BitBake strings -syn match bbContinue "\\$" -syn region bbString matchgroup=3DbbQuote start=3D+"+ skip=3D+\= \$+ end=3D+"+ contained contains=3DbbTodo,bbContinue,bbVarDeref,bbVarPyValu= e,@Spell -syn region bbString matchgroup=3DbbQuote start=3D+'+ skip=3D+\= \$+ end=3D+'+ contained contains=3DbbTodo,bbContinue,bbVarDeref,bbVarPyValu= e,@Spell - -" Vars definition -syn match bbExport "^export" nextgroup=3DbbIdentifier skipwhite -syn keyword bbExportFlag export contained nextgroup=3DbbIdentifier = skipwhite -syn match bbIdentifier "[a-zA-Z0-9\-_\.\/\+]\+" display contained -syn match bbVarDeref "${[a-zA-Z0-9\-_:\.\/\+]\+}" contained -syn match bbVarEq "\(:=3D\|+=3D\|=3D+\|\.=3D\|=3D\.\|?=3D\|?= ?=3D\|=3D\)" contained nextgroup=3DbbVarValue -syn match bbVarDef "^\(export\s*\)\?\([a-zA-Z0-9\-_\.\/\+][${= }a-zA-Z0-9\-_:\.\/\+]*\)\s*\(:=3D\|+=3D\|=3D+\|\.=3D\|=3D\.\|?=3D\|??=3D\|= =3D\)\@=3D" contains=3DbbExportFlag,bbIdentifier,bbOverrideOperator,bbVarDe= ref nextgroup=3DbbVarEq -syn match bbVarValue ".*$" contained contains=3DbbString,bbVarD= eref,bbVarPyValue -syn region bbVarPyValue start=3D+${@+ skip=3D+\\$+ end=3D+}+ conta= ined contains=3D@python - -" Vars metadata flags -syn match bbVarFlagDef "^\([a-zA-Z0-9\-_\.]\+\)\(\[[a-zA-Z0-9\-_\= .+]\+\]\)\@=3D" contains=3DbbIdentifier nextgroup=3DbbVarFlagFlag -syn region bbVarFlagFlag matchgroup=3DbbArrayBrackets start=3D"\[" = end=3D"\]\s*\(:=3D\|=3D\|.=3D\|=3D.|+=3D\|=3D+\|?=3D\)\@=3D" contained cont= ains=3DbbIdentifier nextgroup=3DbbVarEq - -" Includes and requires -syn keyword bbInclude inherit include require contained=20 -syn match bbIncludeRest ".*$" contained contains=3DbbString,bbVarD= eref,bbVarPyValue -syn match bbIncludeLine "^\(inherit\|include\|require\)\s\+" conta= ins=3DbbInclude nextgroup=3DbbIncludeRest - -" Add taks and similar -syn keyword bbStatement addtask deltask addhandler after before EX= PORT_FUNCTIONS contained -syn match bbStatementRest /[^\\]*$/ skipwhite contained contains=3Db= bStatement,bbVarDeref,bbVarPyValue -syn region bbStatementRestCont start=3D/.*\\$/ end=3D/^[^\\]*$/ contained= contains=3DbbStatement,bbVarDeref,bbVarPyValue,bbContinue keepend -syn match bbStatementLine "^\(addtask\|deltask\|addhandler\|after\|b= efore\|EXPORT_FUNCTIONS\)\s\+" contains=3DbbStatement nextgroup=3DbbStateme= ntRest,bbStatementRestCont - -" OE Important Functions -syn keyword bbOEFunctions do_fetch do_unpack do_patch do_configure d= o_compile do_stage do_install do_package contained - -" Generic Functions -syn match bbFunction "\h[0-9A-Za-z_\-\.]*" display contained co= ntains=3DbbOEFunctions - -syn keyword bbOverrideOperator append prepend remove contained - -" BitBake shell metadata -syn include @shell syntax/sh.vim -if exists("b:current_syntax") - unlet b:current_syntax -endif -syn keyword bbShFakeRootFlag fakeroot contained -syn match bbShFuncDef "^\(fakeroot\s*\)\?\([\.0-9A-Za-z_:${}\-\.= ]\+\)\(python\)\@`__: + + .. code-block:: shell + + $ python3 -m venv --clear ./bitbake-setup-venv + $ . ./bitbake-setup-venv/bin/activate + $ pip install bitbake-setup + + .. tip:: + + To update ``bitbake-setup`` when already installed, you can run: + + .. code-block:: console + + $ pip install --upgrade bitbake-setup + +#. To start using ``bitbake-setup``, run: + + .. code-block:: shell + + $ bitbake-setup init + + This command will ask you to choose which configurations to use availab= le as + part of the default BitBake :term:`Configuration Templates `. + + .. note:: + + These default configurations are located in ``./bitbake/default-regi= stry/``. + See the :ref:`ref-bbsetup-section-config-reference` section to learn= more + about ``bitbake-setup`` input configuration files. + +#. Here is an example of files and directories structure created by ``bitb= ake-setup init``:: + + ./bitbake-builds/ + =E2=94=9C=E2=94=80=E2=94=80 site.conf + =E2=94=94=E2=94=80=E2=94=80 poky-master-poky-distro_poky-machine_qem= ux86-64/ + =C2=A0=C2=A0 =E2=94=9C=E2=94=80=E2=94=80 build/ + =C2=A0=C2=A0 =E2=94=9C=E2=94=80=E2=94=80 config/ + =C2=A0=C2=A0 =E2=94=94=E2=94=80=E2=94=80 layers/ + + With: + + - ``./bitbake-builds``: the :term:`Top Directory`, where ``bitbake-set= up`` + configures everything. This directory can be configured with the + :ref:`ref-bbsetup-setting-top-dir-prefix` and + :ref:`ref-bbsetup-setting-top-dir-name` settings. + + - ``poky-master-poky-distro_poky-machine_qemux86-64``: a :term:`Setup` + directory, which holds a :term:`Setup`: a result of the choices made + during the ``bitbake-setup init`` execution. + + The name of the directory will vary depending on the choices. + + - ``config/``: holds the :term:`Configuration Instance`, which embeds = the + :term:`Configuration Template` (first choice of the :ref:`ref-bbsetu= p-command-init` command) + and the choices made during the initialization. + + - ``build/``: the :term:`BitBake Build` directory, where BitBake store= s + its own configuration and outputs for the builds. + + - ``layers/``: the directory where :ref:`layers + ` and other + repositories managed by ``bitbake-setup`` are stored and updated. + + - ``site.conf``: a BitBake configuration file that contains site speci= fic + configurations for your build environment. When it is created, it co= ntains + some variable definitions that are based on your current :term:`sett= ings`. + Comments in this file will help you understand what these variables + correspond to. + + These configurations are shared between the :term:`setups `. + +#. Source the ``init-build-env`` file present in the :term:`BitBake Build` + directory: + + .. code-block:: shell + + $ source ./poky-master-poky-distro_poky-machine_qemux86-64/build/ini= t-build-env + + This command will prepare your current shell with the BitBake environme= nt. + +#. You can then start running ``bitbake`` in the current shell. For more i= nformation + on how to use ``bitbake``, see the :doc:`/bitbake-user-manual/bitbake-u= ser-manual-execution` + section of this manual. + +.. note:: + + The ``bitbake-setup`` tool is developed and maintained in the BitBake + repository and can also be used directly from there: + + .. code-block:: console + + $ git clone https://git.openembedded.org/bitbake ./bitbake + $ ./bitbake/bin/bitbake-setup ... + +Terminology +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +The ``bitbake-setup`` tool revolves around some common terms we define in = this +section. + +``bitbake-setup`` works with a specific hierarchy of files and directories= , that +can be represented as follows:: + + Top Directory + =E2=94=9C=E2=94=80=E2=94=80 site.conf + =E2=94=9C=E2=94=80=E2=94=80 Setup 1 + =E2=94=82=C2=A0=C2=A0 =E2=94=9C=E2=94=80=E2=94=80 build/ + =E2=94=82 =E2=94=9C=E2=94=80=E2=94=80 config/ + =E2=94=82=C2=A0=C2=A0 =E2=94=94=E2=94=80=E2=94=80 layers/ + =E2=94=9C=E2=94=80=E2=94=80 Setup 2 + =E2=94=82=C2=A0=C2=A0 =E2=94=9C=E2=94=80=E2=94=80 build/ + =E2=94=82 =E2=94=9C=E2=94=80=E2=94=80 config/ + =E2=94=82=C2=A0=C2=A0 =E2=94=94=E2=94=80=E2=94=80 layers/ + ... + +The "Top Directory" and "Setup" directories are defined as follows: + +.. glossary:: + :term:`Top Directory` + The top directory is the working directory of ``bitbake-setup``, whe= re its + outputs end-up (unless otherwise configured by :term:`settings` such= as + :ref:`ref-bbsetup-setting-dl-dir`). + + The location of this directory can be changed with the + :ref:`ref-bbsetup-setting-top-dir-prefix` and + :ref:`ref-bbsetup-setting-top-dir-name` settings. + + The top directory contains one or more :term:`Setup` directories, ea= ch of + them containing a :term:`Setup`, and a :term:`Site Configuration Fil= e` + (named ``site.conf``). + + :term:`Setup` + A Setup is the result of the :ref:`ref-bbsetup-command-init` + command, which creates a :term:`Setup` directory. It is constructed = from a + :term:`Configuration Template` and choices made during the ``init`` = command. + + It contains at least: + + - A :term:`BitBake Build` (``build/`` directory). + - A :term:`Configuration Instance` (``config/`` directory). + - Sources such as :ref:`layers + ` or other + repositories managed by ``bitbake-setup`` (``layers/`` directory)= . + + :term:`Site Configuration File` + This file named ``site.conf`` is a unique file located in the :term:= `Top + Directory`, and holds top-level BitBake configuration statements sha= red + between the :term:`Setups `. + + When it is created, it contains some variable definitions that are b= ased + on your current :term:`settings`. Comments in this file will help yo= u + understand what these variables correspond to. + +The following components are involved to create the content of these direc= tories: + +.. glossary:: + :term:`BitBake Build` + A BitBake Build is a sub-tree inside a :term:`Setup` that BitBake it= self + operates on. The files in the ``conf/`` directory of a :term:`BitBak= e + Build` constitute the :ref:`BitBake configuration + `= . + + :term:`Configuration Template` + A Configuration Template is a file in JSON format containing a templ= ate to + create a :term:`Setup`. These files are used during the :ref:`ref-bb= setup-command-init` + command as a starting point to configure the :term:`Setup`. When the + command runs, the user may be prompted with choices to further speci= fy the + :term:`Setup` to create. + + It is also possible to specify the choices on the command line for a + completely non-interactive initialization. + + :term:`Configuration Template` files are stored in :term:`registries + `, and can be listed with the :ref:`ref-bbsetup-command-li= st` + command. + + :term:`Configuration Template` files must end with the ``.conf.json`= ` + suffix for ``bitbake-setup`` to locate them. + + .. note:: + + The default :term:`Configuration Templates ` are + located in the BitBake repository in a local registry. the + ``default-registry/`` directory. This can be modified with the + :ref:`ref-bbsetup-setting-registry` setting. + + :ref:`ref-bbsetup-command-status` will tell if a :term:`Setup` + is in sync with the :term:`Configuration Template` it was constructe= d from + (typically: layer updates). + + :ref:`ref-bbsetup-command-update` will bring a :term:`Setup` + in sync with its :term:`Configuration Template`. + + :term:`Configuration Instance` + The :term:`Configuration Instance` is stored in the ``config/`` dire= ctory + in a :term:`Setup`. It embeds the :term:`Configuration Template` and= the + choices made during the initialization. + + It is also a Git repository, that contains a history of the configur= ation + instance and updates made to it via :ref:`ref-bbsetup-command-update= `. + + :term:`Registry` + A configuration registry is a place where one or more :term:`Configu= ration + Templates ` are stored. + + The directory structure of the registry can be any: ``bitbake-setup`= ` + recursively find files ending with ``.conf.json`` and consider it a + :term:`Configuration Template`. + + The registry location is configured through the + :ref:`ref-bbsetup-setting-registry` setting. This location can be th= e URL to + a Git repository, a local directory, or any URI supported by the Bit= Bake + fetcher (see the :doc:`/bitbake-user-manual/bitbake-user-manual-fetc= hing` + section for more information on fetchers). + + :term:`Settings` + Settings are operational parameters that are global to all builds un= der a + :term:`Top Directory`, stored in a ``settings.conf`` file. For examp= le, + this could be the location of the configuration registry, or where t= he + BitBake fetcher should store the downloads. + + There are also global settings, common to all top directories that a= re + stored in ``~/.config/bitbake-setup/settings.conf``. + + See the :ref:`bitbake-user-manual/bitbake-user-manual-environment-se= tup:Settings` + section to see the supported settings and where they can be stored. + + :term:`Source Override` + A source override is a JSON file that can be used to modify revision= s and + origins of layers or other sources that need to be checked out into = a + :term:`Setup` (in the ``layers/`` directory). It can be useful for e= xample + when the master branches need to be changed to master-next for the p= urpose + of testing, or to set up a CI pipeline that tests code in a pull req= uest + coming from a developer's repository and branch. + + Such a file is specified with a command-line option to + :ref:`ref-bbsetup-command-init`. + + See the :ref:`ref-bbsetup-source-overrides` section for more informa= tion on + the format of these files. + +The ``bitbake-setup`` command +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D + +The ``bitbake-setup`` program has general options and sub-commands. These = can be +obtained using ``bitbake-setup --help``. + +The general options, common to all commands, are: + +- ``-h`` or ``--help``: Show the help message and exit. +- ``-d`` or ``--debug``: Enable debug outputs. +- ``-q`` or ``--quiet``: Print only errors. +- ``--color``: Colorize output (where COLOR is auto, always, never). +- ``--no-network``: Do not check whether configuration repositories and l= ayer + repositories have been updated; use only the local cache. +- ``--global-settings``: Path to the global settings file. +- ``--setting``: Modify a setting (for this bitbake-setup invocation only= ). + For example ``--setting default top-dir-prefix /path/to/top-dir``. + +.. _ref-bbsetup-command-init: + +``bitbake-setup init`` +---------------------- + +The ``bitbake-setup init`` sub-command helps initializing a :term:`Setup`. + +This command can be run without any arguments to prompt the user with +configuration options to choose from. These configuration options are take= n from +the input :term:`Configuration Template` files in the :term:`registry`. + +.. note:: + + The registry location can be set with the :ref:`ref-bbsetup-setting-reg= istry` + setting and the :ref:`ref-bbsetup-command-settings` command. + +Otherwise, the first argument to :ref:`ref-bbsetup-command-init` can be: + +- A Configuration Template ID in the registry. +- A path to a Configuration Template file on a local disk. +- An HTTP URI to the Configuration Template file. + +The choices made during the bare ``bitbake-setup init`` command can also b= e +passed directly on the command-line, for example:: + + bitbake-setup init poky distro/poky-tiny ... + +``bitbake-setup`` will stop and ask to make a choice if the above command = does +not contain all of the required configurations to complete the sequence of +choices. + +In addition, the command can take the following arguments: + +- ``--non-interactive``: can be used to create :term:`Setups ` + without interactions from the user. The command will fail if not all th= e + required choices are provided in the command. + +- ``--source-overrides``: can be used to pass one or more + :ref:`source override `. See the + :ref:`ref-bbsetup-source-overrides` section. + +- ``--setup-dir-name``: can be used to configure the name of the :term:`S= etup` + directory. + +- ``--skip-selection``: can be used to skip some of the choices + (which may result in an incomplete :term:`Setup`!) + +- ``-L`` or ``--use-local-source``: instead of getting a source as prescr= ibed in + a configuration, symlink it into a :term:`Setup` from a path on local d= isk. This + is useful for local development where that particular source directory = is managed + separately, and bitbake-setup will include it in a build but will not o= therwise + touch or modify it. This option can be specified multiple times to spec= ify multiple + local sources. + + The option can be seen as a command line shortcut to providing an overr= ide file + with a ``local`` source in it. See the :ref:`ref-bbsetup-source-overrid= es` section + for more information on source overrides. + +- ``--init-vscode`` / ``--no-init-vscode``: generate (or skip generating)= a + ``bitbake.code-workspace`` file in the :term:`Setup` directory. The wor= kspace + file configures the `Yocto Project BitBake + `_ + VS Code extension with paths to the build directory and the init script= , and + lists the layer directories as workspace folders. Any user-added folder= s or + settings in an existing workspace file are preserved across updates. + The default is ``true`` when ``code`` (the VS Code binary) is found on + ``PATH``, and ``false`` otherwise. + +``bitbake-setup init`` Examples +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- This example prompts the user to choose a :term:`Configuration Template= ` from + a custom registry: + + .. code-block:: shell + + $ bitbake-setup \ + --setting default registry 'git://example.com/bitbake-setup-conf= igurations.git;protocol=3Dhttps;branch=3Dmain;rev=3Dmain' \ + init + +- This example takes a :term:`Configuration Template` from a remote locat= ion + (here, one of the default configuration in BitBake): + + .. code-block:: shell + + $ bitbake-setup init https://git.openembedded.org/bitbake/plain/defa= ult-registry/configurations/oe-nodistro-master.conf.json + +- This example initializes a :term:`Setup` with: + + - A custom :ref:`ref-bbsetup-setting-top-dir-prefix` and :ref:`ref-bbs= etup-setting-top-dir-name` + - A :term:`source override`, and additionally overriding openembedded-= core with a locally managed directory. + - A standalone :term:`Configuration Template` file. + - Choices passed on the command-line, applied non-interactively. + + .. code-block:: shell + + $ bitbake-setup \ + --setting default top-dir-prefix /work/bitbake-setup \ + --setting default top-dir-name custom-project \ + init \ + --non-interactive \ + --source-overrides develop-branch.json \ + -L openembedded-core ~/development/openembedded-core-gadget \ + ./gadget_master.conf.json \ + gadget distro/gadget machine/gadget + +.. _ref-bbsetup-command-list: + +``bitbake-setup list`` +---------------------- + +The ``bitbake-setup list`` sub-command lists the available :term:`Configur= ation +Templates ` in the current :term:`registry`. + +In addition, the command can take the following arguments: + +- ``--with-expired``: list the expired configuration (e.g. older Yocto re= leases + that have reached their End-Of-Life dates). + +- ``--write-json``: write the configurations into a JSON file so they can= be + programmatically processed. + +.. _ref-bbsetup-command-status: + +``bitbake-setup status`` +------------------------ + +The ``bitbake-setup status`` sub-command shows the status of a +:term:`Setup`. Any differences between the local copy of the :term:`Config= uration +Template` and the upstream one are printed on the console. + +If the BitBake environment is sourced and ready to build, the ``bitbake-se= tup +status`` command (without any arguments) will show the status of the curre= nt +:term:`Setup`. + +In addition, the command can take the following arguments: + +- ``--setup-dir``: path to the :term:`Setup` to check to status for. Not + required if the command is invoked from an initialized BitBake environm= ent + that contains :term:`BBPATH`. + +.. _ref-bbsetup-command-update: + +``bitbake-setup update`` +------------------------ + +The ``bitbake-setup update`` sub-command updates a :term:`Setup` to sync w= ith +the latest changes from the :term:`Configuration Template` it was construc= ted from. +The :ref:`ref-bbsetup-command-status` command can be used to show the curr= ent +status of the :term:`Setup` before updating it. + +This command is intended to be run from a shell where the BitBake environm= ent +has been sourced (e.g. after ``source build/init-build-env``), so that +``bitbake-setup`` can automatically identify the current :term:`Setup` wit= hout +requiring the ``--setup-dir`` argument. + +When a layer repository already exists in the :term:`Setup` (i.e. it has b= een +previously checked out by ``bitbake-setup init`` or a prior ``update``), t= he +update is performed *in place* using the fetcher's +:ref:`unpack_update ` method: the new upstream revis= ion +is fetched into the local download cache and then rebased on top of the +checkout's current HEAD. This means any local commits in the layer directo= ry +are preserved and rebased onto the new upstream revision. If the working t= ree +contains staged or unstaged changes to tracked files, the update is blocke= d +until those changes are committed, stashed or discarded. + +.. note:: + + ``bitbake-setup`` performs the rebase to fast-forward local commits ont= o + the new upstream revision, but intentionally does not go further by usi= ng + ``--autostash`` (which would silently stash uncommitted changes before = the + rebase and pop them afterwards). Any uncommitted modifications are surf= aced + to the user before the update proceeds, so there are no surprises from = an + automatic stash/pop cycle. + +In addition, the command can take the following arguments: + +- ``--update-bb-conf``: whether to update the :term:`BitBake Build` + configuration (``local.conf``, ``bblayers.conf``, etc.). This argument = can + take up to three values: + + - ``prompt`` (default): ask the user whether to update. + - ``yes``: update the configuration files. + - ``no``: don't update the configuration files. + +- ``--rebase-conflicts-strategy``: what to do when a layer repository has + local modifications or commits that prevent an in-place update. Accepte= d + values are: + + - ``abort`` (default): stop with an error message describing the probl= em. + The repository is left in its previous state (the failed rebase is + automatically aborted). The error message includes a hint to re-run = with + ``--rebase-conflicts-strategy=3Dbackup``. + - ``backup``: rename the conflicting layer directory to a timestamped + ``-backup-`` path (preserving local work), then + re-clone the layer from upstream into a fresh directory. + +- ``--setup-dir``: path to the :term:`Setup` to update. Not required if t= he + command is invoked from an initialized BitBake environment that contain= s + :term:`BBPATH`. + +``bitbake-setup update`` Examples +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- This example performs a standard update from an initialized BitBake + environment. Local commits in any layer directory are rebased on top of + the new upstream revision and preserved: + + .. code-block:: shell + + $ bitbake-setup update + NOTE: Bitbake-setup is using /path/to/bitbake-builds as top director= y. + NOTE: Looking up config poky-master in configuration registry + NOTE: Layer repository https://git.openembedded.org/openembedded-cor= e + checked out into /path/to/bitbake-builds/poky-master/layers/op= enembedded-core + updated revision master from d383ea3... to b50d6de... + Fetching layer/tool repositories into /path/to/bitbake-builds/poky-m= aster/layers + bitbake + meta-yocto + openembedded-core + +- This example shows what happens when a layer directory contains staged = or + unstaged changes to tracked files. The update is blocked with an error: + + .. code-block:: shell + + $ bitbake-setup update + NOTE: Bitbake-setup is using /path/to/bitbake-builds as top director= y. + NOTE: Looking up config poky-master in configuration registry + Fetching layer/tool repositories into /path/to/bitbake-builds/poky-m= aster/layers + bitbake + meta-yocto + openembedded-core + ERROR: Unpack failure for URL: + 'git://git.openembedded.org/openembedded-core;protocol=3Dhttps;rev= =3Dmaster;branch=3Dmaster;destsuffix=3Dopenembedded-core'. + Repository at /path/to/bitbake-builds/poky-master/layers/openembedd= ed-core has uncommitted changes, unable to update: + M meta/recipes-devtools/ccache/ccache_4.13.1.bb + + Commit, stash or discard your changes and re-run the update. + Use 'bitbake-setup update --rebase-conflicts-strategy=3Dbackup' + to automatically back up the directory and re-clone from upstream, + or use 'bitbake-setup init -L openembedded-core /path/to/local/check= out' + to work with a local checkout instead. + + Stashing the changes and re-running resolves the issue: + + .. code-block:: shell + + $ git -C layers/openembedded-core stash + $ bitbake-setup update + $ git -C layers/openembedded-core stash pop + +- This example shows what happens when a layer directory contains local + commits that conflict with the incoming upstream changes. The failed re= base + is automatically aborted, and the ``dldir`` remote is left in the repos= itory + for manual resolution: + + .. code-block:: shell + + $ bitbake-setup update + ERROR: Repository at layers/openembedded-core has local commits that= could + not be rebased onto the new upstream revision: + ... + Note: the 'dldir' remote points to the local download cache and may = be + used to resolve the conflict manually. + Once resolved, re-run the update. + + The conflict can be resolved manually using the ``dldir`` remote that + ``bitbake-setup`` adds to the repository: + + .. code-block:: shell + + $ git -C layers/openembedded-core rebase dldir/master + # fix conflicts in an editor, then stage the resolved files: + $ git -C layers/openembedded-core add meta/recipes-core/base-files/b= ase-files.bb + $ git -C layers/openembedded-core rebase --continue + $ bitbake-setup update + +- When manual conflict resolution is not desired, the + ``--rebase-conflicts-strategy=3Dbackup`` option can be used instead. It + preserves the conflicting directory under a timestamped backup path and + re-clones the layer cleanly from upstream: + + .. code-block:: shell + + $ bitbake-setup update --rebase-conflicts-strategy=3Dbackup + NOTE: Bitbake-setup is using /path/to/bitbake-builds as top director= y. + NOTE: Looking up config poky-master in configuration registry + NOTE: Layer repository https://git.openembedded.org/openembedded-cor= e checked + out into /path/to/bitbake-builds/poky-master/layers/openembedd= ed-core + updated revision master from 2ec283e... to b50d6de... + Fetching layer/tool repositories into /path/to/bitbake-builds/poky-m= aster/layers + bitbake + meta-yocto + openembedded-core + WARNING: Unpack failure for URL: + 'git://git.openembedded.org/openembedded-core;protocol=3Dhttps;rev= =3Dmaster;branch=3Dmaster;destsuffix=3Dopenembedded-core'. + Repository at /path/to/bitbake-builds/poky-master/layers/openembed= ded-core + has local commits that could not be rebased onto the new upstream = revision: + ... + Note: the 'dldir' remote points to the local download cache and may = be used to resolve the conflict manually. + Once resolved, re-run the update. + Renaming /path/to/bitbake-builds/poky-master/layers/openembedded-cor= e to + /path/to/bitbake-builds/poky-master/layers/openembedded-core-backu= p.20260329160426 + to preserve your work, then re-cloning from upstream. + + The backup directory is a complete git repository. Local commits can be + recovered from it after the update by fetching a branch from the backup + into the fresh clone (git accepts local paths as remote URLs) and then + cherry-picking the desired commits. For example, given a ``my-wip`` bra= nch + with two commits existing in the backup repository and not in the fresh= clone, + the following commands can be used to apply these commits on top of the= new + upstream revision in the fresh clone: + + .. code-block:: shell + + $ git -C layers/openembedded-core-backup.20260329160426 log --onelin= e my-wip + a1b2c3d u-boot: fix compilation with newer GCC + 2ec283e base-files: update version + ... + + $ cd layers/openembedded-core + $ git checkout -b my-wip + Switched to a new branch 'my-wip' + $ git fetch ../openembedded-core-backup.20260329160426 my-wip + + $ git cherry-pick 2ec283e + # resolve any conflicts, then stage the resolved files: + $ git add meta/recipes-core/base-files/base-files.bb + $ git cherry-pick --continue + + $ git cherry-pick a1b2c3d + # resolve any conflicts, then stage the resolved files: + $ git add meta/recipes-devtools/u-boot/u-boot_2026.04.bb + $ git cherry-pick --continue + + $ cd ../.. + + The sequence above is: + + #. Inspect the backup's branch history to identify the commits to recov= er. + #. Change into the fresh clone and create a matching branch. + #. Fetch the backup branch so its objects become available locally. Git + accepts filesystem paths as remote URLs, and from inside + ``layers/openembedded-core/``, ``../`` points to ``layers/``, where = the + backup directory sits. + #. Cherry-pick the commits in oldest-first order. + + Once all desired commits have been recovered and verified, the backup + directory can be removed: + + .. code-block:: shell + + $ rm -rf layers/openembedded-core-backup.20260329160426 + + If a VSCode workspace is in use, the backup directory will appear as an + additional workspace folder until it is cleaned up. It can be removed f= rom + the workspace via the VS Code UI by right-clicking the folder and selec= ting + *Remove Folder from Workspace*. + +.. _ref-bbsetup-command-install-buildtools: + +``bitbake-setup install-buildtools`` +------------------------------------ + +The ``bitbake-setup install-buildtools`` sub-command installs and extracts= a +buildtools tarball into the specified :term:`Setup`. + +After completion, help is printed to help the user on how to use the extra= cted +tarball. + +.. note:: + + The purpose of the Buildtools tarball is to provide tools needed to run + BitBake on build machines where such tools cannot be easily obtained fr= om the + host Linux distribution (for example on older distribution versions tha= t do + not contain a recent enough GCC compiler or Python interpreter, or mach= ines + where the user running BitBake cannot easily install additional softwar= e into + the system). This command requires that the OpenEmbedded-core layer is + present in the BitBake configuration. + + See https://docs.yoctoproject.org/ref-manual/system-requirements.html#r= equired-git-tar-python-make-and-gcc-versions + for more information. + +In addition, the command can take the following arguments: + +- ``--force``: force the re-installation of the tarball. + +- ``--setup-dir``: path to the :term:`Setup` to check to status for. Not + required if :term:`BBPATH` is already configured. + +- ``--url``: full URL to the buildtools SDK installer. Overrides the valu= e + from the configuration file. + +- ``--sha256``: SHA256 checksum of the buildtools installer. Overrides th= e + value from the configuration file. + +When ``url`` and ``sha256sum`` are set in the configuration file, +the installer is downloaded via ``bb.fetch`` (cached in :term:`DL_DIR`) an= d its +checksum is enforced. If no configuration is present, the script falls bac= k +to its built-in defaults:: + + "bitbake-setup": { + "configurations": [ + { + "name": "my-config", + "install-buildtools": { + "url": "https://example.com/buildtools/x86_64-buildtool= s-extended-nativesdk-standalone-5.0.sh", + "sha256sum": "af76648b..." + }, + ... + } + ] + } + +.. _ref-bbsetup-command-settings: + +``bitbake-setup settings`` +-------------------------- + +The ``bitbake-setup settings`` sub-command helps modifying the settings of +``bitbake-setup``. This sub-command has sub-commands itself: + +- ``bitbake-setup settings list`` lists the current settings and their va= lue. +- ``bitbake-setup settings set`` sets a setting. +- ``bitbake-setup settings unset`` removes a setting. + +Settings must be set with a section and a value, for example:: + + bitbake-setup settings set default top-dir-name bitbake-builds + +Will set the value of ``top-dir-name`` in the ``default`` section to +"bitbake-builds". + +In addition, the command can take the following arguments: + +- ``--global``: write to the global settings + (``~/.config/bitbake-setup/settings.conf``) instead of the :term:`Top + Directory` settings. + +See the :ref:`bitbake-user-manual/bitbake-user-manual-environment-setup:Se= ttings` +section to see the supported settings. + +.. note:: + + The supported setting listed in the + :ref:`bitbake-user-manual/bitbake-user-manual-environment-setup:Setting= s` + section are only affected when set in the ``default`` section. + +Settings +=3D=3D=3D=3D=3D=3D=3D=3D + +The settings allow configuring ``bitbake-setup``. Settings are stored in a= file +named ``settings.conf``, in :wikipedia:`INI ` format. + +There are multiple locations for storing settings. Settings in different +locations can override each other, but the final value of a setting is com= puted +from reading the files in this order: + +#. Global settings file: ``~/.config/bitbake-setup/settings.conf``. + +#. Local settings file, taken from a ``settings.conf`` file in the :term:`= Top + Directory`. + +#. Command-line settings, passed with the ``--setting`` argument. + +A valid settings file would for example be: + +.. code-block:: ini + + [default] + top-dir-prefix =3D /path/to/workspace + top-dir-name =3D bitbake-builds + registry =3D /path/to/bitbake/default-registry + dl-dir =3D /path/to/bitbake-setup-downloads + use-full-setup-dir-name =3D yes + common-sstate =3D yes + +Settings and their values can be listed and modified with the ``bitbake-se= tup +settings`` command. See the :ref:`ref-bbsetup-command-settings` section fo= r +more information. + +Below are the available settings. + +.. _ref-bbsetup-setting-top-dir-prefix: + +``top-dir-prefix`` +------------------ + +The :ref:`ref-bbsetup-setting-top-dir-prefix` setting helps configuring th= e +leftmost part of the path to the :term:`Top Directory`. + +For example, with: + +.. code-block:: ini + + [default] + top-dir-prefix =3D /path/to/workspace + +The :term:`top directory` would be ``/path/to/workspace/`` w= ith +the ```` corresponding to the :ref:`ref-bbsetup-setting-top-= dir-name` +setting. + +This is most useful to customize on systems where the default location of = the +:term:`Top Directory` (``~/bitbake-builds``) is not suitable, and there is= a +dedicated directory for builds somewhere else. + +.. _ref-bbsetup-setting-top-dir-name: + +``top-dir-name`` +---------------- + +The :ref:`ref-bbsetup-setting-top-dir-name` setting helps configuring the +rightmost part of the path to the :term:`Top Directory`. + +For example, with: + +.. code-block:: ini + + [default] + top-dir-name =3D builds + +The :term:`top directory` would be ``/builds`` with +the ```` corresponding to the :ref:`ref-bbsetup-setting-to= p-dir-prefix` +setting. + +.. _ref-bbsetup-setting-registry: + +``registry`` +------------ + +The :ref:`ref-bbsetup-setting-registry` setting sets the URI location of t= he +registry. This URI can be any URI supported by the BitBake fetcher. + +A local registry would be configured as follows: + +.. code-block:: ini + + [default] + registry =3D /path/to/registry + +When using another fetcher, it must be specified in the URI scheme. For ex= ample: + +.. code-block:: ini + + [default] + registry =3D git://example.com/bitbake-setup-configurations;protocol=3D= https;branch=3Dmaster;rev=3Dmaster + +This would fetch the remote configurations from a remote Git remote reposi= tory, +on the ``master`` branch. + +See the :doc:`/bitbake-user-manual/bitbake-user-manual-fetching` section f= or more +information on BitBake fetchers. + +.. _ref-bbsetup-setting-dl-dir: + +``dl-dir`` +---------- + +The :ref:`ref-bbsetup-setting-dl-dir` setting sets the location of the dow= nload +cache that ``bitbake-setup`` will configure for the purpose of downloading +configuration repositories, layers and other sources using BitBake fetcher= s. +Please see :doc:`/bitbake-user-manual/bitbake-user-manual-fetching` and th= e +:term:`DL_DIR` variable for more information. + +The location can be set such that it is shared with :term:`DL_DIR` specifi= ed by +BitBake builds, so that there is a single directory containing a copy of +everything needed to set up and run a BitBake build offline in a reproduci= ble +manner. + +.. _ref-bbsetup-setting-use-full-setup-dir-name: + +``use-full-setup-dir-name`` +--------------------------- + +The :ref:`ref-bbsetup-setting-use-full-setup-dir-name` setting, if set to = ``yes`` +will override the suggestions for the :term:`Setup` directory name made by +``setup-dir-name`` entries in :term:`Configuration Template` files. This +will make the directory names longer, but fully specific: they will contai= n +all selections made during initialization. + +.. _ref-bbsetup-setting-common-sstate: + +``common-sstate`` +----------------- + +When this setting is set to ``yes`` (which is also the default), bitbake-s= etup will +set up a common sstate directory and common hash equivalency database for = all the +:term:`setups ` in a :term:`Top Directory`. This is very beneficial= for speeding +up builds as build artefacts will be reused whenever possible between them= . + +Set this to ``no`` for advanced use cases, such as placing the sstate dire= ctory on a NFS +mount and maintaining a separate hash equivalency server, so that sstate a= nd hash equivalency +data can be shared between several computers. For such use cases the sstat= e settings need +to be added to a build configuration separately. + +See https://docs.yoctoproject.org/dev-manual/hashequivserver.html for how = to share sstate +on the network. + +.. _ref-bbsetup-section-config-reference: + +Configuration Template Files Reference +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +:term:`Configuration Templates ` are the input fil= es given +to ``bitbake-setup`` to configure :term:`Setups `. + +These files are written in the JSON file format and are stored in a +:term:`Registry`. They can also be standalone files directly passed to the +:ref:`ref-bbsetup-command-init` command: + +.. code-block:: shell + + $ bitbake-setup init /path/to/config.conf.json + +They contain the following sections: + +- ``version`` (**required**): version of the configuration file. + + Example: + + .. code-block:: json + :force: + + { + "version": "1.0" + } + +- ``description`` (**required**): the description of the configuration. + + Example: + + .. code-block:: json + :force: + + { + "description": "OpenEmbedded - 'nodistro' basic configuration" + } + +- ``sources`` (**required**): sources, such as git repositories that shou= ld + be provided under ``layers/`` directory of a :term:`Setup`. Although th= e + list of sources can be very flexible, at least BitBake should be cloned + to generate a :term:`Setup`. + + Example: + + .. code-block:: json + :force: + + { + "sources": { + "bitbake": { + "git-remote": { + "uri": "https://git.openembedded.org/bitbake", + "branch": "master", + "rev": "master" + }, + "path": "bitbake" + }, + "openembedded-core": { + "local": { + "path": "~/path/to/local/openembedded-core" + } + }, + "meta-private": { + "git-remote": { + "remotes": { + "origin": { + "uri": "ssh://git@git.example.com/meta-privat= e" + } + }, + "branch": "master", + "rev": "master" + }, + "path": "meta-private" + } + } + } + + Sources can be specified with the following options: + + - ``path`` (*optional*): where the source is extracted, relative to th= e + ``layers/`` directory of a :term:`Setup`. If unspecified, the name o= f the + source is used. + + - ``git-remote`` (*optional*): specifies URI, branch and revision of a= git + repository to fetch from. + + ``git-remote`` entries are specified with the following options: + + - ``remotes`` (*optional*): a dictionary of git remote names, each = containing a ``uri`` property + with a URI that follows the git URI syntax. See https://git-scm.c= om/docs/git-clone#_git_urls + and https://git-scm.com/docs/git-remote for more information. + + This structure is useful if you want to provide several remotes. = Please + note that each URI will be cloned, but only the the last treated = one + will be used in the unpacked source. + + Example: + + .. code-block:: json + :force: + + { + "sources": { + "bitbake": { + "git-remote": { + "remotes": { + "origin": { + "uri": "https://git.openembedded.org/bi= tbake" + }, + "contrib": { + "uri": "https://git.openembedded.org/bi= tbake-contrib" + } + }, + "branch": "master", + "rev": "master" + }, + "path": "bitbake" + } + } + } + - ``rev`` (**required**): the revision to checkout. Can be the name= of the + branch to checkout on the latest revision of the specified ``bran= ch``. + + If the value is the branch name, ``bitbake-setup`` will check out= the + latest revision on that branch, and keep it updated when using th= e + :ref:`ref-bbsetup-command-update` command. + + - ``branch`` (**required**): the Git branch, used to check that the + specified ``rev`` is indeed on that branch. + + - ``uri`` (*optional*): a URI that follows the git URI syntax. You = can replace + the ``remotes`` structure if only one URI is provided. Despite th= is, ``uri`` + and ``remotes`` can still be used together. + + See https://git-scm.com/docs/git-clone#_git_urls for more informa= tion. + + - ``local`` (*optional*): specifies a path on local disk that should b= e symlinked + to under ``layers/``. This is useful for local development, where so= me layer + or other component used in a build is managed separately, but should= still be + available for bitbake-setup driven builds. + + ``local`` entries are specified with the following options: + + - ``path`` (**required**): the path on local disk where the externa= lly + managed source tree is. ``~`` and ``~user`` are expanded to that = user's home + directory. Paths in configuration files must be absolute (after p= ossible + ~ expansion), paths in override files can be relative to the dire= ctory where + the override file is. + +- ``expires`` (*optional*): Expiration date of the configuration. This da= te + should be in :wikipedia:`ISO 8601 ` format (``YYYY-MM-DDTHH:M= M:SS``). + +- ``bitbake-setup`` (**required**): contains a list of configurations. + + Example: + + .. code-block:: json + + { + "bitbake-setup": { + "configurations": [ + { + "bb-layers": ["openembedded-core/meta","meta-yocto/meta-yo= cto-bsp","meta-yocto/meta-poky"], + "bb-env-passthrough-additions": ["DL_DIR","SSTATE_DIR"], + "oe-fragments-one-of": { + "machine": { + "description": "Target machines", + "options" : [ + { "name": "machine/qemux86-64", "description":= "Machine configuration for running an x86-64 system on QEMU" }, + { "name": "machine/qemuarm64", "description": = "Machine configuration for running an ARMv8 system on QEMU" }, + { "name": "machine/qemuriscv64", "description"= : "Machine configuration for running an RISC-V system on QEMU" }, + { "name": "machine/genericarm64", "description= ": "Machine configuration for Arm64 SystemReady IR/ES platforms" }, + { "name": "machine/genericx86-64", "descriptio= n": "Machine configuration for generic x86_64 (64-bit) PCs and servers" } + ] + }, + "distro": { + "description": "Target Distributions", + "options" : [ + { "name": "distro/poky", "description": "Distr= o configuration for Poky (Yocto Project Reference Distro)" }, + { "name": "distro/poky-altcfg", "description":= "Distro configuration for Poky (systemd init manager)" }, + { "name": "distro/poky-tiny", "description": "= Distro configuration for Poky (optimized for size)" } + ] + } + }, + "configurations": [ + { + "name": "poky", + "description": "Poky - The Yocto Project testing distr= ibution" + }, + { + "name": "poky-with-sstate", + "description": "Poky - The Yocto Project testing distr= ibution with internet sstate acceleration. Use with caution as it requires = a completely robust local network with sufficient bandwidth.", + "oe-fragments": ["core/yocto/sstate-mirror-cdn"] + } + ] + } + ] + } + } + + Configurations can be specified with the following options: + + - ``name`` (**required**): the name of this configuration snippet. Thi= s is + what is prompted during the :ref:`ref-bbsetup-command-init` command + execution. + + - ``description`` (**required**): the description of this configuratio= n + snippet. This is what is prompted during the + :ref:`ref-bbsetup-command-init` command execution. + + - ``notes`` (*optional*): additional information written to + ``build/conf/conf-notes.txt`` when ``bitbake-setup`` generates the b= uild + configuration from ``bb-layers``. This can be a string, or a list of + strings which will be written one per line. For ``oe-template`` + configurations, this file is provided by the template. + + - ``configurations``: Configurations can recursively contain as many n= ested + configurations as needed. This will create more choices when running= the + :ref:`ref-bbsetup-command-init` command. + + The purpose of such nesting is to be able to scale the configuration= s, for + example when there is a need to create multiple configurations that = share + some parameters (which are specified in their common parent), but di= ffer + between themselves in other parameters. ``bitbake-setup`` will assem= ble + configuration choices by putting together information from a leaf + configuration and all of its ancestors. + + When the same keyword is present in a nested configuration and in on= e of + its ancestors, the values are merged with Python ``+`` semantics. Fo= r + example, lists are appended and strings are concatenated directly. S= tring + values such as ``description`` should include any needed separators = in + the configuration data. + + - ``bb-env-passthrough-additions`` (*optional*): List of environment + variables to include in :term:`BB_ENV_PASSTHROUGH_ADDITIONS`. + + - ``bb-layers`` (*optional*): List of layers to add to the ``bblayers.= conf`` + file. Paths in this list are relative to the ``layers/`` directory o= f a + :term:`Setup`. + + The ``bb-layers`` keyword cannot be used in conjunction with the + ``oe-template`` option, as the ``bblayers.conf`` file comes from the + template itself. + + - ``bb-layers-file-relative`` (*optional*): List of layers that are no= t + managed by ``bitbake-setup`` but that need to be included as part of= the + ``bblayers.conf`` file. Paths in this list are relative to the + configuration file. + + This is useful when (one or more) configuration files and (one or + more) layers are hosted in the same Git repository, which is cloned + and managed independently from bitbake-setup workflows. For example:= : + + =E2=94=9C=E2=94=80=E2=94=80 meta-myproject/ + =E2=94=94=E2=94=80=E2=94=80 myproject.conf.json + + Then ``myproject.conf.json`` can contain the following to add + ``meta-myproject`` to ``bblayers.conf``:: + + { + ... + "bb-layers-file-relative": [ + "meta-myproject" + ], + ... + } + + The ``bb-layers-file-relative`` keyword cannot be used in conjunctio= n with the + ``oe-template`` keyword, as the ``bblayers.conf`` file comes from th= e + template itself. + + - ``oe-template`` (*optional*, OpenEmbedded specific): OpenEmbedded te= mplate + to use. This cannot be used in conjunction with the + ``bb-layers`` or ``bb-layers-file-relative`` keywords as it + already provides a ready ``bblayers.conf`` file to use. + + See https://docs.yoctoproject.org/dev-manual/custom-template-configu= ration-directory.html + for more information of OpenEmbedded templates. + + - ``oe-fragments-one-of`` (*optional*, OpenEmbedded specific): the Ope= nEmbedded + fragments to select as part of the build. + + This can one of two formats: + + - A list of dictionaries with ``name`` and a ``description``, + corresponding to respectively the fragments names and fragment + descriptions (like in the example above). + + - A list of fragment names. For example: + + .. code-block:: json + + "options" : ["machine/qemux86-64", "machine/qemuarm64", "machi= ne/qemuriscv64", "machine/genericarm64", "machine/genericx86-64"] + + This will trigger choices to make during the + :ref:`ref-bbsetup-command-init` command execution. + + See https://docs.yoctoproject.org/dev/ref-manual/fragments.html for + more information of OpenEmbedded configuration fragments. + + - ``oe-fragments`` (*optional*, OpenEmbedded specific): fragments to s= elect + as part of the build. + + See https://docs.yoctoproject.org/dev/ref-manual/fragments.html for + more information of OpenEmbedded configuration fragments. + + - ``setup-dir-name`` (*optional*): a suggestion for the :term:`Setup` + directory name. Bitbake-setup will use it, unless it already exists,= or + the :ref:`ref-bbsetup-setting-use-full-setup-dir-name` setting is se= t + to ``yes`` (in those cases, it will fall back to the built-in full n= ame, + containing the full set of choices made during initialization). + + This key can have variable expansions in the same format as python + string.Template strings (which matches the shell variable expansion = rules): + https://docs.python.org/3/library/string.html#template-strings-strin= gs . + + Variables that can be expanded are any fragment configuration prompt= ed + by the user (e.g. the keys in "oe-fragments-one-of"):: + + { + ... + "oe-fragments-one-of": { + "machine": { + "description": "Target machines", + "options" : ["machine/gadget", "machine/gizmo"] + } + }, + "setup-dir-name": "somebuild-$machine" + ... + } + + would expand to ``somebuild-machine_gadget``. + +Configuration Template Examples +------------------------------- + +OpenEmbedded "nodistro" configuration for master branches: + +.. literalinclude:: ../../default-registry/configurations/oe-nodistro-mast= er.conf.json + :language: json + +Poky distribution configuration for master branches: + +.. literalinclude:: ../../default-registry/configurations/poky-master.conf= .json + :language: json + +.. _ref-bbsetup-source-overrides: + +Source Overrides +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +See the definition of :term:`Source Override` in the Terminology section. + +These files are written in the JSON file format and are optionally passed = to the +``--source-overrides`` argument of the :ref:`ref-bbsetup-command-init` com= mand. +The ``--source-overrides`` option can be passed multiple times, in which c= ase the +overrides are applied in the order specified in the command-line. + +Here is an example file that overrides the branch of the BitBake repositor= y to +"master-next", and provides openembedded-core as a symlink to a path on lo= cal disk: + +.. code-block:: json + + { + "description": "Source override file", + "sources": { + "bitbake": { + "git-remote": { + "branch": "master-next", + "remotes": { + "origin": { + "uri": "https://git.openembedded.org/bitbake" + } + }, + "rev": "master-next" + } + }, + "openembedded-core": { + "local": { + "path": "~/path/to/local/openembedded-core" + } + } + }, + "version": "1.0" + } + +- The ``version`` parameter contains the version of the used configuratio= n, and + should match the one of the :term:`Configuration Template` file in use. + +- The ``sources`` section contains the same options as the ``sources`` op= tion + of a :term:`Configuration Template` file. See the + :ref:`ref-bbsetup-section-config-reference` section for more informatio= n. + +.. _ref-bbsetup-fixed-revisions: + +Fixed source revisions +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +:term:`Configuration Template` can set source revisions in ``rev`` to a ta= g or a branch. +Bitbake-setup will make sure the actual revision will match the tag or bra= nch when performing +initializations or updates, and will capture the revisions in a :ref:`ref-= bbsetup-source-overrides` +file. + +This file is named ``sources-fixed-revisions.json`` and is available in ``= layers/`` +under the :term:`Setup` directory (as well as under ``config/`` for backwa= rds compatibility). +It can be used to keep record of what was checked out when using configura= tions that do not +specify exact revisions. + +Also, as it is an override file, it can be combined with the original conf= igurations +to initialize a setup in a reproducible way that guarantees an exact, neve= r-changing +set of revisions, by using the ``--source-overrides`` option of the +:ref:`ref-bbsetup-command-init` command. + +For example if the original configuration had specified only a master bran= ch +for a source:: + + "bitbake": { + "git-remote": { + "remotes": { + "origin": { + "uri": "https://git.openembedded.org/bitbake" + } + }, + "branch": "master", + "rev": "master" + } + + +the override will contain the exact revisions:: + + "bitbake": { + "git-remote": { + "remotes": { + "origin": { + "uri": "https://git.openembedded.org/bitbake" + } + }, + "branch": "master", + "rev": "720df1a53452983c1c832f624490e255cf389204" + } diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.= rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst index d58fbb32..1638a8f5 100644 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst @@ -64,11 +64,11 @@ data itself is of various types: together. =20 The ``layer.conf`` files are used to construct key variables such as -:term:`BBPATH` and :term:`BBFILES`. -:term:`BBPATH` is used to search for configuration and class files under t= he -``conf`` and ``classes`` directories, respectively. :term:`BBFILES` is use= d -to locate both recipe and recipe append files (``.bb`` and -``.bbappend``). If there is no ``bblayers.conf`` file, it is assumed the +:term:`BBPATH` and :term:`BBFILES`. :term:`BBPATH` is used to search for +configuration files under the ``conf`` directory and class files under the +``classes-global``, ``classes-recipe`` and ``classes`` directories. +:term:`BBFILES` is used to locate both recipe and recipe append files (``.= bb`` +and ``.bbappend``). If there is no ``bblayers.conf`` file, it is assumed t= he user has set the :term:`BBPATH` and :term:`BBFILES` directly in the enviro= nment. =20 Next, the ``bitbake.conf`` file is located using the :term:`BBPATH` variab= le @@ -337,21 +337,22 @@ file that BitBake parses, you can change that prefere= nce:: =20 .. note:: =20 - It is common for a recipe to provide two versions -- a stable, - numbered (and preferred) version, and a version that is automatically - checked out from a source code repository that is considered more - "bleeding edge" but can be selected only explicitly. + It is not uncommon for a given recipe to provide more than one + version -- a stable (and preferred) version that should be selected + by default, and a newer, development or "bleeding edge" version that + will be selected only explicitly by the developer. =20 - For example, in the OpenEmbedded codebase, there is a standard, - versioned recipe file for BusyBox, ``busybox_1.22.1.bb``, but there - is also a Git-based version, ``busybox_git.bb``, which explicitly - contains the line :: + For example, at the moment, in the OpenEmbedded codebase, there are + two versions of the ``nginx`` webserver: a stable "1.28.0" version + that would be selected by default, and a newer "1.29.1" development + version that a developer would need to choose explicitly. =20 - DEFAULT_PREFERENCE =3D "-1" + This can be seen based on this snippet from the ``nginx`` 1.29.1 + recipe file:: =20 - to ensure that the - numbered, stable version is always preferred unless the developer - selects otherwise. + # 1.28.x branch is the current stable branch, the recommended defaul= t + # 1.29.x is the current mainline branches containing all new feature= s + DEFAULT_PREFERENCE =3D "-1" =20 .. _bb-bitbake-dependencies: =20 diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.r= st b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst index fb4f0a23..9f00545d 100644 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst @@ -11,10 +11,11 @@ with the intricacies of downloading source code and fil= es from remote systems. Fetching source code is one of the cornerstones of building software. As such, this module forms an important part of BitBake. =20 -The current fetch module is called "fetch2" and refers to the fact that -it is the second major version of the API. The original version is -obsolete and has been removed from the codebase. Thus, in all cases, -"fetch" refers to "fetch2" in this manual. +The fetch module was previously called ``fetch2`` to distinguish the secon= d +major version of the API from the original version, which is obsolete and = has +been removed from the codebase. The module has since been renamed back to +``fetch``. Existing code using ``bb.fetch2`` remains supported by a +compatibility alias, but new code should use ``bb.fetch``. =20 The Download (Fetch) =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D @@ -30,7 +31,7 @@ The code to execute the first part of this process, a fet= ch, looks something like the following:: =20 src_uri =3D (d.getVar('SRC_URI') or "").split() - fetcher =3D bb.fetch2.Fetch(src_uri, d) + fetcher =3D bb.fetch.Fetch(src_uri, d) fetcher.download() =20 This code sets up an instance of the fetch class. The instance uses a @@ -39,10 +40,10 @@ variable and then calls the ``download`` method to down= load the files. =20 The instantiation of the fetch class is usually followed by:: =20 - rootdir =3D l.getVar('WORKDIR') + rootdir =3D l.getVar('UNPACKDIR') fetcher.unpack(rootdir) =20 -This code unpacks the downloaded files to the specified by ``WORKDIR``. +This code unpacks the downloaded files to the specified by ``UNPACKDIR``. =20 .. note:: =20 @@ -51,7 +52,7 @@ This code unpacks the downloaded files to the specified b= y ``WORKDIR``. examine the OpenEmbedded class file ``base.bbclass`` . =20 -The :term:`SRC_URI` and ``WORKDIR`` variables are not hardcoded into the +The :term:`SRC_URI` and ``UNPACKDIR`` variables are not hardcoded into the fetcher, since those fetcher methods can be (and are) called with different variable names. In OpenEmbedded for example, the shared state (sstate) code uses the fetch module to fetch the sstate files. @@ -84,11 +85,8 @@ fetcher does know how to use HTTP as a transport. Here are some examples that show commonly used mirror definitions:: =20 PREMIRRORS ?=3D "\ - bzr://.*/.\* http://somemirror.org/sources/ \ - cvs://.*/.\* http://somemirror.org/sources/ \ git://.*/.\* http://somemirror.org/sources/ \ hg://.*/.\* http://somemirror.org/sources/ \ - osc://.*/.\* http://somemirror.org/sources/ \ p4://.*/.\* http://somemirror.org/sources/ \ svn://.*/.\* http://somemirror.org/sources/" =20 @@ -126,7 +124,7 @@ in the previous example, or you can name the URLs. The = following syntax shows how you name the URIs:: =20 SRC_URI =3D "http://example.com/foobar.tar.bz2;name=3Dfoo" - SRC_URI[foo.md5sum] =3D 4a8e0f237e961fd7785d19d07fdb994d + SRC_URI[foo.md5sum] =3D "4a8e0f237e961fd7785d19d07fdb994d" =20 After a file has been downloaded and has had its checksum checked, a ".done" stamp is placed in :term:`DL_DIR`. @@ -159,19 +157,25 @@ URLs except Git URLs, BitBake uses the common ``unpac= k`` method. A number of parameters exist that you can specify within the URL to govern the behavior of the unpack stage: =20 -- *unpack:* Controls whether the URL components are unpacked. If set to +- *"unpack":* Controls whether the URL components are unpacked. If set to "1", which is the default, the components are unpacked. If set to "0", the unpack stage leaves the file alone. This parameter is useful when you want an archive to be copied in and not be unpacked. =20 -- *dos:* Applies to ``.zip`` and ``.jar`` files and specifies whether +- *"dos":* Applies to ``.zip`` and ``.jar`` files and specifies whether to use DOS line ending conversion on text files. =20 -- *striplevel:* Strip specified number of leading components (levels) - from file names on extraction +- *"striplevel":* Strip specified number of leading components (levels) + from file names on extraction. =20 -- *subdir:* Unpacks the specific URL to the specified subdirectory - within the root directory. +- *"subdir":* Unpacks the specific URL to the specified subdirectory + within the specified root directory. This path can be further modified + by fetcher specific parameters. + +- *"name":* Assigns a name to a given component of the :term:`SRC_URI`. + This component is later referenced by this name when specifying its + :term:`SRCREV` or :term:`SRC_URI` checksum, or to correctly place its + revision in the package version string with aid of :term:`SRCREV_FORMAT= `. =20 The unpack call automatically decompresses and extracts files with ".Z", ".z", ".gz", ".xz", ".zip", ".jar", ".ipk", ".rpm". ".srpm", ".deb" and @@ -183,6 +187,51 @@ cloning the tree into the final directory. The process= is completed using references so that there is only one central copy of the Git metadata needed. =20 +.. _bb-the-unpack-update: + +The Non-Destructive Update (``unpack_update``) +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +.. note:: + + This is a specialised method intended for tools that manage persistent + layer checkouts on behalf of the user, such as + :ref:`bitbake-setup update `. It is not par= t of the + normal recipe fetch/unpack flow. + +For Git URLs, an alternative ``unpack_update()`` method is available +that updates an existing checkout *in place* rather than removing and +re-cloning it. This is useful when the target directory may contain +local commits that should be preserved across updates. + +The code to call the non-destructive update looks like the following:: + + rootdir =3D l.getVar('UNPACKDIR') + fetcher.unpack_update(rootdir) + +``unpack_update()`` performs the following steps: + +1. A ``dldir`` git remote is added (or updated) in the existing + checkout, pointing at the local download cache. This remote may + also be useful for manually resolving conflicts outside the fetcher. +2. The new upstream revision is fetched from the ``dldir`` remote into + the existing repository. +3. Any local commits are rebased on top of the new upstream revision, + preserving local work. + +The method raises an error if: + +- The working tree contains staged or unstaged changes to tracked + files (``LocalModificationsError``). +- Local commits cannot be cleanly rebased onto the new upstream + revision (``RebaseError``). A failed rebase is automatically aborted + before the exception is raised. +- The download cache does not contain a sufficiently recent clone + of the repository, or the checkout is a shallow clone. + +Currently only the Git fetcher supports ``unpack_update()``. All other +fetcher types raise ``RuntimeError`` if it is called. + .. _bb-fetchers: =20 Fetchers @@ -246,91 +295,19 @@ Some example URLs are as follows:: introduce ambiguity when parsing URLs that also contain semi-colons, for example:: =20 - SRC_URI =3D "http://abc123.org/git/?p=3Dgcc/gcc.git;a=3Dsnapsho= t;h=3Da5dd47" - + SRC_URI =3D "http://abc123.org/git/?p=3Dgcc/gcc.git;a=3Dsnapshot;h= =3Da5dd47" =20 Such URLs should should be modified by replacing semi-colons with '&' characters:: =20 - SRC_URI =3D "http://abc123.org/git/?p=3Dgcc/gcc.git&a=3Dsnapsho= t&h=3Da5dd47" - + SRC_URI =3D "http://abc123.org/git/?p=3Dgcc/gcc.git&a=3Dsnapshot&h= =3Da5dd47" =20 In most cases this should work. Treating semi-colons and '&' in queries identically is recommended by the World Wide Web Consortium (W3C). Note that due to the nature of the URL, you may have to specify the name of the downloaded file as well:: =20 - SRC_URI =3D "http://abc123.org/git/?p=3Dgcc/gcc.git&a=3Dsnapsho= t&h=3Da5dd47;downloadfilename=3Dmyfile.bz2" - - -.. _cvs-fetcher: - -CVS fetcher (``(cvs://``) -------------------------- - -This submodule handles checking out files from the CVS version control -system. You can configure it using a number of different variables: - -- :term:`FETCHCMD_cvs `: The name of the executable to use when= running - the ``cvs`` command. This name is usually "cvs". - -- :term:`SRCDATE`: The date to use when fetching the CVS source code. A - special value of "now" causes the checkout to be updated on every - build. - -- :term:`CVSDIR`: Specifies where a temporary - checkout is saved. The location is often ``DL_DIR/cvs``. - -- CVS_PROXY_HOST: The name to use as a "proxy=3D" parameter to the - ``cvs`` command. - -- CVS_PROXY_PORT: The port number to use as a "proxyport=3D" - parameter to the ``cvs`` command. - -As well as the standard username and password URL syntax, you can also -configure the fetcher with various URL parameters: - -The supported parameters are as follows: - -- *"method":* The protocol over which to communicate with the CVS - server. By default, this protocol is "pserver". If "method" is set to - "ext", BitBake examines the "rsh" parameter and sets ``CVS_RSH``. You - can use "dir" for local directories. - -- *"module":* Specifies the module to check out. You must supply this - parameter. - -- *"tag":* Describes which CVS TAG should be used for the checkout. By - default, the TAG is empty. - -- *"date":* Specifies a date. If no "date" is specified, the - :term:`SRCDATE` of the configuration is used to - checkout a specific date. The special value of "now" causes the - checkout to be updated on every build. - -- *"localdir":* Used to rename the module. Effectively, you are - renaming the output directory to which the module is unpacked. You - are forcing the module into a special directory relative to - :term:`CVSDIR`. - -- *"rsh":* Used in conjunction with the "method" parameter. - -- *"scmdata":* Causes the CVS metadata to be maintained in the tarball - the fetcher creates when set to "keep". The tarball is expanded into - the work directory. By default, the CVS metadata is removed. - -- *"fullpath":* Controls whether the resulting checkout is at the - module level, which is the default, or is at deeper paths. - -- *"norecurse":* Causes the fetcher to only checkout the specified - directory with no recurse into any subdirectories. - -- *"port":* The port to which the CVS server connects. - -Some example URLs are as follows:: - - SRC_URI =3D "cvs://CVSROOT;module=3Dmymodule;tag=3Dsome-version;method= =3Dext" - SRC_URI =3D "cvs://CVSROOT;module=3Dmymodule;date=3D20060126;localdir= =3Dusethat" + SRC_URI =3D "http://abc123.org/git/?p=3Dgcc/gcc.git&a=3Dsnapshot&h= =3Da5dd47;downloadfilename=3Dmyfile.bz2" =20 .. _svn-fetcher: =20 @@ -398,16 +375,16 @@ This fetcher supports the following parameters: =20 .. note:: =20 - When ``protocol`` is "ssh", the URL expected in :term:`SRC_URI` diffe= rs - from the one that is typically passed to ``git clone`` command and pr= ovided - by the Git server to fetch from. For example, the URL returned by Git= Lab - server for ``mesa`` when cloning over SSH is - ``git@gitlab.freedesktop.org:mesa/mesa.git``, however the expected UR= L in - :term:`SRC_URI` is the following:: + When ``protocol`` is "ssh", the URL expected in :term:`SRC_URI` diff= ers + from the one that is typically passed to ``git clone`` command and p= rovided + by the Git server to fetch from. For example, the URL returned by Gi= tLab + server for ``mesa`` when cloning over SSH is + ``git@gitlab.freedesktop.org:mesa/mesa.git``, however the expected U= RL in + :term:`SRC_URI` is the following:: =20 - SRC_URI =3D "git://git@gitlab.freedesktop.org/mesa/mesa.git;branch= =3Dmain;protocol=3Dssh;..." + SRC_URI =3D "git://git@gitlab.freedesktop.org/mesa/mesa.git;branc= h=3Dmain;protocol=3Dssh;..." =20 - Note the ``:`` character changed for a ``/`` before the path to the p= roject. + Note the ``:`` character changed for a ``/`` before the path to the = project. =20 - *"nocheckout":* Tells the fetcher to not checkout source code when unpacking when set to "1". Set this option for the URL where there is @@ -436,13 +413,15 @@ This fetcher supports the following parameters: "nobranch" is set to "1", this is a mandatory parameter. The number of branch parameters must match the number of name parameters. =20 -- *"rev":* The revision to use for the checkout. The default is - "master". +- *"rev":* The revision to use for the checkout. If :term:`SRCREV` is als= o set, + this parameter must match its value. =20 -- *"tag":* Specifies a tag to use for the checkout. To correctly - resolve tags, BitBake must access the network. For that reason, tags - are often not used. As far as Git is concerned, the "tag" parameter - behaves effectively the same as the "rev" parameter. +- *"tag":* Specifies a tag to use when fetching. To correctly resolve + tags, BitBake must access the network. If a ``rev`` parameter or + :term:`SRCREV` is also specified, network access is not necessary to re= solve + the tag and instead, it is verified that they both resolve to the same = commit + SHA at unpack time. The ``tag`` parameter is optional, but strongly + recommended if the checked out revision is a tag. =20 - *"subpath":* Limits the checkout to a specific subpath of the tree. By default, the whole tree is checked out. @@ -451,7 +430,7 @@ This fetcher supports the following parameters: By default, the path is ``git/``. =20 - *"usehead":* Enables local ``git://`` URLs to use the current branch - HEAD as the revision for use with ``AUTOREV``. The "usehead" + HEAD as the revision for use with :term:`AUTOREV`. The "usehead" parameter implies no branch and only works when the transfer protocol is ``file://``. =20 @@ -461,13 +440,6 @@ Here are some example URLs:: SRC_URI =3D "git://github.com/asciidoc/asciidoc-py;protocol=3Dhttps;bra= nch=3Dmain" SRC_URI =3D "git://git@gitlab.freedesktop.org/mesa/mesa.git;branch=3Dma= in;protocol=3Dssh;..." =20 -.. note:: - - When using ``git`` as the fetcher of the main source code of your softw= are, - ``S`` should be set accordingly:: - - S =3D "${WORKDIR}/git" - .. note:: =20 Specifying passwords directly in ``git://`` urls is not supported. @@ -476,7 +448,7 @@ Here are some example URLs:: easy to share metadata without removing passwords. SSH keys, ``~/.netrc= `` and ``~/.ssh/config`` files can be used as alternatives. =20 -Using tags with the git fetcher may cause surprising behaviour. Bitbake ne= eds to +Using tags with the git fetcher may cause surprising behaviour. BitBake ne= eds to resolve the tag to a specific revision and to do that, it has to connect t= o and use the upstream repository. This is because the revision the tags point at ca= n change and we've seen cases of this happening in well known public repositories. This= can mean @@ -530,10 +502,10 @@ The fetcher uses the ``rcleartool`` or =20 Following are options for the :term:`SRC_URI` statement: =20 -- *vob*: The name, which must include the prepending "/" character, +- *"vob":* The name, which must include the prepending "/" character, of the ClearCase VOB. This option is required. =20 -- *module*: The module, which must include the prepending "/" +- *"module":* The module, which must include the prepending "/" character, in the selected VOB. =20 .. note:: @@ -545,7 +517,7 @@ Following are options for the :term:`SRC_URI` statement= : =20 load /example_vob/example_module =20 -- *proto*: The protocol, which can be either ``http`` or ``https``. +- *"proto":* The protocol, which can be either ``http`` or ``https``. =20 By default, the fetcher creates a configuration specification. If you want this specification written to an area other than the default, use @@ -554,9 +526,9 @@ the specification is written. =20 .. note:: =20 - the SRCREV loses its functionality if you specify this variable. Howeve= r, - SRCREV is still used to label the archive after a fetch even though it = does - not define what is fetched. + the :term:`SRCREV` loses its functionality if you specify this variable= . + However, :term:`SRCREV` is still used to label the archive after a fetc= h even + though it does not define what is fetched. =20 Here are a couple of other behaviors worth mentioning: =20 @@ -598,7 +570,7 @@ and port, username, and password, and fetches the Head = Revision:: SRC_URI =3D "p4://example-depot/main/source/..." SRCREV =3D "${AUTOREV}" PV =3D "p4-${SRCPV}" - S =3D "${WORKDIR}/p4" + S =3D "${UNPACKDIR}/p4" =20 Here is an example that specifies the server URL and port, username, and password, and fetches a Revision based on a Label:: @@ -607,25 +579,25 @@ password, and fetches a Revision based on a Label:: SRC_URI =3D "p4://user:passwd@example-depot/main/source/..." SRCREV =3D "release-1.0" PV =3D "p4-${SRCPV}" - S =3D "${WORKDIR}/p4" + S =3D "${UNPACKDIR}/p4" =20 .. note:: =20 - You should always set S to "${WORKDIR}/p4" in your recipe. + You should always set S to "${UNPACKDIR}/p4" in your recipe. =20 By default, the fetcher strips the depot location from the local file path= s. In the above example, the content of ``example-depot/main/source/`` will be p= laced -in ``${WORKDIR}/p4``. For situations where preserving parts of the remote= depot +in ``${UNPACKDIR}/p4``. For situations where preserving parts of the remo= te depot paths locally is desirable, the fetcher supports two parameters: =20 -- *"module":* - The top-level depot location or directory to fetch. The value of this - parameter can also point to a single file within the depot, in which c= ase - the local file path will include the module path. -- *"remotepath":* - When used with the value "``keep``", the fetcher will mirror the full = depot - paths locally for the specified location, even in combination with the - ``module`` parameter. +- *"module":* + The top-level depot location or directory to fetch. The value of this + parameter can also point to a single file within the depot, in which ca= se + the local file path will include the module path. +- *"remotepath":* + When used with the value "``keep``", the fetcher will mirror the full d= epot + paths locally for the specified location, even in combination with the + ``module`` parameter. =20 Here is an example use of the the ``module`` parameter:: =20 @@ -686,9 +658,9 @@ Such functionality is set by the variable: delegate access to resources, if this variable is set, the Az Fetcher w= ill use it when fetching artifacts from the cloud. =20 -You can specify the AZ_SAS variable as shown below:: +You can specify the AZ_SAS variable prefixed with a ? as shown below:: =20 - AZ_SAS =3D "se=3D2021-01-01&sp=3Dr&sv=3D2018-11-09&sr=3Dc&skoid=3D&sig=3D" + AZ_SAS =3D "?se=3D2021-01-01&sp=3Dr&sv=3D2018-11-09&sr=3Dc&skoid=3D&sig=3D" =20 Here is an example URL:: =20 @@ -745,6 +717,11 @@ The format for the :term:`SRC_URI` setting must be:: =20 SRC_URI =3D "crate://REGISTRY/NAME/VERSION" =20 +This fetcher supports the following parameters: + +- *"protocol":* The protocol used to fetch the crates. The default is "ht= tps". + You can also use "http". + Here is an example URL:: =20 SRC_URI =3D "crate://crates.io/glob/0.2.11" @@ -754,6 +731,12 @@ Here is an example URL:: NPM Fetcher (``npm://``) ------------------------ =20 +.. warning:: + + The NPM fetcher is currently disabled due to security concerns. See + `355cd226e0720a9ed7683bb01c8c0a58eee03664 `__ + for more information. + This submodule fetches source code from an `NPM `__ Javascript package registry. @@ -793,6 +776,12 @@ to automatically create a recipe from an NPM URL. NPM shrinkwrap Fetcher (``npmsw://``) ------------------------------------- =20 +.. warning:: + + The NPM fetcher is currently disabled due to security concerns. See + `355cd226e0720a9ed7683bb01c8c0a58eee03664 `__ + for more information. + This submodule fetches source code from an `NPM shrinkwrap `__ description file, which lists the dependencies @@ -828,12 +817,8 @@ Other Fetchers =20 Fetch submodules also exist for the following: =20 -- Bazaar (``bzr://``) - - Mercurial (``hg://``) =20 -- OSC (``osc://``) - - S3 (``s3://``) =20 - Secure FTP (``sftp://``) @@ -848,4 +833,42 @@ submodules. However, you might find the code helpful a= nd readable. Auto Revisions =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 -We need to document ``AUTOREV`` and :term:`SRCREV_FORMAT` here. +For recipes which need to use the latest revision of their source code, +the way to achieve it is to use :term:`AUTOREV` as the value of the +source code repository's :term:`SRCREV`:: + + SRCREV =3D "${AUTOREV}" + +.. note:: + + With :term:`AUTOREV`, BitBake will always need to take the additional s= tep of + querying the remote repository to retrieve the latest available revisio= n. + + Also, recipes using it are not part of the parsing-time cache, + and hence are parsed every time. + +Multiple Source Control Repositories +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +For some recipes it is necessary to make use of more than one +version controlled source code repository. In such case, the recipe +must provide BitBake with information about how it should include +the different SCM revisions in its package version string, instead of its +usual approach with a single :term:`SRCREV`. + +For this purpose, the recipe must set the :term:`SRCREV_FORMAT` +variable. Consider the following example:: + + SRC_URI =3D " \ + git://git.some.example.com/source-tree.git;name=3Dmachine \ + git://git.some.example.com/metadata.git;name=3Dmeta \ + " + SRCREV_machine =3D "3f9db490a81eeb0077be3c5a5aa1388a2372232f" + SRCREV_meta =3D "1ac1d0ff730fe1dd1371823d562db8126750a98c" + SRCREV_FORMAT ?=3D "meta_machine" + +The value given to :term:`SRCREV_FORMAT` references names, which were +assigned using the ``name`` parameter in the :term:`SRC_URI` definition, +and which represent the version controlled source code repositories. +In the above example, the :term:`SRC_URI` contained two URLs named +"meta" and "machine". diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.rst = b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.rst index 35ffb88b..6801323e 100644 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.rst +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-intro.rst @@ -198,7 +198,7 @@ Class files, which are denoted by the ``.bbclass`` exte= nsion, contain information that is useful to share between metadata files. The BitBake source tree currently comes with one class metadata file called ``base.bbclass``. You can find this file in the ``classes`` directory. -The ``base.bbclass`` class files is special since it is always included +The ``base.bbclass`` class file is special since it is always included automatically for all recipes and classes. This class contains definitions for standard basic tasks such as fetching, unpacking, configuring (empty by default), compiling (runs any Makefile present), @@ -206,6 +206,35 @@ installing (empty by default) and packaging (empty by = default). These tasks are often overridden or extended by other classes added during the project development process. =20 +Class Types +~~~~~~~~~~~ + +BitBake supports class files installed in three different directories: + +- ``classes-global/``: these classes must be inherited globally through t= he + :term:`INHERIT` variable in a :ref:`configuration file + `. T= hese + classes are included for every recipe being built. For example, you wou= ld use + the global class named ``myclass`` like so:: + + INHERIT +=3D "myclass" + +- ``classes-recipe/``: these classes must be inherited from a recipe usin= g the + :ref:`inherit ` or + :ref:`inherit_defer ` + directive. They do not support being inherited globally. For example, y= ou + would use the recipe class named ``myclass`` like so:: + + inherit myclass + +- ``classes/``: this final directory is meant for classes that can be use= d in + the two contexts explained above. In other words, they can be inherited= either + globally or in a recipe. + +For details on how BitBake locates class files, see the +:ref:`bitbake-user-manual/bitbake-user-manual-metadata:Locating Class File= s` +section of the BitBake User Manual. + Layers ------ =20 @@ -242,12 +271,9 @@ same root filename. The filenames can differ only in t= he file type suffix used (e.g. ``formfactor_0.0.bb`` and ``formfactor_0.0.bbappend``). =20 -Information in append files extends or overrides the information in the -underlying, similarly-named recipe files. - When you name an append file, you can use the "``%``" wildcard character -to allow for matching recipe names. For example, suppose you have an -append file named as follows:: +to allow for a less strict recipe name matching. For example, suppose you = have +an append file named as follows:: =20 busybox_1.21.%.bbappend =20 @@ -258,12 +284,8 @@ the append file would match the following recipe names= :: busybox_1.21.1.bb busybox_1.21.2.bb busybox_1.21.3.bb - -.. note:: - - The use of the " % " character is limited in that it only works directl= y in - front of the .bbappend portion of the append file's name. You cannot us= e the - wildcard character in any other location of the name. + busybox_1.21.10.bb + busybox_1.21.11.bb =20 If the ``busybox`` recipe was updated to ``busybox_1.3.0.bb``, the append name would not match. However, if you named the append file @@ -272,6 +294,47 @@ append name would not match. However, if you named the= append file In the most general case, you could name the append file something as simple as ``busybox_%.bbappend`` to be entirely version independent. =20 +A recipe named without a version may only be appended by an append file na= med +exactly the same. That is, a recipe named:: + + mesa.bb + +can only be appended by an append file named:: + + mesa.bbappend + +That is, ``mesa_%.bbappend`` append file will not match a ``mesa.bb`` reci= pe. + +.. note:: + + The "``%``" character is a marker for BitBake, and everything after tha= t + marker will not be used for matching recipe names. That is, an append f= ile + named:: + + busybox_1.%.11.bbappend + + will match recipes named:: + + busybox_1.0.11.bb + busybox_1.2.bb + busybox_1.3.48.bb + + Note that the "``%``" character is not specific to the version part of = the + recipe name. That is, an append file named:: + + linux-yocto%.bbappend + + will match recipes named:: + + linux-yocto.bb + linux-yocto_6.18.bb + linux-yocto-tiny_6.12.bb + linux-yocto-rt_6.16.bb + + While it can be useful to append to similar recipes with slightly diffe= rent + names, it is highly discouraged to do so as recipes from other layers m= ay be + named similarly enough that the append file would unwittingly match the= m. + Obtaining BitBake =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 @@ -349,40 +412,84 @@ Usage and syntax Following is the usage and syntax for BitBake:: =20 $ bitbake -h - Usage: bitbake [options] [recipename/target recipe:do_task ...] - - Executes the specified task (default is 'build') for a given set of= target recipes (.bb files). - It is assumed there is a conf/bblayers.conf available in cwd or in = BBPATH which - will provide the layer, BBFILES and other configuration information= . + usage: bitbake [-s] [-e] [-g] [-u UI] [--version] [-h] [-f] [-c CMD] + [-C INVALIDATE_STAMP] [--runall RUNALL] [--runonly RUNON= LY] + [--no-setscene] [--skip-setscene] [--setscene-only] [-n]= [-p] + [-k] [-P] [-S SIGNATURE_HANDLER] [--revisions-changed] + [-b BUILDFILE] [-D] [-l DEBUG_DOMAINS] [-v] [-q] + [-w WRITEEVENTLOG] [-B BIND] [-T SERVER_TIMEOUT] + [--remote-server REMOTE_SERVER] [-m] [--token XMLRPCTOKE= N] + [--observe-only] [--status-only] [--server-only] [-r PRE= FILE] + [-R POSTFILE] [-I EXTRA_ASSUME_PROVIDED] + [recipename/target ...] + + It is assumed there is a conf/bblayers.conf available in cwd or in BBPA= TH + which will provide the layer, BBFILES and other configuration informati= on. + + General options: + recipename/target Execute the specified task (default is 'build')= for + these target recipes (.bb files). + -s, --show-versions Show current and preferred versions of all reci= pes. + -e, --environment Show the global or per-recipe environment compl= ete + with information about where variables were + set/changed. + -g, --graphviz Save dependency tree information for the specif= ied + targets in the dot syntax. + -u UI, --ui UI The user interface to use (knotty, ncurses, tas= kexp, + taskexp_ncurses or teamcity - default knotty). + --version Show programs version and exit. + -h, --help Show this help message and exit. =20 - Options: - --version show program's version number and exit - -h, --help show this help message and exit - -b BUILDFILE, --buildfile=3DBUILDFILE - Execute tasks from a specific .bb recipe direct= ly. - WARNING: Does not handle any dependencies from = other - recipes. - -k, --continue Continue as much as possible after an error. Wh= ile the - target that failed and anything depending on it= cannot - be built, as much as possible will be built bef= ore - stopping. + Task control options: -f, --force Force the specified targets/task to run (invali= dating any existing stamp file). - -c CMD, --cmd=3DCMD Specify the task to execute. The exact option= s + -c CMD, --cmd CMD Specify the task to execute. The exact options available depend on the metadata. Some examples= might be 'compile' or 'populate_sysroot' or 'listtask= s' may give a list of the tasks available. - -C INVALIDATE_STAMP, --clear-stamp=3DINVALIDATE_STAMP + -C INVALIDATE_STAMP, --clear-stamp INVALIDATE_STAMP Invalidate the stamp for the specified task suc= h as 'compile' and then run the default task for the specified target(s). - -r PREFILE, --read=3DPREFILE - Read the specified file before bitbake.conf. - -R POSTFILE, --postread=3DPOSTFILE - Read the specified file after bitbake.conf. - -v, --verbose Enable tracing of shell tasks (with 'set -x'). = Also - print bb.note(...) messages to stdout (in addit= ion to - writing them to ${T}/log.do_<task>). + --runall RUNALL Run the specified task for any recipe in the ta= skgraph + of the specified target (even if it wouldn't ot= herwise + have run). + --runonly RUNONLY Run only the specified task within the taskgrap= h of + the specified targets (and any task dependencie= s those + tasks may have). + --no-setscene Do not run any setscene tasks. sstate will be i= gnored + and everything needed, built. + --skip-setscene Skip setscene tasks if they would be executed. = Tasks + previously restored from sstate will be kept, u= nlike + --no-setscene. + --setscene-only Only run setscene tasks, don't run any real tas= ks. + + Execution control options: + -n, --dry-run Don't execute, just go through the motions. + -p, --parse-only Quit after parsing the BB recipes. + -k, --continue Continue as much as possible after an error. Wh= ile the + target that failed and anything depending on it= cannot + be built, as much as possible will be built bef= ore + stopping. + -P, --profile Profile the command and save reports. + -S SIGNATURE_HANDLER, --dump-signatures SIGNATURE_HANDLER + Dump out the signature construction information= , with + no task execution. The SIGNATURE_HANDLER parame= ter is + passed to the handler. Two common values are no= ne and + printdiff but the handler may define more/less.= none + means only dump the signature, printdiff means + recursively compare the dumped signature with t= he most + recent one in a local build or sstate cache (ca= n be + used to find out why tasks re-run when that is = not + expected) + --revisions-changed Set the exit code depending on whether upstream + floating revisions have changed or not. + -b BUILDFILE, --buildfile BUILDFILE + Execute tasks from a specific .bb recipe direct= ly. + WARNING: Does not handle any dependencies from = other + recipes. + + Logging/output control options: -D, --debug Increase the debug level. You can specify this = more than once. -D sets the debug level to 1, where = only bb.debug(1, ...) messages are printed to stdout= ; -DD @@ -392,65 +499,47 @@ Following is the usage and syntax for BitBake:: -D only affects output to stdout. All debug mes= sages are written to ${T}/log.do_taskname, regardless= of the debug level. + -l DEBUG_DOMAINS, --log-domains DEBUG_DOMAINS + Show debug logging for the specified logging do= mains. + -v, --verbose Enable tracing of shell tasks (with 'set -x'). = Also + print bb.note(...) messages to stdout (in addit= ion to + writing them to ${T}/log.do_). -q, --quiet Output less log message data to the terminal. Y= ou can specify this more than once. - -n, --dry-run Don't execute, just go through the motions. - -S SIGNATURE_HANDLER, --dump-signatures=3DSIGNATURE_HANDLER - Dump out the signature construction information= , with - no task execution. The SIGNATURE_HANDLER parame= ter is - passed to the handler. Two common values are no= ne and - printdiff but the handler may define more/less.= none - means only dump the signature, printdiff means = compare - the dumped signature with the cached one. - -p, --parse-only Quit after parsing the BB recipes. - -s, --show-versions Show current and preferred versions of all reci= pes. - -e, --environment Show the global or per-recipe environment compl= ete - with information about where variables were - set/changed. - -g, --graphviz Save dependency tree information for the specif= ied - targets in the dot syntax. - -I EXTRA_ASSUME_PROVIDED, --ignore-deps=3DEXTRA_ASSUME_PROVIDED - Assume these dependencies don't exist and are a= lready - provided (equivalent to ASSUME_PROVIDED). Usefu= l to - make dependency graphs more appealing - -l DEBUG_DOMAINS, --log-domains=3DDEBUG_DOMAINS - Show debug logging for the specified logging do= mains - -P, --profile Profile the command and save reports. - -u UI, --ui=3DUI The user interface to use (knotty, ncurses, t= askexp or - teamcity - default knotty). - --token=3DXMLRPCTOKEN Specify the connection token to be used when - connecting to a remote server. - --revisions-changed Set the exit code depending on whether upstream - floating revisions have changed or not. - --server-only Run bitbake without a UI, only starting a serve= r - (cooker) process. - -B BIND, --bind=3DBIND The name/address for the bitbake xmlrpc serve= r to bind + -w WRITEEVENTLOG, --write-log WRITEEVENTLOG + Writes the event log of the build to a bitbake = event + json file. Use '' (empty string) to assign the = name + automatically. + + Server options: + -B BIND, --bind BIND The name/address for the bitbake xmlrpc server = to bind to. - -T SERVER_TIMEOUT, --idle-timeout=3DSERVER_TIMEOUT + -T SERVER_TIMEOUT, --idle-timeout SERVER_TIMEOUT Set timeout to unload bitbake server due to inactivity, set to -1 means no unload, default: Environment variable BB_SERVER_TIMEOUT. - --no-setscene Do not run any setscene tasks. sstate will be i= gnored - and everything needed, built. - --skip-setscene Skip setscene tasks if they would be executed. = Tasks - previously restored from sstate will be kept, u= nlike - --no-setscene - --setscene-only Only run setscene tasks, don't run any real tas= ks. - --remote-server=3DREMOTE_SERVER + --remote-server REMOTE_SERVER Connect to the specified server. -m, --kill-server Terminate any running bitbake server. + --token XMLRPCTOKEN Specify the connection token to be used when + connecting to a remote server. --observe-only Connect to a server as an observing-only client= . --status-only Check the status of the remote bitbake server. - -w WRITEEVENTLOG, --write-log=3DWRITEEVENTLOG - Writes the event log of the build to a bitbake = event - json file. Use '' (empty string) to assign the = name - automatically. - --runall=3DRUNALL Run the specified task for any recipe in the = taskgraph - of the specified target (even if it wouldn't ot= herwise - have run). - --runonly=3DRUNONLY Run only the specified task within the taskgr= aph of - the specified targets (and any task dependencie= s those - tasks may have). + --server-only Run bitbake without a UI, only starting a serve= r + (cooker) process. + + Configuration options: + -r PREFILE, --read PREFILE + Read the specified file before bitbake.conf. + -R POSTFILE, --postread POSTFILE + Read the specified file after bitbake.conf. + -I EXTRA_ASSUME_PROVIDED, --ignore-deps EXTRA_ASSUME_PROVIDED + Assume these dependencies don't exist and are a= lready + provided (equivalent to ASSUME_PROVIDED). Usefu= l to + make dependency graphs more appealing. + +.. + BitBake help output generated with "stty columns 80; bin/bitbake -h" =20 .. _bitbake-examples: =20 @@ -481,7 +570,7 @@ The following command runs the clean task on the ``foo.= bb`` recipe file:: =20 .. note:: =20 - The "-b" option explicitly does not handle recipe dependencies. Other + The ``-b`` option explicitly does not handle recipe dependencies. Other than for debugging purposes, it is instead recommended that you use the syntax presented in the next section. =20 @@ -496,7 +585,7 @@ dependencies, both for build-time and runtime. There mu= st be a way for you to express recipe preferences when multiple recipes provide the same functionality, or when there are multiple versions of a recipe. =20 -The ``bitbake`` command, when not using "--buildfile" or "-b" only +The ``bitbake`` command, when not using ``--buildfile`` or ``-b`` only accepts a "PROVIDES". You cannot provide anything else. By default, a recipe file generally "PROVIDES" its "packagename" as shown in the following example:: @@ -504,7 +593,7 @@ following example:: $ bitbake foo =20 This next example "PROVIDES" the -package name and also uses the "-c" option to tell BitBake to just +package name and also uses the ``-c`` option to tell BitBake to just execute the ``do_clean`` task:: =20 $ bitbake -c clean foo diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-library-fu= nctions.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-library-f= unctions.rst new file mode 100644 index 00000000..09e35394 --- /dev/null +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-library-functions= .rst @@ -0,0 +1,59 @@ +.. SPDX-License-Identifier: CC-BY-2.5 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Library Functions +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +| + +This chapter lists common library functions available under the ``lib/`` +directory in BitBake. + +These functions can be used in recipes or configuration files with +:ref:`inline-Python ` or :ref:`Python +` functions. + +Logging utilities +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Different logging utilities can be used from Python code in recipes or +configuration files. + +The strings passed below can be formatted with ``str.format()``, for examp= le:: + + bb.warn("Houston, we have a %s", "bit of a problem") + +Formatted string can also be used directly:: + + bb.error("%s, we have a %s" % ("Houston", "big problem")) + +Python f-strings may also be used:: + + h =3D "Houston" + bb.fatal(f"{h}, we have a critical problem") + +.. automodule:: bb + :members: + debug, + error, + erroronce, + fatal, + note, + plain, + verbnote, + warn, + warnonce, + +``bb.utils`` +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +.. automodule:: bb.utils + :members: + :exclude-members: + LogCatcher, + PrCtlError, + VersionStringException, + better_compile, + better_exec, diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.r= st b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst index 58975f4c..621aab25 100644 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst @@ -172,6 +172,26 @@ assignment, ``BAR`` expands to the literal string "${F= OO}" as long as =20 BAR =3D "${FOO}" =20 +Immediate variable expansion (:=3D) +--------------------------------- + +The ":=3D" operator results in a variable's contents being expanded +immediately, rather than when the variable is actually used:: + + T =3D "123" + A :=3D "test ${T}" + T =3D "456" + B :=3D "${T} ${C}" + C =3D "cval" + C :=3D "${C}append" + +In this example, ``A`` contains "test 123", even though the final value +of :term:`T` is "456". The variable :term:`B` will end up containing "456 +cvalappend". This is because references to undefined variables are +preserved as is during (immediate)expansion. This is in contrast to GNU +Make, where undefined variables expand to nothing. The variable ``C`` +contains "cvalappend" since ``${C}`` immediately expands to "cval". + Setting a default value (?=3D) ---------------------------- =20 @@ -235,26 +255,6 @@ After parsing we will have:: =20 W =3D "xy" =20 -Immediate variable expansion (:=3D) ---------------------------------- - -The ":=3D" operator results in a variable's contents being expanded -immediately, rather than when the variable is actually used:: - - T =3D "123" - A :=3D "test ${T}" - T =3D "456" - B :=3D "${T} ${C}" - C =3D "cval" - C :=3D "${C}append" - -In this example, ``A`` contains "test 123", even though the final value -of :term:`T` is "456". The variable :term:`B` will end up containing "456 -cvalappend". This is because references to undefined variables are -preserved as is during (immediate)expansion. This is in contrast to GNU -Make, where undefined variables expand to nothing. The variable ``C`` -contains "cvalappend" since ``${C}`` immediately expands to "cval". - .. _appending-and-prepending: =20 Appending (+=3D) and prepending (=3D+) With Spaces @@ -608,7 +608,7 @@ variable. =20 DEPENDS =3D "glibc ncurses" OVERRIDES =3D "machine:local" - DEPENDS:append:machine =3D "libmad" + DEPENDS:append:machine =3D " libmad" =20 In this example, :term:`DEPENDS` becomes "glibc ncurses libmad". =20 @@ -617,8 +617,8 @@ variable. ``KERNEL_FEATURES`` variable based on the architecture:: =20 KERNEL_FEATURES:append =3D " ${KERNEL_EXTRA_FEATURES}" - KERNEL_FEATURES:append:qemux86=3D" cfg/sound.scc cfg/paravirt_kvm.sc= c" - KERNEL_FEATURES:append:qemux86-64=3D" cfg/sound.scc cfg/paravirt_kvm= .scc" + KERNEL_FEATURES:append:qemux86 =3D " cfg/sound.scc cfg/paravirt_kvm.= scc" + KERNEL_FEATURES:append:qemux86-64 =3D " cfg/sound.scc cfg/paravirt_k= vm.scc" =20 - *Setting a Variable for a Single Task:* BitBake supports setting a variable just for the duration of a single task. Here is an example:: @@ -753,23 +753,17 @@ share the task. =20 This section presents the mechanisms BitBake provides to allow you to share functionality between recipes. Specifically, the mechanisms -include ``include``, ``inherit``, :term:`INHERIT`, and ``require`` -directives. - -Locating Include and Class Files --------------------------------- - -BitBake uses the :term:`BBPATH` variable to locate -needed include and class files. Additionally, BitBake searches the -current directory for ``include`` and ``require`` directives. - -.. note:: - - The BBPATH variable is analogous to the environment variable PATH . - -In order for include and class files to be found by BitBake, they need -to be located in a "classes" subdirectory that can be found in -:term:`BBPATH`. +include :ref:`inherit`, +:ref:`inherit_defer`, +:ref:`include`, +:ref:`include_all`, +:ref:`require` and +:ref:`INHERIT` +directives. There is also a higher-level abstraction called +`configuration fragments` that is enabled with the +:ref:`bitbake-user-manual/bitbake-user-manual-metadata:\`\`addfragments\`\= ` directive`. + +.. _ref-bitbake-user-manual-metadata-inherit: =20 ``inherit`` Directive --------------------- @@ -809,65 +803,131 @@ An advantage with the inherit directive as compared = to both the :ref:`include ` and :ref:`require ` directives is that you can inherit class files conditionally. You can accomplish this by using a variable expression after the ``inherit`` -statement. Here is an example:: +statement. =20 - inherit ${VARNAME} +For inheriting classes conditionally, using the :ref:`inherit_defer +` directive is advised as +:ref:`inherit_defer ` is +evaluated at the end of parsing. =20 -If ``VARNAME`` is -going to be set, it needs to be set before the ``inherit`` statement is -parsed. One way to achieve a conditional inherit in this case is to use -overrides:: +.. _ref-bitbake-user-manual-metadata-inherit-defer: =20 - VARIABLE =3D "" - VARIABLE:someoverride =3D "myclass" +``inherit_defer`` Directive +--------------------------- =20 -Another method is by using anonymous Python. Here is an example:: +The :ref:`inherit_defer ` +directive works like the :ref:`inherit +` directive, except that it is o= nly +evaluated at the end of parsing. Its usage is recommended when a condition= al +expression is used. =20 - python () { - if condition =3D=3D value: - d.setVar('VARIABLE', 'myclass') - else: - d.setVar('VARIABLE', '') - } +This allows conditional expressions to be evaluated "late", meaning change= s to +the variable after the line is parsed will take effect. With the :ref:`inh= erit +` directive this is not the case= . + +Here is an example:: + + inherit_defer ${VARNAME} + +One way to achieve a conditional inherit in this case is to use +overrides:: =20 -Alternatively, you could use an in-line Python expression in the + VARNAME =3D "" + VARNAME:someoverride =3D "myclass" + +:ref:`inherit_defer ` +defers the evaluation of ``${VARNAME}`` until the end of +parsing. Assuming ``someoverride`` is in :term:`OVERRIDES`, ``${VARNAME}`` +expands to ``myclass``, which is then inherited. + +Alternatively, you could use an inline Python expression in the following form:: =20 - inherit ${@'classname' if condition else ''} - inherit ${@functionname(params)} + inherit_defer ${@'classname' if condition else ''} + +Or:: + + inherit_defer ${@bb.utils.contains('VARIABLE', 'something', 'classname'= , '', d)} =20 In all cases, if the expression evaluates to an empty string, the statement does not trigger a syntax error because it becomes a no-op. =20 +See also :term:`BB_DEFER_BBCLASSES` for automatically promoting classes +``inherit`` calls to ``inherit_defer``. + +.. _ref-include-directive: + ``include`` Directive --------------------- =20 -BitBake understands the ``include`` directive. This directive causes -BitBake to parse whatever file you specify, and to insert that file at -that location. The directive is much like its equivalent in Make except -that if the path specified on the include line is a relative path, -BitBake locates the first file it can find within :term:`BBPATH`. +The ``include`` directive causes BitBake to parse a given file, +and to include that file's contents at the location of the +``include`` statement. This directive is similar to its equivalent +in Make, except that if the path specified on the BitBake ``include`` +line is a relative path, BitBake will search for it on the path designated +by :term:`BBPATH` and will include *only the first matching file*. =20 -The include directive is a more generic method of including +The ``include`` directive is a more generic method of including functionality as compared to the :ref:`inherit ` directive, which is restricted to class (i.e. ``.bbclass``) files. The -include directive is applicable for any other kind of shared or +``include`` directive is applicable for any other kind of shared or encapsulated functionality or configuration that does not suit a ``.bbclass`` file. =20 -As an example, suppose you needed a recipe to include some self-test -definitions:: +For example, if you needed a recipe to include some self-test definitions, +you might write:: =20 include test_defs.inc =20 +The ``include`` directive does not produce an error if the specified file +cannot be found. If the included file *must* exist, then you should use +use :ref:`require ` instead, which will generate an err= or +if the file cannot be found. + +.. note:: + + Note well that the ``include`` directive will include the first matchin= g + file and nothing further (which is almost always the behaviour you want= ). + If you need to include all matching files, you need to use the + ``include_all`` directive, explained below. + +.. _ref-include-all-directive: + +``include_all`` Directive +------------------------- + +The ``include_all`` directive works like the :ref:`include +` +directive but will include *all* of the files that match the specified pat= h in +the enabled layers (layers part of :term:`BBLAYERS`). + +Note that only :term:`BBPATH` will be searched, the parent directory of th= e file +with the ``include_all`` directive will not be searched (unlike for the +``include`` directive). + .. note:: =20 - The include directive does not produce an error when the file cannot be - found. Consequently, it is recommended that if the file you are includ= ing is - expected to exist, you should use :ref:`require ` in= stead - of include . Doing so makes sure that an error is produced if the file = cannot - be found. + This behaviour is rarely what you want in normal operation, and should + be reserved for only those situations when you explicitly want to parse + and include all matching files found across all layers, as the followin= g + example shows. + +As a realistic example of this directive, imagine that all of your active +layers contain a file ``conf/distro/include/maintainers.inc``, containing +maintainer information for the recipes in that layer, and you wanted to +collect all of the content from all of those files across all of those lay= ers. +You could use the statement:: + + include_all conf/distro/include/maintainers.inc + +In this case, BitBake will iterate through all of the directories in +the colon-separated :term:`BBPATH` (from left to right) and collect the +contents of all matching files, so you end up with the maintainer +information of all of your active layers, not just the first one. + +As the ``include_all`` directive uses the ``include`` directive in the +background, as with ``include``, no error is produced if no files are matc= hed. =20 .. _require-inclusion: =20 @@ -933,6 +993,197 @@ the ``autotools`` and ``pkgconfig`` classes:: =20 INHERIT +=3D "autotools pkgconfig" =20 +``addfragments`` Directive +-------------------------- + +This directive allows fine-tuning local configurations with configuration +snippets contained in layers in a structured, controlled way. Typically it= would +go into ``bitbake.conf``, for example:: + + addfragments conf/fragments OE_FRAGMENTS OE_FRAGMENTS_METADATA_VARS OE_= FRAGMENTS_BUILTIN + +``addfragments`` takes four parameters: + +- path prefix for fragment files inside the layer file tree that bitbake + uses to construct full paths to the fragment files + +- name of variable that holds the list of enabled fragments in an + active build + +- name of variable that contains a list of variable names containing + fragment-specific metadata (such as descriptions) + +- name of variable that contains definitions for built-in fragments + +This allows listing enabled configuration fragments in ``OE_FRAGMENTS`` +variable like this:: + + OE_FRAGMENTS =3D "core/domain/somefragment core/someotherfragment anoth= erlayer/anotherdomain/anotherfragment" + +Fragment names listed in this variable must be prefixed by the layer name +where a fragment file is located, defined by :term:`BBFILE_COLLECTIONS` in= ``layer.conf``. + +The implementation then expands this list into +:ref:`require ` +directives with full paths to respective layers:: + + require /path/to/core-layer/conf/fragments/domain/somefragment.conf + require /path/to/core-layer/conf/fragments/someotherfragment.conf + require /path/to/another-layer/conf/fragments/anotherdomain/anotherfrag= ment.conf + +The variable containing a list of fragment metadata variables could look l= ike this:: + + OE_FRAGMENTS_METADATA_VARS =3D "BB_CONF_FRAGMENT_SUMMARY BB_CONF_FRAGME= NT_DESCRIPTION" + +The implementation will add a flag containing the fragment name to each of= those variables +when parsing fragments, so that the variables are namespaced by fragment n= ame, and do not override +each other when several fragments are enabled. + +The variable containing a built-in fragment definitions could look like th= is:: + + OE_FRAGMENTS_BUILTIN =3D "someprefix:SOMEVARIABLE anotherprefix:ANOTHER= VARIABLE" + +and then if 'someprefix/somevalue' is added to the variable that holds the= list +of enabled fragments:: + + OE_FRAGMENTS =3D "... someprefix/somevalue" + +bitbake will treat that as direct value assignment in its configuration:: + + SOMEVARIABLE =3D "somevalue" + +Locating Include Files +---------------------- + +BitBake first searches the current directory for needed include files for +:ref:`include ` and :ref:`require ` +directives. +BitBake also uses the :term:`BBPATH` variable. + +.. note:: + + The :term:`BBPATH` variable is analogous to the environment variable ``= PATH``. + +For these two directives, BitBake includes the first file it finds. + +Let's consider the following statement called from a recipe file located i= n +``/layers/meta-custom2/recipes-example/example/example_0.1.bb``:: + + require myfile.inc + +Where ``myfile.inc`` is located in ``/layers/meta-custom2/recipes-example/= example``. + +And let's assume that the value of :term:`BBPATH` is +``/layers/meta-custom1:/layers/meta-custom2``. Then BitBake will try to fi= nd +``myfile.inc`` in this order:: + + /layers/meta-custom2/recipes-example/example/myfile.inc + /layers/meta-custom1/myfile.inc + /layers/meta-custom2/myfile.inc + +In this case the first path of the list matches and BitBake includes this = file +in ``example_0.1.bb``. + +Another common example would be:: + + require recipes-other/other/otherfile.inc + +Where ``otherfile.inc`` is located in +``/layers/meta-custom1/recipes-other/other/``. + +In this case, the following paths would be searched:: + + /layers/meta-custom2/recipes-example/example/recipes-other/other/otherf= ile.inc + /layers/meta-custom1/recipes-other/other/otherfile.inc + /layers/meta-custom2/recipes-other/other/otherfile.inc + +This time, the second item of this list would be matched. + +Note that the first path is based on the location of the file with the +``require`` (or ``include``) directive. Imagine there's a +``/layers/meta-custom2/recipes-bbappend/example/example_0.1.bbappend`` wit= h:: + + require myappend.inc + +In this case, the following paths would be searched:: + + /layers/meta-custom2/recipes-bbappend/example/myappend.inc + /layers/meta-custom1/myappend.inc + /layers/meta-custom2/myappend.inc + +.. note:: + + In the above examples, the exact same search order applies for the + :ref:`include ` directive. + +It is also possible to include *all* occurences of a file with the same na= me +with the :ref:`include_all ` directive. + +Let's consider the following statement called from a recipe file located i= n +``/layers/meta-custom2/recipes-example/example/exampleall_0.1.bb``:: + + include_all all.inc + +Where multiple ``all.inc`` are in located in ``/layers/meta-custom2`` and +``/layers/meta-custom1``. + +And let's assume that the value of :term:`BBPATH` is +``/layers/meta-custom1:/layers/meta-custom2``. Then BitBake will try to in= clude +all ``all.inc`` in this order:: + + /layers/meta-custom1/all.inc + /layers/meta-custom2/all.inc + +In this case the first ``/layers/meta-custom1/all.inc`` would be included,= and +then ``/layers/meta-custom2/all.inc`` if both are found. + +.. note:: + + The same logic as for the :ref:`ref-include-directive` applies, except = that + the path relative to the file where the directive is specified is not + searched when using the ``include_all`` directive. + +Locating Class Files +-------------------- + +Like include files, class files are located using the :term:`BBPATH` varia= ble. +The classes can be included in the ``classes-recipe``, ``classes-global`` = and +``classes`` directories, as explained in the +:ref:`bitbake-user-manual/bitbake-user-manual-intro:Class types` section o= f the +BitBake User Manual. Like for the :ref:`include ` a= nd +:ref:`require ` directives, BitBake stops and inherits = the +first class that it finds. + +For classes inherited with the :ref:`inherit +` directive, BitBake will try to +locate the class under each ``classes-recipe`` directory for each path in +:term:`BBPATH`, and then do the same for each ``classes`` directory for ea= ch +path in :term:`BBPATH`. + +For example, if the value :term:`BBPATH` is +``/layers/meta-custom1:/layers/meta-custom2`` then the ``hello`` class +would be searched in this order:: + + /layers/meta-custom1/classes-recipe/hello.bbclass + /layers/meta-custom2/classes-recipe/hello.bbclass + /layers/meta-custom1/classes/hello.bbclass + /layers/meta-custom2/classes/hello.bbclass + +.. note:: + + Note that the order of the list above does not depend on where the clas= s is + inherited from. + +Likewise, for classes inherited with the :term:`INHERIT` variable, the +``classes-global`` directory is searched first, and the ``classes`` direct= ory is +searched second. Taking the above example, this would result in the follow= ing +list:: + + /layers/meta-custom1/classes-global/hello.bbclass + /layers/meta-custom2/classes-global/hello.bbclass + /layers/meta-custom1/classes/hello.bbclass + /layers/meta-custom2/classes/hello.bbclass + Functions =3D=3D=3D=3D=3D=3D=3D=3D=3D =20 @@ -972,7 +1223,9 @@ When you create these types of functions in your recipe or class files, you need to follow the shell programming rules. The scripts are executed by ``/bin/sh``, which may not be a bash shell but might be something such as ``dash``. You should not use -Bash-specific script (bashisms). +Bash-specific script (bashisms), such as ``[[`` (use ``[`` instead). The B= itBake +shell parser also has further +:ref:`limitations `. =20 Overrides and override-style operators like ``:append`` and ``:prepend`` can also be applied to shell functions. Most commonly, this application @@ -1013,6 +1266,20 @@ Running ``do_foo`` prints the following:: You can use the ``bitbake -e=C2=A0recipename`` command to view the final assembled function after all overrides have been applied. =20 +Limitations +^^^^^^^^^^^ + +Shell functions must be parsable by BitBake, which cannot handle all +syntax supported by POSIX shells. In particular: + +#. Arithmetic expansion is not supported. This means you cannot do: + + .. code-block:: shell + + VAR=3D$(( i + 1 )) + + See https://bugzilla.yoctoproject.org/show_bug.cgi?id=3D11314. + BitBake-Style Python Functions ------------------------------ =20 @@ -1303,8 +1570,8 @@ the task and other tasks. Here is an example that sho= ws how to define a task and declare some dependencies:: =20 python do_printdate () { - import time - print time.strftime('%Y%m%d', time.gmtime()) + import datetime + bb.plain('Date: %s' % (datetime.date.today())) } addtask printdate after do_fetch before do_build =20 @@ -1496,22 +1763,22 @@ functionality of the task: directory listed is used as the current working directory for the task. =20 -- ``[file-checksums]``: Controls the file dependencies for a task. The - baseline file list is the set of files associated with - :term:`SRC_URI`. May be used to set additional dependencies on - files not associated with :term:`SRC_URI`. +- ``[file-checksums]``: Controls the file dependencies for a task. The + baseline file list is the set of files associated with + :term:`SRC_URI`. May be used to set additional dependencies on + files not associated with :term:`SRC_URI`. =20 - The value set to the list is a file-boolean pair where the first - value is the file name and the second is whether or not it - physically exists on the filesystem. :: + The value set to the list is a file-boolean pair where the first + value is the file name and the second is whether or not it + physically exists on the filesystem. :: =20 - do_configure[file-checksums] +=3D "${MY_DIRPATH}/my-file.txt:True" + do_configure[file-checksums] +=3D "${MY_DIRPATH}/my-file.txt:True" =20 - It is important to record any paths which the task looked at and - which didn't exist. This means that if these do exist at a later - time, the task can be rerun with the new additional files. The - "exists" True or False value after the path allows this to be - handled. + It is important to record any paths which the task looked at and + which didn't exist. This means that if these do exist at a later + time, the task can be rerun with the new additional files. The + "exists" True or False value after the path allows this to be + handled. =20 - ``[lockfiles]``: Specifies one or more lockfiles to lock while the task executes. Only one task may hold a lockfile, and any task that @@ -1524,6 +1791,9 @@ functionality of the task: shouldn't access the network outside of ``do_fetch`` as it usually undermines fetcher source mirroring, image and licence manifests, softw= are auditing and supply chain security. + Tasks without the network flag will run with PR_SET_NO_NEW_PRIVS, + which prevents executables from gaining privileges through + file capabilities. =20 - ``[noexec]``: When set to "1", marks the task as being empty, with no execution required. You can use the ``[noexec]`` flag to set up @@ -1635,8 +1905,9 @@ and the content of the :term:`FILE` variable:: print("The name of the Event is %s" % getName(e)) print("The file we run for is %s" % d.getVar('FILE')) } - myclass_eventhandler[eventmask] =3D "bb.event.BuildStarted - bb.event.BuildCompleted" + myclass_eventhandler[eventmask] =3D "bb.event.BuildStarted \ + bb.event.BuildCompleted \ + " =20 In the previous example, an eventmask has been set so that the handler only sees the "BuildStarted" and @@ -1972,11 +2243,8 @@ access. Here is a list of available operations: Other Functions --------------- =20 -You can find many other functions that can be called from Python by -looking at the source code of the ``bb`` module, which is in -``bitbake/lib/bb``. For example, ``bitbake/lib/bb/utils.py`` includes -the commonly used functions ``bb.utils.contains()`` and -``bb.utils.mkdirhier()``, which come with docstrings. +Other functions are documented in the +:doc:`/bitbake-user-manual/bitbake-user-manual-library-functions` document= . =20 Extending Python Library Code ----------------------------- @@ -2016,12 +2284,12 @@ OpenEmbedded metadata-based example. These checksums are stored in :term:`STAMP`. You can examine the checksums using the following BitBake command:: =20 - $ bitbake-dumpsigs + $ bitbake-dumpsig =20 This command returns the signature data in a readable format that allows you to examine the inputs used when the OpenEmbedded build system generates signatures. For example, using -``bitbake-dumpsigs`` allows you to examine the ``do_compile`` task's +``bitbake-dumpsig`` allows you to examine the ``do_compile`` task's "sigdata" for a C application (e.g. ``bash``). Running the command also reveals that the "CC" variable is part of the inputs that are hashed. Any changes to this variable would invalidate the stamp and cause the diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variab= les.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables= .rst index 899e584f..fd75275b 100644 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst @@ -36,6 +36,14 @@ overview of their function and contents. when specified allows for the Git binary from the host to be used rather than building ``git-native``. =20 + :term:`AUTOREV` + This is a special variable used during fetching. When :term:`SRCREV`= is + set to the value of this variable, the latest revision from the vers= ion + controlled source code repository is used. + It should be set as follows:: + + SRCREV =3D "${AUTOREV}" + :term:`AZ_SAS` Azure Storage Shared Access Signature, when using the :ref:`Azure Storage fetcher ` @@ -108,6 +116,10 @@ overview of their function and contents. Specifies if SSL certificates should be checked when fetching. The d= efault value is ``1`` and certificates are not checked if the value is set = to ``0``. =20 + :term:`BB_CMDLINE` + :term:`BB_CMDLINE` is an internal variable and is read-only. It capt= ures + the exact command line used for the current invocation of BitBake. + :term:`BB_HASH_CODEPARSER_VALS` Specifies values for variables to use when populating the codeparser= cache. This can be used selectively to set dummy values for variables to av= oid @@ -123,21 +135,24 @@ overview of their function and contents. Specifies the path to a log file into which BitBake's user interface writes output during the build. =20 + This variable's value remains persistent during the entire execution= of + the BitBake server. Changes to this variable will only take effect + after a server restart, which can be accomplished with the ``bitbake + --kill-server`` command. + + .. note:: + + For more details on how to make the BitBake server persistent, se= e the + ``BB_SERVER_TIMEOUT`` variable. + :term:`BB_CURRENTTASK` Contains the name of the currently running task. The name does not include the ``do_`` prefix. =20 - :term:`BB_DANGLINGAPPENDS_WARNONLY` - Defines how BitBake handles situations where an append file - (``.bbappend``) has no corresponding recipe file (``.bb``). This - condition often occurs when layers get out of sync (e.g. ``oe-core`` - bumps a recipe version and the old recipe no longer exists and the - other layer has not been updated to the new version of the recipe - yet). - - The default fatal behavior is safest because it is the sane reaction - given something is out of sync. It is important to realize when your - changes are no longer being applied. + :term:`BB_CURRENT_MC` + Contains the name of the current multiconfig a task is being run und= er. + The name is taken from the multiconfig configuration file (a file + ``mc1.conf`` would make this variable equal to ``mc1``). =20 :term:`BB_DEFAULT_TASK` The default task to use when none is specified (e.g. with the ``-c`` @@ -148,6 +163,25 @@ overview of their function and contents. The default umask to apply to tasks if specified and no task specifi= c umask flag is set. =20 + :term:`BB_DEFER_BBCLASSES` + The classes listed in this variable have their :ref:`inherit + ` calls automatically prom= oted + to deferred inherits. See :ref:`inherit_defer + ` for more informati= on on + deferred inherits. + + This means that if :term:`BB_DEFER_BBCLASSES` is set as follows:: + + BB_DEFER_BBCLASSES =3D "foo" + + The following statement:: + + inherit foo + + Will automatically be equal to calling:: + + inherit_defer foo + :term:`BB_DISKMON_DIRS` Monitors disk space and available inodes during the build and allows you to control the build based on these parameters. @@ -262,8 +296,8 @@ overview of their function and contents. Specifies the internal list of variables to allow through from the external environment into BitBake's datastore. If the value of this variable is not specified (which is the default), the following - list is used: :term:`BBPATH`, :term:`BB_PRESERVE_ENV`, - :term:`BB_ENV_PASSTHROUGH`, and :term:`BB_ENV_PASSTHROUGH_ADDITIONS`= . + list is used: :term:`BBPATH`, :term:`BB_PRESERVE_ENV`, and + :term:`BB_ENV_PASSTHROUGH_ADDITIONS`. =20 .. note:: =20 @@ -317,21 +351,42 @@ overview of their function and contents. =20 For example usage, see :term:`BB_GIT_SHALLOW`. =20 + :term:`BB_GIT_DEFAULT_DESTSUFFIX` + The default destination directory where the :ref:`Git fetcher + ` unpacks the source code. If this variable is not set,= the + source code is unpacked in a directory named "git". + :term:`BB_GIT_SHALLOW` Setting this variable to "1" enables the support for fetching, using= and generating mirror tarballs of `shallow git repositories `_. - The external `git-make-shallow `_ - script is used for shallow mirror tarball creation. + A git shallow fetch is used for shallow mirror tarball creation (i.e= . + ``git fetch --depth ...`` or + ``git fetch --shallow-exclude=3D ...``. =20 When :term:`BB_GIT_SHALLOW` is enabled, bitbake will attempt to fetc= h a shallow mirror tarball. If the shallow mirror tarball cannot be fetched, it = will try to fetch the full mirror tarball and use that. =20 - When a mirror tarball is not available, a full git clone will be per= formed - regardless of whether this variable is set or not. Support for shall= ow - clones is not currently implemented as git does not directly support - shallow cloning a particular git commit hash (it only supports cloni= ng - from a tag or branch reference). + This setting causes an initial shallow clone instead of an initial f= ull bare clone. + The amount of data transferred during the initial clone will be sign= ificantly reduced. + + However, every time the source revision (referenced in :term:`SRCREV= `) + changes, regardless of whether the cache within the download directo= ry + (defined by :term:`DL_DIR`) has been cleaned up or not, + the data transfer may be significantly higher because entirely + new shallow clones are required for each source revision change. + + Over time, numerous shallow clones may cumulatively transfer + the same amount of data as an initial full bare clone. + This is especially the case with very large repositories. + + Existing initial full bare clones, created without this setting, + will still be utilized. + + If the Git error "Server does not allow request for unadvertised obj= ect" + occurs, an initial full bare clone is fetched automatically. + This may happen if the Git server does not allow the request + or if the Git client has issues with this functionality. =20 See also :term:`BB_GIT_SHALLOW_DEPTH` and :term:`BB_GENERATE_SHALLOW_TARBALLS`. @@ -359,6 +414,23 @@ overview of their function and contents. =20 For example usage, see :term:`BB_GIT_SHALLOW`. =20 + :term:`BB_GIT_SHALLOW_EXTRA_REFS` + With :term:`BB_GIT_SHALLOW` enabled, by default, all unused refs (br= anches + and tags) are removed from the repository, as shallow processing sca= les + with the number of refs it has to process. + + :term:`BB_GIT_SHALLOW_EXTRA_REFS` allows to explicitly specify addit= ional + refs to keep. This is particularly useful for recipes with custom ch= eckout + processes, or whose Git-based versioning requires a tag be available= (i.e. + for ``git describe --tags``). The :term:`BB_GIT_SHALLOW_EXTRA_REFS` + variable is a space-separated list of refs, fully specified, and sup= ports + wildcards. + + Example usage:: + + BB_GIT_SHALLOW_EXTRA_REFS =3D "refs/tags/v1.0" + BB_GIT_SHALLOW_EXTRA_REFS +=3D "refs/heads/*" + :term:`BB_GLOBAL_PYMODULES` Specifies the list of Python modules to place in the global namespac= e. It is intended that only the core layer should set this and it is me= ant @@ -414,6 +486,29 @@ overview of their function and contents. equivalences that correspond to Share State caches that are only available on specific clients. =20 + :term:`BB_HASHSERVE_DB_DIR` + When :term:`BB_HASHSERVE` is set to ``auto``, bitbake will use + a private location inside a particular build directory for the sqlit= e + database file that holds hash equivalency data. + + This variable allows using a different path, which can be shared + between multiple build directories and bitbake instances + that operate on them. This enables using a common ``${SSTATE_DIR}`` + together with common hash equivalency data for local builds, without= having to + separately manage a hash equivalency server. + + .. note:: + + This variable cannot be set to a NFS mount and bitbake will error= out then. + The reason is that NFS implementations can have file locking issu= es, which + can cause data loss and corruption when there are multiple writer= s operating + on a file at the same time as explained in https://sqlite.org/faq= .html#q5 + + If you'd like to share hash equivalency data between multiple com= puters, you + need to set up a hash equivalency server separately and point :te= rm:`BB_HASHSERVE` + to it. See https://docs.yoctoproject.org/dev-manual/hashequivserv= er.html for + additional information. + :term:`BB_HASHSERVE_UPSTREAM` Specifies an upstream Hash Equivalence server. =20 @@ -424,7 +519,7 @@ overview of their function and contents. =20 Example usage:: =20 - BB_HASHSERVE_UPSTREAM =3D "hashserv.yocto.io:8687" + BB_HASHSERVE_UPSTREAM =3D "hashserv.yoctoproject.org:8686" =20 :term:`BB_INVALIDCONF` Used in combination with the ``ConfigParsed`` event to trigger @@ -525,11 +620,32 @@ overview of their function and contents. version 4.20 expose under ``/proc/pressure``. The threshold represen= ts the difference in "total" pressure from the previous second. The minimum value is 1.0 (extremely slow builds) and the maximum is - 1000000 (a pressure value unlikely to ever be reached). + 1000000 (a pressure value unlikely to ever be reached). See + https://docs.kernel.org/accounting/psi.html for more information. + + A default value to limit the CPU pressure to be set in ``conf/local.= conf`` + could be:: + + BB_PRESSURE_MAX_CPU =3D "15000" + + Multiple values should be tested on the build host to determine what= suits + best, depending on the need for performance versus load average duri= ng + the build. + + .. note:: + + You can track the pressure information while a build is running w= ith: + + .. code-block:: console =20 - This threshold can be set in ``conf/local.conf`` as:: + $ tail -F tmp/log/cooker//console-latest.log | grep "= Pressure status changed" + NOTE: Pressure status changed to CPU: True, IO: False, Mem: Fa= lse (CPU: 1105.9/2.0, IO: 0.0/2.0, Mem: 0.0/2.0) - using 1/64 bitbake threa= ds + ... =20 - BB_PRESSURE_MAX_CPU =3D "500" + If these messages are printed a lot, it means that the :term:`BB_= PRESSURE_MAX_CPU` + should be increased to a reasonable value for limiting the CPU pr= essure + on the system. Monitor the varying value after ``CPU:`` above to = set a + sensible value. =20 :term:`BB_PRESSURE_MAX_IO` Specifies a maximum I/O pressure threshold, above which BitBake's @@ -541,14 +657,38 @@ overview of their function and contents. version 4.20 expose under ``/proc/pressure``. The threshold represen= ts the difference in "total" pressure from the previous second. The minimum value is 1.0 (extremely slow builds) and the maximum is - 1000000 (a pressure value unlikely to ever be reached). + 1000000 (a pressure value unlikely to ever be reached). See + https://docs.kernel.org/accounting/psi.html for more information. =20 At this point in time, experiments show that IO pressure tends to be short-lived and regulating just the CPU with :term:`BB_PRESSURE_MAX_CPU` can help to reduce it. =20 - :term:`BB_PRESSURE_MAX_MEMORY` + A default value to limit the IO pressure to be set in ``conf/local.c= onf`` + could be:: + + BB_PRESSURE_MAX_IO =3D "15000" + + Multiple values should be tested on the build host to determine what= suits + best, depending on the need for performance versus I/O usage during = the + build. + + .. note:: + + You can track the pressure information while a build is running w= ith: + + .. code-block:: console =20 + $ tail -F tmp/log/cooker//console-latest.log | grep "= Pressure status changed" + NOTE: Pressure status changed to CPU: None, IO: True, Mem: Fal= se (CPU: 2236.0/None, IO: 153.6/2.0, Mem: 0.0/2.0) - using 19/64 bitbake th= reads + ... + + If these messages are printed a lot, it means that the :term:`BB_= PRESSURE_MAX_IO`=20 + should be increased to a reasonable value for limiting the I/O pr= essure + on the system. Monitor the varying value after ``IO:`` above to s= et a + sensible value. + + :term:`BB_PRESSURE_MAX_MEMORY` Specifies a maximum memory pressure threshold, above which BitBake's scheduler will not start new tasks (providing there is at least one active task). If no value is set, memory pressure is not @@ -558,7 +698,8 @@ overview of their function and contents. version 4.20 expose under ``/proc/pressure``. The threshold represen= ts the difference in "total" pressure from the previous second. The minimum value is 1.0 (extremely slow builds) and the maximum is - 1000000 (a pressure value unlikely to ever be reached). + 1000000 (a pressure value unlikely to ever be reached). See + https://docs.kernel.org/accounting/psi.html for more information. =20 Memory pressure is experienced when time is spent swapping, refaulting pages from the page cache or performing direct reclaim. @@ -566,6 +707,30 @@ overview of their function and contents. might be useful as a last resort to prevent OOM errors if they are occurring during builds. =20 + A default value to limit the memory pressure to be set in + ``conf/local.conf`` could be:: + + BB_PRESSURE_MAX_MEMORY =3D "15000" + + Multiple values should be tested on the build host to determine what= suits + best, depending on the need for performance versus memory consumptio= n + during the build. + + .. note:: + + You can track the pressure information while a build is running w= ith: + + .. code-block:: console + + $ tail -F tmp/log/cooker//console-latest.log | grep "= Pressure status changed" + NOTE: Pressure status changed to CPU: None, IO: False, Mem: Tr= ue (CPU: 29.5/None, IO: 0.0/2.0, Mem: 2553.3/2.0) - using 17/64 bitbake thr= eads + ... + + If these messages are printed a lot, it means that the :term:`BB_= PRESSURE_MAX_MEMORY` + should be increased to a reasonable value for limiting the memory + pressure on the system. Monitor the varying value after ``Mem:`` = above + to set a sensible value. + :term:`BB_RUNFMT` Specifies the name of the executable script files (i.e. run files) saved into ``${``\ :term:`T`\ ``}``. By default, the @@ -680,11 +845,11 @@ overview of their function and contents. .. note:: =20 In order for your I/O priority settings to take effect, you need = the - Completely Fair Queuing (CFQ) Scheduler selected for the backing = block + Budget Fair Queuing (BFQ) Scheduler selected for the backing bloc= k device. To select the scheduler, use the following command form w= here device is the device (e.g. sda, sdb, and so forth):: =20 - $ sudo sh -c "echo cfq > /sys/block/device/queu/scheduler" + $ sudo sh -c "echo bfq > /sys/block/device/queue/scheduler" =20 :term:`BB_TASK_NICE_LEVEL` Allows specific tasks to change their priority (i.e. nice level). @@ -699,6 +864,12 @@ overview of their function and contents. Within an executing task, this variable holds the hash of the task a= s returned by the currently enabled signature generator. =20 + :term:`BB_USE_HOME_NPMRC` + Controls whether or not BitBake uses the user's .npmrc file within t= heir + home directory within the npm fetcher. This can be used for authenti= cation + of private NPM registries, among other uses. This is turned off by d= efault + and requires the user to explicitly set it to "1" to enable. + :term:`BB_VERBOSE_LOGS` Controls how verbose BitBake is during builds. If set, shell scripts echo commands and shell script output appears on standard out @@ -766,6 +937,10 @@ overview of their function and contents. :term:`BBFILE_PRIORITY` Assigns the priority for recipe files in each layer. =20 + This variable is used in the ``conf/layer.conf`` file and must be + suffixed with a `_` followed by the name of the specific layer (e.g. + ``BBFILE_PRIORITY_emenlow``). Colon as separator is not supported. + This variable is useful in situations where the same recipe appears in more than one layer. Setting this variable allows you to prioritize a layer against other layers that contain the same recipe @@ -780,7 +955,7 @@ overview of their function and contents. higher precedence. For example, the value 6 has a higher precedence than the value 5. If not specified, the :term:`BBFILE_PRIORITY` vari= able is set based on layer dependencies (see the :term:`LAYERDEPENDS` var= iable - for more information. The default priority, if unspecified for a + for more information). The default priority, if unspecified for a layer with no dependencies, is the lowest defined priority + 1 (or 1 if no priorities are defined). =20 @@ -884,28 +1059,101 @@ overview of their function and contents. compiler. Consequently, the syntax follows Python's Regular Expression (re) syntax. The expressions are compared against the ful= l paths to the files. For complete syntax information, see Python's - documentation at http://docs.python.org/3/library/re.html. + documentation at https://docs.python.org/3/library/re.html. =20 The following example uses a complete regular expression to tell - BitBake to ignore all recipe and recipe append files in the - ``meta-ti/recipes-misc/`` directory:: + BitBake to ignore all recipe and recipe append files in + ``recipes-bsp`` directory (recursively) of ``meta-ti-bsp``:: =20 - BBMASK =3D "meta-ti/recipes-misc/" + BBMASK =3D "${BBFILE_PATTERN_meta-ti-bsp}/recipes-bsp/" =20 If you want to mask out multiple directories or recipes, you can specify multiple regular expression fragments. This next example masks out multiple directories and individual recipes:: =20 - BBMASK +=3D "/meta-ti/recipes-misc/ meta-ti/recipes-ti/packagegro= up/" - BBMASK +=3D "/meta-oe/recipes-support/" - BBMASK +=3D "/meta-foo/.*/openldap" - BBMASK +=3D "opencv.*\.bbappend" - BBMASK +=3D "lzma" + BBMASK +=3D "${BBFILE_PATTERN_meta-ti-bsp}/recipes-graphics/libga= l/" + BBMASK +=3D "${BBFILE_PATTERN_openembedded-layer}/recipes-support= /" + BBMASK +=3D "${BBFILE_PATTERN_openembedded-layer}/.*/openldap" + BBMASK +=3D "${BBFILE_PATTERN_meta-ti-bsp}/.*/optee.*\.bbappend" + + This masks: + + - everything under the ``recipes-graphics/libgal/`` directory from + ``meta-ti-bsp``, + - everything under the ``recipes-support/`` directory in ``meta-oe``= , + - everything under a directory whose name starts with ``openldap``, = and + every file with the same naming scheme, in ``meta-oe`` at any dire= ctory + depth > 1 (e.g. in ``meta-oe``, ``recipes-foo/openldap-stuff/`` or + ``recipes-bar/baz/openldap_0.1.bb`` but not ``openldap/``), + - every append file whose name starts with ``optee`` in ``meta-ti-bs= p`` at + any directory depth > 1 (e.g. ``optee/optee-examples_%.bbappend`` = and + ``recipes-security/optee/optee-client_%.bbappend``). =20 .. note:: =20 - When specifying a directory name, use the trailing slash characte= r - to ensure you match just that directory name. + Because these are complete regular expressions, if you want to ma= tch a + directory and not a file, you must end the expression with a trai= ling + slash. That is:: + + BBMASK +=3D "${BBFILE_PATTERN_meta-ti-bsp}/recipes-graphics/li= bgal/" + + Will match anything under ``recipes-graphics/ligbal/`` directory = of + ``meta-ti-bsp``. And:: + + BBMASK +=3D "${BBFILE_PATTERN_meta-ti-bsp}/recipes-graphics/li= bgal" + + Will match in ``meta-ti-bsp`` any file prefixed with ``libgal`` i= n + ``recipes-graphics/`` and any directory (recursively; and its + recipes and recipe append files regardless how they are named) pr= efixed + with ``libgal`` in ``recipes-graphics/``. That is, provided your = layers + are available at ``/bitbake-builds/poky-master/layers/``, it'll m= atch:: + + /bitbake-builds/poky-master/layers/meta-ti/meta-ti-bsp/recipes= -graphics/libgal.bb + /bitbake-builds/poky-master/layers/meta-ti/meta-ti-bsp/recipes= -graphics/libgal_%.bbappend + /bitbake-builds/poky-master/layers/meta-ti/meta-ti-bsp/recipes= -graphics/libgal-foo/foo.bb + /bitbake-builds/poky-master/layers/meta-ti/meta-ti-bsp/recipes= -graphics/libgal-foo/foo/bz.bbappend + /bitbake-builds/poky-master/layers/meta-ti/meta-ti-bsp/recipes= -graphics/libgal/bar.bb + + .. note:: + + Because these are complete regular expressions, failing to start = the + pattern with a ``^`` sign (which is usually the first character i= n + :term:`BBFILE_PATTERN`) means it can match *any* portion of a pat= h. + Take the following as an example:: + + BBMASK =3D "recipes-graphics/libgal/" + + This will match subdirectories and files in any path containing + ``recipes-graphics/libgal/``, meaning (considering your layers ar= e + available at ``/bitbake-builds/poky-master/layers/``):: + + /bitbake-builds/poky-master/layers/meta-ti/meta-ti-bsp/recipes= -graphics/libgal/ + /bitbake-builds/poky-master/layers/my-layer/foo-recipes-graphi= cs/libgal/ + + will be both matched. This may be a more relaxed way of matching + directories, recipes and recipe append files from any third party= layer + but is generally discouraged as it may be casting too wide of a n= et. + + .. note:: + + Because these are complete regular expressions, a leading slash d= oes + not mean the path is absolute. It simply forces the directory to = be + named exactly that. Take:: + + BBMASK =3D "recipes-graphics/libgal/" + + If you happen to have a directory ``foo-recipes-graphics/libgal/`= `, it + will be matched. + + Leading with a slash:: + + BBMASK =3D "/recipes-graphics/libgal/" + + makes sure that doesn't happen. However, this doesn't prevent mat= ching + a directory ``recipes-graphics/libgal/`` from another layer. + + Again, we highly recommend using :term:`BBFILE_PATTERN` in the + expressions to specify absolute paths specific to one layer. =20 :term:`BBMULTICONFIG` Enables BitBake to perform multiple configuration builds and lists @@ -963,18 +1211,10 @@ overview of their function and contents. A name assigned to the build. The name defaults to a datetime stamp of when the build was started but can be defined by the metadata. =20 - :term:`BZRDIR` - The directory in which files checked out of a Bazaar system are - stored. - :term:`CACHE` Specifies the directory BitBake uses to store a cache of the metadat= a so it does not need to be parsed every time BitBake is started. =20 - :term:`CVSDIR` - The directory in which files checked out under the CVS system are - stored. - :term:`DEFAULT_PREFERENCE` Specifies a weak bias for recipe selection priority. =20 @@ -1100,9 +1340,17 @@ overview of their function and contents. =20 :term:`INHERIT` Causes the named class or classes to be inherited globally. Anonymou= s - functions in the class or classes are not executed for the base - configuration and in each individual recipe. The OpenEmbedded build - system ignores changes to :term:`INHERIT` in individual recipes. + functions in the class or classes are executed in two disjoint situa= tions: + + - When only the + :ref:`base configuration ` + is parsed. For example as a result of the following BitBake invoc= ation:: + + $ bitbake -e + + - When recipes are parsed - then for each parsed recipe. + + BitBake ignores changes to :term:`INHERIT` in individual recipes. =20 For more information on :term:`INHERIT`, see the ":ref:`bitbake-user-manual/bitbake-user-manual-metadata:\`\`inherit\= `\` configuration directive`" @@ -1178,6 +1426,58 @@ overview of their function and contents. upstream source, and then locations specified by :term:`MIRRORS` in = that order. =20 + This variable has a specific syntax. Each line must contain a regula= r + expression matching the original source URL and a replacement URL fo= r it. + For example:: + + MIRRORS:prepend =3D " \ + git://git.openembedded.org/.* git:///mirrors/openembedded/BAS= ENAME \ + " + + In the above example: + + - The leftmost part means that any Git repository (``git://`` fetch= er) + whose URL starts with ``git.openembedded.org/`` will be matched. + + - The rightmost part is the replacement URL for the original reposi= tory, + which is hosted locally in the ``/mirrors/openembedded`` director= y. It + also uses the ``BASENAME`` keyword which is detailed below. + + The rightmost part of each line can contain keywords which are + automatically replaced by BitBake. These keywords can contain either= a + sub-part of the original URL or meta-information about it. + + For what follows, let's take the following URL as an example for the + original source (matched by the leftmost part of the line): + ``git://git.openembedded.org/project/repo.git``. + + A replacement for this URL can be created using the following keywor= ds: + + - ``TYPE``: the URL scheme or fetcher type. For the example above: = ``git``. + - ``HOST``: the URL host. For the example above: ``git.openembedded= .org``. + - ``PATH``: the URL path. For the example above: ``/project/repo.gi= t``. + - ``BASENAME``: the URL basename. For the example above: ``repo.git= ``. + - ``MIRRORNAME``: a translation of the host and path with common + characters substituted to dots (``.``). For the example above: + ``git.openembedded.org.project.repo.git``. + + The rightmost part of the line can also be a bare URL that uses no s= pecial + keywords (usually ``https://`` or ``file://``). For example:: + + MIRRORS:prepend =3D " \ + git://git.openembedded.org/.* file:///mirrors/openembedded/ \ + " + + In the above example, the fetcher will try to find a mirror tarball = in the + ``/mirrors/openembedded`` directory. The name of this tarball should= follow + the name of the tarball created by BitBake (for Git repositories, + the :term:`BB_GENERATE_MIRROR_TARBALLS` variable must be set to "1" = to + generate them). + + If we take ``git://git.openembedded.org/project/repo.git`` as an exa= mple, + the tarball would have to be named + ``git2_git.openembedded.org.project.repo.git.tar.gz``. + :term:`OVERRIDES` A colon-separated list that BitBake uses to control what variables a= re overridden after BitBake parses recipes and configuration files. @@ -1213,8 +1513,13 @@ overview of their function and contents. :term:`PERSISTENT_DIR` Specifies the directory BitBake uses to store data that should be preserved between builds. In particular, the data stored is the data - that uses BitBake's persistent data API and the data used by the PR - Server and PR Service. + that uses BitBake's persistent data API, the data used by the PR + Server and PR Service, and the default location of the Hash Equivale= nce + database (when :term:`BB_HASHSERVE` is set to ``auto``). + + This directory should not be shared between different builds. If you= need + to share the Hash Equivalence database, you should setup a Hash + Equivalence server instead. =20 :term:`PF` Specifies the recipe or package name and includes all version and @@ -1253,6 +1558,20 @@ overview of their function and contents. PREFERRED_PROVIDER_xxx =3D "yyy" PREFERRED_PROVIDER_aaa =3D "bbb" =20 + :term:`PREFERRED_RPROVIDER` + Determines which *recipe* should be given preference when + multiple packages declare runtime-providing (:term:`RPROVIDES`) + the same item. Some examples:: + + PREFERRED_RPROVIDER_initd-functions ?=3D "initscripts" + PREFERRED_RPROVIDER_virtual-libegl-icd ?=3D "mesa" + + The former will select as the package built by the `initscripts` + recipe declaring to be runtime-provider for `initd-functions`, + which is in this case `initscripts-functions`:: + + .../initscripts_1.0.bb:RPROVIDES:${PN}-functions =3D "initd-func= tions" + :term:`PREFERRED_VERSION` If there are multiple versions of a recipe available, this variable determines which version should be given preference. You must always @@ -1301,6 +1620,10 @@ overview of their function and contents. use ``file://`` URLs to point to local directories or network shares= as well. =20 + The syntax of the :term:`PREMIRRORS` variable is the same as + :term:`MIRRORS`. See the definition of the :term:`MIRRORS` for more + details. + :term:`PROVIDES` A list of aliases by which a particular recipe can be known. By default, a recipe's own :term:`PN` is implicitly already in its @@ -1460,14 +1783,8 @@ overview of their function and contents. =20 - ``az://``: Fetches files from an Azure Storage account using HTTP= S. =20 - - ``bzr://``: Fetches files from a Bazaar revision control - repository. - - ``ccrc://``: Fetches files from a ClearCase repository. =20 - - ``cvs://``: Fetches files from a CVS revision control - repository. - - ``file://``: Fetches files, which are usually files shipped with the Metadata, from the local machine. The path is relative to the :term:`FILESPATH` @@ -1502,8 +1819,11 @@ overview of their function and contents. =20 - ``npm://``: Fetches JavaScript modules from a registry. =20 - - ``osc://``: Fetches files from an OSC (OpenSUSE Build service) - revision control repository. + .. warning:: + + The NPM fetcher is currently disabled due to security concerns= . See + `355cd226e0720a9ed7683bb01c8c0a58eee03664 `__ + for more information. =20 - ``p4://``: Fetches files from a Perforce (``p4``) revision control repository. @@ -1522,7 +1842,8 @@ overview of their function and contents. =20 - ``name``: Specifies a name to be used for association with :term:`SRC_URI` checksums or :term:`SRCREV` when you have more th= an one - file or git repository specified in :term:`SRC_URI`. For example:= : + file or source control repository specified in :term:`SRC_URI`. + For example:: =20 SRC_URI =3D "git://example.com/foo.git;branch=3Dmain;name=3Dfi= rst \ git://example.com/bar.git;branch=3Dmain;name=3Dseco= nd \ @@ -1535,7 +1856,8 @@ overview of their function and contents. - ``subdir``: Places the file (or extracts its contents) into the specified subdirectory. This option is useful for unusual tarball= s or other archives that do not have their files already in a - subdirectory within the archive. + subdirectory within the archive. This path can be further modifie= d + by fetcher specific parameters. =20 - ``subpath``: Limits the checkout to a specific subpath of the tree when using the Git fetcher is used. @@ -1550,18 +1872,18 @@ overview of their function and contents. =20 :term:`SRCREV` The revision of the source code used to build the package. This - variable applies only when using Subversion, Git, Mercurial and - Bazaar. If you want to build a fixed revision and you want to avoid + variable applies only when using Subversion, Git and Mercurial. + If you want to build a fixed revision and you want to avoid performing a query on the remote repository every time BitBake parse= s your recipe, you should specify a :term:`SRCREV` that is a full revi= sion identifier and not just a tag. =20 :term:`SRCREV_FORMAT` - Helps construct valid :term:`SRCREV` values when + Helps construct a valid package version string when multiple source controlled URLs are used in :term:`SRC_URI`. =20 - The system needs help constructing these values under these + The system needs help constructing this value under these circumstances. Each component in the :term:`SRC_URI` is assigned a n= ame and these are referenced in the :term:`SRCREV_FORMAT` variable. Cons= ider an example with URLs named "machine" and "meta". In this case, @@ -1570,6 +1892,12 @@ overview of their function and contents. ``AUTOINC`` placeholder is added and if needed. And, this placeholde= r is placed at the start of the returned string. =20 + The :term:`SRCREV_FORMAT` can also take the form "_component2". + This assumes that there is a component in the :term:`SRC_URI` that d= oes not + have a name assigned. While this is not considered good practice, it= can be + usefull if a ``.bbappend`` file needs to extend the :term:`SRC_URI` = with + an additional repository. + :term:`STAMP` Specifies the base path used to create recipe stamp files. The path to an actual stamp file is constructed by evaluating this string and diff --git a/bitbake/doc/conf.py b/bitbake/doc/conf.py index fc2ee081..9a53e55c 100644 --- a/bitbake/doc/conf.py +++ b/bitbake/doc/conf.py @@ -6,18 +6,27 @@ =20 # -- Path setup ----------------------------------------------------------= ---- =20 -# If extensions (or modules to document with autodoc) are in another direc= tory, -# add these directories to sys.path here. If the directory is relative to = the -# documentation root, use os.path.abspath to make it absolute, like shown = here. -# -# import os -# import sys -# sys.path.insert(0, os.path.abspath('.')) - -import sys import datetime +import os +import subprocess +import sys + +from pathlib import Path =20 -current_version =3D "dev" +# Test that we are building from a Git repository +try: + subprocess.run(["git", "rev-parse", "--is-inside-work-tree"], + stdout=3Dsubprocess.PIPE, stderr=3Dsubprocess.PIPE, che= ck=3DTrue) +except subprocess.CalledProcessError: + sys.exit("Building bitbake's documentation must be done from its Git r= epository.\n" + "Clone the repository with the following command:\n" + "git clone https://git.openembedded.org/bitbake ") + +sys.path.insert(0, os.path.abspath('.')) +import setversions + +current_version =3D setversions.get_current_version() +print(f"Version calculated to be {current_version}") =20 # String used in sidebar version =3D 'Version: ' + current_version @@ -26,6 +35,12 @@ if current_version =3D=3D 'dev': # Version seen in documentation_options.js and hence in js switchers code release =3D current_version =20 +setversions.write_switchers_js("sphinx-static/switchers.js.in", + "sphinx-static/switchers.js", + current_version) + +setversions.write_releases_rst("releases.rst") + # -- Project information -------------------------------------------------= ---- =20 project =3D 'Bitbake' @@ -37,6 +52,7 @@ author =3D 'Richard Purdie, Chris Larson, and Phil Blunde= ll' extlinks =3D { 'yocto_docs': ('https://docs.yoctoproject.org%s', None), 'oe_lists': ('https://lists.openembedded.org%s', None), + 'wikipedia': ('https://en.wikipedia.org/wiki/%s', None), } =20 # -- General configuration -----------------------------------------------= ---- @@ -47,6 +63,7 @@ extlinks =3D { extensions =3D [ 'sphinx.ext.autosectionlabel', 'sphinx.ext.extlinks', + 'sphinx.ext.autodoc', ] autosectionlabel_prefix_document =3D True =20 @@ -99,3 +116,7 @@ html_last_updated_fmt =3D '%b %d, %Y' =20 # Remove the trailing 'dot' in section numbers html_secnumber_suffix =3D " " + +# autoconf needs the modules available to auto-generate documentation from= the +# code +sys.path.insert(0, str(Path('..', 'lib').resolve())) diff --git a/bitbake/doc/index.rst b/bitbake/doc/index.rst index ee1660ac..9f2a9067 100644 --- a/bitbake/doc/index.rst +++ b/bitbake/doc/index.rst @@ -12,10 +12,12 @@ BitBake User Manual =20 bitbake-user-manual/bitbake-user-manual-intro bitbake-user-manual/bitbake-user-manual-execution + bitbake-user-manual/bitbake-user-manual-environment-setup bitbake-user-manual/bitbake-user-manual-metadata bitbake-user-manual/bitbake-user-manual-ref-variables-context bitbake-user-manual/bitbake-user-manual-fetching bitbake-user-manual/bitbake-user-manual-ref-variables + bitbake-user-manual/bitbake-user-manual-library-functions bitbake-user-manual/bitbake-user-manual-hello =20 .. toctree:: diff --git a/bitbake/doc/releases.rst b/bitbake/doc/releases.rst deleted file mode 100644 index b38b1c06..00000000 --- a/bitbake/doc/releases.rst +++ /dev/null @@ -1,174 +0,0 @@ -.. SPDX-License-Identifier: CC-BY-2.5 - -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D -BitBake Supported Release Manuals -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D - -******************************* -Release Series 4.2 (mickledore) -******************************* - -- :yocto_docs:`BitBake 2.4 User Manual ` - -****************************** -Release Series 4.0 (kirkstone) -****************************** - -- :yocto_docs:`BitBake 2.0 User Manual ` - -**************************** -Release Series 3.1 (dunfell) -**************************** - -- :yocto_docs:`BitBake 1.46 User Manual ` - -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D -BitBake Outdated Release Manuals -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D - -***************************** -Release Series 4.1 (langdale) -***************************** - -- :yocto_docs:`BitBake 2.2 User Manual ` - -****************************** -Release Series 3.4 (honister) -****************************** - -- :yocto_docs:`BitBake 1.52 User Manual ` - -****************************** -Release Series 3.3 (hardknott) -****************************** - -- :yocto_docs:`BitBake 1.50 User Manual ` - -******************************* -Release Series 3.2 (gatesgarth) -******************************* - -- :yocto_docs:`BitBake 1.48 User Manual ` - -******************************************* -Release Series 3.1 (dunfell first versions) -******************************************* - -- :yocto_docs:`3.1 BitBake User Manual ` -- :yocto_docs:`3.1.1 BitBake User Manual ` -- :yocto_docs:`3.1.2 BitBake User Manual ` -- :yocto_docs:`3.1.3 BitBake User Manual ` - -************************* -Release Series 3.0 (zeus) -************************* - -- :yocto_docs:`3.0 BitBake User Manual ` -- :yocto_docs:`3.0.1 BitBake User Manual ` -- :yocto_docs:`3.0.2 BitBake User Manual ` -- :yocto_docs:`3.0.3 BitBake User Manual ` -- :yocto_docs:`3.0.4 BitBake User Manual ` - -**************************** -Release Series 2.7 (warrior) -**************************** - -- :yocto_docs:`2.7 BitBake User Manual ` -- :yocto_docs:`2.7.1 BitBake User Manual ` -- :yocto_docs:`2.7.2 BitBake User Manual ` -- :yocto_docs:`2.7.3 BitBake User Manual ` -- :yocto_docs:`2.7.4 BitBake User Manual ` - -************************* -Release Series 2.6 (thud) -************************* - -- :yocto_docs:`2.6 BitBake User Manual ` -- :yocto_docs:`2.6.1 BitBake User Manual ` -- :yocto_docs:`2.6.2 BitBake User Manual ` -- :yocto_docs:`2.6.3 BitBake User Manual ` -- :yocto_docs:`2.6.4 BitBake User Manual ` - -************************* -Release Series 2.5 (sumo) -************************* - -- :yocto_docs:`2.5 Documentation ` -- :yocto_docs:`2.5.1 Documentation ` -- :yocto_docs:`2.5.2 Documentation ` -- :yocto_docs:`2.5.3 Documentation ` - -************************** -Release Series 2.4 (rocko) -************************** - -- :yocto_docs:`2.4 BitBake User Manual ` -- :yocto_docs:`2.4.1 BitBake User Manual ` -- :yocto_docs:`2.4.2 BitBake User Manual ` -- :yocto_docs:`2.4.3 BitBake User Manual ` -- :yocto_docs:`2.4.4 BitBake User Manual ` - -************************* -Release Series 2.3 (pyro) -************************* - -- :yocto_docs:`2.3 BitBake User Manual ` -- :yocto_docs:`2.3.1 BitBake User Manual ` -- :yocto_docs:`2.3.2 BitBake User Manual ` -- :yocto_docs:`2.3.3 BitBake User Manual ` -- :yocto_docs:`2.3.4 BitBake User Manual ` - -************************** -Release Series 2.2 (morty) -************************** - -- :yocto_docs:`2.2 BitBake User Manual ` -- :yocto_docs:`2.2.1 BitBake User Manual ` -- :yocto_docs:`2.2.2 BitBake User Manual ` -- :yocto_docs:`2.2.3 BitBake User Manual ` - -**************************** -Release Series 2.1 (krogoth) -**************************** - -- :yocto_docs:`2.1 BitBake User Manual ` -- :yocto_docs:`2.1.1 BitBake User Manual ` -- :yocto_docs:`2.1.2 BitBake User Manual ` -- :yocto_docs:`2.1.3 BitBake User Manual ` - -*************************** -Release Series 2.0 (jethro) -*************************** - -- :yocto_docs:`1.9 BitBake User Manual ` -- :yocto_docs:`2.0 BitBake User Manual ` -- :yocto_docs:`2.0.1 BitBake User Manual ` -- :yocto_docs:`2.0.2 BitBake User Manual ` -- :yocto_docs:`2.0.3 BitBake User Manual ` - -************************* -Release Series 1.8 (fido) -************************* - -- :yocto_docs:`1.8 BitBake User Manual ` -- :yocto_docs:`1.8.1 BitBake User Manual ` -- :yocto_docs:`1.8.2 BitBake User Manual ` - -************************** -Release Series 1.7 (dizzy) -************************** - -- :yocto_docs:`1.7 BitBake User Manual ` -- :yocto_docs:`1.7.1 BitBake User Manual ` -- :yocto_docs:`1.7.2 BitBake User Manual ` -- :yocto_docs:`1.7.3 BitBake User Manual ` - -************************** -Release Series 1.6 (daisy) -************************** - -- :yocto_docs:`1.6 BitBake User Manual ` -- :yocto_docs:`1.6.1 BitBake User Manual ` -- :yocto_docs:`1.6.2 BitBake User Manual ` -- :yocto_docs:`1.6.3 BitBake User Manual ` - diff --git a/bitbake/doc/setversions.py b/bitbake/doc/setversions.py new file mode 100755 index 00000000..2df384e9 --- /dev/null +++ b/bitbake/doc/setversions.py @@ -0,0 +1,374 @@ +#!/usr/bin/env python3 +# +# This file defines is used in doc/conf.py to setup the version informatio= n for +# the documentation: +# - get_current_version() used in doc/conf.py computes the current version= by +# trying to guess the approximate versions we're at using git tags and +# branches from the repository. +# - write_switchers_js() write the switchers.js file used for switching be= tween +# versions of the documentation. +# +# Copyright (c) 2026 Antonin Godard +# +# SPDX-License-Identifier: MIT +# + +import argparse +import itertools +import json +import os +import re +import subprocess +import sys +import textwrap + +from urllib.request import urlopen, URLError + +# NOTE: the following variables contain default values in case we are not = able to fetch +# the releases.json file from https://dashboard.yoctoproject.org/releases.= json +DEVBRANCH =3D "2.18" +LTSSERIES =3D ["2.8", "2.0"] +ACTIVERELEASES =3D ["2.16"] + LTSSERIES +YOCTO_MAPPING =3D { + "2.18": "wrynose", + "2.16": "whinlatter", + "2.8": "scarthgap", + "2.0": "kirkstone", +} + +RELEASES_FROM_JSON =3D {} + +# Use the local releases.json file if found, fetch it from the dashboard o= therwise +releases_json_path =3D os.path.join(os.path.dirname(os.path.realpath(__fil= e__)), "releases.json") +try: + with open(releases_json_path, "r") as f: + RELEASES_FROM_JSON =3D json.load(f) +except FileNotFoundError: + print("Fetching releases.json from https://dashboard.yoctoproject.org/= releases.json...", + file=3Dsys.stderr) + try: + with urlopen("https://dashboard.yoctoproject.org/releases.json") a= s r, \ + open(releases_json_path, "w") as f: + RELEASES_FROM_JSON =3D json.load(r) + json.dump(RELEASES_FROM_JSON, f) + except URLError: + print("WARNING: tried to fetch https://dashboard.yoctoproject.org/= releases.json " + "but failed, using default values for active releases", file= =3Dsys.stderr) + pass + +if RELEASES_FROM_JSON: + ACTIVERELEASES =3D [] + DEVBRANCH =3D "" + LTSSERIES =3D [] + YOCTO_MAPPING =3D {} + + for release in RELEASES_FROM_JSON: + bb_ver =3D release["bitbake_version"] + if release["status"] =3D=3D "Active Development": + DEVBRANCH =3D bb_ver + if "LTS until" in release["status"]: + LTSSERIES.append(bb_ver) + if release["bitbake_version"]: + YOCTO_MAPPING[bb_ver] =3D release["release_codename"] + + # Find the first non-dev release, which should be displayed as the def= ault + # page on the docs website. + current_branch =3D "" + for release in RELEASES_FROM_JSON: + if release["status"] !=3D "Active Development": + current_branch =3D release["bitbake_version"] + break + + if not current_branch: + sys.exit("Unable to find a current release! Exiting...") + + # make the list of releases unique, there can be duplication when the + # current releases is also an LTS + ACTIVERELEASES =3D list(dict.fromkeys([current_branch] + LTSSERIES)) + +print(f"ACTIVERELEASES calculated to be {ACTIVERELEASES}", file=3Dsys.stde= rr) +print(f"DEVBRANCH calculated to be {DEVBRANCH}", file=3Dsys.stderr) +print(f"LTSSERIES calculated to be {LTSSERIES}", file=3Dsys.stderr) + +BB_RELEASE_TAG_RE =3D re.compile(r"^[0-9]+\.[0-9]+\.[0-9]+$") + +def main(): + parser =3D argparse.ArgumentParser( + description=3D"Parse https://dashboard.yoctoproject.org/releases.j= son to get current releases information" + ) + parser.add_argument("--get-latest-branch", + help=3D"Print current latest branch and exit", + action=3D"store_true", + default=3DFalse) + args =3D parser.parse_args() + + if args.get_latest_branch: + print(ACTIVERELEASES[0]) + sys.exit(0) + +def get_current_version(): + # Test tags exist and inform the user to fetch if not + try: + subprocess.run(["git", "show", f"{LTSSERIES[-1]}.0"], + stdout=3Dsubprocess.PIPE, stderr=3Dsubprocess.PIPE,= check=3DTrue) + except subprocess.CalledProcessError: + sys.exit("Please run 'git fetch --tags' before building the docume= ntation") + + # Try and figure out what we are + tags =3D subprocess.run(["git", "tag", "--points-at", "HEAD"], + stdout=3Dsubprocess.PIPE, stderr=3Dsubprocess.PI= PE, + universal_newlines=3DTrue).stdout + for t in tags.split(): + if re.match(BB_RELEASE_TAG_RE, t): + return t + + # We're floating on a branch + branch =3D subprocess.run(["git", "branch", "--show-current"], + stdout=3Dsubprocess.PIPE, stderr=3Dsubprocess.= PIPE, + universal_newlines=3DTrue).stdout.strip() + + if branch =3D=3D "" or branch not in list(YOCTO_MAPPING.keys()) + ["ma= ster", "master-next"]: + # We're not on a known release branch so we have to guess. Compare= the + # numbers of commits from each release branch and assume the small= est + # number of commits is the one we're based off + possible_branch =3D None + branch_count =3D 0 + for b in itertools.chain(YOCTO_MAPPING.keys(), ["master"]): + result =3D subprocess.run(["git", "log", "--format=3Doneline",= "HEAD..origin/" + b], + stdout=3Dsubprocess.PIPE, stderr=3Dsub= process.PIPE, + universal_newlines=3DTrue) + if result.returncode =3D=3D 0: + count =3D result.stdout.count('\n') + if not possible_branch or count < branch_count: + print("Branch %s has count %s" % (b, count)) + possible_branch =3D b + branch_count =3D count + if possible_branch: + branch =3D possible_branch + else: + branch =3D "master" + print("Nearest release branch estimated to be %s" % branch) + + if branch =3D=3D "master": + return "dev" + + if branch =3D=3D "master-next": + return "next" + + ourversion =3D branch + head_commit =3D subprocess.run(["git", "rev-parse", "--short", "HEAD"]= , + stdout=3Dsubprocess.PIPE, stderr=3Dsubpro= cess.PIPE, + universal_newlines=3DTrue).stdout.strip() + branch_commit =3D subprocess.run(["git", "rev-parse", "--short", branc= h], + stdout=3Dsubprocess.PIPE, stderr=3Dsubp= rocess.PIPE, + universal_newlines=3DTrue).stdout.strip= () + if head_commit !=3D branch_commit: + ourversion +=3D f" ({head_commit})" + + return ourversion + +def write_switchers_js(js_in, js_out, current_version): + with open(js_in, "r") as r, open(js_out, "w") as w: + lines =3D r.readlines() + for line in lines: + if "VERSIONS_PLACEHOLDER" in line: + if current_version !=3D "dev": + w.write(" 'dev': 'Unstable (dev)',\n") + for series in ACTIVERELEASES: + w.write(f" '{series}': '{series} ({YOCTO_MAPPING[se= ries]})',\n") + else: + w.write(line) + print("switchers.js generated from switchers.js.in") + +def _release_section(series_version: str, codename: str, bitbake_version: = str) -> str: + """ + Helper function to generate a release section, as: + + ******************** + Release Series xxxxx + ******************** + + - + """ + section_length =3D len(series_version) + len(codename) + 18 + return textwrap.dedent( + f"""\ + {'*' * section_length} + Release Series {series_version} ({codename}) + {'*' * section_length} + + - :yocto_docs:`BitBake {bitbake_version} User Manual ` + + """) + +def write_releases_rst(releases_rst_out: str): + """ + Generates the releases.rst file automatically, based on what is found + in the releases.json file. + """ + with open(releases_rst_out, "w") as f: + f.write(textwrap.dedent("""\ + .. SPDX-License-Identifier: CC-BY-2.5 + + =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + BitBake Supported Release Manuals + =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + + """)) + + for release in RELEASES_FROM_JSON: + if release["status"] =3D=3D "Active Development": + continue + + if not release["bitbake_version"]: + continue + + if release["series"] =3D=3D "current": + f.write(_release_section( + release["series_version"], + release["release_codename"], + release["bitbake_version"])) + + f.write(textwrap.dedent("""\ + =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + BitBake Outdated Release Manuals + =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + + """)) + + for release in RELEASES_FROM_JSON: + if not release["series"] =3D=3D "previous": + continue + + if not release["bitbake_version"]: + continue + + f.write(_release_section( + release["series_version"], + release["release_codename"], + release["bitbake_version"])) + + # old legacy links, which cannot be auto-generated + f.write(textwrap.dedent( + """\ + - :yocto_docs:`3.1.2 BitBake User Manual ` + - :yocto_docs:`3.1 BitBake User Manual ` + - :yocto_docs:`3.1.1 BitBake User Manual ` + - :yocto_docs:`3.1.3 BitBake User Manual ` + + ************************* + Release Series 3.0 (Zeus) + ************************* + + - :yocto_docs:`3.0 BitBake User Manual ` + - :yocto_docs:`3.0.1 BitBake User Manual ` + - :yocto_docs:`3.0.2 BitBake User Manual ` + - :yocto_docs:`3.0.3 BitBake User Manual ` + - :yocto_docs:`3.0.4 BitBake User Manual ` + + **************************** + Release Series 2.7 (Warrior) + **************************** + + - :yocto_docs:`2.7 BitBake User Manual ` + - :yocto_docs:`2.7.1 BitBake User Manual ` + - :yocto_docs:`2.7.2 BitBake User Manual ` + - :yocto_docs:`2.7.3 BitBake User Manual ` + - :yocto_docs:`2.7.4 BitBake User Manual ` + + ************************* + Release Series 2.6 (Thud) + ************************* + + - :yocto_docs:`2.6 BitBake User Manual ` + - :yocto_docs:`2.6.1 BitBake User Manual ` + - :yocto_docs:`2.6.2 BitBake User Manual ` + - :yocto_docs:`2.6.3 BitBake User Manual ` + - :yocto_docs:`2.6.4 BitBake User Manual ` + + ************************* + Release Series 2.5 (Sumo) + ************************* + + - :yocto_docs:`2.5 Documentation ` + - :yocto_docs:`2.5.1 Documentation ` + - :yocto_docs:`2.5.2 Documentation ` + - :yocto_docs:`2.5.3 Documentation ` + + ************************** + Release Series 2.4 (Rocko) + ************************** + + - :yocto_docs:`2.4 BitBake User Manual ` + - :yocto_docs:`2.4.1 BitBake User Manual ` + - :yocto_docs:`2.4.2 BitBake User Manual ` + - :yocto_docs:`2.4.3 BitBake User Manual ` + - :yocto_docs:`2.4.4 BitBake User Manual ` + + ************************* + Release Series 2.3 (Pyro) + ************************* + + - :yocto_docs:`2.3 BitBake User Manual ` + - :yocto_docs:`2.3.1 BitBake User Manual ` + - :yocto_docs:`2.3.2 BitBake User Manual ` + - :yocto_docs:`2.3.3 BitBake User Manual ` + - :yocto_docs:`2.3.4 BitBake User Manual ` + + ************************** + Release Series 2.2 (Morty) + ************************** + + - :yocto_docs:`2.2 BitBake User Manual ` + - :yocto_docs:`2.2.1 BitBake User Manual ` + - :yocto_docs:`2.2.2 BitBake User Manual ` + - :yocto_docs:`2.2.3 BitBake User Manual ` + + **************************** + Release Series 2.1 (Krogoth) + **************************** + + - :yocto_docs:`2.1 BitBake User Manual ` + - :yocto_docs:`2.1.1 BitBake User Manual ` + - :yocto_docs:`2.1.2 BitBake User Manual ` + - :yocto_docs:`2.1.3 BitBake User Manual ` + + *************************** + Release Series 2.0 (Jethro) + *************************** + + - :yocto_docs:`1.9 BitBake User Manual ` + - :yocto_docs:`2.0 BitBake User Manual ` + - :yocto_docs:`2.0.1 BitBake User Manual ` + - :yocto_docs:`2.0.2 BitBake User Manual ` + - :yocto_docs:`2.0.3 BitBake User Manual ` + + ************************* + Release Series 1.8 (Fido) + ************************* + + - :yocto_docs:`1.8 BitBake User Manual ` + - :yocto_docs:`1.8.1 BitBake User Manual ` + - :yocto_docs:`1.8.2 BitBake User Manual ` + + ************************** + Release Series 1.7 (Dizzy) + ************************** + + - :yocto_docs:`1.7 BitBake User Manual ` + - :yocto_docs:`1.7.1 BitBake User Manual ` + - :yocto_docs:`1.7.2 BitBake User Manual ` + - :yocto_docs:`1.7.3 BitBake User Manual ` + + ************************** + Release Series 1.6 (Daisy) + ************************** + + - :yocto_docs:`1.6 BitBake User Manual ` + - :yocto_docs:`1.6.1 BitBake User Manual ` + - :yocto_docs:`1.6.2 BitBake User Manual ` + - :yocto_docs:`1.6.3 BitBake User Manual ` + """)) + +if __name__ =3D=3D "__main__": + main() diff --git a/bitbake/doc/sphinx-static/switchers.js b/bitbake/doc/sphinx-st= atic/switchers.js deleted file mode 100644 index 32113cfa..00000000 --- a/bitbake/doc/sphinx-static/switchers.js +++ /dev/null @@ -1,233 +0,0 @@ -(function() { - 'use strict'; - - var all_versions =3D { - 'dev': 'dev (3.2)', - '3.1.2': '3.1.2', - '3.0.3': '3.0.3', - '2.7.4': '2.7.4', - }; - - var all_doctypes =3D { - 'single': 'Individual Webpages', - 'mega': "All-in-one 'Mega' Manual", - }; - - // Simple version comparision - // Return 1 if a > b - // Return -1 if a < b - // Return 0 if a =3D=3D b - function ver_compare(a, b) { - if (a =3D=3D "dev") { - return 1; - } - - if (a =3D=3D=3D b) { - return 0; - } - - var a_components =3D a.split("."); - var b_components =3D b.split("."); - - var len =3D Math.min(a_components.length, b_components.length); - - // loop while the components are equal - for (var i =3D 0; i < len; i++) { - // A bigger than B - if (parseInt(a_components[i]) > parseInt(b_components[i])) { - return 1; - } - - // B bigger than A - if (parseInt(a_components[i]) < parseInt(b_components[i])) { - return -1; - } - } - - // If one's a prefix of the other, the longer one is greater. - if (a_components.length > b_components.length) { - return 1; - } - - if (a_components.length < b_components.length) { - return -1; - } - - // Otherwise they are the same. - return 0; - } - - function build_version_select(current_series, current_version) { - var buf =3D [''); - return buf.join(''); - } - - function build_doctype_select(current_doctype) { - var buf =3D [''); - return buf.join(''); - } - - function navigate_to_first_existing(urls) { - // Navigate to the first existing URL in urls. - var url =3D urls.shift(); - - // Web browsers won't redirect file:// urls to file urls using ajax bu= t - // its useful for local testing - if (url.startsWith("file://")) { - window.location.href =3D url; - return; - } - - if (urls.length =3D=3D 0) { - window.location.href =3D url; - return; - } - $.ajax({ - url: url, - success: function() { - window.location.href =3D url; - }, - error: function() { - navigate_to_first_existing(urls); - } - }); - } - - function get_docroot_url() { - var url =3D window.location.href; - var root =3D DOCUMENTATION_OPTIONS.URL_ROOT; - - var urlarray =3D url.split('/'); - // Trim off anything after '/' - urlarray.pop(); - var depth =3D (root.match(/\.\.\//g) || []).length; - for (var i =3D 0; i < depth; i++) { - urlarray.pop(); - } - - return urlarray.join('/') + '/'; - } - - function on_version_switch() { - var selected_version =3D $(this).children('option:selected').attr('val= ue'); - var url =3D window.location.href; - var current_version =3D DOCUMENTATION_OPTIONS.VERSION; - var docroot =3D get_docroot_url() - - var new_versionpath =3D selected_version + '/'; - if (selected_version =3D=3D "dev") - new_versionpath =3D ''; - - // dev versions have no version prefix - if (current_version =3D=3D "dev") { - var new_url =3D docroot + new_versionpath + url.replace(docroot, "= "); - var fallback_url =3D docroot + new_versionpath; - } else { - var new_url =3D url.replace('/' + current_version + '/', '/' + new= _versionpath); - var fallback_url =3D new_url.replace(url.replace(docroot, ""), "")= ; - } - - console.log(get_docroot_url()) - console.log(url + " to url " + new_url); - console.log(url + " to fallback " + fallback_url); - - if (new_url !=3D url) { - navigate_to_first_existing([ - new_url, - fallback_url, - 'https://www.yoctoproject.org/docs/', - ]); - } - } - - function on_doctype_switch() { - var selected_doctype =3D $(this).children('option:selected').attr('val= ue'); - var url =3D window.location.href; - if (selected_doctype =3D=3D 'mega') { - var docroot =3D get_docroot_url() - var current_version =3D DOCUMENTATION_OPTIONS.VERSION; - // Assume manuals before 3.2 are using old docbook mega-manual - if (ver_compare(current_version, "3.2") < 0) { - var new_url =3D docroot + "mega-manual/mega-manual.html"; - } else { - var new_url =3D docroot + "singleindex.html"; - } - } else { - var new_url =3D url.replace("singleindex.html", "index.html") - } - - if (new_url !=3D url) { - navigate_to_first_existing([ - new_url, - 'https://www.yoctoproject.org/docs/', - ]); - } - } - - // Returns the current doctype based upon the url - function doctype_segment_from_url(url) { - if (url.includes("singleindex") || url.includes("mega-manual")) - return "mega"; - return "single"; - } - - $(document).ready(function() { - var release =3D DOCUMENTATION_OPTIONS.VERSION; - var current_doctype =3D doctype_segment_from_url(window.location.href)= ; - var current_series =3D release.substr(0, 3); - var version_select =3D build_version_select(current_series, release); - - $('.version_switcher_placeholder').html(version_select); - $('.version_switcher_placeholder select').bind('change', on_version_sw= itch); - - var doctype_select =3D build_doctype_select(current_doctype); - - $('.doctype_switcher_placeholder').html(doctype_select); - $('.doctype_switcher_placeholder select').bind('change', on_doctype_sw= itch); - - if (ver_compare(release, "3.1") < 0) { - $('#outdated-warning').html('Version ' + release + ' of the project = is now considered obsolete, please select and use a more recent version'); - $('#outdated-warning').css('padding', '.5em'); - } else if (release !=3D "dev") { - $.each(all_versions, function(version, title) { - var series =3D version.substr(0, 3); - if (series =3D=3D current_series && version !=3D release) { - $('#outdated-warning').html('This document is for outdated versi= on ' + release + ', you should select the latest release version in this se= ries, ' + version + '.'); - $('#outdated-warning').css('padding', '.5em'); - } - }); - } - }); -})(); diff --git a/bitbake/doc/sphinx-static/switchers.js.in b/bitbake/doc/sphinx= -static/switchers.js.in new file mode 100644 index 00000000..0b209e95 --- /dev/null +++ b/bitbake/doc/sphinx-static/switchers.js.in @@ -0,0 +1,249 @@ +// SPDX-License-Identifier: MIT +(function () { + "use strict"; + + var all_versions =3D { + VERSIONS_PLACEHOLDER, + }; + + var all_doctypes =3D { + single: "Individual Webpages", + mega: "All-in-one 'Mega' Manual", + }; + + // Simple version comparision + // Return 1 if a > b + // Return -1 if a < b + // Return 0 if a =3D=3D b + function ver_compare(a, b) { + if (a =3D=3D "dev") { + return 1; + } + + if (a =3D=3D=3D b) { + return 0; + } + + var a_components =3D a.split("."); + var b_components =3D b.split("."); + + var len =3D Math.min(a_components.length, b_components.length); + + // loop while the components are equal + for (var i =3D 0; i < len; i++) { + // A bigger than B + if (parseInt(a_components[i]) > parseInt(b_components[i])) { + return 1; + } + + // B bigger than A + if (parseInt(a_components[i]) < parseInt(b_components[i])) { + return -1; + } + } + + // If one's a prefix of the other, the longer one is greater. + if (a_components.length > b_components.length) { + return 1; + } + + if (a_components.length < b_components.length) { + return -1; + } + + // Otherwise they are the same. + return 0; + } + + function build_version_select(current_version) { + var buf =3D [""); + return buf.join(""); + } + + function build_doctype_select(current_doctype) { + var buf =3D [""); + return buf.join(""); + } + + function navigate_to_first_existing(urls) { + // Navigate to the first existing URL in urls. + var url =3D urls.shift(); + + // Web browsers won't redirect file:// urls to file urls using ajax bu= t + // its useful for local testing + if (url.startsWith("file://")) { + window.location.href =3D url; + return; + } + + if (urls.length =3D=3D 0) { + window.location.href =3D url; + return; + } + $.ajax({ + url: url, + success: function () { + window.location.href =3D url; + }, + error: function () { + navigate_to_first_existing(urls); + }, + }); + } + + function get_docroot_url() { + var url =3D window.location.href; + // Try to get the variable from documentation_options.js + var root =3D DOCUMENTATION_OPTIONS.URL_ROOT; + if (root =3D=3D null) { + // In recent versions of Sphinx, URL_ROOT was removed from + // documentation_options.js, so get it like searchtools.js does. + root =3D document.documentElement.dataset.content_root; + } + + var urlarray =3D url.split("/"); + // Trim off anything after '/' + urlarray.pop(); + var depth =3D (root.match(/\.\.\//g) || []).length; + for (var i =3D 0; i < depth; i++) { + urlarray.pop(); + } + + return urlarray.join("/") + "/"; + } + + function on_version_switch() { + var selected_version =3D $(this).children("option:selected").attr("val= ue"); + var url =3D window.location.href; + var current_version =3D DOCUMENTATION_OPTIONS.VERSION; + var docroot =3D get_docroot_url(); + + var new_versionpath =3D selected_version + "/"; + + // latest tag is also the default page (without version information) + if (docroot.endsWith("dev/")) { + var new_url =3D url.replace("/dev/", "/" + new_versionpath); + var fallback_url =3D new_url.replace(url.replace(docroot, ""), ""); + } else if (docroot.endsWith(current_version + "/") =3D=3D false) { + var new_url =3D docroot + new_versionpath + url.replace(docroot, "")= ; + var fallback_url =3D docroot + new_versionpath; + } else { + var new_url =3D url.replace( + "/" + current_version + "/", + "/" + new_versionpath, + ); + var fallback_url =3D new_url.replace(url.replace(docroot, ""), ""); + } + + console.log(url + " to url " + new_url); + console.log(url + " to fallback " + fallback_url); + + if (new_url !=3D url) { + navigate_to_first_existing([ + new_url, + fallback_url, + "https://www.yoctoproject.org/bitbake/", + ]); + } + } + + function on_doctype_switch() { + var selected_doctype =3D $(this).children("option:selected").attr("val= ue"); + var url =3D window.location.href; + if (selected_doctype =3D=3D "mega") { + var docroot =3D get_docroot_url(); + var current_version =3D DOCUMENTATION_OPTIONS.VERSION; + var new_url =3D docroot + "singleindex.html"; + } else { + var new_url =3D url.replace("singleindex.html", "index.html"); + } + + if (new_url !=3D url) { + navigate_to_first_existing([ + new_url, + "https://www.yoctoproject.org/docs/", + ]); + } + } + + // Returns the current doctype based upon the url + function doctype_segment_from_url(url) { + if (url.includes("singleindex") || url.includes("mega-manual")) + return "mega"; + return "single"; + } + + $(document).ready(function () { + var release =3D DOCUMENTATION_OPTIONS.VERSION; + var current_doctype =3D doctype_segment_from_url(window.location.href)= ; + var current_series =3D release.substr(0, 3); + var version_select =3D build_version_select(release); + + $(".version_switcher_placeholder").html(version_select); + $(".version_switcher_placeholder select").bind("change", on_version_sw= itch); + + var doctype_select =3D build_doctype_select(current_doctype); + + $(".doctype_switcher_placeholder").html(doctype_select); + $(".doctype_switcher_placeholder select").bind("change", on_doctype_sw= itch); + + // if release =3D "X.Y ()", remove the "()" = so that only X.Y is compared + release =3D release.split(" ")[0]; + if (!(["dev", "next"].includes(release)) && !(release in all_versions)= ) { + $("#outdated-warning").html( + "Version " + release + " of the project is now considered obsolete= , please select and use a more recent version", + ); + $("#outdated-warning").css("padding", ".5em"); + } + }); +})(); diff --git a/bitbake/doc/sphinx-static/theme_overrides.css b/bitbake/doc/sp= hinx-static/theme_overrides.css index e362677a..3a303c8e 100644 --- a/bitbake/doc/sphinx-static/theme_overrides.css +++ b/bitbake/doc/sphinx-static/theme_overrides.css @@ -99,14 +99,19 @@ em { [alt=3D'Permalink'] { color: #eee; } [alt=3D'Permalink']:hover { color: black; } =20 +.literal-block { + background: #f8f8f8; +} + @media screen { - /* content column - * - * RTD theme's default is 800px as max width for the content, but we h= ave - * tables with tons of columns, which need the full width of the view-= port. - */ + .wy-nav-content { + max-width: 1000px; + background: #fcfcfc; + } =20 - .wy-nav-content{max-width: none; } + .wy-nav-content-wrap { + background: #efefef; + } =20 /* inline literal: drop the borderbox, padding and red color */ code, .rst-content tt, .rst-content code { diff --git a/bitbake/lib/bb/COW.py b/bitbake/lib/bb/COW.py index 76bc08a3..4af03c54 100644 --- a/bitbake/lib/bb/COW.py +++ b/bitbake/lib/bb/COW.py @@ -36,8 +36,9 @@ class COWDictMeta(COWMeta): __marker__ =3D tuple() =20 def __str__(cls): - # FIXME: I have magic numbers! - return "" % (cls.__count__, le= n(cls.__dict__) - 3) + ignored_keys =3D set(["__count__", "__doc__", "__module__", "__fir= stlineno__", "__static_attributes__"]) + keys =3D set(cls.__dict__.keys()) - ignored_keys + return "" % (cls.__count__, le= n(keys)) =20 __repr__ =3D __str__ =20 @@ -161,8 +162,9 @@ class COWDictMeta(COWMeta): =20 class COWSetMeta(COWDictMeta): def __str__(cls): - # FIXME: I have magic numbers! - return "" % (cls.__count__, len= (cls.__dict__) - 3) + ignored_keys =3D set(["__count__", "__doc__", "__module__", "__fir= stlineno__", "__static_attributes__"]) + keys =3D set(cls.__dict__.keys()) - ignored_keys + return "" % (cls.__count__, len= (keys)) =20 __repr__ =3D __str__ =20 diff --git a/bitbake/lib/bb/__init__.py b/bitbake/lib/bb/__init__.py index 7d9bc147..01c73f1b 100644 --- a/bitbake/lib/bb/__init__.py +++ b/bitbake/lib/bb/__init__.py @@ -9,12 +9,11 @@ # SPDX-License-Identifier: GPL-2.0-only # =20 -__version__ =3D "2.8.0" +__version__ =3D "2.20.0" =20 import sys -# It was 3.8.0 originally but set to 3.7.3 for Debian Buster -if sys.version_info < (3, 7, 3): - raise RuntimeError("Sorry, python 3.7.3 or later is required for this = version of bitbake") +if sys.version_info < (3, 9, 0): + raise RuntimeError("Sorry, python 3.9.0 or later is required for this = version of bitbake") =20 if sys.version_info < (3, 10, 0): # With python 3.8 and 3.9, we see errors of "libgcc_s.so.1 must be ins= talled for pthread_cancel to work" @@ -37,6 +36,35 @@ class BBHandledException(Exception): =20 import os import logging +from collections import namedtuple +import multiprocessing as mp + +# Python 3.14 changes the default multiprocessing context from "fork" to +# "forkserver". However, bitbake heavily relies on "fork" behavior to +# efficiently pass data to the child processes. Places that need this shou= ld do: +# from bb import multiprocessing +# in place of +# import multiprocessing + +class MultiprocessingContext(object): + """ + Multiprocessing proxy object that uses the "fork" context for a proper= ty if + available, otherwise goes to the main multiprocessing module. This all= ows + it to be a drop-in replacement for the multiprocessing module, but use= the + fork context + """ + def __init__(self): + super().__setattr__("_ctx", mp.get_context("fork")) + + def __getattr__(self, name): + if hasattr(self._ctx, name): + return getattr(self._ctx, name) + return getattr(mp, name) + + def __setattr__(self, name, value): + raise AttributeError(f"Unable to set attribute {name}") + +multiprocessing =3D MultiprocessingContext() =20 =20 class NullHandler(logging.Handler): @@ -62,6 +90,7 @@ class BBLoggerMixin(object): return self.bbdebug(3, *args, **kwargs) =20 def bbdebug(self, level, msg, *args, **kwargs): + import bb.event loglevel =3D logging.DEBUG - level + 1 if not bb.event.worker_pid: if self.name in bb.msg.loggerDefaultDomains and loglevel > (bb= .msg.loggerDefaultDomains[self.name]): @@ -104,26 +133,6 @@ class BBLoggerAdapter(logging.LoggerAdapter, BBLoggerM= ixin): self.setup_bblogger(logger.name) super().__init__(logger, *args, **kwargs) =20 - if sys.version_info < (3, 6): - # These properties were added in Python 3.6. Add them in older ver= sions - # for compatibility - @property - def manager(self): - return self.logger.manager - - @manager.setter - def manager(self, value): - self.logger.manager =3D value - - @property - def name(self): - return self.logger.name - - def __repr__(self): - logger =3D self.logger - level =3D logger.getLevelName(logger.getEffectiveLevel()) - return '<%s %s (%s)>' % (self.__class__.__name__, logger.name,= level) - logging.LoggerAdapter =3D BBLoggerAdapter =20 logger =3D logging.getLogger("BitBake") @@ -144,14 +153,39 @@ class PrefixLoggerAdapter(logging.LoggerAdapter): # can result in construction of the various loggers. import bb.msg =20 -from bb import fetch2 as fetch -sys.modules['bb.fetch'] =3D sys.modules['bb.fetch2'] +# There's lots of code (in OE and other layers) that accesses bb.fetch2 wi= thout +# first importing it. It used to work because fetch2 was added to the bb m= odule in +# bb/__init__.py, as part of installing the fetch =3D fetch2 shim. But now= that +# the new fetch2 shim is an actual module, this will break. So, provide la= zy +# access to bb.fetch2 via __getattr__ so layers keep working. +def __getattr__(name): + if name =3D=3D "fetch2": + import bb.fetch2 as fetch2 + return fetch2 + + raise AttributeError(f"module {__name__!r} has no attribute {name!r}") =20 # Messaging convenience functions def plain(*args): + """ + Prints a message at "plain" level (higher level than a ``bb.note()``). + + Arguments: + + - ``args``: one or more strings to print. + """ mainlogger.plain(''.join(args)) =20 def debug(lvl, *args): + """ + Prints a debug message. + + Arguments: + + - ``lvl``: debug level. Higher value increases the debug level + (determined by ``bitbake -D``). + - ``args``: one or more strings to print. + """ if isinstance(lvl, str): mainlogger.warning("Passed invalid debug level '%s' to bb.debug", = lvl) args =3D (lvl,) + args @@ -159,33 +193,81 @@ def debug(lvl, *args): mainlogger.bbdebug(lvl, ''.join(args)) =20 def note(*args): + """ + Prints a message at "note" level. + + Arguments: + + - ``args``: one or more strings to print. + """ mainlogger.info(''.join(args)) =20 -# -# A higher prioity note which will show on the console but isn't a warning -# -# Something is happening the user should be aware of but they probably did -# something to make it happen -# def verbnote(*args): + """ + A higher priority note which will show on the console but isn't a warn= ing. + + Use in contexts when something is happening the user should be aware o= f but + they probably did something to make it happen. + + Arguments: + + - ``args``: one or more strings to print. + """ mainlogger.verbnote(''.join(args)) =20 # # Warnings - things the user likely needs to pay attention to and fix # def warn(*args): + """ + Prints a warning message. + + Arguments: + + - ``args``: one or more strings to print. + """ mainlogger.warning(''.join(args)) =20 def warnonce(*args): + """ + Prints a warning message like ``bb.warn()``, but only prints the messa= ge + once. + + Arguments: + + - ``args``: one or more strings to print. + """ mainlogger.warnonce(''.join(args)) =20 def error(*args, **kwargs): + """ + Prints an error message. + + Arguments: + + - ``args``: one or more strings to print. + """ mainlogger.error(''.join(args), extra=3Dkwargs) =20 def erroronce(*args): + """ + Prints an error message like ``bb.error()``, but only prints the messa= ge + once. + + Arguments: + + - ``args``: one or more strings to print. + """ mainlogger.erroronce(''.join(args)) =20 def fatal(*args, **kwargs): + """ + Prints an error message and stops the BitBake execution. + + Arguments: + + - ``args``: one or more strings to print. + """ mainlogger.critical(''.join(args), extra=3Dkwargs) raise BBHandledException() =20 @@ -214,7 +296,6 @@ def deprecated(func, name=3DNone, advice=3D""): # For compatibility def deprecate_import(current, modulename, fromlist, renames =3D None): """Import objects from one module into another, wrapping them with a D= eprecationWarning""" - import sys =20 module =3D __import__(modulename, fromlist =3D fromlist) for position, objname in enumerate(fromlist): @@ -228,3 +309,14 @@ def deprecate_import(current, modulename, fromlist, re= names =3D None): =20 setattr(sys.modules[current], newname, newobj) =20 +TaskData =3D namedtuple("TaskData", [ + "pn", + "taskname", + "fn", + "deps", + "provides", + "taskhash", + "unihash", + "hashfn", + "taskhash_deps", +]) diff --git a/bitbake/lib/bs4/COPYING.txt b/bitbake/lib/bb/_vendor/beautiful= soup4.LICENSE similarity index 80% rename from bitbake/lib/bs4/COPYING.txt rename to bitbake/lib/bb/_vendor/beautifulsoup4.LICENSE index d668d13f..08e3a9cf 100644 --- a/bitbake/lib/bs4/COPYING.txt +++ b/bitbake/lib/bb/_vendor/beautifulsoup4.LICENSE @@ -1,6 +1,6 @@ Beautiful Soup is made available under the MIT license: =20 - Copyright (c) 2004-2012 Leonard Richardson + Copyright (c) Leonard Richardson =20 Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the @@ -20,7 +20,12 @@ Beautiful Soup is made available under the MIT license: BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE - SOFTWARE, DAMMIT. + SOFTWARE. =20 Beautiful Soup incorporates code from the html5lib library, which is -also made available under the MIT license. +also made available under the MIT license. Copyright (c) James Graham +and other contributors + +Beautiful Soup has an optional dependency on the soupsieve library, +which is also made available under the MIT license. Copyright (c) +Isaac Muse diff --git a/bitbake/lib/bb/_vendor/bs4/__init__.py b/bitbake/lib/bb/_vendo= r/bs4/__init__.py new file mode 100644 index 00000000..58079b9e --- /dev/null +++ b/bitbake/lib/bb/_vendor/bs4/__init__.py @@ -0,0 +1,1177 @@ +"""Beautiful Soup Elixir and Tonic - "The Screen-Scraper's Friend". + +http://www.crummy.com/software/BeautifulSoup/ + +Beautiful Soup uses a pluggable XML or HTML parser to parse a +(possibly invalid) document into a tree representation. Beautiful Soup +provides methods and Pythonic idioms that make it easy to navigate, +search, and modify the parse tree. + +Beautiful Soup works with Python 3.7 and up. It works better if lxml +and/or html5lib is installed, but they are not required. + +For more than you ever wanted to know about Beautiful Soup, see the +documentation: http://www.crummy.com/software/BeautifulSoup/bs4/doc/ +""" + +__author__ =3D "Leonard Richardson (leonardr@segfault.org)" +__version__ =3D "4.15.0" +__copyright__ =3D "Copyright (c) 2004-2026 Leonard Richardson" +# Use of this source code is governed by the MIT license. +__license__ =3D "MIT" + +__all__ =3D [ + "AttributeResemblesVariableWarning", + "BeautifulSoup", + "Comment", + "Declaration", + "ProcessingInstruction", + "ResultSet", + "CSS", + "Script", + "Stylesheet", + "Tag", + "TemplateString", + "ElementFilter", + "UnicodeDammit", + "CData", + "Doctype", + + # Exceptions + "FeatureNotFound", + "ParserRejectedMarkup", + "StopParsing", + + # Warnings + "AttributeResemblesVariableWarning", + "GuessedAtParserWarning", + "MarkupResemblesLocatorWarning", + "UnusualUsageWarning", + "XMLParsedAsHTMLWarning", +] + +from collections import Counter +import io +import sys +import warnings + +# The very first thing we do is give a useful error if someone is +# running this code under Python 2. +if sys.version_info.major < 3: + raise ImportError( + "You are trying to use a Python 3-specific version of Beautiful So= up under Python 2. This will not work. The final version of Beautiful Soup = to support Python 2 was 4.9.3." + ) + +from .builder import ( + builder_registry, + TreeBuilder, +) +from .builder._htmlparser import HTMLParserTreeBuilder +from .dammit import UnicodeDammit +from .css import CSS +from ._deprecation import ( + _deprecated, +) +from .element import ( + CData, + Comment, + DEFAULT_OUTPUT_ENCODING, + Declaration, + Doctype, + NavigableString, + PageElement, + ProcessingInstruction, + PYTHON_SPECIFIC_ENCODINGS, + ResultSet, + Script, + Stylesheet, + Tag, + TemplateString, +) +from .formatter import Formatter +from .filter import ( + ElementFilter, + SoupStrainer, +) +from typing import ( + Any, + cast, + Counter as CounterType, + Dict, + Iterator, + List, + Sequence, + Sized, + Optional, + Type, + Union, +) + +from bb._vendor.bs4._typing import ( + _Encoding, + _Encodings, + _IncomingMarkup, + _InsertableElement, + _RawAttributeValue, + _RawAttributeValues, + _RawMarkup, +) + +# Import all warnings and exceptions into the main package. +from bb._vendor.bs4.exceptions import ( + FeatureNotFound, + ParserRejectedMarkup, + StopParsing, +) +from bb._vendor.bs4._warnings import ( + AttributeResemblesVariableWarning, + GuessedAtParserWarning, + MarkupResemblesLocatorWarning, + UnusualUsageWarning, + XMLParsedAsHTMLWarning, +) + + +class BeautifulSoup(Tag): + """A data structure representing a parsed HTML or XML document. + + Most of the methods you'll call on a BeautifulSoup object are inherite= d from + PageElement or Tag. + + Internally, this class defines the basic interface called by the + tree builders when converting an HTML/XML document into a data + structure. The interface abstracts away the differences between + parsers. To write a new tree builder, you'll need to understand + these methods as a whole. + + These methods will be called by the BeautifulSoup constructor: + * reset() + * feed(markup) + + The tree builder may call these methods from its feed() implementation= : + * handle_starttag(name, attrs) # See note about return value + * handle_endtag(name) + * handle_data(data) # Appends to the current data node + * endData(containerClass) # Ends the current data node + + No matter how complicated the underlying parser is, you should be + able to build a tree using 'start tag' events, 'end tag' events, + 'data' events, and "done with data" events. + + If you encounter an empty-element tag (aka a self-closing tag, + like HTML's
tag), call handle_starttag and then + handle_endtag. + """ + + #: Since `BeautifulSoup` subclasses `Tag`, it's possible to treat it a= s + #: a `Tag` with a `Tag.name`. Hoever, this name makes it clear the + #: `BeautifulSoup` object isn't a real markup tag. + ROOT_TAG_NAME: str =3D "[document]" + + #: If the end-user gives no indication which tree builder they + #: want, look for one with these features. + DEFAULT_BUILDER_FEATURES: Sequence[str] =3D ["html", "fast"] + + #: A string containing all ASCII whitespace characters, used in + #: during parsing to detect data chunks that seem 'empty'. + ASCII_SPACES: str =3D "\x20\x0a\x09\x0c\x0d" + + # FUTURE PYTHON: + element_classes: Dict[Type[PageElement], Type[PageElement]] #: :meta = private: + builder: TreeBuilder #: :meta private: + is_xml: bool + known_xml: Optional[bool] + parse_only: Optional[SoupStrainer] #: :meta private: + + # These members are only used while parsing markup. + markup: Optional[_RawMarkup] #: :meta private: + current_data: List[str] #: :meta private: + currentTag: Optional[Tag] #: :meta private: + tagStack: List[Tag] #: :meta private: + open_tag_counter: CounterType[str] #: :meta private: + preserve_whitespace_tag_stack: List[Tag] #: :meta private: + string_container_stack: List[Tag] #: :meta private: + _most_recent_element: Optional[PageElement] #: :meta private: + + #: Beautiful Soup's best guess as to the character encoding of the + #: original document. + original_encoding: Optional[_Encoding] + + #: The character encoding, if any, that was explicitly defined + #: in the original document. This may or may not match + #: `BeautifulSoup.original_encoding`. + declared_html_encoding: Optional[_Encoding] + + #: This is True if the markup that was parsed contains + #: U+FFFD REPLACEMENT_CHARACTER characters which were not present + #: in the original markup. These mark character sequences that + #: could not be represented in Unicode. + contains_replacement_characters: bool + + def __init__( + self, + markup: _IncomingMarkup =3D "", + features: Optional[Union[str, Sequence[str]]] =3D None, + builder: Optional[Union[TreeBuilder, Type[TreeBuilder]]] =3D None, + parse_only: Optional[SoupStrainer] =3D None, + from_encoding: Optional[_Encoding] =3D None, + exclude_encodings: Optional[_Encodings] =3D None, + element_classes: Optional[Dict[Type[PageElement], Type[PageElement= ]]] =3D None, + **kwargs: Any, + ): + """Constructor. + + :param markup: A string or a file-like object representing + markup to be parsed. + + :param features: Desirable features of the parser to be + used. This may be the name of a specific parser ("lxml", + "lxml-xml", "html.parser", or "html5lib") or it may be the + type of markup to be used ("html", "html5", "xml"). It's + recommended that you name a specific parser, so that + Beautiful Soup gives you the same results across platforms + and virtual environments. + + :param builder: A TreeBuilder subclass to instantiate (or + instance to use) instead of looking one up based on + `features`. You only need to use this if you've implemented a + custom TreeBuilder. + + :param parse_only: A SoupStrainer. Only parts of the document + matching the SoupStrainer will be considered. This is useful + when parsing part of a document that would otherwise be too + large to fit into memory. + + :param from_encoding: A string indicating the encoding of the + document to be parsed. Pass this in if Beautiful Soup is + guessing wrongly about the document's encoding. + + :param exclude_encodings: A list of strings indicating + encodings known to be wrong. Pass this in if you don't know + the document's encoding but you know Beautiful Soup's guess is + wrong. + + :param element_classes: A dictionary mapping BeautifulSoup + classes like Tag and NavigableString, to other classes you'd + like to be instantiated instead as the parse tree is + built. This is useful for subclassing Tag or NavigableString + to modify default behavior. + + :param kwargs: For backwards compatibility purposes, the + constructor accepts certain keyword arguments used in + Beautiful Soup 3. None of these arguments do anything in + Beautiful Soup 4; they will result in a warning and then be + ignored. + + Apart from this, any keyword arguments passed into the + BeautifulSoup constructor are propagated to the TreeBuilder + constructor. This makes it possible to configure a + TreeBuilder by passing in arguments, not just by saying which + one to use. + """ + if "convertEntities" in kwargs: + del kwargs["convertEntities"] + warnings.warn( + "BS4 does not respect the convertEntities argument to the = " + "BeautifulSoup constructor. Entities are always converted = " + "to Unicode characters." + ) + + if "markupMassage" in kwargs: + del kwargs["markupMassage"] + warnings.warn( + "BS4 does not respect the markupMassage argument to the " + "BeautifulSoup constructor. The tree builder is responsibl= e " + "for any necessary markup massage." + ) + + if "smartQuotesTo" in kwargs: + del kwargs["smartQuotesTo"] + warnings.warn( + "BS4 does not respect the smartQuotesTo argument to the " + "BeautifulSoup constructor. Smart quotes are always conver= ted " + "to Unicode characters." + ) + + if "selfClosingTags" in kwargs: + del kwargs["selfClosingTags"] + warnings.warn( + "Beautiful Soup 4 does not respect the selfClosingTags arg= ument to the " + "BeautifulSoup constructor. The tree builder is responsibl= e " + "for understanding self-closing tags." + ) + + if "isHTML" in kwargs: + del kwargs["isHTML"] + warnings.warn( + "Beautiful Soup 4 does not respect the isHTML argument to = the " + "BeautifulSoup constructor. Suggest you use " + "features=3D'lxml' for HTML and features=3D'lxml-xml' for = " + "XML." + ) + + def deprecated_argument(old_name: str, new_name: str) -> Optional[= Any]: + if old_name in kwargs: + warnings.warn( + 'The "%s" argument to the BeautifulSoup constructor ' + 'was renamed to "%s" in Beautiful Soup 4.0.0' + % (old_name, new_name), + DeprecationWarning, + stacklevel=3D3, + ) + return kwargs.pop(old_name) + return None + + parse_only =3D parse_only or deprecated_argument("parseOnlyThese",= "parse_only") + if parse_only is not None: + # Issue a warning if we can tell in advance that + # parse_only will exclude the entire tree. + if parse_only.excludes_everything: + warnings.warn( + f"The given value for parse_only will exclude everythi= ng: {parse_only}", + UserWarning, + stacklevel=3D3, + ) + + from_encoding =3D from_encoding or deprecated_argument( + "fromEncoding", "from_encoding" + ) + + if from_encoding and isinstance(markup, str): + warnings.warn( + "You provided Unicode markup but also provided a value for= from_encoding. Your from_encoding will be ignored." + ) + from_encoding =3D None + + self.element_classes =3D element_classes or dict() + + # We need this information to track whether or not the builder + # was specified well enough that we can omit the 'you need to + # specify a parser' warning. + original_builder =3D builder + original_features =3D features + + builder_class: Optional[Type[TreeBuilder]] =3D None + if isinstance(builder, type): + # A builder class was passed in; it needs to be instantiated. + builder_class =3D builder + builder =3D None + elif builder is None: + if isinstance(features, str): + features =3D [features] + if features is None or len(features) =3D=3D 0: + features =3D self.DEFAULT_BUILDER_FEATURES + possible_builder_class =3D builder_registry.lookup(*features) + if possible_builder_class is None: + raise FeatureNotFound( + "Couldn't find a tree builder with the features you " + "requested: %s. Do you need to install a parser librar= y?" + % ",".join(features) + ) + builder_class =3D possible_builder_class + + # At this point either we have a TreeBuilder instance in + # builder, or we have a builder_class that we can instantiate + # with the remaining **kwargs. + if builder is None: + assert builder_class is not None + builder =3D builder_class(**kwargs) + if ( + not original_builder + and not ( + original_features =3D=3D builder.NAME + or ( + isinstance(original_features, str) + and original_features in builder.ALTERNATE_NAMES + ) + ) + and markup + ): + # The user did not tell us which TreeBuilder to use, + # and we had to guess. Issue a warning. + if builder.is_xml: + markup_type =3D "XML" + else: + markup_type =3D "HTML" + + # This code adapted from warnings.py so that we get the sa= me line + # of code as our warnings.warn() call gets, even if the an= swer is wrong + # (as it may be in a multithreading situation). + caller =3D None + try: + caller =3D sys._getframe(1) + except ValueError: + pass + if caller: + globals =3D caller.f_globals + line_number =3D caller.f_lineno + else: + globals =3D sys.__dict__ + line_number =3D 1 + filename =3D globals.get("__file__") + if filename: + fnl =3D filename.lower() + if fnl.endswith((".pyc", ".pyo")): + filename =3D filename[:-1] + if filename: + # If there is no filename at all, the user is most lik= ely in a REPL, + # and the warning is not necessary. + values =3D dict( + filename=3Dfilename, + line_number=3Dline_number, + parser=3Dbuilder.NAME, + markup_type=3Dmarkup_type, + ) + warnings.warn( + GuessedAtParserWarning.MESSAGE % values, + GuessedAtParserWarning, + stacklevel=3D2, + ) + else: + if kwargs: + warnings.warn( + "Keyword arguments to the BeautifulSoup constructor wi= ll be ignored. These would normally be passed into the TreeBuilder construc= tor, but a TreeBuilder instance was passed in as `builder`." + ) + + self.builder =3D builder + self.is_xml =3D builder.is_xml + self.known_xml =3D self.is_xml + self._namespaces =3D dict() + self.parse_only =3D parse_only + + if hasattr(markup, "read"): # It's a file-type object. + markup =3D cast(io.IOBase, markup).read() + elif not isinstance(markup, (bytes, str)) and not hasattr(markup, = "__len__"): + raise TypeError( + f"Incoming markup is of an invalid type: {markup!r}. Marku= p must be a string, a bytestring, or an open filehandle." + ) + elif isinstance(markup, Sized) and len(markup) <=3D 256 and ( + (isinstance(markup, bytes) and b"<" not in markup and b"\n" no= t in markup) + or (isinstance(markup, str) and "<" not in markup and "\n" not= in markup) + ): + # Issue warnings for a couple beginner problems + # involving passing non-markup to Beautiful Soup. + # Beautiful Soup will still parse the input as markup, + # since that is sometimes the intended behavior. + if not self._markup_is_url(markup): + self._markup_resembles_filename(markup) + + # At this point we know markup is a string or bytestring. If + # it was a file-type object, we've read from it. + markup =3D cast(_RawMarkup, markup) + + rejections =3D [] + success =3D False + for ( + self.markup, + self.original_encoding, + self.declared_html_encoding, + self.contains_replacement_characters, + ) in self.builder.prepare_markup( + markup, from_encoding, exclude_encodings=3Dexclude_encodings + ): + self.reset() + self.builder.initialize_soup(self) + try: + self._feed() + success =3D True + break + except ParserRejectedMarkup as e: + rejections.append(e) + pass + + if not success: + other_exceptions =3D [str(e) for e in rejections] + raise ParserRejectedMarkup( + "The markup you provided was rejected by the parser. Tryin= g a different parser or a different encoding may help.\n\nOriginal exceptio= n(s) from parser:\n " + + "\n ".join(other_exceptions) + ) + + # Clear out the markup and remove the builder's circular + # reference to this object. + self.markup =3D None + self.builder.soup =3D None + + def copy_self(self) -> "BeautifulSoup": + """Create a new BeautifulSoup object with the same TreeBuilder, + but not associated with any markup. + + This is the first step of the deepcopy process. + """ + clone =3D type(self)("", None, self.builder) + + # Keep track of the encoding of the original document, + # since we won't be parsing it again. + clone.original_encoding =3D self.original_encoding + return clone + + def __getstate__(self) -> Dict[str, Any]: + # Frequently a tree builder can't be pickled. + d =3D dict(self.__dict__) + if "builder" in d and d["builder"] is not None and not self.builde= r.picklable: + d["builder"] =3D type(self.builder) + # Store the contents as a Unicode string. + d["contents"] =3D [] + d["markup"] =3D self.decode() + + # If _most_recent_element is present, it's a Tag object left + # over from initial parse. It might not be picklable and we + # don't need it. + if "_most_recent_element" in d: + del d["_most_recent_element"] + return d + + def __setstate__(self, state: Dict[str, Any]) -> None: + # If necessary, restore the TreeBuilder by looking it up. + self.__dict__ =3D state + if isinstance(self.builder, type): + self.builder =3D self.builder() + elif not self.builder: + # We don't know which builder was used to build this + # parse tree, so use a default we know is always available. + self.builder =3D HTMLParserTreeBuilder() + self.builder.soup =3D self + self.reset() + self._feed() + + @property + def _is_root(self): + """Yes, a BeautifulSoup object is the root of its parse tree. Used= by the _root_object internal property.""" + return True + + @classmethod + @_deprecated( + replaced_by=3D"nothing (private method, will be removed)", version= =3D"4.13.0" + ) + def _decode_markup(cls, markup: _RawMarkup) -> str: + """Ensure `markup` is Unicode so it's safe to send into warnings.w= arn. + + warnings.warn had this problem back in 2010 but fortunately + not anymore. This has not been used for a long time; I just + noticed that fact while working on 4.13.0. + """ + if isinstance(markup, bytes): + decoded =3D markup.decode("utf-8", "replace") + else: + decoded =3D markup + return decoded + + @classmethod + def _markup_is_url(cls, markup: _RawMarkup) -> bool: + """Error-handling method to raise a warning if incoming markup loo= ks + like a URL. + + :param markup: A string of markup. + :return: Whether or not the markup resembled a URL + closely enough to justify issuing a warning. + """ + problem: bool =3D False + if isinstance(markup, bytes): + problem =3D ( + any(markup.startswith(prefix) for prefix in (b"http:", b"h= ttps:")) + and b" " not in markup + ) + elif isinstance(markup, str): + problem =3D ( + any(markup.startswith(prefix) for prefix in ("http:", "htt= ps:")) + and " " not in markup + ) + else: + return False + + if not problem: + return False + warnings.warn( + MarkupResemblesLocatorWarning.URL_MESSAGE % dict(what=3D"URL")= , + MarkupResemblesLocatorWarning, + stacklevel=3D3, + ) + return True + + @classmethod + def _markup_resembles_filename(cls, markup: _RawMarkup) -> bool: + """Error-handling method to issue a warning if incoming markup + resembles a filename. + + :param markup: A string of markup. + :return: Whether or not the markup resembled a filename + closely enough to justify issuing a warning. + """ + markup_b: bytes + + # We're only checking ASCII characters, so rather than write + # the same tests twice, convert Unicode to a bytestring and + # operate on the bytestring. + if isinstance(markup, str): + markup_b =3D markup.encode("utf8") + else: + markup_b =3D markup + + # Step 1: does it end with a common textual file extension? + filelike =3D False + lower =3D markup_b.lower() + extensions =3D [b".html", b".htm", b".xml", b".xhtml", b".txt"] + if any(lower.endswith(ext) for ext in extensions): + filelike =3D True + if not filelike: + return False + + # Step 2: it _might_ be a file, but there are a few things + # we can look for that aren't very common in filenames. + + # Characters that have special meaning to Unix shells. (< was + # excluded before this method was called.) + # + # Many of these are also reserved characters that cannot + # appear in Windows filenames. + for byte in markup_b: + if byte in b"?*#&;>$|": + return False + + # Two consecutive forward slashes (as seen in a URL) or two + # consecutive spaces (as seen in fixed-width data). + # + # (Paths to Windows network shares contain consecutive + # backslashes, so checking that doesn't seem as helpful.) + if b"//" in markup_b: + return False + if b" " in markup_b: + return False + + # A colon in any position other than position 1 (e.g. after a + # Windows drive letter). + if markup_b.startswith(b":"): + return False + colon_i =3D markup_b.rfind(b":") + if colon_i not in (-1, 1): + return False + + # Step 3: If it survived all of those checks, it's similar + # enough to a file to justify issuing a warning. + warnings.warn( + MarkupResemblesLocatorWarning.FILENAME_MESSAGE % dict(what=3D"= filename"), + MarkupResemblesLocatorWarning, + stacklevel=3D3, + ) + return True + + def _feed(self) -> None: + """Internal method that parses previously set markup, creating a l= arge + number of Tag and NavigableString objects. + """ + # Convert the document to Unicode. + self.builder.reset() + + if self.markup is not None: + self.builder.feed(self.markup) + # Close out any unfinished strings and close all the open tags. + self.endData() + while ( + self.currentTag is not None and self.currentTag.name !=3D self= .ROOT_TAG_NAME + ): + self.popTag() + + def reset(self) -> None: + """Reset this object to a state as though it had never parsed any + markup. + """ + Tag.__init__(self, self, self.builder, self.ROOT_TAG_NAME) + self.hidden =3D True + self.builder.reset() + self.current_data =3D [] + self.currentTag =3D None + self.tagStack =3D [] + self.open_tag_counter =3D Counter() + self.preserve_whitespace_tag_stack =3D [] + self.string_container_stack =3D [] + self._most_recent_element =3D None + self.pushTag(self) + + def new_tag( + self, + name: str, + namespace: Optional[str] =3D None, + nsprefix: Optional[str] =3D None, + attrs: Optional[_RawAttributeValues] =3D None, + sourceline: Optional[int] =3D None, + sourcepos: Optional[int] =3D None, + string: Optional[str] =3D None, + **kwattrs: _RawAttributeValue, + ) -> Tag: + """Create a new Tag associated with this BeautifulSoup object. + + :param name: The name of the new Tag. + :param namespace: The URI of the new Tag's XML namespace, if any. + :param prefix: The prefix for the new Tag's XML namespace, if any. + :param attrs: A dictionary of this Tag's attribute values; can + be used instead of ``kwattrs`` for attributes like 'class' + that are reserved words in Python. + :param sourceline: The line number where this tag was + (purportedly) found in its source document. + :param sourcepos: The character position within ``sourceline`` whe= re this + tag was (purportedly) found. + :param string: String content for the new Tag, if any. + :param kwattrs: Keyword arguments for the new Tag's attribute valu= es. + + """ + attr_container =3D self.builder.attribute_dict_class(**kwattrs) + if attrs is not None: + attr_container.update(attrs) + tag_class =3D self.element_classes.get(Tag, Tag) + + # Assume that this is either Tag or a subclass of Tag. If not, + # the user brought type-unsafety upon themselves. + tag_class =3D cast(Type[Tag], tag_class) + tag =3D tag_class( + None, + self.builder, + name, + namespace, + nsprefix, + attr_container, + sourceline=3Dsourceline, + sourcepos=3Dsourcepos, + ) + + if string is not None: + tag.string =3D string + return tag + + def string_container( + self, base_class: Optional[Type[NavigableString]] =3D None + ) -> Type[NavigableString]: + """Find the class that should be instantiated to hold a given kind= of + string. + + This may be a built-in Beautiful Soup class or a custom class pass= ed + in to the BeautifulSoup constructor. + """ + container =3D base_class or NavigableString + + # The user may want us to use some other class (hopefully a + # custom subclass) instead of the one we'd use normally. + container =3D cast( + Type[NavigableString], self.element_classes.get(container, con= tainer) + ) + + # On top of that, we may be inside a tag that needs a special + # container class. + if self.string_container_stack and container is NavigableString: + container =3D self.builder.string_containers.get( + self.string_container_stack[-1].name, container + ) + return container + + def new_string( + self, s: str, subclass: Optional[Type[NavigableString]] =3D None + ) -> NavigableString: + """Create a new `NavigableString` associated with this `BeautifulS= oup` + object. + + :param s: The string content of the `NavigableString` + :param subclass: The subclass of `NavigableString`, if any, to + use. If a document is being processed, an appropriate + subclass for the current location in the document will + be determined automatically. + """ + container =3D self.string_container(subclass) + return container(s) + + def insert_before(self, *args: _InsertableElement) -> List[PageElement= ]: + """This method is part of the PageElement API, but `BeautifulSoup`= doesn't implement + it because there is nothing before or after it in the parse tree. + """ + raise NotImplementedError( + "BeautifulSoup objects don't support insert_before()." + ) + + def insert_after(self, *args: _InsertableElement) -> List[PageElement]= : + """This method is part of the PageElement API, but `BeautifulSoup`= doesn't implement + it because there is nothing before or after it in the parse tree. + """ + raise NotImplementedError("BeautifulSoup objects don't support ins= ert_after().") + + def popTag(self) -> Optional[Tag]: + """Internal method called by _popToTag when a tag is closed. + + :meta private: + """ + if not self.tagStack: + # Nothing to pop. This shouldn't happen. + return None + tag =3D self.tagStack.pop() + if tag.name in self.open_tag_counter: + self.open_tag_counter[tag.name] -=3D 1 + if ( + self.preserve_whitespace_tag_stack + and tag =3D=3D self.preserve_whitespace_tag_stack[-1] + ): + self.preserve_whitespace_tag_stack.pop() + if self.string_container_stack and tag =3D=3D self.string_containe= r_stack[-1]: + self.string_container_stack.pop() + # print("Pop", tag.name) + if self.tagStack: + self.currentTag =3D self.tagStack[-1] + return self.currentTag + + def pushTag(self, tag: Tag) -> None: + """Internal method called by handle_starttag when a tag is opened. + + :meta private: + """ + # print("Push", tag.name) + if self.currentTag is not None: + self.currentTag.contents.append(tag) + self.tagStack.append(tag) + self.currentTag =3D self.tagStack[-1] + if tag.name !=3D self.ROOT_TAG_NAME: + self.open_tag_counter[tag.name] +=3D 1 + if tag.name in self.builder.preserve_whitespace_tags: + self.preserve_whitespace_tag_stack.append(tag) + if tag.name in self.builder.string_containers: + self.string_container_stack.append(tag) + + def endData(self, containerClass: Optional[Type[NavigableString]] =3D = None) -> None: + """Method called by the TreeBuilder when the end of a data segment + occurs. + + :param containerClass: The class to use when incorporating the + data segment into the parse tree. + + :meta private: + """ + if self.current_data: + current_data =3D "".join(self.current_data) + # If whitespace is not preserved, and this string contains + # nothing but ASCII spaces, replace it with a single space + # or newline. + if not self.preserve_whitespace_tag_stack: + strippable =3D True + for i in current_data: + if i not in self.ASCII_SPACES: + strippable =3D False + break + if strippable: + if "\n" in current_data: + current_data =3D "\n" + else: + current_data =3D " " + + # Reset the data collector. + self.current_data =3D [] + + # Should we add this string to the tree at all? + if ( + self.parse_only + and len(self.tagStack) <=3D 1 + and (not self.parse_only.allow_string_creation(current_dat= a)) + ): + return + + containerClass =3D self.string_container(containerClass) + o =3D containerClass(current_data) + self.object_was_parsed(o) + + def object_was_parsed( + self, + o: PageElement, + parent: Optional[Tag] =3D None, + most_recent_element: Optional[PageElement] =3D None, + ) -> None: + """Method called by the TreeBuilder to integrate an object into th= e + parse tree. + + :meta private: + """ + if parent is None: + parent =3D self.currentTag + assert parent is not None + previous_element: Optional[PageElement] + if most_recent_element is not None: + previous_element =3D most_recent_element + else: + previous_element =3D self._most_recent_element + + next_element =3D previous_sibling =3D next_sibling =3D None + if isinstance(o, Tag): + next_element =3D o.next_element + next_sibling =3D o.next_sibling + previous_sibling =3D o.previous_sibling + if previous_element is None: + previous_element =3D o.previous_element + + fix =3D parent.next_element is not None + + o.setup(parent, previous_element, next_element, previous_sibling, = next_sibling) + + self._most_recent_element =3D o + parent.contents.append(o) + + # Check if we are inserting into an already parsed node. + if fix: + self._linkage_fixer(parent) + + def _linkage_fixer(self, el: Tag) -> None: + """Make sure linkage of this fragment is sound.""" + + first =3D el.contents[0] + child =3D el.contents[-1] + descendant: PageElement =3D child + + if child is first and el.parent is not None: + # Parent should be linked to first child + el.next_element =3D child + # We are no longer linked to whatever this element is + prev_el =3D child.previous_element + if prev_el is not None and prev_el is not el: + prev_el.next_element =3D None + # First child should be linked to the parent, and no previous = siblings. + child.previous_element =3D el + child.previous_sibling =3D None + + # We have no sibling as we've been appended as the last. + child.next_sibling =3D None + + # This index is a tag, dig deeper for a "last descendant" + if isinstance(child, Tag) and child.contents: + # _last_decendant is typed as returning Optional[PageElement], + # but the value can't be None here, because el is a Tag + # which we know has contents. + descendant =3D cast(PageElement, child._last_descendant(False)= ) + + # As the final step, link last descendant. It should be linked + # to the parent's next sibling (if found), else walk up the chain + # and find a parent with a sibling. It should have no next sibling= . + descendant.next_element =3D None + descendant.next_sibling =3D None + + target: Optional[Tag] =3D el + while True: + if target is None: + break + elif target.next_sibling is not None: + descendant.next_element =3D target.next_sibling + target.next_sibling.previous_element =3D child + break + target =3D target.parent + + def _popToTag( + self, name: str, nsprefix: Optional[str] =3D None, inclusivePop: b= ool =3D True + ) -> Optional[Tag]: + """Pops the tag stack up to and including the most recent + instance of the given tag. + + If there are no open tags with the given name, nothing will be + popped. + + :param name: Pop up to the most recent tag with this name. + :param nsprefix: The namespace prefix that goes with `name`. + :param inclusivePop: It this is false, pops the tag stack up + to but *not* including the most recent instqance of the + given tag. + + :meta private: + """ + # print("Popping to %s" % name) + if name =3D=3D self.ROOT_TAG_NAME: + # The BeautifulSoup object itself can never be popped. + return None + + most_recently_popped =3D None + + stack_size =3D len(self.tagStack) + for i in range(stack_size - 1, 0, -1): + if not self.open_tag_counter.get(name): + break + t =3D self.tagStack[i] + if name =3D=3D t.name and nsprefix =3D=3D t.prefix: + if inclusivePop: + most_recently_popped =3D self.popTag() + break + most_recently_popped =3D self.popTag() + + return most_recently_popped + + def handle_starttag( + self, + name: str, + namespace: Optional[str], + nsprefix: Optional[str], + attrs: _RawAttributeValues, + sourceline: Optional[int] =3D None, + sourcepos: Optional[int] =3D None, + namespaces: Optional[Dict[str, str]] =3D None, + ) -> Optional[Tag]: + """Called by the tree builder when a new tag is encountered. + + :param name: Name of the tag. + :param nsprefix: Namespace prefix for the tag. + :param attrs: A dictionary of attribute values. Note that + attribute values are expected to be simple strings; processing + of multi-valued attributes such as "class" comes later. + :param sourceline: The line number where this tag was found in its + source document. + :param sourcepos: The character position within `sourceline` where= this + tag was found. + :param namespaces: A dictionary of all namespace prefix mappings + currently in scope in the document. + + If this method returns None, the tag was rejected by an active + `ElementFilter`. You should proceed as if the tag had not occurred + in the document. For instance, if this was a self-closing tag, + don't call handle_endtag. + + :meta private: + """ + # print("Start tag %s: %s" % (name, attrs)) + self.endData() + + if ( + self.parse_only + and len(self.tagStack) <=3D 1 + and not self.parse_only.allow_tag_creation(nsprefix, name, att= rs) + ): + return None + + tag_class =3D self.element_classes.get(Tag, Tag) + # Assume that this is either Tag or a subclass of Tag. If not, + # the user brought type-unsafety upon themselves. + tag_class =3D cast(Type[Tag], tag_class) + tag =3D tag_class( + self, + self.builder, + name, + namespace, + nsprefix, + attrs, + self.currentTag, + self._most_recent_element, + sourceline=3Dsourceline, + sourcepos=3Dsourcepos, + namespaces=3Dnamespaces, + ) + if tag is None: + return tag + if self._most_recent_element is not None: + self._most_recent_element.next_element =3D tag + self._most_recent_element =3D tag + self.pushTag(tag) + return tag + + def handle_endtag(self, name: str, nsprefix: Optional[str] =3D None) -= > None: + """Called by the tree builder when an ending tag is encountered. + + :param name: Name of the tag. + :param nsprefix: Namespace prefix for the tag. + + :meta private: + """ + # print("End tag: " + name) + self.endData() + self._popToTag(name, nsprefix) + + def handle_data(self, data: str) -> None: + """Called by the tree builder when a chunk of textual data is + encountered. + + :meta private: + """ + self.current_data.append(data) + + def decode( + self, + indent_level: Optional[int] =3D None, + eventual_encoding: _Encoding =3D DEFAULT_OUTPUT_ENCODING, + formatter: Union[Formatter, str] =3D "minimal", + iterator: Optional[Iterator[PageElement]] =3D None, + **kwargs: Any, + ) -> str: + """Returns a string representation of the parse tree + as a full HTML or XML document. + + :param indent_level: Each line of the rendering will be + indented this many levels. (The ``formatter`` decides what a + 'level' means, in terms of spaces or other characters + output.) This is used internally in recursive calls while + pretty-printing. + :param eventual_encoding: The encoding of the final document. + If this is None, the document will be a Unicode string. + :param formatter: Either a `Formatter` object, or a string naming = one of + the standard formatters. + :param iterator: The iterator to use when navigating over the + parse tree. This is only used by `Tag.decode_contents` and + you probably won't need to use it. + """ + if self.is_xml: + # Print the XML declaration + encoding_part =3D "" + declared_encoding: Optional[str] =3D eventual_encoding + if eventual_encoding in PYTHON_SPECIFIC_ENCODINGS: + # This is a special Python encoding; it can't actually + # go into an XML document because it means nothing + # outside of Python. + declared_encoding =3D None + if declared_encoding is not None: + encoding_part =3D ' encoding=3D"%s"' % declared_encoding + prefix =3D '\n' % encoding_part + else: + prefix =3D "" + + # Prior to 4.13.0, the first argument to this method was a + # bool called pretty_print, which gave the method a different + # signature from its superclass implementation, Tag.decode. + # + # The signatures of the two methods now match, but just in + # case someone is still passing a boolean in as the first + # argument to this method (or a keyword argument with the old + # name), we can handle it and put out a DeprecationWarning. + warning: Optional[str] =3D None + pretty_print: Optional[bool] =3D None + if isinstance(indent_level, bool): + if indent_level is True: + indent_level =3D 0 + elif indent_level is False: + indent_level =3D None + warning =3D f"As of 4.13.0, the first argument to BeautifulSou= p.decode has been changed from bool to int, to match Tag.decode. Pass in a = value of {indent_level} instead." + else: + pretty_print =3D kwargs.pop("pretty_print", None) + assert not kwargs + if pretty_print is not None: + if pretty_print is True: + indent_level =3D 0 + elif pretty_print is False: + indent_level =3D None + warning =3D f"As of 4.13.0, the pretty_print argument to B= eautifulSoup.decode has been removed, to match Tag.decode. Pass in a value = of indent_level=3D{indent_level} instead." + + if warning: + warnings.warn(warning, DeprecationWarning, stacklevel=3D2) + elif indent_level is False or pretty_print is False: + indent_level =3D None + return prefix + super(BeautifulSoup, self).decode( + indent_level, eventual_encoding, formatter, iterator + ) + + +# Aliases to make it easier to get started quickly, e.g. 'from bs4 import = _soup' +_s =3D BeautifulSoup +_soup =3D BeautifulSoup + + +class BeautifulStoneSoup(BeautifulSoup): + """Deprecated interface to an XML parser.""" + + def __init__(self, *args: Any, **kwargs: Any): + kwargs["features"] =3D "xml" + warnings.warn( + "The BeautifulStoneSoup class was deprecated in version 4.0.0.= Instead of using " + 'it, pass features=3D"xml" into the BeautifulSoup constructor.= ', + DeprecationWarning, + stacklevel=3D2, + ) + super(BeautifulStoneSoup, self).__init__(*args, **kwargs) + + +# If this file is run as a script, act as an HTML pretty-printer. +if __name__ =3D=3D "__main__": + soup =3D BeautifulSoup(sys.stdin) + print((soup.prettify())) diff --git a/bitbake/lib/bb/_vendor/bs4/_deprecation.py b/bitbake/lib/bb/_v= endor/bs4/_deprecation.py new file mode 100644 index 00000000..a7b5685b --- /dev/null +++ b/bitbake/lib/bb/_vendor/bs4/_deprecation.py @@ -0,0 +1,80 @@ +"""Helper functions for deprecation. + +This interface is itself unstable and may change without warning. Do +not use these functions yourself, even as a joke. The underscores are +there for a reason. No support will be given. + +In particular, most of this will go away without warning once +Beautiful Soup drops support for Python 3.11, since Python 3.12 +defines a `@typing.deprecated() +decorator. `_ +""" + +import functools +import warnings + +from typing import ( + Any, + Callable, +) + + +def _deprecated_alias(old_name: str, new_name: str, version: str): + """Alias one attribute name to another for backward compatibility + + :meta private: + """ + + @property # type:ignore + def alias(self) -> Any: + ":meta private:" + warnings.warn( + f"Access to deprecated property {old_name}. (Replaced by {new_= name}) -- Deprecated since version {version}.", + DeprecationWarning, + stacklevel=3D2, + ) + return getattr(self, new_name) + + @alias.setter + def alias(self, value: str) -> None: + ":meta private:" + warnings.warn( + f"Write to deprecated property {old_name}. (Replaced by {new_n= ame}) -- Deprecated since version {version}.", + DeprecationWarning, + stacklevel=3D2, + ) + return setattr(self, new_name, value) + + return alias + + +def _deprecated_function_alias( + old_name: str, new_name: str, version: str +) -> Callable[[Any], Any]: + def alias(self, *args: Any, **kwargs: Any) -> Any: + ":meta private:" + warnings.warn( + f"Call to deprecated method {old_name}. (Replaced by {new_name= }) -- Deprecated since version {version}.", + DeprecationWarning, + stacklevel=3D2, + ) + return getattr(self, new_name)(*args, **kwargs) + + return alias + + +def _deprecated(replaced_by: str, version: str) -> Callable: + def deprecate(func: Callable) -> Callable: + @functools.wraps(func) + def with_warning(*args: Any, **kwargs: Any) -> Any: + ":meta private:" + warnings.warn( + f"Call to deprecated method {func.__name__}. (Replaced by = {replaced_by}) -- Deprecated since version {version}.", + DeprecationWarning, + stacklevel=3D2, + ) + return func(*args, **kwargs) + + return with_warning + + return deprecate diff --git a/bitbake/lib/bb/_vendor/bs4/_typing.py b/bitbake/lib/bb/_vendor= /bs4/_typing.py new file mode 100644 index 00000000..caf2567a --- /dev/null +++ b/bitbake/lib/bb/_vendor/bs4/_typing.py @@ -0,0 +1,205 @@ +# Custom type aliases used throughout Beautiful Soup to improve readabilit= y. + +# Notes on improvements to the type system in newer versions of Python +# that can be used once Beautiful Soup drops support for older +# versions: +# +# * ClassVar can be put on class variables now. +# * In 3.10, x|y is an accepted shorthand for Union[x,y]. +# * In 3.10, TypeAlias gains capabilities that can be used to +# improve the tree matching types (I don't remember what, exactly). +# * In 3.9 it's possible to specialize the re.Match type, +# e.g. re.Match[str]. In 3.8 there's a typing.re namespace for this, +# but it's removed in 3.12, so to support the widest possible set of +# versions I'm not using it. + +from bb._vendor.typing_extensions import ( + runtime_checkable, + Protocol, + TypeAlias, +) +from typing import ( + Any, + Callable, + Dict, + IO, + Iterable, + Mapping, + Optional, + Pattern, + TYPE_CHECKING, + Union, +) + +if TYPE_CHECKING: + from bb._vendor.bs4.element import ( + AttributeValueList, + NamespacedAttribute, + NavigableString, + PageElement, + ResultSet, + Tag, + ) + + +@runtime_checkable +class _RegularExpressionProtocol(Protocol): + """A protocol object which can accept either Python's built-in + `re.Pattern` objects, or the similar ``Regex`` objects defined by the + third-party ``regex`` package. + """ + + def search( + self, string: str, pos: int =3D ..., endpos: int =3D ... + ) -> Optional[Any]: ... + + @property + def pattern(self) -> str: ... + + +# Aliases for markup in various stages of processing. +# +#: The rawest form of markup: either a string, bytestring, or an open file= handle. +_IncomingMarkup: TypeAlias =3D Union[str, bytes, IO[str], IO[bytes]] + +#: Markup that is in memory but has (potentially) yet to be converted +#: to Unicode. +_RawMarkup: TypeAlias =3D Union[str, bytes] + +# Aliases for character encodings +# + +#: A data encoding. +_Encoding: TypeAlias =3D str + +#: One or more data encodings. +_Encodings: TypeAlias =3D Iterable[_Encoding] + +# Aliases for XML namespaces +# + +#: The prefix for an XML namespace. +_NamespacePrefix: TypeAlias =3D str + +#: The URL of an XML namespace +_NamespaceURL: TypeAlias =3D str + +#: A mapping of prefixes to namespace URLs. +_NamespaceMapping: TypeAlias =3D Dict[_NamespacePrefix, _NamespaceURL] + +#: A mapping of namespace URLs to prefixes +_InvertedNamespaceMapping: TypeAlias =3D Dict[_NamespaceURL, _NamespacePre= fix] + +# Aliases for the attribute values associated with HTML/XML tags. +# + +#: The value associated with an HTML or XML attribute. This is the +#: relatively unprocessed value Beautiful Soup expects to come from a +#: `TreeBuilder`. +_RawAttributeValue: TypeAlias =3D str + +#: A dictionary of names to `_RawAttributeValue` objects. This is how +#: Beautiful Soup expects a `TreeBuilder` to represent a tag's +#: attribute values. +_RawAttributeValues: TypeAlias =3D ( + "Mapping[Union[str, NamespacedAttribute], _RawAttributeValue]" +) + +#: An attribute value in its final form, as stored in the +# `Tag` class, after it has been processed and (in some cases) +# split into a list of strings. +_AttributeValue: TypeAlias =3D Union[str, "AttributeValueList"] + +#: A dictionary of names to :py:data:`_AttributeValue` objects. This is wh= at +#: a tag's attributes look like after processing. +_AttributeValues: TypeAlias =3D Dict[str, _AttributeValue] + +#: The methods that deal with turning :py:data:`_RawAttributeValue` into +#: :py:data:`_AttributeValue` may be called several times, even after the = values +#: are already processed (e.g. when cloning a tag), so they need to +#: be able to acommodate both possibilities. +_RawOrProcessedAttributeValues: TypeAlias =3D Union[_RawAttributeValues, _= AttributeValues] + +#: A number of tree manipulation methods can take either a `PageElement` o= r a +#: normal Python string (which will be converted to a `NavigableString`). +_InsertableElement: TypeAlias =3D Union["PageElement", str] + +# Aliases to represent the many possibilities for matching bits of a +# parse tree. +# +# This is very complicated because we're applying a formal type system +# to some very DWIM code. The types we end up with will be the types +# of the arguments to the SoupStrainer constructor and (more +# familiarly to Beautiful Soup users) the find* methods. + +#: A function that takes a PageElement and returns a yes-or-no answer. +_PageElementMatchFunction: TypeAlias =3D Callable[["PageElement"], bool] + +#: A function that takes the raw parsed ingredients of a markup tag +#: and returns a yes-or-no answer. +# Not necessary at the moment. +# _AllowTagCreationFunction:TypeAlias =3D Callable[[Optional[str], str, Op= tional[_RawAttributeValues]], bool] + +#: A function that takes the raw parsed ingredients of a markup string nod= e +#: and returns a yes-or-no answer. +# Not necessary at the moment. +# _AllowStringCreationFunction:TypeAlias =3D Callable[[Optional[str]], boo= l] + +#: A function that takes a `Tag` and returns a yes-or-no answer. +#: A `TagNameMatchRule` expects this kind of function, if you're +#: going to pass it a function. +_TagMatchFunction: TypeAlias =3D Callable[["Tag"], bool] + +#: A function that takes a string (or None) and returns a yes-or-no +#: answer. An `AttributeValueMatchRule` expects this kind of function, if +#: you're going to pass it a function. +_NullableStringMatchFunction: TypeAlias =3D Callable[[Optional[str]], bool= ] + +#: A function that takes a string and returns a yes-or-no answer. A +# `StringMatchRule` expects this kind of function, if you're going to +# pass it a function. +_StringMatchFunction: TypeAlias =3D Callable[[str], bool] + +#: Either a tag name, an attribute value or a string can be matched +#: against a string, bytestring, regular expression, or a boolean. +_BaseStrainable: TypeAlias =3D Union[str, bytes, Pattern[str], bool] + +#: A tag can be matched either with the `_BaseStrainable` options, or +#: using a function that takes the `Tag` as its sole argument. +_BaseStrainableElement: TypeAlias =3D Union[_BaseStrainable, _TagMatchFunc= tion] + +#: A tag's attribute value can be matched either with the +#: `_BaseStrainable` options, or using a function that takes that +#: value as its sole argument. +_BaseStrainableAttribute: TypeAlias =3D Union[_BaseStrainable, _NullableSt= ringMatchFunction] + +#: A tag can be matched using either a single criterion or a list of +#: criteria. +_StrainableElement: TypeAlias =3D Union[ + _BaseStrainableElement, Iterable[_BaseStrainableElement] +] + +#: An attribute value can be matched using either a single criterion +#: or a list of criteria. +_StrainableAttribute: TypeAlias =3D Union[ + _BaseStrainableAttribute, Iterable[_BaseStrainableAttribute] +] + +#: An string can be matched using the same techniques as +#: an attribute value. +_StrainableString: TypeAlias =3D _StrainableAttribute + +#: A dictionary may be used to match against multiple attribute vlaues at = once. +_StrainableAttributes: TypeAlias =3D Dict[str, _StrainableAttribute] + +#: Many Beautiful soup methods return a PageElement or an ResultSet of +#: PageElements. A PageElement is either a Tag or a NavigableString. +#: These convenience aliases make it easier for IDE users to see which met= hods +#: are available on the objects they're dealing with. +_OneElement: TypeAlias =3D Union["PageElement", "Tag", "NavigableString"] +_AtMostOneElement: TypeAlias =3D Optional[_OneElement] +_AtMostOneTag: TypeAlias =3D Optional["Tag"] +_AtMostOneNavigableString: TypeAlias =3D Optional["NavigableString"] +_QueryResults: TypeAlias =3D "ResultSet[_OneElement]" +_SomeTags: TypeAlias =3D "ResultSet[Tag]" +_SomeNavigableStrings: TypeAlias =3D "ResultSet[NavigableString]" diff --git a/bitbake/lib/bb/_vendor/bs4/_warnings.py b/bitbake/lib/bb/_vend= or/bs4/_warnings.py new file mode 100644 index 00000000..89041705 --- /dev/null +++ b/bitbake/lib/bb/_vendor/bs4/_warnings.py @@ -0,0 +1,98 @@ +"""Define some custom warnings.""" + + +class GuessedAtParserWarning(UserWarning): + """The warning issued when BeautifulSoup has to guess what parser to + use -- probably because no parser was specified in the constructor. + """ + + MESSAGE: str =3D """No parser was explicitly specified, so I'm using t= he best available %(markup_type)s parser for this system ("%(parser)s"). Th= is usually isn't a problem, but if you run this code on another system, or = in a different virtual environment, it may use a different parser and behav= e differently. + +The code that caused this warning is on line %(line_number)s of the file %= (filename)s. To get rid of this warning, pass the additional argument 'feat= ures=3D"%(parser)s"' to the BeautifulSoup constructor. +""" + + +class UnusualUsageWarning(UserWarning): + """A superclass for warnings issued when Beautiful Soup sees + something that is typically the result of a mistake in the calling + code, but might be intentional on the part of the user. If it is + in fact intentional, you can filter the individual warning class + to get rid of the warning. If you don't like Beautiful Soup + second-guessing what you are doing, you can filter the + UnusualUsageWarningclass itself and get rid of these entirely. + """ + + +class MarkupResemblesLocatorWarning(UnusualUsageWarning): + """The warning issued when BeautifulSoup is given 'markup' that + actually looks like a resource locator -- a URL or a path to a file + on disk. + """ + + #: :meta private: + GENERIC_MESSAGE: str =3D """ + +However, if you want to parse some data that happens to look like a %(what= )s, then nothing has gone wrong: you are using Beautiful Soup correctly, an= d this warning is spurious and can be filtered. To make this warning go awa= y, run this code before calling the BeautifulSoup constructor: + + from bb._vendor.bs4 import MarkupResemblesLocatorWarning + import warnings + + warnings.filterwarnings("ignore", category=3DMarkupResemblesLocatorWar= ning) + """ + + URL_MESSAGE: str =3D ( + """The input passed in on this line looks more like a URL than HTM= L or XML. + +If you meant to use Beautiful Soup to parse the web page found at a certai= n URL, then something has gone wrong. You should use an Python package like= 'requests' to fetch the content behind the URL. Once you have the content = as a string, you can feed that string into Beautiful Soup.""" + + GENERIC_MESSAGE + ) + + FILENAME_MESSAGE: str =3D ( + """The input passed in on this line looks more like a filename tha= n HTML or XML. + +If you meant to use Beautiful Soup to parse the contents of a file on disk= , then something has gone wrong. You should open the file first, using code= like this: + + filehandle =3D open(your filename) + +You can then feed the open filehandle into Beautiful Soup instead of using= the filename.""" + + GENERIC_MESSAGE + ) + + +class AttributeResemblesVariableWarning(UnusualUsageWarning, SyntaxWarning= ): + """The warning issued when Beautiful Soup suspects a provided + attribute name may actually be the misspelled name of a Beautiful + Soup variable. Generally speaking, this is only used in cases like + "_class" where it's very unlikely the user would be referencing an + XML attribute with that name. + """ + + MESSAGE: str =3D """%(original)r is an unusual attribute name and is a= common misspelling for %(autocorrect)r. + +If you meant %(autocorrect)r, change your code to use it, and this warning= will go away. + +If you really did mean to check the %(original)r attribute, this warning i= s spurious and can be filtered. To make it go away, run this code before cr= eating your BeautifulSoup object: + + from bb._vendor.bs4 import AttributeResemblesVariableWarning + import warnings + + warnings.filterwarnings("ignore", category=3DAttributeResemblesVariabl= eWarning) +""" + + +class XMLParsedAsHTMLWarning(UnusualUsageWarning): + """The warning issued when an HTML parser is used to parse + XML that is not (as far as we can tell) XHTML. + """ + + MESSAGE: str =3D """It looks like you're using an HTML parser to parse= an XML document. + +Assuming this really is an XML document, what you're doing might work, but= you should know that using an XML parser will be more reliable. To parse t= his document as XML, make sure you have the Python package 'lxml' installed= , and pass the keyword argument `features=3D"xml"` into the BeautifulSoup c= onstructor. + +If you want or need to use an HTML parser on this document, you can make t= his warning go away by filtering it. To do that, run this code before calli= ng the BeautifulSoup constructor: + + from bb._vendor.bs4 import XMLParsedAsHTMLWarning + import warnings + + warnings.filterwarnings("ignore", category=3DXMLParsedAsHTMLWarning) +""" diff --git a/bitbake/lib/bb/_vendor/bs4/builder/__init__.py b/bitbake/lib/b= b/_vendor/bs4/builder/__init__.py new file mode 100644 index 00000000..c9d5786d --- /dev/null +++ b/bitbake/lib/bb/_vendor/bs4/builder/__init__.py @@ -0,0 +1,848 @@ +from __future__ import annotations + +# Use of this source code is governed by the MIT license. +__license__ =3D "MIT" + +from collections import defaultdict +import re +from types import ModuleType +from typing import ( + Any, + cast, + Dict, + Iterable, + List, + Optional, + Pattern, + Set, + Tuple, + Type, + TYPE_CHECKING, +) +import warnings +import sys +from bb._vendor.bs4.element import ( + AttributeDict, + AttributeValueList, + CharsetMetaAttributeValue, + ContentMetaAttributeValue, + RubyParenthesisString, + RubyTextString, + Stylesheet, + Script, + TemplateString, + nonwhitespace_re, +) + +# Exceptions were moved to their own module in 4.13. Import here for +# backwards compatibility. +from bb._vendor.bs4.exceptions import ParserRejectedMarkup + +from bb._vendor.bs4._typing import ( + _AttributeValues, + _RawAttributeValue, +) + +from bb._vendor.bs4._warnings import XMLParsedAsHTMLWarning + +if TYPE_CHECKING: + from bb._vendor.bs4 import BeautifulSoup + from bb._vendor.bs4.element import ( + NavigableString, + Tag, + ) + from bb._vendor.bs4._typing import ( + _AttributeValue, + _Encoding, + _Encodings, + _RawOrProcessedAttributeValues, + _RawMarkup, + ) + +__all__ =3D [ + "HTMLTreeBuilder", + "SAXTreeBuilder", + "TreeBuilder", + "TreeBuilderRegistry", +] + +# Some useful features for a TreeBuilder to have. +FAST =3D "fast" +PERMISSIVE =3D "permissive" +STRICT =3D "strict" +XML =3D "xml" +HTML =3D "html" +HTML_5 =3D "html5" + +__all__ =3D [ + "TreeBuilderRegistry", + "TreeBuilder", + "HTMLTreeBuilder", + "DetectsXMLParsedAsHTML", + + "ParserRejectedMarkup", # backwards compatibility only as of 4.13.0 +] + +class TreeBuilderRegistry(object): + """A way of looking up TreeBuilder subclasses by their name or by desi= red + features. + """ + + builders_for_feature: Dict[str, List[Type[TreeBuilder]]] + builders: List[Type[TreeBuilder]] + + def __init__(self) -> None: + self.builders_for_feature =3D defaultdict(list) + self.builders =3D [] + + def register(self, treebuilder_class: type[TreeBuilder]) -> None: + """Register a treebuilder based on its advertised features. + + :param treebuilder_class: A subclass of `TreeBuilder`. its + `TreeBuilder.features` attribute should list its features. + """ + for feature in treebuilder_class.features: + self.builders_for_feature[feature].insert(0, treebuilder_class= ) + self.builders.insert(0, treebuilder_class) + + def lookup(self, *features: str) -> Optional[Type[TreeBuilder]]: + """Look up a TreeBuilder subclass with the desired features. + + :param features: A list of features to look for. If none are + provided, the most recently registered TreeBuilder subclass + will be used. + :return: A TreeBuilder subclass, or None if there's no + registered subclass with all the requested features. + """ + if len(self.builders) =3D=3D 0: + # There are no builders at all. + return None + + if len(features) =3D=3D 0: + # They didn't ask for any features. Give them the most + # recently registered builder. + return self.builders[0] + + # Go down the list of features in order, and eliminate any builder= s + # that don't match every feature. + feature_list =3D list(features) + feature_list.reverse() + candidates =3D None + candidate_set =3D None + while len(feature_list) > 0: + feature =3D feature_list.pop() + we_have_the_feature =3D self.builders_for_feature.get(feature,= []) + if len(we_have_the_feature) > 0: + if candidates is None: + candidates =3D we_have_the_feature + candidate_set =3D set(candidates) + elif candidate_set is not None: + # Eliminate any candidates that don't have this featur= e. + candidate_set =3D candidate_set.intersection(set(we_ha= ve_the_feature)) + + # The only valid candidates are the ones in candidate_set. + # Go through the original list of candidates and pick the first on= e + # that's in candidate_set. + if candidate_set is None or candidates is None: + return None + for candidate in candidates: + if candidate in candidate_set: + return candidate + return None + + +#: The `BeautifulSoup` constructor will take a list of features +#: and use it to look up `TreeBuilder` classes in this registry. +builder_registry: TreeBuilderRegistry =3D TreeBuilderRegistry() + + +class TreeBuilder(object): + """Turn a textual document into a Beautiful Soup object tree. + + This is an abstract superclass which smooths out the behavior of + different parser libraries into a single, unified interface. + + :param multi_valued_attributes: If this is set to None, the + TreeBuilder will not turn any values for attributes like + 'class' into lists. Setting this to a dictionary will + customize this behavior; look at :py:attr:`bs4.builder.HTMLTreeBuilde= r.DEFAULT_CDATA_LIST_ATTRIBUTES` + for an example. + + Internally, these are called "CDATA list attributes", but that + probably doesn't make sense to an end-user, so the argument name + is ``multi_valued_attributes``. + + :param preserve_whitespace_tags: A set of tags to treat + the way 
 tags are treated in HTML. Tags in this set
+     are immune from pretty-printing; their contents will always be
+     output as-is.
+
+    :param string_containers: A dictionary mapping tag names to
+     the classes that should be instantiated to contain the textual
+     contents of those tags. The default is to use NavigableString
+     for every tag, no matter what the name. You can override the
+     default by changing :py:attr:`DEFAULT_STRING_CONTAINERS`.
+
+    :param store_line_numbers: If the parser keeps track of the line
+     numbers and positions of the original markup, that information
+     will, by default, be stored in each corresponding
+     :py:class:`bs4.element.Tag` object. You can turn this off by
+     passing store_line_numbers=3DFalse; then Tag.sourcepos and
+     Tag.sourceline will always be None. If the parser you're using
+     doesn't keep track of this information, then store_line_numbers
+     is irrelevant.
+
+    :param attribute_dict_class: The value of a multi-valued attribute
+      (such as HTML's 'class') willl be stored in an instance of this
+      class.  The default is Beautiful Soup's built-in
+      `AttributeValueList`, which is a normal Python list, and you
+      will probably never need to change it.
+    """
+
+    USE_DEFAULT: Any =3D object()  #: :meta private:
+
+    def __init__(
+        self,
+        multi_valued_attributes: Dict[str, Set[str]] =3D USE_DEFAULT,
+        preserve_whitespace_tags: Set[str] =3D USE_DEFAULT,
+        store_line_numbers: bool =3D USE_DEFAULT,
+        string_containers: Dict[str, Type[NavigableString]] =3D USE_DEFAUL=
T,
+        empty_element_tags: Set[str] =3D USE_DEFAULT,
+        attribute_dict_class: Type[AttributeDict] =3D AttributeDict,
+        attribute_value_list_class: Type[AttributeValueList] =3D Attribute=
ValueList,
+    ):
+        self.soup =3D None
+        if multi_valued_attributes is self.USE_DEFAULT:
+            multi_valued_attributes =3D self.DEFAULT_CDATA_LIST_ATTRIBUTES
+        self.cdata_list_attributes =3D multi_valued_attributes
+        if preserve_whitespace_tags is self.USE_DEFAULT:
+            preserve_whitespace_tags =3D self.DEFAULT_PRESERVE_WHITESPACE_=
TAGS
+        self.preserve_whitespace_tags =3D preserve_whitespace_tags
+        if empty_element_tags is self.USE_DEFAULT:
+            self.empty_element_tags =3D self.DEFAULT_EMPTY_ELEMENT_TAGS
+        else:
+            self.empty_element_tags =3D empty_element_tags
+        # TODO: store_line_numbers is probably irrelevant now that
+        # the behavior of sourceline and sourcepos has been made consisten=
t
+        # everywhere.
+        if store_line_numbers =3D=3D self.USE_DEFAULT:
+            store_line_numbers =3D self.TRACKS_LINE_NUMBERS
+        self.store_line_numbers =3D store_line_numbers
+        if string_containers =3D=3D self.USE_DEFAULT:
+            string_containers =3D self.DEFAULT_STRING_CONTAINERS
+        self.string_containers =3D string_containers
+        self.attribute_dict_class =3D attribute_dict_class
+        self.attribute_value_list_class =3D attribute_value_list_class
+
+    NAME: str =3D "[Unknown tree builder]"
+    ALTERNATE_NAMES: Iterable[str] =3D []
+    features: Iterable[str] =3D []
+
+    is_xml: bool =3D False
+    picklable: bool =3D False
+
+    soup: Optional[BeautifulSoup]  #: :meta private:
+
+    #: A tag will be considered an empty-element
+    #: tag when and only when it has no contents.
+    empty_element_tags: Optional[Set[str]] =3D None  #: :meta private:
+    cdata_list_attributes: Dict[str, Set[str]]  #: :meta private:
+    preserve_whitespace_tags: Set[str]  #: :meta private:
+    string_containers: Dict[str, Type[NavigableString]]  #: :meta private:
+    tracks_line_numbers: bool  #: :meta private:
+
+    #: A value for these tag/attribute combinations is a space- or
+    #: comma-separated list of CDATA, rather than a single CDATA.
+    DEFAULT_CDATA_LIST_ATTRIBUTES: Dict[str, Set[str]] =3D defaultdict(set=
)
+
+    #: Whitespace should be preserved inside these tags.
+    DEFAULT_PRESERVE_WHITESPACE_TAGS: Set[str] =3D set()
+
+    #: The textual contents of tags with these names should be
+    #: instantiated with some class other than `bs4.element.NavigableStrin=
g`.
+    DEFAULT_STRING_CONTAINERS: Dict[str, Type[bs4.element.NavigableString]=
] =3D {} # type:ignore
+
+    #: By default, tags are treated as empty-element tags if they have
+    #: no contents--that is, using XML rules. HTMLTreeBuilder
+    #: defines a different set of DEFAULT_EMPTY_ELEMENT_TAGS based on the
+    #: HTML 4 and HTML5 standards.
+    DEFAULT_EMPTY_ELEMENT_TAGS: Optional[Set[str]] =3D None
+
+    #: Most parsers don't keep track of line numbers.
+    TRACKS_LINE_NUMBERS: bool =3D False
+
+    def initialize_soup(self, soup: BeautifulSoup) -> None:
+        """The BeautifulSoup object has been initialized and is now
+        being associated with the TreeBuilder.
+
+        :param soup: A BeautifulSoup object.
+        """
+        self.soup =3D soup
+
+    def reset(self) -> None:
+        """Do any work necessary to reset the underlying parser
+        for a new document.
+
+        By default, this does nothing.
+        """
+        pass
+
+    def can_be_empty_element(self, tag_name: str) -> bool:
+        """Might a tag with this name be an empty-element tag?
+
+        The final markup may or may not actually present this tag as
+        self-closing.
+
+        For instance: an HTMLBuilder does not consider a 
 tag to be
+        an empty-element tag (it's not in
+        HTMLBuilder.empty_element_tags). This means an empty 
 tag
+        will be presented as "
", not "
" or "
".
+
+        The default implementation has no opinion about which tags are
+        empty-element tags, so a tag will be presented as an
+        empty-element tag if and only if it has no children.
+        "" will become "", and "bar" will
+        be left alone.
+
+        :param tag_name: The name of a markup tag.
+        """
+        if self.empty_element_tags is None:
+            return True
+        return tag_name in self.empty_element_tags
+
+    def feed(self, markup: _RawMarkup) -> None:
+        """Run incoming markup through some parsing process."""
+        raise NotImplementedError()
+
+    def prepare_markup(
+        self,
+        markup: _RawMarkup,
+        user_specified_encoding: Optional[_Encoding] =3D None,
+        document_declared_encoding: Optional[_Encoding] =3D None,
+        exclude_encodings: Optional[_Encodings] =3D None,
+    ) -> Iterable[Tuple[_RawMarkup, Optional[_Encoding], Optional[_Encodin=
g], bool]]:
+        """Run any preliminary steps necessary to make incoming markup
+        acceptable to the parser.
+
+        :param markup: The markup that's about to be parsed.
+        :param user_specified_encoding: The user asked to try this encodin=
g
+           to convert the markup into a Unicode string.
+        :param document_declared_encoding: The markup itself claims to be
+            in this encoding. NOTE: This argument is not used by the
+            calling code and can probably be removed.
+        :param exclude_encodings: The user asked *not* to try any of
+            these encodings.
+
+        :yield: A series of 4-tuples: (markup, encoding, declared encoding=
,
+            has undergone character replacement)
+
+            Each 4-tuple represents a strategy that the parser can try
+            to convert the document to Unicode and parse it. Each
+            strategy will be tried in turn.
+
+         By default, the only strategy is to parse the markup
+         as-is. See `LXMLTreeBuilderForXML` and
+         `HTMLParserTreeBuilder` for implementations that take into
+         account the quirks of particular parsers.
+
+        :meta private:
+
+        """
+        yield markup, None, None, False
+
+    def test_fragment_to_document(self, fragment: str) -> str:
+        """Wrap an HTML fragment to make it look like a document.
+
+        Different parsers do this differently. For instance, lxml
+        introduces an empty  tag, and html5lib
+        doesn't. Abstracting this away lets us write simple tests
+        which run HTML fragments through the parser and compare the
+        results against other HTML fragments.
+
+        This method should not be used outside of unit tests.
+
+        :param fragment: A fragment of HTML.
+        :return: A full HTML document.
+        :meta private:
+        """
+        return fragment
+
+    def set_up_substitutions(self, tag: Tag) -> bool:
+        """Set up any substitutions that will need to be performed on
+        a `Tag` when it's output as a string.
+
+        By default, this does nothing. See `HTMLTreeBuilder` for a
+        case where this is used.
+
+        :return: Whether or not a substitution was performed.
+        :meta private:
+        """
+        return False
+
+    def _replace_cdata_list_attribute_values(
+        self, tag_name: str, attrs: _RawOrProcessedAttributeValues
+    ) -> _AttributeValues:
+        """When an attribute value is associated with a tag that can
+        have multiple values for that attribute, convert the string
+        value to a list of strings.
+
+        Basically, replaces class=3D"foo bar" with class=3D["foo", "bar"]
+
+        NOTE: This method modifies its input in place.
+
+        :param tag_name: The name of a tag.
+        :param attrs: A dictionary containing the tag's attributes.
+           Any appropriate attribute values will be modified in place.
+        :return: The modified dictionary that was originally passed in.
+        """
+
+        # First, cast the attrs dict to _AttributeValues. This might
+        # not be accurate yet, but it will be by the time this method
+        # returns.
+        modified_attrs =3D cast(_AttributeValues, attrs)
+        if not modified_attrs or not self.cdata_list_attributes:
+            # Nothing to do.
+            return modified_attrs
+
+        # There is at least a possibility that we need to modify one of
+        # the attribute values.
+        universal: Set[str] =3D self.cdata_list_attributes.get("*", set())
+        tag_specific =3D self.cdata_list_attributes.get(tag_name.lower(), =
None)
+        for attr in list(modified_attrs.keys()):
+            modified_value: _AttributeValue
+            if attr in universal or (tag_specific and attr in tag_specific=
):
+                # We have a "class"-type attribute whose string
+                # value is a whitespace-separated list of
+                # values. Split it into a list.
+                original_value: _AttributeValue =3D modified_attrs[attr]
+                if isinstance(original_value, _RawAttributeValue):
+                    # This is a _RawAttributeValue (a string) that
+                    # needs to be split and converted to a
+                    # AttributeValueList so it can be an
+                    # _AttributeValue.
+                    modified_value =3D self.attribute_value_list_class(
+                        nonwhitespace_re.findall(original_value)
+                    )
+                else:
+                    # html5lib calls setAttributes twice for the
+                    # same tag when rearranging the parse tree. On
+                    # the second call the attribute value here is
+                    # already a list. This can also happen when a
+                    # Tag object is cloned. If this happens, leave
+                    # the value alone rather than trying to split
+                    # it again.
+                    modified_value =3D original_value
+                modified_attrs[attr] =3D modified_value
+        return modified_attrs
+
+
+class SAXTreeBuilder(TreeBuilder):
+    """A Beautiful Soup treebuilder that listens for SAX events.
+
+    This is not currently used for anything, and it will be removed
+    soon. It was a good idea, but it wasn't properly integrated into the
+    rest of Beautiful Soup, so there have been long stretches where it
+    hasn't worked properly.
+    """
+
+    def __init__(self, *args: Any, **kwargs: Any) -> None:
+        warnings.warn(
+            "The SAXTreeBuilder class was deprecated in 4.13.0 and will be=
 removed soon thereafter. It is completely untested and probably doesn't wo=
rk; do not use it.",
+            DeprecationWarning,
+            stacklevel=3D2,
+        )
+        super(SAXTreeBuilder, self).__init__(*args, **kwargs)
+
+    def feed(self, markup: _RawMarkup) -> None:
+        raise NotImplementedError()
+
+    def close(self) -> None:
+        pass
+
+    def startElement(self, name: str, attrs: Dict[str, str]) -> None:
+        attrs =3D AttributeDict((key[1], value) for key, value in list(att=
rs.items()))
+        # print("Start %s, %r" % (name, attrs))
+        assert self.soup is not None
+        self.soup.handle_starttag(name, None, None, attrs)
+
+    def endElement(self, name: str) -> None:
+        # print("End %s" % name)
+        assert self.soup is not None
+        self.soup.handle_endtag(name)
+
+    def startElementNS(
+        self, nsTuple: Tuple[str, str], nodeName: str, attrs: Dict[str, st=
r]
+    ) -> None:
+        # Throw away (ns, nodeName) for now.
+        self.startElement(nodeName, attrs)
+
+    def endElementNS(self, nsTuple: Tuple[str, str], nodeName: str) -> Non=
e:
+        # Throw away (ns, nodeName) for now.
+        self.endElement(nodeName)
+        # handler.endElementNS((ns, node.nodeName), node.nodeName)
+
+    def startPrefixMapping(self, prefix: str, nodeValue: str) -> None:
+        # Ignore the prefix for now.
+        pass
+
+    def endPrefixMapping(self, prefix: str) -> None:
+        # Ignore the prefix for now.
+        # handler.endPrefixMapping(prefix)
+        pass
+
+    def characters(self, content: str) -> None:
+        assert self.soup is not None
+        self.soup.handle_data(content)
+
+    def startDocument(self) -> None:
+        pass
+
+    def endDocument(self) -> None:
+        pass
+
+
+class HTMLTreeBuilder(TreeBuilder):
+    """This TreeBuilder knows facts about HTML, such as which tags are tre=
ated
+    specially by the HTML standard.
+    """
+
+    #: Some HTML tags are defined as having no contents. Beautiful Soup
+    #: treats these specially.
+    DEFAULT_EMPTY_ELEMENT_TAGS: Optional[Set[str]] =3D set(
+        [
+            # These are from HTML5.
+            "area",
+            "base",
+            "br",
+            "col",
+            "embed",
+            "hr",
+            "img",
+            "input",
+            "keygen",
+            "link",
+            "menuitem",
+            "meta",
+            "param",
+            "source",
+            "track",
+            "wbr",
+            # These are from earlier versions of HTML and are removed in H=
TML5.
+            "basefont",
+            "bgsound",
+            "command",
+            "frame",
+            "image",
+            "isindex",
+            "nextid",
+            "spacer",
+        ]
+    )
+
+    #: The HTML standard defines these tags as block-level elements. Beaut=
iful
+    #: Soup does not treat these elements differently from other elements,
+    #: but it may do so eventually, and this information is available if
+    #: you need to use it.
+    DEFAULT_BLOCK_ELEMENTS: Set[str] =3D set(
+        [
+            "address",
+            "article",
+            "aside",
+            "blockquote",
+            "canvas",
+            "dd",
+            "div",
+            "dl",
+            "dt",
+            "fieldset",
+            "figcaption",
+            "figure",
+            "footer",
+            "form",
+            "h1",
+            "h2",
+            "h3",
+            "h4",
+            "h5",
+            "h6",
+            "header",
+            "hr",
+            "li",
+            "main",
+            "nav",
+            "noscript",
+            "ol",
+            "output",
+            "p",
+            "pre",
+            "section",
+            "table",
+            "tfoot",
+            "ul",
+            "video",
+        ]
+    )
+
+    #: These HTML tags need special treatment so they can be
+    #: represented by a string class other than `bs4.element.NavigableStri=
ng`.
+    #:
+    #: For some of these tags, it's because the HTML standard defines
+    #: an unusual content model for them. I made this list by going
+    #: through the HTML spec
+    #: (https://html.spec.whatwg.org/#metadata-content) and looking for
+    #: "metadata content" elements that can contain strings.
+    #:
+    #: The Ruby tags ( and ) are here despite being normal
+    #: "phrasing content" tags, because the content they contain is
+    #: qualitatively different from other text in the document, and it
+    #: can be useful to be able to distinguish it.
+    #:
+    #: TODO: Arguably 
  foo  
  \tbar\n  \n  
  baz  "=
)
-        # Everything outside the 
 tag is reformatted, but everything
-        # inside is left alone.
-        self.assertEqual(
-            '
\n foo\n 
  \tbar\n  \n  
\n baz\n
',
-            soup.div.prettify())
-
-    def test_prettify_accepts_formatter(self):
-        soup =3D BeautifulSoup("foo", 'html.pars=
er')
-        pretty =3D soup.prettify(formatter =3D lambda x: x.upper())
-        self.assertTrue("FOO" in pretty)
-
-    def test_prettify_outputs_unicode_by_default(self):
-        soup =3D self.soup("")
-        self.assertEqual(str, type(soup.prettify()))
-
-    def test_prettify_can_encode_data(self):
-        soup =3D self.soup("")
-        self.assertEqual(bytes, type(soup.prettify("utf-8")))
-
-    def test_html_entity_substitution_off_by_default(self):
-        markup =3D "Sacr\N{LATIN SMALL LETTER E WITH ACUTE} bleu!"
-        soup =3D self.soup(markup)
-        encoded =3D soup.b.encode("utf-8")
-        self.assertEqual(encoded, markup.encode('utf-8'))
-
-    def test_encoding_substitution(self):
-        # Here's the  tag saying that a document is
-        # encoded in Shift-JIS.
-        meta_tag =3D ('')
-        soup =3D self.soup(meta_tag)
-
-        # Parse the document, and the charset apprears unchanged.
-        self.assertEqual(soup.meta['content'], 'text/html; charset=3Dx-sji=
s')
-
-        # Encode the document into some encoding, and the encoding is
-        # substituted into the meta tag.
-        utf_8 =3D soup.encode("utf-8")
-        self.assertTrue(b"charset=3Dutf-8" in utf_8)
-
-        euc_jp =3D soup.encode("euc_jp")
-        self.assertTrue(b"charset=3Deuc_jp" in euc_jp)
-
-        shift_jis =3D soup.encode("shift-jis")
-        self.assertTrue(b"charset=3Dshift-jis" in shift_jis)
-
-        utf_16_u =3D soup.encode("utf-16").decode("utf-16")
-        self.assertTrue("charset=3Dutf-16" in utf_16_u)
-
-    def test_encoding_substitution_doesnt_happen_if_tag_is_strained(self):
-        markup =3D ('
foo
')
-
-        # Beautiful Soup used to try to rewrite the meta tag even if the
-        # meta tag got filtered out by the strainer. This test makes
-        # sure that doesn't happen.
-        strainer =3D SoupStrainer('pre')
-        soup =3D self.soup(markup, parse_only=3Dstrainer)
-        self.assertEqual(soup.contents[0].name, 'pre')
-
-class TestEncoding(SoupTest):
-    """Test the ability to encode objects into strings."""
-
-    def test_unicode_string_can_be_encoded(self):
-        html =3D "\N{SNOWMAN}"
-        soup =3D self.soup(html)
-        self.assertEqual(soup.b.string.encode("utf-8"),
-                          "\N{SNOWMAN}".encode("utf-8"))
-
-    def test_tag_containing_unicode_string_can_be_encoded(self):
-        html =3D "\N{SNOWMAN}"
-        soup =3D self.soup(html)
-        self.assertEqual(
-            soup.b.encode("utf-8"), html.encode("utf-8"))
-
-    def test_encoding_substitutes_unrecognized_characters_by_default(self)=
:
-        html =3D "\N{SNOWMAN}"
-        soup =3D self.soup(html)
-        self.assertEqual(soup.b.encode("ascii"), b"☃")
-
-    def test_encoding_can_be_made_strict(self):
-        html =3D "\N{SNOWMAN}"
-        soup =3D self.soup(html)
-        self.assertRaises(
-            UnicodeEncodeError, soup.encode, "ascii", errors=3D"strict")
-
-    def test_decode_contents(self):
-        html =3D "\N{SNOWMAN}"
-        soup =3D self.soup(html)
-        self.assertEqual("\N{SNOWMAN}", soup.b.decode_contents())
-
-    def test_encode_contents(self):
-        html =3D "\N{SNOWMAN}"
-        soup =3D self.soup(html)
-        self.assertEqual(
-            "\N{SNOWMAN}".encode("utf8"), soup.b.encode_contents(
-                encoding=3D"utf8"))
-
-    def test_deprecated_renderContents(self):
-        html =3D "\N{SNOWMAN}"
-        soup =3D self.soup(html)
-        self.assertEqual(
-            "\N{SNOWMAN}".encode("utf8"), soup.b.renderContents())
-
-    def test_repr(self):
-        html =3D "\N{SNOWMAN}"
-        soup =3D self.soup(html)
-        if PY3K:
-            self.assertEqual(html, repr(soup))
-        else:
-            self.assertEqual(b'\\u2603', repr(soup))
-
-class TestNavigableStringSubclasses(SoupTest):
-
-    def test_cdata(self):
-        # None of the current builders turn CDATA sections into CData
-        # objects, but you can create them manually.
-        soup =3D self.soup("")
-        cdata =3D CData("foo")
-        soup.insert(1, cdata)
-        self.assertEqual(str(soup), "")
-        self.assertEqual(soup.find(text=3D"foo"), "foo")
-        self.assertEqual(soup.contents[0], "foo")
-
-    def test_cdata_is_never_formatted(self):
-        """Text inside a CData object is passed into the formatter.
-
-        But the return value is ignored.
-        """
-
-        self.count =3D 0
-        def increment(*args):
-            self.count +=3D 1
-            return "BITTER FAILURE"
-
-        soup =3D self.soup("")
-        cdata =3D CData("<><><>")
-        soup.insert(1, cdata)
-        self.assertEqual(
-            b"<><>]]>", soup.encode(formatter=3Dincrement))
-        self.assertEqual(1, self.count)
-
-    def test_doctype_ends_in_newline(self):
-        # Unlike other NavigableString subclasses, a DOCTYPE always ends
-        # in a newline.
-        doctype =3D Doctype("foo")
-        soup =3D self.soup("")
-        soup.insert(1, doctype)
-        self.assertEqual(soup.encode(), b"\n")
-
-    def test_declaration(self):
-        d =3D Declaration("foo")
-        self.assertEqual("", d.output_ready())
-
-class TestSoupSelector(TreeTest):
-
-    HTML =3D """
-
-
-
-The title
-
-
-
-Hello there.
-

-

-
An H1

-
Some text

-
Some more text

-
An H2

-
Another

-Bob
-
Another H2

-me
-
-span1a1
-span1a2 test
-
-span2a1
-
-
-
-

-
-

-
-
-
-
-
-
-
-
-
English

-
English UK

-
English US

-
French

-

-
-

-

-"""
-
-    def setUp(self):
-        self.soup =3D BeautifulSoup(self.HTML, 'html.parser')
-
-    def assertSelects(self, selector, expected_ids):
-        el_ids =3D [el['id'] for el in self.soup.select(selector)]
-        el_ids.sort()
-        expected_ids.sort()
-        self.assertEqual(expected_ids, el_ids,
-            "Selector %s, expected [%s], got [%s]" % (
-                selector, ', '.join(expected_ids), ', '.join(el_ids)
-            )
-        )
-
-    assertSelect =3D assertSelects
-
-    def assertSelectMultiple(self, *tests):
-        for selector, expected_ids in tests:
-            self.assertSelect(selector, expected_ids)
-
-    def test_one_tag_one(self):
-        els =3D self.soup.select('title')
-        self.assertEqual(len(els), 1)
-        self.assertEqual(els[0].name, 'title')
-        self.assertEqual(els[0].contents, ['The title'])
-
-    def test_one_tag_many(self):
-        els =3D self.soup.select('div')
-        self.assertEqual(len(els), 4)
-        for div in els:
-            self.assertEqual(div.name, 'div')
-
-        el =3D self.soup.select_one('div')
-        self.assertEqual('main', el['id'])
-
-    def test_select_one_returns_none_if_no_match(self):
-        match =3D self.soup.select_one('nonexistenttag')
-        self.assertEqual(None, match)
-
-
-    def test_tag_in_tag_one(self):
-        els =3D self.soup.select('div div')
-        self.assertSelects('div div', ['inner', 'data1'])
-
-    def test_tag_in_tag_many(self):
-        for selector in ('html div', 'html body div', 'body div'):
-            self.assertSelects(selector, ['data1', 'main', 'inner', 'foote=
r'])
-
-    def test_tag_no_match(self):
-        self.assertEqual(len(self.soup.select('del')), 0)
-
-    def test_invalid_tag(self):
-        self.assertRaises(ValueError, self.soup.select, 'tag%t')
-
-    def test_select_dashed_tag_ids(self):
-        self.assertSelects('custom-dashed-tag', ['dash1', 'dash2'])
-
-    def test_select_dashed_by_id(self):
-        dashed =3D self.soup.select('custom-dashed-tag[id=3D\"dash2\"]')
-        self.assertEqual(dashed[0].name, 'custom-dashed-tag')
-        self.assertEqual(dashed[0]['id'], 'dash2')
-
-    def test_dashed_tag_text(self):
-        self.assertEqual(self.soup.select('body > custom-dashed-tag')[0].t=
ext, 'Hello there.')
-
-    def test_select_dashed_matches_find_all(self):
-        self.assertEqual(self.soup.select('custom-dashed-tag'), self.soup.=
find_all('custom-dashed-tag'))
-
-    def test_header_tags(self):
-        self.assertSelectMultiple(
-            ('h1', ['header1']),
-            ('h2', ['header2', 'header3']),
-        )
-
-    def test_class_one(self):
-        for selector in ('.onep', 'p.onep', 'html p.onep'):
-            els =3D self.soup.select(selector)
-            self.assertEqual(len(els), 1)
-            self.assertEqual(els[0].name, 'p')
-            self.assertEqual(els[0]['class'], ['onep'])
-
-    def test_class_mismatched_tag(self):
-        els =3D self.soup.select('div.onep')
-        self.assertEqual(len(els), 0)
-
-    def test_one_id(self):
-        for selector in ('div#inner', '#inner', 'div div#inner'):
-            self.assertSelects(selector, ['inner'])
-
-    def test_bad_id(self):
-        els =3D self.soup.select('#doesnotexist')
-        self.assertEqual(len(els), 0)
-
-    def test_items_in_id(self):
-        els =3D self.soup.select('div#inner p')
-        self.assertEqual(len(els), 3)
-        for el in els:
-            self.assertEqual(el.name, 'p')
-        self.assertEqual(els[1]['class'], ['onep'])
-        self.assertFalse(els[0].has_attr('class'))
-
-    def test_a_bunch_of_emptys(self):
-        for selector in ('div#main del', 'div#main div.oops', 'div div#mai=
n'):
-            self.assertEqual(len(self.soup.select(selector)), 0)
-
-    def test_multi_class_support(self):
-        for selector in ('.class1', 'p.class1', '.class2', 'p.class2',
-            '.class3', 'p.class3', 'html p.class2', 'div#inner .class2'):
-            self.assertSelects(selector, ['pmulti'])
-
-    def test_multi_class_selection(self):
-        for selector in ('.class1.class3', '.class3.class2',
-                         '.class1.class2.class3'):
-            self.assertSelects(selector, ['pmulti'])
-
-    def test_child_selector(self):
-        self.assertSelects('.s1 > a', ['s1a1', 's1a2'])
-        self.assertSelects('.s1 > a span', ['s1a2s1'])
-
-    def test_child_selector_id(self):
-        self.assertSelects('.s1 > a#s1a2 span', ['s1a2s1'])
-
-    def test_attribute_equals(self):
-        self.assertSelectMultiple(
-            ('p[class=3D"onep"]', ['p1']),
-            ('p[id=3D"p1"]', ['p1']),
-            ('[class=3D"onep"]', ['p1']),
-            ('[id=3D"p1"]', ['p1']),
-            ('link[rel=3D"stylesheet"]', ['l1']),
-            ('link[type=3D"text/css"]', ['l1']),
-            ('link[href=3D"blah.css"]', ['l1']),
-            ('link[href=3D"no-blah.css"]', []),
-            ('[rel=3D"stylesheet"]', ['l1']),
-            ('[type=3D"text/css"]', ['l1']),
-            ('[href=3D"blah.css"]', ['l1']),
-            ('[href=3D"no-blah.css"]', []),
-            ('p[href=3D"no-blah.css"]', []),
-            ('[href=3D"no-blah.css"]', []),
-        )
-
-    def test_attribute_tilde(self):
-        self.assertSelectMultiple(
-            ('p[class~=3D"class1"]', ['pmulti']),
-            ('p[class~=3D"class2"]', ['pmulti']),
-            ('p[class~=3D"class3"]', ['pmulti']),
-            ('[class~=3D"class1"]', ['pmulti']),
-            ('[class~=3D"class2"]', ['pmulti']),
-            ('[class~=3D"class3"]', ['pmulti']),
-            ('a[rel~=3D"friend"]', ['bob']),
-            ('a[rel~=3D"met"]', ['bob']),
-            ('[rel~=3D"friend"]', ['bob']),
-            ('[rel~=3D"met"]', ['bob']),
-        )
-
-    def test_attribute_startswith(self):
-        self.assertSelectMultiple(
-            ('[rel^=3D"style"]', ['l1']),
-            ('link[rel^=3D"style"]', ['l1']),
-            ('notlink[rel^=3D"notstyle"]', []),
-            ('[rel^=3D"notstyle"]', []),
-            ('link[rel^=3D"notstyle"]', []),
-            ('link[href^=3D"bla"]', ['l1']),
-            ('a[href^=3D"http://"]', ['bob', 'me']),
-            ('[href^=3D"http://"]', ['bob', 'me']),
-            ('[id^=3D"p"]', ['pmulti', 'p1']),
-            ('[id^=3D"m"]', ['me', 'main']),
-            ('div[id^=3D"m"]', ['main']),
-            ('a[id^=3D"m"]', ['me']),
-            ('div[data-tag^=3D"dashed"]', ['data1'])
-        )
-
-    def test_attribute_endswith(self):
-        self.assertSelectMultiple(
-            ('[href$=3D".css"]', ['l1']),
-            ('link[href$=3D".css"]', ['l1']),
-            ('link[id$=3D"1"]', ['l1']),
-            ('[id$=3D"1"]', ['data1', 'l1', 'p1', 'header1', 's1a1', 's2a1=
', 's1a2s1', 'dash1']),
-            ('div[id$=3D"1"]', ['data1']),
-            ('[id$=3D"noending"]', []),
-        )
-
-    def test_attribute_contains(self):
-        self.assertSelectMultiple(
-            # From test_attribute_startswith
-            ('[rel*=3D"style"]', ['l1']),
-            ('link[rel*=3D"style"]', ['l1']),
-            ('notlink[rel*=3D"notstyle"]', []),
-            ('[rel*=3D"notstyle"]', []),
-            ('link[rel*=3D"notstyle"]', []),
-            ('link[href*=3D"bla"]', ['l1']),
-            ('[href*=3D"http://"]', ['bob', 'me']),
-            ('[id*=3D"p"]', ['pmulti', 'p1']),
-            ('div[id*=3D"m"]', ['main']),
-            ('a[id*=3D"m"]', ['me']),
-            # From test_attribute_endswith
-            ('[href*=3D".css"]', ['l1']),
-            ('link[href*=3D".css"]', ['l1']),
-            ('link[id*=3D"1"]', ['l1']),
-            ('[id*=3D"1"]', ['data1', 'l1', 'p1', 'header1', 's1a1', 's1a2=
', 's2a1', 's1a2s1', 'dash1']),
-            ('div[id*=3D"1"]', ['data1']),
-            ('[id*=3D"noending"]', []),
-            # New for this test
-            ('[href*=3D"."]', ['bob', 'me', 'l1']),
-            ('a[href*=3D"."]', ['bob', 'me']),
-            ('link[href*=3D"."]', ['l1']),
-            ('div[id*=3D"n"]', ['main', 'inner']),
-            ('div[id*=3D"nn"]', ['inner']),
-            ('div[data-tag*=3D"edval"]', ['data1'])
-        )
-
-    def test_attribute_exact_or_hypen(self):
-        self.assertSelectMultiple(
-            ('p[lang|=3D"en"]', ['lang-en', 'lang-en-gb', 'lang-en-us']),
-            ('[lang|=3D"en"]', ['lang-en', 'lang-en-gb', 'lang-en-us']),
-            ('p[lang|=3D"fr"]', ['lang-fr']),
-            ('p[lang|=3D"gb"]', []),
-        )
-
-    def test_attribute_exists(self):
-        self.assertSelectMultiple(
-            ('[rel]', ['l1', 'bob', 'me']),
-            ('link[rel]', ['l1']),
-            ('a[rel]', ['bob', 'me']),
-            ('[lang]', ['lang-en', 'lang-en-gb', 'lang-en-us', 'lang-fr'])=
,
-            ('p[class]', ['p1', 'pmulti']),
-            ('[blah]', []),
-            ('p[blah]', []),
-            ('div[data-tag]', ['data1'])
-        )
-
-    def test_unsupported_pseudoclass(self):
-        self.assertRaises(
-            NotImplementedError, self.soup.select, "a:no-such-pseudoclass"=
)
-
-        self.assertRaises(
-            NotImplementedError, self.soup.select, "a:nth-of-type(a)")
-
-
-    def test_nth_of_type(self):
-        # Try to select first paragraph
-        els =3D self.soup.select('div#inner p:nth-of-type(1)')
-        self.assertEqual(len(els), 1)
-        self.assertEqual(els[0].string, 'Some text')
-
-        # Try to select third paragraph
-        els =3D self.soup.select('div#inner p:nth-of-type(3)')
-        self.assertEqual(len(els), 1)
-        self.assertEqual(els[0].string, 'Another')
-
-        # Try to select (non-existent!) fourth paragraph
-        els =3D self.soup.select('div#inner p:nth-of-type(4)')
-        self.assertEqual(len(els), 0)
-
-        # Pass in an invalid value.
-        self.assertRaises(
-            ValueError, self.soup.select, 'div p:nth-of-type(0)')
-
-    def test_nth_of_type_direct_descendant(self):
-        els =3D self.soup.select('div#inner > p:nth-of-type(1)')
-        self.assertEqual(len(els), 1)
-        self.assertEqual(els[0].string, 'Some text')
-
-    def test_id_child_selector_nth_of_type(self):
-        self.assertSelects('#inner > p:nth-of-type(2)', ['p1'])
-
-    def test_select_on_element(self):
-        # Other tests operate on the tree; this operates on an element
-        # within the tree.
-        inner =3D self.soup.find("div", id=3D"main")
-        selected =3D inner.select("div")
-        # The 
 tag was selected. The 

-        # tag was not.
-        self.assertSelectsIDs(selected, ['inner', 'data1'])
-
-    def test_overspecified_child_id(self):
-        self.assertSelects(".fancy #inner", ['inner'])
-        self.assertSelects(".normal #inner", [])
-
-    def test_adjacent_sibling_selector(self):
-        self.assertSelects('#p1 + h2', ['header2'])
-        self.assertSelects('#p1 + h2 + p', ['pmulti'])
-        self.assertSelects('#p1 + #header2 + .class1', ['pmulti'])
-        self.assertEqual([], self.soup.select('#p1 + p'))
-
-    def test_general_sibling_selector(self):
-        self.assertSelects('#p1 ~ h2', ['header2', 'header3'])
-        self.assertSelects('#p1 ~ #header2', ['header2'])
-        self.assertSelects('#p1 ~ h2 + a', ['me'])
-        self.assertSelects('#p1 ~ h2 + [rel=3D"me"]', ['me'])
-        self.assertEqual([], self.soup.select('#inner ~ h2'))
-
-    def test_dangling_combinator(self):
-        self.assertRaises(ValueError, self.soup.select, 'h1 >')
-
-    def test_sibling_combinator_wont_select_same_tag_twice(self):
-        self.assertSelects('p[lang] ~ p', ['lang-en-gb', 'lang-en-us', 'la=
ng-fr'])
-
-    # Test the selector grouping operator (the comma)
-    def test_multiple_select(self):
-        self.assertSelects('x, y', ['xid', 'yid'])
-
-    def test_multiple_select_with_no_space(self):
-        self.assertSelects('x,y', ['xid', 'yid'])
-
-    def test_multiple_select_with_more_space(self):
-        self.assertSelects('x,    y', ['xid', 'yid'])
-
-    def test_multiple_select_duplicated(self):
-        self.assertSelects('x, x', ['xid'])
-
-    def test_multiple_select_sibling(self):
-        self.assertSelects('x, y ~ p[lang=3Dfr]', ['xid', 'lang-fr'])
-
-    def test_multiple_select_tag_and_direct_descendant(self):
-        self.assertSelects('x, y > z', ['xid', 'zidb'])
-
-    def test_multiple_select_direct_descendant_and_tags(self):
-        self.assertSelects('div > x, y, z', ['xid', 'yid', 'zida', 'zidb',=
 'zidab', 'zidac'])
-
-    def test_multiple_select_indirect_descendant(self):
-        self.assertSelects('div x,y,  z', ['xid', 'yid', 'zida', 'zidb', '=
zidab', 'zidac'])
-
-    def test_invalid_multiple_select(self):
-        self.assertRaises(ValueError, self.soup.select, ',x, y')
-        self.assertRaises(ValueError, self.soup.select, 'x,,y')
-
-    def test_multiple_select_attrs(self):
-        self.assertSelects('p[lang=3Den], p[lang=3Den-gb]', ['lang-en', 'l=
ang-en-gb'])
-
-    def test_multiple_select_ids(self):
-        self.assertSelects('x, y > z[id=3Dzida], z[id=3Dzidab], z[id=3Dzid=
b]', ['xid', 'zidb', 'zidab'])
-
-    def test_multiple_select_nested(self):
-        self.assertSelects('body > div > x, y > z', ['xid', 'zidb'])
-
-
-
diff --git a/bitbake/lib/codegen.py b/bitbake/lib/codegen.py
deleted file mode 100644
index 018b2831..00000000
--- a/bitbake/lib/codegen.py
+++ /dev/null
@@ -1,564 +0,0 @@
-# -*- coding: utf-8 -*-
-"""
-    codegen
-    ~~~~~~~
-
-    Extension to ast that allow ast -> python code generation.
-
-    :copyright: Copyright 2008 by Armin Ronacher.
-    :license: BSD.
-"""
-from ast import *
-
-BOOLOP_SYMBOLS =3D {
-    And:        'and',
-    Or:         'or'
-}
-
-BINOP_SYMBOLS =3D {
-    Add:        '+',
-    Sub:        '-',
-    Mult:       '*',
-    Div:        '/',
-    FloorDiv:   '//',
-    Mod:        '%',
-    LShift:     '<<',
-    RShift:     '>>',
-    BitOr:      '|',
-    BitAnd:     '&',
-    BitXor:     '^'
-}
-
-CMPOP_SYMBOLS =3D {
-    Eq:         '=3D=3D',
-    Gt:         '>',
-    GtE:        '>=3D',
-    In:         'in',
-    Is:         'is',
-    IsNot:      'is not',
-    Lt:         '<',
-    LtE:        '<=3D',
-    NotEq:      '!=3D',
-    NotIn:      'not in'
-}
-
-UNARYOP_SYMBOLS =3D {
-    Invert:     '~',
-    Not:        'not',
-    UAdd:       '+',
-    USub:       '-'
-}
-
-ALL_SYMBOLS =3D {}
-ALL_SYMBOLS.update(BOOLOP_SYMBOLS)
-ALL_SYMBOLS.update(BINOP_SYMBOLS)
-ALL_SYMBOLS.update(CMPOP_SYMBOLS)
-ALL_SYMBOLS.update(UNARYOP_SYMBOLS)
-
-def to_source(node, indent_with=3D' ' * 4, add_line_information=3DFalse):
-    """This function can convert a node tree back into python sourcecode.
-    This is useful for debugging purposes, especially if you're dealing wi=
th
-    custom asts not generated by python itself.
-
-    It could be that the sourcecode is evaluable when the AST itself is no=
t
-    compilable / evaluable.  The reason for this is that the AST contains =
some
-    more data than regular sourcecode does, which is dropped during
-    conversion.
-
-    Each level of indentation is replaced with `indent_with`.  Per default=
 this
-    parameter is equal to four spaces as suggested by PEP 8, but it might =
be
-    adjusted to match the application's styleguide.
-
-    If `add_line_information` is set to `True` comments for the line numbe=
rs
-    of the nodes are added to the output.  This can be used to spot wrong =
line
-    number information of statement nodes.
-    """
-    generator =3D SourceGenerator(indent_with, add_line_information)
-    generator.visit(node)
-    return ''.join(generator.result)
-
-
-class SourceGenerator(NodeVisitor):
-    """This visitor is able to transform a well formed syntax tree into py=
thon
-    sourcecode.  For more details have a look at the docstring of the
-    `node_to_source` function.
-    """
-
-    def __init__(self, indent_with, add_line_information=3DFalse):
-        self.result =3D []
-        self.indent_with =3D indent_with
-        self.add_line_information =3D add_line_information
-        self.indentation =3D 0
-        self.new_lines =3D 0
-
-    def write(self, x):
-        if self.new_lines:
-            if self.result:
-                self.result.append('\n' * self.new_lines)
-            self.result.append(self.indent_with * self.indentation)
-            self.new_lines =3D 0
-        self.result.append(x)
-
-    def newline(self, node=3DNone, extra=3D0):
-        self.new_lines =3D max(self.new_lines, 1 + extra)
-        if node is not None and self.add_line_information:
-            self.write('# line: %s' % node.lineno)
-            self.new_lines =3D 1
-
-    def body(self, statements):
-        self.new_line =3D True
-        self.indentation +=3D 1
-        for stmt in statements:
-            self.visit(stmt)
-        self.indentation -=3D 1
-
-    def body_or_else(self, node):
-        self.body(node.body)
-        if node.orelse:
-            self.newline()
-            self.write('else:')
-            self.body(node.orelse)
-
-    def signature(self, node):
-        want_comma =3D []
-        def write_comma():
-            if want_comma:
-                self.write(', ')
-            else:
-                want_comma.append(True)
-
-        padding =3D [None] * (len(node.args) - len(node.defaults))
-        for arg, default in zip(node.args, padding + node.defaults):
-            write_comma()
-            self.visit(arg)
-            if default is not None:
-                self.write('=3D')
-                self.visit(default)
-        if node.vararg is not None:
-            write_comma()
-            self.write('*' + node.vararg)
-        if node.kwarg is not None:
-            write_comma()
-            self.write('**' + node.kwarg)
-
-    def decorators(self, node):
-        for decorator in node.decorator_list:
-            self.newline(decorator)
-            self.write('@')
-            self.visit(decorator)
-
-    # Statements
-
-    def visit_Assign(self, node):
-        self.newline(node)
-        for idx, target in enumerate(node.targets):
-            if idx:
-                self.write(', ')
-            self.visit(target)
-        self.write(' =3D ')
-        self.visit(node.value)
-
-    def visit_AugAssign(self, node):
-        self.newline(node)
-        self.visit(node.target)
-        self.write(BINOP_SYMBOLS[type(node.op)] + '=3D')
-        self.visit(node.value)
-
-    def visit_ImportFrom(self, node):
-        self.newline(node)
-        self.write('from %s%s import ' % ('.' * node.level, node.module))
-        for idx, item in enumerate(node.names):
-            if idx:
-                self.write(', ')
-            self.write(item)
-
-    def visit_Import(self, node):
-        self.newline(node)
-        for item in node.names:
-            self.write('import ')
-            self.visit(item)
-
-    def visit_Expr(self, node):
-        self.newline(node)
-        self.generic_visit(node)
-
-    def visit_FunctionDef(self, node):
-        self.newline(extra=3D1)
-        self.decorators(node)
-        self.newline(node)
-        self.write('def %s(' % node.name)
-        self.signature(node.args)
-        self.write('):')
-        self.body(node.body)
-
-    def visit_ClassDef(self, node):
-        have_args =3D []
-        def paren_or_comma():
-            if have_args:
-                self.write(', ')
-            else:
-                have_args.append(True)
-                self.write('(')
-
-        self.newline(extra=3D2)
-        self.decorators(node)
-        self.newline(node)
-        self.write('class %s' % node.name)
-        for base in node.bases:
-            paren_or_comma()
-            self.visit(base)
-        # XXX: the if here is used to keep this module compatible
-        #      with python 2.6.
-        if hasattr(node, 'keywords'):
-            for keyword in node.keywords:
-                paren_or_comma()
-                self.write(keyword.arg + '=3D')
-                self.visit(keyword.value)
-            if hasattr(node, 'starargs') and node.starargs is not None:
-                paren_or_comma()
-                self.write('*')
-                self.visit(node.starargs)
-            if hasattr(node, 'kwargs') and node.kwargs is not None:
-                paren_or_comma()
-                self.write('**')
-                self.visit(node.kwargs)
-        self.write(have_args and '):' or ':')
-        self.body(node.body)
-
-    def visit_If(self, node):
-        self.newline(node)
-        self.write('if ')
-        self.visit(node.test)
-        self.write(':')
-        self.body(node.body)
-        while True:
-            else_ =3D node.orelse
-            if len(else_) =3D=3D 1 and isinstance(else_[0], If):
-                node =3D else_[0]
-                self.newline()
-                self.write('elif ')
-                self.visit(node.test)
-                self.write(':')
-                self.body(node.body)
-            else:
-                self.newline()
-                self.write('else:')
-                self.body(else_)
-                break
-
-    def visit_For(self, node):
-        self.newline(node)
-        self.write('for ')
-        self.visit(node.target)
-        self.write(' in ')
-        self.visit(node.iter)
-        self.write(':')
-        self.body_or_else(node)
-
-    def visit_While(self, node):
-        self.newline(node)
-        self.write('while ')
-        self.visit(node.test)
-        self.write(':')
-        self.body_or_else(node)
-
-    def visit_With(self, node):
-        self.newline(node)
-        self.write('with ')
-        self.visit(node.context_expr)
-        if node.optional_vars is not None:
-            self.write(' as ')
-            self.visit(node.optional_vars)
-        self.write(':')
-        self.body(node.body)
-
-    def visit_Pass(self, node):
-        self.newline(node)
-        self.write('pass')
-
-    def visit_Print(self, node):
-        # XXX: python 2.6 only
-        self.newline(node)
-        self.write('print ')
-        want_comma =3D False
-        if node.dest is not None:
-            self.write(' >> ')
-            self.visit(node.dest)
-            want_comma =3D True
-        for value in node.values:
-            if want_comma:
-                self.write(', ')
-            self.visit(value)
-            want_comma =3D True
-        if not node.nl:
-            self.write(',')
-
-    def visit_Delete(self, node):
-        self.newline(node)
-        self.write('del ')
-        for idx, target in enumerate(node):
-            if idx:
-                self.write(', ')
-            self.visit(target)
-
-    def visit_TryExcept(self, node):
-        self.newline(node)
-        self.write('try:')
-        self.body(node.body)
-        for handler in node.handlers:
-            self.visit(handler)
-
-    def visit_TryFinally(self, node):
-        self.newline(node)
-        self.write('try:')
-        self.body(node.body)
-        self.newline(node)
-        self.write('finally:')
-        self.body(node.finalbody)
-
-    def visit_Global(self, node):
-        self.newline(node)
-        self.write('global ' + ', '.join(node.names))
-
-    def visit_Nonlocal(self, node):
-        self.newline(node)
-        self.write('nonlocal ' + ', '.join(node.names))
-
-    def visit_Return(self, node):
-        self.newline(node)
-        self.write('return ')
-        self.visit(node.value)
-
-    def visit_Break(self, node):
-        self.newline(node)
-        self.write('break')
-
-    def visit_Continue(self, node):
-        self.newline(node)
-        self.write('continue')
-
-    def visit_Raise(self, node):
-        # XXX: Python 2.6 / 3.0 compatibility
-        self.newline(node)
-        self.write('raise')
-        if hasattr(node, 'exc') and node.exc is not None:
-            self.write(' ')
-            self.visit(node.exc)
-            if node.cause is not None:
-                self.write(' from ')
-                self.visit(node.cause)
-        elif hasattr(node, 'type') and node.type is not None:
-            self.visit(node.type)
-            if node.inst is not None:
-                self.write(', ')
-                self.visit(node.inst)
-            if node.tback is not None:
-                self.write(', ')
-                self.visit(node.tback)
-
-    # Expressions
-
-    def visit_Attribute(self, node):
-        self.visit(node.value)
-        self.write('.' + node.attr)
-
-    def visit_Call(self, node):
-        want_comma =3D []
-        def write_comma():
-            if want_comma:
-                self.write(', ')
-            else:
-                want_comma.append(True)
-
-        self.visit(node.func)
-        self.write('(')
-        for arg in node.args:
-            write_comma()
-            self.visit(arg)
-        for keyword in node.keywords:
-            write_comma()
-            self.write(keyword.arg + '=3D')
-            self.visit(keyword.value)
-        if hasattr(node, 'starargs') and node.starargs is not None:
-            write_comma()
-            self.write('*')
-            self.visit(node.starargs)
-        if hasattr(node, 'kwargs') and node.kwargs is not None:
-            write_comma()
-            self.write('**')
-            self.visit(node.kwargs)
-        self.write(')')
-
-    def visit_Name(self, node):
-        self.write(node.id)
-
-    def visit_Constant(self, node):
-        self.write(repr(node.value))
-
-    def visit_Tuple(self, node):
-        self.write('(')
-        idx =3D -1
-        for idx, item in enumerate(node.elts):
-            if idx:
-                self.write(', ')
-            self.visit(item)
-        self.write(idx and ')' or ',)')
-
-    def sequence_visit(left, right):
-        def visit(self, node):
-            self.write(left)
-            for idx, item in enumerate(node.elts):
-                if idx:
-                    self.write(', ')
-                self.visit(item)
-            self.write(right)
-        return visit
-
-    visit_List =3D sequence_visit('[', ']')
-    visit_Set =3D sequence_visit('{', '}')
-    del sequence_visit
-
-    def visit_Dict(self, node):
-        self.write('{')
-        for idx, (key, value) in enumerate(zip(node.keys, node.values)):
-            if idx:
-                self.write(', ')
-            self.visit(key)
-            self.write(': ')
-            self.visit(value)
-        self.write('}')
-
-    def visit_BinOp(self, node):
-        self.visit(node.left)
-        self.write(' %s ' % BINOP_SYMBOLS[type(node.op)])
-        self.visit(node.right)
-
-    def visit_BoolOp(self, node):
-        self.write('(')
-        for idx, value in enumerate(node.values):
-            if idx:
-                self.write(' %s ' % BOOLOP_SYMBOLS[type(node.op)])
-            self.visit(value)
-        self.write(')')
-
-    def visit_Compare(self, node):
-        self.write('(')
-        self.write(node.left)
-        for op, right in zip(node.ops, node.comparators):
-            self.write(' %s %%' % CMPOP_SYMBOLS[type(op)])
-            self.visit(right)
-        self.write(')')
-
-    def visit_UnaryOp(self, node):
-        self.write('(')
-        op =3D UNARYOP_SYMBOLS[type(node.op)]
-        self.write(op)
-        if op =3D=3D 'not':
-            self.write(' ')
-        self.visit(node.operand)
-        self.write(')')
-
-    def visit_Subscript(self, node):
-        self.visit(node.value)
-        self.write('[')
-        self.visit(node.slice)
-        self.write(']')
-
-    def visit_Slice(self, node):
-        if node.lower is not None:
-            self.visit(node.lower)
-        self.write(':')
-        if node.upper is not None:
-            self.visit(node.upper)
-        if node.step is not None:
-            self.write(':')
-            if not (isinstance(node.step, Name) and node.step.id =3D=3D 'N=
one'):
-                self.visit(node.step)
-
-    def visit_ExtSlice(self, node):
-        for idx, item in node.dims:
-            if idx:
-                self.write(', ')
-            self.visit(item)
-
-    def visit_Yield(self, node):
-        self.write('yield ')
-        self.visit(node.value)
-
-    def visit_Lambda(self, node):
-        self.write('lambda ')
-        self.signature(node.args)
-        self.write(': ')
-        self.visit(node.body)
-
-    def visit_Ellipsis(self, node):
-        self.write('Ellipsis')
-
-    def generator_visit(left, right):
-        def visit(self, node):
-            self.write(left)
-            self.visit(node.elt)
-            for comprehension in node.generators:
-                self.visit(comprehension)
-            self.write(right)
-        return visit
-
-    visit_ListComp =3D generator_visit('[', ']')
-    visit_GeneratorExp =3D generator_visit('(', ')')
-    visit_SetComp =3D generator_visit('{', '}')
-    del generator_visit
-
-    def visit_DictComp(self, node):
-        self.write('{')
-        self.visit(node.key)
-        self.write(': ')
-        self.visit(node.value)
-        for comprehension in node.generators:
-            self.visit(comprehension)
-        self.write('}')
-
-    def visit_IfExp(self, node):
-        self.visit(node.body)
-        self.write(' if ')
-        self.visit(node.test)
-        self.write(' else ')
-        self.visit(node.orelse)
-
-    def visit_Starred(self, node):
-        self.write('*')
-        self.visit(node.value)
-
-    def visit_Repr(self, node):
-        # XXX: python 2.6 only
-        self.write('`')
-        self.visit(node.value)
-        self.write('`')
-
-    # Helper Nodes
-
-    def visit_alias(self, node):
-        self.write(node.name)
-        if node.asname is not None:
-            self.write(' as ' + node.asname)
-
-    def visit_comprehension(self, node):
-        self.write(' for ')
-        self.visit(node.target)
-        self.write(' in ')
-        self.visit(node.iter)
-        if node.ifs:
-            for if_ in node.ifs:
-                self.write(' if ')
-                self.visit(if_)
-
-    def visit_excepthandler(self, node):
-        self.newline(node)
-        self.write('except')
-        if node.type is not None:
-            self.write(' ')
-            self.visit(node.type)
-            if node.name is not None:
-                self.write(' as ')
-                self.visit(node.name)
-        self.write(':')
-        self.body(node.body)
diff --git a/bitbake/lib/hashserv/__init__.py b/bitbake/lib/hashserv/__init=
__.py
index 74367eb6..58ab678e 100644
--- a/bitbake/lib/hashserv/__init__.py
+++ b/bitbake/lib/hashserv/__init__.py
@@ -7,12 +7,22 @@ import asyncio
 from contextlib import closing
 import itertools
 import json
+import re
 from collections import namedtuple
 from urllib.parse import urlparse
 from bb.asyncrpc.client import parse_address, ADDR_TYPE_UNIX, ADDR_TYPE_WS
=20
+__version__ =3D "2.20.0"
+
 User =3D namedtuple("User", ("username", "permissions"))
=20
+UNIHASH_REGEX =3D re.compile(r"^[0-9a-f]{64}$")
+
+
+def is_valid_unihash(value):
+    return isinstance(value, str) and UNIHASH_REGEX.fullmatch(value) is no=
t None
+
+
 def create_server(
     addr,
     dbname,
@@ -25,6 +35,7 @@ def create_server(
     anon_perms=3DNone,
     admin_username=3DNone,
     admin_password=3DNone,
+    reuseport=3DFalse,
 ):
     def sqlite_engine():
         from .sqlite import DatabaseEngine
@@ -60,9 +71,9 @@ def create_server(
         s.start_unix_server(*a)
     elif typ =3D=3D ADDR_TYPE_WS:
         url =3D urlparse(a[0])
-        s.start_websocket_server(url.hostname, url.port)
+        s.start_websocket_server(url.hostname, url.port, reuseport=3Dreuse=
port)
     else:
-        s.start_tcp_server(*a)
+        s.start_tcp_server(*a, reuseport=3Dreuseport)
=20
     return s
=20
diff --git a/bitbake/lib/hashserv/client.py b/bitbake/lib/hashserv/client.p=
y
index 0b254bed..b447fb25 100644
--- a/bitbake/lib/hashserv/client.py
+++ b/bitbake/lib/hashserv/client.py
@@ -5,18 +5,262 @@
=20
 import logging
 import socket
+import asyncio
 import bb.asyncrpc
 import json
+from abc import abstractmethod
+from collections.abc import AsyncIterable
+from contextlib import asynccontextmanager
+from dataclasses import dataclass
 from . import create_async_client
=20
-
 logger =3D logging.getLogger("hashserv.client")
=20
=20
+class AsyncQueue(AsyncIterable):
+    class Shutdown(Exception):
+        pass
+
+    SHUTDOWN_SENTINEL =3D object()
+
+    def __init__(self, *args, **kwargs):
+        self.__queue =3D asyncio.Queue()
+        self.__shutdown =3D False
+        self.__is_done =3D False
+
+    async def done(self):
+        if self.__is_done:
+            return
+        self.__is_done =3D True
+        await self.__queue.put(self.SHUTDOWN_SENTINEL)
+
+    async def put(self, item):
+        if self.__is_done:
+            raise self.Shutdown
+        await self.__queue.put(item)
+
+    async def get(self):
+        if self.__shutdown:
+            raise self.Shutdown
+
+        item =3D await self.__queue.get()
+        if item is self.SHUTDOWN_SENTINEL:
+            self.__shutdown =3D True
+            raise self.Shutdown
+
+        return item
+
+    def __aiter__(self):
+        return self
+
+    async def __anext__(self):
+        try:
+            return await self.get()
+        except self.Shutdown:
+            raise StopAsyncIteration
+
+
+@dataclass(eq=3DFalse, frozen=3DTrue)
+class AsyncPipe:
+    send_queue: AsyncQueue
+    recv_queue: AsyncQueue
+
+
+class Stream(AsyncIterable):
+    def __init__(self, pipe):
+        self._pipe =3D pipe
+
+    async def done(self):
+        await self._pipe.send_queue.done()
+
+    def __aiter__(self):
+        return self
+
+    async def __anext__(self):
+        try:
+            return await self.get_result()
+        except AsyncQueue.Shutdown:
+            raise StopAsyncIteration
+
+    @abstractmethod
+    async def _send_batch_input(self, i):
+        raise NotImplementedError("Not implemented")
+
+    @abstractmethod
+    async def get_result(self):
+        raise NotImplementedError("Not implemented")
+
+    async def batch(self, inputs):
+        """
+        Does a "batch" process of stream messages. This sends the query
+        messages as fast as possible, and simultaneously attempts to read =
the
+        messages back. This helps to mitigate the effects of latency to th=
e
+        hash equivalence server be allowing multiple queries to be "in-fli=
ght"
+        at once
+
+        The input may be a generator or an async generator
+        """
+
+        async def get_inputs():
+            if isinstance(inputs, AsyncIterable):
+                async for i in inputs:
+                    yield i
+            else:
+                for i in inputs:
+                    yield i
+
+        async def send():
+            try:
+                async for i in get_inputs():
+                    await self._send_batch_input(i)
+            finally:
+                await self.done()
+
+        results =3D []
+
+        async def recv():
+            async for item in self:
+                results.append(item)
+
+        await bb.asyncrpc.TaskGroup.run(send(), recv())
+        return results
+
+
+class GetUnihashStream(Stream):
+    def __init__(self, pipe):
+        super().__init__(pipe)
+
+    async def _send_batch_input(self, i):
+        method, taskhash =3D i
+        await self.send_query(method, taskhash)
+
+    async def send_query(self, method, taskhash):
+        await self._pipe.send_queue.put(f"{method} {taskhash}")
+
+    async def get_result(self):
+        r =3D await self._pipe.recv_queue.get()
+        return r if r else None
+
+
+class UnihashExistsStream(Stream):
+    def __init__(self, pipe):
+        super().__init__(pipe)
+
+    async def _send_batch_input(self, i):
+        await self.send_query(i)
+
+    async def send_query(self, unihash):
+        await self._pipe.send_queue.put(unihash)
+
+    async def get_result(self):
+        r =3D await self._pipe.recv_queue.get()
+        return r =3D=3D "true"
+
+
+class GcMarkStream(Stream):
+    def __init__(self, pipe, mark):
+        super().__init__(pipe)
+        self.mark =3D mark
+
+    async def _send_batch_input(self, i):
+        def row_to_dict(row):
+            pairs =3D row.split()
+            return dict(zip(pairs[::2], pairs[1::2]))
+
+        await self.send_mark(row_to_dict(i))
+
+    async def send_mark(self, where):
+        await self._pipe.send_queue.put(json.dumps({"mark": self.mark, "wh=
ere": where}))
+
+    async def get_result(self):
+        r =3D await self._pipe.recv_queue.get()
+        return json.loads(r)
+
+
+class Batch(object):
+    def __init__(self, send_queue, recv_queue):
+        self.send_queue =3D send_queue
+        self.recv_queue =3D recv_queue
+        self.fill_done =3D False
+        self.send_done =3D False
+        self.cond =3D asyncio.Condition()
+        self.pending =3D []
+        self.sent_count =3D 0
+        self.recv_count =3D 0
+        self.item =3D None
+
+    async def recv(self, socket):
+        while True:
+            async with self.cond:
+                await self.cond.wait_for(lambda: self.pending or self.send=
_done)
+                if not self.pending:
+                    if self.send_done:
+                        return
+                    continue
+
+            m =3D await socket.recv()
+            await self.recv_queue.put(m)
+
+            async with self.cond:
+                self.recv_count +=3D 1
+                self.pending.pop(0)
+
+    async def fill(self):
+        async for m in self.send_queue:
+            async with self.cond:
+                # Wait for item to be consumed
+                await self.cond.wait_for(lambda: self.item is None)
+                self.item =3D m
+                self.cond.notify_all()
+
+        async with self.cond:
+            self.fill_done =3D True
+            self.cond.notify_all()
+
+    async def send(self, socket):
+        # In the event of a restart due to a reconnect, all in-flight
+        # messages need to be resent first to keep to result count in sync
+        async with self.cond:
+            for m in self.pending:
+                await socket.send(m)
+
+        while True:
+            async with self.cond:
+                await self.cond.wait_for(
+                    lambda: self.item is not None or self.fill_done
+                )
+                if self.item is None:
+                    if self.fill_done:
+                        self.send_done =3D True
+                        self.cond.notify_all()
+                        return
+                    continue
+
+                m =3D self.item
+
+            await socket.send(m)
+
+            async with self.cond:
+                self.item =3D None
+                self.pending.append(m)
+                self.sent_count +=3D 1
+                self.cond.notify_all()
+
+    async def stream(self, socket):
+        await bb.asyncrpc.TaskGroup.run(self.send(socket), self.recv(socke=
t))
+
+    def check(self):
+        if self.sent_count !=3D self.recv_count:
+            raise ConnectionError(
+                f"Sent {self.sent_count} messages but only received {self.=
recv_count}"
+            )
+
+
 class AsyncClient(bb.asyncrpc.AsyncClient):
     MODE_NORMAL =3D 0
     MODE_GET_STREAM =3D 1
     MODE_EXIST_STREAM =3D 2
+    MODE_MARK_STREAM =3D 3
=20
     def __init__(self, username=3DNone, password=3DNone):
         super().__init__("OEHASHEQUIV", "1.1", logger)
@@ -36,32 +280,57 @@ class AsyncClient(bb.asyncrpc.AsyncClient):
             if become:
                 await self.become_user(become)
=20
-    async def send_stream(self, mode, msg):
+    @asynccontextmanager
+    async def send_stream(self, mode):
+        send_queue =3D AsyncQueue()
+        recv_queue =3D AsyncQueue()
+        b =3D Batch(send_queue, recv_queue)
+
         async def proc():
             await self._set_mode(mode)
-            await self.socket.send(msg)
-            return await self.socket.recv()
-
-        return await self._send_wrapper(proc)
-
-    async def invoke(self, *args, **kwargs):
+            await b.stream(self.socket)
+
+        async def process():
+            try:
+                await self._send_wrapper(proc)
+            finally:
+                await recv_queue.done()
+
+        # Create background process to process messages
+        async with bb.asyncrpc.TaskGroup() as group:
+            group.create_task(process())
+            group.create_task(b.fill())
+            try:
+                yield AsyncPipe(send_queue, recv_queue)
+                b.check()
+            except AsyncQueue.Shutdown as e:
+                pass
+            finally:
+                await send_queue.done()
+                await recv_queue.done()
+
+    async def invoke(self, *args, skip_mode=3DFalse, **kwargs):
         # It's OK if connection errors cause a failure here, because the m=
ode
         # is also reset to normal on a new connection
-        await self._set_mode(self.MODE_NORMAL)
+        if not skip_mode:
+            await self._set_mode(self.MODE_NORMAL)
         return await super().invoke(*args, **kwargs)
=20
     async def _set_mode(self, new_mode):
         async def stream_to_normal():
+            # Check if already in normal mode (e.g. due to a connection re=
set)
+            if self.mode =3D=3D self.MODE_NORMAL:
+                return "ok"
             await self.socket.send("END")
             return await self.socket.recv()
=20
         async def normal_to_stream(command):
-            r =3D await self.invoke({command: None})
+            r =3D await self.invoke({command: None}, skip_mode=3DTrue)
             if r !=3D "ok":
+                self.check_invoke_error(r)
                 raise ConnectionError(
                     f"Unable to transition to stream mode: Bad response fr=
om server {r!r}"
                 )
-
             self.logger.debug("Mode is now %s", command)
=20
         if new_mode =3D=3D self.mode:
@@ -83,16 +352,26 @@ class AsyncClient(bb.asyncrpc.AsyncClient):
             await normal_to_stream("get-stream")
         elif new_mode =3D=3D self.MODE_EXIST_STREAM:
             await normal_to_stream("exists-stream")
+        elif new_mode =3D=3D self.MODE_MARK_STREAM:
+            await normal_to_stream("gc-mark-stream")
         elif new_mode !=3D self.MODE_NORMAL:
             raise Exception("Undefined mode transition {self.mode!r} -> {n=
ew_mode!r}")
=20
         self.mode =3D new_mode
=20
     async def get_unihash(self, method, taskhash):
-        r =3D await self.send_stream(self.MODE_GET_STREAM, "%s %s" % (meth=
od, taskhash))
-        if not r:
-            return None
-        return r
+        async with self.get_unihash_stream() as stream:
+            await stream.send_query(method, taskhash)
+            return await stream.get_result()
+
+    async def get_unihash_batch(self, args):
+        async with self.get_unihash_stream() as stream:
+            return await stream.batch(args)
+
+    @asynccontextmanager
+    async def get_unihash_stream(self):
+        async with self.send_stream(self.MODE_GET_STREAM) as pipe:
+            yield GetUnihashStream(pipe)
=20
     async def report_unihash(self, taskhash, method, outhash, unihash, ext=
ra=3D{}):
         m =3D extra.copy()
@@ -115,8 +394,18 @@ class AsyncClient(bb.asyncrpc.AsyncClient):
         )
=20
     async def unihash_exists(self, unihash):
-        r =3D await self.send_stream(self.MODE_EXIST_STREAM, unihash)
-        return r =3D=3D "true"
+        async with self.unihash_exists_stream() as stream:
+            await stream.send_query(unihash)
+            return await stream.get_result()
+
+    async def unihash_exists_batch(self, unihashes):
+        async with self.unihash_exists_stream() as stream:
+            return await stream.batch(unihashes)
+
+    @asynccontextmanager
+    async def unihash_exists_stream(self):
+        async with self.send_stream(self.MODE_EXIST_STREAM) as pipe:
+            yield UnihashExistsStream(pipe)
=20
     async def get_outhash(self, method, outhash, taskhash, with_unihash=3D=
True):
         return await self.invoke(
@@ -216,6 +505,23 @@ class AsyncClient(bb.asyncrpc.AsyncClient):
         """
         return await self.invoke({"gc-mark": {"mark": mark, "where": where=
}})
=20
+    async def gc_mark_batch(self, mark, rows):
+        """
+        Similar to `gc-mark`, but accepts a list of "where" key-value pair
+        conditions. It utilizes stream mode to mark hashes, which helps re=
duce
+        the impact of latency when communicating with the hash equivalence
+        server.
+        """
+        async with self.gc_mark_stream(mark) as stream:
+            results =3D await stream.batch(rows)
+
+        return {"count": sum(int(r["count"]) for r in results)}
+
+    @asynccontextmanager
+    async def gc_mark_stream(self, mark):
+        async with self.send_stream(self.MODE_MARK_STREAM) as pipe:
+            yield GcMarkStream(pipe, mark)
+
     async def gc_sweep(self, mark):
         """
         Finishes garbage collection for "mark". All unihash entries that h=
ave
@@ -237,10 +543,12 @@ class Client(bb.asyncrpc.Client):
             "connect_tcp",
             "connect_websocket",
             "get_unihash",
+            "get_unihash_batch",
             "report_unihash",
             "report_unihash_equiv",
             "get_taskhash",
             "unihash_exists",
+            "unihash_exists_batch",
             "get_outhash",
             "get_stats",
             "reset_stats",
@@ -259,88 +567,9 @@ class Client(bb.asyncrpc.Client):
             "get_db_query_columns",
             "gc_status",
             "gc_mark",
+            "gc_mark_batch",
             "gc_sweep",
         )
=20
     def _get_async_client(self):
         return AsyncClient(self.username, self.password)
-
-
-class ClientPool(bb.asyncrpc.ClientPool):
-    def __init__(
-        self,
-        address,
-        max_clients,
-        *,
-        username=3DNone,
-        password=3DNone,
-        become=3DNone,
-    ):
-        super().__init__(max_clients)
-        self.address =3D address
-        self.username =3D username
-        self.password =3D password
-        self.become =3D become
-
-    async def _new_client(self):
-        client =3D await create_async_client(
-            self.address,
-            username=3Dself.username,
-            password=3Dself.password,
-        )
-        if self.become:
-            await client.become_user(self.become)
-        return client
-
-    def _run_key_tasks(self, queries, call):
-        results =3D {key: None for key in queries.keys()}
-
-        def make_task(key, args):
-            async def task(client):
-                nonlocal results
-                unihash =3D await call(client, args)
-                results[key] =3D unihash
-
-            return task
-
-        def gen_tasks():
-            for key, args in queries.items():
-                yield make_task(key, args)
-
-        self.run_tasks(gen_tasks())
-        return results
-
-    def get_unihashes(self, queries):
-        """
-        Query multiple unihashes in parallel.
-
-        The queries argument is a dictionary with arbitrary key. The value=
s
-        must be a tuple of (method, taskhash).
-
-        Returns a dictionary with a corresponding key for each input key, =
and
-        the value is the queried unihash (which might be none if the query
-        failed)
-        """
-
-        async def call(client, args):
-            method, taskhash =3D args
-            return await client.get_unihash(method, taskhash)
-
-        return self._run_key_tasks(queries, call)
-
-    def unihashes_exist(self, queries):
-        """
-        Query multiple unihash existence checks in parallel.
-
-        The queries argument is a dictionary with arbitrary key. The value=
s
-        must be a unihash.
-
-        Returns a dictionary with a corresponding key for each input key, =
and
-        the value is True or False if the unihash is known by the server (=
or
-        None if there was a failure)
-        """
-
-        async def call(client, unihash):
-            return await client.unihash_exists(unihash)
-
-        return self._run_key_tasks(queries, call)
diff --git a/bitbake/lib/hashserv/server.py b/bitbake/lib/hashserv/server.p=
y
index 68f64f98..d9c30bdb 100644
--- a/bitbake/lib/hashserv/server.py
+++ b/bitbake/lib/hashserv/server.py
@@ -10,8 +10,10 @@ import math
 import time
 import os
 import base64
+import json
 import hashlib
 from . import create_async_client
+from . import is_valid_unihash
 import bb.asyncrpc
=20
 logger =3D logging.getLogger("hashserv.server")
@@ -172,6 +174,11 @@ def hash_token(algo, salt, token):
     return ":".join([algo, salt, h.hexdigest()])
=20
=20
+def validate_unihash(value):
+    if not is_valid_unihash(value):
+        raise bb.asyncrpc.InvokeError("Invalid unihash")
+
+
 def permissions(*permissions, allow_anon=3DTrue, allow_self_service=3DFals=
e):
     """
     Function decorator that can be used to decorate an RPC function call a=
nd
@@ -222,6 +229,68 @@ def permissions(*permissions, allow_anon=3DTrue, allow=
_self_service=3DFalse):
     return wrapper
=20
=20
+class UpstreamQueue(object):
+    UPSTREAM_NONCE =3D object()
+
+    def __init__(
+        self,
+        logger,
+        queue,
+        get_local_result,
+        send_upstream,
+        get_upstream_result,
+    ):
+        self.logger =3D logger
+        self.queue =3D queue
+        self.pending =3D []
+        self.cond =3D asyncio.Condition()
+        self.done =3D False
+        self.get_local_result =3D get_local_result
+        self.send_upstream =3D send_upstream
+        self.get_upstream_result =3D get_upstream_result
+
+    async def process_results(self):
+        try:
+            while True:
+                async with self.cond:
+                    await self.cond.wait_for(lambda: self.pending or self.=
done)
+                    if not self.pending:
+                        if self.done:
+                            return
+                        continue
+
+                    value, m =3D self.pending.pop(0)
+
+                if value is self.UPSTREAM_NONCE:
+                    value =3D await self.get_upstream_result(m)
+
+                if value is None:
+                    self.logger.error(
+                        "None is not allowed as a stream value. Terminatin=
g stream"
+                    )
+                    return
+
+                await self.queue.put(value)
+        finally:
+            await self.queue.put(None)
+
+    async def handler(self, m):
+        if m is None:
+            async with self.cond:
+                self.done =3D True
+                self.cond.notify_all()
+            return
+
+        value =3D await self.get_local_result(m)
+        if value is None:
+            await self.send_upstream(m)
+            value =3D self.UPSTREAM_NONCE
+
+        async with self.cond:
+            self.pending.append((value, m))
+            self.cond.notify_all()
+
+
 class ServerClient(bb.asyncrpc.AsyncServerConnection):
     def __init__(self, socket, server):
         super().__init__(socket, "OEHASHEQUIV", server.logger)
@@ -256,6 +325,7 @@ class ServerClient(bb.asyncrpc.AsyncServerConnection):
                     "backfill-wait": self.handle_backfill_wait,
                     "remove": self.handle_remove,
                     "gc-mark": self.handle_gc_mark,
+                    "gc-mark-stream": self.handle_gc_mark_stream,
                     "gc-sweep": self.handle_gc_sweep,
                     "gc-status": self.handle_gc_status,
                     "clean-unused": self.handle_clean_unused,
@@ -343,7 +413,7 @@ class ServerClient(bb.asyncrpc.AsyncServerConnection):
                 d =3D {k: row[k] for k in row.keys()}
             elif self.upstream_client is not None:
                 d =3D await self.upstream_client.get_taskhash(method, task=
hash)
-                await self.db.insert_unihash(d["method"], d["taskhash"], d=
["unihash"])
+                await self.insert_unihash(d["method"], d["taskhash"], d["u=
nihash"])
=20
         return d
=20
@@ -375,74 +445,131 @@ class ServerClient(bb.asyncrpc.AsyncServerConnection=
):
         if data is None:
             return
=20
-        await self.db.insert_unihash(data["method"], data["taskhash"], dat=
a["unihash"])
+        await self.insert_unihash(data["method"], data["taskhash"], data["=
unihash"])
         await self.db.insert_outhash(data)
=20
-    async def _stream_handler(self, handler):
+    async def insert_unihash(self, method, taskhash, unihash):
+        validate_unihash(unihash)
+        return await self.db.insert_unihash(method, taskhash, unihash)
+
+    async def _stream_queue_handler(self, handler, queue):
         await self.socket.send_message("ok")
=20
-        while True:
-            upstream =3D None
+        async def recv():
+            try:
+                while True:
+                    m =3D await self.socket.recv()
+                    if not m or m =3D=3D "END":
+                        break
=20
-            l =3D await self.socket.recv()
-            if not l:
-                break
+                    await handler(m)
+            finally:
+                await handler(None)
=20
-            try:
-                # This inner loop is very sensitive and must be as fast as
-                # possible (which is why the request sample is handled man=
ually
-                # instead of using 'with', and also why logging statements=
 are
-                # commented out.
-                self.request_sample =3D self.server.request_stats.start_sa=
mple()
-                request_measure =3D self.request_sample.measure()
-                request_measure.start()
-
-                if l =3D=3D "END":
+        async def process():
+            while True:
+                m =3D await queue.get()
+                if m is None:
                     break
=20
-                msg =3D await handler(l)
-                await self.socket.send(msg)
-            finally:
-                request_measure.end()
-                self.request_sample.end()
+                await self.socket.send(m)
=20
+        await bb.asyncrpc.TaskGroup.run(recv(), process())
         await self.socket.send("ok")
+
+    async def _stream_handler(self, handler):
+        queue =3D asyncio.Queue(1000)
+
+        async def h(m):
+            if m is None:
+                await queue.put(None)
+            else:
+                await queue.put(await handler(m))
+
+        await self._stream_queue_handler(h, queue)
         return self.NO_RESPONSE
=20
     @permissions(READ_PERM)
     async def handle_get_stream(self, request):
-        async def handler(l):
-            (method, taskhash) =3D l.split()
-            # self.logger.debug('Looking up %s %s' % (method, taskhash))
-            row =3D await self.db.get_equivalent(method, taskhash)
-
-            if row is not None:
-                # self.logger.debug('Found equivalent task %s -> %s', (row=
['taskhash'], row['unihash']))
+        async def get_unihash(m):
+            method, taskhash =3D m.split()
+            if (row :=3D await self.db.get_equivalent(method, taskhash)) i=
s not None:
                 return row["unihash"]
=20
-            if self.upstream_client is not None:
-                upstream =3D await self.upstream_client.get_unihash(method=
, taskhash)
-                if upstream:
-                    await self.server.backfill_queue.put((method, taskhash=
))
-                    return upstream
-
             return ""
=20
-        return await self._stream_handler(handler)
+        if not self.upstream_client:
+            return await self._stream_handler(get_unihash)
+
+        async with self.upstream_client.get_unihash_stream() as stream:
+
+            async def get_local_result(m):
+                method, taskhash =3D m.split()
+                if (row :=3D await self.db.get_equivalent(method, taskhash=
)) is not None:
+                    return row["unihash"]
+                return None
+
+            async def send_upstream(m):
+                method, taskhash =3D m.split()
+                await stream.send_query(method, taskhash)
+
+            async def get_upstream_result(m):
+                unihash =3D await stream.get_result()
+                if unihash:
+                    method, taskhash =3D m.split()
+                    await self.server.backfill_queue.put((method, taskhash=
))
+                return unihash or ""
+
+            queue =3D asyncio.Queue()
+            upstream =3D UpstreamQueue(
+                self.logger,
+                queue,
+                get_local_result,
+                send_upstream,
+                get_upstream_result,
+            )
+
+            await bb.asyncrpc.TaskGroup.run(
+                self._stream_queue_handler(upstream.handler, queue),
+                upstream.process_results(),
+            )
+        return self.NO_RESPONSE
=20
     @permissions(READ_PERM)
     async def handle_exists_stream(self, request):
-        async def handler(l):
+        async def exists_handler(l):
             if await self.db.unihash_exists(l):
                 return "true"
+            return "false"
=20
-            if self.upstream_client is not None:
-                if await self.upstream_client.unihash_exists(l):
-                    return "true"
+        if not self.upstream_client:
+            return await self._stream_handler(exists_handler)
=20
-            return "false"
+        async with self.upstream_client.unihash_exists_stream() as stream:
=20
-        return await self._stream_handler(handler)
+            async def get_local_result(m):
+                if await self.db.unihash_exists(m):
+                    return "true"
+                return None
+
+            async def get_upstream_result(m):
+                exists =3D await stream.get_result()
+                return "true" if exists else "false"
+
+            queue =3D asyncio.Queue()
+            upstream =3D UpstreamQueue(
+                self.logger,
+                queue,
+                get_local_result,
+                stream.send_query,
+                get_upstream_result,
+            )
+
+            await bb.asyncrpc.TaskGroup.run(
+                self._stream_queue_handler(upstream.handler, queue),
+                upstream.process_results(),
+            )
+        return self.NO_RESPONSE
=20
     async def report_readonly(self, data):
         method =3D data["method"]
@@ -465,6 +592,8 @@ class ServerClient(bb.asyncrpc.AsyncServerConnection):
     # report is made inside the function
     @permissions(READ_PERM)
     async def handle_report(self, data):
+        validate_unihash(data.get("unihash"))
+
         if self.server.read_only or not self.user_has_permissions(REPORT_P=
ERM):
             return await self.report_readonly(data)
=20
@@ -507,7 +636,7 @@ class ServerClient(bb.asyncrpc.AsyncServerConnection):
                     if upstream_data is not None:
                         unihash =3D upstream_data["unihash"]
=20
-            await self.db.insert_unihash(data["method"], data["taskhash"],=
 unihash)
+            await self.insert_unihash(data["method"], data["taskhash"], un=
ihash)
=20
         unihash_data =3D await self.get_unihash(data["method"], data["task=
hash"])
         if unihash_data is not None:
@@ -523,7 +652,9 @@ class ServerClient(bb.asyncrpc.AsyncServerConnection):
=20
     @permissions(READ_PERM, REPORT_PERM)
     async def handle_equivreport(self, data):
-        await self.db.insert_unihash(data["method"], data["taskhash"], dat=
a["unihash"])
+        validate_unihash(data.get("unihash"))
+
+        await self.insert_unihash(data["method"], data["taskhash"], data["=
unihash"])
=20
         # Fetch the unihash that will be reported for the taskhash. If the
         # unihash matches, it means this row was inserted (or the mapping
@@ -583,6 +714,33 @@ class ServerClient(bb.asyncrpc.AsyncServerConnection):
=20
         return {"count": await self.db.gc_mark(mark, condition)}
=20
+    @permissions(DB_ADMIN_PERM)
+    async def handle_gc_mark_stream(self, request):
+        async def handler(line):
+            try:
+                decoded_line =3D json.loads(line)
+            except json.JSONDecodeError as exc:
+                raise bb.asyncrpc.InvokeError(
+                    "Could not decode JSONL input '%s'" % line
+                ) from exc
+
+            try:
+                mark =3D decoded_line["mark"]
+                condition =3D decoded_line["where"]
+                if not isinstance(mark, str):
+                    raise TypeError("Bad mark type %s" % type(mark))
+
+                if not isinstance(condition, dict):
+                    raise TypeError("Bad condition type %s" % type(conditi=
on))
+            except KeyError as exc:
+                raise bb.asyncrpc.InvokeError(
+                    "Input line is missing key '%s' " % exc
+                ) from exc
+
+            return json.dumps({"count": await self.db.gc_mark(mark, condit=
ion)})
+
+        return await self._stream_handler(handler)
+
     @permissions(DB_ADMIN_PERM)
     async def handle_gc_sweep(self, request):
         mark =3D request["mark"]
@@ -603,7 +761,7 @@ class ServerClient(bb.asyncrpc.AsyncServerConnection):
=20
     @permissions(DB_ADMIN_PERM)
     async def handle_gc_status(self, request):
-        (keep_rows, remove_rows, current_mark) =3D await self.db.gc_status=
()
+        keep_rows, remove_rows, current_mark =3D await self.db.gc_status()
         return {
             "keep": keep_rows,
             "remove": remove_rows,
@@ -859,7 +1017,12 @@ class Server(bb.asyncrpc.AsyncServer):
                 method, taskhash =3D item
                 d =3D await client.get_taskhash(method, taskhash)
                 if d is not None:
-                    await db.insert_unihash(d["method"], d["taskhash"], d[=
"unihash"])
+                    if is_valid_unihash(d.get("unihash")):
+                        await db.insert_unihash(
+                            d["method"], d["taskhash"], d["unihash"]
+                        )
+                    else:
+                        self.logger.warning("Upstream server returned inva=
lid unihash")
                 self.backfill_queue.task_done()
=20
     def start(self):
diff --git a/bitbake/lib/hashserv/sqlite.py b/bitbake/lib/hashserv/sqlite.p=
y
index da2e844a..976504d7 100644
--- a/bitbake/lib/hashserv/sqlite.py
+++ b/bitbake/lib/hashserv/sqlite.py
@@ -4,6 +4,7 @@
 #
 # SPDX-License-Identifier: GPL-2.0-only
 #
+from datetime import datetime, timezone
 import sqlite3
 import logging
 from contextlib import closing
@@ -53,6 +54,22 @@ CONFIG_TABLE_DEFINITION =3D (
 CONFIG_TABLE_COLUMNS =3D tuple(name for name, _, _ in CONFIG_TABLE_DEFINIT=
ION)
=20
=20
+def adapt_datetime_iso(val):
+    """Adapt datetime.datetime to UTC ISO 8601 date."""
+    return val.astimezone(timezone.utc).isoformat()
+
+
+sqlite3.register_adapter(datetime, adapt_datetime_iso)
+
+
+def convert_datetime(val):
+    """Convert ISO 8601 datetime to datetime.datetime object."""
+    return datetime.fromisoformat(val.decode())
+
+
+sqlite3.register_converter("DATETIME", convert_datetime)
+
+
 def _make_table(cursor, name, definition):
     cursor.execute(
         """
diff --git a/bitbake/lib/hashserv/tests.py b/bitbake/lib/hashserv/tests.py
index 0809453c..15ee7ecd 100644
--- a/bitbake/lib/hashserv/tests.py
+++ b/bitbake/lib/hashserv/tests.py
@@ -5,13 +5,13 @@
 # SPDX-License-Identifier: GPL-2.0-only
 #
=20
-from . import create_server, create_client
+from . import create_server, create_client, create_async_client
 from .server import DEFAULT_ANON_PERMS, ALL_PERMISSIONS
 from bb.asyncrpc import InvokeError
-from .client import ClientPool
 import hashlib
 import logging
-import multiprocessing
+from bb import multiprocessing
+import asyncio
 import os
 import sys
 import tempfile
@@ -31,7 +31,7 @@ BIN_DIR =3D THIS_DIR.parent.parent / "bin"
=20
 def server_prefunc(server, idx):
     logging.basicConfig(level=3Dlogging.DEBUG, filename=3D'bbhashserv-%d.l=
og' % idx, filemode=3D'w',
-                        format=3D'%(levelname)s %(filename)s:%(lineno)d %(=
message)s')
+                        format=3D'%(levelname)s %(filename)s:%(lineno)d %(=
message)s', force=3DTrue)
     server.logger.debug("Running server %d" % idx)
     sys.stdout =3D open('bbhashserv-stdout-%d.log' % idx, 'w')
     sys.stderr =3D sys.stdout
@@ -42,19 +42,15 @@ class HashEquivalenceTestSetup(object):
     server_index =3D 0
     client_index =3D 0
=20
-    def start_server(self, dbpath=3DNone, upstream=3DNone, read_only=3DFal=
se, prefunc=3Dserver_prefunc, anon_perms=3DDEFAULT_ANON_PERMS, admin_userna=
me=3DNone, admin_password=3DNone):
+    def start_server(self, dbpath=3DNone, upstream=3DNone, read_only=3DFal=
se, prefunc=3Dserver_prefunc, anon_perms=3DDEFAULT_ANON_PERMS, admin_userna=
me=3DNone, admin_password=3DNone, addr=3DNone):
         self.server_index +=3D 1
+        if addr is None:
+            addr =3D self.get_server_addr(self.server_index)
+
         if dbpath is None:
             dbpath =3D self.make_dbpath()
=20
-        def cleanup_server(server):
-            if server.process.exitcode is not None:
-                return
-
-            server.process.terminate()
-            server.process.join()
-
-        server =3D create_server(self.get_server_addr(self.server_index),
+        server =3D create_server(addr,
                                dbpath,
                                upstream=3Dupstream,
                                read_only=3Dread_only,
@@ -64,7 +60,7 @@ class HashEquivalenceTestSetup(object):
         server.dbpath =3D dbpath
=20
         server.serve_as_process(prefunc=3Dprefunc, args=3D(self.server_ind=
ex,))
-        self.addCleanup(cleanup_server, server)
+        self.addCleanup(self.stop_server, server)
=20
         return server
=20
@@ -84,6 +80,13 @@ class HashEquivalenceTestSetup(object):
         self.server =3D self.start_server()
         return self.server.address
=20
+    def stop_server(self, server):
+        if not server or server.process.exitcode is not None:
+            return
+
+        server.process.terminate()
+        server.process.join()
+
     def start_auth_server(self):
         auth_server =3D self.start_server(self.server.dbpath, anon_perms=
=3D[], admin_username=3D"admin", admin_password=3D"password")
         self.auth_server_address =3D auth_server.address
@@ -94,9 +97,8 @@ class HashEquivalenceTestSetup(object):
         return self.start_client(self.auth_server_address, user["username"=
], user["token"])
=20
     def setUp(self):
-        if sys.version_info < (3, 5, 0):
-            self.skipTest('Python 3.5 or later required')
-
+        logging.basicConfig(level=3Dlogging.DEBUG, filename=3D'bbhashtest.=
log', filemode=3D'w',
+                        format=3D'%(levelname)s %(filename)s:%(lineno)d %(=
message)s')
         self.temp_dir =3D tempfile.TemporaryDirectory(prefix=3D'bb-hashser=
v')
         self.addCleanup(self.temp_dir.cleanup)
=20
@@ -128,7 +130,7 @@ class HashEquivalenceTestSetup(object):
         # Simple test that hashes can be created
         taskhash =3D '35788efcb8dfb0a02659d81cf2bfd695fb30faf9'
         outhash =3D '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98f=
d7e5970796f'
-        unihash =3D 'f46d3fbb439bd9b921095da657a4de906510d2cd'
+        unihash =3D 'a69ec97f5af2e21e1a1f9cc8896965515d5559425666f734e245a=
3d40cee33d9'
=20
         self.assertClientGetHash(client, taskhash, None)
=20
@@ -182,7 +184,7 @@ class HashEquivalenceCommonTests(object):
         # assigned the same unihash
         taskhash =3D '53b8dce672cb6d0c73170be43f540460bfc347b4'
         outhash =3D '5a9cb1649625f0bf41fc7791b635cd9c2d7118c7f021ba87dcd03=
f72b67ce7a8'
-        unihash =3D 'f37918cc02eb5a520b1aff86faacbc0a38124646'
+        unihash =3D '46edb5140d2613049332d0bf3745d9fafec9c559dac8cc6181373=
9a28007fcdf'
=20
         result =3D self.client.report_unihash(taskhash, self.METHOD, outha=
sh, unihash)
         self.assertEqual(result['unihash'], unihash, 'Server returned bad =
unihash')
@@ -190,7 +192,7 @@ class HashEquivalenceCommonTests(object):
         # Report a different task with the same outhash. The returned unih=
ash
         # should match the first task
         taskhash2 =3D '3bf6f1e89d26205aec90da04854fbdbf73afe6b4'
-        unihash2 =3D 'af36b199320e611fbb16f1f277d3ee1d619ca58b'
+        unihash2 =3D 'bf6e81926066f770e960f9f777cd088c62bea9addb7745f3e77d=
eaa81a645747'
         result =3D self.client.report_unihash(taskhash2, self.METHOD, outh=
ash, unihash2)
         self.assertEqual(result['unihash'], unihash, 'Server returned bad =
unihash')
=20
@@ -200,19 +202,19 @@ class HashEquivalenceCommonTests(object):
         # taskhash
         taskhash =3D '8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a'
         outhash =3D 'afe240a439959ce86f5e322f8c208e1fedefea9e813f2140c81af=
866cc9edf7e'
-        unihash =3D '218e57509998197d570e2c98512d0105985dffc9'
+        unihash =3D '5b521d8a12683086cc08bc2c6d94a7a2dcff17eba53b9911e145d=
51164689380'
         self.client.report_unihash(taskhash, self.METHOD, outhash, unihash=
)
=20
         self.assertClientGetHash(self.client, taskhash, unihash)
=20
         outhash2 =3D '0904a7fe3dc712d9fd8a74a616ddca2a825a8ee97adf0bd3fc86=
082c7639914d'
-        unihash2 =3D 'ae9a7d252735f0dafcdb10e2e02561ca3a47314c'
+        unihash2 =3D 'a37541b54fd22440e292f617eb30ba07455e88fb0b9f0952eca2=
29b6356290e3'
         self.client.report_unihash(taskhash, self.METHOD, outhash2, unihas=
h2)
=20
         self.assertClientGetHash(self.client, taskhash, unihash)
=20
         outhash3 =3D '77623a549b5b1a31e3732dfa8fe61d7ce5d44b3370f253c5360e=
136b852967b4'
-        unihash3 =3D '9217a7d6398518e5dc002ed58f2cbbbc78696603'
+        unihash3 =3D '6842f1f2daccd96ddef15c9154d4e41ac8a2300d781ac9a9db7f=
8afeb8a96808'
         self.client.report_unihash(taskhash, self.METHOD, outhash3, unihas=
h3)
=20
         self.assertClientGetHash(self.client, taskhash, unihash)
@@ -272,7 +274,7 @@ class HashEquivalenceCommonTests(object):
         # Simple test that hashes can be created
         taskhash =3D 'c665584ee6817aa99edfc77a44dd853828279370'
         outhash =3D '3c979c3db45c569f51ab7626a4651074be3a9d11a84b1db076f5b=
14f7d39db44'
-        unihash =3D '90e9bc1d1f094c51824adca7f8ea79a048d68824'
+        unihash =3D '06f89b8f329ba8124ff73c56d09ef921b42624747c421277bddaf=
5e23f136e57'
=20
         self.assertClientGetHash(self.client, taskhash, None)
=20
@@ -295,6 +297,36 @@ class HashEquivalenceCommonTests(object):
         self.assertEqual(result_outhash['outhash'], outhash)
         self.assertEqual(result_outhash['outhash_siginfo'], siginfo)
=20
+    def test_report_rejects_invalid_unihash(self):
+        taskhash =3D '68a9206490b2321bb033fb3eab013a4ec62c41f9'
+        outhash =3D 'bf5f2efaf1ca351f3b4c3d079363540ab48f7c58db3d23cfbb069=
cf4ff1ea8f7'
+        invalid_unihashes =3D (
+            "${@os.system('true')}",
+            'a' * 63,
+            'a' * 65,
+            'A' * 64,
+            None,
+        )
+
+        for unihash in invalid_unihashes:
+            with self.subTest(unihash=3Dunihash):
+                with self.start_client(self.server_address) as client:
+                    with self.assertRaises(InvokeError) as context:
+                        client.report_unihash(taskhash, self.METHOD, outha=
sh, unihash)
+
+                self.assertEqual(str(context.exception), "Invalid unihash"=
)
+
+        self.assertClientGetHash(self.client, taskhash, None)
+
+    def test_equivreport_rejects_invalid_unihash(self):
+        taskhash =3D 'ae6339531895ddf5b67e663e6a374ad8ec71d81c'
+
+        with self.assertRaises(InvokeError) as context:
+            self.client.report_unihash_equiv(taskhash, self.METHOD, "${@os=
.system('true')}")
+
+        self.assertEqual(str(context.exception), "Invalid unihash")
+        self.assertClientGetHash(self.start_client(self.server_address), t=
askhash, None)
+
     def test_stress(self):
         def query_server(failures):
             client =3D Client(self.server_address)
@@ -346,26 +378,31 @@ class HashEquivalenceCommonTests(object):
             nonlocal side_client
=20
             # check upstream server
+            self.assertTrue(self.client.unihash_exists(unihash))
             self.assertClientGetHash(self.client, taskhash, unihash)
=20
             # Hash should *not* be present on the side server
+            if old_sidehash and unihash !=3D old_sidehash:
+                self.assertFalse(side_client.unihash_exists(unihash))
             self.assertClientGetHash(side_client, taskhash, old_sidehash)
=20
             # Hash should be present on the downstream server, since it
             # will defer to the upstream server. This will trigger
             # the backfill in the downstream server
+            self.assertTrue(down_client.unihash_exists(unihash))
             self.assertClientGetHash(down_client, taskhash, unihash)
=20
             # After waiting for the downstream client to finish backfillin=
g the
             # task from the upstream server, it should appear in the side =
server
             # since the database is populated
             down_client.backfill_wait()
+            self.assertTrue(side_client.unihash_exists(unihash))
             self.assertClientGetHash(side_client, taskhash, unihash)
=20
         # Basic report
         taskhash =3D '8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a'
         outhash =3D 'afe240a439959ce86f5e322f8c208e1fedefea9e813f2140c81af=
866cc9edf7e'
-        unihash =3D '218e57509998197d570e2c98512d0105985dffc9'
+        unihash =3D '5b521d8a12683086cc08bc2c6d94a7a2dcff17eba53b9911e145d=
51164689380'
         self.client.report_unihash(taskhash, self.METHOD, outhash, unihash=
)
=20
         check_hash(taskhash, unihash, None)
@@ -373,7 +410,7 @@ class HashEquivalenceCommonTests(object):
         # Duplicated taskhash with multiple output hashes and unihashes.
         # All servers should agree with the originally reported hash
         outhash2 =3D '0904a7fe3dc712d9fd8a74a616ddca2a825a8ee97adf0bd3fc86=
082c7639914d'
-        unihash2 =3D 'ae9a7d252735f0dafcdb10e2e02561ca3a47314c'
+        unihash2 =3D 'a37541b54fd22440e292f617eb30ba07455e88fb0b9f0952eca2=
29b6356290e3'
         self.client.report_unihash(taskhash, self.METHOD, outhash2, unihas=
h2)
=20
         check_hash(taskhash, unihash, unihash)
@@ -381,7 +418,7 @@ class HashEquivalenceCommonTests(object):
         # Report an equivalent task. The sideload will originally report
         # no unihash until backfilled
         taskhash3 =3D "044c2ec8aaf480685a00ff6ff49e6162e6ad34e1"
-        unihash3 =3D "def64766090d28f627e816454ed46894bb3aab36"
+        unihash3 =3D "aca636d800aef40e6ddcea4b2262cc4ea0d1180a6783e5b4653a=
20c7dd73458d"
         self.client.report_unihash(taskhash3, self.METHOD, outhash, unihas=
h3)
=20
         check_hash(taskhash3, unihash, None)
@@ -390,7 +427,7 @@ class HashEquivalenceCommonTests(object):
         # propagating to the upstream server
         taskhash4 =3D "e3da00593d6a7fb435c7e2114976c59c5fd6d561"
         outhash4 =3D "1cf8713e645f491eb9c959d20b5cae1c47133a292626dda9b107=
09857cbe688a"
-        unihash4 =3D "3b5d3d83f07f259e9086fcb422c855286e18a57d"
+        unihash4 =3D "7aebef07d66a8c0f92d0c4f65ec8b1fbb850a3693c53827b8774=
b64fa9a8a9fe"
         down_client.report_unihash(taskhash4, self.METHOD, outhash4, uniha=
sh4)
         down_client.backfill_wait()
=20
@@ -402,18 +439,18 @@ class HashEquivalenceCommonTests(object):
         # match which was previously reported to the upstream server
         taskhash5 =3D '35788efcb8dfb0a02659d81cf2bfd695fb30faf9'
         outhash5 =3D '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98=
fd7e5970796f'
-        unihash5 =3D 'f46d3fbb439bd9b921095da657a4de906510d2cd'
+        unihash5 =3D 'a69ec97f5af2e21e1a1f9cc8896965515d5559425666f734e245=
a3d40cee33d9'
         result =3D self.client.report_unihash(taskhash5, self.METHOD, outh=
ash5, unihash5)
=20
         taskhash6 =3D '35788efcb8dfb0a02659d81cf2bfd695fb30fafa'
-        unihash6 =3D 'f46d3fbb439bd9b921095da657a4de906510d2ce'
+        unihash6 =3D 'eabc7a98e0c12bbeb8394dbdf055eb81aac60e4a14cca5c1f069=
d36efc933b23'
         result =3D down_client.report_unihash(taskhash6, self.METHOD, outh=
ash5, unihash6)
         self.assertEqual(result['unihash'], unihash5, 'Server failed to co=
py unihash from upstream')
=20
         # Tests read through from server with
         taskhash7 =3D '9d81d76242cc7cfaf7bf74b94b9cd2e29324ed74'
         outhash7 =3D '8470d56547eea6236d7c81a644ce74670ca0bbda998e13c629ef=
6bb3f0d60b69'
-        unihash7 =3D '05d2a63c81e32f0a36542ca677e8ad852365c538'
+        unihash7 =3D '7521a98a0c645341bc51559b234ef37a097e8f3a01665e0303a3=
17925ab7b4d5'
         self.client.report_unihash(taskhash7, self.METHOD, outhash7, uniha=
sh7)
=20
         result =3D down_client.get_taskhash(self.METHOD, taskhash7, True)
@@ -424,7 +461,7 @@ class HashEquivalenceCommonTests(object):
=20
         taskhash8 =3D '86978a4c8c71b9b487330b0152aade10c1ee58aa'
         outhash8 =3D 'ca8c128e9d9e4a28ef24d0508aa20b5cf880604eacd8f65c0e36=
6f7e0cc5fbcf'
-        unihash8 =3D 'd8bcf25369d40590ad7d08c84d538982f2023e01'
+        unihash8 =3D '83386d9385b0bf3ba25693127ddcaaadeaa1c4bf8cb0baecfb53=
14b9a20072a1'
         self.client.report_unihash(taskhash8, self.METHOD, outhash8, uniha=
sh8)
=20
         result =3D down_client.get_outhash(self.METHOD, outhash8, taskhash=
8)
@@ -435,7 +472,7 @@ class HashEquivalenceCommonTests(object):
=20
         taskhash9 =3D 'ae6339531895ddf5b67e663e6a374ad8ec71d81c'
         outhash9 =3D 'afc78172c81880ae10a1fec994b5b4ee33d196a001a1b66212a1=
5ebe573e00b5'
-        unihash9 =3D '6662e699d6e3d894b24408ff9a4031ef9b038ee8'
+        unihash9 =3D 'cc74784b2c0ad5b378a6b783c74c518d2c46b8b52fba29cb39a8=
430d742440d7'
         self.client.report_unihash(taskhash9, self.METHOD, outhash9, uniha=
sh9)
=20
         result =3D down_client.get_taskhash(self.METHOD, taskhash9, False)
@@ -443,6 +480,169 @@ class HashEquivalenceCommonTests(object):
         self.assertEqual(result['taskhash'], taskhash9, 'Server failed to =
copy unihash from upstream')
         self.assertEqual(result['method'], self.METHOD)
=20
+    def test_upstream_batch(self):
+        down_server =3D self.start_server(upstream=3Dself.server.address)
+        down_client =3D self.start_client(down_server.address)
+
+        taskhash1 =3D '8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a'
+        outhash1 =3D 'afe240a439959ce86f5e322f8c208e1fedefea9e813f2140c81a=
f866cc9edf7e'
+        unihash1 =3D '5b521d8a12683086cc08bc2c6d94a7a2dcff17eba53b9911e145=
d51164689380'
+        self.client.report_unihash(taskhash1, self.METHOD, outhash1, uniha=
sh1)
+
+        taskhash2 =3D "e3da00593d6a7fb435c7e2114976c59c5fd6d561"
+        outhash2 =3D "1cf8713e645f491eb9c959d20b5cae1c47133a292626dda9b107=
09857cbe688a"
+        unihash2 =3D "7aebef07d66a8c0f92d0c4f65ec8b1fbb850a3693c53827b8774=
b64fa9a8a9fe"
+        self.client.report_unihash(taskhash2, self.METHOD, outhash2, uniha=
sh2)
+
+        taskhash3 =3D '35788efcb8dfb0a02659d81cf2bfd695fb30faf9'
+        outhash3 =3D '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98=
fd7e5970796f'
+        unihash3 =3D 'a69ec97f5af2e21e1a1f9cc8896965515d5559425666f734e245=
a3d40cee33d9'
+        self.client.report_unihash(taskhash3, self.METHOD, outhash3, uniha=
sh3)
+
+        def query_generator():
+            yield unihash1
+            yield unihash2
+            yield unihash3
+            yield "cc74784b2c0ad5b378a6b783c74c518d2c46b8b52fba29cb39a8430=
d742440d7"
+
+        results =3D down_client.unihash_exists_batch(query_generator())
+        self.assertEqual(results, [True, True, True, False])
+
+    def test_upstream_interrupted(self):
+        up_server =3D self.start_server()
+        down_server =3D self.start_server(upstream=3Dup_server.address)
+
+        def restart_upstream():
+            nonlocal up_server
+
+            self.stop_server(up_server)
+            up_server =3D self.start_server(addr=3Dup_server.address, dbpa=
th=3Dup_server.dbpath)
+
+        # Report some hashes
+        with self.start_client(up_server.address) as up_client:
+            taskhash1 =3D '8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a'
+            outhash1 =3D 'afe240a439959ce86f5e322f8c208e1fedefea9e813f2140=
c81af866cc9edf7e'
+            unihash1 =3D '5b521d8a12683086cc08bc2c6d94a7a2dcff17eba53b9911=
e145d51164689380'
+            up_client.report_unihash(taskhash1, self.METHOD, outhash1, uni=
hash1)
+
+            taskhash2 =3D "e3da00593d6a7fb435c7e2114976c59c5fd6d561"
+            outhash2 =3D "1cf8713e645f491eb9c959d20b5cae1c47133a292626dda9=
b10709857cbe688a"
+            unihash2 =3D "7aebef07d66a8c0f92d0c4f65ec8b1fbb850a3693c53827b=
8774b64fa9a8a9fe"
+            up_client.report_unihash(taskhash2, self.METHOD, outhash2, uni=
hash2)
+
+            taskhash3 =3D '35788efcb8dfb0a02659d81cf2bfd695fb30faf9'
+            outhash3 =3D '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3f=
ce98fd7e5970796f'
+            unihash3 =3D 'a69ec97f5af2e21e1a1f9cc8896965515d5559425666f734=
e245a3d40cee33d9'
+            up_client.report_unihash(taskhash3, self.METHOD, outhash3, uni=
hash3)
+
+        restart_upstream()
+
+        with self.start_client(up_server.address) as up_client:
+            # Verify that reported hashes are correct after restaring serv=
er
+            self.assertTrue(up_client.unihash_exists(unihash1))
+            self.assertClientGetHash(up_client, taskhash1, unihash1)
+
+            self.assertTrue(up_client.unihash_exists(unihash2))
+            self.assertClientGetHash(up_client, taskhash2, unihash2)
+
+        async def check_unihashes():
+            async with await create_async_client(down_server.address) as d=
own_client:
+                async with down_client.unihash_exists_stream() as stream:
+                    await stream.send_query(unihash1)
+                    r =3D await stream.get_result()
+                    self.assertTrue(r)
+
+                    restart_upstream()
+
+                    await stream.send_query(unihash2)
+                    r =3D await stream.get_result()
+                    self.assertTrue(r)
+
+                    await stream.send_query(unihash3)
+                    r =3D await stream.get_result()
+                    self.assertTrue(r)
+
+        asyncio.run(check_unihashes())
+
+    def test_upstream_lost(self):
+        up_server =3D self.start_server()
+        down_server =3D self.start_server(upstream=3Dup_server.address)
+
+        def restart_upstream():
+            nonlocal up_server
+
+            self.stop_server(up_server)
+            up_server =3D self.start_server(addr=3Dup_server.address, dbpa=
th=3Dup_server.dbpath)
+
+        # Report some hashes
+        with self.start_client(up_server.address) as up_client:
+            taskhash1 =3D '8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a'
+            outhash1 =3D 'afe240a439959ce86f5e322f8c208e1fedefea9e813f2140=
c81af866cc9edf7e'
+            unihash1 =3D '5b521d8a12683086cc08bc2c6d94a7a2dcff17eba53b9911=
e145d51164689380'
+            up_client.report_unihash(taskhash1, self.METHOD, outhash1, uni=
hash1)
+
+            taskhash2 =3D "e3da00593d6a7fb435c7e2114976c59c5fd6d561"
+            outhash2 =3D "1cf8713e645f491eb9c959d20b5cae1c47133a292626dda9=
b10709857cbe688a"
+            unihash2 =3D "7aebef07d66a8c0f92d0c4f65ec8b1fbb850a3693c53827b=
8774b64fa9a8a9fe"
+            up_client.report_unihash(taskhash2, self.METHOD, outhash2, uni=
hash2)
+
+            taskhash3 =3D '35788efcb8dfb0a02659d81cf2bfd695fb30faf9'
+            outhash3 =3D '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3f=
ce98fd7e5970796f'
+            unihash3 =3D 'a69ec97f5af2e21e1a1f9cc8896965515d5559425666f734=
e245a3d40cee33d9'
+            up_client.report_unihash(taskhash3, self.METHOD, outhash3, uni=
hash3)
+
+        async def check_unihashes():
+            async with await create_async_client(down_server.address) as d=
own_client:
+                with self.assertRaises(ConnectionError):
+                    async with down_client.unihash_exists_stream() as stre=
am:
+                        await stream.send_query(unihash1)
+                        r =3D await stream.get_result()
+                        self.assertTrue(r)
+
+                        self.stop_server(up_server)
+
+                        await stream.send_query(unihash2)
+                        r =3D await stream.get_result()
+
+        asyncio.run(check_unihashes())
+
+    def test_upstream_get_stream_miss(self):
+        down_server =3D self.start_server(upstream=3Dself.server.address)
+        down_client =3D self.start_client(down_server.address)
+
+        # Two hashes present upstream (hits)
+        taskhash1 =3D '8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a'
+        outhash1 =3D 'afe240a439959ce86f5e322f8c208e1fedefea9e813f2140c81a=
f866cc9edf7e'
+        unihash1 =3D '5b521d8a12683086cc08bc2c6d94a7a2dcff17eba53b9911e145=
d51164689380'
+        self.client.report_unihash(taskhash1, self.METHOD, outhash1, uniha=
sh1)
+
+        taskhash2 =3D 'e3da00593d6a7fb435c7e2114976c59c5fd6d561'
+        outhash2 =3D '1cf8713e645f491eb9c959d20b5cae1c47133a292626dda9b107=
09857cbe688a'
+        unihash2 =3D '7aebef07d66a8c0f92d0c4f65ec8b1fbb850a3693c53827b8774=
b64fa9a8a9fe'
+        self.client.report_unihash(taskhash2, self.METHOD, outhash2, uniha=
sh2)
+
+        # Two taskhashes present nowhere (upstream misses)
+        miss1 =3D '0000000000000000000000000000000000000001'
+        miss2 =3D '0000000000000000000000000000000000000002'
+
+        # Miss interleaved with hits: a miss must not truncate the stream
+        results =3D down_client.get_unihash_batch([
+            (self.METHOD, miss1),
+            (self.METHOD, taskhash1),
+            (self.METHOD, miss2),
+            (self.METHOD, taskhash2),
+        ])
+        self.assertEqual(results, [None, unihash1, None, unihash2])
+
+        # All-miss batch
+        self.assertEqual(
+            down_client.get_unihash_batch([(self.METHOD, miss1), (self.MET=
HOD, miss2)]),
+            [None, None],
+        )
+
+        # Singular get-unihash miss
+        self.assertClientGetHash(down_client, miss1, None)
+
     def test_unihash_exsits(self):
         taskhash, outhash, unihash =3D self.create_test_hash(self.client)
         self.assertTrue(self.client.unihash_exists(unihash))
@@ -458,7 +658,7 @@ class HashEquivalenceCommonTests(object):
         # Report a hash via the read-write server
         taskhash =3D '35788efcb8dfb0a02659d81cf2bfd695fb30faf9'
         outhash =3D '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98f=
d7e5970796f'
-        unihash =3D 'f46d3fbb439bd9b921095da657a4de906510d2cd'
+        unihash =3D 'a69ec97f5af2e21e1a1f9cc8896965515d5559425666f734e245a=
3d40cee33d9'
=20
         result =3D rw_client.report_unihash(taskhash, self.METHOD, outhash=
, unihash)
         self.assertEqual(result['unihash'], unihash, 'Server returned bad =
unihash')
@@ -469,7 +669,7 @@ class HashEquivalenceCommonTests(object):
         # Ensure that reporting via the read-only server fails
         taskhash2 =3D 'c665584ee6817aa99edfc77a44dd853828279370'
         outhash2 =3D '3c979c3db45c569f51ab7626a4651074be3a9d11a84b1db076f5=
b14f7d39db44'
-        unihash2 =3D '90e9bc1d1f094c51824adca7f8ea79a048d68824'
+        unihash2 =3D '06f89b8f329ba8124ff73c56d09ef921b42624747c421277bdda=
f5e23f136e57'
=20
         result =3D ro_client.report_unihash(taskhash2, self.METHOD, outhas=
h2, unihash2)
         self.assertEqual(result['unihash'], unihash2)
@@ -555,57 +755,111 @@ class HashEquivalenceCommonTests(object):
         # shares a taskhash with Task 2
         self.assertClientGetHash(self.client, taskhash2, unihash2)
=20
-
-    def test_client_pool_get_unihashes(self):
+    def test_get_unihash_batch(self):
         TEST_INPUT =3D (
             # taskhash                                   outhash          =
                                                  unihash
-            ('8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a', 'afe240a439959ce8=
6f5e322f8c208e1fedefea9e813f2140c81af866cc9edf7e','218e57509998197d570e2c98=
512d0105985dffc9'),
+            (
+                '8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a',
+                'afe240a439959ce86f5e322f8c208e1fedefea9e813f2140c81af866c=
c9edf7e',
+                '5b521d8a12683086cc08bc2c6d94a7a2dcff17eba53b9911e145d5116=
4689380',
+            ),
             # Duplicated taskhash with multiple output hashes and unihashe=
s.
-            ('8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a', '0904a7fe3dc712d9=
fd8a74a616ddca2a825a8ee97adf0bd3fc86082c7639914d', 'ae9a7d252735f0dafcdb10e=
2e02561ca3a47314c'),
+            (
+                '8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a',
+                '0904a7fe3dc712d9fd8a74a616ddca2a825a8ee97adf0bd3fc86082c7=
639914d',
+                'a37541b54fd22440e292f617eb30ba07455e88fb0b9f0952eca229b63=
56290e3',
+            ),
             # Equivalent hash
-            ("044c2ec8aaf480685a00ff6ff49e6162e6ad34e1", '0904a7fe3dc712d9=
fd8a74a616ddca2a825a8ee97adf0bd3fc86082c7639914d', "def64766090d28f627e8164=
54ed46894bb3aab36"),
-            ("e3da00593d6a7fb435c7e2114976c59c5fd6d561", "1cf8713e645f491e=
b9c959d20b5cae1c47133a292626dda9b10709857cbe688a", "3b5d3d83f07f259e9086fcb=
422c855286e18a57d"),
-            ('35788efcb8dfb0a02659d81cf2bfd695fb30faf9', '2765d4a5884be49b=
28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5970796f', 'f46d3fbb439bd9b921095da=
657a4de906510d2cd'),
-            ('35788efcb8dfb0a02659d81cf2bfd695fb30fafa', '2765d4a5884be49b=
28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5970796f', 'f46d3fbb439bd9b921095da=
657a4de906510d2ce'),
-            ('9d81d76242cc7cfaf7bf74b94b9cd2e29324ed74', '8470d56547eea623=
6d7c81a644ce74670ca0bbda998e13c629ef6bb3f0d60b69', '05d2a63c81e32f0a36542ca=
677e8ad852365c538'),
+            (
+                "044c2ec8aaf480685a00ff6ff49e6162e6ad34e1",
+                '0904a7fe3dc712d9fd8a74a616ddca2a825a8ee97adf0bd3fc86082c7=
639914d',
+                "aca636d800aef40e6ddcea4b2262cc4ea0d1180a6783e5b4653a20c7d=
d73458d",
+            ),
+            (
+                "e3da00593d6a7fb435c7e2114976c59c5fd6d561",
+                "1cf8713e645f491eb9c959d20b5cae1c47133a292626dda9b10709857=
cbe688a",
+                "7aebef07d66a8c0f92d0c4f65ec8b1fbb850a3693c53827b8774b64fa=
9a8a9fe",
+            ),
+            (
+                '35788efcb8dfb0a02659d81cf2bfd695fb30faf9',
+                '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5=
970796f',
+                'a69ec97f5af2e21e1a1f9cc8896965515d5559425666f734e245a3d40=
cee33d9',
+            ),
+            (
+                '35788efcb8dfb0a02659d81cf2bfd695fb30fafa',
+                '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5=
970796f',
+                'eabc7a98e0c12bbeb8394dbdf055eb81aac60e4a14cca5c1f069d36ef=
c933b23',
+            ),
+            (
+                '9d81d76242cc7cfaf7bf74b94b9cd2e29324ed74',
+                '8470d56547eea6236d7c81a644ce74670ca0bbda998e13c629ef6bb3f=
0d60b69',
+                '7521a98a0c645341bc51559b234ef37a097e8f3a01665e0303a317925=
ab7b4d5',
+            ),
         )
         EXTRA_QUERIES =3D (
             "6b6be7a84ab179b4240c4302518dc3f6",
         )
=20
-        with ClientPool(self.server_address, 10) as client_pool:
-            for taskhash, outhash, unihash in TEST_INPUT:
-                self.client.report_unihash(taskhash, self.METHOD, outhash,=
 unihash)
-
-            query =3D {idx: (self.METHOD, data[0]) for idx, data in enumer=
ate(TEST_INPUT)}
-            for idx, taskhash in enumerate(EXTRA_QUERIES):
-                query[idx + len(TEST_INPUT)] =3D (self.METHOD, taskhash)
-
-            result =3D client_pool.get_unihashes(query)
-
-            self.assertDictEqual(result, {
-                0: "218e57509998197d570e2c98512d0105985dffc9",
-                1: "218e57509998197d570e2c98512d0105985dffc9",
-                2: "218e57509998197d570e2c98512d0105985dffc9",
-                3: "3b5d3d83f07f259e9086fcb422c855286e18a57d",
-                4: "f46d3fbb439bd9b921095da657a4de906510d2cd",
-                5: "f46d3fbb439bd9b921095da657a4de906510d2cd",
-                6: "05d2a63c81e32f0a36542ca677e8ad852365c538",
-                7: None,
-            })
+        for taskhash, outhash, unihash in TEST_INPUT:
+            self.client.report_unihash(taskhash, self.METHOD, outhash, uni=
hash)
=20
-    def test_client_pool_unihash_exists(self):
+
+        result =3D self.client.get_unihash_batch(
+            [(self.METHOD, data[0]) for data in TEST_INPUT] +
+            [(self.METHOD, e) for e in EXTRA_QUERIES]
+        )
+
+        self.assertListEqual(result, [
+            "5b521d8a12683086cc08bc2c6d94a7a2dcff17eba53b9911e145d51164689=
380",
+            "5b521d8a12683086cc08bc2c6d94a7a2dcff17eba53b9911e145d51164689=
380",
+            "5b521d8a12683086cc08bc2c6d94a7a2dcff17eba53b9911e145d51164689=
380",
+            "7aebef07d66a8c0f92d0c4f65ec8b1fbb850a3693c53827b8774b64fa9a8a=
9fe",
+            "a69ec97f5af2e21e1a1f9cc8896965515d5559425666f734e245a3d40cee3=
3d9",
+            "a69ec97f5af2e21e1a1f9cc8896965515d5559425666f734e245a3d40cee3=
3d9",
+            "7521a98a0c645341bc51559b234ef37a097e8f3a01665e0303a317925ab7b=
4d5",
+            None,
+        ])
+
+    def test_unihash_exists_batch(self):
         TEST_INPUT =3D (
             # taskhash                                   outhash          =
                                                  unihash
-            ('8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a', 'afe240a439959ce8=
6f5e322f8c208e1fedefea9e813f2140c81af866cc9edf7e','218e57509998197d570e2c98=
512d0105985dffc9'),
+            (
+                '8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a',
+                'afe240a439959ce86f5e322f8c208e1fedefea9e813f2140c81af866c=
c9edf7e',
+                '5b521d8a12683086cc08bc2c6d94a7a2dcff17eba53b9911e145d5116=
4689380',
+            ),
             # Duplicated taskhash with multiple output hashes and unihashe=
s.
-            ('8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a', '0904a7fe3dc712d9=
fd8a74a616ddca2a825a8ee97adf0bd3fc86082c7639914d', 'ae9a7d252735f0dafcdb10e=
2e02561ca3a47314c'),
+            (
+                '8aa96fcffb5831b3c2c0cb75f0431e3f8b20554a',
+                '0904a7fe3dc712d9fd8a74a616ddca2a825a8ee97adf0bd3fc86082c7=
639914d',
+                'a37541b54fd22440e292f617eb30ba07455e88fb0b9f0952eca229b63=
56290e3',
+            ),
             # Equivalent hash
-            ("044c2ec8aaf480685a00ff6ff49e6162e6ad34e1", '0904a7fe3dc712d9=
fd8a74a616ddca2a825a8ee97adf0bd3fc86082c7639914d', "def64766090d28f627e8164=
54ed46894bb3aab36"),
-            ("e3da00593d6a7fb435c7e2114976c59c5fd6d561", "1cf8713e645f491e=
b9c959d20b5cae1c47133a292626dda9b10709857cbe688a", "3b5d3d83f07f259e9086fcb=
422c855286e18a57d"),
-            ('35788efcb8dfb0a02659d81cf2bfd695fb30faf9', '2765d4a5884be49b=
28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5970796f', 'f46d3fbb439bd9b921095da=
657a4de906510d2cd'),
-            ('35788efcb8dfb0a02659d81cf2bfd695fb30fafa', '2765d4a5884be49b=
28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5970796f', 'f46d3fbb439bd9b921095da=
657a4de906510d2ce'),
-            ('9d81d76242cc7cfaf7bf74b94b9cd2e29324ed74', '8470d56547eea623=
6d7c81a644ce74670ca0bbda998e13c629ef6bb3f0d60b69', '05d2a63c81e32f0a36542ca=
677e8ad852365c538'),
+            (
+                "044c2ec8aaf480685a00ff6ff49e6162e6ad34e1",
+                '0904a7fe3dc712d9fd8a74a616ddca2a825a8ee97adf0bd3fc86082c7=
639914d',
+                "aca636d800aef40e6ddcea4b2262cc4ea0d1180a6783e5b4653a20c7d=
d73458d",
+            ),
+            (
+                "e3da00593d6a7fb435c7e2114976c59c5fd6d561",
+                "1cf8713e645f491eb9c959d20b5cae1c47133a292626dda9b10709857=
cbe688a",
+                "7aebef07d66a8c0f92d0c4f65ec8b1fbb850a3693c53827b8774b64fa=
9a8a9fe",
+            ),
+            (
+                '35788efcb8dfb0a02659d81cf2bfd695fb30faf9',
+                '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5=
970796f',
+                'a69ec97f5af2e21e1a1f9cc8896965515d5559425666f734e245a3d40=
cee33d9',
+            ),
+            (
+                '35788efcb8dfb0a02659d81cf2bfd695fb30fafa',
+                '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98fd7e5=
970796f',
+                'eabc7a98e0c12bbeb8394dbdf055eb81aac60e4a14cca5c1f069d36ef=
c933b23',
+            ),
+            (
+                '9d81d76242cc7cfaf7bf74b94b9cd2e29324ed74',
+                '8470d56547eea6236d7c81a644ce74670ca0bbda998e13c629ef6bb3f=
0d60b69',
+                '7521a98a0c645341bc51559b234ef37a097e8f3a01665e0303a317925=
ab7b4d5',
+            ),
         )
         EXTRA_QUERIES =3D (
             "6b6be7a84ab179b4240c4302518dc3f6",
@@ -614,28 +868,24 @@ class HashEquivalenceCommonTests(object):
         result_unihashes =3D set()
=20
=20
-        with ClientPool(self.server_address, 10) as client_pool:
-            for taskhash, outhash, unihash in TEST_INPUT:
-                result =3D self.client.report_unihash(taskhash, self.METHO=
D, outhash, unihash)
-                result_unihashes.add(result["unihash"])
-
-            query =3D {}
-            expected =3D {}
+        for taskhash, outhash, unihash in TEST_INPUT:
+            result =3D self.client.report_unihash(taskhash, self.METHOD, o=
uthash, unihash)
+            result_unihashes.add(result["unihash"])
=20
-            for _, _, unihash in TEST_INPUT:
-                idx =3D len(query)
-                query[idx] =3D unihash
-                expected[idx] =3D unihash in result_unihashes
+        query =3D []
+        expected =3D []
=20
+        for _, _, unihash in TEST_INPUT:
+            query.append(unihash)
+            expected.append(unihash in result_unihashes)
=20
-            for unihash in EXTRA_QUERIES:
-                idx =3D len(query)
-                query[idx] =3D unihash
-                expected[idx] =3D False
=20
-            result =3D client_pool.unihashes_exist(query)
-            self.assertDictEqual(result, expected)
+        for unihash in EXTRA_QUERIES:
+            query.append(unihash)
+            expected.append(False)
=20
+        result =3D self.client.unihash_exists_batch(query)
+        self.assertListEqual(result, expected)
=20
     def test_auth_read_perms(self):
         admin_client =3D self.start_auth_server()
@@ -949,14 +1199,14 @@ class HashEquivalenceCommonTests(object):
     def test_gc(self):
         taskhash =3D '53b8dce672cb6d0c73170be43f540460bfc347b4'
         outhash =3D '5a9cb1649625f0bf41fc7791b635cd9c2d7118c7f021ba87dcd03=
f72b67ce7a8'
-        unihash =3D 'f37918cc02eb5a520b1aff86faacbc0a38124646'
+        unihash =3D '46edb5140d2613049332d0bf3745d9fafec9c559dac8cc6181373=
9a28007fcdf'
=20
         result =3D self.client.report_unihash(taskhash, self.METHOD, outha=
sh, unihash)
         self.assertEqual(result['unihash'], unihash, 'Server returned bad =
unihash')
=20
         taskhash2 =3D '3bf6f1e89d26205aec90da04854fbdbf73afe6b4'
         outhash2 =3D '77623a549b5b1a31e3732dfa8fe61d7ce5d44b3370f253c5360e=
136b852967b4'
-        unihash2 =3D 'af36b199320e611fbb16f1f277d3ee1d619ca58b'
+        unihash2 =3D 'bf6e81926066f770e960f9f777cd088c62bea9addb7745f3e77d=
eaa81a645747'
=20
         result =3D self.client.report_unihash(taskhash2, self.METHOD, outh=
ash2, unihash2)
         self.assertClientGetHash(self.client, taskhash2, unihash2)
@@ -979,17 +1229,59 @@ class HashEquivalenceCommonTests(object):
         # First hash is still present
         self.assertClientGetHash(self.client, taskhash, unihash)
=20
+    def test_gc_batch(self):
+        taskhash =3D '53b8dce672cb6d0c73170be43f540460bfc347b4'
+        outhash =3D '5a9cb1649625f0bf41fc7791b635cd9c2d7118c7f021ba87dcd03=
f72b67ce7a8'
+        unihash =3D '46edb5140d2613049332d0bf3745d9fafec9c559dac8cc6181373=
9a28007fcdf'
+
+        result =3D self.client.report_unihash(taskhash, self.METHOD, outha=
sh, unihash)
+        self.assertEqual(result['unihash'], unihash, 'Server returned bad =
unihash')
+
+        taskhash2 =3D '3bf6f1e89d26205aec90da04854fbdbf73afe6b4'
+        outhash2 =3D '77623a549b5b1a31e3732dfa8fe61d7ce5d44b3370f253c5360e=
136b852967b4'
+        unihash2 =3D 'bf6e81926066f770e960f9f777cd088c62bea9addb7745f3e77d=
eaa81a645747'
+
+        result =3D self.client.report_unihash(taskhash2, self.METHOD, outh=
ash2, unihash2)
+        self.assertClientGetHash(self.client, taskhash2, unihash2)
+
+        taskhash3 =3D 'a1117c1f5a7c9ab2f5a39cc6fe5e6152169d09c0'
+        outhash3 =3D '7289c414905303700a1117c1f5a7c9ab2f5a39cc6fe5e6152169=
d09c04f9a53c'
+        unihash3 =3D 'f5966c8588a69948a89131d3aa63b914c5a63a8cb4820cb1c586=
94135ba77f0b'
+
+        result =3D self.client.report_unihash(taskhash3, self.METHOD, outh=
ash3, unihash3)
+        self.assertClientGetHash(self.client, taskhash3, unihash3)
+
+        # Mark the first unihash to be kept
+        ret =3D self.client.gc_mark_batch("ABC", (f"unihash {h}" for h in =
[unihash, unihash2]))
+        self.assertEqual(ret, {"count": 2})
+
+        ret =3D self.client.gc_status()
+        self.assertEqual(ret, {"mark": "ABC", "keep": 2, "remove": 1})
+
+        # Third hash is still there; mark doesn't delete hashes
+        self.assertClientGetHash(self.client, taskhash3, unihash3)
+
+        ret =3D self.client.gc_sweep("ABC")
+        self.assertEqual(ret, {"count": 1})
+
+        # Hash is gone. Taskhash is returned for second hash
+        self.assertClientGetHash(self.client, taskhash3, None)
+        # First hash is still present
+        self.assertClientGetHash(self.client, taskhash, unihash)
+        # Second hash is still present
+        self.assertClientGetHash(self.client, taskhash2, unihash2)
+
     def test_gc_switch_mark(self):
         taskhash =3D '53b8dce672cb6d0c73170be43f540460bfc347b4'
         outhash =3D '5a9cb1649625f0bf41fc7791b635cd9c2d7118c7f021ba87dcd03=
f72b67ce7a8'
-        unihash =3D 'f37918cc02eb5a520b1aff86faacbc0a38124646'
+        unihash =3D '46edb5140d2613049332d0bf3745d9fafec9c559dac8cc6181373=
9a28007fcdf'
=20
         result =3D self.client.report_unihash(taskhash, self.METHOD, outha=
sh, unihash)
         self.assertEqual(result['unihash'], unihash, 'Server returned bad =
unihash')
=20
         taskhash2 =3D '3bf6f1e89d26205aec90da04854fbdbf73afe6b4'
         outhash2 =3D '77623a549b5b1a31e3732dfa8fe61d7ce5d44b3370f253c5360e=
136b852967b4'
-        unihash2 =3D 'af36b199320e611fbb16f1f277d3ee1d619ca58b'
+        unihash2 =3D 'bf6e81926066f770e960f9f777cd088c62bea9addb7745f3e77d=
eaa81a645747'
=20
         result =3D self.client.report_unihash(taskhash2, self.METHOD, outh=
ash2, unihash2)
         self.assertClientGetHash(self.client, taskhash2, unihash2)
@@ -1027,14 +1319,14 @@ class HashEquivalenceCommonTests(object):
     def test_gc_switch_sweep_mark(self):
         taskhash =3D '53b8dce672cb6d0c73170be43f540460bfc347b4'
         outhash =3D '5a9cb1649625f0bf41fc7791b635cd9c2d7118c7f021ba87dcd03=
f72b67ce7a8'
-        unihash =3D 'f37918cc02eb5a520b1aff86faacbc0a38124646'
+        unihash =3D '46edb5140d2613049332d0bf3745d9fafec9c559dac8cc6181373=
9a28007fcdf'
=20
         result =3D self.client.report_unihash(taskhash, self.METHOD, outha=
sh, unihash)
         self.assertEqual(result['unihash'], unihash, 'Server returned bad =
unihash')
=20
         taskhash2 =3D '3bf6f1e89d26205aec90da04854fbdbf73afe6b4'
         outhash2 =3D '77623a549b5b1a31e3732dfa8fe61d7ce5d44b3370f253c5360e=
136b852967b4'
-        unihash2 =3D 'af36b199320e611fbb16f1f277d3ee1d619ca58b'
+        unihash2 =3D 'bf6e81926066f770e960f9f777cd088c62bea9addb7745f3e77d=
eaa81a645747'
=20
         result =3D self.client.report_unihash(taskhash2, self.METHOD, outh=
ash2, unihash2)
         self.assertClientGetHash(self.client, taskhash2, unihash2)
@@ -1057,7 +1349,7 @@ class HashEquivalenceCommonTests(object):
     def test_gc_new_hashes(self):
         taskhash =3D '53b8dce672cb6d0c73170be43f540460bfc347b4'
         outhash =3D '5a9cb1649625f0bf41fc7791b635cd9c2d7118c7f021ba87dcd03=
f72b67ce7a8'
-        unihash =3D 'f37918cc02eb5a520b1aff86faacbc0a38124646'
+        unihash =3D '46edb5140d2613049332d0bf3745d9fafec9c559dac8cc6181373=
9a28007fcdf'
=20
         result =3D self.client.report_unihash(taskhash, self.METHOD, outha=
sh, unihash)
         self.assertEqual(result['unihash'], unihash, 'Server returned bad =
unihash')
@@ -1074,7 +1366,7 @@ class HashEquivalenceCommonTests(object):
=20
         taskhash2 =3D '3bf6f1e89d26205aec90da04854fbdbf73afe6b4'
         outhash2 =3D '77623a549b5b1a31e3732dfa8fe61d7ce5d44b3370f253c5360e=
136b852967b4'
-        unihash2 =3D 'af36b199320e611fbb16f1f277d3ee1d619ca58b'
+        unihash2 =3D 'bf6e81926066f770e960f9f777cd088c62bea9addb7745f3e77d=
eaa81a645747'
=20
         result =3D self.client.report_unihash(taskhash2, self.METHOD, outh=
ash2, unihash2)
         self.assertClientGetHash(self.client, taskhash2, unihash2)
@@ -1373,14 +1665,14 @@ class TestHashEquivalenceClient(HashEquivalenceTest=
Setup, unittest.TestCase):
     def test_gc(self):
         taskhash =3D '53b8dce672cb6d0c73170be43f540460bfc347b4'
         outhash =3D '5a9cb1649625f0bf41fc7791b635cd9c2d7118c7f021ba87dcd03=
f72b67ce7a8'
-        unihash =3D 'f37918cc02eb5a520b1aff86faacbc0a38124646'
+        unihash =3D '46edb5140d2613049332d0bf3745d9fafec9c559dac8cc6181373=
9a28007fcdf'
=20
         result =3D self.client.report_unihash(taskhash, self.METHOD, outha=
sh, unihash)
         self.assertEqual(result['unihash'], unihash, 'Server returned bad =
unihash')
=20
         taskhash2 =3D '3bf6f1e89d26205aec90da04854fbdbf73afe6b4'
         outhash2 =3D '77623a549b5b1a31e3732dfa8fe61d7ce5d44b3370f253c5360e=
136b852967b4'
-        unihash2 =3D 'af36b199320e611fbb16f1f277d3ee1d619ca58b'
+        unihash2 =3D 'bf6e81926066f770e960f9f777cd088c62bea9addb7745f3e77d=
eaa81a645747'
=20
         result =3D self.client.report_unihash(taskhash2, self.METHOD, outh=
ash2, unihash2)
         self.assertClientGetHash(self.client, taskhash2, unihash2)
@@ -1423,7 +1715,7 @@ class TestHashEquivalenceUnixServerLongPath(HashEquiv=
alenceTestSetup, unittest.T
         # Simple test that hashes can be created
         taskhash =3D '35788efcb8dfb0a02659d81cf2bfd695fb30faf9'
         outhash =3D '2765d4a5884be49b28601445c2760c5f21e7e5c0ee2b7e3fce98f=
d7e5970796f'
-        unihash =3D 'f46d3fbb439bd9b921095da657a4de906510d2cd'
+        unihash =3D 'a69ec97f5af2e21e1a1f9cc8896965515d5559425666f734e245a=
3d40cee33d9'
=20
         self.assertClientGetHash(self.client, taskhash, None)
=20
@@ -1438,6 +1730,102 @@ class TestHashEquivalenceTCPServer(HashEquivalenceT=
estSetup, HashEquivalenceComm
         # case it is more reliable to resolve the IP address explicitly.
         return socket.gethostbyname("localhost") + ":0"
=20
+    def test_get_stream_upstream_pipelined(self):
+        # Verify that, with an upstream configured, the get-stream handler
+        # pipelines its upstream queries instead of doing one blocking
+        # round-trip per task. A latency proxy injects RTT between this se=
rver
+        # and its upstream; a serial (one round-trip per task) implementat=
ion
+        # could not beat N * RTT, so finishing far faster proves the queri=
es
+        # are pipelined.
+        import asyncio
+
+        LATENCY =3D 0.01  # 10ms each direction =3D> ~20ms round trip
+        upstream_host, upstream_port =3D self.server_address.rsplit(":", 1=
)
+        upstream_port =3D int(upstream_port)
+
+        ready =3D threading.Event()
+        proxy =3D {}
+
+        def run_proxy():
+            loop =3D asyncio.new_event_loop()
+            asyncio.set_event_loop(loop)
+
+            async def handle(creader, cwriter):
+                ureader, uwriter =3D await asyncio.open_connection(upstrea=
m_host, upstream_port)
+
+                async def pipe(r, w):
+                    try:
+                        while True:
+                            data =3D await r.read(65536)
+                            if not data:
+                                break
+                            await asyncio.sleep(LATENCY)
+                            w.write(data)
+                            await w.drain()
+                    except Exception:
+                        pass
+                    finally:
+                        try:
+                            w.close()
+                        except Exception:
+                            pass
+
+                await bb.asyncrpc.TaskGroup.run(pipe(creader, uwriter), pi=
pe(ureader, cwriter))
+
+            stop_event =3D asyncio.Event()
+            proxy["stop_event"] =3D stop_event
+            proxy["loop"] =3D loop
+
+            async def main():
+                server =3D await asyncio.start_server(handle, "127.0.0.1",=
 0)
+                proxy["port"] =3D server.sockets[0].getsockname()[1]
+                ready.set()
+                async with server:
+                    await stop_event.wait()
+
+            try:
+                loop.run_until_complete(main())
+            except Exception:
+                pass
+            finally:
+                loop.close()
+
+        proxy_thread =3D threading.Thread(target=3Drun_proxy, daemon=3DTru=
e)
+        proxy_thread.start()
+        self.assertTrue(ready.wait(10), "latency proxy did not start")
+        self.addCleanup(proxy_thread.join, 10)
+        self.addCleanup(lambda: proxy["loop"].call_soon_threadsafe(proxy["=
stop_event"].set))
+        proxy_addr =3D "127.0.0.1:%d" % proxy["port"]
+
+        N =3D 400
+        expected =3D []
+        for i in range(N):
+            taskhash =3D hashlib.sha256(("task%d" % i).encode()).hexdigest=
()
+            outhash =3D hashlib.sha256(("out%d" % i).encode()).hexdigest()
+            unihash =3D hashlib.sha256(("uni%d" % i).encode()).hexdigest()
+            self.client.report_unihash(taskhash, self.METHOD, outhash, uni=
hash)
+            expected.append((self.METHOD, taskhash, unihash))
+
+        # Downstream server with an EMPTY local DB whose upstream is the s=
low proxy.
+        down_server =3D self.start_server(upstream=3Dproxy_addr)
+        down_client =3D self.start_client(down_server.address)
+
+        args =3D [(m, th) for (m, th, _uh) in expected]
+        start =3D time.time()
+        results =3D down_client.get_unihash_batch(args)
+        elapsed =3D time.time() - start
+
+        self.assertEqual(results, [uh for (_m, _th, uh) in expected])
+
+        # A serial (one round-trip per task) implementation cannot beat N*=
RTT.
+        # Allow a generous margin to avoid flakiness on loaded CI machines=
.
+        serial_lower_bound =3D N * 2 * LATENCY
+        self.assertLess(elapsed, serial_lower_bound / 4,
+                        "Upstream queries are not being pipelined "
+                        "(%.3fs for %d queries at %.0fms injected RTT)"
+                        % (elapsed, N, 2 * LATENCY * 1000))
+
+
=20
 class TestHashEquivalenceWebsocketServer(HashEquivalenceTestSetup, HashEqu=
ivalenceCommonTests, unittest.TestCase):
     def setUp(self):
@@ -1510,4 +1898,3 @@ class TestHashEquivalenceExternalServer(HashEquivalen=
ceTestSetup, HashEquivalenc
=20
     def test_auth_get_all_users(self):
         self.skipTest("Cannot test all users with external server")
-
diff --git a/bitbake/lib/layerindexlib/__init__.py b/bitbake/lib/layerindex=
lib/__init__.py
index c3265dda..e693e5c1 100644
--- a/bitbake/lib/layerindexlib/__init__.py
+++ b/bitbake/lib/layerindexlib/__init__.py
@@ -174,15 +174,15 @@ class LayerIndex():
         return res
=20
=20
-    def load_layerindex(self, indexURI, load=3D['layerDependencies', 'reci=
pes', 'machines', 'distros'], reload=3DFalse):
+    def load_layerindex(self, indexURI, load=3D[], reload=3DFalse):
         '''Load the layerindex.
=20
            indexURI - An index to load.  (Use multiple calls to load multi=
ple indexes)
=20
            reload - If reload is True, then any previously loaded indexes =
will be forgotten.
=20
-           load - List of elements to load.  Default loads all items.
-                  Note: plugs may ignore this.
+           load - List of elements to load. By default, an empty list is u=
sed to keep things lean.
+                  Callers need to specify a minimal set of elements to loa=
d, such as ['layerDependencies'] for dependency resolution.
=20
 The format of the indexURI:
=20
diff --git a/bitbake/lib/layerindexlib/cooker.py b/bitbake/lib/layerindexli=
b/cooker.py
index ced3e063..82d77335 100644
--- a/bitbake/lib/layerindexlib/cooker.py
+++ b/bitbake/lib/layerindexlib/cooker.py
@@ -55,10 +55,10 @@ class CookerPlugin(layerindexlib.plugin.IndexPlugin):
         bb_path =3D os.path.dirname(bb_path)  # .../bitbake/lib/layerindex
         bb_path =3D os.path.dirname(bb_path)  # .../bitbake/lib
         bb_path =3D os.path.dirname(bb_path)  # .../bitbake
-        bb_path =3D self._run_command('git rev-parse --show-toplevel', os.=
path.dirname(__file__), default=3Dbb_path)
-        bb_branch =3D self._run_command('git rev-parse --abbrev-ref HEAD',=
 bb_path, default=3D"")
-        bb_rev =3D self._run_command('git rev-parse HEAD', bb_path, defaul=
t=3D"")
-        for remotes in self._run_command('git remote -v', bb_path, default=
=3D"").split("\n"):
+        bb_path =3D self._run_command(['git', 'rev-parse', '--show-topleve=
l'], os.path.dirname(__file__), default=3Dbb_path)
+        bb_branch =3D self._run_command(['git', 'rev-parse', '--abbrev-ref=
', 'HEAD'], bb_path, default=3D"")
+        bb_rev =3D self._run_command(['git', 'rev-parse', 'HEAD'], bb_path=
, default=3D"")
+        for remotes in self._run_command(['git', 'remote', '-v'], bb_path,=
 default=3D"").split("\n"):
             remote =3D remotes.split("\t")[1].split(" ")[0]
             if "(fetch)" =3D=3D remotes.split("\t")[1].split(" ")[1]:
                 bb_remote =3D self._handle_git_remote(remote)
@@ -118,14 +118,14 @@ class CookerPlugin(layerindexlib.plugin.IndexPlugin):
             layerbranch =3D ""
=20
             if os.path.isdir(layerpath):
-                layerbasepath =3D self._run_command('git rev-parse --show-=
toplevel', layerpath, default=3Dlayerpath)
+                layerbasepath =3D self._run_command(['git', 'rev-parse', '=
--show-toplevel'], layerpath, default=3Dlayerpath)
                 if os.path.abspath(layerpath) !=3D os.path.abspath(layerba=
sepath):
                     layersubdir =3D os.path.abspath(layerpath)[len(layerba=
sepath) + 1:]
=20
-                layerbranch =3D self._run_command('git rev-parse --abbrev-=
ref HEAD', layerpath, default=3D"")
-                layerrev =3D self._run_command('git rev-parse HEAD', layer=
path, default=3D"")
+                layerbranch =3D self._run_command(['git', 'rev-parse', '--=
abbrev-ref', 'HEAD'], layerpath, default=3D"")
+                layerrev =3D self._run_command(['git', 'rev-parse', 'HEAD'=
], layerpath, default=3D"")
=20
-                for remotes in self._run_command('git remote -v', layerpat=
h, default=3D"").split("\n"):
+                for remotes in self._run_command(['git', 'remote', '-v'], =
layerpath, default=3D"").split("\n"):
                     if not remotes:
                         layerurl =3D self._handle_git_remote(layerpath)
                     else:
diff --git a/bitbake/lib/layerindexlib/restapi.py b/bitbake/lib/layerindexl=
ib/restapi.py
index 81d99b02..65405cb7 100644
--- a/bitbake/lib/layerindexlib/restapi.py
+++ b/bitbake/lib/layerindexlib/restapi.py
@@ -213,7 +213,7 @@ class RestApiPlugin(layerindexlib.plugin.IndexPlugin):
         pindex['branches'] =3D _get_json_response(index.apilinks['branches=
'] + filter,
                                                     username=3Dup.username=
, password=3Dup.password)
         if not pindex['branches']:
-            logger.debug("No valid branches (%s) found at url %s." % (bran=
ch, url))
+            logger.debug("No valid branches (%s) found at url %s." % (bran=
ches, url))
             return index
         index.add_raw_element("branches", layerindexlib.Branch, pindex['br=
anches'])
=20
diff --git a/bitbake/lib/ply/lex.py b/bitbake/lib/ply/lex.py
deleted file mode 100644
index 182f2e83..00000000
--- a/bitbake/lib/ply/lex.py
+++ /dev/null
@@ -1,1054 +0,0 @@
-# ------------------------------------------------------------------------=
-----
-# ply: lex.py
-#
-# Copyright (C) 2001-2009,
-# David M. Beazley (Dabeaz LLC)
-# All rights reserved.
-#
-# Redistribution and use in source and binary forms, with or without
-# modification, are permitted provided that the following conditions are
-# met:
-#=20
-# * Redistributions of source code must retain the above copyright notice,
-#   this list of conditions and the following disclaimer. =20
-# * Redistributions in binary form must reproduce the above copyright noti=
ce,=20
-#   this list of conditions and the following disclaimer in the documentat=
ion
-#   and/or other materials provided with the distribution. =20
-# * Neither the name of the David Beazley or Dabeaz LLC may be used to
-#   endorse or promote products derived from this software without
-#  specific prior written permission.=20
-#
-# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-# ------------------------------------------------------------------------=
-----
-
-__version__    =3D "3.3"
-__tabversion__ =3D "3.2"       # Version of table file used
-
-import re, sys, types, copy, os
-
-# This tuple contains known string types
-try:
-    # Python 2.6
-    StringTypes =3D (types.StringType, types.UnicodeType)
-except AttributeError:
-    # Python 3.0
-    StringTypes =3D (str, bytes)
-
-# Extract the code attribute of a function. Different implementations
-# are for Python 2/3 compatibility.
-
-if sys.version_info[0] < 3:
-    def func_code(f):
-        return f.func_code
-else:
-    def func_code(f):
-        return f.__code__
-
-# This regular expression is used to match valid token names
-_is_identifier =3D re.compile(r'^[a-zA-Z0-9_]+$')
-
-# Exception thrown when invalid token encountered and no default error
-# handler is defined.
-
-class LexError(Exception):
-    def __init__(self,message,s):
-         self.args =3D (message,)
-         self.text =3D s
-
-# Token class.  This class is used to represent the tokens produced.
-class LexToken(object):
-    def __str__(self):
-        return "LexToken(%s,%r,%d,%d)" % (self.type,self.value,self.lineno=
,self.lexpos)
-    def __repr__(self):
-        return str(self)
-
-# This object is a stand-in for a logging object created by the=20
-# logging module. =20
-
-class PlyLogger(object):
-    def __init__(self,f):
-        self.f =3D f
-    def critical(self,msg,*args,**kwargs):
-        self.f.write((msg % args) + "\n")
-
-    def warning(self,msg,*args,**kwargs):
-        self.f.write("WARNING: "+ (msg % args) + "\n")
-
-    def error(self,msg,*args,**kwargs):
-        self.f.write("ERROR: " + (msg % args) + "\n")
-
-    info =3D critical
-    debug =3D critical
-
-# Null logger is used when no output is generated. Does nothing.
-class NullLogger(object):
-    def __getattribute__(self,name):
-        return self
-    def __call__(self,*args,**kwargs):
-        return self
-
-# ------------------------------------------------------------------------=
-----
-#                        =3D=3D=3D Lexing Engine =3D=3D=3D
-#
-# The following Lexer class implements the lexer runtime.   There are only
-# a few public methods and attributes:
-#
-#    input()          -  Store a new string in the lexer
-#    token()          -  Get the next token
-#    clone()          -  Clone the lexer
-#
-#    lineno           -  Current line number
-#    lexpos           -  Current position in the input string
-# ------------------------------------------------------------------------=
-----
-
-class Lexer:
-    def __init__(self):
-        self.lexre =3D None             # Master regular expression. This =
is a list of
-                                      # tuples (re,findex) where re is a c=
ompiled
-                                      # regular expression and findex is a=
 list
-                                      # mapping regex group numbers to rul=
es
-        self.lexretext =3D None         # Current regular expression strin=
gs
-        self.lexstatere =3D {}          # Dictionary mapping lexer states =
to master regexs
-        self.lexstateretext =3D {}      # Dictionary mapping lexer states =
to regex strings
-        self.lexstaterenames =3D {}     # Dictionary mapping lexer states =
to symbol names
-        self.lexstate =3D "INITIAL"     # Current lexer state
-        self.lexstatestack =3D []       # Stack of lexer states
-        self.lexstateinfo =3D None      # State information
-        self.lexstateignore =3D {}      # Dictionary of ignored characters=
 for each state
-        self.lexstateerrorf =3D {}      # Dictionary of error functions fo=
r each state
-        self.lexreflags =3D 0           # Optional re compile flags
-        self.lexdata =3D None           # Actual input data (as a string)
-        self.lexpos =3D 0               # Current position in input text
-        self.lexlen =3D 0               # Length of the input text
-        self.lexerrorf =3D None         # Error rule (if any)
-        self.lextokens =3D None         # List of valid tokens
-        self.lexignore =3D ""           # Ignored characters
-        self.lexliterals =3D ""         # Literal characters that can be p=
assed through
-        self.lexmodule =3D None         # Module
-        self.lineno =3D 1               # Current line number
-        self.lexoptimize =3D 0          # Optimized mode
-
-    def clone(self,object=3DNone):
-        c =3D copy.copy(self)
-
-        # If the object parameter has been supplied, it means we are attac=
hing the
-        # lexer to a new object.  In this case, we have to rebind all meth=
ods in
-        # the lexstatere and lexstateerrorf tables.
-
-        if object:
-            newtab =3D { }
-            for key, ritem in self.lexstatere.items():
-                newre =3D []
-                for cre, findex in ritem:
-                     newfindex =3D []
-                     for f in findex:
-                         if not f or not f[0]:
-                             newfindex.append(f)
-                             continue
-                         newfindex.append((getattr(object,f[0].__name__),f=
[1]))
-                newre.append((cre,newfindex))
-                newtab[key] =3D newre
-            c.lexstatere =3D newtab
-            c.lexstateerrorf =3D { }
-            for key, ef in self.lexstateerrorf.items():
-                c.lexstateerrorf[key] =3D getattr(object,ef.__name__)
-            c.lexmodule =3D object
-        return c
-
-    # ------------------------------------------------------------
-    # writetab() - Write lexer information to a table file
-    # ------------------------------------------------------------
-    def writetab(self,tabfile,outputdir=3D""):
-        if isinstance(tabfile,types.ModuleType):
-            return
-        basetabfilename =3D tabfile.split(".")[-1]
-        filename =3D os.path.join(outputdir,basetabfilename)+".py"
-        tf =3D open(filename,"w")
-        tf.write("# %s.py. This file automatically created by PLY (version=
 %s). Don't edit!\n" % (tabfile,__version__))
-        tf.write("_tabversion   =3D %s\n" % repr(__version__))
-        tf.write("_lextokens    =3D %s\n" % repr(self.lextokens))
-        tf.write("_lexreflags   =3D %s\n" % repr(self.lexreflags))
-        tf.write("_lexliterals  =3D %s\n" % repr(self.lexliterals))
-        tf.write("_lexstateinfo =3D %s\n" % repr(self.lexstateinfo))
-
-        tabre =3D { }
-        # Collect all functions in the initial state
-        initial =3D self.lexstatere["INITIAL"]
-        initialfuncs =3D []
-        for part in initial:
-            for f in part[1]:
-                if f and f[0]:
-                    initialfuncs.append(f)
-
-        for key, lre in self.lexstatere.items():
-             titem =3D []
-             for i in range(len(lre)):
-                  titem.append((self.lexstateretext[key][i],_funcs_to_name=
s(lre[i][1],self.lexstaterenames[key][i])))
-             tabre[key] =3D titem
-
-        tf.write("_lexstatere   =3D %s\n" % repr(tabre))
-        tf.write("_lexstateignore =3D %s\n" % repr(self.lexstateignore))
-
-        taberr =3D { }
-        for key, ef in self.lexstateerrorf.items():
-             if ef:
-                  taberr[key] =3D ef.__name__
-             else:
-                  taberr[key] =3D None
-        tf.write("_lexstateerrorf =3D %s\n" % repr(taberr))
-        tf.close()
-
-    # ------------------------------------------------------------
-    # readtab() - Read lexer information from a tab file
-    # ------------------------------------------------------------
-    def readtab(self,tabfile,fdict):
-        if isinstance(tabfile,types.ModuleType):
-            lextab =3D tabfile
-        else:
-            if sys.version_info[0] < 3:
-                exec("import %s as lextab" % tabfile)
-            else:
-                env =3D { }
-                exec("import %s as lextab" % tabfile, env,env)
-                lextab =3D env['lextab']
-
-        if getattr(lextab,"_tabversion","0.0") !=3D __version__:
-            raise ImportError("Inconsistent PLY version")
-
-        self.lextokens      =3D lextab._lextokens
-        self.lexreflags     =3D lextab._lexreflags
-        self.lexliterals    =3D lextab._lexliterals
-        self.lexstateinfo   =3D lextab._lexstateinfo
-        self.lexstateignore =3D lextab._lexstateignore
-        self.lexstatere     =3D { }
-        self.lexstateretext =3D { }
-        for key,lre in lextab._lexstatere.items():
-             titem =3D []
-             txtitem =3D []
-             for i in range(len(lre)):
-                  titem.append((re.compile(lre[i][0],lextab._lexreflags | =
re.VERBOSE),_names_to_funcs(lre[i][1],fdict)))
-                  txtitem.append(lre[i][0])
-             self.lexstatere[key] =3D titem
-             self.lexstateretext[key] =3D txtitem
-        self.lexstateerrorf =3D { }
-        for key,ef in lextab._lexstateerrorf.items():
-             self.lexstateerrorf[key] =3D fdict[ef]
-        self.begin('INITIAL')
-
-    # ------------------------------------------------------------
-    # input() - Push a new string into the lexer
-    # ------------------------------------------------------------
-    def input(self,s):
-        # Pull off the first character to see if s looks like a string
-        c =3D s[:1]
-        if not isinstance(c,StringTypes):
-            raise ValueError("Expected a string")
-        self.lexdata =3D s
-        self.lexpos =3D 0
-        self.lexlen =3D len(s)
-
-    # ------------------------------------------------------------
-    # begin() - Changes the lexing state
-    # ------------------------------------------------------------
-    def begin(self,state):
-        if not state in self.lexstatere:
-            raise ValueError("Undefined state")
-        self.lexre =3D self.lexstatere[state]
-        self.lexretext =3D self.lexstateretext[state]
-        self.lexignore =3D self.lexstateignore.get(state,"")
-        self.lexerrorf =3D self.lexstateerrorf.get(state,None)
-        self.lexstate =3D state
-
-    # ------------------------------------------------------------
-    # push_state() - Changes the lexing state and saves old on stack
-    # ------------------------------------------------------------
-    def push_state(self,state):
-        self.lexstatestack.append(self.lexstate)
-        self.begin(state)
-
-    # ------------------------------------------------------------
-    # pop_state() - Restores the previous state
-    # ------------------------------------------------------------
-    def pop_state(self):
-        self.begin(self.lexstatestack.pop())
-
-    # ------------------------------------------------------------
-    # current_state() - Returns the current lexing state
-    # ------------------------------------------------------------
-    def current_state(self):
-        return self.lexstate
-
-    # ------------------------------------------------------------
-    # skip() - Skip ahead n characters
-    # ------------------------------------------------------------
-    def skip(self,n):
-        self.lexpos +=3D n
-
-    # ------------------------------------------------------------
-    # opttoken() - Return the next token from the Lexer
-    #
-    # Note: This function has been carefully implemented to be as fast
-    # as possible.  Don't make changes unless you really know what
-    # you are doing
-    # ------------------------------------------------------------
-    def token(self):
-        # Make local copies of frequently referenced attributes
-        lexpos    =3D self.lexpos
-        lexlen    =3D self.lexlen
-        lexignore =3D self.lexignore
-        lexdata   =3D self.lexdata
-
-        while lexpos < lexlen:
-            # This code provides some short-circuit code for whitespace, t=
abs, and other ignored characters
-            if lexdata[lexpos] in lexignore:
-                lexpos +=3D 1
-                continue
-
-            # Look for a regular expression match
-            for lexre,lexindexfunc in self.lexre:
-                m =3D lexre.match(lexdata,lexpos)
-                if not m: continue
-
-                # Create a token for return
-                tok =3D LexToken()
-                tok.value =3D m.group()
-                tok.lineno =3D self.lineno
-                tok.lexpos =3D lexpos
-
-                i =3D m.lastindex
-                func,tok.type =3D lexindexfunc[i]
-
-                if not func:
-                   # If no token type was set, it's an ignored token
-                   if tok.type:
-                      self.lexpos =3D m.end()
-                      return tok
-                   else:
-                      lexpos =3D m.end()
-                      break
-
-                lexpos =3D m.end()
-
-                # If token is processed by a function, call it
-
-                tok.lexer =3D self      # Set additional attributes useful=
 in token rules
-                self.lexmatch =3D m
-                self.lexpos =3D lexpos
-
-                newtok =3D func(tok)
-
-                # Every function must return a token, if nothing, we just =
move to next token
-                if not newtok:
-                    lexpos    =3D self.lexpos         # This is here in ca=
se user has updated lexpos.
-                    lexignore =3D self.lexignore      # This is here in ca=
se there was a state change
-                    break
-
-                # Verify type of the token.  If not in the token map, rais=
e an error
-                if not self.lexoptimize:
-                    if not newtok.type in self.lextokens:
-                        raise LexError("%s:%d: Rule '%s' returned an unkno=
wn token type '%s'" % (
-                            func_code(func).co_filename, func_code(func).c=
o_firstlineno,
-                            func.__name__, newtok.type),lexdata[lexpos:])
-
-                return newtok
-            else:
-                # No match, see if in literals
-                if lexdata[lexpos] in self.lexliterals:
-                    tok =3D LexToken()
-                    tok.value =3D lexdata[lexpos]
-                    tok.lineno =3D self.lineno
-                    tok.type =3D tok.value
-                    tok.lexpos =3D lexpos
-                    self.lexpos =3D lexpos + 1
-                    return tok
-
-                # No match. Call t_error() if defined.
-                if self.lexerrorf:
-                    tok =3D LexToken()
-                    tok.value =3D self.lexdata[lexpos:]
-                    tok.lineno =3D self.lineno
-                    tok.type =3D "error"
-                    tok.lexer =3D self
-                    tok.lexpos =3D lexpos
-                    self.lexpos =3D lexpos
-                    newtok =3D self.lexerrorf(tok)
-                    if lexpos =3D=3D self.lexpos:
-                        # Error method didn't change text position at all.=
 This is an error.
-                        raise LexError("Scanning error. Illegal character =
'%s'" % (lexdata[lexpos]), lexdata[lexpos:])
-                    lexpos =3D self.lexpos
-                    if not newtok: continue
-                    return newtok
-
-                self.lexpos =3D lexpos
-                raise LexError("Illegal character '%s' at index %d" % (lex=
data[lexpos],lexpos), lexdata[lexpos:])
-
-        self.lexpos =3D lexpos + 1
-        if self.lexdata is None:
-             raise RuntimeError("No input string given with input()")
-        return None
-
-    # Iterator interface
-    def __iter__(self):
-        return self
-
-    def next(self):
-        t =3D self.token()
-        if t is None:
-            raise StopIteration
-        return t
-
-    __next__ =3D next
-
-# ------------------------------------------------------------------------=
-----
-#                           =3D=3D=3D=3D Lex Builder =3D=3D=3D
-#
-# The functions and classes below are used to collect lexing information
-# and build a Lexer object from it.
-# ------------------------------------------------------------------------=
-----
-
-# ------------------------------------------------------------------------=
-----
-# get_caller_module_dict()
-#
-# This function returns a dictionary containing all of the symbols defined=
 within
-# a caller further down the call stack.  This is used to get the environme=
nt
-# associated with the yacc() call if none was provided.
-# ------------------------------------------------------------------------=
-----
-
-def get_caller_module_dict(levels):
-    try:
-        raise RuntimeError
-    except RuntimeError:
-        e,b,t =3D sys.exc_info()
-        f =3D t.tb_frame
-        while levels > 0:
-            f =3D f.f_back                  =20
-            levels -=3D 1
-        ldict =3D f.f_globals.copy()
-        if f.f_globals !=3D f.f_locals:
-            ldict.update(f.f_locals)
-
-        return ldict
-
-# ------------------------------------------------------------------------=
-----
-# _funcs_to_names()
-#
-# Given a list of regular expression functions, this converts it to a list
-# suitable for output to a table file
-# ------------------------------------------------------------------------=
-----
-
-def _funcs_to_names(funclist,namelist):
-    result =3D []
-    for f,name in zip(funclist,namelist):
-         if f and f[0]:
-             result.append((name, f[1]))
-         else:
-             result.append(f)
-    return result
-
-# ------------------------------------------------------------------------=
-----
-# _names_to_funcs()
-#
-# Given a list of regular expression function names, this converts it back=
 to
-# functions.
-# ------------------------------------------------------------------------=
-----
-
-def _names_to_funcs(namelist,fdict):
-     result =3D []
-     for n in namelist:
-          if n and n[0]:
-              result.append((fdict[n[0]],n[1]))
-          else:
-              result.append(n)
-     return result
-
-# ------------------------------------------------------------------------=
-----
-# _form_master_re()
-#
-# This function takes a list of all of the regex components and attempts t=
o
-# form the master regular expression.  Given limitations in the Python re
-# module, it may be necessary to break the master regex into separate expr=
essions.
-# ------------------------------------------------------------------------=
-----
-
-def _form_master_re(relist,reflags,ldict,toknames):
-    if not relist: return []
-    regex =3D "|".join(relist)
-    try:
-        lexre =3D re.compile(regex,re.VERBOSE | reflags)
-
-        # Build the index to function map for the matching engine
-        lexindexfunc =3D [ None ] * (max(lexre.groupindex.values())+1)
-        lexindexnames =3D lexindexfunc[:]
-
-        for f,i in lexre.groupindex.items():
-            handle =3D ldict.get(f,None)
-            if type(handle) in (types.FunctionType, types.MethodType):
-                lexindexfunc[i] =3D (handle,toknames[f])
-                lexindexnames[i] =3D f
-            elif handle is not None:
-                lexindexnames[i] =3D f
-                if f.find("ignore_") > 0:
-                    lexindexfunc[i] =3D (None,None)
-                else:
-                    lexindexfunc[i] =3D (None, toknames[f])
-       =20
-        return [(lexre,lexindexfunc)],[regex],[lexindexnames]
-    except Exception:
-        m =3D int(len(relist)/2)
-        if m =3D=3D 0: m =3D 1
-        llist, lre, lnames =3D _form_master_re(relist[:m],reflags,ldict,to=
knames)
-        rlist, rre, rnames =3D _form_master_re(relist[m:],reflags,ldict,to=
knames)
-        return llist+rlist, lre+rre, lnames+rnames
-
-# ------------------------------------------------------------------------=
-----
-# def _statetoken(s,names)
-#
-# Given a declaration name s of the form "t_" and a dictionary whose keys =
are
-# state names, this function returns a tuple (states,tokenname) where stat=
es
-# is a tuple of state names and tokenname is the name of the token.  For e=
xample,
-# calling this with s =3D "t_foo_bar_SPAM" might return (('foo','bar'),'SP=
AM')
-# ------------------------------------------------------------------------=
-----
-
-def _statetoken(s,names):
-    nonstate =3D 1
-    parts =3D s.split("_")
-    for i in range(1,len(parts)):
-         if not parts[i] in names and parts[i] !=3D 'ANY': break
-    if i > 1:
-       states =3D tuple(parts[1:i])
-    else:
-       states =3D ('INITIAL',)
-
-    if 'ANY' in states:
-       states =3D tuple(names)
-
-    tokenname =3D "_".join(parts[i:])
-    return (states,tokenname)
-
-
-# ------------------------------------------------------------------------=
-----
-# LexerReflect()
-#
-# This class represents information needed to build a lexer as extracted f=
rom a
-# user's input file.
-# ------------------------------------------------------------------------=
-----
-class LexerReflect(object):
-    def __init__(self,ldict,log=3DNone,reflags=3D0):
-        self.ldict      =3D ldict
-        self.error_func =3D None
-        self.tokens     =3D []
-        self.reflags    =3D reflags
-        self.stateinfo  =3D { 'INITIAL' : 'inclusive'}
-        self.files      =3D {}
-        self.error      =3D 0
-
-        if log is None:
-            self.log =3D PlyLogger(sys.stderr)
-        else:
-            self.log =3D log
-
-    # Get all of the basic information
-    def get_all(self):
-        self.get_tokens()
-        self.get_literals()
-        self.get_states()
-        self.get_rules()
-       =20
-    # Validate all of the information
-    def validate_all(self):
-        self.validate_tokens()
-        self.validate_literals()
-        self.validate_rules()
-        return self.error
-
-    # Get the tokens map
-    def get_tokens(self):
-        tokens =3D self.ldict.get("tokens",None)
-        if not tokens:
-            self.log.error("No token list is defined")
-            self.error =3D 1
-            return
-
-        if not isinstance(tokens,(list, tuple)):
-            self.log.error("tokens must be a list or tuple")
-            self.error =3D 1
-            return
-       =20
-        if not tokens:
-            self.log.error("tokens is empty")
-            self.error =3D 1
-            return
-
-        self.tokens =3D tokens
-
-    # Validate the tokens
-    def validate_tokens(self):
-        terminals =3D {}
-        for n in self.tokens:
-            if not _is_identifier.match(n):
-                self.log.error("Bad token name '%s'",n)
-                self.error =3D 1
-            if n in terminals:
-                self.log.warning("Token '%s' multiply defined", n)
-            terminals[n] =3D 1
-
-    # Get the literals specifier
-    def get_literals(self):
-        self.literals =3D self.ldict.get("literals","")
-
-    # Validate literals
-    def validate_literals(self):
-        try:
-            for c in self.literals:
-                if not isinstance(c,StringTypes) or len(c) > 1:
-                    self.log.error("Invalid literal %s. Must be a single c=
haracter", repr(c))
-                    self.error =3D 1
-                    continue
-
-        except TypeError:
-            self.log.error("Invalid literals specification. literals must =
be a sequence of characters")
-            self.error =3D 1
-
-    def get_states(self):
-        self.states =3D self.ldict.get("states",None)
-        # Build statemap
-        if self.states:
-             if not isinstance(self.states,(tuple,list)):
-                  self.log.error("states must be defined as a tuple or lis=
t")
-                  self.error =3D 1
-             else:
-                  for s in self.states:
-                        if not isinstance(s,tuple) or len(s) !=3D 2:
-                               self.log.error("Invalid state specifier %s.=
 Must be a tuple (statename,'exclusive|inclusive')",repr(s))
-                               self.error =3D 1
-                               continue
-                        name, statetype =3D s
-                        if not isinstance(name,StringTypes):
-                               self.log.error("State name %s must be a str=
ing", repr(name))
-                               self.error =3D 1
-                               continue
-                        if not (statetype =3D=3D 'inclusive' or statetype =
=3D=3D 'exclusive'):
-                               self.log.error("State type for state %s mus=
t be 'inclusive' or 'exclusive'",name)
-                               self.error =3D 1
-                               continue
-                        if name in self.stateinfo:
-                               self.log.error("State '%s' already defined"=
,name)
-                               self.error =3D 1
-                               continue
-                        self.stateinfo[name] =3D statetype
-
-    # Get all of the symbols with a t_ prefix and sort them into various
-    # categories (functions, strings, error functions, and ignore characte=
rs)
-
-    def get_rules(self):
-        tsymbols =3D [f for f in self.ldict if f[:2] =3D=3D 't_' ]
-
-        # Now build up a list of functions and a list of strings
-
-        self.toknames =3D { }        # Mapping of symbols to token names
-        self.funcsym =3D  { }        # Symbols defined as functions
-        self.strsym =3D   { }        # Symbols defined as strings
-        self.ignore   =3D { }        # Ignore strings by state
-        self.errorf   =3D { }        # Error functions by state
-
-        for s in self.stateinfo:
-             self.funcsym[s] =3D []
-             self.strsym[s] =3D []
-
-        if len(tsymbols) =3D=3D 0:
-            self.log.error("No rules of the form t_rulename are defined")
-            self.error =3D 1
-            return
-
-        for f in tsymbols:
-            t =3D self.ldict[f]
-            states, tokname =3D _statetoken(f,self.stateinfo)
-            self.toknames[f] =3D tokname
-
-            if hasattr(t,"__call__"):
-                if tokname =3D=3D 'error':
-                    for s in states:
-                        self.errorf[s] =3D t
-                elif tokname =3D=3D 'ignore':
-                    line =3D func_code(t).co_firstlineno
-                    file =3D func_code(t).co_filename
-                    self.log.error("%s:%d: Rule '%s' must be defined as a =
string",file,line,t.__name__)
-                    self.error =3D 1
-                else:
-                    for s in states:=20
-                        self.funcsym[s].append((f,t))
-            elif isinstance(t, StringTypes):
-                if tokname =3D=3D 'ignore':
-                    for s in states:
-                        self.ignore[s] =3D t
-                    if "\\" in t:
-                        self.log.warning("%s contains a literal backslash =
'\\'",f)
-
-                elif tokname =3D=3D 'error':
-                    self.log.error("Rule '%s' must be defined as a functio=
n", f)
-                    self.error =3D 1
-                else:
-                    for s in states:=20
-                        self.strsym[s].append((f,t))
-            else:
-                self.log.error("%s not defined as a function or string", f=
)
-                self.error =3D 1
-
-        # Sort the functions by line number
-        for f in self.funcsym.values():
-            f.sort(key=3Dlambda x: func_code(x[1]).co_firstlineno)
-
-        # Sort the strings by regular expression length
-        for s in self.strsym.values():
-            if sys.version_info[0] < 3:
-                s.sort(lambda x,y: (len(x[1]) < len(y[1])) - (len(x[1]) > =
len(y[1])))
-            else:
-                # Python 3.0
-                s.sort(key=3Dlambda x: len(x[1]),reverse=3DTrue)
-
-    # Validate all of the t_rules collected=20
-    def validate_rules(self):
-        for state in self.stateinfo:
-            # Validate all rules defined by functions
-
-           =20
-
-            for fname, f in self.funcsym[state]:
-                line =3D func_code(f).co_firstlineno
-                file =3D func_code(f).co_filename
-                self.files[file] =3D 1
-
-                tokname =3D self.toknames[fname]
-                if isinstance(f, types.MethodType):
-                    reqargs =3D 2
-                else:
-                    reqargs =3D 1
-                nargs =3D func_code(f).co_argcount
-                if nargs > reqargs:
-                    self.log.error("%s:%d: Rule '%s' has too many argument=
s",file,line,f.__name__)
-                    self.error =3D 1
-                    continue
-
-                if nargs < reqargs:
-                    self.log.error("%s:%d: Rule '%s' requires an argument"=
, file,line,f.__name__)
-                    self.error =3D 1
-                    continue
-
-                if not f.__doc__:
-                    self.log.error("%s:%d: No regular expression defined f=
or rule '%s'",file,line,f.__name__)
-                    self.error =3D 1
-                    continue
-
-                try:
-                    c =3D re.compile("(?P<%s>%s)" % (fname,f.__doc__), re.=
VERBOSE | self.reflags)
-                    if c.match(""):
-                        self.log.error("%s:%d: Regular expression for rule=
 '%s' matches empty string", file,line,f.__name__)
-                        self.error =3D 1
-                except re.error:
-                    _etype, e, _etrace =3D sys.exc_info()
-                    self.log.error("%s:%d: Invalid regular expression for =
rule '%s'. %s", file,line,f.__name__,e)
-                    if '#' in f.__doc__:
-                        self.log.error("%s:%d. Make sure '#' in rule '%s' =
is escaped with '\\#'",file,line, f.__name__)
-                    self.error =3D 1
-
-            # Validate all rules defined by strings
-            for name,r in self.strsym[state]:
-                tokname =3D self.toknames[name]
-                if tokname =3D=3D 'error':
-                    self.log.error("Rule '%s' must be defined as a functio=
n", name)
-                    self.error =3D 1
-                    continue
-
-                if not tokname in self.tokens and tokname.find("ignore_") =
< 0:
-                    self.log.error("Rule '%s' defined for an unspecified t=
oken %s",name,tokname)
-                    self.error =3D 1
-                    continue
-
-                try:
-                    c =3D re.compile("(?P<%s>%s)" % (name,r),re.VERBOSE | =
self.reflags)
-                    if (c.match("")):
-                         self.log.error("Regular expression for rule '%s' =
matches empty string",name)
-                         self.error =3D 1
-                except re.error:
-                    _etype, e, _etrace =3D sys.exc_info()
-                    self.log.error("Invalid regular expression for rule '%=
s'. %s",name,e)
-                    if '#' in r:
-                         self.log.error("Make sure '#' in rule '%s' is esc=
aped with '\\#'",name)
-                    self.error =3D 1
-
-            if not self.funcsym[state] and not self.strsym[state]:
-                self.log.error("No rules defined for state '%s'",state)
-                self.error =3D 1
-
-            # Validate the error function
-            efunc =3D self.errorf.get(state,None)
-            if efunc:
-                f =3D efunc
-                line =3D func_code(f).co_firstlineno
-                file =3D func_code(f).co_filename
-                self.files[file] =3D 1
-
-                if isinstance(f, types.MethodType):
-                    reqargs =3D 2
-                else:
-                    reqargs =3D 1
-                nargs =3D func_code(f).co_argcount
-                if nargs > reqargs:
-                    self.log.error("%s:%d: Rule '%s' has too many argument=
s",file,line,f.__name__)
-                    self.error =3D 1
-
-                if nargs < reqargs:
-                    self.log.error("%s:%d: Rule '%s' requires an argument"=
, file,line,f.__name__)
-                    self.error =3D 1
-
-        for f in self.files:
-            self.validate_file(f)
-
-
-    # --------------------------------------------------------------------=
---------
-    # validate_file()
-    #
-    # This checks to see if there are duplicated t_rulename() functions or=
 strings
-    # in the parser input file.  This is done using a simple regular expre=
ssion
-    # match on each line in the given file. =20
-    # --------------------------------------------------------------------=
---------
-
-    def validate_file(self,filename):
-        import os.path
-        base,ext =3D os.path.splitext(filename)
-        if ext !=3D '.py': return         # No idea what the file is. Retu=
rn OK
-
-        try:
-            f =3D open(filename)
-            lines =3D f.readlines()
-            f.close()
-        except IOError:
-            return                      # Couldn't find the file.  Don't w=
orry about it
-
-        fre =3D re.compile(r'\s*def\s+(t_[a-zA-Z_0-9]*)\(')
-        sre =3D re.compile(r'\s*(t_[a-zA-Z_0-9]*)\s*=3D')
-
-        counthash =3D { }
-        linen =3D 1
-        for l in lines:
-            m =3D fre.match(l)
-            if not m:
-                m =3D sre.match(l)
-            if m:
-                name =3D m.group(1)
-                prev =3D counthash.get(name)
-                if not prev:
-                    counthash[name] =3D linen
-                else:
-                    self.log.error("%s:%d: Rule %s redefined. Previously d=
efined on line %d",filename,linen,name,prev)
-                    self.error =3D 1
-            linen +=3D 1
-           =20
-# ------------------------------------------------------------------------=
-----
-# lex(module)
-#
-# Build all of the regular expression rules from definitions in the suppli=
ed module
-# ------------------------------------------------------------------------=
-----
-def lex(module=3DNone,object=3DNone,debug=3D0,optimize=3D0,lextab=3D"lexta=
b",reflags=3D0,nowarn=3D0,outputdir=3D"", debuglog=3DNone, errorlog=3DNone)=
:
-    global lexer
-    ldict =3D None
-    stateinfo  =3D { 'INITIAL' : 'inclusive'}
-    lexobj =3D Lexer()
-    lexobj.lexoptimize =3D optimize
-    global token,input
-
-    if errorlog is None:
-        errorlog =3D PlyLogger(sys.stderr)
-
-    if debug:
-        if debuglog is None:
-            debuglog =3D PlyLogger(sys.stderr)
-
-    # Get the module dictionary used for the lexer
-    if object: module =3D object
-
-    if module:
-        _items =3D [(k,getattr(module,k)) for k in dir(module)]
-        ldict =3D dict(_items)
-    else:
-        ldict =3D get_caller_module_dict(2)
-
-    # Collect parser information from the dictionary
-    linfo =3D LexerReflect(ldict,log=3Derrorlog,reflags=3Dreflags)
-    linfo.get_all()
-    if not optimize:
-        if linfo.validate_all():
-            raise SyntaxError("Can't build lexer")
-
-    if optimize and lextab:
-        try:
-            lexobj.readtab(lextab,ldict)
-            token =3D lexobj.token
-            input =3D lexobj.input
-            lexer =3D lexobj
-            return lexobj
-
-        except ImportError:
-            pass
-
-    # Dump some basic debugging information
-    if debug:
-        debuglog.info("lex: tokens   =3D %r", linfo.tokens)
-        debuglog.info("lex: literals =3D %r", linfo.literals)
-        debuglog.info("lex: states   =3D %r", linfo.stateinfo)
-
-    # Build a dictionary of valid token names
-    lexobj.lextokens =3D { }
-    for n in linfo.tokens:
-        lexobj.lextokens[n] =3D 1
-
-    # Get literals specification
-    if isinstance(linfo.literals,(list,tuple)):
-        lexobj.lexliterals =3D type(linfo.literals[0])().join(linfo.litera=
ls)
-    else:
-        lexobj.lexliterals =3D linfo.literals
-
-    # Get the stateinfo dictionary
-    stateinfo =3D linfo.stateinfo
-
-    regexs =3D { }
-    # Build the master regular expressions
-    for state in stateinfo:
-        regex_list =3D []
-
-        # Add rules defined by functions first
-        for fname, f in linfo.funcsym[state]:
-            line =3D func_code(f).co_firstlineno
-            file =3D func_code(f).co_filename
-            regex_list.append("(?P<%s>%s)" % (fname,f.__doc__))
-            if debug:
-                debuglog.info("lex: Adding rule %s -> '%s' (state '%s')",f=
name,f.__doc__, state)
-
-        # Now add all of the simple rules
-        for name,r in linfo.strsym[state]:
-            regex_list.append("(?P<%s>%s)" % (name,r))
-            if debug:
-                debuglog.info("lex: Adding rule %s -> '%s' (state '%s')",n=
ame,r, state)
-
-        regexs[state] =3D regex_list
-
-    # Build the master regular expressions
-
-    if debug:
-        debuglog.info("lex: =3D=3D=3D=3D MASTER REGEXS FOLLOW =3D=3D=3D=3D=
")
-
-    for state in regexs:
-        lexre, re_text, re_names =3D _form_master_re(regexs[state],reflags=
,ldict,linfo.toknames)
-        lexobj.lexstatere[state] =3D lexre
-        lexobj.lexstateretext[state] =3D re_text
-        lexobj.lexstaterenames[state] =3D re_names
-        if debug:
-            for i in range(len(re_text)):
-                debuglog.info("lex: state '%s' : regex[%d] =3D '%s'",state=
, i, re_text[i])
-
-    # For inclusive states, we need to add the regular expressions from th=
e INITIAL state
-    for state,stype in stateinfo.items():
-        if state !=3D "INITIAL" and stype =3D=3D 'inclusive':
-             lexobj.lexstatere[state].extend(lexobj.lexstatere['INITIAL'])
-             lexobj.lexstateretext[state].extend(lexobj.lexstateretext['IN=
ITIAL'])
-             lexobj.lexstaterenames[state].extend(lexobj.lexstaterenames['=
INITIAL'])
-
-    lexobj.lexstateinfo =3D stateinfo
-    lexobj.lexre =3D lexobj.lexstatere["INITIAL"]
-    lexobj.lexretext =3D lexobj.lexstateretext["INITIAL"]
-    lexobj.lexreflags =3D reflags
-
-    # Set up ignore variables
-    lexobj.lexstateignore =3D linfo.ignore
-    lexobj.lexignore =3D lexobj.lexstateignore.get("INITIAL","")
-
-    # Set up error functions
-    lexobj.lexstateerrorf =3D linfo.errorf
-    lexobj.lexerrorf =3D linfo.errorf.get("INITIAL",None)
-    if not lexobj.lexerrorf:
-        errorlog.warning("No t_error rule is defined")
-
-    # Check state information for ignore and error rules
-    for s,stype in stateinfo.items():
-        if stype =3D=3D 'exclusive':
-              if not s in linfo.errorf:
-                   errorlog.warning("No error rule is defined for exclusiv=
e state '%s'", s)
-              if not s in linfo.ignore and lexobj.lexignore:
-                   errorlog.warning("No ignore rule is defined for exclusi=
ve state '%s'", s)
-        elif stype =3D=3D 'inclusive':
-              if not s in linfo.errorf:
-                   linfo.errorf[s] =3D linfo.errorf.get("INITIAL",None)
-              if not s in linfo.ignore:
-                   linfo.ignore[s] =3D linfo.ignore.get("INITIAL","")
-
-    # Create global versions of the token() and input() functions
-    token =3D lexobj.token
-    input =3D lexobj.input
-    lexer =3D lexobj
-
-    # If in optimize mode, we write the lextab
-    if lextab and optimize:
-        lexobj.writetab(lextab,outputdir)
-
-    return lexobj
-
-# ------------------------------------------------------------------------=
-----
-# runmain()
-#
-# This runs the lexer as a main program
-# ------------------------------------------------------------------------=
-----
-
-def runmain(lexer=3DNone,data=3DNone):
-    if not data:
-        try:
-            filename =3D sys.argv[1]
-            f =3D open(filename)
-            data =3D f.read()
-            f.close()
-        except IndexError:
-            sys.stdout.write("Reading from standard input (type EOF to end=
):\n")
-            data =3D sys.stdin.read()
-
-    if lexer:
-        _input =3D lexer.input
-    else:
-        _input =3D input
-    _input(data)
-    if lexer:
-        _token =3D lexer.token
-    else:
-        _token =3D token
-
-    while 1:
-        tok =3D _token()
-        if not tok: break
-        sys.stdout.write("(%s,%r,%d,%d)\n" % (tok.type, tok.value, tok.lin=
eno,tok.lexpos))
-
-# ------------------------------------------------------------------------=
-----
-# @TOKEN(regex)
-#
-# This decorator function can be used to set the regex expression on a fun=
ction
-# when its docstring might need to be set in an alternative way
-# ------------------------------------------------------------------------=
-----
-
-def TOKEN(r):
-    def set_doc(f):
-        if hasattr(r,"__call__"):
-            f.__doc__ =3D r.__doc__
-        else:
-            f.__doc__ =3D r
-        return f
-    return set_doc
-
-# Alternative spelling of the TOKEN decorator
-Token =3D TOKEN
-
diff --git a/bitbake/lib/prserv/__init__.py b/bitbake/lib/prserv/__init__.p=
y
index 0e0aa34d..c8c8a982 100644
--- a/bitbake/lib/prserv/__init__.py
+++ b/bitbake/lib/prserv/__init__.py
@@ -4,17 +4,92 @@
 # SPDX-License-Identifier: GPL-2.0-only
 #
=20
-__version__ =3D "1.0.0"
=20
-import os, time
-import sys, logging
+__version__ =3D "2.20.0"
=20
-def init_logger(logfile, loglevel):
-    numeric_level =3D getattr(logging, loglevel.upper(), None)
-    if not isinstance(numeric_level, int):
-        raise ValueError("Invalid log level: %s" % loglevel)
-    FORMAT =3D "%(asctime)-15s %(message)s"
-    logging.basicConfig(level=3Dnumeric_level, filename=3Dlogfile, format=
=3DFORMAT)
+import logging
+logger =3D logging.getLogger("BitBake.PRserv")
=20
-class NotFoundError(Exception):
-    pass
+from bb.asyncrpc.client import parse_address, ADDR_TYPE_UNIX, ADDR_TYPE_WS
+
+def create_server(addr, dbpath, upstream=3DNone, read_only=3DFalse):
+    from . import serv
+
+    s =3D serv.PRServer(dbpath, upstream=3Dupstream, read_only=3Dread_only=
)
+    host, port =3D addr.split(":")
+    s.start_tcp_server(host, int(port))
+
+    return s
+
+def increase_revision(ver):
+    """Take a revision string such as "1" or "1.2.3" or even a number and =
increase its last number
+    This fails if the last number is not an integer"""
+
+    fields=3Dstr(ver).split('.')
+    last =3D fields[-1]
+
+    try:
+         val =3D int(last)
+    except Exception as e:
+         logger.critical("Unable to increase revision value %s: %s" % (ver=
, e))
+         raise e
+
+    return ".".join(fields[0:-1] + [ str(val + 1) ])
+
+def _revision_greater_or_equal(rev1, rev2):
+    """Compares x.y.z revision numbers, using integer comparison
+    Returns True if rev1 is greater or equal to rev2"""
+
+    fields1 =3D rev1.split(".")
+    fields2 =3D rev2.split(".")
+    l1 =3D len(fields1)
+    l2 =3D len(fields2)
+
+    for i in range(l1):
+       val1 =3D int(fields1[i])
+       if i < l2:
+           val2 =3D int(fields2[i])
+           if val2 < val1:
+              return True
+           elif val2 > val1:
+              return False
+       else:
+          return True
+    return True
+
+def revision_smaller(rev1, rev2):
+    """Compares x.y.z revision numbers, using integer comparison
+    Returns True if rev1 is strictly smaller than rev2"""
+    return not(_revision_greater_or_equal(rev1, rev2))
+
+def revision_greater(rev1, rev2):
+    """Compares x.y.z revision numbers, using integer comparison
+    Returns True if rev1 is strictly greater than rev2"""
+    return _revision_greater_or_equal(rev1, rev2) and (rev1 !=3D rev2)
+
+def create_client(addr):
+    from . import client
+
+    c =3D client.PRClient()
+
+    try:
+        (typ, a) =3D parse_address(addr)
+        c.connect_tcp(*a)
+        return c
+    except Exception as e:
+        c.close()
+        raise e
+
+async def create_async_client(addr):
+    from . import client
+
+    c =3D client.PRAsyncClient()
+
+    try:
+        (typ, a) =3D parse_address(addr)
+        await c.connect_tcp(*a)
+        return c
+
+    except Exception as e:
+        await c.close()
+        raise e
diff --git a/bitbake/lib/prserv/client.py b/bitbake/lib/prserv/client.py
index 8471ee30..9f5794c4 100644
--- a/bitbake/lib/prserv/client.py
+++ b/bitbake/lib/prserv/client.py
@@ -6,6 +6,7 @@
=20
 import logging
 import bb.asyncrpc
+from . import create_async_client
=20
 logger =3D logging.getLogger("BitBake.PRserv")
=20
@@ -13,16 +14,16 @@ class PRAsyncClient(bb.asyncrpc.AsyncClient):
     def __init__(self):
         super().__init__("PRSERVICE", "1.0", logger)
=20
-    async def getPR(self, version, pkgarch, checksum):
+    async def getPR(self, version, pkgarch, checksum, history=3DFalse):
         response =3D await self.invoke(
-            {"get-pr": {"version": version, "pkgarch": pkgarch, "checksum"=
: checksum}}
+            {"get-pr": {"version": version, "pkgarch": pkgarch, "checksum"=
: checksum, "history": history}}
         )
         if response:
             return response["value"]
=20
-    async def test_pr(self, version, pkgarch, checksum):
+    async def test_pr(self, version, pkgarch, checksum, history=3DFalse):
         response =3D await self.invoke(
-            {"test-pr": {"version": version, "pkgarch": pkgarch, "checksum=
": checksum}}
+            {"test-pr": {"version": version, "pkgarch": pkgarch, "checksum=
": checksum, "history": history}}
         )
         if response:
             return response["value"]
@@ -48,9 +49,9 @@ class PRAsyncClient(bb.asyncrpc.AsyncClient):
         if response:
             return response["value"]
=20
-    async def export(self, version, pkgarch, checksum, colinfo):
+    async def export(self, version, pkgarch, checksum, colinfo, history=3D=
False):
         response =3D await self.invoke(
-            {"export": {"version": version, "pkgarch": pkgarch, "checksum"=
: checksum, "colinfo": colinfo}}
+            {"export": {"version": version, "pkgarch": pkgarch, "checksum"=
: checksum, "colinfo": colinfo, "history": history}}
         )
         if response:
             return (response["metainfo"], response["datainfo"])
@@ -65,7 +66,7 @@ class PRAsyncClient(bb.asyncrpc.AsyncClient):
 class PRClient(bb.asyncrpc.Client):
     def __init__(self):
         super().__init__()
-        self._add_methods("getPR", "test_pr", "test_package", "importone",=
 "export", "is_readonly")
+        self._add_methods("getPR", "test_pr", "test_package", "max_package=
_pr", "importone", "export", "is_readonly")
=20
     def _get_async_client(self):
         return PRAsyncClient()
diff --git a/bitbake/lib/prserv/db.py b/bitbake/lib/prserv/db.py
index eb415081..2da493dd 100644
--- a/bitbake/lib/prserv/db.py
+++ b/bitbake/lib/prserv/db.py
@@ -8,19 +8,13 @@ import logging
 import os.path
 import errno
 import prserv
-import time
+import sqlite3
=20
-try:
-    import sqlite3
-except ImportError:
-    from pysqlite2 import dbapi2 as sqlite3
+from contextlib import closing
+from . import increase_revision, revision_greater, revision_smaller
=20
 logger =3D logging.getLogger("BitBake.PRserv")
=20
-sqlversion =3D sqlite3.sqlite_version_info
-if sqlversion[0] < 3 or (sqlversion[0] =3D=3D 3 and sqlversion[1] < 3):
-    raise Exception("sqlite3 version 3.3.0 or later is required.")
-
 #
 # "No History" mode - for a given query tuple (version, pkgarch, checksum)=
,
 # the returned value will be the largest among all the values of the same
@@ -29,287 +23,232 @@ if sqlversion[0] < 3 or (sqlversion[0] =3D=3D 3 and s=
qlversion[1] < 3):
 # "History" mode - Return a new higher value for previously unseen query
 # tuple (version, pkgarch, checksum), otherwise return historical value.
 # Value can decrement if returning to a previous build.
-#
=20
 class PRTable(object):
-    def __init__(self, conn, table, nohist, read_only):
+    def __init__(self, conn, table, read_only):
         self.conn =3D conn
-        self.nohist =3D nohist
         self.read_only =3D read_only
-        self.dirty =3D False
-        if nohist:
-            self.table =3D "%s_nohist" % table
-        else:
-            self.table =3D "%s_hist" % table
-
-        if self.read_only:
-            table_exists =3D self._execute(
-                        "SELECT count(*) FROM sqlite_master \
-                        WHERE type=3D'table' AND name=3D'%s'" % (self.tabl=
e))
-            if not table_exists:
-                raise prserv.NotFoundError
-        else:
-            self._execute("CREATE TABLE IF NOT EXISTS %s \
+        self.table =3D table
+
+        # Creating the table even if the server is read-only.
+        # This avoids a race condition if a shared database
+        # is accessed by a read-only server first.
+
+        with closing(self.conn.cursor()) as cursor:
+            cursor.execute("CREATE TABLE IF NOT EXISTS %s \
                         (version TEXT NOT NULL, \
                         pkgarch TEXT NOT NULL,  \
                         checksum TEXT NOT NULL, \
-                        value INTEGER, \
-                        PRIMARY KEY (version, pkgarch, checksum));" % self=
.table)
-
-    def _execute(self, *query):
-        """Execute a query, waiting to acquire a lock if necessary"""
-        start =3D time.time()
-        end =3D start + 20
-        while True:
-            try:
-                return self.conn.execute(*query)
-            except sqlite3.OperationalError as exc:
-                if "is locked" in str(exc) and end > time.time():
-                    continue
-                raise exc
-
-    def sync(self):
-        if not self.read_only:
+                        value TEXT, \
+                        PRIMARY KEY (version, pkgarch, checksum, value));"=
 % self.table)
             self.conn.commit()
-            self._execute("BEGIN EXCLUSIVE TRANSACTION")
=20
-    def sync_if_dirty(self):
-        if self.dirty:
-            self.sync()
-            self.dirty =3D False
+    def _extremum_value(self, rows, is_max):
+        value =3D None
+
+        for row in rows:
+            current_value =3D row[0]
+            if value is None:
+                value =3D current_value
+            else:
+                if is_max:
+                    is_new_extremum =3D revision_greater(current_value, va=
lue)
+                else:
+                    is_new_extremum =3D revision_smaller(current_value, va=
lue)
+                if  is_new_extremum:
+                    value =3D current_value
+        return value
+
+    def _max_value(self, rows):
+        return self._extremum_value(rows, True)
+
+    def _min_value(self, rows):
+        return self._extremum_value(rows, False)
=20
     def test_package(self, version, pkgarch):
         """Returns whether the specified package version is found in the d=
atabase for the specified architecture"""
=20
         # Just returns the value if found or None otherwise
-        data=3Dself._execute("SELECT value FROM %s WHERE version=3D? AND p=
kgarch=3D?;" % self.table,
-                           (version, pkgarch))
-        row=3Ddata.fetchone()
-        if row is not None:
-            return True
-        else:
-            return False
+        with closing(self.conn.cursor()) as cursor:
+            data=3Dcursor.execute("SELECT value FROM %s WHERE version=3D? =
AND pkgarch=3D?;" % self.table,
+                               (version, pkgarch))
+            row=3Ddata.fetchone()
+            if row is not None:
+                return True
+            else:
+                return False
+
+    def test_checksum_value(self, version, pkgarch, checksum, value):
+        """Returns whether the specified value is found in the database fo=
r the specified package, architecture and checksum"""
+
+        with closing(self.conn.cursor()) as cursor:
+            data=3Dcursor.execute("SELECT value FROM %s WHERE version=3D? =
AND pkgarch=3D? and checksum=3D? and value=3D?;" % self.table,
+                               (version, pkgarch, checksum, value))
+            row=3Ddata.fetchone()
+            if row is not None:
+                return True
+            else:
+                return False
=20
     def test_value(self, version, pkgarch, value):
         """Returns whether the specified value is found in the database fo=
r the specified package and architecture"""
=20
         # Just returns the value if found or None otherwise
-        data=3Dself._execute("SELECT value FROM %s WHERE version=3D? AND p=
kgarch=3D? and value=3D?;" % self.table,
-                           (version, pkgarch, value))
-        row=3Ddata.fetchone()
-        if row is not None:
-            return True
-        else:
-            return False
+        with closing(self.conn.cursor()) as cursor:
+            data=3Dcursor.execute("SELECT value FROM %s WHERE version=3D? =
AND pkgarch=3D? and value=3D?;" % self.table,
+                               (version, pkgarch, value))
+            row=3Ddata.fetchone()
+            if row is not None:
+                return True
+            else:
+                return False
=20
-    def find_value(self, version, pkgarch, checksum):
+
+    def find_package_max_value(self, version, pkgarch):
+        """Returns the greatest value for (version, pkgarch), or None if n=
ot found. Doesn't create a new value"""
+
+        with closing(self.conn.cursor()) as cursor:
+            data =3D cursor.execute("SELECT value FROM %s where version=3D=
? AND pkgarch=3D?;" % (self.table),
+                                 (version, pkgarch))
+            rows =3D data.fetchall()
+            value =3D self._max_value(rows)
+            return value
+
+    def find_value(self, version, pkgarch, checksum, history=3DFalse):
         """Returns the value for the specified checksum if found or None o=
therwise."""
=20
-        data=3Dself._execute("SELECT value FROM %s WHERE version=3D? AND p=
kgarch=3D? AND checksum=3D?;" % self.table,
-                           (version, pkgarch, checksum))
-        row=3Ddata.fetchone()
-        if row is not None:
-            return row[0]
+        if history:
+            return self.find_min_value(version, pkgarch, checksum)
         else:
-            return None
+            return self.find_max_value(version, pkgarch, checksum)
=20
-    def find_max_value(self, version, pkgarch):
-        """Returns the greatest value for (version, pkgarch), or None if n=
ot found. Doesn't create a new value"""
=20
-        data =3D self._execute("SELECT max(value) FROM %s where version=3D=
? AND pkgarch=3D?;" % (self.table),
-                             (version, pkgarch))
-        row =3D data.fetchone()
-        if row is not None:
-            return row[0]
-        else:
-            return None
-
-    def _get_value_hist(self, version, pkgarch, checksum):
-        data=3Dself._execute("SELECT value FROM %s WHERE version=3D? AND p=
kgarch=3D? AND checksum=3D?;" % self.table,
-                           (version, pkgarch, checksum))
-        row=3Ddata.fetchone()
-        if row is not None:
-            return row[0]
-        else:
-            #no value found, try to insert
-            if self.read_only:
-                data =3D self._execute("SELECT ifnull(max(value)+1, 0) FRO=
M %s where version=3D? AND pkgarch=3D?;" % (self.table),
-                                   (version, pkgarch))
-                row =3D data.fetchone()
-                if row is not None:
-                    return row[0]
-                else:
-                    return 0
+    def _find_extremum_value(self, version, pkgarch, checksum, is_max):
+        """Returns the maximum (if is_max is True) or minimum (if is_max i=
s False) value
+           for (version, pkgarch, checksum), or None if not found. Doesn't=
 create a new value"""
=20
-            try:
-                self._execute("INSERT INTO %s VALUES (?, ?, ?, (select ifn=
ull(max(value)+1, 0) from %s where version=3D? AND pkgarch=3D?));"
-                           % (self.table, self.table),
-                           (version, pkgarch, checksum, version, pkgarch))
-            except sqlite3.IntegrityError as exc:
-                logger.error(str(exc))
+        with closing(self.conn.cursor()) as cursor:
+            data =3D cursor.execute("SELECT value FROM %s where version=3D=
? AND pkgarch=3D? AND checksum=3D?;" % (self.table),
+                                 (version, pkgarch, checksum))
+            rows =3D data.fetchall()
+            return self._extremum_value(rows, is_max)
=20
-            self.dirty =3D True
+    def find_max_value(self, version, pkgarch, checksum):
+        return self._find_extremum_value(version, pkgarch, checksum, True)
=20
-            data=3Dself._execute("SELECT value FROM %s WHERE version=3D? A=
ND pkgarch=3D? AND checksum=3D?;" % self.table,
-                               (version, pkgarch, checksum))
-            row=3Ddata.fetchone()
-            if row is not None:
-                return row[0]
-            else:
-                raise prserv.NotFoundError
-
-    def _get_value_no_hist(self, version, pkgarch, checksum):
-        data=3Dself._execute("SELECT value FROM %s \
-                            WHERE version=3D? AND pkgarch=3D? AND checksum=
=3D? AND \
-                            value >=3D (select max(value) from %s where ve=
rsion=3D? AND pkgarch=3D?);"
-                            % (self.table, self.table),
-                            (version, pkgarch, checksum, version, pkgarch)=
)
-        row=3Ddata.fetchone()
-        if row is not None:
-            return row[0]
-        else:
-            #no value found, try to insert
-            if self.read_only:
-                data =3D self._execute("SELECT ifnull(max(value)+1, 0) FRO=
M %s where version=3D? AND pkgarch=3D?;" % (self.table),
-                                   (version, pkgarch))
-                return data.fetchone()[0]
-
-            try:
-                self._execute("INSERT OR REPLACE INTO %s VALUES (?, ?, ?, =
(select ifnull(max(value)+1, 0) from %s where version=3D? AND pkgarch=3D?))=
;"
-                               % (self.table, self.table),
-                               (version, pkgarch, checksum, version, pkgar=
ch))
-            except sqlite3.IntegrityError as exc:
-                logger.error(str(exc))
-                self.conn.rollback()
-
-            self.dirty =3D True
-
-            data=3Dself._execute("SELECT value FROM %s WHERE version=3D? A=
ND pkgarch=3D? AND checksum=3D?;" % self.table,
-                               (version, pkgarch, checksum))
-            row=3Ddata.fetchone()
-            if row is not None:
-                return row[0]
+    def find_min_value(self, version, pkgarch, checksum):
+        return self._find_extremum_value(version, pkgarch, checksum, False=
)
+
+    def find_new_subvalue(self, version, pkgarch, base):
+        """Take and increase the greatest ".y" value for (version, p=
kgarch), or return ".0" if not found.
+        This doesn't store a new value."""
+
+        with closing(self.conn.cursor()) as cursor:
+            data =3D cursor.execute("SELECT value FROM %s where version=3D=
? AND pkgarch=3D? AND value LIKE '%s.%%';" % (self.table, base),
+                                 (version, pkgarch))
+            rows =3D data.fetchall()
+            value =3D self._max_value(rows)
+
+            if value is not None:
+                return increase_revision(value)
             else:
-                raise prserv.NotFoundError
+                return base + ".0"
=20
-    def get_value(self, version, pkgarch, checksum):
-        if self.nohist:
-            return self._get_value_no_hist(version, pkgarch, checksum)
-        else:
-            return self._get_value_hist(version, pkgarch, checksum)
-
-    def _import_hist(self, version, pkgarch, checksum, value):
-        if self.read_only:
-            return None
-
-        val =3D None
-        data =3D self._execute("SELECT value FROM %s WHERE version=3D? AND=
 pkgarch=3D? AND checksum=3D?;" % self.table,
-                           (version, pkgarch, checksum))
-        row =3D data.fetchone()
-        if row is not None:
-            val=3Drow[0]
-        else:
-            #no value found, try to insert
-            try:
-                self._execute("INSERT INTO %s VALUES (?, ?, ?, ?);"  % (se=
lf.table),
+    def store_value(self, version, pkgarch, checksum, value):
+        """Store value in the database"""
+
+        if not self.read_only and not self.test_checksum_value(version, pk=
garch, checksum, value):
+            with closing(self.conn.cursor()) as cursor:
+                cursor.execute("INSERT INTO %s VALUES (?, ?, ?, ?);"  % (s=
elf.table),
                            (version, pkgarch, checksum, value))
-            except sqlite3.IntegrityError as exc:
-                logger.error(str(exc))
+                self.conn.commit()
=20
-            self.dirty =3D True
+    def _get_value(self, version, pkgarch, checksum, history):
=20
-            data =3D self._execute("SELECT value FROM %s WHERE version=3D?=
 AND pkgarch=3D? AND checksum=3D?;" % self.table,
-                           (version, pkgarch, checksum))
-            row =3D data.fetchone()
-            if row is not None:
-                val =3D row[0]
-        return val
+        max_value =3D self.find_package_max_value(version, pkgarch)
=20
-    def _import_no_hist(self, version, pkgarch, checksum, value):
-        if self.read_only:
-            return None
+        if max_value is None:
+            # version, pkgarch completely unknown. Return initial value.
+            return "0"
=20
-        try:
-            #try to insert
-            self._execute("INSERT INTO %s VALUES (?, ?, ?, ?);"  % (self.t=
able),
-                           (version, pkgarch, checksum, value))
-        except sqlite3.IntegrityError as exc:
-            #already have the record, try to update
-            try:
-                self._execute("UPDATE %s SET value=3D? WHERE version=3D? A=
ND pkgarch=3D? AND checksum=3D? AND value=3D?;" % self.table,
-                            (version, pkgarch, checksum, value))
-        row=3Ddata.fetchone()
-        if row is not None:
-            return row[0]
+        value =3D self.find_value(version, pkgarch, checksum, history)
+
+        if value is None:
+            # version, pkgarch found but not checksum. Create a new value =
from the maximum one
+            return increase_revision(max_value)
+
+        if history:
+            return value
+
+        # "no history" mode - If the value is not the maximum value for th=
e package, need to increase it.
+        if max_value > value:
+            return increase_revision(max_value)
         else:
-            return None
+            return value
+
+    def get_value(self, version, pkgarch, checksum, history):
+        value =3D self._get_value(version, pkgarch, checksum, history)
+        if not self.read_only:
+            self.store_value(version, pkgarch, checksum, value)
+        return value
=20
     def importone(self, version, pkgarch, checksum, value):
-        if self.nohist:
-            return self._import_no_hist(version, pkgarch, checksum, value)
-        else:
-            return self._import_hist(version, pkgarch, checksum, value)
+        self.store_value(version, pkgarch, checksum, value)
+        return value
=20
-    def export(self, version, pkgarch, checksum, colinfo):
+    def export(self, version, pkgarch, checksum, colinfo, history=3DFalse)=
:
         metainfo =3D {}
-        #column info
-        if colinfo:
-            metainfo["tbl_name"] =3D self.table
-            metainfo["core_ver"] =3D prserv.__version__
-            metainfo["col_info"] =3D []
-            data =3D self._execute("PRAGMA table_info(%s);" % self.table)
+        with closing(self.conn.cursor()) as cursor:
+            #column info
+            if colinfo:
+                metainfo["tbl_name"] =3D self.table
+                metainfo["core_ver"] =3D prserv.__version__
+                metainfo["col_info"] =3D []
+                data =3D cursor.execute("PRAGMA table_info(%s);" % self.ta=
ble)
+                for row in data:
+                    col =3D {}
+                    col["name"] =3D row["name"]
+                    col["type"] =3D row["type"]
+                    col["notnull"] =3D row["notnull"]
+                    col["dflt_value"] =3D row["dflt_value"]
+                    col["pk"] =3D row["pk"]
+                    metainfo["col_info"].append(col)
+
+            #data info
+            datainfo =3D []
+
+            if history:
+                sqlstmt =3D "SELECT * FROM %s as T1 WHERE 1=3D1 " % self.t=
able
+            else:
+                sqlstmt =3D "SELECT T1.version, T1.pkgarch, T1.checksum, T=
1.value FROM %s as T1, \
+                        (SELECT version, pkgarch, max(value) as maxvalue F=
ROM %s GROUP BY version, pkgarch) as T2 \
+                        WHERE T1.version=3DT2.version AND T1.pkgarch=3DT2.=
pkgarch AND T1.value=3DT2.maxvalue " % (self.table, self.table)
+            sqlarg =3D []
+            where =3D ""
+            if version:
+                where +=3D "AND T1.version=3D? "
+                sqlarg.append(str(version))
+            if pkgarch:
+                where +=3D "AND T1.pkgarch=3D? "
+                sqlarg.append(str(pkgarch))
+            if checksum:
+                where +=3D "AND T1.checksum=3D? "
+                sqlarg.append(str(checksum))
+
+            sqlstmt +=3D where + ";"
+
+            if len(sqlarg):
+                data =3D cursor.execute(sqlstmt, tuple(sqlarg))
+            else:
+                data =3D cursor.execute(sqlstmt)
             for row in data:
-                col =3D {}
-                col["name"] =3D row["name"]
-                col["type"] =3D row["type"]
-                col["notnull"] =3D row["notnull"]
-                col["dflt_value"] =3D row["dflt_value"]
-                col["pk"] =3D row["pk"]
-                metainfo["col_info"].append(col)
-
-        #data info
-        datainfo =3D []
-
-        if self.nohist:
-            sqlstmt =3D "SELECT T1.version, T1.pkgarch, T1.checksum, T1.va=
lue FROM %s as T1, \
-                    (SELECT version, pkgarch, max(value) as maxvalue FROM =
%s GROUP BY version, pkgarch) as T2 \
-                    WHERE T1.version=3DT2.version AND T1.pkgarch=3DT2.pkga=
rch AND T1.value=3DT2.maxvalue " % (self.table, self.table)
-        else:
-            sqlstmt =3D "SELECT * FROM %s as T1 WHERE 1=3D1 " % self.table
-        sqlarg =3D []
-        where =3D ""
-        if version:
-            where +=3D "AND T1.version=3D? "
-            sqlarg.append(str(version))
-        if pkgarch:
-            where +=3D "AND T1.pkgarch=3D? "
-            sqlarg.append(str(pkgarch))
-        if checksum:
-            where +=3D "AND T1.checksum=3D? "
-            sqlarg.append(str(checksum))
-
-        sqlstmt +=3D where + ";"
-
-        if len(sqlarg):
-            data =3D self._execute(sqlstmt, tuple(sqlarg))
-        else:
-            data =3D self._execute(sqlstmt)
-        for row in data:
-            if row["version"]:
-                col =3D {}
-                col["version"] =3D row["version"]
-                col["pkgarch"] =3D row["pkgarch"]
-                col["checksum"] =3D row["checksum"]
-                col["value"] =3D row["value"]
-                datainfo.append(col)
+                if row["version"]:
+                    col =3D {}
+                    col["version"] =3D row["version"]
+                    col["pkgarch"] =3D row["pkgarch"]
+                    col["checksum"] =3D row["checksum"]
+                    col["value"] =3D row["value"]
+                    datainfo.append(col)
         return (metainfo, datainfo)
=20
     def dump_db(self, fd):
@@ -322,9 +261,8 @@ class PRTable(object):
=20
 class PRData(object):
     """Object representing the PR database"""
-    def __init__(self, filename, nohist=3DTrue, read_only=3DFalse):
+    def __init__(self, filename, read_only=3DFalse):
         self.filename=3Dos.path.abspath(filename)
-        self.nohist=3Dnohist
         self.read_only =3D read_only
         #build directory hierarchy
         try:
@@ -334,14 +272,15 @@ class PRData(object):
                 raise e
         uri =3D "file:%s%s" % (self.filename, "?mode=3Dro" if self.read_on=
ly else "")
         logger.debug("Opening PRServ database '%s'" % (uri))
-        self.connection=3Dsqlite3.connect(uri, uri=3DTrue, isolation_level=
=3D"EXCLUSIVE", check_same_thread =3D False)
+        self.connection=3Dsqlite3.connect(uri, uri=3DTrue)
         self.connection.row_factory=3Dsqlite3.Row
-        if not self.read_only:
-            self.connection.execute("pragma synchronous =3D off;")
-            self.connection.execute("PRAGMA journal_mode =3D MEMORY;")
+        self.connection.execute("PRAGMA synchronous =3D OFF;")
+        self.connection.execute("PRAGMA journal_mode =3D WAL;")
+        self.connection.commit()
         self._tables=3D{}
=20
     def disconnect(self):
+        self.connection.commit()
         self.connection.close()
=20
     def __getitem__(self, tblname):
@@ -351,7 +290,7 @@ class PRData(object):
         if tblname in self._tables:
             return self._tables[tblname]
         else:
-            tableobj =3D self._tables[tblname] =3D PRTable(self.connection=
, tblname, self.nohist, self.read_only)
+            tableobj =3D self._tables[tblname] =3D PRTable(self.connection=
, tblname, self.read_only)
             return tableobj
=20
     def __delitem__(self, tblname):
@@ -359,3 +298,4 @@ class PRData(object):
             del self._tables[tblname]
         logger.info("drop table %s" % (tblname))
         self.connection.execute("DROP TABLE IF EXISTS %s;" % tblname)
+        self.connection.commit()
diff --git a/bitbake/lib/prserv/serv.py b/bitbake/lib/prserv/serv.py
index dc4be5b6..e1758863 100644
--- a/bitbake/lib/prserv/serv.py
+++ b/bitbake/lib/prserv/serv.py
@@ -12,6 +12,7 @@ import sqlite3
 import prserv
 import prserv.db
 import errno
+from . import create_async_client, revision_smaller, increase_revision
 import bb.asyncrpc
=20
 logger =3D logging.getLogger("BitBake.PRserv")
@@ -41,18 +42,16 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection)=
:
         try:
             return await super().dispatch_message(msg)
         except:
-            self.server.table.sync()
             raise
-        else:
-            self.server.table.sync_if_dirty()
=20
     async def handle_test_pr(self, request):
         '''Finds the PR value corresponding to the request. If not found, =
returns None and doesn't insert a new value'''
         version =3D request["version"]
         pkgarch =3D request["pkgarch"]
         checksum =3D request["checksum"]
+        history =3D request["history"]
=20
-        value =3D self.server.table.find_value(version, pkgarch, checksum)
+        value =3D self.server.table.find_value(version, pkgarch, checksum,=
 history)
         return {"value": value}
=20
     async def handle_test_package(self, request):
@@ -68,22 +67,110 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection=
):
         version =3D request["version"]
         pkgarch =3D request["pkgarch"]
=20
-        value =3D self.server.table.find_max_value(version, pkgarch)
+        value =3D self.server.table.find_package_max_value(version, pkgarc=
h)
         return {"value": value}
=20
     async def handle_get_pr(self, request):
         version =3D request["version"]
         pkgarch =3D request["pkgarch"]
         checksum =3D request["checksum"]
+        history =3D request["history"]
=20
-        response =3D None
-        try:
-            value =3D self.server.table.get_value(version, pkgarch, checks=
um)
-            response =3D {"value": value}
-        except prserv.NotFoundError:
-            self.logger.error("failure storing value in database for (%s, =
%s)",version, checksum)
+        if self.upstream_client is None:
+            value =3D self.server.table.get_value(version, pkgarch, checks=
um, history)
+            return {"value": value}
=20
-        return response
+        # We have an upstream server.
+        # Check whether the local server already knows the requested confi=
guration.
+        # If the configuration is a new one, the generated value we will a=
dd will
+        # depend on what's on the upstream server. That's why we're callin=
g find_value()
+        # instead of get_value() directly.
+
+        value =3D self.server.table.find_value(version, pkgarch, checksum,=
 history)
+        upstream_max =3D await self.upstream_client.max_package_pr(version=
, pkgarch)
+
+        if value is not None:
+
+            # The configuration is already known locally.
+
+            if history:
+                value =3D self.server.table.get_value(version, pkgarch, ch=
ecksum, history)
+            else:
+                existing_value =3D value
+                # In "no history", we need to make sure the value doesn't =
decrease
+                # and is at least greater than the maximum upstream value
+                # and the maximum local value
+
+                local_max =3D self.server.table.find_package_max_value(ver=
sion, pkgarch)
+                if revision_smaller(value, local_max):
+                    value =3D increase_revision(local_max)
+
+                if revision_smaller(value, upstream_max):
+                    # Ask upstream whether it knows the checksum
+                    upstream_value =3D await self.upstream_client.test_pr(=
version, pkgarch, checksum)
+                    if upstream_value is None:
+                        # Upstream doesn't have our checksum, let create a=
 new one
+                        value =3D upstream_max + ".0"
+                    else:
+                        # Fine to take the same value as upstream
+                        value =3D upstream_max
+
+                if not value =3D=3D existing_value and not self.server.rea=
d_only:
+                    self.server.table.store_value(version, pkgarch, checks=
um, value)
+
+            return {"value": value}
+
+        # The configuration is a new one for the local server
+        # Let's ask the upstream server whether it knows it
+
+        known_upstream =3D await self.upstream_client.test_package(version=
, pkgarch)
+
+        if not known_upstream:
+
+            # The package is not known upstream, must be a local-only pack=
age
+            # Let's compute the PR number using the local-only method
+
+            value =3D self.server.table.get_value(version, pkgarch, checks=
um, history)
+            return {"value": value}
+
+        # The package is known upstream, let's ask the upstream server
+        # whether it knows our new output hash
+
+        value =3D await self.upstream_client.test_pr(version, pkgarch, che=
cksum)
+
+        if value is not None:
+
+            # Upstream knows this output hash, let's store it and use it t=
oo.
+
+            if not self.server.read_only:
+                self.server.table.store_value(version, pkgarch, checksum, =
value)
+            # If the local server is read only, won't be able to store the=
 new
+            # value in the database and will have to keep asking the upstr=
eam server
+            return {"value": value}
+
+        # The output hash doesn't exist upstream, get the most recent numb=
er from upstream (x)
+        # Then, we want to have a new PR value for the local server: x.y
+
+        upstream_max =3D await self.upstream_client.max_package_pr(version=
, pkgarch)
+        # Here we know that the package is known upstream, so upstream_max=
 can't be None
+        subvalue =3D self.server.table.find_new_subvalue(version, pkgarch,=
 upstream_max)
+
+        if not self.server.read_only:
+            self.server.table.store_value(version, pkgarch, checksum, subv=
alue)
+
+        return {"value": subvalue}
+
+    async def process_requests(self):
+        if self.server.upstream is not None:
+            self.upstream_client =3D await create_async_client(self.server=
.upstream)
+        else:
+            self.upstream_client =3D None
+
+        try:
+            await super().process_requests()
+        finally:
+            if self.upstream_client is not None:
+                await self.upstream_client.close()
=20
     async def handle_import_one(self, request):
         response =3D None
@@ -104,9 +191,10 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnection=
):
         pkgarch =3D request["pkgarch"]
         checksum =3D request["checksum"]
         colinfo =3D request["colinfo"]
+        history =3D request["history"]
=20
         try:
-            (metainfo, datainfo) =3D self.server.table.export(version, pkg=
arch, checksum, colinfo)
+            (metainfo, datainfo) =3D self.server.table.export(version, pkg=
arch, checksum, colinfo, history)
         except sqlite3.Error as exc:
             self.logger.error(str(exc))
             metainfo =3D datainfo =3D None
@@ -117,11 +205,12 @@ class PRServerClient(bb.asyncrpc.AsyncServerConnectio=
n):
         return {"readonly": self.server.read_only}
=20
 class PRServer(bb.asyncrpc.AsyncServer):
-    def __init__(self, dbfile, read_only=3DFalse):
+    def __init__(self, dbfile, read_only=3DFalse, upstream=3DNone):
         super().__init__(logger)
         self.dbfile =3D dbfile
         self.table =3D None
         self.read_only =3D read_only
+        self.upstream =3D upstream
=20
     def accept_client(self, socket):
         return PRServerClient(socket, self)
@@ -134,27 +223,25 @@ class PRServer(bb.asyncrpc.AsyncServer):
         self.logger.info("Started PRServer with DBfile: %s, Address: %s, P=
ID: %s" %
                      (self.dbfile, self.address, str(os.getpid())))
=20
+        if self.upstream is not None:
+            self.logger.info("And upstream PRServer: %s " % (self.upstream=
))
+
         return tasks
=20
     async def stop(self):
-        self.table.sync_if_dirty()
         self.db.disconnect()
         await super().stop()
=20
-    def signal_handler(self):
-        super().signal_handler()
-        if self.table:
-            self.table.sync()
-
 class PRServSingleton(object):
-    def __init__(self, dbfile, logfile, host, port):
+    def __init__(self, dbfile, logfile, host, port, upstream):
         self.dbfile =3D dbfile
         self.logfile =3D logfile
         self.host =3D host
         self.port =3D port
+        self.upstream =3D upstream
=20
     def start(self):
-        self.prserv =3D PRServer(self.dbfile)
+        self.prserv =3D PRServer(self.dbfile, upstream=3Dself.upstream)
         self.prserv.start_tcp_server(socket.gethostbyname(self.host), self=
.port)
         self.process =3D self.prserv.serve_as_process(log_level=3Dlogging.=
WARNING)
=20
@@ -233,7 +320,7 @@ def run_as_daemon(func, pidfile, logfile):
     os.remove(pidfile)
     os._exit(0)
=20
-def start_daemon(dbfile, host, port, logfile, read_only=3DFalse):
+def start_daemon(dbfile, host, port, logfile, read_only=3DFalse, upstream=
=3DNone):
     ip =3D socket.gethostbyname(host)
     pidfile =3D PIDPREFIX % (ip, port)
     try:
@@ -249,7 +336,7 @@ def start_daemon(dbfile, host, port, logfile, read_only=
=3DFalse):
=20
     dbfile =3D os.path.abspath(dbfile)
     def daemon_main():
-        server =3D PRServer(dbfile, read_only=3Dread_only)
+        server =3D PRServer(dbfile, read_only=3Dread_only, upstream=3Dupst=
ream)
         server.start_tcp_server(ip, port)
         server.serve_forever()
=20
@@ -336,6 +423,9 @@ def auto_start(d):
=20
     host =3D host_params[0].strip().lower()
     port =3D int(host_params[1])
+
+    upstream =3D d.getVar("PRSERV_UPSTREAM") or None
+
     if is_local_special(host, port):
         import bb.utils
         cachedir =3D (d.getVar("PERSISTENT_DIR") or d.getVar("CACHE"))
@@ -350,7 +440,7 @@ def auto_start(d):
                auto_shutdown()
         if not singleton:
             bb.utils.mkdirhier(cachedir)
-            singleton =3D PRServSingleton(os.path.abspath(dbfile), os.path=
.abspath(logfile), host, port)
+            singleton =3D PRServSingleton(os.path.abspath(dbfile), os.path=
.abspath(logfile), host, port, upstream)
             singleton.start()
     if singleton:
         host =3D singleton.host
diff --git a/bitbake/lib/prserv/tests.py b/bitbake/lib/prserv/tests.py
new file mode 100644
index 00000000..df0c0030
--- /dev/null
+++ b/bitbake/lib/prserv/tests.py
@@ -0,0 +1,388 @@
+#! /usr/bin/env python3
+#
+# Copyright (C) 2024 BitBake Contributors
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+from . import create_server, create_client, increase_revision, revision_gr=
eater, revision_smaller, _revision_greater_or_equal
+import prserv.db as db
+from bb.asyncrpc import InvokeError
+import logging
+import os
+import sys
+import tempfile
+import unittest
+import socket
+import subprocess
+from pathlib import Path
+
+THIS_DIR =3D Path(__file__).parent
+BIN_DIR =3D THIS_DIR.parent.parent / "bin"
+
+version =3D "dummy-1.0-r0"
+pkgarch =3D "core2-64"
+other_arch =3D "aarch64"
+
+checksumX =3D "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4=
ab4f0"
+checksum0 =3D "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4=
ab4a0"
+checksum1 =3D "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4=
ab4a1"
+checksum2 =3D "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4=
ab4a2"
+checksum3 =3D "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4=
ab4a3"
+checksum4 =3D "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4=
ab4a4"
+checksum5 =3D "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4=
ab4a5"
+checksum6 =3D "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4=
ab4a6"
+checksum7 =3D "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4=
ab4a7"
+checksum8 =3D "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4=
ab4a8"
+checksum9 =3D "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba4=
ab4a9"
+checksum10 =3D "51bf8189dbe9ea81fa6dd89608bf19380c437a9cf12f6c6239887801ba=
4ab4aa"
+
+def server_prefunc(server, name):
+    logging.basicConfig(level=3Dlogging.DEBUG, filename=3D'prserv-%s.log' =
% name, filemode=3D'w',
+                        format=3D'%(levelname)s %(filename)s:%(lineno)d %(=
message)s')
+    server.logger.debug("Running server %s" % name)
+    sys.stdout =3D open('prserv-stdout-%s.log' % name, 'w')
+    sys.stderr =3D sys.stdout
+
+class PRTestSetup(object):
+
+    def start_server(self, name, dbfile, upstream=3DNone, read_only=3DFals=
e, prefunc=3Dserver_prefunc):
+
+        def cleanup_server(server):
+            if server.process.exitcode is not None:
+                return
+            server.process.terminate()
+            server.process.join()
+
+        server =3D create_server(socket.gethostbyname("localhost") + ":0",
+                               dbfile,
+                               upstream=3Dupstream,
+                               read_only=3Dread_only)
+
+        server.serve_as_process(prefunc=3Dprefunc, args=3D(name,))
+        self.addCleanup(cleanup_server, server)
+
+        return server
+
+    def start_client(self, server_address):
+        def cleanup_client(client):
+            client.close()
+
+        client =3D create_client(server_address)
+        self.addCleanup(cleanup_client, client)
+
+        return client
+
+class FunctionTests(unittest.TestCase):
+
+    def setUp(self):
+        self.temp_dir =3D tempfile.TemporaryDirectory(prefix=3D'bb-prserv'=
)
+        self.addCleanup(self.temp_dir.cleanup)
+
+    def test_increase_revision(self):
+        self.assertEqual(increase_revision("1"), "2")
+        self.assertEqual(increase_revision("1.0"), "1.1")
+        self.assertEqual(increase_revision("1.1.1"), "1.1.2")
+        self.assertEqual(increase_revision("1.1.1.3"), "1.1.1.4")
+        self.assertEqual(increase_revision("9"), "10")
+        self.assertEqual(increase_revision("1.9"), "1.10")
+        self.assertRaises(ValueError, increase_revision, "1.a")
+        self.assertRaises(ValueError, increase_revision, "1.")
+        self.assertRaises(ValueError, increase_revision, "")
+
+    def test_revision_greater_or_equal(self):
+        self.assertTrue(_revision_greater_or_equal("2", "2"))
+        self.assertTrue(_revision_greater_or_equal("2", "1"))
+        self.assertTrue(_revision_greater_or_equal("10", "2"))
+        self.assertTrue(_revision_greater_or_equal("1.10", "1.2"))
+        self.assertFalse(_revision_greater_or_equal("1.2", "1.10"))
+        self.assertTrue(_revision_greater_or_equal("1.10", "1"))
+        self.assertTrue(_revision_greater_or_equal("1.10.1", "1.10"))
+        self.assertFalse(_revision_greater_or_equal("1.10.1", "1.10.2"))
+        self.assertTrue(_revision_greater_or_equal("1.10.1", "1.10.1"))
+        self.assertTrue(_revision_greater_or_equal("1.10.1", "1"))
+        self.assertTrue(revision_greater("1.20", "1.3"))
+        self.assertTrue(revision_smaller("1.3", "1.20"))
+
+    # DB tests
+
+    def test_db(self):
+        dbfile =3D os.path.join(self.temp_dir.name, "testtable.sqlite3")
+
+        self.db =3D db.PRData(dbfile)
+        self.table =3D self.db["PRMAIN"]
+
+        self.table.store_value(version, pkgarch, checksum0, "0")
+        self.table.store_value(version, pkgarch, checksum1, "1")
+        # "No history" mode supports multiple PRs for the same checksum
+        self.table.store_value(version, pkgarch, checksum0, "2")
+        self.table.store_value(version, pkgarch, checksum2, "1.0")
+
+        self.assertTrue(self.table.test_package(version, pkgarch))
+        self.assertFalse(self.table.test_package(version, other_arch))
+
+        self.assertTrue(self.table.test_value(version, pkgarch, "0"))
+        self.assertTrue(self.table.test_value(version, pkgarch, "1"))
+        self.assertTrue(self.table.test_value(version, pkgarch, "2"))
+
+        self.assertEqual(self.table.find_package_max_value(version, pkgarc=
h), "2")
+
+        self.assertEqual(self.table.find_min_value(version, pkgarch, check=
sum0), "0")
+        self.assertEqual(self.table.find_max_value(version, pkgarch, check=
sum0), "2")
+
+        # Test history modes
+        self.assertEqual(self.table.find_value(version, pkgarch, checksum0=
, True), "0")
+        self.assertEqual(self.table.find_value(version, pkgarch, checksum0=
, False), "2")
+
+        self.assertEqual(self.table.find_new_subvalue(version, pkgarch, "3=
"), "3.0")
+        self.assertEqual(self.table.find_new_subvalue(version, pkgarch, "1=
"), "1.1")
+
+        # Revision comparison tests
+        self.table.store_value(version, pkgarch, checksum1, "1.3")
+        self.table.store_value(version, pkgarch, checksum1, "1.20")
+        self.assertEqual(self.table.find_min_value(version, pkgarch, check=
sum1), "1")
+        self.assertEqual(self.table.find_max_value(version, pkgarch, check=
sum1), "1.20")
+
+class PRBasicTests(PRTestSetup, unittest.TestCase):
+
+    def setUp(self):
+        self.temp_dir =3D tempfile.TemporaryDirectory(prefix=3D'bb-prserv'=
)
+        self.addCleanup(self.temp_dir.cleanup)
+
+        dbfile =3D os.path.join(self.temp_dir.name, "prtest-basic.sqlite3"=
)
+
+        self.server1 =3D self.start_server("basic", dbfile)
+        self.client1 =3D self.start_client(self.server1.address)
+
+    def test_basic(self):
+
+        # Checks on non existing configuration
+
+        result =3D self.client1.test_pr(version, pkgarch, checksum0)
+        self.assertIsNone(result, "test_pr should return 'None' for a non =
existing PR")
+
+        result =3D self.client1.test_package(version, pkgarch)
+        self.assertFalse(result, "test_package should return 'False' for a=
 non existing PR")
+
+        result =3D self.client1.max_package_pr(version, pkgarch)
+        self.assertIsNone(result, "max_package_pr should return 'None' for=
 a non existing PR")
+
+        # Add a first configuration
+
+        result =3D self.client1.getPR(version, pkgarch, checksum0)
+        self.assertEqual(result, "0", "getPR: initial PR of a package shou=
ld be '0'")
+
+        result =3D self.client1.test_pr(version, pkgarch, checksum0)
+        self.assertEqual(result, "0", "test_pr should return '0' here, mat=
ching the result of getPR")
+
+        result =3D self.client1.test_package(version, pkgarch)
+        self.assertTrue(result, "test_package should return 'True' for an =
existing PR")
+
+        result =3D self.client1.max_package_pr(version, pkgarch)
+        self.assertEqual(result, "0", "max_package_pr should return '0' in=
 the current test series")
+
+        # Check that the same request gets the same value
+
+        result =3D self.client1.getPR(version, pkgarch, checksum0)
+        self.assertEqual(result, "0", "getPR: asking for the same PR a sec=
ond time in a row should return the same value.")
+
+        # Add new configurations
+
+        result =3D self.client1.getPR(version, pkgarch, checksum1)
+        self.assertEqual(result, "1", "getPR: second PR of a package shoul=
d be '1'")
+
+        result =3D self.client1.test_pr(version, pkgarch, checksum1)
+        self.assertEqual(result, "1", "test_pr should return '1' here, mat=
ching the result of getPR")
+
+        result =3D self.client1.max_package_pr(version, pkgarch)
+        self.assertEqual(result, "1", "max_package_pr should return '1' in=
 the current test series")
+
+        result =3D self.client1.getPR(version, pkgarch, checksum2)
+        self.assertEqual(result, "2", "getPR: second PR of a package shoul=
d be '2'")
+
+        result =3D self.client1.test_pr(version, pkgarch, checksum2)
+        self.assertEqual(result, "2", "test_pr should return '2' here, mat=
ching the result of getPR")
+
+        result =3D self.client1.max_package_pr(version, pkgarch)
+        self.assertEqual(result, "2", "max_package_pr should return '2' in=
 the current test series")
+
+        result =3D self.client1.getPR(version, pkgarch, checksum3)
+        self.assertEqual(result, "3", "getPR: second PR of a package shoul=
d be '3'")
+
+        result =3D self.client1.test_pr(version, pkgarch, checksum3)
+        self.assertEqual(result, "3", "test_pr should return '3' here, mat=
ching the result of getPR")
+
+        result =3D self.client1.max_package_pr(version, pkgarch)
+        self.assertEqual(result, "3", "max_package_pr should return '3' in=
 the current test series")
+
+        # Ask again for the first configuration
+
+        result =3D self.client1.getPR(version, pkgarch, checksum0)
+        self.assertEqual(result, "4", "getPR: should return '4' in this co=
nfiguration")
+
+        # Ask again with explicit "no history" mode
+
+        result =3D self.client1.getPR(version, pkgarch, checksum0, False)
+        self.assertEqual(result, "4", "getPR: should return '4' in this co=
nfiguration")
+
+        # Ask again with explicit "history" mode. This should return the f=
irst recorded PR for checksum0
+
+        result =3D self.client1.getPR(version, pkgarch, checksum0, True)
+        self.assertEqual(result, "0", "getPR: should return '0' in this co=
nfiguration")
+
+        # Check again that another pkgarg resets the counters
+
+        result =3D self.client1.test_pr(version, other_arch, checksum0)
+        self.assertIsNone(result, "test_pr should return 'None' for a non =
existing PR")
+
+        result =3D self.client1.test_package(version, other_arch)
+        self.assertFalse(result, "test_package should return 'False' for a=
 non existing PR")
+
+        result =3D self.client1.max_package_pr(version, other_arch)
+        self.assertIsNone(result, "max_package_pr should return 'None' for=
 a non existing PR")
+
+        # Now add the configuration
+
+        result =3D self.client1.getPR(version, other_arch, checksum0)
+        self.assertEqual(result, "0", "getPR: initial PR of a package shou=
ld be '0'")
+
+        result =3D self.client1.test_pr(version, other_arch, checksum0)
+        self.assertEqual(result, "0", "test_pr should return '0' here, mat=
ching the result of getPR")
+
+        result =3D self.client1.test_package(version, other_arch)
+        self.assertTrue(result, "test_package should return 'True' for an =
existing PR")
+
+        result =3D self.client1.max_package_pr(version, other_arch)
+        self.assertEqual(result, "0", "max_package_pr should return '0' in=
 the current test series")
+
+        result =3D self.client1.is_readonly()
+        self.assertFalse(result, "Server should not be described as 'read-=
only'")
+
+class PRUpstreamTests(PRTestSetup, unittest.TestCase):
+
+    def setUp(self):
+
+        self.temp_dir =3D tempfile.TemporaryDirectory(prefix=3D'bb-prserv'=
)
+        self.addCleanup(self.temp_dir.cleanup)
+
+        dbfile2 =3D os.path.join(self.temp_dir.name, "prtest-upstream2.sql=
ite3")
+        self.server2 =3D self.start_server("upstream2", dbfile2)
+        self.client2 =3D self.start_client(self.server2.address)
+
+        dbfile1 =3D os.path.join(self.temp_dir.name, "prtest-upstream1.sql=
ite3")
+        self.server1 =3D self.start_server("upstream1", dbfile1, upstream=
=3Dself.server2.address)
+        self.client1 =3D self.start_client(self.server1.address)
+
+        dbfile0 =3D os.path.join(self.temp_dir.name, "prtest-local.sqlite3=
")
+        self.server0 =3D self.start_server("local", dbfile0, upstream=3Dse=
lf.server1.address)
+        self.client0 =3D self.start_client(self.server0.address)
+        self.shared_db =3D dbfile0
+
+    def test_upstream_and_readonly(self):
+
+        # For identical checksums, all servers should return the same PR
+
+        result =3D self.client2.getPR(version, pkgarch, checksum0)
+        self.assertEqual(result, "0", "getPR: initial PR of a package shou=
ld be '0'")
+
+        result =3D self.client1.getPR(version, pkgarch, checksum0)
+        self.assertEqual(result, "0", "getPR: initial PR of a package shou=
ld be '0' (same as upstream)")
+
+        result =3D self.client0.getPR(version, pkgarch, checksum0)
+        self.assertEqual(result, "0", "getPR: initial PR of a package shou=
ld be '0' (same as upstream)")
+
+        # Now introduce new checksums on server1 for, same version
+
+        result =3D self.client1.getPR(version, pkgarch, checksum1)
+        self.assertEqual(result, "0.0", "getPR: first PR of a package whic=
h has a different checksum upstream should be '0.0'")
+
+        result =3D self.client1.getPR(version, pkgarch, checksum2)
+        self.assertEqual(result, "0.1", "getPR: second PR of a package tha=
t has a different checksum upstream should be '0.1'")
+
+        # Now introduce checksums on server0 for, same version
+
+        result =3D self.client1.getPR(version, pkgarch, checksum1)
+        self.assertEqual(result, "0.2", "getPR: can't decrease for known P=
R")
+
+        result =3D self.client1.getPR(version, pkgarch, checksum2)
+        self.assertEqual(result, "0.3")
+
+        result =3D self.client1.max_package_pr(version, pkgarch)
+        self.assertEqual(result, "0.3")
+
+        result =3D self.client0.getPR(version, pkgarch, checksum3)
+        self.assertEqual(result, "0.3.0", "getPR: first PR of a package th=
at doesn't exist upstream should be '0.3.0'")
+
+        result =3D self.client0.getPR(version, pkgarch, checksum4)
+        self.assertEqual(result, "0.3.1", "getPR: second PR of a package t=
hat doesn't exist upstream should be '0.3.1'")
+
+        result =3D self.client0.getPR(version, pkgarch, checksum3)
+        self.assertEqual(result, "0.3.2")
+
+        # More upstream updates
+        # Here, we assume no communication between server2 and server0. se=
rver2 only impacts server0
+        # after impacting server1
+
+        self.assertEqual(self.client2.getPR(version, pkgarch, checksum5), =
"1")
+        self.assertEqual(self.client1.getPR(version, pkgarch, checksum6), =
"1.0")
+        self.assertEqual(self.client1.getPR(version, pkgarch, checksum7), =
"1.1")
+        self.assertEqual(self.client0.getPR(version, pkgarch, checksum8), =
"1.1.0")
+        self.assertEqual(self.client0.getPR(version, pkgarch, checksum9), =
"1.1.1")
+
+        # "history" mode tests
+
+        self.assertEqual(self.client2.getPR(version, pkgarch, checksum0, T=
rue), "0")
+        self.assertEqual(self.client1.getPR(version, pkgarch, checksum2, T=
rue), "0.1")
+        self.assertEqual(self.client0.getPR(version, pkgarch, checksum3, T=
rue), "0.3.0")
+
+        # More "no history" mode tests
+
+        self.assertEqual(self.client2.getPR(version, pkgarch, checksum0), =
"2")
+        self.assertEqual(self.client1.getPR(version, pkgarch, checksum0), =
"2") # Same as upstream
+        self.assertEqual(self.client0.getPR(version, pkgarch, checksum0), =
"2") # Same as upstream
+        self.assertEqual(self.client1.getPR(version, pkgarch, checksum7), =
"3") # This could be surprising, but since the previous revision was "2", i=
ncreasing it yields "3".
+                                                                          =
     # We don't know how many upstream servers we have
+        # Start read-only server with server1 as upstream
+        self.server_ro =3D self.start_server("local-ro", self.shared_db, u=
pstream=3Dself.server1.address, read_only=3DTrue)
+        self.client_ro =3D self.start_client(self.server_ro.address)
+
+        self.assertTrue(self.client_ro.is_readonly(), "Database should be =
described as 'read-only'")
+
+        # Checks on non existing configurations
+        self.assertIsNone(self.client_ro.test_pr(version, pkgarch, checksu=
mX))
+        self.assertFalse(self.client_ro.test_package("unknown", pkgarch))
+
+        # Look up existing configurations
+        self.assertEqual(self.client_ro.getPR(version, pkgarch, checksum0)=
, "3") # "no history" mode
+        self.assertEqual(self.client_ro.getPR(version, pkgarch, checksum0,=
 True), "0") # "history" mode
+        self.assertEqual(self.client_ro.getPR(version, pkgarch, checksum3)=
, "3")
+        self.assertEqual(self.client_ro.getPR(version, pkgarch, checksum3,=
 True), "0.3.0")
+        self.assertEqual(self.client_ro.max_package_pr(version, pkgarch), =
"2") # normal as "3" was never saved
+
+        # Try to insert a new value. Here this one is know upstream.
+        self.assertEqual(self.client_ro.getPR(version, pkgarch, checksum7)=
, "3")
+        # Try to insert a completely new value. As the max upstream value =
is already "3", it should be "3.0"
+        self.assertEqual(self.client_ro.getPR(version, pkgarch, checksum10=
), "3.0")
+        # Same with another value which only exists in the upstream upstre=
am server
+        # This time, as the upstream server doesn't know it, it will ask i=
ts upstream server. So that's a known one.
+        self.assertEqual(self.client_ro.getPR(version, pkgarch, checksum9)=
, "3")
+
+class ScriptTests(unittest.TestCase):
+
+    def setUp(self):
+
+        self.temp_dir =3D tempfile.TemporaryDirectory(prefix=3D'bb-prserv'=
)
+        self.addCleanup(self.temp_dir.cleanup)
+        self.dbfile =3D os.path.join(self.temp_dir.name, "prtest.sqlite3")
+
+    def test_1_start_bitbake_prserv(self):
+        try:
+            subprocess.check_call([BIN_DIR / "bitbake-prserv", "--start", =
"-f", self.dbfile])
+        except subprocess.CalledProcessError as e:
+            self.fail("Failed to start bitbake-prserv: %s" % e.returncode)
+
+    def test_2_stop_bitbake_prserv(self):
+        try:
+            subprocess.check_call([BIN_DIR / "bitbake-prserv", "--stop"])
+        except subprocess.CalledProcessError as e:
+            self.fail("Failed to stop bitbake-prserv: %s" % e.returncode)
diff --git a/bitbake/lib/pyinotify.py b/bitbake/lib/pyinotify.py
deleted file mode 100644
index 3c5dab03..00000000
--- a/bitbake/lib/pyinotify.py
+++ /dev/null
@@ -1,2326 +0,0 @@
-#
-# pyinotify.py - python interface to inotify
-# Copyright (c) 2005-2015 Sebastien Martini 
-#
-# SPDX-License-Identifier: MIT
-#
-"""
-pyinotify
-
-@author: Sebastien Martini
-@license: MIT License
-@contact: seb@dbzteam.org
-"""
-
-class PyinotifyError(Exception):
-    """Indicates exceptions raised by a Pyinotify class."""
-    pass
-
-
-class UnsupportedPythonVersionError(PyinotifyError):
-    """
-    Raised on unsupported Python versions.
-    """
-    def __init__(self, version):
-        """
-        @param version: Current Python version
-        @type version: string
-        """
-        PyinotifyError.__init__(self,
-                                ('Python %s is unsupported, requires '
-                                 'at least Python 3.0') % version)
-
-
-# Check Python version
-import sys
-if sys.version_info < (3, 0):
-    raise UnsupportedPythonVersionError(sys.version)
-
-
-# Import directives
-import threading
-import os
-import select
-import struct
-import fcntl
-import errno
-import termios
-import array
-import logging
-import atexit
-from collections import deque
-from datetime import datetime, timedelta
-import time
-import re
-import glob
-import locale
-import subprocess
-
-try:
-    from functools import reduce
-except ImportError:
-    pass  # Will fail on Python 2.4 which has reduce() builtin anyway.
-
-try:
-    import ctypes
-    import ctypes.util
-except ImportError:
-    ctypes =3D None
-
-try:
-    import inotify_syscalls
-except ImportError:
-    inotify_syscalls =3D None
-
-
-__author__ =3D "seb@dbzteam.org (Sebastien Martini)"
-
-__version__ =3D "0.9.6"
-
-
-# Compatibity mode: set to True to improve compatibility with
-# Pyinotify 0.7.1. Do not set this variable yourself, call the
-# function compatibility_mode() instead.
-COMPATIBILITY_MODE =3D False
-
-
-class InotifyBindingNotFoundError(PyinotifyError):
-    """
-    Raised when no inotify support couldn't be found.
-    """
-    def __init__(self):
-        err =3D "Couldn't find any inotify binding"
-        PyinotifyError.__init__(self, err)
-
-
-class INotifyWrapper:
-    """
-    Abstract class wrapping access to inotify's functions. This is an
-    internal class.
-    """
-    @staticmethod
-    def create():
-        """
-        Factory method instanciating and returning the right wrapper.
-        """
-        # First, try to use ctypes.
-        if ctypes:
-            inotify =3D _CtypesLibcINotifyWrapper()
-            if inotify.init():
-                return inotify
-        # Second, see if C extension is compiled.
-        if inotify_syscalls:
-            inotify =3D _INotifySyscallsWrapper()
-            if inotify.init():
-                return inotify
-
-    def get_errno(self):
-        """
-        Return None is no errno code is available.
-        """
-        return self._get_errno()
-
-    def str_errno(self):
-        code =3D self.get_errno()
-        if code is None:
-            return 'Errno: no errno support'
-        return 'Errno=3D%s (%s)' % (os.strerror(code), errno.errorcode[cod=
e])
-
-    def inotify_init(self):
-        return self._inotify_init()
-
-    def inotify_add_watch(self, fd, pathname, mask):
-        # Unicode strings must be encoded to string prior to calling this
-        # method.
-        assert isinstance(pathname, str)
-        return self._inotify_add_watch(fd, pathname, mask)
-
-    def inotify_rm_watch(self, fd, wd):
-        return self._inotify_rm_watch(fd, wd)
-
-
-class _INotifySyscallsWrapper(INotifyWrapper):
-    def __init__(self):
-        # Stores the last errno value.
-        self._last_errno =3D None
-
-    def init(self):
-        assert inotify_syscalls
-        return True
-
-    def _get_errno(self):
-        return self._last_errno
-
-    def _inotify_init(self):
-        try:
-            fd =3D inotify_syscalls.inotify_init()
-        except IOError as err:
-            self._last_errno =3D err.errno
-            return -1
-        return fd
-
-    def _inotify_add_watch(self, fd, pathname, mask):
-        try:
-            wd =3D inotify_syscalls.inotify_add_watch(fd, pathname, mask)
-        except IOError as err:
-            self._last_errno =3D err.errno
-            return -1
-        return wd
-
-    def _inotify_rm_watch(self, fd, wd):
-        try:
-            ret =3D inotify_syscalls.inotify_rm_watch(fd, wd)
-        except IOError as err:
-            self._last_errno =3D err.errno
-            return -1
-        return ret
-
-
-class _CtypesLibcINotifyWrapper(INotifyWrapper):
-    def __init__(self):
-        self._libc =3D None
-        self._get_errno_func =3D None
-
-    def init(self):
-        assert ctypes
-
-        try_libc_name =3D 'c'
-        if sys.platform.startswith('freebsd'):
-            try_libc_name =3D 'inotify'
-
-        libc_name =3D None
-        try:
-            libc_name =3D ctypes.util.find_library(try_libc_name)
-        except (OSError, IOError):
-            pass  # Will attemp to load it with None anyway.
-
-        self._libc =3D ctypes.CDLL(libc_name, use_errno=3DTrue)
-        self._get_errno_func =3D ctypes.get_errno
-
-        # Eventually check that libc has needed inotify bindings.
-        if (not hasattr(self._libc, 'inotify_init') or
-            not hasattr(self._libc, 'inotify_add_watch') or
-            not hasattr(self._libc, 'inotify_rm_watch')):
-            return False
-
-        self._libc.inotify_init.argtypes =3D []
-        self._libc.inotify_init.restype =3D ctypes.c_int
-        self._libc.inotify_add_watch.argtypes =3D [ctypes.c_int, ctypes.c_=
char_p,
-                                                 ctypes.c_uint32]
-        self._libc.inotify_add_watch.restype =3D ctypes.c_int
-        self._libc.inotify_rm_watch.argtypes =3D [ctypes.c_int, ctypes.c_i=
nt]
-        self._libc.inotify_rm_watch.restype =3D ctypes.c_int
-        return True
-
-    def _get_errno(self):
-        assert self._get_errno_func
-        return self._get_errno_func()
-
-    def _inotify_init(self):
-        assert self._libc is not None
-        return self._libc.inotify_init()
-
-    def _inotify_add_watch(self, fd, pathname, mask):
-        assert self._libc is not None
-        # Encodes path to a bytes string. This conversion seems required b=
ecause
-        # ctypes.create_string_buffer seems to manipulate bytes internally=
.
-        # Moreover it seems that inotify_add_watch does not work very well=
 when
-        # it receives an ctypes.create_unicode_buffer instance as argument=
.
-        pathname =3D pathname.encode(sys.getfilesystemencoding())
-        pathname =3D ctypes.create_string_buffer(pathname)
-        return self._libc.inotify_add_watch(fd, pathname, mask)
-
-    def _inotify_rm_watch(self, fd, wd):
-        assert self._libc is not None
-        return self._libc.inotify_rm_watch(fd, wd)
-
-
-# Logging
-def logger_init():
-    """Initialize logger instance."""
-    log =3D logging.getLogger("pyinotify")
-    console_handler =3D logging.StreamHandler()
-    console_handler.setFormatter(
-        logging.Formatter("[%(asctime)s %(name)s %(levelname)s] %(message)=
s"))
-    log.addHandler(console_handler)
-    log.setLevel(20)
-    return log
-
-log =3D logger_init()
-
-
-# inotify's variables
-class ProcINotify:
-    """
-    Access (read, write) inotify's variables through /proc/sys/. Note that
-    usually it requires administrator rights to update them.
-
-    Examples:
-      - Read max_queued_events attribute: myvar =3D max_queued_events.valu=
e
-      - Update max_queued_events attribute: max_queued_events.value =3D 42
-    """
-    def __init__(self, attr):
-        self._base =3D "/proc/sys/fs/inotify"
-        self._attr =3D attr
-
-    def get_val(self):
-        """
-        Gets attribute's value.
-
-        @return: stored value.
-        @rtype: int
-        @raise IOError: if corresponding file in /proc/sys cannot be read.
-        """
-        with open(os.path.join(self._base, self._attr), 'r') as file_obj:
-            return int(file_obj.readline())
-
-    def set_val(self, nval):
-        """
-        Sets new attribute's value.
-
-        @param nval: replaces current value by nval.
-        @type nval: int
-        @raise IOError: if corresponding file in /proc/sys cannot be writt=
en.
-        """
-        with open(os.path.join(self._base, self._attr), 'w') as file_obj:
-            file_obj.write(str(nval) + '\n')
-
-    value =3D property(get_val, set_val)
-
-    def __repr__(self):
-        return '<%s=3D%d>' % (self._attr, self.get_val())
-
-
-# Inotify's variables
-#
-# Note: may raise IOError if the corresponding value in /proc/sys
-#       cannot be accessed.
-#
-# Examples:
-#  - read: myvar =3D max_queued_events.value
-#  - update: max_queued_events.value =3D 42
-#
-for attrname in ('max_queued_events', 'max_user_instances', 'max_user_watc=
hes'):
-    globals()[attrname] =3D ProcINotify(attrname)
-
-
-class EventsCodes:
-    """
-    Set of codes corresponding to each kind of events.
-    Some of these flags are used to communicate with inotify, whereas
-    the others are sent to userspace by inotify notifying some events.
-
-    @cvar IN_ACCESS: File was accessed.
-    @type IN_ACCESS: int
-    @cvar IN_MODIFY: File was modified.
-    @type IN_MODIFY: int
-    @cvar IN_ATTRIB: Metadata changed.
-    @type IN_ATTRIB: int
-    @cvar IN_CLOSE_WRITE: Writtable file was closed.
-    @type IN_CLOSE_WRITE: int
-    @cvar IN_CLOSE_NOWRITE: Unwrittable file closed.
-    @type IN_CLOSE_NOWRITE: int
-    @cvar IN_OPEN: File was opened.
-    @type IN_OPEN: int
-    @cvar IN_MOVED_FROM: File was moved from X.
-    @type IN_MOVED_FROM: int
-    @cvar IN_MOVED_TO: File was moved to Y.
-    @type IN_MOVED_TO: int
-    @cvar IN_CREATE: Subfile was created.
-    @type IN_CREATE: int
-    @cvar IN_DELETE: Subfile was deleted.
-    @type IN_DELETE: int
-    @cvar IN_DELETE_SELF: Self (watched item itself) was deleted.
-    @type IN_DELETE_SELF: int
-    @cvar IN_MOVE_SELF: Self (watched item itself) was moved.
-    @type IN_MOVE_SELF: int
-    @cvar IN_UNMOUNT: Backing fs was unmounted.
-    @type IN_UNMOUNT: int
-    @cvar IN_Q_OVERFLOW: Event queued overflowed.
-    @type IN_Q_OVERFLOW: int
-    @cvar IN_IGNORED: File was ignored.
-    @type IN_IGNORED: int
-    @cvar IN_ONLYDIR: only watch the path if it is a directory (new
-                      in kernel 2.6.15).
-    @type IN_ONLYDIR: int
-    @cvar IN_DONT_FOLLOW: don't follow a symlink (new in kernel 2.6.15).
-                          IN_ONLYDIR we can make sure that we don't watch
-                          the target of symlinks.
-    @type IN_DONT_FOLLOW: int
-    @cvar IN_EXCL_UNLINK: Events are not generated for children after they
-                          have been unlinked from the watched directory.
-                          (new in kernel 2.6.36).
-    @type IN_EXCL_UNLINK: int
-    @cvar IN_MASK_ADD: add to the mask of an already existing watch (new
-                       in kernel 2.6.14).
-    @type IN_MASK_ADD: int
-    @cvar IN_ISDIR: Event occurred against dir.
-    @type IN_ISDIR: int
-    @cvar IN_ONESHOT: Only send event once.
-    @type IN_ONESHOT: int
-    @cvar ALL_EVENTS: Alias for considering all of the events.
-    @type ALL_EVENTS: int
-    """
-
-    # The idea here is 'configuration-as-code' - this way, we get our nice=
 class
-    # constants, but we also get nice human-friendly text mappings to do l=
ookups
-    # against as well, for free:
-    FLAG_COLLECTIONS =3D {'OP_FLAGS': {
-        'IN_ACCESS'        : 0x00000001,  # File was accessed
-        'IN_MODIFY'        : 0x00000002,  # File was modified
-        'IN_ATTRIB'        : 0x00000004,  # Metadata changed
-        'IN_CLOSE_WRITE'   : 0x00000008,  # Writable file was closed
-        'IN_CLOSE_NOWRITE' : 0x00000010,  # Unwritable file closed
-        'IN_OPEN'          : 0x00000020,  # File was opened
-        'IN_MOVED_FROM'    : 0x00000040,  # File was moved from X
-        'IN_MOVED_TO'      : 0x00000080,  # File was moved to Y
-        'IN_CREATE'        : 0x00000100,  # Subfile was created
-        'IN_DELETE'        : 0x00000200,  # Subfile was deleted
-        'IN_DELETE_SELF'   : 0x00000400,  # Self (watched item itself)
-                                          # was deleted
-        'IN_MOVE_SELF'     : 0x00000800,  # Self (watched item itself) was=
 moved
-        },
-                        'EVENT_FLAGS': {
-        'IN_UNMOUNT'       : 0x00002000,  # Backing fs was unmounted
-        'IN_Q_OVERFLOW'    : 0x00004000,  # Event queued overflowed
-        'IN_IGNORED'       : 0x00008000,  # File was ignored
-        },
-                        'SPECIAL_FLAGS': {
-        'IN_ONLYDIR'       : 0x01000000,  # only watch the path if it is a
-                                          # directory
-        'IN_DONT_FOLLOW'   : 0x02000000,  # don't follow a symlink
-        'IN_EXCL_UNLINK'   : 0x04000000,  # exclude events on unlinked obj=
ects
-        'IN_MASK_ADD'      : 0x20000000,  # add to the mask of an already
-                                          # existing watch
-        'IN_ISDIR'         : 0x40000000,  # event occurred against dir
-        'IN_ONESHOT'       : 0x80000000,  # only send event once
-        },
-                        }
-
-    def maskname(mask):
-        """
-        Returns the event name associated to mask. IN_ISDIR is appended to
-        the result when appropriate. Note: only one event is returned, bec=
ause
-        only one event can be raised at a given time.
-
-        @param mask: mask.
-        @type mask: int
-        @return: event name.
-        @rtype: str
-        """
-        ms =3D mask
-        name =3D '%s'
-        if mask & IN_ISDIR:
-            ms =3D mask - IN_ISDIR
-            name =3D '%s|IN_ISDIR'
-        return name % EventsCodes.ALL_VALUES[ms]
-
-    maskname =3D staticmethod(maskname)
-
-
-# So let's now turn the configuration into code
-EventsCodes.ALL_FLAGS =3D {}
-EventsCodes.ALL_VALUES =3D {}
-for flagc, valc in EventsCodes.FLAG_COLLECTIONS.items():
-    # Make the collections' members directly accessible through the
-    # class dictionary
-    setattr(EventsCodes, flagc, valc)
-
-    # Collect all the flags under a common umbrella
-    EventsCodes.ALL_FLAGS.update(valc)
-
-    # Make the individual masks accessible as 'constants' at globals() sco=
pe
-    # and masknames accessible by values.
-    for name, val in valc.items():
-        globals()[name] =3D val
-        EventsCodes.ALL_VALUES[val] =3D name
-
-
-# all 'normal' events
-ALL_EVENTS =3D reduce(lambda x, y: x | y, EventsCodes.OP_FLAGS.values())
-EventsCodes.ALL_FLAGS['ALL_EVENTS'] =3D ALL_EVENTS
-EventsCodes.ALL_VALUES[ALL_EVENTS] =3D 'ALL_EVENTS'
-
-
-class _Event:
-    """
-    Event structure, represent events raised by the system. This
-    is the base class and should be subclassed.
-
-    """
-    def __init__(self, dict_):
-        """
-        Attach attributes (contained in dict_) to self.
-
-        @param dict_: Set of attributes.
-        @type dict_: dictionary
-        """
-        for tpl in dict_.items():
-            setattr(self, *tpl)
-
-    def __repr__(self):
-        """
-        @return: Generic event string representation.
-        @rtype: str
-        """
-        s =3D ''
-        for attr, value in sorted(self.__dict__.items(), key=3Dlambda x: x=
[0]):
-            if attr.startswith('_'):
-                continue
-            if attr =3D=3D 'mask':
-                value =3D hex(getattr(self, attr))
-            elif isinstance(value, str) and not value:
-                value =3D "''"
-            s +=3D ' %s%s%s' % (output_format.field_name(attr),
-                              output_format.punctuation('=3D'),
-                              output_format.field_value(value))
-
-        s =3D '%s%s%s %s' % (output_format.punctuation('<'),
-                           output_format.class_name(self.__class__.__name_=
_),
-                           s,
-                           output_format.punctuation('>'))
-        return s
-
-    def __str__(self):
-        return repr(self)
-
-
-class _RawEvent(_Event):
-    """
-    Raw event, it contains only the informations provided by the system.
-    It doesn't infer anything.
-    """
-    def __init__(self, wd, mask, cookie, name):
-        """
-        @param wd: Watch Descriptor.
-        @type wd: int
-        @param mask: Bitmask of events.
-        @type mask: int
-        @param cookie: Cookie.
-        @type cookie: int
-        @param name: Basename of the file or directory against which the
-                     event was raised in case where the watched directory
-                     is the parent directory. None if the event was raised
-                     on the watched item itself.
-        @type name: string or None
-        """
-        # Use this variable to cache the result of str(self), this object
-        # is immutable.
-        self._str =3D None
-        # name: remove trailing '\0'
-        d =3D {'wd': wd,
-             'mask': mask,
-             'cookie': cookie,
-             'name': name.rstrip('\0')}
-        _Event.__init__(self, d)
-        log.debug(str(self))
-
-    def __str__(self):
-        if self._str is None:
-            self._str =3D _Event.__str__(self)
-        return self._str
-
-
-class Event(_Event):
-    """
-    This class contains all the useful informations about the observed
-    event. However, the presence of each field is not guaranteed and
-    depends on the type of event. In effect, some fields are irrelevant
-    for some kind of event (for example 'cookie' is meaningless for
-    IN_CREATE whereas it is mandatory for IN_MOVE_TO).
-
-    The possible fields are:
-      - wd (int): Watch Descriptor.
-      - mask (int): Mask.
-      - maskname (str): Readable event name.
-      - path (str): path of the file or directory being watched.
-      - name (str): Basename of the file or directory against which the
-              event was raised in case where the watched directory
-              is the parent directory. None if the event was raised
-              on the watched item itself. This field is always provided
-              even if the string is ''.
-      - pathname (str): Concatenation of 'path' and 'name'.
-      - src_pathname (str): Only present for IN_MOVED_TO events and only i=
n
-              the case where IN_MOVED_FROM events are watched too. Holds t=
he
-              source pathname from where pathname was moved from.
-      - cookie (int): Cookie.
-      - dir (bool): True if the event was raised against a directory.
-
-    """
-    def __init__(self, raw):
-        """
-        Concretely, this is the raw event plus inferred infos.
-        """
-        _Event.__init__(self, raw)
-        self.maskname =3D EventsCodes.maskname(self.mask)
-        if COMPATIBILITY_MODE:
-            self.event_name =3D self.maskname
-        try:
-            if self.name:
-                self.pathname =3D os.path.abspath(os.path.join(self.path,
-                                                             self.name))
-            else:
-                self.pathname =3D os.path.abspath(self.path)
-        except AttributeError as err:
-            # Usually it is not an error some events are perfectly valids
-            # despite the lack of these attributes.
-            log.debug(err)
-
-
-class ProcessEventError(PyinotifyError):
-    """
-    ProcessEventError Exception. Raised on ProcessEvent error.
-    """
-    def __init__(self, err):
-        """
-        @param err: Exception error description.
-        @type err: string
-        """
-        PyinotifyError.__init__(self, err)
-
-
-class _ProcessEvent:
-    """
-    Abstract processing event class.
-    """
-    def __call__(self, event):
-        """
-        To behave like a functor the object must be callable.
-        This method is a dispatch method. Its lookup order is:
-          1. process_MASKNAME method
-          2. process_FAMILY_NAME method
-          3. otherwise calls process_default
-
-        @param event: Event to be processed.
-        @type event: Event object
-        @return: By convention when used from the ProcessEvent class:
-                 - Returning False or None (default value) means keep on
-                   executing next chained functors (see chain.py example).
-                 - Returning True instead means do not execute next
-                   processing functions.
-        @rtype: bool
-        @raise ProcessEventError: Event object undispatchable,
-                                  unknown event.
-        """
-        stripped_mask =3D event.mask & ~IN_ISDIR
-        # Bitbake hack - we see event masks of 0x6, i.e., IN_MODIFY & IN_A=
TTRIB.
-        # The kernel inotify code can set more than one of the bits in the=
 mask,
-        # fsnotify_change() in linux/fsnotify.h is quite clear that IN_ATT=
RIB,
-        # IN_MODIFY and IN_ACCESS can arrive together.
-        # This breaks the code below which assume only one mask bit is eve=
r
-        # set in an event. We don't care about attrib or access in bitbake=
 so
-        # drop those.
-        if stripped_mask & IN_MODIFY:
-            stripped_mask &=3D ~(IN_ATTRIB | IN_ACCESS)
-
-        maskname =3D EventsCodes.ALL_VALUES.get(stripped_mask)
-        if maskname is None:
-            raise ProcessEventError("Unknown mask 0x%08x" % stripped_mask)
-
-        # 1- look for process_MASKNAME
-        meth =3D getattr(self, 'process_' + maskname, None)
-        if meth is not None:
-            return meth(event)
-        # 2- look for process_FAMILY_NAME
-        meth =3D getattr(self, 'process_IN_' + maskname.split('_')[1], Non=
e)
-        if meth is not None:
-            return meth(event)
-        # 3- default call method process_default
-        return self.process_default(event)
-
-    def __repr__(self):
-        return '<%s>' % self.__class__.__name__
-
-
-class _SysProcessEvent(_ProcessEvent):
-    """
-    There is three kind of processing according to each event:
-
-      1. special handling (deletion from internal container, bug, ...).
-      2. default treatment: which is applied to the majority of events.
-      3. IN_ISDIR is never sent alone, he is piggybacked with a standard
-         event, he is not processed as the others events, instead, its
-         value is captured and appropriately aggregated to dst event.
-    """
-    def __init__(self, wm, notifier):
-        """
-
-        @param wm: Watch Manager.
-        @type wm: WatchManager instance
-        @param notifier: Notifier.
-        @type notifier: Notifier instance
-        """
-        self._watch_manager =3D wm  # watch manager
-        self._notifier =3D notifier  # notifier
-        self._mv_cookie =3D {}  # {cookie(int): (src_path(str), date), ...=
}
-        self._mv =3D {}  # {src_path(str): (dst_path(str), date), ...}
-
-    def cleanup(self):
-        """
-        Cleanup (delete) old (>1mn) records contained in self._mv_cookie
-        and self._mv.
-        """
-        date_cur_ =3D datetime.now()
-        for seq in (self._mv_cookie, self._mv):
-            for k in list(seq.keys()):
-                if (date_cur_ - seq[k][1]) > timedelta(minutes=3D1):
-                    log.debug('Cleanup: deleting entry %s', seq[k][0])
-                    del seq[k]
-
-    def process_IN_CREATE(self, raw_event):
-        """
-        If the event affects a directory and the auto_add flag of the
-        targetted watch is set to True, a new watch is added on this
-        new directory, with the same attribute values than those of
-        this watch.
-        """
-        if raw_event.mask & IN_ISDIR:
-            watch_ =3D self._watch_manager.get_watch(raw_event.wd)
-            created_dir =3D os.path.join(watch_.path, raw_event.name)
-            if watch_.auto_add and not watch_.exclude_filter(created_dir):
-                addw =3D self._watch_manager.add_watch
-                # The newly monitored directory inherits attributes from i=
ts
-                # parent directory.
-                addw_ret =3D addw(created_dir, watch_.mask,
-                                proc_fun=3Dwatch_.proc_fun,
-                                rec=3DFalse, auto_add=3Dwatch_.auto_add,
-                                exclude_filter=3Dwatch_.exclude_filter)
-
-                # Trick to handle mkdir -p /d1/d2/t3 where d1 is watched a=
nd
-                # d2 and t3 (directory or file) are created.
-                # Since the directory d2 is new, then everything inside it=
 must
-                # also be new.
-                created_dir_wd =3D addw_ret.get(created_dir)
-                if ((created_dir_wd is not None) and (created_dir_wd > 0) =
and
-                    os.path.isdir(created_dir)):
-                    try:
-                        for name in os.listdir(created_dir):
-                            inner =3D os.path.join(created_dir, name)
-                            if self._watch_manager.get_wd(inner) is not No=
ne:
-                                continue
-                            # Generate (simulate) creation events for sub-
-                            # directories and files.
-                            if os.path.isfile(inner):
-                                # symlinks are handled as files.
-                                flags =3D IN_CREATE
-                            elif os.path.isdir(inner):
-                                flags =3D IN_CREATE | IN_ISDIR
-                            else:
-                                # This path should not be taken.
-                                continue
-                            rawevent =3D _RawEvent(created_dir_wd, flags, =
0, name)
-                            self._notifier.append_event(rawevent)
-                    except OSError as err:
-                        msg =3D "process_IN_CREATE, invalid directory: %s"
-                        log.debug(msg % str(err))
-        return self.process_default(raw_event)
-
-    def process_IN_MOVED_FROM(self, raw_event):
-        """
-        Map the cookie with the source path (+ date for cleaning).
-        """
-        watch_ =3D self._watch_manager.get_watch(raw_event.wd)
-        path_ =3D watch_.path
-        src_path =3D os.path.normpath(os.path.join(path_, raw_event.name))
-        self._mv_cookie[raw_event.cookie] =3D (src_path, datetime.now())
-        return self.process_default(raw_event, {'cookie': raw_event.cookie=
})
-
-    def process_IN_MOVED_TO(self, raw_event):
-        """
-        Map the source path with the destination path (+ date for
-        cleaning).
-        """
-        watch_ =3D self._watch_manager.get_watch(raw_event.wd)
-        path_ =3D watch_.path
-        dst_path =3D os.path.normpath(os.path.join(path_, raw_event.name))
-        mv_ =3D self._mv_cookie.get(raw_event.cookie)
-        to_append =3D {'cookie': raw_event.cookie}
-        if mv_ is not None:
-            self._mv[mv_[0]] =3D (dst_path, datetime.now())
-            # Let's assume that IN_MOVED_FROM event is always queued befor=
e
-            # that its associated (they share a common cookie) IN_MOVED_TO
-            # event is queued itself. It is then possible in that scenario
-            # to provide as additional information to the IN_MOVED_TO even=
t
-            # the original pathname of the moved file/directory.
-            to_append['src_pathname'] =3D mv_[0]
-        elif (raw_event.mask & IN_ISDIR and watch_.auto_add and
-              not watch_.exclude_filter(dst_path)):
-            # We got a diretory that's "moved in" from an unknown source a=
nd
-            # auto_add is enabled. Manually add watches to the inner subtr=
ees.
-            # The newly monitored directory inherits attributes from its
-            # parent directory.
-            self._watch_manager.add_watch(dst_path, watch_.mask,
-                                          proc_fun=3Dwatch_.proc_fun,
-                                          rec=3DTrue, auto_add=3DTrue,
-                                          exclude_filter=3Dwatch_.exclude_=
filter)
-        return self.process_default(raw_event, to_append)
-
-    def process_IN_MOVE_SELF(self, raw_event):
-        """
-        STATUS: the following bug has been fixed in recent kernels (FIXME:
-        which version ?). Now it raises IN_DELETE_SELF instead.
-
-        Old kernels were bugged, this event raised when the watched item
-        were moved, so we had to update its path, but under some circumsta=
nces
-        it was impossible: if its parent directory and its destination
-        directory wasn't watched. The kernel (see include/linux/fsnotify.h=
)
-        doesn't bring us enough informations like the destination path of
-        moved items.
-        """
-        watch_ =3D self._watch_manager.get_watch(raw_event.wd)
-        src_path =3D watch_.path
-        mv_ =3D self._mv.get(src_path)
-        if mv_:
-            dest_path =3D mv_[0]
-            watch_.path =3D dest_path
-            # add the separator to the source path to avoid overlapping
-            # path issue when testing with startswith()
-            src_path +=3D os.path.sep
-            src_path_len =3D len(src_path)
-            # The next loop renames all watches with src_path as base path=
.
-            # It seems that IN_MOVE_SELF does not provide IN_ISDIR informa=
tion
-            # therefore the next loop is iterated even if raw_event is a f=
ile.
-            for w in self._watch_manager.watches.values():
-                if w.path.startswith(src_path):
-                    # Note that dest_path is a normalized path.
-                    w.path =3D os.path.join(dest_path, w.path[src_path_len=
:])
-        else:
-            log.error("The pathname '%s' of this watch %s has probably cha=
nged "
-                      "and couldn't be updated, so it cannot be trusted "
-                      "anymore. To fix this error move directories/files o=
nly "
-                      "between watched parents directories, in this case e=
.g. "
-                      "put a watch on '%s'.",
-                      watch_.path, watch_,
-                      os.path.normpath(os.path.join(watch_.path,
-                                                    os.path.pardir)))
-            if not watch_.path.endswith('-unknown-path'):
-                watch_.path +=3D '-unknown-path'
-        return self.process_default(raw_event)
-
-    def process_IN_Q_OVERFLOW(self, raw_event):
-        """
-        Only signal an overflow, most of the common flags are irrelevant
-        for this event (path, wd, name).
-        """
-        return Event({'mask': raw_event.mask})
-
-    def process_IN_IGNORED(self, raw_event):
-        """
-        The watch descriptor raised by this event is now ignored (forever)=
,
-        it can be safely deleted from the watch manager dictionary.
-        After this event we can be sure that neither the event queue nor
-        the system will raise an event associated to this wd again.
-        """
-        event_ =3D self.process_default(raw_event)
-        self._watch_manager.del_watch(raw_event.wd)
-        return event_
-
-    def process_default(self, raw_event, to_append=3DNone):
-        """
-        Commons handling for the followings events:
-
-        IN_ACCESS, IN_MODIFY, IN_ATTRIB, IN_CLOSE_WRITE, IN_CLOSE_NOWRITE,
-        IN_OPEN, IN_DELETE, IN_DELETE_SELF, IN_UNMOUNT.
-        """
-        watch_ =3D self._watch_manager.get_watch(raw_event.wd)
-        if raw_event.mask & (IN_DELETE_SELF | IN_MOVE_SELF):
-            # Unfornulately this information is not provided by the kernel
-            dir_ =3D watch_.dir
-        else:
-            dir_ =3D bool(raw_event.mask & IN_ISDIR)
-        dict_ =3D {'wd': raw_event.wd,
-                 'mask': raw_event.mask,
-                 'path': watch_.path,
-                 'name': raw_event.name,
-                 'dir': dir_}
-        if COMPATIBILITY_MODE:
-            dict_['is_dir'] =3D dir_
-        if to_append is not None:
-            dict_.update(to_append)
-        return Event(dict_)
-
-
-class ProcessEvent(_ProcessEvent):
-    """
-    Process events objects, can be specialized via subclassing, thus its
-    behavior can be overriden:
-
-    Note: you should not override __init__ in your subclass instead define
-    a my_init() method, this method will be called automatically from the
-    constructor of this class with its optionals parameters.
-
-      1. Provide specialized individual methods, e.g. process_IN_DELETE fo=
r
-         processing a precise type of event (e.g. IN_DELETE in this case).
-      2. Or/and provide methods for processing events by 'family', e.g.
-         process_IN_CLOSE method will process both IN_CLOSE_WRITE and
-         IN_CLOSE_NOWRITE events (if process_IN_CLOSE_WRITE and
-         process_IN_CLOSE_NOWRITE aren't defined though).
-      3. Or/and override process_default for catching and processing all
-         the remaining types of events.
-    """
-    pevent =3D None
-
-    def __init__(self, pevent=3DNone, **kargs):
-        """
-        Enable chaining of ProcessEvent instances.
-
-        @param pevent: Optional callable object, will be called on event
-                       processing (before self).
-        @type pevent: callable
-        @param kargs: This constructor is implemented as a template method
-                      delegating its optionals keyworded arguments to the
-                      method my_init().
-        @type kargs: dict
-        """
-        self.pevent =3D pevent
-        self.my_init(**kargs)
-
-    def my_init(self, **kargs):
-        """
-        This method is called from ProcessEvent.__init__(). This method is
-        empty here and must be redefined to be useful. In effect, if you
-        need to specifically initialize your subclass' instance then you
-        just have to override this method in your subclass. Then all the
-        keyworded arguments passed to ProcessEvent.__init__() will be
-        transmitted as parameters to this method. Beware you MUST pass
-        keyword arguments though.
-
-        @param kargs: optional delegated arguments from __init__().
-        @type kargs: dict
-        """
-        pass
-
-    def __call__(self, event):
-        stop_chaining =3D False
-        if self.pevent is not None:
-            # By default methods return None so we set as guideline
-            # that methods asking for stop chaining must explicitely
-            # return non None or non False values, otherwise the default
-            # behavior will be to accept chain call to the corresponding
-            # local method.
-            stop_chaining =3D self.pevent(event)
-        if not stop_chaining:
-            return _ProcessEvent.__call__(self, event)
-
-    def nested_pevent(self):
-        return self.pevent
-
-    def process_IN_Q_OVERFLOW(self, event):
-        """
-        By default this method only reports warning messages, you can over=
redide
-        it by subclassing ProcessEvent and implement your own
-        process_IN_Q_OVERFLOW method. The actions you can take on receivin=
g this
-        event is either to update the variable max_queued_events in order =
to
-        handle more simultaneous events or to modify your code in order to
-        accomplish a better filtering diminishing the number of raised eve=
nts.
-        Because this method is defined, IN_Q_OVERFLOW will never get
-        transmitted as arguments to process_default calls.
-
-        @param event: IN_Q_OVERFLOW event.
-        @type event: dict
-        """
-        log.warning('Event queue overflowed.')
-
-    def process_default(self, event):
-        """
-        Default processing event method. By default does nothing. Subclass
-        ProcessEvent and redefine this method in order to modify its behav=
ior.
-
-        @param event: Event to be processed. Can be of any type of events =
but
-                      IN_Q_OVERFLOW events (see method process_IN_Q_OVERFL=
OW).
-        @type event: Event instance
-        """
-        pass
-
-
-class PrintAllEvents(ProcessEvent):
-    """
-    Dummy class used to print events strings representations. For instance=
 this
-    class is used from command line to print all received events to stdout=
.
-    """
-    def my_init(self, out=3DNone):
-        """
-        @param out: Where events will be written.
-        @type out: Object providing a valid file object interface.
-        """
-        if out is None:
-            out =3D sys.stdout
-        self._out =3D out
-
-    def process_default(self, event):
-        """
-        Writes event string representation to file object provided to
-        my_init().
-
-        @param event: Event to be processed. Can be of any type of events =
but
-                      IN_Q_OVERFLOW events (see method process_IN_Q_OVERFL=
OW).
-        @type event: Event instance
-        """
-        self._out.write(str(event))
-        self._out.write('\n')
-        self._out.flush()
-
-
-class ChainIfTrue(ProcessEvent):
-    """
-    Makes conditional chaining depending on the result of the nested
-    processing instance.
-    """
-    def my_init(self, func):
-        """
-        Method automatically called from base class constructor.
-        """
-        self._func =3D func
-
-    def process_default(self, event):
-        return not self._func(event)
-
-
-class Stats(ProcessEvent):
-    """
-    Compute and display trivial statistics about processed events.
-    """
-    def my_init(self):
-        """
-        Method automatically called from base class constructor.
-        """
-        self._start_time =3D time.time()
-        self._stats =3D {}
-        self._stats_lock =3D threading.Lock()
-
-    def process_default(self, event):
-        """
-        Processes |event|.
-        """
-        self._stats_lock.acquire()
-        try:
-            events =3D event.maskname.split('|')
-            for event_name in events:
-                count =3D self._stats.get(event_name, 0)
-                self._stats[event_name] =3D count + 1
-        finally:
-            self._stats_lock.release()
-
-    def _stats_copy(self):
-        self._stats_lock.acquire()
-        try:
-            return self._stats.copy()
-        finally:
-            self._stats_lock.release()
-
-    def __repr__(self):
-        stats =3D self._stats_copy()
-
-        elapsed =3D int(time.time() - self._start_time)
-        elapsed_str =3D ''
-        if elapsed < 60:
-            elapsed_str =3D str(elapsed) + 'sec'
-        elif 60 <=3D elapsed < 3600:
-            elapsed_str =3D '%dmn%dsec' % (elapsed / 60, elapsed % 60)
-        elif 3600 <=3D elapsed < 86400:
-            elapsed_str =3D '%dh%dmn' % (elapsed / 3600, (elapsed % 3600) =
/ 60)
-        elif elapsed >=3D 86400:
-            elapsed_str =3D '%dd%dh' % (elapsed / 86400, (elapsed % 86400)=
 / 3600)
-        stats['ElapsedTime'] =3D elapsed_str
-
-        l =3D []
-        for ev, value in sorted(stats.items(), key=3Dlambda x: x[0]):
-            l.append(' %s=3D%s' % (output_format.field_name(ev),
-                                 output_format.field_value(value)))
-        s =3D '<%s%s >' % (output_format.class_name(self.__class__.__name_=
_),
-                         ''.join(l))
-        return s
-
-    def dump(self, filename):
-        """
-        Dumps statistics.
-
-        @param filename: filename where stats will be dumped, filename is
-                         created and must not exist prior to this call.
-        @type filename: string
-        """
-        flags =3D os.O_WRONLY|os.O_CREAT|os.O_NOFOLLOW|os.O_EXCL
-        fd =3D os.open(filename, flags, 0o0600)
-        os.write(fd, bytes(self.__str__(), locale.getpreferredencoding()))
-        os.close(fd)
-
-    def __str__(self, scale=3D45):
-        stats =3D self._stats_copy()
-        if not stats:
-            return ''
-
-        m =3D max(stats.values())
-        unity =3D scale / m
-        fmt =3D '%%-26s%%-%ds%%s' % (len(output_format.field_value('@' * s=
cale))
-                                   + 1)
-        def func(x):
-            return fmt % (output_format.field_name(x[0]),
-                          output_format.field_value('@' * int(x[1] * unity=
)),
-                          output_format.simple('%d' % x[1], 'yellow'))
-        s =3D '\n'.join(map(func, sorted(stats.items(), key=3Dlambda x: x[=
0])))
-        return s
-
-
-class NotifierError(PyinotifyError):
-    """
-    Notifier Exception. Raised on Notifier error.
-
-    """
-    def __init__(self, err):
-        """
-        @param err: Exception string's description.
-        @type err: string
-        """
-        PyinotifyError.__init__(self, err)
-
-
-class Notifier:
-    """
-    Read notifications, process events.
-
-    """
-    def __init__(self, watch_manager, default_proc_fun=3DNone, read_freq=
=3D0,
-                 threshold=3D0, timeout=3DNone):
-        """
-        Initialization. read_freq, threshold and timeout parameters are us=
ed
-        when looping.
-
-        @param watch_manager: Watch Manager.
-        @type watch_manager: WatchManager instance
-        @param default_proc_fun: Default processing method. If None, a new
-                                 instance of PrintAllEvents will be assign=
ed.
-        @type default_proc_fun: instance of ProcessEvent
-        @param read_freq: if read_freq =3D=3D 0, events are read asap,
-                          if read_freq is > 0, this thread sleeps
-                          max(0, read_freq - (timeout / 1000)) seconds. Bu=
t if
-                          timeout is None it may be different because
-                          poll is blocking waiting for something to read.
-        @type read_freq: int
-        @param threshold: File descriptor will be read only if the accumul=
ated
-                          size to read becomes >=3D threshold. If !=3D 0, =
you likely
-                          want to use it in combination with an appropriat=
e
-                          value for read_freq because without that you wou=
ld
-                          keep looping without really reading anything and=
 that
-                          until the amount of events to read is >=3D thres=
hold.
-                          At least with read_freq set you might sleep.
-        @type threshold: int
-        @param timeout: see read_freq above. If provided, it must be set i=
n
-                        milliseconds. See
-                        https://docs.python.org/3/library/select.html#sele=
ct.poll.poll
-        @type timeout: int
-        """
-        # Watch Manager instance
-        self._watch_manager =3D watch_manager
-        # File descriptor
-        self._fd =3D self._watch_manager.get_fd()
-        # Poll object and registration
-        self._pollobj =3D select.poll()
-        self._pollobj.register(self._fd, select.POLLIN)
-        # This pipe is correctely initialized and used by ThreadedNotifier
-        self._pipe =3D (-1, -1)
-        # Event queue
-        self._eventq =3D deque()
-        # System processing functor, common to all events
-        self._sys_proc_fun =3D _SysProcessEvent(self._watch_manager, self)
-        # Default processing method
-        self._default_proc_fun =3D default_proc_fun
-        if default_proc_fun is None:
-            self._default_proc_fun =3D PrintAllEvents()
-        # Loop parameters
-        self._read_freq =3D read_freq
-        self._threshold =3D threshold
-        self._timeout =3D timeout
-        # Coalesce events option
-        self._coalesce =3D False
-        # set of str(raw_event), only used when coalesce option is True
-        self._eventset =3D set()
-
-    def append_event(self, event):
-        """
-        Append a raw event to the event queue.
-
-        @param event: An event.
-        @type event: _RawEvent instance.
-        """
-        self._eventq.append(event)
-
-    def proc_fun(self):
-        return self._default_proc_fun
-
-    def coalesce_events(self, coalesce=3DTrue):
-        """
-        Coalescing events. Events are usually processed by batchs, their s=
ize
-        depend on various factors. Thus, before processing them, events re=
ceived
-        from inotify are aggregated in a fifo queue. If this coalescing
-        option is enabled events are filtered based on their unicity, only
-        unique events are enqueued, doublons are discarded. An event is un=
ique
-        when the combination of its fields (wd, mask, cookie, name) is uni=
que
-        among events of a same batch. After a batch of events is processed=
 any
-        events is accepted again. By default this option is disabled, you =
have
-        to explictly call this function to turn it on.
-
-        @param coalesce: Optional new coalescing value. True by default.
-        @type coalesce: Bool
-        """
-        self._coalesce =3D coalesce
-        if not coalesce:
-            self._eventset.clear()
-
-    def check_events(self, timeout=3DNone):
-        """
-        Check for new events available to read, blocks up to timeout
-        milliseconds.
-
-        @param timeout: If specified it overrides the corresponding instan=
ce
-                        attribute _timeout. timeout must be sepcified in
-                        milliseconds.
-        @type timeout: int
-
-        @return: New events to read.
-        @rtype: bool
-        """
-        while True:
-            try:
-                # blocks up to 'timeout' milliseconds
-                if timeout is None:
-                    timeout =3D self._timeout
-                ret =3D self._pollobj.poll(timeout)
-            except select.error as err:
-                if err.args[0] =3D=3D errno.EINTR:
-                    continue # interrupted, retry
-                else:
-                    raise
-            else:
-                break
-
-        if not ret or (self._pipe[0] =3D=3D ret[0][0]):
-            return False
-        # only one fd is polled
-        return ret[0][1] & select.POLLIN
-
-    def read_events(self):
-        """
-        Read events from device, build _RawEvents, and enqueue them.
-        """
-        buf_ =3D array.array('i', [0])
-        # get event queue size
-        if fcntl.ioctl(self._fd, termios.FIONREAD, buf_, 1) =3D=3D -1:
-            return
-        queue_size =3D buf_[0]
-        if queue_size < self._threshold:
-            log.debug('(fd: %d) %d bytes available to read but threshold i=
s '
-                      'fixed to %d bytes', self._fd, queue_size,
-                      self._threshold)
-            return
-
-        try:
-            # Read content from file
-            r =3D os.read(self._fd, queue_size)
-        except Exception as msg:
-            raise NotifierError(msg)
-        log.debug('Event queue size: %d', queue_size)
-        rsum =3D 0  # counter
-        while rsum < queue_size:
-            s_size =3D 16
-            # Retrieve wd, mask, cookie and fname_len
-            wd, mask, cookie, fname_len =3D struct.unpack('iIII',
-                                                        r[rsum:rsum+s_size=
])
-            # Retrieve name
-            bname, =3D struct.unpack('%ds' % fname_len,
-                                   r[rsum + s_size:rsum + s_size + fname_l=
en])
-            # FIXME: should we explictly call sys.getdefaultencoding() her=
e ??
-            uname =3D bname.decode()
-            rawevent =3D _RawEvent(wd, mask, cookie, uname)
-            if self._coalesce:
-                # Only enqueue new (unique) events.
-                raweventstr =3D str(rawevent)
-                if raweventstr not in self._eventset:
-                    self._eventset.add(raweventstr)
-                    self._eventq.append(rawevent)
-            else:
-                self._eventq.append(rawevent)
-            rsum +=3D s_size + fname_len
-
-    def process_events(self):
-        """
-        Routine for processing events from queue by calling their
-        associated proccessing method (an instance of ProcessEvent).
-        It also does internal processings, to keep the system updated.
-        """
-        while self._eventq:
-            raw_event =3D self._eventq.popleft()  # pop next event
-            if self._watch_manager.ignore_events:
-                log.debug("Event ignored: %s" % repr(raw_event))
-                continue
-            watch_ =3D self._watch_manager.get_watch(raw_event.wd)
-            if (watch_ is None) and not (raw_event.mask & IN_Q_OVERFLOW):
-                if not (raw_event.mask & IN_IGNORED):
-                    # Not really sure how we ended up here, nor how we sho=
uld
-                    # handle these types of events and if it is appropriat=
e to
-                    # completly skip them (like we are doing here).
-                    log.warning("Unable to retrieve Watch object associate=
d to %s",
-                                repr(raw_event))
-                continue
-            revent =3D self._sys_proc_fun(raw_event)  # system processings
-            if watch_ and watch_.proc_fun:
-                watch_.proc_fun(revent)  # user processings
-            else:
-                self._default_proc_fun(revent)
-        self._sys_proc_fun.cleanup()  # remove olds MOVED_* events records
-        if self._coalesce:
-            self._eventset.clear()
-
-    def __daemonize(self, pid_file=3DNone, stdin=3Dos.devnull, stdout=3Dos=
.devnull,
-                    stderr=3Dos.devnull):
-        """
-        pid_file: file where the pid will be written. If pid_file=3DNone t=
he pid
-                  is written to /var/run/.pid, if
-                  pid_file=3DFalse no pid_file is written.
-        stdin, stdout, stderr: files associated to common streams.
-        """
-        if pid_file is None:
-            dirname =3D '/var/run/'
-            basename =3D os.path.basename(sys.argv[0]) or 'pyinotify'
-            pid_file =3D os.path.join(dirname, basename + '.pid')
-
-        if pid_file and os.path.lexists(pid_file):
-            err =3D 'Cannot daemonize: pid file %s already exists.' % pid_=
file
-            raise NotifierError(err)
-
-        def fork_daemon():
-            # Adapted from Chad J. Schroeder's recipe
-            # @see http://code.activestate.com/recipes/278731/
-            pid =3D os.fork()
-            if (pid =3D=3D 0):
-                # parent 2
-                os.setsid()
-                pid =3D os.fork()
-                if (pid =3D=3D 0):
-                    # child
-                    os.chdir('/')
-                    os.umask(0o022)
-                else:
-                    # parent 2
-                    os._exit(0)
-            else:
-                # parent 1
-                os._exit(0)
-
-            fd_inp =3D os.open(stdin, os.O_RDONLY)
-            os.dup2(fd_inp, 0)
-            fd_out =3D os.open(stdout, os.O_WRONLY|os.O_CREAT, 0o0600)
-            os.dup2(fd_out, 1)
-            fd_err =3D os.open(stderr, os.O_WRONLY|os.O_CREAT, 0o0600)
-            os.dup2(fd_err, 2)
-
-        # Detach task
-        fork_daemon()
-
-        # Write pid
-        if pid_file:
-            flags =3D os.O_WRONLY|os.O_CREAT|os.O_NOFOLLOW|os.O_EXCL
-            fd_pid =3D os.open(pid_file, flags, 0o0600)
-            os.write(fd_pid,  bytes(str(os.getpid()) + '\n',
-                                    locale.getpreferredencoding()))
-            os.close(fd_pid)
-            # Register unlink function
-            atexit.register(lambda : os.unlink(pid_file))
-
-    def _sleep(self, ref_time):
-        # Only consider sleeping if read_freq is > 0
-        if self._read_freq > 0:
-            cur_time =3D time.time()
-            sleep_amount =3D self._read_freq - (cur_time - ref_time)
-            if sleep_amount > 0:
-                log.debug('Now sleeping %d seconds', sleep_amount)
-                time.sleep(sleep_amount)
-
-    def loop(self, callback=3DNone, daemonize=3DFalse, **args):
-        """
-        Events are read only one time every min(read_freq, timeout)
-        seconds at best and only if the size to read is >=3D threshold.
-        After this method returns it must not be called again for the same
-        instance.
-
-        @param callback: Functor called after each event processing iterat=
ion.
-                         Expects to receive the notifier object (self) as =
first
-                         parameter. If this function returns True the loop=
 is
-                         immediately terminated otherwise the loop method =
keeps
-                         looping.
-        @type callback: callable object or function
-        @param daemonize: This thread is daemonized if set to True.
-        @type daemonize: boolean
-        @param args: Optional and relevant only if daemonize is True. Rema=
ining
-                     keyworded arguments are directly passed to daemonize =
see
-                     __daemonize() method. If pid_file=3DNone or is set to=
 a
-                     pathname the caller must ensure the file does not exi=
st
-                     before this method is called otherwise an exception
-                     pyinotify.NotifierError will be raised. If pid_file=
=3DFalse
-                     it is still daemonized but the pid is not written in =
any
-                     file.
-        @type args: various
-        """
-        if daemonize:
-            self.__daemonize(**args)
-
-        # Read and process events forever
-        while 1:
-            try:
-                self.process_events()
-                if (callback is not None) and (callback(self) is True):
-                    break
-                ref_time =3D time.time()
-                # check_events is blocking
-                if self.check_events():
-                    self._sleep(ref_time)
-                    self.read_events()
-            except KeyboardInterrupt:
-                # Stop monitoring if sigint is caught (Control-C).
-                log.debug('Pyinotify stops monitoring.')
-                break
-        # Close internals
-        self.stop()
-
-    def stop(self):
-        """
-        Close inotify's instance (close its file descriptor).
-        It destroys all existing watches, pending events,...
-        This method is automatically called at the end of loop().
-        Afterward it is invalid to access this instance.
-        """
-        if self._fd is not None:
-            self._pollobj.unregister(self._fd)
-            os.close(self._fd)
-            self._fd =3D None
-        self._sys_proc_fun =3D None
-
-
-class ThreadedNotifier(threading.Thread, Notifier):
-    """
-    This notifier inherits from threading.Thread for instanciating a separ=
ate
-    thread, and also inherits from Notifier, because it is a threaded noti=
fier.
-
-    Note that every functionality provided by this class is also provided
-    through Notifier class. Moreover Notifier should be considered first b=
ecause
-    it is not threaded and could be easily daemonized.
-    """
-    def __init__(self, watch_manager, default_proc_fun=3DNone, read_freq=
=3D0,
-                 threshold=3D0, timeout=3DNone):
-        """
-        Initialization, initialize base classes. read_freq, threshold and
-        timeout parameters are used when looping.
-
-        @param watch_manager: Watch Manager.
-        @type watch_manager: WatchManager instance
-        @param default_proc_fun: Default processing method. See base class=
.
-        @type default_proc_fun: instance of ProcessEvent
-        @param read_freq: if read_freq =3D=3D 0, events are read asap,
-                          if read_freq is > 0, this thread sleeps
-                          max(0, read_freq - (timeout / 1000)) seconds.
-        @type read_freq: int
-        @param threshold: File descriptor will be read only if the accumul=
ated
-                          size to read becomes >=3D threshold. If !=3D 0, =
you likely
-                          want to use it in combination with an appropriat=
e
-                          value set for read_freq because without that you=
 would
-                          keep looping without really reading anything and=
 that
-                          until the amount of events to read is >=3D thres=
hold. At
-                          least with read_freq you might sleep.
-        @type threshold: int
-        @param timeout: see read_freq above. If provided, it must be set i=
n
-                        milliseconds. See
-                        https://docs.python.org/3/library/select.html#sele=
ct.poll.poll
-        @type timeout: int
-        """
-        # Init threading base class
-        threading.Thread.__init__(self)
-        # Stop condition
-        self._stop_event =3D threading.Event()
-        # Init Notifier base class
-        Notifier.__init__(self, watch_manager, default_proc_fun, read_freq=
,
-                          threshold, timeout)
-        # Create a new pipe used for thread termination
-        self._pipe =3D os.pipe()
-        self._pollobj.register(self._pipe[0], select.POLLIN)
-
-    def stop(self):
-        """
-        Stop notifier's loop. Stop notification. Join the thread.
-        """
-        self._stop_event.set()
-        os.write(self._pipe[1], b'stop')
-        threading.Thread.join(self)
-        Notifier.stop(self)
-        self._pollobj.unregister(self._pipe[0])
-        os.close(self._pipe[0])
-        os.close(self._pipe[1])
-
-    def loop(self):
-        """
-        Thread's main loop. Don't meant to be called by user directly.
-        Call inherited start() method instead.
-
-        Events are read only once time every min(read_freq, timeout)
-        seconds at best and only if the size of events to read is >=3D thr=
eshold.
-        """
-        # When the loop must be terminated .stop() is called, 'stop'
-        # is written to pipe fd so poll() returns and .check_events()
-        # returns False which make evaluate the While's stop condition
-        # ._stop_event.isSet() wich put an end to the thread's execution.
-        while not self._stop_event.isSet():
-            self.process_events()
-            ref_time =3D time.time()
-            if self.check_events():
-                self._sleep(ref_time)
-                self.read_events()
-
-    def run(self):
-        """
-        Start thread's loop: read and process events until the method
-        stop() is called.
-        Never call this method directly, instead call the start() method
-        inherited from threading.Thread, which then will call run() in
-        its turn.
-        """
-        self.loop()
-
-
-class TornadoAsyncNotifier(Notifier):
-    """
-    Tornado ioloop adapter.
-
-    """
-    def __init__(self, watch_manager, ioloop, callback=3DNone,
-                 default_proc_fun=3DNone, read_freq=3D0, threshold=3D0, ti=
meout=3DNone,
-                 channel_map=3DNone):
-        """
-        Note that if later you must call ioloop.close() be sure to let the
-        default parameter to all_fds=3DFalse.
-
-        See example tornado_notifier.py for an example using this notifier=
.
-
-        @param ioloop: Tornado's IO loop.
-        @type ioloop: tornado.ioloop.IOLoop instance.
-        @param callback: Functor called at the end of each call to handle_=
read
-                         (IOLoop's read handler). Expects to receive the
-                         notifier object (self) as single parameter.
-        @type callback: callable object or function
-        """
-        self.io_loop =3D ioloop
-        self.handle_read_callback =3D callback
-        Notifier.__init__(self, watch_manager, default_proc_fun, read_freq=
,
-                          threshold, timeout)
-        ioloop.add_handler(self._fd, self.handle_read, ioloop.READ)
-
-    def stop(self):
-        self.io_loop.remove_handler(self._fd)
-        Notifier.stop(self)
-
-    def handle_read(self, *args, **kwargs):
-        """
-        See comment in AsyncNotifier.
-
-        """
-        self.read_events()
-        self.process_events()
-        if self.handle_read_callback is not None:
-            self.handle_read_callback(self)
-
-
-class AsyncioNotifier(Notifier):
-    """
-
-    asyncio/trollius event loop adapter.
-
-    """
-    def __init__(self, watch_manager, loop, callback=3DNone,
-                 default_proc_fun=3DNone, read_freq=3D0, threshold=3D0, ti=
meout=3DNone):
-        """
-
-        See examples/asyncio_notifier.py for an example usage.
-
-        @param loop: asyncio or trollius event loop instance.
-        @type loop: asyncio.BaseEventLoop or trollius.BaseEventLoop instan=
ce.
-        @param callback: Functor called at the end of each call to handle_=
read.
-                         Expects to receive the notifier object (self) as
-                         single parameter.
-        @type callback: callable object or function
-
-        """
-        self.loop =3D loop
-        self.handle_read_callback =3D callback
-        Notifier.__init__(self, watch_manager, default_proc_fun, read_freq=
,
-                          threshold, timeout)
-        loop.add_reader(self._fd, self.handle_read)
-
-    def stop(self):
-        self.loop.remove_reader(self._fd)
-        Notifier.stop(self)
-
-    def handle_read(self, *args, **kwargs):
-        self.read_events()
-        self.process_events()
-        if self.handle_read_callback is not None:
-            self.handle_read_callback(self)
-
-
-class Watch:
-    """
-    Represent a watch, i.e. a file or directory being watched.
-
-    """
-    __slots__ =3D ('wd', 'path', 'mask', 'proc_fun', 'auto_add',
-                 'exclude_filter', 'dir')
-
-    def __init__(self, wd, path, mask, proc_fun, auto_add, exclude_filter)=
:
-        """
-        Initializations.
-
-        @param wd: Watch descriptor.
-        @type wd: int
-        @param path: Path of the file or directory being watched.
-        @type path: str
-        @param mask: Mask.
-        @type mask: int
-        @param proc_fun: Processing callable object.
-        @type proc_fun:
-        @param auto_add: Automatically add watches on new directories.
-        @type auto_add: bool
-        @param exclude_filter: Boolean function, used to exclude new
-                               directories from being automatically watche=
d.
-                               See WatchManager.__init__
-        @type exclude_filter: callable object
-        """
-        self.wd =3D wd
-        self.path =3D path
-        self.mask =3D mask
-        self.proc_fun =3D proc_fun
-        self.auto_add =3D auto_add
-        self.exclude_filter =3D exclude_filter
-        self.dir =3D os.path.isdir(self.path)
-
-    def __repr__(self):
-        """
-        @return: String representation.
-        @rtype: str
-        """
-        s =3D ' '.join(['%s%s%s' % (output_format.field_name(attr),
-                                  output_format.punctuation('=3D'),
-                                  output_format.field_value(getattr(self,
-                                                                    attr))=
) \
-                      for attr in self.__slots__ if not attr.startswith('_=
')])
-
-        s =3D '%s%s %s %s' % (output_format.punctuation('<'),
-                            output_format.class_name(self.__class__.__name=
__),
-                            s,
-                            output_format.punctuation('>'))
-        return s
-
-
-class ExcludeFilter:
-    """
-    ExcludeFilter is an exclusion filter.
-    """
-    def __init__(self, arg_lst):
-        """
-        Examples:
-          ef1 =3D ExcludeFilter(["/etc/rc.*", "/etc/hostname"])
-          ef2 =3D ExcludeFilter("/my/path/exclude.lst")
-          Where exclude.lst contains:
-          /etc/rc.*
-          /etc/hostname
-
-        Note: it is not possible to exclude a file if its encapsulating
-              directory is itself watched. See this issue for more details
-              https://github.com/seb-m/pyinotify/issues/31
-
-        @param arg_lst: is either a list of patterns or a filename from wh=
ich
-                        patterns will be loaded.
-        @type arg_lst: list of str or str
-        """
-        if isinstance(arg_lst, str):
-            lst =3D self._load_patterns_from_file(arg_lst)
-        elif isinstance(arg_lst, list):
-            lst =3D arg_lst
-        else:
-            raise TypeError
-
-        self._lregex =3D []
-        for regex in lst:
-            self._lregex.append(re.compile(regex, re.UNICODE))
-
-    def _load_patterns_from_file(self, filename):
-        lst =3D []
-        with open(filename, 'r') as file_obj:
-            for line in file_obj.readlines():
-                # Trim leading an trailing whitespaces
-                pattern =3D line.strip()
-                if not pattern or pattern.startswith('#'):
-                    continue
-                lst.append(pattern)
-        return lst
-
-    def _match(self, regex, path):
-        return regex.match(path) is not None
-
-    def __call__(self, path):
-        """
-        @param path: Path to match against provided regexps.
-        @type path: str
-        @return: Return True if path has been matched and should
-                 be excluded, False otherwise.
-        @rtype: bool
-        """
-        for regex in self._lregex:
-            if self._match(regex, path):
-                return True
-        return False
-
-
-class WatchManagerError(Exception):
-    """
-    WatchManager Exception. Raised on error encountered on watches
-    operations.
-    """
-    def __init__(self, msg, wmd):
-        """
-        @param msg: Exception string's description.
-        @type msg: string
-        @param wmd: This dictionary contains the wd assigned to paths of t=
he
-                    same call for which watches were successfully added.
-        @type wmd: dict
-        """
-        self.wmd =3D wmd
-        Exception.__init__(self, msg)
-
-
-class WatchManager:
-    """
-    Provide operations for watching files and directories. Its internal
-    dictionary is used to reference watched items. When used inside
-    threaded code, one must instanciate as many WatchManager instances as
-    there are ThreadedNotifier instances.
-
-    """
-    def __init__(self, exclude_filter=3Dlambda path: False):
-        """
-        Initialization: init inotify, init watch manager dictionary.
-        Raise OSError if initialization fails, raise InotifyBindingNotFoun=
dError
-        if no inotify binding was found (through ctypes or from direct acc=
ess to
-        syscalls).
-
-        @param exclude_filter: boolean function, returns True if current
-                               path must be excluded from being watched.
-                               Convenient for providing a common exclusion
-                               filter for every call to add_watch.
-        @type exclude_filter: callable object
-        """
-        self._ignore_events =3D False
-        self._exclude_filter =3D exclude_filter
-        self._wmd =3D {}  # watch dict key: watch descriptor, value: watch
-
-        self._inotify_wrapper =3D INotifyWrapper.create()
-        if self._inotify_wrapper is None:
-            raise InotifyBindingNotFoundError()
-
-        self._fd =3D self._inotify_wrapper.inotify_init() # file descripto=
r
-        if self._fd < 0:
-            err =3D 'Cannot initialize new instance of inotify, %s'
-            raise OSError(err % self._inotify_wrapper.str_errno())
-
-    def close(self):
-        """
-        Close inotify's file descriptor, this action will also automatical=
ly
-        remove (i.e. stop watching) all its associated watch descriptors.
-        After a call to this method the WatchManager's instance become use=
less
-        and cannot be reused, a new instance must then be instanciated. It
-        makes sense to call this method in few situations for instance if
-        several independant WatchManager must be instanciated or if all wa=
tches
-        must be removed and no other watches need to be added.
-        """
-        os.close(self._fd)
-
-    def get_fd(self):
-        """
-        Return assigned inotify's file descriptor.
-
-        @return: File descriptor.
-        @rtype: int
-        """
-        return self._fd
-
-    def get_watch(self, wd):
-        """
-        Get watch from provided watch descriptor wd.
-
-        @param wd: Watch descriptor.
-        @type wd: int
-        """
-        return self._wmd.get(wd)
-
-    def del_watch(self, wd):
-        """
-        Remove watch entry associated to watch descriptor wd.
-
-        @param wd: Watch descriptor.
-        @type wd: int
-        """
-        try:
-            del self._wmd[wd]
-        except KeyError as err:
-            log.error('Cannot delete unknown watch descriptor %s' % str(er=
r))
-
-    @property
-    def watches(self):
-        """
-        Get a reference on the internal watch manager dictionary.
-
-        @return: Internal watch manager dictionary.
-        @rtype: dict
-        """
-        return self._wmd
-
-    def __format_path(self, path):
-        """
-        Format path to its internal (stored in watch manager) representati=
on.
-        """
-        # path must be a unicode string (str) and is just normalized.
-        return os.path.normpath(path)
-
-    def __add_watch(self, path, mask, proc_fun, auto_add, exclude_filter):
-        """
-        Add a watch on path, build a Watch object and insert it in the
-        watch manager dictionary. Return the wd value.
-        """
-        path =3D self.__format_path(path)
-        if auto_add and not mask & IN_CREATE:
-            mask |=3D IN_CREATE
-        wd =3D self._inotify_wrapper.inotify_add_watch(self._fd, path, mas=
k)
-        if wd < 0:
-            return wd
-        watch =3D Watch(wd=3Dwd, path=3Dpath, mask=3Dmask, proc_fun=3Dproc=
_fun,
-                      auto_add=3Dauto_add, exclude_filter=3Dexclude_filter=
)
-        # wd are _always_ indexed with their original unicode paths in wmd=
.
-        self._wmd[wd] =3D watch
-        log.debug('New %s', watch)
-        return wd
-
-    def __glob(self, path, do_glob):
-        if do_glob:
-            return glob.iglob(path)
-        else:
-            return [path]
-
-    def add_watch(self, path, mask, proc_fun=3DNone, rec=3DFalse,
-                  auto_add=3DFalse, do_glob=3DFalse, quiet=3DTrue,
-                  exclude_filter=3DNone):
-        """
-        Add watch(s) on the provided |path|(s) with associated |mask| flag
-        value and optionally with a processing |proc_fun| function and
-        recursive flag |rec| set to True.
-        All |path| components _must_ be str (i.e. unicode) objects.
-        If |path| is already watched it is ignored, but if it is called wi=
th
-        option rec=3DTrue a watch is put on each one of its not-watched
-        subdirectory.
-
-        @param path: Path to watch, the path can either be a file or a
-                     directory. Also accepts a sequence (list) of paths.
-        @type path: string or list of strings
-        @param mask: Bitmask of events.
-        @type mask: int
-        @param proc_fun: Processing object.
-        @type proc_fun: function or ProcessEvent instance or instance of
-                        one of its subclasses or callable object.
-        @param rec: Recursively add watches from path on all its
-                    subdirectories, set to False by default (doesn't
-                    follows symlinks in any case).
-        @type rec: bool
-        @param auto_add: Automatically add watches on newly created
-                         directories in watched parent |path| directory.
-                         If |auto_add| is True, IN_CREATE is ored with |ma=
sk|
-                         when the watch is added.
-        @type auto_add: bool
-        @param do_glob: Do globbing on pathname (see standard globbing
-                        module for more informations).
-        @type do_glob: bool
-        @param quiet: if False raises a WatchManagerError exception on
-                      error. See example not_quiet.py.
-        @type quiet: bool
-        @param exclude_filter: predicate (boolean function), which returns
-                               True if the current path must be excluded
-                               from being watched. This argument has
-                               precedence over exclude_filter passed to
-                               the class' constructor.
-        @type exclude_filter: callable object
-        @return: dict of paths associated to watch descriptors. A wd value
-                 is positive if the watch was added sucessfully, otherwise
-                 the value is negative. If the path was invalid or was alr=
eady
-                 watched it is not included into this returned dictionary.
-        @rtype: dict of {str: int}
-        """
-        ret_ =3D {} # return {path: wd, ...}
-
-        if exclude_filter is None:
-            exclude_filter =3D self._exclude_filter
-
-        # normalize args as list elements
-        for npath in self.__format_param(path):
-            # Require that path be a unicode string
-            if not isinstance(npath, str):
-                ret_[path] =3D -3
-                continue
-
-            # unix pathname pattern expansion
-            for apath in self.__glob(npath, do_glob):
-                # recursively list subdirs according to rec param
-                for rpath in self.__walk_rec(apath, rec):
-                    if not exclude_filter(rpath):
-                        wd =3D ret_[rpath] =3D self.__add_watch(rpath, mas=
k,
-                                                            proc_fun,
-                                                            auto_add,
-                                                            exclude_filter=
)
-                        if wd < 0:
-                            err =3D ('add_watch: cannot watch %s WD=3D%d, =
%s' % \
-                                       (rpath, wd,
-                                        self._inotify_wrapper.str_errno())=
)
-                            if quiet:
-                                log.error(err)
-                            else:
-                                raise WatchManagerError(err, ret_)
-                    else:
-                        # Let's say -2 means 'explicitely excluded
-                        # from watching'.
-                        ret_[rpath] =3D -2
-        return ret_
-
-    def __get_sub_rec(self, lpath):
-        """
-        Get every wd from self._wmd if its path is under the path of
-        one (at least) of those in lpath. Doesn't follow symlinks.
-
-        @param lpath: list of watch descriptor
-        @type lpath: list of int
-        @return: list of watch descriptor
-        @rtype: list of int
-        """
-        for d in lpath:
-            root =3D self.get_path(d)
-            if root is not None:
-                # always keep root
-                yield d
-            else:
-                # if invalid
-                continue
-
-            # nothing else to expect
-            if not os.path.isdir(root):
-                continue
-
-            # normalization
-            root =3D os.path.normpath(root)
-            # recursion
-            lend =3D len(root)
-            for iwd in self._wmd.items():
-                cur =3D iwd[1].path
-                pref =3D os.path.commonprefix([root, cur])
-                if root =3D=3D os.sep or (len(pref) =3D=3D lend and \
-                                      len(cur) > lend and \
-                                      cur[lend] =3D=3D os.sep):
-                    yield iwd[1].wd
-
-    def update_watch(self, wd, mask=3DNone, proc_fun=3DNone, rec=3DFalse,
-                     auto_add=3DFalse, quiet=3DTrue):
-        """
-        Update existing watch descriptors |wd|. The |mask| value, the
-        processing object |proc_fun|, the recursive param |rec| and the
-        |auto_add| and |quiet| flags can all be updated.
-
-        @param wd: Watch Descriptor to update. Also accepts a list of
-                   watch descriptors.
-        @type wd: int or list of int
-        @param mask: Optional new bitmask of events.
-        @type mask: int
-        @param proc_fun: Optional new processing function.
-        @type proc_fun: function or ProcessEvent instance or instance of
-                        one of its subclasses or callable object.
-        @param rec: Optionally adds watches recursively on all
-                    subdirectories contained into |wd| directory.
-        @type rec: bool
-        @param auto_add: Automatically adds watches on newly created
-                         directories in the watch's path corresponding to =
|wd|.
-                         If |auto_add| is True, IN_CREATE is ored with |ma=
sk|
-                         when the watch is updated.
-        @type auto_add: bool
-        @param quiet: If False raises a WatchManagerError exception on
-                      error. See example not_quiet.py
-        @type quiet: bool
-        @return: dict of watch descriptors associated to booleans values.
-                 True if the corresponding wd has been successfully
-                 updated, False otherwise.
-        @rtype: dict of {int: bool}
-        """
-        lwd =3D self.__format_param(wd)
-        if rec:
-            lwd =3D self.__get_sub_rec(lwd)
-
-        ret_ =3D {}  # return {wd: bool, ...}
-        for awd in lwd:
-            apath =3D self.get_path(awd)
-            if not apath or awd < 0:
-                err =3D 'update_watch: invalid WD=3D%d' % awd
-                if quiet:
-                    log.error(err)
-                    continue
-                raise WatchManagerError(err, ret_)
-
-            if mask:
-                wd_ =3D self._inotify_wrapper.inotify_add_watch(self._fd, =
apath,
-                                                              mask)
-                if wd_ < 0:
-                    ret_[awd] =3D False
-                    err =3D ('update_watch: cannot update %s WD=3D%d, %s' =
% \
-                               (apath, wd_, self._inotify_wrapper.str_errn=
o()))
-                    if quiet:
-                        log.error(err)
-                        continue
-                    raise WatchManagerError(err, ret_)
-
-                assert(awd =3D=3D wd_)
-
-            if proc_fun or auto_add:
-                watch_ =3D self._wmd[awd]
-
-            if proc_fun:
-                watch_.proc_fun =3D proc_fun
-
-            if auto_add:
-                watch_.auto_add =3D auto_add
-
-            ret_[awd] =3D True
-            log.debug('Updated watch - %s', self._wmd[awd])
-        return ret_
-
-    def __format_param(self, param):
-        """
-        @param param: Parameter.
-        @type param: string or int
-        @return: wrap param.
-        @rtype: list of type(param)
-        """
-        if isinstance(param, list):
-            for p_ in param:
-                yield p_
-        else:
-            yield param
-
-    def get_wd(self, path):
-        """
-        Returns the watch descriptor associated to path. This method
-        presents a prohibitive cost, always prefer to keep the WD
-        returned by add_watch(). If the path is unknown it returns None.
-
-        @param path: Path.
-        @type path: str
-        @return: WD or None.
-        @rtype: int or None
-        """
-        path =3D self.__format_path(path)
-        for iwd in self._wmd.items():
-            if iwd[1].path =3D=3D path:
-                return iwd[0]
-
-    def get_path(self, wd):
-        """
-        Returns the path associated to WD, if WD is unknown it returns Non=
e.
-
-        @param wd: Watch descriptor.
-        @type wd: int
-        @return: Path or None.
-        @rtype: string or None
-        """
-        watch_ =3D self._wmd.get(wd)
-        if watch_ is not None:
-            return watch_.path
-
-    def __walk_rec(self, top, rec):
-        """
-        Yields each subdirectories of top, doesn't follow symlinks.
-        If rec is false, only yield top.
-
-        @param top: root directory.
-        @type top: string
-        @param rec: recursive flag.
-        @type rec: bool
-        @return: path of one subdirectory.
-        @rtype: string
-        """
-        if not rec or os.path.islink(top) or not os.path.isdir(top):
-            yield top
-        else:
-            for root, dirs, files in os.walk(top):
-                yield root
-
-    def rm_watch(self, wd, rec=3DFalse, quiet=3DTrue):
-        """
-        Removes watch(s).
-
-        @param wd: Watch Descriptor of the file or directory to unwatch.
-                   Also accepts a list of WDs.
-        @type wd: int or list of int.
-        @param rec: Recursively removes watches on every already watched
-                    subdirectories and subfiles.
-        @type rec: bool
-        @param quiet: If False raises a WatchManagerError exception on
-                      error. See example not_quiet.py
-        @type quiet: bool
-        @return: dict of watch descriptors associated to booleans values.
-                 True if the corresponding wd has been successfully
-                 removed, False otherwise.
-        @rtype: dict of {int: bool}
-        """
-        lwd =3D self.__format_param(wd)
-        if rec:
-            lwd =3D self.__get_sub_rec(lwd)
-
-        ret_ =3D {}  # return {wd: bool, ...}
-        for awd in lwd:
-            # remove watch
-            wd_ =3D self._inotify_wrapper.inotify_rm_watch(self._fd, awd)
-            if wd_ < 0:
-                ret_[awd] =3D False
-                err =3D ('rm_watch: cannot remove WD=3D%d, %s' % \
-                           (awd, self._inotify_wrapper.str_errno()))
-                if quiet:
-                    log.error(err)
-                    continue
-                raise WatchManagerError(err, ret_)
-
-            # Remove watch from our dictionary
-            if awd in self._wmd:
-                del self._wmd[awd]
-            ret_[awd] =3D True
-            log.debug('Watch WD=3D%d (%s) removed', awd, self.get_path(awd=
))
-        return ret_
-
-
-    def watch_transient_file(self, filename, mask, proc_class):
-        """
-        Watch a transient file, which will be created and deleted frequent=
ly
-        over time (e.g. pid file).
-
-        @attention: Currently under the call to this function it is not
-        possible to correctly watch the events triggered into the same
-        base directory than the directory where is located this watched
-        transient file. For instance it would be wrong to make these
-        two successive calls: wm.watch_transient_file('/var/run/foo.pid', =
...)
-        and wm.add_watch('/var/run/', ...)
-
-        @param filename: Filename.
-        @type filename: string
-        @param mask: Bitmask of events, should contain IN_CREATE and IN_DE=
LETE.
-        @type mask: int
-        @param proc_class: ProcessEvent (or of one of its subclass), bewar=
e of
-                           accepting a ProcessEvent's instance as argument=
 into
-                           __init__, see transient_file.py example for mor=
e
-                           details.
-        @type proc_class: ProcessEvent's instance or of one of its subclas=
ses.
-        @return: Same as add_watch().
-        @rtype: Same as add_watch().
-        """
-        dirname =3D os.path.dirname(filename)
-        if dirname =3D=3D '':
-            return {}  # Maintains coherence with add_watch()
-        basename =3D os.path.basename(filename)
-        # Assuming we are watching at least for IN_CREATE and IN_DELETE
-        mask |=3D IN_CREATE | IN_DELETE
-
-        def cmp_name(event):
-            if getattr(event, 'name') is None:
-                return False
-            return basename =3D=3D event.name
-        return self.add_watch(dirname, mask,
-                              proc_fun=3Dproc_class(ChainIfTrue(func=3Dcmp=
_name)),
-                              rec=3DFalse,
-                              auto_add=3DFalse, do_glob=3DFalse,
-                              exclude_filter=3Dlambda path: False)
-
-    def get_ignore_events(self):
-        return self._ignore_events
-
-    def set_ignore_events(self, nval):
-        self._ignore_events =3D nval
-
-    ignore_events =3D property(get_ignore_events, set_ignore_events,
-                             "Make watch manager ignoring new events.")
-
-
-class RawOutputFormat:
-    """
-    Format string representations.
-    """
-    def __init__(self, format=3DNone):
-        self.format =3D format or {}
-
-    def simple(self, s, attribute):
-        if not isinstance(s, str):
-            s =3D str(s)
-        return (self.format.get(attribute, '') + s +
-                self.format.get('normal', ''))
-
-    def punctuation(self, s):
-        """Punctuation color."""
-        return self.simple(s, 'normal')
-
-    def field_value(self, s):
-        """Field value color."""
-        return self.simple(s, 'purple')
-
-    def field_name(self, s):
-        """Field name color."""
-        return self.simple(s, 'blue')
-
-    def class_name(self, s):
-        """Class name color."""
-        return self.format.get('red', '') + self.simple(s, 'bold')
-
-output_format =3D RawOutputFormat()
-
-class ColoredOutputFormat(RawOutputFormat):
-    """
-    Format colored string representations.
-    """
-    def __init__(self):
-        f =3D {'normal': '\033[0m',
-             'black': '\033[30m',
-             'red': '\033[31m',
-             'green': '\033[32m',
-             'yellow': '\033[33m',
-             'blue': '\033[34m',
-             'purple': '\033[35m',
-             'cyan': '\033[36m',
-             'bold': '\033[1m',
-             'uline': '\033[4m',
-             'blink': '\033[5m',
-             'invert': '\033[7m'}
-        RawOutputFormat.__init__(self, f)
-
-
-def compatibility_mode():
-    """
-    Use this function to turn on the compatibility mode. The compatibility
-    mode is used to improve compatibility with Pyinotify 0.7.1 (or older)
-    programs. The compatibility mode provides additional variables 'is_dir=
',
-    'event_name', 'EventsCodes.IN_*' and 'EventsCodes.ALL_EVENTS' as
-    Pyinotify 0.7.1 provided. Do not call this function from new programs!=
!
-    Especially if there are developped for Pyinotify >=3D 0.8.x.
-    """
-    setattr(EventsCodes, 'ALL_EVENTS', ALL_EVENTS)
-    for evname in globals():
-        if evname.startswith('IN_'):
-            setattr(EventsCodes, evname, globals()[evname])
-    global COMPATIBILITY_MODE
-    COMPATIBILITY_MODE =3D True
-
-
-def command_line():
-    """
-    By default the watched path is '/tmp' and all types of events are
-    monitored. Events monitoring serves forever, type c^c to stop it.
-    """
-    from optparse import OptionParser
-
-    usage =3D "usage: %prog [options] [path1] [path2] [pathn]"
-
-    parser =3D OptionParser(usage=3Dusage)
-    parser.add_option("-v", "--verbose", action=3D"store_true",
-                      dest=3D"verbose", help=3D"Verbose mode")
-    parser.add_option("-r", "--recursive", action=3D"store_true",
-                      dest=3D"recursive",
-                      help=3D"Add watches recursively on paths")
-    parser.add_option("-a", "--auto_add", action=3D"store_true",
-                      dest=3D"auto_add",
-                      help=3D"Automatically add watches on new directories=
")
-    parser.add_option("-g", "--glob", action=3D"store_true",
-                      dest=3D"glob",
-                      help=3D"Treat paths as globs")
-    parser.add_option("-e", "--events-list", metavar=3D"EVENT[,...]",
-                      dest=3D"events_list",
-                      help=3D("A comma-separated list of events to watch f=
or - "
-                           "see the documentation for valid options (defau=
lts"
-                           " to everything)"))
-    parser.add_option("-s", "--stats", action=3D"store_true",
-                      dest=3D"stats",
-                      help=3D"Display dummy statistics")
-    parser.add_option("-V", "--version", action=3D"store_true",
-                      dest=3D"version",  help=3D"Pyinotify version")
-    parser.add_option("-f", "--raw-format", action=3D"store_true",
-                      dest=3D"raw_format",
-                      help=3D"Disable enhanced output format.")
-    parser.add_option("-c", "--command", action=3D"store",
-                      dest=3D"command",
-                      help=3D"Shell command to run upon event")
-
-    (options, args) =3D parser.parse_args()
-
-    if options.verbose:
-        log.setLevel(10)
-
-    if options.version:
-        print(__version__)
-
-    if not options.raw_format:
-        global output_format
-        output_format =3D ColoredOutputFormat()
-
-    if len(args) < 1:
-        path =3D '/tmp'  # default watched path
-    else:
-        path =3D args
-
-    # watch manager instance
-    wm =3D WatchManager()
-    # notifier instance and init
-    if options.stats:
-        notifier =3D Notifier(wm, default_proc_fun=3DStats(), read_freq=3D=
5)
-    else:
-        notifier =3D Notifier(wm, default_proc_fun=3DPrintAllEvents())
-
-    # What mask to apply
-    mask =3D 0
-    if options.events_list:
-        events_list =3D options.events_list.split(',')
-        for ev in events_list:
-            evcode =3D EventsCodes.ALL_FLAGS.get(ev, 0)
-            if evcode:
-                mask |=3D evcode
-            else:
-                parser.error("The event '%s' specified with option -e"
-                             " is not valid" % ev)
-    else:
-        mask =3D ALL_EVENTS
-
-    # stats
-    cb_fun =3D None
-    if options.stats:
-        def cb(s):
-            sys.stdout.write(repr(s.proc_fun()))
-            sys.stdout.write('\n')
-            sys.stdout.write(str(s.proc_fun()))
-            sys.stdout.write('\n')
-            sys.stdout.flush()
-        cb_fun =3D cb
-
-    # External command
-    if options.command:
-        def cb(s):
-            subprocess.Popen(options.command, shell=3DTrue)
-        cb_fun =3D cb
-
-    log.debug('Start monitoring %s, (press c^c to halt pyinotify)' % path)
-
-    wm.add_watch(path, mask, rec=3Doptions.recursive, auto_add=3Doptions.a=
uto_add, do_glob=3Doptions.glob)
-    # Loop forever (until sigint signal get caught)
-    notifier.loop(callback=3Dcb_fun)
-
-
-if __name__ =3D=3D '__main__':
-    command_line()
diff --git a/bitbake/lib/toaster/bldcollector/admin.py b/bitbake/lib/toaste=
r/bldcollector/admin.py
index feaa888c..e5677a2a 100644
--- a/bitbake/lib/toaster/bldcollector/admin.py
+++ b/bitbake/lib/toaster/bldcollector/admin.py
@@ -8,6 +8,7 @@ from django import forms
 import django.db.models as models
=20
=20
+@admin.register(BitbakeVersion)
 class BitbakeVersionAdmin(admin.ModelAdmin):
=20
     # we override the formfield for db URLField
@@ -20,18 +21,17 @@ class BitbakeVersionAdmin(admin.ModelAdmin):
             db_field, **kwargs)
=20
=20
+@admin.register(Release)
 class ReleaseAdmin(admin.ModelAdmin):
     pass
=20
=20
+@admin.register(ToasterSetting)
 class ToasterSettingAdmin(admin.ModelAdmin):
     pass
=20
=20
+@admin.register(Layer_Version)
 class LayerVersionsAdmin(admin.ModelAdmin):
     pass
=20
-admin.site.register(Layer_Version, LayerVersionsAdmin)
-admin.site.register(BitbakeVersion, BitbakeVersionAdmin)
-admin.site.register(Release, ReleaseAdmin)
-admin.site.register(ToasterSetting, ToasterSettingAdmin)
diff --git a/bitbake/lib/toaster/bldcontrol/admin.py b/bitbake/lib/toaster/=
bldcontrol/admin.py
index 1754bc11..04dd4e16 100644
--- a/bitbake/lib/toaster/bldcontrol/admin.py
+++ b/bitbake/lib/toaster/bldcontrol/admin.py
@@ -5,7 +5,7 @@
 from django.contrib import admin
 from .models import BuildEnvironment
=20
+@admin.register(BuildEnvironment)
 class BuildEnvironmentAdmin(admin.ModelAdmin):
     pass
=20
-admin.site.register(BuildEnvironment, BuildEnvironmentAdmin)
diff --git a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py b/bitb=
ake/lib/toaster/bldcontrol/localhostbecontroller.py
index 577e765f..4e0a6e90 100644
--- a/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/bitbake/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -6,6 +6,11 @@
 # SPDX-License-Identifier: GPL-2.0-only
 #
=20
+# DEBUGGING NOTE: if you need to add debug statements into the files withi=
n a
+# Toaster-imported layer, you must disable the "git fetch ..." command aft=
er
+# the initial clone else any changes will be automatically replaced with t=
he
+# original code
+
 import os
 import re
 import shutil
@@ -35,7 +40,8 @@ class LocalhostBEController(BuildEnvironmentController):
=20
     def __init__(self, be):
         super(LocalhostBEController, self).__init__(be)
-        self.pokydirname =3D None
+        self.bitbake_dirname =3D None
+        self.oecore_dirname =3D None
         self.islayerset =3D False
=20
     def _shellcmd(self, command, cwd=3DNone, nowait=3DFalse,env=3DNone):
@@ -45,7 +51,10 @@ class LocalhostBEController(BuildEnvironmentController):
             env=3Dos.environ.copy()
=20
         logger.debug("lbc_shellcmd: (%s) %s" % (cwd, command))
-        p =3D subprocess.Popen(command, cwd =3D cwd, shell=3DTrue, stdout=
=3Dsubprocess.PIPE, stderr=3Dsubprocess.PIPE, env=3Denv)
+        if isinstance(command, str):
+            p =3D subprocess.Popen(command, cwd =3D cwd, shell=3DTrue, std=
out=3Dsubprocess.PIPE, stderr=3Dsubprocess.PIPE, env=3Denv)
+        else:
+            p =3D subprocess.Popen(command, cwd =3D cwd, stdout=3Dsubproce=
ss.PIPE, stderr=3Dsubprocess.PIPE, env=3Denv)
         if nowait:
             return
         (out,err) =3D p.communicate()
@@ -95,7 +104,7 @@ class LocalhostBEController(BuildEnvironmentController):
         # 1. get a list of repos with branches, and map dirpaths for each =
layer
         gitrepos =3D {}
=20
-        # if we're using a remotely fetched version of bitbake add its git
+        # We are using a remotely fetched version of bitbake - add its git
         # details to the list of repos to clone
         if bitbake.giturl and bitbake.commit:
             gitrepos[(bitbake.giturl, bitbake.commit)] =3D []
@@ -130,7 +139,7 @@ class LocalhostBEController(BuildEnvironmentController)=
:
         cached_layers =3D {}
=20
         try:
-            for remotes in self._shellcmd("git remote -v", self.be.sourced=
ir,env=3Dgit_env).split("\n"):
+            for remotes in self._shellcmd(['git', 'remote', '-v'], self.be=
.sourcedir, env=3Dgit_env).split("\n"):
                 try:
                     remote =3D remotes.split("\t")[1].split(" ")[0]
                     if remote not in cached_layers:
@@ -149,17 +158,24 @@ class LocalhostBEController(BuildEnvironmentControlle=
r):
         clone_total=3Dlen(gitrepos.keys())
         self.setCloneStatus(bitbake,'Started',clone_total,clone_count,'')
         for giturl, commit in gitrepos.keys():
-            self.setCloneStatus(bitbake,'progress',clone_total,clone_count=
,gitrepos[(giturl, commit)][0][0])
+            layer_name =3D gitrepos[(giturl, commit)][0][0]
+            self.setCloneStatus(bitbake,'progress',clone_total,clone_count=
,layer_name)
             clone_count +=3D 1
=20
             localdirname =3D os.path.join(self.be.sourcedir, self.getGitCl=
oneDirectory(giturl, commit))
             logger.debug("localhostbecontroller: giturl %s:%s checking out=
 in current directory %s" % (giturl, commit, localdirname))
=20
+            # Capture local paths to release's bitbake and openembedded-co=
re layers
+            if 'bitbake' =3D=3D layer_name:
+                self.bitbake_dirname =3D localdirname
+            # Capture local path to release's openembedded-core layer (for=
 'oe-init-build-env')
+            if 'openembedded-core' =3D=3D layer_name:
+                self.oecore_dirname =3D localdirname
+
             # see if our directory is a git repository
             if os.path.exists(localdirname):
                 try:
-                    localremotes =3D self._shellcmd("git remote -v",
-                                                  localdirname,env=3Dgit_e=
nv)
+                    localremotes =3D self._shellcmd(['git' ,'remote', '-v'=
], localdirname, env=3Dgit_env)
                     # NOTE: this nice-to-have check breaks when using git =
remaping to get past firewall
                     #       Re-enable later with .gitconfig remapping chec=
ks
                     #if not giturl in localremotes and commit !=3D 'HEAD':
@@ -172,28 +188,21 @@ class LocalhostBEController(BuildEnvironmentControlle=
r):
             else:
                 if giturl in cached_layers:
                     logger.debug("localhostbecontroller git-copying %s to =
%s" % (cached_layers[giturl], localdirname))
-                    self._shellcmd("git clone \"%s\" \"%s\"" % (cached_lay=
ers[giturl], localdirname),env=3Dgit_env)
-                    self._shellcmd("git remote remove origin", localdirnam=
e,env=3Dgit_env)
-                    self._shellcmd("git remote add origin \"%s\"" % giturl=
, localdirname,env=3Dgit_env)
+                    self._shellcmd(['git', 'clone', cached_layers[giturl],=
 localdirname], env=3Dgit_env)
+                    self._shellcmd(['git', 'remote', 'remove', 'origin'], =
localdirname, env=3Dgit_env)
+                    self._shellcmd(['git', 'remote', 'add', 'origin', gitu=
rl], localdirname, env=3Dgit_env)
                 else:
                     logger.debug("localhostbecontroller: cloning %s in %s"=
 % (giturl, localdirname))
-                    self._shellcmd('git clone "%s" "%s"' % (giturl, locald=
irname),env=3Dgit_env)
+                    self._shellcmd(['git', 'clone', giturl, localdirname],=
 env=3Dgit_env)
=20
             # branch magic name "HEAD" will inhibit checkout
             if commit !=3D "HEAD":
                 logger.debug("localhostbecontroller: checking out commit %=
s to %s " % (commit, localdirname))
                 ref =3D commit if re.match('^[a-fA-F0-9]+$', commit) else =
'origin/%s' % commit
-                self._shellcmd('git fetch && git reset --hard "%s"' % ref,=
 localdirname,env=3Dgit_env)
-
-            # take the localdirname as poky dir if we can find the oe-init=
-build-env
-            if self.pokydirname is None and os.path.exists(os.path.join(lo=
caldirname, "oe-init-build-env")):
-                logger.debug("localhostbecontroller: selected poky dir nam=
e %s" % localdirname)
-                self.pokydirname =3D localdirname
-
-                # make sure we have a working bitbake
-                if not os.path.exists(os.path.join(self.pokydirname, 'bitb=
ake')):
-                    logger.debug("localhostbecontroller: checking bitbake =
into the poky dirname %s " % self.pokydirname)
-                    self._shellcmd("git clone -b \"%s\" \"%s\" \"%s\" " % =
(bitbake.commit, bitbake.giturl, os.path.join(self.pokydirname, 'bitbake'))=
,env=3Dgit_env)
+                # DEBUGGING NOTE: this is the 'git fetch" to disable after=
 the initial clone to
+                # prevent inserted debugging commands from being lost
+                self._shellcmd(['git', 'fetch'], localdirname, env=3Dgit_e=
nv)
+                self._shellcmd(['git', 'reset', '--hard', ref], localdirna=
me, env=3Dgit_env)
=20
             # verify our repositories
             for name, dirpath, index in gitrepos[(giturl, commit)]:
@@ -208,21 +217,11 @@ class LocalhostBEController(BuildEnvironmentControlle=
r):
         self.setCloneStatus(bitbake,'complete',clone_total,clone_count,'')
         logger.debug("localhostbecontroller: current layer list %s " % pfo=
rmat(layerlist))
=20
-        # Resolve self.pokydirname if not resolved yet, consider the scena=
rio
-        # where all layers are local, that's the else clause
-        if self.pokydirname is None:
-            if os.path.exists(os.path.join(self.be.sourcedir, "oe-init-bui=
ld-env")):
-                logger.debug("localhostbecontroller: selected poky dir nam=
e %s" % self.be.sourcedir)
-                self.pokydirname =3D self.be.sourcedir
-            else:
-                # Alternatively, scan local layers for relative "oe-init-b=
uild-env" location
-                for layer in layers:
-                    if os.path.exists(os.path.join(layer.layer_version.lay=
er.local_source_dir,"..","oe-init-build-env")):
-                        logger.debug("localhostbecontroller, setting pokyd=
irname to %s" % (layer.layer_version.layer.local_source_dir))
-                        self.pokydirname =3D os.path.join(layer.layer_vers=
ion.layer.local_source_dir,"..")
-                        break
-                else:
-                    logger.error("pokydirname is not set, you will run int=
o trouble!")
+        # If self.bitbake_dirname still missing (e.g. 'local' build), use =
local bitbake
+        if not self.bitbake_dirname:
+            self.bitbake_dirname =3D os.environ.get('BBBASEDIR','')
+        if not self.bitbake_dirname:
+            logger.error("bitbake_dirname is not set, you will run into tr=
ouble!")
=20
         # 5. create custom layer and add custom recipes to it
         for target in targets:
@@ -319,13 +318,37 @@ class LocalhostBEController(BuildEnvironmentControlle=
r):
         layers =3D self.setLayers(bitbake, layers, targets)
         is_merged_attr =3D bitbake.req.project.merged_attr
=20
+        # Is this build for a Bitbake Setup installation?
+        isBBSetup =3D False
+        release_description =3D bitbake.req.project.release.description
+        if 'master' in release_description:
+            # This release going forward is Bitbake Setup
+            isBBSetup =3D True
+        else:
+            try:
+                # Yocto Project 5.3 "Whinlatter"
+                params =3D release_description.split(' ')
+                # Work past YP "10.1"
+                release_params =3D params[2].split('.')
+                if int(release_params[0]) > 5:
+                    isBBSetup =3D True
+                if (release_params[0] =3D=3D '5') and (int(release_params[=
1]) >=3D 3):
+                    isBBSetup =3D True
+            except:
+                # If release schema not recognized, then this release is i=
n the future
+                isBBSetup =3D True
+
         git_env =3D os.environ.copy()
+        toaster_run_dir =3D git_env['BUILDDIR']
         # (note: add custom environment settings here)
         try:
             # insure that the project init/build uses the selected bitbake=
, and not Toaster's
             del git_env['TEMPLATECONF']
-            del git_env['BBBASEDIR']
             del git_env['BUILDDIR']
+            # Insure that oe-init finds the right bitbake (e.g. 'oe-builde=
nv-internal')
+            git_env['BBBASEDIR'] =3D self.bitbake_dirname
+            git_env['BITBAKEDIR'] =3D self.bitbake_dirname
+
         except KeyError:
             pass
=20
@@ -334,29 +357,45 @@ class LocalhostBEController(BuildEnvironmentControlle=
r):
             builddir =3D bitbake.req.project.builddir
         else:
             builddir =3D '%s-toaster-%d' % (self.be.builddir, bitbake.req.=
project.id)
-        oe_init =3D os.path.join(self.pokydirname, 'oe-init-build-env')
-        # init build environment
+        oe_init =3D os.path.join(self.oecore_dirname, 'oe-init-build-env')
+        setup_init =3D os.path.join(builddir, 'init-build-env')
+        # init build environment and build directory structure
         try:
             custom_script =3D ToasterSetting.objects.get(name=3D"CUSTOM_BU=
ILD_INIT_SCRIPT").value
             custom_script =3D custom_script.replace("%BUILDDIR%" ,builddir=
)
             self._shellcmd("bash -c 'source %s'" % (custom_script),env=3Dg=
it_env)
         except ToasterSetting.DoesNotExist:
-            self._shellcmd("bash -c 'source %s %s'" % (oe_init, builddir),
-                       self.be.sourcedir,env=3Dgit_env)
+            # Normal setup
+            if isBBSetup:
+                # Create an 'init-build-env' for the build directory that =
points to respective cloned bitbake
+                os.makedirs(builddir, exist_ok =3D True)
+                build_init_script =3D os.path.join(builddir,'init-build-en=
v')
+                os.system(f"cp {os.path.join(toaster_run_dir,'init-build-e=
nv')} {build_init_script}")
+                os.system(f"sed -i -e 's|^cd .*$|cd {self.oecore_dirname}|=
g' {build_init_script}")
+                os.system(f"sed -i -e 's|^set .*$|set {builddir}|g' {build=
_init_script}")
+                # Execute the init
+                self._shellcmd(f"bash -c 'source {build_init_script}'", en=
v=3Dgit_env)
+            else:
+                self._shellcmd("bash -c 'source %s %s'" % (oe_init, buildd=
ir),
+                           self.be.sourcedir,env=3Dgit_env)
=20
         # update bblayers.conf
         if not is_merged_attr:
             bblconfpath =3D os.path.join(builddir, "conf/toaster-bblayers.=
conf")
             with open(bblconfpath, 'w') as bblayers:
                 bblayers.write('# line added by toaster build control\n'
-                               'BBLAYERS =3D "%s"' % ' '.join(layers))
+                               'BBLAYERS =3D "%s"\n' % ' '.join(layers))
=20
             # write configuration file
             confpath =3D os.path.join(builddir, 'conf/toaster.conf')
             with open(confpath, 'w') as conf:
+                if isBBSetup:
+                    # Force TOPDIR
+                    conf.write(f'# Force TOPDIR\n')
+                    conf.write(f'TOPDIR =3D "{builddir}"\n\n')
                 for var in variables:
-                    conf.write('%s=3D"%s"\n' % (var.name, var.value))
-                conf.write('INHERIT+=3D"toaster buildhistory"')
+                    conf.write('%s =3D "%s"\n' % (var.name, var.value))
+                conf.write('INHERIT +=3D "toaster buildhistory"\n')
         else:
             # Append the Toaster-specific values directly to the bblayers.=
conf
             bblconfpath =3D os.path.join(builddir, "conf/bblayers.conf")
@@ -432,10 +471,10 @@ class LocalhostBEController(BuildEnvironmentControlle=
r):
=20
         # run bitbake server from the clone if available
         # otherwise pick it from the PATH
-        bitbake =3D os.path.join(self.pokydirname, 'bitbake', 'bin', 'bitb=
ake')
+        bitbake =3D os.path.join(self.bitbake_dirname, 'bin', 'bitbake')
         if not os.path.exists(bitbake):
             logger.info("Bitbake not available under %s, will try to use i=
t from PATH" %
-                        self.pokydirname)
+                        self.bitbake_dirname)
             for path in os.environ["PATH"].split(os.pathsep):
                 if os.path.exists(os.path.join(path, 'bitbake')):
                     bitbake =3D os.path.join(path, 'bitbake')
@@ -443,15 +482,25 @@ class LocalhostBEController(BuildEnvironmentControlle=
r):
             else:
                 logger.error("Looks like Bitbake is not available, please =
fix your environment")
=20
+        # NOTE: BB SETUP requires a 'bblayers.conf' even though Toaster im=
mediately overrides it with 'toaster-bblayers.conf'
+        bblayers =3D os.path.join(builddir,"conf/bblayers.conf")
         toasterlayers =3D os.path.join(builddir,"conf/toaster-bblayers.con=
f")
-        if not is_merged_attr:
-            self._shellcmd('%s bash -c \"source %s %s; BITBAKE_UI=3D"knott=
y" %s --read %s --read %s '
-                           '--server-only -B 0.0.0.0:0\"' % (env_clean, oe=
_init,
-                           builddir, bitbake, confpath, toasterlayers), se=
lf.be.sourcedir)
+        if isBBSetup:
+            # Use 'init-build-env' model
+            if not is_merged_attr:
+                self._shellcmd(f'{env_clean} bash -c \"source {setup_init}=
; BITBAKE_UI=3D"knotty" {bitbake} --read {confpath} --read {bblayers} --rea=
d {toasterlayers} '
+                               '--server-only -B 0.0.0.0:0\"', self.be.sou=
rcedir,env=3Dgit_env)
+            else:
+                self._shellcmd(f'{env_clean} bash -c \"source {setup_init}=
; BITBAKE_UI=3D"knotty" {bitbake} '
+                           '--server-only -B 0.0.0.0:0\"', self.be.sourced=
ir,env=3Dgit_env)
         else:
-            self._shellcmd('%s bash -c \"source %s %s; BITBAKE_UI=3D"knott=
y" %s '
-                           '--server-only -B 0.0.0.0:0\"' % (env_clean, oe=
_init,
-                           builddir, bitbake), self.be.sourcedir)
+            # Use 'oe-init-build-env' model
+            if not is_merged_attr:
+                self._shellcmd(f'{env_clean} bash -c \"source {oe_init} {b=
uilddir}; BITBAKE_UI=3D"knotty" {bitbake} --read {confpath} --read {bblayer=
s} --read {toasterlayers} '
+                               '--server-only -B 0.0.0.0:0\"', self.be.sou=
rcedir,env=3Dgit_env)
+            else:
+                self._shellcmd(f'{env_clean} bash -c \"source {oe_init} {b=
uilddir}; BITBAKE_UI=3D"knotty" {bitbake} '
+                           '--server-only -B 0.0.0.0:0\"', self.be.sourced=
ir,env=3Dgit_env)
=20
         # read port number from bitbake.lock
         self.be.bbport =3D -1
@@ -498,19 +547,15 @@ class LocalhostBEController(BuildEnvironmentControlle=
r):
         local_bitbake =3D os.path.join(os.path.dirname(os.getenv('BBBASEDI=
R')),
                                      'bitbake')
         if not is_merged_attr:
-            self._shellcmd(['%s bash -c \"(TOASTER_BRBE=3D"%s" BBSERVER=3D=
"0.0.0.0:%s" '
-                        '%s %s -u toasterui  --read %s --read %s --token=
=3D"" >>%s 2>&1;'
-                        'BITBAKE_UI=3D"knotty" BBSERVER=3D0.0.0.0:%s %s -m=
)&\"' \
-                        % (env_clean, brbe, self.be.bbport, local_bitbake,=
 bbtargets, confpath, toasterlayers, log,
-                        self.be.bbport, bitbake,)],
+            self._shellcmd(f'{env_clean} bash -c \"(TOASTER_BRBE=3D"{brbe}=
" BBSERVER=3D"0.0.0.0:{self.be.bbport}" '
+                        f'{bitbake} {bbtargets} -u toasterui  --read {conf=
path} --read {bblayers} --read {toasterlayers} --token=3D"" >>{log} 2>&1;'
+                        f'BITBAKE_UI=3D"knotty" BBSERVER=3D0.0.0.0:{self.b=
e.bbport} {bitbake} -m)&\"',
                         builddir, nowait=3DTrue)
         else:
-            self._shellcmd(['%s bash -c \"(TOASTER_BRBE=3D"%s" BBSERVER=3D=
"0.0.0.0:%s" '
-                        '%s %s -u toasterui  --token=3D"" >>%s 2>&1;'
-                        'BITBAKE_UI=3D"knotty" BBSERVER=3D0.0.0.0:%s %s -m=
)&\"' \
-                        % (env_clean, brbe, self.be.bbport, local_bitbake,=
 bbtargets, log,
-                        self.be.bbport, bitbake,)],
+            self._shellcmd(f'{env_clean} bash -c \"(TOASTER_BRBE=3D"{brbe}=
" BBSERVER=3D"0.0.0.0:{self.be.bbport}" '
+                        f'{local_bitbake} {bbtargets} -u toasterui  --toke=
n=3D"" >>{log} 2>&1;'
+                        f'BITBAKE_UI=3D"knotty" BBSERVER=3D0.0.0.0:{self.b=
e.bbport} {bitbake} -m)&\"',
                         builddir, nowait=3DTrue)
=20
         logger.debug('localhostbecontroller: Build launched, exiting. '
-                     'Follow build logs at %s' % log)
+                     f'Follow build logs at {log}')
diff --git a/bitbake/lib/toaster/orm/fixtures/check_fixtures.py b/bitbake/l=
ib/toaster/orm/fixtures/check_fixtures.py
new file mode 100755
index 00000000..b33c093d
--- /dev/null
+++ b/bitbake/lib/toaster/orm/fixtures/check_fixtures.py
@@ -0,0 +1,76 @@
+#!/usr/bin/env python3
+#
+# Copyright (C) 2026 Linux Foundation
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+import json
+import urllib.request
+
+import gen_fixtures as fixtures
+
+RELEASE_URL =3D "https://dashboard.yoctoproject.org/releases.json"
+LAYER_INDEX_URL =3D "https://layers.openembedded.org/layerindex/api/branch=
es/"
+
+# Load a JSON file via REST
+def fetch_json(url):
+    with urllib.request.urlopen(url) as response:
+        if response.getcode() =3D=3D 200:
+            data =3D response.read().decode("utf-8")
+            return(json.loads(data))
+        else:
+            print("Couldn't access %s: %s" % (url, reponse.getcode()))
+            exit(1)
+
+# Grab the existing release branches in the Layer Index
+branches =3D fetch_json(LAYER_INDEX_URL)
+index_branches =3D [
+    e["name"].lower() for e in branches
+]
+
+# Grab the recent release branches and add master, so we can ignore old br=
anches
+releases =3D fetch_json(RELEASE_URL)
+active_releases =3D []
+active_releases_bitbake =3D {}
+active_but_not_index =3D []
+for e in releases:
+    if e["series"] !=3D "current":
+        continue
+    release =3D e["release_codename"].lower()
+    if not release in index_branches:
+        active_but_not_index.append(release)
+    else:
+        active_releases.append(release)
+    active_releases_bitbake[release] =3D e["bitbake_version"]
+active_releases.append("master")
+active_releases.append("head")
+active_releases.sort()
+
+if active_but_not_index:
+    print(f"Note: Active releases that are not yet in the Layer Index: {ac=
tive_but_not_index}" )
+
+# Get the list of releases instantiated in the Toaster fixtures
+fixtures_releases =3D []
+fixtures_releases_bitbake =3D {}
+for fixture in fixtures.current_releases:
+    release =3D fixture[0].lower()
+    fixtures_releases.append(release)
+    fixtures_releases_bitbake[release] =3D fixture[6]
+fixtures_releases.sort()
+
+# Report the resulting status
+ok =3D True
+for release in active_releases:
+    if release in ("master","head"):
+        continue
+    if active_releases_bitbake[release] !=3D fixtures_releases_bitbake[rel=
ease]:
+        print(f"ERROR: Bitbake version mismatch for '{release}': {active_r=
eleases_bitbake[release]} versus {fixtures_releases_bitbake[release]}")
+        ok =3D False
+if set(active_releases) !=3D set(fixtures_releases):
+    print(f"WARNING: Active releases don't match toaster configured releas=
es, the difference is: {set(active_releases).difference(set(fixtures_releas=
es))}")
+    print(f"Active releases: {sorted(active_releases)}")
+    print(f"Toaster configured releases: {sorted(fixtures_releases)}")
+    ok =3D False
+if ok:
+    print("Success, configuration matches")
+
diff --git a/bitbake/lib/toaster/orm/fixtures/gen_fixtures.py b/bitbake/lib=
/toaster/orm/fixtures/gen_fixtures.py
index 71afe391..58961d8e 100755
--- a/bitbake/lib/toaster/orm/fixtures/gen_fixtures.py
+++ b/bitbake/lib/toaster/orm/fixtures/gen_fixtures.py
@@ -4,12 +4,12 @@
 #
 # Generate Toaster Fixtures for 'poky.xml' and 'oe-core.xml'
 #
-# Copyright (C) 2022      Wind River Systems
+# Copyright (C) 2026      Wind River Systems
 # SPDX-License-Identifier: GPL-2.0-only
 #
 # Edit the 'current_releases' table for each new release cycle
 #
-# Usage: ./get_fixtures all
+# Usage: ./get_fixtures --all
 #
=20
 import os
@@ -32,22 +32,28 @@ verbose =3D False
 # It is also policy to include all active LTS releases.
 #
=20
-# [Codename, Yocto Project Version, Release Date, Current Version, Support=
 Level, Poky Version, BitBake branch]
+# [0=3DCodename, 1=3DYocto Project Version, 2=3DRelease Date, 3=3DCurrent =
Version, 4=3DSupport Level, 5=3DPoky Version, 6=3DBitBake branch]
 current_releases =3D [
     # Release slot #1
-    ['Kirkstone','4.0','April 2022','4.0.8 (March 2023)','Stable - Long Te=
rm Support (until Apr. 2024)','','2.0'],
+    ['Blacksail','6.1','October 2026','6.1 (October 2026)','Support for 7 =
months (until April 2027)','','2.20'],
     # Release slot #2 'local'
-    ['HEAD','HEAD','','Local Yocto Project','HEAD','','HEAD'],
+    ['Wrynose','6.0','April 2026','6.0 (April 2026)','Long Term Support (u=
ntil April 2030)','','2.18'],
     # Release slot #3 'master'
-    ['Master','master','','Yocto Project master','master','','master'],
+    ['HEAD','HEAD','','Local Yocto Project','HEAD','','HEAD'],
     # Release slot #4
-    ['Mickledore','4.2','April 2023','4.2.0 (April 2023)','Support for 7 m=
onths (until October 2023)','','2.4'],
-#   ['Langdale','4.1','October 2022','4.1.2 (January 2023)','Support for 7=
 months (until May 2023)','','2.2'],
-#   ['Honister','3.4','October 2021','3.4.2 (February 2022)','Support for =
7 months (until May 2022)','26.0','1.52'],
-#   ['Hardknott','3.3','April 2021','3.3.5 (March 2022)','Stable - Support=
 for 13 months (until Apr. 2022)','25.0','1.50'],
-#   ['Gatesgarth','3.2','Oct 2020','3.2.4 (May 2021)','EOL','24.0','1.48']=
,
-    # Optional Release slot #5
-    ['Dunfell','3.1','April 2020','3.1.23 (February 2023)','Stable - Long =
Term Support (until Apr. 2024)','23.0','1.46'],
+    ['Master','master','','Yocto Project master','master','','master'],
+    # Release slot #5
+    ['Scarthgap','5.0','April 2024','5.0 (April 2024)','Long Term Support =
(until April 2028)','','2.8'],
+    #['Whinlatter','5.3','October 2025','5.3 (October 2024)','Support for =
7 months (until May 2026)','','2.14'],
+    #['Styhead','5.1','November 2024','5.1.0 (November 2024)','Support for=
 7 months (until May 2025)','','2.10'],
+    #['Nanbield','4.3','November 2023','4.3.0 (November 2023)','Support fo=
r 7 months (until May 2024)','','2.6'],
+    #['Mickledore','4.2','April 2023','4.2.0 (April 2023)','Support for 7 =
months (until October 2023)','','2.4'],
+    #['Langdale','4.1','October 2022','4.1.2 (January 2023)','Support for =
7 months (until May 2023)','','2.2'],
+    #['Kirkstone','4.0','April 2022','4.0 (March 2023)','Stable - Long Ter=
m Support (until Apr. 2024)','','2.0'],
+    #['Honister','3.4','October 2021','3.4.2 (February 2022)','Support for=
 7 months (until May 2022)','26.0','1.52'],
+    #['Hardknott','3.3','April 2021','3.3.5 (March 2022)','Stable - Suppor=
t for 13 months (until Apr. 2022)','25.0','1.50'],
+    #['Gatesgarth','3.2','Oct 2020','3.2.4 (May 2021)','EOL','24.0','1.48'=
],
+    #['Dunfell','3.1','April 2020','3.1.23 (February 2023)','Stable - Long=
 Term Support (until Apr. 2024)','23.0','1.46'],
 ]
=20
 default_poky_layers =3D [
@@ -77,15 +83,15 @@ prolog_template =3D '''\
 bitbakeversion_poky_template =3D '''\
   
     {{name}}
-    git://git.yoctoproject.org/p=
oky
+    https://git.openembedded.org=
/bitbake
     {{branch}}
-    bitbake
+    
   
 '''
 bitbakeversion_oecore_template =3D '''\
   
     {{name}}
-    git://git.openembedded.org/b=
itbake
+    https://git.openembedded.org=
/bitbake
     {{bitbakeversion}}
   
 '''
@@ -220,7 +226,7 @@ def generate_poky():
             params['release'] =3D params['release'][0].lower() + params['r=
elease'][1:]
         params['name'] =3D params['release']
         params['bitbake_id'] =3D str(i+1)
-        params['branch'] =3D params['release']
+        params['branch'] =3D release[6]   # Bitbake branch
         print_template(bitbakeversion_poky_template,params,fd)
     print_str('',fd)
=20
@@ -243,7 +249,7 @@ def generate_poky():
             params['description'] =3D 'Local Yocto Project'
             params['name'] =3D 'local'
         else:
-            params['help_source'] =3D 'using the tip of the <a href=3D"=
https://git.yoctoproject.org/cgit/cgit.cgi/poky/log/{{h_release}}">Yocto=
 Project {{Release}} branch</a>'
+            params['help_source'] =3D 'using the tip of the <a href=3D"=
https://git.yoctoproject.org/meta-yocto/log/{{h_release}}">Yocto Project=
 {{Release}} branch</a>'
             params['description'] =3D 'Yocto Project {{release_version}} "=
{{Release}}"'
         if 'master' =3D=3D params['release']:
             params['h_release'] =3D ''
@@ -276,10 +282,17 @@ def generate_poky():
         params =3D {}
         params['layer'] =3D layer
         params['layer_id'] =3D str(i+1)
-        params['vcs_url'] =3D 'git://git.yoctoproject.org/poky'
-        params['vcs_web_url'] =3D 'https://git.yoctoproject.org/cgit/cgit.=
cgi/poky'
-        params['vcs_web_tree_base_url'] =3D 'https://git.yoctoproject.org/=
cgit/cgit.cgi/poky/tree/%path%?h=3D%branch%'
-        params['vcs_web_file_base_url'] =3D 'https://git.yoctoproject.org/=
cgit/cgit.cgi/poky/tree/%path%?h=3D%branch%'
+        if 'openembedded-core' =3D=3D layer:
+            params['vcs_url'] =3D 'https://git.openembedded.org/openembedd=
ed-core'
+            params['vcs_web_url'] =3D 'https://git.openembedded.org/openem=
bedded-core'
+            params['vcs_web_tree_base_url'] =3D 'https://git.openembedded.=
org/openembedded-core/tree/%path%?h=3D%branch%'
+            params['vcs_web_file_base_url'] =3D 'https://git.openembedded.=
org/openembedded-core/tree/%path%?h=3D%branch%'
+        else:
+            # Yocto Project layers: meta-poky, meta-yocto-bsp
+            params['vcs_url'] =3D 'https://git.yoctoproject.org/meta-yocto=
'
+            params['vcs_web_url'] =3D 'https://git.yoctoproject.org/meta-y=
octo/tree'
+            params['vcs_web_tree_base_url'] =3D 'https://git.yoctoproject.=
org/meta-yocto/tree/%path%?h=3D%branch%'
+            params['vcs_web_file_base_url'] =3D 'https://git.yoctoproject.=
org/meta-yocto/tree/%path%?h=3D%branch%'
=20
         if i:
             print_str('',fd)
@@ -388,10 +401,10 @@ def generate_oe_core():
         params =3D {}
         params['layer'] =3D layer
         params['layer_id'] =3D str(i+1)
-        params['vcs_url'] =3D 'git://git.openembedded.org/openembedded-cor=
e'
-        params['vcs_web_url'] =3D 'https://cgit.openembedded.org/openembed=
ded-core'
-        params['vcs_web_tree_base_url'] =3D 'https://cgit.openembedded.org=
/openembedded-core/tree/%path%?h=3D%branch%'
-        params['vcs_web_file_base_url'] =3D 'https://cgit.openembedded.org=
/openembedded-core/tree/%path%?h=3D%branch%'
+        params['vcs_url'] =3D 'https://git.openembedded.org/openembedded-c=
ore'
+        params['vcs_web_url'] =3D 'https://git.openembedded.org/openembedd=
ed-core'
+        params['vcs_web_tree_base_url'] =3D 'https://git.openembedded.org/=
openembedded-core/tree/%path%?h=3D%branch%'
+        params['vcs_web_file_base_url'] =3D 'https://git.openembedded.org/=
openembedded-core/tree/%path%?h=3D%branch%'
         if i:
             print_str('',fd)
         print_template(layer_oe_core_template,params,fd)
@@ -402,6 +415,68 @@ def generate_oe_core():
     print_str(epilog_template,fd)
     fd.close()
=20
+#################################
+# Update test project list
+#
+
+def generate_test():
+    this_path =3D os.path.dirname(os.path.abspath(__file__))
+    test_script =3D os.path.join(this_path,"../../tests/functional/test_cr=
eate_new_project.py")
+
+    is_copy =3D True
+    out_lines =3D ""
+    with open(test_script, 'r') as file_fd:
+        lines =3D file_fd.readlines()
+        for line in lines:
+            if 'test_create_new_project_without_name' in line:
+                is_copy =3D True
+                out_lines +=3D '\n'
+            if is_copy:
+                out_lines +=3D line
+            if line.startswith('class TestCreateNewProject'):
+                is_copy =3D False
+                # Insert new project entries
+
+                for id,release in enumerate(current_releases):
+                    release_id =3D id + 1
+                    release_name =3D release[0]
+                    release_merge =3D 'True'
+                    release_version =3D ''
+                    if 'Master' =3D=3D release_name:
+                        release_name =3D 'Master'
+                        release_title =3D 'Yocto Project master'
+                        release_merge =3D 'False'
+                    elif 'HEAD' =3D=3D release_name:
+                        release_name =3D 'Local'
+                        release_title =3D 'Local Yocto Project'
+                    else:
+                        release_title =3D f'Yocto Project {release[1]} "{r=
elease_name}"'
+                        release_version =3D f'{release[1]} '
+                    # Insert project
+                    test_release_def =3D f'''
+    def test_create_new_project_{release_name.lower()}(self):
+        """ Test create new project using:
+          - Project Name: Any string
+          - Release: Yocto Project {release_version}"{release_name}" (opti=
on value: {release_id})
+          - Merge Toaster settings: {release_merge}
+        """
+        release =3D '{release_id}'
+        release_title =3D '{release_title}'
+        project_name =3D 'project{release_name.lower()}'
+        self.create_new_project(
+            project_name,
+            release,
+            release_title,
+            {release_merge},
+        )
+'''
+                    out_lines +=3D test_release_def
+
+    with open('test_create_new_project.py', 'w') as file_fd:
+        file_fd.write(out_lines)
+    print(f"Output: 'test_create_new_project.py'")
+    os.system(f"mv -f test_create_new_project.py {test_script}")
+
 #################################
 # Help
 #
@@ -420,8 +495,9 @@ def main(argv):
     global verbose
=20
     parser =3D argparse.ArgumentParser(description=3D'gen_fixtures.py: tab=
le generate the fixture files')
-    parser.add_argument('--poky', '-p', action=3D'store_const', const=3D'p=
oky', dest=3D'command', help=3D'Generate the poky.xml file')
-    parser.add_argument('--oe-core', '-o', action=3D'store_const', const=
=3D'oe_core', dest=3D'command', help=3D'Generate the oe-core.xml file')
+    parser.add_argument('--poky', '-p', action=3D'store_const', const=3D'p=
oky', dest=3D'command', help=3D"Generate the 'poky.xml' file")
+    parser.add_argument('--oe-core', '-o', action=3D'store_const', const=
=3D'oe_core', dest=3D'command', help=3D"Generate the 'oe-core.xml' file")
+    parser.add_argument('--test', '-t', action=3D'store_const', const=3D't=
est', dest=3D'command', help=3D"Update the 'test_create_new_project.py' fil=
e")
     parser.add_argument('--all', '-a', action=3D'store_const', const=3D'al=
l', dest=3D'command', help=3D'Generate all fixture files')
     parser.add_argument('--list', '-l', action=3D'store_const', const=3D'l=
ist', dest=3D'command', help=3D'List the release table')
     parser.add_argument('--verbose', '-v', action=3D'store_true', dest=3D'=
verbose', help=3D'Enable verbose debugging output')
@@ -432,11 +508,12 @@ def main(argv):
         generate_poky()
     elif 'oe_core' =3D=3D args.command:
         generate_oe_core()
+    elif 'test' =3D=3D args.command:
+        generate_test()
     elif 'all' =3D=3D args.command:
         generate_poky()
         generate_oe_core()
-    elif 'all' =3D=3D args.command:
-        list_releases()
+        generate_test()
     elif 'list' =3D=3D args.command:
         list_releases()
=20
diff --git a/bitbake/lib/toaster/orm/fixtures/oe-core.xml b/bitbake/lib/toa=
ster/orm/fixtures/oe-core.xml
index 950f2a98..ccd8457a 100644
--- a/bitbake/lib/toaster/orm/fixtures/oe-core.xml
+++ b/bitbake/lib/toaster/orm/fixtures/oe-core.xml
@@ -8,66 +8,66 @@
=20
   
   
-    kirkstone
-    git://git.openembedded.org/b=
itbake
-    2.0
+    blacksail
+    https://git.openembedded.org=
/bitbake
+    2.20
   
   
+    wrynose
+    https://git.openembedded.org=
/bitbake
+    2.18
+  
+  
     HEAD
-    git://git.openembedded.org/b=
itbake
+    https://git.openembedded.org=
/bitbake
     HEAD
   
-  
+  
     master
-    git://git.openembedded.org/b=
itbake
+    https://git.openembedded.org=
/bitbake
     master
   
-  
-    mickledore
-    git://git.openembedded.org/b=
itbake
-    2.4
-  
   
-    dunfell
-    git://git.openembedded.org/b=
itbake
-    1.46
+    scarthgap
+    https://git.openembedded.org=
/bitbake
+    2.8
   
=20
   
   
-    kirkstone
-    Openembedded Kirkstone<=
/field>
+    blacksail
+    Openembedded Blacksail<=
/field>
     1
-    kirkstone
-    Toaster will run your buil=
ds using the tip of the <a href=3D\"https://cgit.openembedded.org/openem=
bedded-core/log/?h=3Dkirkstone\">OpenEmbedded Kirkstone</a> branch=
.
+    blacksail
+    Toaster will run your buil=
ds using the tip of the <a href=3D\"https://cgit.openembedded.org/openem=
bedded-core/log/?h=3Dblacksail\">OpenEmbedded Blacksail</a> branch=
.
   
   
+    wrynose
+    Openembedded Wrynose
+    2
+    wrynose
+    Toaster will run your buil=
ds using the tip of the <a href=3D\"https://cgit.openembedded.org/openem=
bedded-core/log/?h=3Dwrynose\">OpenEmbedded Wrynose</a> branch.
+  
+  
     local
     Local Openembedded
-    2
+    3
     HEAD
     Toaster will run your buil=
ds with the version of OpenEmbedded that you have cloned or downloaded to y=
our computer.
   
-  
+  
     master
     OpenEmbedded core maste=
r
-    3
+    4
     master
     Toaster will run your buil=
ds using the tip of the <a href=3D\"https://cgit.openembedded.org/openem=
bedded-core/log/\">OpenEmbedded master</a> branch.
   
-  
-    mickledore
-    Openembedded Mickledore=

-    4
-    mickledore
-    Toaster will run your buil=
ds using the tip of the <a href=3D\"https://cgit.openembedded.org/openem=
bedded-core/log/?h=3Dmickledore\">OpenEmbedded Mickledore</a> bran=
ch.
-  
   
-    dunfell
-    Openembedded Dunfell
+    scarthgap
+    Openembedded Scarthgap<=
/field>
     5
-    dunfell
-    Toaster will run your buil=
ds using the tip of the <a href=3D\"https://cgit.openembedded.org/openem=
bedded-core/log/?h=3Ddunfell\">OpenEmbedded Dunfell</a> branch.
+    scarthgap
+    Toaster will run your buil=
ds using the tip of the <a href=3D\"https://cgit.openembedded.org/openem=
bedded-core/log/?h=3Dscarthgap\">OpenEmbedded Scarthgap</a> branch=
.
   
=20
   
@@ -96,10 +96,10 @@
   
   
     openembedded-core
-    git://git.openembedded.org/=
openembedded-core
-    https://cgit.openembedd=
ed.org/openembedded-core
-    https://cgit.=
openembedded.org/openembedded-core/tree/%path%?h=3D%branch%
-    https://cgit.=
openembedded.org/openembedded-core/tree/%path%?h=3D%branch%
+    https://git.openembedded.or=
g/openembedded-core
+    https://git.openembedde=
d.org/openembedded-core
+    https://git.o=
penembedded.org/openembedded-core/tree/%path%?h=3D%branch%
+    https://git.o=
penembedded.org/openembedded-core/tree/%path%?h=3D%branch%
   
   
     1
diff --git a/bitbake/lib/toaster/orm/fixtures/poky.xml b/bitbake/lib/toaste=
r/orm/fixtures/poky.xml
index 121e52fd..d4c1ddfd 100644
--- a/bitbake/lib/toaster/orm/fixtures/poky.xml
+++ b/bitbake/lib/toaster/orm/fixtures/poky.xml
@@ -8,72 +8,72 @@
=20
   
   
-    kirkstone
-    git://git.yoctoproject.org/p=
oky
-    kirkstone
-    bitbake
+    blacksail
+    https://git.openembedded.org=
/bitbake
+    2.20
+    
   
   
+    wrynose
+    https://git.openembedded.org=
/bitbake
+    2.18
+    
+  
+  
     HEAD
-    git://git.yoctoproject.org/p=
oky
+    https://git.openembedded.org=
/bitbake
     HEAD
-    bitbake
+    
   
-  
+  
     master
-    git://git.yoctoproject.org/p=
oky
+    https://git.openembedded.org=
/bitbake
     master
-    bitbake
-  
-  
-    mickledore
-    git://git.yoctoproject.org/p=
oky
-    mickledore
-    bitbake
+    
   
   
-    dunfell
-    git://git.yoctoproject.org/p=
oky
-    dunfell
-    bitbake
+    scarthgap
+    https://git.openembedded.org=
/bitbake
+    2.8
+    
   
=20
=20
   
   
-    kirkstone
-    Yocto Project 4.0 "Kirk=
stone"
+    blacksail
+    Yocto Project 6.1 "Blac=
ksail"
     1
-    kirkstone
-    Toaster will run your buil=
ds using the tip of the <a href=3D"https://git.yoctoproject.org/cgit/cgi=
t.cgi/poky/log/?h=3Dkirkstone">Yocto Project Kirkstone branch</a>.=

+    blacksail
+    Toaster will run your buil=
ds using the tip of the <a href=3D"https://git.yoctoproject.org/meta-yoc=
to/log/?h=3Dblacksail">Yocto Project Blacksail branch</a>.
   
   
+    wrynose
+    Yocto Project 6.0 "Wryn=
ose"
+    2
+    wrynose
+    Toaster will run your buil=
ds using the tip of the <a href=3D"https://git.yoctoproject.org/meta-yoc=
to/log/?h=3Dwrynose">Yocto Project Wrynose branch</a>.
+  
+  
     local
     Local Yocto Project
-    2
+    3
     HEAD
     Toaster will run your buil=
ds with the version of the Yocto Project you have cloned or downloaded to y=
our computer.
   
-  
+  
     master
     Yocto Project master
-    3
-    master
-    Toaster will run your buil=
ds using the tip of the <a href=3D"https://git.yoctoproject.org/cgit/cgi=
t.cgi/poky/log/">Yocto Project Master branch</a>.
-  
-  
-    mickledore
-    Yocto Project 4.2 "Mick=
ledore"
     4
-    mickledore
-    Toaster will run your buil=
ds using the tip of the <a href=3D"https://git.yoctoproject.org/cgit/cgi=
t.cgi/poky/log/?h=3Dmickledore">Yocto Project Mickledore branch</a>=
;.
+    master
+    Toaster will run your buil=
ds using the tip of the <a href=3D"https://git.yoctoproject.org/meta-yoc=
to/log/">Yocto Project Master branch</a>.
   
   
-    dunfell
-    Yocto Project 3.1 "Dunf=
ell"
+    scarthgap
+    Yocto Project 5.0 "Scar=
thgap"
     5
-    dunfell
-    Toaster will run your buil=
ds using the tip of the <a href=3D"https://git.yoctoproject.org/cgit/cgi=
t.cgi/poky/log/?h=3Ddunfell">Yocto Project Dunfell branch</a>.
+    scarthgap
+    Toaster will run your buil=
ds using the tip of the <a href=3D"https://git.yoctoproject.org/meta-yoc=
to/log/?h=3Dscarthgap">Yocto Project Scarthgap branch</a>.
   
=20
   
@@ -146,135 +146,135 @@
   
     openembedded-core
     
-    git://git.yoctoproject.org/=
poky
-    https://git.yoctoprojec=
t.org/cgit/cgit.cgi/poky
-    https://git.y=
octoproject.org/cgit/cgit.cgi/poky/tree/%path%?h=3D%branch%
-    https://git.y=
octoproject.org/cgit/cgit.cgi/poky/tree/%path%?h=3D%branch%
+    https://git.openembedded.or=
g/openembedded-core
+    https://git.openembedde=
d.org/openembedded-core
+    https://git.o=
penembedded.org/openembedded-core/tree/%path%?h=3D%branch%
+    https://git.o=
penembedded.org/openembedded-core/tree/%path%?h=3D%branch%
   
   
     1
     0
     1
-    kirkstone
+    blacksail
     meta
   
   
     1
     0
     2
-    HEAD
-    HEAD
+    wrynose
     meta
   
   
     1
     0
     3
-    master
+    HEAD
+    HEAD
     meta
   
   
     1
     0
     4
-    mickledore
+    master
     meta
   
   
     1
     0
     5
-    dunfell
+    scarthgap
     meta
   
=20
   
     meta-poky
     
-    git://git.yoctoproject.org/=
poky
-    https://git.yoctoprojec=
t.org/cgit/cgit.cgi/poky
-    https://git.y=
octoproject.org/cgit/cgit.cgi/poky/tree/%path%?h=3D%branch%
-    https://git.y=
octoproject.org/cgit/cgit.cgi/poky/tree/%path%?h=3D%branch%
+    https://git.yoctoproject.or=
g/meta-yocto
+    https://git.yoctoprojec=
t.org/meta-yocto/tree
+    https://git.y=
octoproject.org/meta-yocto/tree/%path%?h=3D%branch%
+    https://git.y=
octoproject.org/meta-yocto/tree/%path%?h=3D%branch%
   
   
     2
     0
     1
-    kirkstone
+    blacksail
     meta-poky
   
   
     2
     0
     2
-    HEAD
-    HEAD
+    wrynose
     meta-poky
   
   
     2
     0
     3
-    master
+    HEAD
+    HEAD
     meta-poky
   
   
     2
     0
     4
-    mickledore
+    master
     meta-poky
   
   
     2
     0
     5
-    dunfell
+    scarthgap
     meta-poky
   
=20
   
     meta-yocto-bsp
     
-    git://git.yoctoproject.org/=
poky
-    https://git.yoctoprojec=
t.org/cgit/cgit.cgi/poky
-    https://git.y=
octoproject.org/cgit/cgit.cgi/poky/tree/%path%?h=3D%branch%
-    https://git.y=
octoproject.org/cgit/cgit.cgi/poky/tree/%path%?h=3D%branch%
+    https://git.yoctoproject.or=
g/meta-yocto
+    https://git.yoctoprojec=
t.org/meta-yocto/tree
+    https://git.y=
octoproject.org/meta-yocto/tree/%path%?h=3D%branch%
+    https://git.y=
octoproject.org/meta-yocto/tree/%path%?h=3D%branch%
   
   
     3
     0
     1
-    kirkstone
+    blacksail
     meta-yocto-bsp
   
   
     3
     0
     2
-    HEAD
-    HEAD
+    wrynose
     meta-yocto-bsp
   
   
     3
     0
     3
-    master
+    HEAD
+    HEAD
     meta-yocto-bsp
   
   
     3
     0
     4
-    mickledore
+    master
     meta-yocto-bsp
   
   
     3
     0
     5
-    dunfell
+    scarthgap
     meta-yocto-bsp
   
 
diff --git a/bitbake/lib/toaster/orm/management/commands/lsupdates.py b/bit=
bake/lib/toaster/orm/management/commands/lsupdates.py
index 6d64830e..1f36906e 100644
--- a/bitbake/lib/toaster/orm/management/commands/lsupdates.py
+++ b/bitbake/lib/toaster/orm/management/commands/lsupdates.py
@@ -105,7 +105,8 @@ class Command(BaseCommand):
             url_branches =3D ";branch=3D%s" % ','.join(allowed_branch_name=
s)
         else:
             url_branches =3D ""
-        layerindex.load_layerindex("%s%s" % (self.apiurl, url_branches))
+        layerindex.load_layerindex("%s%s" % (self.apiurl, url_branches),
+                                   load=3D['layerDependencies', 'recipes',=
 'machines', 'distros'])
=20
         http_progress.stop()
=20
@@ -138,9 +139,7 @@ class Command(BaseCommand):
                 l.summary =3D index.layerItems[id].summary
                 l.description =3D index.layerItems[id].description
=20
-                if created:
-                    # predefined layers in the fixtures (for example poky.=
xml)
-                    # always preempt the Layer Index for these values
+                if True: # (if created:) always override fixture defaults =
now that poky distro is stripped
                     l.vcs_url =3D index.layerItems[id].vcs_url
                     l.vcs_web_url =3D index.layerItems[id].vcs_web_url
                     l.vcs_web_tree_base_url =3D index.layerItems[id].vcs_w=
eb_tree_base_url
@@ -179,7 +178,7 @@ class Command(BaseCommand):
                     index.layerBranches[id].layer_id)
                 continue
=20
-            if created:
+            if True: # (if created:) always override fixture defaults now =
that poky distro is stripped
                 lv.release =3D li_branch_id_to_toaster_release[index.layer=
Branches[id].branch_id]
                 lv.up_date =3D index.layerBranches[id].updated
                 lv.commit =3D index.layerBranches[id].actual_branch
@@ -227,14 +226,18 @@ class Command(BaseCommand):
=20
         total =3D len(index.distros)
         for i, id in enumerate(index.distros):
-            distro, created =3D Distro.objects.get_or_create(
-                name=3Dindex.distros[id].name,
-                layer_version=3DLayer_Version.objects.get(
-                    pk=3Dli_layer_branch_id_to_toaster_lv_id[index.distros=
[id].layerbranch_id]))
-            distro.up_date =3D index.distros[id].updated
-            distro.name =3D index.distros[id].name
-            distro.description =3D index.distros[id].description
-            distro.save()
+            try:
+                distro, created =3D Distro.objects.get_or_create(
+                    name=3Dindex.distros[id].name,
+                    layer_version=3DLayer_Version.objects.get(
+                        pk=3Dli_layer_branch_id_to_toaster_lv_id[index.dis=
tros[id].layerbranch_id]))
+                distro.up_date =3D index.distros[id].updated
+                distro.name =3D index.distros[id].name
+                distro.description =3D index.distros[id].description
+                distro.save()
+            except KeyError as e:
+                logger.warning("Couldn't process %s (%s)" % (i, repr(e)))
+                continue
             self.mini_progress("distros", i, total)
=20
         # update machines
diff --git a/bitbake/lib/toaster/orm/models.py b/bitbake/lib/toaster/orm/mo=
dels.py
index 19c96862..e2f488ed 100644
--- a/bitbake/lib/toaster/orm/models.py
+++ b/bitbake/lib/toaster/orm/models.py
@@ -79,7 +79,6 @@ if 'sqlite' in settings.DATABASES['default']['ENGINE']:
     # end of HACK
=20
 class GitURLValidator(validators.URLValidator):
-    import re
     regex =3D re.compile(
         r'^(?:ssh|git|http|ftp)s?://'  # http:// or https://
         r'(?:(?:[A-Z0-9](?:[A-Z0-9-]{0,61}[A-Z0-9])?\.)+(?:[A-Z]{2,6}\.?|[=
A-Z0-9-]{2,}\.?)|'  # domain...
@@ -1500,7 +1499,7 @@ class Layer_Version(models.Model):
     # code lifted, with adaptations, from the layerindex-web application
     # https://git.yoctoproject.org/cgit/cgit.cgi/layerindex-web/
     def _handle_url_path(self, base_url, path):
-        import re, posixpath
+        import posixpath
         if base_url:
             if self.dirpath:
                 if path:
diff --git a/bitbake/lib/toaster/tests/browser/selenium_helpers_base.py b/b=
itbake/lib/toaster/tests/browser/selenium_helpers_base.py
index 393be754..ce108e2f 100644
--- a/bitbake/lib/toaster/tests/browser/selenium_helpers_base.py
+++ b/bitbake/lib/toaster/tests/browser/selenium_helpers_base.py
@@ -27,7 +27,9 @@ from selenium.webdriver.common.by import By
 from selenium.webdriver.common.desired_capabilities import DesiredCapabili=
ties
 from selenium.common.exceptions import NoSuchElementException, \
         StaleElementReferenceException, TimeoutException, \
-        SessionNotCreatedException
+        SessionNotCreatedException, WebDriverException, \
+        ElementNotInteractableException
+from selenium.webdriver.common.action_chains import ActionChains
=20
 def create_selenium_driver(cls,browser=3D'chrome'):
     # set default browser string based on env (if available)
@@ -90,7 +92,7 @@ class Wait(WebDriverWait):
     Subclass of WebDriverWait with predetermined timeout and poll
     frequency. Also deals with a wider variety of exceptions.
     """
-    _TIMEOUT =3D 10
+    _TIMEOUT =3D 20
     _POLL_FREQUENCY =3D 0.5
=20
     def __init__(self, driver, timeout=3D_TIMEOUT, poll=3D_POLL_FREQUENCY)=
:
@@ -98,7 +100,7 @@ class Wait(WebDriverWait):
         self._POLL_FREQUENCY =3D poll
         super(Wait, self).__init__(driver, self._TIMEOUT, self._POLL_FREQU=
ENCY)
=20
-    def until(self, method, message=3D''):
+    def until(self, method, message=3D'', scroll=3DNone):
         """
         Calls the method provided with the driver as an argument until the
         return value is not False.
@@ -107,13 +109,21 @@ class Wait(WebDriverWait):
         end_time =3D time.time() + self._timeout
         while True:
             try:
+                if scroll:
+                    scroll(self._driver)
                 value =3D method(self._driver)
                 if value:
                     return value
+
+            except ElementNotInteractableException:
+                pass
             except NoSuchElementException:
                 pass
             except StaleElementReferenceException:
                 pass
+            except WebDriverException:
+                # selenium.common.exceptions.WebDriverException: Message: =
unknown error: unhandled inspector error: {"code":-32000,"message":"Node wi=
th given id does not belong to the document"}
+                pass
=20
             time.sleep(self._poll)
             if time.time() > end_time:
@@ -183,7 +193,7 @@ class SeleniumTestCaseBase(unittest.TestCase):
         self.driver.get(abs_url)
=20
         try:  # Ensure page is loaded before proceeding
-            self.wait_until_visible("#global-nav", poll=3D3)
+            self.wait_until_visible("#global-nav")
         except NoSuchElementException:
             self.driver.implicitly_wait(3)
         except TimeoutException:
@@ -208,36 +218,59 @@ class SeleniumTestCaseBase(unittest.TestCase):
         """ Return the element which currently has focus on the page """
         return self.driver.switch_to.active_element
=20
-    def wait_until_present(self, selector, poll=3D0.5):
+    def wait_until_present(self, selector, timeout=3DWait._TIMEOUT):
         """ Wait until element matching CSS selector is on the page """
         is_present =3D lambda driver: self.find(selector)
         msg =3D 'An element matching "%s" should be on the page' % selecto=
r
-        element =3D Wait(self.driver, poll=3Dpoll).until(is_present, msg)
-        if poll > 2:
-            time.sleep(poll)  # element need more delay to be present
+        element =3D Wait(self.driver, timeout=3Dtimeout).until(is_present,=
 msg)
         return element
=20
-    def wait_until_visible(self, selector, poll=3D1):
+    def wait_until_element_visible(self, element, timeout=3DWait._TIMEOUT)=
:
+        """ Wait until element matching CSS selector is visible on the pag=
e """
+        is_visible =3D lambda driver: element.is_displayed()
+        msg =3D 'An element should be visible'
+        Wait(self.driver, timeout=3Dtimeout).until(is_visible, msg)
+        return
+
+    def wait_until_visible(self, selector, timeout=3DWait._TIMEOUT):
         """ Wait until element matching CSS selector is visible on the pag=
e """
         is_visible =3D lambda driver: self.find(selector).is_displayed()
         msg =3D 'An element matching "%s" should be visible' % selector
-        Wait(self.driver, poll=3Dpoll).until(is_visible, msg)
-        time.sleep(poll)  # wait for visibility to settle
+        Wait(self.driver, timeout=3Dtimeout).until(is_visible, msg)
         return self.find(selector)
=20
-    def wait_until_clickable(self, selector, poll=3D1):
+    def wait_until_not_visible(self, selector, timeout=3DWait._TIMEOUT):
+        """ Wait until element matching CSS selector is not visible on the=
 page """
+        is_visible =3D lambda driver: self.find(selector).is_displayed()
+        msg =3D 'An element matching "%s" should be visible' % selector
+        Wait(self.driver, timeout=3Dtimeout).until_not(is_visible, msg)
+        return self.find(selector)
+
+    def wait_until_clickable(self, selector, timeout=3DWait._TIMEOUT):
         """ Wait until element matching CSS selector is visible on the pag=
e """
-        WebDriverWait(
-            self.driver,
-            Wait._TIMEOUT,
-            poll_frequency=3Dpoll
-        ).until(
-            EC.element_to_be_clickable((By.ID, selector.removeprefix('#')
-                                        )
-                                       )
-        )
+        WebDriverWait(self.driver, timeout=3Dtimeout).until(lambda driver:=
 self.driver.execute_script("return jQuery.active =3D=3D 0"))
+        is_clickable =3D lambda driver: (self.find(selector).is_displayed(=
) and self.find(selector).is_enabled())
+        msg =3D 'An element matching "%s" should be clickable' % selector
+        Wait(self.driver, timeout=3Dtimeout).until(is_clickable, msg)
         return self.find(selector)
=20
+    def wait_until_finder_clickable(self, finder, timeout=3DWait._TIMEOUT)=
:
+        """ Wait until element is clickable """
+        WebDriverWait(self.driver, timeout=3Dtimeout).until(lambda driver:=
 self.driver.execute_script("return jQuery.active =3D=3D 0"))
+        is_clickable =3D lambda driver: (finder(driver).is_displayed() and=
 finder(driver).is_enabled())
+        msg =3D 'A matching element never became be clickable'
+        Wait(self.driver, timeout=3Dtimeout).until(is_clickable, msg)
+        return finder(self.driver)
+
+    def wait_until_element_clickable(self, element, timeout=3DWait._TIMEOU=
T):
+        """ Wait until element is clickable """
+        WebDriverWait(self.driver, timeout=3Dtimeout).until(lambda driver:=
 self.driver.execute_script("return jQuery.active =3D=3D 0"))
+        is_clickable =3D lambda driver: (element.is_displayed() and elemen=
t.is_enabled())
+        scroll =3D lambda driver: ActionChains(driver).move_to_element(ele=
ment).perform()
+        msg =3D 'A matching element never became be clickable'
+        Wait(self.driver, timeout=3Dtimeout).until(is_clickable, msg, scro=
ll=3Dscroll)
+        return
+
     def wait_until_focused(self, selector):
         """ Wait until element matching CSS selector has focus """
         is_focused =3D \
diff --git a/bitbake/lib/toaster/tests/browser/test_all_builds_page.py b/bi=
tbake/lib/toaster/tests/browser/test_all_builds_page.py
index b9356a03..9ab81fb1 100644
--- a/bitbake/lib/toaster/tests/browser/test_all_builds_page.py
+++ b/bitbake/lib/toaster/tests/browser/test_all_builds_page.py
@@ -200,6 +200,7 @@ class TestAllBuildsPage(SeleniumTestCase):
=20
         # should see a rebuild button for non-command-line builds
         self.wait_until_visible('#allbuildstable tbody tr')
+        self.wait_until_visible('.rebuild-btn')
         selector =3D 'div[data-latest-build-result=3D"%s"] .rebuild-btn' %=
 build1.id
         run_again_button =3D self.find_all(selector)
         self.assertEqual(len(run_again_button), 1,
@@ -224,7 +225,7 @@ class TestAllBuildsPage(SeleniumTestCase):
=20
         url =3D reverse('all-builds')
         self.get(url)
-        self.wait_until_visible('#allbuildstable', poll=3D3)
+        self.wait_until_visible('#allbuildstable')
=20
         # get the project name cells from the table
         cells =3D self.find_all('#allbuildstable td[class=3D"project"]')
@@ -257,7 +258,7 @@ class TestAllBuildsPage(SeleniumTestCase):
=20
         url =3D reverse('all-builds')
         self.get(url)
-        self.wait_until_visible('#allbuildstable', poll=3D3)
+        self.wait_until_visible('#allbuildstable')
=20
         # test recent builds area for successful build
         element =3D self._get_build_time_element(build1)
@@ -452,7 +453,7 @@ class TestAllBuildsPage(SeleniumTestCase):
         def test_show_rows(row_to_show, show_row_link):
             # Check that we can show rows =3D=3D row_to_show
             show_row_link.select_by_value(str(row_to_show))
-            self.wait_until_visible('#allbuildstable tbody tr', poll=3D3)
+            self.wait_until_visible('#allbuildstable tbody tr')
             # check at least some rows are visible
             self.assertTrue(
                 len(self.find_all('#allbuildstable tbody tr')) > 0
diff --git a/bitbake/lib/toaster/tests/browser/test_all_projects_page.py b/=
bitbake/lib/toaster/tests/browser/test_all_projects_page.py
index 9ed1901c..05e12892 100644
--- a/bitbake/lib/toaster/tests/browser/test_all_projects_page.py
+++ b/bitbake/lib/toaster/tests/browser/test_all_projects_page.py
@@ -81,7 +81,7 @@ class TestAllProjectsPage(SeleniumTestCase):
=20
     def _get_row_for_project(self, project_name):
         """ Get the HTML row for a project, or None if not found """
-        self.wait_until_visible('#projectstable tbody tr', poll=3D3)
+        self.wait_until_visible('#projectstable tbody tr')
         rows =3D self.find_all('#projectstable tbody tr')
=20
         # find the row with a project name matching the one supplied
@@ -236,7 +236,7 @@ class TestAllProjectsPage(SeleniumTestCase):
         self.get(url)
=20
         # Chseck search box is present and works
-        self.wait_until_visible('#projectstable tbody tr', poll=3D3)
+        self.wait_until_visible('#projectstable tbody tr')
         search_box =3D self.find('#search-input-projectstable')
         self.assertTrue(search_box.is_displayed())
=20
@@ -244,7 +244,7 @@ class TestAllProjectsPage(SeleniumTestCase):
         search_box.send_keys('test project 10')
         search_btn =3D self.find('#search-submit-projectstable')
         search_btn.click()
-        self.wait_until_visible('#projectstable tbody tr', poll=3D3)
+        self.wait_until_visible('#projectstable tbody tr')
         rows =3D self.find_all('#projectstable tbody tr')
         self.assertTrue(len(rows) =3D=3D 1)
=20
@@ -290,7 +290,7 @@ class TestAllProjectsPage(SeleniumTestCase):
                 )
         url =3D reverse('all-projects')
         self.get(url)
-        self.wait_until_visible('#projectstable tbody tr', poll=3D3)
+        self.wait_until_visible('#projectstable tbody tr')
=20
         # Check edit column
         edit_column =3D self.find('#edit-columns-button')
@@ -313,7 +313,7 @@ class TestAllProjectsPage(SeleniumTestCase):
         def test_show_rows(row_to_show, show_row_link):
             # Check that we can show rows =3D=3D row_to_show
             show_row_link.select_by_value(str(row_to_show))
-            self.wait_until_visible('#projectstable tbody tr', poll=3D3)
+            self.wait_until_visible('#projectstable tbody tr')
             # check at least some rows are visible
             self.assertTrue(
                 len(self.find_all('#projectstable tbody tr')) > 0
@@ -321,7 +321,7 @@ class TestAllProjectsPage(SeleniumTestCase):
=20
         url =3D reverse('all-projects')
         self.get(url)
-        self.wait_until_visible('#projectstable tbody tr', poll=3D3)
+        self.wait_until_visible('#projectstable tbody tr')
=20
         show_rows =3D self.driver.find_elements(
             By.XPATH,
diff --git a/bitbake/lib/toaster/tests/browser/test_builddashboard_page.py =
b/bitbake/lib/toaster/tests/browser/test_builddashboard_page.py
index d838ce36..82367108 100644
--- a/bitbake/lib/toaster/tests/browser/test_builddashboard_page.py
+++ b/bitbake/lib/toaster/tests/browser/test_builddashboard_page.py
@@ -162,7 +162,7 @@ class TestBuildDashboardPage(SeleniumTestCase):
         """
         url =3D reverse('builddashboard', args=3D(build.id,))
         self.get(url)
-        self.wait_until_visible('#global-nav', poll=3D3)
+        self.wait_until_visible('#global-nav')
=20
     def _get_build_dashboard_errors(self, build):
         """
diff --git a/bitbake/lib/toaster/tests/browser/test_landing_page.py b/bitba=
ke/lib/toaster/tests/browser/test_landing_page.py
index 8fe5fea4..210359d5 100644
--- a/bitbake/lib/toaster/tests/browser/test_landing_page.py
+++ b/bitbake/lib/toaster/tests/browser/test_landing_page.py
@@ -34,6 +34,7 @@ class TestLandingPage(SeleniumTestCase):
     def test_icon_info_visible_and_clickable(self):
         """ Test that the information icon is visible and clickable """
         self.get(reverse('landing'))
+        self.wait_until_visible('#toaster-version-info-sign')
         info_sign =3D self.find('#toaster-version-info-sign')
=20
         # check that the info sign is visible
@@ -43,6 +44,7 @@ class TestLandingPage(SeleniumTestCase):
         # and info modal is appearing when clicking on the info sign
         info_sign.click()  # click on the info sign make attribute 'aria-d=
escribedby' visible
         info_model_id =3D info_sign.get_attribute('aria-describedby')
+        self.wait_until_visible(f'#{info_model_id}')
         info_modal =3D self.find(f'#{info_model_id}')
         self.assertTrue(info_modal.is_displayed())
         self.assertTrue("Toaster version information" in info_modal.text)
@@ -50,6 +52,7 @@ class TestLandingPage(SeleniumTestCase):
     def test_documentation_link_displayed(self):
         """ Test that the documentation link is displayed """
         self.get(reverse('landing'))
+        self.wait_until_visible('#navbar-docs')
         documentation_link =3D self.find('#navbar-docs > a')
=20
         # check that the documentation link is visible
@@ -65,6 +68,7 @@ class TestLandingPage(SeleniumTestCase):
     def test_openembedded_jumbotron_link_visible_and_clickable(self):
         """ Test OpenEmbedded link jumbotron is visible and clickable: """
         self.get(reverse('landing'))
+        self.wait_until_visible('.jumbotron')
         jumbotron =3D self.find('.jumbotron')
=20
         # check OpenEmbedded
@@ -76,6 +80,7 @@ class TestLandingPage(SeleniumTestCase):
     def test_bitbake_jumbotron_link_visible_and_clickable(self):
         """ Test BitBake link jumbotron is visible and clickable: """
         self.get(reverse('landing'))
+        self.wait_until_visible('.jumbotron')
         jumbotron =3D self.find('.jumbotron')
=20
         # check BitBake
@@ -88,6 +93,7 @@ class TestLandingPage(SeleniumTestCase):
     def test_yoctoproject_jumbotron_link_visible_and_clickable(self):
         """ Test Yocto Project link jumbotron is visible and clickable: ""=
"
         self.get(reverse('landing'))
+        self.wait_until_visible('.jumbotron')
         jumbotron =3D self.find('.jumbotron')
=20
         # check Yocto Project
@@ -101,6 +107,7 @@ class TestLandingPage(SeleniumTestCase):
             if visible and clickable
         """
         self.get(reverse('landing'))
+        self.wait_until_visible('.jumbotron')
         jumbotron =3D self.find('.jumbotron')
=20
         # check Big magenta button
@@ -119,6 +126,7 @@ class TestLandingPage(SeleniumTestCase):
         Layer_Version.objects.create(layer=3Dlayer)
=20
         self.get(reverse('landing'))
+        self.wait_until_visible('.jumbotron')
         jumbotron =3D self.find('.jumbotron')
=20
         # check Big Blue button
@@ -132,6 +140,7 @@ class TestLandingPage(SeleniumTestCase):
     def test_toaster_manual_link_visible_and_clickable(self):
         """ Test Read the Toaster manual link jumbotron is visible and cli=
ckable: """
         self.get(reverse('landing'))
+        self.wait_until_visible('.jumbotron')
         jumbotron =3D self.find('.jumbotron')
=20
         # check Read the Toaster manual
@@ -145,6 +154,7 @@ class TestLandingPage(SeleniumTestCase):
     def test_contrib_to_toaster_link_visible_and_clickable(self):
         """ Test Contribute to Toaster link jumbotron is visible and click=
able: """
         self.get(reverse('landing'))
+        self.wait_until_visible('.jumbotron')
         jumbotron =3D self.find('.jumbotron')
=20
         # check Contribute to Toaster
@@ -161,6 +171,7 @@ class TestLandingPage(SeleniumTestCase):
         =3D> should see the landing page
         """
         self.get(reverse('landing'))
+        self.wait_until_visible('.jumbotron')
         self.assertTrue(self.LANDING_PAGE_TITLE in self.get_page_source())
=20
     def test_default_project_has_build(self):
@@ -193,6 +204,7 @@ class TestLandingPage(SeleniumTestCase):
         user_project.save()
=20
         self.get(reverse('landing'))
+        self.wait_until_visible('#projectstable')
=20
         elements =3D self.find_all('#projectstable')
         self.assertEqual(len(elements), 1, 'should redirect to projects')
@@ -213,7 +225,7 @@ class TestLandingPage(SeleniumTestCase):
=20
         self.get(reverse('landing'))
=20
-        self.wait_until_visible("#latest-builds", poll=3D3)
+        self.wait_until_visible("#latest-builds")
         elements =3D self.find_all('#allbuildstable')
         self.assertEqual(len(elements), 1, 'should redirect to builds')
         content =3D self.get_page_source()
diff --git a/bitbake/lib/toaster/tests/browser/test_layerdetails_page.py b/=
bitbake/lib/toaster/tests/browser/test_layerdetails_page.py
index 5c29548b..476a8fa5 100644
--- a/bitbake/lib/toaster/tests/browser/test_layerdetails_page.py
+++ b/bitbake/lib/toaster/tests/browser/test_layerdetails_page.py
@@ -64,7 +64,7 @@ class TestLayerDetailsPage(SeleniumTestCase):
                            args=3D(self.project.pk,
                                  self.imported_layer_version.pk))
=20
-    def _edit_layerdetails(self):
+    def test_edit_layerdetails_page(self):
         """ Edit all the editable fields for the layer refresh the page an=
d
         check that the new values exist"""
=20
@@ -78,17 +78,20 @@ class TestLayerDetailsPage(SeleniumTestCase):
         self.wait_until_visible("#layer-git-repo-url")
=20
         # Open every edit box
-        for btn in self.find_all("dd .glyphicon-edit"):
-            btn.click()
+        self.wait_until_element_clickable(self.find("#edit-layer-descripti=
on"))
+        self.click("#edit-layer-description")
+        self.wait_until_element_clickable(self.find("#edit-layer-summary")=
)
+        self.click("#edit-layer-summary")
=20
         # Wait for the inputs to become visible after animation
-        self.wait_until_visible("#layer-git input[type=3Dtext]")
-        self.wait_until_visible("dd textarea")
+        self.wait_until_visible("#layer-git-repo-url")
         self.wait_until_visible("dd .change-btn")
+        self.wait_until_element_clickable(self.find("#layer-description"))
+        self.wait_until_element_clickable(self.find("#layer-summary"))
=20
         # Edit each value
         for inputs in self.find_all("#layer-git input[type=3Dtext]") + \
-                self.find_all("dd textarea"):
+                [self.find("#layer-summary"), self.find("#layer-descriptio=
n")]:
             # ignore the tt inputs (twitter typeahead input)
             if "tt-" in inputs.get_attribute("class"):
                 continue
@@ -100,24 +103,19 @@ class TestLayerDetailsPage(SeleniumTestCase):
                             (self.initial_values, value))
=20
             # Make sure the input visible beofre sending keys
-            self.wait_until_visible("#layer-git input[type=3Dtext]")
+            self.wait_until_element_clickable(inputs)
             inputs.send_keys("-edited")
=20
         # Save the new values
         for save_btn in self.find_all(".change-btn"):
             save_btn.click()
=20
-        try:
-            self.wait_until_visible("#save-changes-for-switch", poll=3D3)
-            btn_save_chg_for_switch =3D self.wait_until_clickable(
-                "#save-changes-for-switch", poll=3D3)
-            btn_save_chg_for_switch.click()
-        except ElementClickInterceptedException:
-            self.skipTest(
-                "save-changes-for-switch click intercepted. Element not vi=
sible or maybe covered by another element.")
-        except TimeoutException:
-            self.skipTest(
-                "save-changes-for-switch is not clickable within the speci=
fied timeout.")
+        self.wait_until_visible("#save-changes-for-switch")
+        # Ensure scrolled into view
+        self.driver.execute_script('window.scrollTo({behavior: "instant", =
top: 0, left: 0})')
+        btn_save_chg_for_switch =3D self.wait_until_clickable(
+            "#save-changes-for-switch")
+        btn_save_chg_for_switch.click()
=20
         self.wait_until_visible("#edit-layer-source")
=20
@@ -147,17 +145,10 @@ class TestLayerDetailsPage(SeleniumTestCase):
         new_dir =3D "/home/test/my-meta-dir"
         dir_input.send_keys(new_dir)
=20
-        try:
-            self.wait_until_visible("#save-changes-for-switch", poll=3D3)
-            btn_save_chg_for_switch =3D self.wait_until_clickable(
-                "#save-changes-for-switch", poll=3D3)
-            btn_save_chg_for_switch.click()
-        except ElementClickInterceptedException:
-            self.skipTest(
-                "save-changes-for-switch click intercepted. Element not pr=
operly visible or maybe behind another element.")
-        except TimeoutException:
-            self.skipTest(
-                "save-changes-for-switch is not clickable within the speci=
fied timeout.")
+        self.wait_until_visible("#save-changes-for-switch")
+        btn_save_chg_for_switch =3D self.wait_until_clickable(
+            "#save-changes-for-switch")
+        btn_save_chg_for_switch.click()
=20
         self.wait_until_visible("#edit-layer-source")
=20
@@ -168,12 +159,6 @@ class TestLayerDetailsPage(SeleniumTestCase):
                         "Expected %s in the dir value for layer directory"=
 %
                         new_dir)
=20
-    def test_edit_layerdetails_page(self):
-        try:
-            self._edit_layerdetails()
-        except ElementClickInterceptedException:
-            self.skipTest(
-                "ElementClickInterceptedException occured. Element not vis=
ible or maybe covered by another element.")
=20
     def test_delete_layer(self):
         """ Delete the layer """
@@ -211,6 +196,7 @@ class TestLayerDetailsPage(SeleniumTestCase):
         self.get(self.url)
=20
         # Add the layer
+        self.wait_until_clickable("#add-remove-layer-btn")
         self.click("#add-remove-layer-btn")
=20
         notification =3D self.wait_until_visible("#change-notification-msg=
")
@@ -218,12 +204,17 @@ class TestLayerDetailsPage(SeleniumTestCase):
         expected_text =3D "You have added 1 layer to your project: %s" % \
             self.imported_layer_version.layer.name
=20
-        self.assertTrue(expected_text in notification.text,
+        self.assertIn(expected_text, notification.text,
                         "Expected notification text %s not found was "
                         " \"%s\" instead" %
                         (expected_text, notification.text))
=20
+        hide_button =3D self.find('#hide-alert')
+        hide_button.click()
+        self.wait_until_not_visible('#change-notification')
+
         # Remove the layer
+        self.wait_until_clickable("#add-remove-layer-btn")
         self.click("#add-remove-layer-btn")
=20
         notification =3D self.wait_until_visible("#change-notification-msg=
")
@@ -231,7 +222,7 @@ class TestLayerDetailsPage(SeleniumTestCase):
         expected_text =3D "You have removed 1 layer from your project: %s"=
 % \
             self.imported_layer_version.layer.name
=20
-        self.assertTrue(expected_text in notification.text,
+        self.assertIn(expected_text, notification.text,
                         "Expected notification text %s not found was "
                         " \"%s\" instead" %
                         (expected_text, notification.text))
diff --git a/bitbake/lib/toaster/tests/browser/test_new_custom_image_page.p=
y b/bitbake/lib/toaster/tests/browser/test_new_custom_image_page.py
index 9f0b6397..bf0304db 100644
--- a/bitbake/lib/toaster/tests/browser/test_new_custom_image_page.py
+++ b/bitbake/lib/toaster/tests/browser/test_new_custom_image_page.py
@@ -90,7 +90,7 @@ class TestNewCustomImagePage(SeleniumTestCase):
         """
         url =3D reverse('newcustomimage', args=3D(self.project.id,))
         self.get(url)
-        self.wait_until_visible('#global-nav', poll=3D3)
+        self.wait_until_visible('#global-nav')
=20
         self.click('button[data-recipe=3D"%s"]' % self.recipe.id)
=20
diff --git a/bitbake/lib/toaster/tests/browser/test_new_project_page.py b/b=
itbake/lib/toaster/tests/browser/test_new_project_page.py
index 458bb653..e50f236c 100644
--- a/bitbake/lib/toaster/tests/browser/test_new_project_page.py
+++ b/bitbake/lib/toaster/tests/browser/test_new_project_page.py
@@ -47,7 +47,7 @@ class TestNewProjectPage(SeleniumTestCase):
=20
         url =3D reverse('newproject')
         self.get(url)
-        self.wait_until_visible('#new-project-name', poll=3D3)
+        self.wait_until_visible('#new-project-name')
         self.enter_text('#new-project-name', project_name)
=20
         select =3D Select(self.find('#projectversion'))
@@ -58,7 +58,7 @@ class TestNewProjectPage(SeleniumTestCase):
         # We should get redirected to the new project's page with the
         # notification at the top
         element =3D self.wait_until_visible(
-            '#project-created-notification', poll=3D3)
+            '#project-created-notification')
=20
         self.assertTrue(project_name in element.text,
                         "New project name not in new project notification"=
)
@@ -79,7 +79,7 @@ class TestNewProjectPage(SeleniumTestCase):
=20
         url =3D reverse('newproject')
         self.get(url)
-        self.wait_until_visible('#new-project-name', poll=3D3)
+        self.wait_until_visible('#new-project-name')
=20
         self.enter_text('#new-project-name', project_name)
=20
@@ -89,12 +89,10 @@ class TestNewProjectPage(SeleniumTestCase):
         radio =3D self.driver.find_element(By.ID, 'type-new')
         radio.click()
=20
-        self.click("#create-project-button")
-
-        self.wait_until_present('#hint-error-project-name', poll=3D3)
+        self.wait_until_visible('#hint-error-project-name')
         element =3D self.find('#hint-error-project-name')
=20
-        self.assertTrue(("Project names must be unique" in element.text),
+        self.assertIn("Project names must be unique", element.text,
                         "Did not find unique project name error message")
=20
         # Try and click it anyway, if it submits we'll have a new project =
in
diff --git a/bitbake/lib/toaster/tests/builds/buildtest.py b/bitbake/lib/to=
aster/tests/builds/buildtest.py
index cacfccd4..e54d5613 100644
--- a/bitbake/lib/toaster/tests/builds/buildtest.py
+++ b/bitbake/lib/toaster/tests/builds/buildtest.py
@@ -128,7 +128,7 @@ class BuildTest(unittest.TestCase):
         if os.environ.get("TOASTER_TEST_USE_SSTATE_MIRROR"):
             ProjectVariable.objects.get_or_create(
                 name=3D"SSTATE_MIRRORS",
-                value=3D"file://.* http://cdn.jsdelivr.net/yocto/sstate/al=
l/PATH;downloadfilename=3DPATH",
+                value=3D"file://.* http://sstate.yoctoproject.org/all/PATH=
;downloadfilename=3DPATH",
                 project=3Dproject)
=20
         ProjectTarget.objects.create(project=3Dproject,
diff --git a/bitbake/lib/toaster/tests/functional/functional_helpers.py b/b=
itbake/lib/toaster/tests/functional/functional_helpers.py
index 7c20437d..9bcaac61 100644
--- a/bitbake/lib/toaster/tests/functional/functional_helpers.py
+++ b/bitbake/lib/toaster/tests/functional/functional_helpers.py
@@ -12,14 +12,19 @@ import logging
 import subprocess
 import signal
 import re
+import requests
=20
+from django.urls import reverse
 from tests.browser.selenium_helpers_base import SeleniumTestCaseBase
 from selenium.webdriver.common.by import By
+from selenium.webdriver.support.select import Select
 from selenium.common.exceptions import NoSuchElementException
=20
 logger =3D logging.getLogger("toaster")
 toaster_processes =3D []
=20
+MASTER_PROJECT_ID =3D '4'
+
 class SeleniumFunctionalTestCase(SeleniumTestCaseBase):
     wait_toaster_time =3D 10
=20
@@ -136,3 +141,86 @@ class SeleniumFunctionalTestCase(SeleniumTestCaseBase)=
:
         except NoSuchElementException:
             return False
         return element
+
+    def create_new_project(
+        self,
+        project_name,
+        release,
+        release_title,
+        merge_toaster_settings,
+    ):
+        """ Create/Test new project using:
+          - Project Name: Any string
+          - Release: Any string
+          - Merge Toaster settings: True or False
+        """
+
+        # Obtain a CSRF token from a suitable URL
+        projs =3D requests.get(self.live_server_url + reverse('newproject'=
))
+        csrftoken =3D projs.cookies.get('csrftoken')
+
+        # Use the projects typeahead to find out if the project already ex=
ists
+        req =3D requests.get(self.live_server_url + reverse('xhr_projectst=
ypeahead'), {'search': project_name, 'format' : 'json'})
+        data =3D req.json()
+        # Delete any existing projects
+        for result in data['results']:
+            del_url =3D reverse('xhr_project', args=3D(result['id'],))
+            del_response =3D requests.delete(self.live_server_url + del_ur=
l, cookies=3D{'csrftoken': csrftoken}, headers=3D{'X-CSRFToken': csrftoken}=
)
+            self.assertEqual(del_response.status_code, 200)
+
+        self.get(reverse('newproject'))
+        self.wait_until_visible('#new-project-name')
+        self.driver.find_element(By.ID,
+                                 "new-project-name").send_keys(project_nam=
e)
+
+        select =3D Select(self.find('#projectversion'))
+        select.select_by_value(release)
+
+        # check merge toaster settings
+        checkbox =3D self.find('.checkbox-mergeattr')
+        if merge_toaster_settings:
+            if not checkbox.is_selected():
+                checkbox.click()
+        else:
+            if checkbox.is_selected():
+                checkbox.click()
+
+        self.wait_until_clickable('#create-project-button')
+
+        self.driver.find_element(By.ID, "create-project-button").click()
+
+        element =3D self.wait_until_visible('#project-created-notification=
')
+        self.assertTrue(
+            self.element_exists('#project-created-notification'),
+            f"Project:{project_name} creation notification not shown"
+        )
+        self.assertTrue(
+            project_name in element.text,
+            f"New project name:{project_name} not in new project notificat=
ion"
+        )
+
+        # Use the projects typeahead again to check the project now exists
+        req =3D requests.get(self.live_server_url + reverse('xhr_projectst=
ypeahead'), {'search': project_name, 'format' : 'json'})
+        data =3D req.json()
+        self.assertGreater(len(data['results']), 0, f"New project:{project=
_name} not found in database")
+
+        project_id =3D data['results'][0]['id']
+
+        self.wait_until_visible('#project-release-title')
+
+        # check release
+        if release_title is not None:
+            self.assertTrue(re.search(
+                release_title,
+                self.driver.find_element(By.XPATH,
+                                     "//span[@id=3D'project-release-title'=
]"
+                                     ).text),
+                            'The project release is not defined')
+
+        return project_id
+       =20
+    def load_projects_page_helper(self):       =20
+        self.wait_until_present('#projectstable')
+        # Need to wait for some data in the table too
+        self.wait_until_present('td[class=3D"updated"]')
+
diff --git a/bitbake/lib/toaster/tests/functional/test_create_new_project.p=
y b/bitbake/lib/toaster/tests/functional/test_create_new_project.py
index 94d90459..294b6bc9 100644
--- a/bitbake/lib/toaster/tests/functional/test_create_new_project.py
+++ b/bitbake/lib/toaster/tests/functional/test_create_new_project.py
@@ -11,125 +11,84 @@ import pytest
 from django.urls import reverse
 from selenium.webdriver.support.select import Select
 from tests.functional.functional_helpers import SeleniumFunctionalTestCase
-from orm.models import Project
 from selenium.webdriver.common.by import By
=20
-
-@pytest.mark.django_db
-@pytest.mark.order("last")
 class TestCreateNewProject(SeleniumFunctionalTestCase):
=20
-    def _create_test_new_project(
-        self,
-        project_name,
-        release,
-        release_title,
-        merge_toaster_settings,
-    ):
-        """ Create/Test new project using:
+    def test_create_new_project_blacksail(self):
+        """ Test create new project using:
           - Project Name: Any string
-          - Release: Any string
-          - Merge Toaster settings: True or False
+          - Release: Yocto Project 6.1 "Blacksail" (option value: 1)
+          - Merge Toaster settings: True
         """
-        self.get(reverse('newproject'))
-        self.wait_until_visible('#new-project-name', poll=3D3)
-        self.driver.find_element(By.ID,
-                                 "new-project-name").send_keys(project_nam=
e)
-
-        select =3D Select(self.find('#projectversion'))
-        select.select_by_value(release)
-
-        # check merge toaster settings
-        checkbox =3D self.find('.checkbox-mergeattr')
-        if merge_toaster_settings:
-            if not checkbox.is_selected():
-                checkbox.click()
-        else:
-            if checkbox.is_selected():
-                checkbox.click()
-
-        self.driver.find_element(By.ID, "create-project-button").click()
-
-        element =3D self.wait_until_visible('#project-created-notification=
', poll=3D3)
-        self.assertTrue(
-            self.element_exists('#project-created-notification'),
-            f"Project:{project_name} creation notification not shown"
-        )
-        self.assertTrue(
-            project_name in element.text,
-            f"New project name:{project_name} not in new project notificat=
ion"
-        )
-        self.assertTrue(
-            Project.objects.filter(name=3Dproject_name).count(),
-            f"New project:{project_name} not found in database"
-        )
-
-        # check release
-        self.assertTrue(re.search(
+        release =3D '1'
+        release_title =3D 'Yocto Project 6.1 "Blacksail"'
+        project_name =3D 'projectblacksail'
+        self.create_new_project(
+            project_name,
+            release,
             release_title,
-            self.driver.find_element(By.XPATH,
-                                     "//span[@id=3D'project-release-title'=
]"
-                                     ).text),
-                        'The project release is not defined')
+            True,
+        )
=20
-    def test_create_new_project_master(self):
+    def test_create_new_project_wrynose(self):
         """ Test create new project using:
           - Project Name: Any string
-          - Release: Yocto Project master (option value: 3)
-          - Merge Toaster settings: False
+          - Release: Yocto Project 6.0 "Wrynose" (option value: 2)
+          - Merge Toaster settings: True
         """
-        release =3D '3'
-        release_title =3D 'Yocto Project master'
-        project_name =3D 'projectmaster'
-        self._create_test_new_project(
+        release =3D '2'
+        release_title =3D 'Yocto Project 6.0 "Wrynose"'
+        project_name =3D 'projectwrynose'
+        self.create_new_project(
             project_name,
             release,
             release_title,
-            False,
+            True,
         )
=20
-    def test_create_new_project_kirkstone(self):
+    def test_create_new_project_local(self):
         """ Test create new project using:
           - Project Name: Any string
-          - Release: Yocto Project 4.0 "Kirkstone" (option value: 1)
+          - Release: Yocto Project "Local" (option value: 3)
           - Merge Toaster settings: True
         """
-        release =3D '1'
-        release_title =3D 'Yocto Project 4.0 "Kirkstone"'
-        project_name =3D 'projectkirkstone'
-        self._create_test_new_project(
+        release =3D '3'
+        release_title =3D 'Local Yocto Project'
+        project_name =3D 'projectlocal'
+        self.create_new_project(
             project_name,
             release,
             release_title,
             True,
         )
=20
-    def test_create_new_project_dunfell(self):
+    def test_create_new_project_master(self):
         """ Test create new project using:
           - Project Name: Any string
-          - Release: Yocto Project 3.1 "Dunfell" (option value: 5)
+          - Release: Yocto Project "Master" (option value: 4)
           - Merge Toaster settings: False
         """
-        release =3D '5'
-        release_title =3D 'Yocto Project 3.1 "Dunfell"'
-        project_name =3D 'projectdunfell'
-        self._create_test_new_project(
+        release =3D '4'
+        release_title =3D 'Yocto Project master'
+        project_name =3D 'projectmaster'
+        self.create_new_project(
             project_name,
             release,
             release_title,
             False,
         )
=20
-    def test_create_new_project_local(self):
+    def test_create_new_project_scarthgap(self):
         """ Test create new project using:
           - Project Name: Any string
-          - Release: Local Yocto Project (option value: 2)
+          - Release: Yocto Project 5.0 "Scarthgap" (option value: 5)
           - Merge Toaster settings: True
         """
-        release =3D '2'
-        release_title =3D 'Local Yocto Project'
-        project_name =3D 'projectlocal'
-        self._create_test_new_project(
+        release =3D '5'
+        release_title =3D 'Yocto Project 5.0 "Scarthgap"'
+        project_name =3D 'projectscarthgap'
+        self.create_new_project(
             project_name,
             release,
             release_title,
@@ -172,8 +131,10 @@ class TestCreateNewProject(SeleniumFunctionalTestCase)=
:
                                  "import-project-dir").send_keys(wrong_pat=
h)
         self.driver.find_element(By.ID, "create-project-button").click()
=20
+        self.wait_until_visible('.alert-danger')
+
         # check error message
         self.assertTrue(self.element_exists('.alert-danger'),
-                        'Allert message not shown')
+                        'Alert message not shown')
         self.assertTrue(wrong_path in self.find('.alert-danger').text,
                         "Wrong path not in alert message")
diff --git a/bitbake/lib/toaster/tests/functional/test_functional_basic.py =
b/bitbake/lib/toaster/tests/functional/test_functional_basic.py
index e4070fbb..71999ccf 100644
--- a/bitbake/lib/toaster/tests/functional/test_functional_basic.py
+++ b/bitbake/lib/toaster/tests/functional/test_functional_basic.py
@@ -10,152 +10,139 @@
 import re
 from django.urls import reverse
 import pytest
-from tests.functional.functional_helpers import SeleniumFunctionalTestCase
+from tests.functional.functional_helpers import SeleniumFunctionalTestCase=
, MASTER_PROJECT_ID
 from orm.models import Project
 from selenium.webdriver.common.by import By
=20
 from tests.functional.utils import get_projectId_from_url
=20
=20
-@pytest.mark.django_db
-@pytest.mark.order("second_to_last")
 class FuntionalTestBasic(SeleniumFunctionalTestCase):
     """Basic functional tests for Toaster"""
     project_id =3D None
+    project_url =3D None
=20
     def setUp(self):
         super(FuntionalTestBasic, self).setUp()
         if not FuntionalTestBasic.project_id:
-            self._create_slenium_project()
-            current_url =3D self.driver.current_url
-            FuntionalTestBasic.project_id =3D get_projectId_from_url(curre=
nt_url)
-
-#   testcase (1514)
-    def _create_slenium_project(self):
-        project_name =3D 'selenium-project'
-        self.get(reverse('newproject'))
-        self.wait_until_visible('#new-project-name', poll=3D3)
-        self.driver.find_element(By.ID, "new-project-name").send_keys(proj=
ect_name)
-        self.driver.find_element(By.ID, 'projectversion').click()
-        self.driver.find_element(By.ID, "create-project-button").click()
-        element =3D self.wait_until_visible('#project-created-notification=
', poll=3D10)
-        self.assertTrue(self.element_exists('#project-created-notification=
'),'Project creation notification not shown')
-        self.assertTrue(project_name in element.text,
-                        "New project name not in new project notification"=
)
-        self.assertTrue(Project.objects.filter(name=3Dproject_name).count(=
),
-                        "New project not found in database")
-        return Project.objects.last().id
+            FuntionalTestBasic.project_id =3D self.create_new_project('sel=
enium-project', MASTER_PROJECT_ID, None, False)
=20
  #  testcase (1515)
     def test_verify_left_bar_menu(self):
         self.get(reverse('all-projects'))
-        self.wait_until_present('#projectstable', poll=3D10)
+        self.load_projects_page_helper()
         self.find_element_by_link_text_in_table('projectstable', 'selenium=
-project').click()
-        self.wait_until_present('#config-nav', poll=3D10)
+        self.wait_until_present('#config-nav')
         self.assertTrue(self.element_exists('#config-nav'),'Configuration =
Tab does not exist')
         project_URL=3Dself.get_URL()
         self.driver.find_element(By.XPATH, '//a[@href=3D"'+project_URL+'"]=
').click()
-        self.wait_until_present('#config-nav', poll=3D10)
=20
         try:
+            self.wait_until_present('#config-nav')
             self.driver.find_element(By.XPATH, "//*[@id=3D'config-nav']/ul=
/li/a[@href=3D"+'"'+project_URL+'customimages/"'+"]").click()
-            self.wait_until_present('#config-nav', poll=3D10)
-            self.assertTrue(re.search("Custom images",self.driver.find_ele=
ment(By.XPATH, "//div[@class=3D'col-md-10']").text),'Custom images informat=
ion is not loading properly')
+            self.wait_until_present('#filter-modal-customimagestable')
         except:
             self.fail(msg=3D'No Custom images tab available')
+        self.assertTrue(re.search("Custom images",self.driver.find_element=
(By.XPATH, "//div[@class=3D'col-md-10']").text),'Custom images information =
is not loading properly')
=20
         try:
             self.driver.find_element(By.XPATH, "//*[@id=3D'config-nav']/ul=
/li/a[@href=3D"+'"'+project_URL+'images/"'+"]").click()
-            self.wait_until_present('#config-nav', poll=3D10)
-            self.assertTrue(re.search("Compatible image recipes",self.driv=
er.find_element(By.XPATH, "//div[@class=3D'col-md-10']").text),'The Compati=
ble image recipes information is not loading properly')
+            self.wait_until_present('#filter-modal-imagerecipestable')
         except:
             self.fail(msg=3D'No Compatible image tab available')
+        self.assertTrue(re.search("Compatible image recipes",self.driver.f=
ind_element(By.XPATH, "//div[@class=3D'col-md-10']").text),'The Compatible =
image recipes information is not loading properly')
=20
         try:
             self.driver.find_element(By.XPATH, "//*[@id=3D'config-nav']/ul=
/li/a[@href=3D"+'"'+project_URL+'softwarerecipes/"'+"]").click()
-            self.wait_until_present('#config-nav', poll=3D10)
-            self.assertTrue(re.search("Compatible software recipes",self.d=
river.find_element(By.XPATH, "//div[@class=3D'col-md-10']").text),'The Comp=
atible software recipe information is not loading properly')
+            self.wait_until_present('#filter-modal-softwarerecipestable')
         except:
             self.fail(msg=3D'No Compatible software recipe tab available')
+        self.assertTrue(re.search("Compatible software recipes",self.drive=
r.find_element(By.XPATH, "//div[@class=3D'col-md-10']").text),'The Compatib=
le software recipe information is not loading properly')
=20
         try:
             self.driver.find_element(By.XPATH, "//*[@id=3D'config-nav']/ul=
/li/a[@href=3D"+'"'+project_URL+'machines/"'+"]").click()
-            self.wait_until_present('#config-nav', poll=3D10)
-            self.assertTrue(re.search("Compatible machines",self.driver.fi=
nd_element(By.XPATH, "//div[@class=3D'col-md-10']").text),'The Compatible m=
achine information is not loading properly')
+            self.wait_until_present('#filter-modal-machinestable')
         except:
             self.fail(msg=3D'No Compatible machines tab available')
+        self.assertTrue(re.search("Compatible machines",self.driver.find_e=
lement(By.XPATH, "//div[@class=3D'col-md-10']").text),'The Compatible machi=
ne information is not loading properly')
=20
         try:
             self.driver.find_element(By.XPATH, "//*[@id=3D'config-nav']/ul=
/li/a[@href=3D"+'"'+project_URL+'layers/"'+"]").click()
-            self.wait_until_present('#config-nav', poll=3D10)
-            self.assertTrue(re.search("Compatible layers",self.driver.find=
_element(By.XPATH, "//div[@class=3D'col-md-10']").text),'The Compatible lay=
er information is not loading properly')
+            self.wait_until_present('#filter-modal-layerstable')
         except:
             self.fail(msg=3D'No Compatible layers tab available')
+        self.assertTrue(re.search("Compatible layers",self.driver.find_ele=
ment(By.XPATH, "//div[@class=3D'col-md-10']").text),'The Compatible layer i=
nformation is not loading properly')
=20
         try:
             self.driver.find_element(By.XPATH, "//*[@id=3D'config-nav']/ul=
/li/a[@href=3D"+'"'+project_URL+'configuration"'+"]").click()
-            self.wait_until_present('#config-nav', poll=3D10)
-            self.assertTrue(re.search("Bitbake variables",self.driver.find=
_element(By.XPATH, "//div[@class=3D'col-md-10']").text),'The Bitbake variab=
les information is not loading properly')
+            self.wait_until_present('#configvar-list')
         except:
             self.fail(msg=3D'No Bitbake variables tab available')
+        self.assertTrue(re.search("Bitbake variables",self.driver.find_ele=
ment(By.XPATH, "//div[@class=3D'col-md-10']").text),'The Bitbake variables =
information is not loading properly')
=20
 #   testcase (1516)
     def test_review_configuration_information(self):
         self.get(reverse('all-projects'))
-        self.wait_until_present('#projectstable', poll=3D10)
+        self.load_projects_page_helper()
         self.find_element_by_link_text_in_table('projectstable', 'selenium=
-project').click()
         project_URL=3Dself.get_URL()
-        self.wait_until_present('#config-nav', poll=3D10)
+
+        # Machine section of page
+        self.wait_until_visible('#machine-section')
+        self.assertTrue(self.element_exists('#machine-section'),'Machine s=
ection for the project configuration page does not exist')
+        self.assertTrue(re.search("qemux86-64",self.driver.find_element(By=
.XPATH, "//span[@id=3D'project-machine-name']").text),'The machine type is =
not assigned')
         try:
-           self.assertTrue(self.element_exists('#machine-section'),'Machin=
e section for the project configuration page does not exist')
-           self.assertTrue(re.search("qemux86-64",self.driver.find_element=
(By.XPATH, "//span[@id=3D'project-machine-name']").text),'The machine type =
is not assigned')
            self.driver.find_element(By.XPATH, "//span[@id=3D'change-machin=
e-toggle']").click()
-           self.wait_until_visible('#select-machine-form', poll=3D10)
-           self.wait_until_visible('#cancel-machine-change', poll=3D10)
+           self.wait_until_visible('#select-machine-form')
+           self.wait_until_visible('#cancel-machine-change')
            self.driver.find_element(By.XPATH, "//form[@id=3D'select-machin=
e-form']/a[@id=3D'cancel-machine-change']").click()
         except:
            self.fail(msg=3D'The machine information is wrong in the config=
uration page')
=20
+        # Most built recipes section
+        self.wait_until_visible('#no-most-built')
         try:
            self.driver.find_element(By.ID, 'no-most-built')
         except:
            self.fail(msg=3D'No Most built information in project detail pa=
ge')
=20
-        try:
-           self.assertTrue(re.search("Yocto Project master",self.driver.fi=
nd_element(By.XPATH, "//span[@id=3D'project-release-title']").text),'The pr=
oject release is not defined')
-        except:
-           self.fail(msg=3D'No project release title information in projec=
t detail page')
+        # Project Release title
+        self.assertTrue(re.search("Yocto Project master",self.driver.find_=
element(By.XPATH, "//span[@id=3D'project-release-title']").text), 'The proj=
ect release is not defined in the project detail page')
=20
+        # List of layers in project
+        self.wait_until_visible('#layer-container')
+        self.driver.find_element(By.XPATH, "//div[@id=3D'layer-container']=
")
+        self.assertTrue(re.search("3",self.driver.find_element(By.ID, "pro=
ject-layers-count").text),'There should be 3 layers listed in the layer cou=
nt')
         try:
-           self.driver.find_element(By.XPATH, "//div[@id=3D'layer-containe=
r']")
-           self.assertTrue(re.search("3",self.driver.find_element(By.ID, "=
project-layers-count").text),'There should be 3 layers listed in the layer =
count')
            layer_list =3D self.driver.find_element(By.ID, "layers-in-proje=
ct-list")
            layers =3D layer_list.find_elements(By.TAG_NAME, "li")
-           for layer in layers:
-               if re.match ("openembedded-core",layer.text):
-                    print ("openembedded-core layer is a default layer in =
the project configuration")
-               elif re.match ("meta-poky",layer.text):
-                  print ("meta-poky layer is a default layer in the projec=
t configuration")
-               elif re.match ("meta-yocto-bsp",layer.text):
-                  print ("meta-yocto-bsp is a default layer in the project=
 configuratoin")
-               else:
-                  self.fail(msg=3D'default layers are missing from the pro=
ject configuration')
         except:
            self.fail(msg=3D'No Layer information in project detail page')
=20
+        for layer in layers:
+            if re.match ("openembedded-core", layer.text):
+                print ("openembedded-core layer is a default layer in the =
project configuration")
+            elif re.match ("meta-poky", layer.text):
+                 print ("meta-poky layer is a default layer in the project=
 configuration")
+            elif re.match ("meta-yocto-bsp", layer.text):
+                 print ("meta-yocto-bsp is a default layer in the project =
configuratoin")
+            else:
+                 self.fail(msg=3D'default layers are missing from the proj=
ect configuration')
+
 #   testcase (1517)
     def test_verify_machine_information(self):
         self.get(reverse('all-projects'))
-        self.wait_until_present('#projectstable', poll=3D10)
+        self.load_projects_page_helper()
         self.find_element_by_link_text_in_table('projectstable', 'selenium=
-project').click()
-        self.wait_until_present('#config-nav', poll=3D10)
=20
+        self.wait_until_visible('#machine-section')
+        self.assertTrue(self.element_exists('#machine-section'),'Machine s=
ection for the project configuration page does not exist')
+        self.wait_until_visible('#project-machine-name')
+        self.assertTrue(re.search("qemux86-64",self.driver.find_element(By=
.ID, "project-machine-name").text),'The machine type is not assigned')
         try:
-            self.assertTrue(self.element_exists('#machine-section'),'Machi=
ne section for the project configuration page does not exist')
-            self.assertTrue(re.search("qemux86-64",self.driver.find_elemen=
t(By.ID, "project-machine-name").text),'The machine type is not assigned')
             self.driver.find_element(By.ID, "change-machine-toggle").click=
()
-            self.wait_until_visible('#select-machine-form', poll=3D10)
-            self.wait_until_visible('#cancel-machine-change', poll=3D10)
+            self.wait_until_visible('#select-machine-form')
+            self.wait_until_visible('#cancel-machine-change')
             self.driver.find_element(By.ID, "cancel-machine-change").click=
()
         except:
             self.fail(msg=3D'The machine information is wrong in the confi=
guration page')
@@ -163,83 +150,95 @@ class FuntionalTestBasic(SeleniumFunctionalTestCase):
 #   testcase (1518)
     def test_verify_most_built_recipes_information(self):
         self.get(reverse('all-projects'))
-        self.wait_until_present('#projectstable', poll=3D10)
+        self.load_projects_page_helper()
         self.find_element_by_link_text_in_table('projectstable', 'selenium=
-project').click()
-        self.wait_until_present('#config-nav', poll=3D10)
+        self.wait_until_present('#config-nav')
         project_URL=3Dself.get_URL()
+
+        self.wait_until_visible('#no-most-built')
+        self.assertTrue(re.search("You haven't built any recipes yet",self=
.driver.find_element(By.ID, "no-most-built").text),'Default message of no b=
uilds is not present')
         try:
-            self.assertTrue(re.search("You haven't built any recipes yet",=
self.driver.find_element(By.ID, "no-most-built").text),'Default message of =
no builds is not present')
             self.driver.find_element(By.XPATH, "//div[@id=3D'no-most-built=
']/p/a[@href=3D"+'"'+project_URL+'images/"'+"]").click()
-            self.wait_until_present('#config-nav', poll=3D10)
-            self.assertTrue(re.search("Compatible image recipes",self.driv=
er.find_element(By.XPATH, "//div[@class=3D'col-md-10']").text),'The Choose =
a recipe to build link  is not working  properly')
         except:
             self.fail(msg=3D'No Most built information in project detail p=
age')
+        self.wait_until_visible('#config-nav')
+        self.assertTrue(re.search("Compatible image recipes",self.driver.f=
ind_element(By.XPATH, "//div[@class=3D'col-md-10']").text),'The Choose a re=
cipe to build link  is not working  properly')
=20
 #   testcase (1519)
     def test_verify_project_release_information(self):
         self.get(reverse('all-projects'))
-        self.wait_until_present('#projectstable', poll=3D10)
+        self.load_projects_page_helper()
         self.find_element_by_link_text_in_table('projectstable', 'selenium=
-project').click()
-        self.wait_until_present('#config-nav', poll=3D10)
-
-        try:
-            self.assertTrue(re.search("Yocto Project master",self.driver.f=
ind_element(By.ID, "project-release-title").text),'The project release is n=
ot defined')
-        except:
-            self.fail(msg=3D'No project release title information in proje=
ct detail page')
+        self.wait_until_visible('#project-release-title')
+        self.assertTrue(re.search("Yocto Project master",self.driver.find_=
element(By.ID, "project-release-title").text), 'No project release title in=
formation in project detail page')
=20
 #   testcase (1520)
     def test_verify_layer_information(self):
         self.get(reverse('all-projects'))
-        self.wait_until_present('#projectstable', poll=3D10)
+        self.load_projects_page_helper()
         self.find_element_by_link_text_in_table('projectstable', 'selenium=
-project').click()
-        self.wait_until_present('#config-nav', poll=3D10)
+        self.wait_until_present('#config-nav')
         project_URL=3Dself.get_URL()
+        self.wait_until_visible('#layer-container')
+        self.driver.find_element(By.XPATH, "//div[@id=3D'layer-container']=
")
+        self.wait_until_visible('#project-layers-count')
+        self.assertTrue(re.search("3",self.driver.find_element(By.ID, "pro=
ject-layers-count").text),'There should be 3 layers listed in the layer cou=
nt')
+
         try:
-           self.driver.find_element(By.XPATH, "//div[@id=3D'layer-containe=
r']")
-           self.assertTrue(re.search("3",self.driver.find_element(By.ID, "=
project-layers-count").text),'There should be 3 layers listed in the layer =
count')
            layer_list =3D self.driver.find_element(By.ID, "layers-in-proje=
ct-list")
            layers =3D layer_list.find_elements(By.TAG_NAME, "li")
+        except:
+            self.fail(msg=3D'No Layer information in project detail page')
=20
-           for layer in layers:
-               if re.match ("openembedded-core",layer.text):
-                    print ("openembedded-core layer is a default layer in =
the project configuration")
-               elif re.match ("meta-poky",layer.text):
-                  print ("meta-poky layer is a default layer in the projec=
t configuration")
-               elif re.match ("meta-yocto-bsp",layer.text):
-                  print ("meta-yocto-bsp is a default layer in the project=
 configuratoin")
-               else:
-                  self.fail(msg=3D'default layers are missing from the pro=
ject configuration')
+        for layer in layers:
+            if re.match ("openembedded-core",layer.text):
+                print ("openembedded-core layer is a default layer in the =
project configuration")
+            elif re.match ("meta-poky",layer.text):
+                print ("meta-poky layer is a default layer in the project =
configuration")
+            elif re.match ("meta-yocto-bsp",layer.text):
+                print ("meta-yocto-bsp is a default layer in the project c=
onfiguratoin")
+            else:
+                self.fail(msg=3D'default layers are missing from the proje=
ct configuration')
=20
+        try:
            self.driver.find_element(By.XPATH, "//input[@id=3D'layer-add-in=
put']")
            self.driver.find_element(By.XPATH, "//button[@id=3D'add-layer-b=
tn']")
            self.driver.find_element(By.XPATH, "//div[@id=3D'layer-containe=
r']/form[@class=3D'form-inline']/p/a[@id=3D'view-compatible-layers']")
            self.driver.find_element(By.XPATH, "//div[@id=3D'layer-containe=
r']/form[@class=3D'form-inline']/p/a[@href=3D"+'"'+project_URL+'importlayer=
"'+"]")
         except:
-            self.fail(msg=3D'No Layer information in project detail page')
+            self.fail(msg=3D'Layer configuration controls missing')
=20
 #   testcase (1521)
     def test_verify_project_detail_links(self):
         self.get(reverse('all-projects'))
-        self.wait_until_present('#projectstable', poll=3D10)
+        self.load_projects_page_helper()
         self.find_element_by_link_text_in_table('projectstable', 'selenium=
-project').click()
-        self.wait_until_present('#config-nav', poll=3D10)
+        self.wait_until_present('#config-nav')
         project_URL=3Dself.get_URL()
         self.driver.find_element(By.XPATH, "//div[@id=3D'project-topbar']/=
ul[@class=3D'nav nav-tabs']/li[@id=3D'topbar-configuration-tab']/a[@href=3D=
"+'"'+project_URL+'"'+"]").click()
-        self.wait_until_present('#config-nav', poll=3D10)
+        self.wait_until_visible('#topbar-configuration-tab')
         self.assertTrue(re.search("Configuration",self.driver.find_element=
(By.XPATH, "//div[@id=3D'project-topbar']/ul[@class=3D'nav nav-tabs']/li[@i=
d=3D'topbar-configuration-tab']/a[@href=3D"+'"'+project_URL+'"'+"]").text),=
 'Configuration tab in project topbar is misspelled')
=20
         try:
             self.driver.find_element(By.XPATH, "//div[@id=3D'project-topba=
r']/ul[@class=3D'nav nav-tabs']/li/a[@href=3D"+'"'+project_URL+'builds/"'+"=
]").click()
-            self.wait_until_visible('#project-topbar', poll=3D10)
-            self.assertTrue(re.search("Builds",self.driver.find_element(By=
.XPATH, "//div[@id=3D'project-topbar']/ul[@class=3D'nav nav-tabs']/li/a[@hr=
ef=3D"+'"'+project_URL+'builds/"'+"]").text), 'Builds tab in project topbar=
 is misspelled')
+        except:
+            self.fail(msg=3D'Builds tab information is not present')
+
+        self.wait_until_visible('#project-topbar')
+        self.assertTrue(re.search("Builds",self.driver.find_element(By.XPA=
TH, "//div[@id=3D'project-topbar']/ul[@class=3D'nav nav-tabs']/li/a[@href=
=3D"+'"'+project_URL+'builds/"'+"]").text), 'Builds tab in project topbar i=
s misspelled')
+        try:
             self.driver.find_element(By.XPATH, "//div[@id=3D'empty-state-p=
rojectbuildstable']")
         except:
             self.fail(msg=3D'Builds tab information is not present')
=20
         try:
             self.driver.find_element(By.XPATH, "//div[@id=3D'project-topba=
r']/ul[@class=3D'nav nav-tabs']/li/a[@href=3D"+'"'+project_URL+'importlayer=
"'+"]").click()
-            self.wait_until_visible('#project-topbar', poll=3D10)
-            self.assertTrue(re.search("Import layer",self.driver.find_elem=
ent(By.XPATH, "//div[@id=3D'project-topbar']/ul[@class=3D'nav nav-tabs']/li=
/a[@href=3D"+'"'+project_URL+'importlayer"'+"]").text), 'Import layer tab i=
n project topbar is misspelled')
+        except:
+            self.fail(msg=3D'Import layer tab not loading properly')
+
+        self.wait_until_visible('#project-topbar')
+        self.assertTrue(re.search("Import layer",self.driver.find_element(=
By.XPATH, "//div[@id=3D'project-topbar']/ul[@class=3D'nav nav-tabs']/li/a[@=
href=3D"+'"'+project_URL+'importlayer"'+"]").text), 'Import layer tab in pr=
oject topbar is misspelled')
+        try:
             self.driver.find_element(By.XPATH, "//fieldset[@id=3D'repo-sel=
ect']")
             self.driver.find_element(By.XPATH, "//fieldset[@id=3D'git-repo=
']")
         except:
@@ -247,11 +246,12 @@ class FuntionalTestBasic(SeleniumFunctionalTestCase):
=20
         try:
             self.driver.find_element(By.XPATH, "//div[@id=3D'project-topba=
r']/ul[@class=3D'nav nav-tabs']/li/a[@href=3D"+'"'+project_URL+'newcustomim=
age/"'+"]").click()
-            self.wait_until_visible('#project-topbar', poll=3D10)
-            self.assertTrue(re.search("New custom image",self.driver.find_=
element(By.XPATH, "//div[@id=3D'project-topbar']/ul[@class=3D'nav nav-tabs'=
]/li/a[@href=3D"+'"'+project_URL+'newcustomimage/"'+"]").text), 'New custom=
 image tab in project topbar is misspelled')
-            self.assertTrue(re.search("Select the image recipe you want to=
 customise",self.driver.find_element(By.XPATH, "//div[@class=3D'col-md-12']=
/h2").text),'The new custom image tab is not loading correctly')
         except:
             self.fail(msg=3D'New custom image tab not loading properly')
=20
+        self.wait_until_visible('#project-topbar')
+        self.assertTrue(re.search("New custom image",self.driver.find_elem=
ent(By.XPATH, "//div[@id=3D'project-topbar']/ul[@class=3D'nav nav-tabs']/li=
/a[@href=3D"+'"'+project_URL+'newcustomimage/"'+"]").text), 'New custom ima=
ge tab in project topbar is misspelled')
+        self.assertTrue(re.search("Select the image recipe you want to cus=
tomise",self.driver.find_element(By.XPATH, "//div[@class=3D'col-md-12']/h2"=
).text),'The new custom image tab is not loading correctly')
+
=20
=20
diff --git a/bitbake/lib/toaster/tests/functional/test_project_config.py b/=
bitbake/lib/toaster/tests/functional/test_project_config.py
index dbee36aa..8c8450cf 100644
--- a/bitbake/lib/toaster/tests/functional/test_project_config.py
+++ b/bitbake/lib/toaster/tests/functional/test_project_config.py
@@ -7,20 +7,16 @@
 #
=20
 import string
-import random
 import pytest
 from django.urls import reverse
 from selenium.webdriver import Keys
 from selenium.webdriver.support.select import Select
 from selenium.common.exceptions import TimeoutException
-from tests.functional.functional_helpers import SeleniumFunctionalTestCase
+from tests.functional.functional_helpers import SeleniumFunctionalTestCase=
, MASTER_PROJECT_ID
 from selenium.webdriver.common.by import By
=20
 from .utils import get_projectId_from_url
=20
-
-@pytest.mark.django_db
-@pytest.mark.order("last")
 class TestProjectConfig(SeleniumFunctionalTestCase):
     project_id =3D None
     PROJECT_NAME =3D 'TestProjectConfig'
@@ -28,42 +24,6 @@ class TestProjectConfig(SeleniumFunctionalTestCase):
     INVALID_PATH_CHAR_TEXT =3D 'The directory path cannot include spaces o=
r ' \
         'any of these characters'
=20
-    def _create_project(self, project_name):
-        """ Create/Test new project using:
-          - Project Name: Any string
-          - Release: Any string
-          - Merge Toaster settings: True or False
-        """
-        self.get(reverse('newproject'))
-        self.wait_until_visible('#new-project-name', poll=3D2)
-        self.find("#new-project-name").send_keys(project_name)
-        select =3D Select(self.find("#projectversion"))
-        select.select_by_value('3')
-
-        # check merge toaster settings
-        checkbox =3D self.find('.checkbox-mergeattr')
-        if not checkbox.is_selected():
-            checkbox.click()
-
-        if self.PROJECT_NAME !=3D 'TestProjectConfig':
-            # Reset project name if it's not the default one
-            self.PROJECT_NAME =3D 'TestProjectConfig'
-
-        self.find("#create-project-button").click()
-
-        try:
-            self.wait_until_visible('#hint-error-project-name', poll=3D2)
-            url =3D reverse('project', args=3D(TestProjectConfig.project_i=
d, ))
-            self.get(url)
-            self.wait_until_visible('#config-nav', poll=3D3)
-        except TimeoutException:
-            self.wait_until_visible('#config-nav', poll=3D3)
-
-    def _random_string(self, length):
-        return ''.join(
-            random.choice(string.ascii_letters) for _ in range(length)
-        )
-
     def _get_config_nav_item(self, index):
         config_nav =3D self.find('#config-nav')
         return config_nav.find_elements(By.TAG_NAME, 'li')[index]
@@ -72,16 +32,14 @@ class TestProjectConfig(SeleniumFunctionalTestCase):
         """ Navigate to project BitBake variables page """
         # check if the menu is displayed
         if TestProjectConfig.project_id is None:
-            self._create_project(project_name=3Dself._random_string(10))
-            current_url =3D self.driver.current_url
-            TestProjectConfig.project_id =3D get_projectId_from_url(curren=
t_url)
-        else:
-            url =3D reverse('projectconf', args=3D(TestProjectConfig.proje=
ct_id,))
-            self.get(url)
-        self.wait_until_visible('#config-nav', poll=3D3)
+            TestProjectConfig.project_id =3D self.create_new_project(self.=
PROJECT_NAME, MASTER_PROJECT_ID, None, True)
+
+        url =3D reverse('projectconf', args=3D(TestProjectConfig.project_i=
d,))
+        self.get(url)
+        self.wait_until_visible('#config-nav')
         bbv_page_link =3D self._get_config_nav_item(9)
         bbv_page_link.click()
-        self.wait_until_visible('#config-nav', poll=3D3)
+        self.wait_until_visible('#config-nav')
=20
     def test_no_underscore_iamgefs_type(self):
         """
@@ -90,13 +48,13 @@ class TestProjectConfig(SeleniumFunctionalTestCase):
         self._navigate_bbv_page()
         imagefs_type =3D "foo_bar"
=20
-        self.wait_until_visible('#change-image_fstypes-icon', poll=3D2)
+        self.wait_until_visible('#change-image_fstypes-icon')
=20
         self.click('#change-image_fstypes-icon')
=20
         self.enter_text('#new-imagefs_types', imagefs_type)
=20
-        element =3D self.wait_until_visible('#hintError-image-fs_type', po=
ll=3D2)
+        element =3D self.wait_until_visible('#hintError-image-fs_type')
=20
         self.assertTrue(("A valid image type cannot include underscores" i=
n element.text),
                         "Did not find underscore error message")
@@ -110,7 +68,7 @@ class TestProjectConfig(SeleniumFunctionalTestCase):
=20
         imagefs_type =3D "btrfs"
=20
-        self.wait_until_visible('#change-image_fstypes-icon', poll=3D2)
+        self.wait_until_visible('#change-image_fstypes-icon')
=20
         self.click('#change-image_fstypes-icon')
=20
@@ -129,22 +87,20 @@ class TestProjectConfig(SeleniumFunctionalTestCase):
         """
         self._navigate_bbv_page()
=20
-        self.wait_until_visible('#change-image_fstypes-icon', poll=3D2)
-
+        self.wait_until_visible('#change-image_fstypes-icon')
         self.click('#change-image_fstypes-icon')
=20
         checkboxes_selector =3D '.fs-checkbox-fstypes'
=20
-        self.wait_until_visible(checkboxes_selector, poll=3D2)
+        self.wait_until_visible(checkboxes_selector)
         checkboxes =3D self.find_all(checkboxes_selector)
=20
         for checkbox in checkboxes:
             if checkbox.get_attribute("value") =3D=3D "cpio":
                checkbox.click()
+               self.wait_until_visible('#new-imagefs_types')
                element =3D self.driver.find_element(By.ID, 'new-imagefs_ty=
pes')
=20
-               self.wait_until_visible('#new-imagefs_types', poll=3D2)
-
                self.assertTrue(("cpio" in element.get_attribute('value'),
                                "Imagefs not added into the textbox"))
                checkbox.click()
@@ -160,20 +116,19 @@ class TestProjectConfig(SeleniumFunctionalTestCase):
=20
         # activate the input to edit download dir
         try:
-            change_dl_dir_btn =3D self.wait_until_visible('#change-dl_dir-=
icon', poll=3D2)
+            change_dl_dir_btn =3D self.wait_until_visible('#change-dl_dir-=
icon')
         except TimeoutException:
             # If download dir is not displayed, test is skipped
             change_dl_dir_btn =3D None
=20
         if change_dl_dir_btn:
-            change_dl_dir_btn =3D self.wait_until_visible('#change-dl_dir-=
icon', poll=3D2)
             change_dl_dir_btn.click()
=20
             # downloads dir path doesn't start with / or ${...}
-            input_field =3D self.wait_until_visible('#new-dl_dir', poll=3D=
2)
+            input_field =3D self.wait_until_visible('#new-dl_dir')
             input_field.clear()
             self.enter_text('#new-dl_dir', 'home/foo')
-            element =3D self.wait_until_visible('#hintError-initialChar-dl=
_dir', poll=3D2)
+            element =3D self.wait_until_visible('#hintError-initialChar-dl=
_dir')
=20
             msg =3D 'downloads directory path starts with invalid characte=
r but ' \
                 'treated as valid'
@@ -183,7 +138,7 @@ class TestProjectConfig(SeleniumFunctionalTestCase):
             self.driver.find_element(By.ID, 'new-dl_dir').clear()
             self.enter_text('#new-dl_dir', '/foo/bar a')
=20
-            element =3D self.wait_until_visible('#hintError-dl_dir', poll=
=3D2)
+            element =3D self.wait_until_visible('#hintError-dl_dir')
             msg =3D 'downloads directory path characters invalid but treat=
ed as valid'
             self.assertTrue((self.INVALID_PATH_CHAR_TEXT in element.text),=
 msg)
=20
@@ -191,7 +146,7 @@ class TestProjectConfig(SeleniumFunctionalTestCase):
             self.driver.find_element(By.ID,'new-dl_dir').clear()
             self.enter_text('#new-dl_dir', '${TOPDIR}/down foo')
=20
-            element =3D self.wait_until_visible('#hintError-dl_dir', poll=
=3D2)
+            element =3D self.wait_until_visible('#hintError-dl_dir')
             msg =3D 'downloads directory path characters invalid but treat=
ed as valid'
             self.assertTrue((self.INVALID_PATH_CHAR_TEXT in element.text),=
 msg)
=20
@@ -219,10 +174,7 @@ class TestProjectConfig(SeleniumFunctionalTestCase):
         self._navigate_bbv_page()
=20
         try:
-            btn_chg_sstate_dir =3D self.wait_until_visible(
-                '#change-sstate_dir-icon',
-                poll=3D2
-            )
+            btn_chg_sstate_dir =3D self.wait_until_visible('#change-sstate=
_dir-icon')
             self.click('#change-sstate_dir-icon')
         except TimeoutException:
             # If sstate_dir is not displayed, test is skipped
@@ -230,10 +182,10 @@ class TestProjectConfig(SeleniumFunctionalTestCase):
=20
         if btn_chg_sstate_dir:  # Skip continuation if sstate_dir is not d=
isplayed
             # path doesn't start with / or ${...}
-            input_field =3D self.wait_until_visible('#new-sstate_dir', pol=
l=3D2)
+            input_field =3D self.wait_until_visible('#new-sstate_dir')
             input_field.clear()
             self.enter_text('#new-sstate_dir', 'home/foo')
-            element =3D self.wait_until_visible('#hintError-initialChar-ss=
tate_dir', poll=3D2)
+            element =3D self.wait_until_visible('#hintError-initialChar-ss=
tate_dir')
=20
             msg =3D 'sstate directory path starts with invalid character b=
ut ' \
                 'treated as valid'
@@ -243,7 +195,7 @@ class TestProjectConfig(SeleniumFunctionalTestCase):
             self.driver.find_element(By.ID, 'new-sstate_dir').clear()
             self.enter_text('#new-sstate_dir', '/foo/bar a')
=20
-            element =3D self.wait_until_visible('#hintError-sstate_dir', p=
oll=3D2)
+            element =3D self.wait_until_visible('#hintError-sstate_dir')
             msg =3D 'sstate directory path characters invalid but treated =
as valid'
             self.assertTrue((self.INVALID_PATH_CHAR_TEXT in element.text),=
 msg)
=20
@@ -251,7 +203,7 @@ class TestProjectConfig(SeleniumFunctionalTestCase):
             self.driver.find_element(By.ID,'new-sstate_dir').clear()
             self.enter_text('#new-sstate_dir', '${TOPDIR}/down foo')
=20
-            element =3D self.wait_until_visible('#hintError-sstate_dir', p=
oll=3D2)
+            element =3D self.wait_until_visible('#hintError-sstate_dir')
             msg =3D 'sstate directory path characters invalid but treated =
as valid'
             self.assertTrue((self.INVALID_PATH_CHAR_TEXT in element.text),=
 msg)
=20
@@ -275,13 +227,14 @@ class TestProjectConfig(SeleniumFunctionalTestCase):
         var_name, field, btn_id, input_id, value, save_btn, *_ =3D kwargs.=
values()
         """ Change bitbake variable value """
         self._navigate_bbv_page()
-        self.wait_until_visible(f'#{btn_id}', poll=3D2)
+        self.wait_until_visible(f'#{btn_id}')
         if kwargs.get('new_variable'):
             self.find(f"#{btn_id}").clear()
             self.enter_text(f"#{btn_id}", f"{var_name}")
         else:
             self.click(f'#{btn_id}')
-            self.wait_until_visible(f'#{input_id}', poll=3D2)
+
+        self.wait_until_visible(f'#{input_id}')
=20
         if kwargs.get('is_select'):
             select =3D Select(self.find(f'#{input_id}'))
diff --git a/bitbake/lib/toaster/tests/functional/test_project_page.py b/bi=
tbake/lib/toaster/tests/functional/test_project_page.py
index adbe3587..8f67c420 100644
--- a/bitbake/lib/toaster/tests/functional/test_project_page.py
+++ b/bitbake/lib/toaster/tests/functional/test_project_page.py
@@ -7,8 +7,8 @@
 #
=20
 import os
-import random
 import string
+import time
 from unittest import skip
 import pytest
 from django.urls import reverse
@@ -16,64 +16,23 @@ from django.utils import timezone
 from selenium.webdriver.common.keys import Keys
 from selenium.webdriver.support.select import Select
 from selenium.common.exceptions import TimeoutException
-from tests.functional.functional_helpers import SeleniumFunctionalTestCase
+from tests.functional.functional_helpers import SeleniumFunctionalTestCase=
, MASTER_PROJECT_ID
 from orm.models import Build, Project, Target
 from selenium.webdriver.common.by import By
=20
 from .utils import get_projectId_from_url, wait_until_build, wait_until_bu=
ild_cancelled
=20
-
-@pytest.mark.django_db
-@pytest.mark.order("last")
-class TestProjectPage(SeleniumFunctionalTestCase):
+class TestProjectPageBase(SeleniumFunctionalTestCase):
     project_id =3D None
     PROJECT_NAME =3D 'TestProjectPage'
=20
-    def _create_project(self, project_name):
-        """ Create/Test new project using:
-          - Project Name: Any string
-          - Release: Any string
-          - Merge Toaster settings: True or False
-        """
-        self.get(reverse('newproject'))
-        self.wait_until_visible('#new-project-name')
-        self.find("#new-project-name").send_keys(project_name)
-        select =3D Select(self.find("#projectversion"))
-        select.select_by_value('3')
-
-        # check merge toaster settings
-        checkbox =3D self.find('.checkbox-mergeattr')
-        if not checkbox.is_selected():
-            checkbox.click()
-
-        if self.PROJECT_NAME !=3D 'TestProjectPage':
-            # Reset project name if it's not the default one
-            self.PROJECT_NAME =3D 'TestProjectPage'
-
-        self.find("#create-project-button").click()
-
-        try:
-            self.wait_until_visible('#hint-error-project-name')
-            url =3D reverse('project', args=3D(TestProjectPage.project_id,=
 ))
-            self.get(url)
-            self.wait_until_visible('#config-nav', poll=3D3)
-        except TimeoutException:
-            self.wait_until_visible('#config-nav', poll=3D3)
-
-    def _random_string(self, length):
-        return ''.join(
-            random.choice(string.ascii_letters) for _ in range(length)
-        )
-
     def _navigate_to_project_page(self):
         # Navigate to project page
-        if TestProjectPage.project_id is None:
-            self._create_project(project_name=3Dself._random_string(10))
-            current_url =3D self.driver.current_url
-            TestProjectPage.project_id =3D get_projectId_from_url(current_=
url)
-        else:
-            url =3D reverse('project', args=3D(TestProjectPage.project_id,=
))
-            self.get(url)
+        if TestProjectPageBase.project_id is None:
+            TestProjectPageBase.project_id =3D self.create_new_project(sel=
f.PROJECT_NAME, MASTER_PROJECT_ID, None, True)
+
+        url =3D reverse('project', args=3D(TestProjectPageBase.project_id,=
))
+        self.get(url)
         self.wait_until_visible('#config-nav')
=20
     def _get_create_builds(self, **kwargs):
@@ -81,14 +40,14 @@ class TestProjectPage(SeleniumFunctionalTestCase):
         # parameters for builds to associate with the projects
         now =3D timezone.now()
         self.project1_build_success =3D {
-            'project': Project.objects.get(id=3DTestProjectPage.project_id=
),
+            'project': Project.objects.get(id=3DTestProjectPageBase.projec=
t_id),
             'started_on': now,
             'completed_on': now,
             'outcome': Build.SUCCEEDED
         }
=20
         self.project1_build_failure =3D {
-            'project': Project.objects.get(id=3DTestProjectPage.project_id=
),
+            'project': Project.objects.get(id=3DTestProjectPageBase.projec=
t_id),
             'started_on': now,
             'completed_on': now,
             'outcome': Build.FAILED
@@ -133,7 +92,8 @@ class TestProjectPage(SeleniumFunctionalTestCase):
             list_check_box_id: list
     ):
         # Check edit column
-        edit_column =3D self.find(f'#{edit_btn_id}')
+        finder =3D lambda driver: self.find(f'#{edit_btn_id}')
+        edit_column =3D self.wait_until_finder_clickable(finder)
         self.assertTrue(edit_column.is_displayed())
         edit_column.click()
         # Check dropdown is visible
@@ -192,7 +152,7 @@ class TestProjectPage(SeleniumFunctionalTestCase):
         def test_show_rows(row_to_show, show_row_link):
             # Check that we can show rows =3D=3D row_to_show
             show_row_link.select_by_value(str(row_to_show))
-            self.wait_until_visible(f'#{table_selector} tbody tr', poll=3D=
3)
+            self.wait_until_visible(f'#{table_selector} tbody tr')
             # check at least some rows are visible
             self.assertTrue(
                 len(self.find_all(f'#{table_selector} tbody tr')) > 0
@@ -222,34 +182,7 @@ class TestProjectPage(SeleniumFunctionalTestCase):
         rows =3D self.find_all(f'#{table_selector} tbody tr')
         self.assertTrue(len(rows) > 0)
=20
-    def test_create_project(self):
-        """ Create/Test new project using:
-          - Project Name: Any string
-          - Release: Any string
-          - Merge Toaster settings: True or False
-        """
-        self._create_project(project_name=3Dself.PROJECT_NAME)
-
-    def test_image_recipe_editColumn(self):
-        """ Test the edit column feature in image recipe table on project =
page """
-        self._get_create_builds(success=3D10, failure=3D10)
-
-        url =3D reverse('projectimagerecipes', args=3D(TestProjectPage.pro=
ject_id,))
-        self.get(url)
-        self.wait_until_present('#imagerecipestable tbody tr')
-
-        column_list =3D [
-            'get_description_or_summary', 'layer_version__get_vcs_referenc=
e',
-            'layer_version__layer__name', 'license', 'recipe-file', 'secti=
on',
-            'version'
-        ]
-
-        # Check that we can hide the edit column
-        self._mixin_test_table_edit_column(
-            'imagerecipestable',
-            'edit-columns-button',
-            [f'checkbox-{column}' for column in column_list]
-        )
+class TestProjectPage(TestProjectPageBase):
=20
     def test_page_header_on_project_page(self):
         """ Check page header in project page:
@@ -272,8 +205,8 @@ class TestProjectPage(SeleniumFunctionalTestCase):
         logo_img =3D logo.find_element(By.TAG_NAME, 'img')
         self.assertTrue(logo_img.is_displayed(),
                         'Logo of Yocto project not found')
-        self.assertTrue(
-            '/static/img/logo.png' in str(logo_img.get_attribute('src')),
+        self.assertIn(
+            '/static/img/logo.png', str(logo_img.get_attribute('src')),
             'Logo of Yocto project not found'
         )
         # "Toaster"+" Information icon", clickable
@@ -282,34 +215,34 @@ class TestProjectPage(SeleniumFunctionalTestCase):
             "//div[@class=3D'toaster-navbar-brand']//a[@class=3D'brand']",
         )
         self.assertTrue(toaster.is_displayed(), 'Toaster not found')
-        self.assertTrue(toaster.text =3D=3D 'Toaster')
+        self.assertEqual(toaster.text, 'Toaster')
         info_sign =3D self.find('.glyphicon-info-sign')
         self.assertTrue(info_sign.is_displayed())
=20
         # "Server Icon" + "All builds"
         all_builds =3D self.find('#navbar-all-builds')
         all_builds_link =3D all_builds.find_element(By.TAG_NAME, 'a')
-        self.assertTrue("All builds" in all_builds_link.text)
-        self.assertTrue(
-            '/toastergui/builds/' in str(all_builds_link.get_attribute('hr=
ef'))
+        self.assertIn("All builds", all_builds_link.text)
+        self.assertIn(
+            '/toastergui/builds/', str(all_builds_link.get_attribute('href=
'))
         )
         server_icon =3D all_builds.find_element(By.TAG_NAME, 'i')
-        self.assertTrue(
-            server_icon.get_attribute('class') =3D=3D 'glyphicon glyphicon=
-tasks'
+        self.assertEqual(
+            server_icon.get_attribute('class'), 'glyphicon glyphicon-tasks=
'
         )
         self.assertTrue(server_icon.is_displayed())
=20
         # "Directory Icon" + "All projects"
         all_projects =3D self.find('#navbar-all-projects')
         all_projects_link =3D all_projects.find_element(By.TAG_NAME, 'a')
-        self.assertTrue("All projects" in all_projects_link.text)
-        self.assertTrue(
-            '/toastergui/projects/' in str(all_projects_link.get_attribute=
(
+        self.assertIn("All projects", all_projects_link.text)
+        self.assertIn(
+            '/toastergui/projects/', str(all_projects_link.get_attribute(
                 'href'))
         )
         dir_icon =3D all_projects.find_element(By.TAG_NAME, 'i')
-        self.assertTrue(
-            dir_icon.get_attribute('class') =3D=3D 'icon-folder-open'
+        self.assertEqual(
+            dir_icon.get_attribute('class'), 'icon-folder-open'
         )
         self.assertTrue(dir_icon.is_displayed())
=20
@@ -317,23 +250,23 @@ class TestProjectPage(SeleniumFunctionalTestCase):
         toaster_docs_link =3D self.find('#navbar-docs')
         toaster_docs_link_link =3D toaster_docs_link.find_element(By.TAG_N=
AME,
                                                                 'a')
-        self.assertTrue("Documentation" in toaster_docs_link_link.text)
-        self.assertTrue(
-            toaster_docs_link_link.get_attribute('href') =3D=3D 'http://do=
cs.yoctoproject.org/toaster-manual/index.html#toaster-user-manual'
+        self.assertIn("Documentation", toaster_docs_link_link.text)
+        self.assertEqual(
+            toaster_docs_link_link.get_attribute('href'), 'http://docs.yoc=
toproject.org/toaster-manual/index.html#toaster-user-manual'
         )
         book_icon =3D toaster_docs_link.find_element(By.TAG_NAME, 'i')
-        self.assertTrue(
-            book_icon.get_attribute('class') =3D=3D 'glyphicon glyphicon-b=
ook'
+        self.assertEqual(
+            book_icon.get_attribute('class'), 'glyphicon glyphicon-book'
         )
         self.assertTrue(book_icon.is_displayed())
=20
         # AT RIGHT -> button "New project"
         new_project_button =3D self.find('#new-project-button')
         self.assertTrue(new_project_button.is_displayed())
-        self.assertTrue(new_project_button.text =3D=3D 'New project')
+        self.assertEqual(new_project_button.text, 'New project')
         new_project_button.click()
-        self.assertTrue(
-            '/toastergui/newproject/' in str(self.driver.current_url)
+        self.assertIn(
+            '/toastergui/newproject/', str(self.driver.current_url)
         )
=20
     def test_edit_project_name(self):
@@ -348,7 +281,8 @@ class TestProjectPage(SeleniumFunctionalTestCase):
=20
         # click on "Edit" icon button
         self.wait_until_visible('#project-name-container')
-        edit_button =3D self.find('#project-change-form-toggle')
+        finder =3D lambda driver: self.find('#project-change-form-toggle')
+        edit_button =3D self.wait_until_finder_clickable(finder)
         edit_button.click()
         project_name_input =3D self.find('#project-name-change-input')
         self.assertTrue(project_name_input.is_displayed())
@@ -358,8 +292,8 @@ class TestProjectPage(SeleniumFunctionalTestCase):
=20
         # check project name is changed
         self.wait_until_visible('#project-name-container')
-        self.assertTrue(
-            'New Name' in str(self.find('#project-name-container').text)
+        self.assertIn(
+            'New Name', str(self.find('#project-name-container').text)
         )
=20
     def test_project_page_tabs(self):
@@ -376,10 +310,10 @@ class TestProjectPage(SeleniumFunctionalTestCase):
         # check "configuration" tab
         self.wait_until_visible('#topbar-configuration-tab')
         config_tab =3D self.find('#topbar-configuration-tab')
-        self.assertTrue(config_tab.get_attribute('class') =3D=3D 'active')
-        self.assertTrue('Configuration' in str(config_tab.text))
-        self.assertTrue(
-            f"/toastergui/project/{TestProjectPage.project_id}" in str(sel=
f.driver.current_url)
+        self.assertEqual(config_tab.get_attribute('class'), 'active')
+        self.assertIn('Configuration', str(config_tab.text))
+        self.assertIn(
+            f"/toastergui/project/{TestProjectPageBase.project_id}", str(s=
elf.driver.current_url)
         )
=20
         def get_tabs():
@@ -392,9 +326,9 @@ class TestProjectPage(SeleniumFunctionalTestCase):
         def check_tab_link(tab_index, tab_name, url):
             tab =3D get_tabs()[tab_index]
             tab_link =3D tab.find_element(By.TAG_NAME, 'a')
-            self.assertTrue(url in tab_link.get_attribute('href'))
-            self.assertTrue(tab_name in tab_link.text)
-            self.assertTrue(tab.get_attribute('class') =3D=3D 'active')
+            self.assertIn(url, tab_link.get_attribute('href'))
+            self.assertIn(tab_name, tab_link.text)
+            self.assertEqual(tab.get_attribute('class'), 'active')
=20
         # check "Builds" tab
         builds_tab =3D get_tabs()[1]
@@ -402,7 +336,7 @@ class TestProjectPage(SeleniumFunctionalTestCase):
         check_tab_link(
             1,
             'Builds',
-            f"/toastergui/project/{TestProjectPage.project_id}/builds"
+            f"/toastergui/project/{TestProjectPageBase.project_id}/builds"
         )
=20
         # check "Import layers" tab
@@ -411,7 +345,7 @@ class TestProjectPage(SeleniumFunctionalTestCase):
         check_tab_link(
             2,
             'Import layer',
-            f"/toastergui/project/{TestProjectPage.project_id}/importlayer=
"
+            f"/toastergui/project/{TestProjectPageBase.project_id}/importl=
ayer"
         )
=20
         # check "New custom image" tab
@@ -420,7 +354,7 @@ class TestProjectPage(SeleniumFunctionalTestCase):
         check_tab_link(
             3,
             'New custom image',
-            f"/toastergui/project/{TestProjectPage.project_id}/newcustomim=
age"
+            f"/toastergui/project/{TestProjectPageBase.project_id}/newcust=
omimage"
         )
=20
         # check search box can be use to build recipes
@@ -428,13 +362,17 @@ class TestProjectPage(SeleniumFunctionalTestCase):
         search_box.send_keys('core-image-minimal')
         self.find('#build-button').click()
         self.wait_until_visible('#latest-builds')
-        lastest_builds =3D self.driver.find_elements(
-            By.XPATH,
-            '//div[@id=3D"latest-builds"]',
-        )
-        last_build =3D lastest_builds[0]
-        self.assertTrue(
-            'core-image-minimal' in str(last_build.text)
+        buildtext =3D "Loading"
+        while "Loading" in buildtext:
+            time.sleep(1)
+            lastest_builds =3D self.driver.find_elements(
+                By.XPATH,
+                '//div[@id=3D"latest-builds"]',
+            )
+            last_build =3D lastest_builds[0]
+            buildtext =3D last_build.text
+        self.assertIn(
+            'core-image-minimal', str(last_build.text)
         )
=20
     def test_softwareRecipe_page(self):
@@ -446,7 +384,7 @@ class TestProjectPage(SeleniumFunctionalTestCase):
         """
         self._navigate_to_config_nav('softwarerecipestable', 4)
         # check title "Compatible software recipes" is displayed
-        self.assertTrue("Compatible software recipes" in self.get_page_sou=
rce())
+        self.assertIn("Compatible software recipes", self.get_page_source(=
))
         # Test search input
         self._mixin_test_table_search_input(
             input_selector=3D'search-input-softwarerecipestable',
@@ -455,12 +393,8 @@ class TestProjectPage(SeleniumFunctionalTestCase):
             table_selector=3D'softwarerecipestable'
         )
         # check "build recipe" button works
-        rows =3D self.find_all('#softwarerecipestable tbody tr')
-        image_to_build =3D rows[0]
-        build_btn =3D image_to_build.find_element(
-            By.XPATH,
-            '//td[@class=3D"add-del-layers"]//a[1]'
-        )
+        finder =3D lambda driver: self.find_all('#softwarerecipestable tbo=
dy tr')[0].find_element(By.XPATH, '//td[@class=3D"add-del-layers"]/a')
+        build_btn =3D self.wait_until_finder_clickable(finder)
         build_btn.click()
         build_state =3D wait_until_build(self, 'queued cloning starting pa=
rsing failed')
         lastest_builds =3D self.driver.find_elements(
@@ -468,11 +402,10 @@ class TestProjectPage(SeleniumFunctionalTestCase):
             '//div[@id=3D"latest-builds"]/div'
         )
         self.assertTrue(len(lastest_builds) > 0)
-        last_build =3D lastest_builds[0]
-        cancel_button =3D last_build.find_element(
-            By.XPATH,
-            '//span[@class=3D"cancel-build-btn pull-right alert-link"]',
-        )
+        # Find the latest builds, the last build and then the cancel butto=
n
+=20
+        finder =3D lambda driver: driver.find_elements(By.XPATH, '//div[@i=
d=3D"latest-builds"]/div')[0].find_element(By.XPATH, '//span[@class=3D"canc=
el-build-btn pull-right alert-link"]')
+        cancel_button =3D self.wait_until_finder_clickable(finder)
         cancel_button.click()
         if 'starting' not in build_state:  # change build state when cance=
lled in starting state
             wait_until_build_cancelled(self)
@@ -510,7 +443,7 @@ class TestProjectPage(SeleniumFunctionalTestCase):
         """
         self._navigate_to_config_nav('machinestable', 5)
         # check title "Compatible software recipes" is displayed
-        self.assertTrue("Compatible machines" in self.get_page_source())
+        self.assertIn("Compatible machines", self.get_page_source())
         # Test search input
         self._mixin_test_table_search_input(
             input_selector=3D'search-input-machinestable',
@@ -519,17 +452,13 @@ class TestProjectPage(SeleniumFunctionalTestCase):
             table_selector=3D'machinestable'
         )
         # check "Select machine" button works
-        rows =3D self.find_all('#machinestable tbody tr')
-        machine_to_select =3D rows[0]
-        select_btn =3D machine_to_select.find_element(
-            By.XPATH,
-            '//td[@class=3D"add-del-layers"]//a[1]'
-        )
-        select_btn.send_keys(Keys.RETURN)
-        self.wait_until_visible('#config-nav')
+        finder =3D lambda driver: self.find_all('#machinestable tbody tr')=
[0].find_element(By.XPATH, '//td[@class=3D"add-del-layers"]')
+        select_btn =3D self.wait_until_finder_clickable(finder)
+        select_btn.click()
+        self.wait_until_visible('#project-machine-name')
         project_machine_name =3D self.find('#project-machine-name')
-        self.assertTrue(
-            'qemux86-64' in project_machine_name.text
+        self.assertIn(
+            'qemux86-64', project_machine_name.text
         )
         # check "Add layer" button works
         self._navigate_to_config_nav('machinestable', 5)
@@ -540,16 +469,23 @@ class TestProjectPage(SeleniumFunctionalTestCase):
             searchBtn_selector=3D'search-submit-machinestable',
             table_selector=3D'machinestable'
         )
-        self.wait_until_visible('#machinestable tbody tr', poll=3D3)
-        rows =3D self.find_all('#machinestable tbody tr')
-        machine_to_add =3D rows[0]
-        add_btn =3D machine_to_add.find_element(By.XPATH, '//td[@class=3D"=
add-del-layers"]')
+
+        self.wait_until_visible('#machinestable tbody tr')
+        # Locate a machine to add button
+        finder =3D lambda driver: self.find_all('#machinestable tbody tr')=
[0].find_element(By.XPATH, '//td[@class=3D"add-del-layers"]')
+        add_btn =3D self.wait_until_finder_clickable(finder)
         add_btn.click()
         self.wait_until_visible('#change-notification')
         change_notification =3D self.find('#change-notification')
-        self.assertTrue(
-            f'You have added 1 layer to your project' in str(change_notifi=
cation.text)
+        self.assertIn(
+            f'You have added 1 layer to your project', str(change_notifica=
tion.text)
         )
+
+        finder =3D lambda driver: self.find('#hide-alert')
+        hide_button =3D self.wait_until_finder_clickable(finder)
+        hide_button.click()
+        self.wait_until_not_visible('#change-notification')
+
         # check Machine table feature(show/hide column, pagination)
         self._navigate_to_config_nav('machinestable', 5)
         column_list =3D [
@@ -580,7 +516,7 @@ class TestProjectPage(SeleniumFunctionalTestCase):
         """
         self._navigate_to_config_nav('layerstable', 6)
         # check title "Compatible layers" is displayed
-        self.assertTrue("Compatible layers" in self.get_page_source())
+        self.assertIn("Compatible layers", self.get_page_source())
         # Test search input
         input_text=3D'meta-tanowrt'
         self._mixin_test_table_search_input(
@@ -590,42 +526,44 @@ class TestProjectPage(SeleniumFunctionalTestCase):
             table_selector=3D'layerstable'
         )
         # check "Add layer" button works
-        self.wait_until_visible('#layerstable tbody tr', poll=3D3)
-        rows =3D self.find_all('#layerstable tbody tr')
-        layer_to_add =3D rows[0]
-        add_btn =3D layer_to_add.find_element(
-            By.XPATH,
-            '//td[@class=3D"add-del-layers"]'
-        )
+        self.wait_until_visible('#layerstable tbody tr')
+        finder =3D lambda driver: self.find_all('#layerstable tbody tr')[0=
].find_element(By.XPATH, '//td[@class=3D"add-del-layers"]/a[@data-directive=
=3D"add"]')
+        add_btn =3D self.wait_until_finder_clickable(finder)
         add_btn.click()
         # check modal is displayed
-        self.wait_until_visible('#dependencies-modal', poll=3D3)
+        self.wait_until_visible('#dependencies-modal')
         list_dependencies =3D self.find_all('#dependencies-list li')
         # click on add-layers button
-        add_layers_btn =3D self.driver.find_element(
-            By.XPATH,
-            '//form[@id=3D"dependencies-modal-form"]//button[@class=3D"btn=
 btn-primary"]'
-        )
+        finder =3D lambda driver: self.driver.find_element(By.XPATH, '//fo=
rm[@id=3D"dependencies-modal-form"]//button[@class=3D"btn btn-primary"]')
+        add_layers_btn =3D self.wait_until_finder_clickable(finder)
         add_layers_btn.click()
         self.wait_until_visible('#change-notification')
         change_notification =3D self.find('#change-notification')
-        self.assertTrue(
-            f'You have added {len(list_dependencies)+1} layers to your pro=
ject: {input_text} and its dependencies' in str(change_notification.text)
+        self.assertIn(
+            f'You have added {len(list_dependencies)+1} layers to your pro=
ject: {input_text} and its dependencies', str(change_notification.text)
         )
+
+        finder =3D lambda driver: self.find('#hide-alert')
+        hide_button =3D self.wait_until_finder_clickable(finder)
+        hide_button.click()
+        self.wait_until_not_visible('#change-notification')
+
         # check "Remove layer" button works
-        self.wait_until_visible('#layerstable tbody tr', poll=3D3)
-        rows =3D self.find_all('#layerstable tbody tr')
-        layer_to_remove =3D rows[0]
-        remove_btn =3D layer_to_remove.find_element(
-            By.XPATH,
-            '//td[@class=3D"add-del-layers"]'
-        )
+        self.wait_until_visible('#layerstable tbody tr')
+        finder =3D lambda driver: self.find_all('#layerstable tbody tr')[0=
].find_element(By.XPATH, '//td[@class=3D"add-del-layers"]/a[@data-directive=
=3D"remove"]')
+        remove_btn =3D self.wait_until_finder_clickable(finder)
         remove_btn.click()
-        self.wait_until_visible('#change-notification', poll=3D2)
+        self.wait_until_visible('#change-notification')
         change_notification =3D self.find('#change-notification')
-        self.assertTrue(
-            f'You have removed 1 layer from your project: {input_text}' in=
 str(change_notification.text)
+        self.assertIn(
+            f'You have removed 1 layer from your project: {input_text}', s=
tr(change_notification.text)
         )
+
+        finder =3D lambda driver: self.find('#hide-alert')
+        hide_button =3D self.wait_until_finder_clickable(finder)
+        hide_button.click()
+        self.wait_until_not_visible('#change-notification')
+
         # check layers table feature(show/hide column, pagination)
         self._navigate_to_config_nav('layerstable', 6)
         column_list =3D [
@@ -656,7 +594,7 @@ class TestProjectPage(SeleniumFunctionalTestCase):
         """
         self._navigate_to_config_nav('distrostable', 7)
         # check title "Compatible distros" is displayed
-        self.assertTrue("Compatible Distros" in self.get_page_source())
+        self.assertIn("Compatible Distros", self.get_page_source())
         # Test search input
         input_text=3D'poky-altcfg'
         self._mixin_test_table_search_input(
@@ -666,17 +604,14 @@ class TestProjectPage(SeleniumFunctionalTestCase):
             table_selector=3D'distrostable'
         )
         # check "Add distro" button works
-        rows =3D self.find_all('#distrostable tbody tr')
-        distro_to_add =3D rows[0]
-        add_btn =3D distro_to_add.find_element(
-            By.XPATH,
-            '//td[@class=3D"add-del-layers"]//a[1]'
-        )
+        self.wait_until_visible(".add-del-layers")
+        finder =3D lambda driver: self.find_all('#distrostable tbody tr')[=
0].find_element(By.XPATH, '//td[@class=3D"add-del-layers"]')
+        add_btn =3D self.wait_until_finder_clickable(finder)
         add_btn.click()
-        self.wait_until_visible('#change-notification', poll=3D2)
+        self.wait_until_visible('#change-notification')
         change_notification =3D self.find('#change-notification')
-        self.assertTrue(
-            f'You have changed the distro to: {input_text}' in str(change_=
notification.text)
+        self.assertIn(
+            f'You have changed the distro to: {input_text}', str(change_no=
tification.text)
         )
         # check distro table feature(show/hide column, pagination)
         self._navigate_to_config_nav('distrostable', 7)
@@ -699,7 +634,7 @@ class TestProjectPage(SeleniumFunctionalTestCase):
         )
=20
     def test_single_layer_page(self):
-        """ Test layer page
+        """ Test layer details page using meta-poky as an example (assumes=
 is added to start with)
             - Check if title is displayed
             - Check add/remove layer button works
             - Check tabs(layers, recipes, machines) are displayed
@@ -708,45 +643,62 @@ class TestProjectPage(SeleniumFunctionalTestCase):
                 - Check layer summary
                 - Check layer description
         """
-        url =3D reverse("layerdetails", args=3D(TestProjectPage.project_id=
, 8))
-        self.get(url)
+        self._navigate_to_config_nav('layerstable', 6)
+        layer_link =3D self.driver.find_element(By.XPATH, '//tr/td[@class=
=3D"layer__name"]/a[contains(text(),"meta-poky")]')
+        layer_link.click()
         self.wait_until_visible('.page-header')
         # check title is displayed
         self.assertTrue(self.find('.page-header h1').is_displayed())
=20
-        # check add layer button works
-        remove_layer_btn =3D self.find('#add-remove-layer-btn')
+        # check remove layer button works
+        finder =3D lambda driver: self.find('#add-remove-layer-btn')
+        remove_layer_btn =3D self.wait_until_finder_clickable(finder)
         remove_layer_btn.click()
-        self.wait_until_visible('#change-notification', poll=3D2)
+        self.wait_until_visible('#change-notification')
         change_notification =3D self.find('#change-notification')
-        self.assertTrue(
-            f'You have removed 1 layer from your project' in str(change_no=
tification.text)
+        self.assertIn(
+            f'You have removed 1 layer from your project', str(change_noti=
fication.text)
         )
-        # check add layer button works, 18 is the random layer id
-        add_layer_btn =3D self.find('#add-remove-layer-btn')
+        finder =3D lambda driver: self.find('#hide-alert')
+        hide_button =3D self.wait_until_finder_clickable(finder)
+        hide_button.click()
+        # check add layer button works
+        self.wait_until_not_visible('#change-notification')
+        finder =3D lambda driver: self.find('#add-remove-layer-btn')
+        add_layer_btn =3D self.wait_until_finder_clickable(finder)
         add_layer_btn.click()
         self.wait_until_visible('#change-notification')
         change_notification =3D self.find('#change-notification')
-        self.assertTrue(
-            f'You have added 1 layer to your project' in str(change_notifi=
cation.text)
+        self.assertIn(
+            f'You have added 1 layer to your project', str(change_notifica=
tion.text)
         )
+        finder =3D lambda driver: self.find('#hide-alert')
+        hide_button =3D self.wait_until_finder_clickable(finder)
+        hide_button.click()
+        self.wait_until_not_visible('#change-notification')
         # check tabs(layers, recipes, machines) are displayed
         tabs =3D self.find_all('.nav-tabs li')
         self.assertEqual(len(tabs), 3)
         # Check first tab
         tabs[0].click()
-        self.assertTrue(
-            'active' in str(self.find('#information').get_attribute('class=
'))
+        self.assertIn(
+            'active', str(self.find('#information').get_attribute('class')=
)
         )
-        # Check second tab
+        # Check second tab (recipes)
+        self.wait_until_visible('.nav-tabs')
+        # Ensure page is scrolled to the top
+        self.driver.execute_script('window.scrollTo({behavior: "instant", =
top: 0, left: 0})')
         tabs[1].click()
-        self.assertTrue(
-            'active' in str(self.find('#recipes').get_attribute('class'))
+        self.assertIn(
+            'active', str(self.find('#recipes').get_attribute('class'))
         )
-        # Check third tab
+        # Check third tab (machines)
+        self.wait_until_visible('.nav-tabs')
+        # Ensure page is scrolled to the top
+        self.driver.execute_script('window.scrollTo({behavior: "instant", =
top: 0, left: 0})')
         tabs[2].click()
-        self.assertTrue(
-            'active' in str(self.find('#machines').get_attribute('class'))
+        self.assertIn(
+            'active', str(self.find('#machines').get_attribute('class'))
         )
         # Check left section is displayed
         section =3D self.find('.well')
@@ -755,9 +707,13 @@ class TestProjectPage(SeleniumFunctionalTestCase):
             section.find_element(By.XPATH, '//h2[1]').is_displayed()
         )
         # Check layer summary
-        self.assertTrue("Summary" in section.text)
+        self.assertIn("Summary", section.text)
         # Check layer description
-        self.assertTrue("Description" in section.text)
+        self.assertIn("Description", section.text)
+
+@pytest.mark.django_db
+@pytest.mark.order("last")
+class TestProjectPageRecipes(TestProjectPageBase):
=20
     def test_single_recipe_page(self):
         """ Test recipe page
@@ -767,7 +723,12 @@ class TestProjectPage(SeleniumFunctionalTestCase):
                 - Check recipe: name, summary, description, Version, Secti=
on,
                 License, Approx. packages included, Approx. size, Recipe f=
ile
         """
-        url =3D reverse("recipedetails", args=3D(TestProjectPage.project_i=
d, 53428))
+        # Use a recipe which is likely to exist in the layer index but not=
 enabled
+        # in poky out the box - xen-image-minimal from meta-virtualization
+        self._navigate_to_project_page()
+        prj =3D Project.objects.get(pk=3DTestProjectPageBase.project_id)
+        recipe_id =3D prj.get_all_compatible_recipes().get(name=3D"xen-ima=
ge-minimal").pk
+        url =3D reverse("recipedetails", args=3D(TestProjectPageBase.proje=
ct_id, recipe_id))
         self.get(url)
         self.wait_until_visible('.page-header')
         # check title is displayed
@@ -782,11 +743,33 @@ class TestProjectPage(SeleniumFunctionalTestCase):
             section.find_element(By.XPATH, '//h2[1]').is_displayed()
         )
         # Check recipe sections details info are displayed
-        self.assertTrue("Summary" in section.text)
-        self.assertTrue("Description" in section.text)
-        self.assertTrue("Version" in section.text)
-        self.assertTrue("Section" in section.text)
-        self.assertTrue("License" in section.text)
-        self.assertTrue("Approx. packages included" in section.text)
-        self.assertTrue("Approx. package size" in section.text)
-        self.assertTrue("Recipe file" in section.text)
+        self.assertIn("Summary", section.text)
+        self.assertIn("Description", section.text)
+        self.assertIn("Version", section.text)
+        self.assertIn("Section", section.text)
+        self.assertIn("License", section.text)
+        self.assertIn("Approx. packages included", section.text)
+        self.assertIn("Approx. package size", section.text)
+        self.assertIn("Recipe file", section.text)
+
+    def test_image_recipe_editColumn(self):
+        """ Test the edit column feature in image recipe table on project =
page """
+        self._get_create_builds(success=3D10, failure=3D10)
+
+        url =3D reverse('projectimagerecipes', args=3D(TestProjectPageBase=
.project_id,))
+        self.get(url)
+        self.wait_until_present('#imagerecipestable tbody tr')
+
+        column_list =3D [
+            'get_description_or_summary', 'layer_version__get_vcs_referenc=
e',
+            'layer_version__layer__name', 'license', 'recipe-file', 'secti=
on',
+            'version'
+        ]
+
+        # Check that we can hide the edit column
+        self._mixin_test_table_edit_column(
+            'imagerecipestable',
+            'edit-columns-button',
+            [f'checkbox-{column}' for column in column_list]
+        )
+
diff --git a/bitbake/lib/toaster/tests/functional/test_project_page_tab_con=
fig.py b/bitbake/lib/toaster/tests/functional/test_project_page_tab_config.=
py
index eb905ddf..0c1100c5 100644
--- a/bitbake/lib/toaster/tests/functional/test_project_page_tab_config.py
+++ b/bitbake/lib/toaster/tests/functional/test_project_page_tab_config.py
@@ -7,72 +7,27 @@
 #
=20
 import string
-import random
+import time
 import pytest
 from django.urls import reverse
 from selenium.webdriver import Keys
 from selenium.webdriver.support.select import Select
 from selenium.common.exceptions import ElementClickInterceptedException, N=
oSuchElementException, TimeoutException
-from orm.models import Project
-from tests.functional.functional_helpers import SeleniumFunctionalTestCase
+from tests.functional.functional_helpers import SeleniumFunctionalTestCase=
, MASTER_PROJECT_ID
 from selenium.webdriver.common.by import By
=20
 from .utils import get_projectId_from_url, wait_until_build, wait_until_bu=
ild_cancelled
=20
-
-@pytest.mark.django_db
-@pytest.mark.order("last")
-class TestProjectConfigTab(SeleniumFunctionalTestCase):
+class TestProjectConfigTabBase(SeleniumFunctionalTestCase):
     PROJECT_NAME =3D 'TestProjectConfigTab'
     project_id =3D None
=20
-    def _create_project(self, project_name, **kwargs):
-        """ Create/Test new project using:
-          - Project Name: Any string
-          - Release: Any string
-          - Merge Toaster settings: True or False
-        """
-        release =3D kwargs.get('release', '3')
-        self.get(reverse('newproject'))
-        self.wait_until_visible('#new-project-name')
-        self.find("#new-project-name").send_keys(project_name)
-        select =3D Select(self.find("#projectversion"))
-        select.select_by_value(release)
-
-        # check merge toaster settings
-        checkbox =3D self.find('.checkbox-mergeattr')
-        if not checkbox.is_selected():
-            checkbox.click()
-
-        if self.PROJECT_NAME !=3D 'TestProjectConfigTab':
-            # Reset project name if it's not the default one
-            self.PROJECT_NAME =3D 'TestProjectConfigTab'
-
-        self.find("#create-project-button").click()
-
-        try:
-            self.wait_until_visible('#hint-error-project-name', poll=3D3)
-            url =3D reverse('project', args=3D(TestProjectConfigTab.projec=
t_id, ))
-            self.get(url)
-            self.wait_until_visible('#config-nav', poll=3D3)
-        except TimeoutException:
-            self.wait_until_visible('#config-nav', poll=3D3)
-
-    def _random_string(self, length):
-        return ''.join(
-            random.choice(string.ascii_letters) for _ in range(length)
-        )
-
     def _navigate_to_project_page(self):
         # Navigate to project page
-        if TestProjectConfigTab.project_id is None:
-            self._create_project(project_name=3Dself._random_string(10))
-            current_url =3D self.driver.current_url
-            TestProjectConfigTab.project_id =3D get_projectId_from_url(
-                current_url)
-        else:
-            url =3D reverse('project', args=3D(TestProjectConfigTab.projec=
t_id,))
-            self.get(url)
+        if TestProjectConfigTabBase.project_id is None:
+            TestProjectConfigTabBase.project_id =3D self.create_new_projec=
t(self.PROJECT_NAME, MASTER_PROJECT_ID, None, True)
+        url =3D reverse('project', args=3D(TestProjectConfigTabBase.projec=
t_id,))
+        self.get(url)
         self.wait_until_visible('#config-nav')
=20
     def _create_builds(self):
@@ -88,8 +43,8 @@ class TestProjectConfigTab(SeleniumFunctionalTestCase):
             '//div[@id=3D"latest-builds"]/div',
         )
         last_build =3D lastest_builds[0]
-        self.assertTrue(
-            'foo' in str(last_build.text)
+        self.assertIn(
+            'foo', str(last_build.text)
         )
         last_build =3D lastest_builds[0]
         try:
@@ -114,6 +69,8 @@ class TestProjectConfigTab(SeleniumFunctionalTestCase):
         config_nav =3D self.find('#config-nav')
         return config_nav.find_elements(By.TAG_NAME, 'li')[index]
=20
+class TestProjectConfigTab(TestProjectConfigTabBase):
+
     def test_project_config_nav(self):
         """ Test project config tab navigation:
         - Check if the menu is displayed and contains the right elements:
@@ -138,48 +95,48 @@ class TestProjectConfigTab(SeleniumFunctionalTestCase)=
:
=20
         def check_config_nav_item(index, item_name, url):
             item =3D _get_config_nav_item(index)
-            self.assertTrue(item_name in item.text)
-            self.assertTrue(item.get_attribute('class') =3D=3D 'active')
-            self.assertTrue(url in self.driver.current_url)
+            self.assertIn(item_name, item.text)
+            self.assertEqual(item.get_attribute('class'), 'active')
+            self.assertIn(url, self.driver.current_url)
=20
         # check if the menu contains the right elements
         # COMPATIBLE METADATA
         compatible_metadata =3D _get_config_nav_item(1)
-        self.assertTrue(
-            "compatible metadata" in compatible_metadata.text.lower()
+        self.assertIn(
+            "compatible metadata", compatible_metadata.text.lower()
         )
         # EXTRA CONFIGURATION
         extra_configuration =3D _get_config_nav_item(8)
-        self.assertTrue(
-            "extra configuration" in extra_configuration.text.lower()
+        self.assertIn(
+            "extra configuration", extra_configuration.text.lower()
         )
         # Actions
         actions =3D _get_config_nav_item(10)
-        self.assertTrue("actions" in str(actions.text).lower())
+        self.assertIn("actions", str(actions.text).lower())
=20
         conf_nav_list =3D [
             # config
             [0, 'Configuration',
-                f"/toastergui/project/{TestProjectConfigTab.project_id}"],
+                f"/toastergui/project/{TestProjectConfigTabBase.project_id=
}"],
             # custom images
             [2, 'Custom images',
-                f"/toastergui/project/{TestProjectConfigTab.project_id}/cu=
stomimages"],
+                f"/toastergui/project/{TestProjectConfigTabBase.project_id=
}/customimages"],
             # image recipes
             [3, 'Image recipes',
-                f"/toastergui/project/{TestProjectConfigTab.project_id}/im=
ages"],
+                f"/toastergui/project/{TestProjectConfigTabBase.project_id=
}/images"],
             # software recipes
             [4, 'Software recipes',
-                f"/toastergui/project/{TestProjectConfigTab.project_id}/so=
ftwarerecipes"],
+                f"/toastergui/project/{TestProjectConfigTabBase.project_id=
}/softwarerecipes"],
             # machines
             [5, 'Machines',
-                f"/toastergui/project/{TestProjectConfigTab.project_id}/ma=
chines"],
+                f"/toastergui/project/{TestProjectConfigTabBase.project_id=
}/machines"],
             # layers
             [6, 'Layers',
-                f"/toastergui/project/{TestProjectConfigTab.project_id}/la=
yers"],
+                f"/toastergui/project/{TestProjectConfigTabBase.project_id=
}/layers"],
             # distro
             [7, 'Distros',
-                f"/toastergui/project/{TestProjectConfigTab.project_id}/di=
stros"],
-            #  [9, 'BitBake variables', f"/toastergui/project/{TestProject=
ConfigTab.project_id}/configuration"],  # bitbake variables
+                f"/toastergui/project/{TestProjectConfigTabBase.project_id=
}/distros"],
+            #  [9, 'BitBake variables', f"/toastergui/project/{TestProject=
ConfigTabBase.project_id}/configuration"],  # bitbake variables
         ]
         for index, item_name, url in conf_nav_list:
             item =3D _get_config_nav_item(index)
@@ -253,7 +210,7 @@ class TestProjectConfigTab(SeleniumFunctionalTestCase):
         def test_show_rows(row_to_show, show_row_link):
             # Check that we can show rows =3D=3D row_to_show
             show_row_link.select_by_value(str(row_to_show))
-            self.wait_until_visible('#imagerecipestable tbody tr', poll=3D=
3)
+            self.wait_until_visible('#imagerecipestable tbody tr')
             # check at least some rows are visible
             self.assertTrue(
                 len(self.find_all('#imagerecipestable tbody tr'))  > 0
@@ -299,9 +256,11 @@ class TestProjectConfigTab(SeleniumFunctionalTestCase)=
:
                     - meta-poky
                     - meta-yocto-bsp
         """
-        # Create a new project for this test
-        project_name =3D self._random_string(10)
-        self._create_project(project_name=3Dproject_name)
+        project_id =3D self.create_new_project(self.PROJECT_NAME + "-ST", =
MASTER_PROJECT_ID, None, True)
+        url =3D reverse('project', args=3D(project_id,))
+        self.get(url)
+        self.wait_until_visible('#config-nav')
+
         # check if the menu is displayed
         self.wait_until_visible('#project-page')
         block_l =3D self.driver.find_element(
@@ -313,7 +272,7 @@ class TestProjectConfigTab(SeleniumFunctionalTestCase):
         def check_machine_distro(self, item_name, new_item_name, block_id)=
:
             block =3D self.find(f'#{block_id}')
             title =3D block.find_element(By.TAG_NAME, 'h3')
-            self.assertTrue(item_name.capitalize() in title.text)
+            self.assertIn(item_name.capitalize(), title.text)
             edit_btn =3D self.find(f'#change-{item_name}-toggle')
             edit_btn.click()
             self.wait_until_visible(f'#{item_name}-change-input')
@@ -324,12 +283,15 @@ class TestProjectConfigTab(SeleniumFunctionalTestCase=
):
             change_btn.click()
             self.wait_until_visible(f'#project-{item_name}-name')
             project_name =3D self.find(f'#project-{item_name}-name')
-            self.assertTrue(new_item_name in project_name.text)
+            self.assertIn(new_item_name, project_name.text)
             # check change notificaiton is displayed
             change_notification =3D self.find('#change-notification')
-            self.assertTrue(
-                f'You have changed the {item_name} to: {new_item_name}' in=
 change_notification.text
+            self.assertIn(
+                f'You have changed the {item_name} to: {new_item_name}', c=
hange_notification.text
             )
+            hide_button =3D self.find('#hide-alert')
+            hide_button.click()
+            self.wait_until_not_visible('#change-notification')
=20
         # Machine
         check_machine_distro(self, 'machine', 'qemux86-64', 'machine-secti=
on')
@@ -338,97 +300,51 @@ class TestProjectConfigTab(SeleniumFunctionalTestCase=
):
=20
         # Project release
         title =3D project_release.find_element(By.TAG_NAME, 'h3')
-        self.assertTrue("Project release" in title.text)
-        self.assertTrue(
-            "Yocto Project master" in self.find('#project-release-title').=
text
+        self.assertIn("Project release", title.text)
+        self.assertIn(
+            "Yocto Project master", self.find('#project-release-title').te=
xt
         )
         # Layers
         title =3D layers.find_element(By.TAG_NAME, 'h3')
-        self.assertTrue("Layers" in title.text)
+        self.assertIn("Layers", title.text)
+        self.wait_until_clickable('#layer-add-input')
         # check at least three layers are displayed
         # openembedded-core
         # meta-poky
         # meta-yocto-bsp
-        layers_list =3D layers.find_element(By.ID, 'layers-in-project-list=
')
-        layers_list_items =3D layers_list.find_elements(By.TAG_NAME, 'li')
+        layer_list_items =3D []
+        starttime =3D time.time()
+        while len(layer_list_items) < 3:
+            layers_list =3D self.driver.find_element(By.ID, 'layers-in-pro=
ject-list')
+            layer_list_items =3D layers_list.find_elements(By.TAG_NAME, 'l=
i')
+            if time.time() > (starttime + 30):
+                self.fail("Layer list didn't contain at least 3 items with=
in 30s (contained %d)" % len(layer_list_items))
+
         # remove all layers except the first three layers
-        for i in range(3, len(layers_list_items)):
-            layers_list_items[i].find_element(By.TAG_NAME, 'span').click()
+        for i in range(3, len(layer_list_items)):
+            layer_list_items[i].find_element(By.TAG_NAME, 'span').click()
+
         # check can add a layer if exists
         add_layer_input =3D layers.find_element(By.ID, 'layer-add-input')
         add_layer_input.send_keys('meta-oe')
         self.wait_until_visible('#layer-container > form > div > span > di=
v')
-        dropdown_item =3D self.driver.find_element(
-            By.XPATH,
-            '//*[@id=3D"layer-container"]/form/div/span/div'
-        )
-        try:
-            dropdown_item.click()
-        except ElementClickInterceptedException:
-            self.skipTest(
-                "layer-container dropdown item click intercepted. Element =
not properly visible.")
+        self.wait_until_visible('.dropdown-menu')
+        finder =3D lambda driver: driver.find_element(By.XPATH, '//*[@id=
=3D"layer-container"]/form/div/span/div/div/div')
+        dropdown_item =3D self.wait_until_finder_clickable(finder)
+        dropdown_item.click()
+        self.wait_until_clickable('#add-layer-btn')
         add_layer_btn =3D layers.find_element(By.ID, 'add-layer-btn')
         add_layer_btn.click()
         self.wait_until_visible('#layers-in-project-list')
-        # check layer is added
-        layers_list_items =3D layers_list.find_elements(By.TAG_NAME, 'li')
-        self.assertTrue(len(layers_list_items) =3D=3D 4)
=20
-    def test_most_build_recipes(self):
-        """ Test most build recipes block contains"""
-        def rebuild_from_most_build_recipes(recipe_list_items):
-            checkbox =3D recipe_list_items[0].find_element(By.TAG_NAME, 'i=
nput')
-            checkbox.click()
-            build_btn =3D self.find('#freq-build-btn')
-            build_btn.click()
-            self.wait_until_visible('#latest-builds')
-            wait_until_build(self, 'queued cloning starting parsing failed=
')
-            lastest_builds =3D self.driver.find_elements(
-                By.XPATH,
-                '//div[@id=3D"latest-builds"]/div'
-            )
-            self.assertTrue(len(lastest_builds) >=3D 2)
-            last_build =3D lastest_builds[0]
-            try:
-                cancel_button =3D last_build.find_element(
-                    By.XPATH,
-                    '//span[@class=3D"cancel-build-btn pull-right alert-li=
nk"]',
-                )
-                cancel_button.click()
-            except NoSuchElementException:
-                # Skip if the build is already cancelled
-                pass
-            wait_until_build_cancelled(self)
-        # Create a new project for remaining asserts
-        project_name =3D self._random_string(10)
-        self._create_project(project_name=3Dproject_name, release=3D'2')
-        current_url =3D self.driver.current_url
-        TestProjectConfigTab.project_id =3D get_projectId_from_url(current=
_url)
-        url =3D current_url.split('?')[0]
-
-        # Create a new builds
-        self._create_builds()
-
-        # back to project page
-        self.driver.get(url)
-
-        self.wait_until_visible('#project-page', poll=3D3)
-
-        # Most built recipes
-        most_built_recipes =3D self.driver.find_element(
-            By.XPATH, '//*[@id=3D"project-page"]/div[1]/div[3]')
-        title =3D most_built_recipes.find_element(By.TAG_NAME, 'h3')
-        self.assertTrue("Most built recipes" in title.text)
-        # check can select a recipe and build it
-        self.wait_until_visible('#freq-build-list', poll=3D3)
-        recipe_list =3D self.find('#freq-build-list')
-        recipe_list_items =3D recipe_list.find_elements(By.TAG_NAME, 'li')
-        self.assertTrue(
-            len(recipe_list_items) > 0,
-            msg=3D"Any recipes found in the most built recipes list",
-        )
-        rebuild_from_most_build_recipes(recipe_list_items)
-        TestProjectConfigTab.project_id =3D None  # reset project id
+        # check layer is added
+        layer_list_items =3D []
+        starttime =3D time.time()
+        while len(layer_list_items) < 4:
+            layers_list =3D self.driver.find_element(By.ID, 'layers-in-pro=
ject-list')
+            layer_list_items =3D layers_list.find_elements(By.TAG_NAME, 'l=
i')
+            if time.time() > (starttime + 30):
+                self.fail("Layer list didn't contain at least 4 items with=
in 30s (contained %d)" % len(layer_list_items))
=20
     def test_project_page_tab_importlayer(self):
         """ Test project page tab import layer """
@@ -466,42 +382,42 @@ class TestProjectConfigTab(SeleniumFunctionalTestCase=
):
         layers =3D block_l.find_element(By.ID, 'layer-container')
         layers_list =3D layers.find_element(By.ID, 'layers-in-project-list=
')
         layers_list_items =3D layers_list.find_elements(By.TAG_NAME, 'li')
-        self.assertTrue(
-            'meta-fake' in str(layers_list_items[-1].text)
+        self.assertIn(
+            'meta-fake', str(layers_list_items[-1].text)
         )
=20
     def test_project_page_custom_image_no_image(self):
         """ Test project page tab "New custom image" when no custom image =
"""
-        project_name =3D self._random_string(10)
-        self._create_project(project_name=3Dproject_name)
-        current_url =3D self.driver.current_url
-        TestProjectConfigTab.project_id =3D get_projectId_from_url(current=
_url)
+        project_id =3D self.create_new_project(self.PROJECT_NAME + "-Custo=
mImage", MASTER_PROJECT_ID, None, True)
+        url =3D reverse('project', args=3D(project_id,))
+        self.get(url)
+        self.wait_until_visible('#config-nav')
+
         # navigate to "Custom image" tab
         custom_image_section =3D self._get_config_nav_item(2)
         custom_image_section.click()
         self.wait_until_visible('#empty-state-customimagestable')
=20
         # Check message when no custom image
-        self.assertTrue(
-            "You have not created any custom images yet." in str(
+        self.assertIn(
+            "You have not created any custom images yet.", str(
                 self.find('#empty-state-customimagestable').text
             )
         )
         div_empty_msg =3D self.find('#empty-state-customimagestable')
         link_create_custom_image =3D div_empty_msg.find_element(
             By.TAG_NAME, 'a')
-        self.assertTrue(TestProjectConfigTab.project_id is not None)
-        self.assertTrue(
-            f"/toastergui/project/{TestProjectConfigTab.project_id}/newcus=
tomimage" in str(
+        self.assertTrue(TestProjectConfigTabBase.project_id is not None)
+        self.assertIn(
+            f"/toastergui/project/{project_id}/newcustomimage", str(
                 link_create_custom_image.get_attribute('href')
             )
         )
-        self.assertTrue(
-            "Create your first custom image" in str(
+        self.assertIn(
+            "Create your first custom image", str(
                 link_create_custom_image.text
             )
         )
-        TestProjectConfigTab.project_id =3D None  # reset project id
=20
     def test_project_page_image_recipe(self):
         """ Test project page section images
@@ -526,3 +442,66 @@ class TestProjectConfigTab(SeleniumFunctionalTestCase)=
:
         self.wait_until_visible('#imagerecipestable tbody tr')
         rows =3D self.find_all('#imagerecipestable tbody tr')
         self.assertTrue(len(rows) > 0)
+
+@pytest.mark.django_db
+@pytest.mark.order("last")=20
+class TestProjectConfigTabDB(TestProjectConfigTabBase):
+
+    def test_most_build_recipes(self):
+        """ Test most build recipes block contains"""
+        def rebuild_from_most_build_recipes(recipe_list_items):
+            checkbox =3D recipe_list_items[0].find_element(By.TAG_NAME, 'i=
nput')
+            checkbox.click()
+            build_btn =3D self.find('#freq-build-btn')
+            build_btn.click()
+            self.wait_until_visible('#latest-builds')
+            wait_until_build(self, 'queued cloning starting parsing failed=
')
+            lastest_builds =3D self.driver.find_elements(
+                By.XPATH,
+                '//div[@id=3D"latest-builds"]/div'
+            )
+            self.assertTrue(len(lastest_builds) >=3D 2)
+            last_build =3D lastest_builds[0]
+            try:
+                cancel_button =3D last_build.find_element(
+                    By.XPATH,
+                    '//span[@class=3D"cancel-build-btn pull-right alert-li=
nk"]',
+                )
+                cancel_button.click()
+            except NoSuchElementException:
+                # Skip if the build is already cancelled
+                pass
+            wait_until_build_cancelled(self)
+
+        # Create a new project for remaining asserts
+        project_id =3D self.create_new_project(self.PROJECT_NAME + "-MostB=
uilt", MASTER_PROJECT_ID, None, True)
+        url =3D reverse('project', args=3D(project_id,))
+        self.get(url)
+        self.wait_until_visible('#config-nav')
+
+        current_url =3D self.driver.current_url
+        url =3D current_url.split('?')[0]
+
+        # Create a new builds
+        self._create_builds()
+
+        # back to project page
+        self.driver.get(url)
+
+        self.wait_until_visible('#project-page')
+
+        # Most built recipes
+        most_built_recipes =3D self.driver.find_element(
+            By.XPATH, '//*[@id=3D"project-page"]/div[1]/div[3]')
+        title =3D most_built_recipes.find_element(By.TAG_NAME, 'h3')
+        self.assertIn("Most built recipes", title.text)
+        # check can select a recipe and build it
+        self.wait_until_visible('#freq-build-list')
+        recipe_list =3D self.find('#freq-build-list')
+        recipe_list_items =3D recipe_list.find_elements(By.TAG_NAME, 'li')
+        self.assertTrue(
+            len(recipe_list_items) > 0,
+            msg=3D"No recipes found in the most built recipes list",
+        )
+        rebuild_from_most_build_recipes(recipe_list_items)
+
diff --git a/bitbake/lib/toaster/tests/functional/utils.py b/bitbake/lib/to=
aster/tests/functional/utils.py
index 7269fa18..72345aef 100644
--- a/bitbake/lib/toaster/tests/functional/utils.py
+++ b/bitbake/lib/toaster/tests/functional/utils.py
@@ -8,7 +8,7 @@
=20
=20
 from time import sleep
-from selenium.common.exceptions import NoSuchElementException, StaleElemen=
tReferenceException, TimeoutException
+from selenium.common.exceptions import NoSuchElementException, StaleElemen=
tReferenceException, TimeoutException, WebDriverException
 from selenium.webdriver.common.by import By
=20
 from orm.models import Build
@@ -36,7 +36,7 @@ def wait_until_build(test_instance, state):
             if 'failed' in str(build_state).lower():
                 break
         except NoSuchElementException:
-            continue
+            pass
         except TimeoutException:
             break
         start_time +=3D 1
@@ -48,7 +48,6 @@ def wait_until_build_cancelled(test_instance):
     """
     timeout =3D 30
     start_time =3D 0
-    build =3D None
     while True:
         try:
             if start_time > timeout:
@@ -64,19 +63,17 @@ def wait_until_build_cancelled(test_instance):
             if 'failed' in str(build_state).lower():
                 break
             if 'cancelling' in str(build_state).lower():
-                # Change build state to cancelled
-                if not build:  # get build object only once
-                    build =3D Build.objects.last()
-                    build.outcome =3D Build.CANCELLED
-                    build.save()
+                pass
             if 'cancelled' in str(build_state).lower():
                 break
-        except NoSuchElementException:
-            continue
-        except StaleElementReferenceException:
-            continue
         except TimeoutException:
             break
+        except NoSuchElementException:
+            pass
+        except StaleElementReferenceException:
+            pass
+        except WebDriverException:
+            pass
         start_time +=3D 1
         sleep(1) # take a breath and try again
=20
diff --git a/bitbake/lib/toaster/tests/toaster-tests-requirements.txt b/bit=
bake/lib/toaster/tests/toaster-tests-requirements.txt
index 71cc0834..86ba9b42 100644
--- a/bitbake/lib/toaster/tests/toaster-tests-requirements.txt
+++ b/bitbake/lib/toaster/tests/toaster-tests-requirements.txt
@@ -1,7 +1,9 @@
 selenium>=3D4.13.0
-pytest=3D=3D7.4.2
+pytest>=3D9.0.3
 pytest-django=3D=3D4.5.2
 pytest-env=3D=3D1.1.0
 pytest-html=3D=3D4.0.2
 pytest-metadata=3D=3D3.0.0
 pytest-order=3D=3D1.1.0
+requests
+
diff --git a/bitbake/lib/toaster/tests/views/test_views.py b/bitbake/lib/to=
aster/tests/views/test_views.py
index e1adfcf8..5e64e819 100644
--- a/bitbake/lib/toaster/tests/views/test_views.py
+++ b/bitbake/lib/toaster/tests/views/test_views.py
@@ -27,7 +27,7 @@ import toastergui
=20
 from toastergui.tables import SoftwareRecipesTable
 import json
-from bs4 import BeautifulSoup
+from bb._vendor.bs4 import BeautifulSoup
 import string
=20
 PROJECT_NAME =3D "test project"
diff --git a/bitbake/lib/toaster/toastergui/templates/layerdetails.html b/b=
itbake/lib/toaster/toastergui/templates/layerdetails.html
index 923ca3bf..2f23156a 100644
--- a/bitbake/lib/toaster/toastergui/templates/layerdetails.html
+++ b/bitbake/lib/toaster/toastergui/templates/layerdetails.html
@@ -328,12 +328,12 @@
             {{layerversion.layer.summary|def=
ault_if_none:''}}
             

               

-                
+                
               

               
               Cancel
             

-            
+            
             
             
             

@@ -344,12 +344,12 @@
             {{layerversion.layer.description=
|default_if_none:''}}
             

               

-                
               

               
               Cancel
             

-            
+            
             
             
             {% if layerversion.layer_source =3D=3D layer_source.TYPE_LAYER=
INDEX  %}
diff --git a/bitbake/lib/toaster/toastergui/templatetags/projecttags.py b/b=
itbake/lib/toaster/toastergui/templatetags/projecttags.py
index bd398f00..aee9bbcd 100644
--- a/bitbake/lib/toaster/toastergui/templatetags/projecttags.py
+++ b/bitbake/lib/toaster/toastergui/templatetags/projecttags.py
@@ -233,7 +233,6 @@ def filter_sizeovertotal(package_object, total_size):
=20
     return '{:.1%}'.format(float(size)/float(total_size))
=20
-from django.utils.safestring import mark_safe
 @register.filter
 def format_vpackage_rowclass(size):
     if size =3D=3D -1:
diff --git a/bitbake/lib/toaster/toastergui/views.py b/bitbake/lib/toaster/=
toastergui/views.py
index 40aed265..3192b727 100644
--- a/bitbake/lib/toaster/toastergui/views.py
+++ b/bitbake/lib/toaster/toastergui/views.py
@@ -8,6 +8,7 @@
=20
 import ast
 import re
+import shlex
 import subprocess
 import sys
=20
@@ -372,7 +373,6 @@ def _get_parameters_values(request, default_count, defa=
ult_order):
 # set cookies for parameters. this is usefull in case parameters are set
 # manually from the GET values of the link
 def _set_parameters_values(pagesize, orderby, request):
-    from django.urls import resolve
     current_url =3D resolve(request.path_info).url_name
     request.session['%s_count' % current_url] =3D pagesize
     request.session['%s_orderby' % current_url] =3Dorderby
@@ -699,7 +699,6 @@ class LazyEncoder(json.JSONEncoder):
         return super(LazyEncoder, self).default(obj)
=20
 from toastergui.templatetags.projecttags import filtered_filesizeformat
-import os
 def _get_dir_entries(build_id, target_id, start):
     node_str =3D {
         Target_File.ITYPE_REGULAR   : '-',
@@ -1558,7 +1557,7 @@ if True:
     # perform the final actions for the project specific page
     def project_specific_finalize(cmnd, pid):
         project =3D Project.objects.get(pk=3Dpid)
-        callback =3D project.get_variable(Project.PROJECT_SPECIFIC_CALLBAC=
K)
+        callback =3D shlex.split(project.get_variable(Project.PROJECT_SPEC=
IFIC_CALLBACK) or "")
         if "update" =3D=3D cmnd:
             # Delete all '_PROJECT_PREPARE_' builds
             for b in Build.objects.all().filter(project=3Dproject):
@@ -1571,16 +1570,15 @@ if True:
                     management.call_command('builddelete', str(b.id), inte=
ractive=3DFalse)
             # perform callback at this last moment if defined, in case Toa=
ster gets shutdown next
             default_target =3D project.get_variable(Project.PROJECT_SPECIF=
IC_DEFAULTIMAGE)
-            if callback:
-                callback =3D callback.replace("",default_target)
+            for i in range(len(callback)):
+                callback[i] =3D callback[i].replace("", default_tar=
get)
         if "cancel" =3D=3D cmnd:
-            if callback:
-                callback =3D callback.replace("","none")
-                callback =3D callback.replace("--update","--cancel")
+            for i in range(len(callback)):
+                callback[i] =3D callback[i].replace("", "none")
+                callback[i] =3D callback[i].replace("--update", "--cancel"=
)
         # perform callback at this last moment if defined, in case this To=
aster gets shutdown next
-        ret =3D ''
         if callback:
-            ret =3D os.system('bash -c "%s"' % callback)
+            subprocess.run(callback, shell=3DFalse)
             project.set_variable(Project.PROJECT_SPECIFIC_CALLBACK,'')
         # Delete the temp project specific variables
         project.set_variable(Project.PROJECT_SPECIFIC_ISNEW,'')
diff --git a/bitbake/lib/toaster/toastermain/settings.py b/bitbake/lib/toas=
ter/toastermain/settings.py
index e06adc5a..9d2dbc46 100644
--- a/bitbake/lib/toaster/toastermain/settings.py
+++ b/bitbake/lib/toaster/toastermain/settings.py
@@ -229,8 +229,8 @@ from os.path import dirname as DN
 SITE_ROOT=3DDN(DN(os.path.abspath(__file__)))
=20
 import subprocess
-TOASTER_BRANCH =3D subprocess.Popen('git branch | grep "^* " | tr -d "* "'=
, cwd =3D SITE_ROOT, shell=3DTrue, stdout=3Dsubprocess.PIPE, stderr=3Dsubpr=
ocess.PIPE).communicate()[0]
-TOASTER_REVISION =3D subprocess.Popen('git rev-parse HEAD ', cwd =3D SITE_=
ROOT, shell=3DTrue, stdout=3Dsubprocess.PIPE, stderr=3Dsubprocess.PIPE).com=
municate()[0]
+TOASTER_BRANCH =3D subprocess.check_output(['git', 'branch', '--show-curre=
nt'], cwd=3DSITE_ROOT, text=3DTrue).strip()
+TOASTER_REVISION =3D subprocess.check_output(['git', 'rev-parse', 'HEAD'],=
 cwd=3DSITE_ROOT, text=3DTrue).strip()
=20
 ROOT_URLCONF =3D 'toastermain.urls'
=20
@@ -298,7 +298,6 @@ SOUTH_TESTS_MIGRATE =3D False
=20
 # We automatically detect and install applications here if
 # they have a 'models.py' or 'views.py' file
-import os
 currentdir =3D os.path.dirname(__file__)
 for t in os.walk(os.path.dirname(currentdir)):
     modulename =3D os.path.basename(t[0])
diff --git a/bitbake/packaging-pypi/bitbake-setup/BUILD.md b/bitbake/packag=
ing-pypi/bitbake-setup/BUILD.md
new file mode 100644
index 00000000..adb8d183
--- /dev/null
+++ b/bitbake/packaging-pypi/bitbake-setup/BUILD.md
@@ -0,0 +1,67 @@
+# Development Instructions
+
+## Requirements
+
+- Python >=3D 3.9
+- pip >=3D 19 (for installation)
+
+## Testing
+
+To lint the `bitbake-setup` pypi packaging, run the ruff tool.
+```bash
+ruff check bin/bitbake-setup packaging-pypi/bitbake-setup
+```
+
+The steps to build and test the `bitbake-setup` pypi packaging have been a=
utomated with the `bitbake-selftest` tool.  This tool automatically creates=
 a Python virtual environment for you.
+
+Run the bitbake-selftest
+```bash
+BB_SKIP_PYPI_TESTS=3Dno bin/bitbake-selftest -v bb.tests.setup.PyPIPackagi=
ngTest
+```
+
+## Packaging
+
+### Create the development sandbox
+
+To create the development sandbox run:
+```bash
+packaging-pypi/bitbake-setup/package-bitbake-setup.py
+cd packaging-workspace
+```
+
+### Updating the vendored modules
+
+You may use the Python vendoring module to update the vendored modules. It=
s configuration can be found in the pyproject.toml under the tool.vendoring=
 section.
+
+This information includes:
+- the destination directory where the vendored modules are stored (e.g. li=
b/bb/_vendor)
+- the module names and versions to vendor are found in the requirements fi=
le vendor.txt
+- the namespace to place the modules in (eg. bb._vendor)
+- the patches to apply to the pristine source code (eg. vendor/patches)
+
+To run the tool you may install it from pypi and run it from the top-level=
 directory.
+
+```bash
+python3 -m pip install vendoring
+vendoring sync .
+```
+
+### Building the package
+
+To install the development tools manually run:
+```bash
+python3 -m pip install -e '.[dev]'
+```
+
+To build a wheel (.whl) then use:
+```bash
+python3 -m build
+```
+
+This produces a wheel (.whl) file in the dist directory.  This may be inst=
alled using pip.
+
+### Installing the package
+
+```bash
+python3 -m pip install dist/bitbake_setup-*-py3-none-any.whl
+```
diff --git a/bitbake/packaging-pypi/bitbake-setup/LICENSE b/bitbake/packagi=
ng-pypi/bitbake-setup/LICENSE
new file mode 100644
index 00000000..4b4c7367
--- /dev/null
+++ b/bitbake/packaging-pypi/bitbake-setup/LICENSE
@@ -0,0 +1,23 @@
+bitbake-setup is licensed under the GNU General Public License version 2.0=
. See
+LICENSE.GPL-2.0-only for further details.
+
+Individual files contain the following style tags instead of the full lice=
nse text:
+
+    SPDX-License-Identifier:	GPL-2.0-only
+
+This enables machine processing of license information based on the SPDX
+License Identifiers that are here available: http://spdx.org/licenses/
+
+The bitbake-setup package vendorizes the following modules:
+
+beautifulsoup4
+    SPDX-License-Identifier:	MIT
+
+typing_extensions
+    SPDX-License-Identifier:	PSF-2.0
+
+progressbar:
+    SPDX-License-Identifier:	LGPL-2.1-or-later OR BSD-3-Clause-Clear
+
+simplediff:
+    SPDX-License-Identifier:	Zlib
diff --git a/bitbake/packaging-pypi/bitbake-setup/README.md b/bitbake/packa=
ging-pypi/bitbake-setup/README.md
new file mode 100644
index 00000000..aaa70278
--- /dev/null
+++ b/bitbake/packaging-pypi/bitbake-setup/README.md
@@ -0,0 +1,42 @@
+# bitbake-setup
+
+This package provides the `bitbake-setup` command and the Python modules
+required to support BitBake setup and configuration.
+
+## Usage
+
+Instructions on uses of bitbake-setup can be found in
+[Setting Up the Environment With bitbake-setup](https://docs.yoctoproject.=
org/bitbake/dev/bitbake-user-manual/bitbake-user-manual-environment-setup.h=
tml) from the Yocto Project manual.
+
+List the available configurations;
+```bash
+bitbake-setup list
+```
+
+Show the help for the bitbake-setup init subcommand:
+```bash
+bitbake-setup init --help
+usage: bitbake-setup init [-h] [--non-interactive] [--source-overrides SOU=
RCE_OVERRIDES] [--setup-dir-name SETUP_DIR_NAME] [--skip-selection SKIP_SEL=
ECTION] [-L SOURCE_NAME PATH]
+                          [config ...]
+
+positional arguments:
+  config                path/URL/id to a configuration file (use 'list' co=
mmand to get available ids), followed by configuration options. Bitbake-set=
up will ask to choose from available
+                        choices if command line doesn't completely specify=
 them.
+
+options:
+  -h, --help            show this help message and exit
+  --non-interactive     Do not ask to interactively choose from available =
options; if bitbake-setup cannot make a decision it will stop with a failur=
e.
+  --source-overrides SOURCE_OVERRIDES
+                        Override sources information (repositories/revisio=
ns) with values from a local json file.
+  --setup-dir-name SETUP_DIR_NAME
+                        A custom setup directory name under the top direct=
ory.
+  --skip-selection SKIP_SELECTION
+                        Do not select and set an option/fragment from avai=
lable choices; the resulting bitbake configuration may be incomplete.
+  -L SOURCE_NAME PATH, --use-local-source SOURCE_NAME PATH
+                        Symlink local source into a build, instead of gett=
ing it as prescribed by a configuration (useful for local development).
+```
+
+To initialize a workspace for the Poky reference distro using the developm=
ent branch (ie. "master") of OpenEmbedded:
+```bash
+bitbake-setup init poky-master
+```
\ No newline at end of file
diff --git a/bitbake/packaging-pypi/bitbake-setup/package-bitbake-setup.py =
b/bitbake/packaging-pypi/bitbake-setup/package-bitbake-setup.py
new file mode 100755
index 00000000..f5468dc4
--- /dev/null
+++ b/bitbake/packaging-pypi/bitbake-setup/package-bitbake-setup.py
@@ -0,0 +1,79 @@
+#!/usr/bin/env python3
+
+import argparse
+import logging
+import shutil
+from pathlib import Path
+
+
+def create_packaging_workspace(directory):
+    # Create the directory for packaging workspace
+    if len(directory or "") > 0:
+        workspace_dir =3D Path(directory)
+    else:
+        # This script is located in packaging-pypi/bitbake-setup/package-b=
itbake-setup.py
+        workspace_dir =3D Path(__file__).parents[2] / "packaging-workspace=
"
+
+    if not workspace_dir.exists():
+        logging.debug(f"Created packaging workspace at: {workspace_dir}")
+        workspace_dir.mkdir(exist_ok=3DTrue)
+    else:
+        logging.debug(f"Packaging workspace already exists at: {workspace_=
dir}")
+
+    # Copy packaging files to the workspace
+    files_to_copy =3D [
+        "packaging-pypi/bitbake-setup/pyproject.toml",
+        "packaging-pypi/bitbake-setup/README.md",
+        "packaging-pypi/bitbake-setup/LICENSE",
+        "LICENSE.MIT",
+        "LICENSE.GPL-2.0-only"
+    ]
+
+    for file in files_to_copy:
+        src_path =3D Path(__file__).parents[2] / file
+        dest_path =3D workspace_dir / Path(file).name
+
+        if src_path.is_dir():
+            shutil.copytree(src_path, dest_path, dirs_exist_ok=3DTrue)
+            logging.debug(f"Copied directory: {src_path} to {dest_path}")
+        else:
+            shutil.copy2(src_path, dest_path)
+            logging.debug(f"Copied file: {src_path} to {dest_path}")
+
+
+    # Copy necessary modules to the workspace
+    modules_to_bundle =3D [
+        "lib/bb",
+    ]
+
+    for module in modules_to_bundle:
+        src_path =3D Path(__file__).parents[2] / module
+        dest_path =3D workspace_dir / "src" / Path(module).name
+
+        if src_path.is_dir():
+            shutil.copytree(src_path, dest_path, dirs_exist_ok=3DTrue)
+            logging.debug(f"Bundled module directory: {src_path} to {dest_=
path}")
+        else:
+            shutil.copy2(src_path, dest_path)
+            logging.debug(f"Bundled module file: {src_path} to {dest_path}=
")
+
+    # Create bitbake_setup module
+    bitbake_setup_dir =3D Path(workspace_dir / "src" / "bitbake_setup")
+    bitbake_setup_dir.mkdir(exist_ok=3DTrue)
+    Path(bitbake_setup_dir / "__init__.py").touch()
+    shutil.copy2(Path(__file__).parents[2] / "bin" / "bitbake-setup", str(=
bitbake_setup_dir / "__main__.py"))
+
+
+if __name__ =3D=3D "__main__":
+    logging.basicConfig(level=3Dlogging.INFO, format=3D'%(asctime)s - %(le=
velname)s - %(message)s')
+
+    parser =3D argparse.ArgumentParser(description=3D'Package bitbake-setu=
p for PyPI')
+    parser.add_argument('-v', '--verbose', action=3D'store_true', help=3D'=
increase output verbosity.')
+    parser.add_argument('-d', '--directory', type=3Dstr, help=3D'specify t=
he directory to create the packaging workspace in.')
+
+    args =3D parser.parse_args()
+
+    if args.verbose:
+        logging.getLogger().setLevel(logging.DEBUG)
+
+    create_packaging_workspace(args.directory)
diff --git a/bitbake/packaging-pypi/bitbake-setup/pyproject.toml b/bitbake/=
packaging-pypi/bitbake-setup/pyproject.toml
new file mode 100644
index 00000000..90ed1ffb
--- /dev/null
+++ b/bitbake/packaging-pypi/bitbake-setup/pyproject.toml
@@ -0,0 +1,92 @@
+[build-system]
+requires =3D [
+    "setuptools>=3D64",
+]
+build-backend =3D "setuptools.build_meta"
+
+[project]
+name =3D "bitbake-setup"
+dynamic =3D ["version"]
+description =3D "bitbake-setup"
+readme =3D "README.md"
+requires-python =3D ">=3D3.9"
+license =3D "GPL-2.0-only AND MIT AND PSF-2.0 AND Zlib AND LGPL-2.1-or-lat=
er OR BSD-3-Clause-Clear"
+authors =3D [
+  { name =3D "OpenEmbedded BitBake Developers", email =3D "bitbake-devel@l=
ists.openembedded.org" }
+]
+classifiers =3D [
+  "Programming Language :: Python :: 3",
+  "Operating System :: POSIX :: Linux",
+]
+
+dependencies =3D [
+  # bitbake-setup is mostly self-contained
+]
+
+[project.optional-dependencies]
+test =3D [
+  "pytest>=3D7",
+]
+
+lint =3D [
+  "ruff>=3D0.3",
+  "mypy>=3D1.8",
+]
+
+dev =3D [
+  "pytest>=3D7",
+  "ruff>=3D0.3",
+  "mypy>=3D1.8",
+  "build",
+  "twine",
+]
+
+[project.scripts]
+bitbake-setup =3D "bitbake_setup.__main__:main"
+
+[project.urls]
+Homepage =3D "https://git.openembedded.org/bitbake/"
+Documentation =3D "https://docs.yoctoproject.org/bitbake/bitbake-user-manu=
al/bitbake-user-manual-environment-setup.html"
+Repository =3D "https://git.openembedded.org/bitbake/"
+
+[tool.mypy]
+python_version =3D "3.9"
+warn_unused_configs =3D true
+ignore_missing_imports =3D true
+no_implicit_optional =3D true
+check_untyped_defs =3D false
+
+[tool.ruff]
+target-version =3D "py39"
+line-length =3D 88
+src =3D ["src"]
+
+[tool.ruff.lint]
+select =3D [
+  "E",   # pycodestyle
+  "F",   # pyflakes
+  "B",   # flake8-bugbear
+  "I",   # import sorting
+]
+ignore =3D [
+  "E501",  # line length (handled by formatter)
+]
+
+[tool.ruff.format]
+quote-style =3D "double"
+indent-style =3D "space"
+
+[tool.setuptools]
+package-dir =3D { "" =3D "src" }
+zip-safe =3D false
+include-package-data =3D true
+
+[tool.setuptools.dynamic]
+version =3D {attr =3D "bb.__version__"}
+
+[tool.setuptools.packages.find]
+where =3D ["src"]
+include =3D [
+  "bb*",
+  "bitbake_setup"
+]
diff --git a/bitbake/pyproject.toml b/bitbake/pyproject.toml
new file mode 100644
index 00000000..50b6436d
--- /dev/null
+++ b/bitbake/pyproject.toml
@@ -0,0 +1,16 @@
+[project]
+name =3D "bitbake"
+version =3D "2.20.0"
+requires-python =3D ">=3D3.9"
+
+[tool.vendoring]
+destination =3D "lib/bb/_vendor"
+requirements =3D "vendor.txt"
+namespace =3D "bb._vendor"
+patches-dir =3D "vendor/patches"
+
+[tool.vendoring.license.directories]
+ply =3D "vendor/licenses/ply/LICENSE"
+
+[tool.vendoring.license.fallback-urls]
+ply =3D "https://raw.githubusercontent.com/dabeaz/ply/refs/heads/master/sr=
c/ply/lex.py"
diff --git a/bitbake/setup-schema/bitbake-setup.schema.json b/bitbake/setup=
-schema/bitbake-setup.schema.json
new file mode 100644
index 00000000..bb10a925
--- /dev/null
+++ b/bitbake/setup-schema/bitbake-setup.schema.json
@@ -0,0 +1,211 @@
+{
+    "$schema": "https://json-schema.org/draft/2020-12/schema",
+    "description": "Schema for bitbake-setup configuration files",
+    "examples" : [
+        {
+            "description": "Example bitbake-setup configuration file",
+            "sources": {
+                "bitbake": {
+                    "git-remote": {
+                        "uri": "https://git.openembedded.org/bitbake",
+                        "rev": "master"
+                    },
+                    "path": "bitbake"
+                },
+                "openembedded-core": {
+                    "git-remote": {
+                        "uri": "https://git.openembedded.org/openembedded-=
core",
+                        "rev": "master"
+                    },
+                    "path": "openembedded-core"
+                }
+            },
+            "bitbake-setup": {
+                "configurations": [
+                    {
+                        "name" : "qemux86-64-nodistro",
+                        "description": "OpenEmbedded 'nodistro' build for =
qemux86-64",
+                        "setup-dir-name": "oe-nodistro-master",
+                        "bb-layers": ["openembedded-core/meta"],
+                        "oe-fragments": ["machine/qemux86-64"]
+                    }
+                ]
+            },
+            "version": "1.0"
+        }
+    ],
+    "title": "bitbake-setup configuration file",
+    "type": "object",
+    "required": [
+        "description",
+        "bitbake-setup",
+        "version"
+    ],
+    "properties": {
+        "description": {
+            "type": "string",
+            "description": "Description of the bitbake-setup configuration=
 file"
+        },
+        "sources": {
+            "$ref": "layers.schema.json#/properties/sources"
+        },
+        "expires": {
+            "type": "string",
+            "description": "End of life date for this configuration, in IS=
O 8601 format (YYYY-MM-DD)"
+        },
+        "bitbake-setup": {
+            "type": "object",
+            "description": "BitBake-setup configurations",
+            "required": [
+                "configurations"
+            ],
+            "properties": {
+                "configurations": {
+                    "type": "array",
+                    "minItems": 1,
+                    "$anchor": "configurations",
+                    "items": {
+                        "type": "object",
+                        "properties": {
+                            "name": {
+                                "type": "string",
+                                "description": "Name of the configuration"
+                            },
+                            "description": {
+                                "type": "string",
+                                "description": "Human-readable description=
 of the configuration"
+                            },
+                            "notes": {
+                                "anyOf": [
+                                    {
+                                        "type": "string"
+                                    },
+                                    {
+                                        "type": "array",
+                                        "items": {
+                                            "type": "string"
+                                        }
+                                    }
+                                ],
+                                "description": "Extra notes that populate =
conf-notes.txt when bb-layers is used"
+                            },
+                            "bb-layers": {
+                                "type": "array",
+                                "description": "List of BitBake layer path=
s to include, relative to the layers download directory",
+                                "items": {
+                                    "type": "string"
+                                }
+                            },
+                            "bb-layers-file-relative": {
+                                "type": "array",
+                                "description": "List of BitBake layers pat=
hs to include, relative to the directory with the configuration file",
+                                "items": {
+                                    "type": "string"
+                                }
+                            },
+                            "oe-template": {
+                                "type": "string",
+                                "description": "OE-template configuration"
+                            },
+                            "oe-fragments": {
+                                "$anchor": "oe-fragments",
+                                "type": "array",
+                                "description": "List of BitBake configurat=
ion fragments to enable",
+                                "items": {
+                                    "anyOf": [
+                                        {
+                                            "type": "string",
+                                            "description": "Configuration =
fragment name"
+                                        },
+                                        {
+                                            "type": "object",
+                                            "description": "Dictionary of =
BitBake configuration fragments to enable (with description)",
+                                            "properties": {
+                                                "name": {
+                                                    "type": "string",
+                                                    "description": "Config=
uration fragment name"
+                                                },
+                                                "description": {
+                                                    "type": "string",
+                                                    "description": "Human-=
readable description of the fragment"
+                                                }
+                                            }
+                                        }
+                                    ]
+                                }
+                            },
+                            "oe-fragments-one-of": {
+                                "type": "object",
+                                "description": "Mutually exclusive bitbake=
 configuration fragment",
+                                "patternProperties": {
+                                    ".*": {
+                                        "type": "object",
+                                        "required": [
+                                            "description",
+                                            "options"
+                                        ],
+                                        "properties": {
+                                            "description": {
+                                                "type": "string",
+                                                "description": "Human-read=
able description of the fragment category"
+                                            },
+                                            "options": {
+                                                "$ref": "#oe-fragments"
+                                            }
+                                        },
+                                        "additionalProperties": false
+                                    }
+                                },
+                                "additionalProperties": false
+                            },
+                            "configurations": {
+                                "$ref": "#configurations"
+                            },
+                            "bb-env-passthrough-additions": {
+                                "type": "array",
+                                "description": "List of environment variab=
les to include in BB_ENV_PASSTHROUGH_ADDITIONS",
+                                "items": {
+                                    "type": "string"
+                                }
+                            },
+                            "setup-dir-name": {
+                                "type": "string",
+                                "description": "A suggestion for the setup=
 directory name, $-prefixed keys from oe-fragments-one-of will be substitut=
ed with user selections."
+                            },
+                            "install-buildtools": {
+                                "type": "object",
+                                "description": "Settings for downloading b=
uildtools via bb.fetch",
+                                "properties": {
+                                    "url": {
+                                        "type": "string",
+                                        "minLength": 1,
+                                        "description": "Full URL to the bu=
ildtools SDK installer."
+                                    },
+                                    "sha256sum": {
+                                        "type": "string",
+                                        "minLength": 1,
+                                        "description": "SHA256 checksum of=
 the buildtools installer."
+                                    }
+                                },
+                                "required": [
+                                    "url",
+                                    "sha256sum"
+                                ],
+                                "additionalProperties": false
+                            }
+                        },
+                        "additionalProperties": false
+                    }
+                }
+            },
+            "additionalProperties": false
+        },
+        "version": {
+            "description": "The version of this document; currently '1.0'"=
,
+            "enum": [
+                "1.0"
+            ]
+        }
+    },
+    "additionalProperties": false
+}
diff --git a/bitbake/setup-schema/layers.schema.json b/bitbake/setup-schema=
/layers.schema.json
new file mode 100644
index 00000000..3ed40d08
--- /dev/null
+++ b/bitbake/setup-schema/layers.schema.json
@@ -0,0 +1,107 @@
+{
+    "$schema": "https://json-schema.org/draft/2020-12/schema",
+    "description": "A schema for a manifest of layers to setup",
+    "title": "OpenEmbedded Layer Setup Schema",
+    "examples" : [
+        {
+            "sources": {
+                "bitbake": {
+                    "git-remote": {
+                        "uri": "https://git.openembedded.org/bitbake",
+                        "rev": "master"
+                    },
+                    "path": "bitbake"
+                }
+            },
+            "version": "1.0"
+        }
+    ],
+    "type": "object",
+    "additionalProperties": false,
+    "required": [
+        "version"
+    ],
+    "properties": {
+        "version": {
+            "description": "The version of this document; currently '1.0'"=
,
+            "enum": ["1.0"]
+        },
+        "sources": {
+            "description": "The dict of layer sources",
+            "type": "object",
+            "patternProperties": { ".*" : {
+                "type": "object",
+                "description": "The upstream source from which a set of la=
yers may be fetched",
+                "additionalProperties": false,
+                "properties": {
+                    "path": {
+                        "description": "The path where this layer source w=
ill be placed when fetching",
+                        "type": "string"
+                    },
+                    "contains_this_file": {
+                        "description": "Whether the directory with the lay=
er source also contains this json description. Tools may want to skip the c=
heckout of the source then",
+                        "type": "boolean"
+                    },
+                    "git-remote": {
+                        "description": "A remote git source from which to =
fetch",
+                        "type": "object",
+                        "additionalProperties": false,
+                        "required": [
+                            "rev"
+                        ],
+                        "properties": {
+                            "branch": {
+                                "description": "The git branch to fetch (o=
ptional)",
+                                "type": "string"
+                            },
+                            "rev": {
+                                "description": "The git revision to checko=
ut",
+                                "type": "string"
+                            },
+                            "describe": {
+                                "description": "The output of 'git describ=
e' (human readable description of the revision using tags in revision histo=
ry)",
+                                "type": "string"
+                            },
+                            "uri": {
+                                "description": "Specifies the git URI, can=
 be used instead of the more elaborate 'remotes' property when there is onl=
y a single URI",
+                                "type": "string"
+                            },
+                            "remotes": {
+                                "description": "The dict of git remotes to=
 add to this repository",
+                                "type": "object",
+                                "patternProperties": { ".*" : {
+                                    "description": "A git remote",
+                                    "type": "object",
+                                    "additionalProperties": false,
+                                    "required": [
+                                        "uri"
+                                    ],
+                                    "properties": {
+                                        "uri": {
+                                            "description": "The URI for th=
e remote, using git URL syntax",
+                                            "type": "string"
+                                        }
+                                    }
+                                }}
+                            }
+                        }
+                    },
+                    "local": {
+                        "description": "A local directory that should be m=
ade available for builds via symlinking",
+                        "type": "object",
+                        "additionalProperties": false,
+                        "required": [
+                            "path"
+                        ],
+                        "properties": {
+                            "path": {
+                                "description": "The path to the directory"=
,
+                                "type": "string"
+                            }
+                        }
+                    }
+                }
+            }
+        }}
+    }
+}
diff --git a/bitbake/toaster-requirements.txt b/bitbake/toaster-requirement=
s.txt
index d8e48b7f..6941719f 100644
--- a/bitbake/toaster-requirements.txt
+++ b/bitbake/toaster-requirements.txt
@@ -1,4 +1,4 @@
-Django>4.2,<4.3
+Django>5.2,<5.3
 beautifulsoup4>=3D4.4.0
 pytz
 django-log-viewer=3D=3D1.1.7
diff --git a/bitbake/vendor.txt b/bitbake/vendor.txt
new file mode 100644
index 00000000..8d5dcad7
--- /dev/null
+++ b/bitbake/vendor.txt
@@ -0,0 +1,6 @@
+beautifulsoup4=3D=3D4.15.0
+ply=3D=3D3.10
+progressbar=3D=3D2.5
+simplediff=3D=3D1.1
+typing_extensions=3D=3D4.15.0
+tomli=3D=3D2.4.1
diff --git a/bitbake/vendor/licenses/ply/LICENSE b/bitbake/vendor/licenses/=
ply/LICENSE
new file mode 100644
index 00000000..d050b2ef
--- /dev/null
+++ b/bitbake/vendor/licenses/ply/LICENSE
@@ -0,0 +1,32 @@
+# ------------------------------------------------------------------------=
-----
+# ply
+#
+# Copyright (C) 2001-2018
+# David M. Beazley (Dabeaz LLC)
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+# * Redistributions of source code must retain the above copyright notice,
+#   this list of conditions and the following disclaimer.
+# * Redistributions in binary form must reproduce the above copyright noti=
ce,
+#   this list of conditions and the following disclaimer in the documentat=
ion
+#   and/or other materials provided with the distribution.
+# * Neither the name of the David Beazley or Dabeaz LLC may be used to
+#   endorse or promote products derived from this software without
+#  specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# ------------------------------------------------------------------------=
-----
diff --git a/bitbake/vendor/patches/bs4-0010-lib-bs4-Avoid-soupsieve-warnin=
g.patch b/bitbake/vendor/patches/bs4-0010-lib-bs4-Avoid-soupsieve-warning.p=
atch
new file mode 100644
index 00000000..e352473e
--- /dev/null
+++ b/bitbake/vendor/patches/bs4-0010-lib-bs4-Avoid-soupsieve-warning.patch
@@ -0,0 +1,40 @@
+From 8e444cd9913d1ee0672b5583e263e5927c3221df Mon Sep 17 00:00:00 2001
+From: Richard Purdie 
+Date: Fri, 31 May 2024 13:09:44 +0100
+Subject: [PATCH 10/11] lib/bs4: Avoid soupsieve warning
+
+Signed-off-by: Richard Purdie 
+---
+ lib/bs4/css.py | 10 ++--------
+ 1 file changed, 2 insertions(+), 8 deletions(-)
+
+diff --git a/lib/bb/_vendor/bs4/css.py b/lib/bb/_vendor/bs4/css.py
+index 245ac6010..cd1fd2df8 100644
+--- a/lib/bb/_vendor/bs4/css.py
++++ b/lib/bb/_vendor/bs4/css.py
+@@ -24,23 +24,14 @@
+     Optional,
+     TYPE_CHECKING,
+ )
+-import warnings
+ from bs4._typing import _NamespaceMapping
+
+ if TYPE_CHECKING:
+-    from soupsieve import SoupSieve
+     from bs4 import element
+     from bs4.element import ResultSet, Tag
+
+-soupsieve: Optional[ModuleType]
+-try:
+-    import soupsieve
+-except ImportError:
+-    soupsieve =3D None
+-    warnings.warn(
+-        "The soupsieve package is not installed. CSS selectors cannot be =
used."
+-    )
+-
++# We don't use soupsieve
++soupsieve =3D None
+
+ class CSS(object):
+     """A proxy object against the ``soupsieve`` library, to simplify its
diff --git a/bitbake/vendor/patches/bs4-remove-double-imports.patch b/bitba=
ke/vendor/patches/bs4-remove-double-imports.patch
new file mode 100644
index 00000000..2bbfa937
--- /dev/null
+++ b/bitbake/vendor/patches/bs4-remove-double-imports.patch
@@ -0,0 +1,23 @@
+commit 741db6719efca5aa9ef2c15e60cdd624e4aa1a8d
+Author: Michael Estner 
+Date:   Tue Feb 25 19:34:25 2025 +0100
+
+    lib: Remove double imports
+
+    * Remove double imports mentioned by pylint
+
+    Signed-off-by: Michael Estner 
+    Signed-off-by: Richard Purdie 
+
+diff --git a/lib/bb/_vendor/bs4/__init__.py b/lib/bb/_vendor/bs4/__init__.=
py
+index d8ad5e1dc..725203d94 100644
+--- a/lib/bb/_vendor/bs4/__init__.py
++++ b/lib/bb/_vendor/bs4/__init__.py
+@@ -1173,7 +1173,5 @@
+
+ # If this file is run as a script, act as an HTML pretty-printer.
+ if __name__ =3D=3D "__main__":
+-    import sys
+-
+     soup =3D BeautifulSoup(sys.stdin)
+     print((soup.prettify()))
diff --git a/bitbake/vendor/patches/progressbar-0001-lib-implement-basic-ta=
sk-progress-support.patch b/bitbake/vendor/patches/progressbar-0001-lib-imp=
lement-basic-task-progress-support.patch
new file mode 100644
index 00000000..97dd5ca9
--- /dev/null
+++ b/bitbake/vendor/patches/progressbar-0001-lib-implement-basic-task-prog=
ress-support.patch
@@ -0,0 +1,125 @@
+From 0d275fc5b6531957a6189069b04074065bb718a0 Mon Sep 17 00:00:00 2001
+From: Paul Eggleton 
+Date: Thu, 23 Jun 2016 22:59:05 +1200
+Subject: [PATCH 1/6] lib: implement basic task progress support
+
+For long-running tasks where we have some output from the task that
+gives us some idea of the progress of the task (such as a percentage
+complete), provide the means to scrape the output for that progress
+information and show it to the user in the default knotty terminal
+output in the form of a progress bar. This is implemented using a new
+TaskProgress event as well as some code we can insert to do output
+scanning/filtering.
+
+Any task can fire TaskProgress events; however, if you have a shell task
+whose output you wish to scan for progress information, you just need to
+set the "progress" varflag on the task. This can be set to:
+ * "percent" to just look for a number followed by a % sign
+ * "percent:" to specify your own regex matching a percentage
+   value (must have a single group which matches the percentage number)
+ * "outof:" to look for the specified regex matching x out of y
+   items completed (must have two groups - first group needs to be x,
+   second y).
+We can potentially extend this in future but this should be a good
+start.
+
+Part of the implementation for [YOCTO #5383].
+
+Signed-off-by: Paul Eggleton 
+Signed-off-by: Richard Purdie 
+---
+ lib/progressbar/progressbar.py | 16 +++++++++++----
+ lib/progressbar/widgets.py     | 36 ++++++++++++++++++++++++++++++++++
+ 2 files changed, 48 insertions(+), 4 deletions(-)
+
+diff --git a/lib/bb/_vendor/progressbar/progressbar.py b/lib/bb/_vendor/pr=
ogressbar/progressbar.py
+index 0b9dcf763..2873ad6ca 100644
+--- a/lib/bb/_vendor/progressbar/progressbar.py
++++ b/lib/bb/_vendor/progressbar/progressbar.py
+@@ -3,6 +3,8 @@
+ # progressbar  - Text progress bar library for Python.
+ # Copyright (c) 2005 Nilton Volpato
+ #
++# (With some small changes after importing into BitBake)
++#
+ # This library is free software; you can redistribute it and/or
+ # modify it under the terms of the GNU Lesser General Public
+ # License as published by the Free Software Foundation; either
+@@ -261,12 +263,14 @@ class ProgressBar(object):
+         now =3D time.time()
+         self.seconds_elapsed =3D now - self.start_time
+         self.next_update =3D self.currval + self.update_interval
+-        self.fd.write(self._format_line() + '\r')
++        output =3D self._format_line()
++        self.fd.write(output + '\r')
+         self.fd.flush()
+         self.last_update_time =3D now
++        return output
+
+
+-    def start(self):
++    def start(self, update=3DTrue):
+         """Starts measuring time, and prints the bar at 0%.
+
+         It returns self so you can use it like this:
+@@ -289,8 +293,12 @@ class ProgressBar(object):
+             self.update_interval =3D self.maxval / self.num_intervals
+
+
+-        self.start_time =3D self.last_update_time =3D time.time()
+-        self.update(0)
++        self.start_time =3D time.time()
++        if update:
++            self.last_update_time =3D self.start_time
++            self.update(0)
++        else:
++            self.last_update_time =3D 0
+
+         return self
+
+diff --git a/lib/bb/_vendor/progressbar/widgets.py b/lib/bb/_vendor/progre=
ssbar/widgets.py
+index 6434ad559..77285ca7a 100644
+--- a/lib/bb/_vendor/progressbar/widgets.py
++++ b/lib/bb/_vendor/progressbar/widgets.py
+@@ -353,3 +353,39 @@ class BouncingBar(Bar):
+         if not self.fill_left: rpad, lpad =3D lpad, rpad
+
+         return '%s%s%s%s%s' % (left, lpad, marker, rpad, right)
++
++
++class BouncingSlider(Bar):
++    """
++    A slider that bounces back and forth in response to update() calls
++    without reference to the actual value. Based on a combination of
++    BouncingBar from a newer version of this module and RotatingMarker.
++    """
++    def __init__(self, marker=3D'<=3D>'):
++        self.curmark =3D -1
++        self.forward =3D True
++        Bar.__init__(self, marker=3Dmarker)
++    def update(self, pbar, width):
++        left, marker, right =3D (format_updatable(i, pbar) for i in
++                               (self.left, self.marker, self.right))
++
++        width -=3D len(left) + len(right)
++        if width < 0:
++            return ''
++
++        if pbar.finished: return '%s%s%s' % (left, width * '=3D', right)
++
++        self.curmark =3D self.curmark + 1
++        position =3D int(self.curmark % (width * 2 - 1))
++        if position + len(marker) > width:
++            self.forward =3D not self.forward
++            self.curmark =3D 1
++            position =3D 1
++        lpad =3D ' ' * (position - 1)
++        rpad =3D ' ' * (width - len(marker) - len(lpad))
++
++        if not self.forward:
++            temp =3D lpad
++            lpad =3D rpad
++            rpad =3D temp
++        return '%s%s%s%s%s' % (left, lpad, marker, rpad, right)
+--
+2.49.0
diff --git a/bitbake/vendor/patches/progressbar-0002-bitbake-Add-initial-pa=
ss-of-SPDX-license-headers-to-.patch b/bitbake/vendor/patches/progressbar-0=
002-bitbake-Add-initial-pass-of-SPDX-license-headers-to-.patch
new file mode 100644
index 00000000..fd8ccaa3
--- /dev/null
+++ b/bitbake/vendor/patches/progressbar-0002-bitbake-Add-initial-pass-of-S=
PDX-license-headers-to-.patch
@@ -0,0 +1,79 @@
+From ff237c33337f4da2ca06c3a2c49699bc26608a6b Mon Sep 17 00:00:00 2001
+From: Richard Purdie 
+Date: Tue, 30 Apr 2019 11:05:26 +0100
+Subject: [PATCH 2/6] bitbake: Add initial pass of SPDX license headers to
+ source code
+
+This adds the SPDX-License-Identifier license headers to the majority of
+our source files to make it clearer exactly which license files are under.
+
+The bulk of the files are under GPL v2.0 with one found to be under V2.0
+or later, some under MIT and some have dual license. There are some files
+which are potentially harder to classify where we've imported upstream cod=
e
+and those can be handled specifically in later commits.
+
+The COPYING file is replaced with LICENSE.X files which contain the full
+license texts.
+
+Signed-off-by: Richard Purdie 
+---
+ lib/progressbar/__init__.py    | 2 ++
+ lib/progressbar/compat.py      | 2 ++
+ lib/progressbar/progressbar.py | 2 ++
+ lib/progressbar/widgets.py     | 2 ++
+ 4 files changed, 8 insertions(+)
+
+diff --git a/lib/bb/_vendor/progressbar/__init__.py b/lib/bb/_vendor/progr=
essbar/__init__.py
+index fbab744ee..c545a6275 100644
+--- a/lib/bb/_vendor/progressbar/__init__.py
++++ b/lib/bb/_vendor/progressbar/__init__.py
+@@ -4,6 +4,8 @@
+ # progressbar  - Text progress bar library for Python.
+ # Copyright (c) 2005 Nilton Volpato
+ #
++# SPDX-License-Identifier: LGPL-2.1-or-later OR BSD-3-Clause-Clear
++#
+ # This library is free software; you can redistribute it and/or
+ # modify it under the terms of the GNU Lesser General Public
+ # License as published by the Free Software Foundation; either
+diff --git a/lib/bb/_vendor/progressbar/compat.py b/lib/bb/_vendor/progres=
sbar/compat.py
+index a39f4a1f4..9804e0b51 100644
+--- a/lib/bb/_vendor/progressbar/compat.py
++++ b/lib/bb/_vendor/progressbar/compat.py
+@@ -3,6 +3,8 @@
+ # progressbar  - Text progress bar library for Python.
+ # Copyright (c) 2005 Nilton Volpato
+ #
++# SPDX-License-Identifier: LGPL-2.1-or-later OR BSD-3-Clause-Clear
++#
+ # This library is free software; you can redistribute it and/or
+ # modify it under the terms of the GNU Lesser General Public
+ # License as published by the Free Software Foundation; either
+diff --git a/lib/bb/_vendor/progressbar/progressbar.py b/lib/bb/_vendor/pr=
ogressbar/progressbar.py
+index 2873ad6ca..e2b6ba108 100644
+--- a/lib/bb/_vendor/progressbar/progressbar.py
++++ b/lib/bb/_vendor/progressbar/progressbar.py
+@@ -5,6 +5,8 @@
+ #
+ # (With some small changes after importing into BitBake)
+ #
++# SPDX-License-Identifier: LGPL-2.1-or-later OR BSD-3-Clause-Clear
++#
+ # This library is free software; you can redistribute it and/or
+ # modify it under the terms of the GNU Lesser General Public
+ # License as published by the Free Software Foundation; either
+diff --git a/lib/bb/_vendor/progressbar/widgets.py b/lib/bb/_vendor/progre=
ssbar/widgets.py
+index 77285ca7a..0772aa536 100644
+--- a/lib/bb/_vendor/progressbar/widgets.py
++++ b/lib/bb/_vendor/progressbar/widgets.py
+@@ -3,6 +3,8 @@
+ # progressbar  - Text progress bar library for Python.
+ # Copyright (c) 2005 Nilton Volpato
+ #
++# SPDX-License-Identifier: LGPL-2.1-or-later OR BSD-3-Clause-Clear
++#
+ # This library is free software; you can redistribute it and/or
+ # modify it under the terms of the GNU Lesser General Public
+ # License as published by the Free Software Foundation; either
+--
+2.49.0
diff --git a/bitbake/vendor/patches/progressbar-0003-bitbake-progressbar-ac=
cept-value-over-initial-maxval.patch b/bitbake/vendor/patches/progressbar-0=
003-bitbake-progressbar-accept-value-over-initial-maxval.patch
new file mode 100644
index 00000000..aca9e0ed
--- /dev/null
+++ b/bitbake/vendor/patches/progressbar-0003-bitbake-progressbar-accept-va=
lue-over-initial-maxval.patch
@@ -0,0 +1,54 @@
+From 7cea7f7a87da041fc1ad370c5c3d15aabad3a0d4 Mon Sep 17 00:00:00 2001
+From: Enguerrand de Ribaucourt 
+Date: Thu, 22 Feb 2024 16:21:53 +0100
+Subject: [PATCH 3/6] bitbake: progressbar: accept value over initial maxva=
l
+
+There is a very rare case where the maxval is improperly computed
+initially for cache loading progress, and the value will go over.
+
+Explanation from bitbake/lib/bb/cache.py:736 in MulticonfigCache:__init__:=
progress()
+    # we might have calculated incorrect total size because a file
+    # might've been written out just after we checked its size
+
+In that case, progressbar will receive a value over the initial maxval.
+This results in a ValueError stack trace as well as bitbake returning 1.
+    Traceback (most recent call last):
+    File ".../poky/bitbake/lib/bb/ui/knotty.py", line 736, in main
+        cacheprogress.update(event.current)
+    File ".../poky/bitbake/lib/progressbar/progressbar.py", line 256, in u=
pdate
+        raise ValueError('Value out of range')
+    ValueError: Value out of range
+
+This fix mirrors the behavior of MulticonfigCache and accepts the new
+value as the new maxval. This is also what the percentage printout
+is doing in bitbake/lib/progressbar/progressbar.py:191 in ProgressBar:perc=
entage()
+
+I encountered this issue randomly while working on a project with
+VSCode saving files while commands where fired.
+
+Note: This file is a fork from python-progressbar. It hasn't been
+refreshed in 8 years. We did only two commits, 5 years ago with minor
+modifications. This new change is also not how the upstream project is
+behaving.
+
+Signed-off-by: Enguerrand de Ribaucourt 
+Signed-off-by: Richard Purdie 
+---
+ lib/progressbar/progressbar.py | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/lib/bb/_vendor/progressbar/progressbar.py b/lib/bb/_vendor/pr=
ogressbar/progressbar.py
+index e2b6ba108..d4da10ab7 100644
+--- a/lib/bb/_vendor/progressbar/progressbar.py
++++ b/lib/bb/_vendor/progressbar/progressbar.py
+@@ -253,7 +253,7 @@ class ProgressBar(object):
+             if (self.maxval is not widgets.UnknownLength
+                 and not 0 <=3D value <=3D self.maxval):
+
+-                raise ValueError('Value out of range')
++                self.maxval =3D value
+
+             self.currval =3D value
+
+--
+2.49.0
diff --git a/bitbake/vendor/patches/progressbar-0004-progressbar-Add-self._=
fd_console-to-use-for-self._ha.patch b/bitbake/vendor/patches/progressbar-0=
004-progressbar-Add-self._fd_console-to-use-for-self._ha.patch
new file mode 100644
index 00000000..1bf0718f
--- /dev/null
+++ b/bitbake/vendor/patches/progressbar-0004-progressbar-Add-self._fd_cons=
ole-to-use-for-self._ha.patch
@@ -0,0 +1,53 @@
+From f8c76eb89d52b1c28407f0b52dfe4318faa47cd2 Mon Sep 17 00:00:00 2001
+From: =3D?UTF-8?q?Benjamin=3D20Sz=3DC5=3D91ke?=3D 
+Date: Sat, 25 Jan 2025 13:27:49 +0100
+Subject: [PATCH 4/6] progressbar: Add self._fd_console to use for
+ self._handle_resize()
+MIME-Version: 1.0
+Content-Type: text/plain; charset=3DUTF-8
+Content-Transfer-Encoding: 8bit
+
+Introduce self._fd_console as a dedicated attribute of self._handle_resize=
().
+
+Signed-off-by: Benjamin Sz=C5=91ke 
+Signed-off-by: Mathieu Dubois-Briand 
+---
+ lib/progressbar/progressbar.py | 11 +++++++++--
+ 1 file changed, 9 insertions(+), 2 deletions(-)
+
+diff --git a/lib/bb/_vendor/progressbar/progressbar.py b/lib/bb/_vendor/pr=
ogressbar/progressbar.py
+index d4da10ab7..eccc45849 100644
+--- a/lib/bb/_vendor/progressbar/progressbar.py
++++ b/lib/bb/_vendor/progressbar/progressbar.py
+@@ -110,12 +110,20 @@ class ProgressBar(object):
+         self.widgets =3D widgets
+         self.fd =3D fd if fd is not None else sys.stderr
+         self.left_justify =3D left_justify
++        self._fd_console =3D None
+
+         self.signal_set =3D False
+         if term_width is not None:
+             self.term_width =3D term_width
+         else:
+             try:
++                # Check if given file descriptor is resizable for example=
 belong
++                # to a terminal/console as STDOUT or STDERR. If file desc=
riptor
++                # is resizable, let's allow to use for self._handle_resiz=
e()
++                # in a dedicated self._fd_console in order to be able to =
set
++                # temporarily/permanently self.fd to any StringIO or othe=
r
++                # file descriptor later.
++                self._fd_console =3D fd
+                 self._handle_resize()
+                 signal.signal(signal.SIGWINCH, self._handle_resize)
+                 self.signal_set =3D True
+@@ -182,7 +189,7 @@ class ProgressBar(object):
+     def _handle_resize(self, signum=3DNone, frame=3DNone):
+         """Tries to catch resize signals sent from the terminal."""
+
+-        h, w =3D array('h', ioctl(self.fd, termios.TIOCGWINSZ, '\0' * 8))=
[:2]
++        h, w =3D array('h', ioctl(self._fd_console, termios.TIOCGWINSZ, '=
\0' * 8))[:2]
+         self.term_width =3D w
+
+
+--
+2.49.0
diff --git a/bitbake/vendor/patches/progressbar-0005-progressbar-Make-bars-=
show-correctly-with-maxval-0.patch b/bitbake/vendor/patches/progressbar-000=
5-progressbar-Make-bars-show-correctly-with-maxval-0.patch
new file mode 100644
index 00000000..c8592355
--- /dev/null
+++ b/bitbake/vendor/patches/progressbar-0005-progressbar-Make-bars-show-co=
rrectly-with-maxval-0.patch
@@ -0,0 +1,31 @@
+From 736b5b33a5d656a6d5cb6a3180ab529832f77b16 Mon Sep 17 00:00:00 2001
+From: Peter Kjellerstedt 
+Date: Sat, 11 Oct 2025 06:23:10 +0200
+Subject: [PATCH 5/6] progressbar: Make bars show correctly with maxval =3D=
=3D 0
+
+While it can be argued how useful a progressbar created with 0 as
+maximum value is, it should still show two states, started (empty) and
+finished (full). Setting the maxval to _DEFAULT_MAXVAL instead will
+accomplish this.
+
+Signed-off-by: Peter Kjellerstedt 
+Signed-off-by: Mathieu Dubois-Briand 
+---
+ lib/progressbar/progressbar.py | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/lib/bb/_vendor/progressbar/progressbar.py b/lib/bb/_vendor/pr=
ogressbar/progressbar.py
+index eccc45849..a8e2dc09c 100644
+--- a/lib/bb/_vendor/progressbar/progressbar.py
++++ b/lib/bb/_vendor/progressbar/progressbar.py
+@@ -106,7 +106,7 @@ class ProgressBar(object):
+         if widgets is None:
+             widgets =3D list(self._DEFAULT_WIDGETS)
+
+-        self.maxval =3D maxval
++        self.maxval =3D maxval if maxval !=3D 0 else self._DEFAULT_MAXVAL
+         self.widgets =3D widgets
+         self.fd =3D fd if fd is not None else sys.stderr
+         self.left_justify =3D left_justify
+--
+2.49.0
diff --git a/bitbake/vendor/patches/progressbar-0006-progressbar-knotty-All=
ow-mixing-log-messages-and-pro.patch b/bitbake/vendor/patches/progressbar-0=
006-progressbar-knotty-Allow-mixing-log-messages-and-pro.patch
new file mode 100644
index 00000000..96868c76
--- /dev/null
+++ b/bitbake/vendor/patches/progressbar-0006-progressbar-knotty-Allow-mixi=
ng-log-messages-and-pro.patch
@@ -0,0 +1,31 @@
+From e827db50d8f61f0d04c9bd0753c05aeb1ed0715f Mon Sep 17 00:00:00 2001
+From: Richard Purdie 
+Date: Tue, 24 Mar 2026 21:37:25 +0000
+Subject: [PATCH 6/6] progressbar/knotty: Allow mixing log messages and
+ progress bars
+
+If we try and print a log message in the middle of progress bar, the displ=
ay
+can be corrupted. Add a clear() method to progress bars allowing them to b=
e
+removed for the log message, then reprinted using update().
+
+Signed-off-by: Richard Purdie 
+---
+ lib/progressbar/progressbar.py | 3 +++
+ 1 file changed, 3 insertions(+)
+
+diff --git a/lib/bb/_vendor/progressbar/progressbar.py b/lib/bb/_vendor/pr=
ogressbar/progressbar.py
+index a8e2dc09c..1562774ba 100644
+--- a/lib/bb/_vendor/progressbar/progressbar.py
++++ b/lib/bb/_vendor/progressbar/progressbar.py
+@@ -278,6 +278,9 @@ class ProgressBar(object):
+         self.last_update_time =3D now
+         return output
+
++    def clear(self):
++        self.fd.write(" " * self.term_width + '\r')
++        self.fd.flush()
+
+     def start(self, update=3DTrue):
+         """Starts measuring time, and prints the bar at 0%.
+--
+2.49.0
--=20
2.56.0

--=20
You received this message because you are subscribed to the Google Groups "=
isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an e=
mail to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/=
20261109091802.202521-9-clara.kowalsky%40siemens.com.