From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7088694731746574336 X-Received: by 2002:adf:f4c9:0:b0:207:9e3d:cc1d with SMTP id h9-20020adff4c9000000b002079e3dcc1dmr15608734wrp.597.1650466866909; Wed, 20 Apr 2022 08:01:06 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:600c:4f50:b0:391:3728:b2d1 with SMTP id m16-20020a05600c4f5000b003913728b2d1ls3150785wmq.1.canary-gmail; Wed, 20 Apr 2022 08:01:05 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyIOc5uQqaZR18bUuTYGiYSHSEE00biKIXZhISYLt+6xALfA/wrhFvnjGXKrMnccZ86KZDO X-Received: by 2002:a05:600c:3ca7:b0:38e:50d2:27fe with SMTP id bg39-20020a05600c3ca700b0038e50d227femr4151100wmb.159.1650466865771; Wed, 20 Apr 2022 08:01:05 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1650466865; cv=pass; d=google.com; s=arc-20160816; b=wMKtlpbwYI6XaoJGfTbf6B6vx0A0Kci+sP55WUXo5ezouxWluIaQp+a3qgj07SFrJj UgCO9eRtFsbSbCae5frNc01VM5amXDV5/yWT+UUOnKc7fAB7KwbTgZ0ORC1N9eR1TAHy 2hw37Gl1G0nw/ut6RludL69BNXPaJqrYFhhi5KOHnDrXgX5kzGjOBepfv4AubHMRGi2q 6ilCZk8H88ytEO0nDEkhIejh8N8ylzatx4sV7pJHALWfcEhP/sE30WxZj05HLhr9rpD2 Xy2dOjKXBgOz3gxA14REym2eYGzfQHy76UFF/L6+VgqzEFh2D9vH3CFYOP6LAsSk8f/u bPiA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:content-transfer-encoding:references:in-reply-to :message-id:subject:cc:to:from:date:dkim-signature; bh=vnQ3Px1q1QHMxBMsBpeSX7Re1b6qtoOSC2rerTFZcO0=; b=QEpBlFQUoL4PFR2Q6q5ioMqRP2vFzZRbCpXdL3LSu8UStwGCtDYiAFJstwnU8s8EUy xsBzTdUTxNnCQBl8GKwN8GcgSoXPw5Tq7c5U2xqPUujRyJFRzGFya5fN4MznW/BZ0teI kOI92QV3HqoOpbkwegCvrqY0meAtrKO/wzV+Onc7Atz57SKRdfAvDaHuqeCo+REGKUNI jeCWd3GeTdnwlAugDfy2Qh2kkSBeYv0RINuMlNrSkbrl08W+QNkox01Z3NT6f3LH205B UHTkfSPkqdL8qxHCnXo5lxX+BtImo+FYAiHflX6qIPadqa0akcSqYX4Yw2zsD2vdZN5h K7Qw== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=Y1v8i+zp; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of henning.schild@siemens.com designates 2a01:111:f400:7e1a::625 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from EUR05-DB8-obe.outbound.protection.outlook.com (mail-db8eur05on20625.outbound.protection.outlook.com. [2a01:111:f400:7e1a::625]) by gmr-mx.google.com with ESMTPS id 9-20020a056000154900b001f1f8f0f76csi5553wry.3.2022.04.20.08.01.05 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 20 Apr 2022 08:01:05 -0700 (PDT) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 2a01:111:f400:7e1a::625 as permitted sender) client-ip=2a01:111:f400:7e1a::625; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=Y1v8i+zp; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of henning.schild@siemens.com designates 2a01:111:f400:7e1a::625 as permitted sender) smtp.mailfrom=henning.schild@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=SPeoxi0Mo1hIUXLh0Bn+LkZdm1R44mrlRRTx8zzHtGKWk9TFlw2klMN1n+YWzYSa5TXtKTxEnpx7dfuAk//Tr7+bCOVae+H+ESPRztftwk+BW6kH/QnVDxCE3XEPi0z6Wc5hcSsjnk9mBJzqBqzYbGkNFoQjsxqrbfnFtNjKzo5QGD6pee8mql79CMbtt+WDgYkPiDA5pbsn/Ks3O8cOe+hlTQ16tcnohQbMuPcs4nEPB1FF7PCONuLy9sCKykxj3/ORJBUC0S1oVW6HEWKsXWAzyzRabne11ZZmoSWahX/v7wpl/QEUVcLfVzS9InZwpYbpGx3ji3EpIlJVjQudyg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=vnQ3Px1q1QHMxBMsBpeSX7Re1b6qtoOSC2rerTFZcO0=; b=OG5E/k9J5xu5NNik134KfLb4zB6Dn5PS6vopcMr/z0v/MRe/sZmGLSnRW2rh3X2GMTG4pljcymwZlsDUMJblC69SjMMTloS4XiOVEA+AKtG2wHUb/yn5iB98nz8mfZogtP0smbsWDfd3BntC4/6ya+8musd7HAHQIRMfjlVLF8DdA4si6Kv3E3lTG5n/+213FVRYRfRj2A0129fII+NP9u/wBNDEIFV23Kt6QU0UO9WJ1ElDovaUYbPgeenzMJQxzmTEqMS01g9Jug/JrllvV2GTPn/rqP2FSle4pvhVewQdP52dcgM3Dz36X8hEUx26BFjsRkEvCPiK6+icxAR2Lg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=siemens.com; dmarc=pass action=none header.from=siemens.com; dkim=pass header.d=siemens.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=siemens.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vnQ3Px1q1QHMxBMsBpeSX7Re1b6qtoOSC2rerTFZcO0=; b=Y1v8i+zp0sWwLo51hzVQknoo/4ZuviFP0ypFfdGew7/vMp6MrjQqzjrP7/WI3gQy5tQEgQWl4LG/Fi5rLA2UmZaDHvOuSQMIEaw144N2OkwpB/tD6cjeqMdDPUSvRaAVzEVwrWzRI1sD4xDpd/ozcOgAMRaQ5SyCmLEzBLJBDuEYJoSmtoyNgQl1avQy5peKT3fmVv5oP5GImw1jpc0Hj+OhVwViEsCTnH907Lwx4L1Wybe8CpSbH3IHhHF9zCvDna6SwQ7HB9t5vg46u2p6u8DpsTxiel77UbPHGyl2o9hTN88RD/lkZRf8hT1wJHqWWCda92W2TyyN42HOsjtUkw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=siemens.com; Received: from PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:102:269::8) by AM0PR10MB2898.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:208:159::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5186.13; Wed, 20 Apr 2022 15:01:04 +0000 Received: from PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM ([fe80::f4be:8b5d:4314:c2d8]) by PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM ([fe80::f4be:8b5d:4314:c2d8%4]) with mapi id 15.20.5164.025; Wed, 20 Apr 2022 15:01:04 +0000 Date: Wed, 20 Apr 2022 17:00:59 +0200 From: Henning Schild To: isar-users Cc: Felix Moessbauer , Florian Bezdeka Subject: Re: [PATCH] bitbake: Update to 1.50.5 release Message-ID: <20220420170059.23464e65@md1za8fc.ad001.siemens.net> In-Reply-To: <20220420143540.5159-1-henning.schild@siemens.com> References: <20220420143540.5159-1-henning.schild@siemens.com> X-Mailer: Claws Mail 3.18.0 (GTK+ 2.24.33; x86_64-pc-linux-gnu) Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-ClientProxiedBy: AS8P189CA0007.EURP189.PROD.OUTLOOK.COM (2603:10a6:20b:31f::23) To PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:102:269::8) Return-Path: henning.schild@siemens.com MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 805af821-3e1f-42cd-33cb-08da22de96e8 X-MS-TrafficTypeDiagnostic: AM0PR10MB2898:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Lx51KFbVgqcpgTzwq00vT3tp/MSodKe4TLRAPbqPdRfL/eHe5J7+EfKXfhPGELEghGTqpHlTZILQeA0M8GossTazckgbx8MreaFuAjtQORbqWgjrdRoXRiV0wejNaDG14G3Fuea+B8ACupJz2je2sL2rpE+47qV09mreMW5kcsfK9Y31vY4nTd5Bi25ws+AB4eqacMAFI2WLYn7HkM5Vo4Np+GrNUIMfUegZFzDbS6tyViC2ty/VnLj9EI1b9Y3r1hVBCT/OoYV6epVFLFnMHvnuvVmst0kkETKC8Ml1ja2KKrKsHi1VzHVs25xQ+zdA8U0ZLdEDf4RtwqpN487nvgkmN/o6dnX1BBPfJZb+sc7+Xp9HMqniWfcDUsPDzKVIlWYSz9fzob2e/gSKn12AedidZtA4x7jdviu9lRbSEN7rdgM8TarwmuVaAy1hNuBxIrh6SXmwxIH8B8NGq/f9iH0VQp+OFBxTsmssRmG8Nbgd9Au6Kl7TB07ImF7pVPh5qzwajDA/SymGpEq6sBdCpZCgM2JNw+kAt0BzG8+ABmGmZXB7FB79cqdpiOQz9t2R79MmUHW/jYX0r2hfIodjZ76WP1ZY3e9z1Tkd6ilJaWhDxGJ2s077tJYOL3vSbSgydg7kgz6wAimQeFTPp0sez+UKpdrifS/9CLOqB6yW3Rq9ntcbyRV/qiw//2HFCpaMNikPntZJjnfFC8kOjUQb3ty5lsxEvjWjbfT+opToN9NIO7BARPfEcxsjHjS0XHYXSlE+KaZhJHeUcN1htvh9pw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230001)(4636009)(366004)(15650500001)(2906002)(44832011)(54906003)(6916009)(38100700002)(1076003)(186003)(107886003)(30864003)(86362001)(82960400001)(5660300002)(8936002)(6666004)(66476007)(83380400001)(66556008)(66946007)(4326008)(6486002)(508600001)(6506007)(316002)(26005)(9686003)(6512007)(966005)(8676002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?Mw5DXh2HF0ji4gu7rjxJBDchSYuqo22cLX9gImgUe9DlY5PP8pRszZmcPI+N?= =?us-ascii?Q?Bw+rIv1SdlblTbh8l1MD+G0vISgU9l0N8zLR56gNP7PXNfQo53H+rHT0k+A1?= =?us-ascii?Q?trEg9YDnbCgtzMKdTy1/Dc5k3ZbCWmj8HXT9VMxqY7VSQmbfwzbr4MyUsjR+?= =?us-ascii?Q?RIXhmlrXOMxrMTsfB/lgpdJ0xWMrbA+WU8QLKuEDjPuNCLlg+1+LvYNrRFBE?= =?us-ascii?Q?Ncy9myRc86Drycqytkw3pUswjAmO8ZFcX0iFB7KYoc5mFZyEyQmxLJoqqmxk?= =?us-ascii?Q?797x/fm7iI2beCRQgSgJQb+fXDhoAiN7eYvtyLw5hrSRhPMRigyNhtvnFn6v?= =?us-ascii?Q?8gT/1A51bwLBpZzBX5lxawbuvnuGAsgzyyrdDTXUKZNbReWZnQK6ututx0Fj?= =?us-ascii?Q?E/WaFZUSUEP79vb5WpPbq5eN2yN/5pQ4A4mso1tB3dLd6bFtWJ1NOQ6wHTU6?= =?us-ascii?Q?pzNmizEZ/emVtJsINcG7CMc/sFO3ekfvAHsbGj+UgVlcHMce5RcF2hOZ9say?= =?us-ascii?Q?Mcu3jo/XoUtfVarIQbgYHSn2Qp8+OhIn6NeYDGs84kn6HBIUSGEOLz99iUBG?= =?us-ascii?Q?Ags4MprKuGdscVc5HfYlfvQbIVus3nuo3bHCgPrmADHjYdGZdwluPJmQ257b?= =?us-ascii?Q?mbrFf+gLjXSdhBmaE//q9F2eV7LjsqwgvVL/rPqnbE7riOWsOJG4NSGGCVLa?= =?us-ascii?Q?7YOpa7tozpyabvNUQSr0UJJWbxfcqkuIBh0DTvtyGoxkLB7vHvg9fStXnorw?= =?us-ascii?Q?xECnlwr9p1157lJAVhO5FqbJbmsstxWfGEgJyRUIxkPNnwzDv6NqH3i9ht4K?= =?us-ascii?Q?DaVoUf30gdPS4Mt54Odvt9YyYzqq3JH6x2J01hKrHQbxImmKHEsJRHEzllyC?= =?us-ascii?Q?Vc+IEtpB5hx66peu4BTAHEsyBnEf0j8JWA13en3Wx2Rti+BqzoNjtrFsfd9B?= =?us-ascii?Q?1Rl5wYj79FnrafSEL04iKN4uXWTjLT31IGFUPD+78LJuUxIB4hhgAmqd+grz?= =?us-ascii?Q?HzFFo1sT9qSghSo3y64rgjF/l2gBNuzLXfm1dmA206wWHAGWXeZavFUynJTd?= =?us-ascii?Q?xU/FHhEagkq/VtdaGtLbeRTKR/6NeIggSLdiJM5PIaZZ/IrKwWt/yS9sIvkM?= =?us-ascii?Q?8mgYmVN7HEYP92/dRZE3ZXLqxajC91C+AndGclZkP6TqaRAOdjhj6P/R93Cd?= =?us-ascii?Q?yTjJof2YRivDAEnALuweVSAhMzz7OrFuq7E2mgaIrHJOycD2364Zvel7JNwY?= =?us-ascii?Q?j60jmON4MK2s5bWb+70gQOWIfjH4w1VA94y2q0UJBeFDUdGJ+cbyF6uARiQu?= =?us-ascii?Q?dhuJXMyNX+oYrUqKCIuINxbdyF2vUpXzrloxjgCTGVpxXIknbsif0+a2CjJC?= =?us-ascii?Q?StC3NtdslWxTTwJnT20zB+8wBWWXLWCEfoSE9TY53fHLgvSgi80J0oNa6e1F?= =?us-ascii?Q?NHDDAMOmim7k7vHRPKDu77xgOi8b0y9KAxjr5+UoIc6F4sRkYy95yPg2F4Us?= =?us-ascii?Q?xOVHu0wIukWh/1j2S2DRi5t2ZFzkTtSK64VdlZk4Gphgc4CS6ltrSwhugNUZ?= =?us-ascii?Q?LglGRHhDuE/9kktUqQd7Cbq59+6rIUSvLvgx62sAXLs1aH0+o3Y8tfcRDYFo?= =?us-ascii?Q?6gf4l1JV4LI8Dr6g7b68lFSQOqp437PL/U2s+RvKUt2aIwweJZ010HikP/r4?= =?us-ascii?Q?BJaRhEBow6RJlHH9TNqv1lMhyhF5L5Tl8zUeY2kCn4R6H2JvTOyYEM7FwlKP?= =?us-ascii?Q?9DyIFAE6YTcSTo02b2xgrq/Ogp8edGM=3D?= X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-Network-Message-Id: 805af821-3e1f-42cd-33cb-08da22de96e8 X-MS-Exchange-CrossTenant-AuthSource: PA4PR10MB5780.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Apr 2022 15:01:03.8105 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: wpN5OmrQ6Du01cUK8ChQdijsWDcJ8gOjqwfgpnVYg5iM14hNYW60xaM3wt6tQq1ITi+xXgKRJ5FqG3RkZ3Vxur7sXWigh05nC1hh1iW+RHE= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR10MB2898 X-TUID: vSpBmcGWGncJ While that kind of works, the next trouble with python 3.10 will be in wic. I propose to merge my 3.9 patch quickly and deal with the wic and bitbake bump later. bookworm is important and should not be allowed to fail in CI. Henning Am Wed, 20 Apr 2022 16:35:40 +0200 schrieb Henning Schild : > Update bitbake to the latest release in 1.50 branch. This release is > tagged to the commit ID aaa7f7af23d5f89fe4a5ed48c57ea3dfca07c79d > in the bitbake upstream. > > Signed-off-by: Henning Schild > --- > bitbake/lib/bb/cache.py | 3 +- > bitbake/lib/bb/cooker.py | 30 ++++++++++++++-- > bitbake/lib/bb/data_smart.py | 4 +-- > bitbake/lib/bb/fetch2/__init__.py | 4 +++ > bitbake/lib/bb/fetch2/perforce.py | 2 +- > bitbake/lib/bb/fetch2/wget.py | 2 +- > bitbake/lib/bb/persist_data.py | 5 +-- > bitbake/lib/bb/process.py | 2 +- > bitbake/lib/bb/runqueue.py | 34 +++++++++--------- > bitbake/lib/bb/server/process.py | 2 +- > bitbake/lib/bb/tests/fetch.py | 35 > ++++++++++--------- bitbake/lib/bb/utils.py | > 13 +++++-- bitbake/lib/hashserv/server.py | 4 +-- > bitbake/lib/toaster/tests/builds/buildtest.py | 2 +- > 14 files changed, 89 insertions(+), 53 deletions(-) > > diff --git a/bitbake/lib/bb/cache.py b/bitbake/lib/bb/cache.py > index 27eb271798e8..5f9c0a779d75 100644 > --- a/bitbake/lib/bb/cache.py > +++ b/bitbake/lib/bb/cache.py > @@ -19,7 +19,8 @@ > import os > import logging > import pickle > -from collections import defaultdict, Mapping > +from collections import defaultdict > +from collections.abc import Mapping > import bb.utils > from bb import PrefixLoggerAdapter > import re > diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py > index 89f1fad08310..c946800a8c62 100644 > --- a/bitbake/lib/bb/cooker.py > +++ b/bitbake/lib/bb/cooker.py > @@ -388,12 +388,22 @@ class BBCooker: > # Create a new hash server bound to a unix domain socket > if not self.hashserv: > dbfile = (self.data.getVar("PERSISTENT_DIR") or > self.data.getVar("CACHE")) + "/hashserv.db" > + upstream = self.data.getVar("BB_HASHSERVE_UPSTREAM") > or None > + if upstream: > + import socket > + try: > + sock = > socket.create_connection(upstream.split(":"), 5) > + sock.close() > + except socket.error as e: > + bb.warn("BB_HASHSERVE_UPSTREAM is not valid, > unable to connect hash equivalence server at '%s': %s" > + % (upstream, repr(e))) > + > self.hashservaddr = "unix://%s/hashserve.sock" % > self.data.getVar("TOPDIR") self.hashserv = hashserv.create_server( > self.hashservaddr, > dbfile, > sync=False, > - > upstream=self.data.getVar("BB_HASHSERVE_UPSTREAM") or None, > + upstream=upstream, > ) > self.hashserv.process = > multiprocessing.Process(target=self.hashserv.serve_forever) > self.hashserv.process.start() @@ -805,7 +815,9 @@ class BBCooker: > for dep in rq.rqdata.runtaskentries[tid].depends: > (depmc, depfn, _, deptaskfn) = > bb.runqueue.split_tid_mcfn(dep) deppn = > self.recipecaches[depmc].pkg_fn[deptaskfn] > - depend_tree["tdepends"][dotname].append("%s.%s" % > (deppn, bb.runqueue.taskname_from_tid(dep))) > + if depmc: > + depmc = "mc:" + depmc + ":" > + depend_tree["tdepends"][dotname].append("%s%s.%s" % > (depmc, deppn, bb.runqueue.taskname_from_tid(dep))) if taskfn not in > seen_fns: seen_fns.append(taskfn) > packages = [] > @@ -2204,21 +2216,33 @@ class CookerParser(object): > yield not cached, mc, infos > > def parse_generator(self): > - while True: > + empty = False > + while self.processes or not empty: > + for process in self.processes.copy(): > + if not process.is_alive(): > + process.join() > + self.processes.remove(process) > + > if self.parsed >= self.toparse: > break > > try: > result = self.result_queue.get(timeout=0.25) > except queue.Empty: > + empty = True > pass > else: > + empty = False > value = result[1] > if isinstance(value, BaseException): > raise value > else: > yield result > > + if not (self.parsed >= self.toparse): > + raise bb.parse.ParseError("Not all recipes parsed, > parser thread killed/died? Exiting.", None) + > + > def parse_next(self): > result = [] > parsed = None > diff --git a/bitbake/lib/bb/data_smart.py > b/bitbake/lib/bb/data_smart.py index 8291ca65e309..65857a9c7941 100644 > --- a/bitbake/lib/bb/data_smart.py > +++ b/bitbake/lib/bb/data_smart.py > @@ -17,7 +17,7 @@ BitBake build tools. > # Based on functions from the base bb module, Copyright 2003 Holger > Schurig > import copy, re, sys, traceback > -from collections import MutableMapping > +from collections.abc import MutableMapping > import logging > import hashlib > import bb, bb.codeparser > @@ -403,7 +403,7 @@ class DataSmart(MutableMapping): > s = > __expand_python_regexp__.sub(varparse.python_sub, s) except > SyntaxError as e: # Likely unmatched brackets, just don't expand the > expression > - if e.msg != "EOL while scanning string literal": > + if e.msg != "EOL while scanning string literal" > and not e.msg.startswith("unterminated string literal"): raise > if s == olds: > break > diff --git a/bitbake/lib/bb/fetch2/__init__.py > b/bitbake/lib/bb/fetch2/__init__.py index dbf8b50e68a7..1005ec10c639 > 100644 --- a/bitbake/lib/bb/fetch2/__init__.py > +++ b/bitbake/lib/bb/fetch2/__init__.py > @@ -430,6 +430,7 @@ def uri_replace(ud, uri_find, uri_replace, > replacements, d, mirrortarball=None): uri_replace_decoded = > list(decodeurl(uri_replace)) logger.debug2("For url %s comparing %s > to %s" % (uri_decoded, uri_find_decoded, uri_replace_decoded)) > result_decoded = ['', '', '', '', '', {}] > + # 0 - type, 1 - host, 2 - path, 3 - user, 4- pswd, 5 - params > for loc, i in enumerate(uri_find_decoded): > result_decoded[loc] = uri_decoded[loc] > regexp = i > @@ -449,6 +450,9 @@ def uri_replace(ud, uri_find, uri_replace, > replacements, d, mirrortarball=None): for l in replacements: > uri_replace_decoded[loc][k] = > uri_replace_decoded[loc][k].replace(l, replacements[l]) > result_decoded[loc][k] = uri_replace_decoded[loc][k] > + elif (loc == 3 or loc == 4) and uri_replace_decoded[loc]: > + # User/password in the replacement is just a straight > replacement > + result_decoded[loc] = uri_replace_decoded[loc] > elif (re.match(regexp, uri_decoded[loc])): > if not uri_replace_decoded[loc]: > result_decoded[loc] = "" > diff --git a/bitbake/lib/bb/fetch2/perforce.py > b/bitbake/lib/bb/fetch2/perforce.py index e2a41a4a1287..3b6fa4b1ec9a > 100644 --- a/bitbake/lib/bb/fetch2/perforce.py > +++ b/bitbake/lib/bb/fetch2/perforce.py > @@ -134,7 +134,7 @@ class Perforce(FetchMethod): > > ud.setup_revisions(d) > > - ud.localfile = d.expand('%s_%s_%s_%s.tar.gz' % (cleanedhost, > cleanedpath, cleandedmodule, ud.revision)) > + ud.localfile = d.expand('%s_%s_%s_%s.tar.gz' % (cleanedhost, > cleanedpath, cleanedmodule, ud.revision)) > def _buildp4command(self, ud, d, command, depot_filename=None): > """ > diff --git a/bitbake/lib/bb/fetch2/wget.py > b/bitbake/lib/bb/fetch2/wget.py index 784df70c9f62..7fa2a87ffde5 > 100644 --- a/bitbake/lib/bb/fetch2/wget.py > +++ b/bitbake/lib/bb/fetch2/wget.py > @@ -322,7 +322,7 @@ class Wget(FetchMethod): > except (TypeError, ImportError, IOError, > netrc.NetrcParseError): pass > > - with opener.open(r) as response: > + with opener.open(r, timeout=30) as response: > pass > except urllib.error.URLError as e: > if try_again: > diff --git a/bitbake/lib/bb/persist_data.py > b/bitbake/lib/bb/persist_data.py index c6a209fb3fc1..6f32d81afe80 > 100644 --- a/bitbake/lib/bb/persist_data.py > +++ b/bitbake/lib/bb/persist_data.py > @@ -12,6 +12,7 @@ currently, providing a key/value store accessed by > 'domain'. # > > import collections > +import collections.abc > import contextlib > import functools > import logging > @@ -19,7 +20,7 @@ import os.path > import sqlite3 > import sys > import warnings > -from collections import Mapping > +from collections.abc import Mapping > > sqlversion = sqlite3.sqlite_version_info > if sqlversion[0] < 3 or (sqlversion[0] == 3 and sqlversion[1] < 3): > @@ -29,7 +30,7 @@ if sqlversion[0] < 3 or (sqlversion[0] == 3 and > sqlversion[1] < 3): logger = logging.getLogger("BitBake.PersistData") > > @functools.total_ordering > -class SQLTable(collections.MutableMapping): > +class SQLTable(collections.abc.MutableMapping): > class _Decorators(object): > @staticmethod > def retry(*, reconnect=True): > diff --git a/bitbake/lib/bb/process.py b/bitbake/lib/bb/process.py > index d5a1775fcec0..af5d804a1d59 100644 > --- a/bitbake/lib/bb/process.py > +++ b/bitbake/lib/bb/process.py > @@ -60,7 +60,7 @@ class Popen(subprocess.Popen): > "close_fds": True, > "preexec_fn": subprocess_setup, > "stdout": subprocess.PIPE, > - "stderr": subprocess.STDOUT, > + "stderr": subprocess.PIPE, > "stdin": subprocess.PIPE, > "shell": False, > } > diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py > index 10511a09dc1c..cd10da8b3a6f 100644 > --- a/bitbake/lib/bb/runqueue.py > +++ b/bitbake/lib/bb/runqueue.py > @@ -926,38 +926,36 @@ class RunQueueData: > # > # Once all active tasks are marked, prune the ones we don't > need. > - delcount = {} > - for tid in list(self.runtaskentries.keys()): > - if tid not in runq_build: > - delcount[tid] = self.runtaskentries[tid] > - del self.runtaskentries[tid] > - > # Handle --runall > if self.cooker.configuration.runall: > # re-run the mark_active and then drop unused tasks from > new list > + reduced_tasklist = set(self.runtaskentries.keys()) > + for tid in list(self.runtaskentries.keys()): > + if tid not in runq_build: > + reduced_tasklist.remove(tid) > runq_build = {} > > for task in self.cooker.configuration.runall: > if not task.startswith("do_"): > task = "do_{0}".format(task) > runall_tids = set() > - for tid in list(self.runtaskentries): > + for tid in reduced_tasklist: > wanttid = "{0}:{1}".format(fn_from_tid(tid), > task) > - if wanttid in delcount: > - self.runtaskentries[wanttid] = > delcount[wanttid] if wanttid in self.runtaskentries: > runall_tids.add(wanttid) > > for tid in list(runall_tids): > - mark_active(tid,1) > + mark_active(tid, 1) > if self.cooker.configuration.force: > invalidate_task(tid, False) > > - for tid in list(self.runtaskentries.keys()): > - if tid not in runq_build: > - delcount[tid] = self.runtaskentries[tid] > - del self.runtaskentries[tid] > + delcount = set() > + for tid in list(self.runtaskentries.keys()): > + if tid not in runq_build: > + delcount.add(tid) > + del self.runtaskentries[tid] > > + if self.cooker.configuration.runall: > if len(self.runtaskentries) == 0: > bb.msg.fatal("RunQueue", "Could not find any tasks > with the tasknames %s to run within the recipes of the taskgraphs of > the targets %s" % (str(self.cooker.configuration.runall), > str(self.targets))) @@ -971,16 +969,16 @@ class RunQueueData: for > task in self.cooker.configuration.runonly: if not > task.startswith("do_"): task = "do_{0}".format(task) > - runonly_tids = { k: v for k, v in > self.runtaskentries.items() if taskname_from_tid(k) == task } > + runonly_tids = [k for k in > self.runtaskentries.keys() if taskname_from_tid(k) == task] > - for tid in list(runonly_tids): > - mark_active(tid,1) > + for tid in runonly_tids: > + mark_active(tid, 1) > if self.cooker.configuration.force: > invalidate_task(tid, False) > > for tid in list(self.runtaskentries.keys()): > if tid not in runq_build: > - delcount[tid] = self.runtaskentries[tid] > + delcount.add(tid) > del self.runtaskentries[tid] > > if len(self.runtaskentries) == 0: > diff --git a/bitbake/lib/bb/server/process.py > b/bitbake/lib/bb/server/process.py index 07bb785a1822..fcdce19717d2 > 100644 --- a/bitbake/lib/bb/server/process.py > +++ b/bitbake/lib/bb/server/process.py > @@ -659,7 +659,7 @@ class BBUIEventQueue: > self.reader = ConnectionReader(readfd) > > self.t = threading.Thread() > - self.t.setDaemon(True) > + self.t.daemon = True > self.t.run = self.startCallbackHandler > self.t.start() > > diff --git a/bitbake/lib/bb/tests/fetch.py > b/bitbake/lib/bb/tests/fetch.py index f5d557e8c000..3b64584da0e9 > 100644 --- a/bitbake/lib/bb/tests/fetch.py > +++ b/bitbake/lib/bb/tests/fetch.py > @@ -431,6 +431,10 @@ class MirrorUriTest(FetcherTest): > ("git://someserver.org/bitbake;tag=1234567890123456789012345678901234567890;branch=master", > "git://someserver.org/bitbake;branch=master", > "git://git.openembedded.org/bitbake;protocol=http") : > "git://git.openembedded.org/bitbake;tag=1234567890123456789012345678901234567890;branch=master;protocol=http", > + > ("git://user1@someserver.org/bitbake;tag=1234567890123456789012345678901234567890;branch=master", > "git://someserver.org/bitbake;branch=master", > "git://user2@git.openembedded.org/bitbake;protocol=http") > + : > "git://user2@git.openembedded.org/bitbake;tag=1234567890123456789012345678901234567890;branch=master;protocol=http", > + + > #Renaming files doesn't work > #("http://somewhere.org/somedir1/somefile_1.2.3.tar.gz", > "http://somewhere.org/somedir1/somefile_1.2.3.tar.gz", > "http://somewhere2.org/somedir3/somefile_2.3.4.tar.gz") : > "http://somewhere2.org/somedir3/somefile_2.3.4.tar.gz" > #("file://sstate-xyz.tgz", "file://.*/.*", > "file:///somewhere/1234/sstate-cache") : > "file:///somewhere/1234/sstate-cache/sstate-xyz.tgz", @@ -491,7 > +495,7 @@ class GitDownloadDirectoryNamingTest(FetcherTest): > super(GitDownloadDirectoryNamingTest, self).setUp() self.recipe_url = > "git://git.openembedded.org/bitbake" self.recipe_dir = > "git.openembedded.org.bitbake" > - self.mirror_url = "git://github.com/openembedded/bitbake.git" > + self.mirror_url = > "git://github.com/openembedded/bitbake.git;protocol=https" > self.mirror_dir = "github.com.openembedded.bitbake.git" > self.d.setVar('SRCREV', > '82ea737a0b42a8b53e11c9cde141e9e9c0bd8c40') @@ -539,7 +543,7 @@ class > TarballNamingTest(FetcherTest): super(TarballNamingTest, self).setUp() > self.recipe_url = "git://git.openembedded.org/bitbake" > self.recipe_tarball = > "git2_git.openembedded.org.bitbake.tar.gz" > - self.mirror_url = "git://github.com/openembedded/bitbake.git" > + self.mirror_url = > "git://github.com/openembedded/bitbake.git;protocol=https" > self.mirror_tarball = > "git2_github.com.openembedded.bitbake.git.tar.gz" > self.d.setVar('BB_GENERATE_MIRROR_TARBALLS', '1') @@ -573,7 +577,7 @@ > class GitShallowTarballNamingTest(FetcherTest): > super(GitShallowTarballNamingTest, self).setUp() self.recipe_url = > "git://git.openembedded.org/bitbake" self.recipe_tarball = > "gitshallow_git.openembedded.org.bitbake_82ea737-1_master.tar.gz" > - self.mirror_url = "git://github.com/openembedded/bitbake.git" > + self.mirror_url = > "git://github.com/openembedded/bitbake.git;protocol=https" > self.mirror_tarball = > "gitshallow_github.com.openembedded.bitbake.git_82ea737-1_master.tar.gz" > self.d.setVar('BB_GIT_SHALLOW', '1') @@ -985,7 +989,7 @@ class > FetcherNetworkTest(FetcherTest): def > test_git_submodule_dbus_broker(self): # The following external > repositories have show failures in fetch and unpack operations # We > want to avoid regressions! > - url = > "gitsm://github.com/bus1/dbus-broker;protocol=git;rev=fc874afa0992d0c75ec25acb43d344679f0ee7d2;branch=main" > + url = > "gitsm://github.com/bus1/dbus-broker;protocol=https;rev=fc874afa0992d0c75ec25acb43d344679f0ee7d2;branch=main" > fetcher = bb.fetch.Fetch([url], self.d) fetcher.download() > # Previous cwd has been deleted > @@ -1001,7 +1005,7 @@ class FetcherNetworkTest(FetcherTest): > > @skipIfNoNetwork() > def test_git_submodule_CLI11(self): > - url = > "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=bd4dc911847d0cde7a6b41dfa626a85aab213baf;branch=main" > + url = > "gitsm://github.com/CLIUtils/CLI11;protocol=https;rev=bd4dc911847d0cde7a6b41dfa626a85aab213baf;branch=main" > fetcher = bb.fetch.Fetch([url], self.d) fetcher.download() > # Previous cwd has been deleted > @@ -1016,12 +1020,12 @@ class FetcherNetworkTest(FetcherTest): > @skipIfNoNetwork() > def test_git_submodule_update_CLI11(self): > """ Prevent regression on update detection not finding > missing submodule, or modules without needed commits """ > - url = > "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=cf6a99fa69aaefe477cc52e3ef4a7d2d7fa40714;branch=main" > + url = > "gitsm://github.com/CLIUtils/CLI11;protocol=https;rev=cf6a99fa69aaefe477cc52e3ef4a7d2d7fa40714;branch=main" > fetcher = bb.fetch.Fetch([url], self.d) fetcher.download() > > # CLI11 that pulls in a newer nlohmann-json > - url = > "gitsm://github.com/CLIUtils/CLI11;protocol=git;rev=49ac989a9527ee9bb496de9ded7b4872c2e0e5ca;branch=main" > + url = > "gitsm://github.com/CLIUtils/CLI11;protocol=https;rev=49ac989a9527ee9bb496de9ded7b4872c2e0e5ca;branch=main" > fetcher = bb.fetch.Fetch([url], self.d) fetcher.download() > # Previous cwd has been deleted > @@ -1035,7 +1039,7 @@ class FetcherNetworkTest(FetcherTest): > > @skipIfNoNetwork() > def test_git_submodule_aktualizr(self): > - url = > "gitsm://github.com/advancedtelematic/aktualizr;branch=master;protocol=git;rev=d00d1a04cc2366d1a5f143b84b9f507f8bd32c44" > + url = > "gitsm://github.com/advancedtelematic/aktualizr;branch=master;protocol=https;rev=d00d1a04cc2366d1a5f143b84b9f507f8bd32c44" > fetcher = bb.fetch.Fetch([url], self.d) fetcher.download() > # Previous cwd has been deleted > @@ -1055,7 +1059,7 @@ class FetcherNetworkTest(FetcherTest): > """ Prevent regression on deeply nested submodules not being > checked out properly, even though they were fetched. """ > # This repository also has submodules where the module > (name), path and url do not align > - url = > "gitsm://github.com/azure/iotedge.git;protocol=git;rev=d76e0316c6f324345d77c48a83ce836d09392699" > + url = > "gitsm://github.com/azure/iotedge.git;protocol=https;rev=d76e0316c6f324345d77c48a83ce836d09392699" > fetcher = bb.fetch.Fetch([url], self.d) fetcher.download() > # Previous cwd has been deleted > @@ -1113,7 +1117,7 @@ class SVNTest(FetcherTest): > > bb.process.run("svn co %s svnfetch_co" % self.repo_url, > cwd=self.tempdir) # Github will emulate SVN. Use this to check if > we're downloding... > - bb.process.run("svn propset svn:externals 'bitbake > svn://vcs.pcre.org/pcre2/code' .", > + bb.process.run("svn propset svn:externals 'bitbake > https://github.com/PhilipHazel/pcre2.git' .", > cwd=os.path.join(self.tempdir, 'svnfetch_co', 'trunk')) > bb.process.run("svn commit --non-interactive -m 'Add external'", > cwd=os.path.join(self.tempdir, 'svnfetch_co', 'trunk')) @@ -1231,7 > +1235,7 @@ class FetchLatestVersionTest(FetcherTest): > test_git_uris = { > # version pattern "X.Y.Z" > - ("mx-1.0", > "git://github.com/clutter-project/mx.git;branch=mx-1.4", > "9b1db6b8060bd00b121a692f942404a24ae2960f", "") > + ("mx-1.0", > "git://github.com/clutter-project/mx.git;branch=mx-1.4;protocol=https", > "9b1db6b8060bd00b121a692f942404a24ae2960f", "") : "1.99.4", # version > pattern "vX.Y" # mirror of git.infradead.org since network issues > interfered with testing @@ -1258,9 +1262,9 @@ class > FetchLatestVersionTest(FetcherTest): : "0.4.3", > ("build-appliance-image", "git://git.yoctoproject.org/poky", > "b37dd451a52622d5b570183a81583cc34c2ff555", > "(?P(([0-9][\.|_]?)+[0-9]))") : "11.0.0", > - ("chkconfig-alternatives-native", > "git://github.com/kergoth/chkconfig;branch=sysroot", > "cd437ecbd8986c894442f8fce1e0061e20f04dee", > "chkconfig\-(?P((\d+[\.\-_]*)+))") > + ("chkconfig-alternatives-native", > "git://github.com/kergoth/chkconfig;branch=sysroot;protocol=https", > "cd437ecbd8986c894442f8fce1e0061e20f04dee", > "chkconfig\-(?P((\d+[\.\-_]*)+))") : "1.3.59", > - ("remake", "git://github.com/rocky/remake.git", > "f05508e521987c8494c92d9c2871aec46307d51d", > "(?P(\d+\.(\d+\.)*\d*(\+dbg\d+(\.\d+)*)*))") > + ("remake", > "git://github.com/rocky/remake.git;protocol=https", > "f05508e521987c8494c92d9c2871aec46307d51d", > "(?P(\d+\.(\d+\.)*\d*(\+dbg\d+(\.\d+)*)*))") : "3.82+dbg0.9", } > @@ -1354,9 +1358,6 @@ class FetchCheckStatusTest(FetcherTest): > "http://downloads.yoctoproject.org/releases/opkg/opkg-0.1.7.tar.gz", > "http://downloads.yoctoproject.org/releases/opkg/opkg-0.3.0.tar.gz", > "ftp://sourceware.org/pub/libffi/libffi-1.20.tar.gz", > - > "http://ftp.gnu.org/gnu/autoconf/autoconf-2.60.tar.gz", > - > "https://ftp.gnu.org/gnu/chess/gnuchess-5.08.tar.gz", > - "https://ftp.gnu.org/gnu/gmp/gmp-4.0.tar.gz", > # GitHub releases are hosted on Amazon S3, > which doesn't support HEAD > "https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz" > ] @@ -2047,7 +2048,7 @@ class GitShallowTest(FetcherTest): > > @skipIfNoNetwork() > def test_bitbake(self): > - self.git('remote add --mirror=fetch origin > git://github.com/openembedded/bitbake', cwd=self.srcdir) > + self.git('remote add --mirror=fetch origin > https://github.com/openembedded/bitbake', cwd=self.srcdir) > self.git('config core.bare true', cwd=self.srcdir) self.git('fetch', > cwd=self.srcdir) > diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py > index b282d09abfce..2a150fe9c73e 100644 > --- a/bitbake/lib/bb/utils.py > +++ b/bitbake/lib/bb/utils.py > @@ -16,7 +16,8 @@ import bb.msg > import multiprocessing > import fcntl > import importlib > -from importlib import machinery > +import importlib.machinery > +import importlib.util > import itertools > import subprocess > import glob > @@ -451,6 +452,10 @@ def lockfile(name, shared=False, retry=True, > block=False): consider the possibility of sending a signal to the > process to break out - at which point you want block=True rather than > retry=True. """ > + if len(name) > 255: > + root, ext = os.path.splitext(name) > + name = root[:255 - len(ext)] + ext > + > dirname = os.path.dirname(name) > mkdirhier(dirname) > > @@ -487,7 +492,7 @@ def lockfile(name, shared=False, retry=True, > block=False): return lf > lf.close() > except OSError as e: > - if e.errno == errno.EACCES: > + if e.errno == errno.EACCES or e.errno == > errno.ENAMETOOLONG: logger.error("Unable to acquire lock '%s', %s", > e.strerror, name) > sys.exit(1) > @@ -1616,7 +1621,9 @@ def load_plugins(logger, plugins, pluginpath): > logger.debug('Loading plugin %s' % name) > spec = importlib.machinery.PathFinder.find_spec(name, > path=[pluginpath] ) if spec: > - return spec.loader.load_module() > + mod = importlib.util.module_from_spec(spec) > + spec.loader.exec_module(mod) > + return mod > > logger.debug('Loading plugins from %s...' % pluginpath) > > diff --git a/bitbake/lib/hashserv/server.py > b/bitbake/lib/hashserv/server.py index a0dc0c170f2b..df0fa0a07937 > 100644 --- a/bitbake/lib/hashserv/server.py > +++ b/bitbake/lib/hashserv/server.py > @@ -521,7 +521,7 @@ class Server(object): > > def start_tcp_server(self, host, port): > self.server = self.loop.run_until_complete( > - asyncio.start_server(self.handle_client, host, port, > loop=self.loop) > + asyncio.start_server(self.handle_client, host, port) > ) > > for s in self.server.sockets: > @@ -546,7 +546,7 @@ class Server(object): > # Work around path length limits in AF_UNIX > os.chdir(os.path.dirname(path)) > self.server = self.loop.run_until_complete( > - asyncio.start_unix_server(self.handle_client, > os.path.basename(path), loop=self.loop) > + asyncio.start_unix_server(self.handle_client, > os.path.basename(path)) ) > finally: > os.chdir(cwd) > diff --git a/bitbake/lib/toaster/tests/builds/buildtest.py > b/bitbake/lib/toaster/tests/builds/buildtest.py index > 872bbd377510..13b51fb0d8e4 100644 --- > a/bitbake/lib/toaster/tests/builds/buildtest.py +++ > b/bitbake/lib/toaster/tests/builds/buildtest.py @@ -119,7 +119,7 @@ > class BuildTest(unittest.TestCase): if > os.environ.get("TOASTER_TEST_USE_SSTATE_MIRROR"): > ProjectVariable.objects.get_or_create( name="SSTATE_MIRRORS", > - value="file://.* > http://autobuilder.yoctoproject.org/pub/sstate/PATH;downloadfilename=PATH", > + value="file://.* > http://sstate.yoctoproject.org/PATH;downloadfilename=PATH", > project=project) > ProjectTarget.objects.create(project=project,