From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7260007084413943808 X-Received: by 2002:a2e:3305:0:b0:2b9:ac48:d7fe with SMTP id d5-20020a2e3305000000b002b9ac48d7femr1331390ljc.38.1690451492577; Thu, 27 Jul 2023 02:51:32 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:651c:c85:b0:2b9:722f:8f69 with SMTP id bz5-20020a05651c0c8500b002b9722f8f69ls28167ljb.1.-pod-prod-03-eu; Thu, 27 Jul 2023 02:51:30 -0700 (PDT) X-Google-Smtp-Source: APBJJlERt5u6zFFnlc4C7WVVsyudn+WAwoSUhqh5sr58tl2c8h6NyR7ZCw5bh6IzaxVOttec74tr X-Received: by 2002:a2e:8883:0:b0:2b6:cf18:77e0 with SMTP id k3-20020a2e8883000000b002b6cf1877e0mr1354493lji.48.1690451490570; Thu, 27 Jul 2023 02:51:30 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1690451490; cv=pass; d=google.com; s=arc-20160816; b=X+VNas/O7ix63lBkTpgsSCjdI+IYFeUFJgOVMAjU9FmONtqXbG60iww+5eN+TDgp0T DWNIqNQ3ehQxs2FC/JnKwuFO6mMFNifGjmRlGOqhB7TZ32nPEctJrcFD0GZVYqfp2key iygr+p2en8iLuMyx5hNc1VTYo60Ab+BYbE4fXFHx35ehigIq1hctDF9X77PBL1ZIbMzs W32ya1xWNoS5b3Lklvtmz4HmRa4JF6ojjcmCwFPSv1mfqE8k6M4jainwr4WciRM3jFGi YA5w7itmzheXEZ9dbIfl5va1ZC9GFFDqSOOLIIoleRIG7CTlxEz4t7SISE8uB6Q+7U++ bvrA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:content-transfer-encoding:in-reply-to:from:references :to:content-language:subject:user-agent:date:message-id :dkim-signature; bh=r0uNyPsPf7nsxydonPdCeHH09XSxufudkliMGriYtSU=; fh=lBT1xG8sIcSVpYk65+y0SgeX2pmTnaCwJLR76Gk1pwM=; b=u5Pc17lE7gSDmfOjr4eqjevGSFlB6xDDWqseH+f60ODqgXeXyqhdZYLaV+qW0MD0ic Kl5BqkEWZVs+vwUCYSgislPs+dvJtDflDN5POZLDZiXk2fnXL1IiOzrvgJtopMlnR8t1 Ts/axp7KuSa273Rez3NVQ7d9piOEf2kGnRG905KTuJ+87XNXDYWD2wnCJpD5hcCISfMs 1mmv0SQihdfjArE7zW7/0q77T9udqHsPWEVsi/YP5FzFLm928z2QySbuPw0CDUEtuL/I vaCunXxdyCA2xuOjGqOlaBCFFgSqMwxEQAGez9+R1HN/oEk2h+cPstgB8/mdjJ6HKOPT k1lA== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=KnCamFLJ; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of quirin.gylstorff@siemens.com designates 2a01:111:f400:7eaf::61f as permitted sender) smtp.mailfrom=quirin.gylstorff@siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Return-Path: Received: from EUR03-AM7-obe.outbound.protection.outlook.com (mail-am7eur03on2061f.outbound.protection.outlook.com. [2a01:111:f400:7eaf::61f]) by gmr-mx.google.com with ESMTPS id w11-20020a2e160b000000b002b657edbea8si110362ljd.4.2023.07.27.02.51.30 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 27 Jul 2023 02:51:30 -0700 (PDT) Received-SPF: pass (google.com: domain of quirin.gylstorff@siemens.com designates 2a01:111:f400:7eaf::61f as permitted sender) client-ip=2a01:111:f400:7eaf::61f; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=KnCamFLJ; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of quirin.gylstorff@siemens.com designates 2a01:111:f400:7eaf::61f as permitted sender) smtp.mailfrom=quirin.gylstorff@siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=VjclZNTTGC3GdPOqFdoP5WzTuy9YleXSiXQHCUaH9GHbVHknz/yUhnUS6afpX173UEb7CMTsBfZFb/j7VX58tBvWJOZj9KWl92hF9VmLgrRaUBN9aJDZbJHGJytRWJbXoN/1UXZd2p+hXCY4gIsUxFzXl2DMpAmb9I6a7Rn2leQHGmgwTu59u0fH5WVrkaB71Go2ahZOAihnObcj/oFqCn549WaZgkUitjgBrYxvXLQ1v2LnUcbj1Ibo8iux3HklvT4CSRK15ZWJXb+FbxBnwACkSnRGlqw3g0uQW8nxdHF7nmFH1zAvaa3yFQ0ydVKYeL7tapeYhIUum7cjVVE+pA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=r0uNyPsPf7nsxydonPdCeHH09XSxufudkliMGriYtSU=; b=aiLnsD4XjpJrzs9m3ymenuCWix1O26vsq2ThBbfX1lxPrtElNega3q4XPQ35WVA4+LfpbG7aG0GpeOQwV9qrQGsfMacpFis6i8Hbc9hYemKsFjyV51CRo4eg6SFRuYcfwr/P0928FWZ1LMYN4dOOUaDnji2eVLFzBaV/OqEvaLCgPOMtEaHuiaAFkOnjnwUo49fGkX9+qseidXn8fdJdJpBxiweAcrZ13gQBtGjNrQvtjsd0zadD9iESJy9izXOGnOlCpibGmPJW5lfSWv1hBOwEBkTmjL+bbsN+4Ix1OjowHh4FAJ/DNxtBmIo/Ea5I+LOSGlfCLSPAjAbYHMD80w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=siemens.com; dmarc=pass action=none header.from=siemens.com; dkim=pass header.d=siemens.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=siemens.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=r0uNyPsPf7nsxydonPdCeHH09XSxufudkliMGriYtSU=; b=KnCamFLJBB5+FGfx7pa0YACO7R6NRApVBxaz2dVZy8vV1CwUnwR5LPnhl/WuZNpkoOiuNc2HI8ZXcxxqllhElz6JcVo3TLjGvyvufv4hwVlt6gMATmhbqyD3lIYd0TY+kz3ScQSvq6GSYF+BunZW+MgsjMOf6q+G5zxeukPjAY4lqkXBQ6lLv4UZoCZw/g2AyZ7V2FHJeLdmEMytzvV6khH+aCoAYcnnS/+rGVPMWLNykyoYoTE10fOfPSn+9Zm74+WsaufiT5PqjtcInvaIxEs1r6YKQtPcwjKLmwy1UtWECWDimgQmekXHpSmQtfuq4oG33uqO94XrlaiSamKFiw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=siemens.com; Received: from AM9PR10MB4085.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:1f9::22) by AS2PR10MB6926.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:57a::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6631.29; Thu, 27 Jul 2023 09:51:29 +0000 Received: from AM9PR10MB4085.EURPRD10.PROD.OUTLOOK.COM ([fe80::90a8:2018:8363:8691]) by AM9PR10MB4085.EURPRD10.PROD.OUTLOOK.COM ([fe80::90a8:2018:8363:8691%7]) with mapi id 15.20.6631.026; Thu, 27 Jul 2023 09:51:28 +0000 Message-ID: <2838682f-6725-b9df-ad83-d037316afd2a@siemens.com> Date: Thu, 27 Jul 2023 11:51:26 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 Subject: Re: [PATCH 1/1] deduplicate apt sources.list based on filenames Content-Language: en-US, de-DE To: "MOESSBAUER, Felix (T CED INW-CN)" , "isar-users@googlegroups.com" References: <20230726061457.2297344-1-felix.moessbauer@siemens.com> <6f839d5b-4316-6523-368d-e1b3e450c35e@siemens.com> <15630e37d0cd4deab9e9167619e205dedd53820d.camel@siemens.com> From: Gylstorff Quirin In-Reply-To: <15630e37d0cd4deab9e9167619e205dedd53820d.camel@siemens.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-ClientProxiedBy: FR2P281CA0013.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:a::23) To AM9PR10MB4085.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:1f9::22) Return-Path: quirin.gylstorff@siemens.com MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AM9PR10MB4085:EE_|AS2PR10MB6926:EE_ X-MS-Office365-Filtering-Correlation-Id: 3000aae5-9c5d-446a-f906-08db8e870c92 X-LD-Processed: 38ae3bcd-9579-4fd4-adda-b42e1495d55a,ExtAddr X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: hlc2anO1CUD+8b85vNClOX4ZYQG5XYLhLYvLSKy6GIjNQKiR60yJ5Se4yWlgalEs6pMHUSlJZ2YGt2YQOhWyeHstaCp9ZVeqHYdrYvshwS4OcPgDGgGfnc6gWInlUG4k3np7kPv0wgYXQ8Zxrg1jFobvkbIp/rSGRiUf8kVPCOME2N6Jton2VLO+fQaCjulXA79Sj+drGklNXBPSWTcIbmESq8GyUUF8fagqvtgsolHIVbbba01k7f+i2UmWgnQ8nB1Og8iVdWqm1CutoJMO4sEDhjAVKanAwy7GrbvDOTia9H/7PBYhA8AAzrgFlqq2NDv9jDhQ+2FryFVAYcsOApbh/oBfwBi9mxIv2KPW8UxHY9LxjU+H/mJtoKrFwdpukbLWWdGP+7txnUc89nUAqcH7Dxu/ePIougle6LLPUCZHagyO0LmpDTNkWk+lDgna2U0cmK9gO5Z1rfo6ELoL3hVRDl5b1u9QVf5rZ2Fy0KoUpPcQACpKsR+nXdcCWbvYB1QZ3B6PumxeKx17q1gT0/+vmcZ75rvoKMqofIC1My4Rs/QJ+ADBYcVlPfetp0HYdAzMI7r1XMRaTH0PBLIWY+uLSCOnioEoZ6b1xxI/5WD2PgK9dC6PLdWZjDbKSVBNcH1RMsG8vGbphXTFanZu4A== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AM9PR10MB4085.EURPRD10.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230028)(4636009)(366004)(346002)(39860400002)(136003)(376002)(396003)(451199021)(2906002)(316002)(8936002)(8676002)(5660300002)(41300700001)(36756003)(86362001)(31696002)(6486002)(110136005)(478600001)(82960400001)(26005)(6506007)(53546011)(186003)(31686004)(2616005)(66946007)(66556008)(83380400001)(38100700002)(6512007)(66476007)(45980500001)(43740500002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?S3Q5ZVhVSEJ3dXc5OTRlbFlWYUVQdkN2S2FHbTBMZ0NWa0FielgwTndoQVpU?= =?utf-8?B?SGlxeDdhS0NoUXFBTk5VTHF5OTg0RHJlV3pteGtKRXdIQVBMczUyRCt5cWlK?= =?utf-8?B?R2FyUVVtZVBNSjhPbXhzamVFMlVHQlVialk0QWZxOWpOK0pzY1cxZ1dCdkVG?= =?utf-8?B?QXJIbytIa2Yyc2ZiNldyNlY5aHBVOTJTNWNPbkVwakhZNmZXU2E3SFZ5Vm14?= =?utf-8?B?NmY4NkdENG5FcGVjMFNiSHZOOUZYZjc3TEkvVzh6L1lkR09vR1llSmZyUEVx?= =?utf-8?B?bmhCMUN6dlBTazVQZTZ3b010ZHNnNzI5ZUwxVkJqdXZ6ZUVBUGJaZG9SOThC?= =?utf-8?B?cGhZL1ZaNnl2aHZlaDBsQzVQWWp5OEtSRVprS1dhZEpPVU4yOUpOYUxCd2cz?= =?utf-8?B?dVM0MTlYU09tWW0xbUtiU2ZBMFRiZU1TazdYRFd3OHFwNU1Ud3ZvWmxSZnNi?= =?utf-8?B?VGxxU3hUT21id0VFTThBMHdtOE84K0NDYjJ2WFBUNEg4YWRtWEVCaWdmZnVE?= =?utf-8?B?YnNhc2dtQjRWazhVU25JeTZsbkYyV2ZkdVZJamZpRXJUUHJKV1g4MUp2SUVx?= =?utf-8?B?aEZyaytMbThMaDhsL0VnWURiODVOblQxN09CblgwOUUzUzczdFdITnZjaTRF?= =?utf-8?B?Ui9QdWd5ckVaenE5eWg1eG50b3dpbTJyTmJvR2Z5YWFXaGxiQ2tQSzIrMVZF?= =?utf-8?B?WDhMWVRVdmY5VmNIUFE3QTB4bU5TaDdCMFh1aU8yY1ZQSEs0dExLaVlsbERz?= =?utf-8?B?ek9haXZqRHVKaEVZeUpQK3RPZGNDK214a3pTSGtwTEN3TkU3ZHd5SnlBNnBr?= =?utf-8?B?OVp5akN0UXZycmdIT3RXMDFLenAzMERlWkljNFhZZzljaXhhbkkzYkJHQTFT?= =?utf-8?B?ZlFSYXVteVE0QTZ0eFdnQVdreDYxWlR2a0daS2x4cXEwN1ZvYWhIQTFPSVJB?= =?utf-8?B?UGlPRlM0cGROaXZtYmxwMnZiVnloMms2UW1VOGdNRDErQ2NMZkJtSTNKanJ1?= =?utf-8?B?QWNCOXdlZ1BxejNJSmdtbkJNRk14ZjJSbVJFelk1eHdoMnhVZ2JKK0tYRkEr?= =?utf-8?B?VkEydTNzZWFXb2JNYmZZVExKaC8yRWVUYVI4bmlkTTZ6ekhPS1RpbTRvc3NM?= =?utf-8?B?cTl3L0M5cGh2dit2NWRxcFc5L3ZyMzlPdlVRS2daM1lZT3B4SDhnU1hDMVcy?= =?utf-8?B?VjZuY3dIUWpaY1ArUEJPdHpPVDdhL244Qm9iQmN3YnJSY2pnTzZtb1A3ZTg2?= =?utf-8?B?czl2bDNjVndkV2VuZjBZWTBQQmNyVHpwYktzOTZ6N0pGUVZ6MytKTDZ6ZDlx?= =?utf-8?B?SXRidCtIemdwalhMVWltSGFtVzhBQ2wzTjJoZTMrRjRnU0dPY2tOZjFJYW93?= =?utf-8?B?ZEtCOGNELzFyN09LZDM5S2V6RHNVeldmVVRMVDJVN2hwUGUwMG5xK0FQUVgy?= =?utf-8?B?MTlhOVVFN0dDcndkd0Rnc1ZVSEdRekFDdURuSXJiemlaWS9ZS2V6Y1oxdFR0?= =?utf-8?B?RDZwci9kSFBZNWtFc3dodnMybTZkcU9LTFMxUVN2cUVTN3c5cWw1eWNYaUFp?= =?utf-8?B?dEZ4M3ZuSTZZajF4UFpZaG1XNFY1L0N3dUJvanRoQmVWSGZhNEhDWW51Mi9w?= =?utf-8?B?clNyd3RUcDRoZ3lUajRoV0xhUUMrMG0zNU1OTkN6K2JKMmIxTXlTTTNQbkZK?= =?utf-8?B?clllSjV5dGN3V09JYjVtWUNCdkRiSm9ZUmtLb1pXcENXUFRsVHFpSlhzRm5X?= =?utf-8?B?eHVXT0tZZVkwalRibW9WaFZOdEVnTUUzbUNOY2NWM0VHQmhReVRIMWJYaWE4?= =?utf-8?B?YUFKblhZNmV1WmJWbHY5N0VRVE9rNFcwK1daUUxJMlhianYxM3VNNTIyVlFi?= =?utf-8?B?Rmd2YlZlYS9zZ2dwb1R0aDl0bmZZWTJjb1ZOT25GMlFhTEdQWlErWVVJWTNG?= =?utf-8?B?cnJieWNoL3RCVjlUbm5PWWc2N2RXU01QNHNXY0ZtUEpiNDkwV3ljd1dsNHg4?= =?utf-8?B?U0U4NERwVEZFc1hhanEvUGNscVZheWhjdDNhNlZHSmppbGkvR3Voc1ArcUVj?= =?utf-8?B?UHNjUW02aDQrdlF6N0RpNk16SGkxZGgvTWR0S0JGVThBMFNieTNsRnA0d2w3?= =?utf-8?B?R2hpcnh1cHFBaHkyRUlJbHpkeC9TWXJIQ215RDZDVUNTVHFRZERTaWlDbWFq?= =?utf-8?B?N2c9PQ==?= X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3000aae5-9c5d-446a-f906-08db8e870c92 X-MS-Exchange-CrossTenant-AuthSource: AM9PR10MB4085.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Jul 2023 09:51:28.8285 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 7CjeFzxlTtuduLLMr1/onCBmvNIB74YjzYduHK9AmqbI6gTCnFPeCHElXjP+kln43iSkN3oGRaMznQ6bIoHc4pf7JdnbpjB36ZMh1u4f+YE= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS2PR10MB6926 X-TUID: LzHVChpgxKs7 On 7/27/23 05:26, MOESSBAUER, Felix (T CED INW-CN) wrote: > On Wed, 2023-07-26 at 10:59 +0200, 'Gylstorff Quirin' via isar-users > wrote: >> >> >> On 7/26/23 08:14, 'Felix Moessbauer' via isar-users wrote: >>> When combining layers, it can happen that the same >>> DISTRO_APT_SOURCES >>> entry is added multiple times. This creates duplicate entries in >>> the >>> bootstrap.list, which is considered invalid by apt. To solve this, >>> we >>> deduplicate the values in the [HOST_]DISTRO_APT_SOURCES variable. >>> In addition, this also avoids adding the same SRC_URI entry twice. >>> >>> Note, that this only solves the trivial case that the duplicate >>> entries >>> come from the same file. However in case duplicate entries are >>> added >>> from multiple files with different names, the :remove syntax can be >>> used to remove these for the affected targets. >>> >>> Signed-off-by: Felix Moessbauer >>> --- >>> meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 13 ++++++-- >>> ----- >>> 1 file changed, 6 insertions(+), 7 deletions(-) >>> >>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>> index b94ae0bd..8af73a9b 100644 >>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc >>> @@ -59,8 +59,8 @@ python () { >>> filename = os.path.relpath(fetcher.localpath(key), >>> topdir) >>> d.appendVar("THIRD_PARTY_APT_KEYFILES", " ${TOPDIR}/%s" % >>> filename) >>> >>> - distro_apt_sources = d.getVar(d.getVar("DISTRO_VARS_PREFIX") + >>> "DISTRO_APT_SOURCES") or "" >>> - for file in distro_apt_sources.split(): >>> + distro_apt_sources = get_aptsources_list(d) >>> + for file in distro_apt_sources: >>> d.appendVar("SRC_URI", " file://%s" % file) >>> >>> distro_apt_preferences = >>> d.getVar(d.getVar("DISTRO_VARS_PREFIX") + "DISTRO_APT_PREFERENCES") >>> or "" >>> @@ -149,10 +149,11 @@ def aggregate_aptsources_list(d, file_list, >>> file_out): >>> >>> def get_aptsources_list(d): >>> import errno >>> + from collections import OrderedDict >>> apt_sources_var = d.getVar("DISTRO_VARS_PREFIX") + >>> "DISTRO_APT_SOURCES" >>> - list = (d.getVar(apt_sources_var) or "").split() >>> + apt_sources_list = >>> list(OrderedDict.fromkeys((d.getVar(apt_sources_var) or >>> "").split())) >> Why not use `set()` if the goal is to remove all duplicates? > > The set() does not preserve the order of the entries. However, the > order might be relevant as it influences the order of the entries in > the apt sources list. With the OrderedDict approach, the order is > preserved. > > Felix Thanks, the patch looks good to me. Quirin