From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7260007084413943808 X-Received: by 2002:aa7:d49a:0:b0:51d:e2c4:f94a with SMTP id b26-20020aa7d49a000000b0051de2c4f94amr1079410edr.20.1690361991087; Wed, 26 Jul 2023 01:59:51 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:6402:1a45:b0:522:2cdb:ae83 with SMTP id bf5-20020a0564021a4500b005222cdbae83ls104109edb.1.-pod-prod-00-eu; Wed, 26 Jul 2023 01:59:49 -0700 (PDT) X-Google-Smtp-Source: APBJJlFjgvYu62Zp2xSMtZmbk4dXz3WhQBfjhcoy7VISW99fE+hr96lGOuJd28nomFYOTW3HAii3 X-Received: by 2002:a05:6402:5203:b0:522:582f:91af with SMTP id s3-20020a056402520300b00522582f91afmr2954745edd.9.1690361989432; Wed, 26 Jul 2023 01:59:49 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1690361989; cv=pass; d=google.com; s=arc-20160816; b=FXgmDjSmauNEbhZJQrhP/m/6nDgFonb7LO7seeNWD9IQY2d58VGMJ+yFXMasWNiIkM 2agEBUhvAC6JniBWxrKAh/QsjAJnSMlKT21G7Ngh8scTRQ5gjOJlPt1k0IQNHJK3Sso8 A10akMUriKrxrp3IAcGjmycvBzf6g7G5/uups39kq9feJFBoud6yhk5Zhx+Y3+a1prjM 4QuV9M+QTcMosJ3ZjgHkYRUMao7Jct5m3VFEQrhOvHGX7A7YytgKfmNOEVU6fkYpe+2G 5uPb8kg7r1zBVXCEuXuYfqWrsrexy5EPuH4SRKKRCjGJFqDFcGjnwVgyC7Ryg3LKzdAQ tp/Q== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:content-transfer-encoding:in-reply-to:from:references :to:content-language:subject:user-agent:date:message-id :dkim-signature; bh=rSy8VKWoNQ0RGbQ+NkbTza0eFhCPmWeWlVUgDY+GC8k=; fh=SO2BlidHVJZ3BaksvfmFMblhxrBB44z0hoy/rPjWxoM=; b=fMdraGRCAe7hrzz87sXtEdUu/gAqqNAyg9VOV+pFwdZpg6cOv2H/oSrijlWOgOR9+c QolBul7qBjdzXFsOhH5CfviT8cA5NiLCwuamzgZEQWACXqsFr4JZc8KKgMwcNAJzHULq Cjq+eNkYAo+BNxoDGX1z8N/RflcR/VWWdew6NWJHTDOd4E7RRoc4K38kOHKmL6tGWVdK I0cnJTXizIr9oribqNKU0JFApZ0tL46qLa1esi8kinZNa0wk7LbJ+wxKvVLNf0aE1CWq J7+PohVzNJRbtfOrMj0tbhB1Onohum4/Ej4KqUjANUr+1m638Z3F3WJu/6n2jbzByKtj KKgw== ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=rofPJEgz; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of quirin.gylstorff@siemens.com designates 2a01:111:f400:7d00::618 as permitted sender) smtp.mailfrom=quirin.gylstorff@siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Return-Path: Received: from EUR05-VI1-obe.outbound.protection.outlook.com (mail-vi1eur05on20618.outbound.protection.outlook.com. [2a01:111:f400:7d00::618]) by gmr-mx.google.com with ESMTPS id i23-20020a0564020f1700b005222b2cd106si402489eda.3.2023.07.26.01.59.49 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 26 Jul 2023 01:59:49 -0700 (PDT) Received-SPF: pass (google.com: domain of quirin.gylstorff@siemens.com designates 2a01:111:f400:7d00::618 as permitted sender) client-ip=2a01:111:f400:7d00::618; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=selector2 header.b=rofPJEgz; arc=pass (i=1 spf=pass spfdomain=siemens.com dkim=pass dkdomain=siemens.com dmarc=pass fromdomain=siemens.com); spf=pass (google.com: domain of quirin.gylstorff@siemens.com designates 2a01:111:f400:7d00::618 as permitted sender) smtp.mailfrom=quirin.gylstorff@siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=DeNywq8qO2Y4R59CBkgsXHq7n+nnlxzRqW09o3sprsI20AjKJx7vLtMbXL1pxaoOiDH6jAIGNIm0BsAzER7xBcmrSHAUpaELfpoIzuMxzKt6FrTcfEdD3s+bW3NG4gSDavq8YpNSiMhUMry0XhdPujWt8YR3O0ve+BjJYGCXDmhS5crjI6QJkeXnI9s3ct+EAekMbxRWtc3dJHNPnQ7xI+Ce0+W/5E0hyJXFLPj/TeHAEpoUPyBWW58aIunrwFMBnnd7Cy01YRaXtiTnfu2hbSOkLzt8w7J/4MmA6+a2qEyz+MBaEvBxDrI6kWT4qD++4jSx60qXW9PnaqbBBfgNBg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=rSy8VKWoNQ0RGbQ+NkbTza0eFhCPmWeWlVUgDY+GC8k=; b=MLYSJUOaw8luvJCnsj5ELQLnID0Gc1NgwzbrZadLyAs9h0qWJwEBrLggiGyRfhBxoL6eLzA+FZRkuD9KhLFXsaUkUggRhKqU0/NAniTHG1HbIvJU6cURpM3u5F8pE4MlkMA7BaUB9yCEQjWaPKrHKUrH05hvAZnj0SvaNyIO1o0H5hoIql9JU0p86iC5NRuatmQtW1GNuCxsfS/HgNolywS2hW9YRKbSepfN9TJPUKo7pTNPf8+IN9mnHcgJepNTAq3NQSVhBOq3s+5SMEtNlQF9FPhaM4cTim8jql4V6a29f6UI8Jo8UYz4QL6W6kJZ8rBAbwoO5qsDOWHQ3Mzwxg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=siemens.com; dmarc=pass action=none header.from=siemens.com; dkim=pass header.d=siemens.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=siemens.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rSy8VKWoNQ0RGbQ+NkbTza0eFhCPmWeWlVUgDY+GC8k=; b=rofPJEgzVm0PcsfDbzV7J+CCZDssLBD9kkrFOtYTISFam+ZOGFzu7izw3uveb52xtXyY8MWxA4E2iOyPR/7TbS9sd54cUlAJv3kZnZmhkDOi3NLgm3cwestHgXli8sRt4Hefg+jD0xnWqsxQMSXVQLYfH556mSGGo+TA/SCc93lQTwRoOLqDTp/3XVJvX6FOTgzLDtiyvUzGzQ+KUv9NI1gBUXLbhVsDih0OtDiUDxR9AXBaYn6VZZC3ABmo81HFZRziW8pZabmM4f9PyIQQ8O+LijphAJJxLUDxibn7VFiPzORW2Y+pu/FrNEmM13A09yrLrGypNNjghbJcCT3m5w== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=siemens.com; Received: from AM9PR10MB4085.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:1f9::22) by AM7PR10MB3334.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:10b::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6631.29; Wed, 26 Jul 2023 08:59:47 +0000 Received: from AM9PR10MB4085.EURPRD10.PROD.OUTLOOK.COM ([fe80::90a8:2018:8363:8691]) by AM9PR10MB4085.EURPRD10.PROD.OUTLOOK.COM ([fe80::90a8:2018:8363:8691%7]) with mapi id 15.20.6631.026; Wed, 26 Jul 2023 08:59:47 +0000 Message-ID: <6f839d5b-4316-6523-368d-e1b3e450c35e@siemens.com> Date: Wed, 26 Jul 2023 10:59:45 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 Subject: Re: [PATCH 1/1] deduplicate apt sources.list based on filenames Content-Language: en-US, de-DE To: isar-users@googlegroups.com References: <20230726061457.2297344-1-felix.moessbauer@siemens.com> From: Gylstorff Quirin In-Reply-To: <20230726061457.2297344-1-felix.moessbauer@siemens.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-ClientProxiedBy: FR2P281CA0133.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:9e::17) To AM9PR10MB4085.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:1f9::22) Return-Path: quirin.gylstorff@siemens.com MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AM9PR10MB4085:EE_|AM7PR10MB3334:EE_ X-MS-Office365-Filtering-Correlation-Id: d83714ac-b3d5-4506-70ee-08db8db6a9d6 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: vmdDPutffhXjMhvJOVboY3GvHPXe4BJ4gVP22h9hbhXwHE7LPRs+jUsuZcYcJ3a3XlT0rENsZUBgP1r0xXgkI0JH66r4zfKqPx/ouSLtgbEy+5MojfwSqPqb8MCeLjFkOlNsD6lyqKq1A8TiE0BRFWX4hUgjTylOTqqE693GJ2EfAmrAlX6qEHtG4m0wpq8QJ9437vxjesk/4FXdM6xdJplMfDlH00eemouuYf07sDEtKCR7/oEToP2eyXRneXmkn4JC55J3lyQNyZrl4ccPrkD3wFk9g6uZbs846vvdfHyotYYDDMbxA7os0sYOJ0QAhVqeTllB889QKbDp8dZIAK8m6AdPYDqKRzN00RBmMuw684t6XIeJA+i6WhfL/qoBYMAXtIPpuj21nd6xQOVt0tZSaFK4iBeGYvtndx6ZEnK2L3JzAt2X9XV4TulbRExaahlRcQe9Ir1zBEampMWd1DmqkOsVq9pouAz3vpisG6CkJgFWM0JHRU01vgxMozmEZSgkqklWBPjCvf4dPmyb5/uHkRzh+wuAULoY011SsYwM9kJUbtxNgS/5qUJSg1PRIkOJhjDazfw4BVxcyLX+TScTWYD7ggwRwO5J/e1kD97zZpbTDcIYcDzC/0HRHveKI5BGGBBabW49ZIgimG0gCA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AM9PR10MB4085.EURPRD10.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230028)(4636009)(136003)(396003)(39860400002)(346002)(376002)(366004)(451199021)(83380400001)(31686004)(478600001)(31696002)(5660300002)(8676002)(8936002)(36756003)(41300700001)(2906002)(2616005)(6506007)(82960400001)(66556008)(66946007)(66476007)(38100700002)(86362001)(6916009)(316002)(53546011)(6512007)(6486002)(26005)(186003)(45980500001)(43740500002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?eG9xRk1FazFrMVNuSjMzT2Z4SEdsNjNNWlM3Q25ucHRRWnEwMWRQTlNyL3dT?= =?utf-8?B?RWpqZ2pNTkJIWDRrVzVTNEVIVDVIditTOG55dSthRFRHYlA1QVdQM3o1M0FR?= =?utf-8?B?aGxwOVdaSDdnRktySnErTzlSRE9GS2RBZy9NRWdDM2xHYm1kMEdpbXcrMHA3?= =?utf-8?B?SFpkT2NQUXdTdGNCdjdvK3RBbU5TN2xvVkJ3Njh1WDkxN3FNbVI4RnIxb1Vz?= =?utf-8?B?S2lVay93ZkVpdXF1TmtKbTc0eUlZWkFEcXFKM3Q4L2ZRWW1CZzNDNmFtQVEw?= =?utf-8?B?b3laamZRNlVvOVF6TjFDYzhKelNoYXlZUmkxMzFCVHF2a0ladWVzMGNCL0wy?= =?utf-8?B?THVsbzdFSElnL1BCSDg2TFRSV3pWRUlZRlNQUXRCQVlJc2l2S1ZYREUwVDFp?= =?utf-8?B?emZrcUdadFlvRVFlbEh3MUlYbTgyZ0o4aDMwdFNNeit2SmdWdzdPSzBzZG85?= =?utf-8?B?bzJyWlBvV0Y3Ymljdmx2ZTIzUjM5L3ozd3ZhN1FWS1o0N0toNEpKSEZxNXpH?= =?utf-8?B?eWdrdng5aUFqYjkzcUp4VE9nalFjMjlNMGFiM0JGNTYxaWlQRzFOajd2WE1p?= =?utf-8?B?V0hCbThiSUV1SWxUMURON01mOVFodFB0MjI4YVRWZFEwb0piaXo5VEI3Nlcx?= =?utf-8?B?Wi9xWjRycFpMdkIweXo3ZFBYL21jdWtZOU0zTmRVeHg0U25VYUZMTExnNGw5?= =?utf-8?B?M21oVENSYjRzQm5BS01OZXhrUmtNRmZLcUpocjVKOVRqUmQwb2ErbzJmSnNZ?= =?utf-8?B?ZkcybDgramNVKzNmdG5yZmNwZmI0eWR4WDlmeGlBY1lmWkY5V1k2YzBTYUxJ?= =?utf-8?B?S3ZvTzRkUkE0T0xaYkxQRHEyZnZLVWJ1NUY3bEJPeFBwakZkVmptcldVa29i?= =?utf-8?B?d3hSVXM3emlJVXozTnVuNU9xKzBtcXIxa3NJQXZrdFNMbVI4SXFRK3NiK2Jk?= =?utf-8?B?MllHcG1HYUtUUnF5czBkdXVSYjQ3c1c4NndzOURvR3hqb3E4dmNqL3lBdyt4?= =?utf-8?B?RUYrTWlKbUJRdHJmL0FhWERlT2Q5UHEvcDh4bnpqbklnZWhQQUhZR0x4NytH?= =?utf-8?B?VFJ3QzhSTHNsaVdPWmJwQ1ZVOElJRmJWSk5vclZwcW4xZEEzNVNQa3RFSURv?= =?utf-8?B?VCtwdmNyQTJ6UjdDeE5CaGpLQ0FwNGlNdTBCaGdKT3g2UWMyZjJ4WjlFUmgr?= =?utf-8?B?cWFRelNhS3JZb0puVkpMMFFGOXhzMnVmN0tmSWNyWGpHQ2szUnYxTjRzVE1M?= =?utf-8?B?cWxEaDFUKzhlV2kzalhTVi85UmV2Ui9pVTE0UGNiWm1qeEZXSDVDbCs4c2Jn?= =?utf-8?B?QUVFanFwaXp5Z1ZmZVJyV2w5SWtWU1hiazB5MmczMjRrcFNaaGNZTzhZZ0Na?= =?utf-8?B?YTJET1JmOGduSDRCRGpmUEsyNWpmOFZTbDdyRlZrazRqRzVIWENpZmh6MkRL?= =?utf-8?B?SktTYXZUdVN0UUVnWWs0QStWR0t0TnV2RjFvcDIxMjAwSCt3UEpqVks1OHVY?= =?utf-8?B?VE9GSnhqQ1JyVzRyci9MVTl3YnV4a00zZHc1aDdDbTVmalVZLzd1ZUFmYnVY?= =?utf-8?B?SWgvdkc2Q3lDaVkrUmJyelZtTHBQbTU4VU5nRXVya2pHNGFVR1RQeWNxZXpy?= =?utf-8?B?OXVQV29HQnN6eHRrODQxekZMY1lDem5oc0l0Mm1OcmVUSjlhRjJyMXA0NE96?= =?utf-8?B?bjBRNGdHNWFSclBrTXRSSWlVZWJLM1Bjb2Zsb3FraFVTdmJmQXFyaEY4bGJi?= =?utf-8?B?TVlDaFFsODg3V25GS05hMTZkek5Pc29xOTZRdU1vVGFPdUlNT1lNUXBtTXhx?= =?utf-8?B?bzBVRWJPNXJncDhCaXozYXFZSHFVN3JrVVljaWlFWFl6N2J4a3g3VTRkZzc3?= =?utf-8?B?WWlZaXc2cVd4VmZYQlRWdEViTlFnWWVVSXJPWVRWWUlJSnd2QmdkM0txMzNx?= =?utf-8?B?S0J0WUJGUUxTcHJJWWRubGlkVWM0L2Z2azFWWENlc0ZxbWpGOFlSRnhGL2J2?= =?utf-8?B?ODRrWWNIT0R3dm5RK0xUZkRvTFR1cnhkTGQzWEI3eDFpTHpxTHIxZzdQakFn?= =?utf-8?B?cUZYVFRvK2lQTU9qcWxNeHhBRjI1TzFxemF6Wm8zR2dIMGhadFB1NmpZM0tO?= =?utf-8?B?ZVVQd0h1a21EdWE3aDZIS0RScGYzYnc4K0F2ZGhGSmU1NWRXVkp0RmJhMVZT?= =?utf-8?B?Smc9PQ==?= X-OriginatorOrg: siemens.com X-MS-Exchange-CrossTenant-Network-Message-Id: d83714ac-b3d5-4506-70ee-08db8db6a9d6 X-MS-Exchange-CrossTenant-AuthSource: AM9PR10MB4085.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jul 2023 08:59:47.7492 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 38ae3bcd-9579-4fd4-adda-b42e1495d55a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: XZab2rfv6Hbw8Uk8Mi2ZUDfWFuKwiyFHG+t7CtcM6huKb9nvWJXHbAbKmckauRiJhADS0rw/xXvJ7YbE6ZnOLrh7rk/6gzSo03uUNx223MQ= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM7PR10MB3334 X-TUID: JVzFx2XObOns On 7/26/23 08:14, 'Felix Moessbauer' via isar-users wrote: > When combining layers, it can happen that the same DISTRO_APT_SOURCES > entry is added multiple times. This creates duplicate entries in the > bootstrap.list, which is considered invalid by apt. To solve this, we > deduplicate the values in the [HOST_]DISTRO_APT_SOURCES variable. > In addition, this also avoids adding the same SRC_URI entry twice. > > Note, that this only solves the trivial case that the duplicate entries > come from the same file. However in case duplicate entries are added > from multiple files with different names, the :remove syntax can be > used to remove these for the affected targets. > > Signed-off-by: Felix Moessbauer > --- > meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 13 ++++++------- > 1 file changed, 6 insertions(+), 7 deletions(-) > > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > index b94ae0bd..8af73a9b 100644 > --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > @@ -59,8 +59,8 @@ python () { > filename = os.path.relpath(fetcher.localpath(key), topdir) > d.appendVar("THIRD_PARTY_APT_KEYFILES", " ${TOPDIR}/%s" % filename) > > - distro_apt_sources = d.getVar(d.getVar("DISTRO_VARS_PREFIX") + "DISTRO_APT_SOURCES") or "" > - for file in distro_apt_sources.split(): > + distro_apt_sources = get_aptsources_list(d) > + for file in distro_apt_sources: > d.appendVar("SRC_URI", " file://%s" % file) > > distro_apt_preferences = d.getVar(d.getVar("DISTRO_VARS_PREFIX") + "DISTRO_APT_PREFERENCES") or "" > @@ -149,10 +149,11 @@ def aggregate_aptsources_list(d, file_list, file_out): > > def get_aptsources_list(d): > import errno > + from collections import OrderedDict > apt_sources_var = d.getVar("DISTRO_VARS_PREFIX") + "DISTRO_APT_SOURCES" > - list = (d.getVar(apt_sources_var) or "").split() > + apt_sources_list = list(OrderedDict.fromkeys((d.getVar(apt_sources_var) or "").split())) Why not use `set()` if the goal is to remove all duplicates? Quirin > ret = [] > - for p in list: > + for p in apt_sources_list: > try: > f = bb.parse.resolve_file(p, d) > ret.append(f) > @@ -251,9 +252,7 @@ python do_apt_config_prepare() { > > apt_sources_out = d.getVar("APTSRCS") > apt_sources_init_out = d.getVar("APTSRCS_INIT") > - apt_sources_list = ( > - d.getVar(d.getVar("DISTRO_VARS_PREFIX") + "DISTRO_APT_SOURCES") or "" > - ).split() > + apt_sources_list = get_aptsources_list(d) > > aggregate_files(d, apt_sources_list, apt_sources_init_out) > aggregate_aptsources_list(d, apt_sources_list, apt_sources_out)