From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7260007084413943808 X-Received: by 2002:a1c:4b19:0:b0:3fc:8ac:b0e with SMTP id y25-20020a1c4b19000000b003fc08ac0b0emr639778wma.32.1690352122672; Tue, 25 Jul 2023 23:15:22 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a05:600c:5113:b0:3fa:8731:d752 with SMTP id o19-20020a05600c511300b003fa8731d752ls575425wms.0.-pod-prod-08-eu; Tue, 25 Jul 2023 23:15:20 -0700 (PDT) X-Google-Smtp-Source: APBJJlEcFM2V/Odp1N4Rhl0GK5WBVPYQV1E2IaL8bnd+7OKmYpuQqo3MCTb6ZJiJZmY+ZWMOEAB0 X-Received: by 2002:adf:fecf:0:b0:317:643a:4fa7 with SMTP id q15-20020adffecf000000b00317643a4fa7mr668190wrs.26.1690352120704; Tue, 25 Jul 2023 23:15:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690352120; cv=none; d=google.com; s=arc-20160816; b=lkDLdq85ueFh49NBTSgVvMnc9uGcOeB+vUIEbsDVKhQk6JKxV2mWgqP9GR38Wh6GPS tEvoVmFlenosi9HU7O4sjz/mbEFFqQEldXad5CzggsRnWCoWVsJ2ZjBO7y4Ncu7o5xjz ZNLX8lNjn5jZ3XHJp3HHY8g4d2D96Xp1QssRkwJbFwcgZeiJ2eVBGXE/H59c7I1IlTp4 1pJLENhGJwc2eE+oTQBvVDDIuh8VQdwISfL7O7ILL6WOdloNi4DhIDD9YI2sOMqbSMbQ flkiPi86NuuhVXzAUyKGgL3kaJoH57tCsJiG0Q3BO6hCMElf/oyDsvl0QF8yhG8SslKk aiZg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-transfer-encoding:mime-version:message-id:date :subject:cc:to:from:dkim-signature; bh=8QtjhtISr9qSbtTl060N51267L84nMVhcVL9qFNeZwo=; fh=pJ48im6kvLu3Nw+SYlsSQKXN/ec/H+0X8C6odOd+3MU=; b=RUNO89AkWmNgVUd5UVewcaxgJtpP2vXzqDa5BmREOzJF5/uF4d09oWgoqsNd81Y1R7 T7gNPt2kyqTVrJ+G9FoqzxPRXeySB+7OQiIdT1qzlByI/SbGeDQVCKZBjjfH3qX17PzO ef8mnsLniJJLHUAEqyKdbG0kj45bpn2nHsqy6UQ7M04kMXO0CdeIbZQgWMlj0YsHucoU Wfs4sfWcIaQwBH8/Qu7k6Bzsq/NrQEUQnuV1ado/XeBJGT2nD1AWyd4BTC3qhC1Tax+K bWXCkLXBVTjZ//Aq+n+XvPcq2Da+ZEd/vK6M3rzxW8z2Wg3GsNsLg7W7z2l7oAR/s5nd VeIw== ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=ePzJ82YB; spf=pass (google.com: domain of fm-72506-202307260615204c8e238e95951742a6-lnyxeb@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) smtp.mailfrom=fm-72506-202307260615204c8e238e95951742a6-LnyxeB@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Return-Path: Received: from mta-65-226.siemens.flowmailer.net (mta-65-226.siemens.flowmailer.net. [185.136.65.226]) by gmr-mx.google.com with ESMTPS id bn15-20020a056000060f00b00317685aa000si253559wrb.4.2023.07.25.23.15.20 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 25 Jul 2023 23:15:20 -0700 (PDT) Received-SPF: pass (google.com: domain of fm-72506-202307260615204c8e238e95951742a6-lnyxeb@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) client-ip=185.136.65.226; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@siemens.com header.s=fm1 header.b=ePzJ82YB; spf=pass (google.com: domain of fm-72506-202307260615204c8e238e95951742a6-lnyxeb@rts-flowmailer.siemens.com designates 185.136.65.226 as permitted sender) smtp.mailfrom=fm-72506-202307260615204c8e238e95951742a6-LnyxeB@rts-flowmailer.siemens.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=siemens.com Received: by mta-65-226.siemens.flowmailer.net with ESMTPSA id 202307260615204c8e238e95951742a6 for ; Wed, 26 Jul 2023 08:15:20 +0200 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; s=fm1; d=siemens.com; i=felix.moessbauer@siemens.com; h=Date:From:Subject:To:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding:Cc; bh=8QtjhtISr9qSbtTl060N51267L84nMVhcVL9qFNeZwo=; b=ePzJ82YBmTEH+HeCeArrxgAIqFGT7LA4nDA0a2haeJyEjWXthwB8JoqaEeyo7YXKcqEmUT D1seP814XM++Mclg5vHD1adIDmbHEn1Rd5gqFsMn4iW2TsJYRNWqikiLlJd6WMX4e/4T6BkO KpprLcH+70+9HCZIrlmpxdnwAiqE4=; From: Felix Moessbauer To: isar-users@googlegroups.com Cc: jan.kiszka@siemens.com, Felix Moessbauer Subject: [PATCH 1/1] deduplicate apt sources.list based on filenames Date: Wed, 26 Jul 2023 06:14:57 +0000 Message-Id: <20230726061457.2297344-1-felix.moessbauer@siemens.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Flowmailer-Platform: Siemens Feedback-ID: 519:519-72506:519-21489:flowmailer X-TUID: MCvLhaLIo3Fb When combining layers, it can happen that the same DISTRO_APT_SOURCES entry is added multiple times. This creates duplicate entries in the bootstrap.list, which is considered invalid by apt. To solve this, we deduplicate the values in the [HOST_]DISTRO_APT_SOURCES variable. In addition, this also avoids adding the same SRC_URI entry twice. Note, that this only solves the trivial case that the duplicate entries come from the same file. However in case duplicate entries are added from multiple files with different names, the :remove syntax can be used to remove these for the affected targets. Signed-off-by: Felix Moessbauer --- meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index b94ae0bd..8af73a9b 100644 --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -59,8 +59,8 @@ python () { filename = os.path.relpath(fetcher.localpath(key), topdir) d.appendVar("THIRD_PARTY_APT_KEYFILES", " ${TOPDIR}/%s" % filename) - distro_apt_sources = d.getVar(d.getVar("DISTRO_VARS_PREFIX") + "DISTRO_APT_SOURCES") or "" - for file in distro_apt_sources.split(): + distro_apt_sources = get_aptsources_list(d) + for file in distro_apt_sources: d.appendVar("SRC_URI", " file://%s" % file) distro_apt_preferences = d.getVar(d.getVar("DISTRO_VARS_PREFIX") + "DISTRO_APT_PREFERENCES") or "" @@ -149,10 +149,11 @@ def aggregate_aptsources_list(d, file_list, file_out): def get_aptsources_list(d): import errno + from collections import OrderedDict apt_sources_var = d.getVar("DISTRO_VARS_PREFIX") + "DISTRO_APT_SOURCES" - list = (d.getVar(apt_sources_var) or "").split() + apt_sources_list = list(OrderedDict.fromkeys((d.getVar(apt_sources_var) or "").split())) ret = [] - for p in list: + for p in apt_sources_list: try: f = bb.parse.resolve_file(p, d) ret.append(f) @@ -251,9 +252,7 @@ python do_apt_config_prepare() { apt_sources_out = d.getVar("APTSRCS") apt_sources_init_out = d.getVar("APTSRCS_INIT") - apt_sources_list = ( - d.getVar(d.getVar("DISTRO_VARS_PREFIX") + "DISTRO_APT_SOURCES") or "" - ).split() + apt_sources_list = get_aptsources_list(d) aggregate_files(d, apt_sources_list, apt_sources_init_out) aggregate_aptsources_list(d, apt_sources_list, apt_sources_out) -- 2.34.1