From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 7260007084413943808 X-Received: by 2002:a2e:7e0e:0:b0:2b9:f13b:6139 with SMTP id z14-20020a2e7e0e000000b002b9f13b6139mr5936387ljc.20.1691011819879; Wed, 02 Aug 2023 14:30:19 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a2e:a7c7:0:b0:2b9:4f44:a5ed with SMTP id x7-20020a2ea7c7000000b002b94f44a5edls566544ljp.2.-pod-prod-08-eu; Wed, 02 Aug 2023 14:30:17 -0700 (PDT) X-Google-Smtp-Source: APBJJlG6xJy3L24BZB1p8vo+el/g4QGwDQceW3XpC8EuJR5S5ojqBb5CKvXOEF+72aoL/SQjyp0A X-Received: by 2002:a2e:9782:0:b0:2b9:ea5f:36b with SMTP id y2-20020a2e9782000000b002b9ea5f036bmr5711665lji.52.1691011817298; Wed, 02 Aug 2023 14:30:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1691011817; cv=none; d=google.com; s=arc-20160816; b=nezcDn5iakD6bfaQxHZ21G6uXyu2sahjq4EAI8pCmPLS4Dein0R4n/NVH+Rs5gIMYx GgZvLdheQGAaBsnUNPFzx3dIwv7V59VtjhLmxGA7KjMO0COuIfAs+qYF7yqzcKV/0v50 8zCyySqQM8Xb+bMpFyJtX8pe7u71M5MXR4Y4ITg9tt/hwIvHfSlAJmBLb6POGfEVjImk 8G7jK7nTxZTYZtjphO4JV3+khlH5Cmv2IaFLBd9qvZFbgjBohIPbnQxLN0CciwOgaNAB ug2NQyOI73Nu0NSr49C2BOkiFsU/QSpXhhdMsuAFTqS5oOxAt259s8Zz3BgsCQZtF/b9 ib7w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:user-agent:content-transfer-encoding:references :in-reply-to:date:cc:to:from:subject:message-id; bh=kabAmFWEwRYQM21oS9qYYwtxzHJUsVrJHUZIzQwjhlY=; fh=U/+DBc8heL6n6pqHUDvO6xirs8DTghBG4ohN43CLLF4=; b=omegFZv1kjlnuj1mIdym0fb+ouaPm4K+Fst+ODY426QT/5BxGBH9mcnnvr3hWa82OH btScq47OPeaEIdnUMetaQdL3k+tYNV+MNaPUhi86pA4m6d4FbiDWOx31ety5PvZZiM/1 XDcAJhBsGIhZRDP1VubkneEBuA2pImSeIZmlpFHg9CUxmNVP/AW8W8BpgBSiJPzb7MgG X7vRxMpMg+4CcQposunvBc2fQ7TdTjlHzEhi1INHxf18Ozk72EsSwLHBGyYWxBvHv0eN WxL4T8G+Q8/ZPoqAnOlOr2cE323WXpHjeclnY5FDujAogBq10MigbVNHxtIzeghP2gGu /f4Q== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id vk9-20020a170907cbc900b009885c0ef8d2si1042356ejc.1.2023.08.02.14.30.17 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Wed, 02 Aug 2023 14:30:17 -0700 (PDT) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from [127.0.0.1] (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 372LUFD4028456 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 2 Aug 2023 23:30:16 +0200 Message-ID: <5f815882948df104237d35f769a4efe86958d75e.camel@ilbers.de> Subject: Re: [PATCH 1/1] deduplicate apt sources.list based on filenames From: Uladzimir Bely To: Felix Moessbauer , isar-users@googlegroups.com Cc: jan.kiszka@siemens.com Date: Thu, 03 Aug 2023 00:30:14 +0300 In-Reply-To: <20230726061457.2297344-1-felix.moessbauer@siemens.com> References: <20230726061457.2297344-1-felix.moessbauer@siemens.com> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.48.4 (by Flathub.org) MIME-Version: 1.0 X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: 5U115hCDIFd7 On Wed, 2023-07-26 at 06:14 +0000, 'Felix Moessbauer' via isar-users wrote: > When combining layers, it can happen that the same DISTRO_APT_SOURCES > entry is added multiple times. This creates duplicate entries in the > bootstrap.list, which is considered invalid by apt. To solve this, we > deduplicate the values in the [HOST_]DISTRO_APT_SOURCES variable. > In addition, this also avoids adding the same SRC_URI entry twice. >=20 > Note, that this only solves the trivial case that the duplicate > entries > come from the same file. However in case duplicate entries are added > from multiple files with different names, the :remove syntax can be > used to remove these for the affected targets. >=20 > Signed-off-by: Felix Moessbauer > --- > =C2=A0meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 13 ++++++----= - > -- > =C2=A01 file changed, 6 insertions(+), 7 deletions(-) >=20 > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > index b94ae0bd..8af73a9b 100644 > --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc > @@ -59,8 +59,8 @@ python () { > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 filename =3D os.path.rel= path(fetcher.localpath(key), topdir) > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 d.appendVar("THIRD_PARTY= _APT_KEYFILES", " ${TOPDIR}/%s" % > filename) > =C2=A0 > -=C2=A0=C2=A0=C2=A0 distro_apt_sources =3D d.getVar(d.getVar("DISTRO_VARS= _PREFIX") + > "DISTRO_APT_SOURCES") or "" > -=C2=A0=C2=A0=C2=A0 for file in distro_apt_sources.split(): > +=C2=A0=C2=A0=C2=A0 distro_apt_sources =3D get_aptsources_list(d) > +=C2=A0=C2=A0=C2=A0 for file in distro_apt_sources: > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 d.appendVar("SRC_URI", "= file://%s"=C2=A0% file) > =C2=A0 > =C2=A0=C2=A0=C2=A0=C2=A0 distro_apt_preferences =3D d.getVar(d.getVar("DI= STRO_VARS_PREFIX") > + "DISTRO_APT_PREFERENCES") or "" > @@ -149,10 +149,11 @@ def aggregate_aptsources_list(d, file_list, > file_out): > =C2=A0 > =C2=A0def get_aptsources_list(d): > =C2=A0=C2=A0=C2=A0=C2=A0 import errno > +=C2=A0=C2=A0=C2=A0 from collections import OrderedDict > =C2=A0=C2=A0=C2=A0=C2=A0 apt_sources_var =3D d.getVar("DISTRO_VARS_PREFIX= ") + > "DISTRO_APT_SOURCES" > -=C2=A0=C2=A0=C2=A0 list =3D (d.getVar(apt_sources_var) or "").split() > +=C2=A0=C2=A0=C2=A0 apt_sources_list =3D > list(OrderedDict.fromkeys((d.getVar(apt_sources_var) or "").split())) > =C2=A0=C2=A0=C2=A0=C2=A0 ret =3D [] > -=C2=A0=C2=A0=C2=A0 for p in list: > +=C2=A0=C2=A0=C2=A0 for p in apt_sources_list: > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 try: > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = f =3D bb.parse.resolve_file(p, d) > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = ret.append(f) > @@ -251,9 +252,7 @@ python do_apt_config_prepare() { > =C2=A0 > =C2=A0=C2=A0=C2=A0=C2=A0 apt_sources_out =3D d.getVar("APTSRCS") > =C2=A0=C2=A0=C2=A0=C2=A0 apt_sources_init_out =3D d.getVar("APTSRCS_INIT"= ) > -=C2=A0=C2=A0=C2=A0 apt_sources_list =3D ( > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 d.getVar(d.getVar("DISTRO_VAR= S_PREFIX") + > "DISTRO_APT_SOURCES") or "" > -=C2=A0=C2=A0=C2=A0 ).split() > +=C2=A0=C2=A0=C2=A0 apt_sources_list =3D get_aptsources_list(d) > =C2=A0 > =C2=A0=C2=A0=C2=A0=C2=A0 aggregate_files(d, apt_sources_list, apt_sources= _init_out) > =C2=A0=C2=A0=C2=A0=C2=A0 aggregate_aptsources_list(d, apt_sources_list, a= pt_sources_out) > --=20 > 2.34.1 >=20 Applied to next, thanks.