From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from shymkent.ilbers.de ([unix socket]) by shymkent (Cyrus 2.5.10-Debian-2.5.10-3+deb9u2) with LMTPA; Thu, 28 Nov 2024 07:23:43 +0100 X-Sieve: CMU Sieve 2.4 Received: from mail-ed1-f57.google.com (mail-ed1-f57.google.com [209.85.208.57]) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPS id 4AS6NfWf017711 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 28 Nov 2024 07:23:42 +0100 Received: by mail-ed1-f57.google.com with SMTP id 4fb4d7f45d1cf-5d09962822bsf207715a12.1 for ; Wed, 27 Nov 2024 22:23:42 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1732775016; cv=pass; d=google.com; s=arc-20240605; b=HJVywg8UuxfRtOKGuRAQfXrDzCiuqycnG9Vg9tZngZDRxuYJTR1LISGUsKiT83eVQB b+tDuP1+AeuXzSsxdrTzP3mvOn1uMmEC6TlHMm0P3/Y8a16+rWeLKd5Fa1OxtgdoiSwA 1L2K947ZIpN8ezH+v+iy5k2sf4FsPsGI/UXJyOWKs0YMba79NQr1tibgLyZYqjWKTYsi uzu0/2OlNdVph2rFs5aAPzBhuTBmc+zuTJvNeYtxNxeYeNOf31mcTzWL9x/oZkWyBJVW RRpHl9qJ5eQ5mQf0A5xy3+QyXakO+7jGK/slAYrQpoUnD0stUaZForho6f6zdtO+MFpL 7qlg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:user-agent :content-transfer-encoding:references:in-reply-to:date:to:from :subject:message-id:sender:dkim-signature; bh=ggDvnsqGj1tH1Xxg7P1cVfU1oxCWuFHHm1AWUrGn50U=; fh=DI+0ljwyRiuj4SdN1vJvoFjoEC2qy1CdKk67q2hu+WE=; b=HtyZQ3zojnPvJ/Rf2shPvspfYWoIFXN48kWIbNm2FgbVNglZKAYmSz8PBzFHHgd8zE p3KRh4KzXDlfNmM695dQnBFf+1EfBA15o7zujXgAwRuSnBqXapPrXYozpslmR7qGru35 8WZL1+tqZAvnjY50Xf/NDZ/P5u/RukHDgLZpAgO2obUARki6RwEAcms6697wQxxBElaS aUZ6kvUqaGxhvJIf1fY++Sj6msyuO3zyrvZyH1A92Oog/EJGNKV2rpalGMV9sn1P+uq/ lb1LnDmOaUllviCj85axKp9lrn4+R8ybAaxqXjTqiYqlx3kqH2IsgYReAijrISnW+uyG vb1A==; darn=ilbers.de ARC-Authentication-Results: i=2; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1732775016; x=1733379816; darn=ilbers.de; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:user-agent:content-transfer-encoding :references:in-reply-to:date:to:from:subject:message-id:sender:from :to:cc:subject:date:message-id:reply-to; bh=ggDvnsqGj1tH1Xxg7P1cVfU1oxCWuFHHm1AWUrGn50U=; b=L0OWqZC3QneHxQDpW2K8CH3BszgtOxAxD87/eFOFqoc301WiHmKsm1r85XrxlzQtJd S2J44JA1ESRoqGRKFz43icG10+ZErrykKaBJsdQkwtrcaYfW6OdhqcfREz/xz+42oC5f pRXPgwcDvQ9S1EAPiJ11ciLaRzOvV0vR8oShqaNesuXTQoneTCN0GWf2Htfp+ETTDyZh cl1EdkgF95KceyCNy7RaCSkcExE7Pg/OFhvwYyrrBNhC5T013boY2c+6HO5QXTK5At6k BXOdkAXBDWvTB3flmVvq0uBf2ta14hmnyziWdnxa1Y5m2oJjL6XRdanTx4116cGZM6CC 8VhQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732775016; x=1733379816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :user-agent:content-transfer-encoding:references:in-reply-to:date:to :from:subject:message-id:x-beenthere:x-gm-message-state:sender:from :to:cc:subject:date:message-id:reply-to; bh=ggDvnsqGj1tH1Xxg7P1cVfU1oxCWuFHHm1AWUrGn50U=; b=JXg3phFu7uvgg36w24EiWmVH8jp5Xh2OvUAau/9BixxmKqIey+AYj1gXciWYYPG2il uE1y9PNKdX8hBRxh+msf7xf619Is0ShP4cYJ6wLAucUcCphLjUsjHiknDayYtxL17s5F lo6cNMcUVz/EUaMP3dVa+nkF6tH1WhP3pRYNJ1cBjEykU97T5K7i5wNlv/3nTVpNh17A Pn7HdkadPAWL9pfLxl2kbKPyS0aatDOwxi4B5Ji6JbFxEmqbAMtOwG9DBiT9i5NvPEZy gt30Mj0nsgNV9VRkXZVGL6BHA+vOTV+nx7N6Z+KSocP3knqS20CcnBw6/uJfkCWf6o9Y LLMQ== Sender: isar-users@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCXJeslGCtsP6Ea+hOuxZ46o5WOHg4eTGDWUVRHAgOipoAea4MqIDhy7M8SazvnpNVFZB7gT@ilbers.de X-Gm-Message-State: AOJu0YzY9Hd6k10+jxnTtOmUkKOE09fm/m0Ng9H8zF741E/alh9lnXLm 8gTN13/bL8Pc/3Uhq9XXDlqsgjRu2iJW1qSMtmlx0sV4u47NQho5 X-Google-Smtp-Source: AGHT+IF3s3dQHVXRG0BjHilG07PG3vharl4uz+NqcgeCThkER4KwV8a2DGdx/5ZmcSCztd7koxXTrQ== X-Received: by 2002:a05:6402:5255:b0:5d0:832c:92b1 with SMTP id 4fb4d7f45d1cf-5d0832c9323mr3605978a12.25.1732775015828; Wed, 27 Nov 2024 22:23:35 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a50:f60a:0:b0:5cf:801e:4b3d with SMTP id 4fb4d7f45d1cf-5d09546b054ls259915a12.2.-pod-prod-06-eu; Wed, 27 Nov 2024 22:23:33 -0800 (PST) X-Forwarded-Encrypted: i=2; AJvYcCVFEdwcX58nbUuR9xxIrLXkJrWCH4l9vGDULp7NzWOmt/lvnWNe5/Sb+xesjTenXSpcu4DJnzxNcScK@googlegroups.com X-Received: by 2002:a05:6402:13c9:b0:5d0:833d:f85f with SMTP id 4fb4d7f45d1cf-5d0833dfb1dmr4576266a12.0.1732775013196; Wed, 27 Nov 2024 22:23:33 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1732775013; cv=none; d=google.com; s=arc-20240605; b=MkUU9Pcwhc2a1G4jXDYDj2Z5y9BF4LIEO8srPQaOOAJIVofosOLjFMod3jP7MF+4OV 0D+GZxN52g7kV3BveJ+9kuoEjQLlA5c9kEr8GeH7EraieaD7LVEm6n/Ja8FBGKQH824y mwVRPWRb/F4bHpuNHl8KvtdBHOKKd5/xrGQtvshmlrFQ16ujUT3zL+IMowQ/5borJILD XgO2xMTuvhJm0hNVBWIbkcvnpU8nW6Fj2PPKV4G+tBzpsRt5RXSh+YaEYCHH5zfhsT4U hWlcRJmwlACnwa07UfCQo6SlyEQ9Ek76sXL0C7kmSjuQE9Kp0rwKo0vXPX2WNGNo5uVV Gyng== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=mime-version:user-agent:content-transfer-encoding:references :in-reply-to:date:to:from:subject:message-id; bh=iskHItXJ7L38lFM9JfpRpjpZG5GTpoJsKYwi0ZTwnWI=; fh=IxbeHM4vWxalf1XNf2rnGcQL+r1v0Lr0C6ICsGNNkYw=; b=dmGu6kqlF1vvQhrVmwbTU/SZ9vZTlBkgTxgyk1sRBei/AwDiIkjzRP4y0OUw7MSe+E pp8uyBVGl05DSTZtKMLMUIX40OC31bKhxowdh1quhPIKHYNg6bwBlR6UbuGDyzSdXx+q joMiYMGo8rcWmKDq+HFvWKo5nn/ZtcWL49qeAUjC45ehAcrlEOBUIIL2JTo+SxSKC7JO Np+pKScx04K23Is7Esd4qXSSQ3bC48chXH42jqMWArQKWQrdHOSF7XLwVUNHbvqR/nXq BlC5Jv3pXbFVfbOXH0/4vAVqJcVdGPcG8pD+vmPNLipJpxPK+jygxTli+rGtHYBZu4NZ LLLQ==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id 4fb4d7f45d1cf-5d097d9d4d5si26022a12.1.2024.11.27.22.23.33 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Wed, 27 Nov 2024 22:23:33 -0800 (PST) Received-SPF: pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Received: from [127.0.0.1] (host-80-81-17-52.static.customer.m-online.net [80.81.17.52]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8+deb9u1) with ESMTPSA id 4AS6NUd9017705 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 28 Nov 2024 07:23:32 +0100 Message-ID: <87f0b2714fbbb7f2c691c471e744ba105181344a.camel@ilbers.de> Subject: Re: [PATCH v2] Convert apt source fetcher into native bitbake variant From: Uladzimir Bely To: Jan Kiszka , isar-users Date: Thu, 28 Nov 2024 09:23:31 +0300 In-Reply-To: References: <9d1ebee6fe3ffb4040a1c5ea09c2f19700418d64.camel@ilbers.de> <245eca35-04a6-4b72-b528-15b33d661a40@siemens.com> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.54.1 (by Flathub.org) MIME-Version: 1.0 X-Spam-Status: No, score=-4.6 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2, RCVD_IN_RP_CERTIFIED,RCVD_IN_RP_RNBL,RCVD_IN_RP_SAFE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-Original-Sender: ubely@ilbers.de X-Original-Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of ubely@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=ubely@ilbers.de Precedence: list Mailing-list: list isar-users@googlegroups.com; contact isar-users+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: isar-users@googlegroups.com X-Google-Group-Id: 914930254986 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , X-TUID: YO4prAEqWLOT On Thu, 2024-11-28 at 09:03 +0300, Uladzimir Bely wrote: > On Thu, 2024-11-28 at 12:55 +0800, Jan Kiszka wrote: > > On 27.11.24 22:07, Uladzimir Bely wrote: > > > On Fri, 2024-11-15 at 17:40 +0100, Jan Kiszka wrote: > > > > From: Jan Kiszka > > > >=20 > > > > There is no major functional difference, but we no longer have > > > > to > > > > manipulate SRC_URI by registering an official fetcher for > > > > apt://. > > > >=20 > > > > As the fetching no longer takes place in separate tasks, > > > > do_fetch > > > > and > > > > do_unpack need to gain the extra flags that were so far > > > > assigned > > > > to > > > > apt_fetch and apt_unpack. That happens conditionally, i.e. only > > > > if > > > > SRC_URI actually contains an apt URL. > > > >=20 > > > > One difference to the original version is the possibility - > > > > even > > > > if > > > > practically of minor relevance - to unpack multiple apt sources > > > > into > > > > S. > > > > The old version contained a loop but was directing dpkg-source > > > > to > > > > a > > > > pre-existing dir which would have failed on the second > > > > iteration. > > > > The > > > > new version now folds the results together after each step. > > > >=20 > > > > Another minor difference is that unversioned fetches put their > > > > results > > > > into the same subfolder in DL_DIR, also when specifying a > > > > distro > > > > codename. Only versioned fetches get dedicated folders (and > > > > .done > > > > stamps). > > > >=20 > > > > There is no progress report realized because dpkg-source > > > > unfortunately > > > > does not provide information upfront to make this predictable, > > > > thus > > > > expressible in form of percentage. > > > >=20 > > > > Signed-off-by: Jan Kiszka > > > > --- > > > >=20 > > > > Changes in v2: > > > > =C2=A0- rebased, including the removal of isar-apt sources in > > > > apt_unpack > > > >=20 > > > > I'm carefully optimistic that this change also resolves the > > > > previously=20 > > > > seen issue in CI. > > > >=20 > > > > =C2=A0meta/classes/dpkg-base.bbclass | 104 ++++--------------------= - > > > > -- > > > > ---- > > > > -- > > > > =C2=A0meta/lib/aptsrc_fetcher.py=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0 93 > > > > +++++++++++++++++++++++++++++ > > > > =C2=A02 files changed, 104 insertions(+), 93 deletions(-) > > > > =C2=A0create mode 100644 meta/lib/aptsrc_fetcher.py > > > >=20 > > > > diff --git a/meta/classes/dpkg-base.bbclass > > > > b/meta/classes/dpkg- > > > > base.bbclass > > > > index b4ea8e17..c02c07a8 100644 > > > > --- a/meta/classes/dpkg-base.bbclass > > > > +++ b/meta/classes/dpkg-base.bbclass > > > > @@ -79,110 +79,28 @@ do_adjust_git[lockfiles] +=3D > > > > "${DL_DIR}/git/isar.lock" > > > > =C2=A0inherit patch > > > > =C2=A0addtask patch after do_adjust_git > > > > =C2=A0 > > > > -SRC_APT ?=3D "" > > > > - > > > > -# filter out all "apt://" URIs out of SRC_URI and stick them > > > > into > > > > SRC_APT > > > > =C2=A0python() { > > > > -=C2=A0=C2=A0=C2=A0 src_uri =3D (d.getVar('SRC_URI', False) or "").= split() > > > > +=C2=A0=C2=A0=C2=A0 from bb.fetch2 import methods > > > > =C2=A0 > > > > -=C2=A0=C2=A0=C2=A0 prefix =3D "apt://" > > > > -=C2=A0=C2=A0=C2=A0 src_apt =3D [] > > > > -=C2=A0=C2=A0=C2=A0 for u in src_uri: > > > > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 if u.startswith(prefix)= : > > > > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= src_apt.append(u[len(prefix) :]) > > > > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= d.setVar('SRC_URI:remove', u) > > > > +=C2=A0=C2=A0=C2=A0 # apt-src fetcher > > > > +=C2=A0=C2=A0=C2=A0 import aptsrc_fetcher > > > > +=C2=A0=C2=A0=C2=A0 methods.append(aptsrc_fetcher.AptSrc()) > > > > =C2=A0 > > > > -=C2=A0=C2=A0=C2=A0 d.prependVar('SRC_APT', ' '.join(src_apt)) > > > > +=C2=A0=C2=A0=C2=A0 src_uri =3D (d.getVar('SRC_URI', False) or "").= split() > > > > +=C2=A0=C2=A0=C2=A0 for u in src_uri: > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 if u.startswith("apt://= "): > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= d.appendVarFlag('do_fetch', 'depends', > > > > d.getVar('SCHROOT_DEP')) > > > > =C2=A0 > > > > -=C2=A0=C2=A0=C2=A0 if len(d.getVar('SRC_APT').strip()) > 0: > > > > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 bb.build.addtask('apt_u= npack', 'do_patch', '', d) > > > > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 bb.build.addtask('clean= all_apt', 'do_cleanall', '', d) > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= d.appendVarFlag('do_unpack', 'cleandirs', > > > > d.getVar('S')) > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= d.setVarFlag('do_unpack', 'network', > > > > d.getVar('TASK_USE_SUDO')) > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= break > > > > =C2=A0 > > > > =C2=A0=C2=A0=C2=A0=C2=A0 # container docker fetcher > > > > =C2=A0=C2=A0=C2=A0=C2=A0 import container_fetcher > > > > -=C2=A0=C2=A0=C2=A0 from bb.fetch2 import methods > > > > =C2=A0 > > > > =C2=A0=C2=A0=C2=A0=C2=A0 methods.append(container_fetcher.Container= ()) > > > > =C2=A0} > > > > =C2=A0 > > > > -do_apt_fetch() { > > > > -=C2=A0=C2=A0=C2=A0 E=3D"${@ isar_export_proxies(d)}" > > > > -=C2=A0=C2=A0=C2=A0 schroot_create_configs > > > > - > > > > -=C2=A0=C2=A0=C2=A0 session_id=3D$(schroot -q -b -c ${SBUILD_CHROOT= }) > > > > -=C2=A0=C2=A0=C2=A0 echo "Started session: ${session_id}" > > > > - > > > > -=C2=A0=C2=A0=C2=A0 schroot_cleanup() { > > > > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 schroot -q -f -e -c ${s= ession_id} > /dev/null 2>&1 > > > > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 schroot_delete_configs > > > > -=C2=A0=C2=A0=C2=A0 } > > > > -=C2=A0=C2=A0=C2=A0 trap 'exit 1' INT HUP QUIT TERM ALRM USR1 > > > > -=C2=A0=C2=A0=C2=A0 trap 'schroot_cleanup' EXIT > > > > - > > > > -=C2=A0=C2=A0=C2=A0 schroot -r -c ${session_id} -d / -u root -- \ > > > > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 rm /etc/apt/sources.lis= t.d/isar-apt.list > > > > /etc/apt/preferences.d/isar-apt > > > > -=C2=A0=C2=A0=C2=A0 schroot -r -c ${session_id} -d / -- \ > > > > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 sh -c ' > > > > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= set -e > > > > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= for uri in $2; do > > > > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 mkdir -p /downloads/deb-src/"$1"/${uri} > > > > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 cd /downloads/deb-src/"$1"/${uri} > > > > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 apt-get -y --download-only --only-source > > > > source > > > > ${uri} > > > > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= done' \ > > > > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 my_script "${BASE_DISTRO}- > > > > ${BASE_DISTRO_CODENAME}" > > > > "${SRC_APT}" > > > > - > > > > -=C2=A0=C2=A0=C2=A0 schroot -e -c ${session_id} > > > > -=C2=A0=C2=A0=C2=A0 schroot_delete_configs > > > > -} > > > > - > > > > -addtask apt_fetch > > > > -do_apt_fetch[lockfiles] +=3D "${REPO_ISAR_DIR}/isar.lock" > > > > -do_apt_fetch[network] =3D "${TASK_USE_NETWORK_AND_SUDO}" > > > > - > > > > -# Add dependency from the correct schroot: host or target > > > > -do_apt_fetch[depends] +=3D "${SCHROOT_DEP}" > > > > - > > > > -do_apt_unpack() { > > > > -=C2=A0=C2=A0=C2=A0 rm -rf ${S} > > > > -=C2=A0=C2=A0=C2=A0 schroot_create_configs > > > > - > > > > -=C2=A0=C2=A0=C2=A0 session_id=3D$(schroot -q -b -c ${SBUILD_CHROOT= }) > > > > -=C2=A0=C2=A0=C2=A0 echo "Started session: ${session_id}" > > > > - > > > > -=C2=A0=C2=A0=C2=A0 schroot_cleanup() { > > > > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 schroot -q -f -e -c ${s= ession_id} > /dev/null 2>&1 > > > > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 schroot_delete_configs > > > > -=C2=A0=C2=A0=C2=A0 } > > > > -=C2=A0=C2=A0=C2=A0 trap 'exit 1' INT HUP QUIT TERM ALRM USR1 > > > > -=C2=A0=C2=A0=C2=A0 trap 'schroot_cleanup' EXIT > > > > - > > > > -=C2=A0=C2=A0=C2=A0 schroot -r -c ${session_id} -d / -u root -- \ > > > > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 rm /etc/apt/sources.lis= t.d/isar-apt.list > > > > /etc/apt/preferences.d/isar-apt > > > > -=C2=A0=C2=A0=C2=A0 schroot -r -c ${session_id} -d / -- \ > > > > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 sh -c ' > > > > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= set -e > > > > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= for uri in $2; do > > > > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 dscfile=3D"$(apt-get -y -qq --print-uris --only- > > > > source > > > > source $uri | cut -d " " -f2 | grep -E "*.dsc")" > > > > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 cd ${PP} > > > > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 cp /downloads/deb-src/"${1}"/${uri}/* ${PP} > > > > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 dpkg-source -x "${dscfile}" "${PPS}" > > > > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= done' \ > > > > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 my_script "${BASE_DISTRO}- > > > > ${BASE_DISTRO_CODENAME}" > > > > "${SRC_APT}" > > > > - > > > > -=C2=A0=C2=A0=C2=A0 schroot -e -c ${session_id} > > > > -=C2=A0=C2=A0=C2=A0 schroot_delete_configs > > > > -} > > > > -do_apt_unpack[network] =3D "${TASK_USE_SUDO}" > > > > - > > > > -addtask apt_unpack after do_apt_fetch > > > > - > > > > -do_cleanall_apt[nostamp] =3D "1" > > > > -do_cleanall_apt() { > > > > -=C2=A0=C2=A0=C2=A0 for uri in "${SRC_APT}"; do > > > > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 rm -rf "${DEBSRCDIR}/${= BASE_DISTRO}- > > > > ${BASE_DISTRO_CODENAME}/$uri" > > > > -=C2=A0=C2=A0=C2=A0 done > > > > -} > > > > - > > > > =C2=A0def get_package_srcdir(d): > > > > =C2=A0=C2=A0=C2=A0=C2=A0 s =3D os.path.abspath(d.getVar("S")) > > > > =C2=A0=C2=A0=C2=A0=C2=A0 workdir =3D os.path.abspath(d.getVar("WORK= DIR")) > > > > diff --git a/meta/lib/aptsrc_fetcher.py > > > > b/meta/lib/aptsrc_fetcher.py > > > > new file mode 100644 > > > > index 00000000..ee726202 > > > > --- /dev/null > > > > +++ b/meta/lib/aptsrc_fetcher.py > > > > @@ -0,0 +1,93 @@ > > > > +# This software is a part of ISAR. > > > > +# Copyright (c) Siemens AG, 2024 > > > > +# > > > > +# SPDX-License-Identifier: MIT > > > > + > > > > +from bb.fetch2 import FetchError > > > > +from bb.fetch2 import FetchMethod > > > > +from bb.fetch2 import logger > > > > +from bb.fetch2 import runfetchcmd > > > > + > > > > +class AptSrc(FetchMethod): > > > > +=C2=A0=C2=A0=C2=A0 def supports(self, ud, d): > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 return ud.type in ['apt= '] > > > > + > > > > +=C2=A0=C2=A0=C2=A0 def urldata_init(self, ud, d): > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ud.src_package =3D ud.u= rl[len('apt://'):] > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ud.host =3D ud.host.rep= lace('=3D', '_') > > > > + > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 base_distro =3D d.getVa= r('BASE_DISTRO') > > > > + > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 # For these distros we = know that the same version > > > > means > > > > the > > > > same > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 # source package, also = across distro releases. > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 distro_suffix =3D '' if= base_distro in ['debian', > > > > 'ubuntu'] > > > > else \ > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= '-' + d.getVar('BASE_DISTRO_CODENAME') I think, to avoid the issue I mentioned, we should continue using ${BASE_DISTRO}-${BASE_DISTRO_CODENAME} here without exceptions. Also, cache_deb_src() function in rootfs.bbclass still uses this location. > > > > + > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ud.localfile=3D'deb-src= /' + base_distro + distro_suffix > > > > + > > > > '/' > > > > + ud.host > > > > + > > > > +=C2=A0=C2=A0=C2=A0 def download(self, ud, d): > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 bb.utils.exec_flat_pyth= on_func('isar_export_proxies', > > > > d) > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 bb.build.exec_func('sch= root_create_configs', d) > > > > + > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 sbuild_chroot =3D d.get= Var('SBUILD_CHROOT') > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 session_id =3D runfetch= cmd(f'schroot -q -b -c > > > > {sbuild_chroot}', d).strip() > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 logger.info(f'Started s= ession: {session_id}') > > > > + > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 repo_isar_dir =3D d.get= Var('REPO_ISAR_DIR') > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 lockfile =3D > > > > bb.utils.lockfile(f'{repo_isar_dir}/isar.lock') > > > > + > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 try: > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= runfetchcmd(f''' > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 set -e > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 schroot -r -c {session_id} -d / -u root -- \ > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 rm /etc/apt/sources.list.d= /isar-apt.list > > > > /etc/apt/preferences.d/isar-apt > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 schroot -r -c {session_id} -d / -- \ > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 sh -c ' > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 se= t -e > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 mk= dir -p /downloads/{ud.localfile} > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 cd= /downloads/{ud.localfile} > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ap= t-get -y --download-only --only- > > > > source > > > > source {ud.src_package} > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ' > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 ''', d) > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 except (OSError, FetchE= rror): > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= raise > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 finally: > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= bb.utils.unlockfile(lockfile) > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= runfetchcmd(f'schroot -q -f -e -c {session_id}', > > > > d) > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= bb.build.exec_func('schroot_delete_configs', d) > > > > + > > > > +=C2=A0=C2=A0=C2=A0 def unpack(self, ud, rootdir, d): > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 bb.build.exec_func('sch= root_create_configs', d) > > > > + > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 sbuild_chroot =3D d.get= Var('SBUILD_CHROOT') > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 session_id =3D runfetch= cmd(f'schroot -q -b -c > > > > {sbuild_chroot}', d).strip() > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 logger.info(f'Started s= ession: {session_id}') > > > > + > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 pp =3D d.getVar('PP') > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 pps =3D d.getVar('PPS') > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 try: > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= runfetchcmd(f''' > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 set -e > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 schroot -r -c {session_id} -d / -u root -- \ > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 rm /etc/apt/sources.list.d= /isar-apt.list > > > > /etc/apt/preferences.d/isar-apt > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 schroot -r -c {session_id} -d / -- \ > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 sh -c ' > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 se= t -e > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ds= cfile=3D$(apt-get -y -qq --print-uris > > > > -- > > > > only-source source {ud.src_package} | \ > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 cut -d " " -f2 | = grep -E > > > > "\.dsc") > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 cp= /downloads/{ud.localfile}/* {pp} > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 cd= {pp} > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 mv= -f {pps} {pps}.prev > > > > +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 dp= kg-source -x "$dscfile" {pps} > > >=20 > > > Hello. > > >=20 > > > This still fails in CI, but this time I had some time to find the > > > root > > > cause. > > >=20 > > > The problem is that buster(bullseye) and bookworm(trixie) provide > > > different versions of "hello" package. > > >=20 > > > If we first build e.g. `mc:qemuamd64-bookworm:hello`, hello_2.10- > > > 3.dsc > > > is downloaded and the whole "downloads/deb-src/debian/hello/" is > > > considered finished with "downloads/deb-src/debian/hello.done" > > > flag.=C2=A0 > > >=20 > > > So, when e.g. `mc:qemuamd64-bullseye:hello` build follows, it > > > doesn't > > > download hello_2.10-2.dsc an results in dpkg-source error. > > >=20 > > > It doesn't matter if we build both targets in parallel or > > > sequentially, > > > the latest always fails. > > >=20 > >=20 > > Thanks for the analysis. I'll check if I can reproduce und > > understand > > to > > root cause. > >=20 > > Jan > >=20 >=20 > The easy way to reproduce: >=20 > ./kas/kas-container menu # select e.g. qemuamd64-bookworm, save & > exit > ./kas/kas-container shell -c 'bitbake hello' > ./kas/kas-container menu # select e.g. qemuamd64-bullseye, save & > exit > ./kas/kas-container shell -c 'bitbake hello' >=20 > --=20 > Best regards, > Uladzimir. >=20 --=20 Best regards, Uladzimir. --=20 You received this message because you are subscribed to the Google Groups "= isar-users" group. To unsubscribe from this group and stop receiving emails from it, send an e= mail to isar-users+unsubscribe@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/isar-users/= 87f0b2714fbbb7f2c691c471e744ba105181344a.camel%40ilbers.de.