From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6547633542169362432 X-Received: by 2002:a19:6b06:: with SMTP id d6-v6mr1373922lfa.9.1524651063980; Wed, 25 Apr 2018 03:11:03 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 10.46.157.71 with SMTP id y7ls2399468ljj.12.gmail; Wed, 25 Apr 2018 03:11:03 -0700 (PDT) X-Google-Smtp-Source: AIpwx4/qqYecnzugn26d80CJ4QAZXYH5pQTkaxXOiwschLQY7RoKelJFRM5XdlHM85upfgQJUE67 X-Received: by 10.46.155.151 with SMTP id z23mr1692707lji.18.1524651063212; Wed, 25 Apr 2018 03:11:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1524651063; cv=none; d=google.com; s=arc-20160816; b=Lrl9EgrCKA3ulWOTm9MfP3l0cqX2FK3QEDNbpJdVxDDT5yr5DJj4wq9ZEACvdlLU3Y Sjt8Vk3uaV7oJqxDKtSHTHX8JsCqd/HsEtUEawUlG5BJOxoaw0IWeTQcN9PgsgNKaT6m +NHqn6oe89avYsxwSByLvWWAMYDAGR/cd8ZTR7p9CtBJhAh9WBf61SN8KZ6/rzj5zYUq zzSm1KKILuZk1IYHDwJDkxkhL6d4OidNMEg3ff2e6wcSygy+Wph3+hlbVW9djrAbMwr6 OPOyefGEELEFylcl6Pw14HYs+7P5p2AoTaMCNMcg6qnFi8N6w87WSbNVf/RWM+ozJ4At xWaw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:openpgp:from:references:cc:to:subject :arc-authentication-results; bh=gAzIlyDbTkBU0NGysWrQGTj1nzaT5RSe2puAvWLXy9o=; b=wA7kN11Zf2B+6tIW6uehaMmc+tcBC2bUhb1m71ueaQMfiRfIMTCTJej7Ac6t3eG8FO on1xwzcov5Ojy6pZhIXNqjqK5NLWREWGXSAbb7sjoN6ofobdv2UfDYxS1d0ImmTOmEhc z6tJzIqbQKeCvwUa0oi8gibh+bc2AvMvT/C4jfZszjQmasdr2Du9sAV+K2Tn+0ABKBg+ FYAMAfD6U0Fl/hEhIdnAEoHGFqMMabptHyYKihWmsQOZYjBIbkhfwjdb49hpmbwxpCP6 h2czzEedq6unxiXbr24nO2tf9O4v5wg1aIBg1VcpTVtZ14jHqcRRF3vvTGdm7rp/Iikg 1rsQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com Return-Path: Received: from david.siemens.de (david.siemens.de. [192.35.17.14]) by gmr-mx.google.com with ESMTPS id i130-v6si117088lfi.5.2018.04.25.03.11.02 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 25 Apr 2018 03:11:03 -0700 (PDT) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.14 as permitted sender) client-ip=192.35.17.14; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.14 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com Received: from mail3.siemens.de (mail3.siemens.de [139.25.208.14]) by david.siemens.de (8.15.2/8.15.2) with ESMTPS id w3PAB2LN022879 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 25 Apr 2018 12:11:02 +0200 Received: from [139.25.68.37] (md1q0hnc.ad001.siemens.net [139.25.68.37] (may be forged)) by mail3.siemens.de (8.15.2/8.15.2) with ESMTP id w3PAB2Q6027364; Wed, 25 Apr 2018 12:11:02 +0200 Subject: Re: [RFC][PATCH 2/3] classes/dsc: Basic Debian .dsc backend implementation To: "[ext] Henning Schild" , Alexander Smirnov Cc: isar-users@googlegroups.com References: <20180423132146.14743-1-asmirnov@ilbers.de> <20180423132146.14743-3-asmirnov@ilbers.de> <20180423170457.3e25e51a@mmd1pvb1c.ad001.siemens.net> <20180423192731.3af2fa32@mmd1pvb1c.ad001.siemens.net> From: Jan Kiszka Openpgp: preference=signencrypt Message-ID: Date: Wed, 25 Apr 2018 12:11:01 +0200 User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); de; rv:1.8.1.12) Gecko/20080226 SUSE/2.0.0.12-1.1 Thunderbird/2.0.0.12 Mnenhy/0.7.5.666 MIME-Version: 1.0 In-Reply-To: <20180423192731.3af2fa32@mmd1pvb1c.ad001.siemens.net> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: OXd8WUNQ64Jq On 2018-04-23 19:27, [ext] Henning Schild wrote: > Am Mon, 23 Apr 2018 19:11:12 +0300 > schrieb Alexander Smirnov : > >> On 04/23/2018 06:04 PM, Henning Schild wrote: >>> Am Mon, 23 Apr 2018 16:21:45 +0300 >>> schrieb Alexander Smirnov : >>> >>>> This patch introduces class to work with Debian .dsc files. The >>>> main goal is to simplify re-building and customization of original >>>> Debian packages. >>>> >>>> To rebuild package from the upstream the following recipe template >>>> could be used: >>>> >>>> DSC_URI = "http://ftp.de.debian.org/...dsc;md5sum=..." >>>> inherit debian-dsc >>>> >>>> All the remaining information bitbake will derive automatically. >>>> This is only first implementation, so there are still several open >>>> issues like: >>>> - Chose correct upstream package version depending on Debian >>>> distro >>>> - quilt-3.0 source format support >>>> - Cross-recipe dependencies recognition >>>> >>>> Signed-off-by: Alexander Smirnov >>>> --- >>>> meta/classes/debian-dsc.bbclass | 112 >>>> ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 112 >>>> insertions(+) create mode 100644 meta/classes/debian-dsc.bbclass >>>> >>>> diff --git a/meta/classes/debian-dsc.bbclass >>>> b/meta/classes/debian-dsc.bbclass new file mode 100644 >>>> index 0000000..4159e3a >>>> --- /dev/null >>>> +++ b/meta/classes/debian-dsc.bbclass >>>> @@ -0,0 +1,112 @@ >>>> +# Debian .dsc backend >>>> +# >>>> +# This software is a part of ISAR. >>>> +# Copyright (c) ilbers GmbH, 2018 >>>> +# >>>> +# SPDX-License-Identifier: MIT >>>> + >>>> +# List of non-standard variables added to package workspace >>>> +# >>>> +# Set by user: >>>> +# * DSC_URI - uri link to .dsc file in upstream apt >>>> +# >>>> +# Generated automatically: >>>> +# * DEBIAN_URI - uri link to Debian patch in upstream apt >>>> + >>>> +python __anonymous() { >>>> + # Fetch .dsc package file >>>> + dsc_uri = (d.getVar("DSC_URI", True) or "").split() >>>> + if len(dsc_uri) == 0: >>>> + return >>> >>> Can we not use SRC_URI and append more sources as we read .dscs? A >>> copy of the fetching code just reminds me of the broken >>> fetch/unpack in the early Isar days. >>> Because i have got my dscs in tarballs and in git ... and i am >>> behind a proxy. >> >> Here I consider only traditional Debian way, when.dsc file is stored >> directly in apt with .orig tarball and .debian patch: >> >> http://ftp.de.debian.org/debian/pool/main/h/hello/ >> >> So this class intended to work with Debian apts. >> >>> >>>> + try: >>>> + fetcher = bb.fetch2.Fetch(dsc_uri, d) >>>> + fetcher.download() >>>> + except bb.fetch2.BBFetchException as e: >>>> + raise bb.build.FuncFailed(e) >>>> + >>>> + # Open .dsc file from downloads >>>> + dl_dir = d.getVar("DL_DIR", True) >>>> + dsc_file = (dsc_uri[0].split(";")[0]).split("/")[-1] >>>> + filepath = dl_dir + "/" + dsc_file >>>> + >>>> + pv = "" >>>> + src_uri = "" >>>> + >>>> + # Parse .dsc file to get package details >>>> + with open(filepath, "r") as file: >>>> + line = file.readline() >>>> + >>>> + while line: >>>> + # Get package version and export PV >>>> + if line.startswith("Version:") and not pv: >>>> + pv = line.split(": ")[-1].rstrip() >>>> + d.setVar("PV", pv) >>>> + >>>> + # Get package and Debian patch arcives names >>>> + if line.startswith("Files:") and not src_uri: >>>> + line = file.readline() >>>> + src = line.split(" ")[-1].rstrip() >>>> + src_uri = (dsc_uri[0].rsplit("/", 1))[0] + "/" + >>>> src >>>> + src_uri = src_uri + ";md5sum=" + line.split(" >>>> ")[1].rstrip() >>>> + d.setVar("SRC_URI", src_uri) >>>> + >>>> + line = file.readline() >>>> + debian = line.split(" ")[-1].rstrip() >>>> + debian_uri = (dsc_uri[0].rsplit("/", 1))[0] + "/" >>>> + debian >>>> + debian_uri = debian_uri + ";md5sum=" + >>>> line.split(" ")[1].rstrip() >>>> + d.setVar("DEBIAN_URI", debian_uri) >>>> + >>>> + line = file.readline() >>>> + >>>> + file.close() >>>> + >>>> + # Set correct path to unpacked sources >>>> + pv_orig = (src.split("_")[-1]).split(".orig")[0] >>>> + pn = d.getVar("PN", True) >>>> + d.setVar("S", pn + "-" + pv_orig) >>>> +} >>>> + >>>> +python do_fetch_append() { >>>> + debian_uri = (d.getVar("DEBIAN_URI", True) or "").split() >>>> + if len(debian_uri) == 0: >>>> + return >>>> + >>>> + try: >>>> + fetcher = bb.fetch2.Fetch(debian_uri, d) >>>> + fetcher.download() >>>> + except bb.fetch2.BBFetchException as e: >>>> + raise bb.build.FuncFailed(e) >>>> +} >>> >>> Wow another fetcher ... same question as above. >> >> Upstream Debian source package contains 3 parts: >> - .dsc >> - .orig.tar.xz >> - .debian.tar.xz >> >> So: >> - First you need to fetch .dsc to get information about remaining 2 >> artifacts. >> - .orig.tar.xz is fetched by standard fetcher. >> - .debian.tar.xz is fetched here. >> >>> >>>> +python do_unpack_append() { >>>> + import subprocess >>>> + >>>> + s = d.getVar("S", True) >>>> + workdir = d.getVar("WORKDIR", True) >>>> + debian_uri = (d.getVar("DEBIAN_URI", True) or "").split() >>>> + >>>> + try: >>>> + fetcher = bb.fetch2.Fetch(debian_uri, d) >>>> + fetcher.unpack(workdir + "/" + s) >>> >>> fetch^3 this time with unpack >> >> Again: >> - .orig.tar.xz is unpacked by standard fetcher >> - the content of .debian.tar.xz should be unpacked into the folder >> from the .orig.tar.xz archive >> >> So in general, this is an attempt to collect and prepare all the >> artifacts to make it usable with dpkg Isar class. > > Sure, fully understood the moment you mentioned dsc. All i wanted to > find out whether we can use SRC_URI and the one fetcher/unapcker we > have already. Why should it be called DSC_URI only because you download > a .dsc? > The magic that an "unpack" of a .dsc involves downloading even more and > maybe unpacking that ... should go into a central place (do_unpack?) and > SRC_URI should stay SRC_URI. > > i.e.: do_fetch/unpack in base.bbclass get > ... > for uri in src_uri: > if uri ends with .dsc and unpack==true: > add new uris to final_src_uri > else: > add uri to final_src_uri > src_uri = final_src_uri > ... > Ack, we should always try to design from the perspective of our users, and there no reason for them to have separate URI lists. Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux