From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6547633542169362432 X-Received: by 2002:a19:9603:: with SMTP id y3-v6mr776189lfd.18.1524504453933; Mon, 23 Apr 2018 10:27:33 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 10.46.82.84 with SMTP id g81ls1928414ljb.9.gmail; Mon, 23 Apr 2018 10:27:33 -0700 (PDT) X-Google-Smtp-Source: AB8JxZr/Sxl28m7ARZ38j0sv3Xq+sWoOJPgTUaNBauymnBiVETouDPc8ESX7sf5+B+xoBU2NQDnT X-Received: by 10.46.131.201 with SMTP id s9mr51138ljh.15.1524504453243; Mon, 23 Apr 2018 10:27:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1524504453; cv=none; d=google.com; s=arc-20160816; b=uP/NiVOQwVUQKh69qIt/E8lIQTP2T+k9JzqItGkg2hvWxN0We0u5UZLzdqMNooP1hq Xh2v7tDmdcedKm+fv96qZAJCHODzugzXqodfcewix0EcIaV9C5rHZdJM3adA0OgkwDlW F80SCFYOfFFkheNd59ol6cS12UakSfWDZJreqEaRUOJhXsX7zJAa4dA9q0KMpuKdGonX k0W9DaPvQv6vIn9qSV6Mel4tfmrCes5Nyjm0iLt6HQh00ZanCj69f/V/swekRR0fZ7js rHQEK38nBdMU8TWvLVaPKJUUO0T7gJNYoYYlBekmGbTN5Z1XD8yAIch1RF4zDXrZ1uQ2 YZgQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:arc-authentication-results; bh=D6Zs85AJrE/aDldAqksNnHUr2MNB2pd+O/FfEz2VgWU=; b=x5Zf17JT1Q1EzOj2KoHfgdm8/OuoXhvK2n8zTl27EWmHEYgZKF8rqN0rqdeIfYL6fQ orDgtLewFJ92n4UyECdd5SHw0KhHsO9a8dR+g9OD296S/EbQ9VvyXIsfZvGXO7XW/fP7 PMq5UTJOch0BBPRWyd2TSk7RinSbEAug9XlTK26rhxVf9JXyOqlub3h2WRA8//HLe0mV vrvtV74seyFMI9dBrzOLuPiguY3bqVcicDkJ/qxDRkFqKVUdkZdp0iVF/G9TugyJ7T+m XxWZDPH8BMSq+b4eo3S/rRDqxJRs3FSQORjykVrJbGkJJ3C0R2zCwXdvPBUHIQJ006PY Q5bQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.2 as permitted sender) smtp.mailfrom=henning.schild@siemens.com Return-Path: Received: from thoth.sbs.de (thoth.sbs.de. [192.35.17.2]) by gmr-mx.google.com with ESMTPS id i130-v6si1266lfi.5.2018.04.23.10.27.33 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 23 Apr 2018 10:27:33 -0700 (PDT) Received-SPF: pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.2 as permitted sender) client-ip=192.35.17.2; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of henning.schild@siemens.com designates 192.35.17.2 as permitted sender) smtp.mailfrom=henning.schild@siemens.com Received: from mail2.siemens.de (mail2.siemens.de [139.25.208.11]) by thoth.sbs.de (8.15.2/8.15.2) with ESMTPS id w3NHRWKp003689 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 23 Apr 2018 19:27:32 +0200 Received: from mmd1pvb1c.ad001.siemens.net (md1pvb1c.ad001.siemens.net [139.25.68.40] (may be forged)) by mail2.siemens.de (8.15.2/8.15.2) with ESMTP id w3NHRW8F021351; Mon, 23 Apr 2018 19:27:32 +0200 Date: Mon, 23 Apr 2018 19:27:31 +0200 From: Henning Schild To: Alexander Smirnov Cc: Subject: Re: [RFC][PATCH 2/3] classes/dsc: Basic Debian .dsc backend implementation Message-ID: <20180423192731.3af2fa32@mmd1pvb1c.ad001.siemens.net> In-Reply-To: References: <20180423132146.14743-1-asmirnov@ilbers.de> <20180423132146.14743-3-asmirnov@ilbers.de> <20180423170457.3e25e51a@mmd1pvb1c.ad001.siemens.net> X-Mailer: Claws Mail 3.15.0-dirty (GTK+ 2.24.31; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-TUID: /SjOfpA/K8Fb Am Mon, 23 Apr 2018 19:11:12 +0300 schrieb Alexander Smirnov : > On 04/23/2018 06:04 PM, Henning Schild wrote: > > Am Mon, 23 Apr 2018 16:21:45 +0300 > > schrieb Alexander Smirnov : > > > >> This patch introduces class to work with Debian .dsc files. The > >> main goal is to simplify re-building and customization of original > >> Debian packages. > >> > >> To rebuild package from the upstream the following recipe template > >> could be used: > >> > >> DSC_URI = "http://ftp.de.debian.org/...dsc;md5sum=..." > >> inherit debian-dsc > >> > >> All the remaining information bitbake will derive automatically. > >> This is only first implementation, so there are still several open > >> issues like: > >> - Chose correct upstream package version depending on Debian > >> distro > >> - quilt-3.0 source format support > >> - Cross-recipe dependencies recognition > >> > >> Signed-off-by: Alexander Smirnov > >> --- > >> meta/classes/debian-dsc.bbclass | 112 > >> ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 112 > >> insertions(+) create mode 100644 meta/classes/debian-dsc.bbclass > >> > >> diff --git a/meta/classes/debian-dsc.bbclass > >> b/meta/classes/debian-dsc.bbclass new file mode 100644 > >> index 0000000..4159e3a > >> --- /dev/null > >> +++ b/meta/classes/debian-dsc.bbclass > >> @@ -0,0 +1,112 @@ > >> +# Debian .dsc backend > >> +# > >> +# This software is a part of ISAR. > >> +# Copyright (c) ilbers GmbH, 2018 > >> +# > >> +# SPDX-License-Identifier: MIT > >> + > >> +# List of non-standard variables added to package workspace > >> +# > >> +# Set by user: > >> +# * DSC_URI - uri link to .dsc file in upstream apt > >> +# > >> +# Generated automatically: > >> +# * DEBIAN_URI - uri link to Debian patch in upstream apt > >> + > >> +python __anonymous() { > >> + # Fetch .dsc package file > >> + dsc_uri = (d.getVar("DSC_URI", True) or "").split() > >> + if len(dsc_uri) == 0: > >> + return > > > > Can we not use SRC_URI and append more sources as we read .dscs? A > > copy of the fetching code just reminds me of the broken > > fetch/unpack in the early Isar days. > > Because i have got my dscs in tarballs and in git ... and i am > > behind a proxy. > > Here I consider only traditional Debian way, when.dsc file is stored > directly in apt with .orig tarball and .debian patch: > > http://ftp.de.debian.org/debian/pool/main/h/hello/ > > So this class intended to work with Debian apts. > > > > >> + try: > >> + fetcher = bb.fetch2.Fetch(dsc_uri, d) > >> + fetcher.download() > >> + except bb.fetch2.BBFetchException as e: > >> + raise bb.build.FuncFailed(e) > >> + > >> + # Open .dsc file from downloads > >> + dl_dir = d.getVar("DL_DIR", True) > >> + dsc_file = (dsc_uri[0].split(";")[0]).split("/")[-1] > >> + filepath = dl_dir + "/" + dsc_file > >> + > >> + pv = "" > >> + src_uri = "" > >> + > >> + # Parse .dsc file to get package details > >> + with open(filepath, "r") as file: > >> + line = file.readline() > >> + > >> + while line: > >> + # Get package version and export PV > >> + if line.startswith("Version:") and not pv: > >> + pv = line.split(": ")[-1].rstrip() > >> + d.setVar("PV", pv) > >> + > >> + # Get package and Debian patch arcives names > >> + if line.startswith("Files:") and not src_uri: > >> + line = file.readline() > >> + src = line.split(" ")[-1].rstrip() > >> + src_uri = (dsc_uri[0].rsplit("/", 1))[0] + "/" + > >> src > >> + src_uri = src_uri + ";md5sum=" + line.split(" > >> ")[1].rstrip() > >> + d.setVar("SRC_URI", src_uri) > >> + > >> + line = file.readline() > >> + debian = line.split(" ")[-1].rstrip() > >> + debian_uri = (dsc_uri[0].rsplit("/", 1))[0] + "/" > >> + debian > >> + debian_uri = debian_uri + ";md5sum=" + > >> line.split(" ")[1].rstrip() > >> + d.setVar("DEBIAN_URI", debian_uri) > >> + > >> + line = file.readline() > >> + > >> + file.close() > >> + > >> + # Set correct path to unpacked sources > >> + pv_orig = (src.split("_")[-1]).split(".orig")[0] > >> + pn = d.getVar("PN", True) > >> + d.setVar("S", pn + "-" + pv_orig) > >> +} > >> + > >> +python do_fetch_append() { > >> + debian_uri = (d.getVar("DEBIAN_URI", True) or "").split() > >> + if len(debian_uri) == 0: > >> + return > >> + > >> + try: > >> + fetcher = bb.fetch2.Fetch(debian_uri, d) > >> + fetcher.download() > >> + except bb.fetch2.BBFetchException as e: > >> + raise bb.build.FuncFailed(e) > >> +} > > > > Wow another fetcher ... same question as above. > > Upstream Debian source package contains 3 parts: > - .dsc > - .orig.tar.xz > - .debian.tar.xz > > So: > - First you need to fetch .dsc to get information about remaining 2 > artifacts. > - .orig.tar.xz is fetched by standard fetcher. > - .debian.tar.xz is fetched here. > > > > >> +python do_unpack_append() { > >> + import subprocess > >> + > >> + s = d.getVar("S", True) > >> + workdir = d.getVar("WORKDIR", True) > >> + debian_uri = (d.getVar("DEBIAN_URI", True) or "").split() > >> + > >> + try: > >> + fetcher = bb.fetch2.Fetch(debian_uri, d) > >> + fetcher.unpack(workdir + "/" + s) > > > > fetch^3 this time with unpack > > Again: > - .orig.tar.xz is unpacked by standard fetcher > - the content of .debian.tar.xz should be unpacked into the folder > from the .orig.tar.xz archive > > So in general, this is an attempt to collect and prepare all the > artifacts to make it usable with dpkg Isar class. Sure, fully understood the moment you mentioned dsc. All i wanted to find out whether we can use SRC_URI and the one fetcher/unapcker we have already. Why should it be called DSC_URI only because you download a .dsc? The magic that an "unpack" of a .dsc involves downloading even more and maybe unpacking that ... should go into a central place (do_unpack?) and SRC_URI should stay SRC_URI. i.e.: do_fetch/unpack in base.bbclass get ... for uri in src_uri: if uri ends with .dsc and unpack==true: add new uris to final_src_uri else: add uri to final_src_uri src_uri = final_src_uri ... Henning > Alex > > > > > Henning > > > >> + except bb.fetch2.BBFetchException as e: > >> + raise bb.build.FuncFailed(e) > >> +} > >> + > >> +# NOTE: this is workaround to serve quilt-3.0 package format > >> support. +# To build such package, the original tarball > >> should be placed +# in "../" folder. In general, this should > >> be handled by Isar +# core build classes, but for now this > >> class in the only one user +# for quilt-3.0, so for the > >> first implementation let"s keep it here. +# Open issues: > >> +# - How to identify if package has quilt-3.0 source format? > >> +# - How to avoid copying of tarballs from downloads to working > >> dir? +do_build_prepend() { > >> + dl_dir="${@ d.getVar('DL_DIR', True)}" > >> + workdir="${@ d.getVar('WORKDIR', True)}" > >> + file="${@ ((d.getVar('SRC_URI', > >> True)).split(";")[0]).split("/")[-1]}" + > >> + install -m 644 $dl_dir/$file $workdir > >> +} > >> + > >> +inherit dpkg > >