Re: [RFC][PATCH 2/3] classes/dsc: Basic Debian .dsc backend implementation

[Alexander Smirnov <asmirnov@ilbers.de>]

On 04/23/2018 06:04 PM, Henning Schild wrote:
> Am Mon, 23 Apr 2018 16:21:45 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> This patch introduces class to work with Debian .dsc files. The
>> main goal is to simplify re-building and customization of original
>> Debian packages.
>>
>> To rebuild package from the upstream the following recipe template
>> could be used:
>>
>>    DSC_URI = "http://ftp.de.debian.org/...dsc;md5sum=..."
>>    inherit debian-dsc
>>
>> All the remaining information bitbake will derive automatically.
>> This is only first implementation, so there are still several open
>> issues like:
>>   - Chose correct upstream package version depending on Debian distro
>>   - quilt-3.0 source format support
>>   - Cross-recipe dependencies recognition
>>
>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>> ---
>>   meta/classes/debian-dsc.bbclass | 112
>> ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 112
>> insertions(+) create mode 100644 meta/classes/debian-dsc.bbclass
>>
>> diff --git a/meta/classes/debian-dsc.bbclass
>> b/meta/classes/debian-dsc.bbclass new file mode 100644
>> index 0000000..4159e3a
>> --- /dev/null
>> +++ b/meta/classes/debian-dsc.bbclass
>> @@ -0,0 +1,112 @@
>> +# Debian .dsc backend
>> +#
>> +# This software is a part of ISAR.
>> +# Copyright (c) ilbers GmbH, 2018
>> +#
>> +# SPDX-License-Identifier: MIT
>> +
>> +# List of non-standard variables added to package workspace
>> +#
>> +# Set by user:
>> +#  * DSC_URI - uri link to .dsc file in upstream apt
>> +#
>> +# Generated automatically:
>> +#  * DEBIAN_URI - uri link to Debian patch in upstream apt
>> +
>> +python __anonymous() {
>> +    # Fetch .dsc package file
>> +    dsc_uri = (d.getVar("DSC_URI", True) or "").split()
>> +    if len(dsc_uri) == 0:
>> +        return
> 
> Can we not use SRC_URI and append more sources as we read .dscs? A
> copy of the fetching code just reminds me of the broken fetch/unpack in
> the early Isar days.
> Because i have got my dscs in tarballs and in git ... and i am behind a
> proxy.

Here I consider only traditional Debian way, when.dsc file is stored 
directly in apt with .orig tarball and .debian patch:

http://ftp.de.debian.org/debian/pool/main/h/hello/

So this class intended to work with Debian apts.

> 
>> +    try:
>> +        fetcher = bb.fetch2.Fetch(dsc_uri, d)
>> +        fetcher.download()
>> +    except bb.fetch2.BBFetchException as e:
>> +        raise bb.build.FuncFailed(e)
>> +
>> +    # Open .dsc file from downloads
>> +    dl_dir = d.getVar("DL_DIR", True)
>> +    dsc_file = (dsc_uri[0].split(";")[0]).split("/")[-1]
>> +    filepath = dl_dir + "/" + dsc_file
>> +
>> +    pv = ""
>> +    src_uri = ""
>> +
>> +    # Parse .dsc file to get package details
>> +    with open(filepath, "r") as file:
>> +        line = file.readline()
>> +
>> +        while line:
>> +            # Get package version and export PV
>> +            if line.startswith("Version:") and not pv:
>> +                pv = line.split(": ")[-1].rstrip()
>> +                d.setVar("PV", pv)
>> +
>> +            # Get package and Debian patch arcives names
>> +            if line.startswith("Files:") and not src_uri:
>> +                line = file.readline()
>> +                src = line.split(" ")[-1].rstrip()
>> +                src_uri = (dsc_uri[0].rsplit("/", 1))[0] + "/" + src
>> +                src_uri = src_uri + ";md5sum=" + line.split("
>> ")[1].rstrip()
>> +                d.setVar("SRC_URI", src_uri)
>> +
>> +                line = file.readline()
>> +                debian = line.split(" ")[-1].rstrip()
>> +                debian_uri = (dsc_uri[0].rsplit("/", 1))[0] + "/" +
>> debian
>> +                debian_uri = debian_uri + ";md5sum=" + line.split("
>> ")[1].rstrip()
>> +                d.setVar("DEBIAN_URI", debian_uri)
>> +
>> +            line = file.readline()
>> +
>> +        file.close()
>> +
>> +    # Set correct path to unpacked sources
>> +    pv_orig = (src.split("_")[-1]).split(".orig")[0]
>> +    pn = d.getVar("PN", True)
>> +    d.setVar("S", pn + "-" + pv_orig)
>> +}
>> +
>> +python do_fetch_append() {
>> +    debian_uri = (d.getVar("DEBIAN_URI", True) or "").split()
>> +    if len(debian_uri) == 0:
>> +        return
>> +
>> +    try:
>> +        fetcher = bb.fetch2.Fetch(debian_uri, d)
>> +        fetcher.download()
>> +    except bb.fetch2.BBFetchException as e:
>> +        raise bb.build.FuncFailed(e)
>> +}
> 
> Wow another fetcher ... same question as above.

Upstream Debian source package contains 3 parts:
  - .dsc
  - .orig.tar.xz
  - .debian.tar.xz

So:
  - First you need to fetch .dsc to get information about remaining 2 
artifacts.
  - .orig.tar.xz is fetched by standard fetcher.
  - .debian.tar.xz is fetched here.

> 
>> +python do_unpack_append() {
>> +    import subprocess
>> +
>> +    s = d.getVar("S", True)
>> +    workdir = d.getVar("WORKDIR", True)
>> +    debian_uri = (d.getVar("DEBIAN_URI", True) or "").split()
>> +
>> +    try:
>> +        fetcher = bb.fetch2.Fetch(debian_uri, d)
>> +        fetcher.unpack(workdir + "/" + s)
> 
> fetch^3 this time with unpack

Again:
  - .orig.tar.xz is unpacked by standard fetcher
  - the content of .debian.tar.xz should be unpacked into the folder 
from the .orig.tar.xz archive

So in general, this is an attempt to collect and prepare all the 
artifacts to make it usable with dpkg Isar class.

Alex

> 
> Henning
> 
>> +    except bb.fetch2.BBFetchException as e:
>> +        raise bb.build.FuncFailed(e)
>> +}
>> +
>> +# NOTE: this is workaround to serve quilt-3.0 package format support.
>> +#       To build such package, the original tarball should be placed
>> +#       in "../" folder. In general, this should be handled by Isar
>> +#       core build classes, but for now this class in the only one
>> user +#       for quilt-3.0, so for the first implementation let"s
>> keep it here. +# Open issues:
>> +#  - How to identify if package has quilt-3.0 source format?
>> +#  - How to avoid copying of tarballs from downloads to working dir?
>> +do_build_prepend() {
>> +    dl_dir="${@ d.getVar('DL_DIR', True)}"
>> +    workdir="${@ d.getVar('WORKDIR', True)}"
>> +    file="${@ ((d.getVar('SRC_URI',
>> True)).split(";")[0]).split("/")[-1]}" +
>> +    install -m 644 $dl_dir/$file $workdir
>> +}
>> +
>> +inherit dpkg
>