Re: [RFC][PATCH 2/3] classes/dsc: Basic Debian .dsc backend implementation

[Jan Kiszka <jan.kiszka@siemens.com>]

On 2018-04-23 19:27, [ext] Henning Schild wrote:
> Am Mon, 23 Apr 2018 19:11:12 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> On 04/23/2018 06:04 PM, Henning Schild wrote:
>>> Am Mon, 23 Apr 2018 16:21:45 +0300
>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>>>   
>>>> This patch introduces class to work with Debian .dsc files. The
>>>> main goal is to simplify re-building and customization of original
>>>> Debian packages.
>>>>
>>>> To rebuild package from the upstream the following recipe template
>>>> could be used:
>>>>
>>>>    DSC_URI = "http://ftp.de.debian.org/...dsc;md5sum=..."
>>>>    inherit debian-dsc
>>>>
>>>> All the remaining information bitbake will derive automatically.
>>>> This is only first implementation, so there are still several open
>>>> issues like:
>>>>   - Chose correct upstream package version depending on Debian
>>>> distro
>>>>   - quilt-3.0 source format support
>>>>   - Cross-recipe dependencies recognition
>>>>
>>>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>>>> ---
>>>>   meta/classes/debian-dsc.bbclass | 112
>>>> ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 112
>>>> insertions(+) create mode 100644 meta/classes/debian-dsc.bbclass
>>>>
>>>> diff --git a/meta/classes/debian-dsc.bbclass
>>>> b/meta/classes/debian-dsc.bbclass new file mode 100644
>>>> index 0000000..4159e3a
>>>> --- /dev/null
>>>> +++ b/meta/classes/debian-dsc.bbclass
>>>> @@ -0,0 +1,112 @@
>>>> +# Debian .dsc backend
>>>> +#
>>>> +# This software is a part of ISAR.
>>>> +# Copyright (c) ilbers GmbH, 2018
>>>> +#
>>>> +# SPDX-License-Identifier: MIT
>>>> +
>>>> +# List of non-standard variables added to package workspace
>>>> +#
>>>> +# Set by user:
>>>> +#  * DSC_URI - uri link to .dsc file in upstream apt
>>>> +#
>>>> +# Generated automatically:
>>>> +#  * DEBIAN_URI - uri link to Debian patch in upstream apt
>>>> +
>>>> +python __anonymous() {
>>>> +    # Fetch .dsc package file
>>>> +    dsc_uri = (d.getVar("DSC_URI", True) or "").split()
>>>> +    if len(dsc_uri) == 0:
>>>> +        return  
>>>
>>> Can we not use SRC_URI and append more sources as we read .dscs? A
>>> copy of the fetching code just reminds me of the broken
>>> fetch/unpack in the early Isar days.
>>> Because i have got my dscs in tarballs and in git ... and i am
>>> behind a proxy.  
>>
>> Here I consider only traditional Debian way, when.dsc file is stored 
>> directly in apt with .orig tarball and .debian patch:
>>
>> http://ftp.de.debian.org/debian/pool/main/h/hello/
>>
>> So this class intended to work with Debian apts.
>>
>>>   
>>>> +    try:
>>>> +        fetcher = bb.fetch2.Fetch(dsc_uri, d)
>>>> +        fetcher.download()
>>>> +    except bb.fetch2.BBFetchException as e:
>>>> +        raise bb.build.FuncFailed(e)
>>>> +
>>>> +    # Open .dsc file from downloads
>>>> +    dl_dir = d.getVar("DL_DIR", True)
>>>> +    dsc_file = (dsc_uri[0].split(";")[0]).split("/")[-1]
>>>> +    filepath = dl_dir + "/" + dsc_file
>>>> +
>>>> +    pv = ""
>>>> +    src_uri = ""
>>>> +
>>>> +    # Parse .dsc file to get package details
>>>> +    with open(filepath, "r") as file:
>>>> +        line = file.readline()
>>>> +
>>>> +        while line:
>>>> +            # Get package version and export PV
>>>> +            if line.startswith("Version:") and not pv:
>>>> +                pv = line.split(": ")[-1].rstrip()
>>>> +                d.setVar("PV", pv)
>>>> +
>>>> +            # Get package and Debian patch arcives names
>>>> +            if line.startswith("Files:") and not src_uri:
>>>> +                line = file.readline()
>>>> +                src = line.split(" ")[-1].rstrip()
>>>> +                src_uri = (dsc_uri[0].rsplit("/", 1))[0] + "/" +
>>>> src
>>>> +                src_uri = src_uri + ";md5sum=" + line.split("
>>>> ")[1].rstrip()
>>>> +                d.setVar("SRC_URI", src_uri)
>>>> +
>>>> +                line = file.readline()
>>>> +                debian = line.split(" ")[-1].rstrip()
>>>> +                debian_uri = (dsc_uri[0].rsplit("/", 1))[0] + "/"
>>>> + debian
>>>> +                debian_uri = debian_uri + ";md5sum=" +
>>>> line.split(" ")[1].rstrip()
>>>> +                d.setVar("DEBIAN_URI", debian_uri)
>>>> +
>>>> +            line = file.readline()
>>>> +
>>>> +        file.close()
>>>> +
>>>> +    # Set correct path to unpacked sources
>>>> +    pv_orig = (src.split("_")[-1]).split(".orig")[0]
>>>> +    pn = d.getVar("PN", True)
>>>> +    d.setVar("S", pn + "-" + pv_orig)
>>>> +}
>>>> +
>>>> +python do_fetch_append() {
>>>> +    debian_uri = (d.getVar("DEBIAN_URI", True) or "").split()
>>>> +    if len(debian_uri) == 0:
>>>> +        return
>>>> +
>>>> +    try:
>>>> +        fetcher = bb.fetch2.Fetch(debian_uri, d)
>>>> +        fetcher.download()
>>>> +    except bb.fetch2.BBFetchException as e:
>>>> +        raise bb.build.FuncFailed(e)
>>>> +}  
>>>
>>> Wow another fetcher ... same question as above.  
>>
>> Upstream Debian source package contains 3 parts:
>>   - .dsc
>>   - .orig.tar.xz
>>   - .debian.tar.xz
>>
>> So:
>>   - First you need to fetch .dsc to get information about remaining 2 
>> artifacts.
>>   - .orig.tar.xz is fetched by standard fetcher.
>>   - .debian.tar.xz is fetched here.
>>
>>>   
>>>> +python do_unpack_append() {
>>>> +    import subprocess
>>>> +
>>>> +    s = d.getVar("S", True)
>>>> +    workdir = d.getVar("WORKDIR", True)
>>>> +    debian_uri = (d.getVar("DEBIAN_URI", True) or "").split()
>>>> +
>>>> +    try:
>>>> +        fetcher = bb.fetch2.Fetch(debian_uri, d)
>>>> +        fetcher.unpack(workdir + "/" + s)  
>>>
>>> fetch^3 this time with unpack  
>>
>> Again:
>>   - .orig.tar.xz is unpacked by standard fetcher
>>   - the content of .debian.tar.xz should be unpacked into the folder 
>> from the .orig.tar.xz archive
>>
>> So in general, this is an attempt to collect and prepare all the 
>> artifacts to make it usable with dpkg Isar class.
> 
> Sure, fully understood the moment you mentioned dsc. All i wanted to
> find out whether we can use SRC_URI and the one fetcher/unapcker we
> have already. Why should it be called DSC_URI only because you download
> a .dsc?
> The magic that an "unpack" of a .dsc involves downloading even more and
> maybe unpacking that ... should go into a central place (do_unpack?) and
> SRC_URI should stay SRC_URI.
> 
> i.e.: do_fetch/unpack in base.bbclass get
> ...
> for uri in src_uri:
> 	if uri ends with .dsc and unpack==true:
> 		add new uris to final_src_uri
> 	else:
> 		add uri to final_src_uri
> src_uri = final_src_uri
> ...
> 

Ack, we should always try to design from the perspective of our users,
and there no reason for them to have separate URI lists.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux