Re: [RFC][PATCH 2/3] classes/dsc: Basic Debian .dsc backend implementation

[Henning Schild <henning.schild@siemens.com>]

Am Mon, 23 Apr 2018 19:11:12 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> On 04/23/2018 06:04 PM, Henning Schild wrote:
> > Am Mon, 23 Apr 2018 16:21:45 +0300
> > schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >   
> >> This patch introduces class to work with Debian .dsc files. The
> >> main goal is to simplify re-building and customization of original
> >> Debian packages.
> >>
> >> To rebuild package from the upstream the following recipe template
> >> could be used:
> >>
> >>    DSC_URI = "http://ftp.de.debian.org/...dsc;md5sum=..."
> >>    inherit debian-dsc
> >>
> >> All the remaining information bitbake will derive automatically.
> >> This is only first implementation, so there are still several open
> >> issues like:
> >>   - Chose correct upstream package version depending on Debian
> >> distro
> >>   - quilt-3.0 source format support
> >>   - Cross-recipe dependencies recognition
> >>
> >> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> >> ---
> >>   meta/classes/debian-dsc.bbclass | 112
> >> ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 112
> >> insertions(+) create mode 100644 meta/classes/debian-dsc.bbclass
> >>
> >> diff --git a/meta/classes/debian-dsc.bbclass
> >> b/meta/classes/debian-dsc.bbclass new file mode 100644
> >> index 0000000..4159e3a
> >> --- /dev/null
> >> +++ b/meta/classes/debian-dsc.bbclass
> >> @@ -0,0 +1,112 @@
> >> +# Debian .dsc backend
> >> +#
> >> +# This software is a part of ISAR.
> >> +# Copyright (c) ilbers GmbH, 2018
> >> +#
> >> +# SPDX-License-Identifier: MIT
> >> +
> >> +# List of non-standard variables added to package workspace
> >> +#
> >> +# Set by user:
> >> +#  * DSC_URI - uri link to .dsc file in upstream apt
> >> +#
> >> +# Generated automatically:
> >> +#  * DEBIAN_URI - uri link to Debian patch in upstream apt
> >> +
> >> +python __anonymous() {
> >> +    # Fetch .dsc package file
> >> +    dsc_uri = (d.getVar("DSC_URI", True) or "").split()
> >> +    if len(dsc_uri) == 0:
> >> +        return  
> > 
> > Can we not use SRC_URI and append more sources as we read .dscs? A
> > copy of the fetching code just reminds me of the broken
> > fetch/unpack in the early Isar days.
> > Because i have got my dscs in tarballs and in git ... and i am
> > behind a proxy.  
> 
> Here I consider only traditional Debian way, when.dsc file is stored 
> directly in apt with .orig tarball and .debian patch:
> 
> http://ftp.de.debian.org/debian/pool/main/h/hello/
> 
> So this class intended to work with Debian apts.
> 
> >   
> >> +    try:
> >> +        fetcher = bb.fetch2.Fetch(dsc_uri, d)
> >> +        fetcher.download()
> >> +    except bb.fetch2.BBFetchException as e:
> >> +        raise bb.build.FuncFailed(e)
> >> +
> >> +    # Open .dsc file from downloads
> >> +    dl_dir = d.getVar("DL_DIR", True)
> >> +    dsc_file = (dsc_uri[0].split(";")[0]).split("/")[-1]
> >> +    filepath = dl_dir + "/" + dsc_file
> >> +
> >> +    pv = ""
> >> +    src_uri = ""
> >> +
> >> +    # Parse .dsc file to get package details
> >> +    with open(filepath, "r") as file:
> >> +        line = file.readline()
> >> +
> >> +        while line:
> >> +            # Get package version and export PV
> >> +            if line.startswith("Version:") and not pv:
> >> +                pv = line.split(": ")[-1].rstrip()
> >> +                d.setVar("PV", pv)
> >> +
> >> +            # Get package and Debian patch arcives names
> >> +            if line.startswith("Files:") and not src_uri:
> >> +                line = file.readline()
> >> +                src = line.split(" ")[-1].rstrip()
> >> +                src_uri = (dsc_uri[0].rsplit("/", 1))[0] + "/" +
> >> src
> >> +                src_uri = src_uri + ";md5sum=" + line.split("
> >> ")[1].rstrip()
> >> +                d.setVar("SRC_URI", src_uri)
> >> +
> >> +                line = file.readline()
> >> +                debian = line.split(" ")[-1].rstrip()
> >> +                debian_uri = (dsc_uri[0].rsplit("/", 1))[0] + "/"
> >> + debian
> >> +                debian_uri = debian_uri + ";md5sum=" +
> >> line.split(" ")[1].rstrip()
> >> +                d.setVar("DEBIAN_URI", debian_uri)
> >> +
> >> +            line = file.readline()
> >> +
> >> +        file.close()
> >> +
> >> +    # Set correct path to unpacked sources
> >> +    pv_orig = (src.split("_")[-1]).split(".orig")[0]
> >> +    pn = d.getVar("PN", True)
> >> +    d.setVar("S", pn + "-" + pv_orig)
> >> +}
> >> +
> >> +python do_fetch_append() {
> >> +    debian_uri = (d.getVar("DEBIAN_URI", True) or "").split()
> >> +    if len(debian_uri) == 0:
> >> +        return
> >> +
> >> +    try:
> >> +        fetcher = bb.fetch2.Fetch(debian_uri, d)
> >> +        fetcher.download()
> >> +    except bb.fetch2.BBFetchException as e:
> >> +        raise bb.build.FuncFailed(e)
> >> +}  
> > 
> > Wow another fetcher ... same question as above.  
> 
> Upstream Debian source package contains 3 parts:
>   - .dsc
>   - .orig.tar.xz
>   - .debian.tar.xz
> 
> So:
>   - First you need to fetch .dsc to get information about remaining 2 
> artifacts.
>   - .orig.tar.xz is fetched by standard fetcher.
>   - .debian.tar.xz is fetched here.
> 
> >   
> >> +python do_unpack_append() {
> >> +    import subprocess
> >> +
> >> +    s = d.getVar("S", True)
> >> +    workdir = d.getVar("WORKDIR", True)
> >> +    debian_uri = (d.getVar("DEBIAN_URI", True) or "").split()
> >> +
> >> +    try:
> >> +        fetcher = bb.fetch2.Fetch(debian_uri, d)
> >> +        fetcher.unpack(workdir + "/" + s)  
> > 
> > fetch^3 this time with unpack  
> 
> Again:
>   - .orig.tar.xz is unpacked by standard fetcher
>   - the content of .debian.tar.xz should be unpacked into the folder 
> from the .orig.tar.xz archive
> 
> So in general, this is an attempt to collect and prepare all the 
> artifacts to make it usable with dpkg Isar class.

Sure, fully understood the moment you mentioned dsc. All i wanted to
find out whether we can use SRC_URI and the one fetcher/unapcker we
have already. Why should it be called DSC_URI only because you download
a .dsc?
The magic that an "unpack" of a .dsc involves downloading even more and
maybe unpacking that ... should go into a central place (do_unpack?) and
SRC_URI should stay SRC_URI.

i.e.: do_fetch/unpack in base.bbclass get
...
for uri in src_uri:
	if uri ends with .dsc and unpack==true:
		add new uris to final_src_uri
	else:
		add uri to final_src_uri
src_uri = final_src_uri
...

Henning

> Alex
> 
> > 
> > Henning
> >   
> >> +    except bb.fetch2.BBFetchException as e:
> >> +        raise bb.build.FuncFailed(e)
> >> +}
> >> +
> >> +# NOTE: this is workaround to serve quilt-3.0 package format
> >> support. +#       To build such package, the original tarball
> >> should be placed +#       in "../" folder. In general, this should
> >> be handled by Isar +#       core build classes, but for now this
> >> class in the only one user +#       for quilt-3.0, so for the
> >> first implementation let"s keep it here. +# Open issues:
> >> +#  - How to identify if package has quilt-3.0 source format?
> >> +#  - How to avoid copying of tarballs from downloads to working
> >> dir? +do_build_prepend() {
> >> +    dl_dir="${@ d.getVar('DL_DIR', True)}"
> >> +    workdir="${@ d.getVar('WORKDIR', True)}"
> >> +    file="${@ ((d.getVar('SRC_URI',
> >> True)).split(";")[0]).split("/")[-1]}" +
> >> +    install -m 644 $dl_dir/$file $workdir
> >> +}
> >> +
> >> +inherit dpkg  
> >