[RFC][PATCH 0/3] Experiments with .dsc backend

[RFC][PATCH 0/3] Experiments with .dsc backend

[Alexander Smirnov]

Hi everybody,

as it was planned initially, one of the Isar feature is the ability to
rebuild original Debian packages with/without customizations. Doing
this manually in classical Yocto/OE style requires manual setting of
lots of variables and having a copy of Debian patch (*.debian.tar.xz)
for this package in your recipe's folder. Maintenance of this system
becomes very complicated.

I performed a small research regarding bitbake capabilities and found
an interesting trick: during bitbake parsing stage (in anonymous tasks)
we could re-construct whole recipe (including tasks and variable), and
this snapshot will be stored in recipe's dynamic context, that is used
during pipeline execution.

This series is just a PoC, lots of issues are still open, like:
 - Patching upstream packages
 - Automatic cross-recipe dependency detection to avoid duplications in
   recipe and debian/control files
 - etc...

To test this series just run:

 $ bitbake hello

After build, in hello workdir there will be a newly-baked identical copy of
upstream package :-)

Feedbacks, ideas, proposals are welcome!

With best regards,
Alex

Alexander Smirnov (3):
  buildchroot: Include texinfo to base system
  classes/dsc: Basic Debian .dsc backend implementation
  recipes-debian: Add example recipe to test Debian .dsc support

 meta-isar/recipes-debian/hello/hello.bb          |  10 ++
 meta/classes/debian-dsc.bbclass                  | 112 +++++++++++++++++++++++
 meta/recipes-devtools/buildchroot/buildchroot.bb |   3 +-
 3 files changed, 124 insertions(+), 1 deletion(-)
 create mode 100644 meta-isar/recipes-debian/hello/hello.bb
 create mode 100644 meta/classes/debian-dsc.bbclass

-- 
2.9.5

[RFC][PATCH 1/3] buildchroot: Include texinfo to base system

[Alexander Smirnov]

'makeinfo' is the standard tool that used in Debain to generate
man pages. Unfortunately it's not listed in 'debian/control' as
build dependency (at least for hello Debian package), so let's
include it by default.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta/recipes-devtools/buildchroot/buildchroot.bb | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
index b16e63a..67e9198 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
@@ -26,7 +26,8 @@ BUILDCHROOT_PREINSTALL ?= "gcc \
                            apt \
                            automake \
                            devscripts \
-                           equivs"
+                           equivs \
+                           texinfo"
 
 WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
 
-- 
2.9.5

[RFC][PATCH 2/3] classes/dsc: Basic Debian .dsc backend implementation

[Alexander Smirnov]

This patch introduces class to work with Debian .dsc files. The
main goal is to simplify re-building and customization of original
Debian packages.

To rebuild package from the upstream the following recipe template
could be used:

  DSC_URI = "http://ftp.de.debian.org/...dsc;md5sum=..."
  inherit debian-dsc

All the remaining information bitbake will derive automatically.
This is only first implementation, so there are still several open
issues like:
 - Chose correct upstream package version depending on Debian distro
 - quilt-3.0 source format support
 - Cross-recipe dependencies recognition

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta/classes/debian-dsc.bbclass | 112 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 112 insertions(+)
 create mode 100644 meta/classes/debian-dsc.bbclass

diff --git a/meta/classes/debian-dsc.bbclass b/meta/classes/debian-dsc.bbclass
new file mode 100644
index 0000000..4159e3a
--- /dev/null
+++ b/meta/classes/debian-dsc.bbclass
@@ -0,0 +1,112 @@
+# Debian .dsc backend
+#
+# This software is a part of ISAR.
+# Copyright (c) ilbers GmbH, 2018
+#
+# SPDX-License-Identifier: MIT
+
+# List of non-standard variables added to package workspace
+#
+# Set by user:
+#  * DSC_URI - uri link to .dsc file in upstream apt
+#
+# Generated automatically:
+#  * DEBIAN_URI - uri link to Debian patch in upstream apt
+
+python __anonymous() {
+    # Fetch .dsc package file
+    dsc_uri = (d.getVar("DSC_URI", True) or "").split()
+    if len(dsc_uri) == 0:
+        return
+
+    try:
+        fetcher = bb.fetch2.Fetch(dsc_uri, d)
+        fetcher.download()
+    except bb.fetch2.BBFetchException as e:
+        raise bb.build.FuncFailed(e)
+
+    # Open .dsc file from downloads
+    dl_dir = d.getVar("DL_DIR", True)
+    dsc_file = (dsc_uri[0].split(";")[0]).split("/")[-1]
+    filepath = dl_dir + "/" + dsc_file
+
+    pv = ""
+    src_uri = ""
+
+    # Parse .dsc file to get package details
+    with open(filepath, "r") as file:
+        line = file.readline()
+
+        while line:
+            # Get package version and export PV
+            if line.startswith("Version:") and not pv:
+                pv = line.split(": ")[-1].rstrip()
+                d.setVar("PV", pv)
+
+            # Get package and Debian patch arcives names
+            if line.startswith("Files:") and not src_uri:
+                line = file.readline()
+                src = line.split(" ")[-1].rstrip()
+                src_uri = (dsc_uri[0].rsplit("/", 1))[0] + "/" + src
+                src_uri = src_uri + ";md5sum=" + line.split(" ")[1].rstrip()
+                d.setVar("SRC_URI", src_uri)
+
+                line = file.readline()
+                debian = line.split(" ")[-1].rstrip()
+                debian_uri = (dsc_uri[0].rsplit("/", 1))[0] + "/" + debian
+                debian_uri = debian_uri + ";md5sum=" + line.split(" ")[1].rstrip()
+                d.setVar("DEBIAN_URI", debian_uri)
+
+            line = file.readline()
+
+        file.close()
+
+    # Set correct path to unpacked sources
+    pv_orig = (src.split("_")[-1]).split(".orig")[0]
+    pn = d.getVar("PN", True)
+    d.setVar("S", pn + "-" + pv_orig)
+}
+
+python do_fetch_append() {
+    debian_uri = (d.getVar("DEBIAN_URI", True) or "").split()
+    if len(debian_uri) == 0:
+        return
+
+    try:
+        fetcher = bb.fetch2.Fetch(debian_uri, d)
+        fetcher.download()
+    except bb.fetch2.BBFetchException as e:
+        raise bb.build.FuncFailed(e)
+}
+
+python do_unpack_append() {
+    import subprocess
+
+    s = d.getVar("S", True)
+    workdir = d.getVar("WORKDIR", True)
+    debian_uri = (d.getVar("DEBIAN_URI", True) or "").split()
+
+    try:
+        fetcher = bb.fetch2.Fetch(debian_uri, d)
+        fetcher.unpack(workdir + "/" + s)
+    except bb.fetch2.BBFetchException as e:
+        raise bb.build.FuncFailed(e)
+}
+
+# NOTE: this is workaround to serve quilt-3.0 package format support.
+#       To build such package, the original tarball should be placed
+#       in "../" folder. In general, this should be handled by Isar
+#       core build classes, but for now this class in the only one user
+#       for quilt-3.0, so for the first implementation let"s keep it here.
+# Open issues:
+#  - How to identify if package has quilt-3.0 source format?
+#  - How to avoid copying of tarballs from downloads to working dir?
+do_build_prepend() {
+    dl_dir="${@ d.getVar('DL_DIR', True)}"
+    workdir="${@ d.getVar('WORKDIR', True)}"
+    file="${@ ((d.getVar('SRC_URI', True)).split(";")[0]).split("/")[-1]}"
+
+    install -m 644 $dl_dir/$file $workdir
+}
+
+inherit dpkg
-- 
2.9.5

[RFC][PATCH 3/3] recipes-debian: Add example recipe to test Debian .dsc support

[Alexander Smirnov]

This patch adds recipe to build Debian hello package by using .dsc
backend.

To test this, the following command could be used:

 $ bitbake hello

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta-isar/recipes-debian/hello/hello.bb | 10 ++++++++++
 1 file changed, 10 insertions(+)
 create mode 100644 meta-isar/recipes-debian/hello/hello.bb

diff --git a/meta-isar/recipes-debian/hello/hello.bb b/meta-isar/recipes-debian/hello/hello.bb
new file mode 100644
index 0000000..c018141
--- /dev/null
+++ b/meta-isar/recipes-debian/hello/hello.bb
@@ -0,0 +1,10 @@
+# Sample recipe for original Debian package build
+#
+# This software is a part of ISAR.
+# Copyright (c) ilbers GmbH, 2018
+#
+# SPDX-License-Identifier: MIT
+
+DSC_URI = "http://ftp.de.debian.org/debian/pool/main/h/hello/hello_2.9-2+deb8u1.dsc;md5sum=faa827e0ff1d7e76c12488ed77e8fc76"
+
+inherit debian-dsc
-- 
2.9.5

Re: [RFC][PATCH 0/3] Experiments with .dsc backend

[Henning Schild]

Am Mon, 23 Apr 2018 16:21:43 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Hi everybody,
> 
> as it was planned initially, one of the Isar feature is the ability to
> rebuild original Debian packages with/without customizations. Doing
> this manually in classical Yocto/OE style requires manual setting of
> lots of variables and having a copy of Debian patch (*.debian.tar.xz)
> for this package in your recipe's folder. Maintenance of this system
> becomes very complicated.
> 
> I performed a small research regarding bitbake capabilities and found
> an interesting trick: during bitbake parsing stage (in anonymous
> tasks) we could re-construct whole recipe (including tasks and
> variable), and this snapshot will be stored in recipe's dynamic
> context, that is used during pipeline execution.
> 
> This series is just a PoC, lots of issues are still open, like:
>  - Patching upstream packages
>  - Automatic cross-recipe dependency detection to avoid duplications
> in recipe and debian/control files
>  - etc...
> 
> To test this series just run:
> 
>  $ bitbake hello
> 
> After build, in hello workdir there will be a newly-baked identical
> copy of upstream package :-)

Interesting idea, but my personal feeling is that Isar already uses a
lot of bitbake magic and we should maybe try to avoid introducing more
where feasible.

> Feedbacks, ideas, proposals are welcome!

So my feedback would be that the .dsc to recipe step should be put in a
helper tool in the scripts directory. People that want to modifiy
upstream packages would generate an isar-recipe with the tool once.
Modifications in the form of patches could go on top.
Or would this limit the envisioned feature in any way?

I also fear that this could complicate the reproducable build story.

Henning

> With best regards,
> Alex
> 
> Alexander Smirnov (3):
>   buildchroot: Include texinfo to base system
>   classes/dsc: Basic Debian .dsc backend implementation
>   recipes-debian: Add example recipe to test Debian .dsc support
> 
>  meta-isar/recipes-debian/hello/hello.bb          |  10 ++
>  meta/classes/debian-dsc.bbclass                  | 112
> +++++++++++++++++++++++
> meta/recipes-devtools/buildchroot/buildchroot.bb |   3 +- 3 files
> changed, 124 insertions(+), 1 deletion(-) create mode 100644
> meta-isar/recipes-debian/hello/hello.bb create mode 100644
> meta/classes/debian-dsc.bbclass
> 

Re: [RFC][PATCH 2/3] classes/dsc: Basic Debian .dsc backend implementation

[Henning Schild]

Am Mon, 23 Apr 2018 16:21:45 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> This patch introduces class to work with Debian .dsc files. The
> main goal is to simplify re-building and customization of original
> Debian packages.
> 
> To rebuild package from the upstream the following recipe template
> could be used:
> 
>   DSC_URI = "http://ftp.de.debian.org/...dsc;md5sum=..."
>   inherit debian-dsc
> 
> All the remaining information bitbake will derive automatically.
> This is only first implementation, so there are still several open
> issues like:
>  - Chose correct upstream package version depending on Debian distro
>  - quilt-3.0 source format support
>  - Cross-recipe dependencies recognition
> 
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
>  meta/classes/debian-dsc.bbclass | 112
> ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 112
> insertions(+) create mode 100644 meta/classes/debian-dsc.bbclass
> 
> diff --git a/meta/classes/debian-dsc.bbclass
> b/meta/classes/debian-dsc.bbclass new file mode 100644
> index 0000000..4159e3a
> --- /dev/null
> +++ b/meta/classes/debian-dsc.bbclass
> @@ -0,0 +1,112 @@
> +# Debian .dsc backend
> +#
> +# This software is a part of ISAR.
> +# Copyright (c) ilbers GmbH, 2018
> +#
> +# SPDX-License-Identifier: MIT
> +
> +# List of non-standard variables added to package workspace
> +#
> +# Set by user:
> +#  * DSC_URI - uri link to .dsc file in upstream apt
> +#
> +# Generated automatically:
> +#  * DEBIAN_URI - uri link to Debian patch in upstream apt
> +
> +python __anonymous() {
> +    # Fetch .dsc package file
> +    dsc_uri = (d.getVar("DSC_URI", True) or "").split()
> +    if len(dsc_uri) == 0:
> +        return

Can we not use SRC_URI and append more sources as we read .dscs? A
copy of the fetching code just reminds me of the broken fetch/unpack in
the early Isar days.
Because i have got my dscs in tarballs and in git ... and i am behind a
proxy.

> +    try:
> +        fetcher = bb.fetch2.Fetch(dsc_uri, d)
> +        fetcher.download()
> +    except bb.fetch2.BBFetchException as e:
> +        raise bb.build.FuncFailed(e)
> +
> +    # Open .dsc file from downloads
> +    dl_dir = d.getVar("DL_DIR", True)
> +    dsc_file = (dsc_uri[0].split(";")[0]).split("/")[-1]
> +    filepath = dl_dir + "/" + dsc_file
> +
> +    pv = ""
> +    src_uri = ""
> +
> +    # Parse .dsc file to get package details
> +    with open(filepath, "r") as file:
> +        line = file.readline()
> +
> +        while line:
> +            # Get package version and export PV
> +            if line.startswith("Version:") and not pv:
> +                pv = line.split(": ")[-1].rstrip()
> +                d.setVar("PV", pv)
> +
> +            # Get package and Debian patch arcives names
> +            if line.startswith("Files:") and not src_uri:
> +                line = file.readline()
> +                src = line.split(" ")[-1].rstrip()
> +                src_uri = (dsc_uri[0].rsplit("/", 1))[0] + "/" + src
> +                src_uri = src_uri + ";md5sum=" + line.split("
> ")[1].rstrip()
> +                d.setVar("SRC_URI", src_uri)
> +
> +                line = file.readline()
> +                debian = line.split(" ")[-1].rstrip()
> +                debian_uri = (dsc_uri[0].rsplit("/", 1))[0] + "/" +
> debian
> +                debian_uri = debian_uri + ";md5sum=" + line.split("
> ")[1].rstrip()
> +                d.setVar("DEBIAN_URI", debian_uri)
> +
> +            line = file.readline()
> +
> +        file.close()
> +
> +    # Set correct path to unpacked sources
> +    pv_orig = (src.split("_")[-1]).split(".orig")[0]
> +    pn = d.getVar("PN", True)
> +    d.setVar("S", pn + "-" + pv_orig)
> +}
> +
> +python do_fetch_append() {
> +    debian_uri = (d.getVar("DEBIAN_URI", True) or "").split()
> +    if len(debian_uri) == 0:
> +        return
> +
> +    try:
> +        fetcher = bb.fetch2.Fetch(debian_uri, d)
> +        fetcher.download()
> +    except bb.fetch2.BBFetchException as e:
> +        raise bb.build.FuncFailed(e)
> +}

Wow another fetcher ... same question as above.

> +python do_unpack_append() {
> +    import subprocess
> +
> +    s = d.getVar("S", True)
> +    workdir = d.getVar("WORKDIR", True)
> +    debian_uri = (d.getVar("DEBIAN_URI", True) or "").split()
> +
> +    try:
> +        fetcher = bb.fetch2.Fetch(debian_uri, d)
> +        fetcher.unpack(workdir + "/" + s)

fetch^3 this time with unpack

Henning

> +    except bb.fetch2.BBFetchException as e:
> +        raise bb.build.FuncFailed(e)
> +}
> +
> +# NOTE: this is workaround to serve quilt-3.0 package format support.
> +#       To build such package, the original tarball should be placed
> +#       in "../" folder. In general, this should be handled by Isar
> +#       core build classes, but for now this class in the only one
> user +#       for quilt-3.0, so for the first implementation let"s
> keep it here. +# Open issues:
> +#  - How to identify if package has quilt-3.0 source format?
> +#  - How to avoid copying of tarballs from downloads to working dir?
> +do_build_prepend() {
> +    dl_dir="${@ d.getVar('DL_DIR', True)}"
> +    workdir="${@ d.getVar('WORKDIR', True)}"
> +    file="${@ ((d.getVar('SRC_URI',
> True)).split(";")[0]).split("/")[-1]}" +
> +    install -m 644 $dl_dir/$file $workdir
> +}
> +
> +inherit dpkg

Re: [RFC][PATCH 0/3] Experiments with .dsc backend

[Alexander Smirnov]

On 04/23/2018 05:36 PM, Henning Schild wrote:
> Am Mon, 23 Apr 2018 16:21:43 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> Hi everybody,
>>
>> as it was planned initially, one of the Isar feature is the ability to
>> rebuild original Debian packages with/without customizations. Doing
>> this manually in classical Yocto/OE style requires manual setting of
>> lots of variables and having a copy of Debian patch (*.debian.tar.xz)
>> for this package in your recipe's folder. Maintenance of this system
>> becomes very complicated.
>>
>> I performed a small research regarding bitbake capabilities and found
>> an interesting trick: during bitbake parsing stage (in anonymous
>> tasks) we could re-construct whole recipe (including tasks and
>> variable), and this snapshot will be stored in recipe's dynamic
>> context, that is used during pipeline execution.
>>
>> This series is just a PoC, lots of issues are still open, like:
>>   - Patching upstream packages
>>   - Automatic cross-recipe dependency detection to avoid duplications
>> in recipe and debian/control files
>>   - etc...
>>
>> To test this series just run:
>>
>>   $ bitbake hello
>>
>> After build, in hello workdir there will be a newly-baked identical
>> copy of upstream package :-)
> 
> Interesting idea, but my personal feeling is that Isar already uses a
> lot of bitbake magic and we should maybe try to avoid introducing more
> where feasible.
> 

Bitbake was designed for Open Embedded, so it doesn't consider our 
Isar/Debian usecases. And roughly speaking, bitbake usage in Isar is a 
kind of hack at all. In such questions, I'd more prefer to focus on the 
features we need and extend bitbake, instead of applying Yocto 
philosophy to Isar. But this also is only my personal feeling.

>> Feedbacks, ideas, proposals are welcome!
> 
> So my feedback would be that the .dsc to recipe step should be put in a
> helper tool in the scripts directory. People that want to modifiy
> upstream packages would generate an isar-recipe with the tool once.
> Modifications in the form of patches could go on top.
> Or would this limit the envisioned feature in any way?

IMHO this is undesired behavior:
  - Extra step to generate new recipes
  - Extra step to add these recipes to build
  - Headache with maintenance: you touch some dsc -> you need to 
regenerate new recipes etc...
  - You should somehow/somewhere list of .dsc files
So this looks like a completely new ecosystem.

.dsc is standard format to describe Debian source packages. So anyway, 
to be Debian compliant Isar should support this format.

> 
> I also fear that this could complicate the reproducable build story.

In general this doesn't affect reproducibility at all. The key 
difference from typical recipe (build from tarball), that SRC_URI is 
derived dynamically from .dsc file during parsing stage. The remaining 
part of the pipeline stays untouched.

> 
> Henning
> 
>> With best regards,
>> Alex
>>
>> Alexander Smirnov (3):
>>    buildchroot: Include texinfo to base system
>>    classes/dsc: Basic Debian .dsc backend implementation
>>    recipes-debian: Add example recipe to test Debian .dsc support
>>
>>   meta-isar/recipes-debian/hello/hello.bb          |  10 ++
>>   meta/classes/debian-dsc.bbclass                  | 112
>> +++++++++++++++++++++++
>> meta/recipes-devtools/buildchroot/buildchroot.bb |   3 +- 3 files
>> changed, 124 insertions(+), 1 deletion(-) create mode 100644
>> meta-isar/recipes-debian/hello/hello.bb create mode 100644
>> meta/classes/debian-dsc.bbclass
>>
>

Re: [RFC][PATCH 0/3] Experiments with .dsc backend

[Henning Schild]

Am Mon, 23 Apr 2018 18:26:10 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> On 04/23/2018 05:36 PM, Henning Schild wrote:
> > Am Mon, 23 Apr 2018 16:21:43 +0300
> > schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >   
> >> Hi everybody,
> >>
> >> as it was planned initially, one of the Isar feature is the
> >> ability to rebuild original Debian packages with/without
> >> customizations. Doing this manually in classical Yocto/OE style
> >> requires manual setting of lots of variables and having a copy of
> >> Debian patch (*.debian.tar.xz) for this package in your recipe's
> >> folder. Maintenance of this system becomes very complicated.
> >>
> >> I performed a small research regarding bitbake capabilities and
> >> found an interesting trick: during bitbake parsing stage (in
> >> anonymous tasks) we could re-construct whole recipe (including
> >> tasks and variable), and this snapshot will be stored in recipe's
> >> dynamic context, that is used during pipeline execution.
> >>
> >> This series is just a PoC, lots of issues are still open, like:
> >>   - Patching upstream packages
> >>   - Automatic cross-recipe dependency detection to avoid
> >> duplications in recipe and debian/control files
> >>   - etc...
> >>
> >> To test this series just run:
> >>
> >>   $ bitbake hello
> >>
> >> After build, in hello workdir there will be a newly-baked identical
> >> copy of upstream package :-)  
> > 
> > Interesting idea, but my personal feeling is that Isar already uses
> > a lot of bitbake magic and we should maybe try to avoid introducing
> > more where feasible.
> >   
> 
> Bitbake was designed for Open Embedded, so it doesn't consider our 
> Isar/Debian usecases. And roughly speaking, bitbake usage in Isar is
> a kind of hack at all. In such questions, I'd more prefer to focus on
> the features we need and extend bitbake, instead of applying Yocto 
> philosophy to Isar. But this also is only my personal feeling.
> 
> >> Feedbacks, ideas, proposals are welcome!  
> > 
> > So my feedback would be that the .dsc to recipe step should be put
> > in a helper tool in the scripts directory. People that want to
> > modifiy upstream packages would generate an isar-recipe with the
> > tool once. Modifications in the form of patches could go on top.
> > Or would this limit the envisioned feature in any way?  
> 
> IMHO this is undesired behavior:
>   - Extra step to generate new recipes

Yes, but you can see it.

>   - Extra step to add these recipes to build

No, not really. You will have to mention your recipe somewhere if you
want it in your image. What is does and how does not change that fact.

BTW, did you try overloading something like an essential package? Think
of something that is in buildchroot and needs to be updated in there.
(libc in the worst case ...)

>   - Headache with maintenance: you touch some dsc -> you need to 
> regenerate new recipes etc...

True, but major changes in the upstream package will probably make your
patches not apply anymore. So i suspect headache here as well, the one
thing you do with such a tool is story copies of possibly outdated
dscs, which indeed is a bad idea. 
So the online recipe generation is probably the way to go.

>   - You should somehow/somewhere list of .dsc files
> So this looks like a completely new ecosystem.
> 
> .dsc is standard format to describe Debian source packages. So
> anyway, to be Debian compliant Isar should support this format.
> 
> > 
> > I also fear that this could complicate the reproducable build
> > story.  
> 
> In general this doesn't affect reproducibility at all. The key 
> difference from typical recipe (build from tarball), that SRC_URI is 
> derived dynamically from .dsc file during parsing stage. The
> remaining part of the pipeline stays untouched.

It affects it because we will have to redirect every fetch to a local
cache, and it introduces fetching.

Henning

> > 
> > Henning
> >   
> >> With best regards,
> >> Alex
> >>
> >> Alexander Smirnov (3):
> >>    buildchroot: Include texinfo to base system
> >>    classes/dsc: Basic Debian .dsc backend implementation
> >>    recipes-debian: Add example recipe to test Debian .dsc support
> >>
> >>   meta-isar/recipes-debian/hello/hello.bb          |  10 ++
> >>   meta/classes/debian-dsc.bbclass                  | 112
> >> +++++++++++++++++++++++
> >> meta/recipes-devtools/buildchroot/buildchroot.bb |   3 +- 3 files
> >> changed, 124 insertions(+), 1 deletion(-) create mode 100644
> >> meta-isar/recipes-debian/hello/hello.bb create mode 100644
> >> meta/classes/debian-dsc.bbclass
> >>  
> >  

Re: [RFC][PATCH 2/3] classes/dsc: Basic Debian .dsc backend implementation

[Alexander Smirnov]

On 04/23/2018 06:04 PM, Henning Schild wrote:
> Am Mon, 23 Apr 2018 16:21:45 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> This patch introduces class to work with Debian .dsc files. The
>> main goal is to simplify re-building and customization of original
>> Debian packages.
>>
>> To rebuild package from the upstream the following recipe template
>> could be used:
>>
>>    DSC_URI = "http://ftp.de.debian.org/...dsc;md5sum=..."
>>    inherit debian-dsc
>>
>> All the remaining information bitbake will derive automatically.
>> This is only first implementation, so there are still several open
>> issues like:
>>   - Chose correct upstream package version depending on Debian distro
>>   - quilt-3.0 source format support
>>   - Cross-recipe dependencies recognition
>>
>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>> ---
>>   meta/classes/debian-dsc.bbclass | 112
>> ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 112
>> insertions(+) create mode 100644 meta/classes/debian-dsc.bbclass
>>
>> diff --git a/meta/classes/debian-dsc.bbclass
>> b/meta/classes/debian-dsc.bbclass new file mode 100644
>> index 0000000..4159e3a
>> --- /dev/null
>> +++ b/meta/classes/debian-dsc.bbclass
>> @@ -0,0 +1,112 @@
>> +# Debian .dsc backend
>> +#
>> +# This software is a part of ISAR.
>> +# Copyright (c) ilbers GmbH, 2018
>> +#
>> +# SPDX-License-Identifier: MIT
>> +
>> +# List of non-standard variables added to package workspace
>> +#
>> +# Set by user:
>> +#  * DSC_URI - uri link to .dsc file in upstream apt
>> +#
>> +# Generated automatically:
>> +#  * DEBIAN_URI - uri link to Debian patch in upstream apt
>> +
>> +python __anonymous() {
>> +    # Fetch .dsc package file
>> +    dsc_uri = (d.getVar("DSC_URI", True) or "").split()
>> +    if len(dsc_uri) == 0:
>> +        return
> 
> Can we not use SRC_URI and append more sources as we read .dscs? A
> copy of the fetching code just reminds me of the broken fetch/unpack in
> the early Isar days.
> Because i have got my dscs in tarballs and in git ... and i am behind a
> proxy.

Here I consider only traditional Debian way, when.dsc file is stored 
directly in apt with .orig tarball and .debian patch:

http://ftp.de.debian.org/debian/pool/main/h/hello/

So this class intended to work with Debian apts.

> 
>> +    try:
>> +        fetcher = bb.fetch2.Fetch(dsc_uri, d)
>> +        fetcher.download()
>> +    except bb.fetch2.BBFetchException as e:
>> +        raise bb.build.FuncFailed(e)
>> +
>> +    # Open .dsc file from downloads
>> +    dl_dir = d.getVar("DL_DIR", True)
>> +    dsc_file = (dsc_uri[0].split(";")[0]).split("/")[-1]
>> +    filepath = dl_dir + "/" + dsc_file
>> +
>> +    pv = ""
>> +    src_uri = ""
>> +
>> +    # Parse .dsc file to get package details
>> +    with open(filepath, "r") as file:
>> +        line = file.readline()
>> +
>> +        while line:
>> +            # Get package version and export PV
>> +            if line.startswith("Version:") and not pv:
>> +                pv = line.split(": ")[-1].rstrip()
>> +                d.setVar("PV", pv)
>> +
>> +            # Get package and Debian patch arcives names
>> +            if line.startswith("Files:") and not src_uri:
>> +                line = file.readline()
>> +                src = line.split(" ")[-1].rstrip()
>> +                src_uri = (dsc_uri[0].rsplit("/", 1))[0] + "/" + src
>> +                src_uri = src_uri + ";md5sum=" + line.split("
>> ")[1].rstrip()
>> +                d.setVar("SRC_URI", src_uri)
>> +
>> +                line = file.readline()
>> +                debian = line.split(" ")[-1].rstrip()
>> +                debian_uri = (dsc_uri[0].rsplit("/", 1))[0] + "/" +
>> debian
>> +                debian_uri = debian_uri + ";md5sum=" + line.split("
>> ")[1].rstrip()
>> +                d.setVar("DEBIAN_URI", debian_uri)
>> +
>> +            line = file.readline()
>> +
>> +        file.close()
>> +
>> +    # Set correct path to unpacked sources
>> +    pv_orig = (src.split("_")[-1]).split(".orig")[0]
>> +    pn = d.getVar("PN", True)
>> +    d.setVar("S", pn + "-" + pv_orig)
>> +}
>> +
>> +python do_fetch_append() {
>> +    debian_uri = (d.getVar("DEBIAN_URI", True) or "").split()
>> +    if len(debian_uri) == 0:
>> +        return
>> +
>> +    try:
>> +        fetcher = bb.fetch2.Fetch(debian_uri, d)
>> +        fetcher.download()
>> +    except bb.fetch2.BBFetchException as e:
>> +        raise bb.build.FuncFailed(e)
>> +}
> 
> Wow another fetcher ... same question as above.

Upstream Debian source package contains 3 parts:
  - .dsc
  - .orig.tar.xz
  - .debian.tar.xz

So:
  - First you need to fetch .dsc to get information about remaining 2 
artifacts.
  - .orig.tar.xz is fetched by standard fetcher.
  - .debian.tar.xz is fetched here.

> 
>> +python do_unpack_append() {
>> +    import subprocess
>> +
>> +    s = d.getVar("S", True)
>> +    workdir = d.getVar("WORKDIR", True)
>> +    debian_uri = (d.getVar("DEBIAN_URI", True) or "").split()
>> +
>> +    try:
>> +        fetcher = bb.fetch2.Fetch(debian_uri, d)
>> +        fetcher.unpack(workdir + "/" + s)
> 
> fetch^3 this time with unpack

Again:
  - .orig.tar.xz is unpacked by standard fetcher
  - the content of .debian.tar.xz should be unpacked into the folder 
from the .orig.tar.xz archive

So in general, this is an attempt to collect and prepare all the 
artifacts to make it usable with dpkg Isar class.

Alex

> 
> Henning
> 
>> +    except bb.fetch2.BBFetchException as e:
>> +        raise bb.build.FuncFailed(e)
>> +}
>> +
>> +# NOTE: this is workaround to serve quilt-3.0 package format support.
>> +#       To build such package, the original tarball should be placed
>> +#       in "../" folder. In general, this should be handled by Isar
>> +#       core build classes, but for now this class in the only one
>> user +#       for quilt-3.0, so for the first implementation let"s
>> keep it here. +# Open issues:
>> +#  - How to identify if package has quilt-3.0 source format?
>> +#  - How to avoid copying of tarballs from downloads to working dir?
>> +do_build_prepend() {
>> +    dl_dir="${@ d.getVar('DL_DIR', True)}"
>> +    workdir="${@ d.getVar('WORKDIR', True)}"
>> +    file="${@ ((d.getVar('SRC_URI',
>> True)).split(";")[0]).split("/")[-1]}" +
>> +    install -m 644 $dl_dir/$file $workdir
>> +}
>> +
>> +inherit dpkg
>

Re: [RFC][PATCH 2/3] classes/dsc: Basic Debian .dsc backend implementation

[Henning Schild]

Am Mon, 23 Apr 2018 19:11:12 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> On 04/23/2018 06:04 PM, Henning Schild wrote:
> > Am Mon, 23 Apr 2018 16:21:45 +0300
> > schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >   
> >> This patch introduces class to work with Debian .dsc files. The
> >> main goal is to simplify re-building and customization of original
> >> Debian packages.
> >>
> >> To rebuild package from the upstream the following recipe template
> >> could be used:
> >>
> >>    DSC_URI = "http://ftp.de.debian.org/...dsc;md5sum=..."
> >>    inherit debian-dsc
> >>
> >> All the remaining information bitbake will derive automatically.
> >> This is only first implementation, so there are still several open
> >> issues like:
> >>   - Chose correct upstream package version depending on Debian
> >> distro
> >>   - quilt-3.0 source format support
> >>   - Cross-recipe dependencies recognition
> >>
> >> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> >> ---
> >>   meta/classes/debian-dsc.bbclass | 112
> >> ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 112
> >> insertions(+) create mode 100644 meta/classes/debian-dsc.bbclass
> >>
> >> diff --git a/meta/classes/debian-dsc.bbclass
> >> b/meta/classes/debian-dsc.bbclass new file mode 100644
> >> index 0000000..4159e3a
> >> --- /dev/null
> >> +++ b/meta/classes/debian-dsc.bbclass
> >> @@ -0,0 +1,112 @@
> >> +# Debian .dsc backend
> >> +#
> >> +# This software is a part of ISAR.
> >> +# Copyright (c) ilbers GmbH, 2018
> >> +#
> >> +# SPDX-License-Identifier: MIT
> >> +
> >> +# List of non-standard variables added to package workspace
> >> +#
> >> +# Set by user:
> >> +#  * DSC_URI - uri link to .dsc file in upstream apt
> >> +#
> >> +# Generated automatically:
> >> +#  * DEBIAN_URI - uri link to Debian patch in upstream apt
> >> +
> >> +python __anonymous() {
> >> +    # Fetch .dsc package file
> >> +    dsc_uri = (d.getVar("DSC_URI", True) or "").split()
> >> +    if len(dsc_uri) == 0:
> >> +        return  
> > 
> > Can we not use SRC_URI and append more sources as we read .dscs? A
> > copy of the fetching code just reminds me of the broken
> > fetch/unpack in the early Isar days.
> > Because i have got my dscs in tarballs and in git ... and i am
> > behind a proxy.  
> 
> Here I consider only traditional Debian way, when.dsc file is stored 
> directly in apt with .orig tarball and .debian patch:
> 
> http://ftp.de.debian.org/debian/pool/main/h/hello/
> 
> So this class intended to work with Debian apts.
> 
> >   
> >> +    try:
> >> +        fetcher = bb.fetch2.Fetch(dsc_uri, d)
> >> +        fetcher.download()
> >> +    except bb.fetch2.BBFetchException as e:
> >> +        raise bb.build.FuncFailed(e)
> >> +
> >> +    # Open .dsc file from downloads
> >> +    dl_dir = d.getVar("DL_DIR", True)
> >> +    dsc_file = (dsc_uri[0].split(";")[0]).split("/")[-1]
> >> +    filepath = dl_dir + "/" + dsc_file
> >> +
> >> +    pv = ""
> >> +    src_uri = ""
> >> +
> >> +    # Parse .dsc file to get package details
> >> +    with open(filepath, "r") as file:
> >> +        line = file.readline()
> >> +
> >> +        while line:
> >> +            # Get package version and export PV
> >> +            if line.startswith("Version:") and not pv:
> >> +                pv = line.split(": ")[-1].rstrip()
> >> +                d.setVar("PV", pv)
> >> +
> >> +            # Get package and Debian patch arcives names
> >> +            if line.startswith("Files:") and not src_uri:
> >> +                line = file.readline()
> >> +                src = line.split(" ")[-1].rstrip()
> >> +                src_uri = (dsc_uri[0].rsplit("/", 1))[0] + "/" +
> >> src
> >> +                src_uri = src_uri + ";md5sum=" + line.split("
> >> ")[1].rstrip()
> >> +                d.setVar("SRC_URI", src_uri)
> >> +
> >> +                line = file.readline()
> >> +                debian = line.split(" ")[-1].rstrip()
> >> +                debian_uri = (dsc_uri[0].rsplit("/", 1))[0] + "/"
> >> + debian
> >> +                debian_uri = debian_uri + ";md5sum=" +
> >> line.split(" ")[1].rstrip()
> >> +                d.setVar("DEBIAN_URI", debian_uri)
> >> +
> >> +            line = file.readline()
> >> +
> >> +        file.close()
> >> +
> >> +    # Set correct path to unpacked sources
> >> +    pv_orig = (src.split("_")[-1]).split(".orig")[0]
> >> +    pn = d.getVar("PN", True)
> >> +    d.setVar("S", pn + "-" + pv_orig)
> >> +}
> >> +
> >> +python do_fetch_append() {
> >> +    debian_uri = (d.getVar("DEBIAN_URI", True) or "").split()
> >> +    if len(debian_uri) == 0:
> >> +        return
> >> +
> >> +    try:
> >> +        fetcher = bb.fetch2.Fetch(debian_uri, d)
> >> +        fetcher.download()
> >> +    except bb.fetch2.BBFetchException as e:
> >> +        raise bb.build.FuncFailed(e)
> >> +}  
> > 
> > Wow another fetcher ... same question as above.  
> 
> Upstream Debian source package contains 3 parts:
>   - .dsc
>   - .orig.tar.xz
>   - .debian.tar.xz
> 
> So:
>   - First you need to fetch .dsc to get information about remaining 2 
> artifacts.
>   - .orig.tar.xz is fetched by standard fetcher.
>   - .debian.tar.xz is fetched here.
> 
> >   
> >> +python do_unpack_append() {
> >> +    import subprocess
> >> +
> >> +    s = d.getVar("S", True)
> >> +    workdir = d.getVar("WORKDIR", True)
> >> +    debian_uri = (d.getVar("DEBIAN_URI", True) or "").split()
> >> +
> >> +    try:
> >> +        fetcher = bb.fetch2.Fetch(debian_uri, d)
> >> +        fetcher.unpack(workdir + "/" + s)  
> > 
> > fetch^3 this time with unpack  
> 
> Again:
>   - .orig.tar.xz is unpacked by standard fetcher
>   - the content of .debian.tar.xz should be unpacked into the folder 
> from the .orig.tar.xz archive
> 
> So in general, this is an attempt to collect and prepare all the 
> artifacts to make it usable with dpkg Isar class.

Sure, fully understood the moment you mentioned dsc. All i wanted to
find out whether we can use SRC_URI and the one fetcher/unapcker we
have already. Why should it be called DSC_URI only because you download
a .dsc?
The magic that an "unpack" of a .dsc involves downloading even more and
maybe unpacking that ... should go into a central place (do_unpack?) and
SRC_URI should stay SRC_URI.

i.e.: do_fetch/unpack in base.bbclass get
...
for uri in src_uri:
	if uri ends with .dsc and unpack==true:
		add new uris to final_src_uri
	else:
		add uri to final_src_uri
src_uri = final_src_uri
...

Henning

> Alex
> 
> > 
> > Henning
> >   
> >> +    except bb.fetch2.BBFetchException as e:
> >> +        raise bb.build.FuncFailed(e)
> >> +}
> >> +
> >> +# NOTE: this is workaround to serve quilt-3.0 package format
> >> support. +#       To build such package, the original tarball
> >> should be placed +#       in "../" folder. In general, this should
> >> be handled by Isar +#       core build classes, but for now this
> >> class in the only one user +#       for quilt-3.0, so for the
> >> first implementation let"s keep it here. +# Open issues:
> >> +#  - How to identify if package has quilt-3.0 source format?
> >> +#  - How to avoid copying of tarballs from downloads to working
> >> dir? +do_build_prepend() {
> >> +    dl_dir="${@ d.getVar('DL_DIR', True)}"
> >> +    workdir="${@ d.getVar('WORKDIR', True)}"
> >> +    file="${@ ((d.getVar('SRC_URI',
> >> True)).split(";")[0]).split("/")[-1]}" +
> >> +    install -m 644 $dl_dir/$file $workdir
> >> +}
> >> +
> >> +inherit dpkg  
> >  

Re: [RFC][PATCH 2/3] classes/dsc: Basic Debian .dsc backend implementation

[Jan Kiszka]

On 2018-04-23 19:27, [ext] Henning Schild wrote:
> Am Mon, 23 Apr 2018 19:11:12 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> On 04/23/2018 06:04 PM, Henning Schild wrote:
>>> Am Mon, 23 Apr 2018 16:21:45 +0300
>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>>>   
>>>> This patch introduces class to work with Debian .dsc files. The
>>>> main goal is to simplify re-building and customization of original
>>>> Debian packages.
>>>>
>>>> To rebuild package from the upstream the following recipe template
>>>> could be used:
>>>>
>>>>    DSC_URI = "http://ftp.de.debian.org/...dsc;md5sum=..."
>>>>    inherit debian-dsc
>>>>
>>>> All the remaining information bitbake will derive automatically.
>>>> This is only first implementation, so there are still several open
>>>> issues like:
>>>>   - Chose correct upstream package version depending on Debian
>>>> distro
>>>>   - quilt-3.0 source format support
>>>>   - Cross-recipe dependencies recognition
>>>>
>>>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>>>> ---
>>>>   meta/classes/debian-dsc.bbclass | 112
>>>> ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 112
>>>> insertions(+) create mode 100644 meta/classes/debian-dsc.bbclass
>>>>
>>>> diff --git a/meta/classes/debian-dsc.bbclass
>>>> b/meta/classes/debian-dsc.bbclass new file mode 100644
>>>> index 0000000..4159e3a
>>>> --- /dev/null
>>>> +++ b/meta/classes/debian-dsc.bbclass
>>>> @@ -0,0 +1,112 @@
>>>> +# Debian .dsc backend
>>>> +#
>>>> +# This software is a part of ISAR.
>>>> +# Copyright (c) ilbers GmbH, 2018
>>>> +#
>>>> +# SPDX-License-Identifier: MIT
>>>> +
>>>> +# List of non-standard variables added to package workspace
>>>> +#
>>>> +# Set by user:
>>>> +#  * DSC_URI - uri link to .dsc file in upstream apt
>>>> +#
>>>> +# Generated automatically:
>>>> +#  * DEBIAN_URI - uri link to Debian patch in upstream apt
>>>> +
>>>> +python __anonymous() {
>>>> +    # Fetch .dsc package file
>>>> +    dsc_uri = (d.getVar("DSC_URI", True) or "").split()
>>>> +    if len(dsc_uri) == 0:
>>>> +        return  
>>>
>>> Can we not use SRC_URI and append more sources as we read .dscs? A
>>> copy of the fetching code just reminds me of the broken
>>> fetch/unpack in the early Isar days.
>>> Because i have got my dscs in tarballs and in git ... and i am
>>> behind a proxy.  
>>
>> Here I consider only traditional Debian way, when.dsc file is stored 
>> directly in apt with .orig tarball and .debian patch:
>>
>> http://ftp.de.debian.org/debian/pool/main/h/hello/
>>
>> So this class intended to work with Debian apts.
>>
>>>   
>>>> +    try:
>>>> +        fetcher = bb.fetch2.Fetch(dsc_uri, d)
>>>> +        fetcher.download()
>>>> +    except bb.fetch2.BBFetchException as e:
>>>> +        raise bb.build.FuncFailed(e)
>>>> +
>>>> +    # Open .dsc file from downloads
>>>> +    dl_dir = d.getVar("DL_DIR", True)
>>>> +    dsc_file = (dsc_uri[0].split(";")[0]).split("/")[-1]
>>>> +    filepath = dl_dir + "/" + dsc_file
>>>> +
>>>> +    pv = ""
>>>> +    src_uri = ""
>>>> +
>>>> +    # Parse .dsc file to get package details
>>>> +    with open(filepath, "r") as file:
>>>> +        line = file.readline()
>>>> +
>>>> +        while line:
>>>> +            # Get package version and export PV
>>>> +            if line.startswith("Version:") and not pv:
>>>> +                pv = line.split(": ")[-1].rstrip()
>>>> +                d.setVar("PV", pv)
>>>> +
>>>> +            # Get package and Debian patch arcives names
>>>> +            if line.startswith("Files:") and not src_uri:
>>>> +                line = file.readline()
>>>> +                src = line.split(" ")[-1].rstrip()
>>>> +                src_uri = (dsc_uri[0].rsplit("/", 1))[0] + "/" +
>>>> src
>>>> +                src_uri = src_uri + ";md5sum=" + line.split("
>>>> ")[1].rstrip()
>>>> +                d.setVar("SRC_URI", src_uri)
>>>> +
>>>> +                line = file.readline()
>>>> +                debian = line.split(" ")[-1].rstrip()
>>>> +                debian_uri = (dsc_uri[0].rsplit("/", 1))[0] + "/"
>>>> + debian
>>>> +                debian_uri = debian_uri + ";md5sum=" +
>>>> line.split(" ")[1].rstrip()
>>>> +                d.setVar("DEBIAN_URI", debian_uri)
>>>> +
>>>> +            line = file.readline()
>>>> +
>>>> +        file.close()
>>>> +
>>>> +    # Set correct path to unpacked sources
>>>> +    pv_orig = (src.split("_")[-1]).split(".orig")[0]
>>>> +    pn = d.getVar("PN", True)
>>>> +    d.setVar("S", pn + "-" + pv_orig)
>>>> +}
>>>> +
>>>> +python do_fetch_append() {
>>>> +    debian_uri = (d.getVar("DEBIAN_URI", True) or "").split()
>>>> +    if len(debian_uri) == 0:
>>>> +        return
>>>> +
>>>> +    try:
>>>> +        fetcher = bb.fetch2.Fetch(debian_uri, d)
>>>> +        fetcher.download()
>>>> +    except bb.fetch2.BBFetchException as e:
>>>> +        raise bb.build.FuncFailed(e)
>>>> +}  
>>>
>>> Wow another fetcher ... same question as above.  
>>
>> Upstream Debian source package contains 3 parts:
>>   - .dsc
>>   - .orig.tar.xz
>>   - .debian.tar.xz
>>
>> So:
>>   - First you need to fetch .dsc to get information about remaining 2 
>> artifacts.
>>   - .orig.tar.xz is fetched by standard fetcher.
>>   - .debian.tar.xz is fetched here.
>>
>>>   
>>>> +python do_unpack_append() {
>>>> +    import subprocess
>>>> +
>>>> +    s = d.getVar("S", True)
>>>> +    workdir = d.getVar("WORKDIR", True)
>>>> +    debian_uri = (d.getVar("DEBIAN_URI", True) or "").split()
>>>> +
>>>> +    try:
>>>> +        fetcher = bb.fetch2.Fetch(debian_uri, d)
>>>> +        fetcher.unpack(workdir + "/" + s)  
>>>
>>> fetch^3 this time with unpack  
>>
>> Again:
>>   - .orig.tar.xz is unpacked by standard fetcher
>>   - the content of .debian.tar.xz should be unpacked into the folder 
>> from the .orig.tar.xz archive
>>
>> So in general, this is an attempt to collect and prepare all the 
>> artifacts to make it usable with dpkg Isar class.
> 
> Sure, fully understood the moment you mentioned dsc. All i wanted to
> find out whether we can use SRC_URI and the one fetcher/unapcker we
> have already. Why should it be called DSC_URI only because you download
> a .dsc?
> The magic that an "unpack" of a .dsc involves downloading even more and
> maybe unpacking that ... should go into a central place (do_unpack?) and
> SRC_URI should stay SRC_URI.
> 
> i.e.: do_fetch/unpack in base.bbclass get
> ...
> for uri in src_uri:
> 	if uri ends with .dsc and unpack==true:
> 		add new uris to final_src_uri
> 	else:
> 		add uri to final_src_uri
> src_uri = final_src_uri
> ...
> 

Ack, we should always try to design from the perspective of our users,
and there no reason for them to have separate URI lists.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [RFC][PATCH 0/3] Experiments with .dsc backend

[Alexander Smirnov]

On 04/23/2018 07:03 PM, Henning Schild wrote:
> Am Mon, 23 Apr 2018 18:26:10 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> On 04/23/2018 05:36 PM, Henning Schild wrote:
>>> Am Mon, 23 Apr 2018 16:21:43 +0300
>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>>>    
>>>> Hi everybody,
>>>>
>>>> as it was planned initially, one of the Isar feature is the
>>>> ability to rebuild original Debian packages with/without
>>>> customizations. Doing this manually in classical Yocto/OE style
>>>> requires manual setting of lots of variables and having a copy of
>>>> Debian patch (*.debian.tar.xz) for this package in your recipe's
>>>> folder. Maintenance of this system becomes very complicated.
>>>>
>>>> I performed a small research regarding bitbake capabilities and
>>>> found an interesting trick: during bitbake parsing stage (in
>>>> anonymous tasks) we could re-construct whole recipe (including
>>>> tasks and variable), and this snapshot will be stored in recipe's
>>>> dynamic context, that is used during pipeline execution.
>>>>
>>>> This series is just a PoC, lots of issues are still open, like:
>>>>    - Patching upstream packages
>>>>    - Automatic cross-recipe dependency detection to avoid
>>>> duplications in recipe and debian/control files
>>>>    - etc...
>>>>
>>>> To test this series just run:
>>>>
>>>>    $ bitbake hello
>>>>
>>>> After build, in hello workdir there will be a newly-baked identical
>>>> copy of upstream package :-)
>>>
>>> Interesting idea, but my personal feeling is that Isar already uses
>>> a lot of bitbake magic and we should maybe try to avoid introducing
>>> more where feasible.
>>>    
>>
>> Bitbake was designed for Open Embedded, so it doesn't consider our
>> Isar/Debian usecases. And roughly speaking, bitbake usage in Isar is
>> a kind of hack at all. In such questions, I'd more prefer to focus on
>> the features we need and extend bitbake, instead of applying Yocto
>> philosophy to Isar. But this also is only my personal feeling.
>>
>>>> Feedbacks, ideas, proposals are welcome!
>>>
>>> So my feedback would be that the .dsc to recipe step should be put
>>> in a helper tool in the scripts directory. People that want to
>>> modifiy upstream packages would generate an isar-recipe with the
>>> tool once. Modifications in the form of patches could go on top.
>>> Or would this limit the envisioned feature in any way?
>>
>> IMHO this is undesired behavior:
>>    - Extra step to generate new recipes
> 
> Yes, but you can see it.

For now I'd like to try keeping user interface as it is. In general, 
generated files are hard to maintain, especially when you mix them with 
repository content (generate them inside existing meta). If you generate 
these files in a separate meta - then you need to add this meta to 
build, it's also extra step. Moreover in this case the default clear 
source tree is inconsistent, generated part is missed by default.

Anyway this approach could be an option if I'll get in stuck with 
current implementation.

> 
>>    - Extra step to add these recipes to build
> 
> No, not really. You will have to mention your recipe somewhere if you
> want it in your image. What is does and how does not change that fact.
> 
> BTW, did you try overloading something like an essential package? Think
> of something that is in buildchroot and needs to be updated in there.
> (libc in the worst case ...)

Not yet, probably I posted here too early. This is only PoC for 
automatic .dsc parsing. Application use-cases are in todo. Probably it 
makes sense to come with them, so we will have real cases to discuss :-)

> 
>>    - Headache with maintenance: you touch some dsc -> you need to
>> regenerate new recipes etc...
> 
> True, but major changes in the upstream package will probably make your
> patches not apply anymore. So i suspect headache here as well, the one
> thing you do with such a tool is story copies of possibly outdated
> dscs, which indeed is a bad idea.
> So the online recipe generation is probably the way to go.
> 

In case of any Debian update you anyway have to specify new .dsc file, 
because the previous one will be removed (until you use local mirror).
So automatic support of customized upstream package.. It's a kind of "I 
want to believe" :-(

So, let me think a bit more about user interface and use-cases.

>>    - You should somehow/somewhere list of .dsc files
>> So this looks like a completely new ecosystem.
>>
>> .dsc is standard format to describe Debian source packages. So
>> anyway, to be Debian compliant Isar should support this format.
>>
>>>
>>> I also fear that this could complicate the reproducable build
>>> story.
>>
>> In general this doesn't affect reproducibility at all. The key
>> difference from typical recipe (build from tarball), that SRC_URI is
>> derived dynamically from .dsc file during parsing stage. The
>> remaining part of the pipeline stays untouched.
> 
> It affects it because we will have to redirect every fetch to a local
> cache, and it introduces fetching.

Sorry, could you please explain what you mean here? What is the 
difference of calling do_fetch and call fetcher manually?

Alex

> 
> Henning
> 
>>>
>>> Henning
>>>    
>>>> With best regards,
>>>> Alex
>>>>
>>>> Alexander Smirnov (3):
>>>>     buildchroot: Include texinfo to base system
>>>>     classes/dsc: Basic Debian .dsc backend implementation
>>>>     recipes-debian: Add example recipe to test Debian .dsc support
>>>>
>>>>    meta-isar/recipes-debian/hello/hello.bb          |  10 ++
>>>>    meta/classes/debian-dsc.bbclass                  | 112
>>>> +++++++++++++++++++++++
>>>> meta/recipes-devtools/buildchroot/buildchroot.bb |   3 +- 3 files
>>>> changed, 124 insertions(+), 1 deletion(-) create mode 100644
>>>> meta-isar/recipes-debian/hello/hello.bb create mode 100644
>>>> meta/classes/debian-dsc.bbclass
>>>>   
>>>   
> 

Re: [RFC][PATCH 0/3] Experiments with .dsc backend

[Henning Schild]

Am Fri, 27 Apr 2018 18:46:17 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> On 04/23/2018 07:03 PM, Henning Schild wrote:
> > Am Mon, 23 Apr 2018 18:26:10 +0300
> > schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >   
> >> On 04/23/2018 05:36 PM, Henning Schild wrote:  
> >>> Am Mon, 23 Apr 2018 16:21:43 +0300
> >>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >>>      
> >>>> Hi everybody,
> >>>>
> >>>> as it was planned initially, one of the Isar feature is the
> >>>> ability to rebuild original Debian packages with/without
> >>>> customizations. Doing this manually in classical Yocto/OE style
> >>>> requires manual setting of lots of variables and having a copy of
> >>>> Debian patch (*.debian.tar.xz) for this package in your recipe's
> >>>> folder. Maintenance of this system becomes very complicated.
> >>>>
> >>>> I performed a small research regarding bitbake capabilities and
> >>>> found an interesting trick: during bitbake parsing stage (in
> >>>> anonymous tasks) we could re-construct whole recipe (including
> >>>> tasks and variable), and this snapshot will be stored in recipe's
> >>>> dynamic context, that is used during pipeline execution.
> >>>>
> >>>> This series is just a PoC, lots of issues are still open, like:
> >>>>    - Patching upstream packages
> >>>>    - Automatic cross-recipe dependency detection to avoid
> >>>> duplications in recipe and debian/control files
> >>>>    - etc...
> >>>>
> >>>> To test this series just run:
> >>>>
> >>>>    $ bitbake hello
> >>>>
> >>>> After build, in hello workdir there will be a newly-baked
> >>>> identical copy of upstream package :-)  
> >>>
> >>> Interesting idea, but my personal feeling is that Isar already
> >>> uses a lot of bitbake magic and we should maybe try to avoid
> >>> introducing more where feasible.
> >>>      
> >>
> >> Bitbake was designed for Open Embedded, so it doesn't consider our
> >> Isar/Debian usecases. And roughly speaking, bitbake usage in Isar
> >> is a kind of hack at all. In such questions, I'd more prefer to
> >> focus on the features we need and extend bitbake, instead of
> >> applying Yocto philosophy to Isar. But this also is only my
> >> personal feeling. 
> >>>> Feedbacks, ideas, proposals are welcome!  
> >>>
> >>> So my feedback would be that the .dsc to recipe step should be put
> >>> in a helper tool in the scripts directory. People that want to
> >>> modifiy upstream packages would generate an isar-recipe with the
> >>> tool once. Modifications in the form of patches could go on top.
> >>> Or would this limit the envisioned feature in any way?  
> >>
> >> IMHO this is undesired behavior:
> >>    - Extra step to generate new recipes  
> > 
> > Yes, but you can see it.  
> 
> For now I'd like to try keeping user interface as it is. In general, 
> generated files are hard to maintain, especially when you mix them
> with repository content (generate them inside existing meta). If you
> generate these files in a separate meta - then you need to add this
> meta to build, it's also extra step. Moreover in this case the
> default clear source tree is inconsistent, generated part is missed
> by default.
> 
> Anyway this approach could be an option if I'll get in stuck with 
> current implementation.
> 
> >   
> >>    - Extra step to add these recipes to build  
> > 
> > No, not really. You will have to mention your recipe somewhere if
> > you want it in your image. What is does and how does not change
> > that fact.
> > 
> > BTW, did you try overloading something like an essential package?
> > Think of something that is in buildchroot and needs to be updated
> > in there. (libc in the worst case ...)  
> 
> Not yet, probably I posted here too early. This is only PoC for 
> automatic .dsc parsing. Application use-cases are in todo. Probably
> it makes sense to come with them, so we will have real cases to
> discuss :-)
> 
> >   
> >>    - Headache with maintenance: you touch some dsc -> you need to
> >> regenerate new recipes etc...  
> > 
> > True, but major changes in the upstream package will probably make
> > your patches not apply anymore. So i suspect headache here as well,
> > the one thing you do with such a tool is story copies of possibly
> > outdated dscs, which indeed is a bad idea.
> > So the online recipe generation is probably the way to go.
> >   
> 
> In case of any Debian update you anyway have to specify new .dsc
> file, because the previous one will be removed (until you use local
> mirror). So automatic support of customized upstream package.. It's a
> kind of "I want to believe" :-(
> 
> So, let me think a bit more about user interface and use-cases.
> 
> >>    - You should somehow/somewhere list of .dsc files
> >> So this looks like a completely new ecosystem.
> >>
> >> .dsc is standard format to describe Debian source packages. So
> >> anyway, to be Debian compliant Isar should support this format.
> >>  
> >>>
> >>> I also fear that this could complicate the reproducable build
> >>> story.  
> >>
> >> In general this doesn't affect reproducibility at all. The key
> >> difference from typical recipe (build from tarball), that SRC_URI
> >> is derived dynamically from .dsc file during parsing stage. The
> >> remaining part of the pipeline stays untouched.  
> > 
> > It affects it because we will have to redirect every fetch to a
> > local cache, and it introduces fetching.  
> 
> Sorry, could you please explain what you mean here? What is the 
> difference of calling do_fetch and call fetcher manually?

At the moment there is no difference. But patches that introduce
reproducability might modify do_fetch to handle caching centrally.
Maybe it will be done in another way, so just something to keep in mind.

Henning

> Alex
> 
> > 
> > Henning
> >   
> >>>
> >>> Henning
> >>>      
> >>>> With best regards,
> >>>> Alex
> >>>>
> >>>> Alexander Smirnov (3):
> >>>>     buildchroot: Include texinfo to base system
> >>>>     classes/dsc: Basic Debian .dsc backend implementation
> >>>>     recipes-debian: Add example recipe to test Debian .dsc
> >>>> support
> >>>>
> >>>>    meta-isar/recipes-debian/hello/hello.bb          |  10 ++
> >>>>    meta/classes/debian-dsc.bbclass                  | 112
> >>>> +++++++++++++++++++++++
> >>>> meta/recipes-devtools/buildchroot/buildchroot.bb |   3 +- 3 files
> >>>> changed, 124 insertions(+), 1 deletion(-) create mode 100644
> >>>> meta-isar/recipes-debian/hello/hello.bb create mode 100644
> >>>> meta/classes/debian-dsc.bbclass
> >>>>     
> >>>     
> >   

Re: [RFC][PATCH 0/3] Experiments with .dsc backend

[Henning Schild]

Today i wrote a recipe to apply a patch on openssl and something like
that could have been useful. But in fact most of the recipe is the
customization i need to do before build, and i think this series is
missing such a hook-in place.

So you fetch to orig.tar.gz and the debian and now you patch/modify the
code and debian stuff in dpkg_runbuild_prepend() and prepend a new
changelog entry.

Having .dsc support would have saved one entry in SRC_URI. So just
having done that i do not see too much value in having that, but maybe
i missed something.

Henning

Am Mon, 23 Apr 2018 16:21:43 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Hi everybody,
> 
> as it was planned initially, one of the Isar feature is the ability to
> rebuild original Debian packages with/without customizations. Doing
> this manually in classical Yocto/OE style requires manual setting of
> lots of variables and having a copy of Debian patch (*.debian.tar.xz)
> for this package in your recipe's folder. Maintenance of this system
> becomes very complicated.
> 
> I performed a small research regarding bitbake capabilities and found
> an interesting trick: during bitbake parsing stage (in anonymous
> tasks) we could re-construct whole recipe (including tasks and
> variable), and this snapshot will be stored in recipe's dynamic
> context, that is used during pipeline execution.
> 
> This series is just a PoC, lots of issues are still open, like:
>  - Patching upstream packages
>  - Automatic cross-recipe dependency detection to avoid duplications
> in recipe and debian/control files
>  - etc...
> 
> To test this series just run:
> 
>  $ bitbake hello
> 
> After build, in hello workdir there will be a newly-baked identical
> copy of upstream package :-)
> 
> Feedbacks, ideas, proposals are welcome!
> 
> With best regards,
> Alex
> 
> Alexander Smirnov (3):
>   buildchroot: Include texinfo to base system
>   classes/dsc: Basic Debian .dsc backend implementation
>   recipes-debian: Add example recipe to test Debian .dsc support
> 
>  meta-isar/recipes-debian/hello/hello.bb          |  10 ++
>  meta/classes/debian-dsc.bbclass                  | 112
> +++++++++++++++++++++++
> meta/recipes-devtools/buildchroot/buildchroot.bb |   3 +- 3 files
> changed, 124 insertions(+), 1 deletion(-) create mode 100644
> meta-isar/recipes-debian/hello/hello.bb create mode 100644
> meta/classes/debian-dsc.bbclass
> 

Re: [RFC][PATCH 2/3] classes/dsc: Basic Debian .dsc backend implementation

[Henning Schild]

Am Mon, 23 Apr 2018 16:21:45 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> This patch introduces class to work with Debian .dsc files. The
> main goal is to simplify re-building and customization of original
> Debian packages.
> 
> To rebuild package from the upstream the following recipe template
> could be used:
> 
>   DSC_URI = "http://ftp.de.debian.org/...dsc;md5sum=..."
>   inherit debian-dsc
> 
> All the remaining information bitbake will derive automatically.
> This is only first implementation, so there are still several open
> issues like:
>  - Chose correct upstream package version depending on Debian distro
>  - quilt-3.0 source format support
>  - Cross-recipe dependencies recognition
> 
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
>  meta/classes/debian-dsc.bbclass | 112
> ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 112
> insertions(+) create mode 100644 meta/classes/debian-dsc.bbclass
> 
> diff --git a/meta/classes/debian-dsc.bbclass
> b/meta/classes/debian-dsc.bbclass new file mode 100644
> index 0000000..4159e3a
> --- /dev/null
> +++ b/meta/classes/debian-dsc.bbclass
> @@ -0,0 +1,112 @@
> +# Debian .dsc backend
> +#
> +# This software is a part of ISAR.
> +# Copyright (c) ilbers GmbH, 2018
> +#
> +# SPDX-License-Identifier: MIT
> +
> +# List of non-standard variables added to package workspace
> +#
> +# Set by user:
> +#  * DSC_URI - uri link to .dsc file in upstream apt
> +#
> +# Generated automatically:
> +#  * DEBIAN_URI - uri link to Debian patch in upstream apt
> +
> +python __anonymous() {
> +    # Fetch .dsc package file
> +    dsc_uri = (d.getVar("DSC_URI", True) or "").split()
> +    if len(dsc_uri) == 0:
> +        return
> +
> +    try:
> +        fetcher = bb.fetch2.Fetch(dsc_uri, d)
> +        fetcher.download()
> +    except bb.fetch2.BBFetchException as e:
> +        raise bb.build.FuncFailed(e)
> +
> +    # Open .dsc file from downloads
> +    dl_dir = d.getVar("DL_DIR", True)
> +    dsc_file = (dsc_uri[0].split(";")[0]).split("/")[-1]
> +    filepath = dl_dir + "/" + dsc_file
> +
> +    pv = ""
> +    src_uri = ""
> +
> +    # Parse .dsc file to get package details
> +    with open(filepath, "r") as file:
> +        line = file.readline()
> +
> +        while line:
> +            # Get package version and export PV
> +            if line.startswith("Version:") and not pv:
> +                pv = line.split(": ")[-1].rstrip()
> +                d.setVar("PV", pv)
> +
> +            # Get package and Debian patch arcives names
> +            if line.startswith("Files:") and not src_uri:
> +                line = file.readline()
> +                src = line.split(" ")[-1].rstrip()
> +                src_uri = (dsc_uri[0].rsplit("/", 1))[0] + "/" + src
> +                src_uri = src_uri + ";md5sum=" + line.split("
> ")[1].rstrip()
> +                d.setVar("SRC_URI", src_uri)
> +
> +                line = file.readline()
> +                debian = line.split(" ")[-1].rstrip()
> +                debian_uri = (dsc_uri[0].rsplit("/", 1))[0] + "/" +
> debian
> +                debian_uri = debian_uri + ";md5sum=" + line.split("
> ")[1].rstrip()
> +                d.setVar("DEBIAN_URI", debian_uri)
> +
> +            line = file.readline()
> +
> +        file.close()
> +
> +    # Set correct path to unpacked sources
> +    pv_orig = (src.split("_")[-1]).split(".orig")[0]
> +    pn = d.getVar("PN", True)
> +    d.setVar("S", pn + "-" + pv_orig)
> +}
> +
> +python do_fetch_append() {
> +    debian_uri = (d.getVar("DEBIAN_URI", True) or "").split()
> +    if len(debian_uri) == 0:
> +        return
> +
> +    try:
> +        fetcher = bb.fetch2.Fetch(debian_uri, d)
> +        fetcher.download()
> +    except bb.fetch2.BBFetchException as e:
> +        raise bb.build.FuncFailed(e)
> +}
> +
> +python do_unpack_append() {
> +    import subprocess
> +
> +    s = d.getVar("S", True)
> +    workdir = d.getVar("WORKDIR", True)
> +    debian_uri = (d.getVar("DEBIAN_URI", True) or "").split()
> +
> +    try:
> +        fetcher = bb.fetch2.Fetch(debian_uri, d)
> +        fetcher.unpack(workdir + "/" + s)
> +    except bb.fetch2.BBFetchException as e:
> +        raise bb.build.FuncFailed(e)
> +}
> +
> +# NOTE: this is workaround to serve quilt-3.0 package format support.
> +#       To build such package, the original tarball should be placed
> +#       in "../" folder. In general, this should be handled by Isar
> +#       core build classes, but for now this class in the only one
> user +#       for quilt-3.0, so for the first implementation let"s
> keep it here. +# Open issues:
> +#  - How to identify if package has quilt-3.0 source format?

Had the same issue... The format is in the .dsc. We should check which
other formats change what we need to build.

> +#  - How to avoid copying of tarballs from downloads to working dir?

i fetched twice, once with unpack=false

Henning

> +do_build_prepend() {
> +    dl_dir="${@ d.getVar('DL_DIR', True)}"
> +    workdir="${@ d.getVar('WORKDIR', True)}"
> +    file="${@ ((d.getVar('SRC_URI',
> True)).split(";")[0]).split("/")[-1]}" +
> +    install -m 644 $dl_dir/$file $workdir
> +}
> +
> +inherit dpkg

Re: [RFC][PATCH 0/3] Experiments with .dsc backend

[Alexander Smirnov]

Hi Henning,

> Today i wrote a recipe to apply a patch on openssl and something like
> that could have been useful. But in fact most of the recipe is the
> customization i need to do before build, and i think this series is
> missing such a hook-in place.
>
> So you fetch to orig.tar.gz and the debian and now you patch/modify the
> code and debian stuff in dpkg_runbuild_prepend() and prepend a new
> changelog entry.
>
> Having .dsc support would have saved one entry in SRC_URI. So just
> having done that i do not see too much value in having that, but maybe
> i missed something.

As I wrote, the goal of this series was to show, how bitbake could handle 
foreign-format scripts (in this case Debian). It's not for merge.

General goal is to have possibility to partially re-build Debian. Creating 
complete recipe for Debian package is duplication. All the information 
like: SRC_URI, PV, DEPENDS, MD5 is already available in Debian apt, so what 
is the value to duplicate it? This could be an option if you need to 
rebuild one-two packages. But if there are tens of packages to rebuild, for 
example, you want to switch from 'perl' to 'microperl'. For me this looks 
like unnecessary maintaining of excessive information. AFAIK this is the 
key problem in 'meta-debian' project.

Another aspect is Debian package build dependencies from 'debian/control' 
file. Now we have to explicitely duplicate deps between recipes in DEPENDS 
variable inside recipe. So, if I could fetch package in anonymous task, 
then I could also parse its 'debian/control', then I'm able to get list of 
deps and in theory initialize DEPENDS variable in recipe. In this case I'll 
be free from maintaining these deps chains for case with lots of packages.

Alex

>
>
> Henning
>
> Am Mon, 23 Apr 2018 16:21:43 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>
>> Hi everybody,
>>
>> as it was planned initially, one of the Isar feature is the ability to
>> rebuild original Debian packages with/without customizations. Doing
>> this manually in classical Yocto/OE style requires manual setting of
>> lots of variables and having a copy of Debian patch (*.debian.tar.xz)
>> for this package in your recipe's folder. Maintenance of this system
>> becomes very complicated.
>>
>> I performed a small research regarding bitbake capabilities and found
>> an interesting trick: during bitbake parsing stage (in anonymous
>> tasks) we could re-construct whole recipe (including tasks and
>> variable), and this snapshot will be stored in recipe's dynamic
>> context, that is used during pipeline execution.
>>
>> This series is just a PoC, lots of issues are still open, like:
>> - Patching upstream packages
>> - Automatic cross-recipe dependency detection to avoid duplications
>> in recipe and debian/control files
>> - etc...
>>
>> To test this series just run:
>>
>> $ bitbake hello
>>
>> After build, in hello workdir there will be a newly-baked identical
>> copy of upstream package :-)
>>
>> Feedbacks, ideas, proposals are welcome!
>>
>> With best regards,
>> Alex
>>
>> Alexander Smirnov (3):
>> buildchroot: Include texinfo to base system
>> classes/dsc: Basic Debian .dsc backend implementation
>> recipes-debian: Add example recipe to test Debian .dsc support
>>
>> meta-isar/recipes-debian/hello/hello.bb          |  10 ++
>> meta/classes/debian-dsc.bbclass                  | 112
>> +++++++++++++++++++++++
>> meta/recipes-devtools/buildchroot/buildchroot.bb |   3 +- 3 files
>> changed, 124 insertions(+), 1 deletion(-) create mode 100644
>> meta-isar/recipes-debian/hello/hello.bb create mode 100644
>> meta/classes/debian-dsc.bbclass

Re: [RFC][PATCH 0/3] Experiments with .dsc backend

[Henning Schild]

Am Tue, 8 May 2018 21:31:46 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Hi Henning,
> 
> > Today i wrote a recipe to apply a patch on openssl and something
> > like that could have been useful. But in fact most of the recipe is
> > the customization i need to do before build, and i think this
> > series is missing such a hook-in place.
> >
> > So you fetch to orig.tar.gz and the debian and now you patch/modify
> > the code and debian stuff in dpkg_runbuild_prepend() and prepend a
> > new changelog entry.
> >
> > Having .dsc support would have saved one entry in SRC_URI. So just
> > having done that i do not see too much value in having that, but
> > maybe i missed something.  
> 
> As I wrote, the goal of this series was to show, how bitbake could
> handle foreign-format scripts (in this case Debian). It's not for
> merge.

I know, but as i had the need to rebuild/override a debian package i
shortly wrote down some insight for later reference.

> General goal is to have possibility to partially re-build Debian.
> Creating complete recipe for Debian package is duplication. All the
> information like: SRC_URI, PV, DEPENDS, MD5 is already available in
> Debian apt, so what is the value to duplicate it? This could be an
> option if you need to rebuild one-two packages. But if there are tens
> of packages to rebuild, for example, you want to switch from 'perl'
> to 'microperl'. For me this looks like unnecessary maintaining of
> excessive information. AFAIK this is the key problem in 'meta-debian'
> project.

For the partial rebuild some support might be valuable. Imagine a
recipe that mentions a .dsc an overlay directory (i.e. for patches) and
a patch against debian/. A support class behind it could now perform
the right steps.
But i guess we need a few examples to come up with a hopefully general
pattern.

> Another aspect is Debian package build dependencies from
> 'debian/control' file. Now we have to explicitely duplicate deps
> between recipes in DEPENDS variable inside recipe. So, if I could
> fetch package in anonymous task, then I could also parse its
> 'debian/control', then I'm able to get list of deps and in theory
> initialize DEPENDS variable in recipe. In this case I'll be free from
> maintaining these deps chains for case with lots of packages.

If you follow the pattern that you want to build the whole dependency
tree, you probably will end up building everything in most cases.
Another interesting question is what happens with packages that depend
on the one you modified. Say you change a library, do you rebuild all
users?
I guess changing the ABI of a lib would be a very bad idea and most
people will want to replace single packages without going up or down
the tree.

Henning

> Alex
> 
> >
> >
> > Henning
> >
> > Am Mon, 23 Apr 2018 16:21:43 +0300
> > schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >  
> >> Hi everybody,
> >>
> >> as it was planned initially, one of the Isar feature is the
> >> ability to rebuild original Debian packages with/without
> >> customizations. Doing this manually in classical Yocto/OE style
> >> requires manual setting of lots of variables and having a copy of
> >> Debian patch (*.debian.tar.xz) for this package in your recipe's
> >> folder. Maintenance of this system becomes very complicated.
> >>
> >> I performed a small research regarding bitbake capabilities and
> >> found an interesting trick: during bitbake parsing stage (in
> >> anonymous tasks) we could re-construct whole recipe (including
> >> tasks and variable), and this snapshot will be stored in recipe's
> >> dynamic context, that is used during pipeline execution.
> >>
> >> This series is just a PoC, lots of issues are still open, like:
> >> - Patching upstream packages
> >> - Automatic cross-recipe dependency detection to avoid duplications
> >> in recipe and debian/control files
> >> - etc...
> >>
> >> To test this series just run:
> >>
> >> $ bitbake hello
> >>
> >> After build, in hello workdir there will be a newly-baked identical
> >> copy of upstream package :-)
> >>
> >> Feedbacks, ideas, proposals are welcome!
> >>
> >> With best regards,
> >> Alex
> >>
> >> Alexander Smirnov (3):
> >> buildchroot: Include texinfo to base system
> >> classes/dsc: Basic Debian .dsc backend implementation
> >> recipes-debian: Add example recipe to test Debian .dsc support
> >>
> >> meta-isar/recipes-debian/hello/hello.bb          |  10 ++
> >> meta/classes/debian-dsc.bbclass                  | 112
> >> +++++++++++++++++++++++
> >> meta/recipes-devtools/buildchroot/buildchroot.bb |   3 +- 3 files
> >> changed, 124 insertions(+), 1 deletion(-) create mode 100644
> >> meta-isar/recipes-debian/hello/hello.bb create mode 100644
> >> meta/classes/debian-dsc.bbclass  
> 
> 
>