Re: [RFC][PATCH 0/3] Experiments with .dsc backend

[Henning Schild <henning.schild@siemens.com>]

Am Fri, 27 Apr 2018 18:46:17 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> On 04/23/2018 07:03 PM, Henning Schild wrote:
> > Am Mon, 23 Apr 2018 18:26:10 +0300
> > schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >   
> >> On 04/23/2018 05:36 PM, Henning Schild wrote:  
> >>> Am Mon, 23 Apr 2018 16:21:43 +0300
> >>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >>>      
> >>>> Hi everybody,
> >>>>
> >>>> as it was planned initially, one of the Isar feature is the
> >>>> ability to rebuild original Debian packages with/without
> >>>> customizations. Doing this manually in classical Yocto/OE style
> >>>> requires manual setting of lots of variables and having a copy of
> >>>> Debian patch (*.debian.tar.xz) for this package in your recipe's
> >>>> folder. Maintenance of this system becomes very complicated.
> >>>>
> >>>> I performed a small research regarding bitbake capabilities and
> >>>> found an interesting trick: during bitbake parsing stage (in
> >>>> anonymous tasks) we could re-construct whole recipe (including
> >>>> tasks and variable), and this snapshot will be stored in recipe's
> >>>> dynamic context, that is used during pipeline execution.
> >>>>
> >>>> This series is just a PoC, lots of issues are still open, like:
> >>>>    - Patching upstream packages
> >>>>    - Automatic cross-recipe dependency detection to avoid
> >>>> duplications in recipe and debian/control files
> >>>>    - etc...
> >>>>
> >>>> To test this series just run:
> >>>>
> >>>>    $ bitbake hello
> >>>>
> >>>> After build, in hello workdir there will be a newly-baked
> >>>> identical copy of upstream package :-)  
> >>>
> >>> Interesting idea, but my personal feeling is that Isar already
> >>> uses a lot of bitbake magic and we should maybe try to avoid
> >>> introducing more where feasible.
> >>>      
> >>
> >> Bitbake was designed for Open Embedded, so it doesn't consider our
> >> Isar/Debian usecases. And roughly speaking, bitbake usage in Isar
> >> is a kind of hack at all. In such questions, I'd more prefer to
> >> focus on the features we need and extend bitbake, instead of
> >> applying Yocto philosophy to Isar. But this also is only my
> >> personal feeling. 
> >>>> Feedbacks, ideas, proposals are welcome!  
> >>>
> >>> So my feedback would be that the .dsc to recipe step should be put
> >>> in a helper tool in the scripts directory. People that want to
> >>> modifiy upstream packages would generate an isar-recipe with the
> >>> tool once. Modifications in the form of patches could go on top.
> >>> Or would this limit the envisioned feature in any way?  
> >>
> >> IMHO this is undesired behavior:
> >>    - Extra step to generate new recipes  
> > 
> > Yes, but you can see it.  
> 
> For now I'd like to try keeping user interface as it is. In general, 
> generated files are hard to maintain, especially when you mix them
> with repository content (generate them inside existing meta). If you
> generate these files in a separate meta - then you need to add this
> meta to build, it's also extra step. Moreover in this case the
> default clear source tree is inconsistent, generated part is missed
> by default.
> 
> Anyway this approach could be an option if I'll get in stuck with 
> current implementation.
> 
> >   
> >>    - Extra step to add these recipes to build  
> > 
> > No, not really. You will have to mention your recipe somewhere if
> > you want it in your image. What is does and how does not change
> > that fact.
> > 
> > BTW, did you try overloading something like an essential package?
> > Think of something that is in buildchroot and needs to be updated
> > in there. (libc in the worst case ...)  
> 
> Not yet, probably I posted here too early. This is only PoC for 
> automatic .dsc parsing. Application use-cases are in todo. Probably
> it makes sense to come with them, so we will have real cases to
> discuss :-)
> 
> >   
> >>    - Headache with maintenance: you touch some dsc -> you need to
> >> regenerate new recipes etc...  
> > 
> > True, but major changes in the upstream package will probably make
> > your patches not apply anymore. So i suspect headache here as well,
> > the one thing you do with such a tool is story copies of possibly
> > outdated dscs, which indeed is a bad idea.
> > So the online recipe generation is probably the way to go.
> >   
> 
> In case of any Debian update you anyway have to specify new .dsc
> file, because the previous one will be removed (until you use local
> mirror). So automatic support of customized upstream package.. It's a
> kind of "I want to believe" :-(
> 
> So, let me think a bit more about user interface and use-cases.
> 
> >>    - You should somehow/somewhere list of .dsc files
> >> So this looks like a completely new ecosystem.
> >>
> >> .dsc is standard format to describe Debian source packages. So
> >> anyway, to be Debian compliant Isar should support this format.
> >>  
> >>>
> >>> I also fear that this could complicate the reproducable build
> >>> story.  
> >>
> >> In general this doesn't affect reproducibility at all. The key
> >> difference from typical recipe (build from tarball), that SRC_URI
> >> is derived dynamically from .dsc file during parsing stage. The
> >> remaining part of the pipeline stays untouched.  
> > 
> > It affects it because we will have to redirect every fetch to a
> > local cache, and it introduces fetching.  
> 
> Sorry, could you please explain what you mean here? What is the 
> difference of calling do_fetch and call fetcher manually?

At the moment there is no difference. But patches that introduce
reproducability might modify do_fetch to handle caching centrally.
Maybe it will be done in another way, so just something to keep in mind.

Henning

> Alex
> 
> > 
> > Henning
> >   
> >>>
> >>> Henning
> >>>      
> >>>> With best regards,
> >>>> Alex
> >>>>
> >>>> Alexander Smirnov (3):
> >>>>     buildchroot: Include texinfo to base system
> >>>>     classes/dsc: Basic Debian .dsc backend implementation
> >>>>     recipes-debian: Add example recipe to test Debian .dsc
> >>>> support
> >>>>
> >>>>    meta-isar/recipes-debian/hello/hello.bb          |  10 ++
> >>>>    meta/classes/debian-dsc.bbclass                  | 112
> >>>> +++++++++++++++++++++++
> >>>> meta/recipes-devtools/buildchroot/buildchroot.bb |   3 +- 3 files
> >>>> changed, 124 insertions(+), 1 deletion(-) create mode 100644
> >>>> meta-isar/recipes-debian/hello/hello.bb create mode 100644
> >>>> meta/classes/debian-dsc.bbclass
> >>>>     
> >>>     
> >