Re: [RFC][PATCH 0/3] Experiments with .dsc backend

[Alexander Smirnov <asmirnov@ilbers.de>]

On 04/23/2018 07:03 PM, Henning Schild wrote:
> Am Mon, 23 Apr 2018 18:26:10 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> On 04/23/2018 05:36 PM, Henning Schild wrote:
>>> Am Mon, 23 Apr 2018 16:21:43 +0300
>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>>>    
>>>> Hi everybody,
>>>>
>>>> as it was planned initially, one of the Isar feature is the
>>>> ability to rebuild original Debian packages with/without
>>>> customizations. Doing this manually in classical Yocto/OE style
>>>> requires manual setting of lots of variables and having a copy of
>>>> Debian patch (*.debian.tar.xz) for this package in your recipe's
>>>> folder. Maintenance of this system becomes very complicated.
>>>>
>>>> I performed a small research regarding bitbake capabilities and
>>>> found an interesting trick: during bitbake parsing stage (in
>>>> anonymous tasks) we could re-construct whole recipe (including
>>>> tasks and variable), and this snapshot will be stored in recipe's
>>>> dynamic context, that is used during pipeline execution.
>>>>
>>>> This series is just a PoC, lots of issues are still open, like:
>>>>    - Patching upstream packages
>>>>    - Automatic cross-recipe dependency detection to avoid
>>>> duplications in recipe and debian/control files
>>>>    - etc...
>>>>
>>>> To test this series just run:
>>>>
>>>>    $ bitbake hello
>>>>
>>>> After build, in hello workdir there will be a newly-baked identical
>>>> copy of upstream package :-)
>>>
>>> Interesting idea, but my personal feeling is that Isar already uses
>>> a lot of bitbake magic and we should maybe try to avoid introducing
>>> more where feasible.
>>>    
>>
>> Bitbake was designed for Open Embedded, so it doesn't consider our
>> Isar/Debian usecases. And roughly speaking, bitbake usage in Isar is
>> a kind of hack at all. In such questions, I'd more prefer to focus on
>> the features we need and extend bitbake, instead of applying Yocto
>> philosophy to Isar. But this also is only my personal feeling.
>>
>>>> Feedbacks, ideas, proposals are welcome!
>>>
>>> So my feedback would be that the .dsc to recipe step should be put
>>> in a helper tool in the scripts directory. People that want to
>>> modifiy upstream packages would generate an isar-recipe with the
>>> tool once. Modifications in the form of patches could go on top.
>>> Or would this limit the envisioned feature in any way?
>>
>> IMHO this is undesired behavior:
>>    - Extra step to generate new recipes
> 
> Yes, but you can see it.

For now I'd like to try keeping user interface as it is. In general, 
generated files are hard to maintain, especially when you mix them with 
repository content (generate them inside existing meta). If you generate 
these files in a separate meta - then you need to add this meta to 
build, it's also extra step. Moreover in this case the default clear 
source tree is inconsistent, generated part is missed by default.

Anyway this approach could be an option if I'll get in stuck with 
current implementation.

> 
>>    - Extra step to add these recipes to build
> 
> No, not really. You will have to mention your recipe somewhere if you
> want it in your image. What is does and how does not change that fact.
> 
> BTW, did you try overloading something like an essential package? Think
> of something that is in buildchroot and needs to be updated in there.
> (libc in the worst case ...)

Not yet, probably I posted here too early. This is only PoC for 
automatic .dsc parsing. Application use-cases are in todo. Probably it 
makes sense to come with them, so we will have real cases to discuss :-)

> 
>>    - Headache with maintenance: you touch some dsc -> you need to
>> regenerate new recipes etc...
> 
> True, but major changes in the upstream package will probably make your
> patches not apply anymore. So i suspect headache here as well, the one
> thing you do with such a tool is story copies of possibly outdated
> dscs, which indeed is a bad idea.
> So the online recipe generation is probably the way to go.
> 

In case of any Debian update you anyway have to specify new .dsc file, 
because the previous one will be removed (until you use local mirror).
So automatic support of customized upstream package.. It's a kind of "I 
want to believe" :-(

So, let me think a bit more about user interface and use-cases.

>>    - You should somehow/somewhere list of .dsc files
>> So this looks like a completely new ecosystem.
>>
>> .dsc is standard format to describe Debian source packages. So
>> anyway, to be Debian compliant Isar should support this format.
>>
>>>
>>> I also fear that this could complicate the reproducable build
>>> story.
>>
>> In general this doesn't affect reproducibility at all. The key
>> difference from typical recipe (build from tarball), that SRC_URI is
>> derived dynamically from .dsc file during parsing stage. The
>> remaining part of the pipeline stays untouched.
> 
> It affects it because we will have to redirect every fetch to a local
> cache, and it introduces fetching.

Sorry, could you please explain what you mean here? What is the 
difference of calling do_fetch and call fetcher manually?

Alex

> 
> Henning
> 
>>>
>>> Henning
>>>    
>>>> With best regards,
>>>> Alex
>>>>
>>>> Alexander Smirnov (3):
>>>>     buildchroot: Include texinfo to base system
>>>>     classes/dsc: Basic Debian .dsc backend implementation
>>>>     recipes-debian: Add example recipe to test Debian .dsc support
>>>>
>>>>    meta-isar/recipes-debian/hello/hello.bb          |  10 ++
>>>>    meta/classes/debian-dsc.bbclass                  | 112
>>>> +++++++++++++++++++++++
>>>> meta/recipes-devtools/buildchroot/buildchroot.bb |   3 +- 3 files
>>>> changed, 124 insertions(+), 1 deletion(-) create mode 100644
>>>> meta-isar/recipes-debian/hello/hello.bb create mode 100644
>>>> meta/classes/debian-dsc.bbclass
>>>>   
>>>   
>