From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6547633542169362432 X-Received: by 2002:a2e:9e18:: with SMTP id e24-v6mr166513ljk.12.1524843986892; Fri, 27 Apr 2018 08:46:26 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a19:181f:: with SMTP id o31-v6ls205578lfi.6.gmail; Fri, 27 Apr 2018 08:46:26 -0700 (PDT) X-Google-Smtp-Source: AB8JxZqlqyhb+Xcl2hkmFvjVAZBN0FiL1x1oILladpDYe+jINPCfu7O6c0IACrcJItXe5oUdC+oW X-Received: by 2002:a19:e954:: with SMTP id g81-v6mr153243lfh.31.1524843986157; Fri, 27 Apr 2018 08:46:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1524843986; cv=none; d=google.com; s=arc-20160816; b=O645Pa6UTZXE+Bkg9RVsaKW7rRWQh21sHv8lzxDfGzfuJTHoWPB2SiFs7WxL7Q6g51 lEQgAQGpIw/eDC++feQbtW7hhcs7MFB1fHxt8Z++jUzjhIaWRohWb+olqRo2+tDDR4Nf J0+KJX2JVJX/CiQU1bl7xV7ZOOBiucFwCfWBgCzU514DzEZ1g2Bwc2H7hIdUrMdfTYLW RZtTzXBELzEW2kOSNvFyEPv3qvyXI6GnhMx2XuGzyNx1pjng3UOJkT94YwmBzuxcPHTP 8fkqn/6r8gzfx01EeHXpqm2vjsAXwj9dBJhu0p8ejdrCxWXkDS9dqF+llD7vyotqB8dv d1wQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject :arc-authentication-results; bh=ZlB0Wqy4hH2XCV2k2pfrCF0XC0tqG1HQ90f5QQVlwSA=; b=lnVnW8akKeex14DEGHfI0+Z/aQ6CcjBUjAHqD6l/PE3FiexnlJ2bYscdZHM7fhARSD 1CQsDjDm+5soxGotIhfSxrt439my1xB+iDaCGLcM+iZcRfkji3vbcs2SqBsQN7JTxZks TKSs9IsnF7QPBjHio/qZE5rQ0X+h47PQqb2uQvKMojGSPCnrDrTQ2MBIuAjnBUCXU2eN UcB8xw2J5jiXBjg3TDJdN3aCQKmTHTTTYyXGfN2GOU4ZT8ybRO70I2LlDMqsV5q6OYAr b4si55ywHexZSfabXZ3RIp0BCyTaNepA5hdKbz8zcXx/cZPEcPa39j/uxedGD3m9N2QN DoiQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Return-Path: Received: from aqmola.ilbers.de (aqmola.ilbers.de. [85.214.62.211]) by gmr-mx.google.com with ESMTPS id n5-v6si52317ljj.5.2018.04.27.08.46.25 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 27 Apr 2018 08:46:25 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) client-ip=85.214.62.211; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Received: from [10.0.2.15] ([188.227.110.165]) (authenticated bits=0) by aqmola.ilbers.de (8.14.4/8.14.4/Debian-4+deb7u1) with ESMTP id w3RFkMJc032228 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Fri, 27 Apr 2018 17:46:24 +0200 Subject: Re: [RFC][PATCH 0/3] Experiments with .dsc backend To: Henning Schild Cc: isar-users@googlegroups.com References: <20180423132146.14743-1-asmirnov@ilbers.de> <20180423163616.6f038064@mmd1pvb1c.ad001.siemens.net> <52a4801e-ef31-bdc8-5a77-b2def3b90503@ilbers.de> <20180423180307.111a3f62@mmd1pvb1c.ad001.siemens.net> From: Alexander Smirnov Message-ID: Date: Fri, 27 Apr 2018 18:46:17 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: <20180423180307.111a3f62@mmd1pvb1c.ad001.siemens.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: 1naqYnRoXF5h On 04/23/2018 07:03 PM, Henning Schild wrote: > Am Mon, 23 Apr 2018 18:26:10 +0300 > schrieb Alexander Smirnov : > >> On 04/23/2018 05:36 PM, Henning Schild wrote: >>> Am Mon, 23 Apr 2018 16:21:43 +0300 >>> schrieb Alexander Smirnov : >>> >>>> Hi everybody, >>>> >>>> as it was planned initially, one of the Isar feature is the >>>> ability to rebuild original Debian packages with/without >>>> customizations. Doing this manually in classical Yocto/OE style >>>> requires manual setting of lots of variables and having a copy of >>>> Debian patch (*.debian.tar.xz) for this package in your recipe's >>>> folder. Maintenance of this system becomes very complicated. >>>> >>>> I performed a small research regarding bitbake capabilities and >>>> found an interesting trick: during bitbake parsing stage (in >>>> anonymous tasks) we could re-construct whole recipe (including >>>> tasks and variable), and this snapshot will be stored in recipe's >>>> dynamic context, that is used during pipeline execution. >>>> >>>> This series is just a PoC, lots of issues are still open, like: >>>> - Patching upstream packages >>>> - Automatic cross-recipe dependency detection to avoid >>>> duplications in recipe and debian/control files >>>> - etc... >>>> >>>> To test this series just run: >>>> >>>> $ bitbake hello >>>> >>>> After build, in hello workdir there will be a newly-baked identical >>>> copy of upstream package :-) >>> >>> Interesting idea, but my personal feeling is that Isar already uses >>> a lot of bitbake magic and we should maybe try to avoid introducing >>> more where feasible. >>> >> >> Bitbake was designed for Open Embedded, so it doesn't consider our >> Isar/Debian usecases. And roughly speaking, bitbake usage in Isar is >> a kind of hack at all. In such questions, I'd more prefer to focus on >> the features we need and extend bitbake, instead of applying Yocto >> philosophy to Isar. But this also is only my personal feeling. >> >>>> Feedbacks, ideas, proposals are welcome! >>> >>> So my feedback would be that the .dsc to recipe step should be put >>> in a helper tool in the scripts directory. People that want to >>> modifiy upstream packages would generate an isar-recipe with the >>> tool once. Modifications in the form of patches could go on top. >>> Or would this limit the envisioned feature in any way? >> >> IMHO this is undesired behavior: >> - Extra step to generate new recipes > > Yes, but you can see it. For now I'd like to try keeping user interface as it is. In general, generated files are hard to maintain, especially when you mix them with repository content (generate them inside existing meta). If you generate these files in a separate meta - then you need to add this meta to build, it's also extra step. Moreover in this case the default clear source tree is inconsistent, generated part is missed by default. Anyway this approach could be an option if I'll get in stuck with current implementation. > >> - Extra step to add these recipes to build > > No, not really. You will have to mention your recipe somewhere if you > want it in your image. What is does and how does not change that fact. > > BTW, did you try overloading something like an essential package? Think > of something that is in buildchroot and needs to be updated in there. > (libc in the worst case ...) Not yet, probably I posted here too early. This is only PoC for automatic .dsc parsing. Application use-cases are in todo. Probably it makes sense to come with them, so we will have real cases to discuss :-) > >> - Headache with maintenance: you touch some dsc -> you need to >> regenerate new recipes etc... > > True, but major changes in the upstream package will probably make your > patches not apply anymore. So i suspect headache here as well, the one > thing you do with such a tool is story copies of possibly outdated > dscs, which indeed is a bad idea. > So the online recipe generation is probably the way to go. > In case of any Debian update you anyway have to specify new .dsc file, because the previous one will be removed (until you use local mirror). So automatic support of customized upstream package.. It's a kind of "I want to believe" :-( So, let me think a bit more about user interface and use-cases. >> - You should somehow/somewhere list of .dsc files >> So this looks like a completely new ecosystem. >> >> .dsc is standard format to describe Debian source packages. So >> anyway, to be Debian compliant Isar should support this format. >> >>> >>> I also fear that this could complicate the reproducable build >>> story. >> >> In general this doesn't affect reproducibility at all. The key >> difference from typical recipe (build from tarball), that SRC_URI is >> derived dynamically from .dsc file during parsing stage. The >> remaining part of the pipeline stays untouched. > > It affects it because we will have to redirect every fetch to a local > cache, and it introduces fetching. Sorry, could you please explain what you mean here? What is the difference of calling do_fetch and call fetcher manually? Alex > > Henning > >>> >>> Henning >>> >>>> With best regards, >>>> Alex >>>> >>>> Alexander Smirnov (3): >>>> buildchroot: Include texinfo to base system >>>> classes/dsc: Basic Debian .dsc backend implementation >>>> recipes-debian: Add example recipe to test Debian .dsc support >>>> >>>> meta-isar/recipes-debian/hello/hello.bb | 10 ++ >>>> meta/classes/debian-dsc.bbclass | 112 >>>> +++++++++++++++++++++++ >>>> meta/recipes-devtools/buildchroot/buildchroot.bb | 3 +- 3 files >>>> changed, 124 insertions(+), 1 deletion(-) create mode 100644 >>>> meta-isar/recipes-debian/hello/hello.bb create mode 100644 >>>> meta/classes/debian-dsc.bbclass >>>> >>> >