Re: sbuild migration problem: how to jump into SBUILD_CHROOT with build dependencies installed

[Henning Schild <henning.schild@siemens.com>]

Am Fri, 19 Aug 2022 15:05:08 +0200
schrieb Florian Bezdeka <florian.bezdeka@siemens.com>:

> On 19.08.22 12:32, Baurzhan Ismagulov wrote:
> > Hello Florian,
> > 
> > On Fri, Aug 19, 2022 at 10:36:52AM +0200, Florian Bezdeka wrote:  
> >>>> dpkg_runbuild_prepend() {
> >>>>      sudo chroot --userspec=$( id -u ):$( id -g )
> >>>> ${BUILDCHROOT_DIR} \ sh -c "autoreconf -fi"
> >>>> }
> >>>>
> >>>> In the pre-sbuild times this task was executed *after* the build
> >>>> dependencies have been installed but *before* the dpkg_runbild
> >>>> task.
> >>>>
> >>>> With sbuild:
> >>>> - There is no "install build dependencies" task anymore
> >>>>    Installation of build dependencies is done by sbuild
> >>>> internally
> >>>> - The task is now executed *before* build dependencies are
> >>>> installed
> >>>> - I still could chroot into ${SBUILD_CHROOT}, but the build
> >>>> dependencies will not be there
> >>>>
> >>>> Is there any possibility to jump into the ${SBUILD_CHROOT} after
> >>>> build dependencies are installed but before the actual packet
> >>>> build?  
> >>>
> >>> Correct way to implement this is to move autoreconf into
> >>> debian/rules.  
> >>
> >> In this simple example for sure. But: There might be recipes where
> >> debian/rules is not directly "accessable", forcing me to wait for
> >> 3rd parties to update their build process (or patch things in my
> >> layer) and all of this just because I tried to update Isar...
> >>
> >> If there is no way to jump into ${SBUILD_CHROOT} with build
> >> dependencies installed we have a (very minor) regression.  
> > 
> > I'd like to spend a few words on the background: The sbuild change
> > is not about the build tool per se; it's rather about re-using the
> > Debian package building interface. In Debian, a developer prepares
> > a standalone dsc file, uploads it to a build server, and buildd
> > produces binary packages in a clean-room environment. No
> > intermediate steps are taken in between; all necessary information
> > is contained in the source package and its dependencies.
> > 
> > Clean-room building is a requirement because otherwise binaries
> > could be e.g. linked with libraries which were not intended by the
> > developer. There are also architectural reasons like modifiability,
> > etc. Yes, removing interfaces is technically a regression but the
> > resulting functionality is more important. That is why we'd like to
> > follow the Debian way and avoid custom steps. If this is not
> > obviously applicable, we want to see the use cases and discuss them
> > in detail.  
> 
> That's all fine. Clean-room building is definitely the way to go.
> 
> > 
> > Regarding autoreconf, rebuilding the files can be done in
> > debian/rules (there is also dh_autoreconf). That said, I personally
> > prefer to generate the files during the development phase, commit
> > them to the VCS, and disable maintainer mode for building. This
> > saves time, reduces dependencies, and avoids a number of issues.
> >   
> 
> autoreconf was just a simple example. There is more in the wild which
> can not be moved into debian/rules easily. Most of the time it's not
> really a technical issue. See below.

Maybe that can be moved into the rules, even upstream in the project
you are dealing with.

You are using a prepend to an isar internal task, while there is
do_prepare_build.

I would say a _prepend was never an official interface and complaints
about API changes are invalid.

My proposal would be to run that command once and turn the result into
a patch. Patching is a complicated thing to do, i hope you are on
dpkg-gbp. Otherwise you might have to write a patch-patch to smuggle
into existing quilt infra in debian/ and hope you will not have to
patch debian/ itself.

And maybe automate that in do_prepare_build.

> > Regarding the debian/rules being inaccessible, in that case I don't
> > understand yet how you build it with Isar in the first place. I
> > suggest that you share your source package (possibly offline) and
> > we look what could be done. The idea is that any extra steps go
> > either into preparing the source package, or into package patches
> > -- so yes, patching may be required.  
> 
> 
> Sorry, I can't easily share that. But: We have a group of Isar users
> doing integration tasks. They're fetching sources from different
> teams / organization units and building product images out of them.
> 
> If the integrator is unable to update build tools (Isar is one of
> them) because of a broken build, they're stuck. Long term: Try
> pushing all the 3rd parties into the right direction, short term: Do
> not update Isar or start patching.
> 
> Pushing them into the right direction normally takes months...
> Patching is a nightmare. Skipping updates is easy and wins.

I know the pain very well and tend to agree if i try and remember one
of the days where i quickly wanted to bump the isar version under a
layer. But with a bit of distance i would say to anyone that it is
always worth following.
Anyone who decides to skip updates will simply get much worse problems
at some point in time, when the pile of work became too big to work on.

So it is likely good to sit a big change like sstate or sbuilder out
for a few weeks, to wait for bugfixes to slowly enter isar. But once it
seems stable start trying it under your layers and do the work well
before your project really needs that new version on a branch.

And if you see people playing with Isar-internal tasks, variables and
what not. Point out in your reviews that they better should stay away
from that. Its those quick hacks that later cause high cost and blame
on Isar. We could also blame bitbake for being so powerful to allow all
that, but that is also a feature we all value at times.

regards,
Henning

> So whenever we break Isar APIs, we must be careful. Keeping a bitbake
> task but changing the conditions when the task is invoked is dangerous
> and frustrates users.
> 
> Conclusion: Jumping into ${SBUILD_CHROOT} with build dependencies
> already installed is no longer possible.
> 
> Thanks for your comments!
> 
> Let's hope I can somehow solve that issue ;-)
> 
> > 
> > We've seen a number of such source packages in e.g. meta-iot2050
> > and it does require some work if the upstream follows the "let's
> > fetch 128 deps from everywhere and bootstrap ourselves" process. In
> > many cases, you want to have clean source packages yourself to
> > comply with license requirements w.r.t. rebuilding. It is also a
> > big help for the users of the package, since anyone can then
> > apt-get source pkg; sudo apt-get build-dep pkg; cd pkg-ver;
> > dpkg-buildpackage -uc -us for any given package. So, custom package
> > recipes should ideally just build the source package and leave
> > building to Debian tools.
> > 
> > With kind regards,
> > Baurzhan
> >   
>