Re: Question related to Sbuild CLEAN_SOURCE

Re: Question related to Sbuild CLEAN_SOURCE

[quirin.gylstorff]

Hi,

thanks for the fast answer - I will forward the mail to the
ISAR mailing list.

On 2/8/22 21:47, Johannes Schauer Marin Rodrigues wrote:
> Hi,
> 
> please keep me in CC as I'm not subscribed to debian-user@l.d.o.
> 
> Quoting quirin.gylstorff@siemens.com (2022-02-08 15:40:40)
>> we are currently testing the integration of sbuild in ISAR[1] and have a
>> question related to the CLEAN_SOURCE behavior.
>>
>>
>> "When running sbuild from within an unpacked source tree, run the
>> 'clean' target before generating the source package. This might require
>> some of the build dependencies necessary for running the 'clean' target
>> to be installed on the host machine. Only disable if you start from a
>> clean checkout and you know what you are doing."[2]
>>
>>
>> As noted in [2] we observe that we need dh-<helper name> in the host
>> environment to execute the dh clean -with <helper name>.
>>
>> Our questions are:
>>
>> - Is there are reason sbuild executes clean outside of the chroot?
> 
> Yes because if you don't know what you are doing (as it says in the manpage you
> cite) and you run sbuild from within an unpacked dirty source tree without
> running the clean target, then you might be building something that is not the
> source package you expect to be built.
> 
>> - How does the buildd infrastructure deal with this behavior?
> 
> The buildd infrastructure doesn't run sbuild from within an unpacked source
> tree so it will not run the clean target outside the chroot and thus doesn't
> need to have the necessary dependencies for running the clean target installed.
> 
> The fact that sbuild is able to run from inside an unpacked source tree is a
> convenience feature which allows you to run sbuild without having to first
> create the source package. But the actual input to sbuild is the source package
> (the dsc). Don't use sbuild to create a source package. The source package is
> the input to sbuild. The output of sbuild are binary packages, not source
> packages.
> 
> Thanks!
> 
Best regards,

Quirin

Re: Question related to Sbuild CLEAN_SOURCE

[Baurzhan Ismagulov]

On Wed, Feb 09, 2022 at 11:36:09AM +0100, quirin.gylstorff@siemens.com wrote:
> > > - Is there are reason sbuild executes clean outside of the chroot?

It would be interesting to hear what kind of issues you had due to that.


> > But the actual input to sbuild is the source package
> > (the dsc). Don't use sbuild to create a source package. The source package is
> > the input to sbuild. The output of sbuild are binary packages, not source
> > packages.

It's interesting that he is mentioning it; we are currently performing this
exact split for meta-iot2050.


With kind regards,
Baurzhan.

Re: Question related to Sbuild CLEAN_SOURCE

[quirin.gylstorff]

On 2/9/22 12:52, Baurzhan Ismagulov wrote:
> On Wed, Feb 09, 2022 at 11:36:09AM +0100, quirin.gylstorff@siemens.com wrote:
>>>> - Is there are reason sbuild executes clean outside of the chroot?
> 
> It would be interesting to hear what kind of issues you had due to that.
> 

If we rebuild a debian package with dpkg-gbp which uses for example 
`dh-lua`  (swupdate) the build will fail due to the missing installation 
of dh-lua in the host environment. The solution to this is to install 
dh-* in the build environment.

```
gbp:info: Performing the build
dpkg-source: info: using patch list from debian/patches/series
dpkg-source: info: applying use-gcc-compiler.diff
dh clean --with lua
dh: error: unable to load addon lua: Can't locate 
Debian/Debhelper/Sequence/lua.pm in @INC (you may need to install the 
Debian::Debhelper::Sequence::lua module) (@INC contains: /etc/perl 
/usr/local/lib/x86_64-linux-gnu/perl/5.32.1 /usr/local/share/perl/5.32.1 
/usr/lib/x86_64-linux-gnu/perl5/5.32 /usr/share/perl5 
/usr/lib/x86_64-linux-gnu/perl-base /usr/lib/x86_64-linux-gnu/perl/5.32 
/usr/share/perl/5.32 /usr/local/lib/site_perl) at (eval 13) line 1.
BEGIN failed--compilation aborted at (eval 13) line 1.
```

You can fix it with documentation but it needs to be addressed.

A project where this can be tested is isar-cip-core with the current 
preintegrated sbuild[1].
[1]: https://gitlab.com/Quirin.Gy/isar-cip-core/-/tree/test/sbuild
> 
>>> But the actual input to sbuild is the source package
>>> (the dsc). Don't use sbuild to create a source package. The source package is
>>> the input to sbuild. The output of sbuild are binary packages, not source
>>> packages.
> 
> It's interesting that he is mentioning it; we are currently performing this
> exact split for meta-iot2050.
> 
> 
> With kind regards,
> Baurzhan.
Kind regards,
Quirin

Re: Question related to Sbuild CLEAN_SOURCE

[Jan Kiszka]

On 09.02.22 16:14, quirin.gylstorff@siemens.com wrote:
> 
> 
> On 2/9/22 12:52, Baurzhan Ismagulov wrote:
>> On Wed, Feb 09, 2022 at 11:36:09AM +0100, quirin.gylstorff@siemens.com
>> wrote:
>>>>> - Is there are reason sbuild executes clean outside of the chroot?
>>
>> It would be interesting to hear what kind of issues you had due to that.
>>
> 
> If we rebuild a debian package with dpkg-gbp which uses for example
> `dh-lua`  (swupdate) the build will fail due to the missing installation
> of dh-lua in the host environment. The solution to this is to install
> dh-* in the build environment.
> 
> ```
> gbp:info: Performing the build
> dpkg-source: info: using patch list from debian/patches/series
> dpkg-source: info: applying use-gcc-compiler.diff
> dh clean --with lua
> dh: error: unable to load addon lua: Can't locate
> Debian/Debhelper/Sequence/lua.pm in @INC (you may need to install the
> Debian::Debhelper::Sequence::lua module) (@INC contains: /etc/perl
> /usr/local/lib/x86_64-linux-gnu/perl/5.32.1 /usr/local/share/perl/5.32.1
> /usr/lib/x86_64-linux-gnu/perl5/5.32 /usr/share/perl5
> /usr/lib/x86_64-linux-gnu/perl-base /usr/lib/x86_64-linux-gnu/perl/5.32
> /usr/share/perl/5.32 /usr/local/lib/site_perl) at (eval 13) line 1.
> BEGIN failed--compilation aborted at (eval 13) line 1.
> ```
> 
> You can fix it with documentation but it needs to be addressed.
> 

Resolving that concrete issue is one thing, but using sbuild in its
"convenience feature" mode for Isar is another and not that easily fixable.

> A project where this can be tested is isar-cip-core with the current
> preintegrated sbuild[1].
> [1]: https://gitlab.com/Quirin.Gy/isar-cip-core/-/tree/test/sbuild
>>
>>>> But the actual input to sbuild is the source package
>>>> (the dsc). Don't use sbuild to create a source package. The source
>>>> package is
>>>> the input to sbuild. The output of sbuild are binary packages, not
>>>> source
>>>> packages.
>>
>> It's interesting that he is mentioning it; we are currently performing
>> this
>> exact split for meta-iot2050.

What will that mean for layers? Patching Debian packages "ad-hoc" is not
an uncommon use case anymore. In fact, bringing Debianization in via
Isar/bitbake is even more common and would be equally affected.

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

Re: Question related to Sbuild CLEAN_SOURCE

[Uladzimir Bely]

In the email from Wednesday, 9 February 2022 18:14:22 +03 user 
quirin.gylstorff@siemens.com wrote:
> On 2/9/22 12:52, Baurzhan Ismagulov wrote:
> > On Wed, Feb 09, 2022 at 11:36:09AM +0100, quirin.gylstorff@siemens.com 
wrote:
> >>>> - Is there are reason sbuild executes clean outside of the chroot?
> > 
> > It would be interesting to hear what kind of issues you had due to that.
> 
> If we rebuild a debian package with dpkg-gbp which uses for example
> `dh-lua`  (swupdate) the build will fail due to the missing installation
> of dh-lua in the host environment. The solution to this is to install
> dh-* in the build environment.
> 
> ```
> gbp:info: Performing the build
> dpkg-source: info: using patch list from debian/patches/series
> dpkg-source: info: applying use-gcc-compiler.diff
> dh clean --with lua
> dh: error: unable to load addon lua: Can't locate
> Debian/Debhelper/Sequence/lua.pm in @INC (you may need to install the
> Debian::Debhelper::Sequence::lua module) (@INC contains: /etc/perl
> /usr/local/lib/x86_64-linux-gnu/perl/5.32.1 /usr/local/share/perl/5.32.1
> /usr/lib/x86_64-linux-gnu/perl5/5.32 /usr/share/perl5
> /usr/lib/x86_64-linux-gnu/perl-base /usr/lib/x86_64-linux-gnu/perl/5.32
> /usr/share/perl/5.32 /usr/local/lib/site_perl) at (eval 13) line 1.
> BEGIN failed--compilation aborted at (eval 13) line 1.
> ```
> 
> You can fix it with documentation but it needs to be addressed.
> 
> A project where this can be tested is isar-cip-core with the current
> preintegrated sbuild[1].
> [1]: https://gitlab.com/Quirin.Gy/isar-cip-core/-/tree/test/sbuild

Hello.
I've tried this repository (master branch) on my side.
It currently builds OK with Isar revision 33b5aa65 (default mentioned kas-
cip.yml), it works with current isar/next as well. Also, I successfully built 
it wth latest isar/ubely/sbuild revision.

Maybe there are some more specific steps to reproduce the issue you are 
talking about that I could try to check on my side?

> 
> >>> But the actual input to sbuild is the source package
> >>> (the dsc). Don't use sbuild to create a source package. The source
> >>> package is the input to sbuild. The output of sbuild are binary
> >>> packages, not source packages.
> > 
> > It's interesting that he is mentioning it; we are currently performing
> > this
> > exact split for meta-iot2050.
> > 
> > 
> > With kind regards,
> > Baurzhan.
> 
> Kind regards,
> Quirin


-- 
Uladzimir Bely

Re: Question related to Sbuild CLEAN_SOURCE

[quirin.gylstorff]

On 2/10/22 09:47, Uladzimir Bely wrote:
> In the email from Wednesday, 9 February 2022 18:14:22 +03 user
> quirin.gylstorff@siemens.com wrote:
>> On 2/9/22 12:52, Baurzhan Ismagulov wrote:
>>> On Wed, Feb 09, 2022 at 11:36:09AM +0100, quirin.gylstorff@siemens.com
> wrote:
>>>>>> - Is there are reason sbuild executes clean outside of the chroot?
>>>
>>> It would be interesting to hear what kind of issues you had due to that.
>>
>> If we rebuild a debian package with dpkg-gbp which uses for example
>> `dh-lua`  (swupdate) the build will fail due to the missing installation
>> of dh-lua in the host environment. The solution to this is to install
>> dh-* in the build environment.
>>
>> ```
>> gbp:info: Performing the build
>> dpkg-source: info: using patch list from debian/patches/series
>> dpkg-source: info: applying use-gcc-compiler.diff
>> dh clean --with lua
>> dh: error: unable to load addon lua: Can't locate
>> Debian/Debhelper/Sequence/lua.pm in @INC (you may need to install the
>> Debian::Debhelper::Sequence::lua module) (@INC contains: /etc/perl
>> /usr/local/lib/x86_64-linux-gnu/perl/5.32.1 /usr/local/share/perl/5.32.1
>> /usr/lib/x86_64-linux-gnu/perl5/5.32 /usr/share/perl5
>> /usr/lib/x86_64-linux-gnu/perl-base /usr/lib/x86_64-linux-gnu/perl/5.32
>> /usr/share/perl/5.32 /usr/local/lib/site_perl) at (eval 13) line 1.
>> BEGIN failed--compilation aborted at (eval 13) line 1.
>> ```
>>
>> You can fix it with documentation but it needs to be addressed.
>>
>> A project where this can be tested is isar-cip-core with the current
>> preintegrated sbuild[1].
>> [1]: https://gitlab.com/Quirin.Gy/isar-cip-core/-/tree/test/sbuild
> 
> Hello.
> I've tried this repository (master branch) on my side.
> It currently builds OK with Isar revision 33b5aa65 (default mentioned kas-
> cip.yml), it works with current isar/next as well. Also, I successfully built
> it wth latest isar/ubely/sbuild revision.
> 
> Maybe there are some more specific steps to reproduce the issue you are
> talking about that I could try to check on my side?
> 

Ah sorry,

you need to add the SWUpdate option to the build. The kas command would be:

```
/kas-container --isar build 
kas-cip.yml:kas/board/qemu-amd64.yml:kas/opt/ebg-swu.yml
```

or you build SWUpdate from the devshell.

Kind regards
Quirin

Re: Question related to Sbuild CLEAN_SOURCE

[Baurzhan Ismagulov]

On Wed, Feb 09, 2022 at 04:37:55PM +0100, Jan Kiszka wrote:
> Resolving that concrete issue is one thing, but using sbuild in its
> "convenience feature" mode for Isar is another and not that easily fixable.
...
> What will that mean for layers? Patching Debian packages "ad-hoc" is not
> an uncommon use case anymore. In fact, bringing Debianization in via
> Isar/bitbake is even more common and would be equally affected.

I'm not sure what you mean with "sbuild convenience feature mode", could you
elaborate a bit?

For layers, I see only advantages. Debian strictly separates source and binary
packages. We build source packages. sbuild enforces clean-room environment,
which provides reproducibility through specifying all necessary steps in
debian/rules.

Ad-hoc patching and debianization on the fly are not affected; it becomes just
a matter of providing clean Debian source packages (the state after patching /
debianization) automatically by Isar.

With kind regards,
Baurzhan.

Re: Question related to Sbuild CLEAN_SOURCE

[Jan Kiszka]

On 10.02.22 10:59, Baurzhan Ismagulov wrote:
> On Wed, Feb 09, 2022 at 04:37:55PM +0100, Jan Kiszka wrote:
>> Resolving that concrete issue is one thing, but using sbuild in its
>> "convenience feature" mode for Isar is another and not that easily fixable.
> ...
>> What will that mean for layers? Patching Debian packages "ad-hoc" is not
>> an uncommon use case anymore. In fact, bringing Debianization in via
>> Isar/bitbake is even more common and would be equally affected.
> 
> I'm not sure what you mean with "sbuild convenience feature mode", could you
> elaborate a bit?

Read the reply of Josch that Quirin got.

> 
> For layers, I see only advantages. Debian strictly separates source and binary
> packages. We build source packages. sbuild enforces clean-room environment,
> which provides reproducibility through specifying all necessary steps in
> debian/rules.
> 
> Ad-hoc patching and debianization on the fly are not affected; it becomes just
> a matter of providing clean Debian source packages (the state after patching /
> debianization) automatically by Isar.

OK, then we should switch to that and test it. I'd like to see the full
picture, also in context of more layers. It was very helpful, also for
the design discussions, that you converted 2 (besides Isar itself), but
the problem showed up right with the 3rd one.

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

Re: Question related to Sbuild CLEAN_SOURCE

[Baurzhan Ismagulov]

On Thu, Feb 10, 2022 at 12:27:44PM +0100, Jan Kiszka wrote:
> > I'm not sure what you mean with "sbuild convenience feature mode", could you
> > elaborate a bit?
> 
> Read the reply of Josch that Quirin got.

Thanks, I had overlooked that specific wording there.


> OK, then we should switch to that and test it. I'd like to see the full
> picture, also in context of more layers. It was very helpful, also for
> the design discussions, that you converted 2 (besides Isar itself), but
> the problem showed up right with the 3rd one.

We had workarounded this issue also for the Isar kernel (KDIR not set during
cleaning). As cleaning is not necessary when passing a dsc, we can eliminate
the issue as a class and also avoid potential build interference through
garbage in the source directory (assuming the source package is done correctly)
by passing the dsc to sbuild -- currently evaluating.


With kind regards,
Baurzhan.

Re: Question related to Sbuild CLEAN_SOURCE

[Uladzimir Bely]

In the email from Thursday, 10 February 2022 12:10:47 +03 user 
quirin.gylstorff@siemens.com wrote:
> On 2/10/22 09:47, Uladzimir Bely wrote:
> > In the email from Wednesday, 9 February 2022 18:14:22 +03 user
> > 
> > quirin.gylstorff@siemens.com wrote:
> >> On 2/9/22 12:52, Baurzhan Ismagulov wrote:
> >>> On Wed, Feb 09, 2022 at 11:36:09AM +0100, quirin.gylstorff@siemens.com
> > 
> > wrote:
> >>>>>> - Is there are reason sbuild executes clean outside of the chroot?
> >>> 
> >>> It would be interesting to hear what kind of issues you had due to that.
> >> 
> >> If we rebuild a debian package with dpkg-gbp which uses for example
> >> `dh-lua`  (swupdate) the build will fail due to the missing installation
> >> of dh-lua in the host environment. The solution to this is to install
> >> dh-* in the build environment.
> >> 
> >> ```
> >> gbp:info: Performing the build
> >> dpkg-source: info: using patch list from debian/patches/series
> >> dpkg-source: info: applying use-gcc-compiler.diff
> >> dh clean --with lua
> >> dh: error: unable to load addon lua: Can't locate
> >> Debian/Debhelper/Sequence/lua.pm in @INC (you may need to install the
> >> Debian::Debhelper::Sequence::lua module) (@INC contains: /etc/perl
> >> /usr/local/lib/x86_64-linux-gnu/perl/5.32.1 /usr/local/share/perl/5.32.1
> >> /usr/lib/x86_64-linux-gnu/perl5/5.32 /usr/share/perl5
> >> /usr/lib/x86_64-linux-gnu/perl-base /usr/lib/x86_64-linux-gnu/perl/5.32
> >> /usr/share/perl/5.32 /usr/local/lib/site_perl) at (eval 13) line 1.
> >> BEGIN failed--compilation aborted at (eval 13) line 1.
> >> ```
> >> 
> >> You can fix it with documentation but it needs to be addressed.
> >> 
> >> A project where this can be tested is isar-cip-core with the current
> >> preintegrated sbuild[1].
> >> [1]: https://gitlab.com/Quirin.Gy/isar-cip-core/-/tree/test/sbuild
> > 
> > Hello.
> > I've tried this repository (master branch) on my side.
> > It currently builds OK with Isar revision 33b5aa65 (default mentioned kas-
> > cip.yml), it works with current isar/next as well. Also, I successfully
> > built it wth latest isar/ubely/sbuild revision.
> > 
> > Maybe there are some more specific steps to reproduce the issue you are
> > talking about that I could try to check on my side?
> 
> Ah sorry,
> 
> you need to add the SWUpdate option to the build. The kas command would be:
> 
> ```
> /kas-container --isar build
> kas-cip.yml:kas/board/qemu-amd64.yml:kas/opt/ebg-swu.yml
> ```
> 
> or you build SWUpdate from the devshell.
> 
> Kind regards
> Quirin

I tried this - and yes, I also faced this problem with "dh_clean --with lua".

There are 2 ways to deal with it:

1. We can simply add "--no-clean-source" to sbuild parameters. In this case 
sbuild itself doesn't trigger additional "dh_clean" before build. 

2. We can switch to building from .dsc file instead of building from unpacked 
source directory (e.g., firstly create .dsc file by dpkg-source, sencodly, run 
sbuild passing .dsc as argument). In this case sbuild also doesn't call 
dh_clean.

Both methods work, but the second looks more debian-friendly.

BTW, I faced some other build issues with "isar-cip-core":

1. "efibootguard" recipe attempts to deploy "efibootguardx64.efi" and 
"bg_setenv" binaries directly from source tree. This doesn't work with sbuild, 
especially in case of building from .dsc file. Moreover, "efibootguardx64.efi" 
binary doesn't even belong to any output .deb files.

I consider this binary should be added to .deb file and deploying should be 
done from it (e.g. using "dpkg --fsys-tarfile" for extraction). I can submit a 
patch if needed.

2. "swupdate" is built from git branch "debian/master". I had to add "-git-
ignore-branch" to GBP_EXTRA_OPTIONS, to skip the error, like "not master 
branch used".

3. Finally, I still have build error with "swupdate":
"The following packages have unmet dependencies:
 sbuild-build-depends-main-dummy : Depends: sphinx but it is not installable"

-- 
Uladzimir Bely

Re: Question related to Sbuild CLEAN_SOURCE

[Jan Kiszka]

On 11.02.22 11:54, Uladzimir Bely wrote:
> In the email from Thursday, 10 February 2022 12:10:47 +03 user 
> quirin.gylstorff@siemens.com wrote:
>> On 2/10/22 09:47, Uladzimir Bely wrote:
>>> In the email from Wednesday, 9 February 2022 18:14:22 +03 user
>>>
>>> quirin.gylstorff@siemens.com wrote:
>>>> On 2/9/22 12:52, Baurzhan Ismagulov wrote:
>>>>> On Wed, Feb 09, 2022 at 11:36:09AM +0100, quirin.gylstorff@siemens.com
>>>
>>> wrote:
>>>>>>>> - Is there are reason sbuild executes clean outside of the chroot?
>>>>>
>>>>> It would be interesting to hear what kind of issues you had due to that.
>>>>
>>>> If we rebuild a debian package with dpkg-gbp which uses for example
>>>> `dh-lua`  (swupdate) the build will fail due to the missing installation
>>>> of dh-lua in the host environment. The solution to this is to install
>>>> dh-* in the build environment.
>>>>
>>>> ```
>>>> gbp:info: Performing the build
>>>> dpkg-source: info: using patch list from debian/patches/series
>>>> dpkg-source: info: applying use-gcc-compiler.diff
>>>> dh clean --with lua
>>>> dh: error: unable to load addon lua: Can't locate
>>>> Debian/Debhelper/Sequence/lua.pm in @INC (you may need to install the
>>>> Debian::Debhelper::Sequence::lua module) (@INC contains: /etc/perl
>>>> /usr/local/lib/x86_64-linux-gnu/perl/5.32.1 /usr/local/share/perl/5.32.1
>>>> /usr/lib/x86_64-linux-gnu/perl5/5.32 /usr/share/perl5
>>>> /usr/lib/x86_64-linux-gnu/perl-base /usr/lib/x86_64-linux-gnu/perl/5.32
>>>> /usr/share/perl/5.32 /usr/local/lib/site_perl) at (eval 13) line 1.
>>>> BEGIN failed--compilation aborted at (eval 13) line 1.
>>>> ```
>>>>
>>>> You can fix it with documentation but it needs to be addressed.
>>>>
>>>> A project where this can be tested is isar-cip-core with the current
>>>> preintegrated sbuild[1].
>>>> [1]: https://gitlab.com/Quirin.Gy/isar-cip-core/-/tree/test/sbuild
>>>
>>> Hello.
>>> I've tried this repository (master branch) on my side.
>>> It currently builds OK with Isar revision 33b5aa65 (default mentioned kas-
>>> cip.yml), it works with current isar/next as well. Also, I successfully
>>> built it wth latest isar/ubely/sbuild revision.
>>>
>>> Maybe there are some more specific steps to reproduce the issue you are
>>> talking about that I could try to check on my side?
>>
>> Ah sorry,
>>
>> you need to add the SWUpdate option to the build. The kas command would be:
>>
>> ```
>> /kas-container --isar build
>> kas-cip.yml:kas/board/qemu-amd64.yml:kas/opt/ebg-swu.yml
>> ```
>>
>> or you build SWUpdate from the devshell.
>>
>> Kind regards
>> Quirin
> 
> I tried this - and yes, I also faced this problem with "dh_clean --with lua".
> 
> There are 2 ways to deal with it:
> 
> 1. We can simply add "--no-clean-source" to sbuild parameters. In this case 
> sbuild itself doesn't trigger additional "dh_clean" before build. 
> 
> 2. We can switch to building from .dsc file instead of building from unpacked 
> source directory (e.g., firstly create .dsc file by dpkg-source, sencodly, run 
> sbuild passing .dsc as argument). In this case sbuild also doesn't call 
> dh_clean.
> 
> Both methods work, but the second looks more debian-friendly.

Worse: The first one could erode over the time as it is no official
Debian method.

> 
> BTW, I faced some other build issues with "isar-cip-core":
> 
> 1. "efibootguard" recipe attempts to deploy "efibootguardx64.efi" and 
> "bg_setenv" binaries directly from source tree. This doesn't work with sbuild, 
> especially in case of building from .dsc file. Moreover, "efibootguardx64.efi" 
> binary doesn't even belong to any output .deb files.

The first one is a known conversion pattern. The second one is actually
a bug: We should not use the deployment folder, we should use a deb
package installed into the buildchroot that installs the bootloader,
just like we do in all other cases. Seems I missed that in reviews.

> 
> I consider this binary should be added to .deb file and deploying should be 
> done from it (e.g. using "dpkg --fsys-tarfile" for extraction). I can submit a 
> patch if needed.

Yes, please.

> 
> 2. "swupdate" is built from git branch "debian/master". I had to add "-git-
> ignore-branch" to GBP_EXTRA_OPTIONS, to skip the error, like "not master 
> branch used".
> 
> 3. Finally, I still have build error with "swupdate":
> "The following packages have unmet dependencies:
>  sbuild-build-depends-main-dummy : Depends: sphinx but it is not installable"
> 

You will likely need a better tool to get the explanation. apt itself
too primitive in complex cases. Usually, I jump in via devshell_nodeps,
install aptitude, try to install the failing dep and let aptitude
explain to my what goes wrong.

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

Re: Question related to Sbuild CLEAN_SOURCE

[quirin.gylstorff]

On 2/11/22 12:20, Jan Kiszka wrote:
> On 11.02.22 11:54, Uladzimir Bely wrote:
>> In the email from Thursday, 10 February 2022 12:10:47 +03 user
>> quirin.gylstorff@siemens.com wrote:
>>> On 2/10/22 09:47, Uladzimir Bely wrote:
>>>> In the email from Wednesday, 9 February 2022 18:14:22 +03 user
>>>>
>>>> quirin.gylstorff@siemens.com wrote:
>>>>> On 2/9/22 12:52, Baurzhan Ismagulov wrote:
>>>>>> On Wed, Feb 09, 2022 at 11:36:09AM +0100, quirin.gylstorff@siemens.com
>>>>
>>>> wrote:
>>>>>>>>> - Is there are reason sbuild executes clean outside of the chroot?
>>>>>>
>>>>>> It would be interesting to hear what kind of issues you had due to that.
>>>>>
>>>>> If we rebuild a debian package with dpkg-gbp which uses for example
>>>>> `dh-lua`  (swupdate) the build will fail due to the missing installation
>>>>> of dh-lua in the host environment. The solution to this is to install
>>>>> dh-* in the build environment.
>>>>>
>>>>> ```
>>>>> gbp:info: Performing the build
>>>>> dpkg-source: info: using patch list from debian/patches/series
>>>>> dpkg-source: info: applying use-gcc-compiler.diff
>>>>> dh clean --with lua
>>>>> dh: error: unable to load addon lua: Can't locate
>>>>> Debian/Debhelper/Sequence/lua.pm in @INC (you may need to install the
>>>>> Debian::Debhelper::Sequence::lua module) (@INC contains: /etc/perl
>>>>> /usr/local/lib/x86_64-linux-gnu/perl/5.32.1 /usr/local/share/perl/5.32.1
>>>>> /usr/lib/x86_64-linux-gnu/perl5/5.32 /usr/share/perl5
>>>>> /usr/lib/x86_64-linux-gnu/perl-base /usr/lib/x86_64-linux-gnu/perl/5.32
>>>>> /usr/share/perl/5.32 /usr/local/lib/site_perl) at (eval 13) line 1.
>>>>> BEGIN failed--compilation aborted at (eval 13) line 1.
>>>>> ```
>>>>>
>>>>> You can fix it with documentation but it needs to be addressed.
>>>>>
>>>>> A project where this can be tested is isar-cip-core with the current
>>>>> preintegrated sbuild[1].
>>>>> [1]: https://gitlab.com/Quirin.Gy/isar-cip-core/-/tree/test/sbuild
>>>>
>>>> Hello.
>>>> I've tried this repository (master branch) on my side.
>>>> It currently builds OK with Isar revision 33b5aa65 (default mentioned kas-
>>>> cip.yml), it works with current isar/next as well. Also, I successfully
>>>> built it wth latest isar/ubely/sbuild revision.
>>>>
>>>> Maybe there are some more specific steps to reproduce the issue you are
>>>> talking about that I could try to check on my side?
>>>
>>> Ah sorry,
>>>
>>> you need to add the SWUpdate option to the build. The kas command would be:
>>>
>>> ```
>>> /kas-container --isar build
>>> kas-cip.yml:kas/board/qemu-amd64.yml:kas/opt/ebg-swu.yml
>>> ```
>>>
>>> or you build SWUpdate from the devshell.
>>>
>>> Kind regards
>>> Quirin
>>
>> I tried this - and yes, I also faced this problem with "dh_clean --with lua".
>>
>> There are 2 ways to deal with it:
>>
>> 1. We can simply add "--no-clean-source" to sbuild parameters. In this case
>> sbuild itself doesn't trigger additional "dh_clean" before build.
>>
>> 2. We can switch to building from .dsc file instead of building from unpacked
>> source directory (e.g., firstly create .dsc file by dpkg-source, sencodly, run
>> sbuild passing .dsc as argument). In this case sbuild also doesn't call
>> dh_clean.
>>
>> Both methods work, but the second looks more debian-friendly.
> 
> Worse: The first one could erode over the time as it is no official
> Debian method.
> 
>>
>> BTW, I faced some other build issues with "isar-cip-core":
>>
>> 1. "efibootguard" recipe attempts to deploy "efibootguardx64.efi" and
>> "bg_setenv" binaries directly from source tree. This doesn't work with sbuild,
>> especially in case of building from .dsc file. Moreover, "efibootguardx64.efi"
>> binary doesn't even belong to any output .deb files.
> 
> The first one is a known conversion pattern. The second one is actually
> a bug: We should not use the deployment folder, we should use a deb
> package installed into the buildchroot that installs the bootloader,
> just like we do in all other cases. Seems I missed that in reviews.
> 
>>
>> I consider this binary should be added to .deb file and deploying should be
>> done from it (e.g. using "dpkg --fsys-tarfile" for extraction). I can submit a
>> patch if needed.
> 
> Yes, please.

Is this extraction working with sstate? I had some problems with 
rebuilding from sstate with this pattern.

> 
>>
>> 2. "swupdate" is built from git branch "debian/master". I had to add "-git-
>> ignore-branch" to GBP_EXTRA_OPTIONS, to skip the error, like "not master
>> branch used".
>>
>> 3. Finally, I still have build error with "swupdate":
>> "The following packages have unmet dependencies:
>>   sbuild-build-depends-main-dummy : Depends: sphinx but it is not installable"
>>

This looks like 
https://salsa.debian.org/debian/swupdate/-/commit/d666b84435e1b1abebc6070252872ed390ac4df8. 
The profile `nodoc` should be set.


> 
> You will likely need a better tool to get the explanation. apt itself
> too primitive in complex cases. Usually, I jump in via devshell_nodeps,
> install aptitude, try to install the failing dep and let aptitude
> explain to my what goes wrong.


Quirin

Re: Question related to Sbuild CLEAN_SOURCE

[Uladzimir Bely]

In the email from Friday, 11 February 2022 14:20:10 +03 user Jan Kiszka wrote:
> On 11.02.22 11:54, Uladzimir Bely wrote:
> > In the email from Thursday, 10 February 2022 12:10:47 +03 user
> > 
> > quirin.gylstorff@siemens.com wrote:
> >> On 2/10/22 09:47, Uladzimir Bely wrote:
> >>> In the email from Wednesday, 9 February 2022 18:14:22 +03 user
> >>> 
> >>> quirin.gylstorff@siemens.com wrote:
> >>>> On 2/9/22 12:52, Baurzhan Ismagulov wrote:
> >>>>> On Wed, Feb 09, 2022 at 11:36:09AM +0100, quirin.gylstorff@siemens.com
> >>> 
> >>> wrote:
> >>>>>>>> - Is there are reason sbuild executes clean outside of the chroot?
> >>>>> 
> >>>>> It would be interesting to hear what kind of issues you had due to
> >>>>> that.
> >>>> 
> >>>> If we rebuild a debian package with dpkg-gbp which uses for example
> >>>> `dh-lua`  (swupdate) the build will fail due to the missing
> >>>> installation
> >>>> of dh-lua in the host environment. The solution to this is to install
> >>>> dh-* in the build environment.
> >>>> 
> >>>> ```
> >>>> gbp:info: Performing the build
> >>>> dpkg-source: info: using patch list from debian/patches/series
> >>>> dpkg-source: info: applying use-gcc-compiler.diff
> >>>> dh clean --with lua
> >>>> dh: error: unable to load addon lua: Can't locate
> >>>> Debian/Debhelper/Sequence/lua.pm in @INC (you may need to install the
> >>>> Debian::Debhelper::Sequence::lua module) (@INC contains: /etc/perl
> >>>> /usr/local/lib/x86_64-linux-gnu/perl/5.32.1
> >>>> /usr/local/share/perl/5.32.1
> >>>> /usr/lib/x86_64-linux-gnu/perl5/5.32 /usr/share/perl5
> >>>> /usr/lib/x86_64-linux-gnu/perl-base /usr/lib/x86_64-linux-gnu/perl/5.32
> >>>> /usr/share/perl/5.32 /usr/local/lib/site_perl) at (eval 13) line 1.
> >>>> BEGIN failed--compilation aborted at (eval 13) line 1.
> >>>> ```
> >>>> 
> >>>> You can fix it with documentation but it needs to be addressed.
> >>>> 
> >>>> A project where this can be tested is isar-cip-core with the current
> >>>> preintegrated sbuild[1].
> >>>> [1]: https://gitlab.com/Quirin.Gy/isar-cip-core/-/tree/test/sbuild
> >>> 
> >>> Hello.
> >>> I've tried this repository (master branch) on my side.
> >>> It currently builds OK with Isar revision 33b5aa65 (default mentioned
> >>> kas-
> >>> cip.yml), it works with current isar/next as well. Also, I successfully
> >>> built it wth latest isar/ubely/sbuild revision.
> >>> 
> >>> Maybe there are some more specific steps to reproduce the issue you are
> >>> talking about that I could try to check on my side?
> >> 
> >> Ah sorry,
> >> 
> >> you need to add the SWUpdate option to the build. The kas command would
> >> be:
> >> 
> >> ```
> >> /kas-container --isar build
> >> kas-cip.yml:kas/board/qemu-amd64.yml:kas/opt/ebg-swu.yml
> >> ```
> >> 
> >> or you build SWUpdate from the devshell.
> >> 
> >> Kind regards
> >> Quirin
> > 
> > I tried this - and yes, I also faced this problem with "dh_clean --with
> > lua".
> > 
> > There are 2 ways to deal with it:
> > 
> > 1. We can simply add "--no-clean-source" to sbuild parameters. In this
> > case
> > sbuild itself doesn't trigger additional "dh_clean" before build.
> > 
> > 2. We can switch to building from .dsc file instead of building from
> > unpacked source directory (e.g., firstly create .dsc file by dpkg-source,
> > sencodly, run sbuild passing .dsc as argument). In this case sbuild also
> > doesn't call dh_clean.
> > 
> > Both methods work, but the second looks more debian-friendly.
> 
> Worse: The first one could erode over the time as it is no official
> Debian method.
> 
> > BTW, I faced some other build issues with "isar-cip-core":
> > 
> > 1. "efibootguard" recipe attempts to deploy "efibootguardx64.efi" and
> > "bg_setenv" binaries directly from source tree. This doesn't work with
> > sbuild, especially in case of building from .dsc file. Moreover,
> > "efibootguardx64.efi" binary doesn't even belong to any output .deb
> > files.
> 
> The first one is a known conversion pattern. The second one is actually
> a bug: We should not use the deployment folder, we should use a deb
> package installed into the buildchroot that installs the bootloader,
> just like we do in all other cases. Seems I missed that in reviews.
> 
> > I consider this binary should be added to .deb file and deploying should
> > be
> > done from it (e.g. using "dpkg --fsys-tarfile" for extraction). I can
> > submit a patch if needed.
> 
> Yes, please.
> 
Submitted just now.

Seems, my first attempt to send patch (previous week) failed due to missing 
subscription to cip-dev@lists.cip-project.org list.

> > 2. "swupdate" is built from git branch "debian/master". I had to add
> > "-git-
> > ignore-branch" to GBP_EXTRA_OPTIONS, to skip the error, like "not master
> > branch used".
> > 
> > 3. Finally, I still have build error with "swupdate":
> > 
> > "The following packages have unmet dependencies:
> >  sbuild-build-depends-main-dummy : Depends: sphinx but it is not
> >  installable"
> You will likely need a better tool to get the explanation. apt itself
> too primitive in complex cases. Usually, I jump in via devshell_nodeps,
> install aptitude, try to install the failing dep and let aptitude
> explain to my what goes wrong.
> 
> Jan


-- 
Uladzimir Bely