[PATCH 1/2] meta: remove dead code from buildchroot script

[PATCH 1/2] meta: remove dead code from buildchroot script

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

6d0ce719f2a1e8bd01bc4ed3005bb16058a0a9ff dropped code that used the
variable

We do not need it anymore so remove it.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta/recipes-devtools/buildchroot/files/common.sh | 1 -
 1 file changed, 1 deletion(-)

diff --git a/meta/recipes-devtools/buildchroot/files/common.sh b/meta/recipes-devtools/buildchroot/files/common.sh
index f75a84e4bf..0063a38e61 100644
--- a/meta/recipes-devtools/buildchroot/files/common.sh
+++ b/meta/recipes-devtools/buildchroot/files/common.sh
@@ -18,7 +18,6 @@ fi
 # Create human-readable names
 target_arch=$2
 
-debian_version=$(cut -c1 /etc/debian_version)
 set_arch="--host-arch $target_arch"
 
 # Go to build directory
-- 
2.26.2

[PATCH 2/2] meta: make builddep installation verbose

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

We can not rely on the return value of mk-build-deps and do grep to
decide whether it worked. But that way we loose the output for
debugging.

Keep a copy of the output and write it to stdout.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta/recipes-devtools/buildchroot/files/deps.sh | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-devtools/buildchroot/files/deps.sh b/meta/recipes-devtools/buildchroot/files/deps.sh
index 6048305a7a..93bc9cf557 100644
--- a/meta/recipes-devtools/buildchroot/files/deps.sh
+++ b/meta/recipes-devtools/buildchroot/files/deps.sh
@@ -35,8 +35,13 @@ fi
 
 # Install all build deps
 if [ "$3" = "--download-only" ]; then
-    mk-build-deps $set_arch -t "${install_cmd}" -i -r debian/control 2>&1 \
-        | grep "mk-build-deps: Unable to install all build-dep packages"
+    # this will not return 0 even when it worked
+    mk-build-deps $set_arch -t "${install_cmd}" -i -r debian/control &> \
+        mk-build-deps.output || true
+    cat mk-build-deps.output
+    # we assume success when we find this
+    grep "mk-build-deps: Unable to install all build-dep packages" mk-build-deps.output
+    rm -f mk-build-deps.output
 else
     mk-build-deps $set_arch -t "${install_cmd}" -i -r debian/control
 
-- 
2.26.2

Re: [PATCH 1/2] meta: remove dead code from buildchroot script

[Henning Schild]

The first one is a cleanup. And the second one can hopefully help find
out why bullseye currently does not build anymore.

https://github.com/henning-schild-work/isar/tree/henning/staging3

Henning

Am Tue, 5 May 2020 13:00:25 +0200
schrieb Henning Schild <henning.schild@siemens.com>:

> From: Henning Schild <henning.schild@siemens.com>
> 
> 6d0ce719f2a1e8bd01bc4ed3005bb16058a0a9ff dropped code that used the
> variable
> 
> We do not need it anymore so remove it.
> 
> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> ---
>  meta/recipes-devtools/buildchroot/files/common.sh | 1 -
>  1 file changed, 1 deletion(-)
> 
> diff --git a/meta/recipes-devtools/buildchroot/files/common.sh
> b/meta/recipes-devtools/buildchroot/files/common.sh index
> f75a84e4bf..0063a38e61 100644 ---
> a/meta/recipes-devtools/buildchroot/files/common.sh +++
> b/meta/recipes-devtools/buildchroot/files/common.sh @@ -18,7 +18,6 @@
> fi # Create human-readable names
>  target_arch=$2
>  
> -debian_version=$(cut -c1 /etc/debian_version)
>  set_arch="--host-arch $target_arch"
>  
>  # Go to build directory

Re: [PATCH 1/2] meta: remove dead code from buildchroot script

[Baurzhan Ismagulov]

Hello Henning,

On Tue, May 05, 2020 at 01:02:34PM +0200, Henning Schild wrote:
> The first one is a cleanup. And the second one can hopefully help find
> out why bullseye currently does not build anymore.

Thanks. FWIW, at least some of the output leading to the problem can be seen
here:

http://ci.isar-build.org:8080/job/isar_vs_devel_3_fast/9/consoleFull

The relevant part is:

| + mk-build-deps --host-arch armhf -t 'apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends     -y --allow-downgrades ' -i -r debian/control
| dpkg-buildpackage: info: source package isar-exclude-docs-cross-build-deps
| dpkg-buildpackage: info: source version 0.1
| dpkg-buildpackage: info: source distribution unstable
| dpkg-buildpackage: info: source changed by root <root@baighyz.m.ilbers.de>
| dpkg-architecture: warning: specified GNU system type arm-linux-gnueabihf does not match CC system type x86_64-linux-gnu, try setting a correct CC environment variable
|  dpkg-source --before-build .
| dpkg-buildpackage: info: host architecture armhf
| dpkg-checkbuilddeps: error: Unmet build dependencies: build-essential:native
| dpkg-buildpackage: warning: build dependencies/conflicts unsatisfied; aborting
| dpkg-buildpackage: warning: (Use -d flag to override.)
| Error in the build process: exit status 3
| dpkg: error: cannot access archive 'isar-exclude-docs-cross-build-deps_0.1_armhf.deb': No such file or directory
| mk-build-deps: dpkg --unpack failed
| WARNING: exit code 2 from a shell command.
| 
NOTE: recipe isar-exclude-docs-0.1-r0: task do_install_builddeps: Failed

A possible scenario is that the temporary package created by mk-build-deps now
implicitly or explicitly Build-Depends on build-essential:native, which isn't
installed at the time of dpkg-buildpackage. If we can verify this, we could
install the missing Build-Depends before mk-build-deps.

Which would be a hen-and-egg problem, since mk-build-deps has been introduced
to avoid manual parsing of debian/control fields. The ultimate solution could
be to add all packages to isar-apt and apt-get build-deps e.g. libhello-dev. I
don't know whether that could be done today. Meanwhile, I still think that
parsing the fields is a viable option (with the goal to have the solution
upstream).

With kind regards,
Baurzhan.

Re: [PATCH 1/2] meta: remove dead code from buildchroot script

[Henning Schild]

Am Wed, 6 May 2020 10:57:11 +0200
schrieb Baurzhan Ismagulov <ibr@radix50.net>:

> Hello Henning,
> 
> On Tue, May 05, 2020 at 01:02:34PM +0200, Henning Schild wrote:
> > The first one is a cleanup. And the second one can hopefully help
> > find out why bullseye currently does not build anymore.  
> 
> Thanks. FWIW, at least some of the output leading to the problem can
> be seen here:
> 
> http://ci.isar-build.org:8080/job/isar_vs_devel_3_fast/9/consoleFull
> 
> The relevant part is:
> 
> | + mk-build-deps --host-arch armhf -t 'apt-get -o
> Debug::pkgProblemResolver=yes --no-install-recommends     -y
> --allow-downgrades ' -i -r debian/control | dpkg-buildpackage: info:
> source package isar-exclude-docs-cross-build-deps |
> dpkg-buildpackage: info: source version 0.1 | dpkg-buildpackage:
> info: source distribution unstable | dpkg-buildpackage: info: source
> changed by root <root@baighyz.m.ilbers.de> | dpkg-architecture:
> warning: specified GNU system type arm-linux-gnueabihf does not match
> CC system type x86_64-linux-gnu, try setting a correct CC environment
> variable |  dpkg-source --before-build . | dpkg-buildpackage: info:
> host architecture armhf | dpkg-checkbuilddeps: error: Unmet build
> dependencies: build-essential:native | dpkg-buildpackage: warning:
> build dependencies/conflicts unsatisfied; aborting |
> dpkg-buildpackage: warning: (Use -d flag to override.) | Error in the
> build process: exit status 3 | dpkg: error: cannot access archive
> 'isar-exclude-docs-cross-build-deps_0.1_armhf.deb': No such file or
> directory | mk-build-deps: dpkg --unpack failed | WARNING: exit code
> 2 from a shell command. | NOTE: recipe isar-exclude-docs-0.1-r0: task
> do_install_builddeps: Failed
> 
> A possible scenario is that the temporary package created by
> mk-build-deps now implicitly or explicitly Build-Depends on
> build-essential:native, which isn't installed at the time of
> dpkg-buildpackage. If we can verify this, we could install the
> missing Build-Depends before mk-build-deps.
> 
> Which would be a hen-and-egg problem, since mk-build-deps has been
> introduced to avoid manual parsing of debian/control fields. The
> ultimate solution could be to add all packages to isar-apt and
> apt-get build-deps e.g. libhello-dev. I don't know whether that could
> be done today. Meanwhile, I still think that parsing the fields is a
> viable option (with the goal to have the solution upstream).

I would conclude that those patches seem to work. We can merge them
later with the fix, but whoever comes up with the fix might want to use
them.

I think if we find a hen-and-egg problem we found a Debian bug and can
lean back after reporting. But that needs further analysis and someone
taking care of the issue and possible involving upstream.

At the moment i will not be able to do that, i just felt i should fix
my code that swallowed the debug output.

Henning

> With kind regards,
> Baurzhan.
> 

Re: [PATCH 1/2] meta: remove dead code from buildchroot script

[Jan Kiszka]

On 06.05.20 17:40, [ext] Henning Schild wrote:
> Am Wed, 6 May 2020 10:57:11 +0200
> schrieb Baurzhan Ismagulov <ibr@radix50.net>:
> 
>> Hello Henning,
>>
>> On Tue, May 05, 2020 at 01:02:34PM +0200, Henning Schild wrote:
>>> The first one is a cleanup. And the second one can hopefully help
>>> find out why bullseye currently does not build anymore.
>>
>> Thanks. FWIW, at least some of the output leading to the problem can
>> be seen here:
>>
>> http://ci.isar-build.org:8080/job/isar_vs_devel_3_fast/9/consoleFull
>>
>> The relevant part is:
>>
>> | + mk-build-deps --host-arch armhf -t 'apt-get -o
>> Debug::pkgProblemResolver=yes --no-install-recommends     -y
>> --allow-downgrades ' -i -r debian/control | dpkg-buildpackage: info:
>> source package isar-exclude-docs-cross-build-deps |
>> dpkg-buildpackage: info: source version 0.1 | dpkg-buildpackage:
>> info: source distribution unstable | dpkg-buildpackage: info: source
>> changed by root <root@baighyz.m.ilbers.de> | dpkg-architecture:
>> warning: specified GNU system type arm-linux-gnueabihf does not match
>> CC system type x86_64-linux-gnu, try setting a correct CC environment
>> variable |  dpkg-source --before-build . | dpkg-buildpackage: info:
>> host architecture armhf | dpkg-checkbuilddeps: error: Unmet build
>> dependencies: build-essential:native | dpkg-buildpackage: warning:
>> build dependencies/conflicts unsatisfied; aborting |
>> dpkg-buildpackage: warning: (Use -d flag to override.) | Error in the
>> build process: exit status 3 | dpkg: error: cannot access archive
>> 'isar-exclude-docs-cross-build-deps_0.1_armhf.deb': No such file or
>> directory | mk-build-deps: dpkg --unpack failed | WARNING: exit code
>> 2 from a shell command. | NOTE: recipe isar-exclude-docs-0.1-r0: task
>> do_install_builddeps: Failed
>>
>> A possible scenario is that the temporary package created by
>> mk-build-deps now implicitly or explicitly Build-Depends on
>> build-essential:native, which isn't installed at the time of
>> dpkg-buildpackage. If we can verify this, we could install the
>> missing Build-Depends before mk-build-deps.
>>
>> Which would be a hen-and-egg problem, since mk-build-deps has been
>> introduced to avoid manual parsing of debian/control fields. The
>> ultimate solution could be to add all packages to isar-apt and
>> apt-get build-deps e.g. libhello-dev. I don't know whether that could
>> be done today. Meanwhile, I still think that parsing the fields is a
>> viable option (with the goal to have the solution upstream).
> 
> I would conclude that those patches seem to work. We can merge them
> later with the fix, but whoever comes up with the fix might want to use
> them.
> 
> I think if we find a hen-and-egg problem we found a Debian bug and can
> lean back after reporting. But that needs further analysis and someone
> taking care of the issue and possible involving upstream.
> 
> At the moment i will not be able to do that, i just felt i should fix
> my code that swallowed the debug output.

Yes, the issue with bullseye is no single patch regresion, it also 
effects today's next.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH 1/2] meta: remove dead code from buildchroot script

[Baurzhan Ismagulov]

On Wed, May 06, 2020 at 05:40:44PM +0200, Henning Schild wrote:
> I would conclude that those patches seem to work. We can merge them
> later with the fix, but whoever comes up with the fix might want to use
> them.

Not to mean your patches are useless, those look good to me, I'd like to apply
them in the course of the usual review process. My response was aimed at
sharing the current state of the analysis to move forward faster and avoid
duplicated effort.


> I think if we find a hen-and-egg problem we found a Debian bug and can
> lean back after reporting. But that needs further analysis and someone
> taking care of the issue and possible involving upstream.

Seems I haven't described it clearly. The hen-and-egg problem I refer to is:

Current state: Don't parse Build-Depends of e.g. libhello-dev to avoid
maintaining own parser -> use mk-build-deps to generate a dummy package that
Depends on Build-Depends of libhello-dev -> mk-build-deps in bullseye seems to
generate a source package that has Build-Depends itself. What now? We have to
install Build-Depends of libhello-cross-build-deps before we build and install
it. So I don't think this is a Debian bug, it's a matter of how we handle
installing Build-Depends of e.g. libhello-dev. Yes, we'll deal with that in a
second step, this explanation says nothing against your patches.

The solutions I was talking about:

Option 1: If libhello source package is in isar-apt, the Debian way would be to
apt-get build-dep libhello. If not, we should look at having it there.

Option 2: If the above is not possible ATM, we should look at parsing the
strings. Seems that dpkg has a Perl API. From ELBE folks I've heard that they
use python-debian (not sure whether it has this specific feature). I agree that
it should be an upstream solution. However, we might need to maintain it from
time to time.


> At the moment i will not be able to do that, i just felt i should fix
> my code that swallowed the debug output.

Got it, I see no problem with that.


With kind regards,
Baurzhan.

Re: [PATCH 1/2] meta: remove dead code from buildchroot script

[Baurzhan Ismagulov]

On Tue, May 05, 2020 at 01:00:25PM +0200, Henning Schild wrote:
> From: Henning Schild <henning.schild@siemens.com>
> 
> 6d0ce719f2a1e8bd01bc4ed3005bb16058a0a9ff dropped code that used the
> variable
> 
> We do not need it anymore so remove it.

Applied to next with minor commit message updates, thanks.

With kind regards,
Baurzhan.