[PATCH] ci-build: Protect bullseye container with KFAIL

[PATCH] ci-build: Protect bullseye container with KFAIL

[Anton Mikanovich]

Bullseye building issue still not resolved, so container-bullseye fails
with error:

> fakeroot debian/rules clean
>semop(1): encountered an error: Function not implemented

This issue should be already fixed in host qemu/1:5.0-9 but we still
experience the same with qemu/1:5.2+dfsg-3~bpo10+1. Further research
will be done, so temporary fix for ci-build.sh introduced.

Bug report: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=965109

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 scripts/ci_build.sh | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index 6445b10..adc22e4 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -58,6 +58,9 @@ TARGETS_SET_BULLSEYE="\
 TARGETS_CONTAINERS="\
     mc:container-stretch:isar-image-base \
     mc:container-buster:isar-image-base \
+"
+
+TARGETS_CONTAINERS_BULLSEYE="\
     mc:container-bullseye:isar-image-base \
 "
 
@@ -259,6 +262,12 @@ sed -i -e 's/\(IMAGE_INSTALL = .*\) example-module-${KERNEL_NAME}\(.*\)/\1\2/g'
 sed -i -e 's/\(IMAGE_INSTALL = .*\) enable-fsck\(.*\)/\1\2/g' conf/local.conf
 bitbake $BB_ARGS $TARGETS_CONTAINERS
 while [ -e bitbake.sock ]; do sleep 1; done
+if bitbake $BB_ARGS $TARGETS_CONTAINERS_BULLSEYE; then
+    echo "bullseye container: PASSED"
+else
+    echo "bullseye container: KFAIL"
+fi
+while [ -e bitbake.sock ]; do sleep 1; done
 # and SDK container image creation
 echo 'SDK_FORMATS = "docker-archive"' >> conf/local.conf
 bitbake $BB_ARGS -c do_populate_sdk mc:container-stretch:isar-image-base
-- 
2.20.1

RE: [PATCH] ci-build: Protect bullseye container with KFAIL

[Moessbauer, Felix]

Hi,

I stumbled upon that while debugging the build errors on the DKMS module patch.
If possible, we could just bump QEMU in the KAS image to have a proper fix.

Otherwise, we might run into that issue here and there and it's not obvious that this is the root cause.
I don't think that the patch will be integrated into buster, so we can't just wait for it. 

Anyways, at least a warning message should be emitted when running on the broken version.

PS: sending this to the list as well.

Best regards,
Felix

> -----Original Message-----
> From: Anton Mikanovich <amikan@ilbers.de>
> Sent: Thursday, April 29, 2021 4:01 PM
> To: Moessbauer, Felix (T RDA IOT SES-DE) <felix.moessbauer@siemens.com>
> Cc: Kiszka, Jan (T RDA IOT) <jan.kiszka@siemens.com>
> Subject: Re: [PATCH] ci-build: Protect bullseye container with KFAIL
> 
> 29.04.2021 16:42, Moessbauer, Felix wrote:
> > Hi Anton,
> >
> > I can confirm that the CI patch does not solve the issue (on mc:qemuarm64-
> focal).
> > In case of the gitlab-ci, the KAS image provides the qemu-user-static.
> >
> > I just tried to install qemu-user-static from buster-backports into the KAS
> container and that solved the issue.
> > Maybe Jan might have a look into that. IMO this would be way better than
> trying to work-around in ISAR.
> >
> > Best regards,
> > Felix
> 
> Sure that was just an ugly hotfix to not have all CI builds 'red'
> because of that.
> When the original issue will be solved we can remove those KFAILs.
> 
> --
> Anton Mikanovich
> Promwad Ltd.
> External service provider of ilbers GmbH Maria-Merian-Str. 8
> 85521 Ottobrunn, Germany
> +49 (89) 122 67 24-0
> Commercial register Munich, HRB 214197
> General Manager: Baurzhan Ismagulov

Re: [PATCH] ci-build: Protect bullseye container with KFAIL

[Henning Schild]

Am Thu, 29 Apr 2021 14:05:34 +0000
schrieb "Moessbauer, Felix" <felix.moessbauer@siemens.com>:

> Hi,
> 
> I stumbled upon that while debugging the build errors on the DKMS
> module patch. If possible, we could just bump QEMU in the KAS image
> to have a proper fix.

Additions to the kas images need to go to the kas ML. the CI of
ilbers/isar does not use kas so we might need to detect and skip if
possible

ilbers not using kas is a problem that i think is being worked on, but
it also has value to make sure isar does not depend on kas

Henning

> Otherwise, we might run into that issue here and there and it's not
> obvious that this is the root cause. I don't think that the patch
> will be integrated into buster, so we can't just wait for it. 
> 
> Anyways, at least a warning message should be emitted when running on
> the broken version.
> 
> PS: sending this to the list as well.
> 
> Best regards,
> Felix
> 
> > -----Original Message-----
> > From: Anton Mikanovich <amikan@ilbers.de>
> > Sent: Thursday, April 29, 2021 4:01 PM
> > To: Moessbauer, Felix (T RDA IOT SES-DE)
> > <felix.moessbauer@siemens.com> Cc: Kiszka, Jan (T RDA IOT)
> > <jan.kiszka@siemens.com> Subject: Re: [PATCH] ci-build: Protect
> > bullseye container with KFAIL
> > 
> > 29.04.2021 16:42, Moessbauer, Felix wrote:  
> > > Hi Anton,
> > >
> > > I can confirm that the CI patch does not solve the issue (on
> > > mc:qemuarm64-  
> > focal).  
> > > In case of the gitlab-ci, the KAS image provides the
> > > qemu-user-static.
> > >
> > > I just tried to install qemu-user-static from buster-backports
> > > into the KAS  
> > container and that solved the issue.  
> > > Maybe Jan might have a look into that. IMO this would be way
> > > better than  
> > trying to work-around in ISAR.  
> > >
> > > Best regards,
> > > Felix  
> > 
> > Sure that was just an ugly hotfix to not have all CI builds 'red'
> > because of that.
> > When the original issue will be solved we can remove those KFAILs.
> > 
> > --
> > Anton Mikanovich
> > Promwad Ltd.
> > External service provider of ilbers GmbH Maria-Merian-Str. 8
> > 85521 Ottobrunn, Germany
> > +49 (89) 122 67 24-0
> > Commercial register Munich, HRB 214197
> > General Manager: Baurzhan Ismagulov  
> 

RE: [PATCH] ci-build: Protect bullseye container with KFAIL

[Moessbauer, Felix]

Hi Jan, Anton,

Actually I don't think the issue is related to binfmt_misc.
If I get it right, binfmt_misc just dispatches and starts the arm binary with qemu-arm-static.

The bug (missing handling of syscall) is in the qemu binary itself.
Neither mmdebstrap nor namespace support in binfmt_misc can help here.

The good thing is, that we can detect the broken qemu version and issue a warning.
But that just spams logs as the do_install_builddeps is a heavily used task...

Best regards,
Felix

> -----Original Message-----
> From: Jan Kiszka <jan.kiszka@siemens.com>
> Sent: Thursday, April 29, 2021 7:46 PM
> To: Moessbauer, Felix (T RDA IOT SES-DE) <felix.moessbauer@siemens.com>;
> Anton Mikanovich <amikan@ilbers.de>
> Subject: Re: [PATCH] ci-build: Protect bullseye container with KFAIL
> 
> On 29.04.21 15:42, Moessbauer, Felix (T RDA IOT SES-DE) wrote:
> > Hi Anton,
> >
> > I can confirm that the CI patch does not solve the issue (on mc:qemuarm64-
> focal).
> > In case of the gitlab-ci, the KAS image provides the qemu-user-static.
> 
> ...as along as the host didn't have an older version already used (e.g.
> from an older container). By default, binfmt_misc is sticky /wrt the first used
> interpreter binary. When unsure, clean all binfmt settings, then use the latest
> container image.
> 
> >
> > I just tried to install qemu-user-static from buster-backports into the KAS
> container and that solved the issue.
> > Maybe Jan might have a look into that. IMO this would be way better than
> trying to work-around in ISAR.
> 
> Sure, this is a host (or build container) issue, nothing for Isar itself.
> 
> Eventually, mmdebstrap in chrootless mode should help to overcome such
> issues. Or proper namespace support for binfmt_misc.
> 
> Jan
> 
> >
> > Best regards,
> > Felix
> >
> >> -----Original Message-----
> >> From: isar-users@googlegroups.com <isar-users@googlegroups.com> On
> >> Behalf Of Anton Mikanovich
> >> Sent: Friday, April 9, 2021 10:15 AM
> >> To: isar-users@googlegroups.com
> >> Cc: Anton Mikanovich <amikan@ilbers.de>
> >> Subject: [PATCH] ci-build: Protect bullseye container with KFAIL
> >>
> >> Bullseye building issue still not resolved, so container-bullseye fails with error:
> >>
> >>> fakeroot debian/rules clean
> >>> semop(1): encountered an error: Function not implemented
> >>
> >> This issue should be already fixed in host qemu/1:5.0-9 but we still
> >> experience the same with qemu/1:5.2+dfsg-3~bpo10+1. Further research
> >> will be done, so temporary fix for ci-build.sh introduced.
> >>
> >> Bug report:
> >> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbug
> >>
> s.deb%2F&amp;data=04%7C01%7Cfelix.moessbauer%40siemens.com%7C92bd
> 09c5
> >>
> 68bb45136a1008d90b37b6b5%7C38ae3bcd95794fd4addab42e1495d55a%7C1%
> 7C0%7
> >>
> C637553156176955322%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwM
> DAiLCJQI
> >>
> joiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=Q9A9N5
> m%2B
> >> BTq%2BzEJI4Sfkp6aInSDj8qgPZ3SZlh6t7WE%3D&amp;reserved=0
> >> ian.org%2Fcgi-
> >>
> bin%2Fbugreport.cgi%3Fbug%3D965109&amp;data=04%7C01%7Cfelix.moessba
> >>
> uer%40siemens.com%7C4c4f7801dd2b4db1c5a808d8fb2fa8dd%7C38ae3bcd95
> >>
> 794fd4addab42e1495d55a%7C1%7C0%7C637535529408448819%7CUnknown%
> >>
> 7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL
> >>
> CJXVCI6Mn0%3D%7C1000&amp;sdata=pvHkywD1yglmoyG7TOjSaTTYWtFOZL%
> >> 2FqAlrG4UOjpQM%3D&amp;reserved=0
> >>
> >> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
> >> ---
> >>  scripts/ci_build.sh | 9 +++++++++
> >>  1 file changed, 9 insertions(+)
> >>
> >> diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh index
> >> 6445b10..adc22e4
> >> 100755
> >> --- a/scripts/ci_build.sh
> >> +++ b/scripts/ci_build.sh
> >> @@ -58,6 +58,9 @@ TARGETS_SET_BULLSEYE="\
> TARGETS_CONTAINERS="\
> >>      mc:container-stretch:isar-image-base \
> >>      mc:container-buster:isar-image-base \
> >> +"
> >> +
> >> +TARGETS_CONTAINERS_BULLSEYE="\
> >>      mc:container-bullseye:isar-image-base \  "
> >>
> >> @@ -259,6 +262,12 @@ sed -i -e 's/\(IMAGE_INSTALL = .*\)
> >> example-module- ${KERNEL_NAME}\(.*\)/\1\2/g'
> >>  sed -i -e 's/\(IMAGE_INSTALL = .*\) enable-fsck\(.*\)/\1\2/g'
> >> conf/local.conf bitbake $BB_ARGS $TARGETS_CONTAINERS  while [ -e
> >> bitbake.sock ]; do sleep 1; done
> >> +if bitbake $BB_ARGS $TARGETS_CONTAINERS_BULLSEYE; then
> >> +    echo "bullseye container: PASSED"
> >> +else
> >> +    echo "bullseye container: KFAIL"
> >> +fi
> >> +while [ -e bitbake.sock ]; do sleep 1; done
> >>  # and SDK container image creation
> >>  echo 'SDK_FORMATS = "docker-archive"' >> conf/local.conf  bitbake
> >> $BB_ARGS -c do_populate_sdk mc:container-stretch:isar-image-base
> >> --
> >> 2.20.1
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> >> Groups "isar-users" group.
> >> To unsubscribe from this group and stop receiving emails from it,
> >> send an email to isar-users+unsubscribe@googlegroups.com.
> >> To view this discussion on the web visit
> >> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgro
> >>
> ups.g%2F&amp;data=04%7C01%7Cfelix.moessbauer%40siemens.com%7C92bd
> 09c5
> >>
> 68bb45136a1008d90b37b6b5%7C38ae3bcd95794fd4addab42e1495d55a%7C1%
> 7C0%7
> >>
> C637553156176955322%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwM
> DAiLCJQI
> >>
> joiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=u3Q8XDu
> KM0
> >> IxNW3DetVika0M3D73yWxn8Nd3TIfpb1Q%3D&amp;reserved=0
> >> oogle.com%2Fd%2Fmsgid%2Fisar-users%2F20210409081520.399406-1-
> >>
> amikan%2540ilbers.de&amp;data=04%7C01%7Cfelix.moessbauer%40siemens.c
> >>
> om%7C4c4f7801dd2b4db1c5a808d8fb2fa8dd%7C38ae3bcd95794fd4addab42e1
> >>
> 495d55a%7C1%7C0%7C637535529408448819%7CUnknown%7CTWFpbGZsb3d8
> >>
> eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%
> >>
> 7C1000&amp;sdata=th0wxFgLnp3sT51ifrusbltvJ7NKgsDnGUtVXEfXHQs%3D&am
> >> p;reserved=0.

Re: [PATCH] ci-build: Protect bullseye container with KFAIL

[Jan Kiszka]

On 30.04.21 09:49, Moessbauer, Felix (T RDA IOT SES-DE) wrote:
> Hi Jan, Anton,
> 
> Actually I don't think the issue is related to binfmt_misc.
> If I get it right, binfmt_misc just dispatches and starts the arm binary with qemu-arm-static.
> 
> The bug (missing handling of syscall) is in the qemu binary itself.
> Neither mmdebstrap nor namespace support in binfmt_misc can help here.

The problem falls into two parts:

 - we need a newer qemu-user-static package than what buster ships in
   order to extend/fix the syscall emulation
 - binfmt_misc needs to use the right binary, not a stale one due to
   "fix binary" [1]

For former is what kas already does (buster-backports package). But
maybe our version in 2.4 is now too old. I'll check locally.

The latter is a problem when sharing the building kernel with multiple
jobs AND permitting those jobs to bring their own qemu-user-static
binary. That is the model of kas-container, to avoid contamination from
the docker host distro (which may not even be Debian). But it tends to
breaks due to missing isolation of the containers /wrt binfmt_misc.

BTW, if there is a way to detect that a wrong version of
qemu-user-static would actually be used due to "fix binary", I'm happy
to take at least a warning patch for kas-isar (docker entrypoint),
possibly even an overwrite to upgrade (although that may break builds
running in parallel).

Jan

[1] https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html

> 
> The good thing is, that we can detect the broken qemu version and issue a warning.
> But that just spams logs as the do_install_builddeps is a heavily used task...
> 
> Best regards,
> Felix
> 
>> -----Original Message-----
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>> Sent: Thursday, April 29, 2021 7:46 PM
>> To: Moessbauer, Felix (T RDA IOT SES-DE) <felix.moessbauer@siemens.com>;
>> Anton Mikanovich <amikan@ilbers.de>
>> Subject: Re: [PATCH] ci-build: Protect bullseye container with KFAIL
>>
>> On 29.04.21 15:42, Moessbauer, Felix (T RDA IOT SES-DE) wrote:
>>> Hi Anton,
>>>
>>> I can confirm that the CI patch does not solve the issue (on mc:qemuarm64-
>> focal).
>>> In case of the gitlab-ci, the KAS image provides the qemu-user-static.
>>
>> ...as along as the host didn't have an older version already used (e.g.
>> from an older container). By default, binfmt_misc is sticky /wrt the first used
>> interpreter binary. When unsure, clean all binfmt settings, then use the latest
>> container image.
>>
>>>
>>> I just tried to install qemu-user-static from buster-backports into the KAS
>> container and that solved the issue.
>>> Maybe Jan might have a look into that. IMO this would be way better than
>> trying to work-around in ISAR.
>>
>> Sure, this is a host (or build container) issue, nothing for Isar itself.
>>
>> Eventually, mmdebstrap in chrootless mode should help to overcome such
>> issues. Or proper namespace support for binfmt_misc.
>>
>> Jan
>>
>>>
>>> Best regards,
>>> Felix
>>>
>>>> -----Original Message-----
>>>> From: isar-users@googlegroups.com <isar-users@googlegroups.com> On
>>>> Behalf Of Anton Mikanovich
>>>> Sent: Friday, April 9, 2021 10:15 AM
>>>> To: isar-users@googlegroups.com
>>>> Cc: Anton Mikanovich <amikan@ilbers.de>
>>>> Subject: [PATCH] ci-build: Protect bullseye container with KFAIL
>>>>
>>>> Bullseye building issue still not resolved, so container-bullseye fails with error:
>>>>
>>>>> fakeroot debian/rules clean
>>>>> semop(1): encountered an error: Function not implemented
>>>>
>>>> This issue should be already fixed in host qemu/1:5.0-9 but we still
>>>> experience the same with qemu/1:5.2+dfsg-3~bpo10+1. Further research
>>>> will be done, so temporary fix for ci-build.sh introduced.
>>>>
>>>> Bug report:
>>>> https://bug
>>>>
>> s.deb%2F&amp;data=04%7C01%7Cfelix.moessbauer%40siemens.com%7C92bd
>> 09c5
>>>>
>> 68bb45136a1008d90b37b6b5%7C38ae3bcd95794fd4addab42e1495d55a%7C1%
>> 7C0%7
>>>>
>> C637553156176955322%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwM
>> DAiLCJQI
>>>>
>> joiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=Q9A9N5
>> m%2B
>>>> BTq%2BzEJI4Sfkp6aInSDj8qgPZ3SZlh6t7WE%3D&amp;reserved=0
>>>> ian.org%2Fcgi-
>>>>
>> bin%2Fbugreport.cgi%3Fbug%3D965109&amp;data=04%7C01%7Cfelix.moessba
>>>>
>> uer%40siemens.com%7C4c4f7801dd2b4db1c5a808d8fb2fa8dd%7C38ae3bcd95
>>>>
>> 794fd4addab42e1495d55a%7C1%7C0%7C637535529408448819%7CUnknown%
>>>>
>> 7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL
>>>>
>> CJXVCI6Mn0%3D%7C1000&amp;sdata=pvHkywD1yglmoyG7TOjSaTTYWtFOZL%
>>>> 2FqAlrG4UOjpQM%3D&amp;reserved=0
>>>>
>>>> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
>>>> ---
>>>>  scripts/ci_build.sh | 9 +++++++++
>>>>  1 file changed, 9 insertions(+)
>>>>
>>>> diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh index
>>>> 6445b10..adc22e4
>>>> 100755
>>>> --- a/scripts/ci_build.sh
>>>> +++ b/scripts/ci_build.sh
>>>> @@ -58,6 +58,9 @@ TARGETS_SET_BULLSEYE="\
>> TARGETS_CONTAINERS="\
>>>>      mc:container-stretch:isar-image-base \
>>>>      mc:container-buster:isar-image-base \
>>>> +"
>>>> +
>>>> +TARGETS_CONTAINERS_BULLSEYE="\
>>>>      mc:container-bullseye:isar-image-base \  "
>>>>
>>>> @@ -259,6 +262,12 @@ sed -i -e 's/\(IMAGE_INSTALL = .*\)
>>>> example-module- ${KERNEL_NAME}\(.*\)/\1\2/g'
>>>>  sed -i -e 's/\(IMAGE_INSTALL = .*\) enable-fsck\(.*\)/\1\2/g'
>>>> conf/local.conf bitbake $BB_ARGS $TARGETS_CONTAINERS  while [ -e
>>>> bitbake.sock ]; do sleep 1; done
>>>> +if bitbake $BB_ARGS $TARGETS_CONTAINERS_BULLSEYE; then
>>>> +    echo "bullseye container: PASSED"
>>>> +else
>>>> +    echo "bullseye container: KFAIL"
>>>> +fi
>>>> +while [ -e bitbake.sock ]; do sleep 1; done
>>>>  # and SDK container image creation
>>>>  echo 'SDK_FORMATS = "docker-archive"' >> conf/local.conf  bitbake
>>>> $BB_ARGS -c do_populate_sdk mc:container-stretch:isar-image-base
>>>> --
>>>> 2.20.1
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "isar-users" group.
>>>> To unsubscribe from this group and stop receiving emails from it,
>>>> send an email to isar-users+unsubscribe@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://gro
>>>>
>> ups.g%2F&amp;data=04%7C01%7Cfelix.moessbauer%40siemens.com%7C92bd
>> 09c5
>>>>
>> 68bb45136a1008d90b37b6b5%7C38ae3bcd95794fd4addab42e1495d55a%7C1%
>> 7C0%7
>>>>
>> C637553156176955322%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwM
>> DAiLCJQI
>>>>
>> joiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=u3Q8XDu
>> KM0
>>>> IxNW3DetVika0M3D73yWxn8Nd3TIfpb1Q%3D&amp;reserved=0
>>>> oogle.com%2Fd%2Fmsgid%2Fisar-users%2F20210409081520.399406-1-
>>>>
>> amikan%2540ilbers.de&amp;data=04%7C01%7Cfelix.moessbauer%40siemens.c
>>>>
>> om%7C4c4f7801dd2b4db1c5a808d8fb2fa8dd%7C38ae3bcd95794fd4addab42e1
>>>>
>> 495d55a%7C1%7C0%7C637535529408448819%7CUnknown%7CTWFpbGZsb3d8
>>>>
>> eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%
>>>>
>> 7C1000&amp;sdata=th0wxFgLnp3sT51ifrusbltvJ7NKgsDnGUtVXEfXHQs%3D&am
>>>> p;reserved=0.


-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: [PATCH] ci-build: Protect bullseye container with KFAIL

[Jan Kiszka]

On 30.04.21 19:57, Jan Kiszka wrote:
> On 30.04.21 09:49, Moessbauer, Felix (T RDA IOT SES-DE) wrote:
>> Hi Jan, Anton,
>>
>> Actually I don't think the issue is related to binfmt_misc.
>> If I get it right, binfmt_misc just dispatches and starts the arm binary with qemu-arm-static.
>>
>> The bug (missing handling of syscall) is in the qemu binary itself.
>> Neither mmdebstrap nor namespace support in binfmt_misc can help here.
> 
> The problem falls into two parts:
> 
>  - we need a newer qemu-user-static package than what buster ships in
>    order to extend/fix the syscall emulation
>  - binfmt_misc needs to use the right binary, not a stale one due to
>    "fix binary" [1]
> 
> For former is what kas already does (buster-backports package). But
> maybe our version in 2.4 is now too old. I'll check locally.
> 
> The latter is a problem when sharing the building kernel with multiple
> jobs AND permitting those jobs to bring their own qemu-user-static
> binary. That is the model of kas-container, to avoid contamination from
> the docker host distro (which may not even be Debian). But it tends to
> breaks due to missing isolation of the containers /wrt binfmt_misc.
> 
> BTW, if there is a way to detect that a wrong version of
> qemu-user-static would actually be used due to "fix binary", I'm happy
> to take at least a warning patch for kas-isar (docker entrypoint),
> possibly even an overwrite to upgrade (although that may break builds
> running in parallel).
> 

We have an extra mess here:

Because of an incompletely definition of machine/container.conf, we were
building containers for random archs, armhf in this case. I've a patch
for that under test, confining containers to amd64 for now (but that
could be enhanced in the future). It should move
container-amd64-bullseye out of the focus.

Still, I would expect that qemuarmhf-bullseye has the qemu-user-static
issue as well when an incorrect version is fixed-installed on the host.
Validating...

Jan

> Jan
> 
> [1] https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html
> 
>>
>> The good thing is, that we can detect the broken qemu version and issue a warning.
>> But that just spams logs as the do_install_builddeps is a heavily used task...
>>
>> Best regards,
>> Felix
>>
>>> -----Original Message-----
>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>> Sent: Thursday, April 29, 2021 7:46 PM
>>> To: Moessbauer, Felix (T RDA IOT SES-DE) <felix.moessbauer@siemens.com>;
>>> Anton Mikanovich <amikan@ilbers.de>
>>> Subject: Re: [PATCH] ci-build: Protect bullseye container with KFAIL
>>>
>>> On 29.04.21 15:42, Moessbauer, Felix (T RDA IOT SES-DE) wrote:
>>>> Hi Anton,
>>>>
>>>> I can confirm that the CI patch does not solve the issue (on mc:qemuarm64-
>>> focal).
>>>> In case of the gitlab-ci, the KAS image provides the qemu-user-static.
>>>
>>> ...as along as the host didn't have an older version already used (e.g.
>>> from an older container). By default, binfmt_misc is sticky /wrt the first used
>>> interpreter binary. When unsure, clean all binfmt settings, then use the latest
>>> container image.
>>>
>>>>
>>>> I just tried to install qemu-user-static from buster-backports into the KAS
>>> container and that solved the issue.
>>>> Maybe Jan might have a look into that. IMO this would be way better than
>>> trying to work-around in ISAR.
>>>
>>> Sure, this is a host (or build container) issue, nothing for Isar itself.
>>>
>>> Eventually, mmdebstrap in chrootless mode should help to overcome such
>>> issues. Or proper namespace support for binfmt_misc.
>>>
>>> Jan
>>>
>>>>
>>>> Best regards,
>>>> Felix
>>>>
>>>>> -----Original Message-----
>>>>> From: isar-users@googlegroups.com <isar-users@googlegroups.com> On
>>>>> Behalf Of Anton Mikanovich
>>>>> Sent: Friday, April 9, 2021 10:15 AM
>>>>> To: isar-users@googlegroups.com
>>>>> Cc: Anton Mikanovich <amikan@ilbers.de>
>>>>> Subject: [PATCH] ci-build: Protect bullseye container with KFAIL
>>>>>
>>>>> Bullseye building issue still not resolved, so container-bullseye fails with error:
>>>>>
>>>>>> fakeroot debian/rules clean
>>>>>> semop(1): encountered an error: Function not implemented
>>>>>
>>>>> This issue should be already fixed in host qemu/1:5.0-9 but we still
>>>>> experience the same with qemu/1:5.2+dfsg-3~bpo10+1. Further research
>>>>> will be done, so temporary fix for ci-build.sh introduced.
>>>>>
>>>>> Bug report:
>>>>> https://bug
>>>>>
>>> s.deb%2F&amp;data=04%7C01%7Cfelix.moessbauer%40siemens.com%7C92bd
>>> 09c5
>>>>>
>>> 68bb45136a1008d90b37b6b5%7C38ae3bcd95794fd4addab42e1495d55a%7C1%
>>> 7C0%7
>>>>>
>>> C637553156176955322%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwM
>>> DAiLCJQI
>>>>>
>>> joiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=Q9A9N5
>>> m%2B
>>>>> BTq%2BzEJI4Sfkp6aInSDj8qgPZ3SZlh6t7WE%3D&amp;reserved=0
>>>>> ian.org%2Fcgi-
>>>>>
>>> bin%2Fbugreport.cgi%3Fbug%3D965109&amp;data=04%7C01%7Cfelix.moessba
>>>>>
>>> uer%40siemens.com%7C4c4f7801dd2b4db1c5a808d8fb2fa8dd%7C38ae3bcd95
>>>>>
>>> 794fd4addab42e1495d55a%7C1%7C0%7C637535529408448819%7CUnknown%
>>>>>
>>> 7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL
>>>>>
>>> CJXVCI6Mn0%3D%7C1000&amp;sdata=pvHkywD1yglmoyG7TOjSaTTYWtFOZL%
>>>>> 2FqAlrG4UOjpQM%3D&amp;reserved=0
>>>>>
>>>>> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
>>>>> ---
>>>>>  scripts/ci_build.sh | 9 +++++++++
>>>>>  1 file changed, 9 insertions(+)
>>>>>
>>>>> diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh index
>>>>> 6445b10..adc22e4
>>>>> 100755
>>>>> --- a/scripts/ci_build.sh
>>>>> +++ b/scripts/ci_build.sh
>>>>> @@ -58,6 +58,9 @@ TARGETS_SET_BULLSEYE="\
>>> TARGETS_CONTAINERS="\
>>>>>      mc:container-stretch:isar-image-base \
>>>>>      mc:container-buster:isar-image-base \
>>>>> +"
>>>>> +
>>>>> +TARGETS_CONTAINERS_BULLSEYE="\
>>>>>      mc:container-bullseye:isar-image-base \  "
>>>>>
>>>>> @@ -259,6 +262,12 @@ sed -i -e 's/\(IMAGE_INSTALL = .*\)
>>>>> example-module- ${KERNEL_NAME}\(.*\)/\1\2/g'
>>>>>  sed -i -e 's/\(IMAGE_INSTALL = .*\) enable-fsck\(.*\)/\1\2/g'
>>>>> conf/local.conf bitbake $BB_ARGS $TARGETS_CONTAINERS  while [ -e
>>>>> bitbake.sock ]; do sleep 1; done
>>>>> +if bitbake $BB_ARGS $TARGETS_CONTAINERS_BULLSEYE; then
>>>>> +    echo "bullseye container: PASSED"
>>>>> +else
>>>>> +    echo "bullseye container: KFAIL"
>>>>> +fi
>>>>> +while [ -e bitbake.sock ]; do sleep 1; done
>>>>>  # and SDK container image creation
>>>>>  echo 'SDK_FORMATS = "docker-archive"' >> conf/local.conf  bitbake
>>>>> $BB_ARGS -c do_populate_sdk mc:container-stretch:isar-image-base
>>>>> --
>>>>> 2.20.1
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "isar-users" group.
>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>> send an email to isar-users+unsubscribe@googlegroups.com.
>>>>> To view this discussion on the web visit
>>>>> https://gro
>>>>>
>>> ups.g%2F&amp;data=04%7C01%7Cfelix.moessbauer%40siemens.com%7C92bd
>>> 09c5
>>>>>
>>> 68bb45136a1008d90b37b6b5%7C38ae3bcd95794fd4addab42e1495d55a%7C1%
>>> 7C0%7
>>>>>
>>> C637553156176955322%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwM
>>> DAiLCJQI
>>>>>
>>> joiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=u3Q8XDu
>>> KM0
>>>>> IxNW3DetVika0M3D73yWxn8Nd3TIfpb1Q%3D&amp;reserved=0
>>>>> oogle.com%2Fd%2Fmsgid%2Fisar-users%2F20210409081520.399406-1-
>>>>>
>>> amikan%2540ilbers.de&amp;data=04%7C01%7Cfelix.moessbauer%40siemens.c
>>>>>
>>> om%7C4c4f7801dd2b4db1c5a808d8fb2fa8dd%7C38ae3bcd95794fd4addab42e1
>>>>>
>>> 495d55a%7C1%7C0%7C637535529408448819%7CUnknown%7CTWFpbGZsb3d8
>>>>>
>>> eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%
>>>>>
>>> 7C1000&amp;sdata=th0wxFgLnp3sT51ifrusbltvJ7NKgsDnGUtVXEfXHQs%3D&am
>>>>> p;reserved=0.
> 
> 

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux