qemu test with start_vm fails

qemu test with start_vm fails

[Andreas Reichel]

Hi,

I built

multiconfig:qemuarm-stretch:isar-image-base

and wanted to test the image with

start_vm -a arm -d stretch

which failed inside my docker container so I extracted the qemu command
generated and started qemu outside of docker.

This failed to work because the generated ext4 image had no
space left and the kernel root parameter is set to "rw" by the
output of start_vm script, causing systemd to hang.

I thought, there was a CI that tested the builds with qemu on a
regular basis?

I could start the image with "root=/dev/vda ro" parameter.

If such basic things are not tested regularly this can cost developers a
lot of time...

kind regards
Andreas
-- 
Andreas Reichel
Dipl.-Phys. (Univ.)
Software Consultant

Andreas.Reichel@tngtech.com, +49-174-3180074
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterfoehring
Geschaeftsfuehrer: Henrik Klagges, Dr. Robert Dahlke, Gerhard Mueller
Sitz: Unterfoehring * Amtsgericht Muenchen * HRB 135082

Re: qemu test with start_vm fails

[Henning Schild]

Am Wed, 28 Nov 2018 17:25:24 +0100
schrieb "[ext] Andreas Reichel" <andreas.reichel.ext@siemens.com>:

> Hi,
> 
> I built
> 
> multiconfig:qemuarm-stretch:isar-image-base
> 
> and wanted to test the image with
> 
> start_vm -a arm -d stretch
> 
> which failed inside my docker container so I extracted the qemu
> command generated and started qemu outside of docker.
> 
> This failed to work because the generated ext4 image had no
> space left and the kernel root parameter is set to "rw" by the
> output of start_vm script, causing systemd to hang.

That is interesting because we should have 64M of ROOTFS_EXTRA in
there, did you play with that variable?

> I thought, there was a CI that tested the builds with qemu on a
> regular basis?

There is, unfortunately the results are not publicly available.

Have a look at scripts/ci_build.sh and scripts/vm_smoke_test

arm stretch should be covered, there has been some restructuring
lately, but this is a prime target ... not sure it is using the ext4
class

> I could start the image with "root=/dev/vda ro" parameter.
> 
> If such basic things are not tested regularly this can cost
> developers a lot of time...

You can say a lot of bad things about the testing, but i am almost
100% certain that is covered.
If we do not have a test booting an ext4 arm stretch, you could propose
one.

Henning
 
> kind regards
> Andreas

Re: qemu test with start_vm fails

[Maxim Yu. Osipov]

Hi Andreas,

I would suggest to double check your statements before blaming that 
something is not working

1. CI is performed on regular basis on isar-build.org.

The easy way to run it on your debian PC just call
* isar$./scripts/ci_build.sh -q
* isar/build$ source isar-init-build-env
* isar/build$vm_smoke_test -q

You should see something like that:

vm_smoke_test -f -q
-------------------------------------------------
Testing Isar [stretch] image for [arm] machine:
Check login prompt: PASSED
Check example module: PASSED
-------------------------------------------------
Testing Isar [stretch] image for [amd64]

...

Recently the "faster" (to build/try reduced set of configurations) 
version of CI was introduced - just pass additional key '-f' to 
ci_build.sh and vm_smoke_test.

Moreover you are encouraged to run these scripts on your own before 
submitting your patches - this will save a lot of time to maintainer.

Instead launching on your own PC - active developers are granted account 
on Jenkins on isar-build.org - CI is automated by Jenkins. Just ask 
Baurzhan for an account.

2. CI is not launched in docker.

3. On my stretch laptop I've just checked out the latest next, built
multiconfig:qemuarm-stretch:isar-image-base and launched start_vm:

isar/build$ start_vm -a arm -d stretch
+ qemu-system-arm -m 1024M -M virt -cpu cortex-a15 -nographic -kernel 
/home/myo/work/isar/src/isar/build/tmp/deploy/images/qemuarm/isar-image-base-debian-stretch-qemuarm.vmlinuz-4.9.0-8-armmp 
-initrd 
/home/myo/work/isar/src/isar/build/tmp/deploy/images/qemuarm/isar-image-base-debian-stretch-qemuarm.initrd.img-4.9.0-8-armmp 
-append console=ttyAMA0 root=/dev/vda rw -drive 
file=/home/myo/work/isar/src/isar/build/tmp/deploy/images/qemuarm/isar-image-base-debian-stretch-qemuarm.ext4.img,if=none,format=raw,id=hd0 
-device virtio-blk-device,drive=hd0
[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 4.9.0-8-armmp 
(debian-kernel@lists.debian.org) (gcc version 6.3.0 20170516 (Debian 
6.3.0-18+deb9u1) ) #1 SMP Debian 4.9.130-2 (2018-10-27)

[snip]

After a while (wait for some 3-4 min) I've got

Debian GNU/Linux 9 isar ttyAMA0

isar login:


When reporting a problem please provide more details about your 
environment (host/target/distro/Isar tree commit id etc).

Maxim.

On 11/28/18 7:25 PM, Andreas Reichel wrote:
> Hi,
> 
> I built
> 
> multiconfig:qemuarm-stretch:isar-image-base
> 
> and wanted to test the image with
> 
> start_vm -a arm -d stretch
> 
> which failed inside my docker container so I extracted the qemu command
> generated and started qemu outside of docker.
> 
> This failed to work because the generated ext4 image had no
> space left and the kernel root parameter is set to "rw" by the
> output of start_vm script, causing systemd to hang.
> 
> I thought, there was a CI that tested the builds with qemu on a
> regular basis?
> 
> I could start the image with "root=/dev/vda ro" parameter.
> 
> If such basic things are not tested regularly this can cost developers a
> lot of time...
> 
> kind regards
> Andreas
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: qemu test with start_vm fails

[Jan Kiszka]

On 28.11.18 18:34, Maxim Yu. Osipov wrote:
> Hi Andreas,
> 
> I would suggest to double check your statements before blaming that something is
> not working
> 
> 1. CI is performed on regular basis on isar-build.org.
> 
> The easy way to run it on your debian PC just call
> * isar$./scripts/ci_build.sh -q
> * isar/build$ source isar-init-build-env
> * isar/build$vm_smoke_test -q
> 
> You should see something like that:
> 
> vm_smoke_test -f -q
> -------------------------------------------------
> Testing Isar [stretch] image for [arm] machine:
> Check login prompt: PASSED
> Check example module: PASSED
> -------------------------------------------------
> Testing Isar [stretch] image for [amd64]
> 
> ...
> 
> Recently the "faster" (to build/try reduced set of configurations) version of CI
> was introduced - just pass additional key '-f' to ci_build.sh and vm_smoke_test.
> 
> Moreover you are encouraged to run these scripts on your own before submitting
> your patches - this will save a lot of time to maintainer.
> 
> Instead launching on your own PC - active developers are granted account on
> Jenkins on isar-build.org - CI is automated by Jenkins. Just ask Baurzhan for an
> account.
> 
> 2. CI is not launched in docker.
> 
> 3. On my stretch laptop I've just checked out the latest next, built
> multiconfig:qemuarm-stretch:isar-image-base and launched start_vm:
> 
> isar/build$ start_vm -a arm -d stretch
> + qemu-system-arm -m 1024M -M virt -cpu cortex-a15 -nographic -kernel
> /home/myo/work/isar/src/isar/build/tmp/deploy/images/qemuarm/isar-image-base-debian-stretch-qemuarm.vmlinuz-4.9.0-8-armmp
> -initrd
> /home/myo/work/isar/src/isar/build/tmp/deploy/images/qemuarm/isar-image-base-debian-stretch-qemuarm.initrd.img-4.9.0-8-armmp
> -append console=ttyAMA0 root=/dev/vda rw -drive
> file=/home/myo/work/isar/src/isar/build/tmp/deploy/images/qemuarm/isar-image-base-debian-stretch-qemuarm.ext4.img,if=none,format=raw,id=hd0
> -device virtio-blk-device,drive=hd0
> [    0.000000] Booting Linux on physical CPU 0x0
> [    0.000000] Linux version 4.9.0-8-armmp (debian-kernel@lists.debian.org) (gcc
> version 6.3.0 20170516 (Debian 6.3.0-18+deb9u1) ) #1 SMP Debian 4.9.130-2
> (2018-10-27)
> 
> [snip]
> 
> After a while (wait for some 3-4 min) I've got
> 
> Debian GNU/Linux 9 isar ttyAMA0
> 
> isar login:
> 
> 
> When reporting a problem please provide more details about your environment
> (host/target/distro/Isar tree commit id etc).
> 

We discussed this locally a bit, and one explanation could be that something
goes wrong with the overall image size calculation run on the hos due to the
underlying file system. Andreas will follow-up with detailed analysis as time
permits.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: qemu test with start_vm fails

[Andreas Reichel]

On Wed, Nov 28, 2018 at 08:34:58PM +0300, Maxim Yu. Osipov wrote:
> Hi Andreas,
> 
> I would suggest to double check your statements before blaming that
> something is not working
> 
Hi,

well unfortunately I was correct. There is an error in Isar:
the offline cache is included in the root file system :)

There is a /base-apt in the resulting .ext4.img.

This way I got a rootfs of 720 MB with no free space, despite the
64M extra space. (file system overhead).

In your CI you obviously do not use the offline cache feature,
that's why in CI this did not happen.

vG
Andreas
-- 
Andreas Reichel
Dipl.-Phys. (Univ.)
Software Consultant

Andreas.Reichel@tngtech.com, +49-174-3180074
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterfoehring
Geschaeftsfuehrer: Henrik Klagges, Dr. Robert Dahlke, Gerhard Mueller
Sitz: Unterfoehring * Amtsgericht Muenchen * HRB 135082

Re: qemu test with start_vm fails

[Maxim Yu. Osipov]

On 11/29/18 3:20 PM, Andreas Reichel wrote:
> On Wed, Nov 28, 2018 at 08:34:58PM +0300, Maxim Yu. Osipov wrote:
>> Hi Andreas,
>>
>> I would suggest to double check your statements before blaming that
>> something is not working
>>
> Hi,
> 
> well unfortunately I was correct. There is an error in Isar:
> the offline cache is included in the root file system :)

In your initial report you stated:

> -------- Forwarded Message --------
> Subject: qemu test with start_vm fails
> Date: Wed, 28 Nov 2018 17:25:24 +0100
> From: Andreas Reichel <andreas.reichel.ext@siemens.com>
> To: isar-users@googlegroups.com
> 
> Hi,
> 
> I built
> 
> multiconfig:qemuarm-stretch:isar-image-base
> 
> and wanted to test the image with
> 
> start_vm -a arm -d stretch
> 
> which failed inside my docker container so I extracted the qemu command
> generated and started qemu outside of docker.
> 
> This failed to work because the generated ext4 image had no
> space left and the kernel root parameter is set to "rw" by the
> output of start_vm script, causing systemd to hang.

You said that you were unable to launch 'start_vm -a arm -d stretch'
without providing details of environment/feature enabled/commit id used.

At least now I know that you use offline cache (BTW, offline cache 
feature is broken in current next, so I may only guess on which commit 
id you base on).

I hope that it should be clear how to correctly report the problem next 
time.

> There is a /base-apt in the resulting .ext4.img.

I'll look on that and fix it if necessary (previously we had images with 
not cleaned up /var/cache/apt).

> This way I got a rootfs of 720 MB with no free space, despite the
> 64M extra space. (file system overhead).
> 
> In your CI you obviously do not use the offline cache feature,
> that's why in CI this did not happen.

I've just checked-in into 'next' the patch debe3392 "scripts/ci: enable 
use of cached base repository for fast build" and it passes OK in 'next' 
with the reverted patch d40a9ac0.

The easy way to run it on your debian PC just call

* isar$./scripts/ci_build.sh -q -f
* isar$ source isar-init-build-env
* isar/build$vm_smoke_test -q -f


Maxim.


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: qemu test with start_vm fails

[Jan Kiszka]

On 29.11.18 15:25, Maxim Yu. Osipov wrote:
> On 11/29/18 3:20 PM, Andreas Reichel wrote:
>> There is a /base-apt in the resulting .ext4.img.
> 
> I'll look on that and fix it if necessary (previously we had images with not 
> cleaned up /var/cache/apt).

Already fixed, see the patch I sent some minutes ago.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: qemu test with start_vm fails

[Henning Schild]

Am Thu, 29 Nov 2018 13:20:17 +0100
schrieb Andreas Reichel <andreas.reichel.ext@siemens.com>:

> On Wed, Nov 28, 2018 at 08:34:58PM +0300, Maxim Yu. Osipov wrote:
> > Hi Andreas,
> > 
> > I would suggest to double check your statements before blaming that
> > something is not working
> >   
> Hi,
> 
> well unfortunately I was correct. There is an error in Isar:
> the offline cache is included in the root file system :)
> 
> There is a /base-apt in the resulting .ext4.img.
> 
> This way I got a rootfs of 720 MB with no free space, despite the
> 64M extra space. (file system overhead).

I think we now have a fix because we do not include the cache into the
image anymore. But that actually just hides the bug we have in the ext4
class, next time someone builds a 700M image the same thing will happen
again.

So we will also need a patch that prepares the ext4 for that case.
Things that could be done
 - mkfs -m 1 (instead of default 5)
 - multiply the size with an ext4 specific overhead factor
 - df assert the EXTRA while still mounted
 - double check what wic is doing, maybe use wic instead of ext4

Henning

> In your CI you obviously do not use the offline cache feature,
> that's why in CI this did not happen.
> 
> vG
> Andreas

Re: qemu test with start_vm fails

[Henning Schild]

Am Fri, 30 Nov 2018 10:06:39 +0100
schrieb "[ext] Henning Schild" <henning.schild@siemens.com>:

> Am Thu, 29 Nov 2018 13:20:17 +0100
> schrieb Andreas Reichel <andreas.reichel.ext@siemens.com>:
> 
> > On Wed, Nov 28, 2018 at 08:34:58PM +0300, Maxim Yu. Osipov wrote:  
> > > Hi Andreas,
> > > 
> > > I would suggest to double check your statements before blaming
> > > that something is not working
> > >     
> > Hi,
> > 
> > well unfortunately I was correct. There is an error in Isar:
> > the offline cache is included in the root file system :)
> > 
> > There is a /base-apt in the resulting .ext4.img.
> > 
> > This way I got a rootfs of 720 MB with no free space, despite the
> > 64M extra space. (file system overhead).  
> 
> I think we now have a fix because we do not include the cache into the
> image anymore. But that actually just hides the bug we have in the
> ext4 class, next time someone builds a 700M image the same thing will
> happen again.
> 
> So we will also need a patch that prepares the ext4 for that case.
> Things that could be done
>  - mkfs -m 1 (instead of default 5)
>  - multiply the size with an ext4 specific overhead factor
>  - df assert the EXTRA while still mounted
>  - double check what wic is doing, maybe use wic instead of ext4

In addition the "cp -r" will drop a lot of file attributes and should
be "cp -a". In fact it would be best to "mkfs.ext4 -d", like wic.
Using wic would result in a disk and not in a partition image, but that
should not be a problem. Older mkfs.ext4 versions do not support "-d",
maybe we need to finally drop that old raspbian ... 

Henning

> Henning
> 
> > In your CI you obviously do not use the offline cache feature,
> > that's why in CI this did not happen.
> > 
> > vG
> > Andreas  
>