Re: [PATCH v4 0/5] expand-on-first-boot btrfs and CI testing

[Henning Schild <henning.schild@siemens.com>]

Am Wed, 21 Dec 2022 13:25:29 +0300
schrieb Uladzimir Bely <ubely@ilbers.de>:

> In the email from Tuesday, 20 December 2022 18:08:49 +03 user Henning
> Schild wrote:
> > Am Tue, 20 Dec 2022 12:37:24 +0300
> > 
> > schrieb Uladzimir Bely <ubely@ilbers.de>:  
> > > In the email from Tuesday, 13 December 2022 13:25:26 +03 user
> > > Henning
> > > 
> > > Schild wrote:  
> > > > Hi all,
> > > > 
> > > > i decided to merge the re-introduction of btrfs with the CI
> > > > series i had going on. Because that is all so important yet
> > > > seemingly fragile that we really do not want any big change
> > > > without CI.
> > > > 
> > > > this is based on
> > > > "expand-on-first-boot: switch back away from systemd-growfs"
> > > > which should be merged because it breaks several machines like
> > > > rpis, with the udev race we found lately
> > > > 
> > > > It is tested manually in qemu, especially debian-bullseye (btrfs
> > > > candidate) and on that formerly affected rpi.
> > > > In addition i pushed it here to trigger CI runs at ilbers
> > > > already:
> > > > https://github.com/henning-schild-work/isar/tree/henning/ilbers-ci
> > > > 
> > > > I get the feeling we really need a way to log in to the images
> > > > running in CI and run test code there. Only waiting for the
> > > > login prompt and hoping for bits in the kernel log is not
> > > > powerful enough do perform real tests. That is something to
> > > > improve at some point.  
> > > 
> > > Hello Henning.
> > > 
> > > We are currently working on the downstream that requires such
> > > kind of access to the machine running under qemu. We want to
> > > implement the essential parts of functionality in Isar in the
> > > nearest future:
> > > - remove scripts/start_vm in favor of testsuite/start_vm.py (or,
> > > at least, scripts/start_vm will be just a wrapper)
> > > - allow custom image names (not only hardcoded 'isar-image-base')
> > > to be tested
> > > - support checking the image for service status (at least)
> > > - support any custom commands execution (at most)
> > > 
> > > Actually, the latest thing should also cover previous "service
> > > status checking". Also, when implemented, we will be able to
> > > completely remove boot log parsing that is used currently.
> > > 
> > > The main idea of the upcoming implementation is executing custom
> > > commands over SSH in "test" thread while the machine is kept
> > > running in "main" thread.  
> > 
> > For a truly generic way i strongly suggest looking into lava or
> > tbot2. We use lava in many downstream projects.
> >   
> 
> I've looked at some of them and found that lava seems to be used in
> `xenomai- images`. But I didn't find anything using tbot2. Could you
> point at some example projects (if they are not closed-source)?

AFAIK lava is the dominant player in that space, it is also behind
kernel-ci. I just wanted to mention tbot as an alternative i know, but
i do not know users. I called it tbot2 because a very early version of
its old implementation is actually used at Siemens in a project we ran
like this forever and never cared to switch to lava.

Its history is similar to the one of Isar. Once custom made for some
project at Siemens and later it became more abstract and OSS.

> And saying "tbot2" did you mean https://tbot.tools/ or something
> else? At first glance it looks something promising and able to be
> integrated to our python-based testsuite. While lava looks more
> complicated. 

Yes that is the one. I did add Harald maybe he can share some insights
on maturity and users or say a few words on how it compares to lava.

But again, i think in isar CI we might not need such complex things. A
way to start a bunch of qemus might be good enough, only we want to
gain a way to execute code inside them ... via serial or ssh.
I would first look what OE does.

When it comes to testing on Hardware, lava or maybe tbot are what you
want. But as long as it is only qemu ... one might not want either of
the two just yet.

Henning

> > For the mainly CI topics that might be overkill, maybe OEs scripts
> > might be the better option. Not sure how they do that but for some
> > of the fstab changes we got merged there lately we had to adopt
> > test code that was running in the context of the newly assembled
> > rootfs.
> > 
> > Henning
> >   
> > > > regards,
> > > > Henning
> > > > 
> > > > Am Tue, 13 Dec 2022 11:15:04 +0100
> > > > 
> > > > schrieb henning.schild@siemens.com:  
> > > > > From: Henning Schild <henning.schild@siemens.com>
> > > > > 
> > > > > changes since v3:
> > > > >  - rebased on "expand-on-first-boot: switch back away from
> > > > > 
> > > > > systemd-growfs"
> > > > > 
> > > > >  - implement btrfs resizing and CI testing it
> > > > >  - put the padding wks entry into an include
> > > > > 
> > > > > changes since v2:
> > > > >  - add python namespace prefix for the bitbake get functions
> > > > >  - install only in images that use WKS_FILEs that have the
> > > > > "gap"
> > > > > 
> > > > > changes since v1:
> > > > >  - complete rewrite to not focus only on the tests but rather
> > > > > the 
> > > > >    feature for all examples
> > > > > 
> > > > > This first was only about testing but meanwhile has a focus on
> > > > > enabling the expand feature in all example images anyhow, and
> > > > > later look for traces of that happening in CI test where we
> > > > > can. We install the application in any image and in the wic
> > > > > images add some scratch space at the end. That will enable
> > > > > improved interactive use of all those wic images, so makes
> > > > > sense apart from any testing. Later use the log reading
> > > > > feature of the qemu tests to look for traces of successful
> > > > > resizing for non ubuntu images. In ubuntu the
> > > > > console_loglevel is not verbose enough to find traces.
> > > > > 
> > > > > Henning Schild (5):
> > > > >   expand-on-first-boot: support resizing a btrfs
> > > > >   meta-isar: introduce an example to use btrfs
> > > > >   CI: improve cibuilder readability
> > > > >   meta-isar: install expand-on-first-boot in most images and
> > > > > add
> > > > > 
> > > > > space CI: expect a message about filesystem resize vom expand
> > > > > script
> > > > > 
> > > > >  meta-isar/conf/machine/qemuamd64.conf         |  6 ++++
> > > > >  meta-isar/conf/machine/virtualbox.conf        |  1 +
> > > > >  meta-isar/conf/machine/vmware.conf            |  1 +
> > > > >  .../conf/multiconfig/qemuarm-bookworm.conf    |  1 +
> > > > >  .../lib/wic/canned-wks/expand-padding.wks.inc |  5 +++
> > > > >  .../lib/wic/canned-wks/sdimage-efi-btrfs.wks  | 11 ++++++
> > > > >  .../lib/wic/canned-wks/sdimage-efi-sd.wks     |  2 ++
> > > > >  .../lib/wic/canned-wks/sdimage-efi.wks        |  2 ++
> > > > >  .../files/expand-last-partition.sh            | 35
> > > > > 
> > > > > ++++++++++++++++--- testsuite/cibuilder.py
> > > > > 
> > > > > | 28 ++++++++++++--- 10 files changed, 83 insertions(+), 9
> > > > > 
> > > > > deletions(-)
> > > > > 
> > > > >  create mode 100644
> > > > > 
> > > > > meta-isar/scripts/lib/wic/canned-wks/expand-padding.wks.inc
> > > > > create mode 100644
> > > > > meta-isar/scripts/lib/wic/canned-wks/sdimage-efi-btrfs.wks  
> 
>