[PATCH 0/8] CI rework of gitlab fast job

[PATCH 0/8] CI rework of gitlab fast job

[Henning Schild]

This series contains some changes to the CI as done on gitlab but also
affects testing in general. The main idea is to run qemu basically every
time we build an image (only fast for gitlab at the moment), so we do
not just build it without testing.
On the way i also improved shell style and removed some tests from the
fast set, since they seem more advanced and just waste time on "fast"

Henning Schild (8):
  CI: move to avocado to 99.0
  CI: fix shell coding style
  CI: install qemu-system when qemu testing is requested
  testsuite: remove Ccache test from "fast" set
  testsuite: remove Sdk test from "fast" set
  testsuite: remove ContainerImage test from "fast" set
  testsuite: remove ContainerSdk test from "fast" set
  testsuite: remove Sstate test from "fast" set

 .gitlab-ci.yml      |  2 +-
 scripts/ci_build.sh | 26 ++++++++++++++++----------
 testsuite/citest.py | 10 +++++-----
 3 files changed, 22 insertions(+), 16 deletions(-)

-- 
2.38.2

[PATCH 1/8] CI: move to avocado to 99.0

[Henning Schild]

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 scripts/ci_build.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index fa7208c2b64b..84a3e472e784 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -21,7 +21,7 @@ if ! command -v avocado > /dev/null; then
     rm -rf /tmp/avocado_venv
     virtualenv --python python3 /tmp/avocado_venv
     source /tmp/avocado_venv/bin/activate
-    pip install avocado-framework==96.0
+    pip install avocado-framework==99.0
 fi
 
 # Get Avocado build tests path
-- 
2.38.2

[PATCH 2/8] CI: fix shell coding style

[Henning Schild]

Apply some suggestions from shellcheck and make sure that runs fine.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 scripts/ci_build.sh | 19 ++++++++++---------
 1 file changed, 10 insertions(+), 9 deletions(-)

diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index 84a3e472e784..e5f20d4f0a1f 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -20,6 +20,7 @@ if ! command -v avocado > /dev/null; then
     sudo apt-get install -y virtualenv
     rm -rf /tmp/avocado_venv
     virtualenv --python python3 /tmp/avocado_venv
+    # shellcheck source=/dev/null
     source /tmp/avocado_venv/bin/activate
     pip install avocado-framework==99.0
 fi
@@ -33,8 +34,8 @@ BASE_DIR=./build
 # Check dependencies
 DEPENDENCIES="umoci skopeo"
 for prog in ${DEPENDENCIES} ; do
-    if [ ! -x "$(which $prog)" ] ; then
-        echo "missing $prog in PATH" >&2
+    if ! command -v "${prog}" > /dev/null; then
+        echo "missing ${prog} in PATH" >&2
     fi
 done
 
@@ -132,16 +133,16 @@ fi
 mkdir -p .config/avocado
 cat <<EOF > .config/avocado/avocado.conf
 [datadir.paths]
-base_dir = $(realpath $BASE_DIR)/
-test_dir = $(realpath $BASE_DIR)/tests
-data_dir = $(realpath $BASE_DIR)/data
-logs_dir = $(realpath $BASE_DIR)/job-results
+base_dir = $(realpath "${BASE_DIR}")/
+test_dir = $(realpath "${BASE_DIR}")/tests
+data_dir = $(realpath "${BASE_DIR}")/data
+logs_dir = $(realpath "${BASE_DIR}")/job-results
 EOF
 export VIRTUAL_ENV="./"
 
 # the real stuff starts here, trace commands from now on
 set -x
 
-avocado $VERBOSE run "$TESTSUITE_DIR/citest.py" \
-    -t $TAGS --nrunner-max-parallel-tasks=1 --disable-sysinfo \
-    -p quiet=$QUIET -p cross=$CROSS_BUILD -p time_to_wait=$TIMEOUT
+avocado ${VERBOSE} run "${TESTSUITE_DIR}/citest.py" \
+    -t "${TAGS}" --nrunner-max-parallel-tasks=1 --disable-sysinfo \
+    -p quiet="${QUIET}" -p cross="${CROSS_BUILD}" -p time_to_wait="${TIMEOUT}"
-- 
2.38.2

[PATCH 3/8] CI: install qemu-system when qemu testing is requested

[Henning Schild]

And also change gitlab ci config to request qemu testing.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 .gitlab-ci.yml      | 2 +-
 scripts/ci_build.sh | 5 +++++
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index fcdefa87bf66..dc8a01b9d1d6 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -27,7 +27,7 @@ fast-ci:
   except:
     - schedules
   script:
-    - scripts/ci_build.sh -q -f -n
+    - scripts/ci_build.sh -q -f
 
 full-ci:
   <<: *common-build
diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index e5f20d4f0a1f..1704ec17af89 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -127,6 +127,11 @@ fi
 
 if [ -n "$NORUN" ]; then
     TAGS="$TAGS,-startvm"
+else
+    if ! command -v qemu-system-x86 > /dev/null; then
+        sudo apt-get update -qq
+        sudo apt-get install -y qemu-system
+    fi
 fi
 
 # Provide working path
-- 
2.38.2

[PATCH 4/8] testsuite: remove Ccache test from "fast" set

[Henning Schild]

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 testsuite/citest.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 7aa1e6b5d12c..aa6441f5f6c3 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -53,7 +53,7 @@ class CcacheTest(CIBaseTest):
     """
     Test rebuild speed improve with ccache
 
-    :avocado: tags=ccache,fast,full
+    :avocado: tags=ccache,full
     """
     def test_ccache_rebuild(self):
         targets = ['mc:qemuamd64-bullseye:hello-isar']
-- 
2.38.2

Re: [PATCH 4/8] testsuite: remove Ccache test from "fast" set

[Jan Kiszka]

I don't vote against this, but I would like to see some "why" here (as
usual).

On 10.01.23 13:17, Henning Schild wrote:
> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> ---
>  testsuite/citest.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/testsuite/citest.py b/testsuite/citest.py
> index 7aa1e6b5d12c..aa6441f5f6c3 100755
> --- a/testsuite/citest.py
> +++ b/testsuite/citest.py
> @@ -53,7 +53,7 @@ class CcacheTest(CIBaseTest):
>      """
>      Test rebuild speed improve with ccache
>  
> -    :avocado: tags=ccache,fast,full
> +    :avocado: tags=ccache,full
>      """
>      def test_ccache_rebuild(self):
>          targets = ['mc:qemuamd64-bullseye:hello-isar']

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

[PATCH 5/8] testsuite: remove Sdk test from "fast" set

[Henning Schild]

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 testsuite/citest.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index aa6441f5f6c3..0c3e4d0b7835 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -118,7 +118,7 @@ class SdkTest(CIBaseTest):
     """
     In addition test SDK creation
 
-    :avocado: tags=sdk,fast,full
+    :avocado: tags=sdk,full
     """
     def test_sdk(self):
         targets = ['mc:qemuarm-bullseye:isar-image-base']
-- 
2.38.2

Re: [PATCH 5/8] testsuite: remove Sdk test from "fast" set

[Jan Kiszka]

Same here, even if it should be a copy from the previous patch.

On 10.01.23 13:17, Henning Schild wrote:
> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> ---
>  testsuite/citest.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/testsuite/citest.py b/testsuite/citest.py
> index aa6441f5f6c3..0c3e4d0b7835 100755
> --- a/testsuite/citest.py
> +++ b/testsuite/citest.py
> @@ -118,7 +118,7 @@ class SdkTest(CIBaseTest):
>      """
>      In addition test SDK creation
>  
> -    :avocado: tags=sdk,fast,full
> +    :avocado: tags=sdk,full
>      """
>      def test_sdk(self):
>          targets = ['mc:qemuarm-bullseye:isar-image-base']

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

Re: [PATCH 5/8] testsuite: remove Sdk test from "fast" set

[Henning Schild]

Sure i will provide some generic reasoning for all of them, maybe even
squash them.

As of now the one generic reasoning is in the cover letter and the
commits are intentionally separated so we can discuss the tests
individually if need be.

In a v2 i will likely squash them all and provide one generic reason
why they maybe should not be part of "fast".

Henning

Am Tue, 10 Jan 2023 13:38:57 +0100
schrieb Jan Kiszka <jan.kiszka@siemens.com>:

> Same here, even if it should be a copy from the previous patch.
> 
> On 10.01.23 13:17, Henning Schild wrote:
> > Signed-off-by: Henning Schild <henning.schild@siemens.com>
> > ---
> >  testsuite/citest.py | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/testsuite/citest.py b/testsuite/citest.py
> > index aa6441f5f6c3..0c3e4d0b7835 100755
> > --- a/testsuite/citest.py
> > +++ b/testsuite/citest.py
> > @@ -118,7 +118,7 @@ class SdkTest(CIBaseTest):
> >      """
> >      In addition test SDK creation
> >  
> > -    :avocado: tags=sdk,fast,full
> > +    :avocado: tags=sdk,full
> >      """
> >      def test_sdk(self):
> >          targets = ['mc:qemuarm-bullseye:isar-image-base']  
> 
> Jan
> 

[PATCH 6/8] testsuite: remove ContainerImage test from "fast" set

[Henning Schild]

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 testsuite/citest.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 0c3e4d0b7835..2191b6c0e718 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -251,7 +251,7 @@ class ContainerImageTest(CIBaseTest):
     """
     Test containerized images creation
 
-    :avocado: tags=containerbuild,fast,full,container
+    :avocado: tags=containerbuild,full,container
     """
     @skipUnless(UMOCI_AVAILABLE and SKOPEO_AVAILABLE, 'umoci/skopeo not found')
     def test_container_image(self):
-- 
2.38.2

[PATCH 7/8] testsuite: remove ContainerSdk test from "fast" set

[Henning Schild]

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 testsuite/citest.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 2191b6c0e718..13360a48067f 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -269,7 +269,7 @@ class ContainerSdkTest(CIBaseTest):
     """
     Test SDK container image creation
 
-    :avocado: tags=containersdk,fast,full,container
+    :avocado: tags=containersdk,full,container
     """
     @skipUnless(UMOCI_AVAILABLE and SKOPEO_AVAILABLE, 'umoci/skopeo not found')
     def test_container_sdk(self):
-- 
2.38.2

[PATCH 8/8] testsuite: remove Sstate test from "fast" set

[Henning Schild]

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 testsuite/citest.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 13360a48067f..9315fe78e71d 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -283,7 +283,7 @@ class SstateTest(CIBaseTest):
     """
     Test builds with artifacts taken from sstate cache
 
-    :avocado: tags=sstate,fast,full
+    :avocado: tags=sstate,full
     """
     def test_sstate(self):
         image_target = 'mc:qemuamd64-bullseye:isar-image-base'
-- 
2.38.2

Re: [PATCH 0/8] CI rework of gitlab fast job

[Henning Schild]

I did not dare touching any of this for a very long time. But we have
reached a state where the CI is just a better room heater and not
meaningful, at least when done with gitlab and running the fast.

There is much to be improved, this is just a start. It is tested on our
gitlab and we save about 1.5h per run while we now vm-test all images
we build.

Henning

Am Tue, 10 Jan 2023 13:17:40 +0100
schrieb Henning Schild <henning.schild@siemens.com>:

> This series contains some changes to the CI as done on gitlab but also
> affects testing in general. The main idea is to run qemu basically
> every time we build an image (only fast for gitlab at the moment), so
> we do not just build it without testing.
> On the way i also improved shell style and removed some tests from the
> fast set, since they seem more advanced and just waste time on "fast"
> 
> Henning Schild (8):
>   CI: move to avocado to 99.0
>   CI: fix shell coding style
>   CI: install qemu-system when qemu testing is requested
>   testsuite: remove Ccache test from "fast" set
>   testsuite: remove Sdk test from "fast" set
>   testsuite: remove ContainerImage test from "fast" set
>   testsuite: remove ContainerSdk test from "fast" set
>   testsuite: remove Sstate test from "fast" set
> 
>  .gitlab-ci.yml      |  2 +-
>  scripts/ci_build.sh | 26 ++++++++++++++++----------
>  testsuite/citest.py | 10 +++++-----
>  3 files changed, 22 insertions(+), 16 deletions(-)
> 

Re: [PATCH 0/8] CI rework of gitlab fast job

[Baurzhan Ismagulov]

On Tue, Jan 10, 2023 at 01:17:40PM +0100, Henning Schild wrote:
> This series contains some changes to the CI as done on gitlab but also
> affects testing in general. The main idea is to run qemu basically every
> time we build an image (only fast for gitlab at the moment), so we do
> not just build it without testing.
> On the way i also improved shell style and removed some tests from the
> fast set, since they seem more advanced and just waste time on "fast"

We are working on a new "dev" testsuite [1] which should be:

* reasonably fast,
* cover "main" arches (in separate testcases) and common (-ly broken) cases
* and kept green most of the time,

while keeping "fast" and "full" as maintainer suites (massive multiconfigs,
etc.).

So, your series is a good input what is not interesting for a typical
downstream. Let's discuss when Anton is back next week.

1. https://patchwork.isar-build.org/project/isar/list/?series=670&state=*

With kind regards,
Baurzhan

Re: [PATCH 0/8] CI rework of gitlab fast job

[Henning Schild]

Am Thu, 12 Jan 2023 13:23:17 +0100
schrieb Baurzhan Ismagulov <ibr@radix50.net>:

> On Tue, Jan 10, 2023 at 01:17:40PM +0100, Henning Schild wrote:
> > This series contains some changes to the CI as done on gitlab but
> > also affects testing in general. The main idea is to run qemu
> > basically every time we build an image (only fast for gitlab at the
> > moment), so we do not just build it without testing.
> > On the way i also improved shell style and removed some tests from
> > the fast set, since they seem more advanced and just waste time on
> > "fast"  
> 
> We are working on a new "dev" testsuite [1] which should be:
> 
> * reasonably fast,
> * cover "main" arches (in separate testcases) and common (-ly broken)
> cases
> * and kept green most of the time,
> 
> while keeping "fast" and "full" as maintainer suites (massive
> multiconfigs, etc.).
> 
> So, your series is a good input what is not interesting for a typical
> downstream. Let's discuss when Anton is back next week.

Yes the two should be orthogonal at the moment. I am just making fast a
bit faster and go deeper ... because it now boots.

Since there have been no other review comments so far, i will prepare
that v2 where all the "remove from fast" will have a generic reason and
will be squashed.

Henning
 
> 1.
> https://patchwork.isar-build.org/project/isar/list/?series=670&state=*
> 
> With kind regards,
> Baurzhan

Re: [PATCH 0/8] CI rework of gitlab fast job

[Baurzhan Ismagulov]

On Thu, Jan 12, 2023 at 04:12:50PM +0100, Henning Schild wrote:
> Yes the two should be orthogonal at the moment. I am just making fast a
> bit faster and go deeper ... because it now boots.

As "fast" should remain the maintainer testsuite, I'd like to discuss the
coverage implications with Anton as well.


> Since there have been no other review comments so far, i will prepare
> that v2 where all the "remove from fast" will have a generic reason and
> will be squashed.

Yes, would be good.


With kind regards,
Baurzhan.

Re: [PATCH 0/8] CI rework of gitlab fast job

[Henning Schild]

Am Thu, 12 Jan 2023 16:20:56 +0100
schrieb Baurzhan Ismagulov <ibr@radix50.net>:

> On Thu, Jan 12, 2023 at 04:12:50PM +0100, Henning Schild wrote:
> > Yes the two should be orthogonal at the moment. I am just making
> > fast a bit faster and go deeper ... because it now boots.  
> 
> As "fast" should remain the maintainer testsuite, I'd like to discuss
> the coverage implications with Anton as well.

At the end of the day i want any variant of the testsuite to run on any
setup just like it does on ilbers jenkins.

That should be based on the kas containers for anyone not using your
jenkins.

With that any contributor would be able to run it manually or in their
CI.

We currently can not run "full" in gitlab CI because it takes too much
time and our gitlab runners would take such jobs for at most 12h. But
that is a value i could adjust if we find that it really needs to take
that long. I hope we can get much faster and still have the same
coverage.

Henning

> > Since there have been no other review comments so far, i will
> > prepare that v2 where all the "remove from fast" will have a
> > generic reason and will be squashed.  
> 
> Yes, would be good.
> 
> 
> With kind regards,
> Baurzhan.

Re: [PATCH 0/8] CI rework of gitlab fast job

[Moessbauer, Felix]

On Fri, 2023-01-13 at 00:43 +0100, Henning Schild wrote:
> Am Thu, 12 Jan 2023 16:20:56 +0100
> schrieb Baurzhan Ismagulov <ibr@radix50.net>:
> 
> > On Thu, Jan 12, 2023 at 04:12:50PM +0100, Henning Schild wrote:
> > > Yes the two should be orthogonal at the moment. I am just making
> > > fast a bit faster and go deeper ... because it now boots.  
> > 
> > As "fast" should remain the maintainer testsuite, I'd like to
> > discuss
> > the coverage implications with Anton as well.
> 
> At the end of the day i want any variant of the testsuite to run on
> any
> setup just like it does on ilbers jenkins.

I fully agree. Currently it is really hard and time consuming to run
the testsuite, even just for basic tests.
> 
> That should be based on the kas containers for anyone not using your
> jenkins.

This would also help a lot, to run it locally.

> 
> With that any contributor would be able to run it manually or in
> their
> CI.

I guess this would also reduce the maintenance effort at Ilbers as less
patches would be sent to the ML which do not pass the CI.

Felix

> 
> We currently can not run "full" in gitlab CI because it takes too
> much
> time and our gitlab runners would take such jobs for at most 12h. But
> that is a value i could adjust if we find that it really needs to
> take
> that long. I hope we can get much faster and still have the same
> coverage.
> 
> Henning
> 
> > > Since there have been no other review comments so far, i will
> > > prepare that v2 where all the "remove from fast" will have a
> > > generic reason and will be squashed.  
> > 
> > Yes, would be good.
> > 
> > 
> > With kind regards,
> > Baurzhan.
> 

Re: [PATCH 0/8] CI rework of gitlab fast job

[Baurzhan Ismagulov]

On Fri, Jan 13, 2023 at 12:43:15AM +0100, Henning Schild wrote:
> At the end of the day i want any variant of the testsuite to run on any
> setup just like it does on ilbers jenkins.
> 
> That should be based on the kas containers for anyone not using your
> jenkins.
> 
> With that any contributor would be able to run it manually or in their
> CI.
> 
> We currently can not run "full" in gitlab CI because it takes too much
> time and our gitlab runners would take such jobs for at most 12h. But
> that is a value i could adjust if we find that it really needs to take
> that long. I hope we can get much faster and still have the same
> coverage.

Thanks for the summary, that's fully in line with our last off-list discussion:

* The preferred downstream testsuite is "dev" (runs for ~ 2 h IIRC).

* The testsuite runs manually on the host or in kas.

* The testsuite runs automatically in GitLab, Jenkins or Azure using kas.

* Separate testcases for every arch / distro combo. One can run the complete
  testsuite or individual testcases.

* The maintainer CI provides test status and build control on a publicly
  accessible, easily scalable infrastructure.

* The CI setup is easily clonable for the downstreams.

* Downstreams can run Isar testsuites and / or their own testsuites.


So, I propose that we update and apply the "dev" testsuite series, then apply
yours with the following changes:

1. Use "dev" in "[PATCH 3/8] CI: install qemu-system when qemu testing is
   requested".

2. Patches 4-8 "testsuite: Remove X test from "fast" set" will be unnecessary
   (we'd take this into account during the "dev" series update -- those
   testcases are not part of the "dev" v1 anyway).

For the final review of the use cases I'll wait for Anton.


With kind regards,
Baurzhan

Re: [PATCH 0/8] CI rework of gitlab fast job

[Henning Schild]

Am Fri, 13 Jan 2023 14:14:58 +0100
schrieb Baurzhan Ismagulov <ibr@radix50.net>:

> On Fri, Jan 13, 2023 at 12:43:15AM +0100, Henning Schild wrote:
> > At the end of the day i want any variant of the testsuite to run on
> > any setup just like it does on ilbers jenkins.
> > 
> > That should be based on the kas containers for anyone not using your
> > jenkins.
> > 
> > With that any contributor would be able to run it manually or in
> > their CI.
> > 
> > We currently can not run "full" in gitlab CI because it takes too
> > much time and our gitlab runners would take such jobs for at most
> > 12h. But that is a value i could adjust if we find that it really
> > needs to take that long. I hope we can get much faster and still
> > have the same coverage.  
> 
> Thanks for the summary, that's fully in line with our last off-list
> discussion:
> 
> * The preferred downstream testsuite is "dev" (runs for ~ 2 h IIRC).
> 
> * The testsuite runs manually on the host or in kas.
> 
> * The testsuite runs automatically in GitLab, Jenkins or Azure using
> kas.
> 
> * Separate testcases for every arch / distro combo. One can run the
> complete testsuite or individual testcases.
> 
> * The maintainer CI provides test status and build control on a
> publicly accessible, easily scalable infrastructure.
> 
> * The CI setup is easily clonable for the downstreams.
> 
> * Downstreams can run Isar testsuites and / or their own testsuites.
> 
> 
> So, I propose that we update and apply the "dev" testsuite series,
> then apply yours with the following changes:
> 
> 1. Use "dev" in "[PATCH 3/8] CI: install qemu-system when qemu
> testing is requested".
> 
> 2. Patches 4-8 "testsuite: Remove X test from "fast" set" will be
> unnecessary (we'd take this into account during the "dev" series
> update -- those testcases are not part of the "dev" v1 anyway).
> 
> For the final review of the use cases I'll wait for Anton.

Let us see how we merge all that. I never looked at that dev series and
think we should not add a third pipeline. Too many switches and
variations just mean that we will need testing for the tests and nobody
will know which to run when.

I did not try but i bet if "fast" would simply use Sstate caches it
would run in less than 30min for small changes. And we would naturally
test Sstate and actually become much better on "partial rebuild".

So in my book "dev" is "fast" or even "full" with warm caches.

Henning

> 
> With kind regards,
> Baurzhan

Re: [PATCH 0/8] CI rework of gitlab fast job

[Baurzhan Ismagulov]

On Fri, Jan 13, 2023 at 06:14:01PM +0100, Henning Schild wrote:
> think we should not add a third pipeline. Too many switches and
> variations just mean that we will need testing for the tests and nobody
> will know which to run when.

This is a very valid point. Our problem is, we need two (or maybe more :( )
testsuites for efficient handling of maintenance tasks, and they are very
unsuitable for an average contributor. So this has appeared out of necessity.

I fully agree regarding the switches, I also hate them for the same reason.
We've considered moving to the raw avocado command line, but it is neither
short nor obvious, so we left ci_build.sh wrapper at this stage.

So, my proposal is to make "dev" the default and everyone should normally run
ci_build.sh without any switches (unless anyone explicitly wants to experience
some diversity in life). Of course, the maintainers should then regularly run
it.


> I did not try but i bet if "fast" would simply use Sstate caches it
> would run in less than 30min for small changes. And we would naturally
> test Sstate and actually become much better on "partial rebuild".
> 
> So in my book "dev" is "fast" or even "full" with warm caches.

Caching is also a good point. We can run "dev" with sstate by default; we just
need a way to specify local site configuration out of git (e.g., in an
environment variable).

I wouldn't like to have "dev" = "full" + caching because we're evaluating
adding rebuild testcases with and without specific options (like sstate, etc.)
and running both in "full".

"Fast" is rather a misnomer; its goal is to cover some 95+% of functionality in
somewhat reasonable time. The mentioned testcases have been very consciously
included there (e.g., sdk was broken several times); we can discuss what to
include but in general we'd like to keep and increase the current coverage
(execution time permitting) without affecting the users who don't need it.

"Dev" would further diverge from "fast" due to e.g.:

* Omitted "pedantic / infrastructure" bits as you mentioned

* Chosen rebuild testcases as this has often been broken in the past

* To be useful, "dev" should be kept green as far as possible, with all the
  implications.

  E.g., testcases can fail due to Debian issues, so "dev" shouldn't include
  Debian-ports (riscv). But the maintainers need to "quickly" assess the state
  of all supported arches.

* Testcase granularity (one testcase covering one arch + distro) to have a
  clear indication in the test results which specific combos have failed and to
  be able to re-run that failed combo only.

  "Fast" needs to keep the multiconfig building to catch races.


With kind regards,
Baurzhan

Re: [PATCH 0/8] CI rework of gitlab fast job

[Henning Schild]

Am Sat, 14 Jan 2023 15:40:47 +0100
schrieb Baurzhan Ismagulov <ibr@radix50.net>:

> On Fri, Jan 13, 2023 at 06:14:01PM +0100, Henning Schild wrote:
> > think we should not add a third pipeline. Too many switches and
> > variations just mean that we will need testing for the tests and
> > nobody will know which to run when.  
> 
> This is a very valid point. Our problem is, we need two (or maybe
> more :( ) testsuites for efficient handling of maintenance tasks, and
> they are very unsuitable for an average contributor. So this has
> appeared out of necessity.
> 
> I fully agree regarding the switches, I also hate them for the same
> reason. We've considered moving to the raw avocado command line, but
> it is neither short nor obvious, so we left ci_build.sh wrapper at
> this stage.

I would suggest a wrapper for every pipeline, rather than a switch to
one of the wrappers. We can leave switches for advanced users but maybe
only $@ to be passed to avocado.

So a simple ls would tell one which tests are available, where the
files hopefully have good names and the README or CONTRIBUTING files
say which tests to best run after changing anything.

Henning

> 
> So, my proposal is to make "dev" the default and everyone should
> normally run ci_build.sh without any switches (unless anyone
> explicitly wants to experience some diversity in life). Of course,
> the maintainers should then regularly run it.
> 
> 
> > I did not try but i bet if "fast" would simply use Sstate caches it
> > would run in less than 30min for small changes. And we would
> > naturally test Sstate and actually become much better on "partial
> > rebuild".
> > 
> > So in my book "dev" is "fast" or even "full" with warm caches.  
> 
> Caching is also a good point. We can run "dev" with sstate by
> default; we just need a way to specify local site configuration out
> of git (e.g., in an environment variable).
> 
> I wouldn't like to have "dev" = "full" + caching because we're
> evaluating adding rebuild testcases with and without specific options
> (like sstate, etc.) and running both in "full".
> 
> "Fast" is rather a misnomer; its goal is to cover some 95+% of
> functionality in somewhat reasonable time. The mentioned testcases
> have been very consciously included there (e.g., sdk was broken
> several times); we can discuss what to include but in general we'd
> like to keep and increase the current coverage (execution time
> permitting) without affecting the users who don't need it.
> 
> "Dev" would further diverge from "fast" due to e.g.:
> 
> * Omitted "pedantic / infrastructure" bits as you mentioned
> 
> * Chosen rebuild testcases as this has often been broken in the past
> 
> * To be useful, "dev" should be kept green as far as possible, with
> all the implications.
> 
>   E.g., testcases can fail due to Debian issues, so "dev" shouldn't
> include Debian-ports (riscv). But the maintainers need to "quickly"
> assess the state of all supported arches.
> 
> * Testcase granularity (one testcase covering one arch + distro) to
> have a clear indication in the test results which specific combos
> have failed and to be able to re-run that failed combo only.
> 
>   "Fast" needs to keep the multiconfig building to catch races.
> 
> 
> With kind regards,
> Baurzhan
> 

Re: [PATCH 0/8] CI rework of gitlab fast job

[Baurzhan Ismagulov]

On 2023-01-13 14:14, Baurzhan Ismagulov wrote:
> * The maintainer CI provides test status and build control on a publicly
>   accessible, easily scalable infrastructure.

FWIW, we've started with https://gitlab.isar-build.org/ in test mode. Please
report issues. The current low-performance runner will be replaced with a
better machine. We'll continue working on testing improvements.

With kind regards,
Baurzhan