Re: [DISCUSSIONS] CI build

[Baurzhan Ismagulov <ibr@radix50.net>]

Hi Claudius,

thanks for summarizing. I think we'll open github issues at least for some of
the items.

On Wed, May 22, 2019 at 09:26:51AM +0200, Claudius Heine wrote:
> Here are my issues of the current CI with Jenkins in text:
>  - Unreliable build:
>    -> Network and other unrelated issues to the patch
>    -> Contains expected to fail tests

We'll work on these.


>  - Not transparent configuration/environment:
>    -> parameters with which ci_build.sh is called are not controllable from
> the repository

Hmm, the intention actually was that everyone runs the same fixed testsuites.
If desired, we could publish the arguments and gradually move them to the isar
repo.


>    -> difficult to select which test case should be run

This is the motivation for Avocado.


>    -> Configuration of the environment is not controllable from the
> repository: Installed packages, System configuration (locales),
> Distribution, etc.

Here the intention has also been to have one standard testing environment, and
it is Debian stable + instructions from Quick Start, nothing more. I'd welcome
moving to kas + docker. If desired, we could publish the current setup.

I'd like to have the testsuite be runnable from any CI system with the same
script, without arguments. Then people could deploy any system they like
(Jenkins, GitLab, Travis CI, AWS...). The same testsuite should also be
runnable on one's host (already the case with ci_build, vm_smoke_test). Under
the hood, every test case should be defined and runnable individually
(Avocado).


>  - Just one giant build process
>    -> Takes a long time

This will be solved with Avocado.


>    -> Cannot be spread over multiple servers

Haven't looked at that, ideas welcome. Jan told us he is preparing something,
but load balancing would spread multiple pipelines, not tasks within one
pipeline. Depending on the costs, it might or might not be cheaper for us to
throw more physical servers on the task.


>    -> Giant log file, that makes it difficult to figure out exactly which
> command or test case caused an issue to reproduce that locally

This will be solved with Avocado.


>  - Manually triggering of build jobs
>    -> After pushing of CI branch, build has to be triggered via button

I actually prefer working in this mode. Henning has polling enabled. If you
prefer two branches, please send the second branch URL to Maxim.


>    -> Patches on ML should be automatically tested and reported back to ML
> (that only makes sense if the build is reliable though)

I haven't looked at that. Is there a tool that does this?


>  - No "test everything" (acceptance test) build job

Currently, it's the combination of cross and native builds. However, there are
a couple of steps that are still performed manually for various reasons. I'd
like to see them automated if at all possible.


>  - Logs aren't public accessible

Logs are accessible with authentication. Jenkins has public mode, but it lists
the users publicly, that is why we haven't opened it till now. If desired, we
could look at that. Is it that high-priority, given that all major contributors
here have access?


With kind regards,
Baurzhan.