[DISCUSSIONS] CI build

[DISCUSSIONS] CI build

[Claudius Heine]

Hi,

we discussed some issues with the CI offline.

Here are my issues of the current CI with Jenkins in text:
  - Unreliable build:
    -> Network and other unrelated issues to the patch
    -> Contains expected to fail tests
  - Not transparent configuration/environment:
    -> parameters with which ci_build.sh is called are not controllable 
from the repository
    -> difficult to select which test case should be run
    -> Configuration of the environment is not controllable from the 
repository: Installed packages, System configuration (locales), 
Distribution, etc.
  - Just one giant build process
    -> Takes a long time
    -> Cannot be spread over multiple servers
    -> Giant log file, that makes it difficult to figure out exactly 
which command or test case caused an issue to reproduce that locally
  - Manually triggering of build jobs
    -> After pushing of CI branch, build has to be triggered via button
    -> Patches on ML should be automatically tested and reported back to 
ML (that only makes sense if the build is reliable though)
  - No "test everything" (acceptance test) build job
  - Logs aren't public accessible

Anyone has other issues or comments?

regards,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [DISCUSSIONS] CI build

[Maxim Yu. Osipov]

On 5/22/19 9:26 AM, Claudius Heine wrote:
> Hi,
> 
> we discussed some issues with the CI offline.
> 
> Here are my issues of the current CI with Jenkins in text:
>   - Unreliable build:
>     -> Network and other unrelated issues to the patch
>     -> Contains expected to fail tests
>   - Not transparent configuration/environment:
>     -> parameters with which ci_build.sh is called are not controllable 
> from the repository
>     -> difficult to select which test case should be run

Yes. that's why we are going to migrate to avocado test suite.

>     -> Configuration of the environment is not controllable from the 
> repository: Installed packages, System configuration (locales), 
> Distribution, etc.

In my opinion we need to have some "reference" common (better as minimal 
as possible) environment, to avoid a zoo of different environments...

>   - Just one giant build process
>     -> Takes a long time

I fear that it can't be avoided for such kind of build systems - several 
months ago we upgraded/tweaked the server and whole CI build itself 
takes now around 1 hour (which is acceptable in my opinion).
We are improving now smoke test launching.

>     -> Cannot be spread over multiple servers
>     -> Giant log file, that makes it difficult to figure out exactly 
> which command or test case caused an issue to reproduce that locally

Your project is configured with -q (quiet) option and doesn't generate 
huge trace - it becomes verbose only on encountered errors -
I had no problems with such log file as you can easily find the 
encountered error and always analyze the build workspace which is not 
cleaned up for the last build.

Just have a look at your latest build
http://isar-build.org:8080/job/isar_claudius_ilbers-ci/64/console
it took for me 10 seconds to locate the failed recipe.

>   - Manually triggering of build jobs
>     -> After pushing of CI branch, build has to be triggered via button

One may enable such build trigger (I believe that such trigger is 
enabled for Henning's project). but from my user's experience I prefer 
to start the build manually.

Of course, overnights builds are started periodically.

See for details:
https://wiki.jenkins.io/display/JENKINS/Building+a+software+project

>     -> Patches on ML should be automatically tested and reported back to 
> ML (that only makes sense if the build is reliable though)

Do we really want such feature?
My concern is that ML will be polluted by such reports...

>   - No "test everything" (acceptance test) build job

I'll introduce such acceptance test.

>   - Logs aren't public accessible

I think that Baurzhan may comment this.


Regards,
Maxim.

> 
> Anyone has other issues or comments?
> 
> regards,
> Claudius
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: [DISCUSSIONS] CI build

[Claudius Heine]

Hi Maxim,

On 22/05/2019 10.20, Maxim Yu. Osipov wrote:
> On 5/22/19 9:26 AM, Claudius Heine wrote:
[...]
>>     -> Configuration of the environment is not controllable from the 
>> repository: Installed packages, System configuration (locales), 
>> Distribution, etc.
> 
> In my opinion we need to have some "reference" common (better as minimal 
> as possible) environment, to avoid a zoo of different environments...

Well the more different environments to test against the more you can 
find bugs. If we don't want that, then we should deliver a such and 
environment together with isar.

[...]

> 
>>     -> Cannot be spread over multiple servers
>>     -> Giant log file, that makes it difficult to figure out exactly 
>> which command or test case caused an issue to reproduce that locally
> 
> Your project is configured with -q (quiet) option and doesn't generate 
> huge trace - it becomes verbose only on encountered errors -
> I had no problems with such log file as you can easily find the 
> encountered error and always analyze the build workspace which is not 
> cleaned up for the last build.
> 
> Just have a look at your latest build
> http://isar-build.org:8080/job/isar_claudius_ilbers-ci/64/console
> it took for me 10 seconds to locate the failed recipe.

Well its not always about a failed recipe. The ci_build.sh script now 
contains multiple different calls to bitbake, but we don't see those 
calls in the log. There is also no mention about other calls, like sed 
which changes the configuration.

> 
>>   - Manually triggering of build jobs
>>     -> After pushing of CI branch, build has to be triggered via button
> 
> One may enable such build trigger (I believe that such trigger is 
> enabled for Henning's project). but from my user's experience I prefer 
> to start the build manually.

I normally just push to the ci branch when I want to start tests.

> 
> Of course, overnights builds are started periodically.
> 
> See for details:
> https://wiki.jenkins.io/display/JENKINS/Building+a+software+project
> 
>>     -> Patches on ML should be automatically tested and reported back 
>> to ML (that only makes sense if the build is reliable though)
> 
> Do we really want such feature?
> My concern is that ML will be polluted by such reports...

You are currently testing and reporting back manually, if instead of you 
some bot would do that automatically, then I don't how that would 
increase the noise on the ML. But as I said that only makes sense if the 
false positive and negative rates of CI is lessened.

regards,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [DISCUSSIONS] CI build

[Maxim Yu. Osipov]

On 5/22/19 11:10 AM, Claudius Heine wrote:
> Hi Maxim,
> 
> On 22/05/2019 10.20, Maxim Yu. Osipov wrote:
>> On 5/22/19 9:26 AM, Claudius Heine wrote:
> [...]
>>>     -> Configuration of the environment is not controllable from the 
>>> repository: Installed packages, System configuration (locales), 
>>> Distribution, etc.
>>
>> In my opinion we need to have some "reference" common (better as 
>> minimal as possible) environment, to avoid a zoo of different 
>> environments...
> 
> Well the more different environments to test against the more you can 
> find bugs. If we don't want that, then we should deliver a such and 
> environment together with isar.

Please provide your detailed proposals how this can be formalized as
there is an indefinite number of environments.
Again, I vote for some common "reference" environment.


> [...]
> 
>>
>>>     -> Cannot be spread over multiple servers
>>>     -> Giant log file, that makes it difficult to figure out exactly 
>>> which command or test case caused an issue to reproduce that locally
>>
>> Your project is configured with -q (quiet) option and doesn't generate 
>> huge trace - it becomes verbose only on encountered errors -
>> I had no problems with such log file as you can easily find the 
>> encountered error and always analyze the build workspace which is not 
>> cleaned up for the last build.
>>
>> Just have a look at your latest build
>> http://isar-build.org:8080/job/isar_claudius_ilbers-ci/64/console
>> it took for me 10 seconds to locate the failed recipe.
> 
> Well its not always about a failed recipe. The ci_build.sh script now 
> contains multiple different calls to bitbake, but we don't see those 
> calls in the log. There is also no mention about other calls, like sed 
> which changes the configuration.

OK, I'll add `set -x` in ci_build.sh.

> 
>>
>>>   - Manually triggering of build jobs
>>>     -> After pushing of CI branch, build has to be triggered via button
>>
>> One may enable such build trigger (I believe that such trigger is 
>> enabled for Henning's project). but from my user's experience I prefer 
>> to start the build manually.
> 
> I normally just push to the ci branch when I want to start tests.

I've enabled the corresponding options for your projects - please try.

> 
>>
>> Of course, overnights builds are started periodically.
>>
>> See for details:
>> https://wiki.jenkins.io/display/JENKINS/Building+a+software+project
>>
>>>     -> Patches on ML should be automatically tested and reported back 
>>> to ML (that only makes sense if the build is reliable though)
>>
>> Do we really want such feature?
>> My concern is that ML will be polluted by such reports...
> 
> You are currently testing and reporting back manually, if instead of you 
> some bot would do that automatically, then I don't how that would 
> increase the noise on the ML. But as I said that only makes sense if the 
> false positive and negative rates of CI is lessened.

Manual work is unavoidable as sometimes patches are not merged properly.
I'll enable reporting to ML when I push a patch set to maintainers 
branches for CI.

Regards,
Maxim.



> 
> regards,
> Claudius
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: [DISCUSSIONS] CI build

[Claudius Heine]

Hi Maxim,

On 22/05/2019 11.50, Maxim Yu. Osipov wrote:
> On 5/22/19 11:10 AM, Claudius Heine wrote:
>> Hi Maxim,
>>
>> On 22/05/2019 10.20, Maxim Yu. Osipov wrote:
>>> On 5/22/19 9:26 AM, Claudius Heine wrote:
>> [...]
>>>>     -> Configuration of the environment is not controllable from the 
>>>> repository: Installed packages, System configuration (locales), 
>>>> Distribution, etc.
>>>
>>> In my opinion we need to have some "reference" common (better as 
>>> minimal as possible) environment, to avoid a zoo of different 
>>> environments...
>>
>> Well the more different environments to test against the more you can 
>> find bugs. If we don't want that, then we should deliver a such and 
>> environment together with isar.
> 
> Please provide your detailed proposals how this can be formalized as
> there is an indefinite number of environments.

For me containers have proven to provide ways to easily provision and 
configure the content while having a reasonable hit on performance.

> Again, I vote for some common "reference" environment.

If you want to go that way, then maybe choose the kasproject/kas-isar 
docker container as the reference environment, since that is used by 
most contributors when developing AFAIK. And then we can point to that 
container in the documentation as well, so nobody will try to run isar 
in a different environment.

> 
> 
>> [...]
>>
>>>
>>>>     -> Cannot be spread over multiple servers
>>>>     -> Giant log file, that makes it difficult to figure out exactly 
>>>> which command or test case caused an issue to reproduce that locally
>>>
>>> Your project is configured with -q (quiet) option and doesn't 
>>> generate huge trace - it becomes verbose only on encountered errors -
>>> I had no problems with such log file as you can easily find the 
>>> encountered error and always analyze the build workspace which is not 
>>> cleaned up for the last build.
>>>
>>> Just have a look at your latest build
>>> http://isar-build.org:8080/job/isar_claudius_ilbers-ci/64/console
>>> it took for me 10 seconds to locate the failed recipe.
>>
>> Well its not always about a failed recipe. The ci_build.sh script now 
>> contains multiple different calls to bitbake, but we don't see those 
>> calls in the log. There is also no mention about other calls, like sed 
>> which changes the configuration.
> 
> OK, I'll add `set -x` in ci_build.sh.

Hopefully that will not be too noisy on the log.

> 
>>
>>>
>>>>   - Manually triggering of build jobs
>>>>     -> After pushing of CI branch, build has to be triggered via button
>>>
>>> One may enable such build trigger (I believe that such trigger is 
>>> enabled for Henning's project). but from my user's experience I 
>>> prefer to start the build manually.
>>
>> I normally just push to the ci branch when I want to start tests.
> 
> I've enabled the corresponding options for your projects - please try.

Thanks. But I think I should now have two branches for each different 
build parameter set. So that I don't trigger both builds just for one push.

> 
>>
>>>
>>> Of course, overnights builds are started periodically.
>>>
>>> See for details:
>>> https://wiki.jenkins.io/display/JENKINS/Building+a+software+project
>>>
>>>>     -> Patches on ML should be automatically tested and reported 
>>>> back to ML (that only makes sense if the build is reliable though)
>>>
>>> Do we really want such feature?
>>> My concern is that ML will be polluted by such reports...
>>
>> You are currently testing and reporting back manually, if instead of 
>> you some bot would do that automatically, then I don't how that would 
>> increase the noise on the ML. But as I said that only makes sense if 
>> the false positive and negative rates of CI is lessened.
> 
> Manual work is unavoidable as sometimes patches are not merged properly.
> I'll enable reporting to ML when I push a patch set to maintainers 
> branches for CI.

Can you make it so that it emails with the correct email 'in-reply-to' 
header, so it lands in the right thread of the patchset? Otherwise 
ordering that will be annoying.

Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [DISCUSSIONS] CI build

[Claudius Heine]

Hi,

On 22/05/2019 13.28, [ext] Claudius Heine wrote:
[...]
>>
>>>
>>>>
>>>>>   - Manually triggering of build jobs
>>>>>     -> After pushing of CI branch, build has to be triggered via 
>>>>> button
>>>>
>>>> One may enable such build trigger (I believe that such trigger is 
>>>> enabled for Henning's project). but from my user's experience I 
>>>> prefer to start the build manually.
>>>
>>> I normally just push to the ci branch when I want to start tests.
>>
>> I've enabled the corresponding options for your projects - please try.
> 
> Thanks. But I think I should now have two branches for each different 
> build parameter set. So that I don't trigger both builds just for one push.

I pushed a ch/ilbers-ci-fast branch for the fast build.

> 
>>
>>>
>>>>
>>>> Of course, overnights builds are started periodically.
>>>>
>>>> See for details:
>>>> https://wiki.jenkins.io/display/JENKINS/Building+a+software+project
>>>>
>>>>>     -> Patches on ML should be automatically tested and reported 
>>>>> back to ML (that only makes sense if the build is reliable though)
>>>>
>>>> Do we really want such feature?
>>>> My concern is that ML will be polluted by such reports...
>>>
>>> You are currently testing and reporting back manually, if instead of 
>>> you some bot would do that automatically, then I don't how that would 
>>> increase the noise on the ML. But as I said that only makes sense if 
>>> the false positive and negative rates of CI is lessened.
>>
>> Manual work is unavoidable as sometimes patches are not merged properly.
>> I'll enable reporting to ML when I push a patch set to maintainers 
>> branches for CI.
> 
> Can you make it so that it emails with the correct email 'in-reply-to' 
> header, so it lands in the right thread of the patchset? Otherwise 
> ordering that will be annoying.

Just to clarify: In all my points I would prefer a good solution instead 
of a quick one. If some points are quick to solve without any 
disadvantages, then ok. But if some quick solution will probably have 
other undesired side-effects, then take a step back and think about 
other ways to fix those issues without those, even if that might take a 
while to work them out. The current system is ok enough to bare with it 
until a proper solution is in place.

Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [DISCUSSIONS] CI build

[Claudius Heine]

Hi,

On 23/05/2019 10.10, [ext] Claudius Heine wrote:
> Hi,
> 
> On 22/05/2019 13.28, [ext] Claudius Heine wrote:
> [...]
>>>
>>>>
>>>>>
>>>>>>   - Manually triggering of build jobs
>>>>>>     -> After pushing of CI branch, build has to be triggered via 
>>>>>> button
>>>>>
>>>>> One may enable such build trigger (I believe that such trigger is 
>>>>> enabled for Henning's project). but from my user's experience I 
>>>>> prefer to start the build manually.
>>>>
>>>> I normally just push to the ci branch when I want to start tests.
>>>
>>> I've enabled the corresponding options for your projects - please try.
>>
>> Thanks. But I think I should now have two branches for each different 
>> build parameter set. So that I don't trigger both builds just for one 
>> push.
> 
> I pushed a ch/ilbers-ci-fast branch for the fast build.

I have to say the this polling option that jenkins does is suboptimal.
Is there a way to add a git hook that triggers the build exactly when 
the commit arrives on github?

Cheers,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [DISCUSSIONS] CI build

[Maxim Yu. Osipov]

On 5/23/19 3:43 PM, Claudius Heine wrote:
> Hi,
> 
> On 23/05/2019 10.10, [ext] Claudius Heine wrote:
>> Hi,
>>
>> On 22/05/2019 13.28, [ext] Claudius Heine wrote:
>> [...]
>>>>
>>>>>
>>>>>>
>>>>>>>   - Manually triggering of build jobs
>>>>>>>     -> After pushing of CI branch, build has to be triggered via 
>>>>>>> button
>>>>>>
>>>>>> One may enable such build trigger (I believe that such trigger is 
>>>>>> enabled for Henning's project). but from my user's experience I 
>>>>>> prefer to start the build manually.
>>>>>
>>>>> I normally just push to the ci branch when I want to start tests.
>>>>
>>>> I've enabled the corresponding options for your projects - please try.
>>>
>>> Thanks. But I think I should now have two branches for each different 
>>> build parameter set. So that I don't trigger both builds just for one 
>>> push.
>>
>> I pushed a ch/ilbers-ci-fast branch for the fast build.
> 
> I have to say the this polling option that jenkins does is suboptimal.
> Is there a way to add a git hook that triggers the build exactly when 
> the commit arrives on github?

There is an option GitHub hook trigger for GITScm polling:

If Jenkins will receive PUSH GitHub hook from repo defined in Git SCM 
section it will trigger Git SCM polling logic. So polling logic in fact 
belongs to Git SCM.

Should I enable it for you?

Regards,
Maxim.

> 
> Cheers,
> Claudius
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: [DISCUSSIONS] CI build

[Claudius Heine]

On 23/05/2019 15.56, Maxim Yu. Osipov wrote:
> On 5/23/19 3:43 PM, Claudius Heine wrote:
>> Hi,
>>
>> On 23/05/2019 10.10, [ext] Claudius Heine wrote:
>>> Hi,
>>>
>>> On 22/05/2019 13.28, [ext] Claudius Heine wrote:
>>> [...]
>>>>>
>>>>>>
>>>>>>>
>>>>>>>>   - Manually triggering of build jobs
>>>>>>>>     -> After pushing of CI branch, build has to be triggered via 
>>>>>>>> button
>>>>>>>
>>>>>>> One may enable such build trigger (I believe that such trigger is 
>>>>>>> enabled for Henning's project). but from my user's experience I 
>>>>>>> prefer to start the build manually.
>>>>>>
>>>>>> I normally just push to the ci branch when I want to start tests.
>>>>>
>>>>> I've enabled the corresponding options for your projects - please try.
>>>>
>>>> Thanks. But I think I should now have two branches for each 
>>>> different build parameter set. So that I don't trigger both builds 
>>>> just for one push.
>>>
>>> I pushed a ch/ilbers-ci-fast branch for the fast build.
>>
>> I have to say the this polling option that jenkins does is suboptimal.
>> Is there a way to add a git hook that triggers the build exactly when 
>> the commit arrives on github?
> 
> There is an option GitHub hook trigger for GITScm polling:
> 
> If Jenkins will receive PUSH GitHub hook from repo defined in Git SCM 
> section it will trigger Git SCM polling logic. So polling logic in fact 
> belongs to Git SCM.
> 
> Should I enable it for you?

Yes, if you can tell me how to use it.

 From what I read, there should be a curl command that I can add to the 
github repo hook settings. But I don't know the address I should use and 
if I needs some login information etc.

Thanks,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [DISCUSSIONS] CI build

[Baurzhan Ismagulov]

Hi Claudius,

thanks for summarizing. I think we'll open github issues at least for some of
the items.

On Wed, May 22, 2019 at 09:26:51AM +0200, Claudius Heine wrote:
> Here are my issues of the current CI with Jenkins in text:
>  - Unreliable build:
>    -> Network and other unrelated issues to the patch
>    -> Contains expected to fail tests

We'll work on these.


>  - Not transparent configuration/environment:
>    -> parameters with which ci_build.sh is called are not controllable from
> the repository

Hmm, the intention actually was that everyone runs the same fixed testsuites.
If desired, we could publish the arguments and gradually move them to the isar
repo.


>    -> difficult to select which test case should be run

This is the motivation for Avocado.


>    -> Configuration of the environment is not controllable from the
> repository: Installed packages, System configuration (locales),
> Distribution, etc.

Here the intention has also been to have one standard testing environment, and
it is Debian stable + instructions from Quick Start, nothing more. I'd welcome
moving to kas + docker. If desired, we could publish the current setup.

I'd like to have the testsuite be runnable from any CI system with the same
script, without arguments. Then people could deploy any system they like
(Jenkins, GitLab, Travis CI, AWS...). The same testsuite should also be
runnable on one's host (already the case with ci_build, vm_smoke_test). Under
the hood, every test case should be defined and runnable individually
(Avocado).


>  - Just one giant build process
>    -> Takes a long time

This will be solved with Avocado.


>    -> Cannot be spread over multiple servers

Haven't looked at that, ideas welcome. Jan told us he is preparing something,
but load balancing would spread multiple pipelines, not tasks within one
pipeline. Depending on the costs, it might or might not be cheaper for us to
throw more physical servers on the task.


>    -> Giant log file, that makes it difficult to figure out exactly which
> command or test case caused an issue to reproduce that locally

This will be solved with Avocado.


>  - Manually triggering of build jobs
>    -> After pushing of CI branch, build has to be triggered via button

I actually prefer working in this mode. Henning has polling enabled. If you
prefer two branches, please send the second branch URL to Maxim.


>    -> Patches on ML should be automatically tested and reported back to ML
> (that only makes sense if the build is reliable though)

I haven't looked at that. Is there a tool that does this?


>  - No "test everything" (acceptance test) build job

Currently, it's the combination of cross and native builds. However, there are
a couple of steps that are still performed manually for various reasons. I'd
like to see them automated if at all possible.


>  - Logs aren't public accessible

Logs are accessible with authentication. Jenkins has public mode, but it lists
the users publicly, that is why we haven't opened it till now. If desired, we
could look at that. Is it that high-priority, given that all major contributors
here have access?


With kind regards,
Baurzhan.

Re: [DISCUSSIONS] CI build

[Jan Kiszka]

On 22.05.19 21:44, Baurzhan Ismagulov wrote:
> Hi Claudius,
> 
> thanks for summarizing. I think we'll open github issues at least for some of
> the items.
> 
> On Wed, May 22, 2019 at 09:26:51AM +0200, Claudius Heine wrote:
>> Here are my issues of the current CI with Jenkins in text:
>>   - Unreliable build:
>>     -> Network and other unrelated issues to the patch
>>     -> Contains expected to fail tests
> 
> We'll work on these.
> 
> 
>>   - Not transparent configuration/environment:
>>     -> parameters with which ci_build.sh is called are not controllable from
>> the repository
> 
> Hmm, the intention actually was that everyone runs the same fixed testsuites.
> If desired, we could publish the arguments and gradually move them to the isar
> repo.
> 
> 
>>     -> difficult to select which test case should be run
> 
> This is the motivation for Avocado.
> 
> 
>>     -> Configuration of the environment is not controllable from the
>> repository: Installed packages, System configuration (locales),
>> Distribution, etc.
> 
> Here the intention has also been to have one standard testing environment, and
> it is Debian stable + instructions from Quick Start, nothing more. I'd welcome
> moving to kas + docker. If desired, we could publish the current setup.
> 
> I'd like to have the testsuite be runnable from any CI system with the same
> script, without arguments. Then people could deploy any system they like
> (Jenkins, GitLab, Travis CI, AWS...). The same testsuite should also be
> runnable on one's host (already the case with ci_build, vm_smoke_test). Under
> the hood, every test case should be defined and runnable individually
> (Avocado).
> 
> 
>>   - Just one giant build process
>>     -> Takes a long time
> 
> This will be solved with Avocado.
> 

Nope, this requires scale-out into parallel jobs on a build farm, rather than a 
single server. E.g. we would scale better by doing per-arch builds so that arch 
commonality can still be exploited but the rest can work lock-free with own I/O 
bandwidth etc. on separate servers (or nodes in the cloud).

> 
>>     -> Cannot be spread over multiple servers
> 
> Haven't looked at that, ideas welcome. Jan told us he is preparing something,
> but load balancing would spread multiple pipelines, not tasks within one
> pipeline. Depending on the costs, it might or might not be cheaper for us to
> throw more physical servers on the task.
> 

https://lists.cip-project.org/pipermail/cip-dev/2019-May/002371.html

> 
>>     -> Giant log file, that makes it difficult to figure out exactly which
>> command or test case caused an issue to reproduce that locally
> 
> This will be solved with Avocado.
> 
> 
>>   - Manually triggering of build jobs
>>     -> After pushing of CI branch, build has to be triggered via button
> 
> I actually prefer working in this mode. Henning has polling enabled. If you
> prefer two branches, please send the second branch URL to Maxim.
> 

I also prefer standard build-on-push. But when everyone can build his own CI or 
use locally shared ones with personal, repo-controlled settings, we no longer 
depend on having to sync the working modes.

> 
>>     -> Patches on ML should be automatically tested and reported back to ML
>> (that only makes sense if the build is reliable though)
> 
> I haven't looked at that. Is there a tool that does this?
> 
> 
>>   - No "test everything" (acceptance test) build job
> 
> Currently, it's the combination of cross and native builds. However, there are
> a couple of steps that are still performed manually for various reasons. I'd
> like to see them automated if at all possible.
> 
> 
>>   - Logs aren't public accessible
> 
> Logs are accessible with authentication. Jenkins has public mode, but it lists
> the users publicly, that is why we haven't opened it till now. If desired, we
> could look at that. Is it that high-priority, given that all major contributors
> here have access?
> 

Let's stop wasting thoughts on legacy Jenkins, rather discuss how we can move on 
to gitlab-ci, if something is missing there etc.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [DISCUSSIONS] CI build

[Claudius Heine]

Hi Jan, hi Baurzhan,

On 23/05/2019 06.44, [ext] Jan Kiszka wrote:
>>>   - Not transparent configuration/environment:
>>>     -> parameters with which ci_build.sh is called are not 
>>> controllable from
>>> the repository
>>
>> Hmm, the intention actually was that everyone runs the same fixed 
>> testsuites.
>> If desired, we could publish the arguments and gradually move them to 
>> the isar
>> repo.

Which is currently not the case, we have a 'normal' build, a 'fast' 
build, a 'cross' build and a 'repo' build, which can be combined with 
'normal', 'fast' and 'cross' IIUC.

>>
>>
>>>     -> difficult to select which test case should be run
>>
>> This is the motivation for Avocado.
>>
>>
>>>     -> Configuration of the environment is not controllable from the
>>> repository: Installed packages, System configuration (locales),
>>> Distribution, etc.
>>
>> Here the intention has also been to have one standard testing 
>> environment, and
>> it is Debian stable + instructions from Quick Start, nothing more. I'd 
>> welcome
>> moving to kas + docker. If desired, we could publish the current setup.
>>
>> I'd like to have the testsuite be runnable from any CI system with the 
>> same
>> script, without arguments. Then people could deploy any system they like
>> (Jenkins, GitLab, Travis CI, AWS...). The same testsuite should also be
>> runnable on one's host (already the case with ci_build, 
>> vm_smoke_test). Under
>> the hood, every test case should be defined and runnable individually
>> (Avocado).

Personally I don't really care about executing the whole CI process 
locally. I am much more interested in triggering single failed 
test-cases locally to work on them. So the 'ci_build.sh' is not really 
useful for me.

'ci_build.sh' is just to unhandy to use locally.

[...]
>>>     -> Cannot be spread over multiple servers
>>
>> Haven't looked at that, ideas welcome. Jan told us he is preparing 
>> something,
>> but load balancing would spread multiple pipelines, not tasks within one
>> pipeline. Depending on the costs, it might or might not be cheaper for 
>> us to
>> throw more physical servers on the task.
>>
> 
> https://lists.cip-project.org/pipermail/cip-dev/2019-May/002371.html

Would have been nice to have this cross-posted to this ML as well.

Will Michael write patches for isar to integrate this, or does he needs 
anything to do this?

>>
>>>     -> Patches on ML should be automatically tested and reported back 
>>> to ML
>>> (that only makes sense if the build is reliable though)
>>
>> I haven't looked at that. Is there a tool that does this?

Not really. But I posted a patchwork+jenkins integration howto some time 
ago: https://that.guru/blog/patchwork-and-ci-in-a-tree/

>>>   - Logs aren't public accessible
>>
>> Logs are accessible with authentication. Jenkins has public mode, but 
>> it lists
>> the users publicly, that is why we haven't opened it till now. If 
>> desired, we
>> could look at that. Is it that high-priority, given that all major 
>> contributors
>> here have access?
>>
> 
> Let's stop wasting thoughts on legacy Jenkins, rather discuss how we can 
> move on to gitlab-ci, if something is missing there etc.

I can only figure out what is missing after I tried it out.

regards,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [DISCUSSIONS] CI build

[Jan Kiszka]

On 23.05.19 10:32, Claudius Heine wrote:
>>>>     -> Cannot be spread over multiple servers
>>>
>>> Haven't looked at that, ideas welcome. Jan told us he is preparing something,
>>> but load balancing would spread multiple pipelines, not tasks within one
>>> pipeline. Depending on the costs, it might or might not be cheaper for us to
>>> throw more physical servers on the task.
>>>
>>
>> https://lists.cip-project.org/pipermail/cip-dev/2019-May/002371.html
> 
> Would have been nice to have this cross-posted to this ML as well.
> 

We wanted to do that once a build is public, see also the follow-up in that thread.

> Will Michael write patches for isar to integrate this, or does he needs anything 
> to do this?
> 

There is currently no need for patches, gitlab-ci.yml already exists in isar and 
is being used. But there will likely be the desire to tune that further, add the 
qemu-based tests once the infrastructure is running.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [DISCUSSIONS] CI build

[Maxim Yu. Osipov]

On 5/23/19 10:32 AM, Claudius Heine wrote:
> Hi Jan, hi Baurzhan,
> 
> On 23/05/2019 06.44, [ext] Jan Kiszka wrote:
>>>>   - Not transparent configuration/environment:
>>>>     -> parameters with which ci_build.sh is called are not 
>>>> controllable from
>>>> the repository
>>>
>>> Hmm, the intention actually was that everyone runs the same fixed 
>>> testsuites.
>>> If desired, we could publish the arguments and gradually move them to 
>>> the isar
>>> repo.
> 
> Which is currently not the case, we have a 'normal' build, a 'fast' 
> build, a 'cross' build and a 'repo' build, which can be combined with 
> 'normal', 'fast' and 'cross' IIUC.

Just to clarify.

If the patches I've sent yesterday are accepted the
'normal' build will include

* parallel cross compilation of the set
stretch qemuarm/qemuarm64/qemuamd64/de0-nano-soc/rpi and
buster qemuarm
* sdk creation for stretch qemuarm
* parallel native compilation of the set
stretch qemuarm/qemuarm64/qemui386/qemuamd64/rpi and
buster qemuarm/qemui386/qemuamd64/qemuamd64-buster-tgz/nand-ubi-demo

'fast' build excludes native compilation.

If the extra option 'repro' is enabled, additional separate build
to create/use cached base repo for stretch qemuarm/qemuarm64/qemuamd64 
is launched.

To maximize test coverage I've enabled repro for all your projects in 
Jenkins.

So if the patch set is accepted it would be enough to run 'normal' build 
with 'repro' option: 'scripts/ci_build.sh -q -r'

Regards,
Maxim.


>>>
>>>
>>>>     -> difficult to select which test case should be run
>>>
>>> This is the motivation for Avocado.
>>>
>>>
>>>>     -> Configuration of the environment is not controllable from the
>>>> repository: Installed packages, System configuration (locales),
>>>> Distribution, etc.
>>>
>>> Here the intention has also been to have one standard testing 
>>> environment, and
>>> it is Debian stable + instructions from Quick Start, nothing more. 
>>> I'd welcome
>>> moving to kas + docker. If desired, we could publish the current setup.
>>>
>>> I'd like to have the testsuite be runnable from any CI system with 
>>> the same
>>> script, without arguments. Then people could deploy any system they like
>>> (Jenkins, GitLab, Travis CI, AWS...). The same testsuite should also be
>>> runnable on one's host (already the case with ci_build, 
>>> vm_smoke_test). Under
>>> the hood, every test case should be defined and runnable individually
>>> (Avocado).
> 
> Personally I don't really care about executing the whole CI process 
> locally. I am much more interested in triggering single failed 
> test-cases locally to work on them. So the 'ci_build.sh' is not really 
> useful for me.
> 
> 'ci_build.sh' is just to unhandy to use locally.
> 
> [...]
>>>>     -> Cannot be spread over multiple servers
>>>
>>> Haven't looked at that, ideas welcome. Jan told us he is preparing 
>>> something,
>>> but load balancing would spread multiple pipelines, not tasks within one
>>> pipeline. Depending on the costs, it might or might not be cheaper 
>>> for us to
>>> throw more physical servers on the task.
>>>
>>
>> https://lists.cip-project.org/pipermail/cip-dev/2019-May/002371.html
> 
> Would have been nice to have this cross-posted to this ML as well.
> 
> Will Michael write patches for isar to integrate this, or does he needs 
> anything to do this?
> 
>>>
>>>>     -> Patches on ML should be automatically tested and reported 
>>>> back to ML
>>>> (that only makes sense if the build is reliable though)
>>>
>>> I haven't looked at that. Is there a tool that does this?
> 
> Not really. But I posted a patchwork+jenkins integration howto some time 
> ago: https://that.guru/blog/patchwork-and-ci-in-a-tree/
> 
>>>>   - Logs aren't public accessible
>>>
>>> Logs are accessible with authentication. Jenkins has public mode, but 
>>> it lists
>>> the users publicly, that is why we haven't opened it till now. If 
>>> desired, we
>>> could look at that. Is it that high-priority, given that all major 
>>> contributors
>>> here have access?
>>>
>>
>> Let's stop wasting thoughts on legacy Jenkins, rather discuss how we 
>> can move on to gitlab-ci, if something is missing there etc.
> 
> I can only figure out what is missing after I tried it out.
> 
> regards,
> Claudius
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: [DISCUSSIONS] CI build

[Henning Schild]

Am Wed, 22 May 2019 09:26:51 +0200
schrieb "[ext] Claudius Heine" <claudius.heine.ext@siemens.com>:

> Hi,
> 
> we discussed some issues with the CI offline.
> 
> Here are my issues of the current CI with Jenkins in text:
>   - Unreliable build:
>     -> Network and other unrelated issues to the patch
>     -> Contains expected to fail tests 

Agreed, i stopped running the tests locally because i could never come
up with the same errors. Actually the CI itself could never come up
with the same ones .... but it kept failing without giving a real clue
why.
 
>   - Not transparent configuration/environment:
>     -> parameters with which ci_build.sh is called are not
> controllable from the repository
>     -> difficult to select which test case should be run
>     -> Configuration of the environment is not controllable from
> the repository: Installed packages, System configuration (locales), 
> Distribution, etc.
>   - Just one giant build process
>     -> Takes a long time
>     -> Cannot be spread over multiple servers
>     -> Giant log file, that makes it difficult to figure out
> exactly which command or test case caused an issue to reproduce that
> locally

When i use the CI system from Ilbers i usually carry around a few "fast
CI" patches on top of my real changes. In there i configure CI to do
what i want and only build one config to start with. Not elegant but
works somehow.

Henning

>   - Manually triggering of build jobs
>     -> After pushing of CI branch, build has to be triggered via
> button -> Patches on ML should be automatically tested and reported
> back to ML (that only makes sense if the build is reliable though)
>   - No "test everything" (acceptance test) build job
>   - Logs aren't public accessible
> 
> Anyone has other issues or comments?
> 
> regards,
> Claudius
> 

Re: [DISCUSSIONS] CI build

[Baurzhan Ismagulov]

On Thu, May 23, 2019 at 01:35:37PM +0200, Henning Schild wrote:
> >   - Unreliable build:
> >     -> Network and other unrelated issues to the patch
> >     -> Contains expected to fail tests 
> 
> Agreed, i stopped running the tests locally because i could never come
> up with the same errors. Actually the CI itself could never come up
> with the same ones .... but it kept failing without giving a real clue
> why.
...
> >     -> Giant log file, that makes it difficult to figure out
> > exactly which command or test case caused an issue to reproduce that
> > locally
> 
> When i use the CI system from Ilbers i usually carry around a few "fast
> CI" patches on top of my real changes. In there i configure CI to do
> what i want and only build one config to start with. Not elegant but
> works somehow.

Feel free to report problems and / or send patches when your encounter the
issues, otherwise we never know you have one.

With kind regards,
Baurzhan.

Jenkins project configuration by user

[Maxim Yu. Osipov]

Hi isar-build.org users,

On 5/22/19 9:26 AM, Claudius Heine wrote:
<snip>
>   - Not transparent configuration/environment:
>     -> parameters with which ci_build.sh is called are not controllable 
> from the repository

To address this issue I've added the ability to user to configure his 
project - just click button 'Configure' on the left pane of your project
http://isar-build.org:8080/job/<your_project_name>/configure

Here you can change parameters passed to ci_build, tweak build triggers etc.

If you want to initiate the build on push request to your repository

1) Add a webhook in your github isar repository:


Payload URL:
http://ci.isar-build.org:8080/github-webhook/

Content type:
application/json

Secret:
<empty>

Radio-button:
Just the push event

Tick as active.

2) Enable option 'GitHub hook trigger for GITScm polling" in the section 
Build Triggers of project's configuration.

Regards,
Maxim.

-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov