Re: [PATCH 2/2] scripts/ci_build.sh: Set avocado-framework version to 96.0

[Henning Schild <henning.schild@siemens.com>]

Am Wed, 6 Jul 2022 18:27:41 +0200
schrieb Baurzhan Ismagulov <ibr@radix50.net>:

> On Wed, Jul 06, 2022 at 04:01:48PM +0200, Henning Schild wrote:
> > > diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
> > > index 2513f1a0..b1ec6912 100755
> > > --- a/scripts/ci_build.sh
> > > +++ b/scripts/ci_build.sh
> > > @@ -21,7 +21,7 @@ if ! command -v avocado > /dev/null; then
> > >      rm -rf /tmp/avocado_venv
> > >      virtualenv --python python3 /tmp/avocado_venv
> > >      source /tmp/avocado_venv/bin/activate
> > > -    pip install avocado-framework
> > > +    pip install avocado-framework==96.0  
> > 
> > Once the code is reworked so it does not use the legacy feature any
> > longer this pinning should probably not be dropped, it can be moved
> > every once in a while, but getting a random version is simply not
> > good. As we see again here.  
> 
> Not including avocado in kas-container is kas's decision. This
> workaround was requested on this list and we applied it. If you ask
> me, I don't want to have it and still suggest to use debs.

pip is kind of ok when pinning the version, removing the pin was the
problem

> 
> > Especially not when ilbers does not run this bit in their CI, having
> > avocado already installed in some version. Maybe we should always
> > use that venv/pip version no matter if the host is contaminated
> > already with some other random version.  
> 
> This is one of the reasons to avoid pip, which we have repeatedly
> suggested.

That deb of yours does not count as a deb. It is as unmaintained as the
debianization in avocado itself, probably not tested in bookworm ...
and what have you.
The pip way is at least maintained and recent. If one never updates
avocado one will happily write tests using deprecated features ...
forcing oneself to stick to the old version forever.

Not to mention that some people might want to run stuff on non-debian
distros. i.e. the one i tried to get in with the arm64 efi booting, the
build itself done in a container (debian) the boot tests initially
done/developed on the host (gentoo).

> 
> > How is the mainline issue for debian going anyhow?  
> 
> We've contacted Debian, got initial feedback and fixed most of that.
> The remaining issues will be clarified with the upstream. We'll send
> the updated version and ask for the review.
> 
> 
> > Time to grab some tequila to wash down the guacamole and sing some
> > "i told you so" songs. pip pip hurray!  
> 
> I call everyone on this list to report the issues and discuss specific
> solutions. I can also start "told you to use debs" songs which won't
> solve anyone's problems.

Both lack debian upstream quality. Again it is fine as long as it has
the == ... maybe together with a --no-deps if possible.

> 
> > When it comes to testing the maintainer should really make it as
> > simple as possible for people to run the tests, ideally with the
> > very same outcome as the pipelines of the maintainer.  
> 
> I think providing a deb in an apt repo is as simple as possible for
> people to run the tests. Way simpler and cleaner than the pip stuff.
> And using the same deb would result in the same outcome.

I do not think so. What one could do is fetch upstream avocado, build
their package and install it. Do that every single time, also in the
jenkins setups. 

> > Also it should be easy for
> > users to contribute to the test suite, so it will be reasonable to
> > ask people to bring tests with their features.
> > IMHO the whole avocado story made both aspects worse than before.  
> 
> If you have specific difficulties contributing to the testsuite, feel
> free to share them here. From the frameworks that you had suggested,
> any of those would require getting familiar with writing testcases,
> so currently I don't see major differences in this specific aspect,
> but there are important advantages in other aspects. I admit that
> since we spend huge amounts of machine-hours for running CI on every
> patch set, we do have special requirements for the test framework.
> And I think the result is good for downstreams as well.

>From what i saw the tests are pretty convoluted and python is not
properly used. But that has nothing to do with avocado, just the way
the tests have been written.

> > Only
> > for some features that the maintainer wanted, but what they get is
> > ontested patches because they are the only ones able to run that
> > beast.  
> 
> This is not true, you are running avocado in your CI. You can also
> run specific testcases in kas-container, just like with any other
> framework. If desired, we could contribute docs or make a community
> workshop on using the testsuite.

We are trying to ... sometimes ... if it works. But for sure not
always. I am still often finding myself in the place where i send
untested stuff ... well local manual tested and then send.
And one reason people skip tests is still that it is pretty hard and
time consuming to run them. And they are very random and likely to fail
anyhow.

So i just send ... maybe i push to my poller branch of your jenkins ...
but this will always go red anyhow ... like next does all the time. The
testsuite failing in more than 20% in the cases anyways ... means
people stop caring about it. It is "rainy thunderstorm" in jenkins
anyhow or "red" in gitlab ... it was before your patch and it remains
with your patch.
At least the reports ease that a bit, if people still care to look what
is behind "red" or "thunderstorm"

Henning

> 
> With kind regards,
> Baurzhan