[PATCH] testsuite: Cleanup after gpg import in repro_test

[PATCH] testsuite: Cleanup after gpg import in repro_test

[Uladzimir Bely]

After each repro_test a random GNUPGHOME directory and `gpg-agent`
process are left.

After multiple full tests in CI we may come to the situation when
all inotify descriptors are busy:

```
File "<path>/bitbake/lib/pyinotify.py", line 1728, in _init_
    raise OSError(err % self._inotify_wrapper.str_errno())
OSError: Cannot initialize new instance of inotify,
    Errno=Too many open files (EMFILE)
```

This patch provides an appropriate cleanup routine.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 testsuite/cibase.py | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/testsuite/cibase.py b/testsuite/cibase.py
index f2e4e957..7ac8ebc7 100755
--- a/testsuite/cibase.py
+++ b/testsuite/cibase.py
@@ -3,6 +3,7 @@
 import glob
 import os
 import re
+import shutil
 import tempfile
 import time
 
@@ -35,7 +36,7 @@ class CIBaseTest(CIBuilder):
 
         os.chdir(self.build_dir)
 
-        os.environ['GNUPGHOME'] = tempfile.mkdtemp()
+        os.environ['GNUPGHOME'] = gnupg_home = tempfile.mkdtemp()
         result = process.run('gpg --import %s %s' % (gpg_pub_key, gpg_priv_key))
 
         if result.exit_status:
@@ -56,6 +57,10 @@ class CIBaseTest(CIBuilder):
             self.configure(**kwargs)
             self.bitbake(targets, **kwargs)
 
+        # Cleanup
+        process.run('gpgconf --kill gpg-agent')
+        shutil.rmtree(gnupg_home, True)
+
     def perform_ccache_test(self, targets, **kwargs):
         def ccache_stats(dir, field):
             # Look ccache source's 'src/core/Statistic.hpp' for field meanings
-- 
2.20.1

Re: [PATCH] testsuite: Cleanup after gpg import in repro_test

[Henning Schild]

Am Fri,  3 Mar 2023 08:19:07 +0100
schrieb Uladzimir Bely <ubely@ilbers.de>:

> After each repro_test a random GNUPGHOME directory and `gpg-agent`
> process are left.
> 
> After multiple full tests in CI we may come to the situation when
> all inotify descriptors are busy:
> 
> ```
> File "<path>/bitbake/lib/pyinotify.py", line 1728, in _init_
>     raise OSError(err % self._inotify_wrapper.str_errno())
> OSError: Cannot initialize new instance of inotify,
>     Errno=Too many open files (EMFILE)
> ```
> 
> This patch provides an appropriate cleanup routine.
> 
> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> ---
>  testsuite/cibase.py | 7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/testsuite/cibase.py b/testsuite/cibase.py
> index f2e4e957..7ac8ebc7 100755
> --- a/testsuite/cibase.py
> +++ b/testsuite/cibase.py
> @@ -3,6 +3,7 @@
>  import glob
>  import os
>  import re
> +import shutil
>  import tempfile
>  import time
>  
> @@ -35,7 +36,7 @@ class CIBaseTest(CIBuilder):
>  
>          os.chdir(self.build_dir)
>  
> -        os.environ['GNUPGHOME'] = tempfile.mkdtemp()
> +        os.environ['GNUPGHOME'] = gnupg_home = tempfile.mkdtemp()
>          result = process.run('gpg --import %s %s' % (gpg_pub_key,
> gpg_priv_key)) 
>          if result.exit_status:
> @@ -56,6 +57,10 @@ class CIBaseTest(CIBuilder):
>              self.configure(**kwargs)
>              self.bitbake(targets, **kwargs)
>  
> +        # Cleanup
> +        process.run('gpgconf --kill gpg-agent')
> +        shutil.rmtree(gnupg_home, True)

Does that really always run? We have a self.fail('GPG import failed'),
if that returns from this function we might have agents around, and
that directory.

the tearDown() fixture might be better

Henning

> +
>      def perform_ccache_test(self, targets, **kwargs):
>          def ccache_stats(dir, field):
>              # Look ccache source's 'src/core/Statistic.hpp' for
> field meanings

Re: [PATCH] testsuite: Cleanup after gpg import in repro_test

[Uladzimir Bely]

In mail from Friday, 3 March 2023 12:10:56 +03 user Henning Schild wrote:
> Am Fri,  3 Mar 2023 08:19:07 +0100
> 
> schrieb Uladzimir Bely <ubely@ilbers.de>:
> > After each repro_test a random GNUPGHOME directory and `gpg-agent`
> > process are left.
> > 
> > After multiple full tests in CI we may come to the situation when
> > all inotify descriptors are busy:
> > 
> > ```
> > File "<path>/bitbake/lib/pyinotify.py", line 1728, in _init_
> > 
> >     raise OSError(err % self._inotify_wrapper.str_errno())
> > 
> > OSError: Cannot initialize new instance of inotify,
> > 
> >     Errno=Too many open files (EMFILE)
> > 
> > ```
> > 
> > This patch provides an appropriate cleanup routine.
> > 
> > Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> > ---
> > 
> >  testsuite/cibase.py | 7 ++++++-
> >  1 file changed, 6 insertions(+), 1 deletion(-)
> > 
> > diff --git a/testsuite/cibase.py b/testsuite/cibase.py
> > index f2e4e957..7ac8ebc7 100755
> > --- a/testsuite/cibase.py
> > +++ b/testsuite/cibase.py
> > @@ -3,6 +3,7 @@
> > 
> >  import glob
> >  import os
> >  import re
> > 
> > +import shutil
> > 
> >  import tempfile
> >  import time
> > 
> > @@ -35,7 +36,7 @@ class CIBaseTest(CIBuilder):
> >          os.chdir(self.build_dir)
> > 
> > -        os.environ['GNUPGHOME'] = tempfile.mkdtemp()
> > +        os.environ['GNUPGHOME'] = gnupg_home = tempfile.mkdtemp()
> > 
> >          result = process.run('gpg --import %s %s' % (gpg_pub_key,
> > 
> > gpg_priv_key))
> > 
> >          if result.exit_status:
> > @@ -56,6 +57,10 @@ class CIBaseTest(CIBuilder):
> >              self.configure(**kwargs)
> >              self.bitbake(targets, **kwargs)
> > 
> > +        # Cleanup
> > +        process.run('gpgconf --kill gpg-agent')
> > +        shutil.rmtree(gnupg_home, True)
> 
> Does that really always run? We have a self.fail('GPG import failed'),
> if that returns from this function we might have agents around, and
> that directory.
> 

I'm not sure gpg-agent process will be start in this case, but yes, at least 
temporary directory will remain. On the other hand, having this undeleted 
might be helpful for debugging...

> 
> the tearDown() fixture might be better
> Henning

Do you mean just a separate function that executed from 2 places? Or is it 
some kind a method that run automatically?

P.S. Walked through avocado documentation while writing this mail and found 
that it's a method that runs automatically. Will try with this approach, 
thanks for the hint.

> 
> > +
> > 
> >      def perform_ccache_test(self, targets, **kwargs):
> >          def ccache_stats(dir, field):
> >              # Look ccache source's 'src/core/Statistic.hpp' for
> > 
> > field meanings

Re: [PATCH] testsuite: Cleanup after gpg import in repro_test

[Uladzimir Bely]

In the email from Friday, 3 March 2023 16:57:27 +03 user Uladzimir Bely wrote:
> In mail from Friday, 3 March 2023 12:10:56 +03 user Henning Schild wrote:
> > Am Fri,  3 Mar 2023 08:19:07 +0100
> > 
> > schrieb Uladzimir Bely <ubely@ilbers.de>:
> > > After each repro_test a random GNUPGHOME directory and `gpg-agent`
> > > process are left.
> > > 
> > > After multiple full tests in CI we may come to the situation when
> > > all inotify descriptors are busy:
> > > 
> > > ```
> > > File "<path>/bitbake/lib/pyinotify.py", line 1728, in _init_
> > > 
> > >     raise OSError(err % self._inotify_wrapper.str_errno())
> > > 
> > > OSError: Cannot initialize new instance of inotify,
> > > 
> > >     Errno=Too many open files (EMFILE)
> > > 
> > > ```
> > > 
> > > This patch provides an appropriate cleanup routine.
> > > 
> > > Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> > > ---
> > > 
> > >  testsuite/cibase.py | 7 ++++++-
> > >  1 file changed, 6 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/testsuite/cibase.py b/testsuite/cibase.py
> > > index f2e4e957..7ac8ebc7 100755
> > > --- a/testsuite/cibase.py
> > > +++ b/testsuite/cibase.py
> > > @@ -3,6 +3,7 @@
> > > 
> > >  import glob
> > >  import os
> > >  import re
> > > 
> > > +import shutil
> > > 
> > >  import tempfile
> > >  import time
> > > 
> > > @@ -35,7 +36,7 @@ class CIBaseTest(CIBuilder):
> > >          os.chdir(self.build_dir)
> > > 
> > > -        os.environ['GNUPGHOME'] = tempfile.mkdtemp()
> > > +        os.environ['GNUPGHOME'] = gnupg_home = tempfile.mkdtemp()
> > > 
> > >          result = process.run('gpg --import %s %s' % (gpg_pub_key,
> > > 
> > > gpg_priv_key))
> > > 
> > >          if result.exit_status:
> > > @@ -56,6 +57,10 @@ class CIBaseTest(CIBuilder):
> > >              self.configure(**kwargs)
> > >              self.bitbake(targets, **kwargs)
> > > 
> > > +        # Cleanup
> > > +        process.run('gpgconf --kill gpg-agent')
> > > +        shutil.rmtree(gnupg_home, True)
> > 
> > Does that really always run? We have a self.fail('GPG import failed'),
> > if that returns from this function we might have agents around, and
> > that directory.
> > 
> 
> I'm not sure gpg-agent process will be start in this case, but yes, at least 
> temporary directory will remain. On the other hand, having this undeleted 
> might be helpful for debugging...
> 
> > 
> > the tearDown() fixture might be better
> > Henning
> 
> Do you mean just a separate function that executed from 2 places? Or is it 
> some kind a method that run automatically?
> 
> P.S. Walked through avocado documentation while writing this mail and found 
> that it's a method that runs automatically. Will try with this approach, 
> thanks for the hint.
> 

After examining this and discussing internally, we decided to leave it as is. To tearDown here the test should be completely rewritten (at least, moving tmdirs creations to the top level of test (e.g, "setUp" routing), since repro test does "gpg import" twice and requires two different gpg-agent processes to kill. Will leave it as is for now.

> > 
> > > +
> > > 
> > >      def perform_ccache_test(self, targets, **kwargs):
> > >          def ccache_stats(dir, field):
> > >              # Look ccache source's 'src/core/Statistic.hpp' for
> > > 
> > > field meanings
> 
> 
> 
> 
> 

Re: [PATCH] testsuite: Cleanup after gpg import in repro_test

[Uladzimir Bely]

In the email from Friday, 3 March 2023 10:19:07 +03 user Uladzimir Bely wrote:
> After each repro_test a random GNUPGHOME directory and `gpg-agent`
> process are left.
> 
> After multiple full tests in CI we may come to the situation when
> all inotify descriptors are busy:
> 
> ```
> File "<path>/bitbake/lib/pyinotify.py", line 1728, in _init_
>     raise OSError(err % self._inotify_wrapper.str_errno())
> OSError: Cannot initialize new instance of inotify,
>     Errno=Too many open files (EMFILE)
> ```
> 
> This patch provides an appropriate cleanup routine.
> 
> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> ---
>  testsuite/cibase.py | 7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/testsuite/cibase.py b/testsuite/cibase.py
> index f2e4e957..7ac8ebc7 100755
> --- a/testsuite/cibase.py
> +++ b/testsuite/cibase.py
> @@ -3,6 +3,7 @@
>  import glob
>  import os
>  import re
> +import shutil
>  import tempfile
>  import time
>  
> @@ -35,7 +36,7 @@ class CIBaseTest(CIBuilder):
>  
>          os.chdir(self.build_dir)
>  
> -        os.environ['GNUPGHOME'] = tempfile.mkdtemp()
> +        os.environ['GNUPGHOME'] = gnupg_home = tempfile.mkdtemp()
>          result = process.run('gpg --import %s %s' % (gpg_pub_key, gpg_priv_key))
>  
>          if result.exit_status:
> @@ -56,6 +57,10 @@ class CIBaseTest(CIBuilder):
>              self.configure(**kwargs)
>              self.bitbake(targets, **kwargs)
>  
> +        # Cleanup
> +        process.run('gpgconf --kill gpg-agent')
> +        shutil.rmtree(gnupg_home, True)
> +
>      def perform_ccache_test(self, targets, **kwargs):
>          def ccache_stats(dir, field):
>              # Look ccache source's 'src/core/Statistic.hpp' for field meanings
> 

Applied to next.