[PATCH v2] testsuite: Improve SSH ping

[PATCH v2] testsuite: Improve SSH ping

[Uladzimir Bely]

When qemu machine boots, it may happen that consecutive SSH connection
fails right after the previous good one. So, we get a situation when
the command/script fails after we consider SSH is ready.

This patch improves detection of SSH server ready status by making at
least three good consecutive SSH pings.

Example of debug output that shows the case:

```
| Waiting for SSH server ready...
| SSH ping result: 255, left: 300s # <== machine is booting
| SSH ping result: 255, left: 294s
| SSH ping result: 255, left: 288s
| SSH ping result: 255, left: 282s
| SSH ping result: 255, left: 276s
| SSH ping result: 255, left: 270s
| SSH ping result: 255, left: 264s
| SSH ping result: 255, left: 258s
| SSH ping result: 0, left: 253s   # <== SSH server is up...
| SSH ping result: 0, left: 251s
| SSH ping result: 255, left: 250s # <== but one ping failed again
| SSH ping result: 0, left: 248s
| SSH ping result: 0, left: 245s
| SSH ping result: 0, left: 243s   # <== 3 good pings considered
| `lsmod | grep example_module` returned 0
```

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 Changes since v1:
- patch simplified a bit for better readability
- less unnecessary debug logs are produced

 testsuite/cibuilder.py | 14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 9e84c3a3..cd8ff6d4 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -257,7 +257,10 @@ class CIBuilder(Test):
         self.log.debug('Waiting for SSH server ready...')
 
         rc = None
-        while time.time() < timeout:
+        goodcnt = 0
+        # Use 3 good SSH ping attempts to consider SSH connection is stable
+        while time.time() < timeout and goodcnt < 3:
+            goodcnt += 1
             if proc.poll() is not None:
                 self.log.error('Machine is not running')
                 return rc
@@ -265,12 +268,11 @@ class CIBuilder(Test):
             rc = self.exec_cmd('/bin/true', cmd_prefix)
             time.sleep(1)
 
-            if rc == 0:
-                self.log.debug('SSH server is ready')
-                break
+            if rc != 0:
+                goodcnt = 0
 
-        if rc != 0:
-            self.log.error('SSH server is not ready')
+            time_left = timeout - time.time()
+            self.log.debug('SSH ping result: %d, left: %.fs' % (rc, time_left))
 
         return rc
 
-- 
2.20.1

Re: [PATCH v2] testsuite: Improve SSH ping

[Uladzimir Bely]

In mail from Wednesday, 22 March 2023 07:46:14 +03 user Uladzimir Bely wrote:
> When qemu machine boots, it may happen that consecutive SSH connection
> fails right after the previous good one. So, we get a situation when
> the command/script fails after we consider SSH is ready.
> 
> This patch improves detection of SSH server ready status by making at
> least three good consecutive SSH pings.
> 
> Example of debug output that shows the case:
> 

Applied to next.