* [PATCH] CI: Add sstate-cache testcase
@ 2021-10-28 15:20 Anton Mikanovich
2021-10-28 15:34 ` Jan Kiszka
0 siblings, 1 reply; 6+ messages in thread
From: Anton Mikanovich @ 2021-10-28 15:20 UTC (permalink / raw)
To: isar-users; +Cc: Anton Mikanovich
Test rebuild time improve after cleanup to be sure sstate-cache works.
Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
testsuite/build_test/build_test.py | 40 ++++++++++++++++++++++++++++--
1 file changed, 38 insertions(+), 2 deletions(-)
diff --git a/testsuite/build_test/build_test.py b/testsuite/build_test/build_test.py
index d39c10c0..244f6fc0 100644
--- a/testsuite/build_test/build_test.py
+++ b/testsuite/build_test/build_test.py
@@ -1,9 +1,9 @@
#!/usr/bin/env python3
-import os
+import os, time
from avocado import skipUnless
-from avocado.utils import path
+from avocado.utils import path, process
from cibase import CIBaseTest
UMOCI_AVAILABLE = True
@@ -206,3 +206,39 @@ class ContainerSdkTest(CIBaseTest):
targets = ['mc:container-amd64-stretch:isar-image-base']
self.perform_container_test(targets, 'do_populate_sdk')
+
+class SstateTest(CIBaseTest):
+
+ """
+ Test rebuild speed improve with sstate-cache
+
+ :avocado: tags=sstate
+ """
+ def test_sstate_rebuild(self):
+ speedup_k = 2.0
+
+ targets = [
+ 'mc:qemuamd64-bullseye:isar-image-base'
+ ]
+
+ # Cleanup everything before build
+ build_dir = self.params.get('build_dir',
+ default=os.path.dirname(__file__) + '/../../build')
+ process.run('rm -rf ' + build_dir + '/sstate-cache', sudo=True)
+ self.deletetmp(build_dir)
+
+ start = time.time()
+ self.perform_build_test(targets, 1, None)
+ first_time = time.time() - start
+ self.log.info('Non-cached build:' + str(round(first_time, 2)))
+
+ # Cleanup everything but cache files
+ self.deletetmp(build_dir)
+
+ start = time.time()
+ self.perform_build_test(targets, 1, None)
+ second_time = time.time() - start
+ self.log.info('Cached build:' + str(round(second_time, 2)))
+
+ if first_time / second_time < speedup_k:
+ self.fail('No speedup after rebuild with sstate-cache')
--
2.20.1
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] CI: Add sstate-cache testcase
2021-10-28 15:20 [PATCH] CI: Add sstate-cache testcase Anton Mikanovich
@ 2021-10-28 15:34 ` Jan Kiszka
2021-10-28 17:22 ` Anton Mikanovich
0 siblings, 1 reply; 6+ messages in thread
From: Jan Kiszka @ 2021-10-28 15:34 UTC (permalink / raw)
To: Anton Mikanovich, isar-users, Schmidt, Adriaan
On 28.10.21 17:20, Anton Mikanovich wrote:
> Test rebuild time improve after cleanup to be sure sstate-cache works.
>
> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
> ---
> testsuite/build_test/build_test.py | 40 ++++++++++++++++++++++++++++--
> 1 file changed, 38 insertions(+), 2 deletions(-)
>
> diff --git a/testsuite/build_test/build_test.py b/testsuite/build_test/build_test.py
> index d39c10c0..244f6fc0 100644
> --- a/testsuite/build_test/build_test.py
> +++ b/testsuite/build_test/build_test.py
> @@ -1,9 +1,9 @@
> #!/usr/bin/env python3
>
> -import os
> +import os, time
>
> from avocado import skipUnless
> -from avocado.utils import path
> +from avocado.utils import path, process
> from cibase import CIBaseTest
>
> UMOCI_AVAILABLE = True
> @@ -206,3 +206,39 @@ class ContainerSdkTest(CIBaseTest):
> targets = ['mc:container-amd64-stretch:isar-image-base']
>
> self.perform_container_test(targets, 'do_populate_sdk')
> +
> +class SstateTest(CIBaseTest):
> +
> + """
> + Test rebuild speed improve with sstate-cache
> +
> + :avocado: tags=sstate
> + """
> + def test_sstate_rebuild(self):
> + speedup_k = 2.0
> +
> + targets = [
> + 'mc:qemuamd64-bullseye:isar-image-base'
> + ]
> +
> + # Cleanup everything before build
> + build_dir = self.params.get('build_dir',
> + default=os.path.dirname(__file__) + '/../../build')
> + process.run('rm -rf ' + build_dir + '/sstate-cache', sudo=True)
> + self.deletetmp(build_dir)
> +
> + start = time.time()
> + self.perform_build_test(targets, 1, None)
> + first_time = time.time() - start
> + self.log.info('Non-cached build:' + str(round(first_time, 2)))
> +
> + # Cleanup everything but cache files
> + self.deletetmp(build_dir)
> +
> + start = time.time()
> + self.perform_build_test(targets, 1, None)
> + second_time = time.time() - start
> + self.log.info('Cached build:' + str(round(second_time, 2)))
> +
> + if first_time / second_time < speedup_k:
> + self.fail('No speedup after rebuild with sstate-cache')
>
Is there no better way than measuring time to test if sstate was in
place? Maybe some hit/miss statistics?
Jan
--
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] CI: Add sstate-cache testcase
2021-10-28 15:34 ` Jan Kiszka
@ 2021-10-28 17:22 ` Anton Mikanovich
2021-10-28 18:21 ` Henning Schild
0 siblings, 1 reply; 6+ messages in thread
From: Anton Mikanovich @ 2021-10-28 17:22 UTC (permalink / raw)
To: Jan Kiszka, isar-users, Schmidt, Adriaan
28.10.2021 18:34, Jan Kiszka wrote:
> On 28.10.21 17:20, Anton Mikanovich wrote:
>> Test rebuild time improve after cleanup to be sure sstate-cache works.
>>
>> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
>> ---
>> testsuite/build_test/build_test.py | 40 ++++++++++++++++++++++++++++--
>> 1 file changed, 38 insertions(+), 2 deletions(-)
>>
>> diff --git a/testsuite/build_test/build_test.py b/testsuite/build_test/build_test.py
>> index d39c10c0..244f6fc0 100644
>> --- a/testsuite/build_test/build_test.py
>> +++ b/testsuite/build_test/build_test.py
>> @@ -1,9 +1,9 @@
>> #!/usr/bin/env python3
>>
>> -import os
>> +import os, time
>>
>> from avocado import skipUnless
>> -from avocado.utils import path
>> +from avocado.utils import path, process
>> from cibase import CIBaseTest
>>
>> UMOCI_AVAILABLE = True
>> @@ -206,3 +206,39 @@ class ContainerSdkTest(CIBaseTest):
>> targets = ['mc:container-amd64-stretch:isar-image-base']
>>
>> self.perform_container_test(targets, 'do_populate_sdk')
>> +
>> +class SstateTest(CIBaseTest):
>> +
>> + """
>> + Test rebuild speed improve with sstate-cache
>> +
>> + :avocado: tags=sstate
>> + """
>> + def test_sstate_rebuild(self):
>> + speedup_k = 2.0
>> +
>> + targets = [
>> + 'mc:qemuamd64-bullseye:isar-image-base'
>> + ]
>> +
>> + # Cleanup everything before build
>> + build_dir = self.params.get('build_dir',
>> + default=os.path.dirname(__file__) + '/../../build')
>> + process.run('rm -rf ' + build_dir + '/sstate-cache', sudo=True)
>> + self.deletetmp(build_dir)
>> +
>> + start = time.time()
>> + self.perform_build_test(targets, 1, None)
>> + first_time = time.time() - start
>> + self.log.info('Non-cached build:' + str(round(first_time, 2)))
>> +
>> + # Cleanup everything but cache files
>> + self.deletetmp(build_dir)
>> +
>> + start = time.time()
>> + self.perform_build_test(targets, 1, None)
>> + second_time = time.time() - start
>> + self.log.info('Cached build:' + str(round(second_time, 2)))
>> +
>> + if first_time / second_time < speedup_k:
>> + self.fail('No speedup after rebuild with sstate-cache')
>>
> Is there no better way than measuring time to test if sstate was in
> place? Maybe some hit/miss statistics?
>
> Jan
>
I've tried to find some Avocado API to access final testcase statistics
from within other testcase but didn't find so far.
Any advice is welcome.
--
Anton Mikanovich
Promwad Ltd.
External service provider of ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn, Germany
+49 (89) 122 67 24-0
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] CI: Add sstate-cache testcase
2021-10-28 17:22 ` Anton Mikanovich
@ 2021-10-28 18:21 ` Henning Schild
2021-11-02 5:57 ` Schmidt, Adriaan
0 siblings, 1 reply; 6+ messages in thread
From: Henning Schild @ 2021-10-28 18:21 UTC (permalink / raw)
To: Anton Mikanovich; +Cc: Jan Kiszka, isar-users, Schmidt, Adriaan
Am Thu, 28 Oct 2021 20:22:58 +0300
schrieb Anton Mikanovich <amikan@ilbers.de>:
> 28.10.2021 18:34, Jan Kiszka wrote:
> > On 28.10.21 17:20, Anton Mikanovich wrote:
> >> Test rebuild time improve after cleanup to be sure sstate-cache
> >> works.
> >>
> >> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
> >> ---
> >> testsuite/build_test/build_test.py | 40
> >> ++++++++++++++++++++++++++++-- 1 file changed, 38 insertions(+), 2
> >> deletions(-)
> >>
> >> diff --git a/testsuite/build_test/build_test.py
> >> b/testsuite/build_test/build_test.py index d39c10c0..244f6fc0
> >> 100644 --- a/testsuite/build_test/build_test.py
> >> +++ b/testsuite/build_test/build_test.py
> >> @@ -1,9 +1,9 @@
> >> #!/usr/bin/env python3
> >>
> >> -import os
> >> +import os, time
> >>
> >> from avocado import skipUnless
> >> -from avocado.utils import path
> >> +from avocado.utils import path, process
> >> from cibase import CIBaseTest
> >>
> >> UMOCI_AVAILABLE = True
> >> @@ -206,3 +206,39 @@ class ContainerSdkTest(CIBaseTest):
> >> targets = ['mc:container-amd64-stretch:isar-image-base']
> >>
> >> self.perform_container_test(targets, 'do_populate_sdk')
> >> +
> >> +class SstateTest(CIBaseTest):
> >> +
> >> + """
> >> + Test rebuild speed improve with sstate-cache
> >> +
> >> + :avocado: tags=sstate
> >> + """
> >> + def test_sstate_rebuild(self):
> >> + speedup_k = 2.0
> >> +
> >> + targets = [
> >> + 'mc:qemuamd64-bullseye:isar-image-base'
> >> + ]
> >> +
> >> + # Cleanup everything before build
> >> + build_dir = self.params.get('build_dir',
> >> + default=os.path.dirname(__file__) +
> >> '/../../build')
> >> + process.run('rm -rf ' + build_dir + '/sstate-cache',
> >> sudo=True)
> >> + self.deletetmp(build_dir)
> >> +
> >> + start = time.time()
> >> + self.perform_build_test(targets, 1, None)
> >> + first_time = time.time() - start
> >> + self.log.info('Non-cached build:' + str(round(first_time,
> >> 2))) +
> >> + # Cleanup everything but cache files
> >> + self.deletetmp(build_dir)
> >> +
> >> + start = time.time()
> >> + self.perform_build_test(targets, 1, None)
> >> + second_time = time.time() - start
> >> + self.log.info('Cached build:' + str(round(second_time,
> >> 2))) +
> >> + if first_time / second_time < speedup_k:
> >> + self.fail('No speedup after rebuild with
> >> sstate-cache')
> > Is there no better way than measuring time to test if sstate was in
> > place? Maybe some hit/miss statistics?
> >
> > Jan
> >
> I've tried to find some Avocado API to access final testcase
> statistics from within other testcase but didn't find so far.
> Any advice is welcome.
sstate hit rates are in the bitbake logs, i hope avocado can open
artifacts
But this is infra ... we will see when it fails and do not have to test
that.
The interesting problems will happen once you build version+5 on a
cache from version. The only known limitiation there is that your base
rootfss must not be too old. Because your local apt-database will start
containing package versions that might not be available to install
anymore.
So you have to find an eviction strategy. But that is very project
specific.
We currently have two strategies in place on two projects happily
working with this since weeks.
1. nightly clean + full build
2. "tmpreaper"-style age-greater 3 days
Third one you build against a frozen mirror and never clear, or very
infrequent.
Plus this problem is well known to isar guys, you just have to
sometimes do a full build or "bitbake -C task recipe". It just now
moves to runners ... but only if those are specially configured to
offer persistent storage.
Not sure how jenkins deals with this per default, and what sort of
configuration the ilbers jenkins might need.
Henning
^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: [PATCH] CI: Add sstate-cache testcase
2021-10-28 18:21 ` Henning Schild
@ 2021-11-02 5:57 ` Schmidt, Adriaan
2021-11-02 7:55 ` Jan Kiszka
0 siblings, 1 reply; 6+ messages in thread
From: Schmidt, Adriaan @ 2021-11-02 5:57 UTC (permalink / raw)
To: henning.schild, Anton Mikanovich; +Cc: jan.kiszka, isar-users
Henning Schild, Donnerstag, 28. Oktober 2021 20:22:
> Am Thu, 28 Oct 2021 20:22:58 +0300
> schrieb Anton Mikanovich <amikan@ilbers.de>:
>
> > 28.10.2021 18:34, Jan Kiszka wrote:
> > > On 28.10.21 17:20, Anton Mikanovich wrote:
> > >> Test rebuild time improve after cleanup to be sure sstate-cache
> > >> works.
> > >>
> > >> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
> > >> ---
> > >> testsuite/build_test/build_test.py | 40
> > >> ++++++++++++++++++++++++++++-- 1 file changed, 38 insertions(+), 2
> > >> deletions(-)
> > >>
> > >> diff --git a/testsuite/build_test/build_test.py
> > >> b/testsuite/build_test/build_test.py index d39c10c0..244f6fc0
> > >> 100644 --- a/testsuite/build_test/build_test.py
> > >> +++ b/testsuite/build_test/build_test.py
> > >> @@ -1,9 +1,9 @@
> > >> #!/usr/bin/env python3
> > >>
> > >> -import os
> > >> +import os, time
> > >>
> > >> from avocado import skipUnless
> > >> -from avocado.utils import path
> > >> +from avocado.utils import path, process
> > >> from cibase import CIBaseTest
> > >>
> > >> UMOCI_AVAILABLE = True
> > >> @@ -206,3 +206,39 @@ class ContainerSdkTest(CIBaseTest):
> > >> targets = ['mc:container-amd64-stretch:isar-image-base']
> > >>
> > >> self.perform_container_test(targets, 'do_populate_sdk')
> > >> +
> > >> +class SstateTest(CIBaseTest):
> > >> +
> > >> + """
> > >> + Test rebuild speed improve with sstate-cache
> > >> +
> > >> + :avocado: tags=sstate
> > >> + """
> > >> + def test_sstate_rebuild(self):
> > >> + speedup_k = 2.0
> > >> +
> > >> + targets = [
> > >> + 'mc:qemuamd64-bullseye:isar-image-base'
> > >> + ]
> > >> +
> > >> + # Cleanup everything before build
> > >> + build_dir = self.params.get('build_dir',
> > >> + default=os.path.dirname(__file__) +
> > >> '/../../build')
> > >> + process.run('rm -rf ' + build_dir + '/sstate-cache',
> > >> sudo=True)
> > >> + self.deletetmp(build_dir)
> > >> +
> > >> + start = time.time()
> > >> + self.perform_build_test(targets, 1, None)
> > >> + first_time = time.time() - start
> > >> + self.log.info('Non-cached build:' + str(round(first_time,
> > >> 2))) +
> > >> + # Cleanup everything but cache files
> > >> + self.deletetmp(build_dir)
> > >> +
> > >> + start = time.time()
> > >> + self.perform_build_test(targets, 1, None)
> > >> + second_time = time.time() - start
> > >> + self.log.info('Cached build:' + str(round(second_time,
> > >> 2))) +
> > >> + if first_time / second_time < speedup_k:
> > >> + self.fail('No speedup after rebuild with
> > >> sstate-cache')
> > > Is there no better way than measuring time to test if sstate was in
> > > place? Maybe some hit/miss statistics?
> > >
> > > Jan
> > >
> > I've tried to find some Avocado API to access final testcase
> > statistics from within other testcase but didn't find so far.
> > Any advice is welcome.
>
> sstate hit rates are in the bitbake logs, i hope avocado can open
> artifacts
You want to look for the line "Sstate summary", which is shown at the beginning of the bitbake run, e.g.:
Sstate summary: Wanted 121 Local 121 Network 0 Missed 0 Current 0 (100% match, 0% complete)
But in a test it's probably better to examine log.task_order and check which tasks were executed.
> But this is infra ... we will see when it fails and do not have to test
> that.
It's worth running some tests to see that builds succeed with artifacts taken from cache. I can imagine something like:
- Build image to populate cache
- Remove TMPDIR
- Rebuild the image. Should have 100% cache hits. (This would take the target rootfs from cache, and the buildchroot required for image-building)
- Remove TMPDIR
- Build a single package (that was already contained in the image). Should have a cache hit, and only dpkg_build_setscene and deploy_deb should run (plus minimal steps to initialize isar-apt).
And optionally:
- Remove TMPDIR
- Force rebuild of one package (e.g. by deleting its sstate artifact)
- Rebuild the image (should take the buildchroot and all but the one package from cache, and rebuild the one package and the image)
> The interesting problems will happen once you build version+5 on a
> cache from version. The only known limitiation there is that your base
> rootfss must not be too old. Because your local apt-database will start
> containing package versions that might not be available to install
> anymore.
At least IMO this is limitation is unrelated to the sstate cache, and I agree that testing such cases doesn't make much sense.
I'm happy to take over and propose a test case.
Adriaan
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] CI: Add sstate-cache testcase
2021-11-02 5:57 ` Schmidt, Adriaan
@ 2021-11-02 7:55 ` Jan Kiszka
0 siblings, 0 replies; 6+ messages in thread
From: Jan Kiszka @ 2021-11-02 7:55 UTC (permalink / raw)
To: Schmidt, Adriaan (T RDA IOT SES-DE),
Schild, Henning (T RDA IOT SES-DE),
Anton Mikanovich
Cc: isar-users
On 02.11.21 06:57, Schmidt, Adriaan (T RDA IOT SES-DE) wrote:
> Henning Schild, Donnerstag, 28. Oktober 2021 20:22:
>> Am Thu, 28 Oct 2021 20:22:58 +0300
>> schrieb Anton Mikanovich <amikan@ilbers.de>:
>>
>>> 28.10.2021 18:34, Jan Kiszka wrote:
>>>> On 28.10.21 17:20, Anton Mikanovich wrote:
>>>>> Test rebuild time improve after cleanup to be sure sstate-cache
>>>>> works.
>>>>>
>>>>> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
>>>>> ---
>>>>> testsuite/build_test/build_test.py | 40
>>>>> ++++++++++++++++++++++++++++-- 1 file changed, 38 insertions(+), 2
>>>>> deletions(-)
>>>>>
>>>>> diff --git a/testsuite/build_test/build_test.py
>>>>> b/testsuite/build_test/build_test.py index d39c10c0..244f6fc0
>>>>> 100644 --- a/testsuite/build_test/build_test.py
>>>>> +++ b/testsuite/build_test/build_test.py
>>>>> @@ -1,9 +1,9 @@
>>>>> #!/usr/bin/env python3
>>>>>
>>>>> -import os
>>>>> +import os, time
>>>>>
>>>>> from avocado import skipUnless
>>>>> -from avocado.utils import path
>>>>> +from avocado.utils import path, process
>>>>> from cibase import CIBaseTest
>>>>>
>>>>> UMOCI_AVAILABLE = True
>>>>> @@ -206,3 +206,39 @@ class ContainerSdkTest(CIBaseTest):
>>>>> targets = ['mc:container-amd64-stretch:isar-image-base']
>>>>>
>>>>> self.perform_container_test(targets, 'do_populate_sdk')
>>>>> +
>>>>> +class SstateTest(CIBaseTest):
>>>>> +
>>>>> + """
>>>>> + Test rebuild speed improve with sstate-cache
>>>>> +
>>>>> + :avocado: tags=sstate
>>>>> + """
>>>>> + def test_sstate_rebuild(self):
>>>>> + speedup_k = 2.0
>>>>> +
>>>>> + targets = [
>>>>> + 'mc:qemuamd64-bullseye:isar-image-base'
>>>>> + ]
>>>>> +
>>>>> + # Cleanup everything before build
>>>>> + build_dir = self.params.get('build_dir',
>>>>> + default=os.path.dirname(__file__) +
>>>>> '/../../build')
>>>>> + process.run('rm -rf ' + build_dir + '/sstate-cache',
>>>>> sudo=True)
>>>>> + self.deletetmp(build_dir)
>>>>> +
>>>>> + start = time.time()
>>>>> + self.perform_build_test(targets, 1, None)
>>>>> + first_time = time.time() - start
>>>>> + self.log.info('Non-cached build:' + str(round(first_time,
>>>>> 2))) +
>>>>> + # Cleanup everything but cache files
>>>>> + self.deletetmp(build_dir)
>>>>> +
>>>>> + start = time.time()
>>>>> + self.perform_build_test(targets, 1, None)
>>>>> + second_time = time.time() - start
>>>>> + self.log.info('Cached build:' + str(round(second_time,
>>>>> 2))) +
>>>>> + if first_time / second_time < speedup_k:
>>>>> + self.fail('No speedup after rebuild with
>>>>> sstate-cache')
>>>> Is there no better way than measuring time to test if sstate was in
>>>> place? Maybe some hit/miss statistics?
>>>>
>>>> Jan
>>>>
>>> I've tried to find some Avocado API to access final testcase
>>> statistics from within other testcase but didn't find so far.
>>> Any advice is welcome.
>>
>> sstate hit rates are in the bitbake logs, i hope avocado can open
>> artifacts
>
> You want to look for the line "Sstate summary", which is shown at the beginning of the bitbake run, e.g.:
>
> Sstate summary: Wanted 121 Local 121 Network 0 Missed 0 Current 0 (100% match, 0% complete)
>
> But in a test it's probably better to examine log.task_order and check which tasks were executed.
>
>> But this is infra ... we will see when it fails and do not have to test
>> that.
>
> It's worth running some tests to see that builds succeed with artifacts taken from cache. I can imagine something like:
>
> - Build image to populate cache
> - Remove TMPDIR
> - Rebuild the image. Should have 100% cache hits. (This would take the target rootfs from cache, and the buildchroot required for image-building)
> - Remove TMPDIR
> - Build a single package (that was already contained in the image). Should have a cache hit, and only dpkg_build_setscene and deploy_deb should run (plus minimal steps to initialize isar-apt).
>
> And optionally:
> - Remove TMPDIR
> - Force rebuild of one package (e.g. by deleting its sstate artifact)
> - Rebuild the image (should take the buildchroot and all but the one package from cache, and rebuild the one package and the image)
>
BTW, we likely now also need a cleansstate target, like OE has.
Jan
--
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-11-02 7:55 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-10-28 15:20 [PATCH] CI: Add sstate-cache testcase Anton Mikanovich
2021-10-28 15:34 ` Jan Kiszka
2021-10-28 17:22 ` Anton Mikanovich
2021-10-28 18:21 ` Henning Schild
2021-11-02 5:57 ` Schmidt, Adriaan
2021-11-02 7:55 ` Jan Kiszka
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox