[PATCH 0/6] Fix essentials rebuild and cover with CI

[PATCH 0/6] Fix essentials rebuild and cover with CI

[Anton Mikanovich]

Add sed package as an example of rebuiled essential package to be used
during the build inside sbuild chroot.

This patchset fixes the following use case:

$ bitbake -c build sed
$ echo -e "\npython do_rootfs_install:append() {\n\n}" >> /work/meta/classes/rootfs.bbclass
$ bitbake -c build sed

Cover it with full CI, also cover f62de6792 with the following case:

$ bitbake -c local_isarapt cowsay
$ bitbake -c build sed
$ bitbake -c cleansstate sbuild-chroot-target
$ bitbake -c build cowsay

Also introduce new CI APIs:
- Obtaining of package version installed in rootfs.
- Inserting custom sbuild hooks during test case.

Anton Mikanovich (6):
  CI: Add API for checking package version in rootfs
  CI: Add API for sbuild hook insertion
  CI: Add sed as essential rebuild test package
  CI: Add essential packages rebuild test case
  CI: Fix getVars API on fast recalling
  meta: Do not update isar-apt during sbuild chroot prepare

 meta-test/recipes-extended/sed/sed.bb         |  15 +++
 .../sbuild-chroot/sbuild-chroot.inc           |   8 +-
 testsuite/cibase.py                           | 101 ++++++++++++++++++
 testsuite/cibuilder.py                        |  26 ++++-
 testsuite/citest.py                           |  25 ++++-
 testsuite/utils.py                            |   6 ++
 6 files changed, 178 insertions(+), 3 deletions(-)
 create mode 100644 meta-test/recipes-extended/sed/sed.bb

-- 
2.34.1

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250221154843.515037-1-amikan%40ilbers.de.

[PATCH 1/6] CI: Add API for checking package version in rootfs

[Anton Mikanovich]

It will output the version installed in the rootfs.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/cibuilder.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index b5b6a093..e671a4ae 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -1,7 +1,7 @@
 #!/usr/bin/env python3
 #
 # This software is a part of ISAR.
-# Copyright (C) 2022-2024 ilbers GmbH
+# Copyright (C) 2022-2025 ilbers GmbH
 # Copyright (C) 2022-2024 Siemens AG
 #
 # SPDX-License-Identifier: MIT
@@ -335,6 +335,10 @@ class CIBuilder(Test):
         bb.utils.edit_bblayers_conf(bblayersconf_file, None, tmp_layer_dir)
         bb.utils.prunedir(tmp_layer_dir)
 
+    def get_pkg_ver(self, rootfs, package):
+        cmd = f"dpkg-query --root={rootfs} -f='${{Version}}' -W {package}"
+        return process.getoutput(cmd)
+
     def get_ssh_cmd_prefix(self, user, host, port, priv_key):
         cmd_prefix = (
             f"ssh -o ConnectTimeout=5 -o StrictHostKeyChecking=no -p {port} "
-- 
2.34.1

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250221154843.515037-2-amikan%40ilbers.de.

[PATCH 2/6] CI: Add API for sbuild hook insertion

[Anton Mikanovich]

Custom hook will be added to dpkg.bbclass. Method will return the file
name to be restored after the testing to be sure the sources were not
changed after testsuite running.
This API will be used later for build dependencies checking.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/cibuilder.py | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index e671a4ae..d335dbe8 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -339,6 +339,22 @@ class CIBuilder(Test):
         cmd = f"dpkg-query --root={rootfs} -f='${{Version}}' -W {package}"
         return process.getoutput(cmd)
 
+    def set_sbuild_hook(self, hook, type='post-build-commands'):
+        append_data = f"        --{type}=\"{hook}\" \\\n"
+
+        layerdir_core = CIUtils.getVars('LAYERDIR_core')
+        sbuildclass_file = os.path.join(layerdir_core, 'classes/dpkg.bbclass')
+        self.backupfile(sbuildclass_file)
+        with open(sbuildclass_file, 'r') as file:
+            lines = file.readlines()
+        with open(sbuildclass_file, 'w') as file:
+            for line in lines:
+                file.write(line)
+                if 'sbuild -A -n' in line:
+                    file.write(append_data)
+
+        return sbuildclass_file
+
     def get_ssh_cmd_prefix(self, user, host, port, priv_key):
         cmd_prefix = (
             f"ssh -o ConnectTimeout=5 -o StrictHostKeyChecking=no -p {port} "
-- 
2.34.1

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250221154843.515037-3-amikan%40ilbers.de.

[PATCH 3/6] CI: Add sed as essential rebuild test package

[Anton Mikanovich]

It will be used as an example of rebuilding essential package for
using inside sbuild chroot during other package building.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta-test/recipes-extended/sed/sed.bb | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
 create mode 100644 meta-test/recipes-extended/sed/sed.bb

diff --git a/meta-test/recipes-extended/sed/sed.bb b/meta-test/recipes-extended/sed/sed.bb
new file mode 100644
index 00000000..c981defe
--- /dev/null
+++ b/meta-test/recipes-extended/sed/sed.bb
@@ -0,0 +1,15 @@
+# This software is a part of ISAR.
+#
+# Copyright (C) 2025 ilbers GmbH
+#
+# SPDX-License-Identifier: MIT
+
+inherit dpkg
+
+SRC_URI = "apt://${PN}"
+
+CHANGELOG_V = "<orig-version>+isar"
+
+do_prepare_build() {
+	deb_add_changelog
+}
-- 
2.34.1

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250221154843.515037-4-amikan%40ilbers.de.

[PATCH 4/6] CI: Add essential packages rebuild test case

[Anton Mikanovich]

Rebuilding of essential packages (already present in sbuild chroot) was
not covered with CI before. Add a testcase for such a fuctionality.

This test case also covers f62de679 change already merged.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/cibase.py    | 101 +++++++++++++++++++++++++++++++++++++++++
 testsuite/cibuilder.py |   4 ++
 testsuite/citest.py    |  25 +++++++++-
 3 files changed, 129 insertions(+), 1 deletion(-)

diff --git a/testsuite/cibase.py b/testsuite/cibase.py
index 075535b1..c692719a 100755
--- a/testsuite/cibase.py
+++ b/testsuite/cibase.py
@@ -1,4 +1,10 @@
 #!/usr/bin/env python3
+#
+# This software is a part of ISAR.
+# Copyright (C) 2022-2025 ilbers GmbH
+# Copyright (C) 2022-2024 Siemens AG
+#
+# SPDX-License-Identifier: MIT
 
 import glob
 import os
@@ -359,3 +365,98 @@ class CIBaseTest(CIBuilder):
                         self.fail("Source packages are equal")
         finally:
             self.cleanup_tmp_layer(tmp_layer_dir)
+
+
+class CIEssentialTest(CIBaseTest):
+    def init(self, pkg='sed', dep='enable-fsck', **kwargs):
+        self.__pkg = pkg
+        self.__dep = dep
+        self.__flag = 'isar'
+        super().init(**kwargs)
+
+    def perform_build_test(self, targets, **kwargs):
+        package_targets = []
+        dependant_targets = []
+        localapt_targets = []
+        cleanup_targets = []
+        rootfs_dirs = {}
+        work_dirs = {}
+        for target in targets:
+            package_target = target.split(':', 3)
+            dependant_target = package_target.copy()
+            cleanup_target = package_target.copy()
+
+            package_target[-1] = self.__pkg
+            dependant_target[-1] = self.__dep
+            cleanup_target[-1] = 'sbuild-chroot-target:do_cleansstate'
+
+            package_targets.append(':'.join(package_target))
+            dep_str = ':'.join(dependant_target)
+            dependant_targets.append(dep_str)
+            localapt_targets.append(dep_str + ':do_local_isarapt')
+            cleanup_targets.append(':'.join(cleanup_target))
+
+            rootfs_dirs[target] = CIUtils.getVars(
+                'IMAGE_ROOTFS', target=target
+            )
+
+            work_dirs[dep_str] = CIUtils.getVars('WORKDIR', target=dep_str)
+
+        self.configure(essentials=self.__pkg, **kwargs)
+
+        cmd = (
+            "%e dpkg-query -f='\${Version}' -W "
+            f"{self.__pkg} > ${{WORKDIR}}/ci_vercheck"
+        )
+        sbuildclass_file = self.set_sbuild_hook(cmd)
+
+        try:
+            # Build local isar-apt copy
+            self.bitbake(localapt_targets, **kwargs)
+
+            # Build package for target
+            self.bitbake(package_targets, **kwargs)
+
+            # Cleanup existing sbuild chroot
+            self.bitbake(cleanup_targets, **kwargs)
+
+            # Build other package
+            self.bitbake(dependant_targets, **kwargs)
+
+            # Ensure updated build dep used
+            for target in dependant_targets:
+                check_path = os.path.join(work_dirs[target], 'ci_vercheck')
+                with open(check_path) as f:
+                    ver = f.read()
+                if self.__flag not in ver:
+                    self.fail(
+                        f"Package {self.__pkg} of version {ver} used for "
+                        f"building don't have {self.__flag} flag"
+                    )
+        finally:
+            self.restorefile(sbuildclass_file)
+
+        # Change rootfs bbclass and rebuild
+        layerdir_core = CIUtils.getVars('LAYERDIR_core')
+        rfsclass_file = os.path.join(layerdir_core, 'classes/rootfs.bbclass')
+        self.backupfile(rfsclass_file)
+        try:
+            with open(rfsclass_file, 'a') as file:
+                file.write('python do_rootfs_install:append() {\n\n}')
+
+            # Build again package for target
+            self.bitbake(package_targets, **kwargs)
+        finally:
+            self.restorefile(rfsclass_file)
+
+        # Build full image after all
+        self.bitbake(targets, **kwargs)
+
+        # Ensure target rootfs use non-updated package
+        for target in targets:
+            ver = self.get_pkg_ver(rootfs_dirs[target], self.__pkg)
+            if self.__flag in ver:
+                self.fail(
+                    f"Package {self.__pkg} of version {ver} used for "
+                    f"installing have {self.__flag} flag, but shouldn't"
+                )
diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index d335dbe8..5aa64e31 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -109,6 +109,7 @@ class CIBuilder(Test):
         source_date_epoch=None,
         use_apt_snapshot=False,
         image_install=None,
+        essentials=None,
         **kwargs,
     ):
         # write configuration file and set bitbake_args
@@ -155,6 +156,7 @@ class CIBuilder(Test):
             f"  sstate_dir = {sstate_dir}\n"
             f"  ccache_dir = {ccache_dir}\n"
             f"  image_install = {image_install}\n"
+            f"  essentials = {essentials}\n"
             f"==================================================="
         )
 
@@ -216,6 +218,8 @@ class CIBuilder(Test):
                 f.write('IMAGE_INSTALL = "%s"\n' % image_install)
             if fail_on_cleanup == '1':
                 f.write('ISAR_FAIL_ON_CLEANUP = "1"\n')
+            if essentials:
+                f.write('ISAR_REBUILD_ESSENTIAL_PKGS = "%s"\n' % essentials)
 
         # include ci_build.conf in local.conf
         with open(self.build_dir + '/conf/local.conf', 'r+') as f:
diff --git a/testsuite/citest.py b/testsuite/citest.py
index ee965278..9e2c1b43 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -1,9 +1,15 @@
 #!/usr/bin/env python3
+#
+# This software is a part of ISAR.
+# Copyright (C) 2022-2025 ilbers GmbH
+# Copyright (C) 2022-2025 Siemens AG
+#
+# SPDX-License-Identifier: MIT
 
 from avocado import skipUnless
 from avocado.core import exceptions
 from avocado.utils import path
-from cibase import CIBaseTest
+from cibase import CIBaseTest, CIEssentialTest
 from utils import CIUtils
 
 UMOCI_AVAILABLE = True
@@ -400,6 +406,23 @@ class SstateTest(CIBaseTest):
         self.perform_sstate_test(image_target, package_target)
 
 
+class EssentialTest(CIEssentialTest):
+
+    """
+    Essential packages rebuild test
+
+    :avocado: tags=essential,full
+    """
+
+    def test_essential(self):
+        targets = [
+            'mc:qemuamd64-bookworm:isar-image-base',
+        ]
+
+        self.init(pkg="sed", dep="enable-fsck")
+        self.perform_build_test(targets)
+
+
 class SingleTest(CIBaseTest):
 
     """
-- 
2.34.1

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250221154843.515037-5-amikan%40ilbers.de.

[PATCH 5/6] CI: Fix getVars API on fast recalling

[Anton Mikanovich]

Running getVars several times in a row sometimes failing with
'Reconnecting to bitbake server' messages. Fix this by waiting a bit
before trying to check bitbake lock file state.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 testsuite/utils.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/testsuite/utils.py b/testsuite/utils.py
index ffcb5cc7..e82f5a22 100755
--- a/testsuite/utils.py
+++ b/testsuite/utils.py
@@ -1,4 +1,9 @@
 #!/usr/bin/env python3
+#
+# This software is a part of ISAR.
+# Copyright (C) 2024-2025 ilbers GmbH
+#
+# SPDX-License-Identifier: MIT
 
 import os
 import sys
@@ -27,6 +32,7 @@ class CIUtils():
             fixStream(sys.stderr)
 
         # wait until previous bitbake will be finished
+        time.sleep(1)
         lockfile = os.path.join(os.getcwd(), 'bitbake.lock')
         checks = 0
         while os.path.exists(lockfile) and checks < 5:
-- 
2.34.1

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250221154843.515037-6-amikan%40ilbers.de.

[PATCH 6/6] meta: Do not update isar-apt during sbuild chroot prepare

[Anton Mikanovich]

After moving to manual isar-apt update during package build in 0e846829
there is no need in obtaining isar-apt package list during sbuild
rootfs prepare. Moreover, it can results in out of sync between the
data in /var/lib/apt/lists and local isar-apt copy of the recipe.
In case of rebuilding the package already included into sbuild chroot
this sync lost can raise 'File not found' issues with updated debs.
Remove isar-apt update during sbuild chroot prepare and do it later
for every package separately based on local isar-apt copy.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
index 6478d3a6..f5d1f1dd 100644
--- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -1,7 +1,10 @@
 # Common part for build chroot filesystem.
 #
 # This software is a part of ISAR.
-# Copyright (C) 2015-2021 ilbers GmbH
+# Copyright (C) 2015-2025 ilbers GmbH
+# Copyright (C) 2023-2025 Siemens AG
+#
+# SPDX-License-Identifier: MIT
 
 LICENSE = "gpl-2.0"
 LIC_FILES_CHKSUM = "file://${LAYERDIR_core}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
@@ -56,6 +59,9 @@ SBUILD_CHROOT_DIR = "${WORKDIR}/rootfs"
 ROOTFSDIR = "${SBUILD_CHROOT_DIR}"
 ROOTFS_PACKAGES = "${SBUILD_CHROOT_PREINSTALL}"
 
+# We need isar-apt configured but not updated yet
+rootfs_install_pkgs_download[isar-apt-lock] = ""
+ROOTFS_INSTALL_COMMAND:remove = "rootfs_install_pkgs_update"
 ROOTFS_POSTPROCESS_COMMAND:remove = "rootfs_cleanup_isar_apt"
 
 # Do not cleanup base-apt
-- 
2.34.1

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250221154843.515037-7-amikan%40ilbers.de.

Re: [PATCH 6/6] meta: Do not update isar-apt during sbuild chroot prepare

['Jan Kiszka' via isar-users]

On 21.02.25 16:48, Anton Mikanovich wrote:
> After moving to manual isar-apt update during package build in 0e846829
> there is no need in obtaining isar-apt package list during sbuild
> rootfs prepare. Moreover, it can results in out of sync between the
> data in /var/lib/apt/lists and local isar-apt copy of the recipe.
> In case of rebuilding the package already included into sbuild chroot
> this sync lost can raise 'File not found' issues with updated debs.

Why is the sync we do inside sbuild not curing this de-synchronization?

> Remove isar-apt update during sbuild chroot prepare and do it later
> for every package separately based on local isar-apt copy.

What about building a custom schroot that should have a self-built,
isar-apt-originated package included? Won't that fail now?

Practically, users could just pull such packages whenever they build
something in that custom schroot, but having to handle self-built
packages differently than those coming from external repos is at least
not intuitive.

Jan

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/7632446b-0eab-406c-be91-ccabcc4c4711%40siemens.com.

Re: [PATCH 6/6] meta: Do not update isar-apt during sbuild chroot prepare

[Anton Mikanovich]

21/02/2025 18:34, Jan Kiszka wrote:
> On 21.02.25 16:48, Anton Mikanovich wrote:
>> After moving to manual isar-apt update during package build in 0e846829
>> there is no need in obtaining isar-apt package list during sbuild
>> rootfs prepare. Moreover, it can results in out of sync between the
>> data in /var/lib/apt/lists and local isar-apt copy of the recipe.
>> In case of rebuilding the package already included into sbuild chroot
>> this sync lost can raise 'File not found' issues with updated debs.
> Why is the sync we do inside sbuild not curing this de-synchronization?
That's a good question I was unable to answer: if /var/lib/apt/lists has an
information about the package which not exists in the repo, apt still 
tries to
install that package.
>> Remove isar-apt update during sbuild chroot prepare and do it later
>> for every package separately based on local isar-apt copy.
> What about building a custom schroot that should have a self-built,
> isar-apt-originated package included? Won't that fail now?

If it is the case covered with hello-isar application - it is still working.
Isar-apt is populated with libhello-dev because of fixed deps, then manual
apt-get update will be performed on isar-apt only inside sbuild hook, so
libhello-dev will be installed into schroot for hello-isar building.

> Practically, users could just pull such packages whenever they build
> something in that custom schroot, but having to handle self-built
> packages differently than those coming from external repos is at least
> not intuitive.
>
> Jan
>

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/9e9b8292-2150-47e6-a035-4a948b8a4df4%40ilbers.de.

Re: [PATCH 6/6] meta: Do not update isar-apt during sbuild chroot prepare

['Jan Kiszka' via isar-users]

On 03.03.25 11:13, Anton Mikanovich wrote:
> 21/02/2025 18:34, Jan Kiszka wrote:
>> On 21.02.25 16:48, Anton Mikanovich wrote:
>>> After moving to manual isar-apt update during package build in 0e846829
>>> there is no need in obtaining isar-apt package list during sbuild
>>> rootfs prepare. Moreover, it can results in out of sync between the
>>> data in /var/lib/apt/lists and local isar-apt copy of the recipe.
>>> In case of rebuilding the package already included into sbuild chroot
>>> this sync lost can raise 'File not found' issues with updated debs.
>> Why is the sync we do inside sbuild not curing this de-synchronization?
> That's a good question I was unable to answer: if /var/lib/apt/lists has an
> information about the package which not exists in the repo, apt still
> tries to
> install that package.
>>> Remove isar-apt update during sbuild chroot prepare and do it later
>>> for every package separately based on local isar-apt copy.
>> What about building a custom schroot that should have a self-built,
>> isar-apt-originated package included? Won't that fail now?
> 
> If it is the case covered with hello-isar application - it is still
> working.
> Isar-apt is populated with libhello-dev because of fixed deps, then manual
> apt-get update will be performed on isar-apt only inside sbuild hook, so
> libhello-dev will be installed into schroot for hello-isar building.
> 

But will the custom schroot already contain libhello-dev, or will
schroot simply install while setting up the build env for hello-isar?
I'm referring to SBUILD_CHROOT_PREINSTALL_EXTRA for self-built packages.

Jan

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/f8f31958-d62b-4d21-9376-1065733a7dc2%40siemens.com.

Re: [PATCH 6/6] meta: Do not update isar-apt during sbuild chroot prepare

[Anton Mikanovich]

03/03/2025 16:32, Jan Kiszka wrote:
> But will the custom schroot already contain libhello-dev, or will
> schroot simply install while setting up the build env for hello-isar?
> I'm referring to SBUILD_CHROOT_PREINSTALL_EXTRA for self-built packages.
>
> Jan
>
No, using custom packages in SBUILD_CHROOT_PREINSTALL_EXTRA is not covered
in CI (but should be) and probably will not work as expected.
I'm going back to the fix.

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/7b043bce-97db-4eb9-87ea-7bdb91fa6963%40ilbers.de.