[PATCH] buildchroot: Allow downgrades when installing packages

[PATCH] buildchroot: Allow downgrades when installing packages

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This can happen when changing the recipe of a self-built package, and
that already when going from v1.0-rc3 to v1.0.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/image.bbclass                      | 2 +-
 meta/recipes-devtools/buildchroot/files/deps.sh | 4 +++-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 3f4b728..574fb46 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -209,7 +209,7 @@ do_install_imager_deps() {
             -o Dir::Etc::sourceparts="-" \
             -o APT::Get::List-Cleanup="0"
         apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y \
-            --allow-unauthenticated install \
+            --allow-unauthenticated --allow-downgrades install \
             ${IMAGER_INSTALL}'
 }

diff --git a/meta/recipes-devtools/buildchroot/files/deps.sh b/meta/recipes-devtools/buildchroot/files/deps.sh
index 2501f05..1963943 100644
--- a/meta/recipes-devtools/buildchroot/files/deps.sh
+++ b/meta/recipes-devtools/buildchroot/files/deps.sh
@@ -10,7 +10,9 @@ source /isar/common.sh
 # Notes:
 #   1) everything before the -y switch is unchanged from the defaults
 #   2) we add -y to go non-interactive
-install_cmd="apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y"
+#   3) downgrades shall be allowed in case a package recipe was changed
+install_cmd="apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends \
+    -y --allow-downgrades"

 # Make sure that we have latest isar-apt content.
 # Options meaning:
--
2.16.4

Re: [PATCH] buildchroot: Allow downgrades when installing packages

[Henning Schild]

Am Sat, 23 Feb 2019 11:44:21 +0100
schrieb Jan Kiszka <jan.kiszka@web.de>:

> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> This can happen when changing the recipe of a self-built package, and
> that already when going from v1.0-rc3 to v1.0.

Seems pretty risky. If the only motivation is a local package, i would
argue your layer has a bug because it is going "up and down". If it
somehow needs to ... go ahead and do a complete rebuild.
But i guess we are talking about a multiconfig story again ...
I am against this one, we should rather move to the "clean-room" builds

Henning

> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>  meta/classes/image.bbclass                      | 2 +-
>  meta/recipes-devtools/buildchroot/files/deps.sh | 4 +++-
>  2 files changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index 3f4b728..574fb46 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -209,7 +209,7 @@ do_install_imager_deps() {
>              -o Dir::Etc::sourceparts="-" \
>              -o APT::Get::List-Cleanup="0"
>          apt-get -o Debug::pkgProblemResolver=yes
> --no-install-recommends -y \
> -            --allow-unauthenticated install \
> +            --allow-unauthenticated --allow-downgrades install \
>              ${IMAGER_INSTALL}'
>  }
> 
> diff --git a/meta/recipes-devtools/buildchroot/files/deps.sh
> b/meta/recipes-devtools/buildchroot/files/deps.sh index
> 2501f05..1963943 100644 ---
> a/meta/recipes-devtools/buildchroot/files/deps.sh +++
> b/meta/recipes-devtools/buildchroot/files/deps.sh @@ -10,7 +10,9 @@
> source /isar/common.sh # Notes:
>  #   1) everything before the -y switch is unchanged from the defaults
>  #   2) we add -y to go non-interactive
> -install_cmd="apt-get -o Debug::pkgProblemResolver=yes
> --no-install-recommends -y" +#   3) downgrades shall be allowed in
> case a package recipe was changed +install_cmd="apt-get -o
> Debug::pkgProblemResolver=yes --no-install-recommends \
> +    -y --allow-downgrades"
> 
>  # Make sure that we have latest isar-apt content.
>  # Options meaning:
> --
> 2.16.4
> 

Re: [PATCH] buildchroot: Allow downgrades when installing packages

[Jan Kiszka]

On 25.02.19 10:39, [ext] Henning Schild wrote:
> Am Sat, 23 Feb 2019 11:44:21 +0100
> schrieb Jan Kiszka <jan.kiszka@web.de>:
> 
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> This can happen when changing the recipe of a self-built package, and
>> that already when going from v1.0-rc3 to v1.0.
> 
> Seems pretty risky. If the only motivation is a local package, i would
> argue your layer has a bug because it is going "up and down". If it

Look at the example I cited: It was an upgrade.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH] buildchroot: Allow downgrades when installing packages

[Henning Schild]

Am Mon, 25 Feb 2019 10:57:33 +0100
schrieb Jan Kiszka <jan.kiszka@siemens.com>:

> On 25.02.19 10:39, [ext] Henning Schild wrote:
> > Am Sat, 23 Feb 2019 11:44:21 +0100
> > schrieb Jan Kiszka <jan.kiszka@web.de>:
> >   
> >> From: Jan Kiszka <jan.kiszka@siemens.com>
> >>
> >> This can happen when changing the recipe of a self-built package,
> >> and that already when going from v1.0-rc3 to v1.0.  
> > 
> > Seems pretty risky. If the only motivation is a local package, i
> > would argue your layer has a bug because it is going "up and down".
> > If it  
> 
> Look at the example I cited: It was an upgrade.

In that case your version was wrong.

...  The format is: [epoch:]upstream_version[-debian_revision].

... The upstream_version may contain only alphanumerics [6] and the
characters . + - ~ (full stop, plus, hyphen, tilde) and should start
with a digit. If there is no debian_revision then hyphens are not
allowed.

https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-version

So it should have been v1.0-rc3-deb0, or v1.0+rc3 ...
It is a bug in your layer not in Isar.

Henning

> Jan
> 

Re: [PATCH] buildchroot: Allow downgrades when installing packages

[Jan Kiszka]

On 25.02.19 11:15, Henning Schild wrote:
> Am Mon, 25 Feb 2019 10:57:33 +0100
> schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> 
>> On 25.02.19 10:39, [ext] Henning Schild wrote:
>>> Am Sat, 23 Feb 2019 11:44:21 +0100
>>> schrieb Jan Kiszka <jan.kiszka@web.de>:
>>>    
>>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>>
>>>> This can happen when changing the recipe of a self-built package,
>>>> and that already when going from v1.0-rc3 to v1.0.
>>>
>>> Seems pretty risky. If the only motivation is a local package, i
>>> would argue your layer has a bug because it is going "up and down".
>>> If it
>>
>> Look at the example I cited: It was an upgrade.
> 
> In that case your version was wrong.
> 
> ...  The format is: [epoch:]upstream_version[-debian_revision].
> 
> ... The upstream_version may contain only alphanumerics [6] and the
> characters . + - ~ (full stop, plus, hyphen, tilde) and should start
> with a digit. If there is no debian_revision then hyphens are not
> allowed.
> 
> https://www.debian.org/doc/debian-policy/ch-controlfields.html#s-f-version
> 
> So it should have been v1.0-rc3-deb0, or v1.0+rc3 ...
> It is a bug in your layer not in Isar.

Removing -rcX from a version string is lexically a downgrade, logically not. I'm 
not that sure if you can compensate this.

Anyway, a real downgrade is a valid development scenario as well for our isar-apt.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH] buildchroot: Allow downgrades when installing packages

[Claudius Heine]

Hi Jan,

On 23/02/2019 11.44, Jan Kiszka wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> This can happen when changing the recipe of a self-built package, and
> that already when going from v1.0-rc3 to v1.0.

So the issue is that the buildchroot is reused for each recipe/package 
and if a newer one was once installed, it will not be removed. Right?

Then why not use a copy of the buildchroot for each recipe?

That way missing dependencies would also be found more easily.

regards,
Claudius

> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>   meta/classes/image.bbclass                      | 2 +-
>   meta/recipes-devtools/buildchroot/files/deps.sh | 4 +++-
>   2 files changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index 3f4b728..574fb46 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -209,7 +209,7 @@ do_install_imager_deps() {
>               -o Dir::Etc::sourceparts="-" \
>               -o APT::Get::List-Cleanup="0"
>           apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y \
> -            --allow-unauthenticated install \
> +            --allow-unauthenticated --allow-downgrades install \
>               ${IMAGER_INSTALL}'
>   }
> 
> diff --git a/meta/recipes-devtools/buildchroot/files/deps.sh b/meta/recipes-devtools/buildchroot/files/deps.sh
> index 2501f05..1963943 100644
> --- a/meta/recipes-devtools/buildchroot/files/deps.sh
> +++ b/meta/recipes-devtools/buildchroot/files/deps.sh
> @@ -10,7 +10,9 @@ source /isar/common.sh
>   # Notes:
>   #   1) everything before the -y switch is unchanged from the defaults
>   #   2) we add -y to go non-interactive
> -install_cmd="apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y"
> +#   3) downgrades shall be allowed in case a package recipe was changed
> +install_cmd="apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends \
> +    -y --allow-downgrades"
> 
>   # Make sure that we have latest isar-apt content.
>   # Options meaning:
> --
> 2.16.4
> 

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [PATCH] buildchroot: Allow downgrades when installing packages

[Jan Kiszka]

On 04.03.19 10:04, [ext] Claudius Heine wrote:
> Hi Jan,
> 
> On 23/02/2019 11.44, Jan Kiszka wrote:
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> This can happen when changing the recipe of a self-built package, and
>> that already when going from v1.0-rc3 to v1.0.
> 
> So the issue is that the buildchroot is reused for each recipe/package and if a 
> newer one was once installed, it will not be removed. Right?
> 
> Then why not use a copy of the buildchroot for each recipe?
> 
> That way missing dependencies would also be found more easily.

Yes, that is the future direction (post-0.7), possibly achieved by switching to 
sbuilder. For now, we need to deal with the shared buildchroot.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH] buildchroot: Allow downgrades when installing packages

[Baurzhan Ismagulov]

On Mon, Feb 25, 2019 at 12:08:23PM +0100, Jan Kiszka wrote:
> Removing -rcX from a version string is lexically a downgrade, logically not.
> I'm not that sure if you can compensate this.

Debian uses tilde for that, e.g. 1.0~rc1-1+deb9u2. This is addressed in
https://www.debian.org/doc/debian-policy/ch-controlfields.html#version:

  "First the initial part of each string consisting entirely of non-digit
  characters is determined. These two parts (one of which may be empty) are
  compared lexically. If a difference is found it is returned. The lexical
  comparison is a comparison of ASCII values modified so that all the letters
  sort earlier than all the non-letters and so that a tilde sorts before
  anything, even the end of a part. For example, the following parts are in
  sorted order from earliest to latest: ~~, ~~a, ~, the empty part, a. [7]

  [7] One common use of ~ is for upstream pre-releases. For example,
      1.0~beta1~svn1245 sorts earlier than 1.0~beta1, which sorts earlier than
      1.0."


> Anyway, a real downgrade is a valid development scenario as well for our
> isar-apt.

Downgrade is supported by the packaging toolchain (dpkg / apt), but not by the
distribution (packages themselves, notably postinst, etc.).
https://www.debian.org/doc/manuals/debian-reference/ch02.en.html#_emergency_downgrading
says:

  "Downgrading is not officially supported by the Debian by design. It should
  be done only as a part of emergency recovery process. Despite of this
  situation, it is known to work well in many incidents. For critical systems,
  you should backup all important data on the system after the recovery
  operation and re-install the new system from the scratch.

  You may be lucky to downgrade from newer archive to older archive to recover
  from broken system upgrade by manipulating candidate version..."

So I'd be reluctant to do that by default. What about extracting the apt
options into a config file, where the user may add his options?


With kind regards,
Baurzhan.

Re: [PATCH] buildchroot: Allow downgrades when installing packages

[Jan Kiszka]

On 19.03.19 11:10, Baurzhan Ismagulov wrote:
> On Mon, Feb 25, 2019 at 12:08:23PM +0100, Jan Kiszka wrote:
>> Removing -rcX from a version string is lexically a downgrade, logically not.
>> I'm not that sure if you can compensate this.
> 
> Debian uses tilde for that, e.g. 1.0~rc1-1+deb9u2. This is addressed in
> https://www.debian.org/doc/debian-policy/ch-controlfields.html#version:
> 
>    "First the initial part of each string consisting entirely of non-digit
>    characters is determined. These two parts (one of which may be empty) are
>    compared lexically. If a difference is found it is returned. The lexical
>    comparison is a comparison of ASCII values modified so that all the letters
>    sort earlier than all the non-letters and so that a tilde sorts before
>    anything, even the end of a part. For example, the following parts are in
>    sorted order from earliest to latest: ~~, ~~a, ~, the empty part, a. [7]
> 
>    [7] One common use of ~ is for upstream pre-releases. For example,
>        1.0~beta1~svn1245 sorts earlier than 1.0~beta1, which sorts earlier than
>        1.0."
> 

Good to know. We will have to educate people coming from OE about this 
non-intuitive deviation from how things work over there.

> 
>> Anyway, a real downgrade is a valid development scenario as well for our
>> isar-apt.
> 
> Downgrade is supported by the packaging toolchain (dpkg / apt), but not by the
> distribution (packages themselves, notably postinst, etc.).
> https://www.debian.org/doc/manuals/debian-reference/ch02.en.html#_emergency_downgrading
> says:
> 
>    "Downgrading is not officially supported by the Debian by design. It should
>    be done only as a part of emergency recovery process. Despite of this
>    situation, it is known to work well in many incidents. For critical systems,
>    you should backup all important data on the system after the recovery
>    operation and re-install the new system from the scratch.
> 
>    You may be lucky to downgrade from newer archive to older archive to recover
>    from broken system upgrade by manipulating candidate version..."
> 
> So I'd be reluctant to do that by default. What about extracting the apt
> options into a config file, where the user may add his options?

I don't see that we should add a redundant, counter-intuitive option. The user 
already clearly expressed that packages of choice should be installed. This is 
what whole Isar is about: customization. Moreover, the user will get exactly 
that by doing a clean rebuild.

So, please merge.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH] buildchroot: Allow downgrades when installing packages

[Maxim Yu. Osipov]

Hi Jan,

Tried to apply the patch - CI build failed.

See for details:
http://isar-build.org:8080/job/isar_mosipov_next/164/console

Regards,
Maxim.

On 3/19/19 11:31 AM, Jan Kiszka wrote:
> On 19.03.19 11:10, Baurzhan Ismagulov wrote:
>> On Mon, Feb 25, 2019 at 12:08:23PM +0100, Jan Kiszka wrote:
>>> Removing -rcX from a version string is lexically a downgrade, 
>>> logically not.
>>> I'm not that sure if you can compensate this.
>>
>> Debian uses tilde for that, e.g. 1.0~rc1-1+deb9u2. This is addressed in
>> https://www.debian.org/doc/debian-policy/ch-controlfields.html#version:
>>
>>    "First the initial part of each string consisting entirely of 
>> non-digit
>>    characters is determined. These two parts (one of which may be 
>> empty) are
>>    compared lexically. If a difference is found it is returned. The 
>> lexical
>>    comparison is a comparison of ASCII values modified so that all the 
>> letters
>>    sort earlier than all the non-letters and so that a tilde sorts before
>>    anything, even the end of a part. For example, the following parts 
>> are in
>>    sorted order from earliest to latest: ~~, ~~a, ~, the empty part, 
>> a. [7]
>>
>>    [7] One common use of ~ is for upstream pre-releases. For example,
>>        1.0~beta1~svn1245 sorts earlier than 1.0~beta1, which sorts 
>> earlier than
>>        1.0."
>>
> 
> Good to know. We will have to educate people coming from OE about this 
> non-intuitive deviation from how things work over there.
> 
>>
>>> Anyway, a real downgrade is a valid development scenario as well for our
>>> isar-apt.
>>
>> Downgrade is supported by the packaging toolchain (dpkg / apt), but 
>> not by the
>> distribution (packages themselves, notably postinst, etc.).
>> https://www.debian.org/doc/manuals/debian-reference/ch02.en.html#_emergency_downgrading 
>>
>> says:
>>
>>    "Downgrading is not officially supported by the Debian by design. 
>> It should
>>    be done only as a part of emergency recovery process. Despite of this
>>    situation, it is known to work well in many incidents. For critical 
>> systems,
>>    you should backup all important data on the system after the recovery
>>    operation and re-install the new system from the scratch.
>>
>>    You may be lucky to downgrade from newer archive to older archive 
>> to recover
>>    from broken system upgrade by manipulating candidate version..."
>>
>> So I'd be reluctant to do that by default. What about extracting the apt
>> options into a config file, where the user may add his options?
> 
> I don't see that we should add a redundant, counter-intuitive option. 
> The user already clearly expressed that packages of choice should be 
> installed. This is what whole Isar is about: customization. Moreover, 
> the user will get exactly that by doing a clean rebuild.
> 
> So, please merge.
> 
> Jan
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: [PATCH] buildchroot: Allow downgrades when installing packages

[Jan Kiszka]

On 19.03.19 18:33, Maxim Yu. Osipov wrote:
> Hi Jan,
> 
> Tried to apply the patch - CI build failed.
> 
> See for details:
> http://isar-build.org:8080/job/isar_mosipov_next/164/console

There are multiple errors, most of them seem unrelated to my change because they 
are affecting the bootstrap step, and I'm patching the package build:

| debconf: delaying package configuration, since apt-utils is not installed
| Fetched 380 kB in 1s (340 kB/s)
| E: Can not write log (Is /dev/pts mounted?) - posix_openpt (19: No such device)
| (Reading database ... 6945 files and directories currently installed.)
| Preparing to unpack .../libsystemd0_232-25+deb9u9_armhf.deb ...
| Unpacking libsystemd0:armhf (232-25+deb9u9) over (232-25+deb9u8) ...
| Setting up libsystemd0:armhf (232-25+deb9u9) ...
| (Reading database ... 6945 files and directories currently installed.)
| Preparing to unpack .../libudev1_232-25+deb9u9_armhf.deb ...
| Unpacking libudev1:armhf (232-25+deb9u9) over (232-25+deb9u8) ...
| Setting up libudev1:armhf (232-25+deb9u9) ...
| Processing triggers for libc-bin (2.24-11+deb9u4) ...
| umount: 
/workspace/build/isar_mosipov_next/164/build/tmp/work/debian-stretch-armhf/isar-bootstrap-target/rootfs/dev: 
not mounted
| WARNING: exit code 32 from a shell command.
| ERROR: Function failed: do_bootstrap (log file is located at 
/workspace/build/isar_mosipov_next/164/build/tmp/work/debian-stretch-armhf/isar-bootstrap-target/temp/log.do_bootstrap.9153)

You should examine that separately.

The one I'm causing is, sight, once again related to legacy jessie:

| Unpacking libhello-build-deps (0.1) ...
| E: Command line option --allow-downgrades is not understood
| (Reading database ... 16093 files and directories currently installed.)
| Removing libhello-build-deps (0.1) ...
| mk-build-deps: apt-get install call failed
| WARNING: exit code 29 from a shell command.
| ERROR: Function failed: do_install_builddeps (log file is located at 
/workspace/build/isar_mosipov_next/164/build/tmp/work/debian-jessie-i386/libhello-0.1-98f2e41-r0/temp/log.do_install_builddeps.481)

This needs apt 1.1, and while strech is on 1.4, jessie was only on 1.0.

OK, as you plan to keep jessor for this release, I would postpone this patch 
after 0.7 and the final removal of that old Debian version.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH] buildchroot: Allow downgrades when installing packages

[Maxim Yu. Osipov]

On 2/23/19 11:44 AM, Jan Kiszka wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> This can happen when changing the recipe of a self-built package, and
> that already when going from v1.0-rc3 to v1.0.

Applied to the 'next'.

Regards,
Maxim.

> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>   meta/classes/image.bbclass                      | 2 +-
>   meta/recipes-devtools/buildchroot/files/deps.sh | 4 +++-
>   2 files changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index 3f4b728..574fb46 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -209,7 +209,7 @@ do_install_imager_deps() {
>               -o Dir::Etc::sourceparts="-" \
>               -o APT::Get::List-Cleanup="0"
>           apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y \
> -            --allow-unauthenticated install \
> +            --allow-unauthenticated --allow-downgrades install \
>               ${IMAGER_INSTALL}'
>   }
> 
> diff --git a/meta/recipes-devtools/buildchroot/files/deps.sh b/meta/recipes-devtools/buildchroot/files/deps.sh
> index 2501f05..1963943 100644
> --- a/meta/recipes-devtools/buildchroot/files/deps.sh
> +++ b/meta/recipes-devtools/buildchroot/files/deps.sh
> @@ -10,7 +10,9 @@ source /isar/common.sh
>   # Notes:
>   #   1) everything before the -y switch is unchanged from the defaults
>   #   2) we add -y to go non-interactive
> -install_cmd="apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y"
> +#   3) downgrades shall be allowed in case a package recipe was changed
> +install_cmd="apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends \
> +    -y --allow-downgrades"
> 
>   # Make sure that we have latest isar-apt content.
>   # Options meaning:
> --
> 2.16.4
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov