Integration of Pseudo into Isar

Integration of Pseudo into Isar

[Andreas Reichel]

# Integrating pseudo into isar

Idea was to exchange `sudo` by `pseudo`. The function of `pseudo` is to
intercept system calls and file accesses by preloading a library. All
such operations are recorded in a database. For this to work, a
`PSEUDO_PREFIX` variable must be seet, which is `/` if `pseudo` is
installed to the default location.

# Given test configuration #

- Docker container based on debian 9
- `multistrap` from Siemens Debian repository
- `pseudo` from Siemens Debian repository

Using the following multistrap configuration named `simple-config`:

```
[General]
unpack=true
bootstrap=Debian
aptsources=Debian
noauth=true

[Debian]
packages=
source=http://ftp.de.debian.org/debian
suite=stretch
```

Inside chroot (which is inside pseudo):

```
# mkdir rootfs
# multistrap -f simple-config -d rootfs
```

# Results #

* Error during package configuration. (Cannot write to `/etc/ld.so.cache~`)

  This error can be tracked down to `ldconfig`.
  It turned out that `ldconfig` is linked *statically*. Which means, its
  file accesses cannot be intercepted by LDPRELOAD, which is only for
  dynamically linked binaries. Thus, wether being in a pseudo chroot or
  not, `ldconfig` will always access `/etc/ld.so.cache~` on the host,
  which fails.
  This is *NOT* a question of the Debian version and not a bug in `dpkg
  --configure -a`, which calls `ldconfig` internally.

* Extremely odd behaviour within `chroot` within `pseudo`:

```
$ pseudo
# chroot rootfs
# export PATH=/sbin:/bin
# ldconfig
Can't create temporary cache file /etc/ld.so.cache~
```

Idea was then to rename `ldconfig` to `ldconfig_` and create a symbolic
link to `/bin/true` to mimic successful execution of `ldconfig`.

```
$ sudo mv rootfs/sbin/ldconfig rootfs/sbin/ldconfig_
$ sudo ln -s /bin/true rootfs/sbin/ldconfig
```

Here, behavior becomes very odd:

```
$ pseudo
# chroot rootfs
# export PATH=/sbin:/bin
# ldconfig
/bin/sh: 16: ldconfig: not found
```
Although it is in path...
```
# /sbin/ldconfig
/sbin/ldconfig: Can't create temporary cache file /etc/ld.so.cache~:
Permission denied
```

So this is not our symbolic link but the real ldconfig from the host

```
# cd /sbin
# ./ldconfig
#
```

This works and returns `true`.

```
# cd /
# /sbin/ldconfig
/sbin/ldconfig: Can't create temporary cache file /etc/ld.so.cache~:
Permission denied
# sbin/ldconfig
#
```

So a relative path works, but an absolute path does not.

Even more funny is:

```
# exec sh
# exit
$
```

The first `exec` replaces the current shell with `sh` from *OUTSIDE* of
the `chroot`. The `exit` then exits the `pseudo` environment instead of
the `chroot` environment.

# Summary #

* operations with `chroot inside pseudo` are completely messed-up.
* `ldconfig` will never work with `pseudo` since it is static.

But if packets are not configured, initramfs is not generated, etc...

# Ideas #

* Yocto Morty uses pseudo with own patches, which may solve some or all issues
* Do not use multistrap but another tool
* Stracing / Kernel tracing to analyze problem more deeply


Kind regards,
Andreas

-- 
Andreas Reichel
Dipl.-Phys. (Univ.)
Software Consultant

Andreas.Reichel@tngtech.com, +49-174-3180074
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterfoehring
Geschaeftsfuehrer: Henrik Klagges, Dr. Robert Dahlke, Gerhard Mueller
Sitz: Unterfoehring * Amtsgericht Muenchen * HRB 135082

Re: Integration of Pseudo into Isar

[Andreas Reichel]

On Wed, Aug 02, 2017 at 03:24:14PM +0200, [ext] Andreas Reichel wrote:
> # Ideas #
> 
> * Yocto Morty uses pseudo with own patches, which may solve some or all issues

Open embedded has version 1.8.2, which exhibits the same issues. The
latest testing-Debian package is 1.8.1.

Kind regards,
Andreas

-- 
Andreas Reichel
Dipl.-Phys. (Univ.)
Software Consultant

Andreas.Reichel@tngtech.com, +49-174-3180074
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterfoehring
Geschaeftsfuehrer: Henrik Klagges, Dr. Robert Dahlke, Gerhard Mueller
Sitz: Unterfoehring * Amtsgericht Muenchen * HRB 135082

Re: Integration of Pseudo into Isar

[Henning Schild]

Hey,

not very good news. But somehow OE made it work and they do not
control /sbin/ldconfig of the host either.

I suggest you rephrase this mail a bit and send it to the pseudo and
maybe OE community with concrete questions. Having isar-users on CC.

Identify a few people for CC with git-blame if needbe.

Henning

Am Wed, 2 Aug 2017 15:24:14 +0200
schrieb "[ext] Andreas Reichel" <andreas.reichel.ext@siemens.com>:

> # Integrating pseudo into isar
> 
> Idea was to exchange `sudo` by `pseudo`. The function of `pseudo` is
> to intercept system calls and file accesses by preloading a library.
> All such operations are recorded in a database. For this to work, a
> `PSEUDO_PREFIX` variable must be seet, which is `/` if `pseudo` is
> installed to the default location.
> 
> # Given test configuration #
> 
> - Docker container based on debian 9
> - `multistrap` from Siemens Debian repository
> - `pseudo` from Siemens Debian repository
> 
> Using the following multistrap configuration named `simple-config`:
> 
> ```
> [General]
> unpack=true
> bootstrap=Debian
> aptsources=Debian
> noauth=true
> 
> [Debian]
> packages=
> source=http://ftp.de.debian.org/debian
> suite=stretch
> ```
> 
> Inside chroot (which is inside pseudo):
> 
> ```
> # mkdir rootfs
> # multistrap -f simple-config -d rootfs
> ```
> 
> # Results #
> 
> * Error during package configuration. (Cannot write to
> `/etc/ld.so.cache~`)
> 
>   This error can be tracked down to `ldconfig`.
>   It turned out that `ldconfig` is linked *statically*. Which means,
> its file accesses cannot be intercepted by LDPRELOAD, which is only
> for dynamically linked binaries. Thus, wether being in a pseudo
> chroot or not, `ldconfig` will always access `/etc/ld.so.cache~` on
> the host, which fails.
>   This is *NOT* a question of the Debian version and not a bug in
> `dpkg --configure -a`, which calls `ldconfig` internally.
> 
> * Extremely odd behaviour within `chroot` within `pseudo`:
> 
> ```
> $ pseudo
> # chroot rootfs
> # export PATH=/sbin:/bin
> # ldconfig
> Can't create temporary cache file /etc/ld.so.cache~
> ```
> 
> Idea was then to rename `ldconfig` to `ldconfig_` and create a
> symbolic link to `/bin/true` to mimic successful execution of
> `ldconfig`.
> 
> ```
> $ sudo mv rootfs/sbin/ldconfig rootfs/sbin/ldconfig_
> $ sudo ln -s /bin/true rootfs/sbin/ldconfig
> ```
> 
> Here, behavior becomes very odd:
> 
> ```
> $ pseudo
> # chroot rootfs
> # export PATH=/sbin:/bin
> # ldconfig
> /bin/sh: 16: ldconfig: not found
> ```
> Although it is in path...
> ```
> # /sbin/ldconfig
> /sbin/ldconfig: Can't create temporary cache file /etc/ld.so.cache~:
> Permission denied
> ```
> 
> So this is not our symbolic link but the real ldconfig from the host
> 
> ```
> # cd /sbin
> # ./ldconfig
> #
> ```
> 
> This works and returns `true`.
> 
> ```
> # cd /
> # /sbin/ldconfig
> /sbin/ldconfig: Can't create temporary cache file /etc/ld.so.cache~:
> Permission denied
> # sbin/ldconfig
> #
> ```
> 
> So a relative path works, but an absolute path does not.
> 
> Even more funny is:
> 
> ```
> # exec sh
> # exit
> $
> ```
> 
> The first `exec` replaces the current shell with `sh` from *OUTSIDE*
> of the `chroot`. The `exit` then exits the `pseudo` environment
> instead of the `chroot` environment.
> 
> # Summary #
> 
> * operations with `chroot inside pseudo` are completely messed-up.
> * `ldconfig` will never work with `pseudo` since it is static.
> 
> But if packets are not configured, initramfs is not generated, etc...
> 
> # Ideas #
> 
> * Yocto Morty uses pseudo with own patches, which may solve some or
> all issues
> * Do not use multistrap but another tool
> * Stracing / Kernel tracing to analyze problem more deeply
> 
> 
> Kind regards,
> Andreas
> 

Re: Integration of Pseudo into Isar

[Claudius Heine]

On 08/02/2017 03:24 PM, [ext] Andreas Reichel wrote:
> # Integrating pseudo into isar
> 
> Idea was to exchange `sudo` by `pseudo`. The function of `pseudo` is to
> intercept system calls and file accesses by preloading a library. All
> such operations are recorded in a database. For this to work, a
> `PSEUDO_PREFIX` variable must be seet, which is `/` if `pseudo` is
> installed to the default location.
> 
> # Given test configuration #
> 
> - Docker container based on debian 9
> - `multistrap` from Siemens Debian repository
> - `pseudo` from Siemens Debian repository
> 
> Using the following multistrap configuration named `simple-config`:
> 
> ```
> [General]
> unpack=true
> bootstrap=Debian
> aptsources=Debian
> noauth=true
> 
> [Debian]
> packages=
> source=http://ftp.de.debian.org/debian
> suite=stretch
> ```
> 
> Inside chroot (which is inside pseudo):
> 
> ```
> # mkdir rootfs
> # multistrap -f simple-config -d rootfs
> ```
> 
> # Results #
> 
> * Error during package configuration. (Cannot write to `/etc/ld.so.cache~`)
> 
>    This error can be tracked down to `ldconfig`.
>    It turned out that `ldconfig` is linked *statically*. Which means, its
>    file accesses cannot be intercepted by LDPRELOAD, which is only for
>    dynamically linked binaries. Thus, wether being in a pseudo chroot or
>    not, `ldconfig` will always access `/etc/ld.so.cache~` on the host,
>    which fails.
>    This is *NOT* a question of the Debian version and not a bug in `dpkg
>    --configure -a`, which calls `ldconfig` internally.
> 
> * Extremely odd behaviour within `chroot` within `pseudo`:
> 
> ```
> $ pseudo
> # chroot rootfs
> # export PATH=/sbin:/bin
> # ldconfig
> Can't create temporary cache file /etc/ld.so.cache~
> ```
> 
> Idea was then to rename `ldconfig` to `ldconfig_` and create a symbolic
> link to `/bin/true` to mimic successful execution of `ldconfig`.
> 
> ```
> $ sudo mv rootfs/sbin/ldconfig rootfs/sbin/ldconfig_
> $ sudo ln -s /bin/true rootfs/sbin/ldconfig
> ```
> 
> Here, behavior becomes very odd:
> 
> ```
> $ pseudo
> # chroot rootfs
> # export PATH=/sbin:/bin
> # ldconfig
> /bin/sh: 16: ldconfig: not found
> ```
> Although it is in path...
> ```
> # /sbin/ldconfig
> /sbin/ldconfig: Can't create temporary cache file /etc/ld.so.cache~:
> Permission denied
> ```

I also tried using pseudo (from debian sid), with the same success:

$ pseudo -V
pseudo version 1.8.1
$ LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/pseudo/ 
PSEUDO_LOCALSTATEDIR=/tmp/pseudo pseudo -P / -d
$ LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/pseudo/ 
PSEUDO_LOCALSTATEDIR=/tmp/pseudo pseudo -P / -R /mnt/ssd/nfs/rootfs/1 -v 
/bin/bash
# rm /sbin/ldconfig
# ls /sbin/ldconfig
ls: cannot access '/sbin/ldconfig': No such file or directory
# /sbin/ldconfig
/sbin/ldconfig: Can't create temporary cache file /etc/ld.so.cache~: 
Permission denied

Also other commands don't really know if they are running in a chroot or 
outside of it:

# apt update
Ign:1 http://cdn-fastly.deb.debian.org/debian stretch InRelease
Get:2 http://cdn-fastly.deb.debian.org/debian stretch Release [118 kB]
Get:3 http://cdn-fastly.deb.debian.org/debian stretch Release.gpg [2,373 B]
Get:4 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 
Packages [7,095 kB]
Get:5 http://cdn-fastly.deb.debian.org/debian stretch/main 
Translation-en [5,393 kB]
Get:6 http://cdn-fastly.deb.debian.org/debian stretch/contrib amd64 
Packages [50.9 kB]
Get:7 http://cdn-fastly.deb.debian.org/debian stretch/contrib 
Translation-en [45.9 kB]
Get:8 http://cdn-fastly.deb.debian.org/debian stretch/non-free amd64 
Packages [77.9 kB]
Get:9 http://cdn-fastly.deb.debian.org/debian stretch/non-free 
Translation-en [79.2 kB]
Fetched 12.9 MB in 3s (3,524 kB/s)
Reading package lists... Error!
E: Could not open file /mnt/ssd/nfs/rootfs/1/var/lib/dpkg/status - open 
(2: No such file or directory)
E: Problem opening /mnt/ssd/nfs/rootfs/1/var/lib/dpkg/status
E: The package lists or status file could not be parsed or opened.

So with this fail, what other options have we?

- Fixing pseudo and/or reporting all bugs?
- Using something else?
   - Fakeroot
     Only exists within the debian project anymore.
     Not sure which is the most current one:
     https://anonscm.debian.org/cgit/fakeroot/fakeroot.git
     http://packages.debian.org/fakeroot
   - Fakeroot-ng
     dead?
     http://fakeroot-ng.lingnu.com/
   - proot
     Last commit over a year ago, but ?they? are writing currently a
     proot-rs port in rust.
     https://proot-me.github.io

Something else?

Cheers,
Claudius

Re: Integration of Pseudo into Isar

[Henning Schild]

Another thing that might be important is which steps actually fail with
pseudo. Currently we use sudo in several places.

1 multistrap
2 configure.sh
3 ext4fs
4 populate

And maybe more, i do not know by heart.

1 needs to work for sure
2 can be merged into 1 when all configuration is moved into debian
packages
3 will be replaced by wic if you choose to use wic
4 will move into 1 if my second patchq gets merged

But i am guessing 1 already causes problems. Maybe in OE the magic that
makes pseudo work is in the combination of pseudo and wic. In that case
the magic needs to be understood and applied to 1 or maybe 1 can be
done inside wic?

Henning

Am Wed, 2 Aug 2017 15:24:14 +0200
schrieb "Andreas Reichel" <andreas.reichel.ext@siemens.com>:

> # Integrating pseudo into isar
> 
> Idea was to exchange `sudo` by `pseudo`. The function of `pseudo` is
> to intercept system calls and file accesses by preloading a library.
> All such operations are recorded in a database. For this to work, a
> `PSEUDO_PREFIX` variable must be seet, which is `/` if `pseudo` is
> installed to the default location.
> 
> # Given test configuration #
> 
> - Docker container based on debian 9
> - `multistrap` from Siemens Debian repository
> - `pseudo` from Siemens Debian repository
> 
> Using the following multistrap configuration named `simple-config`:
> 
> ```
> [General]
> unpack=true
> bootstrap=Debian
> aptsources=Debian
> noauth=true
> 
> [Debian]
> packages=
> source=http://ftp.de.debian.org/debian
> suite=stretch
> ```
> 
> Inside chroot (which is inside pseudo):
> 
> ```
> # mkdir rootfs
> # multistrap -f simple-config -d rootfs
> ```
> 
> # Results #
> 
> * Error during package configuration. (Cannot write to
> `/etc/ld.so.cache~`)
> 
>   This error can be tracked down to `ldconfig`.
>   It turned out that `ldconfig` is linked *statically*. Which means,
> its file accesses cannot be intercepted by LDPRELOAD, which is only
> for dynamically linked binaries. Thus, wether being in a pseudo
> chroot or not, `ldconfig` will always access `/etc/ld.so.cache~` on
> the host, which fails.
>   This is *NOT* a question of the Debian version and not a bug in
> `dpkg --configure -a`, which calls `ldconfig` internally.
> 
> * Extremely odd behaviour within `chroot` within `pseudo`:
> 
> ```
> $ pseudo
> # chroot rootfs
> # export PATH=/sbin:/bin
> # ldconfig
> Can't create temporary cache file /etc/ld.so.cache~
> ```
> 
> Idea was then to rename `ldconfig` to `ldconfig_` and create a
> symbolic link to `/bin/true` to mimic successful execution of
> `ldconfig`.
> 
> ```
> $ sudo mv rootfs/sbin/ldconfig rootfs/sbin/ldconfig_
> $ sudo ln -s /bin/true rootfs/sbin/ldconfig
> ```
> 
> Here, behavior becomes very odd:
> 
> ```
> $ pseudo
> # chroot rootfs
> # export PATH=/sbin:/bin
> # ldconfig
> /bin/sh: 16: ldconfig: not found
> ```
> Although it is in path...
> ```
> # /sbin/ldconfig
> /sbin/ldconfig: Can't create temporary cache file /etc/ld.so.cache~:
> Permission denied
> ```
> 
> So this is not our symbolic link but the real ldconfig from the host
> 
> ```
> # cd /sbin
> # ./ldconfig
> #
> ```
> 
> This works and returns `true`.
> 
> ```
> # cd /
> # /sbin/ldconfig
> /sbin/ldconfig: Can't create temporary cache file /etc/ld.so.cache~:
> Permission denied
> # sbin/ldconfig
> #
> ```
> 
> So a relative path works, but an absolute path does not.
> 
> Even more funny is:
> 
> ```
> # exec sh
> # exit
> $
> ```
> 
> The first `exec` replaces the current shell with `sh` from *OUTSIDE*
> of the `chroot`. The `exit` then exits the `pseudo` environment
> instead of the `chroot` environment.
> 
> # Summary #
> 
> * operations with `chroot inside pseudo` are completely messed-up.
> * `ldconfig` will never work with `pseudo` since it is static.
> 
> But if packets are not configured, initramfs is not generated, etc...
> 
> # Ideas #
> 
> * Yocto Morty uses pseudo with own patches, which may solve some or
> all issues
> * Do not use multistrap but another tool
> * Stracing / Kernel tracing to analyze problem more deeply
> 
> 
> Kind regards,
> Andreas
> 

Re: Integration of Pseudo into Isar

[Andreas Reichel]

On Thu, Aug 03, 2017 at 09:50:11AM +0200, Henning Schild wrote:
> Another thing that might be important is which steps actually fail with
> pseudo. Currently we use sudo in several places.
> 
> 1 multistrap
> 2 configure.sh
> 3 ext4fs
> 4 populate
> 
> And maybe more, i do not know by heart.
> 
> 1 needs to work for sure

As we just discussed, step 1 fails already since multistrap calls
dpkg --configure and this calls ldconfig.

Since ldconfig is static, this will never work.
https://superuser.com/questions/943221/is-running-ldconfig-inside-fakeroot-possible

(This is about fakeroot, but it also uses LD_PRELOAD)

As we discussed
* I will analyze how bitbake and wic use pseudo
* Have a closer look if this can be solved by user namespaces

Andreas


-- 
Andreas Reichel
Dipl.-Phys. (Univ.)
Software Consultant

Andreas.Reichel@tngtech.com, +49-174-3180074
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterfoehring
Geschaeftsfuehrer: Henrik Klagges, Dr. Robert Dahlke, Gerhard Mueller
Sitz: Unterfoehring * Amtsgericht Muenchen * HRB 135082

RE: Integration of Pseudo into Isar

[Schild, Henning]

[-- Attachment #1: Type: text/plain, Size: 1681 bytes --]

Before diving too deep into namespaces we also discussed contacting the people from oe and pseudo and checking whether the problems come from ld so which you might patch with a custom library in ldpreload.

Von meinem Android-Telefon mit TouchDown (www.symantec.com) gesendet

-----Original Message-----
From: Andreas Reichel [andreas.reichel.ext@siemens.com]
Received: Donnerstag, 03 Aug. 2017, 11:11
To: Schild, Henning (CT RDA ITP SES-DE) [henning.schild@siemens.com]
CC: isar-users@googlegroups.com [isar-users@googlegroups.com]
Subject: Re: Integration of Pseudo into Isar

On Thu, Aug 03, 2017 at 09:50:11AM +0200, Henning Schild wrote:
> Another thing that might be important is which steps actually fail with
> pseudo. Currently we use sudo in several places.
>
> 1 multistrap
> 2 configure.sh
> 3 ext4fs
> 4 populate
>
> And maybe more, i do not know by heart.
>
> 1 needs to work for sure

As we just discussed, step 1 fails already since multistrap calls
dpkg --configure and this calls ldconfig.

Since ldconfig is static, this will never work.
https://superuser.com/questions/943221/is-running-ldconfig-inside-fakeroot-possible

(This is about fakeroot, but it also uses LD_PRELOAD)

As we discussed
* I will analyze how bitbake and wic use pseudo
* Have a closer look if this can be solved by user namespaces

Andreas


--
Andreas Reichel
Dipl.-Phys. (Univ.)
Software Consultant

Andreas.Reichel@tngtech.com, +49-174-3180074
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterfoehring
Geschaeftsfuehrer: Henrik Klagges, Dr. Robert Dahlke, Gerhard Mueller
Sitz: Unterfoehring * Amtsgericht Muenchen * HRB 135082


[-- Attachment #2: Type: text/html, Size: 2704 bytes --]