PRoot Isar summary

PRoot Isar summary

[Alexander Smirnov]

Hello everybody,

I've tried to completely switch Isar to PRoot, so here are the problems 
I've faced with:

1. PRoot doesn't work with UID/GID, all the files in PRoot are owned by 
root. The command 'chown' doesn't have any effect.

2. Some system commands are failed in PRoot: passwd, chpasswd. I see 
message: System error, no other clues (but for Wheezy these commands work).

3. mkfs.ext4 doesn't work under proot, lots of files are dropped in 
resulting image.

So, summary:
============

1. PRoot could be an intermediate option for:
  + Buildchroot creation.
  + Packages building.
  - Drawback: works slowly.

2. For image generation the other tool should be considered.

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov

Re: PRoot Isar summary

[Jan Kiszka]

On 2017-11-09 10:57, Alexander Smirnov wrote:
> Hello everybody,
> 
> I've tried to completely switch Isar to PRoot, so here are the problems
> I've faced with:
> 
> 1. PRoot doesn't work with UID/GID, all the files in PRoot are owned by
> root. The command 'chown' doesn't have any effect.
> 
> 2. Some system commands are failed in PRoot: passwd, chpasswd. I see
> message: System error, no other clues (but for Wheezy these commands work).
> 
> 3. mkfs.ext4 doesn't work under proot, lots of files are dropped in
> resulting image.
> 
> So, summary:
> ============
> 
> 1. PRoot could be an intermediate option for:
>  + Buildchroot creation.
>  + Packages building.
>  - Drawback: works slowly.

Aren't issues 1 and 2 from above affecting these use cases as well?

> 
> 2. For image generation the other tool should be considered.
> 

What is plan B now? Plan C remains falling back to VM builds, I suppose.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: PRoot Isar summary

[Alexander Smirnov]

Hi,

On 11/10/2017 09:59 PM, Jan Kiszka wrote:
> On 2017-11-09 10:57, Alexander Smirnov wrote:
>> Hello everybody,
>>
>> I've tried to completely switch Isar to PRoot, so here are the problems
>> I've faced with:
>>
>> 1. PRoot doesn't work with UID/GID, all the files in PRoot are owned by
>> root. The command 'chown' doesn't have any effect.
>>
>> 2. Some system commands are failed in PRoot: passwd, chpasswd. I see
>> message: System error, no other clues (but for Wheezy these commands work).
>>
>> 3. mkfs.ext4 doesn't work under proot, lots of files are dropped in
>> resulting image.
>>
>> So, summary:
>> ============
>>
>> 1. PRoot could be an intermediate option for:
>>   + Buildchroot creation.
>>   + Packages building.
>>   - Drawback: works slowly.
> 
> Aren't issues 1 and 2 from above affecting these use cases as well?
> 

For now I don't have any facts about problems with buildchroot, but my 
test includes only 'hello' and 'example-raw' applications.

  - Regarding UID/GID, what I've seen for now, these manipulations are 
done in postinst scripts.
  - Passwd/chpasswd commands are also used in postinst scripts (for 
example initrd package), there is no need to have passwords in 
buildchroot because we are working under root.

So, roughly speaking, buildchroot is only needed to compile and pack the 
binary package, what doesn't require multi-UID/GID and passwords support.

But for sure, it needs to build much more real packages to have more 
precise statistics. :-(

So I've created dedicated branch 'asmirnov/proot' for possible 
experiments in future.

>>
>> 2. For image generation the other tool should be considered.
>>
> 
> What is plan B now? Plan C remains falling back to VM builds, I suppose.

So there are 2 options remain for evaluation:
  - fakeroot
  - pseudo

I'd like to evaluate these tools for the features, that are uncovered now:

  - rootfs with UID/GID support: in general PRoot is able to generate 
multistrap rootfs with just *upacked* Debian packages, all the problems 
occur when I try to run 'dpkg-configure -a' inside this rootfs.

  - ext2fs image generation (AFAIK this already is supported by Yocto, 
but unfortunately I don't know too much, I need to take a look first).

 From this evaluation I'd like to get two points:

1. Could we somehow implement quick PoC to drop 'sudo' for Isar. This 
PoC could be based on several tools in parallel.

2. If the item above is possible - then choose one dedicated tool and 
try to adapt it for our needs.

Alex

Re: PRoot Isar summary

[Benedikt Niedermayr]

[-- Attachment #1: Type: text/plain, Size: 4187 bytes --]

Am 10.11.2017 um 20:42 schrieb Alexander Smirnov:
> Hi,
>
> On 11/10/2017 09:59 PM, Jan Kiszka wrote:
>> On 2017-11-09 10:57, Alexander Smirnov wrote:
>>> Hello everybody,
>>>
>>> I've tried to completely switch Isar to PRoot, so here are the problems
>>> I've faced with:
>>>
>>> 1. PRoot doesn't work with UID/GID, all the files in PRoot are owned by
>>> root. The command 'chown' doesn't have any effect.
>>>
>>> 2. Some system commands are failed in PRoot: passwd, chpasswd. I see
>>> message: System error, no other clues (but for Wheezy these commands 
>>> work).
>>>
>>> 3. mkfs.ext4 doesn't work under proot, lots of files are dropped in
>>> resulting image.
>>>
>>> So, summary:
>>> ============
>>>
>>> 1. PRoot could be an intermediate option for:
>>>   + Buildchroot creation.
>>>   + Packages building.
>>>   - Drawback: works slowly.
>>
>> Aren't issues 1 and 2 from above affecting these use cases as well?
>>
>
> For now I don't have any facts about problems with buildchroot, but my 
> test includes only 'hello' and 'example-raw' applications.
>
>  - Regarding UID/GID, what I've seen for now, these manipulations are 
> done in postinst scripts.
>  - Passwd/chpasswd commands are also used in postinst scripts (for 
> example initrd package), there is no need to have passwords in 
> buildchroot because we are working under root.
>
> So, roughly speaking, buildchroot is only needed to compile and pack 
> the binary package, what doesn't require multi-UID/GID and passwords 
> support.
>
> But for sure, it needs to build much more real packages to have more 
> precise statistics. :-(
>
> So I've created dedicated branch 'asmirnov/proot' for possible 
> experiments in future.
>
>>>
>>> 2. For image generation the other tool should be considered.
>>>
>>
>> What is plan B now? Plan C remains falling back to VM builds, I suppose.
>
> So there are 2 options remain for evaluation:
>  - fakeroot
>  - pseudo
>
> I'd like to evaluate these tools for the features, that are uncovered 
> now:
>
>  - rootfs with UID/GID support: in general PRoot is able to generate 
> multistrap rootfs with just *upacked* Debian packages, all the 
> problems occur when I try to run 'dpkg-configure -a' inside this rootfs.
>
>  - ext2fs image generation (AFAIK this already is supported by Yocto, 
> but unfortunately I don't know too much, I need to take a look first).
>
> From this evaluation I'd like to get two points:
>
> 1. Could we somehow implement quick PoC to drop 'sudo' for Isar. This 
> PoC could be based on several tools in parallel.
>
> 2. If the item above is possible - then choose one dedicated tool and 
> try to adapt it for our needs.
>
> Alex
>
Ok bad news, but I faced the same problems, when trying to use one of 
these tools. Each tool has own drawbacks and benefits, but no tool 
combines multiple drawbacks to fit our needs.

I don't want to bother you, but why can't we try to focus on running 
builds with docker support?

Create a wrapper around "bitbake" which first performs a docker 
container setup and then runs the bitbake build.

Using such wrapper can make the docker thing almost transparent.


I know there are some problems getting a docker container secure, but 
maybe a focus on trying to get a docker based isar build secure, is 
easier to reach than the our current approaches?

It is possible to drop some capabilities for docker in order to make it 
more secure (e.g. don't allow to create dev files).

A mount command is also not required since, new versions of mkfs have 
the "-d" option included (specifiy a directory, which copied into the 
filesystem image). So no sys_admin capabilities would be needed.

It is also possible to customize other things within the container to 
make it more secure:

- Add only required commands to sudoers file.

- Modify permissions to files/directories.

- Think about which commands within isar really need root privileges, 
and drop those.


I think, if somebody seriously wants to exploit the container, he will 
also reach that with a non-root based build.

The attachment contains a (very basic and rudimentary) approach of 
running docker based isar builds.


Benedikt








[-- Attachment #2: isar --]
[-- Type: text/plain, Size: 8130 bytes --]

#!/usr/bin/env python3
#
# Copyright (C) 2017 Mixed Mode GmbH
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License version 2 as
# published by the Free Software Foundation.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
# Main ISAR executable. Bitbake itself cannot be executed, instead call this wrapper for
# running builds.
#

import argparse
import sys
import docker
import logging
import traceback
import os
from threading import Thread

# Remove capabilities from container increasing
# host system savety.
CAP_DROP = ['MKNOD']


def get_logger():
    logger = logging.getLogger('isar')
    formatter = logging.Formatter('%(asctime)s-%(name)s-%(levelname)s: %(message)s')

    logger.setLevel(logging.DEBUG)

    shandler = logging.StreamHandler()
    shandler.setLevel(logging.DEBUG)
    shandler.setFormatter(formatter)

    logger.addHandler(shandler)

    return logger

# Add function tracing
def addlog(func):
    def wrapper(*args, **kwargs):
        log.debug('Starting %s """%s"""' % (func.__name__, func.__doc__))
        func(*args, **kwargs)
        log.debug('Finished %s """%s"""' % (func.__name__, func.__doc__))
    return wrapper


log = get_logger()

def docker_log(stream_handler):
    """ Threaded handler for receiving current container and also image build logs.

        Print status lines in place instead of writing those logs into next newlines:

        {"status":"Extracting","progressDetail":{"current":40370176,"total":45129088},"progress":"[============================================\u003e      ] 40.37 MB/45.13 MB","id":"3e17c6eae66c"}
        {"status":"Extracting","progressDetail":{"current":40828928,"total":45129088},"progress":"[=============================================\u003e     ] 40.83 MB/45.13 MB","id":"3e17c6eae66c"}
        {"status":"Extracting","progressDetail":{"current":41287680,"total":45129088},"progress":"[=============================================\u003e     ] 41.29 MB/45.13 MB","id":"3e17c6eae66c"}
        {"status":"Extracting","progressDetail":{"current":41746432,"total":45129088},"progress":"[==============================================\u003e    ] 41.75 MB/45.13 MB","id":"3e17c6eae66c"}
        {"status":"Extracting","progressDetail":{"current":42205184,"total":45129088},"progress":"[==============================================\u003e    ] 42.21 MB/45.13 MB","id":"3e17c6eae66c"}
        {"status":"Extracting","progressDetail":{"current":42663936,"total":45129088},"progress":"[===============================================\u003e   ] 42.66 MB/45.13 MB","id":"3e17c6eae66c"}
        {"status":"Extracting","progressDetail":{"current":43122688,"total":45129088},"progress":"[===============================================\u003e   ] 43.12 MB/45.13 MB","id":"3e17c6eae66c"}
        {"status":"Extracting","progressDetail":{"current":43581440,"total":45129088},"progress":"[================================================\u003e  ] 43.58 MB/45.13 MB","id":"3e17c6eae66c"}
        {"status":"Extracting","progressDetail":{"current":44040192,"total":45129088},"progress":"[================================================\u003e  ] 44.04 MB/45.13 MB","id":"3e17c6eae66c"}

    """
    for i in stream_handler:

        if isinstance(i, dict):
            # Image build logs

            #
            # different keys possible (stream, status)
            #
            #i = i['stream']
            print(i.strip())
        else:
            # Container logs
            print(i.decode().strip())


class IsarDocker():

    def __init__(self, args):
        try:
            self.client = docker.from_env()
        except:
            traceback.print_exc()
            log.error('Cannot connect to the docker socket! Exciting now...')
            exit(2) # ENOENT

        self.builddir   = os.environ['BUILDDIR']
        self.bspdir     = os.path.realpath(os.path.join(self.builddir, '..'))
        self.dockerdir  = args.path

        self.cap_drop   = CAP_DROP
        self.dockerdir
        self.volume_binds = {
            self.bspdir : {
                                'bind' : self.bspdir,
                                'mode' : 'rw',
                          }
        }

        self.hostconfig = self.client.create_host_config(privileged=False, cap_drop=self.cap_drop, binds=self.volume_binds)


    @addlog
    def build(self):
        """ Build the isar docker image. """
        os.chdir(self.builddir)

        self.blogs = self.client.build(path=self.dockerdir, rm=True, tag='isar_image:nanopi', decode=False)
        thread = Thread(target=docker_log, args=(self.blogs, ))
        thread.start()
        thread.join()


    @addlog
    def _create(self, cmd):
        """ Create the container without starting it."""
        try:
            self.container = self.client.create_container(image='isar_image:nanopi', command=cmd, host_config=self.hostconfig)
        except:
            traceback.print_exc()
            log.error('Cannot create container! Exciting now...')
            exit(1)

    @addlog
    def run(self, cmd):
        """ Run a command in the container. """
        #
        # TODO: Use exec_create instead, so _create is not required anymore.
        #       BUT!!: Is it possible to drop capabilities then?
        #
        self._create(cmd)

        try:
            self.client.start(container=self.container.get('Id'))
            self.clogs = self.client.logs(container=self.container.get('Id'), stream=True)

            # Thread for receiving current container logs
            thread = Thread(target=docker_log, args=(self.clogs, ))
            thread.start()

            self.retcode = self.client.wait(container=self.container.get('Id'))
            thread.join()

            if self.retcode != 0:
                log.warning('Command return non zero status!')
        except:
            traceback.print_exc()
            log.error('Executing docker container! Exciting now...')
            exit(1)


def run_docker(args):
    log.debug('Running docker subcommand')
    dc = IsarDocker(args)

    if args.setup:
        dc.build()
    elif args.run:
        dc.run(args.run)

def run_bitbake(args):
    log.debug('Running bitbake subcommand')
    dc = IsarDocker(args)
    builddir = os.path.basename(dc.builddir)

    log.debug('Args: %s' % args.args)
    log.debug('Build directory: %s' % builddir)
    cmd = "bash -c 'cd %s; source setup-environment %s; bitbake %s'" % (dc.bspdir, builddir, args.args)

    log.debug(cmd)
    dc.run(cmd)



#
# Cli
#
parser = argparse.ArgumentParser(prog='isar')
parser.add_argument('--path', type=str, default='./docker', help='Path to the directory holding the dockerfile. Defaults to $BUILDDIR/docker')
subparsers = parser.add_subparsers()

# create the parser for the docker command
docker_parser = subparsers.add_parser('docker', help='docker --help')
docker_parser.add_argument('--create', action='store_true', help='Creates the basic docker image where isar builds run.'
                                                                 'The dockerfile specified by --path is used.')
docker_parser.add_argument('--run', type=str, help='Run a command in the docker container.')
docker_parser.set_defaults(func=run_docker)


# create the parser for the bitbake command
bitbake_parser = subparsers.add_parser('bitbake', help='bitbake --help')
bitbake_parser.add_argument('--args', type=str, required=True, help='Arguments forwarded to bitbake command.'
                                        'The complete argument string will be appended to the bitbake command.')
bitbake_parser.set_defaults(func=run_bitbake)

args = parser.parse_args(sys.argv[1:])

args.func(args)

Re: PRoot Isar summary

[Claudius Heine]

[-- Attachment #1: Type: text/plain, Size: 6549 bytes --]

Hi Ben,

On Sat, 2017-11-11 at 18:22 +0100, 'Benedikt Niedermayr' via isar-users 
wrote:
> Am 10.11.2017 um 20:42 schrieb Alexander Smirnov:
> > Hi,
> > 
> > On 11/10/2017 09:59 PM, Jan Kiszka wrote:
> > > On 2017-11-09 10:57, Alexander Smirnov wrote:
> > > > Hello everybody,
> > > > 
> > > > I've tried to completely switch Isar to PRoot, so here are the
> > > > problems
> > > > I've faced with:
> > > > 
> > > > 1. PRoot doesn't work with UID/GID, all the files in PRoot are
> > > > owned by
> > > > root. The command 'chown' doesn't have any effect.
> > > > 
> > > > 2. Some system commands are failed in PRoot: passwd, chpasswd.
> > > > I see
> > > > message: System error, no other clues (but for Wheezy these
> > > > commands 
> > > > work).
> > > > 
> > > > 3. mkfs.ext4 doesn't work under proot, lots of files are
> > > > dropped in
> > > > resulting image.
> > > > 
> > > > So, summary:
> > > > ============
> > > > 
> > > > 1. PRoot could be an intermediate option for:
> > > >   + Buildchroot creation.
> > > >   + Packages building.
> > > >   - Drawback: works slowly.
> > > 
> > > Aren't issues 1 and 2 from above affecting these use cases as
> > > well?
> > > 
> > 
> > For now I don't have any facts about problems with buildchroot, but
> > my 
> > test includes only 'hello' and 'example-raw' applications.
> > 
> >  - Regarding UID/GID, what I've seen for now, these manipulations
> > are 
> > done in postinst scripts.
> >  - Passwd/chpasswd commands are also used in postinst scripts (for 
> > example initrd package), there is no need to have passwords in 
> > buildchroot because we are working under root.
> > 
> > So, roughly speaking, buildchroot is only needed to compile and
> > pack 
> > the binary package, what doesn't require multi-UID/GID and
> > passwords 
> > support.
> > 
> > But for sure, it needs to build much more real packages to have
> > more 
> > precise statistics. :-(
> > 
> > So I've created dedicated branch 'asmirnov/proot' for possible 
> > experiments in future.
> > 
> > > > 
> > > > 2. For image generation the other tool should be considered.
> > > > 
> > > 
> > > What is plan B now? Plan C remains falling back to VM builds, I
> > > suppose.
> > 
> > So there are 2 options remain for evaluation:
> >  - fakeroot
> >  - pseudo
> > 
> > I'd like to evaluate these tools for the features, that are
> > uncovered 
> > now:
> > 
> >  - rootfs with UID/GID support: in general PRoot is able to
> > generate 
> > multistrap rootfs with just *upacked* Debian packages, all the 
> > problems occur when I try to run 'dpkg-configure -a' inside this
> > rootfs.
> > 
> >  - ext2fs image generation (AFAIK this already is supported by
> > Yocto, 
> > but unfortunately I don't know too much, I need to take a look
> > first).
> > 
> > From this evaluation I'd like to get two points:
> > 
> > 1. Could we somehow implement quick PoC to drop 'sudo' for Isar.
> > This 
> > PoC could be based on several tools in parallel.
> > 
> > 2. If the item above is possible - then choose one dedicated tool
> > and 
> > try to adapt it for our needs.
> > 
> > Alex
> > 
> 
> Ok bad news, but I faced the same problems, when trying to use one
> of 
> these tools. Each tool has own drawbacks and benefits, but no tool 
> combines multiple drawbacks to fit our needs.
> 
> I don't want to bother you, but why can't we try to focus on running 
> builds with docker support?
> 
> Create a wrapper around "bitbake" which first performs a docker 
> container setup and then runs the bitbake build.
> 
> Using such wrapper can make the docker thing almost transparent.

We already using such a wrapper, that does sort of what you want:
https://github.com/siemens/kas
It does not wrap around docker itself, but around bitbake and a ready
prepared docker container is available:
https://hub.docker.com/r/kasproject/kas-isar/

Patches and feature requests welcome.

> I know there are some problems getting a docker container secure,
> but 
> maybe a focus on trying to get a docker based isar build secure, is 
> easier to reach than the our current approaches?
> 
> It is possible to drop some capabilities for docker in order to make
> it 
> more secure (e.g. don't allow to create dev files).
> 
> A mount command is also not required since, new versions of mkfs
> have 
> the "-d" option included (specifiy a directory, which copied into
> the 
> filesystem image). So no sys_admin capabilities would be needed.
> 
> It is also possible to customize other things within the container
> to 
> make it more secure:
> 
> - Add only required commands to sudoers file.

Problem here are the additional layers that might add additional
programs that need to be run with root privileges. That is not
something that can be controlled easily.

> 
> - Modify permissions to files/directories.

Who cares if the container gets corrupted, of course we should mount
every 'source' directory into the container read-only.

> 
> - Think about which commands within isar really need root
> privileges, 
> and drop those.

Don't think that really helps, because there are so many programs that
might require root privileges, depending on the custom recipe.
Also if 'postinst' is run as root, you could put any command there that
is run as root user.

> 
> I think, if somebody seriously wants to exploit the container, he
> will 
> also reach that with a non-root based build.

If we are giving up on trying to remove the root dependency of the
build, then I would rather go with preparing a vagrant file instead of
a container. We would have better isolation there and don't really need
to care about capabilities.

Otherwise I would still suggest to patch proot to support the
additional syscalls (chown, chmod, mknod, ...), save those into a file
and allow proot to load this file via a parameter. Then we would have
something like pseudo, but better. Since pseudo and fakeroot does not
work with static binaries. (And this is what I thought Alex was looking
into.)

Cheers,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

            PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153
                              Keyserver: hkp://pool.sks-keyservers.net

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]