Re: Isar build tree structure

[Claudius Heine <claudius.heine.ext@siemens.com>]

Hi,

On 09/05/2017 01:09 PM, [ext] Claudius Heine wrote:
> Hi,
> 
> On 09/05/2017 11:35 AM, Alexander Smirnov wrote:
>>
>>
>> On 09/05/2017 12:28 PM, Claudius Heine wrote:
>>> On 09/05/2017 11:13 AM, Alexander Smirnov wrote:
>>>>
>>>>
>>>> On 09/05/2017 12:06 PM, Claudius Heine wrote:
>>>>> Hi,
>>>>>
>>>>> On 09/05/2017 10:21 AM, Alexander Smirnov wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> I'd like to discuss build folders tree to somehow chose more 
>>>>>> suitable approach.
>>>>>>
>>>>>> 1. How it's done at the moment.
>>>>>>
>>>>>> The folder tree contains the following main paths:
>>>>>>
>>>>>>   - tmp/work/${PP}
>>>>>>
>>>>>> It's a $WORKDIR for each recipe, here is stored bibtbake logs and 
>>>>>> commands, and data for packages that doesn't require buildchroot 
>>>>>> (for example, buildchroot and isar-image-base).
>>>>>>
>>>>>>   - tmp/work/buildchroot/${DISTRO}-${ARCH}/rootfs/home/build/${PN}
>>>>>>
>>>>>> This is the $BUILDROOT for packages, that require compilation in 
>>>>>> chroot. It is used as destination folder for unpack task.
>>>>>>
>>>>>> Benefits:
>>>>>>
>>>>>>   - Maximal common OE-like folder structure for all kind of 
>>>>>> recipes. You can find the information about specific package in 
>>>>>> 'tmp/work/${PP}'. And it doesn't matter if this package uses dpkg 
>>>>>> class, or not.
>>>>>>
>>>>>>   - After buildchroot cleanup, the build information is not loosed.
>>>>>>
>>>>>>   - For multiple buildchroots in project, there will be single 
>>>>>> place for package meta information.
>>>>>>
>>>>>> 2. Remove $BUILDROOT.
>>>>>>
>>>>>> This would mean that $WORKDIR location may vary depending on 
>>>>>> recipe type. So the recipes data will be distributed across 
>>>>>> buildchroots and tmp/work dir.
>>>>>>
>>>>>> Benefits:
>>>>>>
>>>>>>   - Each recipe will use single folder for everything.
>>>>>>
>>>>>>
>>>>>>
>>>>>> So... Ideas and opinions? :-)
>>>>>>
>>>>>
>>>>> Why not both benefits? Using bind mount to put the workdir of 
>>>>> recipes into the buildchroot?
>>>>
>>>> Mount is very difficult to manage. For example if something failed 
>>>> during build, it's hard to check in next build what is already 
>>>> mounted and what is not yet.
>>>>
>>>> But your idea is good. Could we use symlinks instead?
>>>
>>> If you are in a chroot environment the paths of the symlinks are 
>>> wrong since they reference a different root directory.
>>
>> Yeah, you are right... So probably mount could be an option:
>>
>> = do_unpack =
>> 1. unpack to WORKDIR (tmp/work/${PP})
>>
>> = do_build =
>> 1. mount respective folder in buildchroot
>> 2. build
>> 3. unmount respective folder in buildchroot
>>
>> This approach will be quite good compatible with OE.
> 
> If we go with proot then all this mounting could be done without 
> requiring root or this build step. I could be done like this:
> 
>      $ proot -S ${BUILDROOT} -b ${WORKDIR}:${CHROOT_BUILDDIR} build.sh
> 
> One problem with proot is this:
> 
>      $ mkdir test
>      $ cd test
>      $ proot -0
>      # id
>      uid=0(root) gid=0(root) groups=0(root)
>      # touch testfile
>      # ls -al
>      total 8
>      drwxr-xr-x  2 root root 4096 Sep  5 13:02 .
>      drwxrwxrwt 18 root root 4096 Sep  5 13:02 ..
>      -rw-r--r--  1 root root    0 Sep  5 13:02 testfile
>      # chown 123:123 testfile
>      # ls -al
>      total 8
>      drwxr-xr-x  2 root root 4096 Sep  5 13:02 .
>      drwxrwxrwt 18 root root 4096 Sep  5 13:02 ..
>      -rw-r--r--  1 root root    0 Sep  5 13:02 testfile
>      # exit
>      $ ls -al
>      total 8
>      drwxr-xr-x  2 user user 4096 Sep  5 13:02 .
>      drwxrwxrwt 18 root root 4096 Sep  5 13:02 ..
>      -rw-r--r--  1 user user    0 Sep  5 13:02 testfile
> 
> It maps just everything from the current user to root and back. So 
> multiple users aren't supported here. But maybe this can be patched.

I will go a bit deeper for this proposal.

The roadmap of proot [1] shows that this point is on their TODO list:

     * Emulate chown, chmod, and mknod when -0 (fake_id0) is enabled.

We have to communicate with the proot dev, maybe they already have an 
idea how to solve it themselve, but here is my suggestion:

Add an maybe two additional parameter to proot, maybe '--file-node-map 
map.txt' and maybe '--file-node-map-prefix /sysroot'.
The 'map.txt' file could have this structure:

     /path/to/file f 12 13 0664
     /path d 12 13 0755
     /dev/console c 5 1 0 0 0600

Were the first column is the path with the 'file-node-map-base' as the 
prefix added internally, the next the type 'f' for file, 'd' for 
directory and 'c' for character device. If its a device the next two 
columns are major and minor. the next 3 columns for all node types is 
the uid gid and permissions.

This file is loaded by proot and then it emulates those files. On chown, 
chmod and mknod the internal struct from proot is changed. When proot 
exits it saves the internal struct back to 'map.txt'.

Since all the files seems to have the correct permission from inside the 
proot, calling tar from within would create the right permissions inside 
the resulting tarball.

 From a correct tarball wic should be able to create the correct image 
using their tools (pseudo, ...).

This way we would not depend on the strangeness of pseudo for our tasks.

Cheers,
Claudius

[1] 
https://github.com/proot-me/PRoot/blob/afe225dc804e7926e588e8af22fbc5433b773e2d/doc/proot/roadmap.txt


-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de