Isar build tree structure

Isar build tree structure

[Alexander Smirnov]

Hi all,

I'd like to discuss build folders tree to somehow chose more suitable 
approach.

1. How it's done at the moment.

The folder tree contains the following main paths:

  - tmp/work/${PP}

It's a $WORKDIR for each recipe, here is stored bibtbake logs and 
commands, and data for packages that doesn't require buildchroot (for 
example, buildchroot and isar-image-base).

  - tmp/work/buildchroot/${DISTRO}-${ARCH}/rootfs/home/build/${PN}

This is the $BUILDROOT for packages, that require compilation in chroot. 
It is used as destination folder for unpack task.

Benefits:

  - Maximal common OE-like folder structure for all kind of recipes. You 
can find the information about specific package in 'tmp/work/${PP}'. And 
it doesn't matter if this package uses dpkg class, or not.

  - After buildchroot cleanup, the build information is not loosed.

  - For multiple buildchroots in project, there will be single place for 
package meta information.

2. Remove $BUILDROOT.

This would mean that $WORKDIR location may vary depending on recipe 
type. So the recipes data will be distributed across buildchroots and 
tmp/work dir.

Benefits:

  - Each recipe will use single folder for everything.



So... Ideas and opinions? :-)

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov

Re: Isar build tree structure

[Claudius Heine]

Hi,

On 09/05/2017 10:21 AM, Alexander Smirnov wrote:
> Hi all,
> 
> I'd like to discuss build folders tree to somehow chose more suitable 
> approach.
> 
> 1. How it's done at the moment.
> 
> The folder tree contains the following main paths:
> 
>   - tmp/work/${PP}
> 
> It's a $WORKDIR for each recipe, here is stored bibtbake logs and 
> commands, and data for packages that doesn't require buildchroot (for 
> example, buildchroot and isar-image-base).
> 
>   - tmp/work/buildchroot/${DISTRO}-${ARCH}/rootfs/home/build/${PN}
> 
> This is the $BUILDROOT for packages, that require compilation in chroot. 
> It is used as destination folder for unpack task.
> 
> Benefits:
> 
>   - Maximal common OE-like folder structure for all kind of recipes. You 
> can find the information about specific package in 'tmp/work/${PP}'. And 
> it doesn't matter if this package uses dpkg class, or not.
> 
>   - After buildchroot cleanup, the build information is not loosed.
> 
>   - For multiple buildchroots in project, there will be single place for 
> package meta information.
> 
> 2. Remove $BUILDROOT.
> 
> This would mean that $WORKDIR location may vary depending on recipe 
> type. So the recipes data will be distributed across buildchroots and 
> tmp/work dir.
> 
> Benefits:
> 
>   - Each recipe will use single folder for everything.
> 
> 
> 
> So... Ideas and opinions? :-)
> 

Why not both benefits? Using bind mount to put the workdir of recipes 
into the buildchroot?

Maybe we should take a look at proot [1] and use this instead of chroot, 
because its more flexible, does not require privileges, etc.

[1] https://github.com/proot-me/PRoot/blob/master/doc/proot/manual.txt

Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: Isar build tree structure

[Alexander Smirnov]

On 09/05/2017 12:06 PM, Claudius Heine wrote:
> Hi,
> 
> On 09/05/2017 10:21 AM, Alexander Smirnov wrote:
>> Hi all,
>>
>> I'd like to discuss build folders tree to somehow chose more suitable 
>> approach.
>>
>> 1. How it's done at the moment.
>>
>> The folder tree contains the following main paths:
>>
>>   - tmp/work/${PP}
>>
>> It's a $WORKDIR for each recipe, here is stored bibtbake logs and 
>> commands, and data for packages that doesn't require buildchroot (for 
>> example, buildchroot and isar-image-base).
>>
>>   - tmp/work/buildchroot/${DISTRO}-${ARCH}/rootfs/home/build/${PN}
>>
>> This is the $BUILDROOT for packages, that require compilation in 
>> chroot. It is used as destination folder for unpack task.
>>
>> Benefits:
>>
>>   - Maximal common OE-like folder structure for all kind of recipes. 
>> You can find the information about specific package in 
>> 'tmp/work/${PP}'. And it doesn't matter if this package uses dpkg 
>> class, or not.
>>
>>   - After buildchroot cleanup, the build information is not loosed.
>>
>>   - For multiple buildchroots in project, there will be single place 
>> for package meta information.
>>
>> 2. Remove $BUILDROOT.
>>
>> This would mean that $WORKDIR location may vary depending on recipe 
>> type. So the recipes data will be distributed across buildchroots and 
>> tmp/work dir.
>>
>> Benefits:
>>
>>   - Each recipe will use single folder for everything.
>>
>>
>>
>> So... Ideas and opinions? :-)
>>
> 
> Why not both benefits? Using bind mount to put the workdir of recipes 
> into the buildchroot?

Mount is very difficult to manage. For example if something failed 
during build, it's hard to check in next build what is already mounted 
and what is not yet.

But your idea is good. Could we use symlinks instead?

> 
> Maybe we should take a look at proot [1] and use this instead of chroot, 
> because its more flexible, does not require privileges, etc.
> 
> [1] https://github.com/proot-me/PRoot/blob/master/doc/proot/manual.txt
> 

Do you know if it fully supports dpkg & Co?

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov

Re: Isar build tree structure

[Claudius Heine]

On 09/05/2017 11:13 AM, Alexander Smirnov wrote:
> 
> 
> On 09/05/2017 12:06 PM, Claudius Heine wrote:
>> Hi,
>>
>> On 09/05/2017 10:21 AM, Alexander Smirnov wrote:
>>> Hi all,
>>>
>>> I'd like to discuss build folders tree to somehow chose more suitable 
>>> approach.
>>>
>>> 1. How it's done at the moment.
>>>
>>> The folder tree contains the following main paths:
>>>
>>>   - tmp/work/${PP}
>>>
>>> It's a $WORKDIR for each recipe, here is stored bibtbake logs and 
>>> commands, and data for packages that doesn't require buildchroot (for 
>>> example, buildchroot and isar-image-base).
>>>
>>>   - tmp/work/buildchroot/${DISTRO}-${ARCH}/rootfs/home/build/${PN}
>>>
>>> This is the $BUILDROOT for packages, that require compilation in 
>>> chroot. It is used as destination folder for unpack task.
>>>
>>> Benefits:
>>>
>>>   - Maximal common OE-like folder structure for all kind of recipes. 
>>> You can find the information about specific package in 
>>> 'tmp/work/${PP}'. And it doesn't matter if this package uses dpkg 
>>> class, or not.
>>>
>>>   - After buildchroot cleanup, the build information is not loosed.
>>>
>>>   - For multiple buildchroots in project, there will be single place 
>>> for package meta information.
>>>
>>> 2. Remove $BUILDROOT.
>>>
>>> This would mean that $WORKDIR location may vary depending on recipe 
>>> type. So the recipes data will be distributed across buildchroots and 
>>> tmp/work dir.
>>>
>>> Benefits:
>>>
>>>   - Each recipe will use single folder for everything.
>>>
>>>
>>>
>>> So... Ideas and opinions? :-)
>>>
>>
>> Why not both benefits? Using bind mount to put the workdir of recipes 
>> into the buildchroot?
> 
> Mount is very difficult to manage. For example if something failed 
> during build, it's hard to check in next build what is already mounted 
> and what is not yet.
> 
> But your idea is good. Could we use symlinks instead?

If you are in a chroot environment the paths of the symlinks are wrong 
since they reference a different root directory.

> 
>>
>> Maybe we should take a look at proot [1] and use this instead of 
>> chroot, because its more flexible, does not require privileges, etc.
>>
>> [1] https://github.com/proot-me/PRoot/blob/master/doc/proot/manual.txt
>>
> 
> Do you know if it fully supports dpkg & Co?

I currently use proot in some parts of my isar fork already. And from my 
experience it works rather well. Since it uses the ptrace systemcall, 
static binaries work work as well.

Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: Isar build tree structure

[Alexander Smirnov]

On 09/05/2017 12:28 PM, Claudius Heine wrote:
> On 09/05/2017 11:13 AM, Alexander Smirnov wrote:
>>
>>
>> On 09/05/2017 12:06 PM, Claudius Heine wrote:
>>> Hi,
>>>
>>> On 09/05/2017 10:21 AM, Alexander Smirnov wrote:
>>>> Hi all,
>>>>
>>>> I'd like to discuss build folders tree to somehow chose more 
>>>> suitable approach.
>>>>
>>>> 1. How it's done at the moment.
>>>>
>>>> The folder tree contains the following main paths:
>>>>
>>>>   - tmp/work/${PP}
>>>>
>>>> It's a $WORKDIR for each recipe, here is stored bibtbake logs and 
>>>> commands, and data for packages that doesn't require buildchroot 
>>>> (for example, buildchroot and isar-image-base).
>>>>
>>>>   - tmp/work/buildchroot/${DISTRO}-${ARCH}/rootfs/home/build/${PN}
>>>>
>>>> This is the $BUILDROOT for packages, that require compilation in 
>>>> chroot. It is used as destination folder for unpack task.
>>>>
>>>> Benefits:
>>>>
>>>>   - Maximal common OE-like folder structure for all kind of recipes. 
>>>> You can find the information about specific package in 
>>>> 'tmp/work/${PP}'. And it doesn't matter if this package uses dpkg 
>>>> class, or not.
>>>>
>>>>   - After buildchroot cleanup, the build information is not loosed.
>>>>
>>>>   - For multiple buildchroots in project, there will be single place 
>>>> for package meta information.
>>>>
>>>> 2. Remove $BUILDROOT.
>>>>
>>>> This would mean that $WORKDIR location may vary depending on recipe 
>>>> type. So the recipes data will be distributed across buildchroots 
>>>> and tmp/work dir.
>>>>
>>>> Benefits:
>>>>
>>>>   - Each recipe will use single folder for everything.
>>>>
>>>>
>>>>
>>>> So... Ideas and opinions? :-)
>>>>
>>>
>>> Why not both benefits? Using bind mount to put the workdir of recipes 
>>> into the buildchroot?
>>
>> Mount is very difficult to manage. For example if something failed 
>> during build, it's hard to check in next build what is already mounted 
>> and what is not yet.
>>
>> But your idea is good. Could we use symlinks instead?
> 
> If you are in a chroot environment the paths of the symlinks are wrong 
> since they reference a different root directory.

Yeah, you are right... So probably mount could be an option:

= do_unpack =
1. unpack to WORKDIR (tmp/work/${PP})

= do_build =
1. mount respective folder in buildchroot
2. build
3. unmount respective folder in buildchroot

This approach will be quite good compatible with OE.

> 
>>
>>>
>>> Maybe we should take a look at proot [1] and use this instead of 
>>> chroot, because its more flexible, does not require privileges, etc.
>>>
>>> [1] https://github.com/proot-me/PRoot/blob/master/doc/proot/manual.txt
>>>
>>
>> Do you know if it fully supports dpkg & Co?
> 
> I currently use proot in some parts of my isar fork already. And from my 
> experience it works rather well. Since it uses the ptrace systemcall, 
> static binaries work work as well.
> 

Do you have patch for this, could you please send it as RFC?

> Claudius
> 

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov

Re: Isar build tree structure

[Claudius Heine]

Hi,

On 09/05/2017 11:35 AM, Alexander Smirnov wrote:
> 
> 
> On 09/05/2017 12:28 PM, Claudius Heine wrote:
>> On 09/05/2017 11:13 AM, Alexander Smirnov wrote:
>>>
>>>
>>> On 09/05/2017 12:06 PM, Claudius Heine wrote:
>>>> Hi,
>>>>
>>>> On 09/05/2017 10:21 AM, Alexander Smirnov wrote:
>>>>> Hi all,
>>>>>
>>>>> I'd like to discuss build folders tree to somehow chose more 
>>>>> suitable approach.
>>>>>
>>>>> 1. How it's done at the moment.
>>>>>
>>>>> The folder tree contains the following main paths:
>>>>>
>>>>>   - tmp/work/${PP}
>>>>>
>>>>> It's a $WORKDIR for each recipe, here is stored bibtbake logs and 
>>>>> commands, and data for packages that doesn't require buildchroot 
>>>>> (for example, buildchroot and isar-image-base).
>>>>>
>>>>>   - tmp/work/buildchroot/${DISTRO}-${ARCH}/rootfs/home/build/${PN}
>>>>>
>>>>> This is the $BUILDROOT for packages, that require compilation in 
>>>>> chroot. It is used as destination folder for unpack task.
>>>>>
>>>>> Benefits:
>>>>>
>>>>>   - Maximal common OE-like folder structure for all kind of 
>>>>> recipes. You can find the information about specific package in 
>>>>> 'tmp/work/${PP}'. And it doesn't matter if this package uses dpkg 
>>>>> class, or not.
>>>>>
>>>>>   - After buildchroot cleanup, the build information is not loosed.
>>>>>
>>>>>   - For multiple buildchroots in project, there will be single 
>>>>> place for package meta information.
>>>>>
>>>>> 2. Remove $BUILDROOT.
>>>>>
>>>>> This would mean that $WORKDIR location may vary depending on recipe 
>>>>> type. So the recipes data will be distributed across buildchroots 
>>>>> and tmp/work dir.
>>>>>
>>>>> Benefits:
>>>>>
>>>>>   - Each recipe will use single folder for everything.
>>>>>
>>>>>
>>>>>
>>>>> So... Ideas and opinions? :-)
>>>>>
>>>>
>>>> Why not both benefits? Using bind mount to put the workdir of 
>>>> recipes into the buildchroot?
>>>
>>> Mount is very difficult to manage. For example if something failed 
>>> during build, it's hard to check in next build what is already 
>>> mounted and what is not yet.
>>>
>>> But your idea is good. Could we use symlinks instead?
>>
>> If you are in a chroot environment the paths of the symlinks are wrong 
>> since they reference a different root directory.
> 
> Yeah, you are right... So probably mount could be an option:
> 
> = do_unpack =
> 1. unpack to WORKDIR (tmp/work/${PP})
> 
> = do_build =
> 1. mount respective folder in buildchroot
> 2. build
> 3. unmount respective folder in buildchroot
> 
> This approach will be quite good compatible with OE.

If we go with proot then all this mounting could be done without 
requiring root or this build step. I could be done like this:

     $ proot -S ${BUILDROOT} -b ${WORKDIR}:${CHROOT_BUILDDIR} build.sh

One problem with proot is this:

     $ mkdir test
     $ cd test
     $ proot -0
     # id
     uid=0(root) gid=0(root) groups=0(root)
     # touch testfile
     # ls -al
     total 8
     drwxr-xr-x  2 root root 4096 Sep  5 13:02 .
     drwxrwxrwt 18 root root 4096 Sep  5 13:02 ..
     -rw-r--r--  1 root root    0 Sep  5 13:02 testfile
     # chown 123:123 testfile
     # ls -al
     total 8
     drwxr-xr-x  2 root root 4096 Sep  5 13:02 .
     drwxrwxrwt 18 root root 4096 Sep  5 13:02 ..
     -rw-r--r--  1 root root    0 Sep  5 13:02 testfile
     # exit
     $ ls -al
     total 8
     drwxr-xr-x  2 user user 4096 Sep  5 13:02 .
     drwxrwxrwt 18 root root 4096 Sep  5 13:02 ..
     -rw-r--r--  1 user user    0 Sep  5 13:02 testfile

It maps just everything from the current user to root and back. So 
multiple users aren't supported here. But maybe this can be patched.

>>>>
>>>> Maybe we should take a look at proot [1] and use this instead of 
>>>> chroot, because its more flexible, does not require privileges, etc.
>>>>
>>>> [1] https://github.com/proot-me/PRoot/blob/master/doc/proot/manual.txt
>>>>
>>>
>>> Do you know if it fully supports dpkg & Co?
>>
>> I currently use proot in some parts of my isar fork already. And from 
>> my experience it works rather well. Since it uses the ptrace 
>> systemcall, static binaries work work as well.
>>
> 
> Do you have patch for this, could you please send it as RFC?

I don't have anything worth submitting (even as RFC) concerning proot. 
But AFAIK Johannes has converted more of isar to use proot. Maybe he can 
give some input for this.

Claudius


-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: Isar build tree structure

[Claudius Heine]

Hi,

On 09/05/2017 01:09 PM, [ext] Claudius Heine wrote:
> Hi,
> 
> On 09/05/2017 11:35 AM, Alexander Smirnov wrote:
>>
>>
>> On 09/05/2017 12:28 PM, Claudius Heine wrote:
>>> On 09/05/2017 11:13 AM, Alexander Smirnov wrote:
>>>>
>>>>
>>>> On 09/05/2017 12:06 PM, Claudius Heine wrote:
>>>>> Hi,
>>>>>
>>>>> On 09/05/2017 10:21 AM, Alexander Smirnov wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> I'd like to discuss build folders tree to somehow chose more 
>>>>>> suitable approach.
>>>>>>
>>>>>> 1. How it's done at the moment.
>>>>>>
>>>>>> The folder tree contains the following main paths:
>>>>>>
>>>>>>   - tmp/work/${PP}
>>>>>>
>>>>>> It's a $WORKDIR for each recipe, here is stored bibtbake logs and 
>>>>>> commands, and data for packages that doesn't require buildchroot 
>>>>>> (for example, buildchroot and isar-image-base).
>>>>>>
>>>>>>   - tmp/work/buildchroot/${DISTRO}-${ARCH}/rootfs/home/build/${PN}
>>>>>>
>>>>>> This is the $BUILDROOT for packages, that require compilation in 
>>>>>> chroot. It is used as destination folder for unpack task.
>>>>>>
>>>>>> Benefits:
>>>>>>
>>>>>>   - Maximal common OE-like folder structure for all kind of 
>>>>>> recipes. You can find the information about specific package in 
>>>>>> 'tmp/work/${PP}'. And it doesn't matter if this package uses dpkg 
>>>>>> class, or not.
>>>>>>
>>>>>>   - After buildchroot cleanup, the build information is not loosed.
>>>>>>
>>>>>>   - For multiple buildchroots in project, there will be single 
>>>>>> place for package meta information.
>>>>>>
>>>>>> 2. Remove $BUILDROOT.
>>>>>>
>>>>>> This would mean that $WORKDIR location may vary depending on 
>>>>>> recipe type. So the recipes data will be distributed across 
>>>>>> buildchroots and tmp/work dir.
>>>>>>
>>>>>> Benefits:
>>>>>>
>>>>>>   - Each recipe will use single folder for everything.
>>>>>>
>>>>>>
>>>>>>
>>>>>> So... Ideas and opinions? :-)
>>>>>>
>>>>>
>>>>> Why not both benefits? Using bind mount to put the workdir of 
>>>>> recipes into the buildchroot?
>>>>
>>>> Mount is very difficult to manage. For example if something failed 
>>>> during build, it's hard to check in next build what is already 
>>>> mounted and what is not yet.
>>>>
>>>> But your idea is good. Could we use symlinks instead?
>>>
>>> If you are in a chroot environment the paths of the symlinks are 
>>> wrong since they reference a different root directory.
>>
>> Yeah, you are right... So probably mount could be an option:
>>
>> = do_unpack =
>> 1. unpack to WORKDIR (tmp/work/${PP})
>>
>> = do_build =
>> 1. mount respective folder in buildchroot
>> 2. build
>> 3. unmount respective folder in buildchroot
>>
>> This approach will be quite good compatible with OE.
> 
> If we go with proot then all this mounting could be done without 
> requiring root or this build step. I could be done like this:
> 
>      $ proot -S ${BUILDROOT} -b ${WORKDIR}:${CHROOT_BUILDDIR} build.sh
> 
> One problem with proot is this:
> 
>      $ mkdir test
>      $ cd test
>      $ proot -0
>      # id
>      uid=0(root) gid=0(root) groups=0(root)
>      # touch testfile
>      # ls -al
>      total 8
>      drwxr-xr-x  2 root root 4096 Sep  5 13:02 .
>      drwxrwxrwt 18 root root 4096 Sep  5 13:02 ..
>      -rw-r--r--  1 root root    0 Sep  5 13:02 testfile
>      # chown 123:123 testfile
>      # ls -al
>      total 8
>      drwxr-xr-x  2 root root 4096 Sep  5 13:02 .
>      drwxrwxrwt 18 root root 4096 Sep  5 13:02 ..
>      -rw-r--r--  1 root root    0 Sep  5 13:02 testfile
>      # exit
>      $ ls -al
>      total 8
>      drwxr-xr-x  2 user user 4096 Sep  5 13:02 .
>      drwxrwxrwt 18 root root 4096 Sep  5 13:02 ..
>      -rw-r--r--  1 user user    0 Sep  5 13:02 testfile
> 
> It maps just everything from the current user to root and back. So 
> multiple users aren't supported here. But maybe this can be patched.

I will go a bit deeper for this proposal.

The roadmap of proot [1] shows that this point is on their TODO list:

     * Emulate chown, chmod, and mknod when -0 (fake_id0) is enabled.

We have to communicate with the proot dev, maybe they already have an 
idea how to solve it themselve, but here is my suggestion:

Add an maybe two additional parameter to proot, maybe '--file-node-map 
map.txt' and maybe '--file-node-map-prefix /sysroot'.
The 'map.txt' file could have this structure:

     /path/to/file f 12 13 0664
     /path d 12 13 0755
     /dev/console c 5 1 0 0 0600

Were the first column is the path with the 'file-node-map-base' as the 
prefix added internally, the next the type 'f' for file, 'd' for 
directory and 'c' for character device. If its a device the next two 
columns are major and minor. the next 3 columns for all node types is 
the uid gid and permissions.

This file is loaded by proot and then it emulates those files. On chown, 
chmod and mknod the internal struct from proot is changed. When proot 
exits it saves the internal struct back to 'map.txt'.

Since all the files seems to have the correct permission from inside the 
proot, calling tar from within would create the right permissions inside 
the resulting tarball.

 From a correct tarball wic should be able to create the correct image 
using their tools (pseudo, ...).

This way we would not depend on the strangeness of pseudo for our tasks.

Cheers,
Claudius

[1] 
https://github.com/proot-me/PRoot/blob/afe225dc804e7926e588e8af22fbc5433b773e2d/doc/proot/roadmap.txt


-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de