[PATCH] buildchroot: Align UID and GID of builder user with caller

[PATCH] buildchroot: Align UID and GID of builder user with caller

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This fixes EPERM on rebuild and also some clean builds: We have to align
the IDs of the builder user with the user in the host environment.
Otherwise, files and directories can become unaccessible during the
build.

Fixes: be291cd991bd ("buildchroot: build debian packages as "builder" not "root"")
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/recipes-devtools/buildchroot/buildchroot.inc       | 4 +++-
 meta/recipes-devtools/buildchroot/files/configscript.sh | 4 ++--
 2 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/meta/recipes-devtools/buildchroot/buildchroot.inc b/meta/recipes-devtools/buildchroot/buildchroot.inc
index 7dd909e..2c44db9 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.inc
+++ b/meta/recipes-devtools/buildchroot/buildchroot.inc
@@ -36,7 +36,9 @@ do_build() {
 
     # Configure root filesystem
     sudo install -m 755 ${WORKDIR}/configscript.sh ${BUILDCHROOT_DIR}
-    sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
+    USER_ID=$(id -u)
+    GROUP_ID=$(id -g)
+    sudo chroot ${BUILDCHROOT_DIR} /configscript.sh $USER_ID $GROUP_ID
 
     sudo mount --bind ${DL_DIR} ${BUILDCHROOT_DIR}/downloads
 }
diff --git a/meta/recipes-devtools/buildchroot/files/configscript.sh b/meta/recipes-devtools/buildchroot/files/configscript.sh
index 30660e7..7e49385 100644
--- a/meta/recipes-devtools/buildchroot/files/configscript.sh
+++ b/meta/recipes-devtools/buildchroot/files/configscript.sh
@@ -10,6 +10,6 @@ locales locales/locales_to_be_generated multiselect en_US.UTF-8 UTF-8
 locales locales/default_environment_locale select en_US.UTF-8
 END
 
-addgroup --quiet --system builder
-useradd --system --gid builder --no-create-home --home /home/builder --no-user-group --comment "Isar buildchroot build user" builder
+addgroup --quiet --system builder --gid $2
+useradd --system --uid $1 --gid builder --no-create-home --home /home/builder --no-user-group --comment "Isar buildchroot build user" builder
 chown -R builder:builder /home/builder
-- 
2.16.4

Re: [PATCH] buildchroot: Align UID and GID of builder user with caller

[Henning Schild]

I am afraid that this is not correct. The ids you are taking from the
"host" might be taken inside the chroot. As a result creating the
user/group would fail. Chances might be low ... This also assumes that
ids/hosts will never change and breaks migrating a build to another
host.
If the host fails to remove/overwrite the files, we will have to use
sudo on the host. 

Henning

Am Sat, 10 Nov 2018 08:52:38 +0100
schrieb Jan Kiszka <jan.kiszka@web.de>:

> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> This fixes EPERM on rebuild and also some clean builds: We have to
> align the IDs of the builder user with the user in the host
> environment. Otherwise, files and directories can become unaccessible
> during the build.
> 
> Fixes: be291cd991bd ("buildchroot: build debian packages as "builder"
> not "root"") Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>  meta/recipes-devtools/buildchroot/buildchroot.inc       | 4 +++-
>  meta/recipes-devtools/buildchroot/files/configscript.sh | 4 ++--
>  2 files changed, 5 insertions(+), 3 deletions(-)
> 
> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.inc
> b/meta/recipes-devtools/buildchroot/buildchroot.inc index
> 7dd909e..2c44db9 100644 ---
> a/meta/recipes-devtools/buildchroot/buildchroot.inc +++
> b/meta/recipes-devtools/buildchroot/buildchroot.inc @@ -36,7 +36,9 @@
> do_build() { 
>      # Configure root filesystem
>      sudo install -m 755 ${WORKDIR}/configscript.sh ${BUILDCHROOT_DIR}
> -    sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
> +    USER_ID=$(id -u)
> +    GROUP_ID=$(id -g)
> +    sudo chroot ${BUILDCHROOT_DIR} /configscript.sh $USER_ID
> $GROUP_ID 
>      sudo mount --bind ${DL_DIR} ${BUILDCHROOT_DIR}/downloads
>  }
> diff --git a/meta/recipes-devtools/buildchroot/files/configscript.sh
> b/meta/recipes-devtools/buildchroot/files/configscript.sh index
> 30660e7..7e49385 100644 ---
> a/meta/recipes-devtools/buildchroot/files/configscript.sh +++
> b/meta/recipes-devtools/buildchroot/files/configscript.sh @@ -10,6
> +10,6 @@ locales locales/locales_to_be_generated multiselect
> en_US.UTF-8 UTF-8 locales locales/default_environment_locale select
> en_US.UTF-8 END 
> -addgroup --quiet --system builder
> -useradd --system --gid builder --no-create-home --home /home/builder
> --no-user-group --comment "Isar buildchroot build user" builder
> +addgroup --quiet --system builder --gid $2 +useradd --system --uid
> $1 --gid builder --no-create-home --home /home/builder
> --no-user-group --comment "Isar buildchroot build user" builder chown
> -R builder:builder /home/builder

Re: [PATCH] buildchroot: Align UID and GID of builder user with caller

[Jan Kiszka]

On 12.11.18 10:16, [ext] Henning Schild wrote:
> I am afraid that this is not correct. The ids you are taking from the
> "host" might be taken inside the chroot. As a result creating the
> user/group would fail. Chances might be low ... This also assumes that

Really? I thought that these commands are run very early during bootstrap where 
there are no other users - if not, that would be a bug.

Jan

> ids/hosts will never change and breaks migrating a build to another
> host.
> If the host fails to remove/overwrite the files, we will have to use
> sudo on the host.
> 
> Henning
> 
> Am Sat, 10 Nov 2018 08:52:38 +0100
> schrieb Jan Kiszka <jan.kiszka@web.de>:
> 
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> This fixes EPERM on rebuild and also some clean builds: We have to
>> align the IDs of the builder user with the user in the host
>> environment. Otherwise, files and directories can become unaccessible
>> during the build.
>>
>> Fixes: be291cd991bd ("buildchroot: build debian packages as "builder"
>> not "root"") Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>   meta/recipes-devtools/buildchroot/buildchroot.inc       | 4 +++-
>>   meta/recipes-devtools/buildchroot/files/configscript.sh | 4 ++--
>>   2 files changed, 5 insertions(+), 3 deletions(-)
>>
>> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.inc
>> b/meta/recipes-devtools/buildchroot/buildchroot.inc index
>> 7dd909e..2c44db9 100644 ---
>> a/meta/recipes-devtools/buildchroot/buildchroot.inc +++
>> b/meta/recipes-devtools/buildchroot/buildchroot.inc @@ -36,7 +36,9 @@
>> do_build() {
>>       # Configure root filesystem
>>       sudo install -m 755 ${WORKDIR}/configscript.sh ${BUILDCHROOT_DIR}
>> -    sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
>> +    USER_ID=$(id -u)
>> +    GROUP_ID=$(id -g)
>> +    sudo chroot ${BUILDCHROOT_DIR} /configscript.sh $USER_ID
>> $GROUP_ID
>>       sudo mount --bind ${DL_DIR} ${BUILDCHROOT_DIR}/downloads
>>   }
>> diff --git a/meta/recipes-devtools/buildchroot/files/configscript.sh
>> b/meta/recipes-devtools/buildchroot/files/configscript.sh index
>> 30660e7..7e49385 100644 ---
>> a/meta/recipes-devtools/buildchroot/files/configscript.sh +++
>> b/meta/recipes-devtools/buildchroot/files/configscript.sh @@ -10,6
>> +10,6 @@ locales locales/locales_to_be_generated multiselect
>> en_US.UTF-8 UTF-8 locales locales/default_environment_locale select
>> en_US.UTF-8 END
>> -addgroup --quiet --system builder
>> -useradd --system --gid builder --no-create-home --home /home/builder
>> --no-user-group --comment "Isar buildchroot build user" builder
>> +addgroup --quiet --system builder --gid $2 +useradd --system --uid
>> $1 --gid builder --no-create-home --home /home/builder
>> --no-user-group --comment "Isar buildchroot build user" builder chown
>> -R builder:builder /home/builder
> 

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH] buildchroot: Align UID and GID of builder user with caller

[Henning Schild]

Am Mon, 12 Nov 2018 10:19:54 +0100
schrieb Jan Kiszka <jan.kiszka@siemens.com>:

> On 12.11.18 10:16, [ext] Henning Schild wrote:
> > I am afraid that this is not correct. The ids you are taking from
> > the "host" might be taken inside the chroot. As a result creating
> > the user/group would fail. Chances might be low ... This also
> > assumes that  
> 
> Really? I thought that these commands are run very early during
> bootstrap where there are no other users - if not, that would be a
> bug.

I think the only uid/gid you can really be sure about is 0. 1 could
already be a regular user on the host, and 1 is "daemon" on a current
debian ... probably there right after debootstrap.

1000 being the first "user" is more a convention than something you can
rely on for any host. (/etc/login.defs UID_MIN/MAX etc.)

Henning

> Jan
> 
> > ids/hosts will never change and breaks migrating a build to another
> > host.
> > If the host fails to remove/overwrite the files, we will have to use
> > sudo on the host.
> > 
> > Henning
> > 
> > Am Sat, 10 Nov 2018 08:52:38 +0100
> > schrieb Jan Kiszka <jan.kiszka@web.de>:
> >   
> >> From: Jan Kiszka <jan.kiszka@siemens.com>
> >>
> >> This fixes EPERM on rebuild and also some clean builds: We have to
> >> align the IDs of the builder user with the user in the host
> >> environment. Otherwise, files and directories can become
> >> unaccessible during the build.
> >>
> >> Fixes: be291cd991bd ("buildchroot: build debian packages as
> >> "builder" not "root"") Signed-off-by: Jan Kiszka
> >> <jan.kiszka@siemens.com> ---
> >>   meta/recipes-devtools/buildchroot/buildchroot.inc       | 4 +++-
> >>   meta/recipes-devtools/buildchroot/files/configscript.sh | 4 ++--
> >>   2 files changed, 5 insertions(+), 3 deletions(-)
> >>
> >> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.inc
> >> b/meta/recipes-devtools/buildchroot/buildchroot.inc index
> >> 7dd909e..2c44db9 100644 ---
> >> a/meta/recipes-devtools/buildchroot/buildchroot.inc +++
> >> b/meta/recipes-devtools/buildchroot/buildchroot.inc @@ -36,7 +36,9
> >> @@ do_build() {
> >>       # Configure root filesystem
> >>       sudo install -m 755 ${WORKDIR}/configscript.sh
> >> ${BUILDCHROOT_DIR}
> >> -    sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
> >> +    USER_ID=$(id -u)
> >> +    GROUP_ID=$(id -g)
> >> +    sudo chroot ${BUILDCHROOT_DIR} /configscript.sh $USER_ID
> >> $GROUP_ID
> >>       sudo mount --bind ${DL_DIR} ${BUILDCHROOT_DIR}/downloads
> >>   }
> >> diff --git
> >> a/meta/recipes-devtools/buildchroot/files/configscript.sh
> >> b/meta/recipes-devtools/buildchroot/files/configscript.sh index
> >> 30660e7..7e49385 100644 ---
> >> a/meta/recipes-devtools/buildchroot/files/configscript.sh +++
> >> b/meta/recipes-devtools/buildchroot/files/configscript.sh @@ -10,6
> >> +10,6 @@ locales locales/locales_to_be_generated multiselect
> >> en_US.UTF-8 UTF-8 locales locales/default_environment_locale
> >> select en_US.UTF-8 END -addgroup --quiet --system builder
> >> -useradd --system --gid builder --no-create-home
> >> --home /home/builder --no-user-group --comment "Isar buildchroot
> >> build user" builder +addgroup --quiet --system builder --gid $2
> >> +useradd --system --uid $1 --gid builder --no-create-home
> >> --home /home/builder --no-user-group --comment "Isar buildchroot
> >> build user" builder chown -R builder:builder /home/builder  
> >   
> 

Re: [PATCH] buildchroot: Align UID and GID of builder user with caller

[Jan Kiszka]

On 12.11.18 10:42, Henning Schild wrote:
> Am Mon, 12 Nov 2018 10:19:54 +0100
> schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> 
>> On 12.11.18 10:16, [ext] Henning Schild wrote:
>>> I am afraid that this is not correct. The ids you are taking from
>>> the "host" might be taken inside the chroot. As a result creating
>>> the user/group would fail. Chances might be low ... This also
>>> assumes that
>>
>> Really? I thought that these commands are run very early during
>> bootstrap where there are no other users - if not, that would be a
>> bug.
> 
> I think the only uid/gid you can really be sure about is 0. 1 could
> already be a regular user on the host, and 1 is "daemon" on a current
> debian ... probably there right after debootstrap.

Let me check if we can move the ID assignment earlier, to reduce that risk.

> 
> 1000 being the first "user" is more a convention than something you can
> rely on for any host. (/etc/login.defs UID_MIN/MAX etc.)

We are talking about transferring the ID's from the host Debian to the 
buildchroot Debian - is there really a realistic risk of friction?

If we can't solve that sync problem, we need to revert to running as root, I'm 
afraid. The current model is broken.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH] buildchroot: Align UID and GID of builder user with caller

[Henning Schild]

Am Mon, 12 Nov 2018 10:52:22 +0100
schrieb Jan Kiszka <jan.kiszka@siemens.com>:

> On 12.11.18 10:42, Henning Schild wrote:
> > Am Mon, 12 Nov 2018 10:19:54 +0100
> > schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> >   
> >> On 12.11.18 10:16, [ext] Henning Schild wrote:  
> >>> I am afraid that this is not correct. The ids you are taking from
> >>> the "host" might be taken inside the chroot. As a result creating
> >>> the user/group would fail. Chances might be low ... This also
> >>> assumes that  
> >>
> >> Really? I thought that these commands are run very early during
> >> bootstrap where there are no other users - if not, that would be a
> >> bug.  
> > 
> > I think the only uid/gid you can really be sure about is 0. 1 could
> > already be a regular user on the host, and 1 is "daemon" on a
> > current debian ... probably there right after debootstrap.  
> 
> Let me check if we can move the ID assignment earlier, to reduce that
> risk.

I will look into it. Knowing a problem and reducing the risk is not
good enough.

> > 
> > 1000 being the first "user" is more a convention than something you
> > can rely on for any host. (/etc/login.defs UID_MIN/MAX etc.)  
> 
> We are talking about transferring the ID's from the host Debian to
> the buildchroot Debian - is there really a realistic risk of friction?

Now you are assuming that everyone is using your container ;). While
this is helpful i would like to allow anyone to build without docker,
given they have a few debian utils on their machine.

> If we can't solve that sync problem, we need to revert to running as
> root, I'm afraid. The current model is broken.

I will send a follow up patch ... maybe today. The reproduction build
is already running.

Did you see it in any other package than u-boot? Maybe the u-boot
recipes are broken? I still do not see how a file formerly owned
by root:root can cause problems as 1000:1000 ... but i guess i will
understand that once i can reproduce.

Henning

> Jan
> 

Re: [PATCH] buildchroot: Align UID and GID of builder user with caller

[Jan Kiszka]

On 12.11.18 11:06, Henning Schild wrote:
> Am Mon, 12 Nov 2018 10:52:22 +0100
> schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> 
>> On 12.11.18 10:42, Henning Schild wrote:
>>> Am Mon, 12 Nov 2018 10:19:54 +0100
>>> schrieb Jan Kiszka <jan.kiszka@siemens.com>:
>>>    
>>>> On 12.11.18 10:16, [ext] Henning Schild wrote:
>>>>> I am afraid that this is not correct. The ids you are taking from
>>>>> the "host" might be taken inside the chroot. As a result creating
>>>>> the user/group would fail. Chances might be low ... This also
>>>>> assumes that
>>>>
>>>> Really? I thought that these commands are run very early during
>>>> bootstrap where there are no other users - if not, that would be a
>>>> bug.
>>>
>>> I think the only uid/gid you can really be sure about is 0. 1 could
>>> already be a regular user on the host, and 1 is "daemon" on a
>>> current debian ... probably there right after debootstrap.
>>
>> Let me check if we can move the ID assignment earlier, to reduce that
>> risk.
> 
> I will look into it. Knowing a problem and reducing the risk is not
> good enough.
> 
>>>
>>> 1000 being the first "user" is more a convention than something you
>>> can rely on for any host. (/etc/login.defs UID_MIN/MAX etc.)
>>
>> We are talking about transferring the ID's from the host Debian to
>> the buildchroot Debian - is there really a realistic risk of friction?
> 
> Now you are assuming that everyone is using your container ;). While

No, this is not about the container. We already solved the problem for the 
container long ago (by aligning IDs). This breakage is about the host (in the 
container or on your host) and the buildchroot.

> this is helpful i would like to allow anyone to build without docker,
> given they have a few debian utils on their machine.
> 
>> If we can't solve that sync problem, we need to revert to running as
>> root, I'm afraid. The current model is broken.
> 
> I will send a follow up patch ... maybe today. The reproduction build
> is already running.

TIA!

> 
> Did you see it in any other package than u-boot? Maybe the u-boot
> recipes are broken? I still do not see how a file formerly owned
> by root:root can cause problems as 1000:1000 ... but i guess i will
> understand that once i can reproduce.

U-boot is exposing it, but you see the breakage earlier by getting funny UIDs of 
the artifacts after running a dpkg build.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH] buildchroot: Align UID and GID of builder user with caller

[Henning Schild]

Am Mon, 12 Nov 2018 11:09:27 +0100
schrieb Jan Kiszka <jan.kiszka@siemens.com>:

> On 12.11.18 11:06, Henning Schild wrote:
> > Am Mon, 12 Nov 2018 10:52:22 +0100
> > schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> >   
> >> On 12.11.18 10:42, Henning Schild wrote:  
> >>> Am Mon, 12 Nov 2018 10:19:54 +0100
> >>> schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> >>>      
> >>>> On 12.11.18 10:16, [ext] Henning Schild wrote:  
> >>>>> I am afraid that this is not correct. The ids you are taking
> >>>>> from the "host" might be taken inside the chroot. As a result
> >>>>> creating the user/group would fail. Chances might be low ...
> >>>>> This also assumes that  
> >>>>
> >>>> Really? I thought that these commands are run very early during
> >>>> bootstrap where there are no other users - if not, that would be
> >>>> a bug.  
> >>>
> >>> I think the only uid/gid you can really be sure about is 0. 1
> >>> could already be a regular user on the host, and 1 is "daemon" on
> >>> a current debian ... probably there right after debootstrap.  
> >>
> >> Let me check if we can move the ID assignment earlier, to reduce
> >> that risk.  
> > 
> > I will look into it. Knowing a problem and reducing the risk is not
> > good enough.
> >   
> >>>
> >>> 1000 being the first "user" is more a convention than something
> >>> you can rely on for any host. (/etc/login.defs UID_MIN/MAX etc.)  
> >>
> >> We are talking about transferring the ID's from the host Debian to
> >> the buildchroot Debian - is there really a realistic risk of
> >> friction?  
> > 
> > Now you are assuming that everyone is using your container ;).
> > While  
> 
> No, this is not about the container. We already solved the problem
> for the container long ago (by aligning IDs). This breakage is about
> the host (in the container or on your host) and the buildchroot.

If the container has aligned IDs that was a hack as well. I guess for
the same problem i just found ....
The problem is a "chown -R" in do_unpack that brings the hosts uid into
the chroot.
That should be fixed properly some day ... i just sent a workaround
patch.

Henning

> > this is helpful i would like to allow anyone to build without
> > docker, given they have a few debian utils on their machine.
> >   
> >> If we can't solve that sync problem, we need to revert to running
> >> as root, I'm afraid. The current model is broken.  
> > 
> > I will send a follow up patch ... maybe today. The reproduction
> > build is already running.  
> 
> TIA!
> 
> > 
> > Did you see it in any other package than u-boot? Maybe the u-boot
> > recipes are broken? I still do not see how a file formerly owned
> > by root:root can cause problems as 1000:1000 ... but i guess i will
> > understand that once i can reproduce.  
> 
> U-boot is exposing it, but you see the breakage earlier by getting
> funny UIDs of the artifacts after running a dpkg build.
> 
> Jan
> 

Re: [PATCH] buildchroot: Align UID and GID of builder user with caller

[Jan Kiszka]

On 12.11.18 12:58, Henning Schild wrote:
> Am Mon, 12 Nov 2018 11:09:27 +0100
> schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> 
>> On 12.11.18 11:06, Henning Schild wrote:
>>> Am Mon, 12 Nov 2018 10:52:22 +0100
>>> schrieb Jan Kiszka <jan.kiszka@siemens.com>:
>>>    
>>>> On 12.11.18 10:42, Henning Schild wrote:
>>>>> Am Mon, 12 Nov 2018 10:19:54 +0100
>>>>> schrieb Jan Kiszka <jan.kiszka@siemens.com>:
>>>>>       
>>>>>> On 12.11.18 10:16, [ext] Henning Schild wrote:
>>>>>>> I am afraid that this is not correct. The ids you are taking
>>>>>>> from the "host" might be taken inside the chroot. As a result
>>>>>>> creating the user/group would fail. Chances might be low ...
>>>>>>> This also assumes that
>>>>>>
>>>>>> Really? I thought that these commands are run very early during
>>>>>> bootstrap where there are no other users - if not, that would be
>>>>>> a bug.
>>>>>
>>>>> I think the only uid/gid you can really be sure about is 0. 1
>>>>> could already be a regular user on the host, and 1 is "daemon" on
>>>>> a current debian ... probably there right after debootstrap.
>>>>
>>>> Let me check if we can move the ID assignment earlier, to reduce
>>>> that risk.
>>>
>>> I will look into it. Knowing a problem and reducing the risk is not
>>> good enough.
>>>    
>>>>>
>>>>> 1000 being the first "user" is more a convention than something
>>>>> you can rely on for any host. (/etc/login.defs UID_MIN/MAX etc.)
>>>>
>>>> We are talking about transferring the ID's from the host Debian to
>>>> the buildchroot Debian - is there really a realistic risk of
>>>> friction?
>>>
>>> Now you are assuming that everyone is using your container ;).
>>> While
>>
>> No, this is not about the container. We already solved the problem
>> for the container long ago (by aligning IDs). This breakage is about
>> the host (in the container or on your host) and the buildchroot.
> 
> If the container has aligned IDs that was a hack as well. I guess for
> the same problem i just found ....

No, this is working smoothly for quite a while now, also in many CI setup, 
because we control the container content and the fact that there are practically 
no ID overlaps. It is a mandatory feature there, for the same reasons like here.

> The problem is a "chown -R" in do_unpack that brings the hosts uid into
> the chroot.
> That should be fixed properly some day ... i just sent a workaround
> patch.

I'm rather in favor of a proper fix to the ID mess than more working around it. 
The benefit of going for the calling user ID would be - if implemented properly 
- that we will have less files owned by root or some other users unknown to the 
build host, and can therefore only be purged by means of sudo.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH] buildchroot: Align UID and GID of builder user with caller

[Jan Kiszka]

On 12.11.18 13:11, Jan Kiszka wrote:
> On 12.11.18 12:58, Henning Schild wrote:
>> Am Mon, 12 Nov 2018 11:09:27 +0100
>> schrieb Jan Kiszka <jan.kiszka@siemens.com>:
>>
>>> On 12.11.18 11:06, Henning Schild wrote:
>>>> Am Mon, 12 Nov 2018 10:52:22 +0100
>>>> schrieb Jan Kiszka <jan.kiszka@siemens.com>:
>>>>> On 12.11.18 10:42, Henning Schild wrote:
>>>>>> Am Mon, 12 Nov 2018 10:19:54 +0100
>>>>>> schrieb Jan Kiszka <jan.kiszka@siemens.com>:
>>>>>>> On 12.11.18 10:16, [ext] Henning Schild wrote:
>>>>>>>> I am afraid that this is not correct. The ids you are taking
>>>>>>>> from the "host" might be taken inside the chroot. As a result
>>>>>>>> creating the user/group would fail. Chances might be low ...
>>>>>>>> This also assumes that
>>>>>>>
>>>>>>> Really? I thought that these commands are run very early during
>>>>>>> bootstrap where there are no other users - if not, that would be
>>>>>>> a bug.
>>>>>>
>>>>>> I think the only uid/gid you can really be sure about is 0. 1
>>>>>> could already be a regular user on the host, and 1 is "daemon" on
>>>>>> a current debian ... probably there right after debootstrap.
>>>>>
>>>>> Let me check if we can move the ID assignment earlier, to reduce
>>>>> that risk.
>>>>
>>>> I will look into it. Knowing a problem and reducing the risk is not
>>>> good enough.
>>>>>>
>>>>>> 1000 being the first "user" is more a convention than something
>>>>>> you can rely on for any host. (/etc/login.defs UID_MIN/MAX etc.)
>>>>>
>>>>> We are talking about transferring the ID's from the host Debian to
>>>>> the buildchroot Debian - is there really a realistic risk of
>>>>> friction?
>>>>
>>>> Now you are assuming that everyone is using your container ;).
>>>> While
>>>
>>> No, this is not about the container. We already solved the problem
>>> for the container long ago (by aligning IDs). This breakage is about
>>> the host (in the container or on your host) and the buildchroot.
>>
>> If the container has aligned IDs that was a hack as well. I guess for
>> the same problem i just found ....
> 
> No, this is working smoothly for quite a while now, also in many CI setup, 
> because we control the container content and the fact that there are practically 
> no ID overlaps. It is a mandatory feature there, for the same reasons like here.
> 
>> The problem is a "chown -R" in do_unpack that brings the hosts uid into
>> the chroot.
>> That should be fixed properly some day ... i just sent a workaround
>> patch.
> 
> I'm rather in favor of a proper fix to the ID mess than more working around it. 
> The benefit of going for the calling user ID would be - if implemented properly 
> - that we will have less files owned by root or some other users unknown to the 
> build host, and can therefore only be purged by means of sudo.

Actually, also 624b7c484bf5 could be reverted if we managed to align IDs...

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH] buildchroot: Align UID and GID of builder user with caller

[Henning Schild]

Am Mon, 12 Nov 2018 13:27:34 +0100
schrieb Jan Kiszka <jan.kiszka@siemens.com>:

> On 12.11.18 13:11, Jan Kiszka wrote:
> > On 12.11.18 12:58, Henning Schild wrote:  
> >> Am Mon, 12 Nov 2018 11:09:27 +0100
> >> schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> >>  
> >>> On 12.11.18 11:06, Henning Schild wrote:  
> >>>> Am Mon, 12 Nov 2018 10:52:22 +0100
> >>>> schrieb Jan Kiszka <jan.kiszka@siemens.com>:  
> >>>>> On 12.11.18 10:42, Henning Schild wrote:  
> >>>>>> Am Mon, 12 Nov 2018 10:19:54 +0100
> >>>>>> schrieb Jan Kiszka <jan.kiszka@siemens.com>:  
> >>>>>>> On 12.11.18 10:16, [ext] Henning Schild wrote:  
> >>>>>>>> I am afraid that this is not correct. The ids you are taking
> >>>>>>>> from the "host" might be taken inside the chroot. As a result
> >>>>>>>> creating the user/group would fail. Chances might be low ...
> >>>>>>>> This also assumes that  
> >>>>>>>
> >>>>>>> Really? I thought that these commands are run very early
> >>>>>>> during bootstrap where there are no other users - if not,
> >>>>>>> that would be a bug.  
> >>>>>>
> >>>>>> I think the only uid/gid you can really be sure about is 0. 1
> >>>>>> could already be a regular user on the host, and 1 is "daemon"
> >>>>>> on a current debian ... probably there right after
> >>>>>> debootstrap.  
> >>>>>
> >>>>> Let me check if we can move the ID assignment earlier, to reduce
> >>>>> that risk.  
> >>>>
> >>>> I will look into it. Knowing a problem and reducing the risk is
> >>>> not good enough.  
> >>>>>>
> >>>>>> 1000 being the first "user" is more a convention than something
> >>>>>> you can rely on for any host. (/etc/login.defs UID_MIN/MAX
> >>>>>> etc.)  
> >>>>>
> >>>>> We are talking about transferring the ID's from the host Debian
> >>>>> to the buildchroot Debian - is there really a realistic risk of
> >>>>> friction?  
> >>>>
> >>>> Now you are assuming that everyone is using your container ;).
> >>>> While  
> >>>
> >>> No, this is not about the container. We already solved the problem
> >>> for the container long ago (by aligning IDs). This breakage is
> >>> about the host (in the container or on your host) and the
> >>> buildchroot.  
> >>
> >> If the container has aligned IDs that was a hack as well. I guess
> >> for the same problem i just found ....  
> > 
> > No, this is working smoothly for quite a while now, also in many CI
> > setup, because we control the container content and the fact that
> > there are practically no ID overlaps. It is a mandatory feature
> > there, for the same reasons like here. 
> >> The problem is a "chown -R" in do_unpack that brings the hosts uid
> >> into the chroot.
> >> That should be fixed properly some day ... i just sent a workaround
> >> patch.  
> > 
> > I'm rather in favor of a proper fix to the ID mess than more
> > working around it. The benefit of going for the calling user ID
> > would be - if implemented properly 
> > - that we will have less files owned by root or some other users
> > unknown to the build host, and can therefore only be purged by
> > means of sudo.  
> 
> Actually, also 624b7c484bf5 could be reverted if we managed to align
> IDs...

I did not go all they way back to blame that chown. In fact reverting
624b7c484bf5 should solve the issue.
In a buildchroot we used to have just one user "root", so chowning to
anyone else is just wrong. After my patch you can now chown to
builder. From inside with "builder" or from outside with the id you
extract from etc/passwd. If "file server convenience" is the only reason
behind that patch, please revert that one and forget my update.

Henning

> Jan
> 

Re: [PATCH] buildchroot: Align UID and GID of builder user with caller

[Henning Schild]

Am Mon, 12 Nov 2018 13:11:42 +0100
schrieb Jan Kiszka <jan.kiszka@siemens.com>:

> On 12.11.18 12:58, Henning Schild wrote:
> > Am Mon, 12 Nov 2018 11:09:27 +0100
> > schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> >   
> >> On 12.11.18 11:06, Henning Schild wrote:  
> >>> Am Mon, 12 Nov 2018 10:52:22 +0100
> >>> schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> >>>      
> >>>> On 12.11.18 10:42, Henning Schild wrote:  
> >>>>> Am Mon, 12 Nov 2018 10:19:54 +0100
> >>>>> schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> >>>>>         
> >>>>>> On 12.11.18 10:16, [ext] Henning Schild wrote:  
> >>>>>>> I am afraid that this is not correct. The ids you are taking
> >>>>>>> from the "host" might be taken inside the chroot. As a result
> >>>>>>> creating the user/group would fail. Chances might be low ...
> >>>>>>> This also assumes that  
> >>>>>>
> >>>>>> Really? I thought that these commands are run very early during
> >>>>>> bootstrap where there are no other users - if not, that would
> >>>>>> be a bug.  
> >>>>>
> >>>>> I think the only uid/gid you can really be sure about is 0. 1
> >>>>> could already be a regular user on the host, and 1 is "daemon"
> >>>>> on a current debian ... probably there right after
> >>>>> debootstrap.  
> >>>>
> >>>> Let me check if we can move the ID assignment earlier, to reduce
> >>>> that risk.  
> >>>
> >>> I will look into it. Knowing a problem and reducing the risk is
> >>> not good enough.
> >>>      
> >>>>>
> >>>>> 1000 being the first "user" is more a convention than something
> >>>>> you can rely on for any host. (/etc/login.defs UID_MIN/MAX
> >>>>> etc.)  
> >>>>
> >>>> We are talking about transferring the ID's from the host Debian
> >>>> to the buildchroot Debian - is there really a realistic risk of
> >>>> friction?  
> >>>
> >>> Now you are assuming that everyone is using your container ;).
> >>> While  
> >>
> >> No, this is not about the container. We already solved the problem
> >> for the container long ago (by aligning IDs). This breakage is
> >> about the host (in the container or on your host) and the
> >> buildchroot.  
> > 
> > If the container has aligned IDs that was a hack as well. I guess
> > for the same problem i just found ....  
> 
> No, this is working smoothly for quite a while now, also in many CI
> setup, because we control the container content and the fact that
> there are practically no ID overlaps. It is a mandatory feature
> there, for the same reasons like here.

Ok it is a hack that is working smoothly. The need to do that indicates
a problem and implications on the host. (and the problem is chowns
from two worlds)

> > The problem is a "chown -R" in do_unpack that brings the hosts uid
> > into the chroot.
> > That should be fixed properly some day ... i just sent a workaround
> > patch.  
> 
> I'm rather in favor of a proper fix to the ID mess than more working
> around it.

My patch repaired my other patch. Solving the other - long standing -
issue in another story.

> The benefit of going for the calling user ID would be - if
> implemented properly 
> - that we will have less files owned by root or some other users
> unknown to the build host, and can therefore only be purged by means
> of sudo.

As long as there is at least one file owned by root, you will need to
be root on the host. And by host i mean where Isar called "sudo" for
you. While you probably are talking about your docker/file server. In
order to deal with those files, we will just need a "sudo" enabled
clean target on the host. This way your file server should not see any
root-owned files anymore.

"rm -rf out/" -> "docker run make clean"

Henning 

> Jan
> 

Re: [PATCH] buildchroot: Align UID and GID of builder user with caller

[Jan Kiszka]

On 12.11.18 10:52, Jan Kiszka wrote:
> On 12.11.18 10:42, Henning Schild wrote:
>> Am Mon, 12 Nov 2018 10:19:54 +0100
>> schrieb Jan Kiszka <jan.kiszka@siemens.com>:
>>
>>> On 12.11.18 10:16, [ext] Henning Schild wrote:
>>>> I am afraid that this is not correct. The ids you are taking from
>>>> the "host" might be taken inside the chroot. As a result creating
>>>> the user/group would fail. Chances might be low ... This also
>>>> assumes that
>>>
>>> Really? I thought that these commands are run very early during
>>> bootstrap where there are no other users - if not, that would be a
>>> bug.
>>
>> I think the only uid/gid you can really be sure about is 0. 1 could
>> already be a regular user on the host, and 1 is "daemon" on a current
>> debian ... probably there right after debootstrap.
> 
> Let me check if we can move the ID assignment earlier, to reduce that risk.
> 

The solution is probably much simpler:

useradd -o ...

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux