[PATCH 1/1] wic: disable generation of python bytecode cache

[PATCH 1/1] wic: disable generation of python bytecode cache

['Felix Moessbauer' via isar-users]

Wic is executed as root, so the python bytecode cache is also
created as root. This is problematic as the cache is created inside the
source tree in a folder__pycache__ next to the python script itself. By
that, we end up with files belonging to root inside the repo source
tree, which makes it impossible to delete the source tree as a regular
user.

This problem became visible with the kas purge plugin that removes the
fetched layers as a regular user (these layers are fetched and managed
by kas). Also the read-only mounting of repos in kas does not help here,
as the fetched repos are not mounted ro for obvious reasons.

Anyways, we should not create files inside the source tree that do not
belong to the calling user. To fix this, we just disable the python
cache for the wic task. This is the only task that executes a python
script from the source tree as root.

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/imagetypes_wic.bbclass | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/classes/imagetypes_wic.bbclass b/meta/classes/imagetypes_wic.bbclass
index 7a050e73..fb0b81a9 100644
--- a/meta/classes/imagetypes_wic.bbclass
+++ b/meta/classes/imagetypes_wic.bbclass
@@ -157,6 +157,7 @@ generate_wic_image() {
     export FAKEROOTCMD=${FAKEROOTCMD}
     export BUILDDIR=${TOPDIR}
     export MTOOLS_SKIP_CHECK=1
+    export PYTHONDONTWRITEBYTECODE=1
     mkdir -p ${IMAGE_ROOTFS}/../pseudo
     touch ${IMAGE_ROOTFS}/../pseudo/files.db
 
-- 
2.49.0

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20250605145837.1384007-1-felix.moessbauer%40siemens.com.

Re: [PATCH 1/1] wic: disable generation of python bytecode cache

['Heinisch, Alexander' via isar-users]

On Thu, 2025-06-05 at 16:58 +0200, 'Felix Moessbauer' via isar-users
wrote:
> Wic is executed as root, so the python bytecode cache is also
> created as root. This is problematic as the cache is created inside
> the
> source tree in a folder__pycache__ next to the python script itself.
> By
> that, we end up with files belonging to root inside the repo source
> tree, which makes it impossible to delete the source tree as a
> regular
> user.
> 
> This problem became visible with the kas purge plugin that removes
> the
> fetched layers as a regular user (these layers are fetched and
> managed
> by kas). Also the read-only mounting of repos in kas does not help
> here,
> as the fetched repos are not mounted ro for obvious reasons.
> 
> Anyways, we should not create files inside the source tree that do
> not
> belong to the calling user. To fix this, we just disable the python
> cache for the wic task. This is the only task that executes a python
> script from the source tree as root.

Hi Felix,

What about moving the cache to the build dir?

I just did a quick test using `PYTHONPYCACHEPREFIX` (see [1]) which was
added in Python 3.8 (so works for hosts >= bullseye)

```
diff --git a/meta/classes/imagetypes_wic.bbclass
b/meta/classes/imagetypes_wic.bbclass
index 38b5f0e1..7b8dc38c 100644
--- a/meta/classes/imagetypes_wic.bbclass
+++ b/meta/classes/imagetypes_wic.bbclass
@@ -157,6 +157,7 @@ generate_wic_image() {
     export FAKEROOTCMD=${FAKEROOTCMD}
     export BUILDDIR=${TOPDIR}
     export MTOOLS_SKIP_CHECK=1
+    export PYTHONPYCACHEPREFIX="${TOPDIR}/__pycache__"
     mkdir -p ${IMAGE_ROOTFS}/../pseudo
     touch ${IMAGE_ROOTFS}/../pseudo/files.db```

and it seems to do the job. In depth testing still required!

[1]:
https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPYCACHEPREFIX

BR Alexander

> 
> Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
> ---
>  meta/classes/imagetypes_wic.bbclass | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/meta/classes/imagetypes_wic.bbclass
> b/meta/classes/imagetypes_wic.bbclass
> index 7a050e73..fb0b81a9 100644
> --- a/meta/classes/imagetypes_wic.bbclass
> +++ b/meta/classes/imagetypes_wic.bbclass
> @@ -157,6 +157,7 @@ generate_wic_image() {
>      export FAKEROOTCMD=${FAKEROOTCMD}
>      export BUILDDIR=${TOPDIR}
>      export MTOOLS_SKIP_CHECK=1
> +    export PYTHONDONTWRITEBYTECODE=1
>      mkdir -p ${IMAGE_ROOTFS}/../pseudo
>      touch ${IMAGE_ROOTFS}/../pseudo/files.db
>  
> -- 
> 2.49.0
> 

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/6089b70079752de7da49c81a94c43600912dd801.camel%40siemens.com.

Re: [PATCH 1/1] wic: disable generation of python bytecode cache

['Jan Kiszka' via isar-users]

On 06.06.25 08:19, Heinisch, Alexander (FT RPD CED SES-AT) wrote:
> On Thu, 2025-06-05 at 16:58 +0200, 'Felix Moessbauer' via isar-users
> wrote:
>> Wic is executed as root, so the python bytecode cache is also
>> created as root. This is problematic as the cache is created inside
>> the
>> source tree in a folder__pycache__ next to the python script itself.
>> By
>> that, we end up with files belonging to root inside the repo source
>> tree, which makes it impossible to delete the source tree as a
>> regular
>> user.
>>
>> This problem became visible with the kas purge plugin that removes
>> the
>> fetched layers as a regular user (these layers are fetched and
>> managed
>> by kas). Also the read-only mounting of repos in kas does not help
>> here,
>> as the fetched repos are not mounted ro for obvious reasons.
>>
>> Anyways, we should not create files inside the source tree that do
>> not
>> belong to the calling user. To fix this, we just disable the python
>> cache for the wic task. This is the only task that executes a python
>> script from the source tree as root.
> 
> Hi Felix,
> 
> What about moving the cache to the build dir?
> 
> I just did a quick test using `PYTHONPYCACHEPREFIX` (see [1]) which was
> added in Python 3.8 (so works for hosts >= bullseye)
> 
> ```
> diff --git a/meta/classes/imagetypes_wic.bbclass
> b/meta/classes/imagetypes_wic.bbclass
> index 38b5f0e1..7b8dc38c 100644
> --- a/meta/classes/imagetypes_wic.bbclass
> +++ b/meta/classes/imagetypes_wic.bbclass
> @@ -157,6 +157,7 @@ generate_wic_image() {
>      export FAKEROOTCMD=${FAKEROOTCMD}
>      export BUILDDIR=${TOPDIR}
>      export MTOOLS_SKIP_CHECK=1
> +    export PYTHONPYCACHEPREFIX="${TOPDIR}/__pycache__"
>      mkdir -p ${IMAGE_ROOTFS}/../pseudo
>      touch ${IMAGE_ROOTFS}/../pseudo/files.db```
> 
> and it seems to do the job. In depth testing still required!
> 
> [1]:
> https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPYCACHEPREFIX
> 

What's yocto doing about this BTW? Are we just missing some of their
bits, or are they affected by a similar issue (except for creating the
cache as root)?

Jan

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/319f0cbe-dc36-49eb-82b1-d3a0ed468707%40siemens.com.

Re: [PATCH 1/1] wic: disable generation of python bytecode cache

['MOESSBAUER, Felix' via isar-users]

On Fri, 2025-06-06 at 08:21 +0200, Jan Kiszka wrote:
> On 06.06.25 08:19, Heinisch, Alexander (FT RPD CED SES-AT) wrote:
> > On Thu, 2025-06-05 at 16:58 +0200, 'Felix Moessbauer' via isar-
> > users
> > wrote:
> > > Wic is executed as root, so the python bytecode cache is also
> > > created as root. This is problematic as the cache is created
> > > inside
> > > the
> > > source tree in a folder__pycache__ next to the python script
> > > itself.
> > > By
> > > that, we end up with files belonging to root inside the repo
> > > source
> > > tree, which makes it impossible to delete the source tree as a
> > > regular
> > > user.
> > > 
> > > This problem became visible with the kas purge plugin that
> > > removes
> > > the
> > > fetched layers as a regular user (these layers are fetched and
> > > managed
> > > by kas). Also the read-only mounting of repos in kas does not
> > > help
> > > here,
> > > as the fetched repos are not mounted ro for obvious reasons.
> > > 
> > > Anyways, we should not create files inside the source tree that
> > > do
> > > not
> > > belong to the calling user. To fix this, we just disable the
> > > python
> > > cache for the wic task. This is the only task that executes a
> > > python
> > > script from the source tree as root.
> > 
> > Hi Felix,
> > 
> > What about moving the cache to the build dir?
> > 
> > I just did a quick test using `PYTHONPYCACHEPREFIX` (see [1]) which
> > was
> > added in Python 3.8 (so works for hosts >= bullseye)
> > 
> > ```
> > diff --git a/meta/classes/imagetypes_wic.bbclass
> > b/meta/classes/imagetypes_wic.bbclass
> > index 38b5f0e1..7b8dc38c 100644
> > --- a/meta/classes/imagetypes_wic.bbclass
> > +++ b/meta/classes/imagetypes_wic.bbclass
> > @@ -157,6 +157,7 @@ generate_wic_image() {
> >      export FAKEROOTCMD=${FAKEROOTCMD}
> >      export BUILDDIR=${TOPDIR}
> >      export MTOOLS_SKIP_CHECK=1
> > +    export PYTHONPYCACHEPREFIX="${TOPDIR}/__pycache__"

This script is not called frequently, so there is no real benefit in
caching it. If we want to cache, we should probably cache in the
WORKDIR.

> >      mkdir -p ${IMAGE_ROOTFS}/../pseudo
> >      touch ${IMAGE_ROOTFS}/../pseudo/files.db```
> > 
> > and it seems to do the job. In depth testing still required!
> > 
> > [1]:
> > https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPYCACHEPREFIX
> > 
> 
> What's yocto doing about this BTW? Are we just missing some of their
> bits, or are they affected by a similar issue (except for creating
> the
> cache as root)?

Yocto also disables the python cache for their fakeroot environments:

https://github.com/openembedded/openembedded-core/blob/bd8fc4c59a137a37bd7a54f398949617982d447e/meta/conf/bitbake.conf#L753

But as isar differs here, we either need to add this statement where I
added it (take the patch as-is), or implement this logic in
`wic_fakeroot`.

Felix

> 
> Jan

-- 
Siemens AG
Linux Expert Center
Friedrich-Ludwig-Bauer-Str. 3
85748 Garching, Germany

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/a588fb53f96013a4adb2a35c503dbf09228fae4f.camel%40siemens.com.

Re: [PATCH 1/1] wic: disable generation of python bytecode cache

['Jan Kiszka' via isar-users]

On 06.06.25 09:18, Moessbauer, Felix (FT RPD CED OES-DE) wrote:
> On Fri, 2025-06-06 at 08:21 +0200, Jan Kiszka wrote:
>> On 06.06.25 08:19, Heinisch, Alexander (FT RPD CED SES-AT) wrote:
>>> On Thu, 2025-06-05 at 16:58 +0200, 'Felix Moessbauer' via isar-
>>> users
>>> wrote:
>>>> Wic is executed as root, so the python bytecode cache is also
>>>> created as root. This is problematic as the cache is created
>>>> inside
>>>> the
>>>> source tree in a folder__pycache__ next to the python script
>>>> itself.
>>>> By
>>>> that, we end up with files belonging to root inside the repo
>>>> source
>>>> tree, which makes it impossible to delete the source tree as a
>>>> regular
>>>> user.
>>>>
>>>> This problem became visible with the kas purge plugin that
>>>> removes
>>>> the
>>>> fetched layers as a regular user (these layers are fetched and
>>>> managed
>>>> by kas). Also the read-only mounting of repos in kas does not
>>>> help
>>>> here,
>>>> as the fetched repos are not mounted ro for obvious reasons.
>>>>
>>>> Anyways, we should not create files inside the source tree that
>>>> do
>>>> not
>>>> belong to the calling user. To fix this, we just disable the
>>>> python
>>>> cache for the wic task. This is the only task that executes a
>>>> python
>>>> script from the source tree as root.
>>>
>>> Hi Felix,
>>>
>>> What about moving the cache to the build dir?
>>>
>>> I just did a quick test using `PYTHONPYCACHEPREFIX` (see [1]) which
>>> was
>>> added in Python 3.8 (so works for hosts >= bullseye)
>>>
>>> ```
>>> diff --git a/meta/classes/imagetypes_wic.bbclass
>>> b/meta/classes/imagetypes_wic.bbclass
>>> index 38b5f0e1..7b8dc38c 100644
>>> --- a/meta/classes/imagetypes_wic.bbclass
>>> +++ b/meta/classes/imagetypes_wic.bbclass
>>> @@ -157,6 +157,7 @@ generate_wic_image() {
>>>      export FAKEROOTCMD=${FAKEROOTCMD}
>>>      export BUILDDIR=${TOPDIR}
>>>      export MTOOLS_SKIP_CHECK=1
>>> +    export PYTHONPYCACHEPREFIX="${TOPDIR}/__pycache__"
> 
> This script is not called frequently, so there is no real benefit in
> caching it. If we want to cache, we should probably cache in the
> WORKDIR.
> 
>>>      mkdir -p ${IMAGE_ROOTFS}/../pseudo
>>>      touch ${IMAGE_ROOTFS}/../pseudo/files.db```
>>>
>>> and it seems to do the job. In depth testing still required!
>>>
>>> [1]:
>>> https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPYCACHEPREFIX
>>>
>>
>> What's yocto doing about this BTW? Are we just missing some of their
>> bits, or are they affected by a similar issue (except for creating
>> the
>> cache as root)?
> 
> Yocto also disables the python cache for their fakeroot environments:
> 
> https://github.com/openembedded/openembedded-core/blob/bd8fc4c59a137a37bd7a54f398949617982d447e/meta/conf/bitbake.conf#L753
> 
> But as isar differs here, we either need to add this statement where I
> added it (take the patch as-is), or implement this logic in
> `wic_fakeroot`.

We have more python code than wic in isar. You should then make sure to
disable caching in general, not just for wic.

Jan

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/ff6c120d-a108-4001-bc60-c64af2d4f2d3%40siemens.com.

Re: [PATCH 1/1] wic: disable generation of python bytecode cache

['Heinisch, Alexander' via isar-users]

On Fri, 2025-06-06 at 09:38 +0200, Jan Kiszka wrote:
> On 06.06.25 09:18, Moessbauer, Felix (FT RPD CED OES-DE) wrote:
> > On Fri, 2025-06-06 at 08:21 +0200, Jan Kiszka wrote:
> > > On 06.06.25 08:19, Heinisch, Alexander (FT RPD CED SES-AT) wrote:
> > > > On Thu, 2025-06-05 at 16:58 +0200, 'Felix Moessbauer' via isar-
> > > > users
> > > > wrote:
> > > > > Wic is executed as root, so the python bytecode cache is also
> > > > > created as root. This is problematic as the cache is created
> > > > > inside
> > > > > the
> > > > > source tree in a folder__pycache__ next to the python script
> > > > > itself.
> > > > > By
> > > > > that, we end up with files belonging to root inside the repo
> > > > > source
> > > > > tree, which makes it impossible to delete the source tree as
> > > > > a
> > > > > regular
> > > > > user.
> > > > >
> > > > > This problem became visible with the kas purge plugin that
> > > > > removes
> > > > > the
> > > > > fetched layers as a regular user (these layers are fetched
> > > > > and
> > > > > managed
> > > > > by kas). Also the read-only mounting of repos in kas does not
> > > > > help
> > > > > here,
> > > > > as the fetched repos are not mounted ro for obvious reasons.
> > > > >
> > > > > Anyways, we should not create files inside the source tree

Probably that's already the root cause :-)

> > > > >
> > > > > that
> > > > > do
> > > > > not
> > > > > belong to the calling user. To fix this, we just disable the
> > > > > python
> > > > > cache for the wic task. This is the only task that executes a
> > > > > python
> > > > > script from the source tree as root.
> > > >
> > > > Hi Felix,
> > > >
> > > > What about moving the cache to the build dir?
> > > >
> > > > I just did a quick test using `PYTHONPYCACHEPREFIX` (see [1])
> > > > which
> > > > was
> > > > added in Python 3.8 (so works for hosts >= bullseye)
> > > >
> > > > ```
> > > > diff --git a/meta/classes/imagetypes_wic.bbclass
> > > > b/meta/classes/imagetypes_wic.bbclass
> > > > index 38b5f0e1..7b8dc38c 100644
> > > > --- a/meta/classes/imagetypes_wic.bbclass
> > > > +++ b/meta/classes/imagetypes_wic.bbclass
> > > > @@ -157,6 +157,7 @@ generate_wic_image() {
> > > >      export FAKEROOTCMD=${FAKEROOTCMD}
> > > >      export BUILDDIR=${TOPDIR}
> > > >      export MTOOLS_SKIP_CHECK=1
> > > > +    export PYTHONPYCACHEPREFIX="${TOPDIR}/__pycache__"
> >
> > This script is not called frequently, so there is no real benefit
> > in
> > caching it. If we want to cache, we should probably cache in the
> > WORKDIR.

The chosen directory was just for a quick PoC.

> >
> > > >      mkdir -p ${IMAGE_ROOTFS}/../pseudo
> > > >      touch ${IMAGE_ROOTFS}/../pseudo/files.db```
> > > >
> > > > and it seems to do the job. In depth testing still required!
> > > >
> > > > [1]:
> > > > https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPYCACHEPREFIX
> > > >
> > >
> > > What's yocto doing about this BTW? Are we just missing some of
> > > their
> > > bits, or are they affected by a similar issue (except for
> > > creating
> > > the
> > > cache as root)?
> >
> > Yocto also disables the python cache for their fakeroot
> > environments:
> >
> > https://github.com/openembedded/openembedded-core/blob/bd8fc4c59a137a37bd7a54f398949617982d447e/meta/conf/bitbake.conf#L753
> >
> > But as isar differs here, we either need to add this statement
> > where I
> > added it (take the patch as-is), or implement this logic in
> > `wic_fakeroot`.
>
> We have more python code than wic in isar. You should then make sure
> to
> disable caching in general, not just for wic.

Maybe we should investigate in not adding/modifying files in the source
tree at all? While most buildsystems provide some kind of PREFIX, we
could put all generated artifacts someplace below TOPDIR.
(Same applies to yocto)

>
> Jan
>
BR Alexander

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/ad4f0636ecf99e992b30b9d484a34124c9f04ab4.camel%40siemens.com.

Re: [PATCH 1/1] wic: disable generation of python bytecode cache

['MOESSBAUER, Felix' via isar-users]

On Fri, 2025-06-06 at 07:42 +0000, Heinisch, Alexander (FT RPD CED SES-
AT) wrote:
> On Fri, 2025-06-06 at 09:38 +0200, Jan Kiszka wrote:
> > On 06.06.25 09:18, Moessbauer, Felix (FT RPD CED OES-DE) wrote:
> > > On Fri, 2025-06-06 at 08:21 +0200, Jan Kiszka wrote:
> > > > On 06.06.25 08:19, Heinisch, Alexander (FT RPD CED SES-AT)
> > > > wrote:
> > > > > On Thu, 2025-06-05 at 16:58 +0200, 'Felix Moessbauer' via
> > > > > isar-
> > > > > users
> > > > > wrote:
> > > > > > Wic is executed as root, so the python bytecode cache is
> > > > > > also
> > > > > > created as root. This is problematic as the cache is
> > > > > > created
> > > > > > inside
> > > > > > the
> > > > > > source tree in a folder__pycache__ next to the python
> > > > > > script
> > > > > > itself.
> > > > > > By
> > > > > > that, we end up with files belonging to root inside the
> > > > > > repo
> > > > > > source
> > > > > > tree, which makes it impossible to delete the source tree
> > > > > > as
> > > > > > a
> > > > > > regular
> > > > > > user.
> > > > > > 
> > > > > > This problem became visible with the kas purge plugin that
> > > > > > removes
> > > > > > the
> > > > > > fetched layers as a regular user (these layers are fetched
> > > > > > and
> > > > > > managed
> > > > > > by kas). Also the read-only mounting of repos in kas does
> > > > > > not
> > > > > > help
> > > > > > here,
> > > > > > as the fetched repos are not mounted ro for obvious
> > > > > > reasons.
> > > > > > 
> > > > > > Anyways, we should not create files inside the source tree
> 
> Probably that's already the root cause :-)
> 
> > > > > > 
> > > > > > that
> > > > > > do
> > > > > > not
> > > > > > belong to the calling user. To fix this, we just disable
> > > > > > the
> > > > > > python
> > > > > > cache for the wic task. This is the only task that executes
> > > > > > a
> > > > > > python
> > > > > > script from the source tree as root.
> > > > > 
> > > > > Hi Felix,
> > > > > 
> > > > > What about moving the cache to the build dir?
> > > > > 
> > > > > I just did a quick test using `PYTHONPYCACHEPREFIX` (see [1])
> > > > > which
> > > > > was
> > > > > added in Python 3.8 (so works for hosts >= bullseye)
> > > > > 
> > > > > ```
> > > > > diff --git a/meta/classes/imagetypes_wic.bbclass
> > > > > b/meta/classes/imagetypes_wic.bbclass
> > > > > index 38b5f0e1..7b8dc38c 100644
> > > > > --- a/meta/classes/imagetypes_wic.bbclass
> > > > > +++ b/meta/classes/imagetypes_wic.bbclass
> > > > > @@ -157,6 +157,7 @@ generate_wic_image() {
> > > > >      export FAKEROOTCMD=${FAKEROOTCMD}
> > > > >      export BUILDDIR=${TOPDIR}
> > > > >      export MTOOLS_SKIP_CHECK=1
> > > > > +    export PYTHONPYCACHEPREFIX="${TOPDIR}/__pycache__"
> > > 
> > > This script is not called frequently, so there is no real benefit
> > > in
> > > caching it. If we want to cache, we should probably cache in the
> > > WORKDIR.
> 
> The chosen directory was just for a quick PoC.
> 
> > > 
> > > > >      mkdir -p ${IMAGE_ROOTFS}/../pseudo
> > > > >      touch ${IMAGE_ROOTFS}/../pseudo/files.db```
> > > > > 
> > > > > and it seems to do the job. In depth testing still required!
> > > > > 
> > > > > [1]:
> > > > > https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPYCACHEPREFIX
> > > > > 
> > > > 
> > > > What's yocto doing about this BTW? Are we just missing some of
> > > > their
> > > > bits, or are they affected by a similar issue (except for
> > > > creating
> > > > the
> > > > cache as root)?
> > > 
> > > Yocto also disables the python cache for their fakeroot
> > > environments:
> > > 
> > > https://github.com/openembedded/openembedded-core/blob/bd8fc4c59a137a37bd7a54f398949617982d447e/meta/conf/bitbake.conf#L753
> > > 
> > > But as isar differs here, we either need to add this statement
> > > where I
> > > added it (take the patch as-is), or implement this logic in
> > > `wic_fakeroot`.
> > 
> > We have more python code than wic in isar. You should then make
> > sure
> > to
> > disable caching in general, not just for wic.

We have, but no more python code that is called as root.
We know that, as we exactly know which files in the source tree cannot
be deleted after a build.

> 
> Maybe we should investigate in not adding/modifying files in the
> source
> tree at all? While most buildsystems provide some kind of PREFIX, we
> could put all generated artifacts someplace below TOPDIR.
> (Same applies to yocto)

Yes, but this should be aligned with Yocto as they also only fix the
issue for root environments (or fakeroot in their case). That's a far
bigger topic than what this patch fixes.

Felix

> 
> > 
> > Jan
> > 
> BR Alexander

-- 
Siemens AG
Linux Expert Center
Friedrich-Ludwig-Bauer-Str. 3
85748 Garching, Germany

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/d7566fce4e99fea1ed5ae4ef68d43b850d4ee1bf.camel%40siemens.com.

Re: [PATCH 1/1] wic: disable generation of python bytecode cache

['Heinisch, Alexander' via isar-users]

On Fri, 2025-06-06 at 07:54 +0000, Moessbauer, Felix (FT RPD CED OES-
DE) wrote:
> On Fri, 2025-06-06 at 07:42 +0000, Heinisch, Alexander (FT RPD CED
> SES-
> AT) wrote:
> > On Fri, 2025-06-06 at 09:38 +0200, Jan Kiszka wrote:
> > > On 06.06.25 09:18, Moessbauer, Felix (FT RPD CED OES-DE) wrote:
> > > > On Fri, 2025-06-06 at 08:21 +0200, Jan Kiszka wrote:
> > > > > On 06.06.25 08:19, Heinisch, Alexander (FT RPD CED SES-AT)
> > > > > wrote:
> > > > > > On Thu, 2025-06-05 at 16:58 +0200, 'Felix Moessbauer' via
> > > > > > isar-
> > > > > > users
> > > > > > wrote:
> > > > > > > Wic is executed as root, so the python bytecode cache is
> > > > > > > also
> > > > > > > created as root. This is problematic as the cache is
> > > > > > > created
> > > > > > > inside
> > > > > > > the
> > > > > > > source tree in a folder__pycache__ next to the python
> > > > > > > script
> > > > > > > itself.
> > > > > > > By
> > > > > > > that, we end up with files belonging to root inside the
> > > > > > > repo
> > > > > > > source
> > > > > > > tree, which makes it impossible to delete the source tree
> > > > > > > as
> > > > > > > a
> > > > > > > regular
> > > > > > > user.
> > > > > > >
> > > > > > > This problem became visible with the kas purge plugin
> > > > > > > that
> > > > > > > removes
> > > > > > > the
> > > > > > > fetched layers as a regular user (these layers are
> > > > > > > fetched
> > > > > > > and
> > > > > > > managed
> > > > > > > by kas). Also the read-only mounting of repos in kas does
> > > > > > > not
> > > > > > > help
> > > > > > > here,
> > > > > > > as the fetched repos are not mounted ro for obvious
> > > > > > > reasons.
> > > > > > >
> > > > > > > Anyways, we should not create files inside the source
> > > > > > > tree
> >
> > Probably that's already the root cause :-)
> >
> > > > > > >
> > > > > > > that
> > > > > > > do
> > > > > > > not
> > > > > > > belong to the calling user. To fix this, we just disable
> > > > > > > the
> > > > > > > python
> > > > > > > cache for the wic task. This is the only task that
> > > > > > > executes
> > > > > > > a
> > > > > > > python
> > > > > > > script from the source tree as root.
> > > > > >
> > > > > > Hi Felix,
> > > > > >
> > > > > > What about moving the cache to the build dir?
> > > > > >
> > > > > > I just did a quick test using `PYTHONPYCACHEPREFIX` (see
> > > > > > [1])
> > > > > > which
> > > > > > was
> > > > > > added in Python 3.8 (so works for hosts >= bullseye)
> > > > > >
> > > > > > ```
> > > > > > diff --git a/meta/classes/imagetypes_wic.bbclass
> > > > > > b/meta/classes/imagetypes_wic.bbclass
> > > > > > index 38b5f0e1..7b8dc38c 100644
> > > > > > --- a/meta/classes/imagetypes_wic.bbclass
> > > > > > +++ b/meta/classes/imagetypes_wic.bbclass
> > > > > > @@ -157,6 +157,7 @@ generate_wic_image() {
> > > > > >      export FAKEROOTCMD=${FAKEROOTCMD}
> > > > > >      export BUILDDIR=${TOPDIR}
> > > > > >      export MTOOLS_SKIP_CHECK=1
> > > > > > +    export PYTHONPYCACHEPREFIX="${TOPDIR}/__pycache__"
> > > >
> > > > This script is not called frequently, so there is no real
> > > > benefit
> > > > in
> > > > caching it. If we want to cache, we should probably cache in
> > > > the
> > > > WORKDIR.
> >
> > The chosen directory was just for a quick PoC.
> >
> > > >
> > > > > >      mkdir -p ${IMAGE_ROOTFS}/../pseudo
> > > > > >      touch ${IMAGE_ROOTFS}/../pseudo/files.db```
> > > > > >
> > > > > > and it seems to do the job. In depth testing still
> > > > > > required!
> > > > > >
> > > > > > [1]:
> > > > > > https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPYCACHEPREFIX
> > > > > >
> > > > >
> > > > > What's yocto doing about this BTW? Are we just missing some
> > > > > of
> > > > > their
> > > > > bits, or are they affected by a similar issue (except for
> > > > > creating
> > > > > the
> > > > > cache as root)?
> > > >
> > > > Yocto also disables the python cache for their fakeroot
> > > > environments:
> > > >
> > > > https://github.com/openembedded/openembedded-core/blob/bd8fc4c59a137a37bd7a54f398949617982d447e/meta/conf/bitbake.conf#L753
> > > >
> > > > But as isar differs here, we either need to add this statement
> > > > where I
> > > > added it (take the patch as-is), or implement this logic in
> > > > `wic_fakeroot`.
> > >
> > > We have more python code than wic in isar. You should then make
> > > sure
> > > to
> > > disable caching in general, not just for wic.
>
> We have, but no more python code that is called as root.
> We know that, as we exactly know which files in the source tree
> cannot
> be deleted after a build.
>
> >
> > Maybe we should investigate in not adding/modifying files in the
> > source
> > tree at all? While most buildsystems provide some kind of PREFIX,
> > we
> > could put all generated artifacts someplace below TOPDIR.
> > (Same applies to yocto)
>
> Yes, but this should be aligned with Yocto as they also only fix the
> issue for root environments (or fakeroot in their case). That's a far
> bigger topic than what this patch fixes.
Yes and yes!

Most probably nothing to fix in a single patch, rather a long term
strategy. (Iff we decide to do so (open for discussion) and if we could
align with yocto)

>
> Felix
>
> >
> > >
> > > Jan
> > >
> > BR Alexander
>

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/8b12aa59e8218fbebe0097018b6781697d42e959.camel%40siemens.com.