Re: [PATCH 1/1] wic: disable generation of python bytecode cache

["'MOESSBAUER, Felix' via isar-users" <isar-users@googlegroups.com>]

On Fri, 2025-06-06 at 07:42 +0000, Heinisch, Alexander (FT RPD CED SES-
AT) wrote:
> On Fri, 2025-06-06 at 09:38 +0200, Jan Kiszka wrote:
> > On 06.06.25 09:18, Moessbauer, Felix (FT RPD CED OES-DE) wrote:
> > > On Fri, 2025-06-06 at 08:21 +0200, Jan Kiszka wrote:
> > > > On 06.06.25 08:19, Heinisch, Alexander (FT RPD CED SES-AT)
> > > > wrote:
> > > > > On Thu, 2025-06-05 at 16:58 +0200, 'Felix Moessbauer' via
> > > > > isar-
> > > > > users
> > > > > wrote:
> > > > > > Wic is executed as root, so the python bytecode cache is
> > > > > > also
> > > > > > created as root. This is problematic as the cache is
> > > > > > created
> > > > > > inside
> > > > > > the
> > > > > > source tree in a folder__pycache__ next to the python
> > > > > > script
> > > > > > itself.
> > > > > > By
> > > > > > that, we end up with files belonging to root inside the
> > > > > > repo
> > > > > > source
> > > > > > tree, which makes it impossible to delete the source tree
> > > > > > as
> > > > > > a
> > > > > > regular
> > > > > > user.
> > > > > > 
> > > > > > This problem became visible with the kas purge plugin that
> > > > > > removes
> > > > > > the
> > > > > > fetched layers as a regular user (these layers are fetched
> > > > > > and
> > > > > > managed
> > > > > > by kas). Also the read-only mounting of repos in kas does
> > > > > > not
> > > > > > help
> > > > > > here,
> > > > > > as the fetched repos are not mounted ro for obvious
> > > > > > reasons.
> > > > > > 
> > > > > > Anyways, we should not create files inside the source tree
> 
> Probably that's already the root cause :-)
> 
> > > > > > 
> > > > > > that
> > > > > > do
> > > > > > not
> > > > > > belong to the calling user. To fix this, we just disable
> > > > > > the
> > > > > > python
> > > > > > cache for the wic task. This is the only task that executes
> > > > > > a
> > > > > > python
> > > > > > script from the source tree as root.
> > > > > 
> > > > > Hi Felix,
> > > > > 
> > > > > What about moving the cache to the build dir?
> > > > > 
> > > > > I just did a quick test using `PYTHONPYCACHEPREFIX` (see [1])
> > > > > which
> > > > > was
> > > > > added in Python 3.8 (so works for hosts >= bullseye)
> > > > > 
> > > > > ```
> > > > > diff --git a/meta/classes/imagetypes_wic.bbclass
> > > > > b/meta/classes/imagetypes_wic.bbclass
> > > > > index 38b5f0e1..7b8dc38c 100644
> > > > > --- a/meta/classes/imagetypes_wic.bbclass
> > > > > +++ b/meta/classes/imagetypes_wic.bbclass
> > > > > @@ -157,6 +157,7 @@ generate_wic_image() {
> > > > >      export FAKEROOTCMD=${FAKEROOTCMD}
> > > > >      export BUILDDIR=${TOPDIR}
> > > > >      export MTOOLS_SKIP_CHECK=1
> > > > > +    export PYTHONPYCACHEPREFIX="${TOPDIR}/__pycache__"
> > > 
> > > This script is not called frequently, so there is no real benefit
> > > in
> > > caching it. If we want to cache, we should probably cache in the
> > > WORKDIR.
> 
> The chosen directory was just for a quick PoC.
> 
> > > 
> > > > >      mkdir -p ${IMAGE_ROOTFS}/../pseudo
> > > > >      touch ${IMAGE_ROOTFS}/../pseudo/files.db```
> > > > > 
> > > > > and it seems to do the job. In depth testing still required!
> > > > > 
> > > > > [1]:
> > > > > https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPYCACHEPREFIX
> > > > > 
> > > > 
> > > > What's yocto doing about this BTW? Are we just missing some of
> > > > their
> > > > bits, or are they affected by a similar issue (except for
> > > > creating
> > > > the
> > > > cache as root)?
> > > 
> > > Yocto also disables the python cache for their fakeroot
> > > environments:
> > > 
> > > https://github.com/openembedded/openembedded-core/blob/bd8fc4c59a137a37bd7a54f398949617982d447e/meta/conf/bitbake.conf#L753
> > > 
> > > But as isar differs here, we either need to add this statement
> > > where I
> > > added it (take the patch as-is), or implement this logic in
> > > `wic_fakeroot`.
> > 
> > We have more python code than wic in isar. You should then make
> > sure
> > to
> > disable caching in general, not just for wic.

We have, but no more python code that is called as root.
We know that, as we exactly know which files in the source tree cannot
be deleted after a build.

> 
> Maybe we should investigate in not adding/modifying files in the
> source
> tree at all? While most buildsystems provide some kind of PREFIX, we
> could put all generated artifacts someplace below TOPDIR.
> (Same applies to yocto)

Yes, but this should be aligned with Yocto as they also only fix the
issue for root environments (or fakeroot in their case). That's a far
bigger topic than what this patch fixes.

Felix

> 
> > 
> > Jan
> > 
> BR Alexander

-- 
Siemens AG
Linux Expert Center
Friedrich-Ludwig-Bauer-Str. 3
85748 Garching, Germany

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/d7566fce4e99fea1ed5ae4ef68d43b850d4ee1bf.camel%40siemens.com.