Re: [PATCH 1/1] wic: disable generation of python bytecode cache

["'Heinisch, Alexander' via isar-users" <isar-users@googlegroups.com>]

On Fri, 2025-06-06 at 09:38 +0200, Jan Kiszka wrote:
> On 06.06.25 09:18, Moessbauer, Felix (FT RPD CED OES-DE) wrote:
> > On Fri, 2025-06-06 at 08:21 +0200, Jan Kiszka wrote:
> > > On 06.06.25 08:19, Heinisch, Alexander (FT RPD CED SES-AT) wrote:
> > > > On Thu, 2025-06-05 at 16:58 +0200, 'Felix Moessbauer' via isar-
> > > > users
> > > > wrote:
> > > > > Wic is executed as root, so the python bytecode cache is also
> > > > > created as root. This is problematic as the cache is created
> > > > > inside
> > > > > the
> > > > > source tree in a folder__pycache__ next to the python script
> > > > > itself.
> > > > > By
> > > > > that, we end up with files belonging to root inside the repo
> > > > > source
> > > > > tree, which makes it impossible to delete the source tree as
> > > > > a
> > > > > regular
> > > > > user.
> > > > >
> > > > > This problem became visible with the kas purge plugin that
> > > > > removes
> > > > > the
> > > > > fetched layers as a regular user (these layers are fetched
> > > > > and
> > > > > managed
> > > > > by kas). Also the read-only mounting of repos in kas does not
> > > > > help
> > > > > here,
> > > > > as the fetched repos are not mounted ro for obvious reasons.
> > > > >
> > > > > Anyways, we should not create files inside the source tree

Probably that's already the root cause :-)

> > > > >
> > > > > that
> > > > > do
> > > > > not
> > > > > belong to the calling user. To fix this, we just disable the
> > > > > python
> > > > > cache for the wic task. This is the only task that executes a
> > > > > python
> > > > > script from the source tree as root.
> > > >
> > > > Hi Felix,
> > > >
> > > > What about moving the cache to the build dir?
> > > >
> > > > I just did a quick test using `PYTHONPYCACHEPREFIX` (see [1])
> > > > which
> > > > was
> > > > added in Python 3.8 (so works for hosts >= bullseye)
> > > >
> > > > ```
> > > > diff --git a/meta/classes/imagetypes_wic.bbclass
> > > > b/meta/classes/imagetypes_wic.bbclass
> > > > index 38b5f0e1..7b8dc38c 100644
> > > > --- a/meta/classes/imagetypes_wic.bbclass
> > > > +++ b/meta/classes/imagetypes_wic.bbclass
> > > > @@ -157,6 +157,7 @@ generate_wic_image() {
> > > >      export FAKEROOTCMD=${FAKEROOTCMD}
> > > >      export BUILDDIR=${TOPDIR}
> > > >      export MTOOLS_SKIP_CHECK=1
> > > > +    export PYTHONPYCACHEPREFIX="${TOPDIR}/__pycache__"
> >
> > This script is not called frequently, so there is no real benefit
> > in
> > caching it. If we want to cache, we should probably cache in the
> > WORKDIR.

The chosen directory was just for a quick PoC.

> >
> > > >      mkdir -p ${IMAGE_ROOTFS}/../pseudo
> > > >      touch ${IMAGE_ROOTFS}/../pseudo/files.db```
> > > >
> > > > and it seems to do the job. In depth testing still required!
> > > >
> > > > [1]:
> > > > https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPYCACHEPREFIX
> > > >
> > >
> > > What's yocto doing about this BTW? Are we just missing some of
> > > their
> > > bits, or are they affected by a similar issue (except for
> > > creating
> > > the
> > > cache as root)?
> >
> > Yocto also disables the python cache for their fakeroot
> > environments:
> >
> > https://github.com/openembedded/openembedded-core/blob/bd8fc4c59a137a37bd7a54f398949617982d447e/meta/conf/bitbake.conf#L753
> >
> > But as isar differs here, we either need to add this statement
> > where I
> > added it (take the patch as-is), or implement this logic in
> > `wic_fakeroot`.
>
> We have more python code than wic in isar. You should then make sure
> to
> disable caching in general, not just for wic.

Maybe we should investigate in not adding/modifying files in the source
tree at all? While most buildsystems provide some kind of PREFIX, we
could put all generated artifacts someplace below TOPDIR.
(Same applies to yocto)

>
> Jan
>
BR Alexander

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/ad4f0636ecf99e992b30b9d484a34124c9f04ab4.camel%40siemens.com.