Re: [PATCH 1/1] wic: disable generation of python bytecode cache

["'Heinisch, Alexander' via isar-users" <isar-users@googlegroups.com>]

On Fri, 2025-06-06 at 07:54 +0000, Moessbauer, Felix (FT RPD CED OES-
DE) wrote:
> On Fri, 2025-06-06 at 07:42 +0000, Heinisch, Alexander (FT RPD CED
> SES-
> AT) wrote:
> > On Fri, 2025-06-06 at 09:38 +0200, Jan Kiszka wrote:
> > > On 06.06.25 09:18, Moessbauer, Felix (FT RPD CED OES-DE) wrote:
> > > > On Fri, 2025-06-06 at 08:21 +0200, Jan Kiszka wrote:
> > > > > On 06.06.25 08:19, Heinisch, Alexander (FT RPD CED SES-AT)
> > > > > wrote:
> > > > > > On Thu, 2025-06-05 at 16:58 +0200, 'Felix Moessbauer' via
> > > > > > isar-
> > > > > > users
> > > > > > wrote:
> > > > > > > Wic is executed as root, so the python bytecode cache is
> > > > > > > also
> > > > > > > created as root. This is problematic as the cache is
> > > > > > > created
> > > > > > > inside
> > > > > > > the
> > > > > > > source tree in a folder__pycache__ next to the python
> > > > > > > script
> > > > > > > itself.
> > > > > > > By
> > > > > > > that, we end up with files belonging to root inside the
> > > > > > > repo
> > > > > > > source
> > > > > > > tree, which makes it impossible to delete the source tree
> > > > > > > as
> > > > > > > a
> > > > > > > regular
> > > > > > > user.
> > > > > > >
> > > > > > > This problem became visible with the kas purge plugin
> > > > > > > that
> > > > > > > removes
> > > > > > > the
> > > > > > > fetched layers as a regular user (these layers are
> > > > > > > fetched
> > > > > > > and
> > > > > > > managed
> > > > > > > by kas). Also the read-only mounting of repos in kas does
> > > > > > > not
> > > > > > > help
> > > > > > > here,
> > > > > > > as the fetched repos are not mounted ro for obvious
> > > > > > > reasons.
> > > > > > >
> > > > > > > Anyways, we should not create files inside the source
> > > > > > > tree
> >
> > Probably that's already the root cause :-)
> >
> > > > > > >
> > > > > > > that
> > > > > > > do
> > > > > > > not
> > > > > > > belong to the calling user. To fix this, we just disable
> > > > > > > the
> > > > > > > python
> > > > > > > cache for the wic task. This is the only task that
> > > > > > > executes
> > > > > > > a
> > > > > > > python
> > > > > > > script from the source tree as root.
> > > > > >
> > > > > > Hi Felix,
> > > > > >
> > > > > > What about moving the cache to the build dir?
> > > > > >
> > > > > > I just did a quick test using `PYTHONPYCACHEPREFIX` (see
> > > > > > [1])
> > > > > > which
> > > > > > was
> > > > > > added in Python 3.8 (so works for hosts >= bullseye)
> > > > > >
> > > > > > ```
> > > > > > diff --git a/meta/classes/imagetypes_wic.bbclass
> > > > > > b/meta/classes/imagetypes_wic.bbclass
> > > > > > index 38b5f0e1..7b8dc38c 100644
> > > > > > --- a/meta/classes/imagetypes_wic.bbclass
> > > > > > +++ b/meta/classes/imagetypes_wic.bbclass
> > > > > > @@ -157,6 +157,7 @@ generate_wic_image() {
> > > > > >      export FAKEROOTCMD=${FAKEROOTCMD}
> > > > > >      export BUILDDIR=${TOPDIR}
> > > > > >      export MTOOLS_SKIP_CHECK=1
> > > > > > +    export PYTHONPYCACHEPREFIX="${TOPDIR}/__pycache__"
> > > >
> > > > This script is not called frequently, so there is no real
> > > > benefit
> > > > in
> > > > caching it. If we want to cache, we should probably cache in
> > > > the
> > > > WORKDIR.
> >
> > The chosen directory was just for a quick PoC.
> >
> > > >
> > > > > >      mkdir -p ${IMAGE_ROOTFS}/../pseudo
> > > > > >      touch ${IMAGE_ROOTFS}/../pseudo/files.db```
> > > > > >
> > > > > > and it seems to do the job. In depth testing still
> > > > > > required!
> > > > > >
> > > > > > [1]:
> > > > > > https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPYCACHEPREFIX
> > > > > >
> > > > >
> > > > > What's yocto doing about this BTW? Are we just missing some
> > > > > of
> > > > > their
> > > > > bits, or are they affected by a similar issue (except for
> > > > > creating
> > > > > the
> > > > > cache as root)?
> > > >
> > > > Yocto also disables the python cache for their fakeroot
> > > > environments:
> > > >
> > > > https://github.com/openembedded/openembedded-core/blob/bd8fc4c59a137a37bd7a54f398949617982d447e/meta/conf/bitbake.conf#L753
> > > >
> > > > But as isar differs here, we either need to add this statement
> > > > where I
> > > > added it (take the patch as-is), or implement this logic in
> > > > `wic_fakeroot`.
> > >
> > > We have more python code than wic in isar. You should then make
> > > sure
> > > to
> > > disable caching in general, not just for wic.
>
> We have, but no more python code that is called as root.
> We know that, as we exactly know which files in the source tree
> cannot
> be deleted after a build.
>
> >
> > Maybe we should investigate in not adding/modifying files in the
> > source
> > tree at all? While most buildsystems provide some kind of PREFIX,
> > we
> > could put all generated artifacts someplace below TOPDIR.
> > (Same applies to yocto)
>
> Yes, but this should be aligned with Yocto as they also only fix the
> issue for root environments (or fakeroot in their case). That's a far
> bigger topic than what this patch fixes.
Yes and yes!

Most probably nothing to fix in a single patch, rather a long term
strategy. (Iff we decide to do so (open for discussion) and if we could
align with yocto)

>
> Felix
>
> >
> > >
> > > Jan
> > >
> > BR Alexander
>

-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/8b12aa59e8218fbebe0097018b6781697d42e959.camel%40siemens.com.