RFC: add maximum debootstrap storing time, or check for updates and invalidate

RFC: add maximum debootstrap storing time, or check for updates and invalidate

[Henning Schild]

Hi,

Isar does have the problem that a debootstrap does the only real
"apt-get update". That is in fact intentional but can lead to the
situation where an outdated chrootfs can break a build. I.e. apt-get
wants to install a package in a version that is not available upstream
anymore.

Now when this happens is impossible to say. It depends on the amount of
packages one has, maybe the distro one is using (sid moving faster than
oldstable), and frozen custom mirror never moving.

With local builds people usually clean all, and for sstate people will
also have to find their own way of dealing with it. But one could say
... it is becoming more of a problem. Not something to not merge the
sstate changes for, but something to think about for the future. 

My current idea would be to introduce
DISTRO_MAX_CHROOT_AGE?="3d"

anon python function
  if /var/..apt/ older than DISTRO_MAX_AGE
    cleanall bootstrap recipe, or somehow enforce re-run

An alternative (maybe better) would be to say

check_for_updates after rootfs_install
  copy rootfs to rootfs_tmp
  chroot rootfs_tmp
  apt-get update
  if "apt-get upgrade is not empty"
    bbwarn(image lacking some updates that meanwhile have been
    published)
    cleanall bootstrap recipe
    if ISAR_BE_STRICT_ABOUT_THAT=true
      fail

Henning

Re: RFC: add maximum debootstrap storing time, or check for updates and invalidate

[Jan Kiszka]

On 04.11.21 16:12, Henning Schild wrote:
> Hi,
> 
> Isar does have the problem that a debootstrap does the only real
> "apt-get update". That is in fact intentional but can lead to the
> situation where an outdated chrootfs can break a build. I.e. apt-get
> wants to install a package in a version that is not available upstream
> anymore.
> 
> Now when this happens is impossible to say. It depends on the amount of
> packages one has, maybe the distro one is using (sid moving faster than
> oldstable), and frozen custom mirror never moving.
> 
> With local builds people usually clean all, and for sstate people will
> also have to find their own way of dealing with it. But one could say
> ... it is becoming more of a problem. Not something to not merge the
> sstate changes for, but something to think about for the future. 
> 
> My current idea would be to introduce
> DISTRO_MAX_CHROOT_AGE?="3d"
> 
> anon python function
>   if /var/..apt/ older than DISTRO_MAX_AGE
>     cleanall bootstrap recipe, or somehow enforce re-run
> 
> An alternative (maybe better) would be to say
> 
> check_for_updates after rootfs_install
>   copy rootfs to rootfs_tmp
>   chroot rootfs_tmp
>   apt-get update
>   if "apt-get upgrade is not empty"
>     bbwarn(image lacking some updates that meanwhile have been
>     published)
>     cleanall bootstrap recipe
>     if ISAR_BE_STRICT_ABOUT_THAT=true
>       fail
> 

It's all a trade-off: The above will only run when redoing a rootfs, not
when the buildchroot is considered "done" while a dependency
installation later on may still fail. And running such a check on every
rebuild, even when you only want to build a single package, not a
complete image, may take too much time.

I agree that some update check could be valuable, provided its
controllable whether it is executed or not. And bonus for a simpler way
of triggering the invalidation of all bootstrap tasks, faster than "rm
-rf tmp".

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: RFC: add maximum debootstrap storing time, or check for updates and invalidate

[Henning Schild]

Am Thu, 4 Nov 2021 16:25:02 +0100
schrieb Jan Kiszka <jan.kiszka@siemens.com>:

> On 04.11.21 16:12, Henning Schild wrote:
> > Hi,
> > 
> > Isar does have the problem that a debootstrap does the only real
> > "apt-get update". That is in fact intentional but can lead to the
> > situation where an outdated chrootfs can break a build. I.e. apt-get
> > wants to install a package in a version that is not available
> > upstream anymore.
> > 
> > Now when this happens is impossible to say. It depends on the
> > amount of packages one has, maybe the distro one is using (sid
> > moving faster than oldstable), and frozen custom mirror never
> > moving.
> > 
> > With local builds people usually clean all, and for sstate people
> > will also have to find their own way of dealing with it. But one
> > could say ... it is becoming more of a problem. Not something to
> > not merge the sstate changes for, but something to think about for
> > the future. 
> > 
> > My current idea would be to introduce
> > DISTRO_MAX_CHROOT_AGE?="3d"
> > 
> > anon python function
> >   if /var/..apt/ older than DISTRO_MAX_AGE
> >     cleanall bootstrap recipe, or somehow enforce re-run
> > 
> > An alternative (maybe better) would be to say
> > 
> > check_for_updates after rootfs_install
> >   copy rootfs to rootfs_tmp
> >   chroot rootfs_tmp
> >   apt-get update
> >   if "apt-get upgrade is not empty"
> >     bbwarn(image lacking some updates that meanwhile have been
> >     published)
> >     cleanall bootstrap recipe
> >     if ISAR_BE_STRICT_ABOUT_THAT=true
> >       fail
> >   
> 
> It's all a trade-off: The above will only run when redoing a rootfs,
> not when the buildchroot is considered "done" while a dependency
> installation later on may still fail. And running such a check on
> every rebuild, even when you only want to build a single package, not
> a complete image, may take too much time.

Right, we will also want that for the buildchroots ... but only after
the last package ... not sure how to hook that into the task chain, but
can likely be done

> I agree that some update check could be valuable, provided its
> controllable whether it is executed or not. And bonus for a simpler
> way of triggering the invalidation of all bootstrap tasks, faster
> than "rm -rf tmp".

Yes there needs to be control, on if it is run and if it just warns or
actually removes stuff.

Not sure when i will get around to doing that. Just wanted to discuss
the idea, and do not need to be the one writing the patches.

Henning
 
> Jan
>