[PATCH] Reinstall imager deps if package in IMAGER_INSTALL has changed.

[PATCH] Reinstall imager deps if package in IMAGER_INSTALL has changed.

[Uladzimir Bely]

When debugging packages in `isar-apt` that are in IMAGER_INSTALL
list, they happen not to be updated in the resulting image.

For example, in case of u-boot IMAGE_BOOT_FILES is used to generate
files for `/boot` partition. Buildchroot is currently used for
extracting required files from `u-boot`, `optee-os`, etc. These
packages are installed in buildchroot via IMAGER_INSTALL list.

Since modifications of the packages don't change their versions,
updated packages don't go to buildchroot, so previously used versions
are wrongly used. The following can be seen in log:

"<package> is already the newest version"

This patch forces reinstalling all the packages from IMAGER_INSTALL
list if some of them has changed.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/image-tools-extension.bbclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/image-tools-extension.bbclass b/meta/classes/image-tools-extension.bbclass
index 101704d0..640e1958 100644
--- a/meta/classes/image-tools-extension.bbclass
+++ b/meta/classes/image-tools-extension.bbclass
@@ -35,12 +35,12 @@ do_install_imager_deps() {
             -o APT::Get::List-Cleanup="0"
         apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y \
             --allow-unauthenticated --allow-downgrades --download-only install \
-            ${IMAGER_INSTALL}'
+            --reinstall ${IMAGER_INSTALL}'
 
     deb_dl_dir_export ${BUILDCHROOT_DIR} ${distro}
     sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \
         apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y \
             --allow-unauthenticated --allow-downgrades install \
-            ${IMAGER_INSTALL}'
+            --reinstall ${IMAGER_INSTALL}'
 }
 addtask install_imager_deps before do_image_tools
-- 
2.20.1

Re: [PATCH] Reinstall imager deps if package in IMAGER_INSTALL has changed.

[Henning Schild]

Am Mon, 30 Jan 2023 10:07:41 +0100
schrieb Uladzimir Bely <ubely@ilbers.de>:

> When debugging packages in `isar-apt` that are in IMAGER_INSTALL
> list, they happen not to be updated in the resulting image.
> 
> For example, in case of u-boot IMAGE_BOOT_FILES is used to generate
> files for `/boot` partition. Buildchroot is currently used for
> extracting required files from `u-boot`, `optee-os`, etc. These
> packages are installed in buildchroot via IMAGER_INSTALL list.
> 
> Since modifications of the packages don't change their versions,
> updated packages don't go to buildchroot, so previously used versions
> are wrongly used. The following can be seen in log:
> 
> "<package> is already the newest version"
> 
> This patch forces reinstalling all the packages from IMAGER_INSTALL
> list if some of them has changed.


That makes some sense anyhow. But we should really watch out for such
issues and demand version bumping when contents change. Maybe not for
every recipe. Or we should include all recipes for IMAGER_INSTALL into
the sstate input so the cache gets invalidated and the chroot gets
rebuild from scratch.

Henning

> 
> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> ---
>  meta/classes/image-tools-extension.bbclass | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/classes/image-tools-extension.bbclass
> b/meta/classes/image-tools-extension.bbclass index 101704d0..640e1958
> 100644 --- a/meta/classes/image-tools-extension.bbclass
> +++ b/meta/classes/image-tools-extension.bbclass
> @@ -35,12 +35,12 @@ do_install_imager_deps() {
>              -o APT::Get::List-Cleanup="0"
>          apt-get -o Debug::pkgProblemResolver=yes
> --no-install-recommends -y \ --allow-unauthenticated
> --allow-downgrades --download-only install \
> -            ${IMAGER_INSTALL}'
> +            --reinstall ${IMAGER_INSTALL}'
>  
>      deb_dl_dir_export ${BUILDCHROOT_DIR} ${distro}
>      sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \
>          apt-get -o Debug::pkgProblemResolver=yes
> --no-install-recommends -y \ --allow-unauthenticated
> --allow-downgrades install \
> -            ${IMAGER_INSTALL}'
> +            --reinstall ${IMAGER_INSTALL}'
>  }
>  addtask install_imager_deps before do_image_tools

Re: [PATCH] Reinstall imager deps if package in IMAGER_INSTALL has changed.

[Moessbauer, Felix]

On Mon, 2023-01-30 at 11:53 +0100, Henning Schild wrote:
> Am Mon, 30 Jan 2023 10:07:41 +0100
> schrieb Uladzimir Bely <ubely@ilbers.de>:
> 
> > When debugging packages in `isar-apt` that are in IMAGER_INSTALL
> > list, they happen not to be updated in the resulting image.
> > 
> > For example, in case of u-boot IMAGE_BOOT_FILES is used to generate
> > files for `/boot` partition. Buildchroot is currently used for
> > extracting required files from `u-boot`, `optee-os`, etc. These
> > packages are installed in buildchroot via IMAGER_INSTALL list.
> > 
> > Since modifications of the packages don't change their versions,
> > updated packages don't go to buildchroot, so previously used
> > versions
> > are wrongly used. The following can be seen in log:
> > 
> > "<package> is already the newest version"
> > 
> > This patch forces reinstalling all the packages from IMAGER_INSTALL
> > list if some of them has changed.
> 
> 
> That makes some sense anyhow. But we should really watch out for such
> issues and demand version bumping when contents change. Maybe not for
> every recipe. Or we should include all recipes for IMAGER_INSTALL
> into
> the sstate input so the cache gets invalidated and the chroot gets
> rebuild from scratch.

I agree for packages that we deploy. But especially the imaging
packages are often local-only and specific to the machine and
configuration. By that, we should not require versioning.

Regarding sstate: This is actually a rebuild-only issue. The sstate-
cache is fully transparent in this regard, as the bb hashes change in
case we change an imaging package (including all downstream hashes).

Felix

> 
> Henning
> 
> > 
> > Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> > ---
> >  meta/classes/image-tools-extension.bbclass | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> > 
> > diff --git a/meta/classes/image-tools-extension.bbclass
> > b/meta/classes/image-tools-extension.bbclass index
> > 101704d0..640e1958
> > 100644 --- a/meta/classes/image-tools-extension.bbclass
> > +++ b/meta/classes/image-tools-extension.bbclass
> > @@ -35,12 +35,12 @@ do_install_imager_deps() {
> >              -o APT::Get::List-Cleanup="0"
> >          apt-get -o Debug::pkgProblemResolver=yes
> > --no-install-recommends -y \ --allow-unauthenticated
> > --allow-downgrades --download-only install \
> > -            ${IMAGER_INSTALL}'
> > +            --reinstall ${IMAGER_INSTALL}'
> >  
> >      deb_dl_dir_export ${BUILDCHROOT_DIR} ${distro}
> >      sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \
> >          apt-get -o Debug::pkgProblemResolver=yes
> > --no-install-recommends -y \ --allow-unauthenticated
> > --allow-downgrades install \
> > -            ${IMAGER_INSTALL}'
> > +            --reinstall ${IMAGER_INSTALL}'
> >  }
> >  addtask install_imager_deps before do_image_tools
> 

Re: [PATCH] Reinstall imager deps if package in IMAGER_INSTALL has changed.

[Roberto A. Foglietta]

On Mon, 30 Jan 2023 at 13:28, Moessbauer, Felix
<felix.moessbauer@siemens.com> wrote:
>
> On Mon, 2023-01-30 at 11:53 +0100, Henning Schild wrote:
> > Am Mon, 30 Jan 2023 10:07:41 +0100
> > schrieb Uladzimir Bely <ubely@ilbers.de>:
> >
> > > When debugging packages in `isar-apt` that are in IMAGER_INSTALL
> > > list, they happen not to be updated in the resulting image.
> > >
> > > For example, in case of u-boot IMAGE_BOOT_FILES is used to generate
> > > files for `/boot` partition. Buildchroot is currently used for
> > > extracting required files from `u-boot`, `optee-os`, etc. These
> > > packages are installed in buildchroot via IMAGER_INSTALL list.
> > >
> > > Since modifications of the packages don't change their versions,
> > > updated packages don't go to buildchroot, so previously used
> > > versions
> > > are wrongly used. The following can be seen in log:
> > >
> > > "<package> is already the newest version"
> > >
> > > This patch forces reinstalling all the packages from IMAGER_INSTALL
> > > list if some of them has changed.
> >
> >
> > That makes some sense anyhow. But we should really watch out for such
> > issues and demand version bumping when contents change. Maybe not for
> > every recipe. Or we should include all recipes for IMAGER_INSTALL
> > into
> > the sstate input so the cache gets invalidated and the chroot gets
> > rebuild from scratch.
>
> I agree for packages that we deploy. But especially the imaging
> packages are often local-only and specific to the machine and
> configuration. By that, we should not require versioning.
>

I agree but because possible dependencies between our packages and
those are into the standard distribution,we would need to install all
IMAGER needs and then reinstall our deployments which does not change
very much but is slower in most of the cases. IMHO.

More in general, I think that we need to save - also in the sstate
cache - with our packages a md5sum signature just to have a way to
check if the package has been changed at binary level. This will work
better in conjunction with your patch for reproducibility and
moreover, adopting mount bind for the sbuild, it might give use a
local reproducibility to make the md5sum signature even more useful.
s/md5sum/shaNsum/g

Best regards, R-