Rebuild issues in imaging steps with local packages

Rebuild issues in imaging steps with local packages

[Moessbauer, Felix]

Dear Devs,

I just observed a nasty rebuild issue in the imaging step:

When building a custom package that is later used in the IMAGER_INSTALL
and IMAGER_BUILD_DEPS, this package does ONLY get updated on version
changes. This can be checked by inspecting log.do_install_imager_deps
and checking for a line "<package> is already the newest version".

I found this while debugging why a fresh build of the "Add BSP for
StarFiveTech VisionFive2 Risc-V board" (v1) series on the ML is broken
for another user. The reason was simply that I optimized it locally and
made changes to the visionfive2-u-boot-firmware, which were not
included in the final image.

This behavior is different from how packages in the rootfs are handled,
but I did not debug yet why this is different. Maybe Adriaan can help.
It could also well be, that this issue is automatically fixed when
moving the imaging to the schroot.

Best regards,
Felix

Re: Rebuild issues in imaging steps with local packages

[Uladzimir Bely]

In mail from Monday, 30 January 2023 06:24:37 +03 user Moessbauer, Felix 
wrote:
> Dear Devs,
> 
> I just observed a nasty rebuild issue in the imaging step:
> 
> When building a custom package that is later used in the IMAGER_INSTALL
> and IMAGER_BUILD_DEPS, this package does ONLY get updated on version
> changes. This can be checked by inspecting log.do_install_imager_deps
> and checking for a line "<package> is already the newest version".
> 
> I found this while debugging why a fresh build of the "Add BSP for
> StarFiveTech VisionFive2 Risc-V board" (v1) series on the ML is broken
> for another user. The reason was simply that I optimized it locally and
> made changes to the visionfive2-u-boot-firmware, which were not
> included in the final image.
> 
> This behavior is different from how packages in the rootfs are handled,
> but I did not debug yet why this is different. Maybe Adriaan can help.
> It could also well be, that this issue is automatically fixed when
> moving the imaging to the schroot.
> 
> Best regards,
> Felix


Hello.

We have a similar issue in our internal bug-tracking system, discovered 
recently. In short:

- Image is built 1st time.
- Some patch is applied to U-Boot
- Image is rebuilt but doesn't contain the _updated_ U-Boot package due to the 
same version number comparing to the 1st build.

It happend only with u-boot packages since it is not installed into the output 
rootfs (boot partition has another source), but is handled in buildchroot.

Quick fix with adding '--reinstall' in imager "apt-get install" routines 
worked well, so it may be used for debugging.

Moving imager to schroot (it's under development right now) should also help 
to solve the issue.

Re: Rebuild issues in imaging steps with local packages

[Moessbauer, Felix]

On Mon, 2023-01-30 at 09:43 +0300, Uladzimir Bely wrote:
> In mail from Monday, 30 January 2023 06:24:37 +03 user Moessbauer,
> Felix 
> wrote:
> > Dear Devs,
> > 
> > I just observed a nasty rebuild issue in the imaging step:
> > 
> > When building a custom package that is later used in the
> > IMAGER_INSTALL
> > and IMAGER_BUILD_DEPS, this package does ONLY get updated on
> > version
> > changes. This can be checked by inspecting
> > log.do_install_imager_deps
> > and checking for a line "<package> is already the newest version".
> > 
> > I found this while debugging why a fresh build of the "Add BSP for
> > StarFiveTech VisionFive2 Risc-V board" (v1) series on the ML is
> > broken
> > for another user. The reason was simply that I optimized it locally
> > and
> > made changes to the visionfive2-u-boot-firmware, which were not
> > included in the final image.
> > 
> > This behavior is different from how packages in the rootfs are
> > handled,
> > but I did not debug yet why this is different. Maybe Adriaan can
> > help.
> > It could also well be, that this issue is automatically fixed when
> > moving the imaging to the schroot.
> > 
> > Best regards,
> > Felix
> 
> 
> Hello.
> 
> We have a similar issue in our internal bug-tracking system,
> discovered 
> recently. In short:
> 
> - Image is built 1st time.
> - Some patch is applied to U-Boot
> - Image is rebuilt but doesn't contain the _updated_ U-Boot package
> due to the 
> same version number comparing to the 1st build.
> 
> It happend only with u-boot packages since it is not installed into
> the output 
> rootfs (boot partition has another source), but is handled in
> buildchroot.

Yes, that's exactly the same issue. For me it is the "visionfive2-u-
boot-firmware" package from the mentioned series, that contains the u-
boot FIT image for this board (check Patch 7/7 for details).

> 
> Quick fix with adding '--reinstall' in imager "apt-get install"
> routines 
> worked well, so it may be used for debugging.

While we don't have the schroot based implementation, can we please
integrate that hotfix into current ISAR? This bug is really nasty and
hard to debug as it will likely only show / reproduce on the target.

Felix

> 
> Moving imager to schroot (it's under development right now) should
> also help 
> to solve the issue.
> 
> 
> 

Re: Rebuild issues in imaging steps with local packages

[Uladzimir Bely]

In mail from Monday, 30 January 2023 10:08:54 +03 user Moessbauer, Felix 
wrote:
> On Mon, 2023-01-30 at 09:43 +0300, Uladzimir Bely wrote:
> 
> > In mail from Monday, 30 January 2023 06:24:37 +03 user Moessbauer,
> > Felix 
> > wrote:
> > 
> > > Dear Devs,
> > > 
> > > I just observed a nasty rebuild issue in the imaging step:
> > > 
> > > When building a custom package that is later used in the
> > > IMAGER_INSTALL
> > > and IMAGER_BUILD_DEPS, this package does ONLY get updated on
> > > version
> > > changes. This can be checked by inspecting
> > > log.do_install_imager_deps
> > > and checking for a line "<package> is already the newest version".
> > > 
> > > I found this while debugging why a fresh build of the "Add BSP for
> > > StarFiveTech VisionFive2 Risc-V board" (v1) series on the ML is
> > > broken
> > > for another user. The reason was simply that I optimized it locally
> > > and
> > > made changes to the visionfive2-u-boot-firmware, which were not
> > > included in the final image.
> > > 
> > > This behavior is different from how packages in the rootfs are
> > > handled,
> > > but I did not debug yet why this is different. Maybe Adriaan can
> > > help.
> > > It could also well be, that this issue is automatically fixed when
> > > moving the imaging to the schroot.
> > > 
> > > Best regards,
> > > Felix
> > 
> > 
> > 
> > Hello.
> > 
> > We have a similar issue in our internal bug-tracking system,
> > discovered 
> > recently. In short:
> > 
> > - Image is built 1st time.
> > - Some patch is applied to U-Boot
> > - Image is rebuilt but doesn't contain the _updated_ U-Boot package
> > due to the 
> > same version number comparing to the 1st build.
> > 
> > It happend only with u-boot packages since it is not installed into
> > the output 
> > rootfs (boot partition has another source), but is handled in
> > buildchroot.
> 
> 
> Yes, that's exactly the same issue. For me it is the "visionfive2-u-
> boot-firmware" package from the mentioned series, that contains the u-
> boot FIT image for this board (check Patch 7/7 for details).
> 
> 
> > 
> > Quick fix with adding '--reinstall' in imager "apt-get install"
> > routines 
> > worked well, so it may be used for debugging.
> 
> 
> While we don't have the schroot based implementation, can we please
> integrate that hotfix into current ISAR? This bug is really nasty and
> hard to debug as it will likely only show / reproduce on the target.
> 
> Felix
> 

It would force reinstalling everything listed in IMAGER_INSTALL on every 
rootfs update, even if packages (fs tools, etc) don't need an update. So, I'm 
not sure it is worth applying this way.

But temporary local modification of do_install_imager_deps(), for debugging 
purposes - why not...

> 
> > 
> > Moving imager to schroot (it's under development right now) should
> > also help 
> > to solve the issue.
> > 
> > 
> > 
> 
>