[PATCH 0/1] fix copy out of apt cache after sbuild

[PATCH 0/1] fix copy out of apt cache after sbuild

[Felix Moessbauer]

This series fixes the build issues reported in "potential race
condition in sbuild post commands" by not copying symlinks. This has
been tested on various large layers and seems to do the job. However,
it still remains unclear why packages are missing in the apt cache
under some rebuild scenarios.

While investigating this, I also noticed that we AGAIN have a quadratic
space blow-up in the isar-apt directory that is copied into the build
folder of each package. This makes me wonder if the whole APT handling
is properly understood. IMHO this needs another haul-over, which is also
indicated by the number of apt related issues which were reported in the
last months.

Best regards,
Felix

Felix Moessbauer (1):
  fix copy out of apt cache after sbuild

 meta/classes/dpkg.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
2.39.2

[PATCH 1/1] fix copy out of apt cache after sbuild

[Felix Moessbauer]

During the setup of the sbuild chroot, all packages from the
global apt cache are symlinked into the sbuild chroot. After the build,
the newly downloaded packages should be copied out from the local cache
to the global apt cache. However, the current logic tried to copy out
all packages, including the symlinks. This does not work, as the
symlinks might point to non-existing locations.

This is now fixed by only copying out the newly downloaded debs (the
ones which are not a symlink).

Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
---
 meta/classes/dpkg.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index c699a84d..3057329f 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -121,7 +121,7 @@ dpkg_runbuild() {
         --chroot-setup-commands="mkdir -p ${deb_dir}" \
         --chroot-setup-commands="ln -sf ${ext_deb_dir}/*.deb -t ${deb_dir}/" \
         --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \
-        --finished-build-commands="[ -z \"\$(find ${deb_dir} -maxdepth 1 -name '*.deb' -print -quit)\" ] || cp ${CP_FLAGS} ${deb_dir}/*.deb -t ${ext_deb_dir}/" \
+        --finished-build-commands="find ${deb_dir} -maxdepth 1 -type f -name '*.deb' -print -exec cp ${CP_FLAGS} -t ${ext_deb_dir}/ {} \;" \
         --finished-build-commands="cp /var/log/dpkg.log ${ext_root}/dpkg_partial.log" \
         --debbuildopts="--source-option=-I" \
         --build-dir=${WORKDIR} --dist="isar" ${DSC_FILE}
-- 
2.39.2

RE: [PATCH 0/1] fix copy out of apt cache after sbuild

[Schmidt, Adriaan]

'Felix Moessbauer' via isar-users <isar-users@googlegroups.com>, Monday, January 22, 2024 2:27 PM
> This series fixes the build issues reported in "potential race
> condition in sbuild post commands" by not copying symlinks. This has
> been tested on various large layers and seems to do the job. However,
> it still remains unclear why packages are missing in the apt cache
> under some rebuild scenarios.
> 
> While investigating this, I also noticed that we AGAIN have a quadratic
> space blow-up in the isar-apt directory that is copied into the build
> folder of each package. This makes me wonder if the whole APT handling
> is properly understood. IMHO this needs another haul-over, which is also
> indicated by the number of apt related issues which were reported in the
> last months.

https://github.com/ilbers/isar/blob/master/meta/classes/dpkg-base.bbclass#L198

Looking at the code, the intention is that hard links are used to pass isar-apt
into each package build. Could your measurement be counting those wrong?

And if we do think that another approach at handling the whole apt topic is worth
considering: I posted about that a while back ("[RFC PATCH] Remove isar-apt and
the corresponding lock" from 2022-10-19)...

Adriaan

> 
> Best regards,
> Felix
> 
> Felix Moessbauer (1):
>   fix copy out of apt cache after sbuild
> 
>  meta/classes/dpkg.bbclass | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> --
> 2.39.2
> 
> --
> You received this message because you are subscribed to the Google Groups
> "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to isar-users+unsubscribe@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/isar-users/20240122132647.779902-1-
> felix.moessbauer%40siemens.com.

Re: [PATCH 0/1] fix copy out of apt cache after sbuild

[MOESSBAUER, Felix]

On Mon, 2024-01-22 at 13:47 +0000, Schmidt, Adriaan (T CED EDC-DE)
wrote:
> 'Felix Moessbauer' via isar-users <isar-users@googlegroups.com>,
> Monday, January 22, 2024 2:27 PM
> > This series fixes the build issues reported in "potential race
> > condition in sbuild post commands" by not copying symlinks. This
> > has
> > been tested on various large layers and seems to do the job.
> > However,
> > it still remains unclear why packages are missing in the apt cache
> > under some rebuild scenarios.
> > 
> > While investigating this, I also noticed that we AGAIN have a
> > quadratic
> > space blow-up in the isar-apt directory that is copied into the
> > build
> > folder of each package. This makes me wonder if the whole APT
> > handling
> > is properly understood. IMHO this needs another haul-over, which is
> > also
> > indicated by the number of apt related issues which were reported
> > in the
> > last months.
> 
> https://github.com/ilbers/isar/blob/master/meta/classes/dpkg-base.bbclass#L198
> 
> Looking at the code, the intention is that hard links are used to
> pass isar-apt
> into each package build. Could your measurement be counting those
> wrong?

Right. Thanks for the pointer. That's why I prefer to use the long
options of the commands.

> 
> And if we do think that another approach at handling the whole apt
> topic is worth
> considering: I posted about that a while back ("[RFC PATCH] Remove
> isar-apt and
> the corresponding lock" from 2022-10-19)...
> 

I still believe this is a good idea.

Felix

> Adriaan
> 
> > 
> > Best regards,
> > Felix
> > 
> > Felix Moessbauer (1):
> >   fix copy out of apt cache after sbuild
> > 
> >  meta/classes/dpkg.bbclass | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > --
> > 2.39.2
> > 
> > --
> > You received this message because you are subscribed to the Google
> > Groups
> > "isar-users" group.
> > To unsubscribe from this group and stop receiving emails from it,
> > send an
> > email to isar-users+unsubscribe@googlegroups.com.
> > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/isar-users/20240122132647.779902-1
> > -
> > felix.moessbauer%40siemens.com.

-- 
Siemens AG, Technology
Linux Expert Center

Re: [PATCH 0/1] fix copy out of apt cache after sbuild

[Uladzimir Bely]

On Mon, 2024-01-22 at 14:26 +0100, 'Felix Moessbauer' via isar-users
wrote:
> This series fixes the build issues reported in "potential race
> condition in sbuild post commands" by not copying symlinks. This has
> been tested on various large layers and seems to do the job. However,
> it still remains unclear why packages are missing in the apt cache
> under some rebuild scenarios.
> 
> While investigating this, I also noticed that we AGAIN have a
> quadratic
> space blow-up in the isar-apt directory that is copied into the build
> folder of each package. This makes me wonder if the whole APT
> handling
> is properly understood. IMHO this needs another haul-over, which is
> also
> indicated by the number of apt related issues which were reported in
> the
> last months.
> 
> Best regards,
> Felix
> 
> Felix Moessbauer (1):
>   fix copy out of apt cache after sbuild
> 
>  meta/classes/dpkg.bbclass | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> -- 
> 2.39.2
> 

Applied to next, thanks.