[PATCH] sbuild: Fix low hit ratio for ccache on linux kernel rebuild

[PATCH] sbuild: Fix low hit ratio for ccache on linux kernel rebuild

[Uladzimir Bely]

For some targets defconfig for linux recipe includes option
CONFIG_DEBUG_INFO that embeds absolute path to the source files
into compiled binaries.

While sbuild uses unique directory to place sources for each build
by default, it happens that ccache hits ration is to small on rebuild.

This patch makes sbuild use the same build directory for sequential
builds. While several packages can be built in parallel (e.g. same
package for different architectures), WORKDIR is used to generate
hash part for the build directory of sbuild.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/dpkg.bbclass | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index b726ea9d..016e46c4 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -82,6 +82,8 @@ dpkg_runbuild() {
     sh -c "cd ${WORKDIR}; dpkg-source -q -b ${PPS}"
     DSC_FILE=$(find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -print)
 
+    build_path_hash=$(echo "${WORKDIR}" | md5sum | cut -b1-6)
+
     sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
         --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \
         --no-run-lintian --no-run-piuparts --no-run-autopkgtest --resolve-alternatives \
@@ -91,6 +93,7 @@ dpkg_runbuild() {
         --finished-build-commands="cp -n --no-preserve=owner ${deb_dir}/*.deb -t ${ext_deb_dir}/ || :" \
         --finished-build-commands="cp /var/log/dpkg.log ${ext_root}/dpkg_partial.log" \
         --debbuildopts="--source-option=-I" \
+        --build-path="/build/${PN}-${build_path_hash}" \
         --build-dir=${WORKDIR} --dist="isar" ${DSC_FILE}
 
     sbuild_dpkg_log_export "${WORKDIR}/rootfs/dpkg_partial.log"
-- 
2.20.1

RE: [PATCH] sbuild: Fix low hit ratio for ccache on linux kernel rebuild

[Moessbauer, Felix]

> -----Original Message-----
> From: isar-users@googlegroups.com <isar-users@googlegroups.com> On
> Behalf Of Uladzimir Bely
> Sent: Friday, July 15, 2022 1:27 PM
> To: isar-users@googlegroups.com
> Subject: [PATCH] sbuild: Fix low hit ratio for ccache on linux kernel rebuild
> 
> For some targets defconfig for linux recipe includes option
> CONFIG_DEBUG_INFO that embeds absolute path to the source files into
> compiled binaries.
> 
> While sbuild uses unique directory to place sources for each build by default, it
> happens that ccache hits ration is to small on rebuild.
> 
> This patch makes sbuild use the same build directory for sequential builds. While
> several packages can be built in parallel (e.g. same package for different
> architectures), WORKDIR is used to generate hash part for the build directory of
> sbuild.
> 
> Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> ---
>  meta/classes/dpkg.bbclass | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass index
> b726ea9d..016e46c4 100644
> --- a/meta/classes/dpkg.bbclass
> +++ b/meta/classes/dpkg.bbclass
> @@ -82,6 +82,8 @@ dpkg_runbuild() {
>      sh -c "cd ${WORKDIR}; dpkg-source -q -b ${PPS}"
>      DSC_FILE=$(find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -print)
> 
> +    build_path_hash=$(echo "${WORKDIR}" | md5sum | cut -b1-6)
> +
>      sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
>          --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \
>          --no-run-lintian --no-run-piuparts --no-run-autopkgtest --resolve-
> alternatives \ @@ -91,6 +93,7 @@ dpkg_runbuild() {
>          --finished-build-commands="cp -n --no-preserve=owner ${deb_dir}/*.deb -t
> ${ext_deb_dir}/ || :" \
>          --finished-build-commands="cp /var/log/dpkg.log
> ${ext_root}/dpkg_partial.log" \
>          --debbuildopts="--source-option=-I" \
> +        --build-path="/build/${PN}-${build_path_hash}" \

This is just a workaround that hides the problem.
Debian takes a lot of care to improve the reproducibility of their packages.
IIRC then reproducibility issues will be treated as bugs from bookworm on.
Could we please do the same and fix the corresponding recipes (like the kernel recipe)?

Please also have a look at the reproducibility section of dpkg-buildflags.
The only reason the kernel is affected from this is, that the kbuild only uses KCFLAGS, not CFLAGS.
At least the flags about reproducibility should be added to all kernel builds done via ISAR.

In short: I'm strictly against this workaround.
Having things that are not reproducible should hurt and not become accepted.

Felix

>          --build-dir=${WORKDIR} --dist="isar" ${DSC_FILE}
> 
>      sbuild_dpkg_log_export "${WORKDIR}/rootfs/dpkg_partial.log"
> --
> 2.20.1
> 
> --
> You received this message because you are subscribed to the Google Groups
> "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to isar-users+unsubscribe@googlegroups.com.
> To view this discussion on the web visit
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.g
> oogle.com%2Fd%2Fmsgid%2Fisar-users%2F20220715112716.2460-1-
> ubely%2540ilbers.de&amp;data=05%7C01%7Cfelix.moessbauer%40siemens.co
> m%7C2fbec95c6a0a48bcfa1608da6654fc4b%7C38ae3bcd95794fd4addab42e14
> 95d55a%7C1%7C0%7C637934812445713510%7CUnknown%7CTWFpbGZsb3d8e
> yJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7
> C3000%7C%7C%7C&amp;sdata=H%2FfsTUORIfNFM7jh8CYiCNkuf7T66oxH31mh
> 1v2Leo0%3D&amp;reserved=0.

Re: [PATCH] sbuild: Fix low hit ratio for ccache on linux kernel rebuild

[Uladzimir Bely]

In the email from Friday, 15 July 2022 16:27:27 +03 user Moessbauer, Felix 
wrote:
> > -----Original Message-----
> > From: isar-users@googlegroups.com <isar-users@googlegroups.com> On
> > Behalf Of Uladzimir Bely
> > Sent: Friday, July 15, 2022 1:27 PM
> > To: isar-users@googlegroups.com
> > Subject: [PATCH] sbuild: Fix low hit ratio for ccache on linux kernel
> > rebuild
> > 
> > For some targets defconfig for linux recipe includes option
> > CONFIG_DEBUG_INFO that embeds absolute path to the source files into
> > compiled binaries.
> > 
> > While sbuild uses unique directory to place sources for each build by
> > default, it happens that ccache hits ration is to small on rebuild.
> > 
> > This patch makes sbuild use the same build directory for sequential
> > builds. While several packages can be built in parallel (e.g. same
> > package for different architectures), WORKDIR is used to generate hash
> > part for the build directory of sbuild.
> > 
> > Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
> > ---
> > 
> >  meta/classes/dpkg.bbclass | 3 +++
> >  1 file changed, 3 insertions(+)
> > 
> > diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass index
> > b726ea9d..016e46c4 100644
> > --- a/meta/classes/dpkg.bbclass
> > +++ b/meta/classes/dpkg.bbclass
> > @@ -82,6 +82,8 @@ dpkg_runbuild() {
> > 
> >      sh -c "cd ${WORKDIR}; dpkg-source -q -b ${PPS}"
> >      DSC_FILE=$(find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -print)
> > 
> > +    build_path_hash=$(echo "${WORKDIR}" | md5sum | cut -b1-6)
> > +
> > 
> >      sbuild -A -n -c ${SBUILD_CHROOT}
> >      --extra-repository="${ISAR_APT_REPO}" \
> >      
> >          --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \
> >          --no-run-lintian --no-run-piuparts --no-run-autopkgtest
> >          --resolve-
> > 
> > alternatives \ @@ -91,6 +93,7 @@ dpkg_runbuild() {
> > 
> >          --finished-build-commands="cp -n --no-preserve=owner
> >          ${deb_dir}/*.deb -t
> > 
> > ${ext_deb_dir}/ || :" \
> > 
> >          --finished-build-commands="cp /var/log/dpkg.log
> > 
> > ${ext_root}/dpkg_partial.log" \
> > 
> >          --debbuildopts="--source-option=-I" \
> > 
> > +        --build-path="/build/${PN}-${build_path_hash}" \
> 
> This is just a workaround that hides the problem.
> Debian takes a lot of care to improve the reproducibility of their packages.
> IIRC then reproducibility issues will be treated as bugs from bookworm on.
> Could we please do the same and fix the corresponding recipes (like the
> kernel recipe)?
> 

It can be moved to the kernel recipe, but you left me a bit confused. Why 
changing of build directory from some random (from build to build) path to 
some fixed one (the same hash on every rebuild) makes the build less 
reproducible?

> Please also have a look at the reproducibility section of dpkg-buildflags.
> The only reason the kernel is affected from this is, that the kbuild only
> uses KCFLAGS, not CFLAGS. At least the flags about reproducibility should
> be added to all kernel builds done via ISAR.
> 
> In short: I'm strictly against this workaround.
> Having things that are not reproducible should hurt and not become accepted.
> 
> Felix
> 
> >          --build-dir=${WORKDIR} --dist="isar" ${DSC_FILE}
> >      
> >      sbuild_dpkg_log_export "${WORKDIR}/rootfs/dpkg_partial.log"
> > 
> > --
> > 2.20.1
> > 
> > --
> > You received this message because you are subscribed to the Google Groups
> > "isar-users" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to isar-users+unsubscribe@googlegroups.com.
> > To view this discussion on the web visit
> > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.g
> > oogle.com%2Fd%2Fmsgid%2Fisar-users%2F20220715112716.2460-1-
> > ubely%2540ilbers.de&amp;data=05%7C01%7Cfelix.moessbauer%40siemens.co
> > m%7C2fbec95c6a0a48bcfa1608da6654fc4b%7C38ae3bcd95794fd4addab42e14
> > 95d55a%7C1%7C0%7C637934812445713510%7CUnknown%7CTWFpbGZsb3d8e
> > yJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7
> > C3000%7C%7C%7C&amp;sdata=H%2FfsTUORIfNFM7jh8CYiCNkuf7T66oxH31mh
> > 1v2Leo0%3D&amp;reserved=0.


-- 
Uladzimir Bely

Re: [PATCH] sbuild: Fix low hit ratio for ccache on linux kernel rebuild

[Baurzhan Ismagulov]

On Fri, Jul 15, 2022 at 04:44:29PM +0300, Uladzimir Bely wrote:
> It can be moved to the kernel recipe, but you left me a bit confused. Why 
> changing of build directory from some random (from build to build) path to 
> some fixed one (the same hash on every rebuild) makes the build less 
> reproducible?

Debian wants to have identical binary package contents from different builds,
see https://wiki.debian.org/ReproducibleBuilds. I don't know by heart whether /
how that works for debug info, though. Seems that at least absolute paths
should be a no-go.

With kind regards,
Baurzhan

RE: [PATCH] sbuild: Fix low hit ratio for ccache on linux kernel rebuild

[Moessbauer, Felix]

> -----Original Message-----
> From: isar-users@googlegroups.com <isar-users@googlegroups.com> On
> Behalf Of Baurzhan Ismagulov
> Sent: Friday, July 15, 2022 3:50 PM
> To: isar-users@googlegroups.com
> Subject: Re: [PATCH] sbuild: Fix low hit ratio for ccache on linux kernel rebuild
> 
> On Fri, Jul 15, 2022 at 04:44:29PM +0300, Uladzimir Bely wrote:
> > It can be moved to the kernel recipe, but you left me a bit confused.
> > Why changing of build directory from some random (from build to build)
> > path to some fixed one (the same hash on every rebuild) makes the
> > build less reproducible?
> 
> Debian wants to have identical binary package contents from different builds,
> see
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.debi
> an.org%2FReproducibleBuilds&amp;data=05%7C01%7Cfelix.moessbauer%40sie
> mens.com%7C21efa7957915433beaaf08da6668f548%7C38ae3bcd95794fd4add
> ab42e1495d55a%7C1%7C0%7C637934898219710974%7CUnknown%7CTWFpb
> GZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6M
> n0%3D%7C3000%7C%7C%7C&amp;sdata=1AgsTG9AeVSOlofWSjrURw3%2BsNv
> zI1NcGcJcVZLClwE%3D&amp;reserved=0. I don't know by heart whether / how
> that works for debug info, though. Seems that at least absolute paths should be
> a no-go.

Exactly. We do not want to have any absolute paths in the binary.
By that, the build-dir shall not be relevant (neither for caching, nor for reproducibility).

Regarding relocatable debug info: the debhelpers normally handle that for you.
You can play around by running dpkg-buildflags with various settings (via DEB_BUILD_MAINT_OPTIONS, e.g. DEB_BUILD_MAINT_OPTIONS= reproducible=-timeless).

On bullseye, the -ffile-prefix-map=/tmp/foo=. -fdebug-prefix-map=/tmp/foo=. (building in /tmp/foo) flags are passed to CFLAGS, which strip the specified prefix from the data in the debug info.
However, this does not work for the kernel build, as the CFLAGS are ignored but only KCFLAGS are considered.
By that, we have to manually pass some flags to KCFLAGS.
The official Debian kernel build recipe also does that, but for our custom kernels this might be "too much".

Maybe we could use the dpkg-buildflags to generate some reasonable flags and also enable the reproducibility ones:
export DEB_BUILD_MAINT_OPTIONS="hardening=-all reproducible"
Then, we can pass the generated flags to KCFLAGS.

Felix

> 
> With kind regards,
> Baurzhan
> 
> --
> You received this message because you are subscribed to the Google Groups
> "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to isar-users+unsubscribe@googlegroups.com.
> To view this discussion on the web visit
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.g
> oogle.com%2Fd%2Fmsgid%2Fisar-
> users%2FYtFwl3QBLXdTqVtg%2540ilbers.de&amp;data=05%7C01%7Cfelix.moes
> sbauer%40siemens.com%7C21efa7957915433beaaf08da6668f548%7C38ae3bc
> d95794fd4addab42e1495d55a%7C1%7C0%7C637934898219710974%7CUnkno
> wn%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1ha
> WwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=%2Bvcc0A1dR%2FTIrs
> ojkkCeSEGTbBPBzZgdV%2BndWIMAUGw%3D&amp;reserved=0.

Re: [PATCH] sbuild: Fix low hit ratio for ccache on linux kernel rebuild

[Baurzhan Ismagulov]

Hello Felix,

On Fri, Jul 15, 2022 at 01:27:27PM +0000, Moessbauer, Felix wrote:
> Please also have a look at the reproducibility section of dpkg-buildflags.
> The only reason the kernel is affected from this is, that the kbuild only uses KCFLAGS, not CFLAGS.
> At least the flags about reproducibility should be added to all kernel builds done via ISAR.

IIUC, the non-reproducibility comes from the absolute paths in the debug info
-- we'll have a look, maybe those could be replaced with relative ones.

I haven't deeply looked at reproducible packages, could you please elaborate on
the KCFLAGS vs. CFLAGS distinction mattering here?

With kind regards,
Baurzhan

Re: [PATCH] sbuild: Fix low hit ratio for ccache on linux kernel rebuild

[Jan Kiszka]

On 15.07.22 15:47, Baurzhan Ismagulov wrote:
> Hello Felix,
> 
> On Fri, Jul 15, 2022 at 01:27:27PM +0000, Moessbauer, Felix wrote:
>> Please also have a look at the reproducibility section of dpkg-buildflags.
>> The only reason the kernel is affected from this is, that the kbuild only uses KCFLAGS, not CFLAGS.
>> At least the flags about reproducibility should be added to all kernel builds done via ISAR.
> 
> IIUC, the non-reproducibility comes from the absolute paths in the debug info
> -- we'll have a look, maybe those could be replaced with relative ones.
> 
> I haven't deeply looked at reproducible packages, could you please elaborate on
> the KCFLAGS vs. CFLAGS distinction mattering here?
> 

https://www.kernel.org/doc/html/latest/kbuild/reproducible-builds.html
may explain more. I just learned from Ben Hutchings that following this
in our kernel recipe may not only resolve the absolute path topic but
would also allow to drop slow dh_strip_nondeterminism from our builds.

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux