next: ccache broken for cross-compiling

next: ccache broken for cross-compiling

[Jan Kiszka]

Hi all,

I've noticed that ccache is not doing its job with current next when
cross-building, e.g. mc:qemuarm64-bullseye:linux-mainline. It does work
as expected for native builds (mc:qemuamd64-bullseye:linux-mainline),
and it works fine with v0.9. Seems like an sbuild regression.

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

Re: next: ccache broken for cross-compiling

[Henning Schild]

In order to find what might be going wrong you could enable ccache
logging. setting "log_file = /tmp/ccache.log" in ccache.conf (in the
caches main directory)

Double check it is actually used, i think it would need CCACHE_DIR in
env and some compiler wrappers prefixed in PATH, plus write permissions
on the files.

You can watch the thing from anywhere where it is mounted

CCACHE_DIR=/path/to/cache watch ccache

If you see missing going up but not hits ... it is probably used but
for some reason does not like what it found. Felix once pointed out
that for the kernel that could be if you have debug flags enabled,
where the absolute path of the sources will always end up in the
binaries. Or it could simply be that with sbuild the location of the
sources changes too dramatic ... in which case "hash_dir = false" could
help in the ccache.conf

Henning

Am Sun, 10 Jul 2022 22:51:14 +0200
schrieb Jan Kiszka <jan.kiszka@siemens.com>:

> Hi all,
> 
> I've noticed that ccache is not doing its job with current next when
> cross-building, e.g. mc:qemuarm64-bullseye:linux-mainline. It does
> work as expected for native builds
> (mc:qemuamd64-bullseye:linux-mainline), and it works fine with v0.9.
> Seems like an sbuild regression.
> 
> Jan
> 

Re: next: ccache broken for cross-compiling

[Jan Kiszka]

On 12.07.22 16:20, Henning Schild wrote:
> In order to find what might be going wrong you could enable ccache
> logging. setting "log_file = /tmp/ccache.log" in ccache.conf (in the
> caches main directory)
> 
> Double check it is actually used, i think it would need CCACHE_DIR in
> env and some compiler wrappers prefixed in PATH, plus write permissions
> on the files.
> 
> You can watch the thing from anywhere where it is mounted
> 
> CCACHE_DIR=/path/to/cache watch ccache
> 
> If you see missing going up but not hits ... it is probably used but
> for some reason does not like what it found. Felix once pointed out
> that for the kernel that could be if you have debug flags enabled,
> where the absolute path of the sources will always end up in the
> binaries. Or it could simply be that with sbuild the location of the
> sources changes too dramatic ... in which case "hash_dir = false" could
> help in the ccache.conf
> 

Good hint regarding CONFIG_DEBUG_INFO: It's off for x86 defconfig while
it's on for arm64's defconfig.

Jan

> Henning
> 
> Am Sun, 10 Jul 2022 22:51:14 +0200
> schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> 
>> Hi all,
>>
>> I've noticed that ccache is not doing its job with current next when
>> cross-building, e.g. mc:qemuarm64-bullseye:linux-mainline. It does
>> work as expected for native builds
>> (mc:qemuamd64-bullseye:linux-mainline), and it works fine with v0.9.
>> Seems like an sbuild regression.
>>
>> Jan
>>
> 

-- 
Siemens AG, Technology
Competence Center Embedded Linux

Re: next: ccache broken for cross-compiling

[Uladzimir Bely]

In the email from Sunday, 10 July 2022 23:51:14 +03 user Jan Kiszka wrote:
> Hi all,
> 
> I've noticed that ccache is not doing its job with current next when
> cross-building, e.g. mc:qemuarm64-bullseye:linux-mainline. It does work
> as expected for native builds (mc:qemuamd64-bullseye:linux-mainline),
> and it works fine with v0.9. Seems like an sbuild regression.
> 
> Jan

I did some checks on my side and it seems that it is not related to cross-
build, but mostlry related to something specific to sbuild and kernel.

Just some statistics (taken from 'ccache -s' output)

1. 'hello' cross-build for 'arm64' on 'next' branch (with sbuild)

First run:
  Hits:             415 / 7479 (5.55 %)

Second run:
  Hits:            7353 / 14958 (49.16 %)

Everything OK, most of requests are hits ((7353 - 415) / 7479 = 92.7%)

2. 'linux-mainline' cross build for 'arm64' on 'next' branch (with sbuild)

First run:
  Hits:             262 / 6208 (4.22 %)

Second run:
  Hits:              644 / 12416 (5.19 %)

Cache usage is relatively low ((644 - 262) / 6208 = 5.3%) compared to 'hello'

3. 'linux-mainline' native build for 'arm64' on 'next' branch (with sbuild)

First run:
  Hits:             262 / 6208 (4.22 %)

Second run:
  Hits:              644 / 12416 (5.19 %)

The result is the same as for cross-build ((644 - 262) / 6208 = 5.3%), so the 
low cache usage problem is not related to cross or native build type.

4. 'linux-mainline' native build for 'arm64' on 'master' branch (without 
sbuild)

First run:
  Hits:             262 / 6208 (4.22 %)

Second run:
  Hits:            6469 / 12416 (52.10 %)

Here, ccache usage is high ((6469 - 262) / 6208 = 99.98%)

Most probably, Henning's assumption regarding inclusion of debug information 
(with different absolute paths in case of sbuild) making binaries in cache 
'incompatible' is true. 

-- 
Uladzimir Bely

RE: next: ccache broken for cross-compiling

[Moessbauer, Felix]

> -----Original Message-----
> From: isar-users@googlegroups.com <isar-users@googlegroups.com> On
> Behalf Of Uladzimir Bely
> Sent: Thursday, July 14, 2022 3:41 PM
> To: isar-users <isar-users@googlegroups.com>
> Cc: Kiszka, Jan (T CED) <jan.kiszka@siemens.com>
> Subject: Re: next: ccache broken for cross-compiling
> 
> In the email from Sunday, 10 July 2022 23:51:14 +03 user Jan Kiszka wrote:
> > Hi all,
> >
> > I've noticed that ccache is not doing its job with current next when
> > cross-building, e.g. mc:qemuarm64-bullseye:linux-mainline. It does
> > work as expected for native builds
> > (mc:qemuamd64-bullseye:linux-mainline),
> > and it works fine with v0.9. Seems like an sbuild regression.
> >
> > Jan
> 
> I did some checks on my side and it seems that it is not related to cross- build,
> but mostlry related to something specific to sbuild and kernel.
> 
> Just some statistics (taken from 'ccache -s' output)
> 
> 1. 'hello' cross-build for 'arm64' on 'next' branch (with sbuild)
> 
> First run:
>   Hits:             415 / 7479 (5.55 %)
> 
> Second run:
>   Hits:            7353 / 14958 (49.16 %)
> 
> Everything OK, most of requests are hits ((7353 - 415) / 7479 = 92.7%)
> 
> 2. 'linux-mainline' cross build for 'arm64' on 'next' branch (with sbuild)
> 
> First run:
>   Hits:             262 / 6208 (4.22 %)
> 
> Second run:
>   Hits:              644 / 12416 (5.19 %)
> 
> Cache usage is relatively low ((644 - 262) / 6208 = 5.3%) compared to 'hello'
> 
> 3. 'linux-mainline' native build for 'arm64' on 'next' branch (with sbuild)
> 
> First run:
>   Hits:             262 / 6208 (4.22 %)
> 
> Second run:
>   Hits:              644 / 12416 (5.19 %)
> 
> The result is the same as for cross-build ((644 - 262) / 6208 = 5.3%), so the low
> cache usage problem is not related to cross or native build type.
> 
> 4. 'linux-mainline' native build for 'arm64' on 'master' branch (without
> sbuild)
> 
> First run:
>   Hits:             262 / 6208 (4.22 %)
> 
> Second run:
>   Hits:            6469 / 12416 (52.10 %)
> 
> Here, ccache usage is high ((6469 - 262) / 6208 = 99.98%)
> 
> Most probably, Henning's assumption regarding inclusion of debug information
> (with different absolute paths in case of sbuild) making binaries in cache
> 'incompatible' is true.

At least when I had the issue, the debug information was the root cause for low cache hits.
But instead of guessing, it is better to simply enable the CCache debug mode and look into the generated debug files.
There it is easy to spot the problems.

Yesterday, I sent a patch on the ML to enable the debug mode in ISAR.

BTW: the kernel's debug information not necessarily has to be disabled.
You can also fiddle in the KCFLAG -fdebug-prefix-map=

Felix

> 
> --
> Uladzimir Bely
> 
> 
> 
> --
> You received this message because you are subscribed to the Google Groups
> "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to isar-users+unsubscribe@googlegroups.com.
> To view this discussion on the web visit
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.g
> oogle.com%2Fd%2Fmsgid%2Fisar-
> users%2F6773152.jJDZkT8p0M%2540home&amp;data=05%7C01%7Cfelix.moes
> sbauer%40siemens.com%7C8a0fe8c45b8c4f05f68b08da659e845c%7C38ae3bcd
> 95794fd4addab42e1495d55a%7C1%7C0%7C637934028749181086%7CUnknow
> n%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haW
> wiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=qJm%2Bm1F7EQ9y0wy
> kA1%2BJnooOdJo%2FMG2Va5iUSn8mGzA%3D&amp;reserved=0.

Re: next: ccache broken for cross-compiling

[Jan Kiszka]

On 14.07.22 15:51, Moessbauer, Felix (T CED SES-DE) wrote:
>> -----Original Message-----
>> From: isar-users@googlegroups.com <isar-users@googlegroups.com> On
>> Behalf Of Uladzimir Bely
>> Sent: Thursday, July 14, 2022 3:41 PM
>> To: isar-users <isar-users@googlegroups.com>
>> Cc: Kiszka, Jan (T CED) <jan.kiszka@siemens.com>
>> Subject: Re: next: ccache broken for cross-compiling
>>
>> In the email from Sunday, 10 July 2022 23:51:14 +03 user Jan Kiszka wrote:
>>> Hi all,
>>>
>>> I've noticed that ccache is not doing its job with current next when
>>> cross-building, e.g. mc:qemuarm64-bullseye:linux-mainline. It does
>>> work as expected for native builds
>>> (mc:qemuamd64-bullseye:linux-mainline),
>>> and it works fine with v0.9. Seems like an sbuild regression.
>>>
>>> Jan
>>
>> I did some checks on my side and it seems that it is not related to cross- build,
>> but mostlry related to something specific to sbuild and kernel.
>>
>> Just some statistics (taken from 'ccache -s' output)
>>
>> 1. 'hello' cross-build for 'arm64' on 'next' branch (with sbuild)
>>
>> First run:
>>   Hits:             415 / 7479 (5.55 %)
>>
>> Second run:
>>   Hits:            7353 / 14958 (49.16 %)
>>
>> Everything OK, most of requests are hits ((7353 - 415) / 7479 = 92.7%)
>>
>> 2. 'linux-mainline' cross build for 'arm64' on 'next' branch (with sbuild)
>>
>> First run:
>>   Hits:             262 / 6208 (4.22 %)
>>
>> Second run:
>>   Hits:              644 / 12416 (5.19 %)
>>
>> Cache usage is relatively low ((644 - 262) / 6208 = 5.3%) compared to 'hello'
>>
>> 3. 'linux-mainline' native build for 'arm64' on 'next' branch (with sbuild)
>>
>> First run:
>>   Hits:             262 / 6208 (4.22 %)
>>
>> Second run:
>>   Hits:              644 / 12416 (5.19 %)
>>
>> The result is the same as for cross-build ((644 - 262) / 6208 = 5.3%), so the low
>> cache usage problem is not related to cross or native build type.
>>
>> 4. 'linux-mainline' native build for 'arm64' on 'master' branch (without
>> sbuild)
>>
>> First run:
>>   Hits:             262 / 6208 (4.22 %)
>>
>> Second run:
>>   Hits:            6469 / 12416 (52.10 %)
>>
>> Here, ccache usage is high ((6469 - 262) / 6208 = 99.98%)
>>
>> Most probably, Henning's assumption regarding inclusion of debug information
>> (with different absolute paths in case of sbuild) making binaries in cache
>> 'incompatible' is true.
> 
> At least when I had the issue, the debug information was the root cause for low cache hits.
> But instead of guessing, it is better to simply enable the CCache debug mode and look into the generated debug files.
> There it is easy to spot the problems.

The probability that this effect is rather related to CONFIG_DEBUG_INFO
than cross-compile is high. We use the defconfigs for the archs, and x86
happen to have that off while arm64 has it on. Worth to check that trace
first.

> 
> Yesterday, I sent a patch on the ML to enable the debug mode in ISAR.
> 
> BTW: the kernel's debug information not necessarily has to be disabled.
> You can also fiddle in the KCFLAG -fdebug-prefix-map=

...and then this may become relevant, indeed.

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux