isar-bootstrap failures with debian-buster targets

isar-bootstrap failures with debian-buster targets

[Jan Kiszka]

Hi all,

only on my machine so far, I'm seeing weird failures of buster-targeting
non-x86 builds, primarily armhf and also mipsel. The pattern is always
some out-of-memory inside apt during "apt-get update":

[...]
'/work/build/tmp/work/debian-buster-armhf/isar-bootstrap-target-1.0-r0/chroot-setup.sh' -> '/work/build/tmp/work/debian-buster-armhf/isar-bootstrap-target-1.0-r0/rootfs/chroot-setup.sh'
dpkg-divert: warning: diverting file '/sbin/start-stop-daemon' from an Essential package with rename is dangerous, use --no-rename
Hit:1 http://ftp.debian.org/debian buster InRelease
Get:2 http://ftp.debian.org/debian buster-updates InRelease [49.3 kB]
Get:3 http://security-cdn.debian.org buster/updates InRelease [39.1 kB]
Get:4 http://ftp.debian.org/debian buster/contrib Sources [43.1 kB]
Get:5 http://ftp.debian.org/debian buster/main Sources [7,827 kB]
Get:6 http://ftp.debian.org/debian buster/non-free Sources [86.2 kB]
Get:7 http://ftp.debian.org/debian buster/main Translation-en [5,967 kB]
Get:8 http://ftp.debian.org/debian buster/contrib Translation-en [44.7 kB]
Get:9 http://ftp.debian.org/debian buster/non-free Translation-en [88.3 kB]
Get:10 http://ftp.debian.org/debian buster-updates/main Sources [1,204 B]
Get:11 http://ftp.debian.org/debian buster-updates/main armhf Packages [884 B]
Get:12 http://ftp.debian.org/debian buster-updates/main Translation-en [600 B]
Reading package lists...
E: Could not read from /var/lib/apt/lists/partial/security.debian.org_dists_buster_updates_InRelease - getline (12: Cannot allocate memory)
E: The repository 'http://security.debian.org buster/updates InRelease' provides only weak security information.
WARNING: exit code 100 from a shell command.
ERROR: Function failed: do_bootstrap (log file is located at /work/build/tmp/work/debian-buster-armhf/isar-bootstrap-target-1.0-r0/temp/log.do_bootstrap.1156)

This is apparently caused by the getline here:

https://salsa.debian.org/apt-team/apt/blob/master/apt-pkg/contrib/fileutl.cc#L989

That just reads an absolutely valid line first line from a Release file
that is valid as well (identical to the file an x86 target downloaded
e.g.). ENOMEM makes zero sense /wrt to the overall system memory
available. It also does not correlate with parallel builds, I've also 
seen it when just building that single target. And, of course, it also 
succeeds when retrying often enough.

Anyone any idea? Build host is latest kas-project/kas-isar, i.e. a
Debian 10 instance.

Thanks,
Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: isar-bootstrap failures with debian-buster targets

[Baurzhan Ismagulov]

Hello Jan,

On Thu, Aug 22, 2019 at 04:18:57PM +0200, Jan Kiszka wrote:
> only on my machine so far, I'm seeing weird failures of buster-targeting
> non-x86 builds, primarily armhf and also mipsel. The pattern is always
> some out-of-memory inside apt during "apt-get update":
...
> Anyone any idea? Build host is latest kas-project/kas-isar, i.e. a
> Debian 10 instance.

Is it reproducible with stock Isar?

Is it reproducible outside of docker?

With kind regards,
Baurzhan.

Re: isar-bootstrap failures with debian-buster targets

[Jan Kiszka]

On 22.08.19 16:47, Baurzhan Ismagulov wrote:
> Hello Jan,
> 
> On Thu, Aug 22, 2019 at 04:18:57PM +0200, Jan Kiszka wrote:
>> only on my machine so far, I'm seeing weird failures of buster-targeting
>> non-x86 builds, primarily armhf and also mipsel. The pattern is always
>> some out-of-memory inside apt during "apt-get update":
> ...
>> Anyone any idea? Build host is latest kas-project/kas-isar, i.e. a
>> Debian 10 instance.
> 
> Is it reproducible with stock Isar?

Yes, at least with next.

> 
> Is it reproducible outside of docker?

No idea, I have no such environment.

Interestingly, it does not trigger on our CI system running the exact same 
docker image.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: isar-bootstrap failures with debian-buster targets

[Baurzhan Ismagulov]

On Thu, Aug 22, 2019 at 05:00:42PM +0200, Jan Kiszka wrote:
> > Is it reproducible with stock Isar?
> 
> Yes, at least with next.
> 
> > 
> > Is it reproducible outside of docker?
> 
> No idea, I have no such environment.
> 
> Interestingly, it does not trigger on our CI system running the exact same
> docker image.

I've tried building mc:qemuarm-stretch:i-i-b from next+ under
kasproject/kas-isar:latest once, it succeeded. Once in how many times does it
fail for you? My host is stretch.

With kind regards,
Baurzhan.

Re: isar-bootstrap failures with debian-buster targets

[Jan Kiszka]

On 22.08.19 17:27, Baurzhan Ismagulov wrote:
> On Thu, Aug 22, 2019 at 05:00:42PM +0200, Jan Kiszka wrote:
>>> Is it reproducible with stock Isar?
>>
>> Yes, at least with next.
>>
>>>
>>> Is it reproducible outside of docker?
>>
>> No idea, I have no such environment.
>>
>> Interestingly, it does not trigger on our CI system running the exact same
>> docker image.
> 
> I've tried building mc:qemuarm-stretch:i-i-b from next+ under
> kasproject/kas-isar:latest once, it succeeded. Once in how many times does it
> fail for you? My host is stretch.
> 

A while ago it was 100%: Tried multiconfig:qemuarm-buster:isar-image-base 
multiple times in a row, without passing. Then it worked with 
kasproject/kas-isar:1.0 which was stretch-based. And now it's even passing with 
kas-isar:latest. Weird.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: isar-bootstrap failures with debian-buster targets

[Baurzhan Ismagulov]

On Fri, Aug 23, 2019 at 02:26:57PM +0200, Jan Kiszka wrote:
> A while ago it was 100%: Tried multiconfig:qemuarm-buster:isar-image-base
> multiple times in a row, without passing. Then it worked with
> kasproject/kas-isar:1.0 which was stretch-based. And now it's even passing
> with kas-isar:latest. Weird.

FWIW, this also occurred on an internal Jenkins machine (stretch host, stretch
workspace, no docker). This happening only with buster target, we have only two
candidates: bug in qemu or bug Debian, no?

With kind regards,
Baurzhan.

Re: isar-bootstrap failures with debian-buster targets

[Jan Kiszka]

On 27.08.19 06:33, Baurzhan Ismagulov wrote:
> On Fri, Aug 23, 2019 at 02:26:57PM +0200, Jan Kiszka wrote:
>> A while ago it was 100%: Tried multiconfig:qemuarm-buster:isar-image-base
>> multiple times in a row, without passing. Then it worked with
>> kasproject/kas-isar:1.0 which was stretch-based. And now it's even passing
>> with kas-isar:latest. Weird.
> 
> FWIW, this also occurred on an internal Jenkins machine (stretch host, stretch
> workspace, no docker). This happening only with buster target, we have only two
> candidates: bug in qemu or bug Debian, no?

Yeah, stretch host does not seem to make a difference. I've seen it meanwhile 
also with the stretch-based kas image.

Should we try to file bug? Question is: against which component? debootstrap?

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: isar-bootstrap failures with debian-buster targets

[Baurzhan Ismagulov]

On Tue, Aug 27, 2019 at 09:10:21AM +0200, Jan Kiszka wrote:
> > FWIW, this also occurred on an internal Jenkins machine (stretch host, stretch
> > workspace, no docker). This happening only with buster target, we have only two
> > candidates: bug in qemu or bug Debian, no?
> 
> Yeah, stretch host does not seem to make a difference. I've seen it
> meanwhile also with the stretch-based kas image.
> 
> Should we try to file bug? Question is: against which component? debootstrap?

I think the best candidate would be the package containing the binary that
fails -- presumably apt.

For me, the questions are:

1. Can we rule out qemu in some way?

2. Is it reproducible in a simple way?

I.e., ideally, it should be reproducible from the command line on the host
natively, otherwise we'll probably end up just having reported the problem --
which is admittedly also necessary.


That said, the links below (at least one of which appears to be an automated
translation from English) refer to the same problem in the context of
Singularity, which would point rather into the direction of qemu. What is
actually the size of RAM available to the guest in our case? Apt has the
history of requiring considerable amounts of RAM.

1. https://issue.life/questions/51632062
2. https://bug200.com/post/51632062


With kind regards,
Baurzhan.

Re: isar-bootstrap failures with debian-buster targets

[Jan Kiszka]

On 03.09.19 17:44, Baurzhan Ismagulov wrote:
> On Tue, Aug 27, 2019 at 09:10:21AM +0200, Jan Kiszka wrote:
>>> FWIW, this also occurred on an internal Jenkins machine (stretch host, stretch
>>> workspace, no docker). This happening only with buster target, we have only two
>>> candidates: bug in qemu or bug Debian, no?
>>
>> Yeah, stretch host does not seem to make a difference. I've seen it
>> meanwhile also with the stretch-based kas image.
>>
>> Should we try to file bug? Question is: against which component? debootstrap?
> 
> I think the best candidate would be the package containing the binary that
> fails -- presumably apt.
> 
> For me, the questions are:
> 
> 1. Can we rule out qemu in some way?
> 
> 2. Is it reproducible in a simple way?

Well, for me at least: Just started the OrangePi Zero build of jailhouse-images, 
which is now buster-based, and it failed again. I suspect I can trigger this now 
for a couple of times in a row.

> 
> I.e., ideally, it should be reproducible from the command line on the host
> natively, otherwise we'll probably end up just having reported the problem --
> which is admittedly also necessary.
> 

That's not where I am yet, only seeing it on my SUSE host.

> 
> That said, the links below (at least one of which appears to be an automated
> translation from English) refer to the same problem in the context of
> Singularity, which would point rather into the direction of qemu. What is
> actually the size of RAM available to the guest in our case? Apt has the
> history of requiring considerable amounts of RAM.
> 
> 1. https://issue.life/questions/51632062
> 2. https://bug200.com/post/51632062
> 

Well, we only saw the issue with armhf and mipsel so far, right? I did a lot of 
arm64 builds recently on the very same machine, which are qemu-based, and none 
failed this way. I do not know how much virtual memory qemu-user leaves the 
application, but it is already constrained to 32-bit.

Hmm, there some tunables to qemu-user. I'm currently playing with 
QEMU_RESERVED_VA, will keep you posted...

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: isar-bootstrap failures with debian-buster targets

[Jan Kiszka]

On 03.09.19 18:23, [ext] Jan Kiszka wrote:
> On 03.09.19 17:44, Baurzhan Ismagulov wrote:
>> On Tue, Aug 27, 2019 at 09:10:21AM +0200, Jan Kiszka wrote:
>>>> FWIW, this also occurred on an internal Jenkins machine (stretch host, stretch
>>>> workspace, no docker). This happening only with buster target, we have only two
>>>> candidates: bug in qemu or bug Debian, no?
>>>
>>> Yeah, stretch host does not seem to make a difference. I've seen it
>>> meanwhile also with the stretch-based kas image.
>>>
>>> Should we try to file bug? Question is: against which component? debootstrap?
>>
>> I think the best candidate would be the package containing the binary that
>> fails -- presumably apt.
>>
>> For me, the questions are:
>>
>> 1. Can we rule out qemu in some way?
>>
>> 2. Is it reproducible in a simple way?
> 
> Well, for me at least: Just started the OrangePi Zero build of jailhouse-images, 
> which is now buster-based, and it failed again. I suspect I can trigger this now 
> for a couple of times in a row.
> 
>>
>> I.e., ideally, it should be reproducible from the command line on the host
>> natively, otherwise we'll probably end up just having reported the problem --
>> which is admittedly also necessary.
>>
> 
> That's not where I am yet, only seeing it on my SUSE host.
> 
>>
>> That said, the links below (at least one of which appears to be an automated
>> translation from English) refer to the same problem in the context of
>> Singularity, which would point rather into the direction of qemu. What is
>> actually the size of RAM available to the guest in our case? Apt has the
>> history of requiring considerable amounts of RAM.
>>
>> 1. https://issue.life/questions/51632062
>> 2. https://bug200.com/post/51632062
>>
> 
> Well, we only saw the issue with armhf and mipsel so far, right? I did a lot of 
> arm64 builds recently on the very same machine, which are qemu-based, and none 
> failed this way. I do not know how much virtual memory qemu-user leaves the 
> application, but it is already constrained to 32-bit.
> 
> Hmm, there some tunables to qemu-user. I'm currently playing with 
> QEMU_RESERVED_VA, will keep you posted...
> 

Nope, would have been too easy: Even QEMU_RESERVED_VA=4000M, which is close to 
the maximum, makes no difference.

I also tried to reproduce outside of the container, but that did not work so 
far. At least, that reduces the probability that the kernel is the key.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux