From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6643769936836558848 X-Received: by 2002:ac2:4207:: with SMTP id y7mr1103951lfh.7.1549552713579; Thu, 07 Feb 2019 07:18:33 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 2002:a19:520f:: with SMTP id m15ls271215lfb.5.gmail; Thu, 07 Feb 2019 07:18:33 -0800 (PST) X-Google-Smtp-Source: AHgI3Ial2EVA0Ra6Cy4sFQhZXnLdK9kSbvVWjfQ9Os1ErrfOjd0LYADkpn9Haho45oKaUMU2kW6/ X-Received: by 2002:ac2:5215:: with SMTP id a21mr690306lfl.5.1549552713068; Thu, 07 Feb 2019 07:18:33 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1549552713; cv=none; d=google.com; s=arc-20160816; b=a7e+tAUrVImNDTQcmu31Y6zQJWAdwaHfq6bp6UmTIUXZqlxpv+cJRXQBPm89dytx2l MLeCrzUnYTWWuhFNByZjG9I1iIjjVHQTYDMsBy/F47PKWhAkwBi4HIh66DDD1dJ8B+aK wju3abk6AlcwozitcJXuCsec6RZAlYwlaySa9xWWcVHQSt+awi27qT2yjQhvCIrurnxa mjVpB9eiT8O91nxU4WUcbJpz9uRvHmLXYm4Rye+72Vp4Lb0jn7kELCaEIc6KJVdu+4T1 /kmdNS8zHkTiuaMcTJFml8AyL7oR4whc+t+yiqw/lDgzJcp/kvj9ssN7jjdI1k5BDNK+ SOUA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:organization:from:references:cc:to :subject; bh=I7fptz7zMtfBgIf3829SEwMxY1diNxKG5u/eEum0/wg=; b=P807dJxZgKzRpFbgR8M8PpDUeg8Imx5xrApKNuKP/qLCc3Ah+FQZyt9gdPP2TkEdKX W08B9+91c2tZ/Azxd3JnHiQTAUYEihwivG2yK0YJVA7Uld671uzcuTQBZ8FGJqm3D8Y7 CkevSg4Njm9XGdWITbjocOQu58QVj9f04aq11HdDsYbA8uq9qQLI6vOCSUUx7ayF+YUo i7Ch3hsBOGcnbM+b84vgEhDj83m0/pzlRcT/c9cpLT0I7LiNEJ4A5WCEyqfpXSn/kfSg AJQPPPyBZ44nBkPOK/T1klNIp6hT+1CrNBJ5zPa61I0PchYv3CTLdPzDtt7Y7ItTBaNz jwnA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: best guess record for domain of mosipov@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=mosipov@ilbers.de Return-Path: Received: from shymkent.ilbers.de (shymkent.ilbers.de. [85.214.156.166]) by gmr-mx.google.com with ESMTPS id q13si498079lfd.2.2019.02.07.07.18.32 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 07 Feb 2019 07:18:32 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of mosipov@ilbers.de designates 85.214.156.166 as permitted sender) client-ip=85.214.156.166; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: best guess record for domain of mosipov@ilbers.de designates 85.214.156.166 as permitted sender) smtp.mailfrom=mosipov@ilbers.de Received: from [192.168.50.163] (d51A48A80.access.telenet.be [81.164.138.128]) (authenticated bits=0) by shymkent.ilbers.de (8.15.2/8.15.2/Debian-8) with ESMTPSA id x17FIUlr022301 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 7 Feb 2019 16:18:31 +0100 Subject: Re: Another CI splat: do_cache_base_repo: Failed To: "Hombourger, Cedric" , Jan Kiszka Cc: isar-users References: <2718374f-01fb-0beb-4b31-08d54d24e19a@siemens.com> <4cee612c-4e19-2035-a79b-65aca10f774e@siemens.com> From: "Maxim Yu. Osipov" Organization: ilbers GmbH Message-ID: <79d38b24-9d40-e54c-608d-3e71fc7bf7f3@ilbers.de> Date: Thu, 7 Feb 2019 16:18:31 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on shymkent.ilbers.de X-TUID: XiD4yhYljCpS Hi Cedric, Thank you for fast feedback. I agree with you - race may happen in populate_base_repo. Your patch is very welcome. Thanks, Maxim. On 2/7/19 2:22 PM, Hombourger, Cedric wrote: > > Don't we need to add: > > do_cache_base_repo[lockfiles] = "${REPO_BASE_DIR}/isar.lock" > > Your CI is building multiple images in parallel, this means that we will end-up having multiple processes populate the base apt cache concurrently > We'd therefore need to serialize accesses to the repo > > I am happy to generate a patch if you believe my theory makes sense > > Cedric > > -----Original Message----- > From: Jan Kiszka [mailto:jan.kiszka@siemens.com] > Sent: Wednesday, January 16, 2019 5:29 AM > To: isar-users ; Hombourger, Cedric ; Maksim Osipov > Subject: Re: Another CI splat: do_cache_base_repo: Failed > > On 07.01.19 16:00, [ext] Jan Kiszka wrote: >> Hi, >> >> Familiar (even if rare)? >> >> NOTE: recipe isar-image-base-1.0-r0: task do_cache_base_repo: Started >> ERROR: mc:qemuarm-stretch:isar-image-base-1.0-r0 do_cache_base_repo: >> Function >> failed: do_cache_base_repo (log file is located at >> /builds/ebsy/debian/isar/build/tmp/work/debian-stretch-armhf/isar-imag >> e-base/temp/log.do_cache_base_repo.55127) >> >> ERROR: Logfile of failure stored in: >> /builds/ebsy/debian/isar/build/tmp/work/debian-stretch-armhf/isar-imag >> e-base/temp/log.do_cache_base_repo.55127 >> >> Log data follows: >> | DEBUG: Executing shell function do_cache_base_repo The lock file >> '/builds/ebsy/debian/isar/build/downloads/base-apt/db/debian/lockfile' >> already exists. There might be another instance with the >> | same database dir running. To avoid locking overhead, only one >> | process can access the database at the same time. Do not delete the >> | lock file unless you are sure no other version is still running! >> | There have been errors! >> | WARNING: exit code 239 from a shell command. >> | ERROR: Function failed: do_cache_base_repo (log file is located at >> /builds/ebsy/debian/isar/build/tmp/work/debian-stretch-armhf/isar-imag >> e-base/temp/log.do_cache_base_repo.55127) >> >> NOTE: recipe isar-image-base-1.0-r0: task do_cache_base_repo: Failed >> ERROR: Task >> (multiconfig:qemuarm-stretch:/builds/ebsy/debian/isar/meta-isar/recipe >> s-core/images/isar-image-base.bb:do_cache_base_repo) >> failed with exit code '1' >> >> >> Cedric, you once sent a related patch ("dpkg: acquire lock when >> calling >> reprepro") that seems like it was never merged. Was that related to a >> similar error? > > Happened again yesterday, and even the retry failed now. Seems we have at least a better chance to reproduce this... Any comments on this bug? > > Jan > > -- > Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux > -- Maxim Osipov ilbers GmbH Maria-Merian-Str. 8 85521 Ottobrunn Germany +49 (151) 6517 6917 mosipov@ilbers.de http://ilbers.de/ Commercial register Munich, HRB 214197 General Manager: Baurzhan Ismagulov