From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6834022395126218752 X-Received: by 2002:a19:4344:: with SMTP id m4mr483447lfj.181.1600715869566; Mon, 21 Sep 2020 12:17:49 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:ac2:5c44:: with SMTP id s4ls796199lfp.3.gmail; Mon, 21 Sep 2020 12:17:48 -0700 (PDT) X-Google-Smtp-Source: ABdhPJydRQ9Zj55ldJbgg86VzWP3rli/S/KabArwr9rgVBothqMpPDcEveM2/wTyl15N/ULNS/ik X-Received: by 2002:ac2:5e2b:: with SMTP id o11mr477819lfg.24.1600715868295; Mon, 21 Sep 2020 12:17:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1600715868; cv=none; d=google.com; s=arc-20160816; b=qum+sbQ7PcqGsWpnOxQtWiAvXtdexp5PyHQtPck3v8F6YKOZMct+ajZiYICoBAmRGA Uzx+q++UgJ5dDzHa9Tq/lQiWET2Mh3OxJMMwHHRZ/ZwZzCr1rQYvbonj8XkPdVZBMzKI 2EmRKiVliBsn2BwaPiOhlucii6Nlgker1zjcldiSfBxP2wBIdGNHjyn+k0E6O20LA/Mg 6VMGcG0WSmmj1bJ8px/8YYNPVHa2yqmEEdavBRT1Xs7ysDlr0E8RMxFOjYUd85KuGxDL Z6MWZezLOF9OEJYoBHQtPXN45iGBxkUeJfOWexdsosaexoR85lved8/yaaJD2PWEB0hc selA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:references:to:from:subject; bh=ZDEtJz3P6N7GnYG5Vd0+Dq89lvJ8FWX2n6Ifhz2kimQ=; b=B9b3vr51GJZ8yVXswJb8aTE+oB8uKnITf/iZnOi8A3p86Ga055qG5vK1f7BVhKKqh9 0+Imf1tEKbTk1/+YR/bPK18nL6ADioUGA+03gWFQh6ssJpge/roOKjKtJCrHbFcMDOpe PF6f/kS2vY3yKx/W+ASnlM9R2RSBGqfdykQz+XM1OaYDa81CCJSFquz/ckzKpb2ySvf/ u2nJZOoUJ1Pj5G8XpmodcKcHY5JSQwzwPHDLrZPgF3zm16naxgrIiuZtmijl5/a4CaG/ 7kufrym1qZJubuQ8dy7Iuomspt59ojJ+mFYwvsVD3JhfNU06DBFhmF15sCmoR12j4h3K 9Lbg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from goliath.siemens.de (goliath.siemens.de. [192.35.17.28]) by gmr-mx.google.com with ESMTPS id h22si326634ljh.7.2020.09.21.12.17.48 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 21 Sep 2020 12:17:48 -0700 (PDT) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.28 as permitted sender) client-ip=192.35.17.28; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail2.sbs.de (mail2.sbs.de [192.129.41.66]) by goliath.siemens.de (8.15.2/8.15.2) with ESMTPS id 08LJHl9c019414 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 21 Sep 2020 21:17:47 +0200 Received: from [167.87.129.190] ([167.87.129.190]) by mail2.sbs.de (8.15.2/8.15.2) with ESMTP id 08LJHhCe011732; Mon, 21 Sep 2020 21:17:45 +0200 Subject: Re: [PATCH v2] dpkg-base: Remove newly deployed debs from buildchroots From: Jan Kiszka To: isar-users , Baurzhan Ismagulov References: <20200823160636.GA11779@yssyq.m.ilbers.de> Message-ID: Date: Mon, 21 Sep 2020 21:17:43 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.12.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: /yJKdYXLrUFJ On 24.08.20 09:16, [ext] Jan Kiszka wrote: > On 23.08.20 18:06, Baurzhan Ismagulov wrote: >> Hello Jan, >> >> On Tue, Aug 11, 2020 at 09:05:59AM +0200, Jan Kiszka wrote: >>> This ensures clean reinstallation after partial rebuilds. >>> >>> A typical error pattern so far was that firmware packages pulled by the >>> buildchroot-target were not updated there on rebuilds, causing the old >>> firmware being deployed into the image. >> >> With this applied, we encounter random build failures downstream, e.g.: >> >> 1. ERROR: mc:pixi-cdl100-buster:desktop-shortcuts-1.0-r0 do_install_builddeps: Execution of 'build/tmp/work/debian-buster-armhf/desktop-shortcuts/1.0-r0/temp/run.do_install_builddeps.19582' failed with exit code 1: >> mk-build-deps: You must have equivs installed to use this program. >> WARNING: exit code 1 from a shell command. >> >> 2. Exception: bb.process.ExecutionError: Execution of 'build/tmp/work/debian-buster-armhf/desktop-shortcuts/1.0-r0/temp/run.dpkg_runbuild.16998' failed with exit code 2: >> dpkg-buildpackage: info: source package desktop-shortcuts >> dpkg-buildpackage: info: source version 1.0 >> dpkg-buildpackage: info: source distribution UNRELEASED >> dpkg-buildpackage: info: source changed by isar-users >> dpkg-architecture: warning: specified GNU system type arm-linux-gnueabihf does not match CC system type x86_64-linux-gnu, try setting a correct CC environment variable >> dpkg-source -I --before-build . >> dpkg-buildpackage: info: host architecture armhf >> fakeroot debian/rules clean >> dh clean >> make: dh: Command not found >> >> Seems we have a race, possibly related to removing packages. What I don't >> understand is how it could affect equivs or debhelper -- we don't do anything >> with them. Ideas? > > Which packages are you (re-)building? Debian packages? Or just custom > dependencies of that desktop-shortcuts package? > > In theory, do_deploy_deb should be locked against do_install_builddeps. > Anything that the latter needs should be available after the former. > Maybe you are missing a build dependency in desktop-shortcuts? Try build > that only over a clean buildchroot, that will tell you. > Meanwhile, I understood the issue is it happened on out CI system as well during nightly runs (that shouldn't have happen, but that a local issue). The logs you cited were incomplete: [...] NOTE: recipe cowsay-git-r0: task do_dpkg_build: Succeeded NOTE: Running task 1617 of 1683 (mc:qemumipsel-buster:/builds/ebsy/debian/isar/meta-isar/recipes-app/cowsay/cowsay_git.bb:do_deploy_deb) NOTE: recipe cowsay-git-r0: task do_deploy_deb: Started [...] NOTE: Running noexec task 1637 of 1683 (mc:rpi-stretch:/builds/ebsy/debian/isar/meta-isar/recipes-core/images/isar-image-base.bb:do_build) [...] ERROR: mc:qemuarm-buster:isar-image-base-1.0-r0 do_rootfs_install: Error executing a python function in exec_python_func() autogenerated: [...] E: Failed to fetch file:/isar-apt/pool/main/c/cowsay/cowsay_3.03+dfsg2-4_all.deb File not found - /isar-apt/pool/main/c/cowsay/cowsay_3.03+dfsg2-4_all.deb (2: No such file or directory) E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing? WARNING: exit code 100 from a shell command. ERROR: Logfile of failure stored in: /builds/ebsy/debian/isar/build/tmp/work/debian-buster-armhf/isar-image-base-qemuarm-ext4-img/1.0-r0/temp/log.do_rootfs_install.32804 NOTE: recipe isar-image-base-1.0-r0: task do_rootfs_install: Failed ERROR: Task (mc:qemuarm-buster:/builds/ebsy/debian/isar/meta-isar/recipes-core/images/isar-image-base.bb:do_rootfs_install) failed with exit code '1' NOTE: recipe cowsay-git-r0: task do_deploy_deb: Succeeded The problem is that we have a false sharing between all archs building cowsay for buster. That is because this demo package is arch "all", but the tasks assume it is arch-specific, thus are building and (re-)deploying it happily in parallel. This is similar to the issue I had with [1]. It is not caused by this patch, it is revealed by it, finally. What can we do? - select a different rebuild-example, one that does not generate "all" packages (i.e. stick our head into the sand) - break up isar-apt on a per-arch basis (as considered for [1]) - find a magic spell that allows to express the commonality between all mc:-: recipes that actually produce the same output I suspect the latter will not happen any time soon, so I'm inclined to fix this via the second option. Jan [1] https://groups.google.com/d/msgid/isar-users/e4ca6d44-8b99-24ba-4fcd-c36cedac1ff7%40siemens.com -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux