From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6724331065971834880 X-Received: by 2002:a1c:7e85:: with SMTP id z127mr1724734wmc.95.1566329611977; Tue, 20 Aug 2019 12:33:31 -0700 (PDT) X-BeenThere: isar-users@googlegroups.com Received: by 2002:adf:eb44:: with SMTP id u4ls5830309wrn.2.gmail; Tue, 20 Aug 2019 12:33:31 -0700 (PDT) X-Google-Smtp-Source: APXvYqwNgZDxo5xU6xK8iONOqQUVl/3J9JeSI0eMdD1aTdkGZNt3RDeuDGCVmEbPbrbwlU6p7NWq X-Received: by 2002:a5d:40cd:: with SMTP id b13mr38018677wrq.236.1566329611406; Tue, 20 Aug 2019 12:33:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1566329611; cv=none; d=google.com; s=arc-20160816; b=RcrNoallSgZvE3ynAOu0mXEVlRCUhha1oo+Qxsr3hI647RHkhBLeco7njdHHYZz1TQ MiNpSzG+gu9uuVC/sWrXN+glgL+Yi0iTbBL/FsGCJDsp5y9qwznLb7x+WqtkIFKPkses soboOdHcICLkg55uOI/pxj//ZhdpLkWjpqQ64/o7cgh2Zfgxhug4g4XvZtfsJ7KKpVtv jv2R7ImZmGe2gRNaGhzFYokXsFSbP13MAVq5PTDfH4N+i/Gpqn1VVksP/9klIyroRt05 K00r5gu2SmQJisIiOCRS0H/7DS0vg7uCphu88jCMkqb2l4A0paFFdynl6GTdktwIEl6Z wXcA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:references:to:from:subject; bh=jHEysaWAFTJ/sdf96mJ+6k6aDiDXcKj8zVSSd8d7sbY=; b=jnBa3yQSjXVCcb9GpxiJ7nR3ufIqG4Jg4Grs3m1CwJs/NoENDyzpzMjEqHzrRwx9/T G0f/mZ4YQ/1JXYK2Dm53TSdAeEbufEPgKKeAcRGWk3imBDdTkNvGE+CAul3xotQYza/f lK+s0py/VWaHQwdkw0cL9ZcoReSNdqvB3MYE4sQlX9yH1TyuZ06dG32MUHygvs5aCI3L xNbvDU1D+EXKvfdqs1z2oVPIFU2o/NzP3F3ItPMz5XW/I+fU8yy7b5n/wvJ780m1r0OF cQetdFduG4KHr6Q7B+su0zYGrC5RseHSbwZ90YSiRcOmk35MJ4ZQvGH4ujIwKwtKCKuo Kucg== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Return-Path: Received: from goliath.siemens.de (goliath.siemens.de. [192.35.17.28]) by gmr-mx.google.com with ESMTPS id l9si63250wmc.0.2019.08.20.12.33.31 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 20 Aug 2019 12:33:31 -0700 (PDT) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.28 as permitted sender) client-ip=192.35.17.28; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=siemens.com Received: from mail1.sbs.de (mail1.sbs.de [192.129.41.35]) by goliath.siemens.de (8.15.2/8.15.2) with ESMTPS id x7KJXU2M015006 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 20 Aug 2019 21:33:30 +0200 Received: from [139.25.68.37] (md1q0hnc.ad001.siemens.net [139.25.68.37] (may be forged)) by mail1.sbs.de (8.15.2/8.15.2) with ESMTP id x7KJXULC001872; Tue, 20 Aug 2019 21:33:30 +0200 Subject: Re: isar-bootstrap fails to rebuild after config changes From: Jan Kiszka To: Claudius Heine , isar-users , Claudius Heine , Maksim Osipov References: <52a51b4f-483a-3b42-8edf-3e0db8ffc722@siemens.com> Message-ID: <94e2ed05-76df-2071-9528-e3765e1b8de8@siemens.com> Date: Tue, 20 Aug 2019 21:33:29 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0 MIME-Version: 1.0 In-Reply-To: <52a51b4f-483a-3b42-8edf-3e0db8ffc722@siemens.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-TUID: NLpUiWCdIhhe On 13.08.19 10:14, [ext] Jan Kiszka wrote: > On 13.08.19 10:01, Claudius Heine wrote: >> Hi Jan, >> >> On 12/08/2019 19.18, [ext] Jan Kiszka wrote: >>> Hi, >>> >>> I noticed today that I'm not getting a new bootstrap rootfs after adding a >>> repo to DISTRO_APT_SOURCES. Digging deeper, this revealed missing STAMPCLEAN >>> setting, but also more: >>> >>> Why are we probing for the existence of the DEPLOY_ISAR_BOOTSTRAP link in >>> do_apt_config_prepare(), Claudius? This obviously exists after a first run, >>> and that blocks refreshing all the apt settings if they have changed. >> >> I do not know. That was added by a patch of Maxim >> (ab0a1c8c357a18ddd67fe4c0efc66a296d93dae2), maybe he knows. >> >>      isar-bootstrap: Eliminate multiple debootstraps for the same distro/host >> >>      After applying this patch only single debootstrap for >>      particular platform is created and pointed by corresponding >>      link under DEPLOY_DIR_IMAGE. >> >>      For proper locking of parallel builds shell tasks >>      apt_config_install, set_locale, setup_chroot, apt_update >>      and deploy were collapsed to common debootstrap helper. >>      Due to problems in bitbake with shell functions expansion >>      under quotes these functions were substituted  by their >>      bodies. >> >>      Signed-off-by: Maxim Yu. Osipov >> >> That is by the way a pretty big patch. Also parts of it could probably be >> reverted, since the 'problems in bitbake with shell functions expansion under >> quotes' are resolved in the current version by removing these quotes IIUC. >> >>> There is CLEANFUNCS set with clean_deploy(), but that is only run on true >>> do_clean(), not on rebuilds. >> >> Right. >> >>> >>> And finally, when I look at isar_bootstrap(), run by do_bootstrap(), it >>> protects checking and creating DEPLOY_ISAR_BOOTSTRAP with a lock. But >>> do_apt_config_prepare() runs lockless. This cannot work reliably. >> >> When I initially wrote isar-bootstrap, it didn't contain any looks. IIRC the >> locks where added when I wasn't active with isar and I haven't yet looked into >> why exactly they are needed and what they protect against etc. So you know >> probably more about them than me. > > Ah, sorry, wrongly accounted you for this. > > Maxim, can you comment on the design idea and the issues I found? > Actually, I do not even see the point of locking here: If a bootstrap target is shared, only one multiconfig target will request its build and the others will wait automatically for the result. That's what bitbake is supposed to do for us. If it's not shared, there is no conflict to use locks on. We only need locks if different tasks of different recipes compete for common resources. Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux