From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6521574336665485312 X-Received: by 10.25.202.77 with SMTP id h13mr183855lfj.21.1518548554715; Tue, 13 Feb 2018 11:02:34 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.25.210.77 with SMTP id j74ls412265lfg.4.gmail; Tue, 13 Feb 2018 11:02:34 -0800 (PST) X-Google-Smtp-Source: AH8x2243ZAv7+lbrUfW3+RUZWNtqAb41c6FuGlbMsxhv5KkfppuAU8UY24Wi2JWrnwJiicjmY2QH X-Received: by 10.25.208.132 with SMTP id h126mr190460lfg.33.1518548554121; Tue, 13 Feb 2018 11:02:34 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1518548554; cv=none; d=google.com; s=arc-20160816; b=BoPPRoV7pBOsQBeEdb+SdLrL5tpi5q4DIFnYoRTIPV32ffhe+Tc2zFNEpu1pSLmYjt RtN384QlXOpHXXCAe4TMxFlXpkHOX9N6klkK+mvpihVyte9NMztb/D+vavLSXaVEyWBU NzFeA1DqQXHrl/tFAD/R76OI/6KgoU8ZWvc5gYMZki9RL1Dxooq4xAFGuD75TlLVvFP7 IIlqnRj95fx2QZx7CqB9FZ8KAOiS0L73/wpNrAufiDQYLaJO9CtMnAOqU0os5vxJPltU 0OHIGZBPDyxCxH6c1oZhKxCGqKtbkaXejmnHDFZQgm4x6DHyGDXAliiOCI4K0yG3nK3l jfbg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject :arc-authentication-results; bh=V4pEEQd+RjZqe2slwSkVCq0UZySlnai8mk6M7ts+t6g=; b=geVLgcXOt7FmosGYNCTsrSfvo8vduJg6bJWObLBoUzA5OhWcEyn7c7/lo9HFvYboD3 AtaADTGBsR6klAIkUkyM8N614sMNrJc8SmKNqACvmWq/LBWgRLPwqeid8fIQDE9oaPhh MSAu0UH8XbLtvJSeWfbU0WF62YWDYuakkUKM2IDhQNSeIkJMfiF5ecqhsAVYhd9ajF+f 4bE1JnFLCE7q/jCVYrH7QOgzk3TcfE/OQzmkZjM3H//wA0LNV5ksAzo8OjCo37B+cxdp tbSgR8SejgKISy7+1EftM47gRwim3x8rD6y8E1vo1P6kPl90FodvIWr73v9e7R38giyI TTzw== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Return-Path: Received: from aqmola.ilbers.de (aqmola.ilbers.de. [85.214.62.211]) by gmr-mx.google.com with ESMTPS id 25si659515ljt.1.2018.02.13.11.02.33 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 13 Feb 2018 11:02:33 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) client-ip=85.214.62.211; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Received: from [10.0.2.15] ([188.227.110.165]) (authenticated bits=0) by aqmola.ilbers.de (8.14.4/8.14.4/Debian-4+deb7u1) with ESMTP id w1DJ2UOl007159 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Tue, 13 Feb 2018 20:02:31 +0100 Subject: Re: [PATCH 4/6] Enable proper rebuilds on dependency changes To: Jan Kiszka , isar-users References: <7d0504da7640612abded5ffe1e1b3c6d10168b49.1518422347.git.jan.kiszka@siemens.com> <73bd323b-0dc2-a84a-ae5a-59177cfe6921@ilbers.de> <4b84a0e3-df9e-8343-ff09-7b74474a0e95@siemens.com> <20278b99-b41b-ee89-b5cd-eafc2019875d@siemens.com> From: Alexander Smirnov Message-ID: Date: Tue, 13 Feb 2018 22:02:25 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <20278b99-b41b-ee89-b5cd-eafc2019875d@siemens.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: X85VN1dcurKP On 02/13/2018 09:44 PM, Jan Kiszka wrote: > On 2018-02-13 19:08, Alexander Smirnov wrote: >> On 02/13/2018 08:57 PM, Jan Kiszka wrote: >>> On 2018-02-13 17:21, Jan Kiszka wrote: >>>> On 2018-02-13 14:03, Alexander Smirnov wrote: >>>>> On 02/12/2018 10:59 AM, Jan Kiszka wrote: >>>>>> From: Jan Kiszka >>>>>> >>>>>> Install a basichash as signature handler and set the stamp policy to >>>>>> full - and suddenly we get proper rebuilds of the image and all >>>>>> affected >>>>>> packages when some recipe down the dependency chain changed! >>>>>> >>>>>> We are using the legacy bitbake mechanism here as we do not have >>>>>> setscene machinery like OE. Still good enough for us. >>>>>> >>>>> >>>>> BTW: have you tried this? Two years ago I tried it in Yocto and it >>>>> didn't work properly, not all the dependents were rebuilt. If it works, >>>>> it's a very good feature! >>>> >>>> Yes, I'm using it since this weekend for both Isar as well as >>>> jailhouse-images development, and it helped a lot already. The only >>>> limitation I found, but I do not remember right now if that isn't >>>> inherent to bitbake, is that it does not detect changes in files that >>>> the recipes carries in its SRC_URI (file://...). Then you need "-c >>>> clean" for the affected target - which now works as well. >>>> >>> >>> Found another issue, which worked surprisingly well so far despite being >>> fairly broken: do_setup_mounts is not re-run on rebuilds. >> >> Investigating exactly this issue. It occures for clean builds also. My >> guess is patch #3, currently I'm building with only 2 first applied. >> With 3rd one applied build failed. >> > > That's not the right approach, the concept of the setup_mounts task is > broken. Why you think it's broken? Just tested, it works good: --- a/meta/recipes-devtools/buildchroot/buildchroot.bb +++ b/meta/recipes-devtools/buildchroot/buildchroot.bb @@ -88,6 +88,8 @@ do_setup_mounts() { sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt sudo mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${BUILDCHROOT_DIR}/dev sudo mount -t proc none ${BUILDCHROOT_DIR}/proc + + echo XXX > /tmp/test } Applying this patch I have /tmp/test on re-run. > > I'm still testing, but it looks like as if this massive simplification > can also fix the rebuild issue: The issue is with the patch#2, dropping it fixes the build. Probably the issue is that do_clean has no before/after specifier in the pipeline. Still testing. Alex > > diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass > index a45fbce..aa46f45 100644 > --- a/meta/classes/dpkg-base.bbclass > +++ b/meta/classes/dpkg-base.bbclass > @@ -2,7 +2,7 @@ > # Copyright (C) 2017 Siemens AG > > # Add dependency from buildchroot creation > -do_build[depends] = "buildchroot:do_setup_mounts" > +do_build[depends] = "buildchroot:do_build" > > # Add dependency between Isar recipes > DEPENDS ?= "" > @@ -25,6 +25,14 @@ do_build() { > mkdir -p ${BUILDROOT} > sudo mount --bind ${WORKDIR} ${BUILDROOT} > > + # These tests may race with parallel builds, so ignore errors. > + grep -q ${BUILDCHROOT_DIR}/isar-apt /proc/mounts && > + sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt || true > + grep -q ${BUILDCHROOT_DIR}/dev /proc/mounts && > + sudo mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${BUILDCHROOT_DIR}/dev || true > + grep -q ${BUILDCHROOT_DIR}/proc /proc/mounts && > + sudo mount -t proc none ${BUILDCHROOT_DIR}/proc || true > + > dpkg_runbuild > > sudo umount ${BUILDROOT} 2>/dev/null || true > diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb > index 570c0ad..cd91228 100644 > --- a/meta/recipes-devtools/buildchroot/buildchroot.bb > +++ b/meta/recipes-devtools/buildchroot/buildchroot.bb > @@ -60,7 +60,9 @@ do_build() { > -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \ > "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf" > > - do_setup_mounts > + sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt > + sudo mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${BUILDCHROOT_DIR}/dev > + sudo mount -t proc none ${BUILDCHROOT_DIR}/proc > > # Create root filesystem > sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f "${WORKDIR}/multistrap.conf" > @@ -70,30 +72,4 @@ do_build() { > > # Configure root filesystem > sudo chroot ${BUILDCHROOT_DIR} /configscript.sh > - > - do_cleanup_mounts > -} > - > -# Invalidate stamp for do_setup_mounts before each build start. > -# This will guarantee that this function will be executed once > -# per build. > -python __anonymous() { > - stamp = d.getVar("STAMP") + ".do_setup_mounts." + d.getVarFlag("do_setup_mounts", 'stamp-extra-info') > - os.remove(stamp) if os.path.exists(stamp) else None > -} > - > -do_setup_mounts[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" > - > -do_setup_mounts() { > - sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt > - sudo mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${BUILDCHROOT_DIR}/dev > - sudo mount -t proc none ${BUILDCHROOT_DIR}/proc > -} > - > -addtask setup_mounts after do_build > - > -do_cleanup_mounts() { > - sudo umount ${BUILDCHROOT_DIR}/isar-apt 2>/dev/null || true > - sudo umount ${BUILDCHROOT_DIR}/dev 2>/dev/null || true > - sudo umount ${BUILDCHROOT_DIR}/proc 2>/dev/null || true > } > > > Jan > -- With best regards, Alexander Smirnov ilbers GmbH Baierbrunner Str. 28c D-81379 Munich +49 (89) 122 67 24-0 http://ilbers.de/ Commercial register Munich, HRB 214197 General manager: Baurzhan Ismagulov