From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6519007876157014016 X-Received: by 10.46.54.8 with SMTP id d8mr230909lja.28.1517941900537; Tue, 06 Feb 2018 10:31:40 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.25.16.106 with SMTP id f103ls1177255lfi.9.gmail; Tue, 06 Feb 2018 10:31:39 -0800 (PST) X-Google-Smtp-Source: AH8x2269lLWs6GhSwUMUQw75TdwVojsBHC1l0MxCzeJux/XpcwpO0Hx//iWiY1X++lfpzq17Yy4m X-Received: by 10.25.202.77 with SMTP id h13mr246977lfj.21.1517941899403; Tue, 06 Feb 2018 10:31:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1517941899; cv=none; d=google.com; s=arc-20160816; b=LY3PsUWoRGes2tv8c2zHt6/ySEu2ZWWSC9Tb00PF99BnMrr1ok0SznU1mWbzHUAeWw vM77gs/jLRYKj0cUZVbhRj8Vo+o0BWSUjOWMl3B0ojiYWwTu6GE8qqVM839gk21J6e3o 6FXGte4l5z0ZgkpR/xZEqnncJW32kIVsqSi/fDaMA8Qrk+6oPEby81hLNdj897keoZd7 yGcDoTi0Tu/3jvIP0Kwf52+XO9ZvOu8pOffT3UeE9yCm/o6a7sSysuYjSU7ghXJY3Ics vPxhbAj6A3pE/CYU1nJ6sCBLFMbRdEn2cyHGKOWCf23jGf+mSAzsk9Mwnyqgjn+eNfqM yQKA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject :arc-authentication-results; bh=4BX2v+caI0h2a+uFyCPr8NNYC+r9K878j00DPr4eEyw=; b=PYd/26DPAAmwNw/el8c+Ct435AvMA0ziFzqzCotSlaPsHjzGaIaiO60FVu95YtnfMo HMqrhbpEwRLmA1Bivx7gm/aEZwkpIKO2JCU2/P2ZSm5xcJ+FxRWJz1wqfmOTUQvaG9pf +vWYdsNWdnGSdQE/N58ZkEpHgarwbtH5ZUbF+3A7YUo+JZI0KllG17nFX0y19toKCFHe 3wRoBlYFhzykK5p5x19E77FrH5UJsqBKYj5/3vUj0Qh4xB+xQmMKFWYYhWPiV0LSe0ux Jg1Z1WmQTKLcWK+6oMBaCWDLjr4CTqpZpDQfhxPbZKHb8pbNpNCIZUfCA31Yp105Bsw2 otCA== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com Return-Path: Received: from goliath.siemens.de (goliath.siemens.de. [192.35.17.28]) by gmr-mx.google.com with ESMTPS id w29si837080lfc.5.2018.02.06.10.31.39 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 06 Feb 2018 10:31:39 -0800 (PST) Received-SPF: pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.28 as permitted sender) client-ip=192.35.17.28; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of jan.kiszka@siemens.com designates 192.35.17.28 as permitted sender) smtp.mailfrom=jan.kiszka@siemens.com Received: from mail2.siemens.de (mail2.siemens.de [139.25.208.11]) by goliath.siemens.de (8.15.2/8.15.2) with ESMTPS id w16IVcAe015518 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 6 Feb 2018 19:31:38 +0100 Received: from [139.25.68.37] (md1q0hnc.ad001.siemens.net [139.25.68.37] (may be forged)) by mail2.siemens.de (8.15.2/8.15.2) with ESMTP id w16IVcft005316; Tue, 6 Feb 2018 19:31:38 +0100 Subject: Re: [PATCH 3/5 v4] buildchroot: Enable isar-apt To: Alexander Smirnov , isar-users@googlegroups.com References: <20180205095931.23903-1-asmirnov@ilbers.de> <20180206135725.28502-1-asmirnov@ilbers.de> <3fa05bea-15fc-3caa-24c7-dfc262b9909a@siemens.com> <5cb82a86-2abe-5f54-4650-7f9c65cb252b@ilbers.de> From: Jan Kiszka Message-ID: Date: Tue, 6 Feb 2018 19:31:38 +0100 User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); de; rv:1.8.1.12) Gecko/20080226 SUSE/2.0.0.12-1.1 Thunderbird/2.0.0.12 Mnenhy/0.7.5.666 MIME-Version: 1.0 In-Reply-To: <5cb82a86-2abe-5f54-4650-7f9c65cb252b@ilbers.de> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-TUID: e1kriB4bckx2 On 2018-02-06 19:29, Alexander Smirnov wrote: > On 02/06/2018 08:31 PM, Jan Kiszka wrote: >> On 2018-02-06 14:57, Alexander Smirnov wrote: >>> 8<-- >>> >>> Branch to test: asmirnov/devel >>> >>> 8<-- >>> >>> This patch provides access to isar-apt from buildchroot. It does the >>> following: >>>   - mount isar-apt during buildchroot building. >>>   - umount isar-apt using bitbake events. >>> >>> Also it needs to keep Isar build tree clean from any mounts despite >>> on whether >>> build succeed of failed. bitbake provides various events that could >>> trigger >>> custom python hooks. In this patch BuildCompleted event is used, >>> which happened >>> when bitbake finished its execution despite on the result. >>> >>> Signed-off-by: Alexander Smirnov >>> --- >>>   meta/classes/dpkg-base.bbclass                     |  2 +- >>>   meta/classes/isar-events.bbclass                   | 20 >>> ++++++++++++++++ >>>   meta/conf/isar-bitbake.conf                        |  3 +++ >>>   meta/recipes-devtools/buildchroot/buildchroot.bb   | 28 >>> +++++++++++++++++++++- >>>   .../buildchroot/files/multistrap.conf.in           |  8 ++++++- >>>   5 files changed, 58 insertions(+), 3 deletions(-) >>>   create mode 100644 meta/classes/isar-events.bbclass >>> >>> diff --git a/meta/classes/dpkg-base.bbclass >>> b/meta/classes/dpkg-base.bbclass >>> index 026028f..5d5a924 100644 >>> --- a/meta/classes/dpkg-base.bbclass >>> +++ b/meta/classes/dpkg-base.bbclass >>> @@ -2,7 +2,7 @@ >>>   # Copyright (C) 2017 Siemens AG >>>     # Add dependency from buildchroot creation >>> -do_build[depends] = "buildchroot:do_build" >>> +do_build[depends] = "buildchroot:do_setup_mounts" >>>     # Each package should have its own unique build folder, so use >>>   # recipe name as identifier >>> diff --git a/meta/classes/isar-events.bbclass >>> b/meta/classes/isar-events.bbclass >>> new file mode 100644 >>> index 0000000..55fc106 >>> --- /dev/null >>> +++ b/meta/classes/isar-events.bbclass >>> @@ -0,0 +1,20 @@ >>> +# Isar event handlers. >>> +# >>> +# This software is a part of ISAR. >>> +# Copyright (C) 2015-2017 ilbers GmbH >>> + >>> +addhandler isar_handler >>> + >>> +python isar_handler () { >>> +    import subprocess >>> + >>> +    devnull = open(os.devnull, 'w') >>> + >>> +    if isinstance(e, bb.event.BuildCompleted): >>> +        bchroot = d.getVar('BUILDCHROOT_DIR', True) >>> + >>> +        # Clean up buildchroot >>> +        subprocess.call('/usr/bin/sudo /bin/umount ' + bchroot + >>> '/isar-apt || /bin/true', stdout=devnull, stderr=devnull, shell=True) >>> + >>> +    devnull.close() >>> +} >>> diff --git a/meta/conf/isar-bitbake.conf b/meta/conf/isar-bitbake.conf >>> index b853c88..36cca29 100644 >>> --- a/meta/conf/isar-bitbake.conf >>> +++ b/meta/conf/isar-bitbake.conf >>> @@ -27,6 +27,9 @@ BUILDCHROOT_DIR = >>> "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot/rootfs" >>>   # Setup our default hash policy >>>   BB_SIGNATURE_HANDLER ?= "noop" >>>   +# Add event handlers for bitbake >>> +INHERIT += "isar-events" >>> + >>>   include conf/local.conf >>>   include conf/multiconfig/${BB_CURRENT_MC}.conf >>>   include conf/machine/${MACHINE}.conf >>> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb >>> b/meta/recipes-devtools/buildchroot/buildchroot.bb >>> index 51f9d5d..304c67e 100644 >>> --- a/meta/recipes-devtools/buildchroot/buildchroot.bb >>> +++ b/meta/recipes-devtools/buildchroot/buildchroot.bb >>> @@ -32,7 +32,9 @@ BUILDCHROOT_PREINSTALL ?= "gcc \ >>>   WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}" >>>     do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" >>> -do_build[dirs] = "${WORKDIR}/hooks_multistrap" >>> +do_build[dirs] = "${WORKDIR}/hooks_multistrap \ >>> +                  ${BUILDCHROOT_DIR}/isar-apt" >>> +do_build[depends] = "isar-apt:do_cache_config" >>>     do_build() { >>>       E="${@ bb.utils.export_proxies(d)}" >>> @@ -67,6 +69,8 @@ do_build() { >>>       } >>>       trap '_do_build_cleanup' EXIT >>>   +    do_setup_mounts >>> + >>>       # Create root filesystem >>>       sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f >>> "${WORKDIR}/multistrap.conf" >>>   @@ -76,4 +80,26 @@ do_build() { >>>       # Configure root filesystem >>>       sudo chroot ${BUILDCHROOT_DIR} /configscript.sh >>>       _do_build_cleanup >>> + >>> +    do_cleanup_mounts >>> +} >>> + >>> +# Invalidate stamp for do_setup_mounts before each build start. >>> +# This will guarantee that this function will be executed once >>> +# per build. >>> +python __anonymous() { >>> +    stamp = d.getVar("STAMP") + ".do_setup_mounts." + >>> d.getVarFlag("do_setup_mounts", 'stamp-extra-info') >>> +    os.remove(stamp) if os.path.exists(stamp) else None >>> +} >>> + >>> +do_setup_mounts[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" >>> + >>> +do_setup_mounts() { >>> +    sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} >>> ${BUILDCHROOT_DIR}/isar-apt >>> +} >>> + >>> +addtask setup_mounts after do_build >>> + >>> +do_cleanup_mounts() { >>> +    sudo umount ${BUILDCHROOT_DIR}/isar-apt 2>/dev/null || true >>>   } >>> diff --git >>> a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in >>> b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in >>> index a0b28e3..480a4b8 100644 >>> --- a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in >>> +++ b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in >>> @@ -6,7 +6,7 @@ noauth=true >>>   unpack=true >>>   ignorenativearch=true >>>   bootstrap=##DISTRO_MULTICONF_BOOTSTRAP## >>> -aptsources=##DISTRO_MULTICONF_APTSOURCES## >>> +aptsources=isar-apt ##DISTRO_MULTICONF_APTSOURCES## >>>   configscript=##CONFIG_SCRIPT## >>>   setupscript=##SETUP_SCRIPT## >>>   hookdir=##DIR_HOOKS## >>> @@ -29,3 +29,9 @@ source=##DISTRO_APT_SOURCE_SEC## >>>   suite=##DISTRO_SUITE##/updates >>>   components=##DISTRO_COMPONENTS## >>>   omitdebsrc=true >>> + >>> +[isar-apt] >>> +source=file:///isar-apt >>> +suite=isar >>> +components=main >>> +omitdebsrc=true >>> >> >> Looks good! I'm testing it via your devel branch, and I also >> successfully rebased "Permanently mount '/dev', '/sys' and '/proc'" on >> top. >> > > Thanks! > >> Regarding the remaining cleanup of WORKDIR mounting: Can't each >> dpkg-base class register their own cleanup handler in the same fashion? >> > > I've remembered, that for Jenkins I've used something following to clean > the build automatically: > > 8<-- > > #!/bin/bash > > W=${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH} > > for m in $(cat /proc/mounts | grep $W); do >     m=$(echo $m | cut -d ' ' -f 2) >     umount $m > done > > 8<-- > > There are some facts: > > 1. Proc folder must be available for correct Isar operating. > 2. /proc/mounts contains all the active mounts. > 3. Folder $W contains only mounts created by current build. > > Based on this, I think if we insert the code above to the event handler, > it will be robust enough solution to clean up everything. Also we should > not anymore care if somebody adds some extra mount. > > What do you think? Could work... Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux