From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6519007876157014016 X-Received: by 10.28.4.205 with SMTP id 196mr311949wme.8.1517941787468; Tue, 06 Feb 2018 10:29:47 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.28.170.1 with SMTP id t1ls39459wme.7.gmail; Tue, 06 Feb 2018 10:29:46 -0800 (PST) X-Google-Smtp-Source: AH8x225TLZDiimaK0S1UtjESmgamIF/zY7X2ZM+gQm+fKgx7VtVSl74FYLX1HOl3fZqrDE3MRKAy X-Received: by 10.28.160.18 with SMTP id j18mr328217wme.15.1517941786911; Tue, 06 Feb 2018 10:29:46 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1517941786; cv=none; d=google.com; s=arc-20160816; b=sj4sculEbyA49RS6ifUrW5UKG0zZrbu4wUDYy1kAw9LJ6FgM6BJVmQ6f+YnvIlBchu Iul/XIWpkW6aRbx1bwKdoYPmoKbATYpK7RmIG2jFP20+tsEkqpoqeZo+xY9T3LYNhlgo vLgvd6czYM2MSjWPjTAkzOQ6hIp4PgOxXo6bB81F+a4uT7+jhSmJcXfnN/9DyBTXJXY+ FAgwu82+yzhakFn/v6736TIfK2y6sweQUL0gZ188fskGvERyafglnchLW4DDoLE7ju9c sHBXWcZv7gFUc6FpNblQVypyGZHe/vR5Eo/QaDEE5dzPYPyt2QtueiEH9F0BEjrxeU17 2qKA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject :arc-authentication-results; bh=7GOLfIWc9xb4DREZdmW2tZsg3E8CG61tCLCCzpr/gUY=; b=Qc2NfiMMCJ4BieKbghihOlf1LJhvXaNpIjS6NnQrMUy6uIzqbZ1ZmJan3bYkR9hkxv lzxt0rWXBLCVVGj/KpVXrfU8D1iIq3KP6q8Hg/SsFHJEMiUExOhZ6ClncKHJteIyd5kW Tl3L6DhGbuyLKXgKm67eTl3PDFNR/oXOujYMTrYF3I5L6mTbORNCwogWYLhAfomF4Ld8 JPRBUfn5y/BN+mxqoirX+n+JJUWwGRKIHyG9Y1eAONjXa3K/ckyMbOO3cym9BQwCf8iW iACgK8E4u0BCMKAGe3OyvcCLusZd7dx+KvXMWUB8PH8l+T/Ya8vXXG7iffSYCZinOMAe mfgQ== ARC-Authentication-Results: i=1; gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Return-Path: Received: from aqmola.ilbers.de (aqmola.ilbers.de. [85.214.62.211]) by gmr-mx.google.com with ESMTPS id s5si662251wra.3.2018.02.06.10.29.46 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 06 Feb 2018 10:29:46 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) client-ip=85.214.62.211; Authentication-Results: gmr-mx.google.com; spf=pass (google.com: best guess record for domain of asmirnov@ilbers.de designates 85.214.62.211 as permitted sender) smtp.mailfrom=asmirnov@ilbers.de Received: from [10.0.2.15] ([188.227.110.165]) (authenticated bits=0) by aqmola.ilbers.de (8.14.4/8.14.4/Debian-4+deb7u1) with ESMTP id w16ITgnZ029978 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Tue, 6 Feb 2018 19:29:44 +0100 Subject: Re: [PATCH 3/5 v4] buildchroot: Enable isar-apt To: Jan Kiszka , isar-users@googlegroups.com References: <20180205095931.23903-1-asmirnov@ilbers.de> <20180206135725.28502-1-asmirnov@ilbers.de> <3fa05bea-15fc-3caa-24c7-dfc262b9909a@siemens.com> From: Alexander Smirnov Message-ID: <5cb82a86-2abe-5f54-4650-7f9c65cb252b@ilbers.de> Date: Tue, 6 Feb 2018 21:29:37 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <3fa05bea-15fc-3caa-24c7-dfc262b9909a@siemens.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-TUID: M9GyrYDdq+Ag On 02/06/2018 08:31 PM, Jan Kiszka wrote: > On 2018-02-06 14:57, Alexander Smirnov wrote: >> 8<-- >> >> Branch to test: asmirnov/devel >> >> 8<-- >> >> This patch provides access to isar-apt from buildchroot. It does the >> following: >> - mount isar-apt during buildchroot building. >> - umount isar-apt using bitbake events. >> >> Also it needs to keep Isar build tree clean from any mounts despite on whether >> build succeed of failed. bitbake provides various events that could trigger >> custom python hooks. In this patch BuildCompleted event is used, which happened >> when bitbake finished its execution despite on the result. >> >> Signed-off-by: Alexander Smirnov >> --- >> meta/classes/dpkg-base.bbclass | 2 +- >> meta/classes/isar-events.bbclass | 20 ++++++++++++++++ >> meta/conf/isar-bitbake.conf | 3 +++ >> meta/recipes-devtools/buildchroot/buildchroot.bb | 28 +++++++++++++++++++++- >> .../buildchroot/files/multistrap.conf.in | 8 ++++++- >> 5 files changed, 58 insertions(+), 3 deletions(-) >> create mode 100644 meta/classes/isar-events.bbclass >> >> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass >> index 026028f..5d5a924 100644 >> --- a/meta/classes/dpkg-base.bbclass >> +++ b/meta/classes/dpkg-base.bbclass >> @@ -2,7 +2,7 @@ >> # Copyright (C) 2017 Siemens AG >> >> # Add dependency from buildchroot creation >> -do_build[depends] = "buildchroot:do_build" >> +do_build[depends] = "buildchroot:do_setup_mounts" >> >> # Each package should have its own unique build folder, so use >> # recipe name as identifier >> diff --git a/meta/classes/isar-events.bbclass b/meta/classes/isar-events.bbclass >> new file mode 100644 >> index 0000000..55fc106 >> --- /dev/null >> +++ b/meta/classes/isar-events.bbclass >> @@ -0,0 +1,20 @@ >> +# Isar event handlers. >> +# >> +# This software is a part of ISAR. >> +# Copyright (C) 2015-2017 ilbers GmbH >> + >> +addhandler isar_handler >> + >> +python isar_handler () { >> + import subprocess >> + >> + devnull = open(os.devnull, 'w') >> + >> + if isinstance(e, bb.event.BuildCompleted): >> + bchroot = d.getVar('BUILDCHROOT_DIR', True) >> + >> + # Clean up buildchroot >> + subprocess.call('/usr/bin/sudo /bin/umount ' + bchroot + '/isar-apt || /bin/true', stdout=devnull, stderr=devnull, shell=True) >> + >> + devnull.close() >> +} >> diff --git a/meta/conf/isar-bitbake.conf b/meta/conf/isar-bitbake.conf >> index b853c88..36cca29 100644 >> --- a/meta/conf/isar-bitbake.conf >> +++ b/meta/conf/isar-bitbake.conf >> @@ -27,6 +27,9 @@ BUILDCHROOT_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot/rootfs" >> # Setup our default hash policy >> BB_SIGNATURE_HANDLER ?= "noop" >> >> +# Add event handlers for bitbake >> +INHERIT += "isar-events" >> + >> include conf/local.conf >> include conf/multiconfig/${BB_CURRENT_MC}.conf >> include conf/machine/${MACHINE}.conf >> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb >> index 51f9d5d..304c67e 100644 >> --- a/meta/recipes-devtools/buildchroot/buildchroot.bb >> +++ b/meta/recipes-devtools/buildchroot/buildchroot.bb >> @@ -32,7 +32,9 @@ BUILDCHROOT_PREINSTALL ?= "gcc \ >> WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}" >> >> do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" >> -do_build[dirs] = "${WORKDIR}/hooks_multistrap" >> +do_build[dirs] = "${WORKDIR}/hooks_multistrap \ >> + ${BUILDCHROOT_DIR}/isar-apt" >> +do_build[depends] = "isar-apt:do_cache_config" >> >> do_build() { >> E="${@ bb.utils.export_proxies(d)}" >> @@ -67,6 +69,8 @@ do_build() { >> } >> trap '_do_build_cleanup' EXIT >> >> + do_setup_mounts >> + >> # Create root filesystem >> sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f "${WORKDIR}/multistrap.conf" >> >> @@ -76,4 +80,26 @@ do_build() { >> # Configure root filesystem >> sudo chroot ${BUILDCHROOT_DIR} /configscript.sh >> _do_build_cleanup >> + >> + do_cleanup_mounts >> +} >> + >> +# Invalidate stamp for do_setup_mounts before each build start. >> +# This will guarantee that this function will be executed once >> +# per build. >> +python __anonymous() { >> + stamp = d.getVar("STAMP") + ".do_setup_mounts." + d.getVarFlag("do_setup_mounts", 'stamp-extra-info') >> + os.remove(stamp) if os.path.exists(stamp) else None >> +} >> + >> +do_setup_mounts[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}" >> + >> +do_setup_mounts() { >> + sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt >> +} >> + >> +addtask setup_mounts after do_build >> + >> +do_cleanup_mounts() { >> + sudo umount ${BUILDCHROOT_DIR}/isar-apt 2>/dev/null || true >> } >> diff --git a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in >> index a0b28e3..480a4b8 100644 >> --- a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in >> +++ b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in >> @@ -6,7 +6,7 @@ noauth=true >> unpack=true >> ignorenativearch=true >> bootstrap=##DISTRO_MULTICONF_BOOTSTRAP## >> -aptsources=##DISTRO_MULTICONF_APTSOURCES## >> +aptsources=isar-apt ##DISTRO_MULTICONF_APTSOURCES## >> configscript=##CONFIG_SCRIPT## >> setupscript=##SETUP_SCRIPT## >> hookdir=##DIR_HOOKS## >> @@ -29,3 +29,9 @@ source=##DISTRO_APT_SOURCE_SEC## >> suite=##DISTRO_SUITE##/updates >> components=##DISTRO_COMPONENTS## >> omitdebsrc=true >> + >> +[isar-apt] >> +source=file:///isar-apt >> +suite=isar >> +components=main >> +omitdebsrc=true >> > > Looks good! I'm testing it via your devel branch, and I also > successfully rebased "Permanently mount '/dev', '/sys' and '/proc'" on top. > Thanks! > Regarding the remaining cleanup of WORKDIR mounting: Can't each > dpkg-base class register their own cleanup handler in the same fashion? > I've remembered, that for Jenkins I've used something following to clean the build automatically: 8<-- #!/bin/bash W=${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH} for m in $(cat /proc/mounts | grep $W); do m=$(echo $m | cut -d ' ' -f 2) umount $m done 8<-- There are some facts: 1. Proc folder must be available for correct Isar operating. 2. /proc/mounts contains all the active mounts. 3. Folder $W contains only mounts created by current build. Based on this, I think if we insert the code above to the event handler, it will be robust enough solution to clean up everything. Also we should not anymore care if somebody adds some extra mount. What do you think? Alex