From mboxrd@z Thu Jan 1 00:00:00 1970 X-GM-THRID: 6520117824876183552 X-Received: by 10.28.156.12 with SMTP id f12mr80100wme.4.1518088194220; Thu, 08 Feb 2018 03:09:54 -0800 (PST) X-BeenThere: isar-users@googlegroups.com Received: by 10.28.129.141 with SMTP id c135ls912190wmd.11.canary-gmail; Thu, 08 Feb 2018 03:09:53 -0800 (PST) X-Google-Smtp-Source: AH8x226jJU1mXh2BAIxuvX4Z6TsOOvA5ZC6KYfc6cCS1kEvi8bYhXt/xGhlTJEMSsAUSfGEI/1VK X-Received: by 10.28.231.9 with SMTP id e9mr78704wmh.12.1518088193662; Thu, 08 Feb 2018 03:09:53 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1518088193; cv=none; d=google.com; s=arc-20160816; b=0ZmOI1qr6oLPMc26FVrkbY4hUT33AmDbvZ2KY81tGge3DhW3NEX796o50UE2s+mcUg +EXuEy7xhkz2soTv6dH1mFaRLDn4+H/n3R1ZpqepmS/Xr5ixd58ra0pcamwwklf6+S2y A0YQGEgPRDpYu0eP8B0FZyPfH1x1XirIf94KtPgQsjTtdTYjmNH/amAkLatv5hNftZXX nUEKieEwaNaqCA5t/0ejNdDErkNwAVfUlefuQz5op5ub/MzA/YcyYA+jdzRMVJI7Hzzi qkvoGcAZMggsX3EajDb6QGLc4kD83f3M8pnsJwiooPL12jccJ3PtFP330Z0CRyKvS+cF TlTA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-language:content-transfer-encoding:in-reply-to:mime-version :user-agent:date:message-id:from:references:to:subject :dkim-signature:arc-authentication-results; bh=/OaofF1rCRJ1ND4xfJDX+54nxziiZxhPi00/16AcyIY=; b=MjPgtaUWlK6FvF7mxnPx56QNtoREgf4xiJWZQME9i32GJ+ZepoBWzjipsji48VHK4W vmmazOejalPSDimnEDX5Q7B5RzUxUaG52zM7t1bSp5sgedOW2FLPbGDKVlObNPOKgb33 3UvwLYziOHwwi+wxSo6VUE0lYde2ia9/fs/38BgpWkP52ril9j+a3pHhsCNRjzYc3UH7 2JHIfVEC8tleGbt6fZMreJXWzziLQWJmoHLtWFZDKAb7r4EPrUWMGeW6RHZXPZJs4F0v U13swWbKcyNHieS5XOhxain85nPcPhOnJ+7Xj6HvUy44375mJqEjhFypRJnH4xyRE29/ WLYg== ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@googlemail.com header.s=20161025 header.b=A7OpjwJf; spf=pass (google.com: domain of benbrenson89@googlemail.com designates 2a00:1450:400c:c09::231 as permitted sender) smtp.mailfrom=benbrenson89@googlemail.com; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=googlemail.com Return-Path: Received: from mail-wm0-x231.google.com (mail-wm0-x231.google.com. [2a00:1450:400c:c09::231]) by gmr-mx.google.com with ESMTPS id a12si37387wmg.4.2018.02.08.03.09.53 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 08 Feb 2018 03:09:53 -0800 (PST) Received-SPF: pass (google.com: domain of benbrenson89@googlemail.com designates 2a00:1450:400c:c09::231 as permitted sender) client-ip=2a00:1450:400c:c09::231; Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@googlemail.com header.s=20161025 header.b=A7OpjwJf; spf=pass (google.com: domain of benbrenson89@googlemail.com designates 2a00:1450:400c:c09::231 as permitted sender) smtp.mailfrom=benbrenson89@googlemail.com; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=googlemail.com Received: by mail-wm0-x231.google.com with SMTP id r71so8797931wmd.1 for ; Thu, 08 Feb 2018 03:09:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20161025; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding:content-language; bh=/OaofF1rCRJ1ND4xfJDX+54nxziiZxhPi00/16AcyIY=; b=A7OpjwJfiggLAxU4upi//FkIRdw/EqZabVcn1M8CcpslBgXEegiZnqIjS5f+J0CQuw ZaY5o23ufrrQMDt22QpNNLsaGpCFLK5Nqbhph6b9WNrDsjEsNdTvdkAtN7JkFmdfppIs 44oZ0/TBaxBoos8ZJ4p9e69qniubG1QOfTJvlt4uwz3XGPf06n1Rkyp2rfyP1yBQr741 7x2NAY3kfGnkGawOo8wKXwLpwLHt58b98SMuuBaBTh0pRg7bdHLvJORX0ttlFKgtuECd v9laR/czkg4Dceo7pCHe3rLkpB2g3szue0J8JAyxNl13r0Yvq994xETvfRxdxCAjPa0U gxFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=/OaofF1rCRJ1ND4xfJDX+54nxziiZxhPi00/16AcyIY=; b=eweeRE5vUibSZxkRebJLT8VFO6Y8mkAoD5qziB2q4OYXyRcHygLSRK5J6vBWI1LOe9 R2oi8+nCAENAupBwXIDvgYgmplT3VpoetqFyMX0SPjzFqNUrs5ETGIl1M6vwDuPNiDaX cFxgweWIwvCZ8m+xfwssfWWQdDUnmJKZQC7pvlM523+hxR31e6b6mgZD9fGEFPbdNn20 dBlQXM9NSXAHNpvhi2ZGwCUvXuk5fv+YuXpl3aPkuJeRW4DK4gm8+4YiULeLqZldxIdD TRSsgI0Pv5s9OHi5/Gyr9gcGUsxOnLlHG/Q9UiEL1HXPk+R5SysKv7+4+dSoBPP7BbL6 AhUg== X-Gm-Message-State: APf1xPBwQrX20BswV1MAe+nK30P1UuCawecaOKC2x2vs/Eu4H7xvJ24T 7Ty2DQYzyqmfy/WUt/90Lz8HW+Wr X-Received: by 10.28.15.13 with SMTP id 13mr653787wmp.105.1518088193075; Thu, 08 Feb 2018 03:09:53 -0800 (PST) Return-Path: Received: from [192.168.43.209] (ip-109-41-193-215.web.vodafone.de. [109.41.193.215]) by smtp.gmail.com with ESMTPSA id w83sm2803102wma.47.2018.02.08.03.09.51 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 08 Feb 2018 03:09:52 -0800 (PST) Subject: Re: [RFC PATCH 1/1] RFC-PATCH: partial implementation for support of repos from other layers To: isar-users@googlegroups.com References: <20180208094653.1819-1-claudius.heine.ext@siemens.com> <20180208094653.1819-2-claudius.heine.ext@siemens.com> From: Benedikt Niedermayr Message-ID: Date: Thu, 8 Feb 2018 12:09:50 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: <20180208094653.1819-2-claudius.heine.ext@siemens.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-TUID: CY9qe2179Gwb Am 08.02.2018 um 10:46 schrieb claudius.heine.ext@siemens.com: > From: Claudius Heine > > This patch changes the implementation of isar-image-base to support > the definition of additional repositories from other layers, by addding > apt source list compatible files to the 'DEBIAN_APT_SOURCES' variable. > > This also implement the generation of the multistrap.conf file from > within the recipe. > > Signed-off-by: Claudius Heine > --- > meta-isar/conf/distro/debian-stretch.conf | 6 +- > meta-isar/conf/distro/stretch.list | 3 + > .../recipes-core/images/files/multistrap.conf.in | 38 ------ > meta-isar/recipes-core/images/isar-image-base.bb | 134 +++++++++++++++++---- > 4 files changed, 117 insertions(+), 64 deletions(-) > create mode 100644 meta-isar/conf/distro/stretch.list > delete mode 100644 meta-isar/recipes-core/images/files/multistrap.conf.in > > diff --git a/meta-isar/conf/distro/debian-stretch.conf b/meta-isar/conf/distro/debian-stretch.conf > index 74177fd..083eb86 100644 > --- a/meta-isar/conf/distro/debian-stretch.conf > +++ b/meta-isar/conf/distro/debian-stretch.conf > @@ -1,10 +1,14 @@ > # This software is a part of ISAR. > # Copyright (C) 2017 ilbers GmbH > > +DEBIAN_APT_SOURCES += "conf/distro/stretch.list" > +DEBIAN_APT_PREFERENCES ??= "" > +DISTRO_CONFIG_SCRIPT ??= "debian-configscript.sh" > + > +# No longer needed, after buildchroot is converted: > DISTRO_SUITE ?= "stretch" > DISTRO_COMPONENTS ?= "main contrib non-free" > DISTRO_APT_SOURCE ?= "http://ftp.debian.org/debian" > DISTRO_APT_SOURCE_SEC ?= "http://security.debian.org/debian-security" > -DISTRO_CONFIG_SCRIPT ?= "debian-configscript.sh" > DISTRO_MULTICONF_BOOTSTRAP ?= "base updates security" > DISTRO_MULTICONF_APTSOURCES ?= "${DISTRO_MULTICONF_BOOTSTRAP}" > diff --git a/meta-isar/conf/distro/stretch.list b/meta-isar/conf/distro/stretch.list > new file mode 100644 > index 0000000..a5ed5db > --- /dev/null > +++ b/meta-isar/conf/distro/stretch.list > @@ -0,0 +1,3 @@ > +deb http://ftp.de.debian.org/debian stretch main contrib non-free > +deb http://ftp.de.debian.org/debian stretch-updates main contrib non-free > +deb http://security.debian.org stretch/updates main contrib non-free > diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in b/meta-isar/recipes-core/images/files/multistrap.conf.in > deleted file mode 100644 > index 432b6af..0000000 > --- a/meta-isar/recipes-core/images/files/multistrap.conf.in > +++ /dev/null > @@ -1,38 +0,0 @@ > -# This software is a part of ISAR. > -# Copyright (C) 2015-2017 ilbers GmbH > - > -[General] > -noauth=true > -unpack=true > -ignorenativearch=true > -bootstrap=##DISTRO_MULTICONF_BOOTSTRAP## Isar > -aptsources=##DISTRO_MULTICONF_APTSOURCES## > -configscript=##CONFIG_SCRIPT## > -setupscript=##SETUP_SCRIPT## > -hookdir=##DIR_HOOKS## > - > -[base] > -source=##DISTRO_APT_SOURCE## > -suite=##DISTRO_SUITE## > -components=##DISTRO_COMPONENTS## > -packages=##IMAGE_PREINSTALL## > -omitdebsrc=true > - > -[updates] > -source=##DISTRO_APT_SOURCE## > -suite=##DISTRO_SUITE##-updates > -components=##DISTRO_COMPONENTS## > -omitdebsrc=true > - > -[security] > -source=##DISTRO_APT_SOURCE_SEC## > -suite=##DISTRO_SUITE##/updates > -components=##DISTRO_COMPONENTS## > -omitdebsrc=true > - > -[Isar] > -packages=##IMAGE_INSTALL## > -source=##DEPLOY_DIR_APT## > -suite=##ISAR_DISTRO_SUITE## > -components=main > -omitdebsrc=true > diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb > index e359ac3..655dbb2 100644 > --- a/meta-isar/recipes-core/images/isar-image-base.bb > +++ b/meta-isar/recipes-core/images/isar-image-base.bb > @@ -9,8 +9,7 @@ LICENSE = "gpl-2.0" > LIC_FILES_CHKSUM = "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe" > > FILESPATH =. "${LAYERDIR_isar}/recipes-core/images/files:" > -SRC_URI = "file://multistrap.conf.in \ > - file://${DISTRO_CONFIG_SCRIPT} \ > +SRC_URI = "file://${DISTRO_CONFIG_SCRIPT} \ > file://setup.sh \ > file://download_dev-random" > > @@ -26,34 +25,119 @@ IMAGE_PREINSTALL += "apt \ > > WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}" > > -do_rootfs[dirs] = "${WORKDIR}/hooks_multistrap" > - > -do_rootfs() { > - E="${@ bb.utils.export_proxies(d)}" > +do_multistrap_prepare[dirs] = "${WORKDIR}/hooks_multistrap" > > +do_multistrap_prepare() { > chmod +x "${WORKDIR}/${DISTRO_CONFIG_SCRIPT}" > chmod +x "${WORKDIR}/setup.sh" > install -m 755 "${WORKDIR}/download_dev-random" "${WORKDIR}/hooks_multistrap/" > +} > +addtask multistrap_prepare before do_multistrap_config after do_unpack > + > +python do_multistrap_config() { > + import shutil > + import configparser as cp > + import textwrap > + > + workdir = d.getVar("WORKDIR", True) > + workdir_rel = os.path.relpath(workdir) > + > + multistrap_conf_file = os.path.join(workdir, "multistrap.conf") > + > + apt_sources_list = d.getVar("DEBIAN_APT_SOURCES", True).split() > + > + apt_preferences_list = d.getVar("DEBIAN_APT_PREFERENCES", True).split() > + apt_preferences_out = os.path.join(workdir_rel, "apt-preferences") > + > + image_preinstall = d.getVar("IMAGE_PREINSTALL", True).split() > + image_install = d.getVar("IMAGE_INSTALL", True).split() > + > + repos = [] > + > + msconf = cp.RawConfigParser() > + > + # Set general multistrap configuration: > + msconf.add_section("General") > + msconf.set("General", "noauth", "true") > + msconf.set("General", "unpack", "true") > + msconf.set("General", "ignorenativearch", "true") > + msconf.set("General", "aptpreferences", "./" + apt_preferences_out) > + msconf.set("General", "configscript", "./" + > + os.path.join(workdir_rel, > + d.getVar("DISTRO_CONFIG_SCRIPT", True))) > + msconf.set("General", "setupscript", "./" + > + os.path.join(workdir_rel, > + "setup.sh")) > + msconf.set("General", "hookdir", "./" + > + os.path.join(workdir_rel, > + "hooks_multistrap")) > + > + # Set repository in multistrap configuration: > + repo_counter = 0 > + for aptsrc in apt_sources_list: > + aptsrc_real = bb.parse.resolve_file(aptsrc, d) > + with open(aptsrc_real, "r") as in_fd: > + for line in in_fd: > + entry = line.split() > + > + # Only care about 'deb' repos for now: > + if entry[0] != "deb": > + continue > + > + # Generate on unique name for the section > + repo_name = "repo{}".format(repo_counter) > + repos.append(repo_name) > + > + source = entry[1] > + suite = entry[2] > + components = entry[3:] > + > + msconf.add_section(repo_name) > + msconf.set(repo_name, "source", source) > + msconf.set(repo_name, "suite", suite) > + msconf.set(repo_name, "components", " ".join(components)) > + msconf.set(repo_name, "omitdebsrc", "true") > + > + # The first repo gets assigned all IMAGE_PREINSTALL > + # packages: > + if image_preinstall: > + msconf.set(repo_name, "packages", > + " ".join(image_preinstall)) > + image_preinstall = [] > + > + repo_counter += 1 > + > + msconf.add_section("isar-generated") > + msconf.set("isar-generated", "packages", " ".join(image_install)) > + msconf.set("isar-generated", "source", "copy:///{}/{}".format( > + d.getVar("DEPLOY_DIR_APT", True), d.getVar("DISTRO", True))) > + msconf.set("isar-generated", "suite", d.getVar("DEBDISTRONAME", True)) > + msconf.set("isar-generated", "components", "main") > + msconf.set("isar-generated", "omitdebsrc", "true") > + > + # Generate apt preference file: > + with open(apt_preferences_out, "wb") as out_fd: > + for aptpref in apt_preferences_list: > + artpref_real = bb.parse.resolve_file(aptpref, d) > + with open(aptpref_real, "rb") as in_fd: > + shutil.copyfileobj(in_fd, out_fd, 1024*1024*10) > + out_fd.write("\n".encode()) > + out_fd.write(textwrap.dedent("""\ > + Package: * > + Pin: release n={} > + Pin-Priority: 1000 > + """.format(d.getVar("DEBDISTRONAME", True))).encode()) > + > + msconf.set("General", "bootstrap", " ".join(repos)) > + msconf.set("General", "aptsources", " ".join(repos + ["isar-generated"])) > + > + with open(multistrap_conf_file, "w") as out_fd: > + msconf.write(out_fd) > +} > +addtask multistrap_config after do_multistrap_prepare before do_rootfs > > - # Multistrap accepts only relative path in configuration files, so get it: > - cd ${TOPDIR} > - WORKDIR_REL=${@ os.path.relpath(d.getVar("WORKDIR", True))} > - > - # Adjust multistrap config > - sed -e 's|##IMAGE_PREINSTALL##|${IMAGE_PREINSTALL}|g' \ > - -e 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \ > - -e 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \ > - -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \ > - -e 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \ > - -e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \ > - -e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \ > - -e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/${DISTRO_CONFIG_SCRIPT}|g' \ > - -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \ > - -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \ > - -e 's|##IMAGE_INSTALL##|${IMAGE_INSTALL}|g' \ > - -e 's|##DEPLOY_DIR_APT##|copy:///${DEPLOY_DIR_APT}/${DISTRO}|g' \ > - -e 's|##ISAR_DISTRO_SUITE##|${DEBDISTRONAME}|g' \ > - "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf" > +do_rootfs() { > + E="${@ bb.utils.export_proxies(d)}" > > [ ! -d ${IMAGE_ROOTFS}/proc ] && sudo install -d -o 0 -g 0 -m 555 ${IMAGE_ROOTFS}/proc > sudo mount -t proc none ${IMAGE_ROOTFS}/proc Hi, thanks.... I think it  goes into the right direction. My proposal: - Saving all Parts of  conf/distro/stretch.list in bitbake variables, so we can reuse them in other parts of Isar.   Maybe something like   DISTRO_CONFIG_LIST = "foo"   DISTRO_CONFIG_LIST[type] = "deb"   DISTRO_CONFIG_LIST[url] = "http://ftp.de.debian.org/debian "   DISTRO_CONFIG_LIST[distro] = "stretch"   DISTRO_CONFIG_LIST[components] = "main contrib non-free"   Those vars should be stored in global context which in turn would presuppose   running  do_multistrap_config() in a global context, in order to getting access to those vars from everywhere.   I don't know if this possible? Other proposals are welcome. Benni