* [PATCH 0/2] Consolidate multistrap configuration generation @ 2018-02-14 13:15 claudius.heine.ext 2018-02-14 13:15 ` [PATCH 1/2] meta/classes/base: extend sys.path with lib directory claudius.heine.ext 2018-02-14 13:15 ` [PATCH 2/2] centralize multistrap configuration generation claudius.heine.ext 0 siblings, 2 replies; 23+ messages in thread From: claudius.heine.ext @ 2018-02-14 13:15 UTC (permalink / raw) To: isar-users; +Cc: Claudius Heine From: Claudius Heine <ch@denx.de> Hi, this patchset tries to consolidate the generation of the multistrap configuration in the buildchroot and isar-image-base recipes to one. This patchsetset also represents one part of the implementation of the multi-repo support. Cheers, Claudius Claudius Heine (2): meta/classes/base: extend sys.path with lib directory centralize multistrap configuration generation .../recipes-core/images/files/multistrap.conf.in | 38 ---------- meta-isar/recipes-core/images/isar-image-base.bb | 33 ++++----- meta/classes/base.bbclass | 24 +++++++ meta/lib/isar/__init__.py | 8 +++ meta/lib/isar/multistrap.py | 80 ++++++++++++++++++++++ meta/recipes-devtools/buildchroot/buildchroot.bb | 28 +++----- .../buildchroot/files/multistrap.conf.in | 37 ---------- 7 files changed, 134 insertions(+), 114 deletions(-) delete mode 100644 meta-isar/recipes-core/images/files/multistrap.conf.in create mode 100644 meta/lib/isar/__init__.py create mode 100644 meta/lib/isar/multistrap.py delete mode 100644 meta/recipes-devtools/buildchroot/files/multistrap.conf.in -- 2.15.1 ^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH 1/2] meta/classes/base: extend sys.path with lib directory 2018-02-14 13:15 [PATCH 0/2] Consolidate multistrap configuration generation claudius.heine.ext @ 2018-02-14 13:15 ` claudius.heine.ext 2018-02-15 7:28 ` Jan Kiszka 2018-02-14 13:15 ` [PATCH 2/2] centralize multistrap configuration generation claudius.heine.ext 1 sibling, 1 reply; 23+ messages in thread From: claudius.heine.ext @ 2018-02-14 13:15 UTC (permalink / raw) To: isar-users; +Cc: Claudius Heine From: Claudius Heine <ch@denx.de> Currently scripts and modules that are placed in the 'lib' directory of layers are not available in the recipes. This change adds the required code to the base.bbclass in order to have those imports available. This change contains code from: http://git.openembedded.org/openembedded-core/tree/meta/classes/base.bbclass?id=4a4eff9b9e3c614ff41b17cbec359b72dcbd41bd Signed-off-by: Claudius Heine <ch@denx.de> --- meta/classes/base.bbclass | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass index ae84677..3334789 100644 --- a/meta/classes/base.bbclass +++ b/meta/classes/base.bbclass @@ -37,6 +37,30 @@ bbfatal() { exit 1 } +# Derived from openembedded-core/meta/classes/base.bbclass +ISAR_IMPORTS += "os sys time" + +def isar_imports(d): + import sys + + bbpath = d.getVar("BBPATH").split(":") + sys.path[0:0] = [os.path.join(dir, "lib") for dir in bbpath] + + def inject(name, value): + """Make a python object accessible from the metadata""" + if hasattr(bb.utils, "_context"): + bb.utils._context[name] = value + else: + __builtins__[name] = value + + for toimport in d.getVar("ISAR_IMPORTS", True).split(): + imported = __import__(toimport) + inject(toimport.split(".", 1)[0], imported) + + return "" + +ISAR_IMPORTED := "${@isar_imports(d)}" + # Derived from bitbake: bitbake/classes/base.bbclass addtask showdata do_showdata[nostamp] = "1" -- 2.15.1 ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 1/2] meta/classes/base: extend sys.path with lib directory 2018-02-14 13:15 ` [PATCH 1/2] meta/classes/base: extend sys.path with lib directory claudius.heine.ext @ 2018-02-15 7:28 ` Jan Kiszka 0 siblings, 0 replies; 23+ messages in thread From: Jan Kiszka @ 2018-02-15 7:28 UTC (permalink / raw) To: [ext] claudius.heine.ext@siemens.com, isar-users, Henning Schild Cc: Claudius Heine On 2018-02-14 14:15, [ext] claudius.heine.ext@siemens.com wrote: > From: Claudius Heine <ch@denx.de> > > Currently scripts and modules that are placed in the 'lib' directory of > layers are not available in the recipes. > > This change adds the required code to the base.bbclass in order to have > those imports available. > > This change contains code from: > http://git.openembedded.org/openembedded-core/tree/meta/classes/base.bbclass?id=4a4eff9b9e3c614ff41b17cbec359b72dcbd41bd > > Signed-off-by: Claudius Heine <ch@denx.de> > --- > meta/classes/base.bbclass | 24 ++++++++++++++++++++++++ > 1 file changed, 24 insertions(+) > > diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass > index ae84677..3334789 100644 > --- a/meta/classes/base.bbclass > +++ b/meta/classes/base.bbclass > @@ -37,6 +37,30 @@ bbfatal() { > exit 1 > } > > +# Derived from openembedded-core/meta/classes/base.bbclass > +ISAR_IMPORTS += "os sys time" > + > +def isar_imports(d): > + import sys > + > + bbpath = d.getVar("BBPATH").split(":") > + sys.path[0:0] = [os.path.join(dir, "lib") for dir in bbpath] > + > + def inject(name, value): > + """Make a python object accessible from the metadata""" > + if hasattr(bb.utils, "_context"): > + bb.utils._context[name] = value > + else: > + __builtins__[name] = value > + > + for toimport in d.getVar("ISAR_IMPORTS", True).split(): > + imported = __import__(toimport) > + inject(toimport.split(".", 1)[0], imported) > + > + return "" > + > +ISAR_IMPORTED := "${@isar_imports(d)}" > + > # Derived from bitbake: bitbake/classes/base.bbclass > addtask showdata > do_showdata[nostamp] = "1" > Henning, does this help with selecting wic plugins as well? Or does wic express its search paths differently and would not be affected by this change? Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH 2/2] centralize multistrap configuration generation 2018-02-14 13:15 [PATCH 0/2] Consolidate multistrap configuration generation claudius.heine.ext 2018-02-14 13:15 ` [PATCH 1/2] meta/classes/base: extend sys.path with lib directory claudius.heine.ext @ 2018-02-14 13:15 ` claudius.heine.ext 2018-02-15 7:28 ` Jan Kiszka 2018-02-15 11:42 ` Baurzhan Ismagulov 1 sibling, 2 replies; 23+ messages in thread From: claudius.heine.ext @ 2018-02-14 13:15 UTC (permalink / raw) To: isar-users; +Cc: Claudius Heine From: Claudius Heine <ch@denx.de> Previous there was redundant code between the buildchroot and isar-image-base recipe. This patch moves the generation of the multistrap configuration to a central python script that can be parametrized by both recipes. Signed-off-by: Claudius Heine <ch@denx.de> --- .../recipes-core/images/files/multistrap.conf.in | 38 ---------- meta-isar/recipes-core/images/isar-image-base.bb | 33 ++++----- meta/classes/base.bbclass | 2 +- meta/lib/isar/__init__.py | 8 +++ meta/lib/isar/multistrap.py | 80 ++++++++++++++++++++++ meta/recipes-devtools/buildchroot/buildchroot.bb | 28 +++----- .../buildchroot/files/multistrap.conf.in | 37 ---------- 7 files changed, 111 insertions(+), 115 deletions(-) delete mode 100644 meta-isar/recipes-core/images/files/multistrap.conf.in create mode 100644 meta/lib/isar/__init__.py create mode 100644 meta/lib/isar/multistrap.py delete mode 100644 meta/recipes-devtools/buildchroot/files/multistrap.conf.in diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in b/meta-isar/recipes-core/images/files/multistrap.conf.in deleted file mode 100644 index 432b6af..0000000 --- a/meta-isar/recipes-core/images/files/multistrap.conf.in +++ /dev/null @@ -1,38 +0,0 @@ -# This software is a part of ISAR. -# Copyright (C) 2015-2017 ilbers GmbH - -[General] -noauth=true -unpack=true -ignorenativearch=true -bootstrap=##DISTRO_MULTICONF_BOOTSTRAP## Isar -aptsources=##DISTRO_MULTICONF_APTSOURCES## -configscript=##CONFIG_SCRIPT## -setupscript=##SETUP_SCRIPT## -hookdir=##DIR_HOOKS## - -[base] -source=##DISTRO_APT_SOURCE## -suite=##DISTRO_SUITE## -components=##DISTRO_COMPONENTS## -packages=##IMAGE_PREINSTALL## -omitdebsrc=true - -[updates] -source=##DISTRO_APT_SOURCE## -suite=##DISTRO_SUITE##-updates -components=##DISTRO_COMPONENTS## -omitdebsrc=true - -[security] -source=##DISTRO_APT_SOURCE_SEC## -suite=##DISTRO_SUITE##/updates -components=##DISTRO_COMPONENTS## -omitdebsrc=true - -[Isar] -packages=##IMAGE_INSTALL## -source=##DEPLOY_DIR_APT## -suite=##ISAR_DISTRO_SUITE## -components=main -omitdebsrc=true diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb index 8ddbabb..9913977 100644 --- a/meta-isar/recipes-core/images/isar-image-base.bb +++ b/meta-isar/recipes-core/images/isar-image-base.bb @@ -9,8 +9,7 @@ LICENSE = "gpl-2.0" LIC_FILES_CHKSUM = "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe" FILESPATH =. "${LAYERDIR_isar}/recipes-core/images/files:" -SRC_URI = "file://multistrap.conf.in \ - file://${DISTRO_CONFIG_SCRIPT} \ +SRC_URI = "file://${DISTRO_CONFIG_SCRIPT} \ file://setup.sh \ file://download_dev-random" @@ -28,6 +27,14 @@ WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}" do_rootfs[dirs] = "${WORKDIR}/hooks_multistrap" +MULTISTRAP_CONFIG_SCRIPT = "${DISTRO_CONFIG_SCRIPT}" +MULTISTRAP_APTSOURCES = "${DISTRO_MULTICONF_APTSOURCES}" +MULTISTRAP_BOOTSTRAP = "${DISTRO_MULTICONF_BOOTSTRAP} isar" +MULTISTRAP_PREINSTALL = "${IMAGE_PREINSTALL}" +MULTISTRAP_ISAR_SOURCE = "copy:///${DEPLOY_DIR_APT}/${DISTRO}" +MULTISTRAP_ISAR_INSTALL = "${IMAGE_INSTALL}" +MULTISTRAP_ISAR_SUITE = "${DEBDISTRONAME}" + do_rootfs() { E="${@ bb.utils.export_proxies(d)}" @@ -35,25 +42,8 @@ do_rootfs() { chmod +x "${WORKDIR}/setup.sh" install -m 755 "${WORKDIR}/download_dev-random" "${WORKDIR}/hooks_multistrap/" - # Multistrap accepts only relative path in configuration files, so get it: - cd ${TOPDIR} - WORKDIR_REL=${@ os.path.relpath(d.getVar("WORKDIR", True))} - - # Adjust multistrap config - sed -e 's|##IMAGE_PREINSTALL##|${IMAGE_PREINSTALL}|g' \ - -e 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \ - -e 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \ - -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \ - -e 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \ - -e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \ - -e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \ - -e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/${DISTRO_CONFIG_SCRIPT}|g' \ - -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \ - -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \ - -e 's|##IMAGE_INSTALL##|${IMAGE_INSTALL}|g' \ - -e 's|##DEPLOY_DIR_APT##|copy:///${DEPLOY_DIR_APT}/${DISTRO}|g' \ - -e 's|##ISAR_DISTRO_SUITE##|${DEBDISTRONAME}|g' \ - "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf" + echo "${@ isar.multistrap.generate_multistrap_config(d) }" > \ + "${WORKDIR}/multistrap.conf" # Do not use bitbake flag [dirs] here because this folder should have # specific ownership. @@ -62,6 +52,7 @@ do_rootfs() { # Create root filesystem. We must use sudo -E here to preserve the environment # because of proxy settings + cd "${TOPDIR}" sudo -E multistrap -a ${DISTRO_ARCH} -d "${IMAGE_ROOTFS}" -f "${WORKDIR}/multistrap.conf" # Configure root filesystem diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass index 3334789..2a45a1e 100644 --- a/meta/classes/base.bbclass +++ b/meta/classes/base.bbclass @@ -38,7 +38,7 @@ bbfatal() { } # Derived from openembedded-core/meta/classes/base.bbclass -ISAR_IMPORTS += "os sys time" +ISAR_IMPORTS += "os sys time isar" def isar_imports(d): import sys diff --git a/meta/lib/isar/__init__.py b/meta/lib/isar/__init__.py new file mode 100644 index 0000000..e6aeeaa --- /dev/null +++ b/meta/lib/isar/__init__.py @@ -0,0 +1,8 @@ +# Custom kernel recipe include +# +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2018 +# +# SPDX-License-Identifier: MIT + +from . import multistrap diff --git a/meta/lib/isar/multistrap.py b/meta/lib/isar/multistrap.py new file mode 100644 index 0000000..7022f91 --- /dev/null +++ b/meta/lib/isar/multistrap.py @@ -0,0 +1,80 @@ +# Custom kernel recipe include +# +# This software is a part of ISAR. +# Copyright (c) Siemens AG, 2018 +# +# SPDX-License-Identifier: MIT + +def generate_multistrap_config(d): + import configparser as cp + import io + + workdir = d.getVar("WORKDIR", True) or "" + topdir = d.getVar("TOPDIR", True) or "" + workdir_rel = os.path.relpath(workdir, start=topdir) + + msconf = cp.RawConfigParser() + + # Set general multistrap configuration: + configscript = d.getVar("MULTISTRAP_CONFIG_SCRIPT", True) or "" + setupscript = d.getVar("MULTISTRAP_SETUP_SCRIPT", True) or "setup.sh" + hookdir = d.getVar("MULTISTRAP_HOOK_DIR", True) or "hooks_multistrap" + aptsources = (d.getVar("MULTISTRAP_APTSOURCES", True) or "").split() + bootstrap = (d.getVar("MULTISTRAP_BOOTSTRAP", True) or "").split() + + msconf.add_section("General") + msconf.set("General", "noauth", "true") + msconf.set("General", "unpack", "true") + msconf.set("General", "ignorenativearch", "true") + msconf.set("General", "configscript", "./" + + os.path.join(workdir_rel, configscript)) + msconf.set("General", "setupscript", "./" + + os.path.join(workdir_rel, setupscript)) + msconf.set("General", "hookdir", "./" + + os.path.join(workdir_rel, hookdir)) + msconf.set("General", "aptsources", " ".join(aptsources)) + msconf.set("General", "bootstrap", " ".join(bootstrap)) + + # Setup upstream repositories in multistrap configuration: + apt_source = d.getVar("DISTRO_APT_SOURCE", True) or "" + apt_source_sec = d.getVar("DISTRO_APT_SOURCE_SEC", True) or "" + apt_suite = d.getVar("DISTRO_SUITE", True) or "" + apt_components = (d.getVar("DISTRO_COMPONENTS", True) or "").split() + deb_packages = (d.getVar("MULTISTRAP_PREINSTALL", True) or "").split() + isar_source = d.getVar("MULTISTRAP_ISAR_SOURCE", True) or "" + isar_packages = (d.getVar("MULTISTRAP_ISAR_INSTALL", True) or "").split() + isar_suite = d.getVar("MULTISTRAP_ISAR_SUITE", True) or "isar" + + msconf.add_section("base") + msconf.set("base", "source", apt_source) + msconf.set("base", "suite", apt_suite) + msconf.set("base", "components", " ".join(apt_components)) + msconf.set("base", "omitdebsrc", "true") + msconf.set("base", "packages", " ".join(deb_packages)) + + msconf.add_section("updates") + msconf.set("updates", "source", apt_source) + msconf.set("updates", "suite", apt_suite + "-updates") + msconf.set("updates", "components", " ".join(apt_components)) + msconf.set("updates", "omitdebsrc", "true") + + msconf.add_section("security") + msconf.set("security", "source", apt_source_sec) + msconf.set("security", "suite", apt_suite + "/updates") + msconf.set("security", "components", " ".join(apt_components)) + msconf.set("security", "omitdebsrc", "true") + + msconf.add_section("isar") + msconf.set("isar", "source", isar_source) + msconf.set("isar", "suite", isar_suite) + msconf.set("isar", "components", "main") + msconf.set("isar", "omitdebsrc", "true") + if isar_packages : + msconf.set("isar", "packages", + " ".join(isar_packages)) + + ret = "" + with io.StringIO() as out: + msconf.write(out) + ret = out.getvalue() + return ret diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb index 570c0ad..59b4e7e 100644 --- a/meta/recipes-devtools/buildchroot/buildchroot.bb +++ b/meta/recipes-devtools/buildchroot/buildchroot.bb @@ -9,8 +9,7 @@ LICENSE = "gpl-2.0" LIC_FILES_CHKSUM = "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe" FILESPATH =. "${LAYERDIR_core}/recipes-devtools/buildchroot/files:" -SRC_URI = "file://multistrap.conf.in \ - file://configscript.sh \ +SRC_URI = "file://configscript.sh \ file://setup.sh \ file://build.sh" PV = "1.0" @@ -37,32 +36,25 @@ do_build[dirs] = "${BUILDCHROOT_DIR}/isar-apt \ ${BUILDCHROOT_DIR}/sys" do_build[depends] = "isar-apt:do_cache_config" +MULTISTRAP_CONFIG_SCRIPT = "configscript.sh" +MULTISTRAP_APTSOURCES = "isar ${DISTRO_MULTICONF_APTSOURCES}" +MULTISTRAP_BOOTSTRAP = "${DISTRO_MULTICONF_BOOTSTRAP}" +MULTISTRAP_PREINSTALL = "${BUILDCHROOT_PREINSTALL}" +MULTISTRAP_ISAR_SOURCE = "file:///isar-apt" + do_build() { E="${@ bb.utils.export_proxies(d)}" chmod +x "${WORKDIR}/setup.sh" chmod +x "${WORKDIR}/configscript.sh" - # Multistrap accepts only relative path in configuration files, so get it: - cd ${TOPDIR} - WORKDIR_REL=${@ os.path.relpath(d.getVar("WORKDIR", True))} - - # Adjust multistrap config - sed -e 's|##BUILDCHROOT_PREINSTALL##|${BUILDCHROOT_PREINSTALL}|g' \ - -e 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \ - -e 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \ - -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \ - -e 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \ - -e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \ - -e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \ - -e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/configscript.sh|g' \ - -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \ - -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \ - "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf" + echo "${@ isar.multistrap.generate_multistrap_config(d) }" > \ + "${WORKDIR}/multistrap.conf" do_setup_mounts # Create root filesystem + cd "${TOPDIR}" sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f "${WORKDIR}/multistrap.conf" # Install package builder script diff --git a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in deleted file mode 100644 index 480a4b8..0000000 --- a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in +++ /dev/null @@ -1,37 +0,0 @@ -# This software is a part of ISAR. -# Copyright (C) 2015-2017 ilbers GmbH - -[General] -noauth=true -unpack=true -ignorenativearch=true -bootstrap=##DISTRO_MULTICONF_BOOTSTRAP## -aptsources=isar-apt ##DISTRO_MULTICONF_APTSOURCES## -configscript=##CONFIG_SCRIPT## -setupscript=##SETUP_SCRIPT## -hookdir=##DIR_HOOKS## - -[base] -source=##DISTRO_APT_SOURCE## -suite=##DISTRO_SUITE## -components=##DISTRO_COMPONENTS## -packages=##BUILDCHROOT_PREINSTALL## -omitdebsrc=true - -[updates] -source=##DISTRO_APT_SOURCE## -suite=##DISTRO_SUITE##-updates -components=##DISTRO_COMPONENTS## -omitdebsrc=true - -[security] -source=##DISTRO_APT_SOURCE_SEC## -suite=##DISTRO_SUITE##/updates -components=##DISTRO_COMPONENTS## -omitdebsrc=true - -[isar-apt] -source=file:///isar-apt -suite=isar -components=main -omitdebsrc=true -- 2.15.1 ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 2/2] centralize multistrap configuration generation 2018-02-14 13:15 ` [PATCH 2/2] centralize multistrap configuration generation claudius.heine.ext @ 2018-02-15 7:28 ` Jan Kiszka 2018-02-15 8:16 ` Claudius Heine 2018-02-15 9:34 ` Alexander Smirnov 2018-02-15 11:42 ` Baurzhan Ismagulov 1 sibling, 2 replies; 23+ messages in thread From: Jan Kiszka @ 2018-02-15 7:28 UTC (permalink / raw) To: [ext] claudius.heine.ext@siemens.com, isar-users, Alexander Smirnov, Baurzhan Ismagulov Cc: Claudius Heine On 2018-02-14 14:15, [ext] claudius.heine.ext@siemens.com wrote: > From: Claudius Heine <ch@denx.de> > > Previous there was redundant code between the buildchroot and > isar-image-base recipe. > > This patch moves the generation of the multistrap configuration to a > central python script that can be parametrized by both recipes. > > Signed-off-by: Claudius Heine <ch@denx.de> > --- > .../recipes-core/images/files/multistrap.conf.in | 38 ---------- > meta-isar/recipes-core/images/isar-image-base.bb | 33 ++++----- > meta/classes/base.bbclass | 2 +- > meta/lib/isar/__init__.py | 8 +++ > meta/lib/isar/multistrap.py | 80 ++++++++++++++++++++++ > meta/recipes-devtools/buildchroot/buildchroot.bb | 28 +++----- > .../buildchroot/files/multistrap.conf.in | 37 ---------- > 7 files changed, 111 insertions(+), 115 deletions(-) > delete mode 100644 meta-isar/recipes-core/images/files/multistrap.conf.in > create mode 100644 meta/lib/isar/__init__.py > create mode 100644 meta/lib/isar/multistrap.py > delete mode 100644 meta/recipes-devtools/buildchroot/files/multistrap.conf.in > > diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in b/meta-isar/recipes-core/images/files/multistrap.conf.in > deleted file mode 100644 > index 432b6af..0000000 > --- a/meta-isar/recipes-core/images/files/multistrap.conf.in > +++ /dev/null > @@ -1,38 +0,0 @@ > -# This software is a part of ISAR. > -# Copyright (C) 2015-2017 ilbers GmbH > - > -[General] > -noauth=true > -unpack=true > -ignorenativearch=true > -bootstrap=##DISTRO_MULTICONF_BOOTSTRAP## Isar > -aptsources=##DISTRO_MULTICONF_APTSOURCES## > -configscript=##CONFIG_SCRIPT## > -setupscript=##SETUP_SCRIPT## > -hookdir=##DIR_HOOKS## > - > -[base] > -source=##DISTRO_APT_SOURCE## > -suite=##DISTRO_SUITE## > -components=##DISTRO_COMPONENTS## > -packages=##IMAGE_PREINSTALL## > -omitdebsrc=true > - > -[updates] > -source=##DISTRO_APT_SOURCE## > -suite=##DISTRO_SUITE##-updates > -components=##DISTRO_COMPONENTS## > -omitdebsrc=true > - > -[security] > -source=##DISTRO_APT_SOURCE_SEC## > -suite=##DISTRO_SUITE##/updates > -components=##DISTRO_COMPONENTS## > -omitdebsrc=true > - > -[Isar] > -packages=##IMAGE_INSTALL## > -source=##DEPLOY_DIR_APT## > -suite=##ISAR_DISTRO_SUITE## > -components=main > -omitdebsrc=true > diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb > index 8ddbabb..9913977 100644 > --- a/meta-isar/recipes-core/images/isar-image-base.bb > +++ b/meta-isar/recipes-core/images/isar-image-base.bb > @@ -9,8 +9,7 @@ LICENSE = "gpl-2.0" > LIC_FILES_CHKSUM = "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe" > > FILESPATH =. "${LAYERDIR_isar}/recipes-core/images/files:" > -SRC_URI = "file://multistrap.conf.in \ > - file://${DISTRO_CONFIG_SCRIPT} \ > +SRC_URI = "file://${DISTRO_CONFIG_SCRIPT} \ > file://setup.sh \ > file://download_dev-random" > > @@ -28,6 +27,14 @@ WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}" > > do_rootfs[dirs] = "${WORKDIR}/hooks_multistrap" > > +MULTISTRAP_CONFIG_SCRIPT = "${DISTRO_CONFIG_SCRIPT}" > +MULTISTRAP_APTSOURCES = "${DISTRO_MULTICONF_APTSOURCES}" > +MULTISTRAP_BOOTSTRAP = "${DISTRO_MULTICONF_BOOTSTRAP} isar" > +MULTISTRAP_PREINSTALL = "${IMAGE_PREINSTALL}" > +MULTISTRAP_ISAR_SOURCE = "copy:///${DEPLOY_DIR_APT}/${DISTRO}" > +MULTISTRAP_ISAR_INSTALL = "${IMAGE_INSTALL}" > +MULTISTRAP_ISAR_SUITE = "${DEBDISTRONAME}" > + > do_rootfs() { > E="${@ bb.utils.export_proxies(d)}" > > @@ -35,25 +42,8 @@ do_rootfs() { > chmod +x "${WORKDIR}/setup.sh" > install -m 755 "${WORKDIR}/download_dev-random" "${WORKDIR}/hooks_multistrap/" > > - # Multistrap accepts only relative path in configuration files, so get it: > - cd ${TOPDIR} > - WORKDIR_REL=${@ os.path.relpath(d.getVar("WORKDIR", True))} > - > - # Adjust multistrap config > - sed -e 's|##IMAGE_PREINSTALL##|${IMAGE_PREINSTALL}|g' \ > - -e 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \ > - -e 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \ > - -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \ > - -e 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \ > - -e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \ > - -e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \ > - -e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/${DISTRO_CONFIG_SCRIPT}|g' \ > - -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \ > - -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \ > - -e 's|##IMAGE_INSTALL##|${IMAGE_INSTALL}|g' \ > - -e 's|##DEPLOY_DIR_APT##|copy:///${DEPLOY_DIR_APT}/${DISTRO}|g' \ > - -e 's|##ISAR_DISTRO_SUITE##|${DEBDISTRONAME}|g' \ > - "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf" > + echo "${@ isar.multistrap.generate_multistrap_config(d) }" > \ > + "${WORKDIR}/multistrap.conf" > > # Do not use bitbake flag [dirs] here because this folder should have > # specific ownership. > @@ -62,6 +52,7 @@ do_rootfs() { > > # Create root filesystem. We must use sudo -E here to preserve the environment > # because of proxy settings > + cd "${TOPDIR}" > sudo -E multistrap -a ${DISTRO_ARCH} -d "${IMAGE_ROOTFS}" -f "${WORKDIR}/multistrap.conf" > > # Configure root filesystem > diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass > index 3334789..2a45a1e 100644 > --- a/meta/classes/base.bbclass > +++ b/meta/classes/base.bbclass > @@ -38,7 +38,7 @@ bbfatal() { > } > > # Derived from openembedded-core/meta/classes/base.bbclass > -ISAR_IMPORTS += "os sys time" > +ISAR_IMPORTS += "os sys time isar" > > def isar_imports(d): > import sys > diff --git a/meta/lib/isar/__init__.py b/meta/lib/isar/__init__.py > new file mode 100644 > index 0000000..e6aeeaa > --- /dev/null > +++ b/meta/lib/isar/__init__.py > @@ -0,0 +1,8 @@ > +# Custom kernel recipe include > +# > +# This software is a part of ISAR. > +# Copyright (c) Siemens AG, 2018 > +# > +# SPDX-License-Identifier: MIT > + > +from . import multistrap > diff --git a/meta/lib/isar/multistrap.py b/meta/lib/isar/multistrap.py > new file mode 100644 > index 0000000..7022f91 > --- /dev/null > +++ b/meta/lib/isar/multistrap.py > @@ -0,0 +1,80 @@ > +# Custom kernel recipe include > +# > +# This software is a part of ISAR. > +# Copyright (c) Siemens AG, 2018 > +# > +# SPDX-License-Identifier: MIT > + > +def generate_multistrap_config(d): > + import configparser as cp > + import io > + > + workdir = d.getVar("WORKDIR", True) or "" > + topdir = d.getVar("TOPDIR", True) or "" > + workdir_rel = os.path.relpath(workdir, start=topdir) > + > + msconf = cp.RawConfigParser() > + > + # Set general multistrap configuration: > + configscript = d.getVar("MULTISTRAP_CONFIG_SCRIPT", True) or "" > + setupscript = d.getVar("MULTISTRAP_SETUP_SCRIPT", True) or "setup.sh" > + hookdir = d.getVar("MULTISTRAP_HOOK_DIR", True) or "hooks_multistrap" > + aptsources = (d.getVar("MULTISTRAP_APTSOURCES", True) or "").split() > + bootstrap = (d.getVar("MULTISTRAP_BOOTSTRAP", True) or "").split() > + > + msconf.add_section("General") > + msconf.set("General", "noauth", "true") > + msconf.set("General", "unpack", "true") > + msconf.set("General", "ignorenativearch", "true") > + msconf.set("General", "configscript", "./" + > + os.path.join(workdir_rel, configscript)) > + msconf.set("General", "setupscript", "./" + > + os.path.join(workdir_rel, setupscript)) > + msconf.set("General", "hookdir", "./" + > + os.path.join(workdir_rel, hookdir)) > + msconf.set("General", "aptsources", " ".join(aptsources)) > + msconf.set("General", "bootstrap", " ".join(bootstrap)) > + > + # Setup upstream repositories in multistrap configuration: > + apt_source = d.getVar("DISTRO_APT_SOURCE", True) or "" > + apt_source_sec = d.getVar("DISTRO_APT_SOURCE_SEC", True) or "" > + apt_suite = d.getVar("DISTRO_SUITE", True) or "" > + apt_components = (d.getVar("DISTRO_COMPONENTS", True) or "").split() > + deb_packages = (d.getVar("MULTISTRAP_PREINSTALL", True) or "").split() > + isar_source = d.getVar("MULTISTRAP_ISAR_SOURCE", True) or "" > + isar_packages = (d.getVar("MULTISTRAP_ISAR_INSTALL", True) or "").split() > + isar_suite = d.getVar("MULTISTRAP_ISAR_SUITE", True) or "isar" > + > + msconf.add_section("base") > + msconf.set("base", "source", apt_source) > + msconf.set("base", "suite", apt_suite) > + msconf.set("base", "components", " ".join(apt_components)) > + msconf.set("base", "omitdebsrc", "true") > + msconf.set("base", "packages", " ".join(deb_packages)) > + > + msconf.add_section("updates") > + msconf.set("updates", "source", apt_source) > + msconf.set("updates", "suite", apt_suite + "-updates") > + msconf.set("updates", "components", " ".join(apt_components)) > + msconf.set("updates", "omitdebsrc", "true") > + > + msconf.add_section("security") > + msconf.set("security", "source", apt_source_sec) > + msconf.set("security", "suite", apt_suite + "/updates") > + msconf.set("security", "components", " ".join(apt_components)) > + msconf.set("security", "omitdebsrc", "true") > + > + msconf.add_section("isar") > + msconf.set("isar", "source", isar_source) > + msconf.set("isar", "suite", isar_suite) > + msconf.set("isar", "components", "main") > + msconf.set("isar", "omitdebsrc", "true") > + if isar_packages : > + msconf.set("isar", "packages", > + " ".join(isar_packages)) > + > + ret = "" > + with io.StringIO() as out: > + msconf.write(out) > + ret = out.getvalue() > + return ret > diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb > index 570c0ad..59b4e7e 100644 > --- a/meta/recipes-devtools/buildchroot/buildchroot.bb > +++ b/meta/recipes-devtools/buildchroot/buildchroot.bb > @@ -9,8 +9,7 @@ LICENSE = "gpl-2.0" > LIC_FILES_CHKSUM = "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe" > > FILESPATH =. "${LAYERDIR_core}/recipes-devtools/buildchroot/files:" > -SRC_URI = "file://multistrap.conf.in \ > - file://configscript.sh \ > +SRC_URI = "file://configscript.sh \ > file://setup.sh \ > file://build.sh" > PV = "1.0" > @@ -37,32 +36,25 @@ do_build[dirs] = "${BUILDCHROOT_DIR}/isar-apt \ > ${BUILDCHROOT_DIR}/sys" > do_build[depends] = "isar-apt:do_cache_config" > > +MULTISTRAP_CONFIG_SCRIPT = "configscript.sh" > +MULTISTRAP_APTSOURCES = "isar ${DISTRO_MULTICONF_APTSOURCES}" > +MULTISTRAP_BOOTSTRAP = "${DISTRO_MULTICONF_BOOTSTRAP}" > +MULTISTRAP_PREINSTALL = "${BUILDCHROOT_PREINSTALL}" > +MULTISTRAP_ISAR_SOURCE = "file:///isar-apt" > + > do_build() { > E="${@ bb.utils.export_proxies(d)}" > > chmod +x "${WORKDIR}/setup.sh" > chmod +x "${WORKDIR}/configscript.sh" > > - # Multistrap accepts only relative path in configuration files, so get it: > - cd ${TOPDIR} > - WORKDIR_REL=${@ os.path.relpath(d.getVar("WORKDIR", True))} > - > - # Adjust multistrap config > - sed -e 's|##BUILDCHROOT_PREINSTALL##|${BUILDCHROOT_PREINSTALL}|g' \ > - -e 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \ > - -e 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \ > - -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \ > - -e 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \ > - -e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \ > - -e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \ > - -e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/configscript.sh|g' \ > - -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \ > - -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \ > - "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf" > + echo "${@ isar.multistrap.generate_multistrap_config(d) }" > \ > + "${WORKDIR}/multistrap.conf" > > do_setup_mounts > > # Create root filesystem > + cd "${TOPDIR}" > sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f "${WORKDIR}/multistrap.conf" > > # Install package builder script > diff --git a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in > deleted file mode 100644 > index 480a4b8..0000000 > --- a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in > +++ /dev/null > @@ -1,37 +0,0 @@ > -# This software is a part of ISAR. > -# Copyright (C) 2015-2017 ilbers GmbH > - > -[General] > -noauth=true > -unpack=true > -ignorenativearch=true > -bootstrap=##DISTRO_MULTICONF_BOOTSTRAP## > -aptsources=isar-apt ##DISTRO_MULTICONF_APTSOURCES## > -configscript=##CONFIG_SCRIPT## > -setupscript=##SETUP_SCRIPT## > -hookdir=##DIR_HOOKS## > - > -[base] > -source=##DISTRO_APT_SOURCE## > -suite=##DISTRO_SUITE## > -components=##DISTRO_COMPONENTS## > -packages=##BUILDCHROOT_PREINSTALL## > -omitdebsrc=true > - > -[updates] > -source=##DISTRO_APT_SOURCE## > -suite=##DISTRO_SUITE##-updates > -components=##DISTRO_COMPONENTS## > -omitdebsrc=true > - > -[security] > -source=##DISTRO_APT_SOURCE_SEC## > -suite=##DISTRO_SUITE##/updates > -components=##DISTRO_COMPONENTS## > -omitdebsrc=true > - > -[isar-apt] > -source=file:///isar-apt > -suite=isar > -components=main > -omitdebsrc=true > This requires rebasing. I quickly tried, but the result broke the package build (didn't analyze any details). While this moves us in the right direction, it still leaves some duplications behind (setup.sh) - and the big question if we should invest any further in this horrible multistrap mess. I've just enabled arm64, and it took a while to figure out that I was missing some lines in setup.sh. In contrast, qemu-debootstrap just worked (I used it to debug the multistrap output). So, before we move on here: What all does multistrap currently give us, compared to standard debootstrap? I only see "multiple repositories" so far. But as Claudius is going to massage that part heavily anyway, I really wonder if it's worth building on the multistrap indirection any further. How about something like this? - [qemu-]debootstrap --variant=minbase ... - inject all additional apt sources, including user-provided ones - set pinnings and priority, including user-provided ones - apt-get dist-upgrade That would kick out all the multistrap complications. Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 2/2] centralize multistrap configuration generation 2018-02-15 7:28 ` Jan Kiszka @ 2018-02-15 8:16 ` Claudius Heine 2018-02-15 8:50 ` Jan Kiszka 2018-02-15 9:34 ` Alexander Smirnov 1 sibling, 1 reply; 23+ messages in thread From: Claudius Heine @ 2018-02-15 8:16 UTC (permalink / raw) To: Jan Kiszka, isar-users, Alexander Smirnov, Baurzhan Ismagulov Cc: Claudius Heine Hi Jan, On 02/15/2018 08:28 AM, Jan Kiszka wrote: > On 2018-02-14 14:15, [ext] claudius.heine.ext@siemens.com wrote: >> From: Claudius Heine <ch@denx.de> >> >> Previous there was redundant code between the buildchroot and >> isar-image-base recipe. >> >> This patch moves the generation of the multistrap configuration to a >> central python script that can be parametrized by both recipes. >> >> Signed-off-by: Claudius Heine <ch@denx.de> >> --- >> .../recipes-core/images/files/multistrap.conf.in | 38 ---------- >> meta-isar/recipes-core/images/isar-image-base.bb | 33 ++++----- >> meta/classes/base.bbclass | 2 +- >> meta/lib/isar/__init__.py | 8 +++ >> meta/lib/isar/multistrap.py | 80 ++++++++++++++++++++++ >> meta/recipes-devtools/buildchroot/buildchroot.bb | 28 +++----- >> .../buildchroot/files/multistrap.conf.in | 37 ---------- >> 7 files changed, 111 insertions(+), 115 deletions(-) >> delete mode 100644 meta-isar/recipes-core/images/files/multistrap.conf.in >> create mode 100644 meta/lib/isar/__init__.py >> create mode 100644 meta/lib/isar/multistrap.py >> delete mode 100644 meta/recipes-devtools/buildchroot/files/multistrap.conf.in >> >> diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in b/meta-isar/recipes-core/images/files/multistrap.conf.in >> deleted file mode 100644 >> index 432b6af..0000000 >> --- a/meta-isar/recipes-core/images/files/multistrap.conf.in >> +++ /dev/null >> @@ -1,38 +0,0 @@ >> -# This software is a part of ISAR. >> -# Copyright (C) 2015-2017 ilbers GmbH >> - >> -[General] >> -noauth=true >> -unpack=true >> -ignorenativearch=true >> -bootstrap=##DISTRO_MULTICONF_BOOTSTRAP## Isar >> -aptsources=##DISTRO_MULTICONF_APTSOURCES## >> -configscript=##CONFIG_SCRIPT## >> -setupscript=##SETUP_SCRIPT## >> -hookdir=##DIR_HOOKS## >> - >> -[base] >> -source=##DISTRO_APT_SOURCE## >> -suite=##DISTRO_SUITE## >> -components=##DISTRO_COMPONENTS## >> -packages=##IMAGE_PREINSTALL## >> -omitdebsrc=true >> - >> -[updates] >> -source=##DISTRO_APT_SOURCE## >> -suite=##DISTRO_SUITE##-updates >> -components=##DISTRO_COMPONENTS## >> -omitdebsrc=true >> - >> -[security] >> -source=##DISTRO_APT_SOURCE_SEC## >> -suite=##DISTRO_SUITE##/updates >> -components=##DISTRO_COMPONENTS## >> -omitdebsrc=true >> - >> -[Isar] >> -packages=##IMAGE_INSTALL## >> -source=##DEPLOY_DIR_APT## >> -suite=##ISAR_DISTRO_SUITE## >> -components=main >> -omitdebsrc=true >> diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb >> index 8ddbabb..9913977 100644 >> --- a/meta-isar/recipes-core/images/isar-image-base.bb >> +++ b/meta-isar/recipes-core/images/isar-image-base.bb >> @@ -9,8 +9,7 @@ LICENSE = "gpl-2.0" >> LIC_FILES_CHKSUM = "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe" >> >> FILESPATH =. "${LAYERDIR_isar}/recipes-core/images/files:" >> -SRC_URI = "file://multistrap.conf.in \ >> - file://${DISTRO_CONFIG_SCRIPT} \ >> +SRC_URI = "file://${DISTRO_CONFIG_SCRIPT} \ >> file://setup.sh \ >> file://download_dev-random" >> >> @@ -28,6 +27,14 @@ WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}" >> >> do_rootfs[dirs] = "${WORKDIR}/hooks_multistrap" >> >> +MULTISTRAP_CONFIG_SCRIPT = "${DISTRO_CONFIG_SCRIPT}" >> +MULTISTRAP_APTSOURCES = "${DISTRO_MULTICONF_APTSOURCES}" >> +MULTISTRAP_BOOTSTRAP = "${DISTRO_MULTICONF_BOOTSTRAP} isar" >> +MULTISTRAP_PREINSTALL = "${IMAGE_PREINSTALL}" >> +MULTISTRAP_ISAR_SOURCE = "copy:///${DEPLOY_DIR_APT}/${DISTRO}" >> +MULTISTRAP_ISAR_INSTALL = "${IMAGE_INSTALL}" >> +MULTISTRAP_ISAR_SUITE = "${DEBDISTRONAME}" >> + >> do_rootfs() { >> E="${@ bb.utils.export_proxies(d)}" >> >> @@ -35,25 +42,8 @@ do_rootfs() { >> chmod +x "${WORKDIR}/setup.sh" >> install -m 755 "${WORKDIR}/download_dev-random" "${WORKDIR}/hooks_multistrap/" >> >> - # Multistrap accepts only relative path in configuration files, so get it: >> - cd ${TOPDIR} >> - WORKDIR_REL=${@ os.path.relpath(d.getVar("WORKDIR", True))} >> - >> - # Adjust multistrap config >> - sed -e 's|##IMAGE_PREINSTALL##|${IMAGE_PREINSTALL}|g' \ >> - -e 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \ >> - -e 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \ >> - -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \ >> - -e 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \ >> - -e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \ >> - -e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \ >> - -e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/${DISTRO_CONFIG_SCRIPT}|g' \ >> - -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \ >> - -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \ >> - -e 's|##IMAGE_INSTALL##|${IMAGE_INSTALL}|g' \ >> - -e 's|##DEPLOY_DIR_APT##|copy:///${DEPLOY_DIR_APT}/${DISTRO}|g' \ >> - -e 's|##ISAR_DISTRO_SUITE##|${DEBDISTRONAME}|g' \ >> - "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf" >> + echo "${@ isar.multistrap.generate_multistrap_config(d) }" > \ >> + "${WORKDIR}/multistrap.conf" >> >> # Do not use bitbake flag [dirs] here because this folder should have >> # specific ownership. >> @@ -62,6 +52,7 @@ do_rootfs() { >> >> # Create root filesystem. We must use sudo -E here to preserve the environment >> # because of proxy settings >> + cd "${TOPDIR}" >> sudo -E multistrap -a ${DISTRO_ARCH} -d "${IMAGE_ROOTFS}" -f "${WORKDIR}/multistrap.conf" >> >> # Configure root filesystem >> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass >> index 3334789..2a45a1e 100644 >> --- a/meta/classes/base.bbclass >> +++ b/meta/classes/base.bbclass >> @@ -38,7 +38,7 @@ bbfatal() { >> } >> >> # Derived from openembedded-core/meta/classes/base.bbclass >> -ISAR_IMPORTS += "os sys time" >> +ISAR_IMPORTS += "os sys time isar" >> >> def isar_imports(d): >> import sys >> diff --git a/meta/lib/isar/__init__.py b/meta/lib/isar/__init__.py >> new file mode 100644 >> index 0000000..e6aeeaa >> --- /dev/null >> +++ b/meta/lib/isar/__init__.py >> @@ -0,0 +1,8 @@ >> +# Custom kernel recipe include >> +# >> +# This software is a part of ISAR. >> +# Copyright (c) Siemens AG, 2018 >> +# >> +# SPDX-License-Identifier: MIT >> + >> +from . import multistrap >> diff --git a/meta/lib/isar/multistrap.py b/meta/lib/isar/multistrap.py >> new file mode 100644 >> index 0000000..7022f91 >> --- /dev/null >> +++ b/meta/lib/isar/multistrap.py >> @@ -0,0 +1,80 @@ >> +# Custom kernel recipe include >> +# >> +# This software is a part of ISAR. >> +# Copyright (c) Siemens AG, 2018 >> +# >> +# SPDX-License-Identifier: MIT >> + >> +def generate_multistrap_config(d): >> + import configparser as cp >> + import io >> + >> + workdir = d.getVar("WORKDIR", True) or "" >> + topdir = d.getVar("TOPDIR", True) or "" >> + workdir_rel = os.path.relpath(workdir, start=topdir) >> + >> + msconf = cp.RawConfigParser() >> + >> + # Set general multistrap configuration: >> + configscript = d.getVar("MULTISTRAP_CONFIG_SCRIPT", True) or "" >> + setupscript = d.getVar("MULTISTRAP_SETUP_SCRIPT", True) or "setup.sh" >> + hookdir = d.getVar("MULTISTRAP_HOOK_DIR", True) or "hooks_multistrap" >> + aptsources = (d.getVar("MULTISTRAP_APTSOURCES", True) or "").split() >> + bootstrap = (d.getVar("MULTISTRAP_BOOTSTRAP", True) or "").split() >> + >> + msconf.add_section("General") >> + msconf.set("General", "noauth", "true") >> + msconf.set("General", "unpack", "true") >> + msconf.set("General", "ignorenativearch", "true") >> + msconf.set("General", "configscript", "./" + >> + os.path.join(workdir_rel, configscript)) >> + msconf.set("General", "setupscript", "./" + >> + os.path.join(workdir_rel, setupscript)) >> + msconf.set("General", "hookdir", "./" + >> + os.path.join(workdir_rel, hookdir)) >> + msconf.set("General", "aptsources", " ".join(aptsources)) >> + msconf.set("General", "bootstrap", " ".join(bootstrap)) >> + >> + # Setup upstream repositories in multistrap configuration: >> + apt_source = d.getVar("DISTRO_APT_SOURCE", True) or "" >> + apt_source_sec = d.getVar("DISTRO_APT_SOURCE_SEC", True) or "" >> + apt_suite = d.getVar("DISTRO_SUITE", True) or "" >> + apt_components = (d.getVar("DISTRO_COMPONENTS", True) or "").split() >> + deb_packages = (d.getVar("MULTISTRAP_PREINSTALL", True) or "").split() >> + isar_source = d.getVar("MULTISTRAP_ISAR_SOURCE", True) or "" >> + isar_packages = (d.getVar("MULTISTRAP_ISAR_INSTALL", True) or "").split() >> + isar_suite = d.getVar("MULTISTRAP_ISAR_SUITE", True) or "isar" >> + >> + msconf.add_section("base") >> + msconf.set("base", "source", apt_source) >> + msconf.set("base", "suite", apt_suite) >> + msconf.set("base", "components", " ".join(apt_components)) >> + msconf.set("base", "omitdebsrc", "true") >> + msconf.set("base", "packages", " ".join(deb_packages)) >> + >> + msconf.add_section("updates") >> + msconf.set("updates", "source", apt_source) >> + msconf.set("updates", "suite", apt_suite + "-updates") >> + msconf.set("updates", "components", " ".join(apt_components)) >> + msconf.set("updates", "omitdebsrc", "true") >> + >> + msconf.add_section("security") >> + msconf.set("security", "source", apt_source_sec) >> + msconf.set("security", "suite", apt_suite + "/updates") >> + msconf.set("security", "components", " ".join(apt_components)) >> + msconf.set("security", "omitdebsrc", "true") >> + >> + msconf.add_section("isar") >> + msconf.set("isar", "source", isar_source) >> + msconf.set("isar", "suite", isar_suite) >> + msconf.set("isar", "components", "main") >> + msconf.set("isar", "omitdebsrc", "true") >> + if isar_packages : >> + msconf.set("isar", "packages", >> + " ".join(isar_packages)) >> + >> + ret = "" >> + with io.StringIO() as out: >> + msconf.write(out) >> + ret = out.getvalue() >> + return ret >> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb >> index 570c0ad..59b4e7e 100644 >> --- a/meta/recipes-devtools/buildchroot/buildchroot.bb >> +++ b/meta/recipes-devtools/buildchroot/buildchroot.bb >> @@ -9,8 +9,7 @@ LICENSE = "gpl-2.0" >> LIC_FILES_CHKSUM = "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe" >> >> FILESPATH =. "${LAYERDIR_core}/recipes-devtools/buildchroot/files:" >> -SRC_URI = "file://multistrap.conf.in \ >> - file://configscript.sh \ >> +SRC_URI = "file://configscript.sh \ >> file://setup.sh \ >> file://build.sh" >> PV = "1.0" >> @@ -37,32 +36,25 @@ do_build[dirs] = "${BUILDCHROOT_DIR}/isar-apt \ >> ${BUILDCHROOT_DIR}/sys" >> do_build[depends] = "isar-apt:do_cache_config" >> >> +MULTISTRAP_CONFIG_SCRIPT = "configscript.sh" >> +MULTISTRAP_APTSOURCES = "isar ${DISTRO_MULTICONF_APTSOURCES}" >> +MULTISTRAP_BOOTSTRAP = "${DISTRO_MULTICONF_BOOTSTRAP}" >> +MULTISTRAP_PREINSTALL = "${BUILDCHROOT_PREINSTALL}" >> +MULTISTRAP_ISAR_SOURCE = "file:///isar-apt" >> + >> do_build() { >> E="${@ bb.utils.export_proxies(d)}" >> >> chmod +x "${WORKDIR}/setup.sh" >> chmod +x "${WORKDIR}/configscript.sh" >> >> - # Multistrap accepts only relative path in configuration files, so get it: >> - cd ${TOPDIR} >> - WORKDIR_REL=${@ os.path.relpath(d.getVar("WORKDIR", True))} >> - >> - # Adjust multistrap config >> - sed -e 's|##BUILDCHROOT_PREINSTALL##|${BUILDCHROOT_PREINSTALL}|g' \ >> - -e 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \ >> - -e 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \ >> - -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \ >> - -e 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \ >> - -e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \ >> - -e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \ >> - -e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/configscript.sh|g' \ >> - -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \ >> - -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \ >> - "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf" >> + echo "${@ isar.multistrap.generate_multistrap_config(d) }" > \ >> + "${WORKDIR}/multistrap.conf" >> >> do_setup_mounts >> >> # Create root filesystem >> + cd "${TOPDIR}" >> sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f "${WORKDIR}/multistrap.conf" >> >> # Install package builder script >> diff --git a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in >> deleted file mode 100644 >> index 480a4b8..0000000 >> --- a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in >> +++ /dev/null >> @@ -1,37 +0,0 @@ >> -# This software is a part of ISAR. >> -# Copyright (C) 2015-2017 ilbers GmbH >> - >> -[General] >> -noauth=true >> -unpack=true >> -ignorenativearch=true >> -bootstrap=##DISTRO_MULTICONF_BOOTSTRAP## >> -aptsources=isar-apt ##DISTRO_MULTICONF_APTSOURCES## >> -configscript=##CONFIG_SCRIPT## >> -setupscript=##SETUP_SCRIPT## >> -hookdir=##DIR_HOOKS## >> - >> -[base] >> -source=##DISTRO_APT_SOURCE## >> -suite=##DISTRO_SUITE## >> -components=##DISTRO_COMPONENTS## >> -packages=##BUILDCHROOT_PREINSTALL## >> -omitdebsrc=true >> - >> -[updates] >> -source=##DISTRO_APT_SOURCE## >> -suite=##DISTRO_SUITE##-updates >> -components=##DISTRO_COMPONENTS## >> -omitdebsrc=true >> - >> -[security] >> -source=##DISTRO_APT_SOURCE_SEC## >> -suite=##DISTRO_SUITE##/updates >> -components=##DISTRO_COMPONENTS## >> -omitdebsrc=true >> - >> -[isar-apt] >> -source=file:///isar-apt >> -suite=isar >> -components=main >> -omitdebsrc=true >> > > This requires rebasing. I quickly tried, but the result broke the > package build (didn't analyze any details). I rebased this patch to next before sending it to the mailinglist, but there might have been some other patches applied in the between. If you put this patch on top of: af265bc0731d242afc5bc0ac9e8ebc6ffe6223f6 It should work. (It did at least for me.) > While this moves us in the right direction, it still leaves some > duplications behind (setup.sh) - and the big question if we should > invest any further in this horrible multistrap mess. > > I've just enabled arm64, and it took a while to figure out that I was > missing some lines in setup.sh. In contrast, qemu-debootstrap just > worked (I used it to debug the multistrap output). > > So, before we move on here: What all does multistrap currently give us, > compared to standard debootstrap? I only see "multiple repositories" so > far. But as Claudius is going to massage that part heavily anyway, I > really wonder if it's worth building on the multistrap indirection any > further. > > How about something like this? > > - [qemu-]debootstrap --variant=minbase ... > - inject all additional apt sources, including user-provided ones > - set pinnings and priority, including user-provided ones > - apt-get dist-upgrade > > That would kick out all the multistrap complications. I would prefer using debootstrap as well. I am currently looking at its code, and maybe it could be possible to further split up debootstraps execution into downloading, extracting, configuring steps. This way only configuring has to be done with root permissions (because of chroot) and qemu if needed. Of course overwriting any base packages with generated isar ones might not be as convenient, because packages might have to be replaced at a later step. If some additional functionality in debootstrap is needed, then it could make sense to fetch debootstrap in isar, like any other recipe, and apply some of our own patches to it. Of course we should try to upstream those too. Cheers, Claudius -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 2/2] centralize multistrap configuration generation 2018-02-15 8:16 ` Claudius Heine @ 2018-02-15 8:50 ` Jan Kiszka 0 siblings, 0 replies; 23+ messages in thread From: Jan Kiszka @ 2018-02-15 8:50 UTC (permalink / raw) To: Claudius Heine, isar-users, Alexander Smirnov, Baurzhan Ismagulov Cc: Claudius Heine On 2018-02-15 09:16, Claudius Heine wrote: > Hi Jan, > > On 02/15/2018 08:28 AM, Jan Kiszka wrote: >> On 2018-02-14 14:15, [ext] claudius.heine.ext@siemens.com wrote: >>> From: Claudius Heine <ch@denx.de> >>> >>> Previous there was redundant code between the buildchroot and >>> isar-image-base recipe. >>> >>> This patch moves the generation of the multistrap configuration to a >>> central python script that can be parametrized by both recipes. >>> >>> Signed-off-by: Claudius Heine <ch@denx.de> >>> --- >>> .../recipes-core/images/files/multistrap.conf.in | 38 ---------- >>> meta-isar/recipes-core/images/isar-image-base.bb | 33 ++++----- >>> meta/classes/base.bbclass | 2 +- >>> meta/lib/isar/__init__.py | 8 +++ >>> meta/lib/isar/multistrap.py | 80 >>> ++++++++++++++++++++++ >>> meta/recipes-devtools/buildchroot/buildchroot.bb | 28 +++----- >>> .../buildchroot/files/multistrap.conf.in | 37 ---------- >>> 7 files changed, 111 insertions(+), 115 deletions(-) >>> delete mode 100644 >>> meta-isar/recipes-core/images/files/multistrap.conf.in >>> create mode 100644 meta/lib/isar/__init__.py >>> create mode 100644 meta/lib/isar/multistrap.py >>> delete mode 100644 >>> meta/recipes-devtools/buildchroot/files/multistrap.conf.in >>> >>> diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in >>> b/meta-isar/recipes-core/images/files/multistrap.conf.in >>> deleted file mode 100644 >>> index 432b6af..0000000 >>> --- a/meta-isar/recipes-core/images/files/multistrap.conf.in >>> +++ /dev/null >>> @@ -1,38 +0,0 @@ >>> -# This software is a part of ISAR. >>> -# Copyright (C) 2015-2017 ilbers GmbH >>> - >>> -[General] >>> -noauth=true >>> -unpack=true >>> -ignorenativearch=true >>> -bootstrap=##DISTRO_MULTICONF_BOOTSTRAP## Isar >>> -aptsources=##DISTRO_MULTICONF_APTSOURCES## >>> -configscript=##CONFIG_SCRIPT## >>> -setupscript=##SETUP_SCRIPT## >>> -hookdir=##DIR_HOOKS## >>> - >>> -[base] >>> -source=##DISTRO_APT_SOURCE## >>> -suite=##DISTRO_SUITE## >>> -components=##DISTRO_COMPONENTS## >>> -packages=##IMAGE_PREINSTALL## >>> -omitdebsrc=true >>> - >>> -[updates] >>> -source=##DISTRO_APT_SOURCE## >>> -suite=##DISTRO_SUITE##-updates >>> -components=##DISTRO_COMPONENTS## >>> -omitdebsrc=true >>> - >>> -[security] >>> -source=##DISTRO_APT_SOURCE_SEC## >>> -suite=##DISTRO_SUITE##/updates >>> -components=##DISTRO_COMPONENTS## >>> -omitdebsrc=true >>> - >>> -[Isar] >>> -packages=##IMAGE_INSTALL## >>> -source=##DEPLOY_DIR_APT## >>> -suite=##ISAR_DISTRO_SUITE## >>> -components=main >>> -omitdebsrc=true >>> diff --git a/meta-isar/recipes-core/images/isar-image-base.bb >>> b/meta-isar/recipes-core/images/isar-image-base.bb >>> index 8ddbabb..9913977 100644 >>> --- a/meta-isar/recipes-core/images/isar-image-base.bb >>> +++ b/meta-isar/recipes-core/images/isar-image-base.bb >>> @@ -9,8 +9,7 @@ LICENSE = "gpl-2.0" >>> LIC_FILES_CHKSUM = >>> "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe" >>> >>> FILESPATH =. "${LAYERDIR_isar}/recipes-core/images/files:" >>> -SRC_URI = "file://multistrap.conf.in \ >>> - file://${DISTRO_CONFIG_SCRIPT} \ >>> +SRC_URI = "file://${DISTRO_CONFIG_SCRIPT} \ >>> file://setup.sh \ >>> file://download_dev-random" >>> @@ -28,6 +27,14 @@ WORKDIR = >>> "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}" >>> do_rootfs[dirs] = "${WORKDIR}/hooks_multistrap" >>> +MULTISTRAP_CONFIG_SCRIPT = "${DISTRO_CONFIG_SCRIPT}" >>> +MULTISTRAP_APTSOURCES = "${DISTRO_MULTICONF_APTSOURCES}" >>> +MULTISTRAP_BOOTSTRAP = "${DISTRO_MULTICONF_BOOTSTRAP} isar" >>> +MULTISTRAP_PREINSTALL = "${IMAGE_PREINSTALL}" >>> +MULTISTRAP_ISAR_SOURCE = "copy:///${DEPLOY_DIR_APT}/${DISTRO}" >>> +MULTISTRAP_ISAR_INSTALL = "${IMAGE_INSTALL}" >>> +MULTISTRAP_ISAR_SUITE = "${DEBDISTRONAME}" >>> + >>> do_rootfs() { >>> E="${@ bb.utils.export_proxies(d)}" >>> @@ -35,25 +42,8 @@ do_rootfs() { >>> chmod +x "${WORKDIR}/setup.sh" >>> install -m 755 "${WORKDIR}/download_dev-random" >>> "${WORKDIR}/hooks_multistrap/" >>> - # Multistrap accepts only relative path in configuration >>> files, so get it: >>> - cd ${TOPDIR} >>> - WORKDIR_REL=${@ os.path.relpath(d.getVar("WORKDIR", True))} >>> - >>> - # Adjust multistrap config >>> - sed -e 's|##IMAGE_PREINSTALL##|${IMAGE_PREINSTALL}|g' \ >>> - -e >>> 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \ >>> - -e >>> 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \ >>> - -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \ >>> - -e 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \ >>> - -e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \ >>> - -e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \ >>> - -e >>> 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/${DISTRO_CONFIG_SCRIPT}|g' \ >>> - -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \ >>> - -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \ >>> - -e 's|##IMAGE_INSTALL##|${IMAGE_INSTALL}|g' \ >>> - -e >>> 's|##DEPLOY_DIR_APT##|copy:///${DEPLOY_DIR_APT}/${DISTRO}|g' \ >>> - -e 's|##ISAR_DISTRO_SUITE##|${DEBDISTRONAME}|g' \ >>> - "${WORKDIR}/multistrap.conf.in" > >>> "${WORKDIR}/multistrap.conf" >>> + echo "${@ isar.multistrap.generate_multistrap_config(d) }" > \ >>> + "${WORKDIR}/multistrap.conf" >>> # Do not use bitbake flag [dirs] here because this folder >>> should have >>> # specific ownership. >>> @@ -62,6 +52,7 @@ do_rootfs() { >>> # Create root filesystem. We must use sudo -E here to >>> preserve the environment >>> # because of proxy settings >>> + cd "${TOPDIR}" >>> sudo -E multistrap -a ${DISTRO_ARCH} -d "${IMAGE_ROOTFS}" -f >>> "${WORKDIR}/multistrap.conf" >>> # Configure root filesystem >>> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass >>> index 3334789..2a45a1e 100644 >>> --- a/meta/classes/base.bbclass >>> +++ b/meta/classes/base.bbclass >>> @@ -38,7 +38,7 @@ bbfatal() { >>> } >>> # Derived from openembedded-core/meta/classes/base.bbclass >>> -ISAR_IMPORTS += "os sys time" >>> +ISAR_IMPORTS += "os sys time isar" >>> def isar_imports(d): >>> import sys >>> diff --git a/meta/lib/isar/__init__.py b/meta/lib/isar/__init__.py >>> new file mode 100644 >>> index 0000000..e6aeeaa >>> --- /dev/null >>> +++ b/meta/lib/isar/__init__.py >>> @@ -0,0 +1,8 @@ >>> +# Custom kernel recipe include >>> +# >>> +# This software is a part of ISAR. >>> +# Copyright (c) Siemens AG, 2018 >>> +# >>> +# SPDX-License-Identifier: MIT >>> + >>> +from . import multistrap >>> diff --git a/meta/lib/isar/multistrap.py b/meta/lib/isar/multistrap.py >>> new file mode 100644 >>> index 0000000..7022f91 >>> --- /dev/null >>> +++ b/meta/lib/isar/multistrap.py >>> @@ -0,0 +1,80 @@ >>> +# Custom kernel recipe include >>> +# >>> +# This software is a part of ISAR. >>> +# Copyright (c) Siemens AG, 2018 >>> +# >>> +# SPDX-License-Identifier: MIT >>> + >>> +def generate_multistrap_config(d): >>> + import configparser as cp >>> + import io >>> + >>> + workdir = d.getVar("WORKDIR", True) or "" >>> + topdir = d.getVar("TOPDIR", True) or "" >>> + workdir_rel = os.path.relpath(workdir, start=topdir) >>> + >>> + msconf = cp.RawConfigParser() >>> + >>> + # Set general multistrap configuration: >>> + configscript = d.getVar("MULTISTRAP_CONFIG_SCRIPT", True) or "" >>> + setupscript = d.getVar("MULTISTRAP_SETUP_SCRIPT", True) or >>> "setup.sh" >>> + hookdir = d.getVar("MULTISTRAP_HOOK_DIR", True) or >>> "hooks_multistrap" >>> + aptsources = (d.getVar("MULTISTRAP_APTSOURCES", True) or >>> "").split() >>> + bootstrap = (d.getVar("MULTISTRAP_BOOTSTRAP", True) or "").split() >>> + >>> + msconf.add_section("General") >>> + msconf.set("General", "noauth", "true") >>> + msconf.set("General", "unpack", "true") >>> + msconf.set("General", "ignorenativearch", "true") >>> + msconf.set("General", "configscript", "./" + >>> + os.path.join(workdir_rel, configscript)) >>> + msconf.set("General", "setupscript", "./" + >>> + os.path.join(workdir_rel, setupscript)) >>> + msconf.set("General", "hookdir", "./" + >>> + os.path.join(workdir_rel, hookdir)) >>> + msconf.set("General", "aptsources", " ".join(aptsources)) >>> + msconf.set("General", "bootstrap", " ".join(bootstrap)) >>> + >>> + # Setup upstream repositories in multistrap configuration: >>> + apt_source = d.getVar("DISTRO_APT_SOURCE", True) or "" >>> + apt_source_sec = d.getVar("DISTRO_APT_SOURCE_SEC", True) or "" >>> + apt_suite = d.getVar("DISTRO_SUITE", True) or "" >>> + apt_components = (d.getVar("DISTRO_COMPONENTS", True) or >>> "").split() >>> + deb_packages = (d.getVar("MULTISTRAP_PREINSTALL", True) or >>> "").split() >>> + isar_source = d.getVar("MULTISTRAP_ISAR_SOURCE", True) or "" >>> + isar_packages = (d.getVar("MULTISTRAP_ISAR_INSTALL", True) or >>> "").split() >>> + isar_suite = d.getVar("MULTISTRAP_ISAR_SUITE", True) or "isar" >>> + >>> + msconf.add_section("base") >>> + msconf.set("base", "source", apt_source) >>> + msconf.set("base", "suite", apt_suite) >>> + msconf.set("base", "components", " ".join(apt_components)) >>> + msconf.set("base", "omitdebsrc", "true") >>> + msconf.set("base", "packages", " ".join(deb_packages)) >>> + >>> + msconf.add_section("updates") >>> + msconf.set("updates", "source", apt_source) >>> + msconf.set("updates", "suite", apt_suite + "-updates") >>> + msconf.set("updates", "components", " ".join(apt_components)) >>> + msconf.set("updates", "omitdebsrc", "true") >>> + >>> + msconf.add_section("security") >>> + msconf.set("security", "source", apt_source_sec) >>> + msconf.set("security", "suite", apt_suite + "/updates") >>> + msconf.set("security", "components", " ".join(apt_components)) >>> + msconf.set("security", "omitdebsrc", "true") >>> + >>> + msconf.add_section("isar") >>> + msconf.set("isar", "source", isar_source) >>> + msconf.set("isar", "suite", isar_suite) >>> + msconf.set("isar", "components", "main") >>> + msconf.set("isar", "omitdebsrc", "true") >>> + if isar_packages : >>> + msconf.set("isar", "packages", >>> + " ".join(isar_packages)) >>> + >>> + ret = "" >>> + with io.StringIO() as out: >>> + msconf.write(out) >>> + ret = out.getvalue() >>> + return ret >>> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb >>> b/meta/recipes-devtools/buildchroot/buildchroot.bb >>> index 570c0ad..59b4e7e 100644 >>> --- a/meta/recipes-devtools/buildchroot/buildchroot.bb >>> +++ b/meta/recipes-devtools/buildchroot/buildchroot.bb >>> @@ -9,8 +9,7 @@ LICENSE = "gpl-2.0" >>> LIC_FILES_CHKSUM = >>> "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe" >>> >>> FILESPATH =. "${LAYERDIR_core}/recipes-devtools/buildchroot/files:" >>> -SRC_URI = "file://multistrap.conf.in \ >>> - file://configscript.sh \ >>> +SRC_URI = "file://configscript.sh \ >>> file://setup.sh \ >>> file://build.sh" >>> PV = "1.0" >>> @@ -37,32 +36,25 @@ do_build[dirs] = "${BUILDCHROOT_DIR}/isar-apt \ >>> ${BUILDCHROOT_DIR}/sys" >>> do_build[depends] = "isar-apt:do_cache_config" >>> +MULTISTRAP_CONFIG_SCRIPT = "configscript.sh" >>> +MULTISTRAP_APTSOURCES = "isar ${DISTRO_MULTICONF_APTSOURCES}" >>> +MULTISTRAP_BOOTSTRAP = "${DISTRO_MULTICONF_BOOTSTRAP}" >>> +MULTISTRAP_PREINSTALL = "${BUILDCHROOT_PREINSTALL}" >>> +MULTISTRAP_ISAR_SOURCE = "file:///isar-apt" >>> + >>> do_build() { >>> E="${@ bb.utils.export_proxies(d)}" >>> chmod +x "${WORKDIR}/setup.sh" >>> chmod +x "${WORKDIR}/configscript.sh" >>> - # Multistrap accepts only relative path in configuration >>> files, so get it: >>> - cd ${TOPDIR} >>> - WORKDIR_REL=${@ os.path.relpath(d.getVar("WORKDIR", True))} >>> - >>> - # Adjust multistrap config >>> - sed -e 's|##BUILDCHROOT_PREINSTALL##|${BUILDCHROOT_PREINSTALL}|g' \ >>> - -e >>> 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \ >>> - -e >>> 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \ >>> - -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \ >>> - -e 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \ >>> - -e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \ >>> - -e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \ >>> - -e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/configscript.sh|g' \ >>> - -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \ >>> - -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \ >>> - "${WORKDIR}/multistrap.conf.in" > >>> "${WORKDIR}/multistrap.conf" >>> + echo "${@ isar.multistrap.generate_multistrap_config(d) }" > \ >>> + "${WORKDIR}/multistrap.conf" >>> do_setup_mounts >>> # Create root filesystem >>> + cd "${TOPDIR}" >>> sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f >>> "${WORKDIR}/multistrap.conf" >>> # Install package builder script >>> diff --git >>> a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in >>> b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in >>> deleted file mode 100644 >>> index 480a4b8..0000000 >>> --- a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in >>> +++ /dev/null >>> @@ -1,37 +0,0 @@ >>> -# This software is a part of ISAR. >>> -# Copyright (C) 2015-2017 ilbers GmbH >>> - >>> -[General] >>> -noauth=true >>> -unpack=true >>> -ignorenativearch=true >>> -bootstrap=##DISTRO_MULTICONF_BOOTSTRAP## >>> -aptsources=isar-apt ##DISTRO_MULTICONF_APTSOURCES## >>> -configscript=##CONFIG_SCRIPT## >>> -setupscript=##SETUP_SCRIPT## >>> -hookdir=##DIR_HOOKS## >>> - >>> -[base] >>> -source=##DISTRO_APT_SOURCE## >>> -suite=##DISTRO_SUITE## >>> -components=##DISTRO_COMPONENTS## >>> -packages=##BUILDCHROOT_PREINSTALL## >>> -omitdebsrc=true >>> - >>> -[updates] >>> -source=##DISTRO_APT_SOURCE## >>> -suite=##DISTRO_SUITE##-updates >>> -components=##DISTRO_COMPONENTS## >>> -omitdebsrc=true >>> - >>> -[security] >>> -source=##DISTRO_APT_SOURCE_SEC## >>> -suite=##DISTRO_SUITE##/updates >>> -components=##DISTRO_COMPONENTS## >>> -omitdebsrc=true >>> - >>> -[isar-apt] >>> -source=file:///isar-apt >>> -suite=isar >>> -components=main >>> -omitdebsrc=true >>> >> >> This requires rebasing. I quickly tried, but the result broke the >> package build (didn't analyze any details). > > I rebased this patch to next before sending it to the mailinglist, but > there might have been some other patches applied in the between. > > If you put this patch on top of: > af265bc0731d242afc5bc0ac9e8ebc6ffe6223f6 > It should work. (It did at least for me.) > Next moved on, so this required rebasing, maybe more. But let's first of all decide about the strategy before investing further into multistrap. >> While this moves us in the right direction, it still leaves some >> duplications behind (setup.sh) - and the big question if we should >> invest any further in this horrible multistrap mess. >> >> I've just enabled arm64, and it took a while to figure out that I was >> missing some lines in setup.sh. In contrast, qemu-debootstrap just >> worked (I used it to debug the multistrap output). >> >> So, before we move on here: What all does multistrap currently give us, >> compared to standard debootstrap? I only see "multiple repositories" so >> far. But as Claudius is going to massage that part heavily anyway, I >> really wonder if it's worth building on the multistrap indirection any >> further. >> >> How about something like this? >> >> - [qemu-]debootstrap --variant=minbase ... >> - inject all additional apt sources, including user-provided ones >> - set pinnings and priority, including user-provided ones >> - apt-get dist-upgrade >> >> That would kick out all the multistrap complications. > > I would prefer using debootstrap as well. I am currently looking at its > code, and maybe it could be possible to further split up debootstraps > execution into downloading, extracting, configuring steps. This way only > configuring has to be done with root permissions (because of chroot) and > qemu if needed.> > Of course overwriting any base packages with generated isar ones might > not be as convenient, because packages might have to be replaced at a > later step. That was the idea behind "dist-upgrade", though I cannot say for sure it will work reliably enough. > > If some additional functionality in debootstrap is needed, then it could > make sense to fetch debootstrap in isar, like any other recipe, and > apply some of our own patches to it. Of course we should try to upstream > those too. Sounds promising. I'm sure there will be traps and pitfalls, but so there are with multistrap, and its code become rather a barrier than a help for us. Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 2/2] centralize multistrap configuration generation 2018-02-15 7:28 ` Jan Kiszka 2018-02-15 8:16 ` Claudius Heine @ 2018-02-15 9:34 ` Alexander Smirnov 2018-02-15 10:05 ` Claudius Heine 1 sibling, 1 reply; 23+ messages in thread From: Alexander Smirnov @ 2018-02-15 9:34 UTC (permalink / raw) To: Jan Kiszka, isar-users, Baurzhan Ismagulov; +Cc: Claudius Heine On 02/15/2018 10:28 AM, Jan Kiszka wrote: > On 2018-02-14 14:15, [ext] claudius.heine.ext@siemens.com wrote: >> From: Claudius Heine <ch@denx.de> >> >> Previous there was redundant code between the buildchroot and >> isar-image-base recipe. >> >> This patch moves the generation of the multistrap configuration to a >> central python script that can be parametrized by both recipes. >> >> Signed-off-by: Claudius Heine <ch@denx.de> >> --- >> .../recipes-core/images/files/multistrap.conf.in | 38 ---------- >> meta-isar/recipes-core/images/isar-image-base.bb | 33 ++++----- >> meta/classes/base.bbclass | 2 +- >> meta/lib/isar/__init__.py | 8 +++ >> meta/lib/isar/multistrap.py | 80 ++++++++++++++++++++++ >> meta/recipes-devtools/buildchroot/buildchroot.bb | 28 +++----- >> .../buildchroot/files/multistrap.conf.in | 37 ---------- >> 7 files changed, 111 insertions(+), 115 deletions(-) >> delete mode 100644 meta-isar/recipes-core/images/files/multistrap.conf.in >> create mode 100644 meta/lib/isar/__init__.py >> create mode 100644 meta/lib/isar/multistrap.py >> delete mode 100644 meta/recipes-devtools/buildchroot/files/multistrap.conf.in >> >> diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in b/meta-isar/recipes-core/images/files/multistrap.conf.in >> deleted file mode 100644 >> index 432b6af..0000000 >> --- a/meta-isar/recipes-core/images/files/multistrap.conf.in >> +++ /dev/null >> @@ -1,38 +0,0 @@ >> -# This software is a part of ISAR. >> -# Copyright (C) 2015-2017 ilbers GmbH >> - >> -[General] >> -noauth=true >> -unpack=true >> -ignorenativearch=true >> -bootstrap=##DISTRO_MULTICONF_BOOTSTRAP## Isar >> -aptsources=##DISTRO_MULTICONF_APTSOURCES## >> -configscript=##CONFIG_SCRIPT## >> -setupscript=##SETUP_SCRIPT## >> -hookdir=##DIR_HOOKS## >> - >> -[base] >> -source=##DISTRO_APT_SOURCE## >> -suite=##DISTRO_SUITE## >> -components=##DISTRO_COMPONENTS## >> -packages=##IMAGE_PREINSTALL## >> -omitdebsrc=true >> - >> -[updates] >> -source=##DISTRO_APT_SOURCE## >> -suite=##DISTRO_SUITE##-updates >> -components=##DISTRO_COMPONENTS## >> -omitdebsrc=true >> - >> -[security] >> -source=##DISTRO_APT_SOURCE_SEC## >> -suite=##DISTRO_SUITE##/updates >> -components=##DISTRO_COMPONENTS## >> -omitdebsrc=true >> - >> -[Isar] >> -packages=##IMAGE_INSTALL## >> -source=##DEPLOY_DIR_APT## >> -suite=##ISAR_DISTRO_SUITE## >> -components=main >> -omitdebsrc=true >> diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb >> index 8ddbabb..9913977 100644 >> --- a/meta-isar/recipes-core/images/isar-image-base.bb >> +++ b/meta-isar/recipes-core/images/isar-image-base.bb >> @@ -9,8 +9,7 @@ LICENSE = "gpl-2.0" >> LIC_FILES_CHKSUM = "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe" >> >> FILESPATH =. "${LAYERDIR_isar}/recipes-core/images/files:" >> -SRC_URI = "file://multistrap.conf.in \ >> - file://${DISTRO_CONFIG_SCRIPT} \ >> +SRC_URI = "file://${DISTRO_CONFIG_SCRIPT} \ >> file://setup.sh \ >> file://download_dev-random" >> >> @@ -28,6 +27,14 @@ WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}" >> >> do_rootfs[dirs] = "${WORKDIR}/hooks_multistrap" >> >> +MULTISTRAP_CONFIG_SCRIPT = "${DISTRO_CONFIG_SCRIPT}" >> +MULTISTRAP_APTSOURCES = "${DISTRO_MULTICONF_APTSOURCES}" >> +MULTISTRAP_BOOTSTRAP = "${DISTRO_MULTICONF_BOOTSTRAP} isar" >> +MULTISTRAP_PREINSTALL = "${IMAGE_PREINSTALL}" >> +MULTISTRAP_ISAR_SOURCE = "copy:///${DEPLOY_DIR_APT}/${DISTRO}" >> +MULTISTRAP_ISAR_INSTALL = "${IMAGE_INSTALL}" >> +MULTISTRAP_ISAR_SUITE = "${DEBDISTRONAME}" >> + >> do_rootfs() { >> E="${@ bb.utils.export_proxies(d)}" >> >> @@ -35,25 +42,8 @@ do_rootfs() { >> chmod +x "${WORKDIR}/setup.sh" >> install -m 755 "${WORKDIR}/download_dev-random" "${WORKDIR}/hooks_multistrap/" >> >> - # Multistrap accepts only relative path in configuration files, so get it: >> - cd ${TOPDIR} >> - WORKDIR_REL=${@ os.path.relpath(d.getVar("WORKDIR", True))} >> - >> - # Adjust multistrap config >> - sed -e 's|##IMAGE_PREINSTALL##|${IMAGE_PREINSTALL}|g' \ >> - -e 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \ >> - -e 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \ >> - -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \ >> - -e 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \ >> - -e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \ >> - -e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \ >> - -e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/${DISTRO_CONFIG_SCRIPT}|g' \ >> - -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \ >> - -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \ >> - -e 's|##IMAGE_INSTALL##|${IMAGE_INSTALL}|g' \ >> - -e 's|##DEPLOY_DIR_APT##|copy:///${DEPLOY_DIR_APT}/${DISTRO}|g' \ >> - -e 's|##ISAR_DISTRO_SUITE##|${DEBDISTRONAME}|g' \ >> - "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf" >> + echo "${@ isar.multistrap.generate_multistrap_config(d) }" > \ >> + "${WORKDIR}/multistrap.conf" >> >> # Do not use bitbake flag [dirs] here because this folder should have >> # specific ownership. >> @@ -62,6 +52,7 @@ do_rootfs() { >> >> # Create root filesystem. We must use sudo -E here to preserve the environment >> # because of proxy settings >> + cd "${TOPDIR}" >> sudo -E multistrap -a ${DISTRO_ARCH} -d "${IMAGE_ROOTFS}" -f "${WORKDIR}/multistrap.conf" >> >> # Configure root filesystem >> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass >> index 3334789..2a45a1e 100644 >> --- a/meta/classes/base.bbclass >> +++ b/meta/classes/base.bbclass >> @@ -38,7 +38,7 @@ bbfatal() { >> } >> >> # Derived from openembedded-core/meta/classes/base.bbclass >> -ISAR_IMPORTS += "os sys time" >> +ISAR_IMPORTS += "os sys time isar" >> >> def isar_imports(d): >> import sys >> diff --git a/meta/lib/isar/__init__.py b/meta/lib/isar/__init__.py >> new file mode 100644 >> index 0000000..e6aeeaa >> --- /dev/null >> +++ b/meta/lib/isar/__init__.py >> @@ -0,0 +1,8 @@ >> +# Custom kernel recipe include >> +# >> +# This software is a part of ISAR. >> +# Copyright (c) Siemens AG, 2018 >> +# >> +# SPDX-License-Identifier: MIT >> + >> +from . import multistrap >> diff --git a/meta/lib/isar/multistrap.py b/meta/lib/isar/multistrap.py >> new file mode 100644 >> index 0000000..7022f91 >> --- /dev/null >> +++ b/meta/lib/isar/multistrap.py >> @@ -0,0 +1,80 @@ >> +# Custom kernel recipe include >> +# >> +# This software is a part of ISAR. >> +# Copyright (c) Siemens AG, 2018 >> +# >> +# SPDX-License-Identifier: MIT >> + >> +def generate_multistrap_config(d): >> + import configparser as cp >> + import io >> + >> + workdir = d.getVar("WORKDIR", True) or "" >> + topdir = d.getVar("TOPDIR", True) or "" >> + workdir_rel = os.path.relpath(workdir, start=topdir) >> + >> + msconf = cp.RawConfigParser() >> + >> + # Set general multistrap configuration: >> + configscript = d.getVar("MULTISTRAP_CONFIG_SCRIPT", True) or "" >> + setupscript = d.getVar("MULTISTRAP_SETUP_SCRIPT", True) or "setup.sh" >> + hookdir = d.getVar("MULTISTRAP_HOOK_DIR", True) or "hooks_multistrap" >> + aptsources = (d.getVar("MULTISTRAP_APTSOURCES", True) or "").split() >> + bootstrap = (d.getVar("MULTISTRAP_BOOTSTRAP", True) or "").split() >> + >> + msconf.add_section("General") >> + msconf.set("General", "noauth", "true") >> + msconf.set("General", "unpack", "true") >> + msconf.set("General", "ignorenativearch", "true") >> + msconf.set("General", "configscript", "./" + >> + os.path.join(workdir_rel, configscript)) >> + msconf.set("General", "setupscript", "./" + >> + os.path.join(workdir_rel, setupscript)) >> + msconf.set("General", "hookdir", "./" + >> + os.path.join(workdir_rel, hookdir)) >> + msconf.set("General", "aptsources", " ".join(aptsources)) >> + msconf.set("General", "bootstrap", " ".join(bootstrap)) >> + >> + # Setup upstream repositories in multistrap configuration: >> + apt_source = d.getVar("DISTRO_APT_SOURCE", True) or "" >> + apt_source_sec = d.getVar("DISTRO_APT_SOURCE_SEC", True) or "" >> + apt_suite = d.getVar("DISTRO_SUITE", True) or "" >> + apt_components = (d.getVar("DISTRO_COMPONENTS", True) or "").split() >> + deb_packages = (d.getVar("MULTISTRAP_PREINSTALL", True) or "").split() >> + isar_source = d.getVar("MULTISTRAP_ISAR_SOURCE", True) or "" >> + isar_packages = (d.getVar("MULTISTRAP_ISAR_INSTALL", True) or "").split() >> + isar_suite = d.getVar("MULTISTRAP_ISAR_SUITE", True) or "isar" >> + >> + msconf.add_section("base") >> + msconf.set("base", "source", apt_source) >> + msconf.set("base", "suite", apt_suite) >> + msconf.set("base", "components", " ".join(apt_components)) >> + msconf.set("base", "omitdebsrc", "true") >> + msconf.set("base", "packages", " ".join(deb_packages)) >> + >> + msconf.add_section("updates") >> + msconf.set("updates", "source", apt_source) >> + msconf.set("updates", "suite", apt_suite + "-updates") >> + msconf.set("updates", "components", " ".join(apt_components)) >> + msconf.set("updates", "omitdebsrc", "true") >> + >> + msconf.add_section("security") >> + msconf.set("security", "source", apt_source_sec) >> + msconf.set("security", "suite", apt_suite + "/updates") >> + msconf.set("security", "components", " ".join(apt_components)) >> + msconf.set("security", "omitdebsrc", "true") >> + >> + msconf.add_section("isar") >> + msconf.set("isar", "source", isar_source) >> + msconf.set("isar", "suite", isar_suite) >> + msconf.set("isar", "components", "main") >> + msconf.set("isar", "omitdebsrc", "true") >> + if isar_packages : >> + msconf.set("isar", "packages", >> + " ".join(isar_packages)) >> + >> + ret = "" >> + with io.StringIO() as out: >> + msconf.write(out) >> + ret = out.getvalue() >> + return ret >> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb >> index 570c0ad..59b4e7e 100644 >> --- a/meta/recipes-devtools/buildchroot/buildchroot.bb >> +++ b/meta/recipes-devtools/buildchroot/buildchroot.bb >> @@ -9,8 +9,7 @@ LICENSE = "gpl-2.0" >> LIC_FILES_CHKSUM = "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe" >> >> FILESPATH =. "${LAYERDIR_core}/recipes-devtools/buildchroot/files:" >> -SRC_URI = "file://multistrap.conf.in \ >> - file://configscript.sh \ >> +SRC_URI = "file://configscript.sh \ >> file://setup.sh \ >> file://build.sh" >> PV = "1.0" >> @@ -37,32 +36,25 @@ do_build[dirs] = "${BUILDCHROOT_DIR}/isar-apt \ >> ${BUILDCHROOT_DIR}/sys" >> do_build[depends] = "isar-apt:do_cache_config" >> >> +MULTISTRAP_CONFIG_SCRIPT = "configscript.sh" >> +MULTISTRAP_APTSOURCES = "isar ${DISTRO_MULTICONF_APTSOURCES}" >> +MULTISTRAP_BOOTSTRAP = "${DISTRO_MULTICONF_BOOTSTRAP}" >> +MULTISTRAP_PREINSTALL = "${BUILDCHROOT_PREINSTALL}" >> +MULTISTRAP_ISAR_SOURCE = "file:///isar-apt" >> + >> do_build() { >> E="${@ bb.utils.export_proxies(d)}" >> >> chmod +x "${WORKDIR}/setup.sh" >> chmod +x "${WORKDIR}/configscript.sh" >> >> - # Multistrap accepts only relative path in configuration files, so get it: >> - cd ${TOPDIR} >> - WORKDIR_REL=${@ os.path.relpath(d.getVar("WORKDIR", True))} >> - >> - # Adjust multistrap config >> - sed -e 's|##BUILDCHROOT_PREINSTALL##|${BUILDCHROOT_PREINSTALL}|g' \ >> - -e 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \ >> - -e 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \ >> - -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \ >> - -e 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \ >> - -e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \ >> - -e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \ >> - -e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/configscript.sh|g' \ >> - -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \ >> - -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \ >> - "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf" >> + echo "${@ isar.multistrap.generate_multistrap_config(d) }" > \ >> + "${WORKDIR}/multistrap.conf" >> >> do_setup_mounts >> >> # Create root filesystem >> + cd "${TOPDIR}" >> sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f "${WORKDIR}/multistrap.conf" >> >> # Install package builder script >> diff --git a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in >> deleted file mode 100644 >> index 480a4b8..0000000 >> --- a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in >> +++ /dev/null >> @@ -1,37 +0,0 @@ >> -# This software is a part of ISAR. >> -# Copyright (C) 2015-2017 ilbers GmbH >> - >> -[General] >> -noauth=true >> -unpack=true >> -ignorenativearch=true >> -bootstrap=##DISTRO_MULTICONF_BOOTSTRAP## >> -aptsources=isar-apt ##DISTRO_MULTICONF_APTSOURCES## >> -configscript=##CONFIG_SCRIPT## >> -setupscript=##SETUP_SCRIPT## >> -hookdir=##DIR_HOOKS## >> - >> -[base] >> -source=##DISTRO_APT_SOURCE## >> -suite=##DISTRO_SUITE## >> -components=##DISTRO_COMPONENTS## >> -packages=##BUILDCHROOT_PREINSTALL## >> -omitdebsrc=true >> - >> -[updates] >> -source=##DISTRO_APT_SOURCE## >> -suite=##DISTRO_SUITE##-updates >> -components=##DISTRO_COMPONENTS## >> -omitdebsrc=true >> - >> -[security] >> -source=##DISTRO_APT_SOURCE_SEC## >> -suite=##DISTRO_SUITE##/updates >> -components=##DISTRO_COMPONENTS## >> -omitdebsrc=true >> - >> -[isar-apt] >> -source=file:///isar-apt >> -suite=isar >> -components=main >> -omitdebsrc=true >> > > This requires rebasing. I quickly tried, but the result broke the > package build (didn't analyze any details). > > > While this moves us in the right direction, it still leaves some > duplications behind (setup.sh) - and the big question if we should > invest any further in this horrible multistrap mess. > > I've just enabled arm64, and it took a while to figure out that I was > missing some lines in setup.sh. In contrast, qemu-debootstrap just > worked (I used it to debug the multistrap output). > > So, before we move on here: What all does multistrap currently give us, > compared to standard debootstrap? I only see "multiple repositories" so > far. But as Claudius is going to massage that part heavily anyway, I > really wonder if it's worth building on the multistrap indirection any > further. > > How about something like this? > > - [qemu-]debootstrap --variant=minbase ... > - inject all additional apt sources, including user-provided ones > - set pinnings and priority, including user-provided ones > - apt-get dist-upgrade > I'd say that multistrap is a tool which maps user-friendly config file to specific apt repositories configuration. If you look in the do_build() log for buildchroot, you could see what it exactly does: 1. Creating list of sources in rootfs/etc 2. Get packages.gz for all the sources: apt-get -o Apt::Architecture=armhf -o Dir::Etc::TrustedParts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/trusted.gpg.d -o Dir::Etc::Trusted=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/trusted.gpg.d/trusted.gpg -o Apt::Get::AllowUnauthenticated=true -o Apt::Get::Download-Only=true -o Apt::Install-Recommends=false -o Dir=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/ -o Dir::Etc=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/ -o Dir::Etc::Parts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/apt.conf.d/ -o Dir::Etc::PreferencesParts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/preferences.d/ -o APT::Default-Release=* -o Dir::State=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/lib/apt/ -o Dir::State::Status=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/lib/dpkg/status -o Dir::Cache=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/cache/apt/ update 3. Install packages: apt-get -y -o Apt::Architecture=armhf -o Dir::Etc::TrustedParts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/trusted.gpg.d -o Dir::Etc::Trusted=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/trusted.gpg.d/trusted.gpg -o Apt::Get::AllowUnauthenticated=true -o Apt::Get::Download-Only=true -o Apt::Install-Recommends=false -o Dir=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/ -o Dir::Etc=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/ -o Dir::Etc::Parts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/apt.conf.d/ -o Dir::Etc::PreferencesParts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/preferences.d/ -o APT::Default-Release=* -o Dir::State=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/lib/apt/ -o Dir::State::Status=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/lib/dpkg/status -o Dir::Cache=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/cache/apt/ install ${LIST_OF_PACKAGES} > That would kick out all the multistrap complications. For me the plan above intersects with these multistrap steps, you have to re-implement them from the scratch. In this case I'd vote for the community-proved solution. Alex ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 2/2] centralize multistrap configuration generation 2018-02-15 9:34 ` Alexander Smirnov @ 2018-02-15 10:05 ` Claudius Heine 2018-02-15 10:15 ` Jan Kiszka 0 siblings, 1 reply; 23+ messages in thread From: Claudius Heine @ 2018-02-15 10:05 UTC (permalink / raw) To: Alexander Smirnov, Jan Kiszka, isar-users, Baurzhan Ismagulov Cc: Claudius Heine Hi Alex, On 02/15/2018 10:34 AM, Alexander Smirnov wrote: > > > On 02/15/2018 10:28 AM, Jan Kiszka wrote: >> On 2018-02-14 14:15, [ext] claudius.heine.ext@siemens.com wrote: >>> From: Claudius Heine <ch@denx.de> >>> >>> Previous there was redundant code between the buildchroot and >>> isar-image-base recipe. >>> >>> This patch moves the generation of the multistrap configuration to a >>> central python script that can be parametrized by both recipes. >>> >>> Signed-off-by: Claudius Heine <ch@denx.de> >>> --- >>> .../recipes-core/images/files/multistrap.conf.in | 38 ---------- >>> meta-isar/recipes-core/images/isar-image-base.bb | 33 ++++----- >>> meta/classes/base.bbclass | 2 +- >>> meta/lib/isar/__init__.py | 8 +++ >>> meta/lib/isar/multistrap.py | 80 >>> ++++++++++++++++++++++ >>> meta/recipes-devtools/buildchroot/buildchroot.bb | 28 +++----- >>> .../buildchroot/files/multistrap.conf.in | 37 ---------- >>> 7 files changed, 111 insertions(+), 115 deletions(-) >>> delete mode 100644 >>> meta-isar/recipes-core/images/files/multistrap.conf.in >>> create mode 100644 meta/lib/isar/__init__.py >>> create mode 100644 meta/lib/isar/multistrap.py >>> delete mode 100644 >>> meta/recipes-devtools/buildchroot/files/multistrap.conf.in >>> >>> diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in >>> b/meta-isar/recipes-core/images/files/multistrap.conf.in >>> deleted file mode 100644 >>> index 432b6af..0000000 >>> --- a/meta-isar/recipes-core/images/files/multistrap.conf.in >>> +++ /dev/null >>> @@ -1,38 +0,0 @@ >>> -# This software is a part of ISAR. >>> -# Copyright (C) 2015-2017 ilbers GmbH >>> - >>> -[General] >>> -noauth=true >>> -unpack=true >>> -ignorenativearch=true >>> -bootstrap=##DISTRO_MULTICONF_BOOTSTRAP## Isar >>> -aptsources=##DISTRO_MULTICONF_APTSOURCES## >>> -configscript=##CONFIG_SCRIPT## >>> -setupscript=##SETUP_SCRIPT## >>> -hookdir=##DIR_HOOKS## >>> - >>> -[base] >>> -source=##DISTRO_APT_SOURCE## >>> -suite=##DISTRO_SUITE## >>> -components=##DISTRO_COMPONENTS## >>> -packages=##IMAGE_PREINSTALL## >>> -omitdebsrc=true >>> - >>> -[updates] >>> -source=##DISTRO_APT_SOURCE## >>> -suite=##DISTRO_SUITE##-updates >>> -components=##DISTRO_COMPONENTS## >>> -omitdebsrc=true >>> - >>> -[security] >>> -source=##DISTRO_APT_SOURCE_SEC## >>> -suite=##DISTRO_SUITE##/updates >>> -components=##DISTRO_COMPONENTS## >>> -omitdebsrc=true >>> - >>> -[Isar] >>> -packages=##IMAGE_INSTALL## >>> -source=##DEPLOY_DIR_APT## >>> -suite=##ISAR_DISTRO_SUITE## >>> -components=main >>> -omitdebsrc=true >>> diff --git a/meta-isar/recipes-core/images/isar-image-base.bb >>> b/meta-isar/recipes-core/images/isar-image-base.bb >>> index 8ddbabb..9913977 100644 >>> --- a/meta-isar/recipes-core/images/isar-image-base.bb >>> +++ b/meta-isar/recipes-core/images/isar-image-base.bb >>> @@ -9,8 +9,7 @@ LICENSE = "gpl-2.0" >>> LIC_FILES_CHKSUM = >>> "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe" >>> >>> FILESPATH =. "${LAYERDIR_isar}/recipes-core/images/files:" >>> -SRC_URI = "file://multistrap.conf.in \ >>> - file://${DISTRO_CONFIG_SCRIPT} \ >>> +SRC_URI = "file://${DISTRO_CONFIG_SCRIPT} \ >>> file://setup.sh \ >>> file://download_dev-random" >>> @@ -28,6 +27,14 @@ WORKDIR = >>> "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}" >>> do_rootfs[dirs] = "${WORKDIR}/hooks_multistrap" >>> +MULTISTRAP_CONFIG_SCRIPT = "${DISTRO_CONFIG_SCRIPT}" >>> +MULTISTRAP_APTSOURCES = "${DISTRO_MULTICONF_APTSOURCES}" >>> +MULTISTRAP_BOOTSTRAP = "${DISTRO_MULTICONF_BOOTSTRAP} isar" >>> +MULTISTRAP_PREINSTALL = "${IMAGE_PREINSTALL}" >>> +MULTISTRAP_ISAR_SOURCE = "copy:///${DEPLOY_DIR_APT}/${DISTRO}" >>> +MULTISTRAP_ISAR_INSTALL = "${IMAGE_INSTALL}" >>> +MULTISTRAP_ISAR_SUITE = "${DEBDISTRONAME}" >>> + >>> do_rootfs() { >>> E="${@ bb.utils.export_proxies(d)}" >>> @@ -35,25 +42,8 @@ do_rootfs() { >>> chmod +x "${WORKDIR}/setup.sh" >>> install -m 755 "${WORKDIR}/download_dev-random" >>> "${WORKDIR}/hooks_multistrap/" >>> - # Multistrap accepts only relative path in configuration files, >>> so get it: >>> - cd ${TOPDIR} >>> - WORKDIR_REL=${@ os.path.relpath(d.getVar("WORKDIR", True))} >>> - >>> - # Adjust multistrap config >>> - sed -e 's|##IMAGE_PREINSTALL##|${IMAGE_PREINSTALL}|g' \ >>> - -e >>> 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \ >>> - -e >>> 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \ >>> - -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \ >>> - -e 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \ >>> - -e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \ >>> - -e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \ >>> - -e >>> 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/${DISTRO_CONFIG_SCRIPT}|g' \ >>> - -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \ >>> - -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \ >>> - -e 's|##IMAGE_INSTALL##|${IMAGE_INSTALL}|g' \ >>> - -e >>> 's|##DEPLOY_DIR_APT##|copy:///${DEPLOY_DIR_APT}/${DISTRO}|g' \ >>> - -e 's|##ISAR_DISTRO_SUITE##|${DEBDISTRONAME}|g' \ >>> - "${WORKDIR}/multistrap.conf.in" > >>> "${WORKDIR}/multistrap.conf" >>> + echo "${@ isar.multistrap.generate_multistrap_config(d) }" > \ >>> + "${WORKDIR}/multistrap.conf" >>> # Do not use bitbake flag [dirs] here because this folder >>> should have >>> # specific ownership. >>> @@ -62,6 +52,7 @@ do_rootfs() { >>> # Create root filesystem. We must use sudo -E here to preserve >>> the environment >>> # because of proxy settings >>> + cd "${TOPDIR}" >>> sudo -E multistrap -a ${DISTRO_ARCH} -d "${IMAGE_ROOTFS}" -f >>> "${WORKDIR}/multistrap.conf" >>> # Configure root filesystem >>> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass >>> index 3334789..2a45a1e 100644 >>> --- a/meta/classes/base.bbclass >>> +++ b/meta/classes/base.bbclass >>> @@ -38,7 +38,7 @@ bbfatal() { >>> } >>> # Derived from openembedded-core/meta/classes/base.bbclass >>> -ISAR_IMPORTS += "os sys time" >>> +ISAR_IMPORTS += "os sys time isar" >>> def isar_imports(d): >>> import sys >>> diff --git a/meta/lib/isar/__init__.py b/meta/lib/isar/__init__.py >>> new file mode 100644 >>> index 0000000..e6aeeaa >>> --- /dev/null >>> +++ b/meta/lib/isar/__init__.py >>> @@ -0,0 +1,8 @@ >>> +# Custom kernel recipe include >>> +# >>> +# This software is a part of ISAR. >>> +# Copyright (c) Siemens AG, 2018 >>> +# >>> +# SPDX-License-Identifier: MIT >>> + >>> +from . import multistrap >>> diff --git a/meta/lib/isar/multistrap.py b/meta/lib/isar/multistrap.py >>> new file mode 100644 >>> index 0000000..7022f91 >>> --- /dev/null >>> +++ b/meta/lib/isar/multistrap.py >>> @@ -0,0 +1,80 @@ >>> +# Custom kernel recipe include >>> +# >>> +# This software is a part of ISAR. >>> +# Copyright (c) Siemens AG, 2018 >>> +# >>> +# SPDX-License-Identifier: MIT >>> + >>> +def generate_multistrap_config(d): >>> + import configparser as cp >>> + import io >>> + >>> + workdir = d.getVar("WORKDIR", True) or "" >>> + topdir = d.getVar("TOPDIR", True) or "" >>> + workdir_rel = os.path.relpath(workdir, start=topdir) >>> + >>> + msconf = cp.RawConfigParser() >>> + >>> + # Set general multistrap configuration: >>> + configscript = d.getVar("MULTISTRAP_CONFIG_SCRIPT", True) or "" >>> + setupscript = d.getVar("MULTISTRAP_SETUP_SCRIPT", True) or >>> "setup.sh" >>> + hookdir = d.getVar("MULTISTRAP_HOOK_DIR", True) or >>> "hooks_multistrap" >>> + aptsources = (d.getVar("MULTISTRAP_APTSOURCES", True) or >>> "").split() >>> + bootstrap = (d.getVar("MULTISTRAP_BOOTSTRAP", True) or "").split() >>> + >>> + msconf.add_section("General") >>> + msconf.set("General", "noauth", "true") >>> + msconf.set("General", "unpack", "true") >>> + msconf.set("General", "ignorenativearch", "true") >>> + msconf.set("General", "configscript", "./" + >>> + os.path.join(workdir_rel, configscript)) >>> + msconf.set("General", "setupscript", "./" + >>> + os.path.join(workdir_rel, setupscript)) >>> + msconf.set("General", "hookdir", "./" + >>> + os.path.join(workdir_rel, hookdir)) >>> + msconf.set("General", "aptsources", " ".join(aptsources)) >>> + msconf.set("General", "bootstrap", " ".join(bootstrap)) >>> + >>> + # Setup upstream repositories in multistrap configuration: >>> + apt_source = d.getVar("DISTRO_APT_SOURCE", True) or "" >>> + apt_source_sec = d.getVar("DISTRO_APT_SOURCE_SEC", True) or "" >>> + apt_suite = d.getVar("DISTRO_SUITE", True) or "" >>> + apt_components = (d.getVar("DISTRO_COMPONENTS", True) or >>> "").split() >>> + deb_packages = (d.getVar("MULTISTRAP_PREINSTALL", True) or >>> "").split() >>> + isar_source = d.getVar("MULTISTRAP_ISAR_SOURCE", True) or "" >>> + isar_packages = (d.getVar("MULTISTRAP_ISAR_INSTALL", True) or >>> "").split() >>> + isar_suite = d.getVar("MULTISTRAP_ISAR_SUITE", True) or "isar" >>> + >>> + msconf.add_section("base") >>> + msconf.set("base", "source", apt_source) >>> + msconf.set("base", "suite", apt_suite) >>> + msconf.set("base", "components", " ".join(apt_components)) >>> + msconf.set("base", "omitdebsrc", "true") >>> + msconf.set("base", "packages", " ".join(deb_packages)) >>> + >>> + msconf.add_section("updates") >>> + msconf.set("updates", "source", apt_source) >>> + msconf.set("updates", "suite", apt_suite + "-updates") >>> + msconf.set("updates", "components", " ".join(apt_components)) >>> + msconf.set("updates", "omitdebsrc", "true") >>> + >>> + msconf.add_section("security") >>> + msconf.set("security", "source", apt_source_sec) >>> + msconf.set("security", "suite", apt_suite + "/updates") >>> + msconf.set("security", "components", " ".join(apt_components)) >>> + msconf.set("security", "omitdebsrc", "true") >>> + >>> + msconf.add_section("isar") >>> + msconf.set("isar", "source", isar_source) >>> + msconf.set("isar", "suite", isar_suite) >>> + msconf.set("isar", "components", "main") >>> + msconf.set("isar", "omitdebsrc", "true") >>> + if isar_packages : >>> + msconf.set("isar", "packages", >>> + " ".join(isar_packages)) >>> + >>> + ret = "" >>> + with io.StringIO() as out: >>> + msconf.write(out) >>> + ret = out.getvalue() >>> + return ret >>> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb >>> b/meta/recipes-devtools/buildchroot/buildchroot.bb >>> index 570c0ad..59b4e7e 100644 >>> --- a/meta/recipes-devtools/buildchroot/buildchroot.bb >>> +++ b/meta/recipes-devtools/buildchroot/buildchroot.bb >>> @@ -9,8 +9,7 @@ LICENSE = "gpl-2.0" >>> LIC_FILES_CHKSUM = >>> "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe" >>> >>> FILESPATH =. "${LAYERDIR_core}/recipes-devtools/buildchroot/files:" >>> -SRC_URI = "file://multistrap.conf.in \ >>> - file://configscript.sh \ >>> +SRC_URI = "file://configscript.sh \ >>> file://setup.sh \ >>> file://build.sh" >>> PV = "1.0" >>> @@ -37,32 +36,25 @@ do_build[dirs] = "${BUILDCHROOT_DIR}/isar-apt \ >>> ${BUILDCHROOT_DIR}/sys" >>> do_build[depends] = "isar-apt:do_cache_config" >>> +MULTISTRAP_CONFIG_SCRIPT = "configscript.sh" >>> +MULTISTRAP_APTSOURCES = "isar ${DISTRO_MULTICONF_APTSOURCES}" >>> +MULTISTRAP_BOOTSTRAP = "${DISTRO_MULTICONF_BOOTSTRAP}" >>> +MULTISTRAP_PREINSTALL = "${BUILDCHROOT_PREINSTALL}" >>> +MULTISTRAP_ISAR_SOURCE = "file:///isar-apt" >>> + >>> do_build() { >>> E="${@ bb.utils.export_proxies(d)}" >>> chmod +x "${WORKDIR}/setup.sh" >>> chmod +x "${WORKDIR}/configscript.sh" >>> - # Multistrap accepts only relative path in configuration files, >>> so get it: >>> - cd ${TOPDIR} >>> - WORKDIR_REL=${@ os.path.relpath(d.getVar("WORKDIR", True))} >>> - >>> - # Adjust multistrap config >>> - sed -e 's|##BUILDCHROOT_PREINSTALL##|${BUILDCHROOT_PREINSTALL}|g' \ >>> - -e >>> 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \ >>> - -e >>> 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \ >>> - -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \ >>> - -e 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \ >>> - -e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \ >>> - -e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \ >>> - -e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/configscript.sh|g' \ >>> - -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \ >>> - -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \ >>> - "${WORKDIR}/multistrap.conf.in" > >>> "${WORKDIR}/multistrap.conf" >>> + echo "${@ isar.multistrap.generate_multistrap_config(d) }" > \ >>> + "${WORKDIR}/multistrap.conf" >>> do_setup_mounts >>> # Create root filesystem >>> + cd "${TOPDIR}" >>> sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f >>> "${WORKDIR}/multistrap.conf" >>> # Install package builder script >>> diff --git >>> a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in >>> b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in >>> deleted file mode 100644 >>> index 480a4b8..0000000 >>> --- a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in >>> +++ /dev/null >>> @@ -1,37 +0,0 @@ >>> -# This software is a part of ISAR. >>> -# Copyright (C) 2015-2017 ilbers GmbH >>> - >>> -[General] >>> -noauth=true >>> -unpack=true >>> -ignorenativearch=true >>> -bootstrap=##DISTRO_MULTICONF_BOOTSTRAP## >>> -aptsources=isar-apt ##DISTRO_MULTICONF_APTSOURCES## >>> -configscript=##CONFIG_SCRIPT## >>> -setupscript=##SETUP_SCRIPT## >>> -hookdir=##DIR_HOOKS## >>> - >>> -[base] >>> -source=##DISTRO_APT_SOURCE## >>> -suite=##DISTRO_SUITE## >>> -components=##DISTRO_COMPONENTS## >>> -packages=##BUILDCHROOT_PREINSTALL## >>> -omitdebsrc=true >>> - >>> -[updates] >>> -source=##DISTRO_APT_SOURCE## >>> -suite=##DISTRO_SUITE##-updates >>> -components=##DISTRO_COMPONENTS## >>> -omitdebsrc=true >>> - >>> -[security] >>> -source=##DISTRO_APT_SOURCE_SEC## >>> -suite=##DISTRO_SUITE##/updates >>> -components=##DISTRO_COMPONENTS## >>> -omitdebsrc=true >>> - >>> -[isar-apt] >>> -source=file:///isar-apt >>> -suite=isar >>> -components=main >>> -omitdebsrc=true >>> >> >> This requires rebasing. I quickly tried, but the result broke the >> package build (didn't analyze any details). >> >> >> While this moves us in the right direction, it still leaves some >> duplications behind (setup.sh) - and the big question if we should >> invest any further in this horrible multistrap mess. >> >> I've just enabled arm64, and it took a while to figure out that I was >> missing some lines in setup.sh. In contrast, qemu-debootstrap just >> worked (I used it to debug the multistrap output). >> >> So, before we move on here: What all does multistrap currently give us, >> compared to standard debootstrap? I only see "multiple repositories" so >> far. But as Claudius is going to massage that part heavily anyway, I >> really wonder if it's worth building on the multistrap indirection any >> further. >> >> How about something like this? >> >> - [qemu-]debootstrap --variant=minbase ... >> - inject all additional apt sources, including user-provided ones >> - set pinnings and priority, including user-provided ones >> - apt-get dist-upgrade >> > > I'd say that multistrap is a tool which maps user-friendly config file > to specific apt repositories configuration. If you look in the > do_build() log for buildchroot, you could see what it exactly does: > > 1. Creating list of sources in rootfs/etc > > 2. Get packages.gz for all the sources: > > apt-get -o Apt::Architecture=armhf -o > Dir::Etc::TrustedParts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/trusted.gpg.d > -o > Dir::Etc::Trusted=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/trusted.gpg.d/trusted.gpg > -o Apt::Get::AllowUnauthenticated=true -o Apt::Get::Download-Only=true > -o Apt::Install-Recommends=false -o > Dir=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/ > -o > Dir::Etc=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/ > -o > Dir::Etc::Parts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/apt.conf.d/ > -o > Dir::Etc::PreferencesParts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/preferences.d/ > -o APT::Default-Release=* -o > Dir::State=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/lib/apt/ > -o > Dir::State::Status=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/lib/dpkg/status > -o > Dir::Cache=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/cache/apt/ > update > > 3. Install packages: > > apt-get -y -o Apt::Architecture=armhf -o > Dir::Etc::TrustedParts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/trusted.gpg.d > -o > Dir::Etc::Trusted=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/trusted.gpg.d/trusted.gpg > -o Apt::Get::AllowUnauthenticated=true -o Apt::Get::Download-Only=true > -o Apt::Install-Recommends=false -o > Dir=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/ > -o > Dir::Etc=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/ > -o > Dir::Etc::Parts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/apt.conf.d/ > -o > Dir::Etc::PreferencesParts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/preferences.d/ > -o APT::Default-Release=* -o > Dir::State=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/lib/apt/ > -o > Dir::State::Status=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/lib/dpkg/status > -o > Dir::Cache=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/cache/apt/ > install ${LIST_OF_PACKAGES} > > > That would kick out all the multistrap complications. > > For me the plan above intersects with these multistrap steps, you have > to re-implement them from the scratch. In this case I'd vote for the > community-proved solution. For me that is a reason to use debootstrap instead of multistrap, since multistrap is not community-supported. If we implement some of those feature, we might as well do it in debootstrap inself. Just take a look at their TODO list: Features: ++ second stage via chroot debootstrap/debootstrap ++ debootstrap/deb file to record deb destinations/information -- configuration file -- versus command line -- support for sources (vs mirrors) -- faux-pinning for packages https://anonscm.debian.org/cgit/d-i/debootstrap.git/tree/TODO?id=5d42162bc0fe266b280d81445271e3f56b9560a8 That shows to me that upstream might be interested in some of these improvements we would need to implement. Cheers, Claudius -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 2/2] centralize multistrap configuration generation 2018-02-15 10:05 ` Claudius Heine @ 2018-02-15 10:15 ` Jan Kiszka 2018-02-15 10:29 ` Alexander Smirnov 2018-02-15 11:27 ` Baurzhan Ismagulov 0 siblings, 2 replies; 23+ messages in thread From: Jan Kiszka @ 2018-02-15 10:15 UTC (permalink / raw) To: Claudius Heine, Alexander Smirnov, isar-users, Baurzhan Ismagulov Cc: Claudius Heine On 2018-02-15 11:05, Claudius Heine wrote: >>> While this moves us in the right direction, it still leaves some >>> duplications behind (setup.sh) - and the big question if we should >>> invest any further in this horrible multistrap mess. >>> >>> I've just enabled arm64, and it took a while to figure out that I was >>> missing some lines in setup.sh. In contrast, qemu-debootstrap just >>> worked (I used it to debug the multistrap output). >>> >>> So, before we move on here: What all does multistrap currently give us, >>> compared to standard debootstrap? I only see "multiple repositories" so >>> far. But as Claudius is going to massage that part heavily anyway, I >>> really wonder if it's worth building on the multistrap indirection any >>> further. >>> >>> How about something like this? >>> >>> - [qemu-]debootstrap --variant=minbase ... >>> - inject all additional apt sources, including user-provided ones >>> - set pinnings and priority, including user-provided ones >>> - apt-get dist-upgrade >>> >> >> I'd say that multistrap is a tool which maps user-friendly config file >> to specific apt repositories configuration. If you look in the >> do_build() log for buildchroot, you could see what it exactly does: >> >> 1. Creating list of sources in rootfs/etc >> >> 2. Get packages.gz for all the sources: >> >> apt-get -o Apt::Architecture=armhf -o >> Dir::Etc::TrustedParts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/trusted.gpg.d >> -o >> Dir::Etc::Trusted=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/trusted.gpg.d/trusted.gpg >> -o Apt::Get::AllowUnauthenticated=true -o Apt::Get::Download-Only=true >> -o Apt::Install-Recommends=false -o >> Dir=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/ >> -o >> Dir::Etc=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/ >> -o >> Dir::Etc::Parts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/apt.conf.d/ >> -o >> Dir::Etc::PreferencesParts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/preferences.d/ >> -o APT::Default-Release=* -o >> Dir::State=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/lib/apt/ >> -o >> Dir::State::Status=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/lib/dpkg/status >> -o >> Dir::Cache=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/cache/apt/ >> update >> >> 3. Install packages: >> >> apt-get -y -o Apt::Architecture=armhf -o >> Dir::Etc::TrustedParts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/trusted.gpg.d >> -o >> Dir::Etc::Trusted=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/trusted.gpg.d/trusted.gpg >> -o Apt::Get::AllowUnauthenticated=true -o Apt::Get::Download-Only=true >> -o Apt::Install-Recommends=false -o >> Dir=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/ >> -o >> Dir::Etc=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/ >> -o >> Dir::Etc::Parts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/apt.conf.d/ >> -o >> Dir::Etc::PreferencesParts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/preferences.d/ >> -o APT::Default-Release=* -o >> Dir::State=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/lib/apt/ >> -o >> Dir::State::Status=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/lib/dpkg/status >> -o >> Dir::Cache=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/cache/apt/ >> install ${LIST_OF_PACKAGES} >> >> > That would kick out all the multistrap complications. >> >> For me the plan above intersects with these multistrap steps, you have >> to re-implement them from the scratch. In this case I'd vote for the >> community-proved solution. > > For me that is a reason to use debootstrap instead of multistrap, since > multistrap is not community-supported. If we implement some of those > feature, we might as well do it in debootstrap inself. Just take a look > at their TODO list: > > Features: > ++ second stage via chroot debootstrap/debootstrap > ++ debootstrap/deb file to record deb destinations/information > > -- configuration file > -- versus command line > -- support for sources (vs mirrors) > -- faux-pinning for packages > > https://anonscm.debian.org/cgit/d-i/debootstrap.git/tree/TODO?id=5d42162bc0fe266b280d81445271e3f56b9560a8 > > > That shows to me that upstream might be interested in some of these > improvements we would need to implement. Exactly. We should really use the community-proven, continuously tested official approach of Debian, in that in a way that it foresees for it. Let's bury multistrap. Its a dead horse we rode to long now. Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 2/2] centralize multistrap configuration generation 2018-02-15 10:15 ` Jan Kiszka @ 2018-02-15 10:29 ` Alexander Smirnov 2018-02-15 11:27 ` Baurzhan Ismagulov 1 sibling, 0 replies; 23+ messages in thread From: Alexander Smirnov @ 2018-02-15 10:29 UTC (permalink / raw) To: Jan Kiszka, isar-users, Baurzhan Ismagulov; +Cc: Claudius Heine On 02/15/2018 01:15 PM, Jan Kiszka wrote: > On 2018-02-15 11:05, Claudius Heine wrote: >>>> While this moves us in the right direction, it still leaves some >>>> duplications behind (setup.sh) - and the big question if we should >>>> invest any further in this horrible multistrap mess. >>>> >>>> I've just enabled arm64, and it took a while to figure out that I was >>>> missing some lines in setup.sh. In contrast, qemu-debootstrap just >>>> worked (I used it to debug the multistrap output). >>>> >>>> So, before we move on here: What all does multistrap currently give us, >>>> compared to standard debootstrap? I only see "multiple repositories" so >>>> far. But as Claudius is going to massage that part heavily anyway, I >>>> really wonder if it's worth building on the multistrap indirection any >>>> further. >>>> >>>> How about something like this? >>>> >>>> - [qemu-]debootstrap --variant=minbase ... >>>> - inject all additional apt sources, including user-provided ones >>>> - set pinnings and priority, including user-provided ones >>>> - apt-get dist-upgrade >>>> >>> >>> I'd say that multistrap is a tool which maps user-friendly config file >>> to specific apt repositories configuration. If you look in the >>> do_build() log for buildchroot, you could see what it exactly does: >>> >>> 1. Creating list of sources in rootfs/etc >>> >>> 2. Get packages.gz for all the sources: >>> >>> apt-get -o Apt::Architecture=armhf -o >>> Dir::Etc::TrustedParts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/trusted.gpg.d >>> -o >>> Dir::Etc::Trusted=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/trusted.gpg.d/trusted.gpg >>> -o Apt::Get::AllowUnauthenticated=true -o Apt::Get::Download-Only=true >>> -o Apt::Install-Recommends=false -o >>> Dir=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/ >>> -o >>> Dir::Etc=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/ >>> -o >>> Dir::Etc::Parts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/apt.conf.d/ >>> -o >>> Dir::Etc::PreferencesParts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/preferences.d/ >>> -o APT::Default-Release=* -o >>> Dir::State=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/lib/apt/ >>> -o >>> Dir::State::Status=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/lib/dpkg/status >>> -o >>> Dir::Cache=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/cache/apt/ >>> update >>> >>> 3. Install packages: >>> >>> apt-get -y -o Apt::Architecture=armhf -o >>> Dir::Etc::TrustedParts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/trusted.gpg.d >>> -o >>> Dir::Etc::Trusted=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/trusted.gpg.d/trusted.gpg >>> -o Apt::Get::AllowUnauthenticated=true -o Apt::Get::Download-Only=true >>> -o Apt::Install-Recommends=false -o >>> Dir=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/ >>> -o >>> Dir::Etc=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/ >>> -o >>> Dir::Etc::Parts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/apt.conf.d/ >>> -o >>> Dir::Etc::PreferencesParts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/preferences.d/ >>> -o APT::Default-Release=* -o >>> Dir::State=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/lib/apt/ >>> -o >>> Dir::State::Status=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/lib/dpkg/status >>> -o >>> Dir::Cache=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/cache/apt/ >>> install ${LIST_OF_PACKAGES} >>> >>> > That would kick out all the multistrap complications. >>> >>> For me the plan above intersects with these multistrap steps, you have >>> to re-implement them from the scratch. In this case I'd vote for the >>> community-proved solution. >> >> For me that is a reason to use debootstrap instead of multistrap, since >> multistrap is not community-supported. If we implement some of those >> feature, we might as well do it in debootstrap inself. Just take a look >> at their TODO list: >> >> Features: >> ++ second stage via chroot debootstrap/debootstrap >> ++ debootstrap/deb file to record deb destinations/information >> >> -- configuration file >> -- versus command line >> -- support for sources (vs mirrors) >> -- faux-pinning for packages >> >> https://anonscm.debian.org/cgit/d-i/debootstrap.git/tree/TODO?id=5d42162bc0fe266b280d81445271e3f56b9560a8 >> >> >> That shows to me that upstream might be interested in some of these >> improvements we would need to implement. > > Exactly. We should really use the community-proven, continuously tested > official approach of Debian, in that in a way that it foresees for it. > > Let's bury multistrap. Its a dead horse we rode to long now. > Nobody is against official tools. :-) This question is open for a quite long time, and the conclusion for now: there is *no* tool that fits Isar requirements for 100%. So in general I only want upstream-first way of tools modification for Isar. Alex Alex ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 2/2] centralize multistrap configuration generation 2018-02-15 10:15 ` Jan Kiszka 2018-02-15 10:29 ` Alexander Smirnov @ 2018-02-15 11:27 ` Baurzhan Ismagulov 2018-02-15 13:17 ` Claudius Heine 2018-02-15 13:18 ` Jan Kiszka 1 sibling, 2 replies; 23+ messages in thread From: Baurzhan Ismagulov @ 2018-02-15 11:27 UTC (permalink / raw) To: isar-users On Thu, Feb 15, 2018 at 11:15:02AM +0100, Jan Kiszka wrote: > > That shows to me that upstream might be interested in some of these > > improvements we would need to implement. > > Exactly. We should really use the community-proven, continuously tested > official approach of Debian, in that in a way that it foresees for it. > > Let's bury multistrap. Its a dead horse we rode to long now. First off, thanks for cutting the many lines of code not being referred to :) . Regarding multistrap, both approaches have advantages and disadvantages. Moreover, there are other implementations, such as cdebootstrap. Multistrap is currently useful in the following ways: 1. Generate the list of packages to install using several apt repos. 2. Provide the package paths to download from apt repos. 3. Generate the base system from several apt repos. Unfortunately, all three seem to be unique among the three candidates. (1) and (2) are necessary for reproducibility. My preferred solution would be to use an apt library that would do that for us (e.g., extract it from apt, extend python-debian, or look whether aptly provides that). Moving to *debootstrap would require solving those in a community-unmaintained way for some time. (3) is useful for people with customizations cutting off significant dependency chains from Debian (like SLIND did with Perl dependencies). Moving to *debootstrap would mean first installing those, then apt-getting our layers above the base system, then removing the packages that are no more required. This was the main reason for Neil Williams to create multistrap. After trying to wrap debootstrap for some time, he ended up with problems not solvable in that way (not sure which, maybe some kinds of circular dependencies or dependencies on exact package versions). So, it would be wise to consider his experience here. That said, on DebConf17, he told us that that use case is not interesting for him any more, advised to use debootstrap, and said that he doesn't support multistrap any more. Regarding maintaining multistrap for some time: I personally don't like Perl. That said, Alex looked at that recently; it's some reasonable wrapping around apt-get. If the problem in the end is solved with apt-get, we should possibly evaluate that, too. Regarding debootstrap, the problem is that it is implemented in shell, which is horrible for any serious improvements. Thus the question about cdebootstrap. All in all, I'd like to see a solution that would offer serious benefits while not introducing new problems. Currently, multistrap is an internal issue not directly affecting user-visible features. There are many features we can do for far better Debian support. Changing one bit with another, not better one wouldn't change the big picture. So, I'd rather use this opportunity to start a bigger effort of defining the requirements, choosing the tools to merge with, provide a working PoC, and start working with Debian on reusability of those tools in Isar. IMHO, *debootstrap alone wouldn't bring major architectural improvements. With kind regards, Baurzhan. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 2/2] centralize multistrap configuration generation 2018-02-15 11:27 ` Baurzhan Ismagulov @ 2018-02-15 13:17 ` Claudius Heine 2018-02-15 13:18 ` Jan Kiszka 1 sibling, 0 replies; 23+ messages in thread From: Claudius Heine @ 2018-02-15 13:17 UTC (permalink / raw) To: isar-users Hi Baurzhan, On 02/15/2018 12:27 PM, Baurzhan Ismagulov wrote: > On Thu, Feb 15, 2018 at 11:15:02AM +0100, Jan Kiszka wrote: >>> That shows to me that upstream might be interested in some of these >>> improvements we would need to implement. >> >> Exactly. We should really use the community-proven, continuously tested >> official approach of Debian, in that in a way that it foresees for it. >> >> Let's bury multistrap. Its a dead horse we rode to long now. > > First off, thanks for cutting the many lines of code not being referred to :) . > > Regarding multistrap, both approaches have advantages and disadvantages. > Moreover, there are other implementations, such as cdebootstrap. I am not sure about cdebootstrap, while I do like a C implementation more the one in shell scripts, I am skeptical about its maintenance and if they even accept contributions. Cdebootstrap last version is 0.7.7 from 2017-03-08 while debootstrap latest version 1.0.93 is from 2017-12-07. I could not find any repository containing the development history of cdebootstrap, while debootstraps repo is here [1]. cdebootstrap seems to be developed by the package maintainer himself while debootstrap is maintained and developed by multiple people. If we don't plan on developing our own debian bootstrap method, then trying to get changes pushed there seems to have a higher chance of success then with cdebootstrap. [1] https://anonscm.debian.org/cgit/d-i/debootstrap.git > > > Multistrap is currently useful in the following ways: > > 1. Generate the list of packages to install using several apt repos. > > 2. Provide the package paths to download from apt repos. > > 3. Generate the base system from several apt repos. > > Unfortunately, all three seem to be unique among the three candidates. > > > (1) and (2) are necessary for reproducibility. My preferred solution would be > to use an apt library that would do that for us (e.g., extract it from apt, > extend python-debian, or look whether aptly provides that). Moving to > *debootstrap would require solving those in a community-unmaintained way for > some time. > > (3) is useful for people with customizations cutting off significant dependency > chains from Debian (like SLIND did with Perl dependencies). Moving to > *debootstrap would mean first installing those, then apt-getting our layers > above the base system, then removing the packages that are no more required. IMO apt-preferences could help here. Prios >= 1000 would remove already installed packages. regards, Claudius > > This was the main reason for Neil Williams to create multistrap. After trying > to wrap debootstrap for some time, he ended up with problems not solvable in > that way (not sure which, maybe some kinds of circular dependencies or > dependencies on exact package versions). So, it would be wise to consider his > experience here. > > That said, on DebConf17, he told us that that use case is not interesting for > him any more, advised to use debootstrap, and said that he doesn't support > multistrap any more. > > Regarding maintaining multistrap for some time: I personally don't like Perl. > That said, Alex looked at that recently; it's some reasonable wrapping around > apt-get. If the problem in the end is solved with apt-get, we should possibly > evaluate that, too. > > > Regarding debootstrap, the problem is that it is implemented in shell, which is > horrible for any serious improvements. Thus the question about cdebootstrap. > > > All in all, I'd like to see a solution that would offer serious benefits while > not introducing new problems. Currently, multistrap is an internal issue not > directly affecting user-visible features. There are many features we can do for > far better Debian support. Changing one bit with another, not better one > wouldn't change the big picture. So, I'd rather use this opportunity to start a > bigger effort of defining the requirements, choosing the tools to merge with, > provide a working PoC, and start working with Debian on reusability of those > tools in Isar. IMHO, *debootstrap alone wouldn't bring major architectural > improvements. > > > With kind regards, > Baurzhan. > -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 2/2] centralize multistrap configuration generation 2018-02-15 11:27 ` Baurzhan Ismagulov 2018-02-15 13:17 ` Claudius Heine @ 2018-02-15 13:18 ` Jan Kiszka 1 sibling, 0 replies; 23+ messages in thread From: Jan Kiszka @ 2018-02-15 13:18 UTC (permalink / raw) To: isar-users On 2018-02-15 12:27, Baurzhan Ismagulov wrote: > On Thu, Feb 15, 2018 at 11:15:02AM +0100, Jan Kiszka wrote: >>> That shows to me that upstream might be interested in some of these >>> improvements we would need to implement. >> >> Exactly. We should really use the community-proven, continuously tested >> official approach of Debian, in that in a way that it foresees for it. >> >> Let's bury multistrap. Its a dead horse we rode to long now. > > First off, thanks for cutting the many lines of code not being referred to :) . > > Regarding multistrap, both approaches have advantages and disadvantages. > Moreover, there are other implementations, such as cdebootstrap. > > > Multistrap is currently useful in the following ways: > > 1. Generate the list of packages to install using several apt repos. > > 2. Provide the package paths to download from apt repos. > > 3. Generate the base system from several apt repos. > > Unfortunately, all three seem to be unique among the three candidates. > > > (1) and (2) are necessary for reproducibility. My preferred solution would be > to use an apt library that would do that for us (e.g., extract it from apt, > extend python-debian, or look whether aptly provides that). Moving to > *debootstrap would require solving those in a community-unmaintained way for > some time. > > (3) is useful for people with customizations cutting off significant dependency > chains from Debian (like SLIND did with Perl dependencies). Moving to > *debootstrap would mean first installing those, then apt-getting our layers > above the base system, then removing the packages that are no more required. > > This was the main reason for Neil Williams to create multistrap. After trying > to wrap debootstrap for some time, he ended up with problems not solvable in > that way (not sure which, maybe some kinds of circular dependencies or > dependencies on exact package versions). So, it would be wise to consider his > experience here. > > That said, on DebConf17, he told us that that use case is not interesting for > him any more, advised to use debootstrap, and said that he doesn't support > multistrap any more. > > Regarding maintaining multistrap for some time: I personally don't like Perl. > That said, Alex looked at that recently; it's some reasonable wrapping around > apt-get. If the problem in the end is solved with apt-get, we should possibly > evaluate that, too. > > > Regarding debootstrap, the problem is that it is implemented in shell, which is > horrible for any serious improvements. Thus the question about cdebootstrap. > > > All in all, I'd like to see a solution that would offer serious benefits while > not introducing new problems. Currently, multistrap is an internal issue not > directly affecting user-visible features. There are many features we can do for > far better Debian support. Changing one bit with another, not better one > wouldn't change the big picture. So, I'd rather use this opportunity to start a > bigger effort of defining the requirements, choosing the tools to merge with, > provide a working PoC, and start working with Debian on reusability of those > tools in Isar. IMHO, *debootstrap alone wouldn't bring major architectural > improvements. Let's wait for a first prototype. Maybe it will become clearer that we actually need, if debootstrap can do it, or if we are better of writing the core and best all the bootstarps in Isar recipes. Fact is (as I have been told): debootstarp is Debian standard, all others are secondary or even historic variants. So let's go standard first, and then see if we need more again. Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 2/2] centralize multistrap configuration generation 2018-02-14 13:15 ` [PATCH 2/2] centralize multistrap configuration generation claudius.heine.ext 2018-02-15 7:28 ` Jan Kiszka @ 2018-02-15 11:42 ` Baurzhan Ismagulov 2018-02-15 12:08 ` Claudius Heine 1 sibling, 1 reply; 23+ messages in thread From: Baurzhan Ismagulov @ 2018-02-15 11:42 UTC (permalink / raw) To: isar-users On Wed, Feb 14, 2018 at 02:15:09PM +0100, claudius.heine.ext@siemens.com wrote: > From: Claudius Heine <ch@denx.de> > > Previous there was redundant code between the buildchroot and > isar-image-base recipe. > > This patch moves the generation of the multistrap configuration to a > central python script that can be parametrized by both recipes. > > Signed-off-by: Claudius Heine <ch@denx.de> > --- > .../recipes-core/images/files/multistrap.conf.in | 38 ---------- > meta/lib/isar/multistrap.py | 80 ++++++++++++++++++++++ > .../buildchroot/files/multistrap.conf.in | 37 ---------- Could you please post two final multistrap.confs you'd like to get with your changes? I'd like to understand the reasons for moving from templates to a magic generator. In my experience, magic is usually poorly documented (if at all) and very ununderstandable for users. Simple change of a line requires locating the generator, understanding what it does (they tend to be hairy with tons of conditions) and patching the upstream [Isar] code. Templates have their disadvantages, but are straightforward for users. In general, I'd like to keep the current UX and stick with templates, which is possible most of the time. With kind regards, Baurzhan. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 2/2] centralize multistrap configuration generation 2018-02-15 11:42 ` Baurzhan Ismagulov @ 2018-02-15 12:08 ` Claudius Heine 2018-02-15 12:37 ` Alexander Smirnov 0 siblings, 1 reply; 23+ messages in thread From: Claudius Heine @ 2018-02-15 12:08 UTC (permalink / raw) To: isar-users, Baurzhan Ismagulov Hi Baurzhan, On 02/15/2018 12:42 PM, Baurzhan Ismagulov wrote: > On Wed, Feb 14, 2018 at 02:15:09PM +0100, claudius.heine.ext@siemens.com wrote: >> From: Claudius Heine <ch@denx.de> >> >> Previous there was redundant code between the buildchroot and >> isar-image-base recipe. >> >> This patch moves the generation of the multistrap configuration to a >> central python script that can be parametrized by both recipes. >> >> Signed-off-by: Claudius Heine <ch@denx.de> >> --- >> .../recipes-core/images/files/multistrap.conf.in | 38 ---------- >> meta/lib/isar/multistrap.py | 80 ++++++++++++++++++++++ >> .../buildchroot/files/multistrap.conf.in | 37 ---------- > > Could you please post two final multistrap.confs you'd like to get with your > changes? 'like to get'? I don't want to get any other multistrap.conf than before. This is not a change of the multistrap.conf files, but just how they are generated. The python configparser created ini-file format only puts spaces around the '=' characters, but the resulting deserialized objects is the same. Ok, that is not 100% true, I changed the name of the isar repository section. Now its called 'isar' in the image (previously 'Isar') and buildchroot (previously 'isar-apt') multiconfig. But that is it. It has no different effect outside of multistrap. > I'd like to understand the reasons for moving from templates to a magic > generator. > > In my experience, magic is usually poorly documented (if at all) and very > ununderstandable for users. Simple change of a line requires locating the > generator, understanding what it does (they tend to be hairy with tons of > conditions) and patching the upstream [Isar] code. Templates have their > disadvantages, but are straightforward for users. In general, I'd like to keep > the current UX and stick with templates, which is possible most of the time. You prefer duplicated code? Two slightly different templates for two buildchroot and image recipe? Also this patch is only an intermediate step of the way for multi-repo support. IMO it has value on its own, since it centralized the generation of the multistrap configuration. The current kind of 'template' does not support creating repository entries on the fly. It might be possible to extent the multiconfig with shell scripting, but I prefer using python for such advanced logic. regards, Claudius -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 2/2] centralize multistrap configuration generation 2018-02-15 12:08 ` Claudius Heine @ 2018-02-15 12:37 ` Alexander Smirnov 2018-02-15 13:20 ` Claudius Heine 0 siblings, 1 reply; 23+ messages in thread From: Alexander Smirnov @ 2018-02-15 12:37 UTC (permalink / raw) To: Claudius Heine, isar-users, Baurzhan Ismagulov Claudius Heine <claudius.heine.ext@siemens.com> 15 февраля 2018 г. 15:08:50 написал: > Hi Baurzhan, > > On 02/15/2018 12:42 PM, Baurzhan Ismagulov wrote: >> On Wed, Feb 14, 2018 at 02:15:09PM +0100, claudius.heine.ext@siemens.com wrote: >>> From: Claudius Heine <ch@denx.de> >>> >>> Previous there was redundant code between the buildchroot and >>> isar-image-base recipe. >>> >>> This patch moves the generation of the multistrap configuration to a >>> central python script that can be parametrized by both recipes. >>> >>> Signed-off-by: Claudius Heine <ch@denx.de> >>> --- >>> .../recipes-core/images/files/multistrap.conf.in | 38 ---------- >>> meta/lib/isar/multistrap.py | 80 ++++++++++++++++++++++ >>> .../buildchroot/files/multistrap.conf.in | 37 ---------- >> >> Could you please post two final multistrap.confs you'd like to get with your >> changes? > > 'like to get'? I don't want to get any other multistrap.conf than > before. This is not a change of the multistrap.conf files, but just how > they are generated. The python configparser created ini-file format only > puts spaces around the '=' characters, but the resulting deserialized > objects is the same. The difference that I see IMHO: now you need to feed multistrap by community-standardized input, you propose to add custom tool that eats proprietary input. > > Ok, that is not 100% true, I changed the name of the isar repository > section. Now its called 'isar' in the image (previously 'Isar') and > buildchroot (previously 'isar-apt') multiconfig. But that is it. It has > no different effect outside of multistrap. > >> I'd like to understand the reasons for moving from templates to a magic >> generator. >> >> In my experience, magic is usually poorly documented (if at all) and very >> ununderstandable for users. Simple change of a line requires locating the >> generator, understanding what it does (they tend to be hairy with tons of >> conditions) and patching the upstream [Isar] code. Templates have their >> disadvantages, but are straightforward for users. In general, I'd like to keep >> the current UX and stick with templates, which is possible most of the time. > > You prefer duplicated code? Two slightly different templates for two > buildchroot and image recipe? > AFAIK the key idea is to have the only single multistrap that works with upstream repos. Other multistraps fetch packages from local cache. So after this is done, the problem with duplicacations in configs will go. > Also this patch is only an intermediate step of the way for multi-repo > support. IMO it has value on its own, since it centralized the > generation of the multistrap configuration. > The current kind of 'template' does not support creating repository > entries on the fly. What is the use case to have them "on the fly"? Alex It might be possible to extent the multiconfig with > shell scripting, but I prefer using python for such advanced logic. I don't care about python or bash > > regards, > Claudius > > -- > DENX Software Engineering GmbH, Managing Director: Wolfgang Denk > HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany > Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de > > -- > You received this message because you are subscribed to the Google Groups > "isar-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to isar-users+unsubscribe@googlegroups.com. > To post to this group, send email to isar-users@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/isar-users/d58343b9-c284-81e7-f743-2672c1f5f8ef%40siemens.com. > For more options, visit https://groups.google.com/d/optout. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 2/2] centralize multistrap configuration generation 2018-02-15 12:37 ` Alexander Smirnov @ 2018-02-15 13:20 ` Claudius Heine 2018-02-15 13:39 ` Alexander Smirnov 2018-02-15 13:53 ` Jan Kiszka 0 siblings, 2 replies; 23+ messages in thread From: Claudius Heine @ 2018-02-15 13:20 UTC (permalink / raw) To: Alexander Smirnov, isar-users, Baurzhan Ismagulov On 02/15/2018 01:37 PM, Alexander Smirnov wrote: > > Claudius Heine <claudius.heine.ext@siemens.com> 15 февраля 2018 г. > 15:08:50 написал: > >> Hi Baurzhan, >> >> On 02/15/2018 12:42 PM, Baurzhan Ismagulov wrote: >>> On Wed, Feb 14, 2018 at 02:15:09PM +0100, >>> claudius.heine.ext@siemens.com wrote: >>>> From: Claudius Heine <ch@denx.de> >>>> >>>> Previous there was redundant code between the buildchroot and >>>> isar-image-base recipe. >>>> >>>> This patch moves the generation of the multistrap configuration to a >>>> central python script that can be parametrized by both recipes. >>>> >>>> Signed-off-by: Claudius Heine <ch@denx.de> >>>> --- >>>> .../recipes-core/images/files/multistrap.conf.in | 38 ---------- >>>> meta/lib/isar/multistrap.py | 80 >>>> ++++++++++++++++++++++ >>>> .../buildchroot/files/multistrap.conf.in | 37 ---------- >>> >>> Could you please post two final multistrap.confs you'd like to get >>> with your >>> changes? >> >> 'like to get'? I don't want to get any other multistrap.conf than >> before. This is not a change of the multistrap.conf files, but just how >> they are generated. The python configparser created ini-file format only >> puts spaces around the '=' characters, but the resulting deserialized >> objects is the same. > > The difference that I see IMHO: now you need to feed multistrap by > community-standardized input, you propose to add custom tool that eats > proprietary input. I don't get this point. What do you mean by that? > >> >> Ok, that is not 100% true, I changed the name of the isar repository >> section. Now its called 'isar' in the image (previously 'Isar') and >> buildchroot (previously 'isar-apt') multiconfig. But that is it. It has >> no different effect outside of multistrap. >> >>> I'd like to understand the reasons for moving from templates to a magic >>> generator. >>> >>> In my experience, magic is usually poorly documented (if at all) and >>> very >>> ununderstandable for users. Simple change of a line requires locating >>> the >>> generator, understanding what it does (they tend to be hairy with >>> tons of >>> conditions) and patching the upstream [Isar] code. Templates have their >>> disadvantages, but are straightforward for users. In general, I'd >>> like to keep >>> the current UX and stick with templates, which is possible most of >>> the time. >> >> You prefer duplicated code? Two slightly different templates for two >> buildchroot and image recipe? >> > > AFAIK the key idea is to have the only single multistrap that works with > upstream repos. Other multistraps fetch packages from local cache. So > after this is done, the problem with duplicacations in configs will go. > >> Also this patch is only an intermediate step of the way for multi-repo >> support. IMO it has value on its own, since it centralized the >> generation of the multistrap configuration. >> The current kind of 'template' does not support creating repository >> entries on the fly. > > What is the use case to have them "on the fly"? See: Also this patch is only an intermediate step of the way for multi-repo support. Use-case for multi-repo: docker-ce repository available. > > Alex > > It might be possible to extent the multiconfig with >> shell scripting, but I prefer using python for such advanced logic. > > I don't care about python or bash At least Baurzhan seems to care, because he doesn't like debootstrap because of it being in shell. Cheers, Claudius > >> >> regards, >> Claudius >> >> -- >> DENX Software Engineering GmbH, Managing Director: Wolfgang Denk >> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany >> Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de >> >> -- >> You received this message because you are subscribed to the Google >> Groups "isar-users" group. >> To unsubscribe from this group and stop receiving emails from it, send >> an email to isar-users+unsubscribe@googlegroups.com. >> To post to this group, send email to isar-users@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/isar-users/d58343b9-c284-81e7-f743-2672c1f5f8ef%40siemens.com. >> >> For more options, visit https://groups.google.com/d/optout. > > > -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 2/2] centralize multistrap configuration generation 2018-02-15 13:20 ` Claudius Heine @ 2018-02-15 13:39 ` Alexander Smirnov 2018-02-16 9:31 ` Claudius Heine 2018-02-15 13:53 ` Jan Kiszka 1 sibling, 1 reply; 23+ messages in thread From: Alexander Smirnov @ 2018-02-15 13:39 UTC (permalink / raw) To: Claudius Heine, isar-users, Baurzhan Ismagulov On 02/15/2018 04:20 PM, Claudius Heine wrote: > On 02/15/2018 01:37 PM, Alexander Smirnov wrote: >> >> Claudius Heine <claudius.heine.ext@siemens.com> 15 февраля 2018 г. >> 15:08:50 написал: >> >>> Hi Baurzhan, >>> >>> On 02/15/2018 12:42 PM, Baurzhan Ismagulov wrote: >>>> On Wed, Feb 14, 2018 at 02:15:09PM +0100, >>>> claudius.heine.ext@siemens.com wrote: >>>>> From: Claudius Heine <ch@denx.de> >>>>> >>>>> Previous there was redundant code between the buildchroot and >>>>> isar-image-base recipe. >>>>> >>>>> This patch moves the generation of the multistrap configuration to a >>>>> central python script that can be parametrized by both recipes. >>>>> >>>>> Signed-off-by: Claudius Heine <ch@denx.de> >>>>> --- >>>>> .../recipes-core/images/files/multistrap.conf.in | 38 ---------- >>>>> meta/lib/isar/multistrap.py | 80 >>>>> ++++++++++++++++++++++ >>>>> .../buildchroot/files/multistrap.conf.in | 37 ---------- >>>> >>>> Could you please post two final multistrap.confs you'd like to get >>>> with your >>>> changes? >>> >>> 'like to get'? I don't want to get any other multistrap.conf than >>> before. This is not a change of the multistrap.conf files, but just how >>> they are generated. The python configparser created ini-file format only >>> puts spaces around the '=' characters, but the resulting deserialized >>> objects is the same. >> >> The difference that I see IMHO: now you need to feed multistrap by >> community-standardized input, you propose to add custom tool that eats >> proprietary input. > > I don't get this point. What do you mean by that? You've introduced new language, like: +MULTISTRAP_ISAR_SOURCE = "copy:///${DEPLOY_DIR_APT}/${DISTRO}" +MULTISTRAP_ISAR_INSTALL = "${IMAGE_INSTALL}" +MULTISTRAP_ISAR_SUITE = "${DEBDISTRONAME}" So to have working multistrap config I *have* to write pre-config in this new language, what it's the simplification here, one set of options is replaced by another... > >> >>> >>> Ok, that is not 100% true, I changed the name of the isar repository >>> section. Now its called 'isar' in the image (previously 'Isar') and >>> buildchroot (previously 'isar-apt') multiconfig. But that is it. It has >>> no different effect outside of multistrap. >>> >>>> I'd like to understand the reasons for moving from templates to a magic >>>> generator. >>>> >>>> In my experience, magic is usually poorly documented (if at all) and >>>> very >>>> ununderstandable for users. Simple change of a line requires >>>> locating the >>>> generator, understanding what it does (they tend to be hairy with >>>> tons of >>>> conditions) and patching the upstream [Isar] code. Templates have their >>>> disadvantages, but are straightforward for users. In general, I'd >>>> like to keep >>>> the current UX and stick with templates, which is possible most of >>>> the time. >>> >>> You prefer duplicated code? Two slightly different templates for two >>> buildchroot and image recipe? >>> >> >> AFAIK the key idea is to have the only single multistrap that works >> with upstream repos. Other multistraps fetch packages from local >> cache. So after this is done, the problem with duplicacations in >> configs will go. >> >>> Also this patch is only an intermediate step of the way for multi-repo >>> support. IMO it has value on its own, since it centralized the >>> generation of the multistrap configuration. >>> The current kind of 'template' does not support creating repository >>> entries on the fly. >> >> What is the use case to have them "on the fly"? > > See: Also this patch is only an intermediate step of the way for > multi-repo support. > > Use-case for multi-repo: docker-ce repository available. Sorry, don't know what is docker-ce. Could you please describe whole repositories layout, just to understand the topic? > >> >> Alex >> >> It might be possible to extent the multiconfig with >>> shell scripting, but I prefer using python for such advanced logic. >> >> I don't care about python or bash > > At least Baurzhan seems to care, because he doesn't like debootstrap > because of it being in shell. Aghr, sent the mail on interrupted line, sorry. :-( I meant I don't care whether python or bash is used, my concern what is the input for they. If you prefer to parse multistrap config with python instead of bash, I could understand this. But here you want to introduce another language for doing the same things. So at least for me, it would be much easier to take some template for multistrap config (which syntax is well known and available in man page) and create the custom layout that I need. Alex > > Cheers, > Claudius > >> >>> >>> regards, >>> Claudius >>> >>> -- >>> DENX Software Engineering GmbH, Managing Director: Wolfgang Denk >>> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany >>> Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "isar-users" group. >>> To unsubscribe from this group and stop receiving emails from it, >>> send an email to isar-users+unsubscribe@googlegroups.com. >>> To post to this group, send email to isar-users@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/isar-users/d58343b9-c284-81e7-f743-2672c1f5f8ef%40siemens.com. >>> >>> For more options, visit https://groups.google.com/d/optout. >> >> >> > -- With best regards, Alexander Smirnov ilbers GmbH Baierbrunner Str. 28c D-81379 Munich +49 (89) 122 67 24-0 http://ilbers.de/ Commercial register Munich, HRB 214197 General manager: Baurzhan Ismagulov ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 2/2] centralize multistrap configuration generation 2018-02-15 13:39 ` Alexander Smirnov @ 2018-02-16 9:31 ` Claudius Heine 2018-02-16 10:35 ` Alexander Smirnov 0 siblings, 1 reply; 23+ messages in thread From: Claudius Heine @ 2018-02-16 9:31 UTC (permalink / raw) To: Alexander Smirnov, Claudius Heine, isar-users, Baurzhan Ismagulov [-- Attachment #1.1: Type: text/plain, Size: 7885 bytes --] On 02/15/2018 02:39 PM, Alexander Smirnov wrote: > > > On 02/15/2018 04:20 PM, Claudius Heine wrote: >> On 02/15/2018 01:37 PM, Alexander Smirnov wrote: >>> >>> Claudius Heine <claudius.heine.ext@siemens.com> 15 февраля 2018 г. >>> 15:08:50 написал: >>> >>>> Hi Baurzhan, >>>> >>>> On 02/15/2018 12:42 PM, Baurzhan Ismagulov wrote: >>>>> On Wed, Feb 14, 2018 at 02:15:09PM +0100, >>>>> claudius.heine.ext@siemens.com wrote: >>>>>> From: Claudius Heine <ch@denx.de> >>>>>> >>>>>> Previous there was redundant code between the buildchroot and >>>>>> isar-image-base recipe. >>>>>> >>>>>> This patch moves the generation of the multistrap configuration to a >>>>>> central python script that can be parametrized by both recipes. >>>>>> >>>>>> Signed-off-by: Claudius Heine <ch@denx.de> >>>>>> --- >>>>>> .../recipes-core/images/files/multistrap.conf.in | 38 ---------- >>>>>> meta/lib/isar/multistrap.py | 80 >>>>>> ++++++++++++++++++++++ >>>>>> .../buildchroot/files/multistrap.conf.in | 37 ---------- >>>>> >>>>> Could you please post two final multistrap.confs you'd like to get >>>>> with your >>>>> changes? >>>> >>>> 'like to get'? I don't want to get any other multistrap.conf than >>>> before. This is not a change of the multistrap.conf files, but just how >>>> they are generated. The python configparser created ini-file format >>>> only >>>> puts spaces around the '=' characters, but the resulting deserialized >>>> objects is the same. >>> >>> The difference that I see IMHO: now you need to feed multistrap by >>> community-standardized input, you propose to add custom tool that >>> eats proprietary input. >> >> I don't get this point. What do you mean by that? > > You've introduced new language, like: > > +MULTISTRAP_ISAR_SOURCE = "copy:///${DEPLOY_DIR_APT}/${DISTRO}" > +MULTISTRAP_ISAR_INSTALL = "${IMAGE_INSTALL}" > +MULTISTRAP_ISAR_SUITE = "${DEBDISTRONAME}" > > So to have working multistrap config I *have* to write pre-config in > this new language, what it's the simplification here, one set of options > is replaced by another... ??? Thats not a 'new language' that just bitbake variables. I think I don't get what you are saying. Thats the same stuff that previous to this change happend in this big wall of text sed command. IMO that this is much more to the point and therefore easier to handle than this big search and replace list. >> >>> >>>> >>>> Ok, that is not 100% true, I changed the name of the isar repository >>>> section. Now its called 'isar' in the image (previously 'Isar') and >>>> buildchroot (previously 'isar-apt') multiconfig. But that is it. It has >>>> no different effect outside of multistrap. >>>> >>>>> I'd like to understand the reasons for moving from templates to a >>>>> magic >>>>> generator. >>>>> >>>>> In my experience, magic is usually poorly documented (if at all) >>>>> and very >>>>> ununderstandable for users. Simple change of a line requires >>>>> locating the >>>>> generator, understanding what it does (they tend to be hairy with >>>>> tons of >>>>> conditions) and patching the upstream [Isar] code. Templates have >>>>> their >>>>> disadvantages, but are straightforward for users. In general, I'd >>>>> like to keep >>>>> the current UX and stick with templates, which is possible most of >>>>> the time. >>>> >>>> You prefer duplicated code? Two slightly different templates for two >>>> buildchroot and image recipe? >>>> >>> >>> AFAIK the key idea is to have the only single multistrap that works >>> with upstream repos. Other multistraps fetch packages from local >>> cache. So after this is done, the problem with duplicacations in >>> configs will go. >>> >>>> Also this patch is only an intermediate step of the way for multi-repo >>>> support. IMO it has value on its own, since it centralized the >>>> generation of the multistrap configuration. >>>> The current kind of 'template' does not support creating repository >>>> entries on the fly. >>> >>> What is the use case to have them "on the fly"? >> >> See: Also this patch is only an intermediate step of the way for >> multi-repo support. >> >> Use-case for multi-repo: docker-ce repository available. > > Sorry, don't know what is docker-ce. Could you please describe whole > repositories layout, just to understand the topic? I have the need to install docker community edition from the developers repository in my image. Currently I do it by downloading the 'deb' file from the repo manually and installing it, here I have to take care of updating the version to the latest one. What would be much better if I can just add this repository to the multistrap process and install the most current version like that. For that I need the ability to add additional repositories to the multistrap configuration from other layers. I don't understand what you mean by 'whole repositories layout'. You mean like a tree, but from what repository? > >> >>> >>> Alex >>> >>> It might be possible to extent the multiconfig with >>>> shell scripting, but I prefer using python for such advanced logic. >>> >>> I don't care about python or bash > >> At least Baurzhan seems to care, because he doesn't like debootstrap >> because of it being in shell. > > Aghr, sent the mail on interrupted line, sorry. :-( > > I meant I don't care whether python or bash is used, my concern what is > the input for they. If you prefer to parse multistrap config with python > instead of bash, I could understand this. I don't understand you again. I don't parse the multistrap config. I just generate one. > But here you want to introduce another language for doing the same things. Again, what other language? I don't remember writing or using any language parse in this patch. > So at least for me, it would be much easier to take some template for > multistrap config (which syntax is well known and available in man page) > and create the custom layout that I need. If you prefer the template approach, then we might need to introduce a new templating language, because only search and replace via sed doesn't cut it for the multi-repo feature. Since I know jinja [1] we could do it with this. But it still might result in something more hacky. [1] http://jinja.pocoo.org/ Also since Jan, already decreed the switch to debootstrap I will now continue working on a patchset for this. Cheers, Claudius > > Alex > >> >> Cheers, >> Claudius >> >>> >>>> >>>> regards, >>>> Claudius >>>> >>>> -- >>>> DENX Software Engineering GmbH, Managing Director: Wolfgang Denk >>>> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany >>>> Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "isar-users" group. >>>> To unsubscribe from this group and stop receiving emails from it, >>>> send an email to isar-users+unsubscribe@googlegroups.com. >>>> To post to this group, send email to isar-users@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/isar-users/d58343b9-c284-81e7-f743-2672c1f5f8ef%40siemens.com. >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>> >>> >>> >> > -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153 Keyserver: hkp://pool.sks-keyservers.net [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 2/2] centralize multistrap configuration generation 2018-02-16 9:31 ` Claudius Heine @ 2018-02-16 10:35 ` Alexander Smirnov 2018-02-16 11:35 ` Jan Kiszka 0 siblings, 1 reply; 23+ messages in thread From: Alexander Smirnov @ 2018-02-16 10:35 UTC (permalink / raw) To: Claudius Heine, Claudius Heine, isar-users, Baurzhan Ismagulov On 02/16/2018 12:31 PM, Claudius Heine wrote: > On 02/15/2018 02:39 PM, Alexander Smirnov wrote: >> >> >> On 02/15/2018 04:20 PM, Claudius Heine wrote: >>> On 02/15/2018 01:37 PM, Alexander Smirnov wrote: >>>> >>>> Claudius Heine <claudius.heine.ext@siemens.com> 15 февраля 2018 г. >>>> 15:08:50 написал: >>>> >>>>> Hi Baurzhan, >>>>> >>>>> On 02/15/2018 12:42 PM, Baurzhan Ismagulov wrote: >>>>>> On Wed, Feb 14, 2018 at 02:15:09PM +0100, >>>>>> claudius.heine.ext@siemens.com wrote: >>>>>>> From: Claudius Heine <ch@denx.de> >>>>>>> >>>>>>> Previous there was redundant code between the buildchroot and >>>>>>> isar-image-base recipe. >>>>>>> >>>>>>> This patch moves the generation of the multistrap configuration to a >>>>>>> central python script that can be parametrized by both recipes. >>>>>>> >>>>>>> Signed-off-by: Claudius Heine <ch@denx.de> >>>>>>> --- >>>>>>> .../recipes-core/images/files/multistrap.conf.in | 38 ---------- >>>>>>> meta/lib/isar/multistrap.py | 80 >>>>>>> ++++++++++++++++++++++ >>>>>>> .../buildchroot/files/multistrap.conf.in | 37 ---------- >>>>>> >>>>>> Could you please post two final multistrap.confs you'd like to get >>>>>> with your >>>>>> changes? >>>>> >>>>> 'like to get'? I don't want to get any other multistrap.conf than >>>>> before. This is not a change of the multistrap.conf files, but just how >>>>> they are generated. The python configparser created ini-file format >>>>> only >>>>> puts spaces around the '=' characters, but the resulting deserialized >>>>> objects is the same. >>>> >>>> The difference that I see IMHO: now you need to feed multistrap by >>>> community-standardized input, you propose to add custom tool that >>>> eats proprietary input. >>> >>> I don't get this point. What do you mean by that? >> >> You've introduced new language, like: >> >> +MULTISTRAP_ISAR_SOURCE = "copy:///${DEPLOY_DIR_APT}/${DISTRO}" >> +MULTISTRAP_ISAR_INSTALL = "${IMAGE_INSTALL}" >> +MULTISTRAP_ISAR_SUITE = "${DEBDISTRONAME}" >> >> So to have working multistrap config I *have* to write pre-config in >> this new language, what it's the simplification here, one set of options >> is replaced by another... > > ??? Thats not a 'new language' that just bitbake variables. I think I > don't get what you are saying. Thats the same stuff that previous to > this change happend in this big wall of text sed command. IMO that this > is much more to the point and therefore easier to handle than this big > search and replace list. 1. I suppose your parser eats only *specific* variables, so there is predefined list of extra variables I should somehow learn. 2. Each variable has some meaning, so depending on what I want from multistrap, I should *manually* assign them. For example if I want to use native arch, I have to set something. 3. These variables are complete duplication of multistrap config tokens. So all the above looks like: instead of setting variables directly in multistrap config, you prefer to set them in recipe. A matter of taste, but for me this sounds like unnecessary complication of existing approach. If everything here is only about avoiding duplication, so the multistrap cascading configuration [1] could be used. Common multistrap config part could be split in separate file and included by all rootfs generators. [1] - respective section in multistrap man page. > >>> >>>> >>>>> >>>>> Ok, that is not 100% true, I changed the name of the isar repository >>>>> section. Now its called 'isar' in the image (previously 'Isar') and >>>>> buildchroot (previously 'isar-apt') multiconfig. But that is it. It has >>>>> no different effect outside of multistrap. >>>>> >>>>>> I'd like to understand the reasons for moving from templates to a >>>>>> magic >>>>>> generator. >>>>>> >>>>>> In my experience, magic is usually poorly documented (if at all) >>>>>> and very >>>>>> ununderstandable for users. Simple change of a line requires >>>>>> locating the >>>>>> generator, understanding what it does (they tend to be hairy with >>>>>> tons of >>>>>> conditions) and patching the upstream [Isar] code. Templates have >>>>>> their >>>>>> disadvantages, but are straightforward for users. In general, I'd >>>>>> like to keep >>>>>> the current UX and stick with templates, which is possible most of >>>>>> the time. >>>>> >>>>> You prefer duplicated code? Two slightly different templates for two >>>>> buildchroot and image recipe? >>>>> >>>> >>>> AFAIK the key idea is to have the only single multistrap that works >>>> with upstream repos. Other multistraps fetch packages from local >>>> cache. So after this is done, the problem with duplicacations in >>>> configs will go. >>>> >>>>> Also this patch is only an intermediate step of the way for multi-repo >>>>> support. IMO it has value on its own, since it centralized the >>>>> generation of the multistrap configuration. >>>>> The current kind of 'template' does not support creating repository >>>>> entries on the fly. >>>> >>>> What is the use case to have them "on the fly"? >>> >>> See: Also this patch is only an intermediate step of the way for >>> multi-repo support. >>> >>> Use-case for multi-repo: docker-ce repository available. >> >> Sorry, don't know what is docker-ce. Could you please describe whole >> repositories layout, just to understand the topic? > > I have the need to install docker community edition from the developers > repository in my image. Currently I do it by downloading the 'deb' file > from the repo manually and installing it, here I have to take care of > updating the version to the latest one. What would be much better if I > can just add this repository to the multistrap process and install the > most current version like that. For that I need the ability to add > additional repositories to the multistrap configuration from other layers. Why you can't add respective section to multistrap.conf.in? > > I don't understand what you mean by 'whole repositories layout'. You > mean like a tree, but from what repository? Let's say, final mulstisrap configuration file or the list of sources in /etc/apt. Alex > >> >>> >>>> >>>> Alex >>>> >>>> It might be possible to extent the multiconfig with >>>>> shell scripting, but I prefer using python for such advanced logic. >>>> >>>> I don't care about python or bash > >>> At least Baurzhan seems to care, because he doesn't like debootstrap >>> because of it being in shell. >> >> Aghr, sent the mail on interrupted line, sorry. :-( >> >> I meant I don't care whether python or bash is used, my concern what is >> the input for they. If you prefer to parse multistrap config with python >> instead of bash, I could understand this. > > I don't understand you again. I don't parse the multistrap config. I > just generate one. > >> But here you want to introduce another language for doing the same things. > > Again, what other language? I don't remember writing or using any > language parse in this patch. > >> So at least for me, it would be much easier to take some template for >> multistrap config (which syntax is well known and available in man page) >> and create the custom layout that I need. > > If you prefer the template approach, then we might need to introduce a > new templating language, because only search and replace via sed doesn't > cut it for the multi-repo feature. > > Since I know jinja [1] we could do it with this. But it still might > result in something more hacky. > > [1] http://jinja.pocoo.org/ > > Also since Jan, already decreed the switch to debootstrap I will now > continue working on a patchset for this. I still don't understand why it's so difficult to create new section in multistrap.conf file manually? Moreover that's not the place in Isar which is going to be changed tens times per-build each build. Once you've changed it, you likely never will need to this again within current project. In general, it's impossible to get automatically configure upstream Isar according to your requirements. You some how need to specify what you want to have via some configuration interface: config files, variables, recipes etc... What is the value to change one manual work (populating multistrap.conf.in) to another (populating MULTISTRAP_* variables)? Alex > > Cheers, > Claudius > >> >> Alex >> >>> >>> Cheers, >>> Claudius >>> >>>> >>>>> >>>>> regards, >>>>> Claudius >>>>> >>>>> -- >>>>> DENX Software Engineering GmbH, Managing Director: Wolfgang Denk >>>>> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany >>>>> Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "isar-users" group. >>>>> To unsubscribe from this group and stop receiving emails from it, >>>>> send an email to isar-users+unsubscribe@googlegroups.com. >>>>> To post to this group, send email to isar-users@googlegroups.com. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/isar-users/d58343b9-c284-81e7-f743-2672c1f5f8ef%40siemens.com. >>>>> >>>>> For more options, visit https://groups.google.com/d/optout. >>>> >>>> >>>> >>> >> > -- With best regards, Alexander Smirnov ilbers GmbH Baierbrunner Str. 28c D-81379 Munich +49 (89) 122 67 24-0 http://ilbers.de/ Commercial register Munich, HRB 214197 General manager: Baurzhan Ismagulov ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 2/2] centralize multistrap configuration generation 2018-02-16 10:35 ` Alexander Smirnov @ 2018-02-16 11:35 ` Jan Kiszka 0 siblings, 0 replies; 23+ messages in thread From: Jan Kiszka @ 2018-02-16 11:35 UTC (permalink / raw) To: isar-users Folks, let's not discuss the legacy multistrap topic any further and focus on a clean re-implementation via debootstrap. That will remove all needs to translate into or from intermediate configuration files. Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH 2/2] centralize multistrap configuration generation 2018-02-15 13:20 ` Claudius Heine 2018-02-15 13:39 ` Alexander Smirnov @ 2018-02-15 13:53 ` Jan Kiszka 1 sibling, 0 replies; 23+ messages in thread From: Jan Kiszka @ 2018-02-15 13:53 UTC (permalink / raw) To: [ext] Claudius Heine, Alexander Smirnov, isar-users, Baurzhan Ismagulov On 2018-02-15 14:20, [ext] Claudius Heine wrote: >> It might be possible to extent the multiconfig with >>> shell scripting, but I prefer using python for such advanced logic. >> >> I don't care about python or bash > > At least Baurzhan seems to care, because he doesn't like debootstrap > because of it being in shell. > That's not a major argument. For us, debootstrap a tool with an API. If we can use that API without bending it, it's fine. If we can improve our use case by enhancing it upstream later on, it's perfect. Otherwise, we need to model Debian bootstrap after the community pattern with our own languages (shell and python). Jan -- Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 23+ messages in thread
end of thread, other threads:[~2018-02-16 11:35 UTC | newest] Thread overview: 23+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-02-14 13:15 [PATCH 0/2] Consolidate multistrap configuration generation claudius.heine.ext 2018-02-14 13:15 ` [PATCH 1/2] meta/classes/base: extend sys.path with lib directory claudius.heine.ext 2018-02-15 7:28 ` Jan Kiszka 2018-02-14 13:15 ` [PATCH 2/2] centralize multistrap configuration generation claudius.heine.ext 2018-02-15 7:28 ` Jan Kiszka 2018-02-15 8:16 ` Claudius Heine 2018-02-15 8:50 ` Jan Kiszka 2018-02-15 9:34 ` Alexander Smirnov 2018-02-15 10:05 ` Claudius Heine 2018-02-15 10:15 ` Jan Kiszka 2018-02-15 10:29 ` Alexander Smirnov 2018-02-15 11:27 ` Baurzhan Ismagulov 2018-02-15 13:17 ` Claudius Heine 2018-02-15 13:18 ` Jan Kiszka 2018-02-15 11:42 ` Baurzhan Ismagulov 2018-02-15 12:08 ` Claudius Heine 2018-02-15 12:37 ` Alexander Smirnov 2018-02-15 13:20 ` Claudius Heine 2018-02-15 13:39 ` Alexander Smirnov 2018-02-16 9:31 ` Claudius Heine 2018-02-16 10:35 ` Alexander Smirnov 2018-02-16 11:35 ` Jan Kiszka 2018-02-15 13:53 ` Jan Kiszka
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox