[PATCH 0/2] Consolidate multistrap configuration generation

public inbox for isar-users@googlegroups.com
 help / color / mirror / Atom feed

* [PATCH 0/2] Consolidate multistrap configuration generation
@ 2018-02-14 13:15 claudius.heine.ext
  2018-02-14 13:15 ` [PATCH 1/2] meta/classes/base: extend sys.path with lib directory claudius.heine.ext
  2018-02-14 13:15 ` [PATCH 2/2] centralize multistrap configuration generation claudius.heine.ext
  0 siblings, 2 replies; 23+ messages in thread
From: claudius.heine.ext @ 2018-02-14 13:15 UTC (permalink / raw)
  To: isar-users; +Cc: Claudius Heine

From: Claudius Heine <ch@denx.de>

Hi,

this patchset tries to consolidate the generation of the multistrap
configuration in the buildchroot and isar-image-base recipes to one.

This patchsetset also represents one part of the implementation of the
multi-repo support.

Cheers,
Claudius

Claudius Heine (2):
  meta/classes/base: extend sys.path with lib directory
  centralize multistrap configuration generation

 .../recipes-core/images/files/multistrap.conf.in   | 38 ----------
 meta-isar/recipes-core/images/isar-image-base.bb   | 33 ++++-----
 meta/classes/base.bbclass                          | 24 +++++++
 meta/lib/isar/__init__.py                          |  8 +++
 meta/lib/isar/multistrap.py                        | 80 ++++++++++++++++++++++
 meta/recipes-devtools/buildchroot/buildchroot.bb   | 28 +++-----
 .../buildchroot/files/multistrap.conf.in           | 37 ----------
 7 files changed, 134 insertions(+), 114 deletions(-)
 delete mode 100644 meta-isar/recipes-core/images/files/multistrap.conf.in
 create mode 100644 meta/lib/isar/__init__.py
 create mode 100644 meta/lib/isar/multistrap.py
 delete mode 100644 meta/recipes-devtools/buildchroot/files/multistrap.conf.in

-- 
2.15.1


^ permalink raw reply	[flat|nested] 23+ messages in thread

* [PATCH 1/2] meta/classes/base: extend sys.path with lib directory
  2018-02-14 13:15 [PATCH 0/2] Consolidate multistrap configuration generation claudius.heine.ext
@ 2018-02-14 13:15 ` claudius.heine.ext
  2018-02-15  7:28   ` Jan Kiszka
  2018-02-14 13:15 ` [PATCH 2/2] centralize multistrap configuration generation claudius.heine.ext
  1 sibling, 1 reply; 23+ messages in thread
From: claudius.heine.ext @ 2018-02-14 13:15 UTC (permalink / raw)
  To: isar-users; +Cc: Claudius Heine

From: Claudius Heine <ch@denx.de>

Currently scripts and modules that are placed in the 'lib' directory of
layers are not available in the recipes.

This change adds the required code to the base.bbclass in order to have
those imports available.

This change contains code from:
http://git.openembedded.org/openembedded-core/tree/meta/classes/base.bbclass?id=4a4eff9b9e3c614ff41b17cbec359b72dcbd41bd

Signed-off-by: Claudius Heine <ch@denx.de>
---
 meta/classes/base.bbclass | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index ae84677..3334789 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -37,6 +37,30 @@ bbfatal() {
 	exit 1
 }
 
+# Derived from openembedded-core/meta/classes/base.bbclass
+ISAR_IMPORTS += "os sys time"
+
+def isar_imports(d):
+	import sys
+
+	bbpath = d.getVar("BBPATH").split(":")
+	sys.path[0:0] = [os.path.join(dir, "lib") for dir in bbpath]
+
+	def inject(name, value):
+		"""Make a python object accessible from the metadata"""
+		if hasattr(bb.utils, "_context"):
+			bb.utils._context[name] = value
+		else:
+			__builtins__[name] = value
+
+	for toimport in d.getVar("ISAR_IMPORTS", True).split():
+		imported = __import__(toimport)
+		inject(toimport.split(".", 1)[0], imported)
+
+	return ""
+
+ISAR_IMPORTED := "${@isar_imports(d)}"
+
 # Derived from bitbake: bitbake/classes/base.bbclass
 addtask showdata
 do_showdata[nostamp] = "1"
-- 
2.15.1


^ permalink raw reply	[flat|nested] 23+ messages in thread

* [PATCH 2/2] centralize multistrap configuration generation
  2018-02-14 13:15 [PATCH 0/2] Consolidate multistrap configuration generation claudius.heine.ext
  2018-02-14 13:15 ` [PATCH 1/2] meta/classes/base: extend sys.path with lib directory claudius.heine.ext
@ 2018-02-14 13:15 ` claudius.heine.ext
  2018-02-15  7:28   ` Jan Kiszka
  2018-02-15 11:42   ` Baurzhan Ismagulov
  1 sibling, 2 replies; 23+ messages in thread
From: claudius.heine.ext @ 2018-02-14 13:15 UTC (permalink / raw)
  To: isar-users; +Cc: Claudius Heine

From: Claudius Heine <ch@denx.de>

Previous there was redundant code between the buildchroot and
isar-image-base recipe.

This patch moves the generation of the multistrap configuration to a
central python script that can be parametrized by both recipes.

Signed-off-by: Claudius Heine <ch@denx.de>
---
 .../recipes-core/images/files/multistrap.conf.in   | 38 ----------
 meta-isar/recipes-core/images/isar-image-base.bb   | 33 ++++-----
 meta/classes/base.bbclass                          |  2 +-
 meta/lib/isar/__init__.py                          |  8 +++
 meta/lib/isar/multistrap.py                        | 80 ++++++++++++++++++++++
 meta/recipes-devtools/buildchroot/buildchroot.bb   | 28 +++-----
 .../buildchroot/files/multistrap.conf.in           | 37 ----------
 7 files changed, 111 insertions(+), 115 deletions(-)
 delete mode 100644 meta-isar/recipes-core/images/files/multistrap.conf.in
 create mode 100644 meta/lib/isar/__init__.py
 create mode 100644 meta/lib/isar/multistrap.py
 delete mode 100644 meta/recipes-devtools/buildchroot/files/multistrap.conf.in

diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in b/meta-isar/recipes-core/images/files/multistrap.conf.in
deleted file mode 100644
index 432b6af..0000000
--- a/meta-isar/recipes-core/images/files/multistrap.conf.in
+++ /dev/null
@@ -1,38 +0,0 @@
-# This software is a part of ISAR.
-# Copyright (C) 2015-2017 ilbers GmbH
-
-[General]
-noauth=true
-unpack=true
-ignorenativearch=true
-bootstrap=##DISTRO_MULTICONF_BOOTSTRAP## Isar
-aptsources=##DISTRO_MULTICONF_APTSOURCES##
-configscript=##CONFIG_SCRIPT##
-setupscript=##SETUP_SCRIPT##
-hookdir=##DIR_HOOKS##
-
-[base]
-source=##DISTRO_APT_SOURCE##
-suite=##DISTRO_SUITE##
-components=##DISTRO_COMPONENTS##
-packages=##IMAGE_PREINSTALL##
-omitdebsrc=true
-
-[updates]
-source=##DISTRO_APT_SOURCE##
-suite=##DISTRO_SUITE##-updates
-components=##DISTRO_COMPONENTS##
-omitdebsrc=true
-
-[security]
-source=##DISTRO_APT_SOURCE_SEC##
-suite=##DISTRO_SUITE##/updates
-components=##DISTRO_COMPONENTS##
-omitdebsrc=true
-
-[Isar]
-packages=##IMAGE_INSTALL##
-source=##DEPLOY_DIR_APT##
-suite=##ISAR_DISTRO_SUITE##
-components=main
-omitdebsrc=true
diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb
index 8ddbabb..9913977 100644
--- a/meta-isar/recipes-core/images/isar-image-base.bb
+++ b/meta-isar/recipes-core/images/isar-image-base.bb
@@ -9,8 +9,7 @@ LICENSE = "gpl-2.0"
 LIC_FILES_CHKSUM = "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
 
 FILESPATH =. "${LAYERDIR_isar}/recipes-core/images/files:"
-SRC_URI = "file://multistrap.conf.in \
-	   file://${DISTRO_CONFIG_SCRIPT} \
+SRC_URI = "file://${DISTRO_CONFIG_SCRIPT} \
 	   file://setup.sh \
 	   file://download_dev-random"
 
@@ -28,6 +27,14 @@ WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
 
 do_rootfs[dirs] = "${WORKDIR}/hooks_multistrap"
 
+MULTISTRAP_CONFIG_SCRIPT = "${DISTRO_CONFIG_SCRIPT}"
+MULTISTRAP_APTSOURCES = "${DISTRO_MULTICONF_APTSOURCES}"
+MULTISTRAP_BOOTSTRAP = "${DISTRO_MULTICONF_BOOTSTRAP} isar"
+MULTISTRAP_PREINSTALL = "${IMAGE_PREINSTALL}"
+MULTISTRAP_ISAR_SOURCE = "copy:///${DEPLOY_DIR_APT}/${DISTRO}"
+MULTISTRAP_ISAR_INSTALL = "${IMAGE_INSTALL}"
+MULTISTRAP_ISAR_SUITE = "${DEBDISTRONAME}"
+
 do_rootfs() {
     E="${@ bb.utils.export_proxies(d)}"
 
@@ -35,25 +42,8 @@ do_rootfs() {
     chmod +x "${WORKDIR}/setup.sh"
     install -m 755 "${WORKDIR}/download_dev-random" "${WORKDIR}/hooks_multistrap/"
 
-    # Multistrap accepts only relative path in configuration files, so get it:
-    cd ${TOPDIR}
-    WORKDIR_REL=${@ os.path.relpath(d.getVar("WORKDIR", True))}
-
-    # Adjust multistrap config
-    sed -e 's|##IMAGE_PREINSTALL##|${IMAGE_PREINSTALL}|g' \
-        -e 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \
-        -e 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \
-        -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \
-        -e 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \
-        -e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \
-        -e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \
-        -e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/${DISTRO_CONFIG_SCRIPT}|g' \
-        -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \
-        -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
-        -e 's|##IMAGE_INSTALL##|${IMAGE_INSTALL}|g' \
-        -e 's|##DEPLOY_DIR_APT##|copy:///${DEPLOY_DIR_APT}/${DISTRO}|g' \
-        -e 's|##ISAR_DISTRO_SUITE##|${DEBDISTRONAME}|g' \
-           "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf"
+    echo "${@ isar.multistrap.generate_multistrap_config(d) }" > \
+         "${WORKDIR}/multistrap.conf"
 
     # Do not use bitbake flag [dirs] here because this folder should have
     # specific ownership.
@@ -62,6 +52,7 @@ do_rootfs() {
 
     # Create root filesystem. We must use sudo -E here to preserve the environment
     # because of proxy settings
+    cd "${TOPDIR}"
     sudo -E multistrap -a ${DISTRO_ARCH} -d "${IMAGE_ROOTFS}" -f "${WORKDIR}/multistrap.conf"
 
     # Configure root filesystem
diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 3334789..2a45a1e 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -38,7 +38,7 @@ bbfatal() {
 }
 
 # Derived from openembedded-core/meta/classes/base.bbclass
-ISAR_IMPORTS += "os sys time"
+ISAR_IMPORTS += "os sys time isar"
 
 def isar_imports(d):
 	import sys
diff --git a/meta/lib/isar/__init__.py b/meta/lib/isar/__init__.py
new file mode 100644
index 0000000..e6aeeaa
--- /dev/null
+++ b/meta/lib/isar/__init__.py
@@ -0,0 +1,8 @@
+# Custom kernel recipe include
+#
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2018
+#
+# SPDX-License-Identifier: MIT
+
+from . import multistrap
diff --git a/meta/lib/isar/multistrap.py b/meta/lib/isar/multistrap.py
new file mode 100644
index 0000000..7022f91
--- /dev/null
+++ b/meta/lib/isar/multistrap.py
@@ -0,0 +1,80 @@
+# Custom kernel recipe include
+#
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2018
+#
+# SPDX-License-Identifier: MIT
+
+def generate_multistrap_config(d):
+    import configparser as cp
+    import io
+
+    workdir = d.getVar("WORKDIR", True) or ""
+    topdir = d.getVar("TOPDIR", True) or ""
+    workdir_rel = os.path.relpath(workdir, start=topdir)
+
+    msconf = cp.RawConfigParser()
+
+    # Set general multistrap configuration:
+    configscript = d.getVar("MULTISTRAP_CONFIG_SCRIPT", True) or ""
+    setupscript = d.getVar("MULTISTRAP_SETUP_SCRIPT", True) or "setup.sh"
+    hookdir = d.getVar("MULTISTRAP_HOOK_DIR", True) or "hooks_multistrap"
+    aptsources = (d.getVar("MULTISTRAP_APTSOURCES", True) or "").split()
+    bootstrap = (d.getVar("MULTISTRAP_BOOTSTRAP", True) or "").split()
+
+    msconf.add_section("General")
+    msconf.set("General", "noauth", "true")
+    msconf.set("General", "unpack", "true")
+    msconf.set("General", "ignorenativearch", "true")
+    msconf.set("General", "configscript", "./" +
+               os.path.join(workdir_rel, configscript))
+    msconf.set("General", "setupscript", "./" +
+               os.path.join(workdir_rel, setupscript))
+    msconf.set("General", "hookdir", "./" +
+               os.path.join(workdir_rel, hookdir))
+    msconf.set("General", "aptsources", " ".join(aptsources))
+    msconf.set("General", "bootstrap", " ".join(bootstrap))
+
+    # Setup upstream repositories in multistrap configuration:
+    apt_source = d.getVar("DISTRO_APT_SOURCE", True) or ""
+    apt_source_sec = d.getVar("DISTRO_APT_SOURCE_SEC", True) or ""
+    apt_suite = d.getVar("DISTRO_SUITE", True) or ""
+    apt_components = (d.getVar("DISTRO_COMPONENTS", True) or "").split()
+    deb_packages = (d.getVar("MULTISTRAP_PREINSTALL", True) or "").split()
+    isar_source = d.getVar("MULTISTRAP_ISAR_SOURCE", True) or ""
+    isar_packages = (d.getVar("MULTISTRAP_ISAR_INSTALL", True) or "").split()
+    isar_suite = d.getVar("MULTISTRAP_ISAR_SUITE", True) or "isar"
+
+    msconf.add_section("base")
+    msconf.set("base", "source", apt_source)
+    msconf.set("base", "suite", apt_suite)
+    msconf.set("base", "components", " ".join(apt_components))
+    msconf.set("base", "omitdebsrc", "true")
+    msconf.set("base", "packages", " ".join(deb_packages))
+
+    msconf.add_section("updates")
+    msconf.set("updates", "source", apt_source)
+    msconf.set("updates", "suite", apt_suite + "-updates")
+    msconf.set("updates", "components", " ".join(apt_components))
+    msconf.set("updates", "omitdebsrc", "true")
+
+    msconf.add_section("security")
+    msconf.set("security", "source", apt_source_sec)
+    msconf.set("security", "suite", apt_suite + "/updates")
+    msconf.set("security", "components", " ".join(apt_components))
+    msconf.set("security", "omitdebsrc", "true")
+
+    msconf.add_section("isar")
+    msconf.set("isar", "source", isar_source)
+    msconf.set("isar", "suite", isar_suite)
+    msconf.set("isar", "components", "main")
+    msconf.set("isar", "omitdebsrc", "true")
+    if isar_packages :
+        msconf.set("isar", "packages",
+                   " ".join(isar_packages))
+
+    ret = ""
+    with io.StringIO() as out:
+        msconf.write(out)
+        ret = out.getvalue()
+    return ret
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
index 570c0ad..59b4e7e 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
@@ -9,8 +9,7 @@ LICENSE = "gpl-2.0"
 LIC_FILES_CHKSUM = "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
 
 FILESPATH =. "${LAYERDIR_core}/recipes-devtools/buildchroot/files:"
-SRC_URI = "file://multistrap.conf.in \
-           file://configscript.sh \
+SRC_URI = "file://configscript.sh \
            file://setup.sh \
            file://build.sh"
 PV = "1.0"
@@ -37,32 +36,25 @@ do_build[dirs] = "${BUILDCHROOT_DIR}/isar-apt \
                   ${BUILDCHROOT_DIR}/sys"
 do_build[depends] = "isar-apt:do_cache_config"
 
+MULTISTRAP_CONFIG_SCRIPT = "configscript.sh"
+MULTISTRAP_APTSOURCES = "isar ${DISTRO_MULTICONF_APTSOURCES}"
+MULTISTRAP_BOOTSTRAP = "${DISTRO_MULTICONF_BOOTSTRAP}"
+MULTISTRAP_PREINSTALL = "${BUILDCHROOT_PREINSTALL}"
+MULTISTRAP_ISAR_SOURCE = "file:///isar-apt"
+
 do_build() {
     E="${@ bb.utils.export_proxies(d)}"
 
     chmod +x "${WORKDIR}/setup.sh"
     chmod +x "${WORKDIR}/configscript.sh"
 
-    # Multistrap accepts only relative path in configuration files, so get it:
-    cd ${TOPDIR}
-    WORKDIR_REL=${@ os.path.relpath(d.getVar("WORKDIR", True))}
-
-    # Adjust multistrap config
-    sed -e 's|##BUILDCHROOT_PREINSTALL##|${BUILDCHROOT_PREINSTALL}|g' \
-        -e 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \
-        -e 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \
-        -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \
-        -e 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \
-        -e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \
-        -e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \
-        -e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/configscript.sh|g' \
-        -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \
-        -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
-           "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf"
+    echo "${@ isar.multistrap.generate_multistrap_config(d) }" > \
+         "${WORKDIR}/multistrap.conf"
 
     do_setup_mounts
 
     # Create root filesystem
+    cd "${TOPDIR}"
     sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f "${WORKDIR}/multistrap.conf"
 
     # Install package builder script
diff --git a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
deleted file mode 100644
index 480a4b8..0000000
--- a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
+++ /dev/null
@@ -1,37 +0,0 @@
-# This software is a part of ISAR.
-# Copyright (C) 2015-2017 ilbers GmbH
-
-[General]
-noauth=true
-unpack=true
-ignorenativearch=true
-bootstrap=##DISTRO_MULTICONF_BOOTSTRAP##
-aptsources=isar-apt ##DISTRO_MULTICONF_APTSOURCES##
-configscript=##CONFIG_SCRIPT##
-setupscript=##SETUP_SCRIPT##
-hookdir=##DIR_HOOKS##
-
-[base]
-source=##DISTRO_APT_SOURCE##
-suite=##DISTRO_SUITE##
-components=##DISTRO_COMPONENTS##
-packages=##BUILDCHROOT_PREINSTALL##
-omitdebsrc=true
-
-[updates]
-source=##DISTRO_APT_SOURCE##
-suite=##DISTRO_SUITE##-updates
-components=##DISTRO_COMPONENTS##
-omitdebsrc=true
-
-[security]
-source=##DISTRO_APT_SOURCE_SEC##
-suite=##DISTRO_SUITE##/updates
-components=##DISTRO_COMPONENTS##
-omitdebsrc=true
-
-[isar-apt]
-source=file:///isar-apt
-suite=isar
-components=main
-omitdebsrc=true
-- 
2.15.1


^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [PATCH 2/2] centralize multistrap configuration generation
  2018-02-14 13:15 ` [PATCH 2/2] centralize multistrap configuration generation claudius.heine.ext
@ 2018-02-15  7:28   ` Jan Kiszka
  2018-02-15  8:16     ` Claudius Heine
  2018-02-15  9:34     ` Alexander Smirnov
  2018-02-15 11:42   ` Baurzhan Ismagulov
  1 sibling, 2 replies; 23+ messages in thread
From: Jan Kiszka @ 2018-02-15  7:28 UTC (permalink / raw)
  To: [ext] claudius.heine.ext@siemens.com, isar-users,
	Alexander Smirnov, Baurzhan Ismagulov
  Cc: Claudius Heine

On 2018-02-14 14:15, [ext] claudius.heine.ext@siemens.com wrote:
> From: Claudius Heine <ch@denx.de>
> 
> Previous there was redundant code between the buildchroot and
> isar-image-base recipe.
> 
> This patch moves the generation of the multistrap configuration to a
> central python script that can be parametrized by both recipes.
> 
> Signed-off-by: Claudius Heine <ch@denx.de>
> ---
>  .../recipes-core/images/files/multistrap.conf.in   | 38 ----------
>  meta-isar/recipes-core/images/isar-image-base.bb   | 33 ++++-----
>  meta/classes/base.bbclass                          |  2 +-
>  meta/lib/isar/__init__.py                          |  8 +++
>  meta/lib/isar/multistrap.py                        | 80 ++++++++++++++++++++++
>  meta/recipes-devtools/buildchroot/buildchroot.bb   | 28 +++-----
>  .../buildchroot/files/multistrap.conf.in           | 37 ----------
>  7 files changed, 111 insertions(+), 115 deletions(-)
>  delete mode 100644 meta-isar/recipes-core/images/files/multistrap.conf.in
>  create mode 100644 meta/lib/isar/__init__.py
>  create mode 100644 meta/lib/isar/multistrap.py
>  delete mode 100644 meta/recipes-devtools/buildchroot/files/multistrap.conf.in
> 
> diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in b/meta-isar/recipes-core/images/files/multistrap.conf.in
> deleted file mode 100644
> index 432b6af..0000000
> --- a/meta-isar/recipes-core/images/files/multistrap.conf.in
> +++ /dev/null
> @@ -1,38 +0,0 @@
> -# This software is a part of ISAR.
> -# Copyright (C) 2015-2017 ilbers GmbH
> -
> -[General]
> -noauth=true
> -unpack=true
> -ignorenativearch=true
> -bootstrap=##DISTRO_MULTICONF_BOOTSTRAP## Isar
> -aptsources=##DISTRO_MULTICONF_APTSOURCES##
> -configscript=##CONFIG_SCRIPT##
> -setupscript=##SETUP_SCRIPT##
> -hookdir=##DIR_HOOKS##
> -
> -[base]
> -source=##DISTRO_APT_SOURCE##
> -suite=##DISTRO_SUITE##
> -components=##DISTRO_COMPONENTS##
> -packages=##IMAGE_PREINSTALL##
> -omitdebsrc=true
> -
> -[updates]
> -source=##DISTRO_APT_SOURCE##
> -suite=##DISTRO_SUITE##-updates
> -components=##DISTRO_COMPONENTS##
> -omitdebsrc=true
> -
> -[security]
> -source=##DISTRO_APT_SOURCE_SEC##
> -suite=##DISTRO_SUITE##/updates
> -components=##DISTRO_COMPONENTS##
> -omitdebsrc=true
> -
> -[Isar]
> -packages=##IMAGE_INSTALL##
> -source=##DEPLOY_DIR_APT##
> -suite=##ISAR_DISTRO_SUITE##
> -components=main
> -omitdebsrc=true
> diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb
> index 8ddbabb..9913977 100644
> --- a/meta-isar/recipes-core/images/isar-image-base.bb
> +++ b/meta-isar/recipes-core/images/isar-image-base.bb
> @@ -9,8 +9,7 @@ LICENSE = "gpl-2.0"
>  LIC_FILES_CHKSUM = "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
>  
>  FILESPATH =. "${LAYERDIR_isar}/recipes-core/images/files:"
> -SRC_URI = "file://multistrap.conf.in \
> -	   file://${DISTRO_CONFIG_SCRIPT} \
> +SRC_URI = "file://${DISTRO_CONFIG_SCRIPT} \
>  	   file://setup.sh \
>  	   file://download_dev-random"
>  
> @@ -28,6 +27,14 @@ WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
>  
>  do_rootfs[dirs] = "${WORKDIR}/hooks_multistrap"
>  
> +MULTISTRAP_CONFIG_SCRIPT = "${DISTRO_CONFIG_SCRIPT}"
> +MULTISTRAP_APTSOURCES = "${DISTRO_MULTICONF_APTSOURCES}"
> +MULTISTRAP_BOOTSTRAP = "${DISTRO_MULTICONF_BOOTSTRAP} isar"
> +MULTISTRAP_PREINSTALL = "${IMAGE_PREINSTALL}"
> +MULTISTRAP_ISAR_SOURCE = "copy:///${DEPLOY_DIR_APT}/${DISTRO}"
> +MULTISTRAP_ISAR_INSTALL = "${IMAGE_INSTALL}"
> +MULTISTRAP_ISAR_SUITE = "${DEBDISTRONAME}"
> +
>  do_rootfs() {
>      E="${@ bb.utils.export_proxies(d)}"
>  
> @@ -35,25 +42,8 @@ do_rootfs() {
>      chmod +x "${WORKDIR}/setup.sh"
>      install -m 755 "${WORKDIR}/download_dev-random" "${WORKDIR}/hooks_multistrap/"
>  
> -    # Multistrap accepts only relative path in configuration files, so get it:
> -    cd ${TOPDIR}
> -    WORKDIR_REL=${@ os.path.relpath(d.getVar("WORKDIR", True))}
> -
> -    # Adjust multistrap config
> -    sed -e 's|##IMAGE_PREINSTALL##|${IMAGE_PREINSTALL}|g' \
> -        -e 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \
> -        -e 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \
> -        -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \
> -        -e 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \
> -        -e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \
> -        -e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \
> -        -e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/${DISTRO_CONFIG_SCRIPT}|g' \
> -        -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \
> -        -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
> -        -e 's|##IMAGE_INSTALL##|${IMAGE_INSTALL}|g' \
> -        -e 's|##DEPLOY_DIR_APT##|copy:///${DEPLOY_DIR_APT}/${DISTRO}|g' \
> -        -e 's|##ISAR_DISTRO_SUITE##|${DEBDISTRONAME}|g' \
> -           "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf"
> +    echo "${@ isar.multistrap.generate_multistrap_config(d) }" > \
> +         "${WORKDIR}/multistrap.conf"
>  
>      # Do not use bitbake flag [dirs] here because this folder should have
>      # specific ownership.
> @@ -62,6 +52,7 @@ do_rootfs() {
>  
>      # Create root filesystem. We must use sudo -E here to preserve the environment
>      # because of proxy settings
> +    cd "${TOPDIR}"
>      sudo -E multistrap -a ${DISTRO_ARCH} -d "${IMAGE_ROOTFS}" -f "${WORKDIR}/multistrap.conf"
>  
>      # Configure root filesystem
> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
> index 3334789..2a45a1e 100644
> --- a/meta/classes/base.bbclass
> +++ b/meta/classes/base.bbclass
> @@ -38,7 +38,7 @@ bbfatal() {
>  }
>  
>  # Derived from openembedded-core/meta/classes/base.bbclass
> -ISAR_IMPORTS += "os sys time"
> +ISAR_IMPORTS += "os sys time isar"
>  
>  def isar_imports(d):
>  	import sys
> diff --git a/meta/lib/isar/__init__.py b/meta/lib/isar/__init__.py
> new file mode 100644
> index 0000000..e6aeeaa
> --- /dev/null
> +++ b/meta/lib/isar/__init__.py
> @@ -0,0 +1,8 @@
> +# Custom kernel recipe include
> +#
> +# This software is a part of ISAR.
> +# Copyright (c) Siemens AG, 2018
> +#
> +# SPDX-License-Identifier: MIT
> +
> +from . import multistrap
> diff --git a/meta/lib/isar/multistrap.py b/meta/lib/isar/multistrap.py
> new file mode 100644
> index 0000000..7022f91
> --- /dev/null
> +++ b/meta/lib/isar/multistrap.py
> @@ -0,0 +1,80 @@
> +# Custom kernel recipe include
> +#
> +# This software is a part of ISAR.
> +# Copyright (c) Siemens AG, 2018
> +#
> +# SPDX-License-Identifier: MIT
> +
> +def generate_multistrap_config(d):
> +    import configparser as cp
> +    import io
> +
> +    workdir = d.getVar("WORKDIR", True) or ""
> +    topdir = d.getVar("TOPDIR", True) or ""
> +    workdir_rel = os.path.relpath(workdir, start=topdir)
> +
> +    msconf = cp.RawConfigParser()
> +
> +    # Set general multistrap configuration:
> +    configscript = d.getVar("MULTISTRAP_CONFIG_SCRIPT", True) or ""
> +    setupscript = d.getVar("MULTISTRAP_SETUP_SCRIPT", True) or "setup.sh"
> +    hookdir = d.getVar("MULTISTRAP_HOOK_DIR", True) or "hooks_multistrap"
> +    aptsources = (d.getVar("MULTISTRAP_APTSOURCES", True) or "").split()
> +    bootstrap = (d.getVar("MULTISTRAP_BOOTSTRAP", True) or "").split()
> +
> +    msconf.add_section("General")
> +    msconf.set("General", "noauth", "true")
> +    msconf.set("General", "unpack", "true")
> +    msconf.set("General", "ignorenativearch", "true")
> +    msconf.set("General", "configscript", "./" +
> +               os.path.join(workdir_rel, configscript))
> +    msconf.set("General", "setupscript", "./" +
> +               os.path.join(workdir_rel, setupscript))
> +    msconf.set("General", "hookdir", "./" +
> +               os.path.join(workdir_rel, hookdir))
> +    msconf.set("General", "aptsources", " ".join(aptsources))
> +    msconf.set("General", "bootstrap", " ".join(bootstrap))
> +
> +    # Setup upstream repositories in multistrap configuration:
> +    apt_source = d.getVar("DISTRO_APT_SOURCE", True) or ""
> +    apt_source_sec = d.getVar("DISTRO_APT_SOURCE_SEC", True) or ""
> +    apt_suite = d.getVar("DISTRO_SUITE", True) or ""
> +    apt_components = (d.getVar("DISTRO_COMPONENTS", True) or "").split()
> +    deb_packages = (d.getVar("MULTISTRAP_PREINSTALL", True) or "").split()
> +    isar_source = d.getVar("MULTISTRAP_ISAR_SOURCE", True) or ""
> +    isar_packages = (d.getVar("MULTISTRAP_ISAR_INSTALL", True) or "").split()
> +    isar_suite = d.getVar("MULTISTRAP_ISAR_SUITE", True) or "isar"
> +
> +    msconf.add_section("base")
> +    msconf.set("base", "source", apt_source)
> +    msconf.set("base", "suite", apt_suite)
> +    msconf.set("base", "components", " ".join(apt_components))
> +    msconf.set("base", "omitdebsrc", "true")
> +    msconf.set("base", "packages", " ".join(deb_packages))
> +
> +    msconf.add_section("updates")
> +    msconf.set("updates", "source", apt_source)
> +    msconf.set("updates", "suite", apt_suite + "-updates")
> +    msconf.set("updates", "components", " ".join(apt_components))
> +    msconf.set("updates", "omitdebsrc", "true")
> +
> +    msconf.add_section("security")
> +    msconf.set("security", "source", apt_source_sec)
> +    msconf.set("security", "suite", apt_suite + "/updates")
> +    msconf.set("security", "components", " ".join(apt_components))
> +    msconf.set("security", "omitdebsrc", "true")
> +
> +    msconf.add_section("isar")
> +    msconf.set("isar", "source", isar_source)
> +    msconf.set("isar", "suite", isar_suite)
> +    msconf.set("isar", "components", "main")
> +    msconf.set("isar", "omitdebsrc", "true")
> +    if isar_packages :
> +        msconf.set("isar", "packages",
> +                   " ".join(isar_packages))
> +
> +    ret = ""
> +    with io.StringIO() as out:
> +        msconf.write(out)
> +        ret = out.getvalue()
> +    return ret
> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
> index 570c0ad..59b4e7e 100644
> --- a/meta/recipes-devtools/buildchroot/buildchroot.bb
> +++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
> @@ -9,8 +9,7 @@ LICENSE = "gpl-2.0"
>  LIC_FILES_CHKSUM = "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
>  
>  FILESPATH =. "${LAYERDIR_core}/recipes-devtools/buildchroot/files:"
> -SRC_URI = "file://multistrap.conf.in \
> -           file://configscript.sh \
> +SRC_URI = "file://configscript.sh \
>             file://setup.sh \
>             file://build.sh"
>  PV = "1.0"
> @@ -37,32 +36,25 @@ do_build[dirs] = "${BUILDCHROOT_DIR}/isar-apt \
>                    ${BUILDCHROOT_DIR}/sys"
>  do_build[depends] = "isar-apt:do_cache_config"
>  
> +MULTISTRAP_CONFIG_SCRIPT = "configscript.sh"
> +MULTISTRAP_APTSOURCES = "isar ${DISTRO_MULTICONF_APTSOURCES}"
> +MULTISTRAP_BOOTSTRAP = "${DISTRO_MULTICONF_BOOTSTRAP}"
> +MULTISTRAP_PREINSTALL = "${BUILDCHROOT_PREINSTALL}"
> +MULTISTRAP_ISAR_SOURCE = "file:///isar-apt"
> +
>  do_build() {
>      E="${@ bb.utils.export_proxies(d)}"
>  
>      chmod +x "${WORKDIR}/setup.sh"
>      chmod +x "${WORKDIR}/configscript.sh"
>  
> -    # Multistrap accepts only relative path in configuration files, so get it:
> -    cd ${TOPDIR}
> -    WORKDIR_REL=${@ os.path.relpath(d.getVar("WORKDIR", True))}
> -
> -    # Adjust multistrap config
> -    sed -e 's|##BUILDCHROOT_PREINSTALL##|${BUILDCHROOT_PREINSTALL}|g' \
> -        -e 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \
> -        -e 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \
> -        -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \
> -        -e 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \
> -        -e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \
> -        -e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \
> -        -e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/configscript.sh|g' \
> -        -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \
> -        -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
> -           "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf"
> +    echo "${@ isar.multistrap.generate_multistrap_config(d) }" > \
> +         "${WORKDIR}/multistrap.conf"
>  
>      do_setup_mounts
>  
>      # Create root filesystem
> +    cd "${TOPDIR}"
>      sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f "${WORKDIR}/multistrap.conf"
>  
>      # Install package builder script
> diff --git a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
> deleted file mode 100644
> index 480a4b8..0000000
> --- a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
> +++ /dev/null
> @@ -1,37 +0,0 @@
> -# This software is a part of ISAR.
> -# Copyright (C) 2015-2017 ilbers GmbH
> -
> -[General]
> -noauth=true
> -unpack=true
> -ignorenativearch=true
> -bootstrap=##DISTRO_MULTICONF_BOOTSTRAP##
> -aptsources=isar-apt ##DISTRO_MULTICONF_APTSOURCES##
> -configscript=##CONFIG_SCRIPT##
> -setupscript=##SETUP_SCRIPT##
> -hookdir=##DIR_HOOKS##
> -
> -[base]
> -source=##DISTRO_APT_SOURCE##
> -suite=##DISTRO_SUITE##
> -components=##DISTRO_COMPONENTS##
> -packages=##BUILDCHROOT_PREINSTALL##
> -omitdebsrc=true
> -
> -[updates]
> -source=##DISTRO_APT_SOURCE##
> -suite=##DISTRO_SUITE##-updates
> -components=##DISTRO_COMPONENTS##
> -omitdebsrc=true
> -
> -[security]
> -source=##DISTRO_APT_SOURCE_SEC##
> -suite=##DISTRO_SUITE##/updates
> -components=##DISTRO_COMPONENTS##
> -omitdebsrc=true
> -
> -[isar-apt]
> -source=file:///isar-apt
> -suite=isar
> -components=main
> -omitdebsrc=true
> 

This requires rebasing. I quickly tried, but the result broke the
package build (didn't analyze any details).


While this moves us in the right direction, it still leaves some
duplications behind (setup.sh) - and the big question if we should
invest any further in this horrible multistrap mess.

I've just enabled arm64, and it took a while to figure out that I was
missing some lines in setup.sh. In contrast, qemu-debootstrap just
worked (I used it to debug the multistrap output).

So, before we move on here: What all does multistrap currently give us,
compared to standard debootstrap? I only see "multiple repositories" so
far. But as Claudius is going to massage that part heavily anyway, I
really wonder if it's worth building on the multistrap indirection any
further.

How about something like this?

- [qemu-]debootstrap --variant=minbase ...
- inject all additional apt sources, including user-provided ones
- set pinnings and priority, including user-provided ones
- apt-get dist-upgrade

That would kick out all the multistrap complications.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [PATCH 1/2] meta/classes/base: extend sys.path with lib directory
  2018-02-14 13:15 ` [PATCH 1/2] meta/classes/base: extend sys.path with lib directory claudius.heine.ext
@ 2018-02-15  7:28   ` Jan Kiszka
  0 siblings, 0 replies; 23+ messages in thread
From: Jan Kiszka @ 2018-02-15  7:28 UTC (permalink / raw)
  To: [ext] claudius.heine.ext@siemens.com, isar-users, Henning Schild
  Cc: Claudius Heine

On 2018-02-14 14:15, [ext] claudius.heine.ext@siemens.com wrote:
> From: Claudius Heine <ch@denx.de>
> 
> Currently scripts and modules that are placed in the 'lib' directory of
> layers are not available in the recipes.
> 
> This change adds the required code to the base.bbclass in order to have
> those imports available.
> 
> This change contains code from:
> http://git.openembedded.org/openembedded-core/tree/meta/classes/base.bbclass?id=4a4eff9b9e3c614ff41b17cbec359b72dcbd41bd
> 
> Signed-off-by: Claudius Heine <ch@denx.de>
> ---
>  meta/classes/base.bbclass | 24 ++++++++++++++++++++++++
>  1 file changed, 24 insertions(+)
> 
> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
> index ae84677..3334789 100644
> --- a/meta/classes/base.bbclass
> +++ b/meta/classes/base.bbclass
> @@ -37,6 +37,30 @@ bbfatal() {
>  	exit 1
>  }
>  
> +# Derived from openembedded-core/meta/classes/base.bbclass
> +ISAR_IMPORTS += "os sys time"
> +
> +def isar_imports(d):
> +	import sys
> +
> +	bbpath = d.getVar("BBPATH").split(":")
> +	sys.path[0:0] = [os.path.join(dir, "lib") for dir in bbpath]
> +
> +	def inject(name, value):
> +		"""Make a python object accessible from the metadata"""
> +		if hasattr(bb.utils, "_context"):
> +			bb.utils._context[name] = value
> +		else:
> +			__builtins__[name] = value
> +
> +	for toimport in d.getVar("ISAR_IMPORTS", True).split():
> +		imported = __import__(toimport)
> +		inject(toimport.split(".", 1)[0], imported)
> +
> +	return ""
> +
> +ISAR_IMPORTED := "${@isar_imports(d)}"
> +
>  # Derived from bitbake: bitbake/classes/base.bbclass
>  addtask showdata
>  do_showdata[nostamp] = "1"
> 

Henning, does this help with selecting wic plugins as well? Or does wic
express its search paths differently and would not be affected by this
change?

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [PATCH 2/2] centralize multistrap configuration generation
  2018-02-15  7:28   ` Jan Kiszka
@ 2018-02-15  8:16     ` Claudius Heine
  2018-02-15  8:50       ` Jan Kiszka
  2018-02-15  9:34     ` Alexander Smirnov
  1 sibling, 1 reply; 23+ messages in thread
From: Claudius Heine @ 2018-02-15  8:16 UTC (permalink / raw)
  To: Jan Kiszka, isar-users, Alexander Smirnov, Baurzhan Ismagulov
  Cc: Claudius Heine

Hi Jan,

On 02/15/2018 08:28 AM, Jan Kiszka wrote:
> On 2018-02-14 14:15, [ext] claudius.heine.ext@siemens.com wrote:
>> From: Claudius Heine <ch@denx.de>
>>
>> Previous there was redundant code between the buildchroot and
>> isar-image-base recipe.
>>
>> This patch moves the generation of the multistrap configuration to a
>> central python script that can be parametrized by both recipes.
>>
>> Signed-off-by: Claudius Heine <ch@denx.de>
>> ---
>>   .../recipes-core/images/files/multistrap.conf.in   | 38 ----------
>>   meta-isar/recipes-core/images/isar-image-base.bb   | 33 ++++-----
>>   meta/classes/base.bbclass                          |  2 +-
>>   meta/lib/isar/__init__.py                          |  8 +++
>>   meta/lib/isar/multistrap.py                        | 80 ++++++++++++++++++++++
>>   meta/recipes-devtools/buildchroot/buildchroot.bb   | 28 +++-----
>>   .../buildchroot/files/multistrap.conf.in           | 37 ----------
>>   7 files changed, 111 insertions(+), 115 deletions(-)
>>   delete mode 100644 meta-isar/recipes-core/images/files/multistrap.conf.in
>>   create mode 100644 meta/lib/isar/__init__.py
>>   create mode 100644 meta/lib/isar/multistrap.py
>>   delete mode 100644 meta/recipes-devtools/buildchroot/files/multistrap.conf.in
>>
>> diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in b/meta-isar/recipes-core/images/files/multistrap.conf.in
>> deleted file mode 100644
>> index 432b6af..0000000
>> --- a/meta-isar/recipes-core/images/files/multistrap.conf.in
>> +++ /dev/null
>> @@ -1,38 +0,0 @@
>> -# This software is a part of ISAR.
>> -# Copyright (C) 2015-2017 ilbers GmbH
>> -
>> -[General]
>> -noauth=true
>> -unpack=true
>> -ignorenativearch=true
>> -bootstrap=##DISTRO_MULTICONF_BOOTSTRAP## Isar
>> -aptsources=##DISTRO_MULTICONF_APTSOURCES##
>> -configscript=##CONFIG_SCRIPT##
>> -setupscript=##SETUP_SCRIPT##
>> -hookdir=##DIR_HOOKS##
>> -
>> -[base]
>> -source=##DISTRO_APT_SOURCE##
>> -suite=##DISTRO_SUITE##
>> -components=##DISTRO_COMPONENTS##
>> -packages=##IMAGE_PREINSTALL##
>> -omitdebsrc=true
>> -
>> -[updates]
>> -source=##DISTRO_APT_SOURCE##
>> -suite=##DISTRO_SUITE##-updates
>> -components=##DISTRO_COMPONENTS##
>> -omitdebsrc=true
>> -
>> -[security]
>> -source=##DISTRO_APT_SOURCE_SEC##
>> -suite=##DISTRO_SUITE##/updates
>> -components=##DISTRO_COMPONENTS##
>> -omitdebsrc=true
>> -
>> -[Isar]
>> -packages=##IMAGE_INSTALL##
>> -source=##DEPLOY_DIR_APT##
>> -suite=##ISAR_DISTRO_SUITE##
>> -components=main
>> -omitdebsrc=true
>> diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb
>> index 8ddbabb..9913977 100644
>> --- a/meta-isar/recipes-core/images/isar-image-base.bb
>> +++ b/meta-isar/recipes-core/images/isar-image-base.bb
>> @@ -9,8 +9,7 @@ LICENSE = "gpl-2.0"
>>   LIC_FILES_CHKSUM = "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
>>   
>>   FILESPATH =. "${LAYERDIR_isar}/recipes-core/images/files:"
>> -SRC_URI = "file://multistrap.conf.in \
>> -	   file://${DISTRO_CONFIG_SCRIPT} \
>> +SRC_URI = "file://${DISTRO_CONFIG_SCRIPT} \
>>   	   file://setup.sh \
>>   	   file://download_dev-random"
>>   
>> @@ -28,6 +27,14 @@ WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
>>   
>>   do_rootfs[dirs] = "${WORKDIR}/hooks_multistrap"
>>   
>> +MULTISTRAP_CONFIG_SCRIPT = "${DISTRO_CONFIG_SCRIPT}"
>> +MULTISTRAP_APTSOURCES = "${DISTRO_MULTICONF_APTSOURCES}"
>> +MULTISTRAP_BOOTSTRAP = "${DISTRO_MULTICONF_BOOTSTRAP} isar"
>> +MULTISTRAP_PREINSTALL = "${IMAGE_PREINSTALL}"
>> +MULTISTRAP_ISAR_SOURCE = "copy:///${DEPLOY_DIR_APT}/${DISTRO}"
>> +MULTISTRAP_ISAR_INSTALL = "${IMAGE_INSTALL}"
>> +MULTISTRAP_ISAR_SUITE = "${DEBDISTRONAME}"
>> +
>>   do_rootfs() {
>>       E="${@ bb.utils.export_proxies(d)}"
>>   
>> @@ -35,25 +42,8 @@ do_rootfs() {
>>       chmod +x "${WORKDIR}/setup.sh"
>>       install -m 755 "${WORKDIR}/download_dev-random" "${WORKDIR}/hooks_multistrap/"
>>   
>> -    # Multistrap accepts only relative path in configuration files, so get it:
>> -    cd ${TOPDIR}
>> -    WORKDIR_REL=${@ os.path.relpath(d.getVar("WORKDIR", True))}
>> -
>> -    # Adjust multistrap config
>> -    sed -e 's|##IMAGE_PREINSTALL##|${IMAGE_PREINSTALL}|g' \
>> -        -e 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \
>> -        -e 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \
>> -        -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \
>> -        -e 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \
>> -        -e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \
>> -        -e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \
>> -        -e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/${DISTRO_CONFIG_SCRIPT}|g' \
>> -        -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \
>> -        -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
>> -        -e 's|##IMAGE_INSTALL##|${IMAGE_INSTALL}|g' \
>> -        -e 's|##DEPLOY_DIR_APT##|copy:///${DEPLOY_DIR_APT}/${DISTRO}|g' \
>> -        -e 's|##ISAR_DISTRO_SUITE##|${DEBDISTRONAME}|g' \
>> -           "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf"
>> +    echo "${@ isar.multistrap.generate_multistrap_config(d) }" > \
>> +         "${WORKDIR}/multistrap.conf"
>>   
>>       # Do not use bitbake flag [dirs] here because this folder should have
>>       # specific ownership.
>> @@ -62,6 +52,7 @@ do_rootfs() {
>>   
>>       # Create root filesystem. We must use sudo -E here to preserve the environment
>>       # because of proxy settings
>> +    cd "${TOPDIR}"
>>       sudo -E multistrap -a ${DISTRO_ARCH} -d "${IMAGE_ROOTFS}" -f "${WORKDIR}/multistrap.conf"
>>   
>>       # Configure root filesystem
>> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
>> index 3334789..2a45a1e 100644
>> --- a/meta/classes/base.bbclass
>> +++ b/meta/classes/base.bbclass
>> @@ -38,7 +38,7 @@ bbfatal() {
>>   }
>>   
>>   # Derived from openembedded-core/meta/classes/base.bbclass
>> -ISAR_IMPORTS += "os sys time"
>> +ISAR_IMPORTS += "os sys time isar"
>>   
>>   def isar_imports(d):
>>   	import sys
>> diff --git a/meta/lib/isar/__init__.py b/meta/lib/isar/__init__.py
>> new file mode 100644
>> index 0000000..e6aeeaa
>> --- /dev/null
>> +++ b/meta/lib/isar/__init__.py
>> @@ -0,0 +1,8 @@
>> +# Custom kernel recipe include
>> +#
>> +# This software is a part of ISAR.
>> +# Copyright (c) Siemens AG, 2018
>> +#
>> +# SPDX-License-Identifier: MIT
>> +
>> +from . import multistrap
>> diff --git a/meta/lib/isar/multistrap.py b/meta/lib/isar/multistrap.py
>> new file mode 100644
>> index 0000000..7022f91
>> --- /dev/null
>> +++ b/meta/lib/isar/multistrap.py
>> @@ -0,0 +1,80 @@
>> +# Custom kernel recipe include
>> +#
>> +# This software is a part of ISAR.
>> +# Copyright (c) Siemens AG, 2018
>> +#
>> +# SPDX-License-Identifier: MIT
>> +
>> +def generate_multistrap_config(d):
>> +    import configparser as cp
>> +    import io
>> +
>> +    workdir = d.getVar("WORKDIR", True) or ""
>> +    topdir = d.getVar("TOPDIR", True) or ""
>> +    workdir_rel = os.path.relpath(workdir, start=topdir)
>> +
>> +    msconf = cp.RawConfigParser()
>> +
>> +    # Set general multistrap configuration:
>> +    configscript = d.getVar("MULTISTRAP_CONFIG_SCRIPT", True) or ""
>> +    setupscript = d.getVar("MULTISTRAP_SETUP_SCRIPT", True) or "setup.sh"
>> +    hookdir = d.getVar("MULTISTRAP_HOOK_DIR", True) or "hooks_multistrap"
>> +    aptsources = (d.getVar("MULTISTRAP_APTSOURCES", True) or "").split()
>> +    bootstrap = (d.getVar("MULTISTRAP_BOOTSTRAP", True) or "").split()
>> +
>> +    msconf.add_section("General")
>> +    msconf.set("General", "noauth", "true")
>> +    msconf.set("General", "unpack", "true")
>> +    msconf.set("General", "ignorenativearch", "true")
>> +    msconf.set("General", "configscript", "./" +
>> +               os.path.join(workdir_rel, configscript))
>> +    msconf.set("General", "setupscript", "./" +
>> +               os.path.join(workdir_rel, setupscript))
>> +    msconf.set("General", "hookdir", "./" +
>> +               os.path.join(workdir_rel, hookdir))
>> +    msconf.set("General", "aptsources", " ".join(aptsources))
>> +    msconf.set("General", "bootstrap", " ".join(bootstrap))
>> +
>> +    # Setup upstream repositories in multistrap configuration:
>> +    apt_source = d.getVar("DISTRO_APT_SOURCE", True) or ""
>> +    apt_source_sec = d.getVar("DISTRO_APT_SOURCE_SEC", True) or ""
>> +    apt_suite = d.getVar("DISTRO_SUITE", True) or ""
>> +    apt_components = (d.getVar("DISTRO_COMPONENTS", True) or "").split()
>> +    deb_packages = (d.getVar("MULTISTRAP_PREINSTALL", True) or "").split()
>> +    isar_source = d.getVar("MULTISTRAP_ISAR_SOURCE", True) or ""
>> +    isar_packages = (d.getVar("MULTISTRAP_ISAR_INSTALL", True) or "").split()
>> +    isar_suite = d.getVar("MULTISTRAP_ISAR_SUITE", True) or "isar"
>> +
>> +    msconf.add_section("base")
>> +    msconf.set("base", "source", apt_source)
>> +    msconf.set("base", "suite", apt_suite)
>> +    msconf.set("base", "components", " ".join(apt_components))
>> +    msconf.set("base", "omitdebsrc", "true")
>> +    msconf.set("base", "packages", " ".join(deb_packages))
>> +
>> +    msconf.add_section("updates")
>> +    msconf.set("updates", "source", apt_source)
>> +    msconf.set("updates", "suite", apt_suite + "-updates")
>> +    msconf.set("updates", "components", " ".join(apt_components))
>> +    msconf.set("updates", "omitdebsrc", "true")
>> +
>> +    msconf.add_section("security")
>> +    msconf.set("security", "source", apt_source_sec)
>> +    msconf.set("security", "suite", apt_suite + "/updates")
>> +    msconf.set("security", "components", " ".join(apt_components))
>> +    msconf.set("security", "omitdebsrc", "true")
>> +
>> +    msconf.add_section("isar")
>> +    msconf.set("isar", "source", isar_source)
>> +    msconf.set("isar", "suite", isar_suite)
>> +    msconf.set("isar", "components", "main")
>> +    msconf.set("isar", "omitdebsrc", "true")
>> +    if isar_packages :
>> +        msconf.set("isar", "packages",
>> +                   " ".join(isar_packages))
>> +
>> +    ret = ""
>> +    with io.StringIO() as out:
>> +        msconf.write(out)
>> +        ret = out.getvalue()
>> +    return ret
>> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
>> index 570c0ad..59b4e7e 100644
>> --- a/meta/recipes-devtools/buildchroot/buildchroot.bb
>> +++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
>> @@ -9,8 +9,7 @@ LICENSE = "gpl-2.0"
>>   LIC_FILES_CHKSUM = "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
>>   
>>   FILESPATH =. "${LAYERDIR_core}/recipes-devtools/buildchroot/files:"
>> -SRC_URI = "file://multistrap.conf.in \
>> -           file://configscript.sh \
>> +SRC_URI = "file://configscript.sh \
>>              file://setup.sh \
>>              file://build.sh"
>>   PV = "1.0"
>> @@ -37,32 +36,25 @@ do_build[dirs] = "${BUILDCHROOT_DIR}/isar-apt \
>>                     ${BUILDCHROOT_DIR}/sys"
>>   do_build[depends] = "isar-apt:do_cache_config"
>>   
>> +MULTISTRAP_CONFIG_SCRIPT = "configscript.sh"
>> +MULTISTRAP_APTSOURCES = "isar ${DISTRO_MULTICONF_APTSOURCES}"
>> +MULTISTRAP_BOOTSTRAP = "${DISTRO_MULTICONF_BOOTSTRAP}"
>> +MULTISTRAP_PREINSTALL = "${BUILDCHROOT_PREINSTALL}"
>> +MULTISTRAP_ISAR_SOURCE = "file:///isar-apt"
>> +
>>   do_build() {
>>       E="${@ bb.utils.export_proxies(d)}"
>>   
>>       chmod +x "${WORKDIR}/setup.sh"
>>       chmod +x "${WORKDIR}/configscript.sh"
>>   
>> -    # Multistrap accepts only relative path in configuration files, so get it:
>> -    cd ${TOPDIR}
>> -    WORKDIR_REL=${@ os.path.relpath(d.getVar("WORKDIR", True))}
>> -
>> -    # Adjust multistrap config
>> -    sed -e 's|##BUILDCHROOT_PREINSTALL##|${BUILDCHROOT_PREINSTALL}|g' \
>> -        -e 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \
>> -        -e 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \
>> -        -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \
>> -        -e 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \
>> -        -e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \
>> -        -e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \
>> -        -e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/configscript.sh|g' \
>> -        -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \
>> -        -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
>> -           "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf"
>> +    echo "${@ isar.multistrap.generate_multistrap_config(d) }" > \
>> +         "${WORKDIR}/multistrap.conf"
>>   
>>       do_setup_mounts
>>   
>>       # Create root filesystem
>> +    cd "${TOPDIR}"
>>       sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f "${WORKDIR}/multistrap.conf"
>>   
>>       # Install package builder script
>> diff --git a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
>> deleted file mode 100644
>> index 480a4b8..0000000
>> --- a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
>> +++ /dev/null
>> @@ -1,37 +0,0 @@
>> -# This software is a part of ISAR.
>> -# Copyright (C) 2015-2017 ilbers GmbH
>> -
>> -[General]
>> -noauth=true
>> -unpack=true
>> -ignorenativearch=true
>> -bootstrap=##DISTRO_MULTICONF_BOOTSTRAP##
>> -aptsources=isar-apt ##DISTRO_MULTICONF_APTSOURCES##
>> -configscript=##CONFIG_SCRIPT##
>> -setupscript=##SETUP_SCRIPT##
>> -hookdir=##DIR_HOOKS##
>> -
>> -[base]
>> -source=##DISTRO_APT_SOURCE##
>> -suite=##DISTRO_SUITE##
>> -components=##DISTRO_COMPONENTS##
>> -packages=##BUILDCHROOT_PREINSTALL##
>> -omitdebsrc=true
>> -
>> -[updates]
>> -source=##DISTRO_APT_SOURCE##
>> -suite=##DISTRO_SUITE##-updates
>> -components=##DISTRO_COMPONENTS##
>> -omitdebsrc=true
>> -
>> -[security]
>> -source=##DISTRO_APT_SOURCE_SEC##
>> -suite=##DISTRO_SUITE##/updates
>> -components=##DISTRO_COMPONENTS##
>> -omitdebsrc=true
>> -
>> -[isar-apt]
>> -source=file:///isar-apt
>> -suite=isar
>> -components=main
>> -omitdebsrc=true
>>
> 
> This requires rebasing. I quickly tried, but the result broke the
> package build (didn't analyze any details).

I rebased this patch to next before sending it to the mailinglist, but 
there might have been some other patches applied in the between.

If you put this patch on top of:
af265bc0731d242afc5bc0ac9e8ebc6ffe6223f6
It should work. (It did at least for me.)

> While this moves us in the right direction, it still leaves some
> duplications behind (setup.sh) - and the big question if we should
> invest any further in this horrible multistrap mess.
> 
> I've just enabled arm64, and it took a while to figure out that I was
> missing some lines in setup.sh. In contrast, qemu-debootstrap just
> worked (I used it to debug the multistrap output).
> 
> So, before we move on here: What all does multistrap currently give us,
> compared to standard debootstrap? I only see "multiple repositories" so
> far. But as Claudius is going to massage that part heavily anyway, I
> really wonder if it's worth building on the multistrap indirection any
> further.
> 
> How about something like this?
> 
> - [qemu-]debootstrap --variant=minbase ...
> - inject all additional apt sources, including user-provided ones
> - set pinnings and priority, including user-provided ones
> - apt-get dist-upgrade
> 
> That would kick out all the multistrap complications.

I would prefer using debootstrap as well. I am currently looking at its 
code, and maybe it could be possible to further split up debootstraps 
execution into downloading, extracting, configuring steps. This way only 
configuring has to be done with root permissions (because of chroot) and 
qemu if needed.

Of course overwriting any base packages with generated isar ones might 
not be as convenient, because packages might have to be replaced at a 
later step.

If some additional functionality in debootstrap is needed, then it could 
make sense to fetch debootstrap in isar, like any other recipe, and 
apply some of our own patches to it. Of course we should try to upstream 
those too.

Cheers,
Claudius
-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [PATCH 2/2] centralize multistrap configuration generation
  2018-02-15  8:16     ` Claudius Heine
@ 2018-02-15  8:50       ` Jan Kiszka
  0 siblings, 0 replies; 23+ messages in thread
From: Jan Kiszka @ 2018-02-15  8:50 UTC (permalink / raw)
  To: Claudius Heine, isar-users, Alexander Smirnov, Baurzhan Ismagulov
  Cc: Claudius Heine

On 2018-02-15 09:16, Claudius Heine wrote:
> Hi Jan,
> 
> On 02/15/2018 08:28 AM, Jan Kiszka wrote:
>> On 2018-02-14 14:15, [ext] claudius.heine.ext@siemens.com wrote:
>>> From: Claudius Heine <ch@denx.de>
>>>
>>> Previous there was redundant code between the buildchroot and
>>> isar-image-base recipe.
>>>
>>> This patch moves the generation of the multistrap configuration to a
>>> central python script that can be parametrized by both recipes.
>>>
>>> Signed-off-by: Claudius Heine <ch@denx.de>
>>> ---
>>>   .../recipes-core/images/files/multistrap.conf.in   | 38 ----------
>>>   meta-isar/recipes-core/images/isar-image-base.bb   | 33 ++++-----
>>>   meta/classes/base.bbclass                          |  2 +-
>>>   meta/lib/isar/__init__.py                          |  8 +++
>>>   meta/lib/isar/multistrap.py                        | 80
>>> ++++++++++++++++++++++
>>>   meta/recipes-devtools/buildchroot/buildchroot.bb   | 28 +++-----
>>>   .../buildchroot/files/multistrap.conf.in           | 37 ----------
>>>   7 files changed, 111 insertions(+), 115 deletions(-)
>>>   delete mode 100644
>>> meta-isar/recipes-core/images/files/multistrap.conf.in
>>>   create mode 100644 meta/lib/isar/__init__.py
>>>   create mode 100644 meta/lib/isar/multistrap.py
>>>   delete mode 100644
>>> meta/recipes-devtools/buildchroot/files/multistrap.conf.in
>>>
>>> diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in
>>> b/meta-isar/recipes-core/images/files/multistrap.conf.in
>>> deleted file mode 100644
>>> index 432b6af..0000000
>>> --- a/meta-isar/recipes-core/images/files/multistrap.conf.in
>>> +++ /dev/null
>>> @@ -1,38 +0,0 @@
>>> -# This software is a part of ISAR.
>>> -# Copyright (C) 2015-2017 ilbers GmbH
>>> -
>>> -[General]
>>> -noauth=true
>>> -unpack=true
>>> -ignorenativearch=true
>>> -bootstrap=##DISTRO_MULTICONF_BOOTSTRAP## Isar
>>> -aptsources=##DISTRO_MULTICONF_APTSOURCES##
>>> -configscript=##CONFIG_SCRIPT##
>>> -setupscript=##SETUP_SCRIPT##
>>> -hookdir=##DIR_HOOKS##
>>> -
>>> -[base]
>>> -source=##DISTRO_APT_SOURCE##
>>> -suite=##DISTRO_SUITE##
>>> -components=##DISTRO_COMPONENTS##
>>> -packages=##IMAGE_PREINSTALL##
>>> -omitdebsrc=true
>>> -
>>> -[updates]
>>> -source=##DISTRO_APT_SOURCE##
>>> -suite=##DISTRO_SUITE##-updates
>>> -components=##DISTRO_COMPONENTS##
>>> -omitdebsrc=true
>>> -
>>> -[security]
>>> -source=##DISTRO_APT_SOURCE_SEC##
>>> -suite=##DISTRO_SUITE##/updates
>>> -components=##DISTRO_COMPONENTS##
>>> -omitdebsrc=true
>>> -
>>> -[Isar]
>>> -packages=##IMAGE_INSTALL##
>>> -source=##DEPLOY_DIR_APT##
>>> -suite=##ISAR_DISTRO_SUITE##
>>> -components=main
>>> -omitdebsrc=true
>>> diff --git a/meta-isar/recipes-core/images/isar-image-base.bb
>>> b/meta-isar/recipes-core/images/isar-image-base.bb
>>> index 8ddbabb..9913977 100644
>>> --- a/meta-isar/recipes-core/images/isar-image-base.bb
>>> +++ b/meta-isar/recipes-core/images/isar-image-base.bb
>>> @@ -9,8 +9,7 @@ LICENSE = "gpl-2.0"
>>>   LIC_FILES_CHKSUM =
>>> "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
>>>
>>>     FILESPATH =. "${LAYERDIR_isar}/recipes-core/images/files:"
>>> -SRC_URI = "file://multistrap.conf.in \
>>> -       file://${DISTRO_CONFIG_SCRIPT} \
>>> +SRC_URI = "file://${DISTRO_CONFIG_SCRIPT} \
>>>          file://setup.sh \
>>>          file://download_dev-random"
>>>   @@ -28,6 +27,14 @@ WORKDIR =
>>> "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
>>>     do_rootfs[dirs] = "${WORKDIR}/hooks_multistrap"
>>>   +MULTISTRAP_CONFIG_SCRIPT = "${DISTRO_CONFIG_SCRIPT}"
>>> +MULTISTRAP_APTSOURCES = "${DISTRO_MULTICONF_APTSOURCES}"
>>> +MULTISTRAP_BOOTSTRAP = "${DISTRO_MULTICONF_BOOTSTRAP} isar"
>>> +MULTISTRAP_PREINSTALL = "${IMAGE_PREINSTALL}"
>>> +MULTISTRAP_ISAR_SOURCE = "copy:///${DEPLOY_DIR_APT}/${DISTRO}"
>>> +MULTISTRAP_ISAR_INSTALL = "${IMAGE_INSTALL}"
>>> +MULTISTRAP_ISAR_SUITE = "${DEBDISTRONAME}"
>>> +
>>>   do_rootfs() {
>>>       E="${@ bb.utils.export_proxies(d)}"
>>>   @@ -35,25 +42,8 @@ do_rootfs() {
>>>       chmod +x "${WORKDIR}/setup.sh"
>>>       install -m 755 "${WORKDIR}/download_dev-random"
>>> "${WORKDIR}/hooks_multistrap/"
>>>   -    # Multistrap accepts only relative path in configuration
>>> files, so get it:
>>> -    cd ${TOPDIR}
>>> -    WORKDIR_REL=${@ os.path.relpath(d.getVar("WORKDIR", True))}
>>> -
>>> -    # Adjust multistrap config
>>> -    sed -e 's|##IMAGE_PREINSTALL##|${IMAGE_PREINSTALL}|g' \
>>> -        -e
>>> 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \
>>> -        -e
>>> 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \
>>> -        -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \
>>> -        -e 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \
>>> -        -e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \
>>> -        -e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \
>>> -        -e
>>> 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/${DISTRO_CONFIG_SCRIPT}|g' \
>>> -        -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \
>>> -        -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
>>> -        -e 's|##IMAGE_INSTALL##|${IMAGE_INSTALL}|g' \
>>> -        -e
>>> 's|##DEPLOY_DIR_APT##|copy:///${DEPLOY_DIR_APT}/${DISTRO}|g' \
>>> -        -e 's|##ISAR_DISTRO_SUITE##|${DEBDISTRONAME}|g' \
>>> -           "${WORKDIR}/multistrap.conf.in" >
>>> "${WORKDIR}/multistrap.conf"
>>> +    echo "${@ isar.multistrap.generate_multistrap_config(d) }" > \
>>> +         "${WORKDIR}/multistrap.conf"
>>>         # Do not use bitbake flag [dirs] here because this folder
>>> should have
>>>       # specific ownership.
>>> @@ -62,6 +52,7 @@ do_rootfs() {
>>>         # Create root filesystem. We must use sudo -E here to
>>> preserve the environment
>>>       # because of proxy settings
>>> +    cd "${TOPDIR}"
>>>       sudo -E multistrap -a ${DISTRO_ARCH} -d "${IMAGE_ROOTFS}" -f
>>> "${WORKDIR}/multistrap.conf"
>>>         # Configure root filesystem
>>> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
>>> index 3334789..2a45a1e 100644
>>> --- a/meta/classes/base.bbclass
>>> +++ b/meta/classes/base.bbclass
>>> @@ -38,7 +38,7 @@ bbfatal() {
>>>   }
>>>     # Derived from openembedded-core/meta/classes/base.bbclass
>>> -ISAR_IMPORTS += "os sys time"
>>> +ISAR_IMPORTS += "os sys time isar"
>>>     def isar_imports(d):
>>>       import sys
>>> diff --git a/meta/lib/isar/__init__.py b/meta/lib/isar/__init__.py
>>> new file mode 100644
>>> index 0000000..e6aeeaa
>>> --- /dev/null
>>> +++ b/meta/lib/isar/__init__.py
>>> @@ -0,0 +1,8 @@
>>> +# Custom kernel recipe include
>>> +#
>>> +# This software is a part of ISAR.
>>> +# Copyright (c) Siemens AG, 2018
>>> +#
>>> +# SPDX-License-Identifier: MIT
>>> +
>>> +from . import multistrap
>>> diff --git a/meta/lib/isar/multistrap.py b/meta/lib/isar/multistrap.py
>>> new file mode 100644
>>> index 0000000..7022f91
>>> --- /dev/null
>>> +++ b/meta/lib/isar/multistrap.py
>>> @@ -0,0 +1,80 @@
>>> +# Custom kernel recipe include
>>> +#
>>> +# This software is a part of ISAR.
>>> +# Copyright (c) Siemens AG, 2018
>>> +#
>>> +# SPDX-License-Identifier: MIT
>>> +
>>> +def generate_multistrap_config(d):
>>> +    import configparser as cp
>>> +    import io
>>> +
>>> +    workdir = d.getVar("WORKDIR", True) or ""
>>> +    topdir = d.getVar("TOPDIR", True) or ""
>>> +    workdir_rel = os.path.relpath(workdir, start=topdir)
>>> +
>>> +    msconf = cp.RawConfigParser()
>>> +
>>> +    # Set general multistrap configuration:
>>> +    configscript = d.getVar("MULTISTRAP_CONFIG_SCRIPT", True) or ""
>>> +    setupscript = d.getVar("MULTISTRAP_SETUP_SCRIPT", True) or
>>> "setup.sh"
>>> +    hookdir = d.getVar("MULTISTRAP_HOOK_DIR", True) or
>>> "hooks_multistrap"
>>> +    aptsources = (d.getVar("MULTISTRAP_APTSOURCES", True) or
>>> "").split()
>>> +    bootstrap = (d.getVar("MULTISTRAP_BOOTSTRAP", True) or "").split()
>>> +
>>> +    msconf.add_section("General")
>>> +    msconf.set("General", "noauth", "true")
>>> +    msconf.set("General", "unpack", "true")
>>> +    msconf.set("General", "ignorenativearch", "true")
>>> +    msconf.set("General", "configscript", "./" +
>>> +               os.path.join(workdir_rel, configscript))
>>> +    msconf.set("General", "setupscript", "./" +
>>> +               os.path.join(workdir_rel, setupscript))
>>> +    msconf.set("General", "hookdir", "./" +
>>> +               os.path.join(workdir_rel, hookdir))
>>> +    msconf.set("General", "aptsources", " ".join(aptsources))
>>> +    msconf.set("General", "bootstrap", " ".join(bootstrap))
>>> +
>>> +    # Setup upstream repositories in multistrap configuration:
>>> +    apt_source = d.getVar("DISTRO_APT_SOURCE", True) or ""
>>> +    apt_source_sec = d.getVar("DISTRO_APT_SOURCE_SEC", True) or ""
>>> +    apt_suite = d.getVar("DISTRO_SUITE", True) or ""
>>> +    apt_components = (d.getVar("DISTRO_COMPONENTS", True) or
>>> "").split()
>>> +    deb_packages = (d.getVar("MULTISTRAP_PREINSTALL", True) or
>>> "").split()
>>> +    isar_source = d.getVar("MULTISTRAP_ISAR_SOURCE", True) or ""
>>> +    isar_packages = (d.getVar("MULTISTRAP_ISAR_INSTALL", True) or
>>> "").split()
>>> +    isar_suite = d.getVar("MULTISTRAP_ISAR_SUITE", True) or "isar"
>>> +
>>> +    msconf.add_section("base")
>>> +    msconf.set("base", "source", apt_source)
>>> +    msconf.set("base", "suite", apt_suite)
>>> +    msconf.set("base", "components", " ".join(apt_components))
>>> +    msconf.set("base", "omitdebsrc", "true")
>>> +    msconf.set("base", "packages", " ".join(deb_packages))
>>> +
>>> +    msconf.add_section("updates")
>>> +    msconf.set("updates", "source", apt_source)
>>> +    msconf.set("updates", "suite", apt_suite + "-updates")
>>> +    msconf.set("updates", "components", " ".join(apt_components))
>>> +    msconf.set("updates", "omitdebsrc", "true")
>>> +
>>> +    msconf.add_section("security")
>>> +    msconf.set("security", "source", apt_source_sec)
>>> +    msconf.set("security", "suite", apt_suite + "/updates")
>>> +    msconf.set("security", "components", " ".join(apt_components))
>>> +    msconf.set("security", "omitdebsrc", "true")
>>> +
>>> +    msconf.add_section("isar")
>>> +    msconf.set("isar", "source", isar_source)
>>> +    msconf.set("isar", "suite", isar_suite)
>>> +    msconf.set("isar", "components", "main")
>>> +    msconf.set("isar", "omitdebsrc", "true")
>>> +    if isar_packages :
>>> +        msconf.set("isar", "packages",
>>> +                   " ".join(isar_packages))
>>> +
>>> +    ret = ""
>>> +    with io.StringIO() as out:
>>> +        msconf.write(out)
>>> +        ret = out.getvalue()
>>> +    return ret
>>> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb
>>> b/meta/recipes-devtools/buildchroot/buildchroot.bb
>>> index 570c0ad..59b4e7e 100644
>>> --- a/meta/recipes-devtools/buildchroot/buildchroot.bb
>>> +++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
>>> @@ -9,8 +9,7 @@ LICENSE = "gpl-2.0"
>>>   LIC_FILES_CHKSUM =
>>> "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
>>>
>>>     FILESPATH =. "${LAYERDIR_core}/recipes-devtools/buildchroot/files:"
>>> -SRC_URI = "file://multistrap.conf.in \
>>> -           file://configscript.sh \
>>> +SRC_URI = "file://configscript.sh \
>>>              file://setup.sh \
>>>              file://build.sh"
>>>   PV = "1.0"
>>> @@ -37,32 +36,25 @@ do_build[dirs] = "${BUILDCHROOT_DIR}/isar-apt \
>>>                     ${BUILDCHROOT_DIR}/sys"
>>>   do_build[depends] = "isar-apt:do_cache_config"
>>>   +MULTISTRAP_CONFIG_SCRIPT = "configscript.sh"
>>> +MULTISTRAP_APTSOURCES = "isar ${DISTRO_MULTICONF_APTSOURCES}"
>>> +MULTISTRAP_BOOTSTRAP = "${DISTRO_MULTICONF_BOOTSTRAP}"
>>> +MULTISTRAP_PREINSTALL = "${BUILDCHROOT_PREINSTALL}"
>>> +MULTISTRAP_ISAR_SOURCE = "file:///isar-apt"
>>> +
>>>   do_build() {
>>>       E="${@ bb.utils.export_proxies(d)}"
>>>         chmod +x "${WORKDIR}/setup.sh"
>>>       chmod +x "${WORKDIR}/configscript.sh"
>>>   -    # Multistrap accepts only relative path in configuration
>>> files, so get it:
>>> -    cd ${TOPDIR}
>>> -    WORKDIR_REL=${@ os.path.relpath(d.getVar("WORKDIR", True))}
>>> -
>>> -    # Adjust multistrap config
>>> -    sed -e 's|##BUILDCHROOT_PREINSTALL##|${BUILDCHROOT_PREINSTALL}|g' \
>>> -        -e
>>> 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \
>>> -        -e
>>> 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \
>>> -        -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \
>>> -        -e 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \
>>> -        -e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \
>>> -        -e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \
>>> -        -e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/configscript.sh|g' \
>>> -        -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \
>>> -        -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
>>> -           "${WORKDIR}/multistrap.conf.in" >
>>> "${WORKDIR}/multistrap.conf"
>>> +    echo "${@ isar.multistrap.generate_multistrap_config(d) }" > \
>>> +         "${WORKDIR}/multistrap.conf"
>>>         do_setup_mounts
>>>         # Create root filesystem
>>> +    cd "${TOPDIR}"
>>>       sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f
>>> "${WORKDIR}/multistrap.conf"
>>>         # Install package builder script
>>> diff --git
>>> a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
>>> b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
>>> deleted file mode 100644
>>> index 480a4b8..0000000
>>> --- a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
>>> +++ /dev/null
>>> @@ -1,37 +0,0 @@
>>> -# This software is a part of ISAR.
>>> -# Copyright (C) 2015-2017 ilbers GmbH
>>> -
>>> -[General]
>>> -noauth=true
>>> -unpack=true
>>> -ignorenativearch=true
>>> -bootstrap=##DISTRO_MULTICONF_BOOTSTRAP##
>>> -aptsources=isar-apt ##DISTRO_MULTICONF_APTSOURCES##
>>> -configscript=##CONFIG_SCRIPT##
>>> -setupscript=##SETUP_SCRIPT##
>>> -hookdir=##DIR_HOOKS##
>>> -
>>> -[base]
>>> -source=##DISTRO_APT_SOURCE##
>>> -suite=##DISTRO_SUITE##
>>> -components=##DISTRO_COMPONENTS##
>>> -packages=##BUILDCHROOT_PREINSTALL##
>>> -omitdebsrc=true
>>> -
>>> -[updates]
>>> -source=##DISTRO_APT_SOURCE##
>>> -suite=##DISTRO_SUITE##-updates
>>> -components=##DISTRO_COMPONENTS##
>>> -omitdebsrc=true
>>> -
>>> -[security]
>>> -source=##DISTRO_APT_SOURCE_SEC##
>>> -suite=##DISTRO_SUITE##/updates
>>> -components=##DISTRO_COMPONENTS##
>>> -omitdebsrc=true
>>> -
>>> -[isar-apt]
>>> -source=file:///isar-apt
>>> -suite=isar
>>> -components=main
>>> -omitdebsrc=true
>>>
>>
>> This requires rebasing. I quickly tried, but the result broke the
>> package build (didn't analyze any details).
> 
> I rebased this patch to next before sending it to the mailinglist, but
> there might have been some other patches applied in the between.
> 
> If you put this patch on top of:
> af265bc0731d242afc5bc0ac9e8ebc6ffe6223f6
> It should work. (It did at least for me.)
> 

Next moved on, so this required rebasing, maybe more. But let's first of
all decide about the strategy before investing further into multistrap.

>> While this moves us in the right direction, it still leaves some
>> duplications behind (setup.sh) - and the big question if we should
>> invest any further in this horrible multistrap mess.
>>
>> I've just enabled arm64, and it took a while to figure out that I was
>> missing some lines in setup.sh. In contrast, qemu-debootstrap just
>> worked (I used it to debug the multistrap output).
>>
>> So, before we move on here: What all does multistrap currently give us,
>> compared to standard debootstrap? I only see "multiple repositories" so
>> far. But as Claudius is going to massage that part heavily anyway, I
>> really wonder if it's worth building on the multistrap indirection any
>> further.
>>
>> How about something like this?
>>
>> - [qemu-]debootstrap --variant=minbase ...
>> - inject all additional apt sources, including user-provided ones
>> - set pinnings and priority, including user-provided ones
>> - apt-get dist-upgrade
>>
>> That would kick out all the multistrap complications.
> 
> I would prefer using debootstrap as well. I am currently looking at its
> code, and maybe it could be possible to further split up debootstraps
> execution into downloading, extracting, configuring steps. This way only
> configuring has to be done with root permissions (because of chroot) and
> qemu if needed.>
> Of course overwriting any base packages with generated isar ones might
> not be as convenient, because packages might have to be replaced at a
> later step.

That was the idea behind "dist-upgrade", though I cannot say for sure it
will work reliably enough.

> 
> If some additional functionality in debootstrap is needed, then it could
> make sense to fetch debootstrap in isar, like any other recipe, and
> apply some of our own patches to it. Of course we should try to upstream
> those too.

Sounds promising. I'm sure there will be traps and pitfalls, but so
there are with multistrap, and its code become rather a barrier than a
help for us.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [PATCH 2/2] centralize multistrap configuration generation
  2018-02-15  7:28   ` Jan Kiszka
  2018-02-15  8:16     ` Claudius Heine
@ 2018-02-15  9:34     ` Alexander Smirnov
  2018-02-15 10:05       ` Claudius Heine
  1 sibling, 1 reply; 23+ messages in thread
From: Alexander Smirnov @ 2018-02-15  9:34 UTC (permalink / raw)
  To: Jan Kiszka, isar-users, Baurzhan Ismagulov; +Cc: Claudius Heine



On 02/15/2018 10:28 AM, Jan Kiszka wrote:
> On 2018-02-14 14:15, [ext] claudius.heine.ext@siemens.com wrote:
>> From: Claudius Heine <ch@denx.de>
>>
>> Previous there was redundant code between the buildchroot and
>> isar-image-base recipe.
>>
>> This patch moves the generation of the multistrap configuration to a
>> central python script that can be parametrized by both recipes.
>>
>> Signed-off-by: Claudius Heine <ch@denx.de>
>> ---
>>   .../recipes-core/images/files/multistrap.conf.in   | 38 ----------
>>   meta-isar/recipes-core/images/isar-image-base.bb   | 33 ++++-----
>>   meta/classes/base.bbclass                          |  2 +-
>>   meta/lib/isar/__init__.py                          |  8 +++
>>   meta/lib/isar/multistrap.py                        | 80 ++++++++++++++++++++++
>>   meta/recipes-devtools/buildchroot/buildchroot.bb   | 28 +++-----
>>   .../buildchroot/files/multistrap.conf.in           | 37 ----------
>>   7 files changed, 111 insertions(+), 115 deletions(-)
>>   delete mode 100644 meta-isar/recipes-core/images/files/multistrap.conf.in
>>   create mode 100644 meta/lib/isar/__init__.py
>>   create mode 100644 meta/lib/isar/multistrap.py
>>   delete mode 100644 meta/recipes-devtools/buildchroot/files/multistrap.conf.in
>>
>> diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in b/meta-isar/recipes-core/images/files/multistrap.conf.in
>> deleted file mode 100644
>> index 432b6af..0000000
>> --- a/meta-isar/recipes-core/images/files/multistrap.conf.in
>> +++ /dev/null
>> @@ -1,38 +0,0 @@
>> -# This software is a part of ISAR.
>> -# Copyright (C) 2015-2017 ilbers GmbH
>> -
>> -[General]
>> -noauth=true
>> -unpack=true
>> -ignorenativearch=true
>> -bootstrap=##DISTRO_MULTICONF_BOOTSTRAP## Isar
>> -aptsources=##DISTRO_MULTICONF_APTSOURCES##
>> -configscript=##CONFIG_SCRIPT##
>> -setupscript=##SETUP_SCRIPT##
>> -hookdir=##DIR_HOOKS##
>> -
>> -[base]
>> -source=##DISTRO_APT_SOURCE##
>> -suite=##DISTRO_SUITE##
>> -components=##DISTRO_COMPONENTS##
>> -packages=##IMAGE_PREINSTALL##
>> -omitdebsrc=true
>> -
>> -[updates]
>> -source=##DISTRO_APT_SOURCE##
>> -suite=##DISTRO_SUITE##-updates
>> -components=##DISTRO_COMPONENTS##
>> -omitdebsrc=true
>> -
>> -[security]
>> -source=##DISTRO_APT_SOURCE_SEC##
>> -suite=##DISTRO_SUITE##/updates
>> -components=##DISTRO_COMPONENTS##
>> -omitdebsrc=true
>> -
>> -[Isar]
>> -packages=##IMAGE_INSTALL##
>> -source=##DEPLOY_DIR_APT##
>> -suite=##ISAR_DISTRO_SUITE##
>> -components=main
>> -omitdebsrc=true
>> diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb
>> index 8ddbabb..9913977 100644
>> --- a/meta-isar/recipes-core/images/isar-image-base.bb
>> +++ b/meta-isar/recipes-core/images/isar-image-base.bb
>> @@ -9,8 +9,7 @@ LICENSE = "gpl-2.0"
>>   LIC_FILES_CHKSUM = "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
>>   
>>   FILESPATH =. "${LAYERDIR_isar}/recipes-core/images/files:"
>> -SRC_URI = "file://multistrap.conf.in \
>> -	   file://${DISTRO_CONFIG_SCRIPT} \
>> +SRC_URI = "file://${DISTRO_CONFIG_SCRIPT} \
>>   	   file://setup.sh \
>>   	   file://download_dev-random"
>>   
>> @@ -28,6 +27,14 @@ WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
>>   
>>   do_rootfs[dirs] = "${WORKDIR}/hooks_multistrap"
>>   
>> +MULTISTRAP_CONFIG_SCRIPT = "${DISTRO_CONFIG_SCRIPT}"
>> +MULTISTRAP_APTSOURCES = "${DISTRO_MULTICONF_APTSOURCES}"
>> +MULTISTRAP_BOOTSTRAP = "${DISTRO_MULTICONF_BOOTSTRAP} isar"
>> +MULTISTRAP_PREINSTALL = "${IMAGE_PREINSTALL}"
>> +MULTISTRAP_ISAR_SOURCE = "copy:///${DEPLOY_DIR_APT}/${DISTRO}"
>> +MULTISTRAP_ISAR_INSTALL = "${IMAGE_INSTALL}"
>> +MULTISTRAP_ISAR_SUITE = "${DEBDISTRONAME}"
>> +
>>   do_rootfs() {
>>       E="${@ bb.utils.export_proxies(d)}"
>>   
>> @@ -35,25 +42,8 @@ do_rootfs() {
>>       chmod +x "${WORKDIR}/setup.sh"
>>       install -m 755 "${WORKDIR}/download_dev-random" "${WORKDIR}/hooks_multistrap/"
>>   
>> -    # Multistrap accepts only relative path in configuration files, so get it:
>> -    cd ${TOPDIR}
>> -    WORKDIR_REL=${@ os.path.relpath(d.getVar("WORKDIR", True))}
>> -
>> -    # Adjust multistrap config
>> -    sed -e 's|##IMAGE_PREINSTALL##|${IMAGE_PREINSTALL}|g' \
>> -        -e 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \
>> -        -e 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \
>> -        -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \
>> -        -e 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \
>> -        -e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \
>> -        -e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \
>> -        -e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/${DISTRO_CONFIG_SCRIPT}|g' \
>> -        -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \
>> -        -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
>> -        -e 's|##IMAGE_INSTALL##|${IMAGE_INSTALL}|g' \
>> -        -e 's|##DEPLOY_DIR_APT##|copy:///${DEPLOY_DIR_APT}/${DISTRO}|g' \
>> -        -e 's|##ISAR_DISTRO_SUITE##|${DEBDISTRONAME}|g' \
>> -           "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf"
>> +    echo "${@ isar.multistrap.generate_multistrap_config(d) }" > \
>> +         "${WORKDIR}/multistrap.conf"
>>   
>>       # Do not use bitbake flag [dirs] here because this folder should have
>>       # specific ownership.
>> @@ -62,6 +52,7 @@ do_rootfs() {
>>   
>>       # Create root filesystem. We must use sudo -E here to preserve the environment
>>       # because of proxy settings
>> +    cd "${TOPDIR}"
>>       sudo -E multistrap -a ${DISTRO_ARCH} -d "${IMAGE_ROOTFS}" -f "${WORKDIR}/multistrap.conf"
>>   
>>       # Configure root filesystem
>> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
>> index 3334789..2a45a1e 100644
>> --- a/meta/classes/base.bbclass
>> +++ b/meta/classes/base.bbclass
>> @@ -38,7 +38,7 @@ bbfatal() {
>>   }
>>   
>>   # Derived from openembedded-core/meta/classes/base.bbclass
>> -ISAR_IMPORTS += "os sys time"
>> +ISAR_IMPORTS += "os sys time isar"
>>   
>>   def isar_imports(d):
>>   	import sys
>> diff --git a/meta/lib/isar/__init__.py b/meta/lib/isar/__init__.py
>> new file mode 100644
>> index 0000000..e6aeeaa
>> --- /dev/null
>> +++ b/meta/lib/isar/__init__.py
>> @@ -0,0 +1,8 @@
>> +# Custom kernel recipe include
>> +#
>> +# This software is a part of ISAR.
>> +# Copyright (c) Siemens AG, 2018
>> +#
>> +# SPDX-License-Identifier: MIT
>> +
>> +from . import multistrap
>> diff --git a/meta/lib/isar/multistrap.py b/meta/lib/isar/multistrap.py
>> new file mode 100644
>> index 0000000..7022f91
>> --- /dev/null
>> +++ b/meta/lib/isar/multistrap.py
>> @@ -0,0 +1,80 @@
>> +# Custom kernel recipe include
>> +#
>> +# This software is a part of ISAR.
>> +# Copyright (c) Siemens AG, 2018
>> +#
>> +# SPDX-License-Identifier: MIT
>> +
>> +def generate_multistrap_config(d):
>> +    import configparser as cp
>> +    import io
>> +
>> +    workdir = d.getVar("WORKDIR", True) or ""
>> +    topdir = d.getVar("TOPDIR", True) or ""
>> +    workdir_rel = os.path.relpath(workdir, start=topdir)
>> +
>> +    msconf = cp.RawConfigParser()
>> +
>> +    # Set general multistrap configuration:
>> +    configscript = d.getVar("MULTISTRAP_CONFIG_SCRIPT", True) or ""
>> +    setupscript = d.getVar("MULTISTRAP_SETUP_SCRIPT", True) or "setup.sh"
>> +    hookdir = d.getVar("MULTISTRAP_HOOK_DIR", True) or "hooks_multistrap"
>> +    aptsources = (d.getVar("MULTISTRAP_APTSOURCES", True) or "").split()
>> +    bootstrap = (d.getVar("MULTISTRAP_BOOTSTRAP", True) or "").split()
>> +
>> +    msconf.add_section("General")
>> +    msconf.set("General", "noauth", "true")
>> +    msconf.set("General", "unpack", "true")
>> +    msconf.set("General", "ignorenativearch", "true")
>> +    msconf.set("General", "configscript", "./" +
>> +               os.path.join(workdir_rel, configscript))
>> +    msconf.set("General", "setupscript", "./" +
>> +               os.path.join(workdir_rel, setupscript))
>> +    msconf.set("General", "hookdir", "./" +
>> +               os.path.join(workdir_rel, hookdir))
>> +    msconf.set("General", "aptsources", " ".join(aptsources))
>> +    msconf.set("General", "bootstrap", " ".join(bootstrap))
>> +
>> +    # Setup upstream repositories in multistrap configuration:
>> +    apt_source = d.getVar("DISTRO_APT_SOURCE", True) or ""
>> +    apt_source_sec = d.getVar("DISTRO_APT_SOURCE_SEC", True) or ""
>> +    apt_suite = d.getVar("DISTRO_SUITE", True) or ""
>> +    apt_components = (d.getVar("DISTRO_COMPONENTS", True) or "").split()
>> +    deb_packages = (d.getVar("MULTISTRAP_PREINSTALL", True) or "").split()
>> +    isar_source = d.getVar("MULTISTRAP_ISAR_SOURCE", True) or ""
>> +    isar_packages = (d.getVar("MULTISTRAP_ISAR_INSTALL", True) or "").split()
>> +    isar_suite = d.getVar("MULTISTRAP_ISAR_SUITE", True) or "isar"
>> +
>> +    msconf.add_section("base")
>> +    msconf.set("base", "source", apt_source)
>> +    msconf.set("base", "suite", apt_suite)
>> +    msconf.set("base", "components", " ".join(apt_components))
>> +    msconf.set("base", "omitdebsrc", "true")
>> +    msconf.set("base", "packages", " ".join(deb_packages))
>> +
>> +    msconf.add_section("updates")
>> +    msconf.set("updates", "source", apt_source)
>> +    msconf.set("updates", "suite", apt_suite + "-updates")
>> +    msconf.set("updates", "components", " ".join(apt_components))
>> +    msconf.set("updates", "omitdebsrc", "true")
>> +
>> +    msconf.add_section("security")
>> +    msconf.set("security", "source", apt_source_sec)
>> +    msconf.set("security", "suite", apt_suite + "/updates")
>> +    msconf.set("security", "components", " ".join(apt_components))
>> +    msconf.set("security", "omitdebsrc", "true")
>> +
>> +    msconf.add_section("isar")
>> +    msconf.set("isar", "source", isar_source)
>> +    msconf.set("isar", "suite", isar_suite)
>> +    msconf.set("isar", "components", "main")
>> +    msconf.set("isar", "omitdebsrc", "true")
>> +    if isar_packages :
>> +        msconf.set("isar", "packages",
>> +                   " ".join(isar_packages))
>> +
>> +    ret = ""
>> +    with io.StringIO() as out:
>> +        msconf.write(out)
>> +        ret = out.getvalue()
>> +    return ret
>> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
>> index 570c0ad..59b4e7e 100644
>> --- a/meta/recipes-devtools/buildchroot/buildchroot.bb
>> +++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
>> @@ -9,8 +9,7 @@ LICENSE = "gpl-2.0"
>>   LIC_FILES_CHKSUM = "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
>>   
>>   FILESPATH =. "${LAYERDIR_core}/recipes-devtools/buildchroot/files:"
>> -SRC_URI = "file://multistrap.conf.in \
>> -           file://configscript.sh \
>> +SRC_URI = "file://configscript.sh \
>>              file://setup.sh \
>>              file://build.sh"
>>   PV = "1.0"
>> @@ -37,32 +36,25 @@ do_build[dirs] = "${BUILDCHROOT_DIR}/isar-apt \
>>                     ${BUILDCHROOT_DIR}/sys"
>>   do_build[depends] = "isar-apt:do_cache_config"
>>   
>> +MULTISTRAP_CONFIG_SCRIPT = "configscript.sh"
>> +MULTISTRAP_APTSOURCES = "isar ${DISTRO_MULTICONF_APTSOURCES}"
>> +MULTISTRAP_BOOTSTRAP = "${DISTRO_MULTICONF_BOOTSTRAP}"
>> +MULTISTRAP_PREINSTALL = "${BUILDCHROOT_PREINSTALL}"
>> +MULTISTRAP_ISAR_SOURCE = "file:///isar-apt"
>> +
>>   do_build() {
>>       E="${@ bb.utils.export_proxies(d)}"
>>   
>>       chmod +x "${WORKDIR}/setup.sh"
>>       chmod +x "${WORKDIR}/configscript.sh"
>>   
>> -    # Multistrap accepts only relative path in configuration files, so get it:
>> -    cd ${TOPDIR}
>> -    WORKDIR_REL=${@ os.path.relpath(d.getVar("WORKDIR", True))}
>> -
>> -    # Adjust multistrap config
>> -    sed -e 's|##BUILDCHROOT_PREINSTALL##|${BUILDCHROOT_PREINSTALL}|g' \
>> -        -e 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \
>> -        -e 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \
>> -        -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \
>> -        -e 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \
>> -        -e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \
>> -        -e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \
>> -        -e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/configscript.sh|g' \
>> -        -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \
>> -        -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
>> -           "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf"
>> +    echo "${@ isar.multistrap.generate_multistrap_config(d) }" > \
>> +         "${WORKDIR}/multistrap.conf"
>>   
>>       do_setup_mounts
>>   
>>       # Create root filesystem
>> +    cd "${TOPDIR}"
>>       sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f "${WORKDIR}/multistrap.conf"
>>   
>>       # Install package builder script
>> diff --git a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
>> deleted file mode 100644
>> index 480a4b8..0000000
>> --- a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
>> +++ /dev/null
>> @@ -1,37 +0,0 @@
>> -# This software is a part of ISAR.
>> -# Copyright (C) 2015-2017 ilbers GmbH
>> -
>> -[General]
>> -noauth=true
>> -unpack=true
>> -ignorenativearch=true
>> -bootstrap=##DISTRO_MULTICONF_BOOTSTRAP##
>> -aptsources=isar-apt ##DISTRO_MULTICONF_APTSOURCES##
>> -configscript=##CONFIG_SCRIPT##
>> -setupscript=##SETUP_SCRIPT##
>> -hookdir=##DIR_HOOKS##
>> -
>> -[base]
>> -source=##DISTRO_APT_SOURCE##
>> -suite=##DISTRO_SUITE##
>> -components=##DISTRO_COMPONENTS##
>> -packages=##BUILDCHROOT_PREINSTALL##
>> -omitdebsrc=true
>> -
>> -[updates]
>> -source=##DISTRO_APT_SOURCE##
>> -suite=##DISTRO_SUITE##-updates
>> -components=##DISTRO_COMPONENTS##
>> -omitdebsrc=true
>> -
>> -[security]
>> -source=##DISTRO_APT_SOURCE_SEC##
>> -suite=##DISTRO_SUITE##/updates
>> -components=##DISTRO_COMPONENTS##
>> -omitdebsrc=true
>> -
>> -[isar-apt]
>> -source=file:///isar-apt
>> -suite=isar
>> -components=main
>> -omitdebsrc=true
>>
> 
> This requires rebasing. I quickly tried, but the result broke the
> package build (didn't analyze any details).
> 
> 
> While this moves us in the right direction, it still leaves some
> duplications behind (setup.sh) - and the big question if we should
> invest any further in this horrible multistrap mess.
> 
> I've just enabled arm64, and it took a while to figure out that I was
> missing some lines in setup.sh. In contrast, qemu-debootstrap just
> worked (I used it to debug the multistrap output).
> 
> So, before we move on here: What all does multistrap currently give us,
> compared to standard debootstrap? I only see "multiple repositories" so
> far. But as Claudius is going to massage that part heavily anyway, I
> really wonder if it's worth building on the multistrap indirection any
> further.
> 
> How about something like this?
> 
> - [qemu-]debootstrap --variant=minbase ...
> - inject all additional apt sources, including user-provided ones
> - set pinnings and priority, including user-provided ones
> - apt-get dist-upgrade
> 

I'd say that multistrap is a tool which maps user-friendly config file 
to specific apt repositories configuration. If you look in the 
do_build() log for buildchroot, you could see what it exactly does:

1. Creating list of sources in rootfs/etc

2. Get packages.gz for all the sources:

apt-get  -o Apt::Architecture=armhf -o 
Dir::Etc::TrustedParts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/trusted.gpg.d 
-o 
Dir::Etc::Trusted=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/trusted.gpg.d/trusted.gpg 
-o Apt::Get::AllowUnauthenticated=true -o Apt::Get::Download-Only=true 
-o Apt::Install-Recommends=false -o 
Dir=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/ 
-o 
Dir::Etc=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/ 
-o 
Dir::Etc::Parts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/apt.conf.d/ 
-o 
Dir::Etc::PreferencesParts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/preferences.d/ 
-o APT::Default-Release=* -o 
Dir::State=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/lib/apt/ 
-o 
Dir::State::Status=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/lib/dpkg/status 
-o 
Dir::Cache=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/cache/apt/ 
update

3. Install packages:

apt-get -y  -o Apt::Architecture=armhf -o 
Dir::Etc::TrustedParts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/trusted.gpg.d 
-o 
Dir::Etc::Trusted=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/trusted.gpg.d/trusted.gpg 
-o Apt::Get::AllowUnauthenticated=true -o Apt::Get::Download-Only=true 
-o Apt::Install-Recommends=false -o 
Dir=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/ 
-o 
Dir::Etc=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/ 
-o 
Dir::Etc::Parts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/apt.conf.d/ 
-o 
Dir::Etc::PreferencesParts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/preferences.d/ 
-o APT::Default-Release=* -o 
Dir::State=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/lib/apt/ 
-o 
Dir::State::Status=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/lib/dpkg/status 
-o 
Dir::Cache=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/cache/apt/ 
install ${LIST_OF_PACKAGES}

 > That would kick out all the multistrap complications.

For me the plan above intersects with these multistrap steps, you have 
to re-implement them from the scratch. In this case I'd vote for the 
community-proved solution.

Alex

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [PATCH 2/2] centralize multistrap configuration generation
  2018-02-15  9:34     ` Alexander Smirnov
@ 2018-02-15 10:05       ` Claudius Heine
  2018-02-15 10:15         ` Jan Kiszka
  0 siblings, 1 reply; 23+ messages in thread
From: Claudius Heine @ 2018-02-15 10:05 UTC (permalink / raw)
  To: Alexander Smirnov, Jan Kiszka, isar-users, Baurzhan Ismagulov
  Cc: Claudius Heine

Hi Alex,

On 02/15/2018 10:34 AM, Alexander Smirnov wrote:
> 
> 
> On 02/15/2018 10:28 AM, Jan Kiszka wrote:
>> On 2018-02-14 14:15, [ext] claudius.heine.ext@siemens.com wrote:
>>> From: Claudius Heine <ch@denx.de>
>>>
>>> Previous there was redundant code between the buildchroot and
>>> isar-image-base recipe.
>>>
>>> This patch moves the generation of the multistrap configuration to a
>>> central python script that can be parametrized by both recipes.
>>>
>>> Signed-off-by: Claudius Heine <ch@denx.de>
>>> ---
>>>   .../recipes-core/images/files/multistrap.conf.in   | 38 ----------
>>>   meta-isar/recipes-core/images/isar-image-base.bb   | 33 ++++-----
>>>   meta/classes/base.bbclass                          |  2 +-
>>>   meta/lib/isar/__init__.py                          |  8 +++
>>>   meta/lib/isar/multistrap.py                        | 80 
>>> ++++++++++++++++++++++
>>>   meta/recipes-devtools/buildchroot/buildchroot.bb   | 28 +++-----
>>>   .../buildchroot/files/multistrap.conf.in           | 37 ----------
>>>   7 files changed, 111 insertions(+), 115 deletions(-)
>>>   delete mode 100644 
>>> meta-isar/recipes-core/images/files/multistrap.conf.in
>>>   create mode 100644 meta/lib/isar/__init__.py
>>>   create mode 100644 meta/lib/isar/multistrap.py
>>>   delete mode 100644 
>>> meta/recipes-devtools/buildchroot/files/multistrap.conf.in
>>>
>>> diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in 
>>> b/meta-isar/recipes-core/images/files/multistrap.conf.in
>>> deleted file mode 100644
>>> index 432b6af..0000000
>>> --- a/meta-isar/recipes-core/images/files/multistrap.conf.in
>>> +++ /dev/null
>>> @@ -1,38 +0,0 @@
>>> -# This software is a part of ISAR.
>>> -# Copyright (C) 2015-2017 ilbers GmbH
>>> -
>>> -[General]
>>> -noauth=true
>>> -unpack=true
>>> -ignorenativearch=true
>>> -bootstrap=##DISTRO_MULTICONF_BOOTSTRAP## Isar
>>> -aptsources=##DISTRO_MULTICONF_APTSOURCES##
>>> -configscript=##CONFIG_SCRIPT##
>>> -setupscript=##SETUP_SCRIPT##
>>> -hookdir=##DIR_HOOKS##
>>> -
>>> -[base]
>>> -source=##DISTRO_APT_SOURCE##
>>> -suite=##DISTRO_SUITE##
>>> -components=##DISTRO_COMPONENTS##
>>> -packages=##IMAGE_PREINSTALL##
>>> -omitdebsrc=true
>>> -
>>> -[updates]
>>> -source=##DISTRO_APT_SOURCE##
>>> -suite=##DISTRO_SUITE##-updates
>>> -components=##DISTRO_COMPONENTS##
>>> -omitdebsrc=true
>>> -
>>> -[security]
>>> -source=##DISTRO_APT_SOURCE_SEC##
>>> -suite=##DISTRO_SUITE##/updates
>>> -components=##DISTRO_COMPONENTS##
>>> -omitdebsrc=true
>>> -
>>> -[Isar]
>>> -packages=##IMAGE_INSTALL##
>>> -source=##DEPLOY_DIR_APT##
>>> -suite=##ISAR_DISTRO_SUITE##
>>> -components=main
>>> -omitdebsrc=true
>>> diff --git a/meta-isar/recipes-core/images/isar-image-base.bb 
>>> b/meta-isar/recipes-core/images/isar-image-base.bb
>>> index 8ddbabb..9913977 100644
>>> --- a/meta-isar/recipes-core/images/isar-image-base.bb
>>> +++ b/meta-isar/recipes-core/images/isar-image-base.bb
>>> @@ -9,8 +9,7 @@ LICENSE = "gpl-2.0"
>>>   LIC_FILES_CHKSUM = 
>>> "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe" 
>>>
>>>   FILESPATH =. "${LAYERDIR_isar}/recipes-core/images/files:"
>>> -SRC_URI = "file://multistrap.conf.in \
>>> -       file://${DISTRO_CONFIG_SCRIPT} \
>>> +SRC_URI = "file://${DISTRO_CONFIG_SCRIPT} \
>>>          file://setup.sh \
>>>          file://download_dev-random"
>>> @@ -28,6 +27,14 @@ WORKDIR = 
>>> "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
>>>   do_rootfs[dirs] = "${WORKDIR}/hooks_multistrap"
>>> +MULTISTRAP_CONFIG_SCRIPT = "${DISTRO_CONFIG_SCRIPT}"
>>> +MULTISTRAP_APTSOURCES = "${DISTRO_MULTICONF_APTSOURCES}"
>>> +MULTISTRAP_BOOTSTRAP = "${DISTRO_MULTICONF_BOOTSTRAP} isar"
>>> +MULTISTRAP_PREINSTALL = "${IMAGE_PREINSTALL}"
>>> +MULTISTRAP_ISAR_SOURCE = "copy:///${DEPLOY_DIR_APT}/${DISTRO}"
>>> +MULTISTRAP_ISAR_INSTALL = "${IMAGE_INSTALL}"
>>> +MULTISTRAP_ISAR_SUITE = "${DEBDISTRONAME}"
>>> +
>>>   do_rootfs() {
>>>       E="${@ bb.utils.export_proxies(d)}"
>>> @@ -35,25 +42,8 @@ do_rootfs() {
>>>       chmod +x "${WORKDIR}/setup.sh"
>>>       install -m 755 "${WORKDIR}/download_dev-random" 
>>> "${WORKDIR}/hooks_multistrap/"
>>> -    # Multistrap accepts only relative path in configuration files, 
>>> so get it:
>>> -    cd ${TOPDIR}
>>> -    WORKDIR_REL=${@ os.path.relpath(d.getVar("WORKDIR", True))}
>>> -
>>> -    # Adjust multistrap config
>>> -    sed -e 's|##IMAGE_PREINSTALL##|${IMAGE_PREINSTALL}|g' \
>>> -        -e 
>>> 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \
>>> -        -e 
>>> 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \
>>> -        -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \
>>> -        -e 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \
>>> -        -e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \
>>> -        -e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \
>>> -        -e 
>>> 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/${DISTRO_CONFIG_SCRIPT}|g' \
>>> -        -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \
>>> -        -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
>>> -        -e 's|##IMAGE_INSTALL##|${IMAGE_INSTALL}|g' \
>>> -        -e 
>>> 's|##DEPLOY_DIR_APT##|copy:///${DEPLOY_DIR_APT}/${DISTRO}|g' \
>>> -        -e 's|##ISAR_DISTRO_SUITE##|${DEBDISTRONAME}|g' \
>>> -           "${WORKDIR}/multistrap.conf.in" > 
>>> "${WORKDIR}/multistrap.conf"
>>> +    echo "${@ isar.multistrap.generate_multistrap_config(d) }" > \
>>> +         "${WORKDIR}/multistrap.conf"
>>>       # Do not use bitbake flag [dirs] here because this folder 
>>> should have
>>>       # specific ownership.
>>> @@ -62,6 +52,7 @@ do_rootfs() {
>>>       # Create root filesystem. We must use sudo -E here to preserve 
>>> the environment
>>>       # because of proxy settings
>>> +    cd "${TOPDIR}"
>>>       sudo -E multistrap -a ${DISTRO_ARCH} -d "${IMAGE_ROOTFS}" -f 
>>> "${WORKDIR}/multistrap.conf"
>>>       # Configure root filesystem
>>> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
>>> index 3334789..2a45a1e 100644
>>> --- a/meta/classes/base.bbclass
>>> +++ b/meta/classes/base.bbclass
>>> @@ -38,7 +38,7 @@ bbfatal() {
>>>   }
>>>   # Derived from openembedded-core/meta/classes/base.bbclass
>>> -ISAR_IMPORTS += "os sys time"
>>> +ISAR_IMPORTS += "os sys time isar"
>>>   def isar_imports(d):
>>>       import sys
>>> diff --git a/meta/lib/isar/__init__.py b/meta/lib/isar/__init__.py
>>> new file mode 100644
>>> index 0000000..e6aeeaa
>>> --- /dev/null
>>> +++ b/meta/lib/isar/__init__.py
>>> @@ -0,0 +1,8 @@
>>> +# Custom kernel recipe include
>>> +#
>>> +# This software is a part of ISAR.
>>> +# Copyright (c) Siemens AG, 2018
>>> +#
>>> +# SPDX-License-Identifier: MIT
>>> +
>>> +from . import multistrap
>>> diff --git a/meta/lib/isar/multistrap.py b/meta/lib/isar/multistrap.py
>>> new file mode 100644
>>> index 0000000..7022f91
>>> --- /dev/null
>>> +++ b/meta/lib/isar/multistrap.py
>>> @@ -0,0 +1,80 @@
>>> +# Custom kernel recipe include
>>> +#
>>> +# This software is a part of ISAR.
>>> +# Copyright (c) Siemens AG, 2018
>>> +#
>>> +# SPDX-License-Identifier: MIT
>>> +
>>> +def generate_multistrap_config(d):
>>> +    import configparser as cp
>>> +    import io
>>> +
>>> +    workdir = d.getVar("WORKDIR", True) or ""
>>> +    topdir = d.getVar("TOPDIR", True) or ""
>>> +    workdir_rel = os.path.relpath(workdir, start=topdir)
>>> +
>>> +    msconf = cp.RawConfigParser()
>>> +
>>> +    # Set general multistrap configuration:
>>> +    configscript = d.getVar("MULTISTRAP_CONFIG_SCRIPT", True) or ""
>>> +    setupscript = d.getVar("MULTISTRAP_SETUP_SCRIPT", True) or 
>>> "setup.sh"
>>> +    hookdir = d.getVar("MULTISTRAP_HOOK_DIR", True) or 
>>> "hooks_multistrap"
>>> +    aptsources = (d.getVar("MULTISTRAP_APTSOURCES", True) or 
>>> "").split()
>>> +    bootstrap = (d.getVar("MULTISTRAP_BOOTSTRAP", True) or "").split()
>>> +
>>> +    msconf.add_section("General")
>>> +    msconf.set("General", "noauth", "true")
>>> +    msconf.set("General", "unpack", "true")
>>> +    msconf.set("General", "ignorenativearch", "true")
>>> +    msconf.set("General", "configscript", "./" +
>>> +               os.path.join(workdir_rel, configscript))
>>> +    msconf.set("General", "setupscript", "./" +
>>> +               os.path.join(workdir_rel, setupscript))
>>> +    msconf.set("General", "hookdir", "./" +
>>> +               os.path.join(workdir_rel, hookdir))
>>> +    msconf.set("General", "aptsources", " ".join(aptsources))
>>> +    msconf.set("General", "bootstrap", " ".join(bootstrap))
>>> +
>>> +    # Setup upstream repositories in multistrap configuration:
>>> +    apt_source = d.getVar("DISTRO_APT_SOURCE", True) or ""
>>> +    apt_source_sec = d.getVar("DISTRO_APT_SOURCE_SEC", True) or ""
>>> +    apt_suite = d.getVar("DISTRO_SUITE", True) or ""
>>> +    apt_components = (d.getVar("DISTRO_COMPONENTS", True) or 
>>> "").split()
>>> +    deb_packages = (d.getVar("MULTISTRAP_PREINSTALL", True) or 
>>> "").split()
>>> +    isar_source = d.getVar("MULTISTRAP_ISAR_SOURCE", True) or ""
>>> +    isar_packages = (d.getVar("MULTISTRAP_ISAR_INSTALL", True) or 
>>> "").split()
>>> +    isar_suite = d.getVar("MULTISTRAP_ISAR_SUITE", True) or "isar"
>>> +
>>> +    msconf.add_section("base")
>>> +    msconf.set("base", "source", apt_source)
>>> +    msconf.set("base", "suite", apt_suite)
>>> +    msconf.set("base", "components", " ".join(apt_components))
>>> +    msconf.set("base", "omitdebsrc", "true")
>>> +    msconf.set("base", "packages", " ".join(deb_packages))
>>> +
>>> +    msconf.add_section("updates")
>>> +    msconf.set("updates", "source", apt_source)
>>> +    msconf.set("updates", "suite", apt_suite + "-updates")
>>> +    msconf.set("updates", "components", " ".join(apt_components))
>>> +    msconf.set("updates", "omitdebsrc", "true")
>>> +
>>> +    msconf.add_section("security")
>>> +    msconf.set("security", "source", apt_source_sec)
>>> +    msconf.set("security", "suite", apt_suite + "/updates")
>>> +    msconf.set("security", "components", " ".join(apt_components))
>>> +    msconf.set("security", "omitdebsrc", "true")
>>> +
>>> +    msconf.add_section("isar")
>>> +    msconf.set("isar", "source", isar_source)
>>> +    msconf.set("isar", "suite", isar_suite)
>>> +    msconf.set("isar", "components", "main")
>>> +    msconf.set("isar", "omitdebsrc", "true")
>>> +    if isar_packages :
>>> +        msconf.set("isar", "packages",
>>> +                   " ".join(isar_packages))
>>> +
>>> +    ret = ""
>>> +    with io.StringIO() as out:
>>> +        msconf.write(out)
>>> +        ret = out.getvalue()
>>> +    return ret
>>> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb 
>>> b/meta/recipes-devtools/buildchroot/buildchroot.bb
>>> index 570c0ad..59b4e7e 100644
>>> --- a/meta/recipes-devtools/buildchroot/buildchroot.bb
>>> +++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
>>> @@ -9,8 +9,7 @@ LICENSE = "gpl-2.0"
>>>   LIC_FILES_CHKSUM = 
>>> "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe" 
>>>
>>>   FILESPATH =. "${LAYERDIR_core}/recipes-devtools/buildchroot/files:"
>>> -SRC_URI = "file://multistrap.conf.in \
>>> -           file://configscript.sh \
>>> +SRC_URI = "file://configscript.sh \
>>>              file://setup.sh \
>>>              file://build.sh"
>>>   PV = "1.0"
>>> @@ -37,32 +36,25 @@ do_build[dirs] = "${BUILDCHROOT_DIR}/isar-apt \
>>>                     ${BUILDCHROOT_DIR}/sys"
>>>   do_build[depends] = "isar-apt:do_cache_config"
>>> +MULTISTRAP_CONFIG_SCRIPT = "configscript.sh"
>>> +MULTISTRAP_APTSOURCES = "isar ${DISTRO_MULTICONF_APTSOURCES}"
>>> +MULTISTRAP_BOOTSTRAP = "${DISTRO_MULTICONF_BOOTSTRAP}"
>>> +MULTISTRAP_PREINSTALL = "${BUILDCHROOT_PREINSTALL}"
>>> +MULTISTRAP_ISAR_SOURCE = "file:///isar-apt"
>>> +
>>>   do_build() {
>>>       E="${@ bb.utils.export_proxies(d)}"
>>>       chmod +x "${WORKDIR}/setup.sh"
>>>       chmod +x "${WORKDIR}/configscript.sh"
>>> -    # Multistrap accepts only relative path in configuration files, 
>>> so get it:
>>> -    cd ${TOPDIR}
>>> -    WORKDIR_REL=${@ os.path.relpath(d.getVar("WORKDIR", True))}
>>> -
>>> -    # Adjust multistrap config
>>> -    sed -e 's|##BUILDCHROOT_PREINSTALL##|${BUILDCHROOT_PREINSTALL}|g' \
>>> -        -e 
>>> 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \
>>> -        -e 
>>> 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \
>>> -        -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \
>>> -        -e 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \
>>> -        -e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \
>>> -        -e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \
>>> -        -e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/configscript.sh|g' \
>>> -        -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \
>>> -        -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
>>> -           "${WORKDIR}/multistrap.conf.in" > 
>>> "${WORKDIR}/multistrap.conf"
>>> +    echo "${@ isar.multistrap.generate_multistrap_config(d) }" > \
>>> +         "${WORKDIR}/multistrap.conf"
>>>       do_setup_mounts
>>>       # Create root filesystem
>>> +    cd "${TOPDIR}"
>>>       sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f 
>>> "${WORKDIR}/multistrap.conf"
>>>       # Install package builder script
>>> diff --git 
>>> a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in 
>>> b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
>>> deleted file mode 100644
>>> index 480a4b8..0000000
>>> --- a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
>>> +++ /dev/null
>>> @@ -1,37 +0,0 @@
>>> -# This software is a part of ISAR.
>>> -# Copyright (C) 2015-2017 ilbers GmbH
>>> -
>>> -[General]
>>> -noauth=true
>>> -unpack=true
>>> -ignorenativearch=true
>>> -bootstrap=##DISTRO_MULTICONF_BOOTSTRAP##
>>> -aptsources=isar-apt ##DISTRO_MULTICONF_APTSOURCES##
>>> -configscript=##CONFIG_SCRIPT##
>>> -setupscript=##SETUP_SCRIPT##
>>> -hookdir=##DIR_HOOKS##
>>> -
>>> -[base]
>>> -source=##DISTRO_APT_SOURCE##
>>> -suite=##DISTRO_SUITE##
>>> -components=##DISTRO_COMPONENTS##
>>> -packages=##BUILDCHROOT_PREINSTALL##
>>> -omitdebsrc=true
>>> -
>>> -[updates]
>>> -source=##DISTRO_APT_SOURCE##
>>> -suite=##DISTRO_SUITE##-updates
>>> -components=##DISTRO_COMPONENTS##
>>> -omitdebsrc=true
>>> -
>>> -[security]
>>> -source=##DISTRO_APT_SOURCE_SEC##
>>> -suite=##DISTRO_SUITE##/updates
>>> -components=##DISTRO_COMPONENTS##
>>> -omitdebsrc=true
>>> -
>>> -[isar-apt]
>>> -source=file:///isar-apt
>>> -suite=isar
>>> -components=main
>>> -omitdebsrc=true
>>>
>>
>> This requires rebasing. I quickly tried, but the result broke the
>> package build (didn't analyze any details).
>>
>>
>> While this moves us in the right direction, it still leaves some
>> duplications behind (setup.sh) - and the big question if we should
>> invest any further in this horrible multistrap mess.
>>
>> I've just enabled arm64, and it took a while to figure out that I was
>> missing some lines in setup.sh. In contrast, qemu-debootstrap just
>> worked (I used it to debug the multistrap output).
>>
>> So, before we move on here: What all does multistrap currently give us,
>> compared to standard debootstrap? I only see "multiple repositories" so
>> far. But as Claudius is going to massage that part heavily anyway, I
>> really wonder if it's worth building on the multistrap indirection any
>> further.
>>
>> How about something like this?
>>
>> - [qemu-]debootstrap --variant=minbase ...
>> - inject all additional apt sources, including user-provided ones
>> - set pinnings and priority, including user-provided ones
>> - apt-get dist-upgrade
>>
> 
> I'd say that multistrap is a tool which maps user-friendly config file 
> to specific apt repositories configuration. If you look in the 
> do_build() log for buildchroot, you could see what it exactly does:
> 
> 1. Creating list of sources in rootfs/etc
> 
> 2. Get packages.gz for all the sources:
> 
> apt-get  -o Apt::Architecture=armhf -o 
> Dir::Etc::TrustedParts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/trusted.gpg.d 
> -o 
> Dir::Etc::Trusted=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/trusted.gpg.d/trusted.gpg 
> -o Apt::Get::AllowUnauthenticated=true -o Apt::Get::Download-Only=true 
> -o Apt::Install-Recommends=false -o 
> Dir=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/ 
> -o 
> Dir::Etc=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/ 
> -o 
> Dir::Etc::Parts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/apt.conf.d/ 
> -o 
> Dir::Etc::PreferencesParts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/preferences.d/ 
> -o APT::Default-Release=* -o 
> Dir::State=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/lib/apt/ 
> -o 
> Dir::State::Status=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/lib/dpkg/status 
> -o 
> Dir::Cache=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/cache/apt/ 
> update
> 
> 3. Install packages:
> 
> apt-get -y  -o Apt::Architecture=armhf -o 
> Dir::Etc::TrustedParts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/trusted.gpg.d 
> -o 
> Dir::Etc::Trusted=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/trusted.gpg.d/trusted.gpg 
> -o Apt::Get::AllowUnauthenticated=true -o Apt::Get::Download-Only=true 
> -o Apt::Install-Recommends=false -o 
> Dir=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/ 
> -o 
> Dir::Etc=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/ 
> -o 
> Dir::Etc::Parts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/apt.conf.d/ 
> -o 
> Dir::Etc::PreferencesParts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/preferences.d/ 
> -o APT::Default-Release=* -o 
> Dir::State=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/lib/apt/ 
> -o 
> Dir::State::Status=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/lib/dpkg/status 
> -o 
> Dir::Cache=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/cache/apt/ 
> install ${LIST_OF_PACKAGES}
> 
>  > That would kick out all the multistrap complications.
> 
> For me the plan above intersects with these multistrap steps, you have 
> to re-implement them from the scratch. In this case I'd vote for the 
> community-proved solution.

For me that is a reason to use debootstrap instead of multistrap, since 
multistrap is not community-supported. If we implement some of those 
feature, we might as well do it in debootstrap inself. Just take a look 
at their TODO list:

   Features:
     ++ second stage via chroot debootstrap/debootstrap
     ++ debootstrap/deb file to record deb destinations/information

     -- configuration file
         -- versus command line
         -- support for sources (vs mirrors)
         -- faux-pinning for packages

https://anonscm.debian.org/cgit/d-i/debootstrap.git/tree/TODO?id=5d42162bc0fe266b280d81445271e3f56b9560a8

That shows to me that upstream might be interested in some of these 
improvements we would need to implement.

Cheers,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [PATCH 2/2] centralize multistrap configuration generation
  2018-02-15 10:05       ` Claudius Heine
@ 2018-02-15 10:15         ` Jan Kiszka
  2018-02-15 10:29           ` Alexander Smirnov
  2018-02-15 11:27           ` Baurzhan Ismagulov
  0 siblings, 2 replies; 23+ messages in thread
From: Jan Kiszka @ 2018-02-15 10:15 UTC (permalink / raw)
  To: Claudius Heine, Alexander Smirnov, isar-users, Baurzhan Ismagulov
  Cc: Claudius Heine

On 2018-02-15 11:05, Claudius Heine wrote:
>>> While this moves us in the right direction, it still leaves some
>>> duplications behind (setup.sh) - and the big question if we should
>>> invest any further in this horrible multistrap mess.
>>>
>>> I've just enabled arm64, and it took a while to figure out that I was
>>> missing some lines in setup.sh. In contrast, qemu-debootstrap just
>>> worked (I used it to debug the multistrap output).
>>>
>>> So, before we move on here: What all does multistrap currently give us,
>>> compared to standard debootstrap? I only see "multiple repositories" so
>>> far. But as Claudius is going to massage that part heavily anyway, I
>>> really wonder if it's worth building on the multistrap indirection any
>>> further.
>>>
>>> How about something like this?
>>>
>>> - [qemu-]debootstrap --variant=minbase ...
>>> - inject all additional apt sources, including user-provided ones
>>> - set pinnings and priority, including user-provided ones
>>> - apt-get dist-upgrade
>>>
>>
>> I'd say that multistrap is a tool which maps user-friendly config file
>> to specific apt repositories configuration. If you look in the
>> do_build() log for buildchroot, you could see what it exactly does:
>>
>> 1. Creating list of sources in rootfs/etc
>>
>> 2. Get packages.gz for all the sources:
>>
>> apt-get  -o Apt::Architecture=armhf -o
>> Dir::Etc::TrustedParts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/trusted.gpg.d
>> -o
>> Dir::Etc::Trusted=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/trusted.gpg.d/trusted.gpg
>> -o Apt::Get::AllowUnauthenticated=true -o Apt::Get::Download-Only=true
>> -o Apt::Install-Recommends=false -o
>> Dir=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/
>> -o
>> Dir::Etc=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/
>> -o
>> Dir::Etc::Parts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/apt.conf.d/
>> -o
>> Dir::Etc::PreferencesParts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/preferences.d/
>> -o APT::Default-Release=* -o
>> Dir::State=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/lib/apt/
>> -o
>> Dir::State::Status=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/lib/dpkg/status
>> -o
>> Dir::Cache=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/cache/apt/
>> update
>>
>> 3. Install packages:
>>
>> apt-get -y  -o Apt::Architecture=armhf -o
>> Dir::Etc::TrustedParts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/trusted.gpg.d
>> -o
>> Dir::Etc::Trusted=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/trusted.gpg.d/trusted.gpg
>> -o Apt::Get::AllowUnauthenticated=true -o Apt::Get::Download-Only=true
>> -o Apt::Install-Recommends=false -o
>> Dir=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/
>> -o
>> Dir::Etc=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/
>> -o
>> Dir::Etc::Parts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/apt.conf.d/
>> -o
>> Dir::Etc::PreferencesParts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/preferences.d/
>> -o APT::Default-Release=* -o
>> Dir::State=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/lib/apt/
>> -o
>> Dir::State::Status=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/lib/dpkg/status
>> -o
>> Dir::Cache=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/cache/apt/
>> install ${LIST_OF_PACKAGES}
>>
>>  > That would kick out all the multistrap complications.
>>
>> For me the plan above intersects with these multistrap steps, you have
>> to re-implement them from the scratch. In this case I'd vote for the
>> community-proved solution.
> 
> For me that is a reason to use debootstrap instead of multistrap, since
> multistrap is not community-supported. If we implement some of those
> feature, we might as well do it in debootstrap inself. Just take a look
> at their TODO list:
> 
>   Features:
>     ++ second stage via chroot debootstrap/debootstrap
>     ++ debootstrap/deb file to record deb destinations/information
> 
>     -- configuration file
>         -- versus command line
>         -- support for sources (vs mirrors)
>         -- faux-pinning for packages
> 
> https://anonscm.debian.org/cgit/d-i/debootstrap.git/tree/TODO?id=5d42162bc0fe266b280d81445271e3f56b9560a8
> 
> 
> That shows to me that upstream might be interested in some of these
> improvements we would need to implement.

Exactly. We should really use the community-proven, continuously tested
official approach of Debian, in that in a way that it foresees for it.

Let's bury multistrap. Its a dead horse we rode to long now.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [PATCH 2/2] centralize multistrap configuration generation
  2018-02-15 10:15         ` Jan Kiszka
@ 2018-02-15 10:29           ` Alexander Smirnov
  2018-02-15 11:27           ` Baurzhan Ismagulov
  1 sibling, 0 replies; 23+ messages in thread
From: Alexander Smirnov @ 2018-02-15 10:29 UTC (permalink / raw)
  To: Jan Kiszka, isar-users, Baurzhan Ismagulov; +Cc: Claudius Heine

On 02/15/2018 01:15 PM, Jan Kiszka wrote:
> On 2018-02-15 11:05, Claudius Heine wrote:
>>>> While this moves us in the right direction, it still leaves some
>>>> duplications behind (setup.sh) - and the big question if we should
>>>> invest any further in this horrible multistrap mess.
>>>>
>>>> I've just enabled arm64, and it took a while to figure out that I was
>>>> missing some lines in setup.sh. In contrast, qemu-debootstrap just
>>>> worked (I used it to debug the multistrap output).
>>>>
>>>> So, before we move on here: What all does multistrap currently give us,
>>>> compared to standard debootstrap? I only see "multiple repositories" so
>>>> far. But as Claudius is going to massage that part heavily anyway, I
>>>> really wonder if it's worth building on the multistrap indirection any
>>>> further.
>>>>
>>>> How about something like this?
>>>>
>>>> - [qemu-]debootstrap --variant=minbase ...
>>>> - inject all additional apt sources, including user-provided ones
>>>> - set pinnings and priority, including user-provided ones
>>>> - apt-get dist-upgrade
>>>>
>>>
>>> I'd say that multistrap is a tool which maps user-friendly config file
>>> to specific apt repositories configuration. If you look in the
>>> do_build() log for buildchroot, you could see what it exactly does:
>>>
>>> 1. Creating list of sources in rootfs/etc
>>>
>>> 2. Get packages.gz for all the sources:
>>>
>>> apt-get  -o Apt::Architecture=armhf -o
>>> Dir::Etc::TrustedParts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/trusted.gpg.d
>>> -o
>>> Dir::Etc::Trusted=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/trusted.gpg.d/trusted.gpg
>>> -o Apt::Get::AllowUnauthenticated=true -o Apt::Get::Download-Only=true
>>> -o Apt::Install-Recommends=false -o
>>> Dir=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/
>>> -o
>>> Dir::Etc=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/
>>> -o
>>> Dir::Etc::Parts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/apt.conf.d/
>>> -o
>>> Dir::Etc::PreferencesParts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/preferences.d/
>>> -o APT::Default-Release=* -o
>>> Dir::State=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/lib/apt/
>>> -o
>>> Dir::State::Status=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/lib/dpkg/status
>>> -o
>>> Dir::Cache=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/cache/apt/
>>> update
>>>
>>> 3. Install packages:
>>>
>>> apt-get -y  -o Apt::Architecture=armhf -o
>>> Dir::Etc::TrustedParts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/trusted.gpg.d
>>> -o
>>> Dir::Etc::Trusted=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/trusted.gpg.d/trusted.gpg
>>> -o Apt::Get::AllowUnauthenticated=true -o Apt::Get::Download-Only=true
>>> -o Apt::Install-Recommends=false -o
>>> Dir=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/
>>> -o
>>> Dir::Etc=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/
>>> -o
>>> Dir::Etc::Parts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/apt.conf.d/
>>> -o
>>> Dir::Etc::PreferencesParts=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/etc/apt/preferences.d/
>>> -o APT::Default-Release=* -o
>>> Dir::State=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/lib/apt/
>>> -o
>>> Dir::State::Status=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/lib/dpkg/status
>>> -o
>>> Dir::Cache=/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/buildchroot/rootfs/var/cache/apt/
>>> install ${LIST_OF_PACKAGES}
>>>
>>>   > That would kick out all the multistrap complications.
>>>
>>> For me the plan above intersects with these multistrap steps, you have
>>> to re-implement them from the scratch. In this case I'd vote for the
>>> community-proved solution.
>>
>> For me that is a reason to use debootstrap instead of multistrap, since
>> multistrap is not community-supported. If we implement some of those
>> feature, we might as well do it in debootstrap inself. Just take a look
>> at their TODO list:
>>
>>    Features:
>>      ++ second stage via chroot debootstrap/debootstrap
>>      ++ debootstrap/deb file to record deb destinations/information
>>
>>      -- configuration file
>>          -- versus command line
>>          -- support for sources (vs mirrors)
>>          -- faux-pinning for packages
>>
>> https://anonscm.debian.org/cgit/d-i/debootstrap.git/tree/TODO?id=5d42162bc0fe266b280d81445271e3f56b9560a8
>>
>>
>> That shows to me that upstream might be interested in some of these
>> improvements we would need to implement.
> 
> Exactly. We should really use the community-proven, continuously tested
> official approach of Debian, in that in a way that it foresees for it.
> 
> Let's bury multistrap. Its a dead horse we rode to long now.
> 

Nobody is against official tools. :-) This question is open for a quite 
long time, and the conclusion for now: there is *no* tool that fits Isar 
requirements for 100%. So in general I only want upstream-first way of 
tools modification for Isar.

Alex

Alex


^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [PATCH 2/2] centralize multistrap configuration generation
  2018-02-15 10:15         ` Jan Kiszka
  2018-02-15 10:29           ` Alexander Smirnov
@ 2018-02-15 11:27           ` Baurzhan Ismagulov
  2018-02-15 13:17             ` Claudius Heine
  2018-02-15 13:18             ` Jan Kiszka
  1 sibling, 2 replies; 23+ messages in thread
From: Baurzhan Ismagulov @ 2018-02-15 11:27 UTC (permalink / raw)
  To: isar-users

On Thu, Feb 15, 2018 at 11:15:02AM +0100, Jan Kiszka wrote:
> > That shows to me that upstream might be interested in some of these
> > improvements we would need to implement.
> 
> Exactly. We should really use the community-proven, continuously tested
> official approach of Debian, in that in a way that it foresees for it.
> 
> Let's bury multistrap. Its a dead horse we rode to long now.

First off, thanks for cutting the many lines of code not being referred to :) .

Regarding multistrap, both approaches have advantages and disadvantages.
Moreover, there are other implementations, such as cdebootstrap.

Multistrap is currently useful in the following ways:

1. Generate the list of packages to install using several apt repos.

2. Provide the package paths to download from apt repos.

3. Generate the base system from several apt repos.

Unfortunately, all three seem to be unique among the three candidates.

(1) and (2) are necessary for reproducibility. My preferred solution would be
to use an apt library that would do that for us (e.g., extract it from apt,
extend python-debian, or look whether aptly provides that). Moving to
*debootstrap would require solving those in a community-unmaintained way for
some time.

(3) is useful for people with customizations cutting off significant dependency
chains from Debian (like SLIND did with Perl dependencies). Moving to
*debootstrap would mean first installing those, then apt-getting our layers
above the base system, then removing the packages that are no more required.

This was the main reason for Neil Williams to create multistrap. After trying
to wrap debootstrap for some time, he ended up with problems not solvable in
that way (not sure which, maybe some kinds of circular dependencies or
dependencies on exact package versions). So, it would be wise to consider his
experience here.

That said, on DebConf17, he told us that that use case is not interesting for
him any more, advised to use debootstrap, and said that he doesn't support
multistrap any more.

Regarding maintaining multistrap for some time: I personally don't like Perl.
That said, Alex looked at that recently; it's some reasonable wrapping around
apt-get. If the problem in the end is solved with apt-get, we should possibly
evaluate that, too.

Regarding debootstrap, the problem is that it is implemented in shell, which is
horrible for any serious improvements. Thus the question about cdebootstrap.

All in all, I'd like to see a solution that would offer serious benefits while
not introducing new problems. Currently, multistrap is an internal issue not
directly affecting user-visible features. There are many features we can do for
far better Debian support. Changing one bit with another, not better one
wouldn't change the big picture. So, I'd rather use this opportunity to start a
bigger effort of defining the requirements, choosing the tools to merge with,
provide a working PoC, and start working with Debian on reusability of those
tools in Isar. IMHO, *debootstrap alone wouldn't bring major architectural
improvements.

With kind regards,
Baurzhan.

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [PATCH 2/2] centralize multistrap configuration generation
  2018-02-14 13:15 ` [PATCH 2/2] centralize multistrap configuration generation claudius.heine.ext
  2018-02-15  7:28   ` Jan Kiszka
@ 2018-02-15 11:42   ` Baurzhan Ismagulov
  2018-02-15 12:08     ` Claudius Heine
  1 sibling, 1 reply; 23+ messages in thread
From: Baurzhan Ismagulov @ 2018-02-15 11:42 UTC (permalink / raw)
  To: isar-users

On Wed, Feb 14, 2018 at 02:15:09PM +0100, claudius.heine.ext@siemens.com wrote:
> From: Claudius Heine <ch@denx.de>
> 
> Previous there was redundant code between the buildchroot and
> isar-image-base recipe.
> 
> This patch moves the generation of the multistrap configuration to a
> central python script that can be parametrized by both recipes.
> 
> Signed-off-by: Claudius Heine <ch@denx.de>
> ---
>  .../recipes-core/images/files/multistrap.conf.in   | 38 ----------
>  meta/lib/isar/multistrap.py                        | 80 ++++++++++++++++++++++
>  .../buildchroot/files/multistrap.conf.in           | 37 ----------

Could you please post two final multistrap.confs you'd like to get with your
changes?

I'd like to understand the reasons for moving from templates to a magic
generator.

In my experience, magic is usually poorly documented (if at all) and very
ununderstandable for users. Simple change of a line requires locating the
generator, understanding what it does (they tend to be hairy with tons of
conditions) and patching the upstream [Isar] code. Templates have their
disadvantages, but are straightforward for users. In general, I'd like to keep
the current UX and stick with templates, which is possible most of the time.

With kind regards,
Baurzhan.

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [PATCH 2/2] centralize multistrap configuration generation
  2018-02-15 11:42   ` Baurzhan Ismagulov
@ 2018-02-15 12:08     ` Claudius Heine
  2018-02-15 12:37       ` Alexander Smirnov
  0 siblings, 1 reply; 23+ messages in thread
From: Claudius Heine @ 2018-02-15 12:08 UTC (permalink / raw)
  To: isar-users, Baurzhan Ismagulov

Hi Baurzhan,

On 02/15/2018 12:42 PM, Baurzhan Ismagulov wrote:
> On Wed, Feb 14, 2018 at 02:15:09PM +0100, claudius.heine.ext@siemens.com wrote:
>> From: Claudius Heine <ch@denx.de>
>>
>> Previous there was redundant code between the buildchroot and
>> isar-image-base recipe.
>>
>> This patch moves the generation of the multistrap configuration to a
>> central python script that can be parametrized by both recipes.
>>
>> Signed-off-by: Claudius Heine <ch@denx.de>
>> ---
>>   .../recipes-core/images/files/multistrap.conf.in   | 38 ----------
>>   meta/lib/isar/multistrap.py                        | 80 ++++++++++++++++++++++
>>   .../buildchroot/files/multistrap.conf.in           | 37 ----------
> 
> Could you please post two final multistrap.confs you'd like to get with your
> changes?

'like to get'? I don't want to get any other multistrap.conf than 
before. This is not a change of the multistrap.conf files, but just how 
they are generated. The python configparser created ini-file format only 
puts spaces around the '=' characters, but the resulting deserialized 
objects is the same.

Ok, that is not 100% true, I changed the name of the isar repository 
section. Now its called 'isar' in the image (previously 'Isar') and 
buildchroot (previously 'isar-apt') multiconfig. But that is it. It has 
no different effect outside of multistrap.

> I'd like to understand the reasons for moving from templates to a magic
> generator.
> 
> In my experience, magic is usually poorly documented (if at all) and very
> ununderstandable for users. Simple change of a line requires locating the
> generator, understanding what it does (they tend to be hairy with tons of
> conditions) and patching the upstream [Isar] code. Templates have their
> disadvantages, but are straightforward for users. In general, I'd like to keep
> the current UX and stick with templates, which is possible most of the time.

You prefer duplicated code? Two slightly different templates for two 
buildchroot and image recipe?

Also this patch is only an intermediate step of the way for multi-repo 
support. IMO it has value on its own, since it centralized the 
generation of the multistrap configuration.
The current kind of 'template' does not support creating repository 
entries on the fly. It might be possible to extent the multiconfig with 
shell scripting, but I prefer using python for such advanced logic.

regards,
Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [PATCH 2/2] centralize multistrap configuration generation
  2018-02-15 12:08     ` Claudius Heine
@ 2018-02-15 12:37       ` Alexander Smirnov
  2018-02-15 13:20         ` Claudius Heine
  0 siblings, 1 reply; 23+ messages in thread
From: Alexander Smirnov @ 2018-02-15 12:37 UTC (permalink / raw)
  To: Claudius Heine, isar-users, Baurzhan Ismagulov


Claudius Heine <claudius.heine.ext@siemens.com> 15 февраля 2018 г. 15:08:50 
написал:

> Hi Baurzhan,
>
> On 02/15/2018 12:42 PM, Baurzhan Ismagulov wrote:
>> On Wed, Feb 14, 2018 at 02:15:09PM +0100, claudius.heine.ext@siemens.com wrote:
>>> From: Claudius Heine <ch@denx.de>
>>>
>>> Previous there was redundant code between the buildchroot and
>>> isar-image-base recipe.
>>>
>>> This patch moves the generation of the multistrap configuration to a
>>> central python script that can be parametrized by both recipes.
>>>
>>> Signed-off-by: Claudius Heine <ch@denx.de>
>>> ---
>>>   .../recipes-core/images/files/multistrap.conf.in   | 38 ----------
>>>   meta/lib/isar/multistrap.py                        | 80 ++++++++++++++++++++++
>>>   .../buildchroot/files/multistrap.conf.in           | 37 ----------
>>
>> Could you please post two final multistrap.confs you'd like to get with your
>> changes?
>
> 'like to get'? I don't want to get any other multistrap.conf than
> before. This is not a change of the multistrap.conf files, but just how
> they are generated. The python configparser created ini-file format only
> puts spaces around the '=' characters, but the resulting deserialized
> objects is the same.

The difference that I see IMHO: now you need to feed multistrap by 
community-standardized input, you propose to add custom tool that eats 
proprietary input.

>
> Ok, that is not 100% true, I changed the name of the isar repository
> section. Now its called 'isar' in the image (previously 'Isar') and
> buildchroot (previously 'isar-apt') multiconfig. But that is it. It has
> no different effect outside of multistrap.
>
>> I'd like to understand the reasons for moving from templates to a magic
>> generator.
>>
>> In my experience, magic is usually poorly documented (if at all) and very
>> ununderstandable for users. Simple change of a line requires locating the
>> generator, understanding what it does (they tend to be hairy with tons of
>> conditions) and patching the upstream [Isar] code. Templates have their
>> disadvantages, but are straightforward for users. In general, I'd like to keep
>> the current UX and stick with templates, which is possible most of the time.
>
> You prefer duplicated code? Two slightly different templates for two
> buildchroot and image recipe?
>

AFAIK the key idea is to have the only single multistrap that works with 
upstream repos. Other multistraps fetch packages from local cache. So after 
this is done, the problem with duplicacations in configs will go.

> Also this patch is only an intermediate step of the way for multi-repo
> support. IMO it has value on its own, since it centralized the
> generation of the multistrap configuration.
> The current kind of 'template' does not support creating repository
> entries on the fly.

What is the use case to have them "on the fly"?

Alex

 It might be possible to extent the multiconfig with
> shell scripting, but I prefer using python for such advanced logic.

I don't care about python or bash

>
> regards,
> Claudius
>
> --
> DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
> Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de
>
> --
> You received this message because you are subscribed to the Google Groups 
> "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to isar-users+unsubscribe@googlegroups.com.
> To post to this group, send email to isar-users@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/isar-users/d58343b9-c284-81e7-f743-2672c1f5f8ef%40siemens.com.
> For more options, visit https://groups.google.com/d/optout.




^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [PATCH 2/2] centralize multistrap configuration generation
  2018-02-15 11:27           ` Baurzhan Ismagulov
@ 2018-02-15 13:17             ` Claudius Heine
  2018-02-15 13:18             ` Jan Kiszka
  1 sibling, 0 replies; 23+ messages in thread
From: Claudius Heine @ 2018-02-15 13:17 UTC (permalink / raw)
  To: isar-users

Hi Baurzhan,

On 02/15/2018 12:27 PM, Baurzhan Ismagulov wrote:
> On Thu, Feb 15, 2018 at 11:15:02AM +0100, Jan Kiszka wrote:
>>> That shows to me that upstream might be interested in some of these
>>> improvements we would need to implement.
>>
>> Exactly. We should really use the community-proven, continuously tested
>> official approach of Debian, in that in a way that it foresees for it.
>>
>> Let's bury multistrap. Its a dead horse we rode to long now.
> 
> First off, thanks for cutting the many lines of code not being referred to :) .
> 
> Regarding multistrap, both approaches have advantages and disadvantages.
> Moreover, there are other implementations, such as cdebootstrap.

I am not sure about cdebootstrap, while I do like a C implementation 
more the one in shell scripts, I am skeptical about its maintenance and 
if they even accept contributions.

Cdebootstrap last version is 0.7.7 from 2017-03-08 while debootstrap 
latest version 1.0.93 is from 2017-12-07. I could not find any 
repository containing the development history of cdebootstrap, while 
debootstraps repo is here [1]. cdebootstrap seems to be developed by the 
package maintainer himself while debootstrap is maintained and developed 
by multiple people.

If we don't plan on developing our own debian bootstrap method, then 
trying to get changes pushed there seems to have a higher chance of 
success then with cdebootstrap.

[1] https://anonscm.debian.org/cgit/d-i/debootstrap.git

> 
> 
> Multistrap is currently useful in the following ways:
> 
> 1. Generate the list of packages to install using several apt repos.
> 
> 2. Provide the package paths to download from apt repos.
> 
> 3. Generate the base system from several apt repos.
> 
> Unfortunately, all three seem to be unique among the three candidates.
> 
> 
> (1) and (2) are necessary for reproducibility. My preferred solution would be
> to use an apt library that would do that for us (e.g., extract it from apt,
> extend python-debian, or look whether aptly provides that). Moving to
> *debootstrap would require solving those in a community-unmaintained way for
> some time.
> 
> (3) is useful for people with customizations cutting off significant dependency
> chains from Debian (like SLIND did with Perl dependencies). Moving to
> *debootstrap would mean first installing those, then apt-getting our layers
> above the base system, then removing the packages that are no more required.

IMO apt-preferences could help here. Prios >= 1000 would remove already 
installed packages.

regards,
Claudius

> 
> This was the main reason for Neil Williams to create multistrap. After trying
> to wrap debootstrap for some time, he ended up with problems not solvable in
> that way (not sure which, maybe some kinds of circular dependencies or
> dependencies on exact package versions). So, it would be wise to consider his
> experience here.
> 
> That said, on DebConf17, he told us that that use case is not interesting for
> him any more, advised to use debootstrap, and said that he doesn't support
> multistrap any more.
> 
> Regarding maintaining multistrap for some time: I personally don't like Perl.
> That said, Alex looked at that recently; it's some reasonable wrapping around
> apt-get. If the problem in the end is solved with apt-get, we should possibly
> evaluate that, too.
> 
> 
> Regarding debootstrap, the problem is that it is implemented in shell, which is
> horrible for any serious improvements. Thus the question about cdebootstrap.
> 
> 
> All in all, I'd like to see a solution that would offer serious benefits while
> not introducing new problems. Currently, multistrap is an internal issue not
> directly affecting user-visible features. There are many features we can do for
> far better Debian support. Changing one bit with another, not better one
> wouldn't change the big picture. So, I'd rather use this opportunity to start a
> bigger effort of defining the requirements, choosing the tools to merge with,
> provide a working PoC, and start working with Debian on reusability of those
> tools in Isar. IMHO, *debootstrap alone wouldn't bring major architectural
> improvements.
> 
> 
> With kind regards,
> Baurzhan.
> 

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [PATCH 2/2] centralize multistrap configuration generation
  2018-02-15 11:27           ` Baurzhan Ismagulov
  2018-02-15 13:17             ` Claudius Heine
@ 2018-02-15 13:18             ` Jan Kiszka
  1 sibling, 0 replies; 23+ messages in thread
From: Jan Kiszka @ 2018-02-15 13:18 UTC (permalink / raw)
  To: isar-users

On 2018-02-15 12:27, Baurzhan Ismagulov wrote:
> On Thu, Feb 15, 2018 at 11:15:02AM +0100, Jan Kiszka wrote:
>>> That shows to me that upstream might be interested in some of these
>>> improvements we would need to implement.
>>
>> Exactly. We should really use the community-proven, continuously tested
>> official approach of Debian, in that in a way that it foresees for it.
>>
>> Let's bury multistrap. Its a dead horse we rode to long now.
> 
> First off, thanks for cutting the many lines of code not being referred to :) .
> 
> Regarding multistrap, both approaches have advantages and disadvantages.
> Moreover, there are other implementations, such as cdebootstrap.
> 
> 
> Multistrap is currently useful in the following ways:
> 
> 1. Generate the list of packages to install using several apt repos.
> 
> 2. Provide the package paths to download from apt repos.
> 
> 3. Generate the base system from several apt repos.
> 
> Unfortunately, all three seem to be unique among the three candidates.
> 
> 
> (1) and (2) are necessary for reproducibility. My preferred solution would be
> to use an apt library that would do that for us (e.g., extract it from apt,
> extend python-debian, or look whether aptly provides that). Moving to
> *debootstrap would require solving those in a community-unmaintained way for
> some time.
> 
> (3) is useful for people with customizations cutting off significant dependency
> chains from Debian (like SLIND did with Perl dependencies). Moving to
> *debootstrap would mean first installing those, then apt-getting our layers
> above the base system, then removing the packages that are no more required.
> 
> This was the main reason for Neil Williams to create multistrap. After trying
> to wrap debootstrap for some time, he ended up with problems not solvable in
> that way (not sure which, maybe some kinds of circular dependencies or
> dependencies on exact package versions). So, it would be wise to consider his
> experience here.
> 
> That said, on DebConf17, he told us that that use case is not interesting for
> him any more, advised to use debootstrap, and said that he doesn't support
> multistrap any more.
> 
> Regarding maintaining multistrap for some time: I personally don't like Perl.
> That said, Alex looked at that recently; it's some reasonable wrapping around
> apt-get. If the problem in the end is solved with apt-get, we should possibly
> evaluate that, too.
> 
> 
> Regarding debootstrap, the problem is that it is implemented in shell, which is
> horrible for any serious improvements. Thus the question about cdebootstrap.
> 
> 
> All in all, I'd like to see a solution that would offer serious benefits while
> not introducing new problems. Currently, multistrap is an internal issue not
> directly affecting user-visible features. There are many features we can do for
> far better Debian support. Changing one bit with another, not better one
> wouldn't change the big picture. So, I'd rather use this opportunity to start a
> bigger effort of defining the requirements, choosing the tools to merge with,
> provide a working PoC, and start working with Debian on reusability of those
> tools in Isar. IMHO, *debootstrap alone wouldn't bring major architectural
> improvements.

Let's wait for a first prototype. Maybe it will become clearer that we
actually need, if debootstrap can do it, or if we are better of writing
the core and best all the bootstarps in Isar recipes.

Fact is (as I have been told): debootstarp is Debian standard, all
others are secondary or even historic variants. So let's go standard
first, and then see if we need more again.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [PATCH 2/2] centralize multistrap configuration generation
  2018-02-15 12:37       ` Alexander Smirnov
@ 2018-02-15 13:20         ` Claudius Heine
  2018-02-15 13:39           ` Alexander Smirnov
  2018-02-15 13:53           ` Jan Kiszka
  0 siblings, 2 replies; 23+ messages in thread
From: Claudius Heine @ 2018-02-15 13:20 UTC (permalink / raw)
  To: Alexander Smirnov, isar-users, Baurzhan Ismagulov

On 02/15/2018 01:37 PM, Alexander Smirnov wrote:
> 
> Claudius Heine <claudius.heine.ext@siemens.com> 15 февраля 2018 г. 
> 15:08:50 написал:
> 
>> Hi Baurzhan,
>>
>> On 02/15/2018 12:42 PM, Baurzhan Ismagulov wrote:
>>> On Wed, Feb 14, 2018 at 02:15:09PM +0100, 
>>> claudius.heine.ext@siemens.com wrote:
>>>> From: Claudius Heine <ch@denx.de>
>>>>
>>>> Previous there was redundant code between the buildchroot and
>>>> isar-image-base recipe.
>>>>
>>>> This patch moves the generation of the multistrap configuration to a
>>>> central python script that can be parametrized by both recipes.
>>>>
>>>> Signed-off-by: Claudius Heine <ch@denx.de>
>>>> ---
>>>>   .../recipes-core/images/files/multistrap.conf.in   | 38 ----------
>>>>   meta/lib/isar/multistrap.py                        | 80 
>>>> ++++++++++++++++++++++
>>>>   .../buildchroot/files/multistrap.conf.in           | 37 ----------
>>>
>>> Could you please post two final multistrap.confs you'd like to get 
>>> with your
>>> changes?
>>
>> 'like to get'? I don't want to get any other multistrap.conf than
>> before. This is not a change of the multistrap.conf files, but just how
>> they are generated. The python configparser created ini-file format only
>> puts spaces around the '=' characters, but the resulting deserialized
>> objects is the same.
> 
> The difference that I see IMHO: now you need to feed multistrap by 
> community-standardized input, you propose to add custom tool that eats 
> proprietary input.

I don't get this point. What do you mean by that?

> 
>>
>> Ok, that is not 100% true, I changed the name of the isar repository
>> section. Now its called 'isar' in the image (previously 'Isar') and
>> buildchroot (previously 'isar-apt') multiconfig. But that is it. It has
>> no different effect outside of multistrap.
>>
>>> I'd like to understand the reasons for moving from templates to a magic
>>> generator.
>>>
>>> In my experience, magic is usually poorly documented (if at all) and 
>>> very
>>> ununderstandable for users. Simple change of a line requires locating 
>>> the
>>> generator, understanding what it does (they tend to be hairy with 
>>> tons of
>>> conditions) and patching the upstream [Isar] code. Templates have their
>>> disadvantages, but are straightforward for users. In general, I'd 
>>> like to keep
>>> the current UX and stick with templates, which is possible most of 
>>> the time.
>>
>> You prefer duplicated code? Two slightly different templates for two
>> buildchroot and image recipe?
>>
> 
> AFAIK the key idea is to have the only single multistrap that works with 
> upstream repos. Other multistraps fetch packages from local cache. So 
> after this is done, the problem with duplicacations in configs will go.
> 
>> Also this patch is only an intermediate step of the way for multi-repo
>> support. IMO it has value on its own, since it centralized the
>> generation of the multistrap configuration.
>> The current kind of 'template' does not support creating repository
>> entries on the fly.
> 
> What is the use case to have them "on the fly"?

See: Also this patch is only an intermediate step of the way for 
multi-repo support.

Use-case for multi-repo: docker-ce repository available.

> 
> Alex
> 
> It might be possible to extent the multiconfig with
>> shell scripting, but I prefer using python for such advanced logic.
> 
> I don't care about python or bash

At least Baurzhan seems to care, because he doesn't like debootstrap 
because of it being in shell.

Cheers,
Claudius

> 
>>
>> regards,
>> Claudius
>>
>> -- 
>> DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
>> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
>> Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de
>>
>> -- 
>> You received this message because you are subscribed to the Google 
>> Groups "isar-users" group.
>> To unsubscribe from this group and stop receiving emails from it, send 
>> an email to isar-users+unsubscribe@googlegroups.com.
>> To post to this group, send email to isar-users@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/isar-users/d58343b9-c284-81e7-f743-2672c1f5f8ef%40siemens.com. 
>>
>> For more options, visit https://groups.google.com/d/optout.
> 
> 
> 

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [PATCH 2/2] centralize multistrap configuration generation
  2018-02-15 13:20         ` Claudius Heine
@ 2018-02-15 13:39           ` Alexander Smirnov
  2018-02-16  9:31             ` Claudius Heine
  2018-02-15 13:53           ` Jan Kiszka
  1 sibling, 1 reply; 23+ messages in thread
From: Alexander Smirnov @ 2018-02-15 13:39 UTC (permalink / raw)
  To: Claudius Heine, isar-users, Baurzhan Ismagulov



On 02/15/2018 04:20 PM, Claudius Heine wrote:
> On 02/15/2018 01:37 PM, Alexander Smirnov wrote:
>>
>> Claudius Heine <claudius.heine.ext@siemens.com> 15 февраля 2018 г. 
>> 15:08:50 написал:
>>
>>> Hi Baurzhan,
>>>
>>> On 02/15/2018 12:42 PM, Baurzhan Ismagulov wrote:
>>>> On Wed, Feb 14, 2018 at 02:15:09PM +0100, 
>>>> claudius.heine.ext@siemens.com wrote:
>>>>> From: Claudius Heine <ch@denx.de>
>>>>>
>>>>> Previous there was redundant code between the buildchroot and
>>>>> isar-image-base recipe.
>>>>>
>>>>> This patch moves the generation of the multistrap configuration to a
>>>>> central python script that can be parametrized by both recipes.
>>>>>
>>>>> Signed-off-by: Claudius Heine <ch@denx.de>
>>>>> ---
>>>>>   .../recipes-core/images/files/multistrap.conf.in   | 38 ----------
>>>>>   meta/lib/isar/multistrap.py                        | 80 
>>>>> ++++++++++++++++++++++
>>>>>   .../buildchroot/files/multistrap.conf.in           | 37 ----------
>>>>
>>>> Could you please post two final multistrap.confs you'd like to get 
>>>> with your
>>>> changes?
>>>
>>> 'like to get'? I don't want to get any other multistrap.conf than
>>> before. This is not a change of the multistrap.conf files, but just how
>>> they are generated. The python configparser created ini-file format only
>>> puts spaces around the '=' characters, but the resulting deserialized
>>> objects is the same.
>>
>> The difference that I see IMHO: now you need to feed multistrap by 
>> community-standardized input, you propose to add custom tool that eats 
>> proprietary input.
> 
> I don't get this point. What do you mean by that?

You've introduced new language, like:

+MULTISTRAP_ISAR_SOURCE = "copy:///${DEPLOY_DIR_APT}/${DISTRO}"
+MULTISTRAP_ISAR_INSTALL = "${IMAGE_INSTALL}"
+MULTISTRAP_ISAR_SUITE = "${DEBDISTRONAME}"

So to have working multistrap config I *have* to write pre-config in 
this new language, what it's the simplification here, one set of options 
is replaced by another...

> 
>>
>>>
>>> Ok, that is not 100% true, I changed the name of the isar repository
>>> section. Now its called 'isar' in the image (previously 'Isar') and
>>> buildchroot (previously 'isar-apt') multiconfig. But that is it. It has
>>> no different effect outside of multistrap.
>>>
>>>> I'd like to understand the reasons for moving from templates to a magic
>>>> generator.
>>>>
>>>> In my experience, magic is usually poorly documented (if at all) and 
>>>> very
>>>> ununderstandable for users. Simple change of a line requires 
>>>> locating the
>>>> generator, understanding what it does (they tend to be hairy with 
>>>> tons of
>>>> conditions) and patching the upstream [Isar] code. Templates have their
>>>> disadvantages, but are straightforward for users. In general, I'd 
>>>> like to keep
>>>> the current UX and stick with templates, which is possible most of 
>>>> the time.
>>>
>>> You prefer duplicated code? Two slightly different templates for two
>>> buildchroot and image recipe?
>>>
>>
>> AFAIK the key idea is to have the only single multistrap that works 
>> with upstream repos. Other multistraps fetch packages from local 
>> cache. So after this is done, the problem with duplicacations in 
>> configs will go.
>>
>>> Also this patch is only an intermediate step of the way for multi-repo
>>> support. IMO it has value on its own, since it centralized the
>>> generation of the multistrap configuration.
>>> The current kind of 'template' does not support creating repository
>>> entries on the fly.
>>
>> What is the use case to have them "on the fly"?
> 
> See: Also this patch is only an intermediate step of the way for 
> multi-repo support.
> 
> Use-case for multi-repo: docker-ce repository available.

Sorry, don't know what is docker-ce. Could you please describe whole 
repositories layout, just to understand the topic?

> 
>>
>> Alex
>>
>> It might be possible to extent the multiconfig with
>>> shell scripting, but I prefer using python for such advanced logic.
>>
>> I don't care about python or bash >
> At least Baurzhan seems to care, because he doesn't like debootstrap 
> because of it being in shell.

Aghr, sent the mail on interrupted line, sorry. :-(

I meant I don't care whether python or bash is used, my concern what is 
the input for they. If you prefer to parse multistrap config with python 
instead of bash, I could understand this. But here you want to introduce 
another language for doing the same things.

So at least for me, it would be much easier to take some template for 
multistrap config (which syntax is well known and available in man page) 
and create the custom layout that I need.

Alex

> 
> Cheers,
> Claudius
> 
>>
>>>
>>> regards,
>>> Claudius
>>>
>>> -- 
>>> DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
>>> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
>>> Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "isar-users" group.
>>> To unsubscribe from this group and stop receiving emails from it, 
>>> send an email to isar-users+unsubscribe@googlegroups.com.
>>> To post to this group, send email to isar-users@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/isar-users/d58343b9-c284-81e7-f743-2672c1f5f8ef%40siemens.com. 
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
> 

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [PATCH 2/2] centralize multistrap configuration generation
  2018-02-15 13:20         ` Claudius Heine
  2018-02-15 13:39           ` Alexander Smirnov
@ 2018-02-15 13:53           ` Jan Kiszka
  1 sibling, 0 replies; 23+ messages in thread
From: Jan Kiszka @ 2018-02-15 13:53 UTC (permalink / raw)
  To: [ext] Claudius Heine, Alexander Smirnov, isar-users, Baurzhan Ismagulov

On 2018-02-15 14:20, [ext] Claudius Heine wrote:
>> It might be possible to extent the multiconfig with
>>> shell scripting, but I prefer using python for such advanced logic.
>>
>> I don't care about python or bash
> 
> At least Baurzhan seems to care, because he doesn't like debootstrap
> because of it being in shell.
> 

That's not a major argument. For us, debootstrap a tool with an API. If
we can use that API without bending it, it's fine. If we can improve our
use case by enhancing it upstream later on, it's perfect. Otherwise, we
need to model Debian bootstrap after the community pattern with our own
languages (shell and python).

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [PATCH 2/2] centralize multistrap configuration generation
  2018-02-15 13:39           ` Alexander Smirnov
@ 2018-02-16  9:31             ` Claudius Heine
  2018-02-16 10:35               ` Alexander Smirnov
  0 siblings, 1 reply; 23+ messages in thread
From: Claudius Heine @ 2018-02-16  9:31 UTC (permalink / raw)
  To: Alexander Smirnov, Claudius Heine, isar-users, Baurzhan Ismagulov


[-- Attachment #1.1: Type: text/plain, Size: 7885 bytes --]

On 02/15/2018 02:39 PM, Alexander Smirnov wrote:
> 
> 
> On 02/15/2018 04:20 PM, Claudius Heine wrote:
>> On 02/15/2018 01:37 PM, Alexander Smirnov wrote:
>>>
>>> Claudius Heine <claudius.heine.ext@siemens.com> 15 февраля 2018 г.
>>> 15:08:50 написал:
>>>
>>>> Hi Baurzhan,
>>>>
>>>> On 02/15/2018 12:42 PM, Baurzhan Ismagulov wrote:
>>>>> On Wed, Feb 14, 2018 at 02:15:09PM +0100,
>>>>> claudius.heine.ext@siemens.com wrote:
>>>>>> From: Claudius Heine <ch@denx.de>
>>>>>>
>>>>>> Previous there was redundant code between the buildchroot and
>>>>>> isar-image-base recipe.
>>>>>>
>>>>>> This patch moves the generation of the multistrap configuration to a
>>>>>> central python script that can be parametrized by both recipes.
>>>>>>
>>>>>> Signed-off-by: Claudius Heine <ch@denx.de>
>>>>>> ---
>>>>>>   .../recipes-core/images/files/multistrap.conf.in   | 38 ----------
>>>>>>   meta/lib/isar/multistrap.py                        | 80
>>>>>> ++++++++++++++++++++++
>>>>>>   .../buildchroot/files/multistrap.conf.in           | 37 ----------
>>>>>
>>>>> Could you please post two final multistrap.confs you'd like to get
>>>>> with your
>>>>> changes?
>>>>
>>>> 'like to get'? I don't want to get any other multistrap.conf than
>>>> before. This is not a change of the multistrap.conf files, but just how
>>>> they are generated. The python configparser created ini-file format
>>>> only
>>>> puts spaces around the '=' characters, but the resulting deserialized
>>>> objects is the same.
>>>
>>> The difference that I see IMHO: now you need to feed multistrap by
>>> community-standardized input, you propose to add custom tool that
>>> eats proprietary input.
>>
>> I don't get this point. What do you mean by that?
> 
> You've introduced new language, like:
> 
> +MULTISTRAP_ISAR_SOURCE = "copy:///${DEPLOY_DIR_APT}/${DISTRO}"
> +MULTISTRAP_ISAR_INSTALL = "${IMAGE_INSTALL}"
> +MULTISTRAP_ISAR_SUITE = "${DEBDISTRONAME}"
> 
> So to have working multistrap config I *have* to write pre-config in
> this new language, what it's the simplification here, one set of options
> is replaced by another...

??? Thats not a 'new language' that just bitbake variables. I think I
don't get what you are saying. Thats the same stuff that previous to
this change happend in this big wall of text sed command. IMO that this
is much more to the point and therefore easier to handle than this big
search and replace list.

>>
>>>
>>>>
>>>> Ok, that is not 100% true, I changed the name of the isar repository
>>>> section. Now its called 'isar' in the image (previously 'Isar') and
>>>> buildchroot (previously 'isar-apt') multiconfig. But that is it. It has
>>>> no different effect outside of multistrap.
>>>>
>>>>> I'd like to understand the reasons for moving from templates to a
>>>>> magic
>>>>> generator.
>>>>>
>>>>> In my experience, magic is usually poorly documented (if at all)
>>>>> and very
>>>>> ununderstandable for users. Simple change of a line requires
>>>>> locating the
>>>>> generator, understanding what it does (they tend to be hairy with
>>>>> tons of
>>>>> conditions) and patching the upstream [Isar] code. Templates have
>>>>> their
>>>>> disadvantages, but are straightforward for users. In general, I'd
>>>>> like to keep
>>>>> the current UX and stick with templates, which is possible most of
>>>>> the time.
>>>>
>>>> You prefer duplicated code? Two slightly different templates for two
>>>> buildchroot and image recipe?
>>>>
>>>
>>> AFAIK the key idea is to have the only single multistrap that works
>>> with upstream repos. Other multistraps fetch packages from local
>>> cache. So after this is done, the problem with duplicacations in
>>> configs will go.
>>>
>>>> Also this patch is only an intermediate step of the way for multi-repo
>>>> support. IMO it has value on its own, since it centralized the
>>>> generation of the multistrap configuration.
>>>> The current kind of 'template' does not support creating repository
>>>> entries on the fly.
>>>
>>> What is the use case to have them "on the fly"?
>>
>> See: Also this patch is only an intermediate step of the way for
>> multi-repo support.
>>
>> Use-case for multi-repo: docker-ce repository available.
> 
> Sorry, don't know what is docker-ce. Could you please describe whole
> repositories layout, just to understand the topic?

I have the need to install docker community edition from the developers
repository in my image. Currently I do it by downloading the 'deb' file
from the repo manually and installing it, here I have to take care of
updating the version to the latest one. What would be much better if I
can just add this repository to the multistrap process and install the
most current version like that. For that I need the ability to add
additional repositories to the multistrap configuration from other layers.

I don't understand what you mean by 'whole repositories layout'. You
mean like a tree, but from what repository?

> 
>>
>>>
>>> Alex
>>>
>>> It might be possible to extent the multiconfig with
>>>> shell scripting, but I prefer using python for such advanced logic.
>>>
>>> I don't care about python or bash >
>> At least Baurzhan seems to care, because he doesn't like debootstrap
>> because of it being in shell.
> 
> Aghr, sent the mail on interrupted line, sorry. :-(
> 
> I meant I don't care whether python or bash is used, my concern what is
> the input for they. If you prefer to parse multistrap config with python
> instead of bash, I could understand this.

I don't understand you again. I don't parse the multistrap config. I
just generate one.

> But here you want to introduce another language for doing the same things.

Again, what other language? I don't remember writing or using any
language parse in this patch.

> So at least for me, it would be much easier to take some template for
> multistrap config (which syntax is well known and available in man page)
> and create the custom layout that I need.

If you prefer the template approach, then we might need to introduce a
new templating language, because only search and replace via sed doesn't
cut it for the multi-repo feature.

Since I know jinja [1] we could do it with this. But it still might
result in something more hacky.

[1] http://jinja.pocoo.org/

Also since Jan, already decreed the switch to debootstrap I will now
continue working on a patchset for this.

Cheers,
Claudius

> 
> Alex
> 
>>
>> Cheers,
>> Claudius
>>
>>>
>>>>
>>>> regards,
>>>> Claudius
>>>>
>>>> -- 
>>>> DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
>>>> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
>>>> Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de
>>>>
>>>> -- 
>>>> You received this message because you are subscribed to the Google
>>>> Groups "isar-users" group.
>>>> To unsubscribe from this group and stop receiving emails from it,
>>>> send an email to isar-users+unsubscribe@googlegroups.com.
>>>> To post to this group, send email to isar-users@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/isar-users/d58343b9-c284-81e7-f743-2672c1f5f8ef%40siemens.com.
>>>>
>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>>
>>
> 

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

           PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153
                             Keyserver: hkp://pool.sks-keyservers.net


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [PATCH 2/2] centralize multistrap configuration generation
  2018-02-16  9:31             ` Claudius Heine
@ 2018-02-16 10:35               ` Alexander Smirnov
  2018-02-16 11:35                 ` Jan Kiszka
  0 siblings, 1 reply; 23+ messages in thread
From: Alexander Smirnov @ 2018-02-16 10:35 UTC (permalink / raw)
  To: Claudius Heine, Claudius Heine, isar-users, Baurzhan Ismagulov


On 02/16/2018 12:31 PM, Claudius Heine wrote:
> On 02/15/2018 02:39 PM, Alexander Smirnov wrote:
>>
>>
>> On 02/15/2018 04:20 PM, Claudius Heine wrote:
>>> On 02/15/2018 01:37 PM, Alexander Smirnov wrote:
>>>>
>>>> Claudius Heine <claudius.heine.ext@siemens.com> 15 февраля 2018 г.
>>>> 15:08:50 написал:
>>>>
>>>>> Hi Baurzhan,
>>>>>
>>>>> On 02/15/2018 12:42 PM, Baurzhan Ismagulov wrote:
>>>>>> On Wed, Feb 14, 2018 at 02:15:09PM +0100,
>>>>>> claudius.heine.ext@siemens.com wrote:
>>>>>>> From: Claudius Heine <ch@denx.de>
>>>>>>>
>>>>>>> Previous there was redundant code between the buildchroot and
>>>>>>> isar-image-base recipe.
>>>>>>>
>>>>>>> This patch moves the generation of the multistrap configuration to a
>>>>>>> central python script that can be parametrized by both recipes.
>>>>>>>
>>>>>>> Signed-off-by: Claudius Heine <ch@denx.de>
>>>>>>> ---
>>>>>>>    .../recipes-core/images/files/multistrap.conf.in   | 38 ----------
>>>>>>>    meta/lib/isar/multistrap.py                        | 80
>>>>>>> ++++++++++++++++++++++
>>>>>>>    .../buildchroot/files/multistrap.conf.in           | 37 ----------
>>>>>>
>>>>>> Could you please post two final multistrap.confs you'd like to get
>>>>>> with your
>>>>>> changes?
>>>>>
>>>>> 'like to get'? I don't want to get any other multistrap.conf than
>>>>> before. This is not a change of the multistrap.conf files, but just how
>>>>> they are generated. The python configparser created ini-file format
>>>>> only
>>>>> puts spaces around the '=' characters, but the resulting deserialized
>>>>> objects is the same.
>>>>
>>>> The difference that I see IMHO: now you need to feed multistrap by
>>>> community-standardized input, you propose to add custom tool that
>>>> eats proprietary input.
>>>
>>> I don't get this point. What do you mean by that?
>>
>> You've introduced new language, like:
>>
>> +MULTISTRAP_ISAR_SOURCE = "copy:///${DEPLOY_DIR_APT}/${DISTRO}"
>> +MULTISTRAP_ISAR_INSTALL = "${IMAGE_INSTALL}"
>> +MULTISTRAP_ISAR_SUITE = "${DEBDISTRONAME}"
>>
>> So to have working multistrap config I *have* to write pre-config in
>> this new language, what it's the simplification here, one set of options
>> is replaced by another...
> 
> ??? Thats not a 'new language' that just bitbake variables. I think I
> don't get what you are saying. Thats the same stuff that previous to
> this change happend in this big wall of text sed command. IMO that this
> is much more to the point and therefore easier to handle than this big
> search and replace list.

1. I suppose your parser eats only *specific* variables, so there is 
predefined list of extra variables I should somehow learn.

2. Each variable has some meaning, so depending on what I want from 
multistrap, I should *manually* assign them. For example if I want to 
use native arch, I have to set something.

3. These variables are complete duplication of multistrap config tokens.

So all the above looks like: instead of setting variables directly in 
multistrap config, you prefer to set them in recipe. A matter of taste, 
but for me this sounds like unnecessary complication of existing approach.

If everything here is only about avoiding duplication, so the multistrap 
cascading configuration [1] could be used. Common multistrap config part 
could be split in separate file and included by all rootfs generators.

[1] - respective section in multistrap man page.

> 
>>>
>>>>
>>>>>
>>>>> Ok, that is not 100% true, I changed the name of the isar repository
>>>>> section. Now its called 'isar' in the image (previously 'Isar') and
>>>>> buildchroot (previously 'isar-apt') multiconfig. But that is it. It has
>>>>> no different effect outside of multistrap.
>>>>>
>>>>>> I'd like to understand the reasons for moving from templates to a
>>>>>> magic
>>>>>> generator.
>>>>>>
>>>>>> In my experience, magic is usually poorly documented (if at all)
>>>>>> and very
>>>>>> ununderstandable for users. Simple change of a line requires
>>>>>> locating the
>>>>>> generator, understanding what it does (they tend to be hairy with
>>>>>> tons of
>>>>>> conditions) and patching the upstream [Isar] code. Templates have
>>>>>> their
>>>>>> disadvantages, but are straightforward for users. In general, I'd
>>>>>> like to keep
>>>>>> the current UX and stick with templates, which is possible most of
>>>>>> the time.
>>>>>
>>>>> You prefer duplicated code? Two slightly different templates for two
>>>>> buildchroot and image recipe?
>>>>>
>>>>
>>>> AFAIK the key idea is to have the only single multistrap that works
>>>> with upstream repos. Other multistraps fetch packages from local
>>>> cache. So after this is done, the problem with duplicacations in
>>>> configs will go.
>>>>
>>>>> Also this patch is only an intermediate step of the way for multi-repo
>>>>> support. IMO it has value on its own, since it centralized the
>>>>> generation of the multistrap configuration.
>>>>> The current kind of 'template' does not support creating repository
>>>>> entries on the fly.
>>>>
>>>> What is the use case to have them "on the fly"?
>>>
>>> See: Also this patch is only an intermediate step of the way for
>>> multi-repo support.
>>>
>>> Use-case for multi-repo: docker-ce repository available.
>>
>> Sorry, don't know what is docker-ce. Could you please describe whole
>> repositories layout, just to understand the topic?
> 
> I have the need to install docker community edition from the developers
> repository in my image. Currently I do it by downloading the 'deb' file
> from the repo manually and installing it, here I have to take care of
> updating the version to the latest one. What would be much better if I
> can just add this repository to the multistrap process and install the
> most current version like that. For that I need the ability to add
> additional repositories to the multistrap configuration from other layers.

Why you can't add respective section to multistrap.conf.in?

> 
> I don't understand what you mean by 'whole repositories layout'. You
> mean like a tree, but from what repository?

Let's say, final mulstisrap configuration file or the list of sources in 
/etc/apt.

Alex

> 
>>
>>>
>>>>
>>>> Alex
>>>>
>>>> It might be possible to extent the multiconfig with
>>>>> shell scripting, but I prefer using python for such advanced logic.
>>>>
>>>> I don't care about python or bash >
>>> At least Baurzhan seems to care, because he doesn't like debootstrap
>>> because of it being in shell.
>>
>> Aghr, sent the mail on interrupted line, sorry. :-(
>>
>> I meant I don't care whether python or bash is used, my concern what is
>> the input for they. If you prefer to parse multistrap config with python
>> instead of bash, I could understand this.
> 
> I don't understand you again. I don't parse the multistrap config. I
> just generate one. >
>> But here you want to introduce another language for doing the same things.
> 
> Again, what other language? I don't remember writing or using any
> language parse in this patch.
> 
>> So at least for me, it would be much easier to take some template for
>> multistrap config (which syntax is well known and available in man page)
>> and create the custom layout that I need.
> 
> If you prefer the template approach, then we might need to introduce a
> new templating language, because only search and replace via sed doesn't
> cut it for the multi-repo feature.
> 
> Since I know jinja [1] we could do it with this. But it still might
> result in something more hacky.
> 
> [1] http://jinja.pocoo.org/
> 
> Also since Jan, already decreed the switch to debootstrap I will now
> continue working on a patchset for this.

I still don't understand why it's so difficult to create new section in 
multistrap.conf file manually? Moreover that's not the place in Isar 
which is going to be changed tens times per-build each build. Once 
you've changed it, you likely never will need to this again within 
current project.

In general, it's impossible to get automatically configure upstream Isar 
according to your requirements. You some how need to specify what you 
want to have via some configuration interface: config files, variables, 
recipes etc...

What is the value to change one manual work (populating 
multistrap.conf.in) to another (populating MULTISTRAP_* variables)?

Alex

> 
> Cheers,
> Claudius
> 
>>
>> Alex
>>
>>>
>>> Cheers,
>>> Claudius
>>>
>>>>
>>>>>
>>>>> regards,
>>>>> Claudius
>>>>>
>>>>> -- 
>>>>> DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
>>>>> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
>>>>> Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de
>>>>>
>>>>> -- 
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "isar-users" group.
>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>> send an email to isar-users+unsubscribe@googlegroups.com.
>>>>> To post to this group, send email to isar-users@googlegroups.com.
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/isar-users/d58343b9-c284-81e7-f743-2672c1f5f8ef%40siemens.com.
>>>>>
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>>
>>>>
>>>
>>
> 

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [PATCH 2/2] centralize multistrap configuration generation
  2018-02-16 10:35               ` Alexander Smirnov
@ 2018-02-16 11:35                 ` Jan Kiszka
  0 siblings, 0 replies; 23+ messages in thread
From: Jan Kiszka @ 2018-02-16 11:35 UTC (permalink / raw)
  To: isar-users

Folks,

let's not discuss the legacy multistrap topic any further and focus on a
clean re-implementation via debootstrap. That will remove all needs to
translate into or from intermediate configuration files.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

^ permalink raw reply	[flat|nested] 23+ messages in thread

end of thread, other threads:[~2018-02-16 11:35 UTC | newest]

Thread overview: 23+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-02-14 13:15 [PATCH 0/2] Consolidate multistrap configuration generation claudius.heine.ext
2018-02-14 13:15 ` [PATCH 1/2] meta/classes/base: extend sys.path with lib directory claudius.heine.ext
2018-02-15  7:28   ` Jan Kiszka
2018-02-14 13:15 ` [PATCH 2/2] centralize multistrap configuration generation claudius.heine.ext
2018-02-15  7:28   ` Jan Kiszka
2018-02-15  8:16     ` Claudius Heine
2018-02-15  8:50       ` Jan Kiszka
2018-02-15  9:34     ` Alexander Smirnov
2018-02-15 10:05       ` Claudius Heine
2018-02-15 10:15         ` Jan Kiszka
2018-02-15 10:29           ` Alexander Smirnov
2018-02-15 11:27           ` Baurzhan Ismagulov
2018-02-15 13:17             ` Claudius Heine
2018-02-15 13:18             ` Jan Kiszka
2018-02-15 11:42   ` Baurzhan Ismagulov
2018-02-15 12:08     ` Claudius Heine
2018-02-15 12:37       ` Alexander Smirnov
2018-02-15 13:20         ` Claudius Heine
2018-02-15 13:39           ` Alexander Smirnov
2018-02-16  9:31             ` Claudius Heine
2018-02-16 10:35               ` Alexander Smirnov
2018-02-16 11:35                 ` Jan Kiszka
2018-02-15 13:53           ` Jan Kiszka

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox