* [RFC PATCH 0/1] Multi-Repo and apt preferences support
@ 2018-02-08 9:46 claudius.heine.ext
2018-02-08 9:46 ` [RFC PATCH 1/1] RFC-PATCH: partial implementation for support of repos from other layers claudius.heine.ext
0 siblings, 1 reply; 5+ messages in thread
From: claudius.heine.ext @ 2018-02-08 9:46 UTC (permalink / raw)
To: isar-users; +Cc: Claudius Heine
From: Claudius Heine <ch@denx.de>
Hi,
Here is a small, patch to continue the discussion about how to implement
multi-repo support.
This patch contains just conversion of the isar-image-base recipe to one
that can support adding repositories from other layers.
It also allows to add apt-preferences from other layers.
I just converted debian stretch to this new format.
Comments are welcome, since this is an RFC :)
Cheers,
Claudius
Claudius Heine (1):
RFC-PATCH: partial implementation for support of repos from other
layers
meta-isar/conf/distro/debian-stretch.conf | 6 +-
meta-isar/conf/distro/stretch.list | 3 +
.../recipes-core/images/files/multistrap.conf.in | 38 ------
meta-isar/recipes-core/images/isar-image-base.bb | 134 +++++++++++++++++----
4 files changed, 117 insertions(+), 64 deletions(-)
create mode 100644 meta-isar/conf/distro/stretch.list
delete mode 100644 meta-isar/recipes-core/images/files/multistrap.conf.in
--
2.15.1
^ permalink raw reply [flat|nested] 5+ messages in thread
* [RFC PATCH 1/1] RFC-PATCH: partial implementation for support of repos from other layers
2018-02-08 9:46 [RFC PATCH 0/1] Multi-Repo and apt preferences support claudius.heine.ext
@ 2018-02-08 9:46 ` claudius.heine.ext
2018-02-08 11:09 ` Benedikt Niedermayr
0 siblings, 1 reply; 5+ messages in thread
From: claudius.heine.ext @ 2018-02-08 9:46 UTC (permalink / raw)
To: isar-users; +Cc: Claudius Heine
From: Claudius Heine <ch@denx.de>
This patch changes the implementation of isar-image-base to support
the definition of additional repositories from other layers, by addding
apt source list compatible files to the 'DEBIAN_APT_SOURCES' variable.
This also implement the generation of the multistrap.conf file from
within the recipe.
Signed-off-by: Claudius Heine <ch@denx.de>
---
meta-isar/conf/distro/debian-stretch.conf | 6 +-
meta-isar/conf/distro/stretch.list | 3 +
.../recipes-core/images/files/multistrap.conf.in | 38 ------
meta-isar/recipes-core/images/isar-image-base.bb | 134 +++++++++++++++++----
4 files changed, 117 insertions(+), 64 deletions(-)
create mode 100644 meta-isar/conf/distro/stretch.list
delete mode 100644 meta-isar/recipes-core/images/files/multistrap.conf.in
diff --git a/meta-isar/conf/distro/debian-stretch.conf b/meta-isar/conf/distro/debian-stretch.conf
index 74177fd..083eb86 100644
--- a/meta-isar/conf/distro/debian-stretch.conf
+++ b/meta-isar/conf/distro/debian-stretch.conf
@@ -1,10 +1,14 @@
# This software is a part of ISAR.
# Copyright (C) 2017 ilbers GmbH
+DEBIAN_APT_SOURCES += "conf/distro/stretch.list"
+DEBIAN_APT_PREFERENCES ??= ""
+DISTRO_CONFIG_SCRIPT ??= "debian-configscript.sh"
+
+# No longer needed, after buildchroot is converted:
DISTRO_SUITE ?= "stretch"
DISTRO_COMPONENTS ?= "main contrib non-free"
DISTRO_APT_SOURCE ?= "http://ftp.debian.org/debian"
DISTRO_APT_SOURCE_SEC ?= "http://security.debian.org/debian-security"
-DISTRO_CONFIG_SCRIPT ?= "debian-configscript.sh"
DISTRO_MULTICONF_BOOTSTRAP ?= "base updates security"
DISTRO_MULTICONF_APTSOURCES ?= "${DISTRO_MULTICONF_BOOTSTRAP}"
diff --git a/meta-isar/conf/distro/stretch.list b/meta-isar/conf/distro/stretch.list
new file mode 100644
index 0000000..a5ed5db
--- /dev/null
+++ b/meta-isar/conf/distro/stretch.list
@@ -0,0 +1,3 @@
+deb http://ftp.de.debian.org/debian stretch main contrib non-free
+deb http://ftp.de.debian.org/debian stretch-updates main contrib non-free
+deb http://security.debian.org stretch/updates main contrib non-free
diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in b/meta-isar/recipes-core/images/files/multistrap.conf.in
deleted file mode 100644
index 432b6af..0000000
--- a/meta-isar/recipes-core/images/files/multistrap.conf.in
+++ /dev/null
@@ -1,38 +0,0 @@
-# This software is a part of ISAR.
-# Copyright (C) 2015-2017 ilbers GmbH
-
-[General]
-noauth=true
-unpack=true
-ignorenativearch=true
-bootstrap=##DISTRO_MULTICONF_BOOTSTRAP## Isar
-aptsources=##DISTRO_MULTICONF_APTSOURCES##
-configscript=##CONFIG_SCRIPT##
-setupscript=##SETUP_SCRIPT##
-hookdir=##DIR_HOOKS##
-
-[base]
-source=##DISTRO_APT_SOURCE##
-suite=##DISTRO_SUITE##
-components=##DISTRO_COMPONENTS##
-packages=##IMAGE_PREINSTALL##
-omitdebsrc=true
-
-[updates]
-source=##DISTRO_APT_SOURCE##
-suite=##DISTRO_SUITE##-updates
-components=##DISTRO_COMPONENTS##
-omitdebsrc=true
-
-[security]
-source=##DISTRO_APT_SOURCE_SEC##
-suite=##DISTRO_SUITE##/updates
-components=##DISTRO_COMPONENTS##
-omitdebsrc=true
-
-[Isar]
-packages=##IMAGE_INSTALL##
-source=##DEPLOY_DIR_APT##
-suite=##ISAR_DISTRO_SUITE##
-components=main
-omitdebsrc=true
diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb
index e359ac3..655dbb2 100644
--- a/meta-isar/recipes-core/images/isar-image-base.bb
+++ b/meta-isar/recipes-core/images/isar-image-base.bb
@@ -9,8 +9,7 @@ LICENSE = "gpl-2.0"
LIC_FILES_CHKSUM = "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
FILESPATH =. "${LAYERDIR_isar}/recipes-core/images/files:"
-SRC_URI = "file://multistrap.conf.in \
- file://${DISTRO_CONFIG_SCRIPT} \
+SRC_URI = "file://${DISTRO_CONFIG_SCRIPT} \
file://setup.sh \
file://download_dev-random"
@@ -26,34 +25,119 @@ IMAGE_PREINSTALL += "apt \
WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
-do_rootfs[dirs] = "${WORKDIR}/hooks_multistrap"
-
-do_rootfs() {
- E="${@ bb.utils.export_proxies(d)}"
+do_multistrap_prepare[dirs] = "${WORKDIR}/hooks_multistrap"
+do_multistrap_prepare() {
chmod +x "${WORKDIR}/${DISTRO_CONFIG_SCRIPT}"
chmod +x "${WORKDIR}/setup.sh"
install -m 755 "${WORKDIR}/download_dev-random" "${WORKDIR}/hooks_multistrap/"
+}
+addtask multistrap_prepare before do_multistrap_config after do_unpack
+
+python do_multistrap_config() {
+ import shutil
+ import configparser as cp
+ import textwrap
+
+ workdir = d.getVar("WORKDIR", True)
+ workdir_rel = os.path.relpath(workdir)
+
+ multistrap_conf_file = os.path.join(workdir, "multistrap.conf")
+
+ apt_sources_list = d.getVar("DEBIAN_APT_SOURCES", True).split()
+
+ apt_preferences_list = d.getVar("DEBIAN_APT_PREFERENCES", True).split()
+ apt_preferences_out = os.path.join(workdir_rel, "apt-preferences")
+
+ image_preinstall = d.getVar("IMAGE_PREINSTALL", True).split()
+ image_install = d.getVar("IMAGE_INSTALL", True).split()
+
+ repos = []
+
+ msconf = cp.RawConfigParser()
+
+ # Set general multistrap configuration:
+ msconf.add_section("General")
+ msconf.set("General", "noauth", "true")
+ msconf.set("General", "unpack", "true")
+ msconf.set("General", "ignorenativearch", "true")
+ msconf.set("General", "aptpreferences", "./" + apt_preferences_out)
+ msconf.set("General", "configscript", "./" +
+ os.path.join(workdir_rel,
+ d.getVar("DISTRO_CONFIG_SCRIPT", True)))
+ msconf.set("General", "setupscript", "./" +
+ os.path.join(workdir_rel,
+ "setup.sh"))
+ msconf.set("General", "hookdir", "./" +
+ os.path.join(workdir_rel,
+ "hooks_multistrap"))
+
+ # Set repository in multistrap configuration:
+ repo_counter = 0
+ for aptsrc in apt_sources_list:
+ aptsrc_real = bb.parse.resolve_file(aptsrc, d)
+ with open(aptsrc_real, "r") as in_fd:
+ for line in in_fd:
+ entry = line.split()
+
+ # Only care about 'deb' repos for now:
+ if entry[0] != "deb":
+ continue
+
+ # Generate on unique name for the section
+ repo_name = "repo{}".format(repo_counter)
+ repos.append(repo_name)
+
+ source = entry[1]
+ suite = entry[2]
+ components = entry[3:]
+
+ msconf.add_section(repo_name)
+ msconf.set(repo_name, "source", source)
+ msconf.set(repo_name, "suite", suite)
+ msconf.set(repo_name, "components", " ".join(components))
+ msconf.set(repo_name, "omitdebsrc", "true")
+
+ # The first repo gets assigned all IMAGE_PREINSTALL
+ # packages:
+ if image_preinstall:
+ msconf.set(repo_name, "packages",
+ " ".join(image_preinstall))
+ image_preinstall = []
+
+ repo_counter += 1
+
+ msconf.add_section("isar-generated")
+ msconf.set("isar-generated", "packages", " ".join(image_install))
+ msconf.set("isar-generated", "source", "copy:///{}/{}".format(
+ d.getVar("DEPLOY_DIR_APT", True), d.getVar("DISTRO", True)))
+ msconf.set("isar-generated", "suite", d.getVar("DEBDISTRONAME", True))
+ msconf.set("isar-generated", "components", "main")
+ msconf.set("isar-generated", "omitdebsrc", "true")
+
+ # Generate apt preference file:
+ with open(apt_preferences_out, "wb") as out_fd:
+ for aptpref in apt_preferences_list:
+ artpref_real = bb.parse.resolve_file(aptpref, d)
+ with open(aptpref_real, "rb") as in_fd:
+ shutil.copyfileobj(in_fd, out_fd, 1024*1024*10)
+ out_fd.write("\n".encode())
+ out_fd.write(textwrap.dedent("""\
+ Package: *
+ Pin: release n={}
+ Pin-Priority: 1000
+ """.format(d.getVar("DEBDISTRONAME", True))).encode())
+
+ msconf.set("General", "bootstrap", " ".join(repos))
+ msconf.set("General", "aptsources", " ".join(repos + ["isar-generated"]))
+
+ with open(multistrap_conf_file, "w") as out_fd:
+ msconf.write(out_fd)
+}
+addtask multistrap_config after do_multistrap_prepare before do_rootfs
- # Multistrap accepts only relative path in configuration files, so get it:
- cd ${TOPDIR}
- WORKDIR_REL=${@ os.path.relpath(d.getVar("WORKDIR", True))}
-
- # Adjust multistrap config
- sed -e 's|##IMAGE_PREINSTALL##|${IMAGE_PREINSTALL}|g' \
- -e 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \
- -e 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \
- -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \
- -e 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \
- -e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \
- -e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \
- -e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/${DISTRO_CONFIG_SCRIPT}|g' \
- -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \
- -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
- -e 's|##IMAGE_INSTALL##|${IMAGE_INSTALL}|g' \
- -e 's|##DEPLOY_DIR_APT##|copy:///${DEPLOY_DIR_APT}/${DISTRO}|g' \
- -e 's|##ISAR_DISTRO_SUITE##|${DEBDISTRONAME}|g' \
- "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf"
+do_rootfs() {
+ E="${@ bb.utils.export_proxies(d)}"
[ ! -d ${IMAGE_ROOTFS}/proc ] && sudo install -d -o 0 -g 0 -m 555 ${IMAGE_ROOTFS}/proc
sudo mount -t proc none ${IMAGE_ROOTFS}/proc
--
2.15.1
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [RFC PATCH 1/1] RFC-PATCH: partial implementation for support of repos from other layers
2018-02-08 9:46 ` [RFC PATCH 1/1] RFC-PATCH: partial implementation for support of repos from other layers claudius.heine.ext
@ 2018-02-08 11:09 ` Benedikt Niedermayr
2018-02-08 11:26 ` Jan Kiszka
2018-02-08 11:41 ` Claudius Heine
0 siblings, 2 replies; 5+ messages in thread
From: Benedikt Niedermayr @ 2018-02-08 11:09 UTC (permalink / raw)
To: isar-users
Am 08.02.2018 um 10:46 schrieb claudius.heine.ext@siemens.com:
> From: Claudius Heine <ch@denx.de>
>
> This patch changes the implementation of isar-image-base to support
> the definition of additional repositories from other layers, by addding
> apt source list compatible files to the 'DEBIAN_APT_SOURCES' variable.
>
> This also implement the generation of the multistrap.conf file from
> within the recipe.
>
> Signed-off-by: Claudius Heine <ch@denx.de>
> ---
> meta-isar/conf/distro/debian-stretch.conf | 6 +-
> meta-isar/conf/distro/stretch.list | 3 +
> .../recipes-core/images/files/multistrap.conf.in | 38 ------
> meta-isar/recipes-core/images/isar-image-base.bb | 134 +++++++++++++++++----
> 4 files changed, 117 insertions(+), 64 deletions(-)
> create mode 100644 meta-isar/conf/distro/stretch.list
> delete mode 100644 meta-isar/recipes-core/images/files/multistrap.conf.in
>
> diff --git a/meta-isar/conf/distro/debian-stretch.conf b/meta-isar/conf/distro/debian-stretch.conf
> index 74177fd..083eb86 100644
> --- a/meta-isar/conf/distro/debian-stretch.conf
> +++ b/meta-isar/conf/distro/debian-stretch.conf
> @@ -1,10 +1,14 @@
> # This software is a part of ISAR.
> # Copyright (C) 2017 ilbers GmbH
>
> +DEBIAN_APT_SOURCES += "conf/distro/stretch.list"
> +DEBIAN_APT_PREFERENCES ??= ""
> +DISTRO_CONFIG_SCRIPT ??= "debian-configscript.sh"
> +
> +# No longer needed, after buildchroot is converted:
> DISTRO_SUITE ?= "stretch"
> DISTRO_COMPONENTS ?= "main contrib non-free"
> DISTRO_APT_SOURCE ?= "http://ftp.debian.org/debian"
> DISTRO_APT_SOURCE_SEC ?= "http://security.debian.org/debian-security"
> -DISTRO_CONFIG_SCRIPT ?= "debian-configscript.sh"
> DISTRO_MULTICONF_BOOTSTRAP ?= "base updates security"
> DISTRO_MULTICONF_APTSOURCES ?= "${DISTRO_MULTICONF_BOOTSTRAP}"
> diff --git a/meta-isar/conf/distro/stretch.list b/meta-isar/conf/distro/stretch.list
> new file mode 100644
> index 0000000..a5ed5db
> --- /dev/null
> +++ b/meta-isar/conf/distro/stretch.list
> @@ -0,0 +1,3 @@
> +deb http://ftp.de.debian.org/debian stretch main contrib non-free
> +deb http://ftp.de.debian.org/debian stretch-updates main contrib non-free
> +deb http://security.debian.org stretch/updates main contrib non-free
> diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in b/meta-isar/recipes-core/images/files/multistrap.conf.in
> deleted file mode 100644
> index 432b6af..0000000
> --- a/meta-isar/recipes-core/images/files/multistrap.conf.in
> +++ /dev/null
> @@ -1,38 +0,0 @@
> -# This software is a part of ISAR.
> -# Copyright (C) 2015-2017 ilbers GmbH
> -
> -[General]
> -noauth=true
> -unpack=true
> -ignorenativearch=true
> -bootstrap=##DISTRO_MULTICONF_BOOTSTRAP## Isar
> -aptsources=##DISTRO_MULTICONF_APTSOURCES##
> -configscript=##CONFIG_SCRIPT##
> -setupscript=##SETUP_SCRIPT##
> -hookdir=##DIR_HOOKS##
> -
> -[base]
> -source=##DISTRO_APT_SOURCE##
> -suite=##DISTRO_SUITE##
> -components=##DISTRO_COMPONENTS##
> -packages=##IMAGE_PREINSTALL##
> -omitdebsrc=true
> -
> -[updates]
> -source=##DISTRO_APT_SOURCE##
> -suite=##DISTRO_SUITE##-updates
> -components=##DISTRO_COMPONENTS##
> -omitdebsrc=true
> -
> -[security]
> -source=##DISTRO_APT_SOURCE_SEC##
> -suite=##DISTRO_SUITE##/updates
> -components=##DISTRO_COMPONENTS##
> -omitdebsrc=true
> -
> -[Isar]
> -packages=##IMAGE_INSTALL##
> -source=##DEPLOY_DIR_APT##
> -suite=##ISAR_DISTRO_SUITE##
> -components=main
> -omitdebsrc=true
> diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb
> index e359ac3..655dbb2 100644
> --- a/meta-isar/recipes-core/images/isar-image-base.bb
> +++ b/meta-isar/recipes-core/images/isar-image-base.bb
> @@ -9,8 +9,7 @@ LICENSE = "gpl-2.0"
> LIC_FILES_CHKSUM = "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
>
> FILESPATH =. "${LAYERDIR_isar}/recipes-core/images/files:"
> -SRC_URI = "file://multistrap.conf.in \
> - file://${DISTRO_CONFIG_SCRIPT} \
> +SRC_URI = "file://${DISTRO_CONFIG_SCRIPT} \
> file://setup.sh \
> file://download_dev-random"
>
> @@ -26,34 +25,119 @@ IMAGE_PREINSTALL += "apt \
>
> WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
>
> -do_rootfs[dirs] = "${WORKDIR}/hooks_multistrap"
> -
> -do_rootfs() {
> - E="${@ bb.utils.export_proxies(d)}"
> +do_multistrap_prepare[dirs] = "${WORKDIR}/hooks_multistrap"
>
> +do_multistrap_prepare() {
> chmod +x "${WORKDIR}/${DISTRO_CONFIG_SCRIPT}"
> chmod +x "${WORKDIR}/setup.sh"
> install -m 755 "${WORKDIR}/download_dev-random" "${WORKDIR}/hooks_multistrap/"
> +}
> +addtask multistrap_prepare before do_multistrap_config after do_unpack
> +
> +python do_multistrap_config() {
> + import shutil
> + import configparser as cp
> + import textwrap
> +
> + workdir = d.getVar("WORKDIR", True)
> + workdir_rel = os.path.relpath(workdir)
> +
> + multistrap_conf_file = os.path.join(workdir, "multistrap.conf")
> +
> + apt_sources_list = d.getVar("DEBIAN_APT_SOURCES", True).split()
> +
> + apt_preferences_list = d.getVar("DEBIAN_APT_PREFERENCES", True).split()
> + apt_preferences_out = os.path.join(workdir_rel, "apt-preferences")
> +
> + image_preinstall = d.getVar("IMAGE_PREINSTALL", True).split()
> + image_install = d.getVar("IMAGE_INSTALL", True).split()
> +
> + repos = []
> +
> + msconf = cp.RawConfigParser()
> +
> + # Set general multistrap configuration:
> + msconf.add_section("General")
> + msconf.set("General", "noauth", "true")
> + msconf.set("General", "unpack", "true")
> + msconf.set("General", "ignorenativearch", "true")
> + msconf.set("General", "aptpreferences", "./" + apt_preferences_out)
> + msconf.set("General", "configscript", "./" +
> + os.path.join(workdir_rel,
> + d.getVar("DISTRO_CONFIG_SCRIPT", True)))
> + msconf.set("General", "setupscript", "./" +
> + os.path.join(workdir_rel,
> + "setup.sh"))
> + msconf.set("General", "hookdir", "./" +
> + os.path.join(workdir_rel,
> + "hooks_multistrap"))
> +
> + # Set repository in multistrap configuration:
> + repo_counter = 0
> + for aptsrc in apt_sources_list:
> + aptsrc_real = bb.parse.resolve_file(aptsrc, d)
> + with open(aptsrc_real, "r") as in_fd:
> + for line in in_fd:
> + entry = line.split()
> +
> + # Only care about 'deb' repos for now:
> + if entry[0] != "deb":
> + continue
> +
> + # Generate on unique name for the section
> + repo_name = "repo{}".format(repo_counter)
> + repos.append(repo_name)
> +
> + source = entry[1]
> + suite = entry[2]
> + components = entry[3:]
> +
> + msconf.add_section(repo_name)
> + msconf.set(repo_name, "source", source)
> + msconf.set(repo_name, "suite", suite)
> + msconf.set(repo_name, "components", " ".join(components))
> + msconf.set(repo_name, "omitdebsrc", "true")
> +
> + # The first repo gets assigned all IMAGE_PREINSTALL
> + # packages:
> + if image_preinstall:
> + msconf.set(repo_name, "packages",
> + " ".join(image_preinstall))
> + image_preinstall = []
> +
> + repo_counter += 1
> +
> + msconf.add_section("isar-generated")
> + msconf.set("isar-generated", "packages", " ".join(image_install))
> + msconf.set("isar-generated", "source", "copy:///{}/{}".format(
> + d.getVar("DEPLOY_DIR_APT", True), d.getVar("DISTRO", True)))
> + msconf.set("isar-generated", "suite", d.getVar("DEBDISTRONAME", True))
> + msconf.set("isar-generated", "components", "main")
> + msconf.set("isar-generated", "omitdebsrc", "true")
> +
> + # Generate apt preference file:
> + with open(apt_preferences_out, "wb") as out_fd:
> + for aptpref in apt_preferences_list:
> + artpref_real = bb.parse.resolve_file(aptpref, d)
> + with open(aptpref_real, "rb") as in_fd:
> + shutil.copyfileobj(in_fd, out_fd, 1024*1024*10)
> + out_fd.write("\n".encode())
> + out_fd.write(textwrap.dedent("""\
> + Package: *
> + Pin: release n={}
> + Pin-Priority: 1000
> + """.format(d.getVar("DEBDISTRONAME", True))).encode())
> +
> + msconf.set("General", "bootstrap", " ".join(repos))
> + msconf.set("General", "aptsources", " ".join(repos + ["isar-generated"]))
> +
> + with open(multistrap_conf_file, "w") as out_fd:
> + msconf.write(out_fd)
> +}
> +addtask multistrap_config after do_multistrap_prepare before do_rootfs
>
> - # Multistrap accepts only relative path in configuration files, so get it:
> - cd ${TOPDIR}
> - WORKDIR_REL=${@ os.path.relpath(d.getVar("WORKDIR", True))}
> -
> - # Adjust multistrap config
> - sed -e 's|##IMAGE_PREINSTALL##|${IMAGE_PREINSTALL}|g' \
> - -e 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \
> - -e 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \
> - -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \
> - -e 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \
> - -e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \
> - -e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \
> - -e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/${DISTRO_CONFIG_SCRIPT}|g' \
> - -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \
> - -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
> - -e 's|##IMAGE_INSTALL##|${IMAGE_INSTALL}|g' \
> - -e 's|##DEPLOY_DIR_APT##|copy:///${DEPLOY_DIR_APT}/${DISTRO}|g' \
> - -e 's|##ISAR_DISTRO_SUITE##|${DEBDISTRONAME}|g' \
> - "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf"
> +do_rootfs() {
> + E="${@ bb.utils.export_proxies(d)}"
>
> [ ! -d ${IMAGE_ROOTFS}/proc ] && sudo install -d -o 0 -g 0 -m 555 ${IMAGE_ROOTFS}/proc
> sudo mount -t proc none ${IMAGE_ROOTFS}/proc
Hi,
thanks.... I think it goes into the right direction.
My proposal:
- Saving all Parts of conf/distro/stretch.list in bitbake variables, so
we can reuse them in other parts of Isar.
Maybe something like
DISTRO_CONFIG_LIST = "foo"
DISTRO_CONFIG_LIST[type] = "deb"
DISTRO_CONFIG_LIST[url] = "http://ftp.de.debian.org/debian "
DISTRO_CONFIG_LIST[distro] = "stretch"
DISTRO_CONFIG_LIST[components] = "main contrib non-free"
Those vars should be stored in global context which in turn would
presuppose
running do_multistrap_config() in a global context, in order to
getting access to those vars from everywhere.
I don't know if this possible? Other proposals are welcome.
Benni
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [RFC PATCH 1/1] RFC-PATCH: partial implementation for support of repos from other layers
2018-02-08 11:09 ` Benedikt Niedermayr
@ 2018-02-08 11:26 ` Jan Kiszka
2018-02-08 11:41 ` Claudius Heine
1 sibling, 0 replies; 5+ messages in thread
From: Jan Kiszka @ 2018-02-08 11:26 UTC (permalink / raw)
To: Benedikt Niedermayr, isar-users
On 2018-02-08 12:09, 'Benedikt Niedermayr' via isar-users wrote:
> Am 08.02.2018 um 10:46 schrieb claudius.heine.ext@siemens.com:
>> From: Claudius Heine <ch@denx.de>
>>
>> This patch changes the implementation of isar-image-base to support
>> the definition of additional repositories from other layers, by addding
>> apt source list compatible files to the 'DEBIAN_APT_SOURCES' variable.
>>
>> This also implement the generation of the multistrap.conf file from
>> within the recipe.
>>
>> Signed-off-by: Claudius Heine <ch@denx.de>
>> ---
>> meta-isar/conf/distro/debian-stretch.conf | 6 +-
>> meta-isar/conf/distro/stretch.list | 3 +
>> .../recipes-core/images/files/multistrap.conf.in | 38 ------
>> meta-isar/recipes-core/images/isar-image-base.bb | 134
>> +++++++++++++++++----
>> 4 files changed, 117 insertions(+), 64 deletions(-)
>> create mode 100644 meta-isar/conf/distro/stretch.list
>> delete mode 100644
>> meta-isar/recipes-core/images/files/multistrap.conf.in
>>
>> diff --git a/meta-isar/conf/distro/debian-stretch.conf
>> b/meta-isar/conf/distro/debian-stretch.conf
>> index 74177fd..083eb86 100644
>> --- a/meta-isar/conf/distro/debian-stretch.conf
>> +++ b/meta-isar/conf/distro/debian-stretch.conf
>> @@ -1,10 +1,14 @@
>> # This software is a part of ISAR.
>> # Copyright (C) 2017 ilbers GmbH
>> +DEBIAN_APT_SOURCES += "conf/distro/stretch.list"
>> +DEBIAN_APT_PREFERENCES ??= ""
>> +DISTRO_CONFIG_SCRIPT ??= "debian-configscript.sh"
>> +
>> +# No longer needed, after buildchroot is converted:
>> DISTRO_SUITE ?= "stretch"
>> DISTRO_COMPONENTS ?= "main contrib non-free"
>> DISTRO_APT_SOURCE ?= "http://ftp.debian.org/debian"
>> DISTRO_APT_SOURCE_SEC ?= "http://security.debian.org/debian-security"
>> -DISTRO_CONFIG_SCRIPT ?= "debian-configscript.sh"
>> DISTRO_MULTICONF_BOOTSTRAP ?= "base updates security"
>> DISTRO_MULTICONF_APTSOURCES ?= "${DISTRO_MULTICONF_BOOTSTRAP}"
>> diff --git a/meta-isar/conf/distro/stretch.list
>> b/meta-isar/conf/distro/stretch.list
>> new file mode 100644
>> index 0000000..a5ed5db
>> --- /dev/null
>> +++ b/meta-isar/conf/distro/stretch.list
>> @@ -0,0 +1,3 @@
>> +deb http://ftp.de.debian.org/debian stretch main contrib
>> non-free
>> +deb http://ftp.de.debian.org/debian stretch-updates main
>> contrib non-free
>> +deb http://security.debian.org stretch/updates main contrib
>> non-free
>> diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in
>> b/meta-isar/recipes-core/images/files/multistrap.conf.in
>> deleted file mode 100644
>> index 432b6af..0000000
>> --- a/meta-isar/recipes-core/images/files/multistrap.conf.in
>> +++ /dev/null
>> @@ -1,38 +0,0 @@
>> -# This software is a part of ISAR.
>> -# Copyright (C) 2015-2017 ilbers GmbH
>> -
>> -[General]
>> -noauth=true
>> -unpack=true
>> -ignorenativearch=true
>> -bootstrap=##DISTRO_MULTICONF_BOOTSTRAP## Isar
>> -aptsources=##DISTRO_MULTICONF_APTSOURCES##
>> -configscript=##CONFIG_SCRIPT##
>> -setupscript=##SETUP_SCRIPT##
>> -hookdir=##DIR_HOOKS##
>> -
>> -[base]
>> -source=##DISTRO_APT_SOURCE##
>> -suite=##DISTRO_SUITE##
>> -components=##DISTRO_COMPONENTS##
>> -packages=##IMAGE_PREINSTALL##
>> -omitdebsrc=true
>> -
>> -[updates]
>> -source=##DISTRO_APT_SOURCE##
>> -suite=##DISTRO_SUITE##-updates
>> -components=##DISTRO_COMPONENTS##
>> -omitdebsrc=true
>> -
>> -[security]
>> -source=##DISTRO_APT_SOURCE_SEC##
>> -suite=##DISTRO_SUITE##/updates
>> -components=##DISTRO_COMPONENTS##
>> -omitdebsrc=true
>> -
>> -[Isar]
>> -packages=##IMAGE_INSTALL##
>> -source=##DEPLOY_DIR_APT##
>> -suite=##ISAR_DISTRO_SUITE##
>> -components=main
>> -omitdebsrc=true
>> diff --git a/meta-isar/recipes-core/images/isar-image-base.bb
>> b/meta-isar/recipes-core/images/isar-image-base.bb
>> index e359ac3..655dbb2 100644
>> --- a/meta-isar/recipes-core/images/isar-image-base.bb
>> +++ b/meta-isar/recipes-core/images/isar-image-base.bb
>> @@ -9,8 +9,7 @@ LICENSE = "gpl-2.0"
>> LIC_FILES_CHKSUM =
>> "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
>>
>> FILESPATH =. "${LAYERDIR_isar}/recipes-core/images/files:"
>> -SRC_URI = "file://multistrap.conf.in \
>> - file://${DISTRO_CONFIG_SCRIPT} \
>> +SRC_URI = "file://${DISTRO_CONFIG_SCRIPT} \
>> file://setup.sh \
>> file://download_dev-random"
>> @@ -26,34 +25,119 @@ IMAGE_PREINSTALL += "apt \
>> WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
>> -do_rootfs[dirs] = "${WORKDIR}/hooks_multistrap"
>> -
>> -do_rootfs() {
>> - E="${@ bb.utils.export_proxies(d)}"
>> +do_multistrap_prepare[dirs] = "${WORKDIR}/hooks_multistrap"
>> +do_multistrap_prepare() {
>> chmod +x "${WORKDIR}/${DISTRO_CONFIG_SCRIPT}"
>> chmod +x "${WORKDIR}/setup.sh"
>> install -m 755 "${WORKDIR}/download_dev-random"
>> "${WORKDIR}/hooks_multistrap/"
>> +}
>> +addtask multistrap_prepare before do_multistrap_config after do_unpack
>> +
>> +python do_multistrap_config() {
>> + import shutil
>> + import configparser as cp
>> + import textwrap
>> +
>> + workdir = d.getVar("WORKDIR", True)
>> + workdir_rel = os.path.relpath(workdir)
>> +
>> + multistrap_conf_file = os.path.join(workdir, "multistrap.conf")
>> +
>> + apt_sources_list = d.getVar("DEBIAN_APT_SOURCES", True).split()
>> +
>> + apt_preferences_list = d.getVar("DEBIAN_APT_PREFERENCES",
>> True).split()
>> + apt_preferences_out = os.path.join(workdir_rel, "apt-preferences")
>> +
>> + image_preinstall = d.getVar("IMAGE_PREINSTALL", True).split()
>> + image_install = d.getVar("IMAGE_INSTALL", True).split()
>> +
>> + repos = []
>> +
>> + msconf = cp.RawConfigParser()
>> +
>> + # Set general multistrap configuration:
>> + msconf.add_section("General")
>> + msconf.set("General", "noauth", "true")
>> + msconf.set("General", "unpack", "true")
>> + msconf.set("General", "ignorenativearch", "true")
>> + msconf.set("General", "aptpreferences", "./" + apt_preferences_out)
>> + msconf.set("General", "configscript", "./" +
>> + os.path.join(workdir_rel,
>> + d.getVar("DISTRO_CONFIG_SCRIPT", True)))
>> + msconf.set("General", "setupscript", "./" +
>> + os.path.join(workdir_rel,
>> + "setup.sh"))
>> + msconf.set("General", "hookdir", "./" +
>> + os.path.join(workdir_rel,
>> + "hooks_multistrap"))
>> +
>> + # Set repository in multistrap configuration:
>> + repo_counter = 0
>> + for aptsrc in apt_sources_list:
>> + aptsrc_real = bb.parse.resolve_file(aptsrc, d)
>> + with open(aptsrc_real, "r") as in_fd:
>> + for line in in_fd:
>> + entry = line.split()
>> +
>> + # Only care about 'deb' repos for now:
>> + if entry[0] != "deb":
>> + continue
>> +
>> + # Generate on unique name for the section
>> + repo_name = "repo{}".format(repo_counter)
>> + repos.append(repo_name)
>> +
>> + source = entry[1]
>> + suite = entry[2]
>> + components = entry[3:]
>> +
>> + msconf.add_section(repo_name)
>> + msconf.set(repo_name, "source", source)
>> + msconf.set(repo_name, "suite", suite)
>> + msconf.set(repo_name, "components", "
>> ".join(components))
>> + msconf.set(repo_name, "omitdebsrc", "true")
>> +
>> + # The first repo gets assigned all IMAGE_PREINSTALL
>> + # packages:
>> + if image_preinstall:
>> + msconf.set(repo_name, "packages",
>> + " ".join(image_preinstall))
>> + image_preinstall = []
>> +
>> + repo_counter += 1
>> +
>> + msconf.add_section("isar-generated")
>> + msconf.set("isar-generated", "packages", " ".join(image_install))
>> + msconf.set("isar-generated", "source", "copy:///{}/{}".format(
>> + d.getVar("DEPLOY_DIR_APT", True), d.getVar("DISTRO",
>> True)))
>> + msconf.set("isar-generated", "suite", d.getVar("DEBDISTRONAME",
>> True))
>> + msconf.set("isar-generated", "components", "main")
>> + msconf.set("isar-generated", "omitdebsrc", "true")
>> +
>> + # Generate apt preference file:
>> + with open(apt_preferences_out, "wb") as out_fd:
>> + for aptpref in apt_preferences_list:
>> + artpref_real = bb.parse.resolve_file(aptpref, d)
>> + with open(aptpref_real, "rb") as in_fd:
>> + shutil.copyfileobj(in_fd, out_fd, 1024*1024*10)
>> + out_fd.write("\n".encode())
>> + out_fd.write(textwrap.dedent("""\
>> + Package: *
>> + Pin: release n={}
>> + Pin-Priority: 1000
>> + """.format(d.getVar("DEBDISTRONAME",
>> True))).encode())
>> +
>> + msconf.set("General", "bootstrap", " ".join(repos))
>> + msconf.set("General", "aptsources", " ".join(repos +
>> ["isar-generated"]))
>> +
>> + with open(multistrap_conf_file, "w") as out_fd:
>> + msconf.write(out_fd)
>> +}
>> +addtask multistrap_config after do_multistrap_prepare before do_rootfs
>> - # Multistrap accepts only relative path in configuration files,
>> so get it:
>> - cd ${TOPDIR}
>> - WORKDIR_REL=${@ os.path.relpath(d.getVar("WORKDIR", True))}
>> -
>> - # Adjust multistrap config
>> - sed -e 's|##IMAGE_PREINSTALL##|${IMAGE_PREINSTALL}|g' \
>> - -e
>> 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \
>> - -e
>> 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \
>> - -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \
>> - -e 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \
>> - -e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \
>> - -e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \
>> - -e
>> 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/${DISTRO_CONFIG_SCRIPT}|g' \
>> - -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \
>> - -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
>> - -e 's|##IMAGE_INSTALL##|${IMAGE_INSTALL}|g' \
>> - -e
>> 's|##DEPLOY_DIR_APT##|copy:///${DEPLOY_DIR_APT}/${DISTRO}|g' \
>> - -e 's|##ISAR_DISTRO_SUITE##|${DEBDISTRONAME}|g' \
>> - "${WORKDIR}/multistrap.conf.in" >
>> "${WORKDIR}/multistrap.conf"
>> +do_rootfs() {
>> + E="${@ bb.utils.export_proxies(d)}"
>> [ ! -d ${IMAGE_ROOTFS}/proc ] && sudo install -d -o 0 -g 0 -m
>> 555 ${IMAGE_ROOTFS}/proc
>> sudo mount -t proc none ${IMAGE_ROOTFS}/proc
> Hi,
>
> thanks.... I think it goes into the right direction.
>
> My proposal:
>
> - Saving all Parts of conf/distro/stretch.list in bitbake variables, so
> we can reuse them in other parts of Isar.
> Maybe something like
> DISTRO_CONFIG_LIST = "foo"
> DISTRO_CONFIG_LIST[type] = "deb"
> DISTRO_CONFIG_LIST[url] = "http://ftp.de.debian.org/debian "
> DISTRO_CONFIG_LIST[distro] = "stretch"
> DISTRO_CONFIG_LIST[components] = "main contrib non-free"
>
> Those vars should be stored in global context which in turn would
> presuppose
> running do_multistrap_config() in a global context, in order to
> getting access to those vars from everywhere.
The variables can come from normal config files (distro.conf, multiconf
etc.) which are generally available.
However, I think that distributing the per-repo information in such a
way over multiple bitbake variables will make them ugly to parse and
highly fragile. E.g., when someone adds a repo but forgets to specify
one of the 4 parameters, the array will become unbalanced and the
association will be subtly wrong. Therefore the proposal to list
everything in sources.list format in fragment files.
Jan
--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [RFC PATCH 1/1] RFC-PATCH: partial implementation for support of repos from other layers
2018-02-08 11:09 ` Benedikt Niedermayr
2018-02-08 11:26 ` Jan Kiszka
@ 2018-02-08 11:41 ` Claudius Heine
1 sibling, 0 replies; 5+ messages in thread
From: Claudius Heine @ 2018-02-08 11:41 UTC (permalink / raw)
To: Benedikt Niedermayr, isar-users
Hi Benedikt,
On 02/08/2018 12:09 PM, 'Benedikt Niedermayr' via isar-users wrote:
> Am 08.02.2018 um 10:46 schrieb claudius.heine.ext@siemens.com:
>> From: Claudius Heine <ch@denx.de>
>>
>> This patch changes the implementation of isar-image-base to support
>> the definition of additional repositories from other layers, by addding
>> apt source list compatible files to the 'DEBIAN_APT_SOURCES' variable.
>>
>> This also implement the generation of the multistrap.conf file from
>> within the recipe.
>>
>> Signed-off-by: Claudius Heine <ch@denx.de>
>> ---
>> meta-isar/conf/distro/debian-stretch.conf | 6 +-
>> meta-isar/conf/distro/stretch.list | 3 +
>> .../recipes-core/images/files/multistrap.conf.in | 38 ------
>> meta-isar/recipes-core/images/isar-image-base.bb | 134
>> +++++++++++++++++----
>> 4 files changed, 117 insertions(+), 64 deletions(-)
>> create mode 100644 meta-isar/conf/distro/stretch.list
>> delete mode 100644
>> meta-isar/recipes-core/images/files/multistrap.conf.in
>>
>> diff --git a/meta-isar/conf/distro/debian-stretch.conf
>> b/meta-isar/conf/distro/debian-stretch.conf
>> index 74177fd..083eb86 100644
>> --- a/meta-isar/conf/distro/debian-stretch.conf
>> +++ b/meta-isar/conf/distro/debian-stretch.conf
>> @@ -1,10 +1,14 @@
>> # This software is a part of ISAR.
>> # Copyright (C) 2017 ilbers GmbH
>> +DEBIAN_APT_SOURCES += "conf/distro/stretch.list"
>> +DEBIAN_APT_PREFERENCES ??= ""
>> +DISTRO_CONFIG_SCRIPT ??= "debian-configscript.sh"
>> +
>> +# No longer needed, after buildchroot is converted:
>> DISTRO_SUITE ?= "stretch"
>> DISTRO_COMPONENTS ?= "main contrib non-free"
>> DISTRO_APT_SOURCE ?= "http://ftp.debian.org/debian"
>> DISTRO_APT_SOURCE_SEC ?= "http://security.debian.org/debian-security"
>> -DISTRO_CONFIG_SCRIPT ?= "debian-configscript.sh"
>> DISTRO_MULTICONF_BOOTSTRAP ?= "base updates security"
>> DISTRO_MULTICONF_APTSOURCES ?= "${DISTRO_MULTICONF_BOOTSTRAP}"
>> diff --git a/meta-isar/conf/distro/stretch.list
>> b/meta-isar/conf/distro/stretch.list
>> new file mode 100644
>> index 0000000..a5ed5db
>> --- /dev/null
>> +++ b/meta-isar/conf/distro/stretch.list
>> @@ -0,0 +1,3 @@
>> +deb http://ftp.de.debian.org/debian stretch main contrib
>> non-free
>> +deb http://ftp.de.debian.org/debian stretch-updates main
>> contrib non-free
>> +deb http://security.debian.org stretch/updates main contrib
>> non-free
>> diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in
>> b/meta-isar/recipes-core/images/files/multistrap.conf.in
>> deleted file mode 100644
>> index 432b6af..0000000
>> --- a/meta-isar/recipes-core/images/files/multistrap.conf.in
>> +++ /dev/null
>> @@ -1,38 +0,0 @@
>> -# This software is a part of ISAR.
>> -# Copyright (C) 2015-2017 ilbers GmbH
>> -
>> -[General]
>> -noauth=true
>> -unpack=true
>> -ignorenativearch=true
>> -bootstrap=##DISTRO_MULTICONF_BOOTSTRAP## Isar
>> -aptsources=##DISTRO_MULTICONF_APTSOURCES##
>> -configscript=##CONFIG_SCRIPT##
>> -setupscript=##SETUP_SCRIPT##
>> -hookdir=##DIR_HOOKS##
>> -
>> -[base]
>> -source=##DISTRO_APT_SOURCE##
>> -suite=##DISTRO_SUITE##
>> -components=##DISTRO_COMPONENTS##
>> -packages=##IMAGE_PREINSTALL##
>> -omitdebsrc=true
>> -
>> -[updates]
>> -source=##DISTRO_APT_SOURCE##
>> -suite=##DISTRO_SUITE##-updates
>> -components=##DISTRO_COMPONENTS##
>> -omitdebsrc=true
>> -
>> -[security]
>> -source=##DISTRO_APT_SOURCE_SEC##
>> -suite=##DISTRO_SUITE##/updates
>> -components=##DISTRO_COMPONENTS##
>> -omitdebsrc=true
>> -
>> -[Isar]
>> -packages=##IMAGE_INSTALL##
>> -source=##DEPLOY_DIR_APT##
>> -suite=##ISAR_DISTRO_SUITE##
>> -components=main
>> -omitdebsrc=true
>> diff --git a/meta-isar/recipes-core/images/isar-image-base.bb
>> b/meta-isar/recipes-core/images/isar-image-base.bb
>> index e359ac3..655dbb2 100644
>> --- a/meta-isar/recipes-core/images/isar-image-base.bb
>> +++ b/meta-isar/recipes-core/images/isar-image-base.bb
>> @@ -9,8 +9,7 @@ LICENSE = "gpl-2.0"
>> LIC_FILES_CHKSUM =
>> "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
>>
>> FILESPATH =. "${LAYERDIR_isar}/recipes-core/images/files:"
>> -SRC_URI = "file://multistrap.conf.in \
>> - file://${DISTRO_CONFIG_SCRIPT} \
>> +SRC_URI = "file://${DISTRO_CONFIG_SCRIPT} \
>> file://setup.sh \
>> file://download_dev-random"
>> @@ -26,34 +25,119 @@ IMAGE_PREINSTALL += "apt \
>> WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
>> -do_rootfs[dirs] = "${WORKDIR}/hooks_multistrap"
>> -
>> -do_rootfs() {
>> - E="${@ bb.utils.export_proxies(d)}"
>> +do_multistrap_prepare[dirs] = "${WORKDIR}/hooks_multistrap"
>> +do_multistrap_prepare() {
>> chmod +x "${WORKDIR}/${DISTRO_CONFIG_SCRIPT}"
>> chmod +x "${WORKDIR}/setup.sh"
>> install -m 755 "${WORKDIR}/download_dev-random"
>> "${WORKDIR}/hooks_multistrap/"
>> +}
>> +addtask multistrap_prepare before do_multistrap_config after do_unpack
>> +
>> +python do_multistrap_config() {
>> + import shutil
>> + import configparser as cp
>> + import textwrap
>> +
>> + workdir = d.getVar("WORKDIR", True)
>> + workdir_rel = os.path.relpath(workdir)
>> +
>> + multistrap_conf_file = os.path.join(workdir, "multistrap.conf")
>> +
>> + apt_sources_list = d.getVar("DEBIAN_APT_SOURCES", True).split()
>> +
>> + apt_preferences_list = d.getVar("DEBIAN_APT_PREFERENCES",
>> True).split()
>> + apt_preferences_out = os.path.join(workdir_rel, "apt-preferences")
>> +
>> + image_preinstall = d.getVar("IMAGE_PREINSTALL", True).split()
>> + image_install = d.getVar("IMAGE_INSTALL", True).split()
>> +
>> + repos = []
>> +
>> + msconf = cp.RawConfigParser()
>> +
>> + # Set general multistrap configuration:
>> + msconf.add_section("General")
>> + msconf.set("General", "noauth", "true")
>> + msconf.set("General", "unpack", "true")
>> + msconf.set("General", "ignorenativearch", "true")
>> + msconf.set("General", "aptpreferences", "./" + apt_preferences_out)
>> + msconf.set("General", "configscript", "./" +
>> + os.path.join(workdir_rel,
>> + d.getVar("DISTRO_CONFIG_SCRIPT", True)))
>> + msconf.set("General", "setupscript", "./" +
>> + os.path.join(workdir_rel,
>> + "setup.sh"))
>> + msconf.set("General", "hookdir", "./" +
>> + os.path.join(workdir_rel,
>> + "hooks_multistrap"))
>> +
>> + # Set repository in multistrap configuration:
>> + repo_counter = 0
>> + for aptsrc in apt_sources_list:
>> + aptsrc_real = bb.parse.resolve_file(aptsrc, d)
>> + with open(aptsrc_real, "r") as in_fd:
>> + for line in in_fd:
>> + entry = line.split()
>> +
>> + # Only care about 'deb' repos for now:
>> + if entry[0] != "deb":
>> + continue
>> +
>> + # Generate on unique name for the section
>> + repo_name = "repo{}".format(repo_counter)
>> + repos.append(repo_name)
>> +
>> + source = entry[1]
>> + suite = entry[2]
>> + components = entry[3:]
>> +
>> + msconf.add_section(repo_name)
>> + msconf.set(repo_name, "source", source)
>> + msconf.set(repo_name, "suite", suite)
>> + msconf.set(repo_name, "components", "
>> ".join(components))
>> + msconf.set(repo_name, "omitdebsrc", "true")
>> +
>> + # The first repo gets assigned all IMAGE_PREINSTALL
>> + # packages:
>> + if image_preinstall:
>> + msconf.set(repo_name, "packages",
>> + " ".join(image_preinstall))
>> + image_preinstall = []
>> +
>> + repo_counter += 1
>> +
>> + msconf.add_section("isar-generated")
>> + msconf.set("isar-generated", "packages", " ".join(image_install))
>> + msconf.set("isar-generated", "source", "copy:///{}/{}".format(
>> + d.getVar("DEPLOY_DIR_APT", True), d.getVar("DISTRO",
>> True)))
>> + msconf.set("isar-generated", "suite", d.getVar("DEBDISTRONAME",
>> True))
>> + msconf.set("isar-generated", "components", "main")
>> + msconf.set("isar-generated", "omitdebsrc", "true")
>> +
>> + # Generate apt preference file:
>> + with open(apt_preferences_out, "wb") as out_fd:
>> + for aptpref in apt_preferences_list:
>> + artpref_real = bb.parse.resolve_file(aptpref, d)
>> + with open(aptpref_real, "rb") as in_fd:
>> + shutil.copyfileobj(in_fd, out_fd, 1024*1024*10)
>> + out_fd.write("\n".encode())
>> + out_fd.write(textwrap.dedent("""\
>> + Package: *
>> + Pin: release n={}
>> + Pin-Priority: 1000
>> + """.format(d.getVar("DEBDISTRONAME",
>> True))).encode())
>> +
>> + msconf.set("General", "bootstrap", " ".join(repos))
>> + msconf.set("General", "aptsources", " ".join(repos +
>> ["isar-generated"]))
>> +
>> + with open(multistrap_conf_file, "w") as out_fd:
>> + msconf.write(out_fd)
>> +}
>> +addtask multistrap_config after do_multistrap_prepare before do_rootfs
>> - # Multistrap accepts only relative path in configuration files,
>> so get it:
>> - cd ${TOPDIR}
>> - WORKDIR_REL=${@ os.path.relpath(d.getVar("WORKDIR", True))}
>> -
>> - # Adjust multistrap config
>> - sed -e 's|##IMAGE_PREINSTALL##|${IMAGE_PREINSTALL}|g' \
>> - -e
>> 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \
>> - -e
>> 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \
>> - -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \
>> - -e 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \
>> - -e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \
>> - -e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \
>> - -e
>> 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/${DISTRO_CONFIG_SCRIPT}|g' \
>> - -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \
>> - -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
>> - -e 's|##IMAGE_INSTALL##|${IMAGE_INSTALL}|g' \
>> - -e
>> 's|##DEPLOY_DIR_APT##|copy:///${DEPLOY_DIR_APT}/${DISTRO}|g' \
>> - -e 's|##ISAR_DISTRO_SUITE##|${DEBDISTRONAME}|g' \
>> - "${WORKDIR}/multistrap.conf.in" >
>> "${WORKDIR}/multistrap.conf"
>> +do_rootfs() {
>> + E="${@ bb.utils.export_proxies(d)}"
>> [ ! -d ${IMAGE_ROOTFS}/proc ] && sudo install -d -o 0 -g 0 -m
>> 555 ${IMAGE_ROOTFS}/proc
>> sudo mount -t proc none ${IMAGE_ROOTFS}/proc
> Hi,
>
> thanks.... I think it goes into the right direction.
>
> My proposal:
>
> - Saving all Parts of conf/distro/stretch.list in bitbake variables, so
> we can reuse them in other parts of Isar.
> Maybe something like
> DISTRO_CONFIG_LIST = "foo"
> DISTRO_CONFIG_LIST[type] = "deb"
> DISTRO_CONFIG_LIST[url] = "http://ftp.de.debian.org/debian "
> DISTRO_CONFIG_LIST[distro] = "stretch"
> DISTRO_CONFIG_LIST[components] = "main contrib non-free"
>
> Those vars should be stored in global context which in turn would
> presuppose
> running do_multistrap_config() in a global context, in order to
> getting access to those vars from everywhere.
>
> I don't know if this possible? Other proposals are welcome.
I thought about doing it this way before, but the problem comes when
people start adding repos to this list in their layers. Then the code
has to figure out the matching entries in these lists. Not sure how to
do that in a robust way. For example its pretty easy to forget adding
one entry one of those list and causing difficulties that are hard to debug.
If bitbake would support storing objects in a list, then that would be a
possibility. But since it AFAIK only stores strings, and I don't want to
introduce an additional separation character, I prefer dealing with it
this way.
Also I don't really see a use-case to have those settings available,
because bitbake should only operate on "DISTRO_ARCH" and "DISTRO"
variables, they should be enough in most cases. If you can think of one,
that we can reconsider.
Of course changing the url in some cases to a local mirror might be
useful, but that can be implemented easily later following the same
scheme openembedded does with "PREMIRRORS".
Cheers,
Claudius
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2018-02-08 11:41 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-02-08 9:46 [RFC PATCH 0/1] Multi-Repo and apt preferences support claudius.heine.ext
2018-02-08 9:46 ` [RFC PATCH 1/1] RFC-PATCH: partial implementation for support of repos from other layers claudius.heine.ext
2018-02-08 11:09 ` Benedikt Niedermayr
2018-02-08 11:26 ` Jan Kiszka
2018-02-08 11:41 ` Claudius Heine
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox