[PATCH v3 0/8] Cleanup rootfs creation

[PATCH v3 0/8] Cleanup rootfs creation

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

Hi,

I could not reproduce the raspberrypi issue Maxim found in his CI build:
http://isar-build.org:8080/job/isar_mosipov_develop/91/consoleText

This is my build log:
http://isar-build.org:8080/job/isar_claudius_ilbers-ci/54/consoleText

Only the module loading fails, which Maxim stated he will look into.

This patchset uses still the 'old' style for refactoring/cleanup patchsets,
until I automated the official patch submission process as documented by the
CONTRIBUTERS policy.

You can find the whole patchset in under this url:

https://github.com/cmhe/isar/tree/ch/cleanup

regards,
Claudius

changes from v2:
 - replaced 'ROOTFS_ARCH' with 'HOST_ARCH' in sdkchroot.bb to avoid
   populate_sdk issue

changes from v1:
 - fixed typo in commit message

Claudius Heine (8):
  isar-boostrap-helper: move 'HOST_ARCH' and 'HOST_DISTRO' to
    base.bbclass
  move 'HOST_DISTRO_APT_SOURCES' from bootstrap-helper to isar-bootstrap
  buildchroot.bbclass: only cross build if HOST_ARCH != DISTRO_ARCH
  isar-bootstrap/buildchroot/sdkchroot: refactor PF and WORKDIR
  bitbake.conf: remove unneeded and differently used variables
  image.bbclass: make IMAGE_ROOTFS overwritable
  bitbake.conf: set default QEMU_ARCH variables
  buildchroot/configscript: make creation of builder uid/gid idempotent

 meta/classes/base.bbclass                          |  9 +++++++++
 meta/classes/buildchroot.bbclass                   |  2 +-
 meta/classes/image-sdk-extension.bbclass           |  2 +-
 meta/classes/image.bbclass                         |  2 +-
 meta/classes/isar-bootstrap-helper.bbclass         | 14 --------------
 meta/conf/bitbake.conf                             | 13 ++++++++-----
 .../isar-bootstrap/isar-bootstrap-host.bb          | 11 +++--------
 .../isar-bootstrap/isar-bootstrap-target.bb        |  6 +-----
 .../recipes-core/isar-bootstrap/isar-bootstrap.inc |  1 +
 .../buildchroot/buildchroot-host.bb                |  1 +
 meta/recipes-devtools/buildchroot/buildchroot.inc  |  3 +--
 .../buildchroot/files/configscript.sh              |  4 ++--
 meta/recipes-devtools/sdkchroot/sdkchroot.bb       |  3 +--
 13 files changed, 30 insertions(+), 41 deletions(-)

-- 
2.20.1

[PATCH v3 1/8] isar-boostrap-helper: move 'HOST_ARCH' and 'HOST_DISTRO' to base.bbclass

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

This also sets the 'HOST_DISTRO' to 'DISTRO' per default. (jessie
support was dropped.)

Signed-off-by: Claudius Heine <ch@denx.de>
---
 meta/classes/base.bbclass                  |  9 +++++++++
 meta/classes/isar-bootstrap-helper.bbclass | 12 ------------
 2 files changed, 9 insertions(+), 12 deletions(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 9a1837f..ca80bf8 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -20,6 +20,15 @@
 
 THISDIR = "${@os.path.dirname(d.getVar('FILE', True))}"
 
+def get_deb_host_arch():
+    import subprocess
+    host_arch = subprocess.check_output(
+        ["dpkg", "--print-architecture"]
+    ).decode('utf-8').strip()
+    return host_arch
+HOST_ARCH ??= "${@get_deb_host_arch()}"
+HOST_DISTRO ??= "${DISTRO}"
+
 die() {
 	bbfatal "$*"
 }
diff --git a/meta/classes/isar-bootstrap-helper.bbclass b/meta/classes/isar-bootstrap-helper.bbclass
index 38d90fb..59c447b 100644
--- a/meta/classes/isar-bootstrap-helper.bbclass
+++ b/meta/classes/isar-bootstrap-helper.bbclass
@@ -7,18 +7,6 @@
 
 IMAGE_TRANSIENT_PACKAGES ??= ""
 
-def get_deb_host_arch():
-    import subprocess
-    host_arch = subprocess.check_output(
-        ["dpkg", "--print-architecture"]
-    ).decode('utf-8').strip()
-    return host_arch
-
-#Debian Distribution for SDK host
-HOST_DISTRO ?= "debian-stretch"
-#Determine SDK host architecture if not explicitly set
-HOST_ARCH ?= "${@get_deb_host_arch()}"
-
 HOST_DISTRO_APT_SOURCES += "conf/distro/${HOST_DISTRO}.list"
 
 def reverse_bb_array(d, varname):
-- 
2.20.1

[PATCH v3 2/8] move 'HOST_DISTRO_APT_SOURCES' from bootstrap-helper to isar-bootstrap

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

This variable is only used in 'isar-bootstrap-host.bb' and mentioned in
'isar-bootstrap.inc'. So it does not make sense to have it available
anywhere else.

Signed-off-by: Claudius Heine <ch@denx.de>
---
 meta/classes/isar-bootstrap-helper.bbclass          | 2 --
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 1 +
 2 files changed, 1 insertion(+), 2 deletions(-)

diff --git a/meta/classes/isar-bootstrap-helper.bbclass b/meta/classes/isar-bootstrap-helper.bbclass
index 59c447b..0d77929 100644
--- a/meta/classes/isar-bootstrap-helper.bbclass
+++ b/meta/classes/isar-bootstrap-helper.bbclass
@@ -7,8 +7,6 @@
 
 IMAGE_TRANSIENT_PACKAGES ??= ""
 
-HOST_DISTRO_APT_SOURCES += "conf/distro/${HOST_DISTRO}.list"
-
 def reverse_bb_array(d, varname):
     array = d.getVar(varname, True)
     if array is None:
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index c1b571a..0ec2e2b 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -27,6 +27,7 @@ APTKEYRING = "${WORKDIR}/apt-keyring.gpg"
 DEBOOTSTRAP_KEYRING = ""
 DEPLOY_ISAR_BOOTSTRAP ?= ""
 DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales"
+HOST_DISTRO_APT_SOURCES += "conf/distro/${HOST_DISTRO}.list"
 
 DISTRO_APT_PREMIRRORS ?= "${@ "http://ftp\.(\S+\.)?debian.org  file:///${REPO_BASE_DIR} \n" if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) else "" }"
 
-- 
2.20.1

[PATCH v3 3/8] buildchroot.bbclass: only cross build if HOST_ARCH != DISTRO_ARCH

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

Signed-off-by: Claudius Heine <ch@denx.de>
---
 meta/classes/buildchroot.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/buildchroot.bbclass b/meta/classes/buildchroot.bbclass
index 1a87bf4..ea1538a 100644
--- a/meta/classes/buildchroot.bbclass
+++ b/meta/classes/buildchroot.bbclass
@@ -8,7 +8,7 @@ ISAR_CROSS_COMPILE ??= "0"
 # Add dependency from the correct buildchroot: host or target
 python __anonymous() {
     mode = d.getVar('ISAR_CROSS_COMPILE', True)
-    if mode == "0":
+    if mode == "0" or d.getVar('HOST_ARCH') == d.getVar('DISTRO_ARCH'):
         dep = "buildchroot-target:do_build"
         rootfs = d.getVar('BUILDCHROOT_TARGET_DIR', True)
     else:
-- 
2.20.1

[PATCH v3 4/8] isar-bootstrap/buildchroot/sdkchroot: refactor PF and WORKDIR

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

Each of those are dependent on both HOST_ARCH and DISTRO_ARCH, so the
stamps etc. should not be shared.

Signed-off-by: Claudius Heine <ch@denx.de>
---
 meta/classes/image-sdk-extension.bbclass              |  2 +-
 meta/conf/bitbake.conf                                |  4 ++--
 .../isar-bootstrap/isar-bootstrap-host.bb             | 11 +++--------
 .../isar-bootstrap/isar-bootstrap-target.bb           |  6 +-----
 meta/recipes-devtools/buildchroot/buildchroot-host.bb |  1 +
 meta/recipes-devtools/buildchroot/buildchroot.inc     |  3 +--
 meta/recipes-devtools/sdkchroot/sdkchroot.bb          |  3 +--
 7 files changed, 10 insertions(+), 20 deletions(-)

diff --git a/meta/classes/image-sdk-extension.bbclass b/meta/classes/image-sdk-extension.bbclass
index 1838080..8df3ed1 100644
--- a/meta/classes/image-sdk-extension.bbclass
+++ b/meta/classes/image-sdk-extension.bbclass
@@ -5,7 +5,7 @@
 #
 # This class extends the image.bbclass to supply the creation of a sdk
 
-SDKCHROOT_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/sdkchroot-${HOST_DISTRO}-${HOST_ARCH}"
+SDKCHROOT_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/sdkchroot-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"
 
 do_populate_sdk[stamp-extra-info] = "${DISTRO}-${MACHINE}"
 do_populate_sdk[depends] = "sdkchroot:do_build"
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 0e521bb..7c1c89f 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -50,8 +50,8 @@ DEPLOY_DIR_BOOTSTRAP = "${DEPLOY_DIR}/bootstrap"
 DEPLOY_DIR_IMAGE = "${DEPLOY_DIR}/images/${MACHINE}"
 DL_DIR ?= "${TOPDIR}/downloads"
 SSTATE_DIR ?= "${TMPDIR}/sstate-cache"
-BUILDCHROOT_HOST_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot-host/rootfs"
-BUILDCHROOT_TARGET_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot-target/rootfs"
+BUILDCHROOT_HOST_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot-host-${DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}/rootfs"
+BUILDCHROOT_TARGET_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot-target-${DISTRO}-${DISTRO_ARCH}/rootfs"
 CACHE = "${TMPDIR}/cache"
 
 OVERRIDES = "${DISTRO_ARCH}:${MACHINE}:${DISTRO}:forcevariable"
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
index 08b068f..6d0f5f5 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
@@ -6,17 +6,13 @@
 # SPDX-License-Identifier: MIT
 
 Description = "Minimal host Debian root file system"
+PF = "${PN}-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"
 
-WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}-${HOST_DISTRO}-${HOST_ARCH}"
-DEPLOY_ISAR_BOOTSTRAP = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}"
-ISAR_BOOTSTRAP_LOCK = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}.lock"
+DEPLOY_ISAR_BOOTSTRAP = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"
+ISAR_BOOTSTRAP_LOCK = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}.lock"
 
 require isar-bootstrap.inc
-inherit isar-bootstrap-helper
 
-do_generate_keyring[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
-
-do_apt_config_prepare[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 do_apt_config_prepare[dirs] = "${WORKDIR}"
 do_apt_config_prepare[vardeps] += "\
                                    APTPREFS \
@@ -47,7 +43,6 @@ addtask apt_config_prepare before do_bootstrap after do_unpack
 
 OVERRIDES_append = ":${@get_distro_needs_https_support(d, True)}"
 
-do_bootstrap[stamp-extra-info] = "${HOST_DISTRO}-${HOST_ARCH}"
 do_bootstrap[vardeps] += "HOST_DISTRO_APT_SOURCES"
 do_bootstrap() {
     isar_bootstrap --host
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
index 79f3e34..552a317 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
@@ -6,16 +6,13 @@
 # SPDX-License-Identifier: MIT
 
 Description = "Minimal target Debian root file system"
+PF = "${PN}-${DISTRO}-${DISTRO_ARCH}"
 
-WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
 DEPLOY_ISAR_BOOTSTRAP = "${DEPLOY_DIR_BOOTSTRAP}/${DISTRO}-${DISTRO_ARCH}"
 ISAR_BOOTSTRAP_LOCK = "${DEPLOY_DIR_BOOTSTRAP}/${DISTRO}-${DISTRO_ARCH}.lock"
 
 require isar-bootstrap.inc
 
-do_generate_keyring[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
-
-do_apt_config_prepare[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 do_apt_config_prepare[dirs] = "${WORKDIR}"
 do_apt_config_prepare[vardeps] += "\
                                    APTPREFS \
@@ -44,7 +41,6 @@ addtask apt_config_prepare before do_bootstrap after do_unpack
 
 OVERRIDES_append = ":${@get_distro_needs_https_support(d, False)}"
 
-do_bootstrap[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 do_bootstrap[vardeps] += "DISTRO_APT_SOURCES"
 do_bootstrap() {
     isar_bootstrap
diff --git a/meta/recipes-devtools/buildchroot/buildchroot-host.bb b/meta/recipes-devtools/buildchroot/buildchroot-host.bb
index c8c10c0..e8f7f69 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot-host.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot-host.bb
@@ -6,6 +6,7 @@
 DESCRIPTION = "Isar development filesystem for host"
 
 require buildchroot.inc
+PF = "${PN}-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"
 
 BUILDCHROOT_PREINSTALL ?= "make \
                            debhelper \
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.inc b/meta/recipes-devtools/buildchroot/buildchroot.inc
index 764bc5e..cdb4c07 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.inc
+++ b/meta/recipes-devtools/buildchroot/buildchroot.inc
@@ -12,13 +12,12 @@ SRC_URI = "file://configscript.sh \
            file://common.sh \
            file://deps.sh"
 PV = "1.0"
+PF = "${PN}-${DISTRO}-${DISTRO_ARCH}"
 
 inherit isar-bootstrap-helper
 
-WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
 BUILDCHROOT_DIR = "${WORKDIR}/rootfs"
 
-do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 do_build[root_cleandirs] = "${BUILDCHROOT_DIR} \
                             ${BUILDCHROOT_DIR}/isar-apt \
                             ${BUILDCHROOT_DIR}/downloads \
diff --git a/meta/recipes-devtools/sdkchroot/sdkchroot.bb b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
index 895a454..822d4bb 100644
--- a/meta/recipes-devtools/sdkchroot/sdkchroot.bb
+++ b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
@@ -14,6 +14,7 @@ SRC_URI = " \
 PV = "0.1"
 
 inherit isar-bootstrap-helper
+PF = "${PN}-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"
 
 SDKCHROOT_PREINSTALL := "debhelper \
                            autotools-dev \
@@ -25,11 +26,9 @@ SDKCHROOT_PREINSTALL := "debhelper \
                            devscripts \
                            equivs"
 
-WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}-${HOST_DISTRO}-${HOST_ARCH}"
 S = "${WORKDIR}/rootfs"
 
 do_build[dirs] = "${DEPLOY_DIR_IMAGE}"
-do_build[stamp-extra-info] = "${HOST_DISTRO}-${HOST_ARCH}"
 do_build[root_cleandirs] = "${S} \
                             ${S}/isar-apt"
 
-- 
2.20.1

[PATCH v3 5/8] bitbake.conf: remove unneeded and differently used variables

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

IMAGE_CMD and TARGET_ARCH are not used in Isar or any bitbake code.

IMAGE_ROOTFS is used as the directory where image recipes store their
root file system. Pointing that to a global path like '${TMPDIR}/rootfs'
does not make sense for Isar.

Signed-off-by: Claudius Heine <ch@denx.de>
---
 meta/conf/bitbake.conf | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 7c1c89f..21d13ef 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -26,8 +26,6 @@ DEPLOY_DIR = "${TMPDIR}/deploy"
 FILESPATH = "${FILE_DIRNAME}/${PF}:${FILE_DIRNAME}/${P}:${FILE_DIRNAME}/${PN}:${FILE_DIRNAME}/files:${FILE_DIRNAME}"
 FILE_DIRNAME = "${@os.path.dirname(d.getVar('FILE', False))}"
 GITDIR = "${DL_DIR}/git"
-IMAGE_CMD = "_NO_DEFINED_IMAGE_TYPES_"
-IMAGE_ROOTFS = "${TMPDIR}/rootfs"
 P = "${PN}-${PV}"
 PF = "${PN}-${PV}-${PR}"
 PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}"
@@ -39,7 +37,6 @@ SRC_URI = "file://${FILE}"
 STAMP = "${TMPDIR}/stamps/${PF}"
 SVNDIR = "${DL_DIR}/svn"
 T = "${WORKDIR}/temp"
-TARGET_ARCH = "${BUILD_ARCH}"
 TMPDIR = "${TOPDIR}/tmp"
 PERSISTENT_DIR = "${TMPDIR}/cache"
 GITPKGV = "${@bb.fetch2.get_srcrev(d, 'gitpkgv_revision')}"
-- 
2.20.1

[PATCH v3 6/8] image.bbclass: make IMAGE_ROOTFS overwritable

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

This way recipes that inherit image.bbclass can set this path
themselves. This was previously not possible, because the bitbake.conf
set this variable with '='.

Signed-off-by: Claudius Heine <ch@denx.de>
---
 meta/classes/image.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index f9a3052..6881d37 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -5,7 +5,7 @@ PF = "${PN}-${DISTRO}-${MACHINE}"
 
 IMAGE_INSTALL ?= ""
 IMAGE_TYPE    ?= "ext4-img"
-IMAGE_ROOTFS   = "${WORKDIR}/rootfs"
+IMAGE_ROOTFS ?= "${WORKDIR}/rootfs"
 
 IMAGE_INSTALL += "${@ ("linux-image-" + d.getVar("KERNEL_NAME", True)) if d.getVar("KERNEL_NAME", True) else ""}"
 
-- 
2.20.1

[PATCH v3 7/8] bitbake.conf: set default QEMU_ARCH variables

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

Signed-off-by: Claudius Heine <ch@denx.de>
---
 meta/conf/bitbake.conf | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 21d13ef..37f1963 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -53,6 +53,12 @@ CACHE = "${TMPDIR}/cache"
 
 OVERRIDES = "${DISTRO_ARCH}:${MACHINE}:${DISTRO}:forcevariable"
 
+# Setting default QEMU_ARCH variables for different DISTRO_ARCH:
+QEMU_ARCH_amd64 = "x86_64"
+QEMU_ARCH_i386 = "i386"
+QEMU_ARCH_armhf = "arm"
+QEMU_ARCH_arm64 = "aarch64"
+
 # Codename of the repository created by the caching class
 DEBDISTRONAME = "isar"
 
-- 
2.20.1

[PATCH v3 8/8] buildchroot/configscript: make creation of builder uid/gid idempotent

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

Signed-off-by: Claudius Heine <ch@denx.de>
---
 meta/recipes-devtools/buildchroot/files/configscript.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-devtools/buildchroot/files/configscript.sh b/meta/recipes-devtools/buildchroot/files/configscript.sh
index 25a21ef..c4d8cf8 100644
--- a/meta/recipes-devtools/buildchroot/files/configscript.sh
+++ b/meta/recipes-devtools/buildchroot/files/configscript.sh
@@ -10,6 +10,6 @@ locales locales/locales_to_be_generated multiselect en_US.UTF-8 UTF-8
 locales locales/default_environment_locale select en_US.UTF-8
 END
 
-groupadd --system builder -o --gid $2
-useradd --system -o --uid $1 --gid builder --no-create-home --home /home/builder --no-user-group --comment "Isar buildchroot build user" builder
+grep -q "builder:x:$2" /etc/group || groupadd -f --system builder -o --gid $2
+grep -q "builder:x:$1" /etc/passwd || useradd --system -o --uid $1 --gid builder --no-create-home --home /home/builder --no-user-group --comment "Isar buildchroot build user" builder
 chown -R builder:builder /home/builder
-- 
2.20.1

Re: [PATCH v3 0/8] Cleanup rootfs creation

[Maxim Yu. Osipov]

Tried this series with fast ci build ("scripts/ci_build.sh -q -f"-
similar problems as before.
http://isar-build.org:8080/job/isar_mosipov_develop/98/console

I observe the same errors on my host.

Maxim.

On 4/24/19 11:27 AM, claudius.heine.ext@siemens.com wrote:
> From: Claudius Heine <ch@denx.de>
> 
> Hi,
> 
> I could not reproduce the raspberrypi issue Maxim found in his CI build:
> http://isar-build.org:8080/job/isar_mosipov_develop/91/consoleText
> 
> This is my build log:
> http://isar-build.org:8080/job/isar_claudius_ilbers-ci/54/consoleText
> 
> Only the module loading fails, which Maxim stated he will look into.
> 
> This patchset uses still the 'old' style for refactoring/cleanup patchsets,
> until I automated the official patch submission process as documented by the
> CONTRIBUTERS policy.
> 
> You can find the whole patchset in under this url:
> 
> https://github.com/cmhe/isar/tree/ch/cleanup
> 
> regards,
> Claudius
> 
> changes from v2:
>   - replaced 'ROOTFS_ARCH' with 'HOST_ARCH' in sdkchroot.bb to avoid
>     populate_sdk issue
> 
> changes from v1:
>   - fixed typo in commit message
> 
> Claudius Heine (8):
>    isar-boostrap-helper: move 'HOST_ARCH' and 'HOST_DISTRO' to
>      base.bbclass
>    move 'HOST_DISTRO_APT_SOURCES' from bootstrap-helper to isar-bootstrap
>    buildchroot.bbclass: only cross build if HOST_ARCH != DISTRO_ARCH
>    isar-bootstrap/buildchroot/sdkchroot: refactor PF and WORKDIR
>    bitbake.conf: remove unneeded and differently used variables
>    image.bbclass: make IMAGE_ROOTFS overwritable
>    bitbake.conf: set default QEMU_ARCH variables
>    buildchroot/configscript: make creation of builder uid/gid idempotent
> 
>   meta/classes/base.bbclass                          |  9 +++++++++
>   meta/classes/buildchroot.bbclass                   |  2 +-
>   meta/classes/image-sdk-extension.bbclass           |  2 +-
>   meta/classes/image.bbclass                         |  2 +-
>   meta/classes/isar-bootstrap-helper.bbclass         | 14 --------------
>   meta/conf/bitbake.conf                             | 13 ++++++++-----
>   .../isar-bootstrap/isar-bootstrap-host.bb          | 11 +++--------
>   .../isar-bootstrap/isar-bootstrap-target.bb        |  6 +-----
>   .../recipes-core/isar-bootstrap/isar-bootstrap.inc |  1 +
>   .../buildchroot/buildchroot-host.bb                |  1 +
>   meta/recipes-devtools/buildchroot/buildchroot.inc  |  3 +--
>   .../buildchroot/files/configscript.sh              |  4 ++--
>   meta/recipes-devtools/sdkchroot/sdkchroot.bb       |  3 +--
>   13 files changed, 30 insertions(+), 41 deletions(-)
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: [PATCH v3 0/8] Cleanup rootfs creation

[Claudius Heine]

Hi Maxim,

On 25/04/2019 08.03, Maxim Yu. Osipov wrote:
> 
> Tried this series with fast ci build ("scripts/ci_build.sh -q -f"-
> similar problems as before.
> http://isar-build.org:8080/job/isar_mosipov_develop/98/console
> 
> I observe the same errors on my host.

So what do we do then when the CI is not reproducible?

I am at a loss at how to investigate an error that does not happen when 
I build on isar-build or on my local host and only when you build it on 
isar-build or at your host.

Any suggestions?

Claudius

> 
> Maxim.
> 
> On 4/24/19 11:27 AM, claudius.heine.ext@siemens.com wrote:
>> From: Claudius Heine <ch@denx.de>
>>
>> Hi,
>>
>> I could not reproduce the raspberrypi issue Maxim found in his CI build:
>> http://isar-build.org:8080/job/isar_mosipov_develop/91/consoleText
>>
>> This is my build log:
>> http://isar-build.org:8080/job/isar_claudius_ilbers-ci/54/consoleText
>>
>> Only the module loading fails, which Maxim stated he will look into.
>>
>> This patchset uses still the 'old' style for refactoring/cleanup 
>> patchsets,
>> until I automated the official patch submission process as documented 
>> by the
>> CONTRIBUTERS policy.
>>
>> You can find the whole patchset in under this url:
>>
>> https://github.com/cmhe/isar/tree/ch/cleanup
>>
>> regards,
>> Claudius
>>
>> changes from v2:
>>   - replaced 'ROOTFS_ARCH' with 'HOST_ARCH' in sdkchroot.bb to avoid
>>     populate_sdk issue
>>
>> changes from v1:
>>   - fixed typo in commit message
>>
>> Claudius Heine (8):
>>    isar-boostrap-helper: move 'HOST_ARCH' and 'HOST_DISTRO' to
>>      base.bbclass
>>    move 'HOST_DISTRO_APT_SOURCES' from bootstrap-helper to isar-bootstrap
>>    buildchroot.bbclass: only cross build if HOST_ARCH != DISTRO_ARCH
>>    isar-bootstrap/buildchroot/sdkchroot: refactor PF and WORKDIR
>>    bitbake.conf: remove unneeded and differently used variables
>>    image.bbclass: make IMAGE_ROOTFS overwritable
>>    bitbake.conf: set default QEMU_ARCH variables
>>    buildchroot/configscript: make creation of builder uid/gid idempotent
>>
>>   meta/classes/base.bbclass                          |  9 +++++++++
>>   meta/classes/buildchroot.bbclass                   |  2 +-
>>   meta/classes/image-sdk-extension.bbclass           |  2 +-
>>   meta/classes/image.bbclass                         |  2 +-
>>   meta/classes/isar-bootstrap-helper.bbclass         | 14 --------------
>>   meta/conf/bitbake.conf                             | 13 ++++++++-----
>>   .../isar-bootstrap/isar-bootstrap-host.bb          | 11 +++--------
>>   .../isar-bootstrap/isar-bootstrap-target.bb        |  6 +-----
>>   .../recipes-core/isar-bootstrap/isar-bootstrap.inc |  1 +
>>   .../buildchroot/buildchroot-host.bb                |  1 +
>>   meta/recipes-devtools/buildchroot/buildchroot.inc  |  3 +--
>>   .../buildchroot/files/configscript.sh              |  4 ++--
>>   meta/recipes-devtools/sdkchroot/sdkchroot.bb       |  3 +--
>>   13 files changed, 30 insertions(+), 41 deletions(-)
>>
> 
> 

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [PATCH v3 0/8] Cleanup rootfs creation

[Maxim Yu. Osipov]

Hi Claudius,

On 4/25/19 8:40 AM, Claudius Heine wrote:
> Hi Maxim,
> 
> On 25/04/2019 08.03, Maxim Yu. Osipov wrote:
>>
>> Tried this series with fast ci build ("scripts/ci_build.sh -q -f"-
>> similar problems as before.
>> http://isar-build.org:8080/job/isar_mosipov_develop/98/console
>>
>> I observe the same errors on my host.
> 
> So what do we do then when the CI is not reproducible?
> 
> I am at a loss at how to investigate an error that does not happen when 
> I build on isar-build or on my local host and only when you build it on 
> isar-build or at your host.

Do you mean that this problem is not reproduced when you run "fast" CI 
build (ci_build.sh -q -f) on your PC?

> Any suggestions?

Yes, sure. We will create an additional project for you on 
isar-build.org to launch "fast" CI build.

Maxim.

> Claudius
> 
>>
>> Maxim.
>>
>> On 4/24/19 11:27 AM, claudius.heine.ext@siemens.com wrote:
>>> From: Claudius Heine <ch@denx.de>
>>>
>>> Hi,
>>>
>>> I could not reproduce the raspberrypi issue Maxim found in his CI build:
>>> http://isar-build.org:8080/job/isar_mosipov_develop/91/consoleText
>>>
>>> This is my build log:
>>> http://isar-build.org:8080/job/isar_claudius_ilbers-ci/54/consoleText
>>>
>>> Only the module loading fails, which Maxim stated he will look into.
>>>
>>> This patchset uses still the 'old' style for refactoring/cleanup 
>>> patchsets,
>>> until I automated the official patch submission process as documented 
>>> by the
>>> CONTRIBUTERS policy.
>>>
>>> You can find the whole patchset in under this url:
>>>
>>> https://github.com/cmhe/isar/tree/ch/cleanup
>>>
>>> regards,
>>> Claudius
>>>
>>> changes from v2:
>>>   - replaced 'ROOTFS_ARCH' with 'HOST_ARCH' in sdkchroot.bb to avoid
>>>     populate_sdk issue
>>>
>>> changes from v1:
>>>   - fixed typo in commit message
>>>
>>> Claudius Heine (8):
>>>    isar-boostrap-helper: move 'HOST_ARCH' and 'HOST_DISTRO' to
>>>      base.bbclass
>>>    move 'HOST_DISTRO_APT_SOURCES' from bootstrap-helper to 
>>> isar-bootstrap
>>>    buildchroot.bbclass: only cross build if HOST_ARCH != DISTRO_ARCH
>>>    isar-bootstrap/buildchroot/sdkchroot: refactor PF and WORKDIR
>>>    bitbake.conf: remove unneeded and differently used variables
>>>    image.bbclass: make IMAGE_ROOTFS overwritable
>>>    bitbake.conf: set default QEMU_ARCH variables
>>>    buildchroot/configscript: make creation of builder uid/gid idempotent
>>>
>>>   meta/classes/base.bbclass                          |  9 +++++++++
>>>   meta/classes/buildchroot.bbclass                   |  2 +-
>>>   meta/classes/image-sdk-extension.bbclass           |  2 +-
>>>   meta/classes/image.bbclass                         |  2 +-
>>>   meta/classes/isar-bootstrap-helper.bbclass         | 14 --------------
>>>   meta/conf/bitbake.conf                             | 13 ++++++++-----
>>>   .../isar-bootstrap/isar-bootstrap-host.bb          | 11 +++--------
>>>   .../isar-bootstrap/isar-bootstrap-target.bb        |  6 +-----
>>>   .../recipes-core/isar-bootstrap/isar-bootstrap.inc |  1 +
>>>   .../buildchroot/buildchroot-host.bb                |  1 +
>>>   meta/recipes-devtools/buildchroot/buildchroot.inc  |  3 +--
>>>   .../buildchroot/files/configscript.sh              |  4 ++--
>>>   meta/recipes-devtools/sdkchroot/sdkchroot.bb       |  3 +--
>>>   13 files changed, 30 insertions(+), 41 deletions(-)
>>>
>>
>>
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: [PATCH v3 0/8] Cleanup rootfs creation

[Claudius Heine]

Hi Maxim,

On 25/04/2019 09.55, Maxim Yu. Osipov wrote:
> Hi Claudius,
> 
> On 4/25/19 8:40 AM, Claudius Heine wrote:
>> Hi Maxim,
>>
>> On 25/04/2019 08.03, Maxim Yu. Osipov wrote:
>>>
>>> Tried this series with fast ci build ("scripts/ci_build.sh -q -f"-
>>> similar problems as before.
>>> http://isar-build.org:8080/job/isar_mosipov_develop/98/console
>>>
>>> I observe the same errors on my host.
>>
>> So what do we do then when the CI is not reproducible?
>>
>> I am at a loss at how to investigate an error that does not happen 
>> when I build on isar-build or on my local host and only when you build 
>> it on isar-build or at your host.
> 
> Do you mean that this problem is not reproduced when you run "fast" CI 
> build (ci_build.sh -q -f) on your PC?

I don't do ci_builds locally, because that is the job of the CI. I just 
build one target that might be affected of the specific change.

> 
>> Any suggestions?
> 
> Yes, sure. We will create an additional project for you on 
> isar-build.org to launch "fast" CI build.

So you suggest that I should now test every patch I send two times, once 
for each different CI configuration manually?

Claudius

> 
> Maxim.
> 
>> Claudius
>>
>>>
>>> Maxim.
>>>
>>> On 4/24/19 11:27 AM, claudius.heine.ext@siemens.com wrote:
>>>> From: Claudius Heine <ch@denx.de>
>>>>
>>>> Hi,
>>>>
>>>> I could not reproduce the raspberrypi issue Maxim found in his CI 
>>>> build:
>>>> http://isar-build.org:8080/job/isar_mosipov_develop/91/consoleText
>>>>
>>>> This is my build log:
>>>> http://isar-build.org:8080/job/isar_claudius_ilbers-ci/54/consoleText
>>>>
>>>> Only the module loading fails, which Maxim stated he will look into.
>>>>
>>>> This patchset uses still the 'old' style for refactoring/cleanup 
>>>> patchsets,
>>>> until I automated the official patch submission process as 
>>>> documented by the
>>>> CONTRIBUTERS policy.
>>>>
>>>> You can find the whole patchset in under this url:
>>>>
>>>> https://github.com/cmhe/isar/tree/ch/cleanup
>>>>
>>>> regards,
>>>> Claudius
>>>>
>>>> changes from v2:
>>>>   - replaced 'ROOTFS_ARCH' with 'HOST_ARCH' in sdkchroot.bb to avoid
>>>>     populate_sdk issue
>>>>
>>>> changes from v1:
>>>>   - fixed typo in commit message
>>>>
>>>> Claudius Heine (8):
>>>>    isar-boostrap-helper: move 'HOST_ARCH' and 'HOST_DISTRO' to
>>>>      base.bbclass
>>>>    move 'HOST_DISTRO_APT_SOURCES' from bootstrap-helper to 
>>>> isar-bootstrap
>>>>    buildchroot.bbclass: only cross build if HOST_ARCH != DISTRO_ARCH
>>>>    isar-bootstrap/buildchroot/sdkchroot: refactor PF and WORKDIR
>>>>    bitbake.conf: remove unneeded and differently used variables
>>>>    image.bbclass: make IMAGE_ROOTFS overwritable
>>>>    bitbake.conf: set default QEMU_ARCH variables
>>>>    buildchroot/configscript: make creation of builder uid/gid 
>>>> idempotent
>>>>
>>>>   meta/classes/base.bbclass                          |  9 +++++++++
>>>>   meta/classes/buildchroot.bbclass                   |  2 +-
>>>>   meta/classes/image-sdk-extension.bbclass           |  2 +-
>>>>   meta/classes/image.bbclass                         |  2 +-
>>>>   meta/classes/isar-bootstrap-helper.bbclass         | 14 
>>>> --------------
>>>>   meta/conf/bitbake.conf                             | 13 ++++++++-----
>>>>   .../isar-bootstrap/isar-bootstrap-host.bb          | 11 +++--------
>>>>   .../isar-bootstrap/isar-bootstrap-target.bb        |  6 +-----
>>>>   .../recipes-core/isar-bootstrap/isar-bootstrap.inc |  1 +
>>>>   .../buildchroot/buildchroot-host.bb                |  1 +
>>>>   meta/recipes-devtools/buildchroot/buildchroot.inc  |  3 +--
>>>>   .../buildchroot/files/configscript.sh              |  4 ++--
>>>>   meta/recipes-devtools/sdkchroot/sdkchroot.bb       |  3 +--
>>>>   13 files changed, 30 insertions(+), 41 deletions(-)
>>>>
>>>
>>>
>>
> 
> 

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [PATCH v3 0/8] Cleanup rootfs creation

[Maxim Yu. Osipov]

Hi Claudius,

On 4/25/19 10:01 AM, Claudius Heine wrote:
> Hi Maxim,
> 
> On 25/04/2019 09.55, Maxim Yu. Osipov wrote:
>> Hi Claudius,
>>
>> On 4/25/19 8:40 AM, Claudius Heine wrote:
>>> Hi Maxim,
>>>
>>> On 25/04/2019 08.03, Maxim Yu. Osipov wrote:
>>>>
>>>> Tried this series with fast ci build ("scripts/ci_build.sh -q -f"-
>>>> similar problems as before.
>>>> http://isar-build.org:8080/job/isar_mosipov_develop/98/console
>>>>
>>>> I observe the same errors on my host.
>>>
>>> So what do we do then when the CI is not reproducible?
>>>
>>> I am at a loss at how to investigate an error that does not happen 
>>> when I build on isar-build or on my local host and only when you 
>>> build it on isar-build or at your host.
>>
>> Do you mean that this problem is not reproduced when you run "fast" CI 
>> build (ci_build.sh -q -f) on your PC?
> 
> I don't do ci_builds locally, because that is the job of the CI. I just 
> build one target that might be affected of the specific change.

That's the reason why you can't reproduce the problem.

>>
>>> Any suggestions?
>>
>> Yes, sure. We will create an additional project for you on 
>> isar-build.org to launch "fast" CI build.
> 
> So you suggest that I should now test every patch I send two times, once 
> for each different CI configuration manually?

There is another option. I may reconfigure your current project to fast 
build. Should I do that?

Maxim.

> Claudius
> 
>>
>> Maxim.
>>
>>> Claudius
>>>
>>>>
>>>> Maxim.
>>>>
>>>> On 4/24/19 11:27 AM, claudius.heine.ext@siemens.com wrote:
>>>>> From: Claudius Heine <ch@denx.de>
>>>>>
>>>>> Hi,
>>>>>
>>>>> I could not reproduce the raspberrypi issue Maxim found in his CI 
>>>>> build:
>>>>> http://isar-build.org:8080/job/isar_mosipov_develop/91/consoleText
>>>>>
>>>>> This is my build log:
>>>>> http://isar-build.org:8080/job/isar_claudius_ilbers-ci/54/consoleText
>>>>>
>>>>> Only the module loading fails, which Maxim stated he will look into.
>>>>>
>>>>> This patchset uses still the 'old' style for refactoring/cleanup 
>>>>> patchsets,
>>>>> until I automated the official patch submission process as 
>>>>> documented by the
>>>>> CONTRIBUTERS policy.
>>>>>
>>>>> You can find the whole patchset in under this url:
>>>>>
>>>>> https://github.com/cmhe/isar/tree/ch/cleanup
>>>>>
>>>>> regards,
>>>>> Claudius
>>>>>
>>>>> changes from v2:
>>>>>   - replaced 'ROOTFS_ARCH' with 'HOST_ARCH' in sdkchroot.bb to avoid
>>>>>     populate_sdk issue
>>>>>
>>>>> changes from v1:
>>>>>   - fixed typo in commit message
>>>>>
>>>>> Claudius Heine (8):
>>>>>    isar-boostrap-helper: move 'HOST_ARCH' and 'HOST_DISTRO' to
>>>>>      base.bbclass
>>>>>    move 'HOST_DISTRO_APT_SOURCES' from bootstrap-helper to 
>>>>> isar-bootstrap
>>>>>    buildchroot.bbclass: only cross build if HOST_ARCH != DISTRO_ARCH
>>>>>    isar-bootstrap/buildchroot/sdkchroot: refactor PF and WORKDIR
>>>>>    bitbake.conf: remove unneeded and differently used variables
>>>>>    image.bbclass: make IMAGE_ROOTFS overwritable
>>>>>    bitbake.conf: set default QEMU_ARCH variables
>>>>>    buildchroot/configscript: make creation of builder uid/gid 
>>>>> idempotent
>>>>>
>>>>>   meta/classes/base.bbclass                          |  9 +++++++++
>>>>>   meta/classes/buildchroot.bbclass                   |  2 +-
>>>>>   meta/classes/image-sdk-extension.bbclass           |  2 +-
>>>>>   meta/classes/image.bbclass                         |  2 +-
>>>>>   meta/classes/isar-bootstrap-helper.bbclass         | 14 
>>>>> --------------
>>>>>   meta/conf/bitbake.conf                             | 13 
>>>>> ++++++++-----
>>>>>   .../isar-bootstrap/isar-bootstrap-host.bb          | 11 +++--------
>>>>>   .../isar-bootstrap/isar-bootstrap-target.bb        |  6 +-----
>>>>>   .../recipes-core/isar-bootstrap/isar-bootstrap.inc |  1 +
>>>>>   .../buildchroot/buildchroot-host.bb                |  1 +
>>>>>   meta/recipes-devtools/buildchroot/buildchroot.inc  |  3 +--
>>>>>   .../buildchroot/files/configscript.sh              |  4 ++--
>>>>>   meta/recipes-devtools/sdkchroot/sdkchroot.bb       |  3 +--
>>>>>   13 files changed, 30 insertions(+), 41 deletions(-)
>>>>>
>>>>
>>>>
>>>
>>
>>
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: [PATCH v3 0/8] Cleanup rootfs creation

[Claudius Heine]

On 25/04/2019 10.07, Maxim Yu. Osipov wrote:
> Hi Claudius,
> 
> On 4/25/19 10:01 AM, Claudius Heine wrote:
>> Hi Maxim,
>>
>> On 25/04/2019 09.55, Maxim Yu. Osipov wrote:
>>> Hi Claudius,
>>>
>>> On 4/25/19 8:40 AM, Claudius Heine wrote:
>>>> Hi Maxim,
>>>>
>>>> On 25/04/2019 08.03, Maxim Yu. Osipov wrote:
>>>>>
>>>>> Tried this series with fast ci build ("scripts/ci_build.sh -q -f"-
>>>>> similar problems as before.
>>>>> http://isar-build.org:8080/job/isar_mosipov_develop/98/console
>>>>>
>>>>> I observe the same errors on my host.
>>>>
>>>> So what do we do then when the CI is not reproducible?
>>>>
>>>> I am at a loss at how to investigate an error that does not happen 
>>>> when I build on isar-build or on my local host and only when you 
>>>> build it on isar-build or at your host.
>>>
>>> Do you mean that this problem is not reproduced when you run "fast" 
>>> CI build (ci_build.sh -q -f) on your PC?
>>
>> I don't do ci_builds locally, because that is the job of the CI. I 
>> just build one target that might be affected of the specific change.
> 
> That's the reason why you can't reproduce the problem.

Well I did build the multiconfig+target where the issue occured IIRC.

> 
>>>
>>>> Any suggestions?
>>>
>>> Yes, sure. We will create an additional project for you on 
>>> isar-build.org to launch "fast" CI build.
>>
>> So you suggest that I should now test every patch I send two times, 
>> once for each different CI configuration manually?
> 
> There is another option. I may reconfigure your current project to fast 
> build. Should I do that?

I don't care. I would like to have my CI configured in a way that allows 
my patches to tested for upstream inclusion. If that is in a "fast", 
"slow" or "totally unnoteworthy" velocity does not matter for that. CI 
is an acceptability test and I want the strictest one. If I was mistaken 
that "fast" is less strict than "slow", then switch to "fast". If I now 
need to pass "fast" and "slow" for my patch to be acceptable, then I 
would suggest introducing a "abysmal slow" CI run that combines "slow" 
and "fast".

Claudius

> 
> Maxim.
> 
>> Claudius
>>
>>>
>>> Maxim.
>>>
>>>> Claudius
>>>>
>>>>>
>>>>> Maxim.
>>>>>
>>>>> On 4/24/19 11:27 AM, claudius.heine.ext@siemens.com wrote:
>>>>>> From: Claudius Heine <ch@denx.de>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I could not reproduce the raspberrypi issue Maxim found in his CI 
>>>>>> build:
>>>>>> http://isar-build.org:8080/job/isar_mosipov_develop/91/consoleText
>>>>>>
>>>>>> This is my build log:
>>>>>> http://isar-build.org:8080/job/isar_claudius_ilbers-ci/54/consoleText
>>>>>>
>>>>>> Only the module loading fails, which Maxim stated he will look into.
>>>>>>
>>>>>> This patchset uses still the 'old' style for refactoring/cleanup 
>>>>>> patchsets,
>>>>>> until I automated the official patch submission process as 
>>>>>> documented by the
>>>>>> CONTRIBUTERS policy.
>>>>>>
>>>>>> You can find the whole patchset in under this url:
>>>>>>
>>>>>> https://github.com/cmhe/isar/tree/ch/cleanup
>>>>>>
>>>>>> regards,
>>>>>> Claudius
>>>>>>
>>>>>> changes from v2:
>>>>>>   - replaced 'ROOTFS_ARCH' with 'HOST_ARCH' in sdkchroot.bb to avoid
>>>>>>     populate_sdk issue
>>>>>>
>>>>>> changes from v1:
>>>>>>   - fixed typo in commit message
>>>>>>
>>>>>> Claudius Heine (8):
>>>>>>    isar-boostrap-helper: move 'HOST_ARCH' and 'HOST_DISTRO' to
>>>>>>      base.bbclass
>>>>>>    move 'HOST_DISTRO_APT_SOURCES' from bootstrap-helper to 
>>>>>> isar-bootstrap
>>>>>>    buildchroot.bbclass: only cross build if HOST_ARCH != DISTRO_ARCH
>>>>>>    isar-bootstrap/buildchroot/sdkchroot: refactor PF and WORKDIR
>>>>>>    bitbake.conf: remove unneeded and differently used variables
>>>>>>    image.bbclass: make IMAGE_ROOTFS overwritable
>>>>>>    bitbake.conf: set default QEMU_ARCH variables
>>>>>>    buildchroot/configscript: make creation of builder uid/gid 
>>>>>> idempotent
>>>>>>
>>>>>>   meta/classes/base.bbclass                          |  9 +++++++++
>>>>>>   meta/classes/buildchroot.bbclass                   |  2 +-
>>>>>>   meta/classes/image-sdk-extension.bbclass           |  2 +-
>>>>>>   meta/classes/image.bbclass                         |  2 +-
>>>>>>   meta/classes/isar-bootstrap-helper.bbclass         | 14 
>>>>>> --------------
>>>>>>   meta/conf/bitbake.conf                             | 13 
>>>>>> ++++++++-----
>>>>>>   .../isar-bootstrap/isar-bootstrap-host.bb          | 11 +++--------
>>>>>>   .../isar-bootstrap/isar-bootstrap-target.bb        |  6 +-----
>>>>>>   .../recipes-core/isar-bootstrap/isar-bootstrap.inc |  1 +
>>>>>>   .../buildchroot/buildchroot-host.bb                |  1 +
>>>>>>   meta/recipes-devtools/buildchroot/buildchroot.inc  |  3 +--
>>>>>>   .../buildchroot/files/configscript.sh              |  4 ++--
>>>>>>   meta/recipes-devtools/sdkchroot/sdkchroot.bb       |  3 +--
>>>>>>   13 files changed, 30 insertions(+), 41 deletions(-)
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
> 
> 

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [PATCH v3 0/8] Cleanup rootfs creation

[Maxim Yu. Osipov]

On 4/25/19 10:27 AM, Claudius Heine wrote:
> On 25/04/2019 10.07, Maxim Yu. Osipov wrote:
>> Hi Claudius,
>>
>> On 4/25/19 10:01 AM, Claudius Heine wrote:
>>> Hi Maxim,
>>>
>>> On 25/04/2019 09.55, Maxim Yu. Osipov wrote:
>>>> Hi Claudius,
>>>>
>>>> On 4/25/19 8:40 AM, Claudius Heine wrote:
>>>>> Hi Maxim,
>>>>>
>>>>> On 25/04/2019 08.03, Maxim Yu. Osipov wrote:
>>>>>>
>>>>>> Tried this series with fast ci build ("scripts/ci_build.sh -q -f"-
>>>>>> similar problems as before.
>>>>>> http://isar-build.org:8080/job/isar_mosipov_develop/98/console
>>>>>>
>>>>>> I observe the same errors on my host.
>>>>>
>>>>> So what do we do then when the CI is not reproducible?
>>>>>
>>>>> I am at a loss at how to investigate an error that does not happen 
>>>>> when I build on isar-build or on my local host and only when you 
>>>>> build it on isar-build or at your host.
>>>>
>>>> Do you mean that this problem is not reproduced when you run "fast" 
>>>> CI build (ci_build.sh -q -f) on your PC?
>>>
>>> I don't do ci_builds locally, because that is the job of the CI. I 
>>> just build one target that might be affected of the specific change.
>>
>> That's the reason why you can't reproduce the problem.
> 
> Well I did build the multiconfig+target where the issue occured IIRC.
> 
>>
>>>>
>>>>> Any suggestions?
>>>>
>>>> Yes, sure. We will create an additional project for you on 
>>>> isar-build.org to launch "fast" CI build.
>>>
>>> So you suggest that I should now test every patch I send two times, 
>>> once for each different CI configuration manually?
>>
>> There is another option. I may reconfigure your current project to 
>> fast build. Should I do that?
> 
> I don't care. I would like to have my CI configured in a way that allows 
> my patches to tested for upstream inclusion. If that is in a "fast", 
> "slow" or "totally unnoteworthy" velocity does not matter for that. CI 
> is an acceptability test and I want the strictest one. If I was mistaken 
> that "fast" is less strict than "slow", then switch to "fast". If I now 
> need to pass "fast" and "slow" for my patch to be acceptable, then I 
> would suggest introducing a "abysmal slow" CI run that combines "slow" 
> and "fast".

I've just sent the CI tests procedure in separate email.

The problem that if we include all the features into single CI run it 
will takes hours to test single patch.

Anyway I pointed you to the problem and the way how to reproduce it.
You wrote that you don't want to call CI locally. It's up to you - I 
don't care too - if the problem is not fixed the patchset will not be 
applied.

I didn't understood from your emotional passage what configuration you 
prefer.

I  suggested two options

1) Add additional project for you for "fast" build. Yes. In such cases 
you have to launch the tests in both projects but you will be sure that
integration will go more smoothly.

2) Reconfigure your existing project from "standard" to the "fast" one.

It is up to you to decide how you want to test your patch set.

Greetings,
Maxim.


> Claudius
> 
>>
>> Maxim.
>>
>>> Claudius
>>>
>>>>
>>>> Maxim.
>>>>
>>>>> Claudius
>>>>>
>>>>>>
>>>>>> Maxim.
>>>>>>
>>>>>> On 4/24/19 11:27 AM, claudius.heine.ext@siemens.com wrote:
>>>>>>> From: Claudius Heine <ch@denx.de>
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I could not reproduce the raspberrypi issue Maxim found in his CI 
>>>>>>> build:
>>>>>>> http://isar-build.org:8080/job/isar_mosipov_develop/91/consoleText
>>>>>>>
>>>>>>> This is my build log:
>>>>>>> http://isar-build.org:8080/job/isar_claudius_ilbers-ci/54/consoleText 
>>>>>>>
>>>>>>>
>>>>>>> Only the module loading fails, which Maxim stated he will look into.
>>>>>>>
>>>>>>> This patchset uses still the 'old' style for refactoring/cleanup 
>>>>>>> patchsets,
>>>>>>> until I automated the official patch submission process as 
>>>>>>> documented by the
>>>>>>> CONTRIBUTERS policy.
>>>>>>>
>>>>>>> You can find the whole patchset in under this url:
>>>>>>>
>>>>>>> https://github.com/cmhe/isar/tree/ch/cleanup
>>>>>>>
>>>>>>> regards,
>>>>>>> Claudius
>>>>>>>
>>>>>>> changes from v2:
>>>>>>>   - replaced 'ROOTFS_ARCH' with 'HOST_ARCH' in sdkchroot.bb to avoid
>>>>>>>     populate_sdk issue
>>>>>>>
>>>>>>> changes from v1:
>>>>>>>   - fixed typo in commit message
>>>>>>>
>>>>>>> Claudius Heine (8):
>>>>>>>    isar-boostrap-helper: move 'HOST_ARCH' and 'HOST_DISTRO' to
>>>>>>>      base.bbclass
>>>>>>>    move 'HOST_DISTRO_APT_SOURCES' from bootstrap-helper to 
>>>>>>> isar-bootstrap
>>>>>>>    buildchroot.bbclass: only cross build if HOST_ARCH != DISTRO_ARCH
>>>>>>>    isar-bootstrap/buildchroot/sdkchroot: refactor PF and WORKDIR
>>>>>>>    bitbake.conf: remove unneeded and differently used variables
>>>>>>>    image.bbclass: make IMAGE_ROOTFS overwritable
>>>>>>>    bitbake.conf: set default QEMU_ARCH variables
>>>>>>>    buildchroot/configscript: make creation of builder uid/gid 
>>>>>>> idempotent
>>>>>>>
>>>>>>>   meta/classes/base.bbclass                          |  9 +++++++++
>>>>>>>   meta/classes/buildchroot.bbclass                   |  2 +-
>>>>>>>   meta/classes/image-sdk-extension.bbclass           |  2 +-
>>>>>>>   meta/classes/image.bbclass                         |  2 +-
>>>>>>>   meta/classes/isar-bootstrap-helper.bbclass         | 14 
>>>>>>> --------------
>>>>>>>   meta/conf/bitbake.conf                             | 13 
>>>>>>> ++++++++-----
>>>>>>>   .../isar-bootstrap/isar-bootstrap-host.bb          | 11 
>>>>>>> +++--------
>>>>>>>   .../isar-bootstrap/isar-bootstrap-target.bb        |  6 +-----
>>>>>>>   .../recipes-core/isar-bootstrap/isar-bootstrap.inc |  1 +
>>>>>>>   .../buildchroot/buildchroot-host.bb                |  1 +
>>>>>>>   meta/recipes-devtools/buildchroot/buildchroot.inc  |  3 +--
>>>>>>>   .../buildchroot/files/configscript.sh              |  4 ++--
>>>>>>>   meta/recipes-devtools/sdkchroot/sdkchroot.bb       |  3 +--
>>>>>>>   13 files changed, 30 insertions(+), 41 deletions(-)
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: [PATCH v3 0/8] Cleanup rootfs creation

[Jan Kiszka]

On 25.04.19 08:40, [ext] Claudius Heine wrote:
> Hi Maxim,
> 
> On 25/04/2019 08.03, Maxim Yu. Osipov wrote:
>>
>> Tried this series with fast ci build ("scripts/ci_build.sh -q -f"-
>> similar problems as before.
>> http://isar-build.org:8080/job/isar_mosipov_develop/98/console
>>
>> I observe the same errors on my host.
> 
> So what do we do then when the CI is not reproducible?

Then you likely see a race that you may only trigger by running sufficient 
builds in parallel, and maybe only with the "right" number of cores. Otherwise, 
you need to do offline analysis or push instrumented branches into the same CI 
system

If you take the time to look at the logs, you will likely quickly spot that 
there is some issue with mounting /dev and possibly also /dev/pts. Dig into your 
changes /wrt what may have created a race here - or started to expose a sleeping 
one.

In any case, we need to make CI a community resource. We will come up with a 
proposal soon.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH v3 0/8] Cleanup rootfs creation

[Claudius Heine]

On 25/04/2019 10.39, Maxim Yu. Osipov wrote:
> On 4/25/19 10:27 AM, Claudius Heine wrote:
>> On 25/04/2019 10.07, Maxim Yu. Osipov wrote:
>>> Hi Claudius,
>>>
>>> On 4/25/19 10:01 AM, Claudius Heine wrote:
>>>> Hi Maxim,
>>>>
>>>> On 25/04/2019 09.55, Maxim Yu. Osipov wrote:
>>>>> Hi Claudius,
>>>>>
>>>>> On 4/25/19 8:40 AM, Claudius Heine wrote:
>>>>>> Hi Maxim,
>>>>>>
>>>>>> On 25/04/2019 08.03, Maxim Yu. Osipov wrote:
>>>>>>>
>>>>>>> Tried this series with fast ci build ("scripts/ci_build.sh -q -f"-
>>>>>>> similar problems as before.
>>>>>>> http://isar-build.org:8080/job/isar_mosipov_develop/98/console
>>>>>>>
>>>>>>> I observe the same errors on my host.
>>>>>>
>>>>>> So what do we do then when the CI is not reproducible?
>>>>>>
>>>>>> I am at a loss at how to investigate an error that does not happen 
>>>>>> when I build on isar-build or on my local host and only when you 
>>>>>> build it on isar-build or at your host.
>>>>>
>>>>> Do you mean that this problem is not reproduced when you run "fast" 
>>>>> CI build (ci_build.sh -q -f) on your PC?
>>>>
>>>> I don't do ci_builds locally, because that is the job of the CI. I 
>>>> just build one target that might be affected of the specific change.
>>>
>>> That's the reason why you can't reproduce the problem.
>>
>> Well I did build the multiconfig+target where the issue occured IIRC.

I looked into the "fast" build now, and it cross-compiles 
"multiconfig:rpi-stretch:isar-image-base". Now I tried that and the 
build fails even on the current 'next' so the issue seems to be there.

>>
>>>
>>>>>
>>>>>> Any suggestions?
>>>>>
>>>>> Yes, sure. We will create an additional project for you on 
>>>>> isar-build.org to launch "fast" CI build.
>>>>
>>>> So you suggest that I should now test every patch I send two times, 
>>>> once for each different CI configuration manually?
>>>
>>> There is another option. I may reconfigure your current project to 
>>> fast build. Should I do that?
>>
>> I don't care. I would like to have my CI configured in a way that 
>> allows my patches to tested for upstream inclusion. If that is in a 
>> "fast", "slow" or "totally unnoteworthy" velocity does not matter for 
>> that. CI is an acceptability test and I want the strictest one. If I 
>> was mistaken that "fast" is less strict than "slow", then switch to 
>> "fast". If I now need to pass "fast" and "slow" for my patch to be 
>> acceptable, then I would suggest introducing a "abysmal slow" CI run 
>> that combines "slow" and "fast".
> 
> I've just sent the CI tests procedure in separate email.

That is not a CI test procedure. That is a test procedure for local 
testing, if someone wants to use the ci scripts to do so. I don't since 
my environment is different from what you have on your CI server.

> The problem that if we include all the features into single CI run it 
> will takes hours to test single patch.

Right. "abysmal slow". But in your 'CONTRIBUTERS' you state that 
everyone should do that anyway. Manually.

> 
> Anyway I pointed you to the problem and the way how to reproduce it.
> You wrote that you don't want to call CI locally. It's up to you - I 
> don't care too - if the problem is not fixed the patchset will not be 
> applied.

Well I don't have the same environment as you on your CI server, so 
running scripts build for your CI server is not something I can do. That 
is not choice that is just how things are.

> I didn't understood from your emotional passage what configuration you 
> prefer.

Well CI testing should be done reproducible and transparent. If my 
branch on jenkins has a different configuration than some other branch 
there and now the invisible configuration of those branches changes like 
the weather, then this is not reproducible nor transparent.

Having to explain the basics of what CI is, is pretty nerve wreaking and 
exhausting.

> I  suggested two options
> 
> 1) Add additional project for you for "fast" build. Yes. In such cases 
> you have to launch the tests in both projects but you will be sure that
> integration will go more smoothly.

Well the most acceptable way would be 3 branches. One for "fast", one 
for "slow" and one for "acceptance" (which combines slow and fast).

> 
> 2) Reconfigure your existing project from "standard" to the "fast" one.
> 
> It is up to you to decide how you want to test your patch set.

No its not. It is up to you to test the patchset and decide if that 
should be merged or not. So contributors have to jump through those 
hoops in order for it to be merged reasonable fast.

Claudius

> 
> Greetings,
> Maxim.
> 
> 
>> Claudius
>>
>>>
>>> Maxim.
>>>
>>>> Claudius
>>>>
>>>>>
>>>>> Maxim.
>>>>>
>>>>>> Claudius
>>>>>>
>>>>>>>
>>>>>>> Maxim.
>>>>>>>
>>>>>>> On 4/24/19 11:27 AM, claudius.heine.ext@siemens.com wrote:
>>>>>>>> From: Claudius Heine <ch@denx.de>
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I could not reproduce the raspberrypi issue Maxim found in his 
>>>>>>>> CI build:
>>>>>>>> http://isar-build.org:8080/job/isar_mosipov_develop/91/consoleText
>>>>>>>>
>>>>>>>> This is my build log:
>>>>>>>> http://isar-build.org:8080/job/isar_claudius_ilbers-ci/54/consoleText 
>>>>>>>>
>>>>>>>>
>>>>>>>> Only the module loading fails, which Maxim stated he will look 
>>>>>>>> into.
>>>>>>>>
>>>>>>>> This patchset uses still the 'old' style for refactoring/cleanup 
>>>>>>>> patchsets,
>>>>>>>> until I automated the official patch submission process as 
>>>>>>>> documented by the
>>>>>>>> CONTRIBUTERS policy.
>>>>>>>>
>>>>>>>> You can find the whole patchset in under this url:
>>>>>>>>
>>>>>>>> https://github.com/cmhe/isar/tree/ch/cleanup
>>>>>>>>
>>>>>>>> regards,
>>>>>>>> Claudius
>>>>>>>>
>>>>>>>> changes from v2:
>>>>>>>>   - replaced 'ROOTFS_ARCH' with 'HOST_ARCH' in sdkchroot.bb to 
>>>>>>>> avoid
>>>>>>>>     populate_sdk issue
>>>>>>>>
>>>>>>>> changes from v1:
>>>>>>>>   - fixed typo in commit message
>>>>>>>>
>>>>>>>> Claudius Heine (8):
>>>>>>>>    isar-boostrap-helper: move 'HOST_ARCH' and 'HOST_DISTRO' to
>>>>>>>>      base.bbclass
>>>>>>>>    move 'HOST_DISTRO_APT_SOURCES' from bootstrap-helper to 
>>>>>>>> isar-bootstrap
>>>>>>>>    buildchroot.bbclass: only cross build if HOST_ARCH != 
>>>>>>>> DISTRO_ARCH
>>>>>>>>    isar-bootstrap/buildchroot/sdkchroot: refactor PF and WORKDIR
>>>>>>>>    bitbake.conf: remove unneeded and differently used variables
>>>>>>>>    image.bbclass: make IMAGE_ROOTFS overwritable
>>>>>>>>    bitbake.conf: set default QEMU_ARCH variables
>>>>>>>>    buildchroot/configscript: make creation of builder uid/gid 
>>>>>>>> idempotent
>>>>>>>>
>>>>>>>>   meta/classes/base.bbclass                          |  9 +++++++++
>>>>>>>>   meta/classes/buildchroot.bbclass                   |  2 +-
>>>>>>>>   meta/classes/image-sdk-extension.bbclass           |  2 +-
>>>>>>>>   meta/classes/image.bbclass                         |  2 +-
>>>>>>>>   meta/classes/isar-bootstrap-helper.bbclass         | 14 
>>>>>>>> --------------
>>>>>>>>   meta/conf/bitbake.conf                             | 13 
>>>>>>>> ++++++++-----
>>>>>>>>   .../isar-bootstrap/isar-bootstrap-host.bb          | 11 
>>>>>>>> +++--------
>>>>>>>>   .../isar-bootstrap/isar-bootstrap-target.bb        |  6 +-----
>>>>>>>>   .../recipes-core/isar-bootstrap/isar-bootstrap.inc |  1 +
>>>>>>>>   .../buildchroot/buildchroot-host.bb                |  1 +
>>>>>>>>   meta/recipes-devtools/buildchroot/buildchroot.inc  |  3 +--
>>>>>>>>   .../buildchroot/files/configscript.sh              |  4 ++--
>>>>>>>>   meta/recipes-devtools/sdkchroot/sdkchroot.bb       |  3 +--
>>>>>>>>   13 files changed, 30 insertions(+), 41 deletions(-)
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
> 
> 

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [PATCH v3 0/8] Cleanup rootfs creation

[Maxim Yu. Osipov]

On 4/25/19 11:11 AM, Claudius Heine wrote:
> On 25/04/2019 10.39, Maxim Yu. Osipov wrote:
>> On 4/25/19 10:27 AM, Claudius Heine wrote:
>>> On 25/04/2019 10.07, Maxim Yu. Osipov wrote:
>>>> Hi Claudius,
>>>>
>>>> On 4/25/19 10:01 AM, Claudius Heine wrote:
>>>>> Hi Maxim,
>>>>>
>>>>> On 25/04/2019 09.55, Maxim Yu. Osipov wrote:
>>>>>> Hi Claudius,
>>>>>>
>>>>>> On 4/25/19 8:40 AM, Claudius Heine wrote:
>>>>>>> Hi Maxim,
>>>>>>>
>>>>>>> On 25/04/2019 08.03, Maxim Yu. Osipov wrote:
>>>>>>>>
>>>>>>>> Tried this series with fast ci build ("scripts/ci_build.sh -q -f"-
>>>>>>>> similar problems as before.
>>>>>>>> http://isar-build.org:8080/job/isar_mosipov_develop/98/console
>>>>>>>>
>>>>>>>> I observe the same errors on my host.
>>>>>>>
>>>>>>> So what do we do then when the CI is not reproducible?
>>>>>>>
>>>>>>> I am at a loss at how to investigate an error that does not 
>>>>>>> happen when I build on isar-build or on my local host and only 
>>>>>>> when you build it on isar-build or at your host.
>>>>>>
>>>>>> Do you mean that this problem is not reproduced when you run 
>>>>>> "fast" CI build (ci_build.sh -q -f) on your PC?
>>>>>
>>>>> I don't do ci_builds locally, because that is the job of the CI. I 
>>>>> just build one target that might be affected of the specific change.
>>>>
>>>> That's the reason why you can't reproduce the problem.
>>>
>>> Well I did build the multiconfig+target where the issue occured IIRC.
> 
> I looked into the "fast" build now, and it cross-compiles 
> "multiconfig:rpi-stretch:isar-image-base". Now I tried that and the 
> build fails even on the current 'next' so the issue seems to be there.

No. "Fast" CI build (which in particular tests cross-compilation) in
'next' is OK - I've built it today morning when I tested your patch
"ubi-img/ubifs-img: change fatal error to skip recipe" and my patch 
which includes SDK cretion test into "fast" build.

You may look at the log:
http://isar-build.org:8080/job/isar_mosipov_develop/97/console

>>>
>>>>
>>>>>>
>>>>>>> Any suggestions?
>>>>>>
>>>>>> Yes, sure. We will create an additional project for you on 
>>>>>> isar-build.org to launch "fast" CI build.
>>>>>
>>>>> So you suggest that I should now test every patch I send two times, 
>>>>> once for each different CI configuration manually?
>>>>
>>>> There is another option. I may reconfigure your current project to 
>>>> fast build. Should I do that?
>>>
>>> I don't care. I would like to have my CI configured in a way that 
>>> allows my patches to tested for upstream inclusion. If that is in a 
>>> "fast", "slow" or "totally unnoteworthy" velocity does not matter for 
>>> that. CI is an acceptability test and I want the strictest one. If I 
>>> was mistaken that "fast" is less strict than "slow", then switch to 
>>> "fast". If I now need to pass "fast" and "slow" for my patch to be 
>>> acceptable, then I would suggest introducing a "abysmal slow" CI run 
>>> that combines "slow" and "fast".
>>
>> I've just sent the CI tests procedure in separate email.
> 
> That is not a CI test procedure. That is a test procedure for local 
> testing, if someone wants to use the ci scripts to do so. I don't since 
> my environment is different from what you have on your CI server.
> 
>> The problem that if we include all the features into single CI run it 
>> will takes hours to test single patch.
> 
> Right. "abysmal slow". But in your 'CONTRIBUTERS' you state that 
> everyone should do that anyway. Manually.

It's highly suggested but its' not a must.
Current iteration of patch series just proves that it's worth to run 
these scripts if you want to avoid multiple patches versions iterations.
> 
>>
>> Anyway I pointed you to the problem and the way how to reproduce it.
>> You wrote that you don't want to call CI locally. It's up to you - I 
>> don't care too - if the problem is not fixed the patchset will not be 
>> applied.
> 
> Well I don't have the same environment as you on your CI server, so 
> running scripts build for your CI server is not something I can do. That 
> is not choice that is just how things are.
> 
>> I didn't understood from your emotional passage what configuration you 
>> prefer.
> 
> Well CI testing should be done reproducible and transparent. If my 
> branch on jenkins has a different configuration than some other branch 
> there and now the invisible configuration of those branches changes like 
> the weather, then this is not reproducible nor transparent.
> 
> Having to explain the basics of what CI is, is pretty nerve wreaking and 
> exhausting.

Thank you for explaining what CI is - I'm sorry to make you nervous and 
exhausted.

>> I  suggested two options
>>
>> 1) Add additional project for you for "fast" build. Yes. In such cases 
>> you have to launch the tests in both projects but you will be sure that
>> integration will go more smoothly.
> 
> Well the most acceptable way would be 3 branches. One for "fast", one 
> for "slow" and one for "acceptance" (which combines slow and fast).
> 
>>
>> 2) Reconfigure your existing project from "standard" to the "fast" one.
>>
>> It is up to you to decide how you want to test your patch set.
> 
> No its not. It is up to you to test the patchset and decide if that 
> should be merged or not. So contributors have to jump through those 
> hoops in order for it to be merged reasonable fast.

And I decided to don't merge your patch set because tests were failed.
That's the only story.

Maxim.


> Claudius
> 
>>
>> Greetings,
>> Maxim.
>>
>>
>>> Claudius
>>>
>>>>
>>>> Maxim.
>>>>
>>>>> Claudius
>>>>>
>>>>>>
>>>>>> Maxim.
>>>>>>
>>>>>>> Claudius
>>>>>>>
>>>>>>>>
>>>>>>>> Maxim.
>>>>>>>>
>>>>>>>> On 4/24/19 11:27 AM, claudius.heine.ext@siemens.com wrote:
>>>>>>>>> From: Claudius Heine <ch@denx.de>
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I could not reproduce the raspberrypi issue Maxim found in his 
>>>>>>>>> CI build:
>>>>>>>>> http://isar-build.org:8080/job/isar_mosipov_develop/91/consoleText
>>>>>>>>>
>>>>>>>>> This is my build log:
>>>>>>>>> http://isar-build.org:8080/job/isar_claudius_ilbers-ci/54/consoleText 
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Only the module loading fails, which Maxim stated he will look 
>>>>>>>>> into.
>>>>>>>>>
>>>>>>>>> This patchset uses still the 'old' style for 
>>>>>>>>> refactoring/cleanup patchsets,
>>>>>>>>> until I automated the official patch submission process as 
>>>>>>>>> documented by the
>>>>>>>>> CONTRIBUTERS policy.
>>>>>>>>>
>>>>>>>>> You can find the whole patchset in under this url:
>>>>>>>>>
>>>>>>>>> https://github.com/cmhe/isar/tree/ch/cleanup
>>>>>>>>>
>>>>>>>>> regards,
>>>>>>>>> Claudius
>>>>>>>>>
>>>>>>>>> changes from v2:
>>>>>>>>>   - replaced 'ROOTFS_ARCH' with 'HOST_ARCH' in sdkchroot.bb to 
>>>>>>>>> avoid
>>>>>>>>>     populate_sdk issue
>>>>>>>>>
>>>>>>>>> changes from v1:
>>>>>>>>>   - fixed typo in commit message
>>>>>>>>>
>>>>>>>>> Claudius Heine (8):
>>>>>>>>>    isar-boostrap-helper: move 'HOST_ARCH' and 'HOST_DISTRO' to
>>>>>>>>>      base.bbclass
>>>>>>>>>    move 'HOST_DISTRO_APT_SOURCES' from bootstrap-helper to 
>>>>>>>>> isar-bootstrap
>>>>>>>>>    buildchroot.bbclass: only cross build if HOST_ARCH != 
>>>>>>>>> DISTRO_ARCH
>>>>>>>>>    isar-bootstrap/buildchroot/sdkchroot: refactor PF and WORKDIR
>>>>>>>>>    bitbake.conf: remove unneeded and differently used variables
>>>>>>>>>    image.bbclass: make IMAGE_ROOTFS overwritable
>>>>>>>>>    bitbake.conf: set default QEMU_ARCH variables
>>>>>>>>>    buildchroot/configscript: make creation of builder uid/gid 
>>>>>>>>> idempotent
>>>>>>>>>
>>>>>>>>>   meta/classes/base.bbclass                          |  9 
>>>>>>>>> +++++++++
>>>>>>>>>   meta/classes/buildchroot.bbclass                   |  2 +-
>>>>>>>>>   meta/classes/image-sdk-extension.bbclass           |  2 +-
>>>>>>>>>   meta/classes/image.bbclass                         |  2 +-
>>>>>>>>>   meta/classes/isar-bootstrap-helper.bbclass         | 14 
>>>>>>>>> --------------
>>>>>>>>>   meta/conf/bitbake.conf                             | 13 
>>>>>>>>> ++++++++-----
>>>>>>>>>   .../isar-bootstrap/isar-bootstrap-host.bb          | 11 
>>>>>>>>> +++--------
>>>>>>>>>   .../isar-bootstrap/isar-bootstrap-target.bb        |  6 +-----
>>>>>>>>>   .../recipes-core/isar-bootstrap/isar-bootstrap.inc |  1 +
>>>>>>>>>   .../buildchroot/buildchroot-host.bb                |  1 +
>>>>>>>>>   meta/recipes-devtools/buildchroot/buildchroot.inc  |  3 +--
>>>>>>>>>   .../buildchroot/files/configscript.sh              |  4 ++--
>>>>>>>>>   meta/recipes-devtools/sdkchroot/sdkchroot.bb       |  3 +--
>>>>>>>>>   13 files changed, 30 insertions(+), 41 deletions(-)
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: [PATCH v3 0/8] Cleanup rootfs creation

[Baurzhan Ismagulov]

On Thu, Apr 25, 2019 at 11:11:21AM +0200, Claudius Heine wrote:
> That is not a CI test procedure. That is a test procedure for local testing,
> if someone wants to use the ci scripts to do so. I don't since my
> environment is different from what you have on your CI server.

The CI server doesn't have any magic environment. It's set up according to
doc/user_manual.md#getting-started. scripts/ci_build.sh and
scripts/vm_smoke_test are expected to be able to run locally or on the server.
Of course, you can run any part of that manually if you are not comfortable
running the whole. This is the motivation of moving to Avocado -- to break the
big script into more maintainable and short pieces, so one could directly see
which particular test case has failed and restart just that one.


> > The problem that if we include all the features into single CI run it
> > will takes hours to test single patch.
> 
> Right. "abysmal slow". But in your 'CONTRIBUTERS' you state that everyone
> should do that anyway. Manually.

This is what the maintainer does for every series. Isar supports many features
on a number of platforms; testing all does cost time. We don't ask that every
developer does that routinely (though that is in your interest to get the patch
applied quickly); responding to a specific failure is enough.


With kind regards,
Baurzhan.

Re: [PATCH v3 0/8] Cleanup rootfs creation

[Baurzhan Ismagulov]

On Thu, Apr 25, 2019 at 11:11:21AM +0200, Claudius Heine wrote:
> I looked into the "fast" build now, and it cross-compiles
> "multiconfig:rpi-stretch:isar-image-base". Now I tried that and the build
> fails even on the current 'next' so the issue seems to be there.

How does the problem manifest itself? Please provide some relevant details
(e.g., bitbake output, failure logs, etc.).

With kind regards,
Baurzhan.

Re: [PATCH v3 0/8] Cleanup rootfs creation

[Claudius Heine]

Hi Baurzhan,

On 25/04/2019 12.32, Baurzhan Ismagulov wrote:
> On Thu, Apr 25, 2019 at 11:11:21AM +0200, Claudius Heine wrote:
>> I looked into the "fast" build now, and it cross-compiles
>> "multiconfig:rpi-stretch:isar-image-base". Now I tried that and the build
>> fails even on the current 'next' so the issue seems to be there.
> 
> How does the problem manifest itself? Please provide some relevant details
> (e.g., bitbake output, failure logs, etc.).

Commit ID: 2396d4f44fe1f867ddae389b61cb7017cb2b908c

--- local.conf ---
# cross
ISAR_CROSS_COMPILE = "1"

# standard
CONF_VERSION = "1"

MACHINE ??= "qemu"
DISTRO ??= "raspbian-stretch"
BBMULTICONFIG ?= "rpi-stretch"
--- end local.conf ---

bitbake -k -c build multiconfig:rpi-stretch:isar-image-base

--- error log ---
ERROR: mc:rpi-stretch:isar-bootstrap-host-1.0-r0 do_bootstrap: Function 
failed: do_bootstrap (log file is located at 
/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/temp/log.do_bootstrap.27078)
ERROR: Logfile of failure stored in: 
/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/temp/log.do_bootstrap.27078
Log data follows:
| DEBUG: Executing shell function do_bootstrap
| W: Target architecture is the same as host architecture; disabling 
QEMU support
| I: Running command: debootstrap --arch amd64 --verbose 
--variant=minbase --include=locales --components=main,contrib,non-free 
--keyring 
/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/apt-keyring.gpg 
stretch 
/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs 
http://ftp.debian.org/debian
| I: Retrieving InRelease
| I: Retrieving Release
| I: Retrieving Release.gpg
| I: Checking Release signature
| E: Release signed by unknown key (key id EF0F382A1A7B6500)
| WARNING: 
/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/temp/run.do_bootstrap.27078:1 
exit 1 from 'sudo -E -s  <<'EOSUDO'
|         ( flock 9
|         set -e
|         if [ ! -e 
"/mnt/ssd/isar/upstream/build/tmp/deploy/bootstrap/debian-stretch-amd64" 
]; then
|             rm -rf 
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs"
|
|             if [ ${IS_HOST} ]; then
|                 qemu-debootstrap $debootstrap_args \
|                                --components=main,contrib,non-free \
|                                --keyring 
/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/apt-keyring.gpg 
\
|                                "stretch" \
| 
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs" 
\
|                                "http://ftp.debian.org/debian"
|
|             else
|                  "qemu-debootstrap" $debootstrap_args \
|                                   --arch="armhf" \
| 
--components=main,contrib,non-free,firmware \
|                                   --keyring 
/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/apt-keyring.gpg 
\
|                                   "stretch" \
| 
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs" 
\
|                                   "http://archive.raspbian.org/raspbian"
|             fi
|
|             # Install apt config
|             mkdir -p 
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/etc/apt/preferences.d"
|             install -v -m644 
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/apt-preferences" 
\
| 
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/etc/apt/preferences.d/bootstrap"
|             mkdir -p 
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/etc/apt/sources.list.d"
|             if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
|                 sed -e "s#{BASE_DISTRO}#"raspbian"#g" \
|                     -e "s#{BASE_DISTRO_CODENAME}#"stretch"#g" \
| 	            -i 
/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/base-apt-sources
|                 mkdir -p 
/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/base-apt
|                 sudo mount --bind 
/mnt/ssd/isar/upstream/build/downloads/base-apt/apt 
/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/base-apt
|                 install -v -m644 
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/base-apt-sources" 
\
| 
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/etc/apt/sources.list.d/base-apt.list"
|             else
|                 install -v -m644 
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/apt-sources" 
\
| 
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/etc/apt/sources.list.d/bootstrap.list"
|             fi
|             install -v -m644 
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/apt-sources-init" 
\
| 
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/etc/apt/sources-list"
|             rm -f 
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/etc/apt/sources.list"
|             mkdir -p 
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/etc/apt/apt.conf.d"
|             install -v -m644 
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/isar-apt.conf" 
\
| 
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/etc/apt/apt.conf.d/50isar.conf"
|
|             if [ "stretch" = "stretch" ] && [ "4" -lt "4" ]; then
|                 install -v -m644 
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/isar-apt-fallback.conf" 
\
| 
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/etc/apt/apt.conf.d/55isar-fallback.conf"
|             fi
|
|             # Set locale
|             install -v -m644 
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/locale" 
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/etc/locale"
|
|             sed -i '/en_US.UTF-8 UTF-8/s/^#//g' 
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/etc/locale.gen"
|             chroot 
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs" 
/usr/sbin/locale-gen
|
|             # setup chroot
|             install -v -m755 
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/chroot-setup.sh" 
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/chroot-setup.sh"
| 
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/chroot-setup.sh" 
"setup" 
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs"
|
|             # update APT
|             mount --rbind /dev 
/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/dev
|             mount --make-rslave 
/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/dev
|             mount -t proc none 
/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/proc
|             mount --rbind /sys 
/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/sys
|             mount --make-rslave 
/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/sys
|
|             export DEBIAN_FRONTEND=noninteractive
|             chroot 
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs" 
/usr/bin/apt-get update -y
|             chroot 
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs" 
/usr/bin/apt-get dist-upgrade -y \
|                                   -o Debug::pkgProblemResolver=yes
|
|             umount -l 
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/dev"
|             umount -l 
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/proc"
|             umount -l 
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/sys"
|             umount -l 
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/base-apt" 
|| true
|
|             # Finalize debootstrap by setting the link in deploy
|             ln -Tfsr 
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs" 
"/mnt/ssd/isar/upstream/build/tmp/deploy/bootstrap/debian-stretch-amd64"
|         fi
|         ) 
9>'/mnt/ssd/isar/upstream/build/tmp/deploy/bootstrap/debian-stretch-amd64.lock'
| EOSUDO
| '
| ERROR: Function failed: do_bootstrap (log file is located at 
/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/temp/log.do_bootstrap.27078)
ERROR: Task 
(multiconfig:rpi-stretch:/mnt/data/isar/layers/isar-master/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb:do_bootstrap) 
failed with exit code '1'
--- end error log ---

So pretty much the same error I currently deal with.

Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [PATCH v3 0/8] Cleanup rootfs creation

[Claudius Heine]

On 25/04/2019 13.18, [ext] Claudius Heine wrote:
> Hi Baurzhan,
> 
> On 25/04/2019 12.32, Baurzhan Ismagulov wrote:
>> On Thu, Apr 25, 2019 at 11:11:21AM +0200, Claudius Heine wrote:
>>> I looked into the "fast" build now, and it cross-compiles
>>> "multiconfig:rpi-stretch:isar-image-base". Now I tried that and the 
>>> build
>>> fails even on the current 'next' so the issue seems to be there.
>>
>> How does the problem manifest itself? Please provide some relevant 
>> details
>> (e.g., bitbake output, failure logs, etc.).
> 
> Commit ID: 2396d4f44fe1f867ddae389b61cb7017cb2b908c
> 
> --- local.conf ---
> # cross
> ISAR_CROSS_COMPILE = "1"
> 
> # standard
> CONF_VERSION = "1"
> 
> MACHINE ??= "qemu"
> DISTRO ??= "raspbian-stretch"
> BBMULTICONFIG ?= "rpi-stretch"
> --- end local.conf ---
> 
> bitbake -k -c build multiconfig:rpi-stretch:isar-image-base
> 
> --- error log ---
> ERROR: mc:rpi-stretch:isar-bootstrap-host-1.0-r0 do_bootstrap: Function 
> failed: do_bootstrap (log file is located at 
> /mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/temp/log.do_bootstrap.27078) 
> 
> ERROR: Logfile of failure stored in: 
> /mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/temp/log.do_bootstrap.27078 
> 
> Log data follows:
> | DEBUG: Executing shell function do_bootstrap
> | W: Target architecture is the same as host architecture; disabling 
> QEMU support
> | I: Running command: debootstrap --arch amd64 --verbose 
> --variant=minbase --include=locales --components=main,contrib,non-free 
> --keyring 
> /mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/apt-keyring.gpg 
> stretch 
> /mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs 
> http://ftp.debian.org/debian

Just some speculation here. I find it strange that a normal debian 
stretch bootstrap gets a '--keyring' parameter in that case. What is 
bootstrapped here is a amd64 rootfs and the keyring contains the one for 
raspbian.

But I don't know why this causes an error for me here, but not on the 
CI. But only on the CI when I add my cleanup patches.

Claudius

> | I: Retrieving InRelease
> | I: Retrieving Release
> | I: Retrieving Release.gpg
> | I: Checking Release signature
> | E: Release signed by unknown key (key id EF0F382A1A7B6500)
> | WARNING: 
> /mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/temp/run.do_bootstrap.27078:1 
> exit 1 from 'sudo -E -s  <<'EOSUDO'
> |         ( flock 9
> |         set -e
> |         if [ ! -e 
> "/mnt/ssd/isar/upstream/build/tmp/deploy/bootstrap/debian-stretch-amd64" 
> ]; then
> |             rm -rf 
> "/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs" 
> 
> |
> |             if [ ${IS_HOST} ]; then
> |                 qemu-debootstrap $debootstrap_args \
> |                                --components=main,contrib,non-free \
> |                                --keyring 
> /mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/apt-keyring.gpg 
> \
> |                                "stretch" \
> | 
> "/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs" 
> \
> |                                "http://ftp.debian.org/debian"
> |
> |             else
> |                  "qemu-debootstrap" $debootstrap_args \
> |                                   --arch="armhf" \
> | --components=main,contrib,non-free,firmware \
> |                                   --keyring 
> /mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/apt-keyring.gpg 
> \
> |                                   "stretch" \
> | 
> "/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs" 
> \
> |                                   "http://archive.raspbian.org/raspbian"
> |             fi
> |
> |             # Install apt config
> |             mkdir -p 
> "/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/etc/apt/preferences.d" 
> 
> |             install -v -m644 
> "/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/apt-preferences" 
> \
> | 
> "/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/etc/apt/preferences.d/bootstrap" 
> 
> |             mkdir -p 
> "/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/etc/apt/sources.list.d" 
> 
> |             if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
> |                 sed -e "s#{BASE_DISTRO}#"raspbian"#g" \
> |                     -e "s#{BASE_DISTRO_CODENAME}#"stretch"#g" \
> |                 -i 
> /mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/base-apt-sources 
> 
> |                 mkdir -p 
> /mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/base-apt 
> 
> |                 sudo mount --bind 
> /mnt/ssd/isar/upstream/build/downloads/base-apt/apt 
> /mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/base-apt 
> 
> |                 install -v -m644 
> "/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/base-apt-sources" 
> \
> | 
> "/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/etc/apt/sources.list.d/base-apt.list" 
> 
> |             else
> |                 install -v -m644 
> "/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/apt-sources" 
> \
> | 
> "/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/etc/apt/sources.list.d/bootstrap.list" 
> 
> |             fi
> |             install -v -m644 
> "/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/apt-sources-init" 
> \
> | 
> "/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/etc/apt/sources-list" 
> 
> |             rm -f 
> "/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/etc/apt/sources.list" 
> 
> |             mkdir -p 
> "/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/etc/apt/apt.conf.d" 
> 
> |             install -v -m644 
> "/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/isar-apt.conf" 
> \
> | 
> "/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/etc/apt/apt.conf.d/50isar.conf" 
> 
> |
> |             if [ "stretch" = "stretch" ] && [ "4" -lt "4" ]; then
> |                 install -v -m644 
> "/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/isar-apt-fallback.conf" 
> \
> | 
> "/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/etc/apt/apt.conf.d/55isar-fallback.conf" 
> 
> |             fi
> |
> |             # Set locale
> |             install -v -m644 
> "/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/locale" 
> "/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/etc/locale" 
> 
> |
> |             sed -i '/en_US.UTF-8 UTF-8/s/^#//g' 
> "/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/etc/locale.gen" 
> 
> |             chroot 
> "/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs" 
> /usr/sbin/locale-gen
> |
> |             # setup chroot
> |             install -v -m755 
> "/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/chroot-setup.sh" 
> "/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/chroot-setup.sh" 
> 
> | 
> "/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/chroot-setup.sh" 
> "setup" 
> "/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs" 
> 
> |
> |             # update APT
> |             mount --rbind /dev 
> /mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/dev 
> 
> |             mount --make-rslave 
> /mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/dev 
> 
> |             mount -t proc none 
> /mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/proc 
> 
> |             mount --rbind /sys 
> /mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/sys 
> 
> |             mount --make-rslave 
> /mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/sys 
> 
> |
> |             export DEBIAN_FRONTEND=noninteractive
> |             chroot 
> "/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs" 
> /usr/bin/apt-get update -y
> |             chroot 
> "/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs" 
> /usr/bin/apt-get dist-upgrade -y \
> |                                   -o Debug::pkgProblemResolver=yes
> |
> |             umount -l 
> "/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/dev" 
> 
> |             umount -l 
> "/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/proc" 
> 
> |             umount -l 
> "/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/sys" 
> 
> |             umount -l 
> "/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/base-apt" 
> || true
> |
> |             # Finalize debootstrap by setting the link in deploy
> |             ln -Tfsr 
> "/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs" 
> "/mnt/ssd/isar/upstream/build/tmp/deploy/bootstrap/debian-stretch-amd64"
> |         fi
> |         ) 
> 9>'/mnt/ssd/isar/upstream/build/tmp/deploy/bootstrap/debian-stretch-amd64.lock' 
> 
> | EOSUDO
> | '
> | ERROR: Function failed: do_bootstrap (log file is located at 
> /mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/temp/log.do_bootstrap.27078) 
> 
> ERROR: Task 
> (multiconfig:rpi-stretch:/mnt/data/isar/layers/isar-master/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb:do_bootstrap) 
> failed with exit code '1'
> --- end error log ---
> 
> So pretty much the same error I currently deal with.
> 
> Claudius
> 

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [PATCH v3 1/8] isar-boostrap-helper: move 'HOST_ARCH' and 'HOST_DISTRO' to base.bbclass

[Claudius Heine]

Hi,

On 24/04/2019 11.27, [ext] claudius.heine.ext@siemens.com wrote:
> From: Claudius Heine <ch@denx.de>
> 
> This also sets the 'HOST_DISTRO' to 'DISTRO' per default. (jessie
> support was dropped.)
> 
> Signed-off-by: Claudius Heine <ch@denx.de>
> ---
>   meta/classes/base.bbclass                  |  9 +++++++++
>   meta/classes/isar-bootstrap-helper.bbclass | 12 ------------
>   2 files changed, 9 insertions(+), 12 deletions(-)
> 
> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
> index 9a1837f..ca80bf8 100644
> --- a/meta/classes/base.bbclass
> +++ b/meta/classes/base.bbclass
> @@ -20,6 +20,15 @@
>   
>   THISDIR = "${@os.path.dirname(d.getVar('FILE', True))}"
>   
> +def get_deb_host_arch():
> +    import subprocess
> +    host_arch = subprocess.check_output(
> +        ["dpkg", "--print-architecture"]
> +    ).decode('utf-8').strip()
> +    return host_arch
> +HOST_ARCH ??= "${@get_deb_host_arch()}"
> +HOST_DISTRO ??= "${DISTRO}"

I did find in issue here when debugging.
In case of raspbian-stretch HOST_DISTRO needs to be 'debian-stretch'.

I corrected that for the next version of this patchset.

Currently the same issue in CI issue now reoccurs with the 
'isar-bootstrap/buildchroot/sdkchroot: refactor PF and WORKDIR' patch 
though.

I am currently investigating that.

Claudius

> +
>   die() {
>   	bbfatal "$*"
>   }
> diff --git a/meta/classes/isar-bootstrap-helper.bbclass b/meta/classes/isar-bootstrap-helper.bbclass
> index 38d90fb..59c447b 100644
> --- a/meta/classes/isar-bootstrap-helper.bbclass
> +++ b/meta/classes/isar-bootstrap-helper.bbclass
> @@ -7,18 +7,6 @@
>   
>   IMAGE_TRANSIENT_PACKAGES ??= ""
>   
> -def get_deb_host_arch():
> -    import subprocess
> -    host_arch = subprocess.check_output(
> -        ["dpkg", "--print-architecture"]
> -    ).decode('utf-8').strip()
> -    return host_arch
> -
> -#Debian Distribution for SDK host
> -HOST_DISTRO ?= "debian-stretch"
> -#Determine SDK host architecture if not explicitly set
> -HOST_ARCH ?= "${@get_deb_host_arch()}"
> -
>   HOST_DISTRO_APT_SOURCES += "conf/distro/${HOST_DISTRO}.list"
>   
>   def reverse_bb_array(d, varname):
> 

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de