[PATCH v2 0/7] Fixing base-apt and ci

[PATCH v2 0/7] Fixing base-apt and ci

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

Hi,

here is a small update to the patchset before.

Here is the CI run:
http://isar-build.org:8080/job/isar_claudius_ilbers_fast_ci/35/console

regards,
Claudius

Changes from v1:
 - added warning about usage of IMAGE_TRANSIENT_PACKAGES usage
 - rebased
 - removed unnecessary ci_build changes
 - added build of qemuarm-buster

Claudius Heine (7):
  meta/image.bbclass: add warning if IMAGE_TRANSIENT_PACKAGES is set
  sdkchroot: remove 'copy-package-cache' from ROOTFS_FEATURES
  bitbake: added DISTRO to REPO_BASE_{DB_}DIR
  isar-bootstrap: fix DISTRO_APT_PREMIRRORS regex in case of cached
    build
  base-apt: fix issues with creating and using the base-apt repository
  ci_build: fix shebang
  ci_build/repro: add qemuarm-buster to build configurations

 meta/classes/image-cache-extension.bbclass          | 10 ++++++++++
 meta/classes/image.bbclass                          |  5 +++++
 meta/conf/bitbake.conf                              |  4 ++--
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc |  3 ++-
 meta/recipes-devtools/sdkchroot/sdkchroot.bb        |  2 +-
 scripts/ci_build.sh                                 |  7 ++++---
 6 files changed, 24 insertions(+), 7 deletions(-)

-- 
2.20.1

[PATCH v2 1/7] meta/image.bbclass: add warning if IMAGE_TRANSIENT_PACKAGES is set

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

Signed-off-by: Claudius Heine <ch@denx.de>
---
 meta/classes/image.bbclass | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index ef7983b..fae1e0c 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -27,6 +27,11 @@ BUILDROOT_DEPLOY = "${BUILDCHROOT_DIR}${PP_DEPLOY}"
 BUILDROOT_ROOTFS = "${BUILDCHROOT_DIR}${PP_ROOTFS}"
 BUILDROOT_WORK = "${BUILDCHROOT_DIR}${PP_WORK}"
 
+python(){
+    if (d.getVar('IMAGE_TRANSIENT_PACKAGES')):
+        bb.warn("IMAGE_TRANSIENT_PACKAGES is set and no longer supported")
+}
+
 def cfg_script(d):
     cf = d.getVar('DISTRO_CONFIG_SCRIPT', True) or ''
     if cf:
-- 
2.20.1

[PATCH v2 2/7] sdkchroot: remove 'copy-package-cache' from ROOTFS_FEATURES

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

sdkchroot does not use the packages copied by the `copy-package-cache`
feature anywhere.

Signed-off-by: Claudius Heine <ch@denx.de>
---
 meta/recipes-devtools/sdkchroot/sdkchroot.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-devtools/sdkchroot/sdkchroot.bb b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
index 57d52ba..3cfe05c 100644
--- a/meta/recipes-devtools/sdkchroot/sdkchroot.bb
+++ b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
@@ -23,7 +23,7 @@ ROOTFS_ARCH = "${HOST_ARCH}"
 ROOTFS_DISTRO = "${HOST_DISTRO}"
 ROOTFSDIR = "${S}"
 ROOTFS_PACKAGES = "${SDKCHROOT_PREINSTALL} ${TOOLCHAIN}"
-ROOTFS_FEATURES += "copy-package-cache"
+ROOTFS_FEATURES += ""
 
 python() {
     if d.getVar("HOST_ARCH") not in ['i386', 'amd64']:
-- 
2.20.1

[PATCH v2 3/7] bitbake: added DISTRO to REPO_BASE_{DB_}DIR

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

The base-apt repository can only contain one codename, so the path
should be specific to the DISTRO variable to avoid conflicts.

Signed-off-by: Claudius Heine <ch@denx.de>
---
 meta/conf/bitbake.conf | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 381c121..3d18223 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -69,8 +69,8 @@ REPO_ISAR_DB_DIR = "${DEPLOY_DIR}/isar-apt/db"
 THIRD_PARTY_APT_KEYRING = "/etc/apt/trusted.gpg.d/third_party.gpg"
 
 # Base apt repository paths
-REPO_BASE_DIR = "${DL_DIR}/base-apt/apt"
-REPO_BASE_DB_DIR = "${DL_DIR}/base-apt/db"
+REPO_BASE_DIR = "${DL_DIR}/base-apt/${DISTRO}/apt"
+REPO_BASE_DB_DIR = "${DL_DIR}/base-apt/${DISTRO}/db"
 
 BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DIR \
     SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \
-- 
2.20.1

[PATCH v2 4/7] isar-bootstrap: fix DISTRO_APT_PREMIRRORS regex in case of cached build

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

The regex before did not replace the distribution name in the url. The
new one will replace the whole url with the REPO_BASE_DIR + BASE_DISTRO
path.

Signed-off-by: Claudius Heine <ch@denx.de>
---
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index f8741c2..18e5e8f 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -30,7 +30,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = ",gnupg2"
 DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = ",apt-transport-https,ca-certificates"
 HOST_DISTRO_APT_SOURCES += "conf/distro/${HOST_DISTRO}.list"
 
-DISTRO_APT_PREMIRRORS ?= "${@ "http://ftp\.(\S+\.)?debian.org  file:///${REPO_BASE_DIR} \n" if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) else "" }"
+DISTRO_APT_PREMIRRORS ?= "${@ "http://ftp\.(\S+\.)?debian.org/\S* file:///${REPO_BASE_DIR}/${BASE_DISTRO} \n" if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) else "" }"
 
 inherit base-apt-helper
 
-- 
2.20.1

[PATCH v2 5/7] base-apt: fix issues with creating and using the base-apt repository

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

Currently we only support one repository for one image, but since the
image creation uses two root file systems in case of cross-building,
package of both will land into the same repository. This should not be
an issue if both root file system use the same distribution + codename,
since shared ('all' arch) packages where shared in the upstream
repository as well. But in case those differ, different versions could
be used when building out of the base-apt later.

This fix is two fold, for once it disables the creation of base-apt
repositories if the distribution+codename of the two root file systems
differ and second it adds a 'apt-get install --fix-broken' call before
the 'dist-upgrade' one, in order to remove broken dependencies, that
where installed when such a base-apt repository was used.

Signed-off-by: Claudius Heine <ch@denx.de>
---
 meta/classes/image-cache-extension.bbclass          | 10 ++++++++++
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc |  1 +
 2 files changed, 11 insertions(+)

diff --git a/meta/classes/image-cache-extension.bbclass b/meta/classes/image-cache-extension.bbclass
index af17e0b..30db549 100644
--- a/meta/classes/image-cache-extension.bbclass
+++ b/meta/classes/image-cache-extension.bbclass
@@ -11,6 +11,16 @@ do_cache_base_repo[depends] = "base-apt:do_cache_config"
 do_cache_base_repo[lockfiles] = "${REPO_BASE_DIR}/isar.lock"
 do_cache_base_repo[stamp-extra-info] = "${DISTRO}-${MACHINE}"
 do_cache_base_repo() {
+    if [ -d '${BUILDCHROOT_HOST_DIR}/var/cache/apt' ] &&
+        [ '${DISTRO}' != '${HOST_DISTRO}' ]; then
+        # We would need two separate repository paths for that.
+        # Otherwise packages (especially the 'all' arch ones) from one
+        # distribution can influence the package versions of the other
+        # distribution.
+        bbfatal "Different host and target distributions are currently not supported." \
+                "Try it without cross-build."
+    fi
+
     if [ -d ${WORKDIR}/apt_cache ]; then
         populate_base_apt ${WORKDIR}/apt_cache
     fi
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 18e5e8f..2bc5ea5 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -333,6 +333,7 @@ isar_bootstrap() {
             fi
 
             chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y
+            chroot "${ROOTFSDIR}" /usr/bin/apt-get install -y -f
             chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \
                                   -o Debug::pkgProblemResolver=yes
 
-- 
2.20.1

[PATCH v2 6/7] ci_build: fix shebang

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

Signed-off-by: Claudius Heine <ch@denx.de>
---
 scripts/ci_build.sh | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index 9d370ac..f05dd04 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -1,10 +1,9 @@
+#!/usr/bin/env bash
 # Script for CI system build
 #
 # Alexander Smirnov <asmirnov@ilbers.de>
 # Copyright (c) 2016-2018 ilbers GmbH
 
-#!/bin/sh
-
 set -e
 
 ES_BUG=3
-- 
2.20.1

[PATCH v2 7/7] ci_build/repro: add qemuarm-buster to build configurations

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

Should be possible to build the base-apt cache for multiple codenames
at the same time.

Signed-off-by: Claudius Heine <ch@denx.de>
---
 scripts/ci_build.sh | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index f05dd04..a10d4e8 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -36,7 +36,9 @@ TARGETS_SET="\
 REPRO_TARGETS_SET="\
             multiconfig:qemuarm-stretch:isar-image-base \
             multiconfig:qemuarm64-stretch:isar-image-base \
-            multiconfig:qemuamd64-stretch:isar-image-base"
+            multiconfig:qemuamd64-stretch:isar-image-base \
+            multiconfig:qemuarm-buster:isar-image-base"
+
 
 show_help() {
     echo "This script builds the default Isar images."
-- 
2.20.1

Re: [PATCH v2 0/7] Fixing base-apt and ci

[Maxim Yu. Osipov]

On 5/23/19 5:00 PM, claudius.heine.ext@siemens.com wrote:
> From: Claudius Heine <ch@denx.de>
> 
> Hi,
> 
> here is a small update to the patchset before.
> 
> Here is the CI run:
> http://isar-build.org:8080/job/isar_claudius_ilbers_fast_ci/35/console
> 
> regards,
> Claudius
> 
> Changes from v1:
>   - added warning about usage of IMAGE_TRANSIENT_PACKAGES usage
>   - rebased
>   - removed unnecessary ci_build changes
>   - added build of qemuarm-buster

Applied to the 'next'.

Regards,
Maxim.

> Claudius Heine (7):
>    meta/image.bbclass: add warning if IMAGE_TRANSIENT_PACKAGES is set
>    sdkchroot: remove 'copy-package-cache' from ROOTFS_FEATURES
>    bitbake: added DISTRO to REPO_BASE_{DB_}DIR
>    isar-bootstrap: fix DISTRO_APT_PREMIRRORS regex in case of cached
>      build
>    base-apt: fix issues with creating and using the base-apt repository
>    ci_build: fix shebang
>    ci_build/repro: add qemuarm-buster to build configurations
> 
>   meta/classes/image-cache-extension.bbclass          | 10 ++++++++++
>   meta/classes/image.bbclass                          |  5 +++++
>   meta/conf/bitbake.conf                              |  4 ++--
>   meta/recipes-core/isar-bootstrap/isar-bootstrap.inc |  3 ++-
>   meta/recipes-devtools/sdkchroot/sdkchroot.bb        |  2 +-
>   scripts/ci_build.sh                                 |  7 ++++---
>   6 files changed, 24 insertions(+), 7 deletions(-)
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov