[PATCH 0/6] Fixing base-apt and ci

[PATCH 0/6] Fixing base-apt and ci

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

Hi,

this patchset depends on the "pre-processing pipeline and transient package
replacement" patchset and fixes some issues related to the base-apt
implementation and CI test.

CI test:
http://isar-build.org:8080/job/isar_claudius_ilbers_fast_ci/33/console
(This run was probably triggered with 'ci_build.sh -f -r')

regards,
Claudius

Claudius Heine (6):
  sdkchroot: remove 'copy-package-cache' from ROOTFS_FEATURES
  bitbake: added DISTRO to REPO_BASE_{DB_}DIR
  isar-bootstrap: fix DISTRO_APT_PREMIRRORS regex in case of cached
    build
  base-apt: fix issues with creating and using the base-apt repository
  ci_build.sh: Remove rpi-stretch if CROSS_BUILD and REPRO_BUILD
  ci_build.sh/repro: disable build of non-cached targets

 meta/classes/image-cache-extension.bbclass          | 10 ++++++++++
 meta/conf/bitbake.conf                              |  4 ++--
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc |  3 ++-
 meta/recipes-devtools/sdkchroot/sdkchroot.bb        |  2 +-
 scripts/ci_build.sh                                 | 13 +++++++++++--
 5 files changed, 26 insertions(+), 6 deletions(-)

-- 
2.20.1

[PATCH 1/6] sdkchroot: remove 'copy-package-cache' from ROOTFS_FEATURES

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

sdkchroot does not use the packages copied by the `copy-package-cache`
feature anywhere.

Signed-off-by: Claudius Heine <ch@denx.de>
---
 meta/recipes-devtools/sdkchroot/sdkchroot.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-devtools/sdkchroot/sdkchroot.bb b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
index 57d52ba..3cfe05c 100644
--- a/meta/recipes-devtools/sdkchroot/sdkchroot.bb
+++ b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
@@ -23,7 +23,7 @@ ROOTFS_ARCH = "${HOST_ARCH}"
 ROOTFS_DISTRO = "${HOST_DISTRO}"
 ROOTFSDIR = "${S}"
 ROOTFS_PACKAGES = "${SDKCHROOT_PREINSTALL} ${TOOLCHAIN}"
-ROOTFS_FEATURES += "copy-package-cache"
+ROOTFS_FEATURES += ""
 
 python() {
     if d.getVar("HOST_ARCH") not in ['i386', 'amd64']:
-- 
2.20.1

[PATCH 2/6] bitbake: added DISTRO to REPO_BASE_{DB_}DIR

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

The base-apt repository can only contain one codename, so the path
should be specific to the DISTRO variable to avoid conflicts.

Signed-off-by: Claudius Heine <ch@denx.de>
---
 meta/conf/bitbake.conf | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 381c121..3d18223 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -69,8 +69,8 @@ REPO_ISAR_DB_DIR = "${DEPLOY_DIR}/isar-apt/db"
 THIRD_PARTY_APT_KEYRING = "/etc/apt/trusted.gpg.d/third_party.gpg"
 
 # Base apt repository paths
-REPO_BASE_DIR = "${DL_DIR}/base-apt/apt"
-REPO_BASE_DB_DIR = "${DL_DIR}/base-apt/db"
+REPO_BASE_DIR = "${DL_DIR}/base-apt/${DISTRO}/apt"
+REPO_BASE_DB_DIR = "${DL_DIR}/base-apt/${DISTRO}/db"
 
 BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DIR \
     SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \
-- 
2.20.1

[PATCH 3/6] isar-bootstrap: fix DISTRO_APT_PREMIRRORS regex in case of cached build

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

The regex before did not replace the distribution name in the url. The
new one will replace the whole url with the REPO_BASE_DIR + BASE_DISTRO
path.

Signed-off-by: Claudius Heine <ch@denx.de>
---
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index f8741c2..18e5e8f 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -30,7 +30,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = ",gnupg2"
 DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = ",apt-transport-https,ca-certificates"
 HOST_DISTRO_APT_SOURCES += "conf/distro/${HOST_DISTRO}.list"
 
-DISTRO_APT_PREMIRRORS ?= "${@ "http://ftp\.(\S+\.)?debian.org  file:///${REPO_BASE_DIR} \n" if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) else "" }"
+DISTRO_APT_PREMIRRORS ?= "${@ "http://ftp\.(\S+\.)?debian.org/\S* file:///${REPO_BASE_DIR}/${BASE_DISTRO} \n" if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) else "" }"
 
 inherit base-apt-helper
 
-- 
2.20.1

[PATCH 4/6] base-apt: fix issues with creating and using the base-apt repository

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

Currently we only support one repository for one image, but since the
image creation uses two root file systems in case of cross-building,
package of both will land into the same repository. This should not be
an issue if both root file system use the same distribution + codename,
since shared ('all' arch) packages where shared in the upstream
repository as well. But in case those differ, different versions could
be used when building out of the base-apt later.

This fix is two fold, for once it disables the creation of base-apt
repositories if the distribution+codename of the two root file systems
differ and second it adds a 'apt-get install --fix-broken' call before
the 'dist-upgrade' one, in order to remove broken dependencies, that
where installed when such a base-apt repository was used.

Signed-off-by: Claudius Heine <ch@denx.de>
---
 meta/classes/image-cache-extension.bbclass          | 10 ++++++++++
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc |  1 +
 2 files changed, 11 insertions(+)

diff --git a/meta/classes/image-cache-extension.bbclass b/meta/classes/image-cache-extension.bbclass
index af17e0b..30db549 100644
--- a/meta/classes/image-cache-extension.bbclass
+++ b/meta/classes/image-cache-extension.bbclass
@@ -11,6 +11,16 @@ do_cache_base_repo[depends] = "base-apt:do_cache_config"
 do_cache_base_repo[lockfiles] = "${REPO_BASE_DIR}/isar.lock"
 do_cache_base_repo[stamp-extra-info] = "${DISTRO}-${MACHINE}"
 do_cache_base_repo() {
+    if [ -d '${BUILDCHROOT_HOST_DIR}/var/cache/apt' ] &&
+        [ '${DISTRO}' != '${HOST_DISTRO}' ]; then
+        # We would need two separate repository paths for that.
+        # Otherwise packages (especially the 'all' arch ones) from one
+        # distribution can influence the package versions of the other
+        # distribution.
+        bbfatal "Different host and target distributions are currently not supported." \
+                "Try it without cross-build."
+    fi
+
     if [ -d ${WORKDIR}/apt_cache ]; then
         populate_base_apt ${WORKDIR}/apt_cache
     fi
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 18e5e8f..2bc5ea5 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -333,6 +333,7 @@ isar_bootstrap() {
             fi
 
             chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y
+            chroot "${ROOTFSDIR}" /usr/bin/apt-get install -y -f
             chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \
                                   -o Debug::pkgProblemResolver=yes
 
-- 
2.20.1

[PATCH 5/6] ci_build.sh: Remove rpi-stretch if CROSS_BUILD and REPRO_BUILD

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

Contains also a fix to the shebang. It needs to be on the first line and
since it already uses bashisms it was changed from sh to bash.

Signed-off-by: Claudius Heine <ch@denx.de>
---
 scripts/ci_build.sh | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index a38e2b7..b8a504d 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -1,10 +1,9 @@
+#!/usr/bin/env bash
 # Script for CI system build
 #
 # Alexander Smirnov <asmirnov@ilbers.de>
 # Copyright (c) 2016-2018 ilbers GmbH
 
-#!/bin/sh
-
 set -e
 
 ES_BUG=3
@@ -114,6 +113,10 @@ if [ -n "$CROSS_BUILD" ]; then
 fi
 
 if [ -n "$REPRO_BUILD" ]; then
+    if [ -n "$CROSS_BUILD" ]; then
+        # remove rpi-stretch, since that is currently incompatible
+        TARGETS_SET="${TARGETS_SET//multiconfig:rpi-stretch:isar-image-base/}"
+    fi
     # Enable use of cached base repository
     bitbake $BB_ARGS -c cache_base_repo  $TARGETS_SET
     while [ -e bitbake.sock ]; do sleep 1; done
-- 
2.20.1

[PATCH 6/6] ci_build.sh/repro: disable build of non-cached targets

[claudius.heine.ext]

From: Claudius Heine <ch@denx.de>

Signed-off-by: Claudius Heine <ch@denx.de>
---
 scripts/ci_build.sh | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index b8a504d..3af00be 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -127,6 +127,12 @@ fi
 # Start build for the defined set of configurations
 bitbake $BB_ARGS $TARGETS_SET
 
+# Don't build anything else with 'ISAR_USE_CACHED_BASE_REPO = "1"', since
+# that is not cached:
+if [ -n "$REPRO_BUILD" ]; then
+    exit 0
+fi
+
 # Note: de0-nano-soc build is launched separately as
 # parallel build with the same target arch (armhf) fails.
 # The problem is being investigated
-- 
2.20.1

Re: [PATCH 0/6] Fixing base-apt and ci

[Maxim Yu. Osipov]

Hi Claudius,

It seems that you need to rebase first against the current 'next'.
Definitely, this relates to ci_build.

Regards,
Maxim.

On 5/23/19 1:39 PM, claudius.heine.ext@siemens.com wrote:
> From: Claudius Heine <ch@denx.de>
> 
> Hi,
> 
> this patchset depends on the "pre-processing pipeline and transient package
> replacement" patchset and fixes some issues related to the base-apt
> implementation and CI test.
> 
> CI test:
> http://isar-build.org:8080/job/isar_claudius_ilbers_fast_ci/33/console
> (This run was probably triggered with 'ci_build.sh -f -r')
> 
> regards,
> Claudius
> 
> Claudius Heine (6):
>    sdkchroot: remove 'copy-package-cache' from ROOTFS_FEATURES
>    bitbake: added DISTRO to REPO_BASE_{DB_}DIR
>    isar-bootstrap: fix DISTRO_APT_PREMIRRORS regex in case of cached
>      build
>    base-apt: fix issues with creating and using the base-apt repository
>    ci_build.sh: Remove rpi-stretch if CROSS_BUILD and REPRO_BUILD
>    ci_build.sh/repro: disable build of non-cached targets
> 
>   meta/classes/image-cache-extension.bbclass          | 10 ++++++++++
>   meta/conf/bitbake.conf                              |  4 ++--
>   meta/recipes-core/isar-bootstrap/isar-bootstrap.inc |  3 ++-
>   meta/recipes-devtools/sdkchroot/sdkchroot.bb        |  2 +-
>   scripts/ci_build.sh                                 | 13 +++++++++++--
>   5 files changed, 26 insertions(+), 6 deletions(-)
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: [PATCH 0/6] Fixing base-apt and ci

[Claudius Heine]

Hi Maxim,

On 23/05/2019 13.59, Maxim Yu. Osipov wrote:
> Hi Claudius,
> 
> It seems that you need to rebase first against the current 'next'.
> Definitely, this relates to ci_build.

Ok, if I understand your changes to ci_build, then those two patches 
might no longer be needed, since the they should have resolved those issues.

But I will do a CI run to be sure.

thx,
Claudius

> 
> Regards,
> Maxim.
> 
> On 5/23/19 1:39 PM, claudius.heine.ext@siemens.com wrote:
>> From: Claudius Heine <ch@denx.de>
>>
>> Hi,
>>
>> this patchset depends on the "pre-processing pipeline and transient 
>> package
>> replacement" patchset and fixes some issues related to the base-apt
>> implementation and CI test.
>>
>> CI test:
>> http://isar-build.org:8080/job/isar_claudius_ilbers_fast_ci/33/console
>> (This run was probably triggered with 'ci_build.sh -f -r')
>>
>> regards,
>> Claudius
>>
>> Claudius Heine (6):
>>    sdkchroot: remove 'copy-package-cache' from ROOTFS_FEATURES
>>    bitbake: added DISTRO to REPO_BASE_{DB_}DIR
>>    isar-bootstrap: fix DISTRO_APT_PREMIRRORS regex in case of cached
>>      build
>>    base-apt: fix issues with creating and using the base-apt repository
>>    ci_build.sh: Remove rpi-stretch if CROSS_BUILD and REPRO_BUILD
>>    ci_build.sh/repro: disable build of non-cached targets
>>
>>   meta/classes/image-cache-extension.bbclass          | 10 ++++++++++
>>   meta/conf/bitbake.conf                              |  4 ++--
>>   meta/recipes-core/isar-bootstrap/isar-bootstrap.inc |  3 ++-
>>   meta/recipes-devtools/sdkchroot/sdkchroot.bb        |  2 +-
>>   scripts/ci_build.sh                                 | 13 +++++++++++--
>>   5 files changed, 26 insertions(+), 6 deletions(-)
>>
> 
> 

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [PATCH 0/6] Fixing base-apt and ci

[Maxim Yu. Osipov]

On 5/23/19 2:21 PM, Claudius Heine wrote:
> Hi Maxim,
> 
> On 23/05/2019 13.59, Maxim Yu. Osipov wrote:
>> Hi Claudius,
>>
>> It seems that you need to rebase first against the current 'next'.
>> Definitely, this relates to ci_build.
> 
> Ok, if I understand your changes to ci_build, then those two patches 
> might no longer be needed, since the they should have resolved those 
> issues.

shebang fix is OK.

Thanks,
Maxim.

> But I will do a CI run to be sure.
> 
> thx,
> Claudius
> 
>>
>> Regards,
>> Maxim.
>>
>> On 5/23/19 1:39 PM, claudius.heine.ext@siemens.com wrote:
>>> From: Claudius Heine <ch@denx.de>
>>>
>>> Hi,
>>>
>>> this patchset depends on the "pre-processing pipeline and transient 
>>> package
>>> replacement" patchset and fixes some issues related to the base-apt
>>> implementation and CI test.
>>>
>>> CI test:
>>> http://isar-build.org:8080/job/isar_claudius_ilbers_fast_ci/33/console
>>> (This run was probably triggered with 'ci_build.sh -f -r')
>>>
>>> regards,
>>> Claudius
>>>
>>> Claudius Heine (6):
>>>    sdkchroot: remove 'copy-package-cache' from ROOTFS_FEATURES
>>>    bitbake: added DISTRO to REPO_BASE_{DB_}DIR
>>>    isar-bootstrap: fix DISTRO_APT_PREMIRRORS regex in case of cached
>>>      build
>>>    base-apt: fix issues with creating and using the base-apt repository
>>>    ci_build.sh: Remove rpi-stretch if CROSS_BUILD and REPRO_BUILD
>>>    ci_build.sh/repro: disable build of non-cached targets
>>>
>>>   meta/classes/image-cache-extension.bbclass          | 10 ++++++++++
>>>   meta/conf/bitbake.conf                              |  4 ++--
>>>   meta/recipes-core/isar-bootstrap/isar-bootstrap.inc |  3 ++-
>>>   meta/recipes-devtools/sdkchroot/sdkchroot.bb        |  2 +-
>>>   scripts/ci_build.sh                                 | 13 +++++++++++--
>>>   5 files changed, 26 insertions(+), 6 deletions(-)
>>>
>>
>>
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov