* [PATCH 1/6] sdkchroot: remove 'copy-package-cache' from ROOTFS_FEATURES
2019-05-23 11:39 [PATCH 0/6] Fixing base-apt and ci claudius.heine.ext
@ 2019-05-23 11:39 ` claudius.heine.ext
2019-05-23 11:39 ` [PATCH 2/6] bitbake: added DISTRO to REPO_BASE_{DB_}DIR claudius.heine.ext
` (5 subsequent siblings)
6 siblings, 0 replies; 10+ messages in thread
From: claudius.heine.ext @ 2019-05-23 11:39 UTC (permalink / raw)
To: isar-users; +Cc: Claudius Heine
From: Claudius Heine <ch@denx.de>
sdkchroot does not use the packages copied by the `copy-package-cache`
feature anywhere.
Signed-off-by: Claudius Heine <ch@denx.de>
---
meta/recipes-devtools/sdkchroot/sdkchroot.bb | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/recipes-devtools/sdkchroot/sdkchroot.bb b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
index 57d52ba..3cfe05c 100644
--- a/meta/recipes-devtools/sdkchroot/sdkchroot.bb
+++ b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
@@ -23,7 +23,7 @@ ROOTFS_ARCH = "${HOST_ARCH}"
ROOTFS_DISTRO = "${HOST_DISTRO}"
ROOTFSDIR = "${S}"
ROOTFS_PACKAGES = "${SDKCHROOT_PREINSTALL} ${TOOLCHAIN}"
-ROOTFS_FEATURES += "copy-package-cache"
+ROOTFS_FEATURES += ""
python() {
if d.getVar("HOST_ARCH") not in ['i386', 'amd64']:
--
2.20.1
^ permalink raw reply [flat|nested] 10+ messages in thread* [PATCH 2/6] bitbake: added DISTRO to REPO_BASE_{DB_}DIR
2019-05-23 11:39 [PATCH 0/6] Fixing base-apt and ci claudius.heine.ext
2019-05-23 11:39 ` [PATCH 1/6] sdkchroot: remove 'copy-package-cache' from ROOTFS_FEATURES claudius.heine.ext
@ 2019-05-23 11:39 ` claudius.heine.ext
2019-05-23 11:39 ` [PATCH 3/6] isar-bootstrap: fix DISTRO_APT_PREMIRRORS regex in case of cached build claudius.heine.ext
` (4 subsequent siblings)
6 siblings, 0 replies; 10+ messages in thread
From: claudius.heine.ext @ 2019-05-23 11:39 UTC (permalink / raw)
To: isar-users; +Cc: Claudius Heine
From: Claudius Heine <ch@denx.de>
The base-apt repository can only contain one codename, so the path
should be specific to the DISTRO variable to avoid conflicts.
Signed-off-by: Claudius Heine <ch@denx.de>
---
meta/conf/bitbake.conf | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 381c121..3d18223 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -69,8 +69,8 @@ REPO_ISAR_DB_DIR = "${DEPLOY_DIR}/isar-apt/db"
THIRD_PARTY_APT_KEYRING = "/etc/apt/trusted.gpg.d/third_party.gpg"
# Base apt repository paths
-REPO_BASE_DIR = "${DL_DIR}/base-apt/apt"
-REPO_BASE_DB_DIR = "${DL_DIR}/base-apt/db"
+REPO_BASE_DIR = "${DL_DIR}/base-apt/${DISTRO}/apt"
+REPO_BASE_DB_DIR = "${DL_DIR}/base-apt/${DISTRO}/db"
BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DIR \
SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \
--
2.20.1
^ permalink raw reply [flat|nested] 10+ messages in thread* [PATCH 3/6] isar-bootstrap: fix DISTRO_APT_PREMIRRORS regex in case of cached build
2019-05-23 11:39 [PATCH 0/6] Fixing base-apt and ci claudius.heine.ext
2019-05-23 11:39 ` [PATCH 1/6] sdkchroot: remove 'copy-package-cache' from ROOTFS_FEATURES claudius.heine.ext
2019-05-23 11:39 ` [PATCH 2/6] bitbake: added DISTRO to REPO_BASE_{DB_}DIR claudius.heine.ext
@ 2019-05-23 11:39 ` claudius.heine.ext
2019-05-23 11:39 ` [PATCH 4/6] base-apt: fix issues with creating and using the base-apt repository claudius.heine.ext
` (3 subsequent siblings)
6 siblings, 0 replies; 10+ messages in thread
From: claudius.heine.ext @ 2019-05-23 11:39 UTC (permalink / raw)
To: isar-users; +Cc: Claudius Heine
From: Claudius Heine <ch@denx.de>
The regex before did not replace the distribution name in the url. The
new one will replace the whole url with the REPO_BASE_DIR + BASE_DISTRO
path.
Signed-off-by: Claudius Heine <ch@denx.de>
---
meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index f8741c2..18e5e8f 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -30,7 +30,7 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = ",gnupg2"
DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = ",apt-transport-https,ca-certificates"
HOST_DISTRO_APT_SOURCES += "conf/distro/${HOST_DISTRO}.list"
-DISTRO_APT_PREMIRRORS ?= "${@ "http://ftp\.(\S+\.)?debian.org file:///${REPO_BASE_DIR} \n" if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) else "" }"
+DISTRO_APT_PREMIRRORS ?= "${@ "http://ftp\.(\S+\.)?debian.org/\S* file:///${REPO_BASE_DIR}/${BASE_DISTRO} \n" if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) else "" }"
inherit base-apt-helper
--
2.20.1
^ permalink raw reply [flat|nested] 10+ messages in thread* [PATCH 4/6] base-apt: fix issues with creating and using the base-apt repository
2019-05-23 11:39 [PATCH 0/6] Fixing base-apt and ci claudius.heine.ext
` (2 preceding siblings ...)
2019-05-23 11:39 ` [PATCH 3/6] isar-bootstrap: fix DISTRO_APT_PREMIRRORS regex in case of cached build claudius.heine.ext
@ 2019-05-23 11:39 ` claudius.heine.ext
2019-05-23 11:39 ` [PATCH 5/6] ci_build.sh: Remove rpi-stretch if CROSS_BUILD and REPRO_BUILD claudius.heine.ext
` (2 subsequent siblings)
6 siblings, 0 replies; 10+ messages in thread
From: claudius.heine.ext @ 2019-05-23 11:39 UTC (permalink / raw)
To: isar-users; +Cc: Claudius Heine
From: Claudius Heine <ch@denx.de>
Currently we only support one repository for one image, but since the
image creation uses two root file systems in case of cross-building,
package of both will land into the same repository. This should not be
an issue if both root file system use the same distribution + codename,
since shared ('all' arch) packages where shared in the upstream
repository as well. But in case those differ, different versions could
be used when building out of the base-apt later.
This fix is two fold, for once it disables the creation of base-apt
repositories if the distribution+codename of the two root file systems
differ and second it adds a 'apt-get install --fix-broken' call before
the 'dist-upgrade' one, in order to remove broken dependencies, that
where installed when such a base-apt repository was used.
Signed-off-by: Claudius Heine <ch@denx.de>
---
meta/classes/image-cache-extension.bbclass | 10 ++++++++++
meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 1 +
2 files changed, 11 insertions(+)
diff --git a/meta/classes/image-cache-extension.bbclass b/meta/classes/image-cache-extension.bbclass
index af17e0b..30db549 100644
--- a/meta/classes/image-cache-extension.bbclass
+++ b/meta/classes/image-cache-extension.bbclass
@@ -11,6 +11,16 @@ do_cache_base_repo[depends] = "base-apt:do_cache_config"
do_cache_base_repo[lockfiles] = "${REPO_BASE_DIR}/isar.lock"
do_cache_base_repo[stamp-extra-info] = "${DISTRO}-${MACHINE}"
do_cache_base_repo() {
+ if [ -d '${BUILDCHROOT_HOST_DIR}/var/cache/apt' ] &&
+ [ '${DISTRO}' != '${HOST_DISTRO}' ]; then
+ # We would need two separate repository paths for that.
+ # Otherwise packages (especially the 'all' arch ones) from one
+ # distribution can influence the package versions of the other
+ # distribution.
+ bbfatal "Different host and target distributions are currently not supported." \
+ "Try it without cross-build."
+ fi
+
if [ -d ${WORKDIR}/apt_cache ]; then
populate_base_apt ${WORKDIR}/apt_cache
fi
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 18e5e8f..2bc5ea5 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -333,6 +333,7 @@ isar_bootstrap() {
fi
chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y
+ chroot "${ROOTFSDIR}" /usr/bin/apt-get install -y -f
chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \
-o Debug::pkgProblemResolver=yes
--
2.20.1
^ permalink raw reply [flat|nested] 10+ messages in thread* [PATCH 5/6] ci_build.sh: Remove rpi-stretch if CROSS_BUILD and REPRO_BUILD
2019-05-23 11:39 [PATCH 0/6] Fixing base-apt and ci claudius.heine.ext
` (3 preceding siblings ...)
2019-05-23 11:39 ` [PATCH 4/6] base-apt: fix issues with creating and using the base-apt repository claudius.heine.ext
@ 2019-05-23 11:39 ` claudius.heine.ext
2019-05-23 11:39 ` [PATCH 6/6] ci_build.sh/repro: disable build of non-cached targets claudius.heine.ext
2019-05-23 11:59 ` [PATCH 0/6] Fixing base-apt and ci Maxim Yu. Osipov
6 siblings, 0 replies; 10+ messages in thread
From: claudius.heine.ext @ 2019-05-23 11:39 UTC (permalink / raw)
To: isar-users; +Cc: Claudius Heine
From: Claudius Heine <ch@denx.de>
Contains also a fix to the shebang. It needs to be on the first line and
since it already uses bashisms it was changed from sh to bash.
Signed-off-by: Claudius Heine <ch@denx.de>
---
scripts/ci_build.sh | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index a38e2b7..b8a504d 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -1,10 +1,9 @@
+#!/usr/bin/env bash
# Script for CI system build
#
# Alexander Smirnov <asmirnov@ilbers.de>
# Copyright (c) 2016-2018 ilbers GmbH
-#!/bin/sh
-
set -e
ES_BUG=3
@@ -114,6 +113,10 @@ if [ -n "$CROSS_BUILD" ]; then
fi
if [ -n "$REPRO_BUILD" ]; then
+ if [ -n "$CROSS_BUILD" ]; then
+ # remove rpi-stretch, since that is currently incompatible
+ TARGETS_SET="${TARGETS_SET//multiconfig:rpi-stretch:isar-image-base/}"
+ fi
# Enable use of cached base repository
bitbake $BB_ARGS -c cache_base_repo $TARGETS_SET
while [ -e bitbake.sock ]; do sleep 1; done
--
2.20.1
^ permalink raw reply [flat|nested] 10+ messages in thread* [PATCH 6/6] ci_build.sh/repro: disable build of non-cached targets
2019-05-23 11:39 [PATCH 0/6] Fixing base-apt and ci claudius.heine.ext
` (4 preceding siblings ...)
2019-05-23 11:39 ` [PATCH 5/6] ci_build.sh: Remove rpi-stretch if CROSS_BUILD and REPRO_BUILD claudius.heine.ext
@ 2019-05-23 11:39 ` claudius.heine.ext
2019-05-23 11:59 ` [PATCH 0/6] Fixing base-apt and ci Maxim Yu. Osipov
6 siblings, 0 replies; 10+ messages in thread
From: claudius.heine.ext @ 2019-05-23 11:39 UTC (permalink / raw)
To: isar-users; +Cc: Claudius Heine
From: Claudius Heine <ch@denx.de>
Signed-off-by: Claudius Heine <ch@denx.de>
---
scripts/ci_build.sh | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index b8a504d..3af00be 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -127,6 +127,12 @@ fi
# Start build for the defined set of configurations
bitbake $BB_ARGS $TARGETS_SET
+# Don't build anything else with 'ISAR_USE_CACHED_BASE_REPO = "1"', since
+# that is not cached:
+if [ -n "$REPRO_BUILD" ]; then
+ exit 0
+fi
+
# Note: de0-nano-soc build is launched separately as
# parallel build with the same target arch (armhf) fails.
# The problem is being investigated
--
2.20.1
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [PATCH 0/6] Fixing base-apt and ci
2019-05-23 11:39 [PATCH 0/6] Fixing base-apt and ci claudius.heine.ext
` (5 preceding siblings ...)
2019-05-23 11:39 ` [PATCH 6/6] ci_build.sh/repro: disable build of non-cached targets claudius.heine.ext
@ 2019-05-23 11:59 ` Maxim Yu. Osipov
2019-05-23 12:21 ` Claudius Heine
6 siblings, 1 reply; 10+ messages in thread
From: Maxim Yu. Osipov @ 2019-05-23 11:59 UTC (permalink / raw)
To: claudius.heine.ext, isar-users; +Cc: Claudius Heine
Hi Claudius,
It seems that you need to rebase first against the current 'next'.
Definitely, this relates to ci_build.
Regards,
Maxim.
On 5/23/19 1:39 PM, claudius.heine.ext@siemens.com wrote:
> From: Claudius Heine <ch@denx.de>
>
> Hi,
>
> this patchset depends on the "pre-processing pipeline and transient package
> replacement" patchset and fixes some issues related to the base-apt
> implementation and CI test.
>
> CI test:
> http://isar-build.org:8080/job/isar_claudius_ilbers_fast_ci/33/console
> (This run was probably triggered with 'ci_build.sh -f -r')
>
> regards,
> Claudius
>
> Claudius Heine (6):
> sdkchroot: remove 'copy-package-cache' from ROOTFS_FEATURES
> bitbake: added DISTRO to REPO_BASE_{DB_}DIR
> isar-bootstrap: fix DISTRO_APT_PREMIRRORS regex in case of cached
> build
> base-apt: fix issues with creating and using the base-apt repository
> ci_build.sh: Remove rpi-stretch if CROSS_BUILD and REPRO_BUILD
> ci_build.sh/repro: disable build of non-cached targets
>
> meta/classes/image-cache-extension.bbclass | 10 ++++++++++
> meta/conf/bitbake.conf | 4 ++--
> meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 3 ++-
> meta/recipes-devtools/sdkchroot/sdkchroot.bb | 2 +-
> scripts/ci_build.sh | 13 +++++++++++--
> 5 files changed, 26 insertions(+), 6 deletions(-)
>
--
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [PATCH 0/6] Fixing base-apt and ci
2019-05-23 11:59 ` [PATCH 0/6] Fixing base-apt and ci Maxim Yu. Osipov
@ 2019-05-23 12:21 ` Claudius Heine
2019-05-23 12:26 ` Maxim Yu. Osipov
0 siblings, 1 reply; 10+ messages in thread
From: Claudius Heine @ 2019-05-23 12:21 UTC (permalink / raw)
To: Maxim Yu. Osipov, isar-users; +Cc: Claudius Heine
Hi Maxim,
On 23/05/2019 13.59, Maxim Yu. Osipov wrote:
> Hi Claudius,
>
> It seems that you need to rebase first against the current 'next'.
> Definitely, this relates to ci_build.
Ok, if I understand your changes to ci_build, then those two patches
might no longer be needed, since the they should have resolved those issues.
But I will do a CI run to be sure.
thx,
Claudius
>
> Regards,
> Maxim.
>
> On 5/23/19 1:39 PM, claudius.heine.ext@siemens.com wrote:
>> From: Claudius Heine <ch@denx.de>
>>
>> Hi,
>>
>> this patchset depends on the "pre-processing pipeline and transient
>> package
>> replacement" patchset and fixes some issues related to the base-apt
>> implementation and CI test.
>>
>> CI test:
>> http://isar-build.org:8080/job/isar_claudius_ilbers_fast_ci/33/console
>> (This run was probably triggered with 'ci_build.sh -f -r')
>>
>> regards,
>> Claudius
>>
>> Claudius Heine (6):
>> sdkchroot: remove 'copy-package-cache' from ROOTFS_FEATURES
>> bitbake: added DISTRO to REPO_BASE_{DB_}DIR
>> isar-bootstrap: fix DISTRO_APT_PREMIRRORS regex in case of cached
>> build
>> base-apt: fix issues with creating and using the base-apt repository
>> ci_build.sh: Remove rpi-stretch if CROSS_BUILD and REPRO_BUILD
>> ci_build.sh/repro: disable build of non-cached targets
>>
>> meta/classes/image-cache-extension.bbclass | 10 ++++++++++
>> meta/conf/bitbake.conf | 4 ++--
>> meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 3 ++-
>> meta/recipes-devtools/sdkchroot/sdkchroot.bb | 2 +-
>> scripts/ci_build.sh | 13 +++++++++++--
>> 5 files changed, 26 insertions(+), 6 deletions(-)
>>
>
>
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [PATCH 0/6] Fixing base-apt and ci
2019-05-23 12:21 ` Claudius Heine
@ 2019-05-23 12:26 ` Maxim Yu. Osipov
0 siblings, 0 replies; 10+ messages in thread
From: Maxim Yu. Osipov @ 2019-05-23 12:26 UTC (permalink / raw)
To: Claudius Heine, isar-users; +Cc: Claudius Heine
On 5/23/19 2:21 PM, Claudius Heine wrote:
> Hi Maxim,
>
> On 23/05/2019 13.59, Maxim Yu. Osipov wrote:
>> Hi Claudius,
>>
>> It seems that you need to rebase first against the current 'next'.
>> Definitely, this relates to ci_build.
>
> Ok, if I understand your changes to ci_build, then those two patches
> might no longer be needed, since the they should have resolved those
> issues.
shebang fix is OK.
Thanks,
Maxim.
> But I will do a CI run to be sure.
>
> thx,
> Claudius
>
>>
>> Regards,
>> Maxim.
>>
>> On 5/23/19 1:39 PM, claudius.heine.ext@siemens.com wrote:
>>> From: Claudius Heine <ch@denx.de>
>>>
>>> Hi,
>>>
>>> this patchset depends on the "pre-processing pipeline and transient
>>> package
>>> replacement" patchset and fixes some issues related to the base-apt
>>> implementation and CI test.
>>>
>>> CI test:
>>> http://isar-build.org:8080/job/isar_claudius_ilbers_fast_ci/33/console
>>> (This run was probably triggered with 'ci_build.sh -f -r')
>>>
>>> regards,
>>> Claudius
>>>
>>> Claudius Heine (6):
>>> sdkchroot: remove 'copy-package-cache' from ROOTFS_FEATURES
>>> bitbake: added DISTRO to REPO_BASE_{DB_}DIR
>>> isar-bootstrap: fix DISTRO_APT_PREMIRRORS regex in case of cached
>>> build
>>> base-apt: fix issues with creating and using the base-apt repository
>>> ci_build.sh: Remove rpi-stretch if CROSS_BUILD and REPRO_BUILD
>>> ci_build.sh/repro: disable build of non-cached targets
>>>
>>> meta/classes/image-cache-extension.bbclass | 10 ++++++++++
>>> meta/conf/bitbake.conf | 4 ++--
>>> meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 3 ++-
>>> meta/recipes-devtools/sdkchroot/sdkchroot.bb | 2 +-
>>> scripts/ci_build.sh | 13 +++++++++++--
>>> 5 files changed, 26 insertions(+), 6 deletions(-)
>>>
>>
>>
>
--
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov
^ permalink raw reply [flat|nested] 10+ messages in thread