[PATCHv4 0/9] apt-get source support

[PATCHv4 0/9] apt-get source support

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

diff v3 .. v4:
 - add patch to skip cached ci for hello because of known issue
 - fixed copyright headers
 - changed MAINTAINER of hello to isar-users

v2 was sent by Maxim so i gave it a 3

diff to v1 .. v3:
 - add "apt://" protocol to SRC_URI p7, not squashed
 - add docs
 - change the way i mention Cedric as co-author
 - disable cross-build testing in hello


*** BLURB HERE ***

Henning Schild (9):
  conf: add deb-src entries to all our distro configs
  dpkg-base: introduce an "apt-get source" fetch/unpack step
  meta: move debianization code into a class and into dpkg-base
  debianize: allow changlog version change
  meta-isar/recipes-app: add upstream hello rebuild example
  local.conf: enable rebuilding "hello" for all distros
  meta: change apt source fetcher to hook into SRC_URI
  doc: document "apt://" fetch/unpack feature
  ci: remove "hello" from the offline build, it fails

 doc/user_manual.md                            |  4 +
 .../conf/distro/debian-stretch-backports.list |  1 +
 meta-isar/conf/distro/raspbian-jessie.list    |  1 +
 meta-isar/conf/local.conf.sample              |  2 +-
 meta-isar/recipes-app/hello/hello.inc         | 26 ++++++
 meta-isar/recipes-app/hello/hello_2.10.bb     | 11 +++
 meta-isar/recipes-app/hello/hello_2.9.bb      | 11 +++
 meta/classes/base.bbclass                     | 19 ++++
 meta/classes/debianize.bbclass                | 89 +++++++++++++++++++
 meta/classes/dpkg-base.bbclass                | 20 +++++
 meta/classes/dpkg-raw.bbclass                 | 87 +-----------------
 meta/conf/distro/debian-buster.list           |  3 +
 meta/conf/distro/debian-jessie.list           |  3 +
 meta/conf/distro/debian-stretch.list          |  3 +
 scripts/ci_build.sh                           |  2 +
 15 files changed, 197 insertions(+), 85 deletions(-)
 create mode 100644 meta-isar/recipes-app/hello/hello.inc
 create mode 100644 meta-isar/recipes-app/hello/hello_2.10.bb
 create mode 100644 meta-isar/recipes-app/hello/hello_2.9.bb
 create mode 100644 meta/classes/debianize.bbclass

-- 
2.19.2

[PATCHv4 1/9] conf: add deb-src entries to all our distro configs

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

That will allow us to use "apt-get source" to fetch debian sources from
the mirrors we also get the binary packages from.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta-isar/conf/distro/debian-stretch-backports.list | 1 +
 meta-isar/conf/distro/raspbian-jessie.list          | 1 +
 meta/conf/distro/debian-buster.list                 | 3 +++
 meta/conf/distro/debian-jessie.list                 | 3 +++
 meta/conf/distro/debian-stretch.list                | 3 +++
 5 files changed, 11 insertions(+)

diff --git a/meta-isar/conf/distro/debian-stretch-backports.list b/meta-isar/conf/distro/debian-stretch-backports.list
index 727a835..f606be5 100644
--- a/meta-isar/conf/distro/debian-stretch-backports.list
+++ b/meta-isar/conf/distro/debian-stretch-backports.list
@@ -1 +1,2 @@
 deb	http://ftp.de.debian.org/debian stretch-backports main
+deb-src	http://ftp.de.debian.org/debian stretch-backports main
diff --git a/meta-isar/conf/distro/raspbian-jessie.list b/meta-isar/conf/distro/raspbian-jessie.list
index 792f9c6..0a5af41 100644
--- a/meta-isar/conf/distro/raspbian-jessie.list
+++ b/meta-isar/conf/distro/raspbian-jessie.list
@@ -1 +1,2 @@
 deb	http://archive.raspbian.org/raspbian	jessie	main contrib non-free firmware
+deb-src	http://archive.raspbian.org/raspbian	jessie	main contrib non-free firmware
diff --git a/meta/conf/distro/debian-buster.list b/meta/conf/distro/debian-buster.list
index 18311d8..1e38f1a 100644
--- a/meta/conf/distro/debian-buster.list
+++ b/meta/conf/distro/debian-buster.list
@@ -1,3 +1,6 @@
 deb	http://ftp.de.debian.org/debian buster	main contrib non-free
+deb-src	http://ftp.de.debian.org/debian buster	main contrib non-free
 deb	http://ftp.de.debian.org/debian buster-updates	main contrib non-free
+deb-src	http://ftp.de.debian.org/debian buster-updates	main contrib non-free
 deb	http://security.debian.org	buster/updates	main contrib non-free
+deb-src	http://security.debian.org	buster/updates	main contrib non-free
diff --git a/meta/conf/distro/debian-jessie.list b/meta/conf/distro/debian-jessie.list
index be46a57..5143f4d 100644
--- a/meta/conf/distro/debian-jessie.list
+++ b/meta/conf/distro/debian-jessie.list
@@ -1,3 +1,6 @@
 deb	http://ftp.de.debian.org/debian	jessie	main contrib non-free
+deb-src	http://ftp.de.debian.org/debian	jessie	main contrib non-free
 deb	http://ftp.de.debian.org/debian jessie-updates	main contrib non-free
+deb-src	http://ftp.de.debian.org/debian jessie-updates	main contrib non-free
 deb	http://security.debian.org	jessie/updates	main contrib non-free
+deb-src	http://security.debian.org	jessie/updates	main contrib non-free
diff --git a/meta/conf/distro/debian-stretch.list b/meta/conf/distro/debian-stretch.list
index bfa5cf9..d4f3de0 100644
--- a/meta/conf/distro/debian-stretch.list
+++ b/meta/conf/distro/debian-stretch.list
@@ -1,3 +1,6 @@
 deb	http://ftp.de.debian.org/debian stretch	main contrib non-free
+deb-src	http://ftp.de.debian.org/debian stretch	main contrib non-free
 deb	http://ftp.de.debian.org/debian stretch-updates	main contrib non-free
+deb-src	http://ftp.de.debian.org/debian stretch-updates	main contrib non-free
 deb	http://security.debian.org	stretch/updates	main contrib non-free
+deb-src	http://security.debian.org	stretch/updates	main contrib non-free
-- 
2.19.2

[PATCHv4 2/9] dpkg-base: introduce an "apt-get source" fetch/unpack step

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

This creates a new task to fetch/unpack debian sources from debian
source mirrors. This is done using "apt-get source" and the new variable
APT_SRC to control the arguments.

An example where a original debian package gets rebuild will follow in a
later commit.

based on original patch by Cedric Hombourger <Cedric_Hombourger@mentor.com>

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta/classes/dpkg-base.bbclass | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index f1b127c..2f324f9 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -20,6 +20,25 @@ do_adjust_git[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 inherit patch
 addtask patch after do_adjust_git before do_build
 
+SRC_APT ?= ""
+
+do_apt_fetch[depends] = "buildchroot-target:do_build"
+
+do_apt_fetch() {
+	if [ -z "${@d.getVar("SRC_APT", True).strip()}" ]; then
+		exit
+	fi
+	dpkg_do_mounts
+	E="${@ bb.utils.export_proxies(d)}"
+	sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
+		sh -c 'cd ${PP} && apt-get -y source ${SRC_APT}'
+	dpkg_undo_mounts
+}
+
+addtask apt_fetch after do_unpack before do_patch
+do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
+do_apt_fetch[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
+
 def get_package_srcdir(d):
     s = d.getVar("S", True)
     workdir = d.getVar("WORKDIR", True)
-- 
2.19.2

[PATCHv4 3/9] meta: move debianization code into a class and into dpkg-base

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

This is pure restructuring. Move the code into its own class and include
that in dpkg-base, it was available only in dpkg-raw before.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta/classes/debianize.bbclass | 87 ++++++++++++++++++++++++++++++++++
 meta/classes/dpkg-base.bbclass |  1 +
 meta/classes/dpkg-raw.bbclass  | 87 ++--------------------------------
 3 files changed, 91 insertions(+), 84 deletions(-)
 create mode 100644 meta/classes/debianize.bbclass

diff --git a/meta/classes/debianize.bbclass b/meta/classes/debianize.bbclass
new file mode 100644
index 0000000..c1a2f18
--- /dev/null
+++ b/meta/classes/debianize.bbclass
@@ -0,0 +1,87 @@
+# This software is a part of ISAR.
+# Copyright (C) 2017-2019 Siemens AG
+#
+# SPDX-License-Identifier: MIT
+
+deb_add_changelog() {
+	date=$( LANG=C date -R )
+	cat <<EOF > ${S}/debian/changelog
+${PN} (${PV}) UNRELEASED; urgency=low
+
+  * generated by Isar
+
+ -- ${MAINTAINER}  ${date}
+EOF
+	if [ -f ${WORKDIR}/changelog ]; then
+		echo >> ${S}/debian/changelog
+		cat ${WORKDIR}/changelog >> ${S}/debian/changelog
+	fi
+}
+
+deb_create_compat() {
+	echo 9 > ${S}/debian/compat
+}
+
+deb_create_control() {
+	compat=$( cat ${S}/debian/compat )
+	cat << EOF > ${S}/debian/control
+Source: ${PN}
+Section: misc
+Priority: optional
+Standards-Version: 3.9.6
+Maintainer: ${MAINTAINER}
+Build-Depends: debhelper (>= ${compat})
+
+Package: ${PN}
+Architecture: any
+Depends: ${DEBIAN_DEPENDS}
+Description: ${DESCRIPTION}
+EOF
+}
+
+DH_FIXPERM_EXCLUSIONS = \
+    "${@' '.join(['-X ' + x for x in \
+                  (d.getVar('PRESERVE_PERMS', False) or '').split()])}"
+
+deb_create_rules() {
+	cat << EOF > ${S}/debian/rules
+#!/usr/bin/make -f
+
+override_dh_fixperms:
+	dh_fixperms ${DH_FIXPERM_EXCLUSIONS}
+
+%:
+	dh \$@
+EOF
+	chmod +x ${S}/debian/rules
+}
+
+deb_debianize() {
+	if [ -f ${WORKDIR}/compat ]; then
+		install -v -m 644 ${WORKDIR}/compat ${S}/debian/compat
+	else
+		deb_create_compat
+	fi
+	if [ -f ${WORKDIR}/control ]; then
+		install -v -m 644 ${WORKDIR}/control ${S}/debian/control
+	else
+		deb_create_control
+	fi
+	if [ -f ${WORKDIR}/rules ]; then
+		install -v -m 755 ${WORKDIR}/rules ${S}/debian/rules
+	else
+		deb_create_rules
+	fi
+	deb_add_changelog
+
+	for t in pre post
+	do
+		for a in inst rm
+		do
+			if [ -f ${WORKDIR}/${t}${a} ]; then
+				install -v -m 755 ${WORKDIR}/${t}${a} \
+					${S}/debian/${t}${a}
+			fi
+		done
+	done
+}
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 2f324f9..175dc80 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -4,6 +4,7 @@
 # SPDX-License-Identifier: MIT
 
 inherit buildchroot
+inherit debianize
 
 DEPENDS ?= ""
 
diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
index a9143e9..ea03ea4 100644
--- a/meta/classes/dpkg-raw.bbclass
+++ b/meta/classes/dpkg-raw.bbclass
@@ -1,5 +1,7 @@
 # This software is a part of ISAR.
-# Copyright (C) 2017-2018 Siemens AG
+# Copyright (C) 2017-2019 Siemens AG
+#
+# SPDX-License-Identifier: MIT
 
 inherit dpkg
 
@@ -17,89 +19,6 @@ do_install[cleandirs] = "${D}"
 do_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 addtask install after do_unpack before do_prepare_build
 
-deb_add_changelog() {
-	date=$( LANG=C date -R )
-	cat <<EOF > ${D}/debian/changelog
-${PN} (${PV}) UNRELEASED; urgency=low
-
-  * generated by Isar
-
- -- ${MAINTAINER}  ${date}
-EOF
-	if [ -f ${WORKDIR}/changelog ]; then
-		echo >> ${D}/debian/changelog
-		cat ${WORKDIR}/changelog >> ${D}/debian/changelog
-	fi
-}
-
-deb_create_compat() {
-	echo 9 > ${D}/debian/compat
-}
-
-deb_create_control() {
-	compat=$( cat ${D}/debian/compat )
-	cat << EOF > ${D}/debian/control
-Source: ${PN}
-Section: misc
-Priority: optional
-Standards-Version: 3.9.6
-Maintainer: ${MAINTAINER}
-Build-Depends: debhelper (>= ${compat})
-
-Package: ${PN}
-Architecture: any
-Depends: ${DEBIAN_DEPENDS}
-Description: ${DESCRIPTION}
-EOF
-}
-
-DH_FIXPERM_EXCLUSIONS = \
-    "${@' '.join(['-X ' + x for x in \
-                  (d.getVar('PRESERVE_PERMS', False) or '').split()])}"
-
-deb_create_rules() {
-	cat << EOF > ${S}/debian/rules
-#!/usr/bin/make -f
-
-override_dh_fixperms:
-	dh_fixperms ${DH_FIXPERM_EXCLUSIONS}
-
-%:
-	dh \$@
-EOF
-	chmod +x ${S}/debian/rules
-}
-
-deb_debianize() {
-	if [ -f ${WORKDIR}/compat ]; then
-		install -v -m 644 ${WORKDIR}/compat ${D}/debian/compat
-	else
-		deb_create_compat
-	fi
-	if [ -f ${WORKDIR}/control ]; then
-		install -v -m 644 ${WORKDIR}/control ${D}/debian/control
-	else
-		deb_create_control
-	fi
-	if [ -f ${WORKDIR}/rules ]; then
-		install -v -m 755 ${WORKDIR}/rules ${D}/debian/rules
-	else
-		deb_create_rules
-	fi
-	deb_add_changelog
-
-	for t in pre post
-	do
-		for a in inst rm
-		do
-			if [ -f ${WORKDIR}/${t}${a} ]; then
-				install -v -m 755 ${WORKDIR}/${t}${a} \
-					${D}/debian/${t}${a}
-			fi
-		done
-	done
-}
-
 do_prepare_build[cleandirs] += "${D}/debian"
 do_prepare_build() {
 	cd ${D}
-- 
2.19.2

[PATCHv4 4/9] debianize: allow changlog version change

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

Introduce a variable to control the version string we put into a
generated changelog entry.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta/classes/debianize.bbclass | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/meta/classes/debianize.bbclass b/meta/classes/debianize.bbclass
index c1a2f18..510aebd 100644
--- a/meta/classes/debianize.bbclass
+++ b/meta/classes/debianize.bbclass
@@ -3,10 +3,12 @@
 #
 # SPDX-License-Identifier: MIT
 
+CHANGELOG_V ?= "${PV}"
+
 deb_add_changelog() {
 	date=$( LANG=C date -R )
 	cat <<EOF > ${S}/debian/changelog
-${PN} (${PV}) UNRELEASED; urgency=low
+${PN} (${CHANGELOG_V}) UNRELEASED; urgency=low
 
   * generated by Isar
 
-- 
2.19.2

[PATCHv4 5/9] meta-isar/recipes-app: add upstream hello rebuild example

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

This commit introduces a rebuild example of an upstream package. It also
shows how to deal with multiple versions of such a package, depending on
your distro. And in fact we actually have to patch it to rebuild it,
because its build-deps are wrong.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta-isar/recipes-app/hello/hello.inc     | 26 +++++++++++++++++++++++
 meta-isar/recipes-app/hello/hello_2.10.bb | 11 ++++++++++
 meta-isar/recipes-app/hello/hello_2.9.bb  | 11 ++++++++++
 3 files changed, 48 insertions(+)
 create mode 100644 meta-isar/recipes-app/hello/hello.inc
 create mode 100644 meta-isar/recipes-app/hello/hello_2.10.bb
 create mode 100644 meta-isar/recipes-app/hello/hello_2.9.bb

diff --git a/meta-isar/recipes-app/hello/hello.inc b/meta-isar/recipes-app/hello/hello.inc
new file mode 100644
index 0000000..815211b
--- /dev/null
+++ b/meta-isar/recipes-app/hello/hello.inc
@@ -0,0 +1,26 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2019
+#
+# SPDX-License-Identifier: MIT
+
+inherit dpkg
+
+# this will fetch and unpack the sources from upstream debian
+SRC_APT = "${PN}"
+
+MAINTAINER = "isar-users <isar-users@googlegroups.com>"
+CHANGELOG_V = "${PV}-99+isar"
+
+do_prepare_build() {
+	deb_add_changelog
+	# this seems to be a build dep missing in the upstream control file
+	if ! grep texinfo ${S}/debian/control; then
+		sed -i -e 's/Build-Depends:/Build-Depends: texinfo,/g' ${S}/debian/control
+	fi
+}
+
+dpkg_runbuild_prepend() {
+	if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
+		export DEB_BUILD_OPTIONS="nocheck"
+	fi
+}
diff --git a/meta-isar/recipes-app/hello/hello_2.10.bb b/meta-isar/recipes-app/hello/hello_2.10.bb
new file mode 100644
index 0000000..bfb8722
--- /dev/null
+++ b/meta-isar/recipes-app/hello/hello_2.10.bb
@@ -0,0 +1,11 @@
+# Example recipe to rebuild a debian source package
+#
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2019
+#
+# SPDX-License-Identifier: MIT
+
+require hello.inc
+
+DEFAULT_PREFERENCE_debian-buster = "1"
+DEFAULT_PREFERENCE_debian-stretch = "1"
diff --git a/meta-isar/recipes-app/hello/hello_2.9.bb b/meta-isar/recipes-app/hello/hello_2.9.bb
new file mode 100644
index 0000000..63d1562
--- /dev/null
+++ b/meta-isar/recipes-app/hello/hello_2.9.bb
@@ -0,0 +1,11 @@
+# Example recipe to rebuild a debian source package
+#
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2019
+#
+# SPDX-License-Identifier: MIT
+
+require hello.inc
+
+DEFAULT_PREFERENCE_debian-jessie = "1"
+DEFAULT_PREFERENCE_raspbian-jessie = "1"
-- 
2.19.2

[PATCHv4 6/9] local.conf: enable rebuilding "hello" for all distros

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

This commit includes the rebuild example in our default builds and
therefore into CI.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta-isar/conf/local.conf.sample | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index a671b20..919bb63 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -161,7 +161,7 @@ CONF_VERSION = "1"
 
 #
 # The default list of extra packages to be installed.
-IMAGE_INSTALL = "hello-isar example-raw example-module-${KERNEL_NAME} enable-fsck"
+IMAGE_INSTALL = "hello-isar example-raw example-module-${KERNEL_NAME} enable-fsck hello"
 
 #
 # Enable cross-compilation support
-- 
2.19.2

[PATCHv4 7/9] meta: change apt source fetcher to hook into SRC_URI

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

This patch allows users to use "apt://" URIs in SRC_URI and cleans up
the interface for recipes.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta-isar/recipes-app/hello/hello.inc |  2 +-
 meta/classes/base.bbclass             | 19 +++++++++++++++++++
 2 files changed, 20 insertions(+), 1 deletion(-)

diff --git a/meta-isar/recipes-app/hello/hello.inc b/meta-isar/recipes-app/hello/hello.inc
index 815211b..747b96b 100644
--- a/meta-isar/recipes-app/hello/hello.inc
+++ b/meta-isar/recipes-app/hello/hello.inc
@@ -6,7 +6,7 @@
 inherit dpkg
 
 # this will fetch and unpack the sources from upstream debian
-SRC_APT = "${PN}"
+SRC_URI = "apt://${PN}=${PV}"
 
 MAINTAINER = "isar-users <isar-users@googlegroups.com>"
 CHANGELOG_V = "${PV}-99+isar"
diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index d4082de..6ca93bf 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -96,6 +96,25 @@ python() {
                                   + ws + "root_cleandirs\n")
 }
 
+# filter out all "apt://" URIs out of SRC_URI and stick them into SRC_APT
+python() {
+    src_uri = (d.getVar('SRC_URI', True) or "").split()
+    if len(src_uri) == 0:
+        return
+
+    prefix = "apt://"
+    new_src_uri = []
+    src_apt = []
+    for u in src_uri:
+        if u.startswith(prefix):
+            src_apt.append(u[len(prefix):])
+        else:
+            new_src_uri.append(u)
+
+    d.setVar('SRC_URI', ' '.join(new_src_uri))
+    d.prependVar('SRC_APT', ' '.join(src_apt))
+}
+
 do_fetch[dirs] = "${DL_DIR}"
 do_fetch[file-checksums] = "${@bb.fetch.get_checksum_file_list(d)}"
 do_fetch[vardeps] += "SRCREV"
-- 
2.19.2

[PATCHv4 8/9] doc: document "apt://" fetch/unpack feature

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

Talk about its existance, point to the example, and mention the current
caching limitation.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 doc/user_manual.md | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index eebcaa9..ebc31c6 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -540,6 +540,9 @@ The `deb` packages are built using `dpkg-buildpackage`, so the sources should co
 
 **NOTE:** If the sources do not contain a `debian` directory your recipe can fetch, create, or ship that.
 
+This is also what you do if you want to rebuild/modify an upstream package.
+Isar does understand `SRC_URI` entries starting with "apt://". For an example
+of a customized upstream package have a look at `meta-isar/recipes-app/hello`.
 
 #### Example
 ```
@@ -753,3 +756,4 @@ bitbake multiconfig:qemuarm-stretch:isar-image-base
 ### Limitation
 
 So far the local base-apt repo is not gpg signed.
+Files fetched with the `SRC_URI` protocol "apt://" are not yet cached.
-- 
2.19.2

[PATCHv4 9/9] ci: remove "hello" from the offline build, it fails

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

The recipe uses "apt://" fetching which is not yet supported in the
cache. This patch should be reverted once we cache sources as well.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 scripts/ci_build.sh | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index f3523e8..63e3d21 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -97,6 +97,8 @@ if [ -n "$FAST_BUILD" ]; then
     while [ -e bitbake.sock ]; do sleep 1; done
     sudo rm -rf tmp
     sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?= "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
+    # this packages uses a feature that does not yet work in cached mode
+    sed -i -e 's/ hello//g' conf/local.conf
     bitbake $BB_ARGS \
         multiconfig:qemuarm-stretch:isar-image-base \
         multiconfig:qemuarm64-stretch:isar-image-base \
-- 
2.19.2

Re: [PATCHv4 0/9] apt-get source support

[Henning Schild]

For those who can access it, currently running here:

http://isar-build.org:8080/job/isar_henning_ilbers-ci/73/

Henning

On Thu, 31 Jan 2019 17:42:35 +0100
Henning Schild <henning.schild@siemens.com> wrote:

> From: Henning Schild <henning.schild@siemens.com>
> 
> diff v3 .. v4:
>  - add patch to skip cached ci for hello because of known issue
>  - fixed copyright headers
>  - changed MAINTAINER of hello to isar-users
> 
> v2 was sent by Maxim so i gave it a 3
> 
> diff to v1 .. v3:
>  - add "apt://" protocol to SRC_URI p7, not squashed
>  - add docs
>  - change the way i mention Cedric as co-author
>  - disable cross-build testing in hello
> 
> 
> *** BLURB HERE ***
> 
> Henning Schild (9):
>   conf: add deb-src entries to all our distro configs
>   dpkg-base: introduce an "apt-get source" fetch/unpack step
>   meta: move debianization code into a class and into dpkg-base
>   debianize: allow changlog version change
>   meta-isar/recipes-app: add upstream hello rebuild example
>   local.conf: enable rebuilding "hello" for all distros
>   meta: change apt source fetcher to hook into SRC_URI
>   doc: document "apt://" fetch/unpack feature
>   ci: remove "hello" from the offline build, it fails
> 
>  doc/user_manual.md                            |  4 +
>  .../conf/distro/debian-stretch-backports.list |  1 +
>  meta-isar/conf/distro/raspbian-jessie.list    |  1 +
>  meta-isar/conf/local.conf.sample              |  2 +-
>  meta-isar/recipes-app/hello/hello.inc         | 26 ++++++
>  meta-isar/recipes-app/hello/hello_2.10.bb     | 11 +++
>  meta-isar/recipes-app/hello/hello_2.9.bb      | 11 +++
>  meta/classes/base.bbclass                     | 19 ++++
>  meta/classes/debianize.bbclass                | 89
> +++++++++++++++++++ meta/classes/dpkg-base.bbclass                |
> 20 +++++ meta/classes/dpkg-raw.bbclass                 | 87
> +----------------- meta/conf/distro/debian-buster.list           |  3
> + meta/conf/distro/debian-jessie.list           |  3 +
>  meta/conf/distro/debian-stretch.list          |  3 +
>  scripts/ci_build.sh                           |  2 +
>  15 files changed, 197 insertions(+), 85 deletions(-)
>  create mode 100644 meta-isar/recipes-app/hello/hello.inc
>  create mode 100644 meta-isar/recipes-app/hello/hello_2.10.bb
>  create mode 100644 meta-isar/recipes-app/hello/hello_2.9.bb
>  create mode 100644 meta/classes/debianize.bbclass
> 

Re: [PATCHv4 9/9] ci: remove "hello" from the offline build, it fails

[Maxim Yu. Osipov]

Hi Henning,

I'm OK with your approach/series except this patch as
this patch just hides the problem from CI build and
will immediately pop up in any own user's offline build.

So I suggest to drop from the series patch "local.conf: enable 
rebuilding "hello" for all distros" until
limitation with offline builds will gone.

Thanks,
Maxim.

On 1/31/19 5:42 PM, Henning Schild wrote:
> From: Henning Schild <henning.schild@siemens.com>
> 
> The recipe uses "apt://" fetching which is not yet supported in the
> cache. This patch should be reverted once we cache sources as well.
> 
> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> ---
>   scripts/ci_build.sh | 2 ++
>   1 file changed, 2 insertions(+)
> 
> diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
> index f3523e8..63e3d21 100755
> --- a/scripts/ci_build.sh
> +++ b/scripts/ci_build.sh
> @@ -97,6 +97,8 @@ if [ -n "$FAST_BUILD" ]; then
>       while [ -e bitbake.sock ]; do sleep 1; done
>       sudo rm -rf tmp
>       sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?= "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
> +    # this packages uses a feature that does not yet work in cached mode
> +    sed -i -e 's/ hello//g' conf/local.conf
>       bitbake $BB_ARGS \
>           multiconfig:qemuarm-stretch:isar-image-base \
>           multiconfig:qemuarm64-stretch:isar-image-base \
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: [PATCHv4 9/9] ci: remove "hello" from the offline build, it fails

[Jan Kiszka]

On 04.02.19 18:33, Maxim Yu. Osipov wrote:
> Hi Henning,
> 
> I'm OK with your approach/series except this patch as
> this patch just hides the problem from CI build and
> will immediately pop up in any own user's offline build.
> 
> So I suggest to drop from the series patch "local.conf: enable rebuilding 
> "hello" for all distros" until
> limitation with offline builds will gone.

Which also raises the questions: Who has a proposal how to fix that issue and 
who can work on it?

We definitely need reproducible package rebuild.

Jan

> 
> Thanks,
> Maxim.
> 
> On 1/31/19 5:42 PM, Henning Schild wrote:
>> From: Henning Schild <henning.schild@siemens.com>
>>
>> The recipe uses "apt://" fetching which is not yet supported in the
>> cache. This patch should be reverted once we cache sources as well.
>>
>> Signed-off-by: Henning Schild <henning.schild@siemens.com>
>> ---
>>   scripts/ci_build.sh | 2 ++
>>   1 file changed, 2 insertions(+)
>>
>> diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
>> index f3523e8..63e3d21 100755
>> --- a/scripts/ci_build.sh
>> +++ b/scripts/ci_build.sh
>> @@ -97,6 +97,8 @@ if [ -n "$FAST_BUILD" ]; then
>>       while [ -e bitbake.sock ]; do sleep 1; done
>>       sudo rm -rf tmp
>>       sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?= "1"/ISAR_USE_CACHED_BASE_REPO 
>> ?= "1"/g' conf/local.conf
>> +    # this packages uses a feature that does not yet work in cached mode
>> +    sed -i -e 's/ hello//g' conf/local.conf
>>       bitbake $BB_ARGS \
>>           multiconfig:qemuarm-stretch:isar-image-base \
>>           multiconfig:qemuarm64-stretch:isar-image-base \
>>
> 
> 

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCHv4 9/9] ci: remove "hello" from the offline build, it fails

[Maxim Yu. Osipov]

On 2/4/19 6:37 PM, Jan Kiszka wrote:
> On 04.02.19 18:33, Maxim Yu. Osipov wrote:
>> Hi Henning,
>>
>> I'm OK with your approach/series except this patch as
>> this patch just hides the problem from CI build and
>> will immediately pop up in any own user's offline build.
>>
>> So I suggest to drop from the series patch "local.conf: enable 
>> rebuilding "hello" for all distros" until
>> limitation with offline builds will gone.
> 
> Which also raises the questions: Who has a proposal how to fix that 
> issue and who can work on it?

I'm looking at the problem now.

> We definitely need reproducible package rebuild.

Agreed.

Maxim.

> 
> Jan
> 
>>
>> Thanks,
>> Maxim.
>>
>> On 1/31/19 5:42 PM, Henning Schild wrote:
>>> From: Henning Schild <henning.schild@siemens.com>
>>>
>>> The recipe uses "apt://" fetching which is not yet supported in the
>>> cache. This patch should be reverted once we cache sources as well.
>>>
>>> Signed-off-by: Henning Schild <henning.schild@siemens.com>
>>> ---
>>>   scripts/ci_build.sh | 2 ++
>>>   1 file changed, 2 insertions(+)
>>>
>>> diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
>>> index f3523e8..63e3d21 100755
>>> --- a/scripts/ci_build.sh
>>> +++ b/scripts/ci_build.sh
>>> @@ -97,6 +97,8 @@ if [ -n "$FAST_BUILD" ]; then
>>>       while [ -e bitbake.sock ]; do sleep 1; done
>>>       sudo rm -rf tmp
>>>       sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?= 
>>> "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
>>> +    # this packages uses a feature that does not yet work in cached 
>>> mode
>>> +    sed -i -e 's/ hello//g' conf/local.conf
>>>       bitbake $BB_ARGS \
>>>           multiconfig:qemuarm-stretch:isar-image-base \
>>>           multiconfig:qemuarm64-stretch:isar-image-base \
>>>
>>
>>
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: [PATCHv4 0/9] apt-get source support

[Maxim Yu. Osipov]

On 1/31/19 5:42 PM, Henning Schild wrote:
> From: Henning Schild <henning.schild@siemens.com>
> 
> diff v3 .. v4:
>   - add patch to skip cached ci for hello because of known issue
>   - fixed copyright headers
>   - changed MAINTAINER of hello to isar-users
> 
> v2 was sent by Maxim so i gave it a 3
> 
> diff to v1 .. v3:
>   - add "apt://" protocol to SRC_URI p7, not squashed
>   - add docs
>   - change the way i mention Cedric as co-author
>   - disable cross-build testing in hello


Applied to the 'next', two patches are dropped:
patch #6  "local.conf: enable rebuilding "hello" for all distros" and 
patch #9 "ci: remove "hello" from the offline build, it fail".

Thanks,
Maxim.

> 
> *** BLURB HERE ***
> 
> Henning Schild (9):
>    conf: add deb-src entries to all our distro configs
>    dpkg-base: introduce an "apt-get source" fetch/unpack step
>    meta: move debianization code into a class and into dpkg-base
>    debianize: allow changlog version change
>    meta-isar/recipes-app: add upstream hello rebuild example
>    local.conf: enable rebuilding "hello" for all distros
>    meta: change apt source fetcher to hook into SRC_URI
>    doc: document "apt://" fetch/unpack feature
>    ci: remove "hello" from the offline build, it fails
> 
>   doc/user_manual.md                            |  4 +
>   .../conf/distro/debian-stretch-backports.list |  1 +
>   meta-isar/conf/distro/raspbian-jessie.list    |  1 +
>   meta-isar/conf/local.conf.sample              |  2 +-
>   meta-isar/recipes-app/hello/hello.inc         | 26 ++++++
>   meta-isar/recipes-app/hello/hello_2.10.bb     | 11 +++
>   meta-isar/recipes-app/hello/hello_2.9.bb      | 11 +++
>   meta/classes/base.bbclass                     | 19 ++++
>   meta/classes/debianize.bbclass                | 89 +++++++++++++++++++
>   meta/classes/dpkg-base.bbclass                | 20 +++++
>   meta/classes/dpkg-raw.bbclass                 | 87 +-----------------
>   meta/conf/distro/debian-buster.list           |  3 +
>   meta/conf/distro/debian-jessie.list           |  3 +
>   meta/conf/distro/debian-stretch.list          |  3 +
>   scripts/ci_build.sh                           |  2 +
>   15 files changed, 197 insertions(+), 85 deletions(-)
>   create mode 100644 meta-isar/recipes-app/hello/hello.inc
>   create mode 100644 meta-isar/recipes-app/hello/hello_2.10.bb
>   create mode 100644 meta-isar/recipes-app/hello/hello_2.9.bb
>   create mode 100644 meta/classes/debianize.bbclass
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: [PATCHv4 9/9] ci: remove "hello" from the offline build, it fails

[Henning Schild]

Am Tue, 5 Feb 2019 10:49:49 +0100
schrieb "Maxim Yu. Osipov" <mosipov@ilbers.de>:

> On 2/4/19 6:37 PM, Jan Kiszka wrote:
> > On 04.02.19 18:33, Maxim Yu. Osipov wrote:  
> >> Hi Henning,
> >>
> >> I'm OK with your approach/series except this patch as
> >> this patch just hides the problem from CI build and
> >> will immediately pop up in any own user's offline build.
> >>
> >> So I suggest to drop from the series patch "local.conf: enable 
> >> rebuilding "hello" for all distros" until
> >> limitation with offline builds will gone.  
> > 
> > Which also raises the questions: Who has a proposal how to fix that 
> > issue and who can work on it?  
> 
> I'm looking at the problem now.

Sweet! I left it open because it kind of distracts from the main point
of this series. And i was hoping someone better in reprepro would maybe
pick it up.

> > We definitely need reproducible package rebuild.  
> 
> Agreed.

Ack.

Henning

> Maxim.
> 
> > 
> > Jan
> >   
> >>
> >> Thanks,
> >> Maxim.
> >>
> >> On 1/31/19 5:42 PM, Henning Schild wrote:  
> >>> From: Henning Schild <henning.schild@siemens.com>
> >>>
> >>> The recipe uses "apt://" fetching which is not yet supported in
> >>> the cache. This patch should be reverted once we cache sources as
> >>> well.
> >>>
> >>> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> >>> ---
> >>>   scripts/ci_build.sh | 2 ++
> >>>   1 file changed, 2 insertions(+)
> >>>
> >>> diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
> >>> index f3523e8..63e3d21 100755
> >>> --- a/scripts/ci_build.sh
> >>> +++ b/scripts/ci_build.sh
> >>> @@ -97,6 +97,8 @@ if [ -n "$FAST_BUILD" ]; then
> >>>       while [ -e bitbake.sock ]; do sleep 1; done
> >>>       sudo rm -rf tmp
> >>>       sed -i -e 's/#ISAR_USE_CACHED_BASE_REPO ?= 
> >>> "1"/ISAR_USE_CACHED_BASE_REPO ?= "1"/g' conf/local.conf
> >>> +    # this packages uses a feature that does not yet work in
> >>> cached mode
> >>> +    sed -i -e 's/ hello//g' conf/local.conf
> >>>       bitbake $BB_ARGS \
> >>>           multiconfig:qemuarm-stretch:isar-image-base \
> >>>           multiconfig:qemuarm64-stretch:isar-image-base \
> >>>  
> >>
> >>  
> >   
> 
> 

Re: [PATCHv4 0/9] apt-get source support

[Henning Schild]

Am Tue, 5 Feb 2019 11:42:01 +0100
schrieb "Maxim Yu. Osipov" <mosipov@ilbers.de>:

> On 1/31/19 5:42 PM, Henning Schild wrote:
> > From: Henning Schild <henning.schild@siemens.com>
> > 
> > diff v3 .. v4:
> >   - add patch to skip cached ci for hello because of known issue
> >   - fixed copyright headers
> >   - changed MAINTAINER of hello to isar-users
> > 
> > v2 was sent by Maxim so i gave it a 3
> > 
> > diff to v1 .. v3:
> >   - add "apt://" protocol to SRC_URI p7, not squashed
> >   - add docs
> >   - change the way i mention Cedric as co-author
> >   - disable cross-build testing in hello  
> 
> 
> Applied to the 'next', two patches are dropped:
> patch #6  "local.conf: enable rebuilding "hello" for all distros" and 
> patch #9 "ci: remove "hello" from the offline build, it fail".

I think it would be a good idea to include "hello" into one of the
builds that are not rebuild in offline mode. That way we would have
the apt:// feature at least tested a bit.

As far as i can see qemuamd64-buster would be a candidate. Should i
write a patch for that?

Henning

> Thanks,
> Maxim.
> 
> > 
> > *** BLURB HERE ***
> > 
> > Henning Schild (9):
> >    conf: add deb-src entries to all our distro configs
> >    dpkg-base: introduce an "apt-get source" fetch/unpack step
> >    meta: move debianization code into a class and into dpkg-base
> >    debianize: allow changlog version change
> >    meta-isar/recipes-app: add upstream hello rebuild example
> >    local.conf: enable rebuilding "hello" for all distros
> >    meta: change apt source fetcher to hook into SRC_URI
> >    doc: document "apt://" fetch/unpack feature
> >    ci: remove "hello" from the offline build, it fails
> > 
> >   doc/user_manual.md                            |  4 +
> >   .../conf/distro/debian-stretch-backports.list |  1 +
> >   meta-isar/conf/distro/raspbian-jessie.list    |  1 +
> >   meta-isar/conf/local.conf.sample              |  2 +-
> >   meta-isar/recipes-app/hello/hello.inc         | 26 ++++++
> >   meta-isar/recipes-app/hello/hello_2.10.bb     | 11 +++
> >   meta-isar/recipes-app/hello/hello_2.9.bb      | 11 +++
> >   meta/classes/base.bbclass                     | 19 ++++
> >   meta/classes/debianize.bbclass                | 89
> > +++++++++++++++++++ meta/classes/dpkg-base.bbclass                |
> > 20 +++++ meta/classes/dpkg-raw.bbclass                 | 87
> > +----------------- meta/conf/distro/debian-buster.list           |
> > 3 + meta/conf/distro/debian-jessie.list           |  3 +
> >   meta/conf/distro/debian-stretch.list          |  3 +
> >   scripts/ci_build.sh                           |  2 +
> >   15 files changed, 197 insertions(+), 85 deletions(-)
> >   create mode 100644 meta-isar/recipes-app/hello/hello.inc
> >   create mode 100644 meta-isar/recipes-app/hello/hello_2.10.bb
> >   create mode 100644 meta-isar/recipes-app/hello/hello_2.9.bb
> >   create mode 100644 meta/classes/debianize.bbclass
> >   
> 
> 

Re: [PATCHv4 0/9] apt-get source support

[Maxim Yu. Osipov]

On 2/6/19 10:24 AM, Henning Schild wrote:
> Am Tue, 5 Feb 2019 11:42:01 +0100
> schrieb "Maxim Yu. Osipov" <mosipov@ilbers.de>:
> 
>> On 1/31/19 5:42 PM, Henning Schild wrote:
>>> From: Henning Schild <henning.schild@siemens.com>
>>>
>>> diff v3 .. v4:
>>>    - add patch to skip cached ci for hello because of known issue
>>>    - fixed copyright headers
>>>    - changed MAINTAINER of hello to isar-users
>>>
>>> v2 was sent by Maxim so i gave it a 3
>>>
>>> diff to v1 .. v3:
>>>    - add "apt://" protocol to SRC_URI p7, not squashed
>>>    - add docs
>>>    - change the way i mention Cedric as co-author
>>>    - disable cross-build testing in hello
>>
>>
>> Applied to the 'next', two patches are dropped:
>> patch #6  "local.conf: enable rebuilding "hello" for all distros" and
>> patch #9 "ci: remove "hello" from the offline build, it fail".
> 
> I think it would be a good idea to include "hello" into one of the
> builds that are not rebuild in offline mode. That way we would have
> the apt:// feature at least tested a bit.

Agreed.

> As far as i can see qemuamd64-buster would be a candidate. Should i
> write a patch for that?

Would be fine.

Maxim.


> 
> Henning
> 
>> Thanks,
>> Maxim.
>>
>>>
>>> *** BLURB HERE ***
>>>
>>> Henning Schild (9):
>>>     conf: add deb-src entries to all our distro configs
>>>     dpkg-base: introduce an "apt-get source" fetch/unpack step
>>>     meta: move debianization code into a class and into dpkg-base
>>>     debianize: allow changlog version change
>>>     meta-isar/recipes-app: add upstream hello rebuild example
>>>     local.conf: enable rebuilding "hello" for all distros
>>>     meta: change apt source fetcher to hook into SRC_URI
>>>     doc: document "apt://" fetch/unpack feature
>>>     ci: remove "hello" from the offline build, it fails
>>>
>>>    doc/user_manual.md                            |  4 +
>>>    .../conf/distro/debian-stretch-backports.list |  1 +
>>>    meta-isar/conf/distro/raspbian-jessie.list    |  1 +
>>>    meta-isar/conf/local.conf.sample              |  2 +-
>>>    meta-isar/recipes-app/hello/hello.inc         | 26 ++++++
>>>    meta-isar/recipes-app/hello/hello_2.10.bb     | 11 +++
>>>    meta-isar/recipes-app/hello/hello_2.9.bb      | 11 +++
>>>    meta/classes/base.bbclass                     | 19 ++++
>>>    meta/classes/debianize.bbclass                | 89
>>> +++++++++++++++++++ meta/classes/dpkg-base.bbclass                |
>>> 20 +++++ meta/classes/dpkg-raw.bbclass                 | 87
>>> +----------------- meta/conf/distro/debian-buster.list           |
>>> 3 + meta/conf/distro/debian-jessie.list           |  3 +
>>>    meta/conf/distro/debian-stretch.list          |  3 +
>>>    scripts/ci_build.sh                           |  2 +
>>>    15 files changed, 197 insertions(+), 85 deletions(-)
>>>    create mode 100644 meta-isar/recipes-app/hello/hello.inc
>>>    create mode 100644 meta-isar/recipes-app/hello/hello_2.10.bb
>>>    create mode 100644 meta-isar/recipes-app/hello/hello_2.9.bb
>>>    create mode 100644 meta/classes/debianize.bbclass
>>>    
>>
>>
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov