[PATCH 0/7] "apt-get source" fetch/unpack support

[PATCH 0/7] "apt-get source" fetch/unpack support

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

This series includes support for fetching upstream sources with "apt-get
source". This will make sure we fetch exactly what matches out distro,
without rewriting debian fetch/unpack logic.
I did consider implementing it as an "apt://" extension to the regular
fetcher but decided against that. You have to set SRC_APT and
effectively pass arguement to apt-get. That fetcher can only work in
packages and depends on buildchroot and mounting, so it can not be part
of the general fetcher. But maybe the general fetcher could ignore
"apt://" lines and this task will ignore anything but "apt://" so we can
still use SRC_URI instead of SRC_APT. Let us talk about the interface.
The documentation is still missing, but i thing it is ready for a first
review.

The idea and parts of the code came from Mentor.

Henning Schild (7):
  conf: add deb-src entries to all our distro configs
  dpkg-base: introduce an "apt-get source" fetch/unpack step
  meta: move debianization code into a class and into dpkg-base
  debianize: allow changlog version change
  meta-isar/recipes-app: add upstream hello rebuild example
  local.conf: remove example-hello from the default build
  local.conf: enable rebuilding "hello" for all distros

 meta-isar/conf/distro/debian-buster.list      |  3 +
 meta-isar/conf/distro/debian-jessie.list      |  3 +
 .../conf/distro/debian-stretch-backports.list |  1 +
 meta-isar/conf/distro/debian-stretch.list     |  3 +
 meta-isar/conf/distro/raspbian-jessie.list    |  1 +
 meta-isar/conf/local.conf.sample              |  2 +-
 meta-isar/recipes-app/hello/hello.inc         | 19 +++++
 meta-isar/recipes-app/hello/hello_2.10.bb     |  8 ++
 meta-isar/recipes-app/hello/hello_2.9.bb      |  8 ++
 meta/classes/debianize.bbclass                | 82 +++++++++++++++++++
 meta/classes/dpkg-base.bbclass                | 20 +++++
 meta/classes/dpkg-raw.bbclass                 | 76 -----------------
 12 files changed, 149 insertions(+), 77 deletions(-)
 create mode 100644 meta-isar/recipes-app/hello/hello.inc
 create mode 100644 meta-isar/recipes-app/hello/hello_2.10.bb
 create mode 100644 meta-isar/recipes-app/hello/hello_2.9.bb
 create mode 100644 meta/classes/debianize.bbclass

-- 
2.19.2

[PATCH 1/7] conf: add deb-src entries to all our distro configs

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

That will allow us to use "apt-get source" to fetch debian sources from
the mirrors we also get the binary packages from.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta-isar/conf/distro/debian-buster.list            | 3 +++
 meta-isar/conf/distro/debian-jessie.list            | 3 +++
 meta-isar/conf/distro/debian-stretch-backports.list | 1 +
 meta-isar/conf/distro/debian-stretch.list           | 3 +++
 meta-isar/conf/distro/raspbian-jessie.list          | 1 +
 5 files changed, 11 insertions(+)

diff --git a/meta-isar/conf/distro/debian-buster.list b/meta-isar/conf/distro/debian-buster.list
index 18311d8..1e38f1a 100644
--- a/meta-isar/conf/distro/debian-buster.list
+++ b/meta-isar/conf/distro/debian-buster.list
@@ -1,3 +1,6 @@
 deb	http://ftp.de.debian.org/debian buster	main contrib non-free
+deb-src	http://ftp.de.debian.org/debian buster	main contrib non-free
 deb	http://ftp.de.debian.org/debian buster-updates	main contrib non-free
+deb-src	http://ftp.de.debian.org/debian buster-updates	main contrib non-free
 deb	http://security.debian.org	buster/updates	main contrib non-free
+deb-src	http://security.debian.org	buster/updates	main contrib non-free
diff --git a/meta-isar/conf/distro/debian-jessie.list b/meta-isar/conf/distro/debian-jessie.list
index be46a57..5143f4d 100644
--- a/meta-isar/conf/distro/debian-jessie.list
+++ b/meta-isar/conf/distro/debian-jessie.list
@@ -1,3 +1,6 @@
 deb	http://ftp.de.debian.org/debian	jessie	main contrib non-free
+deb-src	http://ftp.de.debian.org/debian	jessie	main contrib non-free
 deb	http://ftp.de.debian.org/debian jessie-updates	main contrib non-free
+deb-src	http://ftp.de.debian.org/debian jessie-updates	main contrib non-free
 deb	http://security.debian.org	jessie/updates	main contrib non-free
+deb-src	http://security.debian.org	jessie/updates	main contrib non-free
diff --git a/meta-isar/conf/distro/debian-stretch-backports.list b/meta-isar/conf/distro/debian-stretch-backports.list
index 727a835..f606be5 100644
--- a/meta-isar/conf/distro/debian-stretch-backports.list
+++ b/meta-isar/conf/distro/debian-stretch-backports.list
@@ -1 +1,2 @@
 deb	http://ftp.de.debian.org/debian stretch-backports main
+deb-src	http://ftp.de.debian.org/debian stretch-backports main
diff --git a/meta-isar/conf/distro/debian-stretch.list b/meta-isar/conf/distro/debian-stretch.list
index bfa5cf9..d4f3de0 100644
--- a/meta-isar/conf/distro/debian-stretch.list
+++ b/meta-isar/conf/distro/debian-stretch.list
@@ -1,3 +1,6 @@
 deb	http://ftp.de.debian.org/debian stretch	main contrib non-free
+deb-src	http://ftp.de.debian.org/debian stretch	main contrib non-free
 deb	http://ftp.de.debian.org/debian stretch-updates	main contrib non-free
+deb-src	http://ftp.de.debian.org/debian stretch-updates	main contrib non-free
 deb	http://security.debian.org	stretch/updates	main contrib non-free
+deb-src	http://security.debian.org	stretch/updates	main contrib non-free
diff --git a/meta-isar/conf/distro/raspbian-jessie.list b/meta-isar/conf/distro/raspbian-jessie.list
index 792f9c6..0a5af41 100644
--- a/meta-isar/conf/distro/raspbian-jessie.list
+++ b/meta-isar/conf/distro/raspbian-jessie.list
@@ -1 +1,2 @@
 deb	http://archive.raspbian.org/raspbian	jessie	main contrib non-free firmware
+deb-src	http://archive.raspbian.org/raspbian	jessie	main contrib non-free firmware
-- 
2.19.2

[PATCH 2/7] dpkg-base: introduce an "apt-get source" fetch/unpack step

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

This creates a new task to fetch/unpack debian sources from debian
source mirrors. This is done using "apt-get source" and the new variable
APT_SRC to control the arguments.

An example where a original debian package gets rebuild will follow in a
later commit.

Signed-off-by: Cedric Hombourger <Cedric_Hombourger@mentor.com>
Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta/classes/dpkg-base.bbclass | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index f1b127c..2f324f9 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -20,6 +20,25 @@ do_adjust_git[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 inherit patch
 addtask patch after do_adjust_git before do_build
 
+SRC_APT ?= ""
+
+do_apt_fetch[depends] = "buildchroot-target:do_build"
+
+do_apt_fetch() {
+	if [ -z "${@d.getVar("SRC_APT", True).strip()}" ]; then
+		exit
+	fi
+	dpkg_do_mounts
+	E="${@ bb.utils.export_proxies(d)}"
+	sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
+		sh -c 'cd ${PP} && apt-get -y source ${SRC_APT}'
+	dpkg_undo_mounts
+}
+
+addtask apt_fetch after do_unpack before do_patch
+do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
+do_apt_fetch[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
+
 def get_package_srcdir(d):
     s = d.getVar("S", True)
     workdir = d.getVar("WORKDIR", True)
-- 
2.19.2

[PATCH 3/7] meta: move debianization code into a class and into dpkg-base

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

This is pure restructuring. Move the code into its own class and include
that in dpkg-base, it was available only in dpkg-raw before.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta/classes/debianize.bbclass | 80 ++++++++++++++++++++++++++++++++++
 meta/classes/dpkg-base.bbclass |  1 +
 meta/classes/dpkg-raw.bbclass  | 76 --------------------------------
 3 files changed, 81 insertions(+), 76 deletions(-)
 create mode 100644 meta/classes/debianize.bbclass

diff --git a/meta/classes/debianize.bbclass b/meta/classes/debianize.bbclass
new file mode 100644
index 0000000..c771e9c
--- /dev/null
+++ b/meta/classes/debianize.bbclass
@@ -0,0 +1,80 @@
+# This software is a part of ISAR.
+# Copyright (C) 2017-2019 Siemens AG
+
+deb_add_changelog() {
+	date=$( LANG=C date -R )
+	cat <<EOF > ${S}/debian/changelog
+${PN} (${PV}) UNRELEASED; urgency=low
+
+  * generated by Isar
+
+ -- ${MAINTAINER}  ${date}
+EOF
+	if [ -f ${WORKDIR}/changelog ]; then
+		echo >> ${S}/debian/changelog
+		cat ${WORKDIR}/changelog >> ${S}/debian/changelog
+	fi
+}
+
+deb_create_compat() {
+	echo 9 > ${S}/debian/compat
+}
+
+deb_create_control() {
+	compat=$( cat ${S}/debian/compat )
+	cat << EOF > ${S}/debian/control
+Source: ${PN}
+Section: misc
+Priority: optional
+Standards-Version: 3.9.6
+Maintainer: ${MAINTAINER}
+Build-Depends: debhelper (>= ${compat})
+
+Package: ${PN}
+Architecture: any
+Depends: ${DEBIAN_DEPENDS}
+Description: ${DESCRIPTION}
+EOF
+}
+
+deb_create_rules() {
+	cat << EOF > ${S}/debian/rules
+#!/usr/bin/make -f
+%:
+	dh \$@
+
+EOF
+	chmod +x ${S}/debian/rules
+}
+
+deb_debianize() {
+	if [ -f ${WORKDIR}/compat ]; then
+		install -v -m 644 ${WORKDIR}/compat ${S}/debian/compat
+	else
+		deb_create_compat
+	fi
+	if [ -f ${WORKDIR}/control ]; then
+		install -v -m 644 ${WORKDIR}/control ${S}/debian/control
+	else
+		deb_create_control
+	fi
+	if [ -f ${WORKDIR}/rules ]; then
+		install -v -m 755 ${WORKDIR}/rules ${S}/debian/rules
+	else
+		deb_create_rules
+	fi
+	deb_add_changelog
+
+	for t in pre post
+	do
+		for a in inst rm
+		do
+			if [ -f ${WORKDIR}/${t}${a} ]; then
+				install -v -m 755 ${WORKDIR}/${t}${a} \
+					${S}/debian/${t}${a}
+			fi
+		done
+	done
+}
+
+
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 2f324f9..175dc80 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -4,6 +4,7 @@
 # SPDX-License-Identifier: MIT
 
 inherit buildchroot
+inherit debianize
 
 DEPENDS ?= ""
 
diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
index 8d11433..0434313 100644
--- a/meta/classes/dpkg-raw.bbclass
+++ b/meta/classes/dpkg-raw.bbclass
@@ -17,82 +17,6 @@ do_install[cleandirs] = "${D}"
 do_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 addtask install after do_unpack before do_prepare_build
 
-deb_add_changelog() {
-	date=$( LANG=C date -R )
-	cat <<EOF > ${D}/debian/changelog
-${PN} (${PV}) UNRELEASED; urgency=low
-
-  * generated by Isar
-
- -- ${MAINTAINER}  ${date}
-EOF
-	if [ -f ${WORKDIR}/changelog ]; then
-		echo >> ${D}/debian/changelog
-		cat ${WORKDIR}/changelog >> ${D}/debian/changelog
-	fi
-}
-
-deb_create_compat() {
-	echo 9 > ${D}/debian/compat
-}
-
-deb_create_control() {
-	compat=$( cat ${D}/debian/compat )
-	cat << EOF > ${D}/debian/control
-Source: ${PN}
-Section: misc
-Priority: optional
-Standards-Version: 3.9.6
-Maintainer: ${MAINTAINER}
-Build-Depends: debhelper (>= ${compat})
-
-Package: ${PN}
-Architecture: any
-Depends: ${DEBIAN_DEPENDS}
-Description: ${DESCRIPTION}
-EOF
-}
-
-deb_create_rules() {
-	cat << EOF > ${S}/debian/rules
-#!/usr/bin/make -f
-%:
-	dh \$@
-
-EOF
-	chmod +x ${S}/debian/rules
-}
-
-deb_debianize() {
-	if [ -f ${WORKDIR}/compat ]; then
-		install -v -m 644 ${WORKDIR}/compat ${D}/debian/compat
-	else
-		deb_create_compat
-	fi
-	if [ -f ${WORKDIR}/control ]; then
-		install -v -m 644 ${WORKDIR}/control ${D}/debian/control
-	else
-		deb_create_control
-	fi
-	if [ -f ${WORKDIR}/rules ]; then
-		install -v -m 755 ${WORKDIR}/rules ${D}/debian/rules
-	else
-		deb_create_rules
-	fi
-	deb_add_changelog
-
-	for t in pre post
-	do
-		for a in inst rm
-		do
-			if [ -f ${WORKDIR}/${t}${a} ]; then
-				install -v -m 755 ${WORKDIR}/${t}${a} \
-					${D}/debian/${t}${a}
-			fi
-		done
-	done
-}
-
 do_prepare_build[cleandirs] += "${D}/debian"
 do_prepare_build() {
 	cd ${D}
-- 
2.19.2

[PATCH 4/7] debianize: allow changlog version change

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

Introduce a variable to control the version string we put into a
generated changelog entry.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta/classes/debianize.bbclass | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/meta/classes/debianize.bbclass b/meta/classes/debianize.bbclass
index c771e9c..64994e4 100644
--- a/meta/classes/debianize.bbclass
+++ b/meta/classes/debianize.bbclass
@@ -1,10 +1,12 @@
 # This software is a part of ISAR.
 # Copyright (C) 2017-2019 Siemens AG
 
+CHANGELOG_V ?= "${PV}"
+
 deb_add_changelog() {
 	date=$( LANG=C date -R )
 	cat <<EOF > ${S}/debian/changelog
-${PN} (${PV}) UNRELEASED; urgency=low
+${PN} (${CHANGELOG_V}) UNRELEASED; urgency=low
 
   * generated by Isar
 
-- 
2.19.2

[PATCH 5/7] meta-isar/recipes-app: add upstream hello rebuild example

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

This commit introduces a rebuild example of an upstream package. It also
shows how to deal with multiple versions of such a package, depending on
your distro. And in fact we actually have to patch it to rebuild it,
because its build-deps are wrong.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta-isar/recipes-app/hello/hello.inc     | 19 +++++++++++++++++++
 meta-isar/recipes-app/hello/hello_2.10.bb |  8 ++++++++
 meta-isar/recipes-app/hello/hello_2.9.bb  |  8 ++++++++
 3 files changed, 35 insertions(+)
 create mode 100644 meta-isar/recipes-app/hello/hello.inc
 create mode 100644 meta-isar/recipes-app/hello/hello_2.10.bb
 create mode 100644 meta-isar/recipes-app/hello/hello_2.9.bb

diff --git a/meta-isar/recipes-app/hello/hello.inc b/meta-isar/recipes-app/hello/hello.inc
new file mode 100644
index 0000000..49e2eb9
--- /dev/null
+++ b/meta-isar/recipes-app/hello/hello.inc
@@ -0,0 +1,19 @@
+# This software is a part of ISAR.
+
+inherit dpkg
+
+# this will fetch and unpack the sources from upstream debian
+SRC_APT = "${PN}"
+
+MAINTAINER = "Your name here <you@domain.com>"
+CHANGELOG_V = "${PV}-99+isar"
+
+do_prepare_build() {
+	deb_add_changelog
+	# this seems to be a build dep missing in the upstream control file
+	if ! grep texinfo ${S}/debian/control; then
+		sed -i -e 's/Build-Depends:/Build-Depends: texinfo,/g' ${S}/debian/control
+	fi
+}
+
+
diff --git a/meta-isar/recipes-app/hello/hello_2.10.bb b/meta-isar/recipes-app/hello/hello_2.10.bb
new file mode 100644
index 0000000..1598565
--- /dev/null
+++ b/meta-isar/recipes-app/hello/hello_2.10.bb
@@ -0,0 +1,8 @@
+# Example recipe to rebuild a debian source package
+#
+# This software is a part of ISAR.
+
+require hello.inc
+
+DEFAULT_PREFERENCE_debian-buster = "1"
+DEFAULT_PREFERENCE_debian-stretch = "1"
diff --git a/meta-isar/recipes-app/hello/hello_2.9.bb b/meta-isar/recipes-app/hello/hello_2.9.bb
new file mode 100644
index 0000000..b5d5b2e
--- /dev/null
+++ b/meta-isar/recipes-app/hello/hello_2.9.bb
@@ -0,0 +1,8 @@
+# Example recipe to rebuild a debian source package
+#
+# This software is a part of ISAR.
+
+require hello.inc
+
+DEFAULT_PREFERENCE_debian-jessie = "1"
+DEFAULT_PREFERENCE_raspbian-jessie = "1"
-- 
2.19.2

[PATCH 6/7] local.conf: remove example-hello from the default build

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

As reported on the mailinglist, this recipe conflicts with "hello" from
upstream debian. This commit should be reverted once the conflict is
resolved.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta-isar/conf/local.conf.sample | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index fe81f20..6ed4caa 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -161,7 +161,7 @@ CONF_VERSION = "1"
 
 #
 # The default list of extra packages to be installed.
-IMAGE_INSTALL = "example-hello example-raw example-module-${KERNEL_NAME} enable-fsck"
+IMAGE_INSTALL = "example-raw example-module-${KERNEL_NAME} enable-fsck"
 
 #
 # Enable cross-compilation support
-- 
2.19.2

[PATCH 7/7] local.conf: enable rebuilding "hello" for all distros

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

This commit includes the rebuild example in our default builds and
therefore into CI.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta-isar/conf/local.conf.sample | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index 6ed4caa..bdf889b 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -161,7 +161,7 @@ CONF_VERSION = "1"
 
 #
 # The default list of extra packages to be installed.
-IMAGE_INSTALL = "example-raw example-module-${KERNEL_NAME} enable-fsck"
+IMAGE_INSTALL = "example-raw example-module-${KERNEL_NAME} enable-fsck hello"
 
 #
 # Enable cross-compilation support
-- 
2.19.2

Re: [PATCH 0/7] "apt-get source" fetch/unpack support

[Henning Schild]

Am Thu, 17 Jan 2019 17:04:20 +0100
schrieb Henning Schild <henning.schild@siemens.com>:

> From: Henning Schild <henning.schild@siemens.com>
> 
> This series includes support for fetching upstream sources with
> "apt-get source". This will make sure we fetch exactly what matches
> out distro, without rewriting debian fetch/unpack logic.
> I did consider implementing it as an "apt://" extension to the regular
> fetcher but decided against that. You have to set SRC_APT and
> effectively pass arguement to apt-get. That fetcher can only work in
> packages and depends on buildchroot and mounting, so it can not be
> part of the general fetcher. But maybe the general fetcher could
> ignore "apt://" lines and this task will ignore anything but "apt://"
> so we can still use SRC_URI instead of SRC_APT.

Filtering all the "apt://" entries out does not really work. There is a
call to bb.fetch.get_checksum_file_list where i can not filter, at
least i did not yet figure out how.

And it seems that the fetcher can not be extended with a new protocol
without touching the bitbake core. Otherwise an empty implementation of
"apt://" would have been the trick to let debian do it in a completely
different step.
Not sure it is worth forcing it into SRC_URI, we are in two worlds
anyways, no need to pretend otherwise.

Henning

> Let us talk about the
> interface. The documentation is still missing, but i thing it is
> ready for a first review.
> 
> The idea and parts of the code came from Mentor.
> 
> Henning Schild (7):
>   conf: add deb-src entries to all our distro configs
>   dpkg-base: introduce an "apt-get source" fetch/unpack step
>   meta: move debianization code into a class and into dpkg-base
>   debianize: allow changlog version change
>   meta-isar/recipes-app: add upstream hello rebuild example
>   local.conf: remove example-hello from the default build
>   local.conf: enable rebuilding "hello" for all distros
> 
>  meta-isar/conf/distro/debian-buster.list      |  3 +
>  meta-isar/conf/distro/debian-jessie.list      |  3 +
>  .../conf/distro/debian-stretch-backports.list |  1 +
>  meta-isar/conf/distro/debian-stretch.list     |  3 +
>  meta-isar/conf/distro/raspbian-jessie.list    |  1 +
>  meta-isar/conf/local.conf.sample              |  2 +-
>  meta-isar/recipes-app/hello/hello.inc         | 19 +++++
>  meta-isar/recipes-app/hello/hello_2.10.bb     |  8 ++
>  meta-isar/recipes-app/hello/hello_2.9.bb      |  8 ++
>  meta/classes/debianize.bbclass                | 82
> +++++++++++++++++++ meta/classes/dpkg-base.bbclass                |
> 20 +++++ meta/classes/dpkg-raw.bbclass                 | 76
> ----------------- 12 files changed, 149 insertions(+), 77 deletions(-)
>  create mode 100644 meta-isar/recipes-app/hello/hello.inc
>  create mode 100644 meta-isar/recipes-app/hello/hello_2.10.bb
>  create mode 100644 meta-isar/recipes-app/hello/hello_2.9.bb
>  create mode 100644 meta/classes/debianize.bbclass
> 

Re: [PATCH 0/7] "apt-get source" fetch/unpack support

[Claudius Heine]

Hi Henning,

On 17/01/2019 17.38, [ext] Henning Schild wrote:
> Am Thu, 17 Jan 2019 17:04:20 +0100
> schrieb Henning Schild <henning.schild@siemens.com>:
> 
>> From: Henning Schild <henning.schild@siemens.com>
>>
>> This series includes support for fetching upstream sources with
>> "apt-get source". This will make sure we fetch exactly what matches
>> out distro, without rewriting debian fetch/unpack logic.
>> I did consider implementing it as an "apt://" extension to the regular
>> fetcher but decided against that. You have to set SRC_APT and
>> effectively pass arguement to apt-get. That fetcher can only work in
>> packages and depends on buildchroot and mounting, so it can not be
>> part of the general fetcher. But maybe the general fetcher could
>> ignore "apt://" lines and this task will ignore anything but "apt://"
>> so we can still use SRC_URI instead of SRC_APT.
> 
> Filtering all the "apt://" entries out does not really work. There is a
> call to bb.fetch.get_checksum_file_list where i can not filter, at
> least i did not yet figure out how.
> 
> And it seems that the fetcher can not be extended with a new protocol
> without touching the bitbake core. Otherwise an empty implementation of
> "apt://" would have been the trick to let debian do it in a completely
> different step.
> Not sure it is worth forcing it into SRC_URI, we are in two worlds
> anyways, no need to pretend otherwise.

Ok thx! As for variable conflict: There is "APTSRCS" in isar-bootstrap, 
which it IMO far enough away from "SRC_APT", so that name would be OK.

Claudius

> 
> Henning
> 
>> Let us talk about the
>> interface. The documentation is still missing, but i thing it is
>> ready for a first review.
>>
>> The idea and parts of the code came from Mentor.
>>
>> Henning Schild (7):
>>    conf: add deb-src entries to all our distro configs
>>    dpkg-base: introduce an "apt-get source" fetch/unpack step
>>    meta: move debianization code into a class and into dpkg-base
>>    debianize: allow changlog version change
>>    meta-isar/recipes-app: add upstream hello rebuild example
>>    local.conf: remove example-hello from the default build
>>    local.conf: enable rebuilding "hello" for all distros
>>
>>   meta-isar/conf/distro/debian-buster.list      |  3 +
>>   meta-isar/conf/distro/debian-jessie.list      |  3 +
>>   .../conf/distro/debian-stretch-backports.list |  1 +
>>   meta-isar/conf/distro/debian-stretch.list     |  3 +
>>   meta-isar/conf/distro/raspbian-jessie.list    |  1 +
>>   meta-isar/conf/local.conf.sample              |  2 +-
>>   meta-isar/recipes-app/hello/hello.inc         | 19 +++++
>>   meta-isar/recipes-app/hello/hello_2.10.bb     |  8 ++
>>   meta-isar/recipes-app/hello/hello_2.9.bb      |  8 ++
>>   meta/classes/debianize.bbclass                | 82
>> +++++++++++++++++++ meta/classes/dpkg-base.bbclass                |
>> 20 +++++ meta/classes/dpkg-raw.bbclass                 | 76
>> ----------------- 12 files changed, 149 insertions(+), 77 deletions(-)
>>   create mode 100644 meta-isar/recipes-app/hello/hello.inc
>>   create mode 100644 meta-isar/recipes-app/hello/hello_2.10.bb
>>   create mode 100644 meta-isar/recipes-app/hello/hello_2.9.bb
>>   create mode 100644 meta/classes/debianize.bbclass
>>
> 

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [PATCH 0/7] "apt-get source" fetch/unpack support

[Henning Schild]

ping

Am Thu, 17 Jan 2019 17:04:20 +0100
schrieb Henning Schild <henning.schild@siemens.com>:

> From: Henning Schild <henning.schild@siemens.com>
> 
> This series includes support for fetching upstream sources with
> "apt-get source". This will make sure we fetch exactly what matches
> out distro, without rewriting debian fetch/unpack logic.
> I did consider implementing it as an "apt://" extension to the regular
> fetcher but decided against that. You have to set SRC_APT and
> effectively pass arguement to apt-get. That fetcher can only work in
> packages and depends on buildchroot and mounting, so it can not be
> part of the general fetcher. But maybe the general fetcher could
> ignore "apt://" lines and this task will ignore anything but "apt://"
> so we can still use SRC_URI instead of SRC_APT. Let us talk about the
> interface. The documentation is still missing, but i thing it is
> ready for a first review.
> 
> The idea and parts of the code came from Mentor.
> 
> Henning Schild (7):
>   conf: add deb-src entries to all our distro configs
>   dpkg-base: introduce an "apt-get source" fetch/unpack step
>   meta: move debianization code into a class and into dpkg-base
>   debianize: allow changlog version change
>   meta-isar/recipes-app: add upstream hello rebuild example
>   local.conf: remove example-hello from the default build
>   local.conf: enable rebuilding "hello" for all distros
> 
>  meta-isar/conf/distro/debian-buster.list      |  3 +
>  meta-isar/conf/distro/debian-jessie.list      |  3 +
>  .../conf/distro/debian-stretch-backports.list |  1 +
>  meta-isar/conf/distro/debian-stretch.list     |  3 +
>  meta-isar/conf/distro/raspbian-jessie.list    |  1 +
>  meta-isar/conf/local.conf.sample              |  2 +-
>  meta-isar/recipes-app/hello/hello.inc         | 19 +++++
>  meta-isar/recipes-app/hello/hello_2.10.bb     |  8 ++
>  meta-isar/recipes-app/hello/hello_2.9.bb      |  8 ++
>  meta/classes/debianize.bbclass                | 82
> +++++++++++++++++++ meta/classes/dpkg-base.bbclass                |
> 20 +++++ meta/classes/dpkg-raw.bbclass                 | 76
> ----------------- 12 files changed, 149 insertions(+), 77 deletions(-)
>  create mode 100644 meta-isar/recipes-app/hello/hello.inc
>  create mode 100644 meta-isar/recipes-app/hello/hello_2.10.bb
>  create mode 100644 meta-isar/recipes-app/hello/hello_2.9.bb
>  create mode 100644 meta/classes/debianize.bbclass
> 

Re: [PATCH 2/7] dpkg-base: introduce an "apt-get source" fetch/unpack step

[Jan Kiszka]

On 17.01.19 17:04, [ext] Henning Schild wrote:
> From: Henning Schild <henning.schild@siemens.com>
> 
> This creates a new task to fetch/unpack debian sources from debian
> source mirrors. This is done using "apt-get source" and the new variable
> APT_SRC to control the arguments.
> 
> An example where a original debian package gets rebuild will follow in a
> later commit.
> 
> Signed-off-by: Cedric Hombourger <Cedric_Hombourger@mentor.com>

How does Cedric SOB come in here, before your own? Is it his patch that should 
actually be credited by a proper "From:"?

> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> ---
>   meta/classes/dpkg-base.bbclass | 19 +++++++++++++++++++
>   1 file changed, 19 insertions(+)
> 
> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
> index f1b127c..2f324f9 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -20,6 +20,25 @@ do_adjust_git[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>   inherit patch
>   addtask patch after do_adjust_git before do_build
>   
> +SRC_APT ?= ""
> +
> +do_apt_fetch[depends] = "buildchroot-target:do_build"
> +
> +do_apt_fetch() {
> +	if [ -z "${@d.getVar("SRC_APT", True).strip()}" ]; then
> +		exit
> +	fi
> +	dpkg_do_mounts
> +	E="${@ bb.utils.export_proxies(d)}"
> +	sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
> +		sh -c 'cd ${PP} && apt-get -y source ${SRC_APT}'
> +	dpkg_undo_mounts
> +}
> +
> +addtask apt_fetch after do_unpack before do_patch
> +do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
> +do_apt_fetch[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> +
>   def get_package_srcdir(d):
>       s = d.getVar("S", True)
>       workdir = d.getVar("WORKDIR", True)
> 

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH 6/7] local.conf: remove example-hello from the default build

[Jan Kiszka]

On 17.01.19 17:04, [ext] Henning Schild wrote:
> From: Henning Schild <henning.schild@siemens.com>
> 
> As reported on the mailinglist, this recipe conflicts with "hello" from
> upstream debian. This commit should be reverted once the conflict is
> resolved.

I think the resolution is merged already, no?

Jan

> 
> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> ---
>   meta-isar/conf/local.conf.sample | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
> index fe81f20..6ed4caa 100644
> --- a/meta-isar/conf/local.conf.sample
> +++ b/meta-isar/conf/local.conf.sample
> @@ -161,7 +161,7 @@ CONF_VERSION = "1"
>   
>   #
>   # The default list of extra packages to be installed.
> -IMAGE_INSTALL = "example-hello example-raw example-module-${KERNEL_NAME} enable-fsck"
> +IMAGE_INSTALL = "example-raw example-module-${KERNEL_NAME} enable-fsck"
>   
>   #
>   # Enable cross-compilation support
> 

RE: [PATCH 2/7] dpkg-base: introduce an "apt-get source" fetch/unpack step

[Hombourger, Cedric]

Hi Jan,

Henning did a good of rework to the changeset used in our local project tree, he deserves most of the credits :)

Cedric

-----Original Message-----
From: Jan Kiszka [mailto:jan.kiszka@siemens.com] 
Sent: Monday, January 28, 2019 6:07 PM
To: [ext] Henning Schild <henning.schild@siemens.com>; isar-users <isar-users@googlegroups.com>
Cc: Hombourger, Cedric <Cedric_Hombourger@mentor.com>
Subject: Re: [PATCH 2/7] dpkg-base: introduce an "apt-get source" fetch/unpack step

On 17.01.19 17:04, [ext] Henning Schild wrote:
> From: Henning Schild <henning.schild@siemens.com>
> 
> This creates a new task to fetch/unpack debian sources from debian 
> source mirrors. This is done using "apt-get source" and the new 
> variable APT_SRC to control the arguments.
> 
> An example where a original debian package gets rebuild will follow in 
> a later commit.
> 
> Signed-off-by: Cedric Hombourger <Cedric_Hombourger@mentor.com>

How does Cedric SOB come in here, before your own? Is it his patch that should actually be credited by a proper "From:"?

> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> ---
>   meta/classes/dpkg-base.bbclass | 19 +++++++++++++++++++
>   1 file changed, 19 insertions(+)
> 
> diff --git a/meta/classes/dpkg-base.bbclass 
> b/meta/classes/dpkg-base.bbclass index f1b127c..2f324f9 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -20,6 +20,25 @@ do_adjust_git[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>   inherit patch
>   addtask patch after do_adjust_git before do_build
>   
> +SRC_APT ?= ""
> +
> +do_apt_fetch[depends] = "buildchroot-target:do_build"
> +
> +do_apt_fetch() {
> +	if [ -z "${@d.getVar("SRC_APT", True).strip()}" ]; then
> +		exit
> +	fi
> +	dpkg_do_mounts
> +	E="${@ bb.utils.export_proxies(d)}"
> +	sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
> +		sh -c 'cd ${PP} && apt-get -y source ${SRC_APT}'
> +	dpkg_undo_mounts
> +}
> +
> +addtask apt_fetch after do_unpack before do_patch 
> +do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
> +do_apt_fetch[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> +
>   def get_package_srcdir(d):
>       s = d.getVar("S", True)
>       workdir = d.getVar("WORKDIR", True)
> 

Jan

--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux

Re: [PATCH 2/7] dpkg-base: introduce an "apt-get source" fetch/unpack step

[Jan Kiszka]

On 28.01.19 18:12, Hombourger, Cedric wrote:
> Hi Jan,
> 
> Henning did a good of rework to the changeset used in our local project tree, he deserves most of the credits :)
> 

OK. Then let's move a credit like "based on original patch(es) by ..." into the 
commit log but remove the bogus SOB.

Jan

> Cedric
> 
> -----Original Message-----
> From: Jan Kiszka [mailto:jan.kiszka@siemens.com]
> Sent: Monday, January 28, 2019 6:07 PM
> To: [ext] Henning Schild <henning.schild@siemens.com>; isar-users <isar-users@googlegroups.com>
> Cc: Hombourger, Cedric <Cedric_Hombourger@mentor.com>
> Subject: Re: [PATCH 2/7] dpkg-base: introduce an "apt-get source" fetch/unpack step
> 
> On 17.01.19 17:04, [ext] Henning Schild wrote:
>> From: Henning Schild <henning.schild@siemens.com>
>>
>> This creates a new task to fetch/unpack debian sources from debian
>> source mirrors. This is done using "apt-get source" and the new
>> variable APT_SRC to control the arguments.
>>
>> An example where a original debian package gets rebuild will follow in
>> a later commit.
>>
>> Signed-off-by: Cedric Hombourger <Cedric_Hombourger@mentor.com>
> 
> How does Cedric SOB come in here, before your own? Is it his patch that should actually be credited by a proper "From:"?
> 
>> Signed-off-by: Henning Schild <henning.schild@siemens.com>
>> ---
>>    meta/classes/dpkg-base.bbclass | 19 +++++++++++++++++++
>>    1 file changed, 19 insertions(+)
>>
>> diff --git a/meta/classes/dpkg-base.bbclass
>> b/meta/classes/dpkg-base.bbclass index f1b127c..2f324f9 100644
>> --- a/meta/classes/dpkg-base.bbclass
>> +++ b/meta/classes/dpkg-base.bbclass
>> @@ -20,6 +20,25 @@ do_adjust_git[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>>    inherit patch
>>    addtask patch after do_adjust_git before do_build
>>    
>> +SRC_APT ?= ""
>> +
>> +do_apt_fetch[depends] = "buildchroot-target:do_build"
>> +
>> +do_apt_fetch() {
>> +	if [ -z "${@d.getVar("SRC_APT", True).strip()}" ]; then
>> +		exit
>> +	fi
>> +	dpkg_do_mounts
>> +	E="${@ bb.utils.export_proxies(d)}"
>> +	sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
>> +		sh -c 'cd ${PP} && apt-get -y source ${SRC_APT}'
>> +	dpkg_undo_mounts
>> +}
>> +
>> +addtask apt_fetch after do_unpack before do_patch
>> +do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
>> +do_apt_fetch[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>> +
>>    def get_package_srcdir(d):
>>        s = d.getVar("S", True)
>>        workdir = d.getVar("WORKDIR", True)
>>
> 
> Jan
> 
> --
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE Corporate Competence Center Embedded Linux
> 

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH 0/7] "apt-get source" fetch/unpack support

[Jan Kiszka]

On 17.01.19 17:38, [ext] Henning Schild wrote:
> Am Thu, 17 Jan 2019 17:04:20 +0100
> schrieb Henning Schild <henning.schild@siemens.com>:
> 
>> From: Henning Schild <henning.schild@siemens.com>
>>
>> This series includes support for fetching upstream sources with
>> "apt-get source". This will make sure we fetch exactly what matches
>> out distro, without rewriting debian fetch/unpack logic.
>> I did consider implementing it as an "apt://" extension to the regular
>> fetcher but decided against that. You have to set SRC_APT and
>> effectively pass arguement to apt-get. That fetcher can only work in
>> packages and depends on buildchroot and mounting, so it can not be
>> part of the general fetcher. But maybe the general fetcher could
>> ignore "apt://" lines and this task will ignore anything but "apt://"
>> so we can still use SRC_URI instead of SRC_APT.
> 
> Filtering all the "apt://" entries out does not really work. There is a
> call to bb.fetch.get_checksum_file_list where i can not filter, at
> least i did not yet figure out how.
> 
> And it seems that the fetcher can not be extended with a new protocol
> without touching the bitbake core. Otherwise an empty implementation of
> "apt://" would have been the trick to let debian do it in a completely
> different step.
> Not sure it is worth forcing it into SRC_URI, we are in two worlds
> anyways, no need to pretend otherwise.

Given that upstream bitbake supports rpm fetching and even unpacking, that might 
be worth to explore. At least for later, no need to delay moving with this 
approach forward first.

I wonder, though, if we cannot extend BB via some plugin or library. It seems we 
just need to append our own fetcher to bb.fetch2.methods...

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH 6/7] local.conf: remove example-hello from the default build

[Maxim Yu. Osipov]

On 1/28/19 6:08 PM, Jan Kiszka wrote:
> On 17.01.19 17:04, [ext] Henning Schild wrote:
>> From: Henning Schild <henning.schild@siemens.com>
>>
>> As reported on the mailinglist, this recipe conflicts with "hello" from
>> upstream debian. This commit should be reverted once the conflict is
>> resolved.
> 
> I think the resolution is merged already, no?

Yes, it is merged (example-hello was renamed to hello-isar).

Maxim.

> Jan
> 
>>
>> Signed-off-by: Henning Schild <henning.schild@siemens.com>
>> ---
>>   meta-isar/conf/local.conf.sample | 2 +-
>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/meta-isar/conf/local.conf.sample 
>> b/meta-isar/conf/local.conf.sample
>> index fe81f20..6ed4caa 100644
>> --- a/meta-isar/conf/local.conf.sample
>> +++ b/meta-isar/conf/local.conf.sample
>> @@ -161,7 +161,7 @@ CONF_VERSION = "1"
>>   #
>>   # The default list of extra packages to be installed.
>> -IMAGE_INSTALL = "example-hello example-raw 
>> example-module-${KERNEL_NAME} enable-fsck"
>> +IMAGE_INSTALL = "example-raw example-module-${KERNEL_NAME} enable-fsck"
>>   #
>>   # Enable cross-compilation support
>>
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: [PATCH 7/7] local.conf: enable rebuilding "hello" for all distros

[Maxim Yu. Osipov]

Should I drop this patch (with previous one) from the series?

Maxim.

On 1/17/19 5:04 PM, Henning Schild wrote:
> From: Henning Schild <henning.schild@siemens.com>
> 
> This commit includes the rebuild example in our default builds and
> therefore into CI.
> 
> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> ---
>   meta-isar/conf/local.conf.sample | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
> index 6ed4caa..bdf889b 100644
> --- a/meta-isar/conf/local.conf.sample
> +++ b/meta-isar/conf/local.conf.sample
> @@ -161,7 +161,7 @@ CONF_VERSION = "1"
>   
>   #
>   # The default list of extra packages to be installed.
> -IMAGE_INSTALL = "example-raw example-module-${KERNEL_NAME} enable-fsck"
> +IMAGE_INSTALL = "example-raw example-module-${KERNEL_NAME} enable-fsck hello"
>   
>   #
>   # Enable cross-compilation support
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: [PATCH 0/7] "apt-get source" fetch/unpack support

[Maxim Yu. Osipov]

I've rebased the patchset against current 'next' (I've dropped patch #6 
from the series) and ran the "quick" test (with cross-compilation) - 
'ci_build.sh -q -f'. build of 'hello' package failed:
http://isar-build.org:8080/job/isar_mosipov_develop/14/console

FYI: native build ('ci_build.sh -q') passed OK:
http://isar-build.org:8080/job/isar_mosipov_next/133/console

Maxim.

On 1/28/19 5:57 PM, Henning Schild wrote:
> ping
> 
> Am Thu, 17 Jan 2019 17:04:20 +0100
> schrieb Henning Schild <henning.schild@siemens.com>:
> 
>> From: Henning Schild <henning.schild@siemens.com>
>>
>> This series includes support for fetching upstream sources with
>> "apt-get source". This will make sure we fetch exactly what matches
>> out distro, without rewriting debian fetch/unpack logic.
>> I did consider implementing it as an "apt://" extension to the regular
>> fetcher but decided against that. You have to set SRC_APT and
>> effectively pass arguement to apt-get. That fetcher can only work in
>> packages and depends on buildchroot and mounting, so it can not be
>> part of the general fetcher. But maybe the general fetcher could
>> ignore "apt://" lines and this task will ignore anything but "apt://"
>> so we can still use SRC_URI instead of SRC_APT. Let us talk about the
>> interface. The documentation is still missing, but i thing it is
>> ready for a first review.
>>
>> The idea and parts of the code came from Mentor.
>>
>> Henning Schild (7):
>>    conf: add deb-src entries to all our distro configs
>>    dpkg-base: introduce an "apt-get source" fetch/unpack step
>>    meta: move debianization code into a class and into dpkg-base
>>    debianize: allow changlog version change
>>    meta-isar/recipes-app: add upstream hello rebuild example
>>    local.conf: remove example-hello from the default build
>>    local.conf: enable rebuilding "hello" for all distros
>>
>>   meta-isar/conf/distro/debian-buster.list      |  3 +
>>   meta-isar/conf/distro/debian-jessie.list      |  3 +
>>   .../conf/distro/debian-stretch-backports.list |  1 +
>>   meta-isar/conf/distro/debian-stretch.list     |  3 +
>>   meta-isar/conf/distro/raspbian-jessie.list    |  1 +
>>   meta-isar/conf/local.conf.sample              |  2 +-
>>   meta-isar/recipes-app/hello/hello.inc         | 19 +++++
>>   meta-isar/recipes-app/hello/hello_2.10.bb     |  8 ++
>>   meta-isar/recipes-app/hello/hello_2.9.bb      |  8 ++
>>   meta/classes/debianize.bbclass                | 82
>> +++++++++++++++++++ meta/classes/dpkg-base.bbclass                |
>> 20 +++++ meta/classes/dpkg-raw.bbclass                 | 76
>> ----------------- 12 files changed, 149 insertions(+), 77 deletions(-)
>>   create mode 100644 meta-isar/recipes-app/hello/hello.inc
>>   create mode 100644 meta-isar/recipes-app/hello/hello_2.10.bb
>>   create mode 100644 meta-isar/recipes-app/hello/hello_2.9.bb
>>   create mode 100644 meta/classes/debianize.bbclass
>>
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: [PATCH 0/7] "apt-get source" fetch/unpack support

[Henning Schild]

On Wed, 30 Jan 2019 07:43:24 +0100
"Maxim Yu. Osipov" <mosipov@ilbers.de> wrote:

> I've rebased the patchset against current 'next' (I've dropped patch
> #6 from the series) and ran the "quick" test (with cross-compilation)
> - 'ci_build.sh -q -f'. build of 'hello' package failed:
> http://isar-build.org:8080/job/isar_mosipov_develop/14/console

Thanks. Cant quickly spot what happened there. Seems like the compile
went fine and the execution in the testsuite says "file not found". I
am guessing the binary is there but some shared object is missing.
Might actually be a problem in upstream, but i will try to reproduce
this.

Henning

> FYI: native build ('ci_build.sh -q') passed OK:
> http://isar-build.org:8080/job/isar_mosipov_next/133/console
> 
> Maxim.
> 
> On 1/28/19 5:57 PM, Henning Schild wrote:
> > ping
> > 
> > Am Thu, 17 Jan 2019 17:04:20 +0100
> > schrieb Henning Schild <henning.schild@siemens.com>:
> >   
> >> From: Henning Schild <henning.schild@siemens.com>
> >>
> >> This series includes support for fetching upstream sources with
> >> "apt-get source". This will make sure we fetch exactly what matches
> >> out distro, without rewriting debian fetch/unpack logic.
> >> I did consider implementing it as an "apt://" extension to the
> >> regular fetcher but decided against that. You have to set SRC_APT
> >> and effectively pass arguement to apt-get. That fetcher can only
> >> work in packages and depends on buildchroot and mounting, so it
> >> can not be part of the general fetcher. But maybe the general
> >> fetcher could ignore "apt://" lines and this task will ignore
> >> anything but "apt://" so we can still use SRC_URI instead of
> >> SRC_APT. Let us talk about the interface. The documentation is
> >> still missing, but i thing it is ready for a first review.
> >>
> >> The idea and parts of the code came from Mentor.
> >>
> >> Henning Schild (7):
> >>    conf: add deb-src entries to all our distro configs
> >>    dpkg-base: introduce an "apt-get source" fetch/unpack step
> >>    meta: move debianization code into a class and into dpkg-base
> >>    debianize: allow changlog version change
> >>    meta-isar/recipes-app: add upstream hello rebuild example
> >>    local.conf: remove example-hello from the default build
> >>    local.conf: enable rebuilding "hello" for all distros
> >>
> >>   meta-isar/conf/distro/debian-buster.list      |  3 +
> >>   meta-isar/conf/distro/debian-jessie.list      |  3 +
> >>   .../conf/distro/debian-stretch-backports.list |  1 +
> >>   meta-isar/conf/distro/debian-stretch.list     |  3 +
> >>   meta-isar/conf/distro/raspbian-jessie.list    |  1 +
> >>   meta-isar/conf/local.conf.sample              |  2 +-
> >>   meta-isar/recipes-app/hello/hello.inc         | 19 +++++
> >>   meta-isar/recipes-app/hello/hello_2.10.bb     |  8 ++
> >>   meta-isar/recipes-app/hello/hello_2.9.bb      |  8 ++
> >>   meta/classes/debianize.bbclass                | 82
> >> +++++++++++++++++++ meta/classes/dpkg-base.bbclass                |
> >> 20 +++++ meta/classes/dpkg-raw.bbclass                 | 76
> >> ----------------- 12 files changed, 149 insertions(+), 77
> >> deletions(-) create mode 100644
> >> meta-isar/recipes-app/hello/hello.inc create mode 100644
> >> meta-isar/recipes-app/hello/hello_2.10.bb create mode 100644
> >> meta-isar/recipes-app/hello/hello_2.9.bb create mode 100644
> >> meta/classes/debianize.bbclass 
> >   
> 
> 

Re: [PATCH 0/7] "apt-get source" fetch/unpack support

[Henning Schild]

On Wed, 30 Jan 2019 07:43:24 +0100
"Maxim Yu. Osipov" <mosipov@ilbers.de> wrote:

> I've rebased the patchset against current 'next' (I've dropped patch
> #6 from the series) and ran the "quick" test (with cross-compilation)
> - 'ci_build.sh -q -f'. build of 'hello' package failed:
> http://isar-build.org:8080/job/isar_mosipov_develop/14/console
> 
> FYI: native build ('ci_build.sh -q') passed OK:
> http://isar-build.org:8080/job/isar_mosipov_next/133/console

Pretty obvious actually. I can not execute an arm64 binary in an amd64
chroot, so testing needs to be switched off.
This is technically another upstream bug in this package, but hey cross
building is not a debian feature so we are not in the position to
complain.
I will send an update for the recipe.

I actually have a patch that disabled testing for _all_ cross builds in
build.sh ... but decided to not post it. There might actually be some
packages where testing will still work ...

Henning

> Maxim.
> 
> On 1/28/19 5:57 PM, Henning Schild wrote:
> > ping
> > 
> > Am Thu, 17 Jan 2019 17:04:20 +0100
> > schrieb Henning Schild <henning.schild@siemens.com>:
> >   
> >> From: Henning Schild <henning.schild@siemens.com>
> >>
> >> This series includes support for fetching upstream sources with
> >> "apt-get source". This will make sure we fetch exactly what matches
> >> out distro, without rewriting debian fetch/unpack logic.
> >> I did consider implementing it as an "apt://" extension to the
> >> regular fetcher but decided against that. You have to set SRC_APT
> >> and effectively pass arguement to apt-get. That fetcher can only
> >> work in packages and depends on buildchroot and mounting, so it
> >> can not be part of the general fetcher. But maybe the general
> >> fetcher could ignore "apt://" lines and this task will ignore
> >> anything but "apt://" so we can still use SRC_URI instead of
> >> SRC_APT. Let us talk about the interface. The documentation is
> >> still missing, but i thing it is ready for a first review.
> >>
> >> The idea and parts of the code came from Mentor.
> >>
> >> Henning Schild (7):
> >>    conf: add deb-src entries to all our distro configs
> >>    dpkg-base: introduce an "apt-get source" fetch/unpack step
> >>    meta: move debianization code into a class and into dpkg-base
> >>    debianize: allow changlog version change
> >>    meta-isar/recipes-app: add upstream hello rebuild example
> >>    local.conf: remove example-hello from the default build
> >>    local.conf: enable rebuilding "hello" for all distros
> >>
> >>   meta-isar/conf/distro/debian-buster.list      |  3 +
> >>   meta-isar/conf/distro/debian-jessie.list      |  3 +
> >>   .../conf/distro/debian-stretch-backports.list |  1 +
> >>   meta-isar/conf/distro/debian-stretch.list     |  3 +
> >>   meta-isar/conf/distro/raspbian-jessie.list    |  1 +
> >>   meta-isar/conf/local.conf.sample              |  2 +-
> >>   meta-isar/recipes-app/hello/hello.inc         | 19 +++++
> >>   meta-isar/recipes-app/hello/hello_2.10.bb     |  8 ++
> >>   meta-isar/recipes-app/hello/hello_2.9.bb      |  8 ++
> >>   meta/classes/debianize.bbclass                | 82
> >> +++++++++++++++++++ meta/classes/dpkg-base.bbclass                |
> >> 20 +++++ meta/classes/dpkg-raw.bbclass                 | 76
> >> ----------------- 12 files changed, 149 insertions(+), 77
> >> deletions(-) create mode 100644
> >> meta-isar/recipes-app/hello/hello.inc create mode 100644
> >> meta-isar/recipes-app/hello/hello_2.10.bb create mode 100644
> >> meta-isar/recipes-app/hello/hello_2.9.bb create mode 100644
> >> meta/classes/debianize.bbclass 
> >   
> 
> 

Re: [PATCH 7/7] local.conf: enable rebuilding "hello" for all distros

[Henning Schild]

On Tue, 29 Jan 2019 12:20:02 +0100
"Maxim Yu. Osipov" <mosipov@ilbers.de> wrote:

> Should I drop this patch (with previous one) from the series?

This one should be applied/forward-ported. The previous one dropped,
not required anymore.

I think you already did the forward porting right so i will just send
an update to one patch, not the whole series.

Henning

> Maxim.
> 
> On 1/17/19 5:04 PM, Henning Schild wrote:
> > From: Henning Schild <henning.schild@siemens.com>
> > 
> > This commit includes the rebuild example in our default builds and
> > therefore into CI.
> > 
> > Signed-off-by: Henning Schild <henning.schild@siemens.com>
> > ---
> >   meta-isar/conf/local.conf.sample | 2 +-
> >   1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/meta-isar/conf/local.conf.sample
> > b/meta-isar/conf/local.conf.sample index 6ed4caa..bdf889b 100644
> > --- a/meta-isar/conf/local.conf.sample
> > +++ b/meta-isar/conf/local.conf.sample
> > @@ -161,7 +161,7 @@ CONF_VERSION = "1"
> >   
> >   #
> >   # The default list of extra packages to be installed.
> > -IMAGE_INSTALL = "example-raw example-module-${KERNEL_NAME}
> > enable-fsck" +IMAGE_INSTALL = "example-raw
> > example-module-${KERNEL_NAME} enable-fsck hello" 
> >   #
> >   # Enable cross-compilation support
> >   
> 
> 

[PATCHv2 5/7] meta-isar/recipes-app: add upstream hello rebuild example

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

This commit introduces a rebuild example of an upstream package. It also
shows how to deal with multiple versions of such a package, depending on
your distro. And in fact we actually have to patch it to rebuild it,
because its build-deps are wrong.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta-isar/recipes-app/hello/hello.inc     | 23 +++++++++++++++++++++++
 meta-isar/recipes-app/hello/hello_2.10.bb |  8 ++++++++
 meta-isar/recipes-app/hello/hello_2.9.bb  |  8 ++++++++
 3 files changed, 39 insertions(+)
 create mode 100644 meta-isar/recipes-app/hello/hello.inc
 create mode 100644 meta-isar/recipes-app/hello/hello_2.10.bb
 create mode 100644 meta-isar/recipes-app/hello/hello_2.9.bb

diff --git a/meta-isar/recipes-app/hello/hello.inc b/meta-isar/recipes-app/hello/hello.inc
new file mode 100644
index 0000000..3976b15
--- /dev/null
+++ b/meta-isar/recipes-app/hello/hello.inc
@@ -0,0 +1,23 @@
+# This software is a part of ISAR.
+
+inherit dpkg
+
+# this will fetch and unpack the sources from upstream debian
+SRC_APT = "${PN}"
+
+MAINTAINER = "Your name here <you@domain.com>"
+CHANGELOG_V = "${PV}-99+isar"
+
+do_prepare_build() {
+	deb_add_changelog
+	# this seems to be a build dep missing in the upstream control file
+	if ! grep texinfo ${S}/debian/control; then
+		sed -i -e 's/Build-Depends:/Build-Depends: texinfo,/g' ${S}/debian/control
+	fi
+}
+
+dpkg_runbuild_prepend() {
+	if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
+		export DEB_BUILD_OPTIONS="nocheck"
+	fi
+}
diff --git a/meta-isar/recipes-app/hello/hello_2.10.bb b/meta-isar/recipes-app/hello/hello_2.10.bb
new file mode 100644
index 0000000..1598565
--- /dev/null
+++ b/meta-isar/recipes-app/hello/hello_2.10.bb
@@ -0,0 +1,8 @@
+# Example recipe to rebuild a debian source package
+#
+# This software is a part of ISAR.
+
+require hello.inc
+
+DEFAULT_PREFERENCE_debian-buster = "1"
+DEFAULT_PREFERENCE_debian-stretch = "1"
diff --git a/meta-isar/recipes-app/hello/hello_2.9.bb b/meta-isar/recipes-app/hello/hello_2.9.bb
new file mode 100644
index 0000000..b5d5b2e
--- /dev/null
+++ b/meta-isar/recipes-app/hello/hello_2.9.bb
@@ -0,0 +1,8 @@
+# Example recipe to rebuild a debian source package
+#
+# This software is a part of ISAR.
+
+require hello.inc
+
+DEFAULT_PREFERENCE_debian-jessie = "1"
+DEFAULT_PREFERENCE_raspbian-jessie = "1"
-- 
2.19.2

Re: [PATCHv2 5/7] meta-isar/recipes-app: add upstream hello rebuild example

[Henning Schild]

On Wed, 30 Jan 2019 14:06:23 +0100
Henning Schild <henning.schild@siemens.com> wrote:

> From: Henning Schild <henning.schild@siemens.com>
> 
> This commit introduces a rebuild example of an upstream package. It
> also shows how to deal with multiple versions of such a package,
> depending on your distro. And in fact we actually have to patch it to
> rebuild it, because its build-deps are wrong.
> 
> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> ---
>  meta-isar/recipes-app/hello/hello.inc     | 23
> +++++++++++++++++++++++ meta-isar/recipes-app/hello/hello_2.10.bb |
> 8 ++++++++ meta-isar/recipes-app/hello/hello_2.9.bb  |  8 ++++++++
>  3 files changed, 39 insertions(+)
>  create mode 100644 meta-isar/recipes-app/hello/hello.inc
>  create mode 100644 meta-isar/recipes-app/hello/hello_2.10.bb
>  create mode 100644 meta-isar/recipes-app/hello/hello_2.9.bb
> 
> diff --git a/meta-isar/recipes-app/hello/hello.inc
> b/meta-isar/recipes-app/hello/hello.inc new file mode 100644
> index 0000000..3976b15
> --- /dev/null
> +++ b/meta-isar/recipes-app/hello/hello.inc
> @@ -0,0 +1,23 @@
> +# This software is a part of ISAR.
> +
> +inherit dpkg
> +
> +# this will fetch and unpack the sources from upstream debian
> +SRC_APT = "${PN}"
> +
> +MAINTAINER = "Your name here <you@domain.com>"
> +CHANGELOG_V = "${PV}-99+isar"
> +
> +do_prepare_build() {
> +	deb_add_changelog
> +	# this seems to be a build dep missing in the upstream
> control file
> +	if ! grep texinfo ${S}/debian/control; then
> +		sed -i -e 's/Build-Depends:/Build-Depends:
> texinfo,/g' ${S}/debian/control
> +	fi
> +}
> +
> +dpkg_runbuild_prepend() {
> +	if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
> +		export DEB_BUILD_OPTIONS="nocheck"
> +	fi
> +}

This prepend is new in v2, it disables testing in the cross build case.

Henning

> diff --git a/meta-isar/recipes-app/hello/hello_2.10.bb
> b/meta-isar/recipes-app/hello/hello_2.10.bb new file mode 100644
> index 0000000..1598565
> --- /dev/null
> +++ b/meta-isar/recipes-app/hello/hello_2.10.bb
> @@ -0,0 +1,8 @@
> +# Example recipe to rebuild a debian source package
> +#
> +# This software is a part of ISAR.
> +
> +require hello.inc
> +
> +DEFAULT_PREFERENCE_debian-buster = "1"
> +DEFAULT_PREFERENCE_debian-stretch = "1"
> diff --git a/meta-isar/recipes-app/hello/hello_2.9.bb
> b/meta-isar/recipes-app/hello/hello_2.9.bb new file mode 100644
> index 0000000..b5d5b2e
> --- /dev/null
> +++ b/meta-isar/recipes-app/hello/hello_2.9.bb
> @@ -0,0 +1,8 @@
> +# Example recipe to rebuild a debian source package
> +#
> +# This software is a part of ISAR.
> +
> +require hello.inc
> +
> +DEFAULT_PREFERENCE_debian-jessie = "1"
> +DEFAULT_PREFERENCE_raspbian-jessie = "1"

[PATCHv2 2/7] dpkg-base: introduce an "apt-get source" fetch/unpack step

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

This creates a new task to fetch/unpack debian sources from debian
source mirrors. This is done using "apt-get source" and the new variable
APT_SRC to control the arguments.

An example where a original debian package gets rebuild will follow in a
later commit.

based on original patch by Cedric Hombourger <Cedric_Hombourger@mentor.com>

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta/classes/dpkg-base.bbclass | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index f1b127c..2f324f9 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -20,6 +20,25 @@ do_adjust_git[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 inherit patch
 addtask patch after do_adjust_git before do_build
 
+SRC_APT ?= ""
+
+do_apt_fetch[depends] = "buildchroot-target:do_build"
+
+do_apt_fetch() {
+	if [ -z "${@d.getVar("SRC_APT", True).strip()}" ]; then
+		exit
+	fi
+	dpkg_do_mounts
+	E="${@ bb.utils.export_proxies(d)}"
+	sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
+		sh -c 'cd ${PP} && apt-get -y source ${SRC_APT}'
+	dpkg_undo_mounts
+}
+
+addtask apt_fetch after do_unpack before do_patch
+do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
+do_apt_fetch[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
+
 def get_package_srcdir(d):
     s = d.getVar("S", True)
     workdir = d.getVar("WORKDIR", True)
-- 
2.19.2

Re: [PATCHv2 2/7] dpkg-base: introduce an "apt-get source" fetch/unpack step

[Henning Schild]

On Wed, 30 Jan 2019 14:57:38 +0100
Henning Schild <henning.schild@siemens.com> wrote:

> From: Henning Schild <henning.schild@siemens.com>
> 
> This creates a new task to fetch/unpack debian sources from debian
> source mirrors. This is done using "apt-get source" and the new
> variable APT_SRC to control the arguments.
> 
> An example where a original debian package gets rebuild will follow
> in a later commit.
> 
> based on original patch by Cedric Hombourger
> <Cedric_Hombourger@mentor.com>

Changed commit comment to contain the above instead of a
"Signed-off-by".

Henning

> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> ---
>  meta/classes/dpkg-base.bbclass | 19 +++++++++++++++++++
>  1 file changed, 19 insertions(+)
> 
> diff --git a/meta/classes/dpkg-base.bbclass
> b/meta/classes/dpkg-base.bbclass index f1b127c..2f324f9 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -20,6 +20,25 @@ do_adjust_git[stamp-extra-info] =
> "${DISTRO}-${DISTRO_ARCH}" inherit patch
>  addtask patch after do_adjust_git before do_build
>  
> +SRC_APT ?= ""
> +
> +do_apt_fetch[depends] = "buildchroot-target:do_build"
> +
> +do_apt_fetch() {
> +	if [ -z "${@d.getVar("SRC_APT", True).strip()}" ]; then
> +		exit
> +	fi
> +	dpkg_do_mounts
> +	E="${@ bb.utils.export_proxies(d)}"
> +	sudo -E chroot --userspec=$( id -u ):$( id -g )
> ${BUILDCHROOT_DIR} \
> +		sh -c 'cd ${PP} && apt-get -y source ${SRC_APT}'
> +	dpkg_undo_mounts
> +}
> +
> +addtask apt_fetch after do_unpack before do_patch
> +do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
> +do_apt_fetch[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> +
>  def get_package_srcdir(d):
>      s = d.getVar("S", True)
>      workdir = d.getVar("WORKDIR", True)