[PATCH 0/7] "apt-get source" fetch/unpack support

[PATCH 0/7] "apt-get source" fetch/unpack support

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

This series includes support for fetching upstream sources with "apt-get
source". This will make sure we fetch exactly what matches out distro,
without rewriting debian fetch/unpack logic.
I did consider implementing it as an "apt://" extension to the regular
fetcher but decided against that. You have to set SRC_APT and
effectively pass arguement to apt-get. That fetcher can only work in
packages and depends on buildchroot and mounting, so it can not be part
of the general fetcher. But maybe the general fetcher could ignore
"apt://" lines and this task will ignore anything but "apt://" so we can
still use SRC_URI instead of SRC_APT. Let us talk about the interface.
The documentation is still missing, but i thing it is ready for a first
review.

The idea and parts of the code came from Mentor.

Henning Schild (7):
  conf: add deb-src entries to all our distro configs
  dpkg-base: introduce an "apt-get source" fetch/unpack step
  meta: move debianization code into a class and into dpkg-base
  debianize: allow changlog version change
  meta-isar/recipes-app: add upstream hello rebuild example
  local.conf: remove example-hello from the default build
  local.conf: enable rebuilding "hello" for all distros

 meta-isar/conf/distro/debian-buster.list      |  3 +
 meta-isar/conf/distro/debian-jessie.list      |  3 +
 .../conf/distro/debian-stretch-backports.list |  1 +
 meta-isar/conf/distro/debian-stretch.list     |  3 +
 meta-isar/conf/distro/raspbian-jessie.list    |  1 +
 meta-isar/conf/local.conf.sample              |  2 +-
 meta-isar/recipes-app/hello/hello.inc         | 19 +++++
 meta-isar/recipes-app/hello/hello_2.10.bb     |  8 ++
 meta-isar/recipes-app/hello/hello_2.9.bb      |  8 ++
 meta/classes/debianize.bbclass                | 82 +++++++++++++++++++
 meta/classes/dpkg-base.bbclass                | 20 +++++
 meta/classes/dpkg-raw.bbclass                 | 76 -----------------
 12 files changed, 149 insertions(+), 77 deletions(-)
 create mode 100644 meta-isar/recipes-app/hello/hello.inc
 create mode 100644 meta-isar/recipes-app/hello/hello_2.10.bb
 create mode 100644 meta-isar/recipes-app/hello/hello_2.9.bb
 create mode 100644 meta/classes/debianize.bbclass

-- 
2.19.2

[PATCH 1/7] conf: add deb-src entries to all our distro configs

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

That will allow us to use "apt-get source" to fetch debian sources from
the mirrors we also get the binary packages from.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 meta-isar/conf/distro/debian-buster.list            | 3 +++
 meta-isar/conf/distro/debian-jessie.list            | 3 +++
 meta-isar/conf/distro/debian-stretch-backports.list | 1 +
 meta-isar/conf/distro/debian-stretch.list           | 3 +++
 meta-isar/conf/distro/raspbian-jessie.list          | 1 +
 5 files changed, 11 insertions(+)

diff --git a/meta-isar/conf/distro/debian-buster.list b/meta-isar/conf/distro/debian-buster.list
index 18311d8..1e38f1a 100644
--- a/meta-isar/conf/distro/debian-buster.list
+++ b/meta-isar/conf/distro/debian-buster.list
@@ -1,3 +1,6 @@
 deb	http://ftp.de.debian.org/debian buster	main contrib non-free
+deb-src	http://ftp.de.debian.org/debian buster	main contrib non-free
 deb	http://ftp.de.debian.org/debian buster-updates	main contrib non-free
+deb-src	http://ftp.de.debian.org/debian buster-updates	main contrib non-free
 deb	http://security.debian.org	buster/updates	main contrib non-free
+deb-src	http://security.debian.org	buster/updates	main contrib non-free
diff --git a/meta-isar/conf/distro/debian-jessie.list b/meta-isar/conf/distro/debian-jessie.list
index be46a57..5143f4d 100644
--- a/meta-isar/conf/distro/debian-jessie.list
+++ b/meta-isar/conf/distro/debian-jessie.list
@@ -1,3 +1,6 @@
 deb	http://ftp.de.debian.org/debian	jessie	main contrib non-free
+deb-src	http://ftp.de.debian.org/debian	jessie	main contrib non-free
 deb	http://ftp.de.debian.org/debian jessie-updates	main contrib non-free
+deb-src	http://ftp.de.debian.org/debian jessie-updates	main contrib non-free
 deb	http://security.debian.org	jessie/updates	main contrib non-free
+deb-src	http://security.debian.org	jessie/updates	main contrib non-free
diff --git a/meta-isar/conf/distro/debian-stretch-backports.list b/meta-isar/conf/distro/debian-stretch-backports.list
index 727a835..f606be5 100644
--- a/meta-isar/conf/distro/debian-stretch-backports.list
+++ b/meta-isar/conf/distro/debian-stretch-backports.list
@@ -1 +1,2 @@
 deb	http://ftp.de.debian.org/debian stretch-backports main
+deb-src	http://ftp.de.debian.org/debian stretch-backports main
diff --git a/meta-isar/conf/distro/debian-stretch.list b/meta-isar/conf/distro/debian-stretch.list
index bfa5cf9..d4f3de0 100644
--- a/meta-isar/conf/distro/debian-stretch.list
+++ b/meta-isar/conf/distro/debian-stretch.list
@@ -1,3 +1,6 @@
 deb	http://ftp.de.debian.org/debian stretch	main contrib non-free
+deb-src	http://ftp.de.debian.org/debian stretch	main contrib non-free
 deb	http://ftp.de.debian.org/debian stretch-updates	main contrib non-free
+deb-src	http://ftp.de.debian.org/debian stretch-updates	main contrib non-free
 deb	http://security.debian.org	stretch/updates	main contrib non-free
+deb-src	http://security.debian.org	stretch/updates	main contrib non-free
diff --git a/meta-isar/conf/distro/raspbian-jessie.list b/meta-isar/conf/distro/raspbian-jessie.list
index 792f9c6..0a5af41 100644
--- a/meta-isar/conf/distro/raspbian-jessie.list
+++ b/meta-isar/conf/distro/raspbian-jessie.list
@@ -1 +1,2 @@
 deb	http://archive.raspbian.org/raspbian	jessie	main contrib non-free firmware
+deb-src	http://archive.raspbian.org/raspbian	jessie	main contrib non-free firmware
-- 
2.19.2

Re: [PATCH 0/7] "apt-get source" fetch/unpack support

[Henning Schild]

Please ignore this thread, git mail sending aborted because of a
malformed email address.

Henning

Am Thu, 17 Jan 2019 17:02:11 +0100
schrieb Henning Schild <henning.schild@siemens.com>:

> From: Henning Schild <henning.schild@siemens.com>
> 
> This series includes support for fetching upstream sources with
> "apt-get source". This will make sure we fetch exactly what matches
> out distro, without rewriting debian fetch/unpack logic.
> I did consider implementing it as an "apt://" extension to the regular
> fetcher but decided against that. You have to set SRC_APT and
> effectively pass arguement to apt-get. That fetcher can only work in
> packages and depends on buildchroot and mounting, so it can not be
> part of the general fetcher. But maybe the general fetcher could
> ignore "apt://" lines and this task will ignore anything but "apt://"
> so we can still use SRC_URI instead of SRC_APT. Let us talk about the
> interface. The documentation is still missing, but i thing it is
> ready for a first review.
> 
> The idea and parts of the code came from Mentor.
> 
> Henning Schild (7):
>   conf: add deb-src entries to all our distro configs
>   dpkg-base: introduce an "apt-get source" fetch/unpack step
>   meta: move debianization code into a class and into dpkg-base
>   debianize: allow changlog version change
>   meta-isar/recipes-app: add upstream hello rebuild example
>   local.conf: remove example-hello from the default build
>   local.conf: enable rebuilding "hello" for all distros
> 
>  meta-isar/conf/distro/debian-buster.list      |  3 +
>  meta-isar/conf/distro/debian-jessie.list      |  3 +
>  .../conf/distro/debian-stretch-backports.list |  1 +
>  meta-isar/conf/distro/debian-stretch.list     |  3 +
>  meta-isar/conf/distro/raspbian-jessie.list    |  1 +
>  meta-isar/conf/local.conf.sample              |  2 +-
>  meta-isar/recipes-app/hello/hello.inc         | 19 +++++
>  meta-isar/recipes-app/hello/hello_2.10.bb     |  8 ++
>  meta-isar/recipes-app/hello/hello_2.9.bb      |  8 ++
>  meta/classes/debianize.bbclass                | 82
> +++++++++++++++++++ meta/classes/dpkg-base.bbclass                |
> 20 +++++ meta/classes/dpkg-raw.bbclass                 | 76
> ----------------- 12 files changed, 149 insertions(+), 77 deletions(-)
>  create mode 100644 meta-isar/recipes-app/hello/hello.inc
>  create mode 100644 meta-isar/recipes-app/hello/hello_2.10.bb
>  create mode 100644 meta-isar/recipes-app/hello/hello_2.9.bb
>  create mode 100644 meta/classes/debianize.bbclass
> 

Re: [PATCH 0/7] "apt-get source" fetch/unpack support

[Henning Schild]

On Wed, 30 Jan 2019 07:43:24 +0100
"Maxim Yu. Osipov" <mosipov@ilbers.de> wrote:

> I've rebased the patchset against current 'next' (I've dropped patch
> #6 from the series) and ran the "quick" test (with cross-compilation)
> - 'ci_build.sh -q -f'. build of 'hello' package failed:
> http://isar-build.org:8080/job/isar_mosipov_develop/14/console
> 
> FYI: native build ('ci_build.sh -q') passed OK:
> http://isar-build.org:8080/job/isar_mosipov_next/133/console

Pretty obvious actually. I can not execute an arm64 binary in an amd64
chroot, so testing needs to be switched off.
This is technically another upstream bug in this package, but hey cross
building is not a debian feature so we are not in the position to
complain.
I will send an update for the recipe.

I actually have a patch that disabled testing for _all_ cross builds in
build.sh ... but decided to not post it. There might actually be some
packages where testing will still work ...

Henning

> Maxim.
> 
> On 1/28/19 5:57 PM, Henning Schild wrote:
> > ping
> > 
> > Am Thu, 17 Jan 2019 17:04:20 +0100
> > schrieb Henning Schild <henning.schild@siemens.com>:
> >   
> >> From: Henning Schild <henning.schild@siemens.com>
> >>
> >> This series includes support for fetching upstream sources with
> >> "apt-get source". This will make sure we fetch exactly what matches
> >> out distro, without rewriting debian fetch/unpack logic.
> >> I did consider implementing it as an "apt://" extension to the
> >> regular fetcher but decided against that. You have to set SRC_APT
> >> and effectively pass arguement to apt-get. That fetcher can only
> >> work in packages and depends on buildchroot and mounting, so it
> >> can not be part of the general fetcher. But maybe the general
> >> fetcher could ignore "apt://" lines and this task will ignore
> >> anything but "apt://" so we can still use SRC_URI instead of
> >> SRC_APT. Let us talk about the interface. The documentation is
> >> still missing, but i thing it is ready for a first review.
> >>
> >> The idea and parts of the code came from Mentor.
> >>
> >> Henning Schild (7):
> >>    conf: add deb-src entries to all our distro configs
> >>    dpkg-base: introduce an "apt-get source" fetch/unpack step
> >>    meta: move debianization code into a class and into dpkg-base
> >>    debianize: allow changlog version change
> >>    meta-isar/recipes-app: add upstream hello rebuild example
> >>    local.conf: remove example-hello from the default build
> >>    local.conf: enable rebuilding "hello" for all distros
> >>
> >>   meta-isar/conf/distro/debian-buster.list      |  3 +
> >>   meta-isar/conf/distro/debian-jessie.list      |  3 +
> >>   .../conf/distro/debian-stretch-backports.list |  1 +
> >>   meta-isar/conf/distro/debian-stretch.list     |  3 +
> >>   meta-isar/conf/distro/raspbian-jessie.list    |  1 +
> >>   meta-isar/conf/local.conf.sample              |  2 +-
> >>   meta-isar/recipes-app/hello/hello.inc         | 19 +++++
> >>   meta-isar/recipes-app/hello/hello_2.10.bb     |  8 ++
> >>   meta-isar/recipes-app/hello/hello_2.9.bb      |  8 ++
> >>   meta/classes/debianize.bbclass                | 82
> >> +++++++++++++++++++ meta/classes/dpkg-base.bbclass                |
> >> 20 +++++ meta/classes/dpkg-raw.bbclass                 | 76
> >> ----------------- 12 files changed, 149 insertions(+), 77
> >> deletions(-) create mode 100644
> >> meta-isar/recipes-app/hello/hello.inc create mode 100644
> >> meta-isar/recipes-app/hello/hello_2.10.bb create mode 100644
> >> meta-isar/recipes-app/hello/hello_2.9.bb create mode 100644
> >> meta/classes/debianize.bbclass 
> >   
> 
> 

Re: [PATCH 0/7] "apt-get source" fetch/unpack support

[Henning Schild]

On Wed, 30 Jan 2019 07:43:24 +0100
"Maxim Yu. Osipov" <mosipov@ilbers.de> wrote:

> I've rebased the patchset against current 'next' (I've dropped patch
> #6 from the series) and ran the "quick" test (with cross-compilation)
> - 'ci_build.sh -q -f'. build of 'hello' package failed:
> http://isar-build.org:8080/job/isar_mosipov_develop/14/console

Thanks. Cant quickly spot what happened there. Seems like the compile
went fine and the execution in the testsuite says "file not found". I
am guessing the binary is there but some shared object is missing.
Might actually be a problem in upstream, but i will try to reproduce
this.

Henning

> FYI: native build ('ci_build.sh -q') passed OK:
> http://isar-build.org:8080/job/isar_mosipov_next/133/console
> 
> Maxim.
> 
> On 1/28/19 5:57 PM, Henning Schild wrote:
> > ping
> > 
> > Am Thu, 17 Jan 2019 17:04:20 +0100
> > schrieb Henning Schild <henning.schild@siemens.com>:
> >   
> >> From: Henning Schild <henning.schild@siemens.com>
> >>
> >> This series includes support for fetching upstream sources with
> >> "apt-get source". This will make sure we fetch exactly what matches
> >> out distro, without rewriting debian fetch/unpack logic.
> >> I did consider implementing it as an "apt://" extension to the
> >> regular fetcher but decided against that. You have to set SRC_APT
> >> and effectively pass arguement to apt-get. That fetcher can only
> >> work in packages and depends on buildchroot and mounting, so it
> >> can not be part of the general fetcher. But maybe the general
> >> fetcher could ignore "apt://" lines and this task will ignore
> >> anything but "apt://" so we can still use SRC_URI instead of
> >> SRC_APT. Let us talk about the interface. The documentation is
> >> still missing, but i thing it is ready for a first review.
> >>
> >> The idea and parts of the code came from Mentor.
> >>
> >> Henning Schild (7):
> >>    conf: add deb-src entries to all our distro configs
> >>    dpkg-base: introduce an "apt-get source" fetch/unpack step
> >>    meta: move debianization code into a class and into dpkg-base
> >>    debianize: allow changlog version change
> >>    meta-isar/recipes-app: add upstream hello rebuild example
> >>    local.conf: remove example-hello from the default build
> >>    local.conf: enable rebuilding "hello" for all distros
> >>
> >>   meta-isar/conf/distro/debian-buster.list      |  3 +
> >>   meta-isar/conf/distro/debian-jessie.list      |  3 +
> >>   .../conf/distro/debian-stretch-backports.list |  1 +
> >>   meta-isar/conf/distro/debian-stretch.list     |  3 +
> >>   meta-isar/conf/distro/raspbian-jessie.list    |  1 +
> >>   meta-isar/conf/local.conf.sample              |  2 +-
> >>   meta-isar/recipes-app/hello/hello.inc         | 19 +++++
> >>   meta-isar/recipes-app/hello/hello_2.10.bb     |  8 ++
> >>   meta-isar/recipes-app/hello/hello_2.9.bb      |  8 ++
> >>   meta/classes/debianize.bbclass                | 82
> >> +++++++++++++++++++ meta/classes/dpkg-base.bbclass                |
> >> 20 +++++ meta/classes/dpkg-raw.bbclass                 | 76
> >> ----------------- 12 files changed, 149 insertions(+), 77
> >> deletions(-) create mode 100644
> >> meta-isar/recipes-app/hello/hello.inc create mode 100644
> >> meta-isar/recipes-app/hello/hello_2.10.bb create mode 100644
> >> meta-isar/recipes-app/hello/hello_2.9.bb create mode 100644
> >> meta/classes/debianize.bbclass 
> >   
> 
> 

Re: [PATCH 0/7] "apt-get source" fetch/unpack support

[Maxim Yu. Osipov]

I've rebased the patchset against current 'next' (I've dropped patch #6 
from the series) and ran the "quick" test (with cross-compilation) - 
'ci_build.sh -q -f'. build of 'hello' package failed:
http://isar-build.org:8080/job/isar_mosipov_develop/14/console

FYI: native build ('ci_build.sh -q') passed OK:
http://isar-build.org:8080/job/isar_mosipov_next/133/console

Maxim.

On 1/28/19 5:57 PM, Henning Schild wrote:
> ping
> 
> Am Thu, 17 Jan 2019 17:04:20 +0100
> schrieb Henning Schild <henning.schild@siemens.com>:
> 
>> From: Henning Schild <henning.schild@siemens.com>
>>
>> This series includes support for fetching upstream sources with
>> "apt-get source". This will make sure we fetch exactly what matches
>> out distro, without rewriting debian fetch/unpack logic.
>> I did consider implementing it as an "apt://" extension to the regular
>> fetcher but decided against that. You have to set SRC_APT and
>> effectively pass arguement to apt-get. That fetcher can only work in
>> packages and depends on buildchroot and mounting, so it can not be
>> part of the general fetcher. But maybe the general fetcher could
>> ignore "apt://" lines and this task will ignore anything but "apt://"
>> so we can still use SRC_URI instead of SRC_APT. Let us talk about the
>> interface. The documentation is still missing, but i thing it is
>> ready for a first review.
>>
>> The idea and parts of the code came from Mentor.
>>
>> Henning Schild (7):
>>    conf: add deb-src entries to all our distro configs
>>    dpkg-base: introduce an "apt-get source" fetch/unpack step
>>    meta: move debianization code into a class and into dpkg-base
>>    debianize: allow changlog version change
>>    meta-isar/recipes-app: add upstream hello rebuild example
>>    local.conf: remove example-hello from the default build
>>    local.conf: enable rebuilding "hello" for all distros
>>
>>   meta-isar/conf/distro/debian-buster.list      |  3 +
>>   meta-isar/conf/distro/debian-jessie.list      |  3 +
>>   .../conf/distro/debian-stretch-backports.list |  1 +
>>   meta-isar/conf/distro/debian-stretch.list     |  3 +
>>   meta-isar/conf/distro/raspbian-jessie.list    |  1 +
>>   meta-isar/conf/local.conf.sample              |  2 +-
>>   meta-isar/recipes-app/hello/hello.inc         | 19 +++++
>>   meta-isar/recipes-app/hello/hello_2.10.bb     |  8 ++
>>   meta-isar/recipes-app/hello/hello_2.9.bb      |  8 ++
>>   meta/classes/debianize.bbclass                | 82
>> +++++++++++++++++++ meta/classes/dpkg-base.bbclass                |
>> 20 +++++ meta/classes/dpkg-raw.bbclass                 | 76
>> ----------------- 12 files changed, 149 insertions(+), 77 deletions(-)
>>   create mode 100644 meta-isar/recipes-app/hello/hello.inc
>>   create mode 100644 meta-isar/recipes-app/hello/hello_2.10.bb
>>   create mode 100644 meta-isar/recipes-app/hello/hello_2.9.bb
>>   create mode 100644 meta/classes/debianize.bbclass
>>
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: [PATCH 0/7] "apt-get source" fetch/unpack support

[Jan Kiszka]

On 17.01.19 17:38, [ext] Henning Schild wrote:
> Am Thu, 17 Jan 2019 17:04:20 +0100
> schrieb Henning Schild <henning.schild@siemens.com>:
> 
>> From: Henning Schild <henning.schild@siemens.com>
>>
>> This series includes support for fetching upstream sources with
>> "apt-get source". This will make sure we fetch exactly what matches
>> out distro, without rewriting debian fetch/unpack logic.
>> I did consider implementing it as an "apt://" extension to the regular
>> fetcher but decided against that. You have to set SRC_APT and
>> effectively pass arguement to apt-get. That fetcher can only work in
>> packages and depends on buildchroot and mounting, so it can not be
>> part of the general fetcher. But maybe the general fetcher could
>> ignore "apt://" lines and this task will ignore anything but "apt://"
>> so we can still use SRC_URI instead of SRC_APT.
> 
> Filtering all the "apt://" entries out does not really work. There is a
> call to bb.fetch.get_checksum_file_list where i can not filter, at
> least i did not yet figure out how.
> 
> And it seems that the fetcher can not be extended with a new protocol
> without touching the bitbake core. Otherwise an empty implementation of
> "apt://" would have been the trick to let debian do it in a completely
> different step.
> Not sure it is worth forcing it into SRC_URI, we are in two worlds
> anyways, no need to pretend otherwise.

Given that upstream bitbake supports rpm fetching and even unpacking, that might 
be worth to explore. At least for later, no need to delay moving with this 
approach forward first.

I wonder, though, if we cannot extend BB via some plugin or library. It seems we 
just need to append our own fetcher to bb.fetch2.methods...

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH 0/7] "apt-get source" fetch/unpack support

[Henning Schild]

ping

Am Thu, 17 Jan 2019 17:04:20 +0100
schrieb Henning Schild <henning.schild@siemens.com>:

> From: Henning Schild <henning.schild@siemens.com>
> 
> This series includes support for fetching upstream sources with
> "apt-get source". This will make sure we fetch exactly what matches
> out distro, without rewriting debian fetch/unpack logic.
> I did consider implementing it as an "apt://" extension to the regular
> fetcher but decided against that. You have to set SRC_APT and
> effectively pass arguement to apt-get. That fetcher can only work in
> packages and depends on buildchroot and mounting, so it can not be
> part of the general fetcher. But maybe the general fetcher could
> ignore "apt://" lines and this task will ignore anything but "apt://"
> so we can still use SRC_URI instead of SRC_APT. Let us talk about the
> interface. The documentation is still missing, but i thing it is
> ready for a first review.
> 
> The idea and parts of the code came from Mentor.
> 
> Henning Schild (7):
>   conf: add deb-src entries to all our distro configs
>   dpkg-base: introduce an "apt-get source" fetch/unpack step
>   meta: move debianization code into a class and into dpkg-base
>   debianize: allow changlog version change
>   meta-isar/recipes-app: add upstream hello rebuild example
>   local.conf: remove example-hello from the default build
>   local.conf: enable rebuilding "hello" for all distros
> 
>  meta-isar/conf/distro/debian-buster.list      |  3 +
>  meta-isar/conf/distro/debian-jessie.list      |  3 +
>  .../conf/distro/debian-stretch-backports.list |  1 +
>  meta-isar/conf/distro/debian-stretch.list     |  3 +
>  meta-isar/conf/distro/raspbian-jessie.list    |  1 +
>  meta-isar/conf/local.conf.sample              |  2 +-
>  meta-isar/recipes-app/hello/hello.inc         | 19 +++++
>  meta-isar/recipes-app/hello/hello_2.10.bb     |  8 ++
>  meta-isar/recipes-app/hello/hello_2.9.bb      |  8 ++
>  meta/classes/debianize.bbclass                | 82
> +++++++++++++++++++ meta/classes/dpkg-base.bbclass                |
> 20 +++++ meta/classes/dpkg-raw.bbclass                 | 76
> ----------------- 12 files changed, 149 insertions(+), 77 deletions(-)
>  create mode 100644 meta-isar/recipes-app/hello/hello.inc
>  create mode 100644 meta-isar/recipes-app/hello/hello_2.10.bb
>  create mode 100644 meta-isar/recipes-app/hello/hello_2.9.bb
>  create mode 100644 meta/classes/debianize.bbclass
> 

Re: [PATCH 0/7] "apt-get source" fetch/unpack support

[Claudius Heine]

Hi Henning,

On 17/01/2019 17.38, [ext] Henning Schild wrote:
> Am Thu, 17 Jan 2019 17:04:20 +0100
> schrieb Henning Schild <henning.schild@siemens.com>:
> 
>> From: Henning Schild <henning.schild@siemens.com>
>>
>> This series includes support for fetching upstream sources with
>> "apt-get source". This will make sure we fetch exactly what matches
>> out distro, without rewriting debian fetch/unpack logic.
>> I did consider implementing it as an "apt://" extension to the regular
>> fetcher but decided against that. You have to set SRC_APT and
>> effectively pass arguement to apt-get. That fetcher can only work in
>> packages and depends on buildchroot and mounting, so it can not be
>> part of the general fetcher. But maybe the general fetcher could
>> ignore "apt://" lines and this task will ignore anything but "apt://"
>> so we can still use SRC_URI instead of SRC_APT.
> 
> Filtering all the "apt://" entries out does not really work. There is a
> call to bb.fetch.get_checksum_file_list where i can not filter, at
> least i did not yet figure out how.
> 
> And it seems that the fetcher can not be extended with a new protocol
> without touching the bitbake core. Otherwise an empty implementation of
> "apt://" would have been the trick to let debian do it in a completely
> different step.
> Not sure it is worth forcing it into SRC_URI, we are in two worlds
> anyways, no need to pretend otherwise.

Ok thx! As for variable conflict: There is "APTSRCS" in isar-bootstrap, 
which it IMO far enough away from "SRC_APT", so that name would be OK.

Claudius

> 
> Henning
> 
>> Let us talk about the
>> interface. The documentation is still missing, but i thing it is
>> ready for a first review.
>>
>> The idea and parts of the code came from Mentor.
>>
>> Henning Schild (7):
>>    conf: add deb-src entries to all our distro configs
>>    dpkg-base: introduce an "apt-get source" fetch/unpack step
>>    meta: move debianization code into a class and into dpkg-base
>>    debianize: allow changlog version change
>>    meta-isar/recipes-app: add upstream hello rebuild example
>>    local.conf: remove example-hello from the default build
>>    local.conf: enable rebuilding "hello" for all distros
>>
>>   meta-isar/conf/distro/debian-buster.list      |  3 +
>>   meta-isar/conf/distro/debian-jessie.list      |  3 +
>>   .../conf/distro/debian-stretch-backports.list |  1 +
>>   meta-isar/conf/distro/debian-stretch.list     |  3 +
>>   meta-isar/conf/distro/raspbian-jessie.list    |  1 +
>>   meta-isar/conf/local.conf.sample              |  2 +-
>>   meta-isar/recipes-app/hello/hello.inc         | 19 +++++
>>   meta-isar/recipes-app/hello/hello_2.10.bb     |  8 ++
>>   meta-isar/recipes-app/hello/hello_2.9.bb      |  8 ++
>>   meta/classes/debianize.bbclass                | 82
>> +++++++++++++++++++ meta/classes/dpkg-base.bbclass                |
>> 20 +++++ meta/classes/dpkg-raw.bbclass                 | 76
>> ----------------- 12 files changed, 149 insertions(+), 77 deletions(-)
>>   create mode 100644 meta-isar/recipes-app/hello/hello.inc
>>   create mode 100644 meta-isar/recipes-app/hello/hello_2.10.bb
>>   create mode 100644 meta-isar/recipes-app/hello/hello_2.9.bb
>>   create mode 100644 meta/classes/debianize.bbclass
>>
> 

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [PATCH 0/7] "apt-get source" fetch/unpack support

[Henning Schild]

Am Thu, 17 Jan 2019 17:04:20 +0100
schrieb Henning Schild <henning.schild@siemens.com>:

> From: Henning Schild <henning.schild@siemens.com>
> 
> This series includes support for fetching upstream sources with
> "apt-get source". This will make sure we fetch exactly what matches
> out distro, without rewriting debian fetch/unpack logic.
> I did consider implementing it as an "apt://" extension to the regular
> fetcher but decided against that. You have to set SRC_APT and
> effectively pass arguement to apt-get. That fetcher can only work in
> packages and depends on buildchroot and mounting, so it can not be
> part of the general fetcher. But maybe the general fetcher could
> ignore "apt://" lines and this task will ignore anything but "apt://"
> so we can still use SRC_URI instead of SRC_APT.

Filtering all the "apt://" entries out does not really work. There is a
call to bb.fetch.get_checksum_file_list where i can not filter, at
least i did not yet figure out how.

And it seems that the fetcher can not be extended with a new protocol
without touching the bitbake core. Otherwise an empty implementation of
"apt://" would have been the trick to let debian do it in a completely
different step.
Not sure it is worth forcing it into SRC_URI, we are in two worlds
anyways, no need to pretend otherwise.

Henning

> Let us talk about the
> interface. The documentation is still missing, but i thing it is
> ready for a first review.
> 
> The idea and parts of the code came from Mentor.
> 
> Henning Schild (7):
>   conf: add deb-src entries to all our distro configs
>   dpkg-base: introduce an "apt-get source" fetch/unpack step
>   meta: move debianization code into a class and into dpkg-base
>   debianize: allow changlog version change
>   meta-isar/recipes-app: add upstream hello rebuild example
>   local.conf: remove example-hello from the default build
>   local.conf: enable rebuilding "hello" for all distros
> 
>  meta-isar/conf/distro/debian-buster.list      |  3 +
>  meta-isar/conf/distro/debian-jessie.list      |  3 +
>  .../conf/distro/debian-stretch-backports.list |  1 +
>  meta-isar/conf/distro/debian-stretch.list     |  3 +
>  meta-isar/conf/distro/raspbian-jessie.list    |  1 +
>  meta-isar/conf/local.conf.sample              |  2 +-
>  meta-isar/recipes-app/hello/hello.inc         | 19 +++++
>  meta-isar/recipes-app/hello/hello_2.10.bb     |  8 ++
>  meta-isar/recipes-app/hello/hello_2.9.bb      |  8 ++
>  meta/classes/debianize.bbclass                | 82
> +++++++++++++++++++ meta/classes/dpkg-base.bbclass                |
> 20 +++++ meta/classes/dpkg-raw.bbclass                 | 76
> ----------------- 12 files changed, 149 insertions(+), 77 deletions(-)
>  create mode 100644 meta-isar/recipes-app/hello/hello.inc
>  create mode 100644 meta-isar/recipes-app/hello/hello_2.10.bb
>  create mode 100644 meta-isar/recipes-app/hello/hello_2.9.bb
>  create mode 100644 meta/classes/debianize.bbclass
> 

[PATCH 0/7] "apt-get source" fetch/unpack support

[Henning Schild]

From: Henning Schild <henning.schild@siemens.com>

This series includes support for fetching upstream sources with "apt-get
source". This will make sure we fetch exactly what matches out distro,
without rewriting debian fetch/unpack logic.
I did consider implementing it as an "apt://" extension to the regular
fetcher but decided against that. You have to set SRC_APT and
effectively pass arguement to apt-get. That fetcher can only work in
packages and depends on buildchroot and mounting, so it can not be part
of the general fetcher. But maybe the general fetcher could ignore
"apt://" lines and this task will ignore anything but "apt://" so we can
still use SRC_URI instead of SRC_APT. Let us talk about the interface.
The documentation is still missing, but i thing it is ready for a first
review.

The idea and parts of the code came from Mentor.

Henning Schild (7):
  conf: add deb-src entries to all our distro configs
  dpkg-base: introduce an "apt-get source" fetch/unpack step
  meta: move debianization code into a class and into dpkg-base
  debianize: allow changlog version change
  meta-isar/recipes-app: add upstream hello rebuild example
  local.conf: remove example-hello from the default build
  local.conf: enable rebuilding "hello" for all distros

 meta-isar/conf/distro/debian-buster.list      |  3 +
 meta-isar/conf/distro/debian-jessie.list      |  3 +
 .../conf/distro/debian-stretch-backports.list |  1 +
 meta-isar/conf/distro/debian-stretch.list     |  3 +
 meta-isar/conf/distro/raspbian-jessie.list    |  1 +
 meta-isar/conf/local.conf.sample              |  2 +-
 meta-isar/recipes-app/hello/hello.inc         | 19 +++++
 meta-isar/recipes-app/hello/hello_2.10.bb     |  8 ++
 meta-isar/recipes-app/hello/hello_2.9.bb      |  8 ++
 meta/classes/debianize.bbclass                | 82 +++++++++++++++++++
 meta/classes/dpkg-base.bbclass                | 20 +++++
 meta/classes/dpkg-raw.bbclass                 | 76 -----------------
 12 files changed, 149 insertions(+), 77 deletions(-)
 create mode 100644 meta-isar/recipes-app/hello/hello.inc
 create mode 100644 meta-isar/recipes-app/hello/hello_2.10.bb
 create mode 100644 meta-isar/recipes-app/hello/hello_2.9.bb
 create mode 100644 meta/classes/debianize.bbclass

-- 
2.19.2