[PATCH v8 0/7] Fix usage of additional apt keys and repos

[PATCH v8 0/7] Fix usage of additional apt keys and repos

[Andreas J. Reichel]

From: Andreas Reichel <andreas.reichel.ext@siemens.com>

Diff to v7: Complete rework :)

After a hard discussion things turned out to be cleaner and less error
prone. Thank you, Claudius!

* No keys added to host at all
* Keys needed for bootstrapping go to a separated keyring in the
  workdir
* Third party apt keys are added to
  /etc/apt/trusted.gpg.d/third_party.gpg only in the generated rootfs

* The following variables are renamed/created:

  Old name                New name
  -                       DISTRO_BOOTSTRAP_KEYRING
  DISTRO_APT_KEYS         DISTRO_BOOTSTRAP_KEYS
  -                       DISTRO_BOOTSTRAP_KEYFILES
  ISARKEYRING             THIRD_PARTY_APT_KEYRING
  -                       THIRD_PARTY_APT_KEYS
  APTKEYFILES             THIRD_PARTY_APT_KEYFILES

Tested building without error:
  qemuarm64-stretch
  qemuarm64-stretch with 3rd party docker repo
  raspbian-jessie

Not working yet:
  qemuarm64-stretch with BASE_REPO_KEY and do_cache_base_repo

| gpgme gave error GPGME:54:  Unusable secret key

I have created a keypair inside the build container and exported the
public key to a file "blabla.key". Then I set

BASE_REPO_KEY = "file:///build/blabla.key"

Any idea?

Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>

Andreas Reichel (7):
  Revert "isar-bootstrap: Allow to set local keys in DISTRO_APT_KEYS"
  Remove duplicate code from apt-keyring generation
  Fix fetched key location in apt-keyring generator
  Use apt-key to generate keyrings
  If we use a custom keyring debootstrap may fall to https
  raspbian-jessie: Use DISTRO_BOOTSTRAP_KEYS
  docs: Update user_manual.md

 doc/user_manual.md                            |  7 +-
 meta-isar/conf/distro/raspbian-jessie.conf    |  2 +-
 .../conf/multiconfig/qemuamd64-buster.conf    |  1 -
 .../conf/multiconfig/qemuamd64-jessie.conf    |  1 -
 meta/conf/bitbake.conf                        |  1 +
 .../isar-bootstrap/isar-bootstrap-host.bb     |  4 +-
 .../isar-bootstrap/isar-bootstrap-target.bb   |  4 +-
 .../isar-bootstrap/isar-bootstrap.inc         | 95 +++++++++++++------
 8 files changed, 79 insertions(+), 36 deletions(-)

-- 
2.21.0

[PATCH v8 1/7] Revert "isar-bootstrap: Allow to set local keys in DISTRO_APT_KEYS"

[Andreas J. Reichel]

From: Andreas Reichel <andreas.reichel.ext@siemens.com>

This reverts commit af983a13b6f4cee5d4af5e5cf6318231e02775c9.

This commit broke usage of remote keys, where they usually come from.
If fetching "http://example.com/dir1/dir2/key", the file is fetched
into the subdir WORKDIR/dir1/dir2/, which breaks with this code.
However it succeeds with absolute paths.
We do not want to guess where the downloaded file will be. This does
not work anymore if the key is downloaded from remote with a URL.
Furthermore, a user could specify "subdir" as fetcher parameter
or other things, which break this.

This is really fixed in a follow-up commit.

Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
---
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index c1b571a..2910eea 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -40,8 +40,9 @@ python () {
         d.setVar("DEBOOTSTRAP_KEYRING", "--keyring ${APTKEYRING}")
         for key in distro_apt_keys.split():
             url = urlparse(key)
-            d.appendVar("SRC_URI", " " + key)
-            d.appendVar("APTKEYFILES", " " + wd + url.path)
+            filename = os.path.basename(url.path)
+            d.appendVar("SRC_URI", " %s" % key)
+            d.appendVar("APTKEYFILES", " %s" % filename)
     if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
         own_pub_key = d.getVar("BASE_REPO_KEY", False)
         if own_pub_key:
-- 
2.21.0

Re: [PATCH v8 1/7] Revert "isar-bootstrap: Allow to set local keys in DISTRO_APT_KEYS"

[Maxim Yu. Osipov]

Hi Andreas,

On 3/21/19 4:15 PM, Andreas J. Reichel wrote:
> From: Andreas Reichel <andreas.reichel.ext@siemens.com>
> 
> This reverts commit af983a13b6f4cee5d4af5e5cf6318231e02775c9.
> 
> This commit broke usage of remote keys, where they usually come from.
> If fetching "http://example.com/dir1/dir2/key", the file is fetched
> into the subdir WORKDIR/dir1/dir2/, which breaks with this code.
> However it succeeds with absolute paths.
> We do not want to guess where the downloaded file will be. This does
> not work anymore if the key is downloaded from remote with a URL.
> Furthermore, a user could specify "subdir" as fetcher parameter
> or other things, which break this.

In general it's better to avoid reverting the patches.

I disagree with the statement that "commit af983a13  broke usage of 
remote keys" - I wrote you a month ago (see forwarded email below) that 
this patch doesn't break Raspbian build  - the key is put also under 
downloads (DL_DIR) directory:

I've build the current 'master' for target 
multiconfig:rpi-jessie:isar-image-base.

Sorry for copy-paste ;):

<<<<
isar/build$ find -name raspbian.public.key
./downloads/raspbian.public.key
./tmp/work/raspbian-jessie-armhf/isar-bootstrap-target/raspbian.public.key
 >>>

So I don't accept this patch.

Regards,
Maxim.


-------- Forwarded Message --------
Subject: Re: [PATCH 0/1] Fix remote key fetching apt keyring
Date: Wed, 20 Feb 2019 12:58:09 +0100
From: Maxim Yu. Osipov <mosipov@ilbers.de>
Organization: ilbers GmbH
To: Jan Kiszka <jan.kiszka@siemens.com>, [ext] Andreas J. Reichel 
<andreas.reichel.ext@siemens.com>, isar-users@googlegroups.com, Baurzhan 
Ismagulov <ibr@ilbers.de>

On 2/20/19 12:27 PM, Jan Kiszka wrote:
 > On 20.02.19 12:21, [ext] Andreas J. Reichel wrote:
 >> From: Andreas Reichel <andreas.reichel.ext@siemens.com>
 >>
 >> Since my last mail was not answered, but this is an important topic,
 >> here is a patch that shows what the problem is.
 >>
 >> If we fetch the user apt key from remote, we need the basename,
 >> if we fetch it locally we need the absolute path...
 >>
 >> While this might not be the best way to fix this, it works as good
 >> as the rest of this code...
 >>
 >> At least it fixes Isar again up to adding the key to the keyring.
 >>
 >> But this still does not fix the next problem with the docker-ce key:
 >>
 >> | I: Running command: debootstrap --arch arm64 --foreign --verbose
 >> --variant=minbase --include=locales --components=main,contrib,non-free
 >> --keyring
 >> 
/build/build/tmp/work/debian-stretch-arm64/isar-bootstrap-target/apt-keyring.gpg 

 >> stretch
 >> 
/build/build/tmp/work/debian-stretch-arm64/isar-bootstrap-target/rootfs 
http://ftp.debian.org/debian
 >>
 >> | I: Retrieving InRelease
 >> | I: Retrieving Release
 >> | I: Retrieving Release.gpg
 >> | I: Checking Release signature
 >> | E: Release signed by unknown key (key id EF0F382A1A7B6500)
 >>
 >> So something additionally must be done. Since I am not an expert on
 >> debian keyring/debootstrap and dpkg signing I will try to find a
 >> solution but maybe somebody has a good idea already?
 >>
 >
 > Baurzhan, Maxim, any idea?

Strange...I thought that commit af983a13 fixes the reported problem
When testing my patch signing base-apt I've tried both - remote keys 
(used by Raspberry Pi target) and local key.



> 
> This is really fixed in a follow-up commit.
> 
> Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
> ---
>   meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 5 +++--
>   1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> index c1b571a..2910eea 100644
> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> @@ -40,8 +40,9 @@ python () {
>           d.setVar("DEBOOTSTRAP_KEYRING", "--keyring ${APTKEYRING}")
>           for key in distro_apt_keys.split():
>               url = urlparse(key)
> -            d.appendVar("SRC_URI", " " + key)
> -            d.appendVar("APTKEYFILES", " " + wd + url.path)
> +            filename = os.path.basename(url.path)
> +            d.appendVar("SRC_URI", " %s" % key)
> +            d.appendVar("APTKEYFILES", " %s" % filename)
>       if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
>           own_pub_key = d.getVar("BASE_REPO_KEY", False)
>           if own_pub_key:
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: [PATCH v8 1/7] Revert "isar-bootstrap: Allow to set local keys in DISTRO_APT_KEYS"

[Andreas Reichel]

On Mon, Mar 25, 2019 at 12:20:10PM +0100, Maxim Yu. Osipov wrote:
> Hi Andreas,
> 
> On 3/21/19 4:15 PM, Andreas J. Reichel wrote:
> > From: Andreas Reichel <andreas.reichel.ext@siemens.com>
> > 
> > This reverts commit af983a13b6f4cee5d4af5e5cf6318231e02775c9.
> > 
> > This commit broke usage of remote keys, where they usually come from.
> > If fetching "http://example.com/dir1/dir2/key", the file is fetched
> > into the subdir WORKDIR/dir1/dir2/, which breaks with this code.
> > However it succeeds with absolute paths.
> > We do not want to guess where the downloaded file will be. This does
> > not work anymore if the key is downloaded from remote with a URL.
> > Furthermore, a user could specify "subdir" as fetcher parameter
> > or other things, which break this.
> 
> In general it's better to avoid reverting the patches.

Okay.

> 
> I disagree with the statement that "commit af983a13  broke usage of remote
> keys" - I wrote you a month ago (see forwarded email below) that this patch
> doesn't break Raspbian build  - the key is put also under downloads (DL_DIR)

in DL_DIR - Yes, where you can get name collisions with several keys since you
cannot rename the files with your code via bitbake URL parameter.

> directory:
> 
> I've build the current 'master' for target
> multiconfig:rpi-jessie:isar-image-base.

I gave you an example where it broke for me. Raspbian is not one of
these.

 https://archive.raspbian.org/raspbian.public.key

 Is there a subdir after the domain? - No ! Does it fit to my example?
 => No. Does your argument with raspbian make sense now? => No.
> 
> Sorry for copy-paste ;):
> 
> <<<<
> isar/build$ find -name raspbian.public.key
> ./downloads/raspbian.public.key
> ./tmp/work/raspbian-jessie-armhf/isar-bootstrap-target/raspbian.public.key
> >>>
> 
> So I don't accept this patch.
> 
Then I have to rebase - without revert - , which may take some time...

Regards,
Andreas

> Regards,
> Maxim.
> 
> 
> -------- Forwarded Message --------
> Subject: Re: [PATCH 0/1] Fix remote key fetching apt keyring
> Date: Wed, 20 Feb 2019 12:58:09 +0100
> From: Maxim Yu. Osipov <mosipov@ilbers.de>
> Organization: ilbers GmbH
> To: Jan Kiszka <jan.kiszka@siemens.com>, [ext] Andreas J. Reichel
> <andreas.reichel.ext@siemens.com>, isar-users@googlegroups.com, Baurzhan
> Ismagulov <ibr@ilbers.de>
> 
> On 2/20/19 12:27 PM, Jan Kiszka wrote:
> > On 20.02.19 12:21, [ext] Andreas J. Reichel wrote:
> >> From: Andreas Reichel <andreas.reichel.ext@siemens.com>
> >>
> >> Since my last mail was not answered, but this is an important topic,
> >> here is a patch that shows what the problem is.
> >>
> >> If we fetch the user apt key from remote, we need the basename,
> >> if we fetch it locally we need the absolute path...
> >>
> >> While this might not be the best way to fix this, it works as good
> >> as the rest of this code...
> >>
> >> At least it fixes Isar again up to adding the key to the keyring.
> >>
> >> But this still does not fix the next problem with the docker-ce key:
> >>
> >> | I: Running command: debootstrap --arch arm64 --foreign --verbose
> >> --variant=minbase --include=locales --components=main,contrib,non-free
> >> --keyring
> >> /build/build/tmp/work/debian-stretch-arm64/isar-bootstrap-target/apt-keyring.gpg
> 
> >> stretch
> >> /build/build/tmp/work/debian-stretch-arm64/isar-bootstrap-target/rootfs
> http://ftp.debian.org/debian
> >>
> >> | I: Retrieving InRelease
> >> | I: Retrieving Release
> >> | I: Retrieving Release.gpg
> >> | I: Checking Release signature
> >> | E: Release signed by unknown key (key id EF0F382A1A7B6500)
> >>
> >> So something additionally must be done. Since I am not an expert on
> >> debian keyring/debootstrap and dpkg signing I will try to find a
> >> solution but maybe somebody has a good idea already?
> >>
> >
> > Baurzhan, Maxim, any idea?
> 
> Strange...I thought that commit af983a13 fixes the reported problem
> When testing my patch signing base-apt I've tried both - remote keys (used
> by Raspberry Pi target) and local key.
> 
> 
> 
> > 
> > This is really fixed in a follow-up commit.
> > 
> > Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
> > ---
> >   meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 5 +++--
> >   1 file changed, 3 insertions(+), 2 deletions(-)
> > 
> > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> > index c1b571a..2910eea 100644
> > --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> > +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> > @@ -40,8 +40,9 @@ python () {
> >           d.setVar("DEBOOTSTRAP_KEYRING", "--keyring ${APTKEYRING}")
> >           for key in distro_apt_keys.split():
> >               url = urlparse(key)
> > -            d.appendVar("SRC_URI", " " + key)
> > -            d.appendVar("APTKEYFILES", " " + wd + url.path)
> > +            filename = os.path.basename(url.path)
> > +            d.appendVar("SRC_URI", " %s" % key)
> > +            d.appendVar("APTKEYFILES", " %s" % filename)
> >       if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
> >           own_pub_key = d.getVar("BASE_REPO_KEY", False)
> >           if own_pub_key:
> > 
> 
> 
> -- 
> Maxim Osipov
> ilbers GmbH
> Maria-Merian-Str. 8
> 85521 Ottobrunn
> Germany
> +49 (151) 6517 6917
> mosipov@ilbers.de
> http://ilbers.de/
> Commercial register Munich, HRB 214197
> General Manager: Baurzhan Ismagulov

-- 
Andreas Reichel
Dipl.-Phys. (Univ.)
Software Consultant

Andreas.Reichel@tngtech.com, +49-174-3180074
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterfoehring
Geschaeftsfuehrer: Henrik Klagges, Dr. Robert Dahlke, Gerhard Mueller
Sitz: Unterfoehring * Amtsgericht Muenchen * HRB 135082

Re: [PATCH v8 1/7] Revert "isar-bootstrap: Allow to set local keys in DISTRO_APT_KEYS"

[Maxim Yu. Osipov]

On 4/15/19 1:11 PM, Andreas Reichel wrote:
> On Mon, Mar 25, 2019 at 12:20:10PM +0100, Maxim Yu. Osipov wrote:
>> Hi Andreas,
>>
>> On 3/21/19 4:15 PM, Andreas J. Reichel wrote:
>>> From: Andreas Reichel <andreas.reichel.ext@siemens.com>
>>>
>>> This reverts commit af983a13b6f4cee5d4af5e5cf6318231e02775c9.
>>>
>>> This commit broke usage of remote keys, where they usually come from.
>>> If fetching "http://example.com/dir1/dir2/key", the file is fetched
>>> into the subdir WORKDIR/dir1/dir2/, which breaks with this code.
>>> However it succeeds with absolute paths.
>>> We do not want to guess where the downloaded file will be. This does
>>> not work anymore if the key is downloaded from remote with a URL.
>>> Furthermore, a user could specify "subdir" as fetcher parameter
>>> or other things, which break this.
>>
>> In general it's better to avoid reverting the patches.
> 
> Okay.
> 
>>
>> I disagree with the statement that "commit af983a13  broke usage of remote
>> keys" - I wrote you a month ago (see forwarded email below) that this patch
>> doesn't break Raspbian build  - the key is put also under downloads (DL_DIR)
> 
> in DL_DIR - Yes, where you can get name collisions with several keys since you
> cannot rename the files with your code via bitbake URL parameter.

Well...And how this problem is related with my patch?
My patch fixes the problem with locally stored keys, as
Local fetcher module puts the file under
${WORKDIR}/<absolute_path_to_key_file>, so
gpg in do_generate_keyring() task can't find it.


>> directory:
>>
>> I've build the current 'master' for target
>> multiconfig:rpi-jessie:isar-image-base.
> 
> I gave you an example where it broke for me. Raspbian is not one of
> these.
> 
>   https://archive.raspbian.org/raspbian.public.key
> 
>   Is there a subdir after the domain? - No ! Does it fit to my example?
>   => No. Does your argument with raspbian make sense now? => No.

Again, my patch fixes the problem with local keys.
It does nothing about fetching remote keys - it just doesn't break 
existing functionality.


>>
>> Sorry for copy-paste ;):
>>
>> <<<<
>> isar/build$ find -name raspbian.public.key
>> ./downloads/raspbian.public.key
>> ./tmp/work/raspbian-jessie-armhf/isar-bootstrap-target/raspbian.public.key
>>>>>
>>
>> So I don't accept this patch.
>>
> Then I have to rebase - without revert - , which may take some time...

OK.

Regards,
Maxim.

> Regards,
> Andreas
> 
>> Regards,
>> Maxim.
>>
>>
>> -------- Forwarded Message --------
>> Subject: Re: [PATCH 0/1] Fix remote key fetching apt keyring
>> Date: Wed, 20 Feb 2019 12:58:09 +0100
>> From: Maxim Yu. Osipov <mosipov@ilbers.de>
>> Organization: ilbers GmbH
>> To: Jan Kiszka <jan.kiszka@siemens.com>, [ext] Andreas J. Reichel
>> <andreas.reichel.ext@siemens.com>, isar-users@googlegroups.com, Baurzhan
>> Ismagulov <ibr@ilbers.de>
>>
>> On 2/20/19 12:27 PM, Jan Kiszka wrote:
>>> On 20.02.19 12:21, [ext] Andreas J. Reichel wrote:
>>>> From: Andreas Reichel <andreas.reichel.ext@siemens.com>
>>>>
>>>> Since my last mail was not answered, but this is an important topic,
>>>> here is a patch that shows what the problem is.
>>>>
>>>> If we fetch the user apt key from remote, we need the basename,
>>>> if we fetch it locally we need the absolute path...
>>>>
>>>> While this might not be the best way to fix this, it works as good
>>>> as the rest of this code...
>>>>
>>>> At least it fixes Isar again up to adding the key to the keyring.
>>>>
>>>> But this still does not fix the next problem with the docker-ce key:
>>>>
>>>> | I: Running command: debootstrap --arch arm64 --foreign --verbose
>>>> --variant=minbase --include=locales --components=main,contrib,non-free
>>>> --keyring
>>>> /build/build/tmp/work/debian-stretch-arm64/isar-bootstrap-target/apt-keyring.gpg
>>
>>>> stretch
>>>> /build/build/tmp/work/debian-stretch-arm64/isar-bootstrap-target/rootfs
>> http://ftp.debian.org/debian
>>>>
>>>> | I: Retrieving InRelease
>>>> | I: Retrieving Release
>>>> | I: Retrieving Release.gpg
>>>> | I: Checking Release signature
>>>> | E: Release signed by unknown key (key id EF0F382A1A7B6500)
>>>>
>>>> So something additionally must be done. Since I am not an expert on
>>>> debian keyring/debootstrap and dpkg signing I will try to find a
>>>> solution but maybe somebody has a good idea already?
>>>>
>>>
>>> Baurzhan, Maxim, any idea?
>>
>> Strange...I thought that commit af983a13 fixes the reported problem
>> When testing my patch signing base-apt I've tried both - remote keys (used
>> by Raspberry Pi target) and local key.
>>
>>
>>
>>>
>>> This is really fixed in a follow-up commit.
>>>
>>> Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
>>> ---
>>>    meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 5 +++--
>>>    1 file changed, 3 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>>> index c1b571a..2910eea 100644
>>> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>>> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>>> @@ -40,8 +40,9 @@ python () {
>>>            d.setVar("DEBOOTSTRAP_KEYRING", "--keyring ${APTKEYRING}")
>>>            for key in distro_apt_keys.split():
>>>                url = urlparse(key)
>>> -            d.appendVar("SRC_URI", " " + key)
>>> -            d.appendVar("APTKEYFILES", " " + wd + url.path)
>>> +            filename = os.path.basename(url.path)
>>> +            d.appendVar("SRC_URI", " %s" % key)
>>> +            d.appendVar("APTKEYFILES", " %s" % filename)
>>>        if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
>>>            own_pub_key = d.getVar("BASE_REPO_KEY", False)
>>>            if own_pub_key:
>>>
>>
>>
>> -- 
>> Maxim Osipov
>> ilbers GmbH
>> Maria-Merian-Str. 8
>> 85521 Ottobrunn
>> Germany
>> +49 (151) 6517 6917
>> mosipov@ilbers.de
>> http://ilbers.de/
>> Commercial register Munich, HRB 214197
>> General Manager: Baurzhan Ismagulov
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: [PATCH v8 1/7] Revert "isar-bootstrap: Allow to set local keys in DISTRO_APT_KEYS"

[Henning Schild]

Am Tue, 16 Apr 2019 06:54:51 +0200
schrieb "Maxim Yu. Osipov" <mosipov@ilbers.de>:

> On 4/15/19 1:11 PM, Andreas Reichel wrote:
> > On Mon, Mar 25, 2019 at 12:20:10PM +0100, Maxim Yu. Osipov wrote:  
> >> Hi Andreas,
> >>
> >> On 3/21/19 4:15 PM, Andreas J. Reichel wrote:  
> >>> From: Andreas Reichel <andreas.reichel.ext@siemens.com>
> >>>
> >>> This reverts commit af983a13b6f4cee5d4af5e5cf6318231e02775c9.
> >>>
> >>> This commit broke usage of remote keys, where they usually come
> >>> from. If fetching "http://example.com/dir1/dir2/key", the file is
> >>> fetched into the subdir WORKDIR/dir1/dir2/, which breaks with
> >>> this code. However it succeeds with absolute paths.
> >>> We do not want to guess where the downloaded file will be. This
> >>> does not work anymore if the key is downloaded from remote with a
> >>> URL. Furthermore, a user could specify "subdir" as fetcher
> >>> parameter or other things, which break this.  
> >>
> >> In general it's better to avoid reverting the patches.  
> > 
> > Okay.
> >   
> >>
> >> I disagree with the statement that "commit af983a13  broke usage
> >> of remote keys" - I wrote you a month ago (see forwarded email
> >> below) that this patch doesn't break Raspbian build  - the key is
> >> put also under downloads (DL_DIR)  
> > 
> > in DL_DIR - Yes, where you can get name collisions with several
> > keys since you cannot rename the files with your code via bitbake
> > URL parameter.  
> 
> Well...And how this problem is related with my patch?
> My patch fixes the problem with locally stored keys, as
> Local fetcher module puts the file under
> ${WORKDIR}/<absolute_path_to_key_file>, so
> gpg in do_generate_keyring() task can't find it.
> 
> 
> >> directory:
> >>
> >> I've build the current 'master' for target
> >> multiconfig:rpi-jessie:isar-image-base.  
> > 
> > I gave you an example where it broke for me. Raspbian is not one of
> > these.
> > 
> >   https://archive.raspbian.org/raspbian.public.key
> > 
> >   Is there a subdir after the domain? - No ! Does it fit to my
> > example? => No. Does your argument with raspbian make sense now? =>
> > No.  
> 
> Again, my patch fixes the problem with local keys.
> It does nothing about fetching remote keys - it just doesn't break 
> existing functionality.

While i do agree that a revert looks like blaming a patch and an author
for doing something wrong, i fail to get why this comes up so late in
the review. Yes Andreas can rebase yet again and turn the revert into
something else, in the end the code will be the same.

The revert question was already discussed in v3. Let us talk content
and that series is in very good shape.

I would suggest to merge that and accept a test-case / example on top.

Henning

> 
> >>
> >> Sorry for copy-paste ;):
> >>
> >> <<<<
> >> isar/build$ find -name raspbian.public.key
> >> ./downloads/raspbian.public.key
> >> ./tmp/work/raspbian-jessie-armhf/isar-bootstrap-target/raspbian.public.key  
> >>>>>  
> >>
> >> So I don't accept this patch.
> >>  
> > Then I have to rebase - without revert - , which may take some
> > time...  
> 
> OK.
> 
> Regards,
> Maxim.
> 
> > Regards,
> > Andreas
> >   
> >> Regards,
> >> Maxim.
> >>
> >>
> >> -------- Forwarded Message --------
> >> Subject: Re: [PATCH 0/1] Fix remote key fetching apt keyring
> >> Date: Wed, 20 Feb 2019 12:58:09 +0100
> >> From: Maxim Yu. Osipov <mosipov@ilbers.de>
> >> Organization: ilbers GmbH
> >> To: Jan Kiszka <jan.kiszka@siemens.com>, [ext] Andreas J. Reichel
> >> <andreas.reichel.ext@siemens.com>, isar-users@googlegroups.com,
> >> Baurzhan Ismagulov <ibr@ilbers.de>
> >>
> >> On 2/20/19 12:27 PM, Jan Kiszka wrote:  
> >>> On 20.02.19 12:21, [ext] Andreas J. Reichel wrote:  
> >>>> From: Andreas Reichel <andreas.reichel.ext@siemens.com>
> >>>>
> >>>> Since my last mail was not answered, but this is an important
> >>>> topic, here is a patch that shows what the problem is.
> >>>>
> >>>> If we fetch the user apt key from remote, we need the basename,
> >>>> if we fetch it locally we need the absolute path...
> >>>>
> >>>> While this might not be the best way to fix this, it works as
> >>>> good as the rest of this code...
> >>>>
> >>>> At least it fixes Isar again up to adding the key to the keyring.
> >>>>
> >>>> But this still does not fix the next problem with the docker-ce
> >>>> key:
> >>>>
> >>>> | I: Running command: debootstrap --arch arm64 --foreign
> >>>> --verbose --variant=minbase --include=locales
> >>>> --components=main,contrib,non-free --keyring
> >>>> /build/build/tmp/work/debian-stretch-arm64/isar-bootstrap-target/apt-keyring.gpg  
> >>  
> >>>> stretch
> >>>> /build/build/tmp/work/debian-stretch-arm64/isar-bootstrap-target/rootfs  
> >> http://ftp.debian.org/debian  
> >>>>
> >>>> | I: Retrieving InRelease
> >>>> | I: Retrieving Release
> >>>> | I: Retrieving Release.gpg
> >>>> | I: Checking Release signature
> >>>> | E: Release signed by unknown key (key id EF0F382A1A7B6500)
> >>>>
> >>>> So something additionally must be done. Since I am not an expert
> >>>> on debian keyring/debootstrap and dpkg signing I will try to
> >>>> find a solution but maybe somebody has a good idea already?
> >>>>  
> >>>
> >>> Baurzhan, Maxim, any idea?  
> >>
> >> Strange...I thought that commit af983a13 fixes the reported problem
> >> When testing my patch signing base-apt I've tried both - remote
> >> keys (used by Raspberry Pi target) and local key.
> >>
> >>
> >>  
> >>>
> >>> This is really fixed in a follow-up commit.
> >>>
> >>> Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
> >>> ---
> >>>    meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 5 +++--
> >>>    1 file changed, 3 insertions(+), 2 deletions(-)
> >>>
> >>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> >>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> >>> c1b571a..2910eea 100644 ---
> >>> a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++
> >>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -40,8
> >>> +40,9 @@ python () { d.setVar("DEBOOTSTRAP_KEYRING", "--keyring
> >>> ${APTKEYRING}") for key in distro_apt_keys.split():
> >>>                url = urlparse(key)
> >>> -            d.appendVar("SRC_URI", " " + key)
> >>> -            d.appendVar("APTKEYFILES", " " + wd + url.path)
> >>> +            filename = os.path.basename(url.path)
> >>> +            d.appendVar("SRC_URI", " %s" % key)
> >>> +            d.appendVar("APTKEYFILES", " %s" % filename)
> >>>        if
> >>> bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
> >>> own_pub_key = d.getVar("BASE_REPO_KEY", False) if own_pub_key:
> >>>  
> >>
> >>
> >> -- 
> >> Maxim Osipov
> >> ilbers GmbH
> >> Maria-Merian-Str. 8
> >> 85521 Ottobrunn
> >> Germany
> >> +49 (151) 6517 6917
> >> mosipov@ilbers.de
> >> http://ilbers.de/
> >> Commercial register Munich, HRB 214197
> >> General Manager: Baurzhan Ismagulov  
> >   
> 
> 

Re: [PATCH v8 1/7] Revert "isar-bootstrap: Allow to set local keys in DISTRO_APT_KEYS"

[Maxim Yu. Osipov]

On 4/16/19 10:12 AM, Henning Schild wrote:
> Am Tue, 16 Apr 2019 06:54:51 +0200
> schrieb "Maxim Yu. Osipov" <mosipov@ilbers.de>:
> 
>> On 4/15/19 1:11 PM, Andreas Reichel wrote:
>>> On Mon, Mar 25, 2019 at 12:20:10PM +0100, Maxim Yu. Osipov wrote:
>>>> Hi Andreas,
>>>>
>>>> On 3/21/19 4:15 PM, Andreas J. Reichel wrote:
>>>>> From: Andreas Reichel <andreas.reichel.ext@siemens.com>
>>>>>
>>>>> This reverts commit af983a13b6f4cee5d4af5e5cf6318231e02775c9.
>>>>>
>>>>> This commit broke usage of remote keys, where they usually come
>>>>> from. If fetching "http://example.com/dir1/dir2/key", the file is
>>>>> fetched into the subdir WORKDIR/dir1/dir2/, which breaks with
>>>>> this code. However it succeeds with absolute paths.
>>>>> We do not want to guess where the downloaded file will be. This
>>>>> does not work anymore if the key is downloaded from remote with a
>>>>> URL. Furthermore, a user could specify "subdir" as fetcher
>>>>> parameter or other things, which break this.
>>>>
>>>> In general it's better to avoid reverting the patches.
>>>
>>> Okay.
>>>    
>>>>
>>>> I disagree with the statement that "commit af983a13  broke usage
>>>> of remote keys" - I wrote you a month ago (see forwarded email
>>>> below) that this patch doesn't break Raspbian build  - the key is
>>>> put also under downloads (DL_DIR)
>>>
>>> in DL_DIR - Yes, where you can get name collisions with several
>>> keys since you cannot rename the files with your code via bitbake
>>> URL parameter.
>>
>> Well...And how this problem is related with my patch?
>> My patch fixes the problem with locally stored keys, as
>> Local fetcher module puts the file under
>> ${WORKDIR}/<absolute_path_to_key_file>, so
>> gpg in do_generate_keyring() task can't find it.
>>
>>
>>>> directory:
>>>>
>>>> I've build the current 'master' for target
>>>> multiconfig:rpi-jessie:isar-image-base.
>>>
>>> I gave you an example where it broke for me. Raspbian is not one of
>>> these.
>>>
>>>    https://archive.raspbian.org/raspbian.public.key
>>>
>>>    Is there a subdir after the domain? - No ! Does it fit to my
>>> example? => No. Does your argument with raspbian make sense now? =>
>>> No.
>>
>> Again, my patch fixes the problem with local keys.
>> It does nothing about fetching remote keys - it just doesn't break
>> existing functionality.
> 
> While i do agree that a revert looks like blaming a patch and an author
> for doing something wrong, i fail to get why this comes up so late in
> the review. 

My first feedback was ignored by unknown reason.

Yes Andreas can rebase yet again and turn the revert into
> something else, in the end the code will be the same.

> 
> The revert question was already discussed in v3. Let us talk content
> and that series is in very good shape.
> 
> I would suggest to merge that and accept a test-case / example on top.

Not merged, as problems with the latest v9 series were discovered during 
testing.

Maxim.

> 
> Henning
> 
>>
>>>>
>>>> Sorry for copy-paste ;):
>>>>
>>>> <<<<
>>>> isar/build$ find -name raspbian.public.key
>>>> ./downloads/raspbian.public.key
>>>> ./tmp/work/raspbian-jessie-armhf/isar-bootstrap-target/raspbian.public.key
>>>>>>>   
>>>>
>>>> So I don't accept this patch.
>>>>   
>>> Then I have to rebase - without revert - , which may take some
>>> time...
>>
>> OK.
>>
>> Regards,
>> Maxim.
>>
>>> Regards,
>>> Andreas
>>>    
>>>> Regards,
>>>> Maxim.
>>>>
>>>>
>>>> -------- Forwarded Message --------
>>>> Subject: Re: [PATCH 0/1] Fix remote key fetching apt keyring
>>>> Date: Wed, 20 Feb 2019 12:58:09 +0100
>>>> From: Maxim Yu. Osipov <mosipov@ilbers.de>
>>>> Organization: ilbers GmbH
>>>> To: Jan Kiszka <jan.kiszka@siemens.com>, [ext] Andreas J. Reichel
>>>> <andreas.reichel.ext@siemens.com>, isar-users@googlegroups.com,
>>>> Baurzhan Ismagulov <ibr@ilbers.de>
>>>>
>>>> On 2/20/19 12:27 PM, Jan Kiszka wrote:
>>>>> On 20.02.19 12:21, [ext] Andreas J. Reichel wrote:
>>>>>> From: Andreas Reichel <andreas.reichel.ext@siemens.com>
>>>>>>
>>>>>> Since my last mail was not answered, but this is an important
>>>>>> topic, here is a patch that shows what the problem is.
>>>>>>
>>>>>> If we fetch the user apt key from remote, we need the basename,
>>>>>> if we fetch it locally we need the absolute path...
>>>>>>
>>>>>> While this might not be the best way to fix this, it works as
>>>>>> good as the rest of this code...
>>>>>>
>>>>>> At least it fixes Isar again up to adding the key to the keyring.
>>>>>>
>>>>>> But this still does not fix the next problem with the docker-ce
>>>>>> key:
>>>>>>
>>>>>> | I: Running command: debootstrap --arch arm64 --foreign
>>>>>> --verbose --variant=minbase --include=locales
>>>>>> --components=main,contrib,non-free --keyring
>>>>>> /build/build/tmp/work/debian-stretch-arm64/isar-bootstrap-target/apt-keyring.gpg
>>>>   
>>>>>> stretch
>>>>>> /build/build/tmp/work/debian-stretch-arm64/isar-bootstrap-target/rootfs
>>>> http://ftp.debian.org/debian
>>>>>>
>>>>>> | I: Retrieving InRelease
>>>>>> | I: Retrieving Release
>>>>>> | I: Retrieving Release.gpg
>>>>>> | I: Checking Release signature
>>>>>> | E: Release signed by unknown key (key id EF0F382A1A7B6500)
>>>>>>
>>>>>> So something additionally must be done. Since I am not an expert
>>>>>> on debian keyring/debootstrap and dpkg signing I will try to
>>>>>> find a solution but maybe somebody has a good idea already?
>>>>>>   
>>>>>
>>>>> Baurzhan, Maxim, any idea?
>>>>
>>>> Strange...I thought that commit af983a13 fixes the reported problem
>>>> When testing my patch signing base-apt I've tried both - remote
>>>> keys (used by Raspberry Pi target) and local key.
>>>>
>>>>
>>>>   
>>>>>
>>>>> This is really fixed in a follow-up commit.
>>>>>
>>>>> Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
>>>>> ---
>>>>>     meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 5 +++--
>>>>>     1 file changed, 3 insertions(+), 2 deletions(-)
>>>>>
>>>>> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
>>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
>>>>> c1b571a..2910eea 100644 ---
>>>>> a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++
>>>>> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -40,8
>>>>> +40,9 @@ python () { d.setVar("DEBOOTSTRAP_KEYRING", "--keyring
>>>>> ${APTKEYRING}") for key in distro_apt_keys.split():
>>>>>                 url = urlparse(key)
>>>>> -            d.appendVar("SRC_URI", " " + key)
>>>>> -            d.appendVar("APTKEYFILES", " " + wd + url.path)
>>>>> +            filename = os.path.basename(url.path)
>>>>> +            d.appendVar("SRC_URI", " %s" % key)
>>>>> +            d.appendVar("APTKEYFILES", " %s" % filename)
>>>>>         if
>>>>> bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
>>>>> own_pub_key = d.getVar("BASE_REPO_KEY", False) if own_pub_key:
>>>>>   
>>>>
>>>>
>>>> -- 
>>>> Maxim Osipov
>>>> ilbers GmbH
>>>> Maria-Merian-Str. 8
>>>> 85521 Ottobrunn
>>>> Germany
>>>> +49 (151) 6517 6917
>>>> mosipov@ilbers.de
>>>> http://ilbers.de/
>>>> Commercial register Munich, HRB 214197
>>>> General Manager: Baurzhan Ismagulov
>>>    
>>
>>
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

[PATCH v8 2/7] Remove duplicate code from apt-keyring generation

[Andreas J. Reichel]

From: Andreas Reichel <andreas.reichel.ext@siemens.com>

Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
---
 .../isar-bootstrap/isar-bootstrap.inc         | 23 +++++++++----------
 1 file changed, 11 insertions(+), 12 deletions(-)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 2910eea..4526aa7 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -35,22 +35,21 @@ inherit base-apt-helper
 python () {
     from urllib.parse import urlparse
     distro_apt_keys = d.getVar("DISTRO_APT_KEYS", False)
-    wd = d.getVar("WORKDIR", True)
+    aptkeys = []
+
     if distro_apt_keys:
-        d.setVar("DEBOOTSTRAP_KEYRING", "--keyring ${APTKEYRING}")
-        for key in distro_apt_keys.split():
-            url = urlparse(key)
-            filename = os.path.basename(url.path)
-            d.appendVar("SRC_URI", " %s" % key)
-            d.appendVar("APTKEYFILES", " %s" % filename)
+        aptkeys += distro_apt_keys.split()
     if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
         own_pub_key = d.getVar("BASE_REPO_KEY", False)
         if own_pub_key:
-            d.setVar("DEBOOTSTRAP_KEYRING", "--keyring ${APTKEYRING}")
-            for key in own_pub_key.split():
-                url = urlparse(key)
-                d.appendVar("SRC_URI", " " + key)
-                d.appendVar("APTKEYFILES", " " + wd + url.path)
+            aptkeys += own_pub_key.split()
+
+    d.setVar("DEBOOTSTRAP_KEYRING", "--keyring ${APTKEYRING}")
+    for key in aptkeys:
+        url = urlparse(key)
+        filename = os.path.basename(url.path)
+        d.appendVar("SRC_URI", " %s" % key)
+        d.appendVar("APTKEYFILES", " %s" % filename)
 }
 
 def aggregate_files(d, file_list, file_out):
-- 
2.21.0

[PATCH v8 3/7] Fix fetched key location in apt-keyring generator

[Andreas J. Reichel]

From: Andreas Reichel <andreas.reichel.ext@siemens.com>

Use bb.fetch2.Fetch to retrieve the final filename after it is
downloaded. This way we don't have to guess (wrongly), and also
additional SRC_URI parameters like subdir or filename are usable now.

Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
---
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 4526aa7..1f8f178 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -33,7 +33,6 @@ DISTRO_APT_PREMIRRORS ?= "${@ "http://ftp\.(\S+\.)?debian.org  file:///${REPO_BA
 inherit base-apt-helper
 
 python () {
-    from urllib.parse import urlparse
     distro_apt_keys = d.getVar("DISTRO_APT_KEYS", False)
     aptkeys = []
 
@@ -46,9 +45,9 @@ python () {
 
     d.setVar("DEBOOTSTRAP_KEYRING", "--keyring ${APTKEYRING}")
     for key in aptkeys:
-        url = urlparse(key)
-        filename = os.path.basename(url.path)
         d.appendVar("SRC_URI", " %s" % key)
+        fetcher = bb.fetch2.Fetch([key], d)
+        filename = fetcher.localpath(key)
         d.appendVar("APTKEYFILES", " %s" % filename)
 }
 
-- 
2.21.0

[PATCH v8 4/7] Use apt-key to generate keyrings

[Andreas J. Reichel]

From: Andreas Reichel <andreas.reichel.ext@siemens.com>

* Keyring names as well as variable names are now cleanly separated:
  DISTRO_BOOTSTRAP_KEYS, DISTRO_BOOTSTRAP_KEYRING and
  DISTRO_BOOTSTRAP_KEYFILES

  for bootstrapping the distro.

  THIRD_PARTY_APT_KEYS, THIRD_PARTY_APT_KEYRING and
  THIRD_PARTY_APT_KEYFILES

  for installing packages after bootstrapping.

Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
---
 meta/conf/bitbake.conf                        |  1 +
 .../isar-bootstrap/isar-bootstrap-host.bb     |  4 +-
 .../isar-bootstrap/isar-bootstrap-target.bb   |  4 +-
 .../isar-bootstrap/isar-bootstrap.inc         | 76 ++++++++++++++-----
 4 files changed, 60 insertions(+), 25 deletions(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 0e521bb..3782e5c 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -62,6 +62,7 @@ DEBDISTRONAME = "isar"
 # Isar apt repository paths
 REPO_ISAR_DIR = "${DEPLOY_DIR}/isar-apt/apt"
 REPO_ISAR_DB_DIR = "${DEPLOY_DIR}/isar-apt/db"
+THIRD_PARTY_APT_KEYRING = "/etc/apt/trusted.gpg.d/third_party.gpg"
 
 # Base apt repository paths
 REPO_BASE_DIR = "${DL_DIR}/base-apt/apt"
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
index 08b068f..7ee4c61 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
@@ -14,7 +14,7 @@ ISAR_BOOTSTRAP_LOCK = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}.lock"
 require isar-bootstrap.inc
 inherit isar-bootstrap-helper
 
-do_generate_keyring[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
+do_generate_keyrings[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 
 do_apt_config_prepare[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 do_apt_config_prepare[dirs] = "${WORKDIR}"
@@ -52,4 +52,4 @@ do_bootstrap[vardeps] += "HOST_DISTRO_APT_SOURCES"
 do_bootstrap() {
     isar_bootstrap --host
 }
-addtask bootstrap before do_build after do_generate_keyring
+addtask bootstrap before do_build after do_generate_keyrings
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
index 79f3e34..39f12b5 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
@@ -13,7 +13,7 @@ ISAR_BOOTSTRAP_LOCK = "${DEPLOY_DIR_BOOTSTRAP}/${DISTRO}-${DISTRO_ARCH}.lock"
 
 require isar-bootstrap.inc
 
-do_generate_keyring[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
+do_generate_keyrings[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 
 do_apt_config_prepare[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 do_apt_config_prepare[dirs] = "${WORKDIR}"
@@ -49,5 +49,5 @@ do_bootstrap[vardeps] += "DISTRO_APT_SOURCES"
 do_bootstrap() {
     isar_bootstrap
 }
-addtask bootstrap before do_build after do_generate_keyring
+addtask bootstrap before do_build after do_generate_keyrings
 
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 1f8f178..835ad52 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -22,33 +22,41 @@ APTPREFS = "${WORKDIR}/apt-preferences"
 APTSRCS = "${WORKDIR}/apt-sources"
 APTSRCS_INIT = "${WORKDIR}/apt-sources-init"
 BASEAPTSRCS = "${WORKDIR}/base-apt-sources"
-APTKEYFILES = ""
-APTKEYRING = "${WORKDIR}/apt-keyring.gpg"
-DEBOOTSTRAP_KEYRING = ""
+DISTRO_BOOTSTRAP_KEYFILES = ""
+THIRD_PARTY_APT_KEYFILES = ""
 DEPLOY_ISAR_BOOTSTRAP ?= ""
 DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales"
+DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = ",gnupg2"
 
 DISTRO_APT_PREMIRRORS ?= "${@ "http://ftp\.(\S+\.)?debian.org  file:///${REPO_BASE_DIR} \n" if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) else "" }"
 
 inherit base-apt-helper
 
 python () {
-    distro_apt_keys = d.getVar("DISTRO_APT_KEYS", False)
-    aptkeys = []
+    distro_bootstrap_keys = (d.getVar("DISTRO_BOOTSTRAP_KEYS", False) or "").split()
+
+    third_party_apt_keys = (d.getVar("THIRD_PARTY_APT_KEYS", False) or "").split() 
+
+    # The cached repo key can be both for bootstrapping and apt package
+    # installation afterwards. However, debootstrap will include the key into
+    # the rootfs automatically thus the right place is distro_bootstrap_keys.
 
-    if distro_apt_keys:
-        aptkeys += distro_apt_keys.split()
     if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
         own_pub_key = d.getVar("BASE_REPO_KEY", False)
         if own_pub_key:
-            aptkeys += own_pub_key.split()
+            distro_bootstrap_keys += own_pub_key.split()
+
+    for key in distro_bootstrap_keys:
+        d.appendVar("SRC_URI", " %s" % key)
+        fetcher = bb.fetch2.Fetch([key], d)
+        filename = fetcher.localpath(key)
+        d.appendVar("DISTRO_BOOTSTRAP_KEYFILES", " %s" % filename)
 
-    d.setVar("DEBOOTSTRAP_KEYRING", "--keyring ${APTKEYRING}")
-    for key in aptkeys:
+    for key in third_party_apt_keys:
         d.appendVar("SRC_URI", " %s" % key)
         fetcher = bb.fetch2.Fetch([key], d)
         filename = fetcher.localpath(key)
-        d.appendVar("APTKEYFILES", " %s" % filename)
+        d.appendVar("THIRD_PARTY_APT_KEYFILES", " %s" % filename)
 }
 
 def aggregate_files(d, file_list, file_out):
@@ -158,6 +166,14 @@ def get_distro_needs_https_support(d, is_host=False):
     else:
         return ""
 
+def get_distro_needs_gpg_support(d):
+    apt_keys = d.getVar("THIRD_PARTY_APT_KEYS", False)
+    if apt_keys and apt_keys != "":
+        return "gnupg"
+    return ""
+
+OVERRIDES_append = ":${@get_distro_needs_gpg_support(d)}"
+
 def get_distro_source(d, is_host):
     return get_distro_primary_source_entry(d, is_host)[0]
 
@@ -171,17 +187,26 @@ def get_distro_components_argument(d, is_host):
     else:
         return ""
 
-do_generate_keyring[dirs] = "${DL_DIR}"
-do_generate_keyring[vardeps] += "DISTRO_APT_KEYS"
-do_generate_keyring() {
-    if [ -n "${@d.getVar("APTKEYFILES", True) or ""}" ]; then
-        for keyfile in ${@d.getVar("APTKEYFILES", True)}; do
-           gpg --no-default-keyring --keyring "${APTKEYRING}" \
-               --no-tty --homedir "${DL_DIR}"  --import "$keyfile"
+APT_KEYS_DIR := "${WORKDIR}/aptkeys"
+DISTRO_BOOTSTRAP_KEYRING := "${WORKDIR}/distro-keyring.gpg"
+
+do_generate_keyrings[cleandirs] = "${APT_KEYS_DIR}"
+do_generate_keyrings[dirs] = "${DL_DIR}"
+do_generate_keyrings[vardeps] += "DISTRO_BOOTSTRAP_KEYS THIRD_PARTY_APT_KEYS"
+do_generate_keyrings() {
+    if [ -n "${@d.getVar("THIRD_PARTY_APT_KEYFILES", True) or ""}" ]; then
+        chmod 777 "${APT_KEYS_DIR}"
+        for keyfile in ${@d.getVar("THIRD_PARTY_APT_KEYFILES", True)}; do
+           cp "$keyfile" "${APT_KEYS_DIR}"/"$(basename "$keyfile")"
+        done
+    fi
+    if [ -n "${@d.getVar("DISTRO_BOOTSTRAP_KEYFILES", True) or ""}" ]; then
+        for keyfile in ${@d.getVar("DISTRO_BOOTSTRAP_KEYFILES", True)}; do
+           sudo apt-key --keyring "${DISTRO_BOOTSTRAP_KEYRING}" add $keyfile
         done
     fi
 }
-addtask generate_keyring before do_build after do_unpack
+addtask generate_keyrings before do_build after do_unpack
 
 
 
@@ -205,6 +230,9 @@ isar_bootstrap() {
         shift
     done
     debootstrap_args="--verbose --variant=minbase --include=${DISTRO_BOOTSTRAP_BASE_PACKAGES}"
+    if [ ! "x${DISTRO_BOOTSTRAP_KEYS}" = "x" ]; then
+        debootstrap_args="$debootstrap_args --keyring=${DISTRO_BOOTSTRAP_KEYRING}"
+    fi
     if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
         if [ -z "${BASE_REPO_KEY}" ] ; then
             debootstrap_args="$debootstrap_args --no-check-gpg"
@@ -221,7 +249,6 @@ isar_bootstrap() {
             if [ ${IS_HOST} ]; then
                 ${DEBOOTSTRAP} $debootstrap_args \
                                ${@get_distro_components_argument(d, True)} \
-                               ${DEBOOTSTRAP_KEYRING} \
                                "${@get_distro_suite(d, True)}" \
                                "${ROOTFSDIR}" \
                                "${@get_distro_source(d, True)}"
@@ -230,7 +257,6 @@ isar_bootstrap() {
                  "${DEBOOTSTRAP}" $debootstrap_args \
                                   --arch="${DISTRO_ARCH}" \
                                   ${@get_distro_components_argument(d, False)} \
-                                  ${DEBOOTSTRAP_KEYRING} \
                                   "${@get_distro_suite(d, False)}" \
                                   "${ROOTFSDIR}" \
                                   "${@get_distro_source(d, False)}"
@@ -259,6 +285,14 @@ isar_bootstrap() {
             mkdir -p "${ROOTFSDIR}/etc/apt/apt.conf.d"
             install -v -m644 "${WORKDIR}/isar-apt.conf" \
                              "${ROOTFSDIR}/etc/apt/apt.conf.d/50isar.conf"
+            find ${APT_KEYS_DIR}/ -type f | while read keyfile
+            do
+                kfn="$(basename $keyfile)"
+                cp $keyfile "${ROOTFSDIR}/tmp/$kfn"
+                sudo -E chroot "${ROOTFSDIR}" /usr/bin/apt-key \
+                   --keyring ${THIRD_PARTY_APT_KEYRING} add "/tmp/$kfn"
+                rm "${ROOTFSDIR}/tmp/$kfn"
+            done
 
             if [ "${@get_distro_suite(d, True)}" = "stretch" ] && [ "${@get_host_release().split('.')[0]}" -lt "4" ]; then
                 install -v -m644 "${WORKDIR}/isar-apt-fallback.conf" \
-- 
2.21.0

[PATCH v8 5/7] If we use a custom keyring debootstrap may fall to https

[Andreas J. Reichel]

From: Andreas Reichel <andreas.reichel.ext@siemens.com>

See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=891009

So if we have something in the distro bootstrap keyring, append
https-support to OVERRIDES.

Furthermore, the conditional append for https-support was missing
in qemuamd64-stretch.conf, thus, remove this from all the distros
and put it into the isar-bootstrap.inc.

Furthermore, packages are comma-, not space-separated.

Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
---
 meta-isar/conf/multiconfig/qemuamd64-buster.conf    | 1 -
 meta-isar/conf/multiconfig/qemuamd64-jessie.conf    | 1 -
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 8 ++++++++
 3 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/meta-isar/conf/multiconfig/qemuamd64-buster.conf b/meta-isar/conf/multiconfig/qemuamd64-buster.conf
index 63df75c..da90993 100644
--- a/meta-isar/conf/multiconfig/qemuamd64-buster.conf
+++ b/meta-isar/conf/multiconfig/qemuamd64-buster.conf
@@ -18,4 +18,3 @@ QEMU_MACHINE ?= "q35"
 QEMU_CPU ?= ""
 QEMU_DISK_ARGS ?= "-hda ##ROOTFS_IMAGE## -bios /usr/local/share/ovmf/OVMF.fd"
 
-DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = " apt-transport-https ca-certificates"
diff --git a/meta-isar/conf/multiconfig/qemuamd64-jessie.conf b/meta-isar/conf/multiconfig/qemuamd64-jessie.conf
index d1335ff..42c71df 100644
--- a/meta-isar/conf/multiconfig/qemuamd64-jessie.conf
+++ b/meta-isar/conf/multiconfig/qemuamd64-jessie.conf
@@ -15,4 +15,3 @@ QEMU_MACHINE ?= "pc"
 QEMU_CPU ?= ""
 QEMU_DISK_ARGS ?= "-hda ##ROOTFS_IMAGE##"
 
-DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = " apt-transport-https ca-certificates"
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 835ad52..c6c3cde 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -27,6 +27,7 @@ THIRD_PARTY_APT_KEYFILES = ""
 DEPLOY_ISAR_BOOTSTRAP ?= ""
 DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales"
 DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = ",gnupg2"
+DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = ",apt-transport-https,ca-certificates"
 
 DISTRO_APT_PREMIRRORS ?= "${@ "http://ftp\.(\S+\.)?debian.org  file:///${REPO_BASE_DIR} \n" if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) else "" }"
 
@@ -161,6 +162,13 @@ def get_distro_have_https_source(d, is_host=False):
     return any(source[2].startswith("https://") for source in generate_distro_sources(d, is_host))
 
 def get_distro_needs_https_support(d, is_host=False):
+    distro_bootstrap_keys = d.getVar("DISTRO_BOOTSTRAP_KEYS", False)
+    if distro_bootstrap_keys and distro_bootstrap_keys != "":
+        # debootstrap falls back to https if there is no
+        # 'reliable' keyring, whatever that means, but it happened
+        # https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=891009
+        return "https-support"
+
     if get_distro_have_https_source(d, is_host):
         return "https-support"
     else:
-- 
2.21.0

[PATCH v8 6/7] raspbian-jessie: Use DISTRO_BOOTSTRAP_KEYS

[Andreas J. Reichel]

From: Andreas Reichel <andreas.reichel.ext@siemens.com>

Use new DISTRO_BOOTSTRAP_KEYS instead of DISTRO_APT_KEYS

Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
---
 meta-isar/conf/distro/raspbian-jessie.conf | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta-isar/conf/distro/raspbian-jessie.conf b/meta-isar/conf/distro/raspbian-jessie.conf
index 5e907e5..a6a5080 100644
--- a/meta-isar/conf/distro/raspbian-jessie.conf
+++ b/meta-isar/conf/distro/raspbian-jessie.conf
@@ -7,6 +7,6 @@ BASE_DISTRO = "raspbian"
 BASE_DISTRO_CODENAME = "jessie"
 
 DISTRO_APT_SOURCES += "conf/distro/raspbian-jessie.list"
-DISTRO_APT_KEYS += "https://archive.raspbian.org/raspbian.public.key;sha256sum=ca59cd4f2bcbc3a1d41ba6815a02a8dc5c175467a59bd87edeac458f4a5345de"
+DISTRO_BOOTSTRAP_KEYS += "https://archive.raspbian.org/raspbian.public.key;sha256sum=ca59cd4f2bcbc3a1d41ba6815a02a8dc5c175467a59bd87edeac458f4a5345de"
 DISTRO_CONFIG_SCRIPT ?= "${LAYERDIR_isar}/conf/distro/raspbian-configscript.sh"
 DISTRO_KERNELS ?= "rpi rpi2 rpi-rpfv rpi2-rpfv"
-- 
2.21.0

[PATCH v8 7/7] docs: Update user_manual.md

[Andreas J. Reichel]

From: Andreas Reichel <andreas.reichel.ext@siemens.com>

Explain DISTRO_BOOTSTRAP_KEYS and THIRD_PARTY_APT_KEYS.

Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
---
 doc/user_manual.md | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 6fe4d83..8db636b 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -374,14 +374,17 @@ Isar can generate various images types for specific machine. The type of the ima
 The distro is defined by the set of the following variables:
 
  - `DISTRO_APT_SOURCES` - List of apt source files
- - `DISTRO_APT_KEYS` - List of gpg key URIs used to verify apt repos
+ - `DISTRO_BOOTSTRAP_KEYS` - List of gpg key URIs used to verify the apt repo used during bootstrapping
+ - `THIRD_PARTY_APT_KEYS` - List of gpg key URIs used to verify apt repos for apt installation after bootstrapping
  - `DISTRO_APT_PREFERENCES` - List of apt preference files
  - `DISTRO_KERNELS` - List of supported kernel suffixes
 
+The first entry of the apt sources is used for bootstrapping.
+
 Below is an example for Raspbian Jessie:
 ```
 DISTRO_APT_SOURCES += "conf/distro/raspbian-jessie.list"
-DISTRO_APT_KEYS += "https://archive.raspbian.org/raspbian.public.key;sha256sum=ca59cd4f2bcbc3a1d41ba6815a02a8dc5c175467a59bd87edeac458f4a5345de"
+DISTRO_BOOTSTRAP_KEYS += "https://archive.raspbian.org/raspbian.public.key;sha256sum=ca59cd4f2bcbc3a1d41ba6815a02a8dc5c175467a59bd87edeac458f4a5345de"
 DISTRO_CONFIG_SCRIPT?= "raspbian-configscript.sh"
 DISTRO_KERNELS ?= "rpi rpi2 rpi-rpfv rpi2-rpfv"
 ```
-- 
2.21.0

Re: [PATCH v8 0/7] Fix usage of additional apt keys and repos

[Andreas Reichel]

> Not working yet:
>   qemuarm64-stretch with BASE_REPO_KEY and do_cache_base_repo
> 
> | gpgme gave error GPGME:54:  Unusable secret key
> 
> I have created a keypair inside the build container and exported the
> public key to a file "blabla.key". Then I set
> 
> BASE_REPO_KEY = "file:///build/blabla.key"
> 
> Any idea?
> 
There was actually no problem anymore. The KEY had to be in the gpg key
ring which was expected by the normal user in /tmp/tmpb6et85_1/.gnupg, not
/home/builder/.gnupg. After readding the secrect key for the normal
build user, it worked.

I have just triggered a CI build on ilbers-ci. After that is green,
you can apply my patchset.

Regards
Andreas

> Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
> 
> Andreas Reichel (7):
>   Revert "isar-bootstrap: Allow to set local keys in DISTRO_APT_KEYS"
>   Remove duplicate code from apt-keyring generation
>   Fix fetched key location in apt-keyring generator
>   Use apt-key to generate keyrings
>   If we use a custom keyring debootstrap may fall to https
>   raspbian-jessie: Use DISTRO_BOOTSTRAP_KEYS
>   docs: Update user_manual.md
> 
>  doc/user_manual.md                            |  7 +-
>  meta-isar/conf/distro/raspbian-jessie.conf    |  2 +-
>  .../conf/multiconfig/qemuamd64-buster.conf    |  1 -
>  .../conf/multiconfig/qemuamd64-jessie.conf    |  1 -
>  meta/conf/bitbake.conf                        |  1 +
>  .../isar-bootstrap/isar-bootstrap-host.bb     |  4 +-
>  .../isar-bootstrap/isar-bootstrap-target.bb   |  4 +-
>  .../isar-bootstrap/isar-bootstrap.inc         | 95 +++++++++++++------
>  8 files changed, 79 insertions(+), 36 deletions(-)
> 
> -- 
> 2.21.0
> 

-- 
Andreas Reichel
Dipl.-Phys. (Univ.)
Software Consultant

Andreas.Reichel@tngtech.com, +49-174-3180074
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterfoehring
Geschaeftsfuehrer: Henrik Klagges, Dr. Robert Dahlke, Gerhard Mueller
Sitz: Unterfoehring * Amtsgericht Muenchen * HRB 135082

Re: [PATCH v8 0/7] Fix usage of additional apt keys and repos

[Maxim Yu. Osipov]

On 3/25/19 11:19 AM, Andreas Reichel wrote:
>> Not working yet:
>>    qemuarm64-stretch with BASE_REPO_KEY and do_cache_base_repo
>>
>> | gpgme gave error GPGME:54:  Unusable secret key
>>
>> I have created a keypair inside the build container and exported the
>> public key to a file "blabla.key". Then I set
>>
>> BASE_REPO_KEY = "file:///build/blabla.key"
>>
>> Any idea?
>>
> There was actually no problem anymore. The KEY had to be in the gpg key
> ring which was expected by the normal user in /tmp/tmpb6et85_1/.gnupg, not
> /home/builder/.gnupg. After readding the secrect key for the normal
> build user, it worked.
> 
> I have just triggered a CI build on ilbers-ci. After that is green,
> you can apply my patchset.

Just FYI:

I test patchsets independently before applying them into the tree.

Meanwhile I encourage people to use CI build before sending patchset to 
the mailing list (if this is not RFC) to avoid unnecessary patchsets 
iterations.


The automated CI test procedure consists actually from the two steps:

1) "fast" CI build/smoke test (by passing the key '-f' to corresponding 
ci_build.sh and vm_smoke_test scripts) - it tests cross compilation for 
three supported stretch QEMU targets and one de0-nano-soc target.

2) "standard" CI build - it tests native build for the almost full set
of QEMU targets.


If the new feature is added to the ISAR it's always desirable to add 
corresponding test case into the CI.


Regards,
Maxim.



> Regards
> Andreas
> 
>> Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
>>
>> Andreas Reichel (7):
>>    Revert "isar-bootstrap: Allow to set local keys in DISTRO_APT_KEYS"
>>    Remove duplicate code from apt-keyring generation
>>    Fix fetched key location in apt-keyring generator
>>    Use apt-key to generate keyrings
>>    If we use a custom keyring debootstrap may fall to https
>>    raspbian-jessie: Use DISTRO_BOOTSTRAP_KEYS
>>    docs: Update user_manual.md
>>
>>   doc/user_manual.md                            |  7 +-
>>   meta-isar/conf/distro/raspbian-jessie.conf    |  2 +-
>>   .../conf/multiconfig/qemuamd64-buster.conf    |  1 -
>>   .../conf/multiconfig/qemuamd64-jessie.conf    |  1 -
>>   meta/conf/bitbake.conf                        |  1 +
>>   .../isar-bootstrap/isar-bootstrap-host.bb     |  4 +-
>>   .../isar-bootstrap/isar-bootstrap-target.bb   |  4 +-
>>   .../isar-bootstrap/isar-bootstrap.inc         | 95 +++++++++++++------
>>   8 files changed, 79 insertions(+), 36 deletions(-)
>>
>> -- 
>> 2.21.0
>>
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: [PATCH v8 0/7] Fix usage of additional apt keys and repos

[Andreas Reichel]

On Mon, Mar 25, 2019 at 11:35:33AM +0100, Maxim Yu. Osipov wrote:
> On 3/25/19 11:19 AM, Andreas Reichel wrote:
> > > Not working yet:
> > >    qemuarm64-stretch with BASE_REPO_KEY and do_cache_base_repo
> > > 
> > > | gpgme gave error GPGME:54:  Unusable secret key
> > > 
> > > I have created a keypair inside the build container and exported the
> > > public key to a file "blabla.key". Then I set
> > > 
> > > BASE_REPO_KEY = "file:///build/blabla.key"
> > > 
> > > Any idea?
> > > 
> > There was actually no problem anymore. The KEY had to be in the gpg key
> > ring which was expected by the normal user in /tmp/tmpb6et85_1/.gnupg, not
> > /home/builder/.gnupg. After readding the secrect key for the normal
> > build user, it worked.
> > 
> > I have just triggered a CI build on ilbers-ci. After that is green,
> > you can apply my patchset.
> 
> Just FYI:
> 
> I test patchsets independently before applying them into the tree.
> 
> Meanwhile I encourage people to use CI build before sending patchset to the
> mailing list (if this is not RFC) to avoid unnecessary patchsets iterations.
> 
> 
> The automated CI test procedure consists actually from the two steps:
> 
> 1) "fast" CI build/smoke test (by passing the key '-f' to corresponding
> ci_build.sh and vm_smoke_test scripts) - it tests cross compilation for
> three supported stretch QEMU targets and one de0-nano-soc target.
> 
> 2) "standard" CI build - it tests native build for the almost full set
> of QEMU targets.
> 
> 
> If the new feature is added to the ISAR it's always desirable to add
> corresponding test case into the CI.
> 
In this case it means we/I should add a test case where the docker
upstream repo is added and an image with docker is built.

Let's say it is a generalization of an existing feature :)

Regards,
Andreas
> 
> Regards,
> Maxim.
> 
> 
> 
> > Regards
> > Andreas
> > 
> > > Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
> > > 
> > > Andreas Reichel (7):
> > >    Revert "isar-bootstrap: Allow to set local keys in DISTRO_APT_KEYS"
> > >    Remove duplicate code from apt-keyring generation
> > >    Fix fetched key location in apt-keyring generator
> > >    Use apt-key to generate keyrings
> > >    If we use a custom keyring debootstrap may fall to https
> > >    raspbian-jessie: Use DISTRO_BOOTSTRAP_KEYS
> > >    docs: Update user_manual.md
> > > 
> > >   doc/user_manual.md                            |  7 +-
> > >   meta-isar/conf/distro/raspbian-jessie.conf    |  2 +-
> > >   .../conf/multiconfig/qemuamd64-buster.conf    |  1 -
> > >   .../conf/multiconfig/qemuamd64-jessie.conf    |  1 -
> > >   meta/conf/bitbake.conf                        |  1 +
> > >   .../isar-bootstrap/isar-bootstrap-host.bb     |  4 +-
> > >   .../isar-bootstrap/isar-bootstrap-target.bb   |  4 +-
> > >   .../isar-bootstrap/isar-bootstrap.inc         | 95 +++++++++++++------
> > >   8 files changed, 79 insertions(+), 36 deletions(-)
> > > 
> > > -- 
> > > 2.21.0
> > > 
> > 
> 
> 
> -- 
> Maxim Osipov
> ilbers GmbH
> Maria-Merian-Str. 8
> 85521 Ottobrunn
> Germany
> +49 (151) 6517 6917
> mosipov@ilbers.de
> http://ilbers.de/
> Commercial register Munich, HRB 214197
> General Manager: Baurzhan Ismagulov

-- 
Andreas Reichel
Dipl.-Phys. (Univ.)
Software Consultant

Andreas.Reichel@tngtech.com, +49-174-3180074
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterfoehring
Geschaeftsfuehrer: Henrik Klagges, Dr. Robert Dahlke, Gerhard Mueller
Sitz: Unterfoehring * Amtsgericht Muenchen * HRB 135082

Re: [PATCH v8 0/7] Fix usage of additional apt keys and repos

[Maxim Yu. Osipov]

On 3/25/19 12:28 PM, Andreas Reichel wrote:
> On Mon, Mar 25, 2019 at 11:35:33AM +0100, Maxim Yu. Osipov wrote:
>> On 3/25/19 11:19 AM, Andreas Reichel wrote:
>>>> Not working yet:
>>>>     qemuarm64-stretch with BASE_REPO_KEY and do_cache_base_repo
>>>>
>>>> | gpgme gave error GPGME:54:  Unusable secret key
>>>>
>>>> I have created a keypair inside the build container and exported the
>>>> public key to a file "blabla.key". Then I set
>>>>
>>>> BASE_REPO_KEY = "file:///build/blabla.key"
>>>>
>>>> Any idea?
>>>>
>>> There was actually no problem anymore. The KEY had to be in the gpg key
>>> ring which was expected by the normal user in /tmp/tmpb6et85_1/.gnupg, not
>>> /home/builder/.gnupg. After readding the secrect key for the normal
>>> build user, it worked.
>>>
>>> I have just triggered a CI build on ilbers-ci. After that is green,
>>> you can apply my patchset.
>>
>> Just FYI:
>>
>> I test patchsets independently before applying them into the tree.
>>
>> Meanwhile I encourage people to use CI build before sending patchset to the
>> mailing list (if this is not RFC) to avoid unnecessary patchsets iterations.
>>
>>
>> The automated CI test procedure consists actually from the two steps:
>>
>> 1) "fast" CI build/smoke test (by passing the key '-f' to corresponding
>> ci_build.sh and vm_smoke_test scripts) - it tests cross compilation for
>> three supported stretch QEMU targets and one de0-nano-soc target.
>>
>> 2) "standard" CI build - it tests native build for the almost full set
>> of QEMU targets.
>>
>>
>> If the new feature is added to the ISAR it's always desirable to add
>> corresponding test case into the CI.
>>
> In this case it means we/I should add a test case where the docker
> upstream repo is added and an image with docker is built.

I hope that your feature is generic enough to add some simpler (not 
docker) third party repo for testing purposes.

Regards,
Maxim.


> Let's say it is a generalization of an existing feature :)
> 
> Regards,
> Andreas
>>
>> Regards,
>> Maxim.
>>
>>
>>
>>> Regards
>>> Andreas
>>>
>>>> Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
>>>>
>>>> Andreas Reichel (7):
>>>>     Revert "isar-bootstrap: Allow to set local keys in DISTRO_APT_KEYS"
>>>>     Remove duplicate code from apt-keyring generation
>>>>     Fix fetched key location in apt-keyring generator
>>>>     Use apt-key to generate keyrings
>>>>     If we use a custom keyring debootstrap may fall to https
>>>>     raspbian-jessie: Use DISTRO_BOOTSTRAP_KEYS
>>>>     docs: Update user_manual.md
>>>>
>>>>    doc/user_manual.md                            |  7 +-
>>>>    meta-isar/conf/distro/raspbian-jessie.conf    |  2 +-
>>>>    .../conf/multiconfig/qemuamd64-buster.conf    |  1 -
>>>>    .../conf/multiconfig/qemuamd64-jessie.conf    |  1 -
>>>>    meta/conf/bitbake.conf                        |  1 +
>>>>    .../isar-bootstrap/isar-bootstrap-host.bb     |  4 +-
>>>>    .../isar-bootstrap/isar-bootstrap-target.bb   |  4 +-
>>>>    .../isar-bootstrap/isar-bootstrap.inc         | 95 +++++++++++++------
>>>>    8 files changed, 79 insertions(+), 36 deletions(-)
>>>>
>>>> -- 
>>>> 2.21.0
>>>>
>>>
>>
>>
>> -- 
>> Maxim Osipov
>> ilbers GmbH
>> Maria-Merian-Str. 8
>> 85521 Ottobrunn
>> Germany
>> +49 (151) 6517 6917
>> mosipov@ilbers.de
>> http://ilbers.de/
>> Commercial register Munich, HRB 214197
>> General Manager: Baurzhan Ismagulov
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: [PATCH v8 0/7] Fix usage of additional apt keys and repos

[Henning Schild]

Am Mon, 25 Mar 2019 12:39:01 +0100
schrieb "Maxim Yu. Osipov" <mosipov@ilbers.de>:

> On 3/25/19 12:28 PM, Andreas Reichel wrote:
> > On Mon, Mar 25, 2019 at 11:35:33AM +0100, Maxim Yu. Osipov wrote:  
> >> On 3/25/19 11:19 AM, Andreas Reichel wrote:  
> >>>> Not working yet:
> >>>>     qemuarm64-stretch with BASE_REPO_KEY and do_cache_base_repo
> >>>>
> >>>> | gpgme gave error GPGME:54:  Unusable secret key
> >>>>
> >>>> I have created a keypair inside the build container and exported
> >>>> the public key to a file "blabla.key". Then I set
> >>>>
> >>>> BASE_REPO_KEY = "file:///build/blabla.key"
> >>>>
> >>>> Any idea?
> >>>>  
> >>> There was actually no problem anymore. The KEY had to be in the
> >>> gpg key ring which was expected by the normal user
> >>> in /tmp/tmpb6et85_1/.gnupg, not /home/builder/.gnupg. After
> >>> readding the secrect key for the normal build user, it worked.
> >>>
> >>> I have just triggered a CI build on ilbers-ci. After that is
> >>> green, you can apply my patchset.  
> >>
> >> Just FYI:
> >>
> >> I test patchsets independently before applying them into the tree.
> >>
> >> Meanwhile I encourage people to use CI build before sending
> >> patchset to the mailing list (if this is not RFC) to avoid
> >> unnecessary patchsets iterations.
> >>
> >>
> >> The automated CI test procedure consists actually from the two
> >> steps:
> >>
> >> 1) "fast" CI build/smoke test (by passing the key '-f' to
> >> corresponding ci_build.sh and vm_smoke_test scripts) - it tests
> >> cross compilation for three supported stretch QEMU targets and one
> >> de0-nano-soc target.
> >>
> >> 2) "standard" CI build - it tests native build for the almost full
> >> set of QEMU targets.
> >>
> >>
> >> If the new feature is added to the ISAR it's always desirable to
> >> add corresponding test case into the CI.
> >>  
> > In this case it means we/I should add a test case where the docker
> > upstream repo is added and an image with docker is built.  
> 
> I hope that your feature is generic enough to add some simpler (not 
> docker) third party repo for testing purposes.

On the repo level they probably all are equally "simple". However, i
would not trust the docker one to work repeatedly and stable for all
suites/arches. I know it provides broken init scripts, that suggest they
do not do much more than "works for me" testing.

This could be a better example:
https://wiki.x2go.org/doku.php/wiki:repositories:debian

Henning

> Regards,
> Maxim.
> 
> 
> > Let's say it is a generalization of an existing feature :)
> > 
> > Regards,
> > Andreas  
> >>
> >> Regards,
> >> Maxim.
> >>
> >>
> >>  
> >>> Regards
> >>> Andreas
> >>>  
> >>>> Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
> >>>>
> >>>> Andreas Reichel (7):
> >>>>     Revert "isar-bootstrap: Allow to set local keys in
> >>>> DISTRO_APT_KEYS" Remove duplicate code from apt-keyring
> >>>> generation Fix fetched key location in apt-keyring generator
> >>>>     Use apt-key to generate keyrings
> >>>>     If we use a custom keyring debootstrap may fall to https
> >>>>     raspbian-jessie: Use DISTRO_BOOTSTRAP_KEYS
> >>>>     docs: Update user_manual.md
> >>>>
> >>>>    doc/user_manual.md                            |  7 +-
> >>>>    meta-isar/conf/distro/raspbian-jessie.conf    |  2 +-
> >>>>    .../conf/multiconfig/qemuamd64-buster.conf    |  1 -
> >>>>    .../conf/multiconfig/qemuamd64-jessie.conf    |  1 -
> >>>>    meta/conf/bitbake.conf                        |  1 +
> >>>>    .../isar-bootstrap/isar-bootstrap-host.bb     |  4 +-
> >>>>    .../isar-bootstrap/isar-bootstrap-target.bb   |  4 +-
> >>>>    .../isar-bootstrap/isar-bootstrap.inc         | 95
> >>>> +++++++++++++------ 8 files changed, 79 insertions(+), 36
> >>>> deletions(-)
> >>>>
> >>>> -- 
> >>>> 2.21.0
> >>>>  
> >>>  
> >>
> >>
> >> -- 
> >> Maxim Osipov
> >> ilbers GmbH
> >> Maria-Merian-Str. 8
> >> 85521 Ottobrunn
> >> Germany
> >> +49 (151) 6517 6917
> >> mosipov@ilbers.de
> >> http://ilbers.de/
> >> Commercial register Munich, HRB 214197
> >> General Manager: Baurzhan Ismagulov  
> >   
> 
> 

Re: [PATCH v8 0/7] Fix usage of additional apt keys and repos

[Andreas Reichel]

On Fri, Apr 12, 2019 at 02:52:28PM +0200, Henning Schild wrote:
> Am Mon, 25 Mar 2019 12:39:01 +0100
> schrieb "Maxim Yu. Osipov" <mosipov@ilbers.de>:
> 
> > On 3/25/19 12:28 PM, Andreas Reichel wrote:
> > > On Mon, Mar 25, 2019 at 11:35:33AM +0100, Maxim Yu. Osipov wrote:  
> > >> On 3/25/19 11:19 AM, Andreas Reichel wrote:  
> > >>>> Not working yet:
> > >>>>     qemuarm64-stretch with BASE_REPO_KEY and do_cache_base_repo
> > >>>>
> > >>>> | gpgme gave error GPGME:54:  Unusable secret key
> > >>>>
> > >>>> I have created a keypair inside the build container and exported
> > >>>> the public key to a file "blabla.key". Then I set
> > >>>>
> > >>>> BASE_REPO_KEY = "file:///build/blabla.key"
> > >>>>
> > >>>> Any idea?
> > >>>>  
> > >>> There was actually no problem anymore. The KEY had to be in the
> > >>> gpg key ring which was expected by the normal user
> > >>> in /tmp/tmpb6et85_1/.gnupg, not /home/builder/.gnupg. After
> > >>> readding the secrect key for the normal build user, it worked.
> > >>>
> > >>> I have just triggered a CI build on ilbers-ci. After that is
> > >>> green, you can apply my patchset.  
> > >>
> > >> Just FYI:
> > >>
> > >> I test patchsets independently before applying them into the tree.
> > >>
> > >> Meanwhile I encourage people to use CI build before sending
> > >> patchset to the mailing list (if this is not RFC) to avoid
> > >> unnecessary patchsets iterations.
> > >>
> > >>
> > >> The automated CI test procedure consists actually from the two
> > >> steps:
> > >>
> > >> 1) "fast" CI build/smoke test (by passing the key '-f' to
> > >> corresponding ci_build.sh and vm_smoke_test scripts) - it tests
> > >> cross compilation for three supported stretch QEMU targets and one
> > >> de0-nano-soc target.
> > >>
> > >> 2) "standard" CI build - it tests native build for the almost full
> > >> set of QEMU targets.
> > >>
> > >>
> > >> If the new feature is added to the ISAR it's always desirable to
> > >> add corresponding test case into the CI.
> > >>  
> > > In this case it means we/I should add a test case where the docker
> > > upstream repo is added and an image with docker is built.  
> > 
> > I hope that your feature is generic enough to add some simpler (not 
> > docker) third party repo for testing purposes.
> 
> On the repo level they probably all are equally "simple". However, i
> would not trust the docker one to work repeatedly and stable for all
> suites/arches. I know it provides broken init scripts, that suggest they
> do not do much more than "works for me" testing.
> 
> This could be a better example:
> https://wiki.x2go.org/doku.php/wiki:repositories:debian
> 

This seems to be a worst-case example :), since the keys are not provided
via URL but via package/key server, where the key-server protocol is
blocked from Siemens intranet. So this has nothing to do with any apt
key URI, but provides a completely new case to be tested and is out of
scope for my patch set.

Andreas

> Henning
> 
> > Regards,
> > Maxim.
> > 
> > 
> > > Let's say it is a generalization of an existing feature :)
> > > 
> > > Regards,
> > > Andreas  
> > >>
> > >> Regards,
> > >> Maxim.
> > >>
> > >>
> > >>  
> > >>> Regards
> > >>> Andreas
> > >>>  
> > >>>> Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
> > >>>>
> > >>>> Andreas Reichel (7):
> > >>>>     Revert "isar-bootstrap: Allow to set local keys in
> > >>>> DISTRO_APT_KEYS" Remove duplicate code from apt-keyring
> > >>>> generation Fix fetched key location in apt-keyring generator
> > >>>>     Use apt-key to generate keyrings
> > >>>>     If we use a custom keyring debootstrap may fall to https
> > >>>>     raspbian-jessie: Use DISTRO_BOOTSTRAP_KEYS
> > >>>>     docs: Update user_manual.md
> > >>>>
> > >>>>    doc/user_manual.md                            |  7 +-
> > >>>>    meta-isar/conf/distro/raspbian-jessie.conf    |  2 +-
> > >>>>    .../conf/multiconfig/qemuamd64-buster.conf    |  1 -
> > >>>>    .../conf/multiconfig/qemuamd64-jessie.conf    |  1 -
> > >>>>    meta/conf/bitbake.conf                        |  1 +
> > >>>>    .../isar-bootstrap/isar-bootstrap-host.bb     |  4 +-
> > >>>>    .../isar-bootstrap/isar-bootstrap-target.bb   |  4 +-
> > >>>>    .../isar-bootstrap/isar-bootstrap.inc         | 95
> > >>>> +++++++++++++------ 8 files changed, 79 insertions(+), 36
> > >>>> deletions(-)
> > >>>>
> > >>>> -- 
> > >>>> 2.21.0
> > >>>>  
> > >>>  
> > >>
> > >>
> > >> -- 
> > >> Maxim Osipov
> > >> ilbers GmbH
> > >> Maria-Merian-Str. 8
> > >> 85521 Ottobrunn
> > >> Germany
> > >> +49 (151) 6517 6917
> > >> mosipov@ilbers.de
> > >> http://ilbers.de/
> > >> Commercial register Munich, HRB 214197
> > >> General Manager: Baurzhan Ismagulov  
> > >   
> > 
> > 
> 

-- 
Andreas Reichel
Dipl.-Phys. (Univ.)
Software Consultant

Andreas.Reichel@tngtech.com, +49-174-3180074
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterfoehring
Geschaeftsfuehrer: Henrik Klagges, Dr. Robert Dahlke, Gerhard Mueller
Sitz: Unterfoehring * Amtsgericht Muenchen * HRB 135082

Re: [PATCH v8 0/7] Fix usage of additional apt keys and repos

[Henning Schild]

Am Mon, 15 Apr 2019 15:14:39 +0200
schrieb Andreas Reichel <andreas.reichel.ext@siemens.com>:

> On Fri, Apr 12, 2019 at 02:52:28PM +0200, Henning Schild wrote:
> > Am Mon, 25 Mar 2019 12:39:01 +0100
> > schrieb "Maxim Yu. Osipov" <mosipov@ilbers.de>:
> >   
> > > On 3/25/19 12:28 PM, Andreas Reichel wrote:  
> > > > On Mon, Mar 25, 2019 at 11:35:33AM +0100, Maxim Yu. Osipov
> > > > wrote:    
> > > >> On 3/25/19 11:19 AM, Andreas Reichel wrote:    
> > > >>>> Not working yet:
> > > >>>>     qemuarm64-stretch with BASE_REPO_KEY and
> > > >>>> do_cache_base_repo
> > > >>>>
> > > >>>> | gpgme gave error GPGME:54:  Unusable secret key
> > > >>>>
> > > >>>> I have created a keypair inside the build container and
> > > >>>> exported the public key to a file "blabla.key". Then I set
> > > >>>>
> > > >>>> BASE_REPO_KEY = "file:///build/blabla.key"
> > > >>>>
> > > >>>> Any idea?
> > > >>>>    
> > > >>> There was actually no problem anymore. The KEY had to be in
> > > >>> the gpg key ring which was expected by the normal user
> > > >>> in /tmp/tmpb6et85_1/.gnupg, not /home/builder/.gnupg. After
> > > >>> readding the secrect key for the normal build user, it worked.
> > > >>>
> > > >>> I have just triggered a CI build on ilbers-ci. After that is
> > > >>> green, you can apply my patchset.    
> > > >>
> > > >> Just FYI:
> > > >>
> > > >> I test patchsets independently before applying them into the
> > > >> tree.
> > > >>
> > > >> Meanwhile I encourage people to use CI build before sending
> > > >> patchset to the mailing list (if this is not RFC) to avoid
> > > >> unnecessary patchsets iterations.
> > > >>
> > > >>
> > > >> The automated CI test procedure consists actually from the two
> > > >> steps:
> > > >>
> > > >> 1) "fast" CI build/smoke test (by passing the key '-f' to
> > > >> corresponding ci_build.sh and vm_smoke_test scripts) - it tests
> > > >> cross compilation for three supported stretch QEMU targets and
> > > >> one de0-nano-soc target.
> > > >>
> > > >> 2) "standard" CI build - it tests native build for the almost
> > > >> full set of QEMU targets.
> > > >>
> > > >>
> > > >> If the new feature is added to the ISAR it's always desirable
> > > >> to add corresponding test case into the CI.
> > > >>    
> > > > In this case it means we/I should add a test case where the
> > > > docker upstream repo is added and an image with docker is
> > > > built.    
> > > 
> > > I hope that your feature is generic enough to add some simpler
> > > (not docker) third party repo for testing purposes.  
> > 
> > On the repo level they probably all are equally "simple". However, i
> > would not trust the docker one to work repeatedly and stable for all
> > suites/arches. I know it provides broken init scripts, that suggest
> > they do not do much more than "works for me" testing.
> > 
> > This could be a better example:
> > https://wiki.x2go.org/doku.php/wiki:repositories:debian
> >   
> 
> This seems to be a worst-case example :), since the keys are not
> provided via URL but via package/key server, where the key-server
> protocol is blocked from Siemens intranet. So this has nothing to do
> with any apt key URI, but provides a completely new case to be tested
> and is out of scope for my patch set.

Well another mechanism for this one, meaning that this one was not a
good candidate. But meaning that docker should have gone upstream for
CI tests!

Henning

> Andreas
> 
> > Henning
> >   
> > > Regards,
> > > Maxim.
> > > 
> > >   
> > > > Let's say it is a generalization of an existing feature :)
> > > > 
> > > > Regards,
> > > > Andreas    
> > > >>
> > > >> Regards,
> > > >> Maxim.
> > > >>
> > > >>
> > > >>    
> > > >>> Regards
> > > >>> Andreas
> > > >>>    
> > > >>>> Signed-off-by: Andreas Reichel
> > > >>>> <andreas.reichel.ext@siemens.com>
> > > >>>>
> > > >>>> Andreas Reichel (7):
> > > >>>>     Revert "isar-bootstrap: Allow to set local keys in
> > > >>>> DISTRO_APT_KEYS" Remove duplicate code from apt-keyring
> > > >>>> generation Fix fetched key location in apt-keyring generator
> > > >>>>     Use apt-key to generate keyrings
> > > >>>>     If we use a custom keyring debootstrap may fall to https
> > > >>>>     raspbian-jessie: Use DISTRO_BOOTSTRAP_KEYS
> > > >>>>     docs: Update user_manual.md
> > > >>>>
> > > >>>>    doc/user_manual.md                            |  7 +-
> > > >>>>    meta-isar/conf/distro/raspbian-jessie.conf    |  2 +-
> > > >>>>    .../conf/multiconfig/qemuamd64-buster.conf    |  1 -
> > > >>>>    .../conf/multiconfig/qemuamd64-jessie.conf    |  1 -
> > > >>>>    meta/conf/bitbake.conf                        |  1 +
> > > >>>>    .../isar-bootstrap/isar-bootstrap-host.bb     |  4 +-
> > > >>>>    .../isar-bootstrap/isar-bootstrap-target.bb   |  4 +-
> > > >>>>    .../isar-bootstrap/isar-bootstrap.inc         | 95
> > > >>>> +++++++++++++------ 8 files changed, 79 insertions(+), 36
> > > >>>> deletions(-)
> > > >>>>
> > > >>>> -- 
> > > >>>> 2.21.0
> > > >>>>    
> > > >>>    
> > > >>
> > > >>
> > > >> -- 
> > > >> Maxim Osipov
> > > >> ilbers GmbH
> > > >> Maria-Merian-Str. 8
> > > >> 85521 Ottobrunn
> > > >> Germany
> > > >> +49 (151) 6517 6917
> > > >> mosipov@ilbers.de
> > > >> http://ilbers.de/
> > > >> Commercial register Munich, HRB 214197
> > > >> General Manager: Baurzhan Ismagulov    
> > > >     
> > > 
> > >   
> >   
>